From patchwork Thu Feb 16 05:17:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57838 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp116688wrn; Wed, 15 Feb 2023 21:19:11 -0800 (PST) X-Google-Smtp-Source: AK7set/GsziUbSqt2vBmzNefXz1bqZX8ADgBubF1nCwOVfDw//Tk+Vto/w17b/1/8Z3Hc0sd0/qH X-Received: by 2002:a17:90b:3881:b0:234:116a:e172 with SMTP id mu1-20020a17090b388100b00234116ae172mr5541469pjb.27.1676524751641; Wed, 15 Feb 2023 21:19:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676524751; cv=none; d=google.com; s=arc-20160816; b=fw6ptXB3a8S3G2j3XcfqnDQXpbfNqaIm3UHriSV7+ZHfJv/Pb95lfc4RJWhRHKex8u ozCZCW7/u1KS27jJ4Bjj943DNGgVNCPd1mIvuK8WT6/EyPFKwX31Tt/eFSplCdLZWvyU Dz9NdmwSjURqONBqZpSLTuXmd/V12rgC7WOeLTVPWpvWvmoBi6OR4hTzymx8p94uUVHd KkH9Kn3CsV7+NUXJfX4TUc83wJTcuNnTLJZQPr1eVWgBvKGMOfsQyISegWqMicrB/V98 XqZxrLhLitY6JwI+usLixrbcEG1kWPPaRY6pKuxM3fnPBYjkc2Nv9NsnWnle3EYlurbs jyJA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=0nC1+Z5xJ7W1lTs/T0LqT1g+dKiQp9Lc6m1pbhrJE6M=; b=uSSPNMIekW2Hj4lO6xCVYspJTh/yFErgkRH+qghmLvQRNQtzNxm3GdtXXMCmaYDg88 8GIRUT2JqEe+kUJVoI/wPOJnnQHELR+ez6Baxv+ZwNtJOXyc+mPIxmH+V195d3mEtq9B EPF2MW6lhDQBrGSh16DoN/QxHDBNEo6Xm55gXHU6G3onvDVHXlgBscYIQsvLGRsH1+oV 4o2num0LOmjzcRinL0+rpl+CaUa+DJ8Y8HlA8SeNHwt0f9lBmOUYzTxbSMS7AV8sEC8Q W2xPIWmx8Mh3Eswyox/GXm2/z6GBF14f02S6xTuPJDI8kMcs1CvPI5hO2FwqhVL4NuS0 5j/A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=DtrBb0OE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id p11-20020a65490b000000b004fab4df6dfdsi673820pgs.369.2023.02.15.21.18.58; Wed, 15 Feb 2023 21:19:11 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=DtrBb0OE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229568AbjBPFSF (ORCPT + 99 others); Thu, 16 Feb 2023 00:18:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44612 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229516AbjBPFSA (ORCPT ); Thu, 16 Feb 2023 00:18:00 -0500 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 90BAF13DD6 for ; Wed, 15 Feb 2023 21:17:57 -0800 (PST) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-50e79ffba49so8474537b3.9 for ; Wed, 15 Feb 2023 21:17:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=0nC1+Z5xJ7W1lTs/T0LqT1g+dKiQp9Lc6m1pbhrJE6M=; b=DtrBb0OEqSibZGKM+tGrekgia+czdJCmNvwxB9nTfvwGTIHXKlm6TS9Q1D5NfIR4hg c3apVnOjZnfZCRE1YFzTHLthbM0S/IVW1BccnLNsJf21OgviYfhBVHNRB8wtAupWJwTS 1TEYfIbKZCSoMpt+Js0TzB9iPZbPOfnvL//puIai0IRqYdE7nHNdWFvMQyJ5Dt5csXIh 43aWlLJbXNtNaZid0iRHPyqJRyFRPoBCCZ5nKLzs/n4kbMJKA8/P4QRR3m4y665oyagy kgaLTr7SQL3GQMCbK2ePGMttOB2X0C/PFJbSA9edFZkog1VQvtGlD+HG94g+afV+vkdw M8NA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0nC1+Z5xJ7W1lTs/T0LqT1g+dKiQp9Lc6m1pbhrJE6M=; b=ciuTs2y3Dend0HO2GHItrgf4S0BR59O2hkff5KfbdNACW7+0Xo06RXu1PG/1S6Q7Zr kLwrX3CUDsSfQqE/ncqosKBfmLCIm+oSdw52jM187UHL3kUQMp9aIVPPyFdgNEyozrGx Zk5czzBFQvMIiMZ1VdgpdaVtyOJS9j6ymMgv8r1ig0sFDqfrM0DmZjKYzEmyg7ohZtvD 0HO3anGJKVxgeQpj8KCqTmTm1Yor9Aa+k3Ia/sy0Zh2MryWF6ZI2MoZjmzV3nfgyktOc 4dP8ghq2izOcG5S7/rFDDm3sTM6Imb9w6uu8F9GGO0mTIg/47ll7rvdsquWEdQOUFDNE PxgQ== X-Gm-Message-State: AO0yUKX0acTU6DI94CZR04tkVehrXQlUs7EqKTZClkpUDNhH445f++/j Ptuq6BJpLE11QbQ2yevWBoS+44zbUho= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a0d:d88c:0:b0:533:86d8:a551 with SMTP id a134-20020a0dd88c000000b0053386d8a551mr60897ywe.11.1676524676787; Wed, 15 Feb 2023 21:17:56 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:16 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-2-surenb@google.com> Subject: [PATCH v3 01/35] maple_tree: Be more cautious about dead nodes From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Liam Howlett , Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757963617698036841?= X-GMAIL-MSGID: =?utf-8?q?1757963617698036841?= From: Liam Howlett ma_pivots() and ma_data_end() may be called with a dead node. Ensure to that the node isn't dead before using the returned values. This is necessary for RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 52 +++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 43 insertions(+), 9 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 646297cae5d1..cc356b8369ad 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -544,6 +544,7 @@ static inline bool ma_dead_node(const struct maple_node *node) return (parent == node); } + /* * mte_dead_node() - check if the @enode is dead. * @enode: The encoded maple node @@ -625,6 +626,8 @@ static inline unsigned int mas_alloc_req(const struct ma_state *mas) * @node - the maple node * @type - the node type * + * In the event of a dead node, this array may be %NULL + * * Return: A pointer to the maple node pivots */ static inline unsigned long *ma_pivots(struct maple_node *node, @@ -1096,8 +1099,11 @@ static int mas_ascend(struct ma_state *mas) a_type = mas_parent_enum(mas, p_enode); a_node = mte_parent(p_enode); a_slot = mte_parent_slot(p_enode); - pivots = ma_pivots(a_node, a_type); a_enode = mt_mk_node(a_node, a_type); + pivots = ma_pivots(a_node, a_type); + + if (unlikely(ma_dead_node(a_node))) + return 1; if (!set_min && a_slot) { set_min = true; @@ -1401,6 +1407,9 @@ static inline unsigned char ma_data_end(struct maple_node *node, { unsigned char offset; + if (!pivots) + return 0; + if (type == maple_arange_64) return ma_meta_end(node, type); @@ -1436,6 +1445,9 @@ static inline unsigned char mas_data_end(struct ma_state *mas) return ma_meta_end(node, type); pivots = ma_pivots(node, type); + if (unlikely(ma_dead_node(node))) + return 0; + offset = mt_pivots[type] - 1; if (likely(!pivots[offset])) return ma_meta_end(node, type); @@ -4505,6 +4517,9 @@ static inline int mas_prev_node(struct ma_state *mas, unsigned long min) node = mas_mn(mas); slots = ma_slots(node, mt); pivots = ma_pivots(node, mt); + if (unlikely(ma_dead_node(node))) + return 1; + mas->max = pivots[offset]; if (offset) mas->min = pivots[offset - 1] + 1; @@ -4526,6 +4541,9 @@ static inline int mas_prev_node(struct ma_state *mas, unsigned long min) slots = ma_slots(node, mt); pivots = ma_pivots(node, mt); offset = ma_data_end(node, mt, pivots, mas->max); + if (unlikely(ma_dead_node(node))) + return 1; + if (offset) mas->min = pivots[offset - 1] + 1; @@ -4574,6 +4592,7 @@ static inline int mas_next_node(struct ma_state *mas, struct maple_node *node, struct maple_enode *enode; int level = 0; unsigned char offset; + unsigned char node_end; enum maple_type mt; void __rcu **slots; @@ -4597,7 +4616,11 @@ static inline int mas_next_node(struct ma_state *mas, struct maple_node *node, node = mas_mn(mas); mt = mte_node_type(mas->node); pivots = ma_pivots(node, mt); - } while (unlikely(offset == ma_data_end(node, mt, pivots, mas->max))); + node_end = ma_data_end(node, mt, pivots, mas->max); + if (unlikely(ma_dead_node(node))) + return 1; + + } while (unlikely(offset == node_end)); slots = ma_slots(node, mt); pivot = mas_safe_pivot(mas, pivots, ++offset, mt); @@ -4613,6 +4636,9 @@ static inline int mas_next_node(struct ma_state *mas, struct maple_node *node, mt = mte_node_type(mas->node); slots = ma_slots(node, mt); pivots = ma_pivots(node, mt); + if (unlikely(ma_dead_node(node))) + return 1; + offset = 0; pivot = pivots[0]; } @@ -4659,11 +4685,14 @@ static inline void *mas_next_nentry(struct ma_state *mas, return NULL; } - pivots = ma_pivots(node, type); slots = ma_slots(node, type); - mas->index = mas_safe_min(mas, pivots, mas->offset); + pivots = ma_pivots(node, type); count = ma_data_end(node, type, pivots, mas->max); - if (ma_dead_node(node)) + if (unlikely(ma_dead_node(node))) + return NULL; + + mas->index = mas_safe_min(mas, pivots, mas->offset); + if (unlikely(ma_dead_node(node))) return NULL; if (mas->index > max) @@ -4817,6 +4846,11 @@ static inline void *mas_prev_nentry(struct ma_state *mas, unsigned long limit, slots = ma_slots(mn, mt); pivots = ma_pivots(mn, mt); + if (unlikely(ma_dead_node(mn))) { + mas_rewalk(mas, index); + goto retry; + } + if (offset == mt_pivots[mt]) pivot = mas->max; else @@ -6631,11 +6665,11 @@ static inline void *mas_first_entry(struct ma_state *mas, struct maple_node *mn, while (likely(!ma_is_leaf(mt))) { MT_BUG_ON(mas->tree, mte_dead_node(mas->node)); slots = ma_slots(mn, mt); - pivots = ma_pivots(mn, mt); - max = pivots[0]; entry = mas_slot(mas, slots, 0); + pivots = ma_pivots(mn, mt); if (unlikely(ma_dead_node(mn))) return NULL; + max = pivots[0]; mas->node = entry; mn = mas_mn(mas); mt = mte_node_type(mas->node); @@ -6655,13 +6689,13 @@ static inline void *mas_first_entry(struct ma_state *mas, struct maple_node *mn, if (likely(entry)) return entry; - pivots = ma_pivots(mn, mt); - mas->index = pivots[0] + 1; mas->offset = 1; entry = mas_slot(mas, slots, 1); + pivots = ma_pivots(mn, mt); if (unlikely(ma_dead_node(mn))) return NULL; + mas->index = pivots[0] + 1; if (mas->index > limit) goto none; From patchwork Thu Feb 16 05:17:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57839 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp116708wrn; Wed, 15 Feb 2023 21:19:15 -0800 (PST) X-Google-Smtp-Source: AK7set+wEGd6DlEcubTcIlLN5KmxFLUB+cmUGJ6YczoYZzanSLgP3resgAxPp2NDwllWuBpW5rgd X-Received: by 2002:a17:90b:1b48:b0:230:7a31:b9ac with SMTP id nv8-20020a17090b1b4800b002307a31b9acmr5634093pjb.48.1676524754720; Wed, 15 Feb 2023 21:19:14 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676524754; cv=none; d=google.com; s=arc-20160816; b=fRCALGH6z4EfBQpJKhHtYmKDaxxH/gLbwv/Y8ot+fC1fyPcEXnq0PP4S4KJ4zRCGci r5IIj/1By+tVpj2vCZWJN3WaVLmN+I/X3GszjqTu5UoWznKwNYB0BSNj40263NPZIfyR mwg7C9k0R0pEPp8/UwRHBJgEZWhMaWXpceVY4lsHpYZLFYZh0h6Gl5JnSac3xLmPNlSF 8FjAnqorDcM2YFrwABH++ZeXh0LAPU+pj3pXXo6aITPOyIdCjA17KBfeIyfAqS2c0sak DU8AXkwQedZBUoRGLH7pMz5pcaShvx6LWwrQrG9MbOsaTyzHPHZu6E0nEztlgz5irnjQ 1MUA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=eiC+Iexfup+RTroKLCxxsKt1sRroZAPvjqioHV2wMiw=; b=Lx7fDx4laA+WCpKXhTtWaSns58AWgtFSmn/GAtjnDiYVCusWwELQ00S2wjBbo52Z11 RLuMlSm1gPy5m9sCRzBxdlJ1tyIYBjt/Uv0Oqdr2N+kfkIrI9q05/GHZTQKdfj54VByt +YDGUQXfMetNlyKT7sCtq372uQD6oWn3VxjDkTY5+u40ck3EukLLhLsnlBtUmb3Df+Xu FXayd3/c4MnennIV3OTcVv6xldC7i4uKKfofHxz3wFY3H3gCC+NY1DqpZ8it2efyFfuu yiEqnrNSheM1knEHUzF+xCG6kVL0fJl9YaBL4DTsEuIfg5XRRpdJ024T3PxG6mCspFRX LOvQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=gSbKQNPX; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u2-20020a17090add4200b00230c717f66esi3709974pjv.172.2023.02.15.21.19.02; Wed, 15 Feb 2023 21:19:14 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=gSbKQNPX; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229591AbjBPFSK (ORCPT + 99 others); Thu, 16 Feb 2023 00:18:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44620 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229528AbjBPFSC (ORCPT ); Thu, 16 Feb 2023 00:18:02 -0500 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B405A1BACE for ; Wed, 15 Feb 2023 21:18:00 -0800 (PST) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-52ee93d7863so8353547b3.18 for ; Wed, 15 Feb 2023 21:18:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=eiC+Iexfup+RTroKLCxxsKt1sRroZAPvjqioHV2wMiw=; b=gSbKQNPX+j8W3tzxYZsRNAblkFTxWANtMyo2LSqGNkakVHWziV6JTyqlohrqDf6nLO 7F5N7WpIKnXsbFPEflJy25rhat2iqZIJRLZvV94iATncucILtc8zFlPzaxOQpEoApKoJ RWEtB6lTCeOcIkp8qbIxk4V683W1L3zeK3vTsHv9Yla4n9gJP1xuJQs99Ms5w5HF3olN Kli7prX+XczaCsAJ+8hVggWwSYtxp34C+A6/4yuDSchSQcAT4VW0lygmuaxa708qKR/5 zV/LK3WvWxvtgplCG5tRDu/7iPKMsNWXdzFiEgVvRQ3tE0ZzhzdCi73gDraIVPZ+cD1H 1RBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=eiC+Iexfup+RTroKLCxxsKt1sRroZAPvjqioHV2wMiw=; b=HtBka/XnXeDgIdXq2ejdDwuy3vY3xy3ojb8+iyaKytdNa/2lc2izy2fcHN4lzJv2cg c4W+fmKeRC3bXmeGMgxQY/YAKJNx14LEamTrR7icC+gau94X/NyfyGl/HpNVxb4P4saE pO5j9VnaED+mDlRsAQTLOf5m5gamElsIps6q+u0DgIVdBtpa9wdGV77s/YrnqPis5ZnP nm+yHrcG5Nu0fXHs21Gd/qV/pMN7Puv1FzPP8KlZKntuKoXreDFpKN9RCMkyk6gq/fQs xxS/+lwvIEu5pGFdAlApZnNwjGveYSE+GnPkMi2FH7cM/sZdOA9dWuVxrH4te5ZnjaOv dEWQ== X-Gm-Message-State: AO0yUKV9jFICdSO6w4epQYDcm/Ljw362gHscDjCDDbBgZTxRL0MYtcOU PwUaHDALWeBoSP+dM8iXNwUrB47rO5U= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a05:6902:10f:b0:855:fa17:4f5f with SMTP id o15-20020a056902010f00b00855fa174f5fmr78ybh.1.1676524678798; Wed, 15 Feb 2023 21:17:58 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:17 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-3-surenb@google.com> Subject: [PATCH v3 02/35] maple_tree: Detect dead nodes in mas_start() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Liam Howlett , Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757963621334995961?= X-GMAIL-MSGID: =?utf-8?q?1757963621334995961?= From: Liam Howlett When initially starting a search, the root node may already be in the process of being replaced in RCU mode. Detect and restart the walk if this is the case. This is necessary for RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index cc356b8369ad..089cd76ec379 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -1360,12 +1360,16 @@ static inline struct maple_enode *mas_start(struct ma_state *mas) mas->max = ULONG_MAX; mas->depth = 0; +retry: root = mas_root(mas); /* Tree with nodes */ if (likely(xa_is_node(root))) { mas->depth = 1; mas->node = mte_safe_root(root); mas->offset = 0; + if (mte_dead_node(mas->node)) + goto retry; + return NULL; } From patchwork Thu Feb 16 05:17:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57844 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp116880wrn; Wed, 15 Feb 2023 21:19:44 -0800 (PST) X-Google-Smtp-Source: AK7set/Ls2J5cptx3FaqHMmTsp1zi3T6xKH2jYdBq+VRFON9UwTfTA6FabONpCMM1PaP3wOhL37m X-Received: by 2002:a17:90b:3ec8:b0:233:d288:cb0f with SMTP id rm8-20020a17090b3ec800b00233d288cb0fmr5472886pjb.30.1676524784693; Wed, 15 Feb 2023 21:19:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676524784; cv=none; d=google.com; s=arc-20160816; b=ft2GaQ5sueUOm1NbYbuSYbgBfezjGEZMHCL2LtUGfckX/F06RcDcpF/wdAF/x7azHu o6k11i+lWbxpKx8GWmkvI4nFJA75vMN7qY3MmY6MvI7TPVDsfqCyld9uZ2LzfSzM4pst u0KygZSHrpJmWfNfBxx81gz+mQNYO46dKJdJBGaHYI2Ms9kCAZu7KFQzKsUY2QHW1L+x //Sb8IJygaIxJU6nmlRX/2QQnFFmuEbmuWOB1qBGEu7HT8vYnwTMuRlGifeTgAfW2Rn8 agywJv4v111mmG9Z0NXEnGX9YCSmPvN4A4s09ib8Z22nth1x5arb6gfsVfkBJUoNbo63 Uulw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=UHRaAOWRO45il7vCWYolP2vMSHNJY77rtpcEDPLreFo=; b=OLfaBsKQ2a5DRCm8vN2zyoqUtCQCv6EQuNsGd1xP+Tr6g3ac6E9+otAaAlqcBY7Dtm fFa0GNTPQpQ1J4JWCqx0ymASQ04P+2AkTXr15fJlk+8/2wgE2hrj5LaNyTYpVlAAyUDg 7rCIzzLRhIfozJgPZjvLy3ra60FFC61gUgYg3u2yQmQUZAOsdAz9U3EC7icZmyNDOjHq 10hvC9DMHaOyiTJmdfQZhxWVhOgP1rogJdd+637sO+8C9C8bvgaAEqWn7YDCoUJBy7Xg 5kiZqyhTpf6vujIRhrcqCcFksVaR3MZmnxE2/+oGU/zfTcwOM/YIuNKRjzWZBd09/9se rwHQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=JaAJMQvk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u17-20020a656711000000b00477fbd59895si606866pgf.852.2023.02.15.21.19.17; Wed, 15 Feb 2023 21:19:44 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=JaAJMQvk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229600AbjBPFSQ (ORCPT + 99 others); Thu, 16 Feb 2023 00:18:16 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44694 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229564AbjBPFSE (ORCPT ); Thu, 16 Feb 2023 00:18:04 -0500 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 65E1F1F5C0 for ; Wed, 15 Feb 2023 21:18:02 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-4cddba76f55so8292917b3.23 for ; Wed, 15 Feb 2023 21:18:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=UHRaAOWRO45il7vCWYolP2vMSHNJY77rtpcEDPLreFo=; b=JaAJMQvkQYkAsRG/T0d+zxwkrl7mq+q9FZsokrXfiBXR+R1ZXxx2vVNR2lMSqf4OC/ WN5L5m6K8l41JtRKI2HZbVfjAY+cjnkF1BJGHx4Vp8UHSVK5UGOhTfSvkmRDjsxq+IrD 02V2mM9EdjYNQiXsOwUhDaiCYn5z8ljJQUJeFnF2XY5CF0WF7YFz2opQhOGyVfb59slI zMkZhBxatB6Jt2s9icMmk1QAWL7dqweRI7j0fEknZ3XaTRcFBLrS6UTE7S7Q4BJqI8Vo VXsm9/T42jqn+e5iri473tsxrbcsgusYZ9oNJzgxHuxj6F/D7b6ACgioO/upEa2Rro7t uyJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UHRaAOWRO45il7vCWYolP2vMSHNJY77rtpcEDPLreFo=; b=SOijBJvIgOce2EFpNxMyERObgQe9m95TBLQ3pu3qLQXq4wpVCekgjTIMl8Ms850+g2 4lWxnk5yS0+fjV3e9810Yd6Ts7tpOHDuEMgNqrmIx+lFTZ8sawLVux8voJaE8KKrNRVv KWseAChGtAGOz+ulNKYCparpik2ga98nYeZun9wqAlH4ZG7ZhZkiIKlOwT2MaeE2pT0d W2qcMk6JV59pd9oh73knnPj5uPksFNeZ2wnURbjui5B+GfKEbwaVzdrwhOoz6AJDtNlf Msx5YQZhA1WhsgZJnZgOhCi0+ntzImMpesG5a+ulpxa8Yy5l8SUC05WJVk6lvHGLgZcz ACBg== X-Gm-Message-State: AO0yUKUrqKRPpC21ArbpoqtVF4/DpUAZPkeRTxKyrKoWGGEZV1z9JRmQ J/s56K0pz8c0E7mGydiuxLiuoBBv/Q4= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a05:6902:10f:b0:855:fa17:4f5f with SMTP id o15-20020a056902010f00b00855fa174f5fmr80ybh.1.1676524681354; Wed, 15 Feb 2023 21:18:01 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:18 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-4-surenb@google.com> Subject: [PATCH v3 03/35] maple_tree: Fix freeing of nodes in rcu mode From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Liam Howlett , Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757963652906478247?= X-GMAIL-MSGID: =?utf-8?q?1757963652906478247?= From: Liam Howlett The walk to destroy the nodes was not always setting the node type and would result in a destroy method potentially using the values as nodes. Avoid this by setting the correct node types. This is necessary for the RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 73 ++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 62 insertions(+), 11 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 089cd76ec379..44d6ce30b28e 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -902,6 +902,44 @@ static inline void ma_set_meta(struct maple_node *mn, enum maple_type mt, meta->end = end; } +/* + * mas_clear_meta() - clear the metadata information of a node, if it exists + * @mas: The maple state + * @mn: The maple node + * @mt: The maple node type + * @offset: The offset of the highest sub-gap in this node. + * @end: The end of the data in this node. + */ +static inline void mas_clear_meta(struct ma_state *mas, struct maple_node *mn, + enum maple_type mt) +{ + struct maple_metadata *meta; + unsigned long *pivots; + void __rcu **slots; + void *next; + + switch (mt) { + case maple_range_64: + pivots = mn->mr64.pivot; + if (unlikely(pivots[MAPLE_RANGE64_SLOTS - 2])) { + slots = mn->mr64.slot; + next = mas_slot_locked(mas, slots, + MAPLE_RANGE64_SLOTS - 1); + if (unlikely((mte_to_node(next) && mte_node_type(next)))) + return; /* The last slot is a node, no metadata */ + } + fallthrough; + case maple_arange_64: + meta = ma_meta(mn, mt); + break; + default: + return; + } + + meta->gap = 0; + meta->end = 0; +} + /* * ma_meta_end() - Get the data end of a node from the metadata * @mn: The maple node @@ -5455,20 +5493,22 @@ static inline int mas_rev_alloc(struct ma_state *mas, unsigned long min, * mas_dead_leaves() - Mark all leaves of a node as dead. * @mas: The maple state * @slots: Pointer to the slot array + * @type: The maple node type * * Must hold the write lock. * * Return: The number of leaves marked as dead. */ static inline -unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots) +unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots, + enum maple_type mt) { struct maple_node *node; enum maple_type type; void *entry; int offset; - for (offset = 0; offset < mt_slot_count(mas->node); offset++) { + for (offset = 0; offset < mt_slots[mt]; offset++) { entry = mas_slot_locked(mas, slots, offset); type = mte_node_type(entry); node = mte_to_node(entry); @@ -5487,14 +5527,13 @@ unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots) static void __rcu **mas_dead_walk(struct ma_state *mas, unsigned char offset) { - struct maple_node *node, *next; + struct maple_node *next; void __rcu **slots = NULL; next = mas_mn(mas); do { - mas->node = ma_enode_ptr(next); - node = mas_mn(mas); - slots = ma_slots(node, node->type); + mas->node = mt_mk_node(next, next->type); + slots = ma_slots(next, next->type); next = mas_slot_locked(mas, slots, offset); offset = 0; } while (!ma_is_leaf(next->type)); @@ -5558,11 +5597,14 @@ static inline void __rcu **mas_destroy_descend(struct ma_state *mas, node = mas_mn(mas); slots = ma_slots(node, mte_node_type(mas->node)); next = mas_slot_locked(mas, slots, 0); - if ((mte_dead_node(next))) + if ((mte_dead_node(next))) { + mte_to_node(next)->type = mte_node_type(next); next = mas_slot_locked(mas, slots, 1); + } mte_set_node_dead(mas->node); node->type = mte_node_type(mas->node); + mas_clear_meta(mas, node, node->type); node->piv_parent = prev; node->parent_slot = offset; offset = 0; @@ -5582,13 +5624,18 @@ static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, MA_STATE(mas, &mt, 0, 0); - if (mte_is_leaf(enode)) + mas.node = enode; + if (mte_is_leaf(enode)) { + node->type = mte_node_type(enode); goto free_leaf; + } + ma_flags &= ~MT_FLAGS_LOCK_MASK; mt_init_flags(&mt, ma_flags); mas_lock(&mas); - mas.node = start = enode; + mte_to_node(enode)->ma_flags = ma_flags; + start = enode; slots = mas_destroy_descend(&mas, start, 0); node = mas_mn(&mas); do { @@ -5596,7 +5643,8 @@ static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, unsigned char offset; struct maple_enode *parent, *tmp; - node->slot_len = mas_dead_leaves(&mas, slots); + node->type = mte_node_type(mas.node); + node->slot_len = mas_dead_leaves(&mas, slots, node->type); if (free) mt_free_bulk(node->slot_len, slots); offset = node->parent_slot + 1; @@ -5620,7 +5668,8 @@ static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, } while (start != mas.node); node = mas_mn(&mas); - node->slot_len = mas_dead_leaves(&mas, slots); + node->type = mte_node_type(mas.node); + node->slot_len = mas_dead_leaves(&mas, slots, node->type); if (free) mt_free_bulk(node->slot_len, slots); @@ -5630,6 +5679,8 @@ static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, free_leaf: if (free) mt_free_rcu(&node->rcu); + else + mas_clear_meta(&mas, node, node->type); } /* From patchwork Thu Feb 16 05:17:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57840 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp116746wrn; Wed, 15 Feb 2023 21:19:21 -0800 (PST) X-Google-Smtp-Source: AK7set/EU3Y7USQM6raO6gEFyu7Fz4+NxvR9Ak+dCnQxNrgohlQ9TEZnPdX13ja5VlbsygHWiqta X-Received: by 2002:a17:902:fad0:b0:19a:c65d:f91 with SMTP id ld16-20020a170902fad000b0019ac65d0f91mr4657676plb.66.1676524761194; Wed, 15 Feb 2023 21:19:21 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676524761; cv=none; d=google.com; s=arc-20160816; b=SLcM0joksv8uwYZLj+1BnE9tLY4f7BCT/AZLUBr6pUpF4M/K2Spk66m0F9a9Fy4XuH RVKh/sYg8VnCx+Tve3umZKr/vXqooK1UPdBbOuk5pCuaRB6uiPfV2W79b5bB56vHlFE3 BDgFpI4oZ7rs/u0OUtrA3tS2/Er/OSehtIhBXtNnUDqld4FvuXQqztEnIlaB2/RCd1Te odegLeDxGXLf+nhCzr5abbcXAMiyUshLDR9JJIoUDAptyzgrwiYP39fT8qI5UWk0dGoS nR4B6GtTIN5dk2aj8dwk2DDJXw5s4wR6rpB8uJqbBo/DXyV/SBsSi4cOYP/aOlsRTv2z 4ldQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=bYTu7nw04Y/eGAN3i9eCzY/Ke38OAVixcrHXlu+jNFw=; b=0nJcBK/FXJZRAqrefpmnldX3X4qN1JDH9YzUCyNIYCvU9E7WOJohBjgbatp8DYEyQS xZkMKsobZsquXV3wrz5j65qCdvai/30t8GRuLdYEuFRYYzK7p8khGbw/tPnZVvh5O/1t Cm3icwCdiNZzJS0PoNbX3/STlfJqWMVKd5CDAYuvpnDIy/fpAW10zG7r/na50FGmOg3K b50aFcsySHBE/2Zl8KGRCoP3G2BxoqhogSoZQmK8wJALW46b3kS87Fa/wkz3nPdGAyK7 BoLzQsJz83TVwxVWuMqWqo52HKf4/cNzMexjYTrN0IDVdq6GwgcZetICbTZ7qEJaCDZ2 qxEQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=WNV0WuL9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i10-20020a170902c94a00b001990992d68fsi463277pla.518.2023.02.15.21.19.08; Wed, 15 Feb 2023 21:19:21 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=WNV0WuL9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229607AbjBPFSU (ORCPT + 99 others); Thu, 16 Feb 2023 00:18:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44750 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229571AbjBPFSF (ORCPT ); Thu, 16 Feb 2023 00:18:05 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0CBF41BACE for ; Wed, 15 Feb 2023 21:18:05 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id u2-20020a25ab02000000b0090f2c84a6a4so856822ybi.13 for ; Wed, 15 Feb 2023 21:18:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=bYTu7nw04Y/eGAN3i9eCzY/Ke38OAVixcrHXlu+jNFw=; b=WNV0WuL9vmXLIvOmuVEVNJHnS2Qfi+AtiXnEC8qLvBZk4nQfRysiQv+n6V2TYm/oOI 8PVrbtoCrNJAR827RYRUq+srgPH7ettT9CkewSYs2c20qjx3DSR6OCNLU8I9VtlWJ1vK BTnTnTmkn3I2d2LJu6019qpcNY3Rtyhwbcu+BTERQU4CpKL9XQ+m6KDQOlEen5oxaVLd mpn44+vsVlTTH8ejjgOPKl/5T26Yr75D1Q+Pk5P8P+e6MTdUqg4koF2LdV4PYx+3j32Q lvmdVkZra/AZy5zB6Ynbz9xE0KdiJ+yp/dRpPvAGFlnQDPvuVze/cOT0f1MBvOftG1xW MgmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=bYTu7nw04Y/eGAN3i9eCzY/Ke38OAVixcrHXlu+jNFw=; b=Ania1ZLP9q/F4RNkOFXcdEJbZv56ecwm5AGJQ9BhMAx+judzw6GBWo1HF92JV9TOB4 bjILJPSGNkUO7RbZir+GuL9z6gheQem6474yDRPwhbbaDTvHToXpHlX7eQflngYsMFHo XnDDNy2/dZAViLuSu38r5WcJh5Ps4LQY2KXaDOkscenI5vFz8RszXUrGF+cDoLm32rs2 dvrjjr9M9kU34QdZmqTlH9OtRkQCMoI6GbV90CoSPJV25R/UdX//B5ikj55lEXfRdtWU gZgSu81xF/ByoTH9sRWxPFB7FBKiR8BLCTCmbxgIo3OLs4iI+mdoZO6MrJu2uPoXmib2 Usvg== X-Gm-Message-State: AO0yUKVXkf08Q0JCIBm+kndP+PICAZpCLDEP/kIwL2fle1gCqMH2c/dP 0O/EA20CYPlT0fEyvO9HxerCba7I4NY= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a05:6902:1024:b0:8fc:686c:cf87 with SMTP id x4-20020a056902102400b008fc686ccf87mr15ybt.4.1676524683783; Wed, 15 Feb 2023 21:18:03 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:19 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-5-surenb@google.com> Subject: [PATCH v3 04/35] maple_tree: remove extra smp_wmb() from mas_dead_leaves() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Liam Howlett , Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757963627905620510?= X-GMAIL-MSGID: =?utf-8?q?1757963627905620510?= From: Liam Howlett The call to mte_set_dead_node() before the smp_wmb() already calls smp_wmb() so this is not needed. This is an optimization for the RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 1 - 1 file changed, 1 deletion(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 44d6ce30b28e..3d5ab02f981a 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -5517,7 +5517,6 @@ unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots, break; mte_set_node_dead(entry); - smp_wmb(); /* Needed for RCU */ node->type = type; rcu_assign_pointer(slots[offset], node); } From patchwork Thu Feb 16 05:17:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57841 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp116804wrn; Wed, 15 Feb 2023 21:19:30 -0800 (PST) X-Google-Smtp-Source: AK7set/LpyNknRaMCjCn9JaZO7eyFZ68zdRsSgeOd2Y+9MO2jYye+99o0EI5KTWr9cbYBhbO2DwQ X-Received: by 2002:a17:902:e543:b0:19a:89d4:802f with SMTP id n3-20020a170902e54300b0019a89d4802fmr5665054plf.57.1676524769992; Wed, 15 Feb 2023 21:19:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676524769; cv=none; d=google.com; s=arc-20160816; b=xS4+m47tYkcJMUnh1blHKuWSxq7xJGz4TEqhqMyixXOriBUIJbrVrb5IyxJRqR4RZa MWz1ljRbdegbULnBif0N3mqQFpTcCbwOwQBbw5mEGr+pDWkGax6f6Pu+G8pRLdB3Ku+Y OEpK+nWeeknNJ/ge2BRzgyO21f9yY5G/l4pvRo4umIkR+v2BZyzEh/R4LRE0QEEWYLvV GI/SfDrrXHuoWs0Z1CrCk4C2hlYVaz0xoBVFUzfVS7rxS797rSRLv7IwHLtsuLqjXf/G xjF/K4ZDVBhqRp32hbqtuR2B5gan459Gwb+fNis10+KFG6bQG2avatZMy1fnx2AF5BJj PPtA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=rDMlMLTEmJU/Y3crV2atsbCBmpKXyma0yO0OAyc8vDw=; b=RJ8+x1tYftO7YeYchmvJHng/oNYtZgyyZfhWNQ2YzRtb+gRMRgEAjXAAP+Bc1jFJQ5 5TBwHOwQAKrJLZEM1NTCInvAFpLSf7KOTnZ1041eFhIqTzgCQtP0GvGPl09mXQqtWaXu /QvphPDWlPHWLVIubmZzywwiRn+UZ9KMLJewAVi7gUHP3joEeJtxwk05JtqJCu+ffvhm CD7M4uC+3lgF3qrI1B+8XsKBVkds64fqDS2x5OHZ92VxvYMn0iBaRt4l1ZtJiRWF/xZi z/HrXWX9+y8vbpfI3TaEpdWWOGqv3sN7CpTJ7bgzvG7j8IwptCkQpkLawgs0PLS4i3dv JKbQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b="nzJ7S/CJ"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f17-20020a170902f39100b00198d701adb4si353462ple.615.2023.02.15.21.19.17; Wed, 15 Feb 2023 21:19:29 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b="nzJ7S/CJ"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229485AbjBPFSZ (ORCPT + 99 others); Thu, 16 Feb 2023 00:18:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45070 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229587AbjBPFSK (ORCPT ); Thu, 16 Feb 2023 00:18:10 -0500 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AA40A410A1 for ; Wed, 15 Feb 2023 21:18:07 -0800 (PST) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-507aac99fdfso8543797b3.11 for ; Wed, 15 Feb 2023 21:18:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=rDMlMLTEmJU/Y3crV2atsbCBmpKXyma0yO0OAyc8vDw=; b=nzJ7S/CJB/US/rZvNC3fs8J8fXfgfseovx93tCDYzLZ8vkCeR9OXHCG0j7Ki073PZN rg58gpprBwFVBX+ke7DxYuYcgkrSQGYt8YR1LUHO8qrBrJHNZzYIw0fWkCpsF9iepuD5 iMfSL7cYbKTG2WSIKdzRhuGyAIwfPFbrqz4Ug3Wxq2wbPU6P8gVi3bWvRrz9dl4kYYU2 r7gzF5YCRjQDAYOvUYbM1k4rIOQJ+5ffpqKzLjZIEvFTLYyWs4Q62tIpz4Jjx53AOh2l MDbSaQOmeWEl4DEmE5lvlD2J9UkQB4Iedo6XxOd2amw2xyG9CIFDXUp1l5UWlLjXi78n VOeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=rDMlMLTEmJU/Y3crV2atsbCBmpKXyma0yO0OAyc8vDw=; b=rRcURjHgly6C9phVmCfv/Qwn3KA2i2hsOFAed/MKybTvZZ1H5qmL+FstZVXszmjz0s kYPvlU5p1hQs9pDnu6C5oEPfdFkggH6lKVkenQN83A7Fb6qpJuihbPkpivhCpb+SsubO vxcyXiHc5XDC8gN6s7HDbfI1qrL9P3Qj8W388w3UbL9xW3CKKUoJtYDU3vwKKteLN05p 6pMCdCOTj7+AvpGdTinzZ5M0OL04Cr4bJOagOsR0eVSalILZ9XrKz6fG7ukXGEccCgoH gjXNE47x+mNYQbxhcWB4GzAdKhqJnZAk5eq9jqhiS3qPIWrOgEA0tTLZam4qnx5Jjn2T N/wA== X-Gm-Message-State: AO0yUKWy7RDm+0TL1UvYwALTkYC/7DsIlDeLUdpPxM2TGRzCzr5GpnYL M6YmM0fuL2oQGSVGHrWdRn9fhicOiHI= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a81:4309:0:b0:52e:c2c5:b75e with SMTP id q9-20020a814309000000b0052ec2c5b75emr19ywa.2.1676524686391; Wed, 15 Feb 2023 21:18:06 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:20 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-6-surenb@google.com> Subject: [PATCH v3 05/35] maple_tree: Fix write memory barrier of nodes once dead for RCU mode From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, "Liam R. Howlett" , Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757963637014702708?= X-GMAIL-MSGID: =?utf-8?q?1757963637014702708?= From: "Liam R. Howlett" During the development of the maple tree, the strategy of freeing multiple nodes changed and, in the process, the pivots were reused to store pointers to dead nodes. To ensure the readers see accurate pivots, the writers need to mark the nodes as dead and call smp_wmb() to ensure any readers can identify the node as dead before using the pivot values. There were two places where the old method of marking the node as dead without smp_wmb() were being used, which resulted in RCU readers seeing the wrong pivot value before seeing the node was dead. Fix this race condition by using mte_set_node_dead() which has the smp_wmb() call to ensure the race is closed. Add a WARN_ON() to the ma_free_rcu() call to ensure all nodes being freed are marked as dead to ensure there are no other call paths besides the two updated paths. This is necessary for the RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 7 +++++-- tools/testing/radix-tree/maple.c | 16 ++++++++++++++++ 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 3d5ab02f981a..6b6eddadd9d2 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -185,7 +185,7 @@ static void mt_free_rcu(struct rcu_head *head) */ static void ma_free_rcu(struct maple_node *node) { - node->parent = ma_parent_ptr(node); + WARN_ON(node->parent != ma_parent_ptr(node)); call_rcu(&node->rcu, mt_free_rcu); } @@ -1778,8 +1778,10 @@ static inline void mas_replace(struct ma_state *mas, bool advanced) rcu_assign_pointer(slots[offset], mas->node); } - if (!advanced) + if (!advanced) { + mte_set_node_dead(old_enode); mas_free(mas, old_enode); + } } /* @@ -4218,6 +4220,7 @@ static inline bool mas_wr_node_store(struct ma_wr_state *wr_mas) done: mas_leaf_set_meta(mas, newnode, dst_pivots, maple_leaf_64, new_end); if (in_rcu) { + mte_set_node_dead(mas->node); mas->node = mt_mk_node(newnode, wr_mas->type); mas_replace(mas, false); } else { diff --git a/tools/testing/radix-tree/maple.c b/tools/testing/radix-tree/maple.c index 958ee9bdb316..4c89ff333f6f 100644 --- a/tools/testing/radix-tree/maple.c +++ b/tools/testing/radix-tree/maple.c @@ -108,6 +108,7 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, mn->slot[1] != NULL); MT_BUG_ON(mt, mas_allocated(&mas) != 0); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mas.node = MAS_START; mas_nomem(&mas, GFP_KERNEL); @@ -160,6 +161,7 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, mas_allocated(&mas) != i); MT_BUG_ON(mt, !mn); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } @@ -192,6 +194,7 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, not_empty(mn)); MT_BUG_ON(mt, mas_allocated(&mas) != i - 1); MT_BUG_ON(mt, !mn); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } @@ -210,6 +213,7 @@ static noinline void check_new_node(struct maple_tree *mt) mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); MT_BUG_ON(mt, mas_allocated(&mas) != j - 1); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } MT_BUG_ON(mt, mas_allocated(&mas) != 0); @@ -233,6 +237,7 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, mas_allocated(&mas) != i - j); mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); MT_BUG_ON(mt, mas_allocated(&mas) != i - j - 1); } @@ -269,6 +274,7 @@ static noinline void check_new_node(struct maple_tree *mt) mn = mas_pop_node(&mas); /* get the next node. */ MT_BUG_ON(mt, mn == NULL); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } MT_BUG_ON(mt, mas_allocated(&mas) != 0); @@ -294,6 +300,7 @@ static noinline void check_new_node(struct maple_tree *mt) mn = mas_pop_node(&mas2); /* get the next node. */ MT_BUG_ON(mt, mn == NULL); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } MT_BUG_ON(mt, mas_allocated(&mas2) != 0); @@ -334,10 +341,12 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 2); mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); for (i = 1; i <= MAPLE_ALLOC_SLOTS + 1; i++) { mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } MT_BUG_ON(mt, mas_allocated(&mas) != 0); @@ -375,6 +384,7 @@ static noinline void check_new_node(struct maple_tree *mt) mas_node_count(&mas, i); /* Request */ mas_nomem(&mas, GFP_KERNEL); /* Fill request */ mn = mas_pop_node(&mas); /* get the next node. */ + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mas_destroy(&mas); @@ -382,10 +392,13 @@ static noinline void check_new_node(struct maple_tree *mt) mas_node_count(&mas, i); /* Request */ mas_nomem(&mas, GFP_KERNEL); /* Fill request */ mn = mas_pop_node(&mas); /* get the next node. */ + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mn = mas_pop_node(&mas); /* get the next node. */ + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mn = mas_pop_node(&mas); /* get the next node. */ + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mas_destroy(&mas); } @@ -35369,6 +35382,7 @@ static noinline void check_prealloc(struct maple_tree *mt) MT_BUG_ON(mt, allocated != 1 + height * 3); mn = mas_pop_node(&mas); MT_BUG_ON(mt, mas_allocated(&mas) != allocated - 1); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); MT_BUG_ON(mt, mas_preallocate(&mas, GFP_KERNEL) != 0); mas_destroy(&mas); @@ -35386,6 +35400,7 @@ static noinline void check_prealloc(struct maple_tree *mt) mas_destroy(&mas); allocated = mas_allocated(&mas); MT_BUG_ON(mt, allocated != 0); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); MT_BUG_ON(mt, mas_preallocate(&mas, GFP_KERNEL) != 0); @@ -35756,6 +35771,7 @@ void farmer_tests(void) tree.ma_root = mt_mk_node(node, maple_leaf_64); mt_dump(&tree); + node->parent = ma_parent_ptr(node); ma_free_rcu(node); /* Check things that will make lockdep angry */ From patchwork Thu Feb 16 05:17:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57842 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp116820wrn; Wed, 15 Feb 2023 21:19:33 -0800 (PST) X-Google-Smtp-Source: AK7set/f/GG9ODifR0v0u8Kp6rJ0yPMZchWBmrAZfLWF1DeukHP9xQhG8PXyEPsuBauApCU1sPRG X-Received: by 2002:a05:6a20:12d6:b0:be:abb2:66ff with SMTP id v22-20020a056a2012d600b000beabb266ffmr4942027pzg.60.1676524772965; Wed, 15 Feb 2023 21:19:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676524772; cv=none; d=google.com; s=arc-20160816; b=Vp47W8ZZh76holGMFF6VUAzAb702PChCabKxS5Dtv/an5c81BJestCkgvA8laDU5cu P2a2PlrqvYElB7Adi+guwhBjvCmWmzJQmy8KCrJpQQ6LX+Gljn25kEsSdzjso+KVOsw7 gz1r4Bqar5XpjpN0hGTlLxyURmdbjBSWJFdACLD5qpvryv5JBnSXYe6lbbyYKE9h8Pt9 HreeZTy3YY+yKy1yca63278Fylc9RpEG7dOJLgTIXndLSudZqMe7LLIQTOpkBuujU66w TbNfZ4upv0yEP9AvwKxIs+EXA4z/mCtRhIvJh3m5c+rCApK90PV+O++wGdCGFhId9G+3 2t4Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=/TbDrnT4JTA4cMZmAjPRNE76LJNgcWEHHiVobOaA+Y8=; b=q1f8TlocRfBpAwx1cwmZXMfw3NjufgZQi/kKhblZcdkHXL5g0we/i+Vzu/zq5Rshar 6jkPeogBeudWDwlGGlkxCWfkOUBIPfrJzhbL6oAQkJabrgKADxXX3Q1XfDuX6cfGVY+Z HBCGDCNoDw3ssBRNfAPRhM9Lxnn97JPkAnpsXlOipils6IEc6YzDBwwgFJs6dh3gTlA1 cVgfVYUncRE04CgW/BACTb+KwqkxILMQgRai/8A4AttDyKVtptP2JiKoTOw5TRwJiCLj GkEwT8LTDMl+3XehsxlDxxwVIQ32u62E6HsZtD69he7cQzBLvd3oxgdpfhacUGBmKSsK pfow== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=WMbcZd2H; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id j191-20020a6380c8000000b004f1ce43e6b8si716611pgd.22.2023.02.15.21.19.20; Wed, 15 Feb 2023 21:19:32 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=WMbcZd2H; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229664AbjBPFSp (ORCPT + 99 others); Thu, 16 Feb 2023 00:18:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45814 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229585AbjBPFS0 (ORCPT ); Thu, 16 Feb 2023 00:18:26 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F413B42BD8 for ; Wed, 15 Feb 2023 21:18:10 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id e191-20020a2537c8000000b009433a21be0dso835758yba.19 for ; Wed, 15 Feb 2023 21:18:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/TbDrnT4JTA4cMZmAjPRNE76LJNgcWEHHiVobOaA+Y8=; b=WMbcZd2H6xEYgh8jPBjtr3c52H6Pfawu+33AP54ppxn4975bHziKOFepXARIS5jFg+ e6WwvlJoOPdxReXGpPCKc/NAAxfX4rbkO7DDbRqhMho2D2UtZ3EYOUrPrXvxo+yY0eM4 LMQxZ5khHMqm3h0sfllZRxbm/tuunDq1ItomZ38Vc5vIB77wslGrayg6KY3mbC66vptv Do13gHFViUDuIVWARk2CRzZ0n21J2khqCauKfFko2Yy6sWDAGtNFzpXO3m4fv/AYk9g9 lA127TPhorpqkZg7qKdsS4C88Qpbq0ryHT7Lx24g1uyKmp6VAfvqXa/maK1LoDRn6fAx 1DGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/TbDrnT4JTA4cMZmAjPRNE76LJNgcWEHHiVobOaA+Y8=; b=InqrOQHPGViOEwqvQJpC8dX2pF5OhmMmiO6svOisN4Zj+QldjOJbPDmP9KL4H3FvWi JAYsllW/hQpCUtUaOx/9tE1k9uhKH85vwc8mu9n3VJp/LVxfxgYbQKgazDrdDvZRGBmj ZL/9AwlvFXqpooQlGFCfBI/9gcAHRCrheGK2xDKJrAtu3P8xxHssid068b0VWitF+iWl INM7WKd85QzIbSnSWaIp9NX/1TPWdM3KzIWd9FTH4TFS5hJI4Xu/1gAkiyaczr//0O8z XKc9SPPK4xqXCQExs7iU02DRfk4e6jt2toTHGosdDKAmDnr/s8SvzgpGLNTjjgJq80Fd Dj3Q== X-Gm-Message-State: AO0yUKXaUWicOVXr0NyTbw6Xz/L02Wuc7Btdwr9dPNBXAp1XmOlyBpBb XfS3ZIR5Ulnza8JuZuFAOhlb3zL0F9U= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a25:7805:0:b0:8d7:9497:cbb3 with SMTP id t5-20020a257805000000b008d79497cbb3mr12ybc.5.1676524689233; Wed, 15 Feb 2023 21:18:09 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:21 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-7-surenb@google.com> Subject: [PATCH v3 06/35] maple_tree: Add smp_rmb() to dead node detection From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, "Liam R. Howlett" , Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757963640626989059?= X-GMAIL-MSGID: =?utf-8?q?1757963640626989059?= From: "Liam R. Howlett" Add an smp_rmb() before reading the parent pointer to ensure that anything read from the node prior to the parent pointer hasn't been reordered ahead of this check. The is necessary for RCU mode. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 6b6eddadd9d2..8ad2d1669fad 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -539,9 +539,11 @@ static inline struct maple_node *mte_parent(const struct maple_enode *enode) */ static inline bool ma_dead_node(const struct maple_node *node) { - struct maple_node *parent = (void *)((unsigned long) - node->parent & ~MAPLE_NODE_MASK); + struct maple_node *parent; + /* Do not reorder reads from the node prior to the parent check */ + smp_rmb(); + parent = (void *)((unsigned long) node->parent & ~MAPLE_NODE_MASK); return (parent == node); } @@ -556,6 +558,8 @@ static inline bool mte_dead_node(const struct maple_enode *enode) struct maple_node *parent, *node; node = mte_to_node(enode); + /* Do not reorder reads from the node prior to the parent check */ + smp_rmb(); parent = mte_parent(enode); return (parent == node); } From patchwork Thu Feb 16 05:17:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57843 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp116836wrn; Wed, 15 Feb 2023 21:19:37 -0800 (PST) X-Google-Smtp-Source: AK7set9JUo6OBkzXuuIGlVMW70h7bUAf/+94ze+6tBI46p/OlAO7TwvmIo9W8vt5uo+syD9xrKrn X-Received: by 2002:a17:90a:3e07:b0:234:2ac7:6781 with SMTP id j7-20020a17090a3e0700b002342ac76781mr5580292pjc.26.1676524776771; Wed, 15 Feb 2023 21:19:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676524776; cv=none; d=google.com; s=arc-20160816; b=qG+W/1UQDDeXkIWWb6SK2xVdYZJP8NGjIiyA1ISzaoGwS7zISvLKH/Jgfss0mJNN0w s9YEIXIc9S+2nRv9ZD6aJ7ivWR+yaBhLGHIIuq9ghRuAt4Msv2+YQkSWEd0Tg/7eCToL 4DzFqVPqdTMedEdb8H22lI2Rkt7dzD/ZQYeEtmn3//kWGVL98aU9nnVg0oxsis1Lh9Rv xBzu3qUrl9hICtn9kQeBW3cD5HifT7BwY3fKWPdkK4G3hPVOm6mYmYUyOQ8I5TTWEkZb h8Y+8Whu9jg+2+xQorai1aMqzF/ptxQNvRW606ytJ38CD0D+2CKCFWVd7PG35dRzL4Vw yJ2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=hX1DUaFSHqHCHFWD5Crp9NnpH/+9VHFxYkECoHgya1E=; b=BzTcx3T8ecXo+0qRpN33+z5tMOEfGD0VoVViaI8X0G2qJ8aOyj9K2my4J0os+V+Blb YrKiHspm9CS6yY9OKrMGXelaODxrZXRR+Ov7KA6XJ3agxUC01SCp5UKjwcjTfqYMfrz7 /87zjgrtGq4vXZWVouwpXFwLlFMbJxh7jkbBoA5UVVC4hkuBlAeeiKSn+H3r2c+MO7QQ 4pSPqTQgl2yE9RtBqkZWcyI8cqK1uSB/jF09/w34Eq03wtwVaWMUBsul3Wocl6OwTpPU Q1HBlzCC2paWmYRgYu6r5KthWkbn505d2gZXdV/FnjnB0dcMaPYWDar0rRZxF9q6dpRM Xa5w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=ZK5oAM6A; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id bf6-20020a17090b0b0600b00205da45d8a5si3812804pjb.124.2023.02.15.21.19.24; Wed, 15 Feb 2023 21:19:36 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=ZK5oAM6A; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229520AbjBPFSz (ORCPT + 99 others); Thu, 16 Feb 2023 00:18:55 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45816 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229632AbjBPFSk (ORCPT ); Thu, 16 Feb 2023 00:18:40 -0500 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C7838457D5 for ; Wed, 15 Feb 2023 21:18:12 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-52f233aff21so8594177b3.7 for ; Wed, 15 Feb 2023 21:18:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=hX1DUaFSHqHCHFWD5Crp9NnpH/+9VHFxYkECoHgya1E=; b=ZK5oAM6AcKVtxFxzHW9PhD/PdA0+Kv3bSAspA28+ZPD1m3cnM9poVYSoqhonqkIeJU SshDVskllMZgIeSGhnCidLA6ddGXSMxz+S/xoEp08YDrkRg+7RXAXCoS6u4tI7rKulCi ggnzbiGToN5cME4XPhcHuI14WBO6+4EQsud/8F6AxE1deunG+n/R+uM3wq+OYW0Yw/xw 3Mt1GtYUQdIHi9t1kC0vmn2kgnR9+BG6rYf5dtABCjAt9PnVyIFEokbi9zalyZENG7ye R6/8svYYK3xPGp282avPu/RfQWQWw9vT/0uQU8vbTuKv0KxRDKx0B9gWO6ClqS6abquo +1jg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=hX1DUaFSHqHCHFWD5Crp9NnpH/+9VHFxYkECoHgya1E=; b=4lT4jObWMCftKY/lEJ3DG3KxiWu4QIXsWes2ERJUX2WO9NZL+dXBWL5inEStzMehjA rP0MLloHtGHaC2aErKnc7NLhHSi72v61YSNJPvjRCoZNT0v+Arv3FB2Qxb+28idxZuh/ WjsrOCB+Kzwg9MIPKI0G+r3Hmu0h1MRCqHq5XgYTsdoTDnF4R/JP5Bce2b+Ws213CazK 5xqsgimu2Uj4A79/VHM+qQxsJQoj7pMxgdw+16dLgUvO/mBKeQysOeEFrgUGgGx3yjM/ kZ03oWFRviYZ+GSAM2W/0FAzdn32uPVVYgnJj5HwoU+0xvpC8etjZ4vLmFSSmf28zRaU zkNA== X-Gm-Message-State: AO0yUKUjDVVwJ2xtn31VeRoIClNGidaPmL/Ys97+B8pDFcdDfJBN8WNV JjBJe0o5g5x5oj1EgfosgxY7VeGgRaE= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a05:6902:4ea:b0:926:d02a:c18a with SMTP id w10-20020a05690204ea00b00926d02ac18amr43355ybs.233.1676524691938; Wed, 15 Feb 2023 21:18:11 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:22 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-8-surenb@google.com> Subject: [PATCH v3 07/35] maple_tree: Add RCU lock checking to rcu callback functions From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, "Liam R. Howlett" , Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757963644381219843?= X-GMAIL-MSGID: =?utf-8?q?1757963644381219843?= From: "Liam R. Howlett" Dereferencing RCU objects within the RCU callback without the RCU check has caused lockdep to complain. Fix the RCU dereferencing by using the RCU callback lock to ensure the operation is safe. Also stop creating a new lock to use for dereferencing during destruction of the tree or subtree. Instead, pass through a pointer to the tree that has the lock that is held for RCU dereferencing checking. It also does not make sense to use the maple state in the freeing scenario as the tree walk is a special case where the tree no longer has the normal encodings and parent pointers. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Reported-by: Suren Baghdasaryan Signed-off-by: Liam R. Howlett --- lib/maple_tree.c | 188 ++++++++++++++++++++++++----------------------- 1 file changed, 96 insertions(+), 92 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 8ad2d1669fad..2be86368237d 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -824,6 +824,11 @@ static inline void *mt_slot(const struct maple_tree *mt, return rcu_dereference_check(slots[offset], mt_locked(mt)); } +static inline void *mt_slot_locked(struct maple_tree *mt, void __rcu **slots, + unsigned char offset) +{ + return rcu_dereference_protected(slots[offset], mt_locked(mt)); +} /* * mas_slot_locked() - Get the slot value when holding the maple tree lock. * @mas: The maple state @@ -835,7 +840,7 @@ static inline void *mt_slot(const struct maple_tree *mt, static inline void *mas_slot_locked(struct ma_state *mas, void __rcu **slots, unsigned char offset) { - return rcu_dereference_protected(slots[offset], mt_locked(mas->tree)); + return mt_slot_locked(mas->tree, slots, offset); } /* @@ -907,34 +912,35 @@ static inline void ma_set_meta(struct maple_node *mn, enum maple_type mt, } /* - * mas_clear_meta() - clear the metadata information of a node, if it exists - * @mas: The maple state + * mt_clear_meta() - clear the metadata information of a node, if it exists + * @mt: The maple tree * @mn: The maple node - * @mt: The maple node type + * @type: The maple node type * @offset: The offset of the highest sub-gap in this node. * @end: The end of the data in this node. */ -static inline void mas_clear_meta(struct ma_state *mas, struct maple_node *mn, - enum maple_type mt) +static inline void mt_clear_meta(struct maple_tree *mt, struct maple_node *mn, + enum maple_type type) { struct maple_metadata *meta; unsigned long *pivots; void __rcu **slots; void *next; - switch (mt) { + switch (type) { case maple_range_64: pivots = mn->mr64.pivot; if (unlikely(pivots[MAPLE_RANGE64_SLOTS - 2])) { slots = mn->mr64.slot; - next = mas_slot_locked(mas, slots, - MAPLE_RANGE64_SLOTS - 1); - if (unlikely((mte_to_node(next) && mte_node_type(next)))) - return; /* The last slot is a node, no metadata */ + next = mt_slot_locked(mt, slots, + MAPLE_RANGE64_SLOTS - 1); + if (unlikely((mte_to_node(next) && + mte_node_type(next)))) + return; /* no metadata, could be node */ } fallthrough; case maple_arange_64: - meta = ma_meta(mn, mt); + meta = ma_meta(mn, type); break; default: return; @@ -5497,7 +5503,7 @@ static inline int mas_rev_alloc(struct ma_state *mas, unsigned long min, } /* - * mas_dead_leaves() - Mark all leaves of a node as dead. + * mte_dead_leaves() - Mark all leaves of a node as dead. * @mas: The maple state * @slots: Pointer to the slot array * @type: The maple node type @@ -5507,16 +5513,16 @@ static inline int mas_rev_alloc(struct ma_state *mas, unsigned long min, * Return: The number of leaves marked as dead. */ static inline -unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots, - enum maple_type mt) +unsigned char mte_dead_leaves(struct maple_enode *enode, struct maple_tree *mt, + void __rcu **slots) { struct maple_node *node; enum maple_type type; void *entry; int offset; - for (offset = 0; offset < mt_slots[mt]; offset++) { - entry = mas_slot_locked(mas, slots, offset); + for (offset = 0; offset < mt_slot_count(enode); offset++) { + entry = mt_slot(mt, slots, offset); type = mte_node_type(entry); node = mte_to_node(entry); /* Use both node and type to catch LE & BE metadata */ @@ -5531,162 +5537,160 @@ unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots, return offset; } -static void __rcu **mas_dead_walk(struct ma_state *mas, unsigned char offset) +/** + * mte_dead_walk() - Walk down a dead tree to just before the leaves + * @enode: The maple encoded node + * @offset: The starting offset + * + * Note: This can only be used from the RCU callback context. + */ +static void __rcu **mte_dead_walk(struct maple_enode **enode, unsigned char offset) { - struct maple_node *next; + struct maple_node *node, *next; void __rcu **slots = NULL; - next = mas_mn(mas); + next = mte_to_node(*enode); do { - mas->node = mt_mk_node(next, next->type); - slots = ma_slots(next, next->type); - next = mas_slot_locked(mas, slots, offset); + *enode = ma_enode_ptr(next); + node = mte_to_node(*enode); + slots = ma_slots(node, node->type); + next = rcu_dereference_protected(slots[offset], + lock_is_held(&rcu_callback_map)); offset = 0; } while (!ma_is_leaf(next->type)); return slots; } +/** + * mt_free_walk() - Walk & free a tree in the RCU callback context + * @head: The RCU head that's within the node. + * + * Note: This can only be used from the RCU callback context. + */ static void mt_free_walk(struct rcu_head *head) { void __rcu **slots; struct maple_node *node, *start; - struct maple_tree mt; + struct maple_enode *enode; unsigned char offset; enum maple_type type; - MA_STATE(mas, &mt, 0, 0); node = container_of(head, struct maple_node, rcu); if (ma_is_leaf(node->type)) goto free_leaf; - mt_init_flags(&mt, node->ma_flags); - mas_lock(&mas); start = node; - mas.node = mt_mk_node(node, node->type); - slots = mas_dead_walk(&mas, 0); - node = mas_mn(&mas); + enode = mt_mk_node(node, node->type); + slots = mte_dead_walk(&enode, 0); + node = mte_to_node(enode); do { mt_free_bulk(node->slot_len, slots); offset = node->parent_slot + 1; - mas.node = node->piv_parent; - if (mas_mn(&mas) == node) - goto start_slots_free; - - type = mte_node_type(mas.node); - slots = ma_slots(mte_to_node(mas.node), type); - if ((offset < mt_slots[type]) && (slots[offset])) - slots = mas_dead_walk(&mas, offset); - - node = mas_mn(&mas); + enode = node->piv_parent; + if (mte_to_node(enode) == node) + goto free_leaf; + + type = mte_node_type(enode); + slots = ma_slots(mte_to_node(enode), type); + if ((offset < mt_slots[type]) && + rcu_dereference_protected(slots[offset], + lock_is_held(&rcu_callback_map))) + slots = mte_dead_walk(&enode, offset); + node = mte_to_node(enode); } while ((node != start) || (node->slot_len < offset)); slots = ma_slots(node, node->type); mt_free_bulk(node->slot_len, slots); -start_slots_free: - mas_unlock(&mas); free_leaf: mt_free_rcu(&node->rcu); } -static inline void __rcu **mas_destroy_descend(struct ma_state *mas, - struct maple_enode *prev, unsigned char offset) +static inline void __rcu **mte_destroy_descend(struct maple_enode **enode, + struct maple_tree *mt, struct maple_enode *prev, unsigned char offset) { struct maple_node *node; - struct maple_enode *next = mas->node; + struct maple_enode *next = *enode; void __rcu **slots = NULL; + enum maple_type type; + unsigned char next_offset = 0; do { - mas->node = next; - node = mas_mn(mas); - slots = ma_slots(node, mte_node_type(mas->node)); - next = mas_slot_locked(mas, slots, 0); - if ((mte_dead_node(next))) { - mte_to_node(next)->type = mte_node_type(next); - next = mas_slot_locked(mas, slots, 1); - } + *enode = next; + node = mte_to_node(*enode); + type = mte_node_type(*enode); + slots = ma_slots(node, type); + next = mt_slot_locked(mt, slots, next_offset); + if ((mte_dead_node(next))) + next = mt_slot_locked(mt, slots, ++next_offset); - mte_set_node_dead(mas->node); - node->type = mte_node_type(mas->node); - mas_clear_meta(mas, node, node->type); + mte_set_node_dead(*enode); + node->type = type; node->piv_parent = prev; node->parent_slot = offset; - offset = 0; - prev = mas->node; + offset = next_offset; + next_offset = 0; + prev = *enode; } while (!mte_is_leaf(next)); return slots; } -static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, +static void mt_destroy_walk(struct maple_enode *enode, struct maple_tree *mt, bool free) { void __rcu **slots; struct maple_node *node = mte_to_node(enode); struct maple_enode *start; - struct maple_tree mt; - - MA_STATE(mas, &mt, 0, 0); - mas.node = enode; if (mte_is_leaf(enode)) { node->type = mte_node_type(enode); goto free_leaf; } - ma_flags &= ~MT_FLAGS_LOCK_MASK; - mt_init_flags(&mt, ma_flags); - mas_lock(&mas); - - mte_to_node(enode)->ma_flags = ma_flags; start = enode; - slots = mas_destroy_descend(&mas, start, 0); - node = mas_mn(&mas); + slots = mte_destroy_descend(&enode, mt, start, 0); + node = mte_to_node(enode); // Updated in the above call. do { enum maple_type type; unsigned char offset; struct maple_enode *parent, *tmp; - node->type = mte_node_type(mas.node); - node->slot_len = mas_dead_leaves(&mas, slots, node->type); + node->slot_len = mte_dead_leaves(enode, mt, slots); if (free) mt_free_bulk(node->slot_len, slots); offset = node->parent_slot + 1; - mas.node = node->piv_parent; - if (mas_mn(&mas) == node) - goto start_slots_free; + enode = node->piv_parent; + if (mte_to_node(enode) == node) + goto free_leaf; - type = mte_node_type(mas.node); - slots = ma_slots(mte_to_node(mas.node), type); + type = mte_node_type(enode); + slots = ma_slots(mte_to_node(enode), type); if (offset >= mt_slots[type]) goto next; - tmp = mas_slot_locked(&mas, slots, offset); + tmp = mt_slot_locked(mt, slots, offset); if (mte_node_type(tmp) && mte_to_node(tmp)) { - parent = mas.node; - mas.node = tmp; - slots = mas_destroy_descend(&mas, parent, offset); + parent = enode; + enode = tmp; + slots = mte_destroy_descend(&enode, mt, parent, offset); } next: - node = mas_mn(&mas); - } while (start != mas.node); + node = mte_to_node(enode); + } while (start != enode); - node = mas_mn(&mas); - node->type = mte_node_type(mas.node); - node->slot_len = mas_dead_leaves(&mas, slots, node->type); + node = mte_to_node(enode); + node->slot_len = mte_dead_leaves(enode, mt, slots); if (free) mt_free_bulk(node->slot_len, slots); -start_slots_free: - mas_unlock(&mas); - free_leaf: if (free) mt_free_rcu(&node->rcu); else - mas_clear_meta(&mas, node, node->type); + mt_clear_meta(mt, node, node->type); } /* @@ -5702,10 +5706,10 @@ static inline void mte_destroy_walk(struct maple_enode *enode, struct maple_node *node = mte_to_node(enode); if (mt_in_rcu(mt)) { - mt_destroy_walk(enode, mt->ma_flags, false); + mt_destroy_walk(enode, mt, false); call_rcu(&node->rcu, mt_free_walk); } else { - mt_destroy_walk(enode, mt->ma_flags, true); + mt_destroy_walk(enode, mt, true); } } From patchwork Thu Feb 16 05:17:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57845 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp116884wrn; Wed, 15 Feb 2023 21:19:45 -0800 (PST) X-Google-Smtp-Source: AK7set9Gc9tc47DlB1qpprXxKUeYqmdJsJ5VzUUeA5zl/FiemrqCY/H5JVSXCjGSbirn1L2glJzK X-Received: by 2002:a17:90b:3887:b0:234:2d9b:b263 with SMTP id mu7-20020a17090b388700b002342d9bb263mr5812043pjb.17.1676524785159; Wed, 15 Feb 2023 21:19:45 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676524785; cv=none; d=google.com; s=arc-20160816; b=wUvXNYlSAEYPePiBVkm7c8N/pTAbbfksM55wgP4BYdXovV/G1Ymkqll/lZBfKK+tUW 3hSpVgdV0IdyIGRxWGao4f15kwNFdr40K9f1paERFL8EBPJfP1/vm0xzD1NF9jcnNiQT lxbBeoYEQcPuAnqDo72pG99e0nCGjj6UR4IJa1XPixfiKeO4K4p37uXHqTnaFE+eCe7p ZhTD3aw1+H810hvZyyy5wmeozjdeMLU/6wRPnuJnrTiI2beKZt+PAcEDPDaU59MUTDNP arGzhQVm81aIunkz6oWO2Ez5YRjRkbLWVy+o+AjupZzCT59VnUR06tiP2KFNZpB7NMUQ TyxA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=4opi4a+4wBSqIsfjyFGdOCf+RfFv13YZtxtXh5ocOUM=; b=cXgXlgkwjs6y4e9j9qmReg/6IG2BUfoy3v9zvZmCe0FRiXg/he0GszlOSz9kUyZRuD YRvQUGl1Jh3pM1MT4fLq9+iebKT1BW1KrL3mrdE3MCVOtjo1LBGTTCHK+L/3qr3sDw53 xNM02SZaN2/Xreby9R3kE5WCvGNC5mzmie1kiZD8wG3sRzbPG021hXdoEGjh503hXD/P r/ceSfgqLDbUFoUnk3axyt9mErOriBsfz66c553/At6Nx8ppjv6M/5fHq7Xvm3llGSvk lJWxSPJKgUTkt+6bNuwrUluFaomWeaYA/ZkuxWvxdsKHOTGAg4UCGL/elf7VySeSPMPp Q0fQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=Xo1jDDVi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y23-20020a17090a105700b002340e5003e8si556922pjd.45.2023.02.15.21.19.32; Wed, 15 Feb 2023 21:19:45 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=Xo1jDDVi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229673AbjBPFS6 (ORCPT + 99 others); Thu, 16 Feb 2023 00:18:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45976 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229650AbjBPFSn (ORCPT ); Thu, 16 Feb 2023 00:18:43 -0500 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AF4F63E60A for ; Wed, 15 Feb 2023 21:18:15 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-532e96672b3so8513177b3.13 for ; Wed, 15 Feb 2023 21:18:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=4opi4a+4wBSqIsfjyFGdOCf+RfFv13YZtxtXh5ocOUM=; b=Xo1jDDViHSHN/alioaTDnw+zuoNPKrF9ZLXUf3ea0NE3bXPLl0z7WkGrQUW7AOG0Nr J8SsODBneQSeaMKO7nEqhtXELAqyVVzyHag3dRajS0MNlWpB1lyOtsy0oB8MM0CrQEnP Moe4iuqLi8ffYInMsJBu3lDYUcEPw1W16t68/pqdVg/PM7yHOfl4JRIe/AAV288dZ063 0OmpnGUH5/0w4iEIelRXBTS4gWduVmNFBkN/5ItORK5tiSjU8ab3rw3adCz9oFddo8Er mGSeo2MZqEM7cxpZTU0mXlOgRPyGjhgPgDiiUmvS//v9NEkdkZgUTyYfpCAeqd0avQoQ 0u1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4opi4a+4wBSqIsfjyFGdOCf+RfFv13YZtxtXh5ocOUM=; b=dnVvbQuX0z02PvlE2JzCE/26YFqGqDUuIpfJ30U+NHA3qw9ThWwiofaxF09hYN84uM dfyUfd3bxyoDMkeKPTZvNXwwfrHNNvz4XL6Z08lLHaaIQ9OspmlzCC625YorPrElNk5X UxBkLJcvp1dj+jRSWEDvlt+Jo4UKUUDF2GZe7gA0WMdtmFVA1ghDuEG3Ux4+F1iaLOYZ mx48eW/1E2/I867mvHHxdSyTa7hFofNT8kTgYeTJbEl+ZpPJvJehA0iwWMmv43kvBfs9 +w7AD8pmzTvb2ElHOxk0qpcrcqVkzG+bi8+/3p2eZ4TmpG+2D7u6MgS0WdUGskjnaVyP N8Vw== X-Gm-Message-State: AO0yUKXFojpPCu/ngGizB2aOfBSAirllKfpxT8J7j8c5/+krrKJUcxER n0IP5ZUAWcvq5H/Xo1JC1sW8Ym0AJRw= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a05:6902:ca:b0:8f2:9e6:47a4 with SMTP id i10-20020a05690200ca00b008f209e647a4mr3ybs.7.1676524694561; Wed, 15 Feb 2023 21:18:14 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:23 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-9-surenb@google.com> Subject: [PATCH v3 08/35] mm: Enable maple tree RCU mode by default. From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, "Liam R. Howlett" , Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757963653010654781?= X-GMAIL-MSGID: =?utf-8?q?1757963653010654781?= From: "Liam R. Howlett" Use the maple tree in RCU mode for VMA tracking. This is necessary for the use of per-VMA locking. RCU mode is enabled by default but disabled when exiting an mm and for the new tree during a fork. Also enable RCU for the tree used in munmap operations to ensure the nodes remain valid for readers. Signed-off-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- include/linux/mm_types.h | 3 ++- kernel/fork.c | 3 +++ mm/mmap.c | 4 +++- 3 files changed, 8 insertions(+), 2 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 3cd60243c625..3fd5305dbbf9 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -768,7 +768,8 @@ struct mm_struct { unsigned long cpu_bitmap[]; }; -#define MM_MT_FLAGS (MT_FLAGS_ALLOC_RANGE | MT_FLAGS_LOCK_EXTERN) +#define MM_MT_FLAGS (MT_FLAGS_ALLOC_RANGE | MT_FLAGS_LOCK_EXTERN | \ + MT_FLAGS_USE_RCU) extern struct mm_struct init_mm; /* Pointer magic because the dynamic array size confuses some compilers. */ diff --git a/kernel/fork.c b/kernel/fork.c index 5e3029ea8e1e..5f23d5e03362 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -617,6 +617,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, if (retval) goto out; + mt_clear_in_rcu(vmi.mas.tree); for_each_vma(old_vmi, mpnt) { struct file *file; @@ -700,6 +701,8 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, retval = arch_dup_mmap(oldmm, mm); loop_out: vma_iter_free(&vmi); + if (!retval) + mt_set_in_rcu(vmi.mas.tree); out: mmap_write_unlock(mm); flush_tlb_mm(oldmm); diff --git a/mm/mmap.c b/mm/mmap.c index 20f21f0949dd..06c0f6686de8 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2277,7 +2277,8 @@ do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, int count = 0; int error = -ENOMEM; MA_STATE(mas_detach, &mt_detach, 0, 0); - mt_init_flags(&mt_detach, MT_FLAGS_LOCK_EXTERN); + mt_init_flags(&mt_detach, vmi->mas.tree->ma_flags & + (MT_FLAGS_LOCK_MASK | MT_FLAGS_USE_RCU)); mt_set_external_lock(&mt_detach, &mm->mmap_lock); /* @@ -3042,6 +3043,7 @@ void exit_mmap(struct mm_struct *mm) */ set_bit(MMF_OOM_SKIP, &mm->flags); mmap_write_lock(mm); + mt_clear_in_rcu(&mm->mm_mt); free_pgtables(&tlb, &mm->mm_mt, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING); tlb_finish_mmu(&tlb); From patchwork Thu Feb 16 05:17:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57846 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp116907wrn; Wed, 15 Feb 2023 21:19:50 -0800 (PST) X-Google-Smtp-Source: AK7set8liq3C21tBT+8f+pLHtCoJlShA76c5C1DbVZJVhYhlSyOiwgU/B5SGKwMH1iqJ6m3nl3so X-Received: by 2002:a17:90a:319:b0:234:797a:7c5f with SMTP id 25-20020a17090a031900b00234797a7c5fmr2612700pje.33.1676524789863; Wed, 15 Feb 2023 21:19:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676524789; cv=none; d=google.com; s=arc-20160816; b=kjOz8vPlsC/+/MFNwI7acSgNyVsECl8vLStYW9sYVM3ek5fy4n7x8B3j8hQ0Hv0Ms+ WDu8pK7IFXlfIGY8N3TtHI5Hz0dJ5QARHM+vqw+pPjchBncJKgNE4ewJElPudIJRhYrq Ht0jmkGrk0zKlm1c0m2lLlSUzvdf7PQUB9rJIkhkVwjXxB8IaVczCgtLmUNYrYBARWaS L35NR9L1b7Bq4I4V43r8EgOj2vkUsRrdi2IlaAOv1N7xTAGeKqn7RQ9HAMVS72pXC+r5 rfOna5E60cpyDM95L4cdZWzd9/jMQ7r/+3mWNFxB90x/rXHJVrqPiLS4JKVOIxnTKq/d Aq9w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=pFZiiTJ1lM5Y9cE9VJMMN6c4zys62lyfuQWFB/SvSfY=; b=rprj2ev3z7MCqQ7MGV2D+mlB1M078sc3pii9s67ktcHsl5jPrPoZN4yyHHrI3qsl97 +o2vD5WCPW2iwoSxCipKBivSpWRhCJJkwar1qJB3Khz5rQMSpJaxRlwyTcgh9P0jG6cH +WiFGma+wECvmUFxgVPcJ76GPwd+99XMioxlXG9Gh0V/OXYKbxkwpDKxZ22/SvfcGgM8 8Dvj8K9+WYo8DmDHDeRYeemjZISAL/uPtyUNYVhpZITknRU+D2aBS+Y21x054SM3ELEm +ubiJJ1eaQNpUAl7B83f6DZ8Hbk57HzY9UMViXZ3q+Vra8d02l7+lCPJekdmfR/hZNC8 bPbA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=hDLR1pvR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q33-20020a63f961000000b004fb8ed15327si524314pgk.857.2023.02.15.21.19.37; Wed, 15 Feb 2023 21:19:49 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=hDLR1pvR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229706AbjBPFTC (ORCPT + 99 others); Thu, 16 Feb 2023 00:19:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46064 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229670AbjBPFSp (ORCPT ); Thu, 16 Feb 2023 00:18:45 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 50A6243452 for ; Wed, 15 Feb 2023 21:18:18 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id h14-20020a258a8e000000b00827819f87e5so917586ybl.0 for ; Wed, 15 Feb 2023 21:18:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=pFZiiTJ1lM5Y9cE9VJMMN6c4zys62lyfuQWFB/SvSfY=; b=hDLR1pvRXBoNPEsd5rWO1uuL2xRvkViIuLxNT0AXT2es+wKEpveR0MayIpEY9kMwnX yKqoPf3/kr2W8U8aZxHHsI3MeVglv4Uuopqy/RZQvDUL+Iw/1b3yNPaW9UiWpFiktEjT IG17SwGNRtI3Tudjcg57P2Pp6pg3eItytQO0663ElsQFpDjPji34Qnzv66OgBFV/+yja uvKllpdQQMscfNlzsu7K4yE4HNqAZ53EkNxV25Op17ac2k06MkhohfLIKcbxFpec+BdV 695/dYeBJhC1TaWGjUDZda/yGXvfwWspsF3C1nfSkAJuqCo5le3F23yvLzWMlOVw2Nnw Ls0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=pFZiiTJ1lM5Y9cE9VJMMN6c4zys62lyfuQWFB/SvSfY=; b=ZH3ovrhLib0C7VYfFMEp9wUCa1a0sNv43nTZyEj7wvYn8i+tt1UwDkeH0mm+c5GdFV m6U2FqPgN1XTW1NvUcnZoiu5cHbf7pkaCTbdM6Ly9FxwYz+BZKI0YJQQC9OOnYuPchRf pYjy/ldsOWwqAhhbUuYCvaiMiuCG7GVhBdl5imo8PPcSOGNA43Qb+9VVPXHqcM0XSLs8 e3fEdLmjX3zLLVgaxQSvfbdFVi30PGAUS7IsaxuJ5B2FP+QLxtT7n9CAZQFFtmXBAAe8 V/r7eTk/WMi22nDvlW2v0dhD9z43UlHQQi9a9u2z3wUUAaqWE+aDp0W9nN46vnO6aHpV TTnw== X-Gm-Message-State: AO0yUKX7nXSIQWJMT9c0VHqjqeF3J7x+PExHhiOA20f2Co81l7nt81S6 J3k82obhH4VCA8PRLHkNUrmr8IAZjtM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a81:9e45:0:b0:50f:f163:7072 with SMTP id n5-20020a819e45000000b0050ff1637072mr442506ywj.285.1676524697322; Wed, 15 Feb 2023 21:18:17 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:24 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-10-surenb@google.com> Subject: [PATCH v3 09/35] mm: introduce CONFIG_PER_VMA_LOCK From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757963658357990058?= X-GMAIL-MSGID: =?utf-8?q?1757963658357990058?= This configuration variable will be used to build the support for VMA locking during page fault handling. This is enabled on supported architectures with SMP and MMU set. The architecture support is needed since the page fault handler is called from the architecture's page faulting code which needs modifications to handle faults under VMA lock. Signed-off-by: Suren Baghdasaryan --- mm/Kconfig | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index ca98b2072df5..2e4a7e61768a 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1211,6 +1211,18 @@ config LRU_GEN_STATS This option has a per-memcg and per-node memory overhead. # } +config ARCH_SUPPORTS_PER_VMA_LOCK + def_bool n + +config PER_VMA_LOCK + def_bool y + depends on ARCH_SUPPORTS_PER_VMA_LOCK && MMU && SMP + help + Allow per-vma locking during page fault handling. + + This feature allows locking each virtual memory area separately when + handling page faults instead of taking mmap_lock. + source "mm/damon/Kconfig" endmenu From patchwork Thu Feb 16 05:17:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57847 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp116931wrn; Wed, 15 Feb 2023 21:19:54 -0800 (PST) X-Google-Smtp-Source: AK7set84D+EAbTYWGXP/L6kxNaSDGvblz+uEq8NUC0omi/TcjvtpImy8e9TtssOT8qpPTlAtpbZi X-Received: by 2002:a17:90b:1a85:b0:230:c675:787b with SMTP id ng5-20020a17090b1a8500b00230c675787bmr5555475pjb.18.1676524794556; Wed, 15 Feb 2023 21:19:54 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676524794; cv=none; d=google.com; s=arc-20160816; b=V/ro73QXp3GdRDiyJV7DB9+P30e0adyXjBuJdupFaFSNpDBFrvTCk0AO9C0rGHejS2 Ue+nVx1dFChY+gjrjWASQJXd9hN2EdVKlhocbEOIQKWFw7UsHCW5bJPQMYoeLkpI5amM hgxizSrPkt2Cat39AcXdCMIc0ged90L2+yHRwYYNjFd9mRdnS0UDodgxvTQPB1oDHJrU MJPiT3UyimZZrmRfAiHY/S0b+fbYFhpSxYemQNVfd9UjFcACdXhyUnuorswlwJ+DKNBt CywKS+T4/7CSQsHmpQXlZgA0cn9wS4Mtsw1zOwKplAWATNWf5768Ao98Rp4ihfZhMPMp qAGg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=nnxUMMQI6vT0M/Q5sa1IAaTl/WTbc27CfHmxL3/FviM=; b=losE3ex/xO2N+8fYuvbfbz+1MGzjzqBLQAf5IVhQtVeROO61OaHgUg711CWJTFUmYX eHqHLDXoCVv5FJnA48qU9fx7Rni0DcZ4LcrhdoS5FPobQlDGVxfPvf9oWXc4AnYC/qiS /l/fv160lG4xy0tWekf4SaW/2vlYs2No1HWcTP8hMB+BF9wB3sNARNq+bOprXFCsg/Ei A8K+5GWs4EdWr8CHqpHhQvVFyMqsWVAK6LSQiV0ARMvUfP4ns0vZHCRi5ShVe+wChvOp 7XjBYB+Xio0wICva7O34QGgvew4oERtKGjEdzcXtgdVwqAN0iMMtRIxY3BmqzuoNCpJ7 iC0g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=KnhAt7qu; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id cl3-20020a17090af68300b00233f65e8f1csi3990137pjb.126.2023.02.15.21.19.42; Wed, 15 Feb 2023 21:19:54 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=KnhAt7qu; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229651AbjBPFTS (ORCPT + 99 others); Thu, 16 Feb 2023 00:19:18 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46082 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229647AbjBPFSs (ORCPT ); Thu, 16 Feb 2023 00:18:48 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5A62946153 for ; Wed, 15 Feb 2023 21:18:20 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id a4-20020a5b0004000000b006fdc6aaec4fso842945ybp.20 for ; Wed, 15 Feb 2023 21:18:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nnxUMMQI6vT0M/Q5sa1IAaTl/WTbc27CfHmxL3/FviM=; b=KnhAt7quYBwXQRhfuSYaJDek4wttI7RDXOOACIL/F63Nz/58eVv03No3Zqh2UIwNdl TkKGisyfzRrXImR2jOkU2wcQekf+kt9/XxXIsRoXlqskuIeX6hOhHTNZXgdfaMplle7K SAATQyrMS3HXKQy7j0lc9Kc3dK3KueDsfgI8ryy5OQhnNBROeaCbU2Hmg2SLUQOwNbgt XwBV7zK7XX+1ClOK24Y2b0Ex8amo9ljLG0ZokYkKzEniw9mpgT07P+84GOQVR154zkPK 4YJcp2AEGd79KcxB1dsy4RuRGjRlYt9yxaLz/0Wa2WtK7Y2fqnfOErlG1kh3RHBuBBJK 58yQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nnxUMMQI6vT0M/Q5sa1IAaTl/WTbc27CfHmxL3/FviM=; b=cizoPD8FGYc8oxJRSDOtod6y2ksbkM9Nf7KWgULp83n5IriPxwPRahnd4O3DT1HBtx Cq7BTBuNRYLwvbaRm+q2TDaXBlnOv0y1Dgx+rJi4JCaloNIGSMJI2SSO+f+6c2Dvtyik NgEJxQXkQS6sY3c+AnrggVru9x2lictY3igIprMtmgP4WHNz5lIw/leWt4goToMKI5fG ORyew4tAYjNSwcA804jmA8rNHiNtptIZZTXCReLLrMpKR452rgkDQ7akhFJYCSD5Gv06 h3HVRfTLBYXVcn05/EvtplkZw8kaYSylqyHXvDoQFEUG3sTyeDLfkoL2sXrMZyc1Yinp n7cg== X-Gm-Message-State: AO0yUKUKPzTYf3nE/WewVHG8+B4azpm9YHaBTLhUC7pzxdOnfYOY7QiD MF14gF/ag0j+e1tgo2kv26XY7rzLgRI= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a81:a20c:0:b0:52e:f79e:b639 with SMTP id w12-20020a81a20c000000b0052ef79eb639mr389199ywg.283.1676524699450; Wed, 15 Feb 2023 21:18:19 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:25 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-11-surenb@google.com> Subject: [PATCH v3 10/35] mm: rcu safe VMA freeing From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757963662804492015?= X-GMAIL-MSGID: =?utf-8?q?1757963662804492015?= From: Michel Lespinasse This prepares for page faults handling under VMA lock, looking up VMAs under protection of an rcu read lock, instead of the usual mmap read lock. Signed-off-by: Michel Lespinasse Signed-off-by: Suren Baghdasaryan --- include/linux/mm_types.h | 13 ++++++++++--- kernel/fork.c | 20 +++++++++++++++++++- 2 files changed, 29 insertions(+), 4 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 3fd5305dbbf9..fb4e2afad787 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -480,9 +480,16 @@ struct anon_vma_name { struct vm_area_struct { /* The first cache line has the info for VMA tree walking. */ - unsigned long vm_start; /* Our start address within vm_mm. */ - unsigned long vm_end; /* The first byte after our end address - within vm_mm. */ + union { + struct { + /* VMA covers [vm_start; vm_end) addresses within mm */ + unsigned long vm_start; + unsigned long vm_end; + }; +#ifdef CONFIG_PER_VMA_LOCK + struct rcu_head vm_rcu; /* Used for deferred freeing. */ +#endif + }; struct mm_struct *vm_mm; /* The address space we belong to. */ diff --git a/kernel/fork.c b/kernel/fork.c index 5f23d5e03362..314d51eb91da 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -479,12 +479,30 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return new; } -void vm_area_free(struct vm_area_struct *vma) +static void __vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); kmem_cache_free(vm_area_cachep, vma); } +#ifdef CONFIG_PER_VMA_LOCK +static void vm_area_free_rcu_cb(struct rcu_head *head) +{ + struct vm_area_struct *vma = container_of(head, struct vm_area_struct, + vm_rcu); + __vm_area_free(vma); +} +#endif + +void vm_area_free(struct vm_area_struct *vma) +{ +#ifdef CONFIG_PER_VMA_LOCK + call_rcu(&vma->vm_rcu, vm_area_free_rcu_cb); +#else + __vm_area_free(vma); +#endif +} + static void account_kernel_stack(struct task_struct *tsk, int account) { if (IS_ENABLED(CONFIG_VMAP_STACK)) { From patchwork Thu Feb 16 05:17:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57849 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp117061wrn; Wed, 15 Feb 2023 21:20:23 -0800 (PST) X-Google-Smtp-Source: AK7set/XxWKiUeHmXiFv0lcnLZu5P1+HXf/JKyPF67fbkKg7VJa0u7Z7d8cP0w5B1ggdOU+2vK78 X-Received: by 2002:a17:906:af1b:b0:889:d998:1576 with SMTP id lx27-20020a170906af1b00b00889d9981576mr4644773ejb.66.1676524823822; Wed, 15 Feb 2023 21:20:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676524823; cv=none; d=google.com; s=arc-20160816; b=fXCPKZgE92gJffPRjGixU2urw/W8hFzE9u0siwURayLzl1hkQ/PiCa9TTJnWzzMOef TvoKqgCFyeYRXz55y8C8TRzBoSTZMEOqtpSqk0BlnwF2PaysXwiPE7WSMcLG29QMYKke Khn9hZkGERFxJY9T/a22AC1eOSp5PSUm1aGGz1ozdrkBrQ4Tz7ryoYM7dT40otXMw/9J bSiy47OGLJb3c3KT6ko8oeHfFBLZupEtLQE1pe74JXSRO24VnDUd0Vit1WMErpyfeDon 7uCgMAfXtSF6oriEH9qyu/06q1q8KyYjXEeWqZY3j/80BWoliMM6rm4nAc29p9UTbtL0 Fuog== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=OfsRQat1IE0fDeYSs5Psw7qjLLZE8mkiXmSdSDqQs6E=; b=JHNnpQr6fFuMyBpg7UNK4Udx0YiiU7Cw9QsTaCGmxyH+9gNwrrf+SQmZ+ZfgoWQ5RV LdavixxGX0J53JeuiNwcCtDlNen5GE5uKXOv7qiW/xvY4mtZfGud/+flaY3uYxcGPF9V QiM4ougyvFwVkkuczPCLqvxQ8UpxDcYs5BUjeUduizQJ6+zaGtWIdlXTjjkL5VMq5m5V nVIg5ZxO0Ih0KojkVfDVvBU7QPnUfmc0LX7VMNBfRQLcr/cytLrtN/cg8kfqScKi37Aw NuDh2tsOCddKTXnThQbDUlKXwbceWvA+nwmPChiTWQou+z7wBkk2UvE4HB76zY8b2L2n duSA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=MfUNnuRV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id fr37-20020a170906892500b00887f7d97f5bsi758923ejc.29.2023.02.15.21.19.59; Wed, 15 Feb 2023 21:20:23 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=MfUNnuRV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229628AbjBPFTX (ORCPT + 99 others); Thu, 16 Feb 2023 00:19:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45784 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229657AbjBPFSw (ORCPT ); Thu, 16 Feb 2023 00:18:52 -0500 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4285245F6C for ; Wed, 15 Feb 2023 21:18:23 -0800 (PST) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-52a8f97c8ccso8654047b3.6 for ; Wed, 15 Feb 2023 21:18:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=OfsRQat1IE0fDeYSs5Psw7qjLLZE8mkiXmSdSDqQs6E=; b=MfUNnuRVqilTlETV75doNo5d3X2f1I64PkztavWwbIZWW2j+3a5GtwWp8jaeE2FAS7 MLS5o+W6QzlLw73Ic3jcbhKR5fcnDvpJtAm6NUi7rdrBvnG1z5U94zlj5p9Rf6AtiMXB /0WTQWR4hibDfKvJz8o49Ph/XAqUGXxJaWOev01ngzcVh9Xd6JHGOFVoBB6DDs9XGB0U jDPi/dJdwD6mt/I/Iobb8XMkCnfA0C9OXZWj96tLYiXqom79AFLMHYYopE8HfxvEiMuA aI5RyI6LPkvxen53dNb8XfFirBUFuNssxJItswg1uaEv8wS3br6uTFvWft0h1+Cz189K IN3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=OfsRQat1IE0fDeYSs5Psw7qjLLZE8mkiXmSdSDqQs6E=; b=0INJSZZGRj5H2sQcuUY0td4iWSAnT1XYft+W7vQ8n31jU7wcv5lT+ju/qvpI/+FpD1 FfArRVteTZ9cv+jMavW5HeCtFEYrcHI2Ugb+k5X64pLkARHIVHdz+1DhYsvgEBrvuTBV xTZ/i9+TD/KUxEEKaJrwOPLZOt7Tmc8zt/UkfQmu8JCTAqBpqT/LPtj9E7dCSScg7j6c AjhAFC0MFGwSexfHTCmz0MMC1VkMGvB/GN82u5fpZ0wyQl/ge9DvnE5aWZuuDeX5lr5q pwhPENknly0wFvNT7ibOzStD0vlJLrBCIeH8lbiQI3Fx9jRzxV9V8mey9yNWuwNOeoSr 6jnA== X-Gm-Message-State: AO0yUKU8lGwq22vHBgm5Q2nw+NNwVJWO6oDg/HYyQQoONaquZTDnIk6a 3mV6EVCycZ8+CxePqkBE6iLX6YXgw3c= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a25:8e8c:0:b0:965:bac9:d458 with SMTP id q12-20020a258e8c000000b00965bac9d458mr0ybl.11.1676524701625; Wed, 15 Feb 2023 21:18:21 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:26 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-12-surenb@google.com> Subject: [PATCH v3 11/35] mm: move mmap_lock assert function definitions From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757963693738440460?= X-GMAIL-MSGID: =?utf-8?q?1757963693738440460?= Move mmap_lock assert function definitions up so that they can be used by other mmap_lock routines. Signed-off-by: Suren Baghdasaryan --- include/linux/mmap_lock.h | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 96e113e23d04..e49ba91bb1f0 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -60,6 +60,18 @@ static inline void __mmap_lock_trace_released(struct mm_struct *mm, bool write) #endif /* CONFIG_TRACING */ +static inline void mmap_assert_locked(struct mm_struct *mm) +{ + lockdep_assert_held(&mm->mmap_lock); + VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); +} + +static inline void mmap_assert_write_locked(struct mm_struct *mm) +{ + lockdep_assert_held_write(&mm->mmap_lock); + VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); +} + static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); @@ -150,18 +162,6 @@ static inline void mmap_read_unlock_non_owner(struct mm_struct *mm) up_read_non_owner(&mm->mmap_lock); } -static inline void mmap_assert_locked(struct mm_struct *mm) -{ - lockdep_assert_held(&mm->mmap_lock); - VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); -} - -static inline void mmap_assert_write_locked(struct mm_struct *mm) -{ - lockdep_assert_held_write(&mm->mmap_lock); - VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); -} - static inline int mmap_lock_is_contended(struct mm_struct *mm) { return rwsem_is_contended(&mm->mmap_lock); From patchwork Thu Feb 16 05:17:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57851 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp117070wrn; Wed, 15 Feb 2023 21:20:25 -0800 (PST) X-Google-Smtp-Source: AK7set/oqSgKNO4+f1V4sQKMFE+sl3NPmq3YmULk55vuPV2MH+VqwTMTgXDXfqqMhdeNWt5kDwe3 X-Received: by 2002:a17:907:b9d3:b0:877:a7ec:5ff with SMTP id xa19-20020a170907b9d300b00877a7ec05ffmr4749574ejc.10.1676524825431; Wed, 15 Feb 2023 21:20:25 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676524825; cv=none; d=google.com; s=arc-20160816; b=xZVcfjIwRzUcGONMC+PGb9RK24Dj4txiJsoIJ3YTgWXkf83o5oanZt3pvep40fHPME uC4NPBWDtfgXGx+bgCypmBoLwCTs7xYbXMHebiY1fscoJldBh+dYzE3JVgCSpUpSfVjN n8Vu0dpCecGTtxk3//8CTFjycDwRVe87iEpE/BwPZ68Bl9g6ovZYoioMKVs3CX0c3PnR HmsjhIxvg4H4wxuyfAjYrNpOI/kqhL7WGbRIQvBT34ACqFuWAeSPJq8um2GU9RR/Z1/5 msty6ZZrmlQ31urG3b4PwMAKS3SKmMbNs7ektjYi6ycptjrUgBS8zMeTa2XatWbsp2lD loBg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=S0NkKUx4dkIelWS5aTsq7UgpXIDBIhGrMRdd8DrqTMg=; b=cBGJnXtZ5XjURCR8Xa8YE4H5qo/yFelPZt9g7YnvsT406+Mas8igwKjnzRkILmRjKP RenveGXQpPUNId64CRUL3B+xLdYFUPY0YKpRB0B4zHKRd4BvuiMd2stB5Tc8v08PLdN7 X7a6TmPjSghgIZENVMl8/94Ck0ewg3VX/YvtRRigToRJeSgwERe3SJw+ubawldyXH9AO G1G20FZ0STQ+uoY0frk9sdOVxvehlxSRClUGSnLoatddbtHHPhGRhCpmaUDw8U6hV+3a xx6MlJIoXwdc/uZ72CT0VHe0X9HHQ/4ecoFIUPLCHbKwccddOVEhLniYbB00tDCaD5Dh BcEA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=jg6fXXyN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ev23-20020a17090729d700b008b1598f6a24si811784ejc.518.2023.02.15.21.20.02; Wed, 15 Feb 2023 21:20:25 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=jg6fXXyN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229492AbjBPFTb (ORCPT + 99 others); Thu, 16 Feb 2023 00:19:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46626 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229620AbjBPFTE (ORCPT ); Thu, 16 Feb 2023 00:19:04 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C26784608B for ; Wed, 15 Feb 2023 21:18:25 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id n20-20020a25da14000000b008fa1d22bd55so839670ybf.21 for ; Wed, 15 Feb 2023 21:18:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=S0NkKUx4dkIelWS5aTsq7UgpXIDBIhGrMRdd8DrqTMg=; b=jg6fXXyNPA7kGzHuabxhVQS/lUkV32r/2v7x3huSOdcQqflpdfqhoIlNaR0t9wMIZ5 tNsdAuEIvz4tV4UF6ouWYwPH6jjBst45X8ZS39R/or9bhVEvk/hAhXAhGiG6HODkz0Ti 4+W9CepIhu/sbVVhPdRUrFOaALk+XM5cteC/he2x1EN76QEzJ6hbvpDzGk804gLMUejh OhIx76lfAQ2Aj8i3vKfn29ePpUTgzAK6jOUxJ6v4seZ7ANgrIyb45IfAAVauV03+BjjR E1pd5h52L83bbDKmbPDtlCRxv6trm/z+Fa4Qs6PgKsiEdeHNP77Ta8uAxqqtUigosnhI vlvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=S0NkKUx4dkIelWS5aTsq7UgpXIDBIhGrMRdd8DrqTMg=; b=Q/Uo+RLjMFlihdIzbmlPSUbkIaJIJTcb4c1bneOax/ntNJhAvKFQNBA3zN7ZkhUV6E TSq8Ut3FpDf8slbbScMCpDenD3Cd3JES5REoGe3cO5trmE5yibGopWva8oVLtDtD/bEy lejdZd2FZixobs1qwJLUU/dT9wcfOpx58hnC5UoPKa3k9k5JgW+r08WVcaptBZUAabcS kIfK4Djaj1ftiC69xa8dRpF+ci1O9rtH53yRUNPaqx64sbPSjCUcFX94gnFsOZmRe54h R/gYsFnjZrtvpG/G+Ff8UZQi12G20IbytQCfogSXTe1M5sWx+0dV5QCAwc3I0UW8ZfjY T0PQ== X-Gm-Message-State: AO0yUKWsCjeq4Mak6wb73UXpvEvAk8M08+g3nCfQYOvwFXPwMmV+CWxx 0VPXwSTgUoGD2IqroIftbGIHBDYB4mM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a05:6902:13c6:b0:8da:3163:224 with SMTP id y6-20020a05690213c600b008da31630224mr5ybu.0.1676524704439; Wed, 15 Feb 2023 21:18:24 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:27 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-13-surenb@google.com> Subject: [PATCH v3 12/35] mm: add per-VMA lock and helper functions to control it From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757963695230349443?= X-GMAIL-MSGID: =?utf-8?q?1757963695230349443?= Introduce per-VMA locking. The lock implementation relies on a per-vma and per-mm sequence counters to note exclusive locking: - read lock - (implemented by vma_start_read) requires the vma (vm_lock_seq) and mm (mm_lock_seq) sequence counters to differ. If they match then there must be a vma exclusive lock held somewhere. - read unlock - (implemented by vma_end_read) is a trivial vma->lock unlock. - write lock - (vma_start_write) requires the mmap_lock to be held exclusively and the current mm counter is assigned to the vma counter. This will allow multiple vmas to be locked under a single mmap_lock write lock (e.g. during vma merging). The vma counter is modified under exclusive vma lock. - write unlock - (vma_end_write_all) is a batch release of all vma locks held. It doesn't pair with a specific vma_start_write! It is done before exclusive mmap_lock is released by incrementing mm sequence counter (mm_lock_seq). - write downgrade - if the mmap_lock is downgraded to the read lock, all vma write locks are released as well (effectivelly same as write unlock). Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 82 +++++++++++++++++++++++++++++++++++++++ include/linux/mm_types.h | 8 ++++ include/linux/mmap_lock.h | 13 +++++++ kernel/fork.c | 4 ++ mm/init-mm.c | 3 ++ 5 files changed, 110 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 2992a2d55aee..a056ee170e34 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -623,6 +623,87 @@ struct vm_operations_struct { unsigned long addr); }; +#ifdef CONFIG_PER_VMA_LOCK +static inline void vma_init_lock(struct vm_area_struct *vma) +{ + init_rwsem(&vma->lock); + vma->vm_lock_seq = -1; +} + +/* + * Try to read-lock a vma. The function is allowed to occasionally yield false + * locked result to avoid performance overhead, in which case we fall back to + * using mmap_lock. The function should never yield false unlocked result. + */ +static inline bool vma_start_read(struct vm_area_struct *vma) +{ + /* Check before locking. A race might cause false locked result. */ + if (vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq)) + return false; + + if (unlikely(down_read_trylock(&vma->lock) == 0)) + return false; + + /* + * Overflow might produce false locked result. + * False unlocked result is impossible because we modify and check + * vma->vm_lock_seq under vma->lock protection and mm->mm_lock_seq + * modification invalidates all existing locks. + */ + if (unlikely(vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq))) { + up_read(&vma->lock); + return false; + } + return true; +} + +static inline void vma_end_read(struct vm_area_struct *vma) +{ + rcu_read_lock(); /* keeps vma alive till the end of up_read */ + up_read(&vma->lock); + rcu_read_unlock(); +} + +static inline void vma_start_write(struct vm_area_struct *vma) +{ + int mm_lock_seq; + + mmap_assert_write_locked(vma->vm_mm); + + /* + * current task is holding mmap_write_lock, both vma->vm_lock_seq and + * mm->mm_lock_seq can't be concurrently modified. + */ + mm_lock_seq = READ_ONCE(vma->vm_mm->mm_lock_seq); + if (vma->vm_lock_seq == mm_lock_seq) + return; + + down_write(&vma->lock); + vma->vm_lock_seq = mm_lock_seq; + up_write(&vma->lock); +} + +static inline void vma_assert_write_locked(struct vm_area_struct *vma) +{ + mmap_assert_write_locked(vma->vm_mm); + /* + * current task is holding mmap_write_lock, both vma->vm_lock_seq and + * mm->mm_lock_seq can't be concurrently modified. + */ + VM_BUG_ON_VMA(vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); +} + +#else /* CONFIG_PER_VMA_LOCK */ + +static inline void vma_init_lock(struct vm_area_struct *vma) {} +static inline bool vma_start_read(struct vm_area_struct *vma) + { return false; } +static inline void vma_end_read(struct vm_area_struct *vma) {} +static inline void vma_start_write(struct vm_area_struct *vma) {} +static inline void vma_assert_write_locked(struct vm_area_struct *vma) {} + +#endif /* CONFIG_PER_VMA_LOCK */ + static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) { static const struct vm_operations_struct dummy_vm_ops = {}; @@ -631,6 +712,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); + vma_init_lock(vma); } /* Use when VMA is not part of the VMA tree and needs no locking */ diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index fb4e2afad787..e268723eaf44 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -508,6 +508,11 @@ struct vm_area_struct { vm_flags_t __private __vm_flags; }; +#ifdef CONFIG_PER_VMA_LOCK + int vm_lock_seq; + struct rw_semaphore lock; +#endif + /* * For areas with an address space and backing store, * linkage into the address_space->i_mmap interval tree. @@ -633,6 +638,9 @@ struct mm_struct { * init_mm.mmlist, and are protected * by mmlist_lock */ +#ifdef CONFIG_PER_VMA_LOCK + int mm_lock_seq; +#endif unsigned long hiwater_rss; /* High-watermark of RSS usage */ diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index e49ba91bb1f0..aab8f1b28d26 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -72,6 +72,17 @@ static inline void mmap_assert_write_locked(struct mm_struct *mm) VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); } +#ifdef CONFIG_PER_VMA_LOCK +static inline void vma_end_write_all(struct mm_struct *mm) +{ + mmap_assert_write_locked(mm); + /* No races during update due to exclusive mmap_lock being held */ + WRITE_ONCE(mm->mm_lock_seq, mm->mm_lock_seq + 1); +} +#else +static inline void vma_end_write_all(struct mm_struct *mm) {} +#endif + static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); @@ -114,12 +125,14 @@ static inline bool mmap_write_trylock(struct mm_struct *mm) static inline void mmap_write_unlock(struct mm_struct *mm) { __mmap_lock_trace_released(mm, true); + vma_end_write_all(mm); up_write(&mm->mmap_lock); } static inline void mmap_write_downgrade(struct mm_struct *mm) { __mmap_lock_trace_acquire_returned(mm, false, true); + vma_end_write_all(mm); downgrade_write(&mm->mmap_lock); } diff --git a/kernel/fork.c b/kernel/fork.c index 314d51eb91da..9141427a98b2 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -474,6 +474,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) */ data_race(memcpy(new, orig, sizeof(*new))); INIT_LIST_HEAD(&new->anon_vma_chain); + vma_init_lock(new); dup_anon_vma_name(orig, new); } return new; @@ -1147,6 +1148,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, seqcount_init(&mm->write_protect_seq); mmap_init_lock(mm); INIT_LIST_HEAD(&mm->mmlist); +#ifdef CONFIG_PER_VMA_LOCK + mm->mm_lock_seq = 0; +#endif mm_pgtables_bytes_init(mm); mm->map_count = 0; mm->locked_vm = 0; diff --git a/mm/init-mm.c b/mm/init-mm.c index c9327abb771c..33269314e060 100644 --- a/mm/init-mm.c +++ b/mm/init-mm.c @@ -37,6 +37,9 @@ struct mm_struct init_mm = { .page_table_lock = __SPIN_LOCK_UNLOCKED(init_mm.page_table_lock), .arg_lock = __SPIN_LOCK_UNLOCKED(init_mm.arg_lock), .mmlist = LIST_HEAD_INIT(init_mm.mmlist), +#ifdef CONFIG_PER_VMA_LOCK + .mm_lock_seq = 0, +#endif .user_ns = &init_user_ns, .cpu_bitmap = CPU_BITS_NONE, #ifdef CONFIG_IOMMU_SVA From patchwork Thu Feb 16 05:17:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57848 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp117027wrn; Wed, 15 Feb 2023 21:20:18 -0800 (PST) X-Google-Smtp-Source: AK7set/V3zbWNcpz7SGShristD7TKXKcGzTLyRJz3t6BKbzLBFVzb9x259Wq9HLrkeDwJaP4QqWB X-Received: by 2002:a17:902:cecd:b0:199:2f0a:697 with SMTP id d13-20020a170902cecd00b001992f0a0697mr5385164plg.33.1676524818145; Wed, 15 Feb 2023 21:20:18 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676524818; cv=none; d=google.com; s=arc-20160816; b=QLZ674PuxBrHvlOMbs8iE8AMvJr/abS5JIfLyaQu3phH0jOxz0zwj3LH5tmhKGsAzi 5Rp130M3WjdzgPgOxxmOl5y5xrWbFoVq4vJtcW/rfAKvQ374rNI2sg7fB7cAenZ/3gMk BSpvXP7s6veLJG+ZvV+oKSbSLXk9BWdEvbJ7sXF+PSW2uBQaj9A8qyjF7zmz8mfu31jW NfYD3W3rcmivtZxVo11n1/B+tLPdw7pWfSH4kWGJ5a7Y7bCbFh1PP19EXLrGypMv6jCg iXcmstC0wVJdeH+21IsU2ueRAfiHbVrZ0r+CXjxePmrDt+qPVQqkzN+/75432wsLTPfK uf/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=lpf0g6jEgEDruwUAtDYeX37TOF9sV0bBdZYAp5OaPFw=; b=c64x9Z2cBkildgvdx3704ICFiFfn/+15ts8zv1FgOcHoAdjaprgo/Ql30JjO5TZ+eH TbZJz4VxjibEgDSJbcwUc4bp4xXYZI7lNsnS1WwP56bqqqDearQzWOFHlQ3dRoU/oCBm gy1m/iFOSDBEtR6ppTTd/m+GQmlbdiZhS+6h9kuYjdsbuRftfT14TWzX4/kskbDiWYqm YVJ9J98h903okK9dFZcHL1LIZVHjbIf2GZb5uos7QjmNgjMCLh76JW4llzjP7uNbNmha KDE/Ppgh51sn0Hw3T0F9DTvUQ5tLJk4aTNRXmf6Yx4vHPBHKeSsufeJ4urIJ5qqPK8Hm 3Eog== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=QqHoT4KB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a184-20020a6366c1000000b004dfc64889d2si652802pgc.673.2023.02.15.21.20.05; Wed, 15 Feb 2023 21:20:18 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=QqHoT4KB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229640AbjBPFTf (ORCPT + 99 others); Thu, 16 Feb 2023 00:19:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45974 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229587AbjBPFTL (ORCPT ); Thu, 16 Feb 2023 00:19:11 -0500 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6D08039B98 for ; Wed, 15 Feb 2023 21:18:27 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-4cddba76f55so8301457b3.23 for ; Wed, 15 Feb 2023 21:18:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=lpf0g6jEgEDruwUAtDYeX37TOF9sV0bBdZYAp5OaPFw=; b=QqHoT4KBvKUoUdLlMqp51fx+DvEGbEeCQBAnfd6VVaWQ1iGMSVWdcYYT+IcmdoGW6k 3cvpI3DbxmqWzlsZbZtdKxchg+b34bnVwYQM28D7iM4wfdyptvE97biBiYefhiTugMIi 6QLuIfwXdU8feO6tOQjuROgghWTm3q6OIMngPhTxYx3FIBx/Q/vwNR9bBonlFJnVLRif jIgBrdCbBh8/u0HLrcAEynfXgVsHAUyrN5D1sUvBi7o9DTj4LnFYvOofA3wqcNiCIoKK +suiPaGBCd0kL8KKwbFLDtmEf/YUc3gt6F4GpOlLFqmXcdgxQWnJd0Ox4qoA9H1pNQ9Z aRAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=lpf0g6jEgEDruwUAtDYeX37TOF9sV0bBdZYAp5OaPFw=; b=B74ZFrO6R44wFBlyz2MCssrIgopSojVQC8qyy0NXl+Nq87HlQIzydb3FEqU74fQHww qWrR0nwICIbOAtz36fVIX1U1peg5+3wmahF/VHlRswJNW7uZ3bNv2tmURvCmdCLol3Tu e9VAHheIaeoJCijRq/4QatWCPIe25Op9TNhFlIStz5AlyvfHqwcSyY+tPYXRrUtMcirf 4V08cDa+NrFTutD8ehZcRfg5/+JcIrBSh6B/NJdTrHtIgjZVBKTEpZPQ08wnnC/2joA8 smD6UYEL8mpGyzPttkUTRggPq318tOVP5saQPu1AP0EtuGipD1eOZ9GKxkhKYrLcZUwP unIQ== X-Gm-Message-State: AO0yUKX3jE0Qh/2Ss6g7THPB3cVWFfBB3LegEHj/csjMqGeQc10LwBT3 E3nAeAClMMnTUR4zd+THMTH3vAfQt1Y= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a81:5a03:0:b0:527:83f5:3f08 with SMTP id o3-20020a815a03000000b0052783f53f08mr494142ywb.425.1676524707124; Wed, 15 Feb 2023 21:18:27 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:28 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-14-surenb@google.com> Subject: [PATCH v3 13/35] mm: mark VMA as being written when changing vm_flags From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757963687799796810?= X-GMAIL-MSGID: =?utf-8?q?1757963687799796810?= Updates to vm_flags have to be done with VMA marked as being written for preventing concurrent page faults or other modifications. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index a056ee170e34..f4f702224ec5 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -726,28 +726,28 @@ static inline void vm_flags_init(struct vm_area_struct *vma, static inline void vm_flags_reset(struct vm_area_struct *vma, vm_flags_t flags) { - mmap_assert_write_locked(vma->vm_mm); + vma_start_write(vma); vm_flags_init(vma, flags); } static inline void vm_flags_reset_once(struct vm_area_struct *vma, vm_flags_t flags) { - mmap_assert_write_locked(vma->vm_mm); + vma_start_write(vma); WRITE_ONCE(ACCESS_PRIVATE(vma, __vm_flags), flags); } static inline void vm_flags_set(struct vm_area_struct *vma, vm_flags_t flags) { - mmap_assert_write_locked(vma->vm_mm); + vma_start_write(vma); ACCESS_PRIVATE(vma, __vm_flags) |= flags; } static inline void vm_flags_clear(struct vm_area_struct *vma, vm_flags_t flags) { - mmap_assert_write_locked(vma->vm_mm); + vma_start_write(vma); ACCESS_PRIVATE(vma, __vm_flags) &= ~flags; } @@ -768,7 +768,7 @@ static inline void __vm_flags_mod(struct vm_area_struct *vma, static inline void vm_flags_mod(struct vm_area_struct *vma, vm_flags_t set, vm_flags_t clear) { - mmap_assert_write_locked(vma->vm_mm); + vma_start_write(vma); __vm_flags_mod(vma, set, clear); } From patchwork Thu Feb 16 05:17:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57850 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp117065wrn; Wed, 15 Feb 2023 21:20:24 -0800 (PST) X-Google-Smtp-Source: AK7set86sFAi+jtFIzr/FxkIdGxfRND1GSvPCB7ZaSmvKU9Hywy6K+rurzNJHfe+2rqQwMZjVArl X-Received: by 2002:a17:903:2888:b0:198:b99c:f6ee with SMTP id ku8-20020a170903288800b00198b99cf6eemr4062251plb.65.1676524823947; Wed, 15 Feb 2023 21:20:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676524823; cv=none; d=google.com; s=arc-20160816; b=ylOORVRtEdGVV2AZk3oCitWu+RS8FOFqz0EWzzY2dTya4GL/yFaVuE/xvEaurct/mY XZ29UdMUzI8CSYOLUWrkAfBTZnfiqMSgk7TGa1fDuS6VXQWPyMGmIy8r0U+VcRY6rIgb h4WbA4ZBL0+9J7BVg6iVzatXRqC9TsQUwNCk4wpoJo16oUKpnKpKTm8TE2XjWmDcBFWf nIIwEbXVSD1ExnEOso6a9pmvWKqLLIlEp8YIky72Mz4NDwQB01JxrZY3b3C7wcdScYWC Xz10DtPGh2DNr1AMmozsYTDJZzdTb+42z8fWagJ85IAQRluRcrx6Qtbb/gVmsiAdP6Eq 4J7w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=pCPLSRzUXQbjlFnRzNKKqjCLNtpc7ieGcBzd48Gb76w=; b=Ar+JEdEihDGMWOoxwpnB5zXOdL5uQZRZ0GRMfq8UR8ZVJyp/eA0yLmHfYvzBLSxzKk XgqZ0zpfOvCtVBhBWinYiig1wv8Cu4TwEr1+guNvABEYGiGp8TmnxfgbcP5p+x2IhO0s aGRd3AVFcIPlv23tQ9G2pgr1pAKKAtdE1PGlZgps7eU5OYvS+HmGlnXas8ib++ApCbDj ywI9MuE2ExfVqtXAbnwrHV0UpmT9iGugNw2ftwNxc9O7GzoH6hmoQJw2tbibbRRR9ZEN nqFw1oWr3mUxrFBMVvYV4MTWFMZITGTup6YDYyeUca7CjuMpOMGnpjuJWcC+4J3YTVus hM8w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=Aww5hQNu; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id z8-20020a170903018800b001967afda550si624249plg.5.2023.02.15.21.20.11; Wed, 15 Feb 2023 21:20:23 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=Aww5hQNu; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229679AbjBPFTl (ORCPT + 99 others); Thu, 16 Feb 2023 00:19:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46806 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229724AbjBPFTN (ORCPT ); Thu, 16 Feb 2023 00:19:13 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 45E8946178 for ; Wed, 15 Feb 2023 21:18:32 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id p83-20020a25d856000000b0095d2ada3d26so832015ybg.5 for ; Wed, 15 Feb 2023 21:18:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=pCPLSRzUXQbjlFnRzNKKqjCLNtpc7ieGcBzd48Gb76w=; b=Aww5hQNuikXrQshFEtykppDfMBP38yEIiv8YZeHijhJepfYOC8bhl72PA7W7euJsgq UTdE3Yop6yuQ+QJ/imxEBpOD963eWsY1ljkA0jBaqGGg7cF+cENyfejTL2VbGXvtI8Gp zk9pQRaFettmR6L6WtIha8F6CvO6a31a83bm3SVswwIqM5PcX+z7DGQBjncH/5EAGpUc AT/XTvZe03bL8P4vyUJmA5jKJVVP1mrUYpMWnBW8gSUZQLx6c9BBl7Qtd+VfpJh9LBNJ HXZBHaakiRojDQMlDwmDxwqFzGyn5/4IdZKihfypYjTTs63kUZWxAGQtxOSq8MgWRU/t uPxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=pCPLSRzUXQbjlFnRzNKKqjCLNtpc7ieGcBzd48Gb76w=; b=pHXQo1pPLEDZiV1gU1KpP87jqWOjYqrO3mNpuQ1aVjTNbhPQkVG3sivFB6UcS/JPhT BTB+S+4Rmt+4oiV9V3mK1VmK5q20ZZVNkwDi5nDBlnUDchEg9cbvGzLihZT9PzUbtjzF hPyoumH4uxjWdkxXhHDLyD805++3HzDXRM2oJBHPIg89emyyfIiP8TqJGlb1DFgshNbK lgNn4BKHR/towu34S15Dyx8At9r6HcycAJhxUf12ncBSvPqiNkViSedFbp5JAvIX/z1B NotTDVZMHWi/dkNRgLdi+9Jb79RJ4V09cMqq9IdZ6iUW7gC7mnG2eMtwO0fSxzPCu0I2 vQjQ== X-Gm-Message-State: AO0yUKW1/cC3rRElFXiPEzFbK8BxTI8KXb8qKf/AaczCiqDrrRo1D5O0 j5ag6v6Iihbps/4aK5qHiSH40sBtQFQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a0d:d5c6:0:b0:500:e6a3:a141 with SMTP id x189-20020a0dd5c6000000b00500e6a3a141mr575676ywd.470.1676524709201; Wed, 15 Feb 2023 21:18:29 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:29 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-15-surenb@google.com> Subject: [PATCH v3 14/35] mm/mmap: move VMA locking before vma_adjust_trans_huge call From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757963693460181418?= X-GMAIL-MSGID: =?utf-8?q?1757963693460181418?= vma_adjust_trans_huge() modifies the VMA and such modifications should be done after VMA is marked as being written. Therefore move VMA flag modifications before vma_adjust_trans_huge() so that VMA is marked before all these modifications. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mm/mmap.c b/mm/mmap.c index 06c0f6686de8..c5f2ddf17b87 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2910,11 +2910,12 @@ static int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, if (vma_iter_prealloc(vmi)) goto unacct_fail; + /* Set flags first to implicitly lock the VMA before updates */ + vm_flags_set(vma, VM_SOFTDIRTY); vma_adjust_trans_huge(vma, vma->vm_start, addr + len, 0); init_vma_prep(&vp, vma); vma_prepare(&vp); vma->vm_end = addr + len; - vm_flags_set(vma, VM_SOFTDIRTY); vma_iter_store(vmi, vma); vma_complete(&vp, vmi, mm); From patchwork Thu Feb 16 05:17:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57852 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp117142wrn; Wed, 15 Feb 2023 21:20:40 -0800 (PST) X-Google-Smtp-Source: AK7set/fGhMbSwNCCdFjFx4PBhHUII0DhV8RoOryp5VLxFsU+2Ntg/pNs1ZhXeCZoCoss7a8ba7z X-Received: by 2002:a50:ec97:0:b0:4ac:bbb1:e61c with SMTP id e23-20020a50ec97000000b004acbbb1e61cmr4560101edr.1.1676524839994; Wed, 15 Feb 2023 21:20:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676524839; cv=none; d=google.com; s=arc-20160816; b=keErRDpSz5PWMOmunyqjT4Vf+KitqX6xI+TmWYslO8jlyFF5RMlPbXLEptEynB2/Cf sn2c4PLxZg1Pw33ICU1GcaemH1frl7uK0p89ewFeB3xVbYDq4hfMy6yPBzv1uQHxMxmg KnlrQ52E71LNQybiCeMmWwMwB0yd6Vct7hjSowdKg8rA84VhhKOZB15+hjpU1iSXGr5b LIELZbkESQ8z6tQPihgaQXKQ+jg5Bwwma5uMNE11i9f2tq4DP+JcIiDk84NwMy1DalvN ZDN+mZo6Z87V5Tut8YC+eP2MOOOjBNAq6nTN56EED4Frndo/0tcIm+5pU4ZzXl7zEtr+ BqIA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=4tPY63bpwhpT+tcOBCZmVWQ4EsQSsAiOumfwemoUSR4=; b=Am3o0haW5e3vZx8x39pI5GJr3RKemViqVKTRH4aR5Hk9AFIgtZHvhMXajUGLl0tTJH lmofaCp7MZnkypBU47I1CPBOZxzEJciLIbuLKihQfoQicLd9IE7j4pcK55I0wD6jbGlU w4ocjCkIfvs387cWgRDbekKcKsmbBpiS4CaAyjkJ7g5wLd0Sv9OoFQSzfTGxwbQhUXJx 8MdKKPNmFpi/lATBEx8vmOHXrBXWJvV3HIwxkX5zHryJ1xhO2T7NwnlpF9Oxs7s8vY9C HxpWpgUoUUk7zBza9G3F3mybEzc58gHUVWbANhcvkow7ceUaCIyOQoAtLbWqC+VA3X/6 A6xg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=S2sXkiCN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id d12-20020a056402078c00b004acb37caea9si993644edy.344.2023.02.15.21.20.16; Wed, 15 Feb 2023 21:20:39 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=S2sXkiCN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229687AbjBPFT6 (ORCPT + 99 others); Thu, 16 Feb 2023 00:19:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46884 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229736AbjBPFTQ (ORCPT ); Thu, 16 Feb 2023 00:19:16 -0500 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C0B0947416 for ; Wed, 15 Feb 2023 21:18:36 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-52ec8c88d75so8524207b3.12 for ; Wed, 15 Feb 2023 21:18:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=4tPY63bpwhpT+tcOBCZmVWQ4EsQSsAiOumfwemoUSR4=; b=S2sXkiCN7LWOjXaeaMYvz0qNJtP37xcl3aD2Yh+x1ZEOEg3KLdTaCbat1bdYvpjsne ssUpFmsgMZn2ZzaQIQ1nCUMhum1BKo1kVXXpN4w0/hSPSa2H/9hW6SiJ4urWGDrWEDIg /lDLOzKc/0bGJk2Y7O7P1pt9zjSY2dn9UPLftjGBDJCogvybLOdBJ/pVV0l/nLyc8nbd 78hCVT3xeGJZ3GfClpuKXysNAZLItz5CqeYcRfuccmKUwUq5LhC6V5ar511CeqKShlzR g15CQtYg8YR8Te7+dSB+30LV6maBKvWp8iWZepEStLZa94uN9djYv3KGIMR2EUqABQ+Q /xVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4tPY63bpwhpT+tcOBCZmVWQ4EsQSsAiOumfwemoUSR4=; b=1gSp1NhqKDSGVwAYOrs7UiLEtJSXs/IZIItXO+LeS/aSR5jAeqWBV1BD8ll43L61ZD 2mJyESaOQrLAJg7LYujyxfmj8o0Es8EkyaJp8/VAaOE2uU0Zqu5sZyK9PPtqpC0jN78+ KrJLQm1k4oN13qGwlatE4W30dlmUkhQEeRuW2Eqy/B758r7v0nYScKo0RgwoMh5yvTQa idQg6T4r/A6qXoFuY4oLsJ3ZYVY/L+EzAsMvfr0JuYRG7Rv+AaaESpKMvF5+VSL1khRl m8c40SwUTXYvTwvsgTnd51eQ/2Ce/XC1V6tNSlWAQGnJB+8aV0GI+B6WdVB5pJadWK25 /EEw== X-Gm-Message-State: AO0yUKVO2ispUTM5pBxHy2HkofDvcWn6bm6Ec5DczVN9HbhBMrK86TIj 4qgRcRIGlTKDLXQoQ0ooXPuLJ1WvOUA= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a05:6902:10f:b0:855:fa17:4f5f with SMTP id o15-20020a056902010f00b00855fa174f5fmr82ybh.1.1676524711465; Wed, 15 Feb 2023 21:18:31 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:30 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-16-surenb@google.com> Subject: [PATCH v3 15/35] mm/khugepaged: write-lock VMA while collapsing a huge page From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757963710804216537?= X-GMAIL-MSGID: =?utf-8?q?1757963710804216537?= Protect VMA from concurrent page fault handler while collapsing a huge page. Page fault handler needs a stable PMD to use PTL and relies on per-VMA lock to prevent concurrent PMD changes. pmdp_collapse_flush(), set_huge_pmd() and collapse_and_free_pmd() can modify a PMD, which will not be detected by a page fault handler without proper locking. Before this patch, page tables can be walked under any one of the mmap_lock, the mapping lock, and the anon_vma lock; so when khugepaged unlinks and frees page tables, it must ensure that all of those either are locked or don't exist. This patch adds a fourth lock under which page tables can be traversed, and so khugepaged must also lock out that one. Signed-off-by: Suren Baghdasaryan --- mm/khugepaged.c | 5 +++++ mm/rmap.c | 31 ++++++++++++++++--------------- 2 files changed, 21 insertions(+), 15 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 8dbc39896811..d9376acf1fbe 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1140,6 +1140,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, if (result != SCAN_SUCCEED) goto out_up_write; + vma_start_write(vma); anon_vma_lock_write(vma->anon_vma); mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, address, @@ -1607,6 +1608,9 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto drop_hpage; } + /* Lock the vma before taking i_mmap and page table locks */ + vma_start_write(vma); + /* * We need to lock the mapping so that from here on, only GUP-fast and * hardware page walks can access the parts of the page tables that @@ -1812,6 +1816,7 @@ static int retract_page_tables(struct address_space *mapping, pgoff_t pgoff, result = SCAN_PTE_UFFD_WP; goto unlock_next; } + vma_start_write(vma); collapse_and_free_pmd(mm, vma, addr, pmd); if (!cc->is_khugepaged && is_target) result = set_huge_pmd(vma, addr, pmd, hpage); diff --git a/mm/rmap.c b/mm/rmap.c index 15ae24585fc4..8e1a2ad9ca53 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -25,21 +25,22 @@ * mapping->invalidate_lock (in filemap_fault) * page->flags PG_locked (lock_page) * hugetlbfs_i_mmap_rwsem_key (in huge_pmd_share, see hugetlbfs below) - * mapping->i_mmap_rwsem - * anon_vma->rwsem - * mm->page_table_lock or pte_lock - * swap_lock (in swap_duplicate, swap_info_get) - * mmlist_lock (in mmput, drain_mmlist and others) - * mapping->private_lock (in block_dirty_folio) - * folio_lock_memcg move_lock (in block_dirty_folio) - * i_pages lock (widely used) - * lruvec->lru_lock (in folio_lruvec_lock_irq) - * inode->i_lock (in set_page_dirty's __mark_inode_dirty) - * bdi.wb->list_lock (in set_page_dirty's __mark_inode_dirty) - * sb_lock (within inode_lock in fs/fs-writeback.c) - * i_pages lock (widely used, in set_page_dirty, - * in arch-dependent flush_dcache_mmap_lock, - * within bdi.wb->list_lock in __sync_single_inode) + * vma_start_write + * mapping->i_mmap_rwsem + * anon_vma->rwsem + * mm->page_table_lock or pte_lock + * swap_lock (in swap_duplicate, swap_info_get) + * mmlist_lock (in mmput, drain_mmlist and others) + * mapping->private_lock (in block_dirty_folio) + * folio_lock_memcg move_lock (in block_dirty_folio) + * i_pages lock (widely used) + * lruvec->lru_lock (in folio_lruvec_lock_irq) + * inode->i_lock (in set_page_dirty's __mark_inode_dirty) + * bdi.wb->list_lock (in set_page_dirty's __mark_inode_dirty) + * sb_lock (within inode_lock in fs/fs-writeback.c) + * i_pages lock (widely used, in set_page_dirty, + * in arch-dependent flush_dcache_mmap_lock, + * within bdi.wb->list_lock in __sync_single_inode) * * anon_vma->rwsem,mapping->i_mmap_rwsem (memory_failure, collect_procs_anon) * ->tasklist_lock From patchwork Thu Feb 16 05:17:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57855 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp117227wrn; Wed, 15 Feb 2023 21:20:55 -0800 (PST) X-Google-Smtp-Source: AK7set+D1ZAaSVTRhNyZg6ZTjyW8wIMoI341zjXEwE+0vxCDG13h28Whim9PpxS1/pqGg5jSvjZ0 X-Received: by 2002:a17:906:5a90:b0:8b1:3f5b:af5f with SMTP id l16-20020a1709065a9000b008b13f5baf5fmr4248802ejq.73.1676524855313; Wed, 15 Feb 2023 21:20:55 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676524855; cv=none; d=google.com; s=arc-20160816; b=FrclPbp32n984la3zfdRnzx9MKTdgcJT/LhvCt7omaVTULoX+UMMAtWJinzklUSEuK M3DlH93XgKZhJgkoTfxHxY9u54mSPmPkr5p7a9d4VK/11FNcLYAmyd7DH5ohleequAoX EqnQ0150ppAOJj4uJYTiqvL9ituXXYoRGeNLa6DamHtPgLrbigh05VhFPOwDILSQZyr3 3TklNQgaHOxuZVvOvTNCZgbDXsSwEq92gUqyAP81wyTX+eTz0BR3FhgnYbKx2JZvhk+U 6dka287GwBajD23bCLrz3nOFpF8bHxgQX6zNdTbWVtLCxMYrEKYV7LIxxPFc8HOkkdxJ 6CKg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=28uvo2IqceKOssyZjY7i8BHUgiXFo9Fj5hjouTFWFAM=; b=nJdkjbfvuBUq2c++1dmGjmLg3nLboAFN6eBryMZzxl7fWhI8TsAoFuOSFOOVOJvstw F9IWwXqSk9BfuVUjmiRRZMyHKsfs/bhVutIp1SpFNbYHMJG/zkCxWzXxDWdhfTo0xsFz AD1Ly1FnLcgFnMzyTTVO8v37R6k87ATWXZkHOQxlHSZyMRdIJ+SYh5sek1F8dSBBUmUO bhtmdKP51XMcKJvm6ykBlrGrIonoqskoKNfeXyhx6WV6Vz8I1hAg54w8HWtDn153vnUy Q9e4xTnOh4FwVUETetiCZPN9whJMMZx6acb/Rvso6kGONeKOF03OdgFbTTxZ0pz7BDUS i0CA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=ZPrkg6Ab; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id vl10-20020a17090730ca00b008b1484584dfsi904669ejb.4.2023.02.15.21.20.32; Wed, 15 Feb 2023 21:20:55 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=ZPrkg6Ab; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229775AbjBPFUH (ORCPT + 99 others); Thu, 16 Feb 2023 00:20:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46296 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229695AbjBPFTT (ORCPT ); Thu, 16 Feb 2023 00:19:19 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B91EF42BCB for ; Wed, 15 Feb 2023 21:18:45 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id a1-20020a258701000000b008e1de4c1e7dso848079ybl.17 for ; Wed, 15 Feb 2023 21:18:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=28uvo2IqceKOssyZjY7i8BHUgiXFo9Fj5hjouTFWFAM=; b=ZPrkg6AbbvOKBZi1SkD/IxKxJeFUcG0rgRJFt48c6bsRhM/elzC2zYFPNxVAX1Pb/y v8gEIUO6XL4wPMpxHAJU/jtJ7FTBRhHG6n1tq7RCpsfo3BvcbN99La419Vjdnmr4LZUg Gb+YjubKYslJ6dowNgphM8TvAV/LOX1cmszsmnXszpI4IFG2RXQTfj4jwXf+4wC/qln3 8/CHWsl7LtqeSj/ymA2+araRUo+8VrDVeXfGOtKcUcgJXUwZ/d5G40s9s5ABK3v0GdSA ZKM8IbDd/wXIDY67hjDQxG31fd3mnWXrl8BZPhPKFflZXpp+nOaLG/s24K3lMQ1hF8se I12A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=28uvo2IqceKOssyZjY7i8BHUgiXFo9Fj5hjouTFWFAM=; b=tZZnuoU+6DGzX/CGijzGnY/iUPwL3wFTYD2DNuP2aSss06bywYwcoLbbFdJoY1QGT4 Bd4G0HITtMVU6UzG3HFhfYTACD0Hj48ZyqrTCFQt7Kez5P75uw5qiXTiW3b9Yrxusb7I XmarJgOj2j6Kg+rWymZoNhQUP/m05/QYw127mhXTqzLuBPZeSl4wI1LGdhQB42kGPMYT P7S1kVN1YbK41P7TXp1X81JQnUKbAroPpMEEYfZg4KsovPCK0f1cCR52AeaYWrHFu/vU DAW0dMN9ii6EtOuSZ7pnbAYoFM6eqtBjt27RGXQYuVOI+vxUnR0wiFU/zWWgnEGLZuWm pstw== X-Gm-Message-State: AO0yUKXU4bQj/n0NJG+6IX4yGxH1LWdVRoIVVgmpCF/9uZYwJHcLaDSo X6ItIgjNZeV9XB7OJ9WiLUdc4rF8lc8= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a5b:90d:0:b0:94a:ebba:cba1 with SMTP id a13-20020a5b090d000000b0094aebbacba1mr584331ybq.179.1676524713985; Wed, 15 Feb 2023 21:18:33 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:31 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-17-surenb@google.com> Subject: [PATCH v3 16/35] mm/mmap: write-lock VMAs before merging, splitting or expanding them From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757963726994704573?= X-GMAIL-MSGID: =?utf-8?q?1757963726994704573?= Decisions about whether VMAs can be merged, split or expanded must be made while VMAs are protected from the changes which can affect that decision. For example, merge_vma uses vma->anon_vma in its decision whether the VMA can be merged. Meanwhile, page fault handler changes vma->anon_vma during COW operation. Write-lock all VMAs which might be affected by a merge or split operation before making decision how such operations should be performed. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index c5f2ddf17b87..ec2f8d0af280 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -269,8 +269,11 @@ SYSCALL_DEFINE1(brk, unsigned long, brk) */ vma_iter_init(&vmi, mm, oldbrk); next = vma_find(&vmi, newbrk + PAGE_SIZE + stack_guard_gap); - if (next && newbrk + PAGE_SIZE > vm_start_gap(next)) - goto out; + if (next) { + vma_start_write(next); + if (newbrk + PAGE_SIZE > vm_start_gap(next)) + goto out; + } brkvma = vma_prev_limit(&vmi, mm->start_brk); /* Ok, looks good - let it rip. */ @@ -912,10 +915,17 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm, if (vm_flags & VM_SPECIAL) return NULL; + if (prev) + vma_start_write(prev); next = find_vma(mm, prev ? prev->vm_end : 0); + if (next) + vma_start_write(next); mid = next; - if (next && next->vm_end == end) /* cases 6, 7, 8 */ + if (next && next->vm_end == end) { /* cases 6, 7, 8 */ next = find_vma(mm, next->vm_end); + if (next) + vma_start_write(next); + } /* verify some invariant that must be enforced by the caller */ VM_WARN_ON(prev && addr <= prev->vm_start); @@ -2163,6 +2173,7 @@ int __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, WARN_ON(vma->vm_start >= addr); WARN_ON(vma->vm_end <= addr); + vma_start_write(vma); if (vma->vm_ops && vma->vm_ops->may_split) { err = vma->vm_ops->may_split(vma, addr); if (err) @@ -2518,6 +2529,8 @@ unsigned long mmap_region(struct file *file, unsigned long addr, /* Attempt to expand an old mapping */ /* Check next */ + if (next) + vma_start_write(next); if (next && next->vm_start == end && !vma_policy(next) && can_vma_merge_before(next, vm_flags, NULL, file, pgoff+pglen, NULL_VM_UFFD_CTX, NULL)) { @@ -2527,6 +2540,8 @@ unsigned long mmap_region(struct file *file, unsigned long addr, } /* Check prev */ + if (prev) + vma_start_write(prev); if (prev && prev->vm_end == addr && !vma_policy(prev) && (vma ? can_vma_merge_after(prev, vm_flags, vma->anon_vma, file, pgoff, vma->vm_userfaultfd_ctx, NULL) : @@ -2900,6 +2915,8 @@ static int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, if (security_vm_enough_memory_mm(mm, len >> PAGE_SHIFT)) return -ENOMEM; + if (vma) + vma_start_write(vma); /* * Expand the existing vma if possible; Note that singular lists do not * occur after forking, so the expand will only happen on new VMAs. From patchwork Thu Feb 16 05:17:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57854 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp117223wrn; Wed, 15 Feb 2023 21:20:54 -0800 (PST) X-Google-Smtp-Source: AK7set8yGsC4ihXOZJ/ldx2rxoGveBlfYgjhQqfbnVp56wod2GSA68wKwxCvzhePKv5q1tw3M8Ru X-Received: by 2002:aa7:d3cc:0:b0:4ac:bdf3:36ca with SMTP id o12-20020aa7d3cc000000b004acbdf336camr4427328edr.10.1676524854242; Wed, 15 Feb 2023 21:20:54 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676524854; cv=none; d=google.com; s=arc-20160816; b=A9MAIMrdf3lSIjW1AnFLg1NvAVWBOjILeogc+Tj7MYjbMYK29wSjvIPLhZMSXU28MA zlM7JT7FSi3MPOSmxSktas3luEd4MQfVLMJFxKTsKWO0hDgzRJ/oszV/ygLZzoTqkvxi KWdPUrghwR1QafykjJTYOMRwOMHH+X1tPh1mvA6VUAWmmeaZ9TtNDEgVwO6JLyn+spgA YyPKPMwy3tJDG3JylVcB2gOwgMNTtvPhX4ELIb+Og3RQyGdEudJoSTYI82N/3foEgi0h 5toH7SCXUy1VzWhjBItCD5cORJtH8Yw+flEYRXEyCFXHN/zBhA2BMLeabrz2zbVZT1Y0 Dr5w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=Gs5XDlSmqlz144wCEPg25vfe9kudkVyQE95J1Pewv8Q=; b=V9c0hrYVwDDnB4o2fXPx6OWpjDfmRxPrrB1RO03FeX3fWis9zxQauZoOUH4nldCdF0 KYENjEWhtH1oDa984MqAcZodBo4TN1EQbpIPh+E0WX/CAi0iW6JvXPqiNpDb8FSVx47K aE+8ftMDs76e6xufWp4ZMiSibVIMVCTEChNOp65NUSjcM084WFXPNcjKj/0nmHbXanlR 5UG3SoDVas/GMrn5vDuHeZDtbepi5ZoisI2kLvEuZke81/eunfbKcxWKoaNYcTyOhG49 XpfTMtZ8tzTLxwtniNy/ImAOFdmo4tnb/uGiGgXbVZKVyX0e5/AFcayh17bvRmxzrh1J nuig== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=NZSRPVmV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u20-20020a05640207d400b004ad1256d8e3si1070192edy.51.2023.02.15.21.20.30; Wed, 15 Feb 2023 21:20:54 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=NZSRPVmV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229780AbjBPFUD (ORCPT + 99 others); Thu, 16 Feb 2023 00:20:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46294 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229746AbjBPFTT (ORCPT ); Thu, 16 Feb 2023 00:19:19 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B97D742BD8 for ; Wed, 15 Feb 2023 21:18:45 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id p7-20020a257407000000b0091b90b20cd9so869939ybc.6 for ; Wed, 15 Feb 2023 21:18:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Gs5XDlSmqlz144wCEPg25vfe9kudkVyQE95J1Pewv8Q=; b=NZSRPVmVQg07iGqxuaNSoyEyX/JUlE5074LqD7StZA8wPOZx80SXxNMt5E5KxxrtoC 5GmbXen1kftSSNBaHdmndb327iEU+JgvGlsjv+jrut0kw6bBF6uNimcKzz4dFOSEItYW Y4LEl4ERFvkTfh+pRssZBSprx0WHPyqr3iZkWvDeufJqDlNo5Dl9o53Xv9skjZ4WRCg9 pPw2zfwLYTP83BBCg7T3fM4u8tR26MmRDA1XMM3A/clmWDV/BI6rgpWB66aseND2aMHJ sak6MgPcdV5XEwJQs9ETMkutqiCQEhW6TAM4KAr4SAjK7KgYItsJ/rcb5RNyZGDs+f6q Q5lA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Gs5XDlSmqlz144wCEPg25vfe9kudkVyQE95J1Pewv8Q=; b=z5HT4SprMcPNuNcT3lm4djTjfO3QyMjxUIKUeVcycvYqZ9+Yhz5oyJyKR93sl/bIxJ hZbvWKKE1WpGQuDnImLs2vScmX/viRzHYfK7MomFeykWIbeTaA92ell6w58ykjrFdsbb +PtuSkGY6Qq3FCWGCygCUsxir5cHEoSxmZ/Xb5SgZOLlvh7QDmt0Qo1lTyJBP3YoGutp RsEadFbhoNPhtJIrUI8BOyVkOCUfMi+NfOcX9d5yS8or9eE3hizsw8TadZ1AB49L+MA4 AGfMwyUBJQpCCweZk2pCiFgRpvizP8sNebKxIP9njMMHKcSoZwqs7myyMyPUNc/OjzLT iCUw== X-Gm-Message-State: AO0yUKWaWg8YbSJgw9u+VRqFYHL33ZJsinlugrWsfA723BW8PENHxxPN /QrRTGxHx7kZy0WEdE26EllXwamHcrw= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a81:9345:0:b0:52e:e6ed:30bb with SMTP id k66-20020a819345000000b0052ee6ed30bbmr607996ywg.571.1676524715849; Wed, 15 Feb 2023 21:18:35 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:32 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-18-surenb@google.com> Subject: [PATCH v3 17/35] mm/mmap: write-lock VMA before shrinking or expanding it From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757963725116033155?= X-GMAIL-MSGID: =?utf-8?q?1757963725116033155?= vma_expand and vma_shrink change VMA boundaries. Expansion might also result in freeing of an adjacent VMA. Write-lock affected VMAs to prevent concurrent page faults. Signed-off-by: Suren Baghdasaryan Reviewed-by: Liam R. Howlett --- mm/mmap.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index ec2f8d0af280..f079e5bbcd57 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -674,6 +674,9 @@ int vma_expand(struct vma_iterator *vmi, struct vm_area_struct *vma, ret = dup_anon_vma(vma, next); if (ret) return ret; + + /* Lock the VMA before removing it */ + vma_start_write(next); } init_multi_vma_prep(&vp, vma, NULL, remove_next ? next : NULL, NULL); @@ -686,6 +689,7 @@ int vma_expand(struct vma_iterator *vmi, struct vm_area_struct *vma, if (vma_iter_prealloc(vmi)) goto nomem; + vma_start_write(vma); vma_adjust_trans_huge(vma, start, end, 0); /* VMA iterator points to previous, so set to start if necessary */ if (vma_iter_addr(vmi) != start) @@ -725,6 +729,7 @@ int vma_shrink(struct vma_iterator *vmi, struct vm_area_struct *vma, if (vma_iter_prealloc(vmi)) return -ENOMEM; + vma_start_write(vma); init_vma_prep(&vp, vma); vma_adjust_trans_huge(vma, start, end, 0); vma_prepare(&vp); From patchwork Thu Feb 16 05:17:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57856 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp117245wrn; Wed, 15 Feb 2023 21:20:58 -0800 (PST) X-Google-Smtp-Source: AK7set+pIdHGtBFql9qcOXV596+jinQTyJSLOvEARI65aStyRHFOSd2JtMMCHI8Dazbqb/R5OTfK X-Received: by 2002:a17:906:3c5b:b0:881:d1ad:1640 with SMTP id i27-20020a1709063c5b00b00881d1ad1640mr5050813ejg.57.1676524858397; Wed, 15 Feb 2023 21:20:58 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676524858; cv=none; d=google.com; s=arc-20160816; b=HnXIijpJDAwdHhrtpj6AWneBLhG6hzwB+/tV7/LiqtJJKv8Gu7+CWsEenxtOiwcWGu GbkGUIm96H1EEN8bhDGKu108DWG7xIl+iW3zaUc9m45BjhorVhvotQ/3sPoDvQNwlhl2 OYx4wmhitRW0dqu5Bl562qig8yQbV+IjteJ+rVZoz6ayf/mHP9HHZr4GHULPfWuCILiD yhGe+2chmoFRal+cAJjPHUN9/uorn4VnsH5mhpy4UgfLMsXrguuAtKa1pSOwNyqXzrRQ F8qu/ih/umhM9ZCpahYhUrxeoutjqWAG1UxV6jo4HsUtrHe28XDQNRYEISyB87EA4ekk oghA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=jGkERMphMFi5DTXO6AzCtASC+DQ2dhGXIGrkfltmA+k=; b=fGTjOPm7msWDoN3GYjBvWgTHqltGX+akm9+17UGDcX+1AH9fE8g86LUIkTutyRZH3K byMfLpiz/Ww6bpfPY6tRX9bkN4ekkVMbxnmowxqk8E/Y3++je2LZ32S5VHXr6ppbHtzb 8Egaq0QySsKdIqT0mTjqek/ARTERhu2/Xl9Fc8kelAS++X/OLh7BZ2SbWPIEK0QwWqsz BrpSVqrmMncLvB4Ug+jN/IpiAa/WeL+U5Xdhif9eC0pfyiZqtGqiZC5I8+0tEeRic0qK ximP1H2WmzIvjSZoh3zb0R6Xxm6Cj7mB3sct38dKW37gRLbj0aLs4ziN/IrTJgbglF26 0w5g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=ClKHnTmR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id et19-20020a170907295300b008795db50bdcsi748391ejc.934.2023.02.15.21.20.35; Wed, 15 Feb 2023 21:20:58 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=ClKHnTmR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229793AbjBPFUK (ORCPT + 99 others); Thu, 16 Feb 2023 00:20:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46326 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229652AbjBPFTU (ORCPT ); Thu, 16 Feb 2023 00:19:20 -0500 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B942642BCD for ; Wed, 15 Feb 2023 21:18:45 -0800 (PST) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-527501b56ffso8390927b3.15 for ; Wed, 15 Feb 2023 21:18:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=jGkERMphMFi5DTXO6AzCtASC+DQ2dhGXIGrkfltmA+k=; b=ClKHnTmRqrwFsiwdvZoHbs8UrIxrOZcnAlLMKj0DmY4EeItAiZvM3xF/1XkOIILsHj t/ncp3cCJ608udelXLFUJi2J2Bkc4zBNFEFw7AtqnW62B+1ruBsojOMJ+l//Eveg6kMb 9mESDdWGEfVZo4wbqdchVNx7evryh4U8zH6lvFtZPteOd8Fw20lUCwc+q1jxjD4dRCfO fPscX4k3EE8nShVcC7jG6Imy8fv3tD4a5LzQcDPbvgxMAF9FYu61sb1sXCbMYbbYMnzw H0gcnPTwNkiDCSFpe5MW+pIUr82O7ftWGw19nAsT/pwTKfJQo1rJfp9QBdZ/qdttHeJW nqYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=jGkERMphMFi5DTXO6AzCtASC+DQ2dhGXIGrkfltmA+k=; b=qtRDFKEJip7gsG4CTysLnnszyVJIiuQkxWTuCv1I1nfYMUON81OK6T3Vy//J9mff9/ i/XJXM6XqVacJaRsfYxvGXsAsxBQ/HH3YH3GCzeqS6SqT6rcLs3VO3c3GUvX0Wx5zZyo lE2Tq26qVNoMV0E8TS4bpUoHne9Wwg4pli56CK0SzWcRdtyGUq5VfJ4xclJZPHnV+oqu M258AcyziLknuvpaFO5Zv7rbRAH9rLx9ree/UkTEo4ZLxSRV+X5+P98C9Qk+pbEy3Tvy GGhG+XQXRAvjyLXWn3bxq3hYL3VxcEycT2dTM64s5KTUkj3nOEjcYFaF+9XOqqLV7kNT SlQg== X-Gm-Message-State: AO0yUKWHw/wvH2cYthaA07nSK7TRG8MBnNZtkJvsJGvPdx/y5f0iFopj rfsT+I2Ur0X3/7R+yzQmSSALSn/eTbY= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a81:ac54:0:b0:52f:45a:5b00 with SMTP id z20-20020a81ac54000000b0052f045a5b00mr11ywj.2.1676524717784; Wed, 15 Feb 2023 21:18:37 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:33 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-19-surenb@google.com> Subject: [PATCH v3 18/35] mm/mremap: write-lock VMA while remapping it to a new address range From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan , Laurent Dufour X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757963730424924374?= X-GMAIL-MSGID: =?utf-8?q?1757963730424924374?= Write-lock VMA as locked before copying it and when copy_vma produces a new VMA. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- mm/mmap.c | 1 + mm/mremap.c | 1 + 2 files changed, 2 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index f079e5bbcd57..0eaa3d1a6cd1 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3202,6 +3202,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, get_file(new_vma->vm_file); if (new_vma->vm_ops && new_vma->vm_ops->open) new_vma->vm_ops->open(new_vma); + vma_start_write(new_vma); if (vma_link(mm, new_vma)) goto out_vma_link; *need_rmap_locks = false; diff --git a/mm/mremap.c b/mm/mremap.c index 411a85682b58..dd541e59edda 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -623,6 +623,7 @@ static unsigned long move_vma(struct vm_area_struct *vma, return -ENOMEM; } + vma_start_write(vma); new_pgoff = vma->vm_pgoff + ((old_addr - vma->vm_start) >> PAGE_SHIFT); new_vma = copy_vma(&vma, new_addr, new_len, new_pgoff, &need_rmap_locks); From patchwork Thu Feb 16 05:17:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57853 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp117190wrn; Wed, 15 Feb 2023 21:20:49 -0800 (PST) X-Google-Smtp-Source: AK7set8Kzh36+63Pu99WAchvV8sBmq4nDRkY1sdnfCgr1awcw16RUAKABX4vvSv+H6h95GDZlPdP X-Received: by 2002:a5d:6e0a:0:b0:2c5:594b:10d5 with SMTP id h10-20020a5d6e0a000000b002c5594b10d5mr3384795wrz.1.1676524849123; Wed, 15 Feb 2023 21:20:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676524849; cv=none; d=google.com; s=arc-20160816; b=mXtWlExpLSVXn9SgigrDmOgWMlgrfTzpa0oOsI/XoAs2fYFPc9Lnruig10x/wnSf83 nAiWyFjzFrA43ooAum2LWGBqdC0RTS4PoumidFjSQ6FNccUYAqwfu7mLYAPNBd0/1ROg hey9a5OgGaOKoGGZMeIK+5IXifjAe702OonokuQZ9oauRXNWmxjdqmFLLTmBfFYmE69F 81P4eZJZ/L201utUJBwSn2YvCn5oYSy2QTHNYeDr/NRqwwkQFLoAUsxR50hWC7wqKdE+ 8hDJsJjlSlLk77fjN1GXvK4+RGAo97wpC3piYsZ3QMzs+r/BPYuRqgnDOQY8uOY1nLrd GjQw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=rL5PSjChr8wFTlKRWTO498bp0kCwADLZZe/SXHes2h8=; b=k1VyqhdVg7yAqDGdgFZX0vufDj1cIeJm47qoytyBvk9gEVTlCakUXbAk6dr0jnOMdT tH8BIDNKfEpej2WK+YBdSKWhXBLQuakg54jGnqdRxxAwMfW2D1PEjsStlmOdNlXSaU2y 3E6RW3TITbrKcXFqhSYcKzWeHeangVPBGuSULE+ZEzlZ3rFUwihkn6Raav+2sdb7EZRr uPaFfNxX+GF/t6y/s+G9QS8ZvjTS79QfuAYssRlaN+WcCVdfLiPyui/wbLb2NzOW3e3w PXGu6VWdn2Ef8bMTInVjGigVMrRdGFp5/fMXY5RJJ3PgrMlWgQjGuyIXMqB0CEegZSkh Mr/g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=LRD+lTaX; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q15-20020a056000136f00b002c5568e2924si810722wrz.501.2023.02.15.21.20.25; Wed, 15 Feb 2023 21:20:49 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=LRD+lTaX; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229749AbjBPFUB (ORCPT + 99 others); Thu, 16 Feb 2023 00:20:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45784 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229698AbjBPFTT (ORCPT ); Thu, 16 Feb 2023 00:19:19 -0500 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B971D42BCE for ; Wed, 15 Feb 2023 21:18:45 -0800 (PST) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-527501b56ffso8391427b3.15 for ; Wed, 15 Feb 2023 21:18:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=rL5PSjChr8wFTlKRWTO498bp0kCwADLZZe/SXHes2h8=; b=LRD+lTaXr6Pqw8LYMkej56Kfm91pJviFVpFaUDR17WEIFgUKOpU+ogLRfy6oidvw/F 4mOXwIhPJkh7gxaJzacQ5Q2xpA2MxMudctw4pQKYtuqshuSCMBeCw647b99z6JqUfMYU FnWI7aY3RIFvc0EuIZqwF4n5sAACqaxfMc5BUHOhpo7PrLkybX4Nw85DLStMxOu85Den S+x5aTXsnUr3Va/T4IBtQXPzTR4iMs6msv64zgKWL87HLYoK/sTJqZLT5fditOnzScG6 JUQkoM18jflZAeAGFMAkFQi7v0aeD9Evl42tB/J4ec9dW/Jbo4ktb05auTom3OB08iqz 18AA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=rL5PSjChr8wFTlKRWTO498bp0kCwADLZZe/SXHes2h8=; b=oJDdWhN0RQy9I/hsb4XFwsiK2vAh7zeStY3CwjBy3BWD/hn37uUGYL2UJ0GKdWoEaI BnB4TqXN5H2nAdkVmwido6KH6uBNWt55Amd1shMzEMdUDWTGCWMOcz/PoqH3oxBd3RAf qAD+hyCp3M/QasPeWvDxw1O9B9woITynPJZJ1awqJj+/QKz3O0fZGczViUJJ+lRjSOQ8 9PA67umAd/pORsWQMmtkIr8ZcuF0UdsV5o0PvtSLqnSBMooWcYMRcEprKwnw8+p8NpZ1 aouNk1eEhYm4uriF9ly5mLigDEm+ge5jQJknwn6XY5Kc8eE6UbFkIar+4JWbPz9/qsjw 0scw== X-Gm-Message-State: AO0yUKXm3KAlMKeaGDPmp8mMIFRS7KprUtNB0dIVtILxRKA6lMPMbT8W KmT7TbMb6FsQQH/n5RQhUJIWvvwf6wA= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a0d:e506:0:b0:52e:c531:20b3 with SMTP id o6-20020a0de506000000b0052ec53120b3mr448676ywe.147.1676524720915; Wed, 15 Feb 2023 21:18:40 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:34 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-20-surenb@google.com> Subject: [PATCH v3 19/35] mm: write-lock VMAs before removing them from VMA tree From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757963719872185053?= X-GMAIL-MSGID: =?utf-8?q?1757963719872185053?= Write-locking VMAs before isolating them ensures that page fault handlers don't operate on isolated VMAs. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 1 + mm/nommu.c | 5 +++++ 2 files changed, 6 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 0eaa3d1a6cd1..aaa359a147b2 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2261,6 +2261,7 @@ int split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, static inline int munmap_sidetree(struct vm_area_struct *vma, struct ma_state *mas_detach) { + vma_start_write(vma); mas_set_range(mas_detach, vma->vm_start, vma->vm_end - 1); if (mas_store_gfp(mas_detach, vma, GFP_KERNEL)) return -ENOMEM; diff --git a/mm/nommu.c b/mm/nommu.c index 57ba243c6a37..2ab162d773e2 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -588,6 +588,7 @@ static int delete_vma_from_mm(struct vm_area_struct *vma) current->pid); return -ENOMEM; } + vma_start_write(vma); cleanup_vma_from_mm(vma); /* remove from the MM's tree and list */ @@ -1519,6 +1520,10 @@ void exit_mmap(struct mm_struct *mm) */ mmap_write_lock(mm); for_each_vma(vmi, vma) { + /* + * No need to lock VMA because this is the only mm user and no + * page fault handled can race with it. + */ cleanup_vma_from_mm(vma); delete_vma(mm, vma); cond_resched(); From patchwork Thu Feb 16 05:17:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57857 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp117335wrn; Wed, 15 Feb 2023 21:21:13 -0800 (PST) X-Google-Smtp-Source: AK7set+BNfNUaWf5DPPqBr2OXe6d59HUKzQ6AlVC9joZqFx90wzLW4KS/8zDDcTjrem2R6MzzZL2 X-Received: by 2002:aa7:d74a:0:b0:4ac:c125:3f97 with SMTP id a10-20020aa7d74a000000b004acc1253f97mr4384544eds.31.1676524873528; Wed, 15 Feb 2023 21:21:13 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676524873; cv=none; d=google.com; s=arc-20160816; b=mb+lg1WRtjr+QhSF0dIBUV5wtdEJDNc7ftv+RFJtTOjmIJqD82tQ2DfGnMah+oLy0e Uw657AQem83wP+DqBI8/IGi8rKYz3UcPZ6PYRX6QdW6DC1g7p+aSVqUPicqCDL7Rtasi 8OVzhpc1reP2dxew4P4/Uzaz++AfNs84AKeBBs4hACjBr5tjE/Qfw2whbR6LDlFBNzrc ANlCAuvD/t/6I6+LLSQPd6Vd5T16Y5MSzXoiqmlXoumknffyaj+gnr4drF9NbA2YIMDk UyAOd9SgcoHXOCxcL82Zy2rc9iJyT8G5s/SFq05SSbPEbfvfQPa4Riaz7ZX+QppFOl8v UZFA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=65rn53nYPCehqawxI5VHHUH5OXR7IFklQh11X90VOGk=; b=eIjyD3y7J5u9iDOs7WWslz7tR5xgFTQ029qlhpCWs1ceH+LDqL7pgtFOOrqXV9cn9/ /aLZpxA8RcTqLVRq24+Z0B+SWnWzuS4WMt6JGIwa499P9c2mVuRKbPwrc4rj0g/NaB2c qwHiDYuCuzI4gKT9y7P1dWX31bXbk4ELmdi+aMbrJ0LGXhWgyz3WNFpE7D7k4KuuGtCV 5PMxyZ8pH14D0wAPMCWjfjQh9V0q6cqvZnwA/kn9ljpyJ4L2TrQuxqzKnZOYbHt0o5Qz H08+S9gF4xBHPjKFLl3Li7KHIclqVrWKDcqQVsTF0nphEfdx4MhI4QsX1baiqay3UjMf vOIg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=ngrSSnFG; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ca8-20020aa7cd68000000b004acbf1d6be0si1077972edb.107.2023.02.15.21.20.50; Wed, 15 Feb 2023 21:21:13 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=ngrSSnFG; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229478AbjBPFUP (ORCPT + 99 others); Thu, 16 Feb 2023 00:20:15 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46124 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229627AbjBPFTV (ORCPT ); Thu, 16 Feb 2023 00:19:21 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B98BD42DCE for ; Wed, 15 Feb 2023 21:18:45 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id k9-20020a25bec9000000b00944353b6a81so863507ybm.7 for ; Wed, 15 Feb 2023 21:18:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=65rn53nYPCehqawxI5VHHUH5OXR7IFklQh11X90VOGk=; b=ngrSSnFGdDPlYz5zuJYWd2ZcJdDDxwAT/RtnIL5xzVfAFVgHCKIj7A9xwEDcTyX2aB 3vsZSlkeyPVpcwi1aa9l93eu/icfxKL0DIaqQVOS2lNklMJwu2jpUpq532q1GD3mJcXZ OYyJWX25F7RdsRsm0bO+ULQDs6kU+hJx2yHcqT8VTisE8SVpLjiLy4deYJxyu/llv5yd uNDnBwf4OPlsNNTMONtuv4dnuvjdk8hbLw2jK3U2tc3OA2ZTvVJ3/lGJPyy4Jcy7H5Vt +ZqHtuuBA7mxmkHLm02ywaCux2tqtjmQCbr0/F6E0nSC73lfB2u+G7wX3tsYtQZiJlY+ ecqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=65rn53nYPCehqawxI5VHHUH5OXR7IFklQh11X90VOGk=; b=If914xyNZ6E6JKn7JwPiydaFH9m235YeOJJjL/0kh0TjuFOcDzgmXtRrXJhjCMHDBo SewVQlA9iQFZBoC2ydiSQscZ46Vt51irOxbqdbPemMpg6t4QQ9fjhsRimf33SdhFuSVw rE/6vInHrhZLQcPnU590LauLe1Vup2MsFv1Rnb5q8DVXWBZ3YJKLkF37lWqgUfHm9uTx Z3qzTY3yMkJQ4I4Mw9VJOc1bKaiA5eGnJTVjUUrbdkUMH12hp1OWvbTDI/KEhG2nMZQL 58Rs2Fp9jmqQiE4dyZHQS0WDg8/zpwDvWr+KZt4LjyoPcPcbv42jkbo2eDdN9zBK7AwB EM+Q== X-Gm-Message-State: AO0yUKUKFoUUb+u5EjrxgeHJRO/K9ZmhN9fdKH/eDARrjH331nWPjWuO GQcXRMn9bh/ItRfqN+A5OKCVL+/6IAE= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a81:b704:0:b0:52b:fcf0:4fd1 with SMTP id v4-20020a81b704000000b0052bfcf04fd1mr5ywh.1.1676524723168; Wed, 15 Feb 2023 21:18:43 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:35 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-21-surenb@google.com> Subject: [PATCH v3 20/35] mm: conditionally write-lock VMA in free_pgtables From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757963745466411750?= X-GMAIL-MSGID: =?utf-8?q?1757963745466411750?= Normally free_pgtables needs to lock affected VMAs except for the case when VMAs were isolated under VMA write-lock. munmap() does just that, isolating while holding appropriate locks and then downgrading mmap_lock and dropping per-VMA locks before freeing page tables. Add a parameter to free_pgtables for such scenario. Signed-off-by: Suren Baghdasaryan --- mm/internal.h | 2 +- mm/memory.c | 6 +++++- mm/mmap.c | 5 +++-- 3 files changed, 9 insertions(+), 4 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index fc01fd092ea5..400c302fbf47 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -105,7 +105,7 @@ void folio_activate(struct folio *folio); void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, struct vm_area_struct *start_vma, unsigned long floor, - unsigned long ceiling); + unsigned long ceiling, bool mm_wr_locked); void pmd_install(struct mm_struct *mm, pmd_t *pmd, pgtable_t *pte); struct zap_details; diff --git a/mm/memory.c b/mm/memory.c index f456f3b5049c..8177c59ffd2d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -348,7 +348,7 @@ void free_pgd_range(struct mmu_gather *tlb, void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, struct vm_area_struct *vma, unsigned long floor, - unsigned long ceiling) + unsigned long ceiling, bool mm_wr_locked) { MA_STATE(mas, mt, vma->vm_end, vma->vm_end); @@ -366,6 +366,8 @@ void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, * Hide vma from rmap and truncate_pagecache before freeing * pgtables */ + if (mm_wr_locked) + vma_start_write(vma); unlink_anon_vmas(vma); unlink_file_vma(vma); @@ -380,6 +382,8 @@ void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, && !is_vm_hugetlb_page(next)) { vma = next; next = mas_find(&mas, ceiling - 1); + if (mm_wr_locked) + vma_start_write(vma); unlink_anon_vmas(vma); unlink_file_vma(vma); } diff --git a/mm/mmap.c b/mm/mmap.c index aaa359a147b2..118b2246bba9 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2157,7 +2157,8 @@ static void unmap_region(struct mm_struct *mm, struct maple_tree *mt, update_hiwater_rss(mm); unmap_vmas(&tlb, mt, vma, start, end, mm_wr_locked); free_pgtables(&tlb, mt, vma, prev ? prev->vm_end : FIRST_USER_ADDRESS, - next ? next->vm_start : USER_PGTABLES_CEILING); + next ? next->vm_start : USER_PGTABLES_CEILING, + mm_wr_locked); tlb_finish_mmu(&tlb); } @@ -3069,7 +3070,7 @@ void exit_mmap(struct mm_struct *mm) mmap_write_lock(mm); mt_clear_in_rcu(&mm->mm_mt); free_pgtables(&tlb, &mm->mm_mt, vma, FIRST_USER_ADDRESS, - USER_PGTABLES_CEILING); + USER_PGTABLES_CEILING, true); tlb_finish_mmu(&tlb); /* From patchwork Thu Feb 16 05:17:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57858 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp117344wrn; Wed, 15 Feb 2023 21:21:16 -0800 (PST) X-Google-Smtp-Source: AK7set/zI/4joEe1zWsOsLqXd3Z+0I9Hhz19c1xnXNC3TenunPTIfpkEnGQse8x0H7BvwmPnfPYx X-Received: by 2002:a17:906:4e84:b0:889:58bd:86f1 with SMTP id v4-20020a1709064e8400b0088958bd86f1mr4936030eju.14.1676524875912; Wed, 15 Feb 2023 21:21:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676524875; cv=none; d=google.com; s=arc-20160816; b=JbcmhBGfB3PpG9kgSRLwbBZnVt+IIne3dt8OSA/KiCCvwYFdzPuAUqzn0EtHvAjolh riyLzAhjJC5VeYLCtpH7NZxGzwYOXaCozuKD/X5i7liWdg0BQdVTmB0mf+GDPI3UuB1y n2ivD773A1REqzmCfv+5C5XChQg/Iu13RGRwZunKKdEy57mxsuNx4DgNgVMjXPyoCEeR 1c3DC4nQQVgR38dmHw1975Cap9/ypn4P5GgDFlHHs1Rf2ubSdEi1z3NBIq8MK8Qw7nGc KA9+dqn5nj1fmLs9YfRrMR03z+ddLh5/noTlPeM+MUc4sXe4Kjt/C2zl2mX2X7K01k6V nMuQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=zeEKIs6RouZ3NeHI7ZJ8m5Jmv1yLdTyFBRJo2pt5tCE=; b=SKDfHpyEQjARw5WaQvHQfaDulxquPLxfpl09Ex6ZoOZxlvEVtaPHkthP5M5Eb0Q1uk G2fPNwMScbMXB88RbToWxiGhBN0A72a3XU+EJ94yYgdQKYGnm2jECvD+glYCCswRFtIL U8D/1dTjh9MKPS1QxdXcoYzrIqPYFgb3Puyo2gN/GGeJAY2MYUX+8zlpXjp1gPD7waEq SBO3huCuxvLQce1KFJsmjEhKnrTSpJRB++AsuSSaJVZyq8r2i/GlD1iEzf39tKwDnuoo oHmZK0rPaOmaKA6xeutkqSDCMw/kQReGoVsQefrNP+Z+eP1WBAQ/sdNC5XpaJIb4ohVz DTAA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=Hd3NZlon; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id mf8-20020a1709071a4800b008b1384001f8si768368ejc.520.2023.02.15.21.20.52; Wed, 15 Feb 2023 21:21:15 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=Hd3NZlon; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229816AbjBPFUT (ORCPT + 99 others); Thu, 16 Feb 2023 00:20:19 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46396 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229623AbjBPFTW (ORCPT ); Thu, 16 Feb 2023 00:19:22 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5931636FC0 for ; Wed, 15 Feb 2023 21:18:47 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id n20-20020a25da14000000b008fa1d22bd55so840318ybf.21 for ; Wed, 15 Feb 2023 21:18:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=zeEKIs6RouZ3NeHI7ZJ8m5Jmv1yLdTyFBRJo2pt5tCE=; b=Hd3NZlonbpCzy5K7Z5feVeMneevLupNr2wIBk8xrF2F0FeVHuTW7v0p8Iwz465OudO Kz/cwziHnFST/NwsyYv9H1NpjXxfHSGLxO5ukc+3xsoPYXaSl8+RA3RdmvmM4fbLpXN9 gimJ3OxJZ4Ep64JmtioAuz7PGMRMavPtJuthlMhEIfRvRDzYwwXEt+kIY/UhQ/1ohnXV Mp7Yg73p6F/OdwNe1qmUkwSe8yYAoyBy/8c1r/0Zsdj45B+B6DEhDKlW7MvfqHG54uOt TWogzsTy9UemTUaeFAzTdpALJEgq+fwHHnoe+QVimQT+Dp4bg9S2kMJXbPj9+Gb2VnGT o48A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zeEKIs6RouZ3NeHI7ZJ8m5Jmv1yLdTyFBRJo2pt5tCE=; b=FV5sNiMFzNHCNtvK23N/vPhSa88H749KUN2uP8iGhzQx7K+wYFs1fqloQJUvbcXwwg R6OKWNFPaTzj3aQ2TYOncfxBOFMPMgNgy26wQgmT1raRUN6gkZDZsu+ISTS/H0Z67FfD y//3p7Ht8NcQcJOsE0R5EKGDQBbSVAmN5Jp5V/fIvBJFu7WIKaMjqfJeR/XbsZ3ugZ9O Gf7cNJgPhJhHE/ssF9UCDaxtENN77oeH8NbNvBolB80oYXxW+QFw3sgRU+bDxqDmkptv Y19RdCjRxtkZZRhGP3oFjspef3lPMMBzFiMjQ+Cl8BTrRAWDnCYOqQo8/9RKAG8c+PBs lGKA== X-Gm-Message-State: AO0yUKXXbwg/WF7QEdet7cIP1MfWQvGumGsRur2t0XDBnuXUE5A3xB82 /BFlWIVTGqFtpiO7QVMvyvvaD8FvLws= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a5b:604:0:b0:859:2acc:deb6 with SMTP id d4-20020a5b0604000000b008592accdeb6mr561571ybq.79.1676524725671; Wed, 15 Feb 2023 21:18:45 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:36 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-22-surenb@google.com> Subject: [PATCH v3 21/35] mm/mmap: write-lock adjacent VMAs if they can grow into unmapped area From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757963748559215981?= X-GMAIL-MSGID: =?utf-8?q?1757963748559215981?= While unmapping VMAs, adjacent VMAs might be able to grow into the area being unmapped. In such cases write-lock adjacent VMAs to prevent this growth. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index 118b2246bba9..00f8c5798936 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2399,11 +2399,13 @@ do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, * down_read(mmap_lock) and collide with the VMA we are about to unmap. */ if (downgrade) { - if (next && (next->vm_flags & VM_GROWSDOWN)) + if (next && (next->vm_flags & VM_GROWSDOWN)) { + vma_start_write(next); downgrade = false; - else if (prev && (prev->vm_flags & VM_GROWSUP)) + } else if (prev && (prev->vm_flags & VM_GROWSUP)) { + vma_start_write(prev); downgrade = false; - else + } else mmap_write_downgrade(mm); } From patchwork Thu Feb 16 05:17:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57859 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp117371wrn; Wed, 15 Feb 2023 21:21:20 -0800 (PST) X-Google-Smtp-Source: AK7set9i6nbBwyFgXDOU/QVzEJH+5pZW8kbOsGFvnhu0avX//8bEJmGOoBCN6kTbjU/oGphcXfl5 X-Received: by 2002:a17:906:4d49:b0:88f:87f0:c919 with SMTP id b9-20020a1709064d4900b0088f87f0c919mr5453449ejv.64.1676524880748; Wed, 15 Feb 2023 21:21:20 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676524880; cv=none; d=google.com; s=arc-20160816; b=nQrAuAd5sANNzev/XzbEKaHrFASEBH+QxnZ1OvpKb2VjymvMsTrlO7pcpcLAhYd2XZ qODiULFa0lq9A0j8PsjLJN7vdl60Cv/koKyJ/Z9gzbIIn1x9bHVKD2gRQL41imZbCOcd 2MMQ+WcEEHuBPjHJBnjklY2oMpx4fsXqeQdqv6cv2YtKvxhEq4NRyZIDRC72nSKLdghh uSBfRyQqq9JKIGjB3KCK4s5NQ+bpEov2ZPeTKQNMjklhVZdEtX41gxCpauWuHhAlo0nA F4mO+nVlO6GPuHQ2NgWrvLn1Cfz89hqptHZAu79jWBgSFHOtI1yvO7u2F7I75UCb3/NW GNCg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=Cy3b4YdYe1JheypyM1qEqlntx+LNPOiVtlvqoTpRTGk=; b=qJ99wmWBKnJtO6smofqCXb1d0h8tR0zf8ItLeLjP9NGwXDioC642tibGwM54kqOm74 MKMoSj59XRrPCn+HesG+hxrSBRUjm6LQAmYeOoja3jPKXnOatw2b1SNOMONot0KsmvM3 1BDYI6pFaQex0O+WjRc7uaEClyxGDZUIP6NgArTockTYjxDxZd6TlUeW2+inAhPg1f0l wCPZQcfuqG807Jj4kDrlf0cppd8+EouI8mZeAF2tpzznDbM+CrliL6wu0aAzOdANngVG pNkAqus+bpCpnUWOb8n4gh0goGzp8rfpC7LXnMK/yXNq64INPdqewHKhpdoXnuTUbNsH BzFQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=ee09VaaN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ex20-20020a170907955400b008b143bc84ebsi718814ejc.380.2023.02.15.21.20.57; Wed, 15 Feb 2023 21:21:20 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=ee09VaaN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229677AbjBPFU3 (ORCPT + 99 others); Thu, 16 Feb 2023 00:20:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46610 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229672AbjBPFT3 (ORCPT ); Thu, 16 Feb 2023 00:19:29 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 88CC1457D5 for ; Wed, 15 Feb 2023 21:18:49 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id r17-20020a258291000000b008ee1c76c25dso857450ybk.11 for ; Wed, 15 Feb 2023 21:18:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Cy3b4YdYe1JheypyM1qEqlntx+LNPOiVtlvqoTpRTGk=; b=ee09VaaNfMCXdTYs7ho4FL1rz+7Jau0hVrpqtPJwR0LOjVJOOKMHcMbwd/rGxak63H BRVuTQ9tdXJzSR+6NhiK4KAEy3orTDG5A6ugcB1Mu/JZ3TtJ8rFlU/MHE4DwjmulfLqA 4lbez5vLr44VvmK0HXNKetjG61FmR5dtu/ffCFjZaMQfVdd6yPZ6Vqs7cSoCFmCWN2ej U7UJuLArjVJJvFA58VBbTFzUGAlbxAZVBV5EE2zVq6EhbmHffJxz6yeGBWtLnffH9zfX I0SKmeE3zn9vqZPVYyRXu1tk/VZ/LisV9OyoAUd6iOjxLewnK1RWOS7bKO/6E67varMh DYqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Cy3b4YdYe1JheypyM1qEqlntx+LNPOiVtlvqoTpRTGk=; b=S9oRNMN53hXvJ2u1o7n8WUTsVPHgF6cCHA3DtHB81BbxgFZtLfiESNINvU+MrQlQTl 2OQI5aBIlbXTepj2OvGFJy/NxbAg/D2XM9ATKRNyocDpu83/sPUO+aMTUXiMHYwMYhsu s7viF2fnSNjy2HCFJ7RkGPHKAp8Ss1MuTFjQK+KL9rw1EBJpjtFnw/64M0BfxmMu9RRq xzzVmaVRZG63cXb7Kvfjc8EqozIab4Hqa7EdAxGbaAG9/oyXLbYGAnd82croe6EQ396q f3rScoWmoAKeTcCZb0cVCntKD8dMZ2lAwHZXSUOx1q8G0mCya+Dc4SxPLUpeXChTQUAw CafA== X-Gm-Message-State: AO0yUKUz8GvR4fi84hr3SHgZZVjuQxXZI6fJuvveD+V0xnJx0/jIwh/4 Soh49U9H69yCtRj4ZMAt220Ni4XVSXc= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a0d:c347:0:b0:52f:1923:4f9d with SMTP id f68-20020a0dc347000000b0052f19234f9dmr562132ywd.366.1676524727987; Wed, 15 Feb 2023 21:18:47 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:37 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-23-surenb@google.com> Subject: [PATCH v3 22/35] kernel/fork: assert no VMA readers during its destruction From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757963753331622624?= X-GMAIL-MSGID: =?utf-8?q?1757963753331622624?= Assert there are no holders of VMA lock for reading when it is about to be destroyed. Signed-off-by: Suren Baghdasaryan --- kernel/fork.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/fork.c b/kernel/fork.c index 9141427a98b2..a08cc0e2bfde 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -491,6 +491,9 @@ static void vm_area_free_rcu_cb(struct rcu_head *head) { struct vm_area_struct *vma = container_of(head, struct vm_area_struct, vm_rcu); + + /* The vma should not be locked while being destroyed. */ + VM_BUG_ON_VMA(rwsem_is_locked(&vma->lock), vma); __vm_area_free(vma); } #endif From patchwork Thu Feb 16 05:17:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57860 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp117391wrn; Wed, 15 Feb 2023 21:21:26 -0800 (PST) X-Google-Smtp-Source: AK7set8Gkf7epDekS1NoYbxrgSiuP5iP0dzRDISvDjpD6s/MstMaTvotETqFIyI8A3eCzmLhRqUW X-Received: by 2002:a17:907:7e9d:b0:8a9:e031:c4b7 with SMTP id qb29-20020a1709077e9d00b008a9e031c4b7mr6445685ejc.4.1676524886311; Wed, 15 Feb 2023 21:21:26 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676524886; cv=none; d=google.com; s=arc-20160816; b=G+R4nA91O7K/qTDfV7NiM2rwKv4JvOJ9LDcXuGMGcEXpld35cu+y/nOTJd+yx3YJh4 gx7pjtd2XZTx7jOgtHfLa7tgMeh8V21Ms0t0asQgdbaPaw5jSLIB/wkLPKK1PksUQsA8 cJ6+65FCf9f7MGWREkTwrOyb+2PUq+B4x3hlzAYNsMnk2KP9ZJSzLMv4OP3zIW0HhPgD yC2S+HtUTqgw2lbA+yOvFBCyz3RHIf8dHPZaWp/9HSGr1bLtrztPOsJs3W2vy+02dfed uQsNakzQVthrwlaenITwlcxR7hiIEhlKUJ6MJvL/oSMLC4MZsYhplqLjFKztFuiL2Cvw 0GNw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=GTmH+3zLa4DS1pm6XKzoFa9XiN/n/492GQmNm0F2u98=; b=prHdGln7ZQqsK50HJw7fAhgqT2ArHoA2FZdeIUGUt2ZT4O8/8eZptpRBpEPX3k7/Bc OCyV4r6akEkI1pWYwGc7e4Xce8d2DxXxAOoXHej5wGHYfBImZteDi3g0PjucwTDEWulU DvptXD88i9JsNBfXw62zMI71txZ9RV1QOx8gAxu8c3Qr8n0CqXMmq3vUZGwnLt889YNa I42BF1gS28xskjj1epTlKoUw7GDflBPgHjcBDN2Fe5+GtU65TdYuX03jGFXvErnrZxK2 Wlg/OwhiVePf4XKviW+9Xrs+KMh1f9SWYmc+frrrEvZI4VlmJqDsCMxVe7fNnrGqMWWL DlwQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=lxQtFymG; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id if19-20020a170906df5300b008b12da2695bsi843352ejc.993.2023.02.15.21.21.03; Wed, 15 Feb 2023 21:21:26 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=lxQtFymG; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229764AbjBPFUb (ORCPT + 99 others); Thu, 16 Feb 2023 00:20:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46692 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229635AbjBPFTa (ORCPT ); Thu, 16 Feb 2023 00:19:30 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E422645F60 for ; Wed, 15 Feb 2023 21:18:50 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id n20-20020a25da14000000b008fa1d22bd55so840462ybf.21 for ; Wed, 15 Feb 2023 21:18:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GTmH+3zLa4DS1pm6XKzoFa9XiN/n/492GQmNm0F2u98=; b=lxQtFymGNoCMn73Q032qmSTTcj+ve7GubiXjG/S2/wZlI5A+wvms4mpLfeb6oKn81f gVVSU10rEKBXvaTQ4DI3+isYBuVvKtNDNUM4jT8VbxX9prWBulP37D7mcvfTTB0yv5Vd xLNBfAphNgvAxOXoC4bxmXgktQA4hGiUCCaPhABiIUzurEkqvA5yibf0xhi6vP6KsH63 2fMMNvtfnEstPxLz+UPzwP+9hfF+/M/equZBkR4oGxxGpJkcmKAi9IzofVMskpdF8tY7 tvoaCpVKwaaM8HBNFnnIuJry7ZsbTgV8SMkds4HojfN0Cbg+ccWTepT6dYOQ/gKplcjn qubQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GTmH+3zLa4DS1pm6XKzoFa9XiN/n/492GQmNm0F2u98=; b=dWwGXiD833Vh5DcAbp2IOBMXewpJtdezcUeVBwztsoS5bGpNgW8LQSFGxjRlHkclcS ft1uBp84hnEh+mI3muVkuhDeCE4x3idmD9e5wqEl8Apf+W5ZkgcrZ5/Q/COwsWj8VDMI a8yiC3tK8gbuFKQhiKJm9JlJUEv3KZERVAqb93L5bMpnmgAc7tgsqzpRxSS/HNpQaqKB 7P289jaSAZJEUn4bjH7UgU3sI8wYEeYOwvclh1TGiLsbaKNJJAhIpNpIcz+7684GGTSV qIk2KQLff3gNiHP/f3Sv1g7pZ4hSujmlZqoln9gqHBYx6S3jjrOhYe8CjmVkhB5Ys4i5 N0sw== X-Gm-Message-State: AO0yUKXMg/yloDfeGiMUlWXpn2eGKs5NdCujJ+4SELHMABO3sSyU22bZ awTvAqi4892mlH8EelqlV/IO7kglQcc= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a81:441c:0:b0:52e:c79a:cda with SMTP id r28-20020a81441c000000b0052ec79a0cdamr12ywa.10.1676524730090; Wed, 15 Feb 2023 21:18:50 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:38 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-24-surenb@google.com> Subject: [PATCH v3 23/35] mm/mmap: prevent pagefault handler from racing with mmu_notifier registration From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757963759382303504?= X-GMAIL-MSGID: =?utf-8?q?1757963759382303504?= Page fault handlers might need to fire MMU notifications while a new notifier is being registered. Modify mm_take_all_locks to write-lock all VMAs and prevent this race with page fault handlers that would hold VMA locks. VMAs are locked before i_mmap_rwsem and anon_vma to keep the same locking order as in page fault handlers. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 00f8c5798936..801608726be8 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3501,6 +3501,7 @@ static void vm_lock_mapping(struct mm_struct *mm, struct address_space *mapping) * of mm/rmap.c: * - all hugetlbfs_i_mmap_rwsem_key locks (aka mapping->i_mmap_rwsem for * hugetlb mapping); + * - all vmas marked locked * - all i_mmap_rwsem locks; * - all anon_vma->rwseml * @@ -3523,6 +3524,13 @@ int mm_take_all_locks(struct mm_struct *mm) mutex_lock(&mm_all_locks_mutex); + mas_for_each(&mas, vma, ULONG_MAX) { + if (signal_pending(current)) + goto out_unlock; + vma_start_write(vma); + } + + mas_set(&mas, 0); mas_for_each(&mas, vma, ULONG_MAX) { if (signal_pending(current)) goto out_unlock; @@ -3612,6 +3620,7 @@ void mm_drop_all_locks(struct mm_struct *mm) if (vma->vm_file && vma->vm_file->f_mapping) vm_unlock_mapping(vma->vm_file->f_mapping); } + vma_end_write_all(mm); mutex_unlock(&mm_all_locks_mutex); } From patchwork Thu Feb 16 05:17:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57861 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp117421wrn; Wed, 15 Feb 2023 21:21:31 -0800 (PST) X-Google-Smtp-Source: AK7set/2p2YiTzwiDToSMH6oyHNk+md4ZLy6VdYLhOvBU0vljUtqhX0fkEr6vJfnae63jN/99lZ7 X-Received: by 2002:a17:906:8482:b0:88a:d760:19ac with SMTP id m2-20020a170906848200b0088ad76019acmr5596333ejx.1.1676524891634; Wed, 15 Feb 2023 21:21:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676524891; cv=none; d=google.com; s=arc-20160816; b=qaTnr5xVDP827D9CVvfwd4Yw1BsOo3m073BYMeN95B2brtqrB0o4m7d4Hzkm/lZlFf qrtB6X676ZlnHDFNwl8q0ia2vh8mtm6hEnKPkDINlE/zYiBYsUF581z1yLM8s0KQl1Yc b2qbbVtEoT2Xy729bkrXXRsQG9s6qkK9jZg34Y4uMMK3G7Hxy56g/Xs8hl2X/helS1oY gev+L4NvQ0h8wsLmMPCFNvpEknbujAkCfdoJaYZm3yW2osLZZs9TDI3xv0dn2iwBqfAH VGEM9mGehwPp3++2Ryso7zUhEcOhNF4slxVbPETKNEBhrOvr7CewWi+PXhdOpRn3qzDo Vs2g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=u26HcyQZUssNB/wvgmnc8FNLt5Bjt6C27EcCpLSHQ2I=; b=oT18oufHbG6hdSyMSNwA3idLGK6wFwg5g3x2MGlr29U62U39pK54SbvilHqmb0F0mH 7UKpf+a9pjRq7aLuLQqF40mJj+F3UFhHy/POp+aZ4tKeM6vfqb/xEJn5Wjrj5t3QHFsp 6HrX7ZYnu5mFo9aR5tC1DCMarEwyp+7H4dvgZjTM2CEhGov4iJQrICyvBe4dEI46iewn E/GEInzv/by+/JqeuV+tzL853Io3xf37q63r2BmaiaircobGt/RIMCjDCPLRPPwTkpLB IGojVA+26rHOiE8gbkwH2Fil7YXmKbW/zZr09YSjfLp3AgjGLwxyiIS2FbWKBEGkfssn 9tbw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=e8FcGNDH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id gh14-20020a1709073c0e00b008b12ebf0adfsi740464ejc.667.2023.02.15.21.21.08; Wed, 15 Feb 2023 21:21:31 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=e8FcGNDH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229840AbjBPFUd (ORCPT + 99 others); Thu, 16 Feb 2023 00:20:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46194 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229767AbjBPFTj (ORCPT ); Thu, 16 Feb 2023 00:19:39 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BB4FA47423 for ; Wed, 15 Feb 2023 21:18:54 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id v63-20020a254842000000b008f257b16d71so842164yba.15 for ; Wed, 15 Feb 2023 21:18:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=u26HcyQZUssNB/wvgmnc8FNLt5Bjt6C27EcCpLSHQ2I=; b=e8FcGNDH5VkVzBTYRblbtAQ0+ZP2ugsPFuxmDPyoQXwVKDnKFUyAPJn/SJMrsaQ1Ia jo+Cd4OdGYV2adIa7Ps1WdQQvf3M0XkBbQCS2vH4hvx3mR4XIU6AeYNJ4fgFw+1soX5Z Pa1xM0MUFfbg6JS2S95T1SCUKSw+S8anZTC0et5KKeuEZZM6kRmGtxME3D+du+nzcvgG qzTaGZDic4CCJCQNQmsvjPxTw8HSPKOmRJrMlp7St/TdGbLlhdx8Vn9wPFTpNdXttF8Q VA64i9yGakw2MEWw1P/yGjLggzJzYv4qMjifZYLSh4iksjXsYnlNyRAbe89au2hayYpB yYBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=u26HcyQZUssNB/wvgmnc8FNLt5Bjt6C27EcCpLSHQ2I=; b=sNtkjqp3WiwTltWeeHGLdA0Gx0tGlfGR9f59XRhjXQRZf9+1W3vOYF1joJest+20u3 dUlA4P6MSjsw+Dt78EkkqzKGRxxrcNCS5qGXNPM8liu3S+CcrFvc9nM+LpDr9ZfsYwgW pWPkEzPPIHhi7jfo0dhBeTVXgzjPJ5V9g8axCktdhSDHQ9ZSlzTI5C43jllUNcHeVOt8 ls9DNm9Xx42TY+G8sIn8Rha5hzT9cYFRlqpkGNUj2RmDHpmosq9J06lEvUpJdoNUwMdb ka3+VxZN2fUYEMZQ0Fy0FqMpjSz2wpapGkDYJ9pno9HYwF5vNCvBfmNVqzvJwJhDqWzT 1Xow== X-Gm-Message-State: AO0yUKXSrwara7XUqoSGxke4MnoGe97GRepEpNMBc/h3QnheVl9MoNB3 BejSWlE/oVb771b48BhQeTmFqIp7TzU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a05:6902:114f:b0:855:fdcb:4467 with SMTP id p15-20020a056902114f00b00855fdcb4467mr31ybu.0.1676524732481; Wed, 15 Feb 2023 21:18:52 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:39 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-25-surenb@google.com> Subject: [PATCH v3 24/35] mm: introduce vma detached flag From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757963764658458225?= X-GMAIL-MSGID: =?utf-8?q?1757963764658458225?= Per-vma locking mechanism will search for VMA under RCU protection and then after locking it, has to ensure it was not removed from the VMA tree after we found it. To make this check efficient, introduce a vma->detached flag to mark VMAs which were removed from the VMA tree. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 11 +++++++++++ include/linux/mm_types.h | 3 +++ mm/mmap.c | 2 ++ 3 files changed, 16 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index f4f702224ec5..3f98344f829c 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -693,6 +693,14 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma) VM_BUG_ON_VMA(vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); } +static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) +{ + /* When detaching vma should be write-locked */ + if (detached) + vma_assert_write_locked(vma); + vma->detached = detached; +} + #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_init_lock(struct vm_area_struct *vma) {} @@ -701,6 +709,8 @@ static inline bool vma_start_read(struct vm_area_struct *vma) static inline void vma_end_read(struct vm_area_struct *vma) {} static inline void vma_start_write(struct vm_area_struct *vma) {} static inline void vma_assert_write_locked(struct vm_area_struct *vma) {} +static inline void vma_mark_detached(struct vm_area_struct *vma, + bool detached) {} #endif /* CONFIG_PER_VMA_LOCK */ @@ -712,6 +722,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); + vma_mark_detached(vma, false); vma_init_lock(vma); } diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index e268723eaf44..939f4f5a1115 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -511,6 +511,9 @@ struct vm_area_struct { #ifdef CONFIG_PER_VMA_LOCK int vm_lock_seq; struct rw_semaphore lock; + + /* Flag to indicate areas detached from the mm->mm_mt tree */ + bool detached; #endif /* diff --git a/mm/mmap.c b/mm/mmap.c index 801608726be8..adf40177e68f 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -593,6 +593,7 @@ static inline void vma_complete(struct vma_prepare *vp, if (vp->remove) { again: + vma_mark_detached(vp->remove, true); if (vp->file) { uprobe_munmap(vp->remove, vp->remove->vm_start, vp->remove->vm_end); @@ -2267,6 +2268,7 @@ static inline int munmap_sidetree(struct vm_area_struct *vma, if (mas_store_gfp(mas_detach, vma, GFP_KERNEL)) return -ENOMEM; + vma_mark_detached(vma, true); if (vma->vm_flags & VM_LOCKED) vma->vm_mm->locked_vm -= vma_pages(vma); From patchwork Thu Feb 16 05:17:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57862 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp117478wrn; Wed, 15 Feb 2023 21:21:40 -0800 (PST) X-Google-Smtp-Source: AK7set/aF7h4aLhUTBnH11wv/Ni+N3VP62YR9gBaBK1mFSJfPzzZ45MT0NbBuXbtWBHKYD3sSZKe X-Received: by 2002:a05:6402:1e91:b0:4aa:cf80:44af with SMTP id f17-20020a0564021e9100b004aacf8044afmr1138453edf.7.1676524900608; Wed, 15 Feb 2023 21:21:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676524900; cv=none; d=google.com; s=arc-20160816; b=0pHKifRXMLqRSkfl5ubgkyAx6adnCIbqa95HRX9Q5N3y/xukQU6D2H1FUWKSxlMQkB InS7DQBOnnu+bFl2//KDSZVnpascI5np4bAlEXTwb1Qsl0DGSNZMRwr2Nmh948+9HeMl b9l5smWMBI/Cyv0+3ZvkdCEiQRkj53oclqMqFtAf/J9q/YPtaImfy/5ZM5b0LMOz+km+ AzzDsYt/CJ2h45GFtRulz7S48Ahjf8r1RhxNITrfigf/nP9D5PXL0a/J49Q5Bwi1n8xx o2kiZkDU1SDc7RDhlAR6MaBDDzepdetSa9XFlZmH+fsgvUVIEiGhWzN8yOt/HCN/94MD 6MBQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=rll6a/VW4pEw1AhNBjbFoMRbi+XwYRGQoRwSBUD0bh4=; b=bt7lwm51rlr2APkoKpg62JriE1Ca5vl4cD6VwVByUoEcQOPq5/JtQq3NHu+iOJ7/jX Wcl46mMjTQs/exUgBcD/MKAP+8Il97ptqAWlh0WkaB3qQc954SgtpPP7ZJnhd4LvUYg2 cWg3HcyvX3leGhaRZiQ4xzwNlauuGZs3q58IRXVV44uW+eXFh6tX7e726ufloG8Noing kk+4xgTirNilh9mNhEfxb4tqxUhDkULcHcD1hpOoj7H4CW5M0HBcgoyq8EkeAqFrwY5Z XDNWz64FIuKd9BhtPyp9lIO8iCXTVKDv2sKD22h7I892azXRyV2iYR/lz/Wiw71meIax 0VqA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=hu+sMS9o; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q23-20020a056402033700b004acbca07ab5si1097824edw.615.2023.02.15.21.21.17; Wed, 15 Feb 2023 21:21:40 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=hu+sMS9o; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229847AbjBPFUi (ORCPT + 99 others); Thu, 16 Feb 2023 00:20:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46204 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229771AbjBPFTk (ORCPT ); Thu, 16 Feb 2023 00:19:40 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6742B410B9 for ; Wed, 15 Feb 2023 21:18:56 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id 191-20020a2504c8000000b008e2fd9e34e6so855049ybe.9 for ; Wed, 15 Feb 2023 21:18:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=rll6a/VW4pEw1AhNBjbFoMRbi+XwYRGQoRwSBUD0bh4=; b=hu+sMS9oSFHR1r7khxhf1UbXFcs2Z++5UsWdyoPEf98HLaKpCA/5NqLhTjKtFbNcAI bq4IC94z3LeKr+N8fZ27HaBeoB9eV7FJUK2PbLxLqtWEgkZI6C3sySZgRKxnS+2i/7yz RFGxaAOIl8opKirVI8bfAZAj2T7sVtyi7tZlmgl6KCk8dFxbKL8va87izUWyHNnCWm+i wFd5WvNjbROGmsVfKcufeNfiLR4xBKkikebAtyhw//UrLzCCkecCcI/c1WZAFUB1nZ5g ONtU8uW/bb5j1wyQf5MWCYTwREIrfwMyez4h+y24cgMUk3OrfInLJF7DZne80Aw4Hh2T oQOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=rll6a/VW4pEw1AhNBjbFoMRbi+XwYRGQoRwSBUD0bh4=; b=0gKH/Uhd8pfqI3ivqX5DKP0pz/tLKTJDON9T74FLHKTuLdlQ6R59qDSzCG8A4F5BxR RdE3J6vSfZRG2JM9LFNcR3iGACpLNFGMZg6akGpF8yv77NZZEbpDimkKii8BC/7T87eg wFpCbx9IcBtxMPYumKvf4HIhzAkf6Y279xBU98IVfs68JqKespmtTT4SeSekpAdAaUKy LNZD5b4qCSD/7lrJHRuw0nlUSy1GHii33zUA0YQeYVqHD5VzOA4h2v/pSq+F/c/pwhLI uzQV3Z/c+CO0CBiGludJhO1J2u3ICvsEuO/Bir8ZZxC8wRlzaOph84VeDHpBjkigmdJZ iVAA== X-Gm-Message-State: AO0yUKXMOfqZfLcDlbaDfFJJuvb/GBj0IdZXeS3M1fMMslwjyDgtlSMW J1/nZzGMsv2rMhYOwhioG21GyTrQMM8= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a25:9012:0:b0:874:380b:887e with SMTP id s18-20020a259012000000b00874380b887emr545965ybl.239.1676524735071; Wed, 15 Feb 2023 21:18:55 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:40 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-26-surenb@google.com> Subject: [PATCH v3 25/35] mm: introduce lock_vma_under_rcu to be used from arch-specific code From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757963774063931333?= X-GMAIL-MSGID: =?utf-8?q?1757963774063931333?= Introduce lock_vma_under_rcu function to lookup and lock a VMA during page fault handling. When VMA is not found, can't be locked or changes after being locked, the function returns NULL. The lookup is performed under RCU protection to prevent the found VMA from being destroyed before the VMA lock is acquired. VMA lock statistics are updated according to the results. For now only anonymous VMAs can be searched this way. In other cases the function returns NULL. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 3 +++ mm/memory.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 49 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 3f98344f829c..36172bb38e6b 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -701,6 +701,9 @@ static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) vma->detached = detached; } +struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, + unsigned long address); + #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_init_lock(struct vm_area_struct *vma) {} diff --git a/mm/memory.c b/mm/memory.c index 8177c59ffd2d..5e1c124552a1 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5220,6 +5220,52 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, } EXPORT_SYMBOL_GPL(handle_mm_fault); +#ifdef CONFIG_PER_VMA_LOCK +/* + * Lookup and lock a VMA under RCU protection. Returned VMA is guaranteed to be + * stable and not isolated. If the VMA is not found or is being modified the + * function returns NULL. + */ +struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, + unsigned long address) +{ + MA_STATE(mas, &mm->mm_mt, address, address); + struct vm_area_struct *vma; + + rcu_read_lock(); +retry: + vma = mas_walk(&mas); + if (!vma) + goto inval; + + /* Only anonymous vmas are supported for now */ + if (!vma_is_anonymous(vma)) + goto inval; + + if (!vma_start_read(vma)) + goto inval; + + /* Check since vm_start/vm_end might change before we lock the VMA */ + if (unlikely(address < vma->vm_start || address >= vma->vm_end)) { + vma_end_read(vma); + goto inval; + } + + /* Check if the VMA got isolated after we found it */ + if (vma->detached) { + vma_end_read(vma); + /* The area was replaced with another one */ + goto retry; + } + + rcu_read_unlock(); + return vma; +inval: + rcu_read_unlock(); + return NULL; +} +#endif /* CONFIG_PER_VMA_LOCK */ + #ifndef __PAGETABLE_P4D_FOLDED /* * Allocate p4d page table. From patchwork Thu Feb 16 05:17:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57863 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp117491wrn; Wed, 15 Feb 2023 21:21:42 -0800 (PST) X-Google-Smtp-Source: AK7set8MessfJw9VgB2kCZl4nTUiV1nhKlwgcLgHD40NCMvH9jxwYQw3GfalzGM75gh8phm7KOnt X-Received: by 2002:a17:906:f2d8:b0:8b1:2824:90d7 with SMTP id gz24-20020a170906f2d800b008b1282490d7mr4386925ejb.16.1676524902621; Wed, 15 Feb 2023 21:21:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676524902; cv=none; d=google.com; s=arc-20160816; b=OMQu9vVhZW2mN4aHR7QQXAub9JuQvqOKmU463ePlgb4lFjUbuA2ic3RGFZqEdlELyN k+IGoJGnflVb9B5hvGJdFWfrIdS8hcskGoAnvExkZaMRDQ7LiGm8lAwX4GdkahlLfuEw 5xs2oFEajJeXBdN/5IKoboKFnEO1iQlN2FDpnFgQTWAeaxd6yXk+HG/MKq3bORTlr2bE BH8oAk0tMgDM5MLYI6/kcWi9UcpzjMeTIT0nSEelPkqNwKJ00zWESQA8lNhdOxaYleqQ tTQJXOGQ8G46tT6WZeQswIP9HhqCdutaQN2StgBX6nf5muhx6tzNqkmgHCKnSqtoDD0L 7pcg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=76w8C0WtscQXhYR05yNYHvPFGtZWYYoigmdRBKNsAGs=; b=mDU+6HVnVb/Hwda0MZiuMsCJWI9v80CnsIyfNVPqhDp8CPnWXzkwflye7dePl1YzMe o+BJB9c4BKIvC/dp6AsroNiyqVgi/cRzMnHyzI/K5bSCzIz6pP6ke0G5RFIc701EoZpF uswQrNBcp3xb1SMWj+QsHLGSgZZWJ556W9mPmzQ5LaZ6iXjAmo5fYJ59Gw4S6fzYOGTP 41hKGKpivJq8ycbB67VrSkAGzF9Cjl4Z8uf5Bd3J8Bx5YWDG757MLLZKnv/Uz+rSZZEI JesRgEiUbCgBTAcjVxg+uoL+jpsqfQ7DkI/ROmi0T6rQypBao6r9pCHeFlrLEx5yEhBs dFcw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=FCDmrKEx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 18-20020a170906225200b0088d5dc62a29si1053189ejr.200.2023.02.15.21.21.19; Wed, 15 Feb 2023 21:21:42 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=FCDmrKEx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229781AbjBPFUu (ORCPT + 99 others); Thu, 16 Feb 2023 00:20:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46878 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229731AbjBPFTu (ORCPT ); Thu, 16 Feb 2023 00:19:50 -0500 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C3DE7457D4 for ; Wed, 15 Feb 2023 21:18:57 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-4cddba76f55so8311157b3.23 for ; Wed, 15 Feb 2023 21:18:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=76w8C0WtscQXhYR05yNYHvPFGtZWYYoigmdRBKNsAGs=; b=FCDmrKExzLuCArz312mMsT1GG2XBAJpY0IRk4Dudkq8tacxOtpStMHAB4mCqF7cHzr HxmYKbJzh0e/OlcI5tze/pJirqiUzOseaOUlDEtDH80qm7FP3CxiVmnvUlVVgBXp61ej ykGRdx137mYbxOiW4ntwVshjUhNYXqvxbpsd7tR4+U5K8xmf3z6sAC5BmIr5igjBFbE2 Lhh7YXKnyKDhial8w7C+wSOyLz2qDj3wAEjKVPIMU3N4BzLNr3q2F7Jzk8qKKMPeJp0i mhDW6Px8X9de0wrc9+taZBLP7xnnbG62Qe2RXE2u3O0X+3PidgLM2eknvp7ECPYQAyqI 26OQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=76w8C0WtscQXhYR05yNYHvPFGtZWYYoigmdRBKNsAGs=; b=Cl/2dg7px0VteYTm0pbQw/GnSDeS3R4D6VtyflogegSEOZoWEluwyTU/AlNNcDyfU1 qzLngBSYK2fAbFY7MXdonFLKX2iihrOywsQkp196b31AmMQs66bVyZjcqwYGiEKdV2As kI6qNDJKI8hp+4BFj3SHyy9J/bi+BrIIZ9YBAS9ro/c2Kn0CdOlRUj7QBrC+zMpwoepW 76GuwSpmusAEGjcieAX6cJENr51VyktlWN2Kk0V/VCrAJNEYfm/oIBNF4XKDI419JXZD W22h+sTVVExRJ1n9d8uR/y4l9KsxqbH8pwNyrijbcFQizA2cJnhWwpRO3sBcQRoTx6ef 9ENQ== X-Gm-Message-State: AO0yUKVaIfCPc3sKSXOFm2HkN/387vQOxApK9NEv/Wj7DBhMXSgdKfwr xckfzea6zM4CUEz9ld2zPHhxxJQYL8s= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a81:9e0a:0:b0:532:a8a0:8d76 with SMTP id m10-20020a819e0a000000b00532a8a08d76mr235300ywj.85.1676524737340; Wed, 15 Feb 2023 21:18:57 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:41 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-27-surenb@google.com> Subject: [PATCH v3 26/35] mm: fall back to mmap_lock if vma->anon_vma is not yet set From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757963776401405256?= X-GMAIL-MSGID: =?utf-8?q?1757963776401405256?= When vma->anon_vma is not set, page fault handler will set it by either reusing anon_vma of an adjacent VMA if VMAs are compatible or by allocating a new one. find_mergeable_anon_vma() walks VMA tree to find a compatible adjacent VMA and that requires not only the faulting VMA to be stable but also the tree structure and other VMAs inside that tree. Therefore locking just the faulting VMA is not enough for this search. Fall back to taking mmap_lock when vma->anon_vma is not set. This situation happens only on the first page fault and should not affect overall performance. Signed-off-by: Suren Baghdasaryan --- mm/memory.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 5e1c124552a1..13369ff15ec1 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5242,6 +5242,10 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, if (!vma_is_anonymous(vma)) goto inval; + /* find_mergeable_anon_vma uses adjacent vmas which are not locked */ + if (!vma->anon_vma) + goto inval; + if (!vma_start_read(vma)) goto inval; From patchwork Thu Feb 16 05:17:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57864 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp117523wrn; Wed, 15 Feb 2023 21:21:53 -0800 (PST) X-Google-Smtp-Source: AK7set8NaP3JmbTR1IP1X2sp1ir4TaA113WS5l/UtTQBuWlo/0OfLfplXVndAfFfH81vwcJt0cgc X-Received: by 2002:a17:906:6d56:b0:87b:d3f3:dcf3 with SMTP id a22-20020a1709066d5600b0087bd3f3dcf3mr5179697ejt.35.1676524913101; Wed, 15 Feb 2023 21:21:53 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676524913; cv=none; d=google.com; s=arc-20160816; b=JcwbgmQt1qfMxijk/CtfVzHyVjehA9gl+3L8bwCi+NJ25YMI2ilLWoLzCULWgfACsj /EmKreJW7pHvlVoZGwhncVT8bjeHfwEXPqJLI7z4ts2x8vFAvQa5VN40RxOyvUuIxxXK To12c6xpnUgVMvpvwMdVZIR08QfJkTusEsX5EEkyEiT4aAOeifPgfENjOg0sDLZNCEGl zBtlTXp4i+DH0DHXwH5iZ3nJY2SfoVrgyayJtrpvWSQIW+VvGyejobGrvpxKshr677Tk FRW6VnhIAm+E3sNpjqlYjjmJ8QScmqvQU2Ss38iBNHAb7e2v9a/GLmLvelPQsbUKKJNR wdVA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=g+2R9wHJik/ujG7pbEIR7UB7BFuMmi7T4GQjwfs+Slo=; b=ZEx3LoX4HW1DcTjhPa8py7UHcxalHxOxq6+lKJB0tE0VbjaianyL9WUets9Hao/hDi eKwuGBx29kInJEwMiZBP/u+BUOeM70kWMybON+MmndlkYpFG1zQtI8viQYmyDfZZCyBM da3icFdRvVGfH8LrsqWvSGySaJmOHECdKuSrZeYxwpcz80LWzuxeYiPc+9zPUcP15ZIW qT+WsPh0yVKIqEmB8BX937O0fVwowQnJ1OBbQDt4DypbnFWcB7km6XsTwSvWS5pqjPBz tnVVFg0hUZ5YYIC1cWiB2i+Vkeoqgf8qn/MEkYhmFOIJx8Y9jv3q9NiN/IPD7PSutD9c A7xg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=GJ5BxW0Q; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 30-20020a170906011e00b008b1109e0265si1104658eje.112.2023.02.15.21.21.29; Wed, 15 Feb 2023 21:21:53 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=GJ5BxW0Q; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229696AbjBPFUw (ORCPT + 99 others); Thu, 16 Feb 2023 00:20:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46044 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229734AbjBPFT5 (ORCPT ); Thu, 16 Feb 2023 00:19:57 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2FFAA474D1 for ; Wed, 15 Feb 2023 21:19:01 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id q127-20020a25d985000000b009362f0368aeso883604ybg.3 for ; Wed, 15 Feb 2023 21:19:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=g+2R9wHJik/ujG7pbEIR7UB7BFuMmi7T4GQjwfs+Slo=; b=GJ5BxW0Q1u88snEvipBftLmc3+Jf1QItAPirJyrJgB8Mp9ylRIIALZlXmJQ4wNiSwP o865zgbeTpwGE0/miDmLSBWepWhJN06egqwo+YGBO2p834t1tPcrBoNtCdfrXD1jQzaN e7lrVFe77phFTZAotKwLYsW/LOb59LyIPFjYvDx4mPrMSRvD2iWSSQr/TeKwF3SManJP 7zOmcrgCnGJ5t0wPXfGvFFU5Wepw7LHd2B3wtwv+KMRgdeCQNTeRgpwm3UNl2CvWi/nE 3FOidEUJKnpNbe60Or5Um47yum4t5Cm2ARYxCb7pet6vv6oqpLv+fwdHpYVdfug+gwv9 oixA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=g+2R9wHJik/ujG7pbEIR7UB7BFuMmi7T4GQjwfs+Slo=; b=Qi3tRaGOFoNfAfZOXYc5BFe0C7UUqBaV5lquw1ZJ5toVViOcgpfIVIkTr+7eGAzQKM MiP476TRD9/Gi/YCUSfYwC5297Yn0h5OJDxPp9Uu6Sa2gAyo595Fkw6l4ZY2v9WpYUtw BzfrK1e+xzxgFiCKBufoFG82ZcyF2zKF9SS4h6ocgzSmE1I9TdB7AzqJGyKuwTus04/7 sh00bqtKsu/xqYKnQhvg2M7+LxogCzjf7wPFcyblIK0UGh1rDKwZC0ApTfxehir00NJn 5hefN49CeTJLtsRz0zpSedzMfHjoXkFxJMlJW9NAnAaieKokpa0HtDRzd0zjTgq2M3AS +rXg== X-Gm-Message-State: AO0yUKUfxcUdXcQOCiH92eidQJURQsAkgLjGfjpIVT/UmZ63e+T8j9J8 xkTTs7ZOjRQSnz5SvGXFY5aAjXV5YG8= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a5b:144:0:b0:8ec:cc9b:7333 with SMTP id c4-20020a5b0144000000b008eccc9b7333mr463396ybp.205.1676524739514; Wed, 15 Feb 2023 21:18:59 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:42 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-28-surenb@google.com> Subject: [PATCH v3 27/35] mm: add FAULT_FLAG_VMA_LOCK flag From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan , Laurent Dufour X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757963786990959853?= X-GMAIL-MSGID: =?utf-8?q?1757963786990959853?= Add a new flag to distinguish page faults handled under protection of per-vma lock. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- include/linux/mm.h | 3 ++- include/linux/mm_types.h | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 36172bb38e6b..3c9167529417 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -478,7 +478,8 @@ static inline bool fault_flag_allow_retry_first(enum fault_flag flags) { FAULT_FLAG_USER, "USER" }, \ { FAULT_FLAG_REMOTE, "REMOTE" }, \ { FAULT_FLAG_INSTRUCTION, "INSTRUCTION" }, \ - { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" } + { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" }, \ + { FAULT_FLAG_VMA_LOCK, "VMA_LOCK" } /* * vm_fault is filled by the pagefault handler and passed to the vma's diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 939f4f5a1115..212e7f923a69 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1056,6 +1056,7 @@ enum fault_flag { FAULT_FLAG_INTERRUPTIBLE = 1 << 9, FAULT_FLAG_UNSHARE = 1 << 10, FAULT_FLAG_ORIG_PTE_VALID = 1 << 11, + FAULT_FLAG_VMA_LOCK = 1 << 12, }; typedef unsigned int __bitwise zap_flags_t; From patchwork Thu Feb 16 05:17:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57865 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp117541wrn; Wed, 15 Feb 2023 21:21:56 -0800 (PST) X-Google-Smtp-Source: AK7set/2Mvi5U3ImTZfEStqCRM1LNgUHKxbyOvcHZMrKWAUqMhxQIzBs4sV5NPWPEkouTj3fL071 X-Received: by 2002:aa7:d746:0:b0:4ad:316:b4e6 with SMTP id a6-20020aa7d746000000b004ad0316b4e6mr4432107eds.42.1676524916356; Wed, 15 Feb 2023 21:21:56 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676524916; cv=none; d=google.com; s=arc-20160816; b=nSUArGfjOs0EWoz+wUw3MCPuSlz9vFjaAtHlQPGGXB0RYc6cp84GqWdU8eK0hR23+j Lz/QZTPNHy88xlEzXj2VG1z7k7Uj01iKmvv9RvAgHgj7/QN2zgZ/M694EDMgf8rc0++7 jqLYCeXdaW13NVBbiZcVobnDgtAT62JmQ7kwSd5+szcnRJPuG5PGtHE2tIxFvHgdGxuw RKeZKYDmEeDxPq89a7fsW2kGiWUW8ZiZlnDVOCdKtvC5ctVb3GPqQFB+Fr6xF9rve94g rDZ2p+bXubKJMpQCLY4JUHqnZKMNLEEc7LlE+Lov0iCB/OcVQ/K2Kp6VafutODoj8ve/ yFRg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=+haucx/0aUjNKP4B5JYWnKqFhX48IZcct98M2E1/jbw=; b=VaRir0EXMCUI/wuFxOmL5Utf4SMlGIXDcNSZ17UieA60dJ1/4wtqLHuow7R/QrwCjH sfFa5ZUA8ksdZEJttsVQT66WGtYar4NreSpsSg6ogvOA4VvAoUYCVI/HWiwTBkFBv8v3 j56YsR5rvYbUp9iQa8NqSIfB2H5Har7Yh5Mmy5Fip2MuuI/alQ8wMBHsfn2QupVPxN2m QH2ALir1Z1e3feppUOQjdo+wWY50QTBVPBFg+vmyvqMbzE2HSyfgDiIiXR0ONi/1PibQ fCagHgeGd6dg2dXRiByuOAclGlm16RHFB4HXR/+PPiE/hRC+GMczNV3x/DO4rukFPRhO X7Ow== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=aDErRd7k; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id x6-20020aa7d6c6000000b004a2657551e4si1023194edr.287.2023.02.15.21.21.33; Wed, 15 Feb 2023 21:21:56 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=aDErRd7k; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229700AbjBPFU5 (ORCPT + 99 others); Thu, 16 Feb 2023 00:20:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45832 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229685AbjBPFT6 (ORCPT ); Thu, 16 Feb 2023 00:19:58 -0500 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E063E47432 for ; Wed, 15 Feb 2023 21:19:02 -0800 (PST) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-4bdeb1bbeafso8706637b3.4 for ; Wed, 15 Feb 2023 21:19:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=+haucx/0aUjNKP4B5JYWnKqFhX48IZcct98M2E1/jbw=; b=aDErRd7kkY+Kp4AaU6gjM3F503YHpBTPjjocmaNoc4zXVCAuMI2RAQKjzcRAbLtJge uySjaOCoHcGS53W7o3RYdOQoeaAfeHvH08PuG1fh1bxXWOGZDVGDBlzIfGuJz8kR8m0P 4LCu9IW21ZFA5+XCq7hXP9c4LIZtK/mDOJpY3r8EhzjVEo7Ti1XlFspuAz3gf99Vo08U GQhyt4xy4/c9ROtLln2G8n+ixCeeiCqz1U0tO8VqF1k+R16fVXyhOw17lx8zHct64i1Z f89mO/K5qgtX3vlh4Aty7U18axNicxnQBp8nhqlTIla4iaOnphrb06qKzWIf+oeLlfcz N8gg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+haucx/0aUjNKP4B5JYWnKqFhX48IZcct98M2E1/jbw=; b=vXvJ/jmmr8cilTQtDwSdBP8CsN/q9SuLvltAFG8UvI2R0R5hMj+Kq6MUK6SkLZ53Rt XcyyIW9gpU+L5xxQsTfhd77ZxzcOJ4JCdSiHVivWPLbF4A/R8zx9QlWWQlXf/d5VoF8a FgvXXwo66wc2uvZMqNgZO+Fxpqa1D6WmNPYbSqW9EoxmUWr/Jx+F050nEj4kIbB5eTMs LvxnN7MXQO1maObCExB10pge0LyvHy5kMnEqiJThAvyYqngbH83isJTwbCRMEnx4fk86 ya2dLL8+euGIVAtNNEy49vl+fp3ZPK2NEOkrp6XiegjNLcvJAnw3kAFjeNGwH4+e4lrw 4URA== X-Gm-Message-State: AO0yUKVGFbz20h6ipSOryVd1Dya+CyG7oDRmF5XfadjTFb1sSyrg8vvY n5ibRy+z0ce2YWQNEBbuQZIutRIB6BQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a5b:c41:0:b0:91d:ddda:665f with SMTP id d1-20020a5b0c41000000b0091dddda665fmr396601ybr.589.1676524741694; Wed, 15 Feb 2023 21:19:01 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:43 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-29-surenb@google.com> Subject: [PATCH v3 28/35] mm: prevent do_swap_page from handling page faults under VMA lock From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan , Laurent Dufour X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757963790976331659?= X-GMAIL-MSGID: =?utf-8?q?1757963790976331659?= Due to the possibility of do_swap_page dropping mmap_lock, abort fault handling under VMA lock and retry holding mmap_lock. This can be handled more gracefully in the future. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- mm/memory.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 13369ff15ec1..555612d153ad 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3688,6 +3688,11 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (!pte_unmap_same(vmf)) goto out; + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + ret = VM_FAULT_RETRY; + goto out; + } + entry = pte_to_swp_entry(vmf->orig_pte); if (unlikely(non_swap_entry(entry))) { if (is_migration_entry(entry)) { From patchwork Thu Feb 16 05:17:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57866 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp117550wrn; Wed, 15 Feb 2023 21:21:58 -0800 (PST) X-Google-Smtp-Source: AK7set8e2YaGYYdGMGdW+BskZ/osK2QrvyJL5xo5waveTjGTof7W0B0+4IcT+gvHKTB8yNvGxKkP X-Received: by 2002:a05:6402:1209:b0:4ab:ed5:dd46 with SMTP id c9-20020a056402120900b004ab0ed5dd46mr4503383edw.6.1676524918434; Wed, 15 Feb 2023 21:21:58 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676524918; cv=none; d=google.com; s=arc-20160816; b=lc5WfhZIlGxJow/Bb0xLgXdfcdBRHbKoroEkbiy3raLpyM8AJyG5iTvsJ06oFOMLU2 +ATQyoNelrUK5SOYSvcCiwvYCeIHflRlGMwvZVmAI2iNKZx+X59bYOJZdJwyG6zq+jIk dRbU+EfWSsFDHVRbCxJqwdR3C69UE6uL+vPv1DeKF2aojvY0IcEgx5TcqYr658PeHbrq dQtyyQ686fXi/ALbhCE/yzMOP9K0jwBXMa/c134RC3RK6gcIgufl+BjIgKvi+kHYAPRB zJbFDsHBnK45GlFpnHUKQ+jI1n3G4bojz+t3hA0LJaBcU+emmSY7XEQQCUznnExBT555 aKyQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=2G5AxmmOVrsRnXlCrW6/hvh5FUxSjD+0mS6LSWzYfho=; b=TljBfmPRnHisxsPCABwPl4uXb/WLvcc/g1q6o/7HA6Q5LXIjg1m2geIYo2wqKzyC+b 7WsyBtC9rPdjj+whpVfZaqIkYJWp6gdhOCU3+aYlsgNrVbdA/js8bcQFhRhCJHMrTLVp j9SD+Rc9BflrsPeUPjMkJmRdvBstgn0IWfXb+O2aleEPNTB3AopNqRlGWFX1jt2vBOYH oa/YrhH+vzHpvqNIaMv5KsQ2rkzzLl+2OcCy6OVcWL88m0VpWWPd3izKcdO9ZZjk0Zsg NgpnJkmT+KUR4Ljc6HIpbpdUE6JOCwAG13hmCnnF6VLvViz+ANWfF1UITu2rS+LZELx9 jLFA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=sqrDU+BC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ca8-20020aa7cd68000000b004acbf1d6be0si1079821edb.107.2023.02.15.21.21.35; Wed, 15 Feb 2023 21:21:58 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=sqrDU+BC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229808AbjBPFVD (ORCPT + 99 others); Thu, 16 Feb 2023 00:21:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46300 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229744AbjBPFT7 (ORCPT ); Thu, 16 Feb 2023 00:19:59 -0500 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3886A47435 for ; Wed, 15 Feb 2023 21:19:05 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-52f1641b79aso8473097b3.16 for ; Wed, 15 Feb 2023 21:19:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=2G5AxmmOVrsRnXlCrW6/hvh5FUxSjD+0mS6LSWzYfho=; b=sqrDU+BCO7AIuj2EKpC5JjJjchpfs4e7nICzmha43sQtLyfoiEZwLUOwsCD209H5Ui lu0l/EwT0lWrDRHYoglaYWRXNYPJ1g8O+GpgkaLO5UTCsF2sShD98xQNq7PDWtH66r6g ZsB+oua1JfA0KymwRm2/PgdgPfreEDDqnwQyHLX1ndmfndQgCQh8neVqDDo8QNluQIit psaYv9QythR+X2eyBbc1tAjHl/fZRS54lIuiB1Ux7odRjp5k2KlmRL+IIzajgpgWcCV1 agKdJl/mJ1QtI4UVDWAjmp+dEFlVMZYnML3DG7HfRey4EeaADk1hZvmemdIk3Yb/sCoK 4MXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2G5AxmmOVrsRnXlCrW6/hvh5FUxSjD+0mS6LSWzYfho=; b=qcvcJ7dngs44avmgN2LjXdG3axYwu76OjwEDxZMCxitMtZmyibKsvEieuLyIaDIgMT Ux9wBggc/OxN5ps3xj4qIVEzv1iLEt8Qj2FkAkXpspsWYuaSL6zSiuSy1ab/2sS0RNXb KM8f6tyGsD//Z+eT5vtsJJZHDI95ML/IT6riKW13ChKKyTuOOIWwCg2EBjdrN5fsf7X7 KFPkdUlkTBooBc9ux+B4zvNL3AttzdcrIlMsXsBTk7UZsNHeES0g3T5g5Q4W+eThqu4U zFCmeMyvQazWW1qHwvKxGempdmVRABGpRMde9dpQKLqTV8u2UVG+zyhPM5SRK/rgR1R9 esGw== X-Gm-Message-State: AO0yUKXlruymgF53lwLGKjQnHxo2z7Ccm+A2u2ihQPj9RIP0GdTT2KRm 8gFlMHt/odByyC+J98YaIDEquQxG4zw= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a05:6902:13c6:b0:8da:3163:224 with SMTP id y6-20020a05690213c600b008da31630224mr7ybu.0.1676524744013; Wed, 15 Feb 2023 21:19:04 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:44 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-30-surenb@google.com> Subject: [PATCH v3 29/35] mm: prevent userfaults to be handled under per-vma lock From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757963793025781724?= X-GMAIL-MSGID: =?utf-8?q?1757963793025781724?= Due to the possibility of handle_userfault dropping mmap_lock, avoid fault handling under VMA lock and retry holding mmap_lock. This can be handled more gracefully in the future. Signed-off-by: Suren Baghdasaryan Suggested-by: Peter Xu --- mm/memory.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 555612d153ad..751aebc1b29f 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5254,6 +5254,15 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, if (!vma_start_read(vma)) goto inval; + /* + * Due to the possibility of userfault handler dropping mmap_lock, avoid + * it for now and fall back to page fault handling under mmap_lock. + */ + if (userfaultfd_armed(vma)) { + vma_end_read(vma); + goto inval; + } + /* Check since vm_start/vm_end might change before we lock the VMA */ if (unlikely(address < vma->vm_start || address >= vma->vm_end)) { vma_end_read(vma); From patchwork Thu Feb 16 05:17:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57867 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp117685wrn; Wed, 15 Feb 2023 21:22:24 -0800 (PST) X-Google-Smtp-Source: AK7set9s3/VQEOxhDdsodv3M5Y3nNk+0OcXZ70oH5QQyUaSPEXIuyt1vmMIymVOV1Xy9n4+1/KXo X-Received: by 2002:a17:907:988b:b0:881:ad59:97f5 with SMTP id ja11-20020a170907988b00b00881ad5997f5mr4878161ejc.2.1676524944071; Wed, 15 Feb 2023 21:22:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676524944; cv=none; d=google.com; s=arc-20160816; b=rLwc+OZcFydqQb/9fNPx2DlW4g83+qaPMEVxi2cj9S+2JEhExxYC8xYatB2xTgwpOf V0ur1IpQJnVIGjT2HFxeN1lNsbxUFXIMrzkvSta9UJo+XcMtUlLEDjbM0Ks6UUh5+/4D 0QbryomLVNdGlMn8hZ/oaV4sFZRT0NShDOtRDud8vcTE7QWGJoid9dk/B2a9ZfbvGVuj zRLnVvQtoTOuHmSCpjM0ZW7HauX4Qb8R/iQKV4/Mi0BTjHDWQ19SD3uZH6TV0T/9or9f p/WTw31eab1gxjBnyGjRwgrTAfqxRwCBOP0M3mQxMuT31wjU5pq/rYsrsWCn27x43ck2 KVVw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=nAnyYB3Nw50vBlFaI5emapvq5zyyV08CcfGYNR6eiBY=; b=zQAnBfBFuOE1H69Bc8j0e/mda92hOgsphZiCsn+xWjTYh2Ta8RMmc4bmy/riF+3FVZ 85aVHaWUNuLBnOYzFYHjwZ+9R5YqlByRtbzJBDKMYR7yoKBy6RwQtl9brCn3Xo1reDaN l9cLwaEcp5U0jgxkDuZWX2cv0uct9kbTBXEofa22PXEyfnTLT/T6F27OsbI5QIKDpNbN QvR8Tum8ZeN6T/x2vNMc9YkgoMnQ1ydTjYZOWQsektg13ntChY+pTVVtHhhnQIInFlX1 /dPz8VM/xNdbmWkTOvTSHSFZhDv2eiC53CTiBbAU0hskwY3JEx1dTa+tWfFQ7k3dtuQI HEjA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=a3wN6yhQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id fu31-20020a170907b01f00b0088f92a0d234si790593ejc.586.2023.02.15.21.22.01; Wed, 15 Feb 2023 21:22:24 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=a3wN6yhQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229871AbjBPFVI (ORCPT + 99 others); Thu, 16 Feb 2023 00:21:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46610 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229810AbjBPFUO (ORCPT ); Thu, 16 Feb 2023 00:20:14 -0500 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C6B0A474DA for ; Wed, 15 Feb 2023 21:19:08 -0800 (PST) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-4fa63c84621so8412227b3.20 for ; Wed, 15 Feb 2023 21:19:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nAnyYB3Nw50vBlFaI5emapvq5zyyV08CcfGYNR6eiBY=; b=a3wN6yhQzn6ykzBbjResLcztZyUFHxuHfZtRwk9INETFTCK3R+0qznDveEPm+o/T6G wJ3wEiieeEUcjgJFPl0Q1WJhoSjHld23FZEQoY2Aq8Bhu2X6QB+bcNMvcqIxTt4gpYGo 62C0Ezu+RzKHJNHqe8a2WWwzq3BWBtjKwlH9vropbhb+eXqNxVp6cua3PyUJuPb2PSda pEBwm4EDV55dFWt5TRT7+9jNt1+V85COwPZnyzclXtLSHEwvb2nMc9rLXuPiU2TfsCi1 1h3mje1L/1Q/DM34hGjfe9SSEWf4rBhdCLw8Ngy6TcviKT73/jmv3Lf0iRWhfdK1hSLg WjQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nAnyYB3Nw50vBlFaI5emapvq5zyyV08CcfGYNR6eiBY=; b=uYUQxAKDyWERMQyQmXI5wERwaB0DatQd0b7hpuNZDZSUUU/N+8V1LX42K1/2HCt/2m hklzVtBxRmaVEByojFJXUDiSpZVyoNLNfghPj3K1pXv8suL0m3RsObeLyE8RuG9Gm1nl JHluN9PadzpgHw7btQNOEzcHevZ3lDNlT5u9fV6b+TzCQj6vOqWxKqhK431FO9MQuGhb pu7qWFju0v4zOkujUNAe2N+/zUFqFhrrdBVekQUNOZuvBDDctrMuS24i1thIOV3riaw6 MiiB2vZkzipPoke85ipDK0eEWDIgetTrPaZ8IZMlfdAGp+I/FiaOFwGfNFiN8vVMl01h Yu3w== X-Gm-Message-State: AO0yUKXCCqqiQAPrEKLw3HPyLmpvJLvLj1myWeS7UoQmq/OXY9uttLmW M1ZOod600BFtLqzimwhhf52pS7NqJfk= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a05:6902:18cc:b0:8a3:d147:280b with SMTP id ck12-20020a05690218cc00b008a3d147280bmr5ybb.3.1676524746144; Wed, 15 Feb 2023 21:19:06 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:45 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-31-surenb@google.com> Subject: [PATCH v3 30/35] mm: introduce per-VMA lock statistics From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757963819554421286?= X-GMAIL-MSGID: =?utf-8?q?1757963819554421286?= Add a new CONFIG_PER_VMA_LOCK_STATS config option to dump extra statistics about handling page fault under VMA lock. Signed-off-by: Suren Baghdasaryan --- include/linux/vm_event_item.h | 6 ++++++ include/linux/vmstat.h | 6 ++++++ mm/Kconfig.debug | 6 ++++++ mm/memory.c | 2 ++ mm/vmstat.c | 6 ++++++ 5 files changed, 26 insertions(+) diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 7f5d1caf5890..8abfa1240040 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -149,6 +149,12 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, #ifdef CONFIG_X86 DIRECT_MAP_LEVEL2_SPLIT, DIRECT_MAP_LEVEL3_SPLIT, +#endif +#ifdef CONFIG_PER_VMA_LOCK_STATS + VMA_LOCK_SUCCESS, + VMA_LOCK_ABORT, + VMA_LOCK_RETRY, + VMA_LOCK_MISS, #endif NR_VM_EVENT_ITEMS }; diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index 19cf5b6892ce..fed855bae6d8 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -125,6 +125,12 @@ static inline void vm_events_fold_cpu(int cpu) #define count_vm_tlb_events(x, y) do { (void)(y); } while (0) #endif +#ifdef CONFIG_PER_VMA_LOCK_STATS +#define count_vm_vma_lock_event(x) count_vm_event(x) +#else +#define count_vm_vma_lock_event(x) do {} while (0) +#endif + #define __count_zid_vm_events(item, zid, delta) \ __count_vm_events(item##_NORMAL - ZONE_NORMAL + zid, delta) diff --git a/mm/Kconfig.debug b/mm/Kconfig.debug index c3547a373c9c..4965a7333a3f 100644 --- a/mm/Kconfig.debug +++ b/mm/Kconfig.debug @@ -279,3 +279,9 @@ config DEBUG_KMEMLEAK_AUTO_SCAN If unsure, say Y. +config PER_VMA_LOCK_STATS + bool "Statistics for per-vma locks" + depends on PER_VMA_LOCK + default y + help + Statistics for per-vma locks. diff --git a/mm/memory.c b/mm/memory.c index 751aebc1b29f..94194a45ffa7 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5272,6 +5272,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, /* Check if the VMA got isolated after we found it */ if (vma->detached) { vma_end_read(vma); + count_vm_vma_lock_event(VMA_LOCK_MISS); /* The area was replaced with another one */ goto retry; } @@ -5280,6 +5281,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, return vma; inval: rcu_read_unlock(); + count_vm_vma_lock_event(VMA_LOCK_ABORT); return NULL; } #endif /* CONFIG_PER_VMA_LOCK */ diff --git a/mm/vmstat.c b/mm/vmstat.c index 1ea6a5ce1c41..4f1089a1860e 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1399,6 +1399,12 @@ const char * const vmstat_text[] = { "direct_map_level2_splits", "direct_map_level3_splits", #endif +#ifdef CONFIG_PER_VMA_LOCK_STATS + "vma_lock_success", + "vma_lock_abort", + "vma_lock_retry", + "vma_lock_miss", +#endif #endif /* CONFIG_VM_EVENT_COUNTERS || CONFIG_MEMCG */ }; #endif /* CONFIG_PROC_FS || CONFIG_SYSFS || CONFIG_NUMA || CONFIG_MEMCG */ From patchwork Thu Feb 16 05:17:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57872 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp123343wrn; Wed, 15 Feb 2023 21:41:05 -0800 (PST) X-Google-Smtp-Source: AK7set/rqs6KvP/4ZlQVdulYI3Qetqx1LWYKR308kSv4qQeyOqrAz8Yg2KzLnNu2tcMoITZYEdXv X-Received: by 2002:a17:906:aad4:b0:881:a3ec:2b40 with SMTP id kt20-20020a170906aad400b00881a3ec2b40mr4361083ejb.59.1676526065188; Wed, 15 Feb 2023 21:41:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676526065; cv=none; d=google.com; s=arc-20160816; b=LLvW2V55GFWkUc/lcXmzrmu0xxNBnRKfV8a0vIr5o7KqgAbgTeeFmCk412BaHEO7/j lob5qinQrgnPjMmcLA6ipwja5BckkbL/MoJpO8FPtL7UhhKrP104Q/xfDmNmHRWxwsy+ WG+rpLg4eOCoWjtqM5+t25yjhaMCy+eJPx7Riwf3iCdOBAoLOPhevPtUaRQYnHho2UtR cgbmB0beVEsBsXyTY7lkqlqrJC7MbIwrenJt3CHBJovW+2PJB56Z4VX2K6B6eO61KtOj eLh/iq8HT9pxad2smwZbeRqDMJzQ8ou3mqUCfCVBIsGUHyIFG6omq8082jBDnup2nm8y OvFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=IMepWjMdRmbly/mfPnmeELsKhzAKjn/e+MmdnnCshxo=; b=rAeAtFp3yUYsdP1ubtoxAfwAkt9tcWj6zK/cWuHSRqGpobdyeEKWqYf6uAzaqaGsbw YV22F/Mk8rQpkcPRabMQA8RI3kPQg55jgEUcxZUETfXKQBoN+Kib5UcGJ1vQwnqvAIJ4 zjIopySPp+LfhyN+G74pSkZ2nnTPABaMbzptDzk4vp+6LrxPmNQZOm/boH4FW9i6u8Jb sd05phx0/jV/vHTe6qDzYeXq8qunTHo3bOSP7YI6KNuX6IElGU/wgtdUra/+jpw9cpWi Bo2Ga+4+msV9U6h9DRBkavxoOWOwsPNqHs++A539xFwtCeoa7v+geRxkdMsiV1FyHTHe okrg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b="VYBN/Sx6"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id vw20-20020a170907059400b008b138235522si886649ejb.732.2023.02.15.21.40.41; Wed, 15 Feb 2023 21:41:05 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b="VYBN/Sx6"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229824AbjBPFVR (ORCPT + 99 others); Thu, 16 Feb 2023 00:21:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45790 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229820AbjBPFUW (ORCPT ); Thu, 16 Feb 2023 00:20:22 -0500 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8594B43452 for ; Wed, 15 Feb 2023 21:19:11 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-52ee93d7863so8376357b3.18 for ; Wed, 15 Feb 2023 21:19:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=IMepWjMdRmbly/mfPnmeELsKhzAKjn/e+MmdnnCshxo=; b=VYBN/Sx6H1zkhlZQfZvD0Liynh1SzBUBQ9VOXzPL7TQljh9ke7q1U13nsJvyL8jhTT VJdOiIwIjVau+ArgVKuyZBGkikb4wPVpg6tKpQ0o6b7A2xVbgU2vETmFbJMmm/kSYK3W jjQeULRVDfrD7eqjNl25w/HYQSurUi+SuUv5YnVVfpqkT7+a9RL2gs3g5vQZ1Iey2WKK za8Zu9gxbyi43cwHcYzE3Uj7rIHDEJE8sD5soBil1CQae5YChuN7lRgwr3AT9vKkAkhX nNMop/PJ3OYQ+4sMUjA9lss931diz+voYeLd0iyVQtxE8G6e+SWaxHgo0zdZNYDoa0CR el+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=IMepWjMdRmbly/mfPnmeELsKhzAKjn/e+MmdnnCshxo=; b=lXMIWEVuFCi8rc3d7zJAloqNOzmU2mJ/Z1sdb5HEuMj7c2+CDqwd9KMRdn9WNfrK1z rmNmEHTTHVNVP+xqUnVeBji0XiVrCLEIztGGmpXEUK8LBu4RMyfb2A+NLQYje9rM7Tts 6TkaxHVbVRIfQKyqxNUrjrq7GLv0cbwPEyTpdBe4novE2CV9i2IuIDOVIANsMNzMcV+r 9++Sg2ROja3hFt/rxyLIN3jkoAmQfks8NasOQwiPpDZ2qKBx9r5R93Dd3N0Oo/i2DTfU V7gSbCg8k77qsEQw1b1d5ELg+T9jvJPI2iQFFiuJiZMYY7MonZrFINiCXTWt8X5G89sU 6DqA== X-Gm-Message-State: AO0yUKVq/qeBcNOYeZvMFlGgHmxWiV0tBbJFxImHoJSfkm1ds3B+NYzd Hr7VUoTvDhVNEy29WMiliNRxl2a058Q= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a81:4516:0:b0:52e:e718:9641 with SMTP id s22-20020a814516000000b0052ee7189641mr16ywa.9.1676524748625; Wed, 15 Feb 2023 21:19:08 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:46 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-32-surenb@google.com> Subject: [PATCH v3 31/35] x86/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757964995475528250?= X-GMAIL-MSGID: =?utf-8?q?1757964995475528250?= Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Signed-off-by: Suren Baghdasaryan --- arch/x86/Kconfig | 1 + arch/x86/mm/fault.c | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 3604074a878b..3647f7bdb110 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -27,6 +27,7 @@ config X86_64 # Options that are inherently 64-bit kernel only: select ARCH_HAS_GIGANTIC_PAGE select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 + select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_USE_CMPXCHG_LOCKREF select HAVE_ARCH_SOFT_DIRTY select MODULES_USE_ELF_RELA diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index a498ae1fbe66..e4399983c50c 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -19,6 +19,7 @@ #include /* faulthandler_disabled() */ #include /* efi_crash_gracefully_on_page_fault()*/ #include +#include /* find_and_lock_vma() */ #include /* boot_cpu_has, ... */ #include /* dotraplinkage, ... */ @@ -1333,6 +1334,38 @@ void do_user_addr_fault(struct pt_regs *regs, } #endif +#ifdef CONFIG_PER_VMA_LOCK + if (!(flags & FAULT_FLAG_USER)) + goto lock_mmap; + + vma = lock_vma_under_rcu(mm, address); + if (!vma) + goto lock_mmap; + + if (unlikely(access_error(error_code, vma))) { + vma_end_read(vma); + goto lock_mmap; + } + fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); + vma_end_read(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + /* Quick path to respond to signals */ + if (fault_signal_pending(fault, regs)) { + if (!user_mode(regs)) + kernelmode_fixup_or_oops(regs, error_code, address, + SIGBUS, BUS_ADRERR, + ARCH_DEFAULT_PKEY); + return; + } +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ + /* * Kernel-mode access to the user address space should only occur * on well-defined single instructions listed in the exception @@ -1433,6 +1466,9 @@ void do_user_addr_fault(struct pt_regs *regs, } mmap_read_unlock(mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif if (likely(!(fault & VM_FAULT_ERROR))) return; From patchwork Thu Feb 16 05:17:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57871 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp121746wrn; Wed, 15 Feb 2023 21:35:49 -0800 (PST) X-Google-Smtp-Source: AK7set/hMKP0+M9dAleJWqxbDymeZKwKZWMDl3Yutqel/IQvwMScWV8Jhs5Wuv01mkV/7yyoPMQC X-Received: by 2002:a17:906:af1b:b0:88b:5798:2613 with SMTP id lx27-20020a170906af1b00b0088b57982613mr4787784ejb.20.1676525749564; Wed, 15 Feb 2023 21:35:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676525749; cv=none; d=google.com; s=arc-20160816; b=PL2tKkCb56fJVGmno8qy0SjsKNHi+8arrSEXEA8JSjLMH6yPy57xaZOSqn6ePt+SWi MR7seogMUUZ0wub2bKhMDL7wJlAhJLosaF4mjebT02RxFiPkZLvVAWf/FoDL9UgX2Fj7 nxp5NpUGGBLNzUU5VEMrvroEXj6use/a0L4UdRrHVK0EQgYXHP2kB/zIGMoNC7FnOi0j XWCFfvm7zQqEJxeQt4w1ZOJJnO6/xrZBoSMBggJoMVaWqiUS4ptq524FInGU1OPvH09Z 32blC0BQy39yPA84P3GtpoN2yMdkIBQnD3RtGQbutd32oF7j3zpVYLn0mLSg2lN+qPpN 0ExQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=REPTBg+i2LhaxaiXpF8lrF3nvI6zBhAH5JQxrJ1l/Gg=; b=O4S4jptJBtuYvCw7UhfmU4VLA+kDkaXW1VX2WyKC4uHDN9hmY6T4CtXh+hCj7Rxod4 RL9fBeB8WbjCdnPN76VZbBws3b8XFBJTqGsrbGPKhhKtFCCxJvN/m8yoLD+LtSoLt0If kiWZ1vaHlbtfUWLi9vsougek5VMshRNnMjhU3njWxYCH+6TIPSKlTE71z98rJvyo4zHi ZAHHiTwJ3IREP1ki9yhBMlDdkfkap9spr4kkWfXMYcDBs89XSKjgFBtedJ3i1mspFpd/ RlvVW+xffR+Bb46UtCYtyLvTsENsd6p9fSD5vjTaXHBC6ZnQB6FfTpXFxWgyaYcUHeth Iw3Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=aft7yDPM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id gk8-20020a17090790c800b008b152114123si21521ejb.97.2023.02.15.21.34.53; Wed, 15 Feb 2023 21:35:49 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=aft7yDPM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229834AbjBPFVT (ORCPT + 99 others); Thu, 16 Feb 2023 00:21:19 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46710 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229828AbjBPFUX (ORCPT ); Thu, 16 Feb 2023 00:20:23 -0500 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E2413474E7 for ; Wed, 15 Feb 2023 21:19:13 -0800 (PST) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-5005ef73cf3so8815987b3.2 for ; Wed, 15 Feb 2023 21:19:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=REPTBg+i2LhaxaiXpF8lrF3nvI6zBhAH5JQxrJ1l/Gg=; b=aft7yDPM8GMTHB06MFPSYtKh6oWhZ8a8LMFo6PcEI9pnNLntVOlXFNQs/1Yvf52/5v q7Mb9j3IbqGydOQagHUnsSeX/sfiJUssC8Nuv1RApw1dHW5T8YzKas/LnJ1orI0tqnNj 0nSP5pSWAhACQCuzYc7axzYrfhmKcyRBbpYSaW4sNEVU5J76PVU7lF/u3PAFDwnMoS0w 449nI+g12YKzlYuLkXvI0hBRvnRJV+xrScZAqKlTKhgGwG9sA/5/r9CeoVq4yrFt/yH+ v8L5/t9gLYQIZy7UnUIKS96KONHOM4/bkOuXUpcfaqGPz0nskdK5CUeyDODt2k3AI1C9 6jaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=REPTBg+i2LhaxaiXpF8lrF3nvI6zBhAH5JQxrJ1l/Gg=; b=Tj/vMUswQumq0ZHhxk8SiSna+fu3asJHNsf84EX/CF5d9P9yBy/OdYNBsXP5waug2R sHH0fwbEUWHjhS/2OGiHq5l5k2YsyihD8qTbPmsTPy3WLwiCCxOcBb/R8N3ijIZ8i5T6 p78nhsxAvCPU9O+jOiNDPuysgY3erUYTwknG56DBAWQ+cPEqbGJorkNN43k/m60tYli2 qUxIGcFkKixS6Lpr8Oyx9Nk7cluZgsvAr0ebEOWRaaksq35ao6gzVfYgr6DGYrJxBxx7 +lkV4ykQBHJz3dEEwRvPzKOnx3W65iFvIHuf02pNbXEggCFRorVVJKY02XfB7nbwH+eI hxkA== X-Gm-Message-State: AO0yUKVOWaRwqLDftdiBR2RGN2ay2hhw+7D7Qz+Xv2Or063mhy5ZcBvS ZNq5wnMS8Z+ZzHiSC5l2zcnx/U3HnKI= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a5b:d49:0:b0:90e:37e9:1e01 with SMTP id f9-20020a5b0d49000000b0090e37e91e01mr1ybr.1.1676524751194; Wed, 15 Feb 2023 21:19:11 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:47 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-33-surenb@google.com> Subject: [PATCH v3 32/35] arm64/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757964664451361712?= X-GMAIL-MSGID: =?utf-8?q?1757964664451361712?= Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Signed-off-by: Suren Baghdasaryan --- arch/arm64/Kconfig | 1 + arch/arm64/mm/fault.c | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index c5ccca26a408..9f2c0e352da3 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -95,6 +95,7 @@ config ARM64 select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 select ARCH_SUPPORTS_NUMA_BALANCING select ARCH_SUPPORTS_PAGE_TABLE_CHECK + select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT select ARCH_WANT_DEFAULT_BPF_JIT select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index f4cb0f85ccf4..9e0db5c387e3 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -535,6 +535,9 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, unsigned long vm_flags; unsigned int mm_flags = FAULT_FLAG_DEFAULT; unsigned long addr = untagged_addr(far); +#ifdef CONFIG_PER_VMA_LOCK + struct vm_area_struct *vma; +#endif if (kprobe_page_fault(regs, esr)) return 0; @@ -585,6 +588,36 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, addr); +#ifdef CONFIG_PER_VMA_LOCK + if (!(mm_flags & FAULT_FLAG_USER)) + goto lock_mmap; + + vma = lock_vma_under_rcu(mm, addr); + if (!vma) + goto lock_mmap; + + if (!(vma->vm_flags & vm_flags)) { + vma_end_read(vma); + goto lock_mmap; + } + fault = handle_mm_fault(vma, addr & PAGE_MASK, + mm_flags | FAULT_FLAG_VMA_LOCK, regs); + vma_end_read(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + /* Quick path to respond to signals */ + if (fault_signal_pending(fault, regs)) { + if (!user_mode(regs)) + goto no_context; + return 0; + } +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ /* * As per x86, we may deadlock here. However, since the kernel only * validly references user space from well defined areas of the code, @@ -628,6 +661,9 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, } mmap_read_unlock(mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif /* * Handle the "normal" (no error) case first. */ From patchwork Thu Feb 16 05:17:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57868 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp120130wrn; Wed, 15 Feb 2023 21:30:54 -0800 (PST) X-Google-Smtp-Source: AK7set++aUV8bF/69CPwP0tNvlVpKvGGbkQdoGVHdGQCFiWfEpWEVeyJ/EBAK2vTIRz6gO9954nn X-Received: by 2002:a17:906:a1cb:b0:88c:3502:56d4 with SMTP id bx11-20020a170906a1cb00b0088c350256d4mr4480558ejb.46.1676525454760; Wed, 15 Feb 2023 21:30:54 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676525454; cv=none; d=google.com; s=arc-20160816; b=alojLumvcvIjeRzKkBse2PDrvR7xwyxuwaAiGzIHA8IyMnIU5P8eSO9zAkyn4ywCI4 GsTkUfNbskAqrzRm3WRwxwnPhirH1ilpVCsp6lBC87nOI66Rr30OjzcfRHi1h0kpG5Gq bQwhk+2BN42N63vKZyvkLkFs6IX/k1Q0XRerqU8ULhg/kIUOIx/3bSNH/dDXRWg0ZLmG nfTTgEYCybfViuogXQoZa1+Aq48YaJ/ha9N0Z4ptqTDXIkqzxeKFM9SHlH5S8zs2S2dL gDIs9FVQK032Mp/RLtXN+9EBKgxJam5OYBafZwxeic+VLVV+6Hvg02+4HuW5r3O9TRs0 cPJA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=UQ3ySmwJxDlPKOGcVlpRpfqxPlYDfL6UGC9Ax4HzzOw=; b=s/g9AUEbYK/MGIT8EYdB6VER7gU2vqF498HBnsoFd8PoVS4m3QOeIqGoHJ4L2DYDUw WAuhvt5ZoWpV4pO6sMFk2LW/Wqz2wGWp20pD/pIcvM428GQPUmPqV/gAc0QFbtFCayWU 2N3pfEvE7t0yMPchx7vA1MmgghOdfTYJkOay0Mfoxgl7gerDhvzwcQAx53rjCqR7xCw7 pQMlwi6Ck2SGNRAxK7kYPchY4hkd/slijosOpENP96xWwHvLtLQB7ne058p+hcPbb3kT wiQtWCOaHzcmu1kn5TQB1pETcoUXgczZ9nElltd9unawKeaKecdVFyMhNi4DnGxzSIuJ 7ibg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=KGDJpcNA; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id z7-20020a17090674c700b008b14b964bf8si975608ejl.881.2023.02.15.21.30.28; Wed, 15 Feb 2023 21:30:54 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=KGDJpcNA; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229719AbjBPFV3 (ORCPT + 99 others); Thu, 16 Feb 2023 00:21:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46828 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229767AbjBPFUf (ORCPT ); Thu, 16 Feb 2023 00:20:35 -0500 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 718B642BC0 for ; Wed, 15 Feb 2023 21:19:18 -0800 (PST) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-5337e2cbc58so8660367b3.10 for ; Wed, 15 Feb 2023 21:19:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=UQ3ySmwJxDlPKOGcVlpRpfqxPlYDfL6UGC9Ax4HzzOw=; b=KGDJpcNA4oBtC5xQjxkPPkTW0kNBhMrSybgGLHrbBGSwSGEq+rySxMS3rk0dGM+Qo/ JpvUuwvGZIqjAVk+FVRsZ45D9AgvgQzyseylKUSLhPKRWFnmJExh3o/n5uArNAg9pZmc szoMYmExx5uq8bOtsbN9zvU5WJmapQpCTW9YfX5yCKinl4hK2fSO0QEupE6sIVAF6mis 0qvHvTjkF2AHE/zcyOYTuhnORntosRwKL2siaVlbrxc1aSWTq08YWLJWNhdZghaVxqan vIZPE33VZDXNYNH6tC5BE6Aqx0uXP/yBJaWCtfFr0Yvy7z7UrJ4q+Ys5HfXN3+WVCYvy Y1vg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UQ3ySmwJxDlPKOGcVlpRpfqxPlYDfL6UGC9Ax4HzzOw=; b=IlXF2F9OTqesI8pEhb1GvpT2QzL0mUv0EuG6asHPGcNWDlW55ZAk8jtl6h7EyAeBar f36HBv5gRS9mWypoWL7w+uBXfq6LjztmBXDdO1dXuzv5Ud+QxwedSdfqmyJPk+VJJlKP KBUnPQD10Rp8AUBH30r3S5VZ5Bnpe0/xhKgkHLFQU7j0EezIgmnIRkbXYGxjtOc8q8PC L7R9px4no80ymgFirtrIUJ3A4BcFx0L+TXCWKZl6wIfKTtemUmCppX6dgFfG4n1Xuu23 y0A5mAfzFQU2KHUShz+tsU5s8ZV6HTTBF5X4paDdvCSJ1WcRSpMqtvoBMY1KLeB+6ihS 8yoA== X-Gm-Message-State: AO0yUKUNDb5BPAO0acVjbBmFIMrxDt5AOvYgOkttecgIjRJ5039gtUZw lUjyGNHPfnY5CGCYsTDNTOqIcYb4Jtk= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a05:6902:114f:b0:855:fdcb:4467 with SMTP id p15-20020a056902114f00b00855fdcb4467mr33ybu.0.1676524754468; Wed, 15 Feb 2023 21:19:14 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:48 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-34-surenb@google.com> Subject: [PATCH v3 33/35] powerc/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757964354826035300?= X-GMAIL-MSGID: =?utf-8?q?1757964354826035300?= From: Laurent Dufour Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Copied from "x86/mm: try VMA lock-based page fault handling first" Signed-off-by: Laurent Dufour Signed-off-by: Suren Baghdasaryan --- arch/powerpc/mm/fault.c | 41 ++++++++++++++++++++++++++ arch/powerpc/platforms/powernv/Kconfig | 1 + arch/powerpc/platforms/pseries/Kconfig | 1 + 3 files changed, 43 insertions(+) diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c index 2bef19cc1b98..c7ae86b04b8a 100644 --- a/arch/powerpc/mm/fault.c +++ b/arch/powerpc/mm/fault.c @@ -469,6 +469,44 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, if (is_exec) flags |= FAULT_FLAG_INSTRUCTION; +#ifdef CONFIG_PER_VMA_LOCK + if (!(flags & FAULT_FLAG_USER)) + goto lock_mmap; + + vma = lock_vma_under_rcu(mm, address); + if (!vma) + goto lock_mmap; + + if (unlikely(access_pkey_error(is_write, is_exec, + (error_code & DSISR_KEYFAULT), vma))) { + int rc = bad_access_pkey(regs, address, vma); + + vma_end_read(vma); + return rc; + } + + if (unlikely(access_error(is_write, is_exec, vma))) { + int rc = bad_access(regs, address); + + vma_end_read(vma); + return rc; + } + + fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); + vma_end_read(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + if (fault_signal_pending(fault, regs)) + return user_mode(regs) ? 0 : SIGBUS; + +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ + /* When running in the kernel we expect faults to occur only to * addresses in user space. All other faults represent errors in the * kernel and should generate an OOPS. Unfortunately, in the case of an @@ -545,6 +583,9 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, mmap_read_unlock(current->mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif if (unlikely(fault & VM_FAULT_ERROR)) return mm_fault_error(regs, address, fault); diff --git a/arch/powerpc/platforms/powernv/Kconfig b/arch/powerpc/platforms/powernv/Kconfig index ae248a161b43..70a46acc70d6 100644 --- a/arch/powerpc/platforms/powernv/Kconfig +++ b/arch/powerpc/platforms/powernv/Kconfig @@ -16,6 +16,7 @@ config PPC_POWERNV select PPC_DOORBELL select MMU_NOTIFIER select FORCE_SMP + select ARCH_SUPPORTS_PER_VMA_LOCK default y config OPAL_PRD diff --git a/arch/powerpc/platforms/pseries/Kconfig b/arch/powerpc/platforms/pseries/Kconfig index a3b4d99567cb..e036a04ff1ca 100644 --- a/arch/powerpc/platforms/pseries/Kconfig +++ b/arch/powerpc/platforms/pseries/Kconfig @@ -21,6 +21,7 @@ config PPC_PSERIES select HOTPLUG_CPU select FORCE_SMP select SWIOTLB + select ARCH_SUPPORTS_PER_VMA_LOCK default y config PARAVIRT From patchwork Thu Feb 16 05:17:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57870 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp121706wrn; Wed, 15 Feb 2023 21:35:41 -0800 (PST) X-Google-Smtp-Source: AK7set/67QObR8F2AHUFmyCVX0+Z6zE1pMSZB3tJE91uBcQ1s26tldX08g67e7GP/5UT/Etp2fqt X-Received: by 2002:a17:906:134d:b0:8b1:33ba:2911 with SMTP id x13-20020a170906134d00b008b133ba2911mr4651892ejb.54.1676525741076; Wed, 15 Feb 2023 21:35:41 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676525741; cv=none; d=google.com; s=arc-20160816; b=CKifzkegJgpl+GrQLgDjG9FGnKtcRV83QaslriCtUYieociLEYrRHD8fxrX6UYAkFg g6eiHxXDUYnYTPchRK84MaH0v4D3oHKRZHu+aJvtHKrlaDmD48nr5H2vl142TlMB1E6L LyjxJBnAJ3doCxEy7nkot+hfmeR8q3KAqvvWG7seccJcxg0QLGbm91W3F9xOXYsBlrlZ kgbPNAc23Dse2QII+uQX+jA5UFq+XyTppC7hYpKWRLkJc7VO//qkcHWzonutqOFRLzfV KgsS2wWXi9ovCR1fX+8FebFlhXU5tTX7x4uBe1+LV7uyt2UwcmnC/AsrX147+RLYkte/ AYsw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=gCR40Nt3TWmXm3CiMW8diiOUlGd9n6tZbm3lsLizpE8=; b=hDTtAbkXA/v2KWW1aeqb5Ex7zBUtYjP0TqI8l25JCtFsYiGRIYhTFcjb8R8AK3Yt7M sfn3jCYdjByaEnD9suOrOw0jPHI1jsUIWcGhz9gVTtMlT1KUfjjoyjJMVTez0y6ftLjG o/G+Aoy1/coBSqFSWIq4VCkIbApRddWS9jUrvgyBStzcrie9SgGIt0TiEa7B01GFf/rz iJoETSS0v5X32kBUh2pDKHQuFAjJgdsSSup9eXIzyPUTFlFqwdgIxew1232nEWKn8rtu swgG2Ul2bSYq2RxGTuIfOvOCZKWJKqdhoADH5U6vAsC0gdqlzKJY5h7bpNeoX4K71qzP wQ7w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=JAL2Aq1j; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id fr31-20020a170906891f00b008b1293aef18si887631ejc.741.2023.02.15.21.35.18; Wed, 15 Feb 2023 21:35:41 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=JAL2Aq1j; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229782AbjBPFVb (ORCPT + 99 others); Thu, 16 Feb 2023 00:21:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46878 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229717AbjBPFUs (ORCPT ); Thu, 16 Feb 2023 00:20:48 -0500 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 78AF1460A3 for ; Wed, 15 Feb 2023 21:19:19 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-52ec987d600so8366607b3.21 for ; Wed, 15 Feb 2023 21:19:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=gCR40Nt3TWmXm3CiMW8diiOUlGd9n6tZbm3lsLizpE8=; b=JAL2Aq1jm/4r1JqeZYEWBrue/W3MPTrJmtqCzmWLkH/WIKCq/WNMS7CZcuyalyxOey IosycXB+MxcwT4QBmmHaciSLx6z1YK/qs/xuZHAZ6O8LZPapUolphd/QYky6xf7HXG4m TIajJoSTXZu0Yk9TmYuKl91OepBHI9VMsFb9wtZhCPHchjEtdJSeWxPPYnIrHgMp6AHH zReb74rJ9HRrecOXakA/GD9/YkepqNwyoeE0Te0FwV1uNkov7IGEqryUYtJWv/bTKuMJ k46shKzXYlPWW+imaaqXvyzMwcruK2EkfBqoUiODxJ9SCK0t0Bhmm1Z4Ww2xAI1rN1gq fZJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gCR40Nt3TWmXm3CiMW8diiOUlGd9n6tZbm3lsLizpE8=; b=ne2K9KfM1ct5nTmkX7ZN4nf31K0RWOY4fgfp3HjE1tLCv5IRVCzXsWF9smSE+pmDs7 cu49UBbmaUYU8wp6VqKk3mztEfx+B+cgIHtXax+NjwmCul2lnt2XvGlVycCd1fuygooC 06E9MJ0esIG+qcz6SjeyyNKzGt64hAJzEMlxnjlnIgXfP6gzvFqC+fTFSuzDxQhU152m x+NJbYnFYSFWRrYBxXZCLH+bymZShB/E0BA1/++vZMGsmlIKy7hgfb6kBGDw6tMwHsws wm9vQlYqxFfRiUG9EQiCwyFQiT//Kdqh/R/hm6QtZ1u/KAGPVavZJmDWAcTIdYdxEKvm Ns5g== X-Gm-Message-State: AO0yUKUq/dtOs4gzVO6zXEXxAZYPc8QqHZOSck2JtSzb0qpbZG41Lnq0 RHDB7C51Ux3ZvLUhLhtzmlVBfSklfzg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a5b:609:0:b0:80b:adf8:4f9b with SMTP id d9-20020a5b0609000000b0080badf84f9bmr575565ybq.33.1676524756940; Wed, 15 Feb 2023 21:19:16 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:49 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-35-surenb@google.com> Subject: [PATCH v3 34/35] mm/mmap: free vm_area_struct without call_rcu in exit_mmap From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757964655702272729?= X-GMAIL-MSGID: =?utf-8?q?1757964655702272729?= call_rcu() can take a long time when callback offloading is enabled. Its use in the vm_area_free can cause regressions in the exit path when multiple VMAs are being freed. Because exit_mmap() is called only after the last mm user drops its refcount, the page fault handlers can't be racing with it. Any other possible user like oom-reaper or process_mrelease are already synchronized using mmap_lock. Therefore exit_mmap() can free VMAs directly, without the use of call_rcu(). Expose __vm_area_free() and use it from exit_mmap() to avoid possible call_rcu() floods and performance regressions caused by it. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 2 ++ kernel/fork.c | 2 +- mm/mmap.c | 11 +++++++---- 3 files changed, 10 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 3c9167529417..cedef02dfd2b 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -256,6 +256,8 @@ void setup_initial_init_mm(void *start_code, void *end_code, struct vm_area_struct *vm_area_alloc(struct mm_struct *); struct vm_area_struct *vm_area_dup(struct vm_area_struct *); void vm_area_free(struct vm_area_struct *); +/* Use only if VMA has no other users */ +void __vm_area_free(struct vm_area_struct *vma); #ifndef CONFIG_MMU extern struct rb_root nommu_region_tree; diff --git a/kernel/fork.c b/kernel/fork.c index a08cc0e2bfde..d0999de82f94 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -480,7 +480,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return new; } -static void __vm_area_free(struct vm_area_struct *vma) +void __vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); kmem_cache_free(vm_area_cachep, vma); diff --git a/mm/mmap.c b/mm/mmap.c index adf40177e68f..d847be615720 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -133,7 +133,7 @@ void unlink_file_vma(struct vm_area_struct *vma) /* * Close a vm structure and free it. */ -static void remove_vma(struct vm_area_struct *vma) +static void remove_vma(struct vm_area_struct *vma, bool unreachable) { might_sleep(); if (vma->vm_ops && vma->vm_ops->close) @@ -141,7 +141,10 @@ static void remove_vma(struct vm_area_struct *vma) if (vma->vm_file) fput(vma->vm_file); mpol_put(vma_policy(vma)); - vm_area_free(vma); + if (unreachable) + __vm_area_free(vma); + else + vm_area_free(vma); } static inline struct vm_area_struct *vma_prev_limit(struct vma_iterator *vmi, @@ -2135,7 +2138,7 @@ static inline void remove_mt(struct mm_struct *mm, struct ma_state *mas) if (vma->vm_flags & VM_ACCOUNT) nr_accounted += nrpages; vm_stat_account(mm, vma->vm_flags, -nrpages); - remove_vma(vma); + remove_vma(vma, false); } vm_unacct_memory(nr_accounted); validate_mm(mm); @@ -3085,7 +3088,7 @@ void exit_mmap(struct mm_struct *mm) do { if (vma->vm_flags & VM_ACCOUNT) nr_accounted += vma_pages(vma); - remove_vma(vma); + remove_vma(vma, true); count++; cond_resched(); } while ((vma = mas_find(&mas, ULONG_MAX)) != NULL); From patchwork Thu Feb 16 05:17:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 57869 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp121694wrn; Wed, 15 Feb 2023 21:35:39 -0800 (PST) X-Google-Smtp-Source: AK7set+luVrEVN4KsVo/KG5AOFO5cxdvVKEXlPIn4b4J0AQ4wX7QcSBvv84V7odALSrBRboa9Ajh X-Received: by 2002:a17:906:8a64:b0:8aa:c2bd:a71c with SMTP id hy4-20020a1709068a6400b008aac2bda71cmr5099375ejc.75.1676525739144; Wed, 15 Feb 2023 21:35:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676525739; cv=none; d=google.com; s=arc-20160816; b=tlNeagoHutKYuw/wpTAJQtvI3Wl7HbPLBXxctEMCY7ynk3qh0sb3pQZJL+9r9kyFse QI2HsKi858LuhX+WRIDabDLxFLp7lLk2a8zqZLsfU/NA3vV6jfSJXVp/GCgNO9SZg5ty zXNHr5WViqoOraEZjPDLG299iXVzj8Ng+CQtMeR2tDfgDnya7fews0VbQu02uCAdhzoR MAWZ+fzD9OtV47QpB/Of2dYR+cjg3qJHBY5OFvWEqJ0gDkYfpf2ERZi1L18/jz9Arpo3 CvsRB90fXheXqIjkfWHWIWVMINeQTP8ZJ1ilyE4uZIuCCtXALyvJKij74qBtqJtoc1/i Howw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=/bIj4YrLTVPG3QwseSVPiqG0bxdZfORy7sIu/6u8O1k=; b=APCJ+F/yY/9BERYRnNCfkNDqx22k9FPsm6oQyAtOpsrtlrIYgMu9Uq9sBmvBDih4OU VSxLPVakq5Fb7vIRcTg8RlzHwbAdnJ1Pxc7fvds3IZwW6d/NBsyDXEUvQ7OmicPh9gZd eZ87eapK5js4ifgfqvoVbCZVVEcoryU4tukofNYwqeMP6FvUN9F46BJq9nRoHWiY8ep0 PYxX3JjsnlWiQtJrx/yq2yRIFA1WQYUnnxjM6qXdrnpOY9oRORQx+mj56dIoVpnDj0/G 1GlYn/XE2ROg8f7N9JjppyxN7UrfYmim1R/CpPS0f0vpdWsXpAbKaiH1n+yXY/gCWE0O YbaA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=kcrkMPk5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id vs4-20020a170907138400b0086f05817f39si928550ejb.39.2023.02.15.21.35.15; Wed, 15 Feb 2023 21:35:39 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=kcrkMPk5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229787AbjBPFVe (ORCPT + 99 others); Thu, 16 Feb 2023 00:21:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46900 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229776AbjBPFUt (ORCPT ); Thu, 16 Feb 2023 00:20:49 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2AFB643469 for ; Wed, 15 Feb 2023 21:19:21 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id k9-20020a25bec9000000b00944353b6a81so864551ybm.7 for ; Wed, 15 Feb 2023 21:19:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/bIj4YrLTVPG3QwseSVPiqG0bxdZfORy7sIu/6u8O1k=; b=kcrkMPk51QPn4U+Qy9PCMBVvz0+hngRjWyUVg3mDY5eCPo5kdNdGQXXAF5nSKjvtxI ovQfwzu/2jGBzDo4t58Z0TFwD6xaqMMee+NhMptwfny7jSZBnt/+oKNA3aeQgGljd8sS akXety944Mg3EUnDXa5SJO4Zw7aGeTPZtwigMyHbadec4ytdul9kOPOFJ5Xa+9QmQgoM XeWGGTE6ollJyfe3sBlUvV2EFNH4AWaioDHTISUd5H/KMkPUAsjYGzb18bayjRFOkrs2 88EaIEp+Df0ZGeekcwiDAUNprBAPetgMpUou+tgl+Oy9Z03QrXgN/piqimqVkufqqcTw xlTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/bIj4YrLTVPG3QwseSVPiqG0bxdZfORy7sIu/6u8O1k=; b=DQhhhcQxTqFfKwjtkbtZ+npx3UhLA0Tn0hgwW5AgAjcXAzMwobubUJr919IreKbDwE 9AIQddKGd7MLyWfWru1d+xkQblxK2nvp+gdOelQoxm0oD3CaR1rqSti9JJB0Fy1S+AB/ T8pel0u3zVv1HkUWtNLcW+pzpPyt2lRTpEIUDKAhvIRVuKda5FnSoC09mxRXLa/1oDJV w+p7+E1XAxwVz+Oic5JHmoRIwDuH7eNR1DtMmiLyP0PMxa6qSDUs9cKhwLI/bT4Yr/b2 lci9JcY8154OeRbDe+fPwN537do8c2dKCGcAg3wcUMu2uPTn4L/P6Xw4/dOuVw1/DZ8m Es/A== X-Gm-Message-State: AO0yUKU3i4TBtYYC0OvXrl4HbxsBDSYjCsTkBBhNR0isRxQcYrz0OHMP P4KNVfwRfIbuU8f9e596BLNciVDl94Y= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a0d:ea05:0:b0:52e:fbcd:e1b4 with SMTP id t5-20020a0dea05000000b0052efbcde1b4mr590898ywe.0.1676524759267; Wed, 15 Feb 2023 21:19:19 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:50 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-36-surenb@google.com> Subject: [PATCH v3 35/35] mm: separate vma->lock from vm_area_struct From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757964653587838722?= X-GMAIL-MSGID: =?utf-8?q?1757964653587838722?= vma->lock being part of the vm_area_struct causes performance regression during page faults because during contention its count and owner fields are constantly updated and having other parts of vm_area_struct used during page fault handling next to them causes constant cache line bouncing. Fix that by moving the lock outside of the vm_area_struct. All attempts to keep vma->lock inside vm_area_struct in a separate cache line still produce performance regression especially on NUMA machines. Smallest regression was achieved when lock is placed in the fourth cache line but that bloats vm_area_struct to 256 bytes. Considering performance and memory impact, separate lock looks like the best option. It increases memory footprint of each VMA but that can be optimized later if the new size causes issues. Note that after this change vma_init() does not allocate or initialize vma->lock anymore. A number of drivers allocate a pseudo VMA on the stack but they never use the VMA's lock, therefore it does not need to be allocated. The future drivers which might need the VMA lock should use vm_area_alloc()/vm_area_free() to allocate the VMA. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 23 ++++++------- include/linux/mm_types.h | 6 +++- kernel/fork.c | 73 ++++++++++++++++++++++++++++++++-------- 3 files changed, 74 insertions(+), 28 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index cedef02dfd2b..96b18ef3bfa3 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -627,12 +627,6 @@ struct vm_operations_struct { }; #ifdef CONFIG_PER_VMA_LOCK -static inline void vma_init_lock(struct vm_area_struct *vma) -{ - init_rwsem(&vma->lock); - vma->vm_lock_seq = -1; -} - /* * Try to read-lock a vma. The function is allowed to occasionally yield false * locked result to avoid performance overhead, in which case we fall back to @@ -644,17 +638,17 @@ static inline bool vma_start_read(struct vm_area_struct *vma) if (vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq)) return false; - if (unlikely(down_read_trylock(&vma->lock) == 0)) + if (unlikely(down_read_trylock(&vma->vm_lock->lock) == 0)) return false; /* * Overflow might produce false locked result. * False unlocked result is impossible because we modify and check - * vma->vm_lock_seq under vma->lock protection and mm->mm_lock_seq + * vma->vm_lock_seq under vma->vm_lock protection and mm->mm_lock_seq * modification invalidates all existing locks. */ if (unlikely(vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq))) { - up_read(&vma->lock); + up_read(&vma->vm_lock->lock); return false; } return true; @@ -663,7 +657,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) static inline void vma_end_read(struct vm_area_struct *vma) { rcu_read_lock(); /* keeps vma alive till the end of up_read */ - up_read(&vma->lock); + up_read(&vma->vm_lock->lock); rcu_read_unlock(); } @@ -681,9 +675,9 @@ static inline void vma_start_write(struct vm_area_struct *vma) if (vma->vm_lock_seq == mm_lock_seq) return; - down_write(&vma->lock); + down_write(&vma->vm_lock->lock); vma->vm_lock_seq = mm_lock_seq; - up_write(&vma->lock); + up_write(&vma->vm_lock->lock); } static inline void vma_assert_write_locked(struct vm_area_struct *vma) @@ -720,6 +714,10 @@ static inline void vma_mark_detached(struct vm_area_struct *vma, #endif /* CONFIG_PER_VMA_LOCK */ +/* + * WARNING: vma_init does not initialize vma->vm_lock. + * Use vm_area_alloc()/vm_area_free() if vma needs locking. + */ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) { static const struct vm_operations_struct dummy_vm_ops = {}; @@ -729,7 +727,6 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_ops = &dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); vma_mark_detached(vma, false); - vma_init_lock(vma); } /* Use when VMA is not part of the VMA tree and needs no locking */ diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 212e7f923a69..30d4f867ae56 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -471,6 +471,10 @@ struct anon_vma_name { char name[]; }; +struct vma_lock { + struct rw_semaphore lock; +}; + /* * This struct describes a virtual memory area. There is one of these * per VM-area/task. A VM area is any part of the process virtual memory @@ -510,7 +514,7 @@ struct vm_area_struct { #ifdef CONFIG_PER_VMA_LOCK int vm_lock_seq; - struct rw_semaphore lock; + struct vma_lock *vm_lock; /* Flag to indicate areas detached from the mm->mm_mt tree */ bool detached; diff --git a/kernel/fork.c b/kernel/fork.c index d0999de82f94..a152804faa14 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -451,13 +451,49 @@ static struct kmem_cache *vm_area_cachep; /* SLAB cache for mm_struct structures (tsk->mm) */ static struct kmem_cache *mm_cachep; +#ifdef CONFIG_PER_VMA_LOCK + +/* SLAB cache for vm_area_struct.lock */ +static struct kmem_cache *vma_lock_cachep; + +static bool vma_lock_alloc(struct vm_area_struct *vma) +{ + vma->vm_lock = kmem_cache_alloc(vma_lock_cachep, GFP_KERNEL); + if (!vma->vm_lock) + return false; + + init_rwsem(&vma->vm_lock->lock); + vma->vm_lock_seq = -1; + + return true; +} + +static inline void vma_lock_free(struct vm_area_struct *vma) +{ + kmem_cache_free(vma_lock_cachep, vma->vm_lock); +} + +#else /* CONFIG_PER_VMA_LOCK */ + +static inline bool vma_lock_alloc(struct vm_area_struct *vma) { return true; } +static inline void vma_lock_free(struct vm_area_struct *vma) {} + +#endif /* CONFIG_PER_VMA_LOCK */ + struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) { struct vm_area_struct *vma; vma = kmem_cache_alloc(vm_area_cachep, GFP_KERNEL); - if (vma) - vma_init(vma, mm); + if (!vma) + return NULL; + + vma_init(vma, mm); + if (!vma_lock_alloc(vma)) { + kmem_cache_free(vm_area_cachep, vma); + return NULL; + } + return vma; } @@ -465,24 +501,30 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) { struct vm_area_struct *new = kmem_cache_alloc(vm_area_cachep, GFP_KERNEL); - if (new) { - ASSERT_EXCLUSIVE_WRITER(orig->vm_flags); - ASSERT_EXCLUSIVE_WRITER(orig->vm_file); - /* - * orig->shared.rb may be modified concurrently, but the clone - * will be reinitialized. - */ - data_race(memcpy(new, orig, sizeof(*new))); - INIT_LIST_HEAD(&new->anon_vma_chain); - vma_init_lock(new); - dup_anon_vma_name(orig, new); + if (!new) + return NULL; + + ASSERT_EXCLUSIVE_WRITER(orig->vm_flags); + ASSERT_EXCLUSIVE_WRITER(orig->vm_file); + /* + * orig->shared.rb may be modified concurrently, but the clone + * will be reinitialized. + */ + data_race(memcpy(new, orig, sizeof(*new))); + if (!vma_lock_alloc(new)) { + kmem_cache_free(vm_area_cachep, new); + return NULL; } + INIT_LIST_HEAD(&new->anon_vma_chain); + dup_anon_vma_name(orig, new); + return new; } void __vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); + vma_lock_free(vma); kmem_cache_free(vm_area_cachep, vma); } @@ -493,7 +535,7 @@ static void vm_area_free_rcu_cb(struct rcu_head *head) vm_rcu); /* The vma should not be locked while being destroyed. */ - VM_BUG_ON_VMA(rwsem_is_locked(&vma->lock), vma); + VM_BUG_ON_VMA(rwsem_is_locked(&vma->vm_lock->lock), vma); __vm_area_free(vma); } #endif @@ -3089,6 +3131,9 @@ void __init proc_caches_init(void) NULL); vm_area_cachep = KMEM_CACHE(vm_area_struct, SLAB_PANIC|SLAB_ACCOUNT); +#ifdef CONFIG_PER_VMA_LOCK + vma_lock_cachep = KMEM_CACHE(vma_lock, SLAB_PANIC|SLAB_ACCOUNT); +#endif mmap_init(); nsproxy_cache_init(); }