From patchwork Fri Jan 27 19:40:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49660 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1016612wrn; Fri, 27 Jan 2023 11:59:08 -0800 (PST) X-Google-Smtp-Source: AK7set91vIkD2gfPbW5t6q+gx11lqDXE9h2//8Rrz2iSM30VHmtKcv6yCEHCEYKbQLM9PZ/s6s1F X-Received: by 2002:a05:6a00:2289:b0:592:fc9f:8ae5 with SMTP id f9-20020a056a00228900b00592fc9f8ae5mr2410389pfe.23.1674849548465; Fri, 27 Jan 2023 11:59:08 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674849548; cv=none; d=google.com; s=arc-20160816; b=H2QiaW9yQf0tnETk07wCCNrE1hE5rLgMNgjG+pGDhOCKEqx1TzuSb9guuwJDwz2gMk 0wk7hGX6rt5v2TL5tIBBkQ00AsS6IdnP1y9EerBqg6olWW1ux6BCHAqWo7SBmlrOmKbx NstdWMJIBXbuKryh20SP6MPGdIfHTNTHrbs9iUG89z2XgeKIwxBVvc/UHHhLcv+dP10F uhl6dpf42g45zJRL1wAvHBW9azb8BdmyJzzFI4jO1FbBWYK6iymL43JnD9c+VletW7Rg jbA+BtKR9WFh6VNtWyPwtthJl/khtIG2Z254NeJlQwzBUxCJvT6qFkBckQ3xmL99x2aC 4/XA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=SRJKiHKrq/SvrK23nJ6WvAwYz6ftOZ3c7xtMrR/Zkmo=; b=Do/GWK9UMY9FEf3exdj07kGAo85iq+6+dBd+EigogiFVYWlUnkjUCW7pqXjTh8mIr9 8AbFY7vigXJWXGqPhOH6Ko1DR279ySYxLl/py221Nzed5HCi3qap6nD+kiHeTEbf1FTH 1Frz1rnfzgdtHyVgVpkLcBlWM6yZwtKDThApiVL0KrUCRycVdsVhJwA8zJzFUl6J1bnm zEKiQeCTV1w3ECpBmWHCK0lxJqu07xkTtQFYvlPC71E+AGEFq+GFbU8B6PyQ9RxhQnF8 PAO226jUDGboZpbLXiwkEWitbBdgUVWXSKHBT7Ba3RnR2ULlMufglGQ/NCqbNwQxMDiq eNsw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=q++c4JbF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q5-20020a056a0002a500b005627f4b21absi4926381pfs.296.2023.01.27.11.58.54; Fri, 27 Jan 2023 11:59:08 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=q++c4JbF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232622AbjA0T6J (ORCPT + 99 others); Fri, 27 Jan 2023 14:58:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37590 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233105AbjA0T5i (ORCPT ); Fri, 27 Jan 2023 14:57:38 -0500 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 827C68CE0E for ; Fri, 27 Jan 2023 11:55:37 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-4fee82718afso65564437b3.5 for ; Fri, 27 Jan 2023 11:55:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=SRJKiHKrq/SvrK23nJ6WvAwYz6ftOZ3c7xtMrR/Zkmo=; b=q++c4JbFs1cULNmXHJ6C3DczcQ0hXzOKIr6+9Qx2yz39vZWA+rjLc18PKWJkWy6B+C UeBdui8K3sk0l0OIGcxNzLRt494xbxI7WVRAtP+VLTOL1/q+UNWJf0WDLHuQPthduco2 hKKj52t3wKu54Vbflyx5DWFngCga+0YCh2aWf3wXfSgRE76VHyLLWka1QvKlb5EEsVq+ yKsc5SnLc4E9cZlzwXYLALtVVR/qLY2NGEjlF8VIz3gJJ8VVrEVxdVCt2bqpvNVXSzLO bCSv1unz+7C8JA4tTmfPKXKgKgaiBEIUDtx7XKU+dM+xg3yKNZdzkdQzW0TW1MecPnG9 7xHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=SRJKiHKrq/SvrK23nJ6WvAwYz6ftOZ3c7xtMrR/Zkmo=; b=7lYglgmIJOhTygxrBuL2xHFAQv6Cyeu6N45WB8ACorCeqqC+OmB6jswNMJ+VBsA+2t BfR82ptjgHY3mkm9ZVoALZYaK+N6hI0pO+n+6R+00yoec21jB+nwKD3I25ZMjRSNhsF4 E50lXpwQ6Cky4oh300k5CpVFNcbTIhMLxWKG5jXg+22j3FNNPLCcs8HfhsW81Tolf2BC f1Aa3aFDvuUH+7flBPnpHAp4YDyOJAJesEQ0VF08xiCk8LfhtcspvWR16QSW5lFN/WYj R4sZXWztTqFlFZKoapKa7IXdl2ufzAtBJ7o9RS1/d7DrG8Jzu61loAiPzxVIvWhHFRTN aD0A== X-Gm-Message-State: AFqh2koej8TwI34zFPQB3NG0Mm7M5nm4XGoesfk+4YGnPaZzhplMjB+A S+jEfAFGrOgGNMTPGUQz+kpoFTAWxl8= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a25:4e54:0:b0:7fe:6d35:1a28 with SMTP id c81-20020a254e54000000b007fe6d351a28mr4351236ybb.534.1674848476456; Fri, 27 Jan 2023 11:41:16 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:38 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-2-surenb@google.com> Subject: [PATCH v2 01/33] maple_tree: Be more cautious about dead nodes From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, Liam Howlett X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756207040040197961?= X-GMAIL-MSGID: =?utf-8?q?1756207040040197961?= From: Liam Howlett ma_pivots() and ma_data_end() may be called with a dead node. Ensure to that the node isn't dead before using the returned values. This is necessary for RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 52 +++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 43 insertions(+), 9 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 5e9703189259..e5eeecd14eee 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -537,6 +537,7 @@ static inline bool ma_dead_node(const struct maple_node *node) return (parent == node); } + /* * mte_dead_node() - check if the @enode is dead. * @enode: The encoded maple node @@ -618,6 +619,8 @@ static inline unsigned int mas_alloc_req(const struct ma_state *mas) * @node - the maple node * @type - the node type * + * In the event of a dead node, this array may be %NULL + * * Return: A pointer to the maple node pivots */ static inline unsigned long *ma_pivots(struct maple_node *node, @@ -1089,8 +1092,11 @@ static int mas_ascend(struct ma_state *mas) a_type = mas_parent_enum(mas, p_enode); a_node = mte_parent(p_enode); a_slot = mte_parent_slot(p_enode); - pivots = ma_pivots(a_node, a_type); a_enode = mt_mk_node(a_node, a_type); + pivots = ma_pivots(a_node, a_type); + + if (unlikely(ma_dead_node(a_node))) + return 1; if (!set_min && a_slot) { set_min = true; @@ -1394,6 +1400,9 @@ static inline unsigned char ma_data_end(struct maple_node *node, { unsigned char offset; + if (!pivots) + return 0; + if (type == maple_arange_64) return ma_meta_end(node, type); @@ -1429,6 +1438,9 @@ static inline unsigned char mas_data_end(struct ma_state *mas) return ma_meta_end(node, type); pivots = ma_pivots(node, type); + if (unlikely(ma_dead_node(node))) + return 0; + offset = mt_pivots[type] - 1; if (likely(!pivots[offset])) return ma_meta_end(node, type); @@ -4498,6 +4510,9 @@ static inline int mas_prev_node(struct ma_state *mas, unsigned long min) node = mas_mn(mas); slots = ma_slots(node, mt); pivots = ma_pivots(node, mt); + if (unlikely(ma_dead_node(node))) + return 1; + mas->max = pivots[offset]; if (offset) mas->min = pivots[offset - 1] + 1; @@ -4519,6 +4534,9 @@ static inline int mas_prev_node(struct ma_state *mas, unsigned long min) slots = ma_slots(node, mt); pivots = ma_pivots(node, mt); offset = ma_data_end(node, mt, pivots, mas->max); + if (unlikely(ma_dead_node(node))) + return 1; + if (offset) mas->min = pivots[offset - 1] + 1; @@ -4567,6 +4585,7 @@ static inline int mas_next_node(struct ma_state *mas, struct maple_node *node, struct maple_enode *enode; int level = 0; unsigned char offset; + unsigned char node_end; enum maple_type mt; void __rcu **slots; @@ -4590,7 +4609,11 @@ static inline int mas_next_node(struct ma_state *mas, struct maple_node *node, node = mas_mn(mas); mt = mte_node_type(mas->node); pivots = ma_pivots(node, mt); - } while (unlikely(offset == ma_data_end(node, mt, pivots, mas->max))); + node_end = ma_data_end(node, mt, pivots, mas->max); + if (unlikely(ma_dead_node(node))) + return 1; + + } while (unlikely(offset == node_end)); slots = ma_slots(node, mt); pivot = mas_safe_pivot(mas, pivots, ++offset, mt); @@ -4606,6 +4629,9 @@ static inline int mas_next_node(struct ma_state *mas, struct maple_node *node, mt = mte_node_type(mas->node); slots = ma_slots(node, mt); pivots = ma_pivots(node, mt); + if (unlikely(ma_dead_node(node))) + return 1; + offset = 0; pivot = pivots[0]; } @@ -4652,11 +4678,14 @@ static inline void *mas_next_nentry(struct ma_state *mas, return NULL; } - pivots = ma_pivots(node, type); slots = ma_slots(node, type); - mas->index = mas_safe_min(mas, pivots, mas->offset); + pivots = ma_pivots(node, type); count = ma_data_end(node, type, pivots, mas->max); - if (ma_dead_node(node)) + if (unlikely(ma_dead_node(node))) + return NULL; + + mas->index = mas_safe_min(mas, pivots, mas->offset); + if (unlikely(ma_dead_node(node))) return NULL; if (mas->index > max) @@ -4810,6 +4839,11 @@ static inline void *mas_prev_nentry(struct ma_state *mas, unsigned long limit, slots = ma_slots(mn, mt); pivots = ma_pivots(mn, mt); + if (unlikely(ma_dead_node(mn))) { + mas_rewalk(mas, index); + goto retry; + } + if (offset == mt_pivots[mt]) pivot = mas->max; else @@ -6624,11 +6658,11 @@ static inline void *mas_first_entry(struct ma_state *mas, struct maple_node *mn, while (likely(!ma_is_leaf(mt))) { MT_BUG_ON(mas->tree, mte_dead_node(mas->node)); slots = ma_slots(mn, mt); - pivots = ma_pivots(mn, mt); - max = pivots[0]; entry = mas_slot(mas, slots, 0); + pivots = ma_pivots(mn, mt); if (unlikely(ma_dead_node(mn))) return NULL; + max = pivots[0]; mas->node = entry; mn = mas_mn(mas); mt = mte_node_type(mas->node); @@ -6648,13 +6682,13 @@ static inline void *mas_first_entry(struct ma_state *mas, struct maple_node *mn, if (likely(entry)) return entry; - pivots = ma_pivots(mn, mt); - mas->index = pivots[0] + 1; mas->offset = 1; entry = mas_slot(mas, slots, 1); + pivots = ma_pivots(mn, mt); if (unlikely(ma_dead_node(mn))) return NULL; + mas->index = pivots[0] + 1; if (mas->index > limit) goto none; From patchwork Fri Jan 27 19:40:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49620 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1011305wrn; Fri, 27 Jan 2023 11:44:42 -0800 (PST) X-Google-Smtp-Source: AK7set/fkHPBlFbHS4r1c0QTX6Yxv1pf+KKy99OVdZLAJ8HN4T+pst70v14HAoYWwRi2LdJDnq8y X-Received: by 2002:a17:906:3fca:b0:87a:dadd:c5aa with SMTP id k10-20020a1709063fca00b0087adaddc5aamr3620814ejj.2.1674848682088; Fri, 27 Jan 2023 11:44:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674848682; cv=none; d=google.com; s=arc-20160816; b=pu6cI4g4QlU7fDYJYTgj+qyX0b8pTirm3GFLQuH+jLMtPjJkHa1Nd/mnivDmPNtb4M 9mDpbZE9dk/MPtz/ZTObiMM/8Rpq69xhLrGopdb0FqNAfo+1i5dMLdF9MaDdSle3ErE/ H4Vur2941OeBrjsOqXZFy7/b9nyG6KcDEbsaT9iYvke+QHTYvREH/KPxa8TwruVr7wCY eRUTLfjQhBYlf/9WMYJRfQYQM8XXlupk0MrLu7bUtaWc2p+GNQl27OyXHLLec6LOnOrx cyaKijtOgdL6LnVDG6Dhwiy4SPdKqSa+bgO9zOWmZfaeIna/BRxDncJi1T1iZ4misCaZ faMg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=6D7AVbhEFxAo5EKdbJ2HQfTq9cKDp9C6gMDUsIDn2rU=; b=t8eHh3Y19cTpbKG+pQU+Is5xOlwf0yuCZSziVRhKa+rF8F6k/jQOBCzTHldcd+wqax gsa+xKPgYo483KTsukPgpaZsa/fxLUanjtD7xP6fIjxYbXHa+CiYhuNQrqdXPoKFz1mH p2jl3Sbyi1KqEXaB52N8FLgwjzS1SxPOFqRzrI5VYq9mr94HdfvyjZ8rEsS5X5JYhBzt SUuuhitpzyiQimbhiOVKGmGpftOjVf/s/hr7zls81XK30/HOIaowaSKHJOJN+zxek2Gs Xfgsuys07Nslt0dMtcMXxCmC5ljDe9eratCqGypPlHen53WN7r+b7cQ1iWbZNOjXNmi2 RY/Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=ij3G6594; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id vf16-20020a170907239000b0087881827ff9si4319430ejb.469.2023.01.27.11.44.18; Fri, 27 Jan 2023 11:44:42 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=ij3G6594; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231688AbjA0Tm5 (ORCPT + 99 others); Fri, 27 Jan 2023 14:42:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50624 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231681AbjA0Tmx (ORCPT ); Fri, 27 Jan 2023 14:42:53 -0500 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A602B7AE44 for ; Fri, 27 Jan 2023 11:42:21 -0800 (PST) Received: by mail-pj1-x104a.google.com with SMTP id om10-20020a17090b3a8a00b002299e350deaso4781924pjb.1 for ; Fri, 27 Jan 2023 11:42:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6D7AVbhEFxAo5EKdbJ2HQfTq9cKDp9C6gMDUsIDn2rU=; b=ij3G6594G9QMVIcaC7y7IotZlrRGSd461+VUEA/BMKS/DqdJRnleVk5IzJ5gil69bZ nXwMrZpHBfopZhY+dBOM8TntT+xgyMzx0lhn0c4XjFlDdq8hUc51rQLAJayIjAlxU0Vs tSDUCQts0+fWuoHVHrLcPeZOnjtBdS6uOzMiFPrkU4AVT++YMj0+eDXitszbX73gjVet Uj7on6Z68KT42I30IFAHZ8MnQyKVo+KH97KXkIJWbVMmYdISzeeP/HtVHdR/3Pzjnpiz r0YfwFvkG3mRoMMu6hvaQ0O0QqVpO+Mcl7G8itnKrK/kLoHBd5Ey6CcVhBTBp6nBJTd/ 24fw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6D7AVbhEFxAo5EKdbJ2HQfTq9cKDp9C6gMDUsIDn2rU=; b=5nPrbRm9BDFfVpwkFKr5cFjhiDQgUq9n3D4YOhCJTh3mLLYu5Q8UFU4n9C/hQa0Fcv R99XWI96OyBysoEVRtRug/NvqAtSS2YT17qcKIu6Zax+mgQkNIzRXnufNyA7SZ6+EkKj 6CUb0Rfnnsr8mLX1yuw0vJ2evI5Bs/W/uP7qy2WPKIUS2HnQ1DbVkg8dFh+tB+vSOAkA E6b5/o162LD/ITIgYxTjp3UUKUh9hhSM0keF1mx265aRQk21g+O3x8FXTUKE/mfylNHL a8b25BLSXC2cWULM3x6Ztdr+m0KjG+9arJ4HUnoLCuTNbSWL4TC6pdNywR9jp6y9a9Az AX4A== X-Gm-Message-State: AO0yUKXgDJyHl8MsqKy2+pJFUILsmFRX0zIjGI6aGZkx7OvXYcokLyWz siwMAHxBRtY0wzm/ZkPAyJLSU5wopFI= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a05:6a00:1990:b0:593:909f:ed45 with SMTP id d16-20020a056a00199000b00593909fed45mr72889pfl.0.1674848478930; Fri, 27 Jan 2023 11:41:18 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:39 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-3-surenb@google.com> Subject: [PATCH v2 02/33] maple_tree: Detect dead nodes in mas_start() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, Liam Howlett X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756206131924360638?= X-GMAIL-MSGID: =?utf-8?q?1756206131924360638?= From: Liam Howlett When initially starting a search, the root node may already be in the process of being replaced in RCU mode. Detect and restart the walk if this is the case. This is necessary for RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index e5eeecd14eee..482e17a460cb 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -1353,12 +1353,16 @@ static inline struct maple_enode *mas_start(struct ma_state *mas) mas->max = ULONG_MAX; mas->depth = 0; +retry: root = mas_root(mas); /* Tree with nodes */ if (likely(xa_is_node(root))) { mas->depth = 1; mas->node = mte_safe_root(root); mas->offset = 0; + if (mte_dead_node(mas->node)) + goto retry; + return NULL; } From patchwork Fri Jan 27 19:40:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49655 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1016080wrn; Fri, 27 Jan 2023 11:57:49 -0800 (PST) X-Google-Smtp-Source: AK7set8I/TPI291fmenRkD3cWvAJozeTYuCdZ+j7mntpG8Mx2d/pDjPhXP4+M//qG2O926+wfv4d X-Received: by 2002:a17:906:7851:b0:878:4edf:4e06 with SMTP id p17-20020a170906785100b008784edf4e06mr9510820ejm.62.1674849469494; Fri, 27 Jan 2023 11:57:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674849469; cv=none; d=google.com; s=arc-20160816; b=JXUPVzgE5bzXww/+PZcNGlS5hTGzPt2ZdqjaOufFDDsgNrxzV9e/xmiy+GSQq6if/U VFlx9Qupp2juOCw7mmuHPHYevQxsTEKAZgvWMOquf0ONzvSnfdVf/TKoZo/I6JyqBl6B Z+kbnsOs3LQdGL8B6EyzWwrGIdyd0TEsdh/4dN+g4cWWbGYcaQ120KNxP4MwIe5ps5Yy R2zGhPNlcRBUJmp3r8Ka6WNQvbOxMyJYg24NLcD+0ZL4wvAM9y1UlpYmBWHJw5GNaMGH CLVDXFTOrOGk/agTvxTxqJB0QfrjMbcKbABwHysgKbrSeqGWNPS+1XLSYNjAsKO/RB6w bOJQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=FUzzHzCQcxOurh9YuvudjXpNdZx02NIZtoLw3Jde/PE=; b=AYn7//W+pBwOL3LfRyDGIjJJzS9VECrFZFnQBnFfqCv0r/WPXgzzbsBGLegqaRnpSW 1g7LQShOVbMN9grtl7yziyNVSbI0lR02AKsBeK2JXtPm0tOiqYldiLdfDOF30iC4At3Q 7ab8c044l0AEiU+KftIyUofwwfrOFWqWL3gpkYgrDZt+oJHiCEVhW2sqHdI84byDrodo zJyq5oiutHzZqPXruWFDP9GhCLAAE2xkNXe/Zsye4n1K0cHZ8pecVPYHxKDfr2sk58bf cDueIaTCIQj/HxSfHhOZitiJUVVwGIXHm7yA7mya5zQUUnC4QnDlxEn7MOsIY+QVHuMs QP1g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=S2HR++yb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 14-20020a17090601ce00b00871d5ec9073si5577572ejj.756.2023.01.27.11.57.19; Fri, 27 Jan 2023 11:57:49 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=S2HR++yb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231740AbjA0Twz (ORCPT + 99 others); Fri, 27 Jan 2023 14:52:55 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59326 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230120AbjA0Twk (ORCPT ); Fri, 27 Jan 2023 14:52:40 -0500 Received: from mail-oo1-xc4a.google.com (mail-oo1-xc4a.google.com [IPv6:2607:f8b0:4864:20::c4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A4D0B8CE1C for ; Fri, 27 Jan 2023 11:49:45 -0800 (PST) Received: by mail-oo1-xc4a.google.com with SMTP id z14-20020a4ad58e000000b005137826d392so1384149oos.21 for ; Fri, 27 Jan 2023 11:49:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=FUzzHzCQcxOurh9YuvudjXpNdZx02NIZtoLw3Jde/PE=; b=S2HR++ybc/ns4vYecXBlgBA5lh7rKCR5RBxCn2d6nLoproYmP4Y/TuK8A2ykL1TZCy PQi2lsVKc//h5m3TPGFUZRCScYCxf/iBRsPO4Le2L6kyLee01De7/gBctPL5x+AFYdbM +GjlXUavIluJZluW6t/CA9goOPJfDGVnM+Mgt+aMp9WxpRPbD/Tewkr6wZubL9JQbZ/Q D22ZQZJELsrUxamrbo349CKHJL642zBXbi0BO/uzmOjlGEGSeG1j/mwEO9F4yWD/yh+R Z4EUx0fbMYpLh0Co5d8CYSkewP6ymMJoH7lGt/nPXvwtHvtN4R6vwQVewCoOaVtJA9nQ 6Lhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=FUzzHzCQcxOurh9YuvudjXpNdZx02NIZtoLw3Jde/PE=; b=WMLZrewXyt6zBgf/Eo/qML7TW/0vjQFunJ/BWDDLIktSu8dndD4e9yofSBvNEB2R53 EDX1L75md4yvoK7k2gwPStVu4HCgkC8neOiJyox+K2wAtnsaIHqI3JewMX+VCjh0VyQj geTj2ar6N8b3V+1umQq4Puj8A4rv7T9r+8H7Knmfm9pWOH1DySyn/JBbIN9d78HFXSa1 I+wx2a3ONZ/jDNodiI5ktsUfFZO3QS99Jbz09wxudyUVHHEcKcvELWYz4BMqmC6MQQkO h5JUliIg9NhcYYRwuxatHA2jnzuV6XYFRdNxXKgsxsX97O89v6hJr9E15bkC3NpfiBgJ HGJg== X-Gm-Message-State: AFqh2kouhnDE8/Xee9100wqDuiX5j1pVEqy83dxdIciKbxi7Oy+qNLOA 4NNXVw+S71oezQW3askId9tX2L75ZVA= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:7e4b:0:b0:506:4f19:740c with SMTP id p11-20020a817e4b000000b005064f19740cmr1518489ywn.383.1674848481189; Fri, 27 Jan 2023 11:41:21 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:40 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-4-surenb@google.com> Subject: [PATCH v2 03/33] maple_tree: Fix freeing of nodes in rcu mode From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, Liam Howlett X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756206957690973546?= X-GMAIL-MSGID: =?utf-8?q?1756206957690973546?= From: Liam Howlett The walk to destroy the nodes was not always setting the node type and would result in a destroy method potentially using the values as nodes. Avoid this by setting the correct node types. This is necessary for the RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 73 ++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 62 insertions(+), 11 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 482e17a460cb..73917dd2c608 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -895,6 +895,44 @@ static inline void ma_set_meta(struct maple_node *mn, enum maple_type mt, meta->end = end; } +/* + * mas_clear_meta() - clear the metadata information of a node, if it exists + * @mas: The maple state + * @mn: The maple node + * @mt: The maple node type + * @offset: The offset of the highest sub-gap in this node. + * @end: The end of the data in this node. + */ +static inline void mas_clear_meta(struct ma_state *mas, struct maple_node *mn, + enum maple_type mt) +{ + struct maple_metadata *meta; + unsigned long *pivots; + void __rcu **slots; + void *next; + + switch (mt) { + case maple_range_64: + pivots = mn->mr64.pivot; + if (unlikely(pivots[MAPLE_RANGE64_SLOTS - 2])) { + slots = mn->mr64.slot; + next = mas_slot_locked(mas, slots, + MAPLE_RANGE64_SLOTS - 1); + if (unlikely((mte_to_node(next) && mte_node_type(next)))) + return; /* The last slot is a node, no metadata */ + } + fallthrough; + case maple_arange_64: + meta = ma_meta(mn, mt); + break; + default: + return; + } + + meta->gap = 0; + meta->end = 0; +} + /* * ma_meta_end() - Get the data end of a node from the metadata * @mn: The maple node @@ -5448,20 +5486,22 @@ static inline int mas_rev_alloc(struct ma_state *mas, unsigned long min, * mas_dead_leaves() - Mark all leaves of a node as dead. * @mas: The maple state * @slots: Pointer to the slot array + * @type: The maple node type * * Must hold the write lock. * * Return: The number of leaves marked as dead. */ static inline -unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots) +unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots, + enum maple_type mt) { struct maple_node *node; enum maple_type type; void *entry; int offset; - for (offset = 0; offset < mt_slot_count(mas->node); offset++) { + for (offset = 0; offset < mt_slots[mt]; offset++) { entry = mas_slot_locked(mas, slots, offset); type = mte_node_type(entry); node = mte_to_node(entry); @@ -5480,14 +5520,13 @@ unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots) static void __rcu **mas_dead_walk(struct ma_state *mas, unsigned char offset) { - struct maple_node *node, *next; + struct maple_node *next; void __rcu **slots = NULL; next = mas_mn(mas); do { - mas->node = ma_enode_ptr(next); - node = mas_mn(mas); - slots = ma_slots(node, node->type); + mas->node = mt_mk_node(next, next->type); + slots = ma_slots(next, next->type); next = mas_slot_locked(mas, slots, offset); offset = 0; } while (!ma_is_leaf(next->type)); @@ -5551,11 +5590,14 @@ static inline void __rcu **mas_destroy_descend(struct ma_state *mas, node = mas_mn(mas); slots = ma_slots(node, mte_node_type(mas->node)); next = mas_slot_locked(mas, slots, 0); - if ((mte_dead_node(next))) + if ((mte_dead_node(next))) { + mte_to_node(next)->type = mte_node_type(next); next = mas_slot_locked(mas, slots, 1); + } mte_set_node_dead(mas->node); node->type = mte_node_type(mas->node); + mas_clear_meta(mas, node, node->type); node->piv_parent = prev; node->parent_slot = offset; offset = 0; @@ -5575,13 +5617,18 @@ static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, MA_STATE(mas, &mt, 0, 0); - if (mte_is_leaf(enode)) + mas.node = enode; + if (mte_is_leaf(enode)) { + node->type = mte_node_type(enode); goto free_leaf; + } + ma_flags &= ~MT_FLAGS_LOCK_MASK; mt_init_flags(&mt, ma_flags); mas_lock(&mas); - mas.node = start = enode; + mte_to_node(enode)->ma_flags = ma_flags; + start = enode; slots = mas_destroy_descend(&mas, start, 0); node = mas_mn(&mas); do { @@ -5589,7 +5636,8 @@ static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, unsigned char offset; struct maple_enode *parent, *tmp; - node->slot_len = mas_dead_leaves(&mas, slots); + node->type = mte_node_type(mas.node); + node->slot_len = mas_dead_leaves(&mas, slots, node->type); if (free) mt_free_bulk(node->slot_len, slots); offset = node->parent_slot + 1; @@ -5613,7 +5661,8 @@ static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, } while (start != mas.node); node = mas_mn(&mas); - node->slot_len = mas_dead_leaves(&mas, slots); + node->type = mte_node_type(mas.node); + node->slot_len = mas_dead_leaves(&mas, slots, node->type); if (free) mt_free_bulk(node->slot_len, slots); @@ -5623,6 +5672,8 @@ static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, free_leaf: if (free) mt_free_rcu(&node->rcu); + else + mas_clear_meta(&mas, node, node->type); } /* From patchwork Fri Jan 27 19:40:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49623 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1011435wrn; Fri, 27 Jan 2023 11:45:02 -0800 (PST) X-Google-Smtp-Source: AMrXdXslc/qdK04QO6+bmihfXEMuzwm4JKQjfJTqhZNM95UBM7SYCIv5LNoW/IFQQa1mUsnNMNIC X-Received: by 2002:a05:6402:1cb9:b0:49e:ef1f:712e with SMTP id cz25-20020a0564021cb900b0049eef1f712emr28612091edb.18.1674848701799; Fri, 27 Jan 2023 11:45:01 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674848701; cv=none; d=google.com; s=arc-20160816; b=t0AYHuq/1WvMxkTU1b1M30VYzWtSUWJelF2W+nJhD5AbWjMfrzCenplAy+xPPHnKjL TEe/GrX8UfFtG2phkqv33AMPDndtvIjFO6CY9IDrngzi8TierCWB4ZlMVo3Ft4g3/nZh SQ+ch/NOfLKRhXsqu2o+fsdGl3hPsG0vqqJ0GaEHrKCSxebfaY3NSMxxOUdd9HyIOrTL ktDf9L0gUNjMJ8SvM+mVGUHity+cGneZYEACV/QIHfgJ+vfoYT2DKSOZDOor4ufEii32 gm73Xek/8aC3x46cd8ZXpspCfF5aFMv7Jx28OjVIEUEH80x9neiU2NYzyKSlNIicME8N H5PQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=DF9zqRO1prEDm4IKfYXh+Z69pFj8RdYSRvmvJpFQZKM=; b=tYrUUOBU6logzKWccbIPaxWH3IBFDjTNtxY7omJ6xWqOXrNXLrThACPyO8k1eVUzze SkMtrAD2zjy0eRQ4HbGzIoGl13DcO+ozx6Jo4GdHQ4ZjOu4rrwOGNn7mspGHFbzSDb8a p/vazIRHK082piTDBsnYBkV8/vg19e4Q2R+juRLYZrp6pCLypnzkg8j6jazDuT5apDVQ cbKgCUkvKPGXNp0PhUs7hEw7sSnGXkJcxyrptOXxYplgib8sLKiUqkACLRbPPh2dJuZA Z5VtinA8W2Ti42X8aciV5tr0sPfxQsu4qF4KRlULRC3Pr6j1sLwpnWayuWB+jlphPIhP Iq5g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=rnpUdmPc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u1-20020a05640207c100b004a212080564si1828865edy.339.2023.01.27.11.44.37; Fri, 27 Jan 2023 11:45:01 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=rnpUdmPc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231991AbjA0TnK (ORCPT + 99 others); Fri, 27 Jan 2023 14:43:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50662 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231848AbjA0Tm5 (ORCPT ); Fri, 27 Jan 2023 14:42:57 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C966E82421 for ; Fri, 27 Jan 2023 11:42:29 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id p19-20020a25d813000000b0080b78270db5so6347404ybg.15 for ; Fri, 27 Jan 2023 11:42:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=DF9zqRO1prEDm4IKfYXh+Z69pFj8RdYSRvmvJpFQZKM=; b=rnpUdmPcBnY7plbcdmS2pwvwnctQIRIqHumfjs2guCuPsY+Dw6YQ2tfutal8HnrgXz edy5Yrgey9QhoRSbyw/SmRVhoDEfIcnSP0t0LCmLUom2gJSxBSb6RoRoQ6R6E1A4+8T0 XcTb3Fhtcz5YcundAJZb+WYoiQCLWWkiPl+/L7XZyu1tWh8b/ENfJXXDc0COCB4H0HVg yuAJnUBiF1z/T9ta5KOn1AHf3cQZUH/oUuoUg6oOo5mcl+MifhrO/XwhHeojlZjv/4Mm lccC2wtCBVxTksQLP5lorO6u35MCkxakWEmMvAgnG98JEeGI6fsdBcznmI1LtXlDE24U yV4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=DF9zqRO1prEDm4IKfYXh+Z69pFj8RdYSRvmvJpFQZKM=; b=DvFbCFhTP7LJ35OJ5A8wGJSN2PMiyM1gwykU48S9kgEdyQwwVHaQEs5AR6KspwLBF4 AGd5KYEGutxqqY1yB3aUVyL8acEgDBv9IXVjpgek4KI5vSXhpY5GSJUD9hTUGeWN32n9 mXtNH1qMPBye6iitsb56bnWmYAjEmuTLV1xznPoQvlZTmiTvqUrocKtc4Jd8htNlb/2A L//3LCrnwKyb7t+dY66QrN7Zxvfi3AvwAQTGXQsN1HG+bgd5WhLfNF5kzgzHnLCormvy wA8e8qml+MAjDJBltYxP1SFiRSvlxnQl8rlwNXa9diEYM4uZQ7Y7uX27oUAjF71ik+IN 8ZcQ== X-Gm-Message-State: AFqh2koyKgevasGTPFVBILuIewlaB/USuAmoldq80VvJoAVgxFg0yPoD DYLhxVdWMuBFMpIwK+xa3t+uOuj+SEo= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a0d:c8c5:0:b0:4f3:10b1:1351 with SMTP id k188-20020a0dc8c5000000b004f310b11351mr5009568ywd.516.1674848483749; Fri, 27 Jan 2023 11:41:23 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:41 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-5-surenb@google.com> Subject: [PATCH v2 04/33] maple_tree: remove extra smp_wmb() from mas_dead_leaves() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, Liam Howlett X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756206152366174401?= X-GMAIL-MSGID: =?utf-8?q?1756206152366174401?= From: Liam Howlett The call to mte_set_dead_node() before the smp_wmb() already calls smp_wmb() so this is not needed. This is an optimization for the RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 1 - 1 file changed, 1 deletion(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 73917dd2c608..75cce2d4d5da 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -5510,7 +5510,6 @@ unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots, break; mte_set_node_dead(entry); - smp_wmb(); /* Needed for RCU */ node->type = type; rcu_assign_pointer(slots[offset], node); } From patchwork Fri Jan 27 19:40:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49622 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1011397wrn; Fri, 27 Jan 2023 11:44:55 -0800 (PST) X-Google-Smtp-Source: AMrXdXuQgUmPRGMHbyl15sBbrOzy7zIYP/ZqI5dqs3Bqi/qEfTEQLtbuWSPPW+FjeZX0XeI3HGXm X-Received: by 2002:a17:907:1248:b0:84d:45db:b203 with SMTP id wc8-20020a170907124800b0084d45dbb203mr35088082ejb.12.1674848695283; Fri, 27 Jan 2023 11:44:55 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674848695; cv=none; d=google.com; s=arc-20160816; b=a5lE/ekjTPRxc+RSiLGCqarJfu2dr9WMxq9AzdqRdmXuIgZzxBMTewry/5TvLMGVHa afZAC4nZb+nAZFvVU9JtsMB4QKGzyMcbbuCkEloXSaVY22bvCMKf4fBn8h9/k3RsQUdr QhIjGOo5Sw2WhdkKhvsN+DReEmMp1dNEQ4a5wVNG6WDSN+1fuo6GhqVFKZgCPzJq8t50 qX+lrV0ranXTrS9riHMfDnQsbKy8BrDQuDdYLH9RJcSVDyTnbsDH5k7wlcKdZiBOfnYz ys11M2SOHiBJomX5wgcN4TcyNxABSAsMMwRL8ma5cHppNv04+Y4K4ACzqZ7HpTXrx6Xq R2SQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=fluAUEldqgo6Mr/4Su/gpwJVTd/f+QKHo7F0iCGeqOg=; b=XSMGLyDwa2mPEBpBTiS1P98UffvEaF8CbmZ+Ik8+vwE3bXX+s1WPZcp+HkYPpXMNJm hlxJGCX22MPuXoRI7T48cT9OVCRcwTsBqs9ftVhRPfAlK5+pSYzvUKz8ZMrEPPTf8K1o CFeAINoQvnK+5tjxk/m2134zhWJIBKiNKeBk6hwe/iu6z2HzlLTk5wQkGTepkoNB/Ktd BjJtMzuJVhc6Uv2U6olobXnClRoF+auR1pP4SAEwk+sRA/NUC+BYPi+vqdYFkuxRWZuQ o/1cfhtPMGytZFJC9myeVH1Kxp2XBc+aRNtWDBdmYnQWLntTZevPwBd3qz4jMKRrSSYW ZFUA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=B0QFA0zT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id gh4-20020a1709073c0400b008725a1034d3si4953735ejc.85.2023.01.27.11.44.32; Fri, 27 Jan 2023 11:44:55 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=B0QFA0zT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231972AbjA0TnI (ORCPT + 99 others); Fri, 27 Jan 2023 14:43:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50624 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231859AbjA0Tm5 (ORCPT ); Fri, 27 Jan 2023 14:42:57 -0500 Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 497B68242F for ; Fri, 27 Jan 2023 11:42:30 -0800 (PST) Received: by mail-pf1-x44a.google.com with SMTP id cr14-20020a056a000f0e00b0058da951c487so2786838pfb.0 for ; Fri, 27 Jan 2023 11:42:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=fluAUEldqgo6Mr/4Su/gpwJVTd/f+QKHo7F0iCGeqOg=; b=B0QFA0zTTAgJFhNgrdDgKRfLtInL5VmAHE1KFwNYXYIlcWvaJJq1PKzYA8ckdy7Kyn xAwTf5YPOblK6+TJj1tGvdojweaIBaHFiTxRDHhkDnK6qDnyvgtYbq5iiXWxBxOCwUuh WaET5h8CLSuaMj6adU6fJWH6EpE5SqyIKnzFJ/pir1JZwDIco5M5jQ2a/9VWhgwD3iWL QgWDwglZ0wa8AC1rJh0bk86XOm5Sb2wt2duuTU/yqZcdsgUl6ZFLCYWIeCgAmHx4ObRd hHIbD6pxz2ke/VukXpDhoYuOo0GEGrcZQn8DqZwlpC0FiE7wMl7MPAjekSdb7LenTMW0 255w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=fluAUEldqgo6Mr/4Su/gpwJVTd/f+QKHo7F0iCGeqOg=; b=zS84MyA4LiNIUaKu8tV8iiqiew1AwtBjNR252+XwRk/medbzfToEY5qFEQ+zBpSsJj VAi2TtzuhCBPjIui0ouzulTOHIDGtJeeX3ah+kpczFIBLuYLvbq+bcajcBiKW8+Qn4fw Bdl8dok5pWKoRaJICgbTF8uwKj3wELoV0gnx7zoM3K7B9lzTqsK2mTlUumSL31iMxYWj ckOtsW2wpqzHU6ee0rEGprHhJpjc3suqfz22jNb+7F14s7jBeLJ1sW4zxIMFotfHluPR KZhibpltSKLFSBynLXMYlQ7Jsun9rQfZwEuDrP1NIFUc+/SH7hxcuhEOBSHmQrAv8U5v l5aw== X-Gm-Message-State: AO0yUKXtDRsuF/qLUIR2cHZLt3zCNlQALXhEus6lHnEsvexDxod/az77 CYR2Pkg1ef53Db3jlwySejXsmbQ3S9Q= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a17:90b:a16:b0:225:eaa2:3f5d with SMTP id gg22-20020a17090b0a1600b00225eaa23f5dmr12984pjb.2.1674848485998; Fri, 27 Jan 2023 11:41:25 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:42 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-6-surenb@google.com> Subject: [PATCH v2 05/33] maple_tree: Fix write memory barrier of nodes once dead for RCU mode From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, "Liam R. Howlett" X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756206145385321861?= X-GMAIL-MSGID: =?utf-8?q?1756206145385321861?= From: "Liam R. Howlett" During the development of the maple tree, the strategy of freeing multiple nodes changed and, in the process, the pivots were reused to store pointers to dead nodes. To ensure the readers see accurate pivots, the writers need to mark the nodes as dead and call smp_wmb() to ensure any readers can identify the node as dead before using the pivot values. There were two places where the old method of marking the node as dead without smp_wmb() were being used, which resulted in RCU readers seeing the wrong pivot value before seeing the node was dead. Fix this race condition by using mte_set_node_dead() which has the smp_wmb() call to ensure the race is closed. Add a WARN_ON() to the ma_free_rcu() call to ensure all nodes being freed are marked as dead to ensure there are no other call paths besides the two updated paths. This is necessary for the RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 7 +++++-- tools/testing/radix-tree/maple.c | 16 ++++++++++++++++ 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 75cce2d4d5da..49e399e8afaa 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -178,7 +178,7 @@ static void mt_free_rcu(struct rcu_head *head) */ static void ma_free_rcu(struct maple_node *node) { - node->parent = ma_parent_ptr(node); + WARN_ON(node->parent != ma_parent_ptr(node)); call_rcu(&node->rcu, mt_free_rcu); } @@ -1771,8 +1771,10 @@ static inline void mas_replace(struct ma_state *mas, bool advanced) rcu_assign_pointer(slots[offset], mas->node); } - if (!advanced) + if (!advanced) { + mte_set_node_dead(old_enode); mas_free(mas, old_enode); + } } /* @@ -4211,6 +4213,7 @@ static inline bool mas_wr_node_store(struct ma_wr_state *wr_mas) done: mas_leaf_set_meta(mas, newnode, dst_pivots, maple_leaf_64, new_end); if (in_rcu) { + mte_set_node_dead(mas->node); mas->node = mt_mk_node(newnode, wr_mas->type); mas_replace(mas, false); } else { diff --git a/tools/testing/radix-tree/maple.c b/tools/testing/radix-tree/maple.c index 958ee9bdb316..4c89ff333f6f 100644 --- a/tools/testing/radix-tree/maple.c +++ b/tools/testing/radix-tree/maple.c @@ -108,6 +108,7 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, mn->slot[1] != NULL); MT_BUG_ON(mt, mas_allocated(&mas) != 0); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mas.node = MAS_START; mas_nomem(&mas, GFP_KERNEL); @@ -160,6 +161,7 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, mas_allocated(&mas) != i); MT_BUG_ON(mt, !mn); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } @@ -192,6 +194,7 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, not_empty(mn)); MT_BUG_ON(mt, mas_allocated(&mas) != i - 1); MT_BUG_ON(mt, !mn); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } @@ -210,6 +213,7 @@ static noinline void check_new_node(struct maple_tree *mt) mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); MT_BUG_ON(mt, mas_allocated(&mas) != j - 1); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } MT_BUG_ON(mt, mas_allocated(&mas) != 0); @@ -233,6 +237,7 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, mas_allocated(&mas) != i - j); mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); MT_BUG_ON(mt, mas_allocated(&mas) != i - j - 1); } @@ -269,6 +274,7 @@ static noinline void check_new_node(struct maple_tree *mt) mn = mas_pop_node(&mas); /* get the next node. */ MT_BUG_ON(mt, mn == NULL); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } MT_BUG_ON(mt, mas_allocated(&mas) != 0); @@ -294,6 +300,7 @@ static noinline void check_new_node(struct maple_tree *mt) mn = mas_pop_node(&mas2); /* get the next node. */ MT_BUG_ON(mt, mn == NULL); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } MT_BUG_ON(mt, mas_allocated(&mas2) != 0); @@ -334,10 +341,12 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 2); mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); for (i = 1; i <= MAPLE_ALLOC_SLOTS + 1; i++) { mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } MT_BUG_ON(mt, mas_allocated(&mas) != 0); @@ -375,6 +384,7 @@ static noinline void check_new_node(struct maple_tree *mt) mas_node_count(&mas, i); /* Request */ mas_nomem(&mas, GFP_KERNEL); /* Fill request */ mn = mas_pop_node(&mas); /* get the next node. */ + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mas_destroy(&mas); @@ -382,10 +392,13 @@ static noinline void check_new_node(struct maple_tree *mt) mas_node_count(&mas, i); /* Request */ mas_nomem(&mas, GFP_KERNEL); /* Fill request */ mn = mas_pop_node(&mas); /* get the next node. */ + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mn = mas_pop_node(&mas); /* get the next node. */ + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mn = mas_pop_node(&mas); /* get the next node. */ + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mas_destroy(&mas); } @@ -35369,6 +35382,7 @@ static noinline void check_prealloc(struct maple_tree *mt) MT_BUG_ON(mt, allocated != 1 + height * 3); mn = mas_pop_node(&mas); MT_BUG_ON(mt, mas_allocated(&mas) != allocated - 1); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); MT_BUG_ON(mt, mas_preallocate(&mas, GFP_KERNEL) != 0); mas_destroy(&mas); @@ -35386,6 +35400,7 @@ static noinline void check_prealloc(struct maple_tree *mt) mas_destroy(&mas); allocated = mas_allocated(&mas); MT_BUG_ON(mt, allocated != 0); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); MT_BUG_ON(mt, mas_preallocate(&mas, GFP_KERNEL) != 0); @@ -35756,6 +35771,7 @@ void farmer_tests(void) tree.ma_root = mt_mk_node(node, maple_leaf_64); mt_dump(&tree); + node->parent = ma_parent_ptr(node); ma_free_rcu(node); /* Check things that will make lockdep angry */ From patchwork Fri Jan 27 19:40:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49624 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1011437wrn; Fri, 27 Jan 2023 11:45:02 -0800 (PST) X-Google-Smtp-Source: AMrXdXtf4H3CZ0s7JylKjTLKU04RK6FcBkEXpm3+knpLYGh+5YjmzAf5lBTcUGNkIiwDm+wGqjTh X-Received: by 2002:a17:906:d0c8:b0:84d:855:dd02 with SMTP id bq8-20020a170906d0c800b0084d0855dd02mr44516508ejb.46.1674848701957; Fri, 27 Jan 2023 11:45:01 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674848701; cv=none; d=google.com; s=arc-20160816; b=SKNj2F/eDaCzgJoy9Fnq/t6ez0XFiBbQ0A8AtlybXyvrrsEiALFPRFJtFtUU7YjqKF qEvMCkQynpPGRjoz7kJUM3FHY6tUA46eydekRfklElTmmyqIeC7JHjiFxeFfS/mRvV7q TXnGpAPMeGqNKL3qhcGFilg7kQalSY60mIDhVBlfVsOjnqHeFoEQIbM/4oAC549yagDY GcU0u6sa42zmdR4jW28mmIqBzZcHnkvcar8hkyLf3+tG5H7vL7X+MKTq4yP7cwb8YkQh yIoR53DAmhTSGcuWMlpyAM4AULda/1eFFcVE6HAJi2nRfAGSaUqK3NIB3C0cuBIlfPu0 O0+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=A97N+ZL4sr7Moa7VA3MaIfeszXjKUidgl5sbUl4VE3M=; b=vVBW9irJ56AyQyLBwg6CH/EtApgWI3qEqih27i0tftfKZ0pe/0lq6aHwitkEUFRtFC VUt4pYHyXuxPDmWZySLw1q3AidBhfQlpVkzEyVGlQj8SRp064tgSK6MvCwMK7og7wuV5 YuT0p7mrEPoFomveUJikZOqIygwa1WICoX5D6aJmPj8gmW2Qij1SwsCVzv2NtGazHcEG tCuD3jKDj9Dxc5jJK8xJKis+c1G68lPzI4Edy1rA+tykNCtWsVAwBkrh6bRdQZEymlqq 0CNea03pzUtDIa5WZ4dWVOsJDCQqgtskGDo8WVy+Vme0V8QelgdT8rULaueOsmO74pHQ pIog== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=h4z95aWq; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ka5-20020a170907920500b0087cc469b193si1883658ejb.836.2023.01.27.11.44.39; Fri, 27 Jan 2023 11:45:01 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=h4z95aWq; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232095AbjA0TnT (ORCPT + 99 others); Fri, 27 Jan 2023 14:43:19 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50644 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231783AbjA0Tm7 (ORCPT ); Fri, 27 Jan 2023 14:42:59 -0500 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0A3E584B67 for ; Fri, 27 Jan 2023 11:42:35 -0800 (PST) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-507aac99fdfso65529237b3.11 for ; Fri, 27 Jan 2023 11:42:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=A97N+ZL4sr7Moa7VA3MaIfeszXjKUidgl5sbUl4VE3M=; b=h4z95aWqlTijLuw2+pa/O521iwctvYJ/JE9Gk94Om8JNdm1J4qRBJ1EfNZDOaurfUb h0xAr5ZA79m4jhsd5KmptV8L5YTVXkJEqIIo06fTcAgDWyf2Iw1unXEy8R6f3F+h7n0/ eKAno27phPvhqEsCndY+cAQm1TD4BuL7gONpQNAXYFn8Kf2q+ZhvCxmCxJsT4hK4blKF lDmG8J5RutFJ9mwfq4DqWgfTIN8+TeptwgRlAF+qB5k0K+pf2NR8dkIUNhEjxCEKvwc/ GAm3buayP+cfA/HX9e16HQfg7j+WzzAIFSLNgZD3Zm/44+VjSwN/Bplr8AdUjUNKklkq c5Vw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=A97N+ZL4sr7Moa7VA3MaIfeszXjKUidgl5sbUl4VE3M=; b=nMiyFHMJqR8b2ffuEaRwQtF07DKs2Hwn3ZXPEfWZicJGrNC4annzsol19JRM0dBeVW PWAaYNKJCBshwA/YVevYrhwU1mTZBK+Wa/L8ULBJL/I7rXR/XsAWhJdoMi1GdmaBlhsp KutnA6+FtBtxLIXzYs74wUUud94H7A732GOsYTWWRa8Dg/8/IWP2JUxhHqw9SHVfB+gE hVOX0K0aKdNA3lPmgZVqjMSBeQnADpk3FN+rRtMNhfN600qGD/Uwf9TEKMwve909GBW5 C8Qs1mDRRNfTydx5BMh/6Vsu31TooEGJa4+oDunenpSclGnFDl47NUfCd3THcv0icJ9I 1oUw== X-Gm-Message-State: AO0yUKWbdOKiCcfIAT5+/rB0xRlsSBQAXJ83uy5xYMz8sRjqSRkrOxpy +qXaoeCVmYvz57CnHzquaIIfijgV3uQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:9bd6:0:b0:509:5557:c194 with SMTP id s205-20020a819bd6000000b005095557c194mr892323ywg.449.1674848488954; Fri, 27 Jan 2023 11:41:28 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:43 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-7-surenb@google.com> Subject: [PATCH v2 06/33] maple_tree: Add smp_rmb() to dead node detection From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, "Liam R. Howlett" X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756206152383916414?= X-GMAIL-MSGID: =?utf-8?q?1756206152383916414?= From: "Liam R. Howlett" Add an smp_rmb() before reading the parent pointer to ensure that anything read from the node prior to the parent pointer hasn't been reordered ahead of this check. The is necessary for RCU mode. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 49e399e8afaa..859303d2da90 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -532,9 +532,11 @@ static inline struct maple_node *mte_parent(const struct maple_enode *enode) */ static inline bool ma_dead_node(const struct maple_node *node) { - struct maple_node *parent = (void *)((unsigned long) - node->parent & ~MAPLE_NODE_MASK); + struct maple_node *parent; + /* Do not reorder reads from the node prior to the parent check */ + smp_rmb(); + parent = (void *)((unsigned long) node->parent & ~MAPLE_NODE_MASK); return (parent == node); } @@ -549,6 +551,8 @@ static inline bool mte_dead_node(const struct maple_enode *enode) struct maple_node *parent, *node; node = mte_to_node(enode); + /* Do not reorder reads from the node prior to the parent check */ + smp_rmb(); parent = mte_parent(enode); return (parent == node); } From patchwork Fri Jan 27 19:40:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49625 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1011461wrn; Fri, 27 Jan 2023 11:45:04 -0800 (PST) X-Google-Smtp-Source: AMrXdXssT9mQGKOnEMy0GBBHvNxaW2oogkBY7Bd5XaazdlJMPTwwOdn54d4SNCBZigtQzLdsLHS9 X-Received: by 2002:a05:6402:f20:b0:468:58d4:a0f2 with SMTP id i32-20020a0564020f2000b0046858d4a0f2mr50913197eda.23.1674848704284; Fri, 27 Jan 2023 11:45:04 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674848704; cv=none; d=google.com; s=arc-20160816; b=yqe/EzULo39aEADtFYEWoZ7M/K2hpLLJSeeJ/j+zwNQkxl2oAGkjB8NdbjlzEy0C+J R4IhAJvGzeMrg46IUujo48WySIfHUHwea6l4tj0hMCVlImu3TkI0TNofBfmzciJplb9w a8l3R9qd4EfOZT6YA+pPG1QdpA8YRdV4zYsQnkF/ZvKM6AbC5PX74gKCUYPGbDNs/C6E Hf9Lj1fY6IvcRNibLTGDPIgX5T68g57Di/KOMqXfmoK6vL0kfIv4RWXnTiNCI0OFtK+j QqF3PsyHwc9SLmhKSC87ijzpbsR9/IIe1wly7jqaNCtuVS7V3Tbf8GwCjKOW1sM8RrFB Q2Mg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=mDt1g22Fa9gYu2Kb7/F2UNWz7p2znEfhohgEMazkZAE=; b=mfeRN2Ce7EZgm7yPRmnFlQFGkUNOenxmSyFyawtmeOtUUNyyM5FmR51e1gkPD6JQig xR1ckYza9qxMy3f0MDBq+61139p/YVhWwKxzSGtjnNYxRL1mNfjtDz+E9Ax2CsrgwRwY RFJKvfge+Tm7vVjlqD9emwboE2HBMfdWZjxebjTSNERTA6FTZC/TbImNdfRE9qYUtKSg VsYHx2GnO9b7KMHtYaOxDUs9vQFjfF+i9BoCkXeQwqdpVKXmQjrjMU4cjSi30MfiwLX4 TFc3cSEUFe2JzY+ylGzrSYr8nu8CwoBG4NKGu39mMZse9jqlkHpMAytLkW9h/fLRarIM rDzA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=n77zu0oR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b20-20020aa7d494000000b0049dd955ded4si5879390edr.524.2023.01.27.11.44.40; Fri, 27 Jan 2023 11:45:04 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=n77zu0oR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231848AbjA0TnY (ORCPT + 99 others); Fri, 27 Jan 2023 14:43:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50752 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231809AbjA0Tm7 (ORCPT ); Fri, 27 Jan 2023 14:42:59 -0500 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 095CB83244 for ; Fri, 27 Jan 2023 11:42:35 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-50660dd3263so65567817b3.19 for ; Fri, 27 Jan 2023 11:42:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=mDt1g22Fa9gYu2Kb7/F2UNWz7p2znEfhohgEMazkZAE=; b=n77zu0oRsJL7OvjwFNjS7T1FAOtu4hdvxpK7kq3XthnUipBv5szcKq0gW6Eh2qWZ+N VqA54npBtuI1r6BsRB8yCAVHYO39SuVCqNfhqCm/Ya4OMopwGqTAxzjzqXFTaLEmWefn G7d9A2dlsE9NEs9J39+IVmn9jXa2H7iuxd7eUv1r4SdiVN/BsV9E4rX7Fx/7SJnqIKhp 5pljM+LwjnmUAESpDExSAXXjwcvSQjwdpBA+epV9jtEBYonrLt6ilkcmvKbIgLrFIqhL dWwHvu2PRt1rquBCjgEA2IpAdLCPilrM/O5s952uzrtLUUUIX7VUyMYpZs8xK3jyFgDu 2ztw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=mDt1g22Fa9gYu2Kb7/F2UNWz7p2znEfhohgEMazkZAE=; b=3u7GOPP1AKy2zKIq6G7zqCcugp/6Hn3KcNWtsLFQoVHTnfbjqokSGNchF4AuTEJmyE VwbvUkxFuKw8hz57F2R3zoCjRefE+oB240mj7JfAztahEOOL3I3Di51VkTGtgb8OGS+2 58zODxxVpscJ8vj2UbCh93ts+8MjxVcuMRkgE8bsw7AGY1v912HYt44B+vLZcfguvZ69 bytaGPTb58DfVRkx5vIJiCjLs33cL743GT7nukr6Qlj4Lq30RGZDUUO4IfLczLEa+OHR ArWby+d7mFEJ0T1tzm8DKLTNLkpzvYHnD/DhBStRHO8acg2asDn/gBQQSaN7rMAdfMUv Zaxw== X-Gm-Message-State: AO0yUKVZtrPJ0JjIIwuhszcyTk8HzwA3woOP5bu1lJBcRtQ4Tx8H0td0 mtdVE6oCz1pdhL/y4fp7DyKqtX6tnYw= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a25:354:0:b0:80b:543d:2106 with SMTP id 81-20020a250354000000b0080b543d2106mr1234365ybd.10.1674848491227; Fri, 27 Jan 2023 11:41:31 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:44 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-8-surenb@google.com> Subject: [PATCH v2 07/33] mm: Enable maple tree RCU mode by default. From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, "Liam R. Howlett" X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756206154874565602?= X-GMAIL-MSGID: =?utf-8?q?1756206154874565602?= From: "Liam R. Howlett" Use the maple tree in RCU mode for VMA tracking. This is necessary for the use of per-VMA locking. RCU mode is enabled by default but disabled when exiting an mm and for the new tree during a fork. Also enable RCU for the tree used in munmap operations to ensure the nodes remain valid for readers. Signed-off-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- include/linux/mm_types.h | 3 ++- kernel/fork.c | 3 +++ mm/mmap.c | 4 +++- 3 files changed, 8 insertions(+), 2 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index da983aedb741..8410c3052148 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -768,7 +768,8 @@ struct mm_struct { unsigned long cpu_bitmap[]; }; -#define MM_MT_FLAGS (MT_FLAGS_ALLOC_RANGE | MT_FLAGS_LOCK_EXTERN) +#define MM_MT_FLAGS (MT_FLAGS_ALLOC_RANGE | MT_FLAGS_LOCK_EXTERN | \ + MT_FLAGS_USE_RCU) extern struct mm_struct init_mm; /* Pointer magic because the dynamic array size confuses some compilers. */ diff --git a/kernel/fork.c b/kernel/fork.c index 5e3029ea8e1e..5f23d5e03362 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -617,6 +617,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, if (retval) goto out; + mt_clear_in_rcu(vmi.mas.tree); for_each_vma(old_vmi, mpnt) { struct file *file; @@ -700,6 +701,8 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, retval = arch_dup_mmap(oldmm, mm); loop_out: vma_iter_free(&vmi); + if (!retval) + mt_set_in_rcu(vmi.mas.tree); out: mmap_write_unlock(mm); flush_tlb_mm(oldmm); diff --git a/mm/mmap.c b/mm/mmap.c index 9d0dadb55e9a..9efe13d36df7 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2277,7 +2277,8 @@ do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, int count = 0; int error = -ENOMEM; MA_STATE(mas_detach, &mt_detach, 0, 0); - mt_init_flags(&mt_detach, MT_FLAGS_LOCK_EXTERN); + mt_init_flags(&mt_detach, vmi->mas.tree->ma_flags & + (MT_FLAGS_LOCK_MASK | MT_FLAGS_USE_RCU)); mt_set_external_lock(&mt_detach, &mm->mmap_lock); /* @@ -3042,6 +3043,7 @@ void exit_mmap(struct mm_struct *mm) */ set_bit(MMF_OOM_SKIP, &mm->flags); mmap_write_lock(mm); + mt_clear_in_rcu(&mm->mm_mt); free_pgtables(&tlb, &mm->mm_mt, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING); tlb_finish_mmu(&tlb); From patchwork Fri Jan 27 19:40:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49633 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1011744wrn; Fri, 27 Jan 2023 11:45:52 -0800 (PST) X-Google-Smtp-Source: AMrXdXuogoOnb9MP5J6oI4YE+tj/vjHCPIIS+f6Uxpvbb5Z/CShu4KECN3VVoJMEQBkCJTSLZSe4 X-Received: by 2002:aa7:cad3:0:b0:49e:eb11:e40 with SMTP id l19-20020aa7cad3000000b0049eeb110e40mr28969112edt.34.1674848752088; Fri, 27 Jan 2023 11:45:52 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674848752; cv=none; d=google.com; s=arc-20160816; b=cLcQzDibwZZI5o3H4P8+V6htwP5CQsrhU6QYr7Uk+w5/pZcvDwy9WzxK65RtlCmoh5 iQZIq7SbNXpIC/Ab1QnG2ejPoULCOXa5TAuVLmw6ADnDgtPV4PXeGHhf6nbc8XqOdAak WnreDR+o+1uCho8hGaVpRp/l4OnTtCdZvE3EXeEU5ozwnOA/oAnr8bdWRnLm4JF33bHs PrxhAd9QuHZXHmwpPnOnZ1ikMkinjYuCeIuDYn0KULLXL/9kEGBWX82EwVbp7gr7Gn3d JVrfYk9MJ5E06FaD1BZvWsjYJFnZF1u44EKLEA9TXonUwTk8WOo+yPZUWeX/7E+gY+tb SAcg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=pFZiiTJ1lM5Y9cE9VJMMN6c4zys62lyfuQWFB/SvSfY=; b=nBmDctfiTFezW3gyjS8yhEGr8twkBEWLZjLuonw6Hhv6mXEBAhZ87id88p6OiscCfH S/WfDEbcClOqa1ujIhUN+V3T55K0hqrA/q+nVHjlSPzpkcpqFUQlMZ7NaPw2ztrRhGnK qHlTL2rAF4/083U1as0d10asLJDMIU5R5r+EcNkmz2MQ3o6SqI8wInYhKtyiaGAiCwvM MuWv2y2Mppr7kB/5ylwqc3s0iNVPV+cZY5UpZpZabqkxAWjo89PdNUHzH5VauMqIpwAo bnl7YwV0lXXfLvJ+BXqzl4L1ziTDAgT+CffJicpQOsTzJhBO1f0LwECW9TlHeWTO7ric jF3A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=m4Wtor8v; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f13-20020a0564021e8d00b004a0d634cd0asi7286568edf.237.2023.01.27.11.45.22; Fri, 27 Jan 2023 11:45:52 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=m4Wtor8v; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232351AbjA0ToM (ORCPT + 99 others); Fri, 27 Jan 2023 14:44:12 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50932 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232110AbjA0TnT (ORCPT ); Fri, 27 Jan 2023 14:43:19 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CE321BBA8 for ; Fri, 27 Jan 2023 11:42:53 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id b201-20020a2534d2000000b0080b87d910a2so6434989yba.21 for ; Fri, 27 Jan 2023 11:42:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=pFZiiTJ1lM5Y9cE9VJMMN6c4zys62lyfuQWFB/SvSfY=; b=m4Wtor8vj3J0ycRC6QGu22LySA+n2MGlbf/sH76T8N8Pnp/d8SuPXb/kR+Hyj37+ad 48NkH1zhtbcV3yGPaiQ5KPXnB06jsyj37BRmFZaU4H+LJ2YSwMP7BoYjL006YrtxIZWR VsSyGQhSW7cL19AuEXNwVKmULWycfkr7kU4POPAgtbT6hDzlZniUKpxVu1Hu98jAvRgz bcxq0t9OKK2J7/3O0lRl4dNTg073Sp+ojK3rnWcxuyGW0RvPh9anMnsNc+W5/rLzSMsJ Ho/St/2m0p0zsIWZF5QsNOVQeuQXkfsBP43+XpjCQzZI9euWanUB/WzPnytDjlkSAgXd IXOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=pFZiiTJ1lM5Y9cE9VJMMN6c4zys62lyfuQWFB/SvSfY=; b=1U2RGa6dSBim6mLUcCleFv/won4LB+ibPH52TR5dhn53u5bm6Hxi2ZySeG0f1XCGB5 L7AP+dFpW9rWziz2keU1hW23Km7kjmugVGs7j73S9n/pA6FBIrFR3C57iHq/RVc2PUVk plhjFtuVTx0rdWtmF+L4bo0vDngFGDnTTZxL287+RmXN/VxPKITk3yaHELOVGQ9Fsdav ouvp5Bqn1X6wQFxWEXnSmTNPWgsgfniPncduyY18kORPeh2Si0l8n70rWahQ0VEf+0vw zlQ3vt6JMqkNiuaajuKA7AS3if5jBarWzjQ+51tOU43bvh2LnHWCIToB/f5mVUtFXL9l EXow== X-Gm-Message-State: AFqh2krsdCWztM2Yn/OqeUYvXHhJkH/kzCWIgoQ5SOGkb8rgvsZ2T4nQ JBZr+VS/F9+npZuRwE9Tz2qJfB+BYNg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:1d47:0:b0:3f4:6c62:18d1 with SMTP id d68-20020a811d47000000b003f46c6218d1mr3383963ywd.491.1674848493581; Fri, 27 Jan 2023 11:41:33 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:45 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-9-surenb@google.com> Subject: [PATCH v2 08/33] mm: introduce CONFIG_PER_VMA_LOCK From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756206204808513163?= X-GMAIL-MSGID: =?utf-8?q?1756206204808513163?= This configuration variable will be used to build the support for VMA locking during page fault handling. This is enabled on supported architectures with SMP and MMU set. The architecture support is needed since the page fault handler is called from the architecture's page faulting code which needs modifications to handle faults under VMA lock. Signed-off-by: Suren Baghdasaryan --- mm/Kconfig | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index ca98b2072df5..2e4a7e61768a 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1211,6 +1211,18 @@ config LRU_GEN_STATS This option has a per-memcg and per-node memory overhead. # } +config ARCH_SUPPORTS_PER_VMA_LOCK + def_bool n + +config PER_VMA_LOCK + def_bool y + depends on ARCH_SUPPORTS_PER_VMA_LOCK && MMU && SMP + help + Allow per-vma locking during page fault handling. + + This feature allows locking each virtual memory area separately when + handling page faults instead of taking mmap_lock. + source "mm/damon/Kconfig" endmenu From patchwork Fri Jan 27 19:40:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49626 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1011714wrn; Fri, 27 Jan 2023 11:45:47 -0800 (PST) X-Google-Smtp-Source: AK7set81Uem9396Ddn2kZ32DY/+TsgTHmUL7UuQ/c70Ays7FQSx5Vu9B7mVYJKQFylDbFGv8ya/V X-Received: by 2002:a50:d4c9:0:b0:4a0:f786:d887 with SMTP id e9-20020a50d4c9000000b004a0f786d887mr5582929edj.35.1674848747628; Fri, 27 Jan 2023 11:45:47 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674848747; cv=none; d=google.com; s=arc-20160816; b=Ll5Ymq0wS+rjeim31X7vQ2UckAL0rH+i6+LFVqFbDpSBlnT5oZOYUWJSOVX40MH0wA W4R6XTr1Y2r+SgHFVSTMiOQOLqDbf5RlUb3EMid+wQvsed4KaN7oNCX9TinbghUz+dDF ffTM0uixLWOibd7Bwcj/UdHHG6Vvy3ijfx3Hc0ccl/hjNvKHD3zoRHUl8EfmYoly3BCU VMkevM48IlvdMbqVc0bIobOgsRWEbu1mN2CvcEKrNkj5O7V7Q51ZkOFk6mmKqKuy/EXf aHN88hct8JoxsvL+MLLxQOkWT0Up4G7ftg7SOPBq73mAeBelP7RPDcMX1UbnWhVsNy7T lXhA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=Nu9/mdu38G6WtBjXb6Do7xpUyt6CdrDbmAUDhoyp47k=; b=T9oeer1veGnHdU+CFltshAW4UTpsuVFpg5UiKmhjhnCx/94NzK6jedDTV+uk1IHSzd ZDP7QQcZOWuejfEe75IoFL2dmLxwBKF8/NDtkvk1/xhmUSYIFRLoyOz9EYRsobCOVBdH VBdR9P+CwvMNte0bdTWpc2UQyMw496Sp4r6eBJz8zI6PmegvV/7IyAykWxc+8OMYigEL Mr7k2Q1LRLZxNPEe0EAz+lI6wqkMZDloulozGa5n+1ffuf6QbISKlfLS7hCmxD1ijd2p ftapu7KE7WCh/XNHmfpN9jOy78ofBwoT6MpX3nnPi9oUV5RjsXREMJ90zXLkHg5tjc65 aBRg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=oFyOjlhE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i20-20020a0564020f1400b0049ea11f77eesi6099482eda.528.2023.01.27.11.45.18; Fri, 27 Jan 2023 11:45:47 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=oFyOjlhE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232184AbjA0Tn2 (ORCPT + 99 others); Fri, 27 Jan 2023 14:43:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50774 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231846AbjA0TnA (ORCPT ); Fri, 27 Jan 2023 14:43:00 -0500 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C453F82432 for ; Fri, 27 Jan 2023 11:42:36 -0800 (PST) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-4ff7c6679f2so65753727b3.12 for ; Fri, 27 Jan 2023 11:42:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Nu9/mdu38G6WtBjXb6Do7xpUyt6CdrDbmAUDhoyp47k=; b=oFyOjlhEa6jPD1d+pKzu8xqMu9sShcBJM0zkcdXAO9bWZHfs5hDegJ8sEKWFr4AQNl iGPCIH7iC8aR5RiIjRoH9W2KUncZPWl0b5vsJXQEB2UE60mYiP5ldYuNlEW3dM96CgJp +2KdvoCyVN7G6APzAMKWAfXHTUhAA+JaJZhJgfnGwlr68Zr0GLZrYj2YnOQkKo96sWGj Ng0DOWnRANnP+XQns5edhVeWyYl+9EdfDgkHbYKOkVF++Ku/0IaWanDTI7HqCQuL/w5R edw8ZGqmSQJCCmVgs5lV30HhGrOZqx2K3qMJgNbHCqjkT1CdH2dJNnKk0QEKLZZ9jCPS y7/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Nu9/mdu38G6WtBjXb6Do7xpUyt6CdrDbmAUDhoyp47k=; b=h3TEow3zRjG0lfCe5mvSQw5za2RRH4bDNJ2uFc8FE1Vk4O+/vca4y+1MhzhifpHZjF E8iCbuMu3k8njZgssRUSPH0Nd28ho8ZoHuDe/DrMao42v5gYamGAagDmFjMhhxzkzSY/ UGAvLWHmSvZ2kK07e9pNJDPSV5292hZDRv7kt29SfJpizclfJMrE8MiPfp73a38gqvcr 080uEza2rx7zqJmrUmIfF78T30hK+psCMEuX2cREmAgIXE1DQKox+FzA2kkBRXYBECRL BwOusTDNLykSbKbiuSxxKy9DNyBUFR0jTkCS2UlXO7udGjoxuP6PcHGuE+YDWaAhj8Q2 ZRHw== X-Gm-Message-State: AFqh2krEdkqLw+u/U5/61Hh8v+Wvx4W99aaUMVbiwxIw9pXjUSRW9QFK ek0PN/SUSzJ9s7kXfSjfZQEDIW8k5cQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a0d:cdc7:0:b0:4df:f712:5d57 with SMTP id p190-20020a0dcdc7000000b004dff7125d57mr3818080ywd.303.1674848495850; Fri, 27 Jan 2023 11:41:35 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:46 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-10-surenb@google.com> Subject: [PATCH v2 09/33] mm: rcu safe VMA freeing From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756206200734738651?= X-GMAIL-MSGID: =?utf-8?q?1756206200734738651?= From: Michel Lespinasse This prepares for page faults handling under VMA lock, looking up VMAs under protection of an rcu read lock, instead of the usual mmap read lock. Signed-off-by: Michel Lespinasse Signed-off-by: Suren Baghdasaryan --- include/linux/mm_types.h | 13 ++++++++++--- kernel/fork.c | 20 +++++++++++++++++++- 2 files changed, 29 insertions(+), 4 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 8410c3052148..62e413f84011 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -480,9 +480,16 @@ struct anon_vma_name { struct vm_area_struct { /* The first cache line has the info for VMA tree walking. */ - unsigned long vm_start; /* Our start address within vm_mm. */ - unsigned long vm_end; /* The first byte after our end address - within vm_mm. */ + union { + struct { + /* VMA covers [vm_start; vm_end) addresses within mm */ + unsigned long vm_start; + unsigned long vm_end; + }; +#ifdef CONFIG_PER_VMA_LOCK + struct rcu_head vm_rcu; /* Used for deferred freeing. */ +#endif + }; struct mm_struct *vm_mm; /* The address space we belong to. */ diff --git a/kernel/fork.c b/kernel/fork.c index 5f23d5e03362..314d51eb91da 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -479,12 +479,30 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return new; } -void vm_area_free(struct vm_area_struct *vma) +static void __vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); kmem_cache_free(vm_area_cachep, vma); } +#ifdef CONFIG_PER_VMA_LOCK +static void vm_area_free_rcu_cb(struct rcu_head *head) +{ + struct vm_area_struct *vma = container_of(head, struct vm_area_struct, + vm_rcu); + __vm_area_free(vma); +} +#endif + +void vm_area_free(struct vm_area_struct *vma) +{ +#ifdef CONFIG_PER_VMA_LOCK + call_rcu(&vma->vm_rcu, vm_area_free_rcu_cb); +#else + __vm_area_free(vma); +#endif +} + static void account_kernel_stack(struct task_struct *tsk, int account) { if (IS_ENABLED(CONFIG_VMAP_STACK)) { From patchwork Fri Jan 27 19:40:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49627 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1011715wrn; Fri, 27 Jan 2023 11:45:48 -0800 (PST) X-Google-Smtp-Source: AK7set/pesTshyTYA+dPMtk7j02O1nIo3NjbBG95xHjtSo1tuMZcYzF2fq2ntV4n0mZIbXrnMJGm X-Received: by 2002:a17:906:2a9b:b0:878:6755:9089 with SMTP id l27-20020a1709062a9b00b0087867559089mr6690874eje.39.1674848748177; Fri, 27 Jan 2023 11:45:48 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674848748; cv=none; d=google.com; s=arc-20160816; b=BsCp2t9Oq3XLwpeR5mKPx0PcV+evkA+3kh1Lt7bLQh5eLXXgyGVs+v8IrzvlsTd13R 87LkuFGxRXPJMteM5CevGN+qnDR5SJasU0PDZoMZCiMpUDIjhlbO5aCpMN3lwqE5AI3q 2+J5kdJB2S/056L6slBvh6CeaVmhru6UbezalfbcjOfEgGU/HzBM2KMZMavXpCP0PsEc Xv2+kKO5A7sZT8Z3+R5NnFwvU/IYBZnQrbZmQoF5QEelfHVIl4S2/TeDFcMnByQEPPry rL0MtUurk8dhfyOzWmjy7qPRS/u6ECChzUOjUUlyX66jP3r0/ihqjBIDTnOvGoluVbxT +yMw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=OfsRQat1IE0fDeYSs5Psw7qjLLZE8mkiXmSdSDqQs6E=; b=xcwp6BHL5q4Wdden1WPdCzP9H6Xf85rG93MAewOM0aR6aISz6hcFC17XstmIhuhhJ2 ctWz5cmZIkE3p8kWRowVM/AaQ+yyqL8d6KyYR4AmY7NEOPhQjKHWvKeAeWOYjordwaMj MY9zpRvd8SogE3gAZCXc+RvcLyMajLH9i7I1ZfgmqhtjsAuGN9iMd+WlhBXTZHIaDtpd 4R29ljuUnkDHCYwEdu6Ig5P4CruM4KMI014ncUHVb18uv01oThbC+C9Z/t5t70FS0aap 0iDOiNh9TjO82wwzJYWZTZZLyXuXCJ2h97hPk+bvoK8uQ+M084uMA6Hg6IY8cI1ZcB3m M9fA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=Q7jceUjh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 30-20020a170906005e00b008705d5bb5d1si6577487ejg.951.2023.01.27.11.45.19; Fri, 27 Jan 2023 11:45:48 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=Q7jceUjh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232256AbjA0Tng (ORCPT + 99 others); Fri, 27 Jan 2023 14:43:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51056 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231931AbjA0TnF (ORCPT ); Fri, 27 Jan 2023 14:43:05 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9BD478A547 for ; Fri, 27 Jan 2023 11:42:40 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id b201-20020a2534d2000000b0080b87d910a2so6435179yba.21 for ; Fri, 27 Jan 2023 11:42:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=OfsRQat1IE0fDeYSs5Psw7qjLLZE8mkiXmSdSDqQs6E=; b=Q7jceUjhtfhNXIUFz0/MzwmS5ZgtWEAK+cBK0blt69F8+KIFs/AwOKnAqPB+IEDKT/ FyLbx1gLVqD34Rlv7gRilV5FNygR87+qF/41EC627UqwDLTg6NJEFnAj/1s6LMME1gBy h9D9urWoN0e32uyucIiiCDtt59fZ91UJGpShxrAxprHccHpeDkPGS432b78Iuw1rAi+Y PGxS0P/Csnmusy4TpNpOjbr38NhV2uRvkkx/I7vAEJVPhb/fpP3E3BB7l9N5Jq8Ddy4K G0deOaLN7bYZJlZOXJL3xM94KhFiD/71EdmjsD4THVYmj7eB4g9ZYLPfdvfcp6yLyVR8 Oqkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=OfsRQat1IE0fDeYSs5Psw7qjLLZE8mkiXmSdSDqQs6E=; b=Bfr+txlCYis8bHpMFGgq5vlFNPvsd8cpOWeZybDcNlMvo1og1+9BceuR3DW5a4SDL1 0q8r3eG3It+MGgyhtmEa/knqmILVulY/92sQsuoPIz4sSWu73bR5rKjl35AiuhhF8v06 sFfMp8PSLI3ldBvOHQBdT1BTUBpWnRRzhqE4PDxbTq36/SnmYRZ2X+vCIbc3H8urgoex k2lv2x9aeWH1rnqWQaIm1DH0biu6uVW0xT3KozbF1cUmdaFqWjw/JJEg+9ABpCLPmb7N McZlNrNT2YSfIrrtVnaGIZjNhBTkxBBJ36tf5Xic2kW1jn6Hlpu2dnyOCyGbCS+kveLv Du0g== X-Gm-Message-State: AO0yUKU2P/uugDWxMvSyycdgLaCCnWctXTC2zmyAq4ra6epEZTegdGw2 erOvZQYa/DOKK4sFywa94qR21YjlVqA= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a0d:f2c3:0:b0:50a:468d:48d8 with SMTP id b186-20020a0df2c3000000b0050a468d48d8mr731943ywf.104.1674848498315; Fri, 27 Jan 2023 11:41:38 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:47 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-11-surenb@google.com> Subject: [PATCH v2 10/33] mm: move mmap_lock assert function definitions From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756206201205169258?= X-GMAIL-MSGID: =?utf-8?q?1756206201205169258?= Move mmap_lock assert function definitions up so that they can be used by other mmap_lock routines. Signed-off-by: Suren Baghdasaryan --- include/linux/mmap_lock.h | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 96e113e23d04..e49ba91bb1f0 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -60,6 +60,18 @@ static inline void __mmap_lock_trace_released(struct mm_struct *mm, bool write) #endif /* CONFIG_TRACING */ +static inline void mmap_assert_locked(struct mm_struct *mm) +{ + lockdep_assert_held(&mm->mmap_lock); + VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); +} + +static inline void mmap_assert_write_locked(struct mm_struct *mm) +{ + lockdep_assert_held_write(&mm->mmap_lock); + VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); +} + static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); @@ -150,18 +162,6 @@ static inline void mmap_read_unlock_non_owner(struct mm_struct *mm) up_read_non_owner(&mm->mmap_lock); } -static inline void mmap_assert_locked(struct mm_struct *mm) -{ - lockdep_assert_held(&mm->mmap_lock); - VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); -} - -static inline void mmap_assert_write_locked(struct mm_struct *mm) -{ - lockdep_assert_held_write(&mm->mmap_lock); - VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); -} - static inline int mmap_lock_is_contended(struct mm_struct *mm) { return rwsem_is_contended(&mm->mmap_lock); From patchwork Fri Jan 27 19:40:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49630 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1011724wrn; Fri, 27 Jan 2023 11:45:49 -0800 (PST) X-Google-Smtp-Source: AK7set939qO/0mrQestFBRcb/HwKbNl7VeObhR66sdfxYwyZ9k7ooHvA4zA3HUaFHXz71NzhEWaZ X-Received: by 2002:a17:907:a41f:b0:880:d284:11ea with SMTP id sg31-20020a170907a41f00b00880d28411eamr578305ejc.29.1674848749189; Fri, 27 Jan 2023 11:45:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674848749; cv=none; d=google.com; s=arc-20160816; b=sCRwjQFvb7tdQfrfWAF67qbsqxiaWCikxxOORyhEphonUgB9b9IxkFDSPnZbqn2I1B fDG2BRGkaEF4VwAE0NmXKYSPtLNByFUhn+TCvrWebRNGnclJ8+zUV8nGKGPTzrOl2iPE M+0SLxVqlsFc1Ud40D7MPlFykXfEBeYCDiI4x55094x/DT4DWva6Mubj8OyQ+69fs4Wm l0ia6AaXoNpqo/aiI9GGUfDoY4cpEZlTUrkyfju/aAHbog52gW8eZ5nFzKpVYyf13EfZ s2U3G1TL+x0+WCXXjLG57ctLmUtv+M4eYu/XhlzpVYYIf2u2Mju2GruZXiqilSLK1av6 nM2g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=XXZYVjn5Y9alAXdawo6LSRZijnPMVBvcI/Ib+FHEz28=; b=eSM9ZXZG8CWsE5JQJOwYmVqTfxYLrOiFAwqLtWc/VYJ75v+QeSVH8e97VgAUIco62X eXAhaPNw0ID2LlVdAopVyRaqKDiNd+J9oDu6xkn9D1iMDTI+eDiHvLGRhL65rhJNZ7i5 YJ5pR2qP4WAsgaSClyw7e39TQI0Zvhwn1vTQL08DOVYjoCR09S5hhM8Q9LljDU/p/INb 4B/JaP48zALuhp7rFHiC5A0Mogk5BCdec9YcWZI+az4CcWclvoBLesZPTr+P+0KhmBoP J/e+ih07nNxPL5cCby721lfmnkpJQph8oVFJBX2fd9tS+5+/izeoFyQ/T2jBSeE+hzzP bpgQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=dBDLlmFt; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id v9-20020a170906380900b0087859b6c546si6446808ejc.634.2023.01.27.11.45.20; Fri, 27 Jan 2023 11:45:49 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=dBDLlmFt; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232265AbjA0Tnl (ORCPT + 99 others); Fri, 27 Jan 2023 14:43:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51168 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231961AbjA0TnG (ORCPT ); Fri, 27 Jan 2023 14:43:06 -0500 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9BCA28A546 for ; Fri, 27 Jan 2023 11:42:40 -0800 (PST) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-5028c723d28so65352297b3.15 for ; Fri, 27 Jan 2023 11:42:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=XXZYVjn5Y9alAXdawo6LSRZijnPMVBvcI/Ib+FHEz28=; b=dBDLlmFtxDGaSIxnsKi7TuOlfLmJDoa6COq6/jSvBgh64Qerq1YmsLWwhnAAxCyVf/ g582eX0Eo4U8zmrxOFGYnkJmijlE67J9D9xBXGrS1NtU3RqxGXmBh2f947WBs0NX/Zll emlcCuJwSkFMTmaE18K/oCpx+uIqpIqH/tqes3ZD00h73pebtBtpRPZksR0g19LqKl4v xmDEdLHNhpj3CDXqX8SiQjapJ6l/dJ1IUEAjdkc/2LeqJcIWhkIf+A2Qxlo3WsGI6Yzg srHRwUq16G0bEIrEimCIRBcTpWLvsmh9iUkFM5cCzPZIfcbsD+d957wvwGuV97WpJ3D1 vEJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=XXZYVjn5Y9alAXdawo6LSRZijnPMVBvcI/Ib+FHEz28=; b=7PpcU/ReNrz6WTtym2Iw1X+0pHQ+uRBRf+Nd6Ux1HflBbRMN57dOQQV12HewNqvXoN /0u9bFKLdTzClf925X8Pt2VUtAYYmAvXHJzgv8qhYjF+xnYCcf6url8VIMYY10WVCDg7 0xXhzMTf4Oe6yQaekydKGxjuOzbCooXY8+hVJ1MkW27T0WMQEhhghgv18ewGQznGPqVg xR1aR7rXMdQLcsBkGeyiu6E+YZqbQm4OXZ/SfehObJdZRZDZRWW6y9dYPyCP09D7Uj1i QSuDlzw8UPtiXtEx9H9lcF08ZCkjSA0+H5ZxLU345OALWAgemRuTgzZ1raLczzCjPaWg LjCA== X-Gm-Message-State: AFqh2ko0jj7YuEELWcBExjwKBZ9J37jhyD9GGiOP8QqQ8VBiVRZlYATy Tf+ENdTMbEDRSuEeAH9u8/lMn/FkgzY= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a05:6902:10cb:b0:7b8:16c:e66d with SMTP id w11-20020a05690210cb00b007b8016ce66dmr5496306ybu.85.1674848500886; Fri, 27 Jan 2023 11:41:40 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:48 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-12-surenb@google.com> Subject: [PATCH v2 11/33] mm: add per-VMA lock and helper functions to control it From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756206202054815463?= X-GMAIL-MSGID: =?utf-8?q?1756206202054815463?= Introduce per-VMA locking. The lock implementation relies on a per-vma and per-mm sequence counters to note exclusive locking: - read lock - (implemented by vma_start_read) requires the vma (vm_lock_seq) and mm (mm_lock_seq) sequence counters to differ. If they match then there must be a vma exclusive lock held somewhere. - read unlock - (implemented by vma_end_read) is a trivial vma->lock unlock. - write lock - (vma_start_write) requires the mmap_lock to be held exclusively and the current mm counter is assigned to the vma counter. This will allow multiple vmas to be locked under a single mmap_lock write lock (e.g. during vma merging). The vma counter is modified under exclusive vma lock. - write unlock - (vma_end_write_all) is a batch release of all vma locks held. It doesn't pair with a specific vma_start_write! It is done before exclusive mmap_lock is released by incrementing mm sequence counter (mm_lock_seq). - write downgrade - if the mmap_lock is downgraded to the read lock, all vma write locks are released as well (effectivelly same as write unlock). Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 82 +++++++++++++++++++++++++++++++++++++++ include/linux/mm_types.h | 8 ++++ include/linux/mmap_lock.h | 13 +++++++ kernel/fork.c | 4 ++ mm/init-mm.c | 3 ++ 5 files changed, 110 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index dd295c020e85..fee08e8fdce7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -617,6 +617,87 @@ struct vm_operations_struct { unsigned long addr); }; +#ifdef CONFIG_PER_VMA_LOCK +static inline void vma_init_lock(struct vm_area_struct *vma) +{ + init_rwsem(&vma->lock); + vma->vm_lock_seq = -1; +} + +/* + * Try to read-lock a vma. The function is allowed to occasionally yield false + * locked result to avoid performance overhead, in which case we fall back to + * using mmap_lock. The function should never yield false unlocked result. + */ +static inline bool vma_start_read(struct vm_area_struct *vma) +{ + /* Check before locking. A race might cause false locked result. */ + if (vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq)) + return false; + + if (unlikely(down_read_trylock(&vma->lock) == 0)) + return false; + + /* + * Overflow might produce false locked result. + * False unlocked result is impossible because we modify and check + * vma->vm_lock_seq under vma->lock protection and mm->mm_lock_seq + * modification invalidates all existing locks. + */ + if (unlikely(vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq))) { + up_read(&vma->lock); + return false; + } + return true; +} + +static inline void vma_end_read(struct vm_area_struct *vma) +{ + rcu_read_lock(); /* keeps vma alive till the end of up_read */ + up_read(&vma->lock); + rcu_read_unlock(); +} + +static inline void vma_start_write(struct vm_area_struct *vma) +{ + int mm_lock_seq; + + mmap_assert_write_locked(vma->vm_mm); + + /* + * current task is holding mmap_write_lock, both vma->vm_lock_seq and + * mm->mm_lock_seq can't be concurrently modified. + */ + mm_lock_seq = READ_ONCE(vma->vm_mm->mm_lock_seq); + if (vma->vm_lock_seq == mm_lock_seq) + return; + + down_write(&vma->lock); + vma->vm_lock_seq = mm_lock_seq; + up_write(&vma->lock); +} + +static inline void vma_assert_write_locked(struct vm_area_struct *vma) +{ + mmap_assert_write_locked(vma->vm_mm); + /* + * current task is holding mmap_write_lock, both vma->vm_lock_seq and + * mm->mm_lock_seq can't be concurrently modified. + */ + VM_BUG_ON_VMA(vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); +} + +#else /* CONFIG_PER_VMA_LOCK */ + +static inline void vma_init_lock(struct vm_area_struct *vma) {} +static inline bool vma_start_read(struct vm_area_struct *vma) + { return false; } +static inline void vma_end_read(struct vm_area_struct *vma) {} +static inline void vma_start_write(struct vm_area_struct *vma) {} +static inline void vma_assert_write_locked(struct vm_area_struct *vma) {} + +#endif /* CONFIG_PER_VMA_LOCK */ + static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) { static const struct vm_operations_struct dummy_vm_ops = {}; @@ -625,6 +706,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); + vma_init_lock(vma); } /* Use when VMA is not part of the VMA tree and needs no locking */ diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 62e413f84011..88619c6a29a3 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -508,6 +508,11 @@ struct vm_area_struct { vm_flags_t __private __vm_flags; }; +#ifdef CONFIG_PER_VMA_LOCK + int vm_lock_seq; + struct rw_semaphore lock; +#endif + /* * For areas with an address space and backing store, * linkage into the address_space->i_mmap interval tree. @@ -633,6 +638,9 @@ struct mm_struct { * init_mm.mmlist, and are protected * by mmlist_lock */ +#ifdef CONFIG_PER_VMA_LOCK + int mm_lock_seq; +#endif unsigned long hiwater_rss; /* High-watermark of RSS usage */ diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index e49ba91bb1f0..aab8f1b28d26 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -72,6 +72,17 @@ static inline void mmap_assert_write_locked(struct mm_struct *mm) VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); } +#ifdef CONFIG_PER_VMA_LOCK +static inline void vma_end_write_all(struct mm_struct *mm) +{ + mmap_assert_write_locked(mm); + /* No races during update due to exclusive mmap_lock being held */ + WRITE_ONCE(mm->mm_lock_seq, mm->mm_lock_seq + 1); +} +#else +static inline void vma_end_write_all(struct mm_struct *mm) {} +#endif + static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); @@ -114,12 +125,14 @@ static inline bool mmap_write_trylock(struct mm_struct *mm) static inline void mmap_write_unlock(struct mm_struct *mm) { __mmap_lock_trace_released(mm, true); + vma_end_write_all(mm); up_write(&mm->mmap_lock); } static inline void mmap_write_downgrade(struct mm_struct *mm) { __mmap_lock_trace_acquire_returned(mm, false, true); + vma_end_write_all(mm); downgrade_write(&mm->mmap_lock); } diff --git a/kernel/fork.c b/kernel/fork.c index 314d51eb91da..9141427a98b2 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -474,6 +474,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) */ data_race(memcpy(new, orig, sizeof(*new))); INIT_LIST_HEAD(&new->anon_vma_chain); + vma_init_lock(new); dup_anon_vma_name(orig, new); } return new; @@ -1147,6 +1148,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, seqcount_init(&mm->write_protect_seq); mmap_init_lock(mm); INIT_LIST_HEAD(&mm->mmlist); +#ifdef CONFIG_PER_VMA_LOCK + mm->mm_lock_seq = 0; +#endif mm_pgtables_bytes_init(mm); mm->map_count = 0; mm->locked_vm = 0; diff --git a/mm/init-mm.c b/mm/init-mm.c index c9327abb771c..33269314e060 100644 --- a/mm/init-mm.c +++ b/mm/init-mm.c @@ -37,6 +37,9 @@ struct mm_struct init_mm = { .page_table_lock = __SPIN_LOCK_UNLOCKED(init_mm.page_table_lock), .arg_lock = __SPIN_LOCK_UNLOCKED(init_mm.arg_lock), .mmlist = LIST_HEAD_INIT(init_mm.mmlist), +#ifdef CONFIG_PER_VMA_LOCK + .mm_lock_seq = 0, +#endif .user_ns = &init_user_ns, .cpu_bitmap = CPU_BITS_NONE, #ifdef CONFIG_IOMMU_SVA From patchwork Fri Jan 27 19:40:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49629 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1011725wrn; Fri, 27 Jan 2023 11:45:49 -0800 (PST) X-Google-Smtp-Source: AMrXdXuHgPK2kwqi5Msmx+rnyFG8iEN8L9MNTwLKWv/ZjGoujnyKfB+/6eWFNtJbJmDtbxNxe/1m X-Received: by 2002:a17:907:7f93:b0:877:60a2:1c6d with SMTP id qk19-20020a1709077f9300b0087760a21c6dmr48850219ejc.38.1674848749231; Fri, 27 Jan 2023 11:45:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674848749; cv=none; d=google.com; s=arc-20160816; b=ichPwbWrBTnjF32ZK9WuS6ot1umBjHng1rfqtsb5gDBcmQxbzAp5Zhevy5yZDLMU1z DdNT0xrg7lXM99XO7mVQDEg/AtvqFgwGP0d6O3uMqjIiS2jzYNvnNKmn4HhZC7fdZGgo LS+hC+XoRKRrwuNVQGSPlJa9E/LMqaas4JF2XtPDUFsUFlxgoRMMhiyxfxCOvbTsgPnP dhUwC2MdbCWl6oVW6jc4uYsQL7gkvf0KCfOEGkX7PO/u9OcyUaYVTtjAFWSAMKmdYIPM Wxi9tnpRSQv0VNOnRBJt/KeMd8JzfgOi++YIU+MG4m9T6FmJd0Y/Nz39+PHJHthTIZwo fybQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=Hq20e7KS4f4C48PKmYYknVSNimtVVHYzyV8VwmVTJ0Y=; b=bM9eQlfnGPKkXBDX6oXLnumnz/1DPjApRHefh3UR4sp/rVyq26Tv85xgXROhxlMsuj nCJeLP3XHwLzntZSrli4kQ+ovHd114dTgVjIT3V7NML1743UYN/+x7jfQ0LsOqoZbEZ/ SiHKob29a9ooN9RHPzrz3BAtMylNQThIpxfCwHT/EI30HL8V3NPPd+xcit4kfvDAdN8u 38vgQurTKu7EP9/LdiOmITl2YRDWEnXI53kMWhi88zJqGLgRk0jze9MUO0T3kmriNknj 2NOU1jS17zA6W2ClQPWExsHkOYW1Vuhi52BcOKtnTJTOnDyzVdcrE+fAC4DrtmLIW9Gr RG2g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=SD5yhaj9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id z23-20020a17090674d700b007c13387a548si6718708ejl.26.2023.01.27.11.45.21; Fri, 27 Jan 2023 11:45:49 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=SD5yhaj9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232301AbjA0Tnx (ORCPT + 99 others); Fri, 27 Jan 2023 14:43:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50752 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231740AbjA0TnR (ORCPT ); Fri, 27 Jan 2023 14:43:17 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 92F67173C for ; Fri, 27 Jan 2023 11:42:47 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id o1-20020a252801000000b0080b8600bdc9so6361211ybo.3 for ; Fri, 27 Jan 2023 11:42:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Hq20e7KS4f4C48PKmYYknVSNimtVVHYzyV8VwmVTJ0Y=; b=SD5yhaj9WNDyBmH2zVXxIQSAPvkdOOdT/eZZRKlGP77z2J9nVnn6Obfbxv5sauUEV/ LCtaz+vX4qip57892cEocWjVTNmQS4/TdyxbxT8xLbIWgkt6lFG5ZxopwUuxBJEnTkjr vQID9RMKTIQGW2sRQmcGmPfXjoMh0D1eDPyuL/b6rqwKxYdkfEYgEgHjQL0Tw8ZlMbjX ZH2/uoF+k+VRzud7oIym//RQdtO+lmKAHdfjocpDVWhkn9piFYOBRbFkKQo8sM+osjEQ FxFQ0vmeBJ9JfrQeSA2WMn51BintgFmdPrgL1EZjLw25vLNDWAlkic6I+BYJ/3xqK9CW +GQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Hq20e7KS4f4C48PKmYYknVSNimtVVHYzyV8VwmVTJ0Y=; b=xbMAK4Bn4uhVjDHVf3YtwlIXPk0dxw7s513AIEBLb/aFdW7jNTCRS10jOkysBvTZ/v zyIvnaPSx51ikrXwyY16sr6Liz1jwjLJnZRgpXNeWA4QrVBEn3dO0i3Ej3MC9Khp7Q7X JXpuZx87/lVqa0CzO70z3eFtzmoGGl3e503liZJGHU6CfvibO5yussLexm5ekpS/iHlB OLH3sMjTvW7MOQIKOJF1IotNa9YRiURGaXezzFvJBr9tbqpotzJ2v0m+H8BllzcDYbrM KSO/M+5aLYPmp0Oxhz5a7QnoU12smiE2p+yKc6oxiEQqEPO3lDsIs6tR0goNULUH3wCN UgJQ== X-Gm-Message-State: AO0yUKUI+KrJtqbbROnssNiZErS0gVFaBf5G47r63UyqAzpdzpGZOUtP pcfFXwUkPXNXvMtaWOlGIDOT5nxB8Hc= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a25:b283:0:b0:80b:c7cb:6d1f with SMTP id k3-20020a25b283000000b0080bc7cb6d1fmr1156521ybj.298.1674848502930; Fri, 27 Jan 2023 11:41:42 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:49 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-13-surenb@google.com> Subject: [PATCH v2 12/33] mm: mark VMA as being written when changing vm_flags From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756206202238636034?= X-GMAIL-MSGID: =?utf-8?q?1756206202238636034?= Updates to vm_flags have to be done with VMA marked as being written for preventing concurrent page faults or other modifications. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index fee08e8fdce7..66dca140695e 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -720,21 +720,21 @@ static inline void vm_flags_init(struct vm_area_struct *vma, static inline void vm_flags_reset(struct vm_area_struct *vma, vm_flags_t flags) { - mmap_assert_write_locked(vma->vm_mm); + vma_start_write(vma); vm_flags_init(vma, flags); } static inline void vm_flags_set(struct vm_area_struct *vma, vm_flags_t flags) { - mmap_assert_write_locked(vma->vm_mm); + vma_start_write(vma); ACCESS_PRIVATE(vma, __vm_flags) |= flags; } static inline void vm_flags_clear(struct vm_area_struct *vma, vm_flags_t flags) { - mmap_assert_write_locked(vma->vm_mm); + vma_start_write(vma); ACCESS_PRIVATE(vma, __vm_flags) &= ~flags; } @@ -755,7 +755,7 @@ static inline void __vm_flags_mod(struct vm_area_struct *vma, static inline void vm_flags_mod(struct vm_area_struct *vma, vm_flags_t set, vm_flags_t clear) { - mmap_assert_write_locked(vma->vm_mm); + vma_start_write(vma); __vm_flags_mod(vma, set, clear); } From patchwork Fri Jan 27 19:40:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49631 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1011738wrn; Fri, 27 Jan 2023 11:45:51 -0800 (PST) X-Google-Smtp-Source: AMrXdXsxUjuaGH6uAdyPDcXPaAf/1VsoT7tsfmLbzDg+rZ/lFeE9X/Hhwbng8V7yzj/T18WyC0cR X-Received: by 2002:a05:6402:4ce:b0:47f:bc9b:46ec with SMTP id n14-20020a05640204ce00b0047fbc9b46ecmr41285125edw.7.1674848751411; Fri, 27 Jan 2023 11:45:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674848751; cv=none; d=google.com; s=arc-20160816; b=ldEY6papjwkPV3doh7yqFf0MU3GypYemNiHKqQPXB2zSeBeMFkAxgIJGQS9HsQnV4p dK3/FJAmirXqcq5zPWLNAsiiOfX91cfhh2t9VGw84oCdTuhWKjLAbo8XuSBkfnA7pF/9 UtiAsdaeR3cQWJXv79wslmqPnQN/DmBuqL4vKad593y8d3SeJ/5MNxTCW42QeWkCKSul Cneh7dZlopA2bvkywHL4BljrclBrE5tlCYKOmuatENp4Z0+sA7P8NyRDo2S3aJmd9152 6BPrm78E3tfLFda72Uq78IKM4eCK+4HA3VA6/mLOoffS7teko2whgzVnqBLA3BsDWr3+ hn6g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=icGEQDhVNg8AQkGt0Dau1cb43VWZWeb/kUxMjHSAwGA=; b=jUI1Fyr86c361Ped33Q1nNuiIJiv+1g/+f+3vgjZ4ybp53Iw84HgBOWMs1t6T2HPOB sEaEl9ewHEGYz8NB/CLF0V/IaYAjBHRpTU5IDXJaIURRhwyBslOJnEzgv08zd965JuPM R1/mXU8quoSlcCi2oCMYNPfdmie1gsE54GtuaLDelCFG4qoiy62UOpMTP81ZgCdAOcsA bKahB+rtljpNjKodZ/CnEbIN9eMnCyEH+lgu7c8eB08TdzhK0z078uQCOThFWb0AadLc zPjpCeKuIaYl+IBsLXCpPTMSLy9GtO+SR7ewiXCR079RO1laFsYZnOsgJI8jHo3tO4dm TETg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=nhjpt9Vb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c30-20020a17090620de00b0084b9ac8f8a9si5969938ejc.959.2023.01.27.11.45.22; Fri, 27 Jan 2023 11:45:51 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=nhjpt9Vb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232331AbjA0ToH (ORCPT + 99 others); Fri, 27 Jan 2023 14:44:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50624 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231774AbjA0TnT (ORCPT ); Fri, 27 Jan 2023 14:43:19 -0500 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4C5521707 for ; Fri, 27 Jan 2023 11:42:50 -0800 (PST) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-4b34cf67fb6so65244707b3.6 for ; Fri, 27 Jan 2023 11:42:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=icGEQDhVNg8AQkGt0Dau1cb43VWZWeb/kUxMjHSAwGA=; b=nhjpt9VbHDPsHYg33OlJfuq5Iz35VmBZ3kiMxPcXRXZZYdOboFlR0spKtxfGWFFcLn eBkIzKgB8QHaANcEzDHfkdLN1fRlxg2tqUBsMTv/7fNevBLbu/43ueowkRgVsLx1vLRo uHdRFKLtGeWmcojM55WkW4Vbk+IWqqj37nvYWjKvQOLDRhcdlXynMiveqJDQteuGpWZz ETc6Sup2efRPDAlMWE74qe3dlQIJeV8JWIsag7a7AlDm28r/breAdd9LIL/XkRDxD2X3 yud8bBPMeadGbrZ7ftbCoegvJj4wUFszXo3BfVZSLdIr/XgKY9tYqvkkj2bzILo8Fvtn ItZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=icGEQDhVNg8AQkGt0Dau1cb43VWZWeb/kUxMjHSAwGA=; b=y1HZAyEVXuN9pXeCS2t5qJSou9ObpYXCbbNtNA2JwnwOoFIEUN/bECVKKhd07WP/0r 6TNZKusGdxCRcH4XFI7UEr8qFwnbUwhu/M5ebxDPlMuSuY9ysmIIdNTFJn6m3WpoicD6 mQCLFb4BUY79qHln00XP1YbuW3Ljl3If9APCkKwJj9o7kzw9R3EiIcNYQ18QMhS/rmBc 0r9up9QuxckVync0u3RMgFvqJwZa+H1GxodCHr0JnfLIHag4cAKojD8qihR4HQF40zUI 9QP5T18zb8TkCEDcJbM/vzM7eaRaTFszoi31gF25tcWg72GZ4UN6xhA00/r4AsdMI7eN JVig== X-Gm-Message-State: AO0yUKXcFnMYtY8XvKy6qfMYqRNV601DG3sjngfFmok7Mny19cesngMc MbJpJXtj1ereY5NAWMIlKHT/bMURsZk= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:48cb:0:b0:506:4c8b:231e with SMTP id v194-20020a8148cb000000b005064c8b231emr1848392ywa.80.1674848505193; Fri, 27 Jan 2023 11:41:45 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:50 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-14-surenb@google.com> Subject: [PATCH v2 13/33] mm/mmap: move VMA locking before vma_adjust_trans_huge call From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756206204356017782?= X-GMAIL-MSGID: =?utf-8?q?1756206204356017782?= vma_adjust_trans_huge() modifies the VMA and such modifications should be done after VMA is marked as being written. Therefore move VMA flag modifications before vma_adjust_trans_huge() so that VMA is marked before all these modifications. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mm/mmap.c b/mm/mmap.c index 9efe13d36df7..7467d691e357 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2910,11 +2910,12 @@ static int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, if (vma_iter_prealloc(vmi)) goto unacct_fail; + /* Set flags first to implicitly lock the VMA before updates */ + vm_flags_set(vma, VM_SOFTDIRTY); vma_adjust_trans_huge(vma, vma->vm_start, addr + len, 0); init_vma_prep(&vp, vma); vma_prepare(&vp); vma->vm_end = addr + len; - vm_flags_set(vma, VM_SOFTDIRTY); vma_iter_store(vmi, vma); vma_complete(&vp, vmi, mm); From patchwork Fri Jan 27 19:40:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49632 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1011742wrn; Fri, 27 Jan 2023 11:45:52 -0800 (PST) X-Google-Smtp-Source: AMrXdXvydNPzv+SKInDwhs8nHtV3r8EUs+ZQkjwn8WQr0ReoMlP9OlrHzQYowK0KSDv2EZIfBzMA X-Received: by 2002:a05:6402:174d:b0:49e:4b8:f6ab with SMTP id v13-20020a056402174d00b0049e04b8f6abmr41278937edx.8.1674848751910; Fri, 27 Jan 2023 11:45:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674848751; cv=none; d=google.com; s=arc-20160816; b=xJknqr+RSKl79St7oiV9K4Wkhjcvbt/w4qXwGwo9jtcZK8y1hSacJCP2bHDkSEmPpm HU8DOUksvQoF8KCt+C/+svNvH9PI+CzKgxyXXCerD/swlnliDGqLDHXjfhZZBsDH2kav hRximWh5y8z5DOQ9T4xavgSEc2QBrSExb/40ck8JXotWF7+cwIVPQ4ctZ4BfydZOo+y1 CZYZSTiLXZn1OQmN+oWFLQGkY8Xu4T7xNUPwFRyscrGAuRt1SsbqOyR6LYeTSuDefX3o qXl+Bx6jRbtzr4QiCSVtnjn7fTKy92fitFU1tGzdXwjgQEKpme85C35VMHSW559EUA1r utPQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=7Tx+qt9ZwCfzeziH+AxLsfIEHPwRgvIaxjFEfdwiAVs=; b=A1joyl8Dq4hIfV091D8ZnL40d9u0WnuM0u3+5xiYv0+fHPgM1lGNsR46ZduZUaEhJg KMq/cusCblQmzBgJUfMGrEMFZTYhOcxdnvPplLYt2lOa6Kwefv2Jze/Wsjlm8yz3lBkZ Lqvpyfr+0MetXieMBr7HgZwh6hx+3fjt9UcLwWltQ20yUCbWXmwEOO/SoU93XcZFBXcV d6Kg+IOA8CmUOIXhq23fwXcnefsnTr0JjzbAa39AARHJx+2PVFAcpI3LhvtkXFhhz7ZI SgKi50V2jiBNVevn+hWkbnh6Thh+aEza4RWlXqizssWtNoO96jeBHY+JGdcGsdaoxlZj gU9A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=MwDUDcWN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q28-20020a056402249c00b00489e993b8easi6707926eda.630.2023.01.27.11.45.22; Fri, 27 Jan 2023 11:45:51 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=MwDUDcWN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232127AbjA0ToJ (ORCPT + 99 others); Fri, 27 Jan 2023 14:44:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51616 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231838AbjA0TnT (ORCPT ); Fri, 27 Jan 2023 14:43:19 -0500 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 606A14C08 for ; Fri, 27 Jan 2023 11:42:52 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5066df312d7so65757957b3.0 for ; Fri, 27 Jan 2023 11:42:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7Tx+qt9ZwCfzeziH+AxLsfIEHPwRgvIaxjFEfdwiAVs=; b=MwDUDcWNdtFYfmhAuC43tEYj+bx0/7zv78dJ0hpiWDqIAfrnPySCh31pwGtES+llPo g9aJ1BaJcPmj6QIFKuJtNB4+OKnKX4w74YBlWwFtrSbaxVsErI4QpqtlUvrQKvku3x0Y fv8bSpOOjUv/1+/JuPfz5M6ciXa9EK4OZHTaoyyN5QHozfr1J0btP+Cb6s+q8s6DTvgx Eb/YBgAB71iKrit1/2Z+f0K+0VaK6nNNkqEz24OPAMCTiIMwYOn1N+e1hh+02mLm2qhR X0SB7IZ09+W7stQU0DgYwMtwYdWCgN82KCjbLOZQQ4WgzpAt4BTc4SEqK+AY1F70NhR7 prsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7Tx+qt9ZwCfzeziH+AxLsfIEHPwRgvIaxjFEfdwiAVs=; b=V0BaELmMQZ69LYaSXhJDxSjpLq08iRF43H36xf+HTJhHeKFU94MN+Yypdg/U1ec0vy MqoNopfj2scoG2kitq6Lo/Qtnp2SrAGoCWIX1fMpftOlfYhnPnQgoH/+g03vUeYZI5fS Yd2pcWbR+LdMssq8OfmMlzC9feazNCfyRzjUFEKo1mzbCyDkmMOBmiawpsYls4Wi6dUo hyPo4QoPkuICVMcymrxgUXhdHzi14ddFAxFXtxTgle3KKQWOua7vsTFYogcLDhzV7rN1 w3qyFenVl8AozhAo0vUd4/XOZT8u/aKSjJuWKdOg9T1vsyfEsUF9Zuy8aYyvTLKchS2a CL2Q== X-Gm-Message-State: AO0yUKVZKKh0JMCEWiABt0r/SCEcmWcKNEA4V8AAEJdiQbVcddRjAcFo 5QW8M/0oTahlvpg13KbNeT/q2PnYEKs= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:6008:0:b0:506:833a:c32e with SMTP id u8-20020a816008000000b00506833ac32emr1175163ywb.50.1674848507730; Fri, 27 Jan 2023 11:41:47 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:51 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-15-surenb@google.com> Subject: [PATCH v2 14/33] mm/khugepaged: write-lock VMA while collapsing a huge page From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756206205099119202?= X-GMAIL-MSGID: =?utf-8?q?1756206205099119202?= Protect VMA from concurrent page fault handler while collapsing a huge page. Page fault handler needs a stable PMD to use PTL and relies on per-VMA lock to prevent concurrent PMD changes. pmdp_collapse_flush(), set_huge_pmd() and collapse_and_free_pmd() can modify a PMD, which will not be detected by a page fault handler without proper locking. Before this patch, page tables can be walked under any one of the mmap_lock, the mapping lock, and the anon_vma lock; so when khugepaged unlinks and frees page tables, it must ensure that all of those either are locked or don't exist. This patch adds a fourth lock under which page tables can be traversed, and so khugepaged must also lock out that one. Signed-off-by: Suren Baghdasaryan --- mm/khugepaged.c | 5 +++++ mm/rmap.c | 31 ++++++++++++++++--------------- 2 files changed, 21 insertions(+), 15 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index eb38bd1b1b2f..b938b286cdc3 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1134,6 +1134,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, if (result != SCAN_SUCCEED) goto out_up_write; + vma_start_write(vma); anon_vma_lock_write(vma->anon_vma); mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, address, @@ -1601,6 +1602,9 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto drop_hpage; } + /* Lock the vma before taking i_mmap and page table locks */ + vma_start_write(vma); + /* * We need to lock the mapping so that from here on, only GUP-fast and * hardware page walks can access the parts of the page tables that @@ -1806,6 +1810,7 @@ static int retract_page_tables(struct address_space *mapping, pgoff_t pgoff, result = SCAN_PTE_UFFD_WP; goto unlock_next; } + vma_start_write(vma); collapse_and_free_pmd(mm, vma, addr, pmd); if (!cc->is_khugepaged && is_target) result = set_huge_pmd(vma, addr, pmd, hpage); diff --git a/mm/rmap.c b/mm/rmap.c index 15ae24585fc4..8e1a2ad9ca53 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -25,21 +25,22 @@ * mapping->invalidate_lock (in filemap_fault) * page->flags PG_locked (lock_page) * hugetlbfs_i_mmap_rwsem_key (in huge_pmd_share, see hugetlbfs below) - * mapping->i_mmap_rwsem - * anon_vma->rwsem - * mm->page_table_lock or pte_lock - * swap_lock (in swap_duplicate, swap_info_get) - * mmlist_lock (in mmput, drain_mmlist and others) - * mapping->private_lock (in block_dirty_folio) - * folio_lock_memcg move_lock (in block_dirty_folio) - * i_pages lock (widely used) - * lruvec->lru_lock (in folio_lruvec_lock_irq) - * inode->i_lock (in set_page_dirty's __mark_inode_dirty) - * bdi.wb->list_lock (in set_page_dirty's __mark_inode_dirty) - * sb_lock (within inode_lock in fs/fs-writeback.c) - * i_pages lock (widely used, in set_page_dirty, - * in arch-dependent flush_dcache_mmap_lock, - * within bdi.wb->list_lock in __sync_single_inode) + * vma_start_write + * mapping->i_mmap_rwsem + * anon_vma->rwsem + * mm->page_table_lock or pte_lock + * swap_lock (in swap_duplicate, swap_info_get) + * mmlist_lock (in mmput, drain_mmlist and others) + * mapping->private_lock (in block_dirty_folio) + * folio_lock_memcg move_lock (in block_dirty_folio) + * i_pages lock (widely used) + * lruvec->lru_lock (in folio_lruvec_lock_irq) + * inode->i_lock (in set_page_dirty's __mark_inode_dirty) + * bdi.wb->list_lock (in set_page_dirty's __mark_inode_dirty) + * sb_lock (within inode_lock in fs/fs-writeback.c) + * i_pages lock (widely used, in set_page_dirty, + * in arch-dependent flush_dcache_mmap_lock, + * within bdi.wb->list_lock in __sync_single_inode) * * anon_vma->rwsem,mapping->i_mmap_rwsem (memory_failure, collect_procs_anon) * ->tasklist_lock From patchwork Fri Jan 27 19:40:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49640 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1012198wrn; Fri, 27 Jan 2023 11:47:06 -0800 (PST) X-Google-Smtp-Source: AMrXdXtTQpvPTj8kXuAr7wAzMQuY/EJKH8sYPuSh4gkqVPVVMzefbhhYVQl7ic+12XENXI8/wbS1 X-Received: by 2002:a05:6402:3220:b0:49e:1d59:794f with SMTP id g32-20020a056402322000b0049e1d59794fmr50981783eda.22.1674848826769; Fri, 27 Jan 2023 11:47:06 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674848826; cv=none; d=google.com; s=arc-20160816; b=a3tMhdYdfE/PYLGfwLBU2YhS9mCp8XVVc/Z9zzLd4BQwaPIF1Azh2LpUW39afdP94W pvtzoVTmCXf30ockhhj1qTtLY3zwY+w4ST5LZSv1ctAIRmir/fNQKevi2Bxgeh7hrIFj c8vTlR+w2CT1RHI+5q19W2GyiZgMK0AuVcvak8NLoey6HTA+cNKGywaUUlk1Ojyb8Od+ NevASr975j4OR9lFybN48CeB8qCinddm0jtC7bAUAHI8W3hwy7WWano0WKn74Z4Mr5wK o4htf7ftDkBVHKrEA1LQKy/4CRNDujF39iYPClJSMyL+/wq7ckCaszri/xc7/C5pTnLS +qKw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=xM1gV9tZESxmDxiysmAmLZq38ZaKZXSlDI1GeX9HKic=; b=MDhk6xUH+Rn+8V65MYIpE31x9kVgSlCMGrSZxtAf5z6U27+O5fmTtVxywkk1MLRfZh gX8rFW4F1cNkmkr4xaQWavxUJ+yL9xN0PXvuntRp0gOzeo7ZYACwav7PA6+jaaJm+aLe nB5n6GSBtwerYD6BgFeNovgZlgfdvRLMvg6kVsNwid9qUXq8EyMziJdeycnl1bCI0P8s ptxRxkOuGJSVfiElwHzcd+gN28/1k8fK6hfKBcAzIvYMPDBX+iBmII1twWCrMkQzzZVe xzHFECT4+CibmhkRpItugL1ca8mX4CrLw/WWcjI8DfSZvyLSuoUP1vgSFTRKlMmqAbAw sN1Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=lE9qRR84; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 17-20020a508e11000000b004a21579d23esi1797636edw.510.2023.01.27.11.46.42; Fri, 27 Jan 2023 11:47:06 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=lE9qRR84; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232438AbjA0Toa (ORCPT + 99 others); Fri, 27 Jan 2023 14:44:30 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51666 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232144AbjA0TnU (ORCPT ); Fri, 27 Jan 2023 14:43:20 -0500 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4015312851 for ; Fri, 27 Jan 2023 11:42:55 -0800 (PST) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-5063c0b909eso64912297b3.7 for ; Fri, 27 Jan 2023 11:42:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=xM1gV9tZESxmDxiysmAmLZq38ZaKZXSlDI1GeX9HKic=; b=lE9qRR84U5+rYrRmzpPAw7Ud6uFSOEezLerrdXzskk/7N7IWEzIS+P8h33l5/GHLwm pzf0b8p0OfVWGzGLb/dMUWWv/4+TG3PN/VW/HgmJDf4GTYQxz4qhZvzGmSe6iwYXQbEv pmRWin5z3kTr/dQHGmOmQouL9X7gNxEvHMfX/Mw4+j3xs8bwN3ke4zv6/7Jpk/cXGw8a hnXfsAcVwtJnLNS//9cNJnJKu2XJduYEuopAyOxXqZTZsrcp23rvXM55Jk6t7ntayedf QFH2QchxeUajMjFYvRPLkdcrU5vfcJYPLkQ5Wb/mKP36Ds4BBv5V6MGo7zAGbp1SePYq xw1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xM1gV9tZESxmDxiysmAmLZq38ZaKZXSlDI1GeX9HKic=; b=73GImSy7IXwMLb7/l1uQMMn730QaZUY2hwN3cv/BXAsFGU+VFcep1WFHEVFprIE8ma AM+QxCr6025GiRkn/dFaFC49Fgu6jdd8dz8jRJBlkld8WDOGeWa8dwPsRjbuiBwx/3pd C2RINAYmlDXlrHhn+x+lSDByJnheZGPe3BM9e/4whIdeKrXKPVQHuz2azjTJkqwk4e23 kZWf+itTB97msJR8x3/Hgug04Ew7OoFlfEsth30QIe9DZEsYEV3hN0KoCedAFuXFMMB3 TCV2kWFvKKKpcKw9R78wJvtGkblMaNeyfh9Em+u7UJbquxI5YLdnpcLmYLFSny/mKGap tvrw== X-Gm-Message-State: AFqh2koSJTXsnCcicGEuDI+73eFPxZXGEG5rH3qONUUPHNF78i8OpfSG nGfml7JoA78HlNwP9i6WsAI0BXFP/ZI= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a25:b94d:0:b0:801:e503:dd0c with SMTP id s13-20020a25b94d000000b00801e503dd0cmr2559229ybm.384.1674848510225; Fri, 27 Jan 2023 11:41:50 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:52 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-16-surenb@google.com> Subject: [PATCH v2 15/33] mm/mmap: write-lock VMAs before merging, splitting or expanding them From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756206283482751773?= X-GMAIL-MSGID: =?utf-8?q?1756206283482751773?= Decisions about whether VMAs can be merged, split or expanded must be made while VMAs are protected from the changes which can affect that decision. For example, merge_vma uses vma->anon_vma in its decision whether the VMA can be merged. Meanwhile, page fault handler changes vma->anon_vma during COW operation. Write-lock all VMAs which might be affected by a merge or split operation before making decision how such operations should be performed. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index 7467d691e357..6fff76334177 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -269,8 +269,11 @@ SYSCALL_DEFINE1(brk, unsigned long, brk) */ vma_iter_init(&vmi, mm, oldbrk); next = vma_find(&vmi, newbrk + PAGE_SIZE + stack_guard_gap); - if (next && newbrk + PAGE_SIZE > vm_start_gap(next)) - goto out; + if (next) { + vma_start_write(next); + if (newbrk + PAGE_SIZE > vm_start_gap(next)) + goto out; + } brkvma = vma_prev_limit(&vmi, mm->start_brk); /* Ok, looks good - let it rip. */ @@ -912,10 +915,17 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm, if (vm_flags & VM_SPECIAL) return NULL; + if (prev) + vma_start_write(prev); next = find_vma(mm, prev ? prev->vm_end : 0); + if (next) + vma_start_write(next); mid = next; - if (next && next->vm_end == end) /* cases 6, 7, 8 */ + if (next && next->vm_end == end) { /* cases 6, 7, 8 */ next = find_vma(mm, next->vm_end); + if (next) + vma_start_write(next); + } /* verify some invariant that must be enforced by the caller */ VM_WARN_ON(prev && addr <= prev->vm_start); @@ -2163,6 +2173,7 @@ int __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, WARN_ON(vma->vm_start >= addr); WARN_ON(vma->vm_end <= addr); + vma_start_write(vma); if (vma->vm_ops && vma->vm_ops->may_split) { err = vma->vm_ops->may_split(vma, addr); if (err) @@ -2518,6 +2529,8 @@ unsigned long mmap_region(struct file *file, unsigned long addr, /* Attempt to expand an old mapping */ /* Check next */ + if (next) + vma_start_write(next); if (next && next->vm_start == end && !vma_policy(next) && can_vma_merge_before(next, vm_flags, NULL, file, pgoff+pglen, NULL_VM_UFFD_CTX, NULL)) { @@ -2527,6 +2540,8 @@ unsigned long mmap_region(struct file *file, unsigned long addr, } /* Check prev */ + if (prev) + vma_start_write(prev); if (prev && prev->vm_end == addr && !vma_policy(prev) && (vma ? can_vma_merge_after(prev, vm_flags, vma->anon_vma, file, pgoff, vma->vm_userfaultfd_ctx, NULL) : @@ -2900,6 +2915,8 @@ static int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, if (security_vm_enough_memory_mm(mm, len >> PAGE_SHIFT)) return -ENOMEM; + if (vma) + vma_start_write(vma); /* * Expand the existing vma if possible; Note that singular lists do not * occur after forking, so the expand will only happen on new VMAs. From patchwork Fri Jan 27 19:40:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49646 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1012582wrn; Fri, 27 Jan 2023 11:48:11 -0800 (PST) X-Google-Smtp-Source: AMrXdXuMpHdzd0PUbCE7MOXOxupOkyTkQ0EJtVzizWF058SRmIk+H7je0Q4BWzwjhnLtj9qN3kOQ X-Received: by 2002:a05:6402:4305:b0:49c:7aa2:55de with SMTP id m5-20020a056402430500b0049c7aa255demr55581309edc.1.1674848890979; Fri, 27 Jan 2023 11:48:10 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674848890; cv=none; d=google.com; s=arc-20160816; b=Ge03Upe/eo30lotAL3/+DvXOu050Y8ZW0dV7dxGJoSS257LAAmEEdoL6SBKkl3dxmu 0/0r7u3oq5lyJi7E6I25Klo2fkfX5jkZTYmK9xAiHqQNygi4oImqueOvERMit5swLrUn +LuPsKdSFVjIlUGXtRIDw+1UibyRq1wNGjxIDKFK+hRgj28/DrVgyIJzFqAPc0MUpRM5 Qkki2vVhbRQ6EI9r2Cup97f3g4fUHPxxZffTeEzXz9LnHU2v9plZBsDDz5DVf04HzHJB uHguXEl8hYUfhiswEUbE1/w7X4VjjyCDAtRjwFdEWBUqwfU/C1Jt49pXkIjV0ejWqMQ/ smLQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=z2/pTj5C3cgEFiOJp2CeCn46OI8ibDfme0ccGGe2ykQ=; b=ClSyaGcZ4LXVdE84ClXYD1bReln/xWA+1cBHGvqesTuf34/gA/3A8OrUEpJVWik21v ZcGg1vqv1Ls/qeva4UN5I9AySwwD6ghMLg4D3JszEn80AmjkOEYk87OwHVGTdqDOCD7W lhHLetxAUYYLalCpFZf/Za1J14FSc7mQdL/gys7XIRIIN574vfMSb0twGlAiR8f4m1LL W2WcjA53Rk9pm10biTk27xiBwvvjnePWKYjLfqTBCWNbOyQ3Oab0+adY24YcH10SZHPN IhW2Cg5uZ7dgKRbEqKNI5cztvjVRl0pIJiPLaNVZOfy/3kvB0mB2g6XogMjSnUkmEF3B A7Bw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=Cb3p2QcN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ez23-20020a056402451700b004a1b489e80dsi3918882edb.54.2023.01.27.11.47.47; Fri, 27 Jan 2023 11:48:10 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=Cb3p2QcN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232311AbjA0Tpv (ORCPT + 99 others); Fri, 27 Jan 2023 14:45:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51106 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232371AbjA0ToU (ORCPT ); Fri, 27 Jan 2023 14:44:20 -0500 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DBED32ED53 for ; Fri, 27 Jan 2023 11:43:11 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-4bdeb1bbeafso65144387b3.4 for ; Fri, 27 Jan 2023 11:43:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=z2/pTj5C3cgEFiOJp2CeCn46OI8ibDfme0ccGGe2ykQ=; b=Cb3p2QcNsUnk+wL6/K6P4ED35dG3ZUcB734zcAsddvwr/PVSkrRYzepHFsLMU82wqr DCByjFxSWTXuXxKpPzrEyhOes5tD65SfDW58DVPGzZ4bl/GM62EXvV72BYyh+FkQftRL 9ik0iporDuduJD370bXV4KniJvgp40Hs/4xlWBjuzB4w/OEbzQnMEDOcEM26lHODokML 801aFmGgsonG3Ea3SKObTIkdYEG/WUj/siwNQ/Elnoh6/xVR3ypDahdEx2v1JUXFD+l6 cbwz8ftG3Ry5zhC62QLkQy0TB2bZZjts/vu8EdbF1X6krxdoifCb7qgLuiaEVqlGXvIi PdsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=z2/pTj5C3cgEFiOJp2CeCn46OI8ibDfme0ccGGe2ykQ=; b=bdCGFf6cMuLUD8Uq0vdVuagR1jKFPZaK89oZgrnVf+cdT806KX/aFulRurodZ05Wic ZklDK0jmmfqJUDy9eWGIauQ2QqqY4an1oDWRrwiHrqzpaf8jchuTh+2xojazDIv58kPB rK1UhTxJ7H/Kbogudyn2nD7Uqm6iPLCmieD/NLIO1qbrwRtbJV3Pav2EvSvs45Vf8cFl USWIEUwK+q2RmgRgzAgyRPK+4vkuEkIZBQkkiTus+V+9scNtStZZubxAySnUplfaSve9 2RMbFH7y2R4LI72cNrQzIPs3MMRT5PR0kInlbrdjhwjlzcle0Dsl6LTSVib0utmL/FT0 Hasg== X-Gm-Message-State: AFqh2krdyO6GovkVmIQPCfNaF2t0HY+847oOKszqr56nYc+9andaG4jn TnviBoMycSLuODwXZInvKArrBiffd8c= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a0d:f6c2:0:b0:4dc:1241:cc0a with SMTP id g185-20020a0df6c2000000b004dc1241cc0amr6165876ywf.499.1674848512603; Fri, 27 Jan 2023 11:41:52 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:53 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-17-surenb@google.com> Subject: [PATCH v2 16/33] mm/mmap: write-lock VMA before shrinking or expanding it From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756206350878305705?= X-GMAIL-MSGID: =?utf-8?q?1756206350878305705?= vma_expand and vma_shrink change VMA boundaries. Expansion might also result in freeing of an adjacent VMA. Write-lock affected VMAs to prevent concurrent page faults. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 6fff76334177..60038c24d836 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -674,6 +674,9 @@ int vma_expand(struct vma_iterator *vmi, struct vm_area_struct *vma, ret = dup_anon_vma(vma, next); if (ret) return ret; + + /* Lock the VMA before removing it */ + vma_start_write(next); } init_multi_vma_prep(&vp, vma, NULL, remove_next ? next : NULL, NULL); @@ -686,6 +689,7 @@ int vma_expand(struct vma_iterator *vmi, struct vm_area_struct *vma, if (vma_iter_prealloc(vmi)) goto nomem; + vma_start_write(vma); vma_adjust_trans_huge(vma, start, end, 0); /* VMA iterator points to previous, so set to start if necessary */ if (vma_iter_addr(vmi) != start) @@ -725,6 +729,7 @@ int vma_shrink(struct vma_iterator *vmi, struct vm_area_struct *vma, if (vma_iter_prealloc(vmi)) return -ENOMEM; + vma_start_write(vma); init_vma_prep(&vp, vma); vma_adjust_trans_huge(vma, start, end, 0); vma_prepare(&vp); From patchwork Fri Jan 27 19:40:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49635 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1011801wrn; Fri, 27 Jan 2023 11:46:03 -0800 (PST) X-Google-Smtp-Source: AK7set/POckXX4jvDpu0Qa5rygLfHQelZLg0pddih35vWq/qvkcoOEP4Pt0LR40FzUTGoQ33Qo1B X-Received: by 2002:a17:907:9692:b0:7c0:f719:838d with SMTP id hd18-20020a170907969200b007c0f719838dmr469690ejc.36.1674848763501; Fri, 27 Jan 2023 11:46:03 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674848763; cv=none; d=google.com; s=arc-20160816; b=LLKgXcDqSomfUBNYaq9gSlEe96VklQa9jC63l9rVo2NGklwhHoEb7cMX7MEfKWCMRX NKXt5s/AkiF/STNH3OS2vDXKpTY/D2PzJkNf2YzJHaQB1Lm0lL8oNMNIbiDiqvs/SYjB urLizmTmsBg8owpkg6DpYtCeN7ZClAN3/Iv0puclh4TYwrS0P+qLD9xENp8fECqORTym kC99RZpoa6aRzxep/4Lqi5mGIwOzYc/5Dg96wTHUzCAMtQFXgJF4M1QmLxaaSSb1hYW0 +qgqYFRXDp7tsKuOAi0mbH44f8zTveSr7VMQ2IGLABxwxDv+EkysPJIxSIM8BYFzK2Tb ttSg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=L4ezA/DdKPqYrQ5cS+Ke2gtcDFDEWQ5+mTqJ9joAzRk=; b=ePgPA85BMeWM0skjUj5s87p4UWZVFQ6F+4e9a8dDd/ISXpI2h9CyqpAfGkhu9EF4ez D/2OQjbojwuMeiZsAJ5Mem05R0PJhvQv5gqi1F+D69c4Q7n78iEq2JXD8I22l6YYKCYt 0DkL4qwfr/3SyAlDpuYRhpBthZZbnzw31G8yNMF3CpGY6u2ugaIpkRmXek9ysaqJS/fU /LT7eGkjEOQhXgyvecXhY8lfz79NcozBGjFriwacUUgI0u0VmrI8uNzhVC6xJC+SoxQV 5DUaTbKwNbwfyO6MPS/LtRA4rcWXYweh30W64cyzMAPZ1aFzCbF0CI76jUn/OKoJgGHM s+0Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=GKKENfMf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 16-20020a170906021000b0086f0864592dsi6262100ejd.321.2023.01.27.11.45.39; Fri, 27 Jan 2023 11:46:03 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=GKKENfMf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232487AbjA0Toi (ORCPT + 99 others); Fri, 27 Jan 2023 14:44:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51548 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232176AbjA0TnW (ORCPT ); Fri, 27 Jan 2023 14:43:22 -0500 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6C190227A3 for ; Fri, 27 Jan 2023 11:43:03 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-434eb7c6fa5so65876357b3.14 for ; Fri, 27 Jan 2023 11:43:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=L4ezA/DdKPqYrQ5cS+Ke2gtcDFDEWQ5+mTqJ9joAzRk=; b=GKKENfMfMG4N0pZCcl9OmPXOIcATJjfZsJmDxgzcwEK7KGqwCsad+AWtcOXvaoC0br qvO01w86MwzjSygGqi8PA6cojX9dXaSV7GQ7IPy4D+7hoAEyyoJoE4vESGbmz//KocjS XInPKVcNcxUU/7v9tYz8Eq3N+worhIjUrPeLT3iGbzfEm71MChhnFLrlat60IF4TyfFg cT/sx9cN16Dq7Csnqgd8AQwbf9+2MwGgrR42iD3xwMMpBpotxh+QKt5ySOsx98KoL0iq FNYiUMi+Hv+PIq3dZA5NAAwxArkgkUL/euE7T+pJZglO6ldDOiv8hb2KxyDrVWXCBL/Q MOaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=L4ezA/DdKPqYrQ5cS+Ke2gtcDFDEWQ5+mTqJ9joAzRk=; b=aJNtgNfBVDFVITTGk1sfpBqXPwGFJfeWRjHIr0C+EpkZJP/uUnMKjIrtlyoRQ8xbyE k/ejOf/PWYbitVr1vn1ucQyc7ah/yk8ISnqJQmvBy3jbwEyxyq52HgOYn1+1/ZaEhCyb WmAin9d4K0UFhPnzhY5VNinPxM1VV2FtLwIeu62nAopm574UaBAhZjn5RiAH8VEbli5r VU+TlTLLAfZNZdyh2dHrJY5ar4zluW4NNklkrhmiJlYLc1EVW0h+wLJpkOR2GqWtQWgA SR7SzoUKE3eE14v5dvf0janWtMkl8pSh9dzFA47bYaQ9Qy8FWQ6gOugclVOAG+qLIi4B 8+UA== X-Gm-Message-State: AFqh2kp3RUyDPUctKMITWf5kkPCIZ6Cq8ner6woMPoFnPKD7J0RnyEuG LwmHOzqea4N6+5jzPRL/PKwRNXHHi0w= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a25:3c9:0:b0:803:3696:c473 with SMTP id 192-20020a2503c9000000b008033696c473mr2168756ybd.224.1674848514947; Fri, 27 Jan 2023 11:41:54 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:54 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-18-surenb@google.com> Subject: [PATCH v2 17/33] mm/mremap: write-lock VMA while remapping it to a new address range From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, Laurent Dufour X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756206217437356610?= X-GMAIL-MSGID: =?utf-8?q?1756206217437356610?= Write-lock VMA as locked before copying it and when copy_vma produces a new VMA. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- mm/mmap.c | 1 + mm/mremap.c | 1 + 2 files changed, 2 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 60038c24d836..b3c247073aa0 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3202,6 +3202,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, get_file(new_vma->vm_file); if (new_vma->vm_ops && new_vma->vm_ops->open) new_vma->vm_ops->open(new_vma); + vma_start_write(new_vma); if (vma_link(mm, new_vma)) goto out_vma_link; *need_rmap_locks = false; diff --git a/mm/mremap.c b/mm/mremap.c index 411a85682b58..dd541e59edda 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -623,6 +623,7 @@ static unsigned long move_vma(struct vm_area_struct *vma, return -ENOMEM; } + vma_start_write(vma); new_pgoff = vma->vm_pgoff + ((old_addr - vma->vm_start) >> PAGE_SHIFT); new_vma = copy_vma(&vma, new_addr, new_len, new_pgoff, &need_rmap_locks); From patchwork Fri Jan 27 19:40:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49659 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1016604wrn; Fri, 27 Jan 2023 11:59:06 -0800 (PST) X-Google-Smtp-Source: AMrXdXtmKRFXhqKRI94BOu9J844fQUuwElFz5IsxuNe33u9qSZUIQVke69jmfbfzlt0ATnn2VUWM X-Received: by 2002:a17:902:ecc2:b0:194:df47:2b42 with SMTP id a2-20020a170902ecc200b00194df472b42mr37907892plh.48.1674849545254; Fri, 27 Jan 2023 11:59:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674849545; cv=none; d=google.com; s=arc-20160816; b=xLRTdMfaFaHIKIOoG2DJmZmRSjCUQAE5GJnA39TU/45fgR93UZu/MyEuQTV3hYoc4j 4DIJJwyNQeJxdaGrOPM4pjXWjnprU+MjhoX6tS0BGi2HbFoW7Y3rv0tPoMqqEh6cZwwd xDoE2NPsHyBexzjzs/bCOtsRwnLU17SsJZ1nuRX6UHVxEpJcNYbd+lQajK3zuDxJkgAc HXSkkHIijmkqEldHaW8qRRxY2OlBN45mGe0JzvnGMYU7HXrFAHvZIZHnJzRhCTfvJjX9 rFSSbW3hesy4mJXorK4YNdbgaRoZROK4xaRlIZDJL072LUMPbM7a/VHFufJZgt9ybwcK AwLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=vIuOo+WD7hhMM8dbfJD4mGHBD08/Cqf6l0kIi97gWpI=; b=a6/if67Ni6shyo7qK08RlKK4Imgvstqa+5z2V5zhW0yaIS5mkPnsJAhw08TQiYSLYj mwTHZm0cveyOkPPdA6SlFVBbgI8yuwJ8A8relA4to+RD9uZ1mBzcNHWOQQ6wsJN5zKGg 3oJnAE7jhrwepoj/cM+meuxMI4e1jD78gJ4mSw/6eQEsuBfExcAGdvEajgFnqMKrn2v/ xUT6O/H8mZSc27bVEZscy5XXgjCRphOkSKijTLrZrwMiE5S33Drrzy830PPFsGKi4//H 5RGrT1BMluqnbEBv2NFoBLqP0LlgzPzUYSJQpfsxUt6GwEDxKTJ+Lkzcd08E1aiX/F0B LD/A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=iI2a7Mep; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q13-20020a170902788d00b00194a9a78b2bsi4793242pll.68.2023.01.27.11.58.49; Fri, 27 Jan 2023 11:59:05 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=iI2a7Mep; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233098AbjA0T57 (ORCPT + 99 others); Fri, 27 Jan 2023 14:57:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37534 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229776AbjA0T5g (ORCPT ); Fri, 27 Jan 2023 14:57:36 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 25D824A20F for ; Fri, 27 Jan 2023 11:55:34 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id x188-20020a2531c5000000b00716de19d76bso6391663ybx.19 for ; Fri, 27 Jan 2023 11:55:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=vIuOo+WD7hhMM8dbfJD4mGHBD08/Cqf6l0kIi97gWpI=; b=iI2a7MepfOBNDqB7wu+/83gKbYOOOUW8grtwa3CiYxFM9JfwE6yM70Ne3Bb3nWcBVw Yu75uY4jTmGekedOn8ngeIwpB83dCuwgdGAmo3PP8/5H9WQZ6zUPSiR4iOLBbFN7215j t5fY8PD6baQLdLtkjh6u6ca2lGsT7fHu5oxzuRWhyEjLoCc6wJIYtxHPj4OPz1R/r154 +v9JTIHiwr1S1nQG1EzjVKOQzPqMVCb1Q8Qi6vYQuYU/8I9rd1kQnHGC5kdzCbnlOWk7 FPBIbkrQ5+p0qEBCYmhl8CWtOia+KC/VA/z2TgOLyA8KXzLYreAkUsa906GcFZG1qYyr SCmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=vIuOo+WD7hhMM8dbfJD4mGHBD08/Cqf6l0kIi97gWpI=; b=Q+bNhDyyt0TgmdfFWixxRubvRkV0JVtQuIRZEjoVFtEwPCokeMtGgJpAB6vGNCK/Iv auIwJVr7QM34ouwh9SgkIz8KlF/QxrZli5tXnMlmB3mt9amDsLQn7+28tYW9k240saMC Y6UGEWUIJkV5kUWltzkcXzVSFlEBIcTs1W/srBOc0AhMcM/U/PVYF3tk+XVwBnA/ySys azK1kA3l5vWVQJHi5/xPCnUwRbUuUo49L+EQESG4S7itbI9L/Fzr4OX7X+bJj1Ge90rF 2or7hcf5cC57gww/4lhBnUf2QGbC7tqK5tL/RZhDyJB8A1SM6/1B+DchGQtRL/WLERsN w2mw== X-Gm-Message-State: AO0yUKXdcn+owEL6f3C+DlxvoIA7Hw27I29aEq3pQbWRWffDX+HWuU+2 CjRZA5tm1PebCb6hrH0CZiUR8bgNZA4= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:3e15:0:b0:506:653c:db97 with SMTP id l21-20020a813e15000000b00506653cdb97mr1675918ywa.201.1674848517274; Fri, 27 Jan 2023 11:41:57 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:55 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-19-surenb@google.com> Subject: [PATCH v2 18/33] mm: write-lock VMAs before removing them from VMA tree From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756207036824069330?= X-GMAIL-MSGID: =?utf-8?q?1756207036824069330?= Write-locking VMAs before isolating them ensures that page fault handlers don't operate on isolated VMAs. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 1 + mm/nommu.c | 5 +++++ 2 files changed, 6 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index b3c247073aa0..5bdfd087b632 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2261,6 +2261,7 @@ int split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, static inline int munmap_sidetree(struct vm_area_struct *vma, struct ma_state *mas_detach) { + vma_start_write(vma); mas_set_range(mas_detach, vma->vm_start, vma->vm_end - 1); if (mas_store_gfp(mas_detach, vma, GFP_KERNEL)) return -ENOMEM; diff --git a/mm/nommu.c b/mm/nommu.c index 57ba243c6a37..2ab162d773e2 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -588,6 +588,7 @@ static int delete_vma_from_mm(struct vm_area_struct *vma) current->pid); return -ENOMEM; } + vma_start_write(vma); cleanup_vma_from_mm(vma); /* remove from the MM's tree and list */ @@ -1519,6 +1520,10 @@ void exit_mmap(struct mm_struct *mm) */ mmap_write_lock(mm); for_each_vma(vmi, vma) { + /* + * No need to lock VMA because this is the only mm user and no + * page fault handled can race with it. + */ cleanup_vma_from_mm(vma); delete_vma(mm, vma); cond_resched(); From patchwork Fri Jan 27 19:40:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49636 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1011940wrn; Fri, 27 Jan 2023 11:46:22 -0800 (PST) X-Google-Smtp-Source: AMrXdXs9w+1GzLwgtkY8kWaWKTnRLOXaVjEkOCb7FNrFOs1bly35VRWGIWgLW2BfBZCpE5wVO36t X-Received: by 2002:a17:906:94d0:b0:7c4:f0b5:fedc with SMTP id d16-20020a17090694d000b007c4f0b5fedcmr41919366ejy.65.1674848782608; Fri, 27 Jan 2023 11:46:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674848782; cv=none; d=google.com; s=arc-20160816; b=1DZ8CCXUeLwPxnrYwiK8VnhFEGGnjrrjjeo4x45YzNn60znnu6COaL1OyqFIhXIjE/ /YHmK9+wuNYSya6JW5RBPPovH2gr0B0RcYPS4L8qCrlhr/TL42d+swZ1A3ZuH1xzcy/K +J9tro7GXHCwA79UJRUK0F3k7dD8X9cwtN92nP19osJm6F50Rc5iNIHJ9FgHBQ6asH72 z5XkTdrSQ9ffhPFmT6Gn3YsCDJO5r7CpHrawu2ttSY76bInBKRhrLuYnO6BrFHLd/Ckq dyGqYN8WIIWy+x4aANSkpXuj26ZdK416iU3LLMZWp1C4mqy7Adb9RGzgnmKZ+U379B8H 81JQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=PbZvlTacNRMtTYgLZEFyk5N8KO4rHMcD13osXtjRKQc=; b=wP9Ks9TNT0OCC0WecsfRcpIWIU2av0fSPhSTimI6M3+4Hf+SuSs3unVv2WH7W5CjwY W5m0pUFrAcJmnPPulBwzd494jK/3Mw6zxlE55p9vJ/9mXQenl4yT8j4r2K+ijFDY50N5 D1zEVcHUDqvkiwqZiF+iSTqm10ECqZTs12puP2joAAq9vuQXWgh46m8K+BWBJU3xCvPZ ATAKu1ukKawjKKNFnqXxzFEK79G/YQyWhf+KHFrBWwu9Ceccb/n56GydsvgOAzo0plq2 uhgckUU8IoOZlRjkS23J8l9/5OUNsPYYMHPQtlDPzWbYM8ZssoKy/No2TSKHagfzJEit Eyrg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=dIXUHHJy; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f15-20020a1709063f4f00b00870ed0f7caasi5829497ejj.362.2023.01.27.11.45.58; Fri, 27 Jan 2023 11:46:22 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=dIXUHHJy; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232641AbjA0Toq (ORCPT + 99 others); Fri, 27 Jan 2023 14:44:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51764 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229908AbjA0TnY (ORCPT ); Fri, 27 Jan 2023 14:43:24 -0500 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 58F98241DB for ; Fri, 27 Jan 2023 11:43:04 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5066df312d7so65762327b3.0 for ; Fri, 27 Jan 2023 11:43:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=PbZvlTacNRMtTYgLZEFyk5N8KO4rHMcD13osXtjRKQc=; b=dIXUHHJyt3zldvcJ25OA8M/N2ekAkSadxrM4Vc86G1vAxHUN7FQNbSzlWzbH3EF3Ay Np/mC7lgd8yNrbdx+fJZdOiPs5Ax7Y+isaOSeA0W3slHJVunqRE0R5gvKOYJUJLOV6vy eusS7uxTBSUHthbBE/M+AcF029iuITRiXxkeDAE1bHTcC4qSxwHlMi32ImCbpO9h9EMV +wvjzOskJitBDhg128jcOLxlukATYCWhVPwB6m3ynKWMcyTvnD5SZLfuK2oGD5H+Rl4/ hAABUVAQozzvBCIG2lhb5atfzqUlxx9w7Rjl6rs3/T9zVK0hATJevSe9VLcJ9uJd0aXA HxXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PbZvlTacNRMtTYgLZEFyk5N8KO4rHMcD13osXtjRKQc=; b=OKM2kmpli9KDeUFRuIAXDrdxpZOJYp8dv2/63eL69ttgejgCS+EsZUlJA4VrOOM3UW BL6G0wRFMHsNk1RSDZpx7WDYC7LrH0rVKJWGcmsY8VHIWtwQpqJ4aLIfBI6cAG2v9bMy E0oR0cNENUDepWaos+sR3+hfXD5sKgvGkPXgluCn2DWtZKBJPnEXTqqYsHXgiDKUVeav UNRt56YcZS+ZLX0qqyBsqkdp+2x+u5ZHGvxD5XU2TLpy2GwtXE8SpSCX/FyQEOU/XfCn R3mmYNA/ge/p7O/+OeviQeNO2G+7vVqgektrIj5L4DEu38jg3hpfTEeflrqe1/RlAnBG kqKQ== X-Gm-Message-State: AFqh2ko0Ble8j1LpeqmY2aFD5MF7qmLw1UYTHcYIgDrHvynN3MT5BQrb jujQt567/JM1MpY/Fv0KdTbje95qbmo= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:cd4:0:b0:4dc:e929:40fa with SMTP id 203-20020a810cd4000000b004dce92940famr5633708ywm.331.1674848519684; Fri, 27 Jan 2023 11:41:59 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:56 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-20-surenb@google.com> Subject: [PATCH v2 19/33] mm: conditionally write-lock VMA in free_pgtables From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756206236957493929?= X-GMAIL-MSGID: =?utf-8?q?1756206236957493929?= Normally free_pgtables needs to lock affected VMAs except for the case when VMAs were isolated under VMA write-lock. munmap() does just that, isolating while holding appropriate locks and then downgrading mmap_lock and dropping per-VMA locks before freeing page tables. Add a parameter to free_pgtables for such scenario. Signed-off-by: Suren Baghdasaryan --- mm/internal.h | 2 +- mm/memory.c | 6 +++++- mm/mmap.c | 5 +++-- 3 files changed, 9 insertions(+), 4 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 90bb2078444c..52d7e9c2e58f 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -105,7 +105,7 @@ void folio_activate(struct folio *folio); void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, struct vm_area_struct *start_vma, unsigned long floor, - unsigned long ceiling); + unsigned long ceiling, bool mm_wr_locked); void pmd_install(struct mm_struct *mm, pmd_t *pmd, pgtable_t *pte); struct zap_details; diff --git a/mm/memory.c b/mm/memory.c index 7a04a1130ec1..d48c76e9fa57 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -348,7 +348,7 @@ void free_pgd_range(struct mmu_gather *tlb, void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, struct vm_area_struct *vma, unsigned long floor, - unsigned long ceiling) + unsigned long ceiling, bool mm_wr_locked) { MA_STATE(mas, mt, vma->vm_end, vma->vm_end); @@ -366,6 +366,8 @@ void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, * Hide vma from rmap and truncate_pagecache before freeing * pgtables */ + if (mm_wr_locked) + vma_start_write(vma); unlink_anon_vmas(vma); unlink_file_vma(vma); @@ -380,6 +382,8 @@ void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, && !is_vm_hugetlb_page(next)) { vma = next; next = mas_find(&mas, ceiling - 1); + if (mm_wr_locked) + vma_start_write(vma); unlink_anon_vmas(vma); unlink_file_vma(vma); } diff --git a/mm/mmap.c b/mm/mmap.c index 5bdfd087b632..57cb3a2ac9b1 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2157,7 +2157,8 @@ static void unmap_region(struct mm_struct *mm, struct maple_tree *mt, update_hiwater_rss(mm); unmap_vmas(&tlb, mt, vma, start, end, mm_wr_locked); free_pgtables(&tlb, mt, vma, prev ? prev->vm_end : FIRST_USER_ADDRESS, - next ? next->vm_start : USER_PGTABLES_CEILING); + next ? next->vm_start : USER_PGTABLES_CEILING, + mm_wr_locked); tlb_finish_mmu(&tlb); } @@ -3069,7 +3070,7 @@ void exit_mmap(struct mm_struct *mm) mmap_write_lock(mm); mt_clear_in_rcu(&mm->mm_mt); free_pgtables(&tlb, &mm->mm_mt, vma, FIRST_USER_ADDRESS, - USER_PGTABLES_CEILING); + USER_PGTABLES_CEILING, true); tlb_finish_mmu(&tlb); /* From patchwork Fri Jan 27 19:40:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49637 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1011978wrn; Fri, 27 Jan 2023 11:46:28 -0800 (PST) X-Google-Smtp-Source: AK7set/BNFsKKDpIFmw3/A8o3ChJm/LdSJdCP/Th3sObZdHMlu/VNOzQq1HUhy1S83POOVbNtQ49 X-Received: by 2002:aa7:de07:0:b0:4a2:1812:e062 with SMTP id h7-20020aa7de07000000b004a21812e062mr2234693edv.18.1674848788205; Fri, 27 Jan 2023 11:46:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674848788; cv=none; d=google.com; s=arc-20160816; b=hQCaofOX70CNqzk+D83Q7VFXskguzpwslNRbHDU2bsRnX0NQChaebT8lJaLR3KGMU2 OUekK4j6HG19YOTQUW4gfE9pfeC1JMWa/mQOfOB7FkOZjpepIwqiEpBy2Wv+TsjoX0s9 wKe5s7i83OlusbYBaYU8rQFf6eGCgdRswgalcYkkUO6IA1jxlNNGfrMHHrxEehadgGSl pgPF7xfjS9c6IukQ4TyLJkUcy5yasqd9dSuewyCidVKfrxgqr0VuBOdDzAG4SIqlc5D2 fy88KXcJEEfJqvGC7GFKsxrxNMbwkOpceAgCh59pog77/uVc2kEREW8WYXnRfrFWEx8s NEDw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=1VGgj1TmnjNrGwceq1Y9mwtgHdomU/ZJ3zMBBp7twvM=; b=xUb3JMRNTGpgPXg58lqmhmP8o6n4T8nRtT38ilMxttjPXgxjuIQpeUTbQuuP+M/cbw 8Hy9GooZ95UOhiLFb1OFyIxcThECw3tXIFMDGRJxvnNQUPXcxfSphLuVle1ju0iw7wuR gNCj0PBs7XaipSytd5Ya5pGbuafhIZQLmJ8EmnbOMrHk8wxCbemdUQUvgEw+gLpQhEWn 2+pfhTnSVOZTfUmOsXoUVw+zj4CprmE9mca2jhuCxMrytonPdk3jTEDJbY+j229lZWnj vzfXKJGadm/ImTyg/ozCXslCZxgYR+9913q1EJjDV+a574KVX6qqDiK2ACtryDuVk5nN PG3A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=kcxorgjD; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id x5-20020a05640225c500b004a0e1c290cdsi5660045edb.147.2023.01.27.11.46.03; Fri, 27 Jan 2023 11:46:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=kcxorgjD; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232714AbjA0Tov (ORCPT + 99 others); Fri, 27 Jan 2023 14:44:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52018 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232218AbjA0Tng (ORCPT ); Fri, 27 Jan 2023 14:43:36 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AC901241F3 for ; Fri, 27 Jan 2023 11:43:04 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id r8-20020a252b08000000b007b989d5e105so6377360ybr.11 for ; Fri, 27 Jan 2023 11:43:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=1VGgj1TmnjNrGwceq1Y9mwtgHdomU/ZJ3zMBBp7twvM=; b=kcxorgjDvieL4H0gsZ4o6ev4StdVYsVDA7GZVrIORQdjZKoFiGI1KoZQdzzvMshMrE snI0AzaYSvDLC9zEn2IBQumHO6CKLkrL/i3wJ9ueET6bm8oRn7QEPBX+BM7UODPO3hzZ U4MGRqXmViJMsS4HDNU6vnAmwplZNvnqe2D+qu3fGFBMwOJOM+QBc0aicDuQnfczJKX+ vPW/+j1Kiez+csRZc1jn4fVsV2fgBwTNZpRX4guJZVX8gsy3dFik2ggDGPcziixMOIIZ ZlSkjqSVG2I0FEWcD7z9JqxBzo/zAlN1m8/Qkt+nlr6qzU3SYltCmGn1Rp59psmpvaAD Fzjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=1VGgj1TmnjNrGwceq1Y9mwtgHdomU/ZJ3zMBBp7twvM=; b=3iK6K7ENcx0k/Repn0MSoWvvLTAkiaupoqzbJ2J0EsnnXLCAuzJqYfQQrRP6M7tcWS 1+duRey7zR0EK6d4M6MIZf4HwWIlZNJUmHOrefqK4n3Zz+AsNDIoUJKbehzVNPUnic+f 3GOCMoKT4NWQdH+wrdW5qF5ibSVguw3mP6gbZRUAr5loUpU2Tzq7B0zMguGvtOiEePq8 OtpZKFfu2GGuy0MJloCPFF123Xb80fcWttjRiXSEJPQSPFYoDO7TJj2NwsjhbZdlm5je OPmP9dPsnFRqpYPaVKenngglK5pMLJ+RalIVjr89XF0ICpTBB2d0tw2mjemdF3QrtSYH LITw== X-Gm-Message-State: AO0yUKXLeTr7jBoLd3GntH8blmAjJh47+3B5S/nKd1LE/QKrg0aZutaT NggF7seh9q3XZlgAWmAGbWxu5pS+uiY= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a0d:cb8b:0:b0:50b:1b97:f19a with SMTP id n133-20020a0dcb8b000000b0050b1b97f19amr434148ywd.164.1674848522027; Fri, 27 Jan 2023 11:42:02 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:57 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-21-surenb@google.com> Subject: [PATCH v2 20/33] mm/mmap: write-lock adjacent VMAs if they can grow into unmapped area From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756206242823622585?= X-GMAIL-MSGID: =?utf-8?q?1756206242823622585?= While unmapping VMAs, adjacent VMAs might be able to grow into the area being unmapped. In such cases write-lock adjacent VMAs to prevent this growth. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index 57cb3a2ac9b1..3baf218836bb 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2399,11 +2399,13 @@ do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, * down_read(mmap_lock) and collide with the VMA we are about to unmap. */ if (downgrade) { - if (next && (next->vm_flags & VM_GROWSDOWN)) + if (next && (next->vm_flags & VM_GROWSDOWN)) { + vma_start_write(next); downgrade = false; - else if (prev && (prev->vm_flags & VM_GROWSUP)) + } else if (prev && (prev->vm_flags & VM_GROWSUP)) { + vma_start_write(prev); downgrade = false; - else + } else mmap_write_downgrade(mm); } From patchwork Fri Jan 27 19:40:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49644 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1012472wrn; Fri, 27 Jan 2023 11:47:52 -0800 (PST) X-Google-Smtp-Source: AMrXdXtyRr6h1WTzEP4bimwkyb1AgrLF9kQ4HPqmGG7CeLxJOpaxvTakiqHVsLbCz8+ga9ewuoZE X-Received: by 2002:a17:907:971d:b0:870:d15a:c2dc with SMTP id jg29-20020a170907971d00b00870d15ac2dcmr56778013ejc.74.1674848872494; Fri, 27 Jan 2023 11:47:52 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674848872; cv=none; d=google.com; s=arc-20160816; b=VuU66SUWxiq5TZ7TlZwdszG6e+jAcF7bh8k+X6dBeDdpVT+xexzaPEhAou3dT2GTQ7 yc/wwM2+DY/48IQenTqdHBCv1j4cbm1qjTH2eKaF6HllOAt5oQHbPlytaZSFRvsvScX/ QVfhkrjqBOgL+bmfZaxK/vec3/d33kopBktFE6l+f8tNWrhwa0v8kOKKqIYEBkD8Wdb0 kOM/0yeQSh0wQhrgl3DVF9ZQavyWal0J6/qCcB3eLsFXpITy6W+vb7iJfCkz+aFf8Mdp pQu+8mdRYfaHNsDN3t6zfqfdioWmXv6lEDaFydA3N8SxSUny+S1V0Y0G5qTO8ZUjme4L EvlQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=Cy3b4YdYe1JheypyM1qEqlntx+LNPOiVtlvqoTpRTGk=; b=Omn9gg+j7k1MupAJU7sYR19OezbE41ghzPfLQI3ixbI8LYwYY36cwyNvbvBd+v+57F I0YDcCdSaZzBsl8n6z7bQ5exFUeidcIgMVtmE1Vp77HXxJHyRS1lD2751GFVR9WUPXYf VKsq1rM0itFj4AvQwRbPDCH4x9rLRp4LNtW6UHYK2uywCA+xihQCBeDc7k1fsde0yCt1 ITUoOxIveOvWGXk0w46crxjQLlx5kwzI1kkJpztiksnLXo9ffsLxgwwQyhx9QVfRKiFI 8feSiRc9nH75g/jFDAmEwZKpaD91jnEHnxM0I6w9yvaXUKGUjLFgGtFTBGHHi7BBwfC0 j0Mg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=oI0LJkxW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id zu5-20020a17090708c500b0087862d0292bsi5481407ejb.115.2023.01.27.11.47.29; Fri, 27 Jan 2023 11:47:52 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=oI0LJkxW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231962AbjA0TpU (ORCPT + 99 others); Fri, 27 Jan 2023 14:45:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51566 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232287AbjA0Tns (ORCPT ); Fri, 27 Jan 2023 14:43:48 -0500 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8254D2FCC4 for ; Fri, 27 Jan 2023 11:43:05 -0800 (PST) Received: by mail-pl1-x649.google.com with SMTP id z10-20020a170902ccca00b001898329db72so3335005ple.21 for ; Fri, 27 Jan 2023 11:43:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Cy3b4YdYe1JheypyM1qEqlntx+LNPOiVtlvqoTpRTGk=; b=oI0LJkxWJWb4FwrADuA4kk3Z1VuG/PqmB+wfudzVRf1FutZSssy7j1ZDpJRsJeC2x6 AK+2fBcgvA6Z1yebvss3n+ODruOq7UlamXlS7DfsbRapPdytBdlVYqH5VQCwBS0bW5Cs 14NyRgqd5c0dS96F3WtaSc6UYIsY2Q866sZsgDc9kakHc8RR5OiJX50DOvF/naV3sW35 K5NQwDrUGeBVt7kutcZKVO5xLo4MSKoGaFWDyWGZ3caTDR6ZBpyZR8N/O+LnlyPsEjZa UEkxm7VSV5agQqzuF3YgYMg0fZBeMdeXvS4WgRk9vqB/9OKyGK8SEZXO7u7cVN0V9JOR wUhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Cy3b4YdYe1JheypyM1qEqlntx+LNPOiVtlvqoTpRTGk=; b=nH0a9Tz1hFZgirjd/YDxR0lQAY6YbZoxaOIZcQiFuxjz9z6rP/0B6fLXepvpCVKxFS lMualyArBM3DCZ/JaMTB+CYUHEDYNxjCPyQXsp4O1ZvJddq9nb9AhjdNsyk4ZvB0OaY0 KikJn5wc9cV+nqkK5wZ37Gv5mOXeZJA3amaqLoIJFlczkIE3kg/5D36YXMG6zzAr7MYw OIISF59YAnZUX01QKZMdvfnp2HJGuwQbooaC/L/MI6fJYdcmhOdVUgeMEYeKOBTLyPhw 6Ked58SwVaMC+thfsr+k2kXN059RdymwtjfJbVRbDMGUsam1IxI/G9Fe+omduRw6veTh x0NA== X-Gm-Message-State: AO0yUKUu6SI5c84gN2C4n7bggG9LLXw6mskn9v40zGwF3RRmIqTQ6UDz yghLAPPUryuIOKvnNjmGsQDJveHOjI0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a63:ba1a:0:b0:4dc:3b99:6f67 with SMTP id k26-20020a63ba1a000000b004dc3b996f67mr1091417pgf.47.1674848524282; Fri, 27 Jan 2023 11:42:04 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:58 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-22-surenb@google.com> Subject: [PATCH v2 21/33] kernel/fork: assert no VMA readers during its destruction From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756206331562003083?= X-GMAIL-MSGID: =?utf-8?q?1756206331562003083?= Assert there are no holders of VMA lock for reading when it is about to be destroyed. Signed-off-by: Suren Baghdasaryan --- kernel/fork.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/fork.c b/kernel/fork.c index 9141427a98b2..a08cc0e2bfde 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -491,6 +491,9 @@ static void vm_area_free_rcu_cb(struct rcu_head *head) { struct vm_area_struct *vma = container_of(head, struct vm_area_struct, vm_rcu); + + /* The vma should not be locked while being destroyed. */ + VM_BUG_ON_VMA(rwsem_is_locked(&vma->lock), vma); __vm_area_free(vma); } #endif From patchwork Fri Jan 27 19:40:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49645 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1012572wrn; Fri, 27 Jan 2023 11:48:09 -0800 (PST) X-Google-Smtp-Source: AK7set9wVzQNDmRnmthrUXQ269lAtIwBCg4pL2q0Bn31arLhnyPQr4i2An0oQPVT76QHd9dkrXYv X-Received: by 2002:a17:907:a80d:b0:879:ab3:93d1 with SMTP id vo13-20020a170907a80d00b008790ab393d1mr6205421ejc.4.1674848889638; Fri, 27 Jan 2023 11:48:09 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674848889; cv=none; d=google.com; s=arc-20160816; b=WHLvSn8Df569k/2fCIBR08BJ0k8Rfcz71aX3HGJAQHwpU0iPipN/T+Xk0rMgbPD1/9 3LT+3AyrWvyGGCNfA1NkdnCTa1cvSa/UzivDGfVbJncds+Mfgmk0G53uuRGIKBDoaPgQ W07wE5yJAeIAh38v1qP9QWk6hTzmhVBpG+0opCSYt3oXhX6uGo2Ta17KecZ8KjZpziHC 6dRJ2e/wqCqj1U8zzB2H6j0m7vftbNywc5G+Xa6oEOCJA7B5ZC6qv7jgsYtQRedWPKTg KPnpJKgfRxiiFMnZslTwUsLuZFwhxhhLvos1jqSd9PcNKWfCvR8cZFinzPzOUBUyvxik lCRw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=mMW5WtXlwQRVw5jLv0uMPkgP2YoyRT4XQf8L1G2g0sM=; b=hQVyBx2LPU7DY2kgYjUSKAWWW1CreCnDoMe+wWzoFgJoSZ0z+Cq5Gh6pI2MOSCVQCB gEg0rAYgiDXTE0tah+pDjrysTu7004QL6KeOXDvkxwtRq+IrHZo3TjnXm01P6XcBhtKa WY74AIWdTbS1JBaD8pAtryCfaE1sYRzFrKUtqbZj/J4tcpWQfhgjBV/OkP6ZKExN7TfU L6S3rSFLPaTwUA0VHmzwr4a+2tRkoY0n34JnSNlXMXaoTUf9q87fBwDvqaKpJvUozndW lLzX0MwCXvPtl1h+rlyLXR1l+kqYATf2CDBqI5foYUJ/ihti2cXRcfwvByrl3JhGy7CD oN4Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=mdQhCODm; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id e18-20020a170906749200b007877f3132d9si7058267ejl.438.2023.01.27.11.47.45; Fri, 27 Jan 2023 11:48:09 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=mdQhCODm; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232774AbjA0To4 (ORCPT + 99 others); Fri, 27 Jan 2023 14:44:56 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51564 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232244AbjA0Tng (ORCPT ); Fri, 27 Jan 2023 14:43:36 -0500 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6DECC2DE4B for ; Fri, 27 Jan 2023 11:43:05 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-4c8e781bc0aso66308987b3.22 for ; Fri, 27 Jan 2023 11:43:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=mMW5WtXlwQRVw5jLv0uMPkgP2YoyRT4XQf8L1G2g0sM=; b=mdQhCODmER/WcuXnOwjGOPGsluXP611R3K8yAAqGfZO+eojPR0iLkDWsNYoGq20Pxc pLGndC0gNc0k/+qBk7J/uFwYPbCjsW7XmlHDcZCSFndQyDIXmYjY6yxKamOAoeP6UD5R 10GuF5qfONyUFHlhJRc27yht8CtL7+xZMg96kBN6doFn/8DSebIrGsIvYM1XfrVRdrIT Vzd4VicNtC2Hx0ebeUSEC751adwD7ZH4dP9Tg/rHSx32WXBUDV2TcmMYgTc4y3CEq8j+ 2IuM945ICEGJ6/3bquYB/9nBx6UvS9o2Tl6iob7NJEFRIs2YxCbcwKtUsh6O7/MdtiEH lFnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=mMW5WtXlwQRVw5jLv0uMPkgP2YoyRT4XQf8L1G2g0sM=; b=YIac+PxJGZptkH5hsFdPViXAs5WSWvfp2/AQyEb+fMGNv9+3L3/+6EgX+YxG1xIF/H 0zm5S457OYuR9+PXprRr0Gf75fVyCu/0QQRdXkWI0StOn7TEB8Qou/ncjlyt8JTwVERB uDfE7Xzv2HpeCfUyxQ90K5UDTGZpLXEueCNtrpgr8kcwnBFOdcdmunv9smmna4EbfaqL 1zPpCGX6pGnJzRSYkFbIbHIHqFtSobDA9dxhZM4byM1nGOAJlMXxymLKEnGFf+U+5zi0 MGidSYdSrIhMRE/WIHKrnc/Us1tgO+iGGI0Id7h96AHkTnFrrDliOUHVNW5JufhiI8PI N8Fw== X-Gm-Message-State: AFqh2kpOoiFZo+llAgVkjhK3CHgBN1G5zbNvyIFbn37Xyog9Z4akpYvn St41weWyf/OgjepfEUBWZUNw9W+CmH0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a25:1fd4:0:b0:7d4:712e:2852 with SMTP id f203-20020a251fd4000000b007d4712e2852mr4289060ybf.628.1674848526673; Fri, 27 Jan 2023 11:42:06 -0800 (PST) Date: Fri, 27 Jan 2023 11:40:59 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-23-surenb@google.com> Subject: [PATCH v2 22/33] mm/mmap: prevent pagefault handler from racing with mmu_notifier registration From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756206349082536477?= X-GMAIL-MSGID: =?utf-8?q?1756206349082536477?= Page fault handlers might need to fire MMU notifications while a new notifier is being registered. Modify mm_take_all_locks to write-lock all VMAs and prevent this race with page fault handlers that would hold VMA locks. VMAs are locked before i_mmap_rwsem and anon_vma to keep the same locking order as in page fault handlers. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 3baf218836bb..3d0cfbc92745 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3501,6 +3501,7 @@ static void vm_lock_mapping(struct mm_struct *mm, struct address_space *mapping) * of mm/rmap.c: * - all hugetlbfs_i_mmap_rwsem_key locks (aka mapping->i_mmap_rwsem for * hugetlb mapping); + * - all vmas marked locked * - all i_mmap_rwsem locks; * - all anon_vma->rwseml * @@ -3523,6 +3524,13 @@ int mm_take_all_locks(struct mm_struct *mm) mutex_lock(&mm_all_locks_mutex); + mas_for_each(&mas, vma, ULONG_MAX) { + if (signal_pending(current)) + goto out_unlock; + vma_start_write(vma); + } + + mas_set(&mas, 0); mas_for_each(&mas, vma, ULONG_MAX) { if (signal_pending(current)) goto out_unlock; @@ -3612,6 +3620,7 @@ void mm_drop_all_locks(struct mm_struct *mm) if (vma->vm_file && vma->vm_file->f_mapping) vm_unlock_mapping(vma->vm_file->f_mapping); } + vma_end_write_all(mm); mutex_unlock(&mm_all_locks_mutex); } From patchwork Fri Jan 27 19:41:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49638 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1012155wrn; Fri, 27 Jan 2023 11:46:54 -0800 (PST) X-Google-Smtp-Source: AK7set/2QPV9R+02OUgUWX3U6sgKDUFevFM/OMKfZ2fxumPhAWc9rOgsiYcu3kMxxSUYjjlIl6UU X-Received: by 2002:a17:907:7293:b0:880:a42d:dfb4 with SMTP id dt19-20020a170907729300b00880a42ddfb4mr849473ejc.16.1674848814604; Fri, 27 Jan 2023 11:46:54 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674848814; cv=none; d=google.com; s=arc-20160816; b=Q2HAj6uQQEPH3ZU4KX7Y1Jqj2vaOD69o4GvlHDk+t8mkoEgOcjqiUSIRYDk5H+KIo4 sqErlBgyxhPjyyVF2bJByZixv+wqcrGZztp09JqAsNqiZDEE3wrNRDTzQNXzn4R1dyHc A+16Vb6NCvZObADGDPKsfPl/Q4F/qJfPo8IzV/LTjGdv+5acHh82FgNbwIqQH+khxNH5 tNewF04Ksp9POYeX7hLnrj+Jw/RjxKhzhSC/vRmas3UzDLNFxiqxDj5jI3MmWbIoroSZ kvGSi6kH6zcC0cuXsXySRa+eYoIpJjzo9J4i51eoYxbVE0uV0ZOUAgvvAI0jAnCI8EJ1 ohuA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=5PvsAEkfbXSvdKN+lTJ9M9+vqfLQVBlihpxSKUw/t/s=; b=0wYCJaCD5P1Sike/PIuqNAYbJjLzhSos29J+KeD6tOkQQSxi3Nc1skl4ldmm7oBp1s tu4PBE8V6xD3E5Qp+steXDee1nav0Z3nGl3QO1mMiG9Gax10mFh7ixej6J477G7ztTg1 +Qj1UxhwyoU+k2MHD77X+E6jDmbp42IFZVFsqFqEIwxoLCxDFvRpacf8iO0fbge3DZ16 PmXbPhQacZ9C6j4qyqOjsMgtLPbqRrmn06oC6W+QhcgTgcaCJ0XDMxwSxRel8eHgnMKB CgA6PQkVaHzuKyty4Q9W+W64nyEA43GxF3kKTJ/xQYxtl08DICla9su3AbBVCaUjwUHf FyoQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=kp92vFUO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 15-20020a170906208f00b00871a4e8d7bbsi6112157ejq.268.2023.01.27.11.46.30; Fri, 27 Jan 2023 11:46:54 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=kp92vFUO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229510AbjA0TpZ (ORCPT + 99 others); Fri, 27 Jan 2023 14:45:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50640 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229828AbjA0ToC (ORCPT ); Fri, 27 Jan 2023 14:44:02 -0500 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C2CDB1E1FF for ; Fri, 27 Jan 2023 11:43:06 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-4b34cf67fb6so65252337b3.6 for ; Fri, 27 Jan 2023 11:43:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=5PvsAEkfbXSvdKN+lTJ9M9+vqfLQVBlihpxSKUw/t/s=; b=kp92vFUO2N4rCYUenom2PgB/dU7qqaylHAcH6HVVMYT5QMpL/oqQ/DdOm4bVNlYLNw 9rNSkse+dfgOJKwj09yKdpzV2qc7upA8mjHmEpxt3RKsaa3pDRks1QGh3Nb2hG4bp/jU 3eSArvNsdqtxNb6zpVi6gl2vf0iZr8y+PtaslVXMeV9Pu9xS/EmrXvCiajKJa9iCZe4S LpK/JO6vbmew4JxbSzbZQyffZL4sOt9sBlPWbT9WmPY7+pp+Ff0i+9BT7l4WdUaIrrgA btwhF2XkX8IUP7VpoyJ9OAYm06ddrZftAFhV0gBzofVRmYYEJ8dT0O9gMwk7XMh0b2Ct qw3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=5PvsAEkfbXSvdKN+lTJ9M9+vqfLQVBlihpxSKUw/t/s=; b=XqblgBp0ZkG/sgPug/J34bMEfsbtHxL6yOVv+f7o/BSbKodPd6q7cCfA8u461LY5Pv dlK2okGts7uCHAyqMC96o+jPafmSkVuCMIsP65hwWTj08ZfNYWALbmOqYiy1bBRMkQFk W1WRaPTar3im9hcu5qFc4bRz9/S+068dUnpH18sk5TuaRQdc9aWhZwADGBSbGcbUj5ii UV4NGTciJVwmi9KsnDHDS8AxetTQ1AsJhntQfOcQpG12y1r+3am15vYq9gaPJzVouu9O ijiY8n7styqBu+E/wXo8t9aD6owrLAttN8XZ6sQ3xeSZD/vWfnOr0ogsBJ2nkhve/QG+ D2Uw== X-Gm-Message-State: AO0yUKWye3Is3CNTLv9r7Ip5yfYI4YB4Ybd6JRFHC6sCBuZVWc10Su/T JS1vnUejk8RzDriL5IK+Q+EjBwebZN4= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:6d2:0:b0:50e:e91d:f47 with SMTP id 201-20020a8106d2000000b0050ee91d0f47mr27189ywg.61.1674848529047; Fri, 27 Jan 2023 11:42:09 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:00 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-24-surenb@google.com> Subject: [PATCH v2 23/33] mm: introduce lock_vma_under_rcu to be used from arch-specific code From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756206271081503441?= X-GMAIL-MSGID: =?utf-8?q?1756206271081503441?= Introduce lock_vma_under_rcu function to lookup and lock a VMA during page fault handling. When VMA is not found, can't be locked or changes after being locked, the function returns NULL. The lookup is performed under RCU protection to prevent the found VMA from being destroyed before the VMA lock is acquired. VMA lock statistics are updated according to the results. For now only anonymous VMAs can be searched this way. In other cases the function returns NULL. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 3 +++ mm/memory.c | 51 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 66dca140695e..fa2b9d6e665e 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -687,6 +687,9 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma) VM_BUG_ON_VMA(vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); } +struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, + unsigned long address); + #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_init_lock(struct vm_area_struct *vma) {} diff --git a/mm/memory.c b/mm/memory.c index d48c76e9fa57..5568fcb0a46b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5222,6 +5222,57 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, } EXPORT_SYMBOL_GPL(handle_mm_fault); +#ifdef CONFIG_PER_VMA_LOCK +/* + * Lookup and lock a VMA under RCU protection. Returned VMA is guaranteed to be + * stable and not isolated. If the VMA is not found or is being modified the + * function returns NULL. + */ +struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, + unsigned long address) +{ + MA_STATE(mas, &mm->mm_mt, address, address); + struct vm_area_struct *vma, *validate; + + rcu_read_lock(); + vma = mas_walk(&mas); +retry: + if (!vma) + goto inval; + + /* Only anonymous vmas are supported for now */ + if (!vma_is_anonymous(vma)) + goto inval; + + if (!vma_start_read(vma)) + goto inval; + + /* Check since vm_start/vm_end might change before we lock the VMA */ + if (unlikely(address < vma->vm_start || address >= vma->vm_end)) { + vma_end_read(vma); + goto inval; + } + + /* Check if the VMA got isolated after we found it */ + mas.index = address; + validate = mas_walk(&mas); + if (validate != vma) { + vma_end_read(vma); + count_vm_vma_lock_event(VMA_LOCK_MISS); + /* The area was replaced with another one. */ + vma = validate; + goto retry; + } + + rcu_read_unlock(); + return vma; +inval: + rcu_read_unlock(); + count_vm_vma_lock_event(VMA_LOCK_ABORT); + return NULL; +} +#endif /* CONFIG_PER_VMA_LOCK */ + #ifndef __PAGETABLE_P4D_FOLDED /* * Allocate p4d page table. From patchwork Fri Jan 27 19:41:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49643 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1012368wrn; Fri, 27 Jan 2023 11:47:36 -0800 (PST) X-Google-Smtp-Source: AK7set+0nCWC+ctSh5U8L/xYcOPRspKryDWC8oZH8/HJDvwRwr4HATbA2dmxeHWfC2iXhTGQHfNF X-Received: by 2002:a05:6402:4504:b0:4a2:1d08:283b with SMTP id ez4-20020a056402450400b004a21d08283bmr538762edb.11.1674848856394; Fri, 27 Jan 2023 11:47:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674848856; cv=none; d=google.com; s=arc-20160816; b=Z3IGtZdhwyASf1TzmVjvUmLS/CBwebbtONAlTdSLbsKjiwgBfGS0zFpDZPQl81q1XX FMXB1VZRX17iQkds3Sz01CzFXNxV+bdTHNoZS8ZFyokS3huuQRcWpQMBJeg8Bc/SFG3K FEEquz3tQQ6eVhjFHTf5sbdKnl7t3Pkqeq4n4JLg6AOqaxpO8WRTUJgwGsiLb5ee5xpq ZKDpy/W4IC0YNw6gdgE4JXlj+5ABj9Tt69mTge0uGqIj4ZuReemI++9mgtYiNH3uzu01 uoHfo5l4bHD7QlMjJZa9xCUBlN9WG0qmao/SrzAGdjn3n1ndYVlnDkYU+5AxXr2K38sZ 4c2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=gSaEMik01Wf1+Bdt2RmdUMjgYY7lDp9hLxJUYqXXyyM=; b=wYTys9pFKPiNmEpcOCLqgastgKlBpDD+Qy4jltC+Mx7u6uqk+x2r/lzyXHt7DK3AUN +H0OsD3pn54j36A2AwUUCMlQdznkqMR0OhXrHzfF07eosD04Rhkbu2Vu2QGvuQvFHYi7 oUDNVxe19kt2ldGxpA7mY2Gg4zaR72N2oMmfYGQSablGmtKRB+Js53D2kv/e1LUW6w7w idNhQOYSXuS5VUaTGSNQ1PC6Q5IV5vNDEB6oimZjz+/uNOAfdoR9yBqI8tk2sroy+GFK 5eDhxVOVPNurIZUrXGHiiTxe6beuMJXi569c2xNJzECscKfKUoXGu0Dzvg4EaLUv4PJn pyyg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=rcBVm8w9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 3-20020a170906208300b0086e4bf68769si6457801ejq.716.2023.01.27.11.47.13; Fri, 27 Jan 2023 11:47:36 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=rcBVm8w9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232174AbjA0Tpq (ORCPT + 99 others); Fri, 27 Jan 2023 14:45:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51668 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232372AbjA0ToU (ORCPT ); Fri, 27 Jan 2023 14:44:20 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5E5BF53B0C for ; Fri, 27 Jan 2023 11:43:13 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id d21-20020a25add5000000b007ff8112e8b8so6430225ybe.17 for ; Fri, 27 Jan 2023 11:43:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=gSaEMik01Wf1+Bdt2RmdUMjgYY7lDp9hLxJUYqXXyyM=; b=rcBVm8w95my/f0ebkBHOBSdk/g3dXIBnrvyKhE+0lfp8SBBL9UDohndGfm9prz69Nb 1VsGdk5hVjbZdd99cfs1kRzqndkhM72YtTrleRtcoKwDWYV7T956W4hi9P14DNVnfcTE 4dzd8voX/MwzX+niWcYh6zC2gtPII1RsQu2W+o1SzbH/VUzyZkZ+K0mbi6bCuvIO2r/9 q8JUw2FMeYJjbLjibdrizaV+CI77RGRlLfF03fxNpzuIRE1ymkIpYCAH7a7Vh5p4LTWM djOGRW9GCrBrxw2Z+lpBSKPiwjL/b62fESDSi0yRTpPNMfCaBkk0gvqmlpNjM1674+ZN MkCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gSaEMik01Wf1+Bdt2RmdUMjgYY7lDp9hLxJUYqXXyyM=; b=r15GmN1poUZ4dQgu9keNOfHmA9pK8DE5sJA1vaiMC5qTL247YoHvDo7oijSFx/35bd JZp4584jIVWImblKL97t9Yurl7aJFXrxohImUMRERGgABrzmPeGakcBKkd3BfR8AoF/Q 1WTcHq7Kv9pRfr1hAErsHoMhrGQrTQHoT75UaPzIjwrCEwcjTM+5K8uNVyeD1iI/6wIc w7yPO1JSxH7hXSiEwz4pSRzTpNLH0Vmr8kohycaCya4GS7jTDWoURVmIy842vrj7AIG0 GRv4sO29D9bM2ThxDQf9xC+DH4EtAqFtXoonRV3XHrEQ7pKOwheMbmCwr7kSUVXoz1Pn ydmg== X-Gm-Message-State: AO0yUKX0Pkwt6LNpxFwq8GsuuFWMUovf3ja40//j+NJeT2eRrZTWL95I 4KXGpjQvMO3HClo9LM4f0VhRaI/EMY0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a0d:d8d5:0:b0:508:1348:3e54 with SMTP id a204-20020a0dd8d5000000b0050813483e54mr1003857ywe.440.1674848531551; Fri, 27 Jan 2023 11:42:11 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:01 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-25-surenb@google.com> Subject: [PATCH v2 24/33] mm: fall back to mmap_lock if vma->anon_vma is not yet set From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756206314427185644?= X-GMAIL-MSGID: =?utf-8?q?1756206314427185644?= When vma->anon_vma is not set, page fault handler will set it by either reusing anon_vma of an adjacent VMA if VMAs are compatible or by allocating a new one. find_mergeable_anon_vma() walks VMA tree to find a compatible adjacent VMA and that requires not only the faulting VMA to be stable but also the tree structure and other VMAs inside that tree. Therefore locking just the faulting VMA is not enough for this search. Fall back to taking mmap_lock when vma->anon_vma is not set. This situation happens only on the first page fault and should not affect overall performance. Signed-off-by: Suren Baghdasaryan --- mm/memory.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 5568fcb0a46b..593548f24007 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5244,6 +5244,10 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, if (!vma_is_anonymous(vma)) goto inval; + /* find_mergeable_anon_vma uses adjacent vmas which are not locked */ + if (!vma->anon_vma) + goto inval; + if (!vma_start_read(vma)) goto inval; From patchwork Fri Jan 27 19:41:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49641 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1012292wrn; Fri, 27 Jan 2023 11:47:21 -0800 (PST) X-Google-Smtp-Source: AK7set9/EMtqomAFxEtsnphFbgw/jfpIkcoCicfElcijul9iCsqPZFff3XI2MEeyRHhLVxrzDLT2 X-Received: by 2002:a17:906:4999:b0:87d:f1f9:a2fb with SMTP id p25-20020a170906499900b0087df1f9a2fbmr1704423eju.29.1674848841754; Fri, 27 Jan 2023 11:47:21 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674848841; cv=none; d=google.com; s=arc-20160816; b=v1kf4xB0Fq3cnYyt+hIMayKFQuwlAk/UgZFuGK4p5mgJdwDsTlaBOyjfopGm1Amy6V I7XJK9tNrmLQI/nXqx0DzHiOz9zJGSl0bIl/XXmQxmojlQDEDzz6Vw6qCVGbZjafcBJT 9JyV1TBRW6ATQ5+igNK0P9F2rbqy/2cvxogZi1Q1LNsCSEzzN4PlM1rFSkILKf53wrOX 30Z9ECuTSJDbrC3RxixfgyN78nUqEVOgqcoRxIqORtkCDBnB3ij4DBRcFN6mh33byDfD 6jj8eGo1jSHPIxODdwHs5+oIZT3b/zYV7M4DYs1U9eB/5o7KCkl+9k5BsH0OxbPkOn9t hMtQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=2v/Evza9dkU4Ne98Lm6HAWxL8/YazIwD41pmx7joRkE=; b=zzyt4/4i5Ih6qZt9KIS0N16zMKZwUGj3rC2xNWAA5aAvtH4G+b8CxJH7UsYr/ILgCN b7acpKiPHEClRH9QEC+ihcHsO48NKM5yV08r6VFPRVJzlQ0qVZDJUD7y6VYBB6c05Wtj bQ6qXelwWdt+t8ozhma8o1Ht+MxTWypu8cL+hKID+eMRpLIiGWRnV+G0BDa9BHJmCo8z K2klPsHOMDW39Mza885LqcPsLkr4BtYEQtgHatJR4odt900l9CAMq8P2qiPtl1/3hVrp lWGIxT5nPU/9aRe8e5ML1AWMvl/3/YwjnHkNVUhqAYxaLy/eiczv8EbL91wzCg7pPG88 sUZA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=mLwUPJ0n; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id j12-20020a170906104c00b0086a76383bd5si5958943ejj.650.2023.01.27.11.46.58; Fri, 27 Jan 2023 11:47:21 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=mLwUPJ0n; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232062AbjA0Tpj (ORCPT + 99 others); Fri, 27 Jan 2023 14:45:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51634 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232096AbjA0ToH (ORCPT ); Fri, 27 Jan 2023 14:44:07 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E399459249 for ; Fri, 27 Jan 2023 11:43:09 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id x8-20020a25e008000000b007cb4f1e3e57so1753144ybg.8 for ; Fri, 27 Jan 2023 11:43:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=2v/Evza9dkU4Ne98Lm6HAWxL8/YazIwD41pmx7joRkE=; b=mLwUPJ0nc0ulmmwFoyR9amwtEYn4GV75pogWN7ijAK31MqrZCBgfhijsONjhepzopP 2wYY2SkQatHOd+vKpDSxYxHFT1IvhzZTwDWuTcrFWygojUbcZe7xPlEiO9BQnq0w9fKe z4cHjJJqSwoele+PetnaYF8lewRv+be+h1Bgt+SNNsmNqcyEHhiyYbNRieovHRIQnY7Q KgzOHSrbk4QlZX3GRmHUucd8wrsetfG2ey7u2AbUsjqg+O7jZuWXERmenSwIYYomTRZT 0AcxNZ8Sif0L8fvqzJHXb63zST/Gg9+vMF9oOa1KQZBohhw7ZSFfQ9eLw7FWbw/nE4Qf wodg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2v/Evza9dkU4Ne98Lm6HAWxL8/YazIwD41pmx7joRkE=; b=QBkDh/fgibMqfPYkzOeT4m1VgVC0juOZbAcCoOZ57g5udLbkyN890b1oCsdQAxOM3f gTddq7va9/XEZLpWB0rvdUzsS1Ez7x6STtZbhOs6vp5q1Z5Tw2k4gEaJ4Y5rs2Lvd04V Gqzf5TK47TTQ0C7fXMm/2lyz87dWNrgkF8Q4mSNgF895XWd+mrilrNgU77Mgvkelnrx+ Gcam0vhfYHHtRkPHRP8TUUxeD5zme/V42WJAtm7R8CvetJ1l9zWBc9g80X1RuPo1udEq Z+RSyOSXRn4Q48tvMDOb73cjBfdHRveDTo3vFwDGFwR8uaNC6YqRuvBDXkXse3Yf09LQ Izsw== X-Gm-Message-State: AO0yUKWk9EJ52y4Xbqc+d5p7fidh16VmtoDUmMa4x23EsKdkEbGIW74d 2SSj60s/svTQmHfShakIYk8cjkmdSS0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:b2c6:0:b0:509:5b20:ff22 with SMTP id q189-20020a81b2c6000000b005095b20ff22mr917640ywh.328.1674848534136; Fri, 27 Jan 2023 11:42:14 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:02 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-26-surenb@google.com> Subject: [PATCH v2 25/33] mm: add FAULT_FLAG_VMA_LOCK flag From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, Laurent Dufour X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756206299490318853?= X-GMAIL-MSGID: =?utf-8?q?1756206299490318853?= Add a new flag to distinguish page faults handled under protection of per-vma lock. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- include/linux/mm.h | 3 ++- include/linux/mm_types.h | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index fa2b9d6e665e..a6de58bb40c7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -472,7 +472,8 @@ static inline bool fault_flag_allow_retry_first(enum fault_flag flags) { FAULT_FLAG_USER, "USER" }, \ { FAULT_FLAG_REMOTE, "REMOTE" }, \ { FAULT_FLAG_INSTRUCTION, "INSTRUCTION" }, \ - { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" } + { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" }, \ + { FAULT_FLAG_VMA_LOCK, "VMA_LOCK" } /* * vm_fault is filled by the pagefault handler and passed to the vma's diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 88619c6a29a3..c4c43f10344a 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1053,6 +1053,7 @@ enum fault_flag { FAULT_FLAG_INTERRUPTIBLE = 1 << 9, FAULT_FLAG_UNSHARE = 1 << 10, FAULT_FLAG_ORIG_PTE_VALID = 1 << 11, + FAULT_FLAG_VMA_LOCK = 1 << 12, }; typedef unsigned int __bitwise zap_flags_t; From patchwork Fri Jan 27 19:41:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49650 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1014544wrn; Fri, 27 Jan 2023 11:54:01 -0800 (PST) X-Google-Smtp-Source: AK7set/pmNbz7R4fK5o0JkLJDUAlzCcVTY37pKUSMTH7v8TPiUYFZuziTdT79jImQb2VgQP6Y73S X-Received: by 2002:a17:906:65d4:b0:878:5bce:291e with SMTP id z20-20020a17090665d400b008785bce291emr6863017ejn.2.1674849240982; Fri, 27 Jan 2023 11:54:00 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674849240; cv=none; d=google.com; s=arc-20160816; b=E/7CXokPehe475iKWsvCM5oec5BHcUvX/ZhBwFGvAWwyVnQo1uiGPl3tJ73YkkUHie koFwnj9hsiqQGVs4pvGOT/EusKRYJidaN1g86HPSJ67f4OD6lKaq/He00jYzeD4Jch1m QJEpgT9LhOsicEGxrTJo2ZNiA2VPIjP0UMHHy4viChBCX1nL/ev45AY1Xcc6d1PhrUmc gA9hxw8aJZ3ID4SQDCchptJm2cm6ds5Kuy7M0l9iGbBSPYVxazkowgFKFfojN9qNejS2 sz/H9q4TNYOPVse5U/V49lAcxhMpN3vkm1LQtIclQoeX1XA5XITkxG/ZZc+gjwDwRskg qJww== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=sBLmwmiYmTj+OGGVUcLxhx0PU/jBUoUEJpANG8soAfU=; b=ggQL1x9AM645eIFZ3LgMDcRhpNv6EW42JXzyTNOvEVNgMy2YbEYQBHM/4EFoooheOs CZcX1idEvRBxvFqTGxoFEu5R6e+3ElHQV+uBwAJyqyDPLUc8qZjM1I/KnHxB4w9pzDyf GRaRqVTZNn5XzGNe/SGpCGslq1eT5zThnbeEn6K80F76DpA5fJ3G7DKcjbzXJysjAI9r r6nSMiIieIph7U3Ot9xx+RtxZJc1qrzK+1gJcUvWmdLbpazItfNy3PLIwC0EF21CNfPe UIqQz0Bl6FaP/XpMfWF5Aig5E+23RgxNECmAxhuygIKBykUlme5ZG2BWkDMFBjSah2P6 it4Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b="KWtfJ/7x"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l19-20020a056402231300b004a0b5652200si6267702eda.70.2023.01.27.11.53.37; Fri, 27 Jan 2023 11:54:00 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b="KWtfJ/7x"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232853AbjA0Tq5 (ORCPT + 99 others); Fri, 27 Jan 2023 14:46:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51614 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232574AbjA0TqY (ORCPT ); Fri, 27 Jan 2023 14:46:24 -0500 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8841A91FA5 for ; Fri, 27 Jan 2023 11:44:06 -0800 (PST) Received: by mail-yb1-f202.google.com with SMTP id g138-20020a25db90000000b0080c27bde887so2635506ybf.18 for ; Fri, 27 Jan 2023 11:44:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=sBLmwmiYmTj+OGGVUcLxhx0PU/jBUoUEJpANG8soAfU=; b=KWtfJ/7xIlPNyMbJgMly5Sn2+77Pi3kT3bIKcXhSEYdAbccLPUNTncstNEX89dvAhW NM9ImIflxrHOdJ1Mj5fWozRK69MXYGw+crQDfh9Oftap/iRU1kAVPiBZhr/+UuvLj4+U 4LLcR3mdcWgS75oxY+Q2vGwsrXQ6jEpwNWpRZ0m4MKHrCT8iGAtEZyFhrSxdE7CjPr88 djL5aWWS3cMBdV/JlcmaRg0LM4Y33l8LwLfoSQoWCllE6I7RGCcpBFSNwKknGwVlLQii i5R/yajGZAULuwac7zP2AeGWc0Ux4Tm1rMqU81PAQU3nWhlQVK4py4JD6fzjGBwa6RFE gFWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=sBLmwmiYmTj+OGGVUcLxhx0PU/jBUoUEJpANG8soAfU=; b=EScMkfmTrvjSIQhckS3et5hFdVMdAQ7t4NJZWmp79+IeUbRbSrqb2B0WABMWgBOpdG J2E0Wa+I4RowSzWJnaJ7rQHtpz8WzjPBdjAHHM4YdJ59cu1e3m3qFO4i/MKt4QNkXivd ZXWAYvICxo8UububZK1L9JiHP4r+5TFbesLHIamHVSzqdf3ULzXGdQUqdFa6PsEjEwZP 2TDd8gZq+nLNft2mMMeCsQzwBED3wQhhZFDLbhws9wAWP+UBFmCBe5su9Vi2mVi8vBcm G1Roq8+MxAuKZ9aLnpUtbk1cstW3YlsFStCeusgKsDqef4uYOIH657uh54SUEin1xs8j H8fw== X-Gm-Message-State: AFqh2krZtpaFVvgH8g42FNH/X+FR9Z6+3Eks08phyrv5mHqSg77s5lbJ rXT7yfH8enFdw3wPudX4v2Y5Kdz34p8= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:cd7:0:b0:4dd:9ed1:fba0 with SMTP id 206-20020a810cd7000000b004dd9ed1fba0mr5011323ywm.265.1674848536283; Fri, 27 Jan 2023 11:42:16 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:03 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-27-surenb@google.com> Subject: [PATCH v2 26/33] mm: prevent do_swap_page from handling page faults under VMA lock From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, Laurent Dufour X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756206717814908829?= X-GMAIL-MSGID: =?utf-8?q?1756206717814908829?= Due to the possibility of do_swap_page dropping mmap_lock, abort fault handling under VMA lock and retry holding mmap_lock. This can be handled more gracefully in the future. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- mm/memory.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 593548f24007..33ecc850d3cb 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3690,6 +3690,11 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (!pte_unmap_same(vmf)) goto out; + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + ret = VM_FAULT_RETRY; + goto out; + } + entry = pte_to_swp_entry(vmf->orig_pte); if (unlikely(non_swap_entry(entry))) { if (is_migration_entry(entry)) { From patchwork Fri Jan 27 19:41:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49639 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1012167wrn; Fri, 27 Jan 2023 11:46:57 -0800 (PST) X-Google-Smtp-Source: AMrXdXsNtyTeJK1DwNuQA5h/IZqZbRk1mhq1Q8lOGEJT2wIAaCHVkkOQDtOnTHMpC/kibjPlmY2A X-Received: by 2002:a05:6402:294c:b0:48e:c0c3:7974 with SMTP id ed12-20020a056402294c00b0048ec0c37974mr44707052edb.12.1674848817795; Fri, 27 Jan 2023 11:46:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674848817; cv=none; d=google.com; s=arc-20160816; b=U6KhZ3eJEWyuHAtL8STyQFkvPlOtI2Z9H1t5AgWSj3PsNmF6AYD9bfUz00Bao8AQA8 WqncXOpIg1WnmbFOW3AapVAAFXgJh2SPhKXEhpa2ymOtkAHrQCI4VD44yzY/jwdB5uQ6 pXFPX+wEhRL8hi8Y5qm2yYRugw7NzNDngsf/Dy1O7zT3CKNtmhp2yNPpVXZ+8YSyeS3Z mt4vlLmBzGde1pDdZ8Bp/3FHeVgrc4hKQWrLwvCh2Qe9wXhUCaPGGk3Xx9BNhfof8T70 Ohrn9QXMqZFs4FwRGpwwgfvWX5J5d3qkmcATJ91aWB4Lq6ggV68Yg31gIFcEpOhPFja3 7J8g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=dtbb/rgJDVPG5hb8ydKzjEL35WMZYcFrpyL4j03oF8s=; b=ACegVAr9QaBKkwv3rlpqGq7cgSfTbD6GqLPNsFBNbwNndm9yZ5yeXT5k00uAMFfHb9 fc8u5OiIatipAQ1y+Sep8genQnvxaMXb2vwN1T7imwNyJ5ekw406TUN+ErpXiLeTW7QR Z2EvbQSYR+iMkw/+USrdEa78JK5XoNDB6sVpdDsQhr6txDcOJ99X8UGSUATsuLrcZii+ +wWrFSB7Q/fnG0VptFsRMH5dzUfWSnIDHhOJef8Q5FhtsYbCUvE94nLcmmH5yb+fQYXy Tigj9NdEYgMmOn30co1It/kf/h/XEL/bwrE9R4BnW9PZZ0uJJ8jxXUV/tY/xfjjGJRDl FyzQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=GSbfUJd9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u4-20020a50eac4000000b0047f9f11dab3si5744464edp.25.2023.01.27.11.46.33; Fri, 27 Jan 2023 11:46:57 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=GSbfUJd9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232010AbjA0Tpd (ORCPT + 99 others); Fri, 27 Jan 2023 14:45:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50906 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232062AbjA0ToH (ORCPT ); Fri, 27 Jan 2023 14:44:07 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 32A505CD1A for ; Fri, 27 Jan 2023 11:43:09 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id x8-20020a25e008000000b007cb4f1e3e57so1753366ybg.8 for ; Fri, 27 Jan 2023 11:43:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=dtbb/rgJDVPG5hb8ydKzjEL35WMZYcFrpyL4j03oF8s=; b=GSbfUJd90yi2xMLpxJwkNIQoO7AYaRuo1p0W3egBrsgFnTlql/QObCOWtQyWcyCJri L9IkPEewPotUja2mT2dWXCpgA9vVmlCVPbr2KFEVViPcTyej+SHFX8+fhAEaOP6xtrM/ x5eCKOKsLW06Glc5jxntDrg/hIgANLyaiKO4UzofC0FOHh/fDzJT0Gg9GuJf/nEgmEZV hix6GjSSYXSWndo4QypjFazRowiaazjUscTvm2uHMbtPJDABgUnEX51b4PI7vURt3LgP X4xLvpH0mH+5Sv/7D+0hQE/rAXv1nGKdGusu8zdXbWRFOMqJp4iPgA6p21w37iQT/eTP bctw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=dtbb/rgJDVPG5hb8ydKzjEL35WMZYcFrpyL4j03oF8s=; b=qOUIpBPL+rpft3hZySD7KKTVnepWMqlG6hLPEcKmV4cggdRv1DCq+7V1plUbCs951X lxMH/szkt80Br8JxeG/hL0bKBtnz+4Box8316nGao3F1fjd8lEUK6ofTduGEO1YctdHS ejvqbRHp/TIV1v5Q+Y2zmBbFzAG4QwyhrGqUObzpVIDH8EeMjkhYzAA6rZGeS/yA+JBc ggcv7ztumPD6pY+APTxeGzXS2QpTmAJjOqmhtZuLAZnWLN7He9ZsbLfSZO0lfcYZeC1a NKMWQh7o2pLJ9oL98hwmBqsiH5GimfUl5sNpxN1iOqz//I99LkjJt4JRNxQV2DecaMl4 oMSw== X-Gm-Message-State: AFqh2kqfZfJYOvJouHXBZLzvGYCzq190xmj8L7HZdkMWzzivaLCejg0S KEZOYTGRFj2I+TAVWWInPflYWhwxsJM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a0d:fc86:0:b0:4db:3ac0:78a3 with SMTP id m128-20020a0dfc86000000b004db3ac078a3mr5702708ywf.266.1674848538548; Fri, 27 Jan 2023 11:42:18 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:04 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-28-surenb@google.com> Subject: [PATCH v2 27/33] mm: prevent userfaults to be handled under per-vma lock From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756206274317195919?= X-GMAIL-MSGID: =?utf-8?q?1756206274317195919?= Due to the possibility of handle_userfault dropping mmap_lock, avoid fault handling under VMA lock and retry holding mmap_lock. This can be handled more gracefully in the future. Signed-off-by: Suren Baghdasaryan Suggested-by: Peter Xu --- mm/memory.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 33ecc850d3cb..55582c6fa2fd 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5256,6 +5256,15 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, if (!vma_start_read(vma)) goto inval; + /* + * Due to the possibility of userfault handler dropping mmap_lock, avoid + * it for now and fall back to page fault handling under mmap_lock. + */ + if (userfaultfd_armed(vma)) { + vma_end_read(vma); + goto inval; + } + /* Check since vm_start/vm_end might change before we lock the VMA */ if (unlikely(address < vma->vm_start || address >= vma->vm_end)) { vma_end_read(vma); From patchwork Fri Jan 27 19:41:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49648 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1013003wrn; Fri, 27 Jan 2023 11:49:33 -0800 (PST) X-Google-Smtp-Source: AMrXdXtJIfxSIDykCNxsbIYZ1Jk+RldkL9HWAEbnOoBS4eGWHoC1MPMQ5X+/kpV16VX9ZmHBa8Yt X-Received: by 2002:a17:907:2910:b0:86f:9fb1:307b with SMTP id eq16-20020a170907291000b0086f9fb1307bmr37435241ejc.31.1674848972919; Fri, 27 Jan 2023 11:49:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674848972; cv=none; d=google.com; s=arc-20160816; b=Rj7R8uGrvptliE2C+nnZ5YnvL3qgyQkr/22Nofh38nuHdUh52APEHKSQIjJMiBYzKD L2JOoZBG5R1Pq3f8IKy15QPt0K2FesZnefi/Q60o6eikVedwxa+oKdq59sJSgh44tVXW NklDSWlPerU4Fes0NNbaBpsxPEgVWOY24AhlLkKvNi/m38PKH57cJlSxFBRPcKnYQWiS Jr3BRChNQVwQJcoMEpAldoiNZcZjfBrKx0j/Ix7Bt0ltjWKJ8xXIHiDzi5s8yZDaJu7S NYjNx0kxyOX9uSuT1qWLvHjKQdBV7HK+VGBEItuOC+chlOgxu0ldWLtbAb+8e7q2KGpB UIQg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=QTC+ocJD/ZK0d8AWah1kQLEh3QM8J68abwl+vLpX/MY=; b=FFdjF7jGyGlTGkWdmg/eHR/SEzePBbEqzsAQ0i3pJVpdpkpBATYFuoBxfb/o7xVrSK RIJFXbYdNleFO+S6IpD7ye4FpD+L0vXnGb6LaTlAX9EtUcoBrv0WfikqoIUajtns1MhI madfrEakx1ow+7qoGBry0nd2nsQaLgmOMrZuk3FgLm7vNZM6rNcZ8PWC9ZlJtujj3S3E ZVY0wYdol75CG8KOYDVka3JdGVGKwweZQtEPIq9MJ5X1TwoRo4rHN/zM0kLGFEiCxFQv oPFM3HUpEyflrR0dOUqTCE7QMaE4NcC3aFi9/PSXodOKmCTl8o5n//2o/OCL9fcIYnAI w68w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b="mU8gu/pW"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 28-20020a170906015c00b0086b6a3fd247si5782592ejh.557.2023.01.27.11.49.09; Fri, 27 Jan 2023 11:49:32 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b="mU8gu/pW"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232041AbjA0Tpg (ORCPT + 99 others); Fri, 27 Jan 2023 14:45:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50934 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231892AbjA0ToH (ORCPT ); Fri, 27 Jan 2023 14:44:07 -0500 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4103E2E811 for ; Fri, 27 Jan 2023 11:43:10 -0800 (PST) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-50660dd3263so65585197b3.19 for ; Fri, 27 Jan 2023 11:43:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=QTC+ocJD/ZK0d8AWah1kQLEh3QM8J68abwl+vLpX/MY=; b=mU8gu/pWf6zIsQFo9qZJmzO+C9Rz2zywIzJAt0hiLsBb0sfMScn0ga1WT0lJP4+cuU jnlE49/hYrnAMYh/RLEYzocL5Jy7mfCziMb33lJyjegHt186LkbV/WCwUSBxzV0FGK1I xrqhc33jx9MJLtdss02+APSdeRf2XUnFG09IUKfXCIr+hkTGBQo7qLvcD8/EZZarD0c6 9RiTDx0vtERha59paJtaKyAUCno0Vzl4l7lvqpJzE2nTjhpBkjOXcTvShnV68JU8Wm5n IoRxYa+gg9A7UVOaBxUJpIYlD0HzhsBk7iegMRGVNF5IQVYNK1FSZGz91UjWgsue8k5N kYLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=QTC+ocJD/ZK0d8AWah1kQLEh3QM8J68abwl+vLpX/MY=; b=n9nUVyn4N6fMDpHmkxC+XdURepWGHSxPuHIFb3iKjAdaDV6xGuNdCfPfgGkiTDt66J wZJqzfJZU6i4HdFuWDrywN9Zj5miz93KzTq6Tqt9As08EMSG5T5WyWf0c0vj52fqVrGp kOhOLZjJ1KwV3+NhZIcTHtHfumls35Wizc3IGJBg79AgWTFugIo9rzq+QY+Cc85/9GF6 dOKEB7Pf7bmz+q1Ounfx5XkCmKIkksfhTtKEWzUUYdV/PAS0qbpTfPPJXZztpBZ+FIB2 W4HH0glsLr5jySTjGCggSCeDtl+9LfA+6Qqs0YPZ/eb4oaOTVFvOokhfClkPlEmBla2U Y4nA== X-Gm-Message-State: AO0yUKVa6UcDtteJJQ+yq6F6KSQhUaXi8ZiuIUcwnAZK1WR2DKjLHKol tXa82CvKE247czvBGnQ+3zTQ9QZpE7w= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a25:e705:0:b0:80b:bdc5:5060 with SMTP id e5-20020a25e705000000b0080bbdc55060mr930533ybh.27.1674848540634; Fri, 27 Jan 2023 11:42:20 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:05 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-29-surenb@google.com> Subject: [PATCH v2 28/33] mm: introduce per-VMA lock statistics From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756206436573716005?= X-GMAIL-MSGID: =?utf-8?q?1756206436573716005?= Add a new CONFIG_PER_VMA_LOCK_STATS config option to dump extra statistics about handling page fault under VMA lock. Signed-off-by: Suren Baghdasaryan --- include/linux/vm_event_item.h | 6 ++++++ include/linux/vmstat.h | 6 ++++++ mm/Kconfig.debug | 7 +++++++ mm/vmstat.c | 6 ++++++ 4 files changed, 25 insertions(+) diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 7f5d1caf5890..8abfa1240040 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -149,6 +149,12 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, #ifdef CONFIG_X86 DIRECT_MAP_LEVEL2_SPLIT, DIRECT_MAP_LEVEL3_SPLIT, +#endif +#ifdef CONFIG_PER_VMA_LOCK_STATS + VMA_LOCK_SUCCESS, + VMA_LOCK_ABORT, + VMA_LOCK_RETRY, + VMA_LOCK_MISS, #endif NR_VM_EVENT_ITEMS }; diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index 19cf5b6892ce..fed855bae6d8 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -125,6 +125,12 @@ static inline void vm_events_fold_cpu(int cpu) #define count_vm_tlb_events(x, y) do { (void)(y); } while (0) #endif +#ifdef CONFIG_PER_VMA_LOCK_STATS +#define count_vm_vma_lock_event(x) count_vm_event(x) +#else +#define count_vm_vma_lock_event(x) do {} while (0) +#endif + #define __count_zid_vm_events(item, zid, delta) \ __count_vm_events(item##_NORMAL - ZONE_NORMAL + zid, delta) diff --git a/mm/Kconfig.debug b/mm/Kconfig.debug index 9191897e76af..4965a7333a3f 100644 --- a/mm/Kconfig.debug +++ b/mm/Kconfig.debug @@ -278,3 +278,10 @@ config DEBUG_KMEMLEAK_AUTO_SCAN memory leaks. If unsure, say Y. + +config PER_VMA_LOCK_STATS + bool "Statistics for per-vma locks" + depends on PER_VMA_LOCK + default y + help + Statistics for per-vma locks. diff --git a/mm/vmstat.c b/mm/vmstat.c index 1ea6a5ce1c41..4f1089a1860e 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1399,6 +1399,12 @@ const char * const vmstat_text[] = { "direct_map_level2_splits", "direct_map_level3_splits", #endif +#ifdef CONFIG_PER_VMA_LOCK_STATS + "vma_lock_success", + "vma_lock_abort", + "vma_lock_retry", + "vma_lock_miss", +#endif #endif /* CONFIG_VM_EVENT_COUNTERS || CONFIG_MEMCG */ }; #endif /* CONFIG_PROC_FS || CONFIG_SYSFS || CONFIG_NUMA || CONFIG_MEMCG */ From patchwork Fri Jan 27 19:41:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49642 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1012349wrn; Fri, 27 Jan 2023 11:47:31 -0800 (PST) X-Google-Smtp-Source: AK7set9JM7cppwmJ0yEzf6jEDlT5h1lukE4bolS+oNKdfz+MnpxGojumxEcaqJ5BpG9d8Ys1okuV X-Received: by 2002:a17:907:94c8:b0:87b:ded0:75f8 with SMTP id dn8-20020a17090794c800b0087bded075f8mr3133131ejc.37.1674848851432; Fri, 27 Jan 2023 11:47:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674848851; cv=none; d=google.com; s=arc-20160816; b=zHphj4fJfUTDkft2LEPzEyNboiDhV1Kz95K2Jal/c4gtB7AVj/6HgtMiE68VDlJmTw 8F2Wd/WqgwiYG5cgPigtYfvDk8yX9W8fg21IYJLIqImDGdLqjy4S0JYbT1z6Rkp1ZcwL tKQEjhFTHD8yKSIBTNDH5VYZFuqQrXcZ2ghKQhIckp07fx2xqggt6pSJnXUIlKjXd2j3 dUaCWcprKMOPdsBx6oou1xb8dwNUaM3PNAt0MzrFWunAnrKi2bx4nvOVR0geShfmahpp 7yRc9AghUura2vgMvWfuV6CTa0+hPsiZmhhwczHUsuhBy/hDtQ7fFNbURHCcNEzEQ0Bo T9Lw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=zr26Tmb2M01sb1afZfnklHdnYeWn0/HWZzrpjXJYooI=; b=QFcUyRuZxcA1iPHHrRJmOJrUIgBo+/vz823kLEMn64jZlVGDgPEZcaCub2PdlYa6nq MJaETxgPgA4EPEYE77csA+7/ztZCRnxkmnZpWpyyjQQuIxWt20qLidJoZ8q6XtrkDvRI nS7E2uH34tjkTs0ywd21sl9wh9qbILJ66q61x7N/CNT+/a5KdwwXdajHECy84SitKqRi I8lyPEWyLFNsOLhHoO0oYHc6awPKGY8FttVk2BTkI98apwBtA9S8fVlZbxshqYzXq4Bi KlcsfaIbXVs0Sp/F1Rzaz/yREZ6riL04UKqI6yCTFCRv7k6FOuGHFwrXOoXFSXluw4pa OXpg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=I2P3olZB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 23-20020a170906225700b00877a133c02fsi6501525ejr.615.2023.01.27.11.47.08; Fri, 27 Jan 2023 11:47:31 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=I2P3olZB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232140AbjA0Tpn (ORCPT + 99 others); Fri, 27 Jan 2023 14:45:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51650 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231825AbjA0ToI (ORCPT ); Fri, 27 Jan 2023 14:44:08 -0500 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B8C2859769 for ; Fri, 27 Jan 2023 11:43:10 -0800 (PST) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-4c11ae6ab25so65258697b3.8 for ; Fri, 27 Jan 2023 11:43:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=zr26Tmb2M01sb1afZfnklHdnYeWn0/HWZzrpjXJYooI=; b=I2P3olZBrDoCFDP1jHcaHXqtsXyUzxQr/ws4wX6bBu3CUipyuxA31yRpWa3MnFa/SU pcf5+CCx+WnMbozlzDdIg2ALFwq/OFj8s/q24FIJwL7Tizh8I6eeGtbBFib4zJzLB5md /zjFFlwtWXGghsSwviKTLUSRcwBRmcJaXn6b7nEXFweXpvU3ZyCKtcIxikLC8+UQ9pUQ JujyWegWeeB7qz6qsENh+jKg1foSfeMqF01FuGu+Yb/GyFYoIS6vq3ynrguPkj0pOoLx BYJ8cK1I9Z+jOMo/TqTeh+dlSH9FVfZfZ84FkijhY572Ij8GlcP0ii8VORVWe0sgl8Ck Cp6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zr26Tmb2M01sb1afZfnklHdnYeWn0/HWZzrpjXJYooI=; b=Wt59uesTdyZ7RecOO9SwzT8Xwl6yO2QvhuTj8m7BoEQaWSRQOjKdcI3IjZ/g+rUr3j bNenf9ptvCrnBNMwf74exRSsJ5Q3T5uGAYdAO6b4YEMlLY1WX1d81wh/f2x6I5U6GAIF Hjk4PGttpM7orqJ4mnvziEG6ckWhVNvq5KQgrRXtYE7GJbSF2l+2Hm9TudC6/udIAAjw MLK85E7LMHI/cXdlBMI+e6A+HZSuSDzenLCmj9RlUhIyXdaoZ1z6hOC+qGGuSVzQx2Vo 3GDtR25Z++F+NE7B0Ga/hKu/1WPlPUtK+gTlzHuuQHbfk0cvpR4TifYVSfLglVU67BpZ Khkw== X-Gm-Message-State: AO0yUKWipz/yH/5hlrIGEDO6lM+lbHKeq0yj//Qi6W4YsOW0mLAWaveZ rNPT32humK10JIhg5Kgj02jYslr4d7E= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a0d:eb09:0:b0:506:391c:6d37 with SMTP id u9-20020a0deb09000000b00506391c6d37mr2021270ywe.88.1674848542935; Fri, 27 Jan 2023 11:42:22 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:06 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-30-surenb@google.com> Subject: [PATCH v2 29/33] x86/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756206308892018965?= X-GMAIL-MSGID: =?utf-8?q?1756206308892018965?= Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Signed-off-by: Suren Baghdasaryan --- arch/x86/Kconfig | 1 + arch/x86/mm/fault.c | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 3604074a878b..3647f7bdb110 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -27,6 +27,7 @@ config X86_64 # Options that are inherently 64-bit kernel only: select ARCH_HAS_GIGANTIC_PAGE select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 + select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_USE_CMPXCHG_LOCKREF select HAVE_ARCH_SOFT_DIRTY select MODULES_USE_ELF_RELA diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index a498ae1fbe66..122c8fe4af12 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -19,6 +19,7 @@ #include /* faulthandler_disabled() */ #include /* efi_crash_gracefully_on_page_fault()*/ #include +#include /* find_and_lock_vma() */ #include /* boot_cpu_has, ... */ #include /* dotraplinkage, ... */ @@ -1333,6 +1334,38 @@ void do_user_addr_fault(struct pt_regs *regs, } #endif +#ifdef CONFIG_PER_VMA_LOCK + if (!(flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto lock_mmap; + + vma = lock_vma_under_rcu(mm, address); + if (!vma) + goto lock_mmap; + + if (unlikely(access_error(error_code, vma))) { + vma_end_read(vma); + goto lock_mmap; + } + fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); + vma_end_read(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + /* Quick path to respond to signals */ + if (fault_signal_pending(fault, regs)) { + if (!user_mode(regs)) + kernelmode_fixup_or_oops(regs, error_code, address, + SIGBUS, BUS_ADRERR, + ARCH_DEFAULT_PKEY); + return; + } +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ + /* * Kernel-mode access to the user address space should only occur * on well-defined single instructions listed in the exception @@ -1433,6 +1466,9 @@ void do_user_addr_fault(struct pt_regs *regs, } mmap_read_unlock(mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif if (likely(!(fault & VM_FAULT_ERROR))) return; From patchwork Fri Jan 27 19:41:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49651 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1014937wrn; Fri, 27 Jan 2023 11:54:57 -0800 (PST) X-Google-Smtp-Source: AK7set+R9LCwYLQ+9rqjbwAdx+vmk2ayTKnzobPsQ6ESJTziepQmMK/12xEUUX+ZvLBspD0LiCJd X-Received: by 2002:a17:906:31d1:b0:879:2a5:dc40 with SMTP id f17-20020a17090631d100b0087902a5dc40mr4233276ejf.76.1674849297305; Fri, 27 Jan 2023 11:54:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674849297; cv=none; d=google.com; s=arc-20160816; b=UdhjaHZapu+RMw8tZviHE4XCGuTwCSlld412xP1kHD4kcsDvu8b9EyVDgfZGOynQi9 jn1MO+sc+lbxllEHMWvoP2Ior2TRXPXu/CzlRBL3Ynl3lc2XZne9LZKISkvWQEiHbXp/ JuF4kJMfvf3mz/6oY7PD0DH8QV94wTxL107cZuBu6dck4A2LTjLAj9CdNVM2QD2393I+ L/AZfHXmUQssGisqQlODvndbWzlSnM6G/prSWet5/uzYTIMwLpO8wa0SZqOsBExLehOd W8TT5OunhUGZYgLGHXjksMiDbC3RMgyLPOkmQnjChYjNtfUiSn69AS22yEA4uLoDO9nb WDSA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=MNwo0rhG4y8C+tQG7LPM4nLJWTbIHprc8uVQsZA2GSc=; b=Cv/5G/CGoZKEUPVkNFkr4tiJ/YIJBfF/tq61b5Jm3fecEzas4aQR1EhMZ5DDfz+z7h 3Qasp1pYvp93WFEFeKzlg9+aLesGc7jKvJbrTMLsCMP7yO9VIpw0FCmpfIbU9o1mSiHr KMg2kRUZ75bGSTqkdPnKR1b1wyxeYj1QmS5lJ9ObNDVBs5ALhJJ4+z83tK4QGal9J4TY 3fQBRKbN1Hznj2agVOpgg3ugezmHgzRkeqRIwXuRLfE5lIkhnyhKyOBEcwZyr0Jj5OYM rv3ZiIimdkuO8xIm7BU+iB0wvYvi0hCArWHtSBRHGDxnq3JqzhIRtweYMzuz//41vfQa 2Atg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=JxxlO558; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id s8-20020a50ab08000000b004a0bae0bbd9si5996641edc.432.2023.01.27.11.54.33; Fri, 27 Jan 2023 11:54:57 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=JxxlO558; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232694AbjA0TrG (ORCPT + 99 others); Fri, 27 Jan 2023 14:47:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50660 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232683AbjA0Tq3 (ORCPT ); Fri, 27 Jan 2023 14:46:29 -0500 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A25A08D0BD for ; Fri, 27 Jan 2023 11:44:10 -0800 (PST) Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-482d3bf0266so65384517b3.3 for ; Fri, 27 Jan 2023 11:44:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=MNwo0rhG4y8C+tQG7LPM4nLJWTbIHprc8uVQsZA2GSc=; b=JxxlO5586M0z4sVDSxHugqEEADB1cAjEHo2xluVRFI46WqHTOUsszCaJjysGoPqCq8 ztKkJGmntymAGuEGxdM2A7EX8Sykw/reCFrQFVsgm7kaRoHFsG9BNF/PmmjyzPElNA3A u2zVx465PKwxuuDkWTqvf70dFDsT4sE76+Q1i+nyNMyErq//frz9WqLOph65UMZVM1Q3 1JnBBBOhnZpIDf2DrbbDSdhADbiUH2oov3flacw7sejkvu/YDNPNLxViqgxSV8aAiDpN WEe8RN23e0S32q4tATcZ7ZhVfhZ7+NOeU1bzW/fjTQ6HzW6TJlQUp4mHpwvwuOr4rhfO QkKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MNwo0rhG4y8C+tQG7LPM4nLJWTbIHprc8uVQsZA2GSc=; b=4FF1nk0tYt6b6Z3EunkU6EU6kG6eBxGs3ME3FwLhXpLxF38mP7rovlU2by3mKOGy6q Pd6kgN1MTV666QKgtZC1AjGMzWq16tFNdkQPh7HA08EfNbj8a+olr6EkkGBzvh6H6d6d FbyfXvqsFapeMOWOMmnxgxEt6GFsd9YfYJhEh8EuoQLvWSfzFbB4ju7zkXgMKu7BqXdt pDRGrPN7Mo6q1DGjH3roDtkRJKRTFhLVdeX9EX4JnzrXj2tHztIISYOz2fAL4NOTj8sW EdqZWU25C/LfRDN1fYPWZGvWUB8MD5GkaryBxI/Ke0Bkzs8H+gCn0zvglAVA250/yfO0 0bCg== X-Gm-Message-State: AFqh2kpB9jhSGcccYp0gXXc8NN57H5pBxnTR9OBDe2wFuS6D1MPKBouZ FduU3do8ldCdXsDfGJ+nsPevuudIYm0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:5b88:0:b0:462:e278:1a08 with SMTP id p130-20020a815b88000000b00462e2781a08mr5361156ywb.243.1674848545116; Fri, 27 Jan 2023 11:42:25 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:07 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-31-surenb@google.com> Subject: [PATCH v2 30/33] arm64/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756206776614294552?= X-GMAIL-MSGID: =?utf-8?q?1756206776614294552?= Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Signed-off-by: Suren Baghdasaryan --- arch/arm64/Kconfig | 1 + arch/arm64/mm/fault.c | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index c5ccca26a408..9f2c0e352da3 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -95,6 +95,7 @@ config ARM64 select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 select ARCH_SUPPORTS_NUMA_BALANCING select ARCH_SUPPORTS_PAGE_TABLE_CHECK + select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT select ARCH_WANT_DEFAULT_BPF_JIT select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index f4cb0f85ccf4..16588dda9258 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -535,6 +535,9 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, unsigned long vm_flags; unsigned int mm_flags = FAULT_FLAG_DEFAULT; unsigned long addr = untagged_addr(far); +#ifdef CONFIG_PER_VMA_LOCK + struct vm_area_struct *vma; +#endif if (kprobe_page_fault(regs, esr)) return 0; @@ -585,6 +588,36 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, addr); +#ifdef CONFIG_PER_VMA_LOCK + if (!(mm_flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto lock_mmap; + + vma = lock_vma_under_rcu(mm, addr); + if (!vma) + goto lock_mmap; + + if (!(vma->vm_flags & vm_flags)) { + vma_end_read(vma); + goto lock_mmap; + } + fault = handle_mm_fault(vma, addr & PAGE_MASK, + mm_flags | FAULT_FLAG_VMA_LOCK, regs); + vma_end_read(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + /* Quick path to respond to signals */ + if (fault_signal_pending(fault, regs)) { + if (!user_mode(regs)) + goto no_context; + return 0; + } +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ /* * As per x86, we may deadlock here. However, since the kernel only * validly references user space from well defined areas of the code, @@ -628,6 +661,9 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, } mmap_read_unlock(mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif /* * Handle the "normal" (no error) case first. */ From patchwork Fri Jan 27 19:41:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49647 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1012768wrn; Fri, 27 Jan 2023 11:48:47 -0800 (PST) X-Google-Smtp-Source: AMrXdXsYcYqm3OwOx+jYFBcmIlnOjqYf+Vqsccd1ziA5N6XT6crNYX1cbjApvDwBh6UO51xoqLwv X-Received: by 2002:a05:6402:f14:b0:49d:1a59:174d with SMTP id i20-20020a0564020f1400b0049d1a59174dmr38833699eda.30.1674848927414; Fri, 27 Jan 2023 11:48:47 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674848927; cv=none; d=google.com; s=arc-20160816; b=CrKfue4FVa5DwOHstSpTUsxKdvJQhxec4qJWHCaj2HL3gGsB92AquaK7dq5awQrO6S N5TFKJ/Yye/BNAdYQRgflCmy2VtokGilY46qTihiqYaZ9vbZKEmguG5lTC+22LalJtgq t9X0b0a9NwUgCqJrFAhRkrTMv3yVBiGsiMsSyycDQBcqIXWXCBi1V5sCByWsTMTRv/x7 seotBVkLZf9jz1UHitgxsqpRLpUwq3KojXDw6CoROAkak4pUN99CX7zWXuTMWhOq34O9 AIZ0vAvP5Nsh77ot54X7Qu/rSRIhElhMG8mYlVz5N6n8KcoSUqSDnrsJetW0M/Atm4Ma 3mwQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=kzxeHm8qCtyYeQADp+m05TsFOqarDXtVrIERGy2x4bE=; b=Fb9LsJFH8bcYvSlSgSBwXiwVcmMCQuJsS25NpEF2KyzSsTzW8TSTjbjKlHevCLXlPx o/aJiQ7DM8YHxFruWhSSbgKkADlIzjM/sKi4syy2mDvI7NiQ8UasRT0UiribDjdRL+lv tta8F7kqb12uZMJjnBiKbCozlTV+v9wk9c9OqxgjStInvutDx1CrARwWBOPr40YXYXqQ 3zBjZeRH7h2fyomNSbO3up4XhJ0lLW7tf3zcovxE7C/oRpwgvvpISTyVU56TZVLcLIE4 EFw9WjtTVI0fRb8WPe/NrcsMwVCqVOAsSMH3DMln/4xvvn1w8mKEPfpO2dy13WP1mYwQ hd8Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b="Qw/TzqOb"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i20-20020a0564020f1400b0049ea11f77eesi6099482eda.528.2023.01.27.11.48.23; Fri, 27 Jan 2023 11:48:47 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b="Qw/TzqOb"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232421AbjA0Tpy (ORCPT + 99 others); Fri, 27 Jan 2023 14:45:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51148 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232399AbjA0ToV (ORCPT ); Fri, 27 Jan 2023 14:44:21 -0500 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 61CF17750B for ; Fri, 27 Jan 2023 11:43:14 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-50660dd3263so65587577b3.19 for ; Fri, 27 Jan 2023 11:43:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=kzxeHm8qCtyYeQADp+m05TsFOqarDXtVrIERGy2x4bE=; b=Qw/TzqObuxKUrxtkILJdFMJrnRXDl7IhHsfqWYsg15Fht0zJymOStyW6umwGn696mL /vz0L3L+AxNuyNMgLOdykFyo3ffxujr1NJ9PZ/3TCzeykyM3g1PjhFRGoo/e9FoO8fmJ 0gJOei12QFUXaa/Oa6hhy8ENMFQY5aEUa+SKaoSJiwpGU68x3ruM/ipRGHnS2jScV2XA EPWUY+rZwtRmlqA25G6oa8LHcJDQM3bZQ4diO+PFB7fofYZtcMVMGn5EMXtqF1YwdUnI Yur83PIWndlmZU2J2xMV4at0clCg0WXiP4a0wm/tFrIYOxaiWzueFhV10wBQ62TvKtYz 5Efg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kzxeHm8qCtyYeQADp+m05TsFOqarDXtVrIERGy2x4bE=; b=AtQzOB9vPTpJz+EfztU1XkFKoZXi3jM+jIBHL9Jv/WFjn0kPHtGdjxX0hOQMylJMMb qk/eM04IFp7xl5zrinPztwfBqfAtNpfjiTUCfdKBi8zIPv93OKnwVmduvgruRLfXkya9 mYBd2QHacziFw8IhcXs3wHdgzIJLYnxWbleKoZvw8ys6ZrMS+/7tz3m79gpGnrqL0TcV YP6+FTJ7oh1qDXoaBEAjXojDgG+NkD4QUnKayutS+MjG3MoTtn4f1t9hHrRWsWeMYXDP PWTlGWIzA+cphDSXBzZsfMZvy1YcIltSr+OXPRt3H1ec0f7sdsT4uOUlBt2jBL4opNUt oNVQ== X-Gm-Message-State: AFqh2kpKMRZDQ+wNUxiI3DoVJF9BUC5CkeSGJUE1iKnIFk4RbhUBhNLy SuRbyqz33YK9hA4+jzFBF9+v62pUt4c= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:1754:0:b0:4fd:ff18:de6c with SMTP id 81-20020a811754000000b004fdff18de6cmr2568712ywx.227.1674848547503; Fri, 27 Jan 2023 11:42:27 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:08 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-32-surenb@google.com> Subject: [PATCH v2 31/33] powerc/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756206388848217965?= X-GMAIL-MSGID: =?utf-8?q?1756206388848217965?= From: Laurent Dufour Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Copied from "x86/mm: try VMA lock-based page fault handling first" Signed-off-by: Laurent Dufour Signed-off-by: Suren Baghdasaryan --- arch/powerpc/mm/fault.c | 41 ++++++++++++++++++++++++++ arch/powerpc/platforms/powernv/Kconfig | 1 + arch/powerpc/platforms/pseries/Kconfig | 1 + 3 files changed, 43 insertions(+) diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c index 2bef19cc1b98..cab229e75018 100644 --- a/arch/powerpc/mm/fault.c +++ b/arch/powerpc/mm/fault.c @@ -469,6 +469,44 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, if (is_exec) flags |= FAULT_FLAG_INSTRUCTION; +#ifdef CONFIG_PER_VMA_LOCK + if (!(flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto lock_mmap; + + vma = lock_vma_under_rcu(mm, address); + if (!vma) + goto lock_mmap; + + if (unlikely(access_pkey_error(is_write, is_exec, + (error_code & DSISR_KEYFAULT), vma))) { + int rc = bad_access_pkey(regs, address, vma); + + vma_end_read(vma); + return rc; + } + + if (unlikely(access_error(is_write, is_exec, vma))) { + int rc = bad_access(regs, address); + + vma_end_read(vma); + return rc; + } + + fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); + vma_end_read(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + if (fault_signal_pending(fault, regs)) + return user_mode(regs) ? 0 : SIGBUS; + +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ + /* When running in the kernel we expect faults to occur only to * addresses in user space. All other faults represent errors in the * kernel and should generate an OOPS. Unfortunately, in the case of an @@ -545,6 +583,9 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, mmap_read_unlock(current->mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif if (unlikely(fault & VM_FAULT_ERROR)) return mm_fault_error(regs, address, fault); diff --git a/arch/powerpc/platforms/powernv/Kconfig b/arch/powerpc/platforms/powernv/Kconfig index ae248a161b43..70a46acc70d6 100644 --- a/arch/powerpc/platforms/powernv/Kconfig +++ b/arch/powerpc/platforms/powernv/Kconfig @@ -16,6 +16,7 @@ config PPC_POWERNV select PPC_DOORBELL select MMU_NOTIFIER select FORCE_SMP + select ARCH_SUPPORTS_PER_VMA_LOCK default y config OPAL_PRD diff --git a/arch/powerpc/platforms/pseries/Kconfig b/arch/powerpc/platforms/pseries/Kconfig index a3b4d99567cb..e036a04ff1ca 100644 --- a/arch/powerpc/platforms/pseries/Kconfig +++ b/arch/powerpc/platforms/pseries/Kconfig @@ -21,6 +21,7 @@ config PPC_PSERIES select HOTPLUG_CPU select FORCE_SMP select SWIOTLB + select ARCH_SUPPORTS_PER_VMA_LOCK default y config PARAVIRT From patchwork Fri Jan 27 19:41:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49649 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1013030wrn; Fri, 27 Jan 2023 11:49:40 -0800 (PST) X-Google-Smtp-Source: AMrXdXtBqiNKNMb3QaMM/4uoCXx5fpSEqwefBRVacoo+pjLEEFOd983QCwKmI5EVp3MQOWY7jHWC X-Received: by 2002:a17:907:91d3:b0:86e:3e0c:ae50 with SMTP id h19-20020a17090791d300b0086e3e0cae50mr33997210ejz.14.1674848980405; Fri, 27 Jan 2023 11:49:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674848980; cv=none; d=google.com; s=arc-20160816; b=UDq4kKPFC8vR8Y5dFatorKA7XAFoWvXeIcY4D3C9qZ5SzqriJm6uN+vXks/4zH0Flw 1nu6Ryr70CzX+PuKQpHND0y+PJSSeIYReDYFAdNiCtCkYXM8ubMr7N6KyCerYgr62khs dhYxA7cf9Vov2nU1oHarH2lVtUd2ZY1OXgiJuJJSJzi6h7SfeZTYZszvth7H1x5E2Ueg htdVxXdu3yETrS6ZOZuDhg3qsLY5yZ+FazUgZdAvNaORgOv3Jw4Of6Ms7qzAooBtW+W/ I3Z1Xk7DJ1cRbKH+dCqD+9lBSTSzn6QruBnBC4L6KM+dMyl0OflvJvFcEvX+tTQHNmf/ WCfQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=y3NZtkliWQhGwQQ6J+V08tBPQFOX7N0W14jS8GdMVSo=; b=ilC8tfLoP0dqljkp5ULgjd3DbF3gmjw6KF0ePlLS8PD0ZdcY9ZNZUZnDJBmEAo+iJV DFDyBBGqvqzK+cIy2Y9FHYY2nPqESVICRJggqBKhOCnYGwrv9BWBlXAegsFbMMuZ+5kz A7+ZUBq9Q0RuTZlaW7f9s8HXhosRVyXCBCrWQjv4L7ux9gXkpTTEpzgbsjA17UrxUcit 2NDFuFYGPY63h1nATAwTa7IwKgqpZxRqdCtdQr0lRwkraxv+iMBhKvkzpFO4Ednlxhyd GjKxnJSZD/Bme4cvxlxBAZlijcMp957UNU0jgZLXYHjG6vPQubpflrJfCwjzoWPeeNko VLaA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=R8vm5sGe; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f22-20020a1709063f5600b0086ea1c7ae27si6653408ejj.241.2023.01.27.11.49.17; Fri, 27 Jan 2023 11:49:40 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=R8vm5sGe; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232484AbjA0Tp6 (ORCPT + 99 others); Fri, 27 Jan 2023 14:45:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51706 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232148AbjA0ToV (ORCPT ); Fri, 27 Jan 2023 14:44:21 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D1C1923859 for ; Fri, 27 Jan 2023 11:43:14 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id o1-20020a252801000000b0080b8600bdc9so6363168ybo.3 for ; Fri, 27 Jan 2023 11:43:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=y3NZtkliWQhGwQQ6J+V08tBPQFOX7N0W14jS8GdMVSo=; b=R8vm5sGePmLU1Pe4mKnKd7BNiKDqaMQRSE2Uxaf9jHVe/UX+8wZaZnli/NAPrvHh5D Gp2YVdh1UhOUz23uxWg1VsM9bUD5cup3/ZbXShp0DX2DQS4OYSwXrDDdrwmKhIAFH0Hw 74ENSW8vtZuV6jUaLsmkcaEy+yDT+z+95V7siTwW55EWIBuns5aniQb/TRczPRcRq3g4 VhtuB5yMK58BxiemGZ2CqsONqT2FuthDA4119d3zQhl7jpXryXulujWcH7o4rSRWH8Tl w610sqrsMHevwvr509uliYrTdDC7Uzh2VKUBma4LjNdxi7kycc6HPZnQZbUV0crFM3wd Zbgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=y3NZtkliWQhGwQQ6J+V08tBPQFOX7N0W14jS8GdMVSo=; b=U2RH4n3XmSgimihjQh1g+eLe9eLxm8x90wjyM8hU4EiCena1tnn50vrVzQxcLIq1Tf CvuAkAy7nNUkhix2kX0rTB7Aaf0dPW6qMFWGEewhH4ZKPvFt+cjzs7s8Elb+p32/Jfyl wtvKAjgF9+2RfNM0B1If+6qgLnVudCtyvw1Xxs99HSTBkq+BEQ7nDO4LWhrwtlqqSCkD H8ENE4a9cMpSInHOX01ZdDGg2tjDjJnAu4uwdsX2dmE6Q4Sv5QFFs9qm/WRMEDamnfxr ujwnlLxPjg5LFAvD1HpA1bebMGs/X4lKv+RlMuafUyv6jH3nO5LhW7hmEcUsLlhDBijq UeCg== X-Gm-Message-State: AO0yUKXBZn2bXYeyQbUK3ENULzq+MLWv/UFV4b4wSktOqoeY8kmA9e11 xfnOTApstaZKaIrO8fz6frgVoIBvYBw= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a25:db07:0:b0:80b:8dd0:7b35 with SMTP id g7-20020a25db07000000b0080b8dd07b35mr1614607ybf.322.1674848549978; Fri, 27 Jan 2023 11:42:29 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:09 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-33-surenb@google.com> Subject: [PATCH v2 32/33] mm/mmap: free vm_area_struct without call_rcu in exit_mmap From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756206444109975271?= X-GMAIL-MSGID: =?utf-8?q?1756206444109975271?= call_rcu() can take a long time when callback offloading is enabled. Its use in the vm_area_free can cause regressions in the exit path when multiple VMAs are being freed. Because exit_mmap() is called only after the last mm user drops its refcount, the page fault handlers can't be racing with it. Any other possible user like oom-reaper or process_mrelease are already synchronized using mmap_lock. Therefore exit_mmap() can free VMAs directly, without the use of call_rcu(). Expose __vm_area_free() and use it from exit_mmap() to avoid possible call_rcu() floods and performance regressions caused by it. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 2 ++ kernel/fork.c | 2 +- mm/mmap.c | 11 +++++++---- 3 files changed, 10 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index a6de58bb40c7..1c4ddcd6fd84 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -250,6 +250,8 @@ void setup_initial_init_mm(void *start_code, void *end_code, struct vm_area_struct *vm_area_alloc(struct mm_struct *); struct vm_area_struct *vm_area_dup(struct vm_area_struct *); void vm_area_free(struct vm_area_struct *); +/* Use only if VMA has no other users */ +void __vm_area_free(struct vm_area_struct *vma); #ifndef CONFIG_MMU extern struct rb_root nommu_region_tree; diff --git a/kernel/fork.c b/kernel/fork.c index a08cc0e2bfde..d0999de82f94 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -480,7 +480,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return new; } -static void __vm_area_free(struct vm_area_struct *vma) +void __vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); kmem_cache_free(vm_area_cachep, vma); diff --git a/mm/mmap.c b/mm/mmap.c index 3d0cfbc92745..1028fe131bb7 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -133,7 +133,7 @@ void unlink_file_vma(struct vm_area_struct *vma) /* * Close a vm structure and free it. */ -static void remove_vma(struct vm_area_struct *vma) +static void remove_vma(struct vm_area_struct *vma, bool unreachable) { might_sleep(); if (vma->vm_ops && vma->vm_ops->close) @@ -141,7 +141,10 @@ static void remove_vma(struct vm_area_struct *vma) if (vma->vm_file) fput(vma->vm_file); mpol_put(vma_policy(vma)); - vm_area_free(vma); + if (unreachable) + __vm_area_free(vma); + else + vm_area_free(vma); } static inline struct vm_area_struct *vma_prev_limit(struct vma_iterator *vmi, @@ -2134,7 +2137,7 @@ static inline void remove_mt(struct mm_struct *mm, struct ma_state *mas) if (vma->vm_flags & VM_ACCOUNT) nr_accounted += nrpages; vm_stat_account(mm, vma->vm_flags, -nrpages); - remove_vma(vma); + remove_vma(vma, false); } vm_unacct_memory(nr_accounted); validate_mm(mm); @@ -3083,7 +3086,7 @@ void exit_mmap(struct mm_struct *mm) do { if (vma->vm_flags & VM_ACCOUNT) nr_accounted += vma_pages(vma); - remove_vma(vma); + remove_vma(vma, true); count++; cond_resched(); } while ((vma = mas_find(&mas, ULONG_MAX)) != NULL); From patchwork Fri Jan 27 19:41:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49652 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1015016wrn; Fri, 27 Jan 2023 11:55:09 -0800 (PST) X-Google-Smtp-Source: AMrXdXv6YA40L6pakxLis/WMLxNGi2toGQGDFvSCLzw8vzmoOKhL9fGZ/ZTugEeckuNVVgQPKpqB X-Received: by 2002:a17:907:9a09:b0:85d:3771:18b7 with SMTP id kr9-20020a1709079a0900b0085d377118b7mr44655448ejc.70.1674849309438; Fri, 27 Jan 2023 11:55:09 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674849309; cv=none; d=google.com; s=arc-20160816; b=T9a++OwT91MxyNlYKKBeiR1X6u//KrfZexvJ08GdUMrIgVysB6lpGm45P7yYxKGVjV vM0QtwG9+KLXFw8B4Rh1tJBc5Q3fdI69WV6mYAZaf4UdMKaTrwQIDZsgQp3us39wmjRW /GKUeEOuUIuSA3/JOhZzTrKIoGYSLLU7mzh2NmTxGKmrkf/ftId6+elPd48NgSjRkQe4 l6LtZ3lDua6lOBPAmcGA5CdDs+/C+HMriuPB7wGb2DvscTv7ibzHpzl/DhkkJMmuKDCf BT1MMsXMGuo87y4QfJ8Ii05uDGd3T+ilQPmHxW4PEIwb72rZryBIkRNwlmMMJSMyTJ9T J6Hg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=Ud3T37IGHP0g1lmdTa/tq8X3xMhLxr1AXmnGvd1U4CI=; b=n6lF9bBEPRmCDkb4U8nELkhyHlitvBSdQnLxNk/s7M6J43U14aigo9a+Rdj7jt3iWA gMQFYEWpcA+KkUo+hI5c2cDI2N4EJ6x6BpX/bDdCU/24zeaPSM3EmHj5GIXCK8oyx8cS vrBdfwadWF29Ln3koxJrxnx4UPMZYlyfAk+JrBqBvR3ayQOm2rCcKAaBe7/IGKnCZxI7 cYo8gmoQIUPlpLmpaZfTaHblOs4E//eTDCvOwpq7nZnEWO/NGgtm8bCeXi/w1ffBsHi+ Rck6Khs8Oc6T2MEdv72hJ8y0HbjsGvVzARmM5zMOMvcK0q9vG81w4/wNZQLk65K14Nia txGA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=FNDN2+9L; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 26-20020a170906011a00b0086f05817f39si7387369eje.39.2023.01.27.11.54.45; Fri, 27 Jan 2023 11:55:09 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=FNDN2+9L; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232764AbjA0TrK (ORCPT + 99 others); Fri, 27 Jan 2023 14:47:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51706 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232739AbjA0Tqc (ORCPT ); Fri, 27 Jan 2023 14:46:32 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4B5A04C21 for ; Fri, 27 Jan 2023 11:44:14 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id k15-20020a5b0a0f000000b007eba3f8e3baso6329126ybq.4 for ; Fri, 27 Jan 2023 11:44:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Ud3T37IGHP0g1lmdTa/tq8X3xMhLxr1AXmnGvd1U4CI=; b=FNDN2+9L1Vmxr+/mMF6eML5JcVJbNLvQW3fVgvwdePzy3lehe5YuSr9HWsNaMC4a8J FUhI3Xak6vLO90aUXXpIFs6adKFN80x7s3bJgLZLYfmaGEKhkkZLbtXJLX0N/ZyPrSOD vT5YWUMa6D2YWUL+XcKLQxhxuZ8Qre6a0DkbEZ1ct6ULdZqWxPdE1UEo8jeDeLLPMaSB b3W+W/qkByIa+a5SR1bxuJVPYAEoRSrded/1J6aiwU0I/kursScnVvXrI1MYVCgC0Qld xz7h6VUcKpIFgOOniSgMFSpFb+kGVYjmhq6wkneKC/yuMYuw9ACcdFQ59agxDWycdAy0 DJpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Ud3T37IGHP0g1lmdTa/tq8X3xMhLxr1AXmnGvd1U4CI=; b=BgTUrJZRUJmi0hoEVwXVo0m2ZC9fA9u0snfUMNK34bekz+yrU4F3evsVe77MqX9mFL E+aJkMudSBlpfhNonRj4ruMpNuEE3sMYyPGSSr0cLW06/xWkWii2+K7agPKUnC0ZRP1r nt2wGbVaAGCxw/jjzcteXFpyq5iV1y+jIWTiKyGZQqnygrcTKfpXOl6r5g4uxDiEgQxb 9E59Qb3BY7DnRm1gzZTt5EuS1D04Ye8uaUzBA9W4z6LcnqCJBDd6lPmyI4YBmIDzslgm WVYrWugEcyfXObErrTe5UR92peWWGzHRhBHHNV+ehroWG8jO2YmnWF3aR4XbEnh5WBj8 1rbA== X-Gm-Message-State: AFqh2kqZoRkzE+0odlkQSkxUR9Ky0XPbQ+Viyxm7b6qAuNGgJB3Kfq7v pw2xWu0DQWaZDMAfnFhNPosl0Yrr+9g= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a81:be06:0:b0:4b2:fa7c:8836 with SMTP id i6-20020a81be06000000b004b2fa7c8836mr5004644ywn.195.1674848552353; Fri, 27 Jan 2023 11:42:32 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:10 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-34-surenb@google.com> Subject: [PATCH v2 33/33] mm: separate vma->lock from vm_area_struct From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756206789717847472?= X-GMAIL-MSGID: =?utf-8?q?1756206789717847472?= vma->lock being part of the vm_area_struct causes performance regression during page faults because during contention its count and owner fields are constantly updated and having other parts of vm_area_struct used during page fault handling next to them causes constant cache line bouncing. Fix that by moving the lock outside of the vm_area_struct. All attempts to keep vma->lock inside vm_area_struct in a separate cache line still produce performance regression especially on NUMA machines. Smallest regression was achieved when lock is placed in the fourth cache line but that bloats vm_area_struct to 256 bytes. Considering performance and memory impact, separate lock looks like the best option. It increases memory footprint of each VMA but that can be optimized later if the new size causes issues. Note that after this change vma_init() does not allocate or initialize vma->lock anymore. A number of drivers allocate a pseudo VMA on the stack but they never use the VMA's lock, therefore it does not need to be allocated. The future drivers which might need the VMA lock should use vm_area_alloc()/vm_area_free() to allocate the VMA. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 23 ++++++------- include/linux/mm_types.h | 6 +++- kernel/fork.c | 73 ++++++++++++++++++++++++++++++++-------- 3 files changed, 74 insertions(+), 28 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 1c4ddcd6fd84..52e048c31239 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -621,12 +621,6 @@ struct vm_operations_struct { }; #ifdef CONFIG_PER_VMA_LOCK -static inline void vma_init_lock(struct vm_area_struct *vma) -{ - init_rwsem(&vma->lock); - vma->vm_lock_seq = -1; -} - /* * Try to read-lock a vma. The function is allowed to occasionally yield false * locked result to avoid performance overhead, in which case we fall back to @@ -638,17 +632,17 @@ static inline bool vma_start_read(struct vm_area_struct *vma) if (vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq)) return false; - if (unlikely(down_read_trylock(&vma->lock) == 0)) + if (unlikely(down_read_trylock(&vma->vm_lock->lock) == 0)) return false; /* * Overflow might produce false locked result. * False unlocked result is impossible because we modify and check - * vma->vm_lock_seq under vma->lock protection and mm->mm_lock_seq + * vma->vm_lock_seq under vma->vm_lock protection and mm->mm_lock_seq * modification invalidates all existing locks. */ if (unlikely(vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq))) { - up_read(&vma->lock); + up_read(&vma->vm_lock->lock); return false; } return true; @@ -657,7 +651,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) static inline void vma_end_read(struct vm_area_struct *vma) { rcu_read_lock(); /* keeps vma alive till the end of up_read */ - up_read(&vma->lock); + up_read(&vma->vm_lock->lock); rcu_read_unlock(); } @@ -675,9 +669,9 @@ static inline void vma_start_write(struct vm_area_struct *vma) if (vma->vm_lock_seq == mm_lock_seq) return; - down_write(&vma->lock); + down_write(&vma->vm_lock->lock); vma->vm_lock_seq = mm_lock_seq; - up_write(&vma->lock); + up_write(&vma->vm_lock->lock); } static inline void vma_assert_write_locked(struct vm_area_struct *vma) @@ -704,6 +698,10 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma) {} #endif /* CONFIG_PER_VMA_LOCK */ +/* + * WARNING: vma_init does not initialize vma->vm_lock. + * Use vm_area_alloc()/vm_area_free() if vma needs locking. + */ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) { static const struct vm_operations_struct dummy_vm_ops = {}; @@ -712,7 +710,6 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); - vma_init_lock(vma); } /* Use when VMA is not part of the VMA tree and needs no locking */ diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index c4c43f10344a..1e97bb98197c 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -471,6 +471,10 @@ struct anon_vma_name { char name[]; }; +struct vma_lock { + struct rw_semaphore lock; +}; + /* * This struct describes a virtual memory area. There is one of these * per VM-area/task. A VM area is any part of the process virtual memory @@ -510,7 +514,7 @@ struct vm_area_struct { #ifdef CONFIG_PER_VMA_LOCK int vm_lock_seq; - struct rw_semaphore lock; + struct vma_lock *vm_lock; #endif /* diff --git a/kernel/fork.c b/kernel/fork.c index d0999de82f94..a152804faa14 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -451,13 +451,49 @@ static struct kmem_cache *vm_area_cachep; /* SLAB cache for mm_struct structures (tsk->mm) */ static struct kmem_cache *mm_cachep; +#ifdef CONFIG_PER_VMA_LOCK + +/* SLAB cache for vm_area_struct.lock */ +static struct kmem_cache *vma_lock_cachep; + +static bool vma_lock_alloc(struct vm_area_struct *vma) +{ + vma->vm_lock = kmem_cache_alloc(vma_lock_cachep, GFP_KERNEL); + if (!vma->vm_lock) + return false; + + init_rwsem(&vma->vm_lock->lock); + vma->vm_lock_seq = -1; + + return true; +} + +static inline void vma_lock_free(struct vm_area_struct *vma) +{ + kmem_cache_free(vma_lock_cachep, vma->vm_lock); +} + +#else /* CONFIG_PER_VMA_LOCK */ + +static inline bool vma_lock_alloc(struct vm_area_struct *vma) { return true; } +static inline void vma_lock_free(struct vm_area_struct *vma) {} + +#endif /* CONFIG_PER_VMA_LOCK */ + struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) { struct vm_area_struct *vma; vma = kmem_cache_alloc(vm_area_cachep, GFP_KERNEL); - if (vma) - vma_init(vma, mm); + if (!vma) + return NULL; + + vma_init(vma, mm); + if (!vma_lock_alloc(vma)) { + kmem_cache_free(vm_area_cachep, vma); + return NULL; + } + return vma; } @@ -465,24 +501,30 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) { struct vm_area_struct *new = kmem_cache_alloc(vm_area_cachep, GFP_KERNEL); - if (new) { - ASSERT_EXCLUSIVE_WRITER(orig->vm_flags); - ASSERT_EXCLUSIVE_WRITER(orig->vm_file); - /* - * orig->shared.rb may be modified concurrently, but the clone - * will be reinitialized. - */ - data_race(memcpy(new, orig, sizeof(*new))); - INIT_LIST_HEAD(&new->anon_vma_chain); - vma_init_lock(new); - dup_anon_vma_name(orig, new); + if (!new) + return NULL; + + ASSERT_EXCLUSIVE_WRITER(orig->vm_flags); + ASSERT_EXCLUSIVE_WRITER(orig->vm_file); + /* + * orig->shared.rb may be modified concurrently, but the clone + * will be reinitialized. + */ + data_race(memcpy(new, orig, sizeof(*new))); + if (!vma_lock_alloc(new)) { + kmem_cache_free(vm_area_cachep, new); + return NULL; } + INIT_LIST_HEAD(&new->anon_vma_chain); + dup_anon_vma_name(orig, new); + return new; } void __vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); + vma_lock_free(vma); kmem_cache_free(vm_area_cachep, vma); } @@ -493,7 +535,7 @@ static void vm_area_free_rcu_cb(struct rcu_head *head) vm_rcu); /* The vma should not be locked while being destroyed. */ - VM_BUG_ON_VMA(rwsem_is_locked(&vma->lock), vma); + VM_BUG_ON_VMA(rwsem_is_locked(&vma->vm_lock->lock), vma); __vm_area_free(vma); } #endif @@ -3089,6 +3131,9 @@ void __init proc_caches_init(void) NULL); vm_area_cachep = KMEM_CACHE(vm_area_struct, SLAB_PANIC|SLAB_ACCOUNT); +#ifdef CONFIG_PER_VMA_LOCK + vma_lock_cachep = KMEM_CACHE(vma_lock, SLAB_PANIC|SLAB_ACCOUNT); +#endif mmap_init(); nsproxy_cache_init(); }