From patchwork Mon Feb 27 17:36:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 62037 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2552351wrd; Mon, 27 Feb 2023 09:39:44 -0800 (PST) X-Google-Smtp-Source: AK7set/OQyz19/mv2Z0Bvh0aLwN5zNng2tIAkyke39t90U/bNuMp4Zr5IoGUbfAhtetuP1rjp2iI X-Received: by 2002:a17:907:f81:b0:877:a3c4:807b with SMTP id kb1-20020a1709070f8100b00877a3c4807bmr36273190ejc.68.1677519584238; Mon, 27 Feb 2023 09:39:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677519584; cv=none; d=google.com; s=arc-20160816; b=UW2Qw86Hn3urJZSITLj/kaz1ZvdQcrU2+rt+XcMVP+4BPgY5etD162tw1PP4lbooEA 1j72RMK0lRj0uVWBMnYdaH91ULFDjwsi17pVga4x6dcQAEjX5G4EuE7FwIXymOMGDTJG O3xKJ/pseWUYY2ugCdHyNLkVx8hj+u59l8pHmSMxlPGuf8GF2sepID0WC8iztWZe6jjY W8C157tG96LSw4qr79CBkrRPf5ejJiJ3T8lOMOpMFHbbo8NTuPeewS4wq4HtAZAT8Gq0 n6yAaRO0dMmpsn2Y8FO3MsUY25rHed0NCERVvs4U8NlCLwJ9RBXehAQDPn0cqgyoEccA 4PmQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=iRRoEkIKRwl2QQHnq1OJVyPp4L6+mQc5CWukcTSQqHI=; b=zAY9fMnfTwHZf4yNQnH//HswKF6+unMiBtLw579KZ5/5goDEGESMAFmmqy3knNpTX2 U7EWibBdAHp04YMpu8sYE/SEfbMVDF2VB+OM3LiDIOqAiJ9tI4H1+DcPMizHWOL02FS9 vjBluy1kEOgA8h2xBF3mZA+qhuLQivIy8FUqnpfQ6Snc6oYG2ieTNlk5Nyd7n+V+9FY2 al6mUFgSWWIVFG1kIr2TAakU/FnE2IRN+koVENJb9mnhZ0ozKAHai42Sdic0Qjqrkvt5 xxzCEOg3TSRsYUZIYOOl/HlteKZ1OvNQQWpyEf7oY0Qh/Gz2A5ZkfxVsHQdNUjtf00Oj h89w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=M2xu6PJm; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id e12-20020aa7d7cc000000b004ace2521ff4si8734039eds.517.2023.02.27.09.39.20; Mon, 27 Feb 2023 09:39:44 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=M2xu6PJm; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229516AbjB0Rio (ORCPT + 99 others); Mon, 27 Feb 2023 12:38:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49102 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230361AbjB0RiB (ORCPT ); Mon, 27 Feb 2023 12:38:01 -0500 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DF8522449A for ; Mon, 27 Feb 2023 09:37:32 -0800 (PST) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-536cad819c7so153281707b3.6 for ; Mon, 27 Feb 2023 09:37:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=iRRoEkIKRwl2QQHnq1OJVyPp4L6+mQc5CWukcTSQqHI=; b=M2xu6PJmgtYFCFjZtLFUAhDJwsRj1PNvn9tpCCynDkmhHKgKWgNMPKAbFeMB0d7vzQ aKVeG6bXcHl5yxZ6JEXjbkVB2BvokKvbI4LCTQCQPJySbeEhyr0HdzjFd7UV7XC52Nqd IM/Y4Y2oz4P4pwkohu9Bqul5qghmRgidkrq66aJWcYZe5Ah2frrlViyJFqWif5WyIuln lwhBindA8HTOjteANHg0VaUlc+wXF2i0bjESG2yKoDnx3/qzuFoEw7K8TYty79uXoeAT DG0JzJNCfRj4E5rbsNquxz9O5F/H/zOUYZHgnCU0K2ioTwSn7zpOFzVjAn3t3qu4Rbvy YYMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=iRRoEkIKRwl2QQHnq1OJVyPp4L6+mQc5CWukcTSQqHI=; b=y9na5U0qDi21qBAsnJHbPzounsTntLvQkhZlXN9o2VPt93mV9Rc1O9exuCbhlOm3Uo fAmH8/SZB0RBdDhAd9weSIzXnzs5xviA670YxGqJUnRfFErNFZWMsye7R2ZOQKOkYAz+ h273258vMQDT7/xiTtbNrrtzN2vM3yzavsHIgx/nwy1hlmDpUBQYtHZYqx3yG+h+gtzq nUzI3d+3q1QGLJnGVXPugWyJwFlo5rq25xRgCQSCQTZE637ia9lMJDcqIxBNtFdLTGhK akKOjdGtrVYY5TvMxI6c17WoDUe0ma0iIKr62hJhqcm3vBsSWM/DLP3X/BZvJaB2o0n1 qNog== X-Gm-Message-State: AO0yUKXYIVpxuDSm/e204c6V8k/q4qSehtgDtX/aWZC9+fAYoXQvtJnc 2UWeaION7JXYzDjv9ip4ESMp3K2u6Yw= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a5b:6cf:0:b0:a03:da3f:3e65 with SMTP id r15-20020a5b06cf000000b00a03da3f3e65mr9154608ybq.6.1677519451327; Mon, 27 Feb 2023 09:37:31 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:22 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-24-surenb@google.com> Subject: [PATCH v4 23/33] mm: introduce lock_vma_under_rcu to be used from arch-specific code From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759006775380777146?= X-GMAIL-MSGID: =?utf-8?q?1759006775380777146?= Introduce lock_vma_under_rcu function to lookup and lock a VMA during page fault handling. When VMA is not found, can't be locked or changes after being locked, the function returns NULL. The lookup is performed under RCU protection to prevent the found VMA from being destroyed before the VMA lock is acquired. VMA lock statistics are updated according to the results. For now only anonymous VMAs can be searched this way. In other cases the function returns NULL. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 3 +++ mm/memory.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 49 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 895bb3950e8a..46d2db743b1a 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -701,6 +701,9 @@ static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) vma->detached = detached; } +struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, + unsigned long address); + #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_init_lock(struct vm_area_struct *vma) {} diff --git a/mm/memory.c b/mm/memory.c index f7f412833e42..bda4c1a991f0 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5221,6 +5221,52 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, } EXPORT_SYMBOL_GPL(handle_mm_fault); +#ifdef CONFIG_PER_VMA_LOCK +/* + * Lookup and lock a VMA under RCU protection. Returned VMA is guaranteed to be + * stable and not isolated. If the VMA is not found or is being modified the + * function returns NULL. + */ +struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, + unsigned long address) +{ + MA_STATE(mas, &mm->mm_mt, address, address); + struct vm_area_struct *vma; + + rcu_read_lock(); +retry: + vma = mas_walk(&mas); + if (!vma) + goto inval; + + /* Only anonymous vmas are supported for now */ + if (!vma_is_anonymous(vma)) + goto inval; + + if (!vma_start_read(vma)) + goto inval; + + /* Check since vm_start/vm_end might change before we lock the VMA */ + if (unlikely(address < vma->vm_start || address >= vma->vm_end)) { + vma_end_read(vma); + goto inval; + } + + /* Check if the VMA got isolated after we found it */ + if (vma->detached) { + vma_end_read(vma); + /* The area was replaced with another one */ + goto retry; + } + + rcu_read_unlock(); + return vma; +inval: + rcu_read_unlock(); + return NULL; +} +#endif /* CONFIG_PER_VMA_LOCK */ + #ifndef __PAGETABLE_P4D_FOLDED /* * Allocate p4d page table.