From patchwork Mon Feb 19 06:04:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 202909 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp1108794dyc; Sun, 18 Feb 2024 22:24:40 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCXZjDvkr7C/wtga+VwPqBDrw+i7UnNtzIlveTK8E2oeIGjTk3MoC/JgJnZHJcd+Fl9is620t9XBIJQiX7tTSWXvZ843Sg== X-Google-Smtp-Source: AGHT+IH5B+JH1TS7q88r8uUhdRJmFQHi2BlvIH47ZEunpY0YjrNmOtiHtJjHdfPrntDfjXNRw3Ou X-Received: by 2002:aa7:d051:0:b0:563:e047:cb9f with SMTP id n17-20020aa7d051000000b00563e047cb9fmr5521215edo.19.1708323879873; Sun, 18 Feb 2024 22:24:39 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708323879; cv=pass; d=google.com; s=arc-20160816; b=Xs2nrUvKcUAUrEKeTvWs3wm18C4cWQzdNksyUeVqi6hdoJX/cy4RWOpdLrQOdc56iu n7MiJ6hVtzcAAyoee5lUufDDtS2+ENESRff8mOP0I6Nd56s+HSKaLjnQVHGFEm+Xv5sB euXorCeMuP9/dEcOKDLfbTaem8L0nCgiV9iZ5N621epKlw7Wd0G1dXHkolCqu7ZMDPPT oghhZtFjhMwJRShoFSY/SMY1FLyIouySvcsECjtYomK6+0EXPYVwNc1mXtk5MOyJFzOR AN4zj1lyhuAmM9HPl1OliD9R/FCuEvu0hOw0+lmyEigiHBLFsgqsbzgBMFwpsZ6wPELl aDNg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-id:precedence:references :in-reply-to:message-id:date:subject:cc:to:from; bh=eRD1/8o+qtEOTAKMY7ucioDJJtMI3+WpTL+ASvHX4To=; fh=I/fJb8ryqmmGEnsMy9AhDuMN2ego/v5YAM2Gbv30rAg=; b=vFUQ+GnA48DMXiFgUVYFvCuABA1564ulltDz0yXE1lFE9hy7CdohzxUkyeNnpJHYw4 4GUU0ZF+qx2drmWvxtJAy++U0D8/AennvHaFI03XkZfmhYOklu6Mr5onhnTgPuShho8h +4tPGBIvprlPogw3bYC+0GDJFELRuOw+BNcau5ooc8PRwvkWNokMgsIW5soZJAA/+U/B 24bQ2WWgmxHgNNxlpli6K94D35MBacOSwxfoFOqfrcCa7YXsSem4LUF7IXF1LsuOrwn0 hyq4cnOHejN5KerR4VfHICLN39+3Xoelb92tsghLjX+/lpbId6dMbJc+aqyBq1ZDxD0n rTIQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=sk.com); spf=pass (google.com: domain of linux-kernel+bounces-70768-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-70768-ouuuleilei=gmail.com@vger.kernel.org" Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id n10-20020a056402060a00b00563e9723bf1si2100993edv.306.2024.02.18.22.24.39 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Feb 2024 22:24:39 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-70768-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=sk.com); spf=pass (google.com: domain of linux-kernel+bounces-70768-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-70768-ouuuleilei=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 6C5821F219A6 for ; Mon, 19 Feb 2024 06:24:39 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id BFD2D20311; Mon, 19 Feb 2024 06:24:25 +0000 (UTC) Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by smtp.subspace.kernel.org (Postfix) with ESMTP id E7C841F941 for ; Mon, 19 Feb 2024 06:24:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=166.125.252.92 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708323864; cv=none; b=NYoi5MXWRcWz/c59WJsBCpsAgZ38zOru8FgvPgbqaKMU0U/ohUJmnsYlNDk+jYuLdqZjYoek1N/8GqYCKE7ToQvXHTYeI5mGOub4m1KVSzGMB/DK2E6K0Vl6ZhdkVyQ6y6MbgKqEktpi4FpDCTYjzM1a3dPE4HjnHT9rVUCMFOY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708323864; c=relaxed/simple; bh=jCdznYeqTxnAdPqUo3goftYDv+lHb7g10nhlhcikrfU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=Upxw7KNFf0yorZByzhQAda0G3BWn/LZ4hgDZ1dVPu2W/hjj5lMJ9VD4PKMadlnG7IhL0xzbDFdXao/6uizHlbLPx7yxtV9zSpWdxxEbASeZ5L8A88maiSavLfzs5OfhIZJPcfBDYqysdRsGVrtmNziaqVUXxjdJyIpQH/dJEDm4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com; spf=pass smtp.mailfrom=sk.com; arc=none smtp.client-ip=166.125.252.92 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sk.com X-AuditID: a67dfc5b-d6dff70000001748-bd-65d2ef615024 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, namit@vmware.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v8 1/8] x86/tlb: Add APIs manipulating tlb batch's arch data Date: Mon, 19 Feb 2024 15:04:00 +0900 Message-Id: <20240219060407.25254-2-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240219060407.25254-1-byungchul@sk.com> References: <20240219060407.25254-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrALMWRmVeSWpSXmKPExsXC9ZZnkW7i+0upBt8fqFrMWb+GzeLzhn9s Fi82tDNafF3/i9ni6ac+FovLu+awWdxb85/V4vyutawWO5buY7K4dGABk8X1XQ8ZLY73HmCy mH/vM5vF5k1TmS2OT5nKaPH7B1DHyVmTWRwEPb639rF47Jx1l91jwaZSj80rtDwW73nJ5LFp VSebx6ZPk9g93p07x+5xYsZvFo95JwM93u+7yuax9ZedR+PUa2wenzfJebyb/5YtgD+KyyYl NSezLLVI3y6BK+P27ZdMBav4Kpb8aGdrYJzG08XIySEhYCKx8vp0Nhj73PUDYDabgLrEjRs/ mUFsEQEziYOtf9i7GLk4mAU+Mkms/t7BApIQFvCRWHz/H5jNIqAq0T3zHROIzStgKnFq2gYW iKHyEqs3HAAaxMHBCTRo90EhkLAQUMmJq5OZQGZKCLSzS/y7s50Vol5S4uCKGywTGHkXMDKs YhTKzCvLTczMMdHLqMzLrNBLzs/dxAiMh2W1f6J3MH66EHyIUYCDUYmHN0PkUqoQa2JZcWXu IUYJDmYlEV73pgupQrwpiZVVqUX58UWlOanFhxilOViUxHmNvpWnCAmkJ5akZqemFqQWwWSZ ODilGhjl/X6+u7dnlu2ZRwY71ty++sdgFYPOt4qkaT4eF6ZceLywbZVx62wRNpHKr36577VM HOqfH5NYG3WYYcd2tT/x5VIPJ1n6X0ubFbeo7NPtlvVRm9JPy/fNcN6gnb434MgDB8XHt3kZ nl55pz61IK+E33KRjPr+6JxL+Q5XbjKXB5fUKl7bv++oEktxRqKhFnNRcSIAuMXYj4MCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrOLMWRmVeSWpSXmKPExsXC5WfdrJv4/lKqwZevUhZz1q9hs/i84R+b xYsN7YwWX9f/YrZ4+qmPxeLw3JOsFpd3zWGzuLfmP6vF+V1rWS12LN3HZHHpwAImi+u7HjJa HO89wGQx/95nNovNm6YyWxyfMpXR4vcPoI6TsyazOAh5fG/tY/HYOesuu8eCTaUem1doeSze 85LJY9OqTjaPTZ8msXu8O3eO3ePEjN8sHvNOBnq833eVzWPxiw9MHlt/2Xk0Tr3G5vF5k5zH u/lv2QIEorhsUlJzMstSi/TtErgybt9+yVSwiq9iyY92tgbGaTxdjJwcEgImEueuH2ADsdkE 1CVu3PjJDGKLCJhJHGz9w97FyMXBLPCRSWL19w4WkISwgI/E4vv/wGwWAVWJ7pnvmEBsXgFT iVPTNrBADJWXWL3hANAgDg5OoEG7DwqBhIWASk5cncw0gZFrASPDKkaRzLyy3MTMHFO94uyM yrzMCr3k/NxNjMDgXlb7Z+IOxi+X3Q8xCnAwKvHwZohcShViTSwrrsw9xCjBwawkwuvedCFV iDclsbIqtSg/vqg0J7X4EKM0B4uSOK9XeGqCkEB6YklqdmpqQWoRTJaJg1OqgdHK2n/Sdu7V /xe/iok3ct0vt4ZJpKPoyqFNhhYaFd6TWU5/dn/l+ObD9QxrvaN3Z2S4VmhcMhZ8uNh1OZPF +tiAU83Hln/jTmg6bfLN66XQtfu7VkQd4vyxcIPwk+UvT1VHrNiVrfz8zdUHSuvnP128TV9B L1eKZX+k3f8PxRppdi5Vaz3SlfuUWIozEg21mIuKEwGcMv88agIAAA== X-CFilter-Loop: Reflected Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791307420636793121 X-GMAIL-MSGID: 1791307420636793121 This is a preparation for migrc mechanism that needs to recognize read-only TLB entries during batched TLB flush by separating tlb batch's arch data into two, one is for read-only entries and the other is for writable ones, and merging those two when needed. Migrc also needs to optimize CPUs to flush by clearing ones that have already performed TLB flush needed. To support it, added APIs manipulating arch data for x86. Signed-off-by: Byungchul Park --- arch/x86/include/asm/tlbflush.h | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 25726893c6f4..fa7e16dbeb44 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -5,6 +5,7 @@ #include #include #include +#include #include #include @@ -293,6 +294,23 @@ static inline void arch_flush_tlb_batched_pending(struct mm_struct *mm) extern void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); +static inline void arch_tlbbatch_clear(struct arch_tlbflush_unmap_batch *batch) +{ + cpumask_clear(&batch->cpumask); +} + +static inline void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + cpumask_or(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} + +static inline bool arch_tlbbatch_done(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + return cpumask_andnot(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} + static inline bool pte_flags_need_flush(unsigned long oldflags, unsigned long newflags, bool ignore_access) From patchwork Mon Feb 19 06:04:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 202901 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp1103035dyc; Sun, 18 Feb 2024 22:04:48 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCV+XOzGoBnsh/JPfGAlYDHdabDoNtdyDvmm0JWt+tR5id4S0agQKc7FAsrO1gnsvW5WFY3OEwhE0ilKT4A7zLKu7DdeiA== X-Google-Smtp-Source: AGHT+IGAFOjxZtimVMhrCgjGHMFAhgJwYAgcU9BZwQYgwoizyk4lqRrDQPgp/7sacyWZdIIys/Fz X-Received: by 2002:a17:906:b885:b0:a3e:9952:e13f with SMTP id hb5-20020a170906b88500b00a3e9952e13fmr894143ejb.13.1708322688436; Sun, 18 Feb 2024 22:04:48 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708322688; cv=pass; d=google.com; s=arc-20160816; b=EUjPtjrm7CCgpVxDGD3yDwgNmZ6X3elaxNPLamRzX7GMeNaJmBuoHl2Xvrks8te3Pl L8lhmpHmPoLnLL6mqnllC5Bccx+SkwVEq/p1oxBiLWIxYwSTcrAStEB5a2bsXKuIibIo qVklE0jCpZTbXbf7o9J7hglRUf/gkOQAoFPKMU2RMYdSuWUqVJiKUIAZpJWK7o8cwSop +BmujEEniXvTBMcykcMZxud4DISEssqPwAZjj6UeraaIgFv3juB8k8n5J6YYuaSEz71y JBWigOayePm6sXbfRRhq18Tnr2/gSs/Ti38ttzDY5eVMUP064gYZ6SaMgM6YdKJFbmxZ k8Bg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-id:precedence:references :in-reply-to:message-id:date:subject:cc:to:from; bh=23+JjFD9uRPbyMWGqJTLiQYjg6Gkh8EQ/TB1jpahIpQ=; fh=I/fJb8ryqmmGEnsMy9AhDuMN2ego/v5YAM2Gbv30rAg=; b=q+CIBdBbo0MG6FLaIEy4ca4fvjnRVmilcKgrrzD8KGd/N5SUpfBPMSf1WytZe8P1Wr CRmQaXXI1uQjBQkkdKolZzTbFf6m1fVpFJivJR4HfgV0LO0+7X/eBYNNa/Touj7gDOP3 NpNN0YcZbuAYTS0sYe07YO1EjZ4B/k+GQo0fh6vtAN8PHdsooht0JxSkb5lASXD0faLD X1maj07znlQm6BRyWb8vLvjnTAt6p3atlGTAwnz0NhaBdlMuywv9FZqQ4s20h9TBcSbn CpYy/mdyUecLe+WHpwOpjgAEOUlsOksc8x5jhqd+4lxW4a3/OLmwMua7C1i2QaLKmnl7 3EvA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=sk.com); spf=pass (google.com: domain of linux-kernel+bounces-70748-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-70748-ouuuleilei=gmail.com@vger.kernel.org" Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id i19-20020a170906115300b00a3d7bcfb9acsi2184966eja.841.2024.02.18.22.04.48 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Feb 2024 22:04:48 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-70748-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=sk.com); spf=pass (google.com: domain of linux-kernel+bounces-70748-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-70748-ouuuleilei=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 0FFC11F22701 for ; Mon, 19 Feb 2024 06:04:48 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id E4EBE2030D; Mon, 19 Feb 2024 06:04:30 +0000 (UTC) Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 4D4D8200AA for ; Mon, 19 Feb 2024 06:04:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=166.125.252.92 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708322666; cv=none; b=HD8P4n+G5YoXTx/CujnkNvnoJYCHbSb+KcpVhwDDRahpyGK5HHK26OGHsy6ckmHvZwdm2u6dx9tcTydS/ILlhGoZoIQ6xy91Zfd/sC/nlta9NyFJl2XUBGMWSZfnHuSAsPp0hnHEseq9RZeG60Oa1mfxeIhRBfAsQRTfzA0tllQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708322666; c=relaxed/simple; bh=uWSWIWkD3opthclWyKeivuFFBZgLyDaadr0FCXFpy+I=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=Br4HMSObp2CE7G1sotOuPOWx9UQHis0QNGhpJ6YmImet3U9vfqGAQKH3R1Uk8rsSXsaQj5g8tRKj8pkNGS6paCmH6H/8FSyy7eXUBcisLPEuR9nJ8AWSY1iwlcCjWqElS+9JvPdrw3Q3jtC5yP/wrULBKhFeHMWosHHb4Y46KlY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com; spf=pass smtp.mailfrom=sk.com; arc=none smtp.client-ip=166.125.252.92 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sk.com X-AuditID: a67dfc5b-d6dff70000001748-c2-65d2ef61c11d From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, namit@vmware.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v8 2/8] arm64: tlbflush: Add APIs manipulating tlb batch's arch data Date: Mon, 19 Feb 2024 15:04:01 +0900 Message-Id: <20240219060407.25254-3-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240219060407.25254-1-byungchul@sk.com> References: <20240219060407.25254-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrPLMWRmVeSWpSXmKPExsXC9ZZnkW7i+0upBhuf6lnMWb+GzeLzhn9s Fi82tDNafF3/i9ni6ac+FovLu+awWdxb85/V4vyutawWO5buY7K4dGABk8X1XQ8ZLY73HmCy mH/vM5vF5k1TmS2OT5nKaPH7B1DHyVmTWRwEPb639rF47Jx1l91jwaZSj80rtDwW73nJ5LFp VSebx6ZPk9g93p07x+5xYsZvFo95JwM93u+7yuax9ZedR+PUa2wenzfJebyb/5YtgD+KyyYl NSezLLVI3y6BK2NP30P2gtlcFZvet7M1MO7l6GLk5JAQMJH4dPspK4y95ORdJhCbTUBd4saN n8wgtoiAmcTB1j/sXYxcHMwCH5kkVn/vYAFJCAuESPT/PMUGYrMIqErMfbwHLM4rYCqxdhdE s4SAvMTqDQeAbA4OTqBBuw8KgYSFgEpOXJ3MBFHSzC7xZ3I2hC0pcXDFDZYJjLwLGBlWMQpl 5pXlJmbmmOhlVOZlVugl5+duYgRGw7LaP9E7GD9dCD7EKMDBqMTDmyFyKVWINbGsuDL3EKME B7OSCK9704VUId6UxMqq1KL8+KLSnNTiQ4zSHCxK4rxG38pThATSE0tSs1NTC1KLYLJMHJxS DYy1bcxrf3Of9latynp5JX5OwH8TP445ypzzMn6uEvv8pMn7oOe5WTtXMNhrMm1puKsSe0Ps ++oXUmdd1d4kf+NOkD9yoOXNxS9HFuu/fNb/ZbGv7sXjQmxLLkSZzD50UJ912TUmx+Z9a0MV zVX1vn/5PPPPZ2eWhviTMQf3vhA1OlBwdN5p3ZAZSizFGYmGWsxFxYkAZHZPDYICAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrOLMWRmVeSWpSXmKPExsXC5WfdrJv4/lKqwbw+BYs569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfVdDxkt jvceYLKYf+8zm8XmTVOZLY5Pmcpo8fsHUMfJWZNZHIQ8vrf2sXjsnHWX3WPBplKPzSu0PBbv ecnksWlVJ5vHpk+T2D3enTvH7nFixm8Wj3knAz3e77vK5rH4xQcmj62/7Dwap15j8/i8Sc7j 3fy3bAECUVw2Kak5mWWpRfp2CVwZe/oeshfM5qrY9L6drYFxL0cXIyeHhICJxJKTd5lAbDYB dYkbN34yg9giAmYSB1v/sHcxcnEwC3xkklj9vYMFJCEsECLR//MUG4jNIqAqMffxHrA4r4Cp xNpdEM0SAvISqzccALI5ODiBBu0+KAQSFgIqOXF1MtMERq4FjAyrGEUy88pyEzNzTPWKszMq 8zIr9JLzczcxAoN7We2fiTsYv1x2P8QowMGoxMObIXIpVYg1say4MvcQowQHs5IIr3vThVQh 3pTEyqrUovz4otKc1OJDjNIcLErivF7hqQlCAumJJanZqakFqUUwWSYOTqkGRskSc8uF23zK TO9Y9P8PCgibwjbDKcaIebG+E+vNFGGO52H7Snq22SdZCeTnTHLJUNkZE6nTt327UoeyVST/ Q4vy3BnzOwPWBU96bZky69A7d6fP85y05u6t6bmTsVtOcknqevGPp31u17GWnxZ+n3lN+r3x xNaTrTuVHjTyy3oGfn3E4xCoxFKckWioxVxUnAgAtvSl22oCAAA= X-CFilter-Loop: Reflected Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791306170919048039 X-GMAIL-MSGID: 1791306170919048039 This is a preparation for migrc mechanism that requires to manipulate tlb batch's arch data. Even though arm64 does nothing with it, arch with CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH should provide the APIs. Signed-off-by: Byungchul Park --- arch/arm64/include/asm/tlbflush.h | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index bb2c2833a987..4f2094843e7a 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -328,6 +328,25 @@ static inline void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) dsb(ish); } +static inline void arch_tlbbatch_clear(struct arch_tlbflush_unmap_batch *batch) +{ + /* nothing to do */ +} + +static inline void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + /* nothing to do */ +} + +static inline bool arch_tlbbatch_done(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + /* nothing to do */ + + return false; +} + /* * This is meant to avoid soft lock-ups on large TLB flushing ranges and not * necessarily a performance improvement. From patchwork Mon Feb 19 06:04:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 202902 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp1103163dyc; Sun, 18 Feb 2024 22:05:06 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCUHhYxDGdPzXqx/HZQjdb/CHnjZpVQGObiwIgPrtxXL2TxWSsoCNi+N4faXxgajh23Y76NTKziD2Yl3KWLJNPueS8lluA== X-Google-Smtp-Source: AGHT+IHV70Hf6Tlp85t26axkqePmxthKYH8u8RQfIHgyaJbqeZBXSrNDWdBQS7tXXZRnG4QoU2GF X-Received: by 2002:ac8:5787:0:b0:42d:ab95:a55e with SMTP id v7-20020ac85787000000b0042dab95a55emr16744023qta.49.1708322706772; Sun, 18 Feb 2024 22:05:06 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708322706; cv=pass; d=google.com; s=arc-20160816; b=SPPn2hSWV7M3QOY0FY9llTXm6vsp+5IMTZQRX5iFmFcqd+wvzvtMR3cuE+f/LdIX4x KuewFDBlxYA1yU3IJw0XUlmd+GLdO/aI2oJKGV6roFENY/MYOigB3ba4hrqaW5RMhk65 9go5sPCdrpJT1d21XyfoE6tsK31KTu7yvrSMGTl2NwGWwR6f9ZQEpuJ57KI5JIAGQFWu +RJBAAqJsEXLzywpJCPIFWBteZZYwKQnQBkalzHFLAfWh1UdkqOJrsBU8LqwiOQ6M+Wc Jb7XhNw9E96mBxtjTrcJGQmPyQUt/hPfuHOpbJbbcy66f9u1O2stqhJGudU961GKtHv6 X3DQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-id:precedence:references :in-reply-to:message-id:date:subject:cc:to:from; bh=deHIYqkuaMMowWAwSsScVOmIndLCnwrqbZwxQiYz+QA=; fh=I/fJb8ryqmmGEnsMy9AhDuMN2ego/v5YAM2Gbv30rAg=; b=HDkZhLgacnhNVK/KPHy6sL0CHifYbp2t7Ku9UCm2dzg97xg8qUpyXkWDYwLlfHOX8P LNINHtmhhm+JgFthxHV2xZToeYsAIz9Sn5YWs/e/QibcF1uag68cQfoAMasyEcLCV0FI si3EueyRKeSBbDqeTjTuSzP8yB7KwAuvMUICzQnhrh/rwJPod8et5NFVOrMQrGu5Ah1V HYfHKEJFuRi1K3ongjw//GTrVqiXDBHs/+aVbHtXzJlN+PPAaktzpvADEKHM3Tsj6DK1 /XE2grXK5TUfYNvsZBkbNHWoe9FAymWH/huHhWhLkwgddNzTu9tftjdlOVbgjyAG6oCs rpXw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=sk.com); spf=pass (google.com: domain of linux-kernel+bounces-70750-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-70750-ouuuleilei=gmail.com@vger.kernel.org" Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id h9-20020ac85149000000b0042e1845dbc1si369076qtn.77.2024.02.18.22.05.06 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Feb 2024 22:05:06 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-70750-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=sk.com); spf=pass (google.com: domain of linux-kernel+bounces-70750-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-70750-ouuuleilei=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 8FEC61C21057 for ; Mon, 19 Feb 2024 06:05:06 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id A0018208DA; Mon, 19 Feb 2024 06:04:33 +0000 (UTC) Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 3BBB6200BA for ; Mon, 19 Feb 2024 06:04:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=166.125.252.92 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708322671; cv=none; b=ZXopp4TV7ji4rCVJ8vvP0w2EST2l2liQTN02uImbGotTmToSpYb974eVbXDAzKnZ3LbDlYat7ilgz170b2oN3VP842di8dHJmAIt1Te0vqc3WlySNVFe5qgAa6g/6VSWZYa/1FvnI+GJXgV2gUekIYgOg/P795/cE21W8mTvup8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708322671; c=relaxed/simple; bh=WHuF8WryLfu8JX/60dlBPk8CVS136b4yvUfGBE1UZcQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=ThTTRtG8QA70afD+s9595ZI540IyZAfWVcpjlcIwHKGlXxklEd3cFgT93G2rFZhfjzOaplZH0xzglxc4dBkNgthAlA1XbdQ5bEGkQRCiDqTJuPD8kt4rNPJ5urP+gvmH2RmpXlGQJeUn0mU/kpJitaisqLYyOOODiQIrZqh/Huw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com; spf=pass smtp.mailfrom=sk.com; arc=none smtp.client-ip=166.125.252.92 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sk.com X-AuditID: a67dfc5b-d6dff70000001748-c7-65d2ef6120f8 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, namit@vmware.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v8 3/8] mm/rmap: Recognize read-only TLB entries during batched TLB flush Date: Mon, 19 Feb 2024 15:04:02 +0900 Message-Id: <20240219060407.25254-4-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240219060407.25254-1-byungchul@sk.com> References: <20240219060407.25254-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrPLMWRmVeSWpSXmKPExsXC9ZZnkW7i+0upBhsW6lvMWb+GzeLzhn9s Fi82tDNafF3/i9ni6ac+FovLu+awWdxb85/V4vyutawWO5buY7K4dGABk8X1XQ8ZLY73HmCy mH/vM5vF5k1TmS2OT5nKaPH7B1DHyVmTWRwEPb639rF47Jx1l91jwaZSj80rtDwW73nJ5LFp VSebx6ZPk9g93p07x+5xYsZvFo95JwM93u+7yuax9ZedR+PUa2wenzfJebyb/5YtgD+KyyYl NSezLLVI3y6BK+PrPKeC49IVHz7tYGxgfCDWxcjJISFgIrHl3SQmGPv4yimsIDabgLrEjRs/ mUFsEQEziYOtf9i7GLk4mAU+Mkms/t7BApIQFoiUOLTpPJjNIqAq8fTlH7BmXgFTiRuf9kAN lZdYveEA0CAODk6gQbsPCoGEhYBKTlydzAQyU0KgmV3iyNE2Foh6SYmDK26wTGDkXcDIsIpR KDOvLDcxM8dEL6MyL7NCLzk/dxMjMBqW1f6J3sH46ULwIUYBDkYlHt4MkUupQqyJZcWVuYcY JTiYlUR43ZsupArxpiRWVqUW5ccXleakFh9ilOZgURLnNfpWniIkkJ5YkpqdmlqQWgSTZeLg lGpgjNytd05he82hCWc/NTn08igeecM5/fwcB7FbUsf2pPCW+HP1GyfoCsd1XXA40yVUV9h7 0ne6BdefOZUnI9jNcisFirZL2/ZbnuS1/iPOGbUo0qQyZyFXZPBx8xkf05Yv4t6mwyqqHCcs fjI2sWC1/5NPzjVvfW/a+8qmfzuVtUjcPPnxn3dKLMUZiYZazEXFiQD2/cjxggIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrGLMWRmVeSWpSXmKPExsXC5WfdrJv4/lKqQfNKZYs569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfVdDxkt jvceYLKYf+8zm8XmTVOZLY5Pmcpo8fsHUMfJWZNZHIQ8vrf2sXjsnHWX3WPBplKPzSu0PBbv ecnksWlVJ5vHpk+T2D3enTvH7nFixm8Wj3knAz3e77vK5rH4xQcmj62/7Dwap15j8/i8Sc7j 3fy3bAECUVw2Kak5mWWpRfp2CVwZX+c5FRyXrvjwaQdjA+MDsS5GTg4JAROJ4yunsILYbALq Ejdu/GQGsUUEzCQOtv5h72Lk4mAW+Mgksfp7BwtIQlggUuLQpvNgNouAqsTTl3/AmnkFTCVu fNrDBDFUXmL1hgNAgzg4OIEG7T4oBBIWAio5cXUy0wRGrgWMDKsYRTLzynITM3NM9YqzMyrz Miv0kvNzNzECQ3tZ7Z+JOxi/XHY/xCjAwajEw5shcilViDWxrLgy9xCjBAezkgive9OFVCHe lMTKqtSi/Pii0pzU4kOM0hwsSuK8XuGpCUIC6YklqdmpqQWpRTBZJg5OqQbG5swyM7PeKvlC BZMJuvN+7F/6aNG+y5VFn/fe63+0VGOC6RLGaxOauVS28YckLCgpFLU7GXu3oaZA9g3DA1lW pobWLS0neyb/byu+U51Y/u/k2TKfz2G3k2xWnBU+YF/kEVtxWfzns9+b+HQSTe7kzlWcav/l mOqHvrZaF4ZLZ1aFa/14sJRPiaU4I9FQi7moOBEANl9Y0GkCAAA= X-CFilter-Loop: Reflected Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791306190676133414 X-GMAIL-MSGID: 1791306190676133414 Functionally, no change. This is a preparation for migrc mechanism that requires to recognize read-only TLB entries and makes use of them to batch more aggressively. Plus, the newly introduced API, fold_ubc() will be used by migrc mechanism when manipulating tlb batch data. Signed-off-by: Byungchul Park --- include/linux/sched.h | 1 + mm/internal.h | 4 ++++ mm/rmap.c | 31 ++++++++++++++++++++++++++++++- 3 files changed, 35 insertions(+), 1 deletion(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 292c31697248..0317e7a65151 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1328,6 +1328,7 @@ struct task_struct { #endif struct tlbflush_unmap_batch tlb_ubc; + struct tlbflush_unmap_batch tlb_ubc_ro; /* Cache last used pipe for splice(): */ struct pipe_inode_info *splice_pipe; diff --git a/mm/internal.h b/mm/internal.h index b61034bd50f5..b880f1e78700 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -923,6 +923,7 @@ extern struct workqueue_struct *mm_percpu_wq; void try_to_unmap_flush(void); void try_to_unmap_flush_dirty(void); void flush_tlb_batched_pending(struct mm_struct *mm); +void fold_ubc(struct tlbflush_unmap_batch *dst, struct tlbflush_unmap_batch *src); #else static inline void try_to_unmap_flush(void) { @@ -933,6 +934,9 @@ static inline void try_to_unmap_flush_dirty(void) static inline void flush_tlb_batched_pending(struct mm_struct *mm) { } +static inline void fold_ubc(struct tlbflush_unmap_batch *dst, struct tlbflush_unmap_batch *src) +{ +} #endif /* CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ extern const struct trace_print_flags pageflag_names[]; diff --git a/mm/rmap.c b/mm/rmap.c index 7a27a2b41802..da36f23ff7b0 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -605,6 +605,28 @@ struct anon_vma *folio_lock_anon_vma_read(struct folio *folio, } #ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH + +void fold_ubc(struct tlbflush_unmap_batch *dst, + struct tlbflush_unmap_batch *src) +{ + if (!src->flush_required) + return; + + /* + * Fold src to dst. + */ + arch_tlbbatch_fold(&dst->arch, &src->arch); + dst->writable = dst->writable || src->writable; + dst->flush_required = true; + + /* + * Reset src. + */ + arch_tlbbatch_clear(&src->arch); + src->flush_required = false; + src->writable = false; +} + /* * Flush TLB entries for recently unmapped pages from remote CPUs. It is * important if a PTE was dirty when it was unmapped that it's flushed @@ -614,7 +636,9 @@ struct anon_vma *folio_lock_anon_vma_read(struct folio *folio, void try_to_unmap_flush(void) { struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + struct tlbflush_unmap_batch *tlb_ubc_ro = ¤t->tlb_ubc_ro; + fold_ubc(tlb_ubc, tlb_ubc_ro); if (!tlb_ubc->flush_required) return; @@ -645,13 +669,18 @@ void try_to_unmap_flush_dirty(void) static void set_tlb_ubc_flush_pending(struct mm_struct *mm, pte_t pteval, unsigned long uaddr) { - struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + struct tlbflush_unmap_batch *tlb_ubc; int batch; bool writable = pte_dirty(pteval); if (!pte_accessible(mm, pteval)) return; + if (pte_write(pteval) || writable) + tlb_ubc = ¤t->tlb_ubc; + else + tlb_ubc = ¤t->tlb_ubc_ro; + arch_tlbbatch_add_pending(&tlb_ubc->arch, mm, uaddr); tlb_ubc->flush_required = true; From patchwork Mon Feb 19 06:04:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 202904 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp1103628dyc; Sun, 18 Feb 2024 22:06:15 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWqFA/Q7O398MFdjq9LXkG1oOzosIYK6SIs3wkpGIgfVNSBYYh46OdSF9JnO3vo0wzBgqbsYFEQft/jzRYevpQLk9opDw== X-Google-Smtp-Source: AGHT+IG3VGuxIzgjtew1mbKlitlgwHv+hRtcHhOQdwFqPIYOD4xsMHD/zro0p4EHcIN47Cq2apLR X-Received: by 2002:a17:906:c10f:b0:a38:107a:94e9 with SMTP id do15-20020a170906c10f00b00a38107a94e9mr15492266ejc.2.1708322775748; Sun, 18 Feb 2024 22:06:15 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708322775; cv=pass; d=google.com; s=arc-20160816; b=o1/PACyf30Dz15cCTbsxBH/Fi2+pnfo2DvVcFTN53saHNmXPv8kvUU4L9YQmo4cGhq p+U/sgBBGb0zpb/JUtrCvU7eqQpdbFp+qW6YXPOqVa1wKqvnzXo2fY0u8ev/ZWpMQqJU S8mRNbY3QIpd4d+j/SR4XgQlLlG4NtkGBIZ5IhMziFFUQDTtgGXu3QBBFqx1+ZDthx4+ qdQtNx0cjrJEdFRHhJhQn4N4/1Y4Qs6C1JBuV3AktBFeYeX5FrXXh0kbtdy4dBqhN2Vo xyEnZJjMrUDAMEa89s9E/JXcONWexTOKsnbV8libB8fLHENjfOzuxK2dG/te4MqsZliE 1iqQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-id:precedence:references :in-reply-to:message-id:date:subject:cc:to:from; bh=0Xe+86Sg3SvMdEShlZyUiGPZdigVYaGkrosOSTaTIuA=; fh=I/fJb8ryqmmGEnsMy9AhDuMN2ego/v5YAM2Gbv30rAg=; b=RTiwlHT1mR0UmKopKVwTC/M4JM0AA6+mJZ4912BFz/WuRHjnEmn8v/FUanFNv67g+2 9207llYUwi9vujaEv+5ivEPKiJXMC1lZY1q8Wm1Q+7YczoBELyPV7EDIp3lMCUAAadGD UGfRQUB3Mz6/Gk5WAD385nMHCpxCFFvv56HByVOThF8kAUDLJiyK8nSm/Ya+rHSZklSr 77HAVn73aoM2p28xq3e9P+BtAQMYGnc5OiyNNbhNXmlhry0jhIq4sUxrJHbVTdpqiRcF 5OQqQokKT/EE5D+taaptycxDTyBjwa/FeliZ8a2bGd0NSuxgeu3pw0kytb3zlR0f8fcu HoYw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=sk.com); spf=pass (google.com: domain of linux-kernel+bounces-70754-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-70754-ouuuleilei=gmail.com@vger.kernel.org" Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id cw21-20020a170906479500b00a3eabd8fc4csi310716ejc.1028.2024.02.18.22.06.15 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Feb 2024 22:06:15 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-70754-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=sk.com); spf=pass (google.com: domain of linux-kernel+bounces-70754-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-70754-ouuuleilei=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 34FD51F224E2 for ; Mon, 19 Feb 2024 06:06:15 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 07FDF225CC; Mon, 19 Feb 2024 06:04:43 +0000 (UTC) Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by smtp.subspace.kernel.org (Postfix) with ESMTP id E32A820B21 for ; Mon, 19 Feb 2024 06:04:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=166.125.252.92 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708322677; cv=none; b=pZA+WkgyUIjpJ+lEYOBsp8bE3ftfv79aRB5gLUkNRuugMZIuZvuQGwTFEQ0Wli9Yt8mEvtXqSROpIwTQnmrw2xssb5XN3th/A9lbyTEtw/6sxL2pvOCvIqhiPI1EQBFeaHtNNa5NGiiis6MWJq0mU8SosKUCGvfLV6ZqgRJ51bk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708322677; c=relaxed/simple; bh=UUxcFDjs3C/AhgUuMtqawhuQpp/F0UoyeqgQnOj5J+k=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=RkAUBD++Hdi20dUzG0oI+9KvlqDJX2vS3lUaMgXPNW8anb8Vd6+by6RlHMcuWY5nLgpUPoBA3Xa6cyC3W1t/D30K9gGRHGlA7FFn5xLUuLfK0RkRNfpuBy/MxWQA7yeLbuMnUjxoPedqYkqpzFRj2PTKOV1Z5Skzr+1zqCNouY8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com; spf=pass smtp.mailfrom=sk.com; arc=none smtp.client-ip=166.125.252.92 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sk.com X-AuditID: a67dfc5b-d6dff70000001748-dc-65d2ef61c3ba From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, namit@vmware.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v8 7/8] mm: Defer TLB flush by keeping both src and dst folios at migration Date: Mon, 19 Feb 2024 15:04:06 +0900 Message-Id: <20240219060407.25254-8-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240219060407.25254-1-byungchul@sk.com> References: <20240219060407.25254-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrALMWRmVeSWpSXmKPExsXC9ZZnkW7S+0upBu1TmSzmrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeL6roeMFsd7DzBZ zL/3mc1i86apzBbHp0xltPj9A6jj5KzJLA6CHt9b+1g8ds66y+6xYFOpx+YVWh6L97xk8ti0 qpPNY9OnSewe786dY/c4MeM3i8e8k4Ee7/ddZfPY+svOo3HqNTaPz5vkPN7Nf8sWwB/FZZOS mpNZllqkb5fAlbH91wmWgtcrGStm/Q1tYFzfxtjFyMkhIWAise7TFbYuRg4we97uEpAwm4C6 xI0bP5lBbBEBM4mDrX/Yuxi5OJgFPjJJrP7ewQKSEBaIlmibfIIdxGYRUJU4/2MKmM0rYCrx /PxcFoj58hKrNxxgBpnPCTRo90EhkLAQUMmJq5OZQGZKCHSzS+w4N40dol5S4uCKGywTGHkX MDKsYhTKzCvLTczMMdHLqMzLrNBLzs/dxAiMh2W1f6J3MH66EHyIUYCDUYmHN0PkUqoQa2JZ cWXuIUYJDmYlEV73pgupQrwpiZVVqUX58UWlOanFhxilOViUxHmNvpWnCAmkJ5akZqemFqQW wWSZODilGhhlki7mKrxadd0s/2DUKskwRZb11onHzizeYbjM/O3jp/GsmqZpD1zP3VaKy1yt dfm/bNO10H2dmzI+td7Or99bIDvlR8cBaSanG1VSIbNdllQ/mJv7bZvChnO3Jmr885OaHZjO +HlH3ZyCLHuJuGdb/lnv417wMDmqKXSaeJz7ncovhsZzj9xVYinOSDTUYi4qTgQAanq46IMC AAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrOLMWRmVeSWpSXmKPExsXC5WfdrJv4/lKqwan1RhZz1q9hs/i84R+b xYsN7YwWX9f/YrZ4+qmPxeLw3JOsFpd3zWGzuLfmP6vF+V1rWS12LN3HZHHpwAImi+u7HjJa HO89wGQx/95nNovNm6YyWxyfMpXR4vcPoI6TsyazOAh5fG/tY/HYOesuu8eCTaUem1doeSze 85LJY9OqTjaPTZ8msXu8O3eO3ePEjN8sHvNOBnq833eVzWPxiw9MHlt/2Xk0Tr3G5vF5k5zH u/lv2QIEorhsUlJzMstSi/TtErgytv86wVLweiVjxay/oQ2M69sYuxg5OCQETCTm7S7pYuTk YBNQl7hx4ycziC0iYCZxsPUPexcjFwezwEcmidXfO1hAEsIC0RJtk0+wg9gsAqoS539MAbN5 BUwlnp+fC1YjISAvsXrDAWaQ+ZxAg3YfFAIJCwGVnLg6mWkCI9cCRoZVjCKZeWW5iZk5pnrF 2RmVeZkVesn5uZsYgcG9rPbPxB2MXy67H2IU4GBU4uHNELmUKsSaWFZcmXuIUYKDWUmE173p QqoQb0piZVVqUX58UWlOavEhRmkOFiVxXq/w1AQhgfTEktTs1NSC1CKYLBMHp1QD42oP7wvZ /mkTd9+/ycd2d+9sM50jR/oaZFk/JiUv/Cl/4O3t+Xd9Fd4b3tqSfH9VYolayoe5cgd1Lt1Z 1MO7Xco0oJaDr0L79+v9TrGub/1z5j869kLk+Y3yqOpJcyKL9p4NU1z1vn+ixII5Z4p/R7or WdUGdca6vFr0N/nc5/367NNXPI1bxa3EUpyRaKjFXFScCAAxSKAMagIAAA== X-CFilter-Loop: Reflected Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791306263006710707 X-GMAIL-MSGID: 1791306263006710707 Implementation of MIGRC mechanism that stands for 'Migration Read Copy'. We always face the migration overhead at either promotion or demotion, while working with tiered memory e.g. CXL memory and found out TLB shootdown is a quite big one that is needed to get rid of if possible. Fortunately, TLB flush can be defered if both source and destination of folios during migration are kept until all TLB flushes required will have been done, of course, only if the target PTE entries have read-only permission, more precisely speaking, don't have write permission. Otherwise, no doubt the folio might get messed up. To achieve that: 1. For the folios that map only to non-writable TLB entries, prevent TLB flush at migration by keeping both source and destination folios, which will be handled later at a better time. 2. When any non-writable TLB entry changes to writable e.g. through fault handler, give up migrc mechanism so as to perform TLB flush required right away. The following evaluation using XSBench shows the improvement like: 1. itlb flush was reduced by 93.9%. 2. dtlb thread was reduced by 43.5%. 3. stlb flush was reduced by 24.9%. 4. dtlb store misses was reduced by 34.2%. 5. itlb load misses was reduced by 45.5%. 6. The runtime was reduced by 3.5%. --- The measurement result: Architecture - x86_64 QEMU - kvm enabled, host cpu Numa - 2 nodes (16 CPUs 1GB, no CPUs 8GB) Linux Kernel - v6.7, numa balancing tiering on, demotion enabled Benchmark - XSBench -p 100000000 (-p option makes the runtime longer) run 'perf stat' using events: 1) itlb.itlb_flush 2) tlb_flush.dtlb_thread 3) tlb_flush.stlb_any 4) dTLB-load-misses 5) dTLB-store-misses 6) iTLB-load-misses run 'cat /proc/vmstat' and pick: 1) numa_pages_migrated 2) pgmigrate_success 3) nr_tlb_remote_flush 4) nr_tlb_remote_flush_received 5) nr_tlb_local_flush_all 6) nr_tlb_local_flush_one BEFORE - mainline v6.7 ---------------------- $ perf stat -a \ -e itlb.itlb_flush \ -e tlb_flush.dtlb_thread \ -e tlb_flush.stlb_any \ -e dTLB-load-misses \ -e dTLB-store-misses \ -e iTLB-load-misses \ ./XSBench -p 100000000 Performance counter stats for 'system wide': 85647229 itlb.itlb_flush 480981504 tlb_flush.dtlb_thread 323937200 tlb_flush.stlb_any 238381632579 dTLB-load-misses 601514255 dTLB-store-misses 2974157461 iTLB-load-misses 2252.883892112 seconds time elapsed $ cat /proc/vmstat ... numa_pages_migrated 12790664 pgmigrate_success 26835314 nr_tlb_remote_flush 3031412 nr_tlb_remote_flush_received 45234862 nr_tlb_local_flush_all 216584 nr_tlb_local_flush_one 740940 ... AFTER - mainline v6.7 + migrc ----------------------------- $ perf stat -a \ -e itlb.itlb_flush \ -e tlb_flush.dtlb_thread \ -e tlb_flush.stlb_any \ -e dTLB-load-misses \ -e dTLB-store-misses \ -e iTLB-load-misses \ ./XSBench -p 100000000 Performance counter stats for 'system wide': 5240261 itlb.itlb_flush 271581774 tlb_flush.dtlb_thread 243149389 tlb_flush.stlb_any 234502983364 dTLB-load-misses 395673680 dTLB-store-misses 1620215163 iTLB-load-misses 2172.283436287 seconds time elapsed $ cat /proc/vmstat ... numa_pages_migrated 14897064 pgmigrate_success 30825530 nr_tlb_remote_flush 198290 nr_tlb_remote_flush_received 2820156 nr_tlb_local_flush_all 92048 nr_tlb_local_flush_one 741401 ... Signed-off-by: Byungchul Park --- include/linux/mmzone.h | 7 ++ include/linux/sched.h | 8 ++ mm/internal.h | 53 ++++++++ mm/memory.c | 8 ++ mm/migrate.c | 271 +++++++++++++++++++++++++++++++++++++++-- mm/page_alloc.c | 11 +- mm/rmap.c | 12 +- 7 files changed, 358 insertions(+), 12 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 9db36e197712..492111cd1176 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -1002,6 +1002,13 @@ struct zone { /* Zone statistics */ atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS]; atomic_long_t vm_numa_event[NR_VM_NUMA_EVENT_ITEMS]; + +#if defined(CONFIG_MIGRATION) && defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) + /* + * the number of folios pending for TLB flush in the zone + */ + atomic_t migrc_pending_nr; +#endif } ____cacheline_internodealigned_in_smp; enum pgdat_flags { diff --git a/include/linux/sched.h b/include/linux/sched.h index 0317e7a65151..d8c285309a8f 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1330,6 +1330,14 @@ struct task_struct { struct tlbflush_unmap_batch tlb_ubc; struct tlbflush_unmap_batch tlb_ubc_ro; +#if defined(CONFIG_MIGRATION) && defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) + /* + * whether all the mappings of a folio during unmap are read-only + * so that migrc can work on the folio + */ + bool can_migrc; +#endif + /* Cache last used pipe for splice(): */ struct pipe_inode_info *splice_pipe; diff --git a/mm/internal.h b/mm/internal.h index 3be8fd5604e8..ab02cb8306e2 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1284,4 +1284,57 @@ static inline void shrinker_debugfs_remove(struct dentry *debugfs_entry, } #endif /* CONFIG_SHRINKER_DEBUG */ +#if defined(CONFIG_MIGRATION) && defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) +/* + * Reset the indicator indicating there are no writable mappings at the + * beginning of every rmap traverse for unmap. Migrc can work only when + * all the mappings are read-only. + */ +static inline void can_migrc_init(void) +{ + current->can_migrc = true; +} + +/* + * Mark the folio is not applicable to migrc, once it found a writble or + * dirty pte during rmap traverse for unmap. + */ +static inline void can_migrc_fail(void) +{ + current->can_migrc = false; +} + +/* + * Check if all the mappings are read-only and read-only mappings even + * exist. + */ +static inline bool can_migrc_test(void) +{ + return current->can_migrc && current->tlb_ubc_ro.flush_required; +} + +/* + * Return the number of folios pending TLB flush that have yet to get + * freed in the zone. + */ +static inline int migrc_pending_nr_in_zone(struct zone *z) +{ + return atomic_read(&z->migrc_pending_nr); +} + +/* + * Perform TLB flush needed and free the folios under migrc's control. + */ +bool migrc_flush_free_folios(void); +void migrc_flush_start(void); +void migrc_flush_end(struct tlbflush_unmap_batch *batch); +#else /* CONFIG_MIGRATION && CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ +static inline void can_migrc_init(void) {} +static inline void can_migrc_fail(void) {} +static inline bool can_migrc_test(void) { return false; } +static inline int migrc_pending_nr_in_zone(struct zone *z) { return 0; } +static inline bool migrc_flush_free_folios(void) { return false; } +static inline void migrc_flush_start(void) {} +static inline void migrc_flush_end(struct tlbflush_unmap_batch *batch) {} +#endif #endif /* __MM_INTERNAL_H */ diff --git a/mm/memory.c b/mm/memory.c index 6e0712d06cd4..e67de161da8b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3462,6 +3462,14 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) if (vmf->page) folio = page_folio(vmf->page); + /* + * The folio may or may not be one that is under migrc's control + * and about to change its permission from read-only to writable. + * Conservatively give up deferring TLB flush just in case. + */ + if (folio) + migrc_flush_free_folios(); + /* * Shared mapping: we are guaranteed to have VM_WRITE and * FAULT_FLAG_WRITE set at this point. diff --git a/mm/migrate.c b/mm/migrate.c index bbe1ecef4956..cbe5372f159e 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -57,6 +57,194 @@ #include "internal.h" +#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH +static struct tlbflush_unmap_batch migrc_ubc; +static LIST_HEAD(migrc_folios); +static DEFINE_SPINLOCK(migrc_lock); + +static void init_tlb_ubc(struct tlbflush_unmap_batch *ubc) +{ + arch_tlbbatch_clear(&ubc->arch); + ubc->flush_required = false; + ubc->writable = false; +} + +static void migrc_keep_folio(struct folio *f, struct list_head *h) +{ + list_move_tail(&f->lru, h); + folio_get(f); + atomic_inc(&folio_zone(f)->migrc_pending_nr); +} + +static void migrc_undo_folio(struct folio *f) +{ + list_del(&f->lru); + folio_put(f); + atomic_dec(&folio_zone(f)->migrc_pending_nr); +} + +static void migrc_release_folio(struct folio *f) +{ + folio_put_small_nopcp(f); + atomic_dec(&folio_zone(f)->migrc_pending_nr); +} + +/* + * Need to synchronize between TLB flush and managing pending CPUs in + * migrc_ubc. Take a look at the following scenario: + * + * CPU0 CPU1 + * ---- ---- + * TLB flush + * Unmap folios (needing TLB flush) + * Add pending CPUs to migrc_ubc + * Clear the CPUs from migrc_ubc + * + * The pending CPUs added in CPU1 should not be cleared from migrc_ubc + * in CPU0 because the TLB flush for migrc_ubc added in CPU1 has not + * been performed this turn. To avoid this, using 'migrc_flushing' + * variable, prevent adding pending CPUs to migrc_ubc and give up migrc + * mechanism if others are in the middle of TLB flush, like: + * + * CPU0 CPU1 + * ---- ---- + * migrc_flushing++ + * TLB flush + * Unmap folios (needing TLB flush) + * If migrc_flushing == 0: + * Add pending CPUs to migrc_ubc + * Else: <--- hit + * Give up migrc mechanism + * Clear the CPUs from migrc_ubc + * migrc_flush-- + * + * Only the following case would be allowed for migrc mechanism to work: + * + * CPU0 CPU1 + * ---- ---- + * Unmap folios (needing TLB flush) + * If migrc_flushing == 0: <--- hit + * Add pending CPUs to migrc_ubc + * Else: + * Give up migrc mechanism + * migrc_flushing++ + * TLB flush + * Clear the CPUs from migrc_ubc + * migrc_flush-- + */ +static int migrc_flushing; + +static bool migrc_add_pending_ubc(struct tlbflush_unmap_batch *ubc) +{ + struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + unsigned long flags; + + spin_lock_irqsave(&migrc_lock, flags); + if (migrc_flushing) { + spin_unlock_irqrestore(&migrc_lock, flags); + + /* + * Give up migrc mechanism. Just let TLB flush needed + * handled by try_to_unmap_flush() at the caller side. + */ + fold_ubc(tlb_ubc, ubc); + return false; + } + fold_ubc(&migrc_ubc, ubc); + spin_unlock_irqrestore(&migrc_lock, flags); + return true; +} + +static bool migrc_add_pending_folios(struct list_head *folios) +{ + unsigned long flags; + + spin_lock_irqsave(&migrc_lock, flags); + if (migrc_flushing) { + spin_unlock_irqrestore(&migrc_lock, flags); + + /* + * Give up migrc mechanism. The caller should perform + * TLB flush needed using migrc_flush_free_folios() and + * undo some on the folios e.g. restore folios' + * reference count increased by migrc and more. + */ + return false; + } + list_splice(folios, &migrc_folios); + spin_unlock_irqrestore(&migrc_lock, flags); + return true; +} + +void migrc_flush_start(void) +{ + unsigned long flags; + + spin_lock_irqsave(&migrc_lock, flags); + migrc_flushing++; + spin_unlock_irqrestore(&migrc_lock, flags); +} + +void migrc_flush_end(struct tlbflush_unmap_batch *batch) +{ + LIST_HEAD(folios); + struct folio *f, *f2; + unsigned long flags; + + spin_lock_irqsave(&migrc_lock, flags); + if (!arch_tlbbatch_done(&migrc_ubc.arch, &batch->arch)) { + list_splice_init(&migrc_folios, &folios); + migrc_ubc.flush_required = false; + migrc_ubc.writable = false; + } + migrc_flushing--; + spin_unlock_irqrestore(&migrc_lock, flags); + + list_for_each_entry_safe(f, f2, &folios, lru) + migrc_release_folio(f); +} + +bool migrc_flush_free_folios(void) +{ + struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + LIST_HEAD(folios); + struct folio *f, *f2; + unsigned long flags; + bool ret = true; + + spin_lock_irqsave(&migrc_lock, flags); + list_splice_init(&migrc_folios, &folios); + fold_ubc(tlb_ubc, &migrc_ubc); + spin_unlock_irqrestore(&migrc_lock, flags); + + if (list_empty(&folios)) + ret = false; + + try_to_unmap_flush(); + list_for_each_entry_safe(f, f2, &folios, lru) + migrc_release_folio(f); + return ret; +} +#else /* CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ +static void init_tlb_ubc(struct tlbflush_unmap_batch *ubc) +{ +} +static void migrc_keep_folio(struct folio *f, struct list_head *h) +{ +} +static void migrc_undo_folio(struct folio *f) +{ +} +static bool migrc_add_pending_ubc(struct tlbflush_unmap_batch *ubc) +{ + return false; +} +static bool migrc_add_pending_folios(struct list_head *folios) +{ + return false; +} +#endif + bool isolate_movable_page(struct page *page, isolate_mode_t mode) { struct folio *folio = folio_get_nontail_page(page); @@ -1274,7 +1462,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, struct folio *src, struct folio *dst, enum migrate_mode mode, enum migrate_reason reason, - struct list_head *ret) + struct list_head *ret, struct list_head *move_succ) { int rc; int old_page_state = 0; @@ -1321,9 +1509,13 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, /* * A folio that has been migrated has all references removed - * and will be freed. + * and will be freed, unless it's under migrc's control. */ - list_del(&src->lru); + if (move_succ) + migrc_keep_folio(src, move_succ); + else + list_del(&src->lru); + /* Drop an anon_vma reference if we took one */ if (anon_vma) put_anon_vma(anon_vma); @@ -1618,7 +1810,7 @@ static void migrate_folios_move(struct list_head *src_folios, struct list_head *ret_folios, struct migrate_pages_stats *stats, int *retry, int *thp_retry, int *nr_failed, - int *nr_retry_pages) + int *nr_retry_pages, struct list_head *move_succ) { struct folio *folio, *folio2, *dst, *dst2; bool is_thp; @@ -1635,7 +1827,7 @@ static void migrate_folios_move(struct list_head *src_folios, rc = migrate_folio_move(put_new_folio, private, folio, dst, mode, - reason, ret_folios); + reason, ret_folios, move_succ); /* * The rules are: * Success: folio will be freed @@ -1712,17 +1904,34 @@ static int migrate_pages_batch(struct list_head *from, int rc, rc_saved = 0, nr_pages; LIST_HEAD(unmap_folios); LIST_HEAD(dst_folios); + LIST_HEAD(unmap_folios_migrc); + LIST_HEAD(dst_folios_migrc); + LIST_HEAD(move_succ); bool nosplit = (reason == MR_NUMA_MISPLACED); + struct tlbflush_unmap_batch pending_ubc; + struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + struct tlbflush_unmap_batch *tlb_ubc_ro = ¤t->tlb_ubc_ro; + bool do_migrc; + bool migrc_ubc_succ; VM_WARN_ON_ONCE(mode != MIGRATE_ASYNC && !list_empty(from) && !list_is_singular(from)); + /* + * Apply migrc only to numa migration for now. + */ + init_tlb_ubc(&pending_ubc); + do_migrc = IS_ENABLED(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) && + (reason == MR_DEMOTION || reason == MR_NUMA_MISPLACED); + for (pass = 0; pass < nr_pass && retry; pass++) { retry = 0; thp_retry = 0; nr_retry_pages = 0; list_for_each_entry_safe(folio, folio2, from, lru) { + bool can_migrc; + is_large = folio_test_large(folio); is_thp = is_large && folio_test_pmd_mappable(folio); nr_pages = folio_nr_pages(folio); @@ -1752,9 +1961,12 @@ static int migrate_pages_batch(struct list_head *from, continue; } + can_migrc_init(); rc = migrate_folio_unmap(get_new_folio, put_new_folio, private, folio, &dst, mode, reason, ret_folios); + can_migrc = do_migrc && can_migrc_test() && !is_large; + /* * The rules are: * Success: folio will be freed @@ -1800,7 +2012,8 @@ static int migrate_pages_batch(struct list_head *from, /* nr_failed isn't updated for not used */ stats->nr_thp_failed += thp_retry; rc_saved = rc; - if (list_empty(&unmap_folios)) + if (list_empty(&unmap_folios) && + list_empty(&unmap_folios_migrc)) goto out; else goto move; @@ -1814,8 +2027,19 @@ static int migrate_pages_batch(struct list_head *from, stats->nr_thp_succeeded += is_thp; break; case MIGRATEPAGE_UNMAP: - list_move_tail(&folio->lru, &unmap_folios); - list_add_tail(&dst->lru, &dst_folios); + if (can_migrc) { + list_move_tail(&folio->lru, &unmap_folios_migrc); + list_add_tail(&dst->lru, &dst_folios_migrc); + + /* + * Gather ro batch data to add + * to migrc_ubc after unmap. + */ + fold_ubc(&pending_ubc, tlb_ubc_ro); + } else { + list_move_tail(&folio->lru, &unmap_folios); + list_add_tail(&dst->lru, &dst_folios); + } break; default: /* @@ -1829,12 +2053,19 @@ static int migrate_pages_batch(struct list_head *from, stats->nr_failed_pages += nr_pages; break; } + /* + * Done with the current folio. Fold the ro + * batch data gathered, to the normal batch. + */ + fold_ubc(tlb_ubc, tlb_ubc_ro); } } nr_failed += retry; stats->nr_thp_failed += thp_retry; stats->nr_failed_pages += nr_retry_pages; move: + /* Should be before try_to_unmap_flush() */ + migrc_ubc_succ = do_migrc && migrc_add_pending_ubc(&pending_ubc); /* Flush TLBs for all unmapped folios */ try_to_unmap_flush(); @@ -1848,7 +2079,27 @@ static int migrate_pages_batch(struct list_head *from, migrate_folios_move(&unmap_folios, &dst_folios, put_new_folio, private, mode, reason, ret_folios, stats, &retry, &thp_retry, - &nr_failed, &nr_retry_pages); + &nr_failed, &nr_retry_pages, NULL); + migrate_folios_move(&unmap_folios_migrc, &dst_folios_migrc, + put_new_folio, private, mode, reason, + ret_folios, stats, &retry, &thp_retry, + &nr_failed, &nr_retry_pages, migrc_ubc_succ ? + &move_succ : NULL); + } + + /* + * In case that migrc_add_pending_ubc() has been added + * successfully but migrc_add_pending_folios() does not. + */ + if (migrc_ubc_succ && !migrc_add_pending_folios(&move_succ)) { + migrc_flush_free_folios(); + + /* + * Undo src folios that have been successfully added to + * move_succ. + */ + list_for_each_entry_safe(folio, folio2, &move_succ, lru) + migrc_undo_folio(folio); } nr_failed += retry; stats->nr_thp_failed += thp_retry; @@ -1859,6 +2110,8 @@ static int migrate_pages_batch(struct list_head *from, /* Cleanup remaining folios */ migrate_folios_undo(&unmap_folios, &dst_folios, put_new_folio, private, ret_folios); + migrate_folios_undo(&unmap_folios_migrc, &dst_folios_migrc, + put_new_folio, private, ret_folios); return rc; } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 21b8c8cd1673..6ef0c22b1109 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2972,6 +2972,8 @@ bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark, long min = mark; int o; + free_pages += migrc_pending_nr_in_zone(z); + /* free_pages may go negative - that's OK */ free_pages -= __zone_watermark_unusable_free(z, order, alloc_flags); @@ -3066,7 +3068,7 @@ static inline bool zone_watermark_fast(struct zone *z, unsigned int order, long usable_free; long reserved; - usable_free = free_pages; + usable_free = free_pages + migrc_pending_nr_in_zone(z); reserved = __zone_watermark_unusable_free(z, 0, alloc_flags); /* reserved may over estimate high-atomic reserves. */ @@ -3273,6 +3275,13 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags, gfp_mask)) { int ret; + if (migrc_pending_nr_in_zone(zone) && + migrc_flush_free_folios() && + zone_watermark_fast(zone, order, mark, + ac->highest_zoneidx, + alloc_flags, gfp_mask)) + goto try_this_zone; + if (has_unaccepted_memory()) { if (try_to_accept_memory(zone, order)) goto try_this_zone; diff --git a/mm/rmap.c b/mm/rmap.c index b484d659d0c1..39ab0d64665a 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -642,7 +642,9 @@ void try_to_unmap_flush(void) if (!tlb_ubc->flush_required) return; + migrc_flush_start(); arch_tlbbatch_flush(&tlb_ubc->arch); + migrc_flush_end(tlb_ubc); arch_tlbbatch_clear(&tlb_ubc->arch); tlb_ubc->flush_required = false; tlb_ubc->writable = false; @@ -677,9 +679,15 @@ static void set_tlb_ubc_flush_pending(struct mm_struct *mm, pte_t pteval, if (!pte_accessible(mm, pteval)) return; - if (pte_write(pteval) || writable) + if (pte_write(pteval) || writable) { tlb_ubc = ¤t->tlb_ubc; - else + + /* + * Migrc cannot work with the folio, once it found a + * writable or dirty mapping on it. + */ + can_migrc_fail(); + } else tlb_ubc = ¤t->tlb_ubc_ro; arch_tlbbatch_add_pending(&tlb_ubc->arch, mm, uaddr);