From patchwork Mon Feb 26 03:06:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 206185 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp1842069dyb; Sun, 25 Feb 2024 19:07:15 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCX+Hg82ki/ibAC7JhHL9Jur1eJh9KnobRW/SxojNaRmZPIQ2wE013r2T1klg4/ndxyWF0l8qEZabA1zUCZx+FlDyro3Jg== X-Google-Smtp-Source: AGHT+IFnzr16hY93pPL0T5sgPLZvFgE0h9AgkM1QFBRmFGN4rOK7UJyh+QtfDj45LUjMV/3idSF+ X-Received: by 2002:a05:6512:12c5:b0:512:fd8d:98e2 with SMTP id p5-20020a05651212c500b00512fd8d98e2mr1169158lfg.69.1708916835483; Sun, 25 Feb 2024 19:07:15 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708916835; cv=pass; d=google.com; s=arc-20160816; b=TkUmXKhx8V2Z22HtC/d2u65Oi9k2UziWI3QajuVYYHPPgoXvya2fVc/LC64p7uWmn2 w8j/GXMcZSvrem7bEIQjme0zAGOz+WMVNXgOXuJIL26bdiRH3PPYSk1KtPNSD+HEAFNC Y5eVZP9za0QZHM6wp3dGJjmPcgZ7M+JN1NJCYm2g+OLFgT+Ng8IXlMlZGsoplmmftQiA cW+BCnB8io7+YXYKsSae5ZGtrjwbj00UmqbSTiGZ6u/kzWdBJkhQdDBRZwh3hpPzo+Mz CD+ZlZe3HUhpE/rHEOk/bN3+QIqL7SpPG+xQQc4F7H3OAwxbG0P6IPUZC1NAtL3KSf0D eWXw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-id:precedence:references :in-reply-to:message-id:date:subject:cc:to:from; bh=eRD1/8o+qtEOTAKMY7ucioDJJtMI3+WpTL+ASvHX4To=; fh=ZH+M10KtJZRqSjFjp1+uAYTvsWtGX6qxh/YkifVvH1o=; b=hmItM97pO2R7ljyY7Sv2AlACbHr6sz7GjlJzhYdmWN0AnEB9kx/mVVmpFE//3K+CWE wtQEs67E0vshucV3n4SYwxjOpTTOdaQFPqtNZL2fjLY6ySFiTN5DdR++sD12MEE+YXD6 C9ayS8XJcENEDFjNIcqzoyTL3DnD9rnzIHMnLQJv3EyMVm4tKxEFmmdoS4paePnRzTFa o8TR1yKvsBiowqr271suRr6uibmGpcybyK4K1nB/2nevBB9QTBxnHcVf1wTTU7+fSpws pY7HqKSEl/Dyznnvl9+zTMR5mUdJKICqxjvfpAmX178lZQRb/keEoWrSv2Sn9iXeM1dE 6A+g==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=sk.com); spf=pass (google.com: domain of linux-kernel+bounces-80445-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-80445-ouuuleilei=gmail.com@vger.kernel.org" Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id hk8-20020a170906c9c800b00a3ec5a0dd7asi1737264ejb.231.2024.02.25.19.07.15 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 25 Feb 2024 19:07:15 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-80445-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=sk.com); spf=pass (google.com: domain of linux-kernel+bounces-80445-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-80445-ouuuleilei=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 165811F212B5 for ; Mon, 26 Feb 2024 03:07:14 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 9D79D1B7E3; Mon, 26 Feb 2024 03:06:38 +0000 (UTC) Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by smtp.subspace.kernel.org (Postfix) with ESMTP id AD32EC8C7 for ; Mon, 26 Feb 2024 03:06:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=166.125.252.92 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708916797; cv=none; b=CTh+5r9QRhqlJT4dvgxTnakuz8vL8c7WLU2ZLIve1sm4LDVFHAyfhQgeSkMl9nKt+bkUaD4lDTQH2vpKxmokol2M8LqlzHsfVT4cVtpFPMPAh/vjCb033E6BoeqyrTEVYWfMoQ03RQxpbWYFQ3Zng2zJv9unj5z/HW1XWK9MusM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708916797; c=relaxed/simple; bh=jCdznYeqTxnAdPqUo3goftYDv+lHb7g10nhlhcikrfU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=Sf7taoxI7lfq2xqYOtjQRgBkR3pBwTmHm3WJ4qdcxiUlm5KRA+aE26wHqVp6SUNVz65kMq8LdNUxjZRcPc1QSIE9iCQ4FMk3XBXI6V6v3cJf5nA1o3ed84qmCbB4oE5FEelrHdug5U06HROnSKgqarEX1p/1Y5WKvrnAfd9SMHw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com; spf=pass smtp.mailfrom=sk.com; arc=none smtp.client-ip=166.125.252.92 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sk.com X-AuditID: a67dfc5b-d6dff70000001748-27-65dc002fde00 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [RESEND PATCH v8 1/8] x86/tlb: Add APIs manipulating tlb batch's arch data Date: Mon, 26 Feb 2024 12:06:06 +0900 Message-Id: <20240226030613.22366-2-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240226030613.22366-1-byungchul@sk.com> References: <20240226030613.22366-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrKLMWRmVeSWpSXmKPExsXC9ZZnka4+w51UgxdrDS3mrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeJ47wEmi/n3PrNZ bN40ldni+JSpjBa/fwAVn5w1mcVBwON7ax+Lx85Zd9k9Fmwq9di8Qstj8Z6XTB6bVnWyeWz6 NInd4925c+weJ2b8ZvGYdzLQ4/2+q2weW3/ZeTROvcbm8XmTXABfFJdNSmpOZllqkb5dAlfG 7dsvmQpW8VUs+dHO1sA4jaeLkZNDQsBEYvfzC0ww9tqN61lBbDYBdYkbN34yg9giAmYSB1v/ sIPYzAJ3mSQO9LOB2MICwRIvLqwCq2ERUJXYf+QMI4jNK2Aqsf7lfKiZ8hKrNxwAquHg4ASa 8/6/IUhYCKjkb+9VoHIuoJLPbBJP3k9lhaiXlDi44gbLBEbeBYwMqxiFMvPKchMzc0z0Mirz Miv0kvNzNzECA39Z7Z/oHYyfLgQfYhTgYFTi4V3w4XaqEGtiWXFl7iFGCQ5mJRHecJmbqUK8 KYmVValF+fFFpTmpxYcYpTlYlMR5jb6VpwgJpCeWpGanphakFsFkmTg4pRoYuVz6bB9Jzriy X8VMq9yz6u+fJIHjl14utgvvvFLz+vG1NI32YqXOFV2vJ50/LLDgY+mTsIK2WZfiXQ9Ylqms VXUwXRJdkpDhGPPLS7Y//Rjzt6/fqvSSU7dythmc7Z21dKH+x6TGK6rHZ768+PTCrp+Xt1SI 7mS0ckp/fjTwS0lLMWtOttYjJZbijERDLeai4kQANQUZ3XgCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrDLMWRmVeSWpSXmKPExsXC5WfdrKvPcCfV4EKrusWc9WvYLD5v+Mdm 8WJDO6PF1/W/mC2efupjsTg89ySrxeVdc9gs7q35z2pxftdaVosdS/cxWVw6sIDJ4njvASaL +fc+s1ls3jSV2eL4lKmMFr9/ABWfnDWZxUHQ43trH4vHzll32T0WbCr12LxCy2PxnpdMHptW dbJ5bPo0id3j3blz7B4nZvxm8Zh3MtDj/b6rbB6LX3xg8tj6y86jceo1No/Pm+QC+KO4bFJS czLLUov07RK4Mm7ffslUsIqvYsmPdrYGxmk8XYycHBICJhJrN65nBbHZBNQlbtz4yQxiiwiY SRxs/cMOYjML3GWSONDPBmILCwRLvLiwCqyGRUBVYv+RM4wgNq+AqcT6l/OZIGbKS6zecACo hoODE2jO+/+GIGEhoJK/vVcZJzByLWBkWMUokplXlpuYmWOqV5ydUZmXWaGXnJ+7iREYxstq /0zcwfjlsvshRgEORiUe3gUfbqcKsSaWFVfmHmKU4GBWEuENl7mZKsSbklhZlVqUH19UmpNa fIhRmoNFSZzXKzw1QUggPbEkNTs1tSC1CCbLxMEp1cA475ylarTBfS6DxKkLHoZMUPNqZ6s3 WRz5f/LmNdb/N0pH8RdsrHrgPX93wpIshQkpPFs3NSscv3LOdKJmk32G8v6Ie4zpVY93Fs+p VWzeqnklMXD2Go0du2Zfr86Snpe0Lv3CTd5bydZLFm+Ry/BRTzY/NyfwyaOXi75ZlzX8/qAU UhH7KyVciaU4I9FQi7moOBEA6KTWhl8CAAA= X-CFilter-Loop: Reflected Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791929179821397209 X-GMAIL-MSGID: 1791929179821397209 This is a preparation for migrc mechanism that needs to recognize read-only TLB entries during batched TLB flush by separating tlb batch's arch data into two, one is for read-only entries and the other is for writable ones, and merging those two when needed. Migrc also needs to optimize CPUs to flush by clearing ones that have already performed TLB flush needed. To support it, added APIs manipulating arch data for x86. Signed-off-by: Byungchul Park --- arch/x86/include/asm/tlbflush.h | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 25726893c6f4..fa7e16dbeb44 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -5,6 +5,7 @@ #include #include #include +#include #include #include @@ -293,6 +294,23 @@ static inline void arch_flush_tlb_batched_pending(struct mm_struct *mm) extern void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); +static inline void arch_tlbbatch_clear(struct arch_tlbflush_unmap_batch *batch) +{ + cpumask_clear(&batch->cpumask); +} + +static inline void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + cpumask_or(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} + +static inline bool arch_tlbbatch_done(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + return cpumask_andnot(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} + static inline bool pte_flags_need_flush(unsigned long oldflags, unsigned long newflags, bool ignore_access) From patchwork Mon Feb 26 03:06:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 206186 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp1842224dyb; Sun, 25 Feb 2024 19:07:41 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVF2JT6mxRcjVB9YeAJvutYvPvV/qD1Qw3LzFBuIJI7ly9EVbBksaApRwdAT7ffHJd0Xtce6pzx8iaMSQXaIP3udqJ0gQ== X-Google-Smtp-Source: AGHT+IFM85JwozUuT49+NzYv6d00RkeWm06U3qTXVSrmktczVEPncgOUgpSX488h23th1xlIolQ0 X-Received: by 2002:a05:6214:3110:b0:690:79e:78f9 with SMTP id ks16-20020a056214311000b00690079e78f9mr3772697qvb.18.1708916861063; Sun, 25 Feb 2024 19:07:41 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708916861; cv=pass; d=google.com; s=arc-20160816; b=sjsYqSVfxo1pgRzqf17jtISDn1xtfSuItDbckDNltbVQsfrexahiACL3LSK0EP7TtG sp6JqIy3tZrHYmBcbOpYAfXGgPV1PqQlRvaYBaGKnw1mFy1ejKMFZdRpCC6oCQhV+/sA LuqJlU7Fb1ay1QOWkhxGcCyr7lCVRlhPxCXjStRvXfW/iUgXuy+CyvDB4S36Xz+ckVP8 y+HctDY7b/kkMsvC5O9A8CYn+0GdSCBXzJ+M+SAcaG0FqlBqrfRTkjYL7HCLFDu4sj0b wYPVfS06dFEfQGQLZ7IAW5pDxDiqeGs2D1hj/XEVjmAsny87vcOB9fyaBtgptDXC9Qz/ Fnrw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-id:precedence:references :in-reply-to:message-id:date:subject:cc:to:from; bh=deHIYqkuaMMowWAwSsScVOmIndLCnwrqbZwxQiYz+QA=; fh=ZH+M10KtJZRqSjFjp1+uAYTvsWtGX6qxh/YkifVvH1o=; b=Nv9cYReIIyVHBcsM59kR/lEvo1yTpp/IQR5AjfpDp5ImQxO+vb42vQhpx4QvB7EMwF //DAERSx6614NRIF8MVlyBB/Apbxo7xonLl7keRs+v8XnbW05kL+yyB/q2S8tnqUYN5a rCfqE8FtdcPRN8uE0ofjSaHYbblbHUfXDvJlIbysJ4OMw9AQ4abXhlcqCSn3WyFOn4es RcJ7sVdNuLXl/a7tw+pWKhQMfEhJ+aauAI7ZPccUUYw+KSiH7MllwXvXelAe761a+35p Wchm3ZbURe2uQKzzBuVM+TTAPD+sjX5hg4JxrzbTcHc+OpwN7yAZnE0XszU+YcqWC0rM oVCA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=sk.com); spf=pass (google.com: domain of linux-kernel+bounces-80446-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-80446-ouuuleilei=gmail.com@vger.kernel.org" Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id 9-20020a0562140d4900b0068d14cbf19csi4254635qvr.345.2024.02.25.19.07.40 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 25 Feb 2024 19:07:41 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-80446-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=sk.com); spf=pass (google.com: domain of linux-kernel+bounces-80446-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-80446-ouuuleilei=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id D5EB91C21223 for ; Mon, 26 Feb 2024 03:07:40 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 1FC461BDCE; Mon, 26 Feb 2024 03:06:40 +0000 (UTC) Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by smtp.subspace.kernel.org (Postfix) with ESMTP id DEBE411CB8 for ; Mon, 26 Feb 2024 03:06:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=166.125.252.92 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708916798; cv=none; b=Vp5omee6YqENxz431cg4Hbnsk164/JZThK33ddKXlYwPv846H/drYnxbHdX1kxYS49yrycmv6rlhTcZ0C/4AlT/jjarlVJqEaGZATiqNpI/wwHZiBC+ni0yV2QZ7ErcTqrzj2gBQu7GGW1uLDDSqevLS54wpkBAQcn5DWE5tLKY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708916798; c=relaxed/simple; bh=WHuF8WryLfu8JX/60dlBPk8CVS136b4yvUfGBE1UZcQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=ZMezWL0LSDe/gpYg54k3BXfDn2v4zXWJrI+PNuQR6ESQhwXXn6n7eoiqCOBFohdzifItL4h4rgC4DnYY+lllEP3+Wfr6IA94H2bPYfYIH7pIifWlc83pXJUTiowpElDdbI5Nk/QcH5EQ9/5Jt0dtRxkolZWfMzT9owmXrSh6nSA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com; spf=pass smtp.mailfrom=sk.com; arc=none smtp.client-ip=166.125.252.92 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sk.com X-AuditID: a67dfc5b-d6dff70000001748-32-65dc002f6be1 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [RESEND PATCH v8 3/8] mm/rmap: Recognize read-only TLB entries during batched TLB flush Date: Mon, 26 Feb 2024 12:06:08 +0900 Message-Id: <20240226030613.22366-4-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240226030613.22366-1-byungchul@sk.com> References: <20240226030613.22366-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrCLMWRmVeSWpSXmKPExsXC9ZZnoa4+w51Ug8NXLS3mrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeJ47wEmi/n3PrNZ bN40ldni+JSpjBa/fwAVn5w1mcVBwON7ax+Lx85Zd9k9Fmwq9di8Qstj8Z6XTB6bVnWyeWz6 NInd4925c+weJ2b8ZvGYdzLQ4/2+q2weW3/ZeTROvcbm8XmTXABfFJdNSmpOZllqkb5dAlfG 13lOBcelKz582sHYwPhArIuRk0NCwETiy66TrDD2mZe/GUFsNgF1iRs3fjKD2CICZhIHW/+w g9jMAneZJA70s4HYwgIJEg+nzWMCsVkEVCUOn4Go5xUwlTg8YSHUTHmJ1RsOAMU5ODiB5rz/ bwgSFgIq+dt7FWgVF1DJezaJE38OMELUS0ocXHGDZQIj7wJGhlWMQpl5ZbmJmTkmehmVeZkV esn5uZsYgWG/rPZP9A7GTxeCDzEKcDAq8fAu+HA7VYg1say4MvcQowQHs5IIb7jMzVQh3pTE yqrUovz4otKc1OJDjNIcLErivEbfylOEBNITS1KzU1MLUotgskwcnFINjDVc17fHpHDM1vip 1c0s8nhT0EpZt02LFl3InXXdeEMsC/f+t82Fu1aonZkVb8v5j6vsV57bUY6Fwc8sGdZGfbZf 67JQoGOSxJMTjK/S174xDou/mSInM0+fadZjNdZpYaq7p9u0lO65sqdb2MLRuNIzdU/+LsUa jfOWLlrblJwULwpN0GEMUWIpzkg01GIuKk4EALCYTpR3AgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrNLMWRmVeSWpSXmKPExsXC5WfdrKvPcCfV4PZ3PYs569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfHeA0wW 8+99ZrPYvGkqs8XxKVMZLX7/ACo+OWsyi4Ogx/fWPhaPnbPusnss2FTqsXmFlsfiPS+ZPDat 6mTz2PRpErvHu3Pn2D1OzPjN4jHvZKDH+31X2TwWv/jA5LH1l51H49RrbB6fN8kF8Edx2aSk 5mSWpRbp2yVwZXyd51RwXLriw6cdjA2MD8S6GDk5JARMJM68/M0IYrMJqEvcuPGTGcQWETCT ONj6hx3EZha4yyRxoJ8NxBYWSJB4OG0eE4jNIqAqcfgMRD2vgKnE4QkLWSFmykus3nAAKM7B wQk05/1/Q5CwEFDJ396rjBMYuRYwMqxiFMnMK8tNzMwx1SvOzqjMy6zQS87P3cQIDOJltX8m 7mD8ctn9EKMAB6MSD++CD7dThVgTy4orcw8xSnAwK4nwhsvcTBXiTUmsrEotyo8vKs1JLT7E KM3BoiTO6xWemiAkkJ5YkpqdmlqQWgSTZeLglGpgDLfQf3L+fFbP6oyusxLzttvzPdJcJr97 ScDHgA/qEcmONkwr7/z9cOH7/fm1pZfvz/sgFbvYeTG3S/t1QbmXp82FA56ceN7pYWzxU3fH /6YgsQkdSlmvwkVXvJ8iMtPoStaDkj7+locqHgmzLln5O58/HNJVepZVq0Q2YNnvJ8/cDc5M Wqv7U4mlOCPRUIu5qDgRANgmFc1eAgAA X-CFilter-Loop: Reflected Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791929206443852528 X-GMAIL-MSGID: 1791929206443852528 Functionally, no change. This is a preparation for migrc mechanism that requires to recognize read-only TLB entries and makes use of them to batch more aggressively. Plus, the newly introduced API, fold_ubc() will be used by migrc mechanism when manipulating tlb batch data. Signed-off-by: Byungchul Park --- include/linux/sched.h | 1 + mm/internal.h | 4 ++++ mm/rmap.c | 31 ++++++++++++++++++++++++++++++- 3 files changed, 35 insertions(+), 1 deletion(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 292c31697248..0317e7a65151 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1328,6 +1328,7 @@ struct task_struct { #endif struct tlbflush_unmap_batch tlb_ubc; + struct tlbflush_unmap_batch tlb_ubc_ro; /* Cache last used pipe for splice(): */ struct pipe_inode_info *splice_pipe; diff --git a/mm/internal.h b/mm/internal.h index b61034bd50f5..b880f1e78700 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -923,6 +923,7 @@ extern struct workqueue_struct *mm_percpu_wq; void try_to_unmap_flush(void); void try_to_unmap_flush_dirty(void); void flush_tlb_batched_pending(struct mm_struct *mm); +void fold_ubc(struct tlbflush_unmap_batch *dst, struct tlbflush_unmap_batch *src); #else static inline void try_to_unmap_flush(void) { @@ -933,6 +934,9 @@ static inline void try_to_unmap_flush_dirty(void) static inline void flush_tlb_batched_pending(struct mm_struct *mm) { } +static inline void fold_ubc(struct tlbflush_unmap_batch *dst, struct tlbflush_unmap_batch *src) +{ +} #endif /* CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ extern const struct trace_print_flags pageflag_names[]; diff --git a/mm/rmap.c b/mm/rmap.c index 7a27a2b41802..da36f23ff7b0 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -605,6 +605,28 @@ struct anon_vma *folio_lock_anon_vma_read(struct folio *folio, } #ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH + +void fold_ubc(struct tlbflush_unmap_batch *dst, + struct tlbflush_unmap_batch *src) +{ + if (!src->flush_required) + return; + + /* + * Fold src to dst. + */ + arch_tlbbatch_fold(&dst->arch, &src->arch); + dst->writable = dst->writable || src->writable; + dst->flush_required = true; + + /* + * Reset src. + */ + arch_tlbbatch_clear(&src->arch); + src->flush_required = false; + src->writable = false; +} + /* * Flush TLB entries for recently unmapped pages from remote CPUs. It is * important if a PTE was dirty when it was unmapped that it's flushed @@ -614,7 +636,9 @@ struct anon_vma *folio_lock_anon_vma_read(struct folio *folio, void try_to_unmap_flush(void) { struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + struct tlbflush_unmap_batch *tlb_ubc_ro = ¤t->tlb_ubc_ro; + fold_ubc(tlb_ubc, tlb_ubc_ro); if (!tlb_ubc->flush_required) return; @@ -645,13 +669,18 @@ void try_to_unmap_flush_dirty(void) static void set_tlb_ubc_flush_pending(struct mm_struct *mm, pte_t pteval, unsigned long uaddr) { - struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + struct tlbflush_unmap_batch *tlb_ubc; int batch; bool writable = pte_dirty(pteval); if (!pte_accessible(mm, pteval)) return; + if (pte_write(pteval) || writable) + tlb_ubc = ¤t->tlb_ubc; + else + tlb_ubc = ¤t->tlb_ubc_ro; + arch_tlbbatch_add_pending(&tlb_ubc->arch, mm, uaddr); tlb_ubc->flush_required = true; From patchwork Mon Feb 26 03:06:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 206187 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp1842240dyb; Sun, 25 Feb 2024 19:07:42 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCXrTtbLSl8n/dXzL1CDQK7OY/PGxsZzgspbXNPVrPubntJmJWXJlSGc4iUjFCGSplQyLky/15hlj8VYFvBqJcn+fgsBDA== X-Google-Smtp-Source: AGHT+IEcEJ7E+5iM/eaO912dWrC2VM68/D1KFd9joi84U8fiB0T/6AK1BlCNa8V/yDuYX5UOEjVx X-Received: by 2002:a17:906:f286:b0:a3f:4d95:ff69 with SMTP id gu6-20020a170906f28600b00a3f4d95ff69mr3582643ejb.38.1708916862458; Sun, 25 Feb 2024 19:07:42 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708916862; cv=pass; d=google.com; s=arc-20160816; b=s8hm9R3iRPCIBRWXFcaCEz0AvrC0l+QrsOviTfT8vo+Luk7isr5WCCI3zANACs/H7p I4+09B1dXXPFbxb0VggDAV/h5iyG35QwnN+YJqLwiwkQJNfPbMNdKGsvbbrEMqYZUEsS oAH51+/Mwr4I4kBk9dde2AwbyENWknS6y/fkaOWoCtzD+WCFlASD2FjPoU32GHU6K2lF XRd+dc1faB+2VZ8wLnHKV02I9/reEyejUZZaxqM/ZOjWvTprg5gqjgZHvjXI2zsT5xCz v4hpfSNNHVETNoL8BlkZO2mES6Yh5ad5rLnCvR00RRHeu90OFikVGKtmsbryoLrrSP/q SgRQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-id:precedence:references :in-reply-to:message-id:date:subject:cc:to:from; bh=JD5mtfEWkRjuaUV9XVtob8Xf345yQ1B8wbzU19SAuqU=; fh=ZH+M10KtJZRqSjFjp1+uAYTvsWtGX6qxh/YkifVvH1o=; b=guYNtKqkC3vQwGRiPlLSDStTHkeppJiI53k7jOGqvKq3VYY65zpJAj8YJVxgkSF34G 7bR6HQYRNZK2v1z8a9Dcg0xOucYaAALYO2DuEWoFXL6/ESS57PQUX5/ZH8c4/94Hjnbh woPAoE08+H9Ww4AiUMBTy358cYckqSLxtATL6o9g4XdzscUT/ggL80XPgsRa++zH86Wu MPmhdkyQCTYfNOtQSQKKe9ObH5kahA1rixf8OT0dWZGj9H0+82RoF047MfN70o9C0H3B l9lLQDImqZRGA9xK+XZTvV/44mZUM0duLIytwxT0yS5CqJP3R0/IsQPg2Od8RbJLO+Lx jRXA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=sk.com); spf=pass (google.com: domain of linux-kernel+bounces-80447-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-80447-ouuuleilei=gmail.com@vger.kernel.org" Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id x16-20020a170906711000b00a3e3c7a4969si1692952ejj.881.2024.02.25.19.07.42 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 25 Feb 2024 19:07:42 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-80447-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=sk.com); spf=pass (google.com: domain of linux-kernel+bounces-80447-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-80447-ouuuleilei=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 17D7B1F21396 for ; Mon, 26 Feb 2024 03:07:42 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 313BC1BDD0; Mon, 26 Feb 2024 03:06:40 +0000 (UTC) Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 77BFD14A98 for ; Mon, 26 Feb 2024 03:06:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=166.125.252.92 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708916798; cv=none; b=oIca/1t4UWGcj1esqA+48on9q56Cr8AJoik/GwqyHTIY/czuDyNhMbBCWtNpMTWncN4AhNMJ+3/3Q/uglNkHX8F6Ju2l0qcbjEOEj9XkpRY7ST0ScPg65EJjADrJ5tShfgY48AXwcYy4YqSz+ImjM3v5QPbkMp8oCJwLo6eMOiw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708916798; c=relaxed/simple; bh=o7dB2cnQ4DdAIbh4UuP8+5q3qGaOJJ7booLY7nqK98M=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=E8iuh9JmgfZvHFcQ8KfA8h5TEwL/YzYtBmTSxJq70JWeCJOALNmtw67eBLdBsOSCB5zBidO2AeGJ3t26RXxXVKUSw4VLCBhMRYGNfmHN0FHIk0GjDI2I5kx6aB5PoU5389s1xpHwDQh1HI6WXf8zaCrKPft4RCeCYG3b3ju73y4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com; spf=pass smtp.mailfrom=sk.com; arc=none smtp.client-ip=166.125.252.92 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sk.com X-AuditID: a67dfc5b-d6dff70000001748-37-65dc002fa99c From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [RESEND PATCH v8 4/8] x86/tlb, mm/rmap: Separate arch_tlbbatch_clear() out of arch_tlbbatch_flush() Date: Mon, 26 Feb 2024 12:06:09 +0900 Message-Id: <20240226030613.22366-5-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240226030613.22366-1-byungchul@sk.com> References: <20240226030613.22366-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrMLMWRmVeSWpSXmKPExsXC9ZZnka4Bw51Ug69hFnPWr2Gz+LzhH5vF iw3tjBZf1/9itnj6qY/F4vKuOWwW99b8Z7U4v2stq8WOpfuYLC4dWMBkcbz3AJPF/Huf2Sw2 b5rKbHF8ylRGi98/gIpPzprM4iDg8b21j8Vj56y77B4LNpV6bF6h5bF4z0smj02rOtk8Nn2a xO7x7tw5do8TM36zeMw7Gejxft9VNo+tv+w8GqdeY/P4vEkugC+KyyYlNSezLLVI3y6BK+Po nXtsBXM4K87/msrUwHiJvYuRk0NCwETiy4WXrDD2i387wOJsAuoSN278ZAaxRQTMJA62/gGL MwvcZZI40M/WxcjBISyQI7HvaxpImEVAVWL3z6dgJbwCphJrX55mgRgpL7F6wwFmkHJOoDHv /xuChIWASv72XmXsYuQCKvnMJrGqfSYjRL2kxMEVN1gmMPIuYGRYxSiUmVeWm5iZY6KXUZmX WaGXnJ+7iREY9Mtq/0TvYPx0IfgQowAHoxIP74IPt1OFWBPLiitzDzFKcDArifCGy9xMFeJN SaysSi3Kjy8qzUktPsQozcGiJM5r9K08RUggPbEkNTs1tSC1CCbLxMEp1cCoaNMdsOrB5+U9 3HeTbrvb1c+c+/aLqmXDNIFjH5Wu+Nz++fVNOtPBeha2+pf9+rLaV7a+fzFj9s3EELXrDVzM olvN17zXLw336ExxtExueXIyNeHsqeCHBQKTDk2+23kygy1r8/J2MU/ti5uyIj5vuc1h/90l c23cTyN1/5OLHoW2hc5++3y/EktxRqKhFnNRcSIATFW9xnYCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrNLMWRmVeSWpSXmKPExsXC5WfdrKvPcCfVYP09Y4s569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfHeA0wW 8+99ZrPYvGkqs8XxKVMZLX7/ACo+OWsyi4Ogx/fWPhaPnbPusnss2FTqsXmFlsfiPS+ZPDat 6mTz2PRpErvHu3Pn2D1OzPjN4jHvZKDH+31X2TwWv/jA5LH1l51H49RrbB6fN8kF8Edx2aSk 5mSWpRbp2yVwZRy9c4+tYA5nxflfU5kaGC+xdzFyckgImEi8+LcDzGYTUJe4ceMnM4gtImAm cbD1D1icWeAuk8SBfrYuRg4OYYEciX1f00DCLAKqErt/PgUr4RUwlVj78jQLxEh5idUbDjCD lHMCjXn/3xAkLARU8rf3KuMERq4FjAyrGEUy88pyEzNzTPWKszMq8zIr9JLzczcxAoN4We2f iTsYv1x2P8QowMGoxMO74MPtVCHWxLLiytxDjBIczEoivOEyN1OFeFMSK6tSi/Lji0pzUosP MUpzsCiJ83qFpyYICaQnlqRmp6YWpBbBZJk4OKUaGL209/65XXXWKmDGn+p/C/+LPHXn3rCp uvOp7YIZoYk8inLKSj33ZebM2cfpl5Hhf+inK1/xR+v5ZzjOXA/q3Gd443yZlcOa84uuLThy LO6y++o6FgVZgcKfLGJn7dccLRXedev7mYx3Z2bZTU+WL1zryTJlRirjrd9rBWaGRa5+lutY wbx7ursSS3FGoqEWc1FxIgA7DuZGXgIAAA== X-CFilter-Loop: Reflected Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791929207544935225 X-GMAIL-MSGID: 1791929207544935225 This is a preparation for migrc mechanism that requires to avoid redundant TLB flushes by manipulating tlb batch's arch data after arch_tlbbatch_flush(). However, we cannot because the data is getting cleared inside arch_tlbbatch_flush(). So separated the part clearing the tlb batch's arch data out of arch_tlbbatch_flush(). Signed-off-by: Byungchul Park --- arch/x86/mm/tlb.c | 2 -- mm/rmap.c | 1 + 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 453ea95b667d..941f41df02f3 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1268,8 +1268,6 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) local_irq_enable(); } - cpumask_clear(&batch->cpumask); - put_flush_tlb_info(); put_cpu(); } diff --git a/mm/rmap.c b/mm/rmap.c index da36f23ff7b0..b484d659d0c1 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -643,6 +643,7 @@ void try_to_unmap_flush(void) return; arch_tlbbatch_flush(&tlb_ubc->arch); + arch_tlbbatch_clear(&tlb_ubc->arch); tlb_ubc->flush_required = false; tlb_ubc->writable = false; } From patchwork Mon Feb 26 03:06:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 206188 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp1842387dyb; Sun, 25 Feb 2024 19:08:12 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCU1RzXBf3VsVeJaU2GYdc4xFhAkPf1b6q8n1ALjJpYdJDUE+OgVCBS49bnI0U8nKNmQ+uYcWzxIQoxIgeCB1MofQTD4Lw== X-Google-Smtp-Source: AGHT+IHnAgvzTYfSnXTzM3JN3MNnS97upAWY2Pi9k0Ci5vxF4botpYA8zjFaRHUOQP8ZvZG+eYQ2 X-Received: by 2002:a05:620a:47b7:b0:787:c908:791e with SMTP id dt55-20020a05620a47b700b00787c908791emr4346413qkb.33.1708916891964; Sun, 25 Feb 2024 19:08:11 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708916891; cv=pass; d=google.com; s=arc-20160816; b=g4CwYuBRmr+Lr78s/L/rFdyeZYgfWhb6f7+eypLQmNve1VLr6aJfqxkmh0dpZMwREP woWMe0ZedKy3xUpa3S7Kv8DH6xvr2YY8YHrv4Z03U7D9uniOhNyOULU4Kqm0zmYP/3YH qZL8iB2OkxNp9ton5G36HMPVCSuwjILy0cM5SC6Icw8V2Xoy3FqUpGp/6KLBA1k6OhfL LTum0emdVpNkosw1uw0sdVLiiDOa2tcaEbJ8fgZB2HpdxbdhhFJsvzjux1QdrqIfCAzb b5B7N+UXTnYz24Uq1lMdvAm9ViNeVHvOPxIzNCqjPltgwo++HnEDPmhJpBjQITrY42Uu Wzog== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-id:precedence:references :in-reply-to:message-id:date:subject:cc:to:from; bh=sYa6ZVoMA5/dASQ+t66FETLgCEz/UfospKSHXnWzbLI=; fh=ZH+M10KtJZRqSjFjp1+uAYTvsWtGX6qxh/YkifVvH1o=; b=fLAO3ct4t7KlqajJHUdCxCHLgWfc522OX4B1TDM4ePpuZhY8i9qJbyKVyiLwwpuf3t a+gvt5ytQdPn9AHPIHKWin7w74GzJ2SaO55yvylbaiePXJ7MX27yrQr1S5tzVcaq7J3I QrinAUBMrB/I3kZy5N5fCXEVxADnS6aRyBz+A5osJDgqPxGu3T0MnQ2Ar29oyAXSohhq aDshS3ecMLMWRmkVhYUHMVjpeUe62++7egUV/+A/bzDUetPSROJyGMRBLOrDXSaVn3wQ N1+ZWUb8VNNNmQn62HH4svtHK2sMStIhQtCbrTfNmYpLSt60NLw47vqGxUv+HP7GBHvI ffrg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=sk.com); spf=pass (google.com: domain of linux-kernel+bounces-80449-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-80449-ouuuleilei=gmail.com@vger.kernel.org" Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id c23-20020ae9e217000000b00787c1cf0d60si4195294qkc.571.2024.02.25.19.08.11 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 25 Feb 2024 19:08:11 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-80449-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=sk.com); spf=pass (google.com: domain of linux-kernel+bounces-80449-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-80449-ouuuleilei=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id B13571C2152C for ; Mon, 26 Feb 2024 03:08:11 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 63BCA1C2B2; Mon, 26 Feb 2024 03:06:42 +0000 (UTC) Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 93F651B7E2 for ; Mon, 26 Feb 2024 03:06:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=166.125.252.92 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708916800; cv=none; b=bYmbnh9BN0t/A16WIEXoRyr6SgLcsMN+tDA+RcMjhGlNr/gBjd8oVQnHVlseaLa85OEPppOyfIWf+6oWBTStoWNTLUsXWwUh/jVQsILN0lvb7xR7tjoSOj5pI3yb1XWSgl7KfQ37IKgo/O2dAW0r4hCiMhQ8TFsG7qwgQVUf6hA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708916800; c=relaxed/simple; bh=o06PFcoiHrss96x6Trd7RHh2EbWbMqndDf78qMPLNHI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=Lt5nViZSAZoMoAJPuU5yOGFyFBhlPulyZgSy5fv9KXYcclysY4EH9HB8XRGNSDkmCiZ2lwB0DSLlmw6eoip3NHEsUICK+sBSaqBn7SfInSmHTNePiXvI+jyV16z6Ka/kX2RL811JN3xSQU5LmqLOoOq56Tr7vA70LEAgg0c99l8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com; spf=pass smtp.mailfrom=sk.com; arc=none smtp.client-ip=166.125.252.92 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sk.com X-AuditID: a67dfc5b-d6dff70000001748-41-65dc002f0738 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [RESEND PATCH v8 6/8] mm: Add APIs to free a folio directly to the buddy bypassing pcp Date: Mon, 26 Feb 2024 12:06:11 +0900 Message-Id: <20240226030613.22366-7-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240226030613.22366-1-byungchul@sk.com> References: <20240226030613.22366-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrCLMWRmVeSWpSXmKPExsXC9ZZnoa4Bw51Ug/Pn2CzmrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeJ47wEmi/n3PrNZ bN40ldni+JSpjBa/fwAVn5w1mcVBwON7ax+Lx85Zd9k9Fmwq9di8Qstj8Z6XTB6bVnWyeWz6 NInd4925c+weJ2b8ZvGYdzLQ4/2+q2weW3/ZeTROvcbm8XmTXABfFJdNSmpOZllqkb5dAlfG qfuL2AuuyVb8f/uPtYFxskQXIweHhICJxNZHGV2MnGDm0zntLCA2m4C6xI0bP5lBbBEBM4mD rX/YQWxmgbtMEgf62UBsYYF4if2X9oDVsAioSux98QOsl1fAVGLG58lsEDPlJVZvOMAMsooT aM77/4YgYSGgkr+9Vxm7GLmASj6zSWzZepgdol5S4uCKGywTGHkXMDKsYhTKzCvLTczMMdHL qMzLrNBLzs/dxAgM+2W1f6J3MH66EHyIUYCDUYmHd8GH26lCrIllxZW5hxglOJiVRHjDZW6m CvGmJFZWpRblxxeV5qQWH2KU5mBREuc1+laeIiSQnliSmp2aWpBaBJNl4uCUamBs/fwpueHU +a9tKw+2sUW6Xbrza+XE5pfOx8KLlTklC3pe7Dl5W/xH0iLN+GtZapmOzp+F9zow/WKaJquq t/Td3RgtJZXY6nUBOjITfu//bnl8389p/695R/1hzOyZG64y9XTs34X8TNqa66csEjSdH9Zm u73wwNTLs17mLVOZ/P3rydCGyGUflViKMxINtZiLihMBNdP+sncCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrDLMWRmVeSWpSXmKPExsXC5WfdrKvPcCfVYOcJS4s569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfHeA0wW 8+99ZrPYvGkqs8XxKVMZLX7/ACo+OWsyi4Ogx/fWPhaPnbPusnss2FTqsXmFlsfiPS+ZPDat 6mTz2PRpErvHu3Pn2D1OzPjN4jHvZKDH+31X2TwWv/jA5LH1l51H49RrbB6fN8kF8Edx2aSk 5mSWpRbp2yVwZZy6v4i94Jpsxf+3/1gbGCdLdDFyckgImEg8ndPOAmKzCahL3LjxkxnEFhEw kzjY+ocdxGYWuMskcaCfDcQWFoiX2H9pD1gNi4CqxN4XP8B6eQVMJWZ8nswGMVNeYvWGA0A1 HBycQHPe/zcECQsBlfztvco4gZFrASPDKkaRzLyy3MTMHFO94uyMyrzMCr3k/NxNjMAwXlb7 Z+IOxi+X3Q8xCnAwKvHwLvhwO1WINbGsuDL3EKMEB7OSCG+4zM1UId6UxMqq1KL8+KLSnNTi Q4zSHCxK4rxe4akJQgLpiSWp2ampBalFMFkmDk6pBkadtKp9a39EpfhfkeG/yl7+RWGKU8Ar h7QetZ0O31+/at9+Y9qPHSdCr54z+NdVrbB9evJcp3jNm4/Z5V709a477P5QvGKyx1WtmQw9 zwzOBq15tV137Ycfy9IvnZa4HZEYYGsSVuHG98zS5ZL1gbK2ZSzHJ3fMO6gpEPyQ5yTnZY24 2xMfZzEosRRnJBpqMRcVJwIAuYNFwF8CAAA= X-CFilter-Loop: Reflected Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791929239026238203 X-GMAIL-MSGID: 1791929239026238203 This is a preparation for migrc mechanism that frees folios at a better time. The mechanism will defer the use of folio_put*() for source folios of migration, that are unlikely to be used and a group of folios will be freed at once at a later time. However, this will pollute pcp so as to inexpectedly free_pcppages_bulk() fresher folios and make pcp get unstable. To facilitate this new mechanism, an additional API has been added that allows folios under migrc's control to be freed directly to buddy bypassing pcp. Signed-off-by: Byungchul Park --- include/linux/mm.h | 23 +++++++++++++++++++++++ mm/internal.h | 1 + mm/page_alloc.c | 10 ++++++++++ mm/swap.c | 7 +++++++ 4 files changed, 41 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index da5219b48d52..fc0581cce3a7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1284,6 +1284,7 @@ static inline struct folio *virt_to_folio(const void *x) } void __folio_put(struct folio *folio); +void __folio_put_small_nopcp(struct folio *folio); void put_pages_list(struct list_head *pages); @@ -1483,6 +1484,28 @@ static inline void folio_put(struct folio *folio) __folio_put(folio); } +/** + * folio_put_small_nopcp - Decrement the reference count on a folio. + * @folio: The folio. + * + * This is only for a single page folio to release directly to the buddy + * allocator bypassing pcp. + * + * If the folio's reference count reaches zero, the memory will be + * released back to the page allocator and may be used by another + * allocation immediately. Do not access the memory or the struct folio + * after calling folio_put_small_nopcp() unless you can be sure that it + * wasn't the last reference. + * + * Context: May be called in process or interrupt context, but not in NMI + * context. May be called while holding a spinlock. + */ +static inline void folio_put_small_nopcp(struct folio *folio) +{ + if (folio_put_testzero(folio)) + __folio_put_small_nopcp(folio); +} + /** * folio_put_refs - Reduce the reference count on a folio. * @folio: The folio. diff --git a/mm/internal.h b/mm/internal.h index b880f1e78700..3be8fd5604e8 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -451,6 +451,7 @@ extern int user_min_free_kbytes; extern void free_unref_page(struct page *page, unsigned int order); extern void free_unref_page_list(struct list_head *list); +extern void free_pages_nopcp(struct page *page, unsigned int order); extern void zone_pcp_reset(struct zone *zone); extern void zone_pcp_disable(struct zone *zone); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 733732e7e0ba..21b8c8cd1673 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -565,6 +565,16 @@ static inline void free_the_page(struct page *page, unsigned int order) __free_pages_ok(page, order, FPI_NONE); } +void free_pages_nopcp(struct page *page, unsigned int order) +{ + /* + * This function will be used in case that the pages are too + * cold to keep in pcp e.g. migrc mechanism. So it'd better + * release the pages to the tail. + */ + __free_pages_ok(page, order, FPI_TO_TAIL); +} + /* * Higher-order pages are called "compound pages". They are structured thusly: * diff --git a/mm/swap.c b/mm/swap.c index cd8f0150ba3a..3f37496a1184 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -106,6 +106,13 @@ static void __folio_put_small(struct folio *folio) free_unref_page(&folio->page, 0); } +void __folio_put_small_nopcp(struct folio *folio) +{ + __page_cache_release(folio); + mem_cgroup_uncharge(folio); + free_pages_nopcp(&folio->page, 0); +} + static void __folio_put_large(struct folio *folio) { /* From patchwork Mon Feb 26 03:06:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 206190 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp1842487dyb; Sun, 25 Feb 2024 19:08:34 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCXsz7foIAO6liwZvyOtiRg54f/QOPEYy8i07ZgQhfAX8LQQlgZi78Am8rBcKCk0lQuHoegOrTVCa9qfCWE7v997OQu5uw== X-Google-Smtp-Source: AGHT+IFTYU/obeQu2tysQzSacLmf8jqBvd8Qd2kgACx2rA/R9zHW3qvEP+K96oEes0xyeXdpFRND X-Received: by 2002:a05:622a:1354:b0:42e:8b8b:81ca with SMTP id w20-20020a05622a135400b0042e8b8b81camr195626qtk.8.1708916913951; Sun, 25 Feb 2024 19:08:33 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708916913; cv=pass; d=google.com; s=arc-20160816; b=uU7XAn8rKm2w2JS1jfXCxpRdSf8pDklV8dsPWaGaCmFqHjM0SAcpbtrH+43/xrjUPj cN+wBGMPVp6qrd83QXkMTevuat4+DlbyOShLlx8+Z5gRAYLDJnZerZHpuKSyaHW6VIuW mBW3ebku0DXqOoi/w8surVUsz24QOC/7TlUG/tPjJOhqw0VuoNYpZNlYIshFeFR1VIx3 1dszE5x24eZ/hVhVmA22UCwUNIy6Zi5TT2k3IR1/TXyMOZN3O+YSQsXQzm13XwqWlS5F kSqDbMjT9fnXUDyz/nAv75zA8K7cIXNaEQgPmbfLfaJMo2Zwh2uvOU7ZATNtP4MWOC+3 3gvQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-id:precedence:references :in-reply-to:message-id:date:subject:cc:to:from; bh=fZXGApSJ6yOmQMw6O9Qn2JVVKDcgwybFt35sixwjZS8=; fh=ZH+M10KtJZRqSjFjp1+uAYTvsWtGX6qxh/YkifVvH1o=; b=0uuxeTr8kAxYmzF5zBLuWf2Fvd9R2rnsTQrN0NICUxW0lBO0eTOvU2ZdeEOcoQWFP8 j8bLTbmhpw0w5jKklsAeLBxl6rborc4g6aCGSxnGoCg/5mR9UNVTEuPPAlrdzIbyW5Gw UQH7lHP6ovpfTjyy/gNsyu7k/WDMf5ZTO4iGCPgvBaM2+Cs6SDjBUux3nNbnCRd3OHZc AG8sLtTIBNu8/feAwb3Ae65NdXR2K2LPmb8nCWgL+3JRgB1SPIIpO6sF2D5ydZYwR54T uLtHywy3g3vyICUrfq+FAQw7JyjIV50W9tIsnbwH5tDCgE3dX2JcaPlZKYduJXNv+hQi XMMg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=sk.com); spf=pass (google.com: domain of linux-kernel+bounces-80450-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-80450-ouuuleilei=gmail.com@vger.kernel.org" Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id s21-20020ae9f715000000b00787d74b69eesi266699qkg.373.2024.02.25.19.08.33 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 25 Feb 2024 19:08:33 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-80450-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=sk.com); spf=pass (google.com: domain of linux-kernel+bounces-80450-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-80450-ouuuleilei=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id AC8081C21525 for ; Mon, 26 Feb 2024 03:08:33 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 99D5F1CA85; Mon, 26 Feb 2024 03:06:46 +0000 (UTC) Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 060081B94B for ; Mon, 26 Feb 2024 03:06:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=166.125.252.92 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708916801; cv=none; b=Mp0QeXgW35P8D8l5nwR3UWNvI3SHhWcoJI8+NMBGlDctWvntTCnW4JGWadD/N9qEkprYsgYagSd41YDlgmYJhlm9EV3j/3+JAd6l80GAoHDOJp35hijnhcSb/DCUb01o+zqgFTHY4LqwAYrHfwvFk4JYH6YvvM+biZpJYh5CG8I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708916801; c=relaxed/simple; bh=XqhqLP5xp8tj5zmwqb0benp0qdZ7AK8inBBstOQV7PE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=MNqoWxSNEcD87KykLrM2qfknsThunlzldv9aaSIOGWAYrN2qjobLOaQA5BmCtuh/Txts99edVwO8fN2Rqtm+HixweVJ83DDWG5BTpM23ggSgf2aFi8VWeCycjtGpdu6waNV8HeHrTuRR/0CSV7UCAU/no0+SNRmRNUc9Yn/320c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com; spf=pass smtp.mailfrom=sk.com; arc=none smtp.client-ip=166.125.252.92 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sk.com X-AuditID: a67dfc5b-d6dff70000001748-46-65dc00307553 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [RESEND PATCH v8 7/8] mm: Defer TLB flush by keeping both src and dst folios at migration Date: Mon, 26 Feb 2024 12:06:12 +0900 Message-Id: <20240226030613.22366-8-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240226030613.22366-1-byungchul@sk.com> References: <20240226030613.22366-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrOLMWRmVeSWpSXmKPExsXC9ZZnoa4Bw51Ug9NP+CzmrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeJ47wEmi/n3PrNZ bN40ldni+JSpjBa/fwAVn5w1mcVBwON7ax+Lx85Zd9k9Fmwq9di8Qstj8Z6XTB6bVnWyeWz6 NInd4925c+weJ2b8ZvGYdzLQ4/2+q2weW3/ZeTROvcbm8XmTXABfFJdNSmpOZllqkb5dAlfG jEVnWAuerGSsuNpxjL2BcU0bYxcjJ4eEgInEjUWtrDD2z94vzCA2m4C6xI0bP8FsEQEziYOt f9hBbGaBu0wSB/rZQGxhgSSJwzt7mUBsFgFViUsnF4PV8wqYSjzZ/5ANYqa8xOoNB4DiHByc QHPe/zcECQsBlfztvQp0AhdQyWc2iZf73jND1EtKHFxxg2UCI+8CRoZVjEKZeWW5iZk5JnoZ lXmZFXrJ+bmbGIHBv6z2T/QOxk8Xgg8xCnAwKvHwLvhwO1WINbGsuDL3EKMEB7OSCG+4zM1U Id6UxMqq1KL8+KLSnNTiQ4zSHCxK4rxG38pThATSE0tSs1NTC1KLYLJMHJxSDYyuGZkJ0hNc 3c9aPc55qVo4dT6fw4wNB332Fm3pyn+0a3qV6FG2yoodTy3Pe/MvX+M4qT9+0xk/qTqmS64P 5ARUROuDWyvf7jjfqGZy8azqiojd+24cS7/8OtCMa9mXq3zHt97i8jVv3nBxmkjS6eC0pCfb qybsvWVlv3PJCqWwX5+SotbZxaQrsRRnJBpqMRcVJwIAihqzTnoCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrHLMWRmVeSWpSXmKPExsXC5WfdrGvAcCfVYN4hJos569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfHeA0wW 8+99ZrPYvGkqs8XxKVMZLX7/ACo+OWsyi4Ogx/fWPhaPnbPusnss2FTqsXmFlsfiPS+ZPDat 6mTz2PRpErvHu3Pn2D1OzPjN4jHvZKDH+31X2TwWv/jA5LH1l51H49RrbB6fN8kF8Edx2aSk 5mSWpRbp2yVwZcxYdIa14MlKxoqrHcfYGxjXtDF2MXJySAiYSPzs/cIMYrMJqEvcuPETzBYR MJM42PqHHcRmFrjLJHGgnw3EFhZIkji8s5cJxGYRUJW4dHIxWD2vgKnEk/0P2SBmykus3nAA KM7BwQk05/1/Q5CwEFDJ396rjBMYuRYwMqxiFMnMK8tNzMwx1SvOzqjMy6zQS87P3cQIDOVl tX8m7mD8ctn9EKMAB6MSD++CD7dThVgTy4orcw8xSnAwK4nwhsvcTBXiTUmsrEotyo8vKs1J LT7EKM3BoiTO6xWemiAkkJ5YkpqdmlqQWgSTZeLglGpgVGlNtTocvEB9xl8pXhWjjDvbrTJk GvmFZtYK+j9Wzef5GHD7lNzute23ulJZTWcpFvwqbJvWv1NgTb+OYXnXu4/m7zQqXV+s6Jtg dKPf39g123fmt/xvtxqMeT6fNJmZcWfeh9MbTHjMt8wov3vGt+7chpkSH5rvfWBKjhd7z542 Nc3bS3O+EktxRqKhFnNRcSIAWMK1WWECAAA= X-CFilter-Loop: Reflected Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791929262069309347 X-GMAIL-MSGID: 1791929262069309347 Implementation of MIGRC mechanism that stands for 'Migration Read Copy'. We always face the migration overhead at either promotion or demotion, while working with tiered memory e.g. CXL memory and found out TLB shootdown is a quite big one that is needed to get rid of if possible. Fortunately, TLB flush can be defered if both source and destination of folios during migration are kept until all TLB flushes required will have been done, of course, only if the target PTE entries have read-only permission, more precisely speaking, don't have write permission. Otherwise, no doubt the folio might get messed up. To achieve that: 1. For the folios that map only to non-writable TLB entries, prevent TLB flush at migration by keeping both source and destination folios, which will be handled later at a better time. 2. When any non-writable TLB entry changes to writable e.g. through fault handler, give up migrc mechanism so as to perform TLB flush required right away. The following evaluation using XSBench shows the improvement like: 1. itlb flush was reduced by 93.9%. 2. dtlb thread was reduced by 43.5%. 3. stlb flush was reduced by 24.9%. 4. dtlb store misses was reduced by 34.2%. 5. itlb load misses was reduced by 45.5%. 6. The runtime was reduced by 3.5%. The measurement result: Architecture - x86_64 QEMU - kvm enabled, host cpu Numa - 2 nodes (16 CPUs 1GB, no CPUs 8GB) Linux Kernel - v6.7, numa balancing tiering on, demotion enabled Benchmark - XSBench -p 100000000 (-p option makes the runtime longer) run 'perf stat' using events: 1) itlb.itlb_flush 2) tlb_flush.dtlb_thread 3) tlb_flush.stlb_any 4) dTLB-load-misses 5) dTLB-store-misses 6) iTLB-load-misses run 'cat /proc/vmstat' and pick: 1) numa_pages_migrated 2) pgmigrate_success 3) nr_tlb_remote_flush 4) nr_tlb_remote_flush_received 5) nr_tlb_local_flush_all 6) nr_tlb_local_flush_one BEFORE - mainline v6.7 ---------------------- $ perf stat -a \ -e itlb.itlb_flush \ -e tlb_flush.dtlb_thread \ -e tlb_flush.stlb_any \ -e dTLB-load-misses \ -e dTLB-store-misses \ -e iTLB-load-misses \ ./XSBench -p 100000000 Performance counter stats for 'system wide': 85647229 itlb.itlb_flush 480981504 tlb_flush.dtlb_thread 323937200 tlb_flush.stlb_any 238381632579 dTLB-load-misses 601514255 dTLB-store-misses 2974157461 iTLB-load-misses 2252.883892112 seconds time elapsed $ cat /proc/vmstat ... numa_pages_migrated 12790664 pgmigrate_success 26835314 nr_tlb_remote_flush 3031412 nr_tlb_remote_flush_received 45234862 nr_tlb_local_flush_all 216584 nr_tlb_local_flush_one 740940 ... AFTER - mainline v6.7 + migrc ----------------------------- $ perf stat -a \ -e itlb.itlb_flush \ -e tlb_flush.dtlb_thread \ -e tlb_flush.stlb_any \ -e dTLB-load-misses \ -e dTLB-store-misses \ -e iTLB-load-misses \ ./XSBench -p 100000000 Performance counter stats for 'system wide': 5240261 itlb.itlb_flush 271581774 tlb_flush.dtlb_thread 243149389 tlb_flush.stlb_any 234502983364 dTLB-load-misses 395673680 dTLB-store-misses 1620215163 iTLB-load-misses 2172.283436287 seconds time elapsed $ cat /proc/vmstat ... numa_pages_migrated 14897064 pgmigrate_success 30825530 nr_tlb_remote_flush 198290 nr_tlb_remote_flush_received 2820156 nr_tlb_local_flush_all 92048 nr_tlb_local_flush_one 741401 ... Signed-off-by: Byungchul Park --- include/linux/mmzone.h | 7 ++ include/linux/sched.h | 8 ++ mm/internal.h | 53 ++++++++ mm/memory.c | 8 ++ mm/migrate.c | 271 +++++++++++++++++++++++++++++++++++++++-- mm/page_alloc.c | 11 +- mm/rmap.c | 12 +- 7 files changed, 358 insertions(+), 12 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 9db36e197712..492111cd1176 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -1002,6 +1002,13 @@ struct zone { /* Zone statistics */ atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS]; atomic_long_t vm_numa_event[NR_VM_NUMA_EVENT_ITEMS]; + +#if defined(CONFIG_MIGRATION) && defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) + /* + * the number of folios pending for TLB flush in the zone + */ + atomic_t migrc_pending_nr; +#endif } ____cacheline_internodealigned_in_smp; enum pgdat_flags { diff --git a/include/linux/sched.h b/include/linux/sched.h index 0317e7a65151..d8c285309a8f 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1330,6 +1330,14 @@ struct task_struct { struct tlbflush_unmap_batch tlb_ubc; struct tlbflush_unmap_batch tlb_ubc_ro; +#if defined(CONFIG_MIGRATION) && defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) + /* + * whether all the mappings of a folio during unmap are read-only + * so that migrc can work on the folio + */ + bool can_migrc; +#endif + /* Cache last used pipe for splice(): */ struct pipe_inode_info *splice_pipe; diff --git a/mm/internal.h b/mm/internal.h index 3be8fd5604e8..ab02cb8306e2 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1284,4 +1284,57 @@ static inline void shrinker_debugfs_remove(struct dentry *debugfs_entry, } #endif /* CONFIG_SHRINKER_DEBUG */ +#if defined(CONFIG_MIGRATION) && defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) +/* + * Reset the indicator indicating there are no writable mappings at the + * beginning of every rmap traverse for unmap. Migrc can work only when + * all the mappings are read-only. + */ +static inline void can_migrc_init(void) +{ + current->can_migrc = true; +} + +/* + * Mark the folio is not applicable to migrc, once it found a writble or + * dirty pte during rmap traverse for unmap. + */ +static inline void can_migrc_fail(void) +{ + current->can_migrc = false; +} + +/* + * Check if all the mappings are read-only and read-only mappings even + * exist. + */ +static inline bool can_migrc_test(void) +{ + return current->can_migrc && current->tlb_ubc_ro.flush_required; +} + +/* + * Return the number of folios pending TLB flush that have yet to get + * freed in the zone. + */ +static inline int migrc_pending_nr_in_zone(struct zone *z) +{ + return atomic_read(&z->migrc_pending_nr); +} + +/* + * Perform TLB flush needed and free the folios under migrc's control. + */ +bool migrc_flush_free_folios(void); +void migrc_flush_start(void); +void migrc_flush_end(struct tlbflush_unmap_batch *batch); +#else /* CONFIG_MIGRATION && CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ +static inline void can_migrc_init(void) {} +static inline void can_migrc_fail(void) {} +static inline bool can_migrc_test(void) { return false; } +static inline int migrc_pending_nr_in_zone(struct zone *z) { return 0; } +static inline bool migrc_flush_free_folios(void) { return false; } +static inline void migrc_flush_start(void) {} +static inline void migrc_flush_end(struct tlbflush_unmap_batch *batch) {} +#endif #endif /* __MM_INTERNAL_H */ diff --git a/mm/memory.c b/mm/memory.c index 6e0712d06cd4..e67de161da8b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3462,6 +3462,14 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) if (vmf->page) folio = page_folio(vmf->page); + /* + * The folio may or may not be one that is under migrc's control + * and about to change its permission from read-only to writable. + * Conservatively give up deferring TLB flush just in case. + */ + if (folio) + migrc_flush_free_folios(); + /* * Shared mapping: we are guaranteed to have VM_WRITE and * FAULT_FLAG_WRITE set at this point. diff --git a/mm/migrate.c b/mm/migrate.c index bbe1ecef4956..cbe5372f159e 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -57,6 +57,194 @@ #include "internal.h" +#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH +static struct tlbflush_unmap_batch migrc_ubc; +static LIST_HEAD(migrc_folios); +static DEFINE_SPINLOCK(migrc_lock); + +static void init_tlb_ubc(struct tlbflush_unmap_batch *ubc) +{ + arch_tlbbatch_clear(&ubc->arch); + ubc->flush_required = false; + ubc->writable = false; +} + +static void migrc_keep_folio(struct folio *f, struct list_head *h) +{ + list_move_tail(&f->lru, h); + folio_get(f); + atomic_inc(&folio_zone(f)->migrc_pending_nr); +} + +static void migrc_undo_folio(struct folio *f) +{ + list_del(&f->lru); + folio_put(f); + atomic_dec(&folio_zone(f)->migrc_pending_nr); +} + +static void migrc_release_folio(struct folio *f) +{ + folio_put_small_nopcp(f); + atomic_dec(&folio_zone(f)->migrc_pending_nr); +} + +/* + * Need to synchronize between TLB flush and managing pending CPUs in + * migrc_ubc. Take a look at the following scenario: + * + * CPU0 CPU1 + * ---- ---- + * TLB flush + * Unmap folios (needing TLB flush) + * Add pending CPUs to migrc_ubc + * Clear the CPUs from migrc_ubc + * + * The pending CPUs added in CPU1 should not be cleared from migrc_ubc + * in CPU0 because the TLB flush for migrc_ubc added in CPU1 has not + * been performed this turn. To avoid this, using 'migrc_flushing' + * variable, prevent adding pending CPUs to migrc_ubc and give up migrc + * mechanism if others are in the middle of TLB flush, like: + * + * CPU0 CPU1 + * ---- ---- + * migrc_flushing++ + * TLB flush + * Unmap folios (needing TLB flush) + * If migrc_flushing == 0: + * Add pending CPUs to migrc_ubc + * Else: <--- hit + * Give up migrc mechanism + * Clear the CPUs from migrc_ubc + * migrc_flush-- + * + * Only the following case would be allowed for migrc mechanism to work: + * + * CPU0 CPU1 + * ---- ---- + * Unmap folios (needing TLB flush) + * If migrc_flushing == 0: <--- hit + * Add pending CPUs to migrc_ubc + * Else: + * Give up migrc mechanism + * migrc_flushing++ + * TLB flush + * Clear the CPUs from migrc_ubc + * migrc_flush-- + */ +static int migrc_flushing; + +static bool migrc_add_pending_ubc(struct tlbflush_unmap_batch *ubc) +{ + struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + unsigned long flags; + + spin_lock_irqsave(&migrc_lock, flags); + if (migrc_flushing) { + spin_unlock_irqrestore(&migrc_lock, flags); + + /* + * Give up migrc mechanism. Just let TLB flush needed + * handled by try_to_unmap_flush() at the caller side. + */ + fold_ubc(tlb_ubc, ubc); + return false; + } + fold_ubc(&migrc_ubc, ubc); + spin_unlock_irqrestore(&migrc_lock, flags); + return true; +} + +static bool migrc_add_pending_folios(struct list_head *folios) +{ + unsigned long flags; + + spin_lock_irqsave(&migrc_lock, flags); + if (migrc_flushing) { + spin_unlock_irqrestore(&migrc_lock, flags); + + /* + * Give up migrc mechanism. The caller should perform + * TLB flush needed using migrc_flush_free_folios() and + * undo some on the folios e.g. restore folios' + * reference count increased by migrc and more. + */ + return false; + } + list_splice(folios, &migrc_folios); + spin_unlock_irqrestore(&migrc_lock, flags); + return true; +} + +void migrc_flush_start(void) +{ + unsigned long flags; + + spin_lock_irqsave(&migrc_lock, flags); + migrc_flushing++; + spin_unlock_irqrestore(&migrc_lock, flags); +} + +void migrc_flush_end(struct tlbflush_unmap_batch *batch) +{ + LIST_HEAD(folios); + struct folio *f, *f2; + unsigned long flags; + + spin_lock_irqsave(&migrc_lock, flags); + if (!arch_tlbbatch_done(&migrc_ubc.arch, &batch->arch)) { + list_splice_init(&migrc_folios, &folios); + migrc_ubc.flush_required = false; + migrc_ubc.writable = false; + } + migrc_flushing--; + spin_unlock_irqrestore(&migrc_lock, flags); + + list_for_each_entry_safe(f, f2, &folios, lru) + migrc_release_folio(f); +} + +bool migrc_flush_free_folios(void) +{ + struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + LIST_HEAD(folios); + struct folio *f, *f2; + unsigned long flags; + bool ret = true; + + spin_lock_irqsave(&migrc_lock, flags); + list_splice_init(&migrc_folios, &folios); + fold_ubc(tlb_ubc, &migrc_ubc); + spin_unlock_irqrestore(&migrc_lock, flags); + + if (list_empty(&folios)) + ret = false; + + try_to_unmap_flush(); + list_for_each_entry_safe(f, f2, &folios, lru) + migrc_release_folio(f); + return ret; +} +#else /* CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ +static void init_tlb_ubc(struct tlbflush_unmap_batch *ubc) +{ +} +static void migrc_keep_folio(struct folio *f, struct list_head *h) +{ +} +static void migrc_undo_folio(struct folio *f) +{ +} +static bool migrc_add_pending_ubc(struct tlbflush_unmap_batch *ubc) +{ + return false; +} +static bool migrc_add_pending_folios(struct list_head *folios) +{ + return false; +} +#endif + bool isolate_movable_page(struct page *page, isolate_mode_t mode) { struct folio *folio = folio_get_nontail_page(page); @@ -1274,7 +1462,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, struct folio *src, struct folio *dst, enum migrate_mode mode, enum migrate_reason reason, - struct list_head *ret) + struct list_head *ret, struct list_head *move_succ) { int rc; int old_page_state = 0; @@ -1321,9 +1509,13 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, /* * A folio that has been migrated has all references removed - * and will be freed. + * and will be freed, unless it's under migrc's control. */ - list_del(&src->lru); + if (move_succ) + migrc_keep_folio(src, move_succ); + else + list_del(&src->lru); + /* Drop an anon_vma reference if we took one */ if (anon_vma) put_anon_vma(anon_vma); @@ -1618,7 +1810,7 @@ static void migrate_folios_move(struct list_head *src_folios, struct list_head *ret_folios, struct migrate_pages_stats *stats, int *retry, int *thp_retry, int *nr_failed, - int *nr_retry_pages) + int *nr_retry_pages, struct list_head *move_succ) { struct folio *folio, *folio2, *dst, *dst2; bool is_thp; @@ -1635,7 +1827,7 @@ static void migrate_folios_move(struct list_head *src_folios, rc = migrate_folio_move(put_new_folio, private, folio, dst, mode, - reason, ret_folios); + reason, ret_folios, move_succ); /* * The rules are: * Success: folio will be freed @@ -1712,17 +1904,34 @@ static int migrate_pages_batch(struct list_head *from, int rc, rc_saved = 0, nr_pages; LIST_HEAD(unmap_folios); LIST_HEAD(dst_folios); + LIST_HEAD(unmap_folios_migrc); + LIST_HEAD(dst_folios_migrc); + LIST_HEAD(move_succ); bool nosplit = (reason == MR_NUMA_MISPLACED); + struct tlbflush_unmap_batch pending_ubc; + struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + struct tlbflush_unmap_batch *tlb_ubc_ro = ¤t->tlb_ubc_ro; + bool do_migrc; + bool migrc_ubc_succ; VM_WARN_ON_ONCE(mode != MIGRATE_ASYNC && !list_empty(from) && !list_is_singular(from)); + /* + * Apply migrc only to numa migration for now. + */ + init_tlb_ubc(&pending_ubc); + do_migrc = IS_ENABLED(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) && + (reason == MR_DEMOTION || reason == MR_NUMA_MISPLACED); + for (pass = 0; pass < nr_pass && retry; pass++) { retry = 0; thp_retry = 0; nr_retry_pages = 0; list_for_each_entry_safe(folio, folio2, from, lru) { + bool can_migrc; + is_large = folio_test_large(folio); is_thp = is_large && folio_test_pmd_mappable(folio); nr_pages = folio_nr_pages(folio); @@ -1752,9 +1961,12 @@ static int migrate_pages_batch(struct list_head *from, continue; } + can_migrc_init(); rc = migrate_folio_unmap(get_new_folio, put_new_folio, private, folio, &dst, mode, reason, ret_folios); + can_migrc = do_migrc && can_migrc_test() && !is_large; + /* * The rules are: * Success: folio will be freed @@ -1800,7 +2012,8 @@ static int migrate_pages_batch(struct list_head *from, /* nr_failed isn't updated for not used */ stats->nr_thp_failed += thp_retry; rc_saved = rc; - if (list_empty(&unmap_folios)) + if (list_empty(&unmap_folios) && + list_empty(&unmap_folios_migrc)) goto out; else goto move; @@ -1814,8 +2027,19 @@ static int migrate_pages_batch(struct list_head *from, stats->nr_thp_succeeded += is_thp; break; case MIGRATEPAGE_UNMAP: - list_move_tail(&folio->lru, &unmap_folios); - list_add_tail(&dst->lru, &dst_folios); + if (can_migrc) { + list_move_tail(&folio->lru, &unmap_folios_migrc); + list_add_tail(&dst->lru, &dst_folios_migrc); + + /* + * Gather ro batch data to add + * to migrc_ubc after unmap. + */ + fold_ubc(&pending_ubc, tlb_ubc_ro); + } else { + list_move_tail(&folio->lru, &unmap_folios); + list_add_tail(&dst->lru, &dst_folios); + } break; default: /* @@ -1829,12 +2053,19 @@ static int migrate_pages_batch(struct list_head *from, stats->nr_failed_pages += nr_pages; break; } + /* + * Done with the current folio. Fold the ro + * batch data gathered, to the normal batch. + */ + fold_ubc(tlb_ubc, tlb_ubc_ro); } } nr_failed += retry; stats->nr_thp_failed += thp_retry; stats->nr_failed_pages += nr_retry_pages; move: + /* Should be before try_to_unmap_flush() */ + migrc_ubc_succ = do_migrc && migrc_add_pending_ubc(&pending_ubc); /* Flush TLBs for all unmapped folios */ try_to_unmap_flush(); @@ -1848,7 +2079,27 @@ static int migrate_pages_batch(struct list_head *from, migrate_folios_move(&unmap_folios, &dst_folios, put_new_folio, private, mode, reason, ret_folios, stats, &retry, &thp_retry, - &nr_failed, &nr_retry_pages); + &nr_failed, &nr_retry_pages, NULL); + migrate_folios_move(&unmap_folios_migrc, &dst_folios_migrc, + put_new_folio, private, mode, reason, + ret_folios, stats, &retry, &thp_retry, + &nr_failed, &nr_retry_pages, migrc_ubc_succ ? + &move_succ : NULL); + } + + /* + * In case that migrc_add_pending_ubc() has been added + * successfully but migrc_add_pending_folios() does not. + */ + if (migrc_ubc_succ && !migrc_add_pending_folios(&move_succ)) { + migrc_flush_free_folios(); + + /* + * Undo src folios that have been successfully added to + * move_succ. + */ + list_for_each_entry_safe(folio, folio2, &move_succ, lru) + migrc_undo_folio(folio); } nr_failed += retry; stats->nr_thp_failed += thp_retry; @@ -1859,6 +2110,8 @@ static int migrate_pages_batch(struct list_head *from, /* Cleanup remaining folios */ migrate_folios_undo(&unmap_folios, &dst_folios, put_new_folio, private, ret_folios); + migrate_folios_undo(&unmap_folios_migrc, &dst_folios_migrc, + put_new_folio, private, ret_folios); return rc; } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 21b8c8cd1673..6ef0c22b1109 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2972,6 +2972,8 @@ bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark, long min = mark; int o; + free_pages += migrc_pending_nr_in_zone(z); + /* free_pages may go negative - that's OK */ free_pages -= __zone_watermark_unusable_free(z, order, alloc_flags); @@ -3066,7 +3068,7 @@ static inline bool zone_watermark_fast(struct zone *z, unsigned int order, long usable_free; long reserved; - usable_free = free_pages; + usable_free = free_pages + migrc_pending_nr_in_zone(z); reserved = __zone_watermark_unusable_free(z, 0, alloc_flags); /* reserved may over estimate high-atomic reserves. */ @@ -3273,6 +3275,13 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags, gfp_mask)) { int ret; + if (migrc_pending_nr_in_zone(zone) && + migrc_flush_free_folios() && + zone_watermark_fast(zone, order, mark, + ac->highest_zoneidx, + alloc_flags, gfp_mask)) + goto try_this_zone; + if (has_unaccepted_memory()) { if (try_to_accept_memory(zone, order)) goto try_this_zone; diff --git a/mm/rmap.c b/mm/rmap.c index b484d659d0c1..39ab0d64665a 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -642,7 +642,9 @@ void try_to_unmap_flush(void) if (!tlb_ubc->flush_required) return; + migrc_flush_start(); arch_tlbbatch_flush(&tlb_ubc->arch); + migrc_flush_end(tlb_ubc); arch_tlbbatch_clear(&tlb_ubc->arch); tlb_ubc->flush_required = false; tlb_ubc->writable = false; @@ -677,9 +679,15 @@ static void set_tlb_ubc_flush_pending(struct mm_struct *mm, pte_t pteval, if (!pte_accessible(mm, pteval)) return; - if (pte_write(pteval) || writable) + if (pte_write(pteval) || writable) { tlb_ubc = ¤t->tlb_ubc; - else + + /* + * Migrc cannot work with the folio, once it found a + * writable or dirty mapping on it. + */ + can_migrc_fail(); + } else tlb_ubc = ¤t->tlb_ubc_ro; arch_tlbbatch_add_pending(&tlb_ubc->arch, mm, uaddr); From patchwork Mon Feb 26 03:06:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 206189 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp1842446dyb; Sun, 25 Feb 2024 19:08:26 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCUfbe7nxjl5P+3q7/CYPE2rjQ1u3h2W1QHdDtkOJaDT2m7TEhRyiDO5lVxPRkytuR4p6fpIMrzW+N8zW9LsGEu4FQdYmw== X-Google-Smtp-Source: AGHT+IFMRce5JCYOeGLIla3p31kWJsd7OW7DNfft5S+aQS2f9ml5+Hm5Ds9rH7UahhK2Z/E+mMC3 X-Received: by 2002:a17:906:2b53:b0:a43:215b:ff0e with SMTP id b19-20020a1709062b5300b00a43215bff0emr1586919ejg.56.1708916906245; Sun, 25 Feb 2024 19:08:26 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708916906; cv=pass; d=google.com; s=arc-20160816; b=p+OW3IUQGdqlPrNI6KidbSmZkdWXeGCeyPx5GmpcX+5tP3V50mnbKzyfJwcWL/8cqp gOGviYtHdRjdpOBzp5GJRx/+3k9V2SehtQdGtlXtaHLvKUFTBTdHF0YUNS4VBce1JtDq JaiyPS7IKMuCTuDJgaTd/7aH6WXtW+Pzin7gbKZ2uQwQClverVbxyK9WKjhjXOpGtW3u 6b6zc0E6Buajltx9cXMlGwNw9euxKZ5w6doVLz1ETXiE+ypgy8E8Rw3SSQEx9IBhJl3K eGC190vFNfymK913IblcrglNrOmxGTDOl7b6XGamQTtjHUy7UGiiOYrUnXcL2C9TUhSm sAog== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-id:precedence:references :in-reply-to:message-id:date:subject:cc:to:from; bh=z8N3K7cEzP29wp8OmV1x4SKhsG69RUNKzzZ3XrSlbEU=; fh=ZH+M10KtJZRqSjFjp1+uAYTvsWtGX6qxh/YkifVvH1o=; b=sJI8/O94K+Lo8hqS9ij9XgnMqI9/tZDoRsoYxiRO6c8gL4dkJK8CRXC9ypONPSUtMV dloM7OnVUG5GAzXmhVOU0nxudpez69a6T9LbP4z9oNWgm0toxKGXNATtmdjnxs8i/e3L GJaOMh9RkG0xidt4+36NDZnJ1GCyjfZ3tIPbJIqvh/u5Yun6h45asAA2QX2javDegHjO TIpx6G8KTxxkjM4FPQMZKi1/3ecw3kxV8meV0OS863G9KtdLeFg7U5KmUPZgo3ZhSfJB NTwT84gxFWYaQKQ0uPuDUfYwxfc35J1cCQuOuXosL7VWZzG43zcyrsFPg6Bws+Qb4dm+ meJQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=sk.com); spf=pass (google.com: domain of linux-kernel+bounces-80451-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-80451-ouuuleilei=gmail.com@vger.kernel.org" Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id x16-20020a170906711000b00a3e3c7a4969si1692952ejj.881.2024.02.25.19.08.26 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 25 Feb 2024 19:08:26 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-80451-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=sk.com); spf=pass (google.com: domain of linux-kernel+bounces-80451-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-80451-ouuuleilei=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id A74CB1F217CA for ; Mon, 26 Feb 2024 03:08:25 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id B75A81C6AE; Mon, 26 Feb 2024 03:06:43 +0000 (UTC) Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by smtp.subspace.kernel.org (Postfix) with ESMTP id D2E4C1BC57 for ; Mon, 26 Feb 2024 03:06:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=166.125.252.92 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708916801; cv=none; b=BrLNnA79YufVV73TqBcznKhUkZIeceAEoGFCTakT1BWHBdl1L/S/ubbwmn+C2x2HC+JSGdANl+oYz0ZtkjIf1UxwwsaPcYVUOGROR/kZhCN8cmGOpDamUfqcLb8Y4M2IBG4c0usEBAnmfoxisnw1CbbucQ54K7Kvs3+qjJh4UyI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708916801; c=relaxed/simple; bh=+1VdXIPG+erpF/AVFX/h+PqqlvJOGm1O0Tz9NB8RbNk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=fPJrtcfwGBTVorHRT7AkAMktxUdaD6Wqmmi2Pg97bDUaZcTM9he6QVHSWkHuwf+92MUkDtK7tKthAYarjzIHSO++wrLcxu9FXbBL9CogghnlKsJfgvAD+r50ZmVd6m6ZKajbTKZWRz4TSNA5KLbGXn5rX5zTZzCMPI1m0oKlEbA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com; spf=pass smtp.mailfrom=sk.com; arc=none smtp.client-ip=166.125.252.92 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sk.com X-AuditID: a67dfc5b-d6dff70000001748-4b-65dc00305618 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [RESEND PATCH v8 8/8] mm: Pause migrc mechanism at high memory pressure Date: Mon, 26 Feb 2024 12:06:13 +0900 Message-Id: <20240226030613.22366-9-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240226030613.22366-1-byungchul@sk.com> References: <20240226030613.22366-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrCLMWRmVeSWpSXmKPExsXC9ZZnoa4Bw51Ug5PTRS3mrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeJ47wEmi/n3PrNZ bN40ldni+JSpjBa/fwAVn5w1mcVBwON7ax+Lx85Zd9k9Fmwq9di8Qstj8Z6XTB6bVnWyeWz6 NInd4925c+weJ2b8ZvGYdzLQ4/2+q2weW3/ZeTROvcbm8XmTXABfFJdNSmpOZllqkb5dAldG V9t/toKpKhW/N1k1MO6X7WLk5JAQMJH42X+DEcZu+neHCcRmE1CXuHHjJzOILSJgJnGw9Q87 iM0scJdJ4kA/G4gtLBAgsbDzMiuIzSKgKnFmzjUwm1fAVOLOjgvsEDPlJVZvOAA0h4ODE2jO +/+GIGEhoJK/vVeB1nIBlXxmkzjyfB0LRL2kxMEVN1gmMPIuYGRYxSiUmVeWm5iZY6KXUZmX WaGXnJ+7iREY9stq/0TvYPx0IfgQowAHoxIP74IPt1OFWBPLiitzDzFKcDArifCGy9xMFeJN SaysSi3Kjy8qzUktPsQozcGiJM5r9K08RUggPbEkNTs1tSC1CCbLxMEp1cCokaC2g9Mq8Ixo ypMFAUuX3Dtj+C6pXKu1uPzwVqaOQ5+uGCTEH5Xt5FaX/M1ZUG8b1f9egfPBX3ahxQ21f3kv 31ro4dambOOnIjA7eVX+30BW+ZfbQi+ErRCwWd73wEfz0OqEsOe/3M+oXOdf9W6txuklXGms /Z41U7YauOWI9KisY5nPcl+JpTgj0VCLuag4EQCWNApudwIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrNLMWRmVeSWpSXmKPExsXC5WfdrGvAcCfVYMJNdos569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfHeA0wW 8+99ZrPYvGkqs8XxKVMZLX7/ACo+OWsyi4Ogx/fWPhaPnbPusnss2FTqsXmFlsfiPS+ZPDat 6mTz2PRpErvHu3Pn2D1OzPjN4jHvZKDH+31X2TwWv/jA5LH1l51H49RrbB6fN8kF8Edx2aSk 5mSWpRbp2yVwZXS1/WcrmKpS8XuTVQPjftkuRk4OCQETiaZ/d5hAbDYBdYkbN34yg9giAmYS B1v/sIPYzAJ3mSQO9LOB2MICARILOy+zgtgsAqoSZ+ZcA7N5BUwl7uy4wA4xU15i9YYDQHM4 ODiB5rz/bwgSFgIq+dt7lXECI9cCRoZVjCKZeWW5iZk5pnrF2RmVeZkVesn5uZsYgUG8rPbP xB2MXy67H2IU4GBU4uFd8OF2qhBrYllxZe4hRgkOZiUR3nCZm6lCvCmJlVWpRfnxRaU5qcWH GKU5WJTEeb3CUxOEBNITS1KzU1MLUotgskwcnFINjOZfFouliRezzvW/IjXh9MJlbuUnPmo1 9K0T/nN3ysVdU24d335DOrfyjYaUyipdoWO8lzOEqvZ0LuR0XbJrWuzpeYELJKVm61zXW8Um sz14af3MZ/bTebLOu/78Eb/f3/q+2RZTqfrutEnOeu8Cb5yrFtlh0HHpa5gYQ4Dhkuw4g6bv Suf+ciuxFGckGmoxFxUnAgAB/uHHXgIAAA== X-CFilter-Loop: Reflected Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791929253980892041 X-GMAIL-MSGID: 1791929253980892041 Regression was observed when the system is in high memory pressure with swap on, where migrc might keep a number of folios in its pending queue, which possibly makes it worse. So temporarily prevented migrc from working on that condition. Signed-off-by: Byungchul Park --- mm/internal.h | 20 ++++++++++++++++++++ mm/migrate.c | 18 +++++++++++++++++- mm/page_alloc.c | 13 +++++++++++++ 3 files changed, 50 insertions(+), 1 deletion(-) diff --git a/mm/internal.h b/mm/internal.h index ab02cb8306e2..55781f879fb2 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1285,6 +1285,8 @@ static inline void shrinker_debugfs_remove(struct dentry *debugfs_entry, #endif /* CONFIG_SHRINKER_DEBUG */ #if defined(CONFIG_MIGRATION) && defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) +extern atomic_t migrc_pause_cnt; + /* * Reset the indicator indicating there are no writable mappings at the * beginning of every rmap traverse for unmap. Migrc can work only when @@ -1313,6 +1315,21 @@ static inline bool can_migrc_test(void) return current->can_migrc && current->tlb_ubc_ro.flush_required; } +static inline void migrc_pause(void) +{ + atomic_inc(&migrc_pause_cnt); +} + +static inline void migrc_resume(void) +{ + atomic_dec(&migrc_pause_cnt); +} + +static inline bool migrc_paused(void) +{ + return !!atomic_read(&migrc_pause_cnt); +} + /* * Return the number of folios pending TLB flush that have yet to get * freed in the zone. @@ -1332,6 +1349,9 @@ void migrc_flush_end(struct tlbflush_unmap_batch *batch); static inline void can_migrc_init(void) {} static inline void can_migrc_fail(void) {} static inline bool can_migrc_test(void) { return false; } +static inline void migrc_pause(void) {} +static inline void migrc_resume(void) {} +static inline bool migrc_paused(void) { return false; } static inline int migrc_pending_nr_in_zone(struct zone *z) { return 0; } static inline bool migrc_flush_free_folios(void) { return false; } static inline void migrc_flush_start(void) {} diff --git a/mm/migrate.c b/mm/migrate.c index cbe5372f159e..fbc8586ed735 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -62,6 +62,12 @@ static struct tlbflush_unmap_batch migrc_ubc; static LIST_HEAD(migrc_folios); static DEFINE_SPINLOCK(migrc_lock); +/* + * Increase on entry of handling high memory pressure e.g. direct + * reclaim, decrease on the exit. See __alloc_pages_slowpath(). + */ +atomic_t migrc_pause_cnt = ATOMIC_INIT(0); + static void init_tlb_ubc(struct tlbflush_unmap_batch *ubc) { arch_tlbbatch_clear(&ubc->arch); @@ -1922,7 +1928,8 @@ static int migrate_pages_batch(struct list_head *from, */ init_tlb_ubc(&pending_ubc); do_migrc = IS_ENABLED(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) && - (reason == MR_DEMOTION || reason == MR_NUMA_MISPLACED); + (reason == MR_DEMOTION || reason == MR_NUMA_MISPLACED) && + !migrc_paused(); for (pass = 0; pass < nr_pass && retry; pass++) { retry = 0; @@ -1961,6 +1968,15 @@ static int migrate_pages_batch(struct list_head *from, continue; } + /* + * In case that the system is in high memory + * pressure, give up migrc mechanism this turn. + */ + if (unlikely(do_migrc && migrc_paused())) { + fold_ubc(tlb_ubc, &pending_ubc); + do_migrc = false; + } + can_migrc_init(); rc = migrate_folio_unmap(get_new_folio, put_new_folio, private, folio, &dst, mode, reason, diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 6ef0c22b1109..366777afce7f 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4072,6 +4072,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, unsigned int cpuset_mems_cookie; unsigned int zonelist_iter_cookie; int reserve_flags; + bool migrc_paused = false; restart: compaction_retries = 0; @@ -4203,6 +4204,16 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, if (page) goto got_pg; + /* + * The system is in very high memory pressure. Pause migrc from + * expanding its pending queue temporarily. + */ + if (!migrc_paused) { + migrc_pause(); + migrc_paused = true; + migrc_flush_free_folios(); + } + /* Caller is not willing to reclaim, we can't balance anything */ if (!can_direct_reclaim) goto nopage; @@ -4330,6 +4341,8 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, warn_alloc(gfp_mask, ac->nodemask, "page allocation failure: order:%u", order); got_pg: + if (migrc_paused) + migrc_resume(); return page; }