From patchwork Mon Feb 26 20:55:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 206913 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2328821dyb; Mon, 26 Feb 2024 12:57:26 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCXhyyEWxjbs+HHSP78qWHvOKi+qAg5kGM01JWaGot80NXSXW889BXlfhFvAOvdmiqbvFNOd4iMPhpjlG2usCzgmNwt/VQ== X-Google-Smtp-Source: AGHT+IHMsTf95rik/Ea+nyxTvAnQ8Vqd75YUPGr3mtq+ES2/4JyvNBwHfhL0HFDUSbXVKwi05hFp X-Received: by 2002:a05:6214:234d:b0:68f:7ce2:edcd with SMTP id hu13-20020a056214234d00b0068f7ce2edcdmr413937qvb.33.1708981046505; Mon, 26 Feb 2024 12:57:26 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708981046; cv=pass; d=google.com; s=arc-20160816; b=Pjj+lbe91k5uZF8xbYf4S8YmdnTKKg1ayvvPbzkFPF99TDqFO7EEOLnW7hbRcps9b1 /k9CgPFnoS8xhVenhoKy0b3HQTpPwgKRVsnlEL2gYCpbkshkyKYzpuQ0X5pfHDC27TMf L7x3OJls91p0aypxyusChcAIPUX5f8tHq05kmifwc0zJ0dTWFsLHd1k+02VLkit0xshi yXLCIYjGFCQFGmweF1RTca2EZTHj1VYsZAaLz7bvRv1fiZNGysS2sv7JDfGBtJArGkXH 6uHVPp7s2zMteoD0F/4F3wMJYLUM5DoReeWNPIAnUZ0VWVQYjlaF6twOlN0mvkw8zkiY pf3g== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:reply-to:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:dkim-signature :dkim-signature; bh=WJOIcMtfPYUOzzIot1iJ43LJYEvqslRWUKho7a0H8UI=; fh=chAWxDyUzR1fRdzViNE01Bcd9wPKK4ot8WI+7Hca+9w=; b=ap7VtzFd7r0/wBTGrKIE8RAk/H0PFBHehDmvWQMFLpca3DZuqp93AwZwe0xLHekMtT HCutWSLF9UB4DJLFgdrqYO5dQtXsu5ukuNFRiqdDJ4hU1ffsmSX4GqMgIJFvI3We3KwR ruzRLONqBtzzw9hcfCc2ADEY1Pak/B6D30+qUlaSqcvh79ISL9NGefTxWKv0sFk1jWag sRo2gSbnfrsvuoneeVecTTvI0Cne3d7ex18+gatZ3Hit5t7eZF1Dbz122F+oi5LX7uAr NtA5OnCUOY071jJHFaunTEW33aHagXNXycSuh8L6SXGRcpuwSCvInn+TKJ11gShXYObc SwzA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@sent.com header.s=fm1 header.b=hJT3EihE; dkim=pass header.i=@messagingengine.com header.s=fm1 header.b=Q5oLgEgT; arc=pass (i=1 spf=pass spfdomain=sent.com dkim=pass dkdomain=sent.com dkim=pass dkdomain=messagingengine.com dmarc=pass fromdomain=sent.com); spf=pass (google.com: domain of linux-kernel+bounces-82306-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82306-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=sent.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id 3-20020ad45ba3000000b0068ef74704a1si5959027qvq.453.2024.02.26.12.57.26 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 12:57:26 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-82306-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@sent.com header.s=fm1 header.b=hJT3EihE; dkim=pass header.i=@messagingengine.com header.s=fm1 header.b=Q5oLgEgT; arc=pass (i=1 spf=pass spfdomain=sent.com dkim=pass dkdomain=sent.com dkim=pass dkdomain=messagingengine.com dmarc=pass fromdomain=sent.com); spf=pass (google.com: domain of linux-kernel+bounces-82306-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82306-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=sent.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 4A29F1C23D17 for ; Mon, 26 Feb 2024 20:57:26 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 15770132C3F; Mon, 26 Feb 2024 20:56:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sent.com header.i=@sent.com header.b="hJT3EihE"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="Q5oLgEgT" Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7DD7712BEA8; Mon, 26 Feb 2024 20:56:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=66.111.4.29 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708980973; cv=none; b=HpyaGiOyCD6a1BHkyC9VbTI9H17AqWKJrszwGz5k5FcTbGNjOCCZ3/+t63utA027VUcY9pnE0bEbUbt8CwWuieVKktVQe7CAUSzeEkhGoub0HSmOhtABf9aILF7b476AkCbtv4JxBD2uFY/alPIGH5M/p1EPIImltktF6TUSfik= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708980973; c=relaxed/simple; bh=DrLa4gN39OR5eRtafDexilqk38FbJ1y40gc3XkbdbdE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=GUVRCxsS8WDreWGd65SaEDboWZCmepRHjqOOdFq6YlsfbCK5E+swdH/do7cZXkd42ykcXAu9qw4w8Xpsq5MaBYoxOOKy9sEbr8nWg4mEU/0YPQbo2IaYDWW0TyiJqY9a0KnShN7Zarufv3zxGedho2u3E5gje9wFHoa3iTa2TSg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=sent.com; spf=pass smtp.mailfrom=sent.com; dkim=pass (2048-bit key) header.d=sent.com header.i=@sent.com header.b=hJT3EihE; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=Q5oLgEgT; arc=none smtp.client-ip=66.111.4.29 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=sent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sent.com Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 102EB5C0098; Mon, 26 Feb 2024 15:56:10 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Mon, 26 Feb 2024 15:56:10 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:reply-to:subject:subject:to:to; s=fm1; t= 1708980970; x=1709067370; bh=WJOIcMtfPYUOzzIot1iJ43LJYEvqslRWUKh o7a0H8UI=; b=hJT3EihEH3zjGCbTc3s5Q1Im8ArJH/JWJT0hNhjb5JGTqkagU5W Q4hNKBxOinzn7ifNib2G/h16pcUfmI1/bqfx0QW9ipxOVL/z73oO0wIhUCRQrKIj QuyalNfeyGJhvoNoBMmNqOoHD+mZAQ4iOtS8zkzd6wxJ8UyxHfUZ7eiBm3YO68zB bD7v8Gb6j9T3B3OXeUkLj/ck3cczpXMLeakNCkel/zMJnic20K6wkIi/8IhA0LwU ozLTnlOLqs47hU/ehXc616Wd6Xip/Gxu9kBPBoJ7LBix5axVEearJu+HgHnojEdA XYKCy/eTe9paK7oZmye1sK5ka3F4fZ6sVbQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:reply-to:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1708980970; x=1709067370; bh=WJOIcMtfPYUOzzIot1iJ43LJYEvqslRWUKh o7a0H8UI=; b=Q5oLgEgTHIUoilhhh8ikGrW+fLZkXFebzxvzF+BCqVKz0mFbfKG dNwr2FwozKoylm8kaJXYNkoNFkoj/B6kyoT3uz81JV7UwxrafURxKGHdux6Te0b3 Bni3HMV50CWoYfsL+SLnqjBzl4Ui3boCn2vCF+xDO/x5DeJRTh6DLcbYuO+HHwOi AJLzPdTJ3wgzyZECzE9t6vANiyVrnMz4l/+p78bUdO5MVFirqZUmCBiB3alPjaAb 1ztIjR0tLINoL8O8036DDADyCfaJuyE2PsrIAa3kOJ76zPSKzcVqY/XV/ewhmKav mI2WkIBa2+ajP1dPmr7bZ3UmQU+9kDzWP7g== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrgedvgddugedtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkofgjfhhrgggtgfesthekredtredtjeenucfhrhhomhepkghi ucgjrghnuceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepje ekteekffelleekudfftdefvddtjeejuedtuedtteegjefgvedtfedujeekieevnecuvehl uhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepiihirdihrghnse hsvghnthdrtghomh X-ME-Proxy: Feedback-ID: iccd040f4:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 26 Feb 2024 15:56:08 -0500 (EST) From: Zi Yan To: "Pankaj Raghav (Samsung)" , linux-mm@kvack.org Cc: Zi Yan , "Matthew Wilcox (Oracle)" , David Hildenbrand , Yang Shi , Yu Zhao , "Kirill A . Shutemov" , Ryan Roberts , =?utf-8?q?Michal_Koutn=C3=BD?= , Roman Gushchin , "Zach O'Keefe" , Hugh Dickins , Luis Chamberlain , Andrew Morton , linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kselftest@vger.kernel.org Subject: [PATCH v5 1/8] mm/huge_memory: only split PMD mapping when necessary in unmap_folio() Date: Mon, 26 Feb 2024 15:55:27 -0500 Message-ID: <20240226205534.1603748-2-zi.yan@sent.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240226205534.1603748-1-zi.yan@sent.com> References: <20240226205534.1603748-1-zi.yan@sent.com> Reply-To: Zi Yan Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791996509945388219 X-GMAIL-MSGID: 1791996509945388219 From: Zi Yan As multi-size THP support is added, not all THPs are PMD-mapped, thus during a huge page split, there is no need to always split PMD mapping in unmap_folio(). Make it conditional. Signed-off-by: Zi Yan Reviewed-by: David Hildenbrand --- mm/huge_memory.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 28341a5067fb..b20e535e874c 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2727,11 +2727,14 @@ void vma_adjust_trans_huge(struct vm_area_struct *vma, static void unmap_folio(struct folio *folio) { - enum ttu_flags ttu_flags = TTU_RMAP_LOCKED | TTU_SPLIT_HUGE_PMD | - TTU_SYNC | TTU_BATCH_FLUSH; + enum ttu_flags ttu_flags = TTU_RMAP_LOCKED | TTU_SYNC | + TTU_BATCH_FLUSH; VM_BUG_ON_FOLIO(!folio_test_large(folio), folio); + if (folio_test_pmd_mappable(folio)) + ttu_flags |= TTU_SPLIT_HUGE_PMD; + /* * Anon pages need migration entries to preserve them, but file * pages can simply be left unmapped, then faulted back on demand. From patchwork Mon Feb 26 20:55:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 206914 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2328977dyb; Mon, 26 Feb 2024 12:57:49 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCUPsIFc+g9EGfwcG+PgqhZ8xowzN0vb206aD+DfKJoDOylt/jpO947qoHDRwNDq0uFFkLvLXWk8ucc6bnsuppxUueVogA== X-Google-Smtp-Source: AGHT+IEjt8vh3JjKJWQj8gbnpLoHIfom9v/7RN+9an1/kV5Do5ucrS1XxDLE5LV/5zSRpjSukXLP X-Received: by 2002:ac2:419a:0:b0:512:b555:17c4 with SMTP id z26-20020ac2419a000000b00512b55517c4mr4210094lfh.20.1708981069454; Mon, 26 Feb 2024 12:57:49 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708981069; cv=pass; d=google.com; s=arc-20160816; b=ZHsadF9SyixfNjXZ3VcE1B4krJgXto4iAcKEo8BmfMJRO0+kXwXgogF27eI/OZhYQg HFpDPeL4V0UFtb68KNL9M3o2+vLO2BGKByRoNpBWJUo9Gxq2VsDLf6rIhi4NNJR44ylY KdTYB+SJS+WSThSUVP7ut1GRhBtjoIzcYR9iO8rh+RHuBhlUbEmPWrw7Tgh1DY6M5KLL iWdvjrgNra6z+s+DAVf+vZ6WKv3frbYw2WGTEZJvsXaKw1fz6LxmN/QrXmhj5zv9bq5A NI6l0MMCNnDwmE2poqrkYn66UWh/B/F+AE0yY+OEB7vi19o5RztWLUIKxhpkYYwRIBwl F6iQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:reply-to:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:dkim-signature :dkim-signature; bh=ODE0ROICHsjD0yZP7iLxwoRZvmdJZzbZs1wZ6FX7X9k=; fh=chAWxDyUzR1fRdzViNE01Bcd9wPKK4ot8WI+7Hca+9w=; b=Z/auGxNy1uF3xocKWUf7zq4uc+fii/XrEYImuqHERU0b4q80r10yntAfYWMuAxhrHq yWM5ADVu0L56h9na8kPLzOW9w05iHggtZ2rBgFT7Eq55V89tj7LsqzrSNvih3FZbRb2X oVkvZUhRxoIfdF0EUQbRYNSbNFy9BeJ2Va2bqsMBSkhISIEizLsQFUvu+o5DH+tUTH9X W8D68HQT3uRm9/oS7ozYftH6rDJT6jgBYKcs9VJSFbyGBzEWXO8z/ZYhiq2UfWFhIBJW o5GWug6hkhrEo1SAYExbMwOlxVrV1KTK97t4YS9so8GwxE0iArvoLswkLoIe7krUk49k 4UkQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@sent.com header.s=fm1 header.b=T5xeqRqY; dkim=pass header.i=@messagingengine.com header.s=fm1 header.b=bp4JhDO6; arc=pass (i=1 spf=pass spfdomain=sent.com dkim=pass dkdomain=sent.com dkim=pass dkdomain=messagingengine.com dmarc=pass fromdomain=sent.com); spf=pass (google.com: domain of linux-kernel+bounces-82307-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82307-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=sent.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id k5-20020a1709063fc500b00a4300fce0basi82476ejj.489.2024.02.26.12.57.49 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 12:57:49 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-82307-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@sent.com header.s=fm1 header.b=T5xeqRqY; dkim=pass header.i=@messagingengine.com header.s=fm1 header.b=bp4JhDO6; arc=pass (i=1 spf=pass spfdomain=sent.com dkim=pass dkdomain=sent.com dkim=pass dkdomain=messagingengine.com dmarc=pass fromdomain=sent.com); spf=pass (google.com: domain of linux-kernel+bounces-82307-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82307-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=sent.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id DC5FA1F24479 for ; Mon, 26 Feb 2024 20:57:48 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 4E23D133411; Mon, 26 Feb 2024 20:56:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sent.com header.i=@sent.com header.b="T5xeqRqY"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="bp4JhDO6" Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C1223130E3E; Mon, 26 Feb 2024 20:56:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=66.111.4.29 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708980973; cv=none; b=kYmAGOTe7d4JbKWZfhjQwrkKef55gXew71DavrRWFJIjG6fgNy7SyktUS7A17KxB0wSP6zduxKZHTWg7IWyqb7ia+TSIMVCpIUD/CBIXKSqd5sBc60080gd0cEFwQfdBkbMQuoVnUVIdli6R/Et91ireEw4EWi9xZGOmFJjmQ54= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708980973; c=relaxed/simple; bh=6UZCu34iqxVetVOw3hYJVWt8DkK1Q3+qX0MxlTlzkTA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=mVLYRZpiSJ0mc0HCbsA+DjYr397x0l+v2shA9Qn0KyguF5HahBNCQWTGean2fpgHj085kBoheAo3Ce7WMeRnF4QkCRP9MOOokGBfy/abZzNJ11K3oT+5Okf3NzLUEHIOq+nvMHfheMUVMnW5rjIflZfDPhe+c+ZwfzYWF5ST9+g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=sent.com; spf=pass smtp.mailfrom=sent.com; dkim=pass (2048-bit key) header.d=sent.com header.i=@sent.com header.b=T5xeqRqY; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=bp4JhDO6; arc=none smtp.client-ip=66.111.4.29 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=sent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sent.com Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id CE54C5C00A4; Mon, 26 Feb 2024 15:56:10 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Mon, 26 Feb 2024 15:56:10 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:reply-to:subject:subject:to:to; s=fm1; t= 1708980970; x=1709067370; bh=ODE0ROICHsjD0yZP7iLxwoRZvmdJZzbZs1w Z6FX7X9k=; b=T5xeqRqYOCyBPMLYOa3Br+G8KABMxRYqzZHOqItB1AZrzBWc5++ v+Y7er2Z+fEtOY1M/UnNBzfbkuRk6ewQ6+GDS0M+g8TVhKmRbqYnUvTSSYENgpdl 12S7nVi060wPvZWE00vlxyzNriF+sFFCE/S+KC5feMd22qZNmgdU3fy5tMtsHfvV HT2RBYHADtA7sl/kk2Myd4K/s/xtjY5JP3jJkKGHEW4wIWZnQzJzlBw4hVValEJI pF1ogC001UIVIud1NFAAmMHBXEFLAMxinZXdOZVBdBQDgReZA/38hv+a2tRczoQt LOte/EUa97kieD8yiSWrp9rgQ2oOOQERj3A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:reply-to:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1708980970; x=1709067370; bh=ODE0ROICHsjD0yZP7iLxwoRZvmdJZzbZs1w Z6FX7X9k=; b=bp4JhDO6+sHuaMDjp3R19HRVAcTJ3cT1Z+YMXxBHek2OtVWeSIe bSoGWccfFZxHU/vp1hRV/Wx+/NGe2Nwi5VqPnD17ZmX+OO0h9nYYxtd5cYNsJpSZ k6g2gwk3TgcZZC/YqlkL5NwFr75VwxrFdHv/h4jJiey1C3KamfbEuZhqUTmIB1gT VqM+UEkjuQX2hgvT083qH5JYaQ5FeWxw3SNFXooaEzMoAAkw/w+VQ0Z7HSqLEbl8 wYgwWMCVwHlAZFQ5egIrbRKFj6xvh9Sln31XLTY/XDq2FTFETb5D5nn822mnX2BK oyJKkW5yKnszzEI9XSdKRE0NE9iNtLkdlEA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrgedvgddugedtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucdnrfhurhgthhgrshgvucdluddtmdenucfjughrpe fhvfevufffkffojghfrhggtgfgsehtkeertdertdejnecuhfhrohhmpegkihcujggrnhcu oeiiihdrhigrnhesshgvnhhtrdgtohhmqeenucggtffrrghtthgvrhhnpeettdekjeffke ejudelhfeuteekieejieehjeefvdefiefhteevgeejheegkefggfenucffohhmrghinhep khgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrg hilhhfrhhomhepiihirdihrghnsehsvghnthdrtghomh X-ME-Proxy: Feedback-ID: iccd040f4:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 26 Feb 2024 15:56:10 -0500 (EST) From: Zi Yan To: "Pankaj Raghav (Samsung)" , linux-mm@kvack.org Cc: Zi Yan , "Matthew Wilcox (Oracle)" , David Hildenbrand , Yang Shi , Yu Zhao , "Kirill A . Shutemov" , Ryan Roberts , =?utf-8?q?Michal_Koutn=C3=BD?= , Roman Gushchin , "Zach O'Keefe" , Hugh Dickins , Luis Chamberlain , Andrew Morton , linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kselftest@vger.kernel.org Subject: [PATCH v5 2/8] mm: Support order-1 folios in the page cache Date: Mon, 26 Feb 2024 15:55:28 -0500 Message-ID: <20240226205534.1603748-3-zi.yan@sent.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240226205534.1603748-1-zi.yan@sent.com> References: <20240226205534.1603748-1-zi.yan@sent.com> Reply-To: Zi Yan Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791996534025753299 X-GMAIL-MSGID: 1791996534025753299 From: "Matthew Wilcox (Oracle)" Folios of order 1 have no space to store the deferred list. This is not a problem for the page cache as file-backed folios are never placed on the deferred list. All we need to do is prevent the core MM from touching the deferred list for order 1 folios and remove the code which prevented us from allocating order 1 folios. Link: https://lore.kernel.org/linux-mm/90344ea7-4eec-47ee-5996-0c22f42d6a6a@google.com/ Signed-off-by: Matthew Wilcox (Oracle) Signed-off-by: Zi Yan --- mm/filemap.c | 2 -- mm/huge_memory.c | 19 +++++++++++++++---- mm/internal.h | 3 +-- mm/readahead.c | 3 --- 4 files changed, 16 insertions(+), 11 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index b7a21551fbc7..b4858d89f1b1 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1912,8 +1912,6 @@ struct folio *__filemap_get_folio(struct address_space *mapping, pgoff_t index, gfp_t alloc_gfp = gfp; err = -ENOMEM; - if (order == 1) - order = 0; if (order > 0) alloc_gfp |= __GFP_NORETRY | __GFP_NOWARN; folio = filemap_alloc_folio(alloc_gfp, order); diff --git a/mm/huge_memory.c b/mm/huge_memory.c index b20e535e874c..9840f312c08f 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -790,8 +790,10 @@ struct deferred_split *get_deferred_split_queue(struct folio *folio) void folio_prep_large_rmappable(struct folio *folio) { - VM_BUG_ON_FOLIO(folio_order(folio) < 2, folio); - INIT_LIST_HEAD(&folio->_deferred_list); + if (!folio || !folio_test_large(folio)) + return; + if (folio_order(folio) > 1) + INIT_LIST_HEAD(&folio->_deferred_list); folio_set_large_rmappable(folio); } @@ -3114,7 +3116,8 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) /* Prevent deferred_split_scan() touching ->_refcount */ spin_lock(&ds_queue->split_queue_lock); if (folio_ref_freeze(folio, 1 + extra_pins)) { - if (!list_empty(&folio->_deferred_list)) { + if (folio_order(folio) > 1 && + !list_empty(&folio->_deferred_list)) { ds_queue->split_queue_len--; list_del(&folio->_deferred_list); } @@ -3165,6 +3168,9 @@ void folio_undo_large_rmappable(struct folio *folio) struct deferred_split *ds_queue; unsigned long flags; + if (folio_order(folio) <= 1) + return; + /* * At this point, there is no one trying to add the folio to * deferred_list. If folio is not in deferred_list, it's safe @@ -3190,7 +3196,12 @@ void deferred_split_folio(struct folio *folio) #endif unsigned long flags; - VM_BUG_ON_FOLIO(folio_order(folio) < 2, folio); + /* + * Order 1 folios have no space for a deferred list, but we also + * won't waste much memory by not adding them to the deferred list. + */ + if (folio_order(folio) <= 1) + return; /* * The try_to_unmap() in page reclaim path might reach here too, diff --git a/mm/internal.h b/mm/internal.h index 2b7efffbe4d7..c4853ebfa030 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -420,8 +420,7 @@ static inline struct folio *page_rmappable_folio(struct page *page) { struct folio *folio = (struct folio *)page; - if (folio && folio_order(folio) > 1) - folio_prep_large_rmappable(folio); + folio_prep_large_rmappable(folio); return folio; } diff --git a/mm/readahead.c b/mm/readahead.c index 1e74455f908e..130c0e7df99f 100644 --- a/mm/readahead.c +++ b/mm/readahead.c @@ -514,9 +514,6 @@ void page_cache_ra_order(struct readahead_control *ractl, /* Don't allocate pages past EOF */ while (index + (1UL << order) - 1 > limit) order--; - /* THP machinery does not support order-1 */ - if (order == 1) - order = 0; err = ra_alloc_folio(ractl, index, mark, order, gfp); if (err) break; From patchwork Mon Feb 26 20:55:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 206915 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2329026dyb; Mon, 26 Feb 2024 12:57:55 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWnh+oWZgADd3f/yCfkLeDH/tvuVgmIzyp/7Iyx5RrinequIn0ZPPDhnR4mlMJ4SZw67gFS5YeSZxhSGI1gmmkLuWtvjQ== X-Google-Smtp-Source: AGHT+IEYml4kjdQGUiitB0OK4bBdBXhmL/1558yAb0ibRY6iKdNif84zQoGWFbjrc6kwaqgdj+JU X-Received: by 2002:a05:6a20:1206:b0:1a0:e855:b551 with SMTP id v6-20020a056a20120600b001a0e855b551mr249743pzf.21.1708981075745; Mon, 26 Feb 2024 12:57:55 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708981075; cv=pass; d=google.com; s=arc-20160816; b=HNEOEUW0wire3W7ZcKKqrGFV6ejzyVh/ugfM8YTNDndmDGyECzzOoYZzARbC1buP5T 8wQRAyk78OZsUAVgnI8kEVDLcNYDrQMmOCu3wrCTiKBYsxdoGTQx5rHF+haSQxQi66E5 RzKOJ4/peEuDcTTvc10PTCesYmmy6YVBWpvSJEE297aSfBcME/61KP1qCLiZavJqyFgB F6BlZikYoA57LjiEGj1YuVUvAqgXB5+z5B/JtzjfrpOyBgjnu8B68tEON80pXOQGl1yg R3c+BGFhh37x2uoPKHGOHJLfK8243dbW3AI//YSZ0chgZ4Rqmsla08PeamonvIXc0VgQ p/YQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:reply-to:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:dkim-signature :dkim-signature; bh=M9xzwbyGsmIoI4XohAUvU/zFDUrkDEXSepsOn39+bAw=; fh=chAWxDyUzR1fRdzViNE01Bcd9wPKK4ot8WI+7Hca+9w=; b=oBoWarj8SvTnMu/tLQWoMytHPCE8slCS+uylQG13G+4+2N94QR7tmApcidQP1K/G2O OflMurJLN2vwP29sck8meOHBJod8L6qzGzfEf+X1Mzp1nVZm0AOYrCWKKv47b6XejY7W k1u4RJ/3KTpue08idQvorqfzN+ZZuJTV6moN5LnJqzvQGMG7YTRWd0SowbJhQiFrDVIO 2T8+ijFB61aMwAMUaiQr0Qmu0OsiiKyUnXGxKv6/4drFapV0kdveMdeVREjGzVJJohX2 bTXt09YneYW7FLp02rnu3Jasw+NhRxJHeidoMqvHhAUvgrc7Lx7yYo+F732cLu+SZg2W oCbQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@sent.com header.s=fm1 header.b=YuhI4fCt; dkim=pass header.i=@messagingengine.com header.s=fm1 header.b=glx5DJxV; arc=pass (i=1 spf=pass spfdomain=sent.com dkim=pass dkdomain=sent.com dkim=pass dkdomain=messagingengine.com dmarc=pass fromdomain=sent.com); spf=pass (google.com: domain of linux-kernel+bounces-82308-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82308-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=sent.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id n129-20020a634087000000b005dc76f56a18si4234426pga.559.2024.02.26.12.57.55 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 12:57:55 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-82308-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@sent.com header.s=fm1 header.b=YuhI4fCt; dkim=pass header.i=@messagingengine.com header.s=fm1 header.b=glx5DJxV; arc=pass (i=1 spf=pass spfdomain=sent.com dkim=pass dkdomain=sent.com dkim=pass dkdomain=messagingengine.com dmarc=pass fromdomain=sent.com); spf=pass (google.com: domain of linux-kernel+bounces-82308-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82308-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=sent.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 830CA28AC7F for ; Mon, 26 Feb 2024 20:57:55 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id AC39D13342B; Mon, 26 Feb 2024 20:56:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sent.com header.i=@sent.com header.b="YuhI4fCt"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="glx5DJxV" Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AC6F213172E; Mon, 26 Feb 2024 20:56:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=66.111.4.29 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708980974; cv=none; b=a1yJ+ONs5k1wa1PR6FCTBMvHB6bQQKX1LXq4HkCApcUMR2ImG5h8DoeNwALd9Q8eyuqN0wauYHtdp6XjDHpnnjmwe6tqRCdfhLtOzd1oDaybDm8vvDMT3gE+Y6mcrnXWAkrZpYRdYK4gad36yDKJX+1w4Fm21EzYTz7zFX8Jyyk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708980974; c=relaxed/simple; bh=gkVdN6Ju83sqaexnUoOXBzO/NLvcTIiVWHWKKnGtjD0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=g7rhxbKoXo+rx0VZMrZc5IznObuAw6E2TSsxHri3NItTrx3lsBU2+/zABDEVKneCuv7ekNIhPJrRAkO/NEGP1A111FquXFttqapFveKIHvoQifMZX6ikT3cb4Bm77PJ7ivWDOjFLgR+J84g5vNtkGYQ/SQZHSmrHNxjBeYsBw44= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=sent.com; spf=pass smtp.mailfrom=sent.com; dkim=pass (2048-bit key) header.d=sent.com header.i=@sent.com header.b=YuhI4fCt; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=glx5DJxV; arc=none smtp.client-ip=66.111.4.29 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=sent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sent.com Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id B3D865C0048; Mon, 26 Feb 2024 15:56:11 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Mon, 26 Feb 2024 15:56:11 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:reply-to:subject:subject:to:to; s=fm1; t= 1708980971; x=1709067371; bh=M9xzwbyGsmIoI4XohAUvU/zFDUrkDEXSeps On39+bAw=; b=YuhI4fCtxyT7k5hVmSOEXMQbymAhui8JNegspEpvv/VYXQL41Ft pj6ydaFD0U8iL9cudlW457UYRQv8fEQ15V4gtUgJTlULXYCw1W+oHH7tpowniDfd 72lYDV66Ftv8Dihhg5hNlSrvNKyVSES3im67x3NJFo9TOSe5pkEVgSN0kxnXz9iS a2jmX5OXUupg8j2TZ0Wn/rQuxuqFPctJ3PsIQ2HiU53bTckPiZG5ROcg/fK+r2rp J4N6AHpbL4atGUiQ8tXxdfMqi+gal2Qc5fnKvro8o51nEGdfoFOOE+iuAKvvsSRv 7s4EaXf/bQ1fq/pkTQTq+ZndUEfxw8ypyqw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:reply-to:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1708980971; x=1709067371; bh=M9xzwbyGsmIoI4XohAUvU/zFDUrkDEXSeps On39+bAw=; b=glx5DJxVZIKumkxaIHAXIDP5YXKd5Im2fQ/t9CFv2OmBqMUwTJS SnpR5h2I0Ej/cY1kh6BRa4tCtDUgFzifVy7a/5qEul1MiRrxi9sq7VEewBj2bGwE dl13XHV1xbfKMBo+3KRdI7QlYUMK59X0qZ9IVut7HFp7dV5/IoM/cAxgFwQFdVgm Z77LLs9kJU8sYOmfV9Zm8ZvyR7hriEWUq3flyWvA0MM8r3mveLfT/qRIzqOHuvau llFl9HkR6kH1cToAl2JyeQsEJvO7f92ZErYrt3vo0FlHbsQmiSfga0yHN7qGhXbd lq6rkbeM3LiWAQWYeyS6T1JVDO69Ksg7fzw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrgedvgddugedtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkofgjfhhrgggtgfesthekredtredtjeenucfhrhhomhepkghi ucgjrghnuceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepje ekteekffelleekudfftdefvddtjeejuedtuedtteegjefgvedtfedujeekieevnecuvehl uhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepiihirdihrghnse hsvghnthdrtghomh X-ME-Proxy: Feedback-ID: iccd040f4:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 26 Feb 2024 15:56:10 -0500 (EST) From: Zi Yan To: "Pankaj Raghav (Samsung)" , linux-mm@kvack.org Cc: Zi Yan , "Matthew Wilcox (Oracle)" , David Hildenbrand , Yang Shi , Yu Zhao , "Kirill A . Shutemov" , Ryan Roberts , =?utf-8?q?Michal_Koutn=C3=BD?= , Roman Gushchin , "Zach O'Keefe" , Hugh Dickins , Luis Chamberlain , Andrew Morton , linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kselftest@vger.kernel.org Subject: [PATCH v5 3/8] mm/memcg: use order instead of nr in split_page_memcg() Date: Mon, 26 Feb 2024 15:55:29 -0500 Message-ID: <20240226205534.1603748-4-zi.yan@sent.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240226205534.1603748-1-zi.yan@sent.com> References: <20240226205534.1603748-1-zi.yan@sent.com> Reply-To: Zi Yan Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791996540580941257 X-GMAIL-MSGID: 1791996540580941257 From: Zi Yan We do not have non power of two pages, using nr is error prone if nr is not power-of-two. Use page order instead. Signed-off-by: Zi Yan Acked-by: David Hildenbrand --- include/linux/memcontrol.h | 4 ++-- mm/huge_memory.c | 5 +++-- mm/memcontrol.c | 3 ++- mm/page_alloc.c | 4 ++-- 4 files changed, 9 insertions(+), 7 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 4e4caeaea404..173bbb53c1ec 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1163,7 +1163,7 @@ static inline void memcg_memory_event_mm(struct mm_struct *mm, rcu_read_unlock(); } -void split_page_memcg(struct page *head, unsigned int nr); +void split_page_memcg(struct page *head, int order); unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, gfp_t gfp_mask, @@ -1621,7 +1621,7 @@ void count_memcg_event_mm(struct mm_struct *mm, enum vm_event_item idx) { } -static inline void split_page_memcg(struct page *head, unsigned int nr) +static inline void split_page_memcg(struct page *head, int order) { } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 9840f312c08f..96ac7c62c375 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2889,11 +2889,12 @@ static void __split_huge_page(struct page *page, struct list_head *list, struct lruvec *lruvec; struct address_space *swap_cache = NULL; unsigned long offset = 0; - unsigned int nr = thp_nr_pages(head); int i, nr_dropped = 0; + int order = folio_order(folio); + unsigned int nr = 1 << order; /* complete memcg works before add pages to LRU */ - split_page_memcg(head, nr); + split_page_memcg(head, order); if (folio_test_anon(folio) && folio_test_swapcache(folio)) { offset = swp_offset(folio->swap); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 95c3fccb321b..1a09f0e77c44 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3608,11 +3608,12 @@ void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size) /* * Because page_memcg(head) is not set on tails, set it now. */ -void split_page_memcg(struct page *head, unsigned int nr) +void split_page_memcg(struct page *head, int order) { struct folio *folio = page_folio(head); struct mem_cgroup *memcg = folio_memcg(folio); int i; + unsigned int nr = 1 << order; if (mem_cgroup_disabled() || !memcg) return; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 96839b210abe..a7a96bc97e0b 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2653,7 +2653,7 @@ void split_page(struct page *page, unsigned int order) for (i = 1; i < (1 << order); i++) set_page_refcounted(page + i); split_page_owner(page, 1 << order); - split_page_memcg(page, 1 << order); + split_page_memcg(page, order); } EXPORT_SYMBOL_GPL(split_page); @@ -4840,7 +4840,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order, struct page *last = page + nr; split_page_owner(page, 1 << order); - split_page_memcg(page, 1 << order); + split_page_memcg(page, order); while (page < --last) set_page_refcounted(last); From patchwork Mon Feb 26 20:55:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 206916 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2329176dyb; Mon, 26 Feb 2024 12:58:13 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCU+luvvJOeMKRq1uW3me8XmLYtzX1Td0Nckw5rHd55B+zpYHOYH8LyvCx4Oe0SA3Qxjf6s1aVjuzLGbIiRZu+SuynXFRw== X-Google-Smtp-Source: AGHT+IExUcmQJt9fEF+sDEZagIgtMmF9uf2F9/3P5m+sZThvm3e3C2g4jN1W8kHmT+bNn7l0qMnM X-Received: by 2002:a67:f51a:0:b0:470:9874:78b4 with SMTP id u26-20020a67f51a000000b00470987478b4mr5518540vsn.3.1708981093386; Mon, 26 Feb 2024 12:58:13 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708981093; cv=pass; d=google.com; s=arc-20160816; b=sBCJ/LTAJGVVR5wJBTQnd93xyadYhoFoVbu9l9S03KvBr4ndx+eC5qKkrJifQfdFg+ qMOsBTzsn1hv+YPTJJfCqyhK9/lR6fhRUVi/bsnzh7Shat+Xjg62Y3fw9VfMjwTdJe5m bmu5nwi3fqLf7ev434Cq81RPOGoQZmuCV4thN5R9ekUkGeCX2t/18DU5IswBXL4/N2Ax HAVEX7FZZnDt8aOuNJoLksqCEuk+oQAoM7ia8m56LEwiGc7loEylGsgOTTeJax/bg5dS nYK7zZbciJvX7zeoFv3OM5qRtlZG5Xw88et9dAVt5F4ZC5ZPF7Lb5azHxgQJhDaZYvnM PTIg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:reply-to:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:dkim-signature :dkim-signature; bh=k4tW1Ix2BuSNkN1WHXYSmN5X0ZeSMgYeZ0gG0yp9vIY=; fh=chAWxDyUzR1fRdzViNE01Bcd9wPKK4ot8WI+7Hca+9w=; b=hJktpknBZqFnXCBSR9DDBlggf9C18oWB+nJdxzQW6yZjEI7JE3hxR1PpRjMnNTd+dW S0TSD8W07o1/KOT3eKUbUXbJfyU6U7g3SG2yXktVcLlTgsXvICsAUzlfvA1OzAZpx4h5 aY4ROYlk6PViMn1U1b7zkfed3iy5npu9qjuluURm0jnagnzcLlrbWvTAum2N6Zau348o 1qkxgcesxx+/J1ouzr7QiBOmf22gWec5MVMZIKDGKO/WLZB060MfD0jimpVdoKE/wYUO 6icN63M+CTIc0ct+iIaclpT86n+CmgErqZWARmBfgnykcxYUuHnmZa2C2dFUcnE9s8Ci Oa8g==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@sent.com header.s=fm1 header.b=PsGtgEud; dkim=pass header.i=@messagingengine.com header.s=fm1 header.b=MXQaLIK2; arc=pass (i=1 spf=pass spfdomain=sent.com dkim=pass dkdomain=sent.com dkim=pass dkdomain=messagingengine.com dmarc=pass fromdomain=sent.com); spf=pass (google.com: domain of linux-kernel+bounces-82309-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82309-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=sent.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id s28-20020a05620a031c00b00787bf3f02f7si5698441qkm.168.2024.02.26.12.58.13 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 12:58:13 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-82309-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@sent.com header.s=fm1 header.b=PsGtgEud; dkim=pass header.i=@messagingengine.com header.s=fm1 header.b=MXQaLIK2; arc=pass (i=1 spf=pass spfdomain=sent.com dkim=pass dkdomain=sent.com dkim=pass dkdomain=messagingengine.com dmarc=pass fromdomain=sent.com); spf=pass (google.com: domain of linux-kernel+bounces-82309-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82309-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=sent.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 097D31C243C5 for ; Mon, 26 Feb 2024 20:58:13 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 9DB2D130E41; Mon, 26 Feb 2024 20:56:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sent.com header.i=@sent.com header.b="PsGtgEud"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="MXQaLIK2" Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8207113173A; Mon, 26 Feb 2024 20:56:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=66.111.4.29 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708980975; cv=none; b=adio0bnwDO7zp9R+ZmuCsQ/KdMWV6jjod1IuwIV+zKJxgMM1rCZhe1zb7bz2JYl4oEcvmT0ze9uGu7aQXJZxjgICezpqueMDQ6eiQc1TomjOtAKkLnWOW8DshKdS+Z1zzBNzKuRz+d6XkT4wkaOXI1LPSE0Pc1Hpn4XOu6NLfME= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708980975; c=relaxed/simple; bh=HbrSwnkkjzgDPr9Qqs/XsufXHKtUIzBk6wsYMI1kzqY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=OZFTAulElikRLo8Aaybe96XnzAE8A7B/OZRPl6QkejgnPKADrxKakUA3iWIZCa8HtIENNhx6HoDnZcfEQ+r2/dOWORhcnLaelkPzdwoImPrOD12rlK+zUQSTsxkUcGDCGzNJ+FMhqi52w1EyGlMOwn6hjnGzpCwoBLYIfWo7qPc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=sent.com; spf=pass smtp.mailfrom=sent.com; dkim=pass (2048-bit key) header.d=sent.com header.i=@sent.com header.b=PsGtgEud; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=MXQaLIK2; arc=none smtp.client-ip=66.111.4.29 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=sent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sent.com Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 845035C0083; Mon, 26 Feb 2024 15:56:12 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Mon, 26 Feb 2024 15:56:12 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:reply-to:subject:subject:to:to; s=fm1; t= 1708980972; x=1709067372; bh=k4tW1Ix2BuSNkN1WHXYSmN5X0ZeSMgYeZ0g G0yp9vIY=; b=PsGtgEudtl5OWZIMN/QmYcFTzFIwskYJv2SShnXgHjfbIBRHd6c tLFxkANerh0TDSw0aIJ+BRFw3TtwrtnEWmHqtjReaka5XRT4kCVu+/JxTHJUARVP TTaS/ervzETdMoGvlOvQkGU5Vr8k5tDe2HWq4O22UqWyxXceqtEs8oF4xp6w3ILj E+Nn7cDMv2KP+WObR1wfXgZt4SVjLn62IgkbUTVyraMDgD4LC0tnPeSgCrhiQXlI dpijTzey8JMArvsD76watmwYOLbgBoLoa6lmS0djEggreXdhpxLkoa+zAeqDTiVh XpV3/H7xUFtWwTsw2W+RLI8tAthTa0kVP7w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:reply-to:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1708980972; x=1709067372; bh=k4tW1Ix2BuSNkN1WHXYSmN5X0ZeSMgYeZ0g G0yp9vIY=; b=MXQaLIK2tOLWX7tSDG9quAtyz0XtE5uEWLvjXAt6WWz4fuCtIq+ iW5V2mlkR6XtmBDh4IXp0NZH4C8YwuqKervIeM/9EVpHBYcPoWcIkSre/3nQS4z4 deN9QC9eqGj98qO7roEFIt9SV8tO7akXARi4gsDw06jxzAwz5zubvukbV7i4gDO/ zRNBPrfZQzljXqj9vbHt0+V2dLDxewtoVWSQgxCBKRL013TC1g1nj0dxzgkun57/ ZLz8/iUoCZO+anVVQtkHxMUVyw526I4oYS5ikt412Vg+KTvUHxFRAI9uUJp+t7bp th0cOJ+ulQTE92LtOYFZ9c3xS3VyDo9UjLA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrgedvgddugedtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkofgjfhhrgggtgfesthekredtredtjeenucfhrhhomhepkghi ucgjrghnuceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepje ekteekffelleekudfftdefvddtjeejuedtuedtteegjefgvedtfedujeekieevnecuvehl uhhsthgvrhfuihiivgepvdenucfrrghrrghmpehmrghilhhfrhhomhepiihirdihrghnse hsvghnthdrtghomh X-ME-Proxy: Feedback-ID: iccd040f4:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 26 Feb 2024 15:56:11 -0500 (EST) From: Zi Yan To: "Pankaj Raghav (Samsung)" , linux-mm@kvack.org Cc: Zi Yan , "Matthew Wilcox (Oracle)" , David Hildenbrand , Yang Shi , Yu Zhao , "Kirill A . Shutemov" , Ryan Roberts , =?utf-8?q?Michal_Koutn=C3=BD?= , Roman Gushchin , "Zach O'Keefe" , Hugh Dickins , Luis Chamberlain , Andrew Morton , linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kselftest@vger.kernel.org Subject: [PATCH v5 4/8] mm/page_owner: use order instead of nr in split_page_owner() Date: Mon, 26 Feb 2024 15:55:30 -0500 Message-ID: <20240226205534.1603748-5-zi.yan@sent.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240226205534.1603748-1-zi.yan@sent.com> References: <20240226205534.1603748-1-zi.yan@sent.com> Reply-To: Zi Yan Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791996558585307723 X-GMAIL-MSGID: 1791996558585307723 From: Zi Yan We do not have non power of two pages, using nr is error prone if nr is not power-of-two. Use page order instead. Signed-off-by: Zi Yan Acked-by: David Hildenbrand --- include/linux/page_owner.h | 9 ++++----- mm/huge_memory.c | 2 +- mm/page_alloc.c | 4 ++-- mm/page_owner.c | 3 ++- 4 files changed, 9 insertions(+), 9 deletions(-) diff --git a/include/linux/page_owner.h b/include/linux/page_owner.h index 119a0c9d2a8b..2b39c8e19d98 100644 --- a/include/linux/page_owner.h +++ b/include/linux/page_owner.h @@ -11,7 +11,7 @@ extern struct page_ext_operations page_owner_ops; extern void __reset_page_owner(struct page *page, unsigned short order); extern void __set_page_owner(struct page *page, unsigned short order, gfp_t gfp_mask); -extern void __split_page_owner(struct page *page, unsigned int nr); +extern void __split_page_owner(struct page *page, int order); extern void __folio_copy_owner(struct folio *newfolio, struct folio *old); extern void __set_page_owner_migrate_reason(struct page *page, int reason); extern void __dump_page_owner(const struct page *page); @@ -31,10 +31,10 @@ static inline void set_page_owner(struct page *page, __set_page_owner(page, order, gfp_mask); } -static inline void split_page_owner(struct page *page, unsigned int nr) +static inline void split_page_owner(struct page *page, int order) { if (static_branch_unlikely(&page_owner_inited)) - __split_page_owner(page, nr); + __split_page_owner(page, order); } static inline void folio_copy_owner(struct folio *newfolio, struct folio *old) { @@ -59,8 +59,7 @@ static inline void set_page_owner(struct page *page, unsigned int order, gfp_t gfp_mask) { } -static inline void split_page_owner(struct page *page, - unsigned short order) +static inline void split_page_owner(struct page *page, int order) { } static inline void folio_copy_owner(struct folio *newfolio, struct folio *folio) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 96ac7c62c375..2b95dbc95fae 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2933,7 +2933,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, unlock_page_lruvec(lruvec); /* Caller disabled irqs, so they are still disabled here */ - split_page_owner(head, nr); + split_page_owner(head, order); /* See comment in __split_huge_page_tail() */ if (PageAnon(head)) { diff --git a/mm/page_alloc.c b/mm/page_alloc.c index a7a96bc97e0b..eae77e76a671 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2652,7 +2652,7 @@ void split_page(struct page *page, unsigned int order) for (i = 1; i < (1 << order); i++) set_page_refcounted(page + i); - split_page_owner(page, 1 << order); + split_page_owner(page, order); split_page_memcg(page, order); } EXPORT_SYMBOL_GPL(split_page); @@ -4839,7 +4839,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order, struct page *page = virt_to_page((void *)addr); struct page *last = page + nr; - split_page_owner(page, 1 << order); + split_page_owner(page, order); split_page_memcg(page, order); while (page < --last) set_page_refcounted(last); diff --git a/mm/page_owner.c b/mm/page_owner.c index e56c1e92eccf..b678f7a6e702 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -306,11 +306,12 @@ void __set_page_owner_migrate_reason(struct page *page, int reason) page_ext_put(page_ext); } -void __split_page_owner(struct page *page, unsigned int nr) +void __split_page_owner(struct page *page, int order) { int i; struct page_ext *page_ext = page_ext_get(page); struct page_owner *page_owner; + unsigned int nr = 1 << order; if (unlikely(!page_ext)) return; From patchwork Mon Feb 26 20:55:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 206917 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2329223dyb; Mon, 26 Feb 2024 12:58:22 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCXaDYgEJ8OwVVp9hHA8wlEIpa8jFF9SJ1dqU9xnC6g+2bVcHOrrOK1dYhIrLVpZSgHLmzGk22ZG7HtbyDZEo4JITaw+8Q== X-Google-Smtp-Source: AGHT+IG1rwv/ToqDq4TDaH5haHVG3zLcnTBtq7/oh8lA7FnbAA3dka/DKEk5rgwhi39PLKrQb7w6 X-Received: by 2002:a05:6214:21ed:b0:68f:f08e:b64e with SMTP id p13-20020a05621421ed00b0068ff08eb64emr581347qvj.8.1708981101966; Mon, 26 Feb 2024 12:58:21 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708981101; cv=pass; d=google.com; s=arc-20160816; b=LaSr2uMZGzspTF03QLIasFvZAEShIt+LRTjLCKa+nmTRofkdhaGpQpYws+KpoJyTib ygpBAMokMzV4GPo8dr4933qZGopCKjDA9YWQjHA18KTTYDlktLYXeueVyL+PMjN1Zypg tGtzZIShA0Ts3Y2nTCE6FErhXnGgF3ftYv1v0ccpcPd/jvqa2vnbRKUzvfEG4m7YOpeH C532fBsdbwNbvkyyvxKIkKGvKIrhHSQOzBk6Yf1HxLQhuQqkMY7tTAbOw0mz40wLx78k hLECvkxnTgcbQ9pBi2gr5Js/GGgQr1t3fjIumpmRukcTgNS2DcoIJNLGo3+LsugepL+M xtrQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:reply-to:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:dkim-signature :dkim-signature; bh=7C1YvLv3wVMtcqo8/X9e6+Vvqy6Z7pLbOms0IX3NOhM=; fh=chAWxDyUzR1fRdzViNE01Bcd9wPKK4ot8WI+7Hca+9w=; b=CLrKkcrfrDNAwQAUYeNGIpNvoMsWxMU1DU1usL9tMgQT282EP7T4VZZwNnz3r5lWby hqGEa9sHcF2BvwaW1fRY/jNq2IA93/SUzK4EzcTq2BzwnEe3VY1gy7i57K1VGXUsCHND zgDOGtLbHODnQYVBXf/rZBT56CDReEkldePA7JZuoCu5ZtxziRtezFjSk/ETp8MfhnXa werDiqlSakcrZhEgjksEtMbAiznHnqKdT/Frc8+VWyHMG/6p+5dm17OVRZ3egZ8o1d24 6GVbVyXgLeqFfiOPPu1FyIQYy0dSkVfzVDiB6CCyacfLugAijtbkiDF/ItPe5sdd2HJN n6Bw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@sent.com header.s=fm1 header.b=G0xdGaA+; dkim=pass header.i=@messagingengine.com header.s=fm1 header.b="pUr2/5uN"; arc=pass (i=1 spf=pass spfdomain=sent.com dkim=pass dkdomain=sent.com dkim=pass dkdomain=messagingengine.com dmarc=pass fromdomain=sent.com); spf=pass (google.com: domain of linux-kernel+bounces-82310-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82310-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=sent.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id e14-20020a0562141d0e00b0068fe35db61asi5970557qvd.61.2024.02.26.12.58.21 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 12:58:21 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-82310-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@sent.com header.s=fm1 header.b=G0xdGaA+; dkim=pass header.i=@messagingengine.com header.s=fm1 header.b="pUr2/5uN"; arc=pass (i=1 spf=pass spfdomain=sent.com dkim=pass dkdomain=sent.com dkim=pass dkdomain=messagingengine.com dmarc=pass fromdomain=sent.com); spf=pass (google.com: domain of linux-kernel+bounces-82310-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82310-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=sent.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id B0C2E1C24515 for ; Mon, 26 Feb 2024 20:58:21 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 41B6E13472B; Mon, 26 Feb 2024 20:56:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sent.com header.i=@sent.com header.b="G0xdGaA+"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="pUr2/5uN" Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 47D5A131E3A; Mon, 26 Feb 2024 20:56:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=66.111.4.29 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708980976; cv=none; b=MxtOThdKyyLgGZxiiPqsjfzf3dHDvUc/PnaRdwjA2wos6Y0/uwI2iWs3AijMJyef2Awhu/uXnt0O48WK48EKcWQSO+yaViuLK2JobeoPKuAZWSBSHRsgPKn9ZjhPS/2gTbWL9PI5Zbd9p5nXUR4Wlt6Sxj9vBN2g67+ThG5vN8E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708980976; c=relaxed/simple; bh=tQ/FHG8XnUwWFx1UmtP5Vr1SwbTk4PDYio7TLDjvkVU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=nGFF3ev/9elZ5rHsIbMjdMSTJvjbCFcEj6sy2AdW5PdM+wvUr5tfz/8mqemNNOZ4eLuA74KypLQEL2RV0chgKktPFAIMZIDh6oZg9rLPIuCqrXKql8DVaJ0ihhObseMonN8jxB0af0xoY/hYMW63js+vMldD+MveCGjzbglVJb4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=sent.com; spf=pass smtp.mailfrom=sent.com; dkim=pass (2048-bit key) header.d=sent.com header.i=@sent.com header.b=G0xdGaA+; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=pUr2/5uN; arc=none smtp.client-ip=66.111.4.29 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=sent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sent.com Received: from compute7.internal (compute7.nyi.internal [10.202.2.48]) by mailout.nyi.internal (Postfix) with ESMTP id 5D5415C00A6; Mon, 26 Feb 2024 15:56:13 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute7.internal (MEProxy); Mon, 26 Feb 2024 15:56:13 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:reply-to:subject:subject:to:to; s=fm1; t= 1708980973; x=1709067373; bh=7C1YvLv3wVMtcqo8/X9e6+Vvqy6Z7pLbOms 0IX3NOhM=; b=G0xdGaA+uEkADF+wiXa48pt3PQhHb+De9LVKL4DkWp2f+BkyswB Xn0t6TOeQSMlGCZdWmNVwylDGYEGktrxuMzFQ6HjOK0MgYNC/7rkylv7Zf+1m5hz WNatQYLZu4uE3JehluNE2U1hIC66n0giMdDi9+Dw1BhO1Ndg1P82kkDw8ZleMrtm U8A4n7JXXdSwUD4c4a3Ujlj5O3YEjmLM9MAPwBym1VgjGgJyWjEtFDOot+p3UklT HD4LQVHaNLIlmjhmMVr1SI/xsVtZ7vCbEiu6KIwctGI9/rAX38UwjVN8OY/1io20 RqLO8O8vhG7XRsTaPZjD+bm5Fgt+VbrLghw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:reply-to:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1708980973; x=1709067373; bh=7C1YvLv3wVMtcqo8/X9e6+Vvqy6Z7pLbOms 0IX3NOhM=; b=pUr2/5uN/u4JfGEQtK5c/fXMKwAQ1WngoArkr9zDevWWHLryFC2 +D9ry22gMXo9f5eRO4YyqaeFMMZHFj84Cv6cgMhfxiQIi0/0XSQ83NhjHGbF+CXm PZvsiMitxw6gqKPR62nWCTs/I1sLcl0aUjVClavOJ5CWyDFz8+YcmtJY6+oz5ZcU 7j8/Tzrq/ObfcsJ5RuinS2fSzLBOVEeBxv2iiDl0uOxuXBrstQHXJW2+3iu9Kf0H uf8RzWoc+ZGH9l3aKlRJFL6qKLp0Omq0O/7Lv6QvdzHaBH7ziXbcBJ09Ee3iI4fc qLjBZ3qTLfgOzj1Afn7XMInHh76ed8My0/A== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrgedvgddugedtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkofgjfhhrgggtgfesthekredtredtjeenucfhrhhomhepkghi ucgjrghnuceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepje ekteekffelleekudfftdefvddtjeejuedtuedtteegjefgvedtfedujeekieevnecuvehl uhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepiihirdihrghnse hsvghnthdrtghomh X-ME-Proxy: Feedback-ID: iccd040f4:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 26 Feb 2024 15:56:12 -0500 (EST) From: Zi Yan To: "Pankaj Raghav (Samsung)" , linux-mm@kvack.org Cc: Zi Yan , "Matthew Wilcox (Oracle)" , David Hildenbrand , Yang Shi , Yu Zhao , "Kirill A . Shutemov" , Ryan Roberts , =?utf-8?q?Michal_Koutn=C3=BD?= , Roman Gushchin , "Zach O'Keefe" , Hugh Dickins , Luis Chamberlain , Andrew Morton , linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kselftest@vger.kernel.org Subject: [PATCH v5 5/8] mm: memcg: make memcg huge page split support any order split. Date: Mon, 26 Feb 2024 15:55:31 -0500 Message-ID: <20240226205534.1603748-6-zi.yan@sent.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240226205534.1603748-1-zi.yan@sent.com> References: <20240226205534.1603748-1-zi.yan@sent.com> Reply-To: Zi Yan Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791996568094511441 X-GMAIL-MSGID: 1791996568094511441 From: Zi Yan It sets memcg information for the pages after the split. A new parameter new_order is added to tell the order of subpages in the new page, always 0 for now. It prepares for upcoming changes to support split huge page to any lower order. Signed-off-by: Zi Yan Acked-by: David Hildenbrand --- include/linux/memcontrol.h | 4 ++-- mm/huge_memory.c | 2 +- mm/memcontrol.c | 11 ++++++----- mm/page_alloc.c | 4 ++-- 4 files changed, 11 insertions(+), 10 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 173bbb53c1ec..9a2dea92be0e 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1163,7 +1163,7 @@ static inline void memcg_memory_event_mm(struct mm_struct *mm, rcu_read_unlock(); } -void split_page_memcg(struct page *head, int order); +void split_page_memcg(struct page *head, int old_order, int new_order); unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, gfp_t gfp_mask, @@ -1621,7 +1621,7 @@ void count_memcg_event_mm(struct mm_struct *mm, enum vm_event_item idx) { } -static inline void split_page_memcg(struct page *head, int order) +static inline void split_page_memcg(struct page *head, int old_order, int new_order) { } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 2b95dbc95fae..5d4b7c17b9bc 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2894,7 +2894,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, unsigned int nr = 1 << order; /* complete memcg works before add pages to LRU */ - split_page_memcg(head, order); + split_page_memcg(head, order, 0); if (folio_test_anon(folio) && folio_test_swapcache(folio)) { offset = swp_offset(folio->swap); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 1a09f0e77c44..669bc8de3780 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3608,23 +3608,24 @@ void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size) /* * Because page_memcg(head) is not set on tails, set it now. */ -void split_page_memcg(struct page *head, int order) +void split_page_memcg(struct page *head, int old_order, int new_order) { struct folio *folio = page_folio(head); struct mem_cgroup *memcg = folio_memcg(folio); int i; - unsigned int nr = 1 << order; + unsigned int old_nr = 1 << old_order; + unsigned int new_nr = 1 << new_order; if (mem_cgroup_disabled() || !memcg) return; - for (i = 1; i < nr; i++) + for (i = new_nr; i < old_nr; i += new_nr) folio_page(folio, i)->memcg_data = folio->memcg_data; if (folio_memcg_kmem(folio)) - obj_cgroup_get_many(__folio_objcg(folio), nr - 1); + obj_cgroup_get_many(__folio_objcg(folio), old_nr / new_nr - 1); else - css_get_many(&memcg->css, nr - 1); + css_get_many(&memcg->css, old_nr / new_nr - 1); } #ifdef CONFIG_SWAP diff --git a/mm/page_alloc.c b/mm/page_alloc.c index eae77e76a671..c31a468fe317 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2653,7 +2653,7 @@ void split_page(struct page *page, unsigned int order) for (i = 1; i < (1 << order); i++) set_page_refcounted(page + i); split_page_owner(page, order); - split_page_memcg(page, order); + split_page_memcg(page, order, 0); } EXPORT_SYMBOL_GPL(split_page); @@ -4840,7 +4840,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order, struct page *last = page + nr; split_page_owner(page, order); - split_page_memcg(page, order); + split_page_memcg(page, order, 0); while (page < --last) set_page_refcounted(last); From patchwork Mon Feb 26 20:55:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 206918 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2329272dyb; Mon, 26 Feb 2024 12:58:29 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCUue3+6/+LmdMBZN2gwHleEP/iEb8L7RkXIGT+ah/uCCGpmp8N6TbWGNwqu171mOawP6fYkJxFBWEujvUK+9kSM0r9vlQ== X-Google-Smtp-Source: AGHT+IEtUUxOFn0utA1fWr9GdtpAz+lVWf7XJhUn5g1ItNR/S4uXCwR204a9ueBEXdETtmc/m/T7 X-Received: by 2002:a17:902:c951:b0:1dc:a82a:2316 with SMTP id i17-20020a170902c95100b001dca82a2316mr3330958pla.35.1708981109171; Mon, 26 Feb 2024 12:58:29 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708981109; cv=pass; d=google.com; s=arc-20160816; b=v1pM9g/75EiD0WKLI8Pbg41gCpdh3VSZ0HdBvSaULkM7HzocAgYRbFtiSzACxMLmau E6KUPMulQnZivv3xermv+NxWvhd5h1aehm/NI73YdCycSTZl8Lb/h0uxf5RLnUfwQRoX vmtfQQQwliGC2bYU9zSYkUpTC1ID81vxd5Ry9lplbJGWxgkf1RvpKTKoB3wlhppwQEWA 7PZ0vuwC8eohC7/01dxTIprjnMgM00eY+bRWZhXYyoyMV3dEuJW/Kyb4CIWEgdcV7tf0 QaJSpRFobD/l5ZsfNT4kfZv9kVPz6EVU3+HH0xW9akyXaps4XsxxgNqRnhCfjsHIZLw1 K2Pw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:reply-to:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:dkim-signature :dkim-signature; bh=364J4ng7aCg7Ddk79ejWdMNIz/yxFIeeqLKxisnieh8=; fh=chAWxDyUzR1fRdzViNE01Bcd9wPKK4ot8WI+7Hca+9w=; b=DjDVNFHJQWshWQSYAjyuvK6vkJVaLg22SUhbo66dVvsHFWssKLbLbF5hsRrlpj/A1/ R+ffTB2PuI/fMp0JY/LIAMyzJv3NxRfxeOmJ8zeFvyKGsMX1wh1pTvG9qve1VjjcZLVH V3Wy3BJc0MY8Q8BTHkBvGFTIgmA3PSwrOpigpakm07w3Erdi+suWhxrpxLsmvJ/a/oWB I2hlL5IF02reE4er5uxbK3c1LsbaF1gz80R4UCuAXzBgaa0B2ku5XVBKyBq5PkKNHvFY jDY/WpEnW+RNjqXKBzb95pXO+DwkeHf0pMPT1ZLKp9G5WiHaEEF7TRT1qU7kycA1aKFN sPUQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@sent.com header.s=fm1 header.b=iQ1P+dj1; dkim=pass header.i=@messagingengine.com header.s=fm1 header.b=RCNXSmar; arc=pass (i=1 spf=pass spfdomain=sent.com dkim=pass dkdomain=sent.com dkim=pass dkdomain=messagingengine.com dmarc=pass fromdomain=sent.com); spf=pass (google.com: domain of linux-kernel+bounces-82311-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82311-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=sent.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id w16-20020a1709026f1000b001dcb32068c2si173110plk.158.2024.02.26.12.58.29 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 12:58:29 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-82311-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@sent.com header.s=fm1 header.b=iQ1P+dj1; dkim=pass header.i=@messagingengine.com header.s=fm1 header.b=RCNXSmar; arc=pass (i=1 spf=pass spfdomain=sent.com dkim=pass dkdomain=sent.com dkim=pass dkdomain=messagingengine.com dmarc=pass fromdomain=sent.com); spf=pass (google.com: domain of linux-kernel+bounces-82311-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82311-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=sent.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id F26F628B14C for ; Mon, 26 Feb 2024 20:58:28 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 246F213474B; Mon, 26 Feb 2024 20:56:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sent.com header.i=@sent.com header.b="iQ1P+dj1"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="RCNXSmar" Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 13C88132466; Mon, 26 Feb 2024 20:56:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=66.111.4.29 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708980976; cv=none; b=cxN2InN0wWrPpPwy67s894h/sP00tOwyT4QtttVuG8zUpviw6TH69cywDFCxDEg5C8YCOr7rqUjATiE5xTtpUO0KIVwCgmmAW+iq1nj6SUIWuCYQzZmI7epEpJ5nr+gLHex5F0Bo+F6AiLBdVaKS5F461H5I4m9pjkwxWkzlY48= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708980976; c=relaxed/simple; bh=ZtZL/AFG8WZ7Dh5a7UohfkhMN1n6gq+cssXbZdxud4I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=EFu3ByB/Y+uRpdp6BEpfze6iouc/1jNFpS/k2MHLxx4AW/PF08BfMoj8S2tOr36saQdG0A1tVck5y0enFcvdVR3dxXWXAqwHGU1JrlnOPHGg4m+tBUde6WHF/5RSmCm+0LcRQkXKBT+dF0xyGqEraA7+8s7RFwQCZaQAgknqgDE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=sent.com; spf=pass smtp.mailfrom=sent.com; dkim=pass (2048-bit key) header.d=sent.com header.i=@sent.com header.b=iQ1P+dj1; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=RCNXSmar; arc=none smtp.client-ip=66.111.4.29 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=sent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sent.com Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 3360F5C00A7; Mon, 26 Feb 2024 15:56:14 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Mon, 26 Feb 2024 15:56:14 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:reply-to:subject:subject:to:to; s=fm1; t= 1708980974; x=1709067374; bh=364J4ng7aCg7Ddk79ejWdMNIz/yxFIeeqLK xisnieh8=; b=iQ1P+dj1L5/s3YJc1AKNrz35Mc/c79OX4NJp+G2orPz4GZ+6LiT Xl80k8xN21LpRMCUqHw/bricLLciGXJIDhrpnfKRnRXASyRj/2JSFN4K2fnQmLx4 FrzLW8FJUFxwuspkQ+PUdlePmPKJMPJKJuOxU18xO+9IBaOUczy0YQ501H9uXZF7 yx+zV3HFZazJhvZBqCxEJGTRstJ62E0PRnmswRs//ZJfMZigtnVa05ZfTxNl99hp F8TOxc0B9R/IzTp9ciw0z9o7/Obsa81fnnA9hPtSqKwfaqlQoIevp1JrNLQSMGyV UresGBjCtv29Vl3UPpQRa8D52ukRMbrzjZg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:reply-to:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1708980974; x=1709067374; bh=364J4ng7aCg7Ddk79ejWdMNIz/yxFIeeqLK xisnieh8=; b=RCNXSmarRVqb3zv0tke5KZ3fUqf35Wlyl/Cana7FlGKAfeYU0Ni dojnphIkoCOr6OVG+Q2G4sxIQUcSOXY83fToHjB7zmqXFd61QY9M76vppmNyHcUA 9iRsIYi4cifrC6kHwcuTkhhUeeBeNkoqREbH4PP5nNsVc/FJnHFzqwxsASyU0Z2D HqAU3Xz/RIx9Bl7KKQpGkXgXg6PAao9Uas+FDGZcTWJf1E/Itn0q00daT+qTAJAu ZoaTWwAZ7CXgZECYTmPAYAFxhLwBEF3hFD3Oj9PeW2CChuj8fy8S6k1KaqtO/+md TCQZPteVnqHYenzMsDE7tB8TNyNqb2EoIsA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrgedvgddugedtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkofgjfhhrgggtgfesthekredtredtjeenucfhrhhomhepkghi ucgjrghnuceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepje ekteekffelleekudfftdefvddtjeejuedtuedtteegjefgvedtfedujeekieevnecuvehl uhhsthgvrhfuihiivgepvdenucfrrghrrghmpehmrghilhhfrhhomhepiihirdihrghnse hsvghnthdrtghomh X-ME-Proxy: Feedback-ID: iccd040f4:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 26 Feb 2024 15:56:13 -0500 (EST) From: Zi Yan To: "Pankaj Raghav (Samsung)" , linux-mm@kvack.org Cc: Zi Yan , "Matthew Wilcox (Oracle)" , David Hildenbrand , Yang Shi , Yu Zhao , "Kirill A . Shutemov" , Ryan Roberts , =?utf-8?q?Michal_Koutn=C3=BD?= , Roman Gushchin , "Zach O'Keefe" , Hugh Dickins , Luis Chamberlain , Andrew Morton , linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kselftest@vger.kernel.org Subject: [PATCH v5 6/8] mm: page_owner: add support for splitting to any order in split page_owner. Date: Mon, 26 Feb 2024 15:55:32 -0500 Message-ID: <20240226205534.1603748-7-zi.yan@sent.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240226205534.1603748-1-zi.yan@sent.com> References: <20240226205534.1603748-1-zi.yan@sent.com> Reply-To: Zi Yan Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791996575373566270 X-GMAIL-MSGID: 1791996575373566270 From: Zi Yan It adds a new_order parameter to set new page order in page owner. It prepares for upcoming changes to support split huge page to any lower order. Signed-off-by: Zi Yan Acked-by: David Hildenbrand --- include/linux/page_owner.h | 13 ++++++++----- mm/huge_memory.c | 2 +- mm/page_alloc.c | 4 ++-- mm/page_owner.c | 7 +++---- 4 files changed, 14 insertions(+), 12 deletions(-) diff --git a/include/linux/page_owner.h b/include/linux/page_owner.h index 2b39c8e19d98..debdc25f08b9 100644 --- a/include/linux/page_owner.h +++ b/include/linux/page_owner.h @@ -11,7 +11,8 @@ extern struct page_ext_operations page_owner_ops; extern void __reset_page_owner(struct page *page, unsigned short order); extern void __set_page_owner(struct page *page, unsigned short order, gfp_t gfp_mask); -extern void __split_page_owner(struct page *page, int order); +extern void __split_page_owner(struct page *page, int old_order, + int new_order); extern void __folio_copy_owner(struct folio *newfolio, struct folio *old); extern void __set_page_owner_migrate_reason(struct page *page, int reason); extern void __dump_page_owner(const struct page *page); @@ -31,10 +32,11 @@ static inline void set_page_owner(struct page *page, __set_page_owner(page, order, gfp_mask); } -static inline void split_page_owner(struct page *page, int order) +static inline void split_page_owner(struct page *page, int old_order, + int new_order) { if (static_branch_unlikely(&page_owner_inited)) - __split_page_owner(page, order); + __split_page_owner(page, old_order, new_order); } static inline void folio_copy_owner(struct folio *newfolio, struct folio *old) { @@ -56,10 +58,11 @@ static inline void reset_page_owner(struct page *page, unsigned short order) { } static inline void set_page_owner(struct page *page, - unsigned int order, gfp_t gfp_mask) + unsigned short order, gfp_t gfp_mask) { } -static inline void split_page_owner(struct page *page, int order) +static inline void split_page_owner(struct page *page, int old_order, + int new_order) { } static inline void folio_copy_owner(struct folio *newfolio, struct folio *folio) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 5d4b7c17b9bc..b2df788c11fa 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2933,7 +2933,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, unlock_page_lruvec(lruvec); /* Caller disabled irqs, so they are still disabled here */ - split_page_owner(head, order); + split_page_owner(head, order, 0); /* See comment in __split_huge_page_tail() */ if (PageAnon(head)) { diff --git a/mm/page_alloc.c b/mm/page_alloc.c index c31a468fe317..cc41341c08f4 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2652,7 +2652,7 @@ void split_page(struct page *page, unsigned int order) for (i = 1; i < (1 << order); i++) set_page_refcounted(page + i); - split_page_owner(page, order); + split_page_owner(page, order, 0); split_page_memcg(page, order, 0); } EXPORT_SYMBOL_GPL(split_page); @@ -4839,7 +4839,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order, struct page *page = virt_to_page((void *)addr); struct page *last = page + nr; - split_page_owner(page, order); + split_page_owner(page, order, 0); split_page_memcg(page, order, 0); while (page < --last) set_page_refcounted(last); diff --git a/mm/page_owner.c b/mm/page_owner.c index b678f7a6e702..033e349f6479 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -306,19 +306,18 @@ void __set_page_owner_migrate_reason(struct page *page, int reason) page_ext_put(page_ext); } -void __split_page_owner(struct page *page, int order) +void __split_page_owner(struct page *page, int old_order, int new_order) { int i; struct page_ext *page_ext = page_ext_get(page); struct page_owner *page_owner; - unsigned int nr = 1 << order; if (unlikely(!page_ext)) return; - for (i = 0; i < nr; i++) { + for (i = 0; i < (1 << old_order); i++) { page_owner = get_page_owner(page_ext); - page_owner->order = 0; + page_owner->order = new_order; page_ext = page_ext_next(page_ext); } page_ext_put(page_ext); From patchwork Mon Feb 26 20:55:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 206919 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2329356dyb; Mon, 26 Feb 2024 12:58:41 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCU45WfZv/8WeVE/GAYwdfhQOCZGh9R1jHdRQi1f7pSAn4YEgLGfdYdMXFzEEbTI2RHHm+XwGn4SjHSrJywl3AU+JSnUpw== X-Google-Smtp-Source: AGHT+IFuLtSG4slRgikLt1e/DRjkPbZu7GgMUtc8bfgiGRnoqR7UjSpsErW2hrfA430fbTKzQizw X-Received: by 2002:a17:903:48b:b0:1dc:728d:b1de with SMTP id jj11-20020a170903048b00b001dc728db1demr6834564plb.56.1708981121126; Mon, 26 Feb 2024 12:58:41 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708981121; cv=pass; d=google.com; s=arc-20160816; b=OXufZxd8BU3G/HZCph45CN9A8nL9WrEaimU45Q28RrBBAZAv1mj8L16yfxqKNdT3ys SP3L12J0FBYOkFCn+gExqcvIHW1VHTIs3ID2cL9m3+dRM/HCOlS7zpD8zLXU7KATR95f Z09UE9oXXmlhqdvV2sZq3fKG6lPRp7EthJ8uHmpwbemQOUnNzjHD03gBgx/GeiE3A4X1 uaP0aFoRHckpJmWgeC9lu/EhFa+etBqp1uTG1eAaZuf9v7CqxkdS7RvBmwhnd12bkUFk Hw8rtSAqKVSn7QnP0q4BoJ5u17lKDRW+gDgb/Fwp0Rlg3g5nHJBpI/QwAJk1ur+GDd1/ BW3w== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:reply-to:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:dkim-signature :dkim-signature; bh=GUAsk26iYl7MhG3zj5r2ZNKBhryTNauvcp1OLHy1WRg=; fh=chAWxDyUzR1fRdzViNE01Bcd9wPKK4ot8WI+7Hca+9w=; b=hcU15A5WR++N4emPuT6AG0rC7jdiQpThnUteHejPfFNpym48b9z3zFvGs8vrpUEcO8 Ev5qCC0AcVjoVjP7ez1thYuXJcUYyoKmuPLk3lfa9IlQUMRJiZVpKwk6T9K78E9+MO8m ULVLvbHdrdETo1h8uNedphO2QBCmOKNBEiZbMYb4etne3HLlAJemUFkpaJN5JIDacG4R QozJjnLHY+M7JmK3NluWcQuPntydzwaD1gSxY13WUidIVn1NTZUr9HI6rxH5px73z1Od b1sjmm6AIPuerWBz7+92/dV0dl9knrfOh1BW9KCfHOl5mSbE76V0tvPyNguR94l050xY w/5g==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@sent.com header.s=fm1 header.b=Gn40UuZA; dkim=pass header.i=@messagingengine.com header.s=fm1 header.b=S45x2fml; arc=pass (i=1 spf=pass spfdomain=sent.com dkim=pass dkdomain=sent.com dkim=pass dkdomain=messagingengine.com dmarc=pass fromdomain=sent.com); spf=pass (google.com: domain of linux-kernel+bounces-82312-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82312-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=sent.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id u3-20020a170902e5c300b001d8a9162f22si186560plf.60.2024.02.26.12.58.41 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 12:58:41 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-82312-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@sent.com header.s=fm1 header.b=Gn40UuZA; dkim=pass header.i=@messagingengine.com header.s=fm1 header.b=S45x2fml; arc=pass (i=1 spf=pass spfdomain=sent.com dkim=pass dkdomain=sent.com dkim=pass dkdomain=messagingengine.com dmarc=pass fromdomain=sent.com); spf=pass (google.com: domain of linux-kernel+bounces-82312-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82312-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=sent.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id E8BAF28AE51 for ; Mon, 26 Feb 2024 20:58:40 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id BF585132466; Mon, 26 Feb 2024 20:56:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sent.com header.i=@sent.com header.b="Gn40UuZA"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="S45x2fml" Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 183D913249D; Mon, 26 Feb 2024 20:56:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=66.111.4.29 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708980978; cv=none; b=GT0nnU7WRpRalx3bLxos2scg7SPjgchUL9ZWbmramT+YjqeHnSipaaY9F3Ar/qnHvhOzkzVTj7UuKwQcIi/ptmohOrOjNZCrri1cXH0HLvKtBF/H8UFglfWHM9Mqkq+9LfUjxanuP7w+pJ992zhB0vdIpq1eufubIsYyPpmSYJQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708980978; c=relaxed/simple; bh=Gj4etBysvHmpgeUpCQUjRlhkDluUaFmpO3rvp/CVp+I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=MW6O9/zYnkEPXnOp4sziYwU1/aWorA7PQyQB1WwL6BMbZcdTDwDNmQhT2/SsixkKdcwqnA9yknH92nktPVdD/H8UXdSKBFw4/keFPrRSMWAzcJK1vBPsYyD7AvJ98wgf9slUFfwyvzai+IA5p/q4y+q2nfpXQoQRvzIbpM5duaw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=sent.com; spf=pass smtp.mailfrom=sent.com; dkim=pass (2048-bit key) header.d=sent.com header.i=@sent.com header.b=Gn40UuZA; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=S45x2fml; arc=none smtp.client-ip=66.111.4.29 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=sent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sent.com Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 604935C00AB; Mon, 26 Feb 2024 15:56:15 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Mon, 26 Feb 2024 15:56:15 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:reply-to:subject:subject:to:to; s=fm1; t= 1708980975; x=1709067375; bh=GUAsk26iYl7MhG3zj5r2ZNKBhryTNauvcp1 OLHy1WRg=; b=Gn40UuZAhb5ftJv5LRfJMIhXiq3QdWSx1my35TaUWUEu5ycpFVl IS1nKksLaE2+5rqFt9FR61yW0wY7XwvHpf+SUo6qhQhShH+A+FcTOS9wAX2fsPau QoKk6zAp8765ulvAvY4C71FS+BcIRJdBJRNnNSQD4waUHzdg8UlfzS/xhYVMLy2p dkB2jOv9CG5Z5h42xNOoWRnOU4RvVZVDRLQzoiAfcAOXfCUnRv2mdsFTxDwlh33U gUD26EzxwbG+UnchZ/ABk3WgslLi3yyVXLaovONMbYb0pPySoO/QTltve1MhUD/X gI4tSKb+k2mm4W8u+DOD7nRY3ZYLSQfV//Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:reply-to:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1708980975; x=1709067375; bh=GUAsk26iYl7MhG3zj5r2ZNKBhryTNauvcp1 OLHy1WRg=; b=S45x2fml7gmhnMQ2+nlEIwKLi14mFfBfranhr0+4tnF/Zhmlgn4 v0LkmfYJ3ECXF2pAUUe9H3JchTGeWe490haeAxfAt3SlvZhZlHfM1A7oTcxhBtep gsNUSoNiP4Rcf//o9VuNy1gCeFikjL4vGqD66vb/XXztiiuSW+gBQMDMJFSmiaas CUM3gCOiUd8fO8BTNnOWhbwjaAl7dGTRqMum3DwmP2exFekMylhIgyMrK/Rsq3HK CvpHvOvZtc/1ICI47XPcP9xuVAj+cXKgYn3fVBO+C40csgJ7gEAsz1vTC5idS+4F reG2E0mSnFLt4C9LVmZTuCWggrywN5smi5Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrgedvgddugedtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkofgjfhhrgggtgfesthekredtredtjeenucfhrhhomhepkghi ucgjrghnuceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepje ekteekffelleekudfftdefvddtjeejuedtuedtteegjefgvedtfedujeekieevnecuvehl uhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepiihirdihrghnse hsvghnthdrtghomh X-ME-Proxy: Feedback-ID: iccd040f4:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 26 Feb 2024 15:56:14 -0500 (EST) From: Zi Yan To: "Pankaj Raghav (Samsung)" , linux-mm@kvack.org Cc: Zi Yan , "Matthew Wilcox (Oracle)" , David Hildenbrand , Yang Shi , Yu Zhao , "Kirill A . Shutemov" , Ryan Roberts , =?utf-8?q?Michal_Koutn=C3=BD?= , Roman Gushchin , "Zach O'Keefe" , Hugh Dickins , Luis Chamberlain , Andrew Morton , linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kselftest@vger.kernel.org Subject: [PATCH v5 7/8] mm: thp: split huge page to any lower order pages Date: Mon, 26 Feb 2024 15:55:33 -0500 Message-ID: <20240226205534.1603748-8-zi.yan@sent.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240226205534.1603748-1-zi.yan@sent.com> References: <20240226205534.1603748-1-zi.yan@sent.com> Reply-To: Zi Yan Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791996588288208939 X-GMAIL-MSGID: 1791996588288208939 From: Zi Yan To split a THP to any lower order pages, we need to reform THPs on subpages at given order and add page refcount based on the new page order. Also we need to reinitialize page_deferred_list after removing the page from the split_queue, otherwise a subsequent split will see list corruption when checking the page_deferred_list again. Note: Anonymous order-1 folio is not supported because _deferred_list, which is used by partially mapped folios, is stored in subpage 2 and an order-1 folio only has subpage 0 and 1. File-backed order-1 folios are fine, since they do not use _deferred_list. Signed-off-by: Zi Yan --- include/linux/huge_mm.h | 21 +++++--- mm/huge_memory.c | 110 +++++++++++++++++++++++++++++++--------- 2 files changed, 99 insertions(+), 32 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 5adb86af35fc..de0c89105076 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -265,10 +265,11 @@ unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr, void folio_prep_large_rmappable(struct folio *folio); bool can_split_folio(struct folio *folio, int *pextra_pins); -int split_huge_page_to_list(struct page *page, struct list_head *list); +int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, + unsigned int new_order); static inline int split_huge_page(struct page *page) { - return split_huge_page_to_list(page, NULL); + return split_huge_page_to_list_to_order(page, NULL, 0); } void deferred_split_folio(struct folio *folio); @@ -422,7 +423,8 @@ can_split_folio(struct folio *folio, int *pextra_pins) return false; } static inline int -split_huge_page_to_list(struct page *page, struct list_head *list) +split_huge_page_to_list_to_order(struct page *page, struct list_head *list, + unsigned int new_order) { return 0; } @@ -519,17 +521,20 @@ static inline bool thp_migration_supported(void) } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -static inline int split_folio_to_list(struct folio *folio, - struct list_head *list) +static inline int split_folio_to_list_to_order(struct folio *folio, + struct list_head *list, int new_order) { - return split_huge_page_to_list(&folio->page, list); + return split_huge_page_to_list_to_order(&folio->page, list, new_order); } -static inline int split_folio(struct folio *folio) +static inline int split_folio_to_order(struct folio *folio, int new_order) { - return split_folio_to_list(folio, NULL); + return split_folio_to_list_to_order(folio, NULL, new_order); } +#define split_folio_to_list(f, l) split_folio_to_list_to_order(f, l, 0) +#define split_folio(f) split_folio_to_order(f, 0) + /* * archs that select ARCH_WANTS_THP_SWAP but don't support THP_SWP due to * limitations in the implementation like arm64 MTE can override this to diff --git a/mm/huge_memory.c b/mm/huge_memory.c index b2df788c11fa..8b47a96a28f9 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2770,7 +2770,6 @@ static void lru_add_page_tail(struct page *head, struct page *tail, struct lruvec *lruvec, struct list_head *list) { VM_BUG_ON_PAGE(!PageHead(head), head); - VM_BUG_ON_PAGE(PageCompound(tail), head); VM_BUG_ON_PAGE(PageLRU(tail), head); lockdep_assert_held(&lruvec->lru_lock); @@ -2791,7 +2790,8 @@ static void lru_add_page_tail(struct page *head, struct page *tail, } static void __split_huge_page_tail(struct folio *folio, int tail, - struct lruvec *lruvec, struct list_head *list) + struct lruvec *lruvec, struct list_head *list, + unsigned int new_order) { struct page *head = &folio->page; struct page *page_tail = head + tail; @@ -2861,10 +2861,15 @@ static void __split_huge_page_tail(struct folio *folio, int tail, * which needs correct compound_head(). */ clear_compound_head(page_tail); + if (new_order) { + prep_compound_page(page_tail, new_order); + folio_prep_large_rmappable(new_folio); + } /* Finally unfreeze refcount. Additional reference from page cache. */ - page_ref_unfreeze(page_tail, 1 + (!folio_test_anon(folio) || - folio_test_swapcache(folio))); + page_ref_unfreeze(page_tail, + 1 + ((!folio_test_anon(folio) || folio_test_swapcache(folio)) ? + folio_nr_pages(new_folio) : 0)); if (folio_test_young(folio)) folio_set_young(new_folio); @@ -2882,7 +2887,7 @@ static void __split_huge_page_tail(struct folio *folio, int tail, } static void __split_huge_page(struct page *page, struct list_head *list, - pgoff_t end) + pgoff_t end, unsigned int new_order) { struct folio *folio = page_folio(page); struct page *head = &folio->page; @@ -2890,11 +2895,12 @@ static void __split_huge_page(struct page *page, struct list_head *list, struct address_space *swap_cache = NULL; unsigned long offset = 0; int i, nr_dropped = 0; + unsigned int new_nr = 1 << new_order; int order = folio_order(folio); unsigned int nr = 1 << order; /* complete memcg works before add pages to LRU */ - split_page_memcg(head, order, 0); + split_page_memcg(head, order, new_order); if (folio_test_anon(folio) && folio_test_swapcache(folio)) { offset = swp_offset(folio->swap); @@ -2907,8 +2913,8 @@ static void __split_huge_page(struct page *page, struct list_head *list, ClearPageHasHWPoisoned(head); - for (i = nr - 1; i >= 1; i--) { - __split_huge_page_tail(folio, i, lruvec, list); + for (i = nr - new_nr; i >= new_nr; i -= new_nr) { + __split_huge_page_tail(folio, i, lruvec, list, new_order); /* Some pages can be beyond EOF: drop them from page cache */ if (head[i].index >= end) { struct folio *tail = page_folio(head + i); @@ -2929,24 +2935,30 @@ static void __split_huge_page(struct page *page, struct list_head *list, } } - ClearPageCompound(head); + if (!new_order) + ClearPageCompound(head); + else { + struct folio *new_folio = (struct folio *)head; + + folio_set_order(new_folio, new_order); + } unlock_page_lruvec(lruvec); /* Caller disabled irqs, so they are still disabled here */ - split_page_owner(head, order, 0); + split_page_owner(head, order, new_order); /* See comment in __split_huge_page_tail() */ if (PageAnon(head)) { /* Additional pin to swap cache */ if (PageSwapCache(head)) { - page_ref_add(head, 2); + page_ref_add(head, 1 + new_nr); xa_unlock(&swap_cache->i_pages); } else { page_ref_inc(head); } } else { /* Additional pin to page cache */ - page_ref_add(head, 2); + page_ref_add(head, 1 + new_nr); xa_unlock(&head->mapping->i_pages); } local_irq_enable(); @@ -2958,7 +2970,15 @@ static void __split_huge_page(struct page *page, struct list_head *list, if (folio_test_swapcache(folio)) split_swap_cluster(folio->swap); - for (i = 0; i < nr; i++) { + /* + * set page to its compound_head when split to non order-0 pages, so + * we can skip unlocking it below, since PG_locked is transferred to + * the compound_head of the page and the caller will unlock it. + */ + if (new_order) + page = compound_head(page); + + for (i = 0; i < nr; i += new_nr) { struct page *subpage = head + i; if (subpage == page) continue; @@ -2992,29 +3012,36 @@ bool can_split_folio(struct folio *folio, int *pextra_pins) } /* - * This function splits huge page into normal pages. @page can point to any - * subpage of huge page to split. Split doesn't change the position of @page. + * This function splits huge page into pages in @new_order. @page can point to + * any subpage of huge page to split. Split doesn't change the position of + * @page. + * + * NOTE: order-1 anonymous folio is not supported because _deferred_list, + * which is used by partially mapped folios, is stored in subpage 2 and an + * order-1 folio only has subpage 0 and 1. File-backed order-1 folios are OK, + * since they do not use _deferred_list. * * Only caller must hold pin on the @page, otherwise split fails with -EBUSY. * The huge page must be locked. * * If @list is null, tail pages will be added to LRU list, otherwise, to @list. * - * Both head page and tail pages will inherit mapping, flags, and so on from - * the hugepage. + * Pages in new_order will inherit mapping, flags, and so on from the hugepage. * - * GUP pin and PG_locked transferred to @page. Rest subpages can be freed if - * they are not mapped. + * GUP pin and PG_locked transferred to @page or the compound page @page belongs + * to. Rest subpages can be freed if they are not mapped. * * Returns 0 if the hugepage is split successfully. * Returns -EBUSY if the page is pinned or if anon_vma disappeared from under * us. */ -int split_huge_page_to_list(struct page *page, struct list_head *list) +int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, + unsigned int new_order) { struct folio *folio = page_folio(page); struct deferred_split *ds_queue = get_deferred_split_queue(folio); - XA_STATE(xas, &folio->mapping->i_pages, folio->index); + /* reset xarray order to new order after split */ + XA_STATE_ORDER(xas, &folio->mapping->i_pages, folio->index, new_order); struct anon_vma *anon_vma = NULL; struct address_space *mapping = NULL; int extra_pins, ret; @@ -3024,6 +3051,34 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); VM_BUG_ON_FOLIO(!folio_test_large(folio), folio); + /* Cannot split anonymous THP to order-1 */ + if (new_order == 1 && folio_test_anon(folio)) { + VM_WARN_ONCE(1, "Cannot split to order-1 folio"); + return -EINVAL; + } + + if (new_order) { + /* Only swapping a whole PMD-mapped folio is supported */ + if (folio_test_swapcache(folio)) { + VM_WARN_ONCE(1, + "Cannot split swap-cached folio to non-0 order"); + return -EINVAL; + } + /* Split shmem folio to non-zero order not supported */ + if (shmem_mapping(folio->mapping)) { + VM_WARN_ONCE(1, + "Cannot split shmem folio to non-0 order"); + return -EINVAL; + } + /* No split if the file system does not support large folio */ + if (!mapping_large_folio_support(folio->mapping)) { + VM_WARN_ONCE(1, + "Cannot split file folio to non-0 order"); + return -EINVAL; + } + } + + is_hzp = is_huge_zero_page(&folio->page); if (is_hzp) { pr_warn_ratelimited("Called split_huge_page for huge zero page\n"); @@ -3120,14 +3175,21 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) if (folio_order(folio) > 1 && !list_empty(&folio->_deferred_list)) { ds_queue->split_queue_len--; - list_del(&folio->_deferred_list); + /* + * Reinitialize page_deferred_list after removing the + * page from the split_queue, otherwise a subsequent + * split will see list corruption when checking the + * page_deferred_list. + */ + list_del_init(&folio->_deferred_list); } spin_unlock(&ds_queue->split_queue_lock); if (mapping) { int nr = folio_nr_pages(folio); xas_split(&xas, folio, folio_order(folio)); - if (folio_test_pmd_mappable(folio)) { + if (folio_test_pmd_mappable(folio) && + new_order < HPAGE_PMD_ORDER) { if (folio_test_swapbacked(folio)) { __lruvec_stat_mod_folio(folio, NR_SHMEM_THPS, -nr); @@ -3139,7 +3201,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) } } - __split_huge_page(page, list, end); + __split_huge_page(page, list, end, new_order); ret = 0; } else { spin_unlock(&ds_queue->split_queue_lock); From patchwork Mon Feb 26 20:55:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 206920 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2329364dyb; Mon, 26 Feb 2024 12:58:42 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCUAWg6adtBtQN/NJJRrh6C/7CsT87SxC9YQdm0csCJbE4Ss0uIYTHJ5dP5qEKdnjorAGe4fQlOg9ev8Doj5u+yIJbddyA== X-Google-Smtp-Source: AGHT+IGf0KPqiRrn0lMox/JFgYAE+tiDOoaakfPnsLSyxh7hcs3y1VMwQJf6+Xz/428LLdAKVcp3 X-Received: by 2002:a05:6359:4582:b0:178:9393:176c with SMTP id no2-20020a056359458200b001789393176cmr8941567rwb.15.1708981122075; Mon, 26 Feb 2024 12:58:42 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708981122; cv=pass; d=google.com; s=arc-20160816; b=xdbQWkU9xXv/iGJ2+mQwuoG/fO1tsNWz88lufcvKAoIF3ytbgj6LPk88HW+N0SKVIS b453W+W7BxbgILtyZfOHbhzeUVfyVHLgQCBZa2X70flaxTW3Gvg6CZYp3kAuOam61zkr opnEHt3P56KeMMxSoqFHsnUp7NirrR32pG5FTZwHS+ORHDi1MyqjZymM5ddJp8rC/8sn jGZ3yd7xaYCPQw9IhvacNxdtx6kR+1JOHVz0IpOZ298smLA7EQ53HdAEQP3qpdA51lXz Yf1BKFqKsP2jMMmhmJOZdcMWKzK5ioahhTLqMWIkIKW5wC7qWNiLXGI4uAhfmIDqVt1p Q5Pw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:reply-to:references:in-reply-to :message-id:date:subject:cc:to:from:feedback-id:dkim-signature :dkim-signature; bh=eBpC8nLmV3t+tILOOyF9MHkmmffIcEidaEQ0IucvPX4=; fh=chAWxDyUzR1fRdzViNE01Bcd9wPKK4ot8WI+7Hca+9w=; b=t9cF3sF7wsVp3WZwxs1hV0Us5DaqcWxgz2P6rlGteP1Bc2vZbswhwhPuRYWzducgub r1bsbdVVD4F/BdUI8zF5z4L+l7VeLGCaqYFT/xvT801yB4nLe4vQ0l4ZVklpOY4Nxtrt WhBbauF81PRT1CB4KdbkJ3Fgd1ypqZP4Hqe/Se2unSO7mcYkKQ45o0UB/vN/RoIkzswY CtoR8gZaunxU8S7YdycfeJKe/+rbK5x6WljCGTjQkI+62xPYyacfeYVR9k71s+V3hqvp 2w2M4BAvgJ91OVkCeng/yGDN42IRru3uRL+w7DvYNkNPMmd2ViTF6f3X1lUZfWytEN+m scvQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@sent.com header.s=fm1 header.b=OhmXJsbj; dkim=pass header.i=@messagingengine.com header.s=fm1 header.b=HkcqTlDi; arc=pass (i=1 spf=pass spfdomain=sent.com dkim=pass dkdomain=sent.com dkim=pass dkdomain=messagingengine.com dmarc=pass fromdomain=sent.com); spf=pass (google.com: domain of linux-kernel+bounces-82313-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82313-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=sent.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id k64-20020a638443000000b005dc528d603bsi4213608pgd.675.2024.02.26.12.58.41 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 12:58:42 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-82313-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@sent.com header.s=fm1 header.b=OhmXJsbj; dkim=pass header.i=@messagingengine.com header.s=fm1 header.b=HkcqTlDi; arc=pass (i=1 spf=pass spfdomain=sent.com dkim=pass dkdomain=sent.com dkim=pass dkdomain=messagingengine.com dmarc=pass fromdomain=sent.com); spf=pass (google.com: domain of linux-kernel+bounces-82313-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82313-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=sent.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id CDB0F28B220 for ; Mon, 26 Feb 2024 20:58:41 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 1868A134CD0; Mon, 26 Feb 2024 20:56:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sent.com header.i=@sent.com header.b="OhmXJsbj"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="HkcqTlDi" Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 23C1D133282; Mon, 26 Feb 2024 20:56:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=66.111.4.29 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708980979; cv=none; b=M/1sCxn5BpDIo2dm6ltmG+K1AUN+ccOVBcjkNXWFmsXzRH1ZgVnSB44/9YaCIXhmc2wpD/bhyVE5ipu391vv4IYM31MIql+MCJa+2mVgbMRwojbu4kknOjuvOCMtMdON/cEEaVry6uTm5YlTPyFHGTnYOsxdl8+5ylxGvdn9VrY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708980979; c=relaxed/simple; bh=vgsm4k7rgVedH8rRB2J8JRli9rUrF9iO5L/lbj+auXA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=q9HNsnAa1n8lVGFt+jnf0rNWvSzwboN8hz8Oguo3ZMAi9WyBBNYkw+OPuh5IsUCAotje4b+6BVMgiVhoEKSQRdkJMloLvVrsXSq7LE6eDXxgU2RWyr6Mu5EUAU/dNafUz8eXb+JPWX0xOHTgt0s84KKvb7+M/elt35oRDTccsfs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=sent.com; spf=pass smtp.mailfrom=sent.com; dkim=pass (2048-bit key) header.d=sent.com header.i=@sent.com header.b=OhmXJsbj; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=HkcqTlDi; arc=none smtp.client-ip=66.111.4.29 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=sent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sent.com Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 3F4E95C00A8; Mon, 26 Feb 2024 15:56:16 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Mon, 26 Feb 2024 15:56:16 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:reply-to:subject:subject:to:to; s=fm1; t= 1708980976; x=1709067376; bh=eBpC8nLmV3t+tILOOyF9MHkmmffIcEidaEQ 0IucvPX4=; b=OhmXJsbjvWBU33DO8vx9Sxyp+wXJmzSpaj6+LDBZDG49O+6ffyx ZSUltZ/7nhkEFqVsCc3a2BUJ4ZoVi67suV+jcrt7L2a41KVnXgGz7u5C2OQ4vK7Q qe5mbw7Dyw200uoLqf7Y25BxkdETjX2nq02CXm4lZAplslb24g51eTMqbeA1CC0j JE5a5N7H2DfGnr/RAYJJKK13TFEmJ+gq91rNp8+BSLK9z/0fwejQT7dLIWMMKkFA 9cIxCDfrLy93emdUnHrFQdAslKi55y1LFDxW3rqJjzeiJY6ZHYZX2EqCL80oxiim 9LB/ULPPKQBWQm0HacSUH3o3KmmnjeUpoiQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:reply-to:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1708980976; x=1709067376; bh=eBpC8nLmV3t+tILOOyF9MHkmmffIcEidaEQ 0IucvPX4=; b=HkcqTlDiLWonV5rP8T2a91tyNh3gzcRKAFCXLT+izkUiGmAM2tV EqGPzyq9qIzXV+qS6IOEglz7RnR9WR0gQ+/KQDzJjaLsW2j4MS2zASpXMwSEtkeE /8IPpT7yvoXVSdApLjEfvcMxPf2UTtOX04hE40u5hQntCFfKyfCsyZmmoUrsP2gC 1RSn1NOuSDsx9lVkwKOeusqYV2xuixW1d+E8Q1YSZQULNpCZYWHCxKKwFicO2vyo FllPcUGrYbTwUOhkLuvDr0qN88nRM6skelfHLgdoJ30SY7ohg5V9/inmL0zv/t1S 4yDU6cQaB100xO5B1Ub2UmNrWOYZKr6DWPA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrgedvgddugedtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkofgjfhhrgggtgfesthekredtredtjeenucfhrhhomhepkghi ucgjrghnuceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepje ekteekffelleekudfftdefvddtjeejuedtuedtteegjefgvedtfedujeekieevnecuvehl uhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepiihirdihrghnse hsvghnthdrtghomh X-ME-Proxy: Feedback-ID: iccd040f4:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 26 Feb 2024 15:56:15 -0500 (EST) From: Zi Yan To: "Pankaj Raghav (Samsung)" , linux-mm@kvack.org Cc: Zi Yan , "Matthew Wilcox (Oracle)" , David Hildenbrand , Yang Shi , Yu Zhao , "Kirill A . Shutemov" , Ryan Roberts , =?utf-8?q?Michal_Koutn=C3=BD?= , Roman Gushchin , "Zach O'Keefe" , Hugh Dickins , Luis Chamberlain , Andrew Morton , linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kselftest@vger.kernel.org Subject: [PATCH v5 8/8] mm: huge_memory: enable debugfs to split huge pages to any order. Date: Mon, 26 Feb 2024 15:55:34 -0500 Message-ID: <20240226205534.1603748-9-zi.yan@sent.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240226205534.1603748-1-zi.yan@sent.com> References: <20240226205534.1603748-1-zi.yan@sent.com> Reply-To: Zi Yan Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791996588937269270 X-GMAIL-MSGID: 1791996588937269270 From: Zi Yan It is used to test split_huge_page_to_list_to_order for pagecache THPs. Also add test cases for split_huge_page_to_list_to_order via both debugfs. Signed-off-by: Zi Yan --- mm/huge_memory.c | 34 ++++-- .../selftests/mm/split_huge_page_test.c | 115 +++++++++++++++++- 2 files changed, 131 insertions(+), 18 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 8b47a96a28f9..50d146eb248f 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3422,7 +3422,7 @@ static inline bool vma_not_suitable_for_thp_split(struct vm_area_struct *vma) } static int split_huge_pages_pid(int pid, unsigned long vaddr_start, - unsigned long vaddr_end) + unsigned long vaddr_end, unsigned int new_order) { int ret = 0; struct task_struct *task; @@ -3486,13 +3486,19 @@ static int split_huge_pages_pid(int pid, unsigned long vaddr_start, goto next; total++; - if (!can_split_folio(folio, NULL)) + /* + * For folios with private, split_huge_page_to_list_to_order() + * will try to drop it before split and then check if the folio + * can be split or not. So skip the check here. + */ + if (!folio_test_private(folio) && + !can_split_folio(folio, NULL)) goto next; if (!folio_trylock(folio)) goto next; - if (!split_folio(folio)) + if (!split_folio_to_order(folio, new_order)) split++; folio_unlock(folio); @@ -3510,7 +3516,7 @@ static int split_huge_pages_pid(int pid, unsigned long vaddr_start, } static int split_huge_pages_in_file(const char *file_path, pgoff_t off_start, - pgoff_t off_end) + pgoff_t off_end, unsigned int new_order) { struct filename *file; struct file *candidate; @@ -3549,7 +3555,7 @@ static int split_huge_pages_in_file(const char *file_path, pgoff_t off_start, if (!folio_trylock(folio)) goto next; - if (!split_folio(folio)) + if (!split_folio_to_order(folio, new_order)) split++; folio_unlock(folio); @@ -3574,10 +3580,14 @@ static ssize_t split_huge_pages_write(struct file *file, const char __user *buf, { static DEFINE_MUTEX(split_debug_mutex); ssize_t ret; - /* hold pid, start_vaddr, end_vaddr or file_path, off_start, off_end */ + /* + * hold pid, start_vaddr, end_vaddr, new_order or + * file_path, off_start, off_end, new_order + */ char input_buf[MAX_INPUT_BUF_SZ]; int pid; unsigned long vaddr_start, vaddr_end; + unsigned int new_order = 0; ret = mutex_lock_interruptible(&split_debug_mutex); if (ret) @@ -3606,29 +3616,29 @@ static ssize_t split_huge_pages_write(struct file *file, const char __user *buf, goto out; } - ret = sscanf(buf, "0x%lx,0x%lx", &off_start, &off_end); - if (ret != 2) { + ret = sscanf(buf, "0x%lx,0x%lx,%d", &off_start, &off_end, &new_order); + if (ret != 2 && ret != 3) { ret = -EINVAL; goto out; } - ret = split_huge_pages_in_file(file_path, off_start, off_end); + ret = split_huge_pages_in_file(file_path, off_start, off_end, new_order); if (!ret) ret = input_len; goto out; } - ret = sscanf(input_buf, "%d,0x%lx,0x%lx", &pid, &vaddr_start, &vaddr_end); + ret = sscanf(input_buf, "%d,0x%lx,0x%lx,%d", &pid, &vaddr_start, &vaddr_end, &new_order); if (ret == 1 && pid == 1) { split_huge_pages_all(); ret = strlen(input_buf); goto out; - } else if (ret != 3) { + } else if (ret != 3 && ret != 4) { ret = -EINVAL; goto out; } - ret = split_huge_pages_pid(pid, vaddr_start, vaddr_end); + ret = split_huge_pages_pid(pid, vaddr_start, vaddr_end, new_order); if (!ret) ret = strlen(input_buf); out: diff --git a/tools/testing/selftests/mm/split_huge_page_test.c b/tools/testing/selftests/mm/split_huge_page_test.c index 7b698a848bab..cf09fdc9ef22 100644 --- a/tools/testing/selftests/mm/split_huge_page_test.c +++ b/tools/testing/selftests/mm/split_huge_page_test.c @@ -16,6 +16,7 @@ #include #include #include +#include #include "vm_util.h" #include "../kselftest.h" @@ -24,10 +25,12 @@ unsigned int pageshift; uint64_t pmd_pagesize; #define SPLIT_DEBUGFS "/sys/kernel/debug/split_huge_pages" +#define SMAP_PATH "/proc/self/smaps" +#define THP_FS_PATH "/mnt/thp_fs" #define INPUT_MAX 80 -#define PID_FMT "%d,0x%lx,0x%lx" -#define PATH_FMT "%s,0x%lx,0x%lx" +#define PID_FMT "%d,0x%lx,0x%lx,%d" +#define PATH_FMT "%s,0x%lx,0x%lx,%d" #define PFN_MASK ((1UL<<55)-1) #define KPF_THP (1UL<<22) @@ -102,7 +105,7 @@ void split_pmd_thp(void) /* split all THPs */ write_debugfs(PID_FMT, getpid(), (uint64_t)one_page, - (uint64_t)one_page + len); + (uint64_t)one_page + len, 0); for (i = 0; i < len; i++) if (one_page[i] != (char)i) @@ -177,7 +180,7 @@ void split_pte_mapped_thp(void) /* split all remapped THPs */ write_debugfs(PID_FMT, getpid(), (uint64_t)pte_mapped, - (uint64_t)pte_mapped + pagesize * 4); + (uint64_t)pte_mapped + pagesize * 4, 0); /* smap does not show THPs after mremap, use kpageflags instead */ thp_size = 0; @@ -237,7 +240,7 @@ void split_file_backed_thp(void) } /* split the file-backed THP */ - write_debugfs(PATH_FMT, testfile, pgoff_start, pgoff_end); + write_debugfs(PATH_FMT, testfile, pgoff_start, pgoff_end, 0); status = unlink(testfile); if (status) { @@ -265,8 +268,101 @@ void split_file_backed_thp(void) ksft_exit_fail_msg("Error occurred\n"); } +void create_pagecache_thp_and_fd(const char *testfile, size_t fd_size, int *fd, char **addr) +{ + size_t i; + int dummy; + + srand(time(NULL)); + + *fd = open(testfile, O_CREAT | O_RDWR, 0664); + if (*fd == -1) + ksft_exit_fail_msg("Failed to create a file at "THP_FS_PATH); + + for (i = 0; i < fd_size; i++) { + unsigned char byte = (unsigned char)i; + + write(*fd, &byte, sizeof(byte)); + } + close(*fd); + sync(); + *fd = open("/proc/sys/vm/drop_caches", O_WRONLY); + if (*fd == -1) { + ksft_perror("open drop_caches"); + goto err_out_unlink; + } + if (write(*fd, "3", 1) != 1) { + ksft_perror("write to drop_caches"); + goto err_out_unlink; + } + close(*fd); + + *fd = open(testfile, O_RDWR); + if (*fd == -1) { + ksft_perror("Failed to open a file at "THP_FS_PATH); + goto err_out_unlink; + } + + *addr = mmap(NULL, fd_size, PROT_READ|PROT_WRITE, MAP_SHARED, *fd, 0); + if (*addr == (char *)-1) { + ksft_perror("cannot mmap"); + goto err_out_close; + } + madvise(*addr, fd_size, MADV_HUGEPAGE); + + for (size_t i = 0; i < fd_size; i++) + dummy += *(*addr + i); + + if (!check_huge_file(*addr, fd_size / pmd_pagesize, pmd_pagesize)) { + ksft_print_msg("No large pagecache folio generated, please mount a filesystem supporting large folio at "THP_FS_PATH"\n"); + goto err_out_close; + } + return; +err_out_close: + close(*fd); +err_out_unlink: + unlink(testfile); + ksft_exit_fail_msg("Failed to create large pagecache folios\n"); +} + +void split_thp_in_pagecache_to_order(size_t fd_size, int order) +{ + int fd; + char *addr; + size_t i; + const char testfile[] = THP_FS_PATH "/test"; + int err = 0; + + create_pagecache_thp_and_fd(testfile, fd_size, &fd, &addr); + + write_debugfs(PID_FMT, getpid(), (uint64_t)addr, (uint64_t)addr + fd_size, order); + + for (i = 0; i < fd_size; i++) + if (*(addr + i) != (char)i) { + ksft_print_msg("%lu byte corrupted in the file\n", i); + err = EXIT_FAILURE; + goto out; + } + + if (!check_huge_file(addr, 0, pmd_pagesize)) { + ksft_print_msg("Still FilePmdMapped not split\n"); + err = EXIT_FAILURE; + goto out; + } + +out: + close(fd); + unlink(testfile); + if (err) + ksft_exit_fail_msg("Split PMD-mapped pagecache folio to order %d failed\n", order); + ksft_test_result_pass("Split PMD-mapped pagecache folio to order %d passed\n", order); +} + int main(int argc, char **argv) { + int i; + size_t fd_size; + ksft_print_header(); if (geteuid() != 0) { @@ -274,7 +370,7 @@ int main(int argc, char **argv) ksft_finished(); } - ksft_set_plan(3); + ksft_set_plan(3+9); pagesize = getpagesize(); pageshift = ffs(pagesize) - 1; @@ -282,9 +378,16 @@ int main(int argc, char **argv) if (!pmd_pagesize) ksft_exit_fail_msg("Reading PMD pagesize failed\n"); + fd_size = 2 * pmd_pagesize; + split_pmd_thp(); split_pte_mapped_thp(); split_file_backed_thp(); + for (i = 8; i >= 0; i--) + split_thp_in_pagecache_to_order(fd_size, i); + ksft_finished(); + + return 0; }