From patchwork Wed Sep 13 10:53:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 138787 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9ecd:0:b0:3f2:4152:657d with SMTP id t13csp1064048vqx; Wed, 13 Sep 2023 05:57:07 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEK/DdY/sh13qJc570zRhtlLk29Q1sFxdN6HQCp8aW/g5+a3Qka35IZGvtxZOO3I8r9GPSW X-Received: by 2002:a17:903:1207:b0:1c3:b3c7:d67f with SMTP id l7-20020a170903120700b001c3b3c7d67fmr2573355plh.47.1694609827220; Wed, 13 Sep 2023 05:57:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694609827; cv=none; d=google.com; s=arc-20160816; b=gQx4c9QphcgxHuFyk5VVaO/6TXgsl62sadSCgXbx1qnXNY4Jq/836OYihYeYCQRAbl NhKZzy8eLtecMJvrZIPWe3/MIS2G4BO4ChpkRBSf0jsxtkfFQzBzaQGZm6ThSPljF8Sn d7GJrVDBcJ49UENWpbcBIuLaG9gML905ZvgnISAl3CqN/2h99/wAv39JB4g/yaXTHZBf icn2hSBreChSOD7/FcqJrfPmMqvSEN6M2lfQAzCT3eYPo4mo7j9hL5xgTgv/qhETLUs4 YqBB44vx+xriHyAO7Mi7vupwFiubJdvtXkgZfL1aG2FPUKQy8Sd5TYjTR2XmX+jcBcP/ Rbhg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=av0vrlLKsg9xBQaLOcb8CPCy93VQFu46a3TbtxpfU4U=; fh=pwE1+DxwRHQWbcOZzR2ABNMMQVcionEFVt3U0a73p7k=; b=NSIK02gtFMiwEnSHtJqI2IaEe2Im+6xP3zRjlyFz25qATy8dMkGGwD9oHJIdFJjqu8 OCcBlD5NPQ/b7fmMo4mJNbJKqHHRuX4GHzL2IOiLd6Mvi9Q0SoOFyNkfKYZnOOAInJ5q 0+2tvUHdzVnFdb7g/gV7ABXg7S2ViuJP9kKZla/N1TFkvIPJOrsbYJtn0T4Q8PgYWRE6 NVP4b1VWkKbp+T46EUvUjyMnBBWemr6i/njQTzNhzPgnBmARaDxXwow9RsShgG+21p6g RXdx3YikTY1BrYk3LwdvbwEngQQOVECRl42CHT5PyIcg6WtX+q6VS4I06ZJWt4VKlhNv Ga5w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=P5GWihmM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id k9-20020a170902c40900b001bbd0450af8si10815903plk.187.2023.09.13.05.57.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 05:57:07 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=P5GWihmM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 72AEB81D0C5D; Wed, 13 Sep 2023 03:54:38 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239802AbjIMKyi (ORCPT + 36 others); Wed, 13 Sep 2023 06:54:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60198 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239766AbjIMKye (ORCPT ); Wed, 13 Sep 2023 06:54:34 -0400 Received: from mail-lj1-x230.google.com (mail-lj1-x230.google.com [IPv6:2a00:1450:4864:20::230]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 463AC1726 for ; Wed, 13 Sep 2023 03:54:08 -0700 (PDT) Received: by mail-lj1-x230.google.com with SMTP id 38308e7fff4ca-2bfb17435e4so15902811fa.0 for ; Wed, 13 Sep 2023 03:54:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1694602446; x=1695207246; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=av0vrlLKsg9xBQaLOcb8CPCy93VQFu46a3TbtxpfU4U=; b=P5GWihmMHmwcNkj9lfDILOmiDbCAze1/o2zhKzbWGEmPSTV6Vveaa4b96w64JfAGgH UXr2WFXck9FLMeG1WT6GZbzf1dY0VJ7tee9TjuJSGvzLjKfdSc7J7pv8T5I4d1SxIC8P JByflZJXcSNcu7WmFW2H9ub1Ppgi0oYGo0feH64uYnCcKfTojMGVee2IuWkhYXFMW70q TRqXzB/7caGX1dcpqMkotb0p1ELUEMNCICleLJsP3QpsshS90o22q0w5SDW+RF3jj4hr 9zQvnQaiBYEcw1vVXCq+x8xmtHHlgX/YgajPjDRBQsyonoW4ykWYYaMXEN/gNQV3y0pe FW9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694602446; x=1695207246; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=av0vrlLKsg9xBQaLOcb8CPCy93VQFu46a3TbtxpfU4U=; b=bUjMjqZHFrRNWI7FouGKUW7ZBBDYeX7cVgtXgdDJcDz3/KH8ikWxiNuBrNunmNR018 WmkG+GfdWRWqWSglr6e5XNXD/ytGLBuzeWFDzC93kXQb30+M8xUwyQKETCgmdbliVlyX p/oDjtcNdBJWLA28liaqWyEryB0Q/aPtA4HL9CZy5m2pu1aEBFag5AI/da64Pu8ZFUMi z3WeWNJmHsNWuJ0FIeg83/Y97Z8OW3xW/E6YbJb8aqgPamUBsv7vRQkMPaeCDAfpLnjo ieKporHgBGSkKTkYqp6RYpiXgafxnHh057NWQJERa4F2SWwVWI5uCB0csotv6MJUZYCZ PHqQ== X-Gm-Message-State: AOJu0Ywic08p1kjGAsL6vieFVDK5vh95s8iuQbzbDitcYME455NSr8Mt jIR9LOShvvFb0X5CD1SGlU0ptw== X-Received: by 2002:a05:651c:8f:b0:2bf:9576:afd4 with SMTP id 15-20020a05651c008f00b002bf9576afd4mr1875974ljq.16.1694602446317; Wed, 13 Sep 2023 03:54:06 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b5c7:0:59d5:28b9:7f13:cd1b]) by smtp.gmail.com with ESMTPSA id p36-20020a05600c1da400b003ff013a4fd9sm1757055wms.7.2023.09.13.03.54.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 03:54:05 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, songmuchun@bytedance.com, fam.zheng@bytedance.com, liangma@liangbit.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v5 1/4] mm: hugetlb_vmemmap: Use nid of the head page to reallocate it Date: Wed, 13 Sep 2023 11:53:58 +0100 Message-Id: <20230913105401.519709-2-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230913105401.519709-1-usama.arif@bytedance.com> References: <20230913105401.519709-1-usama.arif@bytedance.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 13 Sep 2023 03:54:38 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776927194131319393 X-GMAIL-MSGID: 1776927194131319393 If tail page prep and initialization is skipped, then the "start" page will not contain the correct nid. Use the nid from first vmemap page. Signed-off-by: Usama Arif Reviewed-by: Muchun Song Reviewed-by: Mike Kravetz --- mm/hugetlb_vmemmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index aeb7dd889eee..3cdb38d87a95 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -319,7 +319,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, .reuse_addr = reuse, .vmemmap_pages = &vmemmap_pages, }; - int nid = page_to_nid((struct page *)start); + int nid = page_to_nid((struct page *)reuse); gfp_t gfp_mask = GFP_KERNEL | __GFP_THISNODE | __GFP_NORETRY | __GFP_NOWARN; From patchwork Wed Sep 13 10:53:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 139054 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:a8d:b0:3f2:4152:657d with SMTP id gr13csp43170vqb; Wed, 13 Sep 2023 12:37:19 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEZc+4YAuSLW6lkEQ5vvdcyTxF3IKkSuVtAacZaDnqESwTXS98mPMygbqF4A3J+e8tF6C4J X-Received: by 2002:a17:902:ced1:b0:1c3:a215:a9ba with SMTP id d17-20020a170902ced100b001c3a215a9bamr4802553plg.33.1694633838924; Wed, 13 Sep 2023 12:37:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694633838; cv=none; d=google.com; s=arc-20160816; b=FxdLyuUTEI/MEGlsC/T4eYIRghhRtOZRYZii30auSkghcoEnjmcrsjA+/WOgXJocDU 0MhQX863zUgYKac7tcoHx3LU8KLYDwKHMThK5PriuyDOc3JUKh6HNGdd/cXXOHM92Hl6 SQ4S2WxSxNuaiGHvv2CgbvVEUrxzX7zfjykkO1pmRWJf9T85ZpCe6stpr4fueaO6qp99 G2m7ztY15K36uQUtZF6wL77Wk0d6+HQgIHLaLYGSDs3FpGz6yzatxjOTuTuCuma6n4h+ NWR5KyvP+Aw2T519Fnet85j/IPwtZkv/f2h25ZXpv8hZOylPbiAOfyNAaBpj0pzWtLxw q2rA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=HvAW5HglNzFFKY02dQL2QCkQguw1CPVi9WQHfYQdWVQ=; fh=pwE1+DxwRHQWbcOZzR2ABNMMQVcionEFVt3U0a73p7k=; b=iFB47Je5msQROkb15yR9YUsbZHWie6Q/AICKn/1Bvr21JIlRRWV5MQAWiw3HcOvyAF DDkeRFMmeqVGEfjzHqLfvPB+XjFvtnA9YSMFWQ89BACbOpxL5NMCz1G+w0UdkTm72T7/ XDg5smrAI0Bnd1zw5zVYmIluPrDDlDfpm6dp467W1WvMiOAayiPvW0rN2B/gQvmxxBwQ T/2tzKIZ2qCupkRYwZPpJLlw8xrzo4fBXuDkpA/954VYvZZz55iIBLgJOwju+9PAGyLR /KHZLSzAgL7q+qMmrzN+hX/zY+kjMMNglFHFWFiLOlnVmoVHjIHFMvwPKkHWU4bZFj+l 98ig== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=h0VVpYdN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from pete.vger.email (pete.vger.email. [23.128.96.36]) by mx.google.com with ESMTPS id x15-20020a170902a38f00b001bbc84f3ba7si10418666pla.282.2023.09.13.12.37.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 12:37:18 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=h0VVpYdN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 68C00829E6D3; Wed, 13 Sep 2023 03:54:32 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239802AbjIMKyP (ORCPT + 36 others); Wed, 13 Sep 2023 06:54:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60224 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239795AbjIMKyM (ORCPT ); Wed, 13 Sep 2023 06:54:12 -0400 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D2AA819AF for ; Wed, 13 Sep 2023 03:54:08 -0700 (PDT) Received: by mail-wm1-x32c.google.com with SMTP id 5b1f17b1804b1-401d80f4ef8so72280485e9.1 for ; Wed, 13 Sep 2023 03:54:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1694602447; x=1695207247; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HvAW5HglNzFFKY02dQL2QCkQguw1CPVi9WQHfYQdWVQ=; b=h0VVpYdN84xbMdEtnN1XNkQlKJ+TXQ7/JBBGx9NAvKIGlqjSdcq1QNBIHPvEvmBHZj evJv4iPHL18OT7srF+imsAkmriRjANgq3jGiWZ88pA4i54BdI4LVz37hpNM+NoCuA7ax WoblLdI5f2mpw3cK4qk2tWqgUjT6k49yyX26ODA0O4SMnm/V0vUjxT+Rs9dNMfM/Sr3Q 8GP8BM9rPqeu24zDBGwagsF1A6lRnRIN0+4NrnRM3Sur+43CWeQ4vz22EiqfPG6G7G9A VqCwqiVrQdxLc+kYyTcmi1ePyl9FwlNkwB38zyQjWSeZ5i49JkGAkRAjM7uYW7znQEnR IwDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694602447; x=1695207247; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HvAW5HglNzFFKY02dQL2QCkQguw1CPVi9WQHfYQdWVQ=; b=JYalv5cjFN5df9UpD0G7gc6BeErHwQhUkn64CxLNflMSCF/vYA2StuXKP3dWeHV987 UHDi7k6H30/aa4YEkx8Ddj9ivn0+LCtKWYYB2H2MRy8oIRkHqieonJflc9xoNc8LwuYf 3SuQp4BO6bXwgcc0eBv1SgNYgIYpvQVVU3jflCS8z/k5JUYXflpv1v5EmrzanVNdDt37 DePVmFKIPgOLLSFGCwmt9ZfOqcIdU7bvkMr3xVRqI2QFmMEbBrTPP607aVzdqxi5g8xu VSgfXKvuqv+GqJO+brhvac9iH3fBvoQWC6dp84IVcfzE8cfX53DD5n37hFvf/1LKG7VQ ReWw== X-Gm-Message-State: AOJu0YyU7nU1tG/BAnLAye27CsZ0V+65YOhICfe22bm93s4wDl/rnksL ELqOWnJLHTzsGKSVfAo9vC6T9g== X-Received: by 2002:a05:600c:2196:b0:402:bcac:5773 with SMTP id e22-20020a05600c219600b00402bcac5773mr1720136wme.38.1694602447327; Wed, 13 Sep 2023 03:54:07 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b5c7:0:59d5:28b9:7f13:cd1b]) by smtp.gmail.com with ESMTPSA id p36-20020a05600c1da400b003ff013a4fd9sm1757055wms.7.2023.09.13.03.54.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 03:54:06 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, songmuchun@bytedance.com, fam.zheng@bytedance.com, liangma@liangbit.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v5 2/4] memblock: pass memblock_type to memblock_setclr_flag Date: Wed, 13 Sep 2023 11:53:59 +0100 Message-Id: <20230913105401.519709-3-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230913105401.519709-1-usama.arif@bytedance.com> References: <20230913105401.519709-1-usama.arif@bytedance.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Wed, 13 Sep 2023 03:54:32 -0700 (PDT) X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776952372051638802 X-GMAIL-MSGID: 1776952372051638802 This allows setting flags to both memblock types and is in preparation for setting flags (for e.g. to not initialize struct pages) on reserved memory region. Signed-off-by: Usama Arif Reviewed-by: Muchun Song Reviewed-by: Mike Rapoport (IBM) Acked-by: Mike Kravetz --- mm/memblock.c | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/mm/memblock.c b/mm/memblock.c index 913b2520a9a0..a49efbaee7e0 100644 --- a/mm/memblock.c +++ b/mm/memblock.c @@ -901,10 +901,9 @@ int __init_memblock memblock_physmem_add(phys_addr_t base, phys_addr_t size) * * Return: 0 on success, -errno on failure. */ -static int __init_memblock memblock_setclr_flag(phys_addr_t base, - phys_addr_t size, int set, int flag) +static int __init_memblock memblock_setclr_flag(struct memblock_type *type, + phys_addr_t base, phys_addr_t size, int set, int flag) { - struct memblock_type *type = &memblock.memory; int i, ret, start_rgn, end_rgn; ret = memblock_isolate_range(type, base, size, &start_rgn, &end_rgn); @@ -933,7 +932,7 @@ static int __init_memblock memblock_setclr_flag(phys_addr_t base, */ int __init_memblock memblock_mark_hotplug(phys_addr_t base, phys_addr_t size) { - return memblock_setclr_flag(base, size, 1, MEMBLOCK_HOTPLUG); + return memblock_setclr_flag(&memblock.memory, base, size, 1, MEMBLOCK_HOTPLUG); } /** @@ -945,7 +944,7 @@ int __init_memblock memblock_mark_hotplug(phys_addr_t base, phys_addr_t size) */ int __init_memblock memblock_clear_hotplug(phys_addr_t base, phys_addr_t size) { - return memblock_setclr_flag(base, size, 0, MEMBLOCK_HOTPLUG); + return memblock_setclr_flag(&memblock.memory, base, size, 0, MEMBLOCK_HOTPLUG); } /** @@ -962,7 +961,7 @@ int __init_memblock memblock_mark_mirror(phys_addr_t base, phys_addr_t size) system_has_some_mirror = true; - return memblock_setclr_flag(base, size, 1, MEMBLOCK_MIRROR); + return memblock_setclr_flag(&memblock.memory, base, size, 1, MEMBLOCK_MIRROR); } /** @@ -982,7 +981,7 @@ int __init_memblock memblock_mark_mirror(phys_addr_t base, phys_addr_t size) */ int __init_memblock memblock_mark_nomap(phys_addr_t base, phys_addr_t size) { - return memblock_setclr_flag(base, size, 1, MEMBLOCK_NOMAP); + return memblock_setclr_flag(&memblock.memory, base, size, 1, MEMBLOCK_NOMAP); } /** @@ -994,7 +993,7 @@ int __init_memblock memblock_mark_nomap(phys_addr_t base, phys_addr_t size) */ int __init_memblock memblock_clear_nomap(phys_addr_t base, phys_addr_t size) { - return memblock_setclr_flag(base, size, 0, MEMBLOCK_NOMAP); + return memblock_setclr_flag(&memblock.memory, base, size, 0, MEMBLOCK_NOMAP); } static bool should_skip_region(struct memblock_type *type, From patchwork Wed Sep 13 10:54:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 138924 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9ecd:0:b0:3f2:4152:657d with SMTP id t13csp1198584vqx; Wed, 13 Sep 2023 09:17:29 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHKsY07XrUtm1PpqTsEnC3e8EfkxSBMhAf7wDFxg3hjNpE1hHAMxmlxcXGwxe/BKyKxdmwF X-Received: by 2002:a05:6a00:181b:b0:68f:cdf3:dd54 with SMTP id y27-20020a056a00181b00b0068fcdf3dd54mr3626583pfa.9.1694621849326; Wed, 13 Sep 2023 09:17:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694621849; cv=none; d=google.com; s=arc-20160816; b=pBMc305sfn5HdB9UG6p8sZNsTDoBJDnYgpoAmOv32Oi7Mpa0DRqQ2HuaHLQ6BEF4mh okmzdZ1In3yfMdogzO8h0fzL1aatLg8Jbxd/t6qgrrEbbj8LvVhf76rOVA1Z53DTsZzm eyhOUnHy66rD4kse7XRq7Hqe8wELLvU6WoA7yXFGvP3gEu9AE7e/iduvJb2hLsSgMZlG Wx7YOg3fbFf5tXCxa4ywYOdWCQCGbQLPIezRKfjyhfyD2YuYey/9MM72ClW24eOgTWB6 Fa8I5wkaAGnrJJfQ0eKCc6s5YcPqNf55KgbgyoDjx8pF3jiF75ApNA87+xCugak0R8pp PdPg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Xz27K9YR9w0fQiJ9Te2x4N9NzQgI00KGt4T8XMBUFg0=; fh=pwE1+DxwRHQWbcOZzR2ABNMMQVcionEFVt3U0a73p7k=; b=sAEz5T0h1IRqDrZMmq/XdeaAol3m9grfu9DvOeDWjggCdQqSxMrc8ZhGwy8E8tdk64 frkaOCK3Tobwy/gzjCVVuIN00fVCeqGcoPtrbhYvg0wp8E79NkiqX8bikZswor/5YRuW MR2/LysNkXw3cuXuoe0qReBu3qbtnPpcVorbE/7VnW5kAU9lGvy7qNcWMLlzQSwbhzYf RmGGOtJvy1QRyIP4853RsW7LG6reRuOamc0fyIV/obpIirqYcN/Lh7tlXGVgIPbGdYZa XYfVjPttfdt+FwKCa2TK8ESKLpubak3NfY//Fh9au44Ff/zN2QMfRw0ytu0AT16KQ3ou Lgyw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=WGUJTz8Q; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id r7-20020a63e507000000b00565dfb7d580si9628792pgh.534.2023.09.13.09.17.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 09:17:29 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=WGUJTz8Q; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id B46BD8289535; Wed, 13 Sep 2023 03:54:48 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239807AbjIMKyj (ORCPT + 36 others); Wed, 13 Sep 2023 06:54:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60240 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239795AbjIMKyf (ORCPT ); Wed, 13 Sep 2023 06:54:35 -0400 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F3FB219AD for ; Wed, 13 Sep 2023 03:54:09 -0700 (PDT) Received: by mail-wm1-x333.google.com with SMTP id 5b1f17b1804b1-403012f27e3so48670505e9.3 for ; Wed, 13 Sep 2023 03:54:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1694602448; x=1695207248; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Xz27K9YR9w0fQiJ9Te2x4N9NzQgI00KGt4T8XMBUFg0=; b=WGUJTz8QsszW4Ja7bgpbpod/cBjlUmhVC/9bvnphWkbZPo58rm5UqL9+Tzdy7A+o+G /gRcr87LqxhSEFyxAqf2Fq7lDOzhw/ESxcsrA0xepVDhXnATFm+daRd/d7rNTiFJ1Wq2 FgmvFQ6xSnYBlJqFa3gH5Jm+hQ266Kyhw3vIjAX7OvEX1A0Nli/f9SWkVM7ux0jhEhma 2cwJs9lod8+MBxNBCBSzdiTYI4AIxLjPN/w96DRoC3yPcWl4HynzNyesxLaYGEgl64lX +6j7aS3CuShMFNZRktEGET6U3KslQGuAP1SvSAXTG+8IdarAqQPhpECFOoqDE0WswMXp PMEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694602448; x=1695207248; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Xz27K9YR9w0fQiJ9Te2x4N9NzQgI00KGt4T8XMBUFg0=; b=FhhvYOJUtNQUJmJbITG8citJY4Dc+SeNEyJH7N2LVc2k4gFQcy4W350i6VRt4GNITs cDCoH28wEbe1wWWxzBDpakweCx+oGXfqQS45ZjUedgjqFg/oXZOonsKEcY0+TRn3yZSM 36yA0U2hljQpkn1UYxee+zYQjRp5JuJQvFvmEC/fWyz5BZAYnMCzQ3EwUBKrQjr5UPuG tc3qxyauYbeGAZhL7IO5e9PBQC2dzjCYMMhMUDl62xmM8IskjiO5kTkBBXVzJyAf+GhE +1gg02u5VhnI04YiPrnAtjrhHXvDlE0AG1yRIcwP98PtM6s6jy3IbsWy6/5ubF1286Rj fO1w== X-Gm-Message-State: AOJu0YyraDptqzo4aRe4KjbcyBCz6EU4qYxNjO+ysniJ/oafcLyDEUzX /V3XwcmyfuMCqwxvjVbWnyO9xw== X-Received: by 2002:a05:600c:ac5:b0:401:d803:624f with SMTP id c5-20020a05600c0ac500b00401d803624fmr1620823wmr.4.1694602448422; Wed, 13 Sep 2023 03:54:08 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b5c7:0:59d5:28b9:7f13:cd1b]) by smtp.gmail.com with ESMTPSA id p36-20020a05600c1da400b003ff013a4fd9sm1757055wms.7.2023.09.13.03.54.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 03:54:07 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, songmuchun@bytedance.com, fam.zheng@bytedance.com, liangma@liangbit.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v5 3/4] memblock: introduce MEMBLOCK_RSRV_NOINIT flag Date: Wed, 13 Sep 2023 11:54:00 +0100 Message-Id: <20230913105401.519709-4-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230913105401.519709-1-usama.arif@bytedance.com> References: <20230913105401.519709-1-usama.arif@bytedance.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Wed, 13 Sep 2023 03:54:48 -0700 (PDT) X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776939800262988700 X-GMAIL-MSGID: 1776939800262988700 For reserved memory regions marked with this flag, reserve_bootmem_region is not called during memmap_init_reserved_pages. This can be used to avoid struct page initialization for regions which won't need them, for e.g. hugepages with Hugepage Vmemmap Optimization enabled. Signed-off-by: Usama Arif Acked-by: Muchun Song Reviewed-by: Mike Rapoport (IBM) --- include/linux/memblock.h | 9 +++++++++ mm/memblock.c | 33 ++++++++++++++++++++++++++++----- 2 files changed, 37 insertions(+), 5 deletions(-) diff --git a/include/linux/memblock.h b/include/linux/memblock.h index 1c1072e3ca06..ae3bde302f70 100644 --- a/include/linux/memblock.h +++ b/include/linux/memblock.h @@ -40,6 +40,8 @@ extern unsigned long long max_possible_pfn; * via a driver, and never indicated in the firmware-provided memory map as * system RAM. This corresponds to IORESOURCE_SYSRAM_DRIVER_MANAGED in the * kernel resource tree. + * @MEMBLOCK_RSRV_NOINIT: memory region for which struct pages are + * not initialized (only for reserved regions). */ enum memblock_flags { MEMBLOCK_NONE = 0x0, /* No special request */ @@ -47,6 +49,7 @@ enum memblock_flags { MEMBLOCK_MIRROR = 0x2, /* mirrored region */ MEMBLOCK_NOMAP = 0x4, /* don't add to kernel direct mapping */ MEMBLOCK_DRIVER_MANAGED = 0x8, /* always detected via a driver */ + MEMBLOCK_RSRV_NOINIT = 0x10, /* don't initialize struct pages */ }; /** @@ -125,6 +128,7 @@ int memblock_clear_hotplug(phys_addr_t base, phys_addr_t size); int memblock_mark_mirror(phys_addr_t base, phys_addr_t size); int memblock_mark_nomap(phys_addr_t base, phys_addr_t size); int memblock_clear_nomap(phys_addr_t base, phys_addr_t size); +int memblock_reserved_mark_noinit(phys_addr_t base, phys_addr_t size); void memblock_free_all(void); void memblock_free(void *ptr, size_t size); @@ -259,6 +263,11 @@ static inline bool memblock_is_nomap(struct memblock_region *m) return m->flags & MEMBLOCK_NOMAP; } +static inline bool memblock_is_reserved_noinit(struct memblock_region *m) +{ + return m->flags & MEMBLOCK_RSRV_NOINIT; +} + static inline bool memblock_is_driver_managed(struct memblock_region *m) { return m->flags & MEMBLOCK_DRIVER_MANAGED; diff --git a/mm/memblock.c b/mm/memblock.c index a49efbaee7e0..8f7a0cb668d4 100644 --- a/mm/memblock.c +++ b/mm/memblock.c @@ -996,6 +996,24 @@ int __init_memblock memblock_clear_nomap(phys_addr_t base, phys_addr_t size) return memblock_setclr_flag(&memblock.memory, base, size, 0, MEMBLOCK_NOMAP); } +/** + * memblock_reserved_mark_noinit - Mark a reserved memory region with flag + * MEMBLOCK_RSRV_NOINIT which results in the struct pages not being initialized + * for this region. + * @base: the base phys addr of the region + * @size: the size of the region + * + * struct pages will not be initialized for reserved memory regions marked with + * %MEMBLOCK_RSRV_NOINIT. + * + * Return: 0 on success, -errno on failure. + */ +int __init_memblock memblock_reserved_mark_noinit(phys_addr_t base, phys_addr_t size) +{ + return memblock_setclr_flag(&memblock.reserved, base, size, 1, + MEMBLOCK_RSRV_NOINIT); +} + static bool should_skip_region(struct memblock_type *type, struct memblock_region *m, int nid, int flags) @@ -2112,13 +2130,18 @@ static void __init memmap_init_reserved_pages(void) memblock_set_node(start, end, &memblock.reserved, nid); } - /* initialize struct pages for the reserved regions */ + /* + * initialize struct pages for reserved regions that don't have + * the MEMBLOCK_RSRV_NOINIT flag set + */ for_each_reserved_mem_region(region) { - nid = memblock_get_region_node(region); - start = region->base; - end = start + region->size; + if (!memblock_is_reserved_noinit(region)) { + nid = memblock_get_region_node(region); + start = region->base; + end = start + region->size; - reserve_bootmem_region(start, end, nid); + reserve_bootmem_region(start, end, nid); + } } } From patchwork Wed Sep 13 10:54:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 139439 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp191557vqi; Thu, 14 Sep 2023 01:17:45 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGJIK7DDlhTJAOfDGZYkNTDp/HCqUAja6sCcK4T70l/+oC75R2qpydz4H1BpeagsNMBUsn3 X-Received: by 2002:a05:6300:808c:b0:14c:def1:e728 with SMTP id ap12-20020a056300808c00b0014cdef1e728mr4044369pzc.60.1694679464626; Thu, 14 Sep 2023 01:17:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694679464; cv=none; d=google.com; s=arc-20160816; b=zvHLDdIrTOVtYSPw9ebaSOhtyqcqYD/1XBYYV90T9D9OUjzdwiF7FnWSYOqPiKgO2I 8Gwq9nDq3m9HNKZB8RaxDoYyX/S1s64KsHr3r6TwENd/WX/K5H8lBa0i/6JNS3CAlXn1 M7q/Mhfc8AdknXZHebsbjdfswOojDVyJSGLhb1tFUJ2oXySiPCazt3ks/PUVQK23i88z zLWFpwbzExUTuwTPw8uFpkPsH96bSA3i+nJMok5NcuVh7rRGgpSQ+EXthp6E2TYSsm3X ATOUIlMLJcw2XLW7Dp2MmS2FxQ6hk8qvYzifl9V/RdRtkG/fm1FzJnLf5ObgNQOXvC+R Iqpg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=a6h7k6NjGFuqJLpb7jnCYfQxn4IX1VZuVPP7jz0KAXo=; fh=pwE1+DxwRHQWbcOZzR2ABNMMQVcionEFVt3U0a73p7k=; b=puPsyHRHfPoyp6EjZ10xNkzthyo8wWIIegcPjGEpaAEHxd5vmYaX6r6svPxUI5gWdy OStCi+fDqVj7UKjCmSQGOEZbNK0A0JGHLdbvpNGrxjeA0AOlYThslPvuMo7ZR8XVGh/d d45BDfLHbTVtnUvYEgr/ERFAuh4PFYWcGfSMRPkBzuoBUSfHfVUEGa0scrFN2Zlv0Oi8 0YPap0jON7xje/pz0pMpTLRAV+sHVU9bLZio66qsXhhSEHLQ46Ip0aLHW32PF6Ugt730 sE+EMYknPiW6/IKH7L4mcNhs7EVediEyAA0IVNhSXk+Qm0b6lwi8/OUNdjECDyPYGQ/Z dfpQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=Il2gmVNS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from groat.vger.email (groat.vger.email. [2620:137:e000::3:5]) by mx.google.com with ESMTPS id i17-20020a170902cf1100b001b9c205a860si1254660plg.377.2023.09.14.01.17.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Sep 2023 01:17:44 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) client-ip=2620:137:e000::3:5; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=Il2gmVNS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id B9C538126737; Wed, 13 Sep 2023 03:55:11 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239825AbjIMKym (ORCPT + 36 others); Wed, 13 Sep 2023 06:54:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60254 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239774AbjIMKyf (ORCPT ); Wed, 13 Sep 2023 06:54:35 -0400 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E102C19B0 for ; Wed, 13 Sep 2023 03:54:10 -0700 (PDT) Received: by mail-wm1-x329.google.com with SMTP id 5b1f17b1804b1-403012f276dso45401955e9.0 for ; Wed, 13 Sep 2023 03:54:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1694602449; x=1695207249; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=a6h7k6NjGFuqJLpb7jnCYfQxn4IX1VZuVPP7jz0KAXo=; b=Il2gmVNS1UwjG+jTYRikdvw9VYRUsjgiPbO357R0u8SQfjZLpcqlbI4wgt8pFuqDVq N/1RhK/O+txiCZ/PRH4C+qTzDEt//QVBhbJL2vmWK8I5JYKH4qQYlf3inSUJH4oJDzwk rYhOKutisxQPaptw6+a+Fbjii5dQWWktLkKVBS55KuyNx/tGFFtOE9809FatILUnex62 o129rd0tpcbIqVWfB+EZgU5fsiijYgfn+OYvlwr7N9Ol7lLMZFsAm7hC0R7cpdA+p5jk xCz8440PH3q36JVcq0zqzcSFjQOFTcCHVf7AYQWes1ld6hfyMbJaT/fMa9twqWKV+3bi M0LA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694602449; x=1695207249; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=a6h7k6NjGFuqJLpb7jnCYfQxn4IX1VZuVPP7jz0KAXo=; b=W89F2mgirqA1scnFJyu/omlNCzN7WdRKlLj+Fa/177Gu7bb+C2W2SzLvrJ3l9dE/TB GG0RDuBhD9OpLO95pR8azM+oNcCBZEXxTaBRCG+JV9PSQTPpkk0eNNYR0KnrTn1+Zljg aVXe4orpf6bwh0e9F/S58kLp3+wE640lviIxNzL/fCeZe5BFOJ6sKv5MKwXPpvfNmLm8 tM8Hy8/dxikbFSVuPUqoDheFeNvdrXzILNbiAVs5PSWe+f2d+RwFOpWOVXkSLAN+FoGC ZQaXd5HPYWdEUxuN5sAKYOAkLWkd7Bwpfbwlc8WeK5jfaAXjE1nE3hzdKF4sB5DcYwAP H3VA== X-Gm-Message-State: AOJu0Yxd3mCC2Kj08Snxc8Y/xblMd6+YQmfc3oBpByXa8QfQl2dq2vA/ wWBQs0UwFbbgxAzWAPwBxM72NA== X-Received: by 2002:a1c:4c0d:0:b0:3fd:3006:410b with SMTP id z13-20020a1c4c0d000000b003fd3006410bmr1731104wmf.34.1694602449407; Wed, 13 Sep 2023 03:54:09 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b5c7:0:59d5:28b9:7f13:cd1b]) by smtp.gmail.com with ESMTPSA id p36-20020a05600c1da400b003ff013a4fd9sm1757055wms.7.2023.09.13.03.54.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 03:54:08 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, songmuchun@bytedance.com, fam.zheng@bytedance.com, liangma@liangbit.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v5 4/4] mm: hugetlb: Skip initialization of gigantic tail struct pages if freed by HVO Date: Wed, 13 Sep 2023 11:54:01 +0100 Message-Id: <20230913105401.519709-5-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230913105401.519709-1-usama.arif@bytedance.com> References: <20230913105401.519709-1-usama.arif@bytedance.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Wed, 13 Sep 2023 03:55:11 -0700 (PDT) X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776945173147129086 X-GMAIL-MSGID: 1777000214183859563 The new boot flow when it comes to initialization of gigantic pages is as follows: - At boot time, for a gigantic page during __alloc_bootmem_hugepage, the region after the first struct page is marked as noinit. - This results in only the first struct page to be initialized in reserve_bootmem_region. As the tail struct pages are not initialized at this point, there can be a significant saving in boot time if HVO succeeds later on. - Later on in the boot, the head page is prepped and the first HUGETLB_VMEMMAP_RESERVE_SIZE / sizeof(struct page) - 1 tail struct pages are initialized. - HVO is attempted. If it is not successful, then the rest of the tail struct pages are initialized. If it is successful, no more tail struct pages need to be initialized saving significant boot time. The WARN_ON for increased ref count in gather_bootmem_prealloc was changed to a VM_BUG_ON. This is OK as there should be no speculative references this early in boot process. The VM_BUG_ON's are there just in case such code is introduced. Signed-off-by: Usama Arif Reviewed-by: Muchun Song Reviewed-by: Mike Kravetz --- mm/hugetlb.c | 63 +++++++++++++++++++++++++++++++++++++------- mm/hugetlb_vmemmap.c | 2 +- mm/hugetlb_vmemmap.h | 9 ++++--- mm/internal.h | 3 +++ mm/mm_init.c | 2 +- 5 files changed, 64 insertions(+), 15 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index c32ca241df4b..ed37c6e4e952 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3169,6 +3169,15 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) } found: + + /* + * Only initialize the head struct page in memmap_init_reserved_pages, + * rest of the struct pages will be initialized by the HugeTLB subsystem itself. + * The head struct page is used to get folio information by the HugeTLB + * subsystem like zone id and node id. + */ + memblock_reserved_mark_noinit(virt_to_phys((void *)m + PAGE_SIZE), + huge_page_size(h) - PAGE_SIZE); /* Put them into a private list first because mem_map is not up yet */ INIT_LIST_HEAD(&m->list); list_add(&m->list, &huge_boot_pages); @@ -3176,6 +3185,42 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) return 1; } +/* Initialize [start_page:end_page_number] tail struct pages of a hugepage */ +static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio, + unsigned long start_page_number, + unsigned long end_page_number) +{ + enum zone_type zone = zone_idx(folio_zone(folio)); + int nid = folio_nid(folio); + unsigned long head_pfn = folio_pfn(folio); + unsigned long pfn, end_pfn = head_pfn + end_page_number; + int ret; + + for (pfn = head_pfn + start_page_number; pfn < end_pfn; pfn++) { + struct page *page = pfn_to_page(pfn); + + __init_single_page(page, pfn, zone, nid); + prep_compound_tail((struct page *)folio, pfn - head_pfn); + ret = page_ref_freeze(page, 1); + VM_BUG_ON(!ret); + } +} + +static void __init hugetlb_folio_init_vmemmap(struct folio *folio, struct hstate *h, + unsigned long nr_pages) +{ + int ret; + + /* Prepare folio head */ + __folio_clear_reserved(folio); + __folio_set_head(folio); + ret = page_ref_freeze(&folio->page, 1); + VM_BUG_ON(!ret); + /* Initialize the necessary tail struct pages */ + hugetlb_folio_init_tail_vmemmap(folio, 1, nr_pages); + prep_compound_head((struct page *)folio, huge_page_order(h)); +} + /* * Put bootmem huge pages into the standard lists after mem_map is up. * Note: This only applies to gigantic (order > MAX_ORDER) pages. @@ -3186,19 +3231,19 @@ static void __init gather_bootmem_prealloc(void) list_for_each_entry(m, &huge_boot_pages, list) { struct page *page = virt_to_page(m); - struct folio *folio = page_folio(page); + struct folio *folio = (void *)page; struct hstate *h = m->hstate; VM_BUG_ON(!hstate_is_gigantic(h)); WARN_ON(folio_ref_count(folio) != 1); - if (prep_compound_gigantic_folio(folio, huge_page_order(h))) { - WARN_ON(folio_test_reserved(folio)); - prep_new_hugetlb_folio(h, folio, folio_nid(folio)); - free_huge_folio(folio); /* add to the hugepage allocator */ - } else { - /* VERY unlikely inflated ref count on a tail page */ - free_gigantic_folio(folio, huge_page_order(h)); - } + + hugetlb_folio_init_vmemmap(folio, h, HUGETLB_VMEMMAP_RESERVE_PAGES); + prep_new_hugetlb_folio(h, folio, folio_nid(folio)); + /* If HVO fails, initialize all tail struct pages */ + if (!HPageVmemmapOptimized(&folio->page)) + hugetlb_folio_init_tail_vmemmap(folio, HUGETLB_VMEMMAP_RESERVE_PAGES, + pages_per_huge_page(h)); + free_huge_folio(folio); /* add to the hugepage allocator */ /* * We need to restore the 'stolen' pages to totalram_pages diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 3cdb38d87a95..772a877918d7 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -589,7 +589,7 @@ static int __init hugetlb_vmemmap_init(void) const struct hstate *h; /* HUGETLB_VMEMMAP_RESERVE_SIZE should cover all used struct pages */ - BUILD_BUG_ON(__NR_USED_SUBPAGE * sizeof(struct page) > HUGETLB_VMEMMAP_RESERVE_SIZE); + BUILD_BUG_ON(__NR_USED_SUBPAGE > HUGETLB_VMEMMAP_RESERVE_PAGES); for_each_hstate(h) { if (hugetlb_vmemmap_optimizable(h)) { diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 25bd0e002431..4573899855d7 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -10,15 +10,16 @@ #define _LINUX_HUGETLB_VMEMMAP_H #include -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP -int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head); -void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head); - /* * Reserve one vmemmap page, all vmemmap addresses are mapped to it. See * Documentation/vm/vmemmap_dedup.rst. */ #define HUGETLB_VMEMMAP_RESERVE_SIZE PAGE_SIZE +#define HUGETLB_VMEMMAP_RESERVE_PAGES (HUGETLB_VMEMMAP_RESERVE_SIZE / sizeof(struct page)) + +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head); +void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head); static inline unsigned int hugetlb_vmemmap_size(const struct hstate *h) { diff --git a/mm/internal.h b/mm/internal.h index d1d4bf4e63c0..d74061aa6de7 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1154,4 +1154,7 @@ struct vma_prepare { struct vm_area_struct *remove; struct vm_area_struct *remove2; }; + +void __meminit __init_single_page(struct page *page, unsigned long pfn, + unsigned long zone, int nid); #endif /* __MM_INTERNAL_H */ diff --git a/mm/mm_init.c b/mm/mm_init.c index 50f2f34745af..fed4370b02e1 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -555,7 +555,7 @@ static void __init find_zone_movable_pfns_for_nodes(void) node_states[N_MEMORY] = saved_node_state; } -static void __meminit __init_single_page(struct page *page, unsigned long pfn, +void __meminit __init_single_page(struct page *page, unsigned long pfn, unsigned long zone, int nid) { mm_zero_struct_page(page);