From patchwork Thu Jul 27 20:46:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 127184 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp13995vqg; Thu, 27 Jul 2023 14:10:59 -0700 (PDT) X-Google-Smtp-Source: APBJJlGD69j9awivEUEFpAOQTnlmWQI1qk55b2f3u2HpHp8j9fRngbekqdurr7vy1Hv2hwkfzdGI X-Received: by 2002:a05:6a00:2292:b0:686:2ad5:d11c with SMTP id f18-20020a056a00229200b006862ad5d11cmr337717pfe.33.1690492258892; Thu, 27 Jul 2023 14:10:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690492258; cv=none; d=google.com; s=arc-20160816; b=xurXEg0lfPrQ7msgKHG6vUXP+SPpN5L3W5fUo31UFUk/HrvpNo2unVHszs1yA3zOu3 iMrTqJIWS7/qyRqIJ8o5TdySx1ml0ZMzPNH8d4VqtTTrHCCcghP/pwpYwWkljyguJqWt RkQ4E8ue1jCOeU1Yu7PMX4aXhjXQDoZ73Ka6IhNlar9GXqRoEWZCvy+sdWTnMmfHj6fY lEqIByPEu+DPMcyeUgblrc+1tG6NI7ZC3udmeMPX/X3YciVKBm7bAX9vNL1MVeMskcOq CtHFCe8nmhW51zej0lOptPoagxqhLabx6uwJFrB+9wuKn/ZmxDbJTM14u1QVXT7l74gV XKGw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=oaqNw2Hm1guEkvjK0qvKkO1TrGxzTe44nwo0h0OS4xY=; fh=6ywE3cadAEHt3CqWKbi2najT+v0c7si89uhbZ0SfPzA=; b=Iiep911BEqDa/nUrYMAIYibZD+c/Wp7PU9BZ9G3KIx60SRZpVAtt7hUQ4M/THQSIwy Am6TtYJn8vsMdqFW8pIx5cWE2FweUkJAfejQYHD9hdAzC+Mt8g315fcPeTJPwg1UGi9u m5MYWzEjbrmeezenWvi7MpT8yp8hNBK4eY7GwcqY/4ZB4+Rgu7uyYUuVp8Mc8nPEA57G 2W5jmSoNBKGSB+0PeJmaJcE6I+FJISp3zoKhlWQZmIKbEOLTdAr6C7vfyOiKPpzDvZdq /7vO8cfuaFPUo/wUTAiAC8lp2D5W0XJMSYs1hlzab5Kg1dXb4TXjGMF5s7IVjR35PkOT AUAQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=Qh4G5pkJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id fi35-20020a056a0039a300b00686e7b259fcsi1898449pfb.184.2023.07.27.14.10.46; Thu, 27 Jul 2023 14:10:58 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=Qh4G5pkJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232360AbjG0UrE (ORCPT + 99 others); Thu, 27 Jul 2023 16:47:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34388 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231570AbjG0Uq4 (ORCPT ); Thu, 27 Jul 2023 16:46:56 -0400 Received: from mail-lj1-x232.google.com (mail-lj1-x232.google.com [IPv6:2a00:1450:4864:20::232]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C6597271D for ; Thu, 27 Jul 2023 13:46:33 -0700 (PDT) Received: by mail-lj1-x232.google.com with SMTP id 38308e7fff4ca-2b701e1ca63so21959291fa.1 for ; Thu, 27 Jul 2023 13:46:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690490792; x=1691095592; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=oaqNw2Hm1guEkvjK0qvKkO1TrGxzTe44nwo0h0OS4xY=; b=Qh4G5pkJ2CklC1dn5cBz4RJXwctIF25YGEWE2oQ3ZrOzxnKalFKae/8pznmb31lV7C WTzPyronTTWL8g0r5I8lSbz/AdU7gvJCGq8K2Qd4vWzRvFiQ/FfCwXl6dsQ9CmNizTh+ m0AH/lVmLiMfUecg6qsdvv9XMrT/D7wOGOTWcG9Zt50oRSe1Bavo9Ko7yaB4wjJ+Qnnr BNWwV0vvSuj9U4WuwZrl1x2sIhyQwELI33CtL1QqIzp+Rtzz205zsveFrvcH9XBhLeZX G7lPHvXhJp/f1c2Jmw7kl1BPg6BGcbtzSnM+LE3IJIaKxCmCB8OfQyprqC6Rh3SHBpsg Wpxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690490792; x=1691095592; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oaqNw2Hm1guEkvjK0qvKkO1TrGxzTe44nwo0h0OS4xY=; b=cIWRrNJAxXNGf/cCZhI/WhleJ19H1lVTspF1KEz1+LkNa+BpyAJ+m7YXZSFRX8KH8d pneT9TvEiEmnUN16gKM9HWiLhanRdxidkvdlBx83NWaPgWc7MHxjiNinmoXPLKkoH6fB vzWqrEoM4yFn/SvLONJh7akI9MvWdrNPRS1PNOrGiimjXAJ2F/sWASw2/7LTPI7dnEwF ucIZ/UfvzsJQzMCnPHlc9B4/CNQJA19qkYKkZgHwELSTZ0F8vBGofPZxVfybIitiO6SH IfmZh99jDUMozxHvz5OpiwSAuTAW6R2XGEZ2GUspB2NvXC2BhFijze1zaKQarx1M7mNO VwSA== X-Gm-Message-State: ABy/qLZgHUZWAldkarwnuXffsAWSX4T5nA+KGPwAlHXFCiUCjrncRqVT zta/RDF8HM2Gb6VMjtJo1WXDyg== X-Received: by 2002:a05:651c:1042:b0:2b6:da64:321 with SMTP id x2-20020a05651c104200b002b6da640321mr95486ljm.45.1690490792019; Thu, 27 Jul 2023 13:46:32 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:7e3e:db9e:70fa:9ccb]) by smtp.gmail.com with ESMTPSA id k1-20020a7bc401000000b003fc06169abdsm2701400wmi.2.2023.07.27.13.46.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Jul 2023 13:46:31 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v1 1/6] mm: hugetlb: Skip prep of tail pages when HVO is enabled Date: Thu, 27 Jul 2023 21:46:19 +0100 Message-Id: <20230727204624.1942372-2-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230727204624.1942372-1-usama.arif@bytedance.com> References: <20230727204624.1942372-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772609611021874759 X-GMAIL-MSGID: 1772609611021874759 When vmemmap is optimizable, it will free all the duplicated tail pages in hugetlb_vmemmap_optimize while preparing the new hugepage. Hence, there is no need to prepare them. For 1G x86 hugepages, it avoids preparing 262144 - 64 = 262080 struct pages per hugepage. Signed-off-by: Usama Arif --- mm/hugetlb.c | 32 +++++++++++++++++++++++--------- mm/hugetlb_vmemmap.c | 2 +- mm/hugetlb_vmemmap.h | 7 +++++++ 3 files changed, 31 insertions(+), 10 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 64a3239b6407..58cf5978bee1 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1943,13 +1943,24 @@ static void prep_new_hugetlb_folio(struct hstate *h, struct folio *folio, int ni } static bool __prep_compound_gigantic_folio(struct folio *folio, - unsigned int order, bool demote) + unsigned int order, bool demote, + bool hvo) { int i, j; int nr_pages = 1 << order; struct page *p; __folio_clear_reserved(folio); + +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP + /* + * No need to prep pages that will be freed later by hugetlb_vmemmap_optimize + * in prep_new_huge_page. Hence, reduce nr_pages to the pages that will be kept. + */ + if (hvo) + nr_pages = HUGETLB_VMEMMAP_RESERVE_SIZE / sizeof(struct page); +#endif + for (i = 0; i < nr_pages; i++) { p = folio_page(folio, i); @@ -2020,15 +2031,15 @@ static bool __prep_compound_gigantic_folio(struct folio *folio, } static bool prep_compound_gigantic_folio(struct folio *folio, - unsigned int order) + unsigned int order, bool hvo) { - return __prep_compound_gigantic_folio(folio, order, false); + return __prep_compound_gigantic_folio(folio, order, false, hvo); } static bool prep_compound_gigantic_folio_for_demote(struct folio *folio, - unsigned int order) + unsigned int order, bool hvo) { - return __prep_compound_gigantic_folio(folio, order, true); + return __prep_compound_gigantic_folio(folio, order, true, hvo); } /* @@ -2185,7 +2196,8 @@ static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, if (!folio) return NULL; if (hstate_is_gigantic(h)) { - if (!prep_compound_gigantic_folio(folio, huge_page_order(h))) { + if (!prep_compound_gigantic_folio(folio, huge_page_order(h), + vmemmap_should_optimize(h, &folio->page))) { /* * Rare failure to convert pages to compound page. * Free pages and try again - ONCE! @@ -3201,7 +3213,8 @@ static void __init gather_bootmem_prealloc(void) VM_BUG_ON(!hstate_is_gigantic(h)); WARN_ON(folio_ref_count(folio) != 1); - if (prep_compound_gigantic_folio(folio, huge_page_order(h))) { + if (prep_compound_gigantic_folio(folio, huge_page_order(h), + vmemmap_should_optimize(h, page))) { WARN_ON(folio_test_reserved(folio)); prep_new_hugetlb_folio(h, folio, folio_nid(folio)); free_huge_page(page); /* add to the hugepage allocator */ @@ -3624,8 +3637,9 @@ static int demote_free_hugetlb_folio(struct hstate *h, struct folio *folio) subpage = folio_page(folio, i); inner_folio = page_folio(subpage); if (hstate_is_gigantic(target_hstate)) - prep_compound_gigantic_folio_for_demote(inner_folio, - target_hstate->order); + prep_compound_gigantic_folio_for_demote(folio, + target_hstate->order, + vmemmap_should_optimize(target_hstate, subpage)); else prep_compound_page(subpage, target_hstate->order); folio_change_private(inner_folio, NULL); diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index c2007ef5e9b0..b721e87de2b3 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -486,7 +486,7 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) } /* Return true iff a HugeTLB whose vmemmap should and can be optimized. */ -static bool vmemmap_should_optimize(const struct hstate *h, const struct page *head) +bool vmemmap_should_optimize(const struct hstate *h, const struct page *head) { if (!READ_ONCE(vmemmap_optimize_enabled)) return false; diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 25bd0e002431..07555d2dc0cb 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -13,6 +13,7 @@ #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head); void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head); +bool vmemmap_should_optimize(const struct hstate *h, const struct page *head); /* * Reserve one vmemmap page, all vmemmap addresses are mapped to it. See @@ -51,6 +52,12 @@ static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct hstate { return 0; } + +bool vmemmap_should_optimize(const struct hstate *h, const struct page *head) +{ + return false; +} + #endif /* CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP */ static inline bool hugetlb_vmemmap_optimizable(const struct hstate *h) From patchwork Thu Jul 27 20:46:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 127188 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp19188vqg; Thu, 27 Jul 2023 14:23:46 -0700 (PDT) X-Google-Smtp-Source: APBJJlFT52uMRIhrkoXi1KGRtOmr5sxvsRGh0Znaa3uhkbgIlVJXGqFYeMPAEvnZ8CXpNFSWW2wJ X-Received: by 2002:a17:902:c406:b0:1b9:f7f4:5687 with SMTP id k6-20020a170902c40600b001b9f7f45687mr683246plk.24.1690493026025; Thu, 27 Jul 2023 14:23:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690493026; cv=none; d=google.com; s=arc-20160816; b=tRGqWN9i99tSeDLT2OHQMqq0D9o2BX0rqFnX7HV4tRw5pAt4IUbmFWl+/oSGySUjsm L4kFfcpwsvnkzHrDQeDe6VIYGF1mizrNTwu7D6gwz4OPTHI5lsVS/Stit2msbHp3LTnC 2pKunQ9ceYafuDdalBR2s73lUaFrBC9FNQn+IMhrgTXD4pm0wIvM7u4N4d7vaK5/cplv m2y6aXEfCvfgnkvCAbZVby/AhhcfOvoUoiWMEK2oj2HRRTNPxIposIZLT5YSyD0p36SC +AF7hxDnV7OPFqAPbijN3goZ43eYU0+WjkDvAuE9OSP8t+T7oYjKu3rnfPq/wcyK9HIc 0TGQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=VUFHsk/m/GY1eTNfZx8Py//rj4lEwu1OMjpML8i4XiA=; fh=6ywE3cadAEHt3CqWKbi2najT+v0c7si89uhbZ0SfPzA=; b=HdiYP6KiL1xh2qPQlIesGWUGnCbUG0yRpWQQJ7jO+Xft9+3MG72iBHmc744byLuUnD Ew/TB9JdgphibUBv9DMz5BJ7HvU9PswHeYWQsPcGuoPWHvWffYzwb+mkSIU6zh9QftSo GAfVrKg8WEUNRhFMRTv2RrWPcOHpeNlwrxmyh9NP4NBP5/HJtkCoKatAL5PUQjJy68BP fqPi/4JZGd7AK7+FGaKtftCzUlWqaEnNtLuXutxh266Z9m3pYF4nZNyg2k0YwVoh4DbV xR6GfJGjLVDPAuTveoU5xIKzDzncs/eZ3b3yJXCewMNYyian2dn6H3GzitsP8HF8uTcf 00tQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=ZZUlyW4Y; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id lb14-20020a170902fa4e00b001b896f8ae83si1756607plb.110.2023.07.27.14.23.32; Thu, 27 Jul 2023 14:23:46 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=ZZUlyW4Y; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231219AbjG0Uqh (ORCPT + 99 others); Thu, 27 Jul 2023 16:46:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34390 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229522AbjG0Uqf (ORCPT ); Thu, 27 Jul 2023 16:46:35 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 923272D45 for ; Thu, 27 Jul 2023 13:46:34 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id ffacd0b85a97d-3176a439606so1400731f8f.3 for ; Thu, 27 Jul 2023 13:46:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690490793; x=1691095593; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VUFHsk/m/GY1eTNfZx8Py//rj4lEwu1OMjpML8i4XiA=; b=ZZUlyW4YzysoCHAYMp2/sA6YrKmTeac3t3azsTg4jDgno5140SsxzboZ2DX+sYyTar QBt4NCcbIwVqU6ECOkAxzfNVYTJSA4UoAO9vNAzRgOzSciNduCZzg6hsTy30yZygE/FX Y3IhdC9ei8B4xOpa33YxJ+rAYbRMQmfBhHWbkpR2MzbCVvhFxC5cl2XdP9qY+d1bHhgd lqfn5I8dfHTphHuAk/OGTexXituKKGFsXZtEGV8Pe+xS2H+P9mt/hFkl6RI2Q/3BRFmT 8QbfWy8NIL5JwByXBCIWXDjISwNspvj4RcypyTUIR9HRGShbKOG5H+xfqxUKSelybJ2e i7JQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690490793; x=1691095593; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VUFHsk/m/GY1eTNfZx8Py//rj4lEwu1OMjpML8i4XiA=; b=XGLUlnn1iD+NYgCwrKBTGtwffgNr6WkRbv0VflyZIkqC9iSxHp+mrMKa+L1mFJketw mVA5rEGN6XBAUqxuHxI4b5AjZcVXWXP33T9FmONwd10gEkGiZw+iSfyEq52HZUgvAkx6 1O7hJGnbD90U4ybeRe5UTHPNzBmtYZoW/++QWNLKsYq3bkJOEXQA+4ZyLFi0YNpbweXR xalkJP/GY7OyWKxIKb49CzfvMMuV6xMV61lCVTOfDiP/dS6r6nw4ldDZb73l9sgEupck RfGE36Q5qRBSYLcvdmzbKyfkwhwPCHDpM0N3PUy5ov4O39Y7QBVx2mc6cD+5YJ9HVG1e 7WdA== X-Gm-Message-State: ABy/qLbliNXnrAtc00L9SaqcPaiRNlWyHXfTxAtIxjsiyMVGRo2blJ1M jQJMhG7V1eLcN6MH2gwPWYlEdA== X-Received: by 2002:adf:ef4f:0:b0:314:32b6:af3 with SMTP id c15-20020adfef4f000000b0031432b60af3mr204498wrp.5.1690490792997; Thu, 27 Jul 2023 13:46:32 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:7e3e:db9e:70fa:9ccb]) by smtp.gmail.com with ESMTPSA id k1-20020a7bc401000000b003fc06169abdsm2701400wmi.2.2023.07.27.13.46.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Jul 2023 13:46:32 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v1 2/6] mm: hugetlb_vmemmap: Use nid of the head page to reallocate it Date: Thu, 27 Jul 2023 21:46:20 +0100 Message-Id: <20230727204624.1942372-3-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230727204624.1942372-1-usama.arif@bytedance.com> References: <20230727204624.1942372-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772610415530992773 X-GMAIL-MSGID: 1772610415530992773 If tail page prep and initialization is skipped, then the "start" page will not contain the correct nid. Use the nid from first vmemap page. Signed-off-by: Usama Arif --- mm/hugetlb_vmemmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index b721e87de2b3..bdf750a4786b 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -324,7 +324,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, .reuse_addr = reuse, .vmemmap_pages = &vmemmap_pages, }; - int nid = page_to_nid((struct page *)start); + int nid = page_to_nid((struct page *)reuse); gfp_t gfp_mask = GFP_KERNEL | __GFP_THISNODE | __GFP_NORETRY | __GFP_NOWARN; From patchwork Thu Jul 27 20:46:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 127183 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp13987vqg; Thu, 27 Jul 2023 14:10:58 -0700 (PDT) X-Google-Smtp-Source: APBJJlHh6q80EmDwhH7PYbb1H+GT7MddMdTI9NrdkkbZrTpO9g7PucAXPgL7s/AV3jU4m8LGbxjn X-Received: by 2002:a17:90a:6b85:b0:268:522a:e31 with SMTP id w5-20020a17090a6b8500b00268522a0e31mr382321pjj.42.1690492257725; Thu, 27 Jul 2023 14:10:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690492257; cv=none; d=google.com; s=arc-20160816; b=oXLdyBFl/QthxzlDFZ8kzbZCuys8XYZ01NTkDJwVNLst5/QD8RYBFBICcwKv8l2TYg 29DWBwQwlf3aICvZdzwkYA5oNT2RgIYzMUCagTAsmkXGFdIgCKPB4qagPKGCqqEmy4T/ xKIKuzlJfTCet/EOexQJKOm2pVg7bAff5IB+/hZmvVcn+yUN5k8CbCD3fqoOWT3VOi60 luEvQUEzhZoFMt9rwAJU6toqCYDzmox7IwQB7ttRoqbEGAC54axs9TGIx1doxo8p0Hc1 1yG45OnxpE4qmLmypwePr/GqhFZ4aNtqbHB1gviWTiRgXA8AGoPHn3WYPxTo2hIfLCZG ZaOA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=R/2kTGP5bmExpEckq+8Rz1vqcaG4uRUICK5uorcV/EQ=; fh=6ywE3cadAEHt3CqWKbi2najT+v0c7si89uhbZ0SfPzA=; b=guTHy2RcA/H6aicWNpO4rYeLgieTkSQKFxRHgFaF1y3K+ivCatDAiknczEVviBqQ7g JaObiYyc8G7gIOv7fMxVVncAEVaTT1WJpQMkict9nxr2wuDPmFp104OZcl3of6u8pn+a wNP6dVjIvQ8s445BoSYDXoOfoapnZosgcFhW3aK/OEJOOo4IrRWRtH33FJhpD/4RIRSP agiP7rOyVXESGJcWPpLbmFPA76kD51WLpONXfrWf8rCw0jZoeIwjOWUj/6oEMXS1YFjc WOZMohdgtds+YIG+M1Z9lU2G6MhWmRFdftuD2eD5eQp9PqS3Qo7ngQgL4pcQpRBCaGix wjoQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b="BuN5xyW/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id m6-20020a17090a7f8600b0025027e0ad3dsi1848585pjl.81.2023.07.27.14.10.44; Thu, 27 Jul 2023 14:10:57 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b="BuN5xyW/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231938AbjG0Uq4 (ORCPT + 99 others); Thu, 27 Jul 2023 16:46:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34396 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231773AbjG0Uqy (ORCPT ); Thu, 27 Jul 2023 16:46:54 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A99882D47 for ; Thu, 27 Jul 2023 13:46:35 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id 5b1f17b1804b1-3fde57684d7so14158905e9.2 for ; Thu, 27 Jul 2023 13:46:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690490794; x=1691095594; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=R/2kTGP5bmExpEckq+8Rz1vqcaG4uRUICK5uorcV/EQ=; b=BuN5xyW/6weGNXRlIjeHAIrAgftiJaDdwcAFdinE25QOj+gk0KxhfbPi6NDrWScFMY VMJS1fp/fwlHj4io7fNeedGnL3Vq6LPtuewysLC8Sjo6xuSSwb5C7A3QVA6pUp3BuGal peUgBBXdkLuyAG0r0MtOAu8bJxWihdBSiUkx7+JUoxmU92FDCSc3+a203D99Te3djvrE GFjNEAyNZJnsGcpwIVLN7dd8/TClS2qfeDsn0scV9uZaGKj9iuXLvZP5q+GVR9jZNOl4 uoPcfh3+yJsGI/CKwLu45iuV0qqJWtVosbUaOpv+ylvrytQ2cUfc7YMZnclJk3oiesVt HmMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690490794; x=1691095594; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=R/2kTGP5bmExpEckq+8Rz1vqcaG4uRUICK5uorcV/EQ=; b=frvOzM0GhjlH7wz+Y2s9u7rO5iDv1IBP7Y94bM3QteHwrnIsXQ3p6kq83SxTpfhtWF DEhetDq/R3iPTwmupCO5f6HuxsUf91bs6qtb9VXP5d+7nsCjW9810GM/Ci0zf26t8CrS hSvFdZ0tTKysOaR41PcaMocLtbSlSFyYsvSCzlxCGm7mJMN8MervEdztWpjN3TrPzfEu nVCDdAuOUYrJ1IBI9bNzxaeNZZKN2l+Q2cfmg46NylWd3Q+TQ6Z09DFQTDM4xErgXCWx eJCj+4tAdwZope0VQOBcL/+zHCzEAr2LRrA5yIWn1v7Hm60VhD4SPlVTTR5HxbHO2tWM jW6Q== X-Gm-Message-State: ABy/qLarESVAETB2HGzxazjrEN7FD3r3QL7i1pd9WNo/Vm7xael1K7/6 plbcrkQwKm5ZLpTElt6/qdNwjw== X-Received: by 2002:a05:600c:21cf:b0:3fd:ee50:d6a4 with SMTP id x15-20020a05600c21cf00b003fdee50d6a4mr141098wmj.17.1690490794119; Thu, 27 Jul 2023 13:46:34 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:7e3e:db9e:70fa:9ccb]) by smtp.gmail.com with ESMTPSA id k1-20020a7bc401000000b003fc06169abdsm2701400wmi.2.2023.07.27.13.46.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Jul 2023 13:46:33 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v1 3/6] memblock: add parameter to memblock_setclr_flag for selecting memblock_type Date: Thu, 27 Jul 2023 21:46:21 +0100 Message-Id: <20230727204624.1942372-4-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230727204624.1942372-1-usama.arif@bytedance.com> References: <20230727204624.1942372-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772609609373708739 X-GMAIL-MSGID: 1772609609373708739 This is in preparation for setting flags (for e.g. to not initialize struct pages) on reserved memory region. Signed-off-by: Usama Arif --- mm/memblock.c | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/mm/memblock.c b/mm/memblock.c index f9e61e565a53..4fd431d16ef2 100644 --- a/mm/memblock.c +++ b/mm/memblock.c @@ -897,11 +897,16 @@ int __init_memblock memblock_physmem_add(phys_addr_t base, phys_addr_t size) * Return: 0 on success, -errno on failure. */ static int __init_memblock memblock_setclr_flag(phys_addr_t base, - phys_addr_t size, int set, int flag) + phys_addr_t size, int set, int flag, bool reserved) { - struct memblock_type *type = &memblock.memory; + struct memblock_type *type; int i, ret, start_rgn, end_rgn; + if (reserved) + type = &memblock.reserved; + else + type = &memblock.memory; + ret = memblock_isolate_range(type, base, size, &start_rgn, &end_rgn); if (ret) return ret; @@ -928,7 +933,7 @@ static int __init_memblock memblock_setclr_flag(phys_addr_t base, */ int __init_memblock memblock_mark_hotplug(phys_addr_t base, phys_addr_t size) { - return memblock_setclr_flag(base, size, 1, MEMBLOCK_HOTPLUG); + return memblock_setclr_flag(base, size, 1, MEMBLOCK_HOTPLUG, 0); } /** @@ -940,7 +945,7 @@ int __init_memblock memblock_mark_hotplug(phys_addr_t base, phys_addr_t size) */ int __init_memblock memblock_clear_hotplug(phys_addr_t base, phys_addr_t size) { - return memblock_setclr_flag(base, size, 0, MEMBLOCK_HOTPLUG); + return memblock_setclr_flag(base, size, 0, MEMBLOCK_HOTPLUG, 0); } /** @@ -957,7 +962,7 @@ int __init_memblock memblock_mark_mirror(phys_addr_t base, phys_addr_t size) system_has_some_mirror = true; - return memblock_setclr_flag(base, size, 1, MEMBLOCK_MIRROR); + return memblock_setclr_flag(base, size, 1, MEMBLOCK_MIRROR, 0); } /** @@ -977,7 +982,7 @@ int __init_memblock memblock_mark_mirror(phys_addr_t base, phys_addr_t size) */ int __init_memblock memblock_mark_nomap(phys_addr_t base, phys_addr_t size) { - return memblock_setclr_flag(base, size, 1, MEMBLOCK_NOMAP); + return memblock_setclr_flag(base, size, 1, MEMBLOCK_NOMAP, 0); } /** @@ -989,7 +994,7 @@ int __init_memblock memblock_mark_nomap(phys_addr_t base, phys_addr_t size) */ int __init_memblock memblock_clear_nomap(phys_addr_t base, phys_addr_t size) { - return memblock_setclr_flag(base, size, 0, MEMBLOCK_NOMAP); + return memblock_setclr_flag(base, size, 0, MEMBLOCK_NOMAP, 0); } static bool should_skip_region(struct memblock_type *type, From patchwork Thu Jul 27 20:46:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 127185 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp14035vqg; Thu, 27 Jul 2023 14:11:03 -0700 (PDT) X-Google-Smtp-Source: APBJJlGE+rvRpFsOL042M+AsQdRzI9brDIyScqBTX8LwKRspiPWnH3jUfu/WOEnQb/o/J63yqprB X-Received: by 2002:a17:902:ab5a:b0:1b8:ae12:5610 with SMTP id ij26-20020a170902ab5a00b001b8ae125610mr443664plb.7.1690492263549; Thu, 27 Jul 2023 14:11:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690492263; cv=none; d=google.com; s=arc-20160816; b=OSorLCkltWcIGc6wFkfPda+C6ExujIV6+2LxKFlPmJdQnLVtPdu5dB56fSkyzaIJ9H dWHumEgEbmyPqhOANnQqsWTkHCijOizfgi49ldLI1sCGMjPiTVtZPbkd3br46KuJw+yG ICGkjPVM8cRw4Xf4kaDlu0pK8RQX9i7Mpjc/fsXjTkbt9dxy5GHgZ6/9wkGQEOaCdpYX fXaYmOfqcp6NXJ74oU/7udT5WAMO3yeV0hPAcLTeDuvbjErucVnKg2uF6sagh2dbk6LI QHe9NNjTJRlg/5vb9WDUel+Ncx/1AmOLyNR1rj/GWzU7wcqglwF12wEPhYSF9v58pIL2 txwg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=JQe+LnlORYwelNDP2OagFjwF2/T/1IMvjhocyHCh0HM=; fh=6ywE3cadAEHt3CqWKbi2najT+v0c7si89uhbZ0SfPzA=; b=Hduycj+0F6MrP/4fzfSAJrCgHxbi7YUKcwZ9KbCLotGAh6gMe7WNO++GpKHn0hLmfE 468LtcKC81MI63IjI4nZ0dnQe2aKXRZEOPdASolM04LcdvL1ey8dulp63rnLVhK3qK2o 5o3Vo3r3lGpQvMbsxCB5C8eJv4aWFIAox5Ao1f0MQxlN7239n3Tfi7d4ZWNRlY3asTid kwHt8k7DmQ6Agz6H0HAu1kFzaRxZ52LHzjToU9tZcxXC21Fnx2Fn2STCyJs/f/xKMD4I v2mjlJ8R19e3SaYGTsI8MATqv44mV9d5DR+plK3uVMm9Mewr2f33/dMmN0GxyeVyegFQ zZrA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=gVT8uA85; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k11-20020a170902c40b00b001aafb271d13si1926651plk.235.2023.07.27.14.10.50; Thu, 27 Jul 2023 14:11:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=gVT8uA85; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232849AbjG0UrN (ORCPT + 99 others); Thu, 27 Jul 2023 16:47:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34410 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232213AbjG0UrB (ORCPT ); Thu, 27 Jul 2023 16:47:01 -0400 Received: from mail-lj1-x234.google.com (mail-lj1-x234.google.com [IPv6:2a00:1450:4864:20::234]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B87852D45 for ; Thu, 27 Jul 2023 13:46:36 -0700 (PDT) Received: by mail-lj1-x234.google.com with SMTP id 38308e7fff4ca-2b701e1ca63so21959931fa.1 for ; Thu, 27 Jul 2023 13:46:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690490795; x=1691095595; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JQe+LnlORYwelNDP2OagFjwF2/T/1IMvjhocyHCh0HM=; b=gVT8uA85Q/2RybadFL1WMGqwI8huoiPfSEh3Ga80yhJTkx2wqYddCczcvHucvrLvi3 8LyY62f1ojKZNCIJ1RJJ6AJm/H/D5kfULuGeQgjkEhp/sdQz5z18I7dL/szLmlIfJpYD u61EVQdgSt8wbKxI6DB5hsAcwBuLB2hgvn0jrCvRww9Z0lGEXVNLysQBc2Fe36p46uCT jl2OXbNIY2R4lV5NY2Ex7WgCxOm/ALG64TE59hb/8mRc6i3LDTDk1ah1xfi+hOTG8MYd 4LLgZYdnHkKk+A17Ls6+GXL7nzPHfRkZeRmwu0Uv3hqpWoCBuyANT80yrLWtGgfm1gge gtcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690490795; x=1691095595; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JQe+LnlORYwelNDP2OagFjwF2/T/1IMvjhocyHCh0HM=; b=fIwHg8/52HS2L+/h6TlO/UClI7myuMGNqms80m595MZbW/ka/rULtjlJm4N8KnXJ6m MhRGlKPptJGbH58GvKiXwHeggjCEFm95YAfcP9grwnmuc+c5qI6sYJgJhHqpa2cmRV0x l/5ZsxvA5tnDIuClAJOW54O1Ol2/Ry8ozTb+V5+J/SB+cJavogtQUv+vY0IQDCzcZZA/ 4+xfIMcAg7ABkXtU5i5bHG5psl5Ua+7d5syCXILErkGSPWnsPzfNDFIWPLJaWYQgsP1A KjdR1fr/KgwqdX6DnS0my2vIQDb6OFUCQ9hOhXxwFEXikMyPez0jcxDQZGqJ780RhOIB V28A== X-Gm-Message-State: ABy/qLZe596ymFhynY4Aqp2Gw3heu6nQkznfmI9tmOT7B04RD0ji9BKv d49NuY0uYgUsPPhUCy1XZ6V6fw== X-Received: by 2002:a2e:3003:0:b0:2b5:974f:385 with SMTP id w3-20020a2e3003000000b002b5974f0385mr129660ljw.9.1690490795035; Thu, 27 Jul 2023 13:46:35 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:7e3e:db9e:70fa:9ccb]) by smtp.gmail.com with ESMTPSA id k1-20020a7bc401000000b003fc06169abdsm2701400wmi.2.2023.07.27.13.46.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Jul 2023 13:46:34 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v1 4/6] memblock: introduce MEMBLOCK_RSRV_NOINIT flag Date: Thu, 27 Jul 2023 21:46:22 +0100 Message-Id: <20230727204624.1942372-5-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230727204624.1942372-1-usama.arif@bytedance.com> References: <20230727204624.1942372-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772609615761804595 X-GMAIL-MSGID: 1772609615761804595 For reserved memory regions marked with this flag, reserve_bootmem_region is not called during memmap_init_reserved_pages. This can be used to avoid struct page initialization for regions which won't need them, for e.g. hugepages with HVO enabled. Signed-off-by: Usama Arif --- include/linux/memblock.h | 7 +++++++ mm/memblock.c | 32 ++++++++++++++++++++++++++------ 2 files changed, 33 insertions(+), 6 deletions(-) diff --git a/include/linux/memblock.h b/include/linux/memblock.h index f71ff9f0ec81..7f9d06c08592 100644 --- a/include/linux/memblock.h +++ b/include/linux/memblock.h @@ -47,6 +47,7 @@ enum memblock_flags { MEMBLOCK_MIRROR = 0x2, /* mirrored region */ MEMBLOCK_NOMAP = 0x4, /* don't add to kernel direct mapping */ MEMBLOCK_DRIVER_MANAGED = 0x8, /* always detected via a driver */ + MEMBLOCK_RSRV_NOINIT = 0x10, /* don't call reserve_bootmem_region for this region */ }; /** @@ -125,6 +126,7 @@ int memblock_clear_hotplug(phys_addr_t base, phys_addr_t size); int memblock_mark_mirror(phys_addr_t base, phys_addr_t size); int memblock_mark_nomap(phys_addr_t base, phys_addr_t size); int memblock_clear_nomap(phys_addr_t base, phys_addr_t size); +int memblock_rsrv_mark_noinit(phys_addr_t base, phys_addr_t size); void memblock_free_all(void); void memblock_free(void *ptr, size_t size); @@ -259,6 +261,11 @@ static inline bool memblock_is_nomap(struct memblock_region *m) return m->flags & MEMBLOCK_NOMAP; } +static inline bool memblock_is_noinit(struct memblock_region *m) +{ + return m->flags & MEMBLOCK_RSRV_NOINIT; +} + static inline bool memblock_is_driver_managed(struct memblock_region *m) { return m->flags & MEMBLOCK_DRIVER_MANAGED; diff --git a/mm/memblock.c b/mm/memblock.c index 4fd431d16ef2..3a15708af3b6 100644 --- a/mm/memblock.c +++ b/mm/memblock.c @@ -997,6 +997,22 @@ int __init_memblock memblock_clear_nomap(phys_addr_t base, phys_addr_t size) return memblock_setclr_flag(base, size, 0, MEMBLOCK_NOMAP, 0); } +/** + * memblock_rsrv_mark_noinit - Mark a reserved memory region with flag MEMBLOCK_RSRV_NOINIT. + * @base: the base phys addr of the region + * @size: the size of the region + * + * For memory regions marked with %MEMBLOCK_RSRV_NOINIT, reserve_bootmem_region + * is not called during memmap_init_reserved_pages, hence struct pages are not + * initialized for this region. + * + * Return: 0 on success, -errno on failure. + */ +int __init_memblock memblock_rsrv_mark_noinit(phys_addr_t base, phys_addr_t size) +{ + return memblock_setclr_flag(base, size, 1, MEMBLOCK_RSRV_NOINIT, 1); +} + static bool should_skip_region(struct memblock_type *type, struct memblock_region *m, int nid, int flags) @@ -2113,13 +2129,17 @@ static void __init memmap_init_reserved_pages(void) memblock_set_node(start, end, &memblock.reserved, nid); } - /* initialize struct pages for the reserved regions */ + /* + * initialize struct pages for reserved regions that don't have + * the MEMBLOCK_RSRV_NOINIT flag set + */ for_each_reserved_mem_region(region) { - nid = memblock_get_region_node(region); - start = region->base; - end = start + region->size; - - reserve_bootmem_region(start, end, nid); + if (!memblock_is_noinit(region)) { + nid = memblock_get_region_node(region); + start = region->base; + end = start + region->size; + reserve_bootmem_region(start, end, nid); + } } } From patchwork Thu Jul 27 20:46:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 127182 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp13954vqg; Thu, 27 Jul 2023 14:10:52 -0700 (PDT) X-Google-Smtp-Source: APBJJlFrFe/qhgSJO46cjR9kgKLKRbSh30qlrheJWXCiagzG+xpCFIgcX6flkXYqwV4BQOtioQW4 X-Received: by 2002:a05:6a00:2d83:b0:682:616a:f910 with SMTP id fb3-20020a056a002d8300b00682616af910mr433729pfb.20.1690492252372; Thu, 27 Jul 2023 14:10:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690492252; cv=none; d=google.com; s=arc-20160816; b=bHAuuWDwUD0nvqbM6Y/+XZm0YG+CEFfjy61WeYf9TUCDVbWm9EXLPtZR3I3Fi46xEq rJhpFGfxWsUjWBBIWv5FhvTf55npjqbsf17kZq9+f8AA17b385QAxMg1thOpQSBs67WX bCUlx/fWYrTDpcwGcC2s1fSBtagJQ93CjAKHwQ1yZ5N1dAlNBA3c3PnI1cv6QYN+29Hp v5d5AUjn197u+Uz4Dw8XMzk68Du24q4L0/W6z7AoDGnJ0lb6TbYLy7guwwfNCd5LrmIu jm7EV50qO5MAGhiXrzkbOcLgQx6Z3MjRTjBdvEcQqDicXKZaFq/9QLHLEMSxIL9RH8b/ 9u+A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=cuTqSOqGGusC4+STWjmE7AEvtPXrFRozZqvR7YJO9tM=; fh=6ywE3cadAEHt3CqWKbi2najT+v0c7si89uhbZ0SfPzA=; b=Zu5FINKsol5YvsHTUyKydBd2rFPX3ccSshrkSbHymNQ32/Su7Jav1RHX5Y7TR45VAA UIPW8au5CajRSwvu4u7nHC0xvvCNl+DQzl4kTyVSYAqUgRrU1P89yqRc54bb0fiGI4HD YBDsXGd2ffNBWD4einTMTAFWGgL0BcyEBIpvbFD+qVW2XozsJ/91r+3TJyLv4hLGmSWh XR4Vbha4qcspPdr4qBvgEKRhg2RQoNcxeOuQZDW60dvWYqkK9TX918BqhydkGDOsEBbZ bQfozjype/nCZBe6fAevhWhbfh366nV+vAGr/mNplZiFndVo49Py9vLNPa53MGevlz3M 1bVw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=MYvinfuv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h9-20020a056a001a4900b006546d0d5832si1922423pfv.183.2023.07.27.14.10.39; Thu, 27 Jul 2023 14:10:52 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=MYvinfuv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231781AbjG0Uqj (ORCPT + 99 others); Thu, 27 Jul 2023 16:46:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34412 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231773AbjG0Uqi (ORCPT ); Thu, 27 Jul 2023 16:46:38 -0400 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 492FC2D5B for ; Thu, 27 Jul 2023 13:46:37 -0700 (PDT) Received: by mail-wm1-x32a.google.com with SMTP id 5b1f17b1804b1-3fbc5d5746cso15093815e9.2 for ; Thu, 27 Jul 2023 13:46:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690490796; x=1691095596; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cuTqSOqGGusC4+STWjmE7AEvtPXrFRozZqvR7YJO9tM=; b=MYvinfuvPzSR6QZNB+kO061wB6tpem3LnUdQ68duY2x9Lf58oM1yxzxSQNGFKohFk/ EI1cawtc+AbWjfvWpBvppmYxUriHX3hcCjpQz4l+rQHliweF9u1TU856lSFvZen+b0ac /Iqd2lMerRuef36wCLT08cM3aet4zD4TwG6rd3HjnQ3e8jECc7SC98cUtcttUQ97dfkH 2TCsQq6Rp2S9nECR41e32NdAk0O1RVp9RR8BBusJoiRVB5fSCtegpCAa22SLN6tNpX0M IR0L8UXTdcMg4LsESCJXNbMtMfHWQw91m2aMdBO9Amm4VRFmS4e2CIzdeIHkjpK5IJLK fqUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690490796; x=1691095596; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cuTqSOqGGusC4+STWjmE7AEvtPXrFRozZqvR7YJO9tM=; b=C3UXuym9yAG/L2o+sGCamZRSgWaxLiW2A6fEkpaULf3yf4lgJ63cmZken26jliHGAn 54PBPRKt3Oyyt2LaaFPhuEnqzvCxkXKmsTwOY0oGmga9mspWDXZx7vmXxuO4Y55/A89o VdXuknTwmrEsS8SfMq1WnMQDv2YiKPsF56BapUZ4C0SVQ5kxcDGWfpBpPbb49mfkb49k X59gYpBlnb2m4Yji1S2ffCljCdoCRIkBau9pqpEwXfTvllsHD0C77yghxofaUgr7/DCb kK3GbTDJcBQp8fnJp0Qm6PKclVrgpRj1AFALVLF//Q8EYez5I6784AidfgUyes1hPk18 L21A== X-Gm-Message-State: ABy/qLa0uNh9y8xXTl6nhrrKdvqtAsCvDHWEKAp1aAkXE/0YuphmgpjZ kSz/0WZJ7UXPN0g8oJ9gOpOynA== X-Received: by 2002:a7b:ce16:0:b0:3f9:b748:ff3f with SMTP id m22-20020a7bce16000000b003f9b748ff3fmr151792wmc.1.1690490795841; Thu, 27 Jul 2023 13:46:35 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:7e3e:db9e:70fa:9ccb]) by smtp.gmail.com with ESMTPSA id k1-20020a7bc401000000b003fc06169abdsm2701400wmi.2.2023.07.27.13.46.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Jul 2023 13:46:35 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v1 5/6] mm: move allocation of gigantic hstates to the start of mm_core_init Date: Thu, 27 Jul 2023 21:46:23 +0100 Message-Id: <20230727204624.1942372-6-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230727204624.1942372-1-usama.arif@bytedance.com> References: <20230727204624.1942372-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772609603845474029 X-GMAIL-MSGID: 1772609603845474029 Whether the initialization of tail struct pages of a hugepage happens or not will become dependent on the commandline parameter hugetlb_free_vmemmap in the future. Hence, hugetlb_hstate_alloc_pages needs to be after command line parameters are parsed and the start of mm_core_init is a good point. Signed-off-by: Usama Arif --- include/linux/hugetlb.h | 1 + mm/hugetlb.c | 18 ++++++++++-------- mm/mm_init.c | 4 ++++ 3 files changed, 15 insertions(+), 8 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index ca3c8e10f24a..2b20553deef3 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -1271,4 +1271,5 @@ hugetlb_walk(struct vm_area_struct *vma, unsigned long addr, unsigned long sz) return huge_pte_offset(vma->vm_mm, addr, sz); } +void __init hugetlb_hstate_alloc_gigantic_pages(void); #endif /* _LINUX_HUGETLB_H */ diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 58cf5978bee1..c1fcf2af591a 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4418,14 +4418,6 @@ static int __init hugepages_setup(char *s) } } - /* - * Global state is always initialized later in hugetlb_init. - * But we need to allocate gigantic hstates here early to still - * use the bootmem allocator. - */ - if (hugetlb_max_hstate && hstate_is_gigantic(parsed_hstate)) - hugetlb_hstate_alloc_pages(parsed_hstate); - last_mhp = mhp; return 1; @@ -4437,6 +4429,16 @@ static int __init hugepages_setup(char *s) } __setup("hugepages=", hugepages_setup); +void __init hugetlb_hstate_alloc_gigantic_pages(void) +{ + int i; + + for (i = 0; i < HUGE_MAX_HSTATE; i++) { + if (hstate_is_gigantic(&hstates[i])) + hugetlb_hstate_alloc_pages(&hstates[i]); + } +} + /* * hugepagesz command line processing * A specific huge page size can only be specified once with hugepagesz. diff --git a/mm/mm_init.c b/mm/mm_init.c index a1963c3322af..5585c66c3c42 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -26,6 +26,7 @@ #include #include #include +#include #include "internal.h" #include "slab.h" #include "shuffle.h" @@ -2768,6 +2769,9 @@ static void __init mem_init_print_info(void) */ void __init mm_core_init(void) { +#ifdef CONFIG_HUGETLBFS + hugetlb_hstate_alloc_gigantic_pages(); +#endif /* Initializations relying on SMP setup */ build_all_zonelists(NULL); page_alloc_init_cpuhp(); From patchwork Thu Jul 27 20:46:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 127193 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp25925vqg; Thu, 27 Jul 2023 14:40:28 -0700 (PDT) X-Google-Smtp-Source: APBJJlEq9T76pQuEMNqdzB/s8eZCK9CcOFZRc+BSJ2O/oaxuJmHsfxzXPkAksgqsqUbgLDzUJ9bz X-Received: by 2002:a05:6a00:ac4:b0:682:f48a:726a with SMTP id c4-20020a056a000ac400b00682f48a726amr438103pfl.25.1690494028325; Thu, 27 Jul 2023 14:40:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690494028; cv=none; d=google.com; s=arc-20160816; b=R4TC29kHcZb1KriHU5OwAlM3+2w0g0m2vGdlNZ+5BlQ+Zcdu8PjbCZ7TUK7NlGiGU8 wLBm7d8xYX+tVHNQU3CmoxWKL2IUKOBeBbnATm33sEhwndTgf5eRahd6Uegwe6yR/IvC 1G2I2E/eeqGxV4gKcBekIkueT27WmAtEpKRoegzGEN6WfqHcRp7GJfZTQrIXzpZnnOrr Q1fAecO0off6dajj6ka7sGZYuHq2X93zJbhAFOf/Lm65ewWy0/iI0S3cHdB1E5Oc8nql /zhGjgWdxK6XvnsPBEVuCF1yh0hqlki0fEnV6gxUzVwSa6hHFFmaSEoinCkSmwakBDwQ N5Kw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ckgW/P+ejqydrp6JQqQrrU5HjkGQNsndgwMWGYDB/pc=; fh=6ywE3cadAEHt3CqWKbi2najT+v0c7si89uhbZ0SfPzA=; b=W/Xjl+9hV1mbMmGORpG36lMiVwR4Ywc5YVkvs1B4Q+5ZVsxFtW/O8cUndh9ZsSV7Ae xdzx3/Q+LgQN4Tjv60Po74FptTTds5pevbdSogJiWwzRz9uTdEewHbjWXmKlkqGWhoyv a9eSVChILBrdeCG1oA0LZ6oN8ZnYbFWct+X+Y+ZhjaVtcSdqM5hxfi8r3+IYGy4dLxki oYPYqsJv/RfXWlpUGgeVkI8N+nZF/HOLGVtXhP5/stugleP40QnJrtnlgo3T3Quk9AWG 2lCoHc6o35Zo93QPT2+jB8LEE+BdmYpWV3nE3aERW/XIl8eoq7AN4EPKIDZRvCUIPJwE /d9Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=MHW+96et; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id m12-20020a056a00080c00b00686bbfbdb6dsi1954979pfk.169.2023.07.27.14.40.14; Thu, 27 Jul 2023 14:40:28 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=MHW+96et; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229509AbjG0UrK (ORCPT + 99 others); Thu, 27 Jul 2023 16:47:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34414 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232217AbjG0UrB (ORCPT ); Thu, 27 Jul 2023 16:47:01 -0400 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 386382D60 for ; Thu, 27 Jul 2023 13:46:38 -0700 (PDT) Received: by mail-wr1-x42b.google.com with SMTP id ffacd0b85a97d-3174aac120aso1423313f8f.2 for ; Thu, 27 Jul 2023 13:46:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690490797; x=1691095597; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ckgW/P+ejqydrp6JQqQrrU5HjkGQNsndgwMWGYDB/pc=; b=MHW+96etwxTrHcysNxNv4Jt5Keu/21p5ZeNuDIbQx9GnIkEA1kq7sTTqw6Q/8BayM4 1Acht/ZPiVv8tOChy+mgBTfHG6ixCBKorlj2dJl7YTk0/sTWgbMWE9WJCFXy+OQ9Iijl F797zrYkH1WHN8+kZpWGtO4V6mpRCt3ijeyEEL/x0uHMVkx2JVs2E0JF144vy0BqQwCm brRLxremJQxfBWLsYq+wA+OhATOq4v3hK/HHE03k794S7TLjgAEkUPQNfZuGNCFBakRv 3NI8Jepy+Nx2mW0XqXn6WjNkB/tP3+1AiL7EGltSxqblDgE7Db3FfhwIHlnqEs7zbPpH c/9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690490797; x=1691095597; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ckgW/P+ejqydrp6JQqQrrU5HjkGQNsndgwMWGYDB/pc=; b=HcWvJSbMTkhtzPPMgy1c8PJUZ1VeaGPrrkHY2pYet46UdygVtss9oCzPVKcvTe54Of GFptEERuCyulR0+3BQnd+Xi5c4qTwRUi6vZa9VSV7GSrOhjrkUDsiy8m+l+TLKcMlw/9 04VlTdOcJlrKfOSspUJbWsVWzC084xNSbAhqxYIanqCr3UhYlISQrMwPStMhqRh7NX/8 CiSSsc/NV4fZxhr1qNJYtzYQi8BHZHDqd6TTvHteeJJsXP5CrCiGEHTzUIaTtsOdzXl5 0F3GHHss1U4MpAzRPWUrpSePHQCqUtwcRQYg/xedRpbnbqeiCKsqES12+0aQ/twvGsVw nkjQ== X-Gm-Message-State: ABy/qLbewnaQyyodr5P7r8U4wej9aBuz3Z54VvVNXpAzqORGF8rV2yee tUPAXQWL5CiH5K8vo9HC0Y1u0A== X-Received: by 2002:adf:db44:0:b0:314:34dd:aaec with SMTP id f4-20020adfdb44000000b0031434ddaaecmr229166wrj.8.1690490796779; Thu, 27 Jul 2023 13:46:36 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:7e3e:db9e:70fa:9ccb]) by smtp.gmail.com with ESMTPSA id k1-20020a7bc401000000b003fc06169abdsm2701400wmi.2.2023.07.27.13.46.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Jul 2023 13:46:36 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v1 6/6] mm: hugetlb: Skip initialization of struct pages freed later by HVO Date: Thu, 27 Jul 2023 21:46:24 +0100 Message-Id: <20230727204624.1942372-7-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230727204624.1942372-1-usama.arif@bytedance.com> References: <20230727204624.1942372-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772611466458441448 X-GMAIL-MSGID: 1772611466458441448 This is done by marking the region for which to skip initialization with the MEMBLOCK_RSRV_NOINIT flag. If the region is for hugepages and if HVO is enabled, then those struct pages which will be freed later don't need to be initialized. This can save significant time when a large number of hugepages are allocated at boot time. HUGETLB_VMEMMAP_RESERVE_SIZE struct pages at the start of hugepage still need to be initialized. Signed-off-by: Usama Arif --- mm/hugetlb.c | 21 +++++++++++++++++++++ mm/hugetlb_vmemmap.c | 2 +- mm/hugetlb_vmemmap.h | 3 +++ 3 files changed, 25 insertions(+), 1 deletion(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index c1fcf2af591a..bb2b12f41026 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3166,6 +3166,9 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) { struct huge_bootmem_page *m = NULL; /* initialize for clang */ int nr_nodes, node; + phys_addr_t hugetlb_vmemmap_reserve_size = + HUGETLB_VMEMMAP_RESERVE_SIZE * sizeof(struct page); + phys_addr_t noinit_base; /* do node specific alloc */ if (nid != NUMA_NO_NODE) { @@ -3173,6 +3176,15 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) 0, MEMBLOCK_ALLOC_ACCESSIBLE, nid); if (!m) return 0; + + if (vmemmap_optimize_enabled && hugetlb_vmemmap_optimizable(h)) { + noinit_base = virt_to_phys( + (void *)((phys_addr_t) m + hugetlb_vmemmap_reserve_size)); + memblock_rsrv_mark_noinit( + noinit_base, + huge_page_size(h) - hugetlb_vmemmap_reserve_size); + } + goto found; } /* allocate from next node when distributing huge pages */ @@ -3187,6 +3199,15 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) */ if (!m) return 0; + + if (vmemmap_optimize_enabled && hugetlb_vmemmap_optimizable(h)) { + noinit_base = virt_to_phys( + (void *)((phys_addr_t) m + hugetlb_vmemmap_reserve_size)); + memblock_rsrv_mark_noinit( + noinit_base, + huge_page_size(h) - hugetlb_vmemmap_reserve_size); + } + goto found; } diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index bdf750a4786b..b5b7834e0f42 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -443,7 +443,7 @@ static int vmemmap_remap_alloc(unsigned long start, unsigned long end, DEFINE_STATIC_KEY_FALSE(hugetlb_optimize_vmemmap_key); EXPORT_SYMBOL(hugetlb_optimize_vmemmap_key); -static bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON); +bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON); core_param(hugetlb_free_vmemmap, vmemmap_optimize_enabled, bool, 0); /** diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 07555d2dc0cb..cb5171abe683 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -64,4 +64,7 @@ static inline bool hugetlb_vmemmap_optimizable(const struct hstate *h) { return hugetlb_vmemmap_optimizable_size(h) != 0; } + +extern bool vmemmap_optimize_enabled; + #endif /* _LINUX_HUGETLB_VMEMMAP_H */