From patchwork Sun Jul 30 15:16:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 128240 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp1563197vqg; Sun, 30 Jul 2023 08:59:50 -0700 (PDT) X-Google-Smtp-Source: APBJJlE1FjEBCJQiR2kw+diADFeuee80ucLcePbfGokI+l7X5c1TdJ7O5slMrsiQlYuU9nh4s5oS X-Received: by 2002:a05:6512:280d:b0:4f1:3d7d:409e with SMTP id cf13-20020a056512280d00b004f13d7d409emr5505331lfb.0.1690732789967; Sun, 30 Jul 2023 08:59:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690732789; cv=none; d=google.com; s=arc-20160816; b=za9ZkaOTKU3cERd5C6QTBkrAvNTA8/PRxg+AynWwjP7SqiJGGqouAwZlGs5JjvfPU7 AAh/hW+cComTqpzma84GGA/OziRJN7qKzhMEsGHPY58jifyAnNmKgN8HkbLJ0vumIuwH DZuyr00Na4xQMU5RlO/96fLje9RWe/xmwV0VixKIpif1qsuphEKQoKRQxkabUVcpRz+0 m+DFC27/a9+vf7dzmG1iOsvUsEebLzXdqO9xiUmXTpe44aE7N5q7tsIrb2+Vydlbzc3H 6itaTPaCVZTPemDDtjhv5d+8JU0THDBKDSxn5+z/yFw2t6XeO89MsIgykjMylFHav13R fWkA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=P5VE1b09XGrn3hjFzsE3qaVui4TN6DlsvAzQ6XWShV0=; fh=6ywE3cadAEHt3CqWKbi2najT+v0c7si89uhbZ0SfPzA=; b=Lwz7YEYNcI9a0G4opLSfgfZ0I7XmMzybHc3Dkk6MoTJEu9MYMLc26HCsLlDqkfTNBL pxVAyJKJqS/Xh4eERSZ02j5ibQRng0whYAwHXRkA1NCeLs7rBHpyYf5Q5d3AZu1imKKf 2G8BYcGzieFL040+PXTXImqAdfVSs3lyDTTP0bpT43fcP+uIu/ahzi6+r8L8XZElEB3C 1HXP51kKlPyNEA+KyjuZZ7/zzx4OLq4q9IcpsUAIaWscC++/eo6R7bqnZCPXKEP4Ktig SO0dhfmtsyBBLifJAA/6IjLxh4njZMs47I8VLr7uN6ytX1GxiAD+U1tNFyoClvJAe62H 8QcA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=LhfXuIze; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q2-20020aa7d442000000b0052228b411e4si629390edr.203.2023.07.30.08.59.20; Sun, 30 Jul 2023 08:59:49 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=LhfXuIze; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230033AbjG3PQl (ORCPT + 99 others); Sun, 30 Jul 2023 11:16:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50614 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230024AbjG3PQi (ORCPT ); Sun, 30 Jul 2023 11:16:38 -0400 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1A5AA1713 for ; Sun, 30 Jul 2023 08:16:12 -0700 (PDT) Received: by mail-wm1-x334.google.com with SMTP id 5b1f17b1804b1-3fbc59de0e2so34772095e9.3 for ; Sun, 30 Jul 2023 08:16:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690730170; x=1691334970; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=P5VE1b09XGrn3hjFzsE3qaVui4TN6DlsvAzQ6XWShV0=; b=LhfXuIzeZE5fNFxToBAlvDQA5EoZbK+3WEM+AjQQxsQ0JxOdwHbtoU7CSVU9Y3ua/I YUpJ5p9ziGSrXpohlrNQkGF+742Gm7k1GzsA5f33b42pcc1fc/iAwrWonfQ7MYbv1q7h YVGa2uWWvFfRDEkTBXC8BuX87YIa3rtNxtLUF7dmSJ7zSlON6lGYhX3vs8+xJgQ7inEi v2WUTfER2YUR5IubnbmBUcgeUw1db17u0y+MPFkcMPvVpO4iH/iea96utluPPBbXOwxl JtX5YfmoepWtGVwANwedcUUAaZIqjKeijQsq21YfiLsa2xpb4FFh20dY0yu/BPDgxfy1 opbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690730170; x=1691334970; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=P5VE1b09XGrn3hjFzsE3qaVui4TN6DlsvAzQ6XWShV0=; b=eBSOQhRwN3F3x+//9i/Nz3GBC5czXB1M4mGu8iHcUhxx+eIcIIs5XP12E7ScoVc1HQ UIYqYvJD8kRyjfwxOrmeFO68IRMesRqCFreSh6JixMmRPV3ajP1L7LopY/gmme5c3u9c WJzQ2Rd1FOLh5RuWBhRMEvC+Pxf2FcPVZd+ij//+2m+y9+Q3FA6aRB7iCfcqWQBU+E7A 26RV5XsNC8HC+Gktqd3w8dicV+JeaYfGl4P8o4QH+ceLDv5WmFO9Yu6XBA9LiWK5rk4B FAP/hUyMhWpKKo3+N3YgWb4PDvpl5hvUUgnKx/DCGlxXjqR0H7s8D6wXmituWBpYUy5q f7CA== X-Gm-Message-State: ABy/qLYoAHD4N70d/lLBTBbpfhcNZGD/rcKzZQb6P9N5RWnkbVtb94iX R+NrQM8RNExTlMdI+aOqZUdXpA== X-Received: by 2002:a7b:cc8c:0:b0:3fc:175:ade7 with SMTP id p12-20020a7bcc8c000000b003fc0175ade7mr5638740wma.38.1690730170432; Sun, 30 Jul 2023 08:16:10 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:eda5:aa63:ce24:dac2]) by smtp.gmail.com with ESMTPSA id f17-20020a7bcc11000000b003fd2d33ea53sm9123027wmh.14.2023.07.30.08.16.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Jul 2023 08:16:09 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v2 1/6] mm: hugetlb: Skip prep of tail pages when HVO is enabled Date: Sun, 30 Jul 2023 16:16:01 +0100 Message-Id: <20230730151606.2871391-2-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230730151606.2871391-1-usama.arif@bytedance.com> References: <20230730151606.2871391-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772861825842813955 X-GMAIL-MSGID: 1772861825842813955 When vmemmap is optimizable, it will free all the duplicated tail pages in hugetlb_vmemmap_optimize while preparing the new hugepage. Hence, there is no need to prepare them. For 1G x86 hugepages, it avoids preparing 262144 - 64 = 262080 struct pages per hugepage. The indirection of using __prep_compound_gigantic_folio is also removed, as it just creates extra functions to indicate demote which can be done with the argument. Signed-off-by: Usama Arif --- mm/hugetlb.c | 32 ++++++++++++++------------------ mm/hugetlb_vmemmap.c | 2 +- mm/hugetlb_vmemmap.h | 15 +++++++++++---- 3 files changed, 26 insertions(+), 23 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 64a3239b6407..541c07b6d60f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1942,14 +1942,23 @@ static void prep_new_hugetlb_folio(struct hstate *h, struct folio *folio, int ni spin_unlock_irq(&hugetlb_lock); } -static bool __prep_compound_gigantic_folio(struct folio *folio, - unsigned int order, bool demote) +static bool prep_compound_gigantic_folio(struct folio *folio, struct hstate *h, bool demote) { int i, j; + int order = huge_page_order(h); int nr_pages = 1 << order; struct page *p; __folio_clear_reserved(folio); + + /* + * No need to prep pages that will be freed later by hugetlb_vmemmap_optimize. + * Hence, reduce nr_pages to the pages that will be kept. + */ + if (IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP) && + vmemmap_should_optimize(h, &folio->page)) + nr_pages = HUGETLB_VMEMMAP_RESERVE_SIZE / sizeof(struct page); + for (i = 0; i < nr_pages; i++) { p = folio_page(folio, i); @@ -2019,18 +2028,6 @@ static bool __prep_compound_gigantic_folio(struct folio *folio, return false; } -static bool prep_compound_gigantic_folio(struct folio *folio, - unsigned int order) -{ - return __prep_compound_gigantic_folio(folio, order, false); -} - -static bool prep_compound_gigantic_folio_for_demote(struct folio *folio, - unsigned int order) -{ - return __prep_compound_gigantic_folio(folio, order, true); -} - /* * PageHuge() only returns true for hugetlbfs pages, but not for normal or * transparent huge pages. See the PageTransHuge() documentation for more @@ -2185,7 +2182,7 @@ static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, if (!folio) return NULL; if (hstate_is_gigantic(h)) { - if (!prep_compound_gigantic_folio(folio, huge_page_order(h))) { + if (!prep_compound_gigantic_folio(folio, h, false)) { /* * Rare failure to convert pages to compound page. * Free pages and try again - ONCE! @@ -3201,7 +3198,7 @@ static void __init gather_bootmem_prealloc(void) VM_BUG_ON(!hstate_is_gigantic(h)); WARN_ON(folio_ref_count(folio) != 1); - if (prep_compound_gigantic_folio(folio, huge_page_order(h))) { + if (prep_compound_gigantic_folio(folio, h, false)) { WARN_ON(folio_test_reserved(folio)); prep_new_hugetlb_folio(h, folio, folio_nid(folio)); free_huge_page(page); /* add to the hugepage allocator */ @@ -3624,8 +3621,7 @@ static int demote_free_hugetlb_folio(struct hstate *h, struct folio *folio) subpage = folio_page(folio, i); inner_folio = page_folio(subpage); if (hstate_is_gigantic(target_hstate)) - prep_compound_gigantic_folio_for_demote(inner_folio, - target_hstate->order); + prep_compound_gigantic_folio(inner_folio, target_hstate, true); else prep_compound_page(subpage, target_hstate->order); folio_change_private(inner_folio, NULL); diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index c2007ef5e9b0..b721e87de2b3 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -486,7 +486,7 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) } /* Return true iff a HugeTLB whose vmemmap should and can be optimized. */ -static bool vmemmap_should_optimize(const struct hstate *h, const struct page *head) +bool vmemmap_should_optimize(const struct hstate *h, const struct page *head) { if (!READ_ONCE(vmemmap_optimize_enabled)) return false; diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 25bd0e002431..3e7978a9af73 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -10,16 +10,17 @@ #define _LINUX_HUGETLB_VMEMMAP_H #include -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP -int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head); -void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head); - /* * Reserve one vmemmap page, all vmemmap addresses are mapped to it. See * Documentation/vm/vmemmap_dedup.rst. */ #define HUGETLB_VMEMMAP_RESERVE_SIZE PAGE_SIZE +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head); +void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head); +bool vmemmap_should_optimize(const struct hstate *h, const struct page *head); + static inline unsigned int hugetlb_vmemmap_size(const struct hstate *h) { return pages_per_huge_page(h) * sizeof(struct page); @@ -51,6 +52,12 @@ static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct hstate { return 0; } + +static inline bool vmemmap_should_optimize(const struct hstate *h, const struct page *head) +{ + return false; +} + #endif /* CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP */ static inline bool hugetlb_vmemmap_optimizable(const struct hstate *h) From patchwork Sun Jul 30 15:16:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 128247 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp1591412vqg; Sun, 30 Jul 2023 10:12:28 -0700 (PDT) X-Google-Smtp-Source: APBJJlEvGZ4EhdAE8T5F6pZhvvwHT5vWQeIh6jFMUG7Cx2+Fy6mtU/muXg1yKF3OvmyQrVI41tSL X-Received: by 2002:a17:906:20f:b0:973:d076:67ab with SMTP id 15-20020a170906020f00b00973d07667abmr5018422ejd.42.1690737148416; Sun, 30 Jul 2023 10:12:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690737148; cv=none; d=google.com; s=arc-20160816; b=ViDVB/pv2m+f7y1qCHEXqLGo0gXO4Nix8e5qOp9en2PdX2nY/TJNCqc/GN6rfKJMGb zjJr2vgGF1F6V8ThMjczUKmKZ2kC0YiuGj0NaKsKVTnzOYnrsnn7gB+R0VkshcblHisv 1C16GWQxHe/LLMcNqFtzEMSKeqBCq/rbsOQ0MU6OPr268bu6GmJYNzQG2CQ0J/X+RsS8 d3oYsAjBblLCuQnermDVLHDOXQg292Kk4orOGtScwl7H0d7VfPSnIaejrq6Ents1LzBj TpgJItZhJZWMPcLggDs6iDTSU8xKhlfQQJArC1xbWxhkJjYLhclsVw+ZQf55fWC0GbnJ dqQg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=VUFHsk/m/GY1eTNfZx8Py//rj4lEwu1OMjpML8i4XiA=; fh=6ywE3cadAEHt3CqWKbi2najT+v0c7si89uhbZ0SfPzA=; b=tKr3FV/VRymX+AtEjxcWbG3TdAT+3L+RUT7+kh/iqcX+3QP58qHlWgHNVvlZ0r8djG qc4VBMJz/+Yo4+I5kWt3W/E8KK9o0hR22+9NTtS8K80MItKCuuTuNYVI9RFpZaTUyjD1 HBpANpSBtRn/whJwuP7p4df5oXcqve7LIhZs/sGCdZ3GGBjtD8Zadn7B3+96sYAqKPLx 4RxTy56Dhheoh+xXFb8x6DTE9FOdD0Z4kAtMRSX+rL6lCowHQdgk6gClITWn3e08fwPV SNeb5zHjIyslcmozsrMi8l2liRPrw84HPLuXalB/1AixNt+wCSSGSVHnqsMm7gviBK7H ZqFw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=JoxH10l1; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a13-20020a170906684d00b009886470de61si5939675ejs.857.2023.07.30.10.11.57; Sun, 30 Jul 2023 10:12:28 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=JoxH10l1; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230047AbjG3PQn (ORCPT + 99 others); Sun, 30 Jul 2023 11:16:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50616 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230027AbjG3PQi (ORCPT ); Sun, 30 Jul 2023 11:16:38 -0400 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E56421716 for ; Sun, 30 Jul 2023 08:16:12 -0700 (PDT) Received: by mail-wm1-x334.google.com with SMTP id 5b1f17b1804b1-3fbc5d5742bso41888745e9.2 for ; Sun, 30 Jul 2023 08:16:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690730171; x=1691334971; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VUFHsk/m/GY1eTNfZx8Py//rj4lEwu1OMjpML8i4XiA=; b=JoxH10l13AhAr8ISmwB2i1gOf6hMfDgdag1NYZGf8eqVGmEdh4R3IE2D1prI5+wzm7 vu1QD7xvEBstHjEIbxAMlgCCO00ONBH3G5zPFMoDk4Drvk6SzML5WNLP4tfo6lnQh865 TX13dZrkIeWswvkw6xN/xmei3GAwgiUombSwNiUwcW22Vp57PvkxefJyTKdVVA+u+gzP VbGxb9p/bzQfa6R6GWeFqN5krmkkMZbJxyXakbPqdmWiMydGR5JelRYvQMAxOrQf62pU o4lWtSXohDF8KsxmkSED8zQCamat1slkfvH7Xda+Eyuy3MLasON3KHOua9umK5WY8Nob wW5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690730171; x=1691334971; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VUFHsk/m/GY1eTNfZx8Py//rj4lEwu1OMjpML8i4XiA=; b=RMg2GfdFISe3acNSNOvZ9cm0P9VWd0Zb2Ad1F5XQz4wsNUj+BoWzPGtB9Ei/1o86vC kFPBKuzBs0SIm1Lp1iyQHe0A3triHfOn6CMmdFMKqgWTayOz7W6yWqX+xQV+Fk61M/rJ R4NQ85/KvwOYHs4ilJsMpDJlCU2TL9R6ZycgsE2TiVwixWcHPlUIvxN+jdwNhHQ4Lz3B m/p7OErswrzoSyKku5JU+LUc33wgieqFX3X3P+ySmqB10+VhXwL6Pfsv5uWloqO6v37F LMFT6m8udhud/JaFiGbdnomVGAAmlARW+UznJ2wGHy5DUzC5Mgj0ZkB7MvU/4x7Df3gg x6LA== X-Gm-Message-State: ABy/qLbd6bAzhkiuORD4pY0uxbXQTzFydHg2tc7dA08gWsTyJAH1npUV KwNb+faOysaCcPHTeFG5dQepOQ== X-Received: by 2002:a7b:cbd5:0:b0:3fe:1db2:5168 with SMTP id n21-20020a7bcbd5000000b003fe1db25168mr1370994wmi.11.1690730171283; Sun, 30 Jul 2023 08:16:11 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:eda5:aa63:ce24:dac2]) by smtp.gmail.com with ESMTPSA id f17-20020a7bcc11000000b003fd2d33ea53sm9123027wmh.14.2023.07.30.08.16.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Jul 2023 08:16:10 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v2 2/6] mm: hugetlb_vmemmap: Use nid of the head page to reallocate it Date: Sun, 30 Jul 2023 16:16:02 +0100 Message-Id: <20230730151606.2871391-3-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230730151606.2871391-1-usama.arif@bytedance.com> References: <20230730151606.2871391-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772866396082123234 X-GMAIL-MSGID: 1772866396082123234 If tail page prep and initialization is skipped, then the "start" page will not contain the correct nid. Use the nid from first vmemap page. Signed-off-by: Usama Arif --- mm/hugetlb_vmemmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index b721e87de2b3..bdf750a4786b 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -324,7 +324,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, .reuse_addr = reuse, .vmemmap_pages = &vmemmap_pages, }; - int nid = page_to_nid((struct page *)start); + int nid = page_to_nid((struct page *)reuse); gfp_t gfp_mask = GFP_KERNEL | __GFP_THISNODE | __GFP_NORETRY | __GFP_NOWARN; From patchwork Sun Jul 30 15:16:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 128248 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp1591548vqg; Sun, 30 Jul 2023 10:12:52 -0700 (PDT) X-Google-Smtp-Source: APBJJlGbjHcQzz+JLky96kGUEnrk0VX/EzdcVJC9kzjiLrFygcmwsqdlpPOXYVL5UxX0t6JX71Xf X-Received: by 2002:a17:906:749a:b0:994:1844:caec with SMTP id e26-20020a170906749a00b009941844caecmr4135343ejl.77.1690737172169; Sun, 30 Jul 2023 10:12:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690737172; cv=none; d=google.com; s=arc-20160816; b=fWkihfeAh+973LH9hfhVt4tQSkRLQzyNcCp6Eu/U0I5OJPMujoHl5JnXxgiQm84eyT HSP3rOhZCOG1nbK5C4KoXnf8sr/ziRlmz7rcwf0qQhoSkVM61WqPUFOlqth8k7XQrzbE FA2jx603Mhy5pUns1+cY9c1eMUxQv6r7+/s5KsDzwwER+7bh6fDpybzHawRPBUEaCPW/ Exs18axvrvdWLPCNGuZNvXDj802kRK4wWoC/awbyUO+L+1PWjPCEyUh5aRu7vDAgjzuM xojSyVQYjm5NE2EkGzfzCY3ZZ3fwEp2R2dewaXVkvblLR50RueKzwcRb4X3J+b2c8mFW UnAg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=LjF+zN6TJ1BvEvRrI5C4CD8rIkkkthqR+fjsUt4Q8CQ=; fh=6ywE3cadAEHt3CqWKbi2najT+v0c7si89uhbZ0SfPzA=; b=ZQF3Z/tNc/k5uHyrY7zt1jOQ5MpebdUU8Hp7dMuQGNe70V3CYcQ6UwtyNJcJYkKkUb Y5YKMWA8QylpAheG0J/+zatIs66fjNYGOg/ucl2lYxdy9YEMNDlGxGZ5D3Lmx1OMvAZN v4axSyRQ11vQgO4eHmi0WJoK3PhEl+OEJ8JaOk0QUo1p3IO5g3CxLL1OCJyNAS0Wfz6I PUKy2I0xPYD6mQGlVOCnxiEtmRPYf9eqJey2PihCIuqa8BFYIyYPRAzFYibypUB5eSis YcpnwWohbZZFyVidXfX6VbFzAqvbwlhpaWOkOQvZ8kLKOgE2K7Kyv+OEqOID7Hi/ZqgU 7Uzg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b="Vg/2oAVG"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id mh25-20020a170906eb9900b0097391f75082si2152008ejb.838.2023.07.30.10.12.27; Sun, 30 Jul 2023 10:12:52 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b="Vg/2oAVG"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230000AbjG3PQQ (ORCPT + 99 others); Sun, 30 Jul 2023 11:16:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50622 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229918AbjG3PQO (ORCPT ); Sun, 30 Jul 2023 11:16:14 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9C5F3138 for ; Sun, 30 Jul 2023 08:16:13 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id 5b1f17b1804b1-3fba8e2aa52so42188475e9.1 for ; Sun, 30 Jul 2023 08:16:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690730172; x=1691334972; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LjF+zN6TJ1BvEvRrI5C4CD8rIkkkthqR+fjsUt4Q8CQ=; b=Vg/2oAVG+oc700uPemJAECQqoZhvkHC4+v5G5umPPiGRagPclA/x9n9QgA5SDk8rdg xAP9ReHyrkmHzKD+QCBzbqzaesJvP/4YCVXhkUBDq7oFIh6fXXlQ/W6sesudYok3JgtG i8bNtt0JeBjQ7J+lYkbC40TeksYu8cuU6k1rZccXxVLpTekSilTdDcASxORLzoJggl+u dRvUz2KpmJ9ZuYWecFzFZAf1jF7ZS6Fe1gbbhfyhcMe2hud5pyNq2/bry4VpryWWF7x9 5XjSiGdDOVXVOjoB9TsdMcNi/ewnPLEIYUwh7l9m4DJjOtr0WmlRT+SiTcjbgC7BRS1N Zt9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690730172; x=1691334972; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LjF+zN6TJ1BvEvRrI5C4CD8rIkkkthqR+fjsUt4Q8CQ=; b=a1u51rrFCKwhbysKj25kUQzqMgF1p+lTpRpo/E+2EreLrDNo4by/xnIEI71kLJDM3Y ekrAw3Y4BhzlFyyO2/YUfqmAO7PvAaPBGDOND+z51YTcYjV3S4VOvZhbBMd4Hfi7Mmi8 pwKbeC83i6YGYdDi8a+YcBeMib+H+K74fSlhm49KUQP3qSKDDdvMAhD7B5JMpnWHRz11 U5XPINj2A6CYAKtH9w1Bl4wubqJkwkByee6aHMJB4iXV3hwymBnDKUahszwKcodfj6pJ VIRcNPdHm3X/UTYLZr/BiuT36shaDodImA03BjZQ7LkGFvKhbKhoPCPT3NNCkoDQDFq5 ad0Q== X-Gm-Message-State: ABy/qLbOPXSMQ2MAtfcqdkcpBphnzHeod0VAc3dEUiKiLtLkiZ3IdohO WzQD8C88niBY4LOJAImguKcpGg== X-Received: by 2002:a7b:c450:0:b0:3fb:c990:3b2 with SMTP id l16-20020a7bc450000000b003fbc99003b2mr5960153wmi.34.1690730172214; Sun, 30 Jul 2023 08:16:12 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:eda5:aa63:ce24:dac2]) by smtp.gmail.com with ESMTPSA id f17-20020a7bcc11000000b003fd2d33ea53sm9123027wmh.14.2023.07.30.08.16.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Jul 2023 08:16:11 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v2 3/6] memblock: pass memblock_type to memblock_setclr_flag Date: Sun, 30 Jul 2023 16:16:03 +0100 Message-Id: <20230730151606.2871391-4-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230730151606.2871391-1-usama.arif@bytedance.com> References: <20230730151606.2871391-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772866421289454175 X-GMAIL-MSGID: 1772866421289454175 This allows setting flags to both memblock types and is in preparation for setting flags (for e.g. to not initialize struct pages) on reserved memory region. Signed-off-by: Usama Arif --- mm/memblock.c | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/mm/memblock.c b/mm/memblock.c index f9e61e565a53..43cb4404d94c 100644 --- a/mm/memblock.c +++ b/mm/memblock.c @@ -896,10 +896,9 @@ int __init_memblock memblock_physmem_add(phys_addr_t base, phys_addr_t size) * * Return: 0 on success, -errno on failure. */ -static int __init_memblock memblock_setclr_flag(phys_addr_t base, - phys_addr_t size, int set, int flag) +static int __init_memblock memblock_setclr_flag(struct memblock_type *type, + phys_addr_t base, phys_addr_t size, int set, int flag) { - struct memblock_type *type = &memblock.memory; int i, ret, start_rgn, end_rgn; ret = memblock_isolate_range(type, base, size, &start_rgn, &end_rgn); @@ -928,7 +927,7 @@ static int __init_memblock memblock_setclr_flag(phys_addr_t base, */ int __init_memblock memblock_mark_hotplug(phys_addr_t base, phys_addr_t size) { - return memblock_setclr_flag(base, size, 1, MEMBLOCK_HOTPLUG); + return memblock_setclr_flag(&memblock.memory, base, size, 1, MEMBLOCK_HOTPLUG); } /** @@ -940,7 +939,7 @@ int __init_memblock memblock_mark_hotplug(phys_addr_t base, phys_addr_t size) */ int __init_memblock memblock_clear_hotplug(phys_addr_t base, phys_addr_t size) { - return memblock_setclr_flag(base, size, 0, MEMBLOCK_HOTPLUG); + return memblock_setclr_flag(&memblock.memory, base, size, 0, MEMBLOCK_HOTPLUG); } /** @@ -957,7 +956,7 @@ int __init_memblock memblock_mark_mirror(phys_addr_t base, phys_addr_t size) system_has_some_mirror = true; - return memblock_setclr_flag(base, size, 1, MEMBLOCK_MIRROR); + return memblock_setclr_flag(&memblock.memory, base, size, 1, MEMBLOCK_MIRROR); } /** @@ -977,7 +976,7 @@ int __init_memblock memblock_mark_mirror(phys_addr_t base, phys_addr_t size) */ int __init_memblock memblock_mark_nomap(phys_addr_t base, phys_addr_t size) { - return memblock_setclr_flag(base, size, 1, MEMBLOCK_NOMAP); + return memblock_setclr_flag(&memblock.memory, base, size, 1, MEMBLOCK_NOMAP); } /** @@ -989,7 +988,7 @@ int __init_memblock memblock_mark_nomap(phys_addr_t base, phys_addr_t size) */ int __init_memblock memblock_clear_nomap(phys_addr_t base, phys_addr_t size) { - return memblock_setclr_flag(base, size, 0, MEMBLOCK_NOMAP); + return memblock_setclr_flag(&memblock.memory, base, size, 0, MEMBLOCK_NOMAP); } static bool should_skip_region(struct memblock_type *type, From patchwork Sun Jul 30 15:16:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 128254 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp1603136vqg; Sun, 30 Jul 2023 10:50:34 -0700 (PDT) X-Google-Smtp-Source: APBJJlF/JcknKKtF3nlqP0dNc5UUT8GucQ+3XwrGfYY4t3tj0yOAOjFg9jAfHQDunBaFCZXkb+Sb X-Received: by 2002:a17:90a:b015:b0:268:1489:959e with SMTP id x21-20020a17090ab01500b002681489959emr6636497pjq.33.1690739433908; Sun, 30 Jul 2023 10:50:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690739433; cv=none; d=google.com; s=arc-20160816; b=h9i/UAmJkJnYo2qShehHMuaNyWgU/mC91rIXlyvltSA/8xT3avuepEy6t7hFZwKO8Q aPLndrr8XWy9hFYVUmAuu5jszVIeftMvvtBCuwXNvgtpg2IXv8zjZ6A4bmiEq8NSqXLg nwpPJz9NAQ8xIb1lId2oE9N4F2sdc80rew/NenKaV6+hxzyiew2MC6XX1dvd3kSVBfK2 KXU1Mqg+yU/kTCDfnAnwCTI0Kbus4UXgJpmwXYGcfzVdlfiuDKFByydm7tMLgYTu39VQ vgP8sJJ1TKqU7GmSq58/T/OPzx6BrQgZaQRFOnqSHvVU/FXuPXm3fEMmOhBI77GqWnHy eNCg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=sRlgK3gsc81CRpAMRRPsywIlI5j9OUiW8IxZjNa20o0=; fh=6ywE3cadAEHt3CqWKbi2najT+v0c7si89uhbZ0SfPzA=; b=pg6GttyhmuzFr+MNtv9SjtO/2c8xhbf5XvKqFS+yu6NYVzk2PLdyvYyxL5nP+ahyG3 OwD86n/PyHpu0UsQTbnBJMgQdFUtIb0EP9qIuGgy91LWoEcH387dSZx3rUslI0JcyKRu vxq4x8CL4OatC9jmJbutlJRYP+jS/l2ZTyVHgUvi8/9+KXQCNOFL5KpPoHYXHkJlOYTZ 0d1iPRJSuR8QGC3WyqhATnIZlizAmZMEajiZoYZiVp+WpIY4FJuGk5nnFR4uae6yvMBJ zxxOw134cdDJlFOJqsWyysMA2xiEKy3FeNvHtla6zqKvCkUhviinFNnfD/QzkzcHq1YG eixQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b="W/lVFu8o"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id pg18-20020a17090b1e1200b00262fc8d4dfesi1430137pjb.108.2023.07.30.10.50.13; Sun, 30 Jul 2023 10:50:33 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b="W/lVFu8o"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230074AbjG3PQq (ORCPT + 99 others); Sun, 30 Jul 2023 11:16:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50622 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230035AbjG3PQk (ORCPT ); Sun, 30 Jul 2023 11:16:40 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 89A54171A for ; Sun, 30 Jul 2023 08:16:14 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id 5b1f17b1804b1-3fe2048c910so2025485e9.1 for ; Sun, 30 Jul 2023 08:16:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690730173; x=1691334973; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sRlgK3gsc81CRpAMRRPsywIlI5j9OUiW8IxZjNa20o0=; b=W/lVFu8oBYOHaQON2+IUAE4OeTID9l1Wz5BLxJdEu8LJBB9z/KErJPNGKn4gPKcqET TOz183J9eC9oj+adoUW1wOIA2x17GqKNs+kATE99/ZRY8Fjp8O5VirlHjPixUUm43ku4 z+wT3z1JbZFobFUIlYBs/gN3XFoPKT2xOzlfLL6MoMgEtn/SiRo+2YI4xg86K/JhVl3i MuhecVQM9cEuPwUzqT4PGXvn5nnp3Ror/whrcq0G86C37cVp/BcEGPKbi/b5/niap3Zq DtL+erRe6APzX6GQNORLFAEQj9gidto4zepL9icunXIYosg8Xv4r0ivr6Y0+U81B8I1M t8+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690730173; x=1691334973; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sRlgK3gsc81CRpAMRRPsywIlI5j9OUiW8IxZjNa20o0=; b=YZ3KGxN0H8XlPhQWdPsNzjiPZEJQntgotTMelPIicrugNOD/xy5FzjAwbMCMohbOnT MU4GhI+WjCvJJrn5Ig3sTLnoylcdGB6inONai6y+uHAYOOxXs9gjCfN/0lT5sMcZPbTm A0jnaOrm248kwuMPthVEK5DRwfWqKNS6WJdPtv1Lf5WiS5NdAzWlE0yZ3NFkEG1HeFnt ML7GRs19ORDhLmK22TSigItHLPiV/SmXGHLxl07AjieVhRxLomR7vB5kl1DG2DPpvi9G 7yqEPYcrnpOro+w5KmTWRju+1dQUmZdlt1z5UcEkYDzl+uwanupD3Wrn+Icg0IFqfC0G Ku3g== X-Gm-Message-State: ABy/qLbvk6sCB0Rof9wXWm+4Yk7oNos8J6amGuYvI1idkCPNB+n9xPbk RhPscleu2TXuHL1HeuT6NXZe35/+l3W6viI/nNw= X-Received: by 2002:a05:600c:2809:b0:3fa:9767:c816 with SMTP id m9-20020a05600c280900b003fa9767c816mr5021376wmb.39.1690730173023; Sun, 30 Jul 2023 08:16:13 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:eda5:aa63:ce24:dac2]) by smtp.gmail.com with ESMTPSA id f17-20020a7bcc11000000b003fd2d33ea53sm9123027wmh.14.2023.07.30.08.16.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Jul 2023 08:16:12 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v2 4/6] memblock: introduce MEMBLOCK_RSRV_NOINIT flag Date: Sun, 30 Jul 2023 16:16:04 +0100 Message-Id: <20230730151606.2871391-5-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230730151606.2871391-1-usama.arif@bytedance.com> References: <20230730151606.2871391-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772868792720146913 X-GMAIL-MSGID: 1772868792720146913 For reserved memory regions marked with this flag, reserve_bootmem_region is not called during memmap_init_reserved_pages. This can be used to avoid struct page initialization for regions which won't need them, for e.g. hugepages with HVO enabled. Signed-off-by: Usama Arif --- include/linux/memblock.h | 9 +++++++++ mm/memblock.c | 30 +++++++++++++++++++++++++----- 2 files changed, 34 insertions(+), 5 deletions(-) diff --git a/include/linux/memblock.h b/include/linux/memblock.h index f71ff9f0ec81..de7760aeb870 100644 --- a/include/linux/memblock.h +++ b/include/linux/memblock.h @@ -40,6 +40,8 @@ extern unsigned long long max_possible_pfn; * via a driver, and never indicated in the firmware-provided memory map as * system RAM. This corresponds to IORESOURCE_SYSRAM_DRIVER_MANAGED in the * kernel resource tree. + * @MEMBLOCK_RSRV_NOINIT: memory region for which struct pages are not initialized + * (only for reserved regions). */ enum memblock_flags { MEMBLOCK_NONE = 0x0, /* No special request */ @@ -47,6 +49,7 @@ enum memblock_flags { MEMBLOCK_MIRROR = 0x2, /* mirrored region */ MEMBLOCK_NOMAP = 0x4, /* don't add to kernel direct mapping */ MEMBLOCK_DRIVER_MANAGED = 0x8, /* always detected via a driver */ + MEMBLOCK_RSRV_NOINIT = 0x10, /* don't initialize struct pages */ }; /** @@ -125,6 +128,7 @@ int memblock_clear_hotplug(phys_addr_t base, phys_addr_t size); int memblock_mark_mirror(phys_addr_t base, phys_addr_t size); int memblock_mark_nomap(phys_addr_t base, phys_addr_t size); int memblock_clear_nomap(phys_addr_t base, phys_addr_t size); +int memblock_reserved_mark_noinit(phys_addr_t base, phys_addr_t size); void memblock_free_all(void); void memblock_free(void *ptr, size_t size); @@ -259,6 +263,11 @@ static inline bool memblock_is_nomap(struct memblock_region *m) return m->flags & MEMBLOCK_NOMAP; } +static inline bool memblock_is_noinit(struct memblock_region *m) +{ + return m->flags & MEMBLOCK_RSRV_NOINIT; +} + static inline bool memblock_is_driver_managed(struct memblock_region *m) { return m->flags & MEMBLOCK_DRIVER_MANAGED; diff --git a/mm/memblock.c b/mm/memblock.c index 43cb4404d94c..40690ce0b06e 100644 --- a/mm/memblock.c +++ b/mm/memblock.c @@ -991,6 +991,21 @@ int __init_memblock memblock_clear_nomap(phys_addr_t base, phys_addr_t size) return memblock_setclr_flag(&memblock.memory, base, size, 0, MEMBLOCK_NOMAP); } +/** + * memblock_reserved_mark_noinit - Mark a reserved memory region with flag MEMBLOCK_RSRV_NOINIT. + * @base: the base phys addr of the region + * @size: the size of the region + * + * struct pages will not be initialized for reserved memory regions marked with + * %MEMBLOCK_RSRV_NOINIT. + * + * Return: 0 on success, -errno on failure. + */ +int __init_memblock memblock_reserved_mark_noinit(phys_addr_t base, phys_addr_t size) +{ + return memblock_setclr_flag(&memblock.reserved, base, size, 1, MEMBLOCK_RSRV_NOINIT); +} + static bool should_skip_region(struct memblock_type *type, struct memblock_region *m, int nid, int flags) @@ -2107,13 +2122,18 @@ static void __init memmap_init_reserved_pages(void) memblock_set_node(start, end, &memblock.reserved, nid); } - /* initialize struct pages for the reserved regions */ + /* + * initialize struct pages for reserved regions that don't have + * the MEMBLOCK_RSRV_NOINIT flag set + */ for_each_reserved_mem_region(region) { - nid = memblock_get_region_node(region); - start = region->base; - end = start + region->size; + if (!memblock_is_noinit(region)) { + nid = memblock_get_region_node(region); + start = region->base; + end = start + region->size; - reserve_bootmem_region(start, end, nid); + reserve_bootmem_region(start, end, nid); + } } } From patchwork Sun Jul 30 15:16:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 128249 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp1595279vqg; Sun, 30 Jul 2023 10:24:16 -0700 (PDT) X-Google-Smtp-Source: APBJJlFzwHywWqqMiGL47iggrpWezcePvSEBLIvTPOqmHPcFCtqkCxUJQxY4sO3dtgz5+4pM7vfx X-Received: by 2002:a17:902:c245:b0:1bc:15ea:ced8 with SMTP id 5-20020a170902c24500b001bc15eaced8mr37315plg.54.1690737856479; Sun, 30 Jul 2023 10:24:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690737856; cv=none; d=google.com; s=arc-20160816; b=FWd6pSOc/+jd0J05+jMpYKupPwx7XL05EB8emP/cffKKgAS6wB9kb6YMqffqm+EsiQ NXKZOpIS6eNYdPnitsCzlQ6GC8Jbyz1P8szcPUfLxrv4KDK3oVp3DAdpX8UqUdMTv0sq tdmQvRT6HKoWEqKUhQQmDM3/AEc1OKMgiMPJZZ743FbmovKJKXfe5+dnIKN1YHQrgwlv //tcoS10Q9vKGGzYgbU1mBHksR/VBxVnDN4m+1sRJUh+iUUW/vKYkjpyId+TtaGsyUuK zgN90hDAC3U2qYnf9gk62afdbkvlq9MNiFAKe7ZCqvH8lex3Ujp0dagmA5CGcdMGAUWm /fzg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=SFM4pR1QacGlm/8xlIUmCJcy4VSy/s+0OM0YFxjuA8k=; fh=6ywE3cadAEHt3CqWKbi2najT+v0c7si89uhbZ0SfPzA=; b=mcKnEtqxAUN20SAOJbNTHmsBeDhLKtqbZoKLwAJEuIIMSZtbZjkbQqp4FzPELb3lpJ wWMoQNK6Nc8EvIIbf/15AUjHrsJkm6h/RYsc9XFGXstEGNi4ja7SZ6cJGNcl83bPOKSb bSu9fnOgysET5RBp+hK5i4TwWbKNUt/zqxiWErDmU6S7nyAjhJKLi7uwTT5YaDvaz34Z MwmH3xvDZ/rutaHisdgbhdimrAMMcvsBLXP8Lp/JjCj0rC2vR9gRvYX1nWXQmGBFW9Ed trkiKniKTPoPN/2HFoPrVh1Lgz3Xl9UqRsvD0KhpkBhHKotfGdGRpJAJSuDIiOcclyr0 /vmw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=TkyhfbVG; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id jc9-20020a17090325c900b001b8bab3d5dasi5434767plb.108.2023.07.30.10.24.03; Sun, 30 Jul 2023 10:24:16 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=TkyhfbVG; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230023AbjG3PQU (ORCPT + 99 others); Sun, 30 Jul 2023 11:16:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50632 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229996AbjG3PQQ (ORCPT ); Sun, 30 Jul 2023 11:16:16 -0400 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 45F69138 for ; Sun, 30 Jul 2023 08:16:15 -0700 (PDT) Received: by mail-wm1-x335.google.com with SMTP id 5b1f17b1804b1-3fbc12181b6so42071685e9.2 for ; Sun, 30 Jul 2023 08:16:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690730174; x=1691334974; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SFM4pR1QacGlm/8xlIUmCJcy4VSy/s+0OM0YFxjuA8k=; b=TkyhfbVGe1Y7YHfHksu1wqG6hgQhYRmidCb5+Y61F1QRRs/h47YJFt0AIJh54Q0FKL gByq+10/h43JlVcCEVZjC6aUzw/Omy08uMjBB7CgI/FZQXN3z5WuJzdScj4/HnraGiyu il7QDEWZYGSpogyg92dYwrJa+Eqeb9M65dGnmL7nnCP2F14JH377zSxW/hWjC6XaZnyk O4M74hvreh6XC7+2F9vY3dYi8dCjt4haamygyIK9l6ZMqTa22DtdrGIfwCNlLo2xUbbo yHp+23v36B0XyqElADWdUdmxDLaLZYjFnCSrKJy9muMAQ+U2qOTBbbF4yWjE1y0Ngwfe EPHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690730174; x=1691334974; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SFM4pR1QacGlm/8xlIUmCJcy4VSy/s+0OM0YFxjuA8k=; b=YW4EEIX4ylfJSBse2jsh5Dwd577GK00szMeZY1xSl5zbyNMxecH2sf27ZHgbnuUd3S g9sG+OpoEzfCScytXOPQLSbq0s8n8uASoiwpocWAmFuQKfsu7A2Jy3yofy33877XeIYC DS8shqJVhPQKDToft7zzOyEBie0jcJZDkAPHNQW8IE3G5AaD2F0UEhHjmizplXAvtuc0 tOdT7Nd7tmHPQ21BNpTjz89chSNz+w6Qr5ImFLy8KpY7vkvXAMHLAYmh39+DQhJMCz+x 6/s5n0RDRj0Z3EFyG2lS1FtHfalzEx5Jpo6AjOaML+MDV/AXYWQxVsacSvGvdpIA6TEY bxsw== X-Gm-Message-State: ABy/qLb3o9Dyp3eeJGMUfS/gYDusB9ow2MAgbPmr4WlLVedRFa2D8h/3 myGTE08cEfpcYTXgnYWakTpu2w== X-Received: by 2002:a05:600c:d5:b0:3fb:e206:ca5f with SMTP id u21-20020a05600c00d500b003fbe206ca5fmr5709508wmm.31.1690730173889; Sun, 30 Jul 2023 08:16:13 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:eda5:aa63:ce24:dac2]) by smtp.gmail.com with ESMTPSA id f17-20020a7bcc11000000b003fd2d33ea53sm9123027wmh.14.2023.07.30.08.16.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Jul 2023 08:16:13 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v2 5/6] mm: move allocation of gigantic hstates to the start of mm_core_init Date: Sun, 30 Jul 2023 16:16:05 +0100 Message-Id: <20230730151606.2871391-6-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230730151606.2871391-1-usama.arif@bytedance.com> References: <20230730151606.2871391-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772867138437192475 X-GMAIL-MSGID: 1772867138437192475 Whether the initialization of tail struct pages of a hugepage happens or not will become dependent on the commandline parameter hugetlb_free_vmemmap in the future. Hence, hugetlb_hstate_alloc_pages needs to be after command line parameters are parsed and the start of mm_core_init is a good point. Signed-off-by: Usama Arif --- mm/hugetlb.c | 18 ++++++++++-------- mm/internal.h | 9 +++++++++ mm/mm_init.c | 6 ++++++ 3 files changed, 25 insertions(+), 8 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 541c07b6d60f..bf60545496d7 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4400,14 +4400,6 @@ static int __init hugepages_setup(char *s) } } - /* - * Global state is always initialized later in hugetlb_init. - * But we need to allocate gigantic hstates here early to still - * use the bootmem allocator. - */ - if (hugetlb_max_hstate && hstate_is_gigantic(parsed_hstate)) - hugetlb_hstate_alloc_pages(parsed_hstate); - last_mhp = mhp; return 1; @@ -4419,6 +4411,16 @@ static int __init hugepages_setup(char *s) } __setup("hugepages=", hugepages_setup); +void __init hugetlb_hstate_alloc_gigantic_pages(void) +{ + int i; + + for (i = 0; i < HUGE_MAX_HSTATE; i++) { + if (hstate_is_gigantic(&hstates[i])) + hugetlb_hstate_alloc_pages(&hstates[i]); + } +} + /* * hugepagesz command line processing * A specific huge page size can only be specified once with hugepagesz. diff --git a/mm/internal.h b/mm/internal.h index a7d9e980429a..692bb1136a39 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1102,4 +1102,13 @@ struct vma_prepare { struct vm_area_struct *remove; struct vm_area_struct *remove2; }; + +#ifdef CONFIG_HUGETLBFS +void __init hugetlb_hstate_alloc_gigantic_pages(void); +#else +static inline void __init hugetlb_hstate_alloc_gigantic_pages(void); +{ +} +#endif /* CONFIG_HUGETLBFS */ + #endif /* __MM_INTERNAL_H */ diff --git a/mm/mm_init.c b/mm/mm_init.c index a1963c3322af..f2751ccd7d99 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -26,6 +26,7 @@ #include #include #include +#include #include "internal.h" #include "slab.h" #include "shuffle.h" @@ -2768,6 +2769,11 @@ static void __init mem_init_print_info(void) */ void __init mm_core_init(void) { + /* + * We need to allocate gigantic hstates here early to still use the bootmem + * allocator. Non gigantic hstates are initialized later in hugetlb_init. + */ + hugetlb_hstate_alloc_gigantic_pages(); /* Initializations relying on SMP setup */ build_all_zonelists(NULL); page_alloc_init_cpuhp(); From patchwork Sun Jul 30 15:16:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 128242 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp1572750vqg; Sun, 30 Jul 2023 09:19:54 -0700 (PDT) X-Google-Smtp-Source: APBJJlGLT+a7a2k+Oni5D8Ua3l+HSTAgAW/NnpUrooRts9Frdq0jPuCbMETXG1A0QFniDUC/02ve X-Received: by 2002:a05:6a20:3d13:b0:13d:384f:b6ec with SMTP id y19-20020a056a203d1300b0013d384fb6ecmr5215498pzi.58.1690733994008; Sun, 30 Jul 2023 09:19:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690733993; cv=none; d=google.com; s=arc-20160816; b=o0mqg92eXE1AaLETcn0fkK3Sh40780WISvhGJW6XtxKxjowkB9LxmII68p0n+kGpSP o3wo+8Y863HrG0Z1VgBEm1zPdqNp7yzJrdnsKgitXsVLWD8EV8yZYo4mjybcJFYirHTB ab/ahpi0d5rX4wZ2A4dlh8eKVqR7fnuzNrOw/59KPIkAnxF3atCArgwA/JSqPQGmTjNw ZIcAL1iu8qmUXXX39weu51lYU+YHKFwChnVYb2W8VH7I+Py/hWuuygwuXahHVgnThxmC thpDlovUm04CHCjyPvb9rScElyhW6dN192D89r0/l1NbefP/Ct37ERxhf1QNlIgTJroY jNVQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ed3bTY/DQrtVo38S3b99vtsCN5ydWYuckmFt+jfpXnQ=; fh=6ywE3cadAEHt3CqWKbi2najT+v0c7si89uhbZ0SfPzA=; b=lYpGafakxOFxrdB2g1Y6IcaiOlKaP3KW0IfvydHQkZert1+mkpsJil5r5/StfF2a1c PweccSfeH94DvYNaI/f1ohWQxl1VJRXKEIO/jTJm4nIePhenqH5CyzEGtxZhdV/N6Vqc Kveu9E8h0IlPk77+jsqerYHx0ga2I40j7IwX0RstQggFfVWc9VYd2r+a+YEqEaxtSLTw /0dz5OmScFX76gE1lLvTzJ+4SHiIc1x78pJlymGkM85TrV/04lJD3jZ2xmxJ0GlMPOMy m+rq/gPRGUrd9bGrEB+Ey9g78QsU1yVnH2A0Te/d+tDRGQle8ngfaWL3DSlLeh4vhtUi IsBQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=CsLvLnj1; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id j20-20020a634a54000000b0056357ea1d1dsi5862347pgl.532.2023.07.30.09.19.40; Sun, 30 Jul 2023 09:19:53 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=CsLvLnj1; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230089AbjG3PQu (ORCPT + 99 others); Sun, 30 Jul 2023 11:16:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50638 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230039AbjG3PQl (ORCPT ); Sun, 30 Jul 2023 11:16:41 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1B3D2171B for ; Sun, 30 Jul 2023 08:16:16 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id 5b1f17b1804b1-3fbc5d5742eso42076825e9.3 for ; Sun, 30 Jul 2023 08:16:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690730174; x=1691334974; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ed3bTY/DQrtVo38S3b99vtsCN5ydWYuckmFt+jfpXnQ=; b=CsLvLnj1VBExXRAAglTYPWH2EpYCorZggOAC3yIHs0vybciyxsExVcLP/39obiFOYD FverDpvbx+BWRlvxddS1Ucch0Lo8ySBj5n4n9qsls9ncd4/U0ukC8FCsg5LU0Ps7yXpS 0x73YYFy0A18lccSHORRLeDqKaS8nvvq1tbkN/1SKvxZESadhIFOz/0oIn1npIu6dGrI 2CSL9zF3tMODmGVmKAr7ZywN5DBo03IuiGI33kAtutuURXCt6kQH4LYr2FKRdNpyhVsi OuuSaUD++enIZiLNISEKbcIDH+KKX7DGzU9igsnu/+61mXgL/I4WQRrk8Tlu5bD6f8xb lAyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690730174; x=1691334974; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ed3bTY/DQrtVo38S3b99vtsCN5ydWYuckmFt+jfpXnQ=; b=l5DFJAE0OqA9UJDZYpLQO/ng+i2JvHTjYiEzqZaaun9rcgHVcl1Uxb91s8q+qW0kuJ smr+zYXKp5p15eiiywO+ff9grWdzmeZ4FOjVEZYeguNG301skm9lSRGezrVHYnQPC73/ pZiNDPQ+5RIe+hPA16eZaQcZBEt5fe6K75RL2I4t3weCApIocySPqTje/eR6QyQJIldv la9NwSG9rDRlw7hJaMSd2gOCyoReqC8lnel64Ex+9NDyA1yphvGPQ33BJnbs+AYnNzKb J50d7GRRHTk7kB2WH5I9u+0P0xg9dctsWw9uIR6UnbeGeckIVR/YxJ99coCG+W57Na+7 jDzA== X-Gm-Message-State: ABy/qLZmMb2vUXBzzez8p08zQhbpIfAUk1AsJc7C/LX3sz0BEMu4URxX zDBnaWP7ieN7iy+swL2F3r10/Q== X-Received: by 2002:a05:600c:3787:b0:3fa:97b3:7ce0 with SMTP id o7-20020a05600c378700b003fa97b37ce0mr3917250wmr.26.1690730174681; Sun, 30 Jul 2023 08:16:14 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:eda5:aa63:ce24:dac2]) by smtp.gmail.com with ESMTPSA id f17-20020a7bcc11000000b003fd2d33ea53sm9123027wmh.14.2023.07.30.08.16.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Jul 2023 08:16:14 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v2 6/6] mm: hugetlb: Skip initialization of struct pages freed later by HVO Date: Sun, 30 Jul 2023 16:16:06 +0100 Message-Id: <20230730151606.2871391-7-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230730151606.2871391-1-usama.arif@bytedance.com> References: <20230730151606.2871391-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772863088743578385 X-GMAIL-MSGID: 1772863088743578385 This is done by marking the region for which to skip initialization with the MEMBLOCK_RSRV_NOINIT flag. If the region is for hugepages and if HVO is enabled, then those struct pages which will be freed later don't need to be initialized. This can save significant time when a large number of hugepages are allocated at boot time. HUGETLB_VMEMMAP_RESERVE_SIZE struct pages at the start of hugepage still need to be initialized. Signed-off-by: Usama Arif --- mm/hugetlb.c | 21 +++++++++++++++++++++ mm/hugetlb_vmemmap.c | 2 +- mm/hugetlb_vmemmap.h | 3 +++ 3 files changed, 25 insertions(+), 1 deletion(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index bf60545496d7..8434100f60ae 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3151,6 +3151,9 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) { struct huge_bootmem_page *m = NULL; /* initialize for clang */ int nr_nodes, node; + phys_addr_t hugetlb_vmemmap_reserve_size = + HUGETLB_VMEMMAP_RESERVE_SIZE * sizeof(struct page); + phys_addr_t noinit_base; /* do node specific alloc */ if (nid != NUMA_NO_NODE) { @@ -3158,6 +3161,15 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) 0, MEMBLOCK_ALLOC_ACCESSIBLE, nid); if (!m) return 0; + + if (vmemmap_optimize_enabled && hugetlb_vmemmap_optimizable(h)) { + noinit_base = virt_to_phys( + (void *)((phys_addr_t) m + hugetlb_vmemmap_reserve_size)); + memblock_reserved_mark_noinit( + noinit_base, + huge_page_size(h) - hugetlb_vmemmap_reserve_size); + } + goto found; } /* allocate from next node when distributing huge pages */ @@ -3172,6 +3184,15 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) */ if (!m) return 0; + + if (vmemmap_optimize_enabled && hugetlb_vmemmap_optimizable(h)) { + noinit_base = virt_to_phys( + (void *)((phys_addr_t) m + hugetlb_vmemmap_reserve_size)); + memblock_reserved_mark_noinit( + noinit_base, + huge_page_size(h) - hugetlb_vmemmap_reserve_size); + } + goto found; } diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index bdf750a4786b..b5b7834e0f42 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -443,7 +443,7 @@ static int vmemmap_remap_alloc(unsigned long start, unsigned long end, DEFINE_STATIC_KEY_FALSE(hugetlb_optimize_vmemmap_key); EXPORT_SYMBOL(hugetlb_optimize_vmemmap_key); -static bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON); +bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON); core_param(hugetlb_free_vmemmap, vmemmap_optimize_enabled, bool, 0); /** diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 3e7978a9af73..3fff6f611c19 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -64,4 +64,7 @@ static inline bool hugetlb_vmemmap_optimizable(const struct hstate *h) { return hugetlb_vmemmap_optimizable_size(h) != 0; } + +extern bool vmemmap_optimize_enabled; + #endif /* _LINUX_HUGETLB_VMEMMAP_H */