From patchwork Tue Jan 2 13:12:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gang Li X-Patchwork-Id: 184331 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:6f82:b0:100:9c79:88ff with SMTP id tb2csp4436376dyb; Tue, 2 Jan 2024 05:14:55 -0800 (PST) X-Google-Smtp-Source: AGHT+IELKnbUxFMCFtYUjQgxePb424+z1UKd4hb4rAhSyyUF9eAMljEXYzAUU3GasgLLyNAudXdt X-Received: by 2002:a05:622a:593:b0:427:7bdf:5a87 with SMTP id c19-20020a05622a059300b004277bdf5a87mr25127018qtb.10.1704201294874; Tue, 02 Jan 2024 05:14:54 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1704201294; cv=none; d=google.com; s=arc-20160816; b=Vyl45oPX5zyq4Le+4yAhPsIL0klWMltpmSjnx+KjvIKzdKhLFKSW6/vAYGCUheZET5 K72FlsySKm9Ju/xrYkqMqdC3T9pOuX2BtqYYz39Lxknt445bGPnW9+zZrosPil16PpjO D/kONAlCatIj4sIvZ3y9IpgGZXQsKLDuIi4yA/M42rDDPeBGA5+bgtczyxJ4ZipB3lgt Q/wxcDnxj2MCrpeDn+j5Grq0xu7GzRABa5DJgcjEDi58Ak72V6SSGI+uxHMD/PSc1M4k o/+tpG0M2ruipGq3R6vWWJ3eijCtXTdsVdQAlxOhF7Aue2BkVpZujY596Mc1qJ0EZk8+ P6oQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=lq6odAt2f+wReMUr3LVecCPvhipoB8fbQDkgCZGAfFs=; fh=5n5b5+XOwLgGNDePhvfXysty56rnkWOs9jaHEODdVeE=; b=l9eoX3qZ6D7kUn27ZL0xFK+uCAlOo3HSCPBdToCJqTcqeWMhuY4oINsiiNklLvaR0+ FjD1ZL+lmn7u0Fa8n+1HqGMfbMfcHglPfy9yK+JxFv0Zmp322qIQTttZ1OZi71EVkYMz PEmAOR7jfq+CINf8mZU4TXhf91txrkoAxm2aDT9HqBnWvSpwzCRIi2LjWZksvp1zNIUf bDxMJ8EIo/0QxSJHBP9y2Q5J+gK3+PzFtsAATUVBPqq4rraR2+drBReOguIyRhsXZdAd K5GNu7LgjztpG/0Z2JFFTat6xjl4OKXdetu1pY8hcPKpq1OWM09v0q/MKUT+0wwZ/9p1 Yn9w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b="rOSB7/ml"; spf=pass (google.com: domain of linux-kernel+bounces-14387-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-14387-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id 4-20020ac85704000000b00421b95298d8si26914811qtw.404.2024.01.02.05.14.54 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 05:14:54 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-14387-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b="rOSB7/ml"; spf=pass (google.com: domain of linux-kernel+bounces-14387-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-14387-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 882E91C22088 for ; Tue, 2 Jan 2024 13:14:52 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 8BB9014F73; Tue, 2 Jan 2024 13:13:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="rOSB7/ml" X-Original-To: linux-kernel@vger.kernel.org Received: from out-177.mta1.migadu.com (out-177.mta1.migadu.com [95.215.58.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 936B814AAA for ; Tue, 2 Jan 2024 13:13:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1704201222; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lq6odAt2f+wReMUr3LVecCPvhipoB8fbQDkgCZGAfFs=; b=rOSB7/mlWJwm7Sgj4858g08l3Mah9AIYX19x1bg4TM3sAFEv9Sl0kgizs7/gcRTPswrfp0 vOOopJqL+bE11QDfDg2MV8rTX8iV8fJMNIn3ouKYEcKaZ82/9tLxyXxN5xbIYQpLmzlMAU 1lf390N2ueiFxZGQLNGz7dsM80AvN8Q= From: Gang Li To: David Hildenbrand , David Rientjes , Mike Kravetz , Muchun Song , Andrew Morton , Tim Chen Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, ligang.bdlg@bytedance.com, Gang Li Subject: [PATCH v3 6/7] hugetlb: parallelize 2M hugetlb allocation and initialization Date: Tue, 2 Jan 2024 21:12:48 +0800 Message-Id: <20240102131249.76622-7-gang.li@linux.dev> In-Reply-To: <20240102131249.76622-1-gang.li@linux.dev> References: <20240102131249.76622-1-gang.li@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1786984577307963551 X-GMAIL-MSGID: 1786984577307963551 By distributing both the allocation and the initialization tasks across multiple threads, the initialization of 2M hugetlb will be faster, thereby improving the boot speed. Here are some test results: test no patch(ms) patched(ms) saved ------------------- -------------- ------------- -------- 256c2t(4 node) 2M 3336 1051 68.52% 128c1t(2 node) 2M 1943 716 63.15% Signed-off-by: Gang Li --- mm/hugetlb.c | 72 ++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 53 insertions(+), 19 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index a71bc1622b53b..d1629df5f399f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -35,6 +35,7 @@ #include #include #include +#include #include #include @@ -3510,6 +3511,38 @@ static void __init hugetlb_hstate_alloc_pages_report(unsigned long allocated, st } } +static void __init hugetlb_alloc_node(unsigned long start, unsigned long end, void *arg) +{ + struct hstate *h = (struct hstate *)arg; + int i, num = end - start; + nodemask_t node_alloc_noretry; + unsigned long flags; + int next_nid_to_alloc = 0; + + /* Bit mask controlling how hard we retry per-node allocations.*/ + nodes_clear(node_alloc_noretry); + + for (i = 0; i < num; ++i) { + struct folio *folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY], + &node_alloc_noretry, &next_nid_to_alloc); + if (!folio) + break; + spin_lock_irqsave(&hugetlb_lock, flags); + __prep_account_new_huge_page(h, folio_nid(folio)); + enqueue_hugetlb_folio(h, folio); + spin_unlock_irqrestore(&hugetlb_lock, flags); + cond_resched(); + } +} + +static void __init hugetlb_vmemmap_optimize_node(unsigned long start, unsigned long end, void *arg) +{ + struct hstate *h = (struct hstate *)arg; + int nid = start; + + hugetlb_vmemmap_optimize_folios(h, &h->hugepage_freelists[nid]); +} + static unsigned long __init hugetlb_hstate_alloc_pages_gigantic(struct hstate *h) { unsigned long i; @@ -3529,26 +3562,27 @@ static unsigned long __init hugetlb_hstate_alloc_pages_gigantic(struct hstate *h static unsigned long __init hugetlb_hstate_alloc_pages_non_gigantic(struct hstate *h) { - unsigned long i; - struct folio *folio; - LIST_HEAD(folio_list); - nodemask_t node_alloc_noretry; - - /* Bit mask controlling how hard we retry per-node allocations.*/ - nodes_clear(node_alloc_noretry); - - for (i = 0; i < h->max_huge_pages; ++i) { - folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY], - &node_alloc_noretry); - if (!folio) - break; - list_add(&folio->lru, &folio_list); - cond_resched(); - } - - prep_and_add_allocated_folios(h, &folio_list); + struct padata_mt_job job = { + .fn_arg = h, + .align = 1, + .numa_aware = true + }; - return i; + job.thread_fn = hugetlb_alloc_node; + job.start = 0; + job.size = h->max_huge_pages; + job.min_chunk = h->max_huge_pages / num_node_state(N_MEMORY) / 2; + job.max_threads = num_node_state(N_MEMORY) * 2; + padata_do_multithreaded(&job); + + job.thread_fn = hugetlb_vmemmap_optimize_node; + job.start = 0; + job.size = num_node_state(N_MEMORY); + job.min_chunk = 1; + job.max_threads = num_node_state(N_MEMORY); + padata_do_multithreaded(&job); + + return h->nr_huge_pages; } /*