[RFC,v2,0/5] hugetlb: parallelize hugetlb page init on boot

  Hi all, hugetlb init parallelization has now been updated to v2.

To David Hildenbrand: padata multithread utilities has been used to reduce
code complexity.

To David Rientjes: The patch for measuring time will be separately included
in the reply. Please test during your free time, thanks.

# Introduction
Hugetlb initialization during boot takes up a considerable amount of time.
For instance, on a 2TB system, initializing 1,800 1GB huge pages takes 1-2
seconds out of 10 seconds. Initializing 11,776 1GB pages on a 12TB Intel
host takes 65.2 seconds [1], which is 17.4% of the total 373.78 seconds boot
time. This is a noteworthy figure.

Inspired by [2] and [3], hugetlb initialization can also be accelerated
through parallelization. Kernel already has infrastructure like
padata_do_multithreaded, this patch uses it to achieve effective results
by minimal modifications.

[1] https://lore.kernel.org/all/783f8bac-55b8-5b95-eb6a-11a583675000@google.com/
[2] https://lore.kernel.org/all/20200527173608.2885243-1-daniel.m.jordan@oracle.com/
[3] https://lore.kernel.org/all/20230906112605.2286994-1-usama.arif@bytedance.com/

# Test result
        test          no patch(ms)   patched(ms)   saved   
 ------------------- -------------- ------------- -------- 
  256c2t(4 node) 2M           2624           956   63.57%  
  256c2t(4 node) 1G           2679          1582   40.95%  
  128c1t(2 node) 2M           1788           684   61.74%  
  128c1t(2 node) 1G           3160          1618   48.80%  

# Change log
Changes in v2:
- Reduce complexity with `padata_do_multithreaded`
- Support 1G hugetlb

v1:
- https://lore.kernel.org/all/20231123133036.68540-1-gang.li@linux.dev/
- parallelize 2M hugetlb initialization with workqueue

Gang Li (5):
  hugetlb: code clean for hugetlb_hstate_alloc_pages
  hugetlb: split hugetlb_hstate_alloc_pages
  padata: dispatch works on different nodes
  hugetlb: parallelize 2M hugetlb allocation and initialization
  hugetlb: parallelize 1G hugetlb initialization

 include/linux/hugetlb.h |   2 +-
 include/linux/padata.h  |   2 +
 kernel/padata.c         |   8 +-
 mm/hugetlb.c            | 201 +++++++++++++++++++++++++++-------------
 mm/mm_init.c            |   1 +
 5 files changed, 148 insertions(+), 66 deletions(-)
  

On 12/08/23 10:52, Gang Li wrote:
> Hi all, hugetlb init parallelization has now been updated to v2.

Thanks for your efforts, and sorry for my late comments.

> To David Hildenbrand: padata multithread utilities has been used to reduce
> code complexity.
> 
> To David Rientjes: The patch for measuring time will be separately included
> in the reply. Please test during your free time, thanks.
> 
> # Introduction
> Hugetlb initialization during boot takes up a considerable amount of time.
> For instance, on a 2TB system, initializing 1,800 1GB huge pages takes 1-2
> seconds out of 10 seconds. Initializing 11,776 1GB pages on a 12TB Intel
> host takes 65.2 seconds [1], which is 17.4% of the total 373.78 seconds boot
> time. This is a noteworthy figure.

One issue to be concerned with is hugetlb page allocation on systems with
unbalanced numa node memory.  Commit f60858f9d327 ("hugetlbfs: don't retry
when pool page allocations start to fail") was added to deal with issues
reported on such systems.  So, users are certainly using hugetlb pages
on systems with imbalances.

If performing allocations in parallel, I believe we would want the total
number of hugetlb pages allocated to be the same as today.  For example,
consider a simple 2 node system with 16GB total memory:
node 0:  2GB
node 1: 14GB

With today's code, allocating 6656 2MB pages via the kernel command line
results in:
node 0:  924 pages
node 1: 5732 pages
total:  6656 pages

With code to parallel allocations in this series:
node 0:  924 pages
node 1: 1547 pages
total:  2471 pages
  

On Fri, 8 Dec 2023, Gang Li wrote:

> Hi all, hugetlb init parallelization has now been updated to v2.
> 
> To David Hildenbrand: padata multithread utilities has been used to reduce
> code complexity.
> 
> To David Rientjes: The patch for measuring time will be separately included
> in the reply. Please test during your free time, thanks.
> 

I'd love to, but what kernel is this based on?  :)  I can't get this to 
apply to any kernels that I have recently benchmarked with.

> # Introduction
> Hugetlb initialization during boot takes up a considerable amount of time.
> For instance, on a 2TB system, initializing 1,800 1GB huge pages takes 1-2
> seconds out of 10 seconds. Initializing 11,776 1GB pages on a 12TB Intel
> host takes 65.2 seconds [1], which is 17.4% of the total 373.78 seconds boot
> time. This is a noteworthy figure.
> 
> Inspired by [2] and [3], hugetlb initialization can also be accelerated
> through parallelization. Kernel already has infrastructure like
> padata_do_multithreaded, this patch uses it to achieve effective results
> by minimal modifications.
> 
> [1] https://lore.kernel.org/all/783f8bac-55b8-5b95-eb6a-11a583675000@google.com/
> [2] https://lore.kernel.org/all/20200527173608.2885243-1-daniel.m.jordan@oracle.com/
> [3] https://lore.kernel.org/all/20230906112605.2286994-1-usama.arif@bytedance.com/
> 
> # Test result
>         test          no patch(ms)   patched(ms)   saved   
>  ------------------- -------------- ------------- -------- 
>   256c2t(4 node) 2M           2624           956   63.57%  
>   256c2t(4 node) 1G           2679          1582   40.95%  
>   128c1t(2 node) 2M           1788           684   61.74%  
>   128c1t(2 node) 1G           3160          1618   48.80%  
> 
> # Change log
> Changes in v2:
> - Reduce complexity with `padata_do_multithreaded`
> - Support 1G hugetlb
> 
> v1:
> - https://lore.kernel.org/all/20231123133036.68540-1-gang.li@linux.dev/
> - parallelize 2M hugetlb initialization with workqueue
> 
> Gang Li (5):
>   hugetlb: code clean for hugetlb_hstate_alloc_pages
>   hugetlb: split hugetlb_hstate_alloc_pages
>   padata: dispatch works on different nodes
>   hugetlb: parallelize 2M hugetlb allocation and initialization
>   hugetlb: parallelize 1G hugetlb initialization
> 
>  include/linux/hugetlb.h |   2 +-
>  include/linux/padata.h  |   2 +
>  kernel/padata.c         |   8 +-
>  mm/hugetlb.c            | 201 +++++++++++++++++++++++++++-------------
>  mm/mm_init.c            |   1 +
>  5 files changed, 148 insertions(+), 66 deletions(-)
> 
> -- 
> 2.30.2
> 
>
  

On 12/12/23 14:14, David Rientjes wrote:
> On Fri, 8 Dec 2023, Gang Li wrote:
> 
> > Hi all, hugetlb init parallelization has now been updated to v2.
> > 
> > To David Hildenbrand: padata multithread utilities has been used to reduce
> > code complexity.
> > 
> > To David Rientjes: The patch for measuring time will be separately included
> > in the reply. Please test during your free time, thanks.
> > 
> 
> I'd love to, but what kernel is this based on?  :)  I can't get this to 
> apply to any kernels that I have recently benchmarked with.

I was able to apply and build on top of v6.7-rc5.

Gang Li,
Since hugetlb now depends on CONFIG_PADATA, the Kconfig file should be
updated to reflect this.
  

On Tue, 12 Dec 2023, Mike Kravetz wrote:

> On 12/12/23 14:14, David Rientjes wrote:
> > On Fri, 8 Dec 2023, Gang Li wrote:
> > 
> > > Hi all, hugetlb init parallelization has now been updated to v2.
> > > 
> > > To David Hildenbrand: padata multithread utilities has been used to reduce
> > > code complexity.
> > > 
> > > To David Rientjes: The patch for measuring time will be separately included
> > > in the reply. Please test during your free time, thanks.
> > > 
> > 
> > I'd love to, but what kernel is this based on?  :)  I can't get this to 
> > apply to any kernels that I have recently benchmarked with.
> 
> I was able to apply and build on top of v6.7-rc5.
> 
> Gang Li,
> Since hugetlb now depends on CONFIG_PADATA, the Kconfig file should be
> updated to reflect this.

Gotcha, thanks.

I got this:

ld: error: undefined symbol: padata_do_multithreaded
referenced by hugetlb.c:3470 (./mm/hugetlb.c:3470)
              vmlinux.o:(gather_bootmem_prealloc)
referenced by hugetlb.c:3592 (./mm/hugetlb.c:3592)
              vmlinux.o:(hugetlb_hstate_alloc_pages_non_gigantic)
referenced by hugetlb.c:3599 (./mm/hugetlb.c:3599)
              vmlinux.o:(hugetlb_hstate_alloc_pages_non_gigantic)

So, yeah we need to enable DEFERRED_STRUCT_PAGE_INIT for this to build.

On 6.6 I measured "hugepagesz=1G hugepages=11776" on as 12TB host to be 
77s this time around.

A latest Linus build with this patch set does not boot successfully, so 
I'll need to look into that and try to capture the failure.  Not sure if 
it's related to this patch or the latest Linus build in general.
  

Hi,

On 2023/12/13 08:10, David Rientjes wrote:
> On 6.6 I measured "hugepagesz=1G hugepages=11776" on as 12TB host to be
> 77s this time around.

Thanks for your test! Is this the total kernel boot time, or just the
hugetlb initialization time?

> 
> A latest Linus build with this patch set does not boot successfully, so

Which branch/tag is it compiled on?
I test this patch on v6.7-rc4 and next-20231130.

> I'll need to look into that and try to capture the failure.  Not sure if
> it's related to this patch or the latest Linus build in general.
>
  

On 2023/12/13 04:06, Mike Kravetz wrote:
> With today's code, allocating 6656 2MB pages via the kernel command line
> results in:
> node 0:  924 pages
> node 1: 5732 pages
> total:  6656 pages
> 
> With code to parallel allocations in this series:
> node 0:  924 pages
> node 1: 1547 pages
> total:  2471 pages

Hi Mike,

Disable numa_aware for hugetlb_alloc_node should solve this problem.
I will fix it in v3.
  

On Mon, 18 Dec 2023, Gang Li wrote:

> Hi,
> 
> On 2023/12/13 08:10, David Rientjes wrote:
> > On 6.6 I measured "hugepagesz=1G hugepages=11776" on as 12TB host to be
> > 77s this time around.
> 
> Thanks for your test! Is this the total kernel boot time, or just the
> hugetlb initialization time?
> 

Ah, sorry for not being specific.  It's just the hugetlb preallocation of 
11776 1GB hugetlb pages, total boot takes a few more minutes.

> > A latest Linus build with this patch set does not boot successfully, so
> 
> Which branch/tag is it compiled on?
> I test this patch on v6.7-rc4 and next-20231130.
> 

It was the latest Linus tip of tree.  I'll continue to try again until I 
get a successful boot and report back, serial console won't be possible 
for unrelated reasons.
  

On Thu, 21 Dec 2023, David Rientjes wrote:

> > Hi,
> > 
> > On 2023/12/13 08:10, David Rientjes wrote:
> > > On 6.6 I measured "hugepagesz=1G hugepages=11776" on as 12TB host to be
> > > 77s this time around.
> > 
> > Thanks for your test! Is this the total kernel boot time, or just the
> > hugetlb initialization time?
> > 
> 
> Ah, sorry for not being specific.  It's just the hugetlb preallocation of 
> 11776 1GB hugetlb pages, total boot takes a few more minutes.
> 

I had to apply this to get the patch series to compile on 6.7-rc7:

diff --git a/kernel/padata.c b/kernel/padata.c
--- a/kernel/padata.c
+++ b/kernel/padata.c
@@ -485,7 +485,7 @@ void __init padata_do_multithreaded(struct padata_mt_job *job)
 	struct padata_work my_work, *pw;
 	struct padata_mt_job_state ps;
 	LIST_HEAD(works);
-	int nworks, nid;
+	int nworks, nid = 0;
 
 	if (job->size == 0)
 		return;
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3300,7 +3300,7 @@ int alloc_bootmem_huge_page(struct hstate *h, int nid)
 int __alloc_bootmem_huge_page(struct hstate *h, int nid)
 {
 	struct huge_bootmem_page *m = NULL; /* initialize for clang */
-	int nr_nodes, node;
+	int nr_nodes, node = NUMA_NO_NODE;
 
 	/* do node specific alloc */
 	if (nid != NUMA_NO_NODE) {

With that, I compared "hugepagesz=1G hugepages=11776" before and after on 
a 12TB host with eight NUMA nodes.

Compared to 77s of total initialization time before, with this series I 
measured 18.3s.

Feel free to add this into the changelog once the initialization issues 
are fixed up and I'm happy to ack it.

Thanks!
  

On 2023/12/25 13:21, David Rientjes wrote:
> With that, I compared "hugepagesz=1G hugepages=11776" before and after on
> a 12TB host with eight NUMA nodes.
> 
> Compared to 77s of total initialization time before, with this series I
> measured 18.3s.
> 
> Feel free to add this into the changelog once the initialization issues
> are fixed up and I'm happy to ack it.
> 
> Thanks!

Cool! Thank you ;)