From patchwork Fri Jan 26 15:24:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gang Li X-Patchwork-Id: 192658 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:fa17:b0:106:343:edcb with SMTP id ju23csp8689dyc; Fri, 26 Jan 2024 07:27:39 -0800 (PST) X-Google-Smtp-Source: AGHT+IHct99T0XfF5oZwJhV99lU6A6UmqajxU9ttzuNtKfHkbOUBo4Hvlm13oNCFlqGZ+qRlSpvo X-Received: by 2002:a05:600c:b93:b0:40e:e8e3:40f1 with SMTP id fl19-20020a05600c0b9300b0040ee8e340f1mr83237wmb.369.1706282858899; Fri, 26 Jan 2024 07:27:38 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706282858; cv=pass; d=google.com; s=arc-20160816; b=YsJO65HGMEQKOWHoQm6mrtWzptMKoawJnUSD6qa3RosD8lnH+qptPqZEa/OFKTLyjS 7LWpOiLL+wep0xeSZBGeynTV5FibmHmbLgOwCsxd0V1HIbIxMZ1Viy+2NrErhzqHf5Iw lmbELijxCapLpvGNcXe6HJA+dSnxMKHWzyKTYay1WBKPw6m9dNuvh/i/6enBytPBx4hi +8rztdpkYw+PPmtW1+wdYZHg8FGYF0/1gE2jAfebqS1aEiKVeRfypuLOOQcLi7yy92gF yDAgzGornZe0K/N8IvpGnMguh+ErORyXY3l5u5QsiXNu7OUOlTq6B1iWQyN3oAvdKEWY xrkw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=jedycsf2hoazZl2VD5dKzn3Ogz/yNT2KcxNdOLYavB4=; fh=5n5b5+XOwLgGNDePhvfXysty56rnkWOs9jaHEODdVeE=; b=NvbuIdECICW3V3lidnMNxS7iuH4AK4xiCS6+9/arcfWy8AQqG/8s07+eEmJwjkj+Fc IoGw1FLZXdWz1fMnwnPXRej020/4zNyTdKimG7sOxnjRuh1Mux/iwILnC7B/umIDf65F NkORkOK677GUg+P/YjXRpEpa5E0UxgmC6JJck7uyrirxJ3JjSvRxs9pZOeDRAl1z2Bfc EjIMjxAKzmKKBRTbnSNVKrs8ov/tnuihK9QZ2p+CihVsJrYVG+McYk8Gd/WPqH8AkApz JKxvDLi57ICGWNyOvn1pjGBSr6bBGWCbF7d5vdFQbH1en8ELiB8QD5NTEN/KMv3/AJ/R jjcw== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=qwIcy4O8; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-40274-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-40274-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id i12-20020a170906a28c00b00a310709a5c4si655912ejz.305.2024.01.26.07.27.38 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 26 Jan 2024 07:27:38 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-40274-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=qwIcy4O8; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-40274-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-40274-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id C68B71F25BDF for ; Fri, 26 Jan 2024 15:26:46 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id D49A61DFF7; Fri, 26 Jan 2024 15:25:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="qwIcy4O8" Received: from out-187.mta1.migadu.com (out-187.mta1.migadu.com [95.215.58.187]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 733591D697 for ; Fri, 26 Jan 2024 15:25:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.187 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706282742; cv=none; b=KilYz0wc6iztOE3Ne7WJvZO+6fPaFGrTWIpHuOOgjfGuGTHxVSX0HjbFjb0ohPiivUbCm9vr9MtS4313VNFlYYSlZMuNeq1wbEu++ME2PATfunQyfSWtE0M5P+7wqpw05CegvJ3U2kBb6WUIsZmNa9ycy8W7sNTuhxeErxW72hU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706282742; c=relaxed/simple; bh=1MgYoEYA0TYAFs7M4fp1D81Bw+npp83XpLElfpWHMK8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=eBv3xNdrqoqeOqYEIjz0vH6pBkefL/CqZSOBqNQqBTO6HI0s/d4cJRRFtX+uXwzK3Ih2ISv1GxXJIB0kHFv2tKeguQuZkkIq3fnw+R34JKqYSKmOWAlNspPegdX7PIk4QL2wPuLf1cY8eng/4HSKz5rikYVqHd34nGXwDIRQwLg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=qwIcy4O8; arc=none smtp.client-ip=95.215.58.187 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1706282738; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jedycsf2hoazZl2VD5dKzn3Ogz/yNT2KcxNdOLYavB4=; b=qwIcy4O8oYc6ieAe8OwvpEKH4wyBee3Jlif+nxuPJKFmuXp+E3oYFgTjBUsVhVcTbe3PJO LbG1Yp4MWWpIydaJlX5rrJfhgvOcuD2EKuJO/Yme8P9fG6DM1ACjVdMyPO1FZ4Ag0JyV0D a0k79BNlAPNr4wc8ZdQcdQUIK48SoXc= From: Gang Li To: David Hildenbrand , David Rientjes , Mike Kravetz , Muchun Song , Andrew Morton , Tim Chen Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, ligang.bdlg@bytedance.com, Gang Li Subject: [PATCH v5 6/7] hugetlb: parallelize 2M hugetlb allocation and initialization Date: Fri, 26 Jan 2024 23:24:10 +0800 Message-Id: <20240126152411.1238072-7-gang.li@linux.dev> In-Reply-To: <20240126152411.1238072-1-gang.li@linux.dev> References: <20240126152411.1238072-1-gang.li@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789167254791290107 X-GMAIL-MSGID: 1789167254791290107 By distributing both the allocation and the initialization tasks across multiple threads, the initialization of 2M hugetlb will be faster, thereby improving the boot speed. Here are some test results: test case no patch(ms) patched(ms) saved ------------------- -------------- ------------- -------- 256c2T(4 node) 2M 3336 1051 68.52% 128c1T(2 node) 2M 1943 716 63.15% Signed-off-by: Gang Li Tested-by: David Rientjes Reviewed-by: Muchun Song --- mm/hugetlb.c | 73 ++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 56 insertions(+), 17 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index e4e8ffa1c145a..385840397bce5 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -35,6 +35,7 @@ #include #include #include +#include #include #include @@ -3510,6 +3511,30 @@ static void __init hugetlb_hstate_alloc_pages_errcheck(unsigned long allocated, } } +static void __init hugetlb_pages_alloc_boot_node(unsigned long start, unsigned long end, void *arg) +{ + struct hstate *h = (struct hstate *)arg; + int i, num = end - start; + nodemask_t node_alloc_noretry; + LIST_HEAD(folio_list); + int next_node = first_online_node; + + /* Bit mask controlling how hard we retry per-node allocations.*/ + nodes_clear(node_alloc_noretry); + + for (i = 0; i < num; ++i) { + struct folio *folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY], + &node_alloc_noretry, &next_node); + if (!folio) + break; + + list_move(&folio->lru, &folio_list); + cond_resched(); + } + + prep_and_add_allocated_folios(h, &folio_list); +} + static unsigned long __init hugetlb_gigantic_pages_alloc_boot(struct hstate *h) { unsigned long i; @@ -3525,26 +3550,40 @@ static unsigned long __init hugetlb_gigantic_pages_alloc_boot(struct hstate *h) static unsigned long __init hugetlb_pages_alloc_boot(struct hstate *h) { - unsigned long i; - struct folio *folio; - LIST_HEAD(folio_list); - nodemask_t node_alloc_noretry; - - /* Bit mask controlling how hard we retry per-node allocations.*/ - nodes_clear(node_alloc_noretry); + struct padata_mt_job job = { + .fn_arg = h, + .align = 1, + .numa_aware = true + }; - for (i = 0; i < h->max_huge_pages; ++i) { - folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY], - &node_alloc_noretry); - if (!folio) - break; - list_add(&folio->lru, &folio_list); - cond_resched(); - } + job.thread_fn = hugetlb_pages_alloc_boot_node; + job.start = 0; + job.size = h->max_huge_pages; - prep_and_add_allocated_folios(h, &folio_list); + /* + * job.max_threads is twice the num_node_state(N_MEMORY), + * + * Tests below indicate that a multiplier of 2 significantly improves + * performance, and although larger values also provide improvements, + * the gains are marginal. + * + * Therefore, choosing 2 as the multiplier strikes a good balance between + * enhancing parallel processing capabilities and maintaining efficient + * resource management. + * + * +------------+-------+-------+-------+-------+-------+ + * | multiplier | 1 | 2 | 3 | 4 | 5 | + * +------------+-------+-------+-------+-------+-------+ + * | 256G 2node | 358ms | 215ms | 157ms | 134ms | 126ms | + * | 2T 4node | 979ms | 679ms | 543ms | 489ms | 481ms | + * | 50G 2node | 71ms | 44ms | 37ms | 30ms | 31ms | + * +------------+-------+-------+-------+-------+-------+ + */ + job.max_threads = num_node_state(N_MEMORY) * 2; + job.min_chunk = h->max_huge_pages / num_node_state(N_MEMORY) / 2; + padata_do_multithreaded(&job); - return i; + return h->nr_huge_pages; } /*