From patchwork Fri Jan 26 15:24:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gang Li X-Patchwork-Id: 192654 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:fa17:b0:106:343:edcb with SMTP id ju23csp7389dyc; Fri, 26 Jan 2024 07:25:24 -0800 (PST) X-Google-Smtp-Source: AGHT+IGICch80LkRv8buoP6L0vO6Z6EoyIs9uy0DpJAmBZ5CEUHYaSyajhvpl4KEeaH72O9OpwcA X-Received: by 2002:a17:902:bd82:b0:1d7:c27:2d3c with SMTP id q2-20020a170902bd8200b001d70c272d3cmr1377185pls.6.1706282723858; Fri, 26 Jan 2024 07:25:23 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706282723; cv=pass; d=google.com; s=arc-20160816; b=SrTy+2gLYY6anrPpBIBA0g1zHMIVGQd71r02l8MMyb0Xg29sDv3o3Az1MdRJWVkuBR xSxFcqBxSLJsBsC0oF0P9Ux95Hhj3mv7WX2cV1cQ7beSW3c/PZngeDa1Irs6/c0c6j7D WnrHttx9nEiDRmauvAOK84rVO9LpyMWUOXyxj8+fHd+UogtWJpvaxq+L+zPolZDqc7mI 1vDL/9j3EOSPtNXC8gSqFunaLz7heLinaNcmdCRwDXaH1CbGDKOEdiSoflbtMSF44/oH fwOU8PYrfV5GI8QGQy7V4x6VexWoYdxnnGtPqZZrNA1aIsxo0crsPmve1B2JJRdP2xFC jNiA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=P9aNrm9Rl6VWe92w5Q1+H7U7+x0T3IY85gF9i1XPLD0=; fh=5n5b5+XOwLgGNDePhvfXysty56rnkWOs9jaHEODdVeE=; b=Kl9Bt5wjxbOWxRzOntZK51Prf+U2VfV/o8CNGSe7PpTil4cbBQCmeDs4X6IoTjlCko e/MrqZZxViI8m6oabyDUPIwtVe7hpoqS3EvTyPqOGTDf8AuAC11j1426hVrCH39pxK9k 90wPb1hW6xlf0v4PBjGVw0pyIhxcLkV337SvhrOH2M6oU3GpS+d5Ba4pgaGVKut6ZZ3x mBkqgSJ1+W9U+pbp+Zo6pD9zvQ1zXID6e3J4CDEpJI1E8YTdjQfvgFMPAjS33DexPG4Y VPJzonCa+sjlivFSLAA82fDXZ4HrTBvNUwury9FzJ8NJpXwWl84tgFpimEmkFsAwCIBe B84Q== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=Z2mGLWb9; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-40268-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-40268-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id m4-20020a170902db0400b001d773be7a63si1315415plx.26.2024.01.26.07.25.23 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 26 Jan 2024 07:25:23 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-40268-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=Z2mGLWb9; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-40268-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-40268-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 9BD10282429 for ; Fri, 26 Jan 2024 15:25:23 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 6CFF01D693; Fri, 26 Jan 2024 15:25:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="Z2mGLWb9" Received: from out-171.mta1.migadu.com (out-171.mta1.migadu.com [95.215.58.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1BBBC1CF91 for ; Fri, 26 Jan 2024 15:24:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706282701; cv=none; b=nD7q1HV94BtN6Qk3+J0WB8rWTQBbmrSNm/VIfEcUSrr2CgOXshyyjwEaXZKCmD0xyZVbdKlq31QagUYVNx6BG3Dg0cCk9QaSL5BRV9IGSiVRlOmrKl2ERf8hwgiBeuRXtchbjG9r0b60o+ibqiJwY1ZBhLcD+Emqj993k/53yLQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706282701; c=relaxed/simple; bh=JynJSsrcEmEyDiudVXgdLpdbaxMIxOCbRPacHoaDnbQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=elMGBtMqd3WwB9Ekd6QqBeNQ6tY/uipBvmVc7ELoyYnDM8QeGWqY6FsW2+41m75wS9ea+5/xQZuJjjOCrnAnjT9c3Me0QfAi6cT9oRUQ0tv5nnTu1KNx0+KgFwMtvTJKmePq/HEViyXB3l+n3dn8mnPR/eb4nyr0oBp8YbjeL4s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=Z2mGLWb9; arc=none smtp.client-ip=95.215.58.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1706282697; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=P9aNrm9Rl6VWe92w5Q1+H7U7+x0T3IY85gF9i1XPLD0=; b=Z2mGLWb9xrBuI0vOVD3E/NJFwQiuaje5gheixSf/WVC/8b+43XxkctLePzaM1b5Q45N43H jiACbC9uRwNgXG1Aj/uKYCxw/54Qrw4qopzYmDpDtD28Eu+qzsQAz6NeImrzqeJQQJWWKD oNdPxdCl8DH8xl4A48uiL4uj+fNJQvE= From: Gang Li To: David Hildenbrand , David Rientjes , Mike Kravetz , Muchun Song , Andrew Morton , Tim Chen Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, ligang.bdlg@bytedance.com, Gang Li Subject: [PATCH v5 1/7] hugetlb: code clean for hugetlb_hstate_alloc_pages Date: Fri, 26 Jan 2024 23:24:05 +0800 Message-Id: <20240126152411.1238072-2-gang.li@linux.dev> In-Reply-To: <20240126152411.1238072-1-gang.li@linux.dev> References: <20240126152411.1238072-1-gang.li@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789167113862059573 X-GMAIL-MSGID: 1789167113862059573 The readability of `hugetlb_hstate_alloc_pages` is poor. By cleaning the code, its readability can be improved, facilitating future modifications. This patch extracts two functions to reduce the complexity of `hugetlb_hstate_alloc_pages` and has no functional changes. - hugetlb_hstate_alloc_pages_node_specific() to handle iterates through each online node and performs allocation if necessary. - hugetlb_hstate_alloc_pages_report() report error during allocation. And the value of h->max_huge_pages is updated accordingly. Signed-off-by: Gang Li Tested-by: David Rientjes Reviewed-by: Muchun Song Reviewed-by: Tim Chen --- mm/hugetlb.c | 46 +++++++++++++++++++++++++++++----------------- 1 file changed, 29 insertions(+), 17 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 2cf78218dfe2e..20d0494424780 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3482,6 +3482,33 @@ static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) h->max_huge_pages_node[nid] = i; } +static bool __init hugetlb_hstate_alloc_pages_specific_nodes(struct hstate *h) +{ + int i; + bool node_specific_alloc = false; + + for_each_online_node(i) { + if (h->max_huge_pages_node[i] > 0) { + hugetlb_hstate_alloc_pages_onenode(h, i); + node_specific_alloc = true; + } + } + + return node_specific_alloc; +} + +static void __init hugetlb_hstate_alloc_pages_errcheck(unsigned long allocated, struct hstate *h) +{ + if (allocated < h->max_huge_pages) { + char buf[32]; + + string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, 32); + pr_warn("HugeTLB: allocating %lu of page size %s failed. Only allocated %lu hugepages.\n", + h->max_huge_pages, buf, allocated); + h->max_huge_pages = allocated; + } +} + /* * NOTE: this routine is called in different contexts for gigantic and * non-gigantic pages. @@ -3499,7 +3526,6 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) struct folio *folio; LIST_HEAD(folio_list); nodemask_t *node_alloc_noretry; - bool node_specific_alloc = false; /* skip gigantic hugepages allocation if hugetlb_cma enabled */ if (hstate_is_gigantic(h) && hugetlb_cma_size) { @@ -3508,14 +3534,7 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) } /* do node specific alloc */ - for_each_online_node(i) { - if (h->max_huge_pages_node[i] > 0) { - hugetlb_hstate_alloc_pages_onenode(h, i); - node_specific_alloc = true; - } - } - - if (node_specific_alloc) + if (hugetlb_hstate_alloc_pages_specific_nodes(h)) return; /* below will do all node balanced alloc */ @@ -3558,14 +3577,7 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) /* list will be empty if hstate_is_gigantic */ prep_and_add_allocated_folios(h, &folio_list); - if (i < h->max_huge_pages) { - char buf[32]; - - string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, 32); - pr_warn("HugeTLB: allocating %lu of page size %s failed. Only allocated %lu hugepages.\n", - h->max_huge_pages, buf, i); - h->max_huge_pages = i; - } + hugetlb_hstate_alloc_pages_errcheck(i, h); kfree(node_alloc_noretry); } From patchwork Fri Jan 26 15:24:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gang Li X-Patchwork-Id: 192655 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:fa17:b0:106:343:edcb with SMTP id ju23csp7602dyc; Fri, 26 Jan 2024 07:25:47 -0800 (PST) X-Google-Smtp-Source: AGHT+IGJ4IXzVPhx0Fm3Rmeto2MYV4pOQ3q2RHOIhcgE50o2n9AJRreXCReOm/eC1WYc1wEZmYIp X-Received: by 2002:a05:6a20:6056:b0:19c:6499:b921 with SMTP id s22-20020a056a20605600b0019c6499b921mr1098465pza.92.1706282746927; Fri, 26 Jan 2024 07:25:46 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706282746; cv=pass; d=google.com; s=arc-20160816; b=0lAVFp20L0VdNpLjJjG0umbNzfMhlQOv3H7nvpIvYZeVkE5IMJerBp7e1UP3UD9z+Y ymJthqVrFNpTMSyk91MTmdnMfu7E1W7ZAMHwxgYA7Fws24uNMmYjuHtIGOoAyoXTbpKu 4Z5YJ5CEUasm8LWYdBxTONVsMWFomnvdOee/ycXBrGx4Frse2Uwr0G1wqUjeY6PKGWb0 LzNi2XB2QB0/MAL/da/OJEDmLesIDVhh77d/SglJO1VkUCyd//3sqrBgES56wu/S+Xbv ZfANZpYEnBZ/lLtFSL6j7jSF4Qlm/exEVZbeFd4I3YbRou8DbgDAtZmQOJixv7v+Z4OF qiMw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=EpmnH1heqxiT9L2xvocYTAQyyeV+M8p6qSwgZH/mouw=; fh=5n5b5+XOwLgGNDePhvfXysty56rnkWOs9jaHEODdVeE=; b=gxME1UeBD4yBNIcCuAUZf7JQGk4uUdlKZtSG1PDNPi1iis1kiAOQ+Z968zL0mc2rGa Uih+fZtHZ4swzmqh/MyHbO3pz4x9sGApdUBoayAHNPToWU4adpkiU9iU/f7AtBOrfvRb D3H3+zQYTFZbBxxAGeCEiwPuFClW1NP9u1LP5N67cHAw6sEMXp/Cty91B8s453h7Jlvy 5BHz5hCyZ0zU0BXG722JMxLg54cbuZ9GSo+6ieOuART12041MFSlha9JxmYMz+8EGPbq Qc8JuPVlSWJlEsA/nmed4MORWV7XBsIhEOfxdaouG7domUbSYnOYYe2vdZNnzsBAs4AF +c3A== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=NIN5KgkQ; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-40270-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-40270-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id b30-20020a63715e000000b005b92edaa151si1304815pgn.739.2024.01.26.07.25.46 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 26 Jan 2024 07:25:46 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-40270-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=NIN5KgkQ; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-40270-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-40270-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 97EF1282045 for ; Fri, 26 Jan 2024 15:25:46 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id E01F01DDD5; Fri, 26 Jan 2024 15:25:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="NIN5KgkQ" Received: from out-182.mta1.migadu.com (out-182.mta1.migadu.com [95.215.58.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 810D61D6BD for ; Fri, 26 Jan 2024 15:25:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706282709; cv=none; b=HfbHa9vTwFIagJU/ESYP7lVuiemUeNWKBm6kuAEy++4xNv5l5r8VdqvaFV4Sj4VMKf58DW/RKd2x2MPjlRllKUwE/jnxfOrpehcUoZMXRMKYBG1sV1C2p8xL/qk6w8ZOSIZF2MArFHhE/OJqVj3pP9KKUn4So8u+rgwe+s+nv6U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706282709; c=relaxed/simple; bh=YikDHBkjoqd+ws8wiuocM92jiPsOsAd+AX07ms88HyE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=agKZdm0bJtJfREKQZAuGFmmfv/BCLo4Z8D1KTmMBB//3pJ7FfRBoQFVQ1kJUL9aSge66lDutpDekMfYgEbXhDHpWxZ186GIhddlxSrkw9fTwmdOBCdoLE5U3lAiIIxcMgtCwpOr1Ff/NYjKteG1EJ69OSuuaRJjibJe1ZJt/bNI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=NIN5KgkQ; arc=none smtp.client-ip=95.215.58.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1706282704; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EpmnH1heqxiT9L2xvocYTAQyyeV+M8p6qSwgZH/mouw=; b=NIN5KgkQ6ktwWa4GiDjUdAl7RaNOrgEmptJXjSAwQxyyMDJcoijL2+TDpy7yOmpIZkDhsL OSnefDgiw31H9dPHL9T6oBqYiSboHE9E7/g4GRKZgZb0/NDBZi/sa0bL1SshUOSqTeY3cu /G8p+dcISN/5OT93ypM/kpclSHGR/Kg= From: Gang Li To: David Hildenbrand , David Rientjes , Mike Kravetz , Muchun Song , Andrew Morton , Tim Chen Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, ligang.bdlg@bytedance.com, Gang Li Subject: [PATCH v5 2/7] hugetlb: split hugetlb_hstate_alloc_pages Date: Fri, 26 Jan 2024 23:24:06 +0800 Message-Id: <20240126152411.1238072-3-gang.li@linux.dev> In-Reply-To: <20240126152411.1238072-1-gang.li@linux.dev> References: <20240126152411.1238072-1-gang.li@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789167137740044254 X-GMAIL-MSGID: 1789167137740044254 1G and 2M huge pages have different allocation and initialization logic, which leads to subtle differences in parallelization. Therefore, it is appropriate to split hugetlb_hstate_alloc_pages into gigantic and non-gigantic. This patch has no functional changes. Signed-off-by: Gang Li Tested-by: David Rientjes Reviewed-by: Tim Chen Reviewed-by: Muchun Song --- mm/hugetlb.c | 87 ++++++++++++++++++++++++++-------------------------- 1 file changed, 43 insertions(+), 44 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 20d0494424780..00bbf7442eb6c 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3509,6 +3509,43 @@ static void __init hugetlb_hstate_alloc_pages_errcheck(unsigned long allocated, } } +static unsigned long __init hugetlb_gigantic_pages_alloc_boot(struct hstate *h) +{ + unsigned long i; + + for (i = 0; i < h->max_huge_pages; ++i) { + if (!alloc_bootmem_huge_page(h, NUMA_NO_NODE)) + break; + cond_resched(); + } + + return i; +} + +static unsigned long __init hugetlb_pages_alloc_boot(struct hstate *h) +{ + unsigned long i; + struct folio *folio; + LIST_HEAD(folio_list); + nodemask_t node_alloc_noretry; + + /* Bit mask controlling how hard we retry per-node allocations.*/ + nodes_clear(node_alloc_noretry); + + for (i = 0; i < h->max_huge_pages; ++i) { + folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY], + &node_alloc_noretry); + if (!folio) + break; + list_add(&folio->lru, &folio_list); + cond_resched(); + } + + prep_and_add_allocated_folios(h, &folio_list); + + return i; +} + /* * NOTE: this routine is called in different contexts for gigantic and * non-gigantic pages. @@ -3522,10 +3559,7 @@ static void __init hugetlb_hstate_alloc_pages_errcheck(unsigned long allocated, */ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) { - unsigned long i; - struct folio *folio; - LIST_HEAD(folio_list); - nodemask_t *node_alloc_noretry; + unsigned long allocated; /* skip gigantic hugepages allocation if hugetlb_cma enabled */ if (hstate_is_gigantic(h) && hugetlb_cma_size) { @@ -3538,47 +3572,12 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) return; /* below will do all node balanced alloc */ - if (!hstate_is_gigantic(h)) { - /* - * Bit mask controlling how hard we retry per-node allocations. - * Ignore errors as lower level routines can deal with - * node_alloc_noretry == NULL. If this kmalloc fails at boot - * time, we are likely in bigger trouble. - */ - node_alloc_noretry = kmalloc(sizeof(*node_alloc_noretry), - GFP_KERNEL); - } else { - /* allocations done at boot time */ - node_alloc_noretry = NULL; - } - - /* bit mask controlling how hard we retry per-node allocations */ - if (node_alloc_noretry) - nodes_clear(*node_alloc_noretry); - - for (i = 0; i < h->max_huge_pages; ++i) { - if (hstate_is_gigantic(h)) { - /* - * gigantic pages not added to list as they are not - * added to pools now. - */ - if (!alloc_bootmem_huge_page(h, NUMA_NO_NODE)) - break; - } else { - folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY], - node_alloc_noretry); - if (!folio) - break; - list_add(&folio->lru, &folio_list); - } - cond_resched(); - } - - /* list will be empty if hstate_is_gigantic */ - prep_and_add_allocated_folios(h, &folio_list); + if (hstate_is_gigantic(h)) + allocated = hugetlb_gigantic_pages_alloc_boot(h); + else + allocated = hugetlb_pages_alloc_boot(h); - hugetlb_hstate_alloc_pages_errcheck(i, h); - kfree(node_alloc_noretry); + hugetlb_hstate_alloc_pages_errcheck(allocated, h); } static void __init hugetlb_init_hstates(void) From patchwork Fri Jan 26 15:24:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gang Li X-Patchwork-Id: 192669 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:fa17:b0:106:343:edcb with SMTP id ju23csp18951dyc; Fri, 26 Jan 2024 07:44:32 -0800 (PST) X-Google-Smtp-Source: AGHT+IEuH4I/Y1tzJpQagRBoej/E7wA5P7iIW1Slo2SUaAfVqPKr3Y//hlwvDPDzD4szp2eahqJZ X-Received: by 2002:a05:6870:a452:b0:214:2a08:897d with SMTP id n18-20020a056870a45200b002142a08897dmr1965538oal.46.1706283871973; Fri, 26 Jan 2024 07:44:31 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706283871; cv=pass; d=google.com; s=arc-20160816; b=RwT1I3bcsv9OWMWNTOGldLIl4KM1vun+0X4NBV6KzXMBlnri3u5l3P4KvF1q7DhuN9 3nWtjMfIrAcI3yW+9EFqlabpI0qwDpy7uc6oY0PUcHqmk5jQpprLuAyOYEKbsoQfEEo0 Twa+zQxyWv8lM0SXuUYT6vB0QXaEdUDPmBmWGQQYBCo+6mMU5kOFgeLUvVvVRVo2c4d7 8lbCh1zvywP5F/tnTtHugbmYYdV8HdU9iRvQkWBCgZQT+dJq6HUj7ckb9FKdAIvKAh86 sYkmqRgFgNVs9z35OiCact/ECyIU6cUHvmQmo2tepXLsLLfMvmWA49N0eDL9yM2rtvO5 gvlw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=RmrMLc6E2IVqKhV8ZhRU9VeFjN80rDuhVxh0u6+eJMA=; fh=5n5b5+XOwLgGNDePhvfXysty56rnkWOs9jaHEODdVeE=; b=nSlHuzqGrNWIRAa2boveggY1zk0u3jUG2uQ0qyQXRjGwh7D199/fXbfPXTD6UoKS8Q 0vqfxXBzjno6DjV+3/ygdrp0vj8hvtxLUyhVrw3h8UT4DcRJvW4M9/GotAwnpzItIvfA 627mkofPYWPUdtv9CTVUxwxOpJMLRbRyotgT/E+GKKojKoH2h0v0HoidSc4T3bBEMkOn yjKj4ODBadDVjoIc72f8MVAb1TOpzIZH63Lgs4jDGigXaQdvlYpkmJ6MeFRWHP2c87UO LIXQyE29RgSzUf+MF817wCXawTkg7nipurI1W59iopHX/2fUsDuLzXmV09bXUDhCjiQ9 +47w== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=wyYlzc34; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-40271-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-40271-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [147.75.48.161]) by mx.google.com with ESMTPS id 7-20020a630d47000000b005cfbf651e25si1260310pgn.111.2024.01.26.07.44.31 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 26 Jan 2024 07:44:31 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-40271-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=wyYlzc34; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-40271-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-40271-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 41262B2783D for ; Fri, 26 Jan 2024 15:26:09 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 5B1321D54C; Fri, 26 Jan 2024 15:25:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="wyYlzc34" Received: from out-178.mta1.migadu.com (out-178.mta1.migadu.com [95.215.58.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 162791DFE1 for ; Fri, 26 Jan 2024 15:25:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706282718; cv=none; b=uKyOHJuQF5R6Zb8yvZIOwS5Ct1h5LPl+mmtP5CHczLemnpAv4Li8pLjZSglDan6qnoMKKkODvImBFH0mxZxtO9WF3+gJiOhNBVa9pgoA417rd0K6DjjrfnA6LAtSMdZFSl/N42/aPlG3eCXOh5VG0MOuMwYWw+NNffJQpv0Jgx8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706282718; c=relaxed/simple; bh=A7DeycgaAylfyaGHNXPeWa+Y5alq9KqCBr3DPXdOYWA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=kCeFITXaapkNiwwNqgJxqwXW1fzVogXfo4jlWK+oVR475/z5/thytWCMOefS9q3V0MqCONERFGtf2ib0/FotJ8o646K1ARMimk+VfokEvmL7a5FjI1F5Jtel2XzVdhhBZv9N7fkNry8yuNgdh0TzJMYRU8tRk6UxIxuCotpNrtg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=wyYlzc34; arc=none smtp.client-ip=95.215.58.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1706282715; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RmrMLc6E2IVqKhV8ZhRU9VeFjN80rDuhVxh0u6+eJMA=; b=wyYlzc34oaBoDa42QsBypygqXrsoGKHAfb+deg3SuOPDAmbhiv09g6PCqoW3PG4AJbHsVA hz3+64qyLRhxOGW5ACJiiomiu2sHCxRKxpEOC4gAbDn/H6YePZadVNJJ/H4FtQXG7IOOjG 0L+u9CTOwvqS2DBzIXlE1TKDr+2kLOc= From: Gang Li To: David Hildenbrand , David Rientjes , Mike Kravetz , Muchun Song , Andrew Morton , Tim Chen Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, ligang.bdlg@bytedance.com, Gang Li Subject: [PATCH v5 3/7] padata: dispatch works on different nodes Date: Fri, 26 Jan 2024 23:24:07 +0800 Message-Id: <20240126152411.1238072-4-gang.li@linux.dev> In-Reply-To: <20240126152411.1238072-1-gang.li@linux.dev> References: <20240126152411.1238072-1-gang.li@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789168317610303558 X-GMAIL-MSGID: 1789168317610303558 When a group of tasks that access different nodes are scheduled on the same node, they may encounter bandwidth bottlenecks and access latency. Thus, numa_aware flag is introduced here, allowing tasks to be distributed across different nodes to fully utilize the advantage of multi-node systems. Signed-off-by: Gang Li Tested-by: David Rientjes Reviewed-by: Muchun Song Reviewed-by: Tim Chen --- include/linux/padata.h | 2 ++ kernel/padata.c | 14 ++++++++++++-- mm/mm_init.c | 1 + 3 files changed, 15 insertions(+), 2 deletions(-) diff --git a/include/linux/padata.h b/include/linux/padata.h index 495b16b6b4d72..8f418711351bc 100644 --- a/include/linux/padata.h +++ b/include/linux/padata.h @@ -137,6 +137,7 @@ struct padata_shell { * appropriate for one worker thread to do at once. * @max_threads: Max threads to use for the job, actual number may be less * depending on task size and minimum chunk size. + * @numa_aware: Distribute jobs to different nodes with CPU in a round robin fashion. */ struct padata_mt_job { void (*thread_fn)(unsigned long start, unsigned long end, void *arg); @@ -146,6 +147,7 @@ struct padata_mt_job { unsigned long align; unsigned long min_chunk; int max_threads; + bool numa_aware; }; /** diff --git a/kernel/padata.c b/kernel/padata.c index 179fb1518070c..e3f639ff16707 100644 --- a/kernel/padata.c +++ b/kernel/padata.c @@ -485,7 +485,8 @@ void __init padata_do_multithreaded(struct padata_mt_job *job) struct padata_work my_work, *pw; struct padata_mt_job_state ps; LIST_HEAD(works); - int nworks; + int nworks, nid; + static atomic_t last_used_nid __initdata; if (job->size == 0) return; @@ -517,7 +518,16 @@ void __init padata_do_multithreaded(struct padata_mt_job *job) ps.chunk_size = roundup(ps.chunk_size, job->align); list_for_each_entry(pw, &works, pw_list) - queue_work(system_unbound_wq, &pw->pw_work); + if (job->numa_aware) { + int old_node = atomic_read(&last_used_nid); + + do { + nid = next_node_in(old_node, node_states[N_CPU]); + } while (!atomic_try_cmpxchg(&last_used_nid, &old_node, nid)); + queue_work_node(nid, system_unbound_wq, &pw->pw_work); + } else { + queue_work(system_unbound_wq, &pw->pw_work); + } /* Use the current thread, which saves starting a workqueue worker. */ padata_work_init(&my_work, padata_mt_helper, &ps, PADATA_WORK_ONSTACK); diff --git a/mm/mm_init.c b/mm/mm_init.c index 2c19f5515e36c..549e76af8f82a 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -2231,6 +2231,7 @@ static int __init deferred_init_memmap(void *data) .align = PAGES_PER_SECTION, .min_chunk = PAGES_PER_SECTION, .max_threads = max_threads, + .numa_aware = false, }; padata_do_multithreaded(&job); From patchwork Fri Jan 26 15:24:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gang Li X-Patchwork-Id: 192670 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:fa17:b0:106:343:edcb with SMTP id ju23csp19064dyc; Fri, 26 Jan 2024 07:44:44 -0800 (PST) X-Google-Smtp-Source: AGHT+IH1dQtvR6KlMgSoYrZj73row2EyONFrP07tt7jJVGPIVCB9Cn1C8vVtcpvdR0hfp2LxoDne X-Received: by 2002:a17:90a:6689:b0:294:5a83:8d60 with SMTP id m9-20020a17090a668900b002945a838d60mr23123pjj.71.1706283884285; Fri, 26 Jan 2024 07:44:44 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706283884; cv=pass; d=google.com; s=arc-20160816; b=Nj60FF84HQPuuhhdkLsFRgQ8UOcckA8y66anujusEdjQdf1rPasBnHwxVzNMpn2aQ/ sURb3UzW8ar3DbBypz1U/FwZoLCaecTFL/y3vXgE3xpZrShqCx3TpMk+a3mT0/naYBZB i+BLym/OVaTYSWY+G/0aWmE8NtxFTp6U6baKBvtTJeTaeKUP3XjvmBm8OIZnJMazBjF8 H9hIuKpctl5JAu1BGpCGt3OzfjpN54ZWslz1PUC29bWdPK/txKDyQ5RsQ+U9Q2Sbwfri 0nH8NtRfYYkHzw09LEbFE6NtyWaIpqK+kpDtDIOyxN12bxY42bKy4PU0iX7XYYmVhMsN VITA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=DgUhGSnAajARz2sR8zEsgOZsTNsgDGT2+wOJsJvBtFE=; fh=5n5b5+XOwLgGNDePhvfXysty56rnkWOs9jaHEODdVeE=; b=n3UNbpgoSZunMTmA/NM+wy6l3TX9OphX2V6gDAAcJYtKZdlxf3OfJ4ppCv4tSyZfZO /FTfmZG5vTXum9JiwaROhwKoHBqOcizyhibSmj3JDNog5D6pBthh/1UwPBEmIGAI+D2L GWLREJHP+epV9UGKR7rAsXuu+cjJQOib5PdtM+Il/xAmLBhh3rfyWLjyaXcP4BqMaaK8 mLTujhGBlMyiodf4Q+VoCWA9TQYbjULLxgLW+A5JdwUbnKYUoPgE9HZQwRcDTdyZd6jV TvUvWjylvvX6MSadVMoWsLUrbJQVlvgOGXwiJUXy/HdMEf3ZgrzveqJUHWHfrGfCcNbn Ruxw== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b="gXsmiu/Z"; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-40272-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-40272-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [147.75.48.161]) by mx.google.com with ESMTPS id l8-20020a17090add8800b0029071a5e3fbsi3363670pjv.115.2024.01.26.07.44.43 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 26 Jan 2024 07:44:44 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-40272-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b="gXsmiu/Z"; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-40272-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-40272-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id EF680B27CE1 for ; Fri, 26 Jan 2024 15:26:21 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 090011E533; Fri, 26 Jan 2024 15:25:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="gXsmiu/Z" Received: from out-187.mta1.migadu.com (out-187.mta1.migadu.com [95.215.58.187]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 512CC1DFD8 for ; Fri, 26 Jan 2024 15:25:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.187 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706282730; cv=none; b=JuHPDDXdetb9k0PK/TozbDxdUqE2sFh/j3ziVLBItKtcYziG4qHi9dzbZdTbIwPEzwxnKQEdIFdkjFdNO1oNdGwPTbLsSplZm6GH+IR8baEGfHpw6HmS448Ei3d+Szj4EG2y9ZYPOYaun4GQ4jqZidJ8lLA5oTJNWb2pNuRr88Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706282730; c=relaxed/simple; bh=ibrxswcbU4iJtA1ZyA/2KqNtGq9c0Eu/5DaeuCu3KLM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=WwhyEsKtN1tK9oMJ+hBemfnrWFuXFNn6Q1RsMOgRRUFYa26y+vMKvMODNgb20CZ3RkpsylAn133rPzwVROd3Ei34wgcWnvkbA7WlPkY5WowYhSoqFc0Rvf4PqBT1nDq7q1jYjmrQGdewlfamN13caXv0nnRCSTslFnuGHPCxaGc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=gXsmiu/Z; arc=none smtp.client-ip=95.215.58.187 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1706282726; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DgUhGSnAajARz2sR8zEsgOZsTNsgDGT2+wOJsJvBtFE=; b=gXsmiu/ZCf1nMQFyOdSPcIWiSFw45L9E3U4xHBNZLrlxxSNlwx6qWum6Nk/BZJ5TD6kyb2 jZji4uY+b0TrkNYU777z3nkzvP+RZPRiUs6uA84GSunLLr+1e8DmhHiDWLWBdX0usDWBhv Au8J+F364De+i5U9tAHRO6qcgDJNC1o= From: Gang Li To: David Hildenbrand , David Rientjes , Mike Kravetz , Muchun Song , Andrew Morton , Tim Chen Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, ligang.bdlg@bytedance.com, Gang Li Subject: [PATCH v5 4/7] hugetlb: pass *next_nid_to_alloc directly to for_each_node_mask_to_alloc Date: Fri, 26 Jan 2024 23:24:08 +0800 Message-Id: <20240126152411.1238072-5-gang.li@linux.dev> In-Reply-To: <20240126152411.1238072-1-gang.li@linux.dev> References: <20240126152411.1238072-1-gang.li@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789168330314783405 X-GMAIL-MSGID: 1789168330314783405 With parallelization of hugetlb allocation across different threads, each thread works on a differnet node to allocate pages from, instead of all allocating from a common node h->next_nid_to_alloc. To address this, it's necessary to assign a separate next_nid_to_alloc for each thread. Consequently, the hstate_next_node_to_alloc and for_each_node_mask_to_alloc have been modified to directly accept a *next_nid_to_alloc parameter, ensuring thread-specific allocation and avoiding concurrent access issues. Signed-off-by: Gang Li Tested-by: David Rientjes Reviewed-by: Tim Chen Reviewed-by: Muchun Song --- mm/hugetlb.c | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 00bbf7442eb6c..e4e8ffa1c145a 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1464,15 +1464,15 @@ static int get_valid_node_allowed(int nid, nodemask_t *nodes_allowed) * next node from which to allocate, handling wrap at end of node * mask. */ -static int hstate_next_node_to_alloc(struct hstate *h, +static int hstate_next_node_to_alloc(int *next_node, nodemask_t *nodes_allowed) { int nid; VM_BUG_ON(!nodes_allowed); - nid = get_valid_node_allowed(h->next_nid_to_alloc, nodes_allowed); - h->next_nid_to_alloc = next_node_allowed(nid, nodes_allowed); + nid = get_valid_node_allowed(*next_node, nodes_allowed); + *next_node = next_node_allowed(nid, nodes_allowed); return nid; } @@ -1495,10 +1495,10 @@ static int hstate_next_node_to_free(struct hstate *h, nodemask_t *nodes_allowed) return nid; } -#define for_each_node_mask_to_alloc(hs, nr_nodes, node, mask) \ +#define for_each_node_mask_to_alloc(next_node, nr_nodes, node, mask) \ for (nr_nodes = nodes_weight(*mask); \ nr_nodes > 0 && \ - ((node = hstate_next_node_to_alloc(hs, mask)) || 1); \ + ((node = hstate_next_node_to_alloc(next_node, mask)) || 1); \ nr_nodes--) #define for_each_node_mask_to_free(hs, nr_nodes, node, mask) \ @@ -2350,12 +2350,13 @@ static void prep_and_add_allocated_folios(struct hstate *h, */ static struct folio *alloc_pool_huge_folio(struct hstate *h, nodemask_t *nodes_allowed, - nodemask_t *node_alloc_noretry) + nodemask_t *node_alloc_noretry, + int *next_node) { gfp_t gfp_mask = htlb_alloc_mask(h) | __GFP_THISNODE; int nr_nodes, node; - for_each_node_mask_to_alloc(h, nr_nodes, node, nodes_allowed) { + for_each_node_mask_to_alloc(next_node, nr_nodes, node, nodes_allowed) { struct folio *folio; folio = only_alloc_fresh_hugetlb_folio(h, gfp_mask, node, @@ -3310,7 +3311,7 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) goto found; } /* allocate from next node when distributing huge pages */ - for_each_node_mask_to_alloc(h, nr_nodes, node, &node_states[N_MEMORY]) { + for_each_node_mask_to_alloc(&h->next_nid_to_alloc, nr_nodes, node, &node_states[N_MEMORY]) { m = memblock_alloc_try_nid_raw( huge_page_size(h), huge_page_size(h), 0, MEMBLOCK_ALLOC_ACCESSIBLE, node); @@ -3679,7 +3680,7 @@ static int adjust_pool_surplus(struct hstate *h, nodemask_t *nodes_allowed, VM_BUG_ON(delta != -1 && delta != 1); if (delta < 0) { - for_each_node_mask_to_alloc(h, nr_nodes, node, nodes_allowed) { + for_each_node_mask_to_alloc(&h->next_nid_to_alloc, nr_nodes, node, nodes_allowed) { if (h->surplus_huge_pages_node[node]) goto found; } @@ -3794,7 +3795,8 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, cond_resched(); folio = alloc_pool_huge_folio(h, nodes_allowed, - node_alloc_noretry); + node_alloc_noretry, + &h->next_nid_to_alloc); if (!folio) { prep_and_add_allocated_folios(h, &page_list); spin_lock_irq(&hugetlb_lock); From patchwork Fri Jan 26 15:24:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gang Li X-Patchwork-Id: 192671 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:fa17:b0:106:343:edcb with SMTP id ju23csp19228dyc; Fri, 26 Jan 2024 07:44:59 -0800 (PST) X-Google-Smtp-Source: AGHT+IErLfSK7GvoqmeKGY1ZwPhG+WywmSczOEiQegftrwhoy1Wb4xa+/uJl9SKSNQQh8za+JGg/ X-Received: by 2002:a17:902:dad1:b0:1d8:a827:c338 with SMTP id q17-20020a170902dad100b001d8a827c338mr498918plx.120.1706283898957; Fri, 26 Jan 2024 07:44:58 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706283898; cv=pass; d=google.com; s=arc-20160816; b=jWn67ZUXYz41ejFWku/R7EaL3OghAqFKjiQYFxh3aatsazjKQXTbkpLjZ2tMWd/ZB+ c1uFKPfN7l0TlkteiZ4rDvfP3qskQAWP9dxzvtak9YA9Mpkm5BpXoFTb81Ls7Xargby8 ACOfF8AVx4ag19ArxIF24+RVo3Yh/AOFjtThCOHoYKyGY9ixBf4bm9ykRvsx6Cms8lDS hEZ2B5T0/An5/8tBSEpMnATrxl2jxZkPnC+mZKPbg51eEESNXIjoOlj5C/x8iNcRMRXj 8ng/V9Mm6iqkkOyO/7fuhN2LgsaWqlAFlfTC2A/VXT6pDk//qkxi0oXmoaFDXSI8lp5O hISA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=Fxc6M49wsiZtuAiZJk8Hxt8gV/1ixSfY+RSkangwfeM=; fh=5n5b5+XOwLgGNDePhvfXysty56rnkWOs9jaHEODdVeE=; b=dkQ7Ye8C5KCzGdHVzaLTIlkV93AhBOKRJWPTRRObFLaMk/+eKU17dp8d07Uetkppd3 TQ2DKrt8QILBh0sgCdUJDxY+Snb4fnxb6Ce2TsbEtwMAHmGvwoEJr4kNSZWhHBXiZ26S K5cRTAgZWLakY9eeIntjBvf/JoaVs3Uwj4x20KmNAuco1eAfGgKnS/i1V7M9qzxVibNl VXl5kP8HNUOyT/6xKR54JWU9LRpI6fAOaNv4NvVsd7cP96drpzumfthWhCxZ52pYgubI M6uwOEqrSrxSsOkk4d10I9MAsytQKZD8tH7rIi415RSbJf+PIw48pVtINHM0g+4c5MjN v9yw== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=IkyvnIde; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-40273-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-40273-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [147.75.48.161]) by mx.google.com with ESMTPS id s17-20020a63ff51000000b005d67007b43dsi1312722pgk.571.2024.01.26.07.44.58 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 26 Jan 2024 07:44:58 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-40273-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=IkyvnIde; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-40273-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-40273-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 5AD6AB245E7 for ; Fri, 26 Jan 2024 15:26:34 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 91B701CF94; Fri, 26 Jan 2024 15:25:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="IkyvnIde" Received: from out-178.mta1.migadu.com (out-178.mta1.migadu.com [95.215.58.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E9AA51E873 for ; Fri, 26 Jan 2024 15:25:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706282735; cv=none; b=rHfJ5/p0Hd/rNk43YEeAqA1Skq7vHNaj2W+GCZ78JZoXuyA97nYSQShyrY3D9QAIdUgage1GtUFVMLt5KiBDC6SHDpp0kgYHjkfeM2q8w2JXYcRkp/JlC9VmHH1oVKNuzUHAB8WU+IvTlNL/GSqqwKOLM8qJm5KGlwjewnnSpu0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706282735; c=relaxed/simple; bh=nRdDqIo0CuuDkPFpmOy7jAUoA1VsUwtHB9YZhdW7xbg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=FIClnCYtE2AFBosulzS72LwJKM4IDRni/NGxHV3jIuu1mm38z7gueFF+/a6OxSRYZ+DsFebxVZV9UkFv2VID6mLHZMKdp+tqw98qQ/5JhaVa68ySlKTjsRkyI4miyhgFnbgo7pHdcGjgUnTdeV6mWEA25ltfLpyFJMcPVi1ewyA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=IkyvnIde; arc=none smtp.client-ip=95.215.58.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1706282732; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Fxc6M49wsiZtuAiZJk8Hxt8gV/1ixSfY+RSkangwfeM=; b=IkyvnIdeEAXzyNoF0LqdZSDTbiTOQSu5etCNgZoKti/tjJbcin7DkJcC3ohdXGZT0Tnwyh /g1l+hlYRjpoHgsQS3F7EjYdzCdbyPl6t77EACOC7usJara6gjWSefTrR2k8x+WlfEOi3s WjM+WuvVeiatPmm6FNEI9O1LnuQ1/Ow= From: Gang Li To: David Hildenbrand , David Rientjes , Mike Kravetz , Muchun Song , Andrew Morton , Tim Chen Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, ligang.bdlg@bytedance.com, Gang Li Subject: [PATCH v5 5/7] hugetlb: have CONFIG_HUGETLBFS select CONFIG_PADATA Date: Fri, 26 Jan 2024 23:24:09 +0800 Message-Id: <20240126152411.1238072-6-gang.li@linux.dev> In-Reply-To: <20240126152411.1238072-1-gang.li@linux.dev> References: <20240126152411.1238072-1-gang.li@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789168345708454914 X-GMAIL-MSGID: 1789168345708454914 Allow hugetlb use padata_do_multithreaded for parallel initialization. Select CONFIG_PADATA in this case. Signed-off-by: Gang Li Tested-by: David Rientjes Reviewed-by: Muchun Song --- fs/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/fs/Kconfig b/fs/Kconfig index ea2f77446080e..3abc107ab2fbd 100644 --- a/fs/Kconfig +++ b/fs/Kconfig @@ -261,6 +261,7 @@ menuconfig HUGETLBFS depends on X86 || SPARC64 || ARCH_SUPPORTS_HUGETLBFS || BROKEN depends on (SYSFS || SYSCTL) select MEMFD_CREATE + select PADATA help hugetlbfs is a filesystem backing for HugeTLB pages, based on ramfs. For architectures that support it, say Y here and read From patchwork Fri Jan 26 15:24:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gang Li X-Patchwork-Id: 192658 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:fa17:b0:106:343:edcb with SMTP id ju23csp8689dyc; Fri, 26 Jan 2024 07:27:39 -0800 (PST) X-Google-Smtp-Source: AGHT+IHct99T0XfF5oZwJhV99lU6A6UmqajxU9ttzuNtKfHkbOUBo4Hvlm13oNCFlqGZ+qRlSpvo X-Received: by 2002:a05:600c:b93:b0:40e:e8e3:40f1 with SMTP id fl19-20020a05600c0b9300b0040ee8e340f1mr83237wmb.369.1706282858899; Fri, 26 Jan 2024 07:27:38 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706282858; cv=pass; d=google.com; s=arc-20160816; b=YsJO65HGMEQKOWHoQm6mrtWzptMKoawJnUSD6qa3RosD8lnH+qptPqZEa/OFKTLyjS 7LWpOiLL+wep0xeSZBGeynTV5FibmHmbLgOwCsxd0V1HIbIxMZ1Viy+2NrErhzqHf5Iw lmbELijxCapLpvGNcXe6HJA+dSnxMKHWzyKTYay1WBKPw6m9dNuvh/i/6enBytPBx4hi +8rztdpkYw+PPmtW1+wdYZHg8FGYF0/1gE2jAfebqS1aEiKVeRfypuLOOQcLi7yy92gF yDAgzGornZe0K/N8IvpGnMguh+ErORyXY3l5u5QsiXNu7OUOlTq6B1iWQyN3oAvdKEWY xrkw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=jedycsf2hoazZl2VD5dKzn3Ogz/yNT2KcxNdOLYavB4=; fh=5n5b5+XOwLgGNDePhvfXysty56rnkWOs9jaHEODdVeE=; b=NvbuIdECICW3V3lidnMNxS7iuH4AK4xiCS6+9/arcfWy8AQqG/8s07+eEmJwjkj+Fc IoGw1FLZXdWz1fMnwnPXRej020/4zNyTdKimG7sOxnjRuh1Mux/iwILnC7B/umIDf65F NkORkOK677GUg+P/YjXRpEpa5E0UxgmC6JJck7uyrirxJ3JjSvRxs9pZOeDRAl1z2Bfc EjIMjxAKzmKKBRTbnSNVKrs8ov/tnuihK9QZ2p+CihVsJrYVG+McYk8Gd/WPqH8AkApz JKxvDLi57ICGWNyOvn1pjGBSr6bBGWCbF7d5vdFQbH1en8ELiB8QD5NTEN/KMv3/AJ/R jjcw== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=qwIcy4O8; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-40274-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-40274-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id i12-20020a170906a28c00b00a310709a5c4si655912ejz.305.2024.01.26.07.27.38 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 26 Jan 2024 07:27:38 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-40274-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=qwIcy4O8; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-40274-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-40274-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id C68B71F25BDF for ; Fri, 26 Jan 2024 15:26:46 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id D49A61DFF7; Fri, 26 Jan 2024 15:25:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="qwIcy4O8" Received: from out-187.mta1.migadu.com (out-187.mta1.migadu.com [95.215.58.187]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 733591D697 for ; Fri, 26 Jan 2024 15:25:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.187 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706282742; cv=none; b=KilYz0wc6iztOE3Ne7WJvZO+6fPaFGrTWIpHuOOgjfGuGTHxVSX0HjbFjb0ohPiivUbCm9vr9MtS4313VNFlYYSlZMuNeq1wbEu++ME2PATfunQyfSWtE0M5P+7wqpw05CegvJ3U2kBb6WUIsZmNa9ycy8W7sNTuhxeErxW72hU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706282742; c=relaxed/simple; bh=1MgYoEYA0TYAFs7M4fp1D81Bw+npp83XpLElfpWHMK8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=eBv3xNdrqoqeOqYEIjz0vH6pBkefL/CqZSOBqNQqBTO6HI0s/d4cJRRFtX+uXwzK3Ih2ISv1GxXJIB0kHFv2tKeguQuZkkIq3fnw+R34JKqYSKmOWAlNspPegdX7PIk4QL2wPuLf1cY8eng/4HSKz5rikYVqHd34nGXwDIRQwLg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=qwIcy4O8; arc=none smtp.client-ip=95.215.58.187 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1706282738; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jedycsf2hoazZl2VD5dKzn3Ogz/yNT2KcxNdOLYavB4=; b=qwIcy4O8oYc6ieAe8OwvpEKH4wyBee3Jlif+nxuPJKFmuXp+E3oYFgTjBUsVhVcTbe3PJO LbG1Yp4MWWpIydaJlX5rrJfhgvOcuD2EKuJO/Yme8P9fG6DM1ACjVdMyPO1FZ4Ag0JyV0D a0k79BNlAPNr4wc8ZdQcdQUIK48SoXc= From: Gang Li To: David Hildenbrand , David Rientjes , Mike Kravetz , Muchun Song , Andrew Morton , Tim Chen Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, ligang.bdlg@bytedance.com, Gang Li Subject: [PATCH v5 6/7] hugetlb: parallelize 2M hugetlb allocation and initialization Date: Fri, 26 Jan 2024 23:24:10 +0800 Message-Id: <20240126152411.1238072-7-gang.li@linux.dev> In-Reply-To: <20240126152411.1238072-1-gang.li@linux.dev> References: <20240126152411.1238072-1-gang.li@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789167254791290107 X-GMAIL-MSGID: 1789167254791290107 By distributing both the allocation and the initialization tasks across multiple threads, the initialization of 2M hugetlb will be faster, thereby improving the boot speed. Here are some test results: test case no patch(ms) patched(ms) saved ------------------- -------------- ------------- -------- 256c2T(4 node) 2M 3336 1051 68.52% 128c1T(2 node) 2M 1943 716 63.15% Signed-off-by: Gang Li Tested-by: David Rientjes Reviewed-by: Muchun Song --- mm/hugetlb.c | 73 ++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 56 insertions(+), 17 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index e4e8ffa1c145a..385840397bce5 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -35,6 +35,7 @@ #include #include #include +#include #include #include @@ -3510,6 +3511,30 @@ static void __init hugetlb_hstate_alloc_pages_errcheck(unsigned long allocated, } } +static void __init hugetlb_pages_alloc_boot_node(unsigned long start, unsigned long end, void *arg) +{ + struct hstate *h = (struct hstate *)arg; + int i, num = end - start; + nodemask_t node_alloc_noretry; + LIST_HEAD(folio_list); + int next_node = first_online_node; + + /* Bit mask controlling how hard we retry per-node allocations.*/ + nodes_clear(node_alloc_noretry); + + for (i = 0; i < num; ++i) { + struct folio *folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY], + &node_alloc_noretry, &next_node); + if (!folio) + break; + + list_move(&folio->lru, &folio_list); + cond_resched(); + } + + prep_and_add_allocated_folios(h, &folio_list); +} + static unsigned long __init hugetlb_gigantic_pages_alloc_boot(struct hstate *h) { unsigned long i; @@ -3525,26 +3550,40 @@ static unsigned long __init hugetlb_gigantic_pages_alloc_boot(struct hstate *h) static unsigned long __init hugetlb_pages_alloc_boot(struct hstate *h) { - unsigned long i; - struct folio *folio; - LIST_HEAD(folio_list); - nodemask_t node_alloc_noretry; - - /* Bit mask controlling how hard we retry per-node allocations.*/ - nodes_clear(node_alloc_noretry); + struct padata_mt_job job = { + .fn_arg = h, + .align = 1, + .numa_aware = true + }; - for (i = 0; i < h->max_huge_pages; ++i) { - folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY], - &node_alloc_noretry); - if (!folio) - break; - list_add(&folio->lru, &folio_list); - cond_resched(); - } + job.thread_fn = hugetlb_pages_alloc_boot_node; + job.start = 0; + job.size = h->max_huge_pages; - prep_and_add_allocated_folios(h, &folio_list); + /* + * job.max_threads is twice the num_node_state(N_MEMORY), + * + * Tests below indicate that a multiplier of 2 significantly improves + * performance, and although larger values also provide improvements, + * the gains are marginal. + * + * Therefore, choosing 2 as the multiplier strikes a good balance between + * enhancing parallel processing capabilities and maintaining efficient + * resource management. + * + * +------------+-------+-------+-------+-------+-------+ + * | multiplier | 1 | 2 | 3 | 4 | 5 | + * +------------+-------+-------+-------+-------+-------+ + * | 256G 2node | 358ms | 215ms | 157ms | 134ms | 126ms | + * | 2T 4node | 979ms | 679ms | 543ms | 489ms | 481ms | + * | 50G 2node | 71ms | 44ms | 37ms | 30ms | 31ms | + * +------------+-------+-------+-------+-------+-------+ + */ + job.max_threads = num_node_state(N_MEMORY) * 2; + job.min_chunk = h->max_huge_pages / num_node_state(N_MEMORY) / 2; + padata_do_multithreaded(&job); - return i; + return h->nr_huge_pages; } /* From patchwork Fri Jan 26 15:24:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gang Li X-Patchwork-Id: 192656 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:fa17:b0:106:343:edcb with SMTP id ju23csp8343dyc; Fri, 26 Jan 2024 07:27:04 -0800 (PST) X-Google-Smtp-Source: AGHT+IHaoS6hAmE72yj5Tfe2Y4b9jU2Q9/fgCyBuALEc6GP0fLSv42glN7mJXylQTu+vji+Sm+gN X-Received: by 2002:ad4:4ee3:0:b0:685:977c:2c50 with SMTP id dv3-20020ad44ee3000000b00685977c2c50mr1710384qvb.119.1706282823850; Fri, 26 Jan 2024 07:27:03 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706282823; cv=pass; d=google.com; s=arc-20160816; b=lju/MW+Se8aT1nNgk9pq3xVVz2C6ose0agJj/G3AkdGKZPZFapVohRFSB5RBDqc/Vr I8jwIgAzQTP/sETG84SZWSccAj/DU+du91a8f07QhvXSl3UxtXh/zv5Md6obyfm0/YwP V391c0Yyd/XPUifxOgpDaKIVSCugvUxgIesaPXpqxD+KsMIIiZBQ2HpxSFdXqepb2k0l QEHot2jc3S60W/IQ1IU1aK/4mdaV25l9rS+xxS4DfvvgXL2RKgaZc/OJwaucpkGJ6j52 8dvRuG+0OQSqWFtj7NTM0iCjZaJj0AOkCsXPXy8fqTQAAJHWyg2o8TFWLfPv6rHVl2Sa T1iA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=0iArgVEROeDskUb/co444WxwcA+iCGyxYPbAZNymH1g=; fh=5n5b5+XOwLgGNDePhvfXysty56rnkWOs9jaHEODdVeE=; b=KqmcZg/7asYd10faVk/UjXh9eD5JJTW3VNd2ZTrtBSkcOXtOgoFFKS2YwoVceu3Yex a4P6X1QDWPEvpTv9YyH0E24+9t/Ibf9B4dLad9RyhfGWdhb98OvooNY4RGLaJ+yFMwx0 2kWUTrlmiqLmTK+4IZYUFNxACZGRnSE9HHuNbgKqi9dmHFV6MVTgZp/vMbhXxcC+Tj4p X9Mof2p8/wCu53mUWkvlxg/fzn+JP/IKYjmmcWmEJsRNjspLYUR1pYjvDDa6w6pJ09Yl pWhNTvalrNogZIJbgDiUV/nImuxPw8A6LtuYkNV29Om10MNLopv2QlxoBJWJoqITGrxV uj1g== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=dYyUEugy; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-40275-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-40275-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id kc28-20020a056214411c00b006819bda6ad3si1464212qvb.453.2024.01.26.07.27.03 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 26 Jan 2024 07:27:03 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-40275-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=dYyUEugy; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-40275-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-40275-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id EF2C91C21200 for ; Fri, 26 Jan 2024 15:26:58 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id EEC581EA7A; Fri, 26 Jan 2024 15:25:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="dYyUEugy" Received: from out-172.mta1.migadu.com (out-172.mta1.migadu.com [95.215.58.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A73DF1D55F for ; Fri, 26 Jan 2024 15:25:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706282750; cv=none; b=Ss5iokE7sHK8BmsARIOHM0YlWtY/MokLW3PoYiYBYiNBquuHclaMXw7sVlCf//tciHKo3NaB7+fOPo4hOq0F2YC5oQq93jc+AO3Gt/ejGcJ2LcWzduFEJygCqbVE5ZD84gA10oP0hZ3B3MyHudmScWhfVfXepWenXT50smbOh6A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706282750; c=relaxed/simple; bh=c2eGy8j5jjUuuogbCux3HCSCTf68lnabsdin1Kloo8E=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=aGHeOHZB3lOEiUegydXOiZgm5PRGeGnViaOyPIkYjZRt1gk7iyiMgVuwJ19D/mAz5J1IUCukz33V9FzxDaKir+3CAOiVhioTgmGEb6uqmyMns33T3fxPs4cVzcpli8erQHlWHhx0gzhnPaQb87+syqFx0ic/vATrUoAsOQvYwUU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=dYyUEugy; arc=none smtp.client-ip=95.215.58.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1706282746; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0iArgVEROeDskUb/co444WxwcA+iCGyxYPbAZNymH1g=; b=dYyUEugy+xtDscuDVSmLIJChMKlpJo7hUQlBy8G4LmvOhGPDib/TUY2KXKt7cRInLKAa1I b1QrLCFOaKrSbCu0c2Y3x47nwNUIIzTHzZvBCbhCbYQlzL25YqVImvkO36hN0p1LqAbwUF 8bvuvbI8HTBoL2mr29vGEt4GJQlLJPQ= From: Gang Li To: David Hildenbrand , David Rientjes , Mike Kravetz , Muchun Song , Andrew Morton , Tim Chen Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, ligang.bdlg@bytedance.com, Gang Li Subject: [PATCH v5 7/7] hugetlb: parallelize 1G hugetlb initialization Date: Fri, 26 Jan 2024 23:24:11 +0800 Message-Id: <20240126152411.1238072-8-gang.li@linux.dev> In-Reply-To: <20240126152411.1238072-1-gang.li@linux.dev> References: <20240126152411.1238072-1-gang.li@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789167218602988339 X-GMAIL-MSGID: 1789167218602988339 Optimizing the initialization speed of 1G huge pages through parallelization. 1G hugetlbs are allocated from bootmem, a process that is already very fast and does not currently require optimization. Therefore, we focus on parallelizing only the initialization phase in `gather_bootmem_prealloc`. Here are some test results: test case no patch(ms) patched(ms) saved ------------------- -------------- ------------- -------- 256c2T(4 node) 1G 4745 2024 57.34% 128c1T(2 node) 1G 3358 1712 49.02% 12T 1G 77000 18300 76.23% Signed-off-by: Gang Li Tested-by: David Rientjes Reviewed-by: Muchun Song --- arch/powerpc/mm/hugetlbpage.c | 2 +- include/linux/hugetlb.h | 2 +- mm/hugetlb.c | 44 ++++++++++++++++++++++++++++------- 3 files changed, 38 insertions(+), 10 deletions(-) diff --git a/arch/powerpc/mm/hugetlbpage.c b/arch/powerpc/mm/hugetlbpage.c index 0a540b37aab62..a1651d5471862 100644 --- a/arch/powerpc/mm/hugetlbpage.c +++ b/arch/powerpc/mm/hugetlbpage.c @@ -226,7 +226,7 @@ static int __init pseries_alloc_bootmem_huge_page(struct hstate *hstate) return 0; m = phys_to_virt(gpage_freearray[--nr_gpages]); gpage_freearray[nr_gpages] = 0; - list_add(&m->list, &huge_boot_pages); + list_add(&m->list, &huge_boot_pages[0]); m->hstate = hstate; return 1; } diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index c1ee640d87b11..77b30a8c6076b 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -178,7 +178,7 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma, struct address_space *hugetlb_page_mapping_lock_write(struct page *hpage); extern int sysctl_hugetlb_shm_group; -extern struct list_head huge_boot_pages; +extern struct list_head huge_boot_pages[MAX_NUMNODES]; /* arch callbacks */ diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 385840397bce5..eee0c456f6571 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -69,7 +69,7 @@ static bool hugetlb_cma_folio(struct folio *folio, unsigned int order) #endif static unsigned long hugetlb_cma_size __initdata; -__initdata LIST_HEAD(huge_boot_pages); +__initdata struct list_head huge_boot_pages[MAX_NUMNODES]; /* for command line parsing */ static struct hstate * __initdata parsed_hstate; @@ -3301,7 +3301,7 @@ int alloc_bootmem_huge_page(struct hstate *h, int nid) int __alloc_bootmem_huge_page(struct hstate *h, int nid) { struct huge_bootmem_page *m = NULL; /* initialize for clang */ - int nr_nodes, node; + int nr_nodes, node = nid; /* do node specific alloc */ if (nid != NUMA_NO_NODE) { @@ -3339,7 +3339,7 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) huge_page_size(h) - PAGE_SIZE); /* Put them into a private list first because mem_map is not up yet */ INIT_LIST_HEAD(&m->list); - list_add(&m->list, &huge_boot_pages); + list_add(&m->list, &huge_boot_pages[node]); m->hstate = h; return 1; } @@ -3390,8 +3390,6 @@ static void __init prep_and_add_bootmem_folios(struct hstate *h, /* Send list for bulk vmemmap optimization processing */ hugetlb_vmemmap_optimize_folios(h, folio_list); - /* Add all new pool pages to free lists in one lock cycle */ - spin_lock_irqsave(&hugetlb_lock, flags); list_for_each_entry_safe(folio, tmp_f, folio_list, lru) { if (!folio_test_hugetlb_vmemmap_optimized(folio)) { /* @@ -3404,23 +3402,27 @@ static void __init prep_and_add_bootmem_folios(struct hstate *h, HUGETLB_VMEMMAP_RESERVE_PAGES, pages_per_huge_page(h)); } + /* Subdivide locks to achieve better parallel performance */ + spin_lock_irqsave(&hugetlb_lock, flags); __prep_account_new_huge_page(h, folio_nid(folio)); enqueue_hugetlb_folio(h, folio); + spin_unlock_irqrestore(&hugetlb_lock, flags); } - spin_unlock_irqrestore(&hugetlb_lock, flags); } /* * Put bootmem huge pages into the standard lists after mem_map is up. * Note: This only applies to gigantic (order > MAX_PAGE_ORDER) pages. */ -static void __init gather_bootmem_prealloc(void) +static void __init gather_bootmem_prealloc_node(unsigned long start, unsigned long end, void *arg) + { + int nid = start; LIST_HEAD(folio_list); struct huge_bootmem_page *m; struct hstate *h = NULL, *prev_h = NULL; - list_for_each_entry(m, &huge_boot_pages, list) { + list_for_each_entry(m, &huge_boot_pages[nid], list) { struct page *page = virt_to_page(m); struct folio *folio = (void *)page; @@ -3453,6 +3455,22 @@ static void __init gather_bootmem_prealloc(void) prep_and_add_bootmem_folios(h, &folio_list); } +static void __init gather_bootmem_prealloc(void) +{ + struct padata_mt_job job = { + .thread_fn = gather_bootmem_prealloc_node, + .fn_arg = NULL, + .start = 0, + .size = num_node_state(N_MEMORY), + .align = 1, + .min_chunk = 1, + .max_threads = num_node_state(N_MEMORY), + .numa_aware = true, + }; + + padata_do_multithreaded(&job); +} + static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) { unsigned long i; @@ -3600,6 +3618,7 @@ static unsigned long __init hugetlb_pages_alloc_boot(struct hstate *h) static void __init hugetlb_hstate_alloc_pages(struct hstate *h) { unsigned long allocated; + static bool initialied __initdata; /* skip gigantic hugepages allocation if hugetlb_cma enabled */ if (hstate_is_gigantic(h) && hugetlb_cma_size) { @@ -3607,6 +3626,15 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) return; } + /* hugetlb_hstate_alloc_pages will be called many times, initialize huge_boot_pages once */ + if (!initialied) { + int i = 0; + + for (i = 0; i < MAX_NUMNODES; i++) + INIT_LIST_HEAD(&huge_boot_pages[i]); + initialied = true; + } + /* do node specific alloc */ if (hugetlb_hstate_alloc_pages_specific_nodes(h)) return;