From patchwork Thu Feb 22 14:04:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gang Li X-Patchwork-Id: 204785 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:aa16:b0:108:e6aa:91d0 with SMTP id by22csp273411dyb; Thu, 22 Feb 2024 06:06:44 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWJgln+Tvz2NmkxjsOFrTMc3cDgJYqICSCTWiJ6HcMumYS/zWN5TUpGKR0azTZMJOg4jEf7pUX6k7gxhlieJ6sOi4RUHQ== X-Google-Smtp-Source: AGHT+IGRT4XoAggoBl1GXvfocjv3khub8OliArMFnt21XAl988HSoFbDza4CsQV9kGPz0wUV0Bqx X-Received: by 2002:a92:c912:0:b0:365:1044:4dcd with SMTP id t18-20020a92c912000000b0036510444dcdmr17831697ilp.16.1708610804284; Thu, 22 Feb 2024 06:06:44 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708610804; cv=pass; d=google.com; s=arc-20160816; b=pdA7GvRIcqsn0QPZuTiMmWDtIeW3GOKAC8uLNwvOfgHns6Z916eKxNqnyiOGm/iRa0 +/oul6oE2RhIVpo5bzA52dUDFFPHq1d5ArZWOQIEJDDiNGBSjY+QysiGc9+LN+0/rGjW 0q1Ty0XNEom1d+cKtsbD1/y86YNCMjodZSodYm/BUoHaHoUI06WZl7sUzv5y03LgTq3l CCj5j9wKzMrkjQYhTqR03Bp6jQ+pOygS8BtueO9miUNa1G2/sL3cy4FUM5ejWj4wTMHz otf4FH3GtHTLduk6G7KedprKqLmuYlywQRCoYT3dA1+6xl8u6dfSXGNK7eXq0H+KbzpV hFUw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=qLDJ0oXYyRPh9zz0wE8IXI1bSRdqq7PDnnR6TGU9VgU=; fh=yRWAnXCrDeo1JkXV9iUtee7xVlLig8radlX7vPmVxyI=; b=A/3p43G+uLvVLZRUeFtFA1E/6H7Amki0mfhuApdemzcUbfWk86lKPxiGM4Sx4/ui96 eXGo9/qD6UPxZ3rkFHc7MmaH8vIWdso3ErKAoT9h7PnMbhSgHPz1adVo5fXiScVT8ESP 7FSOqeAgb/SOqYty9NsK8UFy12y4jY1dBYElaqnsUQIJbyomyezkB+KbnnjEo6hAk6y5 NxlS/TdIjvixSL3ELZP0TJwnx6byLmCjIGEltf3W5fO7j7HB+7exaxipKe1FeSiFUBMx mT+GBbMJ42fGzR4oXsrUsoti9j9lrkeiClp8BB5D+lx8KCws0iLZ/RCdjQKd3yhZ6EG0 uGWw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=i3uRvd9J; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-76662-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-76662-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id s20-20020a056638219400b0047438262cc7si2317074jaj.104.2024.02.22.06.06.44 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Feb 2024 06:06:44 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-76662-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=i3uRvd9J; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-76662-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-76662-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 63BB8286E20 for ; Thu, 22 Feb 2024 14:05:35 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 1520D146016; Thu, 22 Feb 2024 14:04:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="i3uRvd9J" Received: from out-177.mta0.migadu.com (out-177.mta0.migadu.com [91.218.175.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 782EC1474A4 for ; Thu, 22 Feb 2024 14:04:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708610690; cv=none; b=sGUFOgsC4egy+wuHmRHyKGNgmxB2uj7P1iSuywJSHRxCkexdtNWlfDd6YjbGsseFfEv+SEsaWiMwkKCtphkX64yPL70zK+rVj6PYCDOjMjrzztO1FPKV8Ho68/c9U1MlM9dL2MsiiCEO+F3Itp15rdLsvletDf8ps2Crl1FIlBE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708610690; c=relaxed/simple; bh=9Nlv5yU9lraOMOB9dfIUMrxAw7ypqQN0Obqwia5pjOo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Tv7tR7D1/QoVuyuZxrH9a9W651ymOGuK9NDh7wCJ+z/oF/dUw8TExGAvetfsvexl7IBu0KNBQmvGtLSl2bDhOhUgAlaOSiR+5NklFEDGUKZF524oNw8KnnOBp9+/CXybxoz1Zpz2QcB4qKFSd9jCOoL+FlYGYNW/ny7+jOgAIBk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=i3uRvd9J; arc=none smtp.client-ip=91.218.175.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1708610686; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qLDJ0oXYyRPh9zz0wE8IXI1bSRdqq7PDnnR6TGU9VgU=; b=i3uRvd9JstCtGlLa8/tyWtExImE0JtGxaojcIRs7n7nhFDNAP1r0mFxWuFGnV0DVJck360 +ybW3ROURTCTJyB8p7i3fvB985dnVQ3LNtm4RXv8GdUo6fpAsYQ3cBv0H6OF+4mHm7dqT7 SiYS5mOQGsqbMvVVng+9nU3mjbwRkQs= From: Gang Li To: Andrew Morton Cc: David Hildenbrand , David Rientjes , Muchun Song , Tim Chen , Steffen Klassert , Daniel Jordan , Jane Chu , "Paul E . McKenney" , Randy Dunlap , linux-mm@kvack.org, linux-kernel@vger.kernel.org, ligang.bdlg@bytedance.com, Gang Li Subject: [PATCH v6 1/8] hugetlb: code clean for hugetlb_hstate_alloc_pages Date: Thu, 22 Feb 2024 22:04:14 +0800 Message-Id: <20240222140422.393911-2-gang.li@linux.dev> In-Reply-To: <20240222140422.393911-1-gang.li@linux.dev> References: <20240222140422.393911-1-gang.li@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791608283025395784 X-GMAIL-MSGID: 1791608283025395784 The readability of `hugetlb_hstate_alloc_pages` is poor. By cleaning the code, its readability can be improved, facilitating future modifications. This patch extracts two functions to reduce the complexity of `hugetlb_hstate_alloc_pages` and has no functional changes. - hugetlb_hstate_alloc_pages_node_specific() to handle iterates through each online node and performs allocation if necessary. - hugetlb_hstate_alloc_pages_report() report error during allocation. And the value of h->max_huge_pages is updated accordingly. Signed-off-by: Gang Li Tested-by: David Rientjes Reviewed-by: Muchun Song Reviewed-by: Tim Chen --- mm/hugetlb.c | 46 +++++++++++++++++++++++++++++----------------- 1 file changed, 29 insertions(+), 17 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index c53a41d07cd3a..794f3e6a19bb6 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3482,6 +3482,33 @@ static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) h->max_huge_pages_node[nid] = i; } +static bool __init hugetlb_hstate_alloc_pages_specific_nodes(struct hstate *h) +{ + int i; + bool node_specific_alloc = false; + + for_each_online_node(i) { + if (h->max_huge_pages_node[i] > 0) { + hugetlb_hstate_alloc_pages_onenode(h, i); + node_specific_alloc = true; + } + } + + return node_specific_alloc; +} + +static void __init hugetlb_hstate_alloc_pages_errcheck(unsigned long allocated, struct hstate *h) +{ + if (allocated < h->max_huge_pages) { + char buf[32]; + + string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, 32); + pr_warn("HugeTLB: allocating %lu of page size %s failed. Only allocated %lu hugepages.\n", + h->max_huge_pages, buf, allocated); + h->max_huge_pages = allocated; + } +} + /* * NOTE: this routine is called in different contexts for gigantic and * non-gigantic pages. @@ -3499,7 +3526,6 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) struct folio *folio; LIST_HEAD(folio_list); nodemask_t *node_alloc_noretry; - bool node_specific_alloc = false; /* skip gigantic hugepages allocation if hugetlb_cma enabled */ if (hstate_is_gigantic(h) && hugetlb_cma_size) { @@ -3508,14 +3534,7 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) } /* do node specific alloc */ - for_each_online_node(i) { - if (h->max_huge_pages_node[i] > 0) { - hugetlb_hstate_alloc_pages_onenode(h, i); - node_specific_alloc = true; - } - } - - if (node_specific_alloc) + if (hugetlb_hstate_alloc_pages_specific_nodes(h)) return; /* below will do all node balanced alloc */ @@ -3558,14 +3577,7 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) /* list will be empty if hstate_is_gigantic */ prep_and_add_allocated_folios(h, &folio_list); - if (i < h->max_huge_pages) { - char buf[32]; - - string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, 32); - pr_warn("HugeTLB: allocating %lu of page size %s failed. Only allocated %lu hugepages.\n", - h->max_huge_pages, buf, i); - h->max_huge_pages = i; - } + hugetlb_hstate_alloc_pages_errcheck(i, h); kfree(node_alloc_noretry); } From patchwork Thu Feb 22 14:04:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gang Li X-Patchwork-Id: 204781 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:aa16:b0:108:e6aa:91d0 with SMTP id by22csp272620dyb; Thu, 22 Feb 2024 06:05:49 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWW12D8KtRqttm4K4Lx7k/cxs6nKBYx8OOH4l1mnRdJsKoCZZezNUWkcnp2fjGHhV+xlwlA0/Wk3XqOvk1yvkvm8rMKQA== X-Google-Smtp-Source: AGHT+IEq+iB2G6/aoJMrWMLHpeW5Jtu0Gnn3TtSF4N9nnM/8l/moD5n4gq4XqZ7ThCRrO2MOQA+7 X-Received: by 2002:a17:906:7155:b0:a3e:5de6:701d with SMTP id z21-20020a170906715500b00a3e5de6701dmr8268626ejj.50.1708610749267; Thu, 22 Feb 2024 06:05:49 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708610749; cv=pass; d=google.com; s=arc-20160816; b=zwdIvysm/aaPInPxC4jTJzZvNKW/bvODT9AafA+KOtFnW85SekP+GwX/6fr3126TVz inEWoUxf16sMOBDaR5p0RS4ujQz5dS92veNpF2durmFGFWNfDVQl0S+igT/S3MeVvpJe hy01moO8JTHSTZOasBH2sGgAU4p+J5qewxcqfHvt/QXX51tKwtu+5oyhkiCTiL9Y8JTg hmrmTZ5DZ3b/LRzlczjmnJr8EyM96+oOzJO6LoV36u6vYoZomQFTp1v0UO6ibUZByRNd ySKfDlZBOpc0KiWEgX0zfpc13zJHYrruXbrz/M+pEMG0INuncL9aFmBsEcoKKSXkh5Co LcnQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=GFm+BsVb+EJLE4T/S60F9nhII3K5Okpm0T3KX+Ypj6g=; fh=yRWAnXCrDeo1JkXV9iUtee7xVlLig8radlX7vPmVxyI=; b=wSBuYpZzgBqGFunmJDs9fF2IK7eHY3Gkr7hyYq6EQbUE7nlszno1alDCW27m7XOxA0 qh+WKCQuH1Eu+2NgClVxVuzm2o0KD+U0mE4e9oPBOi31UGA+gWcmKaH+Grj1nzI08hBU CitiSpWMvfbwDHrirkQYZbAbZrhAt5+XKBEklG6dJ7KiUmlvnfUfaANSZfB6qgxG+ygL PA5vo25ZPBNAF4gcfaD+Le9RkH1hVZ3JTxQCLaXzFUbURaLYzI7waLfYVtMoyLbrGMbh cEfD6DaR8xP82qI2FrkWj3+xiqxUBD3O2an8y1ZUwnSgr78WVSiLgcYE+V2X5haC/O/d yIRQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=ZZ38NdKG; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-76663-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-76663-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id hp14-20020a1709073e0e00b00a3e9176fc9esi4001816ejc.350.2024.02.22.06.05.49 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Feb 2024 06:05:49 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-76663-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=ZZ38NdKG; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-76663-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-76663-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id DDB971F289A3 for ; Thu, 22 Feb 2024 14:05:48 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 666691487FA; Thu, 22 Feb 2024 14:04:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="ZZ38NdKG" Received: from out-187.mta0.migadu.com (out-187.mta0.migadu.com [91.218.175.187]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0E1981474B3 for ; Thu, 22 Feb 2024 14:04:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.187 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708610693; cv=none; b=fnp6E9R21YzuV4+vMmmy6x/eUqtbtgoXr1iM4ERzKPKh2Z/zhxdAjkOP3n9ompLIhAzF4WZMgvn0yiQKL1e45sCp455/xvNpGF3nqbQPFCsP/J+8SR/HBdBisTwsYy+I4TR2ItugOyp8+h4mIfLAaDTb8t/EIquk+ZfqoMWPh84= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708610693; c=relaxed/simple; bh=vRuxUrdqrEjW/kmQKHVnf2KKQjRlpz3t45eHDJuC1Z8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Sf+Fkr8HhW+ZK2vz8JSRm2tGDHbmGizSuWT8yDISkkNjH11VAiKYmWA+5u4yLqe0BUh0Ow+hPPstokk5SyTNPNVytqBgBmYhpjAH9kJ8Jsh2VWgzl2TPtSYkHVKA/UqcJpnATX9i02Zk0GmRa2i7QqvDxBMXEncB9sIThA/81p4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=ZZ38NdKG; arc=none smtp.client-ip=91.218.175.187 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1708610690; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GFm+BsVb+EJLE4T/S60F9nhII3K5Okpm0T3KX+Ypj6g=; b=ZZ38NdKGJjXCAsgG44/Qo//cf/3cc1EknNNpxFahUoIgcacNCf55oBecrLxSCFmnIx/6Yc zc0joimqN7Kl9zD1PYmr5FlyOXa9qomyj3kBYRaaA7C375f+IuMQnFTecYf2S6/jKeHeH3 JIJykJuzg6dwPf8qafigyzC4+gneD6Q= From: Gang Li To: Andrew Morton Cc: David Hildenbrand , David Rientjes , Muchun Song , Tim Chen , Steffen Klassert , Daniel Jordan , Jane Chu , "Paul E . McKenney" , Randy Dunlap , linux-mm@kvack.org, linux-kernel@vger.kernel.org, ligang.bdlg@bytedance.com, Gang Li Subject: [PATCH v6 2/8] hugetlb: split hugetlb_hstate_alloc_pages Date: Thu, 22 Feb 2024 22:04:15 +0800 Message-Id: <20240222140422.393911-3-gang.li@linux.dev> In-Reply-To: <20240222140422.393911-1-gang.li@linux.dev> References: <20240222140422.393911-1-gang.li@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791608225113905965 X-GMAIL-MSGID: 1791608225113905965 1G and 2M huge pages have different allocation and initialization logic, which leads to subtle differences in parallelization. Therefore, it is appropriate to split hugetlb_hstate_alloc_pages into gigantic and non-gigantic. This patch has no functional changes. Signed-off-by: Gang Li Tested-by: David Rientjes Reviewed-by: Tim Chen Reviewed-by: Muchun Song --- mm/hugetlb.c | 87 ++++++++++++++++++++++++++-------------------------- 1 file changed, 43 insertions(+), 44 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 794f3e6a19bb6..647e52596e2da 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3509,6 +3509,43 @@ static void __init hugetlb_hstate_alloc_pages_errcheck(unsigned long allocated, } } +static unsigned long __init hugetlb_gigantic_pages_alloc_boot(struct hstate *h) +{ + unsigned long i; + + for (i = 0; i < h->max_huge_pages; ++i) { + if (!alloc_bootmem_huge_page(h, NUMA_NO_NODE)) + break; + cond_resched(); + } + + return i; +} + +static unsigned long __init hugetlb_pages_alloc_boot(struct hstate *h) +{ + unsigned long i; + struct folio *folio; + LIST_HEAD(folio_list); + nodemask_t node_alloc_noretry; + + /* Bit mask controlling how hard we retry per-node allocations.*/ + nodes_clear(node_alloc_noretry); + + for (i = 0; i < h->max_huge_pages; ++i) { + folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY], + &node_alloc_noretry); + if (!folio) + break; + list_add(&folio->lru, &folio_list); + cond_resched(); + } + + prep_and_add_allocated_folios(h, &folio_list); + + return i; +} + /* * NOTE: this routine is called in different contexts for gigantic and * non-gigantic pages. @@ -3522,10 +3559,7 @@ static void __init hugetlb_hstate_alloc_pages_errcheck(unsigned long allocated, */ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) { - unsigned long i; - struct folio *folio; - LIST_HEAD(folio_list); - nodemask_t *node_alloc_noretry; + unsigned long allocated; /* skip gigantic hugepages allocation if hugetlb_cma enabled */ if (hstate_is_gigantic(h) && hugetlb_cma_size) { @@ -3538,47 +3572,12 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) return; /* below will do all node balanced alloc */ - if (!hstate_is_gigantic(h)) { - /* - * Bit mask controlling how hard we retry per-node allocations. - * Ignore errors as lower level routines can deal with - * node_alloc_noretry == NULL. If this kmalloc fails at boot - * time, we are likely in bigger trouble. - */ - node_alloc_noretry = kmalloc(sizeof(*node_alloc_noretry), - GFP_KERNEL); - } else { - /* allocations done at boot time */ - node_alloc_noretry = NULL; - } - - /* bit mask controlling how hard we retry per-node allocations */ - if (node_alloc_noretry) - nodes_clear(*node_alloc_noretry); - - for (i = 0; i < h->max_huge_pages; ++i) { - if (hstate_is_gigantic(h)) { - /* - * gigantic pages not added to list as they are not - * added to pools now. - */ - if (!alloc_bootmem_huge_page(h, NUMA_NO_NODE)) - break; - } else { - folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY], - node_alloc_noretry); - if (!folio) - break; - list_add(&folio->lru, &folio_list); - } - cond_resched(); - } - - /* list will be empty if hstate_is_gigantic */ - prep_and_add_allocated_folios(h, &folio_list); + if (hstate_is_gigantic(h)) + allocated = hugetlb_gigantic_pages_alloc_boot(h); + else + allocated = hugetlb_pages_alloc_boot(h); - hugetlb_hstate_alloc_pages_errcheck(i, h); - kfree(node_alloc_noretry); + hugetlb_hstate_alloc_pages_errcheck(allocated, h); } static void __init hugetlb_init_hstates(void) From patchwork Thu Feb 22 14:04:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gang Li X-Patchwork-Id: 204782 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:aa16:b0:108:e6aa:91d0 with SMTP id by22csp272821dyb; Thu, 22 Feb 2024 06:06:01 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCXgMLV9KaIqyjCNET5yyeKm04p9fZvcoP/+Qwxnnn9Urv+JmSL8LJab3nDcRuiJYppPwzDgLaGq1tOzPclznjnI23jFVg== X-Google-Smtp-Source: AGHT+IFYRCAOuxVvFBoWoDQIkMoIIuYkZ1sP4WSPeGZLbTCObPPWg88HQWFvysQ+D4I06DoQ2ys6 X-Received: by 2002:a17:906:24db:b0:a3f:9df3:2a4f with SMTP id f27-20020a17090624db00b00a3f9df32a4fmr791854ejb.9.1708610761677; Thu, 22 Feb 2024 06:06:01 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708610761; cv=pass; d=google.com; s=arc-20160816; b=I/iY3PjV6zT1nIHOTUZ5aT5KlXvXRpZHtVqmIp5LBsNNv/tpWCzwI3kCbdp5UgrKFn 0i0S6xGfpOoyfOHlgOcedj1QBg2Q9VmBe+W0Dw26mrWNpnqOuTsk2QuHa92izFkHl0zw dEkR0hjUWW9790JWBokYtZVkWvPHwDjZlmZu+FTzWSO8UZcbsHWXlHKN0Zm73cQkC0A1 743gQNJIAaKXko0DoqLfGMoaHQnWDs3tYHqRx3CX0vEVv/+eqYp9U8hTKR5pOi3QCTYl d1xIR3YK/gqhr2rDEKMgxJNG3eGoSZLaFZOt8McdXKXsx0OgZmxhEmohumxHLWNBlSd3 TCZg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=o+8qigIYBf+CF6nmzaQESLXIfknRBaMPQJkianD0US0=; fh=yRWAnXCrDeo1JkXV9iUtee7xVlLig8radlX7vPmVxyI=; b=LtUTzjopXfo0atttQxFqNY8RwrO71onW98uyD/mm0xUQ71nWI7EYx9tRlBPICN0ukz PJ0efq12G/F/mmht952a5vVQHSxDwGObZMjfsXbK2d1TfftMRbozlHEZVKOFR0gg56UT 7rdSa+2oIfpZX81GgYwmGsOoqex7wfT0gVaayRhii9o8XW6mK9HbjFpfeyi77USMIRZ+ fTj3SwDkmwfiyxDJOO91tRwlLSrEHWqFFbmcXjJXqO7r+PAhTP2iXb/YBFZnf72vHkKd vnBGYMFPKnEono+jD02Z/gtaWkdvvAvlbi49bhUh9bJERA4kOJXpw7H67dpmjspQlgcp kkmg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=GMb+VdCD; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-76664-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-76664-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id mb11-20020a170906eb0b00b00a3f8f20637fsi420426ejb.823.2024.02.22.06.06.01 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Feb 2024 06:06:01 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-76664-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=GMb+VdCD; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-76664-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-76664-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 4CCA51F28883 for ; Thu, 22 Feb 2024 14:06:01 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 13C3F14A096; Thu, 22 Feb 2024 14:05:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="GMb+VdCD" Received: from out-170.mta0.migadu.com (out-170.mta0.migadu.com [91.218.175.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 91E60148FFE for ; Thu, 22 Feb 2024 14:04:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708610697; cv=none; b=q3WlXVQ9i5sGxKTaFuWmWDF7ELpaTPfWelgQe00yMCjyT76JFCvvdGqhFciq7INihQajgVJat+s7tz6nQ3FP7FNGuNaGxEQHPMGrtPUmZ3UMYFcioIpOjI7TTKrufSfd2Kc4EpzjBE2zsdKCbYnlZ6/rPiGUQxVgyHhhUQ8dVe0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708610697; c=relaxed/simple; bh=GJdtD9vyuX0d/G7xkiT5uC/YyxLu8BmsN8PyxCBFDIM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=XwvJ8TUoV7BAnu+mRfT/UpfiXTNxSq/Xons6FK9TZpgWTD+1ovJ8ER5cM96gBmZL9SPMqlK3j51Mxgk1Dp7HpKlx0U081r+GDeUEMYwEgqVrp5zYFSt0kQvd7HyemXQbyZFImfgeYazW58N8uOxRdXcdBAcBd1xar+7Rq6COCtI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=GMb+VdCD; arc=none smtp.client-ip=91.218.175.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1708610693; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=o+8qigIYBf+CF6nmzaQESLXIfknRBaMPQJkianD0US0=; b=GMb+VdCDD/mQRiGk4B7cYFJq6Uvre0TApLBB9BKJ+sRd1zHzqt/ccv8JB7NwYBykKXKcVg yGsJtPCbMDS2g3UmDUDyGkzjAGtoVsFp1hOGY9p3OVE2ZXd6DuJMgkbwGJhkPF9OGmvazP YrigqBnLGrT2H9Joq7NpfDGHlVTmvWM= From: Gang Li To: Andrew Morton Cc: David Hildenbrand , David Rientjes , Muchun Song , Tim Chen , Steffen Klassert , Daniel Jordan , Jane Chu , "Paul E . McKenney" , Randy Dunlap , linux-mm@kvack.org, linux-kernel@vger.kernel.org, ligang.bdlg@bytedance.com, Gang Li Subject: [PATCH v6 3/8] hugetlb: pass *next_nid_to_alloc directly to for_each_node_mask_to_alloc Date: Thu, 22 Feb 2024 22:04:16 +0800 Message-Id: <20240222140422.393911-4-gang.li@linux.dev> In-Reply-To: <20240222140422.393911-1-gang.li@linux.dev> References: <20240222140422.393911-1-gang.li@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791608238013388721 X-GMAIL-MSGID: 1791608238013388721 With parallelization of hugetlb allocation across different threads, each thread works on a differnet node to allocate pages from, instead of all allocating from a common node h->next_nid_to_alloc. To address this, it's necessary to assign a separate next_nid_to_alloc for each thread. Consequently, the hstate_next_node_to_alloc and for_each_node_mask_to_alloc have been modified to directly accept a *next_nid_to_alloc parameter, ensuring thread-specific allocation and avoiding concurrent access issues. Signed-off-by: Gang Li Tested-by: David Rientjes Reviewed-by: Tim Chen Reviewed-by: Muchun Song --- mm/hugetlb.c | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 647e52596e2da..d1ce1a52ad504 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1464,15 +1464,15 @@ static int get_valid_node_allowed(int nid, nodemask_t *nodes_allowed) * next node from which to allocate, handling wrap at end of node * mask. */ -static int hstate_next_node_to_alloc(struct hstate *h, +static int hstate_next_node_to_alloc(int *next_node, nodemask_t *nodes_allowed) { int nid; VM_BUG_ON(!nodes_allowed); - nid = get_valid_node_allowed(h->next_nid_to_alloc, nodes_allowed); - h->next_nid_to_alloc = next_node_allowed(nid, nodes_allowed); + nid = get_valid_node_allowed(*next_node, nodes_allowed); + *next_node = next_node_allowed(nid, nodes_allowed); return nid; } @@ -1495,10 +1495,10 @@ static int hstate_next_node_to_free(struct hstate *h, nodemask_t *nodes_allowed) return nid; } -#define for_each_node_mask_to_alloc(hs, nr_nodes, node, mask) \ +#define for_each_node_mask_to_alloc(next_node, nr_nodes, node, mask) \ for (nr_nodes = nodes_weight(*mask); \ nr_nodes > 0 && \ - ((node = hstate_next_node_to_alloc(hs, mask)) || 1); \ + ((node = hstate_next_node_to_alloc(next_node, mask)) || 1); \ nr_nodes--) #define for_each_node_mask_to_free(hs, nr_nodes, node, mask) \ @@ -2350,12 +2350,13 @@ static void prep_and_add_allocated_folios(struct hstate *h, */ static struct folio *alloc_pool_huge_folio(struct hstate *h, nodemask_t *nodes_allowed, - nodemask_t *node_alloc_noretry) + nodemask_t *node_alloc_noretry, + int *next_node) { gfp_t gfp_mask = htlb_alloc_mask(h) | __GFP_THISNODE; int nr_nodes, node; - for_each_node_mask_to_alloc(h, nr_nodes, node, nodes_allowed) { + for_each_node_mask_to_alloc(next_node, nr_nodes, node, nodes_allowed) { struct folio *folio; folio = only_alloc_fresh_hugetlb_folio(h, gfp_mask, node, @@ -3310,7 +3311,7 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) goto found; } /* allocate from next node when distributing huge pages */ - for_each_node_mask_to_alloc(h, nr_nodes, node, &node_states[N_MEMORY]) { + for_each_node_mask_to_alloc(&h->next_nid_to_alloc, nr_nodes, node, &node_states[N_MEMORY]) { m = memblock_alloc_try_nid_raw( huge_page_size(h), huge_page_size(h), 0, MEMBLOCK_ALLOC_ACCESSIBLE, node); @@ -3679,7 +3680,7 @@ static int adjust_pool_surplus(struct hstate *h, nodemask_t *nodes_allowed, VM_BUG_ON(delta != -1 && delta != 1); if (delta < 0) { - for_each_node_mask_to_alloc(h, nr_nodes, node, nodes_allowed) { + for_each_node_mask_to_alloc(&h->next_nid_to_alloc, nr_nodes, node, nodes_allowed) { if (h->surplus_huge_pages_node[node]) goto found; } @@ -3794,7 +3795,8 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, cond_resched(); folio = alloc_pool_huge_folio(h, nodes_allowed, - node_alloc_noretry); + node_alloc_noretry, + &h->next_nid_to_alloc); if (!folio) { prep_and_add_allocated_folios(h, &page_list); spin_lock_irq(&hugetlb_lock); From patchwork Thu Feb 22 14:04:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gang Li X-Patchwork-Id: 204783 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:aa16:b0:108:e6aa:91d0 with SMTP id by22csp273013dyb; Thu, 22 Feb 2024 06:06:14 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCW08n/MW5GG4ZazylCwRTyddMkTL1bfWmc816mPYRww0UoQOzOMsHx5JNA80rMzegQ/outjltK1MIgXK6M84JbELz+Hvg== X-Google-Smtp-Source: AGHT+IH6Uv59JWgy33JouAF5EOHuj3oB/HBLUdQUaKc9DyCUK1pgYRXUdo2gtHHJuXBDlZTSgEUO X-Received: by 2002:a17:906:8d5:b0:a3d:993e:ad24 with SMTP id o21-20020a17090608d500b00a3d993ead24mr13249369eje.59.1708610774476; Thu, 22 Feb 2024 06:06:14 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708610774; cv=pass; d=google.com; s=arc-20160816; b=t+k7VLP3JgqbDDZtWY5Lm922lth3Zk3YeZrYd0J6cGJl0jQBxs3l3P0711rZQHm7uK fj7DlrYhXUtCgjdWYnthrCpYujgtrk1HV/fnZUEYcrDHVZThunn0U3MpMjF9aUyC7s5O 8ZoqAh36amfwcHWrylIwL+BtUoLSn1n18bXGkzyDlNZMm9y3qlGQeW0FpupCfbs1InmJ 5PrxbMGtWZdhZuCG+jP5hJkghkbXBgrB7I9VEZssiOhjZDbNllh41l4eckL6kZWVARTL 50q/UkBkPeipId+u/L50XxhKYSMns6ZnGF7+5lDL5h9luCS7gqY4Xo/GrMenyalnCBbm PlZQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=MgzBTPzPKbW03C1p+bf5b2VOJPKCTieqosPALSU6d3A=; fh=yRWAnXCrDeo1JkXV9iUtee7xVlLig8radlX7vPmVxyI=; b=iQT8OwnpUYQfXTfx+AwdR0Jz0xP/0c52xCdYHuFGAMTX0hHUygMZkDePSL0rDRk/DL h/Jdl8eetxkqKjPmh7cuJDZXi/MrtCfbImqnTupB/C0uTl82ngh8r97Dor2ujmWwoaVw Ld8F2V7TgHQ9dZZeFxSCEGWofggUT484jRrs6EJDYtgxFPZ0CoqzMfoq9/v3rsA0seH1 FrU8TNDrrA0MBc6F14yNxoeIQs92x/Tq6c1154lwLYH4YC069uuNpjdgbCXdC9Zbg1aw DS3yT7NmOmtfbqr/CW57n+z7XzxvMs1194xadzJv1vY4cvLBfe0xHwIRgQkqWdULFr7H 1lGQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=p1Ko7q4H; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-76665-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-76665-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id sh36-20020a1709076ea400b00a3e3e55bb0asi4875643ejc.454.2024.02.22.06.06.14 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Feb 2024 06:06:14 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-76665-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=p1Ko7q4H; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-76665-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-76665-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 1C4581F289EA for ; Thu, 22 Feb 2024 14:06:14 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id C4ADB1474A2; Thu, 22 Feb 2024 14:05:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="p1Ko7q4H" Received: from out-186.mta0.migadu.com (out-186.mta0.migadu.com [91.218.175.186]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3C32D14A08D for ; Thu, 22 Feb 2024 14:04:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.186 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708610701; cv=none; b=nYubhOsTKrxfl+hltQo8yVCbbdjUlYecC5CW7odlg5neyCt86uf942BSKgSAB8NoZq+ht5YlMEUweb8sEQX46Mc/qaXrAl5s+raQRX99JYdO3RUFDvL8Wf//f9JkV1juvEJ90a/K7t17XnrdefbB+68NRFRqA3gF6KNBSFtjjbU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708610701; c=relaxed/simple; bh=pKjNCBtJRKlytDtOYpIKfNyflGK1XMd1FZLmFDtWJWQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=QPoo8aZI4PLaCnGPrm1+GsPIsvZUI6wj3jVVdKRIKFGwJ8FBaqf9oPDrw2hYzW7SjVKVtUAvA5s2UsES3a5xbehosQc6r145hCtvCROlswi24SLDEUtEzYahTgytNXqRkBExbIYipmd6xliRs2C924wvkE405/NnE8mOiCBIviw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=p1Ko7q4H; arc=none smtp.client-ip=91.218.175.186 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1708610697; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MgzBTPzPKbW03C1p+bf5b2VOJPKCTieqosPALSU6d3A=; b=p1Ko7q4HwLZ8IRyvsRmFPEObb08w0kLTfT8Eo/yUShmw9X8SanRExrBoYN4E8QizQ7JtJr y6CDhqCmusgjbGbPQnWZj7zwS/EdtoiwysOQK8U4EQ2jhWPEDI0N15nmbWXhOwCYYBftee 20rXO2q25jdViRTYxjq4nVcOPEYzQBk= From: Gang Li To: Andrew Morton Cc: David Hildenbrand , David Rientjes , Muchun Song , Tim Chen , Steffen Klassert , Daniel Jordan , Jane Chu , "Paul E . McKenney" , Randy Dunlap , linux-mm@kvack.org, linux-kernel@vger.kernel.org, ligang.bdlg@bytedance.com, Gang Li Subject: [PATCH v6 4/8] padata: dispatch works on different nodes Date: Thu, 22 Feb 2024 22:04:17 +0800 Message-Id: <20240222140422.393911-5-gang.li@linux.dev> In-Reply-To: <20240222140422.393911-1-gang.li@linux.dev> References: <20240222140422.393911-1-gang.li@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791608251436661066 X-GMAIL-MSGID: 1791608251436661066 When a group of tasks that access different nodes are scheduled on the same node, they may encounter bandwidth bottlenecks and access latency. Thus, numa_aware flag is introduced here, allowing tasks to be distributed across different nodes to fully utilize the advantage of multi-node systems. Signed-off-by: Gang Li Tested-by: David Rientjes Reviewed-by: Muchun Song Reviewed-by: Tim Chen --- include/linux/padata.h | 2 ++ kernel/padata.c | 14 ++++++++++++-- mm/mm_init.c | 1 + 3 files changed, 15 insertions(+), 2 deletions(-) diff --git a/include/linux/padata.h b/include/linux/padata.h index 495b16b6b4d72..8f418711351bc 100644 --- a/include/linux/padata.h +++ b/include/linux/padata.h @@ -137,6 +137,7 @@ struct padata_shell { * appropriate for one worker thread to do at once. * @max_threads: Max threads to use for the job, actual number may be less * depending on task size and minimum chunk size. + * @numa_aware: Distribute jobs to different nodes with CPU in a round robin fashion. */ struct padata_mt_job { void (*thread_fn)(unsigned long start, unsigned long end, void *arg); @@ -146,6 +147,7 @@ struct padata_mt_job { unsigned long align; unsigned long min_chunk; int max_threads; + bool numa_aware; }; /** diff --git a/kernel/padata.c b/kernel/padata.c index 179fb1518070c..e3f639ff16707 100644 --- a/kernel/padata.c +++ b/kernel/padata.c @@ -485,7 +485,8 @@ void __init padata_do_multithreaded(struct padata_mt_job *job) struct padata_work my_work, *pw; struct padata_mt_job_state ps; LIST_HEAD(works); - int nworks; + int nworks, nid; + static atomic_t last_used_nid __initdata; if (job->size == 0) return; @@ -517,7 +518,16 @@ void __init padata_do_multithreaded(struct padata_mt_job *job) ps.chunk_size = roundup(ps.chunk_size, job->align); list_for_each_entry(pw, &works, pw_list) - queue_work(system_unbound_wq, &pw->pw_work); + if (job->numa_aware) { + int old_node = atomic_read(&last_used_nid); + + do { + nid = next_node_in(old_node, node_states[N_CPU]); + } while (!atomic_try_cmpxchg(&last_used_nid, &old_node, nid)); + queue_work_node(nid, system_unbound_wq, &pw->pw_work); + } else { + queue_work(system_unbound_wq, &pw->pw_work); + } /* Use the current thread, which saves starting a workqueue worker. */ padata_work_init(&my_work, padata_mt_helper, &ps, PADATA_WORK_ONSTACK); diff --git a/mm/mm_init.c b/mm/mm_init.c index 2c19f5515e36c..549e76af8f82a 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -2231,6 +2231,7 @@ static int __init deferred_init_memmap(void *data) .align = PAGES_PER_SECTION, .min_chunk = PAGES_PER_SECTION, .max_threads = max_threads, + .numa_aware = false, }; padata_do_multithreaded(&job); From patchwork Thu Feb 22 14:04:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gang Li X-Patchwork-Id: 204787 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:aa16:b0:108:e6aa:91d0 with SMTP id by22csp273880dyb; Thu, 22 Feb 2024 06:07:17 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVDIMBcsH/GXgICxG9wAZx1ephn5wsYb7OnOXCMj3lY8f33XmC9mEGLAgUK4mzsrTfe3VFS/pkzuFPjbMzz5nuqZenEuQ== X-Google-Smtp-Source: AGHT+IHqB527InZiy85iEfRgTbGw0CYMU8Hgfw3hjY41i/hMvmdAO2GyOH5AOAAFG/gPgQ1kL4mF X-Received: by 2002:a05:6e02:1143:b0:364:fffe:44c4 with SMTP id o3-20020a056e02114300b00364fffe44c4mr19652588ill.7.1708610836880; Thu, 22 Feb 2024 06:07:16 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708610836; cv=pass; d=google.com; s=arc-20160816; b=b9XLJBLxlAlUiAB7lF1AJSilVYLmU91GuS0IVwcCXAWCKt9kIJEmmWbqqHWgHgEerL hwToshP8hKXoa+Mf3ju6uvuZx8ppeg0naxCw04CIu25A+agatJoJHVUg64ZVuMsU7RCF EaGQK50+Y/gyNP7SUASgWhpb9cVEwN2FMWBPBcso3ahsRE0CHI5+sWJCZytfpUIQE7Sg 4A0jc9Y2vAY742a91najzPfEi15GMr4ZboiU2DRQxx6g9ghcskRuY3fKQNeFYtFjoLme gIGRvyWLMq2Bb3dOJMU8mI3YKBrwqCStz9y9u/ure7xD4FdKZAjjZVNDvZ/quoelBlWs 2/gw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=P5dkM7DuGNd3n4uTkQacavwzunNC39kM4edC7ugWujE=; fh=yRWAnXCrDeo1JkXV9iUtee7xVlLig8radlX7vPmVxyI=; b=dSqSxf9UGm90IbaFJzVRVRt2flkAh9T9hEPy0Eyr6KF6g2gIf/P15w6Jxx7YsnN7We kTHt5O7d2kkbcMftALTKxBehzYj1POLbWoHI4O2XaQ6pqGeNHYGDkXT3/rVt0hwkJrS/ gcGp1pvv6JrNDtmYTHjrRMwhUv5l7hKKwsb2RXrTf9rJoMnG76f2RxULzYEZtk+GFpVK r6UF/w2DqKzx2hNj0uudQ5wiNoKB+IOvHkT8+MktsBNANIRA3J6tK4BR/7eSp4QXqNjG hqAdiIbsvcmaGShJtRxNBqZtr6JyUGPGxD2jRMa9+8BBnrqZC74Vh3rbUX6LSIHmJCYv Y+7A==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=EWql7fnc; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-76666-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-76666-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id y14-20020a92090e000000b00363a6450db7si4497964ilg.85.2024.02.22.06.07.16 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Feb 2024 06:07:16 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-76666-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=EWql7fnc; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-76666-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-76666-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 9E10D2811AF for ; Thu, 22 Feb 2024 14:06:25 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 8BA6114C5A7; Thu, 22 Feb 2024 14:05:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="EWql7fnc" Received: from out-170.mta0.migadu.com (out-170.mta0.migadu.com [91.218.175.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A1D9914691D for ; Thu, 22 Feb 2024 14:05:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708610704; cv=none; b=MUmDAnjJIzS9gCr+IdbGtoE2mu1pcxAxlvnPgaRdkxfatUrfIR0ya3/n2AXwDnz9ojDgeY/crV3BKuH4sXHgJc0snouJSU/mpAUlp6lLeu3lslH5sfHj4fJ/VNEAwPoZ9Cf+8HMj1N2fqx+Cx9ZYWCblJDqSdYKmZY4MbdDfa4g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708610704; c=relaxed/simple; bh=tQai8VYTD5u9kmna2i0v3NDlggEePdlk6wRlzITeLz4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=QU38ab7YDD8SNlGQvhTp+iagjGHdS1muUUjfHjXxfIHPvemrFFJKk4VQLAR3ENbrsKjrM2hHs6hXace8b0eB8XUjwFUjTEAxBspVm0ZTzMf8uT3FufhanKsbn9WYF63VXtxo/4vNO2z3GSWKPOWL70oObWnWPOwhOpYiAbdJPvQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=EWql7fnc; arc=none smtp.client-ip=91.218.175.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1708610701; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=P5dkM7DuGNd3n4uTkQacavwzunNC39kM4edC7ugWujE=; b=EWql7fncR1Nj1MIpD5GRXCTp3WENf4dEY69tv+jJU3JilH4N5PjL/V0Pe5FGwckjEP/Uyc TMSelN5Z8oMXIJ4mqBgNDiz4LmUYpAOxzwz8N/sBA/f6cjS1ksHyDB50z50WXk6Wjh9xGP vTIbBboABNWUcFmQwXUyWrABeLtlfvs= From: Gang Li To: Andrew Morton Cc: David Hildenbrand , David Rientjes , Muchun Song , Tim Chen , Steffen Klassert , Daniel Jordan , Jane Chu , "Paul E . McKenney" , Randy Dunlap , linux-mm@kvack.org, linux-kernel@vger.kernel.org, ligang.bdlg@bytedance.com, Gang Li Subject: [PATCH v6 5/8] padata: downgrade padata_do_multithreaded to serial execution for non-SMP Date: Thu, 22 Feb 2024 22:04:18 +0800 Message-Id: <20240222140422.393911-6-gang.li@linux.dev> In-Reply-To: <20240222140422.393911-1-gang.li@linux.dev> References: <20240222140422.393911-1-gang.li@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791608316539665927 X-GMAIL-MSGID: 1791608316539665927 hugetlb parallelization depends on PADATA, and PADATA depends on SMP. PADATA consists of two distinct functionality: One part is padata_do_multithreaded which disregards order and simply divides tasks into several groups for parallel execution. Hugetlb init parallelization depends on padata_do_multithreaded. The other part is composed of a set of APIs that, while handling data in an out-of-order parallel manner, can eventually return the data with ordered sequence. Currently Only `crypto/pcrypt.c` use them. All users of PADATA of non-SMP case currently only use padata_do_multithreaded. It is easy to implement a serial one in include/linux/padata.h. And it is not necessary to implement another functionality unless the only user of crypto/pcrypt.c does not depend on SMP in the future. Signed-off-by: Gang Li Tested-by: Paul E. McKenney Acked-by: Daniel Jordan --- include/linux/padata.h | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/include/linux/padata.h b/include/linux/padata.h index 8f418711351bc..0146daf344306 100644 --- a/include/linux/padata.h +++ b/include/linux/padata.h @@ -180,10 +180,6 @@ struct padata_instance { #ifdef CONFIG_PADATA extern void __init padata_init(void); -#else -static inline void __init padata_init(void) {} -#endif - extern struct padata_instance *padata_alloc(const char *name); extern void padata_free(struct padata_instance *pinst); extern struct padata_shell *padata_alloc_shell(struct padata_instance *pinst); @@ -194,4 +190,12 @@ extern void padata_do_serial(struct padata_priv *padata); extern void __init padata_do_multithreaded(struct padata_mt_job *job); extern int padata_set_cpumask(struct padata_instance *pinst, int cpumask_type, cpumask_var_t cpumask); +#else +static inline void __init padata_init(void) {} +static inline void __init padata_do_multithreaded(struct padata_mt_job *job) +{ + job->thread_fn(job->start, job->start + job->size, job->fn_arg); +} +#endif + #endif From patchwork Thu Feb 22 14:04:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gang Li X-Patchwork-Id: 204786 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:aa16:b0:108:e6aa:91d0 with SMTP id by22csp273832dyb; Thu, 22 Feb 2024 06:07:12 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCW8IX27nYLLco4nPVecheyJNwxyI69qQ5JBCWrXwvXhXO6KcRPBP8IR+1Xgb28VG4tLBcpFvJ9vHrNI31vhvavdAu2gYA== X-Google-Smtp-Source: AGHT+IFpFQoeoMuUqyvSm1ZKhRDREERCGP2W2gfzF7KmPdxZ/i/zCJj5YLtr9aToFtTwrD1QRX48 X-Received: by 2002:a05:6214:20ad:b0:68f:309d:986 with SMTP id 13-20020a05621420ad00b0068f309d0986mr618077qvd.17.1708610832560; Thu, 22 Feb 2024 06:07:12 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708610832; cv=pass; d=google.com; s=arc-20160816; b=abaPoFLa8GQ1mIpvNYWJHglgiNBY7ReKNkBrn3MXlpB7wdx7gCy2frQYlyRaBIaoxL 1dC4TyBb8XNdnkD7fxNAmC2DhrUEfNF0QJ0WWDhvnGs4adTcX9v5VwfSS5AKM83744+l LcQUrDSgUOVfAaQfAAQFBCgxOllHZ0seXunCZYfU1FLiltSDZEmiw/MnfIGuOY5HTx5Q uzb4wXwtXp2koG36JQXmURBFbmSZ7ylhx8jlHg7qewHmfEQlH2/kpdk9E6ZdAKnOn61e f6rEngHYph1fPfikQ/OYBLxbU3baZYXgJEQehqLHHw206CmEYSWLrrhZOBIK3Qpai4wk 5Mwg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=gKGGJ4keOOfrl00iluvEDQ3Ty93zFU24V1EYV5CxVyE=; fh=yRWAnXCrDeo1JkXV9iUtee7xVlLig8radlX7vPmVxyI=; b=R8xSdo/0RlSv/dJ7Ysz310jdA5QQ+sGE5NaStzOBmtkYuc1CSuOpEJShEsXOG86prb rMZ+ZZzgS89SjOgzDdbXeIzN0Y8O8fxz+sTZ42Nj5X2/aS9uGsY+Bg6i2quRTRHmHOVb qDAZIEmelk/tMDHF/DJ1G8nfin0C6EP+1AgQ/siKSgipOzXsZC7cH7fR9WnubwQjxU4S 18C8m0bqLuEMCfn1+HuHjONxuxEf12le+N4nw4nUGuObIouaHYpjoR9SYvf29BYCL2zZ Zmr07hqseaYWz0+uhCtHv8zjN3GI5jzeU32+PRQIhn9Omb/6TLYh7VZvwTgF548ztd9Q cihw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b="qg/JjqKp"; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-76667-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-76667-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id h11-20020a0562140dab00b0068fb26a9f72si2490182qvh.447.2024.02.22.06.07.12 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Feb 2024 06:07:12 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-76667-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b="qg/JjqKp"; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-76667-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-76667-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 56FD01C25E80 for ; Thu, 22 Feb 2024 14:06:38 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id C6A9E14D435; Thu, 22 Feb 2024 14:05:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="qg/JjqKp" Received: from out-171.mta0.migadu.com (out-171.mta0.migadu.com [91.218.175.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AF5B914C5AF for ; Thu, 22 Feb 2024 14:05:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708610708; cv=none; b=XnVte5m/zxhnlxgpp8OI0FFo9rE4ZD951vtdNoW5VBHIbhJ6ivm//9hnzUWNZl0o1cpZO38ZYXvz5zYGETaj/cWPdR6K6cREc0i4k/iDZjorVKr0WAhlsd/iSfZ9zlzI7gS47eumDeXj07R/0agZaTxmr/Ww0gH2jtHyjxK1MoM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708610708; c=relaxed/simple; bh=R5MnQwc9t4v2E0fp8qGfSsi085XwCBGRqhTGk/VczIo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ESO/7zax262KNBGzZ7VtwWk1XICtKULFYHW2J5b3y0KdmNJwPZA28vPM9Yy+iMjVXXzGu7mEp5SkGbCLBrgOh2BBI0cSEmPat4X24upnFDPlnd8/JWdKPDBhQqoL3Z1+PJlk+yinJVimOPBgoKANMhRw04x9OLbkeGs3bBkrDCI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=qg/JjqKp; arc=none smtp.client-ip=91.218.175.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1708610704; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gKGGJ4keOOfrl00iluvEDQ3Ty93zFU24V1EYV5CxVyE=; b=qg/JjqKph94W1a8idl0Ardw/d1tXM/odKpkHBNYeJX6L3C8F1p+YAZQ9skiGNuOx7apSHg P50H5nQcpGnKjLptZs0X/vUJxmxyGb2wQzDUgVbKsnbaaN5OpGF0RgSBPk7VaTYu7ry88z y+/WrwgLKTRUrw1khfweAOXlhe62gqo= From: Gang Li To: Andrew Morton Cc: David Hildenbrand , David Rientjes , Muchun Song , Tim Chen , Steffen Klassert , Daniel Jordan , Jane Chu , "Paul E . McKenney" , Randy Dunlap , linux-mm@kvack.org, linux-kernel@vger.kernel.org, ligang.bdlg@bytedance.com, Gang Li Subject: [PATCH v6 6/8] hugetlb: have CONFIG_HUGETLBFS select CONFIG_PADATA Date: Thu, 22 Feb 2024 22:04:19 +0800 Message-Id: <20240222140422.393911-7-gang.li@linux.dev> In-Reply-To: <20240222140422.393911-1-gang.li@linux.dev> References: <20240222140422.393911-1-gang.li@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791608312297046031 X-GMAIL-MSGID: 1791608312297046031 Allow hugetlb use padata_do_multithreaded for parallel initialization. Select CONFIG_PADATA in this case. Signed-off-by: Gang Li Tested-by: David Rientjes Reviewed-by: Muchun Song Tested-by: Paul E. McKenney Acked-by: Daniel Jordan --- fs/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/fs/Kconfig b/fs/Kconfig index 89fdbefd1075f..c0b9599bf8e3a 100644 --- a/fs/Kconfig +++ b/fs/Kconfig @@ -262,6 +262,7 @@ menuconfig HUGETLBFS depends on X86 || SPARC64 || ARCH_SUPPORTS_HUGETLBFS || BROKEN depends on (SYSFS || SYSCTL) select MEMFD_CREATE + select PADATA if SMP help hugetlbfs is a filesystem backing for HugeTLB pages, based on ramfs. For architectures that support it, say Y here and read From patchwork Thu Feb 22 14:04:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gang Li X-Patchwork-Id: 204788 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:aa16:b0:108:e6aa:91d0 with SMTP id by22csp274093dyb; Thu, 22 Feb 2024 06:07:33 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCUKYWIn/jIF+539hMO33H1s7u+7H3SEGSLkFokmQ9X4kUpjBPowKXo1PYycccoP16v/TPEqoITrP+fs6w5GzlvZu+KvTA== X-Google-Smtp-Source: AGHT+IFTb5gMDNsyNR284DPA+S+btdAuogF1XdP8SpLXaQsptg90VGZje1h6ikahCqysaWP+dojS X-Received: by 2002:a6b:7d09:0:b0:7c7:2259:8f1e with SMTP id c9-20020a6b7d09000000b007c722598f1emr19003251ioq.6.1708610852901; Thu, 22 Feb 2024 06:07:32 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708610852; cv=pass; d=google.com; s=arc-20160816; b=lb3e3yYZ/Cg6WT+vP4jMT68ghCqzbA+m8cIK/hOP/15tZ0Hy8OSEE2gpU/eJi8d+bt jleJvSn0E6jZnoU59QJPXpknW8EbCSzHSCYKhaBoAg8iypAJKziqYmK6m7cmBc4K6ChP DSYHFVveqZYg1BvRRT6Ih4DxF2ZX/QPrDwTwOh/piNaSFG4Twtn2yq24xmfGHGpcR890 8ZgBeI8WVWd9U9ws4eRuW5IkUiBQxsTlqxrAA6dSeq+JmhtnPzCduIyqLqTQOQpSglFp uzCqBOKq30tJD72x29lQQqO0Ip0N14kfSpqoF20pqEV3JmiLH5J2NCNxt+IX+xxlVdgs sQSw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=N6t1qyq4TDUtkxpVn8LUmZjkdAyqh+tvT81M41wbLKc=; fh=yRWAnXCrDeo1JkXV9iUtee7xVlLig8radlX7vPmVxyI=; b=IA1/LzWVhFBjkGkBvwbaIR7mjhdlr4+YQkA3mBr++p6EniUAJqtXPUScn8f4VvrDUG ccNuaPgJ43gyba+vfQ4/vUUm+DbImq6aJdej2KtgnxjdCu23Tp+bQM7CLKVI9qdy7Mr5 G6hfenWgu0hyA0d5uhxfHAU4KK836hnguxgkFtsre+YV5bIPRPDnXj7NXDLzsygD9J5c /PDBx2djh6lFf5Z5UVAXCiOHW5i8GZJC9K7jGSMsZGWDc3jSlMka8kVHUovj6/rUHrcn LQBRu2UXwIwelBVFvn5cxySxcGGPC+QiP3pGqgJHHl9rsow6SvyLENFZyOe48WiB5cxV E9VA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=X1qraX2V; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-76668-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-76668-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id b7-20020a05660214c700b007c72474171bsi4005668iow.36.2024.02.22.06.07.32 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Feb 2024 06:07:32 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-76668-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=X1qraX2V; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-76668-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-76668-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 175BF28571A for ; Thu, 22 Feb 2024 14:06:51 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 2536C1474C1; Thu, 22 Feb 2024 14:05:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="X1qraX2V" Received: from out-180.mta0.migadu.com (out-180.mta0.migadu.com [91.218.175.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3F5C614C5BD for ; Thu, 22 Feb 2024 14:05:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708610713; cv=none; b=C3lvmeJxYR3yqdC3RPYyAKuWfKduMI/uhBRJISoVcN4fUqwDBV4hb9Y4WCSLUOCZ0izX3mFMSfTAyJTvthRF+yoUuyBGEUntVSggAKcSjBcS3WJt9EPkgNQ0zgtVDgrnUySJYJ6rojr6nGLjG+tTGBbx+eC7xzBiNvZ+LQwNudg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708610713; c=relaxed/simple; bh=wDUtn4D4+R/+Ucrafy1XUZ39rLpK7nGNdp7cuRDIc2A=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=S7HVLBr/QaO0zD9nTSHFob5hB0j8VUHtMuKYKxHyXTvOPWzt5efStkV85a685qpOJL/NBDq4+d/vr0/aEv0XhWlqZYtCRYdWnvEo6KSYcO5MAmLxdt0e8e/s1Ip6I7QBMtRZi2lvjLMfBxLmwqaAivvhRpqnU/ilHo3HY5HYiI4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=X1qraX2V; arc=none smtp.client-ip=91.218.175.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1708610708; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=N6t1qyq4TDUtkxpVn8LUmZjkdAyqh+tvT81M41wbLKc=; b=X1qraX2Vvj0/KBfu6EnJq/+fNR67xug/4C3rpiILP1VuAv3Lrl9M7fpLL3xNyIAibRgjh2 t6IVN2idjt+n0TQ0E+IbU5JFcgU226TBbWzoJOagUVks3iKjgV3ZA0lDtrM/fhY1oX0xtj SGDAg/B2R/m6fm9JvkcoCOxIIyohtc8= From: Gang Li To: Andrew Morton Cc: David Hildenbrand , David Rientjes , Muchun Song , Tim Chen , Steffen Klassert , Daniel Jordan , Jane Chu , "Paul E . McKenney" , Randy Dunlap , linux-mm@kvack.org, linux-kernel@vger.kernel.org, ligang.bdlg@bytedance.com, Gang Li Subject: [PATCH v6 7/8] hugetlb: parallelize 2M hugetlb allocation and initialization Date: Thu, 22 Feb 2024 22:04:20 +0800 Message-Id: <20240222140422.393911-8-gang.li@linux.dev> In-Reply-To: <20240222140422.393911-1-gang.li@linux.dev> References: <20240222140422.393911-1-gang.li@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791608333819389036 X-GMAIL-MSGID: 1791608333819389036 By distributing both the allocation and the initialization tasks across multiple threads, the initialization of 2M hugetlb will be faster, thereby improving the boot speed. Here are some test results: test case no patch(ms) patched(ms) saved ------------------- -------------- ------------- -------- 256c2T(4 node) 2M 3336 1051 68.52% 128c1T(2 node) 2M 1943 716 63.15% Signed-off-by: Gang Li Tested-by: David Rientjes Reviewed-by: Muchun Song --- mm/hugetlb.c | 73 ++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 56 insertions(+), 17 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index d1ce1a52ad504..3ce957b3e350b 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -35,6 +35,7 @@ #include #include #include +#include #include #include @@ -3510,6 +3511,30 @@ static void __init hugetlb_hstate_alloc_pages_errcheck(unsigned long allocated, } } +static void __init hugetlb_pages_alloc_boot_node(unsigned long start, unsigned long end, void *arg) +{ + struct hstate *h = (struct hstate *)arg; + int i, num = end - start; + nodemask_t node_alloc_noretry; + LIST_HEAD(folio_list); + int next_node = first_online_node; + + /* Bit mask controlling how hard we retry per-node allocations.*/ + nodes_clear(node_alloc_noretry); + + for (i = 0; i < num; ++i) { + struct folio *folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY], + &node_alloc_noretry, &next_node); + if (!folio) + break; + + list_move(&folio->lru, &folio_list); + cond_resched(); + } + + prep_and_add_allocated_folios(h, &folio_list); +} + static unsigned long __init hugetlb_gigantic_pages_alloc_boot(struct hstate *h) { unsigned long i; @@ -3525,26 +3550,40 @@ static unsigned long __init hugetlb_gigantic_pages_alloc_boot(struct hstate *h) static unsigned long __init hugetlb_pages_alloc_boot(struct hstate *h) { - unsigned long i; - struct folio *folio; - LIST_HEAD(folio_list); - nodemask_t node_alloc_noretry; - - /* Bit mask controlling how hard we retry per-node allocations.*/ - nodes_clear(node_alloc_noretry); + struct padata_mt_job job = { + .fn_arg = h, + .align = 1, + .numa_aware = true + }; - for (i = 0; i < h->max_huge_pages; ++i) { - folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY], - &node_alloc_noretry); - if (!folio) - break; - list_add(&folio->lru, &folio_list); - cond_resched(); - } + job.thread_fn = hugetlb_pages_alloc_boot_node; + job.start = 0; + job.size = h->max_huge_pages; - prep_and_add_allocated_folios(h, &folio_list); + /* + * job.max_threads is twice the num_node_state(N_MEMORY), + * + * Tests below indicate that a multiplier of 2 significantly improves + * performance, and although larger values also provide improvements, + * the gains are marginal. + * + * Therefore, choosing 2 as the multiplier strikes a good balance between + * enhancing parallel processing capabilities and maintaining efficient + * resource management. + * + * +------------+-------+-------+-------+-------+-------+ + * | multiplier | 1 | 2 | 3 | 4 | 5 | + * +------------+-------+-------+-------+-------+-------+ + * | 256G 2node | 358ms | 215ms | 157ms | 134ms | 126ms | + * | 2T 4node | 979ms | 679ms | 543ms | 489ms | 481ms | + * | 50G 2node | 71ms | 44ms | 37ms | 30ms | 31ms | + * +------------+-------+-------+-------+-------+-------+ + */ + job.max_threads = num_node_state(N_MEMORY) * 2; + job.min_chunk = h->max_huge_pages / num_node_state(N_MEMORY) / 2; + padata_do_multithreaded(&job); - return i; + return h->nr_huge_pages; } /* From patchwork Thu Feb 22 14:04:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gang Li X-Patchwork-Id: 204789 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:aa16:b0:108:e6aa:91d0 with SMTP id by22csp274205dyb; Thu, 22 Feb 2024 06:07:40 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWGU2+0K9GhnQXDTVehm5J96zYtdf0lg2c3UEswd9uY8ZyvUcGM2vV8/nzXs+7pA+ZMgzq7RwXJ6W67TRdh5dQLsdd9JA== X-Google-Smtp-Source: AGHT+IFUPe/5kMg0acHG32GkCk+W6EsbpQ9NFoXBoLoX9YAb+k3hicSCHj2HadssLvJuXz3B9zTs X-Received: by 2002:a92:c54d:0:b0:365:1563:c4e5 with SMTP id a13-20020a92c54d000000b003651563c4e5mr22060667ilj.9.1708610860316; Thu, 22 Feb 2024 06:07:40 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708610860; cv=pass; d=google.com; s=arc-20160816; b=jR4ueThGKpxJq4QEhZi5XFbbPDzs5NU5nIJyWTTxYHdkSBpWfn7qd9dHwzjBwlpLjs nTYrKLj5R7aauyn6fq/Uc62OYjL+Z56Oe4/DCstAnYjmrybue4ijagVvC4AzHmvofX1D IDppaktDK2weGtqjwc2bftdAkCxQow61kM3hZQWSfT6CyZ+d0LlGl4kQNVCoRthJak65 M9IBRmHWJ2uoxgkVG7Y0u2W+Bt+BhcXlA6+cfLuuQc7LLWxJV1oyeeSxVq0BMR4kwDao zhv0wr1UsUUxWH8cY3FRENYaeImgRCUO2eRxorE75UlKYzRH+28HPthEGoNJPq93vBnk GNrA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=PkfbG9EDgmYztgYWXLtb18KzwOAsDRoGbeegEzJG3FQ=; fh=yRWAnXCrDeo1JkXV9iUtee7xVlLig8radlX7vPmVxyI=; b=WMSpOKhsiXjbDROW9Ng5+lX/g79keBEF+BnwGZssn645rMaCAnFIuo1tzSFasTCH3f oscoqKkOrRF7HNQwdfn9xXf4Ww6R2noPZHM+7EVsD0CToaf16UyeBEjsbqLG0vrFMIQc ob2dVzETj8Lc3L7+jMjjsX17emuGkIoCV2Mwgo8UxBBZRGvxY24y652MNacXIdoazfwt puvGJfa6aFKKCr78sA0xWfP+XBPvj91cITyJMci8PZd5K1zB6izn5WH4N1IhwI36Lev9 62syIijegILEn+7/K4p558aIA3oK+RCdwZ/2d77ux/x5Y+NKIqaWbG3FqJVpLVjoWDyb 8kXA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=t41r5HaT; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-76669-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-76669-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id z16-20020a92ced0000000b00363d6c094desi4451632ilq.59.2024.02.22.06.07.40 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Feb 2024 06:07:40 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-76669-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=t41r5HaT; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-76669-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-76669-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id AB588287728 for ; Thu, 22 Feb 2024 14:07:04 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 6CBF814E2EF; Thu, 22 Feb 2024 14:05:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="t41r5HaT" Received: from out-174.mta0.migadu.com (out-174.mta0.migadu.com [91.218.175.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7FDE11474BD for ; Thu, 22 Feb 2024 14:05:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708610716; cv=none; b=BvMd/kzRAZNqrimzKMENJ3S61j5nORVK6jVcY+/IZQ+3tY1CPCnWwqHE9hrciaomPHrDHBcNqYuSav9WHthUUVFDhuLy8Z4LuUGIe2FmwypPBqn4UK0Nt/IUXM77MJyn2gfgKzTy4N3IInkIepgw+hZ1/CYMxk/pW3PdjB7Cbts= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708610716; c=relaxed/simple; bh=RPjBKA6DR5wkHsrMFWa2oUnkIDrSZ9zmAWeJ9svygWA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=q+1gxaKeeV7aFDXH2mwRgmkLzuqSnkOR0hiOKP0tehAeMAWX7qHu4645Xi+32Z7AWP576hxgpDfzns1XN876ksiJXNNGUhd1nIZrtX4zGac451j8lO1KUFhcEQn/ph6MZJLYL4hmLfoId7dzk9fkjV+AqN2ZWYbDdpxTzKDwKKs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=t41r5HaT; arc=none smtp.client-ip=91.218.175.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1708610711; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PkfbG9EDgmYztgYWXLtb18KzwOAsDRoGbeegEzJG3FQ=; b=t41r5HaT0mYHvm84//2lfA9azmzwtZPkLtm5E0DMzyiun/u/Gc1oQ9waBm96kAePKKkDSe Hi+OLO9VM8zbSrf+T3I+INs/MUO9PYJdI1PkVQUVo0g6y2VYj2OdAF9TygrPCwHNCzMTVQ ooYuq9JMPkcTZoUsPL2yAYMTAI4NREU= From: Gang Li To: Andrew Morton Cc: David Hildenbrand , David Rientjes , Muchun Song , Tim Chen , Steffen Klassert , Daniel Jordan , Jane Chu , "Paul E . McKenney" , Randy Dunlap , linux-mm@kvack.org, linux-kernel@vger.kernel.org, ligang.bdlg@bytedance.com, Gang Li Subject: [PATCH v6 8/8] hugetlb: parallelize 1G hugetlb initialization Date: Thu, 22 Feb 2024 22:04:21 +0800 Message-Id: <20240222140422.393911-9-gang.li@linux.dev> In-Reply-To: <20240222140422.393911-1-gang.li@linux.dev> References: <20240222140422.393911-1-gang.li@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791608341679797732 X-GMAIL-MSGID: 1791608341679797732 Optimizing the initialization speed of 1G huge pages through parallelization. 1G hugetlbs are allocated from bootmem, a process that is already very fast and does not currently require optimization. Therefore, we focus on parallelizing only the initialization phase in `gather_bootmem_prealloc`. Here are some test results: test case no patch(ms) patched(ms) saved ------------------- -------------- ------------- -------- 256c2T(4 node) 1G 4745 2024 57.34% 128c1T(2 node) 1G 3358 1712 49.02% 12T 1G 77000 18300 76.23% Signed-off-by: Gang Li Tested-by: David Rientjes Reviewed-by: Muchun Song --- arch/powerpc/mm/hugetlbpage.c | 2 +- include/linux/hugetlb.h | 2 +- mm/hugetlb.c | 51 +++++++++++++++++++++++++++++------ 3 files changed, 45 insertions(+), 10 deletions(-) diff --git a/arch/powerpc/mm/hugetlbpage.c b/arch/powerpc/mm/hugetlbpage.c index 16557d008eef5..594a4b7b2ca24 100644 --- a/arch/powerpc/mm/hugetlbpage.c +++ b/arch/powerpc/mm/hugetlbpage.c @@ -226,7 +226,7 @@ static int __init pseries_alloc_bootmem_huge_page(struct hstate *hstate) return 0; m = phys_to_virt(gpage_freearray[--nr_gpages]); gpage_freearray[nr_gpages] = 0; - list_add(&m->list, &huge_boot_pages); + list_add(&m->list, &huge_boot_pages[0]); m->hstate = hstate; return 1; } diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index c1ee640d87b11..77b30a8c6076b 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -178,7 +178,7 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma, struct address_space *hugetlb_page_mapping_lock_write(struct page *hpage); extern int sysctl_hugetlb_shm_group; -extern struct list_head huge_boot_pages; +extern struct list_head huge_boot_pages[MAX_NUMNODES]; /* arch callbacks */ diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 3ce957b3e350b..1ac38e7606461 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -69,7 +69,7 @@ static bool hugetlb_cma_folio(struct folio *folio, unsigned int order) #endif static unsigned long hugetlb_cma_size __initdata; -__initdata LIST_HEAD(huge_boot_pages); +__initdata struct list_head huge_boot_pages[MAX_NUMNODES]; /* for command line parsing */ static struct hstate * __initdata parsed_hstate; @@ -3301,7 +3301,7 @@ int alloc_bootmem_huge_page(struct hstate *h, int nid) int __alloc_bootmem_huge_page(struct hstate *h, int nid) { struct huge_bootmem_page *m = NULL; /* initialize for clang */ - int nr_nodes, node; + int nr_nodes, node = nid; /* do node specific alloc */ if (nid != NUMA_NO_NODE) { @@ -3339,7 +3339,7 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) huge_page_size(h) - PAGE_SIZE); /* Put them into a private list first because mem_map is not up yet */ INIT_LIST_HEAD(&m->list); - list_add(&m->list, &huge_boot_pages); + list_add(&m->list, &huge_boot_pages[node]); m->hstate = h; return 1; } @@ -3390,8 +3390,6 @@ static void __init prep_and_add_bootmem_folios(struct hstate *h, /* Send list for bulk vmemmap optimization processing */ hugetlb_vmemmap_optimize_folios(h, folio_list); - /* Add all new pool pages to free lists in one lock cycle */ - spin_lock_irqsave(&hugetlb_lock, flags); list_for_each_entry_safe(folio, tmp_f, folio_list, lru) { if (!folio_test_hugetlb_vmemmap_optimized(folio)) { /* @@ -3404,23 +3402,25 @@ static void __init prep_and_add_bootmem_folios(struct hstate *h, HUGETLB_VMEMMAP_RESERVE_PAGES, pages_per_huge_page(h)); } + /* Subdivide locks to achieve better parallel performance */ + spin_lock_irqsave(&hugetlb_lock, flags); __prep_account_new_huge_page(h, folio_nid(folio)); enqueue_hugetlb_folio(h, folio); + spin_unlock_irqrestore(&hugetlb_lock, flags); } - spin_unlock_irqrestore(&hugetlb_lock, flags); } /* * Put bootmem huge pages into the standard lists after mem_map is up. * Note: This only applies to gigantic (order > MAX_PAGE_ORDER) pages. */ -static void __init gather_bootmem_prealloc(void) +static void __init gather_bootmem_prealloc_node(unsigned long nid) { LIST_HEAD(folio_list); struct huge_bootmem_page *m; struct hstate *h = NULL, *prev_h = NULL; - list_for_each_entry(m, &huge_boot_pages, list) { + list_for_each_entry(m, &huge_boot_pages[nid], list) { struct page *page = virt_to_page(m); struct folio *folio = (void *)page; @@ -3453,6 +3453,31 @@ static void __init gather_bootmem_prealloc(void) prep_and_add_bootmem_folios(h, &folio_list); } +static void __init gather_bootmem_prealloc_parallel(unsigned long start, + unsigned long end, void *arg) +{ + int nid; + + for (nid = start; nid < end; nid++) + gather_bootmem_prealloc_node(nid); +} + +static void __init gather_bootmem_prealloc(void) +{ + struct padata_mt_job job = { + .thread_fn = gather_bootmem_prealloc_parallel, + .fn_arg = NULL, + .start = 0, + .size = num_node_state(N_MEMORY), + .align = 1, + .min_chunk = 1, + .max_threads = num_node_state(N_MEMORY), + .numa_aware = true, + }; + + padata_do_multithreaded(&job); +} + static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) { unsigned long i; @@ -3600,6 +3625,7 @@ static unsigned long __init hugetlb_pages_alloc_boot(struct hstate *h) static void __init hugetlb_hstate_alloc_pages(struct hstate *h) { unsigned long allocated; + static bool initialied __initdata; /* skip gigantic hugepages allocation if hugetlb_cma enabled */ if (hstate_is_gigantic(h) && hugetlb_cma_size) { @@ -3607,6 +3633,15 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) return; } + /* hugetlb_hstate_alloc_pages will be called many times, initialize huge_boot_pages once */ + if (!initialied) { + int i = 0; + + for (i = 0; i < MAX_NUMNODES; i++) + INIT_LIST_HEAD(&huge_boot_pages[i]); + initialied = true; + } + /* do node specific alloc */ if (hugetlb_hstate_alloc_pages_specific_nodes(h)) return;