| Message ID | 20230202215625.3248306-1-usama.arif@bytedance.com |
|---|---|
| Headers |
Return-Path: <linux-kernel-owner@vger.kernel.org>
Delivered-To: ouuuleilei@gmail.com
Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp494058wrn;
Thu, 2 Feb 2023 13:58:33 -0800 (PST)
X-Google-Smtp-Source:
AK7set9KEfH+Aq9QKaWIU4Izxc6Q7FPUahYedVp8LUMB2BiADhWY7/vVIhjGDWCfDeDUKD9kMDAm
X-Received: by 2002:a17:906:2ad4:b0:871:dd2:4af0 with SMTP id
m20-20020a1709062ad400b008710dd24af0mr8040942eje.26.1675375113421;
Thu, 02 Feb 2023 13:58:33 -0800 (PST)
ARC-Seal: i=1; a=rsa-sha256; t=1675375113; cv=none;
d=google.com; s=arc-20160816;
b=qmXdDDA9GPjt83xCIieIwqJWhw1mw3mek1OPHbvDTR5cDZ/IDK0VwKnQsend8KBdjZ
BoTrJE565g9y4UVK2YNoZGzFqwZiGNC1jQ/5vQLd62SOmbsj7Ft/E7JwF52ROcaGuG/0
PmXIOszsv8GxtmMe2Rh2sntH/azpgf+e2/KAIva18mMr4Ac+I2o3loZtKMcwFMIC27F1
CFypyCgFCgUqPz/eOJlE/dmkME/8ZzHs+fytbq0DpI7KE2686WkW8Sytr19LZF0uqQ6s
ofxX8R9fcquMXBVkRFApAWZbuGxR/S1RYICjlV0eCXtQ8t1mWcySprf/UXQiyZCGTQKr
RLtg==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com;
s=arc-20160816;
h=list-id:precedence:content-transfer-encoding:mime-version
:message-id:date:subject:cc:to:from:dkim-signature;
bh=nflrbvUgv2oevNGFK8fKYdaOnsJQhpHjx4vgWukFzaw=;
b=ana3pcfincwbuW7vIF1oO8IOyLT3RRsfnYi093UcvzTqEFGx1IUqHdwmRj645XIleL
+tP2HybRGkpadeLDtSOYC4bvuD+/4b1jk48Yb1vhGMAzjFCG6H94nav89SDOrDkGAa4J
cxZKJkUBzGfYRmzkHU7sOmbJ7gY1wlfTq3R+yjxRguLsoLNff6rir41KoCHRJR23CGRr
8/CIt7//sQAmPUlYvBvmgQN0HKLtEXIjfAhjrUP/1opXnLUQA2mADrcJIAo9pUd8cuyM
BOn5ULI8sa40eXEzQhaMQ7n45IgyzJhj52HdueDICP2ibNaimn51BGIv0oKeEytpUCyf
OJdQ==
ARC-Authentication-Results: i=1; mx.google.com;
dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com
header.s=20210112 header.b="Sa/5uWxp";
spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org
designates 2620:137:e000::1:20 as permitted sender)
smtp.mailfrom=linux-kernel-owner@vger.kernel.org;
dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com
Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20])
by mx.google.com with ESMTP id
fi17-20020a1709073ad100b0088091cca1afsi706105ejc.204.2023.02.02.13.58.09;
Thu, 02 Feb 2023 13:58:33 -0800 (PST)
Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org
designates 2620:137:e000::1:20 as permitted sender)
client-ip=2620:137:e000::1:20;
Authentication-Results: mx.google.com;
dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com
header.s=20210112 header.b="Sa/5uWxp";
spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org
designates 2620:137:e000::1:20 as permitted sender)
smtp.mailfrom=linux-kernel-owner@vger.kernel.org;
dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
id S232628AbjBBV4z (ORCPT <rfc822;il.mystafa@gmail.com> + 99 others);
Thu, 2 Feb 2023 16:56:55 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54972 "EHLO
lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
with ESMTP id S233181AbjBBV4p (ORCPT
<rfc822;linux-kernel@vger.kernel.org>);
Thu, 2 Feb 2023 16:56:45 -0500
Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com
[IPv6:2a00:1450:4864:20::435])
by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D36785B5B8
for <linux-kernel@vger.kernel.org>;
Thu, 2 Feb 2023 13:56:31 -0800 (PST)
Received: by mail-wr1-x435.google.com with SMTP id t7so3002630wrp.5
for <linux-kernel@vger.kernel.org>;
Thu, 02 Feb 2023 13:56:31 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=bytedance-com.20210112.gappssmtp.com; s=20210112;
h=content-transfer-encoding:mime-version:message-id:date:subject:cc
:to:from:from:to:cc:subject:date:message-id:reply-to;
bh=nflrbvUgv2oevNGFK8fKYdaOnsJQhpHjx4vgWukFzaw=;
b=Sa/5uWxp0EGHnk2ouNlrcuW+xmdnnOOmo959km5JrXCv4AO8epKqQnW7S+MSBRPljY
HagfH8ZV6YjboeGM+uSniy2prbQbPkLg/u70SRUyCP7PDdxG/htl8pyu/wX9GLn4xWZY
Uz99ExtSpBdxWk1Su6uu35hdI0Q67Tva2lJKlqNIsM5BmypVFG7VBatco2+tEsRZg3cU
qq75MZqcUHEnQdrk6y2Muqf1TTBlex9evMlqsdeVlnOXzOa1IDMf/Oo8eLBJGHHf7DNg
xfw+YnohBTY4wPmgyKJYqs4U9S1DFr731+QV64Vw15bo0jrBZ0rYScSyDnO0wwmoF37c
nPAQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20210112;
h=content-transfer-encoding:mime-version:message-id:date:subject:cc
:to:from:x-gm-message-state:from:to:cc:subject:date:message-id
:reply-to;
bh=nflrbvUgv2oevNGFK8fKYdaOnsJQhpHjx4vgWukFzaw=;
b=jKuW32xFck0TE3O0yHfxktf/2NRvlTT0XYs/pSlJJvrWC1FTIJpnAW2DNYwyFTHCy0
mHFvvVqfhspsM1FhvPXHP2sOekI6fp7b9r6gX9vMY0lDHfCbRA8J8Z/Ibei9ikJddlPq
xwGLV1EnSHbkQJXgyTn1+rmHc6MtRr0uPCFMoa60oca0GBoUdggq/lq8BbeapmNRTjEL
yq/+qKDxmPgLMSsoXS64zoK4n0jfAZMPc3/xtXJ/NB0vw8nbvrbbWGL/fla3fYbIF2hl
7RDGs3n8fnQpVh1RQgAORhlzT5dLZHuSDYaw+2m6Nb0jNlPV0vqWjueSAYYZLA4auNy8
fGPQ==
X-Gm-Message-State: AO0yUKXU60L1MrvLPOh2mc6r0E5dVE4zghKN+QyJ4KdiGovMCv3B9ie/
EsbSV6ufJxuTPK91CnrkDPlk1Q==
X-Received: by 2002:adf:e9ca:0:b0:2c3:c4cd:2e28 with SMTP id
l10-20020adfe9ca000000b002c3c4cd2e28mr3013325wrn.26.1675374990384;
Thu, 02 Feb 2023 13:56:30 -0800 (PST)
Received: from usaari01.cust.communityfibre.co.uk
([2a02:6b6a:b566:0:98fe:e4ee:fc7e:cd71])
by smtp.gmail.com with ESMTPSA id
e8-20020a5d6d08000000b00297dcfdc90fsm506078wrq.24.2023.02.02.13.56.29
(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
Thu, 02 Feb 2023 13:56:29 -0800 (PST)
From: Usama Arif <usama.arif@bytedance.com>
To: dwmw2@infradead.org, tglx@linutronix.de, arjan@linux.intel.com
Cc: mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com,
hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com,
paulmck@kernel.org, linux-kernel@vger.kernel.org,
kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de,
hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com,
pmenzel@molgen.mpg.de, fam.zheng@bytedance.com,
punit.agrawal@bytedance.com, simon.evans@bytedance.com,
liangma@liangbit.com, Usama Arif <usama.arif@bytedance.com>
Subject: [PATCH v6 00/11] Parallel CPU bringup for x86_64
Date: Thu, 2 Feb 2023 21:56:14 +0000
Message-Id: <20230202215625.3248306-1-usama.arif@bytedance.com>
X-Mailer: git-send-email 2.25.1
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED,
DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham
autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?=
X-GMAIL-THRID: =?utf-8?q?1756758134834555664?=
X-GMAIL-MSGID: =?utf-8?q?1756758134834555664?=
|
| Series |
Parallel CPU bringup for x86_64
|
|
Message
Usama Arif
Feb. 2, 2023, 9:56 p.m. UTC
Doing INIT/SIPI/SIPI in parallel brings down the time for smpboot from ~700ms
to 100ms (85% improvement) on a server with 128 CPUs split across 2 NUMA nodes.
The parallel CPU bringup is disabled for all AMD CPUs in this version:
(see discussions: https://lore.kernel.org/all/bc3f2b1332c4bb77558df8aa36493a55542fe5b9.camel@infradead.org/ and
https://lore.kernel.org/all/3b6ac86fdc800cac5806433daf14a9095be101e9.camel@infradead.org/).
The last patch reuses the timer calibration from CPU 0 for secondary CPUs
which brings down the time for parallel smpboot from 100ms to 30ms. It is
missing a sign-off from the author, which hopefully Arjan will add.
Changes across versions:
v2: Cut it back to just INIT/SIPI/SIPI in parallel for now, nothing more
v3: Clean up x2apic patch, add MTRR optimisation, lock topology update
in preparation for more parallelisation.
v4: Fixes to the real mode parallelisation patch spotted by SeanC, to
avoid scribbling on initial_gs in common_cpu_up(), and to allow all
24 bits of the physical X2APIC ID to be used. That patch still needs
a Signed-off-by from its original author, who once claimed not to
remember writing it at all. But now we've fixed it, hopefully he'll
admit it now :)
v5: rebase to v6.1 and remeasure performance, disable parallel bringup
for AMD CPUs.
v6: rebase to v6.2-rc6, disabled parallel boot on amd as a cpu bug and
reused timer calibration for secondary CPUs.
Arjan van de Ven (1):
x86/smpboot: reuse timer calibration
David Woodhouse (9):
x86/apic/x2apic: Fix parallel handling of cluster_mask
cpu/hotplug: Move idle_thread_get() to <linux/smpboot.h>
cpu/hotplug: Add dynamic parallel bringup states before
CPUHP_BRINGUP_CPU
x86/smpboot: Reference count on smpboot_setup_warm_reset_vector()
x86/smpboot: Split up native_cpu_up into separate phases and document
them
x86/smpboot: Disable parallel boot for AMD CPUs
x86/smpboot: Send INIT/SIPI/SIPI to secondary CPUs in parallel
x86/mtrr: Avoid repeated save of MTRRs on boot-time CPU bringup
x86/smpboot: Serialize topology updates for secondary bringup
Thomas Gleixner (1):
x86/smpboot: Support parallel startup of secondary CPUs
arch/x86/include/asm/cpufeatures.h | 1 +
arch/x86/include/asm/realmode.h | 3 +
arch/x86/include/asm/smp.h | 13 +-
arch/x86/include/asm/topology.h | 2 -
arch/x86/kernel/acpi/sleep.c | 1 +
arch/x86/kernel/apic/apic.c | 2 +-
arch/x86/kernel/apic/x2apic_cluster.c | 108 +++++----
arch/x86/kernel/cpu/amd.c | 11 +
arch/x86/kernel/cpu/common.c | 6 +-
arch/x86/kernel/cpu/mtrr/mtrr.c | 9 +
arch/x86/kernel/head_64.S | 75 ++++++
arch/x86/kernel/smpboot.c | 333 ++++++++++++++++++--------
arch/x86/kernel/tsc.c | 3 +
arch/x86/realmode/init.c | 3 +
arch/x86/realmode/rm/trampoline_64.S | 14 ++
arch/x86/xen/smp_pv.c | 4 +-
include/linux/cpuhotplug.h | 2 +
include/linux/smpboot.h | 7 +
kernel/cpu.c | 27 ++-
kernel/smpboot.c | 2 +-
kernel/smpboot.h | 2 -
21 files changed, 469 insertions(+), 159 deletions(-)
Comments
On Thu, Feb 02, 2023 at 09:56:14PM +0000, Usama Arif wrote: > Doing INIT/SIPI/SIPI in parallel brings down the time for smpboot from ~700ms > to 100ms (85% improvement) on a server with 128 CPUs split across 2 NUMA nodes. > The parallel CPU bringup is disabled for all AMD CPUs in this version: > (see discussions: https://lore.kernel.org/all/bc3f2b1332c4bb77558df8aa36493a55542fe5b9.camel@infradead.org/ and > https://lore.kernel.org/all/3b6ac86fdc800cac5806433daf14a9095be101e9.camel@infradead.org/). > > The last patch reuses the timer calibration from CPU 0 for secondary CPUs > which brings down the time for parallel smpboot from 100ms to 30ms. It is > missing a sign-off from the author, which hopefully Arjan will add. > > Changes across versions: > v2: Cut it back to just INIT/SIPI/SIPI in parallel for now, nothing more > v3: Clean up x2apic patch, add MTRR optimisation, lock topology update > in preparation for more parallelisation. > v4: Fixes to the real mode parallelisation patch spotted by SeanC, to > avoid scribbling on initial_gs in common_cpu_up(), and to allow all > 24 bits of the physical X2APIC ID to be used. That patch still needs > a Signed-off-by from its original author, who once claimed not to > remember writing it at all. But now we've fixed it, hopefully he'll > admit it now :) > v5: rebase to v6.1 and remeasure performance, disable parallel bringup > for AMD CPUs. > v6: rebase to v6.2-rc6, disabled parallel boot on amd as a cpu bug and > reused timer calibration for secondary CPUs. > > Arjan van de Ven (1): > x86/smpboot: reuse timer calibration > > David Woodhouse (9): > x86/apic/x2apic: Fix parallel handling of cluster_mask > cpu/hotplug: Move idle_thread_get() to <linux/smpboot.h> > cpu/hotplug: Add dynamic parallel bringup states before > CPUHP_BRINGUP_CPU > x86/smpboot: Reference count on smpboot_setup_warm_reset_vector() > x86/smpboot: Split up native_cpu_up into separate phases and document > them > x86/smpboot: Disable parallel boot for AMD CPUs > x86/smpboot: Send INIT/SIPI/SIPI to secondary CPUs in parallel > x86/mtrr: Avoid repeated save of MTRRs on boot-time CPU bringup > x86/smpboot: Serialize topology updates for secondary bringup > > Thomas Gleixner (1): > x86/smpboot: Support parallel startup of secondary CPUs > > arch/x86/include/asm/cpufeatures.h | 1 + > arch/x86/include/asm/realmode.h | 3 + > arch/x86/include/asm/smp.h | 13 +- > arch/x86/include/asm/topology.h | 2 - > arch/x86/kernel/acpi/sleep.c | 1 + > arch/x86/kernel/apic/apic.c | 2 +- > arch/x86/kernel/apic/x2apic_cluster.c | 108 +++++---- > arch/x86/kernel/cpu/amd.c | 11 + > arch/x86/kernel/cpu/common.c | 6 +- > arch/x86/kernel/cpu/mtrr/mtrr.c | 9 + > arch/x86/kernel/head_64.S | 75 ++++++ > arch/x86/kernel/smpboot.c | 333 ++++++++++++++++++-------- > arch/x86/kernel/tsc.c | 3 + > arch/x86/realmode/init.c | 3 + > arch/x86/realmode/rm/trampoline_64.S | 14 ++ > arch/x86/xen/smp_pv.c | 4 +- > include/linux/cpuhotplug.h | 2 + > include/linux/smpboot.h | 7 + > kernel/cpu.c | 27 ++- > kernel/smpboot.c | 2 +- > kernel/smpboot.h | 2 - > 21 files changed, 469 insertions(+), 159 deletions(-) > > -- > 2.25.1 Gave the v6 patchset a try on a system with 1920 logocal cpus (sixteen 60 core Sapphire Rapids sockets with Hyperthreadding enabled). Without the patchset it took 71 seconds to start all the cpus. With the v6 patchset it took 14 seconds to start all the cpus, a reduction of 57 seconds. That is impressive. Full boot, to root login prompt, without patches takes 223 seconds. This patchset reduces the full boot time by 57 seconds, a 25% reduction. Well done. Output with the patchset: -------------------------------------------------------------------------- 2023-02-05 18:12:19Z [ 1.172564][ T1] smp: Bringing up secondary CPUs ... 2023-02-05 18:12:19Z [ 1.179316][ T1] x86: Booting SMP configuration: 2023-02-05 18:12:19Z [ 1.179634][ T1] .... node #0, CPUs: #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15 #16 #17 #18 #19 #20 #21 #22 #23 #24 #25 #26 #27 #28 #29 #30 #31 #32 #33 #34 #35 #36 #37 #38 #39 #40 #41 #42 #43 #44 #45 #46 #47 #48 #49 #50 #51 #52 #53 #54 #55 #56 #57 #58 #59 2023-02-05 18:12:20Z [ 1.267650][ T1] .... node #1, CPUs: #60 #61 #62 #63 #64 #65 #66 #67 #68 #69 #70 #71 #72 #73 #74 #75 #76 #77 #78 #79 #80 #81 #82 #83 #84 #85 #86 #87 #88 #89 #90 #91 #92 #93 #94 #95 #96 #97 #98 #99 #100 #101 #102 #103 #104 #105 #106 #107 #108 #109 #110 #111 #112 #113 #114 #115 #116 #117 #118 #119 2023-02-05 18:12:20Z [ 1.315633][ T1] .... node #2, CPUs: #120 #121 #122 #123 #124 #125 #126 #127 #128 #129 #130 #131 #132 #133 #134 #135 #136 #137 #138 #139 #140 #141 #142 #143 #144 #145 #146 #147 #148 #149 #150 #151 #152 #153 #154 #155 #156 #157 #158 #159 #160 #161 #162 #163 #164 #165 #166 #167 #168 #169 #170 #171 #172 #173 #174 #175 #176 #177 #178 #179 2023-02-05 18:12:20Z [ 1.363634][ T1] .... node #3, CPUs: #180 #181 #182 #183 #184 #185 #186 #187 #188 #189 #190 #191 #192 #193 #194 #195 #196 #197 #198 #199 #200 #201 #202 #203 #204 #205 #206 #207 #208 #209 #210 #211 #212 #213 #214 #215 #216 #217 #218 #219 #220 #221 #222 #223 #224 #225 #226 #227 #228 #229 #230 #231 #232 #233 #234 #235 #236 #237 #238 #239 2023-02-05 18:12:20Z [ 1.411634][ T1] .... node #4, CPUs: #240 #241 #242 #243 #244 #245 #246 #247 #248 #249 #250 #251 #252 #253 #254 #255 #256 #257 #258 #259 #260 #261 #262 #263 #264 #265 #266 #267 #268 #269 #270 #271 #272 #273 #274 #275 #276 #277 #278 #279 #280 #281 #282 #283 #284 #285 #286 #287 #288 #289 #290 #291 #292 #293 #294 #295 #296 #297 #298 #299 2023-02-05 18:12:20Z [ 1.459634][ T1] .... node #5, CPUs: #300 #301 #302 #303 #304 #305 #306 #307 #308 #309 #310 #311 #312 #313 #314 #315 #316 #317 #318 #319 #320 #321 #322 #323 #324 #325 #326 #327 #328 #329 #330 #331 #332 #333 #334 #335 #336 #337 #338 #339 #340 #341 #342 #343 #344 #345 #346 #347 #348 #349 #350 #351 #352 #353 #354 #355 #356 #357 #358 #359 2023-02-05 18:12:20Z [ 1.507634][ T1] .... node #6, CPUs: #360 #361 #362 #363 #364 #365 #366 #367 #368 #369 #370 #371 #372 #373 #374 #375 #376 #377 #378 #379 #380 #381 #382 #383 #384 #385 #386 #387 #388 #389 #390 #391 #392 #393 #394 #395 #396 #397 #398 #399 #400 #401 #402 #403 #404 #405 #406 #407 #408 #409 #410 #411 #412 #413 #414 #415 #416 #417 #418 #419 2023-02-05 18:12:20Z [ 1.555634][ T1] .... node #7, CPUs: #420 #421 #422 #423 #424 #425 #426 #427 #428 #429 #430 #431 #432 #433 #434 #435 #436 #437 #438 #439 #440 #441 #442 #443 #444 #445 #446 #447 #448 #449 #450 #451 #452 #453 #454 #455 #456 #457 #458 #459 #460 #461 #462 #463 #464 #465 #466 #467 #468 #469 #470 #471 #472 #473 #474 #475 #476 #477 #478 #479 2023-02-05 18:12:20Z [ 1.603634][ T1] .... node #8, CPUs: #480 #481 #482 #483 #484 #485 #486 #487 #488 #489 #490 #491 #492 #493 #494 #495 #496 #497 #498 #499 #500 #501 #502 #503 #504 #505 #506 #507 #508 #509 #510 #511 #512 #513 #514 #515 #516 #517 #518 #519 #520 #521 #522 #523 #524 #525 #526 #527 #528 #529 #530 #531 #532 #533 #534 #535 #536 #537 #538 #539 2023-02-05 18:12:20Z [ 1.655634][ T1] .... node #9, CPUs: #540 #541 #542 #543 #544 #545 #546 #547 #548 #549 #550 #551 #552 #553 #554 #555 #556 #557 #558 #559 #560 #561 #562 #563 #564 #565 #566 #567 #568 #569 #570 #571 #572 #573 #574 #575 #576 #577 #578 #579 #580 #581 #582 #583 #584 #585 #586 #587 #588 #589 #590 #591 #592 #593 #594 #595 #596 #597 #598 #599 2023-02-05 18:12:20Z [ 1.707633][ T1] .... node #10, CPUs: #600 #601 #602 #603 #604 #605 #606 #607 #608 #609 #610 #611 #612 #613 #614 #615 #616 #617 #618 #619 #620 #621 #622 #623 #624 #625 #626 #627 #628 #629 #630 #631 #632 #633 #634 #635 #636 #637 #638 #639 #640 #641 #642 #643 #644 #645 #646 #647 #648 #649 #650 #651 #652 #653 #654 #655 #656 #657 #658 #659 2023-02-05 18:12:20Z [ 1.755634][ T1] .... node #11, CPUs: #660 #661 #662 #663 #664 #665 #666 #667 #668 #669 #670 #671 #672 #673 #674 #675 #676 #677 #678 #679 #680 #681 #682 #683 #684 #685 #686 #687 #688 #689 #690 #691 #692 #693 #694 #695 #696 #697 #698 #699 #700 #701 #702 #703 #704 #705 #706 #707 #708 #709 #710 #711 #712 #713 #714 #715 #716 #717 #718 #719 2023-02-05 18:12:20Z [ 1.803634][ T1] .... node #12, CPUs: #720 #721 #722 #723 #724 #725 #726 #727 #728 #729 #730 #731 #732 #733 #734 #735 #736 #737 #738 #739 #740 #741 #742 #743 #744 #745 #746 #747 #748 #749 #750 #751 #752 #753 #754 #755 #756 #757 #758 #759 #760 #761 #762 #763 #764 #765 #766 #767 #768 #769 #770 #771 #772 #773 #774 #775 #776 #777 #778 #779 2023-02-05 18:12:21Z [ 1.855634][ T1] .... node #13, CPUs: #780 #781 #782 #783 #784 #785 #786 #787 #788 #789 #790 #791 #792 #793 #794 #795 #796 #797 #798 #799 #800 #801 #802 #803 #804 #805 #806 #807 #808 #809 #810 #811 #812 #813 #814 #815 #816 #817 #818 #819 #820 #821 #822 #823 #824 #825 #826 #827 #828 #829 #830 #831 #832 #833 #834 #835 #836 #837 #838 #839 2023-02-05 18:12:21Z [ 1.903641][ T1] .... node #14, CPUs: #840 #841 #842 #843 #844 #845 #846 #847 #848 #849 #850 #851 #852 #853 #854 #855 #856 #857 #858 #859 #860 #861 #862 #863 #864 #865 #866 #867 #868 #869 #870 #871 #872 #873 #874 #875 #876 #877 #878 #879 #880 #881 #882 #883 #884 #885 #886 #887 #888 #889 #890 #891 #892 #893 #894 #895 #896 #897 #898 #899 2023-02-05 18:12:21Z [ 1.951634][ T1] .... node #15, CPUs: #900 #901 #902 #903 #904 #905 #906 #907 #908 #909 #910 #911 #912 #913 #914 #915 #916 #917 #918 #919 #920 #921 #922 #923 #924 #925 #926 #927 #928 #929 #930 #931 #932 #933 #934 #935 #936 #937 #938 #939 #940 #941 #942 #943 #944 #945 #946 #947 #948 #949 #950 #951 #952 #953 #954 #955 #956 #957 #958 #959 2023-02-05 18:12:21Z [ 1.999634][ T1] .... node #0, CPUs: #960 #961 #962 #963 #964 #965 #966 #967 #968 #969 #970 #971 #972 #973 #974 #975 #976 #977 #978 #979 #980 #981 #982 #983 #984 #985 #986 #987 #988 #989 #990 #991 #992 #993 #994 #995 #996 #997 #998 #999 #1000 #1001 #1002 #1003 #1004 #1005 #1006 #1007 #1008 #1009 #1010 #1011 #1012 #1013 #1014 #1015 #1016 #1017 #1018 #1019 2023-02-05 18:12:21Z [ 2.079642][ T1] .... node #1, CPUs: #1020 #1021 #1022 #1023 #1024 #1025 #1026 #1027 #1028 #1029 #1030 #1031 #1032 #1033 #1034 #1035 #1036 #1037 #1038 #1039 #1040 #1041 #1042 #1043 #1044 #1045 #1046 #1047 #1048 #1049 #1050 #1051 #1052 #1053 #1054 #1055 #1056 #1057 #1058 #1059 #1060 #1061 #1062 #1063 #1064 #1065 #1066 #1067 #1068 #1069 #1070 #1071 #1072 #1073 #1074 #1075 #1076 #1077 #1078 #1079 2023-02-05 18:12:21Z [ 2.119634][ T1] .... node #2, CPUs: #1080 #1081 #1082 #1083 #1084 #1085 #1086 #1087 #1088 #1089 #1090 #1091 #1092 #1093 #1094 #1095 #1096 #1097 #1098 #1099 #1100 #1101 #1102 #1103 #1104 #1105 #1106 #1107 #1108 #1109 #1110 #1111 #1112 #1113 #1114 #1115 #1116 #1117 #1118 #1119 #1120 #1121 #1122 #1123 #1124 #1125 #1126 #1127 #1128 #1129 #1130 #1131 #1132 #1133 #1134 #1135 #1136 #1137 #1138 #1139 2023-02-05 18:12:21Z [ 2.155634][ T1] .... node #3, CPUs: #1140 #1141 #1142 #1143 #1144 #1145 #1146 #1147 #1148 #1149 #1150 #1151 #1152 #1153 #1154 #1155 #1156 #1157 #1158 #1159 #1160 #1161 #1162 #1163 #1164 #1165 #1166 #1167 #1168 #1169 #1170 #1171 #1172 #1173 #1174 #1175 #1176 #1177 #1178 #1179 #1180 #1181 #1182 #1183 #1184 #1185 #1186 #1187 #1188 #1189 #1190 #1191 #1192 #1193 #1194 #1195 #1196 #1197 #1198 #1199 2023-02-05 18:12:21Z [ 2.195634][ T1] .... node #4, CPUs: #1200 #1201 #1202 #1203 #1204 #1205 #1206 #1207 #1208 #1209 #1210 #1211 #1212 #1213 #1214 #1215 #1216 #1217 #1218 #1219 #1220 #1221 #1222 #1223 #1224 #1225 #1226 #1227 #1228 #1229 #1230 #1231 #1232 #1233 #1234 #1235 #1236 #1237 #1238 #1239 #1240 #1241 #1242 #1243 #1244 #1245 #1246 #1247 #1248 #1249 #1250 #1251 #1252 #1253 #1254 #1255 #1256 #1257 #1258 #1259 2023-02-05 18:12:21Z [ 2.235635][ T1] .... node #5, CPUs: #1260 #1261 #1262 #1263 #1264 #1265 #1266 #1267 #1268 #1269 #1270 #1271 #1272 #1273 #1274 #1275 #1276 #1277 #1278 #1279 #1280 #1281 #1282 #1283 #1284 #1285 #1286 #1287 #1288 #1289 #1290 #1291 #1292 #1293 #1294 #1295 #1296 #1297 #1298 #1299 #1300 #1301 #1302 #1303 #1304 #1305 #1306 #1307 #1308 #1309 #1310 #1311 #1312 #1313 #1314 #1315 #1316 #1317 #1318 #1319 2023-02-05 18:12:21Z [ 2.275634][ T1] .... node #6, CPUs: #1320 #1321 #1322 #1323 #1324 #1325 #1326 #1327 #1328 #1329 #1330 #1331 #1332 #1333 #1334 #1335 #1336 #1337 #1338 #1339 #1340 #1341 #1342 #1343 #1344 #1345 #1346 #1347 #1348 #1349 #1350 #1351 #1352 #1353 #1354 #1355 #1356 #1357 #1358 #1359 #1360 #1361 #1362 #1363 #1364 #1365 #1366 #1367 #1368 #1369 #1370 #1371 #1372 #1373 #1374 #1375 #1376 #1377 #1378 #1379 2023-02-05 18:12:21Z [ 2.320561][ T1] .... node #7, CPUs: #1380 #1381 #1382 #1383 #1384 #1385 #1386 #1387 #1388 #1389 #1390 #1391 #1392 #1393 #1394 #1395 #1396 #1397 #1398 #1399 #1400 #1401 #1402 #1403 #1404 #1405 #1406 #1407 #1408 #1409 #1410 #1411 #1412 #1413 #1414 #1415 #1416 #1417 #1418 #1419 #1420 #1421 #1422 #1423 #1424 #1425 #1426 #1427 #1428 #1429 #1430 #1431 #1432 #1433 #1434 #1435 #1436 #1437 #1438 #1439 2023-02-05 18:12:21Z [ 2.363647][ T1] .... node #8, CPUs: #1440 #1441 #1442 #1443 #1444 #1445 #1446 #1447 #1448 #1449 #1450 #1451 #1452 #1453 #1454 #1455 #1456 #1457 #1458 #1459 #1460 #1461 #1462 #1463 #1464 #1465 #1466 #1467 #1468 #1469 #1470 #1471 #1472 #1473 #1474 #1475 #1476 #1477 #1478 #1479 #1480 #1481 #1482 #1483 #1484 #1485 #1486 #1487 #1488 #1489 #1490 #1491 #1492 #1493 #1494 #1495 #1496 #1497 #1498 #1499 2023-02-05 18:12:22Z [ 2.407636][ T1] .... node #9, CPUs: #1500 #1501 #1502 #1503 #1504 #1505 #1506 #1507 #1508 #1509 #1510 #1511 #1512 #1513 #1514 #1515 #1516 #1517 #1518 #1519 #1520 #1521 #1522 #1523 #1524 #1525 #1526 #1527 #1528 #1529 #1530 #1531 #1532 #1533 #1534 #1535 #1536 #1537 #1538 #1539 #1540 #1541 #1542 #1543 #1544 #1545 #1546 #1547 #1548 #1549 #1550 #1551 #1552 #1553 #1554 #1555 #1556 #1557 #1558 #1559 2023-02-05 18:12:22Z [ 2.451644][ T1] .... node #10, CPUs: #1560 #1561 #1562 #1563 #1564 #1565 #1566 #1567 #1568 #1569 #1570 #1571 #1572 #1573 #1574 #1575 #1576 #1577 #1578 #1579 #1580 #1581 #1582 #1583 #1584 #1585 #1586 #1587 #1588 #1589 #1590 #1591 #1592 #1593 #1594 #1595 #1596 #1597 #1598 #1599 #1600 #1601 #1602 #1603 #1604 #1605 #1606 #1607 #1608 #1609 #1610 #1611 #1612 #1613 #1614 #1615 #1616 #1617 #1618 #1619 2023-02-05 18:12:22Z [ 2.495648][ T1] .... node #11, CPUs: #1620 #1621 #1622 #1623 #1624 #1625 #1626 #1627 #1628 #1629 #1630 #1631 #1632 #1633 #1634 #1635 #1636 #1637 #1638 #1639 #1640 #1641 #1642 #1643 #1644 #1645 #1646 #1647 #1648 #1649 #1650 #1651 #1652 #1653 #1654 #1655 #1656 #1657 #1658 #1659 #1660 #1661 #1662 #1663 #1664 #1665 #1666 #1667 #1668 #1669 #1670 #1671 #1672 #1673 #1674 #1675 #1676 #1677 #1678 #1679 2023-02-05 18:12:22Z [ 2.539634][ T1] .... node #12, CPUs: #1680 #1681 #1682 #1683 #1684 #1685 #1686 #1687 #1688 #1689 #1690 #1691 #1692 #1693 #1694 #1695 #1696 #1697 #1698 #1699 #1700 #1701 #1702 #1703 #1704 #1705 #1706 #1707 #1708 #1709 #1710 #1711 #1712 #1713 #1714 #1715 #1716 #1717 #1718 #1719 #1720 #1721 #1722 #1723 #1724 #1725 #1726 #1727 #1728 #1729 #1730 #1731 #1732 #1733 #1734 #1735 #1736 #1737 #1738 #1739 2023-02-05 18:12:22Z [ 2.583644][ T1] .... node #13, CPUs: #1740 #1741 #1742 #1743 #1744 #1745 #1746 #1747 #1748 #1749 #1750 #1751 #1752 #1753 #1754 #1755 #1756 #1757 #1758 #1759 #1760 #1761 #1762 #1763 #1764 #1765 #1766 #1767 #1768 #1769 #1770 #1771 #1772 #1773 #1774 #1775 #1776 #1777 #1778 #1779 #1780 #1781 #1782 #1783 #1784 #1785 #1786 #1787 #1788 #1789 #1790 #1791 #1792 #1793 #1794 #1795 #1796 #1797 #1798 #1799 2023-02-05 18:12:22Z [ 2.627635][ T1] .... node #14, CPUs: #1800 #1801 #1802 #1803 #1804 #1805 #1806 #1807 #1808 #1809 #1810 #1811 #1812 #1813 #1814 #1815 #1816 #1817 #1818 #1819 #1820 #1821 #1822 #1823 #1824 #1825 #1826 #1827 #1828 #1829 #1830 #1831 #1832 #1833 #1834 #1835 #1836 #1837 #1838 #1839 #1840 #1841 #1842 #1843 #1844 #1845 #1846 #1847 #1848 #1849 #1850 #1851 #1852 #1853 #1854 #1855 #1856 #1857 #1858 #1859 2023-02-05 18:12:22Z [ 2.675650][ T1] .... node #15, CPUs: #1860 #1861 #1862 #1863 #1864 #1865 #1866 #1867 #1868 #1869 #1870 #1871 #1872 #1873 #1874 #1875 #1876 #1877 #1878 #1879 #1880 #1881 #1882 #1883 #1884 #1885 #1886 #1887 #1888 #1889 #1890 #1891 #1892 #1893 #1894 #1895 #1896 #1897 #1898 #1899 #1900 #1901 #1902 #1903 #1904 #1905 #1906 #1907 #1908 #1909 #1910 #1911 #1912 #1913 #1914 #1915 #1916 #1917 #1918 #1919 2023-02-05 18:12:22Z [ 0.000000][ T0] smpboot: CPU 60 Converting physical 0 to logical die 1 2023-02-05 18:12:22Z [ 0.000000][ T0] smpboot: CPU 120 Converting physical 0 to logical die 2 2023-02-05 18:12:22Z [ 0.000000][ T0] smpboot: CPU 180 Converting physical 0 to logical die 3 2023-02-05 18:12:23Z [ 0.000000][ T0] smpboot: CPU 240 Converting physical 0 to logical die 4 2023-02-05 18:12:23Z [ 0.000000][ T0] smpboot: CPU 300 Converting physical 0 to logical die 5 2023-02-05 18:12:23Z [ 0.000000][ T0] smpboot: CPU 360 Converting physical 0 to logical die 6 2023-02-05 18:12:23Z [ 0.000000][ T0] smpboot: CPU 420 Converting physical 0 to logical die 7 2023-02-05 18:12:23Z [ 0.000000][ T0] smpboot: CPU 480 Converting physical 0 to logical die 8 2023-02-05 18:12:23Z [ 0.000000][ T0] smpboot: CPU 540 Converting physical 0 to logical die 9 2023-02-05 18:12:24Z [ 0.000000][ T0] smpboot: CPU 600 Converting physical 0 to logical die 10 2023-02-05 18:12:24Z [ 0.000000][ T0] smpboot: CPU 660 Converting physical 0 to logical die 11 2023-02-05 18:12:24Z [ 0.000000][ T0] smpboot: CPU 720 Converting physical 0 to logical die 12 2023-02-05 18:12:25Z [ 0.000000][ T0] smpboot: CPU 780 Converting physical 0 to logical die 13 2023-02-05 18:12:25Z [ 0.000000][ T0] smpboot: CPU 840 Converting physical 0 to logical die 14 2023-02-05 18:12:25Z [ 0.000000][ T0] smpboot: CPU 900 Converting physical 0 to logical die 15 2023-02-05 18:12:35Z [ 15.213189][ T1] smp: Brought up 16 nodes, 1920 CPUs 2023-02-05 18:12:35Z [ 15.215637][ T1] smpboot: Max logical packages: 16 2023-02-05 18:12:35Z [ 15.219750][ T1] smpboot: Total of 1920 processors activated (7296000.00 BogoMIPS) -------------------------------------------------------------------------- Output without the patchset: -------------------------------------------------------------------------- 2023-02-03 11:06:49Z [ 1.157637][ T1] smp: Bringing up secondary CPUs ... 2023-02-03 11:06:49Z [ 1.162807][ T1] x86: Booting SMP configuration: 2023-02-03 11:06:51Z [ 1.164299][ T1] .... node #0, CPUs: #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15 #16 #17 #18 #19 #20 #21 #22 #23 #24 #25 #26 #27 #28 #29 #30 #31 #32 #33 #34 #35 #36 #37 #38 #39 #40 #41 #42 #43 #44 #45 #46 #47 #48 #49 #50 #51 #52 #53 #54 #55 #56 #57 #58 #59 2023-02-03 11:06:51Z [ 3.052299][ T1] .... node #1, CPUs: #60 2023-02-03 11:06:51Z [ 0.000000][ T0] smpboot: CPU 60 Converting physical 0 to logical die 1 2023-02-03 11:06:53Z [ 3.133499][ T1] #61 #62 #63 #64 #65 #66 #67 #68 #69 #70 #71 #72 #73 #74 #75 #76 #77 #78 #79 #80 #81 #82 #83 #84 #85 #86 #87 #88 #89 #90 #91 #92 #93 #94 #95 #96 #97 #98 #99 #100 #101 #102 #103 #104 #105 #106 #107 #108 #109 #110 #111 #112 #113 #114 #115 #116 #117 #118 #119 2023-02-03 11:06:53Z [ 4.984298][ T1] .... node #2, CPUs: #120 2023-02-03 11:06:53Z [ 0.000000][ T0] smpboot: CPU 120 Converting physical 0 to logical die 2 2023-02-03 11:06:55Z [ 5.065813][ T1] #121 #122 #123 #124 #125 #126 #127 #128 #129 #130 #131 #132 #133 #134 #135 #136 #137 #138 #139 #140 #141 #142 #143 #144 #145 #146 #147 #148 #149 #150 #151 #152 #153 #154 #155 #156 #157 #158 #159 #160 #161 #162 #163 #164 #165 #166 #167 #168 #169 #170 #171 #172 #173 #174 #175 #176 #177 #178 #179 2023-02-03 11:06:55Z [ 6.920298][ T1] .... node #3, CPUs: #180 2023-02-03 11:06:55Z [ 0.000000][ T0] smpboot: CPU 180 Converting physical 0 to logical die 3 2023-02-03 11:06:57Z [ 7.002019][ T1] #181 #182 #183 #184 #185 #186 #187 #188 #189 #190 #191 #192 #193 #194 #195 #196 #197 #198 #199 #200 #201 #202 #203 #204 #205 #206 #207 #208 #209 #210 #211 #212 #213 #214 #215 #216 #217 #218 #219 #220 #221 #222 #223 #224 #225 #226 #227 #228 #229 #230 #231 #232 #233 #234 #235 #236 #237 #238 #239 2023-02-03 11:06:57Z [ 8.880299][ T1] .... node #4, CPUs: #240 2023-02-03 11:06:57Z [ 0.000000][ T0] smpboot: CPU 240 Converting physical 0 to logical die 4 2023-02-03 11:06:59Z [ 8.991204][ T1] #241 #242 #243 #244 #245 #246 #247 #248 #249 #250 #251 #252 #253 #254 #255 #256 #257 #258 #259 #260 #261 #262 #263 #264 #265 #266 #267 #268 #269 #270 #271 #272 #273 #274 #275 #276 #277 #278 #279 #280 #281 #282 #283 #284 #285 #286 #287 #288 #289 #290 #291 #292 #293 #294 #295 #296 #297 #298 #299 2023-02-03 11:06:59Z [ 10.944299][ T1] .... node #5, CPUs: #300 2023-02-03 11:06:59Z [ 0.000000][ T0] smpboot: CPU 300 Converting physical 0 to logical die 5 2023-02-03 11:07:01Z [ 11.027023][ T1] #301 #302 #303 #304 #305 #306 #307 #308 #309 #310 #311 #312 #313 #314 #315 #316 #317 #318 #319 #320 #321 #322 #323 #324 #325 #326 #327 #328 #329 #330 #331 #332 #333 #334 #335 #336 #337 #338 #339 #340 #341 #342 #343 #344 #345 #346 #347 #348 #349 #350 #351 #352 #353 #354 #355 #356 #357 #358 #359 2023-02-03 11:07:01Z [ 12.964299][ T1] .... node #6, CPUs: #360 2023-02-03 11:07:01Z [ 0.000000][ T0] smpboot: CPU 360 Converting physical 0 to logical die 6 2023-02-03 11:07:03Z [ 13.047169][ T1] #361 #362 #363 #364 #365 #366 #367 #368 #369 #370 #371 #372 #373 #374 #375 #376 #377 #378 #379 #380 #381 #382 #383 #384 #385 #386 #387 #388 #389 #390 #391 #392 #393 #394 #395 #396 #397 #398 #399 #400 #401 #402 #403 #404 #405 #406 #407 #408 #409 #410 #411 #412 #413 #414 #415 #416 #417 #418 #419 2023-02-03 11:07:03Z [ 14.996299][ T1] .... node #7, CPUs: #420 2023-02-03 11:07:03Z [ 0.000000][ T0] smpboot: CPU 420 Converting physical 0 to logical die 7 2023-02-03 11:07:05Z [ 15.079323][ T1] #421 #422 #423 #424 #425 #426 #427 #428 #429 #430 #431 #432 #433 #434 #435 #436 #437 #438 #439 #440 #441 #442 #443 #444 #445 #446 #447 #448 #449 #450 #451 #452 #453 #454 #455 #456 #457 #458 #459 #460 #461 #462 #463 #464 #465 #466 #467 #468 #469 #470 #471 #472 #473 #474 #475 #476 #477 #478 #479 2023-02-03 11:07:06Z [ 17.048299][ T1] .... node #8, CPUs: #480 2023-02-03 11:07:06Z [ 0.000000][ T0] smpboot: CPU 480 Converting physical 0 to logical die 8 2023-02-03 11:07:08Z [ 17.161077][ T1] #481 #482 #483 #484 #485 #486 #487 #488 #489 #490 #491 #492 #493 #494 #495 #496 #497 #498 #499 #500 #501 #502 #503 #504 #505 #506 #507 #508 #509 #510 #511 #512 #513 #514 #515 #516 #517 #518 #519 #520 #521 #522 #523 #524 #525 #526 #527 #528 #529 #530 #531 #532 #533 #534 #535 #536 #537 #538 #539 2023-02-03 11:07:08Z [ 19.196298][ T1] .... node #9, CPUs: #540 2023-02-03 11:07:08Z [ 0.000000][ T0] smpboot: CPU 540 Converting physical 0 to logical die 9 2023-02-03 11:07:10Z [ 19.281098][ T1] #541 #542 #543 #544 #545 #546 #547 #548 #549 #550 #551 #552 #553 #554 #555 #556 #557 #558 #559 #560 #561 #562 #563 #564 #565 #566 #567 #568 #569 #570 #571 #572 #573 #574 #575 #576 #577 #578 #579 #580 #581 #582 #583 #584 #585 #586 #587 #588 #589 #590 #591 #592 #593 #594 #595 #596 #597 #598 #599 2023-02-03 11:07:10Z [ 21.324298][ T1] .... node #10, CPUs: #600 2023-02-03 11:07:10Z [ 0.000000][ T0] smpboot: CPU 600 Converting physical 0 to logical die 10 2023-02-03 11:07:12Z [ 21.409074][ T1] #601 #602 #603 #604 #605 #606 #607 #608 #609 #610 #611 #612 #613 #614 #615 #616 #617 #618 #619 #620 #621 #622 #623 #624 #625 #626 #627 #628 #629 #630 #631 #632 #633 #634 #635 #636 #637 #638 #639 #640 #641 #642 #643 #644 #645 #646 #647 #648 #649 #650 #651 #652 #653 #654 #655 #656 #657 #658 #659 2023-02-03 11:07:12Z [ 23.468298][ T1] .... node #11, CPUs: #660 2023-02-03 11:07:12Z [ 0.000000][ T0] smpboot: CPU 660 Converting physical 0 to logical die 11 2023-02-03 11:07:14Z [ 23.553115][ T1] #661 #662 #663 #664 #665 #666 #667 #668 #669 #670 #671 #672 #673 #674 #675 #676 #677 #678 #679 #680 #681 #682 #683 #684 #685 #686 #687 #688 #689 #690 #691 #692 #693 #694 #695 #696 #697 #698 #699 #700 #701 #702 #703 #704 #705 #706 #707 #708 #709 #710 #711 #712 #713 #714 #715 #716 #717 #718 #719 2023-02-03 11:07:14Z [ 25.624298][ T1] .... node #12, CPUs: #720 2023-02-03 11:07:14Z [ 0.000000][ T0] smpboot: CPU 720 Converting physical 0 to logical die 12 2023-02-03 11:07:17Z [ 25.741550][ T1] #721 #722 #723 #724 #725 #726 #727 #728 #729 #730 #731 #732 #733 #734 #735 #736 #737 #738 #739 #740 #741 #742 #743 #744 #745 #746 #747 #748 #749 #750 #751 #752 #753 #754 #755 #756 #757 #758 #759 #760 #761 #762 #763 #764 #765 #766 #767 #768 #769 #770 #771 #772 #773 #774 #775 #776 #777 #778 #779 2023-02-03 11:07:17Z [ 27.876299][ T1] .... node #13, CPUs: #780 2023-02-03 11:07:17Z [ 0.000000][ T0] smpboot: CPU 780 Converting physical 0 to logical die 13 2023-02-03 11:07:19Z [ 27.962255][ T1] #781 #782 #783 #784 #785 #786 #787 #788 #789 #790 #791 #792 #793 #794 #795 #796 #797 #798 #799 #800 #801 #802 #803 #804 #805 #806 #807 #808 #809 #810 #811 #812 #813 #814 #815 #816 #817 #818 #819 #820 #821 #822 #823 #824 #825 #826 #827 #828 #829 #830 #831 #832 #833 #834 #835 #836 #837 #838 #839 2023-02-03 11:07:19Z [ 30.096299][ T1] .... node #14, CPUs: #840 2023-02-03 11:07:19Z [ 0.000000][ T0] smpboot: CPU 840 Converting physical 0 to logical die 14 2023-02-03 11:07:21Z [ 30.182443][ T1] #841 #842 #843 #844 #845 #846 #847 #848 #849 #850 #851 #852 #853 #854 #855 #856 #857 #858 #859 #860 #861 #862 #863 #864 #865 #866 #867 #868 #869 #870 #871 #872 #873 #874 #875 #876 #877 #878 #879 #880 #881 #882 #883 #884 #885 #886 #887 #888 #889 #890 #891 #892 #893 #894 #895 #896 #897 #898 #899 2023-02-03 11:07:21Z [ 32.332299][ T1] .... node #15, CPUs: #900 2023-02-03 11:07:21Z [ 0.000000][ T0] smpboot: CPU 900 Converting physical 0 to logical die 15 2023-02-03 11:07:23Z [ 32.420700][ T1] #901 #902 #903 #904 #905 #906 #907 #908 #909 #910 #911 #912 #913 #914 #915 #916 #917 #918 #919 #920 #921 #922 #923 #924 #925 #926 #927 #928 #929 #930 #931 #932 #933 #934 #935 #936 #937 #938 #939 #940 #941 #942 #943 #944 #945 #946 #947 #948 #949 #950 #951 #952 #953 #954 #955 #956 #957 #958 #959 2023-02-03 11:07:26Z [ 34.584298][ T1] .... node #0, CPUs: #960 #961 #962 #963 #964 #965 #966 #967 #968 #969 #970 #971 #972 #973 #974 #975 #976 #977 #978 #979 #980 #981 #982 #983 #984 #985 #986 #987 #988 #989 #990 #991 #992 #993 #994 #995 #996 #997 #998 #999 #1000 #1001 #1002 #1003 #1004 #1005 #1006 #1007 #1008 #1009 #1010 #1011 #1012 #1013 #1014 #1015 #1016 #1017 #1018 #1019 2023-02-03 11:07:28Z [ 36.792301][ T1] .... node #1, CPUs: #1020 #1021 #1022 #1023 #1024 #1025 #1026 #1027 #1028 #1029 #1030 #1031 #1032 #1033 #1034 #1035 #1036 #1037 #1038 #1039 #1040 #1041 #1042 #1043 #1044 #1045 #1046 #1047 #1048 #1049 #1050 #1051 #1052 #1053 #1054 #1055 #1056 #1057 #1058 #1059 #1060 #1061 #1062 #1063 #1064 #1065 #1066 #1067 #1068 #1069 #1070 #1071 #1072 #1073 #1074 #1075 #1076 #1077 #1078 #1079 2023-02-03 11:07:30Z [ 38.980298][ T1] .... node #2, CPUs: #1080 #1081 #1082 #1083 #1084 #1085 #1086 #1087 #1088 #1089 #1090 #1091 #1092 #1093 #1094 #1095 #1096 #1097 #1098 #1099 #1100 #1101 #1102 #1103 #1104 #1105 #1106 #1107 #1108 #1109 #1110 #1111 #1112 #1113 #1114 #1115 #1116 #1117 #1118 #1119 #1120 #1121 #1122 #1123 #1124 #1125 #1126 #1127 #1128 #1129 #1130 #1131 #1132 #1133 #1134 #1135 #1136 #1137 #1138 #1139 2023-02-03 11:07:32Z [ 41.180298][ T1] .... node #3, CPUs: #1140 #1141 #1142 #1143 #1144 #1145 #1146 #1147 #1148 #1149 #1150 #1151 #1152 #1153 #1154 #1155 #1156 #1157 #1158 #1159 #1160 #1161 #1162 #1163 #1164 #1165 #1166 #1167 #1168 #1169 #1170 #1171 #1172 #1173 #1174 #1175 #1176 #1177 #1178 #1179 #1180 #1181 #1182 #1183 #1184 #1185 #1186 #1187 #1188 #1189 #1190 #1191 #1192 #1193 #1194 #1195 #1196 #1197 #1198 #1199 2023-02-03 11:07:35Z [ 43.408300][ T1] .... node #4, CPUs: #1200 #1201 #1202 #1203 #1204 #1205 #1206 #1207 #1208 #1209 #1210 #1211 #1212 #1213 #1214 #1215 #1216 #1217 #1218 #1219 #1220 #1221 #1222 #1223 #1224 #1225 #1226 #1227 #1228 #1229 #1230 #1231 #1232 #1233 #1234 #1235 #1236 #1237 #1238 #1239 #1240 #1241 #1242 #1243 #1244 #1245 #1246 #1247 #1248 #1249 #1250 #1251 #1252 #1253 #1254 #1255 #1256 #1257 #1258 #1259 2023-02-03 11:07:37Z [ 45.744298][ T1] .... node #5, CPUs: #1260 #1261 #1262 #1263 #1264 #1265 #1266 #1267 #1268 #1269 #1270 #1271 #1272 #1273 #1274 #1275 #1276 #1277 #1278 #1279 #1280 #1281 #1282 #1283 #1284 #1285 #1286 #1287 #1288 #1289 #1290 #1291 #1292 #1293 #1294 #1295 #1296 #1297 #1298 #1299 #1300 #1301 #1302 #1303 #1304 #1305 #1306 #1307 #1308 #1309 #1310 #1311 #1312 #1313 #1314 #1315 #1316 #1317 #1318 #1319 2023-02-03 11:07:39Z [ 48.056299][ T1] .... node #6, CPUs: #1320 #1321 #1322 #1323 #1324 #1325 #1326 #1327 #1328 #1329 #1330 #1331 #1332 #1333 #1334 #1335 #1336 #1337 #1338 #1339 #1340 #1341 #1342 #1343 #1344 #1345 #1346 #1347 #1348 #1349 #1350 #1351 #1352 #1353 #1354 #1355 #1356 #1357 #1358 #1359 #1360 #1361 #1362 #1363 #1364 #1365 #1366 #1367 #1368 #1369 #1370 #1371 #1372 #1373 #1374 #1375 #1376 #1377 #1378 #1379 2023-02-03 11:07:42Z [ 50.380298][ T1] .... node #7, CPUs: #1380 #1381 #1382 #1383 #1384 #1385 #1386 #1387 #1388 #1389 #1390 #1391 #1392 #1393 #1394 #1395 #1396 #1397 #1398 #1399 #1400 #1401 #1402 #1403 #1404 #1405 #1406 #1407 #1408 #1409 #1410 #1411 #1412 #1413 #1414 #1415 #1416 #1417 #1418 #1419 #1420 #1421 #1422 #1423 #1424 #1425 #1426 #1427 #1428 #1429 #1430 #1431 #1432 #1433 #1434 #1435 #1436 #1437 #1438 #1439 2023-02-03 11:07:44Z [ 52.736298][ T1] .... node #8, CPUs: #1440 #1441 #1442 #1443 #1444 #1445 #1446 #1447 #1448 #1449 #1450 #1451 #1452 #1453 #1454 #1455 #1456 #1457 #1458 #1459 #1460 #1461 #1462 #1463 #1464 #1465 #1466 #1467 #1468 #1469 #1470 #1471 #1472 #1473 #1474 #1475 #1476 #1477 #1478 #1479 #1480 #1481 #1482 #1483 #1484 #1485 #1486 #1487 #1488 #1489 #1490 #1491 #1492 #1493 #1494 #1495 #1496 #1497 #1498 #1499 2023-02-03 11:07:47Z [ 55.164301][ T1] .... node #9, CPUs: #1500 #1501 #1502 #1503 #1504 #1505 #1506 #1507 #1508 #1509 #1510 #1511 #1512 #1513 #1514 #1515 #1516 #1517 #1518 #1519 #1520 #1521 #1522 #1523 #1524 #1525 #1526 #1527 #1528 #1529 #1530 #1531 #1532 #1533 #1534 #1535 #1536 #1537 #1538 #1539 #1540 #1541 #1542 #1543 #1544 #1545 #1546 #1547 #1548 #1549 #1550 #1551 #1552 #1553 #1554 #1555 #1556 #1557 #1558 #1559 2023-02-03 11:07:49Z [ 57.592298][ T1] .... node #10, CPUs: #1560 #1561 #1562 #1563 #1564 #1565 #1566 #1567 #1568 #1569 #1570 #1571 #1572 #1573 #1574 #1575 #1576 #1577 #1578 #1579 #1580 #1581 #1582 #1583 #1584 #1585 #1586 #1587 #1588 #1589 #1590 #1591 #1592 #1593 #1594 #1595 #1596 #1597 #1598 #1599 #1600 #1601 #1602 #1603 #1604 #1605 #1606 #1607 #1608 #1609 #1610 #1611 #1612 #1613 #1614 #1615 #1616 #1617 #1618 #1619 2023-02-03 11:07:52Z [ 60.032299][ T1] .... node #11, CPUs: #1620 #1621 #1622 #1623 #1624 #1625 #1626 #1627 #1628 #1629 #1630 #1631 #1632 #1633 #1634 #1635 #1636 #1637 #1638 #1639 #1640 #1641 #1642 #1643 #1644 #1645 #1646 #1647 #1648 #1649 #1650 #1651 #1652 #1653 #1654 #1655 #1656 #1657 #1658 #1659 #1660 #1661 #1662 #1663 #1664 #1665 #1666 #1667 #1668 #1669 #1670 #1671 #1672 #1673 #1674 #1675 #1676 #1677 #1678 #1679 2023-02-03 11:07:54Z [ 62.484298][ T1] .... node #12, CPUs: #1680 #1681 #1682 #1683 #1684 #1685 #1686 #1687 #1688 #1689 #1690 #1691 #1692 #1693 #1694 #1695 #1696 #1697 #1698 #1699 #1700 #1701 #1702 #1703 #1704 #1705 #1706 #1707 #1708 #1709 #1710 #1711 #1712 #1713 #1714 #1715 #1716 #1717 #1718 #1719 #1720 #1721 #1722 #1723 #1724 #1725 #1726 #1727 #1728 #1729 #1730 #1731 #1732 #1733 #1734 #1735 #1736 #1737 #1738 #1739 2023-02-03 11:07:57Z [ 65.024298][ T1] .... node #13, CPUs: #1740 #1741 #1742 #1743 #1744 #1745 #1746 #1747 #1748 #1749 #1750 #1751 #1752 #1753 #1754 #1755 #1756 #1757 #1758 #1759 #1760 #1761 #1762 #1763 #1764 #1765 #1766 #1767 #1768 #1769 #1770 #1771 #1772 #1773 #1774 #1775 #1776 #1777 #1778 #1779 #1780 #1781 #1782 #1783 #1784 #1785 #1786 #1787 #1788 #1789 #1790 #1791 #1792 #1793 #1794 #1795 #1796 #1797 #1798 #1799 2023-02-03 11:07:59Z [ 67.544299][ T1] .... node #14, CPUs: #1800 #1801 #1802 #1803 #1804 #1805 #1806 #1807 #1808 #1809 #1810 #1811 #1812 #1813 #1814 #1815 #1816 #1817 #1818 #1819 #1820 #1821 #1822 #1823 #1824 #1825 #1826 #1827 #1828 #1829 #1830 #1831 #1832 #1833 #1834 #1835 #1836 #1837 #1838 #1839 #1840 #1841 #1842 #1843 #1844 #1845 #1846 #1847 #1848 #1849 #1850 #1851 #1852 #1853 #1854 #1855 #1856 #1857 #1858 #1859 2023-02-03 11:08:02Z [ 70.084299][ T1] .... node #15, CPUs: #1860 #1861 #1862 #1863 #1864 #1865 #1866 #1867 #1868 #1869 #1870 #1871 #1872 #1873 #1874 #1875 #1876 #1877 #1878 #1879 #1880 #1881 #1882 #1883 #1884 #1885 #1886 #1887 #1888 #1889 #1890 #1891 #1892 #1893 #1894 #1895 #1896 #1897 #1898 #1899 #1900 #1901 #1902 #1903 #1904 #1905 #1906 #1907 #1908 #1909 #1910 #1911 #1912 #1913 #1914 #1915 #1916 #1917 #1918 #1919 2023-02-03 11:08:02Z [ 72.634158][ T1] smp: Brought up 16 nodes, 1920 CPUs 2023-02-03 11:08:02Z [ 72.640299][ T1] smpboot: Max logical packages: 16 2023-02-03 11:08:02Z [ 72.644444][ T1] smpboot: Total of 1920 processors activated (7275848.16 BogoMIPS) -------------------------------------------------------------------------- Thanks.
On Sun, 2023-02-05 at 13:17 -0600, Russ Anderson wrote: > > Gave the v6 patchset a try on a system with 1920 logocal cpus > (sixteen 60 core Sapphire Rapids sockets with Hyperthreadding > enabled). > > Without the patchset it took 71 seconds to start all the cpus. > With the v6 patchset it took 14 seconds to start all the cpus, > a reduction of 57 seconds. That is impressive. > > Full boot, to root login prompt, without patches takes 223 seconds. > This patchset reduces the full boot time by 57 seconds, a 25% > reduction. Nice; thanks for testing. Is that with just the "part1" patch series which has been posted, or also with the 'parallel part 2' still taking shape in the tree at https://git.infradead.org/users/dwmw2/linux.git/shortlog/refs/heads/parallel-6.2-rc6 I believe Usama said the second phase of parallelism didn't really help much in terms of overall timing? Confirming that *without* all the debug prints would be interesting. And we can look for what still *could* be made parallel.
On 06/02/2023 08:28, David Woodhouse wrote: > On Sun, 2023-02-05 at 13:17 -0600, Russ Anderson wrote: >> >> Gave the v6 patchset a try on a system with 1920 logocal cpus >> (sixteen 60 core Sapphire Rapids sockets with Hyperthreadding >> enabled). >> >> Without the patchset it took 71 seconds to start all the cpus. >> With the v6 patchset it took 14 seconds to start all the cpus, >> a reduction of 57 seconds. That is impressive. >> >> Full boot, to root login prompt, without patches takes 223 seconds. >> This patchset reduces the full boot time by 57 seconds, a 25% >> reduction. > > Nice; thanks for testing. > > Is that with just the "part1" patch series which has been posted, or > also with the 'parallel part 2' still taking shape in the tree at > https://git.infradead.org/users/dwmw2/linux.git/shortlog/refs/heads/parallel-6.2-rc6 > > I believe Usama said the second phase of parallelism didn't really help > much in terms of overall timing? Confirming that *without* all the > debug prints would be interesting. And we can look for what still > *could* be made parallel. I think it would be interesting to get the numbers for such a big machine for 3 cases: part1, part1+reuse timer calibration and part1+part2. Russ mentioned testing v6, so I guess the above numbers are for part1+reuse timer calibration. For my machine the smpboot times were: No patches: 700ms part 1:100ms part1+reuse timer calibration: 30ms part1+part2: 30ms Thanks, Usama