From patchwork Thu Feb 9 15:41:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 55021 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp408142wrn; Thu, 9 Feb 2023 07:43:53 -0800 (PST) X-Google-Smtp-Source: AK7set+VlEEnxNm4wzHHi6Ne7geBy2CfIK3tlJMpXV7jQv6Pnrr8pI+iUJI8+NUYXziOLWHcqk6a X-Received: by 2002:a17:903:22cb:b0:19a:5958:15e7 with SMTP id y11-20020a17090322cb00b0019a595815e7mr2694726plg.15.1675957432709; Thu, 09 Feb 2023 07:43:52 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675957432; cv=none; d=google.com; s=arc-20160816; b=ih9EF4pE3yGnmCIKB9ImCe2R82Q6tLIvgAMbXmkY04OC8NF7/AditoqLSbMs/17sTz YTKmMMKjlitRxnkZ9LE9Te0ANR7xa2q5ZxJVOIW3Wu4WZrDK4grUs7FaCAF3dDmTQhQu gnJdelLzp1t9kkFVqjlcda+wx4sXFOoI390SIoHltXwmlgxw9M8FtX/k6lLvU5R3s/lJ H4qLeOVFBmhojMIRQFaZ10RxxMdmRNNdhRjr9hh5IZxhvuLCJnb9S3mQiKwSIzlSoxJQ iz7mA2DihAtyRJY1KTys9g8YYJm99/HHqGL5Kqro92wC8M35b4B7xZxnYexWif6cjQly 08zA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=yFpynNqCsjntY/I1AYfIdfaU4qF01G8hntPs8CGUGJk=; b=mKuxrM9ip7ARGk//wuNl5YbC10jAgdgjEplSQS4u6Gt0kBnmzRYD7mEdrKEgSTwkkt pB5D43/ZqWKLfev6I+ijbcMfqCgK242vgKkM1KNrjvItaWvwHCp1+Uapnl/HDU0TlyuS s2+jw629LrCzrS2ul2Q+f8juP/pmwkgWl+Oyude6usZGW9tcPwcCa8nsi6CC9fAshSym 4VLvZZYzDioXXL9kqTCwswPcMsoshIpy3ysi6h/2+bWU+4K/eHvgY41bnfWltz2ZnxJ3 GDz9ecalSjpqCb82oM0tOyqz2LBgHYbIGIxOR1HxZigpexQ22ZEdm7uKoCUZ5O98F8dj ttPg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=mLQQ5s0L; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id e7-20020a17090301c700b00197918434cbsi2242554plh.593.2023.02.09.07.43.40; Thu, 09 Feb 2023 07:43:52 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=mLQQ5s0L; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231438AbjBIPnI (ORCPT + 99 others); Thu, 9 Feb 2023 10:43:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55080 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231528AbjBIPmp (ORCPT ); Thu, 9 Feb 2023 10:42:45 -0500 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 98A956466B for ; Thu, 9 Feb 2023 07:42:08 -0800 (PST) Received: by mail-wm1-x32e.google.com with SMTP id hn2-20020a05600ca38200b003dc5cb96d46so4164957wmb.4 for ; Thu, 09 Feb 2023 07:42:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yFpynNqCsjntY/I1AYfIdfaU4qF01G8hntPs8CGUGJk=; b=mLQQ5s0LozepNKhlqL43TEbCQto9aNvqir80KAKt+74mLv5eUekNlH6J2dFUIkyPS5 vAVYAwc85IyHxsye1wVlo1qImm0NKXt6CGqpyR0LzbleWqYaB8BNmpcL3g8OT7hNR/Fg 8/wPpP97CewoN77qp2wdaFyTvB7BYAoO2nWwjguUYke1kOQjcN3++FrG7Ubjs4fr2J48 on4ZZ6mrNsv0yODpNt6sHb7BupKPk1v8eYQGw1YTs8rmnIr+1MgTBo3tpl/xx/VBack6 lSyz6KUqPfElaU2p3PZOee4e4Lkuxx7iPoQCI1iHUBeAS0GVSPKklWb11h5G9ptE0PGL CgkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yFpynNqCsjntY/I1AYfIdfaU4qF01G8hntPs8CGUGJk=; b=4f0OGWfJ0CT80IhUZVjvG5CR2DcKx1rKNfg/zP+a+xV9bQIkDl2uMtdN9h+Uj8nb9G lCfTGVzgxgl7sEHtACZ8zqeVh3UhsklKLxQCHXTvmvSIMqwmiJ/23UqSPoeYRJ9pwipo IapGGr48y1jNa3urqB0MBHp4bwW3pU1gn5ZrWyqCFsgueFvJN1UfZzpEyedP8Y2+OhGB aeRgnxwEf3SfWobaFrMFa3ZIVhtCTMYmUfvpKaSnXvpgV/8HUxcTle/pwa2oiUbfqffs pAAoHUPLEQtEhCFP54IHddQfdtSBMPnAw6M9AEBYYDFRssLYEq1dUbQ7bWqp88cspJaZ VNYw== X-Gm-Message-State: AO0yUKUdXgJsLawu2ygwHxXQbHmRfLugZpCC/M/C6g/mf3hLOszzc3aG +8rgh9dlqqhUk5oIP0GBN0hZwg== X-Received: by 2002:a05:600c:992:b0:3df:eedf:f35f with SMTP id w18-20020a05600c099200b003dfeedff35fmr4588500wmp.41.1675957327106; Thu, 09 Feb 2023 07:42:07 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:8009:2525:9580:8db2]) by smtp.gmail.com with ESMTPSA id y6-20020a05600c364600b003df7b40f99fsm5099754wmq.11.2023.02.09.07.42.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 09 Feb 2023 07:42:06 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com Cc: arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse , Usama Arif Subject: [PATCH v8 7/9] x86/smpboot: Send INIT/SIPI/SIPI to secondary CPUs in parallel Date: Thu, 9 Feb 2023 15:41:54 +0000 Message-Id: <20230209154156.266385-8-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230209154156.266385-1-usama.arif@bytedance.com> References: <20230209154156.266385-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757368740655090011?= X-GMAIL-MSGID: =?utf-8?q?1757368740655090011?= From: David Woodhouse When the APs can find their own APIC ID without assistance, perform the AP bringup in parallel. Register a CPUHP_BP_PARALLEL_DYN stage "x86/cpu:kick" which just calls do_boot_cpu() to deliver INIT/SIPI/SIPI to each AP in turn before the normal native_cpu_up() does the rest of the hand-holding. The APs will then take turns through the real mode code (which has its own bitlock for exclusion) until they make it to their own stack, then proceed through the first few lines of start_secondary() and execute these parts in parallel: start_secondary() -> cr4_init() -> (some 32-bit only stuff so not in the parallel cases) -> cpu_init_secondary() -> cpu_init_exception_handling() -> cpu_init() -> wait_for_master_cpu() At this point they wait for the BSP to set their bit in cpu_callout_mask (from do_wait_cpu_initialized()), and release them to continue through the rest of cpu_init() and beyond. This reduces the time taken for bringup on my 28-thread Haswell system from about 120ms to 80ms. On a socket 96-thread Skylake it takes the bringup time from 500ms to 100ms. There is more speedup to be had by doing the remaining parts in parallel too — especially notify_cpu_starting() in which the AP takes itself through all the stages from CPUHP_BRINGUP_CPU to CPUHP_ONLINE. But those require careful auditing to ensure they are reentrant, before we can go that far. [Usama Arif: fixed rebase conflict] Signed-off-by: David Woodhouse Signed-off-by: Usama Arif --- arch/x86/kernel/smpboot.c | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 50621793671d..df839264266b 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -57,6 +57,7 @@ #include #include #include +#include #include #include @@ -1325,9 +1326,12 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) { int ret; - ret = do_cpu_up(cpu, tidle); - if (ret) - return ret; + /* If parallel AP bringup isn't enabled, perform the first steps now. */ + if (!do_parallel_bringup) { + ret = do_cpu_up(cpu, tidle); + if (ret) + return ret; + } ret = do_wait_cpu_initialized(cpu); if (ret) @@ -1349,6 +1353,12 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) return ret; } +/* Bringup step one: Send INIT/SIPI to the target AP */ +static int native_cpu_kick(unsigned int cpu) +{ + return do_cpu_up(cpu, idle_thread_get(cpu)); +} + /** * arch_disable_smp_support() - disables SMP support for x86 at runtime */ @@ -1566,6 +1576,11 @@ void __init native_smp_prepare_cpus(unsigned int max_cpus) smpboot_control = STARTUP_SECONDARY | STARTUP_APICID_CPUID_01; } + if (do_parallel_bringup) { + cpuhp_setup_state_nocalls(CPUHP_BP_PARALLEL_DYN, "x86/cpu:kick", + native_cpu_kick, NULL); + } + snp_set_wakeup_secondary_cpu(); }