From patchwork Tue May 16 09:09:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: tip-bot2 for Thomas Gleixner X-Patchwork-Id: 94583 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp288106vqo; Tue, 16 May 2023 02:21:33 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4CEMOMsDc/pQi1ZF1MFZUDUdwFedGwwHTPetanSI6rai6TjGjKrhLR3Z5ZN/bPfgNlLn8H X-Received: by 2002:a05:6a00:2e1e:b0:64c:c5c0:6e01 with SMTP id fc30-20020a056a002e1e00b0064cc5c06e01mr1189340pfb.31.1684228893272; Tue, 16 May 2023 02:21:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684228893; cv=none; d=google.com; s=arc-20160816; b=YX4j8rfpBX4jYvNXtTqvovZj5KnJKNmZqT6g7yqNqvuIn9fQH4dhrCxTqFyT7SvygA Dvyyi3DA5mzuv4+nxm48hljNBLjCWQa8wCYxmB2tMmdNPsSd98AWALkVrolIOg+iLUY3 dIBxhR+d7VnBPuJCeLpRLxB9XlsYmFH/KLfHIv0SzGpRO9/3NYBPdo4w8DoohZaDve2e YL1HtPHIuZlajSqh6byNktSEN9WSI75Y7MMiYxO9Wc58w7v5RZFg+LgVSGW8q/DmpV6Y S/5rFpkRnX36RyQthQVt75Yzwf4wVthwN5K8TSerV8d7t5wgiS3mB6mGFLKJC5aQBB5D C+ng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:robot-unsubscribe :robot-id:message-id:mime-version:references:in-reply-to:cc:subject :to:reply-to:sender:from:dkim-signature:dkim-signature:date; bh=9rG/4uiCiu4KPZzymCMqQCDAE4nPud2YgKXQRygnnYA=; b=hWvtQPHCTfXBrwRZcodIS7D7Hz21rMUJzqhn/p7IpmQuuFSXt46stAv7SveKHFxeVw nJP1/C50pLNg0NUdhSh5BX9ykwY92zVBMQl4NVMMD6G7UftPC3UB83b0lZjJ0O6dyC0Z PKyFIBJOWPWEfTDNCcn4BmFIlwWe8iGRwVmZd2R3OmUD0Y1jjLw/18Lw5kvQZccs4/PU 3V5EYwMGcfh/FrTFTHPY7ACNHB43spk8MIq4AvgpSvFP/bJGz3h2SjcRLcgfYEkqC8iV 26EK6uENqCkH+xVi3aPgamUiTYTCtyniHZmciNrIWCKeSJXTeCyjXzroN8PhG8H4ofsY eoGA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=qlhSNdWg; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=oMTIVLLc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 12-20020a63010c000000b005132342a587si17482693pgb.610.2023.05.16.02.21.21; Tue, 16 May 2023 02:21:33 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=qlhSNdWg; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=oMTIVLLc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231888AbjEPJKY (ORCPT + 99 others); Tue, 16 May 2023 05:10:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45976 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231601AbjEPJJ7 (ORCPT ); Tue, 16 May 2023 05:09:59 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0D7CF2D53; Tue, 16 May 2023 02:09:56 -0700 (PDT) Date: Tue, 16 May 2023 09:09:54 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684228195; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9rG/4uiCiu4KPZzymCMqQCDAE4nPud2YgKXQRygnnYA=; b=qlhSNdWgAhJy6IOd+pNnB+Xsr6wuH4Fyg8bOeGL4hNgAOYEaAY2uHlM/7zgH98cpa6yasY GEXjsdLnz0FfSwyLpkJbVLrenC/oaB+srBioruXDrrPDSmjgGX2Qoi6w2nTaKSdxgpqV8E m8Wz9QYC0gjSug3Gth1J81j4NwQkhAQYr5NqhnELo+T9e18J0l7obJVCKkkFd2l9lTNoA5 +HrSNZysJkGjI+KoP4snV8oSw/TCfsq6uyPrLD2jPEbck6qX59j+5OtNttolAK9ZjGq4Uk nSoOZyFUImKSggi/jAbitTopD48s49ORwAsW7MhtUHHbZWkKQ5uxi4FfAv5CHQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684228195; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9rG/4uiCiu4KPZzymCMqQCDAE4nPud2YgKXQRygnnYA=; b=oMTIVLLcLyW6Cw035QcFAE3NeV6UwyQumWN7vVOB0pE1n4kTo2liQt7Jhk+RQ2+LQPdZDL 66hRos9O9IqV1ZAQ== From: "tip-bot2 for Thomas Gleixner" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: smp/core] cpu/hotplug: Allow "parallel" bringup up to CPUHP_BP_KICK_AP_STATE Cc: David Woodhouse , Thomas Gleixner , "Peter Zijlstra (Intel)" , Michael Kelley , Oleksandr Natalenko , Helge Deller , "Guilherme G. Piccoli" , x86@kernel.org, linux-kernel@vger.kernel.org In-Reply-To: <20230512205257.240231377@linutronix.de> References: <20230512205257.240231377@linutronix.de> MIME-Version: 1.0 Message-ID: <168422819471.404.16088544163447785024.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765725067295017782?= X-GMAIL-MSGID: =?utf-8?q?1766041996406951743?= The following commit has been merged into the smp/core branch of tip: Commit-ID: 18415f33e2ac4ab382cbca8b5ff82a9036b5bd49 Gitweb: https://git.kernel.org/tip/18415f33e2ac4ab382cbca8b5ff82a9036b5bd49 Author: Thomas Gleixner AuthorDate: Fri, 12 May 2023 23:07:50 +02:00 Committer: Peter Zijlstra CommitterDate: Mon, 15 May 2023 13:45:02 +02:00 cpu/hotplug: Allow "parallel" bringup up to CPUHP_BP_KICK_AP_STATE There is often significant latency in the early stages of CPU bringup, and time is wasted by waking each CPU (e.g. with SIPI/INIT/INIT on x86) and then waiting for it to respond before moving on to the next. Allow a platform to enable parallel setup which brings all to be onlined CPUs up to the CPUHP_BP_KICK_AP state. While this state advancement on the control CPU (BP) is single-threaded the important part is the last state CPUHP_BP_KICK_AP which wakes the to be onlined CPUs up. This allows the CPUs to run up to the first sychronization point cpuhp_ap_sync_alive() where they wait for the control CPU to release them one by one for the full onlining procedure. This parallelism depends on the CPU hotplug core sync mechanism which ensures that the parallel brought up CPUs wait for release before touching any state which would make the CPU visible to anything outside the hotplug control mechanism. To handle the SMT constraints of X86 correctly the bringup happens in two iterations when CONFIG_HOTPLUG_SMT is enabled. The control CPU brings up the primary SMT threads of each core first, which can load the microcode without the need to rendevouz with the thread siblings. Once that's completed it brings up the secondary SMT threads. Co-developed-by: David Woodhouse Signed-off-by: David Woodhouse Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Michael Kelley Tested-by: Oleksandr Natalenko Tested-by: Helge Deller # parisc Tested-by: Guilherme G. Piccoli # Steam Deck Link: https://lore.kernel.org/r/20230512205257.240231377@linutronix.de --- Documentation/admin-guide/kernel-parameters.txt | 6 +- arch/Kconfig | 4 +- include/linux/cpuhotplug.h | 1 +- kernel/cpu.c | 103 ++++++++++++++- 4 files changed, 109 insertions(+), 5 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index ccb91a2..79fb124 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -838,6 +838,12 @@ on every CPU online, such as boot, and resume from suspend. Default: 10000 + cpuhp.parallel= + [SMP] Enable/disable parallel bringup of secondary CPUs + Format: + Default is enabled if CONFIG_HOTPLUG_PARALLEL=y. Otherwise + the parameter has no effect. + crash_kexec_post_notifiers Run kdump after running panic-notifiers and dumping kmsg. This only for the users who doubt kdump always diff --git a/arch/Kconfig b/arch/Kconfig index d3015a6..64d7718 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -53,6 +53,10 @@ config HOTPLUG_SPLIT_STARTUP bool select HOTPLUG_CORE_SYNC_FULL +config HOTPLUG_PARALLEL + bool + select HOTPLUG_SPLIT_STARTUP + config GENERIC_ENTRY bool diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h index bc2d0a1..a5e414c 100644 --- a/include/linux/cpuhotplug.h +++ b/include/linux/cpuhotplug.h @@ -524,6 +524,7 @@ void cpuhp_ap_sync_alive(void); void arch_cpuhp_sync_state_poll(void); void arch_cpuhp_cleanup_kick_cpu(unsigned int cpu); int arch_cpuhp_kick_ap_alive(unsigned int cpu, struct task_struct *tidle); +bool arch_cpuhp_init_parallel_bringup(void); #ifdef CONFIG_HOTPLUG_CORE_SYNC_DEAD void cpuhp_ap_report_dead(void); diff --git a/kernel/cpu.c b/kernel/cpu.c index d2487aa..005f863 100644 --- a/kernel/cpu.c +++ b/kernel/cpu.c @@ -649,8 +649,23 @@ bool cpu_smt_possible(void) cpu_smt_control != CPU_SMT_NOT_SUPPORTED; } EXPORT_SYMBOL_GPL(cpu_smt_possible); + +static inline bool cpuhp_smt_aware(void) +{ + return topology_smt_supported(); +} + +static inline const struct cpumask *cpuhp_get_primary_thread_mask(void) +{ + return cpu_primary_thread_mask; +} #else static inline bool cpu_smt_allowed(unsigned int cpu) { return true; } +static inline bool cpuhp_smt_aware(void) { return false; } +static inline const struct cpumask *cpuhp_get_primary_thread_mask(void) +{ + return cpu_present_mask; +} #endif static inline enum cpuhp_state @@ -1747,18 +1762,96 @@ int bringup_hibernate_cpu(unsigned int sleep_cpu) return 0; } -void __init bringup_nonboot_cpus(unsigned int setup_max_cpus) +static void __init cpuhp_bringup_mask(const struct cpumask *mask, unsigned int ncpus, + enum cpuhp_state target) { unsigned int cpu; - for_each_present_cpu(cpu) { - if (num_online_cpus() >= setup_max_cpus) + for_each_cpu(cpu, mask) { + struct cpuhp_cpu_state *st = per_cpu_ptr(&cpuhp_state, cpu); + + if (!--ncpus) break; - if (!cpu_online(cpu)) - cpu_up(cpu, CPUHP_ONLINE); + + if (cpu_up(cpu, target) && can_rollback_cpu(st)) { + /* + * If this failed then cpu_up() might have only + * rolled back to CPUHP_BP_KICK_AP for the final + * online. Clean it up. NOOP if already rolled back. + */ + WARN_ON(cpuhp_invoke_callback_range(false, cpu, st, CPUHP_OFFLINE)); + } } } +#ifdef CONFIG_HOTPLUG_PARALLEL +static bool __cpuhp_parallel_bringup __ro_after_init = true; + +static int __init parallel_bringup_parse_param(char *arg) +{ + return kstrtobool(arg, &__cpuhp_parallel_bringup); +} +early_param("cpuhp.parallel", parallel_bringup_parse_param); + +/* + * On architectures which have enabled parallel bringup this invokes all BP + * prepare states for each of the to be onlined APs first. The last state + * sends the startup IPI to the APs. The APs proceed through the low level + * bringup code in parallel and then wait for the control CPU to release + * them one by one for the final onlining procedure. + * + * This avoids waiting for each AP to respond to the startup IPI in + * CPUHP_BRINGUP_CPU. + */ +static bool __init cpuhp_bringup_cpus_parallel(unsigned int ncpus) +{ + const struct cpumask *mask = cpu_present_mask; + + if (__cpuhp_parallel_bringup) + __cpuhp_parallel_bringup = arch_cpuhp_init_parallel_bringup(); + if (!__cpuhp_parallel_bringup) + return false; + + if (cpuhp_smt_aware()) { + const struct cpumask *pmask = cpuhp_get_primary_thread_mask(); + static struct cpumask tmp_mask __initdata; + + /* + * X86 requires to prevent that SMT siblings stopped while + * the primary thread does a microcode update for various + * reasons. Bring the primary threads up first. + */ + cpumask_and(&tmp_mask, mask, pmask); + cpuhp_bringup_mask(&tmp_mask, ncpus, CPUHP_BP_KICK_AP); + cpuhp_bringup_mask(&tmp_mask, ncpus, CPUHP_ONLINE); + /* Account for the online CPUs */ + ncpus -= num_online_cpus(); + if (!ncpus) + return true; + /* Create the mask for secondary CPUs */ + cpumask_andnot(&tmp_mask, mask, pmask); + mask = &tmp_mask; + } + + /* Bring the not-yet started CPUs up */ + cpuhp_bringup_mask(mask, ncpus, CPUHP_BP_KICK_AP); + cpuhp_bringup_mask(mask, ncpus, CPUHP_ONLINE); + return true; +} +#else +static inline bool cpuhp_bringup_cpus_parallel(unsigned int ncpus) { return false; } +#endif /* CONFIG_HOTPLUG_PARALLEL */ + +void __init bringup_nonboot_cpus(unsigned int setup_max_cpus) +{ + /* Try parallel bringup optimization if enabled */ + if (cpuhp_bringup_cpus_parallel(setup_max_cpus)) + return; + + /* Full per CPU serialized bringup */ + cpuhp_bringup_mask(cpu_present_mask, setup_max_cpus, CPUHP_ONLINE); +} + #ifdef CONFIG_PM_SLEEP_SMP static cpumask_var_t frozen_cpus;