From patchwork Thu Mar 16 22:21:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 70995 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp25797wrt; Thu, 16 Mar 2023 15:23:45 -0700 (PDT) X-Google-Smtp-Source: AK7set/oWIGe5dDiJQtOxSVjBPOp27/YXdDMk+ZMiZXGxgoXVGW4jd6xxhD89eknHivszvjUuPNa X-Received: by 2002:a17:90b:1c06:b0:23d:3a3f:950d with SMTP id oc6-20020a17090b1c0600b0023d3a3f950dmr5207756pjb.40.1679005425047; Thu, 16 Mar 2023 15:23:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679005425; cv=none; d=google.com; s=arc-20160816; b=bQCrkI+XNVBXWtWU5dLV1OxY2WwXLP69kErPrLvELSocQIothpN6LpfdA3FQRb3VJp BPKCIzz/+6ABkUQ/aI5mzUWJ5nx+u9IDk4B7ByVFWkOtBR8Z2nPEvxMd7R7rhJxAfCF8 uzu/MpP4TwOuo9vxAxF95BYwH4Qy7NGnwuy5siKjPy4ERXJkrp4FbYWwwBPIBE6EQ9jA Gdn6Kx/tXz2sW0I/HenRI6Gk6G3f5JysC1pbDdYDYPM5fb0MltZIMbIMI1j6embF97uN XT6wt1w86+l1elhbHyXFy0lptNgrWY/RwEn/x7FOVw/jSz38LMEJsm8b3PraK8SsKkq7 ilbw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=RcZv3zbGH7k+CBej9wIKrUlEuzG8XkHUgLqLEwBzkYQ=; b=ziQv+I9eBsMM2TS25epmXNbeUdyeHnwfoId4515gMdIB4OawsEwTRVeg3Hi/2dMhRE CeTvl55dOYcFsbZDF6WBhoHYk+TF6UM02Q/SVeARap8AtKk7JKVuVFvbR/Gi9cmssmZQ fFVX7yXn58+dV3LEVWs7HcftPVOeEarV07ySkzgI5KQsStDQBwJ2j5tN5WmuyxQV3dS3 /GTU6OaEFq//iWxuzGjnE5+VSNVSf3ExzpDL77O+nrHa1sC15jYPImI9JTu+HIZhaYSC AxwOU/a63CXMbvYn4a7HHL4KGSuTC2h6aaUD+WTdWgjDJyxRGgQ260P36RTHplQYICmb Eo6Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=a80sFCpT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h10-20020a17090a2eca00b0023d37ca1deesi322511pjs.70.2023.03.16.15.23.30; Thu, 16 Mar 2023 15:23:45 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=a80sFCpT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230310AbjCPWWk (ORCPT + 99 others); Thu, 16 Mar 2023 18:22:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60642 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230287AbjCPWWL (ORCPT ); Thu, 16 Mar 2023 18:22:11 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C6857B6D0A for ; Thu, 16 Mar 2023 15:21:23 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id r29so2851448wra.13 for ; Thu, 16 Mar 2023 15:21:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1679005281; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RcZv3zbGH7k+CBej9wIKrUlEuzG8XkHUgLqLEwBzkYQ=; b=a80sFCpTfQlIpXTjBsvv7ruVIpaS8bqFb+SlrcuHl5I392mri/jymyx8NdqbaTK26C YdzkusJuVGbbG+cerQZAoFtmPsAeG83fpCop2fSPB4j7+NfDKhqyk+UUL/xyRPXD59UL m/JTjGaANhJbkCdXrbqxuO3fcULccPrNcnEkGek1+2CXQxorClocGjcOGE8ByYYHCa97 HhZ2n2NqlKTG8F0FYunRXsj664X3UxvK7D/9mG68dxr1byoc1gYLOUFmoGFsb5FN/GWi +7lWXAqIY8D2vGoHqH8nVgsDFU32g88zJiNgPfmFXQFV2W5wivqgZrlZpo6nQhbIfU+e HfvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679005281; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RcZv3zbGH7k+CBej9wIKrUlEuzG8XkHUgLqLEwBzkYQ=; b=mEofbTxgjMIT5yWQCMoZjCto33lH4Bzg9LJ0xHWEmh25KE+YodDhxfEqlGWXzf4PEQ BqR7KGGeR/OFaVGEjnKkefltrafiKZ/+iOUdwsKt1uRWq85q820pIcAi90Fv4aeW4Mfr uetONcGluYvzz0oP6C02CU5tXg4un4oUEfcH6XxMRq8Mu4iZyMycxK/VSYRWR0qO5ak+ TMzYLjh+pQaY/6MVPlpEunWOEy5vOj4exRq3Q7oNEaBMb0wRkdV7lX+pdrA1jQzM5x5A lrt92oEoX65iPbawIfufWxWjap9ksrtYxyN5Ci1TpZHpH+S9w5BjleDCh7lFuDyUQg+Z Yu/Q== X-Gm-Message-State: AO0yUKX0ylv8JLai6ct6P7YeAyPEd3G7q7zyg4Tkp1mSEf70tC6tYFSw HUD0La1Sylv1ZcDvmZMICuURwg== X-Received: by 2002:a5d:63c3:0:b0:2cf:e868:f789 with SMTP id c3-20020a5d63c3000000b002cfe868f789mr5360294wrw.48.1679005281208; Thu, 16 Mar 2023 15:21:21 -0700 (PDT) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:4b87:78c3:3abe:7b0d]) by smtp.gmail.com with ESMTPSA id f9-20020adff989000000b002cea392f000sm439256wrr.69.2023.03.16.15.21.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 16 Mar 2023 15:21:20 -0700 (PDT) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, gpiccoli@igalia.com, David Woodhouse , Usama Arif Subject: [PATCH v15 10/12] x86/smpboot: Send INIT/SIPI/SIPI to secondary CPUs in parallel Date: Thu, 16 Mar 2023 22:21:07 +0000 Message-Id: <20230316222109.1940300-11-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230316222109.1940300-1-usama.arif@bytedance.com> References: <20230316222109.1940300-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760564792251455865?= X-GMAIL-MSGID: =?utf-8?q?1760564792251455865?= From: David Woodhouse When the APs can find their own APIC ID without assistance, perform the AP bringup in parallel. Register a CPUHP_BP_PARALLEL_DYN stage "x86/cpu:kick" which just calls do_boot_cpu() to deliver INIT/SIPI/SIPI to each AP in turn before the normal native_cpu_up() does the rest of the hand-holding. The APs will then take turns through the real mode code (which has its own bitlock for exclusion) until they make it to their own stack, then proceed through the first few lines of start_secondary() and execute these parts in parallel: start_secondary() -> cr4_init() -> (some 32-bit only stuff so not in the parallel cases) -> cpu_init_secondary() -> cpu_init_exception_handling() -> cpu_init() -> wait_for_master_cpu() At this point they wait for the BSP to set their bit in cpu_callout_mask (from do_wait_cpu_initialized()), and release them to continue through the rest of cpu_init() and beyond. This reduces the time taken for bringup on my 28-thread Haswell system from about 120ms to 80ms. On a socket 96-thread Skylake it takes the bringup time from 500ms to 100ms. There is more speedup to be had by doing the remaining parts in parallel too — especially notify_cpu_starting() in which the AP takes itself through all the stages from CPUHP_BRINGUP_CPU to CPUHP_ONLINE. But those require careful auditing to ensure they are reentrant, before we can go that far. Signed-off-by: David Woodhouse Signed-off-by: Usama Arif Tested-by: Paul E. McKenney Tested-by: Kim Phillips Tested-by: Oleksandr Natalenko Tested-by: Guilherme G. Piccoli --- arch/x86/kernel/smpboot.c | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index a9e48946fc89..69b56c597949 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -57,6 +57,7 @@ #include #include #include +#include #include #include @@ -992,7 +993,8 @@ static void announce_cpu(int cpu, int apicid) node_width = num_digits(num_possible_nodes()) + 1; /* + '#' */ if (cpu == 1) - printk(KERN_INFO "x86: Booting SMP configuration:\n"); + printk(KERN_INFO "x86: Booting SMP configuration in %s:\n", + do_parallel_bringup ? "parallel" : "series"); if (system_state < SYSTEM_RUNNING) { if (node != current_node) { @@ -1325,9 +1327,12 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) { int ret; - ret = do_cpu_up(cpu, tidle); - if (ret) - goto out; + /* If parallel AP bringup isn't enabled, perform the first steps now. */ + if (!do_parallel_bringup) { + ret = do_cpu_up(cpu, tidle); + if (ret) + goto out; + } ret = do_wait_cpu_initialized(cpu); if (ret) @@ -1347,6 +1352,12 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) return ret; } +/* Bringup step one: Send INIT/SIPI to the target AP */ +static int native_cpu_kick(unsigned int cpu) +{ + return do_cpu_up(cpu, idle_thread_get(cpu, true)); +} + /** * arch_disable_smp_support() - disables SMP support for x86 at runtime */ @@ -1515,6 +1526,8 @@ static bool prepare_parallel_bringup(void) smpboot_control = STARTUP_APICID_CPUID_01; } + cpuhp_setup_state_nocalls(CPUHP_BP_PARALLEL_DYN, "x86/cpu:kick", + native_cpu_kick, NULL); return true; }