From patchwork Wed Feb 15 14:54:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 57590 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp236301wrn; Wed, 15 Feb 2023 06:59:25 -0800 (PST) X-Google-Smtp-Source: AK7set/hQ0Cl6GAAS7f7FMWX+9Kz/zp4nKGjERnxipjlRncX0WbsaDtb0RIjhN0uigdXUM5G3RGR X-Received: by 2002:a17:906:af92:b0:878:545b:e540 with SMTP id mj18-20020a170906af9200b00878545be540mr2415730ejb.51.1676473165410; Wed, 15 Feb 2023 06:59:25 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676473165; cv=none; d=google.com; s=arc-20160816; b=rO4LzqlCjNHTTsDJGZbE48iz6t22IaP+FKwkWo9bOhy6uEqFzDgV66+GnBhqVC/6vQ mwGkoIiN3QC4w4DiE7SBRgwUczvwGdPFiJQPet8iScrTd/f+c2VMqZ8zrUawPikTveDF XUzHA8Sw44gu1S81xMU7KVGS40QGccxIp8Jtv8+D/mCC3/ts4EJtsC16eTDUx4eXeuwm vzYZNmY8hf645yyRrAu77PKsEnnvajA01HtHt/tPe+5DgapA1cGMeFepkp4KJJQZaIUw 6LDRlH98rekY8uaeONZS8bcK8dBHquYIdbSyLWUaCQs3dZSAOwsYeZco+WYONm1CCD/J q2vQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=JJkpGg1hd8P6OX4RPHIkrzOMjPz7lmvjqBeU21FPpuA=; b=OguAVJBhrQmRxh933pqeyI4xJ011GkYTfO+g2aIIGPJ1QuB0D/7Rj8gfogGynQw/Rw Cm8yBIP01yBvzND/LJd3weY2kGgVQgpH8VrD1jX61L9iBSAAaZR30Kn08kSh00Pi7DhN LHplggyKmzomQTezX/vNrZv0/0UXe5XhDYN1UpMKOUJrIC8rv7PCd9h+QhXliefUMAiF paeR66JRYvAq5Sd85TJTLwzy3do6kiwArMw75ctUp23E4UEKKTXyPmG2al2AQwzYUTzi dvjvEqMmNQ7yxqf3N8HsmuXLyjMDS1yoii2BGVkLioq8sGRql45bXaI1hDbEnJcPEtKm LsIA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=tcdHg6gL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ui28-20020a170907c91c00b008688996d6bdsi19674246ejc.560.2023.02.15.06.59.01; Wed, 15 Feb 2023 06:59:25 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=tcdHg6gL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230058AbjBOOz3 (ORCPT + 99 others); Wed, 15 Feb 2023 09:55:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51026 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230024AbjBOOzN (ORCPT ); Wed, 15 Feb 2023 09:55:13 -0500 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CF6A23A0BE for ; Wed, 15 Feb 2023 06:54:36 -0800 (PST) Received: by mail-wr1-x42f.google.com with SMTP id by3so18116396wrb.10 for ; Wed, 15 Feb 2023 06:54:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JJkpGg1hd8P6OX4RPHIkrzOMjPz7lmvjqBeU21FPpuA=; b=tcdHg6gL/+kmrDf6TMBwvnFfIcz101nGGz3Inam8TJELR9NPia/44bKe58cCe0wion KHhYTtM5JQHVGWkPIJQvS0Pg801b07Sz0tqEwOWT7VkqDg7FrNYl1ho3vnyTmQJyfBwd AKNb4Qbx/3QPWVMJ+ealo/onIWpVGrNLelRhBmyq9YVt99ApYhJ+jmpXsMkCMwQCvTUR XcHyB3YqBuIx5oALw8ygnOcM4MeZHUhQmNkAysqnAOo8sCMHwQeVvHZR0R+bpWuLe5sq Cr7eqqqA6NjXSDEX5DbcmXdJ5jFBjeC7+t5h+aIFic37yrx73YXF1y5DxCQ8rVp7qUF7 4Pcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JJkpGg1hd8P6OX4RPHIkrzOMjPz7lmvjqBeU21FPpuA=; b=muVZ7rVbTYqycEFWFLy/9pepHHPW1llnfUO9Boi6LwA2HzRA1rAkrC6d79utoh1gHd r7T8Bf7NOHnmw2S7zw8uKD8x6Vk0jdPtMeRLBmqfHpEZoW6F4s/sVJHWM3wo5xzot9TL 5yXf1W3XUBqVlqt5wKTl3hxAxOvlvWuB7iJW4Qjdj+oA+cnFSKrIe1bmkziX4ixttJvm Fzps9I3sRSAYPsP6oR3UHBRPWbBE0qmXPkmwODOeRD4cDEZ+VQs+ICWcXhjrkGgGhqbG gvzUeHi/E6xkUum5saw8ZBlTHR/xY6rw0CRXohXeStt31kgPSCCFp8PQ/cXl/Gnlljya kFHQ== X-Gm-Message-State: AO0yUKU6NthlQuJQtoCuxNCgtqGlwo9CPGSzZYsv7Ei3j/BS2CpnOkaq +sbAvxW8NWnEtaLhqLjIRRjquA== X-Received: by 2002:a5d:68c1:0:b0:2c5:58f5:3c40 with SMTP id p1-20020a5d68c1000000b002c558f53c40mr1698253wrw.47.1676472875295; Wed, 15 Feb 2023 06:54:35 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:8487:6a9a:3a67:11aa]) by smtp.gmail.com with ESMTPSA id t13-20020adfe44d000000b002c557f82e27sm8495508wrm.99.2023.02.15.06.54.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Feb 2023 06:54:34 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com Cc: arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse , Usama Arif Subject: [PATCH v9 7/8] x86/smpboot: Send INIT/SIPI/SIPI to secondary CPUs in parallel Date: Wed, 15 Feb 2023 14:54:24 +0000 Message-Id: <20230215145425.420125-8-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230215145425.420125-1-usama.arif@bytedance.com> References: <20230215145425.420125-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757909526152959931?= X-GMAIL-MSGID: =?utf-8?q?1757909526152959931?= From: David Woodhouse When the APs can find their own APIC ID without assistance, perform the AP bringup in parallel. Register a CPUHP_BP_PARALLEL_DYN stage "x86/cpu:kick" which just calls do_boot_cpu() to deliver INIT/SIPI/SIPI to each AP in turn before the normal native_cpu_up() does the rest of the hand-holding. The APs will then take turns through the real mode code (which has its own bitlock for exclusion) until they make it to their own stack, then proceed through the first few lines of start_secondary() and execute these parts in parallel: start_secondary() -> cr4_init() -> (some 32-bit only stuff so not in the parallel cases) -> cpu_init_secondary() -> cpu_init_exception_handling() -> cpu_init() -> wait_for_master_cpu() At this point they wait for the BSP to set their bit in cpu_callout_mask (from do_wait_cpu_initialized()), and release them to continue through the rest of cpu_init() and beyond. This reduces the time taken for bringup on my 28-thread Haswell system from about 120ms to 80ms. On a socket 96-thread Skylake it takes the bringup time from 500ms to 100ms. There is more speedup to be had by doing the remaining parts in parallel too — especially notify_cpu_starting() in which the AP takes itself through all the stages from CPUHP_BRINGUP_CPU to CPUHP_ONLINE. But those require careful auditing to ensure they are reentrant, before we can go that far. [Usama Arif: fixed rebase conflict] Signed-off-by: David Woodhouse Signed-off-by: Usama Arif --- arch/x86/kernel/smpboot.c | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 49d6563e4c23..4cea3a0ff503 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -57,6 +57,7 @@ #include #include #include +#include #include #include @@ -1325,9 +1326,12 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) { int ret; - ret = do_cpu_up(cpu, tidle); - if (ret) - return ret; + /* If parallel AP bringup isn't enabled, perform the first steps now. */ + if (!do_parallel_bringup) { + ret = do_cpu_up(cpu, tidle); + if (ret) + return ret; + } ret = do_wait_cpu_initialized(cpu); if (ret) @@ -1349,6 +1353,12 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) return ret; } +/* Bringup step one: Send INIT/SIPI to the target AP */ +static int native_cpu_kick(unsigned int cpu) +{ + return do_cpu_up(cpu, idle_thread_get(cpu)); +} + /** * arch_disable_smp_support() - disables SMP support for x86 at runtime */ @@ -1566,6 +1576,11 @@ void __init native_smp_prepare_cpus(unsigned int max_cpus) smpboot_control = STARTUP_SECONDARY | STARTUP_APICID_CPUID_01; } + if (do_parallel_bringup) { + cpuhp_setup_state_nocalls(CPUHP_BP_PARALLEL_DYN, "x86/cpu:kick", + native_cpu_kick, NULL); + } + snp_set_wakeup_secondary_cpu(); }