From patchwork Thu Feb 2 21:56:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 52192 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp494507wrn; Thu, 2 Feb 2023 13:59:51 -0800 (PST) X-Google-Smtp-Source: AK7set9C1mnV0g7ezEyemkkmDh0DFbkG9nIQRS5Ny0lBtVWV5+BI4a8U2HkMtc+FS0cyK0ZOAhuG X-Received: by 2002:a05:6402:3488:b0:494:fae3:c0df with SMTP id v8-20020a056402348800b00494fae3c0dfmr9269989edc.12.1675375191136; Thu, 02 Feb 2023 13:59:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675375191; cv=none; d=google.com; s=arc-20160816; b=I1eiqJAHUfjlnaWm0lKhP2pO2uY9FKIbJU3SjXey/s0f8LzZOR2X5cc4MoQ07nimYD QJJou44iVs/TAtw+E35mBU3lcrWaa3N4zFilrtrt4UX3cj8S3Yfq6+F5IPJPJFHYVLns c6QAB8gTsFiQxxajknBSKSmK7p/Y9Ci5k9DdLkD4AsFynXVztocHKQWN1h05W/zdbt78 6Wu7AToPmoNCpR6BpSt2VfQxuehbR9GpFnCGEbJeroIpZXeMqZE34KLvH01NLkjdvn1J vVif/2uZixy8UZC5cG/LJUQ6L6E8M50H+p6srgpU8FP9FO/B+HqWfejqZEQGJtCFxoYB gB/w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=1hr9lEiKrK81ARA2M0cI5BiENvgrY9Zt5BHSS6rpobc=; b=XoKWRD3i41gpN86qcwFyWxh1el2VKCJETv6B9SD4LqXDyD5vhtR6QyYEB9PW/0EmwJ CBgpzlqFfy+FULLmgt0h8/LPRksaqq2PfQHnHKj7o9E4OD+e1K8hdRc1y5xCb6CdE1mv qRA5Ou3zMrGHjWrBnoveFS6OqV4iz73ht9RXtuC5FUmsiYBZOdVX8rM9m6iHRKDmMy1v mYvNxexKcSvsuKERfjodDgg5r/pgAj9q+ntSKL3q/3FQk+Yx8cMqklBzoRVjgaj3MzSI uJcGTBwJEx5sGBovxy5CRMSJIr0KDw8PxEhH7N0l54JfwRBJ3wqSD5jFY0HomAjGH9uM 2w0w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=u2nq3Ej9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id m2-20020a056402430200b004a2343e90c4si828386edc.380.2023.02.02.13.59.26; Thu, 02 Feb 2023 13:59:51 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=u2nq3Ej9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233221AbjBBV5P (ORCPT + 99 others); Thu, 2 Feb 2023 16:57:15 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54788 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233136AbjBBV4y (ORCPT ); Thu, 2 Feb 2023 16:56:54 -0500 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9CCC6761F3 for ; Thu, 2 Feb 2023 13:56:39 -0800 (PST) Received: by mail-wr1-x436.google.com with SMTP id m14so2960456wrg.13 for ; Thu, 02 Feb 2023 13:56:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1hr9lEiKrK81ARA2M0cI5BiENvgrY9Zt5BHSS6rpobc=; b=u2nq3Ej9mftRp/AibuMcN8opF0/YxTfubYLNVe5VnD252/9mQ6PWzMHa5Wy+zTi5xp CIi3VzipOQ3cUPBGWAYmpslvBtttDPw0YSfsxwwOb8vHCqZjVNxF0VdoPIcVsCTVro9g AguJPXSsPtWzOCPAOcncRsin0t0J5YZ7QOZe8pW0y6LDl7mODkAk59UCuvCsC+MP5sqy q+DG2ZCDTfCUwcSBIxz9PsT6w0i57kcB3wXmbmXODr7XQ/LQ3stW00g1uJPRvgJxl05E tMfTmxwCQSL1qvW3qdy+19PzKFX9DfoV4xwkbCCP0MVT9BliY8KGfNVy4aBXuTJwfNjG w2Iw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1hr9lEiKrK81ARA2M0cI5BiENvgrY9Zt5BHSS6rpobc=; b=GZGAtTDpJaCedREcueohuyIvrwBDj+MCOKGzCVWSdMdrg7WsjixoWUArQDn6L0rrqH c5D9ExVX3Ps8SwIs/CJpSm3ZaryUvOe/w+ivMNuOOyjqxqxJ6I15A13NIL3e03PQ8Vbi BHnagMGtb9t+o+TtJJQE7B2SVvsOP2TSqKwNwMuz6Ycb2DOiKSRC2ELehJ/FZ4JwZi/q RQKSOioW0KBsSrYnAIlElXvDHcoTdLM1rnl2NydTN9+VSP19dGCqCiAelJqQaQ7liRgy f23ozNjnVOWpfD7HdI5fvmXbEwuW5CGE8IuODaoYVBHO+ysps8CE945mGILoZivNjQni JCWQ== X-Gm-Message-State: AO0yUKXa+KbbBlKK3CAthD1CilVyx3rp0b77k+BLZFs5KSZC3Rnfji+S JqfMj2NHRh6TB9zNMTxpi7ECsg== X-Received: by 2002:adf:fa84:0:b0:2bf:ae15:ac15 with SMTP id h4-20020adffa84000000b002bfae15ac15mr6639032wrr.62.1675374998131; Thu, 02 Feb 2023 13:56:38 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:98fe:e4ee:fc7e:cd71]) by smtp.gmail.com with ESMTPSA id e8-20020a5d6d08000000b00297dcfdc90fsm506078wrq.24.2023.02.02.13.56.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Feb 2023 13:56:37 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, arjan@linux.intel.com Cc: mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse , Usama Arif Subject: [PATCH v6 08/11] x86/smpboot: Send INIT/SIPI/SIPI to secondary CPUs in parallel Date: Thu, 2 Feb 2023 21:56:22 +0000 Message-Id: <20230202215625.3248306-9-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230202215625.3248306-1-usama.arif@bytedance.com> References: <20230202215625.3248306-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756758216793564357?= X-GMAIL-MSGID: =?utf-8?q?1756758216793564357?= From: David Woodhouse When the APs can find their own APIC ID without assistance, we can do the AP bringup in parallel. Register a CPUHP_BP_PARALLEL_DYN stage "x86/cpu:kick" which just calls do_boot_cpu() to deliver INIT/SIPI/SIPI to each AP in turn before the normal native_cpu_up() does the rest of the hand-holding. The APs will then take turns through the real mode code (which has its own bitlock for exclusion) until they make it to their own stack, then proceed through the first few lines of start_secondary() and execute these parts in parallel: start_secondary() -> cr4_init() -> (some 32-bit only stuff so not in the parallel cases) -> cpu_init_secondary() -> cpu_init_exception_handling() -> cpu_init() -> wait_for_master_cpu() At this point they wait for the BSP to set their bit in cpu_callout_mask (from do_wait_cpu_initialized()), and release them to continue through the rest of cpu_init() and beyond. This reduces the time taken for bringup on my 28-thread Haswell system from about 120ms to 80ms. On a socket 96-thread Skylake it takes the bringup time from 500ms to 100ms. There is more speedup to be had by doing the remaining parts in parallel too — especially notify_cpu_starting() in which the AP takes itself through all the stages from CPUHP_BRINGUP_CPU to CPUHP_ONLINE. But those require careful auditing to ensure they are reentrant, before we can go that far. [Usama Arif: fixed rebase conflict] Signed-off-by: David Woodhouse Signed-off-by: Usama Arif Signed-off-by: Paul E. McKenney --- arch/x86/kernel/smpboot.c | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 656897b055f5..77c509e95571 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -57,6 +57,7 @@ #include #include #include +#include #include #include @@ -1330,9 +1331,12 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) { int ret; - ret = do_cpu_up(cpu, tidle); - if (ret) - return ret; + /* If parallel AP bringup isn't enabled, perform the first steps now. */ + if (!do_parallel_bringup) { + ret = do_cpu_up(cpu, tidle); + if (ret) + return ret; + } ret = do_wait_cpu_initialized(cpu); if (ret) @@ -1354,6 +1358,12 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) return ret; } +/* Bringup step one: Send INIT/SIPI to the target AP */ +static int native_cpu_kick(unsigned int cpu) +{ + return do_cpu_up(cpu, idle_thread_get(cpu)); +} + /** * arch_disable_smp_support() - disables SMP support for x86 at runtime */ @@ -1550,6 +1560,10 @@ void __init native_smp_prepare_cpus(unsigned int max_cpus) do_parallel_bringup = false; } + if (do_parallel_bringup) + cpuhp_setup_state_nocalls(CPUHP_BP_PARALLEL_DYN, "x86/cpu:kick", + native_cpu_kick, NULL); + snp_set_wakeup_secondary_cpu(); }