From patchwork Sun Feb 26 11:08:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 61594 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp1897985wrd; Sun, 26 Feb 2023 03:09:43 -0800 (PST) X-Google-Smtp-Source: AK7set/yvLOkF/sbCfuunHRmbYgdSedw8lCeJ5L/RdObOYgLFEVL4sAQd5QtwCMI9w7+C2pcbAbs X-Received: by 2002:a05:6a20:3c93:b0:cd:345e:5b10 with SMTP id b19-20020a056a203c9300b000cd345e5b10mr637642pzj.5.1677409783153; Sun, 26 Feb 2023 03:09:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677409783; cv=none; d=google.com; s=arc-20160816; b=Smz8YNbRXFSQv/a2usk7nM2WIQt6B+pT3IE75ot5NlcimPXvkudS82xQQhjyHyd375 +O8MKeeZXBLv47D/Wyu5e89dR3Qs1U1Madl0zjOrfi0E6En6uM51I5yOhOzToXdoUWwt raans0Kw4ILJphulfMKmA1Z8v6iUAqyPQJA7gk9uRJCGHlcGm4SoNZm8HGPCOmuG8+vo SNKfkOaVoOs2dUV2LoQrqaJFHb1Tlw5K8mrnsyXRSmhfGWI+dFMlR9J2cwREozw6VJno Sp7Gz33A/wQOFNwB92yqQeMFLgBpxPomfj5OJmwbQi7egXZHziSkRjSCy7RE2/cdLXi3 plLQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Rv4YBkExnEbzjmvRGSVdLlKzdSs1hoLH3+gZkX3hIJM=; b=AePePLY21P4eC2oBk3JH+qDJ3JTO1/auxj1h7/SqAQcO9HhHJ07kMGIP7IUTNlDFk6 NtB+kWyZY0Ii4hhx1Zass7s+6giuq6hob8FoeMHu0ykxqJDFQjTfDxIJbw4d+lC1PiQv S4h2wEigzT1bNclQGCPVGfSDN2NXPhC+JDOvHyVTzLQt2nNXdwzlidymF3DXPdURZ1t5 0KkI22+4okMIktzD7ZT9FejbjM+qEjJlwQlO1HkobwmNMD/NIb4eEub88ezS8dHAuUXG TQs6nXSucJ4sCkPJQ6B7vUNUrN0bPgjMxY1LO5ptNpOIWPd2PVQEsUcPlyrsq4yQWX4r AnHg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=DJ0qn754; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 65-20020a630044000000b00502f60c6809si4066286pga.358.2023.02.26.03.09.29; Sun, 26 Feb 2023 03:09:43 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=DJ0qn754; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229696AbjBZLIo (ORCPT + 99 others); Sun, 26 Feb 2023 06:08:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36858 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229647AbjBZLIi (ORCPT ); Sun, 26 Feb 2023 06:08:38 -0500 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 32E62EC4F for ; Sun, 26 Feb 2023 03:08:15 -0800 (PST) Received: by mail-wm1-x32a.google.com with SMTP id az36so2545123wmb.1 for ; Sun, 26 Feb 2023 03:08:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Rv4YBkExnEbzjmvRGSVdLlKzdSs1hoLH3+gZkX3hIJM=; b=DJ0qn754KoOY7lszedXqM0cvktd4MIlEDstLHXsvijRzw7VZtckVr4f+/5tRRSmKHU gdDRywi3/sCcc7cHk3Z1pMDmtOJBx42sx44E2WV0E5kwwaYFh0qPiytCfjob5b6Whm6M JoD/FuFOZZRjwBfN/BXF6UWeFn4h1MBx/LQRtZGy6q9qXLvTqOP1wlhSDMW34m8nEXIL 5j4L2Hm+EO5Z2aUvxXx4NkNje0MQfjveISORHATz2z5n4bl/Cjjio/sv53EwV2Lz9yR4 ZLxVoK7Ra+QQNJID4t6OiAb0CvXGuzbUjPqd8warO8sN+vc5QX5e09GyuhHyU4um/Lo4 3BHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Rv4YBkExnEbzjmvRGSVdLlKzdSs1hoLH3+gZkX3hIJM=; b=Acmn+U2eHhWl3G4Cj3yFs1hEKt0y/IHDQs4zD1hdmFULW+dUoxCnKqIoqcsSD87Vfx bHR8FKp1iN0A6k6rGUjF9oW+0wy6JsYTBI0Y78+2xjAfWZfByvTKccqEzHqopyEuUPfg b/NownkzOi8wM20Mi2iG5Wl1ADYA5SkFMn8KVnNbL5X9ENysS0Um2lJELFwJu8zGwLyR fGgwRdTUF/vzMt9q32g4MsPrDJHEOpkj0N8bJUZdaJBpOsXPYXvfV7v16/qS2BgjKk+U DrIhdIFP9kztZ33cq+UU2C8Dpyzxsl5B8aCl/RmsQh2Zx3igi4mXsOVhoVxl2BYQqmo2 lSiw== X-Gm-Message-State: AO0yUKWxJUbeLuvYphb835dz0vf/PDw3KMCJbjWK/+61ke6QcwCJ0QR2 NnaH9jQV31oGNwywfnDuCwKSmw== X-Received: by 2002:a05:600c:4da3:b0:3eb:2da4:f304 with SMTP id v35-20020a05600c4da300b003eb2da4f304mr5025546wmp.17.1677409693572; Sun, 26 Feb 2023 03:08:13 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:df7b:4668:3e23:d0c9]) by smtp.gmail.com with ESMTPSA id v22-20020a1cf716000000b003e1fee8baacsm9157318wmh.25.2023.02.26.03.08.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Feb 2023 03:08:13 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse , Usama Arif Subject: [PATCH v12 10/11] x86/smpboot: Send INIT/SIPI/SIPI to secondary CPUs in parallel Date: Sun, 26 Feb 2023 11:08:01 +0000 Message-Id: <20230226110802.103134-11-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230226110802.103134-1-usama.arif@bytedance.com> References: <20230226110802.103134-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758891640507820420?= X-GMAIL-MSGID: =?utf-8?q?1758891640507820420?= From: David Woodhouse When the APs can find their own APIC ID without assistance, perform the AP bringup in parallel. Register a CPUHP_BP_PARALLEL_DYN stage "x86/cpu:kick" which just calls do_boot_cpu() to deliver INIT/SIPI/SIPI to each AP in turn before the normal native_cpu_up() does the rest of the hand-holding. The APs will then take turns through the real mode code (which has its own bitlock for exclusion) until they make it to their own stack, then proceed through the first few lines of start_secondary() and execute these parts in parallel: start_secondary() -> cr4_init() -> (some 32-bit only stuff so not in the parallel cases) -> cpu_init_secondary() -> cpu_init_exception_handling() -> cpu_init() -> wait_for_master_cpu() At this point they wait for the BSP to set their bit in cpu_callout_mask (from do_wait_cpu_initialized()), and release them to continue through the rest of cpu_init() and beyond. This reduces the time taken for bringup on my 28-thread Haswell system from about 120ms to 80ms. On a socket 96-thread Skylake it takes the bringup time from 500ms to 100ms. There is more speedup to be had by doing the remaining parts in parallel too — especially notify_cpu_starting() in which the AP takes itself through all the stages from CPUHP_BRINGUP_CPU to CPUHP_ONLINE. But those require careful auditing to ensure they are reentrant, before we can go that far. Signed-off-by: David Woodhouse Signed-off-by: Usama Arif Tested-by: Paul E. McKenney Tested-by: Kim Phillips Tested-by: Oleksandr Natalenko --- arch/x86/kernel/smpboot.c | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 19b9b89b7458..711573cd9b87 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -57,6 +57,7 @@ #include #include #include +#include #include #include @@ -1325,9 +1326,12 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) { int ret; - ret = do_cpu_up(cpu, tidle); - if (ret) - return ret; + /* If parallel AP bringup isn't enabled, perform the first steps now. */ + if (!do_parallel_bringup) { + ret = do_cpu_up(cpu, tidle); + if (ret) + return ret; + } ret = do_wait_cpu_initialized(cpu); if (ret) @@ -1349,6 +1353,12 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) return ret; } +/* Bringup step one: Send INIT/SIPI to the target AP */ +static int native_cpu_kick(unsigned int cpu) +{ + return do_cpu_up(cpu, idle_thread_get(cpu)); +} + /** * arch_disable_smp_support() - disables SMP support for x86 at runtime */ @@ -1566,6 +1576,11 @@ void __init native_smp_prepare_cpus(unsigned int max_cpus) smpboot_control = STARTUP_APICID_CPUID_01; } + if (do_parallel_bringup) { + cpuhp_setup_state_nocalls(CPUHP_BP_PARALLEL_DYN, "x86/cpu:kick", + native_cpu_kick, NULL); + } + snp_set_wakeup_secondary_cpu(); }