From patchwork Wed Feb 1 20:43:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 51560 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp502727wrn; Wed, 1 Feb 2023 12:47:02 -0800 (PST) X-Google-Smtp-Source: AK7set/sFxeKHpO6erAz0XX/iDmVCEYy3VuWAfdmfWTO7Z8fl3Ygtp+BLOxRx20n5WMYSvvnYsh0 X-Received: by 2002:a05:6a00:14d0:b0:593:9620:360 with SMTP id w16-20020a056a0014d000b0059396200360mr3910863pfu.7.1675284422281; Wed, 01 Feb 2023 12:47:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675284422; cv=none; d=google.com; s=arc-20160816; b=axSjCQejQnCKq0W9AXZh25WcTefJCPOJojuiEPQfBU+sB4M2PLAMMciwlWyWYqVq+j NiFwkxIe7bsGi23NtVgFTFNqfJUA1yGMRbKDRwkCo83o4XpB3u7C6vdlTVrLLP1OVoCI 6Mf1aUO70SjA0B+8kxWdHFrPNiFr6SVLobig1AeOCRl6eqK5ouMOIUh4+oWsDQe+O8hy h1uL35PNWVPW9db+T4wPNcvFuRvv2pH3RS+b2GrQP0CS5XdG5f8k6aAfI3BbE3yYu3UG Le91iEoRvEFSu8+8l1ljpgE8ZkK0o3b7s5nSd0i8A9KEJPyXQ6OUaVzA5jyNp2YLf3O3 6a2g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=CUSGXtK7n6/6YUwTEjKlWX96IORvZOTW/bJOX05E+p0=; b=XWWg72noLsUn6SFsrn8SSJED5B1KNlB3+y0PC281QynvqgG+2w8G21BAPC5ySBxBUF PL+Yzb0ABXC1JeqqcY10b3cN0wASUACFWb+YCt6JTLbcrQ2aXFXyNCcctj6abDP6aq4G 6aD8z6DfMopzBK1hHmSFx4loW7Q9zSGLYesmAReZ7cjLGGt7EtN7w8xPEel3via6RwET 7pFexDsqwcNkMhBzOhS2ECrRR6W7+8B0AKzrCKmDhf6o7tKsz+DMYSI1feUAB0SeCX3A 3hJrFxMKBlL9RiE/yDnRz+znz+N4X6RpuA7inQmJQKXNSrMYGy25tcTIAT+VErxSeeIA do/w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=SNi6efb2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f12-20020aa79d8c000000b00593b561b6c4si11218450pfq.149.2023.02.01.12.46.49; Wed, 01 Feb 2023 12:47:02 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=SNi6efb2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231691AbjBAUqG (ORCPT + 99 others); Wed, 1 Feb 2023 15:46:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53640 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231215AbjBAUpc (ORCPT ); Wed, 1 Feb 2023 15:45:32 -0500 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4A60574C37 for ; Wed, 1 Feb 2023 12:44:53 -0800 (PST) Received: by mail-wr1-x433.google.com with SMTP id q5so18541950wrv.0 for ; Wed, 01 Feb 2023 12:44:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CUSGXtK7n6/6YUwTEjKlWX96IORvZOTW/bJOX05E+p0=; b=SNi6efb2wXvpa5dRKNfFbHTbohVHxgaFa7TL8cN/uJD2cihgyVfu7V2/hPuG0yfqgv mMqL3dS0ei9J3FfAc9W2vvHBzfK8QBktYWQVbNaR33QRy0t+QLqnSTJLOTCZU/V4qqud 8vVwf5mMFYWnJRLLCtLvdWCfJAFpD3Vv9/cCURR5mbYosPiG6xsPB1yX9ECX2e3UnPLZ MXa8HjoDiLR9hDzxaLIgJqpL6J5Veos5J79U2yra/IlF+3Lz1aeYO4lif1wd0hMAm6IS 5Ny8+mAhu2A4nVW6yF80HpX8bOvqfC3fnsVi0IpNGFS3SJeYm9Me7jC/cijNMEKw94vo Fv6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CUSGXtK7n6/6YUwTEjKlWX96IORvZOTW/bJOX05E+p0=; b=cLI9gIp4REqOvC7lpQvAASTWPH3VzKi6dY6HerdOjjPxWKEV/IgNMDwSsaJ5akscAq QAymZ7s8/9KC/NLFBihKrqJKI4Srx0hbMswIt0lAjTJx9IYdy45paCwyOqLVwmdsCoqD 7U7xl0rxqomVQNqDkJ3rDA5eqWm/lnd2k3STn6MhvAEyohN16FRSuiOXUg0fAb5Jimpy bRXH1oXCseUMvDiQ+rEQPO1YvCyuASZIuLeQZ0UpFLKFIaBItQNfMFiTY+R0J3JX6Ye/ rvXa6AHKb9yaXJWot9TFXXkKWwLru1MSohBT72UrzOX2xgZCRhrEg7Xy5PTlgJyiMgyW YcDA== X-Gm-Message-State: AO0yUKXo07kjkvAkZEHXQG4jve3UJm5cFcxQyAHRPkoLgNWGEw4/01Xg 1SXKnSm7u8828xnCLcpOk4le6g== X-Received: by 2002:a05:6000:16c4:b0:2c3:be89:7c37 with SMTP id h4-20020a05600016c400b002c3be897c37mr129949wrf.26.1675284291708; Wed, 01 Feb 2023 12:44:51 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:7611:c340:3d8d:d46c]) by smtp.gmail.com with ESMTPSA id n15-20020a5d598f000000b002bdff778d87sm19993584wri.34.2023.02.01.12.44.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Feb 2023 12:44:51 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de Cc: mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse , Usama Arif Subject: [PATCH 5/9] x86/smpboot: Split up native_cpu_up into separate phases and document them Date: Wed, 1 Feb 2023 20:43:34 +0000 Message-Id: <20230201204338.1337562-6-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230201204338.1337562-1-usama.arif@bytedance.com> References: <20230201204338.1337562-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756663038404648725?= X-GMAIL-MSGID: =?utf-8?q?1756663038404648725?= From: David Woodhouse There are four logical parts to what native_cpu_up() does on the BSP (or on the controlling CPU for a later hotplug). First it actually wakes the AP by sending the INIT/SIPI/SIPI sequence. Second, it waits for the AP to make it as far as wait_for_master_cpu() which sets that CPU's bit in cpu_initialized_mask, then sets the bit in cpu_callout_mask to let the AP proceed through cpu_init(). Then, it waits for the AP to finish cpu_init() and get as far as the smp_callin() call, which sets that CPU's bit in cpu_callin_mask. Finally, it does the TSC synchronization and waits for the AP to actually mark itself online in cpu_online_mask. This commit should have no behavioural change, but merely splits those phases out into separate functions so that future commits can make them happen in parallel for all APs. And adds some comments around them on both the BSP and AP code paths. [Usama Arif: fixed rebase conflict] Signed-off-by: David Woodhouse Signed-off-by: Usama Arif --- arch/x86/kernel/smpboot.c | 182 +++++++++++++++++++++++++++----------- 1 file changed, 128 insertions(+), 54 deletions(-) diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 2d5014752dc4..98604a3953e6 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -205,6 +205,10 @@ static void smp_callin(void) wmb(); + /* + * This runs the AP through all the cpuhp states to its target + * state (CPUHP_ONLINE in the case of serial bringup). + */ notify_cpu_starting(cpuid); /* @@ -232,17 +236,33 @@ static void notrace start_secondary(void *unused) load_cr3(swapper_pg_dir); __flush_tlb_all(); #endif + /* + * Sync point with do_wait_cpu_initialized(). On boot, all secondary + * CPUs reach this stage after receiving INIT/SIPI from do_cpu_up() + * in the x86/cpu:kick cpuhp stage. At the start of cpu_init() they + * will wait for do_wait_cpu_initialized() to set their bit in + * smp_callout_mask to release them. + */ cpu_init_secondary(); rcu_cpu_starting(raw_smp_processor_id()); x86_cpuinit.early_percpu_clock_init(); + + /* + * Sync point with do_wait_cpu_callin(). The AP doesn't wait here + * but just sets the bit to let the controlling CPU (BSP) know that + * it's got this far. + */ smp_callin(); enable_start_cpu0 = 0; /* otherwise gcc will move up smp_processor_id before the cpu_init */ barrier(); + /* - * Check TSC synchronization with the boot CPU: + * Check TSC synchronization with the boot CPU (or whichever CPU + * is controlling the bringup). It will do its part of this from + * do_wait_cpu_online(), making it an implicit sync point. */ check_tsc_sync_target(); @@ -255,6 +275,7 @@ static void notrace start_secondary(void *unused) * half valid vector space. */ lock_vector_lock(); + /* Sync point with do_wait_cpu_online() */ set_cpu_online(smp_processor_id(), true); lapic_online(); unlock_vector_lock(); @@ -1084,7 +1105,6 @@ static int do_boot_cpu(int apicid, int cpu, struct task_struct *idle, unsigned long start_ip = real_mode_header->trampoline_start; unsigned long boot_error = 0; - unsigned long timeout; #ifdef CONFIG_X86_64 /* If 64-bit wakeup method exists, use the 64-bit mode trampoline IP */ @@ -1145,55 +1165,94 @@ static int do_boot_cpu(int apicid, int cpu, struct task_struct *idle, boot_error = wakeup_cpu_via_init_nmi(cpu, start_ip, apicid, cpu0_nmi_registered); - if (!boot_error) { - /* - * Wait 10s total for first sign of life from AP - */ - boot_error = -1; - timeout = jiffies + 10*HZ; - while (time_before(jiffies, timeout)) { - if (cpumask_test_cpu(cpu, cpu_initialized_mask)) { - /* - * Tell AP to proceed with initialization - */ - cpumask_set_cpu(cpu, cpu_callout_mask); - boot_error = 0; - break; - } - schedule(); - } - } + return boot_error; +} - if (!boot_error) { - /* - * Wait till AP completes initial initialization - */ - while (!cpumask_test_cpu(cpu, cpu_callin_mask)) { - /* - * Allow other tasks to run while we wait for the - * AP to come online. This also gives a chance - * for the MTRR work(triggered by the AP coming online) - * to be completed in the stop machine context. - */ - schedule(); - } +static int do_wait_cpu_cpumask(unsigned int cpu, const struct cpumask *mask) +{ + unsigned long timeout; + + /* + * Wait up to 10s for the CPU to report in. + */ + timeout = jiffies + 10*HZ; + while (time_before(jiffies, timeout)) { + if (cpumask_test_cpu(cpu, mask)) + return 0; + + schedule(); } + return -1; +} - if (x86_platform.legacy.warm_reset) { - /* - * Cleanup possible dangling ends... - */ - smpboot_restore_warm_reset_vector(); +/* + * Bringup step two: Wait for the target AP to reach cpu_init_secondary() + * and thus wait_for_master_cpu(), then set cpu_callout_mask to allow it + * to proceed. The AP will then proceed past setting its 'callin' bit + * and end up waiting in check_tsc_sync_target() until we reach + * do_wait_cpu_online() to tend to it. + */ +static int do_wait_cpu_initialized(unsigned int cpu) +{ + /* + * Wait for first sign of life from AP. + */ + if (do_wait_cpu_cpumask(cpu, cpu_initialized_mask)) + return -1; + + cpumask_set_cpu(cpu, cpu_callout_mask); + return 0; +} + +/* + * Bringup step three: Wait for the target AP to reach smp_callin(). + * The AP is not waiting for us here so we don't need to parallelise + * this step. Not entirely clear why we care about this, since we just + * proceed directly to TSC synchronization which is the next sync + * point with the AP anyway. + */ +static int do_wait_cpu_callin(unsigned int cpu) +{ + /* + * Wait till AP completes initial initialization. + */ + return do_wait_cpu_cpumask(cpu, cpu_callin_mask); +} + +/* + * Bringup step four: Synchronize the TSC and wait for the target AP + * to reach set_cpu_online() in start_secondary(). + */ +static int do_wait_cpu_online(unsigned int cpu) +{ + unsigned long flags; + + /* + * Check TSC synchronization with the AP (keep irqs disabled + * while doing so): + */ + local_irq_save(flags); + check_tsc_sync_source(cpu); + local_irq_restore(flags); + + /* + * Wait for the AP to mark itself online. Not entirely + * clear why we care, since the generic cpuhp code will + * wait for it to each CPUHP_AP_ONLINE_IDLE before going + * ahead with the rest of the bringup anyway. + */ + while (!cpu_online(cpu)) { + cpu_relax(); + touch_nmi_watchdog(); } - return boot_error; + return 0; } -int native_cpu_up(unsigned int cpu, struct task_struct *tidle) +int do_cpu_up(unsigned int cpu, struct task_struct *tidle) { int apicid = apic->cpu_present_to_apicid(cpu); int cpu0_nmi_registered = 0; - unsigned long flags; int err, ret = 0; lockdep_assert_irqs_enabled(); @@ -1240,19 +1299,6 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) goto unreg_nmi; } - /* - * Check TSC synchronization with the AP (keep irqs disabled - * while doing so): - */ - local_irq_save(flags); - check_tsc_sync_source(cpu); - local_irq_restore(flags); - - while (!cpu_online(cpu)) { - cpu_relax(); - touch_nmi_watchdog(); - } - unreg_nmi: /* * Clean up the nmi handler. Do this after the callin and callout sync @@ -1264,6 +1310,34 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) return ret; } +int native_cpu_up(unsigned int cpu, struct task_struct *tidle) +{ + int ret; + + ret = do_cpu_up(cpu, tidle); + if (ret) + return ret; + + ret = do_wait_cpu_initialized(cpu); + if (ret) + return ret; + + ret = do_wait_cpu_callin(cpu); + if (ret) + return ret; + + ret = do_wait_cpu_online(cpu); + + if (x86_platform.legacy.warm_reset) { + /* + * Cleanup possible dangling ends... + */ + smpboot_restore_warm_reset_vector(); + } + + return ret; +} + /** * arch_disable_smp_support() - disables SMP support for x86 at runtime */