From patchwork Thu Feb 2 21:56:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 52185 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp494042wrn; Thu, 2 Feb 2023 13:58:31 -0800 (PST) X-Google-Smtp-Source: AK7set9NtalMCVk8AlmW1E9qF9ifAT95+wH6mRw+6Uu/yAR/ERKY/SsAlIHA09AxApkUch3eQYna X-Received: by 2002:a17:906:9610:b0:878:68e5:e52b with SMTP id s16-20020a170906961000b0087868e5e52bmr8947034ejx.66.1675375111773; Thu, 02 Feb 2023 13:58:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675375111; cv=none; d=google.com; s=arc-20160816; b=0ir4srcnFZnON09Gn3Fd63JkVCRup7cgLiPfEPJHEHXxN+UHenqvo1DWGcnJLmq8dS DnE/j99SDHv6e/yd7FBIvqcv2FvVZ5/pcBeBHg161/ZkDr/IbiMMGGoGgdSZNAPTTsTt lvStNCOYIJ2G4MDBFIK30YGCRpYz6YwK56D7w/RiiDSs4jZZCZ62K1al74054/fGZ/9y bmRoJni7J5Hq2C/F8wDsXnvjelWQ5vdGEfPDrYOEOLStaqHwScWV88vEzPs65ABCgWbX bY4X+6C2VW+sZ6//U0CZswakqu9TnKWPbt8YeuBL0f9xJ6VRmLjQvkXFLg8qm6XaqKhB K9eA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=QLN2DLBaj8FZzm0rkWwaWbCs5Zees0CejLc7Q1oFYAs=; b=JBFc9jnyJA7YzfygRTDvIviJnSeK+WqQsbZwAaccNdlBqKp5oEPCIaM1DLawnU+c1D 0iamh5M/n8U30WANOnsBnLOnKUL85jxIHz7vwsvYkjTI5IN4ptZVPM39DUOJQHXsne7y L6RR2/PADieYZo/KnkgCS4ZPeSjl1LdmG9WLu5GRQMkqg+S5oJaP5rFi2lp5zxSihPy+ 8ZnzzjykhNmkz3oahaLWd/DFY237sL/ZpdjJzeUat+SdEBSDp+QPAqcioFcS+4qUEMDH k1NwMSN+CLQuhITvTTEKxJWrKeGN8bUDqZIC9Bg4AZ3mkmlk36Uvi/29Y2O9ykPLpxoj Ceow== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=ouA5yU5C; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 14-20020a170906008e00b00878621fb6e4si969045ejc.73.2023.02.02.13.58.06; Thu, 02 Feb 2023 13:58:31 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=ouA5yU5C; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233160AbjBBV5D (ORCPT + 99 others); Thu, 2 Feb 2023 16:57:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54772 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233144AbjBBV4s (ORCPT ); Thu, 2 Feb 2023 16:56:48 -0500 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 010FC6779E for ; Thu, 2 Feb 2023 13:56:32 -0800 (PST) Received: by mail-wr1-x433.google.com with SMTP id h16so2972289wrz.12 for ; Thu, 02 Feb 2023 13:56:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QLN2DLBaj8FZzm0rkWwaWbCs5Zees0CejLc7Q1oFYAs=; b=ouA5yU5CD7FQYcPlqDuj/CEpWFiaiZ8la44dP46np9pFtUxTiAjM9MhYmW1JBKJBF1 qCmtmFu4fRdmFvIvTdQC17lmxkMq+MPs9/RSKAPbLJ0kBEEK9t7PH8jofzbmxliFZ1ST 1RY4rNKe/e7xCQ2hkb8BYKK9kH0qbA7Cv4zALFgtB3SZvW06gudaiYgBYZ6BoP1AnV7b re78foFNg+ET9Mun28BU54TsmxuhLbJIIPYS226FRKSmL0rAE7pP9j3Kbar4LLWIFwmC Zg2KUdlDx+x/mIFt9Hq5hUuLjUi4x0m7ZggiOWEcHmiS96teTzhXwXOLIV0UtMmrPuj0 Fu/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QLN2DLBaj8FZzm0rkWwaWbCs5Zees0CejLc7Q1oFYAs=; b=nwsgqrVhrPwPGIlGi7zndesXBbD6TrWI9OPf/9JgwLIvLkQ/CrNZlkL/c6AplsxMFm WaFskd+zLDV5xzN00KauTgh14XmtkPAB8KP5QuoLvNLU0hj8SyoTr5SqKLOQQqJS/aMP TraVcpZC7/zGTIUb5wpaX53B4s2c3PioEJcCFmAIAC/2iLaMSFaltJiddDishL65OKOt F4xKqPndV8qB61GW0PzoKH7zvEypWPkTYTKe51lrABHuPEDgIuOWqarRXu+RsV3IHM9N kaX7PX5177EkejHaY4h0SK53hTpFkm7XO7lBAUKwS+XTAHVqgig1f4A+HnjUE9wu4j+1 s/xA== X-Gm-Message-State: AO0yUKUpedjjGsQH+FxLgC9gbqx051z1gHcf0wSkyp4/M+adxN9kMA// i9mNRKAdw1DKqrCEVFWG/bbYOw== X-Received: by 2002:adf:e706:0:b0:2bf:d7c1:ad95 with SMTP id c6-20020adfe706000000b002bfd7c1ad95mr6593375wrm.16.1675374991446; Thu, 02 Feb 2023 13:56:31 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:98fe:e4ee:fc7e:cd71]) by smtp.gmail.com with ESMTPSA id e8-20020a5d6d08000000b00297dcfdc90fsm506078wrq.24.2023.02.02.13.56.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Feb 2023 13:56:30 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, arjan@linux.intel.com Cc: mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse Subject: [PATCH v6 01/11] x86/apic/x2apic: Fix parallel handling of cluster_mask Date: Thu, 2 Feb 2023 21:56:15 +0000 Message-Id: <20230202215625.3248306-2-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230202215625.3248306-1-usama.arif@bytedance.com> References: <20230202215625.3248306-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756758133205887054?= X-GMAIL-MSGID: =?utf-8?q?1756758133205887054?= From: David Woodhouse For each CPU being brought up, the alloc_clustermask() function allocates a new struct cluster_mask just in case it's needed. Then the target CPU actually runs, and in init_x2apic_ldr() it either uses a cluster_mask from a previous CPU in the same cluster, or consumes the "spare" one and sets the global pointer to NULL. That isn't going to parallelise stunningly well. Ditch the global variable, let alloc_clustermask() install the struct *directly* in the per_cpu data for the CPU being brought up. As an optimisation, actually make it do so for *all* present CPUs in the same cluster, which means only one iteration over for_each_present_cpu() instead of doing so repeatedly, once for each CPU. Now, in fact, there's no point in the 'node' or 'clusterid' members of the struct cluster_mask, so just kill it and use struct cpumask instead. This was a harmless "bug" while CPU bringup wasn't actually happening in parallel. It's about to become less harmless... Fixes: 023a611748fd5 ("x86/apic/x2apic: Simplify cluster management") Signed-off-by: David Woodhouse Signed-off-by: Paul E. McKenney --- arch/x86/kernel/apic/x2apic_cluster.c | 108 +++++++++++++++----------- 1 file changed, 62 insertions(+), 46 deletions(-) diff --git a/arch/x86/kernel/apic/x2apic_cluster.c b/arch/x86/kernel/apic/x2apic_cluster.c index e696e22d0531..e116dfaf5922 100644 --- a/arch/x86/kernel/apic/x2apic_cluster.c +++ b/arch/x86/kernel/apic/x2apic_cluster.c @@ -9,11 +9,7 @@ #include "local.h" -struct cluster_mask { - unsigned int clusterid; - int node; - struct cpumask mask; -}; +#define apic_cluster(apicid) ((apicid) >> 4) /* * __x2apic_send_IPI_mask() possibly needs to read @@ -23,8 +19,7 @@ struct cluster_mask { static u32 *x86_cpu_to_logical_apicid __read_mostly; static DEFINE_PER_CPU(cpumask_var_t, ipi_mask); -static DEFINE_PER_CPU_READ_MOSTLY(struct cluster_mask *, cluster_masks); -static struct cluster_mask *cluster_hotplug_mask; +static DEFINE_PER_CPU_READ_MOSTLY(struct cpumask *, cluster_masks); static int x2apic_acpi_madt_oem_check(char *oem_id, char *oem_table_id) { @@ -60,10 +55,10 @@ __x2apic_send_IPI_mask(const struct cpumask *mask, int vector, int apic_dest) /* Collapse cpus in a cluster so a single IPI per cluster is sent */ for_each_cpu(cpu, tmpmsk) { - struct cluster_mask *cmsk = per_cpu(cluster_masks, cpu); + struct cpumask *cmsk = per_cpu(cluster_masks, cpu); dest = 0; - for_each_cpu_and(clustercpu, tmpmsk, &cmsk->mask) + for_each_cpu_and(clustercpu, tmpmsk, cmsk) dest |= x86_cpu_to_logical_apicid[clustercpu]; if (!dest) @@ -71,7 +66,7 @@ __x2apic_send_IPI_mask(const struct cpumask *mask, int vector, int apic_dest) __x2apic_send_IPI_dest(dest, vector, APIC_DEST_LOGICAL); /* Remove cluster CPUs from tmpmask */ - cpumask_andnot(tmpmsk, tmpmsk, &cmsk->mask); + cpumask_andnot(tmpmsk, tmpmsk, cmsk); } local_irq_restore(flags); @@ -105,55 +100,76 @@ static u32 x2apic_calc_apicid(unsigned int cpu) static void init_x2apic_ldr(void) { - struct cluster_mask *cmsk = this_cpu_read(cluster_masks); - u32 cluster, apicid = apic_read(APIC_LDR); - unsigned int cpu; + struct cpumask *cmsk = this_cpu_read(cluster_masks); - x86_cpu_to_logical_apicid[smp_processor_id()] = apicid; + BUG_ON(!cmsk); - if (cmsk) - goto update; - - cluster = apicid >> 16; - for_each_online_cpu(cpu) { - cmsk = per_cpu(cluster_masks, cpu); - /* Matching cluster found. Link and update it. */ - if (cmsk && cmsk->clusterid == cluster) - goto update; + cpumask_set_cpu(smp_processor_id(), cmsk); +} + +/* + * As an optimisation during boot, set the cluster_mask for *all* + * present CPUs at once, to prevent *each* of them having to iterate + * over the others to find the existing cluster_mask. + */ +static void prefill_clustermask(struct cpumask *cmsk, u32 cluster) +{ + int cpu; + + for_each_present_cpu(cpu) { + u32 apicid = apic->cpu_present_to_apicid(cpu); + if (apicid != BAD_APICID && apic_cluster(apicid) == cluster) { + struct cpumask **cpu_cmsk = &per_cpu(cluster_masks, cpu); + + BUG_ON(*cpu_cmsk && *cpu_cmsk != cmsk); + *cpu_cmsk = cmsk; + } } - cmsk = cluster_hotplug_mask; - cmsk->clusterid = cluster; - cluster_hotplug_mask = NULL; -update: - this_cpu_write(cluster_masks, cmsk); - cpumask_set_cpu(smp_processor_id(), &cmsk->mask); } -static int alloc_clustermask(unsigned int cpu, int node) +static int alloc_clustermask(unsigned int cpu, u32 cluster, int node) { + struct cpumask *cmsk = NULL; + unsigned int cpu_i; + u32 apicid; + if (per_cpu(cluster_masks, cpu)) return 0; - /* - * If a hotplug spare mask exists, check whether it's on the right - * node. If not, free it and allocate a new one. - */ - if (cluster_hotplug_mask) { - if (cluster_hotplug_mask->node == node) - return 0; - kfree(cluster_hotplug_mask); + + /* For the hotplug case, don't always allocate a new one */ + if (system_state >= SYSTEM_RUNNING) { + for_each_present_cpu(cpu_i) { + apicid = apic->cpu_present_to_apicid(cpu_i); + if (apicid != BAD_APICID && apic_cluster(apicid) == cluster) { + cmsk = per_cpu(cluster_masks, cpu_i); + if (cmsk) + break; + } + } + } + if (!cmsk) { + cmsk = kzalloc_node(sizeof(*cmsk), GFP_KERNEL, node); + if (!cmsk) + return -ENOMEM; } - cluster_hotplug_mask = kzalloc_node(sizeof(*cluster_hotplug_mask), - GFP_KERNEL, node); - if (!cluster_hotplug_mask) - return -ENOMEM; - cluster_hotplug_mask->node = node; + per_cpu(cluster_masks, cpu) = cmsk; + + if (system_state < SYSTEM_RUNNING) + prefill_clustermask(cmsk, cluster); + return 0; } static int x2apic_prepare_cpu(unsigned int cpu) { - if (alloc_clustermask(cpu, cpu_to_node(cpu)) < 0) + u32 phys_apicid = apic->cpu_present_to_apicid(cpu); + u32 cluster = apic_cluster(phys_apicid); + u32 logical_apicid = (cluster << 16) | (1 << (phys_apicid & 0xf)); + + x86_cpu_to_logical_apicid[cpu] = logical_apicid; + + if (alloc_clustermask(cpu, cluster, cpu_to_node(cpu)) < 0) return -ENOMEM; if (!zalloc_cpumask_var(&per_cpu(ipi_mask, cpu), GFP_KERNEL)) return -ENOMEM; @@ -162,10 +178,10 @@ static int x2apic_prepare_cpu(unsigned int cpu) static int x2apic_dead_cpu(unsigned int dead_cpu) { - struct cluster_mask *cmsk = per_cpu(cluster_masks, dead_cpu); + struct cpumask *cmsk = per_cpu(cluster_masks, dead_cpu); if (cmsk) - cpumask_clear_cpu(dead_cpu, &cmsk->mask); + cpumask_clear_cpu(dead_cpu, cmsk); free_cpumask_var(per_cpu(ipi_mask, dead_cpu)); return 0; } From patchwork Thu Feb 2 21:56:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 52194 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp494743wrn; Thu, 2 Feb 2023 14:00:29 -0800 (PST) X-Google-Smtp-Source: AK7set83lxml3BznqblWd/H+mxC+WzoqYaFYoYD9JR8NpeGorRMDwRL/quN91HAsUlxOyoLddUAX X-Received: by 2002:a17:906:c089:b0:88f:895f:1ba4 with SMTP id f9-20020a170906c08900b0088f895f1ba4mr2556044ejz.57.1675375228852; Thu, 02 Feb 2023 14:00:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675375228; cv=none; d=google.com; s=arc-20160816; b=Kb+1HpqeWI7tUl9d5hdRPiuIr+5XDoCpyaAnjxj0BSsYKvoPwfL8l0XgtAO87yvfO2 uAuCNaiehS4WQfnFB8F3+c92CVFMA9N7hF06kp2ggfyA6k2NaO7Op3tUnFneflnIXMIN E/iKZQyzlJJ8Ua08fhBJErxNgUREJJBuCQ8fCr7sWyn4YpbCximASkmzzfSdqQMVl+4K Vlo6SJ0/vVlnOdzA4FTzPt1ykqqIXjrDlzCzsVK6WvP65+KEnBLRIsUCcCTNHCTbb5YF 1jwMGQs6+r3kzDEKWcgeF8BJv3Dsw7PZ9JddsCmkTi5FfL0hkQrvw7wpZNVlxcmR9D86 kk1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=fGUMPbv5ehkWjyJkrcI0+C5Cz30zQDLH70pUeAdBLkQ=; b=ecsaq++tzABaY5vJulO/Hft6hKvHaPEsx4zrTgPRB1pRenK1bGWjY5LDzkgeII2PPZ 4Vzl11d1quX3Er0HB4bR3pivoUzF6iiUPEu11szJev4yqlHWYe6lSULUqgNmHR+f6VQJ AKdyNYjgSn1Iq+bn5hnqzI5PSZnrFpwf/bnEdRZtAk8bctg9Vksi9n3GUH2tTlZLxSJr /7yzxQTybITdUDTGx3to5CFot/glZGiElcBWoy+vwu7whhP8h7KXWT4JCldQ460e1NEo 9E7c3vE7a+ARsBonVoVrmduATGtd5a9diw1h+YneDR6Fm8FQNJJ3BjYdYgrvBHirCcee YBcw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=2M4JOPBX; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 29-20020a170906009d00b0088767839e39si1170687ejc.72.2023.02.02.14.00.03; Thu, 02 Feb 2023 14:00:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=2M4JOPBX; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232836AbjBBV5b (ORCPT + 99 others); Thu, 2 Feb 2023 16:57:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54992 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233139AbjBBV5M (ORCPT ); Thu, 2 Feb 2023 16:57:12 -0500 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DAC8A6B02C for ; Thu, 2 Feb 2023 13:56:33 -0800 (PST) Received: by mail-wr1-x434.google.com with SMTP id a2so2718257wrd.6 for ; Thu, 02 Feb 2023 13:56:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fGUMPbv5ehkWjyJkrcI0+C5Cz30zQDLH70pUeAdBLkQ=; b=2M4JOPBXhf4n8GQQdyrzHe3qWIJaL7f9plv5S3ltmqAiKFtsvcGXGUTC2Zuij3BMbs HhlEqG9+INFdfdUkReFlmvH2JPZMrNCztu3WfOAY+06RdcFUOlBB41XvMs68Ya736yK2 nOZn1J7hYgH42tp/gN4crt3m2pxOLEjJAusMRXcc680eWWrSJLJMCjmPufzsVttIRkCW 9QkU3r7GppcN34bu8GaPenanJDgtpVPDONSrvDRiSQXJXkEZApH8tlqbG9jFp0sxPyy2 H0xXvJvm2qthioHCwmimhIU+Q9Nh8Kc7h/9OaMxXMAdSlKwy88YLZFA1gae6hbRvVar+ bsdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fGUMPbv5ehkWjyJkrcI0+C5Cz30zQDLH70pUeAdBLkQ=; b=OnzoFneTumeAw6vXt0+npfg9as+1uKMrzdb1/2ObA2Qbz9opAG2bkfdgIo+L/DzzcS khX7tNB14dJB7Ua9/+jDfvaxQQwJb7MoOR3REK1T5awBJOdTRPV0gmtbbFhg6n0IsnWt 8QeQPwE8C4rt3J9X1lXMPvLDtC74rTI0uczHDhtdb0ftq6hzQMv7lAsJ+35KWTw/I6tj OAJlb+TsSIZS16LoglSPFKxhBX2GUU8/zOLQci1RRcx/vQixt8sdnu+uzkLBSM5NuIpZ ibRtlejtB9zR3pvlhXUdGiBLmUXCSkf8851u6maFaSfft1orF61ZnnCt3epDWsgLxe2s EtYA== X-Gm-Message-State: AO0yUKVpJTvouQJ0TNmdo2Ejdzk+kb8sGfxUi4+DZeAJzjuemFSR7GDY dy1jfWWW3+Vut7m2Mjd4L9WDtw== X-Received: by 2002:a5d:6c6d:0:b0:2bf:d6f9:23d6 with SMTP id r13-20020a5d6c6d000000b002bfd6f923d6mr7783377wrz.45.1675374992316; Thu, 02 Feb 2023 13:56:32 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:98fe:e4ee:fc7e:cd71]) by smtp.gmail.com with ESMTPSA id e8-20020a5d6d08000000b00297dcfdc90fsm506078wrq.24.2023.02.02.13.56.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Feb 2023 13:56:31 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, arjan@linux.intel.com Cc: mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse Subject: [PATCH v6 02/11] cpu/hotplug: Move idle_thread_get() to Date: Thu, 2 Feb 2023 21:56:16 +0000 Message-Id: <20230202215625.3248306-3-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230202215625.3248306-1-usama.arif@bytedance.com> References: <20230202215625.3248306-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756758255656536221?= X-GMAIL-MSGID: =?utf-8?q?1756758255656536221?= From: David Woodhouse Instead of relying purely on the special-case wrapper in bringup_cpu() to pass the idle thread to __cpu_up(), expose idle_thread_get() so that the architecture code can obtain it directly when necessary. This will be useful when the existing __cpu_up() is split into multiple phases, only *one* of which will actually need the idle thread. If the architecture code is to register its new pre-bringup states with the cpuhp core, having a special-case wrapper to pass extra arguments is non-trivial and it's easier just to let the arch register its function pointer to be invoked with the standard API. Signed-off-by: David Woodhouse Signed-off-by: Paul E. McKenney Reviewed-by: Thomas Gleixner --- include/linux/smpboot.h | 7 +++++++ kernel/smpboot.h | 2 -- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/include/linux/smpboot.h b/include/linux/smpboot.h index 9d1bc65d226c..3862addcaa34 100644 --- a/include/linux/smpboot.h +++ b/include/linux/smpboot.h @@ -5,6 +5,13 @@ #include struct task_struct; + +#ifdef CONFIG_GENERIC_SMP_IDLE_THREAD +struct task_struct *idle_thread_get(unsigned int cpu); +#else +static inline struct task_struct *idle_thread_get(unsigned int cpu) { return NULL; } +#endif + /* Cookie handed to the thread_fn*/ struct smpboot_thread_data; diff --git a/kernel/smpboot.h b/kernel/smpboot.h index 34dd3d7ba40b..60c609318ad6 100644 --- a/kernel/smpboot.h +++ b/kernel/smpboot.h @@ -5,11 +5,9 @@ struct task_struct; #ifdef CONFIG_GENERIC_SMP_IDLE_THREAD -struct task_struct *idle_thread_get(unsigned int cpu); void idle_thread_set_boot_cpu(void); void idle_threads_init(void); #else -static inline struct task_struct *idle_thread_get(unsigned int cpu) { return NULL; } static inline void idle_thread_set_boot_cpu(void) { } static inline void idle_threads_init(void) { } #endif From patchwork Thu Feb 2 21:56:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 52187 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp494141wrn; Thu, 2 Feb 2023 13:58:48 -0800 (PST) X-Google-Smtp-Source: AK7set/bQJuZfWNySGvFXLquJLcujPkW9zEVIUBXuKFTG+r4pjoD7TvoqMgbX3hpwLT1KTLaaFds X-Received: by 2002:a17:906:2a0d:b0:878:7819:b8c3 with SMTP id j13-20020a1709062a0d00b008787819b8c3mr7772573eje.46.1675375127886; Thu, 02 Feb 2023 13:58:47 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675375127; cv=none; d=google.com; s=arc-20160816; b=Nafq6BUD9vymf1uEjRg/yEcjgxv6Qo7RAUdHCDhy6sz/pF1KJCG46yoIBeZceITVLq 6oxTLU2FJl6mqdYKkxps7lxYcoG4By5Ivm2rIJe7tr1xYRQ1HtIMT0rc5jcBxDO+3tWe +gxvCKU4FrKKbB0fKfs6K/pVnUS7WfznNwel0hzr/Lz//wMf4pRD+fHVdoep1Rx6nUu4 Bi5yTaDH5Upz3opsMs2JueYOSrd46CRJTOhfWebg9WYUBaMwTGDlV4Era1+oUzHv1+Mj +So01W10csAsbaaKHcqIg/jX7hV32UAd9d3cpx+AGnMKOz3hvvWfMoC14JO7fkMQCc2m GA8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ZN0/JHYpO4HmptKsonBLHsoSegp6yzcZ7Pz1OBnmRqA=; b=Bik25i+hktc3fN7SXoEqq1lyyJYRIfEwNmQNtaBGOvNI7s8Sn64EI/ZhRMmcvTS74i wB9pdHaXAHdZzXEaWKNIxyEUpJ/AArPBisiWzwvi9UDEEQ3ovnqkIRHDVmQZB9HMceqK yxtf4nne/R2N6Vyj8yGyKhkdBIjKGMApK0O4wZGrWA6UWVtoNb5BRDBJWjPtEJ000OuX fR8FR9cwWOXukgoXHYAHKoKcykdu3KBbaNJOlbNtb1pO5zdXxyQ4L0Jrfm0xL8q1xBtJ M1OdeVlCRxV+drSGWq9ByFsSFQcghQQVoh/KDvO5VQITYx7fYFe8LqSAiEIioMVoZ5Ql tTKw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=yqnYiCH8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id eu16-20020a170907299000b0088458045d8asi654159ejc.647.2023.02.02.13.58.24; Thu, 02 Feb 2023 13:58:47 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=yqnYiCH8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233258AbjBBV5f (ORCPT + 99 others); Thu, 2 Feb 2023 16:57:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55014 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233235AbjBBV5R (ORCPT ); Thu, 2 Feb 2023 16:57:17 -0500 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 459C770D63 for ; Thu, 2 Feb 2023 13:56:34 -0800 (PST) Received: by mail-wr1-x42d.google.com with SMTP id m14so2960302wrg.13 for ; Thu, 02 Feb 2023 13:56:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZN0/JHYpO4HmptKsonBLHsoSegp6yzcZ7Pz1OBnmRqA=; b=yqnYiCH8AqjbpexuA4/vbph6hoxbvOnFhN1VxUqRO95nCraYxvYmd29sN+sEN/dpGG qxQDpQJPYq2APbiu5GOOas85BV89BgLqSB51zCXUT7VVHB9vtkWtygPz5StqwzLrzNFA QBdT8ALik941Wvo4RA0zOH+VtJhYzV7HawC2LrqvNxKAhGUtMYsoWHtXeQFZ/jE2LZm3 kH5hPItDQILWhGiBf1ejjmMXVtuQT19dCO/Xwis6fWph+NCB844DcHURiyK3hAHeAdN6 h7CsMWxwW2cRn540KCGUyJiKH6fppnyjLc5mkK8/SO+NTN2/XNJw+JKcDyOM0YLUuPM8 Zu3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZN0/JHYpO4HmptKsonBLHsoSegp6yzcZ7Pz1OBnmRqA=; b=muto+AMWC+A7a/P3E3u8eJZjcG+iQZpO406MXk9Qx6meR8VBWh6SYoI/gMNCSaTRbo T6zHfrB8khnSAbGqCAc0XsG9+Y9K/JqecWdsskkiYMXjCsbtxG0hTVoDXwBFw0OFEwZL kynVqteE6AkHlXjSQ6/idzCrqUHkWJkU9cXEN6KCD3QK3cl5VrIf9yaZErFuExkQR1ce zHTgn/iWdw04NjgFYPeG/+AcbseOBS/1wH7GOuljDuL+/L6SBrR9RslQ0t/CrLhA2fGz N1PjwTPcsms5gVvE+EFf4DhALQFF47ooViXpVnAR2EeBvW0R3BPPCKjgq8KnvHAqkP3/ 935w== X-Gm-Message-State: AO0yUKUVZZJzcvnuPrcZWLCCZctsdof2geVOYr+XJTYnAyAds+O9rgFg 97chHz/TmwsMAiTT19DqE6kJ+Q== X-Received: by 2002:a05:6000:15cd:b0:2bf:d8ed:ba46 with SMTP id y13-20020a05600015cd00b002bfd8edba46mr8519729wry.47.1675374993184; Thu, 02 Feb 2023 13:56:33 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:98fe:e4ee:fc7e:cd71]) by smtp.gmail.com with ESMTPSA id e8-20020a5d6d08000000b00297dcfdc90fsm506078wrq.24.2023.02.02.13.56.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Feb 2023 13:56:32 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, arjan@linux.intel.com Cc: mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse Subject: [PATCH v6 03/11] cpu/hotplug: Add dynamic parallel bringup states before CPUHP_BRINGUP_CPU Date: Thu, 2 Feb 2023 21:56:17 +0000 Message-Id: <20230202215625.3248306-4-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230202215625.3248306-1-usama.arif@bytedance.com> References: <20230202215625.3248306-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756758149762915410?= X-GMAIL-MSGID: =?utf-8?q?1756758149762915410?= From: David Woodhouse If the platform registers these states, bring all CPUs to each registered state in turn, before the final bringup to CPUHP_BRINGUP_CPU. This allows the architecture to parallelise the slow asynchronous tasks like sending INIT/SIPI and waiting for the AP to come to life. There is a subtlety here: even with an empty CPUHP_BP_PARALLEL_DYN step, this means that *all* CPUs are brought through the prepare states and to CPUHP_BP_PREPARE_DYN before any of them are taken to CPUHP_BRINGUP_CPU and then are allowed to run for themselves to CPUHP_ONLINE. So any combination of prepare/start calls which depend on A-B ordering for each CPU in turn, such as the X2APIC code which used to allocate a cluster mask 'just in case' and store it in a global variable in the prep stage, then potentially consume that preallocated structure from the AP and set the global pointer to NULL to be reallocated in CPUHP_X2APIC_PREPARE for the next CPU... would explode horribly. We believe that X2APIC was the only such case, for x86. But this is why it remains an architecture opt-in. For now. Note that the new parallel stages do *not* yet bring each AP to the CPUHP_BRINGUP_CPU state. The final loop in bringup_nonboot_cpus() is untouched, bringing each AP in turn from the final PARALLEL_DYN state (or all the way from CPUHP_OFFLINE) to CPUHP_BRINGUP_CPU and then waiting for that AP to do its own processing and reach CPUHP_ONLINE before releasing the next. Parallelising that part by bringing them all to CPUHP_BRINGUP_CPU and then waiting for them all is an exercise for the future. Signed-off-by: David Woodhouse Signed-off-by: Paul E. McKenney --- include/linux/cpuhotplug.h | 2 ++ kernel/cpu.c | 27 +++++++++++++++++++++++++-- 2 files changed, 27 insertions(+), 2 deletions(-) diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h index 6c6859bfc454..e5a73ae6ccc0 100644 --- a/include/linux/cpuhotplug.h +++ b/include/linux/cpuhotplug.h @@ -133,6 +133,8 @@ enum cpuhp_state { CPUHP_MIPS_SOC_PREPARE, CPUHP_BP_PREPARE_DYN, CPUHP_BP_PREPARE_DYN_END = CPUHP_BP_PREPARE_DYN + 20, + CPUHP_BP_PARALLEL_DYN, + CPUHP_BP_PARALLEL_DYN_END = CPUHP_BP_PARALLEL_DYN + 4, CPUHP_BRINGUP_CPU, /* diff --git a/kernel/cpu.c b/kernel/cpu.c index 6c0a92ca6bb5..5a8f1a93b57c 100644 --- a/kernel/cpu.c +++ b/kernel/cpu.c @@ -1505,6 +1505,24 @@ int bringup_hibernate_cpu(unsigned int sleep_cpu) void bringup_nonboot_cpus(unsigned int setup_max_cpus) { unsigned int cpu; + int n = setup_max_cpus - num_online_cpus(); + + /* ∀ parallel pre-bringup state, bring N CPUs to it */ + if (n > 0) { + enum cpuhp_state st = CPUHP_BP_PARALLEL_DYN; + + while (st <= CPUHP_BP_PARALLEL_DYN_END && + cpuhp_hp_states[st].name) { + int i = n; + + for_each_present_cpu(cpu) { + cpu_up(cpu, st); + if (!--i) + break; + } + st++; + } + } for_each_present_cpu(cpu) { if (num_online_cpus() >= setup_max_cpus) @@ -1882,6 +1900,10 @@ static int cpuhp_reserve_state(enum cpuhp_state state) step = cpuhp_hp_states + CPUHP_BP_PREPARE_DYN; end = CPUHP_BP_PREPARE_DYN_END; break; + case CPUHP_BP_PARALLEL_DYN: + step = cpuhp_hp_states + CPUHP_BP_PARALLEL_DYN; + end = CPUHP_BP_PARALLEL_DYN_END; + break; default: return -EINVAL; } @@ -1906,14 +1928,15 @@ static int cpuhp_store_callbacks(enum cpuhp_state state, const char *name, /* * If name is NULL, then the state gets removed. * - * CPUHP_AP_ONLINE_DYN and CPUHP_BP_PREPARE_DYN are handed out on + * CPUHP_AP_ONLINE_DYN and CPUHP_BP_P*_DYN are handed out on * the first allocation from these dynamic ranges, so the removal * would trigger a new allocation and clear the wrong (already * empty) state, leaving the callbacks of the to be cleared state * dangling, which causes wreckage on the next hotplug operation. */ if (name && (state == CPUHP_AP_ONLINE_DYN || - state == CPUHP_BP_PREPARE_DYN)) { + state == CPUHP_BP_PREPARE_DYN || + state == CPUHP_BP_PARALLEL_DYN)) { ret = cpuhp_reserve_state(state); if (ret < 0) return ret; From patchwork Thu Feb 2 21:56:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 52188 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp494157wrn; Thu, 2 Feb 2023 13:58:50 -0800 (PST) X-Google-Smtp-Source: AK7set8dSbgaR+tOaiOjtpnVlUbKgul8gvSCBuGsbj8+bJ82uMP1mlEeTcgIhJLqEW2ouykd9qhk X-Received: by 2002:a17:906:5054:b0:878:7eb7:3488 with SMTP id e20-20020a170906505400b008787eb73488mr7265612ejk.21.1675375129969; Thu, 02 Feb 2023 13:58:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675375129; cv=none; d=google.com; s=arc-20160816; b=FehDgh/aD+rxD1JRWUsdiPb+HGmkkrDU0e22hoysrUnGpxk9sOAa9GLhD8/T7zOy/o T7RPuNWvew3TFQTztCaYFwO54OIWsU+wtP5aitGNMchDszhmKVlNigHkWTE9S6MkXxnU PH1166saCuGO+bEdj8ns9HEarGheS4PeimwUIL+nx/uGRTR0Dk49w8ncYA6Dr0CAZS8p iohB1V0Ksbe57+Deth+60scPSFtWylWy70DhtFKhl5mX22E1UrNtQzhrfIzwaYavvKE8 YR6OIKhbjW7zFwMbpR5rLAaYZjuejSsHr/vN0sKFC50CTc9T/KC6KG+fRqj8o7I6451p /6HQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=1t64Rjr6YniyNyrn83Uc/qtpVNvig4MY0wcaZ4A3opU=; b=Nt72Awi+jLuK0BEfE5ziTwi6JnMuKf8dK5kHfRSSxa4ZY18i2fXx3rCF+N6jHpxImF r/jMABXGGd63SBG7PofzuBe0ezW/TIDkCQirALNeWwoxIq7muBkKg+Y2V5WT7g7jrbw2 ONZphfVVHxc+dclZUqyRp7Sw4KsV6eRco+VwU3wAgmhC0UqlpJA2hITYiSDEihSCGW9I 04zHAG262nYtWxkSTh6julNWOeTbqbSWzi4WxxK5GnTeSZi27ByRgjYbQRMUx7FR9TKJ bI9UAJzjchNTkLYsH9/IlzmWHFd2JJS3iMfjT2+CWXoUTy/+Y8blHgD+Pq1nPYbKEx8L GrfQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b="XUy0x/p5"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id o13-20020a17090611cd00b0088c0fd7cebcsi871509eja.806.2023.02.02.13.58.25; Thu, 02 Feb 2023 13:58:49 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b="XUy0x/p5"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233136AbjBBV5Z (ORCPT + 99 others); Thu, 2 Feb 2023 16:57:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54996 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232908AbjBBV5L (ORCPT ); Thu, 2 Feb 2023 16:57:11 -0500 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7ECBB70D6E for ; Thu, 2 Feb 2023 13:56:35 -0800 (PST) Received: by mail-wr1-x434.google.com with SMTP id r2so2987058wrv.7 for ; Thu, 02 Feb 2023 13:56:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1t64Rjr6YniyNyrn83Uc/qtpVNvig4MY0wcaZ4A3opU=; b=XUy0x/p5QdV4qoA/2HTEBNCjHnhO7mdM58DCWfmzoZFW5rDKNboAfZ6LAtzKpomG55 eM0FkVsak3pwWr+bZ1NlxDE5LtDtZ6RVEH2qyvRxBe9aSM+bOCJ+ERXyo7ZLACdgGJH6 QAEGDDoSkk9+UDVcm3SrAo4Gl8UrrPERgu+cOQIQ1w00v8K19tm2ECCUullLs9vKvgEG QFPt14rCMgFr7zH5aIQmbVtfcb31sL+nARVUnFY14ZcOJufopoIACFr/WrOU+NSItQkT odm+Y9mrXFoFGiJglBNIH7vaG2MUNECtaO4+JYyT25LuQ0Uqy2JJcpOeFJa828ZmtNqY TnQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1t64Rjr6YniyNyrn83Uc/qtpVNvig4MY0wcaZ4A3opU=; b=7NpQvRQ56OlmcH5wOYvheQHkZxwyD57rsFlDbJWK5cmkw+EN2k0LOEC8JSblzGAxXf bdxnozlD4meyyP3XbIKv62AUmc1umOrcZQsR/nIydxJaBONdegHsAFQjExfpJUNA9aww Opq6jXOHQQrRh2DP11eNBmZN0KuevS1ZRWjKMSusivbKRPofgaJ1U9P1jZvwEEdoxoBo kz4Oq+zNVewEO8hQrmd5M8g+/Y+hxR/+vsJrjvbTiQOtYIgyClaunheyQ2zpOJcOQcUI MiprB++Ob5t2YGiuEnuYUL9KgpjKkSkFrE784jGOtXXhO8kgB4wVjs2350ZF+HDdNWdj 5YsQ== X-Gm-Message-State: AO0yUKXheDC5H8EAPYSHV9JNW/B/ZUMtp7+bHti/rX61T1RNIdKrDx1Y rL6bA7ndsekz4YaSZOZvBF2GlA== X-Received: by 2002:adf:fa84:0:b0:2bf:e449:e72a with SMTP id h4-20020adffa84000000b002bfe449e72amr6584290wrr.60.1675374994014; Thu, 02 Feb 2023 13:56:34 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:98fe:e4ee:fc7e:cd71]) by smtp.gmail.com with ESMTPSA id e8-20020a5d6d08000000b00297dcfdc90fsm506078wrq.24.2023.02.02.13.56.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Feb 2023 13:56:33 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, arjan@linux.intel.com Cc: mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse , Usama Arif Subject: [PATCH v6 04/11] x86/smpboot: Reference count on smpboot_setup_warm_reset_vector() Date: Thu, 2 Feb 2023 21:56:18 +0000 Message-Id: <20230202215625.3248306-5-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230202215625.3248306-1-usama.arif@bytedance.com> References: <20230202215625.3248306-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756758151916545453?= X-GMAIL-MSGID: =?utf-8?q?1756758151916545453?= From: David Woodhouse If we want to do parallel CPU bringup, we're going to need to set this up and leave it until all CPUs are done. Might as well use the RTC spinlock to protect the refcount, as we need to take it anyway. [Usama Arif: fixed rebase conflict] Signed-off-by: David Woodhouse Signed-off-by: Usama Arif Signed-off-by: Paul E. McKenney --- arch/x86/kernel/smpboot.c | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 55cad72715d9..a19eddcdccc2 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -121,17 +121,22 @@ int arch_update_cpu_topology(void) return retval; } + +static unsigned int smpboot_warm_reset_vector_count; + static inline void smpboot_setup_warm_reset_vector(unsigned long start_eip) { unsigned long flags; spin_lock_irqsave(&rtc_lock, flags); - CMOS_WRITE(0xa, 0xf); + if (!smpboot_warm_reset_vector_count++) { + CMOS_WRITE(0xa, 0xf); + *((volatile unsigned short *)phys_to_virt(TRAMPOLINE_PHYS_HIGH)) = + start_eip >> 4; + *((volatile unsigned short *)phys_to_virt(TRAMPOLINE_PHYS_LOW)) = + start_eip & 0xf; + } spin_unlock_irqrestore(&rtc_lock, flags); - *((volatile unsigned short *)phys_to_virt(TRAMPOLINE_PHYS_HIGH)) = - start_eip >> 4; - *((volatile unsigned short *)phys_to_virt(TRAMPOLINE_PHYS_LOW)) = - start_eip & 0xf; } static inline void smpboot_restore_warm_reset_vector(void) @@ -143,10 +148,12 @@ static inline void smpboot_restore_warm_reset_vector(void) * to default values. */ spin_lock_irqsave(&rtc_lock, flags); - CMOS_WRITE(0, 0xf); + if (!--smpboot_warm_reset_vector_count) { + CMOS_WRITE(0, 0xf); + *((volatile u32 *)phys_to_virt(TRAMPOLINE_PHYS_LOW)) = 0; + } spin_unlock_irqrestore(&rtc_lock, flags); - *((volatile u32 *)phys_to_virt(TRAMPOLINE_PHYS_LOW)) = 0; } /* From patchwork Thu Feb 2 21:56:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 52191 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp494479wrn; Thu, 2 Feb 2023 13:59:47 -0800 (PST) X-Google-Smtp-Source: AK7set94FUcssNOV1mY7gZdVxXLvzEEJZqE7zTwqPH8NlmHoQN+ha7weC9aAcR8s+M+3iFDggBoG X-Received: by 2002:a17:907:2bc2:b0:87b:59d9:5a03 with SMTP id gv2-20020a1709072bc200b0087b59d95a03mr7635079ejc.36.1675375187521; Thu, 02 Feb 2023 13:59:47 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675375187; cv=none; d=google.com; s=arc-20160816; b=HXEWEX1+wEjHRLHVntLA/EvKJo2ol9fBO6NLGY83acn1UCh/pdq+Qp00bPHhoShO44 YGmuJu4UZQe6Ya1NV/fTQuaeWersj6fYAa7+ZyuU8sD7ecdntCD4oMRnTmn0JVEBKsri He2aIhcsYCITFK4CxnGD5pjStJHeVGHDI1TFgJRtSk9Fl9uxQwaYQ98UL6TwPWD3MNg6 qocT8PmmjMuGrVNPN4YvPzZh1KtDsRDT+uaprJ3qdDA3uXJ8vuOasmMgabUj7ARXsO35 VAJvdi1oCasQcDPNeOXiuoMDg5HzwGB3TBkWb+94FYMoUte8mBxQy8GPE2Br7wJ1NNfQ 3ckg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=P3b1xgggD+wYGituLwlxcxnoFl/cpgpPrv2JCvE2DdQ=; b=MrTbHllRPs5WG8bvLfhrTTyIc/fThQptS0mGhn7Zy/O+zUracihAWN94yn1yFJkw2c c1jYoUNx8kduRpUCS5WYYneF37f/ng7OUi3JwufZ9B/0fsQTjLSDwXlHW7EZFgOQIOcm CSlyMDfXIZOzFxXfth0lmPt76xJab8WMbhQGl39jTqm1ZbVgaOkHINOzFj50dDny1ibU FHUXEmmehFFukPBrSPm0zsPoLnk0K3mfpmtGfvqFnWpoFhfq1X4QJW+DvvtAQMo8d4QC +HL93rAhw13RhH8MKAzn/6YpvM5Cxr2PocX6wuboYW7L6g9s/uAfnqQlgSuAREet8VNy LwXg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=abVhKq6Q; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 6-20020a170906010600b00882da2a9554si983172eje.592.2023.02.02.13.59.24; Thu, 02 Feb 2023 13:59:47 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=abVhKq6Q; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232916AbjBBV5H (ORCPT + 99 others); Thu, 2 Feb 2023 16:57:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54832 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233152AbjBBV4w (ORCPT ); Thu, 2 Feb 2023 16:56:52 -0500 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8F199721DA for ; Thu, 2 Feb 2023 13:56:36 -0800 (PST) Received: by mail-wr1-x42b.google.com with SMTP id y1so3012723wru.2 for ; Thu, 02 Feb 2023 13:56:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=P3b1xgggD+wYGituLwlxcxnoFl/cpgpPrv2JCvE2DdQ=; b=abVhKq6QTjCUReQpSDjXu9VjJNCsdd8u4ifNmoxDhEWw386xoJPB2ENR9yJxaSImIe De2DRhHr7GcoZ78XSnn+hMZqJuVXYTlY2sN3ahWFQNbC2oCGZN9/HknCOCJfvs86FIVu qP9Zjt0igGTfUQenfalfuDj2uTqoKVCCgK17WLmVnA36vFeuj6kF0QuIS0d+XbRv+O+z UfYzZ5dEuLGo1frWazP0PdLffbfIVWH4azBnxYGtWelXIVU6ub5i+EPLFcP62s5HgjJ4 ZyWBTNJYFiZMXAZR4vqoV5vP1r3Vtb5YKBsMuTfHL+ljOeOe9iKNIaPBF9jx5PSrsmoi YF0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=P3b1xgggD+wYGituLwlxcxnoFl/cpgpPrv2JCvE2DdQ=; b=6s7/cS5Geas9GzZPwDQHNo7h0gFzN+vaHGSfx0HF70laL13LxSfKMIvPrkv2nyW1+1 SfY6zroBFRT8ZUOs4GSfXuUodC0CYXxEsUYZbGnmaL4nf7l8mJx4SdCg1JXedIHxXrIz gCSRGHfr7HHgb7dyzTDGUuHCa/MEbWXV7blYX5wiDwYt94Jtuzrz4G8LOoFhwF6FSFmJ K9soLGDNW0/yAnZd98FJJvJ7nXAX0z3HAnTAFsq+Bi+lpqMIihrvsxTQ3Ll6FUyZ42xA WP6Z/tPKTnKumrJ8O1LXJY6hfGnGML5K7WE6KDY2Wuo4Pahmd/d5yt4JrlLvk+KG9E6t 7yOA== X-Gm-Message-State: AO0yUKXQaayV7NQhnA5wgsWfUZl2WjNgR34fJK8Y/rLwmB2FceQCsO9O +93tiwsU9g7mpkhTsCwe8WngHA== X-Received: by 2002:a05:6000:186a:b0:257:15b9:8b66 with SMTP id d10-20020a056000186a00b0025715b98b66mr8762992wri.62.1675374994908; Thu, 02 Feb 2023 13:56:34 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:98fe:e4ee:fc7e:cd71]) by smtp.gmail.com with ESMTPSA id e8-20020a5d6d08000000b00297dcfdc90fsm506078wrq.24.2023.02.02.13.56.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Feb 2023 13:56:34 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, arjan@linux.intel.com Cc: mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse , Usama Arif Subject: [PATCH v6 05/11] x86/smpboot: Split up native_cpu_up into separate phases and document them Date: Thu, 2 Feb 2023 21:56:19 +0000 Message-Id: <20230202215625.3248306-6-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230202215625.3248306-1-usama.arif@bytedance.com> References: <20230202215625.3248306-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756758212886214861?= X-GMAIL-MSGID: =?utf-8?q?1756758212886214861?= From: David Woodhouse There are four logical parts to what native_cpu_up() does on the BSP (or on the controlling CPU for a later hotplug). First it actually wakes the AP by sending the INIT/SIPI/SIPI sequence. Second, it waits for the AP to make it as far as wait_for_master_cpu() which sets that CPU's bit in cpu_initialized_mask, then sets the bit in cpu_callout_mask to let the AP proceed through cpu_init(). Then, it waits for the AP to finish cpu_init() and get as far as the smp_callin() call, which sets that CPU's bit in cpu_callin_mask. Finally, it does the TSC synchronization and waits for the AP to actually mark itself online in cpu_online_mask. This commit should have no behavioural change, but merely splits those phases out into separate functions so that future commits can make them happen in parallel for all APs. And adds some comments around them on both the BSP and AP code paths. [Usama Arif: fixed rebase conflict] Signed-off-by: David Woodhouse Signed-off-by: Usama Arif Signed-off-by: Paul E. McKenney --- arch/x86/kernel/smpboot.c | 182 +++++++++++++++++++++++++++----------- 1 file changed, 128 insertions(+), 54 deletions(-) diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index a19eddcdccc2..fdcf7c08945f 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -206,6 +206,10 @@ static void smp_callin(void) wmb(); + /* + * This runs the AP through all the cpuhp states to its target + * state (CPUHP_ONLINE in the case of serial bringup). + */ notify_cpu_starting(cpuid); /* @@ -233,17 +237,33 @@ static void notrace start_secondary(void *unused) load_cr3(swapper_pg_dir); __flush_tlb_all(); #endif + /* + * Sync point with do_wait_cpu_initialized(). On boot, all secondary + * CPUs reach this stage after receiving INIT/SIPI from do_cpu_up() + * in the x86/cpu:kick cpuhp stage. At the start of cpu_init() they + * will wait for do_wait_cpu_initialized() to set their bit in + * smp_callout_mask to release them. + */ cpu_init_secondary(); rcu_cpu_starting(raw_smp_processor_id()); x86_cpuinit.early_percpu_clock_init(); + + /* + * Sync point with do_wait_cpu_callin(). The AP doesn't wait here + * but just sets the bit to let the controlling CPU (BSP) know that + * it's got this far. + */ smp_callin(); enable_start_cpu0 = 0; /* otherwise gcc will move up smp_processor_id before the cpu_init */ barrier(); + /* - * Check TSC synchronization with the boot CPU: + * Check TSC synchronization with the boot CPU (or whichever CPU + * is controlling the bringup). It will do its part of this from + * do_wait_cpu_online(), making it an implicit sync point. */ check_tsc_sync_target(); @@ -256,6 +276,7 @@ static void notrace start_secondary(void *unused) * half valid vector space. */ lock_vector_lock(); + /* Sync point with do_wait_cpu_online() */ set_cpu_online(smp_processor_id(), true); lapic_online(); unlock_vector_lock(); @@ -1085,7 +1106,6 @@ static int do_boot_cpu(int apicid, int cpu, struct task_struct *idle, unsigned long start_ip = real_mode_header->trampoline_start; unsigned long boot_error = 0; - unsigned long timeout; #ifdef CONFIG_X86_64 /* If 64-bit wakeup method exists, use the 64-bit mode trampoline IP */ @@ -1146,55 +1166,94 @@ static int do_boot_cpu(int apicid, int cpu, struct task_struct *idle, boot_error = wakeup_cpu_via_init_nmi(cpu, start_ip, apicid, cpu0_nmi_registered); - if (!boot_error) { - /* - * Wait 10s total for first sign of life from AP - */ - boot_error = -1; - timeout = jiffies + 10*HZ; - while (time_before(jiffies, timeout)) { - if (cpumask_test_cpu(cpu, cpu_initialized_mask)) { - /* - * Tell AP to proceed with initialization - */ - cpumask_set_cpu(cpu, cpu_callout_mask); - boot_error = 0; - break; - } - schedule(); - } - } + return boot_error; +} - if (!boot_error) { - /* - * Wait till AP completes initial initialization - */ - while (!cpumask_test_cpu(cpu, cpu_callin_mask)) { - /* - * Allow other tasks to run while we wait for the - * AP to come online. This also gives a chance - * for the MTRR work(triggered by the AP coming online) - * to be completed in the stop machine context. - */ - schedule(); - } +static int do_wait_cpu_cpumask(unsigned int cpu, const struct cpumask *mask) +{ + unsigned long timeout; + + /* + * Wait up to 10s for the CPU to report in. + */ + timeout = jiffies + 10*HZ; + while (time_before(jiffies, timeout)) { + if (cpumask_test_cpu(cpu, mask)) + return 0; + + schedule(); } + return -1; +} - if (x86_platform.legacy.warm_reset) { - /* - * Cleanup possible dangling ends... - */ - smpboot_restore_warm_reset_vector(); +/* + * Bringup step two: Wait for the target AP to reach cpu_init_secondary() + * and thus wait_for_master_cpu(), then set cpu_callout_mask to allow it + * to proceed. The AP will then proceed past setting its 'callin' bit + * and end up waiting in check_tsc_sync_target() until we reach + * do_wait_cpu_online() to tend to it. + */ +static int do_wait_cpu_initialized(unsigned int cpu) +{ + /* + * Wait for first sign of life from AP. + */ + if (do_wait_cpu_cpumask(cpu, cpu_initialized_mask)) + return -1; + + cpumask_set_cpu(cpu, cpu_callout_mask); + return 0; +} + +/* + * Bringup step three: Wait for the target AP to reach smp_callin(). + * The AP is not waiting for us here so we don't need to parallelise + * this step. Not entirely clear why we care about this, since we just + * proceed directly to TSC synchronization which is the next sync + * point with the AP anyway. + */ +static int do_wait_cpu_callin(unsigned int cpu) +{ + /* + * Wait till AP completes initial initialization. + */ + return do_wait_cpu_cpumask(cpu, cpu_callin_mask); +} + +/* + * Bringup step four: Synchronize the TSC and wait for the target AP + * to reach set_cpu_online() in start_secondary(). + */ +static int do_wait_cpu_online(unsigned int cpu) +{ + unsigned long flags; + + /* + * Check TSC synchronization with the AP (keep irqs disabled + * while doing so): + */ + local_irq_save(flags); + check_tsc_sync_source(cpu); + local_irq_restore(flags); + + /* + * Wait for the AP to mark itself online. Not entirely + * clear why we care, since the generic cpuhp code will + * wait for it to each CPUHP_AP_ONLINE_IDLE before going + * ahead with the rest of the bringup anyway. + */ + while (!cpu_online(cpu)) { + cpu_relax(); + touch_nmi_watchdog(); } - return boot_error; + return 0; } -int native_cpu_up(unsigned int cpu, struct task_struct *tidle) +static int do_cpu_up(unsigned int cpu, struct task_struct *tidle) { int apicid = apic->cpu_present_to_apicid(cpu); int cpu0_nmi_registered = 0; - unsigned long flags; int err, ret = 0; lockdep_assert_irqs_enabled(); @@ -1241,19 +1300,6 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) goto unreg_nmi; } - /* - * Check TSC synchronization with the AP (keep irqs disabled - * while doing so): - */ - local_irq_save(flags); - check_tsc_sync_source(cpu); - local_irq_restore(flags); - - while (!cpu_online(cpu)) { - cpu_relax(); - touch_nmi_watchdog(); - } - unreg_nmi: /* * Clean up the nmi handler. Do this after the callin and callout sync @@ -1265,6 +1311,34 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) return ret; } +int native_cpu_up(unsigned int cpu, struct task_struct *tidle) +{ + int ret; + + ret = do_cpu_up(cpu, tidle); + if (ret) + return ret; + + ret = do_wait_cpu_initialized(cpu); + if (ret) + return ret; + + ret = do_wait_cpu_callin(cpu); + if (ret) + return ret; + + ret = do_wait_cpu_online(cpu); + + if (x86_platform.legacy.warm_reset) { + /* + * Cleanup possible dangling ends... + */ + smpboot_restore_warm_reset_vector(); + } + + return ret; +} + /** * arch_disable_smp_support() - disables SMP support for x86 at runtime */ From patchwork Thu Feb 2 21:56:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 52195 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp495001wrn; Thu, 2 Feb 2023 14:00:56 -0800 (PST) X-Google-Smtp-Source: AK7set+3XeCmjkfLr10DPXKjHhcdQFBzrMPIQxbxH75o+uViaoCcsdIhPVdUB0e/VY0yxX/WS9BV X-Received: by 2002:a50:d688:0:b0:49d:2a42:b8c8 with SMTP id r8-20020a50d688000000b0049d2a42b8c8mr8151329edi.26.1675375255970; Thu, 02 Feb 2023 14:00:55 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675375255; cv=none; d=google.com; s=arc-20160816; b=kdTsiy73htqmbCWIl+9ANdNxQBNJTjhlXXtzFhFtEOgWdVLyyTiTjfooHv/Cg2kKmC G92iO8XIVWFSBcxC2EuVCLzrk52Os+6/PHavKsh1K8hOX8pRxnEBwY8HwW/fVWFzuDvE p9GU6Katm3dZyQdsH+Y5m+M1sdrTOrB5LDa1sM0Npg/1l3BmHQd8lFuJSkuRvwJaFvyV ITDbGiO3L+Xqfh7vFvrw1MoannBseiwYFL7Z5yBZyU5c8jCa5Ja2bVkV83BadwFeEKvY lTM+I+63PcVFpRQaGSgyJUQLlbXNvdU1uVGwr1v2tnTxuwJq7Lq1atSunam6TwSfukxS LutA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Uq4IIFgjFjjgLZ+/cLmi0nocsO1Q7ZxNrpkj/NM56gw=; b=z3L8DMAs6u3V9HaTx802vU4pGLhalcfc+KpKXRalcnveRX7iNp7iNB0E5Z/EzIG7vi aBxVjsD1uncQuoGdlHKGByTIX8E0majfJmd1KonJ80T33K1Deyegx+VX4xY1BVpcTwnh RQthEzArLaqJEblGaJrkSiYgQnkfcNWowoDpMjsIX2zY2gjHa9WsOyNaUdtArcZcuYCs I1/fbRjMiiHIXDES3bXwvuQdoJhA9bc3gb3TaynvoACeF6GPoa9i+/mKu5dwacnZfx7E RW/meqZQVp50fRhiuQpuPWHjh3lNIUwtbeqU7I4bFRUNW+3GKGVZfDpTvnignrdGGzQU oacQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=omal6vPR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id co22-20020a0564020c1600b00499b3b2cfbcsi516887edb.586.2023.02.02.14.00.30; Thu, 02 Feb 2023 14:00:55 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=omal6vPR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233236AbjBBV5j (ORCPT + 99 others); Thu, 2 Feb 2023 16:57:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54828 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232085AbjBBV5R (ORCPT ); Thu, 2 Feb 2023 16:57:17 -0500 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8113272651 for ; Thu, 2 Feb 2023 13:56:37 -0800 (PST) Received: by mail-wr1-x42b.google.com with SMTP id o18so3008587wrj.3 for ; Thu, 02 Feb 2023 13:56:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Uq4IIFgjFjjgLZ+/cLmi0nocsO1Q7ZxNrpkj/NM56gw=; b=omal6vPRAFRonalxLv24xaEfbKC/ulBQ4UjQAurWupkCFaMIwhsG6z88M0drVXQuim dSiM341KsSm0Thqujea1m9ci1noUAJv0+wTviAIkog8TUPCYKOWCiqFzMV+WmQ68v1rg nbDvkDFMN1CQx9o4oYWcyntuUp9ufmo+xqRagAG/vDECcFk7We4/ototM2rXrUqvdwGY urTOniol4f1QV0DQirwhlYTNG3LTdLjVCB9me1iGCp2McgTqJM4lcSHhnLhNzxNvMzcc r+4x1YLOSF7/Eao8DrYosokkzKneRWY8hzO8Rl2gun/bnTVWS2gtGJz1hUGEA4zJ9yTw lVwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Uq4IIFgjFjjgLZ+/cLmi0nocsO1Q7ZxNrpkj/NM56gw=; b=qw1Vt+YgWwK6wE7Qbc5AUo2H6r4RNMz7OMlLWR7qU5r+RoPB9xqlrGk9uJJSdIAVt8 NgEJA+vnJIDcs4GrdXTx0jgKdmONGvA+7C0Y3biUuHaU2XBHPruSu0Y2Md0s2WZ8cYv8 v7Td5ZoSVPFV1CgAV6ztf1Xq7Jzedgccx1m2K1GJiVTop6V4Cbp1w7y8Ks3C+VpRN8S1 DQNqc95V5LPXqmW7Wad1j7OV6dcajeRkXLxzA6KM6vDq4oem8m8rT6rAOXOGv+DjUPXf X6fKQCjrx//RlBudlI8Un+m/ci4lKqqvXMk51KDLtOYzxUoNxnRlm2rujdA8/GIUDWkj NlGA== X-Gm-Message-State: AO0yUKWsUJuYF27efSrMdWmvgZkBeh9qqjTDUGRxY02TWVx5g/g/7reg rubuHy/2Jlt6rNADSLKtHYWUyQ== X-Received: by 2002:adf:b612:0:b0:2c3:cf13:c46 with SMTP id f18-20020adfb612000000b002c3cf130c46mr231369wre.53.1675374995961; Thu, 02 Feb 2023 13:56:35 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:98fe:e4ee:fc7e:cd71]) by smtp.gmail.com with ESMTPSA id e8-20020a5d6d08000000b00297dcfdc90fsm506078wrq.24.2023.02.02.13.56.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Feb 2023 13:56:35 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, arjan@linux.intel.com Cc: mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, Usama Arif Subject: [PATCH v6 06/11] x86/smpboot: Support parallel startup of secondary CPUs Date: Thu, 2 Feb 2023 21:56:20 +0000 Message-Id: <20230202215625.3248306-7-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230202215625.3248306-1-usama.arif@bytedance.com> References: <20230202215625.3248306-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756758284439412789?= X-GMAIL-MSGID: =?utf-8?q?1756758284439412789?= From: Thomas Gleixner To allow for parallel AP bringup, we need to avoid the use of global variables for passing information to the APs, as well as preventing them from all trying to use the same real-mode stack simultaneously. So, introduce a 'lock' field in struct trampoline_header to use as a simple bit-spinlock for the real-mode stack. That lock also protects the global variables initial_gs, initial_stack and early_gdt_descr, which can now be calculated... So how do we calculate those addresses? Well, they they can all be found from the per_cpu data for this CPU. Simples! Except... how does it know what its CPU# is? OK, we export the cpuid_to_apicid[] array and it can search it to find its APIC ID in there. But now you whine at me that it doesn't even know its APIC ID? Well, if it's a relatively modern CPU then the APIC ID is in CPUID leaf 0x0B so we can use that. Otherwise... erm... OK, otherwise it can't have parallel CPU bringup for now. We'll still use a global variable for those CPUs and bring them up one at a time. So add a global 'smpboot_control' field which either contains the APIC ID, or a flag indicating that it can be found in CPUID. This adds the 'do_parallel_bringup' flag in preparation but doesn't actually enable parallel bringup yet. [ dwmw2: Minor tweaks, write a commit message ] [ seanc: Fix stray override of initial_gs in common_cpu_up() ] Not-signed-off-by: Thomas Gleixner Signed-off-by: David Woodhouse Signed-off-by: Usama Arif Signed-off-by: Paul E. McKenney --- arch/x86/include/asm/realmode.h | 3 ++ arch/x86/include/asm/smp.h | 9 +++- arch/x86/kernel/acpi/sleep.c | 1 + arch/x86/kernel/apic/apic.c | 2 +- arch/x86/kernel/head_64.S | 75 ++++++++++++++++++++++++++++ arch/x86/kernel/smpboot.c | 34 +++++++++++-- arch/x86/realmode/init.c | 3 ++ arch/x86/realmode/rm/trampoline_64.S | 14 ++++++ kernel/smpboot.c | 2 +- 9 files changed, 136 insertions(+), 7 deletions(-) diff --git a/arch/x86/include/asm/realmode.h b/arch/x86/include/asm/realmode.h index a336feef0af1..f0357cfe2fb0 100644 --- a/arch/x86/include/asm/realmode.h +++ b/arch/x86/include/asm/realmode.h @@ -52,6 +52,7 @@ struct trampoline_header { u64 efer; u32 cr4; u32 flags; + u32 lock; #endif }; @@ -65,6 +66,8 @@ extern unsigned long initial_stack; extern unsigned long initial_vc_handler; #endif +extern u32 *trampoline_lock; + extern unsigned char real_mode_blob[]; extern unsigned char real_mode_relocs[]; diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h index b4dbb20dab1a..9b3c0cd35f5f 100644 --- a/arch/x86/include/asm/smp.h +++ b/arch/x86/include/asm/smp.h @@ -199,5 +199,12 @@ extern void nmi_selftest(void); #define nmi_selftest() do { } while (0) #endif -#endif /* __ASSEMBLY__ */ +extern unsigned int smpboot_control; + +#endif /* !__ASSEMBLY__ */ + +/* Control bits for startup_64 */ +#define STARTUP_PARALLEL 0x80000000 +#define STARTUP_SECONDARY 0x40000000 + #endif /* _ASM_X86_SMP_H */ diff --git a/arch/x86/kernel/acpi/sleep.c b/arch/x86/kernel/acpi/sleep.c index 3b7f4cdbf2e0..06adf340a0f1 100644 --- a/arch/x86/kernel/acpi/sleep.c +++ b/arch/x86/kernel/acpi/sleep.c @@ -115,6 +115,7 @@ int x86_acpi_suspend_lowlevel(void) early_gdt_descr.address = (unsigned long)get_cpu_gdt_rw(smp_processor_id()); initial_gs = per_cpu_offset(smp_processor_id()); + smpboot_control = 0; #endif initial_code = (unsigned long)wakeup_long64; saved_magic = 0x123456789abcdef0L; diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c index 20d9a604da7c..ac1d7e5da1f2 100644 --- a/arch/x86/kernel/apic/apic.c +++ b/arch/x86/kernel/apic/apic.c @@ -2377,7 +2377,7 @@ static int nr_logical_cpuids = 1; /* * Used to store mapping between logical CPU IDs and APIC IDs. */ -static int cpuid_to_apicid[] = { +int cpuid_to_apicid[] = { [0 ... NR_CPUS - 1] = -1, }; diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index 222efd4a09bc..d07f694691d2 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -25,6 +25,7 @@ #include #include #include +#include /* * We are not able to switch in one step to the final KERNEL ADDRESS SPACE @@ -241,6 +242,68 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) UNWIND_HINT_EMPTY ANNOTATE_NOENDBR // above +#ifdef CONFIG_SMP + /* + * Is this the boot CPU coming up? If so everything is available + * in initial_gs, initial_stack and early_gdt_descr. + */ + movl smpboot_control(%rip), %eax + testl %eax, %eax + jz .Lsetup_cpu + + /* + * Secondary CPUs find out the offsets via the APIC ID. For parallel + * boot the APIC ID is retrieved from CPUID, otherwise it's encoded + * in smpboot_control: + * Bit 0-29 APIC ID if STARTUP_PARALLEL flag is not set + * Bit 30 STARTUP_SECONDARY flag + * Bit 31 STARTUP_PARALLEL flag (use CPUID 0x0b for APIC ID) + */ + testl $STARTUP_PARALLEL, %eax + jnz .Luse_cpuid_0b + andl $0x0FFFFFFF, %eax + jmp .Lsetup_AP + +.Luse_cpuid_0b: + mov $0x0B, %eax + xorl %ecx, %ecx + cpuid + mov %edx, %eax + +.Lsetup_AP: + /* EAX contains the APICID of the current CPU */ + xorl %ecx, %ecx + leaq cpuid_to_apicid(%rip), %rbx + +.Lfind_cpunr: + cmpl (%rbx), %eax + jz .Linit_cpu_data + addq $4, %rbx + addq $8, %rcx + jmp .Lfind_cpunr + +.Linit_cpu_data: + /* Get the per cpu offset */ + leaq __per_cpu_offset(%rip), %rbx + addq %rcx, %rbx + movq (%rbx), %rbx + /* Save it for GS BASE setup */ + movq %rbx, initial_gs(%rip) + + /* Calculate the GDT address */ + movq $gdt_page, %rcx + addq %rbx, %rcx + movq %rcx, early_gdt_descr_base(%rip) + + /* Find the idle task stack */ + movq $idle_threads, %rcx + addq %rbx, %rcx + movq (%rcx), %rcx + movq TASK_threadsp(%rcx), %rcx + movq %rcx, initial_stack(%rip) +#endif /* CONFIG_SMP */ + +.Lsetup_cpu: /* * We must switch to a new descriptor in kernel space for the GDT * because soon the kernel won't have access anymore to the userspace @@ -281,6 +344,14 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) */ movq initial_stack(%rip), %rsp + /* Drop the realmode protection. For the boot CPU the pointer is NULL! */ + movq trampoline_lock(%rip), %rax + testq %rax, %rax + jz .Lsetup_idt + lock + btrl $0, (%rax) + +.Lsetup_idt: /* Setup and Load IDT */ pushq %rsi call early_setup_idt @@ -426,6 +497,7 @@ SYM_DATA(initial_vc_handler, .quad handle_vc_boot_ghcb) * reliably detect the end of the stack. */ SYM_DATA(initial_stack, .quad init_thread_union + THREAD_SIZE - FRAME_SIZE) +SYM_DATA(trampoline_lock, .quad 0); __FINITDATA __INIT @@ -660,6 +732,9 @@ SYM_DATA_END(level1_fixmap_pgt) SYM_DATA(early_gdt_descr, .word GDT_ENTRIES*8-1) SYM_DATA_LOCAL(early_gdt_descr_base, .quad INIT_PER_CPU_VAR(gdt_page)) + .align 16 +SYM_DATA(smpboot_control, .long 0) + .align 16 /* This must match the first entry in level2_kernel_pgt */ SYM_DATA(phys_base, .quad 0x0) diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index fdcf7c08945f..741da8d306a4 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -800,6 +800,16 @@ static int __init cpu_init_udelay(char *str) } early_param("cpu_init_udelay", cpu_init_udelay); +static bool do_parallel_bringup = true; + +static int __init no_parallel_bringup(char *str) +{ + do_parallel_bringup = false; + + return 0; +} +early_param("no_parallel_bringup", no_parallel_bringup); + static void __init smp_quirk_init_udelay(void) { /* if cmdline changed it from default, leave it alone */ @@ -1087,8 +1097,6 @@ int common_cpu_up(unsigned int cpu, struct task_struct *idle) #ifdef CONFIG_X86_32 /* Stack for startup_32 can be just as for start_secondary onwards */ per_cpu(pcpu_hot.top_of_stack, cpu) = task_top_of_stack(idle); -#else - initial_gs = per_cpu_offset(cpu); #endif return 0; } @@ -1113,9 +1121,16 @@ static int do_boot_cpu(int apicid, int cpu, struct task_struct *idle, start_ip = real_mode_header->trampoline_start64; #endif idle->thread.sp = (unsigned long)task_pt_regs(idle); - early_gdt_descr.address = (unsigned long)get_cpu_gdt_rw(cpu); initial_code = (unsigned long)start_secondary; - initial_stack = idle->thread.sp; + + if (IS_ENABLED(CONFIG_X86_32)) { + early_gdt_descr.address = (unsigned long)get_cpu_gdt_rw(cpu); + initial_stack = idle->thread.sp; + } else if (do_parallel_bringup) { + smpboot_control = STARTUP_SECONDARY | STARTUP_PARALLEL; + } else { + smpboot_control = STARTUP_SECONDARY | apicid; + } /* Enable the espfix hack for this CPU */ init_espfix_ap(cpu); @@ -1515,6 +1530,17 @@ void __init native_smp_prepare_cpus(unsigned int max_cpus) speculative_store_bypass_ht_init(); + /* + * We can do 64-bit AP bringup in parallel if the CPU reports its + * APIC ID in CPUID leaf 0x0B. Otherwise it's too hard. And not + * for SEV-ES guests because they can't use CPUID that early. + * Also, some AMD CPUs crash when doing parallel cpu bringup, disable + * it for all AMD CPUs to be on the safe side. + */ + if (IS_ENABLED(CONFIG_X86_32) || boot_cpu_data.cpuid_level < 0x0B || + cc_platform_has(CC_ATTR_GUEST_STATE_ENCRYPT)) + do_parallel_bringup = false; + snp_set_wakeup_secondary_cpu(); } diff --git a/arch/x86/realmode/init.c b/arch/x86/realmode/init.c index af565816d2ba..788e5559549f 100644 --- a/arch/x86/realmode/init.c +++ b/arch/x86/realmode/init.c @@ -154,6 +154,9 @@ static void __init setup_real_mode(void) trampoline_header->flags = 0; + trampoline_lock = &trampoline_header->lock; + *trampoline_lock = 0; + trampoline_pgd = (u64 *) __va(real_mode_header->trampoline_pgd); /* Map the real mode stub as virtual == physical */ diff --git a/arch/x86/realmode/rm/trampoline_64.S b/arch/x86/realmode/rm/trampoline_64.S index e38d61d6562e..49ebc1636ffd 100644 --- a/arch/x86/realmode/rm/trampoline_64.S +++ b/arch/x86/realmode/rm/trampoline_64.S @@ -49,6 +49,19 @@ SYM_CODE_START(trampoline_start) mov %ax, %es mov %ax, %ss + /* + * Make sure only one CPU fiddles with the realmode stack + */ +.Llock_rm: + btl $0, tr_lock + jnc 2f + pause + jmp .Llock_rm +2: + lock + btsl $0, tr_lock + jc .Llock_rm + # Setup stack movl $rm_stack_end, %esp @@ -241,6 +254,7 @@ SYM_DATA_START(trampoline_header) SYM_DATA(tr_efer, .space 8) SYM_DATA(tr_cr4, .space 4) SYM_DATA(tr_flags, .space 4) + SYM_DATA(tr_lock, .space 4) SYM_DATA_END(trampoline_header) #include "trampoline_common.S" diff --git a/kernel/smpboot.c b/kernel/smpboot.c index 2c7396da470c..a18a21dff9bc 100644 --- a/kernel/smpboot.c +++ b/kernel/smpboot.c @@ -25,7 +25,7 @@ * For the hotplug case we keep the task structs around and reuse * them. */ -static DEFINE_PER_CPU(struct task_struct *, idle_threads); +DEFINE_PER_CPU(struct task_struct *, idle_threads); struct task_struct *idle_thread_get(unsigned int cpu) { From patchwork Thu Feb 2 21:56:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 52189 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp494208wrn; Thu, 2 Feb 2023 13:59:02 -0800 (PST) X-Google-Smtp-Source: AK7set9BGXWupUsoswfmsimPeWor+XPoCwGgF/gkNehygKVjIzWPftwtd4jLIeATi9Yp1urmLLW/ X-Received: by 2002:a17:906:e09:b0:885:fb8a:7c3f with SMTP id l9-20020a1709060e0900b00885fb8a7c3fmr7593918eji.65.1675375142064; Thu, 02 Feb 2023 13:59:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675375142; cv=none; d=google.com; s=arc-20160816; b=RoTXuYWmoYTOfJurqv6QItJolm2ovkC2N4xa1I0IZNJM2uW8jvAscE2cnigPRFvaNk g5dWV2FRuqtbV+Wup73LTu95WxGM6ze5Q3jn4+gYfAchDND5y8RKD72tQKEBGZLXoy2D VD/zOp6GpIo91KKumNx0kD4CulIXV+jU7F8hujD9Dv+nrV+jNL3QiMtvezI8k9HCM7hK gMBgP281nnegfb4axX7q8fU+AJfxLuaCCZIzj2a20wQ7+RJVnSqi2oyWQAWtDyj9BaeM o1OdLiDzpjkK8BcaI7+SAyBIrsyifCwYVLWwPSQSqIJLtJqyPRaAIkXNPj0MFnxBvDlO q5CQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=NSUUSSxTZ02F+r1LTVXNULWrMNdIZEEBIKeOIraHNX8=; b=zjpkSchsw6UebCo+SW2whOkAMZ3o4DqgCMlLCef+jvrt3Q+nKluDt9cX99fd5R8pan pQDJvVHTZqQIOWuvYHcF9JsA5YGCp6ojQcIQPACzgie8CQlfMllb6hAMjB/uV+3YU5XK 6H4EPveWP2Y6SJJaH+TuAjIYSkcduaW8NEvbZElNuBXdliBOFOLNh9j79sRAPwTeWr5F uF/3zqexRZ6hS98I9d6NPPpmM3fZy1sFPhSdsjGy52R9BloZrBaIikVXr6opteTMo/j3 NsEAHu23AJKvjFJooYo20KzlUIU3VNFe9RlC61j7V44kQpPSfSKtpQUadVLzLGUEuBOy NSEA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=jWrmnXct; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id gc9-20020a1709072b0900b0087a3b3f01ccsi659958ejc.923.2023.02.02.13.58.37; Thu, 02 Feb 2023 13:59:02 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=jWrmnXct; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233274AbjBBV5m (ORCPT + 99 others); Thu, 2 Feb 2023 16:57:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54880 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233237AbjBBV5S (ORCPT ); Thu, 2 Feb 2023 16:57:18 -0500 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0A79A728D5 for ; Thu, 2 Feb 2023 13:56:38 -0800 (PST) Received: by mail-wr1-x42e.google.com with SMTP id q5so3042154wrv.0 for ; Thu, 02 Feb 2023 13:56:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=NSUUSSxTZ02F+r1LTVXNULWrMNdIZEEBIKeOIraHNX8=; b=jWrmnXctFoGJa9Wp7ZeU4q2i+NjqAJ2dYC96bkDbNz/V5jBzNy4cOuVK5EOPrnZJyq /vpwLO/s8e9/wVeW8Shg8jS//TmAwxBJVWnJJERRHdtvez1G1qiZ06wqhIYPy5MPxsSh 8Vb/DlK4m4XAqJokCHT4pGnu7NU/zeYRJsgzo6YI6RkYQqHTnluKdmWVlUNAvqYCG/lx XbgH1M3MTSsxPWLBlAh6scMY/xpQ9kQph0tL23tJYJQUphfb617YuhPngBrGVzw5SWak msODHrjIlagSjaREHtgTxusON6vTobYmR2bISUHElyoTNfwIjvzehB94J43pAdYcCSKS XpFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NSUUSSxTZ02F+r1LTVXNULWrMNdIZEEBIKeOIraHNX8=; b=aGFEfyXRBqPN8E6XZTTZAnH1iPp+KdiJIU+a70A0IJDoYA2jkg1zAMUcvgtk8fKy5S Ihk8K0lshnY6IJZWAJkO88dzUt+UeGad7Uo6WBjdQytXymlbvSHw/muPwX4/XLnMb3UP 3x1Fv/7xKpUwHGGD4N6/dGB55ETVEtanVFTBuymGpqyrskQRzS+qY9zN+OBwCE9ZR3HU G37OJrgkp24qKT/QcSKlMOsHy2nTDlRrB6F2zz696iPn+CF+EQAexWBGGuFVyXh1TeSs laEYDZ3RRjZ5zGej/wBXjjI79gTK35cZIrY/jCSV2PJr3Sex0arG0ySPv/xA2g6ui+24 GIhA== X-Gm-Message-State: AO0yUKW6nukoe2DjZ82Lhaqos2wss0uTIP9kfGOImNkk/jCDxp7l08QG QhreMaRjXSvkVaYJzzxkoINmSg== X-Received: by 2002:a5d:6da3:0:b0:2bf:ae11:c40c with SMTP id u3-20020a5d6da3000000b002bfae11c40cmr8326498wrs.32.1675374996905; Thu, 02 Feb 2023 13:56:36 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:98fe:e4ee:fc7e:cd71]) by smtp.gmail.com with ESMTPSA id e8-20020a5d6d08000000b00297dcfdc90fsm506078wrq.24.2023.02.02.13.56.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Feb 2023 13:56:36 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, arjan@linux.intel.com Cc: mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse Subject: [PATCH v6 07/11] x86/smpboot: Disable parallel boot for AMD CPUs Date: Thu, 2 Feb 2023 21:56:21 +0000 Message-Id: <20230202215625.3248306-8-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230202215625.3248306-1-usama.arif@bytedance.com> References: <20230202215625.3248306-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756758165078612241?= X-GMAIL-MSGID: =?utf-8?q?1756758165078612241?= From: David Woodhouse Signed-off-by: David Woodhouse --- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/kernel/cpu/amd.c | 11 +++++++++++ arch/x86/kernel/smpboot.c | 13 +++++++++++-- 3 files changed, 23 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index 61012476d66e..ed7f32354edc 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -466,5 +466,6 @@ #define X86_BUG_MMIO_UNKNOWN X86_BUG(26) /* CPU is too old and its MMIO Stale Data status is unknown */ #define X86_BUG_RETBLEED X86_BUG(27) /* CPU is affected by RETBleed */ #define X86_BUG_EIBRS_PBRSB X86_BUG(28) /* EIBRS is vulnerable to Post Barrier RSB Predictions */ +#define X86_BUG_NO_PARALLEL_BRINGUP X86_BUG(29) /* CPU has hardware issues with parallel AP bringup */ #endif /* _ASM_X86_CPUFEATURES_H */ diff --git a/arch/x86/kernel/cpu/amd.c b/arch/x86/kernel/cpu/amd.c index f769d6d08b43..19b5c8342d7e 100644 --- a/arch/x86/kernel/cpu/amd.c +++ b/arch/x86/kernel/cpu/amd.c @@ -941,6 +941,17 @@ static void init_amd(struct cpuinfo_x86 *c) case 0x19: init_amd_zn(c); break; } + /* + * Various AMD CPUs appear to not to cope with APs being brought up + * in parallel. In debugging, the AP doesn't even seem to reach an + * outb to port 0x3f8 right at the top of the startup trampoline. + * We don't *think* there are any remaining software issues which + * may contribute to this, although it's possible. So far, attempts + * to get AMD to investigate this have been to no avail. So just + * disable parallel bring up for all AMD CPUs for now. + */ + set_cpu_bug(c, X86_BUG_NO_PARALLEL_BRINGUP); + /* * Enable workaround for FXSAVE leak on CPUs * without a XSaveErPtr feature diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 741da8d306a4..656897b055f5 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -1534,13 +1534,22 @@ void __init native_smp_prepare_cpus(unsigned int max_cpus) * We can do 64-bit AP bringup in parallel if the CPU reports its * APIC ID in CPUID leaf 0x0B. Otherwise it's too hard. And not * for SEV-ES guests because they can't use CPUID that early. - * Also, some AMD CPUs crash when doing parallel cpu bringup, disable - * it for all AMD CPUs to be on the safe side. */ if (IS_ENABLED(CONFIG_X86_32) || boot_cpu_data.cpuid_level < 0x0B || cc_platform_has(CC_ATTR_GUEST_STATE_ENCRYPT)) do_parallel_bringup = false; + /* + * Various AMD CPUs appear not to cope with APs being brought up + * in parallel, so just disable parallel bring up for all AMD CPUs + * for now. + */ + if (do_parallel_bringup && + boot_cpu_has_bug(X86_BUG_NO_PARALLEL_BRINGUP)) { + pr_info("Disabling parallel bringup due to CPU bugs\n"); + do_parallel_bringup = false; + } + snp_set_wakeup_secondary_cpu(); } From patchwork Thu Feb 2 21:56:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 52192 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp494507wrn; Thu, 2 Feb 2023 13:59:51 -0800 (PST) X-Google-Smtp-Source: AK7set9C1mnV0g7ezEyemkkmDh0DFbkG9nIQRS5Ny0lBtVWV5+BI4a8U2HkMtc+FS0cyK0ZOAhuG X-Received: by 2002:a05:6402:3488:b0:494:fae3:c0df with SMTP id v8-20020a056402348800b00494fae3c0dfmr9269989edc.12.1675375191136; Thu, 02 Feb 2023 13:59:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675375191; cv=none; d=google.com; s=arc-20160816; b=I1eiqJAHUfjlnaWm0lKhP2pO2uY9FKIbJU3SjXey/s0f8LzZOR2X5cc4MoQ07nimYD QJJou44iVs/TAtw+E35mBU3lcrWaa3N4zFilrtrt4UX3cj8S3Yfq6+F5IPJPJFHYVLns c6QAB8gTsFiQxxajknBSKSmK7p/Y9Ci5k9DdLkD4AsFynXVztocHKQWN1h05W/zdbt78 6Wu7AToPmoNCpR6BpSt2VfQxuehbR9GpFnCGEbJeroIpZXeMqZE34KLvH01NLkjdvn1J vVif/2uZixy8UZC5cG/LJUQ6L6E8M50H+p6srgpU8FP9FO/B+HqWfejqZEQGJtCFxoYB gB/w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=1hr9lEiKrK81ARA2M0cI5BiENvgrY9Zt5BHSS6rpobc=; b=XoKWRD3i41gpN86qcwFyWxh1el2VKCJETv6B9SD4LqXDyD5vhtR6QyYEB9PW/0EmwJ CBgpzlqFfy+FULLmgt0h8/LPRksaqq2PfQHnHKj7o9E4OD+e1K8hdRc1y5xCb6CdE1mv qRA5Ou3zMrGHjWrBnoveFS6OqV4iz73ht9RXtuC5FUmsiYBZOdVX8rM9m6iHRKDmMy1v mYvNxexKcSvsuKERfjodDgg5r/pgAj9q+ntSKL3q/3FQk+Yx8cMqklBzoRVjgaj3MzSI uJcGTBwJEx5sGBovxy5CRMSJIr0KDw8PxEhH7N0l54JfwRBJ3wqSD5jFY0HomAjGH9uM 2w0w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=u2nq3Ej9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id m2-20020a056402430200b004a2343e90c4si828386edc.380.2023.02.02.13.59.26; Thu, 02 Feb 2023 13:59:51 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=u2nq3Ej9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233221AbjBBV5P (ORCPT + 99 others); Thu, 2 Feb 2023 16:57:15 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54788 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233136AbjBBV4y (ORCPT ); Thu, 2 Feb 2023 16:56:54 -0500 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9CCC6761F3 for ; Thu, 2 Feb 2023 13:56:39 -0800 (PST) Received: by mail-wr1-x436.google.com with SMTP id m14so2960456wrg.13 for ; Thu, 02 Feb 2023 13:56:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1hr9lEiKrK81ARA2M0cI5BiENvgrY9Zt5BHSS6rpobc=; b=u2nq3Ej9mftRp/AibuMcN8opF0/YxTfubYLNVe5VnD252/9mQ6PWzMHa5Wy+zTi5xp CIi3VzipOQ3cUPBGWAYmpslvBtttDPw0YSfsxwwOb8vHCqZjVNxF0VdoPIcVsCTVro9g AguJPXSsPtWzOCPAOcncRsin0t0J5YZ7QOZe8pW0y6LDl7mODkAk59UCuvCsC+MP5sqy q+DG2ZCDTfCUwcSBIxz9PsT6w0i57kcB3wXmbmXODr7XQ/LQ3stW00g1uJPRvgJxl05E tMfTmxwCQSL1qvW3qdy+19PzKFX9DfoV4xwkbCCP0MVT9BliY8KGfNVy4aBXuTJwfNjG w2Iw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1hr9lEiKrK81ARA2M0cI5BiENvgrY9Zt5BHSS6rpobc=; b=GZGAtTDpJaCedREcueohuyIvrwBDj+MCOKGzCVWSdMdrg7WsjixoWUArQDn6L0rrqH c5D9ExVX3Ps8SwIs/CJpSm3ZaryUvOe/w+ivMNuOOyjqxqxJ6I15A13NIL3e03PQ8Vbi BHnagMGtb9t+o+TtJJQE7B2SVvsOP2TSqKwNwMuz6Ycb2DOiKSRC2ELehJ/FZ4JwZi/q RQKSOioW0KBsSrYnAIlElXvDHcoTdLM1rnl2NydTN9+VSP19dGCqCiAelJqQaQ7liRgy f23ozNjnVOWpfD7HdI5fvmXbEwuW5CGE8IuODaoYVBHO+ysps8CE945mGILoZivNjQni JCWQ== X-Gm-Message-State: AO0yUKXa+KbbBlKK3CAthD1CilVyx3rp0b77k+BLZFs5KSZC3Rnfji+S JqfMj2NHRh6TB9zNMTxpi7ECsg== X-Received: by 2002:adf:fa84:0:b0:2bf:ae15:ac15 with SMTP id h4-20020adffa84000000b002bfae15ac15mr6639032wrr.62.1675374998131; Thu, 02 Feb 2023 13:56:38 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:98fe:e4ee:fc7e:cd71]) by smtp.gmail.com with ESMTPSA id e8-20020a5d6d08000000b00297dcfdc90fsm506078wrq.24.2023.02.02.13.56.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Feb 2023 13:56:37 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, arjan@linux.intel.com Cc: mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse , Usama Arif Subject: [PATCH v6 08/11] x86/smpboot: Send INIT/SIPI/SIPI to secondary CPUs in parallel Date: Thu, 2 Feb 2023 21:56:22 +0000 Message-Id: <20230202215625.3248306-9-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230202215625.3248306-1-usama.arif@bytedance.com> References: <20230202215625.3248306-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756758216793564357?= X-GMAIL-MSGID: =?utf-8?q?1756758216793564357?= From: David Woodhouse When the APs can find their own APIC ID without assistance, we can do the AP bringup in parallel. Register a CPUHP_BP_PARALLEL_DYN stage "x86/cpu:kick" which just calls do_boot_cpu() to deliver INIT/SIPI/SIPI to each AP in turn before the normal native_cpu_up() does the rest of the hand-holding. The APs will then take turns through the real mode code (which has its own bitlock for exclusion) until they make it to their own stack, then proceed through the first few lines of start_secondary() and execute these parts in parallel: start_secondary() -> cr4_init() -> (some 32-bit only stuff so not in the parallel cases) -> cpu_init_secondary() -> cpu_init_exception_handling() -> cpu_init() -> wait_for_master_cpu() At this point they wait for the BSP to set their bit in cpu_callout_mask (from do_wait_cpu_initialized()), and release them to continue through the rest of cpu_init() and beyond. This reduces the time taken for bringup on my 28-thread Haswell system from about 120ms to 80ms. On a socket 96-thread Skylake it takes the bringup time from 500ms to 100ms. There is more speedup to be had by doing the remaining parts in parallel too — especially notify_cpu_starting() in which the AP takes itself through all the stages from CPUHP_BRINGUP_CPU to CPUHP_ONLINE. But those require careful auditing to ensure they are reentrant, before we can go that far. [Usama Arif: fixed rebase conflict] Signed-off-by: David Woodhouse Signed-off-by: Usama Arif Signed-off-by: Paul E. McKenney --- arch/x86/kernel/smpboot.c | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 656897b055f5..77c509e95571 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -57,6 +57,7 @@ #include #include #include +#include #include #include @@ -1330,9 +1331,12 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) { int ret; - ret = do_cpu_up(cpu, tidle); - if (ret) - return ret; + /* If parallel AP bringup isn't enabled, perform the first steps now. */ + if (!do_parallel_bringup) { + ret = do_cpu_up(cpu, tidle); + if (ret) + return ret; + } ret = do_wait_cpu_initialized(cpu); if (ret) @@ -1354,6 +1358,12 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) return ret; } +/* Bringup step one: Send INIT/SIPI to the target AP */ +static int native_cpu_kick(unsigned int cpu) +{ + return do_cpu_up(cpu, idle_thread_get(cpu)); +} + /** * arch_disable_smp_support() - disables SMP support for x86 at runtime */ @@ -1550,6 +1560,10 @@ void __init native_smp_prepare_cpus(unsigned int max_cpus) do_parallel_bringup = false; } + if (do_parallel_bringup) + cpuhp_setup_state_nocalls(CPUHP_BP_PARALLEL_DYN, "x86/cpu:kick", + native_cpu_kick, NULL); + snp_set_wakeup_secondary_cpu(); } From patchwork Thu Feb 2 21:56:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 52190 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp494322wrn; Thu, 2 Feb 2023 13:59:19 -0800 (PST) X-Google-Smtp-Source: AK7set/YIGGVII3JqH+nMwrCgbYJNKIMTBzKACRHdHrksUP7HWZzVF/vCykhU1T3w6IEdg0x+wDD X-Received: by 2002:a17:906:dfda:b0:878:545b:e540 with SMTP id jt26-20020a170906dfda00b00878545be540mr7831062ejc.51.1675375159324; Thu, 02 Feb 2023 13:59:19 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675375159; cv=none; d=google.com; s=arc-20160816; b=fJi0sPidityNimpFUO1/cWoLz18SB3hWpEE13PD7LES5jEbO69M5NZZdVWq5o6NHgs 26pRDDmO1LVqXNoLFeIwOeO4n2mDxMxITgmFJIUWwNTDErTYlFJfXfFlXkiJNP5TeDle C5TwsLwM2/wtpfYvPrf5NCdmysJWoS/XX20rfWE2j68NIksjLoMw1iaS7tqXmMP2geei p2qsbTSd+v833lkQ0ck7OutXMM7dp9HleRBx3ezAgTZlPxD2LlemDjADh4qGY5p28VBx zSOjz5Jr5TMDePYS7yrCpI8+SyLwMAvzWuKkGjH+gPs2jAFG9xEDRrxm59eCOFhw96n7 sF+w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=vY0vlTeJqZKE/bujXYv8BkLNeLLbxRNo+nzLcSQF4Yk=; b=LTTEvNPtEhUQAMnM3m6fuvs6FqzI3X8QcbV9sycZp9uWg+kv1OFWFckFTDb0E4gwse b7/CuEuM22ys79P867GpChS/5vKdG3U5eX/ZPLIqLUAR6mFJwbCPbNF+ip5ynSxiUPXr i/qv3GgrDNSsPNQwOFxokJPnoSeBoXNP5BEajw+iTprwWuMXdH/LKn+RvPXuyiEPMDfN 1aHy3VWfNxwn0y9JZ0sv7DYkrk9Z54w+4jHAbHTYkA1Z9u47H/Qgh/p4n2i4+2afrN6F xaYHD7tp7hDvrho2oQl3YnF/Ulv41tC33k8nGx5XrsFIuAj7qOSxss7isX2cWmxnzPpQ bwtQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=mT6B1rsh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 26-20020a170906101a00b0085a483a6fd9si1003897ejm.212.2023.02.02.13.58.54; Thu, 02 Feb 2023 13:59:19 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=mT6B1rsh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233244AbjBBV5v (ORCPT + 99 others); Thu, 2 Feb 2023 16:57:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54946 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231302AbjBBV5T (ORCPT ); Thu, 2 Feb 2023 16:57:19 -0500 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AE24474C0C for ; Thu, 2 Feb 2023 13:56:40 -0800 (PST) Received: by mail-wr1-x434.google.com with SMTP id y1so3012851wru.2 for ; Thu, 02 Feb 2023 13:56:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vY0vlTeJqZKE/bujXYv8BkLNeLLbxRNo+nzLcSQF4Yk=; b=mT6B1rshBPNb/jZfy+ezLKF9F1DIij+0IVLWcPLJlhIuz9rWYW2KEjB1Y7FU0/wz+G 5shof2tQQ1/OeP0PXIEdn+vb45K+y/ZQrIRGyltWRu+W1SbfKZevi8GwRtF2Qqa4MjMB CF604H0RNfdRfoiES6rIq+gBMd40gYDdx0mdoYb52uCQklmOvJi+ZhCc28hUV0VtChrG JMWhH3EZUAkdoBcl+TiGFjVPHAKepRIphktMOO7i9F1crxCsP4nV4wctgZXLbfz+Gqhk BBnBIdbmEtUJurgh33VP9Wjgw42VOF2cDjHx3VbxEkayBjojfkBAayE+dQ1DNOzSKdlB Gk+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vY0vlTeJqZKE/bujXYv8BkLNeLLbxRNo+nzLcSQF4Yk=; b=w0v3ZpTkEi/irSwPYAf0cbBjxMrA6NeZylxdGVAcOvMwpNeGkNkTRX1i+yhfCK++8a 7LUVAA0ArcDKlDk5RpooGuJqflSCps7tJ19LwvxBYaCQKUYz/PIx0WPJ/DiuGhDbdXkn wvyyq56a4wASzJzqKYbsl0AS+IKBnEBTsyadC+c5MkKqVYiPqdqCl0uqMdymHvS8etoy PDU30CA5JeQuXj02mFs99a4QioCfbumUVHaIKKv95Op2iTfW9kivrf5HZvI7XIXmqW4T x44FPM5xTbl9I5rAdCBeE7fsyVjkiiUhfti07qD0IJ852m26LrqOYLPRCSGSvq1XF2qq caYg== X-Gm-Message-State: AO0yUKWBhtAz18adGIvm+7H42DqDo8DrwO0opfhZ0ss7WImFmiU4Qjmq WepqpBxf/MJbAAa/TePdEn+clA== X-Received: by 2002:a5d:560b:0:b0:2bf:cd9f:37f9 with SMTP id l11-20020a5d560b000000b002bfcd9f37f9mr6581774wrv.4.1675374999213; Thu, 02 Feb 2023 13:56:39 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:98fe:e4ee:fc7e:cd71]) by smtp.gmail.com with ESMTPSA id e8-20020a5d6d08000000b00297dcfdc90fsm506078wrq.24.2023.02.02.13.56.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Feb 2023 13:56:38 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, arjan@linux.intel.com Cc: mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse Subject: [PATCH v6 09/11] x86/mtrr: Avoid repeated save of MTRRs on boot-time CPU bringup Date: Thu, 2 Feb 2023 21:56:23 +0000 Message-Id: <20230202215625.3248306-10-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230202215625.3248306-1-usama.arif@bytedance.com> References: <20230202215625.3248306-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756758183410386054?= X-GMAIL-MSGID: =?utf-8?q?1756758183410386054?= From: David Woodhouse There's no need to repeatedly save the BSP's MTRRs for each AP we bring up at boot time. And there's no need to use smp_call_function_single() even for the one time we *do* want to do it. Signed-off-by: David Woodhouse Signed-off-by: Paul E. McKenney --- arch/x86/kernel/cpu/mtrr/mtrr.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/arch/x86/kernel/cpu/mtrr/mtrr.c b/arch/x86/kernel/cpu/mtrr/mtrr.c index 783f3210d582..b6eae3ad4414 100644 --- a/arch/x86/kernel/cpu/mtrr/mtrr.c +++ b/arch/x86/kernel/cpu/mtrr/mtrr.c @@ -721,11 +721,20 @@ void __init mtrr_bp_init(void) */ void mtrr_save_state(void) { + static bool mtrr_saved; int first_cpu; if (!mtrr_enabled()) return; + if (system_state < SYSTEM_RUNNING) { + if (!mtrr_saved) { + mtrr_save_fixed_ranges(NULL); + mtrr_saved = true; + } + return; + } + first_cpu = cpumask_first(cpu_online_mask); smp_call_function_single(first_cpu, mtrr_save_fixed_ranges, NULL, 1); } From patchwork Thu Feb 2 21:56:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 52186 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp494057wrn; Thu, 2 Feb 2023 13:58:33 -0800 (PST) X-Google-Smtp-Source: AK7set/hcT6HjeICv1LYaQN4U6KtAloOmoznLS1FrlMYhU8CZ1ERhb1KnZ24oX7KAvbWHSxG2MXn X-Received: by 2002:a17:906:a397:b0:886:3eb:5164 with SMTP id k23-20020a170906a39700b0088603eb5164mr8021550ejz.9.1675375113367; Thu, 02 Feb 2023 13:58:33 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675375113; cv=none; d=google.com; s=arc-20160816; b=hGjQukd7Dr5bQ5ysub8wFm11B7Cv9onOFbFZMRBpxge+6RXTPD8fsgpQSHvaosietp LOXTF8si9XP+IQqKjMZKRc6qTsk7clTHmZsuZuPbKTD9BKcXvV/Jgwe1i+P/vVxlCUYl 7dX8xriomfSu1LIuifzYXEfxsV1NFC+foNwHS7i1P76ey1Ws5YgAYlgkBChAZpSwNSgP qm9AKCYDbrUMi3ZWLnsDyMaKw0TAcg0v0UBjQ3QhQ+mMe2JNox7gGCCOfruuDUfP1ibr 32UDdVxEKCeeUx2I0r91p7eqMTE1D8HdUQivzKdwyFsChQxNAFsRQg9e2qSQibs3zoBO /VaQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=FZK9BDPDI9bGJd7FPVsAmZEBy79uPnOwINWAWfdQhbY=; b=pX9jOytNyyu5C8hVUBOiLoXPWsch9s8Nh11+yLM4/xh18HyUUQOe/H73NLZqSZALAm gq0f9BXm+JUMc/DFL+bvBzEI1Jlwz/LDtVDICOskfmebvrOBpvNZEurxQIbFGAltzmI4 icUHfKOUiBq7tBBheUg93KrQpVDjElt/70bj5vq3h4lzZSNxynUJJIq+5xaDJ5PsyT08 4a5f7Wf2S+oJKFIHe1cPwi1ZRaO1mX9ZtnUwBUz7NAgy47d0ESEogAqMIj6JUieIU+7M Irz1R/5vhUpF5c8S7SsWLWVhYZpN2kNiaThmgtlzgNmeQmysyQda7/AEQpIQIVi5BliD BQCw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=FoyWNwp5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id vz15-20020a17090704cf00b00889794debc4si823128ejb.775.2023.02.02.13.58.09; Thu, 02 Feb 2023 13:58:33 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=FoyWNwp5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233202AbjBBV5J (ORCPT + 99 others); Thu, 2 Feb 2023 16:57:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55022 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233166AbjBBV4y (ORCPT ); Thu, 2 Feb 2023 16:56:54 -0500 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1904579F30 for ; Thu, 2 Feb 2023 13:56:41 -0800 (PST) Received: by mail-wr1-x434.google.com with SMTP id q5so3042287wrv.0 for ; Thu, 02 Feb 2023 13:56:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FZK9BDPDI9bGJd7FPVsAmZEBy79uPnOwINWAWfdQhbY=; b=FoyWNwp5I/MeCiG6pimyzPx3iy+ufYoh4QnOy3JD/32jlnZYSLVfk0ARdpYczcHN8x i0ncBeAV6j1s8YHGke0eqlYykvsEf4zj2Xe5bPG6U4Q27t6Xcx8JyyZwwVX9t5MsNK6q vqj2DxXD5TQFXkVl5L5i3kl9X8xMemqiStoFicrsrAUS6vSDjBkqneQbpp6Uu7e9Uxyk KAN8qWHeFWxcKRtr2hb0clsc8MbeXAMo0o9/SInfR48f0fqnX9OiZyfUbASgU8eRAT/b KSGgxiBLBqQ23jcWPd+yOrlusMuj8+9SlGVTp+4ln7qf6g/Ft6IzxeseQmoyhOwkDxo6 A8NA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FZK9BDPDI9bGJd7FPVsAmZEBy79uPnOwINWAWfdQhbY=; b=a0eTH8M8rD6aDTSgRU9GnW2fDSfPI+Xk0jrdkTul0gwOoqUh0ZhvknsDoxJPrU3TXI C62qoIBw4nba1i1HttHdrohiLfR0UXh+hoScRGK+H+i+Iq/rsp10EFUFWaWlBwFaM+uq Ru5HsFUAQBRLeS1E+7uEnd4O6/c9+JPrTOku9sXDNmnXjtiIHFiDfoR+n1TSeCH1KtKU 5EAXJ+RWHeb7gCZFFylPoyA1epf3IRDZ29JGmZQzVr6mnmHD9hoPG48Km+4RikG808Fy 5TS9P20z3iKWRL47fBz+TlqFsW4/6KavQnWvBbuWfgY97HvWSQLRPrph91HgDcljVBS6 GLzA== X-Gm-Message-State: AO0yUKXn4BoVif+CV3ygCl4BMWlT7QbQp+I9Ju01Wfy1bVupqjoF5Fbq 40n+Qv24ddIYVLG8PK5TAg51dA== X-Received: by 2002:a5d:6549:0:b0:242:2844:8489 with SMTP id z9-20020a5d6549000000b0024228448489mr8145144wrv.8.1675375000211; Thu, 02 Feb 2023 13:56:40 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:98fe:e4ee:fc7e:cd71]) by smtp.gmail.com with ESMTPSA id e8-20020a5d6d08000000b00297dcfdc90fsm506078wrq.24.2023.02.02.13.56.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Feb 2023 13:56:39 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, arjan@linux.intel.com Cc: mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse , Usama Arif Subject: [PATCH v6 10/11] x86/smpboot: Serialize topology updates for secondary bringup Date: Thu, 2 Feb 2023 21:56:24 +0000 Message-Id: <20230202215625.3248306-11-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230202215625.3248306-1-usama.arif@bytedance.com> References: <20230202215625.3248306-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756758134934026442?= X-GMAIL-MSGID: =?utf-8?q?1756758134934026442?= From: David Woodhouse If we bring up secondaries in parallel they might get confused unless we impose some ordering here: [ 1.360149] x86: Booting SMP configuration: [ 1.360221] .... node #0, CPUs: #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15 #16 #17 #18 #19 #20 #21 #22 #23 [ 1.366225] .... node #1, CPUs: #24 #25 #26 #27 #28 #29 #30 #31 #32 #33 #34 #35 #36 #37 #38 #39 #40 #41 #42 #43 #44 #45 #46 #47 [ 1.370219] .... node #0, CPUs: #48 #49 #50 #51 #52 #53 #54 #55 #56 #57 #58 #59 #60 #61 #62 #63 #64 #65 #66 #67 #68 #69 #70 #71 [ 1.378226] .... node #1, CPUs: #72 #73 #74 #75 #76 #77 #78 #79 #80 #81 #82 #83 #84 #85 #86 #87 #88 #89 #90 #91 #92 #93 #94 #95 [ 1.382037] Brought 96 CPUs to x86/cpu:kick in 72232606 cycles [ 0.104104] smpboot: CPU 26 Converting physical 0 to logical die 1 [ 0.104104] smpboot: CPU 27 Converting physical 1 to logical package 2 [ 0.104104] smpboot: CPU 24 Converting physical 1 to logical package 3 [ 0.104104] smpboot: CPU 27 Converting physical 0 to logical die 2 [ 0.104104] smpboot: CPU 25 Converting physical 1 to logical package 4 [ 1.385609] Brought 96 CPUs to x86/cpu:wait-init in 9269218 cycles [ 1.395285] Brought CPUs online in 28930764 cycles [ 1.395469] smp: Brought up 2 nodes, 96 CPUs [ 1.395689] smpboot: Max logical packages: 2 [ 1.396222] smpboot: Total of 96 processors activated (576000.00 BogoMIPS) Do the full topology update in smp_store_cpu_info() under a spinlock to ensure that things remain consistent. [Usama Arif: fixed rebase conflict] Signed-off-by: David Woodhouse Signed-off-by: Usama Arif Signed-off-by: Paul E. McKenney --- arch/x86/include/asm/smp.h | 4 +- arch/x86/include/asm/topology.h | 2 - arch/x86/kernel/cpu/common.c | 6 +-- arch/x86/kernel/smpboot.c | 73 ++++++++++++++++++++------------- arch/x86/xen/smp_pv.c | 4 +- 5 files changed, 48 insertions(+), 41 deletions(-) diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h index 9b3c0cd35f5f..1760aa269673 100644 --- a/arch/x86/include/asm/smp.h +++ b/arch/x86/include/asm/smp.h @@ -48,8 +48,6 @@ struct smp_ops { }; /* Globals due to paravirt */ -extern void set_cpu_sibling_map(int cpu); - #ifdef CONFIG_SMP extern struct smp_ops smp_ops; @@ -137,7 +135,7 @@ void native_send_call_func_single_ipi(int cpu); void x86_idle_thread_init(unsigned int cpu, struct task_struct *idle); void smp_store_boot_cpu_info(void); -void smp_store_cpu_info(int id); +void smp_store_cpu_info(int id, bool force_single_core); asmlinkage __visible void smp_reboot_interrupt(void); __visible void smp_reschedule_interrupt(struct pt_regs *regs); diff --git a/arch/x86/include/asm/topology.h b/arch/x86/include/asm/topology.h index 458c891a8273..4bccbd949a99 100644 --- a/arch/x86/include/asm/topology.h +++ b/arch/x86/include/asm/topology.h @@ -136,8 +136,6 @@ static inline int topology_max_smt_threads(void) return __max_smt_threads; } -int topology_update_package_map(unsigned int apicid, unsigned int cpu); -int topology_update_die_map(unsigned int dieid, unsigned int cpu); int topology_phys_to_logical_pkg(unsigned int pkg); int topology_phys_to_logical_die(unsigned int die, unsigned int cpu); bool topology_is_primary_thread(unsigned int cpu); diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 9cfca3d7d0e2..cf1a4eff7a76 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -1766,7 +1766,7 @@ static void generic_identify(struct cpuinfo_x86 *c) * Validate that ACPI/mptables have the same information about the * effective APIC id and update the package map. */ -static void validate_apic_and_package_id(struct cpuinfo_x86 *c) +static void validate_apic_id(struct cpuinfo_x86 *c) { #ifdef CONFIG_SMP unsigned int apicid, cpu = smp_processor_id(); @@ -1777,8 +1777,6 @@ static void validate_apic_and_package_id(struct cpuinfo_x86 *c) pr_err(FW_BUG "CPU%u: APIC id mismatch. Firmware: %x APIC: %x\n", cpu, apicid, c->initial_apicid); } - BUG_ON(topology_update_package_map(c->phys_proc_id, cpu)); - BUG_ON(topology_update_die_map(c->cpu_die_id, cpu)); #else c->logical_proc_id = 0; #endif @@ -1969,7 +1967,7 @@ void identify_secondary_cpu(struct cpuinfo_x86 *c) #ifdef CONFIG_X86_32 enable_sep_cpu(); #endif - validate_apic_and_package_id(c); + validate_apic_id(c); x86_spec_ctrl_setup_ap(); update_srbds_msr(); diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 77c509e95571..030eaa84aea0 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -182,16 +182,12 @@ static void smp_callin(void) apic_ap_setup(); /* - * Save our processor parameters. Note: this information - * is needed for clock calibration. - */ - smp_store_cpu_info(cpuid); - - /* + * Save our processor parameters and update topology. + * Note: this information is needed for clock calibration. * The topology information must be up to date before * calibrate_delay() and notify_cpu_starting(). */ - set_cpu_sibling_map(raw_smp_processor_id()); + smp_store_cpu_info(cpuid, false); ap_init_aperfmperf(); @@ -246,6 +242,12 @@ static void notrace start_secondary(void *unused) * smp_callout_mask to release them. */ cpu_init_secondary(); + + /* + * Even though notify_cpu_starting() will do this, it does so too late + * as the AP may already have triggered lockdep splats by then. See + * commit 29368e093 ("x86/smpboot: Move rcu_cpu_starting() earlier"). + */ rcu_cpu_starting(raw_smp_processor_id()); x86_cpuinit.early_percpu_clock_init(); @@ -354,7 +356,7 @@ EXPORT_SYMBOL(topology_phys_to_logical_die); * @pkg: The physical package id as retrieved via CPUID * @cpu: The cpu for which this is updated */ -int topology_update_package_map(unsigned int pkg, unsigned int cpu) +static int topology_update_package_map(unsigned int pkg, unsigned int cpu) { int new; @@ -377,7 +379,7 @@ int topology_update_package_map(unsigned int pkg, unsigned int cpu) * @die: The die id as retrieved via CPUID * @cpu: The cpu for which this is updated */ -int topology_update_die_map(unsigned int die, unsigned int cpu) +static int topology_update_die_map(unsigned int die, unsigned int cpu) { int new; @@ -408,25 +410,7 @@ void __init smp_store_boot_cpu_info(void) c->initialized = true; } -/* - * The bootstrap kernel entry code has set these up. Save them for - * a given CPU - */ -void smp_store_cpu_info(int id) -{ - struct cpuinfo_x86 *c = &cpu_data(id); - - /* Copy boot_cpu_data only on the first bringup */ - if (!c->initialized) - *c = boot_cpu_data; - c->cpu_index = id; - /* - * During boot time, CPU0 has this setup already. Save the info when - * bringing up AP or offlined CPU0. - */ - identify_secondary_cpu(c); - c->initialized = true; -} +static arch_spinlock_t topology_lock = __ARCH_SPIN_LOCK_UNLOCKED; static bool topology_same_node(struct cpuinfo_x86 *c, struct cpuinfo_x86 *o) @@ -632,7 +616,7 @@ static struct sched_domain_topology_level x86_topology[] = { */ static bool x86_has_numa_in_package; -void set_cpu_sibling_map(int cpu) +static void set_cpu_sibling_map(int cpu) { bool has_smt = smp_num_siblings > 1; bool has_mp = has_smt || boot_cpu_data.x86_max_cores > 1; @@ -711,6 +695,37 @@ void set_cpu_sibling_map(int cpu) } } +/* + * The bootstrap kernel entry code has set these up. Save them for + * a given CPU + */ +void smp_store_cpu_info(int id, bool force_single_core) +{ + struct cpuinfo_x86 *c = &cpu_data(id); + + /* Copy boot_cpu_data only on the first bringup */ + if (!c->initialized) + *c = boot_cpu_data; + c->cpu_index = id; + /* + * During boot time, CPU0 has this setup already. Save the info when + * bringing up AP or offlined CPU0. + */ + identify_secondary_cpu(c); + + arch_spin_lock(&topology_lock); + BUG_ON(topology_update_package_map(c->phys_proc_id, id)); + BUG_ON(topology_update_die_map(c->cpu_die_id, id)); + c->initialized = true; + + /* For Xen PV */ + if (force_single_core) + c->x86_max_cores = 1; + + set_cpu_sibling_map(id); + arch_spin_unlock(&topology_lock); +} + /* maps the cpu to the sched domain representing multi-core */ const struct cpumask *cpu_coregroup_mask(int cpu) { diff --git a/arch/x86/xen/smp_pv.c b/arch/x86/xen/smp_pv.c index 6175f2c5c822..09f94f940689 100644 --- a/arch/x86/xen/smp_pv.c +++ b/arch/x86/xen/smp_pv.c @@ -71,9 +71,7 @@ static void cpu_bringup(void) xen_enable_syscall(); } cpu = smp_processor_id(); - smp_store_cpu_info(cpu); - cpu_data(cpu).x86_max_cores = 1; - set_cpu_sibling_map(cpu); + smp_store_cpu_info(cpu, true); speculative_store_bypass_ht_init(); From patchwork Thu Feb 2 21:56:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 52193 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp494538wrn; Thu, 2 Feb 2023 13:59:57 -0800 (PST) X-Google-Smtp-Source: AK7set/iof1m2ML04T8HcxsaJlVzxiXOXMfcRNAevoT/rNgm2zyDAkBobhsLoUl46GrmFM3CWRBp X-Received: by 2002:a05:6402:42ca:b0:4a2:47ac:c5a4 with SMTP id i10-20020a05640242ca00b004a247acc5a4mr10813254edc.23.1675375197090; Thu, 02 Feb 2023 13:59:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675375197; cv=none; d=google.com; s=arc-20160816; b=GfsJwoD9qSB+3SSJNjwPaD8ThGcoMr2p1kQWYhqPd95CLViiQGszOgSnZWbRnqaQs6 TyJbBs0PE0gt1BuPydVlN4p209uE2uDZLci2K4rJGoh/zk+KZCqd827tFYUNm9SbSz57 LON+3k0s/QM06TT1tuGg6o2eKYsdNpFUm2sIPQ35tL7gmAYxB5K1ZRy2gHojpdkMoDUc AsfhnbPldN/1aHz2JI7UdJuJLF6sKgBFW9muwkUP9Yv15GX2Y9VUqPbp6387ziAmW35B 7iIvzxA0PFwJKh7Y5fOIa1tA6IgclEJNiRUhOejlpXXSZYemmmaGmrKaL67aV19h0Fhf ybPw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=mlEYiVjRJdjrX9c53b+gnJjwRka4ZmK3zw3dl+LycL4=; b=vEwSzj1ru/UaeyLk2RkI6flLrGZfcHIvwGt5Fky4CCpEEda6mY1qFTkkt2kHx+A3Uu +aCLdZ4anG/M3/2IAeHUBd4g8dVp5tQx4PeRD/La4oGBVAl2U9Pbc+aSQxe6YG2LkmOC Rgwb23H3LHkObZbyiO3HsiFJNMvmE8TSatd9qfAX8x2aq/NVsZm5eYMVuMt3g5v6/23v cvqShBZWvNn2/5RLHuYecyHRDOqgiNEDqYc6xot7tm81WW4MmZ1Eyi30vMCBXaQEJ+Ro iKqVCI7mKOWJ3ueuGseozJTNM3S2jOE9oMGdPOIrtOGrpffByXbArU8m0fDDdLNaDf6g OTqw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=d28IFYmZ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id d11-20020a056402400b00b0047ff112ee12si603583eda.101.2023.02.02.13.59.33; Thu, 02 Feb 2023 13:59:57 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=d28IFYmZ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233286AbjBBV5q (ORCPT + 99 others); Thu, 2 Feb 2023 16:57:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54772 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233241AbjBBV5S (ORCPT ); Thu, 2 Feb 2023 16:57:18 -0500 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1BA3B7A496 for ; Thu, 2 Feb 2023 13:56:42 -0800 (PST) Received: by mail-wr1-x430.google.com with SMTP id r2so2987301wrv.7 for ; Thu, 02 Feb 2023 13:56:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mlEYiVjRJdjrX9c53b+gnJjwRka4ZmK3zw3dl+LycL4=; b=d28IFYmZuelC4BmIuIijPPFJKpeR6Lv83DAOYl4C9BGbHQihhQo9hhtpVtUE0YuD1x Lr1fxDrkMlmUXMbyUMMeurglo0UW/JarxML1HYH9eZU6g1cuxtYA6wa3kEKN1R5QrMsO DETcwBPXpbIBOEcAOeTrgbNYwfJ0CX0qsEsRmn25jT9BeKrRqH7ioIaIFMDP5yKOpsry HPj4TEQ+A2OvEBANu8jibDM8tqIXtc7o0Z5wcBwo889u11HqaidUFya+h88bCt/WL3R+ n6evjrchUjg1Hf9c/fK6B0eQju+6VN81uY3sL3ZARvXIsO/3/K2UUkluvJzah5kr1VsK u9ZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mlEYiVjRJdjrX9c53b+gnJjwRka4ZmK3zw3dl+LycL4=; b=d1lMJ7U0VgmTyop7Bl4gCim/4m6krgyXCFi8UUZ4NbrhVoDYqP0mOd07mu3hJZkjnn cvydIN7GTxcGjudzDS3krLxge3Z1/mFUpvhdBWlbfuZ2HdHdiduh+cOH824fvZvZ4GuC aYn6tc/v63YQxgq5jBswS4tS5a4/JKpd0YJmNqxYYNe0Lk49kiXozoZfdWG9hg6AI++7 axD6d1eM4Qx5KIrn7oIJbCDAob0kl4GwafKe6F3vf+XAdW8seKTNxxM4m5RkQFfmlbRJ fhLj89mkf+Xna+immG9MFaa1C0mtAWJOV/R+024piiwibVddMK4U5O0XpeibAC2Bsz7s bkZw== X-Gm-Message-State: AO0yUKWY8b3ZRouGvQhdSsXu2PaA7bE6QRTPgM19f5nJqdeGdhTyH8+N HeIPh5sU5GOCJ+11Zoi2uf6h9w== X-Received: by 2002:adf:e351:0:b0:2bf:95d0:fa5d with SMTP id n17-20020adfe351000000b002bf95d0fa5dmr6988999wrj.9.1675375001192; Thu, 02 Feb 2023 13:56:41 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:98fe:e4ee:fc7e:cd71]) by smtp.gmail.com with ESMTPSA id e8-20020a5d6d08000000b00297dcfdc90fsm506078wrq.24.2023.02.02.13.56.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Feb 2023 13:56:40 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, arjan@linux.intel.com Cc: mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, Usama Arif Subject: [PATCH v6 11/11] x86/smpboot: reuse timer calibration Date: Thu, 2 Feb 2023 21:56:25 +0000 Message-Id: <20230202215625.3248306-12-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230202215625.3248306-1-usama.arif@bytedance.com> References: <20230202215625.3248306-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756758222382675884?= X-GMAIL-MSGID: =?utf-8?q?1756758222382675884?= From: Arjan van de Ven No point recalibrating for known-constant tsc. When tested on a 128 core, 2 socket server, this reduces the parallel smpboot time from 100ms to 30ms. Not-signed-off-by: Arjan van de Ven Signed-off-by: Usama Arif --- arch/x86/kernel/tsc.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/arch/x86/kernel/tsc.c b/arch/x86/kernel/tsc.c index a78e73da4a74..bab8a98080cf 100644 --- a/arch/x86/kernel/tsc.c +++ b/arch/x86/kernel/tsc.c @@ -1569,6 +1569,9 @@ unsigned long calibrate_delay_is_known(void) if (!constant_tsc || !mask) return 0; + if (cpu != 0) + return cpu_data(0).loops_per_jiffy; + sibling = cpumask_any_but(mask, cpu); if (sibling < nr_cpu_ids) return cpu_data(sibling).loops_per_jiffy;