From patchwork Tue Feb 7 23:04:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 54105 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp3127765wrn; Tue, 7 Feb 2023 15:07:12 -0800 (PST) X-Google-Smtp-Source: AK7set8+Y6ek3w1zC01uo3QtvhwbYKbGWoD6qsDyI9cplJWRRDH5JMYvtntdvuLRAp/sYHpP789X X-Received: by 2002:a17:906:1e8a:b0:870:6554:92ac with SMTP id e10-20020a1709061e8a00b00870655492acmr4914135ejj.18.1675811232173; Tue, 07 Feb 2023 15:07:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675811232; cv=none; d=google.com; s=arc-20160816; b=GvRwlPSKgnSkLK4Lzw+QJmVBMvsHNlGCSR02VW99mZr08udxH3vf84JjgB1lWQk5+J MAPtS/gMB25IH7mGPapBCi/tbdTD/9TGWslarr0fneR9ZtSu8qY5Jm29nasGhaVTN8EX qTWJ/Fe5UNGKmPsL4I37RLB8WccDokDPCfHkVVRLfsgsbQDccWKigCe4pR9Uh8kz4eMz rAwjOjlBFIWfSgJHVp7oJ9NQ4qr758ICe7T6RmzCpZFK0xb5ZiklX5MXuRGXwYIlBXSm qjpnzgTAqWFF/YeDLSsV0zo7PG/+vL46pY+8T/HOD3CXGm5+PMeyGII9pQea3Oof0OCY 6t2Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=LfQxc38f/Cse36bNk18znDedQFt31nho10D3R7IlJOY=; b=jbUq6oPrwYGBk7+YFzXwpGFZDdRlgkzQNxEL/0VavLNG/wMipi1tfexHfBi61BY3Ap /VBVTow8Aw9TYtV4PdB3nHR3IDf8qQVDyVY/crfp6V9r1anhWTbVNBvK2evP29qFDs4J 4rTEQx5O5XVbjbhmIsUH9tU/fnJJ91tV3Ic7Uj1M+uaCpTjSu/Yb9NTRIWUyitu2hw4x iUmPVfrQ3UfBqkdCoxMmjVdwsMPdFbr4Cc2i1r5rfpTJfXpDjkgXoVCaWAhKnjciP+sh +xetO3A/em/uJUPs1jTEv/IAnDChRRyljDozBskQHfcOVd0jzH9m+XhehQuauNzSbCfk tfgQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=O08Lx+34; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id uo32-20020a170907cc2000b0087cc469b193si17964705ejc.836.2023.02.07.15.06.49; Tue, 07 Feb 2023 15:07:12 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=O08Lx+34; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230196AbjBGXFV (ORCPT + 99 others); Tue, 7 Feb 2023 18:05:21 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48958 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229906AbjBGXFI (ORCPT ); Tue, 7 Feb 2023 18:05:08 -0500 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D3D893E62D for ; Tue, 7 Feb 2023 15:04:42 -0800 (PST) Received: by mail-wm1-x32b.google.com with SMTP id bg13-20020a05600c3c8d00b003d9712b29d2so192003wmb.2 for ; Tue, 07 Feb 2023 15:04:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LfQxc38f/Cse36bNk18znDedQFt31nho10D3R7IlJOY=; b=O08Lx+34VJjE3zF4EYtDLdzuURMM1M3Cd4RDku1Am+8S2qpnzcYGvsPKLMxOad6/sf E7345Ewjb/hTSwtbe4xYG3BkHII7JrShAD684tOrinefdBW+Ew1HKWNPv24bS9Avsfbt ybFRABXEIaX7baRKJiHs0mBYron3cJfju7O1XjfHO5TP9OzJOcuHCq+Fxby+uyknDL5H RfGfmFoKgGOGKHZzThSAwA/mJ/cLvTK89M2ZIw5stEcg5lirJa1fp/RXsyQqMHITVckd ASwQNTExaNs+PyOj2QTjrEvf+7cQN1SBPw7c3XUKGTmDQAAvClmK24kPpen/2clHfMhp p1tw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LfQxc38f/Cse36bNk18znDedQFt31nho10D3R7IlJOY=; b=idT10Sgoj96Z3rQjrVUEldi1MKY5tpnCZjHx/l/9xq4HIHnmqQnykMtvWXmzIDVTCs 5TSGTqBN4LhKpTI+1WXHKcNeRTpH1nfKoLqeB60Pr+GCq2V6F2apKyzGN4O1eLZlhtEn 57CeTdjXqUn5GCqzbTvGDDIMdcHhLYRvZK0ZqT2+7lDHUQAvD4wD5wQIQbr2bRhBY+7q 3dJfuXNkUqX3CFMQX2luiMdmUlvv0/9LLSQgWiTT6U7SY4evYdrZBBHRsx67LAxRBRmd XlfRlalmmmAzU5C0NNJp3yrtO/bGbYgS+0iilecouxbJgcrQWUDBo4gHwyjcQbzl6ShT 4DAw== X-Gm-Message-State: AO0yUKVbweZnoIHYVUwOoMWRQhf3Yzil4dGPhMtLzW4LFEvuZ5sctlw1 uKN0TAGKSibVLB8w4WjaU5/Z8Q== X-Received: by 2002:a05:600c:6003:b0:3d9:f769:2115 with SMTP id az3-20020a05600c600300b003d9f7692115mr4475573wmb.26.1675811081314; Tue, 07 Feb 2023 15:04:41 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:c04f:2463:c151:8b87]) by smtp.gmail.com with ESMTPSA id j14-20020a05600c190e00b003dcc82ce53fsm146485wmq.38.2023.02.07.15.04.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Feb 2023 15:04:40 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com Cc: arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse , Usama Arif Subject: [PATCH v7 1/9] x86/apic/x2apic: Allow CPU cluster_mask to be populated in parallel Date: Tue, 7 Feb 2023 23:04:28 +0000 Message-Id: <20230207230436.2690891-2-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230207230436.2690891-1-usama.arif@bytedance.com> References: <20230207230436.2690891-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757215438230298980?= X-GMAIL-MSGID: =?utf-8?q?1757215438230298980?= From: David Woodhouse Each of the sibling CPUs in a cluster uses the same clustermask. The first CPU in a cluster will need a new clustermask allocated, while subsequent siblings will use the same clustermask as the first. However, the CPU being brought up cannot yet perform memory allocations at the point that this occurs in init_x2apic_ldr(). So at present, the alloc_clustermask() function allocates a clustermask just in case it's needed, storing it in the global cluster_hotplug_mask. A CPU which is the first sibling of a cluster will "take" it from there and set cluster_hotplug_mask to NULL, in order for alloc_clustermask() to allocate a new one before bringing up the next CPU. To facilitate parallel bringup of CPUs in future, switch to a model where alloc_clustermask() prepopulates the clustermask in the per_cpu data for each present CPU in the cluster in advance. All that the CPU needs to do for itself in init_x2apic_ldr() is set its own bit in that mask. The 'node' and 'clusterid' members of struct cluster_mask are thus redundant, and it can become a simple struct cpumask instead. Signed-off-by: David Woodhouse Signed-off-by: Usama Arif --- arch/x86/kernel/apic/x2apic_cluster.c | 130 +++++++++++++++++--------- 1 file changed, 84 insertions(+), 46 deletions(-) diff --git a/arch/x86/kernel/apic/x2apic_cluster.c b/arch/x86/kernel/apic/x2apic_cluster.c index e696e22d0531..d8f513a2287b 100644 --- a/arch/x86/kernel/apic/x2apic_cluster.c +++ b/arch/x86/kernel/apic/x2apic_cluster.c @@ -9,11 +9,7 @@ #include "local.h" -struct cluster_mask { - unsigned int clusterid; - int node; - struct cpumask mask; -}; +#define apic_cluster(apicid) ((apicid) >> 4) /* * __x2apic_send_IPI_mask() possibly needs to read @@ -23,8 +19,7 @@ struct cluster_mask { static u32 *x86_cpu_to_logical_apicid __read_mostly; static DEFINE_PER_CPU(cpumask_var_t, ipi_mask); -static DEFINE_PER_CPU_READ_MOSTLY(struct cluster_mask *, cluster_masks); -static struct cluster_mask *cluster_hotplug_mask; +static DEFINE_PER_CPU_READ_MOSTLY(struct cpumask *, cluster_masks); static int x2apic_acpi_madt_oem_check(char *oem_id, char *oem_table_id) { @@ -60,10 +55,10 @@ __x2apic_send_IPI_mask(const struct cpumask *mask, int vector, int apic_dest) /* Collapse cpus in a cluster so a single IPI per cluster is sent */ for_each_cpu(cpu, tmpmsk) { - struct cluster_mask *cmsk = per_cpu(cluster_masks, cpu); + struct cpumask *cmsk = per_cpu(cluster_masks, cpu); dest = 0; - for_each_cpu_and(clustercpu, tmpmsk, &cmsk->mask) + for_each_cpu_and(clustercpu, tmpmsk, cmsk) dest |= x86_cpu_to_logical_apicid[clustercpu]; if (!dest) @@ -71,7 +66,7 @@ __x2apic_send_IPI_mask(const struct cpumask *mask, int vector, int apic_dest) __x2apic_send_IPI_dest(dest, vector, APIC_DEST_LOGICAL); /* Remove cluster CPUs from tmpmask */ - cpumask_andnot(tmpmsk, tmpmsk, &cmsk->mask); + cpumask_andnot(tmpmsk, tmpmsk, cmsk); } local_irq_restore(flags); @@ -105,55 +100,98 @@ static u32 x2apic_calc_apicid(unsigned int cpu) static void init_x2apic_ldr(void) { - struct cluster_mask *cmsk = this_cpu_read(cluster_masks); - u32 cluster, apicid = apic_read(APIC_LDR); - unsigned int cpu; + struct cpumask *cmsk = this_cpu_read(cluster_masks); - x86_cpu_to_logical_apicid[smp_processor_id()] = apicid; + BUG_ON(!cmsk); - if (cmsk) - goto update; - - cluster = apicid >> 16; - for_each_online_cpu(cpu) { - cmsk = per_cpu(cluster_masks, cpu); - /* Matching cluster found. Link and update it. */ - if (cmsk && cmsk->clusterid == cluster) - goto update; + cpumask_set_cpu(smp_processor_id(), cmsk); +} + +/* + * As an optimisation during boot, set the cluster_mask for all present + * CPUs at once, to prevent each of them having to iterate over the others + * to find the existing cluster_mask. + */ +static void prefill_clustermask(struct cpumask *cmsk, unsigned int cpu, u32 cluster) +{ + int cpu_i; + + for_each_present_cpu(cpu_i) { + struct cpumask **cpu_cmsk = &per_cpu(cluster_masks, cpu_i); + u32 apicid = apic->cpu_present_to_apicid(cpu_i); + + if (apicid == BAD_APICID || cpu_i == cpu || apic_cluster(apicid) != cluster) + continue; + + if (WARN_ON_ONCE(*cpu_cmsk == cmsk)) + continue; + + BUG_ON(*cpu_cmsk); + *cpu_cmsk = cmsk; } - cmsk = cluster_hotplug_mask; - cmsk->clusterid = cluster; - cluster_hotplug_mask = NULL; -update: - this_cpu_write(cluster_masks, cmsk); - cpumask_set_cpu(smp_processor_id(), &cmsk->mask); } -static int alloc_clustermask(unsigned int cpu, int node) +static int alloc_clustermask(unsigned int cpu, u32 cluster, int node) { + struct cpumask *cmsk = NULL; + unsigned int cpu_i; + + /* + * At boot time, the CPU present mask is stable. The cluster mask is + * allocated for the first CPU in the cluster and propagated to all + * present siblings in the cluster. If the cluster mask is already set + * on entry to this function for a given CPU, there is nothing to do. + */ if (per_cpu(cluster_masks, cpu)) return 0; + + if (system_state < SYSTEM_RUNNING) + goto alloc; + /* - * If a hotplug spare mask exists, check whether it's on the right - * node. If not, free it and allocate a new one. + * On post boot hotplug for a CPU which was not present at boot time, + * iterate over all possible CPUs (even those which are not present + * any more) to find any existing cluster mask. */ - if (cluster_hotplug_mask) { - if (cluster_hotplug_mask->node == node) - return 0; - kfree(cluster_hotplug_mask); + for_each_possible_cpu(cpu_i) { + u32 apicid = apic->cpu_present_to_apicid(cpu_i); + + if (apicid != BAD_APICID && apic_cluster(apicid) == cluster) { + cmsk = per_cpu(cluster_masks, cpu_i); + /* + * If the cluster is already initialized, just store + * the mask and return. There's no need to propagate. + */ + if (cmsk) { + per_cpu(cluster_masks, cpu) = cmsk; + return 0; + } + } } - - cluster_hotplug_mask = kzalloc_node(sizeof(*cluster_hotplug_mask), - GFP_KERNEL, node); - if (!cluster_hotplug_mask) - return -ENOMEM; - cluster_hotplug_mask->node = node; - return 0; + /* + * No CPU in the cluster has ever been initialized, so fall through to + * the boot time code which will also populate the cluster mask for any + * other CPU in the cluster which is (now) present. + */ + alloc: + cmsk = kzalloc_node(sizeof(*cmsk), GFP_KERNEL, node); + if (!cmsk) + return -ENOMEM; + per_cpu(cluster_masks, cpu) = cmsk; + prefill_clustermask(cmsk, cpu, cluster); + + return 0; } static int x2apic_prepare_cpu(unsigned int cpu) { - if (alloc_clustermask(cpu, cpu_to_node(cpu)) < 0) + u32 phys_apicid = apic->cpu_present_to_apicid(cpu); + u32 cluster = apic_cluster(phys_apicid); + u32 logical_apicid = (cluster << 16) | (1 << (phys_apicid & 0xf)); + + x86_cpu_to_logical_apicid[cpu] = logical_apicid; + + if (alloc_clustermask(cpu, cluster, cpu_to_node(cpu)) < 0) return -ENOMEM; if (!zalloc_cpumask_var(&per_cpu(ipi_mask, cpu), GFP_KERNEL)) return -ENOMEM; @@ -162,10 +200,10 @@ static int x2apic_prepare_cpu(unsigned int cpu) static int x2apic_dead_cpu(unsigned int dead_cpu) { - struct cluster_mask *cmsk = per_cpu(cluster_masks, dead_cpu); + struct cpumask *cmsk = per_cpu(cluster_masks, dead_cpu); if (cmsk) - cpumask_clear_cpu(dead_cpu, &cmsk->mask); + cpumask_clear_cpu(dead_cpu, cmsk); free_cpumask_var(per_cpu(ipi_mask, dead_cpu)); return 0; } From patchwork Tue Feb 7 23:04:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 54108 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp3127948wrn; Tue, 7 Feb 2023 15:07:35 -0800 (PST) X-Google-Smtp-Source: AK7set/741K/F42h+Uu+1LBcbd+/l8fcqvRsDqRwttwiTYfRegnS0MXsu0041JzDVf6BuLCRCj4V X-Received: by 2002:a17:907:6da6:b0:878:4497:380e with SMTP id sb38-20020a1709076da600b008784497380emr7305288ejc.4.1675811255769; Tue, 07 Feb 2023 15:07:35 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675811255; cv=none; d=google.com; s=arc-20160816; b=qYp6FIQcAXdiMZauHqCS49O8rYwRKgvhw+Zde3qvY1UrcQEMVN8anfIYdjZyQ8g/FP nQhIT6DkrzZCpaMNIsaGUbW0isLxdvmKvTE6bztm8AuKRjTlzvMSQZ+101q/fnDROPfy T2Xf5Oxkhxc9Tw5Ypkhzj0tt7ZpECfqs0MRM96ebQ3UXa/Z00n0q5scOREOOkkGGhnRo VhaH1ek7Ri4RUpxEf8xqK1vs8AFoYSdNwu0xKyXDTmeyBlFiaFKS3og7l397VF5ZSVgy cyyifODIfbrk75kL0y33+2gGSiyqDg9zCvRA5wJfG4d3vpWdz/rUVu4uj8hG2feTQ3xP yN1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=XKydoXD7DOLwoETiq1rVpdNfhGCr3/VFLGt4p4rSeXA=; b=dZmFwVHBQNbAk4UZy19BzFmN7d/WJyZmxgylZkSOzNvTdmtbnv1w++PgGTThrp20VZ 4aadiQ6EF4lxfIL4GVlLIw/Ma5IbTtfV0NaZlgOEbVAB1Xav9LMTzCtpNLGy1UtOVf5M vgjBL8ArNFPyOpeHfjMzQSZg1PtoW5JYv0S4QTHEgeqRG9avz2Ml6Sw2EYt9H5WBpXuP hBxYG0Hwg61JUZAvmuZdLZG+evZkcLGaz+vZKpam6htagn7CkN3qs+Yp3tyjulXFZbGg 4a9rH7TA34UiFLxPb0wuD5Ld2c8XE/5KAkM24Ereu2wZZ3Fqxq3+kCFXeUvowI+wc1ki g/xQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b="Hzld0/Jk"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l6-20020a170907914600b0088adcfbd9basi14482584ejs.988.2023.02.07.15.07.12; Tue, 07 Feb 2023 15:07:35 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b="Hzld0/Jk"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230232AbjBGXFb (ORCPT + 99 others); Tue, 7 Feb 2023 18:05:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48974 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230094AbjBGXFJ (ORCPT ); Tue, 7 Feb 2023 18:05:09 -0500 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B28383EC53 for ; Tue, 7 Feb 2023 15:04:43 -0800 (PST) Received: by mail-wm1-x329.google.com with SMTP id l37-20020a05600c1d2500b003dfe46a9801so221564wms.0 for ; Tue, 07 Feb 2023 15:04:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XKydoXD7DOLwoETiq1rVpdNfhGCr3/VFLGt4p4rSeXA=; b=Hzld0/JklLKS5ElmGesXwglnkEAxlghpu5u+RXXd8eLsmOBqmKGgmg/qpnGfZqokHq xUsE3XHUe3Wb5Gb04q1hiJbGh7DRco334Cn/d4qz0EKEFguht/H01YPLaE854A+cMzQc 1ScdgN27hmzjlVkkipBYathx6d0cXlkCEVlowZvJ45FYP7K7RKFDR1CZh3oiqnt2Lbmo XwDNLs+HdrccUIMu2IPnOs7QW6S/uDGFi+dhru1+sk9ik7jGDE/wwEcNfVVxnes8Ss/5 qjNvJ4ZvZqZ3z1vzSsH3pSwX5od8kU5TKKyoIXJqigf0F0ci353HD5n6dEL1bbiOyOHx wK2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XKydoXD7DOLwoETiq1rVpdNfhGCr3/VFLGt4p4rSeXA=; b=j82Ha1OS4v7/xGtoTKMhFwxuGI/aLNbpNGyQxjQs9+D802K/PVFhY+fHe9wfZofDVR lkEFAKcGjtDrKl+BOFle9lj7q/JYGa5ZjxkpSznatzPLxvf7817FkEdHvA1iNpIk12NN k21hbUdIiy8651URtyp2ZCi4SHWM18Z4A4kra4/odcpb0yxdG33SgxBom96YoKXB3XJ1 BJC5No1NdW1qzKruwa8SO1q8Royy2yFdvF/13w5Uve+pnnmRgxWYvZttG0XS8q94zMb3 UsNGa1qMrz0OwSS6XpxPysj9r5k83ycgjcOpBMKJSdxuGDi8DG1b+LnOCmYgW2ptRj8q 1p8A== X-Gm-Message-State: AO0yUKXrW4+odtFmW2cOTFAmnc5gLOu+WTBhGWuTsSkvjkYFYGsAK72R q+m5Ryfn0dOoM/L5JTjjyAY0Wg== X-Received: by 2002:a05:600c:319d:b0:3df:e1cc:94ff with SMTP id s29-20020a05600c319d00b003dfe1cc94ffmr4565048wmp.28.1675811082324; Tue, 07 Feb 2023 15:04:42 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:c04f:2463:c151:8b87]) by smtp.gmail.com with ESMTPSA id j14-20020a05600c190e00b003dcc82ce53fsm146485wmq.38.2023.02.07.15.04.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Feb 2023 15:04:41 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com Cc: arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse , Usama Arif Subject: [PATCH v7 2/9] cpu/hotplug: Move idle_thread_get() to Date: Tue, 7 Feb 2023 23:04:29 +0000 Message-Id: <20230207230436.2690891-3-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230207230436.2690891-1-usama.arif@bytedance.com> References: <20230207230436.2690891-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757215463356214794?= X-GMAIL-MSGID: =?utf-8?q?1757215463356214794?= From: David Woodhouse Instead of relying purely on the special-case wrapper in bringup_cpu() to pass the idle thread to __cpu_up(), expose idle_thread_get() so that the architecture code can obtain it directly when necessary. This will be useful when the existing __cpu_up() is split into multiple phases, only *one* of which will actually need the idle thread. If the architecture code is to register its new pre-bringup states with the cpuhp core, having a special-case wrapper to pass extra arguments is non-trivial and it's easier just to let the arch register its function pointer to be invoked with the standard API. Signed-off-by: David Woodhouse Reviewed-by: Thomas Gleixner Signed-off-by: Usama Arif --- include/linux/smpboot.h | 7 +++++++ kernel/smpboot.h | 2 -- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/include/linux/smpboot.h b/include/linux/smpboot.h index 9d1bc65d226c..3862addcaa34 100644 --- a/include/linux/smpboot.h +++ b/include/linux/smpboot.h @@ -5,6 +5,13 @@ #include struct task_struct; + +#ifdef CONFIG_GENERIC_SMP_IDLE_THREAD +struct task_struct *idle_thread_get(unsigned int cpu); +#else +static inline struct task_struct *idle_thread_get(unsigned int cpu) { return NULL; } +#endif + /* Cookie handed to the thread_fn*/ struct smpboot_thread_data; diff --git a/kernel/smpboot.h b/kernel/smpboot.h index 34dd3d7ba40b..60c609318ad6 100644 --- a/kernel/smpboot.h +++ b/kernel/smpboot.h @@ -5,11 +5,9 @@ struct task_struct; #ifdef CONFIG_GENERIC_SMP_IDLE_THREAD -struct task_struct *idle_thread_get(unsigned int cpu); void idle_thread_set_boot_cpu(void); void idle_threads_init(void); #else -static inline struct task_struct *idle_thread_get(unsigned int cpu) { return NULL; } static inline void idle_thread_set_boot_cpu(void) { } static inline void idle_threads_init(void) { } #endif From patchwork Tue Feb 7 23:04:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 54107 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp3127879wrn; Tue, 7 Feb 2023 15:07:28 -0800 (PST) X-Google-Smtp-Source: AK7set+nSwYyNhP4ytc61dFSBj9YowUjE5a0PX3MKWMt3u3BD20OBQtzVOeBjYPZ2RFQctTQgC8j X-Received: by 2002:a17:906:380d:b0:879:2a5:dc3c with SMTP id v13-20020a170906380d00b0087902a5dc3cmr5036410ejc.40.1675811248128; Tue, 07 Feb 2023 15:07:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675811248; cv=none; d=google.com; s=arc-20160816; b=q1NBKlL09MODNI99/PS7VKbM6UhB6s63PliCsae3SamM7jPf7eJJ2wtW3lVT6pZBYs waKApotKQmz2xa5qVxasR3zWLl2IYjw+gLDuV9co238jbOY9iiBxoUj0ZtexsnwHBzEu o/NUAgpMvCGyWpNaYQf5KBh5nbv5mDadG+prG9OOhOhTjr5TYZ5dCXxMUS3Tee7VkdLX ssefqq8YiNzEaeVpEbYuXcOejxJDquaIVbT6Oq/wAqsib7+07o2doOTWG2jHdYUX0wL6 bE2cPFB49a9uPosiBeGo5w+qA3wWG0m7+CBzbUgwsvd8jdx+ktnBblmxFQiglzMgXsjr 4I2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=mY6NeUci0HQtTva2v+94F98gv/kbTw7O8GGA4dBnL3M=; b=os3vTwFZV6uJRDlXkGnRwnrVhTD8r6vDLR1eNUBxdHllhOa39wUqFVjQ5+mxEzuP/Z J0T+TX9McR6hqYSH8B82nxTrWGI3+z92PzEdhwou8d04uvMdR9SEREhdBI2se6Z/hRwB BeRguoAJC6lfXGl9bnvjmz9NCK0M4FXLEPeJv1aUXEEOrbY1YbSzHjjH+tXW/1Pufx1k DF46BtQ6olZbklJVX0EmEKgHg002LGbC3TiFiT+s/0MsxB+5HRbaHWCe/8Tb9zQa4+wx midLqKtpcIIw1nYB3jD1z730itouxHjEN1wSuzZvO1NCCmfuKCMCsrO1UoTVjvi8fLVb aZ3A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=Vi2Ppf9k; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id fx14-20020a1709069e8e00b0087859d78883si12901898ejc.823.2023.02.07.15.07.05; Tue, 07 Feb 2023 15:07:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=Vi2Ppf9k; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230230AbjBGXF2 (ORCPT + 99 others); Tue, 7 Feb 2023 18:05:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48988 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229796AbjBGXFK (ORCPT ); Tue, 7 Feb 2023 18:05:10 -0500 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CC3263FF12 for ; Tue, 7 Feb 2023 15:04:44 -0800 (PST) Received: by mail-wm1-x32c.google.com with SMTP id l37-20020a05600c1d2500b003dfe46a9801so221588wms.0 for ; Tue, 07 Feb 2023 15:04:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mY6NeUci0HQtTva2v+94F98gv/kbTw7O8GGA4dBnL3M=; b=Vi2Ppf9keVW26+otagfWa7oYewc9B9pagBPslJ3Kjc68ep9DpstkoM9u12dNmSCOv0 YOY4GifEbRhhywq+8KX/SbPc6kZleF1JQo/8ISWnqj2dPvXW0K4GnQPIxRze1daggjZ1 yHrIG8cKtcmzLBxOXsyJC/VjbAt6XOtj0pVTCjfjRxuNZ3rcBcOWcLp2gMNMnwDxldGN FKHGITBMIrEMFl3f8TAXkoZTflRdGqaOZ5ZQkxfDNstrgaJ1CWnwcHOCecN9sUGv4RFu xkR6wCpqcY8iOCVCgCXGVGtkZhlxaYdjn4SmA5jmdJybaMwHKPJIPBVVLDchVy3f9wpB /c8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mY6NeUci0HQtTva2v+94F98gv/kbTw7O8GGA4dBnL3M=; b=1Q5qSXd+LoHnG8ntxWFjImY++lxUv4wRQinheHq30byn+aU3VnlQlxyzmMOT45U7T5 R4EnHleiFwuJx80S+U5Qpw4R+jFIvqzQj796lujehRTXlzL0pUTRxJe72GWvvDLmXsiN 6s/LfqJe6b4IfUdlsNE1jZ7YXlM31yQ2QFHtuASGD3VWElwoJJDFJ1PS/6YnjCRYTlKI KmheRzC1CQDfTgw5VCqXRwbqQl2MUpbIO+Dby8b+z81q0+M2x+0W1BWPSNRca/CTYtVS jm9rj7v/wpZuevrikSWVPvDVtUl+mJGCi7rNOBu1Hcp5dWFa1dHoZ29Oz4DsXoccPip8 F5QA== X-Gm-Message-State: AO0yUKXsmO4Oad0IvYwGjJRvsSfVxnqPshQefA8Zq3fVHeSq2KzIOipt XwFpLpeTw5PPCLNv/0nyktkWug== X-Received: by 2002:a05:600c:1694:b0:3de:27c3:ef13 with SMTP id k20-20020a05600c169400b003de27c3ef13mr4537840wmn.9.1675811083383; Tue, 07 Feb 2023 15:04:43 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:c04f:2463:c151:8b87]) by smtp.gmail.com with ESMTPSA id j14-20020a05600c190e00b003dcc82ce53fsm146485wmq.38.2023.02.07.15.04.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Feb 2023 15:04:42 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com Cc: arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse , Usama Arif Subject: [PATCH v7 3/9] cpu/hotplug: Add dynamic parallel bringup states before CPUHP_BRINGUP_CPU Date: Tue, 7 Feb 2023 23:04:30 +0000 Message-Id: <20230207230436.2690891-4-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230207230436.2690891-1-usama.arif@bytedance.com> References: <20230207230436.2690891-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757215455031552106?= X-GMAIL-MSGID: =?utf-8?q?1757215455031552106?= From: David Woodhouse There is often significant latency in the early stages of CPU bringup, and time is wasted by waking each CPU (e.g. with SIPI/INIT/INIT on x86) and then waiting for it to respond before moving on to the next. Allow a platform to register a set of pre-bringup CPUHP states to which each CPU can be stepped in parallel, thus absorbing some of that latency. There is a subtlety here: even with an empty CPUHP_BP_PARALLEL_DYN step, this means that *all* CPUs are brought through the prepare states and to CPUHP_BP_PREPARE_DYN before any of them are taken to CPUHP_BRINGUP_CPU and then are allowed to run for themselves to CPUHP_ONLINE. So any combination of prepare/start calls which depend on A-B ordering for each CPU in turn, such as the X2APIC code which used to allocate a cluster mask 'just in case' and store it in a global variable in the prep stage, then potentially consume that preallocated structure from the AP and set the global pointer to NULL to be reallocated in CPUHP_X2APIC_PREPARE for the next CPU... would explode horribly. Any platform enabling the CPUHP_BP_PARALLEL_DYN steps must be reviewed and tested to ensure that such issues do not exist, and the existing behaviour of bringing CPUs to CPUHP_BP_PREPARE_DYN and then immediately to CPUHP_BRINGUP_CPU and CPUHP_ONLINE only one at a time does not change unless such a state is registered. Note that the new parallel stages do *not* yet bring each AP to the CPUHP_BRINGUP_CPU state at the same time, only to the new states which exist before it. The final loop in bringup_nonboot_cpus() is untouched, bringing each AP in turn from the final PARALLEL_DYN state (or all the way from CPUHP_OFFLINE) to CPUHP_BRINGUP_CPU and then waiting for that AP to do its own processing and reach CPUHP_ONLINE before releasing the next. Parallelising that part by bringing them all to CPUHP_BRINGUP_CPU and then waiting for them all is an exercise for the future. Signed-off-by: David Woodhouse Signed-off-by: Usama Arif --- include/linux/cpuhotplug.h | 2 ++ kernel/cpu.c | 31 +++++++++++++++++++++++++++++-- 2 files changed, 31 insertions(+), 2 deletions(-) diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h index 6c6859bfc454..e5a73ae6ccc0 100644 --- a/include/linux/cpuhotplug.h +++ b/include/linux/cpuhotplug.h @@ -133,6 +133,8 @@ enum cpuhp_state { CPUHP_MIPS_SOC_PREPARE, CPUHP_BP_PREPARE_DYN, CPUHP_BP_PREPARE_DYN_END = CPUHP_BP_PREPARE_DYN + 20, + CPUHP_BP_PARALLEL_DYN, + CPUHP_BP_PARALLEL_DYN_END = CPUHP_BP_PARALLEL_DYN + 4, CPUHP_BRINGUP_CPU, /* diff --git a/kernel/cpu.c b/kernel/cpu.c index 6c0a92ca6bb5..fffb0da61ccc 100644 --- a/kernel/cpu.c +++ b/kernel/cpu.c @@ -1504,8 +1504,30 @@ int bringup_hibernate_cpu(unsigned int sleep_cpu) void bringup_nonboot_cpus(unsigned int setup_max_cpus) { + unsigned int n = setup_max_cpus - num_online_cpus(); unsigned int cpu; + /* + * An architecture may have registered parallel pre-bringup states to + * which each CPU may be brought in parallel. For each such state, + * bring N CPUs to it in turn before the final round of bringing them + * online. + */ + if (n > 0) { + enum cpuhp_state st = CPUHP_BP_PARALLEL_DYN; + + while (st <= CPUHP_BP_PARALLEL_DYN_END && cpuhp_hp_states[st].name) { + int i = n; + + for_each_present_cpu(cpu) { + cpu_up(cpu, st); + if (!--i) + break; + } + st++; + } + } + for_each_present_cpu(cpu) { if (num_online_cpus() >= setup_max_cpus) break; @@ -1882,6 +1904,10 @@ static int cpuhp_reserve_state(enum cpuhp_state state) step = cpuhp_hp_states + CPUHP_BP_PREPARE_DYN; end = CPUHP_BP_PREPARE_DYN_END; break; + case CPUHP_BP_PARALLEL_DYN: + step = cpuhp_hp_states + CPUHP_BP_PARALLEL_DYN; + end = CPUHP_BP_PARALLEL_DYN_END; + break; default: return -EINVAL; } @@ -1906,14 +1932,15 @@ static int cpuhp_store_callbacks(enum cpuhp_state state, const char *name, /* * If name is NULL, then the state gets removed. * - * CPUHP_AP_ONLINE_DYN and CPUHP_BP_PREPARE_DYN are handed out on + * CPUHP_AP_ONLINE_DYN and CPUHP_BP_P*_DYN are handed out on * the first allocation from these dynamic ranges, so the removal * would trigger a new allocation and clear the wrong (already * empty) state, leaving the callbacks of the to be cleared state * dangling, which causes wreckage on the next hotplug operation. */ if (name && (state == CPUHP_AP_ONLINE_DYN || - state == CPUHP_BP_PREPARE_DYN)) { + state == CPUHP_BP_PREPARE_DYN || + state == CPUHP_BP_PARALLEL_DYN)) { ret = cpuhp_reserve_state(state); if (ret < 0) return ret; From patchwork Tue Feb 7 23:04:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 54103 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp3127575wrn; Tue, 7 Feb 2023 15:06:48 -0800 (PST) X-Google-Smtp-Source: AK7set/793gSGKEwjYTdrx2NCbXBz9O8BltA+5Y5ewMgZG2w0FDdK7XfEDcfB1KKQdukOB7ZWyLV X-Received: by 2002:a17:906:3950:b0:887:3c7e:5df with SMTP id g16-20020a170906395000b008873c7e05dfmr5383507eje.73.1675811208167; Tue, 07 Feb 2023 15:06:48 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675811208; cv=none; d=google.com; s=arc-20160816; b=oJb+EvQ2zbI23oNV2YNajaf09RnBoiCH57TMx/b1GoZYbrCisnspXLkDmdlyb/Gppa l4wNU9r4M0AprlNyQrSnob9QHV7Q9WkjIEFGKNndwYt2OrK2NPgOmxVUOLwF1ttXs1/y 2Ohrhahr0vq/3U72apRusxSY0iZ52Tsuim7vQkp34R28R5bxQjQN2lLc/9I/VS3cUaOl Fh9Kk2T6q4DYftMgbaj1q3WdrjJYvFdConqObluXtvHplqQN2UnayZd4P2I1STZAQW1l 2yA1rvAqV3JlmGf7zYXW9w9UuXU4cpI1G87QSGMsMZv/xVqFRmyrDO+ijKH6i/mAFGEj 8Kuw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=L+woZryxcmbb99mupmnna5okozeg16DDHF5Mj8G3upM=; b=y1+OjlvTpZxTyD9ww5CTK6HmaVH1IOkLqdMRuTaAd+g6VPGRtWU2dWeSzNpysQzhWu AwLIWfPYl2aLgMFpS3aSa4REsdaGHBQO68lCJd/OFCyMt63cBV5VimespBMb8IqS/Tjb wR0xihTBiiNdcyNnIdG4mKIRZLZLyuxRSj4epHx/27rPexpEJ1ohg7nj5IpP7RjqQBiD 6KtL3dk2gPt8NETYVVaxdY6MXwIzQHWOkHWLzFdipjMzvEFTdfeMxAxLyvXFhNmBeF3P AbpT2k1ufWI+Dad8DzVR92bZtrxYf3RvIFs7fJAKpqLWhU5ZeNR9+21H3+DnQGoYtdH3 lo9A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b="I6OGmp/z"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 17-20020a170906001100b00881984b0f41si21212040eja.291.2023.02.07.15.06.24; Tue, 07 Feb 2023 15:06:48 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b="I6OGmp/z"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229781AbjBGXEt (ORCPT + 99 others); Tue, 7 Feb 2023 18:04:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48942 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230010AbjBGXEq (ORCPT ); Tue, 7 Feb 2023 18:04:46 -0500 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B9BA13FF2B for ; Tue, 7 Feb 2023 15:04:45 -0800 (PST) Received: by mail-wr1-x42c.google.com with SMTP id z11so1090443wrl.8 for ; Tue, 07 Feb 2023 15:04:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=L+woZryxcmbb99mupmnna5okozeg16DDHF5Mj8G3upM=; b=I6OGmp/zf+SXm16RpuuHG0Cs0grAvvCacNPO9dKsk1CnGRmdMFu6+0NJdJhyZFTM4X t222CAl7UR84WcZsi9vxKoozrPUuo3JLOzCIH9sxnbaS2E1xVnpxE5522R2XhnSGXoAq H4dMxa2K3aJcC4yIroI+hFKQ3ShkEp8NGiIlepnXYdQeZXr63YjWL9GWbYxgDjB8SqFq 6hSMRm3TCfJwjZPwZVHhxdN9/3ajo3kOYvz8TDCXvwpUD7+G2VBebFw/w20A52rvxJgq tohCQW4m433T31Ja45QB9CYTeV4/Ljp7ac0MTuuxzvTX9xUgS2fkFqmLhmlrsnEBHe/T tISw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=L+woZryxcmbb99mupmnna5okozeg16DDHF5Mj8G3upM=; b=CfRCNwybFMLY5G3B2mfDrU8ZKkyGRUctmWl3eLT0fqvG/VSq1g2soH5D/Z3WowsHW0 9cxJL9h9MPx+3/MGUworFb+nMFhH0JVhACKoEbdXkiDGiRhkzJVRRlXg2XqY16V/jYIP 1NV1JAebYH7WookmHGyONJyTvisSw4LIO1dN8wIv9uheDJPNVAOQI7X45Q8ZpM4X/TEU U4RgiZcEl5itn5H55PQ4aQKHvEJ1vwF9XlBPSyugBNBc5gNoTkmNW9HnafHsPD3wT339 B4mq1Jf4YNMENhuzcIQ0x8UIrae5OY5NDw8bE89L4LhL9wnAUwTJEVfzxzA/P1BV+opP vJag== X-Gm-Message-State: AO0yUKUEEJ3RkE3OyJDJrQaoaQHbfw5E6X/ri53aq1vXGhJGnPo2hzt6 sIS6zA8GIsfaoVME22XR9PrVlg== X-Received: by 2002:a5d:5183:0:b0:2bf:b5ec:4844 with SMTP id k3-20020a5d5183000000b002bfb5ec4844mr4689320wrv.19.1675811084196; Tue, 07 Feb 2023 15:04:44 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:c04f:2463:c151:8b87]) by smtp.gmail.com with ESMTPSA id j14-20020a05600c190e00b003dcc82ce53fsm146485wmq.38.2023.02.07.15.04.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Feb 2023 15:04:43 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com Cc: arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse , Usama Arif Subject: [PATCH v7 4/9] x86/smpboot: Reference count on smpboot_setup_warm_reset_vector() Date: Tue, 7 Feb 2023 23:04:31 +0000 Message-Id: <20230207230436.2690891-5-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230207230436.2690891-1-usama.arif@bytedance.com> References: <20230207230436.2690891-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757215413339722888?= X-GMAIL-MSGID: =?utf-8?q?1757215413339722888?= From: David Woodhouse When bringing up a secondary CPU from do_boot_cpu(), the warm reset flag is set in CMOS and the starting IP for the trampoline written inside the BDA at 0x467. Once the CPU is running, the CMOS flag is unset and the value in the BDA cleared. To allow for parallel bringup of CPUs, add a reference count to track the number of CPUs currently bring brought up, and clear the state only when the count reaches zero. Since the RTC spinlock is required to write to the CMOS, it can be used for mutual exclusion on the refcount too. [Usama Arif: fixed rebase conflict] Signed-off-by: David Woodhouse Signed-off-by: Usama Arif --- arch/x86/kernel/smpboot.c | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 55cad72715d9..3a793772a2aa 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -121,17 +121,20 @@ int arch_update_cpu_topology(void) return retval; } + +static unsigned int smpboot_warm_reset_vector_count; + static inline void smpboot_setup_warm_reset_vector(unsigned long start_eip) { unsigned long flags; spin_lock_irqsave(&rtc_lock, flags); - CMOS_WRITE(0xa, 0xf); + if (!smpboot_warm_reset_vector_count++) { + CMOS_WRITE(0xa, 0xf); + *((volatile unsigned short *)phys_to_virt(TRAMPOLINE_PHYS_HIGH)) = start_eip >> 4; + *((volatile unsigned short *)phys_to_virt(TRAMPOLINE_PHYS_LOW)) = start_eip & 0xf; + } spin_unlock_irqrestore(&rtc_lock, flags); - *((volatile unsigned short *)phys_to_virt(TRAMPOLINE_PHYS_HIGH)) = - start_eip >> 4; - *((volatile unsigned short *)phys_to_virt(TRAMPOLINE_PHYS_LOW)) = - start_eip & 0xf; } static inline void smpboot_restore_warm_reset_vector(void) @@ -143,10 +146,12 @@ static inline void smpboot_restore_warm_reset_vector(void) * to default values. */ spin_lock_irqsave(&rtc_lock, flags); - CMOS_WRITE(0, 0xf); + if (!--smpboot_warm_reset_vector_count) { + CMOS_WRITE(0, 0xf); + *((volatile u32 *)phys_to_virt(TRAMPOLINE_PHYS_LOW)) = 0; + } spin_unlock_irqrestore(&rtc_lock, flags); - *((volatile u32 *)phys_to_virt(TRAMPOLINE_PHYS_LOW)) = 0; } /* From patchwork Tue Feb 7 23:04:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 54106 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp3127775wrn; Tue, 7 Feb 2023 15:07:14 -0800 (PST) X-Google-Smtp-Source: AK7set/Nyi4wZmGLdzL3dQim9DACg6tVK1LMOUzqwuaHdBajPHu7eUm6uNebswtmbDiDqV3FL9yj X-Received: by 2002:a50:cd13:0:b0:4aa:ab40:385d with SMTP id z19-20020a50cd13000000b004aaab40385dmr5869245edi.1.1675811233980; Tue, 07 Feb 2023 15:07:13 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675811233; cv=none; d=google.com; s=arc-20160816; b=iTorSdrpQCT6UMdU3onO3fdNCt3aKdOrfrtbNsdrxSORszn+d6Wol3SdrNWx+omS3A 90m7UHairX1/79FSyuDEVautuxyiy5K0RyWocXlL6HlurPeiPsnPtiI7dVPJezIc31XU miupBx+9SlSOTaKXJsdHUD+6hHLwDkLyRrFJAMuunNa0LZu98ZOCRpIvT2MoVKAIb+85 9e2fvjxonY0ps11iehl02UIt0IGUlhr86ntgUUE2wHVsd5Id7gRimYD3Sten+j+dB0YR cQChjXDPFx0iQJqnSqg90CUlpds30LsnpmkOyGap1p54ArFysQ+dyVLqH/4ENejTmv7I H5wg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=JMBjZKZYclFXE66iKNBq4sqZjzU651Exc4VIypeE1m8=; b=th5YIUIZs7upz9Zlz1MgLlf9DQHb3uTIKnWu52KbKZ9Zfwezbir8Fb7PePIfptLu8I qy7lljz39Wpsm4HleiNwBUZmaomhY0O9LJ942eyhLNY9bZnx2XRTLdlAnDmG+rMlv+Nr 0GvrFXmKzH0k6ILGmJeqGgoc4nLrSGC2gCA68ZBOy59nDemO6oM1tLj1fagdCHVDfCCE SD7y2QFHVG2xTZ7QHoK7Q3OUoHQu0KP9RuBbzre+qXMMeve62tv2W3zZzbFzen7e5knX w4sfNfGx3uCLpPt/qmySu9dDkHkfd5iEff1RXAAHYbD+zUR/tq008h8s2vx37cMwmkj7 TA2A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=cioo6k3p; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id j14-20020aa7c0ce000000b004aaaa3b7e44si10311162edp.522.2023.02.07.15.06.50; Tue, 07 Feb 2023 15:07:13 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=cioo6k3p; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230123AbjBGXFX (ORCPT + 99 others); Tue, 7 Feb 2023 18:05:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49020 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230168AbjBGXFN (ORCPT ); Tue, 7 Feb 2023 18:05:13 -0500 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A7E393FF3D for ; Tue, 7 Feb 2023 15:04:46 -0800 (PST) Received: by mail-wm1-x332.google.com with SMTP id bg5-20020a05600c3c8500b003e00c739ce4so195627wmb.5 for ; Tue, 07 Feb 2023 15:04:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JMBjZKZYclFXE66iKNBq4sqZjzU651Exc4VIypeE1m8=; b=cioo6k3pMFYX4IiqeQEnvhh2NQU75E2477mIoV0EIox1JWpqiCreM44+qtWsNZ6KmL 07ZbCNr6aSxH8ZFarCNj1HfMgGPaPJTONRvoOYCpRoQmKldQwpbc/tLUxR6OkM3Thk+h vyuRJuB27SOrXSjy5/QiEv7Of3EPVzNJuJIINz5TCY30yuDQHculja4OnXzawe82nS/D I/y64JbK28MBOLWa2wLwyuMOnqqnh085LAxNQBKUJ8Q4U0oAX2JRWqAkpoHSchWNEXqJ TmNUVwj8JbzKpIV2u3yTzJV0b27CV6YLuzLL/x0cM0SQ0J825/NWcnimIDphAkJgLKse Py1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JMBjZKZYclFXE66iKNBq4sqZjzU651Exc4VIypeE1m8=; b=5c4FKcESSksqVXdWnjPTWCtUWJCHPAl4lovoi9+JW4XYgl+BLEBNjmTl6HeXmRQG0u uE7BrinUyzfhDs7LOTU5Vn/Pw6zlUa5eff524raC0eA9oF+gK48nkCHDVCbOyeXwB+vx EYT01apqE8/di21wrk1qhpJvAcojHg7jaWAEwENPM4hUumXOxl4MOVI5uehuXFGANGqP 6UZO6sIXsVINe+3nH1AEnOordiK+ruYbzG3GS3Nv0MjeP2ctU1GD7sFzarU0E5Kxj9rv ImAOAtsJ34lRi0XhFDhagmRm5t+qGPx0XIxtm6Cpy3FSebJ6VpEH+i/CFYhsi8qhO1ur N0fw== X-Gm-Message-State: AO0yUKU8U7DE6i/QK206tFD2MJzTP8OFYhcESOW78nFT2BS9tuvdTBrs hSgOCZGJSP3+mUEcgVd1kTDGLA== X-Received: by 2002:a05:600c:16d4:b0:3e0:fad:675a with SMTP id l20-20020a05600c16d400b003e00fad675amr4333512wmn.38.1675811085153; Tue, 07 Feb 2023 15:04:45 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:c04f:2463:c151:8b87]) by smtp.gmail.com with ESMTPSA id j14-20020a05600c190e00b003dcc82ce53fsm146485wmq.38.2023.02.07.15.04.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Feb 2023 15:04:44 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com Cc: arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse , Usama Arif Subject: [PATCH v7 5/9] x86/smpboot: Split up native_cpu_up into separate phases and document them Date: Tue, 7 Feb 2023 23:04:32 +0000 Message-Id: <20230207230436.2690891-6-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230207230436.2690891-1-usama.arif@bytedance.com> References: <20230207230436.2690891-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757215440430134048?= X-GMAIL-MSGID: =?utf-8?q?1757215440430134048?= From: David Woodhouse There are four logical parts to what native_cpu_up() does on the BSP (or on the controlling CPU for a later hotplug): 1) Wake the AP by sending the INIT/SIPI/SIPI sequence. 2) Wait for the AP to make it as far as wait_for_master_cpu() which sets that CPU's bit in cpu_initialized_mask, then sets the bit in cpu_callout_mask to let the AP proceed through cpu_init(). 3) Waits for the AP to finish cpu_init() and get as far as the smp_callin() call, which sets that CPU's bit in cpu_callin_mask. 4) Perform the TSC synchronization and wait for the AP to actually mark itself online in cpu_online_mask. In preparation to allow these phases to operate in parallel on multiple APs, split them out into separate functions and document the interactions a little more clearly in both the BSP and AP code paths. No functional change intended. [Usama Arif: fixed rebase conflict] Signed-off-by: David Woodhouse Signed-off-by: Usama Arif --- arch/x86/kernel/smpboot.c | 182 +++++++++++++++++++++++++++----------- 1 file changed, 128 insertions(+), 54 deletions(-) diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 3a793772a2aa..eee717086ab7 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -204,6 +204,10 @@ static void smp_callin(void) wmb(); + /* + * This runs the AP through all the cpuhp states to its target + * state (CPUHP_ONLINE in the case of serial bringup). + */ notify_cpu_starting(cpuid); /* @@ -231,17 +235,33 @@ static void notrace start_secondary(void *unused) load_cr3(swapper_pg_dir); __flush_tlb_all(); #endif + /* + * Sync point with do_wait_cpu_initialized(). On boot, all secondary + * CPUs reach this stage after receiving INIT/SIPI from do_cpu_up() + * in the x86/cpu:kick cpuhp stage. At the start of cpu_init() they + * will wait for do_wait_cpu_initialized() to set their bit in + * smp_callout_mask to release them. + */ cpu_init_secondary(); rcu_cpu_starting(raw_smp_processor_id()); x86_cpuinit.early_percpu_clock_init(); + + /* + * Sync point with do_wait_cpu_callin(). The AP doesn't wait here + * but just sets the bit to let the controlling CPU (BSP) know that + * it's got this far. + */ smp_callin(); enable_start_cpu0 = 0; /* otherwise gcc will move up smp_processor_id before the cpu_init */ barrier(); + /* - * Check TSC synchronization with the boot CPU: + * Check TSC synchronization with the boot CPU (or whichever CPU + * is controlling the bringup). It will do its part of this from + * do_wait_cpu_online(), making it an implicit sync point. */ check_tsc_sync_target(); @@ -254,6 +274,7 @@ static void notrace start_secondary(void *unused) * half valid vector space. */ lock_vector_lock(); + /* Sync point with do_wait_cpu_online() */ set_cpu_online(smp_processor_id(), true); lapic_online(); unlock_vector_lock(); @@ -1083,7 +1104,6 @@ static int do_boot_cpu(int apicid, int cpu, struct task_struct *idle, unsigned long start_ip = real_mode_header->trampoline_start; unsigned long boot_error = 0; - unsigned long timeout; #ifdef CONFIG_X86_64 /* If 64-bit wakeup method exists, use the 64-bit mode trampoline IP */ @@ -1144,55 +1164,94 @@ static int do_boot_cpu(int apicid, int cpu, struct task_struct *idle, boot_error = wakeup_cpu_via_init_nmi(cpu, start_ip, apicid, cpu0_nmi_registered); - if (!boot_error) { - /* - * Wait 10s total for first sign of life from AP - */ - boot_error = -1; - timeout = jiffies + 10*HZ; - while (time_before(jiffies, timeout)) { - if (cpumask_test_cpu(cpu, cpu_initialized_mask)) { - /* - * Tell AP to proceed with initialization - */ - cpumask_set_cpu(cpu, cpu_callout_mask); - boot_error = 0; - break; - } - schedule(); - } - } + return boot_error; +} - if (!boot_error) { - /* - * Wait till AP completes initial initialization - */ - while (!cpumask_test_cpu(cpu, cpu_callin_mask)) { - /* - * Allow other tasks to run while we wait for the - * AP to come online. This also gives a chance - * for the MTRR work(triggered by the AP coming online) - * to be completed in the stop machine context. - */ - schedule(); - } +static int do_wait_cpu_cpumask(unsigned int cpu, const struct cpumask *mask) +{ + unsigned long timeout; + + /* + * Wait up to 10s for the CPU to report in. + */ + timeout = jiffies + 10*HZ; + while (time_before(jiffies, timeout)) { + if (cpumask_test_cpu(cpu, mask)) + return 0; + + schedule(); } + return -1; +} - if (x86_platform.legacy.warm_reset) { - /* - * Cleanup possible dangling ends... - */ - smpboot_restore_warm_reset_vector(); +/* + * Bringup step two: Wait for the target AP to reach cpu_init_secondary() + * and thus wait_for_master_cpu(), then set cpu_callout_mask to allow it + * to proceed. The AP will then proceed past setting its 'callin' bit + * and end up waiting in check_tsc_sync_target() until we reach + * do_wait_cpu_online() to tend to it. + */ +static int do_wait_cpu_initialized(unsigned int cpu) +{ + /* + * Wait for first sign of life from AP. + */ + if (do_wait_cpu_cpumask(cpu, cpu_initialized_mask)) + return -1; + + cpumask_set_cpu(cpu, cpu_callout_mask); + return 0; +} + +/* + * Bringup step three: Wait for the target AP to reach smp_callin(). + * The AP is not waiting for us here so we don't need to parallelise + * this step. Not entirely clear why we care about this, since we just + * proceed directly to TSC synchronization which is the next sync + * point with the AP anyway. + */ +static int do_wait_cpu_callin(unsigned int cpu) +{ + /* + * Wait till AP completes initial initialization. + */ + return do_wait_cpu_cpumask(cpu, cpu_callin_mask); +} + +/* + * Bringup step four: Synchronize the TSC and wait for the target AP + * to reach set_cpu_online() in start_secondary(). + */ +static int do_wait_cpu_online(unsigned int cpu) +{ + unsigned long flags; + + /* + * Check TSC synchronization with the AP (keep irqs disabled + * while doing so): + */ + local_irq_save(flags); + check_tsc_sync_source(cpu); + local_irq_restore(flags); + + /* + * Wait for the AP to mark itself online. Not entirely + * clear why we care, since the generic cpuhp code will + * wait for it to each CPUHP_AP_ONLINE_IDLE before going + * ahead with the rest of the bringup anyway. + */ + while (!cpu_online(cpu)) { + cpu_relax(); + touch_nmi_watchdog(); } - return boot_error; + return 0; } -int native_cpu_up(unsigned int cpu, struct task_struct *tidle) +static int do_cpu_up(unsigned int cpu, struct task_struct *tidle) { int apicid = apic->cpu_present_to_apicid(cpu); int cpu0_nmi_registered = 0; - unsigned long flags; int err, ret = 0; lockdep_assert_irqs_enabled(); @@ -1239,19 +1298,6 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) goto unreg_nmi; } - /* - * Check TSC synchronization with the AP (keep irqs disabled - * while doing so): - */ - local_irq_save(flags); - check_tsc_sync_source(cpu); - local_irq_restore(flags); - - while (!cpu_online(cpu)) { - cpu_relax(); - touch_nmi_watchdog(); - } - unreg_nmi: /* * Clean up the nmi handler. Do this after the callin and callout sync @@ -1263,6 +1309,34 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) return ret; } +int native_cpu_up(unsigned int cpu, struct task_struct *tidle) +{ + int ret; + + ret = do_cpu_up(cpu, tidle); + if (ret) + return ret; + + ret = do_wait_cpu_initialized(cpu); + if (ret) + return ret; + + ret = do_wait_cpu_callin(cpu); + if (ret) + return ret; + + ret = do_wait_cpu_online(cpu); + + if (x86_platform.legacy.warm_reset) { + /* + * Cleanup possible dangling ends... + */ + smpboot_restore_warm_reset_vector(); + } + + return ret; +} + /** * arch_disable_smp_support() - disables SMP support for x86 at runtime */ From patchwork Tue Feb 7 23:04:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 54112 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp3128910wrn; Tue, 7 Feb 2023 15:09:34 -0800 (PST) X-Google-Smtp-Source: AK7set92E5JF1IXFxgc06xmn+Kh09PBx9rE3yPeqta1wwuMaGKz7iksf0KgVg2C7L4u/Xej0nu6i X-Received: by 2002:a17:906:f1cc:b0:877:7157:9358 with SMTP id gx12-20020a170906f1cc00b0087771579358mr760242ejb.10.1675811373905; Tue, 07 Feb 2023 15:09:33 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675811373; cv=none; d=google.com; s=arc-20160816; b=ZcE6NdQmiL/iyUAsxZPD/rCr1CShiAXe7k1opVxHYz2OQwa5nf/4p0uFfBiSHoS4B8 h7qE1/pwzsknY8TdoEnnpe29er7OzL4SGUq8Al1aJ3rhySWHJqy4g1j6Fim133vUTDpm 55TT05Wi9tADF/LkAj61xFlMsm/q8YScz0rsPQ+CZ/jLZQIQcwAuP08ULT9UOQ9aRl75 OE7UXDZGijTSol9ikj78GcYGSDbAONYi52lTul3dv7L+6YAbl/e+m6X1XXkP7/sux2/5 vchqVSitFxMEeRiQU85j3W2h258DgaLB+CgzHnOLLVHQZkDbLv+4qK/XNXtb/clSvJlh ETbg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=l3DRX0gFO8Y8Ir/xbOPnEtgu7l9ehHqV085Y1j0CPCQ=; b=M2zhAkXHbUJzvQZETev41iKLcOu2gs3Don2uIU+zYh2LuMzvO/jWuGGJT5Nyr18tNt Xr0M20bfQf55LAAajceQb7nEG5unW9d2ur3WiaEfZ/8Dh+Mda2zeAMQKrunUtMkPS+Et Y+nfHzkn0aqDwF5eJ8kYJEel9XqvPi+V1k+HAGbMQlMqPJJtmRAAGRFm9FZHdPrL7Q12 NWhU0bmKTsgV/iN6f8PM20TnH0kLbvptuHdvh1oHqQdszuy6pQrGeasdRcENtdHRt9Oz a/reRmwGMtojpdPwD0uAJHJA6FTI7FOJs3ZVMU79tPfuAFV73Dj+EhgYPz7nq5cIUScY SAWg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=n4vBM9tI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id gk6-20020a17090790c600b0088bba5ae982si16070671ejb.140.2023.02.07.15.09.10; Tue, 07 Feb 2023 15:09:33 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=n4vBM9tI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230061AbjBGXGB (ORCPT + 99 others); Tue, 7 Feb 2023 18:06:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49506 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230223AbjBGXF2 (ORCPT ); Tue, 7 Feb 2023 18:05:28 -0500 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 93342402E3 for ; Tue, 7 Feb 2023 15:04:47 -0800 (PST) Received: by mail-wm1-x333.google.com with SMTP id j32-20020a05600c1c2000b003dc4fd6e61dso174737wms.5 for ; Tue, 07 Feb 2023 15:04:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=l3DRX0gFO8Y8Ir/xbOPnEtgu7l9ehHqV085Y1j0CPCQ=; b=n4vBM9tIf3osRdbZieCcDBcztSBtz7jQzLvrJSOVzwuTTKB8ZcjI9aklMTiTyfW5no FvAatTbG6LQ/1n+/0IZmzZ5PRaUtpC8SSeBgEHMNa1h9zfYZ+163K00g671k7RPyvZxS lv1fFB0o0/OyfzmX9S/4Bpo6lkIUvWH3dtskdbP93bZi77SgF9aKevs0BpAFppeFWxtG 8nyO5ejdPLfTdg24NKwUP/LhwSOMxJ8sn19JsFLJm6FBIdHAIF0cX3SRecf9lR4UosYK 9gUFWBM+TXV+dt42+TenTErQBgTm+HreRtgoTdpMczFRCT7ey9pScsUZtuEi+V4r5Mu/ SClA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=l3DRX0gFO8Y8Ir/xbOPnEtgu7l9ehHqV085Y1j0CPCQ=; b=3C3I7CQPQ2iH36HER0/9DbMO82Ww1P+nOsdrWFeDYKRSk2NqaMbIntbqGO97ai82zL xkySxNC9NJ8naXYsrLFF8d4A8fZbXSZSmOwocN/4ik8X6hsR2stNgEQNn4cQX2Da5c3s 3ME3s2AeCmqs73p+rzY9jcB3Suue581ucz9CK/a5qZteOZQmHaJSyewMbPPtWtVW5qfn IKaXx6AWA4QNG4/JGlLKi6JVkIhRbA1ZJVLhZvCqBeAWU9zS3BPL6ceFQznQoIbFYwx3 /NdhuDEQklLNdS+zfTCvz4djRhlv9W1iMvBbPWkB+TfZCabJvSqUvpBALYtiyOW3q0SA Oqmg== X-Gm-Message-State: AO0yUKU609mV9L9vP2Y/p5LODK4/OU9kQAK98Vt6r6TW9qhem2C3cI6l y2fG3l8SDLG8BVT5/pr71OQGXA== X-Received: by 2002:a05:600c:1c85:b0:3e0:fad:739a with SMTP id k5-20020a05600c1c8500b003e00fad739amr356458wms.31.1675811086104; Tue, 07 Feb 2023 15:04:46 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:c04f:2463:c151:8b87]) by smtp.gmail.com with ESMTPSA id j14-20020a05600c190e00b003dcc82ce53fsm146485wmq.38.2023.02.07.15.04.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Feb 2023 15:04:45 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com Cc: arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, Usama Arif Subject: [PATCH v7 6/9] x86/smpboot: Support parallel startup of secondary CPUs Date: Tue, 7 Feb 2023 23:04:33 +0000 Message-Id: <20230207230436.2690891-7-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230207230436.2690891-1-usama.arif@bytedance.com> References: <20230207230436.2690891-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757215586864283925?= X-GMAIL-MSGID: =?utf-8?q?1757215586864283925?= From: Thomas Gleixner Rework the real-mode startup code to allow for APs to be brought up in parallel. This is in two parts: 1. Introduce a bit-spinlock to prevent them from all using the real mode stack at the same time. 2. Avoid the use of global variables for passing per-CPU information to the APs. To achieve the latter, export the cpuid_to_apicid[] array so that each AP can find its own per_cpu data (and thus initial_gs, initial_stack and early_gdt_descr) by searching therein based on its APIC ID. Introduce a global variable 'smpboot_control' indicating to the AP how it should find its APIC ID. For a serialized bringup, the APIC ID is explicitly passed in the low bits of smpboot_control, while for parallel mode there are flags directing the AP to find its APIC ID in CPUID leaf 0x0b (for X2APIC mode) or CPUID leaf 0x01 where 8 bits are sufficient. Parallel startup may be disabled by a command line option, and also if: • AMD SEV-ES is in use, since the AP may not use CPUID that early. • X2APIC is enabled, but CPUID leaf 0xb is not present and correect. • X2APIC is not enabled but not even CPUID leaf 0x01 exists. Aside from the fact that APs will now look up their per-cpu data via the newly-exported cpuid_to_apicid[] table, there is no behavioural change intended yet, since new parallel CPUHP states have not — yet — been added. [ dwmw2: Minor tweaks, write a commit message, add CPUID 0x01 support ] [ seanc: Fix stray override of initial_gs in common_cpu_up() ] Not-signed-off-by: Thomas Gleixner Signed-off-by: David Woodhouse Signed-off-by: Usama Arif --- arch/x86/include/asm/realmode.h | 3 + arch/x86/include/asm/smp.h | 10 +++- arch/x86/kernel/acpi/sleep.c | 1 + arch/x86/kernel/apic/apic.c | 2 +- arch/x86/kernel/head_64.S | 84 ++++++++++++++++++++++++++++ arch/x86/kernel/smpboot.c | 60 ++++++++++++++++++-- arch/x86/realmode/init.c | 3 + arch/x86/realmode/rm/trampoline_64.S | 14 +++++ kernel/smpboot.c | 2 +- 9 files changed, 172 insertions(+), 7 deletions(-) diff --git a/arch/x86/include/asm/realmode.h b/arch/x86/include/asm/realmode.h index a336feef0af1..f0357cfe2fb0 100644 --- a/arch/x86/include/asm/realmode.h +++ b/arch/x86/include/asm/realmode.h @@ -52,6 +52,7 @@ struct trampoline_header { u64 efer; u32 cr4; u32 flags; + u32 lock; #endif }; @@ -65,6 +66,8 @@ extern unsigned long initial_stack; extern unsigned long initial_vc_handler; #endif +extern u32 *trampoline_lock; + extern unsigned char real_mode_blob[]; extern unsigned char real_mode_relocs[]; diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h index b4dbb20dab1a..33c0d5fd8af6 100644 --- a/arch/x86/include/asm/smp.h +++ b/arch/x86/include/asm/smp.h @@ -199,5 +199,13 @@ extern void nmi_selftest(void); #define nmi_selftest() do { } while (0) #endif -#endif /* __ASSEMBLY__ */ +extern unsigned int smpboot_control; + +#endif /* !__ASSEMBLY__ */ + +/* Control bits for startup_64 */ +#define STARTUP_SECONDARY 0x80000000 +#define STARTUP_APICID_CPUID_0B 0x40000000 +#define STARTUP_APICID_CPUID_01 0x20000000 + #endif /* _ASM_X86_SMP_H */ diff --git a/arch/x86/kernel/acpi/sleep.c b/arch/x86/kernel/acpi/sleep.c index 3b7f4cdbf2e0..06adf340a0f1 100644 --- a/arch/x86/kernel/acpi/sleep.c +++ b/arch/x86/kernel/acpi/sleep.c @@ -115,6 +115,7 @@ int x86_acpi_suspend_lowlevel(void) early_gdt_descr.address = (unsigned long)get_cpu_gdt_rw(smp_processor_id()); initial_gs = per_cpu_offset(smp_processor_id()); + smpboot_control = 0; #endif initial_code = (unsigned long)wakeup_long64; saved_magic = 0x123456789abcdef0L; diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c index 20d9a604da7c..ac1d7e5da1f2 100644 --- a/arch/x86/kernel/apic/apic.c +++ b/arch/x86/kernel/apic/apic.c @@ -2377,7 +2377,7 @@ static int nr_logical_cpuids = 1; /* * Used to store mapping between logical CPU IDs and APIC IDs. */ -static int cpuid_to_apicid[] = { +int cpuid_to_apicid[] = { [0 ... NR_CPUS - 1] = -1, }; diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index 222efd4a09bc..656e6018b9d4 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -25,6 +25,7 @@ #include #include #include +#include /* * We are not able to switch in one step to the final KERNEL ADDRESS SPACE @@ -241,6 +242,77 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) UNWIND_HINT_EMPTY ANNOTATE_NOENDBR // above +#ifdef CONFIG_SMP + /* + * Is this the boot CPU coming up? If so everything is available + * in initial_gs, initial_stack and early_gdt_descr. + */ + movl smpboot_control(%rip), %edx + testl $STARTUP_SECONDARY, %edx + jz .Lsetup_cpu + + /* + * Secondary CPUs find out the offsets via the APIC ID. For parallel + * boot the APIC ID is retrieved from CPUID, otherwise it's encoded + * in smpboot_control: + * Bit 31 STARTUP_SECONDARY flag (checked above) + * Bit 30 STARTUP_APICID_CPUID_0B flag (use CPUID 0x0b) + * Bit 29 STARTUP_APICID_CPUID_01 flag (use CPUID 0x01) + * Bit 0-24 APIC ID if STARTUP_APICID_CPUID_xx flags are not set + */ + testl $STARTUP_APICID_CPUID_0B, %edx + jnz .Luse_cpuid_0b + testl $STARTUP_APICID_CPUID_01, %edx + jnz .Luse_cpuid_01 + andl $0x0FFFFFFF, %edx + jmp .Lsetup_AP + +.Luse_cpuid_01: + mov $0x01, %eax + cpuid + mov %ebx, %edx + shr $24, %edx + jmp .Lsetup_AP + +.Luse_cpuid_0b: + mov $0x0B, %eax + xorl %ecx, %ecx + cpuid + +.Lsetup_AP: + /* EDX contains the APICID of the current CPU */ + xorl %ecx, %ecx + leaq cpuid_to_apicid(%rip), %rbx + +.Lfind_cpunr: + cmpl (%rbx), %edx + jz .Linit_cpu_data + addq $4, %rbx + addq $8, %rcx + jmp .Lfind_cpunr + +.Linit_cpu_data: + /* Get the per cpu offset */ + leaq __per_cpu_offset(%rip), %rbx + addq %rcx, %rbx + movq (%rbx), %rbx + /* Save it for GS BASE setup */ + movq %rbx, initial_gs(%rip) + + /* Calculate the GDT address */ + movq $gdt_page, %rcx + addq %rbx, %rcx + movq %rcx, early_gdt_descr_base(%rip) + + /* Find the idle task stack */ + movq $idle_threads, %rcx + addq %rbx, %rcx + movq (%rcx), %rcx + movq TASK_threadsp(%rcx), %rcx + movq %rcx, initial_stack(%rip) +#endif /* CONFIG_SMP */ + +.Lsetup_cpu: /* * We must switch to a new descriptor in kernel space for the GDT * because soon the kernel won't have access anymore to the userspace @@ -281,6 +353,14 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) */ movq initial_stack(%rip), %rsp + /* Drop the realmode protection. For the boot CPU the pointer is NULL! */ + movq trampoline_lock(%rip), %rax + testq %rax, %rax + jz .Lsetup_idt + lock + btrl $0, (%rax) + +.Lsetup_idt: /* Setup and Load IDT */ pushq %rsi call early_setup_idt @@ -426,6 +506,7 @@ SYM_DATA(initial_vc_handler, .quad handle_vc_boot_ghcb) * reliably detect the end of the stack. */ SYM_DATA(initial_stack, .quad init_thread_union + THREAD_SIZE - FRAME_SIZE) +SYM_DATA(trampoline_lock, .quad 0); __FINITDATA __INIT @@ -660,6 +741,9 @@ SYM_DATA_END(level1_fixmap_pgt) SYM_DATA(early_gdt_descr, .word GDT_ENTRIES*8-1) SYM_DATA_LOCAL(early_gdt_descr_base, .quad INIT_PER_CPU_VAR(gdt_page)) + .align 16 +SYM_DATA(smpboot_control, .long 0) + .align 16 /* This must match the first entry in level2_kernel_pgt */ SYM_DATA(phys_base, .quad 0x0) diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index eee717086ab7..7abdf347493f 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -798,6 +798,16 @@ static int __init cpu_init_udelay(char *str) } early_param("cpu_init_udelay", cpu_init_udelay); +static bool do_parallel_bringup = true; + +static int __init no_parallel_bringup(char *str) +{ + do_parallel_bringup = false; + + return 0; +} +early_param("no_parallel_bringup", no_parallel_bringup); + static void __init smp_quirk_init_udelay(void) { /* if cmdline changed it from default, leave it alone */ @@ -1085,8 +1095,6 @@ int common_cpu_up(unsigned int cpu, struct task_struct *idle) #ifdef CONFIG_X86_32 /* Stack for startup_32 can be just as for start_secondary onwards */ per_cpu(pcpu_hot.top_of_stack, cpu) = task_top_of_stack(idle); -#else - initial_gs = per_cpu_offset(cpu); #endif return 0; } @@ -1111,9 +1119,14 @@ static int do_boot_cpu(int apicid, int cpu, struct task_struct *idle, start_ip = real_mode_header->trampoline_start64; #endif idle->thread.sp = (unsigned long)task_pt_regs(idle); - early_gdt_descr.address = (unsigned long)get_cpu_gdt_rw(cpu); initial_code = (unsigned long)start_secondary; - initial_stack = idle->thread.sp; + + if (IS_ENABLED(CONFIG_X86_32)) { + early_gdt_descr.address = (unsigned long)get_cpu_gdt_rw(cpu); + initial_stack = idle->thread.sp; + } else if (!do_parallel_bringup) { + smpboot_control = STARTUP_SECONDARY | apicid; + } /* Enable the espfix hack for this CPU */ init_espfix_ap(cpu); @@ -1513,6 +1526,45 @@ void __init native_smp_prepare_cpus(unsigned int max_cpus) speculative_store_bypass_ht_init(); + /* + * We can do 64-bit AP bringup in parallel if the CPU reports + * its APIC ID in CPUID (either leaf 0x0B if we need the full + * APIC ID in X2APIC mode, or leaf 0x01 if 8 bits are + * sufficient). Otherwise it's too hard. And not for SEV-ES + * guests because they can't use CPUID that early. + */ + if (IS_ENABLED(CONFIG_X86_32) || boot_cpu_data.cpuid_level < 1 || + (x2apic_mode && boot_cpu_data.cpuid_level < 0xb) || + cc_platform_has(CC_ATTR_GUEST_STATE_ENCRYPT)) + do_parallel_bringup = false; + + if (do_parallel_bringup && x2apic_mode) { + unsigned int eax, ebx, ecx, edx; + + /* + * To support parallel bringup in x2apic mode, the AP will need + * to obtain its APIC ID from CPUID 0x0B, since CPUID 0x01 has + * only 8 bits. Check that it is present and seems correct. + */ + cpuid_count(0xb, 0, &eax, &ebx, &ecx, &edx); + + /* + * AMD says that if executed with an umimplemented level in + * ECX, then it will return all zeroes in EAX. Intel says it + * will return zeroes in both EAX and EBX. Checking only EAX + * should be sufficient. + */ + if (eax) { + smpboot_control = STARTUP_SECONDARY | STARTUP_APICID_CPUID_0B; + } else { + pr_info("Disabling parallel bringup because CPUID 0xb looks untrustworthy\n"); + do_parallel_bringup = false; + } + } else if (do_parallel_bringup) { + /* Without X2APIC, what's in CPUID 0x01 should suffice. */ + smpboot_control = STARTUP_SECONDARY | STARTUP_APICID_CPUID_01; + } + snp_set_wakeup_secondary_cpu(); } diff --git a/arch/x86/realmode/init.c b/arch/x86/realmode/init.c index af565816d2ba..788e5559549f 100644 --- a/arch/x86/realmode/init.c +++ b/arch/x86/realmode/init.c @@ -154,6 +154,9 @@ static void __init setup_real_mode(void) trampoline_header->flags = 0; + trampoline_lock = &trampoline_header->lock; + *trampoline_lock = 0; + trampoline_pgd = (u64 *) __va(real_mode_header->trampoline_pgd); /* Map the real mode stub as virtual == physical */ diff --git a/arch/x86/realmode/rm/trampoline_64.S b/arch/x86/realmode/rm/trampoline_64.S index e38d61d6562e..49ebc1636ffd 100644 --- a/arch/x86/realmode/rm/trampoline_64.S +++ b/arch/x86/realmode/rm/trampoline_64.S @@ -49,6 +49,19 @@ SYM_CODE_START(trampoline_start) mov %ax, %es mov %ax, %ss + /* + * Make sure only one CPU fiddles with the realmode stack + */ +.Llock_rm: + btl $0, tr_lock + jnc 2f + pause + jmp .Llock_rm +2: + lock + btsl $0, tr_lock + jc .Llock_rm + # Setup stack movl $rm_stack_end, %esp @@ -241,6 +254,7 @@ SYM_DATA_START(trampoline_header) SYM_DATA(tr_efer, .space 8) SYM_DATA(tr_cr4, .space 4) SYM_DATA(tr_flags, .space 4) + SYM_DATA(tr_lock, .space 4) SYM_DATA_END(trampoline_header) #include "trampoline_common.S" diff --git a/kernel/smpboot.c b/kernel/smpboot.c index 2c7396da470c..a18a21dff9bc 100644 --- a/kernel/smpboot.c +++ b/kernel/smpboot.c @@ -25,7 +25,7 @@ * For the hotplug case we keep the task structs around and reuse * them. */ -static DEFINE_PER_CPU(struct task_struct *, idle_threads); +DEFINE_PER_CPU(struct task_struct *, idle_threads); struct task_struct *idle_thread_get(unsigned int cpu) { From patchwork Tue Feb 7 23:04:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 54109 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp3128273wrn; Tue, 7 Feb 2023 15:08:12 -0800 (PST) X-Google-Smtp-Source: AK7set+Q825CgGayxwH15cOb7Fx5y9EApi224zgXJHhd3eIYH9+LszDMOOyBSuUZM8wl2tBUuhrU X-Received: by 2002:a17:906:8477:b0:88d:ba89:1844 with SMTP id hx23-20020a170906847700b0088dba891844mr678633ejc.21.1675811292490; Tue, 07 Feb 2023 15:08:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675811292; cv=none; d=google.com; s=arc-20160816; b=csQnaknqTRAtIVV/VFNDAJ2JFPwvK/X7ASJ/3E9oQJoN4B6lV/sXcUGsEgKanIV2NO mOXZTVGqYUuu+df2Zkzs23jE45rf/RucFE4ATsUPpaVvOLrBm0rKET070PIeZWP0YCTO ZmmSZEEjoZ5QsGYZJKHy3epldbfpyC2Rfyhn5XCkwuYCCHJ97QOXkFC6Vhpc4NlK+fdU W5t0qQsi8elrHDSCfuZgUR6cJy613WnG16R8CR+C/RInDjr19ZFYlP5pru8muMw9GnAW f79IKFrSGfmnYdYBGAq7qUKLXPtpMjwBos6Ki3psJkaF8lfhUSgaskDfT6OP7dhQT3aI 6bew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ZWyZfvHfRMCW/IU8Kqgu9e5Q/3q0z6e4L7Wtj650hu8=; b=tBKJbsxCjyUNkTMazfHwjR/VrMsnO0JrNBmqlgdP7+kH0QScsDgHFbyYAjIN7u0qKd ctTpMmzp9VlPx5JrK4n1qdWmlBpN5kqeN6P6jo43xwEZlnYSBq+me5lUs2JRg3amWg0c evljMo2PPAe0+A+7VOTEREo6ri5uCj5mM4akc2jI4T3XVykmjCoN47uyoUTmSGkkqAJS lmF7Uq+TTKmsM9fwb0kPeFt4hRpIe7B6d17zK8t96Ej9++RmF2x0IC6CXkJlT+1eIiX5 XsjgTZT1Wer6LeFx2yOQ/p6OK6tx9kq6TeYUO3BauTKa9MuGme+36XsnSr/wHyzmYuIE 7nFg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=sj91hFEt; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id g2-20020a50ee02000000b004aadb6f5495si978981eds.205.2023.02.07.15.07.49; Tue, 07 Feb 2023 15:08:12 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=sj91hFEt; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229565AbjBGXF5 (ORCPT + 99 others); Tue, 7 Feb 2023 18:05:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48944 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230168AbjBGXF0 (ORCPT ); Tue, 7 Feb 2023 18:05:26 -0500 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4C13A3FF33 for ; Tue, 7 Feb 2023 15:04:48 -0800 (PST) Received: by mail-wm1-x335.google.com with SMTP id j29-20020a05600c1c1d00b003dc52fed235so204108wms.1 for ; Tue, 07 Feb 2023 15:04:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZWyZfvHfRMCW/IU8Kqgu9e5Q/3q0z6e4L7Wtj650hu8=; b=sj91hFEtXyPTrl/TkrAJ1B+e/sbdEY/hOTom3rh+R8665fhyF94f79s6gfupGiNZsH P1egkAuKBB9SRp/ujvqE4l6LiJfSO4hGFVhcRJFXcoEAAb0tpUFkvhTD6njHhTPPMdF0 am+8Hcm44U9UAql8zZB+dO12E4IFLt+/CP1sh/TGO6W4VhVFeQszvx9qiLNqzpNCYAIJ cIVNCCojklaeWTPO9LloT9pW4B7SRgI1TgXiSzNwqCJKVnFacs+tA/f1cNhyKkt2aUJq dwhDjuWVQj0F2iGMl12PzM5tD0If6DuRXB4tC5AdkZbwghBoD77409US86SpJ62MJRMj dWdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZWyZfvHfRMCW/IU8Kqgu9e5Q/3q0z6e4L7Wtj650hu8=; b=L4pglM89QTxEEf9PIlRQLX94McUEQTtujL/uvm2J8vmWI8MhlNwAjXEkvaYYxGXkOW vbCSOcfyTsWeOUs5nXTBRtLLGg06RfoRrB3dqA+X6UP5vv7dD/yVbJEPnt5CL4BHs8VK 9HJ5b6qhjWNbhvxlprlxUPUZOzBs0BQgQN21uB/sXTQ6fCbiKkw3OcYW7guSc2v+CEaG 1aekPxh7kdc4k2PekpuFgZ5KOXjhmXFMAnyYX5Z+1XaVl7j7VvC8HZSXQQPvwkWlpQAk AQbbWF95y6WdTVSEkfkV7gtqny8651L34NdmlafLU+ATU40vjideuisU7/mCpntAiFYQ 0vqg== X-Gm-Message-State: AO0yUKVnp0swcIzAjxpu1rWRIZqJW0n47K2+ACqnReFP/zrY0bgdFHeZ pv8wKI154wPcmH4LCfCdwrDTAA== X-Received: by 2002:a05:600c:4ecb:b0:3df:9858:c02c with SMTP id g11-20020a05600c4ecb00b003df9858c02cmr533084wmq.1.1675811086915; Tue, 07 Feb 2023 15:04:46 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:c04f:2463:c151:8b87]) by smtp.gmail.com with ESMTPSA id j14-20020a05600c190e00b003dcc82ce53fsm146485wmq.38.2023.02.07.15.04.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Feb 2023 15:04:46 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com Cc: arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse , Usama Arif Subject: [PATCH v7 7/9] x86/smpboot: Send INIT/SIPI/SIPI to secondary CPUs in parallel Date: Tue, 7 Feb 2023 23:04:34 +0000 Message-Id: <20230207230436.2690891-8-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230207230436.2690891-1-usama.arif@bytedance.com> References: <20230207230436.2690891-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757215501815173289?= X-GMAIL-MSGID: =?utf-8?q?1757215501815173289?= From: David Woodhouse When the APs can find their own APIC ID without assistance, perform the AP bringup in parallel. Register a CPUHP_BP_PARALLEL_DYN stage "x86/cpu:kick" which just calls do_boot_cpu() to deliver INIT/SIPI/SIPI to each AP in turn before the normal native_cpu_up() does the rest of the hand-holding. The APs will then take turns through the real mode code (which has its own bitlock for exclusion) until they make it to their own stack, then proceed through the first few lines of start_secondary() and execute these parts in parallel: start_secondary() -> cr4_init() -> (some 32-bit only stuff so not in the parallel cases) -> cpu_init_secondary() -> cpu_init_exception_handling() -> cpu_init() -> wait_for_master_cpu() At this point they wait for the BSP to set their bit in cpu_callout_mask (from do_wait_cpu_initialized()), and release them to continue through the rest of cpu_init() and beyond. This reduces the time taken for bringup on my 28-thread Haswell system from about 120ms to 80ms. On a socket 96-thread Skylake it takes the bringup time from 500ms to 100ms. There is more speedup to be had by doing the remaining parts in parallel too — especially notify_cpu_starting() in which the AP takes itself through all the stages from CPUHP_BRINGUP_CPU to CPUHP_ONLINE. But those require careful auditing to ensure they are reentrant, before we can go that far. [Usama Arif: fixed rebase conflict] Signed-off-by: David Woodhouse Signed-off-by: Usama Arif --- arch/x86/kernel/smpboot.c | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 7abdf347493f..7e7bcab6676e 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -57,6 +57,7 @@ #include #include #include +#include #include #include @@ -1326,9 +1327,12 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) { int ret; - ret = do_cpu_up(cpu, tidle); - if (ret) - return ret; + /* If parallel AP bringup isn't enabled, perform the first steps now. */ + if (!do_parallel_bringup) { + ret = do_cpu_up(cpu, tidle); + if (ret) + return ret; + } ret = do_wait_cpu_initialized(cpu); if (ret) @@ -1350,6 +1354,12 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) return ret; } +/* Bringup step one: Send INIT/SIPI to the target AP */ +static int native_cpu_kick(unsigned int cpu) +{ + return do_cpu_up(cpu, idle_thread_get(cpu)); +} + /** * arch_disable_smp_support() - disables SMP support for x86 at runtime */ @@ -1565,6 +1575,11 @@ void __init native_smp_prepare_cpus(unsigned int max_cpus) smpboot_control = STARTUP_SECONDARY | STARTUP_APICID_CPUID_01; } + if (do_parallel_bringup) { + cpuhp_setup_state_nocalls(CPUHP_BP_PARALLEL_DYN, "x86/cpu:kick", + native_cpu_kick, NULL); + } + snp_set_wakeup_secondary_cpu(); } From patchwork Tue Feb 7 23:04:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 54111 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp3128892wrn; Tue, 7 Feb 2023 15:09:32 -0800 (PST) X-Google-Smtp-Source: AK7set/lLGbmBU5cIEuZSWm4Krs1TvdPxyV6LfE17mX4f2FZf8yNGVRrhG5CBUbGEaktF3iyuwje X-Received: by 2002:a17:906:64b:b0:882:1b70:8962 with SMTP id t11-20020a170906064b00b008821b708962mr5160124ejb.76.1675811372019; Tue, 07 Feb 2023 15:09:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675811372; cv=none; d=google.com; s=arc-20160816; b=qiOSmB311fjL8Hl853MpLhD2kKoFBjy/emf2OsqqHHLr/ZH+B3U9ElBnw+zS/4/dEt Jz0by5gNOI0JzFHCUmXJTtzpFth0C+2UCB6DmZz45aRP6FfewtnVuQiDlE8mPaLA95iu U0hOnbir/oQ9NjmLLeOaBTu0yIwELCjacKlM1rmXLTAl1P0oZnyzm2yjPTIb0sf1Zbgd ygCd6DxJdCiRNyWOQ3EvIi+9X19Me8jS487s123nutot5pFgLH4+pGSSkZCwiBIccxER U3GS4YY8W8M2b9dallMETkKlwm8/kp+9yjOTBLKJeFQi0EyZf6kS7f0IaUQmL5rSRak/ EsYw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=J+ne+m/jA9f2Vf5iapwS72QpCbb1ZulqNbxU7adH9sw=; b=snx45Xs/ZmswrI/S3xQudplN9CLdvC0zHxXEA5//PwRaRZqkXGvkWLLO8cze9io0AD 0NtUX5MMktg6eG11xvR0P0aqKkR6L6QU5p6EFCp0Nwd4Xh9RqadUqccF50tPmpxlYIFD TfX7sY4j5N2cePhHjOEojaOSJAPqN/sqeqyQhwIuJGo6kBW8pHL9VSHWzRSrGRuEoKiW ByH+54Li4pqfophz68kQg0NWk92t64e7nMuZCL2fEx+hpOmUw8gA6Dzi9ovf3tkfxI/+ oJr+wr+oES+8KpwOLf2mQlNoO8pyxPmGo++3K/DVEC9JL8UtTekbLH7VbR+ZLIICTcvU e2Qg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=2C88stP2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f30-20020a170906739e00b0087329ff5934si778408ejl.0.2023.02.07.15.09.09; Tue, 07 Feb 2023 15:09:31 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=2C88stP2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230150AbjBGXFy (ORCPT + 99 others); Tue, 7 Feb 2023 18:05:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49516 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230106AbjBGXF1 (ORCPT ); Tue, 7 Feb 2023 18:05:27 -0500 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 788A3402FA for ; Tue, 7 Feb 2023 15:04:48 -0800 (PST) Received: by mail-wm1-x32c.google.com with SMTP id k8-20020a05600c1c8800b003dc57ea0dfeso215060wms.0 for ; Tue, 07 Feb 2023 15:04:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=J+ne+m/jA9f2Vf5iapwS72QpCbb1ZulqNbxU7adH9sw=; b=2C88stP2TFJvBZHdAdu647aiDvm7KBHA7Lu0B+BBWdqyCEQeJLMGUsMHlB2FlDhDqO wuXthNeCkcLssbQ7FMVcsFA4v0DApgjRF+5hhhVHKkZi69knw/IavDqXyj8ACKgbiBUk O6SNrNzgs36ODAGNF818joieBXKsBnFNJP03FfcvWq2ml1/pXOyJyxk1OgCglFR8V+I1 YbEBq3CONDk/z/EWZrqQ675YTlmRwH6WmgQPS8PaRbeLNQbpnEUs4BgpFstOI3LxrE6/ DWo3OVYj4bbQBFZ4rwqKvIbgC3sfUkq1+qMnZkBmWfAtbd4L9fUTuYhiQEZ2ZdLq02yx pW6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=J+ne+m/jA9f2Vf5iapwS72QpCbb1ZulqNbxU7adH9sw=; b=7qF5J9sw7UspxSg8tYfBfT0GVNsqpSJDDWN0i5hAomr40Pep2cCt4ErAtmMvh2PHSG s1gHibKeRHMLsJq0+Rns7N8YbmY3YoMMX87fAW4pSt7LnqAGt5dJUxw/3kERIBjf5MJP E2Q88lYGyKmQuZ65UJ8iQ67NtrKVtYDLWTlTrEH+fMXz4INxFjgWHaoYOaiRzERh1JDl VqDTzMEsFxPm5PzmeclHtPsPHmZeqAHBNEMK37BFYMpFJGyELoUhBB8Mcf9nwj141V1S WsjTbqBWMm92Tu3IjU6d7TvTMLyJbz6AQvGfjNSx7hyuqU/ONDScgmnjYgzwuBnI+ixl RonA== X-Gm-Message-State: AO0yUKXRha4q4MWDHkif54t4QHOOqds9730Au0zkVOQh7m+7thX+seXe +oFq1Rk9FiuW+EwiZ8LL6k+ZcQ== X-Received: by 2002:a05:600c:807:b0:3dc:eaef:c1bb with SMTP id k7-20020a05600c080700b003dceaefc1bbmr4449336wmp.35.1675811088116; Tue, 07 Feb 2023 15:04:48 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:c04f:2463:c151:8b87]) by smtp.gmail.com with ESMTPSA id j14-20020a05600c190e00b003dcc82ce53fsm146485wmq.38.2023.02.07.15.04.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Feb 2023 15:04:47 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com Cc: arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse , Usama Arif Subject: [PATCH v7 8/9] x86/mtrr: Avoid repeated save of MTRRs on boot-time CPU bringup Date: Tue, 7 Feb 2023 23:04:35 +0000 Message-Id: <20230207230436.2690891-9-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230207230436.2690891-1-usama.arif@bytedance.com> References: <20230207230436.2690891-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757215584762985857?= X-GMAIL-MSGID: =?utf-8?q?1757215584762985857?= From: David Woodhouse There's no need to repeatedly save the BSP's MTRRs for each AP being brought up at boot time. And no need to use smp_call_function_single() even for the one time they do need to be saved. Signed-off-by: David Woodhouse Signed-off-by: Usama Arif --- arch/x86/kernel/cpu/mtrr/mtrr.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/arch/x86/kernel/cpu/mtrr/mtrr.c b/arch/x86/kernel/cpu/mtrr/mtrr.c index 783f3210d582..b6eae3ad4414 100644 --- a/arch/x86/kernel/cpu/mtrr/mtrr.c +++ b/arch/x86/kernel/cpu/mtrr/mtrr.c @@ -721,11 +721,20 @@ void __init mtrr_bp_init(void) */ void mtrr_save_state(void) { + static bool mtrr_saved; int first_cpu; if (!mtrr_enabled()) return; + if (system_state < SYSTEM_RUNNING) { + if (!mtrr_saved) { + mtrr_save_fixed_ranges(NULL); + mtrr_saved = true; + } + return; + } + first_cpu = cpumask_first(cpu_online_mask); smp_call_function_single(first_cpu, mtrr_save_fixed_ranges, NULL, 1); } From patchwork Tue Feb 7 23:04:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 54110 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp3128688wrn; Tue, 7 Feb 2023 15:09:08 -0800 (PST) X-Google-Smtp-Source: AK7set+cByI3ZRWD7C/c5hzDzGqN7BY49pfq7IPyfEWIc0qd+FEzbTmBTm5MTHhHcwvldvhKk78X X-Received: by 2002:a17:906:2ed4:b0:888:a32d:b50 with SMTP id s20-20020a1709062ed400b00888a32d0b50mr5090203eji.40.1675811348273; Tue, 07 Feb 2023 15:09:08 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675811348; cv=none; d=google.com; s=arc-20160816; b=Tdknk5VqYMemzjIlm19IDOTeqxsgmOaSgEamXd2qxvYheJevVg+PpzN3qRApF0VRw2 LCQdPgDZJcI3gXbAeBFo7lZf5sIfcID982j3jevVBkqIonkETYqGL7wih7HkF6BGeCFT 7vlLz7l51SbGRsAe17nbHVsJ9NbJ+2RWay7CtPNtuorTjoHGClOTvx5yhQROUr0owMw3 fvnWJwr3XRLYmSWS/vygf7p4l7jTjJ0pMb7gl3dSyRqqTkjq+2lwcSXvdj6ZBoKBD5Kb RXtOs9pOvbQw2I13ThsgnMZYiMJTljJD5ysEnfKBZH62uZANarfj+4L59NGwfcLQHrvG SPrA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=KVqUJYOx2vTuosUua8hJoGt0geRYeRPq5H8k23kPkBg=; b=WDY33luL/x44xpf2GwVV21BGmz5fsK9129TGEkzX9mNiLJvnvh2BPNj/BCdmyO1bmS TppSxra16/vq7/ApjbObz0eiE2tngDjQD2LEW9tQuSPOZimAkSfASC4gCQEagOZRLeRC +vXykXKTXOrxr0jfJdmhfATYWydwRR4AxfLx1/jaVY7OS8R+9sZk5Tqd62+e1uYcsEmW nnFqK4kAKrdDOlzdb6NZ6m7cS56BBJJ8Q7aAzG6iNs6+2pXVzWKbv0vNIqVgYd3kzt7l w4v26vVgs8RaB/WqoPwttFVMl8oH9tvhSaPK0EZeX1qham1LrO3FvfTZAIjNJrwLqKAs MHpw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b="K/XKPjBx"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id g16-20020a1709065d1000b008778924dd6esi1273765ejt.958.2023.02.07.15.08.44; Tue, 07 Feb 2023 15:09:08 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b="K/XKPjBx"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229709AbjBGXGp (ORCPT + 99 others); Tue, 7 Feb 2023 18:06:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49534 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230224AbjBGXF2 (ORCPT ); Tue, 7 Feb 2023 18:05:28 -0500 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7948D40BC5 for ; Tue, 7 Feb 2023 15:04:49 -0800 (PST) Received: by mail-wm1-x32b.google.com with SMTP id m16-20020a05600c3b1000b003dc4050c94aso196937wms.4 for ; Tue, 07 Feb 2023 15:04:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KVqUJYOx2vTuosUua8hJoGt0geRYeRPq5H8k23kPkBg=; b=K/XKPjBxiVvpGnCBsaHZtP2MmfmWYWpHG1S9ROqPdhk3ZU25g/JOlC+1MkQQErcuGq aONBfjQdDeEo8jMzDH3KrgKg2XEZOAhYuB17+A2g8pVL3/ADw6KQNGnoisVnwPqJ5SFi UXa0Xf/0IgntZAtCY3uyoJA/fMaeKNtZEhGCWxIUnlczw8Gp7HPlvDIeMFGDNVO7buQ8 lhMtml6oj25JiR4LpJ2OdEqpI8kxrAi0e6fDTRKQpWZi6OdDen2rlSYuhPvBunudbzXZ Vg0c59FaCOlEtUptoe12uB9Os165CWXPoSJhwUQ+94sebZ/KKvWoeAGea+d7zhHC1TTl Pdbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KVqUJYOx2vTuosUua8hJoGt0geRYeRPq5H8k23kPkBg=; b=azjYey3qfTNP7vPltG2TqkOUxOOK4FIx1esDizggc0/P4fnh/jpj8EfuHsaBwF/Me9 MKeZ/Ep/hAukdFC6g2BvYzwTdBsfsY7e9YX0XW9p1wR6QSJe1QI/UpZg/pnsR5IwPoDM l9gJCmWaJgV2+YGGthVO+w4KhAIeZ/FFFu5xo64N5eVDO4TJL1ORCAPeziw7ERXRPm3B uQvakn41O2Dvm/EsoRnkMfx4KL9jx2meDqnQNmBwgF2sYi19RgDnln41e6zrVQoDKfJt J+y/M/JEvgN0ETsAuFA3mwu5035uYXD2nGufMKUUF9S/DHmM9VyxWFYif6SA/CVg62BK 0qjQ== X-Gm-Message-State: AO0yUKWX+y+GNdr5Eo1q1PNcOMakAip6TSxMh8lgVtKuKD+g3zNgIiwV mvhesGpfV6GrQo6VHVlP7Eb51w== X-Received: by 2002:a05:600c:807:b0:3dc:eaef:c1bb with SMTP id k7-20020a05600c080700b003dceaefc1bbmr4449372wmp.35.1675811088984; Tue, 07 Feb 2023 15:04:48 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:c04f:2463:c151:8b87]) by smtp.gmail.com with ESMTPSA id j14-20020a05600c190e00b003dcc82ce53fsm146485wmq.38.2023.02.07.15.04.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Feb 2023 15:04:48 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com Cc: arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse , Usama Arif Subject: [PATCH v7 9/9] x86/smpboot: Serialize topology updates for secondary bringup Date: Tue, 7 Feb 2023 23:04:36 +0000 Message-Id: <20230207230436.2690891-10-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230207230436.2690891-1-usama.arif@bytedance.com> References: <20230207230436.2690891-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757215560080552802?= X-GMAIL-MSGID: =?utf-8?q?1757215560080552802?= From: David Woodhouse The toplogy update is performed by the AP via smp_callin() after the BSP has called do_wait_cpu_initialized(), setting the AP's bit in cpu_callout_mask to allow it to proceed. In preparation to enable further parallelism of AP bringup, add locking to serialize the update even if multiple APs are (in future) permitted to proceed through the next stages of bringup in parallel. Without such ordering (and with that future extra parallelism), confusion ensues: [ 1.360149] x86: Booting SMP configuration: [ 1.360221] .... node #0, CPUs: #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15 #16 #17 #18 #19 #20 #21 #22 #23 [ 1.366225] .... node #1, CPUs: #24 #25 #26 #27 #28 #29 #30 #31 #32 #33 #34 #35 #36 #37 #38 #39 #40 #41 #42 #43 #44 #45 #46 #47 [ 1.370219] .... node #0, CPUs: #48 #49 #50 #51 #52 #53 #54 #55 #56 #57 #58 #59 #60 #61 #62 #63 #64 #65 #66 #67 #68 #69 #70 #71 [ 1.378226] .... node #1, CPUs: #72 #73 #74 #75 #76 #77 #78 #79 #80 #81 #82 #83 #84 #85 #86 #87 #88 #89 #90 #91 #92 #93 #94 #95 [ 1.382037] Brought 96 CPUs to x86/cpu:kick in 72232606 cycles [ 0.104104] smpboot: CPU 26 Converting physical 0 to logical die 1 [ 0.104104] smpboot: CPU 27 Converting physical 1 to logical package 2 [ 0.104104] smpboot: CPU 24 Converting physical 1 to logical package 3 [ 0.104104] smpboot: CPU 27 Converting physical 0 to logical die 2 [ 0.104104] smpboot: CPU 25 Converting physical 1 to logical package 4 [ 1.385609] Brought 96 CPUs to x86/cpu:wait-init in 9269218 cycles [ 1.395285] Brought CPUs online in 28930764 cycles [ 1.395469] smp: Brought up 2 nodes, 96 CPUs [ 1.395689] smpboot: Max logical packages: 2 [ 1.396222] smpboot: Total of 96 processors activated (576000.00 BogoMIPS) [Usama Arif: fixed rebase conflict] Signed-off-by: David Woodhouse Signed-off-by: Usama Arif --- arch/x86/include/asm/smp.h | 4 +- arch/x86/include/asm/topology.h | 2 - arch/x86/kernel/cpu/common.c | 6 +-- arch/x86/kernel/smpboot.c | 73 ++++++++++++++++++++------------- arch/x86/xen/smp_pv.c | 4 +- 5 files changed, 48 insertions(+), 41 deletions(-) diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h index 33c0d5fd8af6..b4b29e052b6e 100644 --- a/arch/x86/include/asm/smp.h +++ b/arch/x86/include/asm/smp.h @@ -48,8 +48,6 @@ struct smp_ops { }; /* Globals due to paravirt */ -extern void set_cpu_sibling_map(int cpu); - #ifdef CONFIG_SMP extern struct smp_ops smp_ops; @@ -137,7 +135,7 @@ void native_send_call_func_single_ipi(int cpu); void x86_idle_thread_init(unsigned int cpu, struct task_struct *idle); void smp_store_boot_cpu_info(void); -void smp_store_cpu_info(int id); +void smp_store_cpu_info(int id, bool force_single_core); asmlinkage __visible void smp_reboot_interrupt(void); __visible void smp_reschedule_interrupt(struct pt_regs *regs); diff --git a/arch/x86/include/asm/topology.h b/arch/x86/include/asm/topology.h index 458c891a8273..4bccbd949a99 100644 --- a/arch/x86/include/asm/topology.h +++ b/arch/x86/include/asm/topology.h @@ -136,8 +136,6 @@ static inline int topology_max_smt_threads(void) return __max_smt_threads; } -int topology_update_package_map(unsigned int apicid, unsigned int cpu); -int topology_update_die_map(unsigned int dieid, unsigned int cpu); int topology_phys_to_logical_pkg(unsigned int pkg); int topology_phys_to_logical_die(unsigned int die, unsigned int cpu); bool topology_is_primary_thread(unsigned int cpu); diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 9cfca3d7d0e2..cf1a4eff7a76 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -1766,7 +1766,7 @@ static void generic_identify(struct cpuinfo_x86 *c) * Validate that ACPI/mptables have the same information about the * effective APIC id and update the package map. */ -static void validate_apic_and_package_id(struct cpuinfo_x86 *c) +static void validate_apic_id(struct cpuinfo_x86 *c) { #ifdef CONFIG_SMP unsigned int apicid, cpu = smp_processor_id(); @@ -1777,8 +1777,6 @@ static void validate_apic_and_package_id(struct cpuinfo_x86 *c) pr_err(FW_BUG "CPU%u: APIC id mismatch. Firmware: %x APIC: %x\n", cpu, apicid, c->initial_apicid); } - BUG_ON(topology_update_package_map(c->phys_proc_id, cpu)); - BUG_ON(topology_update_die_map(c->cpu_die_id, cpu)); #else c->logical_proc_id = 0; #endif @@ -1969,7 +1967,7 @@ void identify_secondary_cpu(struct cpuinfo_x86 *c) #ifdef CONFIG_X86_32 enable_sep_cpu(); #endif - validate_apic_and_package_id(c); + validate_apic_id(c); x86_spec_ctrl_setup_ap(); update_srbds_msr(); diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 7e7bcab6676e..8ffec5de2e2e 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -180,16 +180,12 @@ static void smp_callin(void) apic_ap_setup(); /* - * Save our processor parameters. Note: this information - * is needed for clock calibration. - */ - smp_store_cpu_info(cpuid); - - /* + * Save our processor parameters and update topology. + * Note: this information is needed for clock calibration. * The topology information must be up to date before * calibrate_delay() and notify_cpu_starting(). */ - set_cpu_sibling_map(raw_smp_processor_id()); + smp_store_cpu_info(cpuid, false); ap_init_aperfmperf(); @@ -244,6 +240,12 @@ static void notrace start_secondary(void *unused) * smp_callout_mask to release them. */ cpu_init_secondary(); + + /* + * Even though notify_cpu_starting() will do this, it does so too late + * as the AP may already have triggered lockdep splats by then. See + * commit 29368e093 ("x86/smpboot: Move rcu_cpu_starting() earlier"). + */ rcu_cpu_starting(raw_smp_processor_id()); x86_cpuinit.early_percpu_clock_init(); @@ -352,7 +354,7 @@ EXPORT_SYMBOL(topology_phys_to_logical_die); * @pkg: The physical package id as retrieved via CPUID * @cpu: The cpu for which this is updated */ -int topology_update_package_map(unsigned int pkg, unsigned int cpu) +static int topology_update_package_map(unsigned int pkg, unsigned int cpu) { int new; @@ -375,7 +377,7 @@ int topology_update_package_map(unsigned int pkg, unsigned int cpu) * @die: The die id as retrieved via CPUID * @cpu: The cpu for which this is updated */ -int topology_update_die_map(unsigned int die, unsigned int cpu) +static int topology_update_die_map(unsigned int die, unsigned int cpu) { int new; @@ -406,25 +408,7 @@ void __init smp_store_boot_cpu_info(void) c->initialized = true; } -/* - * The bootstrap kernel entry code has set these up. Save them for - * a given CPU - */ -void smp_store_cpu_info(int id) -{ - struct cpuinfo_x86 *c = &cpu_data(id); - - /* Copy boot_cpu_data only on the first bringup */ - if (!c->initialized) - *c = boot_cpu_data; - c->cpu_index = id; - /* - * During boot time, CPU0 has this setup already. Save the info when - * bringing up AP or offlined CPU0. - */ - identify_secondary_cpu(c); - c->initialized = true; -} +static arch_spinlock_t topology_lock = __ARCH_SPIN_LOCK_UNLOCKED; static bool topology_same_node(struct cpuinfo_x86 *c, struct cpuinfo_x86 *o) @@ -630,7 +614,7 @@ static struct sched_domain_topology_level x86_topology[] = { */ static bool x86_has_numa_in_package; -void set_cpu_sibling_map(int cpu) +static void set_cpu_sibling_map(int cpu) { bool has_smt = smp_num_siblings > 1; bool has_mp = has_smt || boot_cpu_data.x86_max_cores > 1; @@ -709,6 +693,37 @@ void set_cpu_sibling_map(int cpu) } } +/* + * The bootstrap kernel entry code has set these up. Save them for + * a given CPU + */ +void smp_store_cpu_info(int id, bool force_single_core) +{ + struct cpuinfo_x86 *c = &cpu_data(id); + + /* Copy boot_cpu_data only on the first bringup */ + if (!c->initialized) + *c = boot_cpu_data; + c->cpu_index = id; + /* + * During boot time, CPU0 has this setup already. Save the info when + * bringing up AP or offlined CPU0. + */ + identify_secondary_cpu(c); + + arch_spin_lock(&topology_lock); + BUG_ON(topology_update_package_map(c->phys_proc_id, id)); + BUG_ON(topology_update_die_map(c->cpu_die_id, id)); + c->initialized = true; + + /* For Xen PV */ + if (force_single_core) + c->x86_max_cores = 1; + + set_cpu_sibling_map(id); + arch_spin_unlock(&topology_lock); +} + /* maps the cpu to the sched domain representing multi-core */ const struct cpumask *cpu_coregroup_mask(int cpu) { diff --git a/arch/x86/xen/smp_pv.c b/arch/x86/xen/smp_pv.c index 6175f2c5c822..09f94f940689 100644 --- a/arch/x86/xen/smp_pv.c +++ b/arch/x86/xen/smp_pv.c @@ -71,9 +71,7 @@ static void cpu_bringup(void) xen_enable_syscall(); } cpu = smp_processor_id(); - smp_store_cpu_info(cpu); - cpu_data(cpu).x86_max_cores = 1; - set_cpu_sibling_map(cpu); + smp_store_cpu_info(cpu, true); speculative_store_bypass_ht_init();