From patchwork Thu Mar 16 22:20:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 70990 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp25645wrt; Thu, 16 Mar 2023 15:23:18 -0700 (PDT) X-Google-Smtp-Source: AK7set/NhE8R1XLioD9vsOJ1oNQ9ErZ9T4CPkvOIeM0PMh6U7mIa+s/xizP7exHfuEJY22zcljii X-Received: by 2002:a17:90b:3b84:b0:234:13a3:6e67 with SMTP id pc4-20020a17090b3b8400b0023413a36e67mr5712204pjb.12.1679005398467; Thu, 16 Mar 2023 15:23:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679005398; cv=none; d=google.com; s=arc-20160816; b=E1UhY/rYCkFG3CJeBHV9RunKqSO73RXirpTE2ruD9sT6V5Z7ENvmo20ToNIkZ5TbPQ +/hfQyG0+LAaLqPOFchU0j3IJqHPvqgDkMc/XFCIoVYP3Sj1YPWRk8vsHjYnyh74Iywk g9Z5Synmt956+BRfcI8300gzfkgx5NykWxTzBRY4ID4N2JGK6toP+szfceUWxqXtbSdE TsyomIWsH/nIQF/qy4RMFwJAyN0kp237skGSMLyX7CgFqneb5o7KTTncJgGcxzHTt7eD zFDLqCZ7vfRpP7yl7tZD485KeXplUpVC5ZH4DYcmIdSojVu+GsL6722B5r5s61P/AgLe B56A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=mbhas2uqRPBmSN9AHCLEH2K+C2q38zTMyGQqhw7QwxQ=; b=JGxEiaS9CJJFXLdcI18XSsnU3U7xWYKDhJgzFNnoVvqe8J0sV+7YrXK8ddBLdnb5LU 0xaNo+M1aM5LLwpt+Op+7spBZW/8/gC5Gw2x29LwAV9AtP6VsCmRW8+Xkz92vfNs2DYU yA/IXwC7ibz6l+En7U7Q3k8zsv80HK51K0VuAvV6VXb9Q+YJpAhtF7/iki/r4TLLcAIa n4sObW9b5zWDQvoYkJhrmCke0E4TfmOIPFu8XAFWgwEYJHF/1SEong6KThjHcw55C/6C EI+QNYJ7CQh0jwzPjwr1OD5n8aYxmeluUR/CDd8JQgdTvFRY4oieM3Pif9FO6QRt/LCE ea9A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=G887ySyT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id mq8-20020a17090b380800b0023a6ea894a0si2885984pjb.88.2023.03.16.15.23.03; Thu, 16 Mar 2023 15:23:18 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=G887ySyT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230227AbjCPWVq (ORCPT + 99 others); Thu, 16 Mar 2023 18:21:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60642 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230133AbjCPWVn (ORCPT ); Thu, 16 Mar 2023 18:21:43 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AF264B06FA for ; Thu, 16 Mar 2023 15:21:14 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id q16so2880557wrw.2 for ; Thu, 16 Mar 2023 15:21:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1679005273; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mbhas2uqRPBmSN9AHCLEH2K+C2q38zTMyGQqhw7QwxQ=; b=G887ySyTaFh7l9kGNOaV9CvLHAce3lWKFnKxqTdjw2vofQU38EOn5pnpjDsI0yDU6v SUwTkKq2r0LiBMG6w1SJHI7HzH2lnEriAHZ9LCL4jhHJ3opc8byF8aic/zk1LaILitSv mJYhzqlQq2CGWJKxnIPRCO1FHG+jDQ3iJR7+D2QJbtAxzs4Fe0/RuVECwISJEfCKAM1e hezGoZjpOwomLa5loLdWbo2bv0RUqVPJ4UgkVz2hg8RJEvrSXK+1x1fWOM/CaRIYeGiB WxHdcQixkuUtJi095NoYhlgeVVYfctKmKJuECCDkQ+6qFHIhn9Py1fRO/tm/mmjRfo81 qyIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679005273; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mbhas2uqRPBmSN9AHCLEH2K+C2q38zTMyGQqhw7QwxQ=; b=UPP3LwE1Oi0o7v0Gjww9QIo1FvDXf1byJtHl8zxp2E37sEoCfr8XIq3t1866u97Wff FX1W1FANo7WhGT5jo/Iqa+8MfI1Y1VlQ3/5BAKcOcCBWnvyvq3BsBHQNFp6LEvmxhHI6 0oskpc0HznQtX0q7xihp1GcQrkkJ4Ypd7bhvK/ss9KkT0D8SajoTu+idJkhfxPlIYloh TUu/VkzsCCZRApPvBdLHN6rEhNNf1iE4Ybn+5rO188mg68PBQFr7pONkWqk1GyBNAN1+ SPP0DM/i9XQ199PuZHUTsSbCe7DSDrHmtDmSUAEG2zNbWh01S518xLqHxo3rXpXEK2/j l1Ug== X-Gm-Message-State: AO0yUKU61aJ6tdjM+xtFQpymygXegxKolRinHBTGUiPRdBLzhF2Y9FkC mrnyrfNjDrQVinxKnza4m0tyvg== X-Received: by 2002:adf:e541:0:b0:2ce:3ae0:55a7 with SMTP id z1-20020adfe541000000b002ce3ae055a7mr5357953wrm.29.1679005273108; Thu, 16 Mar 2023 15:21:13 -0700 (PDT) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:4b87:78c3:3abe:7b0d]) by smtp.gmail.com with ESMTPSA id f9-20020adff989000000b002cea392f000sm439256wrr.69.2023.03.16.15.21.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 16 Mar 2023 15:21:12 -0700 (PDT) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, gpiccoli@igalia.com, David Woodhouse , Usama Arif Subject: [PATCH v15 01/12] x86/apic/x2apic: Allow CPU cluster_mask to be populated in parallel Date: Thu, 16 Mar 2023 22:20:58 +0000 Message-Id: <20230316222109.1940300-2-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230316222109.1940300-1-usama.arif@bytedance.com> References: <20230316222109.1940300-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760564764709889399?= X-GMAIL-MSGID: =?utf-8?q?1760564764709889399?= From: David Woodhouse Each of the sibling CPUs in a cluster uses the same clustermask. The first CPU in a cluster will need a new clustermask allocated, while subsequent siblings will use the same clustermask as the first. However, the CPU being brought up cannot yet perform memory allocations at the point that this occurs in init_x2apic_ldr(). So at present, the alloc_clustermask() function allocates a clustermask just in case it's needed, storing it in the global cluster_hotplug_mask. A CPU which is the first sibling of a cluster will "take" it from there and set cluster_hotplug_mask to NULL, in order for alloc_clustermask() to allocate a new one before bringing up the next CPU. To facilitate parallel bringup of CPUs in future, switch to a model where alloc_clustermask() prepopulates the clustermask in the per_cpu data for each present CPU in the cluster in advance. All that the CPU needs to do for itself in init_x2apic_ldr() is set its own bit in that mask. The 'node' and 'clusterid' members of struct cluster_mask are thus redundant, and it can become a simple struct cpumask instead. Suggested-by: Thomas Gleixner Signed-off-by: David Woodhouse Signed-off-by: Usama Arif Tested-by: Paul E. McKenney Tested-by: Kim Phillips Tested-by: Oleksandr Natalenko Tested-by: Guilherme G. Piccoli --- arch/x86/kernel/apic/x2apic_cluster.c | 126 +++++++++++++++++--------- 1 file changed, 82 insertions(+), 44 deletions(-) diff --git a/arch/x86/kernel/apic/x2apic_cluster.c b/arch/x86/kernel/apic/x2apic_cluster.c index e696e22d0531..b2b2b7f3e03f 100644 --- a/arch/x86/kernel/apic/x2apic_cluster.c +++ b/arch/x86/kernel/apic/x2apic_cluster.c @@ -9,11 +9,7 @@ #include "local.h" -struct cluster_mask { - unsigned int clusterid; - int node; - struct cpumask mask; -}; +#define apic_cluster(apicid) ((apicid) >> 4) /* * __x2apic_send_IPI_mask() possibly needs to read @@ -23,8 +19,7 @@ struct cluster_mask { static u32 *x86_cpu_to_logical_apicid __read_mostly; static DEFINE_PER_CPU(cpumask_var_t, ipi_mask); -static DEFINE_PER_CPU_READ_MOSTLY(struct cluster_mask *, cluster_masks); -static struct cluster_mask *cluster_hotplug_mask; +static DEFINE_PER_CPU_READ_MOSTLY(struct cpumask *, cluster_masks); static int x2apic_acpi_madt_oem_check(char *oem_id, char *oem_table_id) { @@ -60,10 +55,10 @@ __x2apic_send_IPI_mask(const struct cpumask *mask, int vector, int apic_dest) /* Collapse cpus in a cluster so a single IPI per cluster is sent */ for_each_cpu(cpu, tmpmsk) { - struct cluster_mask *cmsk = per_cpu(cluster_masks, cpu); + struct cpumask *cmsk = per_cpu(cluster_masks, cpu); dest = 0; - for_each_cpu_and(clustercpu, tmpmsk, &cmsk->mask) + for_each_cpu_and(clustercpu, tmpmsk, cmsk) dest |= x86_cpu_to_logical_apicid[clustercpu]; if (!dest) @@ -71,7 +66,7 @@ __x2apic_send_IPI_mask(const struct cpumask *mask, int vector, int apic_dest) __x2apic_send_IPI_dest(dest, vector, APIC_DEST_LOGICAL); /* Remove cluster CPUs from tmpmask */ - cpumask_andnot(tmpmsk, tmpmsk, &cmsk->mask); + cpumask_andnot(tmpmsk, tmpmsk, cmsk); } local_irq_restore(flags); @@ -105,55 +100,98 @@ static u32 x2apic_calc_apicid(unsigned int cpu) static void init_x2apic_ldr(void) { - struct cluster_mask *cmsk = this_cpu_read(cluster_masks); - u32 cluster, apicid = apic_read(APIC_LDR); - unsigned int cpu; + struct cpumask *cmsk = this_cpu_read(cluster_masks); - x86_cpu_to_logical_apicid[smp_processor_id()] = apicid; + BUG_ON(!cmsk); - if (cmsk) - goto update; - - cluster = apicid >> 16; - for_each_online_cpu(cpu) { - cmsk = per_cpu(cluster_masks, cpu); - /* Matching cluster found. Link and update it. */ - if (cmsk && cmsk->clusterid == cluster) - goto update; + cpumask_set_cpu(smp_processor_id(), cmsk); +} + +/* + * As an optimisation during boot, set the cluster_mask for all present + * CPUs at once, to prevent each of them having to iterate over the others + * to find the existing cluster_mask. + */ +static void prefill_clustermask(struct cpumask *cmsk, unsigned int cpu, u32 cluster) +{ + int cpu_i; + + for_each_present_cpu(cpu_i) { + struct cpumask **cpu_cmsk = &per_cpu(cluster_masks, cpu_i); + u32 apicid = apic->cpu_present_to_apicid(cpu_i); + + if (apicid == BAD_APICID || cpu_i == cpu || apic_cluster(apicid) != cluster) + continue; + + if (WARN_ON_ONCE(*cpu_cmsk == cmsk)) + continue; + + BUG_ON(*cpu_cmsk); + *cpu_cmsk = cmsk; } - cmsk = cluster_hotplug_mask; - cmsk->clusterid = cluster; - cluster_hotplug_mask = NULL; -update: - this_cpu_write(cluster_masks, cmsk); - cpumask_set_cpu(smp_processor_id(), &cmsk->mask); } -static int alloc_clustermask(unsigned int cpu, int node) +static int alloc_clustermask(unsigned int cpu, u32 cluster, int node) { + struct cpumask *cmsk = NULL; + unsigned int cpu_i; + + /* + * At boot time, the CPU present mask is stable. The cluster mask is + * allocated for the first CPU in the cluster and propagated to all + * present siblings in the cluster. If the cluster mask is already set + * on entry to this function for a given CPU, there is nothing to do. + */ if (per_cpu(cluster_masks, cpu)) return 0; + + if (system_state < SYSTEM_RUNNING) + goto alloc; + /* - * If a hotplug spare mask exists, check whether it's on the right - * node. If not, free it and allocate a new one. + * On post boot hotplug for a CPU which was not present at boot time, + * iterate over all possible CPUs (even those which are not present + * any more) to find any existing cluster mask. */ - if (cluster_hotplug_mask) { - if (cluster_hotplug_mask->node == node) - return 0; - kfree(cluster_hotplug_mask); + for_each_possible_cpu(cpu_i) { + u32 apicid = apic->cpu_present_to_apicid(cpu_i); + + if (apicid != BAD_APICID && apic_cluster(apicid) == cluster) { + cmsk = per_cpu(cluster_masks, cpu_i); + /* + * If the cluster is already initialized, just store + * the mask and return. There's no need to propagate. + */ + if (cmsk) { + per_cpu(cluster_masks, cpu) = cmsk; + return 0; + } + } } - - cluster_hotplug_mask = kzalloc_node(sizeof(*cluster_hotplug_mask), - GFP_KERNEL, node); - if (!cluster_hotplug_mask) + /* + * No CPU in the cluster has ever been initialized, so fall through to + * the boot time code which will also populate the cluster mask for any + * other CPU in the cluster which is (now) present. + */ +alloc: + cmsk = kzalloc_node(sizeof(*cmsk), GFP_KERNEL, node); + if (!cmsk) return -ENOMEM; - cluster_hotplug_mask->node = node; + per_cpu(cluster_masks, cpu) = cmsk; + prefill_clustermask(cmsk, cpu, cluster); + return 0; } static int x2apic_prepare_cpu(unsigned int cpu) { - if (alloc_clustermask(cpu, cpu_to_node(cpu)) < 0) + u32 phys_apicid = apic->cpu_present_to_apicid(cpu); + u32 cluster = apic_cluster(phys_apicid); + u32 logical_apicid = (cluster << 16) | (1 << (phys_apicid & 0xf)); + + x86_cpu_to_logical_apicid[cpu] = logical_apicid; + + if (alloc_clustermask(cpu, cluster, cpu_to_node(cpu)) < 0) return -ENOMEM; if (!zalloc_cpumask_var(&per_cpu(ipi_mask, cpu), GFP_KERNEL)) return -ENOMEM; @@ -162,10 +200,10 @@ static int x2apic_prepare_cpu(unsigned int cpu) static int x2apic_dead_cpu(unsigned int dead_cpu) { - struct cluster_mask *cmsk = per_cpu(cluster_masks, dead_cpu); + struct cpumask *cmsk = per_cpu(cluster_masks, dead_cpu); if (cmsk) - cpumask_clear_cpu(dead_cpu, &cmsk->mask); + cpumask_clear_cpu(dead_cpu, cmsk); free_cpumask_var(per_cpu(ipi_mask, dead_cpu)); return 0; } From patchwork Thu Mar 16 22:20:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 70999 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp27300wrt; Thu, 16 Mar 2023 15:28:03 -0700 (PDT) X-Google-Smtp-Source: AK7set+XOz/jwYbHf0wpB6YPQKvAYv9tmuRarM27/WHOq6rFzCMBSeAQFIzX3daATLrhcDi+l0sq X-Received: by 2002:a17:903:188:b0:19d:323:e68 with SMTP id z8-20020a170903018800b0019d03230e68mr5003350plg.1.1679005683293; Thu, 16 Mar 2023 15:28:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679005683; cv=none; d=google.com; s=arc-20160816; b=LOGystpinoetOFx9dllQdBYx0R811nGcmHjxwROncck0BJX2sb6LvPmCPNEFS2p7Na +rSZ0lT6FEE5g/cftL8hz9S5F1+I2LSiU8mMUTZ4bN37sL1pqMMC9DMjDQRJqt/fGm0e e7uATxBG7xWT1D07aMxD/SqawpjWQA3Qa8s7QeJsKDf71OnN836HO8NgalBNmb/COEso BY9HJJCd5E6dQZLkDOltLMWzEsT6NY6O2iPs5ZiLGbRJgApzqNTVCveFTdHTNOAgSl7T dTrGH+Ml50Rb7fewB9i9ToFOAblNaBJ5xPSZ7SA8n2FoA3PhJoiq+EAK9l6k43aKHVjZ nXKA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=jZOcJlN4hsoET9GB1lz88zELWFf4YxBH7A+4Yjdmpg0=; b=ElcKr0VBpF6ZGxT8nkOpKDxpso29Ie8/l1B1lPWNINgj8WelG1YASw5pzNLSpWN4uO FyDWu27vPT+Yj/OEwyLAt1GUg4HTI50ogDzzGtCaku6PKJjYhrEz8k5IT09+ga20vEzU pFsoq/x0gtXeWz+LotSHjhFiX/oJtAMwNJ9LEJXMCpZ++Up2u8z7vwTkMU3SkpeCEJ+V 1xaiKARYc87dCREqHZMRJ2fOCIR3bdyKZLcCKqk8tt81lClH6b/aQQoxZ32nyqkNph2y Ml3jw1oIPa5RKsukWSYg2CdrczmIGNEJdoUsNwPPfGalv6CNB/gOPTLk2g69zHAdSguu P2WQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=jHDfLcCc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id n3-20020a170903110300b0019f7977c9cfsi642076plh.319.2023.03.16.15.27.30; Thu, 16 Mar 2023 15:28:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=jHDfLcCc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230223AbjCPWVt (ORCPT + 99 others); Thu, 16 Mar 2023 18:21:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60664 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230149AbjCPWVn (ORCPT ); Thu, 16 Mar 2023 18:21:43 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 986D3B0B8E for ; Thu, 16 Mar 2023 15:21:15 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id p4so2861321wre.11 for ; Thu, 16 Mar 2023 15:21:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1679005274; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jZOcJlN4hsoET9GB1lz88zELWFf4YxBH7A+4Yjdmpg0=; b=jHDfLcCchiqdnozdtUvd3AIzSPgmu8CKRQyfMZhtMflGqrA4sIZVBV11LttR2/WRVQ j6CPPcpqg4vZd+OlXmy1/pRWVAisVuiFqwtLTvO3oTeuan72LXYJ+1Q3PFcPJeLFsUqL MER8KlDOQBcISMTImXF1w3+CZsw9iemukQHpHNfzDc3RIdfsWm+EXE9sTiubnYMTyZRr vkbmS4MJljqZGRK2MUKVO8rbpMle2zqdi6M34Fc9tWxSr2HU5waK82B5J9QwG22kAwB/ CehZoT7fehMX1KmYU3h8N0EhL9xS5LaSrZiNtTY9aH7km5ZeZJYR6ywGwK8T23gdWHjn VrWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679005274; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jZOcJlN4hsoET9GB1lz88zELWFf4YxBH7A+4Yjdmpg0=; b=EY5zQZn5An2CJuVut1/zVjpB+QqgaeGyreL/L8+eRdVzhX2UQxB2Sjy0+6qFV0tn6U n2diNgJJXvUeidIPYMOmUoAt91lWjWdOr4zmQ8qLe/R0EXCMRxhGG6DIAXPsX7RWeC3u wp8F+vR5QEjuEvNCf721B3pNlbfCuqwfMF0/ayx1Ijw9nMqK/8JjAA5F3rbdEcZRbPGf 22znBH/FbTK25ORsbRhEKBjBGP9M6lKTOUNUXukQ2OO7e1TawJFUcsVk2uWlFzvuBlEp /qqprqj/zmRQdJaWv2Ni++I0JBgX5Cj+0PfvPYjOHsXnlQiWq0Ld3smdOqaIV8AbdetB JUhg== X-Gm-Message-State: AO0yUKUZE2sHktn+DB58ff3vzVEY14bGMZh9iYMJ1b0dvHv/sLcmVMmK WfPeLstSAvlC+jog/7eoZUbwCw== X-Received: by 2002:a5d:530d:0:b0:2cf:e2f3:ef46 with SMTP id e13-20020a5d530d000000b002cfe2f3ef46mr5192145wrv.52.1679005273994; Thu, 16 Mar 2023 15:21:13 -0700 (PDT) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:4b87:78c3:3abe:7b0d]) by smtp.gmail.com with ESMTPSA id f9-20020adff989000000b002cea392f000sm439256wrr.69.2023.03.16.15.21.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 16 Mar 2023 15:21:13 -0700 (PDT) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, gpiccoli@igalia.com, David Woodhouse , Usama Arif Subject: [PATCH v15 02/12] cpu/hotplug: Move idle_thread_get() to Date: Thu, 16 Mar 2023 22:20:59 +0000 Message-Id: <20230316222109.1940300-3-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230316222109.1940300-1-usama.arif@bytedance.com> References: <20230316222109.1940300-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760565063412440834?= X-GMAIL-MSGID: =?utf-8?q?1760565063412440834?= From: David Woodhouse Instead of relying purely on the special-case wrapper in bringup_cpu() to pass the idle thread to __cpu_up(), expose idle_thread_get() so that the architecture code can obtain it directly when necessary. This will be useful when the existing __cpu_up() is split into multiple phases, only *one* of which will actually need the idle thread. If the architecture code is to register its new pre-bringup states with the cpuhp core, having a special-case wrapper to pass extra arguments is non-trivial and it's easier just to let the arch register its function pointer to be invoked with the standard API. To reduce duplication, move the shadow stack reset and kasan unpoisoning into idle_thread_get() too. Signed-off-by: David Woodhouse Reviewed-by: Thomas Gleixner Signed-off-by: Usama Arif Tested-by: Paul E. McKenney Tested-by: Kim Phillips Tested-by: Oleksandr Natalenko Tested-by: Guilherme G. Piccoli --- include/linux/smpboot.h | 10 ++++++++++ kernel/cpu.c | 13 +++---------- kernel/smpboot.c | 11 ++++++++++- kernel/smpboot.h | 2 -- 4 files changed, 23 insertions(+), 13 deletions(-) diff --git a/include/linux/smpboot.h b/include/linux/smpboot.h index 9d1bc65d226c..df6417703e4c 100644 --- a/include/linux/smpboot.h +++ b/include/linux/smpboot.h @@ -5,6 +5,16 @@ #include struct task_struct; + +#ifdef CONFIG_GENERIC_SMP_IDLE_THREAD +struct task_struct *idle_thread_get(unsigned int cpu, bool unpoison); +#else +static inline struct task_struct *idle_thread_get(unsigned int cpu, bool unpoison) +{ + return NULL; +} +#endif + /* Cookie handed to the thread_fn*/ struct smpboot_thread_data; diff --git a/kernel/cpu.c b/kernel/cpu.c index 6c0a92ca6bb5..6b3dccb4a888 100644 --- a/kernel/cpu.c +++ b/kernel/cpu.c @@ -31,7 +31,6 @@ #include #include #include -#include #include #include #include @@ -588,15 +587,9 @@ static int bringup_wait_for_ap(unsigned int cpu) static int bringup_cpu(unsigned int cpu) { - struct task_struct *idle = idle_thread_get(cpu); + struct task_struct *idle = idle_thread_get(cpu, true); int ret; - /* - * Reset stale stack state from the last time this CPU was online. - */ - scs_task_reset(idle); - kasan_unpoison_task_stack(idle); - /* * Some architectures have to walk the irq descriptors to * setup the vector space for the cpu which comes online. @@ -614,7 +607,7 @@ static int bringup_cpu(unsigned int cpu) static int finish_cpu(unsigned int cpu) { - struct task_struct *idle = idle_thread_get(cpu); + struct task_struct *idle = idle_thread_get(cpu, false); struct mm_struct *mm = idle->active_mm; /* @@ -1378,7 +1371,7 @@ static int _cpu_up(unsigned int cpu, int tasks_frozen, enum cpuhp_state target) if (st->state == CPUHP_OFFLINE) { /* Let it fail before we try to bring the cpu up */ - idle = idle_thread_get(cpu); + idle = idle_thread_get(cpu, false); if (IS_ERR(idle)) { ret = PTR_ERR(idle); goto out; diff --git a/kernel/smpboot.c b/kernel/smpboot.c index 2c7396da470c..24e81c725e7b 100644 --- a/kernel/smpboot.c +++ b/kernel/smpboot.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include #include @@ -27,12 +28,20 @@ */ static DEFINE_PER_CPU(struct task_struct *, idle_threads); -struct task_struct *idle_thread_get(unsigned int cpu) +struct task_struct *idle_thread_get(unsigned int cpu, bool unpoison) { struct task_struct *tsk = per_cpu(idle_threads, cpu); if (!tsk) return ERR_PTR(-ENOMEM); + + if (unpoison) { + /* + * Reset stale stack state from last time this CPU was online. + */ + scs_task_reset(tsk); + kasan_unpoison_task_stack(tsk); + } return tsk; } diff --git a/kernel/smpboot.h b/kernel/smpboot.h index 34dd3d7ba40b..60c609318ad6 100644 --- a/kernel/smpboot.h +++ b/kernel/smpboot.h @@ -5,11 +5,9 @@ struct task_struct; #ifdef CONFIG_GENERIC_SMP_IDLE_THREAD -struct task_struct *idle_thread_get(unsigned int cpu); void idle_thread_set_boot_cpu(void); void idle_threads_init(void); #else -static inline struct task_struct *idle_thread_get(unsigned int cpu) { return NULL; } static inline void idle_thread_set_boot_cpu(void) { } static inline void idle_threads_init(void) { } #endif From patchwork Thu Mar 16 22:21:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 71000 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp28089wrt; Thu, 16 Mar 2023 15:30:25 -0700 (PDT) X-Google-Smtp-Source: AK7set9nTfIwCePnZrk7fSthpQgL8ZlspDUYkYN1l30VpisYn6kSXG4zDcmzd+qrLWWgzVAzxjAN X-Received: by 2002:a17:90b:4a87:b0:23a:6be8:9446 with SMTP id lp7-20020a17090b4a8700b0023a6be89446mr5117071pjb.48.1679005825676; Thu, 16 Mar 2023 15:30:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679005825; cv=none; d=google.com; s=arc-20160816; b=jBif/n9ac9nBA+hmj4Gr9kF5hwRbElMxDJPfeuL1lm/xF4sxF+zmExwOuiVEgRdklr Xj1FtgM8GKuc7mHFuYa3v/V2GwU2Em6KG3G2iVolaODKm9kAzvR2VIlkfM1B451HNKYb ITchlPdklwp/MzGS5Snkz8YjleouEaSGpwuuhMJpAcCv5RCasowDuCcsSD1JM23U4KpZ popKQHq7NRJqXP5/mgRgsJYcebo1Oy1ZeMpTiRXlfRLSV7sQAIPWjawqCdP1OVCuIVkp X15FYJWyclPLjSMq2qU8FEuqijeDRGueDRNCjq+G1CAX0eK9h+zdbT4zSX+YpjT7biS2 FT2g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ySNFLglM8SJEJlQ1FHHmy3ZddsPF9igYAJti/ai7CSk=; b=ZxyQelvXchTvpvmZBQBdna8kOqKiX1u8SqdJhZLGsdQrwDz3qTM2zQrlmPFEOnLhE9 jiOj2rJDDA6p8KfajCp6fetwl70fN0AGl/tsui7uFakI/4o4iZmNAyUwMHdUgAP2X/7v 2qf9fFfz4Zz0tdkq6g12Y2OSL0YqP2cVgnS5Y85kUlhbXsQ/UiJsrxU+LyJNbmQ8uLMi ewRvTuB8kdC7/9PjnQBgIfnHeAAu75hE/BfX3RzpZXLukk5rmTh23Hq+1PWUBXBw+kN3 aPRf6Zc0NIuFefWd9fkWXUSQRQpQY7652klz5TgCnsdcHAHe0o+H/brIsUaDyVIbgF8a ElGg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=E8lXhN7x; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ds19-20020a17090b08d300b0023750b695e1si5860552pjb.156.2023.03.16.15.30.10; Thu, 16 Mar 2023 15:30:25 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=E8lXhN7x; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229948AbjCPWWE (ORCPT + 99 others); Thu, 16 Mar 2023 18:22:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60700 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230197AbjCPWVo (ORCPT ); Thu, 16 Mar 2023 18:21:44 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 78461B0BA5 for ; Thu, 16 Mar 2023 15:21:16 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id v16so2892529wrn.0 for ; Thu, 16 Mar 2023 15:21:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1679005275; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ySNFLglM8SJEJlQ1FHHmy3ZddsPF9igYAJti/ai7CSk=; b=E8lXhN7xCG0/Utl/fWZI1ZNtGqI9EJIVGD+gymWMWoRdPZAPfvFi+5QKhLol1oOGHn BpYddX/YMFiEjf0YMIdZHef4jpIdVZzFrtXEM+6qB29huW2jgbb2mnXjVF3wKR8EhvD9 Xzixo1CR0PuIHdZiKI/ReOwtirzLrxUlWuGfc73YebJwwIICs7XRmdiUi5G1JO+Bs55z 84uOF5NsaJ9x8USZRZHm+RPpwK9gX9dCuqASSP+NT8+etsVloO37LzZ8ldj+/Mz5yzrj MFnCA11jiCSq0o8Cb0RjIg0Ds/3zIiecSY6GeEgKiOPQXAUuOB0lKRPtRBXgejW4EsAX kz+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679005275; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ySNFLglM8SJEJlQ1FHHmy3ZddsPF9igYAJti/ai7CSk=; b=QQTaflMAdZHAv2WrzOb/r1JyjyLgDvfBQc0ZCd2ws6FcXDmSQC4RqyzYECzHyo6DrV sfXrGPG8O+fd2SJSBJx12VO36PIOpoHWnnlEl8NNDR2jnYr8Jeqe1Ob9QZhVxxGh7c6e kB1pghnPu1qSaIPYK6FbBdBoqOCrbYhNrG+0zrfBSIbrQvi9eHjXXppFiJgo6qP/NPBL nnmIEo2DRRYHW/67Xni5Na1CWTxZPN+x58X+4kNpN63tZIJSQp7KvY6j3K+uPocsgY4Y yMRwitSyaA3prWxiYUXt12v2S95DkIYks/IvL0KH1zuHizYF3Uljv4GiPpRuXri7hsRk 1R+w== X-Gm-Message-State: AO0yUKXpciIslNM0BBmPrIJ9hbF5N8xk7j9hqFrvSgKfsWaPpm3jPouf Lp2nUpsEmBA5CpRzXwyZvDu3eQ== X-Received: by 2002:adf:dcc2:0:b0:2ce:aa2d:c625 with SMTP id x2-20020adfdcc2000000b002ceaa2dc625mr5869244wrm.22.1679005274770; Thu, 16 Mar 2023 15:21:14 -0700 (PDT) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:4b87:78c3:3abe:7b0d]) by smtp.gmail.com with ESMTPSA id f9-20020adff989000000b002cea392f000sm439256wrr.69.2023.03.16.15.21.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 16 Mar 2023 15:21:14 -0700 (PDT) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, gpiccoli@igalia.com, David Woodhouse , Usama Arif Subject: [PATCH v15 03/12] cpu/hotplug: Add dynamic parallel bringup states before CPUHP_BRINGUP_CPU Date: Thu, 16 Mar 2023 22:21:00 +0000 Message-Id: <20230316222109.1940300-4-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230316222109.1940300-1-usama.arif@bytedance.com> References: <20230316222109.1940300-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760565212606506315?= X-GMAIL-MSGID: =?utf-8?q?1760565212606506315?= From: David Woodhouse There is often significant latency in the early stages of CPU bringup, and time is wasted by waking each CPU (e.g. with SIPI/INIT/INIT on x86) and then waiting for it to respond before moving on to the next. Allow a platform to register a set of pre-bringup CPUHP states to which each CPU can be stepped in parallel, thus absorbing some of that latency. There is a subtlety here: even with an empty CPUHP_BP_PARALLEL_DYN step, this means that *all* CPUs are brought through the prepare states and to CPUHP_BP_PREPARE_DYN before any of them are taken to CPUHP_BRINGUP_CPU and then are allowed to run for themselves to CPUHP_ONLINE. So any combination of prepare/start calls which depend on A-B ordering for each CPU in turn, such as the X2APIC code which used to allocate a cluster mask 'just in case' and store it in a global variable in the prep stage, then potentially consume that preallocated structure from the AP and set the global pointer to NULL to be reallocated in CPUHP_X2APIC_PREPARE for the next CPU... would explode horribly. Any platform enabling the CPUHP_BP_PARALLEL_DYN steps must be reviewed and tested to ensure that such issues do not exist, and the existing behaviour of bringing CPUs to CPUHP_BP_PREPARE_DYN and then immediately to CPUHP_BRINGUP_CPU and CPUHP_ONLINE only one at a time does not change unless such a state is registered. Note that the new parallel stages do *not* yet bring each AP to the CPUHP_BRINGUP_CPU state at the same time, only to the new states which exist before it. The final loop in bringup_nonboot_cpus() is untouched, bringing each AP in turn from the final PARALLEL_DYN state (or all the way from CPUHP_OFFLINE) to CPUHP_BRINGUP_CPU and then waiting for that AP to do its own processing and reach CPUHP_ONLINE before releasing the next. Parallelising that part by bringing them all to CPUHP_BRINGUP_CPU and then waiting for them all is an exercise for the future. Signed-off-by: David Woodhouse Signed-off-by: Usama Arif Tested-by: Paul E. McKenney Tested-by: Kim Phillips Tested-by: Oleksandr Natalenko Tested-by: Guilherme G. Piccoli --- include/linux/cpuhotplug.h | 2 ++ kernel/cpu.c | 31 +++++++++++++++++++++++++++++-- 2 files changed, 31 insertions(+), 2 deletions(-) diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h index 6c6859bfc454..e5a73ae6ccc0 100644 --- a/include/linux/cpuhotplug.h +++ b/include/linux/cpuhotplug.h @@ -133,6 +133,8 @@ enum cpuhp_state { CPUHP_MIPS_SOC_PREPARE, CPUHP_BP_PREPARE_DYN, CPUHP_BP_PREPARE_DYN_END = CPUHP_BP_PREPARE_DYN + 20, + CPUHP_BP_PARALLEL_DYN, + CPUHP_BP_PARALLEL_DYN_END = CPUHP_BP_PARALLEL_DYN + 4, CPUHP_BRINGUP_CPU, /* diff --git a/kernel/cpu.c b/kernel/cpu.c index 6b3dccb4a888..6ccc64defd47 100644 --- a/kernel/cpu.c +++ b/kernel/cpu.c @@ -1497,8 +1497,30 @@ int bringup_hibernate_cpu(unsigned int sleep_cpu) void bringup_nonboot_cpus(unsigned int setup_max_cpus) { + unsigned int n = setup_max_cpus - num_online_cpus(); unsigned int cpu; + /* + * An architecture may have registered parallel pre-bringup states to + * which each CPU may be brought in parallel. For each such state, + * bring N CPUs to it in turn before the final round of bringing them + * online. + */ + if (n > 0) { + enum cpuhp_state st = CPUHP_BP_PARALLEL_DYN; + + while (st <= CPUHP_BP_PARALLEL_DYN_END && cpuhp_hp_states[st].name) { + int i = n; + + for_each_present_cpu(cpu) { + cpu_up(cpu, st); + if (!--i) + break; + } + st++; + } + } + for_each_present_cpu(cpu) { if (num_online_cpus() >= setup_max_cpus) break; @@ -1875,6 +1897,10 @@ static int cpuhp_reserve_state(enum cpuhp_state state) step = cpuhp_hp_states + CPUHP_BP_PREPARE_DYN; end = CPUHP_BP_PREPARE_DYN_END; break; + case CPUHP_BP_PARALLEL_DYN: + step = cpuhp_hp_states + CPUHP_BP_PARALLEL_DYN; + end = CPUHP_BP_PARALLEL_DYN_END; + break; default: return -EINVAL; } @@ -1899,14 +1925,15 @@ static int cpuhp_store_callbacks(enum cpuhp_state state, const char *name, /* * If name is NULL, then the state gets removed. * - * CPUHP_AP_ONLINE_DYN and CPUHP_BP_PREPARE_DYN are handed out on + * CPUHP_AP_ONLINE_DYN and CPUHP_BP_P*_DYN are handed out on * the first allocation from these dynamic ranges, so the removal * would trigger a new allocation and clear the wrong (already * empty) state, leaving the callbacks of the to be cleared state * dangling, which causes wreckage on the next hotplug operation. */ if (name && (state == CPUHP_AP_ONLINE_DYN || - state == CPUHP_BP_PREPARE_DYN)) { + state == CPUHP_BP_PREPARE_DYN || + state == CPUHP_BP_PARALLEL_DYN)) { ret = cpuhp_reserve_state(state); if (ret < 0) return ret; From patchwork Thu Mar 16 22:21:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 70989 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp25644wrt; Thu, 16 Mar 2023 15:23:18 -0700 (PDT) X-Google-Smtp-Source: AK7set8MkfQ4brP+SSSsuP4hr5xEoXHwmtckXjU+k2BwEygCfPxwzgdto1EgvXpZp6hKFk6klnta X-Received: by 2002:a17:903:2111:b0:1a0:7584:f46f with SMTP id o17-20020a170903211100b001a07584f46fmr667971ple.9.1679005398504; Thu, 16 Mar 2023 15:23:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679005398; cv=none; d=google.com; s=arc-20160816; b=BL/G4RM7RN0sISWIfAhUYgrYqBnwuCn8KoQNw3dqUFPxoPiATgETuU11PMC8TNqMKa oCX0whFQDOfFuQH68rQ486Jd3DHoYKj7VWS15CvmgdAnf+GtHDB5OXNpx/TwTsfWJe85 3vCMSvA7HNBzY9TKwFFRVuuO5BHJTXd/ngfwQpbMH7Ci/SqqoPhlAtWSx/TOLkLaDM0F SOGv1e01b/Fab7moqq1rT0Z4oXaZ0kBIF4Ne3woPqeUcuY8gy3n9wpIr+8Ld1cplToJN a6MorbO7actrwt+5HSvJiR8BrFk3wrOg5FKnf4tiM++cUMK70GrbNKU7x+/PfSt24xDe xpIQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=QbZumu4dMY0Za38bxLjojwYOtrQ4rJ/sJrcRrt0N9M0=; b=RGo6en0QO7GnIskVKwIMAVtyez4Z98HJjNLyygoAc+r6vv+UCtRk3BmTZaIBPEzjzj zZskcK0p+EUSrgRr4qIglIf/IUGQYickS23V0vmWGdXHcAb4Vfz/x8P0vcl5tfCKjiGX IB8np09qP3TvStVDroca2IqhmMyMZ8Ba1zMxBk/5uL1XslqaNE20wiSRHJJJVbqPW/0A l9uvmEbncFa7cRvSdGO5fZ/swOstFlYRLTGuNaUQwSJY1EDCvEQLBeysYYlEj2uVBzCB nwNMbTtQaZ6lnU0IJhZU+1aQOp2uI3+38+Dly4fNRV7fagJGHm4y1cFJtwMEYu8f8Of/ S5EQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=akDU+NON; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l20-20020a17090ac59400b0022c1c376f57si649423pjt.33.2023.03.16.15.23.03; Thu, 16 Mar 2023 15:23:18 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=akDU+NON; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229820AbjCPWV6 (ORCPT + 99 others); Thu, 16 Mar 2023 18:21:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60684 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230107AbjCPWVo (ORCPT ); Thu, 16 Mar 2023 18:21:44 -0400 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2F3DBB1A43 for ; Thu, 16 Mar 2023 15:21:17 -0700 (PDT) Received: by mail-wm1-x32a.google.com with SMTP id m18-20020a05600c3b1200b003ed2a3d635eso2172490wms.4 for ; Thu, 16 Mar 2023 15:21:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1679005275; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QbZumu4dMY0Za38bxLjojwYOtrQ4rJ/sJrcRrt0N9M0=; b=akDU+NONCJCdGBGVARuhAyYEyH1e56zvKd1pHTUrwEbXIROtqT3NRhj/nP1vMj3Ji4 whfGxuIXnEs9OPIXC7ByNlQRB01JoZidvSqL+LZfNQbYOVKfMa9wQr6or2mQUc4oOSFO JFzz/rb/5RproOJxOSDYRXu/q96ur4PJhWRJM8SOgQgp2pK/fbIuafKBV1QiZz/56R2V bOBSZmjMZoOVMiwuL8oEE6CEVYm+C7Ob3sqfSe1nlrxHZzfHhPSQGtSqtyXrjG2GThID ibe8eJWI7Du11Jjes+OCpBrQ67FEzvvIbS4ZdvE+tMLEBhUGcYeBi7giasJxLASf9I+d bNtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679005275; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QbZumu4dMY0Za38bxLjojwYOtrQ4rJ/sJrcRrt0N9M0=; b=rhyqjQk/iJLvFv0l/GeTWZwTy5GLXOqHiohWQbZnasObGxhwB0QsuierE7EXFiUnE3 y3LHRvqeuVKa5zNF9nAlZco7E5r6vTIXNZ6uWYEHFF/UaIiteaQ00PVx4oyV177FlbfH MocZy9Pb0/E1vXp0/X3WVITI1XPMyRullgdUynmQ3Hg36pAnzvQo3EtKQDr+8Ew04a8H TLrdfE3WwzIotMmLG7UGhAce68Eerdf9pKtrN+3JKXsCg57XYaiIjKHwh8F+Zu3WAAn9 2OwNXMgPhU8S8OJv2Tk7ICSDvpvLmzAueGumwaf8sknyjpjR+fmlVnK/fYCJs8kQUYbR edpQ== X-Gm-Message-State: AO0yUKX8FBYSgjvZNGDbHIZPALnObOwjVHZl/bSvBDNAHbPQxb0MP/Ba xu7HwTgmh2HuOCNA3ExgzK6jrw== X-Received: by 2002:a05:600c:1f0f:b0:3df:ef18:b0a1 with SMTP id bd15-20020a05600c1f0f00b003dfef18b0a1mr23732968wmb.12.1679005275673; Thu, 16 Mar 2023 15:21:15 -0700 (PDT) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:4b87:78c3:3abe:7b0d]) by smtp.gmail.com with ESMTPSA id f9-20020adff989000000b002cea392f000sm439256wrr.69.2023.03.16.15.21.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 16 Mar 2023 15:21:15 -0700 (PDT) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, gpiccoli@igalia.com, David Woodhouse , Usama Arif Subject: [PATCH v15 04/12] x86/smpboot: Reference count on smpboot_setup_warm_reset_vector() Date: Thu, 16 Mar 2023 22:21:01 +0000 Message-Id: <20230316222109.1940300-5-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230316222109.1940300-1-usama.arif@bytedance.com> References: <20230316222109.1940300-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760564764685302012?= X-GMAIL-MSGID: =?utf-8?q?1760564764685302012?= From: David Woodhouse When bringing up a secondary CPU from do_boot_cpu(), the warm reset flag is set in CMOS and the starting IP for the trampoline written inside the BDA at 0x467. Once the CPU is running, the CMOS flag is unset and the value in the BDA cleared. To allow for parallel bringup of CPUs, add a reference count to track the number of CPUs currently bring brought up, and clear the state only when the count reaches zero. Since the RTC spinlock is required to write to the CMOS, it can be used for mutual exclusion on the refcount too. Signed-off-by: David Woodhouse Signed-off-by: Usama Arif Tested-by: Paul E. McKenney Tested-by: Kim Phillips Tested-by: Oleksandr Natalenko Tested-by: Guilherme G. Piccoli --- arch/x86/kernel/smpboot.c | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 55cad72715d9..3a793772a2aa 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -121,17 +121,20 @@ int arch_update_cpu_topology(void) return retval; } + +static unsigned int smpboot_warm_reset_vector_count; + static inline void smpboot_setup_warm_reset_vector(unsigned long start_eip) { unsigned long flags; spin_lock_irqsave(&rtc_lock, flags); - CMOS_WRITE(0xa, 0xf); + if (!smpboot_warm_reset_vector_count++) { + CMOS_WRITE(0xa, 0xf); + *((volatile unsigned short *)phys_to_virt(TRAMPOLINE_PHYS_HIGH)) = start_eip >> 4; + *((volatile unsigned short *)phys_to_virt(TRAMPOLINE_PHYS_LOW)) = start_eip & 0xf; + } spin_unlock_irqrestore(&rtc_lock, flags); - *((volatile unsigned short *)phys_to_virt(TRAMPOLINE_PHYS_HIGH)) = - start_eip >> 4; - *((volatile unsigned short *)phys_to_virt(TRAMPOLINE_PHYS_LOW)) = - start_eip & 0xf; } static inline void smpboot_restore_warm_reset_vector(void) @@ -143,10 +146,12 @@ static inline void smpboot_restore_warm_reset_vector(void) * to default values. */ spin_lock_irqsave(&rtc_lock, flags); - CMOS_WRITE(0, 0xf); + if (!--smpboot_warm_reset_vector_count) { + CMOS_WRITE(0, 0xf); + *((volatile u32 *)phys_to_virt(TRAMPOLINE_PHYS_LOW)) = 0; + } spin_unlock_irqrestore(&rtc_lock, flags); - *((volatile u32 *)phys_to_virt(TRAMPOLINE_PHYS_LOW)) = 0; } /* From patchwork Thu Mar 16 22:21:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 70992 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp25736wrt; Thu, 16 Mar 2023 15:23:35 -0700 (PDT) X-Google-Smtp-Source: AK7set8kw1ya+r5JP+xYT3R0tA4vZ/YYrdrGDLMz/t0nVzUUtH6qV7F9eTk5RixfFbOZnjR0CRHb X-Received: by 2002:a62:7bce:0:b0:625:1487:f06c with SMTP id w197-20020a627bce000000b006251487f06cmr4483631pfc.29.1679005415548; Thu, 16 Mar 2023 15:23:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679005415; cv=none; d=google.com; s=arc-20160816; b=MgLxKnWwf8cLgiISOWP5ROLnzszsucd/DXHX6zvQBLH5ZxWKkaFDAAAU9SrFXIy6HI r+nes8Jhd/g70etn5RXjv2ifES/en/gg5FTR2/5y0/JJv+daP7KWRTNlqTUc7hms+X5t bFY84025Z2SHPPNa2vuUiNAbK/0Itk2ER59GkbzLcjdi5/+O34zAin9j4PWgY+7YJk8n IAtUvgCwOGkpOwPKtjREWgyut+tM1qY5ligkfJHOA5Ej1AWCgIO0lnfRPj7NoQBTKsxm B2qgw0x+rwpTOMnNSU1FRUI2EumSrX21bMpngap7WtMfk799wfEaZRDcBZRFeY0bs3rL dgRA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=UpfktZXX532ImgFvpuvdQggoHIm2ccR81vMIQOZNM0M=; b=wNhlAaEUysjJYqkNLlOOrVLv66yeeTwysWn+Y8sev4vQ/+zHKd1c9oOdWlRmwYiEan WiRifvudCfaVrJFN38rEMmZ9Dn6uxtEMB+rhm4iGmomMte8zGVJQX3gupg5R9bXR06kX Cs3tmV+WcDNDrq4gZRQAZuNDNoJWZgIjceOuQwo0z9Tm4N4tp/e2wFvglDI6yJeJHGut wyzXrvzgFjxdly4D/tj5L4oWAfX55txVSimbY+kH41gai+3xLE668n1M/XFMREUDNL13 kBxVvpZVO0HOk66p8hD5CWEa3IXPexa2Ibl5fia9mrqNScAwcBDwSZ8nfRGF2sGUYD/Q q6JQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=U6HnDfkT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u22-20020a056a00125600b0062508c81c6csi605489pfi.222.2023.03.16.15.23.21; Thu, 16 Mar 2023 15:23:35 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=U6HnDfkT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230378AbjCPWWX (ORCPT + 99 others); Thu, 16 Mar 2023 18:22:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33166 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230150AbjCPWWI (ORCPT ); Thu, 16 Mar 2023 18:22:08 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0D705B1A47 for ; Thu, 16 Mar 2023 15:21:18 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id y14so2872981wrq.4 for ; Thu, 16 Mar 2023 15:21:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1679005276; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UpfktZXX532ImgFvpuvdQggoHIm2ccR81vMIQOZNM0M=; b=U6HnDfkTM/OM/2zyWzzWnY++N1S4crZglaLNA8MqHnqyE6bMBvnmDPOkBVu6pL0DcL ESax/wo5d9HyQUiNjMsI+7Nr5Dd9LUuTFjHaZeR398+cmWdYAop8Z2M+gXKUXJRy9LfP r6h8phgBRk2hdD7UfMxJmiI4KSkDrFYUl8MuUYX1ILX5dJeR5WLwwe11yG0mn9jIamSx 84BccIJi0WwfRxKmCM1u6ZUYjjMq2QVeCrKKzOr+If9wgpciAKrV+CAaWPf2xyV9Lfy8 1NJnpfWRGJ9YkOmpH1Y3YDXLVBhcjFiUHcjhgpRYc9lbuRV8LwdLZinn7RIj3IGDqryH qqVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679005276; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UpfktZXX532ImgFvpuvdQggoHIm2ccR81vMIQOZNM0M=; b=2gqt5B0yOVq285XzQDB7kqFcJL8eXAMo0NvErgRLmUptrZkvJsa/NtUmipdHXpn/s4 B3aSMs4ZaK8+j171hfX27zSJvhMNY2vDWtYl8Z6e3FuMMiR5HY9Mo+1A8G+n0F4v0StL i5gCoUqmpE9RHsxScZxYysBhyzGhJXo5U2STT55nQBVBGnJW+89TyLiIFI9HfGZ41fg0 tMIMRV+trOYI/ZdSklsxYUgCPXfZmVdU2MtVj1Zopp8euqDmEH+7QylTZXaLKe4FnhHl cuD37wFivx7Bl1TF9hOfCeZud2uXB49+HUJwGP2EKP+mnsVYnuinaHgfbT7835RRii9S xBSw== X-Gm-Message-State: AO0yUKWnXgFbTNSkXnBPmOboRU/Nt9CQ3C2W3MgjU/JpKM2L41tAaVyL TrXTc2VtujtjVlzGjuBaAOmulw== X-Received: by 2002:adf:f390:0:b0:2cf:e18f:7e7f with SMTP id m16-20020adff390000000b002cfe18f7e7fmr5741356wro.25.1679005276433; Thu, 16 Mar 2023 15:21:16 -0700 (PDT) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:4b87:78c3:3abe:7b0d]) by smtp.gmail.com with ESMTPSA id f9-20020adff989000000b002cea392f000sm439256wrr.69.2023.03.16.15.21.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 16 Mar 2023 15:21:16 -0700 (PDT) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, gpiccoli@igalia.com, David Woodhouse , Usama Arif Subject: [PATCH v15 05/12] x86/smpboot: Split up native_cpu_up into separate phases and document them Date: Thu, 16 Mar 2023 22:21:02 +0000 Message-Id: <20230316222109.1940300-6-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230316222109.1940300-1-usama.arif@bytedance.com> References: <20230316222109.1940300-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760564782491367178?= X-GMAIL-MSGID: =?utf-8?q?1760564782491367178?= From: David Woodhouse There are four logical parts to what native_cpu_up() does on the BSP (or on the controlling CPU for a later hotplug): 1) Wake the AP by sending the INIT/SIPI/SIPI sequence. 2) Wait for the AP to make it as far as wait_for_master_cpu() which sets that CPU's bit in cpu_initialized_mask, then sets the bit in cpu_callout_mask to let the AP proceed through cpu_init(). 3) Wait for the AP to finish cpu_init() and get as far as the smp_callin() call, which sets that CPU's bit in cpu_callin_mask. 4) Perform the TSC synchronization and wait for the AP to actually mark itself online in cpu_online_mask. In preparation to allow these phases to operate in parallel on multiple APs, split them out into separate functions and document the interactions a little more clearly in both the BSP and AP code paths. No functional change intended. Signed-off-by: David Woodhouse Signed-off-by: Usama Arif Tested-by: Paul E. McKenney Tested-by: Kim Phillips Tested-by: Oleksandr Natalenko Tested-by: Guilherme G. Piccoli --- arch/x86/kernel/smpboot.c | 179 ++++++++++++++++++++++++++------------ 1 file changed, 125 insertions(+), 54 deletions(-) diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 3a793772a2aa..fa1e35b0747c 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -204,6 +204,10 @@ static void smp_callin(void) wmb(); + /* + * This runs the AP through all the cpuhp states to its target + * state (CPUHP_ONLINE in the case of serial bringup). + */ notify_cpu_starting(cpuid); /* @@ -231,17 +235,32 @@ static void notrace start_secondary(void *unused) load_cr3(swapper_pg_dir); __flush_tlb_all(); #endif + /* + * Sync point with do_wait_cpu_initialized(). Before proceeding through + * cpu_init(), the AP will call wait_for_master_cpu() which sets its + * own bit in cpu_initialized_mask and then waits for the BSP to set + * its bit in cpu_callout_mask to release it. + */ cpu_init_secondary(); rcu_cpu_starting(raw_smp_processor_id()); x86_cpuinit.early_percpu_clock_init(); + + /* + * Sync point with do_wait_cpu_callin(). The AP doesn't wait here + * but just sets the bit to let the controlling CPU (BSP) know that + * it's got this far. + */ smp_callin(); enable_start_cpu0 = 0; /* otherwise gcc will move up smp_processor_id before the cpu_init */ barrier(); + /* - * Check TSC synchronization with the boot CPU: + * Check TSC synchronization with the boot CPU (or whichever CPU + * is controlling the bringup). It will do its part of this from + * do_wait_cpu_online(), making it an implicit sync point. */ check_tsc_sync_target(); @@ -254,6 +273,7 @@ static void notrace start_secondary(void *unused) * half valid vector space. */ lock_vector_lock(); + /* Sync point with do_wait_cpu_online() */ set_cpu_online(smp_processor_id(), true); lapic_online(); unlock_vector_lock(); @@ -1083,7 +1103,6 @@ static int do_boot_cpu(int apicid, int cpu, struct task_struct *idle, unsigned long start_ip = real_mode_header->trampoline_start; unsigned long boot_error = 0; - unsigned long timeout; #ifdef CONFIG_X86_64 /* If 64-bit wakeup method exists, use the 64-bit mode trampoline IP */ @@ -1144,55 +1163,94 @@ static int do_boot_cpu(int apicid, int cpu, struct task_struct *idle, boot_error = wakeup_cpu_via_init_nmi(cpu, start_ip, apicid, cpu0_nmi_registered); - if (!boot_error) { - /* - * Wait 10s total for first sign of life from AP - */ - boot_error = -1; - timeout = jiffies + 10*HZ; - while (time_before(jiffies, timeout)) { - if (cpumask_test_cpu(cpu, cpu_initialized_mask)) { - /* - * Tell AP to proceed with initialization - */ - cpumask_set_cpu(cpu, cpu_callout_mask); - boot_error = 0; - break; - } - schedule(); - } - } + return boot_error; +} - if (!boot_error) { - /* - * Wait till AP completes initial initialization - */ - while (!cpumask_test_cpu(cpu, cpu_callin_mask)) { - /* - * Allow other tasks to run while we wait for the - * AP to come online. This also gives a chance - * for the MTRR work(triggered by the AP coming online) - * to be completed in the stop machine context. - */ - schedule(); - } +static int do_wait_cpu_cpumask(unsigned int cpu, const struct cpumask *mask) +{ + unsigned long timeout; + + /* + * Wait up to 10s for the CPU to report in. + */ + timeout = jiffies + 10*HZ; + while (time_before(jiffies, timeout)) { + if (cpumask_test_cpu(cpu, mask)) + return 0; + + schedule(); } + return -1; +} - if (x86_platform.legacy.warm_reset) { - /* - * Cleanup possible dangling ends... - */ - smpboot_restore_warm_reset_vector(); +/* + * Bringup step two: Wait for the target AP to reach cpu_init_secondary() + * and thus wait_for_master_cpu(), then set cpu_callout_mask to allow it + * to proceed. The AP will then proceed past setting its 'callin' bit + * and end up waiting in check_tsc_sync_target() until we reach + * do_wait_cpu_online() to tend to it. + */ +static int do_wait_cpu_initialized(unsigned int cpu) +{ + /* + * Wait for first sign of life from AP. + */ + if (do_wait_cpu_cpumask(cpu, cpu_initialized_mask)) + return -1; + + cpumask_set_cpu(cpu, cpu_callout_mask); + return 0; +} + +/* + * Bringup step three: Wait for the target AP to reach smp_callin(). + * The AP is not waiting for us here so we don't need to parallelise + * this step. Not entirely clear why we care about this, since we just + * proceed directly to TSC synchronization which is the next sync + * point with the AP anyway. + */ +static int do_wait_cpu_callin(unsigned int cpu) +{ + /* + * Wait till AP completes initial initialization. + */ + return do_wait_cpu_cpumask(cpu, cpu_callin_mask); +} + +/* + * Bringup step four: Synchronize the TSC and wait for the target AP + * to reach set_cpu_online() in start_secondary(). + */ +static int do_wait_cpu_online(unsigned int cpu) +{ + unsigned long flags; + + /* + * Check TSC synchronization with the AP (keep irqs disabled + * while doing so): + */ + local_irq_save(flags); + check_tsc_sync_source(cpu); + local_irq_restore(flags); + + /* + * Wait for the AP to mark itself online. Not entirely + * clear why we care, since the generic cpuhp code will + * wait for it to each CPUHP_AP_ONLINE_IDLE before going + * ahead with the rest of the bringup anyway. + */ + while (!cpu_online(cpu)) { + cpu_relax(); + touch_nmi_watchdog(); } - return boot_error; + return 0; } -int native_cpu_up(unsigned int cpu, struct task_struct *tidle) +static int do_cpu_up(unsigned int cpu, struct task_struct *tidle) { int apicid = apic->cpu_present_to_apicid(cpu); int cpu0_nmi_registered = 0; - unsigned long flags; int err, ret = 0; lockdep_assert_irqs_enabled(); @@ -1239,19 +1297,6 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) goto unreg_nmi; } - /* - * Check TSC synchronization with the AP (keep irqs disabled - * while doing so): - */ - local_irq_save(flags); - check_tsc_sync_source(cpu); - local_irq_restore(flags); - - while (!cpu_online(cpu)) { - cpu_relax(); - touch_nmi_watchdog(); - } - unreg_nmi: /* * Clean up the nmi handler. Do this after the callin and callout sync @@ -1263,6 +1308,32 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) return ret; } +int native_cpu_up(unsigned int cpu, struct task_struct *tidle) +{ + int ret; + + ret = do_cpu_up(cpu, tidle); + if (ret) + goto out; + + ret = do_wait_cpu_initialized(cpu); + if (ret) + goto out; + + ret = do_wait_cpu_callin(cpu); + if (ret) + goto out; + + ret = do_wait_cpu_online(cpu); + + out: + /* Cleanup possible dangling ends... */ + if (x86_platform.legacy.warm_reset) + smpboot_restore_warm_reset_vector(); + + return ret; +} + /** * arch_disable_smp_support() - disables SMP support for x86 at runtime */ From patchwork Thu Mar 16 22:21:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 70991 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp25700wrt; Thu, 16 Mar 2023 15:23:27 -0700 (PDT) X-Google-Smtp-Source: AK7set/9uC1HJU3YXJuK2T9zcZvMLBCl6pidDRd4Jq49AgprbYeb9KYwEb96910a+t+AycA48ypn X-Received: by 2002:a05:6a20:6595:b0:d7:2dbd:b65f with SMTP id p21-20020a056a20659500b000d72dbdb65fmr3091272pzh.57.1679005407629; Thu, 16 Mar 2023 15:23:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679005407; cv=none; d=google.com; s=arc-20160816; b=APZ18CL6zwbwMinNLz9lhNMgJtVKMJhOQbzgJYqHrxi26r/3ASvxn0XQyxLFKCucZU RcxQVR5DoMsuYaXVixJyzHEnwpDI5Tua6lkxlzccFx7wUPcz2yMcpP4Ve/W8HVhaVHYW Vn4MIMX89NLuanvC4B4wLGlJA7aDKjo7iUbOE5tnIgTUlDoTVO/gvlNanP1yguQaZSQp 7qEAhPdaIHPXUwzUswlfr2BPy4VCMHiU0BdBWDgBo2U1HBck2zhlpevE751sUO+tHcV3 aSl09ckQUsdgEsfIX1Scbx4ry1as5S62qr2h9UEcf30j5KxqhrFxNWX0FjUkCCa42MlU WRIQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=WCd04cCjdDd9yY9uOauqDe6tLaGHwa1xS5Ta04nWC9w=; b=T2QJaP3tXBlhOQruPirbF0sT4q79ef1hgPhIzE1MWRZzUtJghdm2huzPEmjuMgy9Ks XgFtt1BioV+O5jZQCizJiuRIeD/0QZRTR0BMcMjM9gefPEYy0TvFT0s2UvIHw1cRu6oU H2xtKBA1J4hK9lKFbsUJGloxBCsBEjTAhp610WC6bbbFu+ZKzErx0urAfZFwrgRBH7Ji RVMLQmw9Qd7oEdDOPe1rrhYioiM57AZSjrguk/Z12UgSNXaxxSX3lnucPfkJS4+RJhWZ u8oGdstX45BnbFsE9r1L4RVT9QUVzNAIocl6DQATwpJ+zH5IGUYEZwO8DULxG5LOd55D M4/w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=YDJJw+pS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q13-20020a632a0d000000b00502d7fad4d2si358269pgq.595.2023.03.16.15.23.13; Thu, 16 Mar 2023 15:23:27 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=YDJJw+pS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230357AbjCPWWU (ORCPT + 99 others); Thu, 16 Mar 2023 18:22:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33172 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229813AbjCPWWH (ORCPT ); Thu, 16 Mar 2023 18:22:07 -0400 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 02355B1B12 for ; Thu, 16 Mar 2023 15:21:18 -0700 (PDT) Received: by mail-wm1-x32a.google.com with SMTP id fm20-20020a05600c0c1400b003ead37e6588so4062784wmb.5 for ; Thu, 16 Mar 2023 15:21:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1679005277; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WCd04cCjdDd9yY9uOauqDe6tLaGHwa1xS5Ta04nWC9w=; b=YDJJw+pSikaVs3fCOY91ao1Vu7r06p5XLVjpiPhJLZbcY3qDH/HQVuOf3vyW1Lj/R/ Oxci52Cw+uWEx+VOeZ0Q9UZqvaJr8nyxF5AAlpLzyt3JNB5p9AfDs4Gsz4kXa7bBdl4K mGzeSojARtKfwMnehoj/ZNB9EBhGsqavxNOiFSO34L3bmzjR1BYye4+IcahI42zPONo0 HTJA1YZiqDJYcJgR2TLQ285JR+FTqbZgXXnpyqrm8c7IfDrXFt6XzQJzbsXcAj+k6WGU fb8BhOprzTate3FGN8awPwD3EWsAyb9Lf3YY10fuMwrAmvfq0gEDV2G0C6LFDG6Cc5ku ECNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679005277; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WCd04cCjdDd9yY9uOauqDe6tLaGHwa1xS5Ta04nWC9w=; b=M8NA2R+efGbCKtaQYD6Fvk0vtnMCvGI0nvWFno4i4245j4bV+CrtrojV2/kIamWwfE txT0wn0FR6M/amdZRjLAOx15P7Sez3/yfSp+sJaY7nSwVTkTEoAXHaTcxjUb5KHpZ8Yq x9Rv6l+gVYEfbSD24n4TOQTCKXkknL6C7YDXVTYK6e9ffZqvtWQiEFRzl51fL5eZ/bVC /PEG0Hn1cp7uUGXmwJDDE4FjUA6Qu4FeMS/zgmhTa98EjfsV7Iv5KjXMR4Hq5ZJaaD+W PVmewYxGdZ5J54LyMltP0ULp44SP5/Kw06Pf6Kv4xIX8xpxsrS5TDvR4cgXXNxPFLxIj xCmg== X-Gm-Message-State: AO0yUKX8nk1LJNT3rE4o2mK88FAGB4SK6C3bjP5MPYZqcXxt4GR/bCq9 Z5oPzODH+gN728ynXcwAkcmXgA== X-Received: by 2002:a7b:c4da:0:b0:3ed:70df:37 with SMTP id g26-20020a7bc4da000000b003ed70df0037mr1475010wmk.21.1679005277390; Thu, 16 Mar 2023 15:21:17 -0700 (PDT) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:4b87:78c3:3abe:7b0d]) by smtp.gmail.com with ESMTPSA id f9-20020adff989000000b002cea392f000sm439256wrr.69.2023.03.16.15.21.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 16 Mar 2023 15:21:17 -0700 (PDT) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, gpiccoli@igalia.com, David Woodhouse , Usama Arif Subject: [PATCH v15 06/12] x86/smpboot: Remove initial_stack on 64-bit Date: Thu, 16 Mar 2023 22:21:03 +0000 Message-Id: <20230316222109.1940300-7-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230316222109.1940300-1-usama.arif@bytedance.com> References: <20230316222109.1940300-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760564774133220932?= X-GMAIL-MSGID: =?utf-8?q?1760564774133220932?= From: Brian Gerst In order to facilitate parallel startup, start to eliminate some of the global variables passing information to CPUs in the startup path. However, start by introducing one more: smpboot_control. For now this merely holds the CPU# of the CPU which is coming up. Each CPU can then find its own per-cpu data, and everything else it needs can be found from there, allowing the other global variables to be removed. First to be removed is initial_stack. Each CPU can load %rsp from its current_task->thread.sp instead. That is already set up with the correct idle thread for APs. Set up the .sp field in INIT_THREAD on x86 so that the BSP also finds a suitable stack pointer in the static per-cpu data when coming up on first boot. On resume from S3, the CPU needs a temporary stack because its idle task is already active. Instead of setting initial_stack, the sleep code can simply set its own current->thread.sp to point to the temporary stack. Nobody else cares about ->thread.sp for a thread which is currently on a CPU, because the true value is actually in the %rsp register. Which is restored with the rest of the CPU context in do_suspend_lowlevel(). Signed-off-by: Brian Gerst Reviewed-by: David Woodhouse Signed-off-by: David Woodhouse Tested-by: Usama Arif Signed-off-by: Usama Arif Tested-by: Guilherme G. Piccoli --- arch/x86/include/asm/processor.h | 6 ++++- arch/x86/include/asm/smp.h | 5 +++- arch/x86/kernel/acpi/sleep.c | 20 +++++++++++++-- arch/x86/kernel/asm-offsets.c | 1 + arch/x86/kernel/head_64.S | 43 +++++++++++++++++++++----------- arch/x86/kernel/smpboot.c | 7 +++++- arch/x86/xen/xen-head.S | 2 +- 7 files changed, 63 insertions(+), 21 deletions(-) diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h index 4e35c66edeb7..bdde7316e75b 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -648,7 +648,11 @@ static inline void spin_lock_prefetch(const void *x) #define KSTK_ESP(task) (task_pt_regs(task)->sp) #else -#define INIT_THREAD { } +extern unsigned long __end_init_task[]; + +#define INIT_THREAD { \ + .sp = (unsigned long)&__end_init_task - sizeof(struct pt_regs), \ +} extern unsigned long KSTK_ESP(struct task_struct *task); diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h index b4dbb20dab1a..bf2c51df9e0b 100644 --- a/arch/x86/include/asm/smp.h +++ b/arch/x86/include/asm/smp.h @@ -199,5 +199,8 @@ extern void nmi_selftest(void); #define nmi_selftest() do { } while (0) #endif -#endif /* __ASSEMBLY__ */ +extern unsigned int smpboot_control; + +#endif /* !__ASSEMBLY__ */ + #endif /* _ASM_X86_SMP_H */ diff --git a/arch/x86/kernel/acpi/sleep.c b/arch/x86/kernel/acpi/sleep.c index 3b7f4cdbf2e0..1b4c43d0819a 100644 --- a/arch/x86/kernel/acpi/sleep.c +++ b/arch/x86/kernel/acpi/sleep.c @@ -111,13 +111,29 @@ int x86_acpi_suspend_lowlevel(void) saved_magic = 0x12345678; #else /* CONFIG_64BIT */ #ifdef CONFIG_SMP - initial_stack = (unsigned long)temp_stack + sizeof(temp_stack); + /* + * As each CPU starts up, it will find its own stack pointer + * from its current_task->thread.sp. Typically that will be + * the idle thread for a newly-started AP, or even the boot + * CPU which will find it set to &init_task in the static + * per-cpu data. + * + * Make the resuming CPU use the temporary stack at startup + * by setting current->thread.sp to point to that. The true + * %rsp will be restored with the rest of the CPU context, + * by do_suspend_lowlevel(). And unwinders don't care about + * the abuse of ->thread.sp because it's a dead variable + * while the thread is running on the CPU anyway; the true + * value is in the actual %rsp register. + */ + current->thread.sp = (unsigned long)temp_stack + sizeof(temp_stack); early_gdt_descr.address = (unsigned long)get_cpu_gdt_rw(smp_processor_id()); initial_gs = per_cpu_offset(smp_processor_id()); + smpboot_control = smp_processor_id(); #endif initial_code = (unsigned long)wakeup_long64; - saved_magic = 0x123456789abcdef0L; + saved_magic = 0x123456789abcdef0L; #endif /* CONFIG_64BIT */ /* diff --git a/arch/x86/kernel/asm-offsets.c b/arch/x86/kernel/asm-offsets.c index 82c783da16a8..797ae1a15c91 100644 --- a/arch/x86/kernel/asm-offsets.c +++ b/arch/x86/kernel/asm-offsets.c @@ -108,6 +108,7 @@ static void __used common(void) OFFSET(TSS_sp1, tss_struct, x86_tss.sp1); OFFSET(TSS_sp2, tss_struct, x86_tss.sp2); OFFSET(X86_top_of_stack, pcpu_hot, top_of_stack); + OFFSET(X86_current_task, pcpu_hot, current_task); #ifdef CONFIG_CALL_DEPTH_TRACKING OFFSET(X86_call_depth, pcpu_hot, call_depth); #endif diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index 222efd4a09bc..cc1b145055ac 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -61,8 +61,8 @@ SYM_CODE_START_NOALIGN(startup_64) * tables and then reload them. */ - /* Set up the stack for verify_cpu(), similar to initial_stack below */ - leaq (__end_init_task - FRAME_SIZE)(%rip), %rsp + /* Set up the stack for verify_cpu() */ + leaq (__end_init_task - PTREGS_SIZE)(%rip), %rsp leaq _text(%rip), %rdi @@ -241,6 +241,24 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) UNWIND_HINT_EMPTY ANNOTATE_NOENDBR // above +#ifdef CONFIG_SMP + movl smpboot_control(%rip), %ecx + + /* Get the per cpu offset for the given CPU# which is in ECX */ + movq __per_cpu_offset(,%rcx,8), %rdx +#else + xorl %edx, %edx /* zero-extended to clear all of RDX */ +#endif /* CONFIG_SMP */ + + /* + * Setup a boot time stack - Any secondary CPU will have lost its stack + * by now because the cr3-switch above unmaps the real-mode stack. + * + * RDX contains the per-cpu offset + */ + movq pcpu_hot + X86_current_task(%rdx), %rax + movq TASK_threadsp(%rax), %rsp + /* * We must switch to a new descriptor in kernel space for the GDT * because soon the kernel won't have access anymore to the userspace @@ -275,12 +293,6 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) movl initial_gs+4(%rip),%edx wrmsr - /* - * Setup a boot time stack - Any secondary CPU will have lost its stack - * by now because the cr3-switch above unmaps the real-mode stack - */ - movq initial_stack(%rip), %rsp - /* Setup and Load IDT */ pushq %rsi call early_setup_idt @@ -372,7 +384,11 @@ SYM_CODE_END(secondary_startup_64) SYM_CODE_START(start_cpu0) ANNOTATE_NOENDBR UNWIND_HINT_EMPTY - movq initial_stack(%rip), %rsp + + /* Find the idle task stack */ + movq PER_CPU_VAR(pcpu_hot) + X86_current_task, %rcx + movq TASK_threadsp(%rcx), %rsp + jmp .Ljump_to_C_code SYM_CODE_END(start_cpu0) #endif @@ -420,12 +436,6 @@ SYM_DATA(initial_gs, .quad INIT_PER_CPU_VAR(fixed_percpu_data)) #ifdef CONFIG_AMD_MEM_ENCRYPT SYM_DATA(initial_vc_handler, .quad handle_vc_boot_ghcb) #endif - -/* - * The FRAME_SIZE gap is a convention which helps the in-kernel unwinder - * reliably detect the end of the stack. - */ -SYM_DATA(initial_stack, .quad init_thread_union + THREAD_SIZE - FRAME_SIZE) __FINITDATA __INIT @@ -660,6 +670,9 @@ SYM_DATA_END(level1_fixmap_pgt) SYM_DATA(early_gdt_descr, .word GDT_ENTRIES*8-1) SYM_DATA_LOCAL(early_gdt_descr_base, .quad INIT_PER_CPU_VAR(gdt_page)) + .align 16 +SYM_DATA(smpboot_control, .long 0) + .align 16 /* This must match the first entry in level2_kernel_pgt */ SYM_DATA(phys_base, .quad 0x0) diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index fa1e35b0747c..ab8decac6294 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -1112,7 +1112,12 @@ static int do_boot_cpu(int apicid, int cpu, struct task_struct *idle, idle->thread.sp = (unsigned long)task_pt_regs(idle); early_gdt_descr.address = (unsigned long)get_cpu_gdt_rw(cpu); initial_code = (unsigned long)start_secondary; - initial_stack = idle->thread.sp; + + if (IS_ENABLED(CONFIG_X86_32)) { + initial_stack = idle->thread.sp; + } else { + smpboot_control = cpu; + } /* Enable the espfix hack for this CPU */ init_espfix_ap(cpu); diff --git a/arch/x86/xen/xen-head.S b/arch/x86/xen/xen-head.S index ffaa62167f6e..6bd391476656 100644 --- a/arch/x86/xen/xen-head.S +++ b/arch/x86/xen/xen-head.S @@ -49,7 +49,7 @@ SYM_CODE_START(startup_xen) ANNOTATE_NOENDBR cld - mov initial_stack(%rip), %rsp + leaq (__end_init_task - PTREGS_SIZE)(%rip), %rsp /* Set up %gs. * From patchwork Thu Mar 16 22:21:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 70997 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp26041wrt; Thu, 16 Mar 2023 15:24:27 -0700 (PDT) X-Google-Smtp-Source: AK7set/GwrnumeIaPfO20Om/PoYRYdDVAwwy8nhyLXq0vkXrarDuCiJGGcIWjUgLxPfDcFPB7nme X-Received: by 2002:a17:90b:3a83:b0:237:a500:eca6 with SMTP id om3-20020a17090b3a8300b00237a500eca6mr749152pjb.22.1679005467468; Thu, 16 Mar 2023 15:24:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679005467; cv=none; d=google.com; s=arc-20160816; b=WrJi3F9J1OMG2JPYvvq6QYjuIAsuhuh8o1r0YlgqXSe5JtqADBPyRGdN2g8n25zgAP FHhKiKJwe9Pt4aLIEPvDIjz+dXZkVxlW1XGr3qa7R3d3eGJCZmdROo7LhULipjAH2wAN a+B2gmonerq+laYkrewH22TcxT8QXBdnJA4XzfnjYXoA8fYnyEh3aSmi1Qjq92ks+GMj 86JRUMgXp0h+/ULT1YuiNu6pjb5v8QMx9HWHSK7ROI37x9Mk+9ubwO9dVP9PiEscsFRx sA+E9V5uK348D4ru6Sb5wEbpmAnHbvzBrTmLTjS682q6eqNQfcLyi7h48+UVpXGcKpJV 30KQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=A1Nso70j7TcOLvyUcSWwhrfhUH4RnAbpnDIgeFc4TIk=; b=szHWUhWaxFk2fg/Yai1DXthL3MqIAk9JVYXy9/s6w0ciRO2OfqxnXEZ+UpJiOYw5aJ g/AvLAbknhY0sFi6zjELM+XihgayfKUU21Gx0z8KAFkHwNMRbdWDNqdAdxisDlHRer0t Drqeiau8y+hjVN9dGZPB7VoV4zLYeGyCRhZEYHYb1KFawx8cdrhS74dcSQ9aCjHGwQEK GhkEwm3uOVwXKkcxPNtXZ2/E/6jOBB5TZb/FwzCa6JxbrV+sKJA8WOHpnuA6HNJW/79+ leD2xcmaFTrdSfSOzFyAY+2x5Wi1fixqQfBu0Uxb3PhzKz4sfP2BHiz72rog5qglssyx waxQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=kUrA8WNT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id v3-20020a17090a6b0300b00225c981fee5si5731695pjj.112.2023.03.16.15.24.12; Thu, 16 Mar 2023 15:24:27 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=kUrA8WNT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230149AbjCPWVx (ORCPT + 99 others); Thu, 16 Mar 2023 18:21:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33198 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229985AbjCPWVo (ORCPT ); Thu, 16 Mar 2023 18:21:44 -0400 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8DA6CB5AAE for ; Thu, 16 Mar 2023 15:21:20 -0700 (PDT) Received: by mail-wr1-x42b.google.com with SMTP id o7so2872083wrg.5 for ; Thu, 16 Mar 2023 15:21:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1679005278; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=A1Nso70j7TcOLvyUcSWwhrfhUH4RnAbpnDIgeFc4TIk=; b=kUrA8WNTnCZR7ZN8Y0MlY/qQ5ggU/8FvF9S6J9RiR8OsjerRCbOCfFcUrWEG9OWok2 shAGMGXwdC0cfh8K+fST9Hd04rigk8xYonifLmItP2Uo0hYOKUGn6SDNCV5HSkavNxsQ lIZ6+KGIgc3p6so7ukucolEoG0m65RHF1FSn4br3vbx/RZgJMlhRPusonBL+Jo1KmUHu CqCseNQ2/ivXBykeEPFerRkcZHzXzus9v7qJy/E9WXGSFk/f9MYmzcpW9yfFGmuHIx8+ 5jy+s+dimL5VaC7CkVSv5YpOhVikFmNqcATCn4hCJZdoSAegg3AJ3rZuUiEmyGIOjsTk cYGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679005278; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=A1Nso70j7TcOLvyUcSWwhrfhUH4RnAbpnDIgeFc4TIk=; b=0MfYoaCcSR6Zhgz8eN5D0QEft8rPn/u/yU22uq6jYZ9g2dohk8Vu0pSsAPUnz8LM3r +FuQHX3Mgt9szQP79FhH4kkYr0LLAMvm1A5zUWen3s5zwGG+XSe/PiBz7961yzvTjNNS eUDMX2eDqK5Hp+hAWmlFvpOfrbbhPIyWtLzOVvVIqVxHUOoHehnhYGxO7UBSv4cw/aVB 5eqmeUjD2G0fGhRU5oAnG+dds2G6kRMpvAgMl9nKFSwTmByaXjGY+zkL6tAagIzoXHD4 AMOhRiAWuZJPtZLxxrsq8DgryEvOuauqQZz261wF0KDhQ8BjayDc4TgfgJt8nQBIc4y3 2Kuw== X-Gm-Message-State: AO0yUKVU18fmplAC9v18xLIMqHJD39xpQPw68EY5Qdxt3ruO6Qdx2HJS dY3pSNsE3k1eQxUmd6L0Z8URAQ== X-Received: by 2002:a05:6000:1111:b0:2cf:e315:10b8 with SMTP id z17-20020a056000111100b002cfe31510b8mr6039732wrw.10.1679005278388; Thu, 16 Mar 2023 15:21:18 -0700 (PDT) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:4b87:78c3:3abe:7b0d]) by smtp.gmail.com with ESMTPSA id f9-20020adff989000000b002cea392f000sm439256wrr.69.2023.03.16.15.21.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 16 Mar 2023 15:21:18 -0700 (PDT) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, gpiccoli@igalia.com, David Woodhouse , Usama Arif Subject: [PATCH v15 07/12] x86/smpboot: Remove early_gdt_descr on 64-bit Date: Thu, 16 Mar 2023 22:21:04 +0000 Message-Id: <20230316222109.1940300-8-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230316222109.1940300-1-usama.arif@bytedance.com> References: <20230316222109.1940300-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760564836725363446?= X-GMAIL-MSGID: =?utf-8?q?1760564836725363446?= From: Brian Gerst Build the GDT descriptor on the stack instead. Signed-off-by: Brian Gerst Reviewed-by: David Woodhouse Signed-off-by: David Woodhouse Tested-by: Usama Arif Signed-off-by: Usama Arif Tested-by: Guilherme G. Piccoli --- arch/x86/kernel/acpi/sleep.c | 2 -- arch/x86/kernel/head_64.S | 11 ++++++----- arch/x86/kernel/smpboot.c | 2 +- 3 files changed, 7 insertions(+), 8 deletions(-) diff --git a/arch/x86/kernel/acpi/sleep.c b/arch/x86/kernel/acpi/sleep.c index 1b4c43d0819a..de89bb4719d0 100644 --- a/arch/x86/kernel/acpi/sleep.c +++ b/arch/x86/kernel/acpi/sleep.c @@ -127,8 +127,6 @@ int x86_acpi_suspend_lowlevel(void) * value is in the actual %rsp register. */ current->thread.sp = (unsigned long)temp_stack + sizeof(temp_stack); - early_gdt_descr.address = - (unsigned long)get_cpu_gdt_rw(smp_processor_id()); initial_gs = per_cpu_offset(smp_processor_id()); smpboot_control = smp_processor_id(); #endif diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index cc1b145055ac..a5b46c2fba05 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -265,7 +265,12 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) * addresses where we're currently running on. We have to do that here * because in 32bit we couldn't load a 64bit linear address. */ - lgdt early_gdt_descr(%rip) + subq $16, %rsp + movw $(GDT_SIZE-1), (%rsp) + leaq gdt_page(%rdx), %rax + movq %rax, 2(%rsp) + lgdt (%rsp) + addq $16, %rsp /* set up data segments */ xorl %eax,%eax @@ -667,10 +672,6 @@ SYM_DATA_END(level1_fixmap_pgt) .data .align 16 -SYM_DATA(early_gdt_descr, .word GDT_ENTRIES*8-1) -SYM_DATA_LOCAL(early_gdt_descr_base, .quad INIT_PER_CPU_VAR(gdt_page)) - - .align 16 SYM_DATA(smpboot_control, .long 0) .align 16 diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index ab8decac6294..67224e61310c 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -1110,10 +1110,10 @@ static int do_boot_cpu(int apicid, int cpu, struct task_struct *idle, start_ip = real_mode_header->trampoline_start64; #endif idle->thread.sp = (unsigned long)task_pt_regs(idle); - early_gdt_descr.address = (unsigned long)get_cpu_gdt_rw(cpu); initial_code = (unsigned long)start_secondary; if (IS_ENABLED(CONFIG_X86_32)) { + early_gdt_descr.address = (unsigned long)get_cpu_gdt_rw(cpu); initial_stack = idle->thread.sp; } else { smpboot_control = cpu; From patchwork Thu Mar 16 22:21:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 70998 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp26847wrt; Thu, 16 Mar 2023 15:26:44 -0700 (PDT) X-Google-Smtp-Source: AK7set9CDdIgTfGTs4nmWFdMVynHIH/0Rnz4XrnElAKrpdyO9cPLmIXADaSU3jS1a/1KUF8VQmgw X-Received: by 2002:a17:90a:192:b0:23d:4188:ad86 with SMTP id 18-20020a17090a019200b0023d4188ad86mr5471522pjc.18.1679005604645; Thu, 16 Mar 2023 15:26:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679005604; cv=none; d=google.com; s=arc-20160816; b=Tga7jqcdeD4Bhv13VLXeBDQ+gHC/d3KoCvYudBGLzlZOPaNq08+ItojP/c66NQkdYF 4zK1o1yEPmLC/hncB8U3B5KC/1gQV0dSbDvN9R92FG29eixBN7J133Za0aTv7OxCv6nI GawC9ykEnrZ4juf+3hsVXVrwduoabov/Ag/8YwLfK4WxD3/J1ihgBPQuyWrMVNdn6mqE JNMvxwR4wShhSh+YuuRQOtdcQhy+afGfd1esGw+7pkkFuUs2ipgRtEJ6UkSKX96wWfow d8MjQ2fKUvAKiWdZCIrcugl5qzJaYXRbri5RtlJ+scyvs+hM7/ZTqNgIgNNJaScVl5LQ 6/Jw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=FULGw2JzzQsOYOrWK/IeWDfn0l3ZyQtosjohoUs52UM=; b=rdPAIMcbL8zt04zIAAEjBVsjeQCWGZpwv8LvjM68JR3aVh+Za00NH/L2NXM/h2LdBw MF63jtxrr4xbaw44LoJwfCL/dU9N40SGbztK8Ks0TtyunCqfteDr0MWE9wtGARbsVtUo PdGnMAwV3t/zK0+XSwXyvf7RXqxsq1/W1TAS4hTkt6ybbzPDgGKknnsUGaa/bosD9zgr KN4lLhNzOn1CtzTPftzDRZv26lNCUUz4xPn4MD71pKbklLdAy3YAeuJVw+0MyCi/d6yB h5irmDeeCqNROsIgvvRL5RIYB423IFkx4j/R7ckHiYdRTvmvTnhUCsFRLEJLm1sqhv3A laOA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=XJkZiiYe; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id kk2-20020a17090b4a0200b0022c83e223desi6802583pjb.58.2023.03.16.15.26.29; Thu, 16 Mar 2023 15:26:44 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=XJkZiiYe; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230082AbjCPWWB (ORCPT + 99 others); Thu, 16 Mar 2023 18:22:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33212 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229888AbjCPWVo (ORCPT ); Thu, 16 Mar 2023 18:21:44 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 32135B2558 for ; Thu, 16 Mar 2023 15:21:21 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id h17so2862894wrt.8 for ; Thu, 16 Mar 2023 15:21:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1679005279; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FULGw2JzzQsOYOrWK/IeWDfn0l3ZyQtosjohoUs52UM=; b=XJkZiiYe6qukEyVX/ujwP5etOZk592Q68jftwrVXpL4JMdoU251Sic4GKKxqoURB75 gfXMd+jOXjW5LK3dYpkUe8S2pEZcRggenTmD9+KE8eZgb7yDzwQ3EMBwGdAl5IPg5Svn 1adM4NXK26p396oaoPI2OcwyTLd5TOZ1YG6WQ6WmrT6yfgHUpju2g8gPjeQRxBtKMwN6 f/t0MCNQSMV0DnRW6aFDulqBtXy3RiaUMqnmpi5FlWc+3X8EyGqzmzqkqNLeFBvPnhqe IFrtkO2JXGZY1L74uXsj4Kv61nAuvl270HxBVyJPFpasxMgN+NUXU8heKCaGYPurMp14 C9jA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679005279; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FULGw2JzzQsOYOrWK/IeWDfn0l3ZyQtosjohoUs52UM=; b=riWkjyuVe9lEt2215+k1De30jJaQH0mcyw4lTCjpXFRvVS3jrENWv/+yFCY6xcK9nk e4tSPbfTrQFMNdKkGU4KIC0XYgGsUvY2Oub4qcMtnJUq9yESZWnnL/dbCoeCg60eAyiA 4CS3noF2kLyMYnGBqACiwYsyebt/EtojqfMrHI0nN3OIopgFss5q7C+3Aylh9A2h9snb H5NnzlAUbvYV6fQB86cIPKFgR3ymVN1/Ly9Zkc8rm4eQXcwAegTxruKcTw/cpw+QaXzE uQbbKn7OvCALXxSk6q60CMdI+GYLIQLAFfgFeJCZTM4iUjqszqXNRLo3oGtTtdTmCblH z38g== X-Gm-Message-State: AO0yUKXr/aK7vuau0wJx+C7ukOsln7ELGm/2AtwjmysWhAO54w7csub8 l5Xx+L0PPqIJSvnbAdSgAv5fLA== X-Received: by 2002:adf:dc0f:0:b0:2cf:f0c3:79ba with SMTP id t15-20020adfdc0f000000b002cff0c379bamr5429317wri.67.1679005279217; Thu, 16 Mar 2023 15:21:19 -0700 (PDT) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:4b87:78c3:3abe:7b0d]) by smtp.gmail.com with ESMTPSA id f9-20020adff989000000b002cea392f000sm439256wrr.69.2023.03.16.15.21.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 16 Mar 2023 15:21:18 -0700 (PDT) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, gpiccoli@igalia.com, David Woodhouse , Usama Arif Subject: [PATCH v15 08/12] x86/smpboot: Remove initial_gs Date: Thu, 16 Mar 2023 22:21:05 +0000 Message-Id: <20230316222109.1940300-9-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230316222109.1940300-1-usama.arif@bytedance.com> References: <20230316222109.1940300-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760564980697133778?= X-GMAIL-MSGID: =?utf-8?q?1760564980697133778?= From: Brian Gerst Given its CPU#, each CPU can find its own per-cpu offset, and directly set GSBASE accordingly. The global variable can be eliminated. Signed-off-by: Brian Gerst Reviewed-by: David Woodhouse Signed-off-by: David Woodhouse Tested-by: Usama Arif Signed-off-by: Usama Arif Tested-by: Guilherme G. Piccoli --- arch/x86/include/asm/realmode.h | 1 - arch/x86/kernel/acpi/sleep.c | 1 - arch/x86/kernel/head_64.S | 22 ++++++++-------------- arch/x86/kernel/smpboot.c | 2 -- 4 files changed, 8 insertions(+), 18 deletions(-) diff --git a/arch/x86/include/asm/realmode.h b/arch/x86/include/asm/realmode.h index a336feef0af1..f6a1737c77be 100644 --- a/arch/x86/include/asm/realmode.h +++ b/arch/x86/include/asm/realmode.h @@ -59,7 +59,6 @@ extern struct real_mode_header *real_mode_header; extern unsigned char real_mode_blob_end[]; extern unsigned long initial_code; -extern unsigned long initial_gs; extern unsigned long initial_stack; #ifdef CONFIG_AMD_MEM_ENCRYPT extern unsigned long initial_vc_handler; diff --git a/arch/x86/kernel/acpi/sleep.c b/arch/x86/kernel/acpi/sleep.c index de89bb4719d0..1328c221af30 100644 --- a/arch/x86/kernel/acpi/sleep.c +++ b/arch/x86/kernel/acpi/sleep.c @@ -127,7 +127,6 @@ int x86_acpi_suspend_lowlevel(void) * value is in the actual %rsp register. */ current->thread.sp = (unsigned long)temp_stack + sizeof(temp_stack); - initial_gs = per_cpu_offset(smp_processor_id()); smpboot_control = smp_processor_id(); #endif initial_code = (unsigned long)wakeup_long64; diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index a5b46c2fba05..6a8238702eab 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -66,18 +66,10 @@ SYM_CODE_START_NOALIGN(startup_64) leaq _text(%rip), %rdi - /* - * initial_gs points to initial fixed_percpu_data struct with storage for - * the stack protector canary. Global pointer fixups are needed at this - * stage, so apply them as is done in fixup_pointer(), and initialize %gs - * such that the canary can be accessed at %gs:40 for subsequent C calls. - */ + /* Setup GSBASE to allow stack canary access for C code */ movl $MSR_GS_BASE, %ecx - movq initial_gs(%rip), %rax - movq $_text, %rdx - subq %rdx, %rax - addq %rdi, %rax - movq %rax, %rdx + leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx + movl %edx, %eax shrq $32, %rdx wrmsr @@ -294,8 +286,11 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) * the per cpu areas are set up. */ movl $MSR_GS_BASE,%ecx - movl initial_gs(%rip),%eax - movl initial_gs+4(%rip),%edx +#ifndef CONFIG_SMP + leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx +#endif + movl %edx, %eax + shrq $32, %rdx wrmsr /* Setup and Load IDT */ @@ -437,7 +432,6 @@ SYM_CODE_END(vc_boot_ghcb) __REFDATA .balign 8 SYM_DATA(initial_code, .quad x86_64_start_kernel) -SYM_DATA(initial_gs, .quad INIT_PER_CPU_VAR(fixed_percpu_data)) #ifdef CONFIG_AMD_MEM_ENCRYPT SYM_DATA(initial_vc_handler, .quad handle_vc_boot_ghcb) #endif diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 67224e61310c..28d1643eee99 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -1084,8 +1084,6 @@ int common_cpu_up(unsigned int cpu, struct task_struct *idle) #ifdef CONFIG_X86_32 /* Stack for startup_32 can be just as for start_secondary onwards */ per_cpu(pcpu_hot.top_of_stack, cpu) = task_top_of_stack(idle); -#else - initial_gs = per_cpu_offset(cpu); #endif return 0; } From patchwork Thu Mar 16 22:21:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 70994 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp25781wrt; Thu, 16 Mar 2023 15:23:41 -0700 (PDT) X-Google-Smtp-Source: AK7set8KAU9kZEq8X80znT66ZG/O29p9AWx58UG9dBFTz7ZXrBxQFAyOa/r5jLcxj5VTYbW8Rkok X-Received: by 2002:a05:6a21:7891:b0:cc:c5db:ea4a with SMTP id bf17-20020a056a21789100b000ccc5dbea4amr6873498pzc.33.1679005421508; Thu, 16 Mar 2023 15:23:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679005421; cv=none; d=google.com; s=arc-20160816; b=jqv33Dq5S6FqwKHhxRg4erXJDLWWsl5Fd3upW+iM6zZlRIoVG2ONlmz9hUSM2CJ4Rz qqnhDuyv9XN9Awg+wx4PXUEj4XdY0GYLW0h40/27Mz5kwaO5QfvxzFdeUcbF4W0mOyZw uxTRe6600RUUBqY5m9znTh6clqAeEz9uGMP9y+TtEFydCivvpDiYeVBXd6wZX4yOx/aZ 4AVEkY+AEu3ZOQfiudW0mAxqrRuN2/DOsJIfZ5yWd8JJnLphGfknoqD3Qx219XccVd4E E9H35WGws9Of5rcuWuXAdTEXZR/XM5fv7//MOFw+V78D1+dUtQOD1U84VuhjMEEEsANr LjtQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=v0n30u7JAEzq/8YxEy0E6GMO580Q3zBlmdKg/T/xx0Y=; b=qauYZbZbfg0ZO38qFiTXGm3th/elitLValrSU61ymwoaUNi4lM+rcnCoO89uh3K3TA scXwyy2L1jKqJCljcfBeUTy1ac6QjTLEmZTBC9q47OZ8KzC1e4pgYf6Q5O/yxgHz2YSk knbmMncorOpVJM9qxEJd0mv4LDJDHrO1xanfHGJrzolTSsZ4CPJ73bJ8LOikOGHpDJH2 E+lJiI3SD1ucRgHv7xgppYMm1pCNhB9HI7tqbKTojeCsmHSsvrhcNsqwllYEbqQdjaaW nxBjRXIjKrYOjw+HFMqh/VZRmDt4I1AF0Imbpb1tTQZqFgwKpNedo3JMTCKhsMcGtQJO YInQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=ejbUgJ+Z; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l28-20020a635b5c000000b0050726756ee8si466619pgm.93.2023.03.16.15.23.27; Thu, 16 Mar 2023 15:23:41 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=ejbUgJ+Z; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229815AbjCPWWd (ORCPT + 99 others); Thu, 16 Mar 2023 18:22:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33184 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230297AbjCPWWL (ORCPT ); Thu, 16 Mar 2023 18:22:11 -0400 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D0924B1A63 for ; Thu, 16 Mar 2023 15:21:20 -0700 (PDT) Received: by mail-wm1-x32b.google.com with SMTP id l15-20020a05600c4f0f00b003ed58a9a15eso2164727wmq.5 for ; Thu, 16 Mar 2023 15:21:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1679005280; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=v0n30u7JAEzq/8YxEy0E6GMO580Q3zBlmdKg/T/xx0Y=; b=ejbUgJ+ZdWl9323hvtrT6oe36zrypriL4x3Xs3haidUgUYHXrccoaR3G8seQuYjS0a eQCNFh5UNTAYV8U3FSQI0f7s6gXJLrq6CVdtS+m7iwZEBAfqgwmEOmNxTeQSjrU0xIh0 S2OK3BPSSXUMclbNQDLqwuUU1E+RqNrkwV0c0DsO51ypV0VcOTNTAwvxiGASnLvYnVR5 xOTF4WbRsXBexOfK6yL7Qenj5/W25lYrzc3DUEewxqURRARcieKUXtj5uuG65uVOpEoC FuIZShXlP8pRA9zgyQLy5RSepCYHwyYvP/DPtBS865YOlRNZZGo48l4FwquiWDHQpLjo glyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679005280; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=v0n30u7JAEzq/8YxEy0E6GMO580Q3zBlmdKg/T/xx0Y=; b=2Mlv3Bx8Bzwwoqe2ZjrtpvQp5T/X/LYTrL6gplar9KQKEMxNxA0BehCqAl7ANDFm3e OxfT+wxfY5aOeu1Qnf9Q0MjYGvzd3xzfDhq7tVEZifGCwTuPE164v3dk+d9srl6OFoQa +84akTtyzwKQk7bVwpJuWn/4VXQX0mBc9x/4jKRmS9+bkqQTarPBXo9f3zOKnjoW/Ak0 4nVR3NgBXjigb83YxPwItfu1voFf0lScj2tKsh2tMM9LhqNhglAgypGvWkDulw5T/Rme 9BqEHpEz0CxD5Zzf0ejHLPUsicHFl4IVY0PqU3xQF4uOylThgW6zzK9O8VgaV676bJW7 V2YA== X-Gm-Message-State: AO0yUKUzpMff2I2eRENLmUQGA23iFzcnAfgYxLRt1bPOaDK4mox5S+6z EYktLjONrL0CSQhjKuIpjGSqYg== X-Received: by 2002:a05:600c:3b07:b0:3ed:301d:9f86 with SMTP id m7-20020a05600c3b0700b003ed301d9f86mr8382105wms.9.1679005280180; Thu, 16 Mar 2023 15:21:20 -0700 (PDT) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:4b87:78c3:3abe:7b0d]) by smtp.gmail.com with ESMTPSA id f9-20020adff989000000b002cea392f000sm439256wrr.69.2023.03.16.15.21.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 16 Mar 2023 15:21:19 -0700 (PDT) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, gpiccoli@igalia.com, David Woodhouse , Usama Arif Subject: [PATCH v15 09/12] x86/smpboot: Support parallel startup of secondary CPUs Date: Thu, 16 Mar 2023 22:21:06 +0000 Message-Id: <20230316222109.1940300-10-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230316222109.1940300-1-usama.arif@bytedance.com> References: <20230316222109.1940300-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760564788487790948?= X-GMAIL-MSGID: =?utf-8?q?1760564788487790948?= From: David Woodhouse Rework the real-mode startup code to allow for APs to be brought up in parallel. This is in two parts: 1. Introduce a bit-spinlock to prevent them from all using the real mode stack at the same time. 2. Avoid needing to use the global smpboot_control variable to pass each AP its CPU#. To achieve the latter, export the cpuid_to_apicid[] array so that each AP can find its own CPU# by searching therein based on its APIC ID. Introduce flags in the top bits of smpboot_control which indicate methods by which an AP should find its CPU#. For a serialized bringup, the CPU# is explicitly passed in the low bits of smpboot_control as before. For parallel mode there are flags directing the AP to find its APIC ID in CPUID leaf 0x0b (for X2APIC mode) or CPUID leaf 0x01 where 8 bits are sufficient, then perform the cpuid_to_apicid[] lookup with that. Parallel startup may be disabled by a command line option, and also if: • AMD SEV-ES is in use, since the AP may not use CPUID that early. • X2APIC is enabled, but CPUID leaf 0xb is not present and correct. • X2APIC is not enabled but not even CPUID leaf 0x01 exists. Aside from the fact that APs will now look up their CPU# via the newly-exported cpuid_to_apicid[] table, there is no behavioural change intended yet, since new parallel CPUHP states have not — yet — been added. [ tglx: Initial proof of concept patch with bitlock and APIC ID lookup ] [ dwmw2: Rework and testing, commit message, CPUID 0x1 and CPU0 support ] [ seanc: Fix stray override of initial_gs in common_cpu_up() ] [ Oleksandr Natalenko: reported suspend/resume issue fixed in x86_acpi_suspend_lowlevel ] Co-developed-by: Thomas Gleixner Co-developed-by: Brian Gerst Signed-off-by: Thomas Gleixner Signed-off-by: Brian Gerst Signed-off-by: David Woodhouse Signed-off-by: Usama Arif Tested-by: Paul E. McKenney Tested-by: Kim Phillips Tested-by: Oleksandr Natalenko Tested-by: Guilherme G. Piccoli --- .../admin-guide/kernel-parameters.txt | 3 + arch/x86/include/asm/cpu.h | 1 + arch/x86/include/asm/realmode.h | 3 + arch/x86/include/asm/smp.h | 6 ++ arch/x86/kernel/acpi/sleep.c | 9 ++- arch/x86/kernel/apic/apic.c | 2 +- arch/x86/kernel/cpu/topology.c | 3 +- arch/x86/kernel/head_64.S | 65 +++++++++++++++++++ arch/x86/kernel/smpboot.c | 50 +++++++++++++- arch/x86/realmode/init.c | 3 + arch/x86/realmode/rm/trampoline_64.S | 27 ++++++-- 11 files changed, 163 insertions(+), 9 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 6cfa6e3996cf..7bb7020f97e2 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3819,6 +3819,9 @@ nomodule Disable module load + no_parallel_bringup + [X86,SMP] Disable parallel bring-up of secondary cores. + nopat [X86] Disable PAT (page attribute table extension of pagetables) support. diff --git a/arch/x86/include/asm/cpu.h b/arch/x86/include/asm/cpu.h index 78796b98a544..ef8ba318dca1 100644 --- a/arch/x86/include/asm/cpu.h +++ b/arch/x86/include/asm/cpu.h @@ -97,5 +97,6 @@ static inline bool intel_cpu_signatures_match(unsigned int s1, unsigned int p1, extern u64 x86_read_arch_cap_msr(void); int intel_find_matching_signature(void *mc, unsigned int csig, int cpf); int intel_microcode_sanity_check(void *mc, bool print_err, int hdr_type); +int check_extended_topology_leaf(int leaf); #endif /* _ASM_X86_CPU_H */ diff --git a/arch/x86/include/asm/realmode.h b/arch/x86/include/asm/realmode.h index f6a1737c77be..87e5482acd0d 100644 --- a/arch/x86/include/asm/realmode.h +++ b/arch/x86/include/asm/realmode.h @@ -52,6 +52,7 @@ struct trampoline_header { u64 efer; u32 cr4; u32 flags; + u32 lock; #endif }; @@ -64,6 +65,8 @@ extern unsigned long initial_stack; extern unsigned long initial_vc_handler; #endif +extern u32 *trampoline_lock; + extern unsigned char real_mode_blob[]; extern unsigned char real_mode_relocs[]; diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h index bf2c51df9e0b..1cf4f1e57570 100644 --- a/arch/x86/include/asm/smp.h +++ b/arch/x86/include/asm/smp.h @@ -203,4 +203,10 @@ extern unsigned int smpboot_control; #endif /* !__ASSEMBLY__ */ +/* Control bits for startup_64 */ +#define STARTUP_APICID_CPUID_0B 0x80000000 +#define STARTUP_APICID_CPUID_01 0x40000000 + +#define STARTUP_PARALLEL_MASK (STARTUP_APICID_CPUID_01 | STARTUP_APICID_CPUID_0B) + #endif /* _ASM_X86_SMP_H */ diff --git a/arch/x86/kernel/acpi/sleep.c b/arch/x86/kernel/acpi/sleep.c index 1328c221af30..6dfecb27b846 100644 --- a/arch/x86/kernel/acpi/sleep.c +++ b/arch/x86/kernel/acpi/sleep.c @@ -16,6 +16,7 @@ #include #include #include +#include #include #include "../../realmode/rm/wakeup.h" @@ -127,7 +128,13 @@ int x86_acpi_suspend_lowlevel(void) * value is in the actual %rsp register. */ current->thread.sp = (unsigned long)temp_stack + sizeof(temp_stack); - smpboot_control = smp_processor_id(); + /* + * Ensure the CPU knows which one it is when it comes back, if + * it isn't in parallel mode and expected to work that out for + * itself. + */ + if (!(smpboot_control & STARTUP_PARALLEL_MASK)) + smpboot_control = smp_processor_id(); #endif initial_code = (unsigned long)wakeup_long64; saved_magic = 0x123456789abcdef0L; diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c index 20d9a604da7c..ac1d7e5da1f2 100644 --- a/arch/x86/kernel/apic/apic.c +++ b/arch/x86/kernel/apic/apic.c @@ -2377,7 +2377,7 @@ static int nr_logical_cpuids = 1; /* * Used to store mapping between logical CPU IDs and APIC IDs. */ -static int cpuid_to_apicid[] = { +int cpuid_to_apicid[] = { [0 ... NR_CPUS - 1] = -1, }; diff --git a/arch/x86/kernel/cpu/topology.c b/arch/x86/kernel/cpu/topology.c index 5e868b62a7c4..4373442e500a 100644 --- a/arch/x86/kernel/cpu/topology.c +++ b/arch/x86/kernel/cpu/topology.c @@ -9,6 +9,7 @@ #include #include #include +#include #include "cpu.h" @@ -32,7 +33,7 @@ EXPORT_SYMBOL(__max_die_per_package); /* * Check if given CPUID extended topology "leaf" is implemented */ -static int check_extended_topology_leaf(int leaf) +int check_extended_topology_leaf(int leaf) { unsigned int eax, ebx, ecx, edx; diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index 6a8238702eab..65bca47d84a1 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -25,6 +25,7 @@ #include #include #include +#include /* * We are not able to switch in one step to the final KERNEL ADDRESS SPACE @@ -234,8 +235,61 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) ANNOTATE_NOENDBR // above #ifdef CONFIG_SMP + /* + * For parallel boot, the APIC ID is retrieved from CPUID, and then + * used to look up the CPU number. For booting a single CPU, the + * CPU number is encoded in smpboot_control. + * + * Bit 31 STARTUP_APICID_CPUID_0B flag (use CPUID 0x0b) + * Bit 30 STARTUP_APICID_CPUID_01 flag (use CPUID 0x01) + * Bit 0-24 CPU# if STARTUP_APICID_CPUID_xx flags are not set + */ movl smpboot_control(%rip), %ecx + testl $STARTUP_APICID_CPUID_0B, %ecx + jnz .Luse_cpuid_0b + testl $STARTUP_APICID_CPUID_01, %ecx + jnz .Luse_cpuid_01 + andl $0x0FFFFFFF, %ecx + jmp .Lsetup_cpu + +.Luse_cpuid_01: + mov $0x01, %eax + cpuid + mov %ebx, %edx + shr $24, %edx + jmp .Lsetup_AP +.Luse_cpuid_0b: + mov $0x0B, %eax + xorl %ecx, %ecx + cpuid + +.Lsetup_AP: + /* EDX contains the APIC ID of the current CPU */ + xorq %rcx, %rcx + leaq cpuid_to_apicid(%rip), %rbx + +.Lfind_cpunr: + cmpl (%rbx,%rcx,4), %edx + jz .Lsetup_cpu + inc %ecx +#ifdef CONFIG_FORCE_NR_CPUS + cmpl $NR_CPUS, %ecx +#else + cmpl nr_cpu_ids(%rip), %ecx +#endif + jb .Lfind_cpunr + + /* APIC ID not found in the table. Drop the trampoline lock and bail. */ + movq trampoline_lock(%rip), %rax + lock + btrl $0, (%rax) + +1: cli + hlt + jmp 1b + +.Lsetup_cpu: /* Get the per cpu offset for the given CPU# which is in ECX */ movq __per_cpu_offset(,%rcx,8), %rdx #else @@ -264,6 +318,14 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) lgdt (%rsp) addq $16, %rsp + /* Drop the realmode protection. For the boot CPU the pointer is NULL! */ + movq trampoline_lock(%rip), %rax + testq %rax, %rax + jz .Lsetup_data_segments + lock + btrl $0, (%rax) + +.Lsetup_data_segments: /* set up data segments */ xorl %eax,%eax movl %eax,%ds @@ -293,6 +355,7 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) shrq $32, %rdx wrmsr +.Lsetup_idt: /* Setup and Load IDT */ pushq %rsi call early_setup_idt @@ -435,6 +498,8 @@ SYM_DATA(initial_code, .quad x86_64_start_kernel) #ifdef CONFIG_AMD_MEM_ENCRYPT SYM_DATA(initial_vc_handler, .quad handle_vc_boot_ghcb) #endif + +SYM_DATA(trampoline_lock, .quad 0); __FINITDATA __INIT diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 28d1643eee99..a9e48946fc89 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -797,6 +797,16 @@ static int __init cpu_init_udelay(char *str) } early_param("cpu_init_udelay", cpu_init_udelay); +static bool do_parallel_bringup __ro_after_init = true; + +static int __init no_parallel_bringup(char *str) +{ + do_parallel_bringup = false; + + return 0; +} +early_param("no_parallel_bringup", no_parallel_bringup); + static void __init smp_quirk_init_udelay(void) { /* if cmdline changed it from default, leave it alone */ @@ -1113,7 +1123,7 @@ static int do_boot_cpu(int apicid, int cpu, struct task_struct *idle, if (IS_ENABLED(CONFIG_X86_32)) { early_gdt_descr.address = (unsigned long)get_cpu_gdt_rw(cpu); initial_stack = idle->thread.sp; - } else { + } else if (!do_parallel_bringup) { smpboot_control = cpu; } @@ -1473,6 +1483,41 @@ void __init smp_prepare_cpus_common(void) set_cpu_sibling_map(0); } +/* + * We can do 64-bit AP bringup in parallel if the CPU reports its APIC + * ID in CPUID (either leaf 0x0B if we need the full APIC ID in X2APIC + * mode, or leaf 0x01 if 8 bits are sufficient). Otherwise it's too + * hard. And not for SEV-ES guests because they can't use CPUID that + * early. + */ +static bool prepare_parallel_bringup(void) +{ + if (IS_ENABLED(CONFIG_X86_32) || cc_platform_has(CC_ATTR_GUEST_STATE_ENCRYPT)) + return false; + + if (x2apic_mode) { + if (boot_cpu_data.cpuid_level < 0x0b) + return false; + + if (check_extended_topology_leaf(0x0b) != 0) { + pr_info("Disabling parallel bringup because CPUID 0xb looks untrustworthy\n"); + return false; + } + + pr_debug("Using CPUID 0xb for parallel CPU startup\n"); + smpboot_control = STARTUP_APICID_CPUID_0B; + } else { + /* Without X2APIC, what's in CPUID 0x01 should suffice. */ + if (boot_cpu_data.cpuid_level < 0x01) + return false; + + pr_debug("Using CPUID 0x1 for parallel CPU startup\n"); + smpboot_control = STARTUP_APICID_CPUID_01; + } + + return true; +} + /* * Prepare for SMP bootup. * @max_cpus: configured maximum number of CPUs, It is a legacy parameter @@ -1513,6 +1558,9 @@ void __init native_smp_prepare_cpus(unsigned int max_cpus) speculative_store_bypass_ht_init(); + if (do_parallel_bringup) + do_parallel_bringup = prepare_parallel_bringup(); + snp_set_wakeup_secondary_cpu(); } diff --git a/arch/x86/realmode/init.c b/arch/x86/realmode/init.c index af565816d2ba..788e5559549f 100644 --- a/arch/x86/realmode/init.c +++ b/arch/x86/realmode/init.c @@ -154,6 +154,9 @@ static void __init setup_real_mode(void) trampoline_header->flags = 0; + trampoline_lock = &trampoline_header->lock; + *trampoline_lock = 0; + trampoline_pgd = (u64 *) __va(real_mode_header->trampoline_pgd); /* Map the real mode stub as virtual == physical */ diff --git a/arch/x86/realmode/rm/trampoline_64.S b/arch/x86/realmode/rm/trampoline_64.S index e38d61d6562e..2dfb1c400167 100644 --- a/arch/x86/realmode/rm/trampoline_64.S +++ b/arch/x86/realmode/rm/trampoline_64.S @@ -37,6 +37,24 @@ .text .code16 +.macro LOAD_REALMODE_ESP + /* + * Make sure only one CPU fiddles with the realmode stack + */ +.Llock_rm\@: + btl $0, tr_lock + jnc 2f + pause + jmp .Llock_rm\@ +2: + lock + btsl $0, tr_lock + jc .Llock_rm\@ + + # Setup stack + movl $rm_stack_end, %esp +.endm + .balign PAGE_SIZE SYM_CODE_START(trampoline_start) cli # We should be safe anyway @@ -49,8 +67,7 @@ SYM_CODE_START(trampoline_start) mov %ax, %es mov %ax, %ss - # Setup stack - movl $rm_stack_end, %esp + LOAD_REALMODE_ESP call verify_cpu # Verify the cpu supports long mode testl %eax, %eax # Check for return code @@ -93,8 +110,7 @@ SYM_CODE_START(sev_es_trampoline_start) mov %ax, %es mov %ax, %ss - # Setup stack - movl $rm_stack_end, %esp + LOAD_REALMODE_ESP jmp .Lswitch_to_protected SYM_CODE_END(sev_es_trampoline_start) @@ -177,7 +193,7 @@ SYM_CODE_START(pa_trampoline_compat) * In compatibility mode. Prep ESP and DX for startup_32, then disable * paging and complete the switch to legacy 32-bit mode. */ - movl $rm_stack_end, %esp + LOAD_REALMODE_ESP movw $__KERNEL_DS, %dx movl $(CR0_STATE & ~X86_CR0_PG), %eax @@ -241,6 +257,7 @@ SYM_DATA_START(trampoline_header) SYM_DATA(tr_efer, .space 8) SYM_DATA(tr_cr4, .space 4) SYM_DATA(tr_flags, .space 4) + SYM_DATA(tr_lock, .space 4) SYM_DATA_END(trampoline_header) #include "trampoline_common.S" From patchwork Thu Mar 16 22:21:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 70995 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp25797wrt; Thu, 16 Mar 2023 15:23:45 -0700 (PDT) X-Google-Smtp-Source: AK7set/oWIGe5dDiJQtOxSVjBPOp27/YXdDMk+ZMiZXGxgoXVGW4jd6xxhD89eknHivszvjUuPNa X-Received: by 2002:a17:90b:1c06:b0:23d:3a3f:950d with SMTP id oc6-20020a17090b1c0600b0023d3a3f950dmr5207756pjb.40.1679005425047; Thu, 16 Mar 2023 15:23:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679005425; cv=none; d=google.com; s=arc-20160816; b=bQCrkI+XNVBXWtWU5dLV1OxY2WwXLP69kErPrLvELSocQIothpN6LpfdA3FQRb3VJp BPKCIzz/+6ABkUQ/aI5mzUWJ5nx+u9IDk4B7ByVFWkOtBR8Z2nPEvxMd7R7rhJxAfCF8 uzu/MpP4TwOuo9vxAxF95BYwH4Qy7NGnwuy5siKjPy4ERXJkrp4FbYWwwBPIBE6EQ9jA Gdn6Kx/tXz2sW0I/HenRI6Gk6G3f5JysC1pbDdYDYPM5fb0MltZIMbIMI1j6embF97uN XT6wt1w86+l1elhbHyXFy0lptNgrWY/RwEn/x7FOVw/jSz38LMEJsm8b3PraK8SsKkq7 ilbw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=RcZv3zbGH7k+CBej9wIKrUlEuzG8XkHUgLqLEwBzkYQ=; b=ziQv+I9eBsMM2TS25epmXNbeUdyeHnwfoId4515gMdIB4OawsEwTRVeg3Hi/2dMhRE CeTvl55dOYcFsbZDF6WBhoHYk+TF6UM02Q/SVeARap8AtKk7JKVuVFvbR/Gi9cmssmZQ fFVX7yXn58+dV3LEVWs7HcftPVOeEarV07ySkzgI5KQsStDQBwJ2j5tN5WmuyxQV3dS3 /GTU6OaEFq//iWxuzGjnE5+VSNVSf3ExzpDL77O+nrHa1sC15jYPImI9JTu+HIZhaYSC AxwOU/a63CXMbvYn4a7HHL4KGSuTC2h6aaUD+WTdWgjDJyxRGgQ260P36RTHplQYICmb Eo6Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=a80sFCpT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h10-20020a17090a2eca00b0023d37ca1deesi322511pjs.70.2023.03.16.15.23.30; Thu, 16 Mar 2023 15:23:45 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=a80sFCpT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230310AbjCPWWk (ORCPT + 99 others); Thu, 16 Mar 2023 18:22:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60642 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230287AbjCPWWL (ORCPT ); Thu, 16 Mar 2023 18:22:11 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C6857B6D0A for ; Thu, 16 Mar 2023 15:21:23 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id r29so2851448wra.13 for ; Thu, 16 Mar 2023 15:21:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1679005281; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RcZv3zbGH7k+CBej9wIKrUlEuzG8XkHUgLqLEwBzkYQ=; b=a80sFCpTfQlIpXTjBsvv7ruVIpaS8bqFb+SlrcuHl5I392mri/jymyx8NdqbaTK26C YdzkusJuVGbbG+cerQZAoFtmPsAeG83fpCop2fSPB4j7+NfDKhqyk+UUL/xyRPXD59UL m/JTjGaANhJbkCdXrbqxuO3fcULccPrNcnEkGek1+2CXQxorClocGjcOGE8ByYYHCa97 HhZ2n2NqlKTG8F0FYunRXsj664X3UxvK7D/9mG68dxr1byoc1gYLOUFmoGFsb5FN/GWi +7lWXAqIY8D2vGoHqH8nVgsDFU32g88zJiNgPfmFXQFV2W5wivqgZrlZpo6nQhbIfU+e HfvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679005281; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RcZv3zbGH7k+CBej9wIKrUlEuzG8XkHUgLqLEwBzkYQ=; b=mEofbTxgjMIT5yWQCMoZjCto33lH4Bzg9LJ0xHWEmh25KE+YodDhxfEqlGWXzf4PEQ BqR7KGGeR/OFaVGEjnKkefltrafiKZ/+iOUdwsKt1uRWq85q820pIcAi90Fv4aeW4Mfr uetONcGluYvzz0oP6C02CU5tXg4un4oUEfcH6XxMRq8Mu4iZyMycxK/VSYRWR0qO5ak+ TMzYLjh+pQaY/6MVPlpEunWOEy5vOj4exRq3Q7oNEaBMb0wRkdV7lX+pdrA1jQzM5x5A lrt92oEoX65iPbawIfufWxWjap9ksrtYxyN5Ci1TpZHpH+S9w5BjleDCh7lFuDyUQg+Z Yu/Q== X-Gm-Message-State: AO0yUKX0ylv8JLai6ct6P7YeAyPEd3G7q7zyg4Tkp1mSEf70tC6tYFSw HUD0La1Sylv1ZcDvmZMICuURwg== X-Received: by 2002:a5d:63c3:0:b0:2cf:e868:f789 with SMTP id c3-20020a5d63c3000000b002cfe868f789mr5360294wrw.48.1679005281208; Thu, 16 Mar 2023 15:21:21 -0700 (PDT) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:4b87:78c3:3abe:7b0d]) by smtp.gmail.com with ESMTPSA id f9-20020adff989000000b002cea392f000sm439256wrr.69.2023.03.16.15.21.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 16 Mar 2023 15:21:20 -0700 (PDT) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, gpiccoli@igalia.com, David Woodhouse , Usama Arif Subject: [PATCH v15 10/12] x86/smpboot: Send INIT/SIPI/SIPI to secondary CPUs in parallel Date: Thu, 16 Mar 2023 22:21:07 +0000 Message-Id: <20230316222109.1940300-11-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230316222109.1940300-1-usama.arif@bytedance.com> References: <20230316222109.1940300-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760564792251455865?= X-GMAIL-MSGID: =?utf-8?q?1760564792251455865?= From: David Woodhouse When the APs can find their own APIC ID without assistance, perform the AP bringup in parallel. Register a CPUHP_BP_PARALLEL_DYN stage "x86/cpu:kick" which just calls do_boot_cpu() to deliver INIT/SIPI/SIPI to each AP in turn before the normal native_cpu_up() does the rest of the hand-holding. The APs will then take turns through the real mode code (which has its own bitlock for exclusion) until they make it to their own stack, then proceed through the first few lines of start_secondary() and execute these parts in parallel: start_secondary() -> cr4_init() -> (some 32-bit only stuff so not in the parallel cases) -> cpu_init_secondary() -> cpu_init_exception_handling() -> cpu_init() -> wait_for_master_cpu() At this point they wait for the BSP to set their bit in cpu_callout_mask (from do_wait_cpu_initialized()), and release them to continue through the rest of cpu_init() and beyond. This reduces the time taken for bringup on my 28-thread Haswell system from about 120ms to 80ms. On a socket 96-thread Skylake it takes the bringup time from 500ms to 100ms. There is more speedup to be had by doing the remaining parts in parallel too — especially notify_cpu_starting() in which the AP takes itself through all the stages from CPUHP_BRINGUP_CPU to CPUHP_ONLINE. But those require careful auditing to ensure they are reentrant, before we can go that far. Signed-off-by: David Woodhouse Signed-off-by: Usama Arif Tested-by: Paul E. McKenney Tested-by: Kim Phillips Tested-by: Oleksandr Natalenko Tested-by: Guilherme G. Piccoli --- arch/x86/kernel/smpboot.c | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index a9e48946fc89..69b56c597949 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -57,6 +57,7 @@ #include #include #include +#include #include #include @@ -992,7 +993,8 @@ static void announce_cpu(int cpu, int apicid) node_width = num_digits(num_possible_nodes()) + 1; /* + '#' */ if (cpu == 1) - printk(KERN_INFO "x86: Booting SMP configuration:\n"); + printk(KERN_INFO "x86: Booting SMP configuration in %s:\n", + do_parallel_bringup ? "parallel" : "series"); if (system_state < SYSTEM_RUNNING) { if (node != current_node) { @@ -1325,9 +1327,12 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) { int ret; - ret = do_cpu_up(cpu, tidle); - if (ret) - goto out; + /* If parallel AP bringup isn't enabled, perform the first steps now. */ + if (!do_parallel_bringup) { + ret = do_cpu_up(cpu, tidle); + if (ret) + goto out; + } ret = do_wait_cpu_initialized(cpu); if (ret) @@ -1347,6 +1352,12 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) return ret; } +/* Bringup step one: Send INIT/SIPI to the target AP */ +static int native_cpu_kick(unsigned int cpu) +{ + return do_cpu_up(cpu, idle_thread_get(cpu, true)); +} + /** * arch_disable_smp_support() - disables SMP support for x86 at runtime */ @@ -1515,6 +1526,8 @@ static bool prepare_parallel_bringup(void) smpboot_control = STARTUP_APICID_CPUID_01; } + cpuhp_setup_state_nocalls(CPUHP_BP_PARALLEL_DYN, "x86/cpu:kick", + native_cpu_kick, NULL); return true; } From patchwork Thu Mar 16 22:21:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 70996 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp25799wrt; Thu, 16 Mar 2023 15:23:45 -0700 (PDT) X-Google-Smtp-Source: AK7set9PaIk6mLvLZO/1QItHMWjMp5Fj3APMyu3sz5p637LvKo8YRAavhovqaFO6IGndqcU28heK X-Received: by 2002:a17:902:d4ce:b0:1a1:8860:70d7 with SMTP id o14-20020a170902d4ce00b001a1886070d7mr6531746plg.48.1679005425101; Thu, 16 Mar 2023 15:23:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679005425; cv=none; d=google.com; s=arc-20160816; b=DXGark1/64wfZPsr1ekU/naq0B3RBgBlozQ+fxDt+tmuwTiLb7ramu2PslYVuZDFua b2N18ON1elecuW17aLWFUXCv4Q9rpK9sJBXQgfBXKBVEY2xsTrciKVG62iGy0SqiEr9F C8K+uSPoRio0mC5DZESFfIaOgqBiKjhebPxFWkgcTwof5Eo3DoXC2xtFhqauIfIgb2Dt tARD4K5+4seSYpTFZWJT2T0RbBGHcfzmpqp9iVz6Apfxtprn48Qs0KfMf7qY7E7lIBiv 2Z/f8HF9oxYO6JL8jOxF4fbKS7SjaOtdwSWJC5jMWmjUgezCH88s4G/LoRSRCc24KciO lxkg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=abU/HviepGZu6336k1MSiJADTLApJ+GQsge41I7AWUg=; b=ixnlxHWDGXGGJHwXZ8uBftyHZuQLYopW9VerVsmPv1QW8ilvbRVUgmXrhPL9/p3li6 IAwAc6KXOgmGri/9icL9NqulKY+QaI7w8ea3k3UkE5xKwwZLuLyYkZLcWKIf+/Pf8N3J NH1dYauLbWslzsT6tUnE4tuX9w0iO+XrkDgQm+QrOhKJxPWEIKOKfKYhyFs1jJvaVOhS 1ZjCLqJuz7LZ7diZ9FrjDB3MfnbXr1h8KN2Jjjgjf50EwVOxf95NLLBAy6IORTktbEVG J1DaAwWCUq6M/i14PDOOMdkFB2/QI/Y+BFVJHSekY0uCsSbDDhLnkj6mvt3ml2MZiNpw h8iQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=QXlA+Dod; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h10-20020a170902f54a00b0019f239f4966si608085plf.568.2023.03.16.15.23.30; Thu, 16 Mar 2023 15:23:45 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=QXlA+Dod; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230125AbjCPWWi (ORCPT + 99 others); Thu, 16 Mar 2023 18:22:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33284 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230299AbjCPWWL (ORCPT ); Thu, 16 Mar 2023 18:22:11 -0400 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 12122B719C for ; Thu, 16 Mar 2023 15:21:23 -0700 (PDT) Received: by mail-wm1-x32e.google.com with SMTP id j42-20020a05600c1c2a00b003ed363619ddso2576125wms.1 for ; Thu, 16 Mar 2023 15:21:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1679005282; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=abU/HviepGZu6336k1MSiJADTLApJ+GQsge41I7AWUg=; b=QXlA+DodXTz82wK9foe4XP1PtTD5CMXj0hFHQEpnOFZGQfFt1o6qSOGXFp3n0uzm3v ZCrQDPlwEzX9wfhQOGpVrZ7U51gESJBoJbP5J3BxkyKPpMezeJosSfsOD88vc5PML5F9 FwQHOlIJQRt9Lu54qnbCkHIkdiBqw+zF3WMTFXchJ/K7yoSUiyWeBr98gh3j4dgoufoT uZz7KUJH3zqPlfKgwxwE2H59iGJIw77vUi7kRJpXY+Qe0cdjpd4W6zm/ejWPlNmRejhd 3sEsYx4YxawLzDiA2bv/2ZR0A+KaUY7FShCZOPNRgHcTASpwO5pCTqZQC7jkyB+p/2J4 F0ug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679005282; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=abU/HviepGZu6336k1MSiJADTLApJ+GQsge41I7AWUg=; b=gLqxfSLWM8ROQiLD+TEPhcy4QSjblX8YmNi+6VrBI6Q27HBJfZ5X/x3xdBXQjSochR HWKlLTMlaMf4KyouuKHKKUN486sp7XWBKUUrrKX63gC9GeUcW9RMv/Ie9aQht9LEzEQI Y0wa1LwTQB/h/FMg4HPy9JrNGN3mTOO4Sc+y5iiauGgcNWotN+1H6eeBAhc2WjtyFFWj k+2GIQgsXkEuOFIxaMkdWPAY+tyaYHCIvdCmgLHlp4MEkQIrSTNb9nPE8x2KasXEKXh5 pOs9L8IW2pWn/VNWTISO+EoRPBdWyRzEf00xRm7vcVwxrNlVRr6V+REdbhhN/fOe2P2d 2KTg== X-Gm-Message-State: AO0yUKX4l2mhEYPtup7s8lTje90GQipA3uVI6dSR+OWV/L8HZwJeRQ0N jd529yAmSW2jCRLX3XiXfjGJ4A== X-Received: by 2002:a05:600c:4e87:b0:3eb:38a2:2bcd with SMTP id f7-20020a05600c4e8700b003eb38a22bcdmr22744902wmq.28.1679005282469; Thu, 16 Mar 2023 15:21:22 -0700 (PDT) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:4b87:78c3:3abe:7b0d]) by smtp.gmail.com with ESMTPSA id f9-20020adff989000000b002cea392f000sm439256wrr.69.2023.03.16.15.21.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 16 Mar 2023 15:21:21 -0700 (PDT) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, gpiccoli@igalia.com, David Woodhouse , Usama Arif Subject: [PATCH v15 11/12] x86/smpboot: Serialize topology updates for secondary bringup Date: Thu, 16 Mar 2023 22:21:08 +0000 Message-Id: <20230316222109.1940300-12-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230316222109.1940300-1-usama.arif@bytedance.com> References: <20230316222109.1940300-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760564792192306201?= X-GMAIL-MSGID: =?utf-8?q?1760564792192306201?= From: David Woodhouse The toplogy update is performed by the AP via smp_callin() after the BSP has called do_wait_cpu_initialized(), setting the AP's bit in cpu_callout_mask to allow it to proceed. In preparation to enable further parallelism of AP bringup, add locking to serialize the update even if multiple APs are (in future) permitted to proceed through the next stages of bringup in parallel. Without such ordering (and with that future extra parallelism), confusion ensues: [ 1.360149] x86: Booting SMP configuration: [ 1.360221] .... node #0, CPUs: #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15 #16 #17 #18 #19 #20 #21 #22 #23 [ 1.366225] .... node #1, CPUs: #24 #25 #26 #27 #28 #29 #30 #31 #32 #33 #34 #35 #36 #37 #38 #39 #40 #41 #42 #43 #44 #45 #46 #47 [ 1.370219] .... node #0, CPUs: #48 #49 #50 #51 #52 #53 #54 #55 #56 #57 #58 #59 #60 #61 #62 #63 #64 #65 #66 #67 #68 #69 #70 #71 [ 1.378226] .... node #1, CPUs: #72 #73 #74 #75 #76 #77 #78 #79 #80 #81 #82 #83 #84 #85 #86 #87 #88 #89 #90 #91 #92 #93 #94 #95 [ 1.382037] Brought 96 CPUs to x86/cpu:kick in 72232606 cycles [ 0.104104] smpboot: CPU 26 Converting physical 0 to logical die 1 [ 0.104104] smpboot: CPU 27 Converting physical 1 to logical package 2 [ 0.104104] smpboot: CPU 24 Converting physical 1 to logical package 3 [ 0.104104] smpboot: CPU 27 Converting physical 0 to logical die 2 [ 0.104104] smpboot: CPU 25 Converting physical 1 to logical package 4 [ 1.385609] Brought 96 CPUs to x86/cpu:wait-init in 9269218 cycles [ 1.395285] Brought CPUs online in 28930764 cycles [ 1.395469] smp: Brought up 2 nodes, 96 CPUs [ 1.395689] smpboot: Max logical packages: 2 [ 1.396222] smpboot: Total of 96 processors activated (576000.00 BogoMIPS) Signed-off-by: David Woodhouse Signed-off-by: Usama Arif Tested-by: Paul E. McKenney Tested-by: Kim Phillips Tested-by: Oleksandr Natalenko Tested-by: Guilherme G. Piccoli --- arch/x86/include/asm/smp.h | 4 +- arch/x86/include/asm/topology.h | 2 - arch/x86/kernel/cpu/common.c | 6 +-- arch/x86/kernel/smpboot.c | 73 ++++++++++++++++++++------------- arch/x86/xen/smp_pv.c | 4 +- 5 files changed, 48 insertions(+), 41 deletions(-) diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h index 1cf4f1e57570..defe76ee9e64 100644 --- a/arch/x86/include/asm/smp.h +++ b/arch/x86/include/asm/smp.h @@ -48,8 +48,6 @@ struct smp_ops { }; /* Globals due to paravirt */ -extern void set_cpu_sibling_map(int cpu); - #ifdef CONFIG_SMP extern struct smp_ops smp_ops; @@ -137,7 +135,7 @@ void native_send_call_func_single_ipi(int cpu); void x86_idle_thread_init(unsigned int cpu, struct task_struct *idle); void smp_store_boot_cpu_info(void); -void smp_store_cpu_info(int id); +void smp_store_cpu_info(int id, bool force_single_core); asmlinkage __visible void smp_reboot_interrupt(void); __visible void smp_reschedule_interrupt(struct pt_regs *regs); diff --git a/arch/x86/include/asm/topology.h b/arch/x86/include/asm/topology.h index 458c891a8273..4bccbd949a99 100644 --- a/arch/x86/include/asm/topology.h +++ b/arch/x86/include/asm/topology.h @@ -136,8 +136,6 @@ static inline int topology_max_smt_threads(void) return __max_smt_threads; } -int topology_update_package_map(unsigned int apicid, unsigned int cpu); -int topology_update_die_map(unsigned int dieid, unsigned int cpu); int topology_phys_to_logical_pkg(unsigned int pkg); int topology_phys_to_logical_die(unsigned int die, unsigned int cpu); bool topology_is_primary_thread(unsigned int cpu); diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index f3cc7699e1e1..06d7f9e55d45 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -1771,7 +1771,7 @@ static void generic_identify(struct cpuinfo_x86 *c) * Validate that ACPI/mptables have the same information about the * effective APIC id and update the package map. */ -static void validate_apic_and_package_id(struct cpuinfo_x86 *c) +static void validate_apic_id(struct cpuinfo_x86 *c) { #ifdef CONFIG_SMP unsigned int apicid, cpu = smp_processor_id(); @@ -1782,8 +1782,6 @@ static void validate_apic_and_package_id(struct cpuinfo_x86 *c) pr_err(FW_BUG "CPU%u: APIC id mismatch. Firmware: %x APIC: %x\n", cpu, apicid, c->initial_apicid); } - BUG_ON(topology_update_package_map(c->phys_proc_id, cpu)); - BUG_ON(topology_update_die_map(c->cpu_die_id, cpu)); #else c->logical_proc_id = 0; #endif @@ -1974,7 +1972,7 @@ void identify_secondary_cpu(struct cpuinfo_x86 *c) #ifdef CONFIG_X86_32 enable_sep_cpu(); #endif - validate_apic_and_package_id(c); + validate_apic_id(c); x86_spec_ctrl_setup_ap(); update_srbds_msr(); diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 69b56c597949..0960fdec43c2 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -180,16 +180,12 @@ static void smp_callin(void) apic_ap_setup(); /* - * Save our processor parameters. Note: this information - * is needed for clock calibration. - */ - smp_store_cpu_info(cpuid); - - /* + * Save our processor parameters and update topology. + * Note: this information is needed for clock calibration. * The topology information must be up to date before * calibrate_delay() and notify_cpu_starting(). */ - set_cpu_sibling_map(raw_smp_processor_id()); + smp_store_cpu_info(cpuid, false); ap_init_aperfmperf(); @@ -243,6 +239,12 @@ static void notrace start_secondary(void *unused) * its bit in cpu_callout_mask to release it. */ cpu_init_secondary(); + + /* + * Even though notify_cpu_starting() will do this, it does so too late + * as the AP may already have triggered lockdep splats by then. See + * commit 29368e093 ("x86/smpboot: Move rcu_cpu_starting() earlier"). + */ rcu_cpu_starting(raw_smp_processor_id()); x86_cpuinit.early_percpu_clock_init(); @@ -351,7 +353,7 @@ EXPORT_SYMBOL(topology_phys_to_logical_die); * @pkg: The physical package id as retrieved via CPUID * @cpu: The cpu for which this is updated */ -int topology_update_package_map(unsigned int pkg, unsigned int cpu) +static int topology_update_package_map(unsigned int pkg, unsigned int cpu) { int new; @@ -374,7 +376,7 @@ int topology_update_package_map(unsigned int pkg, unsigned int cpu) * @die: The die id as retrieved via CPUID * @cpu: The cpu for which this is updated */ -int topology_update_die_map(unsigned int die, unsigned int cpu) +static int topology_update_die_map(unsigned int die, unsigned int cpu) { int new; @@ -405,25 +407,7 @@ void __init smp_store_boot_cpu_info(void) c->initialized = true; } -/* - * The bootstrap kernel entry code has set these up. Save them for - * a given CPU - */ -void smp_store_cpu_info(int id) -{ - struct cpuinfo_x86 *c = &cpu_data(id); - - /* Copy boot_cpu_data only on the first bringup */ - if (!c->initialized) - *c = boot_cpu_data; - c->cpu_index = id; - /* - * During boot time, CPU0 has this setup already. Save the info when - * bringing up AP or offlined CPU0. - */ - identify_secondary_cpu(c); - c->initialized = true; -} +static arch_spinlock_t topology_lock = __ARCH_SPIN_LOCK_UNLOCKED; static bool topology_same_node(struct cpuinfo_x86 *c, struct cpuinfo_x86 *o) @@ -629,7 +613,7 @@ static struct sched_domain_topology_level x86_topology[] = { */ static bool x86_has_numa_in_package; -void set_cpu_sibling_map(int cpu) +static void set_cpu_sibling_map(int cpu) { bool has_smt = smp_num_siblings > 1; bool has_mp = has_smt || boot_cpu_data.x86_max_cores > 1; @@ -708,6 +692,37 @@ void set_cpu_sibling_map(int cpu) } } +/* + * The bootstrap kernel entry code has set these up. Save them for + * a given CPU + */ +void smp_store_cpu_info(int id, bool force_single_core) +{ + struct cpuinfo_x86 *c = &cpu_data(id); + + /* Copy boot_cpu_data only on the first bringup */ + if (!c->initialized) + *c = boot_cpu_data; + c->cpu_index = id; + /* + * During boot time, CPU0 has this setup already. Save the info when + * bringing up AP or offlined CPU0. + */ + identify_secondary_cpu(c); + + arch_spin_lock(&topology_lock); + BUG_ON(topology_update_package_map(c->phys_proc_id, id)); + BUG_ON(topology_update_die_map(c->cpu_die_id, id)); + c->initialized = true; + + /* For Xen PV */ + if (force_single_core) + c->x86_max_cores = 1; + + set_cpu_sibling_map(id); + arch_spin_unlock(&topology_lock); +} + /* maps the cpu to the sched domain representing multi-core */ const struct cpumask *cpu_coregroup_mask(int cpu) { diff --git a/arch/x86/xen/smp_pv.c b/arch/x86/xen/smp_pv.c index 6175f2c5c822..09f94f940689 100644 --- a/arch/x86/xen/smp_pv.c +++ b/arch/x86/xen/smp_pv.c @@ -71,9 +71,7 @@ static void cpu_bringup(void) xen_enable_syscall(); } cpu = smp_processor_id(); - smp_store_cpu_info(cpu); - cpu_data(cpu).x86_max_cores = 1; - set_cpu_sibling_map(cpu); + smp_store_cpu_info(cpu, true); speculative_store_bypass_ht_init(); From patchwork Thu Mar 16 22:21:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 70993 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp25768wrt; Thu, 16 Mar 2023 15:23:40 -0700 (PDT) X-Google-Smtp-Source: AK7set+ZorC7OAG/e/yBTr+BNmBp3uweD3XjgXMspLdAtVugkR1OGBdq36BiBGZlQufStoDqDDZi X-Received: by 2002:a17:903:2342:b0:19c:be03:d0ba with SMTP id c2-20020a170903234200b0019cbe03d0bamr5575647plh.18.1679005419822; Thu, 16 Mar 2023 15:23:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679005419; cv=none; d=google.com; s=arc-20160816; b=VBzIN9qn9AH7N36DUKqeoAcp62bNknpqTenj0k3Ce3+y/UqIzRo9Jg0DI1LMoLEaqg YIhFtNOUqo/+6VvADmegoylycTyDpJdN109UX596zu4xZYSNMBC/I4otQgGcUuGLqyfd qfpyqPx3NDZ2SsQzK1ACwI800169v8zqLYZUQcRgQ3k/AV4syQ5Yr0akpW9KF3LCN5QD RxQG/ojNgLTLY3P5TjNyfAwUxaSr/DDfpxQjNIogDOl9Qgbxhkbq1A0xY+rIbMGrGbH/ sIk+HGl0PWcpxil6W/jjyBlF0bPOgL4B4LYgaJ2xgkSBJ5n/lmYEQFxGBpT89OMvW//q ALyA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=/pc8Fr19lZeoT7q7aaMJd1wTCs0Rtv3VmfL+mjQqn4Q=; b=oTQq26yDNbRIuNXGm0OJZGR2j33y3tzcxDAemNq2hCgLXxKOXklopN9nliEcnspw8x t9GOGOGdwzZKWTSukehRchx3YwPLAIVuwtDSLZHzmBxIhUfxlp/6PDV4Q3IHKNcQduP2 vQPt00wTYMHpuF19lhsMweQZXNQdPkk96wf/SducJXKSdWCIb+JEq84nDit5kgUVtv02 CtLtfjP5eRS5Remg2Hakco7iv95bMXqULPzsTcRd/tHEtQ6NaNrix7/fT7IZXzKLPLXA AQlNY9GbvlKCK261Pew+n6dpS3Eg1UlZnI9PwDz3wgYHS87s2frozO6QjN3PhMaMSm/H XG6w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b="F/pB0cmP"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t19-20020a170902e1d300b0019e8c3cd415si518587pla.260.2023.03.16.15.23.25; Thu, 16 Mar 2023 15:23:39 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b="F/pB0cmP"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229516AbjCPWW1 (ORCPT + 99 others); Thu, 16 Mar 2023 18:22:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60684 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230289AbjCPWWK (ORCPT ); Thu, 16 Mar 2023 18:22:10 -0400 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EBE1EB7182 for ; Thu, 16 Mar 2023 15:21:23 -0700 (PDT) Received: by mail-wr1-x42f.google.com with SMTP id h17so2863012wrt.8 for ; Thu, 16 Mar 2023 15:21:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1679005283; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/pc8Fr19lZeoT7q7aaMJd1wTCs0Rtv3VmfL+mjQqn4Q=; b=F/pB0cmPjqelnxt+mi/Zlr5KL9GUCOiNpfTq70y8JbScgxcD8gwHy50I3vtgFZCfDc bWO6LjVQr8TzFMPsstU3ti6ksZNXSJQyoaapyKVEWD5xn8LUt8rYmn/B2Lnp2BMvBWBk SmiIDi1Na/x6DGh2AIhenDkOCjWxESZD6aD1SLnaufjCgEYPrd1rXLRczw/yXnecdoA3 81G7GHTCd/lpSWzu7hS3/EGzq7vRCYHbBRLE8o3jqxMfTPavhji+rCY2P/UDGBRzRHxM nOWkJt6XtT2rwVF7p665nvPeFaiKbsigW6DAoRj0R8nMiq+xhD/0HXfpGn1pz9MTOcu/ 1jSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679005283; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/pc8Fr19lZeoT7q7aaMJd1wTCs0Rtv3VmfL+mjQqn4Q=; b=UDt9UzXevy0WQ2oTvHP22niULzrZN/r/SGxrlK1nh7Nwii8CVy9+89B/KpYXCygrrK qHKe23MmahfYx/gCJ4ZeA3Dy5WQ9VkpI/ScnJ4Z4resi5jgxktuMMcX5vTdqeJEU4ecI 2VUprUjvCOrtO0xznss9eDhMIyhkZiSciEKo7G3gn0OVFBh4Q5MR50cJ7ESL6uKU9llY mdCAbZzF0AJ7rDLhuas8nLdUFeoUIEvtwWgdoPSaog2gt1b/fP73Ddc4fMU0inRt9DiT dkhyJ7vEuDdIPNXQR7ESywc35vN9GS1TFhvSQB9f1uqvLJjSfaokyUb65tDi5UsxeXd8 snZA== X-Gm-Message-State: AO0yUKX7gfflxOuet0jxY3wvyOk9hEmZ6y0GUKUH/+z4yd+jO0ByCd05 xSiHmssk1ltQSTanAwWsBoHUGw== X-Received: by 2002:adf:dc0f:0:b0:2cf:f0c3:79ba with SMTP id t15-20020adfdc0f000000b002cff0c379bamr5429379wri.67.1679005283474; Thu, 16 Mar 2023 15:21:23 -0700 (PDT) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:4b87:78c3:3abe:7b0d]) by smtp.gmail.com with ESMTPSA id f9-20020adff989000000b002cea392f000sm439256wrr.69.2023.03.16.15.21.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 16 Mar 2023 15:21:23 -0700 (PDT) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, gpiccoli@igalia.com, David Woodhouse , Sabin Rapan , Usama Arif Subject: [PATCH v15 12/12] x86/smpboot: Allow parallel bringup for SEV-ES Date: Thu, 16 Mar 2023 22:21:09 +0000 Message-Id: <20230316222109.1940300-13-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230316222109.1940300-1-usama.arif@bytedance.com> References: <20230316222109.1940300-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760564787138529665?= X-GMAIL-MSGID: =?utf-8?q?1760564787138529665?= From: David Woodhouse Enable parallel bringup for SEV-ES guests. The APs can't actually execute the CPUID instruction directly during early startup, but they can make the GHCB call directly instead, just as the VC trap handler would do. Thanks to Sabin for talking me through the way this works. Suggested-by: Sabin Rapan Signed-off-by: David Woodhouse Signed-off-by: Usama Arif Reviewed-by: Tom Lendacky --- arch/x86/coco/core.c | 5 ++++ arch/x86/include/asm/coco.h | 1 + arch/x86/include/asm/sev-common.h | 3 +++ arch/x86/include/asm/smp.h | 5 +++- arch/x86/kernel/head_64.S | 30 ++++++++++++++++++++++++ arch/x86/kernel/smpboot.c | 39 ++++++++++++++++++++++++++----- 6 files changed, 76 insertions(+), 7 deletions(-) diff --git a/arch/x86/coco/core.c b/arch/x86/coco/core.c index 49b44f881484..0bab38efb15a 100644 --- a/arch/x86/coco/core.c +++ b/arch/x86/coco/core.c @@ -129,6 +129,11 @@ u64 cc_mkdec(u64 val) } EXPORT_SYMBOL_GPL(cc_mkdec); +enum cc_vendor cc_get_vendor(void) +{ + return vendor; +} + __init void cc_set_vendor(enum cc_vendor v) { vendor = v; diff --git a/arch/x86/include/asm/coco.h b/arch/x86/include/asm/coco.h index 3d98c3a60d34..0428d9712c96 100644 --- a/arch/x86/include/asm/coco.h +++ b/arch/x86/include/asm/coco.h @@ -12,6 +12,7 @@ enum cc_vendor { }; void cc_set_vendor(enum cc_vendor v); +enum cc_vendor cc_get_vendor(void); void cc_set_mask(u64 mask); #ifdef CONFIG_ARCH_HAS_CC_PLATFORM diff --git a/arch/x86/include/asm/sev-common.h b/arch/x86/include/asm/sev-common.h index b8357d6ecd47..f25df4bd318e 100644 --- a/arch/x86/include/asm/sev-common.h +++ b/arch/x86/include/asm/sev-common.h @@ -70,6 +70,7 @@ /* GHCBData[63:12] */ \ (((u64)(v) & GENMASK_ULL(63, 12)) >> 12) +#ifndef __ASSEMBLY__ /* * SNP Page State Change Operation * @@ -160,6 +161,8 @@ struct snp_psc_desc { #define GHCB_RESP_CODE(v) ((v) & GHCB_MSR_INFO_MASK) +#endif /* __ASSEMBLY__ */ + /* * Error codes related to GHCB input that can be communicated back to the guest * by setting the lower 32-bits of the GHCB SW_EXITINFO1 field to 2. diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h index defe76ee9e64..1584f04a7007 100644 --- a/arch/x86/include/asm/smp.h +++ b/arch/x86/include/asm/smp.h @@ -204,7 +204,10 @@ extern unsigned int smpboot_control; /* Control bits for startup_64 */ #define STARTUP_APICID_CPUID_0B 0x80000000 #define STARTUP_APICID_CPUID_01 0x40000000 +#define STARTUP_APICID_SEV_ES 0x20000000 -#define STARTUP_PARALLEL_MASK (STARTUP_APICID_CPUID_01 | STARTUP_APICID_CPUID_0B) +#define STARTUP_PARALLEL_MASK (STARTUP_APICID_CPUID_01 | \ + STARTUP_APICID_CPUID_0B | \ + STARTUP_APICID_SEV_ES) #endif /* _ASM_X86_SMP_H */ diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index 65bca47d84a1..79a17e705f4e 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -26,6 +26,7 @@ #include #include #include +#include /* * We are not able to switch in one step to the final KERNEL ADDRESS SPACE @@ -242,6 +243,7 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) * * Bit 31 STARTUP_APICID_CPUID_0B flag (use CPUID 0x0b) * Bit 30 STARTUP_APICID_CPUID_01 flag (use CPUID 0x01) + * Bit 29 STARTUP_APICID_SEV_ES flag (CPUID 0x0b via GHCB MSR) * Bit 0-24 CPU# if STARTUP_APICID_CPUID_xx flags are not set */ movl smpboot_control(%rip), %ecx @@ -249,6 +251,10 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) jnz .Luse_cpuid_0b testl $STARTUP_APICID_CPUID_01, %ecx jnz .Luse_cpuid_01 +#ifdef CONFIG_AMD_MEM_ENCRYPT + testl $STARTUP_APICID_SEV_ES, %ecx + jnz .Luse_sev_cpuid_0b +#endif andl $0x0FFFFFFF, %ecx jmp .Lsetup_cpu @@ -259,6 +265,30 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) shr $24, %edx jmp .Lsetup_AP +#ifdef CONFIG_AMD_MEM_ENCRYPT +.Luse_sev_cpuid_0b: + /* Set the GHCB MSR to request CPUID 0xB_EDX */ + movl $MSR_AMD64_SEV_ES_GHCB, %ecx + movl $(GHCB_CPUID_REQ_EDX << 30) | GHCB_MSR_CPUID_REQ, %eax + movl $0x0B, %edx + wrmsr + + /* Perform GHCB MSR protocol */ + rep; vmmcall /* vmgexit */ + + /* + * Get the result. After the RDMSR: + * EAX should be 0xc0000005 + * EDX should have the CPUID register value and since EDX + * is the target register, no need to move the result. + */ + rdmsr + andl $GHCB_MSR_INFO_MASK, %eax + cmpl $GHCB_MSR_CPUID_RESP, %eax + jne 1f + jmp .Lsetup_AP +#endif + .Luse_cpuid_0b: mov $0x0B, %eax xorl %ecx, %ecx diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 0960fdec43c2..895395787afc 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -85,6 +85,7 @@ #include #include #include +#include /* representing HT siblings of each logical CPU */ DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_sibling_map); @@ -1513,15 +1514,36 @@ void __init smp_prepare_cpus_common(void) * We can do 64-bit AP bringup in parallel if the CPU reports its APIC * ID in CPUID (either leaf 0x0B if we need the full APIC ID in X2APIC * mode, or leaf 0x01 if 8 bits are sufficient). Otherwise it's too - * hard. And not for SEV-ES guests because they can't use CPUID that - * early. + * hard. */ static bool prepare_parallel_bringup(void) { - if (IS_ENABLED(CONFIG_X86_32) || cc_platform_has(CC_ATTR_GUEST_STATE_ENCRYPT)) + bool has_sev_es = false; + + if (IS_ENABLED(CONFIG_X86_32)) return false; - if (x2apic_mode) { + /* + * Encrypted guests other than SEV-ES (in the future) will need to + * implement an early way of finding the APIC ID, since they will + * presumably block direct CPUID too. Be kind to our future selves + * by warning here instead of just letting them break. Parallel + * startup doesn't have to be in the first round of enabling patches + * for any such technology. + */ + if (cc_platform_has(CC_ATTR_GUEST_STATE_ENCRYPT)) { + switch (cc_get_vendor()) { + case CC_VENDOR_AMD: + has_sev_es = true; + break; + + default: + pr_info("Disabling parallel bringup due to guest state encryption\n"); + return false; + } + } + + if (x2apic_mode || has_sev_es) { if (boot_cpu_data.cpuid_level < 0x0b) return false; @@ -1530,8 +1552,13 @@ static bool prepare_parallel_bringup(void) return false; } - pr_debug("Using CPUID 0xb for parallel CPU startup\n"); - smpboot_control = STARTUP_APICID_CPUID_0B; + if (has_sev_es) { + pr_debug("Using SEV-ES CPUID 0xb for parallel CPU startup\n"); + smpboot_control = STARTUP_APICID_SEV_ES; + } else { + pr_debug("Using CPUID 0xb for parallel CPU startup\n"); + smpboot_control = STARTUP_APICID_CPUID_0B; + } } else { /* Without X2APIC, what's in CPUID 0x01 should suffice. */ if (boot_cpu_data.cpuid_level < 0x01)