From patchwork Thu Feb 23 19:11:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 61056 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp484449wrd; Thu, 23 Feb 2023 11:13:32 -0800 (PST) X-Google-Smtp-Source: AK7set9rH1xoAk5leIyPnv2kbz7gLJriwafw4jzWnW7UwTsknPeZ93U8o203SVAS30kCp52FKWkh X-Received: by 2002:a17:906:196:b0:8c5:3625:ffc5 with SMTP id 22-20020a170906019600b008c53625ffc5mr17669027ejb.7.1677179611941; Thu, 23 Feb 2023 11:13:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677179611; cv=none; d=google.com; s=arc-20160816; b=JYDD5QIaY1rEZsadwXbOpspL5PThqqyDU67HgJKlrQsmkUFbmrW2VHqdrPJz+felb0 P/YV/1PlL/EZ3x65ISkej3RW7xUHMP9p2/MfCLVIL/qGB9lyD0wPHDaBPnxT05G9zoLi PBzbPGcowfpe0QZ/gHV9LvIs0XUVHcjgX2MwG3t8uwfHsWbvaXqTjPlAwx9hPqpEfvEH +NEKIjrf8E57CvL2v0H2tEkJGg5wB6hxI25G0A2jMecbrquwqfYc73+j/uiWUdgq9/v2 sxkoA9Hcyw9HNyY5PDI2LYQDM7bl+abLFgdmI1F1v0kQdPaKpUMkIPVEpCZiuRkdxqbQ 4jPw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=8SFcyAg12RLJRTJdp4mtixGxiIqYpnMS3znV8hvpPts=; b=GX9Ch3k+sbwkOorYDCHYKliFOiqgfNOP0PoeoaJnfgb2eVOu5910z1VSaEtHZ69ykC Z7jIlpw/0IjvEdOwYUVpdJJv+Dvz71w5jtmnlg9BXAu9mGMgWXXmmX8DpG+dmOmyLA46 442DECXDmqnDP/Hoif6o+t1s7qjoOczZQcslrZo+fkLUxU7U+k0nDni6VQet6SHvJ/SF jDUk3Q9OPmSpdX+QZnNqlB7snapjlFh1teIa28vFuRgSkcAGKcuFUu91mzdTBfTN4VoN +BOgj/IL9MVrOc3XBLCJrIml83tN8EgUeS/v3J1YRTlSWDtYR+CBbSbCCsi46oLFNyyj TCZg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b="FR/5U6b0"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id vs5-20020a170907138500b008b17a861395si23731320ejb.905.2023.02.23.11.13.08; Thu, 23 Feb 2023 11:13:31 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b="FR/5U6b0"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231871AbjBWTMg (ORCPT + 99 others); Thu, 23 Feb 2023 14:12:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60008 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231836AbjBWTMa (ORCPT ); Thu, 23 Feb 2023 14:12:30 -0500 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DABE95457E for ; Thu, 23 Feb 2023 11:12:04 -0800 (PST) Received: by mail-wr1-x430.google.com with SMTP id bw19so928173wrb.13 for ; Thu, 23 Feb 2023 11:12:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8SFcyAg12RLJRTJdp4mtixGxiIqYpnMS3znV8hvpPts=; b=FR/5U6b0lIE7/j0QhdY8FJ3gh1ZCMnFiksYDWQDIygBy44Q+xclp3ilXgQDFvVC5NS fldEL2mL4p90CJrR64KBR74+hK7+enriGwh5cMZQD3YpNXT00rBh7nkyIIqDEuQmQ3ol g2vfbVIz59GT6D8Ikd5T2++kPrbc31Bw7cse9/rlr/Iyl2Zen17rMZ3ZlgLU8eY63ZBF bmyNLk+UxKujTUSvlCBUFuDYiHMS++NBvBTrpyaGvxSCJuQAJzHrpEJbJQdZ9ZOvunj7 qREWQt6lvoCbPu58O08h64QO5RdEYxrTEnX+0GNsZE0r0qVbVZF8IipcCAqq2k0IaoJa 6SNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8SFcyAg12RLJRTJdp4mtixGxiIqYpnMS3znV8hvpPts=; b=Q6sfQSrmrxMqcufAfl5AKCJgsIZLAzaaboYYfL3++CKmZ0CK0fa4nsp/mXFWMiUqWP PRZSilEHXLWD151UOg0tyRUeqTBxX5WF7ARF3POGkvXiNDFF4HTjS2+wB5PSTCARWIrb RgiRgk//mtLJTuYFNMotnak//HvRotoafXa7pHIYyoUdRL+lzrALPTjEhaR74cn+biA9 qS8jNyqKNekWMnlF9p8eQ9P8tzCJi6WbhRoea91id8aRbgRqwJ2H/KW6CQkBPG0JOkgv t5x7ijmzh7r9/lBZdnHDdytdFUPYscnW0Jsjh1aLTLSxF2/JqKsWi2dIedatQLqZDt/7 1xxw== X-Gm-Message-State: AO0yUKW2rs54dHwDBj8z/cSrXtXl//Kl8RfYHMilbcbE4CulHJ7tuQgk HDTAH40fqcnKA+Rm1Ta05rhZ6g== X-Received: by 2002:a5d:6090:0:b0:2c5:9990:288e with SMTP id w16-20020a5d6090000000b002c59990288emr10862999wrt.18.1677179504124; Thu, 23 Feb 2023 11:11:44 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:5ee0:5af0:64bd:6198]) by smtp.gmail.com with ESMTPSA id b15-20020a5d4b8f000000b002c561805a4csm12957286wrt.45.2023.02.23.11.11.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Feb 2023 11:11:43 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse , Usama Arif Subject: [PATCH v11 01/12] x86/apic/x2apic: Allow CPU cluster_mask to be populated in parallel Date: Thu, 23 Feb 2023 19:11:29 +0000 Message-Id: <20230223191140.4155012-2-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230223191140.4155012-1-usama.arif@bytedance.com> References: <20230223191140.4155012-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758650288482260617?= X-GMAIL-MSGID: =?utf-8?q?1758650288482260617?= From: David Woodhouse Each of the sibling CPUs in a cluster uses the same clustermask. The first CPU in a cluster will need a new clustermask allocated, while subsequent siblings will use the same clustermask as the first. However, the CPU being brought up cannot yet perform memory allocations at the point that this occurs in init_x2apic_ldr(). So at present, the alloc_clustermask() function allocates a clustermask just in case it's needed, storing it in the global cluster_hotplug_mask. A CPU which is the first sibling of a cluster will "take" it from there and set cluster_hotplug_mask to NULL, in order for alloc_clustermask() to allocate a new one before bringing up the next CPU. To facilitate parallel bringup of CPUs in future, switch to a model where alloc_clustermask() prepopulates the clustermask in the per_cpu data for each present CPU in the cluster in advance. All that the CPU needs to do for itself in init_x2apic_ldr() is set its own bit in that mask. The 'node' and 'clusterid' members of struct cluster_mask are thus redundant, and it can become a simple struct cpumask instead. Suggested-by: Thomas Gleixner Signed-off-by: David Woodhouse Signed-off-by: Usama Arif Tested-by: Paul E. McKenney Tested-by: Kim Phillips Tested-by: Oleksandr Natalenko --- arch/x86/kernel/apic/x2apic_cluster.c | 126 +++++++++++++++++--------- 1 file changed, 82 insertions(+), 44 deletions(-) diff --git a/arch/x86/kernel/apic/x2apic_cluster.c b/arch/x86/kernel/apic/x2apic_cluster.c index e696e22d0531..b2b2b7f3e03f 100644 --- a/arch/x86/kernel/apic/x2apic_cluster.c +++ b/arch/x86/kernel/apic/x2apic_cluster.c @@ -9,11 +9,7 @@ #include "local.h" -struct cluster_mask { - unsigned int clusterid; - int node; - struct cpumask mask; -}; +#define apic_cluster(apicid) ((apicid) >> 4) /* * __x2apic_send_IPI_mask() possibly needs to read @@ -23,8 +19,7 @@ struct cluster_mask { static u32 *x86_cpu_to_logical_apicid __read_mostly; static DEFINE_PER_CPU(cpumask_var_t, ipi_mask); -static DEFINE_PER_CPU_READ_MOSTLY(struct cluster_mask *, cluster_masks); -static struct cluster_mask *cluster_hotplug_mask; +static DEFINE_PER_CPU_READ_MOSTLY(struct cpumask *, cluster_masks); static int x2apic_acpi_madt_oem_check(char *oem_id, char *oem_table_id) { @@ -60,10 +55,10 @@ __x2apic_send_IPI_mask(const struct cpumask *mask, int vector, int apic_dest) /* Collapse cpus in a cluster so a single IPI per cluster is sent */ for_each_cpu(cpu, tmpmsk) { - struct cluster_mask *cmsk = per_cpu(cluster_masks, cpu); + struct cpumask *cmsk = per_cpu(cluster_masks, cpu); dest = 0; - for_each_cpu_and(clustercpu, tmpmsk, &cmsk->mask) + for_each_cpu_and(clustercpu, tmpmsk, cmsk) dest |= x86_cpu_to_logical_apicid[clustercpu]; if (!dest) @@ -71,7 +66,7 @@ __x2apic_send_IPI_mask(const struct cpumask *mask, int vector, int apic_dest) __x2apic_send_IPI_dest(dest, vector, APIC_DEST_LOGICAL); /* Remove cluster CPUs from tmpmask */ - cpumask_andnot(tmpmsk, tmpmsk, &cmsk->mask); + cpumask_andnot(tmpmsk, tmpmsk, cmsk); } local_irq_restore(flags); @@ -105,55 +100,98 @@ static u32 x2apic_calc_apicid(unsigned int cpu) static void init_x2apic_ldr(void) { - struct cluster_mask *cmsk = this_cpu_read(cluster_masks); - u32 cluster, apicid = apic_read(APIC_LDR); - unsigned int cpu; + struct cpumask *cmsk = this_cpu_read(cluster_masks); - x86_cpu_to_logical_apicid[smp_processor_id()] = apicid; + BUG_ON(!cmsk); - if (cmsk) - goto update; - - cluster = apicid >> 16; - for_each_online_cpu(cpu) { - cmsk = per_cpu(cluster_masks, cpu); - /* Matching cluster found. Link and update it. */ - if (cmsk && cmsk->clusterid == cluster) - goto update; + cpumask_set_cpu(smp_processor_id(), cmsk); +} + +/* + * As an optimisation during boot, set the cluster_mask for all present + * CPUs at once, to prevent each of them having to iterate over the others + * to find the existing cluster_mask. + */ +static void prefill_clustermask(struct cpumask *cmsk, unsigned int cpu, u32 cluster) +{ + int cpu_i; + + for_each_present_cpu(cpu_i) { + struct cpumask **cpu_cmsk = &per_cpu(cluster_masks, cpu_i); + u32 apicid = apic->cpu_present_to_apicid(cpu_i); + + if (apicid == BAD_APICID || cpu_i == cpu || apic_cluster(apicid) != cluster) + continue; + + if (WARN_ON_ONCE(*cpu_cmsk == cmsk)) + continue; + + BUG_ON(*cpu_cmsk); + *cpu_cmsk = cmsk; } - cmsk = cluster_hotplug_mask; - cmsk->clusterid = cluster; - cluster_hotplug_mask = NULL; -update: - this_cpu_write(cluster_masks, cmsk); - cpumask_set_cpu(smp_processor_id(), &cmsk->mask); } -static int alloc_clustermask(unsigned int cpu, int node) +static int alloc_clustermask(unsigned int cpu, u32 cluster, int node) { + struct cpumask *cmsk = NULL; + unsigned int cpu_i; + + /* + * At boot time, the CPU present mask is stable. The cluster mask is + * allocated for the first CPU in the cluster and propagated to all + * present siblings in the cluster. If the cluster mask is already set + * on entry to this function for a given CPU, there is nothing to do. + */ if (per_cpu(cluster_masks, cpu)) return 0; + + if (system_state < SYSTEM_RUNNING) + goto alloc; + /* - * If a hotplug spare mask exists, check whether it's on the right - * node. If not, free it and allocate a new one. + * On post boot hotplug for a CPU which was not present at boot time, + * iterate over all possible CPUs (even those which are not present + * any more) to find any existing cluster mask. */ - if (cluster_hotplug_mask) { - if (cluster_hotplug_mask->node == node) - return 0; - kfree(cluster_hotplug_mask); + for_each_possible_cpu(cpu_i) { + u32 apicid = apic->cpu_present_to_apicid(cpu_i); + + if (apicid != BAD_APICID && apic_cluster(apicid) == cluster) { + cmsk = per_cpu(cluster_masks, cpu_i); + /* + * If the cluster is already initialized, just store + * the mask and return. There's no need to propagate. + */ + if (cmsk) { + per_cpu(cluster_masks, cpu) = cmsk; + return 0; + } + } } - - cluster_hotplug_mask = kzalloc_node(sizeof(*cluster_hotplug_mask), - GFP_KERNEL, node); - if (!cluster_hotplug_mask) + /* + * No CPU in the cluster has ever been initialized, so fall through to + * the boot time code which will also populate the cluster mask for any + * other CPU in the cluster which is (now) present. + */ +alloc: + cmsk = kzalloc_node(sizeof(*cmsk), GFP_KERNEL, node); + if (!cmsk) return -ENOMEM; - cluster_hotplug_mask->node = node; + per_cpu(cluster_masks, cpu) = cmsk; + prefill_clustermask(cmsk, cpu, cluster); + return 0; } static int x2apic_prepare_cpu(unsigned int cpu) { - if (alloc_clustermask(cpu, cpu_to_node(cpu)) < 0) + u32 phys_apicid = apic->cpu_present_to_apicid(cpu); + u32 cluster = apic_cluster(phys_apicid); + u32 logical_apicid = (cluster << 16) | (1 << (phys_apicid & 0xf)); + + x86_cpu_to_logical_apicid[cpu] = logical_apicid; + + if (alloc_clustermask(cpu, cluster, cpu_to_node(cpu)) < 0) return -ENOMEM; if (!zalloc_cpumask_var(&per_cpu(ipi_mask, cpu), GFP_KERNEL)) return -ENOMEM; @@ -162,10 +200,10 @@ static int x2apic_prepare_cpu(unsigned int cpu) static int x2apic_dead_cpu(unsigned int dead_cpu) { - struct cluster_mask *cmsk = per_cpu(cluster_masks, dead_cpu); + struct cpumask *cmsk = per_cpu(cluster_masks, dead_cpu); if (cmsk) - cpumask_clear_cpu(dead_cpu, &cmsk->mask); + cpumask_clear_cpu(dead_cpu, cmsk); free_cpumask_var(per_cpu(ipi_mask, dead_cpu)); return 0; } From patchwork Thu Feb 23 19:11:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 61058 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp484474wrd; Thu, 23 Feb 2023 11:13:34 -0800 (PST) X-Google-Smtp-Source: AK7set96k8+srtUwzVGPX0psEBhNh0zQ5f2yKSYWZafKZmhQ2PWr5RdgDbjkUWJSXJ2oJCDHPN29 X-Received: by 2002:a05:6402:35c9:b0:4ad:9b2:5e70 with SMTP id z9-20020a05640235c900b004ad09b25e70mr14885523edc.1.1677179614254; Thu, 23 Feb 2023 11:13:34 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677179614; cv=none; d=google.com; s=arc-20160816; b=wWrgJQe62EYdRVxW7INa444adfRSqLhcYCRwQYTv8pUIKsafjNiAXc+7R+repRBn1H As1oH0yEeTmrbFIv4m1pQvSuw7NVM7SqKYJf6uFI+iKiNHXOIYUOsjN5AUss6cjOhiMb +fzxPMzLvDxk4SY/a84qKdsMTMYSuVQOKCjkdxnPi4invdXuH3DWnr0TpDdNG5mwC4Kr 4h0nPqtvcLINIlJOQ14zBoLBWiL1x7y3+Pnij0zbPbx1/4uAHfdanq+Q0LF/ZTdQ6cbO heAfXmzFwLTZ08DUj0crKIPJ8fJeRSND/AFFAtnXdLXtVHFhOJgpaYqUnNUPLFj3INYu IE2Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=7yne835l8M0Qgk6uaQXrJLgxrNefMQSePYhKpKknocM=; b=EQ5o9nNm+fk7pfJ/bCMw4r4Wn1yHjoQlJwOFm3zOf8RXMXiB2v9Zexn+dQwNFb+a/9 +tJq38+jhX7HPsAClwNkYgIzn06koF4UjLBU1FoLPbSS/mRLTfNgZj3LJbQZavB46Wa9 rJX1j45mRyIa1yR5vQ2N9PkwVqc4B6ykBEcOc+PNLzWuol5iPFfQ44tdFVGUeKxGVN2t 04iaxUBTf0wuItFw4DGd98k+Vg/kJe4UZMv7ZiA/8o1EwiGd1oqcvXgw8rQ9hIod9rJv YvTa+pAgA6srcncA+WJnKyBDH2KqZcykUt1TjzVLshTC811F+FJiUBEznjU54jd8jSll RdGg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=V7OoY+Xk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id v16-20020a056402185000b004acdef7baa9si15490500edy.522.2023.02.23.11.13.10; Thu, 23 Feb 2023 11:13:34 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=V7OoY+Xk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231991AbjBWTMj (ORCPT + 99 others); Thu, 23 Feb 2023 14:12:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60056 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231695AbjBWTMa (ORCPT ); Thu, 23 Feb 2023 14:12:30 -0500 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 97FFA56521 for ; Thu, 23 Feb 2023 11:12:06 -0800 (PST) Received: by mail-wr1-x42b.google.com with SMTP id p8so11689029wrt.12 for ; Thu, 23 Feb 2023 11:12:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7yne835l8M0Qgk6uaQXrJLgxrNefMQSePYhKpKknocM=; b=V7OoY+Xkb1vzkS5nj+lYutSsnw1LLzyIz6O92ioTe4bcq67o38+hPJ+XTpkDTM3bx+ pcuZkF8TIS5mtmNsNeKwBd/4jBIuSnQAu79D9emHf5V2xG4utkfDG478Jz9txH4BVwwH UovpWvDcaLRxN6Si7ssBQMsqY8iMmhB0ZlhOsaLjGFoRk/pDnp8kxjtVxEnAb3v9rMm6 WrvS03lvV2I84wKNfON28EiIXpS5n2cwPIsjNToKGkr4cssGxeHmd7ijVPfpCYGyHATD u/QUatv6Xxkd0zj9CH/ixQO4vyC2tknuYhgwhKszsJWW7IaKfzW8NQA9cpIAcLA1QtuF 9RfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7yne835l8M0Qgk6uaQXrJLgxrNefMQSePYhKpKknocM=; b=uVVYjyY0NMaAL6wuNwfAKFxmRHXWqL6VjMT/NcTMf6wRUo6sQAUvhG9QV14yh4Fyrd n7N3YtHKGD1GIrNwK+QrwH572FWhqm9+C5MgV2IZFwky9gUjO81AkXU2BQuwMYWIm6aF D2Bu8bEe78HZ7r1nsPo/zAnqRu6BMKlGWmvpcNjXYaTp0G15jsga5YwpNc1cu+ErtT9l TU7vqNe2ozs+7xC2sr9dQCOuTZXfc2BmKluZxlQtlSb5u/OULe9IZgAlWZj8wGMAxeih 8OsBCVjjsu323AtH+vAzxTPvgTbzIDKG1PEOOwxGWnkvok2FmNC37hD1+6OEeTTjBHNf FOvg== X-Gm-Message-State: AO0yUKWQVMhSmjxx/0WIjpOyhVbOLwNHqIgsGRWctBbVh+Pp6tevoe9f a1tkIlJLyjm0cHFtjOW7dq9xvA== X-Received: by 2002:a5d:5309:0:b0:2c7:1b6e:5a9c with SMTP id e9-20020a5d5309000000b002c71b6e5a9cmr13124wrv.2.1677179505012; Thu, 23 Feb 2023 11:11:45 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:5ee0:5af0:64bd:6198]) by smtp.gmail.com with ESMTPSA id b15-20020a5d4b8f000000b002c561805a4csm12957286wrt.45.2023.02.23.11.11.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Feb 2023 11:11:44 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse , Usama Arif Subject: [PATCH v11 02/12] cpu/hotplug: Move idle_thread_get() to Date: Thu, 23 Feb 2023 19:11:30 +0000 Message-Id: <20230223191140.4155012-3-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230223191140.4155012-1-usama.arif@bytedance.com> References: <20230223191140.4155012-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758650291292417752?= X-GMAIL-MSGID: =?utf-8?q?1758650291292417752?= From: David Woodhouse Instead of relying purely on the special-case wrapper in bringup_cpu() to pass the idle thread to __cpu_up(), expose idle_thread_get() so that the architecture code can obtain it directly when necessary. This will be useful when the existing __cpu_up() is split into multiple phases, only *one* of which will actually need the idle thread. If the architecture code is to register its new pre-bringup states with the cpuhp core, having a special-case wrapper to pass extra arguments is non-trivial and it's easier just to let the arch register its function pointer to be invoked with the standard API. Signed-off-by: David Woodhouse Reviewed-by: Thomas Gleixner Signed-off-by: Usama Arif Tested-by: Paul E. McKenney Tested-by: Kim Phillips Tested-by: Oleksandr Natalenko --- include/linux/smpboot.h | 7 +++++++ kernel/smpboot.h | 2 -- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/include/linux/smpboot.h b/include/linux/smpboot.h index 9d1bc65d226c..3862addcaa34 100644 --- a/include/linux/smpboot.h +++ b/include/linux/smpboot.h @@ -5,6 +5,13 @@ #include struct task_struct; + +#ifdef CONFIG_GENERIC_SMP_IDLE_THREAD +struct task_struct *idle_thread_get(unsigned int cpu); +#else +static inline struct task_struct *idle_thread_get(unsigned int cpu) { return NULL; } +#endif + /* Cookie handed to the thread_fn*/ struct smpboot_thread_data; diff --git a/kernel/smpboot.h b/kernel/smpboot.h index 34dd3d7ba40b..60c609318ad6 100644 --- a/kernel/smpboot.h +++ b/kernel/smpboot.h @@ -5,11 +5,9 @@ struct task_struct; #ifdef CONFIG_GENERIC_SMP_IDLE_THREAD -struct task_struct *idle_thread_get(unsigned int cpu); void idle_thread_set_boot_cpu(void); void idle_threads_init(void); #else -static inline struct task_struct *idle_thread_get(unsigned int cpu) { return NULL; } static inline void idle_thread_set_boot_cpu(void) { } static inline void idle_threads_init(void) { } #endif From patchwork Thu Feb 23 19:11:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 61060 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp484631wrd; Thu, 23 Feb 2023 11:13:56 -0800 (PST) X-Google-Smtp-Source: AK7set8cOKBglSP1NNLAhnVS8z1orNw5lfQlt8BN8rrvkrO8p0uUOINkJJhIRGhvoTHzf3y0MpLL X-Received: by 2002:a17:906:196:b0:88f:1255:59c with SMTP id 22-20020a170906019600b0088f1255059cmr18881031ejb.1.1677179636597; Thu, 23 Feb 2023 11:13:56 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677179636; cv=none; d=google.com; s=arc-20160816; b=Bp1gi2gAiIkpC0JYDOr3gFKopZ9j3/nBxRnjByUeYJtMHC00jgCgSiBHw0WrvwDf7s g2NJ7eWLSlQJ1KmEuFEP//O4D+mOyIs09IvO0vhfqE0uQ5+538lh7CV4mcz7CbRZSm5J d4xLa5mVdOsQ70WAcyOk0fp+jWXaPjBBYRIghTizJ9aFiz6gDQWiIR16jVSkADjGA+XV OG5w07fIev1P5Gjs94H7iBnTplFPizzYeU2vnt2Zu2QY0Fn4kH1h2hSgx88VEJ5n1G/p dmVLb+vQSScOLUW09mMeXf8nGzrOI2mSxAXbjkYr6T0W9q9cs1WkueQtOlvlXGNVhd2k pL5Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=+bBfIfpR8W2u8MvrmlDVru58AVxTNXaOUl1C+oqeDgw=; b=YjnY0D26vvVGxRVp6Wrx9cgdem7AgB8fVDKID4yvyqGpDNaoiH2i+WLHxeOWexEtX3 NPiWHhXZfIjfHjN5CbTACZgMLqrM4mHMr+gRwXjk8PMgC7FHQFq+x+uuGQG09jBpnJy+ YKlrYs3mEBF6+iWW91KkCiqGYh/a/VaOZFXn06+gMx8hUGBwFc3A3EMhar+QqCFGND5d t2RtedgjqOdWnCCB0grRW2eOeJQZ+ypn+oLdUjGh6AXDT0rBY+v6ujgmO9rXlN44BBpv yM2zilTeJ7RdTHdnTCz+pkC2if+FBC6yJQpe+TvyBATedC+60ZoLv6l5ADU9W9vI/DXJ yd1A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=UE46E5BX; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 28-20020a170906029c00b008b17e88ee87si28331093ejf.537.2023.02.23.11.13.33; Thu, 23 Feb 2023 11:13:56 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=UE46E5BX; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232070AbjBWTMm (ORCPT + 99 others); Thu, 23 Feb 2023 14:12:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59998 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231681AbjBWTMc (ORCPT ); Thu, 23 Feb 2023 14:12:32 -0500 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 698644ECF5 for ; Thu, 23 Feb 2023 11:12:09 -0800 (PST) Received: by mail-wr1-x42d.google.com with SMTP id c12so11561419wrw.1 for ; Thu, 23 Feb 2023 11:12:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+bBfIfpR8W2u8MvrmlDVru58AVxTNXaOUl1C+oqeDgw=; b=UE46E5BXTtS7vFvWGFT2AdaSl8TrYTYxGeVYdAYXiVQNr2k3cSU48ZjJ2fODdNa61H 7TJR/N6tZQylBTO7IpJ9cp5d/mIATGMWuRkIblEm8nQCyh2PETBEnHqzubZR/2DXj7/m wgeJczPN9rvrxjjHKhjwNltHZgWXORQUqCByNQajBrZPFxPZCUYxuJMVlAOY6TX6kEwd Qgvr6Gv/Mpm2BG81YG7apxdwuxhq8lkJJUzsz7TCCkIJabFCY+t0LoaQq3J5p8uBqlkQ H+jNwZ1Ca7vqi8kFhQGF8Y/R9rNXxqedDqvz2ym6wLDTh5wsxy8yYaMHmDQU0TcW3Kby QyUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+bBfIfpR8W2u8MvrmlDVru58AVxTNXaOUl1C+oqeDgw=; b=OdjXteqoVl6JZE9bcts/R3naLAj91BN81L/8C8o+s/fR6nrOuEiUv36OyxsHwzAxV/ lOv3kZyhXwByLRqG147eBHG3WTrq1Cr+ipTT2xMIqTBnXJZoLlLZYYj2TvcUH5AivlXd 64Pcd1AtEn0LivUW5pzEw8ABH2BfLEud8igL6lggfo/WUzAJCIyCG9NPxjRjbGSG6m2L vN9qU47qXS0o7mONseeIr37u9c8nyTBB3oo8hB0Z3pFzTJSoN75gtdn5bmIhi7NAQF0w 9JTm/IjBpJiE60/iYJSqYKLfkAd1JFv/XIIO1tWxvPHNf6ZwwHzE7hebjOk/5qSGzpYr kzKQ== X-Gm-Message-State: AO0yUKWRUsRdWUgT5M6Jx4sB3GR2MCJXpdOqNqcHGdL98+U7BNeqMF2y ByQNzTvzLh36aV9+szj94w9gIw== X-Received: by 2002:a5d:67d2:0:b0:2c5:a6c4:7fd3 with SMTP id n18-20020a5d67d2000000b002c5a6c47fd3mr10856571wrw.8.1677179505968; Thu, 23 Feb 2023 11:11:45 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:5ee0:5af0:64bd:6198]) by smtp.gmail.com with ESMTPSA id b15-20020a5d4b8f000000b002c561805a4csm12957286wrt.45.2023.02.23.11.11.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Feb 2023 11:11:45 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse , Usama Arif Subject: [PATCH v11 03/12] cpu/hotplug: Add dynamic parallel bringup states before CPUHP_BRINGUP_CPU Date: Thu, 23 Feb 2023 19:11:31 +0000 Message-Id: <20230223191140.4155012-4-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230223191140.4155012-1-usama.arif@bytedance.com> References: <20230223191140.4155012-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758650314469875526?= X-GMAIL-MSGID: =?utf-8?q?1758650314469875526?= From: David Woodhouse There is often significant latency in the early stages of CPU bringup, and time is wasted by waking each CPU (e.g. with SIPI/INIT/INIT on x86) and then waiting for it to respond before moving on to the next. Allow a platform to register a set of pre-bringup CPUHP states to which each CPU can be stepped in parallel, thus absorbing some of that latency. There is a subtlety here: even with an empty CPUHP_BP_PARALLEL_DYN step, this means that *all* CPUs are brought through the prepare states and to CPUHP_BP_PREPARE_DYN before any of them are taken to CPUHP_BRINGUP_CPU and then are allowed to run for themselves to CPUHP_ONLINE. So any combination of prepare/start calls which depend on A-B ordering for each CPU in turn, such as the X2APIC code which used to allocate a cluster mask 'just in case' and store it in a global variable in the prep stage, then potentially consume that preallocated structure from the AP and set the global pointer to NULL to be reallocated in CPUHP_X2APIC_PREPARE for the next CPU... would explode horribly. Any platform enabling the CPUHP_BP_PARALLEL_DYN steps must be reviewed and tested to ensure that such issues do not exist, and the existing behaviour of bringing CPUs to CPUHP_BP_PREPARE_DYN and then immediately to CPUHP_BRINGUP_CPU and CPUHP_ONLINE only one at a time does not change unless such a state is registered. Note that the new parallel stages do *not* yet bring each AP to the CPUHP_BRINGUP_CPU state at the same time, only to the new states which exist before it. The final loop in bringup_nonboot_cpus() is untouched, bringing each AP in turn from the final PARALLEL_DYN state (or all the way from CPUHP_OFFLINE) to CPUHP_BRINGUP_CPU and then waiting for that AP to do its own processing and reach CPUHP_ONLINE before releasing the next. Parallelising that part by bringing them all to CPUHP_BRINGUP_CPU and then waiting for them all is an exercise for the future. Signed-off-by: David Woodhouse Signed-off-by: Usama Arif Tested-by: Paul E. McKenney Tested-by: Kim Phillips Tested-by: Oleksandr Natalenko --- include/linux/cpuhotplug.h | 2 ++ kernel/cpu.c | 31 +++++++++++++++++++++++++++++-- 2 files changed, 31 insertions(+), 2 deletions(-) diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h index 6c6859bfc454..e5a73ae6ccc0 100644 --- a/include/linux/cpuhotplug.h +++ b/include/linux/cpuhotplug.h @@ -133,6 +133,8 @@ enum cpuhp_state { CPUHP_MIPS_SOC_PREPARE, CPUHP_BP_PREPARE_DYN, CPUHP_BP_PREPARE_DYN_END = CPUHP_BP_PREPARE_DYN + 20, + CPUHP_BP_PARALLEL_DYN, + CPUHP_BP_PARALLEL_DYN_END = CPUHP_BP_PARALLEL_DYN + 4, CPUHP_BRINGUP_CPU, /* diff --git a/kernel/cpu.c b/kernel/cpu.c index 6c0a92ca6bb5..fffb0da61ccc 100644 --- a/kernel/cpu.c +++ b/kernel/cpu.c @@ -1504,8 +1504,30 @@ int bringup_hibernate_cpu(unsigned int sleep_cpu) void bringup_nonboot_cpus(unsigned int setup_max_cpus) { + unsigned int n = setup_max_cpus - num_online_cpus(); unsigned int cpu; + /* + * An architecture may have registered parallel pre-bringup states to + * which each CPU may be brought in parallel. For each such state, + * bring N CPUs to it in turn before the final round of bringing them + * online. + */ + if (n > 0) { + enum cpuhp_state st = CPUHP_BP_PARALLEL_DYN; + + while (st <= CPUHP_BP_PARALLEL_DYN_END && cpuhp_hp_states[st].name) { + int i = n; + + for_each_present_cpu(cpu) { + cpu_up(cpu, st); + if (!--i) + break; + } + st++; + } + } + for_each_present_cpu(cpu) { if (num_online_cpus() >= setup_max_cpus) break; @@ -1882,6 +1904,10 @@ static int cpuhp_reserve_state(enum cpuhp_state state) step = cpuhp_hp_states + CPUHP_BP_PREPARE_DYN; end = CPUHP_BP_PREPARE_DYN_END; break; + case CPUHP_BP_PARALLEL_DYN: + step = cpuhp_hp_states + CPUHP_BP_PARALLEL_DYN; + end = CPUHP_BP_PARALLEL_DYN_END; + break; default: return -EINVAL; } @@ -1906,14 +1932,15 @@ static int cpuhp_store_callbacks(enum cpuhp_state state, const char *name, /* * If name is NULL, then the state gets removed. * - * CPUHP_AP_ONLINE_DYN and CPUHP_BP_PREPARE_DYN are handed out on + * CPUHP_AP_ONLINE_DYN and CPUHP_BP_P*_DYN are handed out on * the first allocation from these dynamic ranges, so the removal * would trigger a new allocation and clear the wrong (already * empty) state, leaving the callbacks of the to be cleared state * dangling, which causes wreckage on the next hotplug operation. */ if (name && (state == CPUHP_AP_ONLINE_DYN || - state == CPUHP_BP_PREPARE_DYN)) { + state == CPUHP_BP_PREPARE_DYN || + state == CPUHP_BP_PARALLEL_DYN)) { ret = cpuhp_reserve_state(state); if (ret < 0) return ret; From patchwork Thu Feb 23 19:11:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 61062 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp484788wrd; Thu, 23 Feb 2023 11:14:12 -0800 (PST) X-Google-Smtp-Source: AK7set/cLf/Yys2E1Iqhj2rqe382UgvcOIck82qAJYNT9181FjlADai0uXPNxtD6qs/38WhAX/PW X-Received: by 2002:a17:906:51d8:b0:8ae:cd8e:3957 with SMTP id v24-20020a17090651d800b008aecd8e3957mr18699620ejk.4.1677179652258; Thu, 23 Feb 2023 11:14:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677179652; cv=none; d=google.com; s=arc-20160816; b=aoApDePW7IhbvrYQt4EgHcL4ZBtFQ+c0yXVF2CjmXRBT4IVGBFrX4YtfjH1eJaXItG CnopWXsAWblWylsUT7CF/Z1tTEyihdd7qGEN9+j819yFaaMnVDl4YCz3+ZqxO1VOp6Rn OODGK4Dwg25Y4DBWkVcpaxXxeM0zu5JlzWQLerSHv0wmPJ/U/KmsqPIqT/vI2kymlT0o N9QJfv0jSfNMFsoeBtn2MQguS/eio0vKaM7RMtk0Vf0NZ86QiGmm/xxQQYPi0oTzuVuD /UtR+igB40x+Ms3esJGKhmGYBSrp7tsgGZ+4r+Zwor6Qdqe33DCbLYS15/pthHrouUBc lXng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=OivBWw2fC0XPAqVFrKNNGqM4Zo9U9YwXqHaW8wBeEGc=; b=HdT/T4PevEUd/PpUuqARCkrBZQ9RRTVOER+fUqyEbVnqZUR6z281C7rfgvOTNaxfpk thcx9SudvLTT8X3aw+HeI3BXielng0b7JJeq49Zhjd1zP4EycmwTR0U/cnBbSXcrZHl0 SqiSEFijIBat7h1I67kaWVINmofYohb5RA20k/3ALi5vMzmDJ9BonvJ9V0bgaLckI6Op oWpyJRWvCA+xf8JnJc8t5wIcHidCb6NrGrvo/Kbv+QgWTpjDjdAm3aNWT3I1aMVBDFAt g2ibDU/BtKzA9s3VnaeuOdBCMcaetrnC98woImfE4sHaI0SNpYr5RtRomfsLHpsKpz5e vPVg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=ZDG3NgAB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 9-20020a17090600c900b008c25e1c05desi24538658eji.392.2023.02.23.11.13.49; Thu, 23 Feb 2023 11:14:12 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=ZDG3NgAB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232165AbjBWTNH (ORCPT + 99 others); Thu, 23 Feb 2023 14:13:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60088 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231890AbjBWTNA (ORCPT ); Thu, 23 Feb 2023 14:13:00 -0500 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D572E59E4F for ; Thu, 23 Feb 2023 11:12:14 -0800 (PST) Received: by mail-wr1-x429.google.com with SMTP id bw19so928278wrb.13 for ; Thu, 23 Feb 2023 11:12:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OivBWw2fC0XPAqVFrKNNGqM4Zo9U9YwXqHaW8wBeEGc=; b=ZDG3NgABSuo3eKqXu8k0fCIFzS7x293O1xHnTdlbWTZzDzRsDo5w2fxICTyGIU9SZd La3hgbWjwAUViIFtj5nzdi9l286i+1BiiR8wgacSaNXsw/d9B2uf546kgw7pRzUOoU/O I9KjbX2NEBuvYhL7QbfZxgmJ52GtQy63SHB8mJdwQjIqrjEBSKFYLsgRjjr31IhELq8K +jVoVRVpaHAthXJtNr1V4FcLmQ+AKDPpCH7mcBHVUDKSMDSt9OamVRZ3xwxtubEoCSf2 oIPFuxYBjE/iDAjsJ2hW8lj2oGMrSUTAZif/uT3Llh8ahCoiI45510hGVXSicsDBQ/he p4rw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OivBWw2fC0XPAqVFrKNNGqM4Zo9U9YwXqHaW8wBeEGc=; b=tlua53Wgv23prtyQTbfiZVe06QMipMVNal8W3oks+VfauRLqg/Jsy0WB8R+5DYb1i1 xgsMGhsYNfOBMONG/YK1MLwoI+kE1ZsJ+CviJcluOWQ1AJ7JeHzSjsUPyUJigJUYoI28 51JQ2pRFX04xVb7KVoarzxXut6qpZm/I19GgIG72sKWXnHx590ISytK8z9QpDA7rx3cm D3psf4dWS+Nz8Z98CIWEnahYVDKnaLNBjqFeJHnE3n6gmYi5USL6TwuhM6BSBdgtVPQc /grHy+d15SDAaaa01L9h/lALMlaIny7ox+Udn70UErpYWdMN1PrHb+SQl9DeKq/imHel 7+IQ== X-Gm-Message-State: AO0yUKWcVB+CPKv0WQ/feGxXhswKIh9rTRRzLORgshVwd3gDHtXie/y/ 8HGsep1+8WKbVZIOJsXMba61og== X-Received: by 2002:adf:dd86:0:b0:2c7:e60:a41d with SMTP id x6-20020adfdd86000000b002c70e60a41dmr4335140wrl.61.1677179506858; Thu, 23 Feb 2023 11:11:46 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:5ee0:5af0:64bd:6198]) by smtp.gmail.com with ESMTPSA id b15-20020a5d4b8f000000b002c561805a4csm12957286wrt.45.2023.02.23.11.11.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Feb 2023 11:11:46 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse , Usama Arif Subject: [PATCH v11 04/12] x86/smpboot: Reference count on smpboot_setup_warm_reset_vector() Date: Thu, 23 Feb 2023 19:11:32 +0000 Message-Id: <20230223191140.4155012-5-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230223191140.4155012-1-usama.arif@bytedance.com> References: <20230223191140.4155012-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758650331249698660?= X-GMAIL-MSGID: =?utf-8?q?1758650331249698660?= From: David Woodhouse When bringing up a secondary CPU from do_boot_cpu(), the warm reset flag is set in CMOS and the starting IP for the trampoline written inside the BDA at 0x467. Once the CPU is running, the CMOS flag is unset and the value in the BDA cleared. To allow for parallel bringup of CPUs, add a reference count to track the number of CPUs currently bring brought up, and clear the state only when the count reaches zero. Since the RTC spinlock is required to write to the CMOS, it can be used for mutual exclusion on the refcount too. Signed-off-by: David Woodhouse Signed-off-by: Usama Arif Tested-by: Paul E. McKenney Tested-by: Kim Phillips Tested-by: Oleksandr Natalenko --- arch/x86/kernel/smpboot.c | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 55cad72715d9..3a793772a2aa 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -121,17 +121,20 @@ int arch_update_cpu_topology(void) return retval; } + +static unsigned int smpboot_warm_reset_vector_count; + static inline void smpboot_setup_warm_reset_vector(unsigned long start_eip) { unsigned long flags; spin_lock_irqsave(&rtc_lock, flags); - CMOS_WRITE(0xa, 0xf); + if (!smpboot_warm_reset_vector_count++) { + CMOS_WRITE(0xa, 0xf); + *((volatile unsigned short *)phys_to_virt(TRAMPOLINE_PHYS_HIGH)) = start_eip >> 4; + *((volatile unsigned short *)phys_to_virt(TRAMPOLINE_PHYS_LOW)) = start_eip & 0xf; + } spin_unlock_irqrestore(&rtc_lock, flags); - *((volatile unsigned short *)phys_to_virt(TRAMPOLINE_PHYS_HIGH)) = - start_eip >> 4; - *((volatile unsigned short *)phys_to_virt(TRAMPOLINE_PHYS_LOW)) = - start_eip & 0xf; } static inline void smpboot_restore_warm_reset_vector(void) @@ -143,10 +146,12 @@ static inline void smpboot_restore_warm_reset_vector(void) * to default values. */ spin_lock_irqsave(&rtc_lock, flags); - CMOS_WRITE(0, 0xf); + if (!--smpboot_warm_reset_vector_count) { + CMOS_WRITE(0, 0xf); + *((volatile u32 *)phys_to_virt(TRAMPOLINE_PHYS_LOW)) = 0; + } spin_unlock_irqrestore(&rtc_lock, flags); - *((volatile u32 *)phys_to_virt(TRAMPOLINE_PHYS_LOW)) = 0; } /* From patchwork Thu Feb 23 19:11:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 61063 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp484871wrd; Thu, 23 Feb 2023 11:14:22 -0800 (PST) X-Google-Smtp-Source: AK7set+MqctfcMOg1k5YGoA3N0a+X3B0TSzdvNEqU285NcOoLC5vDTjaXEEkRpsm8E6KJX1qo7mW X-Received: by 2002:a17:906:5d:b0:88c:f97c:7c87 with SMTP id 29-20020a170906005d00b0088cf97c7c87mr19501136ejg.2.1677179662827; Thu, 23 Feb 2023 11:14:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677179662; cv=none; d=google.com; s=arc-20160816; b=iz6ouvYrDEYC5WfQVGHJ0SQqLlvZJCp3EyyyKNE4VEe+F3MyBEDH5wkOVMGO/fcZgQ jD0QEKspXneZv5nbcHZlgz/r2k6Iiiw6UWZN33WoD2mRNw9Cv1aam76mmGP/JPUG2w8O eYuxYh/wiW/NVt7eamb0KhH11ZTKbq2bMoLZizJdJMM4BKG4sR/mhNGgvkcQVxTgOLls l5iYgaJXBoameGhPE75ys4J+GEC6KEI7w2g43cpV6GkvJW51sVaqoSTQ6a4T7fEsnpGp 1o9z/k9G8rhQFBoa6sj70Ea/cFW1sn7x2AokjFWVBALV2pru0auL6Opcpg6/mlX5qvhx DwcA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=kmlw8LHyg0IiYBrAdMu6qbL5s9Zil0lzZnE7VOWUkds=; b=u2RYpdTGRSLol4IGHfjhfFhwPHSyihh+knJ2xyDAo5yYtkqrdFtHsjTV1gLlA0A/ZQ B4CKHilotzlglvsHk8fR4EspUJHg/TtL+rxwXskZQdkMqkT5/TXs9ZucLSNEktVWMLUF wGBHb9C2NsRAmtNy3mmwTusuPh7Ry1bqSC61sNlMmvMdDFMIoRvcylbnUyGyTLDtIvtC XuxclWe77olNHd9M9oWPYY9idm6d0ufxW6fXbTc8hDsj3CllbiZHdhFqv/TZcQhUBhZA XUulWdiN+U8vc/uuYQ+QIZb2QthO4bFk5akpDrY3X9SOsaaQ3vHqzEpU+OiKmRnu82Yy OxnQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=h0jwJEmc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id fv22-20020a170907509600b008dcebb5a122si9743487ejc.191.2023.02.23.11.13.58; Thu, 23 Feb 2023 11:14:22 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=h0jwJEmc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232158AbjBWTNK (ORCPT + 99 others); Thu, 23 Feb 2023 14:13:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60208 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232155AbjBWTNB (ORCPT ); Thu, 23 Feb 2023 14:13:01 -0500 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B803C580F0 for ; Thu, 23 Feb 2023 11:12:16 -0800 (PST) Received: by mail-wm1-x330.google.com with SMTP id ay29-20020a05600c1e1d00b003e9f4c2b623so260728wmb.3 for ; Thu, 23 Feb 2023 11:12:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kmlw8LHyg0IiYBrAdMu6qbL5s9Zil0lzZnE7VOWUkds=; b=h0jwJEmcH5tV9UWF6cp7pYSf2onqVv7QczvptBTmGJoDCrK3+AaDP1laBKbBLWlA7L ZnJC1piyjahgIHSLN+Yegos469i8AoudF/qCDJ3+e52KviFKfvdq4+nMfudgjPXE3v9v eV2TcfEl+uu7Wii+R/+3JMQCRF08PsoUWRv3LMlXWWTlyDpIBZnyvL/ZUAGxq+5/yH4i nA7Jaz4MCOGAr/2QmyEjATqAlyAkxM4wmV3RADtVfDhvqqzX24z9sIUU0h0dbx+NybrP L0jPR2XiJUBlPqor05BFNoQk7G7L5RA1dCgegAtj6B0kn6kmRVxDtIUFleGhbkefcX6y H+hg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kmlw8LHyg0IiYBrAdMu6qbL5s9Zil0lzZnE7VOWUkds=; b=CE84KiSXlrVtpebMI9knsEBDiMwtH/Sy5KiUO4otUhsBG1m7TTATz1LcN9ZHP9Oauu 04VYKVBq5kfvny+81hiwagRWXNEPbxUZO0qnNh9jWUYID6Os8EBw0tr+3upHunTDxylN 9+R6h/F6oC6eBZZlTyaY3iMC8WJoEJRucH67Los/d5o9F3K2QGefn3QIJoGE5F9mRPUd UOVqzRXir7xsUHeXxP+d3Z+VmXG96kCO9aQkRRiESB84ENqpFlhHjSExxixiRfb01pIA PvE3Jg8XnQK4GDOw91v7mm7ldR7XupuDbeE8m3x9x7T8GzXdYNhaUhFDHB6I3HVrSWsP xkgQ== X-Gm-Message-State: AO0yUKWrfZv1d3aKKC/HI/5o7WT4D9SSXMrN3hWjtPKsPr306cPzqp0m D0Mxd1AlEUHGjZpSv8ytQMikYA== X-Received: by 2002:a05:600c:1c88:b0:3ea:dbdd:66df with SMTP id k8-20020a05600c1c8800b003eadbdd66dfmr1065387wms.2.1677179507753; Thu, 23 Feb 2023 11:11:47 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:5ee0:5af0:64bd:6198]) by smtp.gmail.com with ESMTPSA id b15-20020a5d4b8f000000b002c561805a4csm12957286wrt.45.2023.02.23.11.11.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Feb 2023 11:11:47 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse , Usama Arif Subject: [PATCH v11 05/12] x86/smpboot: Split up native_cpu_up into separate phases and document them Date: Thu, 23 Feb 2023 19:11:33 +0000 Message-Id: <20230223191140.4155012-6-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230223191140.4155012-1-usama.arif@bytedance.com> References: <20230223191140.4155012-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758650341963508859?= X-GMAIL-MSGID: =?utf-8?q?1758650341963508859?= From: David Woodhouse There are four logical parts to what native_cpu_up() does on the BSP (or on the controlling CPU for a later hotplug): 1) Wake the AP by sending the INIT/SIPI/SIPI sequence. 2) Wait for the AP to make it as far as wait_for_master_cpu() which sets that CPU's bit in cpu_initialized_mask, then sets the bit in cpu_callout_mask to let the AP proceed through cpu_init(). 3) Wait for the AP to finish cpu_init() and get as far as the smp_callin() call, which sets that CPU's bit in cpu_callin_mask. 4) Perform the TSC synchronization and wait for the AP to actually mark itself online in cpu_online_mask. In preparation to allow these phases to operate in parallel on multiple APs, split them out into separate functions and document the interactions a little more clearly in both the BSP and AP code paths. No functional change intended. Signed-off-by: David Woodhouse Signed-off-by: Usama Arif Tested-by: Paul E. McKenney Tested-by: Kim Phillips Tested-by: Oleksandr Natalenko --- arch/x86/kernel/smpboot.c | 181 ++++++++++++++++++++++++++------------ 1 file changed, 127 insertions(+), 54 deletions(-) diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 3a793772a2aa..b18c1385e181 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -204,6 +204,10 @@ static void smp_callin(void) wmb(); + /* + * This runs the AP through all the cpuhp states to its target + * state (CPUHP_ONLINE in the case of serial bringup). + */ notify_cpu_starting(cpuid); /* @@ -231,17 +235,32 @@ static void notrace start_secondary(void *unused) load_cr3(swapper_pg_dir); __flush_tlb_all(); #endif + /* + * Sync point with do_wait_cpu_initialized(). Before proceeding through + * cpu_init(), the AP will call wait_for_master_cpu() which sets its + * own bit in cpu_initialized_mask and then waits for the BSP to set + * its bit in cpu_callout_mask to release it. + */ cpu_init_secondary(); rcu_cpu_starting(raw_smp_processor_id()); x86_cpuinit.early_percpu_clock_init(); + + /* + * Sync point with do_wait_cpu_callin(). The AP doesn't wait here + * but just sets the bit to let the controlling CPU (BSP) know that + * it's got this far. + */ smp_callin(); enable_start_cpu0 = 0; /* otherwise gcc will move up smp_processor_id before the cpu_init */ barrier(); + /* - * Check TSC synchronization with the boot CPU: + * Check TSC synchronization with the boot CPU (or whichever CPU + * is controlling the bringup). It will do its part of this from + * do_wait_cpu_online(), making it an implicit sync point. */ check_tsc_sync_target(); @@ -254,6 +273,7 @@ static void notrace start_secondary(void *unused) * half valid vector space. */ lock_vector_lock(); + /* Sync point with do_wait_cpu_online() */ set_cpu_online(smp_processor_id(), true); lapic_online(); unlock_vector_lock(); @@ -1083,7 +1103,6 @@ static int do_boot_cpu(int apicid, int cpu, struct task_struct *idle, unsigned long start_ip = real_mode_header->trampoline_start; unsigned long boot_error = 0; - unsigned long timeout; #ifdef CONFIG_X86_64 /* If 64-bit wakeup method exists, use the 64-bit mode trampoline IP */ @@ -1144,55 +1163,94 @@ static int do_boot_cpu(int apicid, int cpu, struct task_struct *idle, boot_error = wakeup_cpu_via_init_nmi(cpu, start_ip, apicid, cpu0_nmi_registered); - if (!boot_error) { - /* - * Wait 10s total for first sign of life from AP - */ - boot_error = -1; - timeout = jiffies + 10*HZ; - while (time_before(jiffies, timeout)) { - if (cpumask_test_cpu(cpu, cpu_initialized_mask)) { - /* - * Tell AP to proceed with initialization - */ - cpumask_set_cpu(cpu, cpu_callout_mask); - boot_error = 0; - break; - } - schedule(); - } - } + return boot_error; +} - if (!boot_error) { - /* - * Wait till AP completes initial initialization - */ - while (!cpumask_test_cpu(cpu, cpu_callin_mask)) { - /* - * Allow other tasks to run while we wait for the - * AP to come online. This also gives a chance - * for the MTRR work(triggered by the AP coming online) - * to be completed in the stop machine context. - */ - schedule(); - } +static int do_wait_cpu_cpumask(unsigned int cpu, const struct cpumask *mask) +{ + unsigned long timeout; + + /* + * Wait up to 10s for the CPU to report in. + */ + timeout = jiffies + 10*HZ; + while (time_before(jiffies, timeout)) { + if (cpumask_test_cpu(cpu, mask)) + return 0; + + schedule(); } + return -1; +} - if (x86_platform.legacy.warm_reset) { - /* - * Cleanup possible dangling ends... - */ - smpboot_restore_warm_reset_vector(); +/* + * Bringup step two: Wait for the target AP to reach cpu_init_secondary() + * and thus wait_for_master_cpu(), then set cpu_callout_mask to allow it + * to proceed. The AP will then proceed past setting its 'callin' bit + * and end up waiting in check_tsc_sync_target() until we reach + * do_wait_cpu_online() to tend to it. + */ +static int do_wait_cpu_initialized(unsigned int cpu) +{ + /* + * Wait for first sign of life from AP. + */ + if (do_wait_cpu_cpumask(cpu, cpu_initialized_mask)) + return -1; + + cpumask_set_cpu(cpu, cpu_callout_mask); + return 0; +} + +/* + * Bringup step three: Wait for the target AP to reach smp_callin(). + * The AP is not waiting for us here so we don't need to parallelise + * this step. Not entirely clear why we care about this, since we just + * proceed directly to TSC synchronization which is the next sync + * point with the AP anyway. + */ +static int do_wait_cpu_callin(unsigned int cpu) +{ + /* + * Wait till AP completes initial initialization. + */ + return do_wait_cpu_cpumask(cpu, cpu_callin_mask); +} + +/* + * Bringup step four: Synchronize the TSC and wait for the target AP + * to reach set_cpu_online() in start_secondary(). + */ +static int do_wait_cpu_online(unsigned int cpu) +{ + unsigned long flags; + + /* + * Check TSC synchronization with the AP (keep irqs disabled + * while doing so): + */ + local_irq_save(flags); + check_tsc_sync_source(cpu); + local_irq_restore(flags); + + /* + * Wait for the AP to mark itself online. Not entirely + * clear why we care, since the generic cpuhp code will + * wait for it to each CPUHP_AP_ONLINE_IDLE before going + * ahead with the rest of the bringup anyway. + */ + while (!cpu_online(cpu)) { + cpu_relax(); + touch_nmi_watchdog(); } - return boot_error; + return 0; } -int native_cpu_up(unsigned int cpu, struct task_struct *tidle) +static int do_cpu_up(unsigned int cpu, struct task_struct *tidle) { int apicid = apic->cpu_present_to_apicid(cpu); int cpu0_nmi_registered = 0; - unsigned long flags; int err, ret = 0; lockdep_assert_irqs_enabled(); @@ -1239,19 +1297,6 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) goto unreg_nmi; } - /* - * Check TSC synchronization with the AP (keep irqs disabled - * while doing so): - */ - local_irq_save(flags); - check_tsc_sync_source(cpu); - local_irq_restore(flags); - - while (!cpu_online(cpu)) { - cpu_relax(); - touch_nmi_watchdog(); - } - unreg_nmi: /* * Clean up the nmi handler. Do this after the callin and callout sync @@ -1263,6 +1308,34 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) return ret; } +int native_cpu_up(unsigned int cpu, struct task_struct *tidle) +{ + int ret; + + ret = do_cpu_up(cpu, tidle); + if (ret) + return ret; + + ret = do_wait_cpu_initialized(cpu); + if (ret) + return ret; + + ret = do_wait_cpu_callin(cpu); + if (ret) + return ret; + + ret = do_wait_cpu_online(cpu); + + if (x86_platform.legacy.warm_reset) { + /* + * Cleanup possible dangling ends... + */ + smpboot_restore_warm_reset_vector(); + } + + return ret; +} + /** * arch_disable_smp_support() - disables SMP support for x86 at runtime */ From patchwork Thu Feb 23 19:11:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 61065 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp485106wrd; Thu, 23 Feb 2023 11:14:54 -0800 (PST) X-Google-Smtp-Source: AK7set/BEYMOal9pKl8ExaS5NsbjgnLM3GJHyB7csxLu5O3o6t2Y6h+dsFKHuJQe2lQySQu4c0gW X-Received: by 2002:a05:6402:2551:b0:4ac:b9e3:33bc with SMTP id l17-20020a056402255100b004acb9e333bcmr12732790edb.1.1677179694628; Thu, 23 Feb 2023 11:14:54 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677179694; cv=none; d=google.com; s=arc-20160816; b=biJlhHTS1gwiRu96Crhi6C5oL1+umO//+BczEqQqh3i1oKsT9Mz4SsN0Mq9cSKvHKn shrLHnQu8nzgRZmxocfBHyM/SO150QHSAPH8Chmqo4b00foj4iAre/WjVSAi5xJ3FkIH xSVe+vVgTVKh0YqbZjK+HV5+7YdEIzhtzy8GafkjE+Jf7TB2Cw7DH7Ma6zMInas8Vzg/ I8zqkVY+8SAyf8ItZJqu1Fsa+kuI3RCDXT1fPqealeLK7ibNrjYXvxuf7RQS0MRHAvKg ecXRHqG1LCZ3KFjRleTi8OJkTyIRiOhNN10FUYGcUVi3oEn2ZS1nSsXnBXRfhSuGKXer oMpA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=0eWCnrG5+k8d70ipSCEgGk7Fcy+1iacsbarHxi4tH0Q=; b=hboC0ZVDPCVnLW24gkC3RmjwSVQHfgjN6sYbZRPY1QSe6lT8Zmf++ie0bU3VHFbvIN 1t0ugUC3NfW3j8l4k6cLkB6lMGGMN59MX3NKCaSdcl3vXr6fgqDz4ehR/nlphsZmX26p Ogc6b2WF4ZcgluZ+Cbslv8lqOd2FLp5U21ntrMF3b3yUD/fImwNphB0BVLrRKS5eAEsq lLiicPa4tV9m51DVTUxxjrT40RTXBw/BQtCPXZ4tZ6dOG7pUFG9ufnSlrDyefk2h5U/G JxxV7UnV1zEgtSQ1Cxd9Pz7Qc6Z3uioB+SB60kJsfD02u4U1GjPx9xe0JfS9+UWCzzHH oeQw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=Dm7gVPd2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id d26-20020aa7d69a000000b004af6e956bd7si2518236edr.421.2023.02.23.11.14.31; Thu, 23 Feb 2023 11:14:54 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=Dm7gVPd2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232588AbjBWTNi (ORCPT + 99 others); Thu, 23 Feb 2023 14:13:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60026 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232310AbjBWTNM (ORCPT ); Thu, 23 Feb 2023 14:13:12 -0500 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F3D7C5C147 for ; Thu, 23 Feb 2023 11:12:20 -0800 (PST) Received: by mail-wr1-x42d.google.com with SMTP id 6so11156104wrb.11 for ; Thu, 23 Feb 2023 11:12:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0eWCnrG5+k8d70ipSCEgGk7Fcy+1iacsbarHxi4tH0Q=; b=Dm7gVPd2n+lpCkDY/8JTZhOyM2P2bPk7Rl2MVfwRHoYhsbBPQFP/7DKQL6JD3280SV 0rhiI8tMP58UdeTjS3mlMsxDBipsOq9hvixk1iD3BBx0Jz2g2LGONkICZK/l62GCFwCh XiQKpfnXcyRsLpcGf19K92gdCC7VeBc+XtOmKAhiDO6UsG6fTnhZUPV5MWTbQSKhsl9O yc4Q+FzJUhmeLxL/h/fiQQQH/AH0ZSqUXKJyTSvSSR9XNHqdbs2cfvWoxK9fHO1PZopk /jZdnbZxYDzhsPQbGRuIWMm/JqlHOv5VZPvmsfAorJfaHHK5bvirfS7Ewo0YQ4eZDWya 4pAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0eWCnrG5+k8d70ipSCEgGk7Fcy+1iacsbarHxi4tH0Q=; b=YiDPAzRDoodiuBvTVKaWdc5Bqss/Nu5i8DOzjqMufL/wWzJasQNsAp98V2VL//AkqL J+tyOZA3xOLQRzyW/frwayJCUP2PPYozc8xsE5x/kE1DKtFOe19mUeRZK8O/S0gjrlf/ OyTYLwjRCtfisv5vu4Aql1JHBtYm+UNV8SmfN4p0TK7vHC8haZAlBbY+9B01ykcCASra qlUGxorziub5N9esIMyacoyia822GvGo2VBn8iYkEBhNZnCT3MzPIb+8ZEWsTYYebVaF An2c1BhKYPpNFz7BhStcPFqyNafGxdCT1d/NPZvek6AkgRAWqD5cjwnnHhTEuBuTFKI9 rOOA== X-Gm-Message-State: AO0yUKW6J8A9XbBNvw3AjnItcT7XLii/2YedCHkUCU+davkB89FAwdkx eF8P099pLqvEc3JhfRJFlKrw4g== X-Received: by 2002:a5d:4d50:0:b0:2c6:e7ff:8f17 with SMTP id a16-20020a5d4d50000000b002c6e7ff8f17mr11090018wru.25.1677179508850; Thu, 23 Feb 2023 11:11:48 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:5ee0:5af0:64bd:6198]) by smtp.gmail.com with ESMTPSA id b15-20020a5d4b8f000000b002c561805a4csm12957286wrt.45.2023.02.23.11.11.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Feb 2023 11:11:48 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse , Usama Arif Subject: [PATCH v11 06/12] x86/smpboot: Support parallel startup of secondary CPUs Date: Thu, 23 Feb 2023 19:11:34 +0000 Message-Id: <20230223191140.4155012-7-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230223191140.4155012-1-usama.arif@bytedance.com> References: <20230223191140.4155012-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758650375498349314?= X-GMAIL-MSGID: =?utf-8?q?1758650375498349314?= From: David Woodhouse Rework the real-mode startup code to allow for APs to be brought up in parallel. This is in two parts: 1. Introduce a bit-spinlock to prevent them from all using the real mode stack at the same time. 2. Avoid the use of global variables for passing per-CPU information to the APs. To achieve the latter, export the cpuid_to_apicid[] array so that each AP can find its own per_cpu data (and thus initial_gs, initial_stack and early_gdt_descr) by searching therein based on its APIC ID. Introduce a global variable 'smpboot_control' indicating to the AP how it should find its APIC ID. For a serialized bringup, the APIC ID is explicitly passed in the low bits of smpboot_control, while for parallel mode there are flags directing the AP to find its APIC ID in CPUID leaf 0x0b (for X2APIC mode) or CPUID leaf 0x01 where 8 bits are sufficient. Parallel startup may be disabled by a command line option, and also if: • AMD SEV-ES is in use, since the AP may not use CPUID that early. • X2APIC is enabled, but CPUID leaf 0xb is not present and correect. • X2APIC is not enabled but not even CPUID leaf 0x01 exists. Aside from the fact that APs will now look up their per-cpu data via the newly-exported cpuid_to_apicid[] table, there is no behavioural change intended yet, since new parallel CPUHP states have not — yet — been added. [ tglx: Initial proof of concept patch with bitlock and APIC ID lookup ] [ dwmw2: Rework and testing, commit message, CPUID 0x1 and CPU0 support ] [ seanc: Fix stray override of initial_gs in common_cpu_up() ] [ Oleksandr Natalenko: reported suspend/resume issue fixed in x86_acpi_suspend_lowlevel ] Co-developed-by: Thomas Gleixner Co-developed-by: Brian Gerst Signed-off-by: Thomas Gleixner Signed-off-by: Brian Gerst Signed-off-by: David Woodhouse Signed-off-by: Usama Arif Tested-by: Paul E. McKenney Tested-by: Kim Phillips Tested-by: Oleksandr Natalenko --- .../admin-guide/kernel-parameters.txt | 3 + arch/x86/include/asm/realmode.h | 3 + arch/x86/include/asm/smp.h | 10 +- arch/x86/kernel/acpi/sleep.c | 7 ++ arch/x86/kernel/apic/apic.c | 2 +- arch/x86/kernel/asm-offsets.c | 1 + arch/x86/kernel/head_64.S | 98 ++++++++++++++++++- arch/x86/kernel/smpboot.c | 62 +++++++++++- arch/x86/realmode/init.c | 3 + arch/x86/realmode/rm/trampoline_64.S | 14 +++ 10 files changed, 196 insertions(+), 7 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 6cfa6e3996cf..ee099b8aac6d 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3819,6 +3819,9 @@ nomodule Disable module load + no_parallel_bringup + [X86,SMP] Disable parallel brinugp of secondary cores. + nopat [X86] Disable PAT (page attribute table extension of pagetables) support. diff --git a/arch/x86/include/asm/realmode.h b/arch/x86/include/asm/realmode.h index a336feef0af1..f0357cfe2fb0 100644 --- a/arch/x86/include/asm/realmode.h +++ b/arch/x86/include/asm/realmode.h @@ -52,6 +52,7 @@ struct trampoline_header { u64 efer; u32 cr4; u32 flags; + u32 lock; #endif }; @@ -65,6 +66,8 @@ extern unsigned long initial_stack; extern unsigned long initial_vc_handler; #endif +extern u32 *trampoline_lock; + extern unsigned char real_mode_blob[]; extern unsigned char real_mode_relocs[]; diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h index b4dbb20dab1a..33c0d5fd8af6 100644 --- a/arch/x86/include/asm/smp.h +++ b/arch/x86/include/asm/smp.h @@ -199,5 +199,13 @@ extern void nmi_selftest(void); #define nmi_selftest() do { } while (0) #endif -#endif /* __ASSEMBLY__ */ +extern unsigned int smpboot_control; + +#endif /* !__ASSEMBLY__ */ + +/* Control bits for startup_64 */ +#define STARTUP_SECONDARY 0x80000000 +#define STARTUP_APICID_CPUID_0B 0x40000000 +#define STARTUP_APICID_CPUID_01 0x20000000 + #endif /* _ASM_X86_SMP_H */ diff --git a/arch/x86/kernel/acpi/sleep.c b/arch/x86/kernel/acpi/sleep.c index 3b7f4cdbf2e0..47e75c056cb5 100644 --- a/arch/x86/kernel/acpi/sleep.c +++ b/arch/x86/kernel/acpi/sleep.c @@ -16,6 +16,7 @@ #include #include #include +#include #include #include "../../realmode/rm/wakeup.h" @@ -57,6 +58,7 @@ asmlinkage acpi_status __visible x86_acpi_enter_sleep_state(u8 state) */ int x86_acpi_suspend_lowlevel(void) { + unsigned int __maybe_unused saved_smpboot_ctrl; struct wakeup_header *header = (struct wakeup_header *) __va(real_mode_header->wakeup_header); @@ -115,6 +117,8 @@ int x86_acpi_suspend_lowlevel(void) early_gdt_descr.address = (unsigned long)get_cpu_gdt_rw(smp_processor_id()); initial_gs = per_cpu_offset(smp_processor_id()); + /* Force the startup into boot mode */ + saved_smpboot_ctrl = xchg(&smpboot_control, 0); #endif initial_code = (unsigned long)wakeup_long64; saved_magic = 0x123456789abcdef0L; @@ -127,6 +131,9 @@ int x86_acpi_suspend_lowlevel(void) pause_graph_tracing(); do_suspend_lowlevel(); unpause_graph_tracing(); + + if (IS_ENABLED(CONFIG_64BIT) && IS_ENABLED(CONFIG_SMP)) + smpboot_control = saved_smpboot_ctrl; return 0; } diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c index 20d9a604da7c..ac1d7e5da1f2 100644 --- a/arch/x86/kernel/apic/apic.c +++ b/arch/x86/kernel/apic/apic.c @@ -2377,7 +2377,7 @@ static int nr_logical_cpuids = 1; /* * Used to store mapping between logical CPU IDs and APIC IDs. */ -static int cpuid_to_apicid[] = { +int cpuid_to_apicid[] = { [0 ... NR_CPUS - 1] = -1, }; diff --git a/arch/x86/kernel/asm-offsets.c b/arch/x86/kernel/asm-offsets.c index 82c783da16a8..797ae1a15c91 100644 --- a/arch/x86/kernel/asm-offsets.c +++ b/arch/x86/kernel/asm-offsets.c @@ -108,6 +108,7 @@ static void __used common(void) OFFSET(TSS_sp1, tss_struct, x86_tss.sp1); OFFSET(TSS_sp2, tss_struct, x86_tss.sp2); OFFSET(X86_top_of_stack, pcpu_hot, top_of_stack); + OFFSET(X86_current_task, pcpu_hot, current_task); #ifdef CONFIG_CALL_DEPTH_TRACKING OFFSET(X86_call_depth, pcpu_hot, call_depth); #endif diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index 222efd4a09bc..c32e5b06a9ce 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -25,6 +25,7 @@ #include #include #include +#include /* * We are not able to switch in one step to the final KERNEL ADDRESS SPACE @@ -241,6 +242,85 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) UNWIND_HINT_EMPTY ANNOTATE_NOENDBR // above +#ifdef CONFIG_SMP + /* + * Is this the boot CPU coming up? If so everything is available + * in initial_gs, initial_stack and early_gdt_descr. + */ + movl smpboot_control(%rip), %edx + testl $STARTUP_SECONDARY, %edx + jz .Lsetup_cpu + + /* + * For parallel boot, the APIC ID is retrieved from CPUID, and then + * used to look up the CPU number. For booting a single CPU, the + * CPU number is encoded in smpboot_control. + * + * Bit 31 STARTUP_SECONDARY flag (checked above) + * Bit 30 STARTUP_APICID_CPUID_0B flag (use CPUID 0x0b) + * Bit 29 STARTUP_APICID_CPUID_01 flag (use CPUID 0x01) + * Bit 0-24 CPU# if STARTUP_APICID_CPUID_xx flags are not set + */ + testl $STARTUP_APICID_CPUID_0B, %edx + jnz .Luse_cpuid_0b + testl $STARTUP_APICID_CPUID_01, %edx + jnz .Luse_cpuid_01 + andl $0x0FFFFFFF, %edx + movl %edx, %ecx + jmp .Linit_cpu_data + +.Luse_cpuid_01: + mov $0x01, %eax + cpuid + mov %ebx, %edx + shr $24, %edx + jmp .Lsetup_AP + +.Luse_cpuid_0b: + mov $0x0B, %eax + xorl %ecx, %ecx + cpuid + +.Lsetup_AP: + /* EDX contains the APIC ID of the current CPU */ + xorq %rcx, %rcx + leaq cpuid_to_apicid(%rip), %rbx + +.Lfind_cpunr: + cmpl (%rbx,%rcx,4), %edx + jz .Linit_cpu_data + inc %ecx + cmpl nr_cpu_ids(%rip), %ecx + jb .Lfind_cpunr + + /* APIC ID not found in the table. Drop the trampoline lock and bail. */ + movq trampoline_lock(%rip), %rax + lock + btrl $0, (%rax) + +1: cli + hlt + jmp 1b + +.Linit_cpu_data: + /* Get the per cpu offset for the given CPU# which is in ECX */ + leaq __per_cpu_offset(%rip), %rbx + movq (%rbx,%rcx,8), %rbx + /* Save it for GS BASE setup */ + movq %rbx, initial_gs(%rip) + + /* Calculate the GDT address */ + movq $gdt_page, %rcx + addq %rbx, %rcx + movq %rcx, early_gdt_descr_base(%rip) + + /* Find the idle task stack */ + movq pcpu_hot + X86_current_task(%rbx), %rcx + movq TASK_threadsp(%rcx), %rcx + movq %rcx, initial_stack(%rip) +#endif /* CONFIG_SMP */ + +.Lsetup_cpu: /* * We must switch to a new descriptor in kernel space for the GDT * because soon the kernel won't have access anymore to the userspace @@ -281,6 +361,14 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) */ movq initial_stack(%rip), %rsp + /* Drop the realmode protection. For the boot CPU the pointer is NULL! */ + movq trampoline_lock(%rip), %rax + testq %rax, %rax + jz .Lsetup_idt + lock + btrl $0, (%rax) + +.Lsetup_idt: /* Setup and Load IDT */ pushq %rsi call early_setup_idt @@ -372,7 +460,11 @@ SYM_CODE_END(secondary_startup_64) SYM_CODE_START(start_cpu0) ANNOTATE_NOENDBR UNWIND_HINT_EMPTY - movq initial_stack(%rip), %rsp + + /* Find the idle task stack */ + movq PER_CPU_VAR(pcpu_hot) + X86_current_task, %rcx + movq TASK_threadsp(%rcx), %rsp + jmp .Ljump_to_C_code SYM_CODE_END(start_cpu0) #endif @@ -426,6 +518,7 @@ SYM_DATA(initial_vc_handler, .quad handle_vc_boot_ghcb) * reliably detect the end of the stack. */ SYM_DATA(initial_stack, .quad init_thread_union + THREAD_SIZE - FRAME_SIZE) +SYM_DATA(trampoline_lock, .quad 0); __FINITDATA __INIT @@ -660,6 +753,9 @@ SYM_DATA_END(level1_fixmap_pgt) SYM_DATA(early_gdt_descr, .word GDT_ENTRIES*8-1) SYM_DATA_LOCAL(early_gdt_descr_base, .quad INIT_PER_CPU_VAR(gdt_page)) + .align 16 +SYM_DATA(smpboot_control, .long 0) + .align 16 /* This must match the first entry in level2_kernel_pgt */ SYM_DATA(phys_base, .quad 0x0) diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index b18c1385e181..74c76c78f7d2 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -797,6 +797,16 @@ static int __init cpu_init_udelay(char *str) } early_param("cpu_init_udelay", cpu_init_udelay); +static bool do_parallel_bringup __ro_after_init = true; + +static int __init no_parallel_bringup(char *str) +{ + do_parallel_bringup = false; + + return 0; +} +early_param("no_parallel_bringup", no_parallel_bringup); + static void __init smp_quirk_init_udelay(void) { /* if cmdline changed it from default, leave it alone */ @@ -1084,8 +1094,6 @@ int common_cpu_up(unsigned int cpu, struct task_struct *idle) #ifdef CONFIG_X86_32 /* Stack for startup_32 can be just as for start_secondary onwards */ per_cpu(pcpu_hot.top_of_stack, cpu) = task_top_of_stack(idle); -#else - initial_gs = per_cpu_offset(cpu); #endif return 0; } @@ -1110,9 +1118,14 @@ static int do_boot_cpu(int apicid, int cpu, struct task_struct *idle, start_ip = real_mode_header->trampoline_start64; #endif idle->thread.sp = (unsigned long)task_pt_regs(idle); - early_gdt_descr.address = (unsigned long)get_cpu_gdt_rw(cpu); initial_code = (unsigned long)start_secondary; - initial_stack = idle->thread.sp; + + if (IS_ENABLED(CONFIG_X86_32)) { + early_gdt_descr.address = (unsigned long)get_cpu_gdt_rw(cpu); + initial_stack = idle->thread.sp; + } else if (!do_parallel_bringup) { + smpboot_control = STARTUP_SECONDARY | cpu; + } /* Enable the espfix hack for this CPU */ init_espfix_ap(cpu); @@ -1512,6 +1525,47 @@ void __init native_smp_prepare_cpus(unsigned int max_cpus) speculative_store_bypass_ht_init(); + /* + * We can do 64-bit AP bringup in parallel if the CPU reports + * its APIC ID in CPUID (either leaf 0x0B if we need the full + * APIC ID in X2APIC mode, or leaf 0x01 if 8 bits are + * sufficient). Otherwise it's too hard. And not for SEV-ES + * guests because they can't use CPUID that early. + */ + if (IS_ENABLED(CONFIG_X86_32) || boot_cpu_data.cpuid_level < 1 || + (x2apic_mode && boot_cpu_data.cpuid_level < 0xb) || + cc_platform_has(CC_ATTR_GUEST_STATE_ENCRYPT)) + do_parallel_bringup = false; + + if (do_parallel_bringup && x2apic_mode) { + unsigned int eax, ebx, ecx, edx; + + /* + * To support parallel bringup in x2apic mode, the AP will need + * to obtain its APIC ID from CPUID 0x0B, since CPUID 0x01 has + * only 8 bits. Check that it is present and seems correct. + */ + cpuid_count(0xb, 0, &eax, &ebx, &ecx, &edx); + + /* + * AMD says that if executed with an umimplemented level in + * ECX, then it will return all zeroes in EAX. Intel says it + * will return zeroes in both EAX and EBX. Checking only EAX + * should be sufficient. + */ + if (eax) { + pr_debug("Using CPUID 0xb for parallel CPU startup\n"); + smpboot_control = STARTUP_SECONDARY | STARTUP_APICID_CPUID_0B; + } else { + pr_info("Disabling parallel bringup because CPUID 0xb looks untrustworthy\n"); + do_parallel_bringup = false; + } + } else if (do_parallel_bringup) { + /* Without X2APIC, what's in CPUID 0x01 should suffice. */ + pr_debug("Using CPUID 0x1 for parallel CPU startup\n"); + smpboot_control = STARTUP_SECONDARY | STARTUP_APICID_CPUID_01; + } + snp_set_wakeup_secondary_cpu(); } diff --git a/arch/x86/realmode/init.c b/arch/x86/realmode/init.c index af565816d2ba..788e5559549f 100644 --- a/arch/x86/realmode/init.c +++ b/arch/x86/realmode/init.c @@ -154,6 +154,9 @@ static void __init setup_real_mode(void) trampoline_header->flags = 0; + trampoline_lock = &trampoline_header->lock; + *trampoline_lock = 0; + trampoline_pgd = (u64 *) __va(real_mode_header->trampoline_pgd); /* Map the real mode stub as virtual == physical */ diff --git a/arch/x86/realmode/rm/trampoline_64.S b/arch/x86/realmode/rm/trampoline_64.S index e38d61d6562e..49ebc1636ffd 100644 --- a/arch/x86/realmode/rm/trampoline_64.S +++ b/arch/x86/realmode/rm/trampoline_64.S @@ -49,6 +49,19 @@ SYM_CODE_START(trampoline_start) mov %ax, %es mov %ax, %ss + /* + * Make sure only one CPU fiddles with the realmode stack + */ +.Llock_rm: + btl $0, tr_lock + jnc 2f + pause + jmp .Llock_rm +2: + lock + btsl $0, tr_lock + jc .Llock_rm + # Setup stack movl $rm_stack_end, %esp @@ -241,6 +254,7 @@ SYM_DATA_START(trampoline_header) SYM_DATA(tr_efer, .space 8) SYM_DATA(tr_cr4, .space 4) SYM_DATA(tr_flags, .space 4) + SYM_DATA(tr_lock, .space 4) SYM_DATA_END(trampoline_header) #include "trampoline_common.S" From patchwork Thu Feb 23 19:11:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 61064 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp484963wrd; Thu, 23 Feb 2023 11:14:36 -0800 (PST) X-Google-Smtp-Source: AK7set/R0HESdnbZdfpF34K18vStzkJMBQVCY5CjO1oD86UEpaZcCJ6b1YNx5ETZwAZeX/HIX8cl X-Received: by 2002:a05:6402:35d0:b0:4ad:6f56:a362 with SMTP id z16-20020a05640235d000b004ad6f56a362mr15291677edc.4.1677179676795; Thu, 23 Feb 2023 11:14:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677179676; cv=none; d=google.com; s=arc-20160816; b=wXnPD3RnRv7Fh5cAyXut5YaFDo4PP+6xkKMq52Nuh5oBga4tjwTQxlP0+8VPQ0fHiL 34gL+IJmiLxYhmoeK+hDivpeQWHkeXgn9G+Kwu+gk2BiYKlMKCl0H1Da0NzLPfTqvXPh YQHtTX36QCZuHYh7yUZr1qNNO6ocBWTUYDD+o9K8W86/jEJEjIAtCF88s8kOv+ekT3cr 1tmGEKM7y3yAaAf9L7tEvx6wVI+BpGowKNJG1pUOayau6cgoAXLB1mkl6AuX0USGT+Q3 6T9393uco8q13i12av24aXMCgfFWmI9nUsF9006KnLWsecnO3bd/6xb5Yr8yclD7llyM TyIQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=0cOxOkpCD+vr2n7vBER3mbkwWwlvY2KHB3oBnHbXr/U=; b=JuXSGfXWlu6Qzx2SSPQzxyfCt+Ni+P6SNyYZfQvyMKikJcyolZ6Qj3fuVwMCqq8X1w g7h27SK6VvCo0+dUbVWh8LpwpALsN/JMglMLWTW85bFrDrpWaJPMalKQLZCUL852nMmm um5LdvibEI4X7IJj8VCpr2fokV2ceMg/lrdZvOKDpA5yy6ywd2pJPZEoUMJxtto9ioWP EhJ9AkC4WhV8KStuiXIXvNAGUOebiAAghWYitsVMVe8hr9M79gCCirHLFNOV3F2BsI9f dEUtZw4LAO5zxX0P+Yphms5Xeid9xgpIYwyvZ/9dU8RyLeU0oHbsNd2oAcyfaDn0qZfE +1CA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=ErntIUgA; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id r13-20020aa7d14d000000b0045fca739593si8948421edo.188.2023.02.23.11.14.12; Thu, 23 Feb 2023 11:14:36 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=ErntIUgA; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232537AbjBWTNe (ORCPT + 99 others); Thu, 23 Feb 2023 14:13:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60202 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232312AbjBWTNM (ORCPT ); Thu, 23 Feb 2023 14:13:12 -0500 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E3AA75BBA2 for ; Thu, 23 Feb 2023 11:12:22 -0800 (PST) Received: by mail-wr1-x42e.google.com with SMTP id i11so5397702wrp.5 for ; Thu, 23 Feb 2023 11:12:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0cOxOkpCD+vr2n7vBER3mbkwWwlvY2KHB3oBnHbXr/U=; b=ErntIUgAGEyRKXt4RMT66Pr0KsOFVw3vBflYFgfNggAyszQJLjPK+0lmvNVh2KXY7g wLgsBRVHOmbmRWbenYGMIxstY0+viUEq09wJYgsdb0oa72GZHZyRNICptybAqmyCeY/4 JSfHCnSohGbanQEtvvBqQOiPK3IGTQsbhuRvCIkXAtcI/l1zNfcYBX53buZ7xaGJm9Si tIx8lFSAKy+fGR9TzyT8jccKHgDF22drQHBcbFUumT4w4Cm+cupSv8AmEqAky0qy3J34 Dsakgq/7oGhgzOOA/T+PDYwDyEqdsiF1RVugLC3zmLNw4vIThARfDuXru7zf/cuVHvwj jtaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0cOxOkpCD+vr2n7vBER3mbkwWwlvY2KHB3oBnHbXr/U=; b=bTwqbJj+YmeJuObeSRUN9rJR/W5XRgu9lVl2mjgJup/WHUmZUJlXRlouOwkAWt2Ln1 1nheTiZhY8Y7U+MqZvlYRdHJ3dkhZjLmXxVKb0sh9YAnYbwO41Anz8bGuxxPoGCTZdcd y0LihAAbUSmzj+2lHlGlcSQuUZngWuMaJon5lfX1nkbACPN9HXERVGV+LRfXCzbQKp3f bb70MFsFWMplIGbzTNF1dAzwFlzMjhUi+8rETMx2iv6qLbp1huIM/koHCLuCYguUm8NA Hz98ZEpvxY47EwnzpO2mo8F8jKZn080ZliBk4W8TgS7u5Dmuar2UhVfdhStP5CqN11J0 253g== X-Gm-Message-State: AO0yUKX/wmPb1Vg1ZroOdRryIx+jiioRzb0h+2C/deCX8gRLrjFFV/Ya W0bFCCuXQ1kBYp/QFPiLC0qaHA== X-Received: by 2002:adf:ea11:0:b0:2c7:a39:7453 with SMTP id q17-20020adfea11000000b002c70a397453mr6424268wrm.54.1677179509837; Thu, 23 Feb 2023 11:11:49 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:5ee0:5af0:64bd:6198]) by smtp.gmail.com with ESMTPSA id b15-20020a5d4b8f000000b002c561805a4csm12957286wrt.45.2023.02.23.11.11.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Feb 2023 11:11:49 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse , Usama Arif Subject: [PATCH v11 07/12] x86/smpboot: Send INIT/SIPI/SIPI to secondary CPUs in parallel Date: Thu, 23 Feb 2023 19:11:35 +0000 Message-Id: <20230223191140.4155012-8-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230223191140.4155012-1-usama.arif@bytedance.com> References: <20230223191140.4155012-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758650357038102469?= X-GMAIL-MSGID: =?utf-8?q?1758650357038102469?= From: David Woodhouse When the APs can find their own APIC ID without assistance, perform the AP bringup in parallel. Register a CPUHP_BP_PARALLEL_DYN stage "x86/cpu:kick" which just calls do_boot_cpu() to deliver INIT/SIPI/SIPI to each AP in turn before the normal native_cpu_up() does the rest of the hand-holding. The APs will then take turns through the real mode code (which has its own bitlock for exclusion) until they make it to their own stack, then proceed through the first few lines of start_secondary() and execute these parts in parallel: start_secondary() -> cr4_init() -> (some 32-bit only stuff so not in the parallel cases) -> cpu_init_secondary() -> cpu_init_exception_handling() -> cpu_init() -> wait_for_master_cpu() At this point they wait for the BSP to set their bit in cpu_callout_mask (from do_wait_cpu_initialized()), and release them to continue through the rest of cpu_init() and beyond. This reduces the time taken for bringup on my 28-thread Haswell system from about 120ms to 80ms. On a socket 96-thread Skylake it takes the bringup time from 500ms to 100ms. There is more speedup to be had by doing the remaining parts in parallel too — especially notify_cpu_starting() in which the AP takes itself through all the stages from CPUHP_BRINGUP_CPU to CPUHP_ONLINE. But those require careful auditing to ensure they are reentrant, before we can go that far. Signed-off-by: David Woodhouse Signed-off-by: Usama Arif Tested-by: Paul E. McKenney Tested-by: Kim Phillips Tested-by: Oleksandr Natalenko Tested-by: Michael Kelley --- arch/x86/kernel/smpboot.c | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 74c76c78f7d2..85ce6a8978ff 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -57,6 +57,7 @@ #include #include #include +#include #include #include @@ -1325,9 +1326,12 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) { int ret; - ret = do_cpu_up(cpu, tidle); - if (ret) - return ret; + /* If parallel AP bringup isn't enabled, perform the first steps now. */ + if (!do_parallel_bringup) { + ret = do_cpu_up(cpu, tidle); + if (ret) + return ret; + } ret = do_wait_cpu_initialized(cpu); if (ret) @@ -1349,6 +1353,12 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) return ret; } +/* Bringup step one: Send INIT/SIPI to the target AP */ +static int native_cpu_kick(unsigned int cpu) +{ + return do_cpu_up(cpu, idle_thread_get(cpu)); +} + /** * arch_disable_smp_support() - disables SMP support for x86 at runtime */ @@ -1566,6 +1576,11 @@ void __init native_smp_prepare_cpus(unsigned int max_cpus) smpboot_control = STARTUP_SECONDARY | STARTUP_APICID_CPUID_01; } + if (do_parallel_bringup) { + cpuhp_setup_state_nocalls(CPUHP_BP_PARALLEL_DYN, "x86/cpu:kick", + native_cpu_kick, NULL); + } + snp_set_wakeup_secondary_cpu(); } From patchwork Thu Feb 23 19:11:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 61069 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp495655wrd; Thu, 23 Feb 2023 11:38:17 -0800 (PST) X-Google-Smtp-Source: AK7set/zpyZj8OmU7h3O5qvjphWtz3LYa3WxSnxPv0lASbSM5zhNqX6oEDKwe4tBtIx6AI5UEmi2 X-Received: by 2002:a05:6a00:1804:b0:5b2:5466:34e1 with SMTP id y4-20020a056a00180400b005b2546634e1mr12180820pfa.3.1677181096997; Thu, 23 Feb 2023 11:38:16 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677181096; cv=none; d=google.com; s=arc-20160816; b=oEv18v8vOWmuowfJuaKhCIvSz70Cw/NtolKGVcfZmucJoU66vocQhr35q+7dUFbRJW XsHGt5/7gAvNSMIiEG7GGO4DlUX9kdX2SC9dYbgxtHVeSUwaSl38jPNvEL93ZxvK9uW0 dXjhmN+fGT8NRc1ASaqcWb8OJrx/AoeBQyLxXaRKqqqnuFHEdBNFsmUgNtIRSnz0lJSR AFdNJWzOfj0X0uflzs1BBqNUUDnBZFwNHurBZzRFBmMldquwd6fdsFDhNxNYnPdpM4fK AIWaNFn/f5eg+NHgwwuyOPIMefk8g7Hl0QpQINM5MVLFl0Uh2sQyoppRkHbvbJNJ8a5k DNwg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=pCJsFyg6Hssbg4gqmP6Vz6XI/w3eaP9DOOa8tGXjhs8=; b=ncxPzVWdruWLFYKZZSrgobQPfBevkATL+1Tg5K4l/wH5psYZKYMls8uv8oTDAri2DJ LI1QPOUaFveKPSgXMeNWk+SqisxnmRknxwjjuNgLzGnKloJPqLz6gftYg4iiLRZIIc5f OZFG0FQXQIjIUP4PoEH1zZR0q3lW84IP+68WG6WOdsJ54W3ER26l6u5bnmp9+gfSfj9Y /pT39sOMOZWSmLb7H86JJmUHsCGiJEB7Qm60ISdOH7UMPcJ4iTvZ6rp7LANHG7KS1RCd 0pyP6/AVURrlHvFtCl484S1YivT7V8Z6f5PG1xP0SiRiqF6EEhVZm3w5hNrJtZovX1ZD YmgA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=YiASnhnJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t76-20020a63784f000000b004fd21719183si5307407pgc.605.2023.02.23.11.38.03; Thu, 23 Feb 2023 11:38:16 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=YiASnhnJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232690AbjBWTNw (ORCPT + 99 others); Thu, 23 Feb 2023 14:13:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60660 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232149AbjBWTNO (ORCPT ); Thu, 23 Feb 2023 14:13:14 -0500 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E3B775BBBF for ; Thu, 23 Feb 2023 11:12:22 -0800 (PST) Received: by mail-wr1-x435.google.com with SMTP id c12so11561614wrw.1 for ; Thu, 23 Feb 2023 11:12:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pCJsFyg6Hssbg4gqmP6Vz6XI/w3eaP9DOOa8tGXjhs8=; b=YiASnhnJ1jtnV2xO1mohlm7SFA5yOYPv34i9HrdBCz2KDwzP75JhqoFEnQdw9/t3+G jajzF89nU5uTbtXyrxlV5+Pu0gaGY4jxhkQFKGjMNzuf+0KNRPGBtCJtgX/2zEr24yBp fmlJh20Iq0/Qvkb41zSGrtA8/oosM5pWcSL4tqpFDbGP0PuqoDAn/gr4pUV0IcwsyZxN z3MDIHglXH8FUKmQFhrTeK8oPi4hVkEQrzEkDnsPEr1iaCGyvDEBcb6JzCH2PSMGt5zq Z9gg36jk99xbKahnESEWvrzcFkNL9RP2E6RHuW6/T9D6e0ewN+trOfkGmbv5XOE+4DTD 1XuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pCJsFyg6Hssbg4gqmP6Vz6XI/w3eaP9DOOa8tGXjhs8=; b=OUjgT09wSF+4P/qcYJA7ge13mf8TAuTdXJUdD701MId7w8x0wGlM1dTlTIy9Q8Kbgq I1TDCH5yxDIKGa86+L2PLghGMx81dty1gru0KJZtQPApVOPV8xfs91V21G4TMJhyoxuN EgaBUW9OdfoqD/aiyMOt28bSSwUooU/H8CzdC/5XZyJLkNKFCPLyI0ffQZHiEpK0OpWk 2kXmSFNQMJqXp80Zzzdl9vAV8VTClZsdArgAZOjHWiCD970idVhDRAKD4vJLitKwhl/u 5jWRIjjvvdNE7aKQ7CTi4B0rvk1HdZipp9nXHe0A/ziZyWL27sqbTHPRqmgBfTgJ48vi QJjQ== X-Gm-Message-State: AO0yUKVecCJPOLvI8OpYX96BQTlEZjpeY24iYtrgiztjyeDEWGGZvBCx 7+HwDqilzRdskCBrW4X522jioA== X-Received: by 2002:a5d:5383:0:b0:2c7:169b:c57d with SMTP id d3-20020a5d5383000000b002c7169bc57dmr1544252wrv.18.1677179510696; Thu, 23 Feb 2023 11:11:50 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:5ee0:5af0:64bd:6198]) by smtp.gmail.com with ESMTPSA id b15-20020a5d4b8f000000b002c561805a4csm12957286wrt.45.2023.02.23.11.11.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Feb 2023 11:11:50 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse , Usama Arif Subject: [PATCH v11 08/12] x86/smpboot: Serialize topology updates for secondary bringup Date: Thu, 23 Feb 2023 19:11:36 +0000 Message-Id: <20230223191140.4155012-9-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230223191140.4155012-1-usama.arif@bytedance.com> References: <20230223191140.4155012-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758651846190397497?= X-GMAIL-MSGID: =?utf-8?q?1758651846190397497?= From: David Woodhouse The toplogy update is performed by the AP via smp_callin() after the BSP has called do_wait_cpu_initialized(), setting the AP's bit in cpu_callout_mask to allow it to proceed. In preparation to enable further parallelism of AP bringup, add locking to serialize the update even if multiple APs are (in future) permitted to proceed through the next stages of bringup in parallel. Without such ordering (and with that future extra parallelism), confusion ensues: [ 1.360149] x86: Booting SMP configuration: [ 1.360221] .... node #0, CPUs: #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15 #16 #17 #18 #19 #20 #21 #22 #23 [ 1.366225] .... node #1, CPUs: #24 #25 #26 #27 #28 #29 #30 #31 #32 #33 #34 #35 #36 #37 #38 #39 #40 #41 #42 #43 #44 #45 #46 #47 [ 1.370219] .... node #0, CPUs: #48 #49 #50 #51 #52 #53 #54 #55 #56 #57 #58 #59 #60 #61 #62 #63 #64 #65 #66 #67 #68 #69 #70 #71 [ 1.378226] .... node #1, CPUs: #72 #73 #74 #75 #76 #77 #78 #79 #80 #81 #82 #83 #84 #85 #86 #87 #88 #89 #90 #91 #92 #93 #94 #95 [ 1.382037] Brought 96 CPUs to x86/cpu:kick in 72232606 cycles [ 0.104104] smpboot: CPU 26 Converting physical 0 to logical die 1 [ 0.104104] smpboot: CPU 27 Converting physical 1 to logical package 2 [ 0.104104] smpboot: CPU 24 Converting physical 1 to logical package 3 [ 0.104104] smpboot: CPU 27 Converting physical 0 to logical die 2 [ 0.104104] smpboot: CPU 25 Converting physical 1 to logical package 4 [ 1.385609] Brought 96 CPUs to x86/cpu:wait-init in 9269218 cycles [ 1.395285] Brought CPUs online in 28930764 cycles [ 1.395469] smp: Brought up 2 nodes, 96 CPUs [ 1.395689] smpboot: Max logical packages: 2 [ 1.396222] smpboot: Total of 96 processors activated (576000.00 BogoMIPS) Signed-off-by: David Woodhouse Signed-off-by: Usama Arif Tested-by: Paul E. McKenney Tested-by: Kim Phillips Tested-by: Oleksandr Natalenko --- arch/x86/include/asm/smp.h | 4 +- arch/x86/include/asm/topology.h | 2 - arch/x86/kernel/cpu/common.c | 6 +-- arch/x86/kernel/smpboot.c | 73 ++++++++++++++++++++------------- arch/x86/xen/smp_pv.c | 4 +- 5 files changed, 48 insertions(+), 41 deletions(-) diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h index 33c0d5fd8af6..b4b29e052b6e 100644 --- a/arch/x86/include/asm/smp.h +++ b/arch/x86/include/asm/smp.h @@ -48,8 +48,6 @@ struct smp_ops { }; /* Globals due to paravirt */ -extern void set_cpu_sibling_map(int cpu); - #ifdef CONFIG_SMP extern struct smp_ops smp_ops; @@ -137,7 +135,7 @@ void native_send_call_func_single_ipi(int cpu); void x86_idle_thread_init(unsigned int cpu, struct task_struct *idle); void smp_store_boot_cpu_info(void); -void smp_store_cpu_info(int id); +void smp_store_cpu_info(int id, bool force_single_core); asmlinkage __visible void smp_reboot_interrupt(void); __visible void smp_reschedule_interrupt(struct pt_regs *regs); diff --git a/arch/x86/include/asm/topology.h b/arch/x86/include/asm/topology.h index 458c891a8273..4bccbd949a99 100644 --- a/arch/x86/include/asm/topology.h +++ b/arch/x86/include/asm/topology.h @@ -136,8 +136,6 @@ static inline int topology_max_smt_threads(void) return __max_smt_threads; } -int topology_update_package_map(unsigned int apicid, unsigned int cpu); -int topology_update_die_map(unsigned int dieid, unsigned int cpu); int topology_phys_to_logical_pkg(unsigned int pkg); int topology_phys_to_logical_die(unsigned int die, unsigned int cpu); bool topology_is_primary_thread(unsigned int cpu); diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index f3cc7699e1e1..06d7f9e55d45 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -1771,7 +1771,7 @@ static void generic_identify(struct cpuinfo_x86 *c) * Validate that ACPI/mptables have the same information about the * effective APIC id and update the package map. */ -static void validate_apic_and_package_id(struct cpuinfo_x86 *c) +static void validate_apic_id(struct cpuinfo_x86 *c) { #ifdef CONFIG_SMP unsigned int apicid, cpu = smp_processor_id(); @@ -1782,8 +1782,6 @@ static void validate_apic_and_package_id(struct cpuinfo_x86 *c) pr_err(FW_BUG "CPU%u: APIC id mismatch. Firmware: %x APIC: %x\n", cpu, apicid, c->initial_apicid); } - BUG_ON(topology_update_package_map(c->phys_proc_id, cpu)); - BUG_ON(topology_update_die_map(c->cpu_die_id, cpu)); #else c->logical_proc_id = 0; #endif @@ -1974,7 +1972,7 @@ void identify_secondary_cpu(struct cpuinfo_x86 *c) #ifdef CONFIG_X86_32 enable_sep_cpu(); #endif - validate_apic_and_package_id(c); + validate_apic_id(c); x86_spec_ctrl_setup_ap(); update_srbds_msr(); diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 85ce6a8978ff..69ef0860feea 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -180,16 +180,12 @@ static void smp_callin(void) apic_ap_setup(); /* - * Save our processor parameters. Note: this information - * is needed for clock calibration. - */ - smp_store_cpu_info(cpuid); - - /* + * Save our processor parameters and update topology. + * Note: this information is needed for clock calibration. * The topology information must be up to date before * calibrate_delay() and notify_cpu_starting(). */ - set_cpu_sibling_map(raw_smp_processor_id()); + smp_store_cpu_info(cpuid, false); ap_init_aperfmperf(); @@ -243,6 +239,12 @@ static void notrace start_secondary(void *unused) * its bit in cpu_callout_mask to release it. */ cpu_init_secondary(); + + /* + * Even though notify_cpu_starting() will do this, it does so too late + * as the AP may already have triggered lockdep splats by then. See + * commit 29368e093 ("x86/smpboot: Move rcu_cpu_starting() earlier"). + */ rcu_cpu_starting(raw_smp_processor_id()); x86_cpuinit.early_percpu_clock_init(); @@ -351,7 +353,7 @@ EXPORT_SYMBOL(topology_phys_to_logical_die); * @pkg: The physical package id as retrieved via CPUID * @cpu: The cpu for which this is updated */ -int topology_update_package_map(unsigned int pkg, unsigned int cpu) +static int topology_update_package_map(unsigned int pkg, unsigned int cpu) { int new; @@ -374,7 +376,7 @@ int topology_update_package_map(unsigned int pkg, unsigned int cpu) * @die: The die id as retrieved via CPUID * @cpu: The cpu for which this is updated */ -int topology_update_die_map(unsigned int die, unsigned int cpu) +static int topology_update_die_map(unsigned int die, unsigned int cpu) { int new; @@ -405,25 +407,7 @@ void __init smp_store_boot_cpu_info(void) c->initialized = true; } -/* - * The bootstrap kernel entry code has set these up. Save them for - * a given CPU - */ -void smp_store_cpu_info(int id) -{ - struct cpuinfo_x86 *c = &cpu_data(id); - - /* Copy boot_cpu_data only on the first bringup */ - if (!c->initialized) - *c = boot_cpu_data; - c->cpu_index = id; - /* - * During boot time, CPU0 has this setup already. Save the info when - * bringing up AP or offlined CPU0. - */ - identify_secondary_cpu(c); - c->initialized = true; -} +static arch_spinlock_t topology_lock = __ARCH_SPIN_LOCK_UNLOCKED; static bool topology_same_node(struct cpuinfo_x86 *c, struct cpuinfo_x86 *o) @@ -629,7 +613,7 @@ static struct sched_domain_topology_level x86_topology[] = { */ static bool x86_has_numa_in_package; -void set_cpu_sibling_map(int cpu) +static void set_cpu_sibling_map(int cpu) { bool has_smt = smp_num_siblings > 1; bool has_mp = has_smt || boot_cpu_data.x86_max_cores > 1; @@ -708,6 +692,37 @@ void set_cpu_sibling_map(int cpu) } } +/* + * The bootstrap kernel entry code has set these up. Save them for + * a given CPU + */ +void smp_store_cpu_info(int id, bool force_single_core) +{ + struct cpuinfo_x86 *c = &cpu_data(id); + + /* Copy boot_cpu_data only on the first bringup */ + if (!c->initialized) + *c = boot_cpu_data; + c->cpu_index = id; + /* + * During boot time, CPU0 has this setup already. Save the info when + * bringing up AP or offlined CPU0. + */ + identify_secondary_cpu(c); + + arch_spin_lock(&topology_lock); + BUG_ON(topology_update_package_map(c->phys_proc_id, id)); + BUG_ON(topology_update_die_map(c->cpu_die_id, id)); + c->initialized = true; + + /* For Xen PV */ + if (force_single_core) + c->x86_max_cores = 1; + + set_cpu_sibling_map(id); + arch_spin_unlock(&topology_lock); +} + /* maps the cpu to the sched domain representing multi-core */ const struct cpumask *cpu_coregroup_mask(int cpu) { diff --git a/arch/x86/xen/smp_pv.c b/arch/x86/xen/smp_pv.c index 6175f2c5c822..09f94f940689 100644 --- a/arch/x86/xen/smp_pv.c +++ b/arch/x86/xen/smp_pv.c @@ -71,9 +71,7 @@ static void cpu_bringup(void) xen_enable_syscall(); } cpu = smp_processor_id(); - smp_store_cpu_info(cpu); - cpu_data(cpu).x86_max_cores = 1; - set_cpu_sibling_map(cpu); + smp_store_cpu_info(cpu, true); speculative_store_bypass_ht_init(); From patchwork Thu Feb 23 19:11:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 61066 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp485574wrd; Thu, 23 Feb 2023 11:15:53 -0800 (PST) X-Google-Smtp-Source: AK7set9fupYXmBFxogMoENhVDhe8wRSxNICC0Kv3wyaJVGtWTkeSEVRt/AdftQZSx3YN20g5ybsq X-Received: by 2002:a17:906:5d:b0:88c:f97c:7c87 with SMTP id 29-20020a170906005d00b0088cf97c7c87mr19505332ejg.2.1677179752916; Thu, 23 Feb 2023 11:15:52 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677179752; cv=none; d=google.com; s=arc-20160816; b=VvbyjyG2XC+wNkBJUYaGgLS2iu3wrgnv5phg1wDVkkdZiM9wCdrd1U7PYITYN+tyc/ 7vJJX/ujrI46uVW8qtZthHF6LRKsiej43dF6cI4TBpg2otaKFFKAAUXuuIVkfHQfdhJG G/nc9b3MWefYAJRbd6/lcyNjxyxlGc9mH0hV9jrz2YoBwVWelXnKV3FK2YvJrcB9u3b2 i/VYg440qTKjBRcIJ03IIoRDUAlHmkOUGto6yMEUUm9W0foibULV2BxTx0F4fD91BaQa QT+0OdK6qq7/y/hs3BkwvLxJx3aLnEQePeo5eJEDCr8I0VX/S2CMCisCabIh8UldUqdq pQWA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=wL4H6dkwiqQh7dcmIrLqt3HZrti5VJpIfMc44frmzHc=; b=cfgI5/p1o4ZPPIoOdkm+UtpK8aGNw/GNT1itiAc94XIGhF2aSuHAuG5K9NIM6rT9cL 0zPzcoRtnjmgrMbdBajl77Fbj0zukryF/PCszdHDiNK/nYaTugNa7Q0kzXeb49fPNoGy IQnVMdJWnFAVnPGZu3UzvH6lIZzzQwi1rGEhOWgSELVjD3+UGuN3DnBvKCJodHZYz15X qpOXD3HJvl/1NZILyVwCHGMF7VILE+Ji/AlNUdcYJcHIk4PtloQ7Zxo8w5GgCFrOKpD7 dKb5XE3F8dhFryx6Pf4zB47NeLecE8VA/gafIZ7y2mr6XoGxLm32jcKDfVdwTYbdB+HQ rO4w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=Z+5BzDDr; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ep5-20020a1709069b4500b008c9851d114asi15785342ejc.607.2023.02.23.11.15.29; Thu, 23 Feb 2023 11:15:52 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=Z+5BzDDr; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232670AbjBWTNr (ORCPT + 99 others); Thu, 23 Feb 2023 14:13:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60658 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231937AbjBWTNN (ORCPT ); Thu, 23 Feb 2023 14:13:13 -0500 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3B6155BBBB for ; Thu, 23 Feb 2023 11:12:23 -0800 (PST) Received: by mail-wm1-x32e.google.com with SMTP id l7-20020a05600c4f0700b003e79fa98ce1so194802wmq.2 for ; Thu, 23 Feb 2023 11:12:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wL4H6dkwiqQh7dcmIrLqt3HZrti5VJpIfMc44frmzHc=; b=Z+5BzDDrPRuwK1aoLnVaWnL5dhdfm1qA4XC8szYEgLwxcNNkKR7N1jrdZUW8KEWp0r ooYGJUtsbba00Y/hcT4cdNn2uc+6w59DOT51XHq8wkNlj1qgwWVTvjDjUiFC3ZFQToAM Ms/4rjIXTj2kdOLZklcn7MemtWoO5VOYBGMjSb5QcoQKZg2bNnOZRgeY3otdukJX0KNl BffMhFJReoM18BJOkeYVYcZVKuaGFxn5NJNeSccfuIgvSMAAOL6Gq4tvOWjQeorYSe8d JHZI4Woj5seaPOAbsH+RKe4gXlSOJ5xsX4e0EjXjAxBad9J0hkle5NxDW9QiNVwK7Dd4 3Jng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wL4H6dkwiqQh7dcmIrLqt3HZrti5VJpIfMc44frmzHc=; b=WXY848qiE8f7JCNeCRy2fzHG5k29GatnNIrWmU2HMBeMMWz/6z+mR3Q8ergrhYIOGa D91NihJx5/9vzqfPwKFZEcZViAH/RYVI0NtNGHCDc8ayDBOvXVJN7Zop/TmTIIT5AaZL YVq9em0MR5o9J3C9V+CI3iaPAL8TYXd/62XqySimmdZBhKP7HW2IW1i+ZaQk0O88WnEq afsIpqtKC1XiUEA7E47flV+PHfJYUfvKvVfmAJFyEC7F9SAhe9tB+CWdTCxOVTav4qWF lZjNu8EE7ryKTRdUMirvvQIvGzcc8G3ik9LsHO2RmIPilmPKuXPKqpTbCywd9h8Q5KId iriQ== X-Gm-Message-State: AO0yUKV7+vyb5Km3iLoz7MiY0QPVjNTGOxQjbqgBB84CfK+so+ZBfqo+ zibZ2maOFLIYHnMIsAfBSjnMrw== X-Received: by 2002:a05:600c:3b9f:b0:3e9:9fc5:2c6d with SMTP id n31-20020a05600c3b9f00b003e99fc52c6dmr4257178wms.14.1677179511475; Thu, 23 Feb 2023 11:11:51 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:5ee0:5af0:64bd:6198]) by smtp.gmail.com with ESMTPSA id b15-20020a5d4b8f000000b002c561805a4csm12957286wrt.45.2023.02.23.11.11.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Feb 2023 11:11:51 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse , Usama Arif Subject: [PATCH v11 09/12] x86/smpboot: Remove initial_stack on 64-bit Date: Thu, 23 Feb 2023 19:11:37 +0000 Message-Id: <20230223191140.4155012-10-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230223191140.4155012-1-usama.arif@bytedance.com> References: <20230223191140.4155012-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758650436140892984?= X-GMAIL-MSGID: =?utf-8?q?1758650436140892984?= From: Brian Gerst Load RSP from current_task->thread.sp instead. Signed-off-by: Brian Gerst Reviewed-by: David Woodhouse Signed-off-by: David Woodhouse Tested-by: Usama Arif Signed-off-by: Usama Arif --- arch/x86/include/asm/processor.h | 6 +++++- arch/x86/kernel/acpi/sleep.c | 2 +- arch/x86/kernel/head_64.S | 35 ++++++++++++++++++-------------- arch/x86/xen/xen-head.S | 2 +- 4 files changed, 27 insertions(+), 18 deletions(-) diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h index 4e35c66edeb7..9c4a5c4d46c1 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -648,7 +648,11 @@ static inline void spin_lock_prefetch(const void *x) #define KSTK_ESP(task) (task_pt_regs(task)->sp) #else -#define INIT_THREAD { } +extern unsigned long __end_init_task[]; + +#define INIT_THREAD { \ + .sp = (unsigned long)&__end_init_task - PTREGS_SIZE, \ +} extern unsigned long KSTK_ESP(struct task_struct *task); diff --git a/arch/x86/kernel/acpi/sleep.c b/arch/x86/kernel/acpi/sleep.c index 47e75c056cb5..008fda8b1982 100644 --- a/arch/x86/kernel/acpi/sleep.c +++ b/arch/x86/kernel/acpi/sleep.c @@ -113,7 +113,7 @@ int x86_acpi_suspend_lowlevel(void) saved_magic = 0x12345678; #else /* CONFIG_64BIT */ #ifdef CONFIG_SMP - initial_stack = (unsigned long)temp_stack + sizeof(temp_stack); + current->thread.sp = (unsigned long)temp_stack + sizeof(temp_stack); early_gdt_descr.address = (unsigned long)get_cpu_gdt_rw(smp_processor_id()); initial_gs = per_cpu_offset(smp_processor_id()); diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index c32e5b06a9ce..f7905ba4b992 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -62,8 +62,8 @@ SYM_CODE_START_NOALIGN(startup_64) * tables and then reload them. */ - /* Set up the stack for verify_cpu(), similar to initial_stack below */ - leaq (__end_init_task - FRAME_SIZE)(%rip), %rsp + /* Set up the stack for verify_cpu() */ + leaq (__end_init_task - PTREGS_SIZE)(%rip), %rsp leaq _text(%rip), %rdi @@ -245,11 +245,11 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) #ifdef CONFIG_SMP /* * Is this the boot CPU coming up? If so everything is available - * in initial_gs, initial_stack and early_gdt_descr. + * in initial_gs and early_gdt_descr. */ movl smpboot_control(%rip), %edx testl $STARTUP_SECONDARY, %edx - jz .Lsetup_cpu + jz .Linit_cpu0_data /* * For parallel boot, the APIC ID is retrieved from CPUID, and then @@ -302,6 +302,10 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) hlt jmp 1b +.Linit_cpu0_data: + movq __per_cpu_offset(%rip), %rdx + jmp .Lsetup_cpu + .Linit_cpu_data: /* Get the per cpu offset for the given CPU# which is in ECX */ leaq __per_cpu_offset(%rip), %rbx @@ -314,13 +318,21 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) addq %rbx, %rcx movq %rcx, early_gdt_descr_base(%rip) - /* Find the idle task stack */ - movq pcpu_hot + X86_current_task(%rbx), %rcx - movq TASK_threadsp(%rcx), %rcx - movq %rcx, initial_stack(%rip) + movq %rbx, %rdx +#else + xorl %edx, %edx #endif /* CONFIG_SMP */ .Lsetup_cpu: + /* + * Setup a boot time stack - Any secondary CPU will have lost its stack + * by now because the cr3-switch above unmaps the real-mode stack + * + * RDX contains the per-cpu offset + */ + movq pcpu_hot + X86_current_task(%rdx), %rax + movq TASK_threadsp(%rax), %rsp + /* * We must switch to a new descriptor in kernel space for the GDT * because soon the kernel won't have access anymore to the userspace @@ -355,12 +367,6 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) movl initial_gs+4(%rip),%edx wrmsr - /* - * Setup a boot time stack - Any secondary CPU will have lost its stack - * by now because the cr3-switch above unmaps the real-mode stack - */ - movq initial_stack(%rip), %rsp - /* Drop the realmode protection. For the boot CPU the pointer is NULL! */ movq trampoline_lock(%rip), %rax testq %rax, %rax @@ -517,7 +523,6 @@ SYM_DATA(initial_vc_handler, .quad handle_vc_boot_ghcb) * The FRAME_SIZE gap is a convention which helps the in-kernel unwinder * reliably detect the end of the stack. */ -SYM_DATA(initial_stack, .quad init_thread_union + THREAD_SIZE - FRAME_SIZE) SYM_DATA(trampoline_lock, .quad 0); __FINITDATA diff --git a/arch/x86/xen/xen-head.S b/arch/x86/xen/xen-head.S index ffaa62167f6e..6bd391476656 100644 --- a/arch/x86/xen/xen-head.S +++ b/arch/x86/xen/xen-head.S @@ -49,7 +49,7 @@ SYM_CODE_START(startup_xen) ANNOTATE_NOENDBR cld - mov initial_stack(%rip), %rsp + leaq (__end_init_task - PTREGS_SIZE)(%rip), %rsp /* Set up %gs. * From patchwork Thu Feb 23 19:11:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 61067 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp485688wrd; Thu, 23 Feb 2023 11:16:09 -0800 (PST) X-Google-Smtp-Source: AK7set87ShHHLi7/EjPYDk/GvcNuwrFMfuT43FTCLx3WvKdkSI+cVD1zdZAY+l6Ej/em0rKOlAd7 X-Received: by 2002:a17:906:196:b0:88f:1255:59c with SMTP id 22-20020a170906019600b0088f1255059cmr18888451ejb.1.1677179769268; Thu, 23 Feb 2023 11:16:09 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677179769; cv=none; d=google.com; s=arc-20160816; b=aqX7Es39Lrcta1aYkDCEp7H3zdokwgh4bzxyMogGmwT3phiyqAVwuMcMiQReA6J8il 6tTfr0f9ctlSlIqmhVm+0+0aeXXAEkCsbCQ9+Dp/a85g8IexKA0/N5GMRBH8WxveT1NA ddeX+kYn2kczw43AJ/hCaRZfkDlzFE1ypQ7G/rehN/Tj6+IEwcKcbYzyzEoXyPX4q1fA 2I++xuePPIjs/CmPalvF69aScXFvkOOQzUV/ERCE8ha1GSZc9SHaaCy5X43qfdLyJDfB gk5/rI9GFWPSxoABLQG1y0LZRYWGH2ctHJbyNEbLZ1+kRF3q4UZ9sVUb8Mtk8YJpZj0H WqPQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Ydn+N8Ta2/l5mmEEZ9zQwSQU4fOYRVeCJFh5IKZvP2o=; b=tohMrvP25Uk4jH6ptlvFHRu+aU3reAjF1QNLaiKTF/rGWcmzQQqy+T1Eq8Ii56C7Vj FHguSM/htx9J60AH5c2yyDFu2JSw1bLOi/o/8LfvhTHH5a2bxOy9EZhkCUlavXdxKqKx ZlucNYGOkrnIMlW+4TP+Ri0o0xn6s2A88p4U54Cj4dq/vys4+4iS1vYoIiATbAiEj+D6 1pMPmWbaA8L/Qqy0LKuSetbC/Ax2/Fgz8Ky/WFNnagfltewIYjNQb6xXhDxMmW/b2+bE vlnFbgkqBBpi/YgQ4AzX75xS8NaJc/q0IjBwx8E52CfTWSbPEwahPJWEoBZ0jVrfgGL5 CYZw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=XMP+3c7u; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ma25-20020a1709079f1900b008e2bd861068si7700744ejc.697.2023.02.23.11.15.45; Thu, 23 Feb 2023 11:16:09 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=XMP+3c7u; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232627AbjBWTNl (ORCPT + 99 others); Thu, 23 Feb 2023 14:13:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60730 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231882AbjBWTNN (ORCPT ); Thu, 23 Feb 2023 14:13:13 -0500 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 33E385C15F for ; Thu, 23 Feb 2023 11:12:27 -0800 (PST) Received: by mail-wr1-x433.google.com with SMTP id i11so5397837wrp.5 for ; Thu, 23 Feb 2023 11:12:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Ydn+N8Ta2/l5mmEEZ9zQwSQU4fOYRVeCJFh5IKZvP2o=; b=XMP+3c7uJ0e3rUuqE3xP3n2bcdkHaye4xq0doQzwnAtiZpHBt0EhXEMjBGyfdkJC8L XdmwsDBHG5CJlyC7g/3t/QfN3zXpeVAwxeCVNr5p+a6HiWka/VvuOsIibLUKaMN3eYz7 A+Z5tUhdCDd9x2Tetmn6oO8DU3AUjzxV5kNC4yA1qaEmjknk3zJoKfv6pCtCJ6dG25Y1 LzKe2S/ieY3a+EZQCkkI2msPad7Kphbtlfo/Aum/h/WBHYqyeLQR4CUZRxjypWZcVWwl fCaY119EFubhBbWFfX3bJEW3UWJotvmlglZvJHStOoUXGebQaJZWF7P0X7ZOnIZvFLJT 0DMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ydn+N8Ta2/l5mmEEZ9zQwSQU4fOYRVeCJFh5IKZvP2o=; b=H4xGtR9n6ZGLfc9yktblREtZk58dFm2KOGDSgqwIxG/KfJtfjMhAgf2Vo0E4ovkgJi dJw/dCBaIIl6RsfkIATdV8diIXYX8H5QTdUxikJ8q9kteUYZLEsBygf6XhycrDu9usmY cSK4j5bl1i8YMnLkZWsfPgGC/hMLgKf+nJ839/GdYo4rabDgFxHIbSPU4mYI/3VxAWmp GJkjg5+QXKV0s6x4INeYZfy62peCMckGhN9e1L9jKRDpavGllWzT/eWibO4WHoFaB0Yq E4vh6NHUF6s7MZvKaUao9voXLoB2CO9Cw/wNq9bf+iaocslJsyPcxaN9LrECzCwL2NBN SdcQ== X-Gm-Message-State: AO0yUKXdA5ARot62v9msh6M7181Nukpvw75jy4zatYEpJ79O7MjvP4nh la4TL57QRpk1bToZHghSC85fzw== X-Received: by 2002:a05:6000:1007:b0:2c3:da8a:192 with SMTP id a7-20020a056000100700b002c3da8a0192mr10542698wrx.15.1677179512258; Thu, 23 Feb 2023 11:11:52 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:5ee0:5af0:64bd:6198]) by smtp.gmail.com with ESMTPSA id b15-20020a5d4b8f000000b002c561805a4csm12957286wrt.45.2023.02.23.11.11.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Feb 2023 11:11:51 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse , Usama Arif Subject: [PATCH v11 10/12] x86/smpboot: Remove early_gdt_descr on 64-bit Date: Thu, 23 Feb 2023 19:11:38 +0000 Message-Id: <20230223191140.4155012-11-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230223191140.4155012-1-usama.arif@bytedance.com> References: <20230223191140.4155012-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758650453724845264?= X-GMAIL-MSGID: =?utf-8?q?1758650453724845264?= From: Brian Gerst Build the GDT descriptor on the stack instead. Signed-off-by: Brian Gerst Reviewed-by: David Woodhouse Signed-off-by: David Woodhouse Tested-by: Usama Arif Signed-off-by: Usama Arif --- arch/x86/kernel/acpi/sleep.c | 2 -- arch/x86/kernel/head_64.S | 19 +++++++------------ 2 files changed, 7 insertions(+), 14 deletions(-) diff --git a/arch/x86/kernel/acpi/sleep.c b/arch/x86/kernel/acpi/sleep.c index 008fda8b1982..6538ddb55f28 100644 --- a/arch/x86/kernel/acpi/sleep.c +++ b/arch/x86/kernel/acpi/sleep.c @@ -114,8 +114,6 @@ int x86_acpi_suspend_lowlevel(void) #else /* CONFIG_64BIT */ #ifdef CONFIG_SMP current->thread.sp = (unsigned long)temp_stack + sizeof(temp_stack); - early_gdt_descr.address = - (unsigned long)get_cpu_gdt_rw(smp_processor_id()); initial_gs = per_cpu_offset(smp_processor_id()); /* Force the startup into boot mode */ saved_smpboot_ctrl = xchg(&smpboot_control, 0); diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index f7905ba4b992..0dd57d573a0e 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -245,7 +245,7 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) #ifdef CONFIG_SMP /* * Is this the boot CPU coming up? If so everything is available - * in initial_gs and early_gdt_descr. + * in initial_gs. */ movl smpboot_control(%rip), %edx testl $STARTUP_SECONDARY, %edx @@ -313,11 +313,6 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) /* Save it for GS BASE setup */ movq %rbx, initial_gs(%rip) - /* Calculate the GDT address */ - movq $gdt_page, %rcx - addq %rbx, %rcx - movq %rcx, early_gdt_descr_base(%rip) - movq %rbx, %rdx #else xorl %edx, %edx @@ -339,7 +334,12 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) * addresses where we're currently running on. We have to do that here * because in 32bit we couldn't load a 64bit linear address. */ - lgdt early_gdt_descr(%rip) + subq $16, %rsp + movw $(GDT_SIZE-1), (%rsp) + leaq gdt_page(%rdx), %rax + movq %rax, 2(%rsp) + lgdt (%rsp) + addq $16, %rsp /* set up data segments */ xorl %eax,%eax @@ -754,11 +754,6 @@ SYM_DATA_END(level1_fixmap_pgt) .data .align 16 - -SYM_DATA(early_gdt_descr, .word GDT_ENTRIES*8-1) -SYM_DATA_LOCAL(early_gdt_descr_base, .quad INIT_PER_CPU_VAR(gdt_page)) - - .align 16 SYM_DATA(smpboot_control, .long 0) .align 16 From patchwork Thu Feb 23 19:11:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 61068 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp488327wrd; Thu, 23 Feb 2023 11:22:02 -0800 (PST) X-Google-Smtp-Source: AK7set87MRVek/jlXMrURCx7zKcXtQWDb15RtjnutBPQMgcOb2EMqZ9zCV7JH0dksEjuL/FnJXom X-Received: by 2002:a05:6a20:3d24:b0:be:cd93:66cd with SMTP id y36-20020a056a203d2400b000becd9366cdmr15667844pzi.2.1677180122100; Thu, 23 Feb 2023 11:22:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677180122; cv=none; d=google.com; s=arc-20160816; b=1Gcb//gfbdzBiHni83zBnMMUJhRe2I1RdmPE+/O+dXlduYn7ZuEglQGzLFQxmBm3pz A7osvQbZv2a6eSkXZ4nUj3cfjLRbLTHSTjlEWn4H8peqQWwkbfF7qYt99bRAodKLoEP6 08TfErpSEW08spcUCxv2jHBZVPHhXVIyOkkQ1WZVHhxnLAcJSSZpWfX/ypU0GVAAwZvF AyaLFKNIsqLxz7wdl9lAlBlPaSaJgi6F034OxFgSLWr+kQ33qVo/mi8eHSWSXh7H0zSU 5UkJSj10rdWyOS/JGw1yk6DNDz/x8HdY8BPqtogooyVyCJk17EFbafraEt8F6zUiMtCW qvNw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Ef9SnPny8CPfpZmm6N0jiCHwpvy9x8nd/PRiLPI7xag=; b=g83gqwgKqPn0KUHiIKdimXJZfY+A0r8MwLN6pfChG61tkM1CeUysGS0DC8Elhndn2Q O3yppbp2nZPnCTnrde1bLU1O8LLN7OAOW0FJBvAaF3joTd3CMOnyuOmitC4nADpW9xxB ktQ86457kjuCabTsAvU0W9gm5kww9V29+Q2+UDkafH4+/mqHKvNPXwJ/kvrrJm3eSW+E fMbphOMGv0Y3+HygYcQlgP5QmKh5PPSTk28KSrhn/OR96hZIsGevQb6vHaCrx4/W527R 1bxaOTitapfrd1DFRzqoTqf8T9l/v9jjAh+AuOb6wU4okpAY41ZsPcrjDtgv2CntkC6U XUSw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=XrQnR2cf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t140-20020a635f92000000b00502effd3bb9si4163467pgb.521.2023.02.23.11.21.48; Thu, 23 Feb 2023 11:22:02 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=XrQnR2cf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232728AbjBWTOJ (ORCPT + 99 others); Thu, 23 Feb 2023 14:14:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60256 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232389AbjBWTNR (ORCPT ); Thu, 23 Feb 2023 14:13:17 -0500 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 769945D44D for ; Thu, 23 Feb 2023 11:12:28 -0800 (PST) Received: by mail-wr1-x436.google.com with SMTP id h14so2893343wru.4 for ; Thu, 23 Feb 2023 11:12:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Ef9SnPny8CPfpZmm6N0jiCHwpvy9x8nd/PRiLPI7xag=; b=XrQnR2cfEdgLwqJ0o6NE4oQo94jXmalIs45LryMzC+zupXpNrQFyQaJDH9RhoY4NO/ X/U5UAZk9WIGO8WHqPCfOQy+GntieRIMVjX2CQO+K+o5U5aUtGC7UKKBzKuMbm21D8lp u0cZi//QcFT5PIFqhoirDUIPE4UgvBV1nDVlTma2g2xOeU/86sFo7mb1nakDRvdHP19X b6Hy4TfrowFaRJqrFtRTUHv3cCDJZXsk3yJr4KG4iRmgLGfdgrVmlvq6hEmItm3DhhlW um39DZUjO4+pDqLuLxhSJqXCoyrzn21aC2w8ZXkjEK+FWBK2QkpLgcwgQVEbVKEkuKEt bN/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ef9SnPny8CPfpZmm6N0jiCHwpvy9x8nd/PRiLPI7xag=; b=yBCh51JoQwQyFOv7v8GFtrc4DQpZByIkRm5wGU84nRHGx757LXBzFsNBeo0vkIISxP 7djnurvwM1+00WUkyTl03QV1LFZ6RY3kZQwu88Fdnx/cLGwxXcSdJugmSeSIYVNKOpAU LRCSq2UkYwasiymw8/1nyTsbTkNO2pxvHffNZwI78zK4avdf+WGTmnjgR2TMA5Q5oX/6 I4fQK/PMEgskbvBQjnIxiJx7EaEm8s/NDfG1v1n+BTA7NtcjhW9R1DruGOuh48m7IEwp YCrXHmjeYy3q8lzc2iOjJPWJknVsgFypYGN555vG2YUQ2EjtyRcP67Je7XWGLlbdPNok ANPw== X-Gm-Message-State: AO0yUKVDZm2q7MHeh4KkY+MjHh3yDGlbCyD1fH7lWOUQBy2LoXBwBZhf jXBQRw7aXEW5OIIh4lR5dELZ+A== X-Received: by 2002:a5d:410b:0:b0:2c5:58f5:3c40 with SMTP id l11-20020a5d410b000000b002c558f53c40mr9571433wrp.47.1677179513058; Thu, 23 Feb 2023 11:11:53 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:5ee0:5af0:64bd:6198]) by smtp.gmail.com with ESMTPSA id b15-20020a5d4b8f000000b002c561805a4csm12957286wrt.45.2023.02.23.11.11.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Feb 2023 11:11:52 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse , Usama Arif Subject: [PATCH v11 11/12] x86/smpboot: Remove initial_gs Date: Thu, 23 Feb 2023 19:11:39 +0000 Message-Id: <20230223191140.4155012-12-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230223191140.4155012-1-usama.arif@bytedance.com> References: <20230223191140.4155012-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758650823338113910?= X-GMAIL-MSGID: =?utf-8?q?1758650823338113910?= From: Brian Gerst Use the percpu offset directly to set GSBASE. Signed-off-by: Brian Gerst Reviewed-by: David Woodhouse Signed-off-by: David Woodhouse Tested-by: Usama Arif Signed-off-by: Usama Arif --- arch/x86/include/asm/realmode.h | 1 - arch/x86/kernel/acpi/sleep.c | 1 - arch/x86/kernel/head_64.S | 34 ++++++++++----------------------- 3 files changed, 10 insertions(+), 26 deletions(-) diff --git a/arch/x86/include/asm/realmode.h b/arch/x86/include/asm/realmode.h index f0357cfe2fb0..87e5482acd0d 100644 --- a/arch/x86/include/asm/realmode.h +++ b/arch/x86/include/asm/realmode.h @@ -60,7 +60,6 @@ extern struct real_mode_header *real_mode_header; extern unsigned char real_mode_blob_end[]; extern unsigned long initial_code; -extern unsigned long initial_gs; extern unsigned long initial_stack; #ifdef CONFIG_AMD_MEM_ENCRYPT extern unsigned long initial_vc_handler; diff --git a/arch/x86/kernel/acpi/sleep.c b/arch/x86/kernel/acpi/sleep.c index 6538ddb55f28..214dd4a79860 100644 --- a/arch/x86/kernel/acpi/sleep.c +++ b/arch/x86/kernel/acpi/sleep.c @@ -114,7 +114,6 @@ int x86_acpi_suspend_lowlevel(void) #else /* CONFIG_64BIT */ #ifdef CONFIG_SMP current->thread.sp = (unsigned long)temp_stack + sizeof(temp_stack); - initial_gs = per_cpu_offset(smp_processor_id()); /* Force the startup into boot mode */ saved_smpboot_ctrl = xchg(&smpboot_control, 0); #endif diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index 0dd57d573a0e..9ed87ba0609f 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -67,18 +67,10 @@ SYM_CODE_START_NOALIGN(startup_64) leaq _text(%rip), %rdi - /* - * initial_gs points to initial fixed_percpu_data struct with storage for - * the stack protector canary. Global pointer fixups are needed at this - * stage, so apply them as is done in fixup_pointer(), and initialize %gs - * such that the canary can be accessed at %gs:40 for subsequent C calls. - */ + /* Setup GSBASE to allow stack canary access for C code */ movl $MSR_GS_BASE, %ecx - movq initial_gs(%rip), %rax - movq $_text, %rdx - subq %rdx, %rax - addq %rdi, %rax - movq %rax, %rdx + leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx + movl %edx, %eax shrq $32, %rdx wrmsr @@ -243,10 +235,7 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) ANNOTATE_NOENDBR // above #ifdef CONFIG_SMP - /* - * Is this the boot CPU coming up? If so everything is available - * in initial_gs. - */ + /* Is this the boot CPU coming up? */ movl smpboot_control(%rip), %edx testl $STARTUP_SECONDARY, %edx jz .Linit_cpu0_data @@ -308,12 +297,7 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) .Linit_cpu_data: /* Get the per cpu offset for the given CPU# which is in ECX */ - leaq __per_cpu_offset(%rip), %rbx - movq (%rbx,%rcx,8), %rbx - /* Save it for GS BASE setup */ - movq %rbx, initial_gs(%rip) - - movq %rbx, %rdx + movq __per_cpu_offset(,%rcx,8), %rdx #else xorl %edx, %edx #endif /* CONFIG_SMP */ @@ -363,8 +347,11 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) * the per cpu areas are set up. */ movl $MSR_GS_BASE,%ecx - movl initial_gs(%rip),%eax - movl initial_gs+4(%rip),%edx +#ifndef CONFIG_SMP + leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx +#endif + movl %edx, %eax + shrq $32, %rdx wrmsr /* Drop the realmode protection. For the boot CPU the pointer is NULL! */ @@ -514,7 +501,6 @@ SYM_CODE_END(vc_boot_ghcb) __REFDATA .balign 8 SYM_DATA(initial_code, .quad x86_64_start_kernel) -SYM_DATA(initial_gs, .quad INIT_PER_CPU_VAR(fixed_percpu_data)) #ifdef CONFIG_AMD_MEM_ENCRYPT SYM_DATA(initial_vc_handler, .quad handle_vc_boot_ghcb) #endif From patchwork Thu Feb 23 19:11:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 61061 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp484721wrd; Thu, 23 Feb 2023 11:14:05 -0800 (PST) X-Google-Smtp-Source: AK7set+57wZ+qR/esF118Q6Xsl2OqJlFlxd8sQZ4qEbkFcTECbz5pDAVjOmRrA6hX5iEqlBXqHnH X-Received: by 2002:a17:906:10a:b0:8b1:78bc:3103 with SMTP id 10-20020a170906010a00b008b178bc3103mr22864439eje.7.1677179645221; Thu, 23 Feb 2023 11:14:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677179645; cv=none; d=google.com; s=arc-20160816; b=WKVs+I6YFJcGdfdpUnz0G5mqhxF7f9531u7aS2XcnV7ljaFhZW8Q+tk6PxXU9vG5q4 1mzmB7/9s6LLZuOSX79WUaf+1T1AVaZZGvzTpdePNg9WuRC6vz2zc1tNceYS7SXpm7Ml C4i9YdeIGdTYfIAtyXg5WGMX+e+DmSIVfXY0+91c91XiYXu9ZjyFuVZwCOowaVpLjXD3 grmzvN0gla/5cb342y5BirmGaCwh1zWkRcNcGuPd9OZf5cbq+YKFZ+pfFQvgbLTPG1t1 cNywREabsKDqIC0iqGGc6uOJIK+fmnmXzBToSJ7UhZr2JccB64fQ/CoD0vet84QzhJiy 1gGw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=EwjBftfc3L0BBR8aaJYAfM/W208AZmDnuvLkbcbpYBc=; b=SrQ6j+wCB3v62nmDVb5874oTkiBfFHMdA7+RAkbCoAAacHA3O5p85VICGPRXT5IxA3 vgv3waLYpnInjVFdAn26B5amTqkP1r++bHaAdJXSyMwMrmMBctGW5U5jA8TCIi8FxZw4 DwW8/74X9U7H1nCKli3QkW1M6nOWHnl7VlKf1hvXqpUpCE0HlLsaX/SrGjmxz8KIypuc iTatlVfB0EMkARQEHjp2gOjMwYmx0Ha83d5lRpL1mrX0e+5z/jchfjM0tzwkOOhOfhi7 T/C2szniUhayXMHfc9t0psknSOnqlIJQRhtQauxY6nH2WyFyKJAolgh8yaOj1VdYHtuN 0w+w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=D6yXbUB9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 24-20020a170906009800b008d41e170aa3si15380571ejc.557.2023.02.23.11.13.42; Thu, 23 Feb 2023 11:14:05 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=D6yXbUB9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231974AbjBWTNA (ORCPT + 99 others); Thu, 23 Feb 2023 14:13:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60852 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232155AbjBWTMx (ORCPT ); Thu, 23 Feb 2023 14:12:53 -0500 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 31DFA3E081 for ; Thu, 23 Feb 2023 11:12:32 -0800 (PST) Received: by mail-wr1-x42a.google.com with SMTP id h14so2893384wru.4 for ; Thu, 23 Feb 2023 11:12:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=EwjBftfc3L0BBR8aaJYAfM/W208AZmDnuvLkbcbpYBc=; b=D6yXbUB9sco4X1VpFV6XERHwY2Jrdg2C/gnoW/DRwjyLLVrrxI1ywrohxmZRrJm091 pyvN1Hju8nZVLvJpgR9MegSvJlbBhzhD4yTgTKONC35c++3h/vbAyqsGVIcCxYjtPJb/ tggWnyVAoBa94JfCwJ6wYvG1/pp2c2t+m2b+McHBtAG54vrfvV8PsUW/GRtA+Wlg1RBV GBLI60u93gjpV6vPweKG3CwxbRegUHgsMZ2+2kfjBBN4dujoujM87ABiPwktktjExmO7 B/4Xuz+/aG30TSdTs0orGRkS9Bv/ooYZTvwdgfM4oBWX7zEfFUxgbwLY3cQKWTXxUFA3 Sjig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EwjBftfc3L0BBR8aaJYAfM/W208AZmDnuvLkbcbpYBc=; b=HruLMlweTKzCG8xIxJfYUpgIe+zkis8Vhl6nXLJpCcSsbQn5oM/Y2ReENYimcMCALr kWzTqctFDxhh6qsI5BaHmSuvqqeDVxlZTIkr1WMWFJLcB47ZP63jZZaaMnQSqiqssoyP 4QW2yXMdjJdH1nV1F3Rrg8x2YZ+cqWA2gQWlGi19itANSUWZxBY8fbw9lQZ7FuEmk+I4 3R4NubPoSqSFbSlzbyOCWCDl0vEndehVwG68Vp7eC7POVrb/aiyAzkioe/kncX+wktrU oyb7g1eohzJKMuQIHCvVJE3P3T4IQarEN7e9EPBf36JZ/+oEq/wbmnqpbWc0lcXYDAnK QtuQ== X-Gm-Message-State: AO0yUKURw4z5cDJWqrXNWHcay3bgmCyp+e/2+ZAlXbhRzoPK7GQvXiKI gC0OigyEHAmhOJC8dNMcMDmOmg== X-Received: by 2002:a5d:6743:0:b0:2c7:a3b:4e76 with SMTP id l3-20020a5d6743000000b002c70a3b4e76mr5897330wrw.6.1677179513936; Thu, 23 Feb 2023 11:11:53 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:5ee0:5af0:64bd:6198]) by smtp.gmail.com with ESMTPSA id b15-20020a5d4b8f000000b002c561805a4csm12957286wrt.45.2023.02.23.11.11.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Feb 2023 11:11:53 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse , Usama Arif Subject: [PATCH v11 12/12] x86/smpboot: Simplify boot CPU setup Date: Thu, 23 Feb 2023 19:11:40 +0000 Message-Id: <20230223191140.4155012-13-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230223191140.4155012-1-usama.arif@bytedance.com> References: <20230223191140.4155012-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758650323807128860?= X-GMAIL-MSGID: =?utf-8?q?1758650323807128860?= From: Brian Gerst Now that the per-cpu GSBASE, stack, and GDT descriptor can be derived dynamically by CPU number, the boot CPU can use a fixed CPU number and take the same path as secondary CPUs. Signed-off-by: Brian Gerst Reviewed-by: David Woodhouse Signed-off-by: David Woodhouse Tested-by: Usama Arif Signed-off-by: Usama Arif --- arch/x86/include/asm/smp.h | 5 ++--- arch/x86/kernel/head_64.S | 25 +++++++------------------ arch/x86/kernel/smpboot.c | 6 +++--- 3 files changed, 12 insertions(+), 24 deletions(-) diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h index b4b29e052b6e..97a36d029b0e 100644 --- a/arch/x86/include/asm/smp.h +++ b/arch/x86/include/asm/smp.h @@ -202,8 +202,7 @@ extern unsigned int smpboot_control; #endif /* !__ASSEMBLY__ */ /* Control bits for startup_64 */ -#define STARTUP_SECONDARY 0x80000000 -#define STARTUP_APICID_CPUID_0B 0x40000000 -#define STARTUP_APICID_CPUID_01 0x20000000 +#define STARTUP_APICID_CPUID_0B 0x80000000 +#define STARTUP_APICID_CPUID_01 0x40000000 #endif /* _ASM_X86_SMP_H */ diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index 9ed87ba0609f..949c13b26811 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -235,28 +235,22 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) ANNOTATE_NOENDBR // above #ifdef CONFIG_SMP - /* Is this the boot CPU coming up? */ - movl smpboot_control(%rip), %edx - testl $STARTUP_SECONDARY, %edx - jz .Linit_cpu0_data - /* * For parallel boot, the APIC ID is retrieved from CPUID, and then * used to look up the CPU number. For booting a single CPU, the * CPU number is encoded in smpboot_control. * - * Bit 31 STARTUP_SECONDARY flag (checked above) * Bit 30 STARTUP_APICID_CPUID_0B flag (use CPUID 0x0b) * Bit 29 STARTUP_APICID_CPUID_01 flag (use CPUID 0x01) * Bit 0-24 CPU# if STARTUP_APICID_CPUID_xx flags are not set */ - testl $STARTUP_APICID_CPUID_0B, %edx + movl smpboot_control(%rip), %ecx + testl $STARTUP_APICID_CPUID_0B, %ecx jnz .Luse_cpuid_0b - testl $STARTUP_APICID_CPUID_01, %edx + testl $STARTUP_APICID_CPUID_01, %ecx jnz .Luse_cpuid_01 - andl $0x0FFFFFFF, %edx - movl %edx, %ecx - jmp .Linit_cpu_data + andl $0x0FFFFFFF, %ecx + jmp .Lsetup_cpu .Luse_cpuid_01: mov $0x01, %eax @@ -277,7 +271,7 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) .Lfind_cpunr: cmpl (%rbx,%rcx,4), %edx - jz .Linit_cpu_data + jz .Lsetup_cpu inc %ecx cmpl nr_cpu_ids(%rip), %ecx jb .Lfind_cpunr @@ -291,18 +285,13 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) hlt jmp 1b -.Linit_cpu0_data: - movq __per_cpu_offset(%rip), %rdx - jmp .Lsetup_cpu - -.Linit_cpu_data: +.Lsetup_cpu: /* Get the per cpu offset for the given CPU# which is in ECX */ movq __per_cpu_offset(,%rcx,8), %rdx #else xorl %edx, %edx #endif /* CONFIG_SMP */ -.Lsetup_cpu: /* * Setup a boot time stack - Any secondary CPU will have lost its stack * by now because the cr3-switch above unmaps the real-mode stack diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 69ef0860feea..9d956571ecc1 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -1140,7 +1140,7 @@ static int do_boot_cpu(int apicid, int cpu, struct task_struct *idle, early_gdt_descr.address = (unsigned long)get_cpu_gdt_rw(cpu); initial_stack = idle->thread.sp; } else if (!do_parallel_bringup) { - smpboot_control = STARTUP_SECONDARY | cpu; + smpboot_control = cpu; } /* Enable the espfix hack for this CPU */ @@ -1580,7 +1580,7 @@ void __init native_smp_prepare_cpus(unsigned int max_cpus) */ if (eax) { pr_debug("Using CPUID 0xb for parallel CPU startup\n"); - smpboot_control = STARTUP_SECONDARY | STARTUP_APICID_CPUID_0B; + smpboot_control = STARTUP_APICID_CPUID_0B; } else { pr_info("Disabling parallel bringup because CPUID 0xb looks untrustworthy\n"); do_parallel_bringup = false; @@ -1588,7 +1588,7 @@ void __init native_smp_prepare_cpus(unsigned int max_cpus) } else if (do_parallel_bringup) { /* Without X2APIC, what's in CPUID 0x01 should suffice. */ pr_debug("Using CPUID 0x1 for parallel CPU startup\n"); - smpboot_control = STARTUP_SECONDARY | STARTUP_APICID_CPUID_01; + smpboot_control = STARTUP_APICID_CPUID_01; } if (do_parallel_bringup) {