From patchwork Wed Feb 1 20:43:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 51555 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp502484wrn; Wed, 1 Feb 2023 12:46:30 -0800 (PST) X-Google-Smtp-Source: AK7set8rCA8Dz18g+6DvcZ0sghuKHQZ8mFriIINTWFZi2xxM5K4QXygk2UKOIpi46fpMFQftVmtU X-Received: by 2002:a05:6a20:a026:b0:bf:233b:c1d5 with SMTP id p38-20020a056a20a02600b000bf233bc1d5mr4881495pzj.61.1675284389914; Wed, 01 Feb 2023 12:46:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675284389; cv=none; d=google.com; s=arc-20160816; b=mEpzqyhr0yjxS2fuGIHkazPUWS0z8tOZ6MAtFTNkj4m0YTwqz3Jj4WLHi5aRYItNUg QuH3OTS1auTzjhGcMnvJEBIRoPnmK0m8mzUU52G7dWNu/6Q9B26iPn2ilvwEBoOvj+Ur B6kuqD/hfDJTuwwjU+yOAz6jTTl5TfHWmdAm52O+91SMb9GUNOa4YRB2zNgpztmHz3mm u0eFVDBuIe0DHEziqG7/LSNIeWRcAPgkrAHVh7b0nPsUgUcYgYdxdQ34CqsmYENEA+bG 3+US/F4aZnS78gaVPAf4sI8/QuJ4YMsgaLqE8fOgpfP/nFItMS7zYgLF0Ehlj1XcRh+Q ygkA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=m3/r2SgGGjsfHuJnNSpZFlNUSXv8jvj1KYLkjQUwAnE=; b=TWX/O7bsYViqDQHlEy5IToXGYVkr5PNaY0/ipbJV5gyL06RS6fl+pl/WZsbWpc3a/4 YJ1obYsdD9+UnrGugZzvAh4i8YQlsahctEJTFR/Z84R6L8dgwNF/AEbHs3lwJF/WtTJK hI1HjN65XBh25TVNheVhqLzB9o62lmqZaQPaR12JyQjIu+R+oiXUCen+BeQqawRODHDP KUqcpTGKdxY/sMn73JjZfihyJBDHbcIFijynpqn3diq/SXz6Qgbn6h4YOSX/+MT0Ajl5 +rpqERlnSRDR7SDxG2nnUgenUnVpVQ3edOO5Zykvb0bF9sIVJlGnlYrmYPDJU7eerw3Z V7WQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=uZok0F+N; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i69-20020a639d48000000b004f1bfbf42d0si249174pgd.135.2023.02.01.12.45.51; Wed, 01 Feb 2023 12:46:29 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=uZok0F+N; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230287AbjBAUpb (ORCPT + 99 others); Wed, 1 Feb 2023 15:45:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53342 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230213AbjBAUpU (ORCPT ); Wed, 1 Feb 2023 15:45:20 -0500 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 954EE790B7 for ; Wed, 1 Feb 2023 12:44:49 -0800 (PST) Received: by mail-wr1-x435.google.com with SMTP id m14so18029271wrg.13 for ; Wed, 01 Feb 2023 12:44:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=m3/r2SgGGjsfHuJnNSpZFlNUSXv8jvj1KYLkjQUwAnE=; b=uZok0F+Nsacmd6VTqhDBt1ovy1sIk4RPXpwjQjKo9fsXZWBIziS84xJQtuDb9Iz4wr vYYUQYiM+djqSZ0mE03uhIYxvfPNhxlC2p8rcHxrLgBjaq6jvDZ0ZgFGlidQpYHmZ3z4 q1abOM2IcvZgBu7eLzI3l5nxnFblZB2ayYJkeYW0RF/pR6iuDslQ7vFhPWncB3aEbXC5 heTsA8QO7iBDgU6UZtXDP0UwUpswRERaFWWCvLWUSm0CDcnXfNWvAJ8jCYgUimmzj9vW co9TLqo8cqgZTeXhJQ+M+ZNwzUn6jjbhsFaFpEQ2yphfH6WIaPMhRX8nuX4F1SeaNA4t 5yxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=m3/r2SgGGjsfHuJnNSpZFlNUSXv8jvj1KYLkjQUwAnE=; b=HkFB++h0vVnShwwJrpdqpMvsT9ojbubTn8rb1Mp327v6imXPpoBPefoccXb1oRwZa4 6VCYzWd3+50b48P4DuprbdLWPOEyxRNBV/IUDqw9qBLJVUr+r71aCC77XeL7kkIu/TAT GMQv0fP7ULGCLqbA7qfS3Dt5z0sNg3OBZSe72qpZU7p7Ppo2PsF/Cz66trWOySO7wSEA R6y+N2i1Z8quz+T4dIa03rsxPe5XA2e+goSUcrJPEgwLHzdXmWgwolV3HFyxs8lQNic6 yrWCpHA2fthEL8INbFp9vRkeaeiReEZY55LQCokmf3RSAd7FeUTESHViM9OBfgrYqRHy Jn/g== X-Gm-Message-State: AO0yUKUT5rvWbary7N1kqTpacEXLcSqbarDBHKCtUr4kYQlgb+IW5jqH V5vC3r2CJHELdANI9J9k2fvEQQ== X-Received: by 2002:a05:6000:a0e:b0:2bf:b389:ae6 with SMTP id co14-20020a0560000a0e00b002bfb3890ae6mr4088125wrb.42.1675284288032; Wed, 01 Feb 2023 12:44:48 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:7611:c340:3d8d:d46c]) by smtp.gmail.com with ESMTPSA id n15-20020a5d598f000000b002bdff778d87sm19993584wri.34.2023.02.01.12.44.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Feb 2023 12:44:47 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de Cc: mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse Subject: [PATCH 1/9] x86/apic/x2apic: Fix parallel handling of cluster_mask Date: Wed, 1 Feb 2023 20:43:30 +0000 Message-Id: <20230201204338.1337562-2-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230201204338.1337562-1-usama.arif@bytedance.com> References: <20230201204338.1337562-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756663004894694416?= X-GMAIL-MSGID: =?utf-8?q?1756663004894694416?= From: David Woodhouse For each CPU being brought up, the alloc_clustermask() function allocates a new struct cluster_mask just in case it's needed. Then the target CPU actually runs, and in init_x2apic_ldr() it either uses a cluster_mask from a previous CPU in the same cluster, or consumes the "spare" one and sets the global pointer to NULL. That isn't going to parallelise stunningly well. Ditch the global variable, let alloc_clustermask() install the struct *directly* in the per_cpu data for the CPU being brought up. As an optimisation, actually make it do so for *all* present CPUs in the same cluster, which means only one iteration over for_each_present_cpu() instead of doing so repeatedly, once for each CPU. Now, in fact, there's no point in the 'node' or 'clusterid' members of the struct cluster_mask, so just kill it and use struct cpumask instead. This was a harmless "bug" while CPU bringup wasn't actually happening in parallel. It's about to become less harmless... Fixes: 023a611748fd5 ("x86/apic/x2apic: Simplify cluster management") Signed-off-by: David Woodhouse --- arch/x86/kernel/apic/x2apic_cluster.c | 108 +++++++++++++++----------- 1 file changed, 62 insertions(+), 46 deletions(-) diff --git a/arch/x86/kernel/apic/x2apic_cluster.c b/arch/x86/kernel/apic/x2apic_cluster.c index e696e22d0531..e116dfaf5922 100644 --- a/arch/x86/kernel/apic/x2apic_cluster.c +++ b/arch/x86/kernel/apic/x2apic_cluster.c @@ -9,11 +9,7 @@ #include "local.h" -struct cluster_mask { - unsigned int clusterid; - int node; - struct cpumask mask; -}; +#define apic_cluster(apicid) ((apicid) >> 4) /* * __x2apic_send_IPI_mask() possibly needs to read @@ -23,8 +19,7 @@ struct cluster_mask { static u32 *x86_cpu_to_logical_apicid __read_mostly; static DEFINE_PER_CPU(cpumask_var_t, ipi_mask); -static DEFINE_PER_CPU_READ_MOSTLY(struct cluster_mask *, cluster_masks); -static struct cluster_mask *cluster_hotplug_mask; +static DEFINE_PER_CPU_READ_MOSTLY(struct cpumask *, cluster_masks); static int x2apic_acpi_madt_oem_check(char *oem_id, char *oem_table_id) { @@ -60,10 +55,10 @@ __x2apic_send_IPI_mask(const struct cpumask *mask, int vector, int apic_dest) /* Collapse cpus in a cluster so a single IPI per cluster is sent */ for_each_cpu(cpu, tmpmsk) { - struct cluster_mask *cmsk = per_cpu(cluster_masks, cpu); + struct cpumask *cmsk = per_cpu(cluster_masks, cpu); dest = 0; - for_each_cpu_and(clustercpu, tmpmsk, &cmsk->mask) + for_each_cpu_and(clustercpu, tmpmsk, cmsk) dest |= x86_cpu_to_logical_apicid[clustercpu]; if (!dest) @@ -71,7 +66,7 @@ __x2apic_send_IPI_mask(const struct cpumask *mask, int vector, int apic_dest) __x2apic_send_IPI_dest(dest, vector, APIC_DEST_LOGICAL); /* Remove cluster CPUs from tmpmask */ - cpumask_andnot(tmpmsk, tmpmsk, &cmsk->mask); + cpumask_andnot(tmpmsk, tmpmsk, cmsk); } local_irq_restore(flags); @@ -105,55 +100,76 @@ static u32 x2apic_calc_apicid(unsigned int cpu) static void init_x2apic_ldr(void) { - struct cluster_mask *cmsk = this_cpu_read(cluster_masks); - u32 cluster, apicid = apic_read(APIC_LDR); - unsigned int cpu; + struct cpumask *cmsk = this_cpu_read(cluster_masks); - x86_cpu_to_logical_apicid[smp_processor_id()] = apicid; + BUG_ON(!cmsk); - if (cmsk) - goto update; - - cluster = apicid >> 16; - for_each_online_cpu(cpu) { - cmsk = per_cpu(cluster_masks, cpu); - /* Matching cluster found. Link and update it. */ - if (cmsk && cmsk->clusterid == cluster) - goto update; + cpumask_set_cpu(smp_processor_id(), cmsk); +} + +/* + * As an optimisation during boot, set the cluster_mask for *all* + * present CPUs at once, to prevent *each* of them having to iterate + * over the others to find the existing cluster_mask. + */ +static void prefill_clustermask(struct cpumask *cmsk, u32 cluster) +{ + int cpu; + + for_each_present_cpu(cpu) { + u32 apicid = apic->cpu_present_to_apicid(cpu); + if (apicid != BAD_APICID && apic_cluster(apicid) == cluster) { + struct cpumask **cpu_cmsk = &per_cpu(cluster_masks, cpu); + + BUG_ON(*cpu_cmsk && *cpu_cmsk != cmsk); + *cpu_cmsk = cmsk; + } } - cmsk = cluster_hotplug_mask; - cmsk->clusterid = cluster; - cluster_hotplug_mask = NULL; -update: - this_cpu_write(cluster_masks, cmsk); - cpumask_set_cpu(smp_processor_id(), &cmsk->mask); } -static int alloc_clustermask(unsigned int cpu, int node) +static int alloc_clustermask(unsigned int cpu, u32 cluster, int node) { + struct cpumask *cmsk = NULL; + unsigned int cpu_i; + u32 apicid; + if (per_cpu(cluster_masks, cpu)) return 0; - /* - * If a hotplug spare mask exists, check whether it's on the right - * node. If not, free it and allocate a new one. - */ - if (cluster_hotplug_mask) { - if (cluster_hotplug_mask->node == node) - return 0; - kfree(cluster_hotplug_mask); + + /* For the hotplug case, don't always allocate a new one */ + if (system_state >= SYSTEM_RUNNING) { + for_each_present_cpu(cpu_i) { + apicid = apic->cpu_present_to_apicid(cpu_i); + if (apicid != BAD_APICID && apic_cluster(apicid) == cluster) { + cmsk = per_cpu(cluster_masks, cpu_i); + if (cmsk) + break; + } + } + } + if (!cmsk) { + cmsk = kzalloc_node(sizeof(*cmsk), GFP_KERNEL, node); + if (!cmsk) + return -ENOMEM; } - cluster_hotplug_mask = kzalloc_node(sizeof(*cluster_hotplug_mask), - GFP_KERNEL, node); - if (!cluster_hotplug_mask) - return -ENOMEM; - cluster_hotplug_mask->node = node; + per_cpu(cluster_masks, cpu) = cmsk; + + if (system_state < SYSTEM_RUNNING) + prefill_clustermask(cmsk, cluster); + return 0; } static int x2apic_prepare_cpu(unsigned int cpu) { - if (alloc_clustermask(cpu, cpu_to_node(cpu)) < 0) + u32 phys_apicid = apic->cpu_present_to_apicid(cpu); + u32 cluster = apic_cluster(phys_apicid); + u32 logical_apicid = (cluster << 16) | (1 << (phys_apicid & 0xf)); + + x86_cpu_to_logical_apicid[cpu] = logical_apicid; + + if (alloc_clustermask(cpu, cluster, cpu_to_node(cpu)) < 0) return -ENOMEM; if (!zalloc_cpumask_var(&per_cpu(ipi_mask, cpu), GFP_KERNEL)) return -ENOMEM; @@ -162,10 +178,10 @@ static int x2apic_prepare_cpu(unsigned int cpu) static int x2apic_dead_cpu(unsigned int dead_cpu) { - struct cluster_mask *cmsk = per_cpu(cluster_masks, dead_cpu); + struct cpumask *cmsk = per_cpu(cluster_masks, dead_cpu); if (cmsk) - cpumask_clear_cpu(dead_cpu, &cmsk->mask); + cpumask_clear_cpu(dead_cpu, cmsk); free_cpumask_var(per_cpu(ipi_mask, dead_cpu)); return 0; } From patchwork Wed Feb 1 20:43:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 51553 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp502431wrn; Wed, 1 Feb 2023 12:46:20 -0800 (PST) X-Google-Smtp-Source: AK7set+98+mDwktre9w56JEiusHHS83b79jxLcvq7VR9IgJ6du/sFvsQ67GxB0kQxjPTdz82itlz X-Received: by 2002:a17:90b:180f:b0:22b:e5b9:43e9 with SMTP id lw15-20020a17090b180f00b0022be5b943e9mr3861529pjb.25.1675284380516; Wed, 01 Feb 2023 12:46:20 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675284380; cv=none; d=google.com; s=arc-20160816; b=w7BvzsjptKwGShyDYC38QEUJtbgbxGVHNh6Ix3OkqRK0sD88dUCYzBNQgwzRMehraM iXk6IPv9iiGszli3rDTyvv8Ct+0xsYHJwv0wRbCyugYuztIKqiBCyuJr31xh8QovdOqE 3gSeKQO33D1cYKpWIGa/GpPK44qGYXXRX385nb6OYYv5ZqsVd6oRlD3ax5C4VNbe4G2V s2BAccmZxmBk6CCZr9xE+ZXHBHqo3C6Gj41E22n3NpesSze9Lgbwrh98CZhx7TUw9Fa1 iOPF7QkabFdLnIWor6BvBuk4f+cwFoaqXOxpGTINGmx4Xa5uCHKt8kqMQaAzz9DI6rcF 0BBg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=cVGzQu3A2f4Qgo78q6jxGuOz9gIN1MMwXavNYjSHKBU=; b=ODKXFXacRvhaj/3aODg/tywafw0Ke5A/iSMFZEkeghzlxQdi8T/7q885gPhP3GJzKB 4XKvVBkuCs3uPnDca61QGex2FJrxp2JwXFvsMyEMgJ5a5+9SwEtrENGB2iF+xFGCSwro WNXcAKWVs7gTThGEmzGJ3JeeK+Um/pAZfXdfqN3Dp57PLDJJ/xw2NVaCuzlRSuVKNT7d IY9rJ5TycxvDurNiTkaWx2Sj/+jTkTDIqiCAmxWcUvE90dNKBB8l8WvcnF9ExQHEinIt 1wFeytNUCEXIiRezo/SyYMs6Z1ymCSbf0f3/rZA8E9H3xy9fY5efCsflRuU28DSPhnuK 3XTQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=1pmygf+5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id gg24-20020a17090b0a1800b0022915c0c7e2si2611099pjb.64.2023.02.01.12.46.08; Wed, 01 Feb 2023 12:46:20 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=1pmygf+5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230095AbjBAUpe (ORCPT + 99 others); Wed, 1 Feb 2023 15:45:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53356 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230218AbjBAUpU (ORCPT ); Wed, 1 Feb 2023 15:45:20 -0500 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6A02D77DF0 for ; Wed, 1 Feb 2023 12:44:50 -0800 (PST) Received: by mail-wr1-x432.google.com with SMTP id m7so18494373wru.8 for ; Wed, 01 Feb 2023 12:44:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cVGzQu3A2f4Qgo78q6jxGuOz9gIN1MMwXavNYjSHKBU=; b=1pmygf+5X1mOOTRudl4VcLIfZwAStujjk3QFGa6j7KBal48TQf/affncmOD3//kkMI rcmGFccJCavXlkXytWAbcm6emsOHDwsjt6vCZUF43+rfAZGCQ79xzEj6NExy3AWl2TIH yqq4qgKbDrRXWSdk/Q6LTGijK2ZwPQRhSwcI44pdzEE7QTuapkzHDgxfAFcajF4pFeiU c+DydIzDTjs9qzHUYHbj4pjAqVKeSBkn3qUBcz10GgDBL1+qKVKQYV54cTXisUVhB59K zisPp3vUaSdjviwrmlQIpnH9yGM9tfJ3vGj1rRhx0PEZ2z+5ua/QBLLFDvzwKxEmVQh9 qy/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cVGzQu3A2f4Qgo78q6jxGuOz9gIN1MMwXavNYjSHKBU=; b=7OfAyEn47Zg174E3qhMxxu1noSQsz2DWAbNPQrvdQ0J5s75eCWvrkChWv8iU75/j/D ZIvXAGWipTuL2+gzTHzmBc942393mcwFNzDrv1G+opJAoHnMke3DU3MhVAoyXbY6ziZJ N8XI6rR/Amszc33CxrIIl+N94LVLXedXKxfJrdybx5vi4VAjPaqsvKdv1foC2qGS4kI+ wifDnMgyBXoLMR7/tm41ucPm/bM8a0YjTT88APoYYc5KT3/7fty8hpZK/Dhb9K++Y7Jq AVKmoY3M9O0YN5s2djA/fh1lwTHEdLT7r6LfFjYOgN/w1ASI5W+5sxt5L5uy61e5RR7c hABQ== X-Gm-Message-State: AO0yUKW+6arreEgvogjcFKky2iiveMSCoF1q4jKD6aEJyaS1VIqC7lYJ HSY4A8jaO8zGGA0xpmJHxKp3FA== X-Received: by 2002:a05:6000:1004:b0:2bf:d332:e123 with SMTP id a4-20020a056000100400b002bfd332e123mr3000451wrx.62.1675284288852; Wed, 01 Feb 2023 12:44:48 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:7611:c340:3d8d:d46c]) by smtp.gmail.com with ESMTPSA id n15-20020a5d598f000000b002bdff778d87sm19993584wri.34.2023.02.01.12.44.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Feb 2023 12:44:48 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de Cc: mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse Subject: [PATCH 2/9] cpu/hotplug: Move idle_thread_get() to Date: Wed, 1 Feb 2023 20:43:31 +0000 Message-Id: <20230201204338.1337562-3-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230201204338.1337562-1-usama.arif@bytedance.com> References: <20230201204338.1337562-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756662994443695910?= X-GMAIL-MSGID: =?utf-8?q?1756662994443695910?= From: David Woodhouse Instead of relying purely on the special-case wrapper in bringup_cpu() to pass the idle thread to __cpu_up(), expose idle_thread_get() so that the architecture code can obtain it directly when necessary. This will be useful when the existing __cpu_up() is split into multiple phases, only *one* of which will actually need the idle thread. If the architecture code is to register its new pre-bringup states with the cpuhp core, having a special-case wrapper to pass extra arguments is non-trivial and it's easier just to let the arch register its function pointer to be invoked with the standard API. Signed-off-by: David Woodhouse --- include/linux/smpboot.h | 7 +++++++ kernel/smpboot.h | 2 -- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/include/linux/smpboot.h b/include/linux/smpboot.h index 9d1bc65d226c..3862addcaa34 100644 --- a/include/linux/smpboot.h +++ b/include/linux/smpboot.h @@ -5,6 +5,13 @@ #include struct task_struct; + +#ifdef CONFIG_GENERIC_SMP_IDLE_THREAD +struct task_struct *idle_thread_get(unsigned int cpu); +#else +static inline struct task_struct *idle_thread_get(unsigned int cpu) { return NULL; } +#endif + /* Cookie handed to the thread_fn*/ struct smpboot_thread_data; diff --git a/kernel/smpboot.h b/kernel/smpboot.h index 34dd3d7ba40b..60c609318ad6 100644 --- a/kernel/smpboot.h +++ b/kernel/smpboot.h @@ -5,11 +5,9 @@ struct task_struct; #ifdef CONFIG_GENERIC_SMP_IDLE_THREAD -struct task_struct *idle_thread_get(unsigned int cpu); void idle_thread_set_boot_cpu(void); void idle_threads_init(void); #else -static inline struct task_struct *idle_thread_get(unsigned int cpu) { return NULL; } static inline void idle_thread_set_boot_cpu(void) { } static inline void idle_threads_init(void) { } #endif From patchwork Wed Feb 1 20:43:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 51550 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp502301wrn; Wed, 1 Feb 2023 12:46:05 -0800 (PST) X-Google-Smtp-Source: AK7set9yozmEJerJ2ac05NucQeJd6O0BSpo2IXHtFuf6h4kpjERuJlRsYoytfJPCW6lZYGXjNNHb X-Received: by 2002:a17:907:2c43:b0:878:5f8e:26c0 with SMTP id hf3-20020a1709072c4300b008785f8e26c0mr218146ejc.7.1675284365618; Wed, 01 Feb 2023 12:46:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675284365; cv=none; d=google.com; s=arc-20160816; b=knDHEwTeRoFzvEvjMPdGJxMyTcHc1bIJLISYBtPXuB+J1ElgzfZiHZPE208BiIYavJ SCadjI5isnHpn901L6gnXElrsAC5v/fPbJuEdq0rRoyLOLRs/boipcN3RFC3aEd8uviG 4APNsjNv8ZiZ0x1OVgmIovBCo8v71sO7LSQ7FO0HpVI8pyo/819wqPezO2WAmT202OvT jgEs04nhcF5yBYlV8wNANesb423Rf87EHBuNsUPwCK9omzZIIaPjW0Sg6QxQ4u5T8Unn /nv5x4XZ6GFBp4FnGN+W9lbI93mXtXmxLHetN+ziNB06qf0i5S/DPAC1zFt2aXI1vdyF TfxQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=D35a783y9+r4vRds/PMwj8fmKYL9Opm+FKXtqgXxLHg=; b=g+M+c/s8EVu3x7HHGByCBkaaHvrXuMhjDFoF84CyVMCKunXkZs+pyCI17yf5sgcnRy Bz9ewOWoY6wiWwZoSQrGuiDXbEqrEA/CpmCdWTXrKJX9Jab30DEEFfLAXxGehdwduJR5 nvnmXY8WN5k75s0855v15kwkGm3SmVK4lM28AQaz6uxIeg+TIxgCwOs5T0ULbI0KdYUF 1YLGT/UoXjjiE6eis32vu/E3NPVh8r2/JZzkOwtwIKz2nGRJ0XDtybfLWfasbQsp4Juz yt3UJNruxPdPqilSkhXby7yS691j3LgM3kWCmQHvItomfXwt1/kQMYlZ7ED8geB8GlSg af1Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=eF8Rk+Mt; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id v3-20020a17090651c300b0087f21349a65si19159077ejk.809.2023.02.01.12.45.17; Wed, 01 Feb 2023 12:46:05 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=eF8Rk+Mt; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229926AbjBAUpA (ORCPT + 99 others); Wed, 1 Feb 2023 15:45:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53446 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229479AbjBAUo6 (ORCPT ); Wed, 1 Feb 2023 15:44:58 -0500 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 43D13783F7 for ; Wed, 1 Feb 2023 12:44:51 -0800 (PST) Received: by mail-wr1-x436.google.com with SMTP id o18so9025229wrj.3 for ; Wed, 01 Feb 2023 12:44:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=D35a783y9+r4vRds/PMwj8fmKYL9Opm+FKXtqgXxLHg=; b=eF8Rk+MtnUKV+dEzht46DG3sSGDcWvQ+IioOradRjywouK3nVCjQwpj3Pr2GEwQsba 7tLkNcl3tQT+CsojvjzsmHXw3l5ju/hSDjM55geghkRVpzOcX64DsACf3gT/2f6tBq1v aR57e9OqJVLpNokFXvga1NUbihdw4wjpjf5RlKzKlJk8kcwPRTcI7hteq2tOGSP3JSdz lagaOVJTlIP6+d3zuFyuNZrTVYnYUJJG9+gb44942jTHDWCeqZ3hVcrv7et+Jd0yXjy4 T9U/Rk+fw81fCu0qbz7UvgD4bw0sBXqpupRoGdTndDVubTi0zk6EvndWP0ijiSZw+jB1 wjPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=D35a783y9+r4vRds/PMwj8fmKYL9Opm+FKXtqgXxLHg=; b=cDIzKge3i/4MbEK6QrCf0F1b9fApkEhv414JwOMWdoCc7WyNFNBkUVKK3iP29Y5zEk ZkA7ioWOsfojSACbl6JuTjml2ooGpPMEKp8PRLagLEF+eyAggsuVsC509uLvYVzHexG8 L14QW753Ezug6PYspU7tJuNT4M0ILp0iKDndpjA6yZRtlis9s3PF4CnK3ABJlu+xUGtx Gt55O7Azj/TjQVo4IC+a/xfZsLB2n60d/V6ceKH2k+S5IWiJBr8prviC7pLA/YZZcx2i 46Rlpjk6QYfMAox5O3iBvhVC16fRS2OuJqYg9/Jd9TrQtLIjVVZJ/AjK/Nifg84iRzQP efaw== X-Gm-Message-State: AO0yUKVWWKlXapltqXiwCASXVWPlDye5wjFCCDyNTk/71IE40mrfNRc1 lrhLzJ5ozQKMVDgRDFYygJNo2A== X-Received: by 2002:adf:f5ca:0:b0:2bf:d686:c873 with SMTP id k10-20020adff5ca000000b002bfd686c873mr123344wrp.28.1675284289627; Wed, 01 Feb 2023 12:44:49 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:7611:c340:3d8d:d46c]) by smtp.gmail.com with ESMTPSA id n15-20020a5d598f000000b002bdff778d87sm19993584wri.34.2023.02.01.12.44.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Feb 2023 12:44:49 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de Cc: mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse Subject: [PATCH 3/9] cpu/hotplug: Add dynamic parallel bringup states before CPUHP_BRINGUP_CPU Date: Wed, 1 Feb 2023 20:43:32 +0000 Message-Id: <20230201204338.1337562-4-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230201204338.1337562-1-usama.arif@bytedance.com> References: <20230201204338.1337562-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756662979108061169?= X-GMAIL-MSGID: =?utf-8?q?1756662979108061169?= From: David Woodhouse If the platform registers these states, bring all CPUs to each registered state in turn, before the final bringup to CPUHP_BRINGUP_CPU. This allows the architecture to parallelise the slow asynchronous tasks like sending INIT/SIPI and waiting for the AP to come to life. There is a subtlety here: even with an empty CPUHP_BP_PARALLEL_DYN step, this means that *all* CPUs are brought through the prepare states and to CPUHP_BP_PREPARE_DYN before any of them are taken to CPUHP_BRINGUP_CPU and then are allowed to run for themselves to CPUHP_ONLINE. So any combination of prepare/start calls which depend on A-B ordering for each CPU in turn, such as the X2APIC code which used to allocate a cluster mask 'just in case' and store it in a global variable in the prep stage, then potentially consume that preallocated structure from the AP and set the global pointer to NULL to be reallocated in CPUHP_X2APIC_PREPARE for the next CPU... would explode horribly. We believe that X2APIC was the only such case, for x86. But this is why it remains an architecture opt-in. For now. Note that the new parallel stages do *not* yet bring each AP to the CPUHP_BRINGUP_CPU state. The final loop in bringup_nonboot_cpus() is untouched, bringing each AP in turn from the final PARALLEL_DYN state (or all the way from CPUHP_OFFLINE) to CPUHP_BRINGUP_CPU and then waiting for that AP to do its own processing and reach CPUHP_ONLINE before releasing the next. Parallelising that part by bringing them all to CPUHP_BRINGUP_CPU and then waiting for them all is an exercise for the future. Signed-off-by: David Woodhouse --- include/linux/cpuhotplug.h | 2 ++ kernel/cpu.c | 27 +++++++++++++++++++++++++-- 2 files changed, 27 insertions(+), 2 deletions(-) diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h index f61447913db9..3c08d9acc7b1 100644 --- a/include/linux/cpuhotplug.h +++ b/include/linux/cpuhotplug.h @@ -132,6 +132,8 @@ enum cpuhp_state { CPUHP_MIPS_SOC_PREPARE, CPUHP_BP_PREPARE_DYN, CPUHP_BP_PREPARE_DYN_END = CPUHP_BP_PREPARE_DYN + 20, + CPUHP_BP_PARALLEL_DYN, + CPUHP_BP_PARALLEL_DYN_END = CPUHP_BP_PARALLEL_DYN + 4, CPUHP_BRINGUP_CPU, /* diff --git a/kernel/cpu.c b/kernel/cpu.c index bbad5e375d3b..67e4db9f334b 100644 --- a/kernel/cpu.c +++ b/kernel/cpu.c @@ -1481,6 +1481,24 @@ int bringup_hibernate_cpu(unsigned int sleep_cpu) void bringup_nonboot_cpus(unsigned int setup_max_cpus) { unsigned int cpu; + int n = setup_max_cpus - num_online_cpus(); + + /* ∀ parallel pre-bringup state, bring N CPUs to it */ + if (n > 0) { + enum cpuhp_state st = CPUHP_BP_PARALLEL_DYN; + + while (st <= CPUHP_BP_PARALLEL_DYN_END && + cpuhp_hp_states[st].name) { + int i = n; + + for_each_present_cpu(cpu) { + cpu_up(cpu, st); + if (!--i) + break; + } + st++; + } + } for_each_present_cpu(cpu) { if (num_online_cpus() >= setup_max_cpus) @@ -1858,6 +1876,10 @@ static int cpuhp_reserve_state(enum cpuhp_state state) step = cpuhp_hp_states + CPUHP_BP_PREPARE_DYN; end = CPUHP_BP_PREPARE_DYN_END; break; + case CPUHP_BP_PARALLEL_DYN: + step = cpuhp_hp_states + CPUHP_BP_PARALLEL_DYN; + end = CPUHP_BP_PARALLEL_DYN_END; + break; default: return -EINVAL; } @@ -1882,14 +1904,15 @@ static int cpuhp_store_callbacks(enum cpuhp_state state, const char *name, /* * If name is NULL, then the state gets removed. * - * CPUHP_AP_ONLINE_DYN and CPUHP_BP_PREPARE_DYN are handed out on + * CPUHP_AP_ONLINE_DYN and CPUHP_BP_P*_DYN are handed out on * the first allocation from these dynamic ranges, so the removal * would trigger a new allocation and clear the wrong (already * empty) state, leaving the callbacks of the to be cleared state * dangling, which causes wreckage on the next hotplug operation. */ if (name && (state == CPUHP_AP_ONLINE_DYN || - state == CPUHP_BP_PREPARE_DYN)) { + state == CPUHP_BP_PREPARE_DYN || + state == CPUHP_BP_PARALLEL_DYN)) { ret = cpuhp_reserve_state(state); if (ret < 0) return ret; From patchwork Wed Feb 1 20:43:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 51554 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp502480wrn; Wed, 1 Feb 2023 12:46:29 -0800 (PST) X-Google-Smtp-Source: AK7set8p/xR+nC8/KiolSKFsk2BBOiwOLrZFsduLEyX0sFlhx+WV2kdjE8Rao9qmMU34FI6eG8e8 X-Received: by 2002:a17:902:d48a:b0:196:8bd6:2398 with SMTP id c10-20020a170902d48a00b001968bd62398mr4949818plg.18.1675284389453; Wed, 01 Feb 2023 12:46:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675284389; cv=none; d=google.com; s=arc-20160816; b=efVEKBuSFd+LCa9IKeMhYazXgzkJ7tK5MwvYO+shAqLq/1I2aJRYVSnlsY2nev6wMO vlIjrSMFtBw3LXaQGvNSw91BofVkN00apn4QPEx8YXJkERXB6MNVZk1k35XYmZRKj1+U NRZdfHedGaGtapOqHQR02ARD9RNfaL1JmasdTL+FKVXQ8a1P8WobaRPOH7R6u5xExy0U X2sqs9xe37rVH78gKgkEJfbf3aTR+vt3++i+7ASwWd4xajS+q7GBbZ2RWsKFKmmIFwUa Q7WZCGRp+/n/pMulaboWXIUN+h1EIQePhkimTUk0i3NORB3qqqN4IMMOFBhTaKald03N Y0eA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=i6e9FsuHypszT7EvXQeEpKp5FLlszNay99UK4BoH4rM=; b=maBcYFB4SYBs348ekzKYtOhuwtgwr5AWgpFQ5u5pMb0Gq9Kk/MOyeDXyVaDcw5ng0h uczcUzpU0JJ8aoyu4GXzOiIidxcjjCw3SClPNMZ9dqF4EDorms8UK6bq8+7ObmQRNWlb z23rAGBwthBtb17JDuTqoQJDyNb6kOkCfj2744JqqNWlg0P1y4SAsgxMGlFEn4BYvCG5 h3Uy6InH10BsYgijDsM/K+w/5QhBg9BDVTNH/V4FKB68L9fgUegmDzyCQUfE1e9kNxRB vOpB9svhFsb1XtUi3bnO44LKn4+jlNmnf6fk24BMLdknZQifo3+BkaadKe5Ch/dA4dtp /OBA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=CwNSnHQi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l4-20020a170902d04400b0019326002309si18494385pll.602.2023.02.01.12.46.14; Wed, 01 Feb 2023 12:46:29 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=CwNSnHQi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231454AbjBAUph (ORCPT + 99 others); Wed, 1 Feb 2023 15:45:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53564 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230231AbjBAUpW (ORCPT ); Wed, 1 Feb 2023 15:45:22 -0500 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 29D0C78AE9 for ; Wed, 1 Feb 2023 12:44:52 -0800 (PST) Received: by mail-wr1-x433.google.com with SMTP id y1so18518665wru.2 for ; Wed, 01 Feb 2023 12:44:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=i6e9FsuHypszT7EvXQeEpKp5FLlszNay99UK4BoH4rM=; b=CwNSnHQiMWxYA/AjxMfHHlxHH6vbja8TRT1y9aTu12HxkVYye6Siks4+Jaoz8DdYxQ 6BN+MgUbghZ8pXP0F5EbX5ssnDuePd0zdBTOYLLR8v22Ii2gWTUTGBprCQpHnhfHBWvk PqKjfY4iLnId8XY6A4ykMVbHrxoldwAKGVzmzZpkg0nn0wwbaSGA4cnQUaM1Ni9KbNaD 2RK+CsdP0BEyIf9mghg++Zqpek1O2bkzOcfh8r5xI2mhqrPe10iuvsiCKyuusr9771iT mEc3DngzAFDNe3VmiJM4yva9zuZhtHvXT1kl2z1yXbyf2tDSPhRTgy3GdMVg7AI8NUti lpYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=i6e9FsuHypszT7EvXQeEpKp5FLlszNay99UK4BoH4rM=; b=BvcwjeSdcNS1U1GbjSerzQp2y4n+vZVmp0izwxPB8nQ/IXlh09kFi4vx0tzKpTKwig yDeQS5lwtxy6ltefkA6/txegfnAKfvEVDqLIbXMW9FP1paKG+lyp+mSLYemd3vfq5Q0Z 1vJScCxEDBYM0dH2v63EzjoVNOAe9PW2HPEzrDIyNNlHP6EZOFm3Xk//6w9Px8IAJ5tX E1SlVlgRHmXhgF7L6zRhu9j1gFDlrXKxAlxMNkB+XE0rRcfbtoWP1NW9bBq42sNN4TC2 H8wRh5oR6wz3Q//xqEqqeZA7BaHjmlAmiZS48bSoUEfrYrnvxIicU+U0w2z0B7NE3L6V LK9w== X-Gm-Message-State: AO0yUKXVlXywEqVx5DIxGB6vEAM3H/sjIJy475ykNO7AKCPviTHJFnzN 9LjNjy2fkZOO8oQQrXenuKtkWg== X-Received: by 2002:a5d:5643:0:b0:2c0:227d:ca48 with SMTP id j3-20020a5d5643000000b002c0227dca48mr3283516wrw.63.1675284290622; Wed, 01 Feb 2023 12:44:50 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:7611:c340:3d8d:d46c]) by smtp.gmail.com with ESMTPSA id n15-20020a5d598f000000b002bdff778d87sm19993584wri.34.2023.02.01.12.44.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Feb 2023 12:44:50 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de Cc: mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse , Usama Arif Subject: [PATCH 4/9] x86/smpboot: Reference count on smpboot_setup_warm_reset_vector() Date: Wed, 1 Feb 2023 20:43:33 +0000 Message-Id: <20230201204338.1337562-5-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230201204338.1337562-1-usama.arif@bytedance.com> References: <20230201204338.1337562-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756663004306176212?= X-GMAIL-MSGID: =?utf-8?q?1756663004306176212?= From: David Woodhouse If we want to do parallel CPU bringup, we're going to need to set this up and leave it until all CPUs are done. Might as well use the RTC spinlock to protect the refcount, as we need to take it anyway. [Usama Arif: fixed rebase conflict] Signed-off-by: David Woodhouse Signed-off-by: Usama Arif --- arch/x86/kernel/smpboot.c | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index dd2f61c3797f..2d5014752dc4 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -120,17 +120,22 @@ int arch_update_cpu_topology(void) return retval; } + +static unsigned int smpboot_warm_reset_vector_count; + static inline void smpboot_setup_warm_reset_vector(unsigned long start_eip) { unsigned long flags; spin_lock_irqsave(&rtc_lock, flags); - CMOS_WRITE(0xa, 0xf); + if (!smpboot_warm_reset_vector_count++) { + CMOS_WRITE(0xa, 0xf); + *((volatile unsigned short *)phys_to_virt(TRAMPOLINE_PHYS_HIGH)) = + start_eip >> 4; + *((volatile unsigned short *)phys_to_virt(TRAMPOLINE_PHYS_LOW)) = + start_eip & 0xf; + } spin_unlock_irqrestore(&rtc_lock, flags); - *((volatile unsigned short *)phys_to_virt(TRAMPOLINE_PHYS_HIGH)) = - start_eip >> 4; - *((volatile unsigned short *)phys_to_virt(TRAMPOLINE_PHYS_LOW)) = - start_eip & 0xf; } static inline void smpboot_restore_warm_reset_vector(void) @@ -142,10 +147,12 @@ static inline void smpboot_restore_warm_reset_vector(void) * to default values. */ spin_lock_irqsave(&rtc_lock, flags); - CMOS_WRITE(0, 0xf); + if (!--smpboot_warm_reset_vector_count) { + CMOS_WRITE(0, 0xf); + *((volatile u32 *)phys_to_virt(TRAMPOLINE_PHYS_LOW)) = 0; + } spin_unlock_irqrestore(&rtc_lock, flags); - *((volatile u32 *)phys_to_virt(TRAMPOLINE_PHYS_LOW)) = 0; } /* From patchwork Wed Feb 1 20:43:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 51560 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp502727wrn; Wed, 1 Feb 2023 12:47:02 -0800 (PST) X-Google-Smtp-Source: AK7set/sFxeKHpO6erAz0XX/iDmVCEYy3VuWAfdmfWTO7Z8fl3Ygtp+BLOxRx20n5WMYSvvnYsh0 X-Received: by 2002:a05:6a00:14d0:b0:593:9620:360 with SMTP id w16-20020a056a0014d000b0059396200360mr3910863pfu.7.1675284422281; Wed, 01 Feb 2023 12:47:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675284422; cv=none; d=google.com; s=arc-20160816; b=axSjCQejQnCKq0W9AXZh25WcTefJCPOJojuiEPQfBU+sB4M2PLAMMciwlWyWYqVq+j NiFwkxIe7bsGi23NtVgFTFNqfJUA1yGMRbKDRwkCo83o4XpB3u7C6vdlTVrLLP1OVoCI 6Mf1aUO70SjA0B+8kxWdHFrPNiFr6SVLobig1AeOCRl6eqK5ouMOIUh4+oWsDQe+O8hy h1uL35PNWVPW9db+T4wPNcvFuRvv2pH3RS+b2GrQP0CS5XdG5f8k6aAfI3BbE3yYu3UG Le91iEoRvEFSu8+8l1ljpgE8ZkK0o3b7s5nSd0i8A9KEJPyXQ6OUaVzA5jyNp2YLf3O3 6a2g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=CUSGXtK7n6/6YUwTEjKlWX96IORvZOTW/bJOX05E+p0=; b=XWWg72noLsUn6SFsrn8SSJED5B1KNlB3+y0PC281QynvqgG+2w8G21BAPC5ySBxBUF PL+Yzb0ABXC1JeqqcY10b3cN0wASUACFWb+YCt6JTLbcrQ2aXFXyNCcctj6abDP6aq4G 6aD8z6DfMopzBK1hHmSFx4loW7Q9zSGLYesmAReZ7cjLGGt7EtN7w8xPEel3via6RwET 7pFexDsqwcNkMhBzOhS2ECrRR6W7+8B0AKzrCKmDhf6o7tKsz+DMYSI1feUAB0SeCX3A 3hJrFxMKBlL9RiE/yDnRz+znz+N4X6RpuA7inQmJQKXNSrMYGy25tcTIAT+VErxSeeIA do/w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=SNi6efb2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f12-20020aa79d8c000000b00593b561b6c4si11218450pfq.149.2023.02.01.12.46.49; Wed, 01 Feb 2023 12:47:02 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=SNi6efb2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231691AbjBAUqG (ORCPT + 99 others); Wed, 1 Feb 2023 15:46:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53640 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231215AbjBAUpc (ORCPT ); Wed, 1 Feb 2023 15:45:32 -0500 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4A60574C37 for ; Wed, 1 Feb 2023 12:44:53 -0800 (PST) Received: by mail-wr1-x433.google.com with SMTP id q5so18541950wrv.0 for ; Wed, 01 Feb 2023 12:44:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CUSGXtK7n6/6YUwTEjKlWX96IORvZOTW/bJOX05E+p0=; b=SNi6efb2wXvpa5dRKNfFbHTbohVHxgaFa7TL8cN/uJD2cihgyVfu7V2/hPuG0yfqgv mMqL3dS0ei9J3FfAc9W2vvHBzfK8QBktYWQVbNaR33QRy0t+QLqnSTJLOTCZU/V4qqud 8vVwf5mMFYWnJRLLCtLvdWCfJAFpD3Vv9/cCURR5mbYosPiG6xsPB1yX9ECX2e3UnPLZ MXa8HjoDiLR9hDzxaLIgJqpL6J5Veos5J79U2yra/IlF+3Lz1aeYO4lif1wd0hMAm6IS 5Ny8+mAhu2A4nVW6yF80HpX8bOvqfC3fnsVi0IpNGFS3SJeYm9Me7jC/cijNMEKw94vo Fv6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CUSGXtK7n6/6YUwTEjKlWX96IORvZOTW/bJOX05E+p0=; b=cLI9gIp4REqOvC7lpQvAASTWPH3VzKi6dY6HerdOjjPxWKEV/IgNMDwSsaJ5akscAq QAymZ7s8/9KC/NLFBihKrqJKI4Srx0hbMswIt0lAjTJx9IYdy45paCwyOqLVwmdsCoqD 7U7xl0rxqomVQNqDkJ3rDA5eqWm/lnd2k3STn6MhvAEyohN16FRSuiOXUg0fAb5Jimpy bRXH1oXCseUMvDiQ+rEQPO1YvCyuASZIuLeQZ0UpFLKFIaBItQNfMFiTY+R0J3JX6Ye/ rvXa6AHKb9yaXJWot9TFXXkKWwLru1MSohBT72UrzOX2xgZCRhrEg7Xy5PTlgJyiMgyW YcDA== X-Gm-Message-State: AO0yUKXo07kjkvAkZEHXQG4jve3UJm5cFcxQyAHRPkoLgNWGEw4/01Xg 1SXKnSm7u8828xnCLcpOk4le6g== X-Received: by 2002:a05:6000:16c4:b0:2c3:be89:7c37 with SMTP id h4-20020a05600016c400b002c3be897c37mr129949wrf.26.1675284291708; Wed, 01 Feb 2023 12:44:51 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:7611:c340:3d8d:d46c]) by smtp.gmail.com with ESMTPSA id n15-20020a5d598f000000b002bdff778d87sm19993584wri.34.2023.02.01.12.44.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Feb 2023 12:44:51 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de Cc: mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse , Usama Arif Subject: [PATCH 5/9] x86/smpboot: Split up native_cpu_up into separate phases and document them Date: Wed, 1 Feb 2023 20:43:34 +0000 Message-Id: <20230201204338.1337562-6-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230201204338.1337562-1-usama.arif@bytedance.com> References: <20230201204338.1337562-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756663038404648725?= X-GMAIL-MSGID: =?utf-8?q?1756663038404648725?= From: David Woodhouse There are four logical parts to what native_cpu_up() does on the BSP (or on the controlling CPU for a later hotplug). First it actually wakes the AP by sending the INIT/SIPI/SIPI sequence. Second, it waits for the AP to make it as far as wait_for_master_cpu() which sets that CPU's bit in cpu_initialized_mask, then sets the bit in cpu_callout_mask to let the AP proceed through cpu_init(). Then, it waits for the AP to finish cpu_init() and get as far as the smp_callin() call, which sets that CPU's bit in cpu_callin_mask. Finally, it does the TSC synchronization and waits for the AP to actually mark itself online in cpu_online_mask. This commit should have no behavioural change, but merely splits those phases out into separate functions so that future commits can make them happen in parallel for all APs. And adds some comments around them on both the BSP and AP code paths. [Usama Arif: fixed rebase conflict] Signed-off-by: David Woodhouse Signed-off-by: Usama Arif --- arch/x86/kernel/smpboot.c | 182 +++++++++++++++++++++++++++----------- 1 file changed, 128 insertions(+), 54 deletions(-) diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 2d5014752dc4..98604a3953e6 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -205,6 +205,10 @@ static void smp_callin(void) wmb(); + /* + * This runs the AP through all the cpuhp states to its target + * state (CPUHP_ONLINE in the case of serial bringup). + */ notify_cpu_starting(cpuid); /* @@ -232,17 +236,33 @@ static void notrace start_secondary(void *unused) load_cr3(swapper_pg_dir); __flush_tlb_all(); #endif + /* + * Sync point with do_wait_cpu_initialized(). On boot, all secondary + * CPUs reach this stage after receiving INIT/SIPI from do_cpu_up() + * in the x86/cpu:kick cpuhp stage. At the start of cpu_init() they + * will wait for do_wait_cpu_initialized() to set their bit in + * smp_callout_mask to release them. + */ cpu_init_secondary(); rcu_cpu_starting(raw_smp_processor_id()); x86_cpuinit.early_percpu_clock_init(); + + /* + * Sync point with do_wait_cpu_callin(). The AP doesn't wait here + * but just sets the bit to let the controlling CPU (BSP) know that + * it's got this far. + */ smp_callin(); enable_start_cpu0 = 0; /* otherwise gcc will move up smp_processor_id before the cpu_init */ barrier(); + /* - * Check TSC synchronization with the boot CPU: + * Check TSC synchronization with the boot CPU (or whichever CPU + * is controlling the bringup). It will do its part of this from + * do_wait_cpu_online(), making it an implicit sync point. */ check_tsc_sync_target(); @@ -255,6 +275,7 @@ static void notrace start_secondary(void *unused) * half valid vector space. */ lock_vector_lock(); + /* Sync point with do_wait_cpu_online() */ set_cpu_online(smp_processor_id(), true); lapic_online(); unlock_vector_lock(); @@ -1084,7 +1105,6 @@ static int do_boot_cpu(int apicid, int cpu, struct task_struct *idle, unsigned long start_ip = real_mode_header->trampoline_start; unsigned long boot_error = 0; - unsigned long timeout; #ifdef CONFIG_X86_64 /* If 64-bit wakeup method exists, use the 64-bit mode trampoline IP */ @@ -1145,55 +1165,94 @@ static int do_boot_cpu(int apicid, int cpu, struct task_struct *idle, boot_error = wakeup_cpu_via_init_nmi(cpu, start_ip, apicid, cpu0_nmi_registered); - if (!boot_error) { - /* - * Wait 10s total for first sign of life from AP - */ - boot_error = -1; - timeout = jiffies + 10*HZ; - while (time_before(jiffies, timeout)) { - if (cpumask_test_cpu(cpu, cpu_initialized_mask)) { - /* - * Tell AP to proceed with initialization - */ - cpumask_set_cpu(cpu, cpu_callout_mask); - boot_error = 0; - break; - } - schedule(); - } - } + return boot_error; +} - if (!boot_error) { - /* - * Wait till AP completes initial initialization - */ - while (!cpumask_test_cpu(cpu, cpu_callin_mask)) { - /* - * Allow other tasks to run while we wait for the - * AP to come online. This also gives a chance - * for the MTRR work(triggered by the AP coming online) - * to be completed in the stop machine context. - */ - schedule(); - } +static int do_wait_cpu_cpumask(unsigned int cpu, const struct cpumask *mask) +{ + unsigned long timeout; + + /* + * Wait up to 10s for the CPU to report in. + */ + timeout = jiffies + 10*HZ; + while (time_before(jiffies, timeout)) { + if (cpumask_test_cpu(cpu, mask)) + return 0; + + schedule(); } + return -1; +} - if (x86_platform.legacy.warm_reset) { - /* - * Cleanup possible dangling ends... - */ - smpboot_restore_warm_reset_vector(); +/* + * Bringup step two: Wait for the target AP to reach cpu_init_secondary() + * and thus wait_for_master_cpu(), then set cpu_callout_mask to allow it + * to proceed. The AP will then proceed past setting its 'callin' bit + * and end up waiting in check_tsc_sync_target() until we reach + * do_wait_cpu_online() to tend to it. + */ +static int do_wait_cpu_initialized(unsigned int cpu) +{ + /* + * Wait for first sign of life from AP. + */ + if (do_wait_cpu_cpumask(cpu, cpu_initialized_mask)) + return -1; + + cpumask_set_cpu(cpu, cpu_callout_mask); + return 0; +} + +/* + * Bringup step three: Wait for the target AP to reach smp_callin(). + * The AP is not waiting for us here so we don't need to parallelise + * this step. Not entirely clear why we care about this, since we just + * proceed directly to TSC synchronization which is the next sync + * point with the AP anyway. + */ +static int do_wait_cpu_callin(unsigned int cpu) +{ + /* + * Wait till AP completes initial initialization. + */ + return do_wait_cpu_cpumask(cpu, cpu_callin_mask); +} + +/* + * Bringup step four: Synchronize the TSC and wait for the target AP + * to reach set_cpu_online() in start_secondary(). + */ +static int do_wait_cpu_online(unsigned int cpu) +{ + unsigned long flags; + + /* + * Check TSC synchronization with the AP (keep irqs disabled + * while doing so): + */ + local_irq_save(flags); + check_tsc_sync_source(cpu); + local_irq_restore(flags); + + /* + * Wait for the AP to mark itself online. Not entirely + * clear why we care, since the generic cpuhp code will + * wait for it to each CPUHP_AP_ONLINE_IDLE before going + * ahead with the rest of the bringup anyway. + */ + while (!cpu_online(cpu)) { + cpu_relax(); + touch_nmi_watchdog(); } - return boot_error; + return 0; } -int native_cpu_up(unsigned int cpu, struct task_struct *tidle) +int do_cpu_up(unsigned int cpu, struct task_struct *tidle) { int apicid = apic->cpu_present_to_apicid(cpu); int cpu0_nmi_registered = 0; - unsigned long flags; int err, ret = 0; lockdep_assert_irqs_enabled(); @@ -1240,19 +1299,6 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) goto unreg_nmi; } - /* - * Check TSC synchronization with the AP (keep irqs disabled - * while doing so): - */ - local_irq_save(flags); - check_tsc_sync_source(cpu); - local_irq_restore(flags); - - while (!cpu_online(cpu)) { - cpu_relax(); - touch_nmi_watchdog(); - } - unreg_nmi: /* * Clean up the nmi handler. Do this after the callin and callout sync @@ -1264,6 +1310,34 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) return ret; } +int native_cpu_up(unsigned int cpu, struct task_struct *tidle) +{ + int ret; + + ret = do_cpu_up(cpu, tidle); + if (ret) + return ret; + + ret = do_wait_cpu_initialized(cpu); + if (ret) + return ret; + + ret = do_wait_cpu_callin(cpu); + if (ret) + return ret; + + ret = do_wait_cpu_online(cpu); + + if (x86_platform.legacy.warm_reset) { + /* + * Cleanup possible dangling ends... + */ + smpboot_restore_warm_reset_vector(); + } + + return ret; +} + /** * arch_disable_smp_support() - disables SMP support for x86 at runtime */ From patchwork Wed Feb 1 20:43:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 51563 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp502887wrn; Wed, 1 Feb 2023 12:47:24 -0800 (PST) X-Google-Smtp-Source: AK7set8DupeBuTB8IVw9Pjhy5F0iSRRnVE7aWVH2GrB7EqfauyHvNAmM+3dtfQOcj+4q1x6VvRi/ X-Received: by 2002:a17:903:1251:b0:196:88e0:ea11 with SMTP id u17-20020a170903125100b0019688e0ea11mr4766138plh.35.1675284444452; Wed, 01 Feb 2023 12:47:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675284444; cv=none; d=google.com; s=arc-20160816; b=T34pnZVHHlMAGOqaZSYbolGwD8WNzWYN0doA+05IkEt0S+n4OEcERG1nq+mFz2OWKs zyyjTC4RwUEN+4oHTiPigQlx6BLKlK8kAuq5xyP0a2vMrWPj1IgeoVCiflzUlcv1v4y/ 3pKObEyw/fCTv0MGD5MVY8mj2nHDDo/CojLO5OGhKBp6RZWgl/KDUC0pg41M0QtlvZ+E 0iDhpZIvCzp0GhgVcWnPAbh87Na5O5wNk1dTjeSiufd4S/SzIN2t+aofIxCkCKxmcfNp aHszxyLnGFYgS/M6UAdD3flN9IAz6Zw+bDQwbQUCl9UZ+PL71Wnu/zd3WNPb8adstTyz G9cQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=1hZJEiSfBX2wQvagNcOApmVWpKDmG3BeofJQJZakBEQ=; b=CpaQL6nt8XRsS1dFa2+F5+pBo0NWZ35711fdASq1tG1zTlkThIXe+UFPatbrh67cET opj+fbw+PJyi11jmJw7dQYevtNDfekEs+4nmDpAV0sZ8RlpDvA/EFMbePmjXMdQt7XBQ O+o4ni6a3HF2JX7Vn5jG6AxtOA0H2QyOV0c3IdWs6MkVgTEmZQN3ncYhBV+LRxFLRtsH ClKIItnJsDBPLc8DuR3PTFt9lbq+HBeBlYLAHQrjCS3oWnQ6XNrmTCdIIAfZ+5MyOEee 6eutI9TQfDbtG4yH+pRKb28u57L8pmUvjY1ujvQRSLZcC6xlFXkiKFGkrCcJjDQ9Tgk5 KmYw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=zJiYzcq9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id x11-20020a170902e04b00b001978f58e452si7222173plx.309.2023.02.01.12.47.12; Wed, 01 Feb 2023 12:47:24 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=zJiYzcq9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231790AbjBAUqP (ORCPT + 99 others); Wed, 1 Feb 2023 15:46:15 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53638 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231339AbjBAUpf (ORCPT ); Wed, 1 Feb 2023 15:45:35 -0500 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6C5587A4B1 for ; Wed, 1 Feb 2023 12:44:54 -0800 (PST) Received: by mail-wr1-x42a.google.com with SMTP id y1so18518744wru.2 for ; Wed, 01 Feb 2023 12:44:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1hZJEiSfBX2wQvagNcOApmVWpKDmG3BeofJQJZakBEQ=; b=zJiYzcq95uUppV8U6o/cbPusjbNQE0o3HbAIMVOiv8LCM8VygpLlZ5GDbSegiWpg5m DUK+f3RMDblHEw4eyuXVDoIFxoNlPu8uoDWi/ziyTsnnSXX/5/mckEK7GJv3jXVAycei 3RyzBNj9GkalObeHryYwS606ZU84B5O5gnysw3h3/xYwE+GqaUz2c5mwVsaUQYnIjaOb f9RSwG8hsrn65hvOHCQzy+z60nj9clVFtrJnWBacrMFVyvOyZHVb+LOmncIYzy5IJ9os tbeRKOWqMwAJRJD7s+R0mbYHgcuAqd708npbNyExf2ytnGFoZasNO9AkGu1tzWaGRTXt BQdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1hZJEiSfBX2wQvagNcOApmVWpKDmG3BeofJQJZakBEQ=; b=lx5mz7UwwWP4gJr2z3zoI6xc9Ak27ODo0x80hlE1xc4ukYKbKnnNQz5Xoml18LjUg/ ryiJsMDYPONzdXxxWDP/k75HOh/7hV2kHACeuzfPKOtjzcRCDAi0k3yexCE2UDQlvhi3 TCPpv3Qqw+edcE7fVOiIZwWqI3cyDxk+UQ6UK6PCTzvrej/d0wmR9ObeBdDnOiyIDsrt 8Gi0z0B3Wryjhb0lcAJ/Tr/EaJCz0dUH4rm5SNqAXqi8QkoS0URQI0ZsE0Sg1iH1QsP6 DfQcErk0xMqC2hmckn3hshme6t5XysMxwbmGbfMdBxAOwirMMVB8GW+SHV1NVjiSGgNB 5emA== X-Gm-Message-State: AO0yUKVN7YSUzr5fFXMMBQ8FmapfHNVdALaJPyKgNFZoah3OgKpxQkZA RbylminGEHiYVdbQm7MRPIQxxA== X-Received: by 2002:adf:f911:0:b0:2bd:e7a0:6b5e with SMTP id b17-20020adff911000000b002bde7a06b5emr3423929wrr.40.1675284292702; Wed, 01 Feb 2023 12:44:52 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:7611:c340:3d8d:d46c]) by smtp.gmail.com with ESMTPSA id n15-20020a5d598f000000b002bdff778d87sm19993584wri.34.2023.02.01.12.44.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Feb 2023 12:44:52 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de Cc: mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, Usama Arif Subject: [PATCH 6/9] x86/smpboot: Support parallel startup of secondary CPUs Date: Wed, 1 Feb 2023 20:43:35 +0000 Message-Id: <20230201204338.1337562-7-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230201204338.1337562-1-usama.arif@bytedance.com> References: <20230201204338.1337562-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756663061805841292?= X-GMAIL-MSGID: =?utf-8?q?1756663061805841292?= From: Thomas Gleixner To allow for parallel AP bringup, we need to avoid the use of global variables for passing information to the APs, as well as preventing them from all trying to use the same real-mode stack simultaneously. So, introduce a 'lock' field in struct trampoline_header to use as a simple bit-spinlock for the real-mode stack. That lock also protects the global variables initial_gs, initial_stack and early_gdt_descr, which can now be calculated... So how do we calculate those addresses? Well, they they can all be found from the per_cpu data for this CPU. Simples! Except... how does it know what its CPU# is? OK, we export the cpuid_to_apicid[] array and it can search it to find its APIC ID in there. But now you whine at me that it doesn't even know its APIC ID? Well, if it's a relatively modern CPU then the APIC ID is in CPUID leaf 0x0B so we can use that. Otherwise... erm... OK, otherwise it can't have parallel CPU bringup for now. We'll still use a global variable for those CPUs and bring them up one at a time. So add a global 'smpboot_control' field which either contains the APIC ID, or a flag indicating that it can be found in CPUID. This adds the 'do_parallel_bringup' flag in preparation but doesn't actually enable parallel bringup yet. [ dwmw2: Minor tweaks, write a commit message ] [ seanc: Fix stray override of initial_gs in common_cpu_up() ] [ Usama Arif: Disable parallel bringup for AMD CPUs] Not-signed-off-by: Thomas Gleixner Signed-off-by: David Woodhouse Signed-off-by: Usama Arif --- arch/x86/include/asm/realmode.h | 3 ++ arch/x86/include/asm/smp.h | 9 +++- arch/x86/kernel/acpi/sleep.c | 1 + arch/x86/kernel/apic/apic.c | 2 +- arch/x86/kernel/head_64.S | 73 ++++++++++++++++++++++++++++ arch/x86/kernel/smpboot.c | 35 +++++++++++-- arch/x86/realmode/init.c | 3 ++ arch/x86/realmode/rm/trampoline_64.S | 14 ++++++ kernel/smpboot.c | 2 +- 9 files changed, 135 insertions(+), 7 deletions(-) diff --git a/arch/x86/include/asm/realmode.h b/arch/x86/include/asm/realmode.h index fd6f6e5b755a..503336462264 100644 --- a/arch/x86/include/asm/realmode.h +++ b/arch/x86/include/asm/realmode.h @@ -52,6 +52,7 @@ struct trampoline_header { u64 efer; u32 cr4; u32 flags; + u32 lock; #endif }; @@ -65,6 +66,8 @@ extern unsigned long initial_stack; extern unsigned long initial_vc_handler; #endif +extern u32 *trampoline_lock; + extern unsigned char real_mode_blob[]; extern unsigned char real_mode_relocs[]; diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h index a73bced40e24..58ddb2f78092 100644 --- a/arch/x86/include/asm/smp.h +++ b/arch/x86/include/asm/smp.h @@ -201,5 +201,12 @@ extern void nmi_selftest(void); #define nmi_selftest() do { } while (0) #endif -#endif /* __ASSEMBLY__ */ +extern unsigned int smpboot_control; + +#endif /* !__ASSEMBLY__ */ + +/* Control bits for startup_64 */ +#define STARTUP_PARALLEL 0x80000000 +#define STARTUP_SECONDARY 0x40000000 + #endif /* _ASM_X86_SMP_H */ diff --git a/arch/x86/kernel/acpi/sleep.c b/arch/x86/kernel/acpi/sleep.c index 3b7f4cdbf2e0..06adf340a0f1 100644 --- a/arch/x86/kernel/acpi/sleep.c +++ b/arch/x86/kernel/acpi/sleep.c @@ -115,6 +115,7 @@ int x86_acpi_suspend_lowlevel(void) early_gdt_descr.address = (unsigned long)get_cpu_gdt_rw(smp_processor_id()); initial_gs = per_cpu_offset(smp_processor_id()); + smpboot_control = 0; #endif initial_code = (unsigned long)wakeup_long64; saved_magic = 0x123456789abcdef0L; diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c index c6876d3ea4b1..a841c4b4083f 100644 --- a/arch/x86/kernel/apic/apic.c +++ b/arch/x86/kernel/apic/apic.c @@ -2374,7 +2374,7 @@ static int nr_logical_cpuids = 1; /* * Used to store mapping between logical CPU IDs and APIC IDs. */ -static int cpuid_to_apicid[] = { +int cpuid_to_apicid[] = { [0 ... NR_CPUS - 1] = -1, }; diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index d860d437631b..dd931c049acb 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -25,6 +25,7 @@ #include #include #include +#include /* * We are not able to switch in one step to the final KERNEL ADDRESS SPACE @@ -241,6 +242,66 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) UNWIND_HINT_EMPTY ANNOTATE_NOENDBR // above + /* + * Is this the boot CPU coming up? If so everything is available + * in initial_gs, initial_stack and early_gdt_descr. + */ + movl smpboot_control(%rip), %eax + testl %eax, %eax + jz .Lsetup_cpu + + /* + * Secondary CPUs find out the offsets via the APIC ID. For parallel + * boot the APIC ID is retrieved from CPUID, otherwise it's encoded + * in smpboot_control: + * Bit 0-29 APIC ID if STARTUP_PARALLEL flag is not set + * Bit 30 STARTUP_SECONDARY flag + * Bit 31 STARTUP_PARALLEL flag (use CPUID 0x0b for APIC ID) + */ + testl $STARTUP_PARALLEL, %eax + jnz .Luse_cpuid_0b + andl $0x0FFFFFFF, %eax + jmp .Lsetup_AP + +.Luse_cpuid_0b: + mov $0x0B, %eax + xorl %ecx, %ecx + cpuid + mov %edx, %eax + +.Lsetup_AP: + /* EAX contains the APICID of the current CPU */ + xorl %ecx, %ecx + leaq cpuid_to_apicid(%rip), %rbx + +.Lfind_cpunr: + cmpl (%rbx), %eax + jz .Linit_cpu_data + addq $4, %rbx + addq $8, %rcx + jmp .Lfind_cpunr + +.Linit_cpu_data: + /* Get the per cpu offset */ + leaq __per_cpu_offset(%rip), %rbx + addq %rcx, %rbx + movq (%rbx), %rbx + /* Save it for GS BASE setup */ + movq %rbx, initial_gs(%rip) + + /* Calculate the GDT address */ + movq $gdt_page, %rcx + addq %rbx, %rcx + movq %rcx, early_gdt_descr_base(%rip) + + /* Find the idle task stack */ + movq $idle_threads, %rcx + addq %rbx, %rcx + movq (%rcx), %rcx + movq TASK_threadsp(%rcx), %rcx + movq %rcx, initial_stack(%rip) + +.Lsetup_cpu: /* * We must switch to a new descriptor in kernel space for the GDT * because soon the kernel won't have access anymore to the userspace @@ -281,6 +342,14 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) */ movq initial_stack(%rip), %rsp + /* Drop the realmode protection. For the boot CPU the pointer is NULL! */ + movq trampoline_lock(%rip), %rax + testq %rax, %rax + jz .Lsetup_idt + lock + btrl $0, (%rax) + +.Lsetup_idt: /* Setup and Load IDT */ pushq %rsi call early_setup_idt @@ -425,6 +494,7 @@ SYM_DATA(initial_vc_handler, .quad handle_vc_boot_ghcb) * reliably detect the end of the stack. */ SYM_DATA(initial_stack, .quad init_thread_union + THREAD_SIZE - FRAME_SIZE) +SYM_DATA(trampoline_lock, .quad 0); __FINITDATA __INIT @@ -659,6 +729,9 @@ SYM_DATA_END(level1_fixmap_pgt) SYM_DATA(early_gdt_descr, .word GDT_ENTRIES*8-1) SYM_DATA_LOCAL(early_gdt_descr_base, .quad INIT_PER_CPU_VAR(gdt_page)) + .align 16 +SYM_DATA(smpboot_control, .long 0) + .align 16 /* This must match the first entry in level2_kernel_pgt */ SYM_DATA(phys_base, .quad 0x0) diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 98604a3953e6..44e9f7ae5afc 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -799,6 +799,16 @@ static int __init cpu_init_udelay(char *str) } early_param("cpu_init_udelay", cpu_init_udelay); +static bool do_parallel_bringup = true; + +static int __init no_parallel_bringup(char *str) +{ + do_parallel_bringup = false; + + return 0; +} +early_param("no_parallel_bringup", no_parallel_bringup); + static void __init smp_quirk_init_udelay(void) { /* if cmdline changed it from default, leave it alone */ @@ -1086,8 +1096,6 @@ int common_cpu_up(unsigned int cpu, struct task_struct *idle) #ifdef CONFIG_X86_32 /* Stack for startup_32 can be just as for start_secondary onwards */ per_cpu(cpu_current_top_of_stack, cpu) = task_top_of_stack(idle); -#else - initial_gs = per_cpu_offset(cpu); #endif return 0; } @@ -1112,9 +1120,16 @@ static int do_boot_cpu(int apicid, int cpu, struct task_struct *idle, start_ip = real_mode_header->trampoline_start64; #endif idle->thread.sp = (unsigned long)task_pt_regs(idle); - early_gdt_descr.address = (unsigned long)get_cpu_gdt_rw(cpu); initial_code = (unsigned long)start_secondary; - initial_stack = idle->thread.sp; + + if (IS_ENABLED(CONFIG_X86_32)) { + early_gdt_descr.address = (unsigned long)get_cpu_gdt_rw(cpu); + initial_stack = idle->thread.sp; + } else if (do_parallel_bringup) { + smpboot_control = STARTUP_SECONDARY | STARTUP_PARALLEL; + } else { + smpboot_control = STARTUP_SECONDARY | apicid; + } /* Enable the espfix hack for this CPU */ init_espfix_ap(cpu); @@ -1516,6 +1531,18 @@ void __init native_smp_prepare_cpus(unsigned int max_cpus) speculative_store_bypass_ht_init(); + /* + * We can do 64-bit AP bringup in parallel if the CPU reports its + * APIC ID in CPUID leaf 0x0B. Otherwise it's too hard. And not + * for SEV-ES guests because they can't use CPUID that early. + * Also, some AMD CPUs crash when doing parallel cpu bringup, disable + * it for all AMD CPUs to be on the safe side. + */ + if (IS_ENABLED(CONFIG_X86_32) || boot_cpu_data.cpuid_level < 0x0B || + cc_platform_has(CC_ATTR_GUEST_STATE_ENCRYPT) || + boot_cpu_data.x86_vendor == X86_VENDOR_AMD) + do_parallel_bringup = false; + snp_set_wakeup_secondary_cpu(); } diff --git a/arch/x86/realmode/init.c b/arch/x86/realmode/init.c index 41d7669a97ad..8658ba456254 100644 --- a/arch/x86/realmode/init.c +++ b/arch/x86/realmode/init.c @@ -154,6 +154,9 @@ static void __init setup_real_mode(void) trampoline_header->flags = 0; + trampoline_lock = &trampoline_header->lock; + *trampoline_lock = 0; + trampoline_pgd = (u64 *) __va(real_mode_header->trampoline_pgd); /* Map the real mode stub as virtual == physical */ diff --git a/arch/x86/realmode/rm/trampoline_64.S b/arch/x86/realmode/rm/trampoline_64.S index e38d61d6562e..49ebc1636ffd 100644 --- a/arch/x86/realmode/rm/trampoline_64.S +++ b/arch/x86/realmode/rm/trampoline_64.S @@ -49,6 +49,19 @@ SYM_CODE_START(trampoline_start) mov %ax, %es mov %ax, %ss + /* + * Make sure only one CPU fiddles with the realmode stack + */ +.Llock_rm: + btl $0, tr_lock + jnc 2f + pause + jmp .Llock_rm +2: + lock + btsl $0, tr_lock + jc .Llock_rm + # Setup stack movl $rm_stack_end, %esp @@ -241,6 +254,7 @@ SYM_DATA_START(trampoline_header) SYM_DATA(tr_efer, .space 8) SYM_DATA(tr_cr4, .space 4) SYM_DATA(tr_flags, .space 4) + SYM_DATA(tr_lock, .space 4) SYM_DATA_END(trampoline_header) #include "trampoline_common.S" diff --git a/kernel/smpboot.c b/kernel/smpboot.c index 2c7396da470c..a18a21dff9bc 100644 --- a/kernel/smpboot.c +++ b/kernel/smpboot.c @@ -25,7 +25,7 @@ * For the hotplug case we keep the task structs around and reuse * them. */ -static DEFINE_PER_CPU(struct task_struct *, idle_threads); +DEFINE_PER_CPU(struct task_struct *, idle_threads); struct task_struct *idle_thread_get(unsigned int cpu) { From patchwork Wed Feb 1 20:43:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 51552 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp502403wrn; Wed, 1 Feb 2023 12:46:16 -0800 (PST) X-Google-Smtp-Source: AK7set8Ayb6GWFuqSeOj5JQagxsDapCut1c5jgg0nmr7VY0NFMsIaEPWOCRjFDqxy/EteoFy6dfq X-Received: by 2002:aa7:c403:0:b0:46c:8544:42be with SMTP id j3-20020aa7c403000000b0046c854442bemr3500326edq.5.1675284376406; Wed, 01 Feb 2023 12:46:16 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675284376; cv=none; d=google.com; s=arc-20160816; b=oeR9Hjk8g3M8RfpFy4Zhmdno6lYCY8q4KT4UNGRgkjoR5C1LqzCb5FM2nm5Maqi4vh 2sSzcMInYAgn1irZ2XzkTEsDvjQTG3c8CijT47VNM64ibi7nX6vHloXkXpQnnFpNXUHF 7X5pnxgXvki+3MCKSLj/sQh4g1IhKI1MwjE0PkvrQ8VhxYV77kDrPIXOqlxlYVzT25ON +DpTTHmaFX6ztYPTPzsEjWquaYOd7HUhYMvkokEikg5yadffEeUuowSVkthpapiQo4Za lCOWtfBxl3CcXYdqNhY5Dbog2AkQtkZEQs7v6j88qcS5q0i2U2aO0FVspnZ6DLfi3tSl p5Ag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=iTNF1JvfCDajsaqBdlIe78iIn+/3qna+HTWlbtJK4ls=; b=TWRIj66CoVJ1dHh/1oqXck9iNIS88JfA1wZ6M0DKt4NnrqOenaskOT7iU5hP05d5B/ ORBpNPAL9QbAwf7Sw8AUPRkWU1gfPs4xFNV0imFLz0yOQYlu5xu4o3nNC3ZRJaSE9V8Y Md57dlFC8hGw2FTCBge1TjhmjjonEUmZDsYoIBjiQdShoWEqE3BdKVbpjnp7FUVTgk+a Loqx3ZmpWx4+SLKz7qr0iWfZDm9EI23ua2SN7JfwzJ6DkWQ4UOuDW5skdrF2lwiehJCY Wn8W5eVdOBvj1Y40TmMYNO3Z5Og92rEbatD0zL9wGvu7YFXQLou+g4rzm5dCM79vJjqe LSJQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=PHsc2s0l; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a22-20020aa7cf16000000b0049defeaec6dsi389513edy.530.2023.02.01.12.45.29; Wed, 01 Feb 2023 12:46:16 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=PHsc2s0l; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230011AbjBAUpM (ORCPT + 99 others); Wed, 1 Feb 2023 15:45:12 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53698 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229957AbjBAUpJ (ORCPT ); Wed, 1 Feb 2023 15:45:09 -0500 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2873178ACE for ; Wed, 1 Feb 2023 12:44:55 -0800 (PST) Received: by mail-wr1-x42a.google.com with SMTP id m7so18494526wru.8 for ; Wed, 01 Feb 2023 12:44:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=iTNF1JvfCDajsaqBdlIe78iIn+/3qna+HTWlbtJK4ls=; b=PHsc2s0l5yODha3GL0699EVouPw2t2VyZ/NpijjKErd/Wse0mkuoPmiTOTBFK5druj O+8z9IC64HBK0PtRhvS00L1CHOP2Tt1XQgtckGVJrZCTenGyOpEdLF2JkiQ5W6uThjvg tMwTIAehAFNjejO5Fr1XdrMjLCqVEk0WzwMrrB1yFj9MMznXnzgGoYBfY9Q/uxOzjYYW iEpO5m6XjptcngDjwzD6aUIAmWwlDi3xWVhjEBAqmAIBiPGHAO6Z+AQUXn6HQwvkOdfm I8kjD6IWRNSzffMbkewI0/3WzZrr5qqOUkRNV+5PWo7liabF2u7hWJKLTAXVYKwlCHV0 w+/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iTNF1JvfCDajsaqBdlIe78iIn+/3qna+HTWlbtJK4ls=; b=sSbDhwZVULgb0lnpZTIYRO3V4b+qwL3s4mW4O0qZmzz+jgZc5qnnY7y2j7bxLAX0DL MbkYgadzfdJRCOLRNOS1Bn/B+ZgjviLlOwqs4fyD6X5n4+0m5Tv71uFbu+i2HnzCuMRn o5kP5oou0nYGufa86RICLvCpXKkKdRCMCZHJTHMWOICLacQ3E+mceM+5YrC3IRZVwrqA NDoX2roaWMNuZ+0JMG/UKpt5PmlJSCTsbrJ7xAbG3zQY8ed4eZ5J6cyhVEsTPqcXSLZi YHuBIE06pYOmq2K6TK3obR2uGXNUymz884flqv1Pau8nHaQVJjiqeVkMl3aU6Mf+8IZh I8iA== X-Gm-Message-State: AO0yUKW1Y7mVJqb1QcdWjG4xWT9U2XaY3mbRnOR+PSqJn2uW4iTM+zpl +/LSjzEvVpbaIHiY2cPaGoIvcA== X-Received: by 2002:a5d:560e:0:b0:2bf:cfb4:817c with SMTP id l14-20020a5d560e000000b002bfcfb4817cmr3484247wrv.35.1675284293632; Wed, 01 Feb 2023 12:44:53 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:7611:c340:3d8d:d46c]) by smtp.gmail.com with ESMTPSA id n15-20020a5d598f000000b002bdff778d87sm19993584wri.34.2023.02.01.12.44.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Feb 2023 12:44:53 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de Cc: mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse , Usama Arif Subject: [PATCH 7/9] x86/smpboot: Send INIT/SIPI/SIPI to secondary CPUs in parallel Date: Wed, 1 Feb 2023 20:43:36 +0000 Message-Id: <20230201204338.1337562-8-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230201204338.1337562-1-usama.arif@bytedance.com> References: <20230201204338.1337562-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756662990435677322?= X-GMAIL-MSGID: =?utf-8?q?1756662990435677322?= From: David Woodhouse When the APs can find their own APIC ID without assistance, we can do the AP bringup in parallel. Register a CPUHP_BP_PARALLEL_DYN stage "x86/cpu:kick" which just calls do_boot_cpu() to deliver INIT/SIPI/SIPI to each AP in turn before the normal native_cpu_up() does the rest of the hand-holding. The APs will then take turns through the real mode code (which has its own bitlock for exclusion) until they make it to their own stack, then proceed through the first few lines of start_secondary() and execute these parts in parallel: start_secondary() -> cr4_init() -> (some 32-bit only stuff so not in the parallel cases) -> cpu_init_secondary() -> cpu_init_exception_handling() -> cpu_init() -> wait_for_master_cpu() At this point they wait for the BSP to set their bit in cpu_callout_mask (from do_wait_cpu_initialized()), and release them to continue through the rest of cpu_init() and beyond. This reduces the time taken for bringup on my 28-thread Haswell system from about 120ms to 80ms. On a socket 96-thread Skylake it takes the bringup time from 500ms to 100ms. There is more speedup to be had by doing the remaining parts in parallel too — especially notify_cpu_starting() in which the AP takes itself through all the stages from CPUHP_BRINGUP_CPU to CPUHP_ONLINE. But those require careful auditing to ensure they are reentrant, before we can go that far. [Usama Arif: fixed rebase conflict] Signed-off-by: David Woodhouse Signed-off-by: Usama Arif --- arch/x86/kernel/smpboot.c | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 44e9f7ae5afc..060813411f85 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -1329,9 +1329,12 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) { int ret; - ret = do_cpu_up(cpu, tidle); - if (ret) - return ret; + /* If parallel AP bringup isn't enabled, perform the first steps now. */ + if (!do_parallel_bringup) { + ret = do_cpu_up(cpu, tidle); + if (ret) + return ret; + } ret = do_wait_cpu_initialized(cpu); if (ret) @@ -1353,6 +1356,12 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) return ret; } +/* Bringup step one: Send INIT/SIPI to the target AP */ +static int native_cpu_kick(unsigned int cpu) +{ + return do_cpu_up(cpu, idle_thread_get(cpu)); +} + /** * arch_disable_smp_support() - disables SMP support for x86 at runtime */ @@ -1543,6 +1552,10 @@ void __init native_smp_prepare_cpus(unsigned int max_cpus) boot_cpu_data.x86_vendor == X86_VENDOR_AMD) do_parallel_bringup = false; + if (do_parallel_bringup) + cpuhp_setup_state_nocalls(CPUHP_BP_PARALLEL_DYN, "x86/cpu:kick", + native_cpu_kick, NULL); + snp_set_wakeup_secondary_cpu(); } From patchwork Wed Feb 1 20:43:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 51561 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp502797wrn; Wed, 1 Feb 2023 12:47:12 -0800 (PST) X-Google-Smtp-Source: AK7set9Icg1K838VnYv9RwnwoRtm/1DjhB3yPmwewLnNmWIZNIJ5yQ5Y1dXTrkGdODInL866O7/q X-Received: by 2002:a05:6a00:2491:b0:593:ec73:424d with SMTP id c17-20020a056a00249100b00593ec73424dmr4439921pfv.8.1675284432623; Wed, 01 Feb 2023 12:47:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675284432; cv=none; d=google.com; s=arc-20160816; b=DeDR++zk+dZdWuTCIckWueSgSd1+GPcSrESaGZdnBePkg+TLA0JBfDiGNLIif2fY9m Ejb6+At9htaO0LPLMdGjYdbHRtpYU0KGE2jt8V+es6RsXtgcjjBNWIp/yx0H1w70oGbc avoSbs+CP7NOPuJ3brayu7sQeDXF6ponC/XbHtGMANqcVI/B6iUG8uEP3LCZOoP5ut14 cDWzBRSWixzr3f/GBh9plbBWR2lZQx25aayuT2rb3qgoPtCKJs8qVNI/33op9LFVOOtY 29fCuEjBSn6EFh/nG2VUz6gMS3JkQ3L012HWfOP2PK60waAK8jEuXV+Sd7U07X+PXYRH hBpQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Si9kzr7Ck9J5FJkWCPsjkhUgmegMqi5GAbSuKxlmi/Y=; b=LXI6bvcIDFhYM33ixDObh1KkZ9fCEJZNgKfwohJIY0lHMST8UYVqZC5ciRxv3te5Fo KPkDoE+Om7uv8REGP/ftC7Gt6eMcUJge0FJrgCGuE+qqbs/WDzr0O/SKmAImd9/ZO1tQ 8tqlMAfcEJsHzo7DPdq01Y6tZ3DWCW6SDLzvaqa8SO3tYrBh+lj9ikoD1lX7s57EIay/ GAZ5ccFHJJwEHbAmqqgP63BndsWxCO9nz0rG5EFVlOTB3l0ApEk3OaJwjQzRV1MhaPjc 3TgMwaZtCxbjL/44oBLrrYG2kbfRVxC86qdKcBfJWTA8xMmjU4gJv8oRXX1dQEBDs9+W fb+Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b="sXMxj/CK"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a204-20020a621ad5000000b00592591f1972si19480878pfa.64.2023.02.01.12.46.56; Wed, 01 Feb 2023 12:47:12 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b="sXMxj/CK"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231778AbjBAUqL (ORCPT + 99 others); Wed, 1 Feb 2023 15:46:11 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53726 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231357AbjBAUpf (ORCPT ); Wed, 1 Feb 2023 15:45:35 -0500 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4148978AEB for ; Wed, 1 Feb 2023 12:44:56 -0800 (PST) Received: by mail-wr1-x42f.google.com with SMTP id t7so10140833wrp.5 for ; Wed, 01 Feb 2023 12:44:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Si9kzr7Ck9J5FJkWCPsjkhUgmegMqi5GAbSuKxlmi/Y=; b=sXMxj/CKJBGFdLzPqQuDQiOPwKF/J04CKNI5iVbGlsHYa+yHZsQXZ/welJivcDAh78 03XN/aIk9smRRSJBbVcD8A7xCJQW537kgAuRtyai14FS8t0ukXFaqkwlkudC99dkmvdN rtlhlvW0kPQwaNg8Yz96FxnRXUt/hZZ4kDgRUlcTj6wrqyrl6sv/drdH7ZlZp7N0xpgr 7/F+R8t/koV++BDvOKRHqSKkhTRC4E1+ivjSfL+Y8IvbVH6WudoBt9K7CLfIp39iEZO0 iRMQLwcQy13UYZ9od7HzZ4VJNpiv6qQnkv1bEA/lGY/7VjTyfJn1TzJcGWgdw61ICEHi 8K6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Si9kzr7Ck9J5FJkWCPsjkhUgmegMqi5GAbSuKxlmi/Y=; b=mSl9Ss1kYMYSWhn52Oie8qyeElB5+MZGhihCtrtDR3yt3YOF0VQybmXSSfRo17jV6m FmONZgnZMjRGHCvSSBa+9AxFyQvsw8cRzEa06nYIyL/M22sdcqJHL1XGKes7pyDkc2Ie /PnNBSHxrDmgTX//ak8QNApIxUS35xL93m9c0HMLiadwFOkBnaYfNawJ2gc0hX/BeuoJ uMsR9dgDMU4AThXUWfbTdILs8BlaS68zaM8NCzQRMndFj3p7WIV3gbng8s8sXBXHlEua jq5jn9AxoyjEMmXYF5OEQEUVpcd+oTweo+cZoO+7TaroXo8S1eUvmbTBWSCT6hhuFnGi VatA== X-Gm-Message-State: AO0yUKXI7S/7qIMNpVOBT3pwG9tqbuVq7KwXg0b+TnpLQey5QDyF/b5E 0izwzK0UsJMCriR/LSVgxrctIQ== X-Received: by 2002:adf:fb8f:0:b0:2c1:28dc:1566 with SMTP id a15-20020adffb8f000000b002c128dc1566mr99026wrr.41.1675284294722; Wed, 01 Feb 2023 12:44:54 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:7611:c340:3d8d:d46c]) by smtp.gmail.com with ESMTPSA id n15-20020a5d598f000000b002bdff778d87sm19993584wri.34.2023.02.01.12.44.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Feb 2023 12:44:54 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de Cc: mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse Subject: [PATCH 8/9] x86/mtrr: Avoid repeated save of MTRRs on boot-time CPU bringup Date: Wed, 1 Feb 2023 20:43:37 +0000 Message-Id: <20230201204338.1337562-9-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230201204338.1337562-1-usama.arif@bytedance.com> References: <20230201204338.1337562-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756663049415156923?= X-GMAIL-MSGID: =?utf-8?q?1756663049415156923?= From: David Woodhouse There's no need to repeatedly save the BSP's MTRRs for each AP we bring up at boot time. And there's no need to use smp_call_function_single() even for the one time we *do* want to do it. Signed-off-by: David Woodhouse --- arch/x86/kernel/cpu/mtrr/mtrr.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/arch/x86/kernel/cpu/mtrr/mtrr.c b/arch/x86/kernel/cpu/mtrr/mtrr.c index 2746cac9d8a9..2884017586f1 100644 --- a/arch/x86/kernel/cpu/mtrr/mtrr.c +++ b/arch/x86/kernel/cpu/mtrr/mtrr.c @@ -814,11 +814,20 @@ void mtrr_ap_init(void) */ void mtrr_save_state(void) { + static bool mtrr_saved; int first_cpu; if (!mtrr_enabled()) return; + if (system_state < SYSTEM_RUNNING) { + if (!mtrr_saved) { + mtrr_save_fixed_ranges(NULL); + mtrr_saved = true; + } + return; + } + first_cpu = cpumask_first(cpu_online_mask); smp_call_function_single(first_cpu, mtrr_save_fixed_ranges, NULL, 1); } From patchwork Wed Feb 1 20:43:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 51551 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp502340wrn; Wed, 1 Feb 2023 12:46:09 -0800 (PST) X-Google-Smtp-Source: AK7set9YIwIFGv9+typSU/xEmWCmhDIU2mXhF8qsYsjuGOEnd2VRaYe3e2el4+l1pO5jnRTPOUZ6 X-Received: by 2002:a17:902:e2c3:b0:196:2313:12c6 with SMTP id l3-20020a170902e2c300b00196231312c6mr3217876plc.47.1675284369509; Wed, 01 Feb 2023 12:46:09 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675284369; cv=none; d=google.com; s=arc-20160816; b=TWHYOLEbt7wfpdH2Y/GamP6pPvZYqzvDObqMXAmm+t/cL/UbeWtMu0TMAHCjTFKt8V 6nRRRtJukZDiFoOg0627Aqa6ERX/B26gufQoCvdVCgJeax+vC4jNlNSyjgC3w+p7NhTf TNl93RdMSIPNw7Qxw++rlWIqFJZvNDTbYHiGvN/eGUNXZArzOuOnNFixrPGGj4RmbLEf 2+3vYpgY4h969q1DNy7tCw1L+XqnPM0C8eq2wSqq1ihJ/ieu6vItK24Oz+BnaDms/1OT 6+yHuG1PKKAY31y5oy9BQcoFxdBJ6jBMy49t8S7RXqt1MvsrMmZ7nZCDK+KVPhmpCj0d PDpA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=hgM55XxpY35W22axaOPhRqoufjGi9YkYop3PvYcIKxw=; b=sRNwO1Rzoo3ZNY/H/CyQEnDPsaPvAMMKPT6RyEY9ohgyaaDVsPrYfJhZPzWPQjG77I 2522Zp+VRMbBeAZh/1UB9pJCNLbrgZ3yUwNjXCgHzf7McEasUjgA4s9m5VQuUQzqv8R8 nk0zrbWL1/Z8SREkO2d+74InlZzh+rYFBqKs1klzLvQHKULxnBX3uebOjD4D2dvd3XVN UxLb5GG2gZkdqYFJCKqDtHeljkI4Fc80H8KK4Zrkc/8LJ6LWRDUROGxpxBmPsGCOWxYg 6Zq+MxLqdIK+ZlBezu8aii/Lr2qy51a+ZMEwNhEHAzAV/rWGdu/C1hQHURPTkHyiGZcQ OiVA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=xWr4+4P1; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a4-20020a170902ecc400b001968b68d6aasi9804564plh.448.2023.02.01.12.45.34; Wed, 01 Feb 2023 12:46:09 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=xWr4+4P1; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230156AbjBAUpS (ORCPT + 99 others); Wed, 1 Feb 2023 15:45:18 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53742 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230046AbjBAUpN (ORCPT ); Wed, 1 Feb 2023 15:45:13 -0500 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3D45378AC3 for ; Wed, 1 Feb 2023 12:44:57 -0800 (PST) Received: by mail-wr1-x435.google.com with SMTP id y1so18518842wru.2 for ; Wed, 01 Feb 2023 12:44:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hgM55XxpY35W22axaOPhRqoufjGi9YkYop3PvYcIKxw=; b=xWr4+4P1cRv+8rYBvo5f6LU8uVU3/jfybo/4gOp7QcHTptt60DLyNSy0shHIb4Ro8y TMVNdbUNSOq09AVWo88YdasD9O8dP4UlDRyWk368j7WBIvk+ibAmcVOGI3OMD8ZzY3bu Fu+r/Vt1kqQ9vpFqJ6kW16KaYFDHqdOz7d4YycsrCYAxfEj1FMB/cncdhzxz52HLYP7O j6z/36RxhG/gS7w36IJGHZclBojKi/KMZkDbiDUv0l2SdYEgAfwuOVE1Qqx0LK26iygy mvFzy+s7VNAQKvG7erbKSyFj7YqnpjgVZUsuK8GzzOnW77kVdVMHOX/+wVtNHPGGrSmK PZ1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hgM55XxpY35W22axaOPhRqoufjGi9YkYop3PvYcIKxw=; b=JaadPMpvwpcLTEYvwpZF8Iwo8hyubvvmCkzH8pl3sTromPT0oV8ocCZfG9U6dAnHZM w2XKzWhTqyibJK2UrpBkpTtCtoz/t870zeTYIvpYnmcJODqAYk3b2Fy6Ep74kZZFYEwJ iZ9r1sKf1p2f0Bi+llDvaswFV0rXHICg3S4EcR59Muuzh9Vovud68SptaGzuGjXm+Ge2 mrGldJKBpvqpUAvqJD4TbWLijqWw29Lw88qmgEGUdRE66xRthGWTsCYevz9pXoRpOVkQ ml/Hf9vkJ2aM2LyTR6dnbjaWsd5n4E8eW7S80SETpAz6jIEOB4ln9WtaLzQKMHzbvenS CwGQ== X-Gm-Message-State: AO0yUKWolFZudEcSEcyRDRGFuwVyNW69wW8SVUj88hiQs3W/N+EYQwO6 Gs5zRioM4M8+jr1zjIcLpxEELA== X-Received: by 2002:adf:dc0a:0:b0:2bf:f15c:e331 with SMTP id t10-20020adfdc0a000000b002bff15ce331mr130538wri.10.1675284295618; Wed, 01 Feb 2023 12:44:55 -0800 (PST) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:7611:c340:3d8d:d46c]) by smtp.gmail.com with ESMTPSA id n15-20020a5d598f000000b002bdff778d87sm19993584wri.34.2023.02.01.12.44.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Feb 2023 12:44:55 -0800 (PST) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de Cc: mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, David Woodhouse , Usama Arif Subject: [PATCH 9/9] x86/smpboot: Serialize topology updates for secondary bringup Date: Wed, 1 Feb 2023 20:43:38 +0000 Message-Id: <20230201204338.1337562-10-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230201204338.1337562-1-usama.arif@bytedance.com> References: <20230201204338.1337562-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756662983134169425?= X-GMAIL-MSGID: =?utf-8?q?1756662983134169425?= From: David Woodhouse If we bring up secondaries in parallel they might get confused unless we impose some ordering here: [ 1.360149] x86: Booting SMP configuration: [ 1.360221] .... node #0, CPUs: #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15 #16 #17 #18 #19 #20 #21 #22 #23 [ 1.366225] .... node #1, CPUs: #24 #25 #26 #27 #28 #29 #30 #31 #32 #33 #34 #35 #36 #37 #38 #39 #40 #41 #42 #43 #44 #45 #46 #47 [ 1.370219] .... node #0, CPUs: #48 #49 #50 #51 #52 #53 #54 #55 #56 #57 #58 #59 #60 #61 #62 #63 #64 #65 #66 #67 #68 #69 #70 #71 [ 1.378226] .... node #1, CPUs: #72 #73 #74 #75 #76 #77 #78 #79 #80 #81 #82 #83 #84 #85 #86 #87 #88 #89 #90 #91 #92 #93 #94 #95 [ 1.382037] Brought 96 CPUs to x86/cpu:kick in 72232606 cycles [ 0.104104] smpboot: CPU 26 Converting physical 0 to logical die 1 [ 0.104104] smpboot: CPU 27 Converting physical 1 to logical package 2 [ 0.104104] smpboot: CPU 24 Converting physical 1 to logical package 3 [ 0.104104] smpboot: CPU 27 Converting physical 0 to logical die 2 [ 0.104104] smpboot: CPU 25 Converting physical 1 to logical package 4 [ 1.385609] Brought 96 CPUs to x86/cpu:wait-init in 9269218 cycles [ 1.395285] Brought CPUs online in 28930764 cycles [ 1.395469] smp: Brought up 2 nodes, 96 CPUs [ 1.395689] smpboot: Max logical packages: 2 [ 1.396222] smpboot: Total of 96 processors activated (576000.00 BogoMIPS) Do the full topology update in smp_store_cpu_info() under a spinlock to ensure that things remain consistent. [Usama Arif: fixed rebase conflict] Signed-off-by: David Woodhouse Signed-off-by: Usama Arif --- arch/x86/include/asm/smp.h | 4 +- arch/x86/include/asm/topology.h | 2 - arch/x86/kernel/cpu/common.c | 6 +-- arch/x86/kernel/smpboot.c | 73 ++++++++++++++++++++------------- arch/x86/xen/smp_pv.c | 4 +- 5 files changed, 48 insertions(+), 41 deletions(-) diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h index 58ddb2f78092..177532b392b1 100644 --- a/arch/x86/include/asm/smp.h +++ b/arch/x86/include/asm/smp.h @@ -49,8 +49,6 @@ struct smp_ops { }; /* Globals due to paravirt */ -extern void set_cpu_sibling_map(int cpu); - #ifdef CONFIG_SMP extern struct smp_ops smp_ops; @@ -138,7 +136,7 @@ void native_send_call_func_single_ipi(int cpu); void x86_idle_thread_init(unsigned int cpu, struct task_struct *idle); void smp_store_boot_cpu_info(void); -void smp_store_cpu_info(int id); +void smp_store_cpu_info(int id, bool force_single_core); asmlinkage __visible void smp_reboot_interrupt(void); __visible void smp_reschedule_interrupt(struct pt_regs *regs); diff --git a/arch/x86/include/asm/topology.h b/arch/x86/include/asm/topology.h index 458c891a8273..4bccbd949a99 100644 --- a/arch/x86/include/asm/topology.h +++ b/arch/x86/include/asm/topology.h @@ -136,8 +136,6 @@ static inline int topology_max_smt_threads(void) return __max_smt_threads; } -int topology_update_package_map(unsigned int apicid, unsigned int cpu); -int topology_update_die_map(unsigned int dieid, unsigned int cpu); int topology_phys_to_logical_pkg(unsigned int pkg); int topology_phys_to_logical_die(unsigned int die, unsigned int cpu); bool topology_is_primary_thread(unsigned int cpu); diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 3e508f239098..1561c0abdb93 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -1745,7 +1745,7 @@ static void generic_identify(struct cpuinfo_x86 *c) * Validate that ACPI/mptables have the same information about the * effective APIC id and update the package map. */ -static void validate_apic_and_package_id(struct cpuinfo_x86 *c) +static void validate_apic_id(struct cpuinfo_x86 *c) { #ifdef CONFIG_SMP unsigned int apicid, cpu = smp_processor_id(); @@ -1756,8 +1756,6 @@ static void validate_apic_and_package_id(struct cpuinfo_x86 *c) pr_err(FW_BUG "CPU%u: APIC id mismatch. Firmware: %x APIC: %x\n", cpu, apicid, c->initial_apicid); } - BUG_ON(topology_update_package_map(c->phys_proc_id, cpu)); - BUG_ON(topology_update_die_map(c->cpu_die_id, cpu)); #else c->logical_proc_id = 0; #endif @@ -1949,7 +1947,7 @@ void identify_secondary_cpu(struct cpuinfo_x86 *c) enable_sep_cpu(); #endif mtrr_ap_init(); - validate_apic_and_package_id(c); + validate_apic_id(c); x86_spec_ctrl_setup_ap(); update_srbds_msr(); diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 060813411f85..e563a61904eb 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -180,16 +180,12 @@ static void smp_callin(void) apic_ap_setup(); /* - * Save our processor parameters. Note: this information - * is needed for clock calibration. - */ - smp_store_cpu_info(cpuid); - - /* + * Save our processor parameters and update topology. + * Note: this information is needed for clock calibration. * The topology information must be up to date before * calibrate_delay() and notify_cpu_starting(). */ - set_cpu_sibling_map(raw_smp_processor_id()); + smp_store_cpu_info(cpuid, false); ap_init_aperfmperf(); @@ -244,6 +240,12 @@ static void notrace start_secondary(void *unused) * smp_callout_mask to release them. */ cpu_init_secondary(); + + /* + * Even though notify_cpu_starting() will do this, it does so too late + * as the AP may already have triggered lockdep splats by then. See + * commit 29368e093 ("x86/smpboot: Move rcu_cpu_starting() earlier"). + */ rcu_cpu_starting(raw_smp_processor_id()); x86_cpuinit.early_percpu_clock_init(); @@ -352,7 +354,7 @@ EXPORT_SYMBOL(topology_phys_to_logical_die); * @pkg: The physical package id as retrieved via CPUID * @cpu: The cpu for which this is updated */ -int topology_update_package_map(unsigned int pkg, unsigned int cpu) +static int topology_update_package_map(unsigned int pkg, unsigned int cpu) { int new; @@ -375,7 +377,7 @@ int topology_update_package_map(unsigned int pkg, unsigned int cpu) * @die: The die id as retrieved via CPUID * @cpu: The cpu for which this is updated */ -int topology_update_die_map(unsigned int die, unsigned int cpu) +static int topology_update_die_map(unsigned int die, unsigned int cpu) { int new; @@ -406,25 +408,7 @@ void __init smp_store_boot_cpu_info(void) c->initialized = true; } -/* - * The bootstrap kernel entry code has set these up. Save them for - * a given CPU - */ -void smp_store_cpu_info(int id) -{ - struct cpuinfo_x86 *c = &cpu_data(id); - - /* Copy boot_cpu_data only on the first bringup */ - if (!c->initialized) - *c = boot_cpu_data; - c->cpu_index = id; - /* - * During boot time, CPU0 has this setup already. Save the info when - * bringing up AP or offlined CPU0. - */ - identify_secondary_cpu(c); - c->initialized = true; -} +static arch_spinlock_t topology_lock = __ARCH_SPIN_LOCK_UNLOCKED; static bool topology_same_node(struct cpuinfo_x86 *c, struct cpuinfo_x86 *o) @@ -630,7 +614,7 @@ static struct sched_domain_topology_level x86_topology[] = { */ static bool x86_has_numa_in_package; -void set_cpu_sibling_map(int cpu) +static void set_cpu_sibling_map(int cpu) { bool has_smt = smp_num_siblings > 1; bool has_mp = has_smt || boot_cpu_data.x86_max_cores > 1; @@ -709,6 +693,37 @@ void set_cpu_sibling_map(int cpu) } } +/* + * The bootstrap kernel entry code has set these up. Save them for + * a given CPU + */ +void smp_store_cpu_info(int id, bool force_single_core) +{ + struct cpuinfo_x86 *c = &cpu_data(id); + + /* Copy boot_cpu_data only on the first bringup */ + if (!c->initialized) + *c = boot_cpu_data; + c->cpu_index = id; + /* + * During boot time, CPU0 has this setup already. Save the info when + * bringing up AP or offlined CPU0. + */ + identify_secondary_cpu(c); + + arch_spin_lock(&topology_lock); + BUG_ON(topology_update_package_map(c->phys_proc_id, id)); + BUG_ON(topology_update_die_map(c->cpu_die_id, id)); + c->initialized = true; + + /* For Xen PV */ + if (force_single_core) + c->x86_max_cores = 1; + + set_cpu_sibling_map(id); + arch_spin_unlock(&topology_lock); +} + /* maps the cpu to the sched domain representing multi-core */ const struct cpumask *cpu_coregroup_mask(int cpu) { diff --git a/arch/x86/xen/smp_pv.c b/arch/x86/xen/smp_pv.c index 480be82e9b7b..b2c045a37f39 100644 --- a/arch/x86/xen/smp_pv.c +++ b/arch/x86/xen/smp_pv.c @@ -71,9 +71,7 @@ static void cpu_bringup(void) xen_enable_syscall(); } cpu = smp_processor_id(); - smp_store_cpu_info(cpu); - cpu_data(cpu).x86_max_cores = 1; - set_cpu_sibling_map(cpu); + smp_store_cpu_info(cpu, true); speculative_store_bypass_ht_init();