From patchwork Tue May 16 09:09:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: tip-bot2 for Thomas Gleixner X-Patchwork-Id: 94551 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp283687vqo; Tue, 16 May 2023 02:11:33 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5stJPt/rd6SrPRMFt17glY+dUYW8voNHmNfGK6RbYT3k1BQs1ctP3MYUQcWSvkknPW+RSW X-Received: by 2002:a05:6a20:1584:b0:100:eb6e:8d0a with SMTP id h4-20020a056a20158400b00100eb6e8d0amr34972196pzj.44.1684228293654; Tue, 16 May 2023 02:11:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684228293; cv=none; d=google.com; s=arc-20160816; b=rhoFBDMNIQeXo9n7I8wxNf5cq4LtDd7gfX8BOYZwygfbUsBCEYxnpsybvUM+LkQqUj pwHkGwHnjvcX9z4a3gv/cfu8JAPknIcXQ0iRB4RJvsEc2SsKNT+nSs29nSAlyYtY230J vyf811IpITT/33KOlTUK7GSe+qW+F7fPN1Z1F8nBEePkxMA0Gtv1JfQxWhnLK+M9wwLb ZKM0cLVHuo5jH3Vy+exIk2vfVMbW2uhBaV4VQTpvLv/U85F0Sj461cq898O6vlwz3zVQ ZkkVMYF1xVkdarCVfLvMaAFdzqN/OuASeBw8zKSpV6SbbqmS1trBPOogbPsPIxHLHyX7 EvpA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:robot-unsubscribe :robot-id:message-id:mime-version:references:in-reply-to:cc:subject :to:reply-to:sender:from:dkim-signature:dkim-signature:date; bh=696lBPZ0E7Ky3WNzJYp8aRcVPxvYq/pZKvShQXB/sW0=; b=TIenAiLvOwD5CYo/RoL7m7kMYgSznXIgKsP3xsFDy1g74W8yKMHJ2bgfQ1KzaaRWbb WneBHVLVTTFRdJOAd8G9G5uwv9+OGTGPMsaqM+SS/h5lR3N6Fx+9cQbJ3FcUIWFndE2Z IADWEx+EUns4UcUrkYFmBV+L7VlMXllqOKAtckKnr7nC5ATD64/Ytm8p6i9XZcpoA0IG WH/WFDt8h6gRNJHiaDpDFvkJRLFyxG3F6HnMXfYKtmBHC2tdJQu4J8/Q3eKDK28l4u8Z QK/GqDaX8DdGs/l7kJ5+FLJl24e0BXj8PYSQR7v5UU3+P6DjOEY0KovFjaPVkMblcL+H esSg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=FrKpfgRt; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y3-20020aa793c3000000b0063b8685bab4si18960814pff.113.2023.05.16.02.11.16; Tue, 16 May 2023 02:11:33 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=FrKpfgRt; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231811AbjEPJKL (ORCPT + 99 others); Tue, 16 May 2023 05:10:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45942 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231183AbjEPJJ7 (ORCPT ); Tue, 16 May 2023 05:09:59 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0D9AD2D7F; Tue, 16 May 2023 02:09:56 -0700 (PDT) Date: Tue, 16 May 2023 09:09:53 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684228194; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=696lBPZ0E7Ky3WNzJYp8aRcVPxvYq/pZKvShQXB/sW0=; b=FrKpfgRt0pku/1t/5eQU/khiTM40EvgQkNtAhDi+IvrAylTpK24O2g0uzCk4kl2Fji7mWS yff7YiQciJWc7zroWOHEh/QKyQMWWyKbZdBtsQtO/GmPaMeTOcBhXtBd3+2kzQOCTKGYA9 1IQPq+yCfetNF+/JUtv4V/WPpyhNPlLOYYDYu/U4o7NmqqP7B4YpaVCZQYSqxpKvKY+Mrr MCIvperF5RvjVwszldw0tZ7Q4eshAYsL97h/MMAB7LO4HqFMVW47AT1sRL626hgFjAAy5Z UvxTB6Vo9hhaM6E3vmTH6baKeQT293AqqP5nFIy5oItJxCFJfZHffPW2kSrCnw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684228194; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=696lBPZ0E7Ky3WNzJYp8aRcVPxvYq/pZKvShQXB/sW0=; b=KnaLwSkOxGiaEc7Ikq18XUTMZHV1eKnWiPKXPZuo90VbuqFujMl2F69DhZ26F/XEI4R+Iz C/CZ0RmhZu0piUCA== From: "tip-bot2 for David Woodhouse" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: smp/core] x86/smpboot: Support parallel startup of secondary CPUs Cc: Thomas Gleixner , Brian Gerst , David Woodhouse , "Peter Zijlstra (Intel)" , Michael Kelley , Oleksandr Natalenko , Helge Deller , "Guilherme G. Piccoli" , x86@kernel.org, linux-kernel@vger.kernel.org In-Reply-To: <20230512205257.411554373@linutronix.de> References: <20230512205257.411554373@linutronix.de> MIME-Version: 1.0 Message-ID: <168422819355.404.16336847409734430711.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765724835861619282?= X-GMAIL-MSGID: =?utf-8?q?1766041367289366669?= The following commit has been merged into the smp/core branch of tip: Commit-ID: 7e75178a0950c5ceffa2ca3225701b69752f7d3a Gitweb: https://git.kernel.org/tip/7e75178a0950c5ceffa2ca3225701b69752f7d3a Author: David Woodhouse AuthorDate: Fri, 12 May 2023 23:07:55 +02:00 Committer: Peter Zijlstra CommitterDate: Mon, 15 May 2023 13:45:04 +02:00 x86/smpboot: Support parallel startup of secondary CPUs In parallel startup mode the APs are kicked alive by the control CPU quickly after each other and run through the early startup code in parallel. The real-mode startup code is already serialized with a bit-spinlock to protect the real-mode stack. In parallel startup mode the smpboot_control variable obviously cannot contain the Linux CPU number so the APs have to determine their Linux CPU number on their own. This is required to find the CPUs per CPU offset in order to find the idle task stack and other per CPU data. To achieve this, export the cpuid_to_apicid[] array so that each AP can find its own CPU number by searching therein based on its APIC ID. Introduce a flag in the top bits of smpboot_control which indicates that the AP should find its CPU number by reading the APIC ID from the APIC. This is required because CPUID based APIC ID retrieval can only provide the initial APIC ID, which might have been overruled by the firmware. Some AMD APUs come up with APIC ID = initial APIC ID + 0x10, so the APIC ID to CPU number lookup would fail miserably if based on CPUID. Also virtualization can make its own APIC ID assignements. The only requirement is that the APIC IDs are consistent with the APCI/MADT table. For the boot CPU or in case parallel bringup is disabled the control bits are empty and the CPU number is directly available in bit 0-23 of smpboot_control. [ tglx: Initial proof of concept patch with bitlock and APIC ID lookup ] [ dwmw2: Rework and testing, commit message, CPUID 0x1 and CPU0 support ] [ seanc: Fix stray override of initial_gs in common_cpu_up() ] [ Oleksandr Natalenko: reported suspend/resume issue fixed in x86_acpi_suspend_lowlevel ] [ tglx: Make it read the APIC ID from the APIC instead of using CPUID, split the bitlock part out ] Co-developed-by: Thomas Gleixner Co-developed-by: Brian Gerst Signed-off-by: Thomas Gleixner Signed-off-by: Brian Gerst Signed-off-by: David Woodhouse Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Tested-by: Michael Kelley Tested-by: Oleksandr Natalenko Tested-by: Helge Deller # parisc Tested-by: Guilherme G. Piccoli # Steam Deck Link: https://lore.kernel.org/r/20230512205257.411554373@linutronix.de --- arch/x86/include/asm/apic.h | 2 +- arch/x86/include/asm/apicdef.h | 5 ++- arch/x86/include/asm/smp.h | 6 +++- arch/x86/kernel/acpi/sleep.c | 9 ++++- arch/x86/kernel/apic/apic.c | 2 +- arch/x86/kernel/head_64.S | 61 +++++++++++++++++++++++++++++++++- arch/x86/kernel/smpboot.c | 2 +- 7 files changed, 83 insertions(+), 4 deletions(-) diff --git a/arch/x86/include/asm/apic.h b/arch/x86/include/asm/apic.h index 030f5fb..98c32aa 100644 --- a/arch/x86/include/asm/apic.h +++ b/arch/x86/include/asm/apic.h @@ -55,6 +55,8 @@ extern int local_apic_timer_c2_ok; extern int disable_apic; extern unsigned int lapic_timer_period; +extern int cpuid_to_apicid[]; + extern enum apic_intr_mode_id apic_intr_mode; enum apic_intr_mode_id { APIC_PIC, diff --git a/arch/x86/include/asm/apicdef.h b/arch/x86/include/asm/apicdef.h index 68d213e..bf546df 100644 --- a/arch/x86/include/asm/apicdef.h +++ b/arch/x86/include/asm/apicdef.h @@ -138,7 +138,8 @@ #define APIC_EILVT_MASKED (1 << 16) #define APIC_BASE (fix_to_virt(FIX_APIC_BASE)) -#define APIC_BASE_MSR 0x800 +#define APIC_BASE_MSR 0x800 +#define APIC_X2APIC_ID_MSR 0x802 #define XAPIC_ENABLE (1UL << 11) #define X2APIC_ENABLE (1UL << 10) @@ -162,6 +163,7 @@ #define APIC_CPUID(apicid) ((apicid) & XAPIC_DEST_CPUS_MASK) #define NUM_APIC_CLUSTERS ((BAD_APICID + 1) >> XAPIC_DEST_CPUS_SHIFT) +#ifndef __ASSEMBLY__ /* * the local APIC register structure, memory mapped. Not terribly well * tested, but we might eventually use this one in the future - the @@ -435,4 +437,5 @@ enum apic_delivery_modes { APIC_DELIVERY_MODE_EXTINT = 7, }; +#endif /* !__ASSEMBLY__ */ #endif /* _ASM_X86_APICDEF_H */ diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h index c6d5b65..4206077 100644 --- a/arch/x86/include/asm/smp.h +++ b/arch/x86/include/asm/smp.h @@ -200,4 +200,10 @@ extern unsigned long apic_mmio_base; #endif /* !__ASSEMBLY__ */ +/* Control bits for startup_64 */ +#define STARTUP_READ_APICID 0x80000000 + +/* Top 8 bits are reserved for control */ +#define STARTUP_PARALLEL_MASK 0xFF000000 + #endif /* _ASM_X86_SMP_H */ diff --git a/arch/x86/kernel/acpi/sleep.c b/arch/x86/kernel/acpi/sleep.c index 1328c22..6dfecb2 100644 --- a/arch/x86/kernel/acpi/sleep.c +++ b/arch/x86/kernel/acpi/sleep.c @@ -16,6 +16,7 @@ #include #include #include +#include #include #include "../../realmode/rm/wakeup.h" @@ -127,7 +128,13 @@ int x86_acpi_suspend_lowlevel(void) * value is in the actual %rsp register. */ current->thread.sp = (unsigned long)temp_stack + sizeof(temp_stack); - smpboot_control = smp_processor_id(); + /* + * Ensure the CPU knows which one it is when it comes back, if + * it isn't in parallel mode and expected to work that out for + * itself. + */ + if (!(smpboot_control & STARTUP_PARALLEL_MASK)) + smpboot_control = smp_processor_id(); #endif initial_code = (unsigned long)wakeup_long64; saved_magic = 0x123456789abcdef0L; diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c index d3f6c18..209c505 100644 --- a/arch/x86/kernel/apic/apic.c +++ b/arch/x86/kernel/apic/apic.c @@ -2380,7 +2380,7 @@ static int nr_logical_cpuids = 1; /* * Used to store mapping between logical CPU IDs and APIC IDs. */ -static int cpuid_to_apicid[] = { +int cpuid_to_apicid[] = { [0 ... NR_CPUS - 1] = -1, }; diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index f99e9ab..9cd77d3 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -24,7 +24,9 @@ #include "../entry/calling.h" #include #include +#include #include +#include /* * We are not able to switch in one step to the final KERNEL ADDRESS SPACE @@ -234,8 +236,67 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) ANNOTATE_NOENDBR // above #ifdef CONFIG_SMP + /* + * For parallel boot, the APIC ID is read from the APIC, and then + * used to look up the CPU number. For booting a single CPU, the + * CPU number is encoded in smpboot_control. + * + * Bit 31 STARTUP_READ_APICID (Read APICID from APIC) + * Bit 0-23 CPU# if STARTUP_xx flags are not set + */ movl smpboot_control(%rip), %ecx + testl $STARTUP_READ_APICID, %ecx + jnz .Lread_apicid + /* + * No control bit set, single CPU bringup. CPU number is provided + * in bit 0-23. This is also the boot CPU case (CPU number 0). + */ + andl $(~STARTUP_PARALLEL_MASK), %ecx + jmp .Lsetup_cpu + +.Lread_apicid: + /* Check whether X2APIC mode is already enabled */ + mov $MSR_IA32_APICBASE, %ecx + rdmsr + testl $X2APIC_ENABLE, %eax + jnz .Lread_apicid_msr + + /* Read the APIC ID from the fix-mapped MMIO space. */ + movq apic_mmio_base(%rip), %rcx + addq $APIC_ID, %rcx + movl (%rcx), %eax + shr $24, %eax + jmp .Llookup_AP + +.Lread_apicid_msr: + mov $APIC_X2APIC_ID_MSR, %ecx + rdmsr + +.Llookup_AP: + /* EAX contains the APIC ID of the current CPU */ + xorq %rcx, %rcx + leaq cpuid_to_apicid(%rip), %rbx + +.Lfind_cpunr: + cmpl (%rbx,%rcx,4), %eax + jz .Lsetup_cpu + inc %ecx +#ifdef CONFIG_FORCE_NR_CPUS + cmpl $NR_CPUS, %ecx +#else + cmpl nr_cpu_ids(%rip), %ecx +#endif + jb .Lfind_cpunr + + /* APIC ID not found in the table. Drop the trampoline lock and bail. */ + movq trampoline_lock(%rip), %rax + movl $0, (%rax) + +1: cli + hlt + jmp 1b +.Lsetup_cpu: /* Get the per cpu offset for the given CPU# which is in ECX */ movq __per_cpu_offset(,%rcx,8), %rdx #else diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 51122f0..4b97373 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -996,7 +996,7 @@ static int do_boot_cpu(int apicid, int cpu, struct task_struct *idle) if (IS_ENABLED(CONFIG_X86_32)) { early_gdt_descr.address = (unsigned long)get_cpu_gdt_rw(cpu); initial_stack = idle->thread.sp; - } else { + } else if (!(smpboot_control & STARTUP_PARALLEL_MASK)) { smpboot_control = cpu; }