From patchwork Wed Dec 13 12:47:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 17979 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:3b04:b0:fb:cd0c:d3e with SMTP id c4csp7748038dys; Wed, 13 Dec 2023 04:48:02 -0800 (PST) X-Google-Smtp-Source: AGHT+IH2A7diIWCGCwngNrldp0ESEEQHm19qK+Rpx+2g/StMc9OgS3LJHL2SFlqboOl499hFH9GO X-Received: by 2002:a17:902:e747:b0:1d3:4162:17f3 with SMTP id p7-20020a170902e74700b001d3416217f3mr1581966plf.50.1702471681839; Wed, 13 Dec 2023 04:48:01 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702471681; cv=none; d=google.com; s=arc-20160816; b=oGLbVW5bnXWe5I3+NthLo/IEjnGxsjzY4Jp0mRxTluv+IOhuspPMz9BU1ngu1vgC4J TapKmfCqFHEeviRbaGnaKIiyIcZh4X/Wnl7TN1O3rJ1DxfG0TQlzT7PONlJ2SP8jgVlS tgfEOHYArNfDLpfEvjKxtwyQsJ4JYT/+tPLkZX1IjsEu6iP5Un/kYgDlPLmCjfPn6uPY BmtyiIbBDZdk7hteI+h39AMkgeQ0N0dMRmufJUPSqu5SBgxartUox1xO8HMQzF42DC5O 26H/hp1yAoD0cULxWL08L1NBJpxnWlBUIye4QYpHyLyclF6FUHQRiC8QbftRUzFynA3B zb0w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-disposition:mime-version :message-id:subject:cc:to:from:date:dkim-signature; bh=ZHjgHD/Gl8l5KieVxhVjocoVHd90ATqZhHg2Tfmk6v4=; fh=YWIBRcP8cdLYR894sEToaan3lN1VSUITe+B678IO0WY=; b=kMkNALEatiO5yMmAgi7CxVFVB3diopcFdimtLZFcgV8vHNcK2F6KARF2rXyOKip6AZ hwR5/Wi3hz6iAo8dmZwAFyhMb061yGK6C0LMcgllIcj3L8NPq7V5DG3CgpGazqRt9A27 sWq+Kezt5e3hTlnSnYSNDBKizLnJ+BhATPDagFEvZ4j6FICm+QrFUJjfcFchv85go5K1 Pw11qFCWEA8pJrquJwS2DpESXDd4geycYizQl3dUDg0PJkjfemUJTDkQyMOE4Fzk/+MU M0KWAROsRTQ2cwbYX+lFfhXVOv83/865H3X6ckki+vxshWCpVzyp0UMMqSZ5e8+CxeOW 57dA== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail (test mode) header.i=@armlinux.org.uk header.s=pandora-2019 header.b="m/S0b+qe"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=armlinux.org.uk Received: from morse.vger.email (morse.vger.email. [2620:137:e000::3:1]) by mx.google.com with ESMTPS id s9-20020a170902988900b001cfafa3a897si9294522plp.646.2023.12.13.04.48.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Dec 2023 04:48:01 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1; Authentication-Results: mx.google.com; dkim=fail (test mode) header.i=@armlinux.org.uk header.s=pandora-2019 header.b="m/S0b+qe"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=armlinux.org.uk Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 070688201B06; Wed, 13 Dec 2023 04:47:54 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378833AbjLMMrp (ORCPT + 99 others); Wed, 13 Dec 2023 07:47:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38520 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233416AbjLMMro (ORCPT ); Wed, 13 Dec 2023 07:47:44 -0500 Received: from pandora.armlinux.org.uk (pandora.armlinux.org.uk [IPv6:2001:4d48:ad52:32c8:5054:ff:fe00:142]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A66CBA4; Wed, 13 Dec 2023 04:47:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Sender:Content-Type:MIME-Version: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=ZHjgHD/Gl8l5KieVxhVjocoVHd90ATqZhHg2Tfmk6v4=; b=m/S0b+qed0K3dcmWhCc0n9Zs1F G2H5oNb9p4vERMgroCQuwVGt1Uv7foYe43DitlbwjiozNi10Xu2Vrh72WT440BqM40+LvddApHWhY zGwM7W4RFWJqLMb1g/6Z/kp2C5Undrnq3hDzu9HMXqCCK2eKCGgtXQe7a/yj7PObb9/i9YKrWM545 dlr023WeZ4zVoeMqA6vpdHxm7L4HBZ1qAld4lja7+MfvzUS4JXoMSyYrYoBpvIhgR6ZfPWXvbovu7 OkDAC+ItJuxc62bHBQpFIDmyj139RXmf/MWWUpecB7N4sByYaYHjO6TSbTjzDXFQGHkICjketuZJE HonJMDRw==; Received: from shell.armlinux.org.uk ([fd8f:7570:feb6:1:5054:ff:fe00:4ec]:44482) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1rDOeE-0008CM-2c; Wed, 13 Dec 2023 12:47:34 +0000 Received: from linux by shell.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1rDOeB-0001UM-IO; Wed, 13 Dec 2023 12:47:31 +0000 Date: Wed, 13 Dec 2023 12:47:31 +0000 From: "Russell King (Oracle)" To: linux-pm@vger.kernel.org, loongarch@lists.linux.dev, linux-acpi@vger.kernel.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-riscv@lists.infradead.org, kvmarm@lists.linux.dev, x86@kernel.org, acpica-devel@lists.linuxfoundation.org, linux-csky@vger.kernel.org, linux-doc@vger.kernel.org, linux-ia64@vger.kernel.org, linux-parisc@vger.kernel.org Cc: Salil Mehta , Jean-Philippe Brucker , jianyong.wu@arm.com, justin.he@arm.com, James Morse Subject: [RFC PATCH v3 00/21] ACPI/arm64: add support for virtual cpu hotplug Message-ID: MIME-Version: 1.0 Content-Disposition: inline Sender: Russell King (Oracle) X-Spam-Status: No, score=-0.6 required=5.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Wed, 13 Dec 2023 04:47:54 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785170946110506637 X-GMAIL-MSGID: 1785170946110506637 Hi, This is this remaining patches for ARM64 virtual cpu hotplug, which follows on from the previous set of 21 patches that GregKH has recently queued up, and "x86: intel_epb: Don't rely on link order" which can be found at: https://lore.kernel.org/r/E1r6SeD-00DCuK-M6@rmk-PC.armlinux.org.uk https://lore.kernel.org/r/ZVyz/Ve5pPu8AWoA@shell.armlinux.org.uk The entire series can be found at: git://git.armlinux.org.uk/~rmk/linux-arm.git aarch64/hotplug-vcpu/head The original cover message from the entire series is below the diffstat. Documentation/arch/arm64/cpu-hotplug.rst | 79 ++++++++++++++++ Documentation/arch/arm64/index.rst | 1 + arch/arm64/include/asm/acpi.h | 11 +++ arch/arm64/kernel/acpi_numa.c | 11 --- arch/arm64/kernel/psci.c | 2 +- arch/arm64/kernel/smp.c | 3 +- arch/loongarch/Kconfig | 2 +- arch/loongarch/configs/loongson3_defconfig | 2 +- arch/loongarch/kernel/acpi.c | 4 +- arch/x86/Kconfig | 3 +- arch/x86/kernel/acpi/boot.c | 4 +- drivers/acpi/Kconfig | 13 ++- drivers/acpi/acpi_processor.c | 141 ++++++++++++++++++++++++++--- drivers/acpi/bus.c | 16 ++++ drivers/acpi/device_pm.c | 2 +- drivers/acpi/device_sysfs.c | 2 +- drivers/acpi/internal.h | 1 - drivers/acpi/property.c | 2 +- drivers/acpi/scan.c | 140 ++++++++++++++++++---------- drivers/base/cpu.c | 16 +++- drivers/irqchip/irq-gic-v3.c | 32 ++++--- include/acpi/acpi_bus.h | 1 + include/acpi/actbl2.h | 1 + include/linux/acpi.h | 10 +- include/linux/cpumask.h | 25 +++++ kernel/cpu.c | 3 + 26 files changed, 421 insertions(+), 106 deletions(-) On Tue, Oct 24, 2023 at 04:15:28PM +0100, Russell King (Oracle) wrote: > Hi, > > I'm posting James' patch set updated with most of the review comments > from his RFC v2 series back in September. Individual patches have a > changelog attached at the bottom of the commit message. Those which > I have finished updating have my S-o-b on them, those which still have > outstanding review comments from RFC v2 do not. In some of these cases > I've asked questions and am waiting for responses. > > I'm posting this as RFC v3 because there's still some unaddressed > comments and it's clearly not ready for merging. Even if it was ready > to be merged, it is too late in this development cycle to be taking > this change in, so there would be little point posting it non-RFC. > Also James stated that he's waiting for confirmation from the > Kubernetes/Kata folk - I have no idea what the status is there. > > I will be sending each patch individually to a wider audience > appropriate for that patch - apologies to those missing out on this > cover message. I have added more mailing lists to the series with the > exception of the acpica list in a hope of this cover message also > reaching those folk. > > The changes that aren't included are: > > 1. Updates for my patch that was merged via Thomas (thanks!): > c4dd854f740c cpu-hotplug: Provide prototypes for arch CPU registration > rather than having this change spread through James' patches. > > 2. New patch - simplification of PA-RISC's smp_prepare_boot_cpu() > > 3. Moved "ACPI: Use the acpi_device_is_present() helper in more places" > and "ACPI: Rename acpi_scan_device_not_present() to be about > enumeration" to the beginning of the series - these two patches are > already queued up for merging into 6.7. > > 4. Moved "arm64, irqchip/gic-v3, ACPI: Move MADT GICC enabled check into > a helper" to the beginning of the series, which has been submitted, > but as yet the fate of that posting isn't known. > > The first four patches in this series are provided for completness only. > > There is an additional patch in James' git tree that isn't in the set > of patches that James posted: "ACPI: processor: Only call > arch_unregister_cpu() if HOTPLUG_CPU is selected" which looks to me to > be a workaround for arch_unregister_cpu() being under the ifdef. I've > commented on this on the RFC v2 posting making a suggestion, but as yet > haven't had any response. > > I've included almost all of James' original covering body below the > diffstat. > > The reason that I'm doing this is to help move this code forward so > hopefully it can be merged - which is why I have been keen to dig out > from James' patches anything that can be merged and submit it > separately, since this is a feature for which some users have a > definite need for. > > Please note that I haven't tested this beyond building for aarch64 at > the present time. > > The series can be found at: > > git://git.armlinux.org.uk/~rmk/linux-arm.git aarch64/hotplug-vcpu/v6.6-rc7 > > Documentation/arch/arm64/cpu-hotplug.rst | 79 +++++++++++++++ > Documentation/arch/arm64/index.rst | 1 + > arch/arm64/Kconfig | 1 + > arch/arm64/include/asm/acpi.h | 11 +++ > arch/arm64/include/asm/cpu.h | 1 - > arch/arm64/kernel/acpi_numa.c | 11 --- > arch/arm64/kernel/psci.c | 2 +- > arch/arm64/kernel/setup.c | 13 +-- > arch/arm64/kernel/smp.c | 5 +- > arch/ia64/Kconfig | 3 + > arch/ia64/include/asm/acpi.h | 2 +- > arch/ia64/include/asm/cpu.h | 6 -- > arch/ia64/kernel/acpi.c | 6 +- > arch/ia64/kernel/setup.c | 2 +- > arch/ia64/kernel/topology.c | 35 +------ > arch/loongarch/Kconfig | 2 + > arch/loongarch/configs/loongson3_defconfig | 2 +- > arch/loongarch/kernel/acpi.c | 4 +- > arch/loongarch/kernel/topology.c | 38 +------- > arch/parisc/kernel/smp.c | 8 +- > arch/riscv/Kconfig | 1 + > arch/riscv/kernel/setup.c | 19 +--- > arch/x86/Kconfig | 3 + > arch/x86/include/asm/cpu.h | 4 - > arch/x86/kernel/acpi/boot.c | 4 +- > arch/x86/kernel/cpu/intel_epb.c | 2 +- > arch/x86/kernel/topology.c | 27 +----- > drivers/acpi/Kconfig | 14 ++- > drivers/acpi/acpi_processor.c | 151 +++++++++++++++++++++++------ > drivers/acpi/bus.c | 16 +++ > drivers/acpi/device_pm.c | 2 +- > drivers/acpi/device_sysfs.c | 2 +- > drivers/acpi/internal.h | 1 - > drivers/acpi/processor_core.c | 2 +- > drivers/acpi/property.c | 2 +- > drivers/acpi/scan.c | 148 ++++++++++++++++++---------- > drivers/base/arch_topology.c | 38 +++++--- > drivers/base/cpu.c | 44 +++++++-- > drivers/base/init.c | 2 +- > drivers/base/node.c | 7 -- > drivers/firmware/psci/psci.c | 2 + > drivers/irqchip/irq-gic-v3.c | 38 +++++--- > include/acpi/acpi_bus.h | 1 + > include/acpi/actbl2.h | 1 + > include/linux/acpi.h | 13 ++- > include/linux/cpu.h | 4 + > include/linux/cpumask.h | 25 +++++ > kernel/cpu.c | 3 + > 48 files changed, 516 insertions(+), 292 deletions(-) > > > On Wed, Sep 13, 2023 at 04:37:48PM +0000, James Morse wrote: > > Hello! > > > > Changes since RFC-v1: > > * riscv is new, ia64 is gone > > * The KVM support is different, and upstream - no need to patch the host. > > > > --- > > > > This series adds what looks like cpuhotplug support to arm64 for use in > > virtual machines. It does this by moving the cpu_register() calls for > > architectures that support ACPI out of the arch code by using > > GENERIC_CPU_DEVICES, then into the ACPI processor driver. > > > > The kubernetes folk really want to be able to add CPUs to an existing VM, > > in exactly the same way they do on x86. The use-case is pre-booting guests > > with one CPU, then adding the number that were actually needed when the > > workload is provisioned. > > > > Wait? Doesn't arm64 support cpuhotplug already!? > > In the arm world, cpuhotplug gets used to mean removing the power from a CPU. > > The CPU is offline, and remains present. For x86, and ACPI, cpuhotplug > > has the additional step of physically removing the CPU, so that it isn't > > present anymore. > > > > Arm64 doesn't support this, and can't support it: CPUs are really a slice > > of the SoC, and there is not enough information in the existing ACPI tables > > to describe which bits of the slice also got removed. Without a reference > > machine: adding this support to the spec is a wild goose chase. > > > > Critically: everything described in the firmware tables must remain present. > > > > For a virtual machine this is easy as all the other bits of 'virtual SoC' > > are emulated, so they can (and do) remain present when a vCPU is 'removed'. > > > > On a system that supports cpuhotplug the MADT has to describe every possible > > CPU at boot. Under KVM, the vGIC needs to know about every possible vCPU before > > the guest is started. > > With these constraints, virtual-cpuhotplug is really just a hypervisor/firmware > > policy about which CPUs can be brought online. > > > > This series adds support for virtual-cpuhotplug as exactly that: firmware > > policy. This may even work on a physical machine too; for a guest the part of > > firmware is played by the VMM. (typically Qemu). > > > > PSCI support is modified to return 'DENIED' if the CPU can't be brought > > online/enabled yet. The CPU object's _STA method's enabled bit is used to > > indicate firmware's current disposition. If the CPU has its enabled bit clear, > > it will not be registered with sysfs, and attempts to bring it online will > > fail. The notifications that _STA has changed its value then work in the same > > way as physical hotplug, and firmware can cause the CPU to be registered some > > time later, allowing it to be brought online. > > > > This creates something that looks like cpuhotplug to user-space, as the sysfs > > files appear and disappear, and the udev notifications look the same. > > > > One notable difference is the CPU present mask, which is exposed via sysfs. > > Because the CPUs remain present throughout, they can still be seen in that mask. > > This value does get used by webbrowsers to estimate the number of CPUs > > as the CPU online mask is constantly changed on mobile phones. > > > > Linux is tolerant of PSCI returning errors, as its always been allowed to do > > that. To avoid confusing OS that can't tolerate this, we needed an additional > > bit in the MADT GICC flags. This series copies ACPI_MADT_ONLINE_CAPABLE, which > > appears to be for this purpose, but calls it ACPI_MADT_GICC_CPU_CAPABLE as it > > has a different bit position in the GICC. > > > > This code is unconditionally enabled for all ACPI architectures. > > If there are problems with firmware tables on some devices, the CPUs will > > already be online by the time the acpi_processor_make_enabled() is called. > > A mismatch here causes a firmware-bug message and kernel taint. This should > > only affect people with broken firmware who also boot with maxcpus=1, and > > bring CPUs online later. > > > > I had a go at switching the remaining architectures over to GENERIC_CPU_DEVICES, > > so that the Kconfig symbol can be removed, but I got stuck with powerpc > > and s390. > > > > I've only build tested Loongarch and riscv. I've removed the ia64 specific > > patches, but left the changes in other patches to make git-grep review of > > renames easier. > > > > If folk want to play along at home, you'll need a copy of Qemu that supports this. > > https://github.com/salil-mehta/qemu.git salil/virt-cpuhp-armv8/rfc-v2-rc6 > > > > Replace your '-smp' argument with something like: > > | -smp cpus=1,maxcpus=3,cores=3,threads=1,sockets=1 > > > > then feed the following to the Qemu montior; > > | (qemu) device_add driver=host-arm-cpu,core-id=1,id=cpu1 > > | (qemu) device_del cpu1 > > > > > > Why is this still an RFC? I'm still looking for confirmation from the > > kubernetes/kata folk that this works for them. Because of this I've culled > > the CC list... > > > > > > This series is based on v6.6-rc1, and can be retrieved from: > > https://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git/ virtual_cpu_hotplug/rfc/v2 > > > > > > Thanks, > > > > James Morse (34): > > ACPI: Move ACPI_HOTPLUG_CPU to be disabled on arm64 and riscv > > drivers: base: Use present CPUs in GENERIC_CPU_DEVICES > > drivers: base: Allow parts of GENERIC_CPU_DEVICES to be overridden > > drivers: base: Move cpu_dev_init() after node_dev_init() > > drivers: base: Print a warning instead of panic() when register_cpu() > > fails > > arm64: setup: Switch over to GENERIC_CPU_DEVICES using > > arch_register_cpu() > > x86: intel_epb: Don't rely on link order > > x86/topology: Switch over to GENERIC_CPU_DEVICES > > LoongArch: Switch over to GENERIC_CPU_DEVICES > > riscv: Switch over to GENERIC_CPU_DEVICES > > arch_topology: Make register_cpu_capacity_sysctl() tolerant to late > > CPUs > > ACPI: Use the acpi_device_is_present() helper in more places > > ACPI: Rename acpi_scan_device_not_present() to be about enumeration > > ACPI: Only enumerate enabled (or functional) devices > > ACPI: processor: Add support for processors described as container > > packages > > ACPI: processor: Register CPUs that are online, but not described in > > the DSDT > > ACPI: processor: Register all CPUs from acpi_processor_get_info() > > ACPI: Rename ACPI_HOTPLUG_CPU to include 'present' > > ACPI: Move acpi_bus_trim_one() before acpi_scan_hot_remove() > > ACPI: Rename acpi_processor_hotadd_init and remove pre-processor > > guards > > ACPI: Add post_eject to struct acpi_scan_handler for cpu hotplug > > ACPI: Check _STA present bit before making CPUs not present > > ACPI: Warn when the present bit changes but the feature is not enabled > > drivers: base: Implement weak arch_unregister_cpu() > > LoongArch: Use the __weak version of arch_unregister_cpu() > > arm64: acpi: Move get_cpu_for_acpi_id() to a header > > ACPICA: Add new MADT GICC flags fields [code first?] > > arm64, irqchip/gic-v3, ACPI: Move MADT GICC enabled check into a > > helper > > irqchip/gic-v3: Don't return errors from gic_acpi_match_gicc() > > irqchip/gic-v3: Add support for ACPI's disabled but 'online capable' > > CPUs > > ACPI: add support to register CPUs based on the _STA enabled bit > > arm64: document virtual CPU hotplug's expectations > > ACPI: Add _OSC bits to advertise OS support for toggling CPU > > present/enabled > > cpumask: Add enabled cpumask for present CPUs that can be brought > > online > > > > Jean-Philippe Brucker (1): > > arm64: psci: Ignore DENIED CPUs > > -- > RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ > FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last! >