From patchwork Fri Feb 3 13:50:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Morse X-Patchwork-Id: 52511 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp851185wrn; Fri, 3 Feb 2023 05:58:37 -0800 (PST) X-Google-Smtp-Source: AK7set9pRcsbi+j9dGmG5iY8hmkkaeOLhC/U0dg/Iuh02HgYVRO0BY7L2hzVpoqAtY4TsGkwpS4W X-Received: by 2002:a05:6402:1909:b0:49e:210a:65f3 with SMTP id e9-20020a056402190900b0049e210a65f3mr12991757edz.0.1675432717110; Fri, 03 Feb 2023 05:58:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675432717; cv=none; d=google.com; s=arc-20160816; b=neYUEU+Eu9bixjofiLYmhTmiHvNdVa2PaRrG4on4/cB1tImMxGtubwQhhBbr3MgmsS 0qU8h5YHKMlk0L/YgO9Zj8PD+7Lgzf7tq9J5x6hWC10Jy3pmYCDX5Rua8O+beWRHngw5 EnIUv2ATFKJylBcgRsHAEfoqh8+wvkJv5xUb3+w3QU8b0yG6QCCDB+4OXM9pomWuwBiv O1nuVYf9VpRtE7bnY0fJW8wKjKRnwG5p2CoxBK8dkZ5HkRfd7rqZQ8GTJCm0K+GHwC/5 nDlvAVee8yMB1Ezz1fITGapisfEMgYBBKGdE+fi4ReJC+HiYBx9wbifzJdALO3T0dqif 2SUA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=3aC7J43CXW9gD2mCJVLEjoBTrV5t2eQdWvxRkqNR/Tg=; b=Lae9wC6cpTUsTthFLJoxfafNpdfb/r6RaQl0To5Xi4LHSg+0cFFsrLLw8UDQppBqdz de51eRYNN1ORTZAKOOShhg3GGvI1W2eqdiADR6rZ9uNbJKHUtkUffgM8neS067eSm7h3 ZUiEsosZilvy9E/h+7XNARt2R7VCtuRPQce2kUYV05lzCFYoX1Q6o145KYOwnckKT+nY XPDHYqbStzTfdKRPENhTQQVFy6e6M39/zFt6fQ9KHavmo0qen9vB8cNXBSnQYJdMvovj FISpKeTElSJnnd5pOmmsK8mfp1j7rhqgUek4j2N5uA+IUG3oKd4QhLr6Tx/NncIW/jhq kyMw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id o13-20020a17090611cd00b0087875900d30si2863228eja.802.2023.02.03.05.58.13; Fri, 03 Feb 2023 05:58:37 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233716AbjBCN5J (ORCPT + 99 others); Fri, 3 Feb 2023 08:57:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34138 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233482AbjBCN43 (ORCPT ); Fri, 3 Feb 2023 08:56:29 -0500 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 5EF77A2A78; Fri, 3 Feb 2023 05:54:08 -0800 (PST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6D40B15DB; Fri, 3 Feb 2023 05:54:50 -0800 (PST) Received: from eglon.cambridge.arm.com (eglon.cambridge.arm.com [10.1.196.177]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 9E6EA3F71E; Fri, 3 Feb 2023 05:54:04 -0800 (PST) From: James Morse To: linux-pm@vger.kernel.org, loongarch@lists.linux.dev, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-acpi@vger.kernel.org, linux-arch@vger.kernel.org, linux-ia64@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, x86@kernel.org Cc: Marc Zyngier , Thomas Gleixner , Lorenzo Pieralisi , Mark Rutland , Sudeep Holla , Borislav Petkov , H Peter Anvin , Dave Hansen , Ingo Molnar , Will Deacon , Catalin Marinas , Huacai Chen , James Morse , Suzuki K Poulose , Oliver Upton , Len Brown , Rafael Wysocki , WANG Xuerui , Salil Mehta , Russell King , Jean-Philippe Brucker Subject: [RFC PATCH 31/32] arm64: document virtual CPU hotplug's expectations Date: Fri, 3 Feb 2023 13:50:42 +0000 Message-Id: <20230203135043.409192-32-james.morse@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230203135043.409192-1-james.morse@arm.com> References: <20230203135043.409192-1-james.morse@arm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756818536851554425?= X-GMAIL-MSGID: =?utf-8?q?1756818536851554425?= Add a description of physical and virtual CPU hotplug, explain the differences and elaborate on what is required in ACPI for a working virtual hotplug system. Signed-off-by: James Morse --- Documentation/arm64/cpu-hotplug.rst | 79 +++++++++++++++++++++++++++++ Documentation/arm64/index.rst | 1 + 2 files changed, 80 insertions(+) create mode 100644 Documentation/arm64/cpu-hotplug.rst diff --git a/Documentation/arm64/cpu-hotplug.rst b/Documentation/arm64/cpu-hotplug.rst new file mode 100644 index 000000000000..76ba8d932c72 --- /dev/null +++ b/Documentation/arm64/cpu-hotplug.rst @@ -0,0 +1,79 @@ +.. SPDX-License-Identifier: GPL-2.0 +.. _cpuhp_index: + +==================== +CPU Hotplug and ACPI +==================== + +CPU hotplug in the arm64 world is commonly used to describe the kernel taking +CPUs online/offline using PSCI. This document is about ACPI firmware allowing +CPUs that were not available during boot to be added to the system later. + +``possible`` and ``present`` refer to the state of the CPU as seen by linux. + + +CPU Hotplug on physical systems - CPUs not present at boot +---------------------------------------------------------- + +Physical systems need to mark a CPU that is ``possible`` but not ``present`` as +being ``present``. An example would be a dual socket machine, where the package +in one of the sockets can be replaced while the system is running. + +This is not supported. + +In the arm64 world CPUs are not a single device but a slice of the system. +There are no systems that support the physical addition (or removal) of CPUs +while the system is running, and ACPI is not able to sufficiently describe +them. + +e.g. New CPUs come with new caches, but the platform's cache toplogy is +described in a static table, the PPTT. How caches are shared between CPUs is +not discoverable, and must be described by firmware. + +e.g. The GIC redistributor for each CPU must be accessed by the driver during +boot to discover the system wide supported features. ACPI's MADT GICC +structures can describe a redistributor associated with a disabled CPU, but +can't describe whether the redistributor is accessible, only that it is not +'always on'. + +arm64's ACPI tables assume that everything described is ``present``. + + +CPU Hotplug on virtual systems - CPUs not enabled at boot +--------------------------------------------------------- + +Virtual systems have the advantage that all the properties the system will +ever have can be described at boot. There are no power-domain considerations +as such devices are emulated. + +CPU Hotplug on virtual systems is supported. It is distinct from physical +CPU Hotplug as all resources are described as ``present``, but CPUs may be +marked as disabled by firmware. Only the CPU's online/offline behaviour is +influenced by firmware. An example is where a virtual machine boots with a +single CPU, and additional CPUs are added once a cloud orchestrator deploys +the workload. + +For a virtual machine, the VMM (e.g. Qemu) plays the part of firmware. + +Virtual hotplug is implemented as a firmware policy affecting which CPUs can be +brought online. Firmware can enforce its policy via PSCI's return codes. e.g. +``DENIED``. + +The ACPI tables must describe all the resources of the virtual machine. CPUs +that firmware wishes to disable either from boot (or later) should not be +``enabled`` in the MADT GICC structures, but should have the ``online capable`` +bit set, to indicate they can be enabled later. The boot CPU must be marked as +``enabled``. The 'always on' GICR structure must be used to describe the +redistributors. + +CPUs described as ``online capable`` but not ``enabled`` can be set to enabled +by the DSDT's Processor object's _STA method. On virtual systems the _STA method +must always report the CPU as ``present``. Changes to the firmware policy can +be notified to the OS via device-check or eject-request. + +CPUs described as ``enabled`` in the static table, should not have their _STA +modified dynamically by firmware. Soft-restart features such as kexec will +re-read the static properties of the system from these static tables, and +may malfunction if these no longer describe the running system. Linux will +re-discover the dynamic properties of the system from the _STA method later +during boot. diff --git a/Documentation/arm64/index.rst b/Documentation/arm64/index.rst index ae21f8118830..54da62534871 100644 --- a/Documentation/arm64/index.rst +++ b/Documentation/arm64/index.rst @@ -13,6 +13,7 @@ ARM64 Architecture asymmetric-32bit booting cpu-feature-registers + cpu-hotplug elf_hwcaps hugetlbpage legacy_instructions