From patchwork Wed Dec 13 12:50:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 177973 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:3b04:b0:fb:cd0c:d3e with SMTP id c4csp7750493dys; Wed, 13 Dec 2023 04:53:04 -0800 (PST) X-Google-Smtp-Source: AGHT+IF8dPr0Oj4Ydk8zz/TTbrq6k2lsD1xx7DuGG4Cv2JRJSRVzxldTJNT0c9+o6qzlYu4eRY2n X-Received: by 2002:a17:903:228e:b0:1d0:7761:31f6 with SMTP id b14-20020a170903228e00b001d0776131f6mr4836514plh.32.1702471984088; Wed, 13 Dec 2023 04:53:04 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702471984; cv=none; d=google.com; s=arc-20160816; b=uEEy4ozG5GK0bEmfj0kviE/KI/f0Gi6Au/2pQmYSal0dTLQcNY0Dl3+VA4WR8X97PJ b+++haxmx/x2yUj96NodFxsavLXaI4vSwkiGQU/IGxOfegbxi4omCDBFRE83J9vmKZb5 NVnVUJNWfExtBnvyrSWgXH0w2V31D9Z0kLe8eQoHa6gMX73E5q+EgStDoC+i9uZD/fv7 JKsJhQ+GK1+YTVmXrT3Pwjc1yWHYcdppAlGDfKytvQQ/QDCo2JvEW36Uv4zBQ/7uLrJH zOG+IKrZhDAJaaJPH2ZTDMdOLW2tK2GL8Lhzk7xigenNrH0PpyXcTv1l0DKUz/pUh9ax 1yiA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:date:sender:message-id:content-transfer-encoding :content-disposition:mime-version:subject:cc:to:from:references :in-reply-to:dkim-signature; bh=dR7JNhitZsYYNsJnRBxnMA0zB4zlIQmz4Z+T0N0ONL4=; fh=YWIBRcP8cdLYR894sEToaan3lN1VSUITe+B678IO0WY=; b=iTW4zULqgnvHwkZhBg65hJLRiAOuVMcpS2oek7idPcGjZtCDYrb7KT5PK38H/lj/pc qtDGItKy4CWDNrqzBkh6+FOXjXGxI1wSOviIUmqHObZcBtRzBCYbLDn2JQUCCLwwL3oY 0fnGNw5aMVXdg+T2tzpRTSiwkOJwAqB7gt/z+UKnI/M/Bj3hFwWlWUyzdqa0vbPstFH0 u3pW0gMORJITpSg0WiUCGGFfJBr742C669DtigIMsJ4vw52A1v4nlSEFZfDr2IM2gmvP 9GwqVYguiuDK0oXjVRcveRZjFKpkfZmS0oxKKRE7oRHjsjE7My98+tILGTRe3R9xi+l2 AXJA== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail (test mode) header.i=@armlinux.org.uk header.s=pandora-2019 header.b=edybxLfq; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=armlinux.org.uk Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id k10-20020a170902694a00b001cc221d40e1si9281078plt.635.2023.12.13.04.53.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Dec 2023 04:53:04 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=fail (test mode) header.i=@armlinux.org.uk header.s=pandora-2019 header.b=edybxLfq; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=armlinux.org.uk Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id E7DD8807F64F; Wed, 13 Dec 2023 04:52:45 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1379050AbjLMMwB (ORCPT + 99 others); Wed, 13 Dec 2023 07:52:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34546 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1379044AbjLMMve (ORCPT ); Wed, 13 Dec 2023 07:51:34 -0500 Received: from pandora.armlinux.org.uk (pandora.armlinux.org.uk [IPv6:2001:4d48:ad52:32c8:5054:ff:fe00:142]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BBFC319E; Wed, 13 Dec 2023 04:50:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Date:Sender:Message-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:Subject:Cc:To:From:References: In-Reply-To:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=dR7JNhitZsYYNsJnRBxnMA0zB4zlIQmz4Z+T0N0ONL4=; b=edybxLfqLwDaq9HY6CNb9zw0Jr pjBeF6oG5grGntw4FYgF5CREBzDB5dQDEtjzB3UZQUl+wbostnK/9T02VUHkT0JYG2jBM6RVHOMn5 Bh94I73D++Zeo6Jc7kZ6c5ETvudIKqevgd2Dr0/kJFbjoLlCoJOiHdKcsVEyct0DmxDTuXgThoZZg tuxfi6iWKWjuyyE+YZJK1d1Z/ENuelO5+CZXSCAcjSafhCSRkN24DfSaHCW2DWepIU5IMPK55DT/D n9+M/ibWloQmZI3ayPPRY1E2fF1rT7rEb9ehWZaOsxryJVKcLNEMSRRHtSceBX8paS8oD2O3twZkH QYyyICPQ==; Received: from e0022681537dd.dyn.armlinux.org.uk ([fd8f:7570:feb6:1:222:68ff:fe15:37dd]:44930 helo=rmk-PC.armlinux.org.uk) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1rDOhK-0008Hg-2l; Wed, 13 Dec 2023 12:50:46 +0000 Received: from rmk by rmk-PC.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1rDOhN-00DvlU-2e; Wed, 13 Dec 2023 12:50:49 +0000 In-Reply-To: References: From: Russell King (Oracle) To: linux-pm@vger.kernel.org, loongarch@lists.linux.dev, linux-acpi@vger.kernel.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-riscv@lists.infradead.org, kvmarm@lists.linux.dev, x86@kernel.org, acpica-devel@lists.linuxfoundation.org, linux-csky@vger.kernel.org, linux-doc@vger.kernel.org, linux-ia64@vger.kernel.org, linux-parisc@vger.kernel.org Cc: Salil Mehta , Jean-Philippe Brucker , jianyong.wu@arm.com, justin.he@arm.com, James Morse Subject: [PATCH RFC v3 19/21] arm64: document virtual CPU hotplug's expectations MIME-Version: 1.0 Content-Disposition: inline Message-Id: Sender: Russell King Date: Wed, 13 Dec 2023 12:50:49 +0000 X-Spam-Status: No, score=-0.6 required=5.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Wed, 13 Dec 2023 04:52:46 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785171262980121022 X-GMAIL-MSGID: 1785171262980121022 From: James Morse Add a description of physical and virtual CPU hotplug, explain the differences and elaborate on what is required in ACPI for a working virtual hotplug system. Signed-off-by: James Morse --- Outstanding comment: https://lore.kernel.org/r/20230914174137.00000a62@Huawei.com --- Documentation/arch/arm64/cpu-hotplug.rst | 79 ++++++++++++++++++++++++ Documentation/arch/arm64/index.rst | 1 + 2 files changed, 80 insertions(+) create mode 100644 Documentation/arch/arm64/cpu-hotplug.rst diff --git a/Documentation/arch/arm64/cpu-hotplug.rst b/Documentation/arch/arm64/cpu-hotplug.rst new file mode 100644 index 000000000000..76ba8d932c72 --- /dev/null +++ b/Documentation/arch/arm64/cpu-hotplug.rst @@ -0,0 +1,79 @@ +.. SPDX-License-Identifier: GPL-2.0 +.. _cpuhp_index: + +==================== +CPU Hotplug and ACPI +==================== + +CPU hotplug in the arm64 world is commonly used to describe the kernel taking +CPUs online/offline using PSCI. This document is about ACPI firmware allowing +CPUs that were not available during boot to be added to the system later. + +``possible`` and ``present`` refer to the state of the CPU as seen by linux. + + +CPU Hotplug on physical systems - CPUs not present at boot +---------------------------------------------------------- + +Physical systems need to mark a CPU that is ``possible`` but not ``present`` as +being ``present``. An example would be a dual socket machine, where the package +in one of the sockets can be replaced while the system is running. + +This is not supported. + +In the arm64 world CPUs are not a single device but a slice of the system. +There are no systems that support the physical addition (or removal) of CPUs +while the system is running, and ACPI is not able to sufficiently describe +them. + +e.g. New CPUs come with new caches, but the platform's cache toplogy is +described in a static table, the PPTT. How caches are shared between CPUs is +not discoverable, and must be described by firmware. + +e.g. The GIC redistributor for each CPU must be accessed by the driver during +boot to discover the system wide supported features. ACPI's MADT GICC +structures can describe a redistributor associated with a disabled CPU, but +can't describe whether the redistributor is accessible, only that it is not +'always on'. + +arm64's ACPI tables assume that everything described is ``present``. + + +CPU Hotplug on virtual systems - CPUs not enabled at boot +--------------------------------------------------------- + +Virtual systems have the advantage that all the properties the system will +ever have can be described at boot. There are no power-domain considerations +as such devices are emulated. + +CPU Hotplug on virtual systems is supported. It is distinct from physical +CPU Hotplug as all resources are described as ``present``, but CPUs may be +marked as disabled by firmware. Only the CPU's online/offline behaviour is +influenced by firmware. An example is where a virtual machine boots with a +single CPU, and additional CPUs are added once a cloud orchestrator deploys +the workload. + +For a virtual machine, the VMM (e.g. Qemu) plays the part of firmware. + +Virtual hotplug is implemented as a firmware policy affecting which CPUs can be +brought online. Firmware can enforce its policy via PSCI's return codes. e.g. +``DENIED``. + +The ACPI tables must describe all the resources of the virtual machine. CPUs +that firmware wishes to disable either from boot (or later) should not be +``enabled`` in the MADT GICC structures, but should have the ``online capable`` +bit set, to indicate they can be enabled later. The boot CPU must be marked as +``enabled``. The 'always on' GICR structure must be used to describe the +redistributors. + +CPUs described as ``online capable`` but not ``enabled`` can be set to enabled +by the DSDT's Processor object's _STA method. On virtual systems the _STA method +must always report the CPU as ``present``. Changes to the firmware policy can +be notified to the OS via device-check or eject-request. + +CPUs described as ``enabled`` in the static table, should not have their _STA +modified dynamically by firmware. Soft-restart features such as kexec will +re-read the static properties of the system from these static tables, and +may malfunction if these no longer describe the running system. Linux will +re-discover the dynamic properties of the system from the _STA method later +during boot. diff --git a/Documentation/arch/arm64/index.rst b/Documentation/arch/arm64/index.rst index d08e924204bf..78544de0a8a9 100644 --- a/Documentation/arch/arm64/index.rst +++ b/Documentation/arch/arm64/index.rst @@ -13,6 +13,7 @@ ARM64 Architecture asymmetric-32bit booting cpu-feature-registers + cpu-hotplug elf_hwcaps hugetlbpage kdump