From patchwork Tue Mar 21 19:40:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 73029 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp1975877wrt; Tue, 21 Mar 2023 12:48:09 -0700 (PDT) X-Google-Smtp-Source: AK7set/I5gxz7Kr2L1o3TiYvLvZbgRo5QLevZrAlb9t0QxpS5rFfa5Fe/VmO9Scyfuagn8sVMDp9 X-Received: by 2002:a17:90b:1b12:b0:23f:7625:49b6 with SMTP id nu18-20020a17090b1b1200b0023f762549b6mr853276pjb.37.1679428089268; Tue, 21 Mar 2023 12:48:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679428089; cv=none; d=google.com; s=arc-20160816; b=c3bFTcjr049lFoinzgiD+Ue3kxiCDJygIDAVc2rrhmbuAA5lWtUW/f8sWPwWZbw9lA KctBzyrXZX7FGU/DJDCQhatRxzYciY8sbVSGETtQwVMljnMexSeXlTrQnFdCrqhc+s18 hncPE+rqMWLk4DmXAg8nzdaTOQ7SzNMa63uzemD5yHI5UdaM+Qc9FTJQ6pI59WykMcRW rVzzRwOn7PS2/7pIwPEnh6J9SMMqkwjCml2akHhbuyYOhpAD6k3QPpNGw49GysFH5+7g Nlq8rN/3kKxDbRcUO7p/S5jbXh7WEgs+TpSiMMoIBHhYWmPX2UezY5LM1osSfC4MVpV8 T4wg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=5ygAcJKnTKCTvVciunh9HcyMwipcH5PYmlVu4YEhebA=; b=HhQS4xSCvp8BIwU3iAXpx7mY9e05Y5iY3DRtbxbePnJMLbWc1mU9rOKFBFkIOOhZ8p NGaPzSkytlFOs+BICM+sDkcq2Osxhe8pgk1GIUfrQZAD8wyqw+YEKo71C2ltXLPocNk4 QDKhN8zWGgXEtpvyIyZIJuWU/aeGfq4tb6R19Y5L/OYMITZHuSYaDF8MebPcatlSNHiz o5w9FSrMemJeBzOC/vsm0rexFmHw8fMen1pVW51RRBQxRRvxc2KQ+DJ1AufNd+tuynTy gZg97GxEO/lwm6mqFvgEXVlw+qaTidhVRPw0zwon0j3dhJempnw+3W7Zu9Hbn2f5cuwa UDUA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=Iexkp6x3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t14-20020a635f0e000000b0050300a5cf3csi14491805pgb.469.2023.03.21.12.47.56; Tue, 21 Mar 2023 12:48:09 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=Iexkp6x3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230181AbjCUTmC (ORCPT + 99 others); Tue, 21 Mar 2023 15:42:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33788 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230231AbjCUTlx (ORCPT ); Tue, 21 Mar 2023 15:41:53 -0400 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 061EA3BC7B for ; Tue, 21 Mar 2023 12:40:47 -0700 (PDT) Received: by mail-wr1-x42b.google.com with SMTP id t15so14874599wrz.7 for ; Tue, 21 Mar 2023 12:40:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1679427635; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5ygAcJKnTKCTvVciunh9HcyMwipcH5PYmlVu4YEhebA=; b=Iexkp6x3yyHqDDL6OaZInLzZ93bK+l7wJcqEc9pszCs17KfJta9MRhbYNehOvs8q6D XKUaQuZ2obQke50TshqprcQQZw3dKHyHYgWrm/pzzQPvWpEcQSNPXDZHWw0agky2U+ps ogF9bsf4QHdfeveEvlewrX/zdZYyqcFn8X9/RYcrS0ujc35Kmtm0vgw9+o+KtBZvusTh +rf69A/3woBAMVVa67Agt4vUljhc6SgDCXli6nYiJPzMFpu8uKbrQoUXSg+t94/iROyb Bshdjqa2hgalSv9JjLbbPTgXk8fBP+fK4thtjwGtYVs/MJhLRADue+B62bdcQ1MLig72 6IbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679427635; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5ygAcJKnTKCTvVciunh9HcyMwipcH5PYmlVu4YEhebA=; b=UbUK9obk2Bfu9HPLrHD1pHpgpZ9BabFlg0vNlIFw6XcC1132lMTQvFOORVsz87XHRM V9ycQi6gbeiPASGDMAVkXcYojZJWkSdvYkunzb84kiKlxkvC6uVPFnqgfPX7VLQiK2wQ gN0AyMXCHvSV7dNDh5fOGIc/I95QhvjB29nVND5MqSGIlD7ArEPtxnDLHi8BHcOxYmuz TBKLBBNAfcExCoLji+IX1IEk3gDIZV3EZGXSdC5lSOQkIzpwJL+KqDXFk/nmNfCqTOwJ +sIUpaZZBRR1cftT9a/Rj0wReDBn/6UE0wrRQ6MFWnZ9HRnqb091aMjw3j3H30/j4Xbm XLxQ== X-Gm-Message-State: AO0yUKWhaX8rXqLy+gk9uNIs/nwQPiKIeZWO9eX+F/QgAgmqy4IWaR4K ip5wo4vR02K7Hpwh7C0oGkhgxA== X-Received: by 2002:a5d:474d:0:b0:2d4:62ed:e7cf with SMTP id o13-20020a5d474d000000b002d462ede7cfmr3301491wrs.43.1679427635199; Tue, 21 Mar 2023 12:40:35 -0700 (PDT) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:7a8a:d679:ba4e:61cf]) by smtp.gmail.com with ESMTPSA id a3-20020adffb83000000b002c561805a4csm12005026wrr.45.2023.03.21.12.40.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Mar 2023 12:40:34 -0700 (PDT) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, gpiccoli@igalia.com, David Woodhouse , Usama Arif Subject: [PATCH v16 1/8] cpu/hotplug: Move idle_thread_get() to Date: Tue, 21 Mar 2023 19:40:01 +0000 Message-Id: <20230321194008.785922-2-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230321194008.785922-1-usama.arif@bytedance.com> References: <20230321194008.785922-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761007988204879552?= X-GMAIL-MSGID: =?utf-8?q?1761007988204879552?= From: David Woodhouse Instead of relying purely on the special-case wrapper in bringup_cpu() to pass the idle thread to __cpu_up(), expose idle_thread_get() so that the architecture code can obtain it directly when necessary. This will be useful when the existing __cpu_up() is split into multiple phases, only *one* of which will actually need the idle thread. If the architecture code is to register its new pre-bringup states with the cpuhp core, having a special-case wrapper to pass extra arguments is non-trivial and it's easier just to let the arch register its function pointer to be invoked with the standard API. Signed-off-by: David Woodhouse Reviewed-by: Thomas Gleixner Signed-off-by: Usama Arif Tested-by: Paul E. McKenney Tested-by: Kim Phillips Tested-by: Oleksandr Natalenko Tested-by: Guilherme G. Piccoli --- include/linux/smpboot.h | 7 +++++++ kernel/smpboot.h | 2 -- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/include/linux/smpboot.h b/include/linux/smpboot.h index 9d1bc65d226c..3862addcaa34 100644 --- a/include/linux/smpboot.h +++ b/include/linux/smpboot.h @@ -5,6 +5,13 @@ #include struct task_struct; + +#ifdef CONFIG_GENERIC_SMP_IDLE_THREAD +struct task_struct *idle_thread_get(unsigned int cpu); +#else +static inline struct task_struct *idle_thread_get(unsigned int cpu) { return NULL; } +#endif + /* Cookie handed to the thread_fn*/ struct smpboot_thread_data; diff --git a/kernel/smpboot.h b/kernel/smpboot.h index 34dd3d7ba40b..60c609318ad6 100644 --- a/kernel/smpboot.h +++ b/kernel/smpboot.h @@ -5,11 +5,9 @@ struct task_struct; #ifdef CONFIG_GENERIC_SMP_IDLE_THREAD -struct task_struct *idle_thread_get(unsigned int cpu); void idle_thread_set_boot_cpu(void); void idle_threads_init(void); #else -static inline struct task_struct *idle_thread_get(unsigned int cpu) { return NULL; } static inline void idle_thread_set_boot_cpu(void) { } static inline void idle_threads_init(void) { } #endif From patchwork Tue Mar 21 19:40:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 73024 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp1974342wrt; Tue, 21 Mar 2023 12:44:33 -0700 (PDT) X-Google-Smtp-Source: AK7set/vU023zsEobT2DA4a0egFhiYbsX4unqcTRnlXHcS/B8F5Cp1PWBKvo2VU6deQKTB+WNyxK X-Received: by 2002:a05:6a20:3541:b0:d8:d720:7fa with SMTP id f1-20020a056a20354100b000d8d72007famr3050925pze.0.1679427873294; Tue, 21 Mar 2023 12:44:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679427873; cv=none; d=google.com; s=arc-20160816; b=0ZwbIx8OEVwH9Hqy0uKhF3W8kIv3ITrsvV+vVHGBBdc3RZc6oXoLgMHxuC0x208mUc BabPEY89bmN9yFfSYlntjtgRuYRHS6adQxBgCYvm7ZnDFHSf5onzX2KbWyeqSLSBo+xr 0wVEGc0yGJkHSxZqNYphUmmETaQXIBim1MMMtz4H7mhYXK+imFvZJbAr33+ine0yb2Wq dHz2+NTD7gMM4tTBhDVOMe1/BPItoy2DKwpgmAXpf1bdw7NzTlk4jO2AAfc+rdCtKrOv zpY4f3tp642hl2bv9pEMz74SvwxqPUY2IcCM3ob2baf3SaB5Ze/rtyaW8nWaeGf3t4f9 1EdQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=XuLu3HGQcjaytP31qHwhZDi8gSxR4JzJz3SEvAV55zo=; b=G8who4c8nKBFLLQhzB0hXA+YINZ16UtnfXRCj7ExRDaaOSGpglEwOzAVYvquzn0Ets yW2o7kYA1HECvgPh+ZMlstscdFkluOppAtfXh+s9QErdIy50PybSoVzdOU/NYtuXl2uo c5CMaMS/UA7npNdP4NvnHft0Ex/s5yfdnagjdflYF8ibVWEg8iXpwKSS96lRgmpMxj6T v+sCs7KrPYR8aQcAs06uvdWjU/MkMdi1TwNCyjf2ZqmfXABpxIqBg6gxZujAaTj2ZCKy 6JEPwuoeaAzBJHZSo3rg9SOfcViZO0NUt7ZmnfKzxmx+5DZW+EXTnb0Hi93FFtYG2Pud qxEg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=f2Jpe6Q2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 7-20020a630307000000b004df43225600si14153839pgd.709.2023.03.21.12.44.20; Tue, 21 Mar 2023 12:44:33 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=f2Jpe6Q2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230114AbjCUTld (ORCPT + 99 others); Tue, 21 Mar 2023 15:41:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33786 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230025AbjCUTl3 (ORCPT ); Tue, 21 Mar 2023 15:41:29 -0400 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 53B79574F3 for ; Tue, 21 Mar 2023 12:40:49 -0700 (PDT) Received: by mail-wr1-x42f.google.com with SMTP id t15so14874641wrz.7 for ; Tue, 21 Mar 2023 12:40:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1679427636; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XuLu3HGQcjaytP31qHwhZDi8gSxR4JzJz3SEvAV55zo=; b=f2Jpe6Q2tY3Gz7GXpj/6SRsTP1/2EY+sStePEHknHyPHUxW7vMGCIi3haEE+eKVzV4 QxHXFaOm6A1jMa600e1jsjRZ2Hl+D9DQQr0Bgolx3k08dShHr+eiu/5y5bB7KAzrD99x A74GnQ92Y32buwVyIEdp5aKzgQ5tEvOHrd2SrkqmqfwKv5ICTLu1GPaPjiJWttr7ilx+ alkfHz7qtXfU5Ad++n6WadIP8+htGhANDetMan0r+uogeKgOQPqKmkn1HZV9RBV3Hruu ulLcOvl51E6Y2+nBPbaRyk0hBhtVT36fUx02Y/ymgMI+7vPPdXEEA4Fw6J7ivKsDukgV I7+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679427636; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XuLu3HGQcjaytP31qHwhZDi8gSxR4JzJz3SEvAV55zo=; b=hMc6w3DY+iGpyIRjI32ehL2+x5+4+S/heBIwgcNpYVekgxmJxapc8Ou5ehK7rU/W21 vv8/qUXFHCKs0NkwENV3TBWRyAs6S8VmfYLKJkb/u7m0vTESxO1o8pbS8q6gEE7p/XSj aJILlXvUdvcnIhkLvavZbNJMeyPAHE1EFjhdGgH32dXTPXD2M1D8W72JDe+n5l4h062B Ad8zpCfpD96/A8FnIT8cl2RlRxf79Xlz1TBiWM+2MyZaC6xHSm0FvlWVcaFvm0n/J15b 61dQjVW/yzkWcajpSSkfDXbd3xN95MEBY6OR8cPov4DATV2JZ8sMnrOFtd44bVCkMWyv PPCQ== X-Gm-Message-State: AO0yUKU4P9bYXvTXyt36LVEUmKAdYVDGW9v4E/PngKhHvcEhU8rtiT85 vEiQK/P3fk+3jaYiM/F7mzoVfg== X-Received: by 2002:a5d:538c:0:b0:2d0:d73a:7671 with SMTP id d12-20020a5d538c000000b002d0d73a7671mr3066653wrv.22.1679427636150; Tue, 21 Mar 2023 12:40:36 -0700 (PDT) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:7a8a:d679:ba4e:61cf]) by smtp.gmail.com with ESMTPSA id a3-20020adffb83000000b002c561805a4csm12005026wrr.45.2023.03.21.12.40.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Mar 2023 12:40:35 -0700 (PDT) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, gpiccoli@igalia.com, David Woodhouse Subject: [PATCH v16 2/8] cpu/hotplug: Reset task stack state in _cpu_up() Date: Tue, 21 Mar 2023 19:40:02 +0000 Message-Id: <20230321194008.785922-3-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230321194008.785922-1-usama.arif@bytedance.com> References: <20230321194008.785922-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761007761267043083?= X-GMAIL-MSGID: =?utf-8?q?1761007761267043083?= From: David Woodhouse Commit dce1ca0525bf ("sched/scs: Reset task stack state in bringup_cpu()") ensured that the shadow call stack and KASAN poisoning were removed from a CPU's stack each time that CPU is brought up, not just once. This is not incorrect. However, with parallel bringup, an architecture may obtain the idle thread for a new CPU from a pre-bringup stage, by calling idle_thread_get() for itself. This would mean that the cleanup in bringup_cpu() would be too late. Move the SCS/KASAN cleanup to the generic _cpu_up() function instead, which already ensures that the new CPU's stack is available, purely to allow for early failure. This occurs when the CPU to be brought up is in the CPUHP_OFFLINE state, which should correctly do the cleanup any time the CPU has been taken down to the point where such is needed. Signed-off-by: David Woodhouse Reviewed-by: Mark Rutland Tested-by: Mark Rutland [arm64] --- kernel/cpu.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/kernel/cpu.c b/kernel/cpu.c index 6c0a92ca6bb5..43e0a77f21e8 100644 --- a/kernel/cpu.c +++ b/kernel/cpu.c @@ -591,12 +591,6 @@ static int bringup_cpu(unsigned int cpu) struct task_struct *idle = idle_thread_get(cpu); int ret; - /* - * Reset stale stack state from the last time this CPU was online. - */ - scs_task_reset(idle); - kasan_unpoison_task_stack(idle); - /* * Some architectures have to walk the irq descriptors to * setup the vector space for the cpu which comes online. @@ -1383,6 +1377,12 @@ static int _cpu_up(unsigned int cpu, int tasks_frozen, enum cpuhp_state target) ret = PTR_ERR(idle); goto out; } + + /* + * Reset stale stack state from the last time this CPU was online. + */ + scs_task_reset(idle); + kasan_unpoison_task_stack(idle); } cpuhp_tasks_frozen = tasks_frozen; From patchwork Tue Mar 21 19:40:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 73031 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp1977091wrt; Tue, 21 Mar 2023 12:51:06 -0700 (PDT) X-Google-Smtp-Source: AK7set8gtXr9/i8/WXb5fI9VhOLDEJLP+P34qqwDpuIC22DgjvQv2p7rey0kmgvAUhTn0KAbA3ld X-Received: by 2002:a05:6a20:4a23:b0:d9:bf06:cd85 with SMTP id fr35-20020a056a204a2300b000d9bf06cd85mr3285175pzb.25.1679428266056; Tue, 21 Mar 2023 12:51:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679428266; cv=none; d=google.com; s=arc-20160816; b=UVhEVXkk06RC7nml4uWt4Vj4ecQWsP85MhEQ/BeoxaxFNi5n+Q4trcSTPcz14F6u8u XIGo9YlTp5rEYFzkoE4+O/4yHUVDhhpug3Oq6YCHpC8MGj4nVy78XWVTgLLNVusaemj9 TmMRr/9/wefrgO9AyVxli61bWT/Njjt3fiSVlIMGpyLFcz71crXoF2dWAdB1RKT/zrCW qrcYIoymlxgYdroMUqHmeTHpYej8JqokhmJDcbVpO/oX64QfU6ymoL8Vf9BtGe4Up43y /eI0JrhHvZmTejU+pAKildJ9wqLpug0oJBtnLROTNPUhWT7iwHoIxpmOl3Yh6O9JeFzT PGZA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=DktZ/R8hIfvx1jODzCxrB4AmB9U5V9FbvM8VCV2fQU8=; b=pUhiHgj2hOs/gJuPWlldPWKyN6ZgkO39L/D0EQn3nKDV2g9tKqODOruiZQ3iK6gaeE MWaAgJG7XROA3ApE3zDif21BKl4zq0wy8SQEyAhSxJCChpVTX3AysebHjisMZNlxVkmB 0b5WtkqGW/8uXoClN6wqsws4R5vsKNGwCiAa1gD1SFtXg2pvBcIg2hN0S6EPojXi/JV8 21qHn0Z0O0SIjOdMYKzVLYEP7/YBESTjkp0AL0DZ0v0gXc/BSP5EQpVYnDOpY+sNp6LE ezjZ7AHZZUIUfnFmlF3Wo/UC/lk+2DYpqICbHFG/q2H21SK4sHk0SysMItUG0m4waRbc hPAQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=PMusNAMz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k17-20020a056a00169100b006281ec442absi1053174pfc.309.2023.03.21.12.50.52; Tue, 21 Mar 2023 12:51:06 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=PMusNAMz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230348AbjCUTlk (ORCPT + 99 others); Tue, 21 Mar 2023 15:41:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33794 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230231AbjCUTla (ORCPT ); Tue, 21 Mar 2023 15:41:30 -0400 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 489E63E0B3 for ; Tue, 21 Mar 2023 12:40:53 -0700 (PDT) Received: by mail-wr1-x42b.google.com with SMTP id m2so14884658wrh.6 for ; Tue, 21 Mar 2023 12:40:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1679427637; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DktZ/R8hIfvx1jODzCxrB4AmB9U5V9FbvM8VCV2fQU8=; b=PMusNAMzrj2S5E/UImuGRw0Y2hKIAn7ciHXdgf+MQYDpFK6JCTiXnL5RAgYe5jJ19K 7qOo5eum1pTPHY9mJtl6U1n2DsKZCpsltLeI5Ub07faBl6tSZB2jRL8yE6FVnRFjaQzk Uv2kgoJdnXR9KWI2BH7w32fsAcXNbpmVCZE6JThofYZnTcHp8Astk6Mp7Lt2fR7zxSHI NWBOTQ8FkVqATaktP2aLnfBpkCFwmeQRiORCUzoNXCVrAR/Et3YIx55B06RZTncQZ0mO F8aqMEg4pcvm3JqMoP6gdf+hlv3NxXvz7MxePYGvh8TG8/QGo9i26oIwCj7HLjd9o4NR PkiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679427637; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DktZ/R8hIfvx1jODzCxrB4AmB9U5V9FbvM8VCV2fQU8=; b=EkeJVyQK5/wNa+CptM5X8LyO0dK/MUMFCm4o+VYJfVRblqCj5eS8gsxoo1hCm+BGdW t76TDbKEI1FYJ8FdCC3l0TfE9qeEX99vQwSCtejZhsZF93rZsIbROOhCV6AUjbpZC2oN kSNBO6XjsE6Fd4XZ6FiLY4TOyEmPavM5VcuDBim0Spp9ph+Ycphv+qJ58LrtLPaDYgSW LiFc2rsthBLHU87lWdw83mXVEhCbo6ebj0GWcFI3f4Pi6GNu5BtFy4TVp5l27+6kTb3M UYhIZVPqWiPShT5HXJClVOP9HfGkxk2f16hZLxG++uVGd7unBi7qGOlMA3dvavplNtFe ybwA== X-Gm-Message-State: AO0yUKVt6Dqwdy3XUBZquKSe4lnjBhH6V+I1pJ2XBL/budz90qlF0lYp KO9wBQmewuUDEu2bx6/mZYRb7g== X-Received: by 2002:a5d:4ccf:0:b0:2ce:a13b:76b4 with SMTP id c15-20020a5d4ccf000000b002cea13b76b4mr2941937wrt.24.1679427637108; Tue, 21 Mar 2023 12:40:37 -0700 (PDT) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:7a8a:d679:ba4e:61cf]) by smtp.gmail.com with ESMTPSA id a3-20020adffb83000000b002c561805a4csm12005026wrr.45.2023.03.21.12.40.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Mar 2023 12:40:36 -0700 (PDT) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, gpiccoli@igalia.com, David Woodhouse , Usama Arif Subject: [PATCH v16 3/8] cpu/hotplug: Add dynamic parallel bringup states before CPUHP_BRINGUP_CPU Date: Tue, 21 Mar 2023 19:40:03 +0000 Message-Id: <20230321194008.785922-4-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230321194008.785922-1-usama.arif@bytedance.com> References: <20230321194008.785922-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761008173770032702?= X-GMAIL-MSGID: =?utf-8?q?1761008173770032702?= From: David Woodhouse There is often significant latency in the early stages of CPU bringup, and time is wasted by waking each CPU (e.g. with SIPI/INIT/INIT on x86) and then waiting for it to respond before moving on to the next. Allow a platform to register a set of pre-bringup CPUHP states to which each CPU can be stepped in parallel, thus absorbing some of that latency. There is a subtlety here: even with an empty CPUHP_BP_PARALLEL_DYN step, this means that *all* CPUs are brought through the prepare states and to CPUHP_BP_PREPARE_DYN before any of them are taken to CPUHP_BRINGUP_CPU and then are allowed to run for themselves to CPUHP_ONLINE. So any combination of prepare/start calls which depend on A-B ordering for each CPU in turn, such as the X2APIC code which used to allocate a cluster mask 'just in case' and store it in a global variable in the prep stage, then potentially consume that preallocated structure from the AP and set the global pointer to NULL to be reallocated in CPUHP_X2APIC_PREPARE for the next CPU... would explode horribly. Any platform enabling the CPUHP_BP_PARALLEL_DYN steps must be reviewed and tested to ensure that such issues do not exist, and the existing behaviour of bringing CPUs to CPUHP_BP_PREPARE_DYN and then immediately to CPUHP_BRINGUP_CPU and CPUHP_ONLINE only one at a time does not change unless such a state is registered. Note that the new parallel stages do *not* yet bring each AP to the CPUHP_BRINGUP_CPU state at the same time, only to the new states which exist before it. The final loop in bringup_nonboot_cpus() is untouched, bringing each AP in turn from the final PARALLEL_DYN state (or all the way from CPUHP_OFFLINE) to CPUHP_BRINGUP_CPU and then waiting for that AP to do its own processing and reach CPUHP_ONLINE before releasing the next. Parallelising that part by bringing them all to CPUHP_BRINGUP_CPU and then waiting for them all is an exercise for the future. Signed-off-by: David Woodhouse Signed-off-by: Usama Arif Tested-by: Paul E. McKenney Tested-by: Kim Phillips Tested-by: Oleksandr Natalenko Tested-by: Guilherme G. Piccoli Signed-off-by: David Woodhouse Signed-off-by: Usama Arif Tested-by: Paul E. McKenney Tested-by: Kim Phillips Tested-by: Oleksandr Natalenko Tested-by: Guilherme G. Piccoli Reviewed-by: Mark Rutland Tested-by: Mark Rutland [arm64] --- include/linux/cpuhotplug.h | 2 ++ kernel/cpu.c | 49 ++++++++++++++++++++++++++++++++++---- 2 files changed, 47 insertions(+), 4 deletions(-) diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h index c6fab004104a..ef3cf69a3d5b 100644 --- a/include/linux/cpuhotplug.h +++ b/include/linux/cpuhotplug.h @@ -133,6 +133,8 @@ enum cpuhp_state { CPUHP_MIPS_SOC_PREPARE, CPUHP_BP_PREPARE_DYN, CPUHP_BP_PREPARE_DYN_END = CPUHP_BP_PREPARE_DYN + 20, + CPUHP_BP_PARALLEL_DYN, + CPUHP_BP_PARALLEL_DYN_END = CPUHP_BP_PARALLEL_DYN + 4, CPUHP_BRINGUP_CPU, /* diff --git a/kernel/cpu.c b/kernel/cpu.c index 43e0a77f21e8..cf3c1c6f0710 100644 --- a/kernel/cpu.c +++ b/kernel/cpu.c @@ -1504,13 +1504,49 @@ int bringup_hibernate_cpu(unsigned int sleep_cpu) void bringup_nonboot_cpus(unsigned int setup_max_cpus) { + unsigned int n = setup_max_cpus - num_online_cpus(); unsigned int cpu; + /* + * An architecture may have registered parallel pre-bringup states to + * which each CPU may be brought in parallel. For each such state, + * bring N CPUs to it in turn before the final round of bringing them + * online. + */ + if (n > 0) { + enum cpuhp_state st = CPUHP_BP_PARALLEL_DYN; + + while (st <= CPUHP_BP_PARALLEL_DYN_END && cpuhp_hp_states[st].name) { + int i = n; + + for_each_present_cpu(cpu) { + cpu_up(cpu, st); + if (!--i) + break; + } + st++; + } + } + for_each_present_cpu(cpu) { if (num_online_cpus() >= setup_max_cpus) break; - if (!cpu_online(cpu)) - cpu_up(cpu, CPUHP_ONLINE); + if (!cpu_online(cpu)) { + int ret = cpu_up(cpu, CPUHP_ONLINE); + + /* + * For the parallel bringup case, roll all the way back + * to CPUHP_OFFLINE on failure; don't leave them in the + * parallel stages. This happens in the nosmt case for + * non-primary threads. + */ + if (ret && cpuhp_hp_states[CPUHP_BP_PARALLEL_DYN].name) { + struct cpuhp_cpu_state *st = per_cpu_ptr(&cpuhp_state, cpu); + if (can_rollback_cpu(st)) + WARN_ON(cpuhp_invoke_callback_range(false, cpu, st, + CPUHP_OFFLINE)); + } + } } } @@ -1882,6 +1918,10 @@ static int cpuhp_reserve_state(enum cpuhp_state state) step = cpuhp_hp_states + CPUHP_BP_PREPARE_DYN; end = CPUHP_BP_PREPARE_DYN_END; break; + case CPUHP_BP_PARALLEL_DYN: + step = cpuhp_hp_states + CPUHP_BP_PARALLEL_DYN; + end = CPUHP_BP_PARALLEL_DYN_END; + break; default: return -EINVAL; } @@ -1906,14 +1946,15 @@ static int cpuhp_store_callbacks(enum cpuhp_state state, const char *name, /* * If name is NULL, then the state gets removed. * - * CPUHP_AP_ONLINE_DYN and CPUHP_BP_PREPARE_DYN are handed out on + * CPUHP_AP_ONLINE_DYN and CPUHP_BP_P*_DYN are handed out on * the first allocation from these dynamic ranges, so the removal * would trigger a new allocation and clear the wrong (already * empty) state, leaving the callbacks of the to be cleared state * dangling, which causes wreckage on the next hotplug operation. */ if (name && (state == CPUHP_AP_ONLINE_DYN || - state == CPUHP_BP_PREPARE_DYN)) { + state == CPUHP_BP_PREPARE_DYN || + state == CPUHP_BP_PARALLEL_DYN)) { ret = cpuhp_reserve_state(state); if (ret < 0) return ret; From patchwork Tue Mar 21 19:40:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 73025 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp1974512wrt; Tue, 21 Mar 2023 12:44:54 -0700 (PDT) X-Google-Smtp-Source: AK7set+xpGeHLy0JOgbOtK6pzmG/uZEaXZrsKYt5Mz8dQcM5JSJ/4qcuR754saizFMEvWNfhpZok X-Received: by 2002:a17:903:2903:b0:1a0:44e7:59dc with SMTP id lh3-20020a170903290300b001a044e759dcmr217574plb.40.1679427893875; Tue, 21 Mar 2023 12:44:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679427893; cv=none; d=google.com; s=arc-20160816; b=A7/GlIv0INrDztw8Ex61EGCilkWI9R76o7CjmSHtFl6J1x1mae/f4kPTgBALhjufuQ D0GE1C2G5cxNUr81tUcQ5Aj16U5vxg3fXZeTEmhKsndU71RUhcADrq2ywolSKtFp1OO5 uZWvCa1+z7ZMNBxihLlDlPjev9tdh5IfztOkTffm+BRmRWiwl03S3UN79WdF9kbvKc5v e8FK23CheIeagPOSmS3jvrTOzW53jRQk3FPCSS7mC/AQWGz+Ob03cf6wljWMXt+/eFws 5twaSs5bbzy2WDHbn13a1X/Yqgx1l0hEl/Se+nYR/CMq9nYN0GT1beTOBD+gcD9LISUG bMDQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=InZdH9g7WgaH3myKo4e9txmH25mdwzb46V9DlSyz+Qw=; b=GQROcSxsM0710z8+lQPJGa4I6MqCpaEROUq2DhKNqjcrIv/EZdNGbdOcX475NAr3Bg 6nYsSVk+305WCT2O9I33udl0jfBCYJe2fkwnsO6Gmg9NKd/IIB5oDx9+1rO9LpdKFzHx guqap8mkTh07xqY5KlHdc1DgxSinP/jYZRJHJsasYLE1WEylu1l/HSm560IS4prHYz8c FLtQxXAAoMMcZez2BuUrjSL5ymn68fV/z7Kjal2OyuSrhxBdV2KwQXhB9KgauS69h9qa Em3PYZrH0qeh5zzuABx2eL8iSsMMui2ZoNobr44BvpM4kYZwkIBr08+swDh64X84oXWA z6oQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=iHntaj0f; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id d18-20020a170902729200b0019f73eb630fsi13459979pll.637.2023.03.21.12.44.41; Tue, 21 Mar 2023 12:44:53 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=iHntaj0f; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230288AbjCUTlg (ORCPT + 99 others); Tue, 21 Mar 2023 15:41:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33852 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230253AbjCUTlc (ORCPT ); Tue, 21 Mar 2023 15:41:32 -0400 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CA61651FB2 for ; Tue, 21 Mar 2023 12:40:54 -0700 (PDT) Received: by mail-wr1-x42d.google.com with SMTP id l27so6563296wrb.2 for ; Tue, 21 Mar 2023 12:40:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1679427638; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=InZdH9g7WgaH3myKo4e9txmH25mdwzb46V9DlSyz+Qw=; b=iHntaj0fw4OiyphpXTxsruryaSfzgK6xqb6AxgCx5a8pL34IS2YoH93cKuIlYNJeuA 8HLvn+q7/8NGsLDnQpeWnSoSOPO76m+5Eer1LHHJovenoqwse8L8OYvFy1CQzh3gPlwl 8TEeMmQ4XpC7IOtTuIMwwF61G3L6+fgWC9tSSZVp5mExbIyub6dv4807UiqXg74huSOO j3s8jLwO4fMzALWTAUWtxsKFYw/HD2MljdWf6IolqUoqCggt2WePksUSbzBpeBGGp5Iz v/R6K8TfRbrROnG4RDN5LRLruc8+fF3EzAendFiL6tKvRyiDEinNGyzLQfSf78A8QGc/ PpKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679427638; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=InZdH9g7WgaH3myKo4e9txmH25mdwzb46V9DlSyz+Qw=; b=DoR5wHG0Np+aTvnzZq4PgKSjAck7ek0jW4FoWZUOOF6YOrUM7O3vdscmerDum3cERA UEvpDVyreDhZAuOcDKL9f5g5Vz+eaY5hg1l6hm+Ws1Kl39S0AhRRD9ewlEwE3NwEaJ0X Z/MCjzGYEpl+CPKbEz57oeq97vKhH4rNB8CdDgPhfduMw/eGqVIfSaX9H7vFoUWGH0UG /GbyXow9rKPrfwAOteIc/Lhh21HgF8v16ZBRw5BeZ08XrviR//IaAx3yMtbiI6PGc1o6 W+HjtUtQQnEK90imYGAp0ssCrL2jPCd4/EW1oxNI3QY5AeIhzGLLkJlkc9ZxFsI9sS5c Wiqg== X-Gm-Message-State: AO0yUKVUIgBKEDOnQTrLUDa/SPIgxEoq/fkEECxNmerNnNqiMs1+u8T8 J3Ahs7zTSrqjnzqOdq85FWIZLQ== X-Received: by 2002:a5d:58d4:0:b0:2ce:a85b:174 with SMTP id o20-20020a5d58d4000000b002cea85b0174mr3302705wrf.44.1679427638062; Tue, 21 Mar 2023 12:40:38 -0700 (PDT) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:7a8a:d679:ba4e:61cf]) by smtp.gmail.com with ESMTPSA id a3-20020adffb83000000b002c561805a4csm12005026wrr.45.2023.03.21.12.40.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Mar 2023 12:40:37 -0700 (PDT) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, gpiccoli@igalia.com, David Woodhouse , Usama Arif Subject: [PATCH v16 4/8] x86/smpboot: Split up native_cpu_up into separate phases and document them Date: Tue, 21 Mar 2023 19:40:04 +0000 Message-Id: <20230321194008.785922-5-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230321194008.785922-1-usama.arif@bytedance.com> References: <20230321194008.785922-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761007783634213780?= X-GMAIL-MSGID: =?utf-8?q?1761007783634213780?= From: David Woodhouse There are four logical parts to what native_cpu_up() does on the BSP (or on the controlling CPU for a later hotplug): 1) Wake the AP by sending the INIT/SIPI/SIPI sequence. 2) Wait for the AP to make it as far as wait_for_master_cpu() which sets that CPU's bit in cpu_initialized_mask, then sets the bit in cpu_callout_mask to let the AP proceed through cpu_init(). 3) Wait for the AP to finish cpu_init() and get as far as the smp_callin() call, which sets that CPU's bit in cpu_callin_mask. 4) Perform the TSC synchronization and wait for the AP to actually mark itself online in cpu_online_mask. In preparation to allow these phases to operate in parallel on multiple APs, split them out into separate functions and document the interactions a little more clearly in both the BSP and AP code paths. No functional change intended. Signed-off-by: David Woodhouse Signed-off-by: Usama Arif Tested-by: Paul E. McKenney Tested-by: Kim Phillips Tested-by: Oleksandr Natalenko Tested-by: Guilherme G. Piccoli --- arch/x86/kernel/smpboot.c | 179 ++++++++++++++++++++++++++------------ 1 file changed, 125 insertions(+), 54 deletions(-) diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 851477f7d728..2a8a18e23cc6 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -204,6 +204,10 @@ static void smp_callin(void) wmb(); + /* + * This runs the AP through all the cpuhp states to its target + * state (CPUHP_ONLINE in the case of serial bringup). + */ notify_cpu_starting(cpuid); /* @@ -231,17 +235,32 @@ static void notrace start_secondary(void *unused) load_cr3(swapper_pg_dir); __flush_tlb_all(); #endif + /* + * Sync point with do_wait_cpu_initialized(). Before proceeding through + * cpu_init(), the AP will call wait_for_master_cpu() which sets its + * own bit in cpu_initialized_mask and then waits for the BSP to set + * its bit in cpu_callout_mask to release it. + */ cpu_init_secondary(); rcu_cpu_starting(raw_smp_processor_id()); x86_cpuinit.early_percpu_clock_init(); + + /* + * Sync point with do_wait_cpu_callin(). The AP doesn't wait here + * but just sets the bit to let the controlling CPU (BSP) know that + * it's got this far. + */ smp_callin(); enable_start_cpu0 = 0; /* otherwise gcc will move up smp_processor_id before the cpu_init */ barrier(); + /* - * Check TSC synchronization with the boot CPU: + * Check TSC synchronization with the boot CPU (or whichever CPU + * is controlling the bringup). It will do its part of this from + * do_wait_cpu_online(), making it an implicit sync point. */ check_tsc_sync_target(); @@ -254,6 +273,7 @@ static void notrace start_secondary(void *unused) * half valid vector space. */ lock_vector_lock(); + /* Sync point with do_wait_cpu_online() */ set_cpu_online(smp_processor_id(), true); lapic_online(); unlock_vector_lock(); @@ -1081,7 +1101,6 @@ static int do_boot_cpu(int apicid, int cpu, struct task_struct *idle, unsigned long start_ip = real_mode_header->trampoline_start; unsigned long boot_error = 0; - unsigned long timeout; #ifdef CONFIG_X86_64 /* If 64-bit wakeup method exists, use the 64-bit mode trampoline IP */ @@ -1147,55 +1166,94 @@ static int do_boot_cpu(int apicid, int cpu, struct task_struct *idle, boot_error = wakeup_cpu_via_init_nmi(cpu, start_ip, apicid, cpu0_nmi_registered); - if (!boot_error) { - /* - * Wait 10s total for first sign of life from AP - */ - boot_error = -1; - timeout = jiffies + 10*HZ; - while (time_before(jiffies, timeout)) { - if (cpumask_test_cpu(cpu, cpu_initialized_mask)) { - /* - * Tell AP to proceed with initialization - */ - cpumask_set_cpu(cpu, cpu_callout_mask); - boot_error = 0; - break; - } - schedule(); - } - } + return boot_error; +} - if (!boot_error) { - /* - * Wait till AP completes initial initialization - */ - while (!cpumask_test_cpu(cpu, cpu_callin_mask)) { - /* - * Allow other tasks to run while we wait for the - * AP to come online. This also gives a chance - * for the MTRR work(triggered by the AP coming online) - * to be completed in the stop machine context. - */ - schedule(); - } +static int do_wait_cpu_cpumask(unsigned int cpu, const struct cpumask *mask) +{ + unsigned long timeout; + + /* + * Wait up to 10s for the CPU to report in. + */ + timeout = jiffies + 10*HZ; + while (time_before(jiffies, timeout)) { + if (cpumask_test_cpu(cpu, mask)) + return 0; + + schedule(); } + return -1; +} - if (x86_platform.legacy.warm_reset) { - /* - * Cleanup possible dangling ends... - */ - smpboot_restore_warm_reset_vector(); +/* + * Bringup step two: Wait for the target AP to reach cpu_init_secondary() + * and thus wait_for_master_cpu(), then set cpu_callout_mask to allow it + * to proceed. The AP will then proceed past setting its 'callin' bit + * and end up waiting in check_tsc_sync_target() until we reach + * do_wait_cpu_online() to tend to it. + */ +static int do_wait_cpu_initialized(unsigned int cpu) +{ + /* + * Wait for first sign of life from AP. + */ + if (do_wait_cpu_cpumask(cpu, cpu_initialized_mask)) + return -1; + + cpumask_set_cpu(cpu, cpu_callout_mask); + return 0; +} + +/* + * Bringup step three: Wait for the target AP to reach smp_callin(). + * The AP is not waiting for us here so we don't need to parallelise + * this step. Not entirely clear why we care about this, since we just + * proceed directly to TSC synchronization which is the next sync + * point with the AP anyway. + */ +static int do_wait_cpu_callin(unsigned int cpu) +{ + /* + * Wait till AP completes initial initialization. + */ + return do_wait_cpu_cpumask(cpu, cpu_callin_mask); +} + +/* + * Bringup step four: Synchronize the TSC and wait for the target AP + * to reach set_cpu_online() in start_secondary(). + */ +static int do_wait_cpu_online(unsigned int cpu) +{ + unsigned long flags; + + /* + * Check TSC synchronization with the AP (keep irqs disabled + * while doing so): + */ + local_irq_save(flags); + check_tsc_sync_source(cpu); + local_irq_restore(flags); + + /* + * Wait for the AP to mark itself online. Not entirely + * clear why we care, since the generic cpuhp code will + * wait for it to each CPUHP_AP_ONLINE_IDLE before going + * ahead with the rest of the bringup anyway. + */ + while (!cpu_online(cpu)) { + cpu_relax(); + touch_nmi_watchdog(); } - return boot_error; + return 0; } -int native_cpu_up(unsigned int cpu, struct task_struct *tidle) +static int do_cpu_up(unsigned int cpu, struct task_struct *tidle) { int apicid = apic->cpu_present_to_apicid(cpu); int cpu0_nmi_registered = 0; - unsigned long flags; int err, ret = 0; lockdep_assert_irqs_enabled(); @@ -1242,19 +1300,6 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) goto unreg_nmi; } - /* - * Check TSC synchronization with the AP (keep irqs disabled - * while doing so): - */ - local_irq_save(flags); - check_tsc_sync_source(cpu); - local_irq_restore(flags); - - while (!cpu_online(cpu)) { - cpu_relax(); - touch_nmi_watchdog(); - } - unreg_nmi: /* * Clean up the nmi handler. Do this after the callin and callout sync @@ -1266,6 +1311,32 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) return ret; } +int native_cpu_up(unsigned int cpu, struct task_struct *tidle) +{ + int ret; + + ret = do_cpu_up(cpu, tidle); + if (ret) + goto out; + + ret = do_wait_cpu_initialized(cpu); + if (ret) + goto out; + + ret = do_wait_cpu_callin(cpu); + if (ret) + goto out; + + ret = do_wait_cpu_online(cpu); + + out: + /* Cleanup possible dangling ends... */ + if (x86_platform.legacy.warm_reset) + smpboot_restore_warm_reset_vector(); + + return ret; +} + /** * arch_disable_smp_support() - disables SMP support for x86 at runtime */ From patchwork Tue Mar 21 19:40:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 73027 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp1975648wrt; Tue, 21 Mar 2023 12:47:30 -0700 (PDT) X-Google-Smtp-Source: AK7set8jyrOGQdPrps7reP17aMEI9m812ydkvkGWZisPlY1OcIb5NcQCCrm3FQiUvSuaImPh5SFw X-Received: by 2002:aa7:9e04:0:b0:626:dc8:b004 with SMTP id y4-20020aa79e04000000b006260dc8b004mr768058pfq.26.1679428049713; Tue, 21 Mar 2023 12:47:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679428049; cv=none; d=google.com; s=arc-20160816; b=bFKsdA4sbI86HbSg8+pW8opBsSO1NXotCkqpQ+Ma3nrpXgtEQ4mH4elZW1z8FKJbPi ksGkSUW7eRRZNSCxkCd3zIvzkwV20i0NmqBcxaJlN5xkNjIxo65OvwhADciPt3J3ijcO 2xuds/hhHfkVrM3omnnLknTrH5mhjaIJ+6OrQKUkwed6JGcL1d9Ww/wv/7+qu38W+bUk pfrW8CYQUFNKQJAQhx3LvD256YWEEDAEiQ6hA8ANOxr4V9DKPXrog+C2Pevgl5G83MHX zth+GcQK9gC75SODAA0sx8CeFFEHHCQarCPmJBIi0wgil7umpUVFhKnhJk+awNMVNqAt qwNw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=WMdNWTJBH3JMn7SBa4haHVmNvEwQaMIEDuQZl+CDNU8=; b=Lz/0iFj8pnyUuDiY+C8RNHDqnwKxZOfgntte9w+2DtqNUWRcXx+zxW63LKUm2A/phj IQRbDt6Gyj71mEZMqxezzimI52O/1MFHcx8kpugV1ffsa9iStOCf2xnS3+YJzYKOWOST kq1XXzbE3PuBjBZOu5h/3XNkWMnZdrGrwvucPitH2wM6MM101kvdRO2aiPtQYCOvPZVl A6AdsmB4wckSTm2sCLi1D5Kib15z8NhKBMvNUtWdwF+pI0fmTG2aIBCv0f+hfjDMc29c 5ELtbtqqmRT/3O5BfEspedbhfZY2nP/2MLm1RWu1nNcRNbpm36+rbOA57niPhFRPA1Z9 GQxg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=GvnLWsLM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c5-20020a056a000ac500b0059072f087b7si14272135pfl.135.2023.03.21.12.47.17; Tue, 21 Mar 2023 12:47:29 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=GvnLWsLM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229801AbjCUTmG (ORCPT + 99 others); Tue, 21 Mar 2023 15:42:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33852 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230346AbjCUTl7 (ORCPT ); Tue, 21 Mar 2023 15:41:59 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0DD32D520 for ; Tue, 21 Mar 2023 12:41:02 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id y14so14884176wrq.4 for ; Tue, 21 Mar 2023 12:41:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1679427639; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WMdNWTJBH3JMn7SBa4haHVmNvEwQaMIEDuQZl+CDNU8=; b=GvnLWsLM1oGKQf9HBqqePCaBRgY9kZajm1lIOBw75VKVvocgkPDdbIMRL74PPiybpd ni9YmQxegJOAWceDRKWr6d1xPCe78+PnMlO31zaXkBxK5491LKTehza0t9+lTP6Q0+8B kX81sTcA9YaxEuXbJKHRmSDUbomq3W2eAzys7JQ75uJAT65I+1FordZJIkQVsAMNStwF faFjuH6WCvgNEVk91NNIp2yJ84DuYxes5k/MOhSqfRlHstuoi9Mfy+XIaaigqGXNvA3P lxdxjOHsQAzYebHkImpq+S+o/cto/edwVvoHl6RpZ3FFqajjIuGd7dco99i4xMea7to5 Q0rQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679427639; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WMdNWTJBH3JMn7SBa4haHVmNvEwQaMIEDuQZl+CDNU8=; b=By5+qZ/0ij0Uc9SI9cWDvMYX3dwAxqsFNBJVKTGATYXECClh+MmjSSmnHOV39q7wjV 58UtxHdWJ5NGOGf5USBmaaoBQw6HDSAy6ELZQOcG/UINX+v2S2OLmJPhr+NE8osuSdt9 Ibp93mNgzl9EV0sXkpgnoDkmP5gJA8/XSMkITDdDnNEmeIgRY+63QbyQjXRFWb1/L39V q4/ZxQsUetdmOOefHrcLync0p/b61e3DliV905lb+HovZf3pQu2TnyG+XL+xHyqjwJc0 7W3UqxvMwKOGZh/Os4q1pKTTTD07ZMn8s6bCfZXA2yhsUcK9K/NaxBZW0pgYMMgZC6NU 1RRA== X-Gm-Message-State: AO0yUKXm70WX+ZKHS2J5eU4vHXYl3Pyx6QQq+VQnXU0ttnSS9DJUhuyl EDi6u+0sr00iw1YyWfTxjVfoEQ== X-Received: by 2002:adf:d0cb:0:b0:2c7:efb:dded with SMTP id z11-20020adfd0cb000000b002c70efbddedmr3007834wrh.24.1679427639097; Tue, 21 Mar 2023 12:40:39 -0700 (PDT) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:7a8a:d679:ba4e:61cf]) by smtp.gmail.com with ESMTPSA id a3-20020adffb83000000b002c561805a4csm12005026wrr.45.2023.03.21.12.40.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Mar 2023 12:40:38 -0700 (PDT) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, gpiccoli@igalia.com, David Woodhouse , Usama Arif Subject: [PATCH v16 5/8] x86/smpboot: Support parallel startup of secondary CPUs Date: Tue, 21 Mar 2023 19:40:05 +0000 Message-Id: <20230321194008.785922-6-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230321194008.785922-1-usama.arif@bytedance.com> References: <20230321194008.785922-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761007946365072490?= X-GMAIL-MSGID: =?utf-8?q?1761007946365072490?= From: David Woodhouse Rework the real-mode startup code to allow for APs to be brought up in parallel. This is in two parts: 1. Introduce a bit-spinlock to prevent them from all using the real mode stack at the same time. 2. Avoid needing to use the global smpboot_control variable to pass each AP its CPU#. To achieve the latter, export the cpuid_to_apicid[] array so that each AP can find its own CPU# by searching therein based on its APIC ID. Introduce flags in the top bits of smpboot_control which indicate methods by which an AP should find its CPU#. For a serialized bringup, the CPU# is explicitly passed in the low bits of smpboot_control as before. For parallel mode there are flags directing the AP to find its APIC ID in CPUID leaf 0x0b (for X2APIC mode) or CPUID leaf 0x01 where 8 bits are sufficient, then perform the cpuid_to_apicid[] lookup with that. Parallel startup may be disabled by a command line option, and also if: • AMD SEV-ES is in use, since the AP may not use CPUID that early. • X2APIC is enabled, but CPUID leaf 0xb is not present and correct. • X2APIC is not enabled but not even CPUID leaf 0x01 exists. Aside from the fact that APs will now look up their CPU# via the newly-exported cpuid_to_apicid[] table, there is no behavioural change intended yet, since new parallel CPUHP states have not — yet — been added. [ tglx: Initial proof of concept patch with bitlock and APIC ID lookup ] [ dwmw2: Rework and testing, commit message, CPUID 0x1 and CPU0 support ] [ seanc: Fix stray override of initial_gs in common_cpu_up() ] [ Oleksandr Natalenko: reported suspend/resume issue fixed in x86_acpi_suspend_lowlevel ] Co-developed-by: Thomas Gleixner Co-developed-by: Brian Gerst Signed-off-by: Thomas Gleixner Signed-off-by: Brian Gerst Signed-off-by: David Woodhouse Signed-off-by: Usama Arif Tested-by: Paul E. McKenney Tested-by: Kim Phillips Tested-by: Oleksandr Natalenko Tested-by: Guilherme G. Piccoli --- .../admin-guide/kernel-parameters.txt | 3 + arch/x86/include/asm/cpu.h | 1 + arch/x86/include/asm/realmode.h | 3 + arch/x86/include/asm/smp.h | 6 ++ arch/x86/kernel/acpi/sleep.c | 9 ++- arch/x86/kernel/apic/apic.c | 2 +- arch/x86/kernel/cpu/topology.c | 3 +- arch/x86/kernel/head_64.S | 67 +++++++++++++++++++ arch/x86/kernel/smpboot.c | 50 +++++++++++++- arch/x86/realmode/init.c | 3 + arch/x86/realmode/rm/trampoline_64.S | 27 ++++++-- 11 files changed, 165 insertions(+), 9 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 6221a1d057dd..424151f296ff 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3822,6 +3822,9 @@ nomodule Disable module load + no_parallel_bringup + [X86,SMP] Disable parallel bring-up of secondary cores. + nopat [X86] Disable PAT (page attribute table extension of pagetables) support. diff --git a/arch/x86/include/asm/cpu.h b/arch/x86/include/asm/cpu.h index 78796b98a544..ef8ba318dca1 100644 --- a/arch/x86/include/asm/cpu.h +++ b/arch/x86/include/asm/cpu.h @@ -97,5 +97,6 @@ static inline bool intel_cpu_signatures_match(unsigned int s1, unsigned int p1, extern u64 x86_read_arch_cap_msr(void); int intel_find_matching_signature(void *mc, unsigned int csig, int cpf); int intel_microcode_sanity_check(void *mc, bool print_err, int hdr_type); +int check_extended_topology_leaf(int leaf); #endif /* _ASM_X86_CPU_H */ diff --git a/arch/x86/include/asm/realmode.h b/arch/x86/include/asm/realmode.h index f6a1737c77be..87e5482acd0d 100644 --- a/arch/x86/include/asm/realmode.h +++ b/arch/x86/include/asm/realmode.h @@ -52,6 +52,7 @@ struct trampoline_header { u64 efer; u32 cr4; u32 flags; + u32 lock; #endif }; @@ -64,6 +65,8 @@ extern unsigned long initial_stack; extern unsigned long initial_vc_handler; #endif +extern u32 *trampoline_lock; + extern unsigned char real_mode_blob[]; extern unsigned char real_mode_relocs[]; diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h index bf2c51df9e0b..1cf4f1e57570 100644 --- a/arch/x86/include/asm/smp.h +++ b/arch/x86/include/asm/smp.h @@ -203,4 +203,10 @@ extern unsigned int smpboot_control; #endif /* !__ASSEMBLY__ */ +/* Control bits for startup_64 */ +#define STARTUP_APICID_CPUID_0B 0x80000000 +#define STARTUP_APICID_CPUID_01 0x40000000 + +#define STARTUP_PARALLEL_MASK (STARTUP_APICID_CPUID_01 | STARTUP_APICID_CPUID_0B) + #endif /* _ASM_X86_SMP_H */ diff --git a/arch/x86/kernel/acpi/sleep.c b/arch/x86/kernel/acpi/sleep.c index 1328c221af30..6dfecb27b846 100644 --- a/arch/x86/kernel/acpi/sleep.c +++ b/arch/x86/kernel/acpi/sleep.c @@ -16,6 +16,7 @@ #include #include #include +#include #include #include "../../realmode/rm/wakeup.h" @@ -127,7 +128,13 @@ int x86_acpi_suspend_lowlevel(void) * value is in the actual %rsp register. */ current->thread.sp = (unsigned long)temp_stack + sizeof(temp_stack); - smpboot_control = smp_processor_id(); + /* + * Ensure the CPU knows which one it is when it comes back, if + * it isn't in parallel mode and expected to work that out for + * itself. + */ + if (!(smpboot_control & STARTUP_PARALLEL_MASK)) + smpboot_control = smp_processor_id(); #endif initial_code = (unsigned long)wakeup_long64; saved_magic = 0x123456789abcdef0L; diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c index 20d9a604da7c..ac1d7e5da1f2 100644 --- a/arch/x86/kernel/apic/apic.c +++ b/arch/x86/kernel/apic/apic.c @@ -2377,7 +2377,7 @@ static int nr_logical_cpuids = 1; /* * Used to store mapping between logical CPU IDs and APIC IDs. */ -static int cpuid_to_apicid[] = { +int cpuid_to_apicid[] = { [0 ... NR_CPUS - 1] = -1, }; diff --git a/arch/x86/kernel/cpu/topology.c b/arch/x86/kernel/cpu/topology.c index 5e868b62a7c4..4373442e500a 100644 --- a/arch/x86/kernel/cpu/topology.c +++ b/arch/x86/kernel/cpu/topology.c @@ -9,6 +9,7 @@ #include #include #include +#include #include "cpu.h" @@ -32,7 +33,7 @@ EXPORT_SYMBOL(__max_die_per_package); /* * Check if given CPUID extended topology "leaf" is implemented */ -static int check_extended_topology_leaf(int leaf) +int check_extended_topology_leaf(int leaf) { unsigned int eax, ebx, ecx, edx; diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index 6a8238702eab..ff3a5f008d8a 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -25,6 +25,7 @@ #include #include #include +#include /* * We are not able to switch in one step to the final KERNEL ADDRESS SPACE @@ -234,8 +235,61 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) ANNOTATE_NOENDBR // above #ifdef CONFIG_SMP + /* + * For parallel boot, the APIC ID is retrieved from CPUID, and then + * used to look up the CPU number. For booting a single CPU, the + * CPU number is encoded in smpboot_control. + * + * Bit 31 STARTUP_APICID_CPUID_0B flag (use CPUID 0x0b) + * Bit 30 STARTUP_APICID_CPUID_01 flag (use CPUID 0x01) + * Bit 0-24 CPU# if STARTUP_APICID_CPUID_xx flags are not set + */ movl smpboot_control(%rip), %ecx + testl $STARTUP_APICID_CPUID_0B, %ecx + jnz .Luse_cpuid_0b + testl $STARTUP_APICID_CPUID_01, %ecx + jnz .Luse_cpuid_01 + andl $0x0FFFFFFF, %ecx + jmp .Lsetup_cpu + +.Luse_cpuid_01: + mov $0x01, %eax + cpuid + mov %ebx, %edx + shr $24, %edx + jmp .Lsetup_AP +.Luse_cpuid_0b: + mov $0x0B, %eax + xorl %ecx, %ecx + cpuid + +.Lsetup_AP: + /* EDX contains the APIC ID of the current CPU */ + xorq %rcx, %rcx + leaq cpuid_to_apicid(%rip), %rbx + +.Lfind_cpunr: + cmpl (%rbx,%rcx,4), %edx + jz .Lsetup_cpu + inc %ecx +#ifdef CONFIG_FORCE_NR_CPUS + cmpl $NR_CPUS, %ecx +#else + cmpl nr_cpu_ids(%rip), %ecx +#endif + jb .Lfind_cpunr + + /* APIC ID not found in the table. Drop the trampoline lock and bail. */ + movq trampoline_lock(%rip), %rax + lock + btrl $0, (%rax) + +1: cli + hlt + jmp 1b + +.Lsetup_cpu: /* Get the per cpu offset for the given CPU# which is in ECX */ movq __per_cpu_offset(,%rcx,8), %rdx #else @@ -251,6 +305,17 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) movq pcpu_hot + X86_current_task(%rdx), %rax movq TASK_threadsp(%rax), %rsp + /* + * Now that this CPU is running on its own stack, drop the realmode + * protection. For the boot CPU the pointer is NULL! + */ + movq trampoline_lock(%rip), %rax + testq %rax, %rax + jz .Lsetup_gdt + lock + btrl $0, (%rax) + +.Lsetup_gdt: /* * We must switch to a new descriptor in kernel space for the GDT * because soon the kernel won't have access anymore to the userspace @@ -435,6 +500,8 @@ SYM_DATA(initial_code, .quad x86_64_start_kernel) #ifdef CONFIG_AMD_MEM_ENCRYPT SYM_DATA(initial_vc_handler, .quad handle_vc_boot_ghcb) #endif + +SYM_DATA(trampoline_lock, .quad 0); __FINITDATA __INIT diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 2a8a18e23cc6..8f7f36721ea8 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -797,6 +797,16 @@ static int __init cpu_init_udelay(char *str) } early_param("cpu_init_udelay", cpu_init_udelay); +static bool do_parallel_bringup __ro_after_init = true; + +static int __init no_parallel_bringup(char *str) +{ + do_parallel_bringup = false; + + return 0; +} +early_param("no_parallel_bringup", no_parallel_bringup); + static void __init smp_quirk_init_udelay(void) { /* if cmdline changed it from default, leave it alone */ @@ -1113,7 +1123,7 @@ static int do_boot_cpu(int apicid, int cpu, struct task_struct *idle, if (IS_ENABLED(CONFIG_X86_32)) { early_gdt_descr.address = (unsigned long)get_cpu_gdt_rw(cpu); initial_stack = idle->thread.sp; - } else { + } else if (!do_parallel_bringup) { smpboot_control = cpu; } @@ -1473,6 +1483,41 @@ void __init smp_prepare_cpus_common(void) set_cpu_sibling_map(0); } +/* + * We can do 64-bit AP bringup in parallel if the CPU reports its APIC + * ID in CPUID (either leaf 0x0B if we need the full APIC ID in X2APIC + * mode, or leaf 0x01 if 8 bits are sufficient). Otherwise it's too + * hard. And not for SEV-ES guests because they can't use CPUID that + * early. + */ +static bool prepare_parallel_bringup(void) +{ + if (IS_ENABLED(CONFIG_X86_32) || cc_platform_has(CC_ATTR_GUEST_STATE_ENCRYPT)) + return false; + + if (x2apic_mode) { + if (boot_cpu_data.cpuid_level < 0x0b) + return false; + + if (check_extended_topology_leaf(0x0b) != 0) { + pr_info("Disabling parallel bringup because CPUID 0xb looks untrustworthy\n"); + return false; + } + + pr_debug("Using CPUID 0xb for parallel CPU startup\n"); + smpboot_control = STARTUP_APICID_CPUID_0B; + } else { + /* Without X2APIC, what's in CPUID 0x01 should suffice. */ + if (boot_cpu_data.cpuid_level < 0x01) + return false; + + pr_debug("Using CPUID 0x1 for parallel CPU startup\n"); + smpboot_control = STARTUP_APICID_CPUID_01; + } + + return true; +} + /* * Prepare for SMP bootup. * @max_cpus: configured maximum number of CPUs, It is a legacy parameter @@ -1513,6 +1558,9 @@ void __init native_smp_prepare_cpus(unsigned int max_cpus) speculative_store_bypass_ht_init(); + if (do_parallel_bringup) + do_parallel_bringup = prepare_parallel_bringup(); + snp_set_wakeup_secondary_cpu(); } diff --git a/arch/x86/realmode/init.c b/arch/x86/realmode/init.c index af565816d2ba..788e5559549f 100644 --- a/arch/x86/realmode/init.c +++ b/arch/x86/realmode/init.c @@ -154,6 +154,9 @@ static void __init setup_real_mode(void) trampoline_header->flags = 0; + trampoline_lock = &trampoline_header->lock; + *trampoline_lock = 0; + trampoline_pgd = (u64 *) __va(real_mode_header->trampoline_pgd); /* Map the real mode stub as virtual == physical */ diff --git a/arch/x86/realmode/rm/trampoline_64.S b/arch/x86/realmode/rm/trampoline_64.S index e38d61d6562e..2dfb1c400167 100644 --- a/arch/x86/realmode/rm/trampoline_64.S +++ b/arch/x86/realmode/rm/trampoline_64.S @@ -37,6 +37,24 @@ .text .code16 +.macro LOAD_REALMODE_ESP + /* + * Make sure only one CPU fiddles with the realmode stack + */ +.Llock_rm\@: + btl $0, tr_lock + jnc 2f + pause + jmp .Llock_rm\@ +2: + lock + btsl $0, tr_lock + jc .Llock_rm\@ + + # Setup stack + movl $rm_stack_end, %esp +.endm + .balign PAGE_SIZE SYM_CODE_START(trampoline_start) cli # We should be safe anyway @@ -49,8 +67,7 @@ SYM_CODE_START(trampoline_start) mov %ax, %es mov %ax, %ss - # Setup stack - movl $rm_stack_end, %esp + LOAD_REALMODE_ESP call verify_cpu # Verify the cpu supports long mode testl %eax, %eax # Check for return code @@ -93,8 +110,7 @@ SYM_CODE_START(sev_es_trampoline_start) mov %ax, %es mov %ax, %ss - # Setup stack - movl $rm_stack_end, %esp + LOAD_REALMODE_ESP jmp .Lswitch_to_protected SYM_CODE_END(sev_es_trampoline_start) @@ -177,7 +193,7 @@ SYM_CODE_START(pa_trampoline_compat) * In compatibility mode. Prep ESP and DX for startup_32, then disable * paging and complete the switch to legacy 32-bit mode. */ - movl $rm_stack_end, %esp + LOAD_REALMODE_ESP movw $__KERNEL_DS, %dx movl $(CR0_STATE & ~X86_CR0_PG), %eax @@ -241,6 +257,7 @@ SYM_DATA_START(trampoline_header) SYM_DATA(tr_efer, .space 8) SYM_DATA(tr_cr4, .space 4) SYM_DATA(tr_flags, .space 4) + SYM_DATA(tr_lock, .space 4) SYM_DATA_END(trampoline_header) #include "trampoline_common.S" From patchwork Tue Mar 21 19:40:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 73030 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp1976015wrt; Tue, 21 Mar 2023 12:48:32 -0700 (PDT) X-Google-Smtp-Source: AK7set++kx+s0+9d5G2qLWEVpJZhQiwnO5PYT/2zkBwSV4nK5ymqCvwHBfMKl6lvuUSd7NG2mMrJ X-Received: by 2002:a17:90b:4fc7:b0:23d:3264:16bf with SMTP id qa7-20020a17090b4fc700b0023d326416bfmr962006pjb.0.1679428112421; Tue, 21 Mar 2023 12:48:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679428112; cv=none; d=google.com; s=arc-20160816; b=V/8yPUy3rr+5SgPdB/nEgFA8szPXszL7w+6eWqkYBgeOqSd6Catd9ljmEQDLVCrMqF wXOZZhOh9Wn3rO6zQCcVffjXdizTD9AbuJbBF1oFK8RFFPm++7HpQVw3kITkj59ZJ4ww t4zdktbMpIgKPNEnBy0RWGnZb59bA6sbAK+miw8rM6MIkx6MCZ2htezPEzztsJm/WDRk xMtDmLK+SZw9qOvdceIvxxhExO3LH6eLaoGkdBhmtyrkxrgaevXvl68YbbQiT93RVD0a +vgKlQZI2qk6I7vE/vferne+Y17iya0vNy02rJc4iR3bvWF6VOCLdvliZWRoy4csMD8Y xzwA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=713eNkzIsW4JnC6r4Qq0A/MbdlkjMzmV9oMocjSXM2A=; b=WXNBu8NoH7y4joFT1wQ9ZeWiJpWVUljyO8xCOSlZ+PitQhWIPfXEOa364fYMK4hUWU +FdPmuV1u2TewTQ4nlkWvk+04NuvNitu1K5TYy4DHx6zSCzvlfRgrWYNbB/PYu9tJtBV 1bJhQRJ3E9wB5z8d7WE2diHK1C5u4A0WaxskgWM2uKPj/sdCi0kqhMLJS5WivrxjLv/8 5ivvsZE0GjKXz4efm0uUud1IfZ1AHlNNdwZl8DbFle2Tmx7L42nLbDaWkn8iSOs1LvAR PGhqrSmUUEIh0ldRG6U5UqfTxHZUJsk1lWn3pbpa9l77lxQ3Lb2WjP/beQ/cnHmeXsjQ I+bQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=GFvc9lFJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f29-20020a63555d000000b0050bf0f1b79csi14036692pgm.699.2023.03.21.12.48.19; Tue, 21 Mar 2023 12:48:32 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=GFvc9lFJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229839AbjCUTmN (ORCPT + 99 others); Tue, 21 Mar 2023 15:42:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33996 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229635AbjCUTmB (ORCPT ); Tue, 21 Mar 2023 15:42:01 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5E9EF40DB for ; Tue, 21 Mar 2023 12:41:04 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id i9so14873572wrp.3 for ; Tue, 21 Mar 2023 12:41:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1679427640; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=713eNkzIsW4JnC6r4Qq0A/MbdlkjMzmV9oMocjSXM2A=; b=GFvc9lFJiKr6fg+/7WOD6Eg3fbx6ow2SRG8rgbJntZLZLX+FtpF8NJOlDIwAvuRAj2 l/b8hvQlrsWeIAQuX6rFqT//lsZwWhQhM5clGIwcmO6+zDArNTA8TyQTGvT9ipKqeaiV Amv9m971asKm1ZZX+HYMV67R5qaSo7mHxYObjaASdmSoATbWAqcRfru9Rpswjn9oBAAd y6K67ADIOEUNJmUsnkLfX4CTq5BF+xJbwo7012Xq/3IosVQMV2ksBi+2de42En5A/mTb mnSMXqt6oFclJ/9XZHTe4upXc+exRMxyMUfLsOfCMmwTiLFonGkJ9pqNrSUCrAJ/vspj mVXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679427640; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=713eNkzIsW4JnC6r4Qq0A/MbdlkjMzmV9oMocjSXM2A=; b=kqK7/EgiYm1f+LooEWgkHPvObRfYp2s1a0OUuqoAeeaPDWISNjKHNCtWpEZ24/l3nP 2LBYnW8yF89jX21JLxfjXQZJJ1PdjeImhOjtUe/DnzuvXN9dGriKzZFHjZdoNpB8of4C /CHiN+p97usethfZLLs+sccmXfDEYi5LTcWSPdSooS2pgMjeyXXeS+Sc6kOsYFsFq/J2 tSmxo32j62Bp4dpLrHQz0ErInMGq0ufBFXvVogZaVg11k6Vt8W8qjj56Jto5ZfNpEc4+ SY8kqPmuMilwwY58Jjup7Wf3tjDm6Sy6E6V7bYE749ReuUmNxbpf9xYS2BkTyc5U9Dl3 MxrA== X-Gm-Message-State: AO0yUKXU/i+fpaTrtABpHZtwA5dOApvfPiu1tIPitUwlVvIGtMnD74zb 2kvWX0ZtwnWZOpcFxHqu8dhOtQ== X-Received: by 2002:a5d:66c7:0:b0:2c7:1e43:e578 with SMTP id k7-20020a5d66c7000000b002c71e43e578mr3202990wrw.41.1679427640021; Tue, 21 Mar 2023 12:40:40 -0700 (PDT) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:7a8a:d679:ba4e:61cf]) by smtp.gmail.com with ESMTPSA id a3-20020adffb83000000b002c561805a4csm12005026wrr.45.2023.03.21.12.40.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Mar 2023 12:40:39 -0700 (PDT) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, gpiccoli@igalia.com, David Woodhouse , Usama Arif Subject: [PATCH v16 6/8] x86/smpboot: Send INIT/SIPI/SIPI to secondary CPUs in parallel Date: Tue, 21 Mar 2023 19:40:06 +0000 Message-Id: <20230321194008.785922-7-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230321194008.785922-1-usama.arif@bytedance.com> References: <20230321194008.785922-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761008012793982348?= X-GMAIL-MSGID: =?utf-8?q?1761008012793982348?= From: David Woodhouse When the APs can find their own APIC ID without assistance, perform the AP bringup in parallel. Register a CPUHP_BP_PARALLEL_DYN stage "x86/cpu:kick" which just calls do_boot_cpu() to deliver INIT/SIPI/SIPI to each AP in turn before the normal native_cpu_up() does the rest of the hand-holding. The APs will then take turns through the real mode code (which has its own bitlock for exclusion) until they make it to their own stack, then proceed through the first few lines of start_secondary() and execute these parts in parallel: start_secondary() -> cr4_init() -> (some 32-bit only stuff so not in the parallel cases) -> cpu_init_secondary() -> cpu_init_exception_handling() -> cpu_init() -> wait_for_master_cpu() At this point they wait for the BSP to set their bit in cpu_callout_mask (from do_wait_cpu_initialized()), and release them to continue through the rest of cpu_init() and beyond. This reduces the time taken for bringup on my 28-thread Haswell system from about 120ms to 80ms. On a socket 96-thread Skylake it takes the bringup time from 500ms to 100ms. There is more speedup to be had by doing the remaining parts in parallel too — especially notify_cpu_starting() in which the AP takes itself through all the stages from CPUHP_BRINGUP_CPU to CPUHP_ONLINE. But those require careful auditing to ensure they are reentrant, before we can go that far. Signed-off-by: David Woodhouse Signed-off-by: Usama Arif Tested-by: Paul E. McKenney Tested-by: Kim Phillips Tested-by: Oleksandr Natalenko Tested-by: Guilherme G. Piccoli --- arch/x86/kernel/smpboot.c | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 8f7f36721ea8..bb818e6d8701 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -57,6 +57,7 @@ #include #include #include +#include #include #include @@ -992,7 +993,8 @@ static void announce_cpu(int cpu, int apicid) node_width = num_digits(num_possible_nodes()) + 1; /* + '#' */ if (cpu == 1) - printk(KERN_INFO "x86: Booting SMP configuration:\n"); + printk(KERN_INFO "x86: Booting SMP configuration in %s:\n", + do_parallel_bringup ? "parallel" : "series"); if (system_state < SYSTEM_RUNNING) { if (node != current_node) { @@ -1325,9 +1327,12 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) { int ret; - ret = do_cpu_up(cpu, tidle); - if (ret) - goto out; + /* If parallel AP bringup isn't enabled, perform the first steps now. */ + if (!do_parallel_bringup) { + ret = do_cpu_up(cpu, tidle); + if (ret) + goto out; + } ret = do_wait_cpu_initialized(cpu); if (ret) @@ -1347,6 +1352,12 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) return ret; } +/* Bringup step one: Send INIT/SIPI to the target AP */ +static int native_cpu_kick(unsigned int cpu) +{ + return do_cpu_up(cpu, idle_thread_get(cpu)); +} + /** * arch_disable_smp_support() - disables SMP support for x86 at runtime */ @@ -1515,6 +1526,8 @@ static bool prepare_parallel_bringup(void) smpboot_control = STARTUP_APICID_CPUID_01; } + cpuhp_setup_state_nocalls(CPUHP_BP_PARALLEL_DYN, "x86/cpu:kick", + native_cpu_kick, NULL); return true; } From patchwork Tue Mar 21 19:40:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 73026 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp1975399wrt; Tue, 21 Mar 2023 12:46:50 -0700 (PDT) X-Google-Smtp-Source: AK7set+Jsv61bF9szlojgqn5UUd7BtUIswPHQPdTKkx3US6Of7FRyIjmHb8BTBqPriquzcoaSDts X-Received: by 2002:a17:902:ea0f:b0:19e:b2ed:6fe1 with SMTP id s15-20020a170902ea0f00b0019eb2ed6fe1mr357664plg.11.1679428010271; Tue, 21 Mar 2023 12:46:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679428010; cv=none; d=google.com; s=arc-20160816; b=F/CJxizOM8XBnPpJh0aI17xnNBouIAdccqU/Y0Piwfzj7LIDNx1sGDzIflnH4HHr0w ba6D2+TGNYPtoybSuba8ZdQbeSuyJF+4pNi6yfUkLGUAkkIPxSH26NKn8C3CUS/1W59W 9UHbTXxV1L+nipim0eCqaotMi+a2uBPcVNF8fWD8f8Q7y3xs+UoE1DmORZys97/+B+QI dWz3/IW+4qL/6+CBHRSXPvzxZ9cTODutDjQq7CNsda3uneR8suqhawoU7oMK51bRFGxx /RFXYDUmVrqsj0VeX2BCu5zzDv9lUxgwvhdnysBSPFyqfm4mhjTKrAUJQbFZwFZAvzl4 5F3Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=z39vbfAMOm3rZ8q2wnuNPhf0GbJ+pogxjBguM3zqrEM=; b=WEw2GUXFaThckWJNTikbKJKGFeGTyCSUMLl9sVy/YCdXIddUfuqD5UM7dC10tiosoG j4dvN1Fv3ryj4itIbvn6pXzQAAAtl0Z+xPVhwBj7EB4RawPKluZlCW7lvcYC8rBsg9fC wzYTZxw7ikpGu4fpHVwMOShUxmfA6mwGBT0UTVAWr2wfJFOgHJ80l0Wr76cTZH2Fys1V MkbXg1T6SaivMc7lj+3qZBSbLOkF9i7wSbKN9rDHVKBbgfWFNXA4/qtSZu3vSqWk0au5 2QDhuxjkc/2qt44M06jkL8Q2BPhpY8MzJsm0UyTgjIQ0iM81zVwKfSBEy6WwHaZCIOzq grHQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=JLZtWFXP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id j7-20020a170903024700b001948d9c2ca8si14929966plh.179.2023.03.21.12.46.35; Tue, 21 Mar 2023 12:46:50 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=JLZtWFXP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230429AbjCUTls (ORCPT + 99 others); Tue, 21 Mar 2023 15:41:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34098 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230346AbjCUTll (ORCPT ); Tue, 21 Mar 2023 15:41:41 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7401AA5D7 for ; Tue, 21 Mar 2023 12:41:05 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id y14so14884237wrq.4 for ; Tue, 21 Mar 2023 12:41:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1679427641; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=z39vbfAMOm3rZ8q2wnuNPhf0GbJ+pogxjBguM3zqrEM=; b=JLZtWFXPimB0vIW7FOULDvjYcRProdVX8b0R7w/Wq1bSQpTKeqzOM/X6IPNVTmo2ky 0kf1tajLwDUinlJpxGjgsPIITQUG2Q1BJ6Qscg7j3Ba0S7M6ECnFBqUGt//fml7wenD4 2pwRG/OAOeK7u63IqHKTIihpkd9/YgdW4xA5Zr81r5eKibdqD98dTLXvAZ1mhd03vM7U nMEK0ntmbghvfKvjOIVW9mS2sE2B9mHK/36kyKajolkWSlZJ+DtNUMuKITTwvzSBGlRe 6c4lR0Kg28AXIJijW087IHQByouvVU8Ig0EtZfjJZhE/cFIf9jysb6u92H/kKdPF1rh2 X/Ww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679427641; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=z39vbfAMOm3rZ8q2wnuNPhf0GbJ+pogxjBguM3zqrEM=; b=JSB9Toz9bNAAFdQ4tABTcKEsis0UMx78tSSX5WktHNxyHAeQ9AwJTBWW5KNQRwKyZA N0tLEztORbcHcmXXHHMzL4CCGCkEIfAmHS7GwsnNv6WoKpKML6AYFwmlwGWKj/UW5jsm 0C9lTar0ytINd2A8OenXON6N5QJgx/ngr7vPlcA4yeeVsXHi9pVnI533+ZhNtBFf1wpS 2pmdIyhIPB7b20eYddjR1zzeM1ih+CIEFU8KogNphElEayJvssrnmRawFLvBaSiVwfA+ d6tPu9GUs4T9uU8ikBzO6UY/6wrCHkWZd5pdHPZjkn9WYY6Vjzr0DOJD9iwAr5SYIn1T ymcA== X-Gm-Message-State: AO0yUKXYLG5iR2jLTHl7H/Oy5yLOg64V1JwI5Ee8Ys43YCvu18gOfqQO +09w9Qp4tk1AFmJHOiM1x68QqA== X-Received: by 2002:a5d:61ce:0:b0:2c7:fc61:12d4 with SMTP id q14-20020a5d61ce000000b002c7fc6112d4mr3492964wrv.47.1679427640924; Tue, 21 Mar 2023 12:40:40 -0700 (PDT) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:7a8a:d679:ba4e:61cf]) by smtp.gmail.com with ESMTPSA id a3-20020adffb83000000b002c561805a4csm12005026wrr.45.2023.03.21.12.40.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Mar 2023 12:40:40 -0700 (PDT) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, gpiccoli@igalia.com, David Woodhouse , Usama Arif Subject: [PATCH v16 7/8] x86/smpboot: Serialize topology updates for secondary bringup Date: Tue, 21 Mar 2023 19:40:07 +0000 Message-Id: <20230321194008.785922-8-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230321194008.785922-1-usama.arif@bytedance.com> References: <20230321194008.785922-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761007905446155968?= X-GMAIL-MSGID: =?utf-8?q?1761007905446155968?= From: David Woodhouse The toplogy update is performed by the AP via smp_callin() after the BSP has called do_wait_cpu_initialized(), setting the AP's bit in cpu_callout_mask to allow it to proceed. In preparation to enable further parallelism of AP bringup, add locking to serialize the update even if multiple APs are (in future) permitted to proceed through the next stages of bringup in parallel. Without such ordering (and with that future extra parallelism), confusion ensues: [ 1.360149] x86: Booting SMP configuration: [ 1.360221] .... node #0, CPUs: #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15 #16 #17 #18 #19 #20 #21 #22 #23 [ 1.366225] .... node #1, CPUs: #24 #25 #26 #27 #28 #29 #30 #31 #32 #33 #34 #35 #36 #37 #38 #39 #40 #41 #42 #43 #44 #45 #46 #47 [ 1.370219] .... node #0, CPUs: #48 #49 #50 #51 #52 #53 #54 #55 #56 #57 #58 #59 #60 #61 #62 #63 #64 #65 #66 #67 #68 #69 #70 #71 [ 1.378226] .... node #1, CPUs: #72 #73 #74 #75 #76 #77 #78 #79 #80 #81 #82 #83 #84 #85 #86 #87 #88 #89 #90 #91 #92 #93 #94 #95 [ 1.382037] Brought 96 CPUs to x86/cpu:kick in 72232606 cycles [ 0.104104] smpboot: CPU 26 Converting physical 0 to logical die 1 [ 0.104104] smpboot: CPU 27 Converting physical 1 to logical package 2 [ 0.104104] smpboot: CPU 24 Converting physical 1 to logical package 3 [ 0.104104] smpboot: CPU 27 Converting physical 0 to logical die 2 [ 0.104104] smpboot: CPU 25 Converting physical 1 to logical package 4 [ 1.385609] Brought 96 CPUs to x86/cpu:wait-init in 9269218 cycles [ 1.395285] Brought CPUs online in 28930764 cycles [ 1.395469] smp: Brought up 2 nodes, 96 CPUs [ 1.395689] smpboot: Max logical packages: 2 [ 1.396222] smpboot: Total of 96 processors activated (576000.00 BogoMIPS) Signed-off-by: David Woodhouse Signed-off-by: Usama Arif Tested-by: Paul E. McKenney Tested-by: Kim Phillips Tested-by: Oleksandr Natalenko Tested-by: Guilherme G. Piccoli --- arch/x86/include/asm/smp.h | 4 +- arch/x86/include/asm/topology.h | 2 - arch/x86/kernel/cpu/common.c | 6 +-- arch/x86/kernel/smpboot.c | 73 ++++++++++++++++++++------------- arch/x86/xen/smp_pv.c | 4 +- 5 files changed, 48 insertions(+), 41 deletions(-) diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h index 1cf4f1e57570..defe76ee9e64 100644 --- a/arch/x86/include/asm/smp.h +++ b/arch/x86/include/asm/smp.h @@ -48,8 +48,6 @@ struct smp_ops { }; /* Globals due to paravirt */ -extern void set_cpu_sibling_map(int cpu); - #ifdef CONFIG_SMP extern struct smp_ops smp_ops; @@ -137,7 +135,7 @@ void native_send_call_func_single_ipi(int cpu); void x86_idle_thread_init(unsigned int cpu, struct task_struct *idle); void smp_store_boot_cpu_info(void); -void smp_store_cpu_info(int id); +void smp_store_cpu_info(int id, bool force_single_core); asmlinkage __visible void smp_reboot_interrupt(void); __visible void smp_reschedule_interrupt(struct pt_regs *regs); diff --git a/arch/x86/include/asm/topology.h b/arch/x86/include/asm/topology.h index 458c891a8273..4bccbd949a99 100644 --- a/arch/x86/include/asm/topology.h +++ b/arch/x86/include/asm/topology.h @@ -136,8 +136,6 @@ static inline int topology_max_smt_threads(void) return __max_smt_threads; } -int topology_update_package_map(unsigned int apicid, unsigned int cpu); -int topology_update_die_map(unsigned int dieid, unsigned int cpu); int topology_phys_to_logical_pkg(unsigned int pkg); int topology_phys_to_logical_die(unsigned int die, unsigned int cpu); bool topology_is_primary_thread(unsigned int cpu); diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 8cd4126d8253..80a688295ffa 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -1776,7 +1776,7 @@ static void generic_identify(struct cpuinfo_x86 *c) * Validate that ACPI/mptables have the same information about the * effective APIC id and update the package map. */ -static void validate_apic_and_package_id(struct cpuinfo_x86 *c) +static void validate_apic_id(struct cpuinfo_x86 *c) { #ifdef CONFIG_SMP unsigned int apicid, cpu = smp_processor_id(); @@ -1787,8 +1787,6 @@ static void validate_apic_and_package_id(struct cpuinfo_x86 *c) pr_err(FW_BUG "CPU%u: APIC id mismatch. Firmware: %x APIC: %x\n", cpu, apicid, c->initial_apicid); } - BUG_ON(topology_update_package_map(c->phys_proc_id, cpu)); - BUG_ON(topology_update_die_map(c->cpu_die_id, cpu)); #else c->logical_proc_id = 0; #endif @@ -1979,7 +1977,7 @@ void identify_secondary_cpu(struct cpuinfo_x86 *c) #ifdef CONFIG_X86_32 enable_sep_cpu(); #endif - validate_apic_and_package_id(c); + validate_apic_id(c); x86_spec_ctrl_setup_ap(); update_srbds_msr(); diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index bb818e6d8701..0dba5c247be0 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -180,16 +180,12 @@ static void smp_callin(void) apic_ap_setup(); /* - * Save our processor parameters. Note: this information - * is needed for clock calibration. - */ - smp_store_cpu_info(cpuid); - - /* + * Save our processor parameters and update topology. + * Note: this information is needed for clock calibration. * The topology information must be up to date before * calibrate_delay() and notify_cpu_starting(). */ - set_cpu_sibling_map(raw_smp_processor_id()); + smp_store_cpu_info(cpuid, false); ap_init_aperfmperf(); @@ -243,6 +239,12 @@ static void notrace start_secondary(void *unused) * its bit in cpu_callout_mask to release it. */ cpu_init_secondary(); + + /* + * Even though notify_cpu_starting() will do this, it does so too late + * as the AP may already have triggered lockdep splats by then. See + * commit 29368e093 ("x86/smpboot: Move rcu_cpu_starting() earlier"). + */ rcu_cpu_starting(raw_smp_processor_id()); x86_cpuinit.early_percpu_clock_init(); @@ -351,7 +353,7 @@ EXPORT_SYMBOL(topology_phys_to_logical_die); * @pkg: The physical package id as retrieved via CPUID * @cpu: The cpu for which this is updated */ -int topology_update_package_map(unsigned int pkg, unsigned int cpu) +static int topology_update_package_map(unsigned int pkg, unsigned int cpu) { int new; @@ -374,7 +376,7 @@ int topology_update_package_map(unsigned int pkg, unsigned int cpu) * @die: The die id as retrieved via CPUID * @cpu: The cpu for which this is updated */ -int topology_update_die_map(unsigned int die, unsigned int cpu) +static int topology_update_die_map(unsigned int die, unsigned int cpu) { int new; @@ -405,25 +407,7 @@ void __init smp_store_boot_cpu_info(void) c->initialized = true; } -/* - * The bootstrap kernel entry code has set these up. Save them for - * a given CPU - */ -void smp_store_cpu_info(int id) -{ - struct cpuinfo_x86 *c = &cpu_data(id); - - /* Copy boot_cpu_data only on the first bringup */ - if (!c->initialized) - *c = boot_cpu_data; - c->cpu_index = id; - /* - * During boot time, CPU0 has this setup already. Save the info when - * bringing up AP or offlined CPU0. - */ - identify_secondary_cpu(c); - c->initialized = true; -} +static arch_spinlock_t topology_lock = __ARCH_SPIN_LOCK_UNLOCKED; static bool topology_same_node(struct cpuinfo_x86 *c, struct cpuinfo_x86 *o) @@ -629,7 +613,7 @@ static struct sched_domain_topology_level x86_topology[] = { */ static bool x86_has_numa_in_package; -void set_cpu_sibling_map(int cpu) +static void set_cpu_sibling_map(int cpu) { bool has_smt = smp_num_siblings > 1; bool has_mp = has_smt || boot_cpu_data.x86_max_cores > 1; @@ -708,6 +692,37 @@ void set_cpu_sibling_map(int cpu) } } +/* + * The bootstrap kernel entry code has set these up. Save them for + * a given CPU + */ +void smp_store_cpu_info(int id, bool force_single_core) +{ + struct cpuinfo_x86 *c = &cpu_data(id); + + /* Copy boot_cpu_data only on the first bringup */ + if (!c->initialized) + *c = boot_cpu_data; + c->cpu_index = id; + /* + * During boot time, CPU0 has this setup already. Save the info when + * bringing up AP or offlined CPU0. + */ + identify_secondary_cpu(c); + + arch_spin_lock(&topology_lock); + BUG_ON(topology_update_package_map(c->phys_proc_id, id)); + BUG_ON(topology_update_die_map(c->cpu_die_id, id)); + c->initialized = true; + + /* For Xen PV */ + if (force_single_core) + c->x86_max_cores = 1; + + set_cpu_sibling_map(id); + arch_spin_unlock(&topology_lock); +} + /* maps the cpu to the sched domain representing multi-core */ const struct cpumask *cpu_coregroup_mask(int cpu) { diff --git a/arch/x86/xen/smp_pv.c b/arch/x86/xen/smp_pv.c index a9cf8c8fa074..ea6c60514092 100644 --- a/arch/x86/xen/smp_pv.c +++ b/arch/x86/xen/smp_pv.c @@ -71,9 +71,7 @@ static void cpu_bringup(void) xen_enable_syscall(); } cpu = smp_processor_id(); - smp_store_cpu_info(cpu); - cpu_data(cpu).x86_max_cores = 1; - set_cpu_sibling_map(cpu); + smp_store_cpu_info(cpu, true); speculative_store_bypass_ht_init(); From patchwork Tue Mar 21 19:40:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 73028 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp1975730wrt; Tue, 21 Mar 2023 12:47:42 -0700 (PDT) X-Google-Smtp-Source: AK7set9BuxJdCi4AKKlFIvbEJwnpg+vxcTETCu1GD7OcW86ae/fGlHOUD/kzoSHmv14lSU0F19yx X-Received: by 2002:a05:6a20:7b22:b0:db:1b41:6fe with SMTP id s34-20020a056a207b2200b000db1b4106femr930539pzh.6.1679428061698; Tue, 21 Mar 2023 12:47:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679428061; cv=none; d=google.com; s=arc-20160816; b=VJJriY4TvwOXcOBle5uKP64W7J2G5ee48xmmi9N5wU0/9uCYGK01GYyGg1buZXiD++ 6lQwpQSGQQxvDoBSeeXYJPXmUnhlMxlLjn+hjv8T0fhOvRb5hqcO/s7BY7Sssa6CV6fH gN2XGC5fPKnTfcHEil0qCdyI6I0ovc3oI0zEnZlTly/Hlq5kovRq2wdKOKA5mkRnoRpp c3bMTtRXNQrgmovUbeeXNA94THhDOAjSii58cJAbdlvx1AMbW7cJbH+g3J1WPcsEh9d+ Rzgi+xTXZ43zi45mBLrvLi1eIzv4w0q28NQee1K/LPj+RzYOs5ipzuWwfKzxmpRUZFgz xBYA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=l3gQK6+MXZzclW4d0GGLnTn3p9hIu0ffrS0A+Sd8IEM=; b=nM15SyW+kR9iI5lQoCZBQK6D96KIrkYXVHajReoMSCNgPagKkEYkaBStAjmHueKzzS sGRS3c+aw8N11PDZxFf2pMfSHeiiLY9ImzWDDELAzET6SjjrwgKlOWgecI3vOsV+4Wzt cbfJdEJn6v5T7GzUsZLHkRjYX8UT7FAdu4CPi5qUdePHrQzROxUtEwIHLglrCe/1vVnr B/Qy4CFBXC290zIb/LeHyUVASn8xZbtaguWo79u/wn5eecV/DTRbYfJ8ewILJQT/YZTd AwzCh7VleIRsUYnZjghv1MCWo8u7xQaPM0Uo8WkxV7RnHrPPJGSc319mp1uKRwkTK5un yNfQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=iDIMZpNV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 7-20020a630307000000b004df43225600si14153839pgd.709.2023.03.21.12.47.29; Tue, 21 Mar 2023 12:47:41 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=iDIMZpNV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230473AbjCUTmR (ORCPT + 99 others); Tue, 21 Mar 2023 15:42:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34006 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229890AbjCUTmF (ORCPT ); Tue, 21 Mar 2023 15:42:05 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8D8A52FCE0 for ; Tue, 21 Mar 2023 12:41:06 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id r29so14855623wra.13 for ; Tue, 21 Mar 2023 12:41:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1679427642; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=l3gQK6+MXZzclW4d0GGLnTn3p9hIu0ffrS0A+Sd8IEM=; b=iDIMZpNVu/rJq2uyMdfotQHp+XfP7xkWQekYnCD7Mpayj6O5zMcoQHrk2DVTxASKAf WFIrUo72HOrWyBGizUQkIydoteGJDnDt6pVL8Q9Hmee1ZnkGR6PA+KRBQqgB0sDendod I9iUk3S02eV9sWO5GDS4kWCFiwfLf3JN3iO/MK8vIIq4ZcmVqIrUY/r/8ythxjfz2lLC X9cfoGa0pKOBUlBjmq8KK6B8nZFkKLqUbvybSbx7J7P0xzxdDvNmmhN0KDqab6tBmyof a8InYJmLone6yIGk1ZEHbHK2HsAJDO98363Iyfc6eMehMiiptxIk4tTU15ym77+l1ipJ T9vQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679427642; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=l3gQK6+MXZzclW4d0GGLnTn3p9hIu0ffrS0A+Sd8IEM=; b=ZU19xL4XOULQK3AZi6sqxS03TNkJEv2kVrX4Zi1fluaW2gSoY0rtaja3GoAKL/+nXw MSXrguIleGBhyNhZ643wp6qL5sd4yewmp7t3et99Sk67o54+VKR+EJtOjdv3Oi5p2ECc N0Lh5Qrcg2N7RXrkb/2zm1l12WBN9zlGOV3DgmyjtwToP1BFf+OF+R6GH+JLKnLoZBvo Gur4zfIAolF847RUrnuw0VqTRvJDs1M5XVtb9vX7h39zrcTRz6gKj0NnxZyjEdTiQhok CLa6TyftR3Rv98qLEh6cCc+3lkjPxuuC7ghhI2dyo3pudUN47lO1FZdhY0qCwgyIQ0p9 XAGw== X-Gm-Message-State: AO0yUKWiNaApB4d/XxoRkFRh795v2GYO/HBv0GCZjFwq9r1dxymzfG4d h5iomuyF751XPzJ5wV9HD4zB9w== X-Received: by 2002:adf:db86:0:b0:2d5:5610:e7b4 with SMTP id u6-20020adfdb86000000b002d55610e7b4mr3135776wri.64.1679427642103; Tue, 21 Mar 2023 12:40:42 -0700 (PDT) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:7a8a:d679:ba4e:61cf]) by smtp.gmail.com with ESMTPSA id a3-20020adffb83000000b002c561805a4csm12005026wrr.45.2023.03.21.12.40.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Mar 2023 12:40:41 -0700 (PDT) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, gpiccoli@igalia.com, David Woodhouse , Sabin Rapan , Usama Arif Subject: [PATCH v16 8/8] x86/smpboot: Allow parallel bringup for SEV-ES Date: Tue, 21 Mar 2023 19:40:08 +0000 Message-Id: <20230321194008.785922-9-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230321194008.785922-1-usama.arif@bytedance.com> References: <20230321194008.785922-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761007958837078885?= X-GMAIL-MSGID: =?utf-8?q?1761007958837078885?= From: David Woodhouse Enable parallel bringup for SEV-ES guests. The APs can't actually execute the CPUID instruction directly during early startup, but they can make the GHCB call directly instead, just as the VC trap handler would do. Thanks to Sabin for talking me through the way this works. Suggested-by: Sabin Rapan Signed-off-by: David Woodhouse Signed-off-by: Usama Arif Reviewed-by: Tom Lendacky --- arch/x86/coco/core.c | 5 ++++ arch/x86/include/asm/coco.h | 1 + arch/x86/include/asm/sev-common.h | 3 +++ arch/x86/include/asm/smp.h | 5 +++- arch/x86/kernel/head_64.S | 30 ++++++++++++++++++++++++ arch/x86/kernel/smpboot.c | 39 ++++++++++++++++++++++++++----- 6 files changed, 76 insertions(+), 7 deletions(-) diff --git a/arch/x86/coco/core.c b/arch/x86/coco/core.c index 49b44f881484..0bab38efb15a 100644 --- a/arch/x86/coco/core.c +++ b/arch/x86/coco/core.c @@ -129,6 +129,11 @@ u64 cc_mkdec(u64 val) } EXPORT_SYMBOL_GPL(cc_mkdec); +enum cc_vendor cc_get_vendor(void) +{ + return vendor; +} + __init void cc_set_vendor(enum cc_vendor v) { vendor = v; diff --git a/arch/x86/include/asm/coco.h b/arch/x86/include/asm/coco.h index 3d98c3a60d34..0428d9712c96 100644 --- a/arch/x86/include/asm/coco.h +++ b/arch/x86/include/asm/coco.h @@ -12,6 +12,7 @@ enum cc_vendor { }; void cc_set_vendor(enum cc_vendor v); +enum cc_vendor cc_get_vendor(void); void cc_set_mask(u64 mask); #ifdef CONFIG_ARCH_HAS_CC_PLATFORM diff --git a/arch/x86/include/asm/sev-common.h b/arch/x86/include/asm/sev-common.h index b63be696b776..0abf8a39cee1 100644 --- a/arch/x86/include/asm/sev-common.h +++ b/arch/x86/include/asm/sev-common.h @@ -70,6 +70,7 @@ /* GHCBData[63:12] */ \ (((u64)(v) & GENMASK_ULL(63, 12)) >> 12) +#ifndef __ASSEMBLY__ /* * SNP Page State Change Operation * @@ -161,6 +162,8 @@ struct snp_psc_desc { #define GHCB_RESP_CODE(v) ((v) & GHCB_MSR_INFO_MASK) +#endif /* __ASSEMBLY__ */ + /* * Error codes related to GHCB input that can be communicated back to the guest * by setting the lower 32-bits of the GHCB SW_EXITINFO1 field to 2. diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h index defe76ee9e64..1584f04a7007 100644 --- a/arch/x86/include/asm/smp.h +++ b/arch/x86/include/asm/smp.h @@ -204,7 +204,10 @@ extern unsigned int smpboot_control; /* Control bits for startup_64 */ #define STARTUP_APICID_CPUID_0B 0x80000000 #define STARTUP_APICID_CPUID_01 0x40000000 +#define STARTUP_APICID_SEV_ES 0x20000000 -#define STARTUP_PARALLEL_MASK (STARTUP_APICID_CPUID_01 | STARTUP_APICID_CPUID_0B) +#define STARTUP_PARALLEL_MASK (STARTUP_APICID_CPUID_01 | \ + STARTUP_APICID_CPUID_0B | \ + STARTUP_APICID_SEV_ES) #endif /* _ASM_X86_SMP_H */ diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index ff3a5f008d8a..9c38849fcac8 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -26,6 +26,7 @@ #include #include #include +#include /* * We are not able to switch in one step to the final KERNEL ADDRESS SPACE @@ -242,6 +243,7 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) * * Bit 31 STARTUP_APICID_CPUID_0B flag (use CPUID 0x0b) * Bit 30 STARTUP_APICID_CPUID_01 flag (use CPUID 0x01) + * Bit 29 STARTUP_APICID_SEV_ES flag (CPUID 0x0b via GHCB MSR) * Bit 0-24 CPU# if STARTUP_APICID_CPUID_xx flags are not set */ movl smpboot_control(%rip), %ecx @@ -249,6 +251,10 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) jnz .Luse_cpuid_0b testl $STARTUP_APICID_CPUID_01, %ecx jnz .Luse_cpuid_01 +#ifdef CONFIG_AMD_MEM_ENCRYPT + testl $STARTUP_APICID_SEV_ES, %ecx + jnz .Luse_sev_cpuid_0b +#endif andl $0x0FFFFFFF, %ecx jmp .Lsetup_cpu @@ -259,6 +265,30 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) shr $24, %edx jmp .Lsetup_AP +#ifdef CONFIG_AMD_MEM_ENCRYPT +.Luse_sev_cpuid_0b: + /* Set the GHCB MSR to request CPUID 0xB_EDX */ + movl $MSR_AMD64_SEV_ES_GHCB, %ecx + movl $(GHCB_CPUID_REQ_EDX << 30) | GHCB_MSR_CPUID_REQ, %eax + movl $0x0B, %edx + wrmsr + + /* Perform GHCB MSR protocol */ + rep; vmmcall /* vmgexit */ + + /* + * Get the result. After the RDMSR: + * EAX should be 0xc0000005 + * EDX should have the CPUID register value and since EDX + * is the target register, no need to move the result. + */ + rdmsr + andl $GHCB_MSR_INFO_MASK, %eax + cmpl $GHCB_MSR_CPUID_RESP, %eax + jne 1f + jmp .Lsetup_AP +#endif + .Luse_cpuid_0b: mov $0x0B, %eax xorl %ecx, %ecx diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 0dba5c247be0..ef37356ab695 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -85,6 +85,7 @@ #include #include #include +#include /* representing HT siblings of each logical CPU */ DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_sibling_map); @@ -1513,15 +1514,36 @@ void __init smp_prepare_cpus_common(void) * We can do 64-bit AP bringup in parallel if the CPU reports its APIC * ID in CPUID (either leaf 0x0B if we need the full APIC ID in X2APIC * mode, or leaf 0x01 if 8 bits are sufficient). Otherwise it's too - * hard. And not for SEV-ES guests because they can't use CPUID that - * early. + * hard. */ static bool prepare_parallel_bringup(void) { - if (IS_ENABLED(CONFIG_X86_32) || cc_platform_has(CC_ATTR_GUEST_STATE_ENCRYPT)) + bool has_sev_es = false; + + if (IS_ENABLED(CONFIG_X86_32)) return false; - if (x2apic_mode) { + /* + * Encrypted guests other than SEV-ES (in the future) will need to + * implement an early way of finding the APIC ID, since they will + * presumably block direct CPUID too. Be kind to our future selves + * by warning here instead of just letting them break. Parallel + * startup doesn't have to be in the first round of enabling patches + * for any such technology. + */ + if (cc_platform_has(CC_ATTR_GUEST_STATE_ENCRYPT)) { + switch (cc_get_vendor()) { + case CC_VENDOR_AMD: + has_sev_es = true; + break; + + default: + pr_info("Disabling parallel bringup due to guest state encryption\n"); + return false; + } + } + + if (x2apic_mode || has_sev_es) { if (boot_cpu_data.cpuid_level < 0x0b) return false; @@ -1530,8 +1552,13 @@ static bool prepare_parallel_bringup(void) return false; } - pr_debug("Using CPUID 0xb for parallel CPU startup\n"); - smpboot_control = STARTUP_APICID_CPUID_0B; + if (has_sev_es) { + pr_debug("Using SEV-ES CPUID 0xb for parallel CPU startup\n"); + smpboot_control = STARTUP_APICID_SEV_ES; + } else { + pr_debug("Using CPUID 0xb for parallel CPU startup\n"); + smpboot_control = STARTUP_APICID_CPUID_0B; + } } else { /* Without X2APIC, what's in CPUID 0x01 should suffice. */ if (boot_cpu_data.cpuid_level < 0x01)