From patchwork Mon Feb 20 03:28:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Zhang, Rui" X-Patchwork-Id: 5621 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1116366wrn; Sun, 19 Feb 2023 19:57:10 -0800 (PST) X-Google-Smtp-Source: AK7set/Ynhjrol7GU31RNt3JXTl3y3mqVY/zdVcCPNblSq08Cenv+gQianMXuIF9ihSbAAK+cqmK X-Received: by 2002:aa7:96e9:0:b0:5a8:cbcc:4b58 with SMTP id i9-20020aa796e9000000b005a8cbcc4b58mr2799168pfq.12.1676865429798; Sun, 19 Feb 2023 19:57:09 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676865429; cv=none; d=google.com; s=arc-20160816; b=SRoLSTE1o7c0iZRiB7m45VnblHg1n9IoKGrOzdRKaVYv85+ob0XPUemArVqldfoo4A jBcXle7PSAztNIge+Zl9tl9xnkoIFS+AN/lXIMbj0Y6qUdJKCrMeDwtKkOIiIbYAPKXf AiO6DHn4CJgtmHbXIVkQMU2kygFUP88hms4CvVB0YfNPU+KD5aADTVokB9a9dHk3FoTn WDViBmPW7NBU0RfzixAHPvNAvgwnbeWaExakPt/mPc+LwgXeRP53Yt9mn+QRm8OexGr7 D2zGfw4nG6G8TdMxcv4+KBoXEW340jtwEijiMUYyNMVQaeoDFcOCOckyzlZqolT+Rr5m pOWQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=EzLVtBwm0JdbV4d9A0wywbtKldMSqj8scWRp08No2B0=; b=fJpIm+P65b0MIBLob/oBQxAdRLc9ZulzAxWG6R6xtwYfr6rWimvLPM6fTzYwHqFqxS Lf/f/F+1HQA+/23JwdCNYLGS+w2eXV5Xfx5nfLqUW05TWQ7J9NHZT3i5oxuKvVuPXRV2 lCWurNW2dMPGpp7XhlX4ajTGDmvnzH5/R0HsTUhl9juy8ioV1Nu+MgLB+O2w9hHgeuyL MYq9sfevXjkCM9joSkVFV/ZG8P/sWB3oLt4ixicRJHpWazgugtAkVvqGppQDMLMY1W5/ FKne26DxJunh4Y2xGhCKWCqmx5vmrZsTKMgDprXH+98zDdV8TZM+zm9qPZtP6pnvMiuI A1XQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=ZJzkVI7G; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b75-20020a621b4e000000b005a8bebbdb3esi1895720pfb.105.2023.02.19.19.56.56; Sun, 19 Feb 2023 19:57:09 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=ZJzkVI7G; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229736AbjBTD3O (ORCPT + 99 others); Sun, 19 Feb 2023 22:29:14 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35286 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229642AbjBTD3M (ORCPT ); Sun, 19 Feb 2023 22:29:12 -0500 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EE4A1C66B for ; Sun, 19 Feb 2023 19:29:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1676863750; x=1708399750; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=UbFOTIk6kgGuUkd1NfbpBJotjTmWKsF4mSJ46RkdXNI=; b=ZJzkVI7GphW5CWel6MK8fobdh0nRjVYfeehQXspmDPEhIaRlHezMjdjz EcW9EewJC8lNHaCzAy4zuJaYwB366ntMIDzwHc0y9Chf3klD2YK57oF16 j0JH6vhADV0KJhxtew16rgycg6k8a9MdevAMz3FHIYcX59RSJ+CHcFs7S qSPHatT69l03u4CWExJoMjGzyY+p6QlqtngQ0TEDfPPYJ4Mfsi7WzE2ho zoYU9jnvs2MVi0/7Tq5eWanm95ydb8WPQnKue7XaX+j+kVzYjOZMRUt7O XuD0SbwASzPbsUQpNeYX+VWkRcl0C4Sgn3R2ID4LfjOayXS5l+YXrhZ95 Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10626"; a="418537672" X-IronPort-AV: E=Sophos;i="5.97,311,1669104000"; d="scan'208";a="418537672" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Feb 2023 19:29:10 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10626"; a="813986499" X-IronPort-AV: E=Sophos;i="5.97,311,1669104000"; d="scan'208";a="813986499" Received: from cyi-mobl.ccr.corp.intel.com (HELO rzhang1-DESK.intel.com) ([10.255.29.242]) by fmsmga001-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Feb 2023 19:29:07 -0800 From: Zhang Rui To: tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com Cc: x86@kernel.org, linux-kernel@vger.kernel.org, zhang.jia@linux.alibaba.com, len.brown@intel.com, rui.zhang@intel.com Subject: [RFC PATCH V2 0/1] x86: cpu topology fix and question on x86_max_cores Date: Mon, 20 Feb 2023 11:28:55 +0800 Message-Id: <20230220032856.661884-1-rui.zhang@intel.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758320844816505011?= X-GMAIL-MSGID: =?utf-8?q?1758320844816505011?= Hi, All, There are two kernel issues observed on Intel Hybrid platform named MeteorLake. And these two issues altogether bring broken CPU topology. This patch series aims to 1. provide a solution for the first issue, as an urgent fix and -stable candidate, so that this problem is not exposed to end users. 2. get feedback on how to fix the second issue. Any comments on this are really appreciated. Problem details on MeteorLake ----------------------------- On Intel Hybrid platforms like AlderLake-P/S, both smp_num_siblings and cpuinfo_x86.x86_max_cores are broken like below [ 0.201005] detect_extended_topology: CPU APICID 0x0, smp_num_siblings 2, x86_max_cores 10 [ 0.201117] start_kernel->check_bugs->cpu_smt_check_topology: smp_num_siblings 2 ... [ 0.010146] detect_extended_topology: CPU APICID 0x8, smp_num_siblings 2, x86_max_cores 10 ... [ 0.010146] detect_extended_topology: CPU APICID 0x39, smp_num_siblings 2, x86_max_cores 10 [ 0.010146] detect_extended_topology: CPU APICID 0x48, smp_num_siblings 1, x86_max_cores 20 ... [ 0.010146] detect_extended_topology: CPU APICID 0x4e, smp_num_siblings 1, x86_max_cores 20 [ 2.583800] sched_set_itmt_core_prio: smp_num_siblings 1 This is because the SMT siblings value returned by CPUID.1F SMT level EBX differs among CPUs. It returns 2 for Pcore CPUs which have HT sibling and returns 1 for Ecore CPUs which do not have SMT sibling. This brings several potential issues: 1. some kernel configuration like cpu_smt_control, as set in start_kernel()->check_bugs()->cpu_smt_check_topology(), depends on smp_num_siblings set by boot cpu. It is pure luck that all the current hybrid platforms use Pcore as cpu0 and hide this problem. 2. some per CPU data like cpuinfo_x86.x86_max_cores that depends on smp_num_siblings becomes inconsistent and bogus. 3. the final smp_num_siblings value after boot depends on the last CPU enumerated, which could either be Pcore or Ecore CPU. Previously, there is no functional issue observed on AlderLake-P/S. However, on MeteorLake, this becomes worse. a). smp_num_siblings varies like AlderLake and it is set to 1 for Ecore. b). x86_max_cores is totally broken and it is set to 1 for the boot cpu. Altogether, these two issues make the system being treated as an UP system in set_cpu_sibling_map() when probing Ecore CPUs, and the Ecore CPUs are not updated in any cpu sibling maps erroneously. Below shows part of the CPU topology information before and after the fix, for both Pcore and Ecore CPU (cpu0 is Pcore, cpu 12 is Ecore). ... -/sys/devices/system/cpu/cpu0/topology/package_cpus:000fff -/sys/devices/system/cpu/cpu0/topology/package_cpus_list:0-11 +/sys/devices/system/cpu/cpu0/topology/package_cpus:3fffff +/sys/devices/system/cpu/cpu0/topology/package_cpus_list:0-21 ... -/sys/devices/system/cpu/cpu12/topology/package_cpus:001000 -/sys/devices/system/cpu/cpu12/topology/package_cpus_list:12 +/sys/devices/system/cpu/cpu12/topology/package_cpus:3fffff +/sys/devices/system/cpu/cpu12/topology/package_cpus_list:0-21 And this also breaks userspace tools like lscpu -Core(s) per socket: 1 -Socket(s): 11 +Core(s) per socket: 16 +Socket(s): 1 Solution for fix smp_num_sibling -------------------------------- Patch 1/1 ensures that smp_num_siblings represents the system-wide maximum number of siblings by always increasing its value. Never allow it to decrease. It is sufficient to make the problem go away. However, there is a pontenial problem left. That is, when boot CPU is an Ecore CPU, smp_num_sibling is set to 1 during BSP probe, kernel disables SMT support by setting cpu_smt_control to CPU_SMT_NOT_SUPPORTED in start_kernel()->check_bugs()->cpu_smt_check_topology(). So far, we don't have such platforms. Questions on how to fix cpuinfo_x86.x86_max_cores ------------------------------------------------- Fixing x86_max_cores is more complex. Current kernel uses below logic to get x86_max_cores x86_max_cores = cpus_in_a_package / smp_num_siblings But 1. There is a known bug in CPUID.1F handling code. Thus cpus_in_a_package can be bogus. To fix it, I will add CPUID.1F Module level support. 2. x86_max_cores is set and used in an inconsistent way in current kernel. In short, smp_num_siblings/x86_max_cores 2.1 represents the number of maximum *addressable* threads/cores in a core/package when retrieved via CPUID 1 and 4 on old platforms. CPUID.1 EBX 23:16 "Maximum number of addressable IDs for logical processors in this physical package". CPUID.4 EAX 31:26 "Maximum number of addressable IDs for processor cores in the physical package". 2.2 represents the number of maximum *possible* threads/cores in a core/package, when retrieved via CPUID.B/1F on non-Hybrid platforms. CPUID.B/1F EBX 15:0 "Number of logical processors at this level type. The number reflects configuration as shipped by Intel". For example, in calc_llc_size_per_core() do_div(llc_size, c->x86_max_cores); x86_max_cores is used as the max *possible* cores in a package. 2.3 is used in a conflict way on other vendors like AMD by checking the code. I need help on confirming the proper behavior for AMD. For example, in amd_get_topology(), c->x86_coreid_bits = get_count_order(c->x86_max_cores); x86_max_cores is used as the max *addressable* cores in a package. in get_nbc_for_node(), cores_per_node = (c->x86_max_cores * smp_num_siblings) / amd_get_nodes_per_socket(); x86_max_cores is used as the max *possible* cores in a package. 3. using x86_max_cores = cpus_in_a_package / smp_num_siblings to get the number of maximum *possible* cores in a package during boot cpu bringup is not applicable on platforms with asymmetric cores. Because, for a given number of threads, we don't know how many of the threads are the master thread or the only thread of a core, and how many of them are SMT siblings. For example, on a platform with 6 Pcore and 8 Ecore, there are 20 threads. But setting x86_max_cores to 10 is apparently wrong. Given the above situation, I have below question and any input is really appreciated. Is this inconsistency a problem or not? If we want to keep it consistent, it has to represent the max *addressable* cores in a package. Because max *possible* cores in a package is not available on Intel Hybrid platform. I have proposed a patch for this purpose, but gave up because I didn't address the above scenarios that uses x86_max_core differently. https://lore.kernel.org/all/20220922133800.12918-7-rui.zhang@intel.com/ Does this break the expectation on AMD platforms using CPUID.B? If yes, what should we do? BTW, this also fixes the potential issue that kernel runs as SMT disabled when cpu0 is a Ecore cpu. The downside is that, on a platform that does not have SMT, like AlderLake-N, which has Ecore CPUs only, kernel will still run as SMT-capable with this solution. Because SMT ID still takes 1 bit in APIC-ID and smp_num_siblings is set to 2. But given that kernel has already optimized for non-SMT case at runtime in places like scheduler, by checking sibling maps, so my current understanding is that the overhead won't be big. thanks, rui --- Changes in V2: - improve changelog to more focus on the smp_num_siblings issue.