From patchwork Wed Feb 28 19:36:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Luck, Tony" X-Patchwork-Id: 21107 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp3568251dyb; Wed, 28 Feb 2024 11:38:26 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCV5T7tb61dBiyBRruL5ip0N7jlymEOgumTMgRoQabX1ujZWWeFXdF5pejJUlcEbetPYQ9WlYAa7vjOBgtZQ9E7OulGRbA== X-Google-Smtp-Source: AGHT+IGtoG8fVjWGbffLz7KOE5haNQxVcmkZjUpRyQZoouGOGMT/SHjdVaZ6lvWejprDaPWM0Kz7 X-Received: by 2002:aa7:cfd1:0:b0:563:c54e:f1 with SMTP id r17-20020aa7cfd1000000b00563c54e00f1mr305406edy.2.1709149106582; Wed, 28 Feb 2024 11:38:26 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1709149106; cv=pass; d=google.com; s=arc-20160816; b=jfz8AwXqwcfSOe69Bv2Ksjl1iC7SGvjsre4+kZmWhqMg26gxN22pda52rD3KuCm7vj eAK/ATMSiGa/7kpwSRPbwiD6dit5knsaB9c78sTSzgaJcMVcb6vmH4eXnNpOd8whVf7E fUQ3uhlRi/L8R0qZqRC9KBj0omXR1Ap9rZa0XWvh5rJlpbbXsiGfP9tTO3JzxFfLE1Xi 562RSUKfQ5Y/ivo8ZCYjBLNmxyt7kPvNutsuc2t+VoexRwKIyrz9ArH2Za/4ulFEfKIC 8+yulbohgL/ZMTL9pC5IR7gAFfbPRoCyaYQFDmZ2D79YpCVLtIl59YK/8tDR6Rvh6OqV Ving== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=/sCoLoDm3ArfwoN4xS33VKUtT3oHWQN2/cjm09/l3uI=; fh=kDu4WSgSKQjfqS0+uR7gDD47kd+EMQAW47hbuuOMwvI=; b=oVytuSEo2PdkUoTLVIRSgXeJ2z0fOdeR0M0AONavgbxghJjaU68La6yhG56K/srNSo aivzB3147Todfp+z/PhFnBI0r2RSbuvWD7rRgveRSwBgZLwsEnG6Lhv/Ls0cGsh5ypFt A+EFYob9zDBomIBQQdREUzVi/pP6fM8i3IYniN8146hhmzwGxxfEFcEtn/sRF+lgk8kH 4RtBakl1u9G82j3OsBBRmOsKYi+Tgg3zfXSQ+OymyiDYxDsXLdabimq1X7Bg6sicfB4w JiNCtu3wyoZ/W1+rVzZCjiYkn8XChXDW9vFQU41jAslIGcE+1mDzfg9LPwndTFpRTrQD n98w==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=UxvFWK4X; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-85616-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-85616-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id z2-20020a05640240c200b005668feb45fdsi103948edb.36.2024.02.28.11.38.26 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Feb 2024 11:38:26 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-85616-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=UxvFWK4X; arc=pass (i=1 spf=pass spfdomain=intel.com dkim=pass dkdomain=intel.com dmarc=pass fromdomain=intel.com); spf=pass (google.com: domain of linux-kernel+bounces-85616-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-85616-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 089701F2625A for ; Wed, 28 Feb 2024 19:38:26 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id CDD8D1649D5; Wed, 28 Feb 2024 19:37:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="UxvFWK4X" Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.21]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9C91515D5CC; Wed, 28 Feb 2024 19:37:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.21 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709149042; cv=none; b=keJwmy7OTjeVZsAtj71ag6s/BBSiNPRc9JCLy/RK6kaX7XLXb6G24B8yLYkpIjf2h0mxIJi0L5g1FHoGdR0ZxQxgUBpGg4ELgSudu66FsoCMX+Ik+G1++G3NY4hCFMajf09/0O+6pUaZwyzZ8+38Rsr1FqzOHFjo57iE581RilQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709149042; c=relaxed/simple; bh=CCNsGpyPB60rmAZbLn5fA0IiWeGHXWTCmie41OkzAKU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=XDJvtVcgWOBMZln3ISzzN0rW85gPQdqUAyHGQRiFaC9GlEkEz2w6qMI76G4Alpl33PoVmlmkwu2dXnc7KIsoO/VPBEMIUPXORGdKjhqcZQmnfhhn/WInujTxk0u+CvC0c/OYNrLiTJ5JwpaGIEM+ImxKZklGE3pnfa2SUnR10kQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=UxvFWK4X; arc=none smtp.client-ip=198.175.65.21 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1709149041; x=1740685041; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=CCNsGpyPB60rmAZbLn5fA0IiWeGHXWTCmie41OkzAKU=; b=UxvFWK4XgBNMIVpYWTOol1rSTFFROCcXE1zCU2EL228ja6QoEEJvBTfT hVAeih+3LbaUtBC2VcyqPP9AZhx19VDcLZau7h5Xs7gVJcHtihSwQccVx eHdzYSoFjbKRDdUbnyJ/6H+Qt/1F+Hu2K2LVg7x0Hrw/BR+qGTzk/Txnu bkuJ6nOugzzP0+QvSMV5YHuRjDyy4LMtNtLkZIX1N/7QvgjKjN9tve6TT 5smW1dYipBPsyYQrULUkU+2y6IAK8e09S4NfdneCSYpDIeo4svKrJ+GyK c5TZ2xPbhsfePl41c3QHg8b+9cOozUaaya4OKAH7ZH3DmR3Cp5JfLpHdk A==; X-IronPort-AV: E=McAfee;i="6600,9927,10998"; a="3495491" X-IronPort-AV: E=Sophos;i="6.06,191,1705392000"; d="scan'208";a="3495491" Received: from orviesa010.jf.intel.com ([10.64.159.150]) by orvoesa113.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Feb 2024 11:37:19 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,191,1705392000"; d="scan'208";a="7485376" Received: from agluck-desk3.sc.intel.com ([172.25.222.105]) by orviesa010-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Feb 2024 11:37:18 -0800 From: Tony Luck To: Fenghua Yu , Reinette Chatre , Peter Newman , Jonathan Corbet , Shuah Khan , x86@kernel.org Cc: Shaopeng Tan , James Morse , Jamie Iles , Babu Moger , Randy Dunlap , Drew Fustini , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, patches@lists.linux.dev, Tony Luck Subject: [PATCH v15 0/8] Add support for Sub-NUMA cluster (SNC) systems Date: Wed, 28 Feb 2024 11:36:55 -0800 Message-ID: <20240228112935.8087-tony.luck@intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240228112215.8044-tony.luck@intel.com> References: <20240228112215.8044-tony.luck@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1792172733284360516 X-GMAIL-MSGID: 1792172733284360516 The Sub-NUMA cluster feature on some Intel processors partitions the CPUs that share an L3 cache into two or more sets. This plays havoc with the Resource Director Technology (RDT) monitoring features. Prior to this patch Intel has advised that SNC and RDT are incompatible. Some of these CPU support an MSR that can partition the RMID counters in the same way. This allows monitoring features to be used. With the caveat that users must be aware that Linux may migrate tasks more frequently between SNC nodes than between "regular" NUMA nodes, so reading counters from all SNC nodes may be needed to get a complete picture of activity for tasks. Cache and memory bandwidth allocation features continue to operate at the scope of the L3 cache. Signed-off-by: Tony Luck --- Changes since v14: https://lore.kernel.org/all/20240126223837.21835-1-tony.luck@intel.com/ 1) Rebase to TIP x86/cache + my 2-patch cleanup 2) Dropped all Reviewed/Tested tags (enough changed in TIP that this needs looking at, and testing, again). 3) Added ATOM_CRESTMONT_X (Sierra Forest) to list of SNC supporting CPUs. 4) Added a console "INFO" message when SNC detected: pr_info("Sub-NUMA cluster detected with %d nodes per L3 cache\n", ret); Note that my alternate approach posted as v15-RFC https://lore.kernel.org/all/20240130222034.37181-1-tony.luck@intel.com/ has been abandoned. I'd still like to explore splitting the L3 rdt_resource into separate control/monitor elements, but that can wait for another series of patches. Tony Luck (8): x86/resctrl: Prepare for new domain scope x86/resctrl: Prepare to split rdt_domain structure x86/resctrl: Prepare for different scope for control/monitor operations x86/resctrl: Split the rdt_domain and rdt_hw_domain structures x86/resctrl: Add node-scope to the options for feature scope x86/resctrl: Introduce snc_nodes_per_l3_cache x86/resctrl: Sub NUMA Cluster detection and enable x86/resctrl: Update documentation with Sub-NUMA cluster changes Documentation/arch/x86/resctrl.rst | 25 +- include/linux/resctrl.h | 85 +++-- arch/x86/include/asm/msr-index.h | 1 + arch/x86/kernel/cpu/resctrl/internal.h | 67 ++-- arch/x86/kernel/cpu/resctrl/core.c | 428 ++++++++++++++++++---- arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 56 +-- arch/x86/kernel/cpu/resctrl/monitor.c | 70 ++-- arch/x86/kernel/cpu/resctrl/pseudo_lock.c | 26 +- arch/x86/kernel/cpu/resctrl/rdtgroup.c | 156 ++++---- 9 files changed, 642 insertions(+), 272 deletions(-)