From patchwork Thu Nov 9 23:09:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Luck, Tony" X-Patchwork-Id: 163631 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b129:0:b0:403:3b70:6f57 with SMTP id q9csp764732vqs; Thu, 9 Nov 2023 15:12:11 -0800 (PST) X-Google-Smtp-Source: AGHT+IHsHCHy103c5z2aM44EZYQ/dHSQL7CSoodKhdwsuNAdzuwufld1GnIPlafN6JEpJYPRwizJ X-Received: by 2002:a17:90b:38c6:b0:27d:24d6:7343 with SMTP id nn6-20020a17090b38c600b0027d24d67343mr3248692pjb.19.1699571531007; Thu, 09 Nov 2023 15:12:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699571530; cv=none; d=google.com; s=arc-20160816; b=t9ODcl8czWvcwuRYGzZXyg08GOvScFsk1UGuMPYGrdTvKFb7+i/Xk47w7mj8VA4IyD tKfKcrC++OCWR0LPWZALcJBu1vCQay3ZKZdkpFQRhy8DJ2kDQ6+5THtaz3s4V/RVqIzC Kl+t/rvGTw9tle5XvvWr2qhd83V6Rya5mvHoUNrzmMhu7/Woqz0I0vBbNWCGYbPuZh7a f4y8jM3nGoZFJts3d7KonlIn1PhXWJOt2ydDcpffWbaIfVD51M2s8DPSyqGNe9FnrsVW qb0MwqfCqcu8HXe0OvlFSdbIELXEQwaet35v7+LvnlyA5sppyZrSc60meRgOdQBkFPk/ osVQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=A6gxxQAkV82qEZX8jr1Ex1II3XfKn9k2sYTU1RUofBE=; fh=EIH9XAmicvPIUSP7TBeBhZ/WaoqG49JQ3xV1i3Gl7Co=; b=xiyoVqMruv4MnXNFjINqIOaOwuorbUAfPOVhCxAVHGl1CEYrS4bM5bhrWzU8trYqr2 sJa9RBNS4iydvyVxnxzpzuN9N2evQplKvp8nfDv5hZ1yKKrhPFo04rnw/ywjNViBiJ8M EdyoajERxx+YSg4sbH/3zOJtyvMUt8XbqFr5JKZqLGAS1HSB/wyI34H7IA+rXQqHwIzz 6nDnToa0BI/pFTBZHGIiCxH0zJ0o3WoBcW2XlAtbsrlFjnPT7u+7FwEOvzR8Hl0jgbS4 hR0lntbt41ZQ234NFdCUAGgO7jAJObByMOORpKZ3S6hTGdk2WxWmeQBMIWHhw5sCZFku IjEg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=eaWnBDtI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from groat.vger.email (groat.vger.email. [2620:137:e000::3:5]) by mx.google.com with ESMTPS id z4-20020a17090ad78400b0028103abc77csi2618809pju.18.2023.11.09.15.12.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 09 Nov 2023 15:12:10 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) client-ip=2620:137:e000::3:5; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=eaWnBDtI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 1C4DF809880A; Thu, 9 Nov 2023 15:10:18 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345371AbjKIXJi (ORCPT + 31 others); Thu, 9 Nov 2023 18:09:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44606 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229971AbjKIXJf (ORCPT ); Thu, 9 Nov 2023 18:09:35 -0500 Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 36E144239; Thu, 9 Nov 2023 15:09:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1699571373; x=1731107373; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=d4//eOg83Mx1w5bc2Vnzq/3wHqfRX9iPAHd6VXBHp60=; b=eaWnBDtIxqZ11K61OMEyXP/3b82xJR4HCTkgHUFLLWQOMU05BnKojLhV 1VFx0OCZcDnVcyC3FJS4DxgIhzgRuCnBwy+qYpD4n1anuIrWAo6A0jCXq 6DcRr31UyGrzapfHjCW94b2+zp0rBhHIA/Z/VLdUHns60EYWVejnzeVgQ A8PqD6vF35RVPJ+yY8qSJVpmJ5Dn0gn7bvBBtfNxcTS6qb3fhZelg4BHI h/W4OZySNqY3Mqu7L7WFKTbk2njVmcMwlh+gP5f4uvw2cv8ueXmwu2I8n ncePfeBmqPcBxFKQkAZ6toMcbUwXeX7u3f6jl5XIEpAyUuB1VADOsdYOt w==; X-IronPort-AV: E=McAfee;i="6600,9927,10889"; a="370298097" X-IronPort-AV: E=Sophos;i="6.03,290,1694761200"; d="scan'208";a="370298097" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2023 15:09:31 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10889"; a="833984558" X-IronPort-AV: E=Sophos;i="6.03,290,1694761200"; d="scan'208";a="833984558" Received: from agluck-desk3.sc.intel.com ([172.25.222.74]) by fmsmga004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2023 15:09:31 -0800 From: Tony Luck To: Fenghua Yu , Reinette Chatre , Peter Newman , Jonathan Corbet , Shuah Khan , x86@kernel.org Cc: Shaopeng Tan , James Morse , Jamie Iles , Babu Moger , Randy Dunlap , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, patches@lists.linux.dev, Tony Luck Subject: [PATCH v11 1/8] x86/resctrl: Prepare for new domain scope Date: Thu, 9 Nov 2023 15:09:08 -0800 Message-ID: <20231109230915.73600-2-tony.luck@intel.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231109230915.73600-1-tony.luck@intel.com> References: <20231031211708.37390-1-tony.luck@intel.com> <20231109230915.73600-1-tony.luck@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Thu, 09 Nov 2023 15:10:18 -0800 (PST) X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778061806902623891 X-GMAIL-MSGID: 1782129917855670195 Resctrl resources operate on subsets of CPUs in the system with the defining attribute of each subset being an instance of a particular level of cache. E.g. all CPUs sharing an L3 cache would be part of the same domain. In preparation for features that are scoped at the NUMA node level change the code from explicit references to "cache_level" to a more generic scope. At this point the only options for this scope are groups of CPUs that share an L2 cache or L3 cache. Clean up the error handling when looking up domains. Report invalid id's before calling rdt_find_domain() in preparation for better messages when scope can be other than cache scope. This means that rdt_find_domain() will never return an error. So remove checks for error from the callsites. Signed-off-by: Tony Luck Reviewed-by: Reinette Chatre --- Changes since v10: Reinette: Move a blank line in domain_add_cpu() Reniette: Fix warning message in domain_remove_cpu() include/linux/resctrl.h | 9 +++-- arch/x86/kernel/cpu/resctrl/core.c | 42 +++++++++++++++-------- arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 2 +- arch/x86/kernel/cpu/resctrl/pseudo_lock.c | 6 +++- arch/x86/kernel/cpu/resctrl/rdtgroup.c | 5 ++- 5 files changed, 45 insertions(+), 19 deletions(-) diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h index 66942d7fba7f..7d4eb7df611d 100644 --- a/include/linux/resctrl.h +++ b/include/linux/resctrl.h @@ -144,13 +144,18 @@ struct resctrl_membw { struct rdt_parse_data; struct resctrl_schema; +enum resctrl_scope { + RESCTRL_L2_CACHE = 2, + RESCTRL_L3_CACHE = 3, +}; + /** * struct rdt_resource - attributes of a resctrl resource * @rid: The index of the resource * @alloc_capable: Is allocation available on this machine * @mon_capable: Is monitor feature available on this machine * @num_rmid: Number of RMIDs available - * @cache_level: Which cache level defines scope of this resource + * @scope: Scope of this resource * @cache: Cache allocation related data * @membw: If the component has bandwidth controls, their properties. * @domains: All domains for this resource @@ -168,7 +173,7 @@ struct rdt_resource { bool alloc_capable; bool mon_capable; int num_rmid; - int cache_level; + enum resctrl_scope scope; struct resctrl_cache cache; struct resctrl_membw membw; struct list_head domains; diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c index 19e0681f0435..9f8d87abd751 100644 --- a/arch/x86/kernel/cpu/resctrl/core.c +++ b/arch/x86/kernel/cpu/resctrl/core.c @@ -65,7 +65,7 @@ struct rdt_hw_resource rdt_resources_all[] = { .r_resctrl = { .rid = RDT_RESOURCE_L3, .name = "L3", - .cache_level = 3, + .scope = RESCTRL_L3_CACHE, .domains = domain_init(RDT_RESOURCE_L3), .parse_ctrlval = parse_cbm, .format_str = "%d=%0*x", @@ -79,7 +79,7 @@ struct rdt_hw_resource rdt_resources_all[] = { .r_resctrl = { .rid = RDT_RESOURCE_L2, .name = "L2", - .cache_level = 2, + .scope = RESCTRL_L2_CACHE, .domains = domain_init(RDT_RESOURCE_L2), .parse_ctrlval = parse_cbm, .format_str = "%d=%0*x", @@ -93,7 +93,7 @@ struct rdt_hw_resource rdt_resources_all[] = { .r_resctrl = { .rid = RDT_RESOURCE_MBA, .name = "MB", - .cache_level = 3, + .scope = RESCTRL_L3_CACHE, .domains = domain_init(RDT_RESOURCE_MBA), .parse_ctrlval = parse_bw, .format_str = "%d=%*u", @@ -105,7 +105,7 @@ struct rdt_hw_resource rdt_resources_all[] = { .r_resctrl = { .rid = RDT_RESOURCE_SMBA, .name = "SMBA", - .cache_level = 3, + .scope = RESCTRL_L3_CACHE, .domains = domain_init(RDT_RESOURCE_SMBA), .parse_ctrlval = parse_bw, .format_str = "%d=%*u", @@ -401,9 +401,6 @@ struct rdt_domain *rdt_find_domain(struct rdt_resource *r, int id, struct rdt_domain *d; struct list_head *l; - if (id < 0) - return ERR_PTR(-ENODEV); - list_for_each(l, &r->domains) { d = list_entry(l, struct rdt_domain, list); /* When id is found, return its domain. */ @@ -491,6 +488,19 @@ static int arch_domain_mbm_alloc(u32 num_rmid, struct rdt_hw_domain *hw_dom) return 0; } +static int get_domain_id_from_scope(int cpu, enum resctrl_scope scope) +{ + switch (scope) { + case RESCTRL_L2_CACHE: + case RESCTRL_L3_CACHE: + return get_cpu_cacheinfo_id(cpu, scope); + default: + break; + } + + return -EINVAL; +} + /* * domain_add_cpu - Add a cpu to a resource's domain list. * @@ -506,18 +516,19 @@ static int arch_domain_mbm_alloc(u32 num_rmid, struct rdt_hw_domain *hw_dom) */ static void domain_add_cpu(int cpu, struct rdt_resource *r) { - int id = get_cpu_cacheinfo_id(cpu, r->cache_level); + int id = get_domain_id_from_scope(cpu, r->scope); struct list_head *add_pos = NULL; struct rdt_hw_domain *hw_dom; struct rdt_domain *d; int err; - d = rdt_find_domain(r, id, &add_pos); - if (IS_ERR(d)) { - pr_warn("Couldn't find cache id for CPU %d\n", cpu); + if (id < 0) { + pr_warn_once("Can't find domain id for CPU:%d scope:%d for resource %s\n", + cpu, r->scope, r->name); return; } + d = rdt_find_domain(r, id, &add_pos); if (d) { cpumask_set_cpu(cpu, &d->cpu_mask); if (r->cache.arch_has_per_cpu_cfg) @@ -556,13 +567,16 @@ static void domain_add_cpu(int cpu, struct rdt_resource *r) static void domain_remove_cpu(int cpu, struct rdt_resource *r) { - int id = get_cpu_cacheinfo_id(cpu, r->cache_level); + int id = get_domain_id_from_scope(cpu, r->scope); struct rdt_hw_domain *hw_dom; struct rdt_domain *d; + if (id < 0) + return; + d = rdt_find_domain(r, id, NULL); - if (IS_ERR_OR_NULL(d)) { - pr_warn("Couldn't find cache id for CPU %d\n", cpu); + if (!d) { + pr_warn("Couldn't find domain with id=%d for CPU %d\n", id, cpu); return; } hw_dom = resctrl_to_arch_dom(d); diff --git a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c index beccb0e87ba7..3f8891d57fac 100644 --- a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c +++ b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c @@ -563,7 +563,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg) r = &rdt_resources_all[resid].r_resctrl; d = rdt_find_domain(r, domid, NULL); - if (IS_ERR_OR_NULL(d)) { + if (!d) { ret = -ENOENT; goto out; } diff --git a/arch/x86/kernel/cpu/resctrl/pseudo_lock.c b/arch/x86/kernel/cpu/resctrl/pseudo_lock.c index 8f559eeae08e..2a682da9f43a 100644 --- a/arch/x86/kernel/cpu/resctrl/pseudo_lock.c +++ b/arch/x86/kernel/cpu/resctrl/pseudo_lock.c @@ -292,10 +292,14 @@ static void pseudo_lock_region_clear(struct pseudo_lock_region *plr) */ static int pseudo_lock_region_init(struct pseudo_lock_region *plr) { + enum resctrl_scope scope = plr->s->res->scope; struct cpu_cacheinfo *ci; int ret; int i; + if (WARN_ON_ONCE(scope != RESCTRL_L2_CACHE && scope != RESCTRL_L3_CACHE)) + return -ENODEV; + /* Pick the first cpu we find that is associated with the cache. */ plr->cpu = cpumask_first(&plr->d->cpu_mask); @@ -311,7 +315,7 @@ static int pseudo_lock_region_init(struct pseudo_lock_region *plr) plr->size = rdtgroup_cbm_to_size(plr->s->res, plr->d, plr->cbm); for (i = 0; i < ci->num_leaves; i++) { - if (ci->info_list[i].level == plr->s->res->cache_level) { + if (ci->info_list[i].level == scope) { plr->line_size = ci->info_list[i].coherency_line_size; return 0; } diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c index 69a1de92384a..c44be64d65ec 100644 --- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c +++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c @@ -1413,10 +1413,13 @@ unsigned int rdtgroup_cbm_to_size(struct rdt_resource *r, unsigned int size = 0; int num_b, i; + if (WARN_ON_ONCE(r->scope != RESCTRL_L2_CACHE && r->scope != RESCTRL_L3_CACHE)) + return size; + num_b = bitmap_weight(&cbm, r->cache.cbm_len); ci = get_cpu_cacheinfo(cpumask_any(&d->cpu_mask)); for (i = 0; i < ci->num_leaves; i++) { - if (ci->info_list[i].level == r->cache_level) { + if (ci->info_list[i].level == r->scope) { size = ci->info_list[i].size / r->cache.cbm_len * num_b; break; }