From patchwork Fri Oct 28 06:42:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: tip-bot2 for Thomas Gleixner X-Patchwork-Id: 12110 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp659183wru; Thu, 27 Oct 2022 23:44:58 -0700 (PDT) X-Google-Smtp-Source: AMsMyM5AxY59pYTHsCD/xd+1aeV4jwEexJJJZQlnhqopzKbmxFQcl6wNIfKEh+3EzSj17cmgud4o X-Received: by 2002:a17:90b:4b4a:b0:213:35c:bf7 with SMTP id mi10-20020a17090b4b4a00b00213035c0bf7mr14340189pjb.14.1666939498359; Thu, 27 Oct 2022 23:44:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666939498; cv=none; d=google.com; s=arc-20160816; b=FP6AbhiP+6JgvG7bj2f0gq1vi+BvP9r2wjLczmfDy+TvKusKPgEQ82nhZFdhnGeKiv tw1ixsYY07CR1hDWHcVhREhVtFzmzDRE6mjaYaVQVzKuLf+7mTXUXtjTVEhjN8aXWawp u0LgLtFXz6mqDJOO8a6eFPLjMubrVOOBGnNOsM8pbzGLS0lQJbQKR7+EjeYA7uRxMkuf yn+YRgIhMYwTw0e53lSjTftiyA0K0shMdBHWzsdkesYg40qi5OQtQu+hQj9BdtagPpxh sAjhgtfA5Cv4D7od67r6gB+Xk2QUiRuuxCl5h5MS8dZ6c/KzusQ5umZH6qmyttnIZ+2U W7Og== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:robot-unsubscribe :robot-id:message-id:mime-version:references:in-reply-to:cc:subject :to:reply-to:sender:from:dkim-signature:dkim-signature:date; bh=+V1Zjg7R6j1LUTb4PqfZFg9+ZMgRl0HrtQtwBXBU7g8=; b=Dv1zAr9HvnNfd9DUUPy3h+sDIYX60F+ZBG3cx4U8CFb14dMTzUaryBacI6wnz7vMJ6 G30QKNAGT1OuInEw7SHuQ51Tprqrthi923YKsEiYzOSjA79Muyog4BuyZXLjtyGWTte5 eCNzlkfhAarQhteMagHlysN2gIOgpoXJ2rmvmqyOdtHtYiobEpTt3z3flCpFLsiXQvue ti0ciGTi0n6r5JMEw5M3GbA/5f8HWTIBIdafOG49MvKKZrxZpDDBzycWbnt/zUpw4Rpj kcGG/sSuum1pGdtb2pK5FH8GBmWNWMnibWyddDB0xfeTnhZRJ8Z99m0bUzrWmt25jSaw CfAA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=0YenhQqM; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=IYgc+Oa9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id z15-20020a170902cccf00b0017cc894c78esi4221558ple.447.2022.10.27.23.44.45; Thu, 27 Oct 2022 23:44:58 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=0YenhQqM; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=IYgc+Oa9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229978AbiJ1GoN (ORCPT + 99 others); Fri, 28 Oct 2022 02:44:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48540 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229972AbiJ1GmZ (ORCPT ); Fri, 28 Oct 2022 02:42:25 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 62A501BB573; Thu, 27 Oct 2022 23:42:20 -0700 (PDT) Date: Fri, 28 Oct 2022 06:42:17 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1666939339; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+V1Zjg7R6j1LUTb4PqfZFg9+ZMgRl0HrtQtwBXBU7g8=; b=0YenhQqMKhGZML5/VgfZybC3xgpQL0TSSsxYpNGPmKqp/FYfSuPjXTfXhTUpKw2cT2wkq2 nXV7tvaqC9hp856N+wJBJ4iRXJrtAdbGaDlgAA1WTxu0BE7wy0Za7S8gZGFq3X4GXSG5Lq RKdYFWI/x83dBdCvakjf94a62lHPrDln8r5WAZzzK8FC16az7sUo+ndRR7lul2m1Wd+6ps yJFG4Pm2RIbxIGn7vpszcvfDeElUL9bI+dIPUZ6FdXITAxmsjA5ufMI5z6ypqinlrZEkzC eMvZZK6bISRaBOe8kBRnn/XbK9Z/xK9b4S44GNU7n1sDwlvNmnZ9lHJ7Ws916Q== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1666939339; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+V1Zjg7R6j1LUTb4PqfZFg9+ZMgRl0HrtQtwBXBU7g8=; b=IYgc+Oa9xJzewmRNPO62GVAcKBuGHhOJHZh1jWHKeaKGjVmL0kSWHYZSw+5/pcYYw6xev+ D+kg2VAUXMKFgTDw== From: "tip-bot2 for Qais Yousef" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: sched/core] sched/fair: Detect capacity inversion Cc: Qais Yousef , "Peter Zijlstra (Intel)" , x86@kernel.org, linux-kernel@vger.kernel.org In-Reply-To: <20220804143609.515789-9-qais.yousef@arm.com> References: <20220804143609.515789-9-qais.yousef@arm.com> MIME-Version: 1.0 Message-ID: <166693933776.29415.4930965019951446484.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747912751277585540?= X-GMAIL-MSGID: =?utf-8?q?1747912751277585540?= The following commit has been merged into the sched/core branch of tip: Commit-ID: 44c7b80bffc3a657a36857098d5d9c49d94e652b Gitweb: https://git.kernel.org/tip/44c7b80bffc3a657a36857098d5d9c49d94e652b Author: Qais Yousef AuthorDate: Thu, 04 Aug 2022 15:36:08 +01:00 Committer: Peter Zijlstra CommitterDate: Thu, 27 Oct 2022 11:01:20 +02:00 sched/fair: Detect capacity inversion Check each performance domain to see if thermal pressure is causing its capacity to be lower than another performance domain. We assume that each performance domain has CPUs with the same capacities, which is similar to an assumption made in energy_model.c We also assume that thermal pressure impacts all CPUs in a performance domain equally. If there're multiple performance domains with the same capacity_orig, we will trigger a capacity inversion if the domain is under thermal pressure. The new cpu_in_capacity_inversion() should help users to know when information about capacity_orig are not reliable and can opt in to use the inverted capacity as the 'actual' capacity_orig. Signed-off-by: Qais Yousef Signed-off-by: Peter Zijlstra (Intel) Link: https://lore.kernel.org/r/20220804143609.515789-9-qais.yousef@arm.com --- kernel/sched/fair.c | 63 ++++++++++++++++++++++++++++++++++++++++--- kernel/sched/sched.h | 19 +++++++++++++- 2 files changed, 79 insertions(+), 3 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 0f32acb..4c4ea47 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -8824,16 +8824,73 @@ static unsigned long scale_rt_capacity(int cpu) static void update_cpu_capacity(struct sched_domain *sd, int cpu) { + unsigned long capacity_orig = arch_scale_cpu_capacity(cpu); unsigned long capacity = scale_rt_capacity(cpu); struct sched_group *sdg = sd->groups; + struct rq *rq = cpu_rq(cpu); - cpu_rq(cpu)->cpu_capacity_orig = arch_scale_cpu_capacity(cpu); + rq->cpu_capacity_orig = capacity_orig; if (!capacity) capacity = 1; - cpu_rq(cpu)->cpu_capacity = capacity; - trace_sched_cpu_capacity_tp(cpu_rq(cpu)); + rq->cpu_capacity = capacity; + + /* + * Detect if the performance domain is in capacity inversion state. + * + * Capacity inversion happens when another perf domain with equal or + * lower capacity_orig_of() ends up having higher capacity than this + * domain after subtracting thermal pressure. + * + * We only take into account thermal pressure in this detection as it's + * the only metric that actually results in *real* reduction of + * capacity due to performance points (OPPs) being dropped/become + * unreachable due to thermal throttling. + * + * We assume: + * * That all cpus in a perf domain have the same capacity_orig + * (same uArch). + * * Thermal pressure will impact all cpus in this perf domain + * equally. + */ + if (static_branch_unlikely(&sched_asym_cpucapacity)) { + unsigned long inv_cap = capacity_orig - thermal_load_avg(rq); + struct perf_domain *pd = rcu_dereference(rq->rd->pd); + + rq->cpu_capacity_inverted = 0; + + for (; pd; pd = pd->next) { + struct cpumask *pd_span = perf_domain_span(pd); + unsigned long pd_cap_orig, pd_cap; + + cpu = cpumask_any(pd_span); + pd_cap_orig = arch_scale_cpu_capacity(cpu); + + if (capacity_orig < pd_cap_orig) + continue; + + /* + * handle the case of multiple perf domains have the + * same capacity_orig but one of them is under higher + * thermal pressure. We record it as capacity + * inversion. + */ + if (capacity_orig == pd_cap_orig) { + pd_cap = pd_cap_orig - thermal_load_avg(cpu_rq(cpu)); + + if (pd_cap > inv_cap) { + rq->cpu_capacity_inverted = inv_cap; + break; + } + } else if (pd_cap_orig > inv_cap) { + rq->cpu_capacity_inverted = inv_cap; + break; + } + } + } + + trace_sched_cpu_capacity_tp(rq); sdg->sgc->capacity = capacity; sdg->sgc->min_capacity = capacity; diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index d6d488e..5f18460 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1041,6 +1041,7 @@ struct rq { unsigned long cpu_capacity; unsigned long cpu_capacity_orig; + unsigned long cpu_capacity_inverted; struct balance_callback *balance_callback; @@ -2878,6 +2879,24 @@ static inline unsigned long capacity_orig_of(int cpu) return cpu_rq(cpu)->cpu_capacity_orig; } +/* + * Returns inverted capacity if the CPU is in capacity inversion state. + * 0 otherwise. + * + * Capacity inversion detection only considers thermal impact where actual + * performance points (OPPs) gets dropped. + * + * Capacity inversion state happens when another performance domain that has + * equal or lower capacity_orig_of() becomes effectively larger than the perf + * domain this CPU belongs to due to thermal pressure throttling it hard. + * + * See comment in update_cpu_capacity(). + */ +static inline unsigned long cpu_in_capacity_inversion(int cpu) +{ + return cpu_rq(cpu)->cpu_capacity_inverted; +} + /** * enum cpu_util_type - CPU utilization type * @FREQUENCY_UTIL: Utilization used to select frequency