From patchwork Fri Nov 3 13:18:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 161301 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp1025131vqu; Fri, 3 Nov 2023 06:19:24 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHmiKpCDX9iO3xV3ufpRT1vWJba6Ba6Xk2UMFbm0odIYrJyfp2SxQnX7JioF6QRpXczUzTm X-Received: by 2002:a05:6a20:d90b:b0:181:98d6:6b01 with SMTP id jd11-20020a056a20d90b00b0018198d66b01mr5430836pzb.5.1699017564192; Fri, 03 Nov 2023 06:19:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1699017564; cv=none; d=google.com; s=arc-20160816; b=W5bjzQV1Hvq92XeX7VtYUV6C5xrF//L+W3KdyxOg3igtv0gbSmEuaiE1PdmN26W/Yz iZf1VxEOHUrmnqM/0CBi+e6uqUNzvcv9Dncb/xE6FxMdSf1438BmX4zp8Zce1R3YTLeM RgoNB5eO2zzY0/OSY2GbUCXqF7Nqmv5PBUENO6BPhnSnGHNT1aEWQiN4RLBjTvqjaQtf VaUJFFnhteouCdnohtjPkucKq1/9tftHK3bbS34J+XQLgTA9jT1PcQSxYXo7Dl7BG4IV vKGszdSfiNE4h/2shdjNf32Tu3UdVVUMgKVU3kBTdMzSAJ5kWyNzCoZFRPijkRZmbXt1 So+w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=2mIt+lv4yPEO9Yhocb1mj5iSx33QbDf2o5rRTl2H5Vw=; fh=3TTsVLri5uVG5k6QxKPRU5Fzak9FoO4yptSAdhI1VBc=; b=SxK9DA28LGKieujlm8QhXNDuA5kBRK6orGtWXKG8smWgMsFY6MGP/g/ZAGoVVqbjwF tXlYFzAtsOUZL33JlLGFsBnARoru8NmyS8z6+O2h7S7JZT1FVsNeJ9Acp2LWMj4C/Utn 3/OFy3hp9ZoZW0rnuSSar1v6goW55GTebjXDE/YRUxHOZipbQAVNi4d/iN3xVYWAARE5 lqNKVLCoXs44oviBSf+eXmtE5UCvEoiDdtLLwnF7rDOyewEba606rBU10wy4iD2q6j5A 6g2GpGxAyTxW84BPAaLp9m72600Em8fpbKSYLTEMNeNz7tH9vxkgrvwVJORIjFoSo0qM YHzw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=FO0kh7KI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from agentk.vger.email (agentk.vger.email. [23.128.96.32]) by mx.google.com with ESMTPS id t64-20020a638143000000b005b7fb04f1f9si1474972pgd.734.2023.11.03.06.19.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Nov 2023 06:19:24 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=FO0kh7KI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 5CE008039A1B; Fri, 3 Nov 2023 06:19:19 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1377687AbjKCNSm (ORCPT + 36 others); Fri, 3 Nov 2023 09:18:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39594 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1377628AbjKCNSj (ORCPT ); Fri, 3 Nov 2023 09:18:39 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EDF44134 for ; Fri, 3 Nov 2023 06:18:27 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id ffacd0b85a97d-32f7abbb8b4so1257509f8f.0 for ; Fri, 03 Nov 2023 06:18:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1699017506; x=1699622306; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2mIt+lv4yPEO9Yhocb1mj5iSx33QbDf2o5rRTl2H5Vw=; b=FO0kh7KI7yQmR3ciYXFn3OKucucTMKHrAF3aAVgOcXGBYd1ginF/OqXTsHe//MVW8M fTyCzftilpzu9XhgCEMaxgTJpZB+mNe/6tmuuf5q+VDo94GvHxJZXoMlCtIX5AHQ249U deSahFClMFwYGdmZeIDedpqvVAh8/8SXNxmTXnLVq8W3TLahFzOD0/5kIRcoeQz4+HMv 6zOqkG17PzzdnkRtAjrmzOTQVd0v3q9MjKbeqe3RJbahbqw8U5wlibXVDdrkm+kF/no1 Ej0KRuWZotawytML4kvQdpFibLjjGnidqTdCtfnUIzEy6sy1mRdk4mi9Kh862SCwcEYz UO0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699017506; x=1699622306; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2mIt+lv4yPEO9Yhocb1mj5iSx33QbDf2o5rRTl2H5Vw=; b=b9QSh8GqQp5LRN3bL0wKLnkApBE/dDCa15QQeTKGMQfs1juGKAAPj355zb125MNG/2 NlhCO2dCaTLJvI49n+Jii6rSvbbRrIoz51ARmhHWYh/xK30eB7YHPy0ICAvyqLHfTlkJ CG0IutYzT11Dl2/ca6UkcEivEcWr9xNf97HRDl9u77gKX3tNwVrT9HZ6S09Chrm2xY69 6F8lhzdGViQtd0UL1fDKjA+bEp0GywVIRdDLN8Ybk/iOAZnn70NtfgEQvW6vrI8tkhfY NbTcS/aZtqVP7VB+FkWtSxOrSSJULhEiC9peM448j+/YQg4zSU2GYu9OviArESPtwJRT SsaA== X-Gm-Message-State: AOJu0YwfJhmvFyDn8bojfg2xXmYiYaJN60ZLLxYIEXwoRcKLDxh1VkUG bNwJpD4VCOLZ8o+kP2BLvYnjSw== X-Received: by 2002:a5d:5981:0:b0:32f:7767:eab2 with SMTP id n1-20020a5d5981000000b0032f7767eab2mr19356640wri.14.1699017506222; Fri, 03 Nov 2023 06:18:26 -0700 (PDT) Received: from vingu-book.. ([2a01:e0a:f:6020:3172:eb13:5bac:126a]) by smtp.gmail.com with ESMTPSA id p14-20020a5d68ce000000b0032daf848f68sm1877565wrw.59.2023.11.03.06.18.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Nov 2023 06:18:25 -0700 (PDT) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, rafael@kernel.org, viresh.kumar@linaro.org, qyousef@layalina.io, linux-kernel@vger.kernel.org, linux-pm@vger.kernel.org Cc: lukasz.luba@arm.com, wyes.karny@amd.com, beata.michalska@arm.com, Vincent Guittot Subject: [PATCH v3 1/2] sched/schedutil: Rework performance estimation Date: Fri, 3 Nov 2023 14:18:20 +0100 Message-Id: <20231103131821.1176294-2-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231103131821.1176294-1-vincent.guittot@linaro.org> References: <20231103131821.1176294-1-vincent.guittot@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Fri, 03 Nov 2023 06:19:19 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781549041883325275 X-GMAIL-MSGID: 1781549041883325275 The current method to take into account uclamp hints when estimating the target frequency can end into situation where the selected target frequency is finally higher than uclamp hints whereas there are no real needs. Such cases mainly happen because we are currently mixing the traditional scheduler utilization signal with the uclamp performance hints. By adding these 2 metrics, we loose an important information when it comes to select the target frequency and we have to make some assumptions which can't fit all cases. Rework the interface between the scheduler and schedutil governor in order to propagate all information down to the cpufreq governor. effective_cpu_util() interface changes and now returns the actual utilization of the CPU with 2 optional inputs: - The minimum performance for this CPU; typically the capacity to handle the deadline task and the interrupt pressure. But also uclamp_min request when available. - The maximum targeting performance for this CPU which reflects the maximum level that we would like to not exceed. By default it will be the CPU capacity but can be reduced because of some performance hints set with uclamp. The value can be lower than actual utilization and/or min performance level. A new sugov_effective_cpu_perf() interface is also available to compute the final performance level that is targeted for the CPU after applying some cpufreq headroom and taking into account all inputs. With these 2 functions, schedutil is now able to decide when it must go above uclamp hints. It now also have a generic way to get the min perfromance level. The dependency between energy model and cpufreq governor and its headroom policy doesn't exist anymore. eenv_pd_max_util asks schedutil for the targeted performance after applying the impact of the waking task. Signed-off-by: Vincent Guittot --- include/linux/energy_model.h | 1 - kernel/sched/core.c | 82 ++++++++++++-------------------- kernel/sched/cpufreq_schedutil.c | 40 ++++++++++++---- kernel/sched/fair.c | 22 +++++++-- kernel/sched/sched.h | 24 +++------- 5 files changed, 86 insertions(+), 83 deletions(-) diff --git a/include/linux/energy_model.h b/include/linux/energy_model.h index b9caa01dfac4..adec808b371a 100644 --- a/include/linux/energy_model.h +++ b/include/linux/energy_model.h @@ -243,7 +243,6 @@ static inline unsigned long em_cpu_energy(struct em_perf_domain *pd, scale_cpu = arch_scale_cpu_capacity(cpu); ps = &pd->table[pd->nr_perf_states - 1]; - max_util = map_util_perf(max_util); max_util = min(max_util, allowed_cpu_cap); freq = map_util_freq(max_util, ps->frequency, scale_cpu); diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 7a0c16115b79..af5333327493 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -7391,18 +7391,13 @@ int sched_core_idle_cpu(int cpu) * required to meet deadlines. */ unsigned long effective_cpu_util(int cpu, unsigned long util_cfs, - enum cpu_util_type type, - struct task_struct *p) + unsigned long *min, + unsigned long *max) { - unsigned long dl_util, util, irq, max; + unsigned long util, irq, scale; struct rq *rq = cpu_rq(cpu); - max = arch_scale_cpu_capacity(cpu); - - if (!uclamp_is_used() && - type == FREQUENCY_UTIL && rt_rq_is_runnable(&rq->rt)) { - return max; - } + scale = arch_scale_cpu_capacity(cpu); /* * Early check to see if IRQ/steal time saturates the CPU, can be @@ -7410,45 +7405,41 @@ unsigned long effective_cpu_util(int cpu, unsigned long util_cfs, * update_irq_load_avg(). */ irq = cpu_util_irq(rq); - if (unlikely(irq >= max)) - return max; + if (unlikely(irq >= scale)) { + if (min) + *min = scale; + if (max) + *max = scale; + return scale; + } + + /* + * The minimum utilization returns the highest level between: + * - the computed DL bandwidth needed with the irq pressure which + * steals time to the deadline task. + * - The minimum bandwidth requirement for CFS and/or RT. + */ + if (min) + *min = max(irq + cpu_bw_dl(rq), uclamp_rq_get(rq, UCLAMP_MIN)); /* * Because the time spend on RT/DL tasks is visible as 'lost' time to * CFS tasks and we use the same metric to track the effective * utilization (PELT windows are synchronized) we can directly add them * to obtain the CPU's actual utilization. - * - * CFS and RT utilization can be boosted or capped, depending on - * utilization clamp constraints requested by currently RUNNABLE - * tasks. - * When there are no CFS RUNNABLE tasks, clamps are released and - * frequency will be gracefully reduced with the utilization decay. */ util = util_cfs + cpu_util_rt(rq); - if (type == FREQUENCY_UTIL) - util = uclamp_rq_util_with(rq, util, p); - - dl_util = cpu_util_dl(rq); + util += cpu_util_dl(rq); /* - * For frequency selection we do not make cpu_util_dl() a permanent part - * of this sum because we want to use cpu_bw_dl() later on, but we need - * to check if the CFS+RT+DL sum is saturated (ie. no idle time) such - * that we select f_max when there is no idle time. - * - * NOTE: numerical errors or stop class might cause us to not quite hit - * saturation when we should -- something for later. + * The maximum hint is a soft bandwidth requirement which can be lower + * than the actual utilization because of uclamp_max requirements */ - if (util + dl_util >= max) - return max; + if (max) + *max = min(scale, uclamp_rq_get(rq, UCLAMP_MAX)); - /* - * OTOH, for energy computation we need the estimated running time, so - * include util_dl and ignore dl_bw. - */ - if (type == ENERGY_UTIL) - util += dl_util; + if (util >= scale) + return scale; /* * There is still idle time; further improve the number by using the @@ -7459,28 +7450,15 @@ unsigned long effective_cpu_util(int cpu, unsigned long util_cfs, * U' = irq + --------- * U * max */ - util = scale_irq_capacity(util, irq, max); + util = scale_irq_capacity(util, irq, scale); util += irq; - /* - * Bandwidth required by DEADLINE must always be granted while, for - * FAIR and RT, we use blocked utilization of IDLE CPUs as a mechanism - * to gracefully reduce the frequency when no tasks show up for longer - * periods of time. - * - * Ideally we would like to set bw_dl as min/guaranteed freq and util + - * bw_dl as requested freq. However, cpufreq is not yet ready for such - * an interface. So, we only do the latter for now. - */ - if (type == FREQUENCY_UTIL) - util += cpu_bw_dl(rq); - - return min(max, util); + return min(scale, util); } unsigned long sched_cpu_util(int cpu) { - return effective_cpu_util(cpu, cpu_util_cfs(cpu), ENERGY_UTIL, NULL); + return effective_cpu_util(cpu, cpu_util_cfs(cpu), NULL, NULL); } #endif /* CONFIG_SMP */ diff --git a/kernel/sched/cpufreq_schedutil.c b/kernel/sched/cpufreq_schedutil.c index 458d359f5991..38accd8c854b 100644 --- a/kernel/sched/cpufreq_schedutil.c +++ b/kernel/sched/cpufreq_schedutil.c @@ -47,7 +47,7 @@ struct sugov_cpu { u64 last_update; unsigned long util; - unsigned long bw_dl; + unsigned long bw_min; /* The field below is for single-CPU policies only: */ #ifdef CONFIG_NO_HZ_COMMON @@ -143,7 +143,6 @@ static unsigned int get_next_freq(struct sugov_policy *sg_policy, unsigned int freq = arch_scale_freq_invariant() ? policy->cpuinfo.max_freq : policy->cur; - util = map_util_perf(util); freq = map_util_freq(util, freq, max); if (freq == sg_policy->cached_raw_freq && !sg_policy->need_freq_update) @@ -153,14 +152,35 @@ static unsigned int get_next_freq(struct sugov_policy *sg_policy, return cpufreq_driver_resolve_freq(policy, freq); } +unsigned long sugov_effective_cpu_perf(int cpu, unsigned long actual, + unsigned long min, + unsigned long max) +{ + struct rq *rq = cpu_rq(cpu); + + if (rt_rq_is_runnable(&rq->rt)) + return max; + + /* Add dvfs headroom to actual utilization */ + actual = map_util_perf(actual); + /* Actually we don't need to target the max performance */ + if (actual < max) + max = actual; + + /* + * Ensure at least minimum performance while providing more compute + * capacity when possible. + */ + return max(min, max); +} + static void sugov_get_util(struct sugov_cpu *sg_cpu) { - unsigned long util = cpu_util_cfs_boost(sg_cpu->cpu); - struct rq *rq = cpu_rq(sg_cpu->cpu); + unsigned long min, max, util = cpu_util_cfs_boost(sg_cpu->cpu); - sg_cpu->bw_dl = cpu_bw_dl(rq); - sg_cpu->util = effective_cpu_util(sg_cpu->cpu, util, - FREQUENCY_UTIL, NULL); + util = effective_cpu_util(sg_cpu->cpu, util, &min, &max); + sg_cpu->bw_min = map_util_perf(min); + sg_cpu->util = sugov_effective_cpu_perf(sg_cpu->cpu, util, min, max); } /** @@ -306,7 +326,7 @@ static inline bool sugov_cpu_is_busy(struct sugov_cpu *sg_cpu) { return false; } */ static inline void ignore_dl_rate_limit(struct sugov_cpu *sg_cpu) { - if (cpu_bw_dl(cpu_rq(sg_cpu->cpu)) > sg_cpu->bw_dl) + if (cpu_bw_dl(cpu_rq(sg_cpu->cpu)) > sg_cpu->bw_min) sg_cpu->sg_policy->limits_changed = true; } @@ -407,8 +427,8 @@ static void sugov_update_single_perf(struct update_util_data *hook, u64 time, sugov_cpu_is_busy(sg_cpu) && sg_cpu->util < prev_util) sg_cpu->util = prev_util; - cpufreq_driver_adjust_perf(sg_cpu->cpu, map_util_perf(sg_cpu->bw_dl), - map_util_perf(sg_cpu->util), max_cap); + cpufreq_driver_adjust_perf(sg_cpu->cpu, sg_cpu->bw_min, + sg_cpu->util, max_cap); sg_cpu->sg_policy->last_freq_update_time = time; } diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 8767988242ee..ed64f2eaaa2a 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -7671,7 +7671,7 @@ static inline void eenv_pd_busy_time(struct energy_env *eenv, for_each_cpu(cpu, pd_cpus) { unsigned long util = cpu_util(cpu, p, -1, 0); - busy_time += effective_cpu_util(cpu, util, ENERGY_UTIL, NULL); + busy_time += effective_cpu_util(cpu, util, NULL, NULL); } eenv->pd_busy_time = min(eenv->pd_cap, busy_time); @@ -7694,7 +7694,7 @@ eenv_pd_max_util(struct energy_env *eenv, struct cpumask *pd_cpus, for_each_cpu(cpu, pd_cpus) { struct task_struct *tsk = (cpu == dst_cpu) ? p : NULL; unsigned long util = cpu_util(cpu, p, dst_cpu, 1); - unsigned long eff_util; + unsigned long eff_util, min, max; /* * Performance domain frequency: utilization clamping @@ -7703,7 +7703,23 @@ eenv_pd_max_util(struct energy_env *eenv, struct cpumask *pd_cpus, * NOTE: in case RT tasks are running, by default the * FREQUENCY_UTIL's utilization can be max OPP. */ - eff_util = effective_cpu_util(cpu, util, FREQUENCY_UTIL, tsk); + eff_util = effective_cpu_util(cpu, util, &min, &max); + + /* Task's uclamp can modify min and max value */ + if (tsk && uclamp_is_used()) { + min = max(min, uclamp_eff_value(p, UCLAMP_MIN)); + + /* + * If there is no active max uclamp constraint, + * directly use task's one otherwise keep max + */ + if (uclamp_rq_is_idle(cpu_rq(cpu))) + max = uclamp_eff_value(p, UCLAMP_MAX); + else + max = max(max, uclamp_eff_value(p, UCLAMP_MAX)); + } + + eff_util = sugov_effective_cpu_perf(cpu, eff_util, min, max); max_util = max(max_util, eff_util); } diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 2e5a95486a42..302b451a3fd8 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2961,24 +2961,14 @@ static inline void cpufreq_update_util(struct rq *rq, unsigned int flags) {} #endif #ifdef CONFIG_SMP -/** - * enum cpu_util_type - CPU utilization type - * @FREQUENCY_UTIL: Utilization used to select frequency - * @ENERGY_UTIL: Utilization used during energy calculation - * - * The utilization signals of all scheduling classes (CFS/RT/DL) and IRQ time - * need to be aggregated differently depending on the usage made of them. This - * enum is used within effective_cpu_util() to differentiate the types of - * utilization expected by the callers, and adjust the aggregation accordingly. - */ -enum cpu_util_type { - FREQUENCY_UTIL, - ENERGY_UTIL, -}; - unsigned long effective_cpu_util(int cpu, unsigned long util_cfs, - enum cpu_util_type type, - struct task_struct *p); + unsigned long *min, + unsigned long *max); + +unsigned long sugov_effective_cpu_perf(int cpu, unsigned long actual, + unsigned long min, + unsigned long max); + /* * Verify the fitness of task @p to run on @cpu taking into account the From patchwork Fri Nov 3 13:18:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 161302 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp1025266vqu; Fri, 3 Nov 2023 06:19:35 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEfxuvCgun2tIoR1h4iPD5VsVowiFDOTfT5k2t8s1Zhv4nk8+rzUf208141CydheHrYNFUx X-Received: by 2002:a05:6a21:47c6:b0:180:edee:151e with SMTP id as6-20020a056a2147c600b00180edee151emr11349002pzc.22.1699017575018; Fri, 03 Nov 2023 06:19:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1699017575; cv=none; d=google.com; s=arc-20160816; b=b6kLayeblO7C5bqbmYnMbuDVyV6bMEVXKo35His4YX+VmL3QRSgyCBTJu7CJyCcg8F iCVhG7ZYfIkXORaETQ1MGyPXjtkx7IwY8RSc4Mo9Y5yECHZ3m37ek36TOO5hd1q60h2s l2ElHyYmFY8g4PMsow51ncMd4usb+jDuHL3n90Cz5MpvLNJOKRHZ8jlU0371ThTDY4jx VvZu7kqFTnS6Py9Y/LlgxbrwUzBJD5bd3N9xz4iI1OvPh72aM21UWebGIZlpMiJGGXAZ 7CeUlwnu8MrWAbeea1sVlx4RA1eILmh6MbSBbaiGB3yAfnjuav5SGFTXca+wSbrVBG2t sfPA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=MMhZpNg5qvSUb3GObDAJDoBlrGoMrTREaqDW44ExX70=; fh=3TTsVLri5uVG5k6QxKPRU5Fzak9FoO4yptSAdhI1VBc=; b=YKOuZGEH9xGGCaAzMAgI9DlzFDL9WNDhJkfRzLiq9+ChX95mLjQ9ZcmQiP3A3zvC50 alMzZQVrsuw+9+SioNTaYoBKN49yHwgqG469Eudb7CasKcvyONqzd8CObxwOzqTNEAnx 5ACc6RxUVrm40RPda6HSnk66sFfuqY5NtbkdjgJFkSuGmuiQeQUkBSB2R18HhbYsa9Pp T+ETWp4Piq0piKQU8zM9ApdR2c9xFjXVcsw7+yDNS9JZi11uPM0irm/4bDBHIb0vcxg2 ZEOWHUdNdRbJVnP1e7remy6Pb3iKxmkytD5CikOblCbLGBlMMqg9qzNfAGZ6C8b5u6fc Q12g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=KldbSS+E; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id x185-20020a6386c2000000b00573f885c4easi1444939pgd.753.2023.11.03.06.19.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Nov 2023 06:19:34 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=KldbSS+E; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 16B04829A577; Fri, 3 Nov 2023 06:19:30 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1377705AbjKCNSp (ORCPT + 36 others); Fri, 3 Nov 2023 09:18:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39640 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1377627AbjKCNSk (ORCPT ); Fri, 3 Nov 2023 09:18:40 -0400 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 83CE3D62 for ; Fri, 3 Nov 2023 06:18:29 -0700 (PDT) Received: by mail-wm1-x332.google.com with SMTP id 5b1f17b1804b1-4094301d505so16060875e9.2 for ; Fri, 03 Nov 2023 06:18:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1699017508; x=1699622308; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MMhZpNg5qvSUb3GObDAJDoBlrGoMrTREaqDW44ExX70=; b=KldbSS+EFCCP7HVG4VI4DBGVIrdCV/Hhtuf5rN3ln30/zJ63Q397n0YgVmnBmIy8ea /m0Eg8LOi9hQRn/g28Pxs28AQlkl6sB/yjRM3zEPQ6sbCHt9nnks/MT1uOaxGR/QyBcL Uj08CVfG+twi5E1aK16l4Cjh7W1CRNS/lrIsLXT9yRqzDmK7F5ChFP2de++Vd1JO9sUR Jj4nSPEgvxcxOKNo/clNX+j1RHc85HdQ1g9wEt4d/g3G5bYeWqPhOT8xdIjb4uTJqB/6 ++SOQtkh7lwnbTwtqZzF7tD7GS9EPfkLfG/3R9x79cMpfN2RBOSTOZ67pUeuG+Nb3shj JTTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699017508; x=1699622308; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MMhZpNg5qvSUb3GObDAJDoBlrGoMrTREaqDW44ExX70=; b=dKM3MKQd2N/xcU6PdvB7Erm2hwieKP3X+vU5bXrXdKEO8lPIl4iXtUjSS4UHhw2t5x 2I8Dj/Nk3uzsJc5EFv6a6Qhbw9KU3aImsds5poaj0uRLuuUI3s/k+ppOuEjKA/hAw6q+ pbKlExgvY3ICr+Mw7P3I+JfgMtkTwJ89YJJeKUuWSyb2viDEuKFjNyFb5V8msxU98ZRn nOHZh6nZnEdQ7qarZYetwt4bJxv3/oeBObAWf+rVCClWGdXEiMWYluu9B6vDM0cotTcO yItM9gXR1g3k0GZ4jl/f7lGxNulhsAPVhW2N7/LbF7Oddf8C6A2Q/8mDIlnBKm4zwEAC 5sQQ== X-Gm-Message-State: AOJu0YweQ7qQ3nDUaOLDf9inkjq5A53NJxYhNY1kxGNXIihSdogsGqpF emHgj71RxUjUGwlP/shk1JlxcQ== X-Received: by 2002:a05:600c:3d9a:b0:408:5ba9:d707 with SMTP id bi26-20020a05600c3d9a00b004085ba9d707mr17721293wmb.16.1699017507814; Fri, 03 Nov 2023 06:18:27 -0700 (PDT) Received: from vingu-book.. ([2a01:e0a:f:6020:3172:eb13:5bac:126a]) by smtp.gmail.com with ESMTPSA id p14-20020a5d68ce000000b0032daf848f68sm1877565wrw.59.2023.11.03.06.18.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Nov 2023 06:18:26 -0700 (PDT) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, rafael@kernel.org, viresh.kumar@linaro.org, qyousef@layalina.io, linux-kernel@vger.kernel.org, linux-pm@vger.kernel.org Cc: lukasz.luba@arm.com, wyes.karny@amd.com, beata.michalska@arm.com, Vincent Guittot Subject: [PATCH v3 2/2] sched/schedutil: Rework iowait boost Date: Fri, 3 Nov 2023 14:18:21 +0100 Message-Id: <20231103131821.1176294-3-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231103131821.1176294-1-vincent.guittot@linaro.org> References: <20231103131821.1176294-1-vincent.guittot@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Fri, 03 Nov 2023 06:19:30 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781549053116350802 X-GMAIL-MSGID: 1781549053116350802 Use the max value that has already been computed inside sugov_get_util() to cap the iowait boost and remove dependency with uclamp_rq_util_with() which is not used anymore. Signed-off-by: Vincent Guittot Reviewed-by: Qais Yousef --- kernel/sched/cpufreq_schedutil.c | 29 ++++++++------- kernel/sched/sched.h | 60 -------------------------------- 2 files changed, 14 insertions(+), 75 deletions(-) diff --git a/kernel/sched/cpufreq_schedutil.c b/kernel/sched/cpufreq_schedutil.c index 38accd8c854b..068c895517fb 100644 --- a/kernel/sched/cpufreq_schedutil.c +++ b/kernel/sched/cpufreq_schedutil.c @@ -174,11 +174,12 @@ unsigned long sugov_effective_cpu_perf(int cpu, unsigned long actual, return max(min, max); } -static void sugov_get_util(struct sugov_cpu *sg_cpu) +static void sugov_get_util(struct sugov_cpu *sg_cpu, unsigned long boost) { unsigned long min, max, util = cpu_util_cfs_boost(sg_cpu->cpu); util = effective_cpu_util(sg_cpu->cpu, util, &min, &max); + util = max(util, boost); sg_cpu->bw_min = map_util_perf(min); sg_cpu->util = sugov_effective_cpu_perf(sg_cpu->cpu, util, min, max); } @@ -271,18 +272,16 @@ static void sugov_iowait_boost(struct sugov_cpu *sg_cpu, u64 time, * This mechanism is designed to boost high frequently IO waiting tasks, while * being more conservative on tasks which does sporadic IO operations. */ -static void sugov_iowait_apply(struct sugov_cpu *sg_cpu, u64 time, +static unsigned long sugov_iowait_apply(struct sugov_cpu *sg_cpu, u64 time, unsigned long max_cap) { - unsigned long boost; - /* No boost currently required */ if (!sg_cpu->iowait_boost) - return; + return 0; /* Reset boost if the CPU appears to have been idle enough */ if (sugov_iowait_reset(sg_cpu, time, false)) - return; + return 0; if (!sg_cpu->iowait_boost_pending) { /* @@ -291,7 +290,7 @@ static void sugov_iowait_apply(struct sugov_cpu *sg_cpu, u64 time, sg_cpu->iowait_boost >>= 1; if (sg_cpu->iowait_boost < IOWAIT_BOOST_MIN) { sg_cpu->iowait_boost = 0; - return; + return 0; } } @@ -301,10 +300,7 @@ static void sugov_iowait_apply(struct sugov_cpu *sg_cpu, u64 time, * sg_cpu->util is already in capacity scale; convert iowait_boost * into the same scale so we can compare. */ - boost = (sg_cpu->iowait_boost * max_cap) >> SCHED_CAPACITY_SHIFT; - boost = uclamp_rq_util_with(cpu_rq(sg_cpu->cpu), boost, NULL); - if (sg_cpu->util < boost) - sg_cpu->util = boost; + return (sg_cpu->iowait_boost * max_cap) >> SCHED_CAPACITY_SHIFT; } #ifdef CONFIG_NO_HZ_COMMON @@ -334,6 +330,8 @@ static inline bool sugov_update_single_common(struct sugov_cpu *sg_cpu, u64 time, unsigned long max_cap, unsigned int flags) { + unsigned long boost; + sugov_iowait_boost(sg_cpu, time, flags); sg_cpu->last_update = time; @@ -342,8 +340,8 @@ static inline bool sugov_update_single_common(struct sugov_cpu *sg_cpu, if (!sugov_should_update_freq(sg_cpu->sg_policy, time)) return false; - sugov_get_util(sg_cpu); - sugov_iowait_apply(sg_cpu, time, max_cap); + boost = sugov_iowait_apply(sg_cpu, time, max_cap); + sugov_get_util(sg_cpu, boost); return true; } @@ -444,9 +442,10 @@ static unsigned int sugov_next_freq_shared(struct sugov_cpu *sg_cpu, u64 time) for_each_cpu(j, policy->cpus) { struct sugov_cpu *j_sg_cpu = &per_cpu(sugov_cpu, j); + unsigned long boost; - sugov_get_util(j_sg_cpu); - sugov_iowait_apply(j_sg_cpu, time, max_cap); + boost = sugov_iowait_apply(j_sg_cpu, time, max_cap); + sugov_get_util(j_sg_cpu, boost); util = max(j_sg_cpu->util, util); } diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 302b451a3fd8..e3cb8e004bd1 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -3025,59 +3025,6 @@ static inline bool uclamp_rq_is_idle(struct rq *rq) return rq->uclamp_flags & UCLAMP_FLAG_IDLE; } -/** - * uclamp_rq_util_with - clamp @util with @rq and @p effective uclamp values. - * @rq: The rq to clamp against. Must not be NULL. - * @util: The util value to clamp. - * @p: The task to clamp against. Can be NULL if you want to clamp - * against @rq only. - * - * Clamps the passed @util to the max(@rq, @p) effective uclamp values. - * - * If sched_uclamp_used static key is disabled, then just return the util - * without any clamping since uclamp aggregation at the rq level in the fast - * path is disabled, rendering this operation a NOP. - * - * Use uclamp_eff_value() if you don't care about uclamp values at rq level. It - * will return the correct effective uclamp value of the task even if the - * static key is disabled. - */ -static __always_inline -unsigned long uclamp_rq_util_with(struct rq *rq, unsigned long util, - struct task_struct *p) -{ - unsigned long min_util = 0; - unsigned long max_util = 0; - - if (!static_branch_likely(&sched_uclamp_used)) - return util; - - if (p) { - min_util = uclamp_eff_value(p, UCLAMP_MIN); - max_util = uclamp_eff_value(p, UCLAMP_MAX); - - /* - * Ignore last runnable task's max clamp, as this task will - * reset it. Similarly, no need to read the rq's min clamp. - */ - if (uclamp_rq_is_idle(rq)) - goto out; - } - - min_util = max_t(unsigned long, min_util, uclamp_rq_get(rq, UCLAMP_MIN)); - max_util = max_t(unsigned long, max_util, uclamp_rq_get(rq, UCLAMP_MAX)); -out: - /* - * Since CPU's {min,max}_util clamps are MAX aggregated considering - * RUNNABLE tasks with _different_ clamps, we can end up with an - * inversion. Fix it now when the clamps are applied. - */ - if (unlikely(min_util >= max_util)) - return min_util; - - return clamp(util, min_util, max_util); -} - /* Is the rq being capped/throttled by uclamp_max? */ static inline bool uclamp_rq_is_capped(struct rq *rq) { @@ -3115,13 +3062,6 @@ static inline unsigned long uclamp_eff_value(struct task_struct *p, return SCHED_CAPACITY_SCALE; } -static inline -unsigned long uclamp_rq_util_with(struct rq *rq, unsigned long util, - struct task_struct *p) -{ - return util; -} - static inline bool uclamp_rq_is_capped(struct rq *rq) { return false; } static inline bool uclamp_is_used(void)