From patchwork Mon Oct 31 12:13:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kajetan Puchalski X-Patchwork-Id: 13253 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp2276690wru; Mon, 31 Oct 2022 05:16:22 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6I5GRt0cj/aYRQmA57HUrW5Oejk2D9RavCG0ZpHrC854XPjjdxhViuh+wTJ3EU+5wxOpnr X-Received: by 2002:a17:90a:e81:b0:213:d1ce:2580 with SMTP id 1-20020a17090a0e8100b00213d1ce2580mr7986730pjx.162.1667218582454; Mon, 31 Oct 2022 05:16:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667218582; cv=none; d=google.com; s=arc-20160816; b=xTsYAlM3Aw2Nhh3w5BVXkUu7EjoI4iicxObOoOuZffce5ubF9PZQs08IuxmGW2aDig NsYBO7reKzPI5WpNycHPWTN/N3LYyiBQVPfmkdx49fKVgIF/wP5N51rPG4P3arR2rS8G htDGnvbBqXo3b577W6XNcftDfCZp+G9JTEiTfYLQfNjk0YGkba9LfbrSFYcibQn9d0nV NJy0DRnDIymP4ObUwR09nSK+oJbFP1pKehEM+PDqsx/XaPQ/xo85opvjAmd/nvth6l28 vOvoU5yxUFuSRFpPvVq4svvRK07YSOB3/cNzYxVLjYLZK/jlFy/ED8DC2czvDwStxFTh qgsw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=anVv06K4voEx8glJ2FPNjL9SygdM/whii5BW5GdM7ks=; b=SHDcgCrKtBdAcRwKo0ZGMwvSoRJ6dbjqX/QkPO71LTzBHOESKGeNBsYAZ6qZhJZv+1 h5T+jIRb7s0alLT9Z7EEHA8P7xZhKrfMreXcUGQE3gAHG0GjxM9XB8F+OyXriofoymlu bBvj8oDWkbemYXs3fB3PEC5651/JrxmenbaQvtBKZ3p3/F4muly+wfY9b7VB8/82PPGV aBHPsBEaPcJLkwa5Y4VYAumL2Gx6dh9U6d7sE6KKwNxnMuobTKL1dNa6LzZLVZKKjcnY d6eY4NAtvjZsUrVwBSPlUU998eIPDud4XMOx1FP2EiC4eqzeF46ry3NiTA0BetaEcUun sVAw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id x9-20020a17090aca0900b00205d6cc25cfsi7607792pjt.72.2022.10.31.05.16.08; Mon, 31 Oct 2022 05:16:22 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231273AbiJaMOw (ORCPT + 99 others); Mon, 31 Oct 2022 08:14:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42864 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231253AbiJaMOn (ORCPT ); Mon, 31 Oct 2022 08:14:43 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id EE70CF011; Mon, 31 Oct 2022 05:14:31 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0DC3623A; Mon, 31 Oct 2022 05:14:38 -0700 (PDT) Received: from e126311.arm.com (unknown [10.57.68.155]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 931D33F5A1; Mon, 31 Oct 2022 05:14:29 -0700 (PDT) From: Kajetan Puchalski To: rafael@kernel.org Cc: daniel.lezcano@linaro.org, lukasz.luba@arm.com, Dietmar.Eggemann@arm.com, dsmythies@telus.net, yu.chen.surf@gmail.com, kajetan.puchalski@arm.com, linux-pm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v3 1/2] cpuidle: teo: Optionally skip polling states in teo_find_shallower_state() Date: Mon, 31 Oct 2022 12:13:13 +0000 Message-Id: <20221031121314.1381472-2-kajetan.puchalski@arm.com> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20221031121314.1381472-1-kajetan.puchalski@arm.com> References: <20221031121314.1381472-1-kajetan.puchalski@arm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748205392077597175?= X-GMAIL-MSGID: =?utf-8?q?1748205392077597175?= Add a no_poll flag to teo_find_shallower_state() that will let the function optionally not consider polling states. This allows the caller to guard against the function inadvertently resulting in TEO putting the CPU in a polling state when that behaviour is undesirable. Signed-off-by: Kajetan Puchalski --- drivers/cpuidle/governors/teo.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/drivers/cpuidle/governors/teo.c b/drivers/cpuidle/governors/teo.c index d9262db79cae..e2864474a98d 100644 --- a/drivers/cpuidle/governors/teo.c +++ b/drivers/cpuidle/governors/teo.c @@ -258,15 +258,17 @@ static s64 teo_middle_of_bin(int idx, struct cpuidle_driver *drv) * @dev: Target CPU. * @state_idx: Index of the capping idle state. * @duration_ns: Idle duration value to match. + * @no_poll: Don't consider polling states. */ static int teo_find_shallower_state(struct cpuidle_driver *drv, struct cpuidle_device *dev, int state_idx, - s64 duration_ns) + s64 duration_ns, bool no_poll) { int i; for (i = state_idx - 1; i >= 0; i--) { - if (dev->states_usage[i].disable) + if (dev->states_usage[i].disable || + (no_poll && drv->states[i].flags & CPUIDLE_FLAG_POLLING)) continue; state_idx = i; @@ -469,7 +471,7 @@ static int teo_select(struct cpuidle_driver *drv, struct cpuidle_device *dev, */ if (idx > idx0 && drv->states[idx].target_residency_ns > delta_tick) - idx = teo_find_shallower_state(drv, dev, idx, delta_tick); + idx = teo_find_shallower_state(drv, dev, idx, delta_tick, false); } return idx; From patchwork Mon Oct 31 12:13:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kajetan Puchalski X-Patchwork-Id: 13251 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp2276066wru; Mon, 31 Oct 2022 05:15:21 -0700 (PDT) X-Google-Smtp-Source: AMsMyM5smEEYQCtEAKztJEr0eUoeMShB/WJRsjGN5b6f2ws5O5mwbuVpRH099D49nkM9ZVNR+K1t X-Received: by 2002:a17:902:e352:b0:187:c4c:26ff with SMTP id p18-20020a170902e35200b001870c4c26ffmr10774885plc.162.1667218520745; Mon, 31 Oct 2022 05:15:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667218520; cv=none; d=google.com; s=arc-20160816; b=BG5foLYx+KHc6tuhh/r+wIb0JwmepQytmJvUxECR4YfGWzanK+JqobVdYsQJPIpwaa 65o1VNq+aATi5B0pQk5ArFLphEO98AmGzj5FJicOd7p0EL/+mQLXia36D1+VALWGwwoY KLqVshNpJXkYPQ2Vbs8n6K395WysGsdL+LOQXbfNXJxGlK+zlYCRZMeCafK7NOLGILp5 NLRWlvziY+I3+dVnkLsNtGIiG06vjBQWlyJTw6l5+rLmxGna5qevmwu8/nhRJh7ZzsqL xC+Yf0GddE5O7zOExcKsB1dvC4+sOErwf29cSOzVB9Z3VjAHwBObFd6Fh45x3hKpJtUQ oomQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=LX8kojHiWmP6t3EhRScoI5AMdyQc7zeZ+YWi/8nOQA8=; b=Vw6arFE/qr951ethi1bKyhSDI0OuB5PDGOduBk/La7Jx3PGMkccZun5LPull7ETbVq hshQBHQzcRm1AZGp2F1SXkCBsmptEOp9OK0KDB9KiOcAkBBLcyL0mp613PQADGpc+/Jf zct8qLsAoZR7lb5PZ0Mzann2irYpceFIB+1HO3FOTgvY9+gtz+jkJupLaxhswDzZ3+9r 018KSx/jhyLTgIpWWAbEBtrgPuMD/95y6SyCxyQs3CRTIh7MS2RaSlYcovKQlWaJ4TIH wh401wYuffiRFS2PIxAi2OkFK28EksHIBrliLnP8ztfNKT1iyjzWv3dtcCqRUGV/rC0j OqnA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k3-20020a170902ce0300b00186a8c18856si8678459plg.451.2022.10.31.05.15.05; Mon, 31 Oct 2022 05:15:20 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231263AbiJaMOs (ORCPT + 99 others); Mon, 31 Oct 2022 08:14:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42850 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231199AbiJaMOn (ORCPT ); Mon, 31 Oct 2022 08:14:43 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 98E31F013; Mon, 31 Oct 2022 05:14:34 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id AD17F1FB; Mon, 31 Oct 2022 05:14:40 -0700 (PDT) Received: from e126311.arm.com (unknown [10.57.68.155]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 284333F5A1; Mon, 31 Oct 2022 05:14:31 -0700 (PDT) From: Kajetan Puchalski To: rafael@kernel.org Cc: daniel.lezcano@linaro.org, lukasz.luba@arm.com, Dietmar.Eggemann@arm.com, dsmythies@telus.net, yu.chen.surf@gmail.com, kajetan.puchalski@arm.com, linux-pm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v3 2/2] cpuidle: teo: Introduce util-awareness Date: Mon, 31 Oct 2022 12:13:14 +0000 Message-Id: <20221031121314.1381472-3-kajetan.puchalski@arm.com> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20221031121314.1381472-1-kajetan.puchalski@arm.com> References: <20221031121314.1381472-1-kajetan.puchalski@arm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748205327707135710?= X-GMAIL-MSGID: =?utf-8?q?1748205327707135710?= Modern interactive systems, such as recent Android phones, tend to have power efficient shallow idle states. Selecting deeper idle states on a device while a latency-sensitive workload is running can adversely impact performance due to increased latency. Additionally, if the CPU wakes up from a deeper sleep before its target residency as is often the case, it results in a waste of energy on top of that. This patch extends the TEO governor with a mechanism adding util-awareness, effectively providing a way for the governor to reduce the selected idle state by 1 when the CPU is being utilized over a certain threshold while still trying to select the deepest possible state using TEO's metrics when the CPU is not being utilized. This is now possible since the CPU utilization is exported from the scheduler with the sched_cpu_util function and already used e.g. in the thermal governor IPA. Under this implementation, when the CPU is being utilised and the selected candidate state is C1, it will be reduced to C0 as long as C0 is not a polling state. This effectively should make the patch have no effect on most Intel systems. This can provide drastically decreased latency and performance benefits in certain types of mobile workloads that are sensitive to latency, such as Geekbench 5. Signed-off-by: Kajetan Puchalski --- drivers/cpuidle/governors/teo.c | 84 ++++++++++++++++++++++++++++++++- 1 file changed, 82 insertions(+), 2 deletions(-) diff --git a/drivers/cpuidle/governors/teo.c b/drivers/cpuidle/governors/teo.c index e2864474a98d..0f38b3ecab3c 100644 --- a/drivers/cpuidle/governors/teo.c +++ b/drivers/cpuidle/governors/teo.c @@ -2,8 +2,13 @@ /* * Timer events oriented CPU idle governor * + * TEO governor: * Copyright (C) 2018 - 2021 Intel Corporation * Author: Rafael J. Wysocki + * + * Util-awareness mechanism: + * Copyright (C) 2022 Arm Ltd. + * Author: Kajetan Puchalski */ /** @@ -99,14 +104,49 @@ * select the given idle state instead of the candidate one. * * 3. By default, select the candidate state. + * + * Util-awareness mechanism: + * + * The idea behind the util-awareness extension is that there are two distinct + * scenarios for the CPU which should result in two different approaches to idle + * state selection - utilized and not utilized. + * + * In this case, 'utilized' means that the average runqueue util of the CPU is + * above a certain threshold. + * + * When the CPU is utilized while going into idle, more likely than not it will + * be woken up to do more work soon and so a shallower idle state should be + * selected to minimise latency and maximise performance. When the CPU is not + * being utilized, the usual metrics-based approach to selecting the deepest + * available idle state should be preferred to take advantage of the power + * saving. + * + * In order to achieve this, the governor uses a utilization threshold. + * The threshold is computed per-cpu as a percentage of the CPU's capacity + * by bit shifting the capacity value. Based on testing, the shift of 6 (~1.56%) + * seems to be getting the best results. + * + * Before selecting the next idle state, the governor compares the current CPU + * util to the precomputed util threshold. If it's below, it defaults to the + * TEO metrics mechanism. If it's above and the currently selected candidate is + * C1, the idle state will be reduced to C0 as long as C0 is not a polling state. */ #include #include #include +#include #include +#include #include +/* + * The number of bits to shift the cpu's capacity by in order to determine + * the utilized threshold + */ +#define UTIL_THRESHOLD_SHIFT 6 + + /* * The PULSE value is added to metrics when they grow and the DECAY_SHIFT value * is used for decreasing metrics on a regular basis. @@ -137,9 +177,11 @@ struct teo_bin { * @time_span_ns: Time between idle state selection and post-wakeup update. * @sleep_length_ns: Time till the closest timer event (at the selection time). * @state_bins: Idle state data bins for this CPU. - * @total: Grand total of the "intercepts" and "hits" mertics for all bins. + * @total: Grand total of the "intercepts" and "hits" metrics for all bins. * @next_recent_idx: Index of the next @recent_idx entry to update. * @recent_idx: Indices of bins corresponding to recent "intercepts". + * @util_threshold: Threshold above which the CPU is considered utilized + * @utilized: Whether the last sleep on the CPU happened while utilized */ struct teo_cpu { s64 time_span_ns; @@ -148,10 +190,24 @@ struct teo_cpu { unsigned int total; int next_recent_idx; int recent_idx[NR_RECENT]; + unsigned long util_threshold; + bool utilized; }; static DEFINE_PER_CPU(struct teo_cpu, teo_cpus); +/** + * teo_get_util - Update the CPU utilized status + * @dev: Target CPU + * @cpu_data: Governor CPU data for the target CPU + */ +static void teo_get_util(struct cpuidle_device *dev, struct teo_cpu *cpu_data) +{ + unsigned long util = sched_cpu_util(dev->cpu); + + cpu_data->utilized = util > cpu_data->util_threshold; +} + /** * teo_update - Update CPU metrics after wakeup. * @drv: cpuidle driver containing state data. @@ -303,7 +359,9 @@ static int teo_select(struct cpuidle_driver *drv, struct cpuidle_device *dev, int i; if (dev->last_state_idx >= 0) { - teo_update(drv, dev); + /* don't update metrics if the cpu was utilized during the last sleep */ + if (!cpu_data->utilized) + teo_update(drv, dev); dev->last_state_idx = -1; } @@ -323,6 +381,21 @@ static int teo_select(struct cpuidle_driver *drv, struct cpuidle_device *dev, goto end; } + teo_get_util(dev, cpu_data); + /* the cpu is being utilized and there's only 2 states to choose from */ + /* no need to consider metrics, choose the shallowest non-polling state and exit */ + if (drv->state_count < 3 && cpu_data->utilized) { + for (i = 0; i < drv->state_count; ++i) { + if (dev->states_usage[i].disable || + drv->states[i].flags & CPUIDLE_FLAG_POLLING) + continue; + break; + } + + idx = i; + goto end; + } + /* * Find the deepest idle state whose target residency does not exceed * the current sleep length and the deepest idle state not deeper than @@ -454,6 +527,11 @@ static int teo_select(struct cpuidle_driver *drv, struct cpuidle_device *dev, if (idx > constraint_idx) idx = constraint_idx; + /* if the CPU is being utilized and C1 is the selected candidate */ + /* choose a shallower non-polling state to improve latency */ + if (cpu_data->utilized && idx == 1) + idx = teo_find_shallower_state(drv, dev, idx, duration_ns, true); + end: /* * Don't stop the tick if the selected state is a polling one or if the @@ -510,9 +588,11 @@ static int teo_enable_device(struct cpuidle_driver *drv, struct cpuidle_device *dev) { struct teo_cpu *cpu_data = per_cpu_ptr(&teo_cpus, dev->cpu); + unsigned long max_capacity = arch_scale_cpu_capacity(dev->cpu); int i; memset(cpu_data, 0, sizeof(*cpu_data)); + cpu_data->util_threshold = max_capacity >> UTIL_THRESHOLD_SHIFT; for (i = 0; i < NR_RECENT; i++) cpu_data->recent_idx[i] = -1;