From patchwork Fri Feb 24 09:34:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 61207 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:7b8e:b0:9f:8c76:fea4 with SMTP id j14csp1222359dyk; Fri, 24 Feb 2023 01:38:47 -0800 (PST) X-Google-Smtp-Source: AK7set+cOZAhw77B92dawUOgo6Im5ddS7Nqw4aOTTAyezlQBxt1RegL3zSgIGTlvSVPiWsjbcgMy X-Received: by 2002:a17:906:d99:b0:8b2:2141:6de8 with SMTP id m25-20020a1709060d9900b008b221416de8mr24206819eji.73.1677231527569; Fri, 24 Feb 2023 01:38:47 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677231527; cv=none; d=google.com; s=arc-20160816; b=RBdBQeLI9fVP6IrAGNLV1pYeDZMT81W2I/0BmvvTWXfg+pvV+FWMNTnOL/RlrglBqI 2cTXe8Sund+OIphQq0fCAUySU/6ZhPzI364PxUT9vUf/DfXJPKzgE0GZwns1BvmGTTFf i3hvrD9mtZ1CNicrigwar/FYkCYq0v25hy2Trjrs0VbTBImP5hAIUyBq3rydDbV9pyr7 KllIlyWn9ryyjcp+OzWiE3A1OfrqsraA7TYCSDF7NW7d6yqrgGw61CdEPGv8NW/iCaP/ Z6ON2uZfZSg6c0OgkQaathvyDrshGrRyr5v6E9KhKm5UGMfN4ws2S7jsnNrF9ulBh0oj UfYw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=WRK9CVl+wp5UA9FM0KiXaS6Hv0eR7fUWtXzYt4onYBI=; b=iAM8Nfk8Txx4hlrz8fRPVQeOTCG58PoBfvL/dTX96RROXVKK3E5zAnr+mtcWQ3l8iz PTeeoWjO8pc+KOwO59L81DWYwi1dkQm/DHHfqAm7xXL3UZ4DMTkdNsN0FoyMW706svbG AC5H6XhpyzD0zVvVod3Qk4uVvZ38V/J61iOw5ZdzoTr4X8MDTiWbt3J4ir4ptnVXqJ9u HueHpEt0qGcbuw0DHeNO7vxhngQPfS2SehmF9nEKceeKax6s2KgdeI3wPCnRu0GH5N/B 56P07OS4rZbSswJ+wlNx7CxVRGzkXHwVSSyl+01HLkPfjds/+vjDpzpDVULK1eitwCtB wRNw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=aeaP+8Gp; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id vs2-20020a170907138200b008e7916f1952si7483362ejb.822.2023.02.24.01.38.24; Fri, 24 Feb 2023 01:38:47 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=aeaP+8Gp; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229737AbjBXJfH (ORCPT + 99 others); Fri, 24 Feb 2023 04:35:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45214 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229646AbjBXJfC (ORCPT ); Fri, 24 Feb 2023 04:35:02 -0500 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 76EDD4FC8F for ; Fri, 24 Feb 2023 01:35:00 -0800 (PST) Received: by mail-wm1-x335.google.com with SMTP id c18so6343825wmr.3 for ; Fri, 24 Feb 2023 01:35:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WRK9CVl+wp5UA9FM0KiXaS6Hv0eR7fUWtXzYt4onYBI=; b=aeaP+8GpSlBrTz0Ck22w37L8jNZXuLY7XzqRiV7TMNqQIraO8lIsG2uI0QxBG6sYSO YaCzS2VTd72to5fFRzTyrxDRqfVmkG/1xLZcqcTROHtnd8QziUlJZ5/GaA2l7chieDrd KfMNbvgA/KZbYLoZm5v/8oLyQRhmDC3fLmX8ke34ZZYBBq3ouVuLw2JMfGK4874hBrwh 5dCJ9xvzrd4k/tpGfHGW/dtUoljWWPlVPVbZkJ0YWxdbO7l3KaAj44H76grIkeXb7Z6/ 0Z0Z3UcS8m81iXcoxjO2GXo0rqHYT6GPKb1TvBOGozY192erQKt1rLmS3DMx/G7T+JYp 2oTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WRK9CVl+wp5UA9FM0KiXaS6Hv0eR7fUWtXzYt4onYBI=; b=FC+NCZFpeGKJqmm222vOe/TIySVL1Me2XHKpB94yIj2lcDy3hH0z6bYG0YVyg35HDi NyDHFYdiY0/nLSvyzy06fceAcTfS1QACypwxzjPgYyYZGRlmJGBbIUyNPeveoE/ffGs3 et6LAyD3tPRzaNIO7I+/o4hgvT49NvaERj8ubgcvLLXW8k9CiqE77V3VJKJNZIiD2BfE MpI1HFCOnahcbqRpDr+xjURuQQekvltbBe/uRJjRNW0nLyIjpAb3bczsgQ8dlvZpcfn3 cFq4SDo4jKTWh6AcFzJ+9eFcmKU3ajAhgVgOefdq85MWCS5XFsHBbVFpaNCXmZ11uXqJ HMuA== X-Gm-Message-State: AO0yUKUqwbeKwDEnII8hsZkobnHMm5APj8JbN7izXkty0VUg2wR3GBQt 9W24c50bXx/8IC0gb72t5BrH9A== X-Received: by 2002:a05:600c:4aaa:b0:3ea:f710:bdad with SMTP id b42-20020a05600c4aaa00b003eaf710bdadmr1899360wmp.29.1677231298788; Fri, 24 Feb 2023 01:34:58 -0800 (PST) Received: from vingu-book.. ([2a01:e0a:f:6020:a6f0:4ee9:c103:44cb]) by smtp.gmail.com with ESMTPSA id d18-20020a05600c34d200b003e6dcd562a6sm2239179wmq.28.2023.02.24.01.34.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 Feb 2023 01:34:58 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com, tj@kernel.org, lizefan.x@bytedance.com, hannes@cmpxchg.org, cgroups@vger.kernel.org, corbet@lwn.net, linux-doc@vger.kernel.org Cc: qyousef@layalina.io, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v12 1/8] sched/fair: fix unfairness at wakeup Date: Fri, 24 Feb 2023 10:34:47 +0100 Message-Id: <20230224093454.956298-2-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230224093454.956298-1-vincent.guittot@linaro.org> References: <20230224093454.956298-1-vincent.guittot@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DIET_1,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758704726206469718?= X-GMAIL-MSGID: =?utf-8?q?1758704726206469718?= At wake up, the vruntime of a task is updated to not be more older than a sched_latency period behind the min_vruntime. This prevents long sleeping task to get unlimited credit at wakeup. Such waking task should preempt current one to use its CPU bandwidth but wakeup_gran() can be larger than sched_latency, filter out the wakeup preemption and as a results steals some CPU bandwidth to the waking task. Make sure that a task, which vruntime has been capped, will preempt current task and use its CPU bandwidth even if wakeup_gran() is in the same range as sched_latency. If the waking task failed to preempt current it could to wait up to sysctl_sched_min_granularity before preempting it during next tick. Strictly speaking, we should use cfs->min_vruntime instead of curr->vruntime but it doesn't worth the additional overhead and complexity as the vruntime of current should be close to min_vruntime if not equal. Reported-by: Youssef Esmat Signed-off-by: Vincent Guittot Reviewed-by: Joel Fernandes (Google) Tested-by: K Prateek Nayak --- kernel/sched/fair.c | 46 ++++++++++++++++++++------------------------ kernel/sched/sched.h | 34 +++++++++++++++++++++++++++++++- 2 files changed, 54 insertions(+), 26 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index ff4dbbae3b10..81bef11eb660 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4654,33 +4654,17 @@ place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int initial) u64 vruntime = cfs_rq->min_vruntime; u64 sleep_time; - /* - * The 'current' period is already promised to the current tasks, - * however the extra weight of the new task will slow them down a - * little, place the new task so that it fits in the slot that - * stays open at the end. - */ - if (initial && sched_feat(START_DEBIT)) - vruntime += sched_vslice(cfs_rq, se); - - /* sleeps up to a single latency don't count. */ - if (!initial) { - unsigned long thresh; - - if (se_is_idle(se)) - thresh = sysctl_sched_min_granularity; - else - thresh = sysctl_sched_latency; - + if (!initial) + /* sleeps up to a single latency don't count. */ + vruntime -= get_sleep_latency(se_is_idle(se)); + else if (sched_feat(START_DEBIT)) /* - * Halve their sleep time's effect, to allow - * for a gentler effect of sleepers: + * The 'current' period is already promised to the current tasks, + * however the extra weight of the new task will slow them down a + * little, place the new task so that it fits in the slot that + * stays open at the end. */ - if (sched_feat(GENTLE_FAIR_SLEEPERS)) - thresh >>= 1; - - vruntime -= thresh; - } + vruntime += sched_vslice(cfs_rq, se); /* * Pull vruntime of the entity being placed to the base level of @@ -7721,6 +7705,18 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se) return -1; gran = wakeup_gran(se); + + /* + * At wake up, the vruntime of a task is capped to not be older than + * a sched_latency period compared to min_vruntime. This prevents long + * sleeping task to get unlimited credit at wakeup. Such waking up task + * has to preempt current in order to not lose its share of CPU + * bandwidth but wakeup_gran() can become higher than scheduling period + * for low priority task. Make sure that long sleeping task will get a + * chance to preempt current. + */ + gran = min_t(s64, gran, get_latency_max()); + if (vdiff > gran) return 1; diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 3e8df6d31c1e..51ba0af7fb27 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2458,9 +2458,9 @@ extern void check_preempt_curr(struct rq *rq, struct task_struct *p, int flags); extern const_debug unsigned int sysctl_sched_nr_migrate; extern const_debug unsigned int sysctl_sched_migration_cost; -#ifdef CONFIG_SCHED_DEBUG extern unsigned int sysctl_sched_latency; extern unsigned int sysctl_sched_min_granularity; +#ifdef CONFIG_SCHED_DEBUG extern unsigned int sysctl_sched_idle_min_granularity; extern unsigned int sysctl_sched_wakeup_granularity; extern int sysctl_resched_latency_warn_ms; @@ -2475,6 +2475,38 @@ extern unsigned int sysctl_numa_balancing_scan_size; extern unsigned int sysctl_numa_balancing_hot_threshold; #endif +static inline unsigned long get_sleep_latency(bool idle) +{ + unsigned long thresh; + + if (idle) + thresh = sysctl_sched_min_granularity; + else + thresh = sysctl_sched_latency; + + /* + * Halve their sleep time's effect, to allow + * for a gentler effect of sleepers: + */ + if (sched_feat(GENTLE_FAIR_SLEEPERS)) + thresh >>= 1; + + return thresh; +} + +static inline unsigned long get_latency_max(void) +{ + unsigned long thresh = get_sleep_latency(false); + + /* + * If the waking task failed to preempt current it could to wait up to + * sysctl_sched_min_granularity before preempting it during next tick. + */ + thresh -= sysctl_sched_min_granularity; + + return thresh; +} + #ifdef CONFIG_SCHED_HRTICK /* From patchwork Fri Feb 24 09:34:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 61208 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:7b8e:b0:9f:8c76:fea4 with SMTP id j14csp1222452dyk; Fri, 24 Feb 2023 01:39:03 -0800 (PST) X-Google-Smtp-Source: AK7set9JFZOJpDhmWqvkdOdfY3+HqXe40XozB3/3WWC11XpQCjApfB+D8BlIzSOCKJzdXY5+Xhx7 X-Received: by 2002:a17:906:5d0f:b0:8f0:143d:ee28 with SMTP id g15-20020a1709065d0f00b008f0143dee28mr6112243ejt.16.1677231542918; Fri, 24 Feb 2023 01:39:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677231542; cv=none; d=google.com; s=arc-20160816; b=1J09ls1VWkvCN/eCoagD9aBvskPUrebdV7aQ3yZMB2P3BeO2bo948fJtW7CpBCirJ4 UTb8+PG1hz8AP/TaYaMlMzDTTeUgFR29C+8M3RW1OPupq0+FGQKb1gPdo+GVf//qFZus K3hf8CxNkbh8AVL/m/dtsAgdmmzsoh5fc1B+jXXrG59iLOSIOnPfvfn1hTNxPOYH2x/m cpzimJaI34/RDz7a0rpMGQhlK5xYMv3nS0PwXFF+29sFpev5bYC7pxeTaKMsNsCTQ1R7 Qypuqw+Gf+fWlVD2oW65Bky2JdOY+1tcBDv9B03+GClumpirP1pPL/VNMkhHU43bWrn2 FMnQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=kiVS63wbAWm+QNHUVgj2Zht/JYfZt2Iaghb+UJPmGJA=; b=jGSU2w5xzgM9+KHJg7U7wlDM/le2cv9UHHM1OfUQjq0GZXVbZBVL//4Y5RR0+23sRy WY7yIbSp7PHbS+j9A5KIg2GCCm6sYpSNIl7fPIsAEHZYp+p5vxnsj0ejz02RBizCwPfe GpZk0Ema8oK+ICW3nTnU1W8fVCl9/AgFUdJbvYaXxnvwNKcJAlbh1P49Iqg43tAvKmem wlz5kkfYvNwiBpgzwk/8VLIcbp9VewGnZy+ZspFI7BwCPDmhcrsfpKn6CPzL6M1o3OF+ XE73JNv+N6SQtisC8/vtu1uPzIZkaGnIHR7WfseBzi87spbHhHqNoQfjt1Jf+ANmNO/T AaUA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=QrTYP59U; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id fq38-20020a1709069da600b008dfead50664si11233137ejc.114.2023.02.24.01.38.40; Fri, 24 Feb 2023 01:39:02 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=QrTYP59U; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229551AbjBXJfK (ORCPT + 99 others); Fri, 24 Feb 2023 04:35:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45228 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229694AbjBXJfD (ORCPT ); Fri, 24 Feb 2023 04:35:03 -0500 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9A5F45193C for ; Fri, 24 Feb 2023 01:35:01 -0800 (PST) Received: by mail-wm1-x332.google.com with SMTP id o38-20020a05600c512600b003e8320d1c11so3870076wms.1 for ; Fri, 24 Feb 2023 01:35:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kiVS63wbAWm+QNHUVgj2Zht/JYfZt2Iaghb+UJPmGJA=; b=QrTYP59UXLyEyZXXYAeeXw2AFfFauFp/C2n/IOyN3xqbAk8jjk7e20YwjW4/43PsBi 4QPrI/QTUmePR3SOLNczxnpoBs5aDDxAEopDH+YvwI2cwTf4AF65FfzRnZObTfbN+Hec WFZO5EQCCOfR7bZFaP2tJLma9p8iRxA7DlUro5Z4PqHmwHBB+u2FhNfrug7WVn7wubU3 GzPZ6F/YVjhhCLmMkMui/c6pr2O5+yI85Yufy2TgrbyHe/rdl02i5nnCM0km4exhBi3P cJhw7ozhoXe+F0JqtenV03G1YlpZGakYFvT8fC0bLeE7ExceapOsAmy4aB88jY7A/oFs 835g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kiVS63wbAWm+QNHUVgj2Zht/JYfZt2Iaghb+UJPmGJA=; b=mCnaB2o5Qy77vWFPB7nD8fjXcAmmC+6Vus1lNr63iw0PdzFAdO2+rbrNfbW0UGlc+8 wJShK8kxLFnjar6tl9qnDTNyregjTa1V/y7HHu5/dRkfrJ8VrhJR8QmGgKfGz9nLxPas MSW5DV7cxqFHN3qd2eyvkMl3n5qGcwUzHYDMFMbJrC0LThqEoCFKJ8f7GQ4WxANpEDEy rp6toz7YpDipCtsl2Jl4Gwz5JXswuXzdvONxrfggq40OZdxWmRRNHKQc6UVghjQxzEul kFCW3o0ya1dowy1b4GR1hVEWa5Y1+mRhpfsSl5hk9Z6GNq34DcLCGvw8zzJsyBMDvOwB ybCQ== X-Gm-Message-State: AO0yUKVpMFBeFXYc2nkOJ++NSqu1TrjXuP54oiSonTc4xfW1d10N7wfV OwbwLx/gCYZpI9dJht6WtHQy7w== X-Received: by 2002:a05:600c:4929:b0:3e2:1f63:4beb with SMTP id f41-20020a05600c492900b003e21f634bebmr11748426wmp.19.1677231300127; Fri, 24 Feb 2023 01:35:00 -0800 (PST) Received: from vingu-book.. ([2a01:e0a:f:6020:a6f0:4ee9:c103:44cb]) by smtp.gmail.com with ESMTPSA id d18-20020a05600c34d200b003e6dcd562a6sm2239179wmq.28.2023.02.24.01.34.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 Feb 2023 01:34:59 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com, tj@kernel.org, lizefan.x@bytedance.com, hannes@cmpxchg.org, cgroups@vger.kernel.org, corbet@lwn.net, linux-doc@vger.kernel.org Cc: qyousef@layalina.io, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v12 2/8] sched: Introduce latency-nice as a per-task attribute Date: Fri, 24 Feb 2023 10:34:48 +0100 Message-Id: <20230224093454.956298-3-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230224093454.956298-1-vincent.guittot@linaro.org> References: <20230224093454.956298-1-vincent.guittot@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758704742340097029?= X-GMAIL-MSGID: =?utf-8?q?1758704742340097029?= From: Parth Shah Latency-nice indicates the latency requirements of a task with respect to the other tasks in the system. The value of the attribute can be within the range of [-20, 19] both inclusive to be in-line with the values just like task nice values. latency_nice = -20 indicates the task to have the least latency as compared to the tasks having latency_nice = +19. The latency_nice may affect only the CFS SCHED_CLASS by getting latency requirements from the userspace. Additionally, add debugging bits for newly added latency_nice attribute. Signed-off-by: Parth Shah [rebase, move defines in sched/prio.h] Signed-off-by: Vincent Guittot Tested-by: K Prateek Nayak --- include/linux/sched.h | 1 + include/linux/sched/prio.h | 18 ++++++++++++++++++ kernel/sched/debug.c | 1 + 3 files changed, 20 insertions(+) diff --git a/include/linux/sched.h b/include/linux/sched.h index 4df2b3e76b30..6c61bde49152 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -784,6 +784,7 @@ struct task_struct { int static_prio; int normal_prio; unsigned int rt_priority; + int latency_nice; struct sched_entity se; struct sched_rt_entity rt; diff --git a/include/linux/sched/prio.h b/include/linux/sched/prio.h index ab83d85e1183..bfcd7f1d1e11 100644 --- a/include/linux/sched/prio.h +++ b/include/linux/sched/prio.h @@ -42,4 +42,22 @@ static inline long rlimit_to_nice(long prio) return (MAX_NICE - prio + 1); } +/* + * Latency nice is meant to provide scheduler hints about the relative + * latency requirements of a task with respect to other tasks. + * Thus a task with latency_nice == 19 can be hinted as the task with no + * latency requirements, in contrast to the task with latency_nice == -20 + * which should be given priority in terms of lower latency. + */ +#define MAX_LATENCY_NICE 19 +#define MIN_LATENCY_NICE -20 + +#define LATENCY_NICE_WIDTH \ + (MAX_LATENCY_NICE - MIN_LATENCY_NICE + 1) + +/* + * Default tasks should be treated as a task with latency_nice = 0. + */ +#define DEFAULT_LATENCY_NICE 0 + #endif /* _LINUX_SCHED_PRIO_H */ diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c index 1637b65ba07a..68be7a3e42a3 100644 --- a/kernel/sched/debug.c +++ b/kernel/sched/debug.c @@ -1043,6 +1043,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns, #endif P(policy); P(prio); + P(latency_nice); if (task_has_dl_policy(p)) { P(dl.runtime); P(dl.deadline); From patchwork Fri Feb 24 09:34:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 61209 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:7b8e:b0:9f:8c76:fea4 with SMTP id j14csp1222491dyk; Fri, 24 Feb 2023 01:39:06 -0800 (PST) X-Google-Smtp-Source: AK7set/vTTTkeP9FEMI0tFMe6WwFTbi23CFR6EYIMMKGDmOq4HzlFDeKil3d3LsnbQhNQZ5ip7Ks X-Received: by 2002:a17:907:1c0f:b0:8d7:6699:3ba9 with SMTP id nc15-20020a1709071c0f00b008d766993ba9mr18340843ejc.29.1677231545992; Fri, 24 Feb 2023 01:39:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677231545; cv=none; d=google.com; s=arc-20160816; b=nZOvlw3yvtKAaFHWV5pHIKymTykgJZZSe1y7hVtfNAqPkcxKqjrJNh1byMs9jAC6jV w4P+0NI12A9iDOdn8O/LswakvABSH7lF9yTeqd264QMM0KBcCrTehPUYj6rw+GC+z1oZ uAEABnajMPDk4sPgaU3sMaSTBSYN6YXHlTNezVKW6jKaU/lnBlT0hZg5Qc7HYEg37cH1 xqVW5rRCaiYBwCHxkOSjh1KyJ9+oTk0h0gicTgp7Uz1EdoXoXOlz1yp613NW+KNfc94T 1a5+wIPo79iLMk0kHVV7AjhNe5R6W8TyXdJorV1uQe4AAkxsohynuEF53S0/cz4Ur4jz MhXQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Oe0Uu935GUv7Os5h4xnlj3hjmH+1d59YLoRXBZZGIAY=; b=Y7OWs4koW91wwdQunVfYP+OF0o/Anl5qnWsGZXyo3eTrI5KflgFx6UFFvEsShLdu4c JbjN8BHZdXsxKOJCfk+e1OjtPf+5kMlPyudc6nF0WwNLht+Qhj2vNJl/oPtt1VTNjKZz E86n8FxF56Q4fyV601ZKQ7/+7PZCou9C3IrIcBlV5f+8M/UXJj2z9qNosodk46vHw+8P Y0ex5k9b2/6Swqf9fQZ8sunotTj46N0EtnbEm6HwR5zK8yCh11e/8wdYwZcTgwBxfEQu r6rO6IMoJnktAQmYss6rLCrzEt9/ERwSzOYgxR343Z+0+Zjj/dDwTmTN1R5aJAzEWjFV l7iA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="M/oQDDEh"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id j3-20020a170906534300b008bd243fe7easi27275838ejo.776.2023.02.24.01.38.43; Fri, 24 Feb 2023 01:39:05 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="M/oQDDEh"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229798AbjBXJfN (ORCPT + 99 others); Fri, 24 Feb 2023 04:35:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45292 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229684AbjBXJfF (ORCPT ); Fri, 24 Feb 2023 04:35:05 -0500 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 45BBD4DBCA for ; Fri, 24 Feb 2023 01:35:03 -0800 (PST) Received: by mail-wm1-x32a.google.com with SMTP id az36so1396295wmb.1 for ; Fri, 24 Feb 2023 01:35:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Oe0Uu935GUv7Os5h4xnlj3hjmH+1d59YLoRXBZZGIAY=; b=M/oQDDEh1x3rok6xB3rswBzsh2CRxoaZDID2byS65RkMGb0GDD3MqQmZH9MppTnquq fTby+i2kplGA5hVWZsaxpw1fs5xuLjf142dgQ0zIhorxZwv1qQC743lB+ZBvxIB3XpgX ejykMcqofVFzvaWswtOlR/1nyi3ch4k59aW9F2s1awuNUohXaJ6NmHbZ042zD6GD2Lwm 8ex+yj1V149qNPNyUdSrZ1rxeAVrMLA081+RP1BnvvM8je801xgRaPigur0shh0gwhxr PCqmbsxGGIhXYM9IQZM/+ChhpzH7cxxHHVenlx5XW5jPf/1IM8YtzbrnZziibwY+MF2w zS0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Oe0Uu935GUv7Os5h4xnlj3hjmH+1d59YLoRXBZZGIAY=; b=Es51guq9AlM+fpaR3dadBmfupp3E+yM9aVkxUJz2W+qrvweaqTTHJCK+BjG3SRo78m JkH0thsIRWBTkVUoF5+8MyiToktKnM1RIt+5bjC+xukETy1/4ALD+dJzCaIBHp7d1Bon rDXNfDLKmky9cVGweHmOmBjYqpsToLpXlq3jqyGNOAlPPAigfXCn5WhWIJk3GG0nJmln eifUG4syrrWPMautVR0xcTwF9FFt6GqQ0xFPc1lLB0wbXK4feiFoCDCoxsmFnvzXVKa0 2ztMuomqgv1hG/9PLhyokIIzChBMba8DH1pJNBS2ddkZSH7zqri/bgAwVDuvsxUYS+tN BKTw== X-Gm-Message-State: AO0yUKUWnpxas0ZOdizexXcxdz5IlOc0COhnN8ecnWc7CvHcRxM5wWiG VM5/5fd+pQ2mUBcH2yqbRdqUTTgpCqYz3jwT X-Received: by 2002:a05:600c:4d21:b0:3ea:e4bb:340c with SMTP id u33-20020a05600c4d2100b003eae4bb340cmr2641552wmp.18.1677231301533; Fri, 24 Feb 2023 01:35:01 -0800 (PST) Received: from vingu-book.. ([2a01:e0a:f:6020:a6f0:4ee9:c103:44cb]) by smtp.gmail.com with ESMTPSA id d18-20020a05600c34d200b003e6dcd562a6sm2239179wmq.28.2023.02.24.01.35.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 Feb 2023 01:35:00 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com, tj@kernel.org, lizefan.x@bytedance.com, hannes@cmpxchg.org, cgroups@vger.kernel.org, corbet@lwn.net, linux-doc@vger.kernel.org Cc: qyousef@layalina.io, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v12 3/8] sched/core: Propagate parent task's latency requirements to the child task Date: Fri, 24 Feb 2023 10:34:49 +0100 Message-Id: <20230224093454.956298-4-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230224093454.956298-1-vincent.guittot@linaro.org> References: <20230224093454.956298-1-vincent.guittot@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758704745656593307?= X-GMAIL-MSGID: =?utf-8?q?1758704745656593307?= From: Parth Shah Clone parent task's latency_nice attribute to the forked child task. Reset the latency_nice value to default value when the child task is set to sched_reset_on_fork. Also, initialize init_task.latency_nice value with DEFAULT_LATENCY_NICE value Signed-off-by: Parth Shah [rebase] Signed-off-by: Vincent Guittot Tested-by: K Prateek Nayak --- init/init_task.c | 1 + kernel/sched/core.c | 1 + 2 files changed, 2 insertions(+) diff --git a/init/init_task.c b/init/init_task.c index ff6c4b9bfe6b..7dd71dd2d261 100644 --- a/init/init_task.c +++ b/init/init_task.c @@ -78,6 +78,7 @@ struct task_struct init_task .prio = MAX_PRIO - 20, .static_prio = MAX_PRIO - 20, .normal_prio = MAX_PRIO - 20, + .latency_nice = DEFAULT_LATENCY_NICE, .policy = SCHED_NORMAL, .cpus_ptr = &init_task.cpus_mask, .user_cpus_ptr = NULL, diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 4580fe3e1d0c..28b397f9698b 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4681,6 +4681,7 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p) p->prio = p->normal_prio = p->static_prio; set_load_weight(p, false); + p->latency_nice = DEFAULT_LATENCY_NICE; /* * We don't need the reset flag anymore after the fork. It has * fulfilled its duty: From patchwork Fri Feb 24 09:34:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 61210 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:7b8e:b0:9f:8c76:fea4 with SMTP id j14csp1222770dyk; Fri, 24 Feb 2023 01:39:49 -0800 (PST) X-Google-Smtp-Source: AK7set/dzpT1Wpy/DmJM09mUHufz+jfIiCS3GFuSQ/TCgiBLha9TVW/ArXl3iyvHq4FqR9lsnPQG X-Received: by 2002:a17:907:9491:b0:8f3:9ee9:f1e3 with SMTP id dm17-20020a170907949100b008f39ee9f1e3mr1750810ejc.37.1677231588839; Fri, 24 Feb 2023 01:39:48 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677231588; cv=none; d=google.com; s=arc-20160816; b=AP3pStWONMJaaV+CyPyGrg6aabeDjKSOP+RRBw0TXeWxkf0zqEXFgw9ut8b3wwhSnB Mg3uK2h/wkae03TtUOWiPRDtSJAyF1eKdyrcK2/io9DEh5znY0T82LD2ce/D+tRqhF6+ WealJIwdwg/pWzykJ4ptTHrIXJpePSHEqfSsdkq54goheuC4O9ureWQ+uqxrsiO98VcX Orb6+p8X0X5FTX93icmDFls38Z6YuT/V5Y2GfyetRkZjR0iCWhDHiagH38jy6xGp/dGg 3qcBArsfJzFoLlQu0YXcuuk7kjc3QH55qIDlyH11/HoV5BhBX42Lv5EygBgNqW30uVGJ LxJQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=onBZrkVc33fv3ZoSz4aRQrXEiepEJotpGYY7iF4GxXI=; b=t5Vkqw04Cx4b/XoTb/5aJB2XQCmB1ujNTQC4bO29tfBh7hYBNrvgfxPt+3bR2Of1t8 KJ+oUXBS3XEQ9VAkQ2jyc5a30j3V44pj6x/I4UiS4yq1h/CrjLUeytdiUoadfZyWHd8B ljzYj2Lp+VlQX/ndkwJoTjqzwW4BbNd9nJeySxvN3iBX/QHiY0XXiWaKH05doMjWRsyx N8vz3zhIg81r7vEV3qRi1oPqrMBmBrtXV5NTDcWln5G9BZiUayYAen9yPAwVRqwSxXg4 3A4dS9XCeHTnhXUgEHMRjn2CV3JXfEd1SQ6kEkKJ+q0/Z8E2lF8K2dR7lKpjapqin3Pk 8Nyw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=zuPHFQM4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a11-20020aa7cf0b000000b004acc77edc4asi14772651edy.124.2023.02.24.01.39.26; Fri, 24 Feb 2023 01:39:48 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=zuPHFQM4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229874AbjBXJfV (ORCPT + 99 others); Fri, 24 Feb 2023 04:35:21 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45322 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229746AbjBXJfG (ORCPT ); Fri, 24 Feb 2023 04:35:06 -0500 Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 75C0D51F9B for ; Fri, 24 Feb 2023 01:35:04 -0800 (PST) Received: by mail-wm1-x336.google.com with SMTP id ay29-20020a05600c1e1d00b003e9f4c2b623so1665522wmb.3 for ; Fri, 24 Feb 2023 01:35:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=onBZrkVc33fv3ZoSz4aRQrXEiepEJotpGYY7iF4GxXI=; b=zuPHFQM4lpOyTl/tjD5lIptYFglWeIROXBm/pQPjVBH/RlONdIiyW4KYzlKHtlMMwg 0q2Fjvi0r2owWxWCYiaOpUYF+t8IMzLKn8HXosSJC32uaxIc6eqYVmy9s4qOu56hpXku lATUHrqBVHTu5ExLzSGCp8KXY+F6XajZcDsJgfFv/eLWwbdc0DdZjNtHPrhc+2treOZM ZHCbZhkWNokrB+S1zPcjX3f3z9tmDGRq9NhKv1/HuXe8lGWFG8OnbxfEpbBGUMVwIeVV gMV7SmG5NPGDcctKL7pxwcxAnNK8nU83QrfqEXBfsM9n5noZTXPToJoRedCPZNFfa0n7 p+2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=onBZrkVc33fv3ZoSz4aRQrXEiepEJotpGYY7iF4GxXI=; b=BlaLHERZqkGTBVKFbAmI/dTtYZuI6rizvP5BmH5ostNmCF5feRE/QffLbxYFlZ/2ca lGnDzbR6T/uhhmUsTbGc0B/cv3xlYlAaoFozeGUd3Jcf62IRJg4bkzOJFutNlhWLh780 M5zu9qiIjLhPCFfoJbcwjaS7o8fjtct26cg15Xd7bRJqq6uAjoyDOilP343LapNXQSqO PH94lPIJ+G5bkNdYcbbAyyez60G7Jw+sNWE+COI53sRlrSMus0EjcdtvzlRJHQgQsMhN r4AhHkTWtwN7M9r4xLZOYooqupg+D8Rj0p1sl4ajrmJcq6UeKdCJDljfFwu5AFUfuFAy XP1A== X-Gm-Message-State: AO0yUKW0rKLOLBK2MpEFhMflr5m2vL2RKb//ADDvYCP+altqOJ8l6dhQ XTBR4/rUGL5w8cmsh9zwtTE5mw== X-Received: by 2002:a05:600c:3d85:b0:3df:efdc:6505 with SMTP id bi5-20020a05600c3d8500b003dfefdc6505mr12645122wmb.0.1677231302912; Fri, 24 Feb 2023 01:35:02 -0800 (PST) Received: from vingu-book.. ([2a01:e0a:f:6020:a6f0:4ee9:c103:44cb]) by smtp.gmail.com with ESMTPSA id d18-20020a05600c34d200b003e6dcd562a6sm2239179wmq.28.2023.02.24.01.35.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 Feb 2023 01:35:02 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com, tj@kernel.org, lizefan.x@bytedance.com, hannes@cmpxchg.org, cgroups@vger.kernel.org, corbet@lwn.net, linux-doc@vger.kernel.org Cc: qyousef@layalina.io, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v12 4/8] sched: Allow sched_{get,set}attr to change latency_nice of the task Date: Fri, 24 Feb 2023 10:34:50 +0100 Message-Id: <20230224093454.956298-5-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230224093454.956298-1-vincent.guittot@linaro.org> References: <20230224093454.956298-1-vincent.guittot@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758704790027712162?= X-GMAIL-MSGID: =?utf-8?q?1758704790027712162?= From: Parth Shah Introduce the latency_nice attribute to sched_attr and provide a mechanism to change the value with the use of sched_setattr/sched_getattr syscall. Also add new flag "SCHED_FLAG_LATENCY_NICE" to hint the change in latency_nice of the task on every sched_setattr syscall. Signed-off-by: Parth Shah [rebase and add a dedicated __setscheduler_latency ] Signed-off-by: Vincent Guittot Tested-by: K Prateek Nayak --- include/uapi/linux/sched.h | 4 +++- include/uapi/linux/sched/types.h | 19 +++++++++++++++++++ kernel/sched/core.c | 24 ++++++++++++++++++++++++ tools/include/uapi/linux/sched.h | 4 +++- 4 files changed, 49 insertions(+), 2 deletions(-) diff --git a/include/uapi/linux/sched.h b/include/uapi/linux/sched.h index 3bac0a8ceab2..b2e932c25be6 100644 --- a/include/uapi/linux/sched.h +++ b/include/uapi/linux/sched.h @@ -132,6 +132,7 @@ struct clone_args { #define SCHED_FLAG_KEEP_PARAMS 0x10 #define SCHED_FLAG_UTIL_CLAMP_MIN 0x20 #define SCHED_FLAG_UTIL_CLAMP_MAX 0x40 +#define SCHED_FLAG_LATENCY_NICE 0x80 #define SCHED_FLAG_KEEP_ALL (SCHED_FLAG_KEEP_POLICY | \ SCHED_FLAG_KEEP_PARAMS) @@ -143,6 +144,7 @@ struct clone_args { SCHED_FLAG_RECLAIM | \ SCHED_FLAG_DL_OVERRUN | \ SCHED_FLAG_KEEP_ALL | \ - SCHED_FLAG_UTIL_CLAMP) + SCHED_FLAG_UTIL_CLAMP | \ + SCHED_FLAG_LATENCY_NICE) #endif /* _UAPI_LINUX_SCHED_H */ diff --git a/include/uapi/linux/sched/types.h b/include/uapi/linux/sched/types.h index f2c4589d4dbf..db1e8199e8c8 100644 --- a/include/uapi/linux/sched/types.h +++ b/include/uapi/linux/sched/types.h @@ -10,6 +10,7 @@ struct sched_param { #define SCHED_ATTR_SIZE_VER0 48 /* sizeof first published struct */ #define SCHED_ATTR_SIZE_VER1 56 /* add: util_{min,max} */ +#define SCHED_ATTR_SIZE_VER2 60 /* add: latency_nice */ /* * Extended scheduling parameters data structure. @@ -98,6 +99,22 @@ struct sched_param { * scheduled on a CPU with no more capacity than the specified value. * * A task utilization boundary can be reset by setting the attribute to -1. + * + * Latency Tolerance Attributes + * =========================== + * + * A subset of sched_attr attributes allows to specify the relative latency + * requirements of a task with respect to the other tasks running/queued in the + * system. + * + * @ sched_latency_nice task's latency_nice value + * + * The latency_nice of a task can have any value in a range of + * [MIN_LATENCY_NICE..MAX_LATENCY_NICE]. + * + * A task with latency_nice with the value of LATENCY_NICE_MIN can be + * taken for a task requiring a lower latency as opposed to the task with + * higher latency_nice. */ struct sched_attr { __u32 size; @@ -120,6 +137,8 @@ struct sched_attr { __u32 sched_util_min; __u32 sched_util_max; + /* latency requirement hints */ + __s32 sched_latency_nice; }; #endif /* _UAPI_LINUX_SCHED_TYPES_H */ diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 28b397f9698b..d327614c70b0 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -7443,6 +7443,14 @@ static void __setscheduler_params(struct task_struct *p, p->rt_priority = attr->sched_priority; p->normal_prio = normal_prio(p); set_load_weight(p, true); + +} + +static void __setscheduler_latency(struct task_struct *p, + const struct sched_attr *attr) +{ + if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) + p->latency_nice = attr->sched_latency_nice; } /* @@ -7585,6 +7593,13 @@ static int __sched_setscheduler(struct task_struct *p, return retval; } + if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) { + if (attr->sched_latency_nice > MAX_LATENCY_NICE) + return -EINVAL; + if (attr->sched_latency_nice < MIN_LATENCY_NICE) + return -EINVAL; + } + if (pi) cpuset_read_lock(); @@ -7619,6 +7634,9 @@ static int __sched_setscheduler(struct task_struct *p, goto change; if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP) goto change; + if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE && + attr->sched_latency_nice != p->latency_nice) + goto change; p->sched_reset_on_fork = reset_on_fork; retval = 0; @@ -7707,6 +7725,7 @@ static int __sched_setscheduler(struct task_struct *p, __setscheduler_params(p, attr); __setscheduler_prio(p, newprio); } + __setscheduler_latency(p, attr); __setscheduler_uclamp(p, attr); if (queued) { @@ -7917,6 +7936,9 @@ static int sched_copy_attr(struct sched_attr __user *uattr, struct sched_attr *a size < SCHED_ATTR_SIZE_VER1) return -EINVAL; + if ((attr->sched_flags & SCHED_FLAG_LATENCY_NICE) && + size < SCHED_ATTR_SIZE_VER2) + return -EINVAL; /* * XXX: Do we want to be lenient like existing syscalls; or do we want * to be strict and return an error on out-of-bounds values? @@ -8154,6 +8176,8 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sched_attr __user *, uattr, get_params(p, &kattr); kattr.sched_flags &= SCHED_FLAG_ALL; + kattr.sched_latency_nice = p->latency_nice; + #ifdef CONFIG_UCLAMP_TASK /* * This could race with another potential updater, but this is fine diff --git a/tools/include/uapi/linux/sched.h b/tools/include/uapi/linux/sched.h index 3bac0a8ceab2..b2e932c25be6 100644 --- a/tools/include/uapi/linux/sched.h +++ b/tools/include/uapi/linux/sched.h @@ -132,6 +132,7 @@ struct clone_args { #define SCHED_FLAG_KEEP_PARAMS 0x10 #define SCHED_FLAG_UTIL_CLAMP_MIN 0x20 #define SCHED_FLAG_UTIL_CLAMP_MAX 0x40 +#define SCHED_FLAG_LATENCY_NICE 0x80 #define SCHED_FLAG_KEEP_ALL (SCHED_FLAG_KEEP_POLICY | \ SCHED_FLAG_KEEP_PARAMS) @@ -143,6 +144,7 @@ struct clone_args { SCHED_FLAG_RECLAIM | \ SCHED_FLAG_DL_OVERRUN | \ SCHED_FLAG_KEEP_ALL | \ - SCHED_FLAG_UTIL_CLAMP) + SCHED_FLAG_UTIL_CLAMP | \ + SCHED_FLAG_LATENCY_NICE) #endif /* _UAPI_LINUX_SCHED_H */ From patchwork Fri Feb 24 09:34:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 61206 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:7b8e:b0:9f:8c76:fea4 with SMTP id j14csp1222202dyk; Fri, 24 Feb 2023 01:38:28 -0800 (PST) X-Google-Smtp-Source: AK7set+WJ7dOTESkLYMVRRNoCM6jo/1WjBF3etZgQpzoA0/RMdOc8asoHJ1ruTKDkSjwPRruVuiE X-Received: by 2002:a17:906:9f1d:b0:8e4:dd4d:7b07 with SMTP id fy29-20020a1709069f1d00b008e4dd4d7b07mr10199893ejc.15.1677231508229; Fri, 24 Feb 2023 01:38:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677231508; cv=none; d=google.com; s=arc-20160816; b=BeN2bS0Pd2b74cdxA1TPea80oGyye4YQyvjdEF3PkpzyVyLFp+P65EP5RcJKgXNSnZ mruInWB3DDpZqRinxDoCORHwct/Jb6bBQQCx5qQyz/FzNX63UR/exE7r8DbORKr/HzlH NHRsFrtBwtnrorutD/ONzKoKHKcYC9aYayDJDhcP3IxBK5Ew3o07cusP6Zyi/3SX4lky viNLwWyEfcqTj9MfySYkKTkhIUxmYZ1WEX0L3y7rjsv/dNZBMkw/2/a+b73B+mT+/Xlu fFmRSk5UzsTAJS0kvQ2h4u1SKaJD+Oly+E+E9zhJh0+dsTcMLeMPbuXjKycnsrHYIytJ wVLQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=NluA/bvTWT8jzIvRWktAUoRtEbCr/07FC+kKT5m9N3I=; b=wKVLsAQC5NWZ4zRgGdmzCRQQL0hzoS68TD8gs/lrMwPzlbs2gr3+tbqLjG6k+OS/Z2 vXrmrhN5/d5s4z2Dp3w4z8Oyh0urRsE9Asp40lCRPpK1bI5zAji5w1iFTG6h5lX1OT7W DNoodQIsdfHQ0qQTwwHCNkz20c3zObcus2GIQPecGmGEVcMjC1NXCCONNBHkT2xuFHix wdftJnEpkaKFw0Y2rMw9qGyhI2+1Cm3+Z2HS3hINFYdp3okpGp8Epml05tDqJXfdk1zJ aMpIvIGoQ1So+qq8R1BdvZU1NeomCIQCh3A7lSCRPb6ZoBDMcetpKXjsx5su+rQBQbUy bMrg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=bXEFaFOB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q4-20020aa7da84000000b004aef546359bsi17267878eds.585.2023.02.24.01.38.05; Fri, 24 Feb 2023 01:38:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=bXEFaFOB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229849AbjBXJfY (ORCPT + 99 others); Fri, 24 Feb 2023 04:35:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45416 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229744AbjBXJfI (ORCPT ); Fri, 24 Feb 2023 04:35:08 -0500 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E604D5C163 for ; Fri, 24 Feb 2023 01:35:05 -0800 (PST) Received: by mail-wm1-x331.google.com with SMTP id fm20-20020a05600c0c1400b003ead37e6588so1654836wmb.5 for ; Fri, 24 Feb 2023 01:35:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=NluA/bvTWT8jzIvRWktAUoRtEbCr/07FC+kKT5m9N3I=; b=bXEFaFOB9BPaiuYAnX28FoKXJwLBFME8Z0H4fAAK09KJ6cnhZqM+px+95cHkt/E1H9 mzO0qMICK+rmPU+3L2R3Q49mwTs02hN4TZb04rBrZVJxeNbwjDmcvZ1km9Iumvo+mv/f rlBCggY8qAahQf11CVTY0+doSlW7M9Jb1u4+SEBXJo4fyxdcgdazH8l7uNhWwzZnPvVJ 9WxcRp0cpqS/9JfXsnUblnwj8DnWsqCFMhHvLnLQMDG1jL1AhYbn/vWvSyUR/nr2i0VV Y5Cn1wQ6e+N94jzs3vJadHtt0qc8jC+VQpWENu05knB7LTB7aLFXSXK11vrCW83KfSws Gtfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NluA/bvTWT8jzIvRWktAUoRtEbCr/07FC+kKT5m9N3I=; b=FaM+KeYiiODgDvvUyqjl8uoAefQ3Qg1Ob8+XshGkCxjln6XKSCKKICOeURB4fp1FAV 79QdBrgj23XDB9Gcuelnw/J/QiMtzHK2UkzXfBXjQOpMGKbOVWXClon+DKLS1Dwhpo2o Z1GInmvvWUGB1lcAgBSu/PdjwROEbvpf5SUDdrC5oygrjZqxCLS96pSY5zL+zy+lQ3Rf E9RabKIRMU9avtLuei8AoU0qF+NrqAPuD1f7bzyg27bc2MLXz4pnAFLaGyuzFWBdJE0k orD/bc/hIs63LDSf1YKGNNjmKbGuqamhp1sUXUOTjMhgxKTh/YSdhkwAUI6bsC8CgMQt ybvg== X-Gm-Message-State: AO0yUKUKjYF16GMFr5tvCMSlPIeyXAQPLUyWV4SPkE35mKg4kO1049yE HeJdu05192O8K9M2Sq1Ab7PBWQ== X-Received: by 2002:a05:600c:1c8e:b0:3ea:e677:5017 with SMTP id k14-20020a05600c1c8e00b003eae6775017mr2399245wms.8.1677231304337; Fri, 24 Feb 2023 01:35:04 -0800 (PST) Received: from vingu-book.. ([2a01:e0a:f:6020:a6f0:4ee9:c103:44cb]) by smtp.gmail.com with ESMTPSA id d18-20020a05600c34d200b003e6dcd562a6sm2239179wmq.28.2023.02.24.01.35.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 Feb 2023 01:35:03 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com, tj@kernel.org, lizefan.x@bytedance.com, hannes@cmpxchg.org, cgroups@vger.kernel.org, corbet@lwn.net, linux-doc@vger.kernel.org Cc: qyousef@layalina.io, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v12 5/8] sched/fair: Take into account latency priority at wakeup Date: Fri, 24 Feb 2023 10:34:51 +0100 Message-Id: <20230224093454.956298-6-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230224093454.956298-1-vincent.guittot@linaro.org> References: <20230224093454.956298-1-vincent.guittot@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-1.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,PDS_BTC_ID,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758704705838555615?= X-GMAIL-MSGID: =?utf-8?q?1758704705838555615?= Take into account the latency priority of a thread when deciding to preempt the current running thread. We don't want to provide more CPU bandwidth to a thread but reorder the scheduling to run latency sensitive task first whenever possible. As long as a thread didn't use its bandwidth, it will be able to preempt the current thread. At the opposite, a thread with a low latency priority will preempt current thread at wakeup only to keep fair CPU bandwidth sharing. Otherwise it will wait for the tick to get its sched slice. curr vruntime | sysctl_sched_wakeup_granularity <--> ----------------------------------|----|-----------------------|--------------- | |<---------------------> | . sysctl_sched_latency | . default/current latency entity | . | . 1111111111111111111111111111111111|0000|-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1- se preempts curr at wakeup ------>|<- se doesn't preempt curr ----------------- | . | . | . low latency entity | . ---------------------->| % of sysctl_sched_latency | 1111111111111111111111111111111111111111111111111111111111|0000|-1-1-1-1-1-1-1- preempt ------------------------------------------------->|<- do not preempt -- | . | . | . high latency entity | . |<-----------------------|----. | % of sysctl_sched_latency . 111111111|0000|-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1 preempt->|<- se doesn't preempt curr ------------------------------------------ Tests results of nice latency impact on heavy load like hackbench: hackbench -l (2560 / group) -g group group latency 0 latency 19 1 1.378(+/- 1%) 1.337(+/- 1%) + 3% 4 1.393(+/- 3%) 1.312(+/- 3%) + 6% 8 1.308(+/- 2%) 1.279(+/- 1%) + 2% 16 1.347(+/- 1%) 1.317(+/- 1%) + 2% hackbench -p -l (2560 / group) -g group group 1 1.836(+/- 17%) 1.148(+/- 5%) +37% 4 1.586(+/- 6%) 1.109(+/- 8%) +30% 8 1.209(+/- 4%) 0.780(+/- 4%) +35% 16 0.805(+/- 5%) 0.728(+/- 4%) +10% By deacreasing the latency prio, we reduce the number of preemption at wakeup and help hackbench making progress. Test results of nice latency impact on short live load like cyclictest while competing with heavy load like hackbench: hackbench -l 10000 -g $group & cyclictest --policy other -D 5 -q -n latency 0 latency -20 group min avg max min avg max 0 16 19 29 17 18 29 1 43 299 7359 63 84 3422 4 56 449 14806 45 83 284 8 63 820 51123 63 83 283 16 64 1326 70684 41 157 26852 group = 0 means that hackbench is not running. The avg is significantly improved with nice latency -20 especially with large number of groups but min and max remain quite similar. If we add the histogram parameter to get details of latency, we have : hackbench -l 10000 -g 16 & cyclictest --policy other -D 5 -q -n -H 20000 --histfile data.txt latency 0 latency -20 Min Latencies: 64 62 Avg Latencies: 1170 107 Max Latencies: 88069 10417 50% latencies: 122 86 75% latencies: 614 91 85% latencies: 961 94 90% latencies: 1225 97 95% latencies: 6120 102 99% latencies: 18328 159 With percentile details, we see the benefit of nice latency -20 as only 1% of the latencies are above 159us whereas the default latency has got 15% around ~1ms or above and 5% over the 6ms. Signed-off-by: Vincent Guittot Tested-by: K Prateek Nayak --- include/linux/sched.h | 4 +++- include/linux/sched/prio.h | 9 +++++++++ init/init_task.c | 2 +- kernel/sched/core.c | 19 ++++++++++++++----- kernel/sched/debug.c | 2 +- kernel/sched/fair.c | 32 +++++++++++++++++++++++++++----- kernel/sched/sched.h | 11 +++++++++++ 7 files changed, 66 insertions(+), 13 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 6c61bde49152..38decae3e156 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -568,6 +568,8 @@ struct sched_entity { /* cached value of my_q->h_nr_running */ unsigned long runnable_weight; #endif + /* preemption offset in ns */ + long latency_offset; #ifdef CONFIG_SMP /* @@ -784,7 +786,7 @@ struct task_struct { int static_prio; int normal_prio; unsigned int rt_priority; - int latency_nice; + int latency_prio; struct sched_entity se; struct sched_rt_entity rt; diff --git a/include/linux/sched/prio.h b/include/linux/sched/prio.h index bfcd7f1d1e11..be79503d86af 100644 --- a/include/linux/sched/prio.h +++ b/include/linux/sched/prio.h @@ -59,5 +59,14 @@ static inline long rlimit_to_nice(long prio) * Default tasks should be treated as a task with latency_nice = 0. */ #define DEFAULT_LATENCY_NICE 0 +#define DEFAULT_LATENCY_PRIO (DEFAULT_LATENCY_NICE + LATENCY_NICE_WIDTH/2) + +/* + * Convert user-nice values [ -20 ... 0 ... 19 ] + * to static latency [ 0..39 ], + * and back. + */ +#define NICE_TO_LATENCY(nice) ((nice) + DEFAULT_LATENCY_PRIO) +#define LATENCY_TO_NICE(prio) ((prio) - DEFAULT_LATENCY_PRIO) #endif /* _LINUX_SCHED_PRIO_H */ diff --git a/init/init_task.c b/init/init_task.c index 7dd71dd2d261..071deff8dbd1 100644 --- a/init/init_task.c +++ b/init/init_task.c @@ -78,7 +78,7 @@ struct task_struct init_task .prio = MAX_PRIO - 20, .static_prio = MAX_PRIO - 20, .normal_prio = MAX_PRIO - 20, - .latency_nice = DEFAULT_LATENCY_NICE, + .latency_prio = DEFAULT_LATENCY_PRIO, .policy = SCHED_NORMAL, .cpus_ptr = &init_task.cpus_mask, .user_cpus_ptr = NULL, diff --git a/kernel/sched/core.c b/kernel/sched/core.c index d327614c70b0..d5b7e237d79b 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -1285,6 +1285,11 @@ static void set_load_weight(struct task_struct *p, bool update_load) } } +static void set_latency_offset(struct task_struct *p) +{ + p->se.latency_offset = calc_latency_offset(p->latency_prio); +} + #ifdef CONFIG_UCLAMP_TASK /* * Serializes updates of utilization clamp values @@ -4681,7 +4686,9 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p) p->prio = p->normal_prio = p->static_prio; set_load_weight(p, false); - p->latency_nice = DEFAULT_LATENCY_NICE; + p->latency_prio = NICE_TO_LATENCY(0); + set_latency_offset(p); + /* * We don't need the reset flag anymore after the fork. It has * fulfilled its duty: @@ -7449,8 +7456,10 @@ static void __setscheduler_params(struct task_struct *p, static void __setscheduler_latency(struct task_struct *p, const struct sched_attr *attr) { - if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) - p->latency_nice = attr->sched_latency_nice; + if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) { + p->latency_prio = NICE_TO_LATENCY(attr->sched_latency_nice); + set_latency_offset(p); + } } /* @@ -7635,7 +7644,7 @@ static int __sched_setscheduler(struct task_struct *p, if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP) goto change; if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE && - attr->sched_latency_nice != p->latency_nice) + attr->sched_latency_nice != LATENCY_TO_NICE(p->latency_prio)) goto change; p->sched_reset_on_fork = reset_on_fork; @@ -8176,7 +8185,7 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sched_attr __user *, uattr, get_params(p, &kattr); kattr.sched_flags &= SCHED_FLAG_ALL; - kattr.sched_latency_nice = p->latency_nice; + kattr.sched_latency_nice = LATENCY_TO_NICE(p->latency_prio); #ifdef CONFIG_UCLAMP_TASK /* diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c index 68be7a3e42a3..b3922184af91 100644 --- a/kernel/sched/debug.c +++ b/kernel/sched/debug.c @@ -1043,7 +1043,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns, #endif P(policy); P(prio); - P(latency_nice); + P(latency_prio); if (task_has_dl_policy(p)) { P(dl.runtime); P(dl.deadline); diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 81bef11eb660..414b6243208b 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4877,6 +4877,8 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) update_idle_cfs_rq_clock_pelt(cfs_rq); } +static long wakeup_latency_gran(struct sched_entity *curr, struct sched_entity *se); + /* * Preempt the current task with a newly woken task if needed: */ @@ -4885,7 +4887,7 @@ check_preempt_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr) { unsigned long ideal_runtime, delta_exec; struct sched_entity *se; - s64 delta; + s64 delta, offset; /* * When many tasks blow up the sched_period; it is possible that @@ -4916,10 +4918,12 @@ check_preempt_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr) se = __pick_first_entity(cfs_rq); delta = curr->vruntime - se->vruntime; - if (delta < 0) + offset = wakeup_latency_gran(curr, se); + if (delta < offset) return; - if (delta > ideal_runtime) + if ((delta > ideal_runtime) || + (delta > get_latency_max())) resched_curr(rq_of(cfs_rq)); } @@ -7662,6 +7666,23 @@ balance_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) } #endif /* CONFIG_SMP */ +static long wakeup_latency_gran(struct sched_entity *curr, struct sched_entity *se) +{ + long latency_offset = se->latency_offset; + + /* + * A negative latency offset means that the sched_entity has latency + * requirement that needs to be evaluated versus other entity. + * Otherwise, use the latency weight to evaluate how much scheduling + * delay is acceptable by se. + */ + if ((latency_offset < 0) || (curr->latency_offset < 0)) + latency_offset -= curr->latency_offset; + latency_offset = min_t(long, latency_offset, get_latency_max()); + + return latency_offset; +} + static unsigned long wakeup_gran(struct sched_entity *se) { unsigned long gran = sysctl_sched_wakeup_granularity; @@ -7700,11 +7721,12 @@ static int wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se) { s64 gran, vdiff = curr->vruntime - se->vruntime; + s64 offset = wakeup_latency_gran(curr, se); - if (vdiff <= 0) + if (vdiff < offset) return -1; - gran = wakeup_gran(se); + gran = offset + wakeup_gran(se); /* * At wake up, the vruntime of a task is capped to not be older than diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 51ba0af7fb27..3f42f86105d4 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2494,6 +2494,17 @@ static inline unsigned long get_sleep_latency(bool idle) return thresh; } +/* + * Calculate the latency offset for a priority level. + * We use a linear mapping of the priority in the range: + * [-sysctl_sched_latency:sysctl_sched_latency] + */ +static inline long calc_latency_offset(int prio) +{ + return (long)get_sleep_latency(false) * LATENCY_TO_NICE(prio) / + (LATENCY_NICE_WIDTH/2); +} + static inline unsigned long get_latency_max(void) { unsigned long thresh = get_sleep_latency(false); From patchwork Fri Feb 24 09:34:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 61205 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:7b8e:b0:9f:8c76:fea4 with SMTP id j14csp1222168dyk; Fri, 24 Feb 2023 01:38:23 -0800 (PST) X-Google-Smtp-Source: AK7set/dDD+bh6u8ZGzz54xO+IYPdydYEg1b93tylAZRLioKfMeWzBZ4fZ3EpsnkmlYo4lrCnwmY X-Received: by 2002:a17:906:af85:b0:888:db6b:5fa9 with SMTP id mj5-20020a170906af8500b00888db6b5fa9mr19931771ejb.67.1677231503384; Fri, 24 Feb 2023 01:38:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677231503; cv=none; d=google.com; s=arc-20160816; b=OFQmGKSVoQWSuDpHYAQ4pCtk+Y0gK5ymb9Wf1vQ+UDCPmUC+iwhZdifeD79cHQQr5f WgGGWL85d+MD3Hai2DropgA7h8nvGukU/+befNz1mxYjTiNo1NusHAJ/aSFMZ37gZNFj fJC5Dp/J95UnZxoMH/M+beCHXOqXXiVb+5liZa05IUaSKm2NrPletzQ9gqxIbYFtfzX6 i9I7d6GT30FQPCofMpB6qwxity8NOxraWygQH5aymteBRSFHrDIRUaPxNKJ41GUil2Ca EwomoMq8JL8iaANYHfSODtVLyVBLoDSjeMTDB0l8tfKDtCU5inHadhqKt90xlZp4pOKs 4y0A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=m64jqOrcIRXu/umTrAuoSeK+k4rXGOgELajP5IUVuyY=; b=PrYN2JEUfT/wAxrHicZ+jKHLHSP39vQ+dwIfNLq91M4fAnGiHZ/+gQx5V5HvIQ6dAw 4QJdhCxm0QvXAT1b+EjUBBcjZBKICNNHCdhj1cnD396OwILcGuBRcTJlzKWa5DuETPn9 4NbekgSMBHq0BiUtiajiiWt7Ajmk/Ng0nJ17EW22ZrpEp9EMYVIkbpnwAhZEfeA9Ofjh ffk72K1TgULvZlwD7O7lAa9zgv3acpoQsU0sLF6aXaEYbVQBwe+kBZI8SBXRkoa0jHUj ZjqewpNz+T8FGh7t3SF4GKgJcPtvg25MyVaPSL6OiUDPNYrU1nSV3xkvrUBWCzuu4aBY TocA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=pYehFX1b; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 24-20020a170906015800b008cc17a6e394si22484771ejh.779.2023.02.24.01.38.00; Fri, 24 Feb 2023 01:38:23 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=pYehFX1b; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229901AbjBXJf2 (ORCPT + 99 others); Fri, 24 Feb 2023 04:35:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45356 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229810AbjBXJfT (ORCPT ); Fri, 24 Feb 2023 04:35:19 -0500 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5315F5EECC for ; Fri, 24 Feb 2023 01:35:07 -0800 (PST) Received: by mail-wm1-x334.google.com with SMTP id fm20-20020a05600c0c1400b003ead37e6588so1654873wmb.5 for ; Fri, 24 Feb 2023 01:35:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=m64jqOrcIRXu/umTrAuoSeK+k4rXGOgELajP5IUVuyY=; b=pYehFX1bNNfwjhqedPBDus23aDHSLj1CXv7Icw4jLz2wc0T84AoEZo3r/sIRSZVce3 FtOSNxYg74KuifAm9s89teyegf76CDzqV8TC2BvujoX5fUKQ0AtQDlIATj5KR4yrZAsG SjQeie1J0O3NNLbXgn1+6uVPT9AT0BQ3yAzGUvnYzFf+JLq9+522QPd1MnaBJlEDUobi Hh0YeEEuLGgu2UK33METSkIWcX5w+WBbiYdG3afy50VMjNvvLxdL5X3OIT/lhvUH6x43 xTPaoZjEox86Bn8gMBaHvOZB70Wa8ymL5X1n0YbgUmzYkD1be2ra+f9bKBHkRhyYbHWa Cm3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=m64jqOrcIRXu/umTrAuoSeK+k4rXGOgELajP5IUVuyY=; b=D1B4fs8JM1ZgvFIYmbcQvxEPX2pIXTLj2XH30Zbxqs53YxPdmZ4H1wYSBtDtYGSCBK QPY9oENBph3VdT1QOtcewbvNxT50s2NhdCJ/doe92RkE0+An3xUnavZU+FwppeDi3XIR fj5v6mrg7VECGzNF+YVHsxcoJQCl+SvpgbhHp6TA3Zrsze0oA0Z4qnu4HPTKH9m23ha6 PGMFoqsqXvV1o3g6GV8R+Hs1dbHO193i4pFRst58sgFPv8+ZnmmnCRS/EelKRFfcKEZ3 CgHk3e1SVuUQuwVO0Lchrg/bQ26b3098IChV+f2K8nLRkzF5FyG2A3/JjFLrNwtFWYAV pN7g== X-Gm-Message-State: AO0yUKUmPhtpWrQC3EZiOa+EVMZWSPs3e+jscOQAGtP/z7J9qGjhy6mb S5LegiTyqLPwGnp1OkSudJqTFA== X-Received: by 2002:a05:600c:1d09:b0:3ea:f882:bb40 with SMTP id l9-20020a05600c1d0900b003eaf882bb40mr1902858wms.17.1677231305771; Fri, 24 Feb 2023 01:35:05 -0800 (PST) Received: from vingu-book.. ([2a01:e0a:f:6020:a6f0:4ee9:c103:44cb]) by smtp.gmail.com with ESMTPSA id d18-20020a05600c34d200b003e6dcd562a6sm2239179wmq.28.2023.02.24.01.35.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 Feb 2023 01:35:05 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com, tj@kernel.org, lizefan.x@bytedance.com, hannes@cmpxchg.org, cgroups@vger.kernel.org, corbet@lwn.net, linux-doc@vger.kernel.org Cc: qyousef@layalina.io, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v12 6/8] sched/fair: Add sched group latency support Date: Fri, 24 Feb 2023 10:34:52 +0100 Message-Id: <20230224093454.956298-7-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230224093454.956298-1-vincent.guittot@linaro.org> References: <20230224093454.956298-1-vincent.guittot@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758704701257417978?= X-GMAIL-MSGID: =?utf-8?q?1758704701257417978?= Task can set its latency priority with sched_setattr(), which is then used to set the latency offset of its sched_enity, but sched group entities still have the default latency offset value. Add a latency.nice field in cpu cgroup controller to set the latency priority of the group similarly to sched_setattr(). The latency priority is then used to set the offset of the sched_entities of the group. Signed-off-by: Vincent Guittot Tested-by: K Prateek Nayak --- Documentation/admin-guide/cgroup-v2.rst | 10 ++++++++ kernel/sched/core.c | 30 +++++++++++++++++++++++ kernel/sched/fair.c | 32 +++++++++++++++++++++++++ kernel/sched/sched.h | 4 ++++ 4 files changed, 76 insertions(+) diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst index 1b3ed1c3b3f1..c08424593e4a 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -1121,6 +1121,16 @@ All time durations are in microseconds. values similar to the sched_setattr(2). This maximum utilization value is used to clamp the task specific maximum utilization clamp. + cpu.latency.nice + A read-write single value file which exists on non-root + cgroups. The default is "0". + + The nice value is in the range [-20, 19]. + + This interface file allows reading and setting latency using the + same values used by sched_setattr(2). The latency_nice of a group is + used to limit the impact of the latency_nice of a task outside the + group. Memory diff --git a/kernel/sched/core.c b/kernel/sched/core.c index d5b7e237d79b..093cc1af73dc 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -11059,6 +11059,25 @@ static int cpu_idle_write_s64(struct cgroup_subsys_state *css, { return sched_group_set_idle(css_tg(css), idle); } + +static s64 cpu_latency_nice_read_s64(struct cgroup_subsys_state *css, + struct cftype *cft) +{ + return LATENCY_TO_NICE(css_tg(css)->latency_prio); +} + +static int cpu_latency_nice_write_s64(struct cgroup_subsys_state *css, + struct cftype *cft, s64 nice) +{ + int prio; + + if (nice < MIN_LATENCY_NICE || nice > MAX_LATENCY_NICE) + return -ERANGE; + + prio = NICE_TO_LATENCY(nice); + + return sched_group_set_latency(css_tg(css), prio); +} #endif static struct cftype cpu_legacy_files[] = { @@ -11073,6 +11092,11 @@ static struct cftype cpu_legacy_files[] = { .read_s64 = cpu_idle_read_s64, .write_s64 = cpu_idle_write_s64, }, + { + .name = "latency.nice", + .read_s64 = cpu_latency_nice_read_s64, + .write_s64 = cpu_latency_nice_write_s64, + }, #endif #ifdef CONFIG_CFS_BANDWIDTH { @@ -11290,6 +11314,12 @@ static struct cftype cpu_files[] = { .read_s64 = cpu_idle_read_s64, .write_s64 = cpu_idle_write_s64, }, + { + .name = "latency.nice", + .flags = CFTYPE_NOT_ON_ROOT, + .read_s64 = cpu_latency_nice_read_s64, + .write_s64 = cpu_latency_nice_write_s64, + }, #endif #ifdef CONFIG_CFS_BANDWIDTH { diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 414b6243208b..dc7570f43ebe 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -12274,6 +12274,7 @@ int alloc_fair_sched_group(struct task_group *tg, struct task_group *parent) goto err; tg->shares = NICE_0_LOAD; + tg->latency_prio = DEFAULT_LATENCY_PRIO; init_cfs_bandwidth(tg_cfs_bandwidth(tg)); @@ -12372,6 +12373,9 @@ void init_tg_cfs_entry(struct task_group *tg, struct cfs_rq *cfs_rq, } se->my_q = cfs_rq; + + se->latency_offset = calc_latency_offset(tg->latency_prio); + /* guarantee group entities always have weight */ update_load_set(&se->load, NICE_0_LOAD); se->parent = parent; @@ -12502,6 +12506,34 @@ int sched_group_set_idle(struct task_group *tg, long idle) return 0; } +int sched_group_set_latency(struct task_group *tg, int prio) +{ + long latency_offset; + int i; + + if (tg == &root_task_group) + return -EINVAL; + + mutex_lock(&shares_mutex); + + if (tg->latency_prio == prio) { + mutex_unlock(&shares_mutex); + return 0; + } + + tg->latency_prio = prio; + latency_offset = calc_latency_offset(prio); + + for_each_possible_cpu(i) { + struct sched_entity *se = tg->se[i]; + + WRITE_ONCE(se->latency_offset, latency_offset); + } + + mutex_unlock(&shares_mutex); + return 0; +} + #else /* CONFIG_FAIR_GROUP_SCHED */ void free_fair_sched_group(struct task_group *tg) { } diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 3f42f86105d4..9a2e71231083 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -378,6 +378,8 @@ struct task_group { /* A positive value indicates that this is a SCHED_IDLE group. */ int idle; + /* latency priority of the group. */ + int latency_prio; #ifdef CONFIG_SMP /* @@ -488,6 +490,8 @@ extern int sched_group_set_shares(struct task_group *tg, unsigned long shares); extern int sched_group_set_idle(struct task_group *tg, long idle); +extern int sched_group_set_latency(struct task_group *tg, int prio); + #ifdef CONFIG_SMP extern void set_task_rq_fair(struct sched_entity *se, struct cfs_rq *prev, struct cfs_rq *next); From patchwork Fri Feb 24 09:34:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 61212 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:7b8e:b0:9f:8c76:fea4 with SMTP id j14csp1222953dyk; Fri, 24 Feb 2023 01:40:13 -0800 (PST) X-Google-Smtp-Source: AK7set+QsYxGQYZX3oaZpdN9o7qGGPoEw23U/aHhRB8SmxmiR7KzzN2QmlY9c+Bdy8SkvSpdNCMq X-Received: by 2002:a17:906:739d:b0:8b1:7b5c:bde7 with SMTP id f29-20020a170906739d00b008b17b5cbde7mr29479650ejl.77.1677231613692; Fri, 24 Feb 2023 01:40:13 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677231613; cv=none; d=google.com; s=arc-20160816; b=wNxIEzii6erGNSw9G3OMRK/6o6KEYV8MmXu5exasrnXRKaM9xFahO+8Jlhrp5e0SES CF1HdlRKw3H0evLKsJaVrXgobyCYZG8L/8fmiUf3YSVDh329VLUEzPacXVo6Y4ObGAhr Jqg1g7vfIgkbvHoznJ+suNxDTbue6jWl9xUP2Xo2BxrxIVlybl6BdyujZzzc1Y8Kkgg8 +x84orDalRAS6ZZm3NZ8TD1kkeyB5/d9kr0DAHv9C+0EFQo1LsO1Z9GDg3vGX0mJwQX8 DC9LnFMjsPUFfRHB1BHx0b6FJ25svprnIiXxUiot9OQTOVd5N/AVcaIErHpzdq2vezdN pRsQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=jFfq5teius6pAK1nmf1oSiYVaS3V8Wiqx9qF8KZNMCQ=; b=u7Qz4GD7An04KPfGGgMFwto82WnGcspvRmngi6gnDoLBDBz/PD8umrSeO8sEjTAKSD UWbT5DjpkUQ92Me1lF4Yyss0WZJT7yiQRAh/2w+6z1TeJFScPbPTDleZK+i3+VL4C2B/ LVZwmhajjnOkf7JCQFZXbNXB13kAscRBLUCVE1vXC65dKNtcBbzpW6kIn98AbJh4v7YH 6NfFPGQWN5Oc06PVdPnxVoiScmm/2a2vzmFoYqHU4LOF3dyri4QTzf5/JahQSPmz56pG Wo1vaUegXN0yPmIXmow1TBQ9t0ULTkKLnBTA+n+kcMWaTU1ejc1GUYZg7IxF8iqvv39F 0JZA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=UPtzcx8E; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id wj20-20020a170907051400b008bbcd4d9a0csi21723439ejb.858.2023.02.24.01.39.51; Fri, 24 Feb 2023 01:40:13 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=UPtzcx8E; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229784AbjBXJfa (ORCPT + 99 others); Fri, 24 Feb 2023 04:35:30 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46038 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229816AbjBXJfT (ORCPT ); Fri, 24 Feb 2023 04:35:19 -0500 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DE06565CF2 for ; Fri, 24 Feb 2023 01:35:07 -0800 (PST) Received: by mail-wm1-x32a.google.com with SMTP id az36so1396481wmb.1 for ; Fri, 24 Feb 2023 01:35:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jFfq5teius6pAK1nmf1oSiYVaS3V8Wiqx9qF8KZNMCQ=; b=UPtzcx8EFj1zlSxQ6lMU0mkanMgYYtKeaO2lWxoPNm90eUsDg5dZ1+YuhJFLZ9eA1C GwVX/XBU0wi8bbZqS4brymorHHXxUfabheM649o6i0SQ/aVibdS9bgWV9lK6GXWoyCxV DDBMfSwJFsDs+VxTsL6tGbeInl0ZqUq5wfYvFDMQ9DRbPWoCbA/6Cvincqhb8n8A/JMv yF2u9O9DHpm/uv9CcBKmrH4ca42lGrIqrseUJRNneFVCwHJafcuPzxjA18lvOlidR6dV 8ry4CjCYix+BYGvJUnF6KeUvdaZAfzJwDqJXYi1r+dRV7oZAoBl3+BLYwTygm+t5PEOr SgfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jFfq5teius6pAK1nmf1oSiYVaS3V8Wiqx9qF8KZNMCQ=; b=HqAXP07bCE0W6t8HOHH6n9Xq0Y3aj3Td6ofdZ1H+iXrxemNRnYBqx25oMs2OiyvdzX /Zt/6b99i08nUTrfhF0Ouq/90B7+3pY5Zmk2i1g5KBRI6rwFxucMN6gemzhujKkF6/2a xVYtBWmCBzkMItx+8/BEv1JiCuV8u9r3/M+A/4TNtQHYCoxdmz5DWNqoKPNDBYX7GPu7 3272jp8QH8aZkHFAn3UmIA4HiGBCsxe3xI/5qQKxHpdDtLqBAsZqlvMZvdxAPKJVkRWr I06hPSVvDoOXtHeZfEKYQMKEzSlJI9znOwr4TEotsljU00RUh3jFvX3F8zlAQFH2uxxY XIWA== X-Gm-Message-State: AO0yUKVfD7Kvh8MU0vi71wWi8Rn/w73a4LRjB2w2nIyEuPv4jWHcZC3X KdLhT+0TNHZWj0qoFhLVio2lQA== X-Received: by 2002:a05:600c:1da2:b0:3ea:ed4d:38fc with SMTP id p34-20020a05600c1da200b003eaed4d38fcmr2179606wms.31.1677231307168; Fri, 24 Feb 2023 01:35:07 -0800 (PST) Received: from vingu-book.. ([2a01:e0a:f:6020:a6f0:4ee9:c103:44cb]) by smtp.gmail.com with ESMTPSA id d18-20020a05600c34d200b003e6dcd562a6sm2239179wmq.28.2023.02.24.01.35.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 Feb 2023 01:35:06 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com, tj@kernel.org, lizefan.x@bytedance.com, hannes@cmpxchg.org, cgroups@vger.kernel.org, corbet@lwn.net, linux-doc@vger.kernel.org Cc: qyousef@layalina.io, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v12 7/8] sched/core: Support latency priority with sched core Date: Fri, 24 Feb 2023 10:34:53 +0100 Message-Id: <20230224093454.956298-8-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230224093454.956298-1-vincent.guittot@linaro.org> References: <20230224093454.956298-1-vincent.guittot@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758704816331691078?= X-GMAIL-MSGID: =?utf-8?q?1758704816331691078?= Take into account wakeup_latency_gran() when ordering the cfs threads. Signed-off-by: Vincent Guittot Tested-by: K Prateek Nayak --- kernel/sched/fair.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index dc7570f43ebe..125a6ff53378 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -11949,6 +11949,9 @@ bool cfs_prio_less(const struct task_struct *a, const struct task_struct *b, delta = (s64)(sea->vruntime - seb->vruntime) + (s64)(cfs_rqb->min_vruntime_fi - cfs_rqa->min_vruntime_fi); + /* Take into account latency offset */ + delta -= wakeup_latency_gran(sea, seb); + return delta > 0; } #else From patchwork Fri Feb 24 09:34:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 61211 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:7b8e:b0:9f:8c76:fea4 with SMTP id j14csp1222783dyk; Fri, 24 Feb 2023 01:39:51 -0800 (PST) X-Google-Smtp-Source: AK7set+WycPYyYYkhISslaUj3peUwX9KfzzQgbi/4FvfwMoKmooxMWQ7POIZaZSVQ2F2Ro0PMRDi X-Received: by 2002:a17:907:980e:b0:8ae:ea17:bae4 with SMTP id ji14-20020a170907980e00b008aeea17bae4mr23047601ejc.30.1677231591156; Fri, 24 Feb 2023 01:39:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677231591; cv=none; d=google.com; s=arc-20160816; b=vEA1RfZvJhJZVRsi4ErF39Xmt/3lwTaxTJARAzJkd1cBzOgPokA5QdvSMe5XyKtTyZ oLU31vyWjkzy6eVNwSit7ZdLdCRJRgVbZmgeiIRpYvILtf6/svIDIn58nHHV2ETH6TUK 5tMMDzVtAa+oY2H+iYPSeHXDt5Os4REehJA4vZBsyMI3NQqyot7NespTd5U6AFezfT0o 0A4Gd4VlIrDIR+/bhl2c2bdzwCztXHepG2+17yTtgCC3gq/cD4F2/yqMufp8uaGFGU9c DCZWfPuKDnRbfUKwOOdczWEcyFC8GH55uHFp3XwLhz46/3843A4MkQARK6DBp+ILX9R+ w5pw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=3qtQ6Hb1NcMmUdmeDB55iw9V1Tm3Jd0QSWgTcrurHX8=; b=GoFTC0BiJKYZDTuN/Q7duEepby4SQpMwai5I4wznS05w0sRHRLx2RjHmf1WFpHdeTy VctsWdPS2ltYSNdF7cLEqORTYq540HqX96pG9ikB0w7Nhp/06f5R8BpxlRXulKtqQ7Y6 UpT8O1bivoOmBO7RxYuLkq8qbURkwCgNf6Z9PZmJsv+eruNkhoZuWVqgusG1TfTQGEyX txzx7Szva/qBPSTofC4gmbjzpZVBgWOquI/ozIm6LNN2Haj55S74PUgtAfNVJGi4R65L N2/AsWg906HFm0m4AZL49CruFjfWk3JD7a6FZ0BTr7nkpSIw0gOEeF06XOqpXG8FIUQA PbSA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=IaqygHpr; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id r3-20020aa7cfc3000000b004acbbc70421si14528823edy.366.2023.02.24.01.39.28; Fri, 24 Feb 2023 01:39:51 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=IaqygHpr; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229927AbjBXJfh (ORCPT + 99 others); Fri, 24 Feb 2023 04:35:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45356 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229690AbjBXJfU (ORCPT ); Fri, 24 Feb 2023 04:35:20 -0500 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 14F1965318 for ; Fri, 24 Feb 2023 01:35:09 -0800 (PST) Received: by mail-wm1-x334.google.com with SMTP id p16so8758826wmq.5 for ; Fri, 24 Feb 2023 01:35:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3qtQ6Hb1NcMmUdmeDB55iw9V1Tm3Jd0QSWgTcrurHX8=; b=IaqygHprsEYiu7vlcVABSQcoFLIzT2pf2we+OcE8esJq5iLpeExRK3wGg7dypYfAoo I+DLxUpNQVLlr01nJbxYBl13k+C0nd6WC10n7614S5U3hCry/AXb7CUKiv+wNTjNyNCz fZ8mzLBKVmQnyRwyL9rGzexQhBHHVhGGvuAPbpM0MVst1ve+yFdCBwVtlK8dZVF46yne rWGyCVlw6lGHGBxeugK3JiMo3IKWfEspcAk6sqEjop7ObswC83ot9Ep1t73Pb1nGD/9p Sq6EusFG5WzOglWvwvMAkidavnmT5fylR0EHcwtMGYCLG5hgQub1Y3adTjBGoId2FEmy 6/Og== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3qtQ6Hb1NcMmUdmeDB55iw9V1Tm3Jd0QSWgTcrurHX8=; b=vkI3H4W3Rp4vk00ScCP60G0LJDwOkdYI5THbsizHDIOyOGU8/MEExHKBzF6iN2S60X /kn2Gw0PfSOUZPZNALeEUWiXAhE4dfDqCYUFt4bBRSWZowwmbfntShScBKtgbjY05ZhP nh8OxCNRtqHy8G0zuM5Kf5qiQniKtm8m2H5XRK7OGIZhwK+DS/H1D4guJSW6mHUMQOEV XeXBdCFda5QKVdwiaamBFrtOOcOmG9JL/sUCXuRbxM4ciAiJaeL5/2SD4WtpJS5pLyLZ gJZ9ZqUOxQPGyc9t6ZmPdIE95IK2J1RzcvMLzPjQ9ZW29/ODea6vsjgruIIQgmaoM5yA 2STQ== X-Gm-Message-State: AO0yUKWZKBZ1/ONPWzDHUUgBYuHpE84DD737boyzFBScYq+gMJpKT5dZ i2gUGJI24JiKjYulQnp0ERhkig== X-Received: by 2002:a05:600c:198e:b0:3e2:1f00:bff7 with SMTP id t14-20020a05600c198e00b003e21f00bff7mr13128798wmq.12.1677231308585; Fri, 24 Feb 2023 01:35:08 -0800 (PST) Received: from vingu-book.. ([2a01:e0a:f:6020:a6f0:4ee9:c103:44cb]) by smtp.gmail.com with ESMTPSA id d18-20020a05600c34d200b003e6dcd562a6sm2239179wmq.28.2023.02.24.01.35.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 Feb 2023 01:35:08 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com, tj@kernel.org, lizefan.x@bytedance.com, hannes@cmpxchg.org, cgroups@vger.kernel.org, corbet@lwn.net, linux-doc@vger.kernel.org Cc: qyousef@layalina.io, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v12 8/8] sched/fair: Add latency list Date: Fri, 24 Feb 2023 10:34:54 +0100 Message-Id: <20230224093454.956298-9-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230224093454.956298-1-vincent.guittot@linaro.org> References: <20230224093454.956298-1-vincent.guittot@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758704792722058381?= X-GMAIL-MSGID: =?utf-8?q?1758704792722058381?= Add a rb tree for latency sensitive entities so we can schedule the most sensitive one first even when it failed to preempt current at wakeup or when it got quickly preempted by another entity of higher priority. In order to keep fairness, the latency is used once at wakeup to get a minimum slice and not during the following scheduling slice to prevent long running entity to got more running time than allocated to his nice priority. The rb tree enables to cover the last corner case where latency sensitive entity can't got schedule quickly after the wakeup. Signed-off-by: Vincent Guittot Tested-by: K Prateek Nayak --- include/linux/sched.h | 1 + kernel/sched/core.c | 1 + kernel/sched/fair.c | 109 ++++++++++++++++++++++++++++++++++++++++-- kernel/sched/sched.h | 1 + 4 files changed, 109 insertions(+), 3 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 38decae3e156..41bb92be5ecc 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -548,6 +548,7 @@ struct sched_entity { /* For load-balancing: */ struct load_weight load; struct rb_node run_node; + struct rb_node latency_node; struct list_head group_node; unsigned int on_rq; diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 093cc1af73dc..752fd364216c 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4434,6 +4434,7 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p) p->se.nr_migrations = 0; p->se.vruntime = 0; INIT_LIST_HEAD(&p->se.group_node); + RB_CLEAR_NODE(&p->se.latency_node); #ifdef CONFIG_FAIR_GROUP_SCHED p->se.cfs_rq = NULL; diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 125a6ff53378..e2aeb4511686 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -680,7 +680,85 @@ struct sched_entity *__pick_last_entity(struct cfs_rq *cfs_rq) return __node_2_se(last); } +#endif +/************************************************************** + * Scheduling class tree data structure manipulation methods: + * for latency + */ + +static inline bool latency_before(struct sched_entity *a, + struct sched_entity *b) +{ + return (s64)(a->vruntime + a->latency_offset - b->vruntime - b->latency_offset) < 0; +} + +#define __latency_node_2_se(node) \ + rb_entry((node), struct sched_entity, latency_node) + +static inline bool __latency_less(struct rb_node *a, const struct rb_node *b) +{ + return latency_before(__latency_node_2_se(a), __latency_node_2_se(b)); +} + +/* + * Enqueue an entity into the latency rb-tree: + */ +static void __enqueue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) +{ + + /* Only latency sensitive entity can be added to the list */ + if (se->latency_offset >= 0) + return; + + if (!RB_EMPTY_NODE(&se->latency_node)) + return; + + /* + * The entity is always added the latency list at wakeup. + * Then, a not waking up entity that is put back in the list after an + * execution time less than sysctl_sched_min_granularity, means that + * the entity has been preempted by a higher sched class or an entity + * with higher latency constraint. In thi case, the entity is also put + * back in the latency list so it gets a chance to run 1st during the + * next slice. + */ + if (!(flags & ENQUEUE_WAKEUP)) { + u64 delta_exec = se->sum_exec_runtime - se->prev_sum_exec_runtime; + + if (delta_exec >= sysctl_sched_min_granularity) + return; + } + + rb_add_cached(&se->latency_node, &cfs_rq->latency_timeline, __latency_less); +} + +/* + * Dequeue an entity from the latency rb-tree and return true if it was really + * part of the rb-tree: + */ +static bool __dequeue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se) +{ + if (!RB_EMPTY_NODE(&se->latency_node)) { + rb_erase_cached(&se->latency_node, &cfs_rq->latency_timeline); + RB_CLEAR_NODE(&se->latency_node); + return true; + } + + return false; +} + +static struct sched_entity *__pick_first_latency(struct cfs_rq *cfs_rq) +{ + struct rb_node *left = rb_first_cached(&cfs_rq->latency_timeline); + + if (!left) + return NULL; + + return __latency_node_2_se(left); +} + +#ifdef CONFIG_SCHED_DEBUG /************************************************************** * Scheduling class statistics methods: */ @@ -4758,8 +4836,10 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) check_schedstat_required(); update_stats_enqueue_fair(cfs_rq, se, flags); check_spread(cfs_rq, se); - if (!curr) + if (!curr) { __enqueue_entity(cfs_rq, se); + __enqueue_latency(cfs_rq, se, flags); + } se->on_rq = 1; if (cfs_rq->nr_running == 1) { @@ -4845,8 +4925,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) clear_buddies(cfs_rq, se); - if (se != cfs_rq->curr) + if (se != cfs_rq->curr) { __dequeue_entity(cfs_rq, se); + __dequeue_latency(cfs_rq, se); + } se->on_rq = 0; account_entity_dequeue(cfs_rq, se); @@ -4941,6 +5023,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se) */ update_stats_wait_end_fair(cfs_rq, se); __dequeue_entity(cfs_rq, se); + __dequeue_latency(cfs_rq, se); update_load_avg(cfs_rq, se, UPDATE_TG); } @@ -4979,7 +5062,7 @@ static struct sched_entity * pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr) { struct sched_entity *left = __pick_first_entity(cfs_rq); - struct sched_entity *se; + struct sched_entity *latency, *se; /* * If curr is set we have to see if its left of the leftmost entity @@ -5021,6 +5104,12 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr) se = cfs_rq->last; } + /* Check for latency sensitive entity waiting for running */ + latency = __pick_first_latency(cfs_rq); + if (latency && (latency != se) && + wakeup_preempt_entity(latency, se) < 1) + se = latency; + return se; } @@ -5044,6 +5133,7 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev) update_stats_wait_start_fair(cfs_rq, prev); /* Put 'current' back into the tree. */ __enqueue_entity(cfs_rq, prev); + __enqueue_latency(cfs_rq, prev, 0); /* in !on_rq case, update occurred at dequeue */ update_load_avg(cfs_rq, prev, 0); } @@ -12222,6 +12312,7 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first) void init_cfs_rq(struct cfs_rq *cfs_rq) { cfs_rq->tasks_timeline = RB_ROOT_CACHED; + cfs_rq->latency_timeline = RB_ROOT_CACHED; u64_u32_store(cfs_rq->min_vruntime, (u64)(-(1LL << 20))); #ifdef CONFIG_SMP raw_spin_lock_init(&cfs_rq->removed.lock); @@ -12378,6 +12469,7 @@ void init_tg_cfs_entry(struct task_group *tg, struct cfs_rq *cfs_rq, se->my_q = cfs_rq; se->latency_offset = calc_latency_offset(tg->latency_prio); + RB_CLEAR_NODE(&se->latency_node); /* guarantee group entities always have weight */ update_load_set(&se->load, NICE_0_LOAD); @@ -12529,8 +12621,19 @@ int sched_group_set_latency(struct task_group *tg, int prio) for_each_possible_cpu(i) { struct sched_entity *se = tg->se[i]; + struct rq *rq = cpu_rq(i); + struct rq_flags rf; + bool queued; + + rq_lock_irqsave(rq, &rf); + queued = __dequeue_latency(se->cfs_rq, se); WRITE_ONCE(se->latency_offset, latency_offset); + if (queued) + __enqueue_latency(se->cfs_rq, se, ENQUEUE_WAKEUP); + + + rq_unlock_irqrestore(rq, &rf); } mutex_unlock(&shares_mutex); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 9a2e71231083..21dd309e98a9 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -570,6 +570,7 @@ struct cfs_rq { #endif struct rb_root_cached tasks_timeline; + struct rb_root_cached latency_timeline; /* * 'curr' points to currently running entity on this cfs_rq.