From patchwork Thu Nov 10 17:50:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 18245 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp285674wru; Thu, 10 Nov 2022 09:51:08 -0800 (PST) X-Google-Smtp-Source: AMsMyM6bFeQEf8abe0AWInk/u0SnrajetoXtSo1zWf96kBFK9GtuIZpNiVeTFLRrNLJ5zn/i55cL X-Received: by 2002:a17:902:f28b:b0:186:b069:63fc with SMTP id k11-20020a170902f28b00b00186b06963fcmr66846516plc.38.1668102667716; Thu, 10 Nov 2022 09:51:07 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668102667; cv=none; d=google.com; s=arc-20160816; b=u1fEYsU/huUksDLgm5fNOVe9ddLMoMrFp7wLeUPE1kv9D9DFq9jkb6LZ2pDTicELTQ d7R7x4SnuV1uDfaG0vWDgmHUuw1/0R5bn90u/2X8rR+HoPvUl9W0xKRINK5yXw77dZSf RjVbEblh9A9/FH40VvENzah4EGw2yaTvFChgWLgwro25I9ldNPLrEh3mC1powltgoAEK DSThoPopHCrJo9VDdKTJWPMYnYSuo9FTBemvcanV5opGU+wJmoQJEj+8/iII4tsdDRun Ov7t1SQ88uukSSM/w+kGg/+/9ci1VVSnCwBKyBQ8Nb1P4TAomE+W4gBhey0a1q0XIVnl R2TQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=FulDXK+MjqUjf+jM0x0PxczJh4rRajiCQKKkLNt7L9g=; b=uO3dnbyO9dcPJnJrmMCuTUKdqQZozUqKVu0VJDVCaX9v7WZ9RuIQImBwY+mya7lZ17 nu2ybwrleE3L3VHBUqqZdE3G08IRcRTaud8rUtO+7AmR3IcvGDjdcQWw+J8RppGLY1aJ RnOjQFZVpmtrj4uqz88cxlASqnngS9Ey+BRuc0YvRAM2RcGxW4wviMo7akMxfNFGySYJ 8LRj9SqKfqWYRGMHIRAORcRRY/rJcNJlmkBUeI5CZonbGXiSiCg3G9CXUhTQ087/Yf/v na/c6P1Uqsb59YtF6J5QMHj0w9RT6aX3evGyxYZUFXK6u+ewrJZVZ+ZP67u31N//tfgS KmLg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=uqF7wYco; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h24-20020a63c018000000b004639c772888si18630488pgg.225.2022.11.10.09.50.54; Thu, 10 Nov 2022 09:51:07 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=uqF7wYco; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231356AbiKJRuZ (ORCPT + 99 others); Thu, 10 Nov 2022 12:50:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33768 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231270AbiKJRuT (ORCPT ); Thu, 10 Nov 2022 12:50:19 -0500 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9C2D83C6E4 for ; Thu, 10 Nov 2022 09:50:18 -0800 (PST) Received: by mail-wr1-x42a.google.com with SMTP id j15so3386552wrq.3 for ; Thu, 10 Nov 2022 09:50:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=FulDXK+MjqUjf+jM0x0PxczJh4rRajiCQKKkLNt7L9g=; b=uqF7wYcoS1Vb/LeU+zmM+bWQUS+qz9nsrF2qaBasvtiFv8WBliA8otbDoW4M5B9cwq WyYJBfZgarq006JA96KlmAOBsILI2m/yIIl9TH5XlQZlQhqbt6L0C48uOa/lQwUqlb0e dUEkyyfWPCWzb+C/qL/jJNhesEP81sWPU+S8IemO62dyKhLOadbQ0Yr2ot3jnaHMpT4w HHv8T5aKurGZqFSIB3DDJJ26EOYDze3hRakZOMfuwqBmxYDsyNZ/qwWF//7Iw3R5Vifz ScMvIIPk2DaRqHmlyUGZPSx/MYUMdWgrq368eMR1B9K7LSAzkpWfxKENO404JPYFVvPJ 8uRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=FulDXK+MjqUjf+jM0x0PxczJh4rRajiCQKKkLNt7L9g=; b=Vct2NnNgKo7IMqunMAQ8X5FxXGZ3mD+GfUeQ31rvdsllvTjVB6tHUZqSSi4FSG9TRK ZF3JhQFkKkYHkiYXYqQwxyd+iXK5Q2hNBtaLmkRoPYANUNQFz5aI+wSnilQepzzxosfM ywAFTD9RYv4+Fed54bQcsJTC6SqeX4Py+QMxZrBaTmk00kotY9bheSvmZa/CK9oBqc4H mBU7OXUdoK1ET16C1P7oF00Bv9jK7WITC1iVzhiriKxgOeIGK8Zust+uduq79fd+oujA ZJNGdLisbU0aq1nE8+J0lFO9S4v6deT2D2VASw6JvIfRzPfaArQBFxvY44gBOndgxewB 77mA== X-Gm-Message-State: ACrzQf1RyCenywJyHYnGYWiIerlTH/KEqzjYDR33zD80yvhWLY1Gr4aM e/w0TY7kbz2d7AKzMqdeq077CA== X-Received: by 2002:adf:e104:0:b0:236:dce2:35dc with SMTP id t4-20020adfe104000000b00236dce235dcmr35917753wrz.675.1668102617044; Thu, 10 Nov 2022 09:50:17 -0800 (PST) Received: from localhost.localdomain ([2a01:e0a:f:6020:342a:468f:562a:9cc4]) by smtp.gmail.com with ESMTPSA id f6-20020a05600c4e8600b003cfc02ab8basm6514677wmq.33.2022.11.10.09.50.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Nov 2022 09:50:15 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com Cc: qyousef@layalina.io, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v8 1/9] sched/fair: fix unfairness at wakeup Date: Thu, 10 Nov 2022 18:50:01 +0100 Message-Id: <20221110175009.18458-2-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221110175009.18458-1-vincent.guittot@linaro.org> References: <20221110175009.18458-1-vincent.guittot@linaro.org> X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DIET_1,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749132422442660334?= X-GMAIL-MSGID: =?utf-8?q?1749132422442660334?= At wake up, the vruntime of a task is updated to not be more older than a sched_latency period behind the min_vruntime. This prevents long sleeping task to get unlimited credit at wakeup. Such waking task should preempt current one to use its CPU bandwidth but wakeup_gran() can be larger than sched_latency, filter out the wakeup preemption and as a results steals some CPU bandwidth to the waking task. Make sure that a task, which vruntime has been capped, will preempt current task and use its CPU bandwidth even if wakeup_gran() is in the same range as sched_latency. If the waking task failed to preempt current it could to wait up to sysctl_sched_min_granularity before preempting it during next tick. Strictly speaking, we should use cfs->min_vruntime instead of curr->vruntime but it doesn't worth the additional overhead and complexity as the vruntime of current should be close to min_vruntime if not equal. Signed-off-by: Vincent Guittot Reported-by: Youssef Esmat Reviewed-by: Joel Fernandes (Google) --- kernel/sched/fair.c | 46 ++++++++++++++++++++------------------------ kernel/sched/sched.h | 30 ++++++++++++++++++++++++++++- 2 files changed, 50 insertions(+), 26 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 5ffec4370602..eb04c83112a0 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4345,33 +4345,17 @@ place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int initial) { u64 vruntime = cfs_rq->min_vruntime; - /* - * The 'current' period is already promised to the current tasks, - * however the extra weight of the new task will slow them down a - * little, place the new task so that it fits in the slot that - * stays open at the end. - */ - if (initial && sched_feat(START_DEBIT)) - vruntime += sched_vslice(cfs_rq, se); - - /* sleeps up to a single latency don't count. */ - if (!initial) { - unsigned long thresh; - - if (se_is_idle(se)) - thresh = sysctl_sched_min_granularity; - else - thresh = sysctl_sched_latency; - + if (!initial) + /* sleeps up to a single latency don't count. */ + vruntime -= get_sched_latency(se_is_idle(se)); + else if (sched_feat(START_DEBIT)) /* - * Halve their sleep time's effect, to allow - * for a gentler effect of sleepers: + * The 'current' period is already promised to the current tasks, + * however the extra weight of the new task will slow them down a + * little, place the new task so that it fits in the slot that + * stays open at the end. */ - if (sched_feat(GENTLE_FAIR_SLEEPERS)) - thresh >>= 1; - - vruntime -= thresh; - } + vruntime += sched_vslice(cfs_rq, se); /* ensure we never gain time by being placed backwards. */ se->vruntime = max_vruntime(se->vruntime, vruntime); @@ -7187,6 +7171,18 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se) return -1; gran = wakeup_gran(se); + + /* + * At wake up, the vruntime of a task is capped to not be older than + * a sched_latency period compared to min_vruntime. This prevents long + * sleeping task to get unlimited credit at wakeup. Such waking up task + * has to preempt current in order to not lose its share of CPU + * bandwidth but wakeup_gran() can become higher than scheduling period + * for low priority task. Make sure that long sleeping task will get a + * chance to preempt current. + */ + gran = min_t(s64, gran, get_latency_max()); + if (vdiff > gran) return 1; diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 1fc198be1ffd..14879d429919 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2432,9 +2432,9 @@ extern void check_preempt_curr(struct rq *rq, struct task_struct *p, int flags); extern const_debug unsigned int sysctl_sched_nr_migrate; extern const_debug unsigned int sysctl_sched_migration_cost; -#ifdef CONFIG_SCHED_DEBUG extern unsigned int sysctl_sched_latency; extern unsigned int sysctl_sched_min_granularity; +#ifdef CONFIG_SCHED_DEBUG extern unsigned int sysctl_sched_idle_min_granularity; extern unsigned int sysctl_sched_wakeup_granularity; extern int sysctl_resched_latency_warn_ms; @@ -2448,6 +2448,34 @@ extern unsigned int sysctl_numa_balancing_scan_period_max; extern unsigned int sysctl_numa_balancing_scan_size; #endif +static inline unsigned long get_sched_latency(bool idle) +{ + unsigned long thresh; + + if (idle) + thresh = sysctl_sched_min_granularity; + else + thresh = sysctl_sched_latency; + + /* + * Halve their sleep time's effect, to allow + * for a gentler effect of sleepers: + */ + if (sched_feat(GENTLE_FAIR_SLEEPERS)) + thresh >>= 1; + + return thresh; +} + +static inline unsigned long get_latency_max(void) +{ + unsigned long thresh = get_sched_latency(false); + + thresh -= sysctl_sched_min_granularity; + + return thresh; +} + #ifdef CONFIG_SCHED_HRTICK /* From patchwork Thu Nov 10 17:50:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 18246 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp285715wru; Thu, 10 Nov 2022 09:51:14 -0800 (PST) X-Google-Smtp-Source: AMsMyM72vYyu49Kx9OoHsof4F0KtYPd3EKn4x7yzu1fNEjKTiWlH4e7teJze3M+GKHy7MFlB6l5d X-Received: by 2002:a17:902:db12:b0:187:2596:8b0c with SMTP id m18-20020a170902db1200b0018725968b0cmr54888951plx.172.1668102673964; Thu, 10 Nov 2022 09:51:13 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668102673; cv=none; d=google.com; s=arc-20160816; b=dCEgsuOpgZZCgrPumTas/eYojnwnjEqnvQMlMBHoejHVFPWVgEa+3ibXZXvi9tZFkX gDNtxQiYLGAluEJu4zG2o24F1s7nOBY7buJMz+wJUUKGVlxZz1b6SoOySGCZfL+QkCch wb8gjlcNVdE5XSffipMextDyBBcfiY7sgUbzRuUhRLkHvr4QxSdCBLVMJ8cmjgyAK1kr eR12QB2jrsxXO29qPFD/+bGjz5igU426DBhxwyY2PdHrkvvkQebBDdMWDLadkPTgTBT5 w+E5oW0TRWKxb2ZxsFBueMnf/SqAq6GADVh8NyBlTohdND6MMLmb0wmI09k+yQ5Un+5/ //uQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=kTjYYbmOvEwchQQqZbLCLNfz1JMqxHsTlxBK50VB9/E=; b=ieltvfGxZAQwvYTp/jsq//zKoDpvLPS0tPRAe+KmO5q+OnqNSBQCxOeUH/4dlh9C5o G8no1iyemMrGZ3JvpR6FjzGF8j9XwNuvMGXTCp9gLfjD9uokE3rUhdvpy+hbd+HcE+qe AA07L20mucqRzrVmTa8G8eU4chIBD+4o6ZRaVpc9wwSrS7BgeKe6b/prgav6OOBjv4XF KjifUkyN5SevHmv3a+RKwT3v8pkSDnZVaH261R/4eagWrYbb8lY1UTKAVHRmKm/v93ms hSMd19DxlMpMW+a3yjSiw8mn0eLNhFEbFZpeVFjG1P2Wo6rXeLlpxSQoRfe3a2kIxXit 1nYg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=qH89BUBg; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id r8-20020a632048000000b004347b49d045si19840084pgm.631.2022.11.10.09.51.00; Thu, 10 Nov 2022 09:51:13 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=qH89BUBg; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231368AbiKJRua (ORCPT + 99 others); Thu, 10 Nov 2022 12:50:30 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33906 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231305AbiKJRuX (ORCPT ); Thu, 10 Nov 2022 12:50:23 -0500 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4AC04303FF for ; Thu, 10 Nov 2022 09:50:21 -0800 (PST) Received: by mail-wr1-x430.google.com with SMTP id k8so3390506wrh.1 for ; Thu, 10 Nov 2022 09:50:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=kTjYYbmOvEwchQQqZbLCLNfz1JMqxHsTlxBK50VB9/E=; b=qH89BUBgJl4VFhynofvsQ2sQni4rCZBBOn8OKk2XdXbz/RsioKXkLySMkmDMKVgDyj RU5rLwRY2brdjghKJ/NkmM7ayZBNvmf8ZCEHLj7DcLLZb4815z1dXSo1xxpXpsKGuIVJ owqapyZC2XspP1M2u38vfzoLOzTUTeOjbfzR0kxls4ZEBD+OL3dzweM2sRzLF6O1Ihx0 csjk5jKZW+TnCPTpATOo5/PSzrBMn7GKELzlUsI/QGudky0PDAh0bpGP9TfN5fivqI5t Em0u87DGeW0ouPlOAsYOTFKxK+O/9cS1i41AL2EbyFg1UYfQ3wueMGD1Y17NZgap7Yb6 MDTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kTjYYbmOvEwchQQqZbLCLNfz1JMqxHsTlxBK50VB9/E=; b=LcyJwgUa0BiELHzdnpzp5zH0Nnk1kp/IqI+NAwCI+CS9tg5rmL0rPlJ6XFphfPXhbw YCZDaMRrRvTRzcyxIRcaKjQNJXjN1hoUvLMsMoNys+2bGxg9qr2qcVca9UeHBiPDGmyz 49Ed2wcU6HzKhC4KHjaEjkVssVo/sYGcIw4mb1iMqFyMYKgn6uVPs5FCxxh7+/WHbQ+s /3/WLkNn/Gr4f/bcIaynC6rennmuhm/9z9uypN/8sLt0A5ubGeSNSnPqJKdIFz8hQbsw JqdEkJVCQ+GhBPi5vLy+8CYsiJmtncP4z5/wAUcQkNi7MbP7NYKvZYUvlWBQdjZNM5XQ h6fw== X-Gm-Message-State: ACrzQf26CgsCvZwpespXzmK5OKq9Yz12+6TEYJ4fRWVUhUD+S4CGy4jz LzGwI0dsl94rddbAHXEnQGs6yw== X-Received: by 2002:adf:d226:0:b0:235:d9ae:1de9 with SMTP id k6-20020adfd226000000b00235d9ae1de9mr43167752wrh.599.1668102619818; Thu, 10 Nov 2022 09:50:19 -0800 (PST) Received: from localhost.localdomain ([2a01:e0a:f:6020:342a:468f:562a:9cc4]) by smtp.gmail.com with ESMTPSA id f6-20020a05600c4e8600b003cfc02ab8basm6514677wmq.33.2022.11.10.09.50.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Nov 2022 09:50:18 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com Cc: qyousef@layalina.io, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v8 2/9] sched: Introduce latency-nice as a per-task attribute Date: Thu, 10 Nov 2022 18:50:02 +0100 Message-Id: <20221110175009.18458-3-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221110175009.18458-1-vincent.guittot@linaro.org> References: <20221110175009.18458-1-vincent.guittot@linaro.org> X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749132429594447254?= X-GMAIL-MSGID: =?utf-8?q?1749132429594447254?= From: Parth Shah Latency-nice indicates the latency requirements of a task with respect to the other tasks in the system. The value of the attribute can be within the range of [-20, 19] both inclusive to be in-line with the values just like task nice values. latency_nice = -20 indicates the task to have the least latency as compared to the tasks having latency_nice = +19. The latency_nice may affect only the CFS SCHED_CLASS by getting latency requirements from the userspace. Additionally, add debugging bits for newly added latency_nice attribute. Signed-off-by: Parth Shah [rebase] Signed-off-by: Vincent Guittot --- include/linux/sched.h | 1 + kernel/sched/debug.c | 1 + kernel/sched/sched.h | 18 ++++++++++++++++++ 3 files changed, 20 insertions(+) diff --git a/include/linux/sched.h b/include/linux/sched.h index 15e3bd96e4ce..6805f378a9c3 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -783,6 +783,7 @@ struct task_struct { int static_prio; int normal_prio; unsigned int rt_priority; + int latency_nice; struct sched_entity se; struct sched_rt_entity rt; diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c index bb3d63bdf4ae..a3f7876217a6 100644 --- a/kernel/sched/debug.c +++ b/kernel/sched/debug.c @@ -1042,6 +1042,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns, #endif P(policy); P(prio); + P(latency_nice); if (task_has_dl_policy(p)) { P(dl.runtime); P(dl.deadline); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 14879d429919..4bf9d7777f99 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -125,6 +125,24 @@ extern int sched_rr_timeslice; */ #define NS_TO_JIFFIES(TIME) ((unsigned long)(TIME) / (NSEC_PER_SEC / HZ)) +/* + * Latency nice is meant to provide scheduler hints about the relative + * latency requirements of a task with respect to other tasks. + * Thus a task with latency_nice == 19 can be hinted as the task with no + * latency requirements, in contrast to the task with latency_nice == -20 + * which should be given priority in terms of lower latency. + */ +#define MAX_LATENCY_NICE 19 +#define MIN_LATENCY_NICE -20 + +#define LATENCY_NICE_WIDTH \ + (MAX_LATENCY_NICE - MIN_LATENCY_NICE + 1) + +/* + * Default tasks should be treated as a task with latency_nice = 0. + */ +#define DEFAULT_LATENCY_NICE 0 + /* * Increase resolution of nice-level calculations for 64-bit architectures. * The extra resolution improves shares distribution and load balancing of From patchwork Thu Nov 10 17:50:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 18248 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp285764wru; Thu, 10 Nov 2022 09:51:22 -0800 (PST) X-Google-Smtp-Source: AMsMyM4U8N48kVT6UcAYhM2pf1PmSMhqqzlkPh/Jz1Jy6yER3zyG325IN5QJ9K/yRMEIgpIHo+ug X-Received: by 2002:a17:902:f612:b0:186:6c10:7260 with SMTP id n18-20020a170902f61200b001866c107260mr1584592plg.26.1668102682717; Thu, 10 Nov 2022 09:51:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668102682; cv=none; d=google.com; s=arc-20160816; b=oR5AWAFi6lK46idkTo/0rNezDyc2msijK104mjDNW0Vfs+MzQEdhLd7K0zl6lE2zpF SOViY/FUfHiJ63T9OFeHYOonKRXbMafg4h6nQpHGOapqXKDEXsKxT8ZQ7muMIYtpDsPj MJwgRzyeZOpDFc0cWc9hYoPyS5LRzgHwGPStVg02gY62YmwdhMmaCcrTAKPkqk6dQ2W+ +PJ5RC4RDSKZljEiNIEJPUfGsSOaOwjuJd5XkBt1yZrjcJo5MXTrGQUCmCE6ykgmbUvS +cbb+vAk1sHPEca7hg4+cblEF7OL0EpSZUbT/taM+fLVcjMkQWnwyR0EgrqeKaDfxtGr 2DFw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=ySpaS8Fkj5Miw0XHBXwZ3O8DcUQ428wDyrUI4Txxz7I=; b=alPiLL2ER6P1a0u6subCKensTuB2BIy1P5t5P8oTjsYsKlqEDZ5GMeDGnWDpm0EIwZ y8m23NQlumI6Y3+1DzuLqic4y2HZmdWCD1hCgGQIvNmIAlMhMCybkPgOt5JqDQ0wsN0p J7ggFkF+LRlyVXEuTxBFDXE61iaRpWGeYB3AXp4BOjMbg7+6P1XgIWl5BI64Tkzn4oj9 fpJgYrzcp71ImTnaAzBN5WHskZMetjTM69gjdeir5/OYTa1TYJJf6Y6+1zd7hRbqaW/Z f6xz1znY/GxT9OwgkC2puThR1eP+dpnM3iG8H++mSi0e4Lbx+vS6aysrs5gF7AeF8Qfu ji6w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=hf1Bmm3q; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a20-20020a170902b59400b0017f8c07eaf8si17369092pls.44.2022.11.10.09.51.08; Thu, 10 Nov 2022 09:51:22 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=hf1Bmm3q; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231300AbiKJRul (ORCPT + 99 others); Thu, 10 Nov 2022 12:50:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33904 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231342AbiKJRuZ (ORCPT ); Thu, 10 Nov 2022 12:50:25 -0500 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 487222FC30 for ; Thu, 10 Nov 2022 09:50:23 -0800 (PST) Received: by mail-wm1-x32a.google.com with SMTP id p13-20020a05600c468d00b003cf8859ed1bso1657313wmo.1 for ; Thu, 10 Nov 2022 09:50:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=ySpaS8Fkj5Miw0XHBXwZ3O8DcUQ428wDyrUI4Txxz7I=; b=hf1Bmm3qTQNsSIOlKLTd6ZGi0sOW06EIWnJhpFoBtnO4ELVAcPyLDmB/C+/Kenmi+O /a5SepUDLDQT2hP91MYsYZqSRuLuGrZtsGKJdsOQ5etk4yXI/epOeIVHbWRVOLlbDCRP gyVszqcKDTWCGw3Ylbn/k6Uua0S2q/81i5sd9xTvZ+OKl1UtNTDTI5NjaQ4Vwzq2QClT YGF7pfamPZ0CgQ9nr1d1lXDQwtLCuDQfLyW+urIORrM9MxNrnhJgdY6wyuJUn07CyZrF qnKVR0a3ULjrVM1pU3jdXZRCArdlJfLLFnFP3aeHdTfjV/5QEvArJXzAXUBDguTmKNq9 YFlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ySpaS8Fkj5Miw0XHBXwZ3O8DcUQ428wDyrUI4Txxz7I=; b=Pimn2pXwi16McZiORGsFcmzclRAJ9SLGRILSlNGhU9rbVk1qdRkw1kEJ0DXUucOFlc V1MbLFZljFGpuUhWEsa4NozbaYtNxfbVuyNCJhcgWUJDZPqS/UEMO2YtmLpT9//IXimt u643JN43a9mFHJGAMJ2ZOTzcv5+5kLXaMkQQWoXSIW0IS9dT5V3GLftn5SY3/Azs0xCL zUF7RyWybGC6qW/Z+X55/H5wxSw19bUFbCfbbrMDeMweClrodzJFgIk5m266riWIJHfo IiDn5uqjIOUHfv/XxUJBHvCOOgmHvHB4gzrzg0dje1eqCYNKF8BT8aYIdwzPeeiSD7Wq QWpQ== X-Gm-Message-State: ACrzQf3IxmXFUzmuFZqX4OgbSHYJrGBEfPfJLwMcKe2z4zZVV9Asip01 DudlOVurmOLCWynm1Mxy3hbGGg== X-Received: by 2002:a05:600c:818:b0:3cf:7385:7609 with SMTP id k24-20020a05600c081800b003cf73857609mr38335386wmp.186.1668102621748; Thu, 10 Nov 2022 09:50:21 -0800 (PST) Received: from localhost.localdomain ([2a01:e0a:f:6020:342a:468f:562a:9cc4]) by smtp.gmail.com with ESMTPSA id f6-20020a05600c4e8600b003cfc02ab8basm6514677wmq.33.2022.11.10.09.50.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Nov 2022 09:50:20 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com Cc: qyousef@layalina.io, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v8 3/9] sched/core: Propagate parent task's latency requirements to the child task Date: Thu, 10 Nov 2022 18:50:03 +0100 Message-Id: <20221110175009.18458-4-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221110175009.18458-1-vincent.guittot@linaro.org> References: <20221110175009.18458-1-vincent.guittot@linaro.org> X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749132439032598607?= X-GMAIL-MSGID: =?utf-8?q?1749132439032598607?= From: Parth Shah Clone parent task's latency_nice attribute to the forked child task. Reset the latency_nice value to default value when the child task is set to sched_reset_on_fork. Also, initialize init_task.latency_nice value with DEFAULT_LATENCY_NICE value Signed-off-by: Parth Shah [rebase] Signed-off-by: Vincent Guittot --- init/init_task.c | 1 + kernel/sched/core.c | 1 + 2 files changed, 2 insertions(+) diff --git a/init/init_task.c b/init/init_task.c index ff6c4b9bfe6b..7dd71dd2d261 100644 --- a/init/init_task.c +++ b/init/init_task.c @@ -78,6 +78,7 @@ struct task_struct init_task .prio = MAX_PRIO - 20, .static_prio = MAX_PRIO - 20, .normal_prio = MAX_PRIO - 20, + .latency_nice = DEFAULT_LATENCY_NICE, .policy = SCHED_NORMAL, .cpus_ptr = &init_task.cpus_mask, .user_cpus_ptr = NULL, diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 02dc1b8e3cb6..54544353025b 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4559,6 +4559,7 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p) p->prio = p->normal_prio = p->static_prio; set_load_weight(p, false); + p->latency_nice = DEFAULT_LATENCY_NICE; /* * We don't need the reset flag anymore after the fork. It has * fulfilled its duty: From patchwork Thu Nov 10 17:50:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 18249 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp285788wru; Thu, 10 Nov 2022 09:51:25 -0800 (PST) X-Google-Smtp-Source: AMsMyM6dlNW5n5sdCJy8ok+iP1tZKuQrcKQb9u/pg8oiTDpTJhN4TGgO0csydqg7eQolTwDALoKp X-Received: by 2002:a17:902:d38c:b0:179:e1f6:d24d with SMTP id e12-20020a170902d38c00b00179e1f6d24dmr1626732pld.91.1668102684961; Thu, 10 Nov 2022 09:51:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668102684; cv=none; d=google.com; s=arc-20160816; b=GaETk6OXzqZYwtUea/1xywDvkzeoVq03r6aE8PlOBUWIXTpHk03LTy15TNGF7mbbvz NDNwac9CV2is5uaFUhdfMa2ZxBxM9JR9JK76X1KkFg785JgOxmK2iS5T0q6D2rh6zRM+ edIwnvqgPAho3ZHRLyrx+lr8BS6zT53EULdYvedA8YfEXdQoTnbdtzOkG56XHVdC9QYh 2+PFUmZg8bukoVWADk0xK2qeCAHm6CRREu8wieSnurelpGj5an3Uya6nMO/ChWhZGiBa Z3/D5Z4pBH3LAn3GUEwuPwwHdH+RTe1KlYlHPkI5iyxsy8pMvT1N2vyDP/gCeSjdNSJ9 agGA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=Oeg4H6o/dcPx5AQqGWzAkKIEqb6LjGQnX/XTYoDflfQ=; b=A25xupJaDyyz2kIhJELZAJ2TH04JYS+d6GqXlwrd8sE1KV9l+rZoE1Bf520Bd17MOV 2n24VP/XIbWJBX0FOJqP7+1GZZk43dB92IvtJtPzjDJbQF1JrS6yEBa9+qiJJcdTMTFq gLVFHonhGcLHRvxbo4Wueyj+2oGo/f4tRZ+saGYBKhvs+8861bWi0oMi21IHNq0QD8kJ 7bPMEWUdIRdmiL+mxrsSHUEn4ZMx/NxTu2phCQuPQvRQ8nAo5eBoqP3/IOdAQrfvVlNj mN/AbEjjYv/FExn2B8nzdY0KAzGLgwNAjU6mJNz2UW/gkIejmcMZENxIV1Sv8Rvtlupa 5f7A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=OacGgf30; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f1-20020a631001000000b00456b507fb2asi20537162pgl.446.2022.11.10.09.51.11; Thu, 10 Nov 2022 09:51:24 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=OacGgf30; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231423AbiKJRuq (ORCPT + 99 others); Thu, 10 Nov 2022 12:50:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33954 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231286AbiKJRu0 (ORCPT ); Thu, 10 Nov 2022 12:50:26 -0500 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4CF683C6E6 for ; Thu, 10 Nov 2022 09:50:25 -0800 (PST) Received: by mail-wm1-x32f.google.com with SMTP id o30so1608387wms.2 for ; Thu, 10 Nov 2022 09:50:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=Oeg4H6o/dcPx5AQqGWzAkKIEqb6LjGQnX/XTYoDflfQ=; b=OacGgf30bByrvVz1+tlosG27nEni7YDntuF+2Z0Q0dpuqjl7PIsMxFYqNUBsLfeCOP i5stUX9so7amoY0ktRBJ/FXMiEfypgv5cgIG7GGzs0stvJdMfch25M0HRRZsn1r5vJIL dPXiWBKK/2HZNRnOfmFlvne07+deZv1P/nsi7XH64NTARTUdJAOIYiQSPNMCVVqk/5/h 3k7hZYvZoY2IrcNMnHLXTTwUFPKGlj6HbT/PjnZqjWmiWFfrgEyzjkE0tTkBMm38Llv1 n3Juoa39/+zw8uCrVq9A5y6zAX2qAOcqLwi6w/z3+XHGP6I6a1oHiAtj/6qpdYeg6gt3 mwIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Oeg4H6o/dcPx5AQqGWzAkKIEqb6LjGQnX/XTYoDflfQ=; b=kNZcukAQ1RS8CqkHtiumB/81gN3YHI99jGnyBs4kmpM4XbD45q91vIl9WSGWhzHg+J bTU26+EUFRkRUyYLzvqC5H+D9/XjKsvP0xPfUnDTdkMxhSp0NfcabMHd6rPYxHU2/16o sgebuTnbJTl9INn+xKIuuDii0O7Yil3o4BxHKNdn9juhtUxSft3d5NmKDHnvkB/QrIcF MmPKdD7ztWn1BIfLqufQHi/hdknjG+C93+yNEj1uDUaat8b2p9lDZlwRjYZ8nTlu2Xlg U1QUlRVmwfoeNjxUy2T4T9VDotA4FFmGN0EoTRIvpwhDl3OBY6qseRghcGn/8kykzRw+ uR8Q== X-Gm-Message-State: ACrzQf3PvZv9cZZZe1GZvN9IgfN4MWuOcu0lq6lt/QL3GwrICEXUSoeA aAHBWRi04Yzcbz1ib69o5frPMA== X-Received: by 2002:a05:600c:4586:b0:3c6:fbb0:bf2d with SMTP id r6-20020a05600c458600b003c6fbb0bf2dmr43639707wmo.13.1668102623717; Thu, 10 Nov 2022 09:50:23 -0800 (PST) Received: from localhost.localdomain ([2a01:e0a:f:6020:342a:468f:562a:9cc4]) by smtp.gmail.com with ESMTPSA id f6-20020a05600c4e8600b003cfc02ab8basm6514677wmq.33.2022.11.10.09.50.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Nov 2022 09:50:22 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com Cc: qyousef@layalina.io, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v8 4/9] sched: Allow sched_{get,set}attr to change latency_nice of the task Date: Thu, 10 Nov 2022 18:50:04 +0100 Message-Id: <20221110175009.18458-5-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221110175009.18458-1-vincent.guittot@linaro.org> References: <20221110175009.18458-1-vincent.guittot@linaro.org> X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749132441213000256?= X-GMAIL-MSGID: =?utf-8?q?1749132441213000256?= From: Parth Shah Introduce the latency_nice attribute to sched_attr and provide a mechanism to change the value with the use of sched_setattr/sched_getattr syscall. Also add new flag "SCHED_FLAG_LATENCY_NICE" to hint the change in latency_nice of the task on every sched_setattr syscall. Signed-off-by: Parth Shah [rebase and add a dedicated __setscheduler_latency ] Signed-off-by: Vincent Guittot --- include/uapi/linux/sched.h | 4 +++- include/uapi/linux/sched/types.h | 19 +++++++++++++++++++ kernel/sched/core.c | 24 ++++++++++++++++++++++++ tools/include/uapi/linux/sched.h | 4 +++- 4 files changed, 49 insertions(+), 2 deletions(-) diff --git a/include/uapi/linux/sched.h b/include/uapi/linux/sched.h index 3bac0a8ceab2..b2e932c25be6 100644 --- a/include/uapi/linux/sched.h +++ b/include/uapi/linux/sched.h @@ -132,6 +132,7 @@ struct clone_args { #define SCHED_FLAG_KEEP_PARAMS 0x10 #define SCHED_FLAG_UTIL_CLAMP_MIN 0x20 #define SCHED_FLAG_UTIL_CLAMP_MAX 0x40 +#define SCHED_FLAG_LATENCY_NICE 0x80 #define SCHED_FLAG_KEEP_ALL (SCHED_FLAG_KEEP_POLICY | \ SCHED_FLAG_KEEP_PARAMS) @@ -143,6 +144,7 @@ struct clone_args { SCHED_FLAG_RECLAIM | \ SCHED_FLAG_DL_OVERRUN | \ SCHED_FLAG_KEEP_ALL | \ - SCHED_FLAG_UTIL_CLAMP) + SCHED_FLAG_UTIL_CLAMP | \ + SCHED_FLAG_LATENCY_NICE) #endif /* _UAPI_LINUX_SCHED_H */ diff --git a/include/uapi/linux/sched/types.h b/include/uapi/linux/sched/types.h index f2c4589d4dbf..db1e8199e8c8 100644 --- a/include/uapi/linux/sched/types.h +++ b/include/uapi/linux/sched/types.h @@ -10,6 +10,7 @@ struct sched_param { #define SCHED_ATTR_SIZE_VER0 48 /* sizeof first published struct */ #define SCHED_ATTR_SIZE_VER1 56 /* add: util_{min,max} */ +#define SCHED_ATTR_SIZE_VER2 60 /* add: latency_nice */ /* * Extended scheduling parameters data structure. @@ -98,6 +99,22 @@ struct sched_param { * scheduled on a CPU with no more capacity than the specified value. * * A task utilization boundary can be reset by setting the attribute to -1. + * + * Latency Tolerance Attributes + * =========================== + * + * A subset of sched_attr attributes allows to specify the relative latency + * requirements of a task with respect to the other tasks running/queued in the + * system. + * + * @ sched_latency_nice task's latency_nice value + * + * The latency_nice of a task can have any value in a range of + * [MIN_LATENCY_NICE..MAX_LATENCY_NICE]. + * + * A task with latency_nice with the value of LATENCY_NICE_MIN can be + * taken for a task requiring a lower latency as opposed to the task with + * higher latency_nice. */ struct sched_attr { __u32 size; @@ -120,6 +137,8 @@ struct sched_attr { __u32 sched_util_min; __u32 sched_util_max; + /* latency requirement hints */ + __s32 sched_latency_nice; }; #endif /* _UAPI_LINUX_SCHED_TYPES_H */ diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 54544353025b..b2accc9da4fe 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -7318,6 +7318,14 @@ static void __setscheduler_params(struct task_struct *p, p->rt_priority = attr->sched_priority; p->normal_prio = normal_prio(p); set_load_weight(p, true); + +} + +static void __setscheduler_latency(struct task_struct *p, + const struct sched_attr *attr) +{ + if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) + p->latency_nice = attr->sched_latency_nice; } /* @@ -7460,6 +7468,13 @@ static int __sched_setscheduler(struct task_struct *p, return retval; } + if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) { + if (attr->sched_latency_nice > MAX_LATENCY_NICE) + return -EINVAL; + if (attr->sched_latency_nice < MIN_LATENCY_NICE) + return -EINVAL; + } + if (pi) cpuset_read_lock(); @@ -7494,6 +7509,9 @@ static int __sched_setscheduler(struct task_struct *p, goto change; if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP) goto change; + if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE && + attr->sched_latency_nice != p->latency_nice) + goto change; p->sched_reset_on_fork = reset_on_fork; retval = 0; @@ -7582,6 +7600,7 @@ static int __sched_setscheduler(struct task_struct *p, __setscheduler_params(p, attr); __setscheduler_prio(p, newprio); } + __setscheduler_latency(p, attr); __setscheduler_uclamp(p, attr); if (queued) { @@ -7792,6 +7811,9 @@ static int sched_copy_attr(struct sched_attr __user *uattr, struct sched_attr *a size < SCHED_ATTR_SIZE_VER1) return -EINVAL; + if ((attr->sched_flags & SCHED_FLAG_LATENCY_NICE) && + size < SCHED_ATTR_SIZE_VER2) + return -EINVAL; /* * XXX: Do we want to be lenient like existing syscalls; or do we want * to be strict and return an error on out-of-bounds values? @@ -8029,6 +8051,8 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sched_attr __user *, uattr, get_params(p, &kattr); kattr.sched_flags &= SCHED_FLAG_ALL; + kattr.sched_latency_nice = p->latency_nice; + #ifdef CONFIG_UCLAMP_TASK /* * This could race with another potential updater, but this is fine diff --git a/tools/include/uapi/linux/sched.h b/tools/include/uapi/linux/sched.h index 3bac0a8ceab2..b2e932c25be6 100644 --- a/tools/include/uapi/linux/sched.h +++ b/tools/include/uapi/linux/sched.h @@ -132,6 +132,7 @@ struct clone_args { #define SCHED_FLAG_KEEP_PARAMS 0x10 #define SCHED_FLAG_UTIL_CLAMP_MIN 0x20 #define SCHED_FLAG_UTIL_CLAMP_MAX 0x40 +#define SCHED_FLAG_LATENCY_NICE 0x80 #define SCHED_FLAG_KEEP_ALL (SCHED_FLAG_KEEP_POLICY | \ SCHED_FLAG_KEEP_PARAMS) @@ -143,6 +144,7 @@ struct clone_args { SCHED_FLAG_RECLAIM | \ SCHED_FLAG_DL_OVERRUN | \ SCHED_FLAG_KEEP_ALL | \ - SCHED_FLAG_UTIL_CLAMP) + SCHED_FLAG_UTIL_CLAMP | \ + SCHED_FLAG_LATENCY_NICE) #endif /* _UAPI_LINUX_SCHED_H */ From patchwork Thu Nov 10 17:50:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 18252 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp286544wru; Thu, 10 Nov 2022 09:53:02 -0800 (PST) X-Google-Smtp-Source: AMsMyM5vtJALjP3A90dy+nzx12yju8upObnSJEmK79/4WFY8BBMhrIxkRR1DZKA7nZfzGRL0D3oR X-Received: by 2002:a17:90a:7808:b0:214:1a8a:a40a with SMTP id w8-20020a17090a780800b002141a8aa40amr49295426pjk.137.1668102782118; Thu, 10 Nov 2022 09:53:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668102782; cv=none; d=google.com; s=arc-20160816; b=MfRMOzJwmtV4JebkeorX2ev3Ky4MP+foEoU1rvoGYYFncbjnOqt7GrbFYtmjA/C/3n yQD15KiB8m7ByjzXyzl4z6OfUn8xKYZ92GKGceQpTGA0T88LjqDvbgGzk1GKCmkw8+fh A+2oolfjo5IPGCE9yhIFsipvZythbTlwNSyr/vPixNwFpiUO/faWgBifUGbu6ZX6j871 LLVu2esXJKbJ/fVAMYqW1bIM4F2P9N24Y7imjSuKudmY4+EwHwbDBatNmcSCeDaa39P3 ANpf5Y1HIEF1CzxeB3jTbLlhOjgE3YbsNeExETr24naIgbS2NBFa5Ps60jyzMLIovmqv XMng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=l+29IGkaCXOYpZn2jHPEOLz7ZMwnGQwxoqUEqxvOsgk=; b=aEjvH2ID794ZdOiNqIgexRn0pbCkz2xcVFljtJJdy0IiEE/TKMQj+AOCAuMUcCBwl/ vyVxOKquXdYOONQXrczczX/f5wsw+vN2amlSLv5LJ9tJi+vklVwtfJ6i4n3RhZ2kr5N/ XmkIyaH4Z26ln76V6mXMymd/nf774iwqv1bzJGg+pCeGkq96QzqQM0+vdBmqN6OrNQg0 AjRkdoI/F3EGn1WiiqPvB8BpAXwfaSfkG4CIQhPbc7RIKFAkJ+vjg7Eq3pSeIIC2+8AI vn2KqXiL/jqK9YDNV7uQsSaSHuWweBV4lW7Mq/VKULG6TuaDDpx4h6+UqKuJCPV7KIv/ 0I+A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=IQZHxebG; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l7-20020a17090a850700b0020d2286c30csi153145pjn.134.2022.11.10.09.52.48; Thu, 10 Nov 2022 09:53:02 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=IQZHxebG; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231443AbiKJRuu (ORCPT + 99 others); Thu, 10 Nov 2022 12:50:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33906 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231390AbiKJRuj (ORCPT ); Thu, 10 Nov 2022 12:50:39 -0500 Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 304693F05A for ; Thu, 10 Nov 2022 09:50:28 -0800 (PST) Received: by mail-wm1-x336.google.com with SMTP id h133-20020a1c218b000000b003cf4d389c41so3997095wmh.3 for ; Thu, 10 Nov 2022 09:50:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=l+29IGkaCXOYpZn2jHPEOLz7ZMwnGQwxoqUEqxvOsgk=; b=IQZHxebG+swqyRL1GrlaPg7XGoqXLFverqCbZRvsHp4AuYjtfCvMaTiyKhK6BySHNc XNx1PdKShf0d9gGAMcc4EtNUvRN04c1skxzQsSrXICz6n9lapx6ZLP/q09VsEMgQLzfq Dk+yVLBzHJEHaR1ZlkG48BWay/dFVlR5NcDx/fQkqrT4asTtvhLSXTlb9tFyBu7H8kw/ gUzGba38uMlvi7RZODngbj5RWr6Mlk+2Su5F3aht1t2BWXfdhXuyLNgxnaocQIdd+8DC yTmryx2kXVKcJh6byEOTnZKbZUDCICzP8xKOmvHl6g05DzSEVbRKEEVU0JUogqIWfraN NMyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=l+29IGkaCXOYpZn2jHPEOLz7ZMwnGQwxoqUEqxvOsgk=; b=YZ7J5VTz3tX8zkunRa77fD784DLMBTD2hc3+f0uxnsROj9H7CNcyfhDh9vaKk4JJSJ boePrBhIEOaIJJtQDcTbbfrIMhLs29AbiL7Nx0vdo3I7pEuzn+wwHGh0UYgWCpOUPips M55iSTqDZTnKmAMvngy+Xty7HeuevOu01eXI9RfE//iF4oAgfKDufMp96E6gH/rTAHWA t9WEo2CJ4ZIhJwA6OisTk1ll9/QzYSTdPKvarkBlPqjKJmzM+DjHBOk2CZcLXQqVqHUl YsTUn8xlxq8acMjfkD166/u3qLuKlP10goykuf54JgVxx3gBNfVB/TsEDqxincQSfhiE +/bQ== X-Gm-Message-State: ACrzQf0ZNtGXOtHnLFHIvA+9dnqMbOhyRjdI9vxJQcyJKwAuIELyMkAw IS2jpPQ9bAgHIuKNJHxTKd+axA== X-Received: by 2002:a05:600c:3546:b0:3cf:a18e:12e5 with SMTP id i6-20020a05600c354600b003cfa18e12e5mr18865370wmq.112.1668102626631; Thu, 10 Nov 2022 09:50:26 -0800 (PST) Received: from localhost.localdomain ([2a01:e0a:f:6020:342a:468f:562a:9cc4]) by smtp.gmail.com with ESMTPSA id f6-20020a05600c4e8600b003cfc02ab8basm6514677wmq.33.2022.11.10.09.50.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Nov 2022 09:50:24 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com Cc: qyousef@layalina.io, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v8 5/9] sched/fair: Take into account latency priority at wakeup Date: Thu, 10 Nov 2022 18:50:05 +0100 Message-Id: <20221110175009.18458-6-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221110175009.18458-1-vincent.guittot@linaro.org> References: <20221110175009.18458-1-vincent.guittot@linaro.org> X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,PDS_BTC_ID,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749132543157477692?= X-GMAIL-MSGID: =?utf-8?q?1749132543157477692?= Take into account the latency priority of a thread when deciding to preempt the current running thread. We don't want to provide more CPU bandwidth to a thread but reorder the scheduling to run latency sensitive task first whenever possible. As long as a thread didn't use its bandwidth, it will be able to preempt the current thread. At the opposite, a thread with a low latency priority will preempt current thread at wakeup only to keep fair CPU bandwidth sharing. Otherwise it will wait for the tick to get its sched slice. curr vruntime | sysctl_sched_wakeup_granularity <--> ----------------------------------|----|-----------------------|--------------- | |<---------------------> | . sysctl_sched_latency | . default/current latency entity | . | . 1111111111111111111111111111111111|0000|-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1- se preempts curr at wakeup ------>|<- se doesn't preempt curr ----------------- | . | . | . low latency entity | . ---------------------->| % of sysctl_sched_latency | 1111111111111111111111111111111111111111111111111111111111|0000|-1-1-1-1-1-1-1- preempt ------------------------------------------------->|<- do not preempt -- | . | . | . high latency entity | . |<-----------------------|----. | % of sysctl_sched_latency . 111111111|0000|-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1 preempt->|<- se doesn't preempt curr ------------------------------------------ Tests results of nice latency impact on heavy load like hackbench: hackbench -l (2560 / group) -g group group latency 0 latency 19 1 1.378(+/- 1%) 1.337(+/- 1%) + 3% 4 1.393(+/- 3%) 1.312(+/- 3%) + 6% 8 1.308(+/- 2%) 1.279(+/- 1%) + 2% 16 1.347(+/- 1%) 1.317(+/- 1%) + 2% hackbench -p -l (2560 / group) -g group group 1 1.836(+/- 17%) 1.148(+/- 5%) +37% 4 1.586(+/- 6%) 1.109(+/- 8%) +30% 8 1.209(+/- 4%) 0.780(+/- 4%) +35% 16 0.805(+/- 5%) 0.728(+/- 4%) +10% By deacreasing the latency prio, we reduce the number of preemption at wakeup and help hackbench making progress. Test results of nice latency impact on short live load like cyclictest while competing with heavy load like hackbench: hackbench -l 10000 -g $group & cyclictest --policy other -D 5 -q -n latency 0 latency -20 group min avg max min avg max 0 16 19 29 17 18 29 1 43 299 7359 63 84 3422 4 56 449 14806 45 83 284 8 63 820 51123 63 83 283 16 64 1326 70684 41 157 26852 group = 0 means that hackbench is not running. The avg is significantly improved with nice latency -20 especially with large number of groups but min and max remain quite similar. If we add the histogram parameter to get details of latency, we have : hackbench -l 10000 -g 16 & cyclictest --policy other -D 5 -q -n -H 20000 --histfile data.txt latency 0 latency -20 Min Latencies: 64 62 Avg Latencies: 1170 107 Max Latencies: 88069 10417 50% latencies: 122 86 75% latencies: 614 91 85% latencies: 961 94 90% latencies: 1225 97 95% latencies: 6120 102 99% latencies: 18328 159 With percentile details, we see the benefit of nice latency -20 as only 1% of the latencies are above 159us whereas the default latency has got 15% around ~1ms or above and 5% over the 6ms. Signed-off-by: Vincent Guittot --- include/linux/sched.h | 4 ++- init/init_task.c | 2 +- kernel/sched/core.c | 38 +++++++++++++++++++++---- kernel/sched/debug.c | 2 +- kernel/sched/fair.c | 66 ++++++++++++++++++++++++++++++++++++++----- kernel/sched/sched.h | 12 ++++++++ 6 files changed, 109 insertions(+), 15 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 6805f378a9c3..a74cad08e91e 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -567,6 +567,8 @@ struct sched_entity { /* cached value of my_q->h_nr_running */ unsigned long runnable_weight; #endif + /* preemption offset in ns */ + long latency_offset; #ifdef CONFIG_SMP /* @@ -783,7 +785,7 @@ struct task_struct { int static_prio; int normal_prio; unsigned int rt_priority; - int latency_nice; + int latency_prio; struct sched_entity se; struct sched_rt_entity rt; diff --git a/init/init_task.c b/init/init_task.c index 7dd71dd2d261..b8ddf403bc62 100644 --- a/init/init_task.c +++ b/init/init_task.c @@ -78,7 +78,7 @@ struct task_struct init_task .prio = MAX_PRIO - 20, .static_prio = MAX_PRIO - 20, .normal_prio = MAX_PRIO - 20, - .latency_nice = DEFAULT_LATENCY_NICE, + .latency_prio = NICE_WIDTH - 20, .policy = SCHED_NORMAL, .cpus_ptr = &init_task.cpus_mask, .user_cpus_ptr = NULL, diff --git a/kernel/sched/core.c b/kernel/sched/core.c index b2accc9da4fe..caf54e54a74f 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -1284,6 +1284,16 @@ static void set_load_weight(struct task_struct *p, bool update_load) } } +static void set_latency_offset(struct task_struct *p) +{ + long weight = sched_latency_to_weight[p->latency_prio]; + s64 offset; + + offset = weight * get_sched_latency(false); + offset = div_s64(offset, NICE_LATENCY_WEIGHT_MAX); + p->se.latency_offset = (long)offset; +} + #ifdef CONFIG_UCLAMP_TASK /* * Serializes updates of utilization clamp values @@ -4559,7 +4569,9 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p) p->prio = p->normal_prio = p->static_prio; set_load_weight(p, false); - p->latency_nice = DEFAULT_LATENCY_NICE; + p->latency_prio = NICE_TO_LATENCY(0); + set_latency_offset(p); + /* * We don't need the reset flag anymore after the fork. It has * fulfilled its duty: @@ -7324,8 +7336,10 @@ static void __setscheduler_params(struct task_struct *p, static void __setscheduler_latency(struct task_struct *p, const struct sched_attr *attr) { - if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) - p->latency_nice = attr->sched_latency_nice; + if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) { + p->latency_prio = NICE_TO_LATENCY(attr->sched_latency_nice); + set_latency_offset(p); + } } /* @@ -7510,7 +7524,7 @@ static int __sched_setscheduler(struct task_struct *p, if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP) goto change; if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE && - attr->sched_latency_nice != p->latency_nice) + attr->sched_latency_nice != LATENCY_TO_NICE(p->latency_prio)) goto change; p->sched_reset_on_fork = reset_on_fork; @@ -8051,7 +8065,7 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sched_attr __user *, uattr, get_params(p, &kattr); kattr.sched_flags &= SCHED_FLAG_ALL; - kattr.sched_latency_nice = p->latency_nice; + kattr.sched_latency_nice = LATENCY_TO_NICE(p->latency_prio); #ifdef CONFIG_UCLAMP_TASK /* @@ -11204,6 +11218,20 @@ const u32 sched_prio_to_wmult[40] = { /* 15 */ 119304647, 148102320, 186737708, 238609294, 286331153, }; +/* + * latency weight for wakeup preemption + */ +const int sched_latency_to_weight[40] = { + /* -20 */ -1024, -973, -922, -870, -819, + /* -15 */ -768, -717, -666, -614, -563, + /* -10 */ -512, -461, -410, -358, -307, + /* -5 */ -256, -205, -154, -102, -51, + /* 0 */ 0, 51, 102, 154, 205, + /* 5 */ 256, 307, 358, 410, 461, + /* 10 */ 512, 563, 614, 666, 717, + /* 15 */ 768, 819, 870, 922, 973, +}; + void call_trace_sched_update_nr_running(struct rq *rq, int count) { trace_sched_update_nr_running_tp(rq, count); diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c index a3f7876217a6..06aaa0c81d4b 100644 --- a/kernel/sched/debug.c +++ b/kernel/sched/debug.c @@ -1042,7 +1042,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns, #endif P(policy); P(prio); - P(latency_nice); + P(latency_prio); if (task_has_dl_policy(p)) { P(dl.runtime); P(dl.deadline); diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index eb04c83112a0..4299d5108dc7 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4558,6 +4558,8 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) update_idle_cfs_rq_clock_pelt(cfs_rq); } +static long wakeup_latency_gran(struct sched_entity *curr, struct sched_entity *se); + /* * Preempt the current task with a newly woken task if needed: */ @@ -4566,7 +4568,7 @@ check_preempt_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr) { unsigned long ideal_runtime, delta_exec; struct sched_entity *se; - s64 delta; + s64 delta, offset; ideal_runtime = sched_slice(cfs_rq, curr); delta_exec = curr->sum_exec_runtime - curr->prev_sum_exec_runtime; @@ -4591,10 +4593,12 @@ check_preempt_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr) se = __pick_first_entity(cfs_rq); delta = curr->vruntime - se->vruntime; - if (delta < 0) + offset = wakeup_latency_gran(curr, se); + if (delta < offset) return; - if (delta > ideal_runtime) + if ((delta > ideal_runtime) || + (delta > get_latency_max())) resched_curr(rq_of(cfs_rq)); } @@ -5716,6 +5720,35 @@ static int sched_idle_cpu(int cpu) } #endif +static void set_next_buddy(struct sched_entity *se); + +static void check_preempt_from_others(struct cfs_rq *cfs, struct sched_entity *se) +{ + struct sched_entity *next; + + if (se->latency_offset >= 0) + return; + + if (cfs->nr_running <= 1) + return; + /* + * When waking from another class, we don't need to check to preempt at + * wakeup and don't set next buddy as a candidate for being picked in + * priority. + * In case of simultaneous wakeup when current is another class, the + * latency sensitive tasks lost opportunity to preempt non sensitive + * tasks which woke up simultaneously. + */ + + if (cfs->next) + next = cfs->next; + else + next = __pick_first_entity(cfs); + + if (next && wakeup_preempt_entity(next, se) == 1) + set_next_buddy(se); +} + /* * The enqueue_task method is called before nr_running is * increased. Here we update the fair scheduling stats and @@ -5802,14 +5835,15 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) if (!task_new) update_overutilized_status(rq); + if (rq->curr->sched_class != &fair_sched_class) + check_preempt_from_others(cfs_rq_of(&p->se), &p->se); + enqueue_throttle: assert_list_leaf_cfs_rq(rq); hrtick_update(rq); } -static void set_next_buddy(struct sched_entity *se); - /* * The dequeue_task method is called before nr_running is * decreased. We remove the task from the rbtree and @@ -7128,6 +7162,23 @@ balance_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) } #endif /* CONFIG_SMP */ +static long wakeup_latency_gran(struct sched_entity *curr, struct sched_entity *se) +{ + long latency_offset = se->latency_offset; + + /* + * A negative latency offset means that the sched_entity has latency + * requirement that needs to be evaluated versus other entity. + * Otherwise, use the latency weight to evaluate how much scheduling + * delay is acceptable by se. + */ + if ((latency_offset < 0) || (curr->latency_offset < 0)) + latency_offset -= curr->latency_offset; + latency_offset = min_t(long, latency_offset, get_latency_max()); + + return latency_offset; +} + static unsigned long wakeup_gran(struct sched_entity *se) { unsigned long gran = sysctl_sched_wakeup_granularity; @@ -7166,11 +7217,12 @@ static int wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se) { s64 gran, vdiff = curr->vruntime - se->vruntime; + s64 offset = wakeup_latency_gran(curr, se); - if (vdiff <= 0) + if (vdiff < offset) return -1; - gran = wakeup_gran(se); + gran = offset + wakeup_gran(se); /* * At wake up, the vruntime of a task is capped to not be older than diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 4bf9d7777f99..99f10b4dc230 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -142,6 +142,17 @@ extern int sched_rr_timeslice; * Default tasks should be treated as a task with latency_nice = 0. */ #define DEFAULT_LATENCY_NICE 0 +#define DEFAULT_LATENCY_PRIO (DEFAULT_LATENCY_NICE + LATENCY_NICE_WIDTH/2) + +/* + * Convert user-nice values [ -20 ... 0 ... 19 ] + * to static latency [ 0..39 ], + * and back. + */ +#define NICE_TO_LATENCY(nice) ((nice) + DEFAULT_LATENCY_PRIO) +#define LATENCY_TO_NICE(prio) ((prio) - DEFAULT_LATENCY_PRIO) +#define NICE_LATENCY_SHIFT (SCHED_FIXEDPOINT_SHIFT) +#define NICE_LATENCY_WEIGHT_MAX (1L << NICE_LATENCY_SHIFT) /* * Increase resolution of nice-level calculations for 64-bit architectures. @@ -2116,6 +2127,7 @@ static_assert(WF_TTWU == SD_BALANCE_WAKE); extern const int sched_prio_to_weight[40]; extern const u32 sched_prio_to_wmult[40]; +extern const int sched_latency_to_weight[40]; /* * {de,en}queue flags: From patchwork Thu Nov 10 17:50:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 18253 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp286550wru; Thu, 10 Nov 2022 09:53:03 -0800 (PST) X-Google-Smtp-Source: AMsMyM5WCywQt4zSu3M1mXCRcmtzyoR+OPQ1Cg+NX1AkgHgDp8j4En3JRfAszxodE4/MM06vky6k X-Received: by 2002:a17:903:2452:b0:186:99e0:672d with SMTP id l18-20020a170903245200b0018699e0672dmr66028162pls.95.1668102782817; Thu, 10 Nov 2022 09:53:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668102782; cv=none; d=google.com; s=arc-20160816; b=pUHA95V+Bh1pdQwPwqERC8/sbB1HOHagxftphmTqd7lOL5wbal7sI7zyNzXpKV2X+Y xTVBbU71WqFlBP7zSGYcMjZ2X3aC12GvItK6tN+NBBbXSEbwvgdJSieY//jmBpwjZ4iR /8oOdth19qv1BuuDAV77D6hdn4xYLlQeg3vFzpJC9YmqBJ++ivd965LusaN0ISaZ3OC7 zlqh77u+rfo9+I1CM2r8g4Vves7pdUjxHoTIqPYtjfczcXBncpCP0Nhe/T91D4KVYaGc KwyPSUQgv0xIqkO48Uzwc+3Khqq3cd/cVYW4jnBVyfbwM26zDaaJ94O+fEw0apjenr+q CXvQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=1+roSIgbqvGq/+OOlHqHUt+Rf8MwdSQP0WyWz8JU2fs=; b=HKZSZu4ir91fybqJrU3GoN5JE0MvcUvrmmch5C4w3brIsxd72VZMF3IPi+fT7Pp8aI cpaCF2YYiW1A0zl1jVvG0g9IqUqv7rBrOUqEFPD/JRagm2b0VN9l6RvTyTwLaOIpTNTG 6nrAlL7sFke+cGe5uQJDd10I9zliZu+fb5pYfs0S2Ah2Eik/VGimRDawJSpP1lhA4+m3 ljGKgqj5zE1uSCY3wfooutjdexx8PmhDQpPjktC1uLwQdj7FkwRVWlRI6Irk4SZig3XH /II2nEz9WUB3IusQuUalx47wIPGJdd8NCsTKjwjwalbCpLuQXrxz49MUjwuTGFnm6+rb FD8g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=wALwYXKW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id d16-20020a170902729000b001873af01dcasi18790770pll.85.2022.11.10.09.52.48; Thu, 10 Nov 2022 09:53:02 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=wALwYXKW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231666AbiKJRu6 (ORCPT + 99 others); Thu, 10 Nov 2022 12:50:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33952 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231426AbiKJRuk (ORCPT ); Thu, 10 Nov 2022 12:50:40 -0500 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 734BC45A3D for ; Thu, 10 Nov 2022 09:50:30 -0800 (PST) Received: by mail-wr1-x431.google.com with SMTP id a14so3370628wru.5 for ; Thu, 10 Nov 2022 09:50:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=1+roSIgbqvGq/+OOlHqHUt+Rf8MwdSQP0WyWz8JU2fs=; b=wALwYXKWKLKsnMyjlz7MkUR91OBeeuRK1zURuBDKEIjHgmCPt+UjBYKledjkp882qA cnUMY/+SWT4HR1VMK9IKylvlgVF/VAsJ1OHakTPcgExBTPL5TLF9v9lejWdfjWvWaHPN FhmD5zSgXODtQemZ/VqMh2bDy2YVsvW/UXwEsOtoIBImcz11xQgtBdCeOkFfkB8IHO2Z 6ifJ7vlBq0qHXXlslPirV/ZBg39TAxvsH/1mZSa/TuPoPfYdV0VkMZBUdtk2FRYIG/L3 X0eQjn5gwnWchXIx9Zrxal4Vx6MS3+WbP3Ib49WLmPYsPm/KbJeBkLsEqErNQ/BZ+WK7 I8hQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=1+roSIgbqvGq/+OOlHqHUt+Rf8MwdSQP0WyWz8JU2fs=; b=A09fmT99khLr2OcbNWvHj/+y0FQXQs2YNjNcUJLyL24KbaHxEneuXkZzaK66vmuzeC 3uBUM18Ov7wijTXACICL3KoDHHAtCdVNX+hUEbdysOKGrb9GrUQ10kZZ+YUTxYd5cIrj uz8YxOxNo0pVEnhDqj6oYqOsuK+s5LyxZt3BH5bq5qUFWNqlFotQj/m5Bn2IceHg04GF KzUZ8B7z+6HDefghzGa0ATS3qIHRwLx4iO7r0RAqWP3iHoVIC/LEPNQCuuE3+aFu2s2T i0HRKr890PLGi6IlkBMT38Eg7nZhnnKKxMU1eYdgIlXepNkYg/RpA/grqr8N1wTLQOjm jGLw== X-Gm-Message-State: ACrzQf2ZEYt2gXiRw2Cpch4FifPgGEyoH7F3TBD4eFc9CkiGzAVy6yQX Rt03bfORdQvYCm53kv7NLw+2SQ== X-Received: by 2002:adf:fc05:0:b0:236:63fa:c792 with SMTP id i5-20020adffc05000000b0023663fac792mr1046633wrr.476.1668102628909; Thu, 10 Nov 2022 09:50:28 -0800 (PST) Received: from localhost.localdomain ([2a01:e0a:f:6020:342a:468f:562a:9cc4]) by smtp.gmail.com with ESMTPSA id f6-20020a05600c4e8600b003cfc02ab8basm6514677wmq.33.2022.11.10.09.50.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Nov 2022 09:50:27 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com Cc: qyousef@layalina.io, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v8 6/9] sched/fair: Add sched group latency support Date: Thu, 10 Nov 2022 18:50:06 +0100 Message-Id: <20221110175009.18458-7-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221110175009.18458-1-vincent.guittot@linaro.org> References: <20221110175009.18458-1-vincent.guittot@linaro.org> X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749132543360069579?= X-GMAIL-MSGID: =?utf-8?q?1749132543360069579?= Task can set its latency priority with sched_setattr(), which is then used to set the latency offset of its sched_enity, but sched group entities still have the default latency offset value. Add a latency.nice field in cpu cgroup controller to set the latency priority of the group similarly to sched_setattr(). The latency priority is then used to set the offset of the sched_entities of the group. Signed-off-by: Vincent Guittot --- Documentation/admin-guide/cgroup-v2.rst | 10 +++++ kernel/sched/core.c | 52 +++++++++++++++++++++++++ kernel/sched/fair.c | 33 ++++++++++++++++ kernel/sched/sched.h | 4 ++ 4 files changed, 99 insertions(+) diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst index be4a77baf784..a4866cd4e58c 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -1095,6 +1095,16 @@ All time durations are in microseconds. values similar to the sched_setattr(2). This maximum utilization value is used to clamp the task specific maximum utilization clamp. + cpu.latency.nice + A read-write single value file which exists on non-root + cgroups. The default is "0". + + The nice value is in the range [-20, 19]. + + This interface file allows reading and setting latency using the + same values used by sched_setattr(2). The latency_nice of a group is + used to limit the impact of the latency_nice of a task outside the + group. Memory diff --git a/kernel/sched/core.c b/kernel/sched/core.c index caf54e54a74f..3f42b1f61a7e 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -10890,6 +10890,47 @@ static int cpu_idle_write_s64(struct cgroup_subsys_state *css, { return sched_group_set_idle(css_tg(css), idle); } + +static s64 cpu_latency_nice_read_s64(struct cgroup_subsys_state *css, + struct cftype *cft) +{ + int prio, delta, last_delta = INT_MAX; + s64 weight; + + weight = css_tg(css)->latency_offset * NICE_LATENCY_WEIGHT_MAX; + weight = div_s64(weight, get_sched_latency(false)); + + /* Find the closest nice value to the current weight */ + for (prio = 0; prio < ARRAY_SIZE(sched_latency_to_weight); prio++) { + delta = abs(sched_latency_to_weight[prio] - weight); + if (delta >= last_delta) + break; + last_delta = delta; + } + + return LATENCY_TO_NICE(prio-1); +} + +static int cpu_latency_nice_write_s64(struct cgroup_subsys_state *css, + struct cftype *cft, s64 nice) +{ + s64 latency_offset; + long weight; + int idx; + + if (nice < MIN_LATENCY_NICE || nice > MAX_LATENCY_NICE) + return -ERANGE; + + idx = NICE_TO_LATENCY(nice); + idx = array_index_nospec(idx, LATENCY_NICE_WIDTH); + weight = sched_latency_to_weight[idx]; + + latency_offset = weight * get_sched_latency(false); + latency_offset = div_s64(latency_offset, NICE_LATENCY_WEIGHT_MAX); + + return sched_group_set_latency(css_tg(css), latency_offset); +} + #endif static struct cftype cpu_legacy_files[] = { @@ -10904,6 +10945,11 @@ static struct cftype cpu_legacy_files[] = { .read_s64 = cpu_idle_read_s64, .write_s64 = cpu_idle_write_s64, }, + { + .name = "latency.nice", + .read_s64 = cpu_latency_nice_read_s64, + .write_s64 = cpu_latency_nice_write_s64, + }, #endif #ifdef CONFIG_CFS_BANDWIDTH { @@ -11121,6 +11167,12 @@ static struct cftype cpu_files[] = { .read_s64 = cpu_idle_read_s64, .write_s64 = cpu_idle_write_s64, }, + { + .name = "latency.nice", + .flags = CFTYPE_NOT_ON_ROOT, + .read_s64 = cpu_latency_nice_read_s64, + .write_s64 = cpu_latency_nice_write_s64, + }, #endif #ifdef CONFIG_CFS_BANDWIDTH { diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 4299d5108dc7..9583936ce30c 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -11764,6 +11764,7 @@ int alloc_fair_sched_group(struct task_group *tg, struct task_group *parent) goto err; tg->shares = NICE_0_LOAD; + tg->latency_offset = 0; init_cfs_bandwidth(tg_cfs_bandwidth(tg)); @@ -11862,6 +11863,9 @@ void init_tg_cfs_entry(struct task_group *tg, struct cfs_rq *cfs_rq, } se->my_q = cfs_rq; + + se->latency_offset = tg->latency_offset; + /* guarantee group entities always have weight */ update_load_set(&se->load, NICE_0_LOAD); se->parent = parent; @@ -11992,6 +11996,35 @@ int sched_group_set_idle(struct task_group *tg, long idle) return 0; } +int sched_group_set_latency(struct task_group *tg, s64 latency) +{ + int i; + + if (tg == &root_task_group) + return -EINVAL; + + if (abs(latency) > sysctl_sched_latency) + return -EINVAL; + + mutex_lock(&shares_mutex); + + if (tg->latency_offset == latency) { + mutex_unlock(&shares_mutex); + return 0; + } + + tg->latency_offset = latency; + + for_each_possible_cpu(i) { + struct sched_entity *se = tg->se[i]; + + WRITE_ONCE(se->latency_offset, latency); + } + + mutex_unlock(&shares_mutex); + return 0; +} + #else /* CONFIG_FAIR_GROUP_SCHED */ void free_fair_sched_group(struct task_group *tg) { } diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 99f10b4dc230..95d4be4f3af6 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -407,6 +407,8 @@ struct task_group { /* A positive value indicates that this is a SCHED_IDLE group. */ int idle; + /* latency constraint of the group. */ + int latency_offset; #ifdef CONFIG_SMP /* @@ -517,6 +519,8 @@ extern int sched_group_set_shares(struct task_group *tg, unsigned long shares); extern int sched_group_set_idle(struct task_group *tg, long idle); +extern int sched_group_set_latency(struct task_group *tg, s64 latency); + #ifdef CONFIG_SMP extern void set_task_rq_fair(struct sched_entity *se, struct cfs_rq *prev, struct cfs_rq *next); From patchwork Thu Nov 10 17:50:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 18251 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp286416wru; Thu, 10 Nov 2022 09:52:46 -0800 (PST) X-Google-Smtp-Source: AMsMyM4hD2yYFu5369oDTwoVNViR07bZZVzD1b5BIeQwFCSKuGwmhLevzmb1x/W+Nx2cwxOogHxa X-Received: by 2002:a17:90a:f507:b0:211:e623:da5b with SMTP id cs7-20020a17090af50700b00211e623da5bmr85651159pjb.37.1668102766126; Thu, 10 Nov 2022 09:52:46 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668102766; cv=none; d=google.com; s=arc-20160816; b=xMZ7Yto9sPyRFiDuLdd4VKoMox2zxIE5s+RKlXxXj3oXKr7lTWOkv8kAl7pNwEwn0z JDBqyzp2kj7mRc0vE5oWKkExbsZ1R3z01GnW45dEPYNLBJ37snhPHfcd2GaleDdjGaBW qZEPebjAxtDvgfhZAp3EInSYMcC4ClHUbXvub4vcFK9J2tBfMH1lJbQd6mn+eb0o0MI7 Huy3BAgoKG2LrBVwCoLCJv7ZsySKKElH/GmmB3nrKe+CbGsSURin8X3rKZ1AY83+0fn0 0J59j9i68psHqTnhv1BoW1Xmmxe9Nw+n6aa2GVwE3xAQ2ZxJ4FGsZfhfP6CCHaDEnLli vJWg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=BdpGOqqRR51x8v+WZgI/bsaGNfHn7spXAbo4XhwMNf4=; b=NlVHiLAuWtDxvOHpPSVfMVv2tgko/dsstEDdVMUh9y8G4SY4pvYiDvDrl+yhG6IFl0 Wj0gPOWhvq+D3QpZx/R1b5uMKqE9W49BMGE58eSP9QiFMdyh5tDj37yB9IuiBUW3zAL5 zRoqBraqsipfekZpPwQJ1VLcCZbe1aGL8lKpfhB205top9gNFwVUI6ZiVQwtfrOh+IqR Qj+sp5lCtSPDss14j+Ffe/RAoTfQMBk25qqItjmHYS5/kaYB/n1LOFAKPB+VZD5td2y/ wGJla5jhBHgDM4H86mjmC3NqiRw4HA4naamecqNNcyrcqvnIMuaRgFisZUJUxRGPnmHa N9ag== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=MfRrRFFN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id z16-20020aa78890000000b00557e99763d3si22892900pfe.93.2022.11.10.09.52.31; Thu, 10 Nov 2022 09:52:46 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=MfRrRFFN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231487AbiKJRuy (ORCPT + 99 others); Thu, 10 Nov 2022 12:50:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34192 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231419AbiKJRuk (ORCPT ); Thu, 10 Nov 2022 12:50:40 -0500 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B0E654B983 for ; Thu, 10 Nov 2022 09:50:32 -0800 (PST) Received: by mail-wm1-x329.google.com with SMTP id a11-20020a05600c2d4b00b003cf6f5fd9f1so1660477wmg.2 for ; Thu, 10 Nov 2022 09:50:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=BdpGOqqRR51x8v+WZgI/bsaGNfHn7spXAbo4XhwMNf4=; b=MfRrRFFNTQcZnBIPJiun3xx2yPDz6mhtY8iK0hjwh8shwr6WuAXyJAVwIZdhJhiy2/ H6TalqqeWvAmhwv5EHslh2+56IV8ZXdFhOWtTT1mJAPuy7uoJq+ibsy2w1R0NWhhf3Ne uu8IxmNsOs/9ya2GlPxF+Rl+9hX/mshANAFmqihmGxQ8OQ9OEp5Uzt3gZC+MtYSKDrUa i40+tWAZoz1SXLvKA9+5soeZOMgz3ZA2x8iSjvENiKfjnycnSzKFKylJgVMXF3r/TjIz itA1GWBbSniDilCTj8N3iB5e8OYnj43RByhhcvaInKu9HwrfqEjDBBNaw4TACdWxI0go FYvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BdpGOqqRR51x8v+WZgI/bsaGNfHn7spXAbo4XhwMNf4=; b=FY3bW4QOKuW1AUG7CEbyzaORlrgAbkLYIUD32MK575Nhrru4SrGO+mefq7obfFXhFS tw1zUjIrW4ufDSMUW/gfsKxiR+KavNnU8ApvwvB0oUP/ad+6cFOpD9r6QfMOz8GopDO0 yHb/mOxrUPiFferNbNZTbw0QVOpZ3mL0nFYhyPZNeFv8t48zL8TIBg4TaFjwK3qF8pN0 NkPIjN4B9OS5EEzuZJS99bZjqQ180sWFM1m1l8WnuNhuspNcAaNANjtOXRnzw/IXSzr9 3y/FnXNLQtiD0KjgXNE7jnryuUGq9c56yyCskMZUt9I4X23IfOOrrebiEfwMsQaQyS7T bz0A== X-Gm-Message-State: ACrzQf2jrYK3HZ66UrJQrYkWv4FUDioNuuqpviKvsnv4s1B8sftLroON 6KN8Wy2STpCRgVtTixuC3d1jrA== X-Received: by 2002:a05:600c:1508:b0:3cf:6cc1:c3b4 with SMTP id b8-20020a05600c150800b003cf6cc1c3b4mr41599484wmg.156.1668102631192; Thu, 10 Nov 2022 09:50:31 -0800 (PST) Received: from localhost.localdomain ([2a01:e0a:f:6020:342a:468f:562a:9cc4]) by smtp.gmail.com with ESMTPSA id f6-20020a05600c4e8600b003cfc02ab8basm6514677wmq.33.2022.11.10.09.50.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Nov 2022 09:50:30 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com Cc: qyousef@layalina.io, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v8 7/9] sched/core: Support latency priority with sched core Date: Thu, 10 Nov 2022 18:50:07 +0100 Message-Id: <20221110175009.18458-8-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221110175009.18458-1-vincent.guittot@linaro.org> References: <20221110175009.18458-1-vincent.guittot@linaro.org> X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749132526053879480?= X-GMAIL-MSGID: =?utf-8?q?1749132526053879480?= Take into account wakeup_latency_gran() when ordering the cfs threads. Signed-off-by: Vincent Guittot --- kernel/sched/fair.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 9583936ce30c..a7372f80b1ea 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -11439,6 +11439,9 @@ bool cfs_prio_less(struct task_struct *a, struct task_struct *b, bool in_fi) delta = (s64)(sea->vruntime - seb->vruntime) + (s64)(cfs_rqb->min_vruntime_fi - cfs_rqa->min_vruntime_fi); + /* Take into account latency prio */ + delta -= wakeup_latency_gran(sea, seb); + return delta > 0; } #else From patchwork Thu Nov 10 17:50:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 18254 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp286561wru; Thu, 10 Nov 2022 09:53:04 -0800 (PST) X-Google-Smtp-Source: AMsMyM4Xy8RLJQy4QJmUn3TIFQ2UiaORe1CBToB0cvg52k/L6w2WrLfXRhILZPecDLnLYFIcCE9R X-Received: by 2002:a17:903:1306:b0:17e:539:c405 with SMTP id iy6-20020a170903130600b0017e0539c405mr1592640plb.53.1668102783725; Thu, 10 Nov 2022 09:53:03 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668102783; cv=none; d=google.com; s=arc-20160816; b=yRlJLJP/naKOinSPaBfo3wAIJQQ44jKnumdJx8pgD3E/93w9MdmPnb7meq5KUbOz5R pH1PcVlVuB8zJVgKOXHjj6ASXuLH2f1jpBq9LfvFNPWGKceNjEjcZDFCj3QIYibPToqE qHw8SRlyC3iptoh4LZhDvInwLWW3araCm1ptnbfsrP4RIop+MnLTEqOzaBYRkBteDBAL mqX1+6466ddagp32n1qRx0YkWJLKsQjSU46c9hBhQndooPUvixR7M529o/P/oA9rl7Xb 6KVL3fZ4l6+xJejCWl/Q7kRm/bky44DreC2NnA0mTdgDXV/PjahAVGfQrnXfMnR9jgjR lsiQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=jbAeTvweSkPP6agxaj22JMn30F3trtvlIj14CMAmqB0=; b=nsPSHyAsrt4ViBOrF4BhTbfzxtJAUHisyT/rT4/H/F0GVnvz+QUqhgcRWuiC0gveXh FXanuDnC8I+XC/6MI+LP7YwD6yHhqhsgaoQzcSAHTXGfRW8oTfUDvmvyRw391LeErMV/ tdmYwSvv7j/lrEBQgV0Ys47Ozcw3tQrxKHxmf0dVdPqT4WZCQBhbMYSkvybLKoPavzcr GtOthFFngrIk7Z32aprXBavb24yW5C0mYO+ZLVBx93n7kzpcVnV61ogGg8hDC/Pg1t+8 eoDynV76HpYaVxqPwLcQSM6siRxvvIas0aK/SpTBeatj0uUsjH1gFh++PiXq34BBjHxS 1b7g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=bmdUgTf4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h6-20020a170902f7c600b00176d5b20ebesi18617880plw.355.2022.11.10.09.52.49; Thu, 10 Nov 2022 09:53:03 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=bmdUgTf4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231678AbiKJRvH (ORCPT + 99 others); Thu, 10 Nov 2022 12:51:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33902 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231364AbiKJRul (ORCPT ); Thu, 10 Nov 2022 12:50:41 -0500 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 15E1945A1D for ; Thu, 10 Nov 2022 09:50:35 -0800 (PST) Received: by mail-wr1-x42d.google.com with SMTP id z14so3357354wrn.7 for ; Thu, 10 Nov 2022 09:50:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=jbAeTvweSkPP6agxaj22JMn30F3trtvlIj14CMAmqB0=; b=bmdUgTf4gcVpCxglkDR3uxYUaXJknyaQYgzpJiDFLJ94DvHOtzANYn0e72SN5bOsI0 ojUXdf3NQWy04FTS9ZPuggZUteO8WqEaaVT9cDmhl304L432LjMFKnQQ19dBwPhwWIx9 mRZkCG/++jW9SjfRQbwYF4RorYRtkC/OGOic/Rj8pUV3g9XBbfi1Vy/7MVod8/g90dJ5 sAoGhNKvRFQN/IbJQZBs9K6XZ4qaNaVexOqedCTk5yvu7y+TXnZtgwNwhjk2O6IwkY50 Eauybi+D1ZS6mm4gWf6CxzPoE+TmfI8dsNGWWZ3+Ufp96eTmmcUlkdN9ZbtJ/3c3zguK a8Hg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=jbAeTvweSkPP6agxaj22JMn30F3trtvlIj14CMAmqB0=; b=Oka5Z0/ktkG9G67XRPSgqYnpbcKiLujOiahWgcRDhkMb3nkjUwb9GRxaK/BDsOQD90 jd33juuPNJAlRqnjA9SeSEAFGk/SYr8nxNwSjW8pfGDDe/n7wBqw4CHKZqe37S3kQWDq hRBRMt1Mi9QSxmHJRlTmkTFA8mg/EprfSgvJveNOjGoUX/WRwvMIoUYbvwRGNCz3xvri 8T9x7HvE1VE63H4fASj1rPnCJ6zhhOErjatZMBeZM6ojtjl2Lrnv+Pz+GaovZjelT+aH 9OS0v5YBc4EeBthbw+djqhmorXD1pp27McSBwx6CUp/TMNHza/HV90ahvOfQj3o/qHux dfzw== X-Gm-Message-State: ACrzQf2WeECQ/vP9iSg/aPrWHUIWVCV7T5QMAEu9MKpLCPiDUO+ZzkbP FcSSxY1PzwwCFCwG1z1CPMNoeQ== X-Received: by 2002:a5d:4107:0:b0:236:58e5:290d with SMTP id l7-20020a5d4107000000b0023658e5290dmr1030819wrp.2.1668102633422; Thu, 10 Nov 2022 09:50:33 -0800 (PST) Received: from localhost.localdomain ([2a01:e0a:f:6020:342a:468f:562a:9cc4]) by smtp.gmail.com with ESMTPSA id f6-20020a05600c4e8600b003cfc02ab8basm6514677wmq.33.2022.11.10.09.50.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Nov 2022 09:50:32 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com Cc: qyousef@layalina.io, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v8 8/9] sched/fair: Add latency list Date: Thu, 10 Nov 2022 18:50:08 +0100 Message-Id: <20221110175009.18458-9-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221110175009.18458-1-vincent.guittot@linaro.org> References: <20221110175009.18458-1-vincent.guittot@linaro.org> X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749132544395624608?= X-GMAIL-MSGID: =?utf-8?q?1749132544395624608?= Add a rb tree for latency sensitive entities so we can schedule the most sensitive one first even when it failed to preempt current at wakeup or when it got quickly preempted by another entity of higher priority. In order to keep fairness, the latency is used once at wakeup to get a minimum slice and not during the following scheduling slice to prevent long running entity to got more running time than allocated to his nice priority. The rb tree enables to cover the last corner case where latency sensitive entity can't got schedule quickly after the wakeup. Signed-off-by: Vincent Guittot --- include/linux/sched.h | 1 + kernel/sched/core.c | 1 + kernel/sched/fair.c | 95 +++++++++++++++++++++++++++++++++++++++++-- kernel/sched/sched.h | 1 + 4 files changed, 95 insertions(+), 3 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index a74cad08e91e..45b8c36e64cc 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -547,6 +547,7 @@ struct sched_entity { /* For load-balancing: */ struct load_weight load; struct rb_node run_node; + struct rb_node latency_node; struct list_head group_node; unsigned int on_rq; diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 3f42b1f61a7e..c6c67677d71f 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4342,6 +4342,7 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p) p->se.nr_migrations = 0; p->se.vruntime = 0; INIT_LIST_HEAD(&p->se.group_node); + RB_CLEAR_NODE(&p->se.latency_node); #ifdef CONFIG_FAIR_GROUP_SCHED p->se.cfs_rq = NULL; diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index a7372f80b1ea..fb4973a87f25 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -664,7 +664,76 @@ struct sched_entity *__pick_last_entity(struct cfs_rq *cfs_rq) return __node_2_se(last); } +#endif +/************************************************************** + * Scheduling class tree data structure manipulation methods: + * for latency + */ + +static inline bool latency_before(struct sched_entity *a, + struct sched_entity *b) +{ + return (s64)(a->vruntime + a->latency_offset - b->vruntime - b->latency_offset) < 0; +} + +#define __latency_node_2_se(node) \ + rb_entry((node), struct sched_entity, latency_node) + +static inline bool __latency_less(struct rb_node *a, const struct rb_node *b) +{ + return latency_before(__latency_node_2_se(a), __latency_node_2_se(b)); +} + +/* + * Enqueue an entity into the latency rb-tree: + */ +static void __enqueue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) +{ + + /* Only latency sensitive entity can be added to the list */ + if (se->latency_offset >= 0) + return; + + if (!RB_EMPTY_NODE(&se->latency_node)) + return; + + /* + * An execution time less than sysctl_sched_min_granularity means that + * the entity has been preempted by a higher sched class or an entity + * with higher latency constraint. + * Put it back in the list so it gets a chance to run 1st during the + * next slice. + */ + if (!(flags & ENQUEUE_WAKEUP)) { + u64 delta_exec = se->sum_exec_runtime - se->prev_sum_exec_runtime; + + if (delta_exec >= sysctl_sched_min_granularity) + return; + } + + rb_add_cached(&se->latency_node, &cfs_rq->latency_timeline, __latency_less); +} + +static void __dequeue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se) +{ + if (!RB_EMPTY_NODE(&se->latency_node)) { + rb_erase_cached(&se->latency_node, &cfs_rq->latency_timeline); + RB_CLEAR_NODE(&se->latency_node); + } +} + +static struct sched_entity *__pick_first_latency(struct cfs_rq *cfs_rq) +{ + struct rb_node *left = rb_first_cached(&cfs_rq->latency_timeline); + + if (!left) + return NULL; + + return __latency_node_2_se(left); +} + +#ifdef CONFIG_SCHED_DEBUG /************************************************************** * Scheduling class statistics methods: */ @@ -4439,8 +4508,10 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) check_schedstat_required(); update_stats_enqueue_fair(cfs_rq, se, flags); check_spread(cfs_rq, se); - if (!curr) + if (!curr) { __enqueue_entity(cfs_rq, se); + __enqueue_latency(cfs_rq, se, flags); + } se->on_rq = 1; if (cfs_rq->nr_running == 1) { @@ -4526,8 +4597,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) clear_buddies(cfs_rq, se); - if (se != cfs_rq->curr) + if (se != cfs_rq->curr) { __dequeue_entity(cfs_rq, se); + __dequeue_latency(cfs_rq, se); + } se->on_rq = 0; account_entity_dequeue(cfs_rq, se); @@ -4616,6 +4689,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se) */ update_stats_wait_end_fair(cfs_rq, se); __dequeue_entity(cfs_rq, se); + __dequeue_latency(cfs_rq, se); update_load_avg(cfs_rq, se, UPDATE_TG); } @@ -4654,7 +4728,7 @@ static struct sched_entity * pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr) { struct sched_entity *left = __pick_first_entity(cfs_rq); - struct sched_entity *se; + struct sched_entity *latency, *se; /* * If curr is set we have to see if its left of the leftmost entity @@ -4696,6 +4770,12 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr) se = cfs_rq->last; } + /* Check for latency sensitive entity waiting for running */ + latency = __pick_first_latency(cfs_rq); + if (latency && (latency != se) && + wakeup_preempt_entity(latency, se) < 1) + se = latency; + return se; } @@ -4719,6 +4799,7 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev) update_stats_wait_start_fair(cfs_rq, prev); /* Put 'current' back into the tree. */ __enqueue_entity(cfs_rq, prev); + __enqueue_latency(cfs_rq, prev, 0); /* in !on_rq case, update occurred at dequeue */ update_load_avg(cfs_rq, prev, 0); } @@ -11712,6 +11793,7 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first) void init_cfs_rq(struct cfs_rq *cfs_rq) { cfs_rq->tasks_timeline = RB_ROOT_CACHED; + cfs_rq->latency_timeline = RB_ROOT_CACHED; u64_u32_store(cfs_rq->min_vruntime, (u64)(-(1LL << 20))); #ifdef CONFIG_SMP raw_spin_lock_init(&cfs_rq->removed.lock); @@ -12020,8 +12102,15 @@ int sched_group_set_latency(struct task_group *tg, s64 latency) for_each_possible_cpu(i) { struct sched_entity *se = tg->se[i]; + struct rq *rq = cpu_rq(i); + struct rq_flags rf; + + rq_lock_irqsave(rq, &rf); + __dequeue_latency(se->cfs_rq, se); WRITE_ONCE(se->latency_offset, latency); + + rq_unlock_irqrestore(rq, &rf); } mutex_unlock(&shares_mutex); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 95d4be4f3af6..91ec36c1158b 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -599,6 +599,7 @@ struct cfs_rq { #endif struct rb_root_cached tasks_timeline; + struct rb_root_cached latency_timeline; /* * 'curr' points to currently running entity on this cfs_rq. From patchwork Thu Nov 10 17:50:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 18255 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp290477wru; Thu, 10 Nov 2022 10:01:47 -0800 (PST) X-Google-Smtp-Source: AMsMyM7AaqbZG8b8KnlnGg2hU8IqepGomie0mXQdFJyWGY6d//DKkDL8/CKI6KiWMw01sCtpXEbx X-Received: by 2002:a17:902:988e:b0:186:907a:a029 with SMTP id s14-20020a170902988e00b00186907aa029mr66449323plp.97.1668103306724; Thu, 10 Nov 2022 10:01:46 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668103306; cv=none; d=google.com; s=arc-20160816; b=nJtjNW27Vzxws7l3NWU7qbXM6q81CxSfSlmVHhFwdEmap68uqv1TLNtXJXfPoWPPHV qfQhbz6aLsd6b7Rv6DPKFUKX+wdPd6QwxOQa4/INwVzklwiKrmNmxRJtL+vDkDFKQxtZ 1XVttgtXCtQnHhdaB+gUNWs1fQnGVIvD2BKsCkf6nFJXQhRZrIlUfDI47FxWWSGIBV4f g9+DMkrn0Q+cpTNBA3z+bwdv+ERvyo3fqbGb236I/+/vhIRQ+3hbxG2SHYHCuHCROHkk pOLvPNFvi1kq0iMgN5lgOT1HL5nZDINn0y2TXpYj3GpKoxMwKnWeXXN5nrSpTwJnTnIa a0Tw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=eKRDIrK6+9JpxczLCDYf+I+TauedG4U/HEai4Cf8rvo=; b=GUDgbGQ0OHX5ESIKOChkF4rLtKIQHHRFnQ4CunB0xsgbYCGtM/gxcxixW5Wlz7WKZ1 jCJZmFDiNfmjd3QgH4eVcPmzNFohNFOXYjKeK/RuJSk8tI3J/j7Utyov98/fU2xtv/w9 f+FjIllIFCRGkcmGZz6W0oXTMvt9lns3wSJyo7rU4HxhOYOOMj8jHcGkBymC321voGLU tPcLCOv1F6Wery9+pomsfevkrdRLLbl2QYq/QEzesiaWclrQfXtn0xZGS/gi7D4J2/Ru xhc5y2bCDw4MdWh7LXNduHml4z3obgNd8i+KZXyHFNpRFgmICCl26Zbp19P6Ed4bYEdw UmOA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=cai+412m; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 33-20020a630e61000000b004610dceef13si22235369pgo.336.2022.11.10.10.01.30; Thu, 10 Nov 2022 10:01:46 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=cai+412m; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231696AbiKJRvJ (ORCPT + 99 others); Thu, 10 Nov 2022 12:51:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34220 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231472AbiKJRul (ORCPT ); Thu, 10 Nov 2022 12:50:41 -0500 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2F8E24B9A4 for ; Thu, 10 Nov 2022 09:50:37 -0800 (PST) Received: by mail-wr1-x42c.google.com with SMTP id v1so3337746wrt.11 for ; Thu, 10 Nov 2022 09:50:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=eKRDIrK6+9JpxczLCDYf+I+TauedG4U/HEai4Cf8rvo=; b=cai+412mPBIM/ZU05LnrpGJuBoUbQunUn+PqMS7ewH09C41D7pj4tPDKTiMzWcjVPt pxlV3Rs2lT0I4VGRLZdYdHxEKOVWY1SV/Hp8tFBqOZnMOcTXIXiRGw16ooPxkxeGn1Cf AdZQ0TmPKk3NAiFU8QU6PB5tms2YbQyUKyV7BY5HLibiiiiT19kwmGapDL3fBTSPgTvP WZGIZvZiA8q7ovYLCGtWjjjVj/Cep8vzhSAAynNdz5yEQJOPma/acfgYk47KeOqpe4Zs 99qsBxxVIL5kCLDsXXxkjLxo2se2TrsrkQKD04e7jheo1vC+QzG2Z55YyXiviDx3+D/f fjyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=eKRDIrK6+9JpxczLCDYf+I+TauedG4U/HEai4Cf8rvo=; b=CQ/o1scj8AZ0MPi/znrYv3yrSNM1kF3J3Qsen5a9G1+GAy9Jyp62xn9Mn5OphtIJ3N D/7T4Arc3ECBgiLiYlEswfXwujRfdH+FwofYzhdFGjNuJnv9xxTc2fP0RfIC353fcBbK 2nmCKWLvARVNs4mm0rglbthSbtFEnbCKyuUW/irZWzYgd6xiQMQZBMYD7Og4jQrSClez kwg+rGTgiQrm8qkshoJphl7UGje6wQGJ7PVO5aFS7CJotr0ZJdz6FilxqCtgBw/COpdH Zbp9EQHFwD/1XSwqQfR0imIejzacTYiXgqhaa8Pai+C8f2Lae34KV0fCt+8qe6pyktHw 54pg== X-Gm-Message-State: ACrzQf2vRo2dHYGZaQiBEvrZVJ5q6cF3Mzbjz1RvhZzw/+CK1DciGof0 hdO+20lcVyj795D9oeJqcg53UA== X-Received: by 2002:adf:f84c:0:b0:236:6e52:504 with SMTP id d12-20020adff84c000000b002366e520504mr40614748wrq.564.1668102635719; Thu, 10 Nov 2022 09:50:35 -0800 (PST) Received: from localhost.localdomain ([2a01:e0a:f:6020:342a:468f:562a:9cc4]) by smtp.gmail.com with ESMTPSA id f6-20020a05600c4e8600b003cfc02ab8basm6514677wmq.33.2022.11.10.09.50.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Nov 2022 09:50:34 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com Cc: qyousef@layalina.io, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v8 9/9] sched/fair: remove check_preempt_from_others Date: Thu, 10 Nov 2022 18:50:09 +0100 Message-Id: <20221110175009.18458-10-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221110175009.18458-1-vincent.guittot@linaro.org> References: <20221110175009.18458-1-vincent.guittot@linaro.org> X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749133092550687096?= X-GMAIL-MSGID: =?utf-8?q?1749133092550687096?= With the dedicated latency list, we don't have to take care of this special case anymore as pick_next_entity checks for a runnable latency sensitive task. Signed-off-by: Vincent Guittot --- kernel/sched/fair.c | 34 ++-------------------------------- 1 file changed, 2 insertions(+), 32 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index fb4973a87f25..c2c75d531612 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -5801,35 +5801,6 @@ static int sched_idle_cpu(int cpu) } #endif -static void set_next_buddy(struct sched_entity *se); - -static void check_preempt_from_others(struct cfs_rq *cfs, struct sched_entity *se) -{ - struct sched_entity *next; - - if (se->latency_offset >= 0) - return; - - if (cfs->nr_running <= 1) - return; - /* - * When waking from another class, we don't need to check to preempt at - * wakeup and don't set next buddy as a candidate for being picked in - * priority. - * In case of simultaneous wakeup when current is another class, the - * latency sensitive tasks lost opportunity to preempt non sensitive - * tasks which woke up simultaneously. - */ - - if (cfs->next) - next = cfs->next; - else - next = __pick_first_entity(cfs); - - if (next && wakeup_preempt_entity(next, se) == 1) - set_next_buddy(se); -} - /* * The enqueue_task method is called before nr_running is * increased. Here we update the fair scheduling stats and @@ -5916,15 +5887,14 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) if (!task_new) update_overutilized_status(rq); - if (rq->curr->sched_class != &fair_sched_class) - check_preempt_from_others(cfs_rq_of(&p->se), &p->se); - enqueue_throttle: assert_list_leaf_cfs_rq(rq); hrtick_update(rq); } +static void set_next_buddy(struct sched_entity *se); + /* * The dequeue_task method is called before nr_running is * decreased. We remove the task from the rbtree and