From patchwork Fri Jan 13 14:12:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 43323 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp293566wrt; Fri, 13 Jan 2023 06:17:02 -0800 (PST) X-Google-Smtp-Source: AMrXdXvdAt7LsUQ9jnQ/Vi/Wvrt4pUdB1SuqKavSvZEbSfgIwmjfCSdOZa+XKgoTr5UE4nqOzbVW X-Received: by 2002:a17:907:9006:b0:84d:411d:64a4 with SMTP id ay6-20020a170907900600b0084d411d64a4mr2949171ejc.38.1673619422698; Fri, 13 Jan 2023 06:17:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673619422; cv=none; d=google.com; s=arc-20160816; b=icVoMK4BGa1OPkH9QTLVUgw1tGiTn4+Os9cOYnp/kw+Q50BI25wN4uI9AGFvg5UI8p 0493kJHqVVj/ocSdqgf/yIwP5KRZaJrZxRCHjypSFraKdJ6kpNr+QEU2VZUJ1glqVYUB T+wC1+VMzgQRzyVNuVU92C46qdx+Djl9TJ1esnE/OKzo2x85u27rIcEIAl7E8aKfukkB ZzwWsQYoE77VYjBDzlU8E6f5Atz4Mybx5d5DFqv0MuK1PyJ7Z3g5g2fNxMdbnr1OcL9n 0vPq1FXDptEHbmEfLf2Ep8qn7OHiW3QE6ppQBiwrC3G9HRs8VdifK2/wW9p+QYw/3pk6 IGig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=zlloFS7Bj1vz8MB/gY8ujw3edPCeBj7FeQ8wX2SHm40=; b=08QSwMWN0CIh2vpxtRVvY8jyOZCdOcKBAfvs0pEoOV9Guvcy5jrzOSD+7T+076ofPB KJmXFn8FHbWhTZCBxLjTRwIUldPP+twC60Tn7EBUH+m1E6+NgcmKcgADjJlxgJNTSLig Qw93ZpjorATy6cP/uDatds5m5KZQIc+HpIORiX9Kcgf0tswAJmZasJVQiKRg6t3noPrF gT//Wh8M2X77zxqjHLjvHcnSncWXUYHes2wQ/LxoXljp6xwmy+ess9M43OQ8ssUWDXLp 0pLfkf3TxCzOsT2NDqD9O4OIswYkCsuRvxuBx3dKT2m2/0Nz8pFzAB6xdKmgCHhoRE7+ as5w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=XM70w7KZ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id bb6-20020a1709070a0600b0084e3eac46d5si12675344ejc.729.2023.01.13.06.16.37; Fri, 13 Jan 2023 06:17:02 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=XM70w7KZ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241332AbjAMOO5 (ORCPT + 99 others); Fri, 13 Jan 2023 09:14:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40476 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242013AbjAMONs (ORCPT ); Fri, 13 Jan 2023 09:13:48 -0500 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 47F0DCD9 for ; Fri, 13 Jan 2023 06:12:41 -0800 (PST) Received: by mail-wr1-x42f.google.com with SMTP id bs20so21200179wrb.3 for ; Fri, 13 Jan 2023 06:12:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=zlloFS7Bj1vz8MB/gY8ujw3edPCeBj7FeQ8wX2SHm40=; b=XM70w7KZMtRcIGeyzBW5f5aEzTdbfuDdmw3koOm1IKpbkMOynQ0dDy1sXObUQxUUWD ylsbugkfXukjgr2ZrF4CzFZZTDMJZZA1RLYrwIZA/c4qJD3Ne3i15wxjLNQoDiDI7u4O S6WBxFtLE52OjuIVEbFG3wzgvhrTd+GYe+5fW+thyzxDa58mJvNW3vuKNoZ7JCaSo7c9 BM7MoMBTJQIDEUqt5PxdWglEbTdmA5eU27MKeQEmEJGzRP5loLiU+kHmIctlVzW8XtFt PVHXElyLfJnlio7LdY04U36p+JckjuSQhhcDlFuFeLD+PVqecnaxEnK66LD+w7XOZ0NP 9XmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zlloFS7Bj1vz8MB/gY8ujw3edPCeBj7FeQ8wX2SHm40=; b=YxtAhLVGZW/+/gllAW79MWR1Atf/28AyN0ad3yE3TjV70L1cN3/KxX6cJfIyTgt63y QPnPwfcY9nv+cmkN1lEyfKSNoVxh/+EpY5KLit4h46vbJIrigvrFwiAgdJqpHUXU6A6h MiImsUfyihyKWPco+OejCP41ZQ9FM2/pjbqvOlKhXtkhTAMEL87Yxfm2Z2XHenBKnpJY 9k8ktILcDEgCiGUJwW6x1WK2GkVxs35sovwuScyU9hUR3vhcehIG3fHxDsUOIjNvU/pE NrT34mPpZ5uY3cAgtUO5x8CuDvOK5PeGC3AX009IC0zE8o4HgmDKwc44pFL04m7F07WN kSjg== X-Gm-Message-State: AFqh2kpd/KNVqlJ8CF5iN6JsxXrCAUHjnRuYh88EMLDqz0GgRr9LpJqE lNN85IX+lmpudkYPZlOIPtAE9w== X-Received: by 2002:a5d:42c7:0:b0:2bc:7f49:d276 with SMTP id t7-20020a5d42c7000000b002bc7f49d276mr10712356wrr.46.1673619159779; Fri, 13 Jan 2023 06:12:39 -0800 (PST) Received: from vingu-book.. ([2a01:e0a:f:6020:1563:65bf:c344:661e]) by smtp.gmail.com with ESMTPSA id f7-20020a5d6647000000b002bbeb700c38sm13869919wrw.91.2023.01.13.06.12.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Jan 2023 06:12:39 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com, cgroups@vger.kernel.org Cc: qyousef@layalina.io, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v10 1/9] sched/fair: fix unfairness at wakeup Date: Fri, 13 Jan 2023 15:12:26 +0100 Message-Id: <20230113141234.260128-2-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230113141234.260128-1-vincent.guittot@linaro.org> References: <20230113141234.260128-1-vincent.guittot@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DIET_1,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754917159737326612?= X-GMAIL-MSGID: =?utf-8?q?1754917159737326612?= At wake up, the vruntime of a task is updated to not be more older than a sched_latency period behind the min_vruntime. This prevents long sleeping task to get unlimited credit at wakeup. Such waking task should preempt current one to use its CPU bandwidth but wakeup_gran() can be larger than sched_latency, filter out the wakeup preemption and as a results steals some CPU bandwidth to the waking task. Make sure that a task, which vruntime has been capped, will preempt current task and use its CPU bandwidth even if wakeup_gran() is in the same range as sched_latency. If the waking task failed to preempt current it could to wait up to sysctl_sched_min_granularity before preempting it during next tick. Strictly speaking, we should use cfs->min_vruntime instead of curr->vruntime but it doesn't worth the additional overhead and complexity as the vruntime of current should be close to min_vruntime if not equal. Reported-by: Youssef Esmat Signed-off-by: Vincent Guittot Reviewed-by: Joel Fernandes (Google) Tested-by: K Prateek Nayak --- kernel/sched/fair.c | 46 ++++++++++++++++++++------------------------ kernel/sched/sched.h | 34 +++++++++++++++++++++++++++++++- 2 files changed, 54 insertions(+), 26 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index e9d906a9bba9..8a85c6cf781e 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4657,33 +4657,17 @@ place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int initial) { u64 vruntime = cfs_rq->min_vruntime; - /* - * The 'current' period is already promised to the current tasks, - * however the extra weight of the new task will slow them down a - * little, place the new task so that it fits in the slot that - * stays open at the end. - */ - if (initial && sched_feat(START_DEBIT)) - vruntime += sched_vslice(cfs_rq, se); - - /* sleeps up to a single latency don't count. */ - if (!initial) { - unsigned long thresh; - - if (se_is_idle(se)) - thresh = sysctl_sched_min_granularity; - else - thresh = sysctl_sched_latency; - + if (!initial) + /* sleeps up to a single latency don't count. */ + vruntime -= get_sleep_latency(se_is_idle(se)); + else if (sched_feat(START_DEBIT)) /* - * Halve their sleep time's effect, to allow - * for a gentler effect of sleepers: + * The 'current' period is already promised to the current tasks, + * however the extra weight of the new task will slow them down a + * little, place the new task so that it fits in the slot that + * stays open at the end. */ - if (sched_feat(GENTLE_FAIR_SLEEPERS)) - thresh >>= 1; - - vruntime -= thresh; - } + vruntime += sched_vslice(cfs_rq, se); /* ensure we never gain time by being placed backwards. */ se->vruntime = max_vruntime(se->vruntime, vruntime); @@ -7656,6 +7640,18 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se) return -1; gran = wakeup_gran(se); + + /* + * At wake up, the vruntime of a task is capped to not be older than + * a sched_latency period compared to min_vruntime. This prevents long + * sleeping task to get unlimited credit at wakeup. Such waking up task + * has to preempt current in order to not lose its share of CPU + * bandwidth but wakeup_gran() can become higher than scheduling period + * for low priority task. Make sure that long sleeping task will get a + * chance to preempt current. + */ + gran = min_t(s64, gran, get_latency_max()); + if (vdiff > gran) return 1; diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 1072502976df..df7db06c9943 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2459,9 +2459,9 @@ extern void check_preempt_curr(struct rq *rq, struct task_struct *p, int flags); extern const_debug unsigned int sysctl_sched_nr_migrate; extern const_debug unsigned int sysctl_sched_migration_cost; -#ifdef CONFIG_SCHED_DEBUG extern unsigned int sysctl_sched_latency; extern unsigned int sysctl_sched_min_granularity; +#ifdef CONFIG_SCHED_DEBUG extern unsigned int sysctl_sched_idle_min_granularity; extern unsigned int sysctl_sched_wakeup_granularity; extern int sysctl_resched_latency_warn_ms; @@ -2476,6 +2476,38 @@ extern unsigned int sysctl_numa_balancing_scan_size; extern unsigned int sysctl_numa_balancing_hot_threshold; #endif +static inline unsigned long get_sleep_latency(bool idle) +{ + unsigned long thresh; + + if (idle) + thresh = sysctl_sched_min_granularity; + else + thresh = sysctl_sched_latency; + + /* + * Halve their sleep time's effect, to allow + * for a gentler effect of sleepers: + */ + if (sched_feat(GENTLE_FAIR_SLEEPERS)) + thresh >>= 1; + + return thresh; +} + +static inline unsigned long get_latency_max(void) +{ + unsigned long thresh = get_sleep_latency(false); + + /* + * If the waking task failed to preempt current it could to wait up to + * sysctl_sched_min_granularity before preempting it during next tick. + */ + thresh -= sysctl_sched_min_granularity; + + return thresh; +} + #ifdef CONFIG_SCHED_HRTICK /* From patchwork Fri Jan 13 14:12:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 43324 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp293658wrt; Fri, 13 Jan 2023 06:17:10 -0800 (PST) X-Google-Smtp-Source: AMrXdXsdcu3gYPvbZOFCM0QS0z70suvGk+ls4dDIc6xuSZFtomfur0IYzS2BWW0UjsbqUI6pZd0F X-Received: by 2002:a17:906:2816:b0:7c0:d452:2e74 with SMTP id r22-20020a170906281600b007c0d4522e74mr69581684ejc.4.1673619430576; Fri, 13 Jan 2023 06:17:10 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673619430; cv=none; d=google.com; s=arc-20160816; b=FSw2pXN9qV9G0kX+LRhxo8ADlUaWCl3VMyMMauSyu4pRRyAaS3UWJSXStUOTTBdEMP pHq4o3HJa3/ZcqNPJOaRWxkdicT6r/2plefmlLkhLgNJcvKgyvsGiDSX4mOINqh8uE2J SGzWY78ehZcq1C4t0eIUZzqxlmmE7/ABfl6gfZy99pplIVVcD6V47n14AUsqZHLyFVtV gK4FtsV5OGzgnaTcmOVkmhbBiM2/rEt6tYl18n4vniw/fbW4r/TuSdsczTEQrED4Xdvz 5dTWT1fTiZTTR+wH5IozRP/TahCapY5juYrL5n3PTxbxe1Rtn08GLs/qHimtKxccaHo2 8p2g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=kiVS63wbAWm+QNHUVgj2Zht/JYfZt2Iaghb+UJPmGJA=; b=h/l9a41gadZ699/j3GwwpwYkiW4bu4juXbExYXxlZx3N67t0UgcZweDGu4F7PvC2IW QfkdZKPocFGs+FhGAPQRFZco904wxbN/kjR8KviC9waJiV0MVUstLmI5B/2NxuT7j+aI RoNVF1YiZo8W0czwj109SCQc4ODFoIy/s9RHGjOxgL09XarrJIuScEFOO6I2wR9ZunzA aW22gAHlnXb/JZT7i/B3q6d8weh8mMOy8R0aj0cCPdKrhTPCU8gxTuhbZftomLNm+hc6 85uDRtx5e4CGSGgY0SltPUkrXS/H515L432TEz5H3/3lFr+DBKdrdJ6qPpI8Ycvmy25A 7lsg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Qh85uD1Y; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id qb11-20020a1709077e8b00b0079dccbbf0c0si21017393ejc.146.2023.01.13.06.16.44; Fri, 13 Jan 2023 06:17:10 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Qh85uD1Y; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241565AbjAMOPF (ORCPT + 99 others); Fri, 13 Jan 2023 09:15:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40408 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242017AbjAMONs (ORCPT ); Fri, 13 Jan 2023 09:13:48 -0500 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D8B9A32E for ; Fri, 13 Jan 2023 06:12:42 -0800 (PST) Received: by mail-wr1-x432.google.com with SMTP id r2so21192064wrv.7 for ; Fri, 13 Jan 2023 06:12:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kiVS63wbAWm+QNHUVgj2Zht/JYfZt2Iaghb+UJPmGJA=; b=Qh85uD1YsNWfsXSd0d73mm0u8hDaGmsaRhIZGgLJv8V9BgvN+HigCnzW5b9dMcw03V LcFIYpE7cA4lzuSa8n9vvPMFUvGTtVjfNY9gpmAGy9CGlYRhkqXxwXn8Nl0IrWDqRUhe S4uaRLOt5G79qMoW8ItaBtopKB9BDhstDF+PQKqqhRIuTheyU/YuaGxhtjsfZivHQwVl 5epEhNrXjIJhyXgi3pMDMbwWSwcglnz+3DzHSPNhas1TcW4iyDzZwzhHFYm4S6884GVj SszmSBo/CQH+3x+o+201a6zSuaD3xxy9PeGmUEZCnVzq3ykO+k+v8MbDsyoeNyR1JihL In1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kiVS63wbAWm+QNHUVgj2Zht/JYfZt2Iaghb+UJPmGJA=; b=i/DQ/obIA0Sk7NFoHj7jTua7m8SPm4I/XeDn+uD8ViR1IwUt+M+pW5JmjLr5vye0IL bJFYNo3Rce6J/gfrvAb0VK4wKXHz599ROCGd++8YBLDhazPWJBkWhKgMdXU0vIPq5bHZ K3ltn2MdJxGs5BSxN3Yt9acWB6l5CJxpzzlwDAcBFGnSfIeTruupClc/Lf95/xfMWjtv PfYE76ZM3y4bDq5cf3EA8lq4+m+ckBt9v54iXg3F4zck+XLGQf3YyC+8ZUJgfKKUdvN0 yO4TvTt9ARBPj/BfCrXB2M7fNo7SBS866FGFuY8f63fAx+0PmsMuOzkDBv5T2kI3/ZeB zqxA== X-Gm-Message-State: AFqh2koQDWjNiAtV3MvyXI/gECkYUnrNXkE879gRuaiivQLQQ2XtQYwj kB5l2APRjnx4Lesp8UX785h/ZA== X-Received: by 2002:a5d:63c1:0:b0:2bd:bdb5:baf1 with SMTP id c1-20020a5d63c1000000b002bdbdb5baf1mr9347506wrw.10.1673619161369; Fri, 13 Jan 2023 06:12:41 -0800 (PST) Received: from vingu-book.. ([2a01:e0a:f:6020:1563:65bf:c344:661e]) by smtp.gmail.com with ESMTPSA id f7-20020a5d6647000000b002bbeb700c38sm13869919wrw.91.2023.01.13.06.12.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Jan 2023 06:12:40 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com, cgroups@vger.kernel.org Cc: qyousef@layalina.io, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v10 2/9] sched: Introduce latency-nice as a per-task attribute Date: Fri, 13 Jan 2023 15:12:27 +0100 Message-Id: <20230113141234.260128-3-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230113141234.260128-1-vincent.guittot@linaro.org> References: <20230113141234.260128-1-vincent.guittot@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754917168063858961?= X-GMAIL-MSGID: =?utf-8?q?1754917168063858961?= From: Parth Shah Latency-nice indicates the latency requirements of a task with respect to the other tasks in the system. The value of the attribute can be within the range of [-20, 19] both inclusive to be in-line with the values just like task nice values. latency_nice = -20 indicates the task to have the least latency as compared to the tasks having latency_nice = +19. The latency_nice may affect only the CFS SCHED_CLASS by getting latency requirements from the userspace. Additionally, add debugging bits for newly added latency_nice attribute. Signed-off-by: Parth Shah [rebase, move defines in sched/prio.h] Signed-off-by: Vincent Guittot Tested-by: K Prateek Nayak --- include/linux/sched.h | 1 + include/linux/sched/prio.h | 18 ++++++++++++++++++ kernel/sched/debug.c | 1 + 3 files changed, 20 insertions(+) diff --git a/include/linux/sched.h b/include/linux/sched.h index 4df2b3e76b30..6c61bde49152 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -784,6 +784,7 @@ struct task_struct { int static_prio; int normal_prio; unsigned int rt_priority; + int latency_nice; struct sched_entity se; struct sched_rt_entity rt; diff --git a/include/linux/sched/prio.h b/include/linux/sched/prio.h index ab83d85e1183..bfcd7f1d1e11 100644 --- a/include/linux/sched/prio.h +++ b/include/linux/sched/prio.h @@ -42,4 +42,22 @@ static inline long rlimit_to_nice(long prio) return (MAX_NICE - prio + 1); } +/* + * Latency nice is meant to provide scheduler hints about the relative + * latency requirements of a task with respect to other tasks. + * Thus a task with latency_nice == 19 can be hinted as the task with no + * latency requirements, in contrast to the task with latency_nice == -20 + * which should be given priority in terms of lower latency. + */ +#define MAX_LATENCY_NICE 19 +#define MIN_LATENCY_NICE -20 + +#define LATENCY_NICE_WIDTH \ + (MAX_LATENCY_NICE - MIN_LATENCY_NICE + 1) + +/* + * Default tasks should be treated as a task with latency_nice = 0. + */ +#define DEFAULT_LATENCY_NICE 0 + #endif /* _LINUX_SCHED_PRIO_H */ diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c index 1637b65ba07a..68be7a3e42a3 100644 --- a/kernel/sched/debug.c +++ b/kernel/sched/debug.c @@ -1043,6 +1043,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns, #endif P(policy); P(prio); + P(latency_nice); if (task_has_dl_policy(p)) { P(dl.runtime); P(dl.deadline); From patchwork Fri Jan 13 14:12:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 43331 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp294814wrt; Fri, 13 Jan 2023 06:19:26 -0800 (PST) X-Google-Smtp-Source: AMrXdXuUoNUHAE2mOrQkzB38neQx3bP6IG9/FGzo2TVWBRJ5oL4ayCvLUfrDsMxpbLHwLkZafPYh X-Received: by 2002:a05:6402:f04:b0:46d:ca42:2e59 with SMTP id i4-20020a0564020f0400b0046dca422e59mr77572091eda.11.1673619566324; Fri, 13 Jan 2023 06:19:26 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673619566; cv=none; d=google.com; s=arc-20160816; b=jYG9raRuYj40Cq3BJG3I+KBLkYDrPdUT/0Wt5k9nQOFBnppxjlXZSu6e28ZL8WPKmB f5Lo7G+NaKE5OdGrpjcWLlH/XRFaXdXGwwAUayhoapNMjN8XZaQnk8N8EKAl76ZSIwTC +3GkQhgaz9oK7Xnn0cxJtP9CYAxVY3/LlN+zxdcd5/deYxQYFLKXb9KW52o392Oeg6EW 6Gd7MCQAnpkqwqMUfTaqFZIsAj+R3bhib7kl/AechiwkCPZh6Dy1165YFPBVw3WSpL5+ 5/EOvCP2HcfS2cACrW6wO6/P+wk+BoG3uSZlj15sn36WA1Ai9nT4vPZTOp8g7MGQoInG 8PNw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=QwoFVJIW+0bL7iox9o7Mb8U1WD+e1JDaKsR1FcoEM8g=; b=yr6jsQPZ0o7MnxqmCyG0BTMS2l9t9Uqf1X72BF3vN/3AdGNeDzBPCAMJcQUN8phToc CIyuC+249i3uEQU3y2GEpYML5+fBbRjNQ9ODwjqANaBS9NRN1zzsq/rmSYM6OcjR5D1+ ZfzYZLSALzyA0+7SZYASiLnfgGQNmn0YtyAk2UPLOCdq9XvCAf/gZLLapIIz73QhLCMc j2f8FSD9JYNl1+lU83U9UetxnNzoFQH9g25y2KpX817pwxOgoD/kauxi5wdBi4obsxNR TUuKk3rhQ8IKpSmA1Vpl8TUEyP3E+PEdqFOrmRUehA5a+ZDqa+H9Sn9Stu69MudjSbsf Qbbw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=MhhrAgOC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id fj27-20020a0564022b9b00b00499c322a9c5si10560116edb.85.2023.01.13.06.18.58; Fri, 13 Jan 2023 06:19:26 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=MhhrAgOC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241477AbjAMOPM (ORCPT + 99 others); Fri, 13 Jan 2023 09:15:12 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39702 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242022AbjAMONt (ORCPT ); Fri, 13 Jan 2023 09:13:49 -0500 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4F14AFF for ; Fri, 13 Jan 2023 06:12:44 -0800 (PST) Received: by mail-wr1-x432.google.com with SMTP id z5so20134306wrt.6 for ; Fri, 13 Jan 2023 06:12:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QwoFVJIW+0bL7iox9o7Mb8U1WD+e1JDaKsR1FcoEM8g=; b=MhhrAgOCEacYOqL0IH0/5NYFagxjKcXil2qT+PzY/z6IHGPK6LEnYt7xNEt6Q9tlFu XhP/eOXVU7KfBx4MKIwGS5k3DadIIxLYdR5SZQ+EuZ8NF64OIb8s4e43SBd35hr0FcJm 9yW3MFwcjOZWFus2MWaGvdV5+c82Ae4mhjhzf3aqdukTTqo2VMWPrFKwJhgXCjXpmW65 bOtLzu9qRbgLQgDZ9RV1CVTveptWR16CeqriY4Hwj46dYCkdWiu2ZdAU7XT1KmwKpuIe /3KssFjWpG5vEH5sY3Doxs/3BAi8MhMEz536cReTf4TAbzMQIZMMnCdAmwiXm19uw7gv XIIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QwoFVJIW+0bL7iox9o7Mb8U1WD+e1JDaKsR1FcoEM8g=; b=EMO3e3an4CDk8aLKqqF2Z9vz2usOaW7A2iwQshRp3B1uk3X3tXu8McoNfsaxHPYcpJ 1UH4m4fVA798yX6vox3iVMIy9TjIPcVSHEHVRzKvHIxnBJcAjek9QCfcsfaa7D1xQabB n6Oh8NWOF+bCYRfsLX9jwNpICF47e7qQDwEDRU7mMejBTif8pWppWdDUSYQH3DcsRWPq ecLtz/gZbJlMw1TLClt/A67DVrkIdix27BpbM0sk5xpHLH9WEylkcBhKSsmjv44zILS2 Fv+jHxnR3l7CJV0FynsdmEJTfNIJHxuTA+Ww+xxgSAYAuQMxJngMTKEjFxgml6SbqPEY Iseg== X-Gm-Message-State: AFqh2koiEvcZhQ4D2JDFbRXg3D1mquReyXMyQjBceyWEsLo91VYLT9Si rdILhS5aFkkO409XiPBmHds2uw== X-Received: by 2002:a5d:604f:0:b0:2bd:dc5c:7e4c with SMTP id j15-20020a5d604f000000b002bddc5c7e4cmr2431910wrt.15.1673619162861; Fri, 13 Jan 2023 06:12:42 -0800 (PST) Received: from vingu-book.. ([2a01:e0a:f:6020:1563:65bf:c344:661e]) by smtp.gmail.com with ESMTPSA id f7-20020a5d6647000000b002bbeb700c38sm13869919wrw.91.2023.01.13.06.12.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Jan 2023 06:12:42 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com, cgroups@vger.kernel.org Cc: qyousef@layalina.io, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v10 3/9] sched/core: Propagate parent task's latency requirements to the child task Date: Fri, 13 Jan 2023 15:12:28 +0100 Message-Id: <20230113141234.260128-4-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230113141234.260128-1-vincent.guittot@linaro.org> References: <20230113141234.260128-1-vincent.guittot@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754917310514220635?= X-GMAIL-MSGID: =?utf-8?q?1754917310514220635?= From: Parth Shah Clone parent task's latency_nice attribute to the forked child task. Reset the latency_nice value to default value when the child task is set to sched_reset_on_fork. Also, initialize init_task.latency_nice value with DEFAULT_LATENCY_NICE value Signed-off-by: Parth Shah [rebase] Signed-off-by: Vincent Guittot Tested-by: K Prateek Nayak --- init/init_task.c | 1 + kernel/sched/core.c | 1 + 2 files changed, 2 insertions(+) diff --git a/init/init_task.c b/init/init_task.c index ff6c4b9bfe6b..7dd71dd2d261 100644 --- a/init/init_task.c +++ b/init/init_task.c @@ -78,6 +78,7 @@ struct task_struct init_task .prio = MAX_PRIO - 20, .static_prio = MAX_PRIO - 20, .normal_prio = MAX_PRIO - 20, + .latency_nice = DEFAULT_LATENCY_NICE, .policy = SCHED_NORMAL, .cpus_ptr = &init_task.cpus_mask, .user_cpus_ptr = NULL, diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 03b8529db73f..012a1f551f4f 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4632,6 +4632,7 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p) p->prio = p->normal_prio = p->static_prio; set_load_weight(p, false); + p->latency_nice = DEFAULT_LATENCY_NICE; /* * We don't need the reset flag anymore after the fork. It has * fulfilled its duty: From patchwork Fri Jan 13 14:12:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 43325 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp293902wrt; Fri, 13 Jan 2023 06:17:38 -0800 (PST) X-Google-Smtp-Source: AMrXdXtT+Z9IFItIocV0zKWuKrUcqBvKvmfIHKGXQnBjF9De5VNeEkiuN1V8PEZPocE6/Vx9NUzH X-Received: by 2002:a17:906:7d4d:b0:862:1709:a06a with SMTP id l13-20020a1709067d4d00b008621709a06amr7492197ejp.72.1673619458591; Fri, 13 Jan 2023 06:17:38 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673619458; cv=none; d=google.com; s=arc-20160816; b=b4+IX5IEXkg768hvZZ1hySrtMVRKAD9OxHrRpg4R6UxhSpQOdb9ZSw/PJfA/fzy6wc hXSEVA4OArpoYdcJsXvFxHi4WyjnnlshrxAyUFkX5GqnBioJSNyqfzIEhsbndl7HLMEg c6SXc42uGmcHASPxJ/KqSc9Odl14WvlVp53E4ZzgzGQQjuKkZ1eFJWtRNIVKEHsoWvU8 wWSmpFTKkL803xx6Kn8wVHJQ/tKNvNIq/PZnovWqoDwr0xtlQqKoqnbUCN9MeKTvClz1 0JiO9UgQSc28RPPrEibLT6vgJxVhgP/QlQtmHyWt1lWk1Y0Fx3Ny8rLPllNSgFl8+ueV zeuA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=2Am1q/bEvRn6pheYQNrR7HS5jvhXqf4XcHEu+7BBCYA=; b=vkDHaxxaf6ui0mL/E5biImmlwv2JcCCynLuvfWT7u79bQi9FFA5pq3BhIaNYpSdhG3 mIkyqKzGl5V45usdp4Y9n7oVTeey/2/KZCNuZ08Jxjig7QOSZn2cez71ZKJz+pjpHVPe 1sDGREArvsVcod06P5RQcY+qihDGwlZsEaaMuikEA2jK88JwwSp4HGLcqcPCXjvkqg7U RN/J03wpFDZMS7YBWb+H5bVmXuvB3hW4zjDjIbYshJ/Rqa6CExvwp3PI6EYvFyWp9P0k mKQ5sorWp9Xu0Q395UiKtcUMPtWfotiPwGv48jj7iZIX/77sk40DT9QA69hcz+gzhPku 8Qag== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=NbVbWvwm; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id sc22-20020a1709078a1600b00865b943e999si6485159ejc.10.2023.01.13.06.17.13; Fri, 13 Jan 2023 06:17:38 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=NbVbWvwm; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233059AbjAMOPj (ORCPT + 99 others); Fri, 13 Jan 2023 09:15:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40478 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242026AbjAMONu (ORCPT ); Fri, 13 Jan 2023 09:13:50 -0500 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BA129B12 for ; Fri, 13 Jan 2023 06:12:45 -0800 (PST) Received: by mail-wm1-x333.google.com with SMTP id p3-20020a05600c1d8300b003d9ee5f125bso12372251wms.4 for ; Fri, 13 Jan 2023 06:12:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2Am1q/bEvRn6pheYQNrR7HS5jvhXqf4XcHEu+7BBCYA=; b=NbVbWvwm3dS9NPvSb1fzuMi3VKuQ6EhjoB3vCoplqZQl2sh2bg01jMlbprGg5xfKg/ B+4PqQkCl03dIKYokB+i6GxtDdLgzQv0k+mzoEO6cuAlF9FLmnT3LCTI6O9f6OrR+Peh kgb6gBsfsZfRVsB+gcnD2Ylh9O4eQRpMPsDv/4n7CsoPIjCup2yz3U3ZBflSbLqyOIbP ZQ2Wk3xGXJ+Ank/cocUu2n+RI44sbzx5stllUYTh1uTb7UmMDudTj6QMibXj7OFXfVd/ PSXwF+mFKr8M+92q2RlyHDx6/IoVvjxAqwyCu6G2lTs74gRz5iYi5ymBR/CWyqp8bNGk pZHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2Am1q/bEvRn6pheYQNrR7HS5jvhXqf4XcHEu+7BBCYA=; b=nP3XFec5JZp1+89+CAmLqygnhkDB5Kks68xgvE5MGrcEJQNVcC9AP6PTar9xlYOSYh L2BjsmqKvKtigOPrgleCU30u5SvOx37VpICnjmxCGMjlG+nIa/k7yL3xz0P03z9zsiE1 7tjX/jxgc5mv7Vb538hSrhvEHhe5hHht4++/FQ0jdEkU1YxRvgF7p58j5cqGcsLyuMlW UAmHHLlM9NNfdEc4y1jE6ts7KlHIM3X5oWeeYgH7nvRCZPaST3dYL9+KFH/XSsoY6CH+ Hm97Hm2LOhG/YUSGolSt/M1k2+wjU+He/Qvfx/XdqwbrARTqPGq1pUFA8e601hSjeX+B TNyg== X-Gm-Message-State: AFqh2kp51vNf5O/954tqu93VK+xET5yfN+8oEQ6zdp6//EMYhiYgnoo9 rBGmReApf5jqikpuDPzBQZHBkA== X-Received: by 2002:a05:600c:3d91:b0:3d9:103d:9078 with SMTP id bi17-20020a05600c3d9100b003d9103d9078mr57517685wmb.22.1673619164260; Fri, 13 Jan 2023 06:12:44 -0800 (PST) Received: from vingu-book.. ([2a01:e0a:f:6020:1563:65bf:c344:661e]) by smtp.gmail.com with ESMTPSA id f7-20020a5d6647000000b002bbeb700c38sm13869919wrw.91.2023.01.13.06.12.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Jan 2023 06:12:43 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com, cgroups@vger.kernel.org Cc: qyousef@layalina.io, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v10 4/9] sched: Allow sched_{get,set}attr to change latency_nice of the task Date: Fri, 13 Jan 2023 15:12:29 +0100 Message-Id: <20230113141234.260128-5-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230113141234.260128-1-vincent.guittot@linaro.org> References: <20230113141234.260128-1-vincent.guittot@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754917197406059150?= X-GMAIL-MSGID: =?utf-8?q?1754917197406059150?= From: Parth Shah Introduce the latency_nice attribute to sched_attr and provide a mechanism to change the value with the use of sched_setattr/sched_getattr syscall. Also add new flag "SCHED_FLAG_LATENCY_NICE" to hint the change in latency_nice of the task on every sched_setattr syscall. Signed-off-by: Parth Shah [rebase and add a dedicated __setscheduler_latency ] Signed-off-by: Vincent Guittot Tested-by: K Prateek Nayak --- include/uapi/linux/sched.h | 4 +++- include/uapi/linux/sched/types.h | 19 +++++++++++++++++++ kernel/sched/core.c | 24 ++++++++++++++++++++++++ tools/include/uapi/linux/sched.h | 4 +++- 4 files changed, 49 insertions(+), 2 deletions(-) diff --git a/include/uapi/linux/sched.h b/include/uapi/linux/sched.h index 3bac0a8ceab2..b2e932c25be6 100644 --- a/include/uapi/linux/sched.h +++ b/include/uapi/linux/sched.h @@ -132,6 +132,7 @@ struct clone_args { #define SCHED_FLAG_KEEP_PARAMS 0x10 #define SCHED_FLAG_UTIL_CLAMP_MIN 0x20 #define SCHED_FLAG_UTIL_CLAMP_MAX 0x40 +#define SCHED_FLAG_LATENCY_NICE 0x80 #define SCHED_FLAG_KEEP_ALL (SCHED_FLAG_KEEP_POLICY | \ SCHED_FLAG_KEEP_PARAMS) @@ -143,6 +144,7 @@ struct clone_args { SCHED_FLAG_RECLAIM | \ SCHED_FLAG_DL_OVERRUN | \ SCHED_FLAG_KEEP_ALL | \ - SCHED_FLAG_UTIL_CLAMP) + SCHED_FLAG_UTIL_CLAMP | \ + SCHED_FLAG_LATENCY_NICE) #endif /* _UAPI_LINUX_SCHED_H */ diff --git a/include/uapi/linux/sched/types.h b/include/uapi/linux/sched/types.h index f2c4589d4dbf..db1e8199e8c8 100644 --- a/include/uapi/linux/sched/types.h +++ b/include/uapi/linux/sched/types.h @@ -10,6 +10,7 @@ struct sched_param { #define SCHED_ATTR_SIZE_VER0 48 /* sizeof first published struct */ #define SCHED_ATTR_SIZE_VER1 56 /* add: util_{min,max} */ +#define SCHED_ATTR_SIZE_VER2 60 /* add: latency_nice */ /* * Extended scheduling parameters data structure. @@ -98,6 +99,22 @@ struct sched_param { * scheduled on a CPU with no more capacity than the specified value. * * A task utilization boundary can be reset by setting the attribute to -1. + * + * Latency Tolerance Attributes + * =========================== + * + * A subset of sched_attr attributes allows to specify the relative latency + * requirements of a task with respect to the other tasks running/queued in the + * system. + * + * @ sched_latency_nice task's latency_nice value + * + * The latency_nice of a task can have any value in a range of + * [MIN_LATENCY_NICE..MAX_LATENCY_NICE]. + * + * A task with latency_nice with the value of LATENCY_NICE_MIN can be + * taken for a task requiring a lower latency as opposed to the task with + * higher latency_nice. */ struct sched_attr { __u32 size; @@ -120,6 +137,8 @@ struct sched_attr { __u32 sched_util_min; __u32 sched_util_max; + /* latency requirement hints */ + __s32 sched_latency_nice; }; #endif /* _UAPI_LINUX_SCHED_TYPES_H */ diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 012a1f551f4f..981665550f8c 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -7392,6 +7392,14 @@ static void __setscheduler_params(struct task_struct *p, p->rt_priority = attr->sched_priority; p->normal_prio = normal_prio(p); set_load_weight(p, true); + +} + +static void __setscheduler_latency(struct task_struct *p, + const struct sched_attr *attr) +{ + if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) + p->latency_nice = attr->sched_latency_nice; } /* @@ -7534,6 +7542,13 @@ static int __sched_setscheduler(struct task_struct *p, return retval; } + if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) { + if (attr->sched_latency_nice > MAX_LATENCY_NICE) + return -EINVAL; + if (attr->sched_latency_nice < MIN_LATENCY_NICE) + return -EINVAL; + } + if (pi) cpuset_read_lock(); @@ -7568,6 +7583,9 @@ static int __sched_setscheduler(struct task_struct *p, goto change; if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP) goto change; + if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE && + attr->sched_latency_nice != p->latency_nice) + goto change; p->sched_reset_on_fork = reset_on_fork; retval = 0; @@ -7656,6 +7674,7 @@ static int __sched_setscheduler(struct task_struct *p, __setscheduler_params(p, attr); __setscheduler_prio(p, newprio); } + __setscheduler_latency(p, attr); __setscheduler_uclamp(p, attr); if (queued) { @@ -7866,6 +7885,9 @@ static int sched_copy_attr(struct sched_attr __user *uattr, struct sched_attr *a size < SCHED_ATTR_SIZE_VER1) return -EINVAL; + if ((attr->sched_flags & SCHED_FLAG_LATENCY_NICE) && + size < SCHED_ATTR_SIZE_VER2) + return -EINVAL; /* * XXX: Do we want to be lenient like existing syscalls; or do we want * to be strict and return an error on out-of-bounds values? @@ -8103,6 +8125,8 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sched_attr __user *, uattr, get_params(p, &kattr); kattr.sched_flags &= SCHED_FLAG_ALL; + kattr.sched_latency_nice = p->latency_nice; + #ifdef CONFIG_UCLAMP_TASK /* * This could race with another potential updater, but this is fine diff --git a/tools/include/uapi/linux/sched.h b/tools/include/uapi/linux/sched.h index 3bac0a8ceab2..b2e932c25be6 100644 --- a/tools/include/uapi/linux/sched.h +++ b/tools/include/uapi/linux/sched.h @@ -132,6 +132,7 @@ struct clone_args { #define SCHED_FLAG_KEEP_PARAMS 0x10 #define SCHED_FLAG_UTIL_CLAMP_MIN 0x20 #define SCHED_FLAG_UTIL_CLAMP_MAX 0x40 +#define SCHED_FLAG_LATENCY_NICE 0x80 #define SCHED_FLAG_KEEP_ALL (SCHED_FLAG_KEEP_POLICY | \ SCHED_FLAG_KEEP_PARAMS) @@ -143,6 +144,7 @@ struct clone_args { SCHED_FLAG_RECLAIM | \ SCHED_FLAG_DL_OVERRUN | \ SCHED_FLAG_KEEP_ALL | \ - SCHED_FLAG_UTIL_CLAMP) + SCHED_FLAG_UTIL_CLAMP | \ + SCHED_FLAG_LATENCY_NICE) #endif /* _UAPI_LINUX_SCHED_H */ From patchwork Fri Jan 13 14:12:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 43332 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp294915wrt; Fri, 13 Jan 2023 06:19:34 -0800 (PST) X-Google-Smtp-Source: AMrXdXubATrLDFpjlaS4qjEh7KIrC5kwFWoNhMxaxHi7kfjlmb9zh4FTWsNT09eOiUVOXMwZjT9x X-Received: by 2002:a17:906:9f07:b0:7ec:27d7:1838 with SMTP id fy7-20020a1709069f0700b007ec27d71838mr85509681ejc.22.1673619573834; Fri, 13 Jan 2023 06:19:33 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673619573; cv=none; d=google.com; s=arc-20160816; b=CmMiCnRHN3GN0xeY2YU71i4qGctT1Z32uJW3yRudyn/7jIAhEnilEHTn0ipju6X+iY WUJ3YmuhFbSvBGb4Wk6RMcPk4iX+x3XkBssRS07jfb76YigpPBkjtkqBNnuC7z33mIUg VWxby0axSoaP4ai/4lNNCBh1ekO/pUogkhGvJQQIz8LOyYshnf9i8h5SUm4Vuz1L75Cj sONnnIebGh64wKuZXS8p+98LxPgcistSIQfDqj1Sclq/D9tOkD/Hei2hkUiww2nb0LKH DpZDvaSGRHT/CYC5u/cnxUTI5LXNvw2gDqS1t+3mAA2XzMjUBcqWjL8tv/z0X3H8b6ai k1yg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=v5xxDzbGl7sCo78cCnADO2gQFU/vv6X0hEszTOboieA=; b=acGjGmnNJE3bGexHiDD2Gar9lAOT7pfUVlKoK8ME4h8iNloeMlYWYNPG0JUmhkXg2y ALw/e/aNyGZKNl9f+o2dbTMFefz4R3/A4YzvtWvNl2cBWWB7xcdtq++QeXIlxjWiPEqs V37/ToVN8CnzqNEknbmBEs6TfQGRYULpf4EocJzxRp2x4KK6+4p6IwTIJecI1lVuH4Sb Pqzws/CLVP8gmJCo/8L5ymIG6eghbXK6HAOGCIp4e+vddNtBWFRfe9oFhbUYugeS0iMZ SJzZnFxGBRG7dhDLOMDKcySaGgamBbK0uHvOmow/tKjRnegx+8IVwbjEDP0xd0PPHjMW xnew== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=sOcFHJRO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id dm8-20020a170907948800b007c0fa2d86d1si23371473ejc.906.2023.01.13.06.19.07; Fri, 13 Jan 2023 06:19:33 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=sOcFHJRO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241747AbjAMOPU (ORCPT + 99 others); Fri, 13 Jan 2023 09:15:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40422 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242034AbjAMONv (ORCPT ); Fri, 13 Jan 2023 09:13:51 -0500 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 924F85CFB5 for ; Fri, 13 Jan 2023 06:12:47 -0800 (PST) Received: by mail-wr1-x42c.google.com with SMTP id v2so10197618wrw.10 for ; Fri, 13 Jan 2023 06:12:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=v5xxDzbGl7sCo78cCnADO2gQFU/vv6X0hEszTOboieA=; b=sOcFHJRO3tzMdITMarIyeG54hQ9Z/8+xami3g461hCBYHEHEoaWYo6RoJbqK1aj3td vBaZqktAhE/Otrxe0Wi4OGxpmAOE3GJBPRMPzsVwTWBwA3uG0VXPLeaDk9wu62h2xmVz LUXMCyjeIsxjhD7AQJRkh9GcHeBKQyewpnbSe8TvoK1hgp+hpTEeWU9f9FNvWccw36I0 uGHuW004HyfQN5eBLSIycAjFuNolTDQv8GoZbjaIMz0lOpmC1udBFAgfg7Z6jWM6D/Nh ZYHgmxSRcReFHB9Yp6UPFj0aof/Bmh7xlg7asRA4yX3HaMRDsFgUrDCZs9mqB93L9mf2 QBnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=v5xxDzbGl7sCo78cCnADO2gQFU/vv6X0hEszTOboieA=; b=4jAhDHOkyoa2yVwynVEy7Kbv2rJ4Sga/Q3w/nThuCg6Uj62HQ1TWt0vA3aHNSyuiEt pNF6nXnwR8a7Loc40xANGri9XuQAuGS1XjVS2lUh56YhGthzU2pQ4I1VVntM1jJtQCu7 OAg6O9S9gaBI0I0MSHNCvTfRkQMYkQHSegy8jYP46UlrjZBF70IiYFekWm0Uop9s1znH 9Elx1g+m7wQCvPnoalUlW/hL737LbohbEzmdKWLcCQvqM84s4BpWbylKTxTZTBzoGC9Y rHKDK+ZpOFfVuueP1y4F3fVo5TepYKkvwwvsgom5svmCOZ8JSkGcXCMmM9R9gw2U4/2/ c5rQ== X-Gm-Message-State: AFqh2kr7zZ5wQ8iszpF8QJ//g0rjf1ov6mwHp+D4VyLXhCgahSTcl3SU uY9uELlnrkklusSfhcxB4ODOfQ== X-Received: by 2002:adf:ffcd:0:b0:2bd:db3b:33f5 with SMTP id x13-20020adfffcd000000b002bddb3b33f5mr3501837wrs.3.1673619165957; Fri, 13 Jan 2023 06:12:45 -0800 (PST) Received: from vingu-book.. ([2a01:e0a:f:6020:1563:65bf:c344:661e]) by smtp.gmail.com with ESMTPSA id f7-20020a5d6647000000b002bbeb700c38sm13869919wrw.91.2023.01.13.06.12.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Jan 2023 06:12:45 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com, cgroups@vger.kernel.org Cc: qyousef@layalina.io, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v10 5/9] sched/fair: Take into account latency priority at wakeup Date: Fri, 13 Jan 2023 15:12:30 +0100 Message-Id: <20230113141234.260128-6-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230113141234.260128-1-vincent.guittot@linaro.org> References: <20230113141234.260128-1-vincent.guittot@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-1.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,PDS_BTC_ID,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754917318270377182?= X-GMAIL-MSGID: =?utf-8?q?1754917318270377182?= Take into account the latency priority of a thread when deciding to preempt the current running thread. We don't want to provide more CPU bandwidth to a thread but reorder the scheduling to run latency sensitive task first whenever possible. As long as a thread didn't use its bandwidth, it will be able to preempt the current thread. At the opposite, a thread with a low latency priority will preempt current thread at wakeup only to keep fair CPU bandwidth sharing. Otherwise it will wait for the tick to get its sched slice. curr vruntime | sysctl_sched_wakeup_granularity <--> ----------------------------------|----|-----------------------|--------------- | |<---------------------> | . sysctl_sched_latency | . default/current latency entity | . | . 1111111111111111111111111111111111|0000|-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1- se preempts curr at wakeup ------>|<- se doesn't preempt curr ----------------- | . | . | . low latency entity | . ---------------------->| % of sysctl_sched_latency | 1111111111111111111111111111111111111111111111111111111111|0000|-1-1-1-1-1-1-1- preempt ------------------------------------------------->|<- do not preempt -- | . | . | . high latency entity | . |<-----------------------|----. | % of sysctl_sched_latency . 111111111|0000|-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1 preempt->|<- se doesn't preempt curr ------------------------------------------ Tests results of nice latency impact on heavy load like hackbench: hackbench -l (2560 / group) -g group group latency 0 latency 19 1 1.378(+/- 1%) 1.337(+/- 1%) + 3% 4 1.393(+/- 3%) 1.312(+/- 3%) + 6% 8 1.308(+/- 2%) 1.279(+/- 1%) + 2% 16 1.347(+/- 1%) 1.317(+/- 1%) + 2% hackbench -p -l (2560 / group) -g group group 1 1.836(+/- 17%) 1.148(+/- 5%) +37% 4 1.586(+/- 6%) 1.109(+/- 8%) +30% 8 1.209(+/- 4%) 0.780(+/- 4%) +35% 16 0.805(+/- 5%) 0.728(+/- 4%) +10% By deacreasing the latency prio, we reduce the number of preemption at wakeup and help hackbench making progress. Test results of nice latency impact on short live load like cyclictest while competing with heavy load like hackbench: hackbench -l 10000 -g $group & cyclictest --policy other -D 5 -q -n latency 0 latency -20 group min avg max min avg max 0 16 19 29 17 18 29 1 43 299 7359 63 84 3422 4 56 449 14806 45 83 284 8 63 820 51123 63 83 283 16 64 1326 70684 41 157 26852 group = 0 means that hackbench is not running. The avg is significantly improved with nice latency -20 especially with large number of groups but min and max remain quite similar. If we add the histogram parameter to get details of latency, we have : hackbench -l 10000 -g 16 & cyclictest --policy other -D 5 -q -n -H 20000 --histfile data.txt latency 0 latency -20 Min Latencies: 64 62 Avg Latencies: 1170 107 Max Latencies: 88069 10417 50% latencies: 122 86 75% latencies: 614 91 85% latencies: 961 94 90% latencies: 1225 97 95% latencies: 6120 102 99% latencies: 18328 159 With percentile details, we see the benefit of nice latency -20 as only 1% of the latencies are above 159us whereas the default latency has got 15% around ~1ms or above and 5% over the 6ms. Signed-off-by: Vincent Guittot Tested-by: K Prateek Nayak --- include/linux/sched.h | 4 ++- include/linux/sched/prio.h | 9 ++++++ init/init_task.c | 2 +- kernel/sched/core.c | 38 +++++++++++++++++++--- kernel/sched/debug.c | 2 +- kernel/sched/fair.c | 66 ++++++++++++++++++++++++++++++++++---- kernel/sched/sched.h | 6 ++++ 7 files changed, 112 insertions(+), 15 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 6c61bde49152..38decae3e156 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -568,6 +568,8 @@ struct sched_entity { /* cached value of my_q->h_nr_running */ unsigned long runnable_weight; #endif + /* preemption offset in ns */ + long latency_offset; #ifdef CONFIG_SMP /* @@ -784,7 +786,7 @@ struct task_struct { int static_prio; int normal_prio; unsigned int rt_priority; - int latency_nice; + int latency_prio; struct sched_entity se; struct sched_rt_entity rt; diff --git a/include/linux/sched/prio.h b/include/linux/sched/prio.h index bfcd7f1d1e11..be79503d86af 100644 --- a/include/linux/sched/prio.h +++ b/include/linux/sched/prio.h @@ -59,5 +59,14 @@ static inline long rlimit_to_nice(long prio) * Default tasks should be treated as a task with latency_nice = 0. */ #define DEFAULT_LATENCY_NICE 0 +#define DEFAULT_LATENCY_PRIO (DEFAULT_LATENCY_NICE + LATENCY_NICE_WIDTH/2) + +/* + * Convert user-nice values [ -20 ... 0 ... 19 ] + * to static latency [ 0..39 ], + * and back. + */ +#define NICE_TO_LATENCY(nice) ((nice) + DEFAULT_LATENCY_PRIO) +#define LATENCY_TO_NICE(prio) ((prio) - DEFAULT_LATENCY_PRIO) #endif /* _LINUX_SCHED_PRIO_H */ diff --git a/init/init_task.c b/init/init_task.c index 7dd71dd2d261..071deff8dbd1 100644 --- a/init/init_task.c +++ b/init/init_task.c @@ -78,7 +78,7 @@ struct task_struct init_task .prio = MAX_PRIO - 20, .static_prio = MAX_PRIO - 20, .normal_prio = MAX_PRIO - 20, - .latency_nice = DEFAULT_LATENCY_NICE, + .latency_prio = DEFAULT_LATENCY_PRIO, .policy = SCHED_NORMAL, .cpus_ptr = &init_task.cpus_mask, .user_cpus_ptr = NULL, diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 981665550f8c..402c8d622b76 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -1285,6 +1285,16 @@ static void set_load_weight(struct task_struct *p, bool update_load) } } +static void set_latency_offset(struct task_struct *p) +{ + long weight = sched_latency_to_weight[p->latency_prio]; + s64 offset; + + offset = weight * get_sleep_latency(false); + offset = div_s64(offset, NICE_LATENCY_WEIGHT_MAX); + p->se.latency_offset = (long)offset; +} + #ifdef CONFIG_UCLAMP_TASK /* * Serializes updates of utilization clamp values @@ -4632,7 +4642,9 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p) p->prio = p->normal_prio = p->static_prio; set_load_weight(p, false); - p->latency_nice = DEFAULT_LATENCY_NICE; + p->latency_prio = NICE_TO_LATENCY(0); + set_latency_offset(p); + /* * We don't need the reset flag anymore after the fork. It has * fulfilled its duty: @@ -7398,8 +7410,10 @@ static void __setscheduler_params(struct task_struct *p, static void __setscheduler_latency(struct task_struct *p, const struct sched_attr *attr) { - if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) - p->latency_nice = attr->sched_latency_nice; + if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) { + p->latency_prio = NICE_TO_LATENCY(attr->sched_latency_nice); + set_latency_offset(p); + } } /* @@ -7584,7 +7598,7 @@ static int __sched_setscheduler(struct task_struct *p, if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP) goto change; if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE && - attr->sched_latency_nice != p->latency_nice) + attr->sched_latency_nice != LATENCY_TO_NICE(p->latency_prio)) goto change; p->sched_reset_on_fork = reset_on_fork; @@ -8125,7 +8139,7 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sched_attr __user *, uattr, get_params(p, &kattr); kattr.sched_flags &= SCHED_FLAG_ALL; - kattr.sched_latency_nice = p->latency_nice; + kattr.sched_latency_nice = LATENCY_TO_NICE(p->latency_prio); #ifdef CONFIG_UCLAMP_TASK /* @@ -11334,6 +11348,20 @@ const u32 sched_prio_to_wmult[40] = { /* 15 */ 119304647, 148102320, 186737708, 238609294, 286331153, }; +/* + * latency weight for wakeup preemption + */ +const int sched_latency_to_weight[40] = { + /* -20 */ -1024, -973, -922, -870, -819, + /* -15 */ -768, -717, -666, -614, -563, + /* -10 */ -512, -461, -410, -358, -307, + /* -5 */ -256, -205, -154, -102, -51, + /* 0 */ 0, 51, 102, 154, 205, + /* 5 */ 256, 307, 358, 410, 461, + /* 10 */ 512, 563, 614, 666, 717, + /* 15 */ 768, 819, 870, 922, 973, +}; + void call_trace_sched_update_nr_running(struct rq *rq, int count) { trace_sched_update_nr_running_tp(rq, count); diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c index 68be7a3e42a3..b3922184af91 100644 --- a/kernel/sched/debug.c +++ b/kernel/sched/debug.c @@ -1043,7 +1043,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns, #endif P(policy); P(prio); - P(latency_nice); + P(latency_prio); if (task_has_dl_policy(p)) { P(dl.runtime); P(dl.deadline); diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 8a85c6cf781e..e87a863a2aa6 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4870,6 +4870,8 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) update_idle_cfs_rq_clock_pelt(cfs_rq); } +static long wakeup_latency_gran(struct sched_entity *curr, struct sched_entity *se); + /* * Preempt the current task with a newly woken task if needed: */ @@ -4878,7 +4880,7 @@ check_preempt_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr) { unsigned long ideal_runtime, delta_exec; struct sched_entity *se; - s64 delta; + s64 delta, offset; ideal_runtime = sched_slice(cfs_rq, curr); delta_exec = curr->sum_exec_runtime - curr->prev_sum_exec_runtime; @@ -4903,10 +4905,12 @@ check_preempt_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr) se = __pick_first_entity(cfs_rq); delta = curr->vruntime - se->vruntime; - if (delta < 0) + offset = wakeup_latency_gran(curr, se); + if (delta < offset) return; - if (delta > ideal_runtime) + if ((delta > ideal_runtime) || + (delta > get_latency_max())) resched_curr(rq_of(cfs_rq)); } @@ -6155,6 +6159,35 @@ static int sched_idle_cpu(int cpu) } #endif +static void set_next_buddy(struct sched_entity *se); + +static void check_preempt_from_others(struct cfs_rq *cfs, struct sched_entity *se) +{ + struct sched_entity *next; + + if (se->latency_offset >= 0) + return; + + if (cfs->nr_running <= 1) + return; + /* + * When waking from another class, we don't need to check to preempt at + * wakeup and don't set next buddy as a candidate for being picked in + * priority. + * In case of simultaneous wakeup when current is another class, the + * latency sensitive tasks lost opportunity to preempt non sensitive + * tasks which woke up simultaneously. + */ + + if (cfs->next) + next = cfs->next; + else + next = __pick_first_entity(cfs); + + if (next && wakeup_preempt_entity(next, se) == 1) + set_next_buddy(se); +} + /* * The enqueue_task method is called before nr_running is * increased. Here we update the fair scheduling stats and @@ -6241,14 +6274,15 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) if (!task_new) update_overutilized_status(rq); + if (rq->curr->sched_class != &fair_sched_class) + check_preempt_from_others(cfs_rq_of(&p->se), &p->se); + enqueue_throttle: assert_list_leaf_cfs_rq(rq); hrtick_update(rq); } -static void set_next_buddy(struct sched_entity *se); - /* * The dequeue_task method is called before nr_running is * decreased. We remove the task from the rbtree and @@ -7597,6 +7631,23 @@ balance_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) } #endif /* CONFIG_SMP */ +static long wakeup_latency_gran(struct sched_entity *curr, struct sched_entity *se) +{ + long latency_offset = se->latency_offset; + + /* + * A negative latency offset means that the sched_entity has latency + * requirement that needs to be evaluated versus other entity. + * Otherwise, use the latency weight to evaluate how much scheduling + * delay is acceptable by se. + */ + if ((latency_offset < 0) || (curr->latency_offset < 0)) + latency_offset -= curr->latency_offset; + latency_offset = min_t(long, latency_offset, get_latency_max()); + + return latency_offset; +} + static unsigned long wakeup_gran(struct sched_entity *se) { unsigned long gran = sysctl_sched_wakeup_granularity; @@ -7635,11 +7686,12 @@ static int wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se) { s64 gran, vdiff = curr->vruntime - se->vruntime; + s64 offset = wakeup_latency_gran(curr, se); - if (vdiff <= 0) + if (vdiff < offset) return -1; - gran = wakeup_gran(se); + gran = offset + wakeup_gran(se); /* * At wake up, the vruntime of a task is capped to not be older than diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index df7db06c9943..fd099f3961e4 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -125,6 +125,11 @@ extern int sched_rr_timeslice; */ #define NS_TO_JIFFIES(TIME) ((unsigned long)(TIME) / (NSEC_PER_SEC / HZ)) +/* Maximum nice latency weight used to scale the latency_offset */ + +#define NICE_LATENCY_SHIFT (SCHED_FIXEDPOINT_SHIFT) +#define NICE_LATENCY_WEIGHT_MAX (1L << NICE_LATENCY_SHIFT) + /* * Increase resolution of nice-level calculations for 64-bit architectures. * The extra resolution improves shares distribution and load balancing of @@ -2121,6 +2126,7 @@ static_assert(WF_TTWU == SD_BALANCE_WAKE); extern const int sched_prio_to_weight[40]; extern const u32 sched_prio_to_wmult[40]; +extern const int sched_latency_to_weight[40]; /* * {de,en}queue flags: From patchwork Fri Jan 13 14:12:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 43327 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp293906wrt; Fri, 13 Jan 2023 06:17:39 -0800 (PST) X-Google-Smtp-Source: AMrXdXvcsukCnCh1vDL4VBh8jy74GNS7B2B2XWOfgkpWnk6IdocksbR8yacW+FAor8Z/jJONtzpM X-Received: by 2002:a05:6402:3908:b0:49b:65cc:faa6 with SMTP id fe8-20020a056402390800b0049b65ccfaa6mr5381241edb.16.1673619459178; Fri, 13 Jan 2023 06:17:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673619459; cv=none; d=google.com; s=arc-20160816; b=O+jeuU2MM5J4TzSPLn2gjwf8wXd1vr7sovFXjqnmh5rXCJK/0cg6uFWlyPgXsOBjcU cp8qlYGuTTILQlAdaIwnj2xJ4JfNjPiKXLsQwP9EDlVuRJQgQ0HbRSiZkR8zeRlsSpSS 2RcuRTzaSt1gY5YNaw7nXkKnLAcJqN7cUrUvAtKrr9GxyNAYS92iF+4RZVdcREzmeIr9 sDf2lHm4DsclFqnsWKzecG+/3wIosMLqZWnmQRYVEWTtbKpArEvap+hMsvvx4ZhOIdx5 9DnLIsl6+VDIOVgqtv/39AJAXv1B93aD8JoY5U397KUx/JawXbDj9w+x61nV+U1/WGqg vNTQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=5wKU8uiBTq3EBSEKs7HoauBnDIh2Fc9iGzBeoPlJTDY=; b=OKgIoMdUDOb3O/1RleSTQhe2rj9Te0uKIuFzgxG/LMDTrGKt0ejmMITvcgDfybcPAO LwnAxAmhqXwmOW934MHCegAaCNw7vnh/3x5Ons5qiks3mRnPX8xHxjAfjVQJDQNYdjZU Z1c66LIHVu8RaidgI2AOBEQpYkiBNill3fEvrznQWL7pY+cu5QhZQpzp5sg8gRYFMRUY xLs4Ch1q1igsbaHU02OeEjAXljp3l4I9kemgBv9VTCrg+B4IAfL9ZAQm5DZ/hkMvF8Jd DL3QNZ/pyJHf3ydXMntXIFdEfeSn2TOkrFjf3Gxw1epOyJh4VHBDt9FUjh6MHOUBfp1L EC0w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Y4VJe97f; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b14-20020a056402084e00b004842aaeea0bsi26254573edz.603.2023.01.13.06.17.14; Fri, 13 Jan 2023 06:17:39 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Y4VJe97f; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233216AbjAMOP2 (ORCPT + 99 others); Fri, 13 Jan 2023 09:15:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40310 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242038AbjAMONw (ORCPT ); Fri, 13 Jan 2023 09:13:52 -0500 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1CB8B54DB1 for ; Fri, 13 Jan 2023 06:12:48 -0800 (PST) Received: by mail-wr1-x42c.google.com with SMTP id b5so29350wrn.0 for ; Fri, 13 Jan 2023 06:12:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5wKU8uiBTq3EBSEKs7HoauBnDIh2Fc9iGzBeoPlJTDY=; b=Y4VJe97fZY3loO27wJAyXfrN3a+KVh5Zuffg0d4rOcwZDSmq3CSYIZFG3xDZcfwMYo xwxgY1zulnkbJyf9oSgwf0zje7+WKqKSmyU6BOvPTy03kxdFJ+x5eeqsqRoLbY5Edb+s VMOoJefJhpGY+yjXT18tQ+KPR9GLt3BE51lpmpUE3E9cHbDvnukSoaD6w0N0dyEGOXgp GvVkFtpBnJblg1D2oxP7hByCNefNkcHOE5eQgEvRLFhxVZfbFETpyopGlEcY5G0nLWPU GiqDDJX6Cj6VUBbx0uHbxewx4vQXY5oOMQcv3P5FBERiMmSeYeLGzW0739e3uZnDvMQj mLdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5wKU8uiBTq3EBSEKs7HoauBnDIh2Fc9iGzBeoPlJTDY=; b=HQpHK7MR+iaFSr4zsWiGSI0fjZU7nK1VyHf9/hg2U9WX0Z6qluA1HSDgzfPCnrsqix SymjT1kp1cQS0vvQZPEIYgN9wfSGr8n6nKpVYe9mML4SJXdHhdEg3k/1lHYKn3bjUkCH mcuEf2Rg2J/8+Xq8amoNJtSwgDs17Rf0m1wEDT+roSVUbn+p4ECe6X8OuQa6jQwOxuq/ FHx/nr/VNmtPAk0xsIixjBkeQl923M5onooPxQ0W2UabqZfDnDfnCvEFVteZEbw+ZFVv 9sRvajW84en7lO134rmYeHTvR1peKqY4Gf2mXJvUCSFRYlRh2hzRSIGfrVUikXwnjkOp RfyA== X-Gm-Message-State: AFqh2kqAlRSppfwxSSiBVSJnJR+cGK8IbnAFwfzbG2z92miXjmDU7HDb G8ksFHmLHXo9Wvj/IAUQI6vXhQ== X-Received: by 2002:adf:e2c6:0:b0:26d:f5d2:4efa with SMTP id d6-20020adfe2c6000000b0026df5d24efamr60211064wrj.21.1673619167559; Fri, 13 Jan 2023 06:12:47 -0800 (PST) Received: from vingu-book.. ([2a01:e0a:f:6020:1563:65bf:c344:661e]) by smtp.gmail.com with ESMTPSA id f7-20020a5d6647000000b002bbeb700c38sm13869919wrw.91.2023.01.13.06.12.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Jan 2023 06:12:46 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com, cgroups@vger.kernel.org Cc: qyousef@layalina.io, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v10 6/9] sched/fair: Add sched group latency support Date: Fri, 13 Jan 2023 15:12:31 +0100 Message-Id: <20230113141234.260128-7-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230113141234.260128-1-vincent.guittot@linaro.org> References: <20230113141234.260128-1-vincent.guittot@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754917198132437021?= X-GMAIL-MSGID: =?utf-8?q?1754917198132437021?= Task can set its latency priority with sched_setattr(), which is then used to set the latency offset of its sched_enity, but sched group entities still have the default latency offset value. Add a latency.nice field in cpu cgroup controller to set the latency priority of the group similarly to sched_setattr(). The latency priority is then used to set the offset of the sched_entities of the group. Signed-off-by: Vincent Guittot Tested-by: K Prateek Nayak --- Documentation/admin-guide/cgroup-v2.rst | 10 +++++ kernel/sched/core.c | 52 +++++++++++++++++++++++++ kernel/sched/fair.c | 33 ++++++++++++++++ kernel/sched/sched.h | 4 ++ 4 files changed, 99 insertions(+) diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst index 1b3ed1c3b3f1..c08424593e4a 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -1121,6 +1121,16 @@ All time durations are in microseconds. values similar to the sched_setattr(2). This maximum utilization value is used to clamp the task specific maximum utilization clamp. + cpu.latency.nice + A read-write single value file which exists on non-root + cgroups. The default is "0". + + The nice value is in the range [-20, 19]. + + This interface file allows reading and setting latency using the + same values used by sched_setattr(2). The latency_nice of a group is + used to limit the impact of the latency_nice of a task outside the + group. Memory diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 402c8d622b76..6798c9a297d6 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -11007,6 +11007,47 @@ static int cpu_idle_write_s64(struct cgroup_subsys_state *css, { return sched_group_set_idle(css_tg(css), idle); } + +static s64 cpu_latency_nice_read_s64(struct cgroup_subsys_state *css, + struct cftype *cft) +{ + int prio, delta, last_delta = INT_MAX; + s64 weight; + + weight = css_tg(css)->latency_offset * NICE_LATENCY_WEIGHT_MAX; + weight = div_s64(weight, get_sleep_latency(false)); + + /* Find the closest nice value to the current weight */ + for (prio = 0; prio < ARRAY_SIZE(sched_latency_to_weight); prio++) { + delta = abs(sched_latency_to_weight[prio] - weight); + if (delta >= last_delta) + break; + last_delta = delta; + } + + return LATENCY_TO_NICE(prio-1); +} + +static int cpu_latency_nice_write_s64(struct cgroup_subsys_state *css, + struct cftype *cft, s64 nice) +{ + s64 latency_offset; + long weight; + int idx; + + if (nice < MIN_LATENCY_NICE || nice > MAX_LATENCY_NICE) + return -ERANGE; + + idx = NICE_TO_LATENCY(nice); + idx = array_index_nospec(idx, LATENCY_NICE_WIDTH); + weight = sched_latency_to_weight[idx]; + + latency_offset = weight * get_sleep_latency(false); + latency_offset = div_s64(latency_offset, NICE_LATENCY_WEIGHT_MAX); + + return sched_group_set_latency(css_tg(css), latency_offset); +} + #endif static struct cftype cpu_legacy_files[] = { @@ -11021,6 +11062,11 @@ static struct cftype cpu_legacy_files[] = { .read_s64 = cpu_idle_read_s64, .write_s64 = cpu_idle_write_s64, }, + { + .name = "latency.nice", + .read_s64 = cpu_latency_nice_read_s64, + .write_s64 = cpu_latency_nice_write_s64, + }, #endif #ifdef CONFIG_CFS_BANDWIDTH { @@ -11238,6 +11284,12 @@ static struct cftype cpu_files[] = { .read_s64 = cpu_idle_read_s64, .write_s64 = cpu_idle_write_s64, }, + { + .name = "latency.nice", + .flags = CFTYPE_NOT_ON_ROOT, + .read_s64 = cpu_latency_nice_read_s64, + .write_s64 = cpu_latency_nice_write_s64, + }, #endif #ifdef CONFIG_CFS_BANDWIDTH { diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index e87a863a2aa6..80ad27ddb4a1 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -12296,6 +12296,7 @@ int alloc_fair_sched_group(struct task_group *tg, struct task_group *parent) goto err; tg->shares = NICE_0_LOAD; + tg->latency_offset = 0; init_cfs_bandwidth(tg_cfs_bandwidth(tg)); @@ -12394,6 +12395,9 @@ void init_tg_cfs_entry(struct task_group *tg, struct cfs_rq *cfs_rq, } se->my_q = cfs_rq; + + se->latency_offset = tg->latency_offset; + /* guarantee group entities always have weight */ update_load_set(&se->load, NICE_0_LOAD); se->parent = parent; @@ -12524,6 +12528,35 @@ int sched_group_set_idle(struct task_group *tg, long idle) return 0; } +int sched_group_set_latency(struct task_group *tg, s64 latency) +{ + int i; + + if (tg == &root_task_group) + return -EINVAL; + + if (abs(latency) > sysctl_sched_latency) + return -EINVAL; + + mutex_lock(&shares_mutex); + + if (tg->latency_offset == latency) { + mutex_unlock(&shares_mutex); + return 0; + } + + tg->latency_offset = latency; + + for_each_possible_cpu(i) { + struct sched_entity *se = tg->se[i]; + + WRITE_ONCE(se->latency_offset, latency); + } + + mutex_unlock(&shares_mutex); + return 0; +} + #else /* CONFIG_FAIR_GROUP_SCHED */ void free_fair_sched_group(struct task_group *tg) { } diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index fd099f3961e4..5a4ce8e61f47 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -383,6 +383,8 @@ struct task_group { /* A positive value indicates that this is a SCHED_IDLE group. */ int idle; + /* latency constraint of the group. */ + int latency_offset; #ifdef CONFIG_SMP /* @@ -493,6 +495,8 @@ extern int sched_group_set_shares(struct task_group *tg, unsigned long shares); extern int sched_group_set_idle(struct task_group *tg, long idle); +extern int sched_group_set_latency(struct task_group *tg, s64 latency); + #ifdef CONFIG_SMP extern void set_task_rq_fair(struct sched_entity *se, struct cfs_rq *prev, struct cfs_rq *next); From patchwork Fri Jan 13 14:12:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 43326 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp293908wrt; Fri, 13 Jan 2023 06:17:39 -0800 (PST) X-Google-Smtp-Source: AMrXdXusPsshtUcqzVI8OEKobpvjf1av9crC5wM4uKiuDVaMsk0EEjEFJSUVw1XC7/cGvzX1BhtJ X-Received: by 2002:a17:906:6892:b0:7c1:7010:f413 with SMTP id n18-20020a170906689200b007c17010f413mr3266701ejr.19.1673619459134; Fri, 13 Jan 2023 06:17:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673619459; cv=none; d=google.com; s=arc-20160816; b=0HMZ0BdRaacFKC7KuD0Dtc6L7Kvh7DzXHx99wYs+UVX4HUnDFKuzwr/5tVvt9tWliC HxdCbMjFGtIVA5RK0JvB+dAPJKXOia60YOF3vprwE6fpJ6ALImGECPvjbIGOOHYam0ye F5jBdTCLmem6fQjDV/SZS7oRm+XdX2+a+e4M8V/O1o6ilm1PbmgWaFYSP0PzivNpbpsL tvf7MpFK2ZRVx+1/ZLwU70ubYTyTxZNr8ryfS+UepsCS0FWYyZ9/wMCt5uhSppgEgVV3 /4qYX2HOhpXB8xQ6aMr7kzW8PXkvEcYlPXK0JzK3YLyWvM3ebEfsHyzcFF8nhXI/G/gZ z4Kg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=V6TGebPqxsiNn0n5diQ98jmCY4ZYb1DFPf0O/f9N2dY=; b=TANuqxb5OiUrA+fCu8VMckicI6Ji0POdBHsIz9vfu4pcP3u4XFL9QwO0VMBgBMJDKz Qkiq/Y752X97PTHilAQZ9W9Xh8TaFYRJ39XVVcwUpXyRA5gW4fBcUCIaSfI9bHKhqYsT SlBA8qq6iRt4SzPpeQHF4PIOaIP3HW3LXHCXTATX/U3TrzcIwl/CWpvE5DUuzt8nl23v +Nf2RQHFWE7v7sEybKUQb7rJgGuOyWdL2U7YsEHRqHEBuJEqJNGuwPH3KtLg9VMmCm9o i3H8281mSXCmAzMvgScwS2frjVPuTglfjlCy8oAm8WYj5qLBa4RTi8oL6vmdDJNsa94I D8Nw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=kyoXFhhH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id hc41-20020a17090716a900b007c0dcc62e49si26023476ejc.56.2023.01.13.06.17.15; Fri, 13 Jan 2023 06:17:39 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=kyoXFhhH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240603AbjAMOPr (ORCPT + 99 others); Fri, 13 Jan 2023 09:15:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40434 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242056AbjAMONx (ORCPT ); Fri, 13 Jan 2023 09:13:53 -0500 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5CD6E6A0F6 for ; Fri, 13 Jan 2023 06:12:49 -0800 (PST) Received: by mail-wr1-x42d.google.com with SMTP id h16so21184469wrz.12 for ; Fri, 13 Jan 2023 06:12:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=V6TGebPqxsiNn0n5diQ98jmCY4ZYb1DFPf0O/f9N2dY=; b=kyoXFhhH4LSSWG6wMnAiaJ07jZSSGb0HZosWr9qzMojEznxN8iqtffNopToVMH/hUL HSaY1Cobj6yL8RFFk3xyrySF23tDrhRt8LAziuIYTh00/5o47whlA/ZFBFM9ylDKTxFb 9EFinVeA81cLatWd6SqRGbpWGmlkmVG8EZed4HEvr+AN0C2rW8x4chbzfrnIS54JyJ6I Qbm/O6OMzFBNLWTO+oMb7JJhCWtZCEJCnmTlUCFxMNkAiUK2rwHwiU60qPQhW6lbzFyp +lDDdKUk9x57PDc9PGqF9Kmc9FzBaQJdyWVIrUZK4jp/KaNKJubP3jBIDRjMvoCUz+D4 rAyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=V6TGebPqxsiNn0n5diQ98jmCY4ZYb1DFPf0O/f9N2dY=; b=k82yvB5Xxw/6RTpk12h6EeC4qlcphnng6mLF896j1r9rDA/bPtvD4iKcePwz4ThK4z 6q8mvTJCCgecbzULzNVFMrigrsNzRADSSmBKJthhCK+yDgngJR4rGo4jw0J04NNTxsAp lWjeyCuBpH/KfX/dJh8bM55dguBD36HlXl9SA+0lwRACKVFT0wm3fYnSAAhMQl5eoJhs RPn1KnmLPcYZrh40Bn0oo9YZEZTKjYSd0EfHu5d9Dk27v2O9oeW+KcfBuu45qqhfBGuH M+9SmhGar71lgospG2886DRpH4VyhYMl913B9DjPDRWbP/a2Pb5cYLaIgXIclh2H5P9d 7ugA== X-Gm-Message-State: AFqh2krh/cHksp7Q7dIf05Qoy0Zfnb4Qq2Px0rnvLXrEFQlt4+dMZ+Vh WIgAvGKgcCXcRG0h1t7y6l/w/Q== X-Received: by 2002:a5d:4692:0:b0:2bb:e94c:fcbf with SMTP id u18-20020a5d4692000000b002bbe94cfcbfmr13297857wrq.52.1673619168889; Fri, 13 Jan 2023 06:12:48 -0800 (PST) Received: from vingu-book.. ([2a01:e0a:f:6020:1563:65bf:c344:661e]) by smtp.gmail.com with ESMTPSA id f7-20020a5d6647000000b002bbeb700c38sm13869919wrw.91.2023.01.13.06.12.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Jan 2023 06:12:48 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com, cgroups@vger.kernel.org Cc: qyousef@layalina.io, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v10 7/9] sched/core: Support latency priority with sched core Date: Fri, 13 Jan 2023 15:12:32 +0100 Message-Id: <20230113141234.260128-8-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230113141234.260128-1-vincent.guittot@linaro.org> References: <20230113141234.260128-1-vincent.guittot@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754917198365179922?= X-GMAIL-MSGID: =?utf-8?q?1754917198365179922?= Take into account wakeup_latency_gran() when ordering the cfs threads. Signed-off-by: Vincent Guittot Tested-by: K Prateek Nayak --- kernel/sched/fair.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 80ad27ddb4a1..a4bfa03d096c 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -11971,6 +11971,9 @@ bool cfs_prio_less(const struct task_struct *a, const struct task_struct *b, delta = (s64)(sea->vruntime - seb->vruntime) + (s64)(cfs_rqb->min_vruntime_fi - cfs_rqa->min_vruntime_fi); + /* Take into account latency prio */ + delta -= wakeup_latency_gran(sea, seb); + return delta > 0; } #else From patchwork Fri Jan 13 14:12:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 43329 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp294355wrt; Fri, 13 Jan 2023 06:18:27 -0800 (PST) X-Google-Smtp-Source: AMrXdXvquwI47aznFJEe0VQ7YuaPMI9kln2Y0N+xtKO30w8+4pvlsVIJE29tknCQRH57VNoBcI98 X-Received: by 2002:a17:907:d681:b0:7c0:eb38:f8af with SMTP id wf1-20020a170907d68100b007c0eb38f8afmr68881539ejc.2.1673619506905; Fri, 13 Jan 2023 06:18:26 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673619506; cv=none; d=google.com; s=arc-20160816; b=tA2rvYGA37rPJ4QWmC6hR0jw+kc0NQxA735tK4vNrnHJ7FL3ubAaFHTts1W3qZMNOP nN9Q3o1RtewnhamtfuSgsK+/nnykTzIyIYyDPOcH44fk7jsf9HhXwX3pN3N2zYXWSoOx ibKjM5sUFanlFm7b1wEwHVX4/mVj7TRW4YEyyCbv0wrPXms5rGZ+w4rjKfXeO15dq/Os ANwNyZFI3vb+BfCFnmB+d/VI07m3xFCE6sfb329gQWmtgb8aRgPkAVBT/X8Ry6UaO9ZW 5NPjQB5gvR4beLY/5lXHsl4tIaiX/iXM7pz153UVWujV7maQi+ZAeCZn2ANhfEQxW+Mc HphQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=YWiIpOe2nNdT5aWZRMfGm3NcSfCw2CxcRRTzhzpvxrA=; b=XGnylhQPi0t5WZETigWwDdb/TqPqgulg2fWgVHOrZKTfjF0mbO3xCqtvk0uplGvzLL FqpnB4ZyrNKVIeXyhRz6DjxMWKrx6Jja6QZzGqYexssXYtvaPSvNp2g1eUU1k+XgGpvP vlCrVnnK7bD9UFIweeozT0TJ4QyjOoh/uBxwfCsSqDXL+FYiMJdUZz3NvRI8NV5ag2RJ agbDQYZENviyUbTMvhmWi1As7tE0oof6Ubw/3jDxQYO6S7a9QYNUkxlkDNQxHqhjq61o w5sEPDJ0v4aYqyBMSC69h66u/Pw0jcnbJjY5epdQhCUepH3slW3F3EvDjuBv82buLpA/ tFyQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=BywjmuYY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id qo14-20020a170907874e00b007c4fb1332c0si17696500ejc.174.2023.01.13.06.17.54; Fri, 13 Jan 2023 06:18:26 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=BywjmuYY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241149AbjAMOP7 (ORCPT + 99 others); Fri, 13 Jan 2023 09:15:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40432 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242066AbjAMONy (ORCPT ); Fri, 13 Jan 2023 09:13:54 -0500 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0F4FB3590A for ; Fri, 13 Jan 2023 06:12:52 -0800 (PST) Received: by mail-wm1-x331.google.com with SMTP id q8so3556792wmo.5 for ; Fri, 13 Jan 2023 06:12:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YWiIpOe2nNdT5aWZRMfGm3NcSfCw2CxcRRTzhzpvxrA=; b=BywjmuYYDQL0B21BIKa7iT2z+zdJTdpWyljP9Agfj2597r0VBHFUwD+R67fS51aQp7 VbuIrgy3kWGXYiQ+AbS40a9O3nOPS9zQ7D3TFtpaiZfn/uqD3EHq44UHZjX3is0nupNv N3xfnI864WldAvYuq/c2/aof7J0TL/NuZUi6kM22NjmqFjasPhF2FPEtAdcms4XvNet8 itm+i4Y4ADmVmnsCKI+/u+MNMrocBIza8Wircmv8MtKQ4+e0VcHPCl4ioHcGfoq3dnWP fI0NUB55RCG2tu31OHi5U1koTyaUjudyfB8StE8p0F8t3cp3XWaNYF7wC76RWyz9yC5D lgqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YWiIpOe2nNdT5aWZRMfGm3NcSfCw2CxcRRTzhzpvxrA=; b=bJ791cHXtPzZfMGMUYx9iY5h4em9Fus5AT1ALo+Aaha8uz5R2i69b5UUC35MLbhkEZ Jdwc+X5URABogMyKk+esSFf17KOpRt7QWFjqI0SGnlvh59opR8a63NpPv68Fm7kqj16o 43F12nzAi3t/dLzPjjQCck+kwGoB2CTrAAHqnymxN0F2x5/1NDFZ2KLkyi4q1aiISR8Y LQVALTqfbDizTFokKEZISWpH+T/ORpBzSZ6/47WZ984uLMOmdx6BWirolUyoQokxcq6e Xmu/b85Jcsn1AY+t3j/3h4juhea77Cras3S4G55KN9vmNi482y28vmXkyGgEDQIOOFYZ kngA== X-Gm-Message-State: AFqh2kooOHOV9z5HjyEK22lkY1zRIfUhPle7pntWXniR7TFGvrK0VG1S CIoSw0ZbQUueVNHooB/ei/IJHg== X-Received: by 2002:a05:600c:510b:b0:3da:2090:d404 with SMTP id o11-20020a05600c510b00b003da2090d404mr2041306wms.18.1673619170540; Fri, 13 Jan 2023 06:12:50 -0800 (PST) Received: from vingu-book.. ([2a01:e0a:f:6020:1563:65bf:c344:661e]) by smtp.gmail.com with ESMTPSA id f7-20020a5d6647000000b002bbeb700c38sm13869919wrw.91.2023.01.13.06.12.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Jan 2023 06:12:49 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com, cgroups@vger.kernel.org Cc: qyousef@layalina.io, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v10 8/9] sched/fair: Add latency list Date: Fri, 13 Jan 2023 15:12:33 +0100 Message-Id: <20230113141234.260128-9-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230113141234.260128-1-vincent.guittot@linaro.org> References: <20230113141234.260128-1-vincent.guittot@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754917248066803425?= X-GMAIL-MSGID: =?utf-8?q?1754917248066803425?= Add a rb tree for latency sensitive entities so we can schedule the most sensitive one first even when it failed to preempt current at wakeup or when it got quickly preempted by another entity of higher priority. In order to keep fairness, the latency is used once at wakeup to get a minimum slice and not during the following scheduling slice to prevent long running entity to got more running time than allocated to his nice priority. The rb tree enables to cover the last corner case where latency sensitive entity can't got schedule quickly after the wakeup. Signed-off-by: Vincent Guittot Tested-by: K Prateek Nayak --- include/linux/sched.h | 1 + kernel/sched/core.c | 1 + kernel/sched/fair.c | 95 +++++++++++++++++++++++++++++++++++++++++-- kernel/sched/sched.h | 1 + 4 files changed, 95 insertions(+), 3 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 38decae3e156..41bb92be5ecc 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -548,6 +548,7 @@ struct sched_entity { /* For load-balancing: */ struct load_weight load; struct rb_node run_node; + struct rb_node latency_node; struct list_head group_node; unsigned int on_rq; diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 6798c9a297d6..5c99f67c7e7b 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4390,6 +4390,7 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p) p->se.nr_migrations = 0; p->se.vruntime = 0; INIT_LIST_HEAD(&p->se.group_node); + RB_CLEAR_NODE(&p->se.latency_node); #ifdef CONFIG_FAIR_GROUP_SCHED p->se.cfs_rq = NULL; diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index a4bfa03d096c..a8f0e32431e2 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -680,7 +680,76 @@ struct sched_entity *__pick_last_entity(struct cfs_rq *cfs_rq) return __node_2_se(last); } +#endif + +/************************************************************** + * Scheduling class tree data structure manipulation methods: + * for latency + */ + +static inline bool latency_before(struct sched_entity *a, + struct sched_entity *b) +{ + return (s64)(a->vruntime + a->latency_offset - b->vruntime - b->latency_offset) < 0; +} + +#define __latency_node_2_se(node) \ + rb_entry((node), struct sched_entity, latency_node) + +static inline bool __latency_less(struct rb_node *a, const struct rb_node *b) +{ + return latency_before(__latency_node_2_se(a), __latency_node_2_se(b)); +} + +/* + * Enqueue an entity into the latency rb-tree: + */ +static void __enqueue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) +{ + + /* Only latency sensitive entity can be added to the list */ + if (se->latency_offset >= 0) + return; + + if (!RB_EMPTY_NODE(&se->latency_node)) + return; + + /* + * An execution time less than sysctl_sched_min_granularity means that + * the entity has been preempted by a higher sched class or an entity + * with higher latency constraint. + * Put it back in the list so it gets a chance to run 1st during the + * next slice. + */ + if (!(flags & ENQUEUE_WAKEUP)) { + u64 delta_exec = se->sum_exec_runtime - se->prev_sum_exec_runtime; + + if (delta_exec >= sysctl_sched_min_granularity) + return; + } + + rb_add_cached(&se->latency_node, &cfs_rq->latency_timeline, __latency_less); +} + +static void __dequeue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se) +{ + if (!RB_EMPTY_NODE(&se->latency_node)) { + rb_erase_cached(&se->latency_node, &cfs_rq->latency_timeline); + RB_CLEAR_NODE(&se->latency_node); + } +} + +static struct sched_entity *__pick_first_latency(struct cfs_rq *cfs_rq) +{ + struct rb_node *left = rb_first_cached(&cfs_rq->latency_timeline); + + if (!left) + return NULL; + + return __latency_node_2_se(left); +} +#ifdef CONFIG_SCHED_DEBUG /************************************************************** * Scheduling class statistics methods: */ @@ -4751,8 +4820,10 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) check_schedstat_required(); update_stats_enqueue_fair(cfs_rq, se, flags); check_spread(cfs_rq, se); - if (!curr) + if (!curr) { __enqueue_entity(cfs_rq, se); + __enqueue_latency(cfs_rq, se, flags); + } se->on_rq = 1; if (cfs_rq->nr_running == 1) { @@ -4838,8 +4909,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) clear_buddies(cfs_rq, se); - if (se != cfs_rq->curr) + if (se != cfs_rq->curr) { __dequeue_entity(cfs_rq, se); + __dequeue_latency(cfs_rq, se); + } se->on_rq = 0; account_entity_dequeue(cfs_rq, se); @@ -4928,6 +5001,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se) */ update_stats_wait_end_fair(cfs_rq, se); __dequeue_entity(cfs_rq, se); + __dequeue_latency(cfs_rq, se); update_load_avg(cfs_rq, se, UPDATE_TG); } @@ -4966,7 +5040,7 @@ static struct sched_entity * pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr) { struct sched_entity *left = __pick_first_entity(cfs_rq); - struct sched_entity *se; + struct sched_entity *latency, *se; /* * If curr is set we have to see if its left of the leftmost entity @@ -5008,6 +5082,12 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr) se = cfs_rq->last; } + /* Check for latency sensitive entity waiting for running */ + latency = __pick_first_latency(cfs_rq); + if (latency && (latency != se) && + wakeup_preempt_entity(latency, se) < 1) + se = latency; + return se; } @@ -5031,6 +5111,7 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev) update_stats_wait_start_fair(cfs_rq, prev); /* Put 'current' back into the tree. */ __enqueue_entity(cfs_rq, prev); + __enqueue_latency(cfs_rq, prev, 0); /* in !on_rq case, update occurred at dequeue */ update_load_avg(cfs_rq, prev, 0); } @@ -12244,6 +12325,7 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first) void init_cfs_rq(struct cfs_rq *cfs_rq) { cfs_rq->tasks_timeline = RB_ROOT_CACHED; + cfs_rq->latency_timeline = RB_ROOT_CACHED; u64_u32_store(cfs_rq->min_vruntime, (u64)(-(1LL << 20))); #ifdef CONFIG_SMP raw_spin_lock_init(&cfs_rq->removed.lock); @@ -12552,8 +12634,15 @@ int sched_group_set_latency(struct task_group *tg, s64 latency) for_each_possible_cpu(i) { struct sched_entity *se = tg->se[i]; + struct rq *rq = cpu_rq(i); + struct rq_flags rf; + + rq_lock_irqsave(rq, &rf); + __dequeue_latency(se->cfs_rq, se); WRITE_ONCE(se->latency_offset, latency); + + rq_unlock_irqrestore(rq, &rf); } mutex_unlock(&shares_mutex); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 5a4ce8e61f47..186873bb41e2 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -575,6 +575,7 @@ struct cfs_rq { #endif struct rb_root_cached tasks_timeline; + struct rb_root_cached latency_timeline; /* * 'curr' points to currently running entity on this cfs_rq. From patchwork Fri Jan 13 14:12:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 43328 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp294000wrt; Fri, 13 Jan 2023 06:17:49 -0800 (PST) X-Google-Smtp-Source: AMrXdXsHtayigPvEObFGL/a9jO7R62TvwlwQ/9Lk6w6M/GtphYL4M7vD6N1NH3VCTCp8rTbLj4qQ X-Received: by 2002:a17:906:c418:b0:7c1:fbea:c5ce with SMTP id u24-20020a170906c41800b007c1fbeac5cemr3023005ejz.7.1673619469233; Fri, 13 Jan 2023 06:17:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673619469; cv=none; d=google.com; s=arc-20160816; b=fziZmir2eJ4FnpEOW8nYhdw4o121yco1YSvegeUTrivDTybJfHv8y0xZL+HkwV3C8t j1TWf5OaYgBc4alJfrF/3hNb8e22fXcoU5sq6k8rqCC9cqsWDZy0rXe4TdfT4bb04/tN YfttKltNx7QuPgf4QK2wWvdV5F0GSCrKu+dp29fN/jddmp3bMBxF2yKSBTspJFq0SEZc biI+HkIojx+WQwBEFyL9y0QAeev/PJ2f+YAB95WGq96rSXGH45xQcO9Uuhi1RBBT4Mwb SYgZmHV5bzj03+Y+UZSF0wGC9i30Eobj3CxX2aB0TrdGu/9HrAGhyk1GP+TWWKLofoAE AXIQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=rfA7fOK3LCcmN6R2CxD4VildndRtCjudzidz68b+vdE=; b=F9T0365WFfIPanfZeAxC7Nxtdk16sgg7BZt8V2UBlVtoOZtCnssi1TFOYObubPn6h5 YNp6grCNIoDV0Luu8fy4583/Os5lR3kXYwUhb34Z3MfCijRw+2DJs/NLtCAWbv9UlrBY Fe86CM7hSci6s53Tydbn7hkm7W5hxz6ZmObAFTzwTB2Fry0KtnT1C5IafQGHO7m3RwpV tz3ANYDiO65fCsek2nyWdhO9LnHo3uTVdbRYuqPtlLGjpIkqXNLZlN09K1gW0XeMOQBs Pv2uBTTlILWXibrM1ApYVXyQwHTiPw1UEzFYnW+TeWmGxiPetr0twT5fjT7bCKvq4lyJ cFzg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=fO5NzU8X; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id hu9-20020a170907a08900b007c09df6e6a1si21497437ejc.923.2023.01.13.06.17.24; Fri, 13 Jan 2023 06:17:49 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=fO5NzU8X; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241599AbjAMOQN (ORCPT + 99 others); Fri, 13 Jan 2023 09:16:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40916 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242073AbjAMONz (ORCPT ); Fri, 13 Jan 2023 09:13:55 -0500 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 11559687BF for ; Fri, 13 Jan 2023 06:12:54 -0800 (PST) Received: by mail-wr1-x42e.google.com with SMTP id e3so11975715wru.13 for ; Fri, 13 Jan 2023 06:12:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rfA7fOK3LCcmN6R2CxD4VildndRtCjudzidz68b+vdE=; b=fO5NzU8X5twvuoxI8ACLZI8Gu+yGI2YSIMh067qxr7DL8dFlZI5g+mHS2UiYb3j4to viZ2CnpjxEKrL4kusySj8Ic40GZF0LCq6Pid4UYvGWpYSlfZG3oDxZMImm0B0G/GpiaG +jZNS0FPJfq1VMX4bBosB3QxN3Uu+giZgBLYATIUEDdo3b5Zo+EYbPhB8lqESdE6UhEF 3sBoCYdB68/uJ7bLSp6cFBWw+8VO5THrtEPNMT2PIq8tCHB+0A24FKS+30Kp1NY3pW6c 3pxx1WaKh3wiEW0dHOvJhV7cLSmKPwk9PSPtYeI1T3EH1zPMSYWrY+l5SHSQrsvI2htA /ekw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rfA7fOK3LCcmN6R2CxD4VildndRtCjudzidz68b+vdE=; b=a+p1UVd2gcQ6Q5jcGofkRAowPGtQCsxVk/aoF+hzw34Wjzmqf2UERF1LJwrF0Y7OqK w0ohXo20f8HN4Y2vBdDUZt7/clAhE02dJ/2t5z0o0tAeP4CTPD1CCk7dwqAw1ISSqev/ Uvb5Ym5fkp19J6hxetXMIaA0Bp+9jrhPAeot4w0pRib7o+anHG6GQxAzXw1BghVkB5Mo 1UxAvDOOJcnCdaGGdgD12DYBs+ybLkXs2c6QVm3jzwIQGZlRjSnOzjwNgFWABkug7Fdz p7+K7ZchXT+4oCZr5JCjHS4ucgJS3m5q01n5/Jg5l6ZmSQuEBVpmeuECemdSwygqbutO O15w== X-Gm-Message-State: AFqh2krQTqsOSN7T9NICFjMDNbPb8Kbb82vmjE/pciBQGjksgHJN5J3P 4Xpl8regNeXQx2KDt0Q4XzqbhA== X-Received: by 2002:a05:6000:806:b0:293:1868:3a14 with SMTP id bt6-20020a056000080600b0029318683a14mr32347854wrb.0.1673619173276; Fri, 13 Jan 2023 06:12:53 -0800 (PST) Received: from vingu-book.. ([2a01:e0a:f:6020:1563:65bf:c344:661e]) by smtp.gmail.com with ESMTPSA id f7-20020a5d6647000000b002bbeb700c38sm13869919wrw.91.2023.01.13.06.12.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Jan 2023 06:12:51 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com, cgroups@vger.kernel.org Cc: qyousef@layalina.io, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v10 9/9] sched/fair: remove check_preempt_from_others Date: Fri, 13 Jan 2023 15:12:34 +0100 Message-Id: <20230113141234.260128-10-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230113141234.260128-1-vincent.guittot@linaro.org> References: <20230113141234.260128-1-vincent.guittot@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754917208223155260?= X-GMAIL-MSGID: =?utf-8?q?1754917208223155260?= With the dedicated latency list, we don't have to take care of this special case anymore as pick_next_entity checks for a runnable latency sensitive task. Signed-off-by: Vincent Guittot Tested-by: K Prateek Nayak --- kernel/sched/fair.c | 34 ++-------------------------------- 1 file changed, 2 insertions(+), 32 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index a8f0e32431e2..fb2a5b2e3440 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6240,35 +6240,6 @@ static int sched_idle_cpu(int cpu) } #endif -static void set_next_buddy(struct sched_entity *se); - -static void check_preempt_from_others(struct cfs_rq *cfs, struct sched_entity *se) -{ - struct sched_entity *next; - - if (se->latency_offset >= 0) - return; - - if (cfs->nr_running <= 1) - return; - /* - * When waking from another class, we don't need to check to preempt at - * wakeup and don't set next buddy as a candidate for being picked in - * priority. - * In case of simultaneous wakeup when current is another class, the - * latency sensitive tasks lost opportunity to preempt non sensitive - * tasks which woke up simultaneously. - */ - - if (cfs->next) - next = cfs->next; - else - next = __pick_first_entity(cfs); - - if (next && wakeup_preempt_entity(next, se) == 1) - set_next_buddy(se); -} - /* * The enqueue_task method is called before nr_running is * increased. Here we update the fair scheduling stats and @@ -6355,15 +6326,14 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) if (!task_new) update_overutilized_status(rq); - if (rq->curr->sched_class != &fair_sched_class) - check_preempt_from_others(cfs_rq_of(&p->se), &p->se); - enqueue_throttle: assert_list_leaf_cfs_rq(rq); hrtick_update(rq); } +static void set_next_buddy(struct sched_entity *se); + /* * The dequeue_task method is called before nr_running is * decreased. We remove the task from the rbtree and