From patchwork Fri Oct 28 07:36:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 12136 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp679280wru; Fri, 28 Oct 2022 00:41:49 -0700 (PDT) X-Google-Smtp-Source: AMsMyM4aEait8dw21zjTyzKy2OimZkyrfxpDsoxZoSZeQNAqRG7VnOWWDfuPpAAChP/bdm/zHt4Y X-Received: by 2002:aa7:d341:0:b0:461:4db9:24f6 with SMTP id m1-20020aa7d341000000b004614db924f6mr34211848edr.87.1666942909826; Fri, 28 Oct 2022 00:41:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666942909; cv=none; d=google.com; s=arc-20160816; b=XpbGZOXsEAYh9dT+uPQZhQ8GoA99iIeSbXrhBCDN6skGNto3FdxSL2HvmQ4VOI9TaX Mh3tsgKWVu5alK1zJ5seDi6p4UlNpfQxm/qv4m194DeteHF+OxrfSiZVjkOaj3V/A67y Z8RWkThUOOr/GNTn5VTzoDYs9OBtW2Rnh+YyKlgw50fty+/58MFgzg+2VhQecCqFoGXR heFIwzcW0jM38kkkvgXVMv87PKij8I95AZN+eOgHbzBKHc+RfqsCl9pnAmvE6jQRIZrY AaoTlegw+wYLWQP+FBHvvkxCbs8aKcoz3mUni795lP2xjHolcXzlfTt+KseKnOMhhOD0 AHTQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=ps1HJOKEnWCS/7lhYsGYqdzZOp8bQxvEb3UkDJ00+rE=; b=GjL0d2Ml8PSMMFXOjadhoCyYoItVh2UthTN2PHoMiF7UZ7WcCqIstK0sW4/OPhEsiH oy77BgQ1CcG9SxooAfQq1NAyhABcwjn7HGrm9iLMrFDulPjJc8wAuGYEbMh0kwppw1NH 3m4o/44Zm5+EOp/XleY66XMrVsoWl3BvJi/s3Czrrc7kyNDc0jxuv/0mP0yYqWooJbn0 dLueYN+m/LF7qZP4F4/1W9SFnDry4BO/eFL6tLYtlVrdhGryjj6CYj7naZzXr7qarkAq Zl0u6qA6085t+Rn26nF9KbFYZTtwUTLUiXMbdqTP0axvqnP+g3rH8UgUb+V7v6PbXTNH ZHqQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="r9vIh/ry"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id s17-20020a50ab11000000b0045d189ac612si3354735edc.402.2022.10.28.00.41.26; Fri, 28 Oct 2022 00:41:49 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="r9vIh/ry"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230180AbiJ1Hgx (ORCPT + 99 others); Fri, 28 Oct 2022 03:36:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42872 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229543AbiJ1Hgv (ORCPT ); Fri, 28 Oct 2022 03:36:51 -0400 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4EE4EA2AAE for ; Fri, 28 Oct 2022 00:36:50 -0700 (PDT) Received: by mail-wm1-x32e.google.com with SMTP id y10so2512368wma.0 for ; Fri, 28 Oct 2022 00:36:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=ps1HJOKEnWCS/7lhYsGYqdzZOp8bQxvEb3UkDJ00+rE=; b=r9vIh/ryRXdC+rppRMO3FTwE+0X1f+lWJeXTpplDr1lwrJHn/xrHVLoKDD6kZMVagk j93h8jTAYJtL8vF9kjq21yJrD8slyw3iXGnirmuMXPajEwK1VVsjMArBtJix0T8+fabz I2lXoSYDdErFeK8SoY6C3JedlwVBUNAx2vPXC7o7rm5AvK2o8Gqlrhsv2Gsm8/wNAh6L fb7JLIckOITXrPF8D4a80BUAZxOhMDSApK91MR4HX9keJ35xz0wwIpbbMwE6i6aF/c5K 1p/IDsnj6LX9KvyNNu3C/swsXaAnzurA7j1nXH6b9sRXCAiMrlPXhkaxfaN8DkrB8eZS w+1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ps1HJOKEnWCS/7lhYsGYqdzZOp8bQxvEb3UkDJ00+rE=; b=zQKVXmjdre5KcPpw7FoyDXGgao9xWskTKxD/4ZQoHoFEb5tCfOEAd/6jNMeykAOsbC IBZaX9xaBY6r9OLz73fosb9vN09XiK63sQMzqjx3TRJEMmAuwAW6ksVO/2e+BM+8PaCR aeFBvlLjBvi+0M9gF+te7kSn0ZCT6kkCTnPRcmAk10TqLPrLiDjoZ/qba7dGIIh+HbnM SxX4XzCEMLYsfW+8baC2GNVKz6d6Y11MDAY4ADh+pU7eB9FPfZe3NO7jnk4fzn3gXgsv rqom72fTCYyl5ug0oFKmroAqjkQ5P8dXZAFql+f4/ypLYK5vBbgd/K68g5cn6uEbq/+Z o1kA== X-Gm-Message-State: ACrzQf2yAjJtduBRiLc6sIaFNP2HW/ABIQmd7On2JGBwjAxeo3HHsziD zgMvfDR995D9nptfJ+G7OzfO5g== X-Received: by 2002:a1c:c90c:0:b0:3b4:adc7:9766 with SMTP id f12-20020a1cc90c000000b003b4adc79766mr8555482wmb.66.1666942608656; Fri, 28 Oct 2022 00:36:48 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:f:6020:c12b:b448:f0a9:83ef]) by smtp.gmail.com with ESMTPSA id j8-20020a05600c190800b003b47e8a5d22sm7783129wmq.23.2022.10.28.00.36.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Oct 2022 00:36:47 -0700 (PDT) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com Cc: qais.yousef@arm.com, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v6 1/9] sched/fair: fix unfairness at wakeup Date: Fri, 28 Oct 2022 09:36:29 +0200 Message-Id: <20221028073637.31195-2-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221028073637.31195-1-vincent.guittot@linaro.org> References: <20221028073637.31195-1-vincent.guittot@linaro.org> X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DIET_1,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747916328376208008?= X-GMAIL-MSGID: =?utf-8?q?1747916328376208008?= At wake up, the vruntime of a task is updated to not be more older than a sched_latency period behind the min_vruntime. This prevents long sleeping task to get unlimited credit at wakeup. Such waking task should preempt current one to use its CPU bandwidth but wakeup_gran() can be larger than sched_latency, filter out the wakeup preemption and as a results steals some CPU bandwidth to the waking task. Make sure that a task, which vruntime has been capped, will preempt current task and use its CPU bandwidth even if wakeup_gran() is in the same range as sched_latency. If the waking task failed to preempt current it could to wait up to sysctl_sched_min_granularity before preempting it during next tick. Strictly speaking, we should use cfs->min_vruntime instead of curr->vruntime but it doesn't worth the additional overhead and complexity as the vruntime of current should be close to min_vruntime if not equal. Signed-off-by: Vincent Guittot --- kernel/sched/fair.c | 46 ++++++++++++++++++++------------------------ kernel/sched/sched.h | 28 +++++++++++++++++++++++++++ 2 files changed, 49 insertions(+), 25 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 5ffec4370602..eb04c83112a0 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4345,33 +4345,17 @@ place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int initial) { u64 vruntime = cfs_rq->min_vruntime; - /* - * The 'current' period is already promised to the current tasks, - * however the extra weight of the new task will slow them down a - * little, place the new task so that it fits in the slot that - * stays open at the end. - */ - if (initial && sched_feat(START_DEBIT)) - vruntime += sched_vslice(cfs_rq, se); - - /* sleeps up to a single latency don't count. */ - if (!initial) { - unsigned long thresh; - - if (se_is_idle(se)) - thresh = sysctl_sched_min_granularity; - else - thresh = sysctl_sched_latency; - + if (!initial) + /* sleeps up to a single latency don't count. */ + vruntime -= get_sched_latency(se_is_idle(se)); + else if (sched_feat(START_DEBIT)) /* - * Halve their sleep time's effect, to allow - * for a gentler effect of sleepers: + * The 'current' period is already promised to the current tasks, + * however the extra weight of the new task will slow them down a + * little, place the new task so that it fits in the slot that + * stays open at the end. */ - if (sched_feat(GENTLE_FAIR_SLEEPERS)) - thresh >>= 1; - - vruntime -= thresh; - } + vruntime += sched_vslice(cfs_rq, se); /* ensure we never gain time by being placed backwards. */ se->vruntime = max_vruntime(se->vruntime, vruntime); @@ -7187,6 +7171,18 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se) return -1; gran = wakeup_gran(se); + + /* + * At wake up, the vruntime of a task is capped to not be older than + * a sched_latency period compared to min_vruntime. This prevents long + * sleeping task to get unlimited credit at wakeup. Such waking up task + * has to preempt current in order to not lose its share of CPU + * bandwidth but wakeup_gran() can become higher than scheduling period + * for low priority task. Make sure that long sleeping task will get a + * chance to preempt current. + */ + gran = min_t(s64, gran, get_latency_max()); + if (vdiff > gran) return 1; diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 1fc198be1ffd..cdb84aec8ed5 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2448,6 +2448,34 @@ extern unsigned int sysctl_numa_balancing_scan_period_max; extern unsigned int sysctl_numa_balancing_scan_size; #endif +static inline unsigned long get_sched_latency(bool idle) +{ + unsigned long thresh; + + if (idle) + thresh = sysctl_sched_min_granularity; + else + thresh = sysctl_sched_latency; + + /* + * Halve their sleep time's effect, to allow + * for a gentler effect of sleepers: + */ + if (sched_feat(GENTLE_FAIR_SLEEPERS)) + thresh >>= 1; + + return thresh; +} + +static inline unsigned long get_latency_max(void) +{ + unsigned long thresh = get_sched_latency(false); + + thresh -= sysctl_sched_min_granularity; + + return thresh; +} + #ifdef CONFIG_SCHED_HRTICK /* From patchwork Fri Oct 28 07:36:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 12134 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp679070wru; Fri, 28 Oct 2022 00:41:10 -0700 (PDT) X-Google-Smtp-Source: AMsMyM5imsGp0DZHlhF61nX8tlA259GPMpdeDxj4K7NKQp3bw8M5NK9qJykmnMKYhpgDQhEQW7dX X-Received: by 2002:a17:907:a42c:b0:7aa:97e5:fac6 with SMTP id sg44-20020a170907a42c00b007aa97e5fac6mr19659236ejc.378.1666942869879; Fri, 28 Oct 2022 00:41:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666942869; cv=none; d=google.com; s=arc-20160816; b=LbkMWLT4+WdttG69I5qQvQOqDN7yi/D4NT9kkM8B35SYOB95nsyxxNvAgkwT4AxI8F /AzVeWrOfb8lfAJ9dILVe4eD+VpvdueelQRVRwe+fTBDcN6NOREeP/tyk4vWL3n2DfL6 qlYPOAHsGHEn+SDU4h5oerFXPWhnvJHJYKOKssnwMzEhwjrljvWetVZVBtgv6xh38RMg 1RGWBr9O3r3TXpxDPIXtyMf5UAkIJz3+u+TpEUE4OykS+F2jN7BKWlFxgFxh++bpADwI bk/6KSsXrYcjUyH+oNM/wAgDiVBluG+e8WxKX6NoML23DpmgF3J8D47xtm1UD7h0sgfi wqHw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=B0XWvRu1fVdTl/QBgfNPXdbzdRti+Q21J6ft5UqwBWk=; b=frAn0A6YoQy0xQDe2TKpKACAnWpZUHq88WqmpYucYyWOHpwkJgRB6mnTZIVYUNJEuW f64RjuGnM7h9POgzNuTHSpfNF6Fo0GLU1PLDzk3oMCMA3Y6DqUCD6FsIrMlAC+ItVFHv +UmVtJgWyz0cFw1QeVsKAvAgzgS6Eac9yURcqhuJ4eEUdsgKB4C8CXrx5OKCNNTpdUiU 6PGM7rlp7z7lA4PX/ZPlLbb9KVZsoJGEnvlgsvyAVUBTkCYPmpe8SKNYfPBOO6usNknl /wkDzaF5d0g5pE7NvQL7HUu3IidoHm02mY0rkKCDgJ2eRAl+tIkmVVmUlph9orTKvXXB NyNw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ubrUnxCv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id hd7-20020a170907968700b00777581091d9si3877901ejc.634.2022.10.28.00.40.45; Fri, 28 Oct 2022 00:41:09 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ubrUnxCv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230194AbiJ1Hg6 (ORCPT + 99 others); Fri, 28 Oct 2022 03:36:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42902 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230182AbiJ1Hgy (ORCPT ); Fri, 28 Oct 2022 03:36:54 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D01C71ABA1E for ; Fri, 28 Oct 2022 00:36:52 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id g12so5819764wrs.10 for ; Fri, 28 Oct 2022 00:36:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=B0XWvRu1fVdTl/QBgfNPXdbzdRti+Q21J6ft5UqwBWk=; b=ubrUnxCv59FBkGL/I9iLuY7+6esUZPHBfx+yAm96qCrlGCvlGAcCtfUVoiutjB6GPs g1fvjamhlKaJQb1aMbtj/MHtmroYKlcNJLvES3rZi7WuwrD/pB8YT/GqMS+sLiU6T1so 3u8YAP15ZgGwPVgXIoFiEjHOO2VFZBMg+D86M7u5J4D5AOS9Ecd/SQ4jgutIgOFgrw/9 NHYywtWvxg+tnHzGJ7xevv6Rj0U76deop0og/hDZaAWdWm4dXlMxhMFTlDyYd3I0db0+ 5HR4XYfZcGhSrxp7s7Ml9dIayVTdKG6a+Yz7pujJTiCTxrGW6PB0jevR2rtKTo5q6BpZ ouPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=B0XWvRu1fVdTl/QBgfNPXdbzdRti+Q21J6ft5UqwBWk=; b=dAAZJzEbOGGj0hkoZDDV8lsACIWFApojTqjt62OL1HXWypPuNYX5cSaa6ivn4jLl1H xzJjDtOfK0o1Xt2sZ2bmciQO06c19ndPruuag2X78IzanzSInxPSvaJ9dCvcAoMYMHzY IJkolxs7OcaKNAUQm44k4eJZ5ZySphAKQQ/Ll2plbuO9LcNMJ8TwDyhFZtyIz9FGMYyk vyWYEQgz0HlcT48kVkWakL19qAfSqXxFoZ8Jldsg7vW5qlHNRY5rJYkXY1kWdhXRCLLm GIWsQXS2e1IVidONRT9rth2qYNhjsHgljNCnbWci52ArWiiVPe1KTYWGc45ZP6aP7L5w Dnww== X-Gm-Message-State: ACrzQf0R7YDqXq//MEAqs9fAd582UjkXJcdYylm7MdLNggPLeRCGDnbV wpvwmJIIY5e5LD4a3l3GS4YtyA== X-Received: by 2002:adf:ba8f:0:b0:22c:def3:1179 with SMTP id p15-20020adfba8f000000b0022cdef31179mr34940134wrg.571.1666942611258; Fri, 28 Oct 2022 00:36:51 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:f:6020:c12b:b448:f0a9:83ef]) by smtp.gmail.com with ESMTPSA id j8-20020a05600c190800b003b47e8a5d22sm7783129wmq.23.2022.10.28.00.36.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Oct 2022 00:36:49 -0700 (PDT) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com Cc: qais.yousef@arm.com, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v6 2/9] sched: Introduce latency-nice as a per-task attribute Date: Fri, 28 Oct 2022 09:36:30 +0200 Message-Id: <20221028073637.31195-3-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221028073637.31195-1-vincent.guittot@linaro.org> References: <20221028073637.31195-1-vincent.guittot@linaro.org> X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747916286674890170?= X-GMAIL-MSGID: =?utf-8?q?1747916286674890170?= From: Parth Shah Latency-nice indicates the latency requirements of a task with respect to the other tasks in the system. The value of the attribute can be within the range of [-20, 19] both inclusive to be in-line with the values just like task nice values. latency_nice = -20 indicates the task to have the least latency as compared to the tasks having latency_nice = +19. The latency_nice may affect only the CFS SCHED_CLASS by getting latency requirements from the userspace. Additionally, add debugging bits for newly added latency_nice attribute. Signed-off-by: Parth Shah [rebase] Signed-off-by: Vincent Guittot --- include/linux/sched.h | 1 + kernel/sched/debug.c | 1 + kernel/sched/sched.h | 18 ++++++++++++++++++ 3 files changed, 20 insertions(+) diff --git a/include/linux/sched.h b/include/linux/sched.h index 15e3bd96e4ce..6805f378a9c3 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -783,6 +783,7 @@ struct task_struct { int static_prio; int normal_prio; unsigned int rt_priority; + int latency_nice; struct sched_entity se; struct sched_rt_entity rt; diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c index bb3d63bdf4ae..a3f7876217a6 100644 --- a/kernel/sched/debug.c +++ b/kernel/sched/debug.c @@ -1042,6 +1042,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns, #endif P(policy); P(prio); + P(latency_nice); if (task_has_dl_policy(p)) { P(dl.runtime); P(dl.deadline); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index cdb84aec8ed5..5445a894babd 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -125,6 +125,24 @@ extern int sched_rr_timeslice; */ #define NS_TO_JIFFIES(TIME) ((unsigned long)(TIME) / (NSEC_PER_SEC / HZ)) +/* + * Latency nice is meant to provide scheduler hints about the relative + * latency requirements of a task with respect to other tasks. + * Thus a task with latency_nice == 19 can be hinted as the task with no + * latency requirements, in contrast to the task with latency_nice == -20 + * which should be given priority in terms of lower latency. + */ +#define MAX_LATENCY_NICE 19 +#define MIN_LATENCY_NICE -20 + +#define LATENCY_NICE_WIDTH \ + (MAX_LATENCY_NICE - MIN_LATENCY_NICE + 1) + +/* + * Default tasks should be treated as a task with latency_nice = 0. + */ +#define DEFAULT_LATENCY_NICE 0 + /* * Increase resolution of nice-level calculations for 64-bit architectures. * The extra resolution improves shares distribution and load balancing of From patchwork Fri Oct 28 07:36:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 12135 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp679146wru; Fri, 28 Oct 2022 00:41:25 -0700 (PDT) X-Google-Smtp-Source: AMsMyM4Rq8iukC8LMB4yguK66aLgF2KRQZzz6R73qyRXDjhlab601F5FXrP+zyLrFIWl0YH40dFr X-Received: by 2002:a17:907:7627:b0:78d:b6f5:9f15 with SMTP id jy7-20020a170907762700b0078db6f59f15mr46575074ejc.149.1666942885094; Fri, 28 Oct 2022 00:41:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666942885; cv=none; d=google.com; s=arc-20160816; b=SkiFqhB9+JkoP/uOxNRYtd6kgk5m/rxnCFFvK9IFu0G4n0fAAfeW8iOtMyeIx8fObf T053iKXYN2y2Pw4ZFAJqH1jgbvrh88kpGQaPRoFmdzehl6/u4UhErwLjTnyFf8Z/AoOb ZOF3UhbZJn3HJ6zZZ3pND7vLbPtLkEXp0dwYN8+mmFL9W6GUUoxwy9aJWURkB1JmNu2e qsQVMnUsZYVFddluV4L/JYzmcGj92IrsEuW5tpSvyVZlO+FL42Zdp8M2ByXUFXJoDaX7 zpZCSCdC/yljJ9ELYPKGI+reLhfW33x+1dKTdAQL9bsIXr0c+2k3/oyN+lzrenWDunib 0dmA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=ySpaS8Fkj5Miw0XHBXwZ3O8DcUQ428wDyrUI4Txxz7I=; b=lzloDYayVw2Vn7kJh363lV7/XcyjKxXWx3no3s1cS6PjZXYoyNGR1cFZte+JglrPjC LWYrkajTGLCj16UWQwGcQWt3tH+s/7CV58RmUBWxE3JzZ4EsJjfRcM0BcBHmsU37CkwX Ps+bYQ/BL9K8i4RyvvtZSPmATEazQ7Z5dVhBXm53cPeKvj6VbS2q105SEt7QFXYKbKVA AUVJ0Nl2vjLlj6LrP7sEndGL2pJJh6V9hTJI+d4nkyf4X7o9Wnzqu0hUlwR6BGz9YHxA uL2m6m83/HUvceG0NZx9A68lMPDtUOWyDJuLCv2/AE7/Ep8iINiuL11t68rj5Jp/3mGm TxeQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=wqSeFbRd; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ji13-20020a170907980d00b0078d85777c4fsi4390918ejc.700.2022.10.28.00.41.01; Fri, 28 Oct 2022 00:41:25 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=wqSeFbRd; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230206AbiJ1HhC (ORCPT + 99 others); Fri, 28 Oct 2022 03:37:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42936 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230185AbiJ1Hg4 (ORCPT ); Fri, 28 Oct 2022 03:36:56 -0400 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 39B2C1BB579 for ; Fri, 28 Oct 2022 00:36:55 -0700 (PDT) Received: by mail-wr1-x434.google.com with SMTP id h9so5894025wrt.0 for ; Fri, 28 Oct 2022 00:36:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=ySpaS8Fkj5Miw0XHBXwZ3O8DcUQ428wDyrUI4Txxz7I=; b=wqSeFbRdq2adahV/mdUOa1fqJD6aVJJovMUxYS+oMGCv12Q3FUv9AHXl4ggCN9Eo2j 96+U5EEOxOoQZ+FUtLT0YZhrHzFKzFUx9VjE6IzHV2wkARJeZBxUAJfB1VSsKDLs5BSO 5iqzHmI3yJT/4wMQ8sQ5l9fV6ReSQDtOEJfvUAxGwdiIQTaBuL89u4KmgRmWik2XLXIj zefQBzdrz6vtgunooANIBekXTuye4ROpVYuFsXCK98gV1PW5hzRkDB90d/Q+myXBBfXk ltabfUODHhh33AjDCP46QLX32FN7Jn59jmZcoy6SrGYwxVVVZ7D9WW8beT27BrFLYH44 Utpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ySpaS8Fkj5Miw0XHBXwZ3O8DcUQ428wDyrUI4Txxz7I=; b=UmPuyyARMpFtRiab/cg5WDEwrDW5K8nCLxhgiKVBV7R/oLeYBHxez0K+9MZsAq68qj TY3dQiv/dAfCB2II8Wkte221leQ65+m1yI2xcdY9hAYuPmaIXXammXhdQqwllWM0Vtas mmpXrgEMaQbDHjTLqjQAVxFn0cM24zfp4mCJ3NkB0swvVh7HUol725HmAmCKMm7nVFxZ t1UfhDc6fIYJchxPWLO7LrGDdm++w4IPbOUqyBh73lfD7RSBgqwrEGUFVVI1dT1YLu5i dAL4AXYAHnZl1N40I8psVXp6f4h8MEoclSbGjNdBVz9uJgs7DYPBx/bvVV9Qlnouuglr hFWw== X-Gm-Message-State: ACrzQf3TfBJST/uZRj8ZUziMbp46Ltp/dlyR9LfPZTdJuyedesLWXkPE omszR1T5Y8ZbCCJskZlp1waQrQ== X-Received: by 2002:a05:6000:719:b0:236:73ff:9ca3 with SMTP id bs25-20020a056000071900b0023673ff9ca3mr15200834wrb.603.1666942613661; Fri, 28 Oct 2022 00:36:53 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:f:6020:c12b:b448:f0a9:83ef]) by smtp.gmail.com with ESMTPSA id j8-20020a05600c190800b003b47e8a5d22sm7783129wmq.23.2022.10.28.00.36.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Oct 2022 00:36:52 -0700 (PDT) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com Cc: qais.yousef@arm.com, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v6 3/9] sched/core: Propagate parent task's latency requirements to the child task Date: Fri, 28 Oct 2022 09:36:31 +0200 Message-Id: <20221028073637.31195-4-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221028073637.31195-1-vincent.guittot@linaro.org> References: <20221028073637.31195-1-vincent.guittot@linaro.org> X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747916302913893679?= X-GMAIL-MSGID: =?utf-8?q?1747916302913893679?= From: Parth Shah Clone parent task's latency_nice attribute to the forked child task. Reset the latency_nice value to default value when the child task is set to sched_reset_on_fork. Also, initialize init_task.latency_nice value with DEFAULT_LATENCY_NICE value Signed-off-by: Parth Shah [rebase] Signed-off-by: Vincent Guittot --- init/init_task.c | 1 + kernel/sched/core.c | 1 + 2 files changed, 2 insertions(+) diff --git a/init/init_task.c b/init/init_task.c index ff6c4b9bfe6b..7dd71dd2d261 100644 --- a/init/init_task.c +++ b/init/init_task.c @@ -78,6 +78,7 @@ struct task_struct init_task .prio = MAX_PRIO - 20, .static_prio = MAX_PRIO - 20, .normal_prio = MAX_PRIO - 20, + .latency_nice = DEFAULT_LATENCY_NICE, .policy = SCHED_NORMAL, .cpus_ptr = &init_task.cpus_mask, .user_cpus_ptr = NULL, diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 02dc1b8e3cb6..54544353025b 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4559,6 +4559,7 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p) p->prio = p->normal_prio = p->static_prio; set_load_weight(p, false); + p->latency_nice = DEFAULT_LATENCY_NICE; /* * We don't need the reset flag anymore after the fork. It has * fulfilled its duty: From patchwork Fri Oct 28 07:36:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 12137 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp679298wru; Fri, 28 Oct 2022 00:41:53 -0700 (PDT) X-Google-Smtp-Source: AMsMyM5pY9Pdzdzi308SzYVjsCJfebj0lT5rA8KcU8ZEn27j01RfNHhZN/8/W8uQrq4JUCJnsXz+ X-Received: by 2002:a05:6402:b56:b0:461:91aa:11b7 with SMTP id bx22-20020a0564020b5600b0046191aa11b7mr28610939edb.220.1666942913468; Fri, 28 Oct 2022 00:41:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666942913; cv=none; d=google.com; s=arc-20160816; b=hPgHZaHH5MCWp6cHFYUwqxMqI3mnVAJG4q5jvHm8ndBjs/jIoluDHuQfVdqzLpgzG2 HwYbPaSazBJwNk+kRgTAXcZAtyaQPdoAsWkkpc75RIUSP3NQp/4FFUpJjnBA59K+Q0ID Qd7fkREZ44RA+NhKeylRgVmP+zZ68QDFqFdQlziXQupYb3G67cWmOl/WdJVntM/fOU2t qozBsfKWpiWsUb9fcRc2s/wTPoKkEVNhiX1MnqaTfgBEmJ6aBzJxvYHcV7/b/IM24xYG WW/B66JpqWxKIAJOTm5XBf58ByhQ+NVMaQ0pr9TCQHQmjCA7lVx3yeSAG7WCo+N4fQ1i QSEQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=VaAhG4KgLjRKvDa8P9MUGAqTUpov8Fc/6Ib8AC8OEG0=; b=RITcp2fw1Lfgc3DLIbZyfZJrF7LYcMtWBc9pk0yRYTwqKTkhmWuI9Ypi6OG469T9Vy bzbTZUklkT9WW2cA5AVy/OkSXFIdUDv2nUY2Y3K63A3r4xYUTrjwJd5EUy6+HN34sS9e d+FTAJG40buE8G6baXV/Y7H4j017XWGVeYJAXqY5o0VvuMkaPdJyqwdSyJEMuukN+Yq8 qT0OecsEZH9mrnOPCnzqIadtdsmfc7YeWL4KQGL2bQOEDyM8CdPm/VlkgkIjR1gnV0Fo 7AYEnR+25awkU3HnonRb4AbiNrHCDeRrThHTcvUDtHYXRv9B9vePrK7u9G0CzC8PqNv1 ZJIA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=KC0d30CV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id gn14-20020a1709070d0e00b007414886601asi4110420ejc.25.2022.10.28.00.41.30; Fri, 28 Oct 2022 00:41:53 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=KC0d30CV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230200AbiJ1HhH (ORCPT + 99 others); Fri, 28 Oct 2022 03:37:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42980 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230186AbiJ1Hg5 (ORCPT ); Fri, 28 Oct 2022 03:36:57 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 59C62EA9D8 for ; Fri, 28 Oct 2022 00:36:56 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id g12so5819998wrs.10 for ; Fri, 28 Oct 2022 00:36:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=VaAhG4KgLjRKvDa8P9MUGAqTUpov8Fc/6Ib8AC8OEG0=; b=KC0d30CVZ0BaVdqHlWW2XQleYHfGn+zRSEQLi3cfoXxI2rUbB/3Ym3UJAkgrpAQoW1 K6rdwX4hpTxJjZw9imF7M7m/K/tWtAUqh4UDi5GuY0X7OhogKkyM5iP1JRyrMn5v2Lb+ amZRqaYXpxeRiCsBUoxoMaEjD2dyWHl933h1g1qpYX86S7zI77VO9Vjz1Dqeeh7Ijv0t m1Hgz3GZEmg+tI/2Q/lkpjajSg8w2FQXYE5G/8xgrp0iE8oNtVOPI7sTMJUOoY9k+GTh DyzjynK6WBJS/xTvgv+js9nGrvN3jH0vwFy3YlQBGIqzeJoRvKPOm2UZ1WT3ZLKIxlbu CMZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=VaAhG4KgLjRKvDa8P9MUGAqTUpov8Fc/6Ib8AC8OEG0=; b=CMSID/corSPDtZbXmULWxaJTX9KwH3Ccf15UEP9j4X81BZXSM7wfR0XoIE+hnriv8o YGcLohXFsXA+QZ30IaCVi48W+upAAtBkTPragf/h+mk/uqtHUFx/9/DEk23pTCPVQJEk 9wWUhHj7nYieb/Z3Aj04JUnChgh+PVxYlNVdXs02IrQ/SpajGiODuL/mSazLu3DjyLK/ 2IHoyv9nOGv9MOC3kh7AFPx6j3lEIOsJBtQLYHvDWrtjFrLmo1JD4VeL421N4I8Re9v/ OJNarBaA2dUf3nXYAG0cOzCZqWkzjSNUDpcvTyLfT14Y4a7VbGghQuRaN7dFdSlBRhmF idwg== X-Gm-Message-State: ACrzQf1HjXKpuEB6whgrIUbCXOmvfrMSXmCeyqfKfXBeUJzgVomf75MM lCzXzyNfX+FKWSG3AJTfy/uH2A== X-Received: by 2002:a5d:6145:0:b0:236:a948:9e35 with SMTP id y5-20020a5d6145000000b00236a9489e35mr169891wrt.185.1666942615790; Fri, 28 Oct 2022 00:36:55 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:f:6020:c12b:b448:f0a9:83ef]) by smtp.gmail.com with ESMTPSA id j8-20020a05600c190800b003b47e8a5d22sm7783129wmq.23.2022.10.28.00.36.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Oct 2022 00:36:54 -0700 (PDT) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com Cc: qais.yousef@arm.com, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v6 4/9] sched: Allow sched_{get,set}attr to change latency_nice of the task Date: Fri, 28 Oct 2022 09:36:32 +0200 Message-Id: <20221028073637.31195-5-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221028073637.31195-1-vincent.guittot@linaro.org> References: <20221028073637.31195-1-vincent.guittot@linaro.org> X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747916332029929474?= X-GMAIL-MSGID: =?utf-8?q?1747916332029929474?= From: Parth Shah Introduce the latency_nice attribute to sched_attr and provide a mechanism to change the value with the use of sched_setattr/sched_getattr syscall. Also add new flag "SCHED_FLAG_LATENCY_NICE" to hint the change in latency_nice of the task on every sched_setattr syscall. Signed-off-by: Parth Shah [rebase and add a dedicated __setscheduler_latency ] Signed-off-by: Vincent Guittot --- include/uapi/linux/sched.h | 4 +++- include/uapi/linux/sched/types.h | 19 +++++++++++++++++++ kernel/sched/core.c | 24 ++++++++++++++++++++++++ tools/include/uapi/linux/sched.h | 4 +++- 4 files changed, 49 insertions(+), 2 deletions(-) diff --git a/include/uapi/linux/sched.h b/include/uapi/linux/sched.h index 3bac0a8ceab2..b2e932c25be6 100644 --- a/include/uapi/linux/sched.h +++ b/include/uapi/linux/sched.h @@ -132,6 +132,7 @@ struct clone_args { #define SCHED_FLAG_KEEP_PARAMS 0x10 #define SCHED_FLAG_UTIL_CLAMP_MIN 0x20 #define SCHED_FLAG_UTIL_CLAMP_MAX 0x40 +#define SCHED_FLAG_LATENCY_NICE 0x80 #define SCHED_FLAG_KEEP_ALL (SCHED_FLAG_KEEP_POLICY | \ SCHED_FLAG_KEEP_PARAMS) @@ -143,6 +144,7 @@ struct clone_args { SCHED_FLAG_RECLAIM | \ SCHED_FLAG_DL_OVERRUN | \ SCHED_FLAG_KEEP_ALL | \ - SCHED_FLAG_UTIL_CLAMP) + SCHED_FLAG_UTIL_CLAMP | \ + SCHED_FLAG_LATENCY_NICE) #endif /* _UAPI_LINUX_SCHED_H */ diff --git a/include/uapi/linux/sched/types.h b/include/uapi/linux/sched/types.h index f2c4589d4dbf..db1e8199e8c8 100644 --- a/include/uapi/linux/sched/types.h +++ b/include/uapi/linux/sched/types.h @@ -10,6 +10,7 @@ struct sched_param { #define SCHED_ATTR_SIZE_VER0 48 /* sizeof first published struct */ #define SCHED_ATTR_SIZE_VER1 56 /* add: util_{min,max} */ +#define SCHED_ATTR_SIZE_VER2 60 /* add: latency_nice */ /* * Extended scheduling parameters data structure. @@ -98,6 +99,22 @@ struct sched_param { * scheduled on a CPU with no more capacity than the specified value. * * A task utilization boundary can be reset by setting the attribute to -1. + * + * Latency Tolerance Attributes + * =========================== + * + * A subset of sched_attr attributes allows to specify the relative latency + * requirements of a task with respect to the other tasks running/queued in the + * system. + * + * @ sched_latency_nice task's latency_nice value + * + * The latency_nice of a task can have any value in a range of + * [MIN_LATENCY_NICE..MAX_LATENCY_NICE]. + * + * A task with latency_nice with the value of LATENCY_NICE_MIN can be + * taken for a task requiring a lower latency as opposed to the task with + * higher latency_nice. */ struct sched_attr { __u32 size; @@ -120,6 +137,8 @@ struct sched_attr { __u32 sched_util_min; __u32 sched_util_max; + /* latency requirement hints */ + __s32 sched_latency_nice; }; #endif /* _UAPI_LINUX_SCHED_TYPES_H */ diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 54544353025b..b2accc9da4fe 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -7318,6 +7318,14 @@ static void __setscheduler_params(struct task_struct *p, p->rt_priority = attr->sched_priority; p->normal_prio = normal_prio(p); set_load_weight(p, true); + +} + +static void __setscheduler_latency(struct task_struct *p, + const struct sched_attr *attr) +{ + if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) + p->latency_nice = attr->sched_latency_nice; } /* @@ -7460,6 +7468,13 @@ static int __sched_setscheduler(struct task_struct *p, return retval; } + if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) { + if (attr->sched_latency_nice > MAX_LATENCY_NICE) + return -EINVAL; + if (attr->sched_latency_nice < MIN_LATENCY_NICE) + return -EINVAL; + } + if (pi) cpuset_read_lock(); @@ -7494,6 +7509,9 @@ static int __sched_setscheduler(struct task_struct *p, goto change; if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP) goto change; + if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE && + attr->sched_latency_nice != p->latency_nice) + goto change; p->sched_reset_on_fork = reset_on_fork; retval = 0; @@ -7582,6 +7600,7 @@ static int __sched_setscheduler(struct task_struct *p, __setscheduler_params(p, attr); __setscheduler_prio(p, newprio); } + __setscheduler_latency(p, attr); __setscheduler_uclamp(p, attr); if (queued) { @@ -7792,6 +7811,9 @@ static int sched_copy_attr(struct sched_attr __user *uattr, struct sched_attr *a size < SCHED_ATTR_SIZE_VER1) return -EINVAL; + if ((attr->sched_flags & SCHED_FLAG_LATENCY_NICE) && + size < SCHED_ATTR_SIZE_VER2) + return -EINVAL; /* * XXX: Do we want to be lenient like existing syscalls; or do we want * to be strict and return an error on out-of-bounds values? @@ -8029,6 +8051,8 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sched_attr __user *, uattr, get_params(p, &kattr); kattr.sched_flags &= SCHED_FLAG_ALL; + kattr.sched_latency_nice = p->latency_nice; + #ifdef CONFIG_UCLAMP_TASK /* * This could race with another potential updater, but this is fine diff --git a/tools/include/uapi/linux/sched.h b/tools/include/uapi/linux/sched.h index 3bac0a8ceab2..ecc4884bfe4b 100644 --- a/tools/include/uapi/linux/sched.h +++ b/tools/include/uapi/linux/sched.h @@ -132,6 +132,7 @@ struct clone_args { #define SCHED_FLAG_KEEP_PARAMS 0x10 #define SCHED_FLAG_UTIL_CLAMP_MIN 0x20 #define SCHED_FLAG_UTIL_CLAMP_MAX 0x40 +#define SCHED_FLAG_LATENCY_NICE 0x80 #define SCHED_FLAG_KEEP_ALL (SCHED_FLAG_KEEP_POLICY | \ SCHED_FLAG_KEEP_PARAMS) @@ -143,6 +144,7 @@ struct clone_args { SCHED_FLAG_RECLAIM | \ SCHED_FLAG_DL_OVERRUN | \ SCHED_FLAG_KEEP_ALL | \ - SCHED_FLAG_UTIL_CLAMP) + SCHED_FLAG_UTIL_CLAMP | \ + SCHED_FLAG_LATENCY_NICE) #endif /* _UAPI_LINUX_SCHED_H */ From patchwork Fri Oct 28 07:36:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 12139 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp679527wru; Fri, 28 Oct 2022 00:42:32 -0700 (PDT) X-Google-Smtp-Source: AMsMyM741PcIUia8EKh21RjZU+dnO6gGFSxSx+1VSepHgquVVC4k/g/FzbsnQguARI9qPnjoSlWd X-Received: by 2002:aa7:d506:0:b0:461:565e:8673 with SMTP id y6-20020aa7d506000000b00461565e8673mr34070332edq.416.1666942952771; Fri, 28 Oct 2022 00:42:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666942952; cv=none; d=google.com; s=arc-20160816; b=kXszY1dHi5Lk/xSEO4SNupoFRuGwId/GvQICjck31EFowyBw7M4oQflnrbdx8DUqIe VIpIWmV/07jCYo+17UQ0fNbMO4U4/pUzc8eb0vMhbTzZMciEh/zxbTJ/Lpk+uL6gvJec oqUe3et23G5mcTRsB5j/Pqzov5JbnX587yk1e38T+WBC8IyS6+39049AwSHuP1K1LGGG vyzG0xlC990NtHeygnoopSFzfpQZMtLIgBrCtneDbCfyjt677jDhuQIstyI0Aw0e1YX5 xfQExjUkwmJz2OZ+S5D85TVQrlfgDkp+PiEL9Iwx9CYNFOWKtGGgSlPFqi+p5DXViLyH cTAg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=4kgrsJEqYaa/F19eJ53q5XCqUaTM5noPZmGTo2LpZn4=; b=f7Xzp9sw1cVzAMl9PewGLhaZaeLzJAto3lUaEox3wu6a1DI/8CoFuqYhMA0k7PO4XB NcifnEYbkMTF/chRnWNEcUb4oqX0ba6ZP1U+WT7fg22uF7y6f0Cf5cWtzYNLRog9gylV Rx+AJB7I5kR/eVksRSjzvK3OtSb9Yz9VrWoPxO/5oCmdiiZ8C4ihJcybaJsX1xdymXam tILxdpW3kl3BRrWgrPKDg5vYlL7ltAulalaw67nD2N1P5pipQPaBQA3hyqfN/C2ej+7n aeS+0G5BIO6yn75Mt0HF1Yr5qZE+8jrSfwuHyPwoD0WxItoY7gmmXTBMW7JiEY5FBVar 7QwQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="qUhaL/1B"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id hv17-20020a17090760d100b0078c5d28e27esi1560026ejc.604.2022.10.28.00.42.08; Fri, 28 Oct 2022 00:42:32 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="qUhaL/1B"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230233AbiJ1HhR (ORCPT + 99 others); Fri, 28 Oct 2022 03:37:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42932 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230224AbiJ1HhF (ORCPT ); Fri, 28 Oct 2022 03:37:05 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B55ABC96DE for ; Fri, 28 Oct 2022 00:36:59 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id y16so5810310wrt.12 for ; Fri, 28 Oct 2022 00:36:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=4kgrsJEqYaa/F19eJ53q5XCqUaTM5noPZmGTo2LpZn4=; b=qUhaL/1BjAG9E/aX/qxYLgPWXz1+UlNNOkpSYM0P8w8s+xlfbdFY7y1lhSj/lXlpv5 Ngvki6GcQDiU7NKf24Kq1FQUJUGP9/+EHyFsAZlYwBNKPOoTF6DIVwMXDtKZ8wFSNkMb IPusg/6BielrC0OGbPVAJOAZ7z29UfKwZzi59mQMmtSerma3Io4l+7eSLEIkH1Ag9scl YpZfa0XVyhDXX5j6SYjTqfRq0zGMwIUZ8STs8MaMXE7LMJrHmdhsCNWoqncr5sIWMycj gPYiemNQ0+5zw5ko2lx5LyfODMPKKVvI6kFDBv4JTjpmez6iCH3SO70I6CS8nKz6ZMUw BsXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4kgrsJEqYaa/F19eJ53q5XCqUaTM5noPZmGTo2LpZn4=; b=60PkYov9jsMdC6q5mwQWmMD0qi6dYo0c2U/PlNcwKnFpNWw9hwbhfL3/VpruI42iuX f1uesrF2rHlZDfCWBtbXJS6ewELsgU++AP9ZVBjHU0dlrmp4EftETOxqSb7w2lNESlVL zEO0f0sioK5FPfH0hqkdVXUcogMwvdqljEurG6svR33WLrN4xhXOyNCIJtULDHoEoWEm VFaSmsVE+OmbC7eoLJCrOCvijE4WVjIb19yX8tEyjaPb0UMkoKSvIGBerUv9GwaEnTUs +Tn9swQMh/zP8kevilCJbVnnFIwDQ+g67vBUuTK7Fvx/DY3paQQXb6oSYRHz+T8N+Z07 Vtog== X-Gm-Message-State: ACrzQf0JQ0iKntexIMykwtsOAdJsji6bmh2awOiCBVg1ZVCuyuEKAq9w cdH8o46PCw3KtacQf2+RNzeCYw== X-Received: by 2002:adf:efc5:0:b0:236:6848:7c28 with SMTP id i5-20020adfefc5000000b0023668487c28mr18305278wrp.263.1666942618115; Fri, 28 Oct 2022 00:36:58 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:f:6020:c12b:b448:f0a9:83ef]) by smtp.gmail.com with ESMTPSA id j8-20020a05600c190800b003b47e8a5d22sm7783129wmq.23.2022.10.28.00.36.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Oct 2022 00:36:56 -0700 (PDT) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com Cc: qais.yousef@arm.com, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v6 5/9] sched/fair: Take into account latency priority at wakeup Date: Fri, 28 Oct 2022 09:36:33 +0200 Message-Id: <20221028073637.31195-6-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221028073637.31195-1-vincent.guittot@linaro.org> References: <20221028073637.31195-1-vincent.guittot@linaro.org> X-Spam-Status: No, score=-1.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,PDS_BTC_ID,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747916373833295676?= X-GMAIL-MSGID: =?utf-8?q?1747916373833295676?= Take into account the latency priority of a thread when deciding to preempt the current running thread. We don't want to provide more CPU bandwidth to a thread but reorder the scheduling to run latency sensitive task first whenever possible. As long as a thread didn't use its bandwidth, it will be able to preempt the current thread. At the opposite, a thread with a low latency priority will preempt current thread at wakeup only to keep fair CPU bandwidth sharing. Otherwise it will wait for the tick to get its sched slice. curr vruntime | sysctl_sched_wakeup_granularity <--> ----------------------------------|----|-----------------------|--------------- | |<---------------------> | . sysctl_sched_latency | . default/current latency entity | . | . 1111111111111111111111111111111111|0000|-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1- se preempts curr at wakeup ------>|<- se doesn't preempt curr ----------------- | . | . | . low latency entity | . ---------------------->| % of sysctl_sched_latency | 1111111111111111111111111111111111111111111111111111111111|0000|-1-1-1-1-1-1-1- preempt ------------------------------------------------->|<- do not preempt -- | . | . | . high latency entity | . |<-----------------------|----. | % of sysctl_sched_latency . 111111111|0000|-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1 preempt->|<- se doesn't preempt curr ------------------------------------------ Tests results of nice latency impact on heavy load like hackbench: hackbench -l (2560 / group) -g group group latency 0 latency 19 1 1.378(+/- 1%) 1.337(+/- 1%) + 3% 4 1.393(+/- 3%) 1.312(+/- 3%) + 6% 8 1.308(+/- 2%) 1.279(+/- 1%) + 2% 16 1.347(+/- 1%) 1.317(+/- 1%) + 2% hackbench -p -l (2560 / group) -g group group 1 1.836(+/- 17%) 1.148(+/- 5%) +37% 4 1.586(+/- 6%) 1.109(+/- 8%) +30% 8 1.209(+/- 4%) 0.780(+/- 4%) +35% 16 0.805(+/- 5%) 0.728(+/- 4%) +10% By deacreasing the latency prio, we reduce the number of preemption at wakeup and help hackbench making progress. Test results of nice latency impact on short live load like cyclictest while competing with heavy load like hackbench: hackbench -l 10000 -g $group & cyclictest --policy other -D 5 -q -n latency 0 latency -20 group min avg max min avg max 0 16 19 29 17 18 29 1 43 299 7359 63 84 3422 4 56 449 14806 45 83 284 8 63 820 51123 63 83 283 16 64 1326 70684 41 157 26852 group = 0 means that hackbench is not running. The avg is significantly improved with nice latency -20 especially with large number of groups but min and max remain quite similar. If we add the histogram parameter to get details of latency, we have : hackbench -l 10000 -g 16 & cyclictest --policy other -D 5 -q -n -H 20000 --histfile data.txt latency 0 latency -20 Min Latencies: 64 62 Avg Latencies: 1170 107 Max Latencies: 88069 10417 50% latencies: 122 86 75% latencies: 614 91 85% latencies: 961 94 90% latencies: 1225 97 95% latencies: 6120 102 99% latencies: 18328 159 With percentile details, we see the benefit of nice latency -20 as only 1% of the latencies are above 159us whereas the default latency has got 15% around ~1ms or above and 5% over the 6ms. Signed-off-by: Vincent Guittot --- The figures above include the whole patchset. include/linux/sched.h | 4 ++- init/init_task.c | 2 +- kernel/sched/core.c | 38 +++++++++++++++++++++---- kernel/sched/debug.c | 2 +- kernel/sched/fair.c | 66 ++++++++++++++++++++++++++++++++++++++----- kernel/sched/sched.h | 14 ++++++++- 6 files changed, 110 insertions(+), 16 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 6805f378a9c3..a74cad08e91e 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -567,6 +567,8 @@ struct sched_entity { /* cached value of my_q->h_nr_running */ unsigned long runnable_weight; #endif + /* preemption offset in ns */ + long latency_offset; #ifdef CONFIG_SMP /* @@ -783,7 +785,7 @@ struct task_struct { int static_prio; int normal_prio; unsigned int rt_priority; - int latency_nice; + int latency_prio; struct sched_entity se; struct sched_rt_entity rt; diff --git a/init/init_task.c b/init/init_task.c index 7dd71dd2d261..b8ddf403bc62 100644 --- a/init/init_task.c +++ b/init/init_task.c @@ -78,7 +78,7 @@ struct task_struct init_task .prio = MAX_PRIO - 20, .static_prio = MAX_PRIO - 20, .normal_prio = MAX_PRIO - 20, - .latency_nice = DEFAULT_LATENCY_NICE, + .latency_prio = NICE_WIDTH - 20, .policy = SCHED_NORMAL, .cpus_ptr = &init_task.cpus_mask, .user_cpus_ptr = NULL, diff --git a/kernel/sched/core.c b/kernel/sched/core.c index b2accc9da4fe..caf54e54a74f 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -1284,6 +1284,16 @@ static void set_load_weight(struct task_struct *p, bool update_load) } } +static void set_latency_offset(struct task_struct *p) +{ + long weight = sched_latency_to_weight[p->latency_prio]; + s64 offset; + + offset = weight * get_sched_latency(false); + offset = div_s64(offset, NICE_LATENCY_WEIGHT_MAX); + p->se.latency_offset = (long)offset; +} + #ifdef CONFIG_UCLAMP_TASK /* * Serializes updates of utilization clamp values @@ -4559,7 +4569,9 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p) p->prio = p->normal_prio = p->static_prio; set_load_weight(p, false); - p->latency_nice = DEFAULT_LATENCY_NICE; + p->latency_prio = NICE_TO_LATENCY(0); + set_latency_offset(p); + /* * We don't need the reset flag anymore after the fork. It has * fulfilled its duty: @@ -7324,8 +7336,10 @@ static void __setscheduler_params(struct task_struct *p, static void __setscheduler_latency(struct task_struct *p, const struct sched_attr *attr) { - if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) - p->latency_nice = attr->sched_latency_nice; + if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) { + p->latency_prio = NICE_TO_LATENCY(attr->sched_latency_nice); + set_latency_offset(p); + } } /* @@ -7510,7 +7524,7 @@ static int __sched_setscheduler(struct task_struct *p, if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP) goto change; if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE && - attr->sched_latency_nice != p->latency_nice) + attr->sched_latency_nice != LATENCY_TO_NICE(p->latency_prio)) goto change; p->sched_reset_on_fork = reset_on_fork; @@ -8051,7 +8065,7 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sched_attr __user *, uattr, get_params(p, &kattr); kattr.sched_flags &= SCHED_FLAG_ALL; - kattr.sched_latency_nice = p->latency_nice; + kattr.sched_latency_nice = LATENCY_TO_NICE(p->latency_prio); #ifdef CONFIG_UCLAMP_TASK /* @@ -11204,6 +11218,20 @@ const u32 sched_prio_to_wmult[40] = { /* 15 */ 119304647, 148102320, 186737708, 238609294, 286331153, }; +/* + * latency weight for wakeup preemption + */ +const int sched_latency_to_weight[40] = { + /* -20 */ -1024, -973, -922, -870, -819, + /* -15 */ -768, -717, -666, -614, -563, + /* -10 */ -512, -461, -410, -358, -307, + /* -5 */ -256, -205, -154, -102, -51, + /* 0 */ 0, 51, 102, 154, 205, + /* 5 */ 256, 307, 358, 410, 461, + /* 10 */ 512, 563, 614, 666, 717, + /* 15 */ 768, 819, 870, 922, 973, +}; + void call_trace_sched_update_nr_running(struct rq *rq, int count) { trace_sched_update_nr_running_tp(rq, count); diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c index a3f7876217a6..06aaa0c81d4b 100644 --- a/kernel/sched/debug.c +++ b/kernel/sched/debug.c @@ -1042,7 +1042,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns, #endif P(policy); P(prio); - P(latency_nice); + P(latency_prio); if (task_has_dl_policy(p)) { P(dl.runtime); P(dl.deadline); diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index eb04c83112a0..4299d5108dc7 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4558,6 +4558,8 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) update_idle_cfs_rq_clock_pelt(cfs_rq); } +static long wakeup_latency_gran(struct sched_entity *curr, struct sched_entity *se); + /* * Preempt the current task with a newly woken task if needed: */ @@ -4566,7 +4568,7 @@ check_preempt_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr) { unsigned long ideal_runtime, delta_exec; struct sched_entity *se; - s64 delta; + s64 delta, offset; ideal_runtime = sched_slice(cfs_rq, curr); delta_exec = curr->sum_exec_runtime - curr->prev_sum_exec_runtime; @@ -4591,10 +4593,12 @@ check_preempt_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr) se = __pick_first_entity(cfs_rq); delta = curr->vruntime - se->vruntime; - if (delta < 0) + offset = wakeup_latency_gran(curr, se); + if (delta < offset) return; - if (delta > ideal_runtime) + if ((delta > ideal_runtime) || + (delta > get_latency_max())) resched_curr(rq_of(cfs_rq)); } @@ -5716,6 +5720,35 @@ static int sched_idle_cpu(int cpu) } #endif +static void set_next_buddy(struct sched_entity *se); + +static void check_preempt_from_others(struct cfs_rq *cfs, struct sched_entity *se) +{ + struct sched_entity *next; + + if (se->latency_offset >= 0) + return; + + if (cfs->nr_running <= 1) + return; + /* + * When waking from another class, we don't need to check to preempt at + * wakeup and don't set next buddy as a candidate for being picked in + * priority. + * In case of simultaneous wakeup when current is another class, the + * latency sensitive tasks lost opportunity to preempt non sensitive + * tasks which woke up simultaneously. + */ + + if (cfs->next) + next = cfs->next; + else + next = __pick_first_entity(cfs); + + if (next && wakeup_preempt_entity(next, se) == 1) + set_next_buddy(se); +} + /* * The enqueue_task method is called before nr_running is * increased. Here we update the fair scheduling stats and @@ -5802,14 +5835,15 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) if (!task_new) update_overutilized_status(rq); + if (rq->curr->sched_class != &fair_sched_class) + check_preempt_from_others(cfs_rq_of(&p->se), &p->se); + enqueue_throttle: assert_list_leaf_cfs_rq(rq); hrtick_update(rq); } -static void set_next_buddy(struct sched_entity *se); - /* * The dequeue_task method is called before nr_running is * decreased. We remove the task from the rbtree and @@ -7128,6 +7162,23 @@ balance_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) } #endif /* CONFIG_SMP */ +static long wakeup_latency_gran(struct sched_entity *curr, struct sched_entity *se) +{ + long latency_offset = se->latency_offset; + + /* + * A negative latency offset means that the sched_entity has latency + * requirement that needs to be evaluated versus other entity. + * Otherwise, use the latency weight to evaluate how much scheduling + * delay is acceptable by se. + */ + if ((latency_offset < 0) || (curr->latency_offset < 0)) + latency_offset -= curr->latency_offset; + latency_offset = min_t(long, latency_offset, get_latency_max()); + + return latency_offset; +} + static unsigned long wakeup_gran(struct sched_entity *se) { unsigned long gran = sysctl_sched_wakeup_granularity; @@ -7166,11 +7217,12 @@ static int wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se) { s64 gran, vdiff = curr->vruntime - se->vruntime; + s64 offset = wakeup_latency_gran(curr, se); - if (vdiff <= 0) + if (vdiff < offset) return -1; - gran = wakeup_gran(se); + gran = offset + wakeup_gran(se); /* * At wake up, the vruntime of a task is capped to not be older than diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 5445a894babd..cca09ce8d0d6 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -142,6 +142,17 @@ extern int sched_rr_timeslice; * Default tasks should be treated as a task with latency_nice = 0. */ #define DEFAULT_LATENCY_NICE 0 +#define DEFAULT_LATENCY_PRIO (DEFAULT_LATENCY_NICE + LATENCY_NICE_WIDTH/2) + +/* + * Convert user-nice values [ -20 ... 0 ... 19 ] + * to static latency [ 0..39 ], + * and back. + */ +#define NICE_TO_LATENCY(nice) ((nice) + DEFAULT_LATENCY_PRIO) +#define LATENCY_TO_NICE(prio) ((prio) - DEFAULT_LATENCY_PRIO) +#define NICE_LATENCY_SHIFT (SCHED_FIXEDPOINT_SHIFT) +#define NICE_LATENCY_WEIGHT_MAX (1L << NICE_LATENCY_SHIFT) /* * Increase resolution of nice-level calculations for 64-bit architectures. @@ -2116,6 +2127,7 @@ static_assert(WF_TTWU == SD_BALANCE_WAKE); extern const int sched_prio_to_weight[40]; extern const u32 sched_prio_to_wmult[40]; +extern const int sched_latency_to_weight[40]; /* * {de,en}queue flags: @@ -2450,8 +2462,8 @@ extern void check_preempt_curr(struct rq *rq, struct task_struct *p, int flags); extern const_debug unsigned int sysctl_sched_nr_migrate; extern const_debug unsigned int sysctl_sched_migration_cost; -#ifdef CONFIG_SCHED_DEBUG extern unsigned int sysctl_sched_latency; +#ifdef CONFIG_SCHED_DEBUG extern unsigned int sysctl_sched_min_granularity; extern unsigned int sysctl_sched_idle_min_granularity; extern unsigned int sysctl_sched_wakeup_granularity; From patchwork Fri Oct 28 07:36:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 12138 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp679526wru; Fri, 28 Oct 2022 00:42:32 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6CrESmcIYAf33Zgo0FYq1Tiz8LW1UGwej2n8Fq3vdtXEg2oFaCfuHBZrsX9NX82qnUW1e8 X-Received: by 2002:a17:907:7b86:b0:78d:cb21:da0c with SMTP id ne6-20020a1709077b8600b0078dcb21da0cmr47409886ejc.24.1666942952610; Fri, 28 Oct 2022 00:42:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666942952; cv=none; d=google.com; s=arc-20160816; b=qOlwlAAmj1K2j7j1QY8/P6jAoPR3auCuvYbvtJVF+0f5On7oeWM+h6u3XMi8D2ls/0 C4Ae2vgbx/XsS+e/txsMFxqdvDAgnMrKE5cAZM1creJxD+tAmVBSAxZ1pgkSrSOln/8m lSpqfxMjqmAoE9AyLFrwsO65QYtZcYd3n7NhaTBkThKlrmfSe+fKCwZ/5J868pug/Wxp pfMLDWsLdnBCxcrc/a3G/zwfiw3WdOnR8VC9j5DN2UV5dYu0XpzTxzXJFoIV31UoawGW pWrlDTgQBVfxR++B6aDly/iL/Tv3x/RehO0676YNho2xFwj1o26KpGnvca8/fRlEXVLJ yJmA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=t3r9zn7TCuvGykxEk/g32GjM6BAQKOwVz31FtxjXSRY=; b=PLAaiXQoC34aouXLAKfK/km3CiYCOz7+c7qmQ6g3HM8KJPg30PVwNExw99bHd4evcP 75BmAsLKsTxx7Cuo0Zgh24qXRvUPO9q0C94RcI4pBz6Oeizof0dNEyabvNA6YUXwJ2or h1YA5P5eJf/ZQi3jk3MaGEG+415z5K7te+xEMAaNEqHE4yXQZw7qNr3zt8ADBAdcQ8Qm WgXP2qNlb9X4Af9ZU39MVpcdWVdVf1Qr6A1dihtgxkII+ZYXfBAbIe2McJmhgOsHzYlc fnDmXg7IcF+/BcYPyz3c2byFk/wrFuQbtAeZ1KjLTPnv93mUR0ysFp7m+26Ca1ic4GG/ tR9g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=oUxnh8+s; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id du17-20020a17090772d100b0077b83f3dc20si4102924ejc.63.2022.10.28.00.42.09; Fri, 28 Oct 2022 00:42:32 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=oUxnh8+s; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230211AbiJ1HhU (ORCPT + 99 others); Fri, 28 Oct 2022 03:37:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43092 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230187AbiJ1HhF (ORCPT ); Fri, 28 Oct 2022 03:37:05 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CE3AD1BC148 for ; Fri, 28 Oct 2022 00:37:01 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id bp11so5826970wrb.9 for ; Fri, 28 Oct 2022 00:37:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=t3r9zn7TCuvGykxEk/g32GjM6BAQKOwVz31FtxjXSRY=; b=oUxnh8+sy2cgAkT3Y6CJ0Q1GBMMkUDOp19hE+ohIJsgSG3AiN7i8Vydo4AzMexODgN kupn/n/FxTPhN64X4dqYr5wTx8e5xA0Zl3Uil946yAp9cED7a9UN+D5Zkx/dPD9AIMUU JXvWBT08CP3mQYrkqwOZuLCjBlYM2zGcqWuvwQsGbd/+A6F9nCdKxDn0vZQnSelWY29z jIBhT8IAxcQ8+zT0rPnBQge2kJ070WRnKjfiakNz8zo24qNUJNCJly85bVmaBEZtQyS4 WePJ1zQU/LELR3qXFGzRZorE7t0uCF1qcQgJpnpBks8fe8/u/FtKml4NG/T1smjHUBiC GNCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=t3r9zn7TCuvGykxEk/g32GjM6BAQKOwVz31FtxjXSRY=; b=ho/ZMa8hN6QQRj8Wmau42y582oa2JUcN3KzwzKW2RESBpFRqOiF7yxj+Gpy+DGZTCz 4qV5tffhN30DbtoRVpz7ciYHfie5AJ96WlkqqLbxFJMxwm2iaDcq1g3tMyN8zMX5MN92 /3sRwBP7/9DeXTkJOIXzC7LkFbkS3r+TaMYEW3ffhjJaFUrY/Bpq1mVSpwlK7Ap3xZfh kGNhaesrDz90rHurs6IKM85ohkWEiQw4I+PiLxMpMwFWPfu+t/ZcoGsPh/s0iQyEjgZn vh1HCKmeG24IRqiGWdzH2lcJzkeJXU+eNTizehVTbvR/99wFDunmMkr638jkWoBOfWyE eMkg== X-Gm-Message-State: ACrzQf136kFI8El5m2Ii+a0GCOsq5k9uZUX23ug9+jZuRn2iA4Wkejyj VrS7Gonrx9Ub5gOoec/CNL3IWA== X-Received: by 2002:a05:6000:1d94:b0:22c:c4d1:3622 with SMTP id bk20-20020a0560001d9400b0022cc4d13622mr33678684wrb.63.1666942620323; Fri, 28 Oct 2022 00:37:00 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:f:6020:c12b:b448:f0a9:83ef]) by smtp.gmail.com with ESMTPSA id j8-20020a05600c190800b003b47e8a5d22sm7783129wmq.23.2022.10.28.00.36.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Oct 2022 00:36:59 -0700 (PDT) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com Cc: qais.yousef@arm.com, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v6 6/9] sched/fair: Add sched group latency support Date: Fri, 28 Oct 2022 09:36:34 +0200 Message-Id: <20221028073637.31195-7-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221028073637.31195-1-vincent.guittot@linaro.org> References: <20221028073637.31195-1-vincent.guittot@linaro.org> X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747916373431723807?= X-GMAIL-MSGID: =?utf-8?q?1747916373431723807?= Task can set its latency priority with sched_setattr(), which is then used to set the latency offset of its sched_enity, but sched group entities still have the default latency offset value. Add a latency.nice field in cpu cgroup controller to set the latency priority of the group similarly to sched_setattr(). The latency priority is then used to set the offset of the sched_entities of the group. Signed-off-by: Vincent Guittot --- Documentation/admin-guide/cgroup-v2.rst | 8 ++++ kernel/sched/core.c | 52 +++++++++++++++++++++++++ kernel/sched/fair.c | 33 ++++++++++++++++ kernel/sched/sched.h | 4 ++ 4 files changed, 97 insertions(+) diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst index be4a77baf784..d8ae7e411f9c 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -1095,6 +1095,14 @@ All time durations are in microseconds. values similar to the sched_setattr(2). This maximum utilization value is used to clamp the task specific maximum utilization clamp. + cpu.latency.nice + A read-write single value file which exists on non-root + cgroups. The default is "0". + + The nice value is in the range [-20, 19]. + + This interface file allows reading and setting latency using the + same values used by sched_setattr(2). Memory diff --git a/kernel/sched/core.c b/kernel/sched/core.c index caf54e54a74f..3f42b1f61a7e 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -10890,6 +10890,47 @@ static int cpu_idle_write_s64(struct cgroup_subsys_state *css, { return sched_group_set_idle(css_tg(css), idle); } + +static s64 cpu_latency_nice_read_s64(struct cgroup_subsys_state *css, + struct cftype *cft) +{ + int prio, delta, last_delta = INT_MAX; + s64 weight; + + weight = css_tg(css)->latency_offset * NICE_LATENCY_WEIGHT_MAX; + weight = div_s64(weight, get_sched_latency(false)); + + /* Find the closest nice value to the current weight */ + for (prio = 0; prio < ARRAY_SIZE(sched_latency_to_weight); prio++) { + delta = abs(sched_latency_to_weight[prio] - weight); + if (delta >= last_delta) + break; + last_delta = delta; + } + + return LATENCY_TO_NICE(prio-1); +} + +static int cpu_latency_nice_write_s64(struct cgroup_subsys_state *css, + struct cftype *cft, s64 nice) +{ + s64 latency_offset; + long weight; + int idx; + + if (nice < MIN_LATENCY_NICE || nice > MAX_LATENCY_NICE) + return -ERANGE; + + idx = NICE_TO_LATENCY(nice); + idx = array_index_nospec(idx, LATENCY_NICE_WIDTH); + weight = sched_latency_to_weight[idx]; + + latency_offset = weight * get_sched_latency(false); + latency_offset = div_s64(latency_offset, NICE_LATENCY_WEIGHT_MAX); + + return sched_group_set_latency(css_tg(css), latency_offset); +} + #endif static struct cftype cpu_legacy_files[] = { @@ -10904,6 +10945,11 @@ static struct cftype cpu_legacy_files[] = { .read_s64 = cpu_idle_read_s64, .write_s64 = cpu_idle_write_s64, }, + { + .name = "latency.nice", + .read_s64 = cpu_latency_nice_read_s64, + .write_s64 = cpu_latency_nice_write_s64, + }, #endif #ifdef CONFIG_CFS_BANDWIDTH { @@ -11121,6 +11167,12 @@ static struct cftype cpu_files[] = { .read_s64 = cpu_idle_read_s64, .write_s64 = cpu_idle_write_s64, }, + { + .name = "latency.nice", + .flags = CFTYPE_NOT_ON_ROOT, + .read_s64 = cpu_latency_nice_read_s64, + .write_s64 = cpu_latency_nice_write_s64, + }, #endif #ifdef CONFIG_CFS_BANDWIDTH { diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 4299d5108dc7..9583936ce30c 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -11764,6 +11764,7 @@ int alloc_fair_sched_group(struct task_group *tg, struct task_group *parent) goto err; tg->shares = NICE_0_LOAD; + tg->latency_offset = 0; init_cfs_bandwidth(tg_cfs_bandwidth(tg)); @@ -11862,6 +11863,9 @@ void init_tg_cfs_entry(struct task_group *tg, struct cfs_rq *cfs_rq, } se->my_q = cfs_rq; + + se->latency_offset = tg->latency_offset; + /* guarantee group entities always have weight */ update_load_set(&se->load, NICE_0_LOAD); se->parent = parent; @@ -11992,6 +11996,35 @@ int sched_group_set_idle(struct task_group *tg, long idle) return 0; } +int sched_group_set_latency(struct task_group *tg, s64 latency) +{ + int i; + + if (tg == &root_task_group) + return -EINVAL; + + if (abs(latency) > sysctl_sched_latency) + return -EINVAL; + + mutex_lock(&shares_mutex); + + if (tg->latency_offset == latency) { + mutex_unlock(&shares_mutex); + return 0; + } + + tg->latency_offset = latency; + + for_each_possible_cpu(i) { + struct sched_entity *se = tg->se[i]; + + WRITE_ONCE(se->latency_offset, latency); + } + + mutex_unlock(&shares_mutex); + return 0; +} + #else /* CONFIG_FAIR_GROUP_SCHED */ void free_fair_sched_group(struct task_group *tg) { } diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index cca09ce8d0d6..b22113c8939b 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -407,6 +407,8 @@ struct task_group { /* A positive value indicates that this is a SCHED_IDLE group. */ int idle; + /* latency constraint of the group. */ + int latency_offset; #ifdef CONFIG_SMP /* @@ -517,6 +519,8 @@ extern int sched_group_set_shares(struct task_group *tg, unsigned long shares); extern int sched_group_set_idle(struct task_group *tg, long idle); +extern int sched_group_set_latency(struct task_group *tg, s64 latency); + #ifdef CONFIG_SMP extern void set_task_rq_fair(struct sched_entity *se, struct cfs_rq *prev, struct cfs_rq *next); From patchwork Fri Oct 28 07:36:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 12140 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp679796wru; Fri, 28 Oct 2022 00:43:31 -0700 (PDT) X-Google-Smtp-Source: AMsMyM7IJtsiD+flHdwsZ1oVINb/CAkGdBV/ycm/+/6JWxjPMFfBqCAIHA0sGrQslS1i3bATAxh0 X-Received: by 2002:a17:907:a0c6:b0:7ad:a030:7499 with SMTP id hw6-20020a170907a0c600b007ada0307499mr1228719ejc.552.1666943010965; Fri, 28 Oct 2022 00:43:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666943010; cv=none; d=google.com; s=arc-20160816; b=yVptwPbgnhCqqwRGmXfCDXBlWbK38wHZ8utA3KSfNOy2TX6/5Mv6MinOnMXifrhcL+ 2Sr6f+7ALWdosYPoXxbDWhA10VQ84YhhlYk6kjNcUEPbL6KmCHk8N1BsJaDuvmMoz2AQ Ir7sG3t/JRt4Xnm8LJFTRgO89puvlP9Vo9mUUgcCX1XAUrRyl6qEt0QLVcN3pdXgjT1S Z7VD4mQrkyyoXNundRqvOlA08V1X3IcaXfChyKQBTVQfreIL2S7rXpdFOqALs3R2RCmZ V9VoB/Tf78I1/zO7gu8PUeq7xP+iB3jNwG/MVTfHFVu3P7A4OxYmXWpaTFFtPMzwStVa cH/w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=BdpGOqqRR51x8v+WZgI/bsaGNfHn7spXAbo4XhwMNf4=; b=pqpvE2DP/VeiXL5IWLIBOWWQGRR5lKfmSqlHsTdqivGp8pQNssYMb+OBtw9bUVzEUo /HC2oFjudLH5gTvcC7XPt2LNcSRlONHvfOMH6kvdeXVpWK3lEuDWudLSglx6sGk7j78S I/ETQZ3DxwBn5jTGx7vGgtxGju+1wqos5LbeLWj3VW0o1rgcYni800wDPj1D6fNgAmtX tIwzgk8foM/+uMJd26A1V+Lbt91eUCRvrM7iRXTHnlhwixMQxt04VkAX+67+v0iauK8c hM4Scj9agd7YdwGTw7ONiuKmcm0gRfrI7uQiMPXcts742x1nxEh5+K9RuG9wDQsc12Wu OaHw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="qe09/nTv"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f25-20020a50d559000000b00461ac01753fsi3811970edj.369.2022.10.28.00.43.07; Fri, 28 Oct 2022 00:43:30 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="qe09/nTv"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229473AbiJ1Hh0 (ORCPT + 99 others); Fri, 28 Oct 2022 03:37:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43200 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230232AbiJ1HhF (ORCPT ); Fri, 28 Oct 2022 03:37:05 -0400 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 09C0D1BC157 for ; Fri, 28 Oct 2022 00:37:03 -0700 (PDT) Received: by mail-wm1-x32c.google.com with SMTP id l16-20020a05600c4f1000b003c6c0d2a445so2909270wmq.4 for ; Fri, 28 Oct 2022 00:37:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=BdpGOqqRR51x8v+WZgI/bsaGNfHn7spXAbo4XhwMNf4=; b=qe09/nTvE0G/K4zkvg/DsQq8oHH7SjmRLoGHceZha1Qm5MqWolIk7mFG0thT+nMLKF KmysZi86J4U/5hXr25vRhPcDCIKGkkN4zND+Y6vzOHSZIiS616i1X/GWi7nrmvq4ie7m lLtii7gv4j4T1PY82iCgfwruhI3BLqBkJDpGb11OxRWiYl3aCY0s/q0KJRN/tjm6C5gN 3voZULnWa+hrVgtCi0+hNUcAckmglcRPmebAzm21WrSsNcHebeMwX3ob8HJ8XM7ubRLs DYzs111nItJbuSSoUYYd+orcq/2ILdjTkCxJlJ4xOlJan/kuxb/YNFDB6+OI2htyxuDZ 1xXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BdpGOqqRR51x8v+WZgI/bsaGNfHn7spXAbo4XhwMNf4=; b=7HEOvTEgpDQy4QZVWLSUyvaAbAjt7i2l39M5nLbnHFHuvVkm8LvZlrLaL6OS3q956b 1RKPYrDsDPbYA8EFr8+6cvgRECPot0vA2eBER02MKlp3ZGedQmcZGwjJCynlT8y06sni T0h5CiSoRRzIUFfVeidn5Kb+mql/11NM4wblT6KzzUgxhMg/KraENseCJ/R6tb/hCeMQ +XXW7ztGBEbHbMDkRXnRpaq+B5BYEPQtf/KXqN1CzTsaFxk2TDU0vToBl7k4ooILZV12 TJMHTndd5chHbzIeoSO7D3Y1Mo+cZ0AH5PWQu78HRhEFmohUXVCKhUGY5v9nc+XUpxsT HS2w== X-Gm-Message-State: ACrzQf2LndxuOztrLoN+FQMrQpseIBaWLrqc7tzEyalpVgqox0cY2psj z0m5m9SD3Pe05bw/v8qIfmJVchcDNcujlg== X-Received: by 2002:a7b:c341:0:b0:3c4:552d:2ea7 with SMTP id l1-20020a7bc341000000b003c4552d2ea7mr8571521wmj.82.1666942622585; Fri, 28 Oct 2022 00:37:02 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:f:6020:c12b:b448:f0a9:83ef]) by smtp.gmail.com with ESMTPSA id j8-20020a05600c190800b003b47e8a5d22sm7783129wmq.23.2022.10.28.00.37.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Oct 2022 00:37:01 -0700 (PDT) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com Cc: qais.yousef@arm.com, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v6 7/9] sched/core: Support latency priority with sched core Date: Fri, 28 Oct 2022 09:36:35 +0200 Message-Id: <20221028073637.31195-8-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221028073637.31195-1-vincent.guittot@linaro.org> References: <20221028073637.31195-1-vincent.guittot@linaro.org> X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747916434538424673?= X-GMAIL-MSGID: =?utf-8?q?1747916434538424673?= Take into account wakeup_latency_gran() when ordering the cfs threads. Signed-off-by: Vincent Guittot --- kernel/sched/fair.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 9583936ce30c..a7372f80b1ea 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -11439,6 +11439,9 @@ bool cfs_prio_less(struct task_struct *a, struct task_struct *b, bool in_fi) delta = (s64)(sea->vruntime - seb->vruntime) + (s64)(cfs_rqb->min_vruntime_fi - cfs_rqa->min_vruntime_fi); + /* Take into account latency prio */ + delta -= wakeup_latency_gran(sea, seb); + return delta > 0; } #else From patchwork Fri Oct 28 07:36:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 12141 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp679800wru; Fri, 28 Oct 2022 00:43:32 -0700 (PDT) X-Google-Smtp-Source: AMsMyM45tVU8WP/mYom8siL+6BWaXlftZ+38VXiVmb6zmMHdJDFQk7Q1hEbKU2wipB9SwH0c9VwZ X-Received: by 2002:a05:6402:2201:b0:44f:443e:2a78 with SMTP id cq1-20020a056402220100b0044f443e2a78mr48614976edb.76.1666943011889; Fri, 28 Oct 2022 00:43:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666943011; cv=none; d=google.com; s=arc-20160816; b=LNCICjLHzkeTJXT/xGBNsh+drGDNvpr0c1lfIjLGTmVWb/yqSlhnl2Tc+sIKD0sAiP aRVStz/lb/d7w0KF3PIwOxQnx31MkWlJQ3HqwW7q8f7IGPWlcO56EohWXGQRgPAqlUOS b93dbksWdPEngupbdvQ6QwQ3oDC1rcrSz3anDIR8L3ywQhq+s7r29DzC+vkeVc9qo6JF /uUPxLAjSH7wrbtp0KOmcbixXLJyD8LkgQvfBIfItBzFC/TOUnHdMROgpCNpzNpZHxen hr3KR6GvMXFDzEEkIIGZ6x1bMhZwRK5PwcEYRlL810HMGdKRufHIK1SbGZZ6Q4NEo2fo J/mw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=ENYyRamoy8ZlslgxQpSlD1U36+EC0S6QAIMpH75DwoY=; b=fB5SMNaVDZm/gunB9xyly9j9pFvNjYb6Dl76odS7m1kkz9hV8cCjoozrKj/rR7ebOC gNkxV9xhHnAcsd9mJRzK5cZKQPJ0d3abfiBRT/loKmNzqEt78DdtOJrxSVXfOYCAsRbP OaeIdc17R7lZjZOXt8ia4JA36rmUFZAWEVwIXQSiKb4V80IxwGdQiE5NVj+rmSJKJflg phSRWKVdg2LlXu2OXCePzeIIXGB4NYTxaTAYUCuyfUlxOVebDjPQTc5vWV6Fb2p8bDxV /ZtH4HXQdzl7Rtead53Bq3w4m+Rv99MMrbnIZ3BnaEj3jQuky8VNGKxS3CeRwg2F1nAS t7rQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=McbZwsPK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id nd8-20020a170907628800b007ad86f97b36si4223649ejc.560.2022.10.28.00.43.08; Fri, 28 Oct 2022 00:43:31 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=McbZwsPK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230300AbiJ1Hh3 (ORCPT + 99 others); Fri, 28 Oct 2022 03:37:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43378 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230186AbiJ1HhM (ORCPT ); Fri, 28 Oct 2022 03:37:12 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BBF911BC15B for ; Fri, 28 Oct 2022 00:37:06 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id k8so5872176wrh.1 for ; Fri, 28 Oct 2022 00:37:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=ENYyRamoy8ZlslgxQpSlD1U36+EC0S6QAIMpH75DwoY=; b=McbZwsPKlZfVbGKcEq4/662OAB9E+k3f5GdHKRbqIWoXpwDvFzIp+NGqIkEmuuojsM 9gNIdzoo+Ct9st6crajDxF/HEJYv7ipUNh1EbPDxdWKcH8uVBNOVWH9C+pmDEe1W+Lv6 x+YJ4oDs1/oqyFthfp7Su9rqEn3XEH+rBPXp9kT1I4DCXzDgZ/zGkfW/HNtAESBY/Uf1 7ohzrUFED3GuZm8JBpt+Z9c9tSNcENdu7/JPtKqYBMWNuaQvScKhqSuLlvxo0xrCVbta /5ZbpHq0MQGBuOZOfsFNIPBsSzMFHrFIjFd2ClyEshxSigUL8VhioiQIpZRABNIv7i4c Py7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ENYyRamoy8ZlslgxQpSlD1U36+EC0S6QAIMpH75DwoY=; b=M3Hd5+NHh5QF1JD8h6GaOxXk1RbVtbLcdtf0TIkjpueBATwVDOIzuadOkN9r9lLiyR hTkQu49uwyXYb2W8cOPNVpmWxPGdWXq4k+JNifOGXmBM7dmf+J3XNobl+u1FOcTFIvcC p6Qi7NGXju9f142zCIztx1wOoIK/CAHaHKYpC5We+9wyfoQv0NesRhABHlf5Z4fM/sad NcCXjmzF5CyJCZT7F5lCh/CGWWIBIzNLYuXrI3Is1qVzu9/Ba1Cs7cD9zfjLTJA9pM/x 6Ftc6wplQvqAPXxA7cxfO8AL0Vkqdt5NZPKMUAmlVagiz/T8pAH1XPdciK5hbtqewl6I ZrQg== X-Gm-Message-State: ACrzQf0EMwrkMI0pKI2waGsnhnF8brmIJI86XeJ/ZyuynUNBMw9O9OMC opCzEYjgLJa5vRKJxS5pFpW4ew== X-Received: by 2002:adf:e785:0:b0:236:5998:67a0 with SMTP id n5-20020adfe785000000b00236599867a0mr22987923wrm.414.1666942625227; Fri, 28 Oct 2022 00:37:05 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:f:6020:c12b:b448:f0a9:83ef]) by smtp.gmail.com with ESMTPSA id j8-20020a05600c190800b003b47e8a5d22sm7783129wmq.23.2022.10.28.00.37.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Oct 2022 00:37:03 -0700 (PDT) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com Cc: qais.yousef@arm.com, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v6 8/9] sched/fair: Add latency list Date: Fri, 28 Oct 2022 09:36:36 +0200 Message-Id: <20221028073637.31195-9-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221028073637.31195-1-vincent.guittot@linaro.org> References: <20221028073637.31195-1-vincent.guittot@linaro.org> X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747916435665225227?= X-GMAIL-MSGID: =?utf-8?q?1747916435665225227?= Add a rb tree for latency sensitive entities so we can schedule the most sensitive one first even when it failed to preempt current at wakeup or when it got quickly preempted by another entity of higher priority. In order to keep fairness, the latency is used once at wakeup to get a minimum slice and not during the following scheduling slice to prevent long running entity to got more running time than allocated to his nice priority. The rb tree enables to cover the last corner case where latency sensitive entity can't got schedule quickly after the wakeup. Signed-off-by: Vincent Guittot --- include/linux/sched.h | 2 + kernel/sched/fair.c | 96 +++++++++++++++++++++++++++++++++++++++++-- kernel/sched/sched.h | 1 + 3 files changed, 96 insertions(+), 3 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index a74cad08e91e..0b92674e3664 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -547,6 +547,8 @@ struct sched_entity { /* For load-balancing: */ struct load_weight load; struct rb_node run_node; + struct rb_node latency_node; + unsigned int on_latency; struct list_head group_node; unsigned int on_rq; diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index a7372f80b1ea..c28992b7d1a6 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -664,7 +664,77 @@ struct sched_entity *__pick_last_entity(struct cfs_rq *cfs_rq) return __node_2_se(last); } +#endif +/************************************************************** + * Scheduling class tree data structure manipulation methods: + * for latency + */ + +static inline bool latency_before(struct sched_entity *a, + struct sched_entity *b) +{ + return (s64)(a->vruntime + a->latency_offset - b->vruntime - b->latency_offset) < 0; +} + +#define __latency_node_2_se(node) \ + rb_entry((node), struct sched_entity, latency_node) + +static inline bool __latency_less(struct rb_node *a, const struct rb_node *b) +{ + return latency_before(__latency_node_2_se(a), __latency_node_2_se(b)); +} + +/* + * Enqueue an entity into the latency rb-tree: + */ +static void __enqueue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) +{ + + /* Only latency sensitive entity can be added to the list */ + if (se->latency_offset >= 0) + return; + + if (se->on_latency) + return; + + /* + * An execution time less than sysctl_sched_min_granularity means that + * the entity has been preempted by a higher sched class or an entity + * with higher latency constraint. + * Put it back in the list so it gets a chance to run 1st during the + * next slice. + */ + if (!(flags & ENQUEUE_WAKEUP)) { + u64 delta_exec = se->sum_exec_runtime - se->prev_sum_exec_runtime; + + if (delta_exec >= sysctl_sched_min_granularity) + return; + } + + rb_add_cached(&se->latency_node, &cfs_rq->latency_timeline, __latency_less); + se->on_latency = 1; +} + +static void __dequeue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se) +{ + if (se->on_latency) { + rb_erase_cached(&se->latency_node, &cfs_rq->latency_timeline); + se->on_latency = 0; + } +} + +static struct sched_entity *__pick_first_latency(struct cfs_rq *cfs_rq) +{ + struct rb_node *left = rb_first_cached(&cfs_rq->latency_timeline); + + if (!left) + return NULL; + + return __latency_node_2_se(left); +} + +#ifdef CONFIG_SCHED_DEBUG /************************************************************** * Scheduling class statistics methods: */ @@ -4439,8 +4509,10 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) check_schedstat_required(); update_stats_enqueue_fair(cfs_rq, se, flags); check_spread(cfs_rq, se); - if (!curr) + if (!curr) { __enqueue_entity(cfs_rq, se); + __enqueue_latency(cfs_rq, se, flags); + } se->on_rq = 1; if (cfs_rq->nr_running == 1) { @@ -4526,8 +4598,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) clear_buddies(cfs_rq, se); - if (se != cfs_rq->curr) + if (se != cfs_rq->curr) { __dequeue_entity(cfs_rq, se); + __dequeue_latency(cfs_rq, se); + } se->on_rq = 0; account_entity_dequeue(cfs_rq, se); @@ -4616,6 +4690,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se) */ update_stats_wait_end_fair(cfs_rq, se); __dequeue_entity(cfs_rq, se); + __dequeue_latency(cfs_rq, se); update_load_avg(cfs_rq, se, UPDATE_TG); } @@ -4654,7 +4729,7 @@ static struct sched_entity * pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr) { struct sched_entity *left = __pick_first_entity(cfs_rq); - struct sched_entity *se; + struct sched_entity *latency, *se; /* * If curr is set we have to see if its left of the leftmost entity @@ -4696,6 +4771,12 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr) se = cfs_rq->last; } + /* Check for latency sensitive entity waiting for running */ + latency = __pick_first_latency(cfs_rq); + if (latency && (latency != se) && + wakeup_preempt_entity(latency, se) < 1) + se = latency; + return se; } @@ -4719,6 +4800,7 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev) update_stats_wait_start_fair(cfs_rq, prev); /* Put 'current' back into the tree. */ __enqueue_entity(cfs_rq, prev); + __enqueue_latency(cfs_rq, prev, 0); /* in !on_rq case, update occurred at dequeue */ update_load_avg(cfs_rq, prev, 0); } @@ -11712,6 +11794,7 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first) void init_cfs_rq(struct cfs_rq *cfs_rq) { cfs_rq->tasks_timeline = RB_ROOT_CACHED; + cfs_rq->latency_timeline = RB_ROOT_CACHED; u64_u32_store(cfs_rq->min_vruntime, (u64)(-(1LL << 20))); #ifdef CONFIG_SMP raw_spin_lock_init(&cfs_rq->removed.lock); @@ -12020,8 +12103,15 @@ int sched_group_set_latency(struct task_group *tg, s64 latency) for_each_possible_cpu(i) { struct sched_entity *se = tg->se[i]; + struct rq *rq = cpu_rq(i); + struct rq_flags rf; + + rq_lock_irqsave(rq, &rf); + __dequeue_latency(se->cfs_rq, se); WRITE_ONCE(se->latency_offset, latency); + + rq_unlock_irqrestore(rq, &rf); } mutex_unlock(&shares_mutex); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index b22113c8939b..c8206945868d 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -599,6 +599,7 @@ struct cfs_rq { #endif struct rb_root_cached tasks_timeline; + struct rb_root_cached latency_timeline; /* * 'curr' points to currently running entity on this cfs_rq. From patchwork Fri Oct 28 07:36:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 12142 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp679881wru; Fri, 28 Oct 2022 00:43:46 -0700 (PDT) X-Google-Smtp-Source: AMsMyM676hWcSktj9DiH8YmWGUust4v6ScdmnfBVx0RCEasYckVtlXihBYArUR2hPFE7CRFNEHoo X-Received: by 2002:a05:6402:c4d:b0:457:99ec:1837 with SMTP id cs13-20020a0564020c4d00b0045799ec1837mr48893506edb.86.1666943026229; Fri, 28 Oct 2022 00:43:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666943026; cv=none; d=google.com; s=arc-20160816; b=cLW9FAKZtKwqApOEnFQwXutwYUAC3iXt8Z1uXw4FFIHL13TotKCcqvusMKoSJ15Qs4 jzuwtBsKRWI1dMj9fsbZJqCao6TuJSan9dFRMcgQKIh/SdGbrFtCBhfhitbVYYFFfWzR JWohB/CFToGpO2C/bqc9N5cjJ4jjiNZ2ua2UZ01arahhk/xjAxVRH40nYbkrC16kSH9m MmMkx/f6mOk9tjGwNz02v2xYxgMkwDjuPztP44/ijIhKxv8xHk0ca1KSa/zduNsOInIT QniS6y5aLeVTTXIuh5RESV/EnOsThw1b8QHNAiD58nA5Dg7vie2bgfEKeNZtcGQOJRrG Ruwg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=zy7QY6HAum4eocAiGxC3tA1yZA7zRnpZ3kdrW18hPm0=; b=hvgvprl9C5eLMZ3o6kRssoS9WnFhm/IrHitLt5XKgasifK43quLZaEp2ppllG9AkmS t+EwOujmH4BGyADc14EZG5yRHdLvK70S4uWX/z9uFA5IpKNy7juNssTt8ihTlKm42Sgj XHGD8v3lvrwG4vmPgFFej9QZ+RY0h54yj/c2syMP8EtuhIgTm8aTxam59wdyeZuLtomy 2bVJCpnslBACzoaGKcfa9gQSqnV7d5jss+qtKrSgI/5wUe25HVeJ6VGxflSmQ10JkVA2 WDUO3y04xLTzOq6fA7aTKKqoK2B5jI520X26kZdJBbCn1OQ6rf4cwUAm2li3zwX/BIWp n/gw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=VLe7YWFt; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id qw34-20020a1709066a2200b0078da3cd3073si1876105ejc.628.2022.10.28.00.43.23; Fri, 28 Oct 2022 00:43:46 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=VLe7YWFt; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230264AbiJ1Hhr (ORCPT + 99 others); Fri, 28 Oct 2022 03:37:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43092 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230226AbiJ1HhP (ORCPT ); Fri, 28 Oct 2022 03:37:15 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A25921BE1D6 for ; Fri, 28 Oct 2022 00:37:09 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id z14so5830391wrn.7 for ; Fri, 28 Oct 2022 00:37:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=zy7QY6HAum4eocAiGxC3tA1yZA7zRnpZ3kdrW18hPm0=; b=VLe7YWFtcwIvALbj2GLX0Md1taqlPqb82YAbGh6QGYJTtbgEU4+3dtSWNFLSgbx4yI Qcf+VVGUxYSdtKyLIa2gsj0R8gSwuvBup3mvo3kxxwRd5fG4iRv5/QX0+eL4DwnWTO4g MkD/YgWag9h/tQRxFb8J5LFYNcvMwxyE9LE8tZTH4g+mTPTiGSr1P6d6XPrbeVJtIV6x IElsLB2eVc/pEcwJctzSOAUEW6AnvJ7u46WxgstpyX6Gz5ztSRaaiSejyLA0uXfvZ0O9 PADRkkwSJ+bMoZ+wjjzRbzYZdOnJOr/n/2AOhr1eP/oJncvK5Ez2CuZkHMax1j+RWhzW n0mQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zy7QY6HAum4eocAiGxC3tA1yZA7zRnpZ3kdrW18hPm0=; b=durP6NWS6xyMHERg+qt48ojJXfGbBGYdh8MP9mLsPvkfi9AqikYDqLtifbbbRcIFK2 KOcaCrvOAgK8jWF55Lp5SEnh2HchnFuzLCHpOyuqIbsHHOSL7yXVO4+2rcX7MDu2vGmj 4xiKE3lIHDfbSZVpzhaWTWXaEMB+l4xSgPRWasFKO/9TvRUCfGhKM8qWzVxwB1YTzC3N WQ/k1YkezT3bOL1I8wZaolOI32B0k6c8AtNkiIZKD4SsGzPvFHUFx8CM08Sxrfmk2lll RO2BB6+B8VovrDpaiBxNo/IZYIXdf3ztQWNR+nGnFONmH4VGf6SgLLfQLl53JaUDsV7P GyDA== X-Gm-Message-State: ACrzQf2z4Z6Rp5vGutjc8I+PbgS3PpN0irzh5Vz6fOq2lLJ+N1K26CIB jItzwnDv+eZTfpyHoSjnxufiqg== X-Received: by 2002:adf:ec8a:0:b0:236:5b80:da83 with SMTP id z10-20020adfec8a000000b002365b80da83mr20949051wrn.509.1666942627666; Fri, 28 Oct 2022 00:37:07 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:f:6020:c12b:b448:f0a9:83ef]) by smtp.gmail.com with ESMTPSA id j8-20020a05600c190800b003b47e8a5d22sm7783129wmq.23.2022.10.28.00.37.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Oct 2022 00:37:06 -0700 (PDT) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com Cc: qais.yousef@arm.com, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v6 9/9] sched/fair: remove check_preempt_from_others Date: Fri, 28 Oct 2022 09:36:37 +0200 Message-Id: <20221028073637.31195-10-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221028073637.31195-1-vincent.guittot@linaro.org> References: <20221028073637.31195-1-vincent.guittot@linaro.org> X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747916451132416642?= X-GMAIL-MSGID: =?utf-8?q?1747916451132416642?= With the dedicated latency list, we don't have to take care of this special case anymore as pick_next_entity checks for a runnable latency sensitive task. Signed-off-by: Vincent Guittot --- kernel/sched/fair.c | 34 ++-------------------------------- 1 file changed, 2 insertions(+), 32 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index c28992b7d1a6..9a421b49dbfd 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -5802,35 +5802,6 @@ static int sched_idle_cpu(int cpu) } #endif -static void set_next_buddy(struct sched_entity *se); - -static void check_preempt_from_others(struct cfs_rq *cfs, struct sched_entity *se) -{ - struct sched_entity *next; - - if (se->latency_offset >= 0) - return; - - if (cfs->nr_running <= 1) - return; - /* - * When waking from another class, we don't need to check to preempt at - * wakeup and don't set next buddy as a candidate for being picked in - * priority. - * In case of simultaneous wakeup when current is another class, the - * latency sensitive tasks lost opportunity to preempt non sensitive - * tasks which woke up simultaneously. - */ - - if (cfs->next) - next = cfs->next; - else - next = __pick_first_entity(cfs); - - if (next && wakeup_preempt_entity(next, se) == 1) - set_next_buddy(se); -} - /* * The enqueue_task method is called before nr_running is * increased. Here we update the fair scheduling stats and @@ -5917,15 +5888,14 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) if (!task_new) update_overutilized_status(rq); - if (rq->curr->sched_class != &fair_sched_class) - check_preempt_from_others(cfs_rq_of(&p->se), &p->se); - enqueue_throttle: assert_list_leaf_cfs_rq(rq); hrtick_update(rq); } +static void set_next_buddy(struct sched_entity *se); + /* * The dequeue_task method is called before nr_running is * decreased. We remove the task from the rbtree and