From patchwork Thu May 18 01:34:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josh Don X-Patchwork-Id: 95610 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp173865vqo; Wed, 17 May 2023 18:45:11 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4dNbqxmH/wN8d7UjpzPYkkrfMwD15Xjuez+3XjEBoHzR142wZHUAuHM8TinaeH3SxjBoTn X-Received: by 2002:a05:6a20:9144:b0:ff:ca91:68ee with SMTP id x4-20020a056a20914400b000ffca9168eemr148033pzc.9.1684374311087; Wed, 17 May 2023 18:45:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684374311; cv=none; d=google.com; s=arc-20160816; b=GN8ui/eSibWPfjJCX/bI7sRvHPjycsJZf0T0WxXLUYE9rs1etQBG9F9wRyxJRT34nd aHkdEIDNDTZhsRhnPwD5kT1upA9bFHkJsRyXcC2h2V2CT0Z14MRmGQ9pPECHKnvh+6Rb o+TBZBXvWzKNXFbpgAlF9m0nV4VdwT5/xaSY1ehE/yjsgNs/4HqU5sSNwNXhTYiZ+jh2 vCoLfhpXiOa3vY9H5/AE7mshyxo5+YjwzNkoTqZPz05jOJ7aapNsaoAmigwMpADurYZ8 klSDYMwghOhTi46I8w6r21Earh4dHtbvRBRNoYtl2UgaBKRdqALgz6RfsfkQtphPviOW mO0g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:mime-version:date :dkim-signature; bh=YCkOGnN5VV+G0vc4oBgEXNSPP/cS4uflXB2VdW0nTDc=; b=uaj8bBbkhvOfudkLjKwMymIgScHMnqqiIdJO9Tw7n/gLBQb73wbX+PkEzH4VwbPpPn VbBrI/0xtH9kVW3tmvtTdCxWqIfHlVfI3IS/KUwY008G2w3u8JlAVGBBwSbIARM4Fe/x l9N3/BmzgXSUuU+XBcMq72ImSg9YfszWzQIgrBEwtWcmrN7HFyztgXAQobJE5MvwBNP6 H2tVo5p3SCg1i52SCakBHU62ygmcm10iGNZR1H7RWPtHj9Gofhq3nbgUr9WpTjZ/9+/5 pdxZjDUlbBVtEMroIYDJn0wEvzh9qmH31hXNk2+0E3T//IRcsyKeZ46gQit0NLMweb/5 PHvQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=LV9SKqc9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 3-20020a621703000000b006435b9bd9f0si452291pfx.45.2023.05.17.18.44.56; Wed, 17 May 2023 18:45:11 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=LV9SKqc9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229676AbjERBem (ORCPT + 99 others); Wed, 17 May 2023 21:34:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36284 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229621AbjERBek (ORCPT ); Wed, 17 May 2023 21:34:40 -0400 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 987D535BF for ; Wed, 17 May 2023 18:34:37 -0700 (PDT) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-561abf855afso15762157b3.2 for ; Wed, 17 May 2023 18:34:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684373677; x=1686965677; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=YCkOGnN5VV+G0vc4oBgEXNSPP/cS4uflXB2VdW0nTDc=; b=LV9SKqc9LLji/8d1D1nfFc5i5jRcnJLn+AfdqPiE8q/ZsW+08nHc91HNb4jIFg5d6H jgdmF4AsVk+sjFNbe6dNmj/yjWzAvlQoPbqzFzG1yAFkYJTm0tSE4JRPp2pkAbpk8Yzl sMwx5yX3xjLDUYEbTRyn1YgUs8/4CyPkDnd5i1xEg8U1TP4klX+n1KZZQP7nQ2rdYwLc sIkYfAezE/CJwAl1KHldUuPcIRJf1cXLC5Q2taC1S6LpVTa83YOcpVTVkwZz0l9o+njR AndVecG6UkuOC+ZGe3GQEYiXzhMOPnOp8lV1m5u//s0N+M6h7dNTW4jeFc87BXnBz6yA wdNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684373677; x=1686965677; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=YCkOGnN5VV+G0vc4oBgEXNSPP/cS4uflXB2VdW0nTDc=; b=kI2c7SMIZzfwqPmR1IvP9sLx9OjG7I36bndLiZYyyZmizg4EVb+nPLEv6WsdFxvCby 2u0futUAKUR+Vw3zqLXa+xcklGQhc7SWNfikNEZEmvv36kYS+61p11AtjgDgczNK8u4i mJSoRWOFbdUaSB4s2VSrBH6gNACylgfimXR4mBEvHUQc1OVIGQmlUwNqp2i+AJE9F30/ vnojCtHPGxNve1D+Gg1u+n6dE2tRgkdu1qJ47SvhtQJZNr+hNrq+0YHLvwoxV6qUpQWh VY0C0wjPuCVD0rEEztrbZcj3uQBO12lKgK4AFniuesgBPBqEZEYywQZJhBS4bptdpeQ1 t77g== X-Gm-Message-State: AC+VfDymT0+9j45GWoX+Pp3CO1nzeVenvhvKWCyC30WQT8Oifqt7+BBj fJLd05gyyATQATsB0LwJ6xCX1PY4/T78 X-Received: from joshdon-desktop.svl.corp.google.com ([2620:15c:2d4:203:53e2:12a7:c986:1d79]) (user=joshdon job=sendgmr) by 2002:a81:ae64:0:b0:561:2d82:7f08 with SMTP id g36-20020a81ae64000000b005612d827f08mr131225ywk.0.1684373676864; Wed, 17 May 2023 18:34:36 -0700 (PDT) Date: Wed, 17 May 2023 18:34:13 -0700 Mime-Version: 1.0 X-Mailer: git-send-email 2.40.1.606.ga4b1b128d6-goog Message-ID: <20230518013414.3053254-1-joshdon@google.com> Subject: [PATCH 1/2] sched: don't account throttle time for empty groups From: Josh Don To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot Cc: Tejun Heo , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , linux-kernel@vger.kernel.org, Xiangling Kong , Josh Don X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766194477971609020?= X-GMAIL-MSGID: =?utf-8?q?1766194477971609020?= It is easy for a cfs_rq to become throttled even when it has no enqueued entities (for example, if we have just put_prev()'d the last runnable task of the cfs_rq, and the cfs_rq is out of quota). Avoid accounting this time towards total throttle time, since it otherwise falsely inflates the stats. Signed-off-by: Josh Don --- kernel/sched/fair.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index f4b8b906d30a..85c2c0c3cab6 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4768,6 +4768,7 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) { bool renorm = !(flags & ENQUEUE_WAKEUP) || (flags & ENQUEUE_MIGRATED); bool curr = cfs_rq->curr == se; + struct rq *rq = rq_of(cfs_rq); /* * If we're the current task, we must renormalise before calling @@ -4816,8 +4817,12 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) if (cfs_rq->nr_running == 1) { check_enqueue_throttle(cfs_rq); - if (!throttled_hierarchy(cfs_rq)) + if (!throttled_hierarchy(cfs_rq)) { list_add_leaf_cfs_rq(cfs_rq); + } else { + if (!cfs_rq->throttled_clock) + cfs_rq->throttled_clock = rq_clock(rq); + } } } @@ -5423,7 +5428,9 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq) * throttled-list. rq->lock protects completion. */ cfs_rq->throttled = 1; - cfs_rq->throttled_clock = rq_clock(rq); + SCHED_WARN_ON(cfs_rq->throttled_clock); + if (cfs_rq->nr_running) + cfs_rq->throttled_clock = rq_clock(rq); return true; } @@ -5441,7 +5448,10 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) update_rq_clock(rq); raw_spin_lock(&cfs_b->lock); - cfs_b->throttled_time += rq_clock(rq) - cfs_rq->throttled_clock; + if (cfs_rq->throttled_clock) { + cfs_b->throttled_time += rq_clock(rq) - cfs_rq->throttled_clock; + cfs_rq->throttled_clock = 0; + } list_del_rcu(&cfs_rq->throttled_list); raw_spin_unlock(&cfs_b->lock); From patchwork Thu May 18 01:34:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josh Don X-Patchwork-Id: 95611 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp173961vqo; Wed, 17 May 2023 18:45:26 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7TQUZgv8GyppjoaW2RBRlum0vRKU/HNDNrfjwIbp87CMLZGbfWKn3OpL8P9dMFrU6cERkQ X-Received: by 2002:a05:6a00:2355:b0:647:4dee:62b7 with SMTP id j21-20020a056a00235500b006474dee62b7mr2203643pfj.29.1684374326160; Wed, 17 May 2023 18:45:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684374326; cv=none; d=google.com; s=arc-20160816; b=FoIufMKgEJK3Vb4YXnTFNNO8yl44IIsmFdnKblNopdUSc0iJi24v1peSP9clJTwg3f e0wlhgTQv+AUBnx59nCB7/uesXr51k7h/odMgu4gjX6zGGjNUOglIni60avikbwxM/Kp HEuB/5Ip+tQIHqveRNmV4zgzG0Z2LUVSuCaDtonSgjm466JnNhlBwbuhdu0KuJevCBd4 yXaYJhm0Jka2lEcBBaiiAhReiZP63fRyOJfo9P8zW100fqMrHbLfQXr9jo8hkvxsCCA6 nRI766xiem71NGRHaDhtdOpltdNeY+emcba65eV1E+QD4HvueRJu3vv7+eIDjncVCprH qEFw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=zGd11uJrEauFdMc+x3tyZ2RnE2rDj1U9cEd6S0knPbY=; b=vn5h4WgS+ECu4dJCq8+CfrXKnP9Sa/sV/JagV9OVP9LhmjbkVszdOAhlq96pgVqyra zJfNB0U9hdf0OWmcMX9SvgEglLmtfKEFehPASQJbVniPZIAzE092WKejETH6h0tB1Qc1 GeO6GAtmY20UFvdwYsgXTdJFKndxBVc71iQtg2g65TnjdSQmaQ9hY1Sx2WvznUp9jixC XI+tvZ+f3JdoxGLdyIgjd3JiMXsrMG+mEg5hWpF1qjnqwYk4DD5ZMJHqTkV3onNL2tgu utF+i9wMOf5hbFldxvraXLQ6Qsj/j6gxTGkmb7K7ULwORrSL8wJn224lOMlb4pkT11vO wmHQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=EM6+JMyN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id g27-20020aa79f1b000000b0063f1cb928bdsi396006pfr.313.2023.05.17.18.45.11; Wed, 17 May 2023 18:45:26 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=EM6+JMyN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229708AbjERBeu (ORCPT + 99 others); Wed, 17 May 2023 21:34:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36296 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229621AbjERBeo (ORCPT ); Wed, 17 May 2023 21:34:44 -0400 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4C02319BD for ; Wed, 17 May 2023 18:34:43 -0700 (PDT) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-559d36a91a9so14836547b3.1 for ; Wed, 17 May 2023 18:34:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684373682; x=1686965682; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=zGd11uJrEauFdMc+x3tyZ2RnE2rDj1U9cEd6S0knPbY=; b=EM6+JMyNggS40H+yrDfKc942QmVVl0BSiAmnmIdr1/kkXic9IJcQtltRWoogIsU1JP yizy17CMGSaLm87+DRT8IW5KwhsIvUklVd3Zg+cdMbNJzyGyic057eS3d77CkMBxJf3x TOf8Gg6xvn+aTzjCD0kmDzCbEKp1VvLO4HGLm3Fk+K0u8OuJYzQbUQ2ifKVRQN4HhLrc z0AM3nsccsVc0GtAFgIOJ6PprJp3r2bM3aezhzo5QOL7wi1W4rRRPmKmx1WXLTfKscCK YqMmeAs1h1ouryvDB/6hU1nT3uNsrTbmKiCdAZmS+w0WbXhTJ5KaCEj2gZe3wwS8/Bzy UQhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684373682; x=1686965682; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zGd11uJrEauFdMc+x3tyZ2RnE2rDj1U9cEd6S0knPbY=; b=S9qDcyiKuBVd3K2/GFH4bPEmynrz1krZTr0EZYyjhyQOrkIJSJ0v9E+DRbd1NNhnP1 s8ihy625IQoD6futX9r0pOrhzfdF7jUKf1YIjFje2jSF0sLg48JU71NI5BLmlhVNf/xw X8Y52DZn3QO+o4VpLd1Ug4EY3mWqNH9vfSN6cyfG1bkxZxtImdcDrGqPg3v4cpiQCaQF YE/E82lAKqsAxXYjPCVosglFQGZp7XBlLd4vj6uAtP+/dJg/nZs7zxnj2NlNQs8SLrCV Fjtbyzaq2otMneFn+45oQ5uJd6eAW84f1Ucb7vAgZy4NL9laxQv7o7joNNCFs065pUJP 6aJg== X-Gm-Message-State: AC+VfDxoVhCTokDIE1I8rBoJ5vS/XtzkVuE5c/K/6tAsJGmvO/lVWE8h +ABYxbUF0yBsWFveyMJVsGPc/Eb5Tfu0 X-Received: from joshdon-desktop.svl.corp.google.com ([2620:15c:2d4:203:53e2:12a7:c986:1d79]) (user=joshdon job=sendgmr) by 2002:a81:af15:0:b0:561:8bfb:feb1 with SMTP id n21-20020a81af15000000b005618bfbfeb1mr74532ywh.10.1684373682560; Wed, 17 May 2023 18:34:42 -0700 (PDT) Date: Wed, 17 May 2023 18:34:14 -0700 In-Reply-To: <20230518013414.3053254-1-joshdon@google.com> Mime-Version: 1.0 References: <20230518013414.3053254-1-joshdon@google.com> X-Mailer: git-send-email 2.40.1.606.ga4b1b128d6-goog Message-ID: <20230518013414.3053254-2-joshdon@google.com> Subject: [PATCH 2/2] sched: add throttled time stat for throttled children From: Josh Don To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot Cc: Tejun Heo , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , linux-kernel@vger.kernel.org, Xiangling Kong , Josh Don X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766194493326736884?= X-GMAIL-MSGID: =?utf-8?q?1766194493326736884?= We currently export the total throttled time for cgroups that are given a bandwidth limit. This patch extends this accounting to also account the total time that each children cgroup has been throttled. This is useful to understand the degree to which children have been affected by the throttling control. Children which are not runnable during the entire throttled period, for example, will not show any self-throttling time during this period. Signed-off-by: Josh Don --- kernel/sched/core.c | 21 +++++++++++++++++++-- kernel/sched/fair.c | 17 +++++++++++++++++ kernel/sched/sched.h | 2 ++ 3 files changed, 38 insertions(+), 2 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index a384344ec9ee..fcd702889fcb 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -11064,6 +11064,18 @@ static int __cfs_schedulable(struct task_group *tg, u64 period, u64 quota) return ret; } +static u64 throttled_time_self(struct task_group *tg) +{ + int i; + u64 total = 0; + + for_each_possible_cpu(i) { + total += READ_ONCE(tg->cfs_rq[i]->throttled_clock_self_time); + } + + return total; +} + static int cpu_cfs_stat_show(struct seq_file *sf, void *v) { struct task_group *tg = css_tg(seq_css(sf)); @@ -11072,6 +11084,7 @@ static int cpu_cfs_stat_show(struct seq_file *sf, void *v) seq_printf(sf, "nr_periods %d\n", cfs_b->nr_periods); seq_printf(sf, "nr_throttled %d\n", cfs_b->nr_throttled); seq_printf(sf, "throttled_time %llu\n", cfs_b->throttled_time); + seq_printf(sf, "throttled_time_self %llu\n", throttled_time_self(tg)); if (schedstat_enabled() && tg != &root_task_group) { struct sched_statistics *stats; @@ -11204,20 +11217,24 @@ static int cpu_extra_stat_show(struct seq_file *sf, { struct task_group *tg = css_tg(css); struct cfs_bandwidth *cfs_b = &tg->cfs_bandwidth; - u64 throttled_usec, burst_usec; + u64 throttled_usec, burst_usec, throttled_self_usec; throttled_usec = cfs_b->throttled_time; do_div(throttled_usec, NSEC_PER_USEC); + throttled_self_usec = throttled_time_self(tg); + do_div(throttled_self_usec, NSEC_PER_USEC); burst_usec = cfs_b->burst_time; do_div(burst_usec, NSEC_PER_USEC); seq_printf(sf, "nr_periods %d\n" "nr_throttled %d\n" "throttled_usec %llu\n" + "throttled_self_usec %llu\n" "nr_bursts %d\n" "burst_usec %llu\n", cfs_b->nr_periods, cfs_b->nr_throttled, - throttled_usec, cfs_b->nr_burst, burst_usec); + throttled_usec, throttled_self_usec, cfs_b->nr_burst, + burst_usec); } #endif return 0; diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 85c2c0c3cab6..fd348b9421c1 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4822,6 +4822,8 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) } else { if (!cfs_rq->throttled_clock) cfs_rq->throttled_clock = rq_clock(rq); + if (!cfs_rq->throttled_clock_self) + cfs_rq->throttled_clock_self = rq_clock(rq); } } } @@ -5327,6 +5329,17 @@ static int tg_unthrottle_up(struct task_group *tg, void *data) list_add_leaf_cfs_rq(cfs_rq); } + if (cfs_rq->throttled_clock_self) { + u64 delta = rq_clock(rq) - cfs_rq->throttled_clock_self; + + cfs_rq->throttled_clock_self = 0; + + if (SCHED_WARN_ON((s64)delta < 0)) + delta = 0; + + cfs_rq->throttled_clock_self_time += delta; + } + return 0; } @@ -5342,6 +5355,10 @@ static int tg_throttle_down(struct task_group *tg, void *data) } cfs_rq->throttle_count++; + SCHED_WARN_ON(cfs_rq->throttled_clock_self); + if (cfs_rq->nr_running) + cfs_rq->throttled_clock_self = rq_clock(rq); + return 0; } diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 060616944d7a..ddc48b2f1fd0 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -642,6 +642,8 @@ struct cfs_rq { u64 throttled_clock; u64 throttled_clock_pelt; u64 throttled_clock_pelt_time; + u64 throttled_clock_self; + u64 throttled_clock_self_time; int throttled; int throttle_count; struct list_head throttled_list;