From patchwork Tue Jun 20 18:32:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josh Don X-Patchwork-Id: 110647 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp3871408vqr; Tue, 20 Jun 2023 11:54:51 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4zZvFHiVc3+Yc/+RFYjI9Gv1xT2Awj/JTmEsXm502SXtDmxiWAonyfbjkGENQD3gUI7VLa X-Received: by 2002:a05:6359:282:b0:12b:c390:ac01 with SMTP id ek2-20020a056359028200b0012bc390ac01mr3325815rwb.30.1687287291012; Tue, 20 Jun 2023 11:54:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1687287290; cv=none; d=google.com; s=arc-20160816; b=Evzqi8VAPbTB/dP/tP+wZ0a8vltZwgXR3law6mBOUmmbMa9xP4iKeHbtgjAZ9JvtBW Bpaskv5WEA8JYkXYP9z9+yV2WN8K6MCHi8xunfWCIeMDgCyZubA+TGSMYCGig/7HUrUy 8rqgm41g+STVz7ZqR5CxReEFhJKsS8vkO/rbFFWuMZVPgTtf2mvJupfDdsvaeuhyKDxH RC58XnPX8auxQH2mxhNJr5EsTNYirgVBEWV/HMrHY1mlMR1yfj05cb2qI86o/ijILD94 r601cm/IBKA3Nkpl9IS9Isok8PGX6bhGAmoewkptx8TlL9SEdKLZVTEPCpVVX3yWK1VI 613g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:mime-version:date :dkim-signature; bh=sBPwNjci+mVKgvTGGSvenUmGifZY3Myji5qjgOuCzdY=; b=FoqmRUfpKfq9cqADUAFLPckkUmvCKuSWnfe670j9/t2kmRIw1NBvnT7DUXeeLRBaRM xtV767FJp0V0Ch0yGLXp4SKSQlT+UNCYVStWL6tmNikbmsP8eSvjlBJWXYcj0yH9VdU5 yYS/+FnIjn5veEPR1VTssp9atsqtSoslxoazEMRjGGN5qWJ8WEmwtEpggakKT6001LzY nXucA6kTWuCOcd+ouoi6mKtPInf865SwA/tQftfYIBpxSMxqtq/Afiz/pID7n8uUDi+Y l+1UExRaGhlQ51d6FU3vky/9NiXhTRTOXUMP+x3HYYyh7sj1HeFzlFdtN2zv72NB0pVt tjdQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=CV163j2P; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id d129-20020a633687000000b00553ebb05d24si240035pga.69.2023.06.20.11.54.36; Tue, 20 Jun 2023 11:54:50 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=CV163j2P; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229527AbjFTSdN (ORCPT + 99 others); Tue, 20 Jun 2023 14:33:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60258 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229567AbjFTSdM (ORCPT ); Tue, 20 Jun 2023 14:33:12 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EF4AF1737 for ; Tue, 20 Jun 2023 11:33:01 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-bd5059e5347so12061876276.0 for ; Tue, 20 Jun 2023 11:33:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687285981; x=1689877981; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=sBPwNjci+mVKgvTGGSvenUmGifZY3Myji5qjgOuCzdY=; b=CV163j2PUGaavyM9SvGsrlqZLN2xBHrKdDynwlJOjsbY0BnCYcw8OuXxIYqTcePd/4 75NNiIVKAORJDQqeVbt8R+SXRThRGQvhEqlcna+1G+zhYFac2tqz+Kx++pU6u41NWRnI IzUsvovlGgT4EJdq0PTaZ4Dm0WIXjBtD0wLcnJDoVjgRFlPCmyTU8VsNYLZJDmPqxxCL OT/TGrpF+2O7PaMxGBI6yeul6VwI9pwRx5/fUXqA3PQO4hd7oy3cRJyi66jJCV40Dd2x EjZ+GR+fkHQuIwCMlkwh6W3+frRhMueAcukods0XG7+nZMO2shq9cUUTv+JNlJgKJmLI hdkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687285981; x=1689877981; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=sBPwNjci+mVKgvTGGSvenUmGifZY3Myji5qjgOuCzdY=; b=i8sGZBGR0ZvxFIKuZKApA62FeRzY1pSb3pEHgTwVFhwGGN2Hj0oW/vl4advXGtEYD7 d7Puta3CXQZPhx050yFvJmkjBk/YiYWM5SF1idXAMTmmqmh43eEZfyqKQRBmZ9y9sqSS DsrMin6cmdamF1J4g116jsa70Rrp1qGv+0/Tz3S+tWjaS4FMWwsIULAb3aEejel96Do4 tqh35Cjy9s9q0kgxYsmClk6oIiZQt1ckBw9dNjcQKxhVBg74kcYzpgBxRHOkSnioBIuW DjOxvtFUYIdmn5FzgZOwy1bgysIQIXlCa62jlXocRWxJEloJTuv+5Cth3bN9NyC0Uamk EQOQ== X-Gm-Message-State: AC+VfDzRN7oyVD1DGU3pJYfLYpc9+YO1F9+N9vU1yRZZOjADEJCNJb/A vKS+Akkb/OczqoOd9YukqljJlDtf+ny5 X-Received: from joshdon-desktop.svl.corp.google.com ([2620:15c:2a3:200:7f30:1fa1:c3c6:d39a]) (user=joshdon job=sendgmr) by 2002:a05:6902:561:b0:bad:2b06:da3 with SMTP id a1-20020a056902056100b00bad2b060da3mr4934249ybt.3.1687285981208; Tue, 20 Jun 2023 11:33:01 -0700 (PDT) Date: Tue, 20 Jun 2023 11:32:46 -0700 Mime-Version: 1.0 X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230620183247.737942-1-joshdon@google.com> Subject: [PATCH v3 1/2] sched: don't account throttle time for empty groups From: Josh Don To: Peter Zijlstra Cc: Tejun Heo , Zefan Li , Johannes Weiner , Ingo Molnar , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, mkoutny@suse.com, Josh Don X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1769248958658257140?= X-GMAIL-MSGID: =?utf-8?q?1769248958658257140?= It is easy for a cfs_rq to become throttled even when it has no enqueued entities (for example, if we have just put_prev()'d the last runnable task of the cfs_rq, and the cfs_rq is out of quota). Avoid accounting this time towards total throttle time, since it otherwise falsely inflates the stats. Note that the dequeue path is special, since we normally disallow migrations when a task is in a throttled hierarchy (see throttled_lb_pair()). Signed-off-by: Josh Don --- v3: fix bug with nested limits kernel/sched/fair.c | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 6189d1a45635..72571d48559a 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4787,6 +4787,7 @@ place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int initial) } static void check_enqueue_throttle(struct cfs_rq *cfs_rq); +static inline int cfs_rq_throttled(struct cfs_rq *cfs_rq); static inline bool cfs_bandwidth_used(void); @@ -4873,8 +4874,14 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) if (cfs_rq->nr_running == 1) { check_enqueue_throttle(cfs_rq); - if (!throttled_hierarchy(cfs_rq)) + if (!throttled_hierarchy(cfs_rq)) { list_add_leaf_cfs_rq(cfs_rq); + } else { +#ifdef CONFIG_CFS_BANDWIDTH + if (cfs_rq_throttled(cfs_rq) && !cfs_rq->throttled_clock) + cfs_rq->throttled_clock = rq_clock(rq_of(cfs_rq)); +#endif + } } } @@ -5480,7 +5487,9 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq) * throttled-list. rq->lock protects completion. */ cfs_rq->throttled = 1; - cfs_rq->throttled_clock = rq_clock(rq); + SCHED_WARN_ON(cfs_rq->throttled_clock); + if (cfs_rq->nr_running) + cfs_rq->throttled_clock = rq_clock(rq); return true; } @@ -5498,7 +5507,10 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) update_rq_clock(rq); raw_spin_lock(&cfs_b->lock); - cfs_b->throttled_time += rq_clock(rq) - cfs_rq->throttled_clock; + if (cfs_rq->throttled_clock) { + cfs_b->throttled_time += rq_clock(rq) - cfs_rq->throttled_clock; + cfs_rq->throttled_clock = 0; + } list_del_rcu(&cfs_rq->throttled_list); raw_spin_unlock(&cfs_b->lock); From patchwork Tue Jun 20 18:32:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josh Don X-Patchwork-Id: 110656 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp3897354vqr; Tue, 20 Jun 2023 12:41:10 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5b9OIGNklPVC3x21QS+U/ZxaSz89wprjK9OSzGWUiGSsQUjIg5HJPk7kXy4qvOfcwGRb3I X-Received: by 2002:a17:903:18e:b0:1b3:f8f4:5ff with SMTP id z14-20020a170903018e00b001b3f8f405ffmr3291609plg.69.1687290049286; Tue, 20 Jun 2023 12:40:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1687290049; cv=none; d=google.com; s=arc-20160816; b=zWTvP6oxIvlh0N3Xfkw1FdvcKf+tzmx0dpTv/RyBddimwiRUMGs1MCPGqC11mQYL4C t6HLR9ZPC180FtT1WmqmfiHeRY3HgT4suCrE95wuKYxLUOYDFjJY8KaFTIdLIkXH4uHR nn/c1vepVG0ritXqeM+yH9BvqK+1/W25w69pZ3GSjfBsjK1P7Xrq4F/tqXMYl81oFYB3 KvE88cViEwrfflXesVMgPg6eecPNjivxEgwnn9uYn3DdNh45i2DBItAO59uh5gaKoxVu wvKWrI1/sji0OMrA43n+QmkLQ8i5LG+CGBHuxZtJ0xMRq1DJILkCOwXKlrdu0mz8bw0l mtlg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=RCaOAN1lAgFX63nyqwqYs4paj62tn+2HK5slSbPJUks=; b=FA0qIy7l47eWHUwmfBIfykIISFseFP0/H6q045F7mHHv9FWcF32ZAflTko5gkYQFKs vPg99NixnRDqyL1U/sIAz7m8g9g+oY+vNhWAf6Du6+wC87+5MUtf9pWW77UzdQHbqsNn nMgKdTo3qtDJvP8HRDOeu8rgw4hyi6b5VXkpT8LuTxcVhdDamfgdb7zcfVOuyWZ7vaor jHYL4M+3G3cUKmLa1Iac/5NgXokTYPW8CMO//RkRS+gUlb2YvVdZnH1de9Fto0Xt2kvv JxdGr0dLv3UzDGPVwcyWPitZdHJ45GmQoA+uZOmfK+9XbOD/eIOvmz4CfEGmmdf2AKps TXpQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=Inu+37zq; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a7-20020a170902ecc700b001b694ecf48fsi458088plh.71.2023.06.20.12.40.36; Tue, 20 Jun 2023 12:40:49 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=Inu+37zq; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229864AbjFTSdQ (ORCPT + 99 others); Tue, 20 Jun 2023 14:33:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60284 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229665AbjFTSdO (ORCPT ); Tue, 20 Jun 2023 14:33:14 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6F1E810CE for ; Tue, 20 Jun 2023 11:33:05 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-bce829aca1bso5447812276.3 for ; Tue, 20 Jun 2023 11:33:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687285984; x=1689877984; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=RCaOAN1lAgFX63nyqwqYs4paj62tn+2HK5slSbPJUks=; b=Inu+37zqf+xqVO90CaUGqjRGPqQl9Wdhyj9ZNMYgA8LimfGK0MuzHhG0c9J01rXMnZ sCOlDztWnQa/zeQ7WinObNQviCcQC2hTDAeA3mlIa2T/UOf5kIpu+zfTQLrTGXOAnbu3 yuhnxbaozl/KopC8Fb4KpMnjCtSphJIlpzwiHjK3JzFF7uK1J3WK4Gk07Qp+lxY4IhuT czS35YL1cKqJLd9/bpO8k0cklSaM7ZUTNmZLJIABfFSWEgXolY4ZC6wihqGQ7Q/spMol EEay8r9zsoMhavsJXVl6sE6+CS8SuBGgbbu/lefYpY/Qgt88tbaqnV92ZkVeet5eJrNz e8Aw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687285984; x=1689877984; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=RCaOAN1lAgFX63nyqwqYs4paj62tn+2HK5slSbPJUks=; b=f2OVHRyqDCZUrgSrqxJAbzA2FHP8zyIUxCTp8c929rvYvuJW4TzDnpyxKN3UUXWzG8 84zuIIXfHo0UZxtKDHKnjtKMzWxs+0EdEW6GBOubr98uYzQfscP8GMuhj5XreHAJFAw0 1/3/3G3Q9ECJktZ98EvIxOgUuEd56kceFD2txFZzyqZOGlLo/b9Yrb2+1qE4pA1OmPeq N3iVdr2X5HDwwH5BbW1567oovZKqDe9nIma3Z47RYb4kmVzusaZHPs9vQ6oQfqHVvoC1 Db7zhDZw1ma2rRh+rNURIGGq8g8+adnUNPknIgZorMF15tUiYPX0g9uHrVBwgsouq6gK 3AhA== X-Gm-Message-State: AC+VfDwkygD5JPXCOmTXDvztmyxpVNxcC3UanHlimRDJrgLri8Haw0VB Uub8dYduHJ2wAaBGk4EzohIwWyFxxk83 X-Received: from joshdon-desktop.svl.corp.google.com ([2620:15c:2a3:200:7f30:1fa1:c3c6:d39a]) (user=joshdon job=sendgmr) by 2002:a25:418f:0:b0:bea:918f:2ef1 with SMTP id o137-20020a25418f000000b00bea918f2ef1mr1201669yba.7.1687285984733; Tue, 20 Jun 2023 11:33:04 -0700 (PDT) Date: Tue, 20 Jun 2023 11:32:47 -0700 In-Reply-To: <20230620183247.737942-1-joshdon@google.com> Mime-Version: 1.0 References: <20230620183247.737942-1-joshdon@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230620183247.737942-2-joshdon@google.com> Subject: [PATCH v3 2/2] sched: add throttled time stat for throttled children From: Josh Don To: Peter Zijlstra Cc: Tejun Heo , Zefan Li , Johannes Weiner , Ingo Molnar , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, mkoutny@suse.com, Josh Don X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1769251850682909960?= X-GMAIL-MSGID: =?utf-8?q?1769251850682909960?= We currently export the total throttled time for cgroups that are given a bandwidth limit. This patch extends this accounting to also account the total time that each children cgroup has been throttled. This is useful to understand the degree to which children have been affected by the throttling control. Children which are not runnable during the entire throttled period, for example, will not show any self-throttling time during this period. Expose this in a new interface, 'cpu.stat.local', which is similar to how non-hierarchical events are accounted in 'memory.events.local'. Signed-off-by: Josh Don Acked-by: Tejun Heo --- v3: fix bug with nested limits v2: - moved export to new cpu.stat.local file, per Tejun's recommendation include/linux/cgroup-defs.h | 2 ++ kernel/cgroup/cgroup.c | 34 ++++++++++++++++++++++++++++ kernel/sched/core.c | 44 +++++++++++++++++++++++++++++++++++++ kernel/sched/fair.c | 21 +++++++++++++++++- kernel/sched/sched.h | 2 ++ 5 files changed, 102 insertions(+), 1 deletion(-) diff --git a/include/linux/cgroup-defs.h b/include/linux/cgroup-defs.h index 8a0d5466c7be..ae20dbb885d6 100644 --- a/include/linux/cgroup-defs.h +++ b/include/linux/cgroup-defs.h @@ -661,6 +661,8 @@ struct cgroup_subsys { void (*css_rstat_flush)(struct cgroup_subsys_state *css, int cpu); int (*css_extra_stat_show)(struct seq_file *seq, struct cgroup_subsys_state *css); + int (*css_local_stat_show)(struct seq_file *seq, + struct cgroup_subsys_state *css); int (*can_attach)(struct cgroup_taskset *tset); void (*cancel_attach)(struct cgroup_taskset *tset); diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c index 2d3d13e52333..bfe063225d3b 100644 --- a/kernel/cgroup/cgroup.c +++ b/kernel/cgroup/cgroup.c @@ -3726,6 +3726,36 @@ static int cpu_stat_show(struct seq_file *seq, void *v) return ret; } +static int __maybe_unused cgroup_local_stat_show(struct seq_file *seq, + struct cgroup *cgrp, int ssid) +{ + struct cgroup_subsys *ss = cgroup_subsys[ssid]; + struct cgroup_subsys_state *css; + int ret; + + if (!ss->css_local_stat_show) + return 0; + + css = cgroup_tryget_css(cgrp, ss); + if (!css) + return 0; + + ret = ss->css_local_stat_show(seq, css); + css_put(css); + return ret; +} + +static int cpu_local_stat_show(struct seq_file *seq, void *v) +{ + struct cgroup __maybe_unused *cgrp = seq_css(seq)->cgroup; + int ret = 0; + +#ifdef CONFIG_CGROUP_SCHED + ret = cgroup_local_stat_show(seq, cgrp, cpu_cgrp_id); +#endif + return ret; +} + #ifdef CONFIG_PSI static int cgroup_io_pressure_show(struct seq_file *seq, void *v) { @@ -5276,6 +5306,10 @@ static struct cftype cgroup_base_files[] = { .name = "cpu.stat", .seq_show = cpu_stat_show, }, + { + .name = "cpu.stat.local", + .seq_show = cpu_local_stat_show, + }, { } /* terminate */ }; diff --git a/kernel/sched/core.c b/kernel/sched/core.c index d9a04faa48a1..0de483190a57 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -11140,6 +11140,27 @@ static int cpu_cfs_stat_show(struct seq_file *sf, void *v) return 0; } + +static u64 throttled_time_self(struct task_group *tg) +{ + int i; + u64 total = 0; + + for_each_possible_cpu(i) { + total += READ_ONCE(tg->cfs_rq[i]->throttled_clock_self_time); + } + + return total; +} + +static int cpu_cfs_local_stat_show(struct seq_file *sf, void *v) +{ + struct task_group *tg = css_tg(seq_css(sf)); + + seq_printf(sf, "throttled_time %llu\n", throttled_time_self(tg)); + + return 0; +} #endif /* CONFIG_CFS_BANDWIDTH */ #endif /* CONFIG_FAIR_GROUP_SCHED */ @@ -11216,6 +11237,10 @@ static struct cftype cpu_legacy_files[] = { .name = "stat", .seq_show = cpu_cfs_stat_show, }, + { + .name = "stat.local", + .seq_show = cpu_cfs_local_stat_show, + }, #endif #ifdef CONFIG_RT_GROUP_SCHED { @@ -11272,6 +11297,24 @@ static int cpu_extra_stat_show(struct seq_file *sf, return 0; } +static int cpu_local_stat_show(struct seq_file *sf, + struct cgroup_subsys_state *css) +{ +#ifdef CONFIG_CFS_BANDWIDTH + { + struct task_group *tg = css_tg(css); + u64 throttled_self_usec; + + throttled_self_usec = throttled_time_self(tg); + do_div(throttled_self_usec, NSEC_PER_USEC); + + seq_printf(sf, "throttled_usec %llu\n", + throttled_self_usec); + } +#endif + return 0; +} + #ifdef CONFIG_FAIR_GROUP_SCHED static u64 cpu_weight_read_u64(struct cgroup_subsys_state *css, struct cftype *cft) @@ -11450,6 +11493,7 @@ struct cgroup_subsys cpu_cgrp_subsys = { .css_released = cpu_cgroup_css_released, .css_free = cpu_cgroup_css_free, .css_extra_stat_show = cpu_extra_stat_show, + .css_local_stat_show = cpu_local_stat_show, #ifdef CONFIG_RT_GROUP_SCHED .can_attach = cpu_cgroup_can_attach, #endif diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 72571d48559a..b1bccef78b90 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4878,8 +4878,12 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) list_add_leaf_cfs_rq(cfs_rq); } else { #ifdef CONFIG_CFS_BANDWIDTH + struct rq *rq = rq_of(cfs_rq); + if (cfs_rq_throttled(cfs_rq) && !cfs_rq->throttled_clock) - cfs_rq->throttled_clock = rq_clock(rq_of(cfs_rq)); + cfs_rq->throttled_clock = rq_clock(rq); + if (!cfs_rq->throttled_clock_self) + cfs_rq->throttled_clock_self = rq_clock(rq); #endif } } @@ -5384,6 +5388,17 @@ static int tg_unthrottle_up(struct task_group *tg, void *data) /* Add cfs_rq with load or one or more already running entities to the list */ if (!cfs_rq_is_decayed(cfs_rq)) list_add_leaf_cfs_rq(cfs_rq); + + if (cfs_rq->throttled_clock_self) { + u64 delta = rq_clock(rq) - cfs_rq->throttled_clock_self; + + cfs_rq->throttled_clock_self = 0; + + if (SCHED_WARN_ON((s64)delta < 0)) + delta = 0; + + cfs_rq->throttled_clock_self_time += delta; + } } return 0; @@ -5398,6 +5413,10 @@ static int tg_throttle_down(struct task_group *tg, void *data) if (!cfs_rq->throttle_count) { cfs_rq->throttled_clock_pelt = rq_clock_pelt(rq); list_del_leaf_cfs_rq(cfs_rq); + + SCHED_WARN_ON(cfs_rq->throttled_clock_self); + if (cfs_rq->nr_running) + cfs_rq->throttled_clock_self = rq_clock(rq); } cfs_rq->throttle_count++; diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 556496c77dc2..46f8a01ff405 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -636,6 +636,8 @@ struct cfs_rq { u64 throttled_clock; u64 throttled_clock_pelt; u64 throttled_clock_pelt_time; + u64 throttled_clock_self; + u64 throttled_clock_self_time; int throttled; int throttle_count; struct list_head throttled_list;