From patchwork Wed Nov 15 11:33:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: cruzzhao X-Patchwork-Id: 165282 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b909:0:b0:403:3b70:6f57 with SMTP id t9csp2474076vqg; Wed, 15 Nov 2023 03:34:34 -0800 (PST) X-Google-Smtp-Source: AGHT+IFgnn/pArDIUBRWmBvfV+k11cKr1jmi56nhMVDzdMFnXtM+rRwexdxNrK/CSO94H5aanwrB X-Received: by 2002:a05:6a00:1d12:b0:690:c306:151a with SMTP id a18-20020a056a001d1200b00690c306151amr11959536pfx.0.1700048073942; Wed, 15 Nov 2023 03:34:33 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700048073; cv=none; d=google.com; s=arc-20160816; b=P48vXxtJx8XFGMilMjb4VFTiDNSrwADr/5LgUQLrW0S3VwyeBcCCjUatUhkiZcuNOS yKbIl+l1gXXY+3rjh6T7pM+GQ7xS4MO/xJbrwzbXi17B5GTj0gSLMEu7PFmDIgu3XpvQ 09q16k3KDBojzRctoATNBC3m0SVhuCC3pPxIKvDwNzi6lTl1FECG+y5eEvR0oFDEhTYY UE2ecDRNIb05jwTPB6koloC75nrUZAd21EroMlSjJ7roD4HbHldPYyg4EFF2gK1+90Tk gKgfn6N5nBEU++sm4Wwqd+H8MSzAtPNmVY4nEMH8BgpYnNX3M6XOPlol+axmJrOoLW+9 en6Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=NUnhdCG+7KbFoQU9OYQBLYrhUcdPEju09Aw8MVxFNsk=; fh=QlmRtWuTYJDhaxF0GCu0nvK7RUqeTMNKD6hMg0kv7eo=; b=lK6FY/+MInfgO2EbVaMV+rTGq8jrLcTob5ZOzp/+EMxCFJcvj65p9q0gQZw9r4z+sw bZLogEmj8Oex+3jcXw8a5O80UwzcM9XrluT2OCgI8t8IxdmvcxkfmT/8B2FtMwQ/iKUD T2P/TpkmWj9jiEycYaecPvA0t+dgQ0jTZBsHjf6ICq61KxAn/1u9Ek3rk6nyvP+iWYSi 8epYgEIXENo/mFaHBGePrwlSImshmUo0PzJHxJW6Ybatj3kGMLr9goQqTd87y3tGGCnv IxCxTvBslmI7++Ra7VLXMcmyu98vbVbd159igeayjet2Cn22Wj+XUd5NvAmI4r+PxlDT uoLA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id fc5-20020a056a002e0500b006be1dd2313asi10216457pfb.210.2023.11.15.03.34.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 03:34:33 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 663CC802644A; Wed, 15 Nov 2023 03:34:21 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234939AbjKOLeA (ORCPT + 28 others); Wed, 15 Nov 2023 06:34:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41202 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229796AbjKOLd5 (ORCPT ); Wed, 15 Nov 2023 06:33:57 -0500 Received: from out30-100.freemail.mail.aliyun.com (out30-100.freemail.mail.aliyun.com [115.124.30.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5EC8ACC for ; Wed, 15 Nov 2023 03:33:54 -0800 (PST) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R161e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046059;MF=cruzzhao@linux.alibaba.com;NM=1;PH=DS;RN=12;SR=0;TI=SMTPD_---0VwSxxN2_1700048031; Received: from localhost.localdomain(mailfrom:CruzZhao@linux.alibaba.com fp:SMTPD_---0VwSxxN2_1700048031) by smtp.aliyun-inc.com; Wed, 15 Nov 2023 19:33:51 +0800 From: Cruz Zhao To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, joel@joelfernandes.org Cc: linux-kernel@vger.kernel.org Subject: [PATCH 1/4] sched/core: introduce core_id to struct rq Date: Wed, 15 Nov 2023 19:33:38 +0800 Message-Id: <20231115113341.13261-2-CruzZhao@linux.alibaba.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20231115113341.13261-1-CruzZhao@linux.alibaba.com> References: <20231115113341.13261-1-CruzZhao@linux.alibaba.com> MIME-Version: 1.0 X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Wed, 15 Nov 2023 03:34:21 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1782629609198699763 X-GMAIL-MSGID: 1782629609198699763 Introduce core_id to struct rq, indates the cpu id of the core, which is used for getting cpu id of rq->core quickly. Signed-off-by: Cruz Zhao --- kernel/sched/core.c | 16 ++++++++++++---- kernel/sched/sched.h | 1 + 2 files changed, 13 insertions(+), 4 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index a708d225c28e..7a685fae73c4 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6400,7 +6400,7 @@ static void sched_core_cpu_starting(unsigned int cpu) { const struct cpumask *smt_mask = cpu_smt_mask(cpu); struct rq *rq = cpu_rq(cpu), *core_rq = NULL; - int t; + int t, core_id; guard(core_lock)(&cpu); @@ -6417,6 +6417,7 @@ static void sched_core_cpu_starting(unsigned int cpu) rq = cpu_rq(t); if (rq->core == rq) { core_rq = rq; + core_id = t; break; } } @@ -6428,8 +6429,10 @@ static void sched_core_cpu_starting(unsigned int cpu) for_each_cpu(t, smt_mask) { rq = cpu_rq(t); - if (t == cpu) + if (t == cpu) { rq->core = core_rq; + rq->core_id = core_id; + } WARN_ON_ONCE(rq->core != core_rq); } @@ -6439,7 +6442,7 @@ static void sched_core_cpu_deactivate(unsigned int cpu) { const struct cpumask *smt_mask = cpu_smt_mask(cpu); struct rq *rq = cpu_rq(cpu), *core_rq = NULL; - int t; + int t, core_id; guard(core_lock)(&cpu); @@ -6458,6 +6461,7 @@ static void sched_core_cpu_deactivate(unsigned int cpu) if (t == cpu) continue; core_rq = cpu_rq(t); + core_id = t; break; } @@ -6483,6 +6487,7 @@ static void sched_core_cpu_deactivate(unsigned int cpu) for_each_cpu(t, smt_mask) { rq = cpu_rq(t); rq->core = core_rq; + rq->core_id = core_id; } } @@ -6490,8 +6495,10 @@ static inline void sched_core_cpu_dying(unsigned int cpu) { struct rq *rq = cpu_rq(cpu); - if (rq->core != rq) + if (rq->core != rq) { rq->core = rq; + rq->core_id = cpu; + } } #else /* !CONFIG_SCHED_CORE */ @@ -10008,6 +10015,7 @@ void __init sched_init(void) #ifdef CONFIG_SCHED_CORE rq->core = rq; + rq->core_id = i; rq->core_pick = NULL; rq->core_enabled = 0; rq->core_tree = RB_ROOT; diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 2e5a95486a42..1b62165fc840 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1121,6 +1121,7 @@ struct rq { #ifdef CONFIG_SCHED_CORE /* per rq */ struct rq *core; + unsigned int core_id; struct task_struct *core_pick; unsigned int core_enabled; unsigned int core_sched_seq; From patchwork Wed Nov 15 11:33:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: cruzzhao X-Patchwork-Id: 165283 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b909:0:b0:403:3b70:6f57 with SMTP id t9csp2474119vqg; Wed, 15 Nov 2023 03:34:41 -0800 (PST) X-Google-Smtp-Source: AGHT+IGIJe7yFN2bEVZz8ESEnTeDpq772LOHu1z6dzZUrQpqODMRxAoVLW6+YELsr2grzcWdlSxP X-Received: by 2002:a05:6808:13c7:b0:3a3:e638:78d2 with SMTP id d7-20020a05680813c700b003a3e63878d2mr14874483oiw.12.1700048081018; Wed, 15 Nov 2023 03:34:41 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700048080; cv=none; d=google.com; s=arc-20160816; b=h2JIuS+kN0YU7Xhuymc45UsOYA9xySSp9dFqulHCL8/x1I1JI28h/WWGCROCB+yz7u aVAXL2OLSzgPp2COtIdDzEgT9ioi00mPHGg/y4GmStSQxXgvxIQS27r+LPzk7iABRIWy OsudymyOjGipqSV9hc56jZ0jGGsjcJQI2bNyA4nET8EUxj3cDL/gcBg0UyUqLOQHS80b 84FUNkF2pGHK4bs0/CgTDTHcje7UhYAVgOUTRiHd4Lqiyhb8y95nbnGrWafEF+Lu4/Ix SXMoXpLNYv9lN5eouFv5u5oq18adFm0kFfLOF4oxSeBsaM8sH47UaEe2bQN5uiPcfvRk +gIQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=A1KlAhQNPumT0YG8taAGyNgkIK9PUlK6jEwnMeYE2aA=; fh=QlmRtWuTYJDhaxF0GCu0nvK7RUqeTMNKD6hMg0kv7eo=; b=onzq3dHDNQMC14o3xv1E9Sa5xeKP4GOIuWDRLXSZaZdQ73uUCZaq6hAyetQteQXrN6 Zkwa/9MDc/qDmQ/Au8lZJoxYTV4GLOwQaFFGdAhAF9CCSK46/uXJzkB38rKQ35Usb7MZ jQQEW/7hjIx+fR3nA89cq3XGIiIThnQsxtg5ynnLeSQWvIz7mN187R77sk0/lgKi+W23 /LWDVK1QRKdiGfFYJiuC7vsSj9VyHAxzLNVCmmm4Avg9pi1wOyP9XC6i8FH4KEetS/41 0eb1tZPuZiRiUHRmOhkBOv1RGgmvvDh6HT0pgTFZ+GiYt457mQ/tkFgAUpvT0i6mKSeN yYug== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from fry.vger.email (fry.vger.email. [23.128.96.38]) by mx.google.com with ESMTPS id l189-20020a6388c6000000b005b8eaeabf49si10152488pgd.178.2023.11.15.03.34.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 03:34:40 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) client-ip=23.128.96.38; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 2040F801C19D; Wed, 15 Nov 2023 03:34:32 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234948AbjKOLeC (ORCPT + 28 others); Wed, 15 Nov 2023 06:34:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41216 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234680AbjKOLd6 (ORCPT ); Wed, 15 Nov 2023 06:33:58 -0500 Received: from out30-124.freemail.mail.aliyun.com (out30-124.freemail.mail.aliyun.com [115.124.30.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BCB9AE9 for ; Wed, 15 Nov 2023 03:33:54 -0800 (PST) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R611e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046056;MF=cruzzhao@linux.alibaba.com;NM=1;PH=DS;RN=12;SR=0;TI=SMTPD_---0VwSxxNQ_1700048031; Received: from localhost.localdomain(mailfrom:CruzZhao@linux.alibaba.com fp:SMTPD_---0VwSxxNQ_1700048031) by smtp.aliyun-inc.com; Wed, 15 Nov 2023 19:33:52 +0800 From: Cruz Zhao To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, joel@joelfernandes.org Cc: linux-kernel@vger.kernel.org Subject: [PATCH 2/4] sched/core: introduce core to struct cfs_rq Date: Wed, 15 Nov 2023 19:33:39 +0800 Message-Id: <20231115113341.13261-3-CruzZhao@linux.alibaba.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20231115113341.13261-1-CruzZhao@linux.alibaba.com> References: <20231115113341.13261-1-CruzZhao@linux.alibaba.com> MIME-Version: 1.0 X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Wed, 15 Nov 2023 03:34:32 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1782629616638335882 X-GMAIL-MSGID: 1782629616638335882 Introduce core to struct cfs_rq, indicates the corresponding cfs_rq of rq->core. Signed-off-by: Cruz Zhao --- kernel/sched/core.c | 4 ++++ kernel/sched/fair.c | 11 +++++++++++ kernel/sched/sched.h | 1 + 3 files changed, 16 insertions(+) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 7a685fae73c4..647a12af9172 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6432,6 +6432,7 @@ static void sched_core_cpu_starting(unsigned int cpu) if (t == cpu) { rq->core = core_rq; rq->core_id = core_id; + rq->cfs.core = &core_rq->cfs; } WARN_ON_ONCE(rq->core != core_rq); @@ -6488,6 +6489,7 @@ static void sched_core_cpu_deactivate(unsigned int cpu) rq = cpu_rq(t); rq->core = core_rq; rq->core_id = core_id; + rq->cfs.core = &core_rq->cfs; } } @@ -6498,6 +6500,7 @@ static inline void sched_core_cpu_dying(unsigned int cpu) if (rq->core != rq) { rq->core = rq; rq->core_id = cpu; + rq->cfs.core = &rq->cfs; } } @@ -10016,6 +10019,7 @@ void __init sched_init(void) #ifdef CONFIG_SCHED_CORE rq->core = rq; rq->core_id = i; + rq->cfs.core = &rq->cfs; rq->core_pick = NULL; rq->core_enabled = 0; rq->core_tree = RB_ROOT; diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 2048138ce54b..61cbaa3cc385 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -12420,6 +12420,16 @@ bool cfs_prio_less(const struct task_struct *a, const struct task_struct *b, return delta > 0; } +void sched_core_init_cfs_rq(struct task_group *tg, struct cfs_rq *cfs_rq) +{ +#ifdef CONFIG_FAIR_GROUP_SCHED + struct rq *rq = rq_of(cfs_rq); + int core_id = rq->core_id; + + cfs_rq->core = tg->cfs_rq[core_id]; +#endif +} + static int task_is_throttled_fair(struct task_struct *p, int cpu) { struct cfs_rq *cfs_rq; @@ -12715,6 +12725,7 @@ int alloc_fair_sched_group(struct task_group *tg, struct task_group *parent) init_cfs_rq(cfs_rq); init_tg_cfs_entry(tg, cfs_rq, se, i, parent->se[i]); + sched_core_init_cfs_rq(tg, cfs_rq); init_entity_runnable_average(se); } diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 1b62165fc840..62fca54223a1 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -547,6 +547,7 @@ struct cfs_rq { #ifdef CONFIG_SCHED_CORE unsigned int forceidle_seq; u64 min_vruntime_fi; + struct cfs_rq *core; #endif #ifndef CONFIG_64BIT From patchwork Wed Nov 15 11:33:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: cruzzhao X-Patchwork-Id: 165286 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b909:0:b0:403:3b70:6f57 with SMTP id t9csp2474883vqg; Wed, 15 Nov 2023 03:36:21 -0800 (PST) X-Google-Smtp-Source: AGHT+IF+l6ZAyT1ZZGb63UBvcYVoLxyk8PKxkphiWmHg71SoclpkjBLa+oODyr4/uD0W6KQfLQvu X-Received: by 2002:a05:6a00:2d82:b0:6c3:45bc:41f8 with SMTP id fb2-20020a056a002d8200b006c345bc41f8mr12092212pfb.33.1700048181479; Wed, 15 Nov 2023 03:36:21 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700048181; cv=none; d=google.com; s=arc-20160816; b=Z1R0P/Ge0ErenxcAx1fpiFPkOJjDMVb2Ifse2vUppOnmezzqE5vVHFnjuVd4Volxt7 JOJJ166X4QzsKofZbao1SIKSzdlWyFlmXM+bgk8TBcOtAjc16GoM1RDT0FTvfyxJARyE G22UgsnJuW3oAYAkllYpzzezi1kQXgvIryst4kp88MOqEPyrmSWY125uQtRyYICcPxCX EVw34K2OiSEmt+nspzRnX4/kY9hNOWR3OiLPvhGQM+gKp9f6X5lS4U04i+gs+2lWz6Td AG1SPVBJ0ESxv7nc0/xct9b5zAhSfvBSNZ2+DdeEKZIajaclkpJg4Ix+LVgnHjGmJvC7 IBNg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=uaycZAxoOJkvDTnIxYpCXWAhUOBGd20vZXyAVgu3ZtQ=; fh=QlmRtWuTYJDhaxF0GCu0nvK7RUqeTMNKD6hMg0kv7eo=; b=sKSLjDO1rC9HYOc7Hz+Eh6u2UPRghJWko0QmTa1xEkkwgYM4zA3ymJTdwvTCFlbp5H Q36+MXY9Hq4hfCunPY0UeztYwTgVpyLaJcjfR7X7zTrStNGBqYVzxiLfevIoplhcLSnM In2anofVgbMx92Wjq70YMFgWIUJNKRUzBlUkxCe7dlUo7SUjSOb2r0sNs/f+1G169STY o6hXodVgcw1tY6EnBXdjCYpiPLrX69aNiFhswDXukXBhBsx5C1uml0+bUzqZHHyDyOHu i0cJzDJO+3fDSYccEnctiujMBdZw10k46DAQcxdXE7M+BQsYlmeTkYlQuxuIwjKTHzeG fqnQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id s12-20020a056a00194c00b006bddac4b018si10144196pfk.107.2023.11.15.03.36.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 03:36:21 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 7CB7A80C2545; Wed, 15 Nov 2023 03:35:24 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234954AbjKOLeG (ORCPT + 28 others); Wed, 15 Nov 2023 06:34:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41224 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234922AbjKOLd6 (ORCPT ); Wed, 15 Nov 2023 06:33:58 -0500 Received: from out30-133.freemail.mail.aliyun.com (out30-133.freemail.mail.aliyun.com [115.124.30.133]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0F7CD11C for ; Wed, 15 Nov 2023 03:33:54 -0800 (PST) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R191e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045176;MF=cruzzhao@linux.alibaba.com;NM=1;PH=DS;RN=12;SR=0;TI=SMTPD_---0VwSxxNu_1700048032; Received: from localhost.localdomain(mailfrom:CruzZhao@linux.alibaba.com fp:SMTPD_---0VwSxxNu_1700048032) by smtp.aliyun-inc.com; Wed, 15 Nov 2023 19:33:52 +0800 From: Cruz Zhao To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, joel@joelfernandes.org Cc: linux-kernel@vger.kernel.org Subject: [PATCH 3/4] sched/fair: introduce core_vruntime and core_min_vruntime Date: Wed, 15 Nov 2023 19:33:40 +0800 Message-Id: <20231115113341.13261-4-CruzZhao@linux.alibaba.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20231115113341.13261-1-CruzZhao@linux.alibaba.com> References: <20231115113341.13261-1-CruzZhao@linux.alibaba.com> MIME-Version: 1.0 X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Wed, 15 Nov 2023 03:35:24 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1782629721794731989 X-GMAIL-MSGID: 1782629721794731989 To compare the priority of sched_entity from different cpus of a core, we introduce core_vruntime to struct sched_entity and core_min_vruntime to struct cfs_rq. cfs_rq->core->core_min_vruntime records the min vruntime of the cfs_rqs of the same task_group among the core, and se->core_vruntime is the vruntime relative to se->cfs_rq->core->core_min_vruntime. Signed-off-by: Cruz Zhao --- include/linux/sched.h | 3 +++ kernel/sched/fair.c | 52 ++++++++++++++++++++++++++++++++++++++----- kernel/sched/sched.h | 1 + 3 files changed, 51 insertions(+), 5 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 292c31697248..df481a8ebc07 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -562,6 +562,9 @@ struct sched_entity { u64 sum_exec_runtime; u64 prev_sum_exec_runtime; u64 vruntime; +#ifdef CONFIG_SCHED_CORE + u64 core_vruntime; +#endif s64 vlag; u64 slice; diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 61cbaa3cc385..60b2fd437474 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -750,30 +750,58 @@ static u64 __update_min_vruntime(struct cfs_rq *cfs_rq, u64 vruntime) return min_vruntime; } +#ifdef CONFIG_SCHED_CORE +static u64 __update_core_min_vruntime(struct cfs_rq *cfs_rq, u64 vruntime) +{ + u64 min_vruntime = cfs_rq->core_min_vruntime; + s64 delta = (s64)(vruntime - min_vruntime); + + return delta > 0 ? vruntime : min_vruntime; +} +#endif + static void update_min_vruntime(struct cfs_rq *cfs_rq) { struct sched_entity *se = __pick_first_entity(cfs_rq); struct sched_entity *curr = cfs_rq->curr; u64 vruntime = cfs_rq->min_vruntime; +#ifdef CONFIG_SCHED_CORE + u64 core_vruntime = cfs_rq->core->min_vruntime; +#endif if (curr) { - if (curr->on_rq) + if (curr->on_rq) { vruntime = curr->vruntime; - else +#ifdef CONFIG_SCHED_CORE + core_vruntime = curr->core_vruntime; +#endif + } else { curr = NULL; + } } if (se) { - if (!curr) + if (!curr) { vruntime = se->vruntime; - else +#ifdef CONFIG_SCHED_CORE + core_vruntime = se->core_vruntime; +#endif + } else { vruntime = min_vruntime(vruntime, se->vruntime); +#ifdef CONFIG_SCHED_CORE + core_vruntime = min_vruntime(core_vruntime, se->core_vruntime); +#endif + } } /* ensure we never gain time by being placed backwards. */ u64_u32_store(cfs_rq->min_vruntime, __update_min_vruntime(cfs_rq, vruntime)); +#ifdef CONFIG_SCHED_CORE + u64_u32_store(cfs_rq->core->core_min_vruntime, + __update_core_min_vruntime(cfs_rq->core, vruntime)); +#endif } static inline bool __entity_less(struct rb_node *a, const struct rb_node *b) @@ -1137,6 +1165,7 @@ static void update_curr(struct cfs_rq *cfs_rq) struct sched_entity *curr = cfs_rq->curr; u64 now = rq_clock_task(rq_of(cfs_rq)); u64 delta_exec; + u64 delta_exec_fair; if (unlikely(!curr)) return; @@ -1158,7 +1187,11 @@ static void update_curr(struct cfs_rq *cfs_rq) curr->sum_exec_runtime += delta_exec; schedstat_add(cfs_rq->exec_clock, delta_exec); - curr->vruntime += calc_delta_fair(delta_exec, curr); + delta_exec_fair = calc_delta_fair(delta_exec, curr); + curr->vruntime += delta_exec_fair; +#ifdef CONFIG_SCHED_CORE + curr->core_vruntime += delta_exec_fair; +#endif update_deadline(cfs_rq, curr); update_min_vruntime(cfs_rq); @@ -5009,6 +5042,9 @@ static void place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) { u64 vslice, vruntime = avg_vruntime(cfs_rq); +#ifdef CONFIG_SCHED_CORE + u64 core_vruntime = cfs_rq->core->core_min_vruntime + vruntime - cfs_rq->min_vruntime; +#endif s64 lag = 0; se->slice = sysctl_sched_base_slice; @@ -5091,6 +5127,9 @@ place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) } se->vruntime = vruntime - lag; +#ifdef CONFIG_SCHED_CORE + se->core_vruntime = core_vruntime - lag; +#endif /* * When joining the competition; the exisiting tasks will be, @@ -12655,6 +12694,9 @@ void init_cfs_rq(struct cfs_rq *cfs_rq) { cfs_rq->tasks_timeline = RB_ROOT_CACHED; u64_u32_store(cfs_rq->min_vruntime, (u64)(-(1LL << 20))); +#ifdef CONFIG_SCHED_CORE + u64_u32_store(cfs_rq->core_min_vruntime, (u64)(-(1LL << 20))); +#endif #ifdef CONFIG_SMP raw_spin_lock_init(&cfs_rq->removed.lock); #endif diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 62fca54223a1..f9d3701481f1 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -545,6 +545,7 @@ struct cfs_rq { u64 exec_clock; u64 min_vruntime; #ifdef CONFIG_SCHED_CORE + u64 core_min_vruntime; unsigned int forceidle_seq; u64 min_vruntime_fi; struct cfs_rq *core; From patchwork Wed Nov 15 11:33:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: cruzzhao X-Patchwork-Id: 165284 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b909:0:b0:403:3b70:6f57 with SMTP id t9csp2474171vqg; Wed, 15 Nov 2023 03:34:48 -0800 (PST) X-Google-Smtp-Source: AGHT+IHhkXfVXpKtN0PMg/S/py46+Rr7mCfc4klqvtIV/vnDFTUNv1IYq97BtBLISMGRx8/J/A/8 X-Received: by 2002:a05:6830:c5:b0:6d6:5577:4922 with SMTP id x5-20020a05683000c500b006d655774922mr5051357oto.35.1700048088025; Wed, 15 Nov 2023 03:34:48 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700048087; cv=none; d=google.com; s=arc-20160816; b=pEvbLIWOH28m+KSWWhP3EELNiIaZtYLnFS47Qffydyi4YoE8NplH2QqsQJJj9M3k8x NRrdHLrb10EEArepA5d9eIvcGkEBJ3czjby1igStIvfLRx1xo/6x2RJ/RHuTcd3RQLxg PsFD3QSs2hWmJXOtQmGaIKGUTwWb18/w6G8lt4GO98Rudcnakzr9ssSkvTIJTIMheN5f phxm1QZmvqUx4gN+dhFrcwbkVxUY3QZX6lPVvF/sKC2c6sJsoWN/G0nLWkrLiFBpIo0c bPFDAO86cG90ETX0aDNrWLyk7lToswsHsfekWZI5sYIftwYLTxX47AVVJbvGU9JJhOcO 799w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=kwMYU9ctakzbaK1/FBL50k7Kw4tkHQ2eQOkyaySJmHo=; fh=QlmRtWuTYJDhaxF0GCu0nvK7RUqeTMNKD6hMg0kv7eo=; b=UQWeJDvzc0s6imEYb81aUNoCBzjt15W/i4YV0QmrI4IEYaUFK5/h3ogGgLK7n/lgUO FZquEVFV6H3KwtSNqV1SKcRScJT9IIiGaffCfxjmO5oAJt6i0cofKOl1eoqRGAU5JQ0D mH3ZAYzx1rJr0y6S2OGwECriuXcS0tPTHdXDx8s72cGrJq/za4ugeQOOLiCVSv2dMT0t xWaBibUMrjJ/9zAF/SV2EXNGVVPxaZUphcHZywfV8lPbh+I8f15s+DtU+mgXn0GM+Q5R 9lYE3ShsaHa4HvUbHHud3/CSbH4Qq2TWPb5pzIs8qTJvXKEstOJa6l9T/JrbZRz0TYKB 44RQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id q19-20020a632a13000000b005b8f0ff9b35si10125284pgq.406.2023.11.15.03.34.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 03:34:47 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id E85AD80286AF; Wed, 15 Nov 2023 03:34:44 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234978AbjKOLeI (ORCPT + 28 others); Wed, 15 Nov 2023 06:34:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41240 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234934AbjKOLd7 (ORCPT ); Wed, 15 Nov 2023 06:33:59 -0500 Received: from out30-113.freemail.mail.aliyun.com (out30-113.freemail.mail.aliyun.com [115.124.30.113]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ED369CC for ; Wed, 15 Nov 2023 03:33:55 -0800 (PST) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R841e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045170;MF=cruzzhao@linux.alibaba.com;NM=1;PH=DS;RN=12;SR=0;TI=SMTPD_---0VwSxxO8_1700048033; Received: from localhost.localdomain(mailfrom:CruzZhao@linux.alibaba.com fp:SMTPD_---0VwSxxO8_1700048033) by smtp.aliyun-inc.com; Wed, 15 Nov 2023 19:33:53 +0800 From: Cruz Zhao To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, joel@joelfernandes.org Cc: linux-kernel@vger.kernel.org Subject: [PATCH 4/4] sched/core: fix cfs_prio_less Date: Wed, 15 Nov 2023 19:33:41 +0800 Message-Id: <20231115113341.13261-5-CruzZhao@linux.alibaba.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20231115113341.13261-1-CruzZhao@linux.alibaba.com> References: <20231115113341.13261-1-CruzZhao@linux.alibaba.com> MIME-Version: 1.0 X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Wed, 15 Nov 2023 03:34:45 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1782629598665677537 X-GMAIL-MSGID: 1782629623650718885 The update of vruntime snapshot will cause unfair sched, especially when tasks enqueue/dequeue frequently. Consider the following case: - Task A1 and A2 share a cookie, and task B has another cookie. - A1 is a short task, waking up frequently but running short everytime. - A2 and B are long tasks. - A1 and B runs on ht0 and A2 runs on ht1. ht0 ht1 fi_before fi update switch to A1 switch to A2 0 0 1 A1 sleeps switch to B A2 force idle 0 1 1 A1 wakes up switch to A1 switch to A1 1 0 1 A1 sleeps switch to B A2 force idle 0 1 1 In this case, cfs_rq->min_vruntime_fi will update every schedule, and prio of B and A2 will be pulled to the same level, no matter how long A2 and B have run before, which is not fair enough. Extramely, we observed that the latency of a task became several minutes due to this reason, which should be 100ms. To fix this problem, we compare the priority of ses using core_vruntime directly, instead of vruntime snapshot. Fixes: c6047c2e3af6 ("sched/fair: Snapshot the min_vruntime of CPUs on force idle") Signed-off-by: Cruz Zhao --- kernel/sched/core.c | 17 ----------------- kernel/sched/fair.c | 35 +---------------------------------- kernel/sched/sched.h | 2 -- 3 files changed, 1 insertion(+), 53 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 647a12af9172..22edf4bcc7e8 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6052,8 +6052,6 @@ static inline struct task_struct *pick_task(struct rq *rq) BUG(); /* The idle class should always have a runnable task. */ } -extern void task_vruntime_update(struct rq *rq, struct task_struct *p, bool in_fi); - static void queue_core_balance(struct rq *rq); static struct task_struct * @@ -6154,7 +6152,6 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) * unconstrained picks as well. */ WARN_ON_ONCE(fi_before); - task_vruntime_update(rq, next, false); goto out_set_next; } } @@ -6204,8 +6201,6 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) if (p == rq_i->idle) { if (rq_i->nr_running) { rq->core->core_forceidle_count++; - if (!fi_before) - rq->core->core_forceidle_seq++; } } else { occ++; @@ -6245,17 +6240,6 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) if (!rq_i->core_pick) continue; - /* - * Update for new !FI->FI transitions, or if continuing to be in !FI: - * fi_before fi update? - * 0 0 1 - * 0 1 1 - * 1 0 1 - * 1 1 0 - */ - if (!(fi_before && rq->core->core_forceidle_count)) - task_vruntime_update(rq_i, rq_i->core_pick, !!rq->core->core_forceidle_count); - rq_i->core_pick->core_occupation = occ; if (i == cpu) { @@ -6474,7 +6458,6 @@ static void sched_core_cpu_deactivate(unsigned int cpu) core_rq->core_pick_seq = rq->core_pick_seq; core_rq->core_cookie = rq->core_cookie; core_rq->core_forceidle_count = rq->core_forceidle_count; - core_rq->core_forceidle_seq = rq->core_forceidle_seq; core_rq->core_forceidle_occupation = rq->core_forceidle_occupation; /* diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 60b2fd437474..15c350b7c34a 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -12382,35 +12382,6 @@ static inline void task_tick_core(struct rq *rq, struct task_struct *curr) resched_curr(rq); } -/* - * se_fi_update - Update the cfs_rq->min_vruntime_fi in a CFS hierarchy if needed. - */ -static void se_fi_update(const struct sched_entity *se, unsigned int fi_seq, - bool forceidle) -{ - for_each_sched_entity(se) { - struct cfs_rq *cfs_rq = cfs_rq_of(se); - - if (forceidle) { - if (cfs_rq->forceidle_seq == fi_seq) - break; - cfs_rq->forceidle_seq = fi_seq; - } - - cfs_rq->min_vruntime_fi = cfs_rq->min_vruntime; - } -} - -void task_vruntime_update(struct rq *rq, struct task_struct *p, bool in_fi) -{ - struct sched_entity *se = &p->se; - - if (p->sched_class != &fair_sched_class) - return; - - se_fi_update(se, rq->core->core_forceidle_seq, in_fi); -} - bool cfs_prio_less(const struct task_struct *a, const struct task_struct *b, bool in_fi) { @@ -12438,9 +12409,6 @@ bool cfs_prio_less(const struct task_struct *a, const struct task_struct *b, seb = parent_entity(seb); } - se_fi_update(sea, rq->core->core_forceidle_seq, in_fi); - se_fi_update(seb, rq->core->core_forceidle_seq, in_fi); - cfs_rqa = sea->cfs_rq; cfs_rqb = seb->cfs_rq; #else @@ -12453,8 +12421,7 @@ bool cfs_prio_less(const struct task_struct *a, const struct task_struct *b, * min_vruntime_fi, which would have been updated in prior calls * to se_fi_update(). */ - delta = (s64)(sea->vruntime - seb->vruntime) + - (s64)(cfs_rqb->min_vruntime_fi - cfs_rqa->min_vruntime_fi); + delta = (s64)(sea->core_vruntime - seb->core_vruntime); return delta > 0; } diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index f9d3701481f1..2ac89eb20973 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -546,7 +546,6 @@ struct cfs_rq { u64 min_vruntime; #ifdef CONFIG_SCHED_CORE u64 core_min_vruntime; - unsigned int forceidle_seq; u64 min_vruntime_fi; struct cfs_rq *core; #endif @@ -1134,7 +1133,6 @@ struct rq { unsigned int core_pick_seq; unsigned long core_cookie; unsigned int core_forceidle_count; - unsigned int core_forceidle_seq; unsigned int core_forceidle_occupation; u64 core_forceidle_start; #endif