From patchwork Mon Sep 18 06:21:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: tip-bot2 for Thomas Gleixner X-Patchwork-Id: 141336 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp2546359vqi; Mon, 18 Sep 2023 03:11:43 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFhGzoqvoHqyjwW6sKBzt5Ucb1CmMrdRzbN5Ebhssfb3pvyw4xt9wyEAMGsB7ptuUozzeAz X-Received: by 2002:a05:6358:718d:b0:132:d42f:8e19 with SMTP id t13-20020a056358718d00b00132d42f8e19mr7110112rwt.31.1695031903472; Mon, 18 Sep 2023 03:11:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695031903; cv=none; d=google.com; s=arc-20160816; b=Oh9l7jfP5Qh2MBoRxGQUycpc0x9wkzKq9R1P2p4QzrK6ksNLgh9aOACVEb6AqmXINW B3AXVIJHuCGR8/U56OLHP6VPJcxuSJX92qphSCt6rY7f1X0bXq0E+Q7t6LazXU6t6IXh grnbLOs7J7wWqfzUZ9NJJE3BLKuHv6ITV33WoYarfDhzBQ5IK2YqgtqIV5+7aFsxHM5y 6tfZJhU/z4XuV37HpSsVBDNHGhrUEDb5DI1h3bAwEXo3OXT9F3pdfs+LxbMD9nW3qWeq FL/0iGAO0xGcgcb+OIl7e4vI346Fd1Ogl4sZEwcErPTAQjI1mu3KsXnHqMwj7EZXiC2H EcjQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:robot-unsubscribe :robot-id:message-id:mime-version:cc:subject:to:reply-to:sender:from :dkim-signature:dkim-signature:date; bh=/tv/cW4VFXXfUEoCJ6/DJ7tbok7IFkyfL1VEDSRPrGA=; fh=mxbxVtcUt51+XC2/hBDGJJdKEWUZAtDCzBS4GQ7L0nA=; b=yB85raC1CVCx9n2YH4hi4TeVX44JYJXKY2CE6YVqcsIGuSe2WRFLkBEO6zVItdKkQy Htd2gyYSkFC7kQYLKS/KGCChYDLorhG/P1yzMyAwTjRmEYeNzfm/QuKH9SO0qrt2LpqQ sUK8DBofN94FwOeEKFz2SjBlR6dVtkS9Zgq0PxcadmxoOKvVeOwGx+NDelFOJm6bjpHF JalzgWchCE3eFmxvzs6a/MgdGDRbR+DgUxsheqwwwO53rXdommfrhwv+BqJmv+lIUsdQ yQTipZ6f7bkvAzDoLsxUtXyDc++Dm9HDRRbgDOBTs4cv/IZ50UZJXIiYmY1PgKBTGl38 HO+Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=1js5R5mJ; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=+Vob4eVN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from morse.vger.email (morse.vger.email. [2620:137:e000::3:1]) by mx.google.com with ESMTPS id m16-20020a63fd50000000b00563fac86c55si5450216pgj.134.2023.09.18.03.11.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Sep 2023 03:11:43 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=1js5R5mJ; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=+Vob4eVN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 01605801DB05; Mon, 18 Sep 2023 00:41:23 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240008AbjIRHkb (ORCPT + 27 others); Mon, 18 Sep 2023 03:40:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49926 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240282AbjIRHkL (ORCPT ); Mon, 18 Sep 2023 03:40:11 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3342D1711; Mon, 18 Sep 2023 00:38:14 -0700 (PDT) Date: Mon, 18 Sep 2023 06:21:23 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1695022692; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=/tv/cW4VFXXfUEoCJ6/DJ7tbok7IFkyfL1VEDSRPrGA=; b=1js5R5mJ09MR6PiFc2eVBg1KyFicOxVVRuCXVru8M9l6BnmgTFqce+hoQHnGjDaG+Py/CG b65H4AJUytNizW+nA9WudS8FtbZ+kWPHIJpCbh6VQwKW3q3OxXhUJUowf27+LKW87gCSc0 3Qc15AakQH2H54UmPLwva75FnAecGUqbAnvniSjUCElhWJx/EodeJC+W7KIS7GszpDTGcT YZwJ9AYUrR+KCbDc2taJQVGenpYmljSzavpuWzl3fGWafXmkUcUgplEU1VAx/UBBxBX0mY RSKqbLfwCFnUs38zRIlQ1Mol/tjN/uqk2XYzOKyY5/1aNPodnm/6jYBByiz+hA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1695022692; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=/tv/cW4VFXXfUEoCJ6/DJ7tbok7IFkyfL1VEDSRPrGA=; b=+Vob4eVNTy4pqABzyf7WcOEMHT2jPQU9T68BJ2yX6xbWHKUACU6HnrRNs7y6w3toWL6LtU br4p+94QsLOeDaBg== From: "tip-bot2 for Elliot Berman" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: sched/core] freezer,sched: Use saved_state to reduce some spurious wakeups Cc: Prakash Viswalingam , Elliot Berman , "Peter Zijlstra (Intel)" , Ingo Molnar , x86@kernel.org, linux-kernel@vger.kernel.org MIME-Version: 1.0 Message-ID: <169501808337.27769.14753741283379327202.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Mon, 18 Sep 2023 00:41:24 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777369772973681942 X-GMAIL-MSGID: 1777369772973681942 The following commit has been merged into the sched/core branch of tip: Commit-ID: 8f0eed4a78a81668bc78923ea09f51a7a663c2b0 Gitweb: https://git.kernel.org/tip/8f0eed4a78a81668bc78923ea09f51a7a663c2b0 Author: Elliot Berman AuthorDate: Fri, 08 Sep 2023 15:49:16 -07:00 Committer: Ingo Molnar CommitterDate: Mon, 18 Sep 2023 08:14:36 +02:00 freezer,sched: Use saved_state to reduce some spurious wakeups After commit f5d39b020809 ("freezer,sched: Rewrite core freezer logic"), tasks that transition directly from TASK_FREEZABLE to TASK_FROZEN are always woken up on the thaw path. Prior to that commit, tasks could ask freezer to consider them "frozen enough" via freezer_do_not_count(). The commit replaced freezer_do_not_count() with a TASK_FREEZABLE state which allows freezer to immediately mark the task as TASK_FROZEN without waking up the task. This is efficient for the suspend path, but on the thaw path, the task is always woken up even if the task didn't need to wake up and goes back to its TASK_(UN)INTERRUPTIBLE state. Although these tasks are capable of handling of the wakeup, we can observe a power/perf impact from the extra wakeup. We observed on Android many tasks wait in the TASK_FREEZABLE state (particularly due to many of them being binder clients). We observed nearly 4x the number of tasks and a corresponding linear increase in latency and power consumption when thawing the system. The latency increased from ~15ms to ~50ms. Avoid the spurious wakeups by saving the state of TASK_FREEZABLE tasks. If the task was running before entering TASK_FROZEN state (__refrigerator()) or if the task received a wake up for the saved state, then the task is woken on thaw. saved_state from PREEMPT_RT locks can be re-used because freezer would not stomp on the rtlock wait flow: TASK_RTLOCK_WAIT isn't considered freezable. Reported-by: Prakash Viswalingam Signed-off-by: Elliot Berman Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Ingo Molnar --- kernel/freezer.c | 41 +++++++++++++++++++---------------------- kernel/sched/core.c | 23 ++++++++++++++--------- 2 files changed, 33 insertions(+), 31 deletions(-) diff --git a/kernel/freezer.c b/kernel/freezer.c index 4fad0e6..c450fa8 100644 --- a/kernel/freezer.c +++ b/kernel/freezer.c @@ -71,7 +71,11 @@ bool __refrigerator(bool check_kthr_stop) for (;;) { bool freeze; + raw_spin_lock_irq(¤t->pi_lock); set_current_state(TASK_FROZEN); + /* unstale saved_state so that __thaw_task() will wake us up */ + current->saved_state = TASK_RUNNING; + raw_spin_unlock_irq(¤t->pi_lock); spin_lock_irq(&freezer_lock); freeze = freezing(current) && !(check_kthr_stop && kthread_should_stop()); @@ -129,6 +133,7 @@ static int __set_task_frozen(struct task_struct *p, void *arg) WARN_ON_ONCE(debug_locks && p->lockdep_depth); #endif + p->saved_state = p->__state; WRITE_ONCE(p->__state, TASK_FROZEN); return TASK_FROZEN; } @@ -170,42 +175,34 @@ bool freeze_task(struct task_struct *p) } /* - * The special task states (TASK_STOPPED, TASK_TRACED) keep their canonical - * state in p->jobctl. If either of them got a wakeup that was missed because - * TASK_FROZEN, then their canonical state reflects that and the below will - * refuse to restore the special state and instead issue the wakeup. + * Restore the saved_state before the task entered freezer. For typical task + * in the __refrigerator(), saved_state == TASK_RUNNING so nothing happens + * here. For tasks which were TASK_NORMAL | TASK_FREEZABLE, their initial state + * is restored unless they got an expected wakeup (see ttwu_state_match()). + * Returns 1 if the task state was restored. */ -static int __set_task_special(struct task_struct *p, void *arg) +static int __restore_freezer_state(struct task_struct *p, void *arg) { - unsigned int state = 0; + unsigned int state = p->saved_state; - if (p->jobctl & JOBCTL_TRACED) - state = TASK_TRACED; - - else if (p->jobctl & JOBCTL_STOPPED) - state = TASK_STOPPED; - - if (state) + if (state != TASK_RUNNING) { WRITE_ONCE(p->__state, state); + return 1; + } - return state; + return 0; } void __thaw_task(struct task_struct *p) { - unsigned long flags, flags2; + unsigned long flags; spin_lock_irqsave(&freezer_lock, flags); if (WARN_ON_ONCE(freezing(p))) goto unlock; - if (lock_task_sighand(p, &flags2)) { - /* TASK_FROZEN -> TASK_{STOPPED,TRACED} */ - bool ret = task_call_func(p, __set_task_special, NULL); - unlock_task_sighand(p, &flags2); - if (ret) - goto unlock; - } + if (task_call_func(p, __restore_freezer_state, NULL)) + goto unlock; wake_up_state(p, TASK_FROZEN); unlock: diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 49541e3..5a50c4e 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -2242,8 +2242,8 @@ static __always_inline int task_state_match(struct task_struct *p, unsigned int state) { /* - * Serialize against current_save_and_set_rtlock_wait_state() and - * current_restore_rtlock_saved_state(). + * Serialize against current_save_and_set_rtlock_wait_state(), + * current_restore_rtlock_saved_state(), and __refrigerator(). */ guard(raw_spinlock_irq)(&p->pi_lock); return __task_state_match(p, state); @@ -4015,13 +4015,17 @@ static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags) * The caller holds p::pi_lock if p != current or has preemption * disabled when p == current. * - * The rules of PREEMPT_RT saved_state: + * The rules of saved_state: * * The related locking code always holds p::pi_lock when updating * p::saved_state, which means the code is fully serialized in both cases. * - * The lock wait and lock wakeups happen via TASK_RTLOCK_WAIT. No other - * bits set. This allows to distinguish all wakeup scenarios. + * For PREEMPT_RT, the lock wait and lock wakeups happen via TASK_RTLOCK_WAIT. + * No other bits set. This allows to distinguish all wakeup scenarios. + * + * For FREEZER, the wakeup happens via TASK_FROZEN. No other bits set. This + * allows us to prevent early wakeup of tasks before they can be run on + * asymmetric ISA architectures (eg ARMv9). */ static __always_inline bool ttwu_state_match(struct task_struct *p, unsigned int state, int *success) @@ -4037,10 +4041,11 @@ bool ttwu_state_match(struct task_struct *p, unsigned int state, int *success) /* * Saved state preserves the task state across blocking on - * an RT lock. If the state matches, set p::saved_state to - * TASK_RUNNING, but do not wake the task because it waits - * for a lock wakeup. Also indicate success because from - * the regular waker's point of view this has succeeded. + * an RT lock or TASK_FREEZABLE tasks. If the state matches, + * set p::saved_state to TASK_RUNNING, but do not wake the task + * because it waits for a lock wakeup or __thaw_task(). Also + * indicate success because from the regular waker's point of + * view this has succeeded. * * After acquiring the lock the task will restore p::__state * from p::saved_state which ensures that the regular