From patchwork Mon Feb 20 12:41:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 59427 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1285860wrn; Mon, 20 Feb 2023 04:45:42 -0800 (PST) X-Google-Smtp-Source: AK7set9KlMge2r6IXIRyeHqY4LzTUeqQ7XGPkrnC91QSnG/GASJTdCpcmnnJxeR1SgRQb7nH4ceI X-Received: by 2002:a17:90b:4c87:b0:233:d181:5b75 with SMTP id my7-20020a17090b4c8700b00233d1815b75mr3592671pjb.31.1676897141758; Mon, 20 Feb 2023 04:45:41 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676897141; cv=none; d=google.com; s=arc-20160816; b=ApGLloZgWQLCjJBFVOH7hiH9eb9v0nKtqeB+vIvNKHV/Obu4H6g05A+ubLdEjqp8VZ N0Z4JhvSqBHvZ0amMFKXFjBf+fTjVtbfgXq9ebuCjl+c5HdZKVAt1TMo8De6O7o82f/O G9CJn5b897fV1gPp23jJCUYoeY5XJZOKjCZJV3BtAo9K3EVMWp3fDVfL5v+oazQIhOjB cpt+HlYRQlW2V6tRDarZ0LGw5RJoKUOUoFcvgjQjNMpcX+v5t8zDX3mA/0F9dolCzcOf X29zqwxk0qCo8Pn3PA5Yi0NZ5KR0lnPk/Ta+G34EWb54RdUw59lD2BYwZAbNDZa0CVn9 cEXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=egoUl+UrCX7i4gPVYmA9o3lWb9zRaE4swrZUPBYXKnY=; b=MjFJlfy3WLQdIPdHz+XfwfRgoJjTBm/qeIEr+zlJ9gaJZdLOAtz9QaNwpoQUmXPawz 0/RQAyZUCT2FwZzgGP0XVX5T4Nc67ah5F1MmJH7opBRjlHQTldrKoEaGo56uRfeRz1iq 6/pKkEHLa0GwzB+qg7dGAam7jaTr8eOwFRQBlHg1dPX5NJ1NRm3Ycuv60MDPUiaXWs1u q7Sx5aX46WEvMSSQrzh7MWJmKDkm5f/ecFP7mMmlLHsaojja2gUcspBk356Xoa/AH1ts C8qflT1t1LoAtVfTMrJVPDZm4ER8qrf6EZQjcqohM21jbggNNWfuTLFVg36wov8Sno/o Xwpg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=UJ8Yqu4T; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y1-20020a17090a134100b0022be3049e52si6925197pjf.34.2023.02.20.04.45.29; Mon, 20 Feb 2023 04:45:41 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=UJ8Yqu4T; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232129AbjBTMlv (ORCPT + 99 others); Mon, 20 Feb 2023 07:41:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60336 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232112AbjBTMlr (ORCPT ); Mon, 20 Feb 2023 07:41:47 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5A80A1C5B0 for ; Mon, 20 Feb 2023 04:41:45 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id F414DB80D1C for ; Mon, 20 Feb 2023 12:41:43 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 922B6C4339B; Mon, 20 Feb 2023 12:41:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1676896902; bh=Wy5hbT4ewIbxtuRGZo3BNmMi3nK/RbjQJn9sH1g4yFs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UJ8Yqu4TY3MNStEp3/quqlin7WksyZLlHo4C3AoWniZJ/dh/ikKCGQXs/rWPS3mNr 1B2+prIa8HCPmavA6vpDDclIuRUV3Oasst3UtDObPJo/xDlGbh5AEHo6W8i6dlx9yV o71lgevA2iwWZHH88EgcBoMAvi21YdIiqzBiSKJxX/az66jrPBfbFh/gi344T764aD R8VQSIlMOW2sKS5aX7VJZmxrqyQuYEeU+XSr9MZLHAGhwEumyNe+JO0yMtwb7nhIym MIOKDje7m1Qu9VJzy2B+T2tDuwh3Rd/zYQfk0yQ1XX6UYm3l4xWqpqpd/GuKHaU3t/ L/K40Nm3tv0tQ== From: Frederic Weisbecker To: Thomas Gleixner Cc: LKML , Frederic Weisbecker , Alexey Dobriyan , Peter Zijlstra , Wei Li , Mirsad Goran Todorovac , Yu Liao , Hillf Danton , Ingo Molnar Subject: [PATCH 2/7] timers/nohz: Only ever update sleeptime from idle exit Date: Mon, 20 Feb 2023 13:41:24 +0100 Message-Id: <20230220124129.519477-3-frederic@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230220124129.519477-1-frederic@kernel.org> References: <20230220124129.519477-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758354097216478685?= X-GMAIL-MSGID: =?utf-8?q?1758354097216478685?= The idle and io sleeptime statistics appearing in /proc/stat can be currently updated from two sites: locally on idle exit and remotely by cpufreq. However there is no synchronization mechanism protecting concurrent updates. It is therefore possible to account the sleeptime twice, among all the possible broken scenarios. To prevent from breaking the sleeptime accounting source, restrict the sleeptime updates to the local idle exit site. If there is a delta to add since the last update, IO/Idle sleep time readers will now only compute the delta without actually writing it back to the internal idle statistic fields. This fixes a writer VS writer race. Note there are still two known reader VS writer races to handle. A subsequent patch will fix one. Reported-by: Yu Liao Acked-by: Peter Zijlstra (Intel) Cc: Hillf Danton Cc: Yu Liao Cc: Ingo Molnar Cc: Thomas Gleixner Cc: Wei Li Cc: Alexey Dobriyan Cc: Mirsad Goran Todorovac Signed-off-by: Frederic Weisbecker --- kernel/time/tick-sched.c | 103 ++++++++++++++++----------------------- 1 file changed, 41 insertions(+), 62 deletions(-) diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index b0e3c9205946..9058b9eb8bc1 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -637,31 +637,21 @@ static void tick_nohz_update_jiffies(ktime_t now) touch_softlockup_watchdog_sched(); } -/* - * Updates the per-CPU time idle statistics counters - */ -static void -update_ts_time_stats(int cpu, struct tick_sched *ts, ktime_t now, u64 *last_update_time) -{ - ktime_t delta; - - if (ts->idle_active) { - delta = ktime_sub(now, ts->idle_entrytime); - if (nr_iowait_cpu(cpu) > 0) - ts->iowait_sleeptime = ktime_add(ts->iowait_sleeptime, delta); - else - ts->idle_sleeptime = ktime_add(ts->idle_sleeptime, delta); - ts->idle_entrytime = now; - } - - if (last_update_time) - *last_update_time = ktime_to_us(now); - -} - static void tick_nohz_stop_idle(struct tick_sched *ts, ktime_t now) { - update_ts_time_stats(smp_processor_id(), ts, now, NULL); + ktime_t delta; + + if (WARN_ON_ONCE(!ts->idle_active)) + return; + + delta = ktime_sub(now, ts->idle_entrytime); + + if (nr_iowait_cpu(smp_processor_id()) > 0) + ts->iowait_sleeptime = ktime_add(ts->iowait_sleeptime, delta); + else + ts->idle_sleeptime = ktime_add(ts->idle_sleeptime, delta); + + ts->idle_entrytime = now; ts->idle_active = 0; sched_clock_idle_wakeup_event(); @@ -674,6 +664,30 @@ static void tick_nohz_start_idle(struct tick_sched *ts) sched_clock_idle_sleep_event(); } +static u64 get_cpu_sleep_time_us(struct tick_sched *ts, ktime_t *sleeptime, + bool compute_delta, u64 *last_update_time) +{ + ktime_t now, idle; + + if (!tick_nohz_active) + return -1; + + now = ktime_get(); + if (last_update_time) + *last_update_time = ktime_to_us(now); + + if (ts->idle_active && compute_delta) { + ktime_t delta = ktime_sub(now, ts->idle_entrytime); + + idle = ktime_add(*sleeptime, delta); + } else { + idle = *sleeptime; + } + + return ktime_to_us(idle); + +} + /** * get_cpu_idle_time_us - get the total idle time of a CPU * @cpu: CPU number to query @@ -691,27 +705,9 @@ static void tick_nohz_start_idle(struct tick_sched *ts) u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time) { struct tick_sched *ts = &per_cpu(tick_cpu_sched, cpu); - ktime_t now, idle; - - if (!tick_nohz_active) - return -1; - - now = ktime_get(); - if (last_update_time) { - update_ts_time_stats(cpu, ts, now, last_update_time); - idle = ts->idle_sleeptime; - } else { - if (ts->idle_active && !nr_iowait_cpu(cpu)) { - ktime_t delta = ktime_sub(now, ts->idle_entrytime); - - idle = ktime_add(ts->idle_sleeptime, delta); - } else { - idle = ts->idle_sleeptime; - } - } - - return ktime_to_us(idle); + return get_cpu_sleep_time_us(ts, &ts->idle_sleeptime, + !nr_iowait_cpu(cpu), last_update_time); } EXPORT_SYMBOL_GPL(get_cpu_idle_time_us); @@ -732,26 +728,9 @@ EXPORT_SYMBOL_GPL(get_cpu_idle_time_us); u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time) { struct tick_sched *ts = &per_cpu(tick_cpu_sched, cpu); - ktime_t now, iowait; - if (!tick_nohz_active) - return -1; - - now = ktime_get(); - if (last_update_time) { - update_ts_time_stats(cpu, ts, now, last_update_time); - iowait = ts->iowait_sleeptime; - } else { - if (ts->idle_active && nr_iowait_cpu(cpu) > 0) { - ktime_t delta = ktime_sub(now, ts->idle_entrytime); - - iowait = ktime_add(ts->iowait_sleeptime, delta); - } else { - iowait = ts->iowait_sleeptime; - } - } - - return ktime_to_us(iowait); + return get_cpu_sleep_time_us(ts, &ts->iowait_sleeptime, + nr_iowait_cpu(cpu), last_update_time); } EXPORT_SYMBOL_GPL(get_cpu_iowait_time_us);