From patchwork Wed Feb 22 14:46:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 60565 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp618718wrd; Wed, 22 Feb 2023 06:53:28 -0800 (PST) X-Google-Smtp-Source: AK7set85Rgd9pU+pU9E2spMCmFfmkTAc8k7aDfZGY9ZMtICrw1mUjExTLowLHhOsNGn2XodRrLMw X-Received: by 2002:a05:6402:785:b0:4af:62ad:a36 with SMTP id d5-20020a056402078500b004af62ad0a36mr2095375edy.27.1677077608380; Wed, 22 Feb 2023 06:53:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677077608; cv=none; d=google.com; s=arc-20160816; b=yb6a/oZWPyVRNdGihK+p5i3N2sZcBWdxD0bLoJViJPwXtrDVn3bX8WxxoXQgpQsGRu BcVDHDVusKj0PGh4d+OH0Cpwcwach7TXQOaGkdEYcAZNubV2y8YVahjOy10DsDLLoGTv pFdOM2XH8GC9pVgUMtz9llWHLqSAMGBZ5SmCPdrNXbtNJz/f70ufpDTFnPZxZUBIX1Yi Lx8CQNm0LWJ9yzSwdaT6Z0ZKjaN8GR3Z6iicUrylvUGIY0RfIvKFJHpOBYdlur+znSWb 5jfZ8Kddu0BPRd/y0eLuU+G2ZG0nxqNNJ5R+53D3iliU1yCjoA0OtByBMcic/BJ4fUpY LaTQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=egoUl+UrCX7i4gPVYmA9o3lWb9zRaE4swrZUPBYXKnY=; b=Fn1utU5kXhwPStE9KbuJlli6Xyl05f+U0cJ+BKcSENzbXKEfizsPoFcgrb4ultgRYq VM41W3psNyc448DX9t98K5uFFfnHnMgzBKBnfPt83v/h/Y7kvQbU16LnSLeudX9Hdzq0 v0tG9E5Mrou+8C4IVYgmqCn/cpf20RlpWTFQbs2zV6QtPttkl8cAGBRkcrK2lQXARTz6 QhUlAIqjTJNFdtAxvAnyU2uBJFj3rZjndU0ZwesxlMNhdmt2KcZiz5N7XYyoiNuhZ1zu OH8CYRmb01aDVU3t5y8qVcAI7EW7JyU56DAw3Gf5FNot81+lKr+N1cW80l2uKeNYV8qh +vZQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=FDmOTfUD; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y17-20020a056402135100b004a2758d0cc5si4504515edw.226.2023.02.22.06.53.04; Wed, 22 Feb 2023 06:53:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=FDmOTfUD; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232253AbjBVOrg (ORCPT + 99 others); Wed, 22 Feb 2023 09:47:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59486 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232169AbjBVOrZ (ORCPT ); Wed, 22 Feb 2023 09:47:25 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BF94E3BDAF for ; Wed, 22 Feb 2023 06:47:02 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 3544661464 for ; Wed, 22 Feb 2023 14:47:02 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 819ADC4339B; Wed, 22 Feb 2023 14:46:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1677077221; bh=Wy5hbT4ewIbxtuRGZo3BNmMi3nK/RbjQJn9sH1g4yFs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=FDmOTfUDDAvB318Tim6Svmwl4prd0CvCrO2yMhterj811AW25jGCizagfRmMU3PuO TtwrV6OOwv0mDAN5hY6twOv0xbamI50e31QG5hOBllfU0tfy1WiuNPI1PFNylJku9E QN4/p1q4Mkdo26fnClKlfGr9+zZk2dvpCo1jxbOYzQ+c677jHx0+V9SrSnUeintpfl JfyivG+S4iGlZNMP70TqKR+DSDUdc7Wbb+gTYayl3WGKBFkf2IXoLn4WuTbr4EivDM MdnybAwi2AGGOFmy382kl+pG1nF6aGuLLq2BGD/GXMWKOpgw5S3CvPPD7yfdLzqFb8 UabOOMYAYTN6w== From: Frederic Weisbecker To: Thomas Gleixner Cc: LKML , Frederic Weisbecker , Alexey Dobriyan , Wei Li , Peter Zijlstra , Mirsad Goran Todorovac , Yu Liao , Hillf Danton , Ingo Molnar Subject: [PATCH 2/8] timers/nohz: Only ever update sleeptime from idle exit Date: Wed, 22 Feb 2023 15:46:43 +0100 Message-Id: <20230222144649.624380-3-frederic@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230222144649.624380-1-frederic@kernel.org> References: <20230222144649.624380-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758543330286671622?= X-GMAIL-MSGID: =?utf-8?q?1758543330286671622?= The idle and io sleeptime statistics appearing in /proc/stat can be currently updated from two sites: locally on idle exit and remotely by cpufreq. However there is no synchronization mechanism protecting concurrent updates. It is therefore possible to account the sleeptime twice, among all the possible broken scenarios. To prevent from breaking the sleeptime accounting source, restrict the sleeptime updates to the local idle exit site. If there is a delta to add since the last update, IO/Idle sleep time readers will now only compute the delta without actually writing it back to the internal idle statistic fields. This fixes a writer VS writer race. Note there are still two known reader VS writer races to handle. A subsequent patch will fix one. Reported-by: Yu Liao Acked-by: Peter Zijlstra (Intel) Cc: Hillf Danton Cc: Yu Liao Cc: Ingo Molnar Cc: Thomas Gleixner Cc: Wei Li Cc: Alexey Dobriyan Cc: Mirsad Goran Todorovac Signed-off-by: Frederic Weisbecker --- kernel/time/tick-sched.c | 103 ++++++++++++++++----------------------- 1 file changed, 41 insertions(+), 62 deletions(-) diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index b0e3c9205946..9058b9eb8bc1 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -637,31 +637,21 @@ static void tick_nohz_update_jiffies(ktime_t now) touch_softlockup_watchdog_sched(); } -/* - * Updates the per-CPU time idle statistics counters - */ -static void -update_ts_time_stats(int cpu, struct tick_sched *ts, ktime_t now, u64 *last_update_time) -{ - ktime_t delta; - - if (ts->idle_active) { - delta = ktime_sub(now, ts->idle_entrytime); - if (nr_iowait_cpu(cpu) > 0) - ts->iowait_sleeptime = ktime_add(ts->iowait_sleeptime, delta); - else - ts->idle_sleeptime = ktime_add(ts->idle_sleeptime, delta); - ts->idle_entrytime = now; - } - - if (last_update_time) - *last_update_time = ktime_to_us(now); - -} - static void tick_nohz_stop_idle(struct tick_sched *ts, ktime_t now) { - update_ts_time_stats(smp_processor_id(), ts, now, NULL); + ktime_t delta; + + if (WARN_ON_ONCE(!ts->idle_active)) + return; + + delta = ktime_sub(now, ts->idle_entrytime); + + if (nr_iowait_cpu(smp_processor_id()) > 0) + ts->iowait_sleeptime = ktime_add(ts->iowait_sleeptime, delta); + else + ts->idle_sleeptime = ktime_add(ts->idle_sleeptime, delta); + + ts->idle_entrytime = now; ts->idle_active = 0; sched_clock_idle_wakeup_event(); @@ -674,6 +664,30 @@ static void tick_nohz_start_idle(struct tick_sched *ts) sched_clock_idle_sleep_event(); } +static u64 get_cpu_sleep_time_us(struct tick_sched *ts, ktime_t *sleeptime, + bool compute_delta, u64 *last_update_time) +{ + ktime_t now, idle; + + if (!tick_nohz_active) + return -1; + + now = ktime_get(); + if (last_update_time) + *last_update_time = ktime_to_us(now); + + if (ts->idle_active && compute_delta) { + ktime_t delta = ktime_sub(now, ts->idle_entrytime); + + idle = ktime_add(*sleeptime, delta); + } else { + idle = *sleeptime; + } + + return ktime_to_us(idle); + +} + /** * get_cpu_idle_time_us - get the total idle time of a CPU * @cpu: CPU number to query @@ -691,27 +705,9 @@ static void tick_nohz_start_idle(struct tick_sched *ts) u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time) { struct tick_sched *ts = &per_cpu(tick_cpu_sched, cpu); - ktime_t now, idle; - - if (!tick_nohz_active) - return -1; - - now = ktime_get(); - if (last_update_time) { - update_ts_time_stats(cpu, ts, now, last_update_time); - idle = ts->idle_sleeptime; - } else { - if (ts->idle_active && !nr_iowait_cpu(cpu)) { - ktime_t delta = ktime_sub(now, ts->idle_entrytime); - - idle = ktime_add(ts->idle_sleeptime, delta); - } else { - idle = ts->idle_sleeptime; - } - } - - return ktime_to_us(idle); + return get_cpu_sleep_time_us(ts, &ts->idle_sleeptime, + !nr_iowait_cpu(cpu), last_update_time); } EXPORT_SYMBOL_GPL(get_cpu_idle_time_us); @@ -732,26 +728,9 @@ EXPORT_SYMBOL_GPL(get_cpu_idle_time_us); u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time) { struct tick_sched *ts = &per_cpu(tick_cpu_sched, cpu); - ktime_t now, iowait; - if (!tick_nohz_active) - return -1; - - now = ktime_get(); - if (last_update_time) { - update_ts_time_stats(cpu, ts, now, last_update_time); - iowait = ts->iowait_sleeptime; - } else { - if (ts->idle_active && nr_iowait_cpu(cpu) > 0) { - ktime_t delta = ktime_sub(now, ts->idle_entrytime); - - iowait = ktime_add(ts->iowait_sleeptime, delta); - } else { - iowait = ts->iowait_sleeptime; - } - } - - return ktime_to_us(iowait); + return get_cpu_sleep_time_us(ts, &ts->iowait_sleeptime, + nr_iowait_cpu(cpu), last_update_time); } EXPORT_SYMBOL_GPL(get_cpu_iowait_time_us);