From patchwork Mon Nov 14 23:28:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 20090 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp2421189wru; Mon, 14 Nov 2022 15:39:50 -0800 (PST) X-Google-Smtp-Source: AA0mqf4QN6+fShqUgCja4OG83VjQ+vd/OMrYHAwkjR3iNWvTRn1poCkfRSeddsX+cdgX9dIGM1TH X-Received: by 2002:a05:6402:25c7:b0:467:f948:6df4 with SMTP id x7-20020a05640225c700b00467f9486df4mr3823729edb.217.1668469190175; Mon, 14 Nov 2022 15:39:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668469190; cv=none; d=google.com; s=arc-20160816; b=ZXo6FJeAPAgXKM11e+xPEMXfgahSHjYHlJNAcfA4OWIwu0CjsXH6+BmcD6RBKyNF71 6B/18N3leSNQSv0W4w0hovZVJzZPu1KjX51yroOptPw7BRzvYgDO5b2lpT5IqqrpLYFY jPBWrujdWo8N9WInynQTAUAn2TbhWEaupNhiGNu4yJnWAu9vTKQLRlqj9vAAqGRdfzWK fOx0EXuultI3p93BUMERTyLuDXZlo7JxCplbwhNOTxiTWc3j0kvAeeQ69rmHIOySTib3 d6e6WEvNNANJ5gKEEMYINZZPyjC1SPiFmdNHSsiK+Yvaaf/FuJIeycvfW3joS6yc94e8 1e4Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=m6BtVwquu1iMi9rgw1jwVjIMeYawj9MxMCCgbKrrqLA=; b=k1M6BQn1yQL1cdTYOLZtDGrKExSrjIIM4fSu0GXMZ7g0NXsc0SaCtBJhXCokOHalYw YbIQZYCxYJvQkwx+iSA8JfgoUR6GDudaNfEwzG2Dz/da+W6J+qkt+a9WjeELVm0YF8dE NyP0fO8/9C/LYv+mvsdW2qhM9SsP/CXsJ85LNhnMuF+ewH8XFSjG+E0VKbMSygKaLCeW gwpuwwdPku4bJZWZQmaCvyFn4OZppGl8bDhF79QLQk0FNB4cxIdghT2yf3N4hHgj4jab 9VdC1DWWTlap5yNwAxDHU17ZPcjHGFnO0qqzYi8AvRFiSlbtiSfh9P+a3X6QGtjVjZrH 9xhA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="A/X+O8aX"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id qf14-20020a1709077f0e00b007add269f2c4si10822526ejc.740.2022.11.14.15.39.26; Mon, 14 Nov 2022 15:39:50 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="A/X+O8aX"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236432AbiKNX2j (ORCPT + 99 others); Mon, 14 Nov 2022 18:28:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39900 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236173AbiKNX2d (ORCPT ); Mon, 14 Nov 2022 18:28:33 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7C0691157 for ; Mon, 14 Nov 2022 15:28:29 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 7E4EB614B9 for ; Mon, 14 Nov 2022 23:28:29 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id DE716C433D7; Mon, 14 Nov 2022 23:28:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668468508; bh=KYiqFd5L+cDLkyT6epEDOfkYEWTRr614tu/0hHbUz6U=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=A/X+O8aXEOSxP8x3DYS+35d+nm6T3IXGs6J4bGbc9VfVQJy6ELwhDCbAljCbdcbW+ Ya5GV1QLglrEjUrSTQZIv6iqQxhsq0oyu7YU/sNFM7Gw+9kVv/ShrNhJoovoAOcZH0 5uDZXAVidcgeYyG2M8Ey/jRi0JwE4Pmjk9a46yFg6Cj+6EHyfMYQJFxKiArml6sDHY TQAeKK6DnJnLxNRN5rUd+GHXuMV7vRPOyMhCZ/utvlTHSTX5aBElinuRp/lAQLkaO7 YeySi0M0Dlq8AeCCEeZtOjQXttBbN3owtuSf7eOnLeTrmeqPViADFlD4VDtkTAsIXa N9YdWvJoIzvKQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 8F9315C14FC; Mon, 14 Nov 2022 15:28:28 -0800 (PST) From: "Paul E. McKenney" To: tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, john.stultz@linaro.org, sboyd@kernel.org, corbet@lwn.net, Mark.Rutland@arm.com, maz@kernel.org, kernel-team@meta.com, neeraju@codeaurora.org, ak@linux.intel.com, feng.tang@intel.com, zhengjun.xing@intel.com, "Paul E. McKenney" , Chris Mason , John Stultz , Waiman Long Subject: [PATCH clocksource 1/3] clocksource: Reject bogus watchdog clocksource measurements Date: Mon, 14 Nov 2022 15:28:24 -0800 Message-Id: <20221114232827.835599-1-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221114232807.GA834337@paulmck-ThinkPad-P17-Gen-1> References: <20221114232807.GA834337@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749516749748247033?= X-GMAIL-MSGID: =?utf-8?q?1749516749748247033?= One remaining clocksource-skew issue involves extreme CPU overcommit, which can cause the clocksource watchdog measurements to be delayed by tens of seconds. This in turn means that a clock-skew criterion that is appropriate for a 500-millisecond interval will instead give lots of false positives. Therefore, check for the watchdog clocksource reporting much larger or much less than the time specified by WATCHDOG_INTERVAL. In these cases, print a pr_warn() warning and refrain from marking the clocksource under test as being unstable. Reported-by: Chris Mason Signed-off-by: Paul E. McKenney Cc: John Stultz Cc: Thomas Gleixner Cc: Stephen Boyd Cc: Feng Tang Cc: Waiman Long --- kernel/time/clocksource.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/kernel/time/clocksource.c b/kernel/time/clocksource.c index 8058bec87acee..dcaf38c062161 100644 --- a/kernel/time/clocksource.c +++ b/kernel/time/clocksource.c @@ -386,7 +386,7 @@ EXPORT_SYMBOL_GPL(clocksource_verify_percpu); static void clocksource_watchdog(struct timer_list *unused) { - u64 csnow, wdnow, cslast, wdlast, delta; + u64 csnow, wdnow, cslast, wdlast, delta, wdi; int next_cpu, reset_pending; int64_t wd_nsec, cs_nsec; struct clocksource *cs; @@ -440,6 +440,17 @@ static void clocksource_watchdog(struct timer_list *unused) if (atomic_read(&watchdog_reset_pending)) continue; + /* Check for bogus measurements. */ + wdi = jiffies_to_nsecs(WATCHDOG_INTERVAL); + if (wd_nsec < (wdi >> 2)) { + pr_warn("timekeeping watchdog on CPU%d: Watchdog clocksource '%s' advanced only %lld ns during %d-jiffy time interval, skipping watchdog check.\n", smp_processor_id(), watchdog->name, wd_nsec, WATCHDOG_INTERVAL); + continue; + } + if (wd_nsec > (wdi << 2)) { + pr_warn("timekeeping watchdog on CPU%d: Watchdog clocksource '%s' advanced an excessive %lld ns during %d-jiffy time interval, probable CPU overutilization, skipping watchdog check.\n", smp_processor_id(), watchdog->name, wd_nsec, WATCHDOG_INTERVAL); + continue; + } + /* Check the deviation from the watchdog clocksource. */ md = cs->uncertainty_margin + watchdog->uncertainty_margin; if (abs(cs_nsec - wd_nsec) > md) { From patchwork Mon Nov 14 23:28:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 20095 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp2422103wru; Mon, 14 Nov 2022 15:42:48 -0800 (PST) X-Google-Smtp-Source: AA0mqf6uuPlgHRfXw3Ft9O3wz5+A56lLNeoOWuA1OGY8crehC8Otg+0XxsrF7FjM9noaXoAUMI18 X-Received: by 2002:a17:907:7672:b0:7ad:c35a:ad76 with SMTP id kk18-20020a170907767200b007adc35aad76mr12005189ejc.705.1668469368136; Mon, 14 Nov 2022 15:42:48 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668469368; cv=none; d=google.com; s=arc-20160816; b=X+tgiwXZieLCs8AFHBG35SGvV6aNBjGupdt0bO46juKv4YntB0KFS2VRwPQBoMIoz9 Ju6qgjMSAEzCcPDqvYzf/uan3x3G80QKnOEZl83YfgoGbU8r6XO11UJlBlF+u9bQm6rk brorvOL7iLoEsx88QuQvyvDm2JzhUBkxzHY9cEazVF3BRXqjFzwk+gqiIg2lYSrUiPdh 8CYxx4YrvSU9sBngSNQ9yH04kA5NPA8nbv/lxNk4aDbGH6LUwX3g+zAwobGVq5ag+hpq GiwT+HPGfOTVgpCcpaAmXufLM2AAQiPP2NM/2hpU2tFOkP5j7RyXUdvYS6pbl9OWHb4I ZGBw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=JyxJr/Nc9bEaMpzDzRhoMQQze00aSiEji1chdk8jcpA=; b=JY99rhgFqRg25BZuC2tNnpST1Vxf1FqC0elQWLScz7I6pgFur3BNyF1EZM7vlBhHfj zOEK+ma/v+7LVDipF0zBjnCXQnEgJVoKdrIC7SKfuGZBy7PW/WH0yBNO06Y/s4BIakv/ FLwzJpp9UniumPLtDEMbxosKip8s1m8tjZbAkkDPhg71/VW4mVlKJgtuPoOSRRGReChM hCR/ux909uxBmg+GPj2VBFj4o1SiSZAuoqsa2tP7ao2lt9cmCS9Imn1VwF1rXAXoyuMs aqjFfXS1n0/+8h4GaYdj4nAMINIZporjMev8jbUjHYU8uMFoUgQ9MIQ1QjR+WidTzaSP pSAw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=s8NlDZxi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f6-20020a17090624c600b00783a0e64f8asi7916603ejb.246.2022.11.14.15.42.24; Mon, 14 Nov 2022 15:42:48 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=s8NlDZxi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231790AbiKNX2n (ORCPT + 99 others); Mon, 14 Nov 2022 18:28:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40336 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236642AbiKNX2d (ORCPT ); Mon, 14 Nov 2022 18:28:33 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 935E92A4 for ; Mon, 14 Nov 2022 15:28:31 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 4CD4AB815FC for ; Mon, 14 Nov 2022 23:28:30 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E2D6BC433B5; Mon, 14 Nov 2022 23:28:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668468509; bh=ZjS+OKGtE62BjR1npWFo6xbmcHIB1kAdTslp2K20OB8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=s8NlDZxiV/u067KU7C3YtF9zAQMKW5UVHP2/5BJ7oZc7OZXi1teKrv56PH4rgoqtB /Xz4IV2JIkrnzU5g7ZBl50S2PzuQxzvcmZJTOKm42B+H4fOmRkTcM+VmG7B/Lz4pQ8 5/tJVZPu/8o2rzZrFMryalyu7GfsPoCLeb1aNi9fL/sZVMpMQ31ZRPL1tV0F/LaNDA 4T8whGVExx4YhJUByU82/jCN/4R5EYdQRTNACcfmO1NonrIzHq3OkSi6tWL2AkfrZC Kt/jjSIPIilpZclEqR5fCHZGZ09PplUdCguPGAxD+oEJrURs3jRRABI8nmix3lyU9q TRzDPILoFgNLQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 91E705C16C5; Mon, 14 Nov 2022 15:28:28 -0800 (PST) From: "Paul E. McKenney" To: tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, john.stultz@linaro.org, sboyd@kernel.org, corbet@lwn.net, Mark.Rutland@arm.com, maz@kernel.org, kernel-team@meta.com, neeraju@codeaurora.org, ak@linux.intel.com, feng.tang@intel.com, zhengjun.xing@intel.com, "Paul E. McKenney" , Waiman Long , John Stultz Subject: [PATCH clocksource 2/3] clocksource: Add comments to classify bogus measurements Date: Mon, 14 Nov 2022 15:28:25 -0800 Message-Id: <20221114232827.835599-2-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221114232807.GA834337@paulmck-ThinkPad-P17-Gen-1> References: <20221114232807.GA834337@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749516936138436145?= X-GMAIL-MSGID: =?utf-8?q?1749516936138436145?= An extremely busy system can delay the clocksource watchdog, so that the corresponding too-long bogus-measurement error does not necessarily imply an error in the system. However, a too-short bogus-measurement error likely indicates a bug in hardware, firmware or software. Therefore, add comments clarifying these bogus-measurement pr_warn()s. Reported-by: Feng Tang Signed-off-by: Paul E. McKenney Reviewed-by: Waiman Long Reviewed-by: Feng Tang Cc: John Stultz Cc: Thomas Gleixner Cc: Stephen Boyd --- kernel/time/clocksource.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/time/clocksource.c b/kernel/time/clocksource.c index dcaf38c062161..3f5317faf891f 100644 --- a/kernel/time/clocksource.c +++ b/kernel/time/clocksource.c @@ -443,10 +443,12 @@ static void clocksource_watchdog(struct timer_list *unused) /* Check for bogus measurements. */ wdi = jiffies_to_nsecs(WATCHDOG_INTERVAL); if (wd_nsec < (wdi >> 2)) { + /* This usually indicates broken timer code or hardware. */ pr_warn("timekeeping watchdog on CPU%d: Watchdog clocksource '%s' advanced only %lld ns during %d-jiffy time interval, skipping watchdog check.\n", smp_processor_id(), watchdog->name, wd_nsec, WATCHDOG_INTERVAL); continue; } if (wd_nsec > (wdi << 2)) { + /* This can happen on busy systems, which can delay the watchdog. */ pr_warn("timekeeping watchdog on CPU%d: Watchdog clocksource '%s' advanced an excessive %lld ns during %d-jiffy time interval, probable CPU overutilization, skipping watchdog check.\n", smp_processor_id(), watchdog->name, wd_nsec, WATCHDOG_INTERVAL); continue; } From patchwork Mon Nov 14 23:28:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 20091 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp2421651wru; Mon, 14 Nov 2022 15:41:15 -0800 (PST) X-Google-Smtp-Source: AA0mqf5nsC9y5yz1p/9lqaoD3f4ri3tcrnH1XCGM6LTNLan1NU+MRFSFHTUa8uWtiiduVUQnqEIV X-Received: by 2002:aa7:dc03:0:b0:462:6a0c:cfa with SMTP id b3-20020aa7dc03000000b004626a0c0cfamr12990947edu.349.1668469275551; Mon, 14 Nov 2022 15:41:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668469275; cv=none; d=google.com; s=arc-20160816; b=K+VYEvlPSMJ7nFm2SzV63XuupjCTZDgWyjyeKf3m0jqQpucjxbLNUrWcjepVdCvU02 kINzJhzGWYqB9XnR0AIi6etgRhyx3LW4V6DpJ6L14nh7kHx7W/o8VyW45OVCEcIKEG/N eoFtTE5yUW6+YEg/or47sWvisAhuVKtOe3ghd2GH7LQljT+1ibM6XtkvleyZJ9abFs29 RJDMgvph51cwwoFyy4KACzNQLSHMPT3caYO2h6sZvGNAP43pLwPab0y/VFGpBu6HWT9s AUHvG5qNr7KiFpqcOL3xhAuosRFKthcj4KEDL0tm4h5ZbHS3JqNGjwMAOZmeIJXit2oX C1dQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=z3uRyW9e1y9P6ZGvduzRozd7XFV95okur8nQ8Syr3c4=; b=wKhnLTytpLY8jJ6NFMHAF57KXDhhFPc30sBBSsNZP5YOypC1SCk4HU/0osYGpTsN87 8+JkbvOxtz3x5kV9bxTzZXcz/Y/SxvnH+YYy7K/M8dycHXBN4TAZphaRcY/b9OW3NgiJ mqgZhuF4ob1SQR74M2FqDtitGsBjyPZVrTuhhpUvrtTmfJjYw6Gdx8v0OLx9jR0tApAC Kvk1dy221qcVxWFJA4fDQKmHaE4Z6nL49+OnXbHnW3lvG+ZQMFEvdt58g/k11P65NnlJ XQTS/XSVsDjkThTKgf64i0HJLl8PDj3KZECy4n/ZiEZ//Zdk8ni2ifKpngNjUySmUYTP kNBw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=U240jKJW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q15-20020a1709064c8f00b0078d550c0272si8377144eju.269.2022.11.14.15.40.51; Mon, 14 Nov 2022 15:41:15 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=U240jKJW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237118AbiKNX2s (ORCPT + 99 others); Mon, 14 Nov 2022 18:28:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40338 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236856AbiKNX2e (ORCPT ); Mon, 14 Nov 2022 18:28:34 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AD1E92AB for ; Mon, 14 Nov 2022 15:28:31 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 4E647B8133B for ; Mon, 14 Nov 2022 23:28:30 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D9CD9C433C1; Mon, 14 Nov 2022 23:28:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668468508; bh=yWPbFrTyk/Pat7LzvV8D6SH/ODiafdLuMh9HWpa0Mu0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=U240jKJW3sGDmowlbl/h6Mpt1cLwUTBap5/9nPpthjXqufdIfx3lsVzDx64c4n4U6 lqhJEx33abLRxkuqwVzQ0P+CIA0CtLmgw3pa6ppp9r3SQYG2RZrFPcvunWrMMkT95W iLXI+bL723pmkqztOnufnepz4/RqzkbCmRynMvy62SH7ClaXgmkTNpzHwEUahn/6PE KYXQJtyBlNztfhYW1XsJbTX3Q0w2LK6rUEJu4AmRQWQqtTHjD+x0o70rT5b9MuS836 2z+L/5AI0wxGBRFD34YsWct/iuPJyN2g5d5EV4cXYCqFnGj9mgdfE0grCP0u4rPekl gJFxfBgHRSvng== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 940665C1718; Mon, 14 Nov 2022 15:28:28 -0800 (PST) From: "Paul E. McKenney" To: tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, john.stultz@linaro.org, sboyd@kernel.org, corbet@lwn.net, Mark.Rutland@arm.com, maz@kernel.org, kernel-team@meta.com, neeraju@codeaurora.org, ak@linux.intel.com, feng.tang@intel.com, zhengjun.xing@intel.com, "Paul E. McKenney" , Waiman Long , John Stultz Subject: [PATCH clocksource 3/3] clocksource: Exponential backoff for load-induced bogus watchdog reads Date: Mon, 14 Nov 2022 15:28:26 -0800 Message-Id: <20221114232827.835599-3-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221114232807.GA834337@paulmck-ThinkPad-P17-Gen-1> References: <20221114232807.GA834337@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749516839159564580?= X-GMAIL-MSGID: =?utf-8?q?1749516839159564580?= The clocksource watchdog will reject measurements that are excessively delayed, that is, by more than 1.5 seconds beyond the intended 0.5-second watchdog interval. On an extremely busy system, this can result in a console message being printed every two seconds. This is excessively noisy for a non-error condition. Therefore, apply exponential backoff to these messages. This exponential backoff is capped at 1024 times the watchdog interval, which comes to not quite one message per ten minutes. Please note that the bogus watchdog reads that occur when the watchdog interval is less than 0.125 seconds are still printed unconditionally because these likely correspond to a serious error condition in the timer code or hardware. [ paulmck: Apply Feng Tang feedback. ] [ paulmck: Apply Waiman Long feedback. ] Reported-by: Waiman Long Reported-by: Feng Tang Signed-off-by: Paul E. McKenney Reviewed-by: Feng Tang Reviewed-by: Waiman Long Cc: John Stultz Cc: Thomas Gleixner Cc: Stephen Boyd --- include/linux/clocksource.h | 3 +++ kernel/time/clocksource.c | 31 +++++++++++++++++++++++++------ 2 files changed, 28 insertions(+), 6 deletions(-) diff --git a/include/linux/clocksource.h b/include/linux/clocksource.h index 1d42d4b173271..23b73f2293d6d 100644 --- a/include/linux/clocksource.h +++ b/include/linux/clocksource.h @@ -125,6 +125,9 @@ struct clocksource { struct list_head wd_list; u64 cs_last; u64 wd_last; + u64 wd_last_bogus; + int wd_bogus_shift; + unsigned long wd_bogus_count; #endif struct module *owner; }; diff --git a/kernel/time/clocksource.c b/kernel/time/clocksource.c index 3f5317faf891f..4015ec6503a52 100644 --- a/kernel/time/clocksource.c +++ b/kernel/time/clocksource.c @@ -442,14 +442,33 @@ static void clocksource_watchdog(struct timer_list *unused) /* Check for bogus measurements. */ wdi = jiffies_to_nsecs(WATCHDOG_INTERVAL); - if (wd_nsec < (wdi >> 2)) { - /* This usually indicates broken timer code or hardware. */ - pr_warn("timekeeping watchdog on CPU%d: Watchdog clocksource '%s' advanced only %lld ns during %d-jiffy time interval, skipping watchdog check.\n", smp_processor_id(), watchdog->name, wd_nsec, WATCHDOG_INTERVAL); + if (wd_nsec > (wdi << 2) || cs_nsec > (wdi << 2)) { + bool needwarn = false; + u64 wd_lb; + + cs->wd_bogus_count++; + if (!cs->wd_bogus_shift) { + needwarn = true; + } else { + delta = clocksource_delta(wdnow, cs->wd_last_bogus, watchdog->mask); + wd_lb = clocksource_cyc2ns(delta, watchdog->mult, watchdog->shift); + if ((1 << cs->wd_bogus_shift) * wdi <= wd_lb) + needwarn = true; + } + if (needwarn) { + /* This can happen on busy systems, which can delay the watchdog. */ + pr_warn("timekeeping watchdog on CPU%d: Watchdog clocksource '%s' advanced an excessive %lld ns during %d-jiffy time interval (%lu since last message), probable CPU overutilization, skipping watchdog check.\n", smp_processor_id(), watchdog->name, wd_nsec, WATCHDOG_INTERVAL, cs->wd_bogus_count); + cs->wd_last_bogus = wdnow; + if (cs->wd_bogus_shift < 10) + cs->wd_bogus_shift++; + cs->wd_bogus_count = 0; + } continue; } - if (wd_nsec > (wdi << 2)) { - /* This can happen on busy systems, which can delay the watchdog. */ - pr_warn("timekeeping watchdog on CPU%d: Watchdog clocksource '%s' advanced an excessive %lld ns during %d-jiffy time interval, probable CPU overutilization, skipping watchdog check.\n", smp_processor_id(), watchdog->name, wd_nsec, WATCHDOG_INTERVAL); + /* Check too-short measurements second to handle wrap. */ + if (wd_nsec < (wdi >> 2) || cs_nsec < (wdi >> 2)) { + /* This usually indicates broken timer code or hardware. */ + pr_warn("timekeeping watchdog on CPU%d: Watchdog clocksource '%s' advanced only %lld ns during %d-jiffy time interval, skipping watchdog check.\n", smp_processor_id(), watchdog->name, wd_nsec, WATCHDOG_INTERVAL); continue; }