From patchwork Thu Jan 5 01:06:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 39265 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp56360wrt; Wed, 4 Jan 2023 17:11:31 -0800 (PST) X-Google-Smtp-Source: AMrXdXtvh5SWLA4t+mEdrU1SdDpEGuXYNuBJi2HhLu5TcVik1gWR1elrqrceXOlznPrzV8upSO1V X-Received: by 2002:a17:90b:35c7:b0:226:17f9:8025 with SMTP id nb7-20020a17090b35c700b0022617f98025mr29254489pjb.40.1672881090861; Wed, 04 Jan 2023 17:11:30 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672881090; cv=none; d=google.com; s=arc-20160816; b=r7SnzDCV6jw3d5wrPrXoMbuE8uyD0emMxSbnFOs3eNM0XOaTnJA4shSE30oweqtfBE k2SagqD1RrjTxHyoW/C4NnCviL4ZaO9P7ZhUcvNaa8ZesF5g+sL69INCVM4WCJaRsNso 3tBTFU7A7BlKK+ixWMLpsySqmqKsUZDjDNim7laondNcZbgHt2AOn0XlDmfWFwZSjyEp B3P8WUkoxHTkBz6bOQJ61y4cd9/SL2sScEKFco4SlIAuAtFcJW/Sjya816sI69dDpvjc YWNRcdAhCCED/j4agCxkzzcNegsyfwpeME7ptMYwXrrG/w8fI93ODGEFA8K4UUR+VtDr DksA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=leyAfntoPyncGeLmBFFQvj2xV9SgXtKjZS2Ph6m63vk=; b=zqnTeL+qoNRyzE3J3a5/lp+DnkGup0r2EdTG49c0aN3IsIKYeZtBOc91JbPGmdYmX1 BbR3PifLbKRZjPklphP0bTTWJZnsL9IamrcSUOdXHSLggGDgaNCqttkFsDVX3LZd+WNh ztUT1tWiV4irV0k+DKJIWMVrFX26mUzUePkpdvd8OES/ED+A9DLsJlZR00BMQE85BT2p H5u8HTj0nEL1qaW1BWJYzAASxo4wz1ea2fpTotgRqyTA48kwBiJGVxGwmS4EN56L5Pqw f/Jk1V3UywGibi8iVnl1jB26nM8fMzQ4Mbv4JqJ0buJnssedBKlS843NVl1h/dZM5sWG hEfg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=GN77Hc6X; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id lx8-20020a17090b4b0800b002199a9891desi483984pjb.141.2023.01.04.17.11.17; Wed, 04 Jan 2023 17:11:30 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=GN77Hc6X; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229927AbjAEBHG (ORCPT + 99 others); Wed, 4 Jan 2023 20:07:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44100 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229727AbjAEBHE (ORCPT ); Wed, 4 Jan 2023 20:07:04 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E8E1B2F7B2 for ; Wed, 4 Jan 2023 17:07:03 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 9053C61881 for ; Thu, 5 Jan 2023 01:07:03 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id EAF85C433EF; Thu, 5 Jan 2023 01:07:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1672880823; bh=tNnAiq5HYDDhjHdWky6iQVYABLIWoBenLL5L50zs8N4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GN77Hc6XS1nlTuMi12zL+N+XWGpzdO2v30sgdGDbi9bCgHju5MURna2GwsoIifsPV vc1CaTS3N8FZqPl7Loq6Dtw53crh8h4cwiRccmYhE3kihvi4neBVQHbZZ6A6kpJVxC nT/3hb/XQZwWRMgaTlmuBIb+c9zK90NbpP5svTCaqW6JT2FB1izLxtejExLy3bEGHR e8Fj1/gQ8pg+zZ2twpmPJljotcSA0bMgvp6APi6jrVKUB39Dw3gQs3jPCaKGjvrkzp AnVY3pcTkWw1xmILRbyU7UlcoTOr5lhAEK+B76viUgUg7snuN05dvC2/HgYO5R/pz1 G1YoEbd+wY8CQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id A84285C05CA; Wed, 4 Jan 2023 17:07:02 -0800 (PST) From: "Paul E. McKenney" To: tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, john.stultz@linaro.org, sboyd@kernel.org, corbet@lwn.net, Mark.Rutland@arm.com, maz@kernel.org, kernel-team@meta.com, neeraju@codeaurora.org, ak@linux.intel.com, feng.tang@intel.com, zhengjun.xing@intel.com, Yunying Sun , "Paul E . McKenney" Subject: [PATCH clocksource 1/6] clocksource: Print clocksource name when clocksource is tested unstable Date: Wed, 4 Jan 2023 17:06:56 -0800 Message-Id: <20230105010701.1773895-1-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20230105010429.GA1773522@paulmck-ThinkPad-P17-Gen-1> References: <20230105010429.GA1773522@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754142962533508672?= X-GMAIL-MSGID: =?utf-8?q?1754142962533508672?= From: Yunying Sun Some "TSC fall back to HPET" messages appear on systems having more than 2 NUMA nodes: clocksource: timekeeping watchdog on CPU168: hpet read-back delay of 4296200ns, attempt 4, marking unstable The "hpet" here is misleading the clocksource watchdog is really doing repeated reads of "hpet" in order to check for unrelated delays. Therefore, print the name of the clocksource under test, prefixed by "wd-" and suffixed by "-wd", for example, "wd-tsc-wd". Signed-off-by: Yunying Sun Signed-off-by: Paul E. McKenney --- kernel/time/clocksource.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/time/clocksource.c b/kernel/time/clocksource.c index 9cf32ccda715d..4a2c3bb92e2e9 100644 --- a/kernel/time/clocksource.c +++ b/kernel/time/clocksource.c @@ -257,8 +257,8 @@ static enum wd_read_status cs_watchdog_read(struct clocksource *cs, u64 *csnow, goto skip_test; } - pr_warn("timekeeping watchdog on CPU%d: %s read-back delay of %lldns, attempt %d, marking unstable\n", - smp_processor_id(), watchdog->name, wd_delay, nretries); + pr_warn("timekeeping watchdog on CPU%d: wd-%s-wd read-back delay of %lldns, attempt %d, marking unstable\n", + smp_processor_id(), cs->name, wd_delay, nretries); return WD_READ_UNSTABLE; skip_test: From patchwork Thu Jan 5 01:06:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 39269 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp56576wrt; Wed, 4 Jan 2023 17:12:07 -0800 (PST) X-Google-Smtp-Source: AMrXdXu1u+K/1dEJYYHMJ+vEq7kK4tuf6KHQd5qrZLciuny9SX4XD7y2WM/3yRpbiVo3rl1PUTOd X-Received: by 2002:a17:90a:f614:b0:225:c3ab:3137 with SMTP id bw20-20020a17090af61400b00225c3ab3137mr44652236pjb.44.1672881127368; Wed, 04 Jan 2023 17:12:07 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672881127; cv=none; d=google.com; s=arc-20160816; b=Wo9biJRs5rmPlqku1Qmux8Dk+mBYyqXeJb8nUDyYU4k/5sLmMWGRC+YoaIVxJkvz59 KCK5nqzd0NFqIiedbjQheSJXizE1evm1j1pHacVTYcLHCqdv+4DwsEr4PBBcb9MgLgc2 8pwh58OyEcPd6rhJlpFQNeHUYdd0cD/0fI0xKu7wyBbNxgvRbGELwqMxIl9uo4kt88Sl A6QGQ+U55I6dW1FjN5WDKk3g0fp/0pSWsQ33GPr96eHQ3OfMbkcGBIkQgeBMvRne13i2 wOQ5coSgBMGlEwT0RKGNYVeKZ8osDcMmLkhipPG/3YW2RYN23GxiJ6EXA4LPzAYf45NX IhPA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=pS5sTazU1rZslFt6OqU/kU4RJoGUmrP+rWuC47g6Wuc=; b=hnUv6b7Vm4JCmLWFwUYoNy/OYDy/FlO9NE7h6Q1X2/7Stxn/+tsYleVYx5TXmYCcIj zMHvQ5WqY7M7CyDMLcT1I6p/3EhF2v+hXunW4DrBf4P8M9AKAG7Oxw41L0VTuhHlZgXl D85W2VmuU/sRgBRanG83bTkpsuZoh5P5jSd8gPcoPCaE0ZFiaWdMJx67yVX2Mc/AN4ID 2iLbF4fpEI3WSp/S3T3H/dEOHXrbzfOqN3GLfD3BrYXizLYxq8RQcWU+lO9UhleD+Nte vJGTCVkUI3jYgQGNsYR7szcWG5YH8K/7LTIaWzD2h9ycCEE4baXxoxu9IMdJXQ1dZPK3 wfUw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Dpm8So2J; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u15-20020a17090a6a8f00b0020b2101908asi496146pjj.16.2023.01.04.17.11.55; Wed, 04 Jan 2023 17:12:07 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Dpm8So2J; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230116AbjAEBHX (ORCPT + 99 others); Wed, 4 Jan 2023 20:07:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44180 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230029AbjAEBHH (ORCPT ); Wed, 4 Jan 2023 20:07:07 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B640C30563 for ; Wed, 4 Jan 2023 17:07:05 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 5E548B81984 for ; Thu, 5 Jan 2023 01:07:04 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 027F7C433F1; Thu, 5 Jan 2023 01:07:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1672880823; bh=yfdddVAsVm0wKfhUXEcuvIc8bj3pYmqk8I+8WWgBY/w=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Dpm8So2JL3X9PMhoZWVSdv/dq+i0y0lDDjxvZNzRpeCudBE03jip4QbBJGQxMHeSw pv/k+fHdtkYX+fYjpzTNThW48lBT5ckHs88usJbEF0VwgXRbMPs/F4Zg1jt7u4yZGW mIOjb0uHBU00ZaMyp0O887Ahv97F9p++x/6vM4EK0afW5hvRUubYuVoxhFN8efGeIo ucpSdcvf9jdsYe1lFNjh7BYHsw4CeHsBODFyNsHCk0QhB036Clj72vZZicDG44I+kB ks2Q75ka9F7YVDqMfOVEN1yJuL0rnOfl3k1tLjHEOgwb+KK2TI9c7Ae1TCfL1NNLh5 87KrokWCUWMrg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id AA98E5C086D; Wed, 4 Jan 2023 17:07:02 -0800 (PST) From: "Paul E. McKenney" To: tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, john.stultz@linaro.org, sboyd@kernel.org, corbet@lwn.net, Mark.Rutland@arm.com, maz@kernel.org, kernel-team@meta.com, neeraju@codeaurora.org, ak@linux.intel.com, feng.tang@intel.com, zhengjun.xing@intel.com, "Paul E. McKenney" Subject: [PATCH clocksource 2/6] clocksource: Loosen clocksource watchdog constraints Date: Wed, 4 Jan 2023 17:06:57 -0800 Message-Id: <20230105010701.1773895-2-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20230105010429.GA1773522@paulmck-ThinkPad-P17-Gen-1> References: <20230105010429.GA1773522@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754143000612117881?= X-GMAIL-MSGID: =?utf-8?q?1754143000612117881?= Currently, MAX_SKEW_USEC is set to 100 microseconds, which has worked reasonably well. However, NTP is willing to tolerate 500 microseconds of skew per second, and a clocksource that is good enough for NTP should be good enough for the clocksource watchdog. The watchdog's skew is controlled by MAX_SKEW_USEC and the CLOCKSOURCE_WATCHDOG_MAX_SKEW_US Kconfig option. However, these values are doubled before being associated with a clocksource's ->uncertainty_margin, and the ->uncertainty_margin values of the pair of clocksource's being compared are summed before checking against the skew. Therefore, set both MAX_SKEW_USEC and the default for the CLOCKSOURCE_WATCHDOG_MAX_SKEW_US Kconfig option to 125 microseconds of skew per second, resulting in 500 microseconds of skew per second in the clocksource watchdog's skew comparison. Suggested-by Rik van Riel Signed-off-by: Paul E. McKenney --- kernel/time/Kconfig | 6 +++++- kernel/time/clocksource.c | 15 +++++++++------ 2 files changed, 14 insertions(+), 7 deletions(-) diff --git a/kernel/time/Kconfig b/kernel/time/Kconfig index a41753be1a2bf..bae8f11070bef 100644 --- a/kernel/time/Kconfig +++ b/kernel/time/Kconfig @@ -200,10 +200,14 @@ config CLOCKSOURCE_WATCHDOG_MAX_SKEW_US int "Clocksource watchdog maximum allowable skew (in μs)" depends on CLOCKSOURCE_WATCHDOG range 50 1000 - default 100 + default 125 help Specify the maximum amount of allowable watchdog skew in microseconds before reporting the clocksource to be unstable. + The default is based on a half-second clocksource watchdog + interval and NTP's maximum frequency drift of 500 parts + per million. If the clocksource is good enough for NTP, + it is good enough for the clocksource watchdog! endmenu endif diff --git a/kernel/time/clocksource.c b/kernel/time/clocksource.c index 4a2c3bb92e2e9..a3d19f6660ac7 100644 --- a/kernel/time/clocksource.c +++ b/kernel/time/clocksource.c @@ -95,6 +95,11 @@ static char override_name[CS_NAME_LEN]; static int finished_booting; static u64 suspend_start; +/* + * Interval: 0.5sec. + */ +#define WATCHDOG_INTERVAL (HZ >> 1) + /* * Threshold: 0.0312s, when doubled: 0.0625s. * Also a default for cs->uncertainty_margin when registering clocks. @@ -106,11 +111,14 @@ static u64 suspend_start; * clocksource surrounding a read of the clocksource being validated. * This delay could be due to SMIs, NMIs, or to VCPU preemptions. Used as * a lower bound for cs->uncertainty_margin values when registering clocks. + * + * The default of 500 parts per million is based on NTP's limits. + * If a clocksource is good enough for NTP, it is good enough for us! */ #ifdef CONFIG_CLOCKSOURCE_WATCHDOG_MAX_SKEW_US #define MAX_SKEW_USEC CONFIG_CLOCKSOURCE_WATCHDOG_MAX_SKEW_US #else -#define MAX_SKEW_USEC 100 +#define MAX_SKEW_USEC (125 * WATCHDOG_INTERVAL / HZ) #endif #define WATCHDOG_MAX_SKEW (MAX_SKEW_USEC * NSEC_PER_USEC) @@ -140,11 +148,6 @@ static inline void clocksource_watchdog_unlock(unsigned long *flags) static int clocksource_watchdog_kthread(void *data); static void __clocksource_change_rating(struct clocksource *cs, int rating); -/* - * Interval: 0.5sec. - */ -#define WATCHDOG_INTERVAL (HZ >> 1) - static void clocksource_watchdog_work(struct work_struct *work) { /* From patchwork Thu Jan 5 01:06:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 39268 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp56439wrt; Wed, 4 Jan 2023 17:11:43 -0800 (PST) X-Google-Smtp-Source: AMrXdXuCCv62JnOBdgMwKEzFw5JHCI38cA4JR82NerPcX2eCpKnMllY+ODjKyuEyTYqQuxZ5WVTD X-Received: by 2002:a17:90b:485:b0:223:ffd0:b2a1 with SMTP id bh5-20020a17090b048500b00223ffd0b2a1mr55025333pjb.48.1672881102760; Wed, 04 Jan 2023 17:11:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672881102; cv=none; d=google.com; s=arc-20160816; b=fE7jtewng7rM93yqn1VdA6RSHgXZSrieeGOes3LXURPLtC1gUOUFWGQ2+pdRh9SnF1 W6+SMTrMaehhJDtMTlevVKU8kziVcg0dXEiaCMp1w5aawVbKFdea6XCiEwDkFpPr0+0t JvzEutrUaI9PqkloCraMYm2d+WkbSKrQ1SZuHOw/rqeFhLDe//kV55c/sVwNK7WdHnVS pijH12+VIe9nKZ+UVm6p8tnRco7chmDa2ky8M/jQltlAddp8lXeHtmF8tV6O7proSnSH WLVwcEYvZi4onQOGuw5Bubj/bm8OE8s9ZTh9p6K4np1BkhRGCKAI+oBhTx3nwRzJafp3 D47Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=+OndCycZAVKgCubNF85TmtspvEXQeM4SpV0jmeM03ZM=; b=sUQ2BB50Ji2LBjV9apR3FQtYivn7RamWAEGCvxaTo4vq7SMyq+air9IsnV1KZVwlop S0XD51Ap7/WWcGT3BcojKg2OlcVsKY9ICP35YMnqprSALvAerR42iHsoJvt2qy/Id6AH fpLLiJFWTaYA2ZlKJqC08B57YjmY7vwyn9F6vV8+pR1NsfRJd1gq/S6h1bKq07mapucJ kJLbfi557V3xJ7XPeM6zoMXw3Nendio0knA7fHkWkoTqMDgrtm8OracCGezVjhZ3OG/9 SzlOaAJPJ82GRD20fIGKhzhdrfKchUOFcdYKvaao25Q5O/m8fPUpB+FEW3gZZXiAGYid PCSw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=r+YiZtfF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h16-20020a635310000000b00479391226edsi19014082pgb.328.2023.01.04.17.11.28; Wed, 04 Jan 2023 17:11:42 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=r+YiZtfF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230093AbjAEBHQ (ORCPT + 99 others); Wed, 4 Jan 2023 20:07:16 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44156 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229923AbjAEBHG (ORCPT ); Wed, 4 Jan 2023 20:07:06 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 87C7530562 for ; Wed, 4 Jan 2023 17:07:05 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 47BA3B81980 for ; Thu, 5 Jan 2023 01:07:04 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id F1E9DC433F0; Thu, 5 Jan 2023 01:07:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1672880823; bh=ZxflhOZe+HfSeKoe2YE5lW7UCVsim4bFg/Q32iI5pZk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=r+YiZtfFBZfzgNj7s7ovifNLZtJ4uYwpTFf2uaMsVsEhbEAqq8IBqdUsdYNYAvKKB jtWs0m44JrzvmW7SKXdV56RWB8TYOTl68yfgziyO3MXmSxd12eRpSYGuu3gHnyp3Mz HoSkfWWe/Rct3zheX2cXZC9bvaesj6OrP30Rc+iFK6Jdm82dHEehZ2ryOL0bsWUzwk g9C7GuLc9uU7WeFuEbztQGZECqJZVai9rxCrx/E+r7MG6+ua0d5kXPaG9HfBRkjFHg kQRMrauSSJPMXMJs/D7i9Mqavt4UmiBI6qOqD3ZxhyD03asnfwC9o3NLxhDnJt4uXl DJp/0cIRqjZFA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id AC8935C08E5; Wed, 4 Jan 2023 17:07:02 -0800 (PST) From: "Paul E. McKenney" To: tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, john.stultz@linaro.org, sboyd@kernel.org, corbet@lwn.net, Mark.Rutland@arm.com, maz@kernel.org, kernel-team@meta.com, neeraju@codeaurora.org, ak@linux.intel.com, feng.tang@intel.com, zhengjun.xing@intel.com, "Paul E. McKenney" , John Stultz Subject: [PATCH clocksource 3/6] clocksource: Improve read-back-delay message Date: Wed, 4 Jan 2023 17:06:58 -0800 Message-Id: <20230105010701.1773895-3-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20230105010429.GA1773522@paulmck-ThinkPad-P17-Gen-1> References: <20230105010429.GA1773522@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754142975183159883?= X-GMAIL-MSGID: =?utf-8?q?1754142975183159883?= When cs_watchdog_read() is unable to get a qualifying clocksource read within the limit set by max_cswd_read_retries, it prints a message and marks the clocksource under test as unstable. But that message is unclear to anyone unfamiliar with the code: clocksource: timekeeping watchdog on CPU13: wd-tsc-wd read-back delay 1000614ns, attempt 3, marking unstable Therefore, add some context so that the message appears as follows: clocksource: timekeeping watchdog on CPU13: wd-tsc-wd excessive read-back delay of 1000614ns vs. limit of 125000ns, wd-wd read-back delay only 27ns, attempt 3, marking tsc unstable Signed-off-by: Paul E. McKenney Cc: John Stultz Cc: Thomas Gleixner Cc: Stephen Boyd Cc: Feng Tang --- kernel/time/clocksource.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/time/clocksource.c b/kernel/time/clocksource.c index a3d19f6660ac7..b59914953809f 100644 --- a/kernel/time/clocksource.c +++ b/kernel/time/clocksource.c @@ -260,8 +260,8 @@ static enum wd_read_status cs_watchdog_read(struct clocksource *cs, u64 *csnow, goto skip_test; } - pr_warn("timekeeping watchdog on CPU%d: wd-%s-wd read-back delay of %lldns, attempt %d, marking unstable\n", - smp_processor_id(), cs->name, wd_delay, nretries); + pr_warn("timekeeping watchdog on CPU%d: wd-%s-wd excessive read-back delay of %lldns vs. limit of %ldns, wd-wd read-back delay only %lldns, attempt %d, marking %s unstable\n", + smp_processor_id(), cs->name, wd_delay, WATCHDOG_MAX_SKEW, wd_seq_delay, nretries, cs->name); return WD_READ_UNSTABLE; skip_test: From patchwork Thu Jan 5 01:06:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 39266 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp56395wrt; Wed, 4 Jan 2023 17:11:37 -0800 (PST) X-Google-Smtp-Source: AMrXdXvWkjwn/C5lJgkTrGT4uM6VU58RaJ/lxU5nCcSMYk9K2SusW/rrc2nE+Q8bvZRYbPmYCYhv X-Received: by 2002:a17:90b:5112:b0:226:7b3e:dea5 with SMTP id sc18-20020a17090b511200b002267b3edea5mr11700918pjb.36.1672881097048; Wed, 04 Jan 2023 17:11:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672881097; cv=none; d=google.com; s=arc-20160816; b=T/UyMN6JuxXy0un/LViFzcIKV3SrjcaLQNQlBoUTdZtwo/7g0ZWO1Jf26L0KrA9K87 QyoNOyGTBhblufol8GxANhdJSz6Cz9IDW7p0KjvjxS/EwAI+XSa4k0/DS+vcsVhwdOcb aWHyrAvlRJRNukIEQfHppdicYz4cWl1oa4ytYdwoquAIS5fy614SCxeNCw+CdVemova7 Na/ShjsJ4GlpQTxf7tyxA/1jT2/CT9AkdaxoTbM0vZarCgs0jrE5X9MDG+/OnvubF82T Buas2ZBWnA4sdP8U23ZNPo/bLd0Uv5wNliue/XVcxEiZ/ZK4N3QRSWidYmKvzTmnELVM huvQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=RQxIpGJT4sfcpMxaoESE8FxdxjTkXJ1xjq85g/brPxM=; b=HZPcAPAF2cwluIqrgjsJA1PT6NTr+mGjKaJVUaamS9H1DYAPAO0S1W8qpUTtOcfJpF cPDpN+1sl/QP+ufLWJM6ymxS4akSuCe5Aax6E+NUKqg6f8ZyZva5If6eV0dilswUrzeB lOsBMvcTqgxCeBWarPUV3v37R95ShiVKauj8e/xXSyU00BfN5ZXV4YJGFv6DcmFZQs0N HhHaIVREbowizlgn2VcNntQeLa0RsjSSIxub6GmHOfUL9dkHtOLlLMYJAFC2n6pKvEvt Qz6/u3y7zXnFUc+8OWDUsPiFVH92z/I+SPjd6+oE8a2ZdtBtRqvQnwbjk6Jom7ggx0KC iBoQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=O1KAGj6R; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id hf21-20020a17090aff9500b00226ab1b3b8dsi459053pjb.83.2023.01.04.17.11.24; Wed, 04 Jan 2023 17:11:37 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=O1KAGj6R; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230061AbjAEBHJ (ORCPT + 99 others); Wed, 4 Jan 2023 20:07:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44110 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229773AbjAEBHF (ORCPT ); Wed, 4 Jan 2023 20:07:05 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1F8EB2F7B5 for ; Wed, 4 Jan 2023 17:07:04 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id A533A618A0 for ; Thu, 5 Jan 2023 01:07:03 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 06928C433D2; Thu, 5 Jan 2023 01:07:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1672880823; bh=AJeR1T6OoNxbdITdFAHVVUIQjCuiYAgVS5ruYDD+l3E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=O1KAGj6RD8nDceB2/3J13gdWOW0YT7n2SIQMEb4PaMuUAM7LnhCfDq3js9BjHKySb Z7ykct1b8BpsoWmDMN2qkDIwX8zsr8wJ64WPQXUJ14Bz7G8IvKUjxG4RLRukCSBzv8 +/kzgQERoNL95VW4luKNjAvIPPLkeMKskJjvONzgvcehd2wPAG0mGUsoosTLb5KfUh TLCY7rNPV5EQOFxvSxJWhjeyLJLzfIy81E/CB3VvfGNTQL5oPZhVDWnGkEeCJ2CYs1 4uq6xtdNPEdNIazbr7T+uGCOUt+t+SbVSznSRCJ7HdQZ2zUDL1OG0LnmFW8Zryh9ME EOYPSWsm5zvPg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id AE78E5C1456; Wed, 4 Jan 2023 17:07:02 -0800 (PST) From: "Paul E. McKenney" To: tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, john.stultz@linaro.org, sboyd@kernel.org, corbet@lwn.net, Mark.Rutland@arm.com, maz@kernel.org, kernel-team@meta.com, neeraju@codeaurora.org, ak@linux.intel.com, feng.tang@intel.com, zhengjun.xing@intel.com, "Paul E. McKenney" , John Stultz Subject: [PATCH clocksource 4/6] clocksource: Improve "skew is too large" messages Date: Wed, 4 Jan 2023 17:06:59 -0800 Message-Id: <20230105010701.1773895-4-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20230105010429.GA1773522@paulmck-ThinkPad-P17-Gen-1> References: <20230105010429.GA1773522@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754142968945328195?= X-GMAIL-MSGID: =?utf-8?q?1754142968945328195?= When clocksource_watchdog() detects excessive clocksource skew compared to the watchdog clocksource, it marks the clocksource under test as unstable and prints several lines worth of message. But that message is unclear to anyone unfamiliar with the code: clocksource: timekeeping watchdog on CPU2: Marking clocksource 'wdtest-ktime' as unstable because the skew is too large: clocksource: 'kvm-clock' wd_nsec: 400744390 wd_now: 612625c2c wd_last: 5fa7f7c66 mask: ffffffffffffffff clocksource: 'wdtest-ktime' cs_nsec: 600744034 cs_now: 173081397a292d4f cs_last: 17308139565a8ced mask: ffffffffffffffff clocksource: 'kvm-clock' (not 'wdtest-ktime') is current clocksource. Therefore, add the following line near the end of that message: Clocksource 'wdtest-ktime' skewed 199999644 ns (199 ms) over watchdog 'kvm-clock' interval of 400744390 ns (400 ms) This new line clearly indicates the amount of skew between the two clocksources, along with the duration of the time interval over which the skew occurred, both in nanoseconds and milliseconds. Cc: John Stultz Cc: Thomas Gleixner Cc: Stephen Boyd Cc: Feng Tang Signed-off-by: Paul E. McKenney --- kernel/time/clocksource.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/kernel/time/clocksource.c b/kernel/time/clocksource.c index b59914953809f..fc486cd972635 100644 --- a/kernel/time/clocksource.c +++ b/kernel/time/clocksource.c @@ -446,12 +446,20 @@ static void clocksource_watchdog(struct timer_list *unused) /* Check the deviation from the watchdog clocksource. */ md = cs->uncertainty_margin + watchdog->uncertainty_margin; if (abs(cs_nsec - wd_nsec) > md) { + u64 cs_wd_msec; + u64 wd_msec; + u32 wd_rem; + pr_warn("timekeeping watchdog on CPU%d: Marking clocksource '%s' as unstable because the skew is too large:\n", smp_processor_id(), cs->name); pr_warn(" '%s' wd_nsec: %lld wd_now: %llx wd_last: %llx mask: %llx\n", watchdog->name, wd_nsec, wdnow, wdlast, watchdog->mask); pr_warn(" '%s' cs_nsec: %lld cs_now: %llx cs_last: %llx mask: %llx\n", cs->name, cs_nsec, csnow, cslast, cs->mask); + cs_wd_msec = div_u64_rem(cs_nsec - wd_nsec, 1000U * 1000U, &wd_rem); + wd_msec = div_u64_rem(wd_nsec, 1000U * 1000U, &wd_rem); + pr_warn(" Clocksource '%s' skewed %lld ns (%lld ms) over watchdog '%s' interval of %lld ns (%lld ms)\n", + cs->name, cs_nsec - wd_nsec, cs_wd_msec, watchdog->name, wd_nsec, wd_msec); if (curr_clocksource == cs) pr_warn(" '%s' is current clocksource.\n", cs->name); else if (curr_clocksource) From patchwork Thu Jan 5 01:07:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 39274 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp56853wrt; Wed, 4 Jan 2023 17:12:51 -0800 (PST) X-Google-Smtp-Source: AMrXdXuALinHKdFAh8618mWxk5P4srbcGVFMVyo0Y+2ZHk3P2hAPAqj2zkWv4HJpheB6K/qfZVsD X-Received: by 2002:a17:902:b08e:b0:192:6195:44b6 with SMTP id p14-20020a170902b08e00b00192619544b6mr41511527plr.58.1672881171472; Wed, 04 Jan 2023 17:12:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672881171; cv=none; d=google.com; s=arc-20160816; b=x6uR5PTR7pT66AnRHeiPtn1yonotDY5lr49I4lFmGMdFEWHzitH80hpPJ38+BWcz0R J3/9VKrap1+14N9SElR/FNKGhAQbvozW+KdLQCL8MWpXPEIekrs6dZJaKe/AG5ccjb61 0Cbz9BtRNCufQZJRgMhr4PyxuE2oCXQZcN7qI/plVlpqlIckVPr0gp3lJowZB/kQMeZk wEhmCwllsxbIktPBhf+TrofNanuSFMEQLvwAjIhXpYNZH5X+wC9ruIhztbkEfE1mz2vu 6o/FyUX4YXioqEd+uXsIBvICUK/ZxkaySyFAkQo40dL7p+Vnc1WuF5MkJYu37U5dK/kR PMMg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=2AyDK4NDcvtUNTtF6RCfDl995m3SzxJWJC/92AqlVI4=; b=RtDjavfuB/kHlfOPvYIo3mh/d4qYEAKZ2d5pi7xpf1ufZ+rljwsszaeAIpYqnUgO53 KqdNz+huiwOU6dvPu2QiWyYu/Du+cpWR/E09tmcIM7Ks2KW7F5Spi5brkjQFYSupjAZr HQtiqGQuCe+quOa7jelW4tWxolBJCX+IpXBMjU4/xoJal+NDA2+TF85msSKAn2GTSd40 xpODw3+d41pkvrcuG7kAvPjn7o/Ao8PeZEFErezdDgqNRVnfSz2WYInn6ROIZ4uSEyMU UjTq45JltaEV7M/DcfuoDEs6vJ8CLMnGhnV36Oqui3ljltH+sYXqCYqUwDy3sgNW06Jy H6rA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=nKZYdpk4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id n4-20020a170902968400b0019260195003si32446844plp.599.2023.01.04.17.12.39; Wed, 04 Jan 2023 17:12:51 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=nKZYdpk4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230106AbjAEBHU (ORCPT + 99 others); Wed, 4 Jan 2023 20:07:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44166 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229978AbjAEBHG (ORCPT ); Wed, 4 Jan 2023 20:07:06 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B63302F7B5 for ; Wed, 4 Jan 2023 17:07:05 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 68230B81986 for ; Thu, 5 Jan 2023 01:07:04 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0A0E3C43396; Thu, 5 Jan 2023 01:07:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1672880823; bh=hcTQTn+b1sZvh82B5YLHkY2GewljRr5BEWwiBULEmq8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nKZYdpk4iFosWUndbKPesO16saq43HPQBKWgqd0ZxFasfORyaxeWX8oHsN7AINfAz Q/NEgFjgVN0j3K5ZD9jORuWllNjETVsSmotvuwb44S1v3Et16P8p4ls0IBGnwfomYA hYCW+TdYcLb7xe7iws8bzMcgRkYpTeSqe/HicfJDoEA/GFMdByZV3kGsYZtjY+7L9Y JFH28SlVq0gfV26lCY6/GOSBMgf6S5BWlByAyHR5fYFKteNFlth1IyPCidCrx2bDc/ uWl61E2m+8X0K4gAnOpM4pjmvNm71Fc7SkhUgWVAoYttSzbk7RARmy7Qx9BIS70rf1 JbhlatiIXeeEQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id B0B315C149B; Wed, 4 Jan 2023 17:07:02 -0800 (PST) From: "Paul E. McKenney" To: tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, john.stultz@linaro.org, sboyd@kernel.org, corbet@lwn.net, Mark.Rutland@arm.com, maz@kernel.org, kernel-team@meta.com, neeraju@codeaurora.org, ak@linux.intel.com, feng.tang@intel.com, zhengjun.xing@intel.com, Waiman Long , John Stultz , "Paul E . McKenney" Subject: [PATCH clocksource 5/6] clocksource: Suspend the watchdog temporarily when high read latency detected Date: Wed, 4 Jan 2023 17:07:00 -0800 Message-Id: <20230105010701.1773895-5-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20230105010429.GA1773522@paulmck-ThinkPad-P17-Gen-1> References: <20230105010429.GA1773522@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754143047042245252?= X-GMAIL-MSGID: =?utf-8?q?1754143047042245252?= From: Feng Tang Bugs have been reported on 8 sockets x86 machines in which the TSC was wrongly disabled when the system is under heavy workload. [ 818.380354] clocksource: timekeeping watchdog on CPU336: hpet wd-wd read-back delay of 1203520ns [ 818.436160] clocksource: wd-tsc-wd read-back delay of 181880ns, clock-skew test skipped! [ 819.402962] clocksource: timekeeping watchdog on CPU338: hpet wd-wd read-back delay of 324000ns [ 819.448036] clocksource: wd-tsc-wd read-back delay of 337240ns, clock-skew test skipped! [ 819.880863] clocksource: timekeeping watchdog on CPU339: hpet read-back delay of 150280ns, attempt 3, marking unstable [ 819.936243] tsc: Marking TSC unstable due to clocksource watchdog [ 820.068173] TSC found unstable after boot, most likely due to broken BIOS. Use 'tsc=unstable'. [ 820.092382] sched_clock: Marking unstable (818769414384, 1195404998) [ 820.643627] clocksource: Checking clocksource tsc synchronization from CPU 267 to CPUs 0,4,25,70,126,430,557,564. [ 821.067990] clocksource: Switched to clocksource hpet This can be reproduced by running memory intensive 'stream' tests, or some of the stress-ng subcases such as 'ioport'. The reason for these issues is the when system is under heavy load, the read latency of the clocksources can be very high. Even lightweight TSC reads can show high latencies, and latencies are much worse for external clocksources such as HPET or the APIC PM timer. These latencies can result in false-positive clocksource-unstable determinations. Given that the clocksource watchdog is a continual diagnostic check with frequency of twice a second, there is no need to rush it when the system is under heavy load. Therefore, when high clocksource read latencies are detected, suspend the watchdog timer for 5 minutes. Signed-off-by: Feng Tang Acked-by: Waiman Long Cc: John Stultz Cc: Thomas Gleixner Cc: Stephen Boyd Cc: Feng Tang Signed-off-by: Paul E. McKenney --- kernel/time/clocksource.c | 45 ++++++++++++++++++++++++++++----------- 1 file changed, 32 insertions(+), 13 deletions(-) diff --git a/kernel/time/clocksource.c b/kernel/time/clocksource.c index fc486cd972635..91836b727cef5 100644 --- a/kernel/time/clocksource.c +++ b/kernel/time/clocksource.c @@ -387,6 +387,15 @@ void clocksource_verify_percpu(struct clocksource *cs) } EXPORT_SYMBOL_GPL(clocksource_verify_percpu); +static inline void clocksource_reset_watchdog(void) +{ + struct clocksource *cs; + + list_for_each_entry(cs, &watchdog_list, wd_list) + cs->flags &= ~CLOCK_SOURCE_WATCHDOG; +} + + static void clocksource_watchdog(struct timer_list *unused) { u64 csnow, wdnow, cslast, wdlast, delta; @@ -394,6 +403,7 @@ static void clocksource_watchdog(struct timer_list *unused) int64_t wd_nsec, cs_nsec; struct clocksource *cs; enum wd_read_status read_ret; + unsigned long extra_wait = 0; u32 md; spin_lock(&watchdog_lock); @@ -413,13 +423,30 @@ static void clocksource_watchdog(struct timer_list *unused) read_ret = cs_watchdog_read(cs, &csnow, &wdnow); - if (read_ret != WD_READ_SUCCESS) { - if (read_ret == WD_READ_UNSTABLE) - /* Clock readout unreliable, so give it up. */ - __clocksource_unstable(cs); + if (read_ret == WD_READ_UNSTABLE) { + /* Clock readout unreliable, so give it up. */ + __clocksource_unstable(cs); continue; } + /* + * When WD_READ_SKIP is returned, it means the system is likely + * under very heavy load, where the latency of reading + * watchdog/clocksource is very big, and affect the accuracy of + * watchdog check. So give system some space and suspend the + * watchdog check for 5 minutes. + */ + if (read_ret == WD_READ_SKIP) { + /* + * As the watchdog timer will be suspended, and + * cs->last could keep unchanged for 5 minutes, reset + * the counters. + */ + clocksource_reset_watchdog(); + extra_wait = HZ * 300; + break; + } + /* Clocksource initialized ? */ if (!(cs->flags & CLOCK_SOURCE_WATCHDOG) || atomic_read(&watchdog_reset_pending)) { @@ -523,7 +550,7 @@ static void clocksource_watchdog(struct timer_list *unused) * pair clocksource_stop_watchdog() clocksource_start_watchdog(). */ if (!timer_pending(&watchdog_timer)) { - watchdog_timer.expires += WATCHDOG_INTERVAL; + watchdog_timer.expires += WATCHDOG_INTERVAL + extra_wait; add_timer_on(&watchdog_timer, next_cpu); } out: @@ -548,14 +575,6 @@ static inline void clocksource_stop_watchdog(void) watchdog_running = 0; } -static inline void clocksource_reset_watchdog(void) -{ - struct clocksource *cs; - - list_for_each_entry(cs, &watchdog_list, wd_list) - cs->flags &= ~CLOCK_SOURCE_WATCHDOG; -} - static void clocksource_resume_watchdog(void) { atomic_inc(&watchdog_reset_pending); From patchwork Thu Jan 5 01:07:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 39267 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp56415wrt; Wed, 4 Jan 2023 17:11:39 -0800 (PST) X-Google-Smtp-Source: AMrXdXuGbQRQ5H+SfGm45joqzp0o26Ruhl3U8/X78jjLrRzrUAvnWCXlpnMRA9hYYGJ7PVwO5ah+ X-Received: by 2002:a17:902:f08a:b0:189:efe8:1e with SMTP id p10-20020a170902f08a00b00189efe8001emr48250382pla.68.1672881098974; Wed, 04 Jan 2023 17:11:38 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672881098; cv=none; d=google.com; s=arc-20160816; b=yGfAvotOIAuJhuEHnRjlgVYWRuPjSsg0eqdfS+cF24J9xnWEtdszkcNH7NPGcBbi36 f1nvLB8Xs8CbP19pk4qKz9nyhR3LPOe6ExDLCcosb0zFWh+3BlYqFahpL6IuYOIRk4rY eo9yCF0HioWgDXwBM3rANuEiu62CvMMmFI2PT7zgB9LOw8x75ILwHBaYTwJHiSIL2T3L vIFswEvlB1YuqkX0+9Kf8PCZEQyFiJmMCOoo00jkfmM5T6j+9SqUu1sXxkJj2MHdnoA4 STZnll9wB5PBolcPlzdOcfUqUxGKw1zSX8JCPTp96ej9QzvJWbcJwJX1DTZazyfhPTGt mmGQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=+5lfXKrb94RmbGe46p5qw4mDYOVWIvCqBBUItnDAHpM=; b=xwp4YBw5nzw/X0sHxgsfKUfzKBxD1dtZKG+uhPtnDeSvRAEyVHWTLQvWdTrskqU9o0 1S8nyQ9zWprtsMiSRI+xK0D0ivuCNYO9oHe7YDuNuE208PrJQqv9S+9vn8atXgZfscm9 FR53GodDz/WEhBDSFcCHPg6zVZP5BXv/0GM8Ljf4avCszjWQC2aOsQBRiuXeIP2yJMyJ bY9z3MIAPwB68mXRrZuwX6bOqE/1Lk64wHgjOFPIQjl3IddgyJmxuUgqbR/iZSDHZWFF NITwbHe5K3vHhbdRE3dRTm1tw+clSLT8J3hNfxk/Wfd6adM5d9xBmN4TmahZtoswJluC 3kNg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=m0KxLN5J; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b1-20020a170903228100b0017f75654a51si3888128plh.245.2023.01.04.17.11.25; Wed, 04 Jan 2023 17:11:38 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=m0KxLN5J; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230083AbjAEBHN (ORCPT + 99 others); Wed, 4 Jan 2023 20:07:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44116 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229906AbjAEBHF (ORCPT ); Wed, 4 Jan 2023 20:07:05 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8AC343055C for ; Wed, 4 Jan 2023 17:07:04 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 26D1761896 for ; Thu, 5 Jan 2023 01:07:04 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5F3B5C433AE; Thu, 5 Jan 2023 01:07:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1672880823; bh=/IAu4A8czS9Ro0tYmgqlY7WbRqRFFlgq0vaHiJqT+No=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=m0KxLN5JEXw/4+Ba7IOnj0Hvvm8LbLPFx8JJyYLkkOc4RsFGjK9/EVnQ8t4CafMNb sybTvhPX02Y3nDI301MUT3axdSlaKktepbaJVwCKVHKAWEUyvlrFFYhthot5cBzPuM muTumVrEzFWUSiwwvTuxND99dVqp0TsI3xVSjnJkI4iCLjnyj/91ovEheWlMKwzWKc DHqFpEoDCsUtPkQi5OsDRicxJez4Culc9bt97KSyEAZSxKxSzGnL53Cz/qAFddzwA3 zRcomGuCLcvWEQPXrx7sm3rvJTUf+pO3UtNQPuic+ISDBIGMwJIxmFrSef2Q8HfO3n gA5gwJdR7VZ8A== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id B29EB5C1ADF; Wed, 4 Jan 2023 17:07:02 -0800 (PST) From: "Paul E. McKenney" To: tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, john.stultz@linaro.org, sboyd@kernel.org, corbet@lwn.net, Mark.Rutland@arm.com, maz@kernel.org, kernel-team@meta.com, neeraju@codeaurora.org, ak@linux.intel.com, feng.tang@intel.com, zhengjun.xing@intel.com, "Paul E. McKenney" , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Daniel Lezcano , Waiman Long , x86@kernel.org Subject: [PATCH clocksource 6/6] clocksource: Verify HPET and PMTMR when TSC unverified Date: Wed, 4 Jan 2023 17:07:01 -0800 Message-Id: <20230105010701.1773895-6-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20230105010429.GA1773522@paulmck-ThinkPad-P17-Gen-1> References: <20230105010429.GA1773522@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754142971158058299?= X-GMAIL-MSGID: =?utf-8?q?1754142971158058299?= On systems with two or fewer sockets, when the boot CPU has CONSTANT_TSC, NONSTOP_TSC, and TSC_ADJUST, clocksource watchdog verification of the TSC is disabled. This works well much of the time, but there is the occasional system that meets all of these criteria, but which still has a TSC that skews significantly from atomic-clock time. This is usually attributed to a firmware or hardware fault. Yes, the various NTP daemons do express their opinions of userspace-to-atomic-clock time skew, but they put them in various places, depending on the daemon and distro in question. It would therefore be good for the kernel to have some clue that there is a problem. The old behavior of marking the TSC unstable is a non-starter because a great many workloads simply cannot tolerate the overheads and latencies of the various non-TSC clocksources. In addition, NTP-corrected systems often seem to be able to tolerate significant kernel-space time skew as long as the userspace time sources are within epsilon of atomic-clock time. Therefore, when watchdog verification of TSC is disabled, enable it for HPET and PMTMR (AKA ACPI PM timer). This provides the needed in-kernel time-skew diagnostic without degrading the system's performance. Signed-off-by: Paul E. McKenney Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: "H. Peter Anvin" Cc: Daniel Lezcano Cc: Waiman Long Cc: Tested-by: Feng Tang --- arch/x86/include/asm/time.h | 1 + arch/x86/kernel/hpet.c | 2 ++ arch/x86/kernel/tsc.c | 5 +++++ drivers/clocksource/acpi_pm.c | 6 ++++-- 4 files changed, 12 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/time.h b/arch/x86/include/asm/time.h index 8ac563abb567b..a53961c64a567 100644 --- a/arch/x86/include/asm/time.h +++ b/arch/x86/include/asm/time.h @@ -8,6 +8,7 @@ extern void hpet_time_init(void); extern void time_init(void); extern bool pit_timer_init(void); +extern bool tsc_clocksource_watchdog_disabled(void); extern struct clock_event_device *global_clock_event; diff --git a/arch/x86/kernel/hpet.c b/arch/x86/kernel/hpet.c index 71f336425e58a..c8eb1ac5125ab 100644 --- a/arch/x86/kernel/hpet.c +++ b/arch/x86/kernel/hpet.c @@ -1091,6 +1091,8 @@ int __init hpet_enable(void) if (!hpet_counting()) goto out_nohpet; + if (tsc_clocksource_watchdog_disabled()) + clocksource_hpet.flags |= CLOCK_SOURCE_MUST_VERIFY; clocksource_register_hz(&clocksource_hpet, (u32)hpet_freq); if (id & HPET_ID_LEGSUP) { diff --git a/arch/x86/kernel/tsc.c b/arch/x86/kernel/tsc.c index a78e73da4a74b..af3782fb6200c 100644 --- a/arch/x86/kernel/tsc.c +++ b/arch/x86/kernel/tsc.c @@ -1186,6 +1186,11 @@ static void __init tsc_disable_clocksource_watchdog(void) clocksource_tsc.flags &= ~CLOCK_SOURCE_MUST_VERIFY; } +bool tsc_clocksource_watchdog_disabled(void) +{ + return !(clocksource_tsc.flags & CLOCK_SOURCE_MUST_VERIFY); +} + static void __init check_system_tsc_reliable(void) { #if defined(CONFIG_MGEODEGX1) || defined(CONFIG_MGEODE_LX) || defined(CONFIG_X86_GENERIC) diff --git a/drivers/clocksource/acpi_pm.c b/drivers/clocksource/acpi_pm.c index 279ddff81ab49..82338773602ca 100644 --- a/drivers/clocksource/acpi_pm.c +++ b/drivers/clocksource/acpi_pm.c @@ -23,6 +23,7 @@ #include #include #include +#include /* * The I/O port the PMTMR resides at. @@ -210,8 +211,9 @@ static int __init init_acpi_pm_clocksource(void) return -ENODEV; } - return clocksource_register_hz(&clocksource_acpi_pm, - PMTMR_TICKS_PER_SEC); + if (tsc_clocksource_watchdog_disabled()) + clocksource_acpi_pm.flags |= CLOCK_SOURCE_MUST_VERIFY; + return clocksource_register_hz(&clocksource_acpi_pm, PMTMR_TICKS_PER_SEC); } /* We use fs_initcall because we want the PCI fixups to have run