From patchwork Wed Oct 25 21:31:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Gleixner X-Patchwork-Id: 158304 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:d641:0:b0:403:3b70:6f57 with SMTP id cy1csp250462vqb; Wed, 25 Oct 2023 14:32:04 -0700 (PDT) X-Google-Smtp-Source: AGHT+IE0UXRgW7YFJoA9vUpjDjX+XVfs6CqMbXKEqkQOfT/DRaQzYrVEQ2gCP/LUMWVDoccaOpPY X-Received: by 2002:a05:6830:4d3:b0:6ce:2733:f71b with SMTP id s19-20020a05683004d300b006ce2733f71bmr17253249otd.26.1698269523877; Wed, 25 Oct 2023 14:32:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698269523; cv=none; d=google.com; s=arc-20160816; b=07T5ZXt7R1wnivlI+9AM6HGVNy5LYV/4r16gPibQBeh0c2w0lm3vI9kjDHDjelEKOn 8f/3+1na6Cv+q+eL4HKht4WVc4pkjVQwWMoeXEViqgeouBNraEriYuBW39ZagXakxLVa rAw3HZgpmVDA/PPR7LpQ3wt50FGKtHJr/oOr5lkJkNRrmb36zJncn8fUn7m1x6TnzS+j rdWmAP40N3xNixGX1z/nVeuMZtZKWEJCWKlbJf08i5PdsC/Kh6xgvE4U056wMfsklJCI 7WhYCBAusdk5GYzZP0KpFnlDuVIU4od0gynnA/rarTflUDKe3poe7JvgrRH3tw2vTlcG QtMA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:message-id:date:references :in-reply-to:subject:cc:to:dkim-signature:dkim-signature:from; bh=xcDcsO81M9vK0hceytiQ1PbO/SvCwbp0ScdGs67d9so=; fh=CMEqvHWMR129yJD6ZDMNtvwByWOnDu4oIuwgun12RTY=; b=vntNM4cSwrTLkdMUUsfq9FyajLUAVWQ5/RQjG/OWqrBSvy6r09bfRtHdeilGwIini3 UYZj7YHgQ697KIAlcqCnQyS/BQEonQjsek5AwNfMAYsJZm0yashloJdw76Vkq6VdeaBY CeFuwYO9VDks/7suWRJYO0ujw+4rNDglZ5+dqGyev798aMNNC9rwBPa86AW6HFtI7Ner 1evALC5/nWuNClHX9rK55U7IQk6N/qcQn6x5QmPMWfBJ4SZPdde0SoHLG7RuvkwV3hxK m2oMSJp5sQNv0pMJtUoC3+5vEibPsjgEB6wjyv7O7djA49I2FM4FowYhpAubZzktzHvP p1Cw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=1co65Z8h; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from agentk.vger.email (agentk.vger.email. [23.128.96.32]) by mx.google.com with ESMTPS id k66-20020a819345000000b005920925b1e0si13991109ywg.184.2023.10.25.14.31.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Oct 2023 14:32:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=1co65Z8h; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 1350980873BB; Wed, 25 Oct 2023 14:31:54 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230046AbjJYVbk (ORCPT + 25 others); Wed, 25 Oct 2023 17:31:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52078 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230120AbjJYVbj (ORCPT ); Wed, 25 Oct 2023 17:31:39 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 18DDD132 for ; Wed, 25 Oct 2023 14:31:37 -0700 (PDT) From: Thomas Gleixner DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1698269495; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=xcDcsO81M9vK0hceytiQ1PbO/SvCwbp0ScdGs67d9so=; b=1co65Z8hfV62olLoJAmJt46iO9SecDx8c35M7OrFrsM4FCcQAuoiMJANaJaTElgaxQERf+ Xchc0f1sgogBRqm1N0lXxbS7YJ2wCOgYmkwOM92aV6m4h0q+9QGnutI2wMs8gGmeA7RRB8 BAP/I+sjDF1h5zRYDiIJioh9OuhZM1NsKkpaY5RYabeov4UjAuGUU5Hfp39PksyoUw+4Qt Dq6c16+EYhwr28H+7U8HlWYa4XFFa7l6jkRcep7Avt1xW+CJH5oAaBFAA26sI19ORuF4H2 sJdtPeJm1xX69Jij+PjKq7/2LrgJaxvJrXzYIhAgLFnTkuNr/K9dA7GcbJcq2Q== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1698269495; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=xcDcsO81M9vK0hceytiQ1PbO/SvCwbp0ScdGs67d9so=; b=bjd5BGpVaX8zgazGbahtwdKQbJPM0SYQOw1EsizBKJffYkuf8fQD1iOkAHzMZ651JgNNYf 7Myq4/OyHH9sU/Ag== To: Tetsuo Handa , John Stultz Cc: Stephen Boyd , LKML , Sebastian Andrzej Siewior , "Paul E. McKenney" , x86@kernel.org Subject: [PATCH] x86/tsc: Defer marking TSC unstable to a worker In-Reply-To: <90361195-4309-4a02-bd3f-8ee606e6d35b@I-love.SAKURA.ne.jp> References: <80ff5036-8449-44a6-ba2f-0130d3be6b57@I-love.SAKURA.ne.jp> <878r826xys.ffs@tglx> <90361195-4309-4a02-bd3f-8ee606e6d35b@I-love.SAKURA.ne.jp> Date: Wed, 25 Oct 2023 23:31:35 +0200 Message-ID: <87zg064ceg.ffs@tglx> MIME-Version: 1.0 X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Wed, 25 Oct 2023 14:31:54 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780764664303549376 X-GMAIL-MSGID: 1780764664303549376 Tetsuo reported the following lockdep splat when the TSC synchronization fails during CPU hotplug: tsc: Marking TSC unstable due to check_tsc_sync_source failed WARNING: inconsistent lock state inconsistent {IN-HARDIRQ-W} -> {HARDIRQ-ON-W} usage. ffffffff8cfa1c78 (watchdog_lock){?.-.}-{2:2}, at: clocksource_watchdog+0x23/0x5a0 {IN-HARDIRQ-W} state was registered at: _raw_spin_lock_irqsave+0x3f/0x60 clocksource_mark_unstable+0x1b/0x90 mark_tsc_unstable+0x41/0x50 check_tsc_sync_source+0x14f/0x180 sysvec_call_function_single+0x69/0x90 Possible unsafe locking scenario: lock(watchdog_lock); lock(watchdog_lock); stack backtrace: _raw_spin_lock+0x30/0x40 clocksource_watchdog+0x23/0x5a0 run_timer_softirq+0x2a/0x50 sysvec_apic_timer_interrupt+0x6e/0x90 The reason is the recent conversion of the TSC synchronization function during CPU hotplug on the control CPU to a SMP function call. In case that the synchronization with the upcoming CPU fails, the TSC has to be marked unstable via clocksource_mark_unstable(). clocksource_mark_unstable() acquires 'watchdog_lock', but that lock is taken with interrupts enabled in the watchdog timer callback to minimize interrupt disabled time. That's obviously a possible deadlock scenario, Before that change the synchronization function was invoked in thread context so this could not happen. As it is not crucical whether the unstable marking happens slightly delayed, defer the call to a worker thread which avoids the lock context problem. Fixes: 9d349d47f0e3 ("x86/smpboot: Make TSC synchronization function call based") Reported-by: Tetsuo Handa Signed-off-by: Thomas Gleixner Tested-by: Tetsuo Handa Cc: stable@vger.kernel.org --- arch/x86/kernel/tsc_sync.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) --- a/arch/x86/kernel/tsc_sync.c +++ b/arch/x86/kernel/tsc_sync.c @@ -15,6 +15,7 @@ * ( The serial nature of the boot logic and the CPU hotplug lock * protects against more than 2 CPUs entering this code. ) */ +#include #include #include #include @@ -342,6 +343,13 @@ static inline unsigned int loop_timeout( return (cpumask_weight(topology_core_cpumask(cpu)) > 1) ? 2 : 20; } +static void tsc_sync_mark_tsc_unstable(struct work_struct *work) +{ + mark_tsc_unstable("check_tsc_sync_source failed"); +} + +static DECLARE_WORK(tsc_sync_work, tsc_sync_mark_tsc_unstable); + /* * The freshly booted CPU initiates this via an async SMP function call. */ @@ -395,7 +403,7 @@ static void check_tsc_sync_source(void * "turning off TSC clock.\n", max_warp); if (random_warps) pr_warn("TSC warped randomly between CPUs\n"); - mark_tsc_unstable("check_tsc_sync_source failed"); + schedule_work(&tsc_sync_work); } /*