From patchwork Wed Dec 20 21:15:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Doug Anderson X-Patchwork-Id: 181760 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:2483:b0:fb:cd0c:d3e with SMTP id q3csp15861dyi; Wed, 20 Dec 2023 13:17:39 -0800 (PST) X-Google-Smtp-Source: AGHT+IFxe9N3mG4qNRztHmEB03UNS5bLfkPV8xxhcEforKeeETu5IR/jZVNO32l/5N2HNFvkhIjs X-Received: by 2002:a05:6214:f21:b0:67a:359:a82c with SMTP id iw1-20020a0562140f2100b0067a0359a82cmr5876957qvb.22.1703107058836; Wed, 20 Dec 2023 13:17:38 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1703107058; cv=none; d=google.com; s=arc-20160816; b=kgjMScKRrjPkDRSy6NvbtUT4W3vr1tjv7nJDpfhtxqZ/eSGzwJo6Xd/lQ6SeKjFBGq UoygdXPoBHmFXKADdVtJx64Whe2x/F84g1tqmLiU34YKoqyQdnExBqLotI8SRcofE2w6 9iv4i6Wk9a24ahIVKNe757mUIbVWk4R86rsDC7OVsW3poO80UKGVwOjzUM+s52iW1i9m eul1ytG0NqwtFpbJ/ZWgVKBIPgt/Jkt41UkuKQ0fQLYm09mQ11vAtJWXCToxxlt6kiJo EsxlmbRYsyqvvVsC9QR3JUIwrAtbpQU3i/lyHbIN6gPJMwPzQWJEYB2S1vYO4TAPNeyD LKRw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=t1CDklyqCRANoyYKv7Digk6bbgZ534wMCluTfdczIuw=; fh=xd3xzeCwEWmmWDqIPUTcLHu9oRFfvngZDI2DlEXKYEU=; b=g7kA5+SxUOR6wlQ1t+92iO7O+8u0IgAEyIsSCIfVDviNe/3Yeb6Xys5S4T5V/maaya jLWSvFHZCA6fZhves17da3cbvECRVpGoqCHcyULSD3SJVr340nkjZDwLfcrzGb7+xMcg /4zLiM4/ZhHQUYdOFAvcDWme1a908jsgdDKtCBnSCOcdzz9cjP42nutMuMWyl/+Ns7i7 ZWzK2s8sI+fIi1Nzzzf6ZEZ+gqCaHib6m50yy1Z5PQj5sXDP+w7+YKWYd5+P0HFM3qJT WI4516Ps/yV3/4LCViZ53uJo/HPnQDx7oV9E/2SM8VWHKse4CJ1JlTKtYy6axOuIjoBu Xdgw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@chromium.org header.s=google header.b="MgJD/Z3U"; spf=pass (google.com: domain of linux-kernel+bounces-7436-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-7436-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=chromium.org Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id e4-20020a0cf344000000b0067f28c68f6fsi570799qvm.513.2023.12.20.13.17.38 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Dec 2023 13:17:38 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-7436-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@chromium.org header.s=google header.b="MgJD/Z3U"; spf=pass (google.com: domain of linux-kernel+bounces-7436-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-7436-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=chromium.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 9D8F01C22C11 for ; Wed, 20 Dec 2023 21:17:38 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 0D5554A995; Wed, 20 Dec 2023 21:17:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="MgJD/Z3U" X-Original-To: linux-kernel@vger.kernel.org Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 18E7C495CE for ; Wed, 20 Dec 2023 21:17:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-1d3dee5f534so9098485ad.1 for ; Wed, 20 Dec 2023 13:17:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1703107026; x=1703711826; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=t1CDklyqCRANoyYKv7Digk6bbgZ534wMCluTfdczIuw=; b=MgJD/Z3UnTeItdFfsGqEyxxSFlGTUtQ6xL2moU4MO/TcdgyUQRXTU+jRyvUDLTaMkP sOcjH3DXZ+mc3XTVAK567F5grC4UZufmQNXphmdjfI3OqkoYLys0AnYUC+HvJpQkEbdd 0wO1G+a1lsQIyhsQnDZCVCXQ7XA3zARvR71Fc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703107026; x=1703711826; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=t1CDklyqCRANoyYKv7Digk6bbgZ534wMCluTfdczIuw=; b=UZ4lZxiR4d4OieR1I+jLOukEPSOaaWiYYXuWhvnUG6rayLBweyWbAXwS+oYpdlZv4p iXy7Ux0ndKNJj/zvN5kIzdXWP2f+xLnG4SW106CxETyJFKNlv3pUMwmslZavpH9IhnIT 540r+G9yNZKwvWm5fpEsCNAKWzrw1o8HeDwhG1CB+4abwDGjqqD7hHBzkGzQIlvpfO91 lkEikQxGPKupTIovF2Iow5sCIW3M9OQdmRsKr6tLsp7xT5Qlp9dYptpX5tkE2Ujqs2HE 2V+2yJCpnFn58gFLdEQIPn/DtDgPbGYvpmseNIhdV73b2Yfpctg4tLSJv89I1X4RQXGh w4sA== X-Gm-Message-State: AOJu0YypVMTZYn7HtEWMfECMoWqOwwedWsDNl+PygCwPGlWHxWK/FwiC 47y6V7WzmuqNnVhR76Aj1SiFeg== X-Received: by 2002:a17:902:e543:b0:1d0:bf4c:166c with SMTP id n3-20020a170902e54300b001d0bf4c166cmr4634775plf.1.1703107026411; Wed, 20 Dec 2023 13:17:06 -0800 (PST) Received: from tictac2.mtv.corp.google.com ([2620:15c:9d:2:5a22:d46c:eec1:e5d4]) by smtp.gmail.com with ESMTPSA id u10-20020a170902b28a00b001d3dfebc05esm175023plr.21.2023.12.20.13.17.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Dec 2023 13:17:05 -0800 (PST) From: Douglas Anderson To: Andrew Morton Cc: Petr Mladek , Li Zhe , Pingfan Liu , John Ogness , Lecopzer Chen , Douglas Anderson , linux-kernel@vger.kernel.org Subject: [PATCH 1/4] watchdog/hardlockup: Adopt softlockup logic avoiding double-dumps Date: Wed, 20 Dec 2023 13:15:34 -0800 Message-ID: <20231220131534.1.I4f35a69fbb124b5f0c71f75c631e11fabbe188ff@changeid> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog In-Reply-To: <20231220211640.2023645-1-dianders@chromium.org> References: <20231220211640.2023645-1-dianders@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785837187538641729 X-GMAIL-MSGID: 1785837187538641729 The hardlockup detector and softlockup detector both have the ability to dump the stack of all CPUs (`kernel.hardlockup_all_cpu_backtrace` and `kernel.softlockup_all_cpu_backtrace`). Both detectors also have some logic to attempt to avoid interleaving printouts if two CPUs were trying to do dumps of all CPUs at the same time. However: - The hardlockup detector's logic still allowed interleaving some information. Specifically another CPU could print modules and dump the stack of the locked CPU at the same time we were dumping all CPUs. - In the case where `kernel.hardlockup_panic` was set in addition to `kernel.hardlockup_all_cpu_backtrace`, when two CPUs both detected hardlockups at the same time the second CPU could call panic() while the first was still dumping stacks. This was especially bad if the locked up CPU wasn't responding to the request for a backtrace since the function nmi_trigger_cpumask_backtrace() can wait up to 10 seconds. Let's resolve this by adopting the softlockup logic in the hardlockup handler. NOTES: - As part of this, one might think that we should make a helper function that both the hard and softlockup detectors call. This turns out not to be super trivial since it would have to be parameterized quite a bit since there are separate global variables controlling each lockup detector and they print log messages that are just different enough that it would be a pain. We probably don't want to change the messages that are printed without good reason to avoid throwing log parsers for a loop. - One might also think that it would be a good idea to have the hardlockup and softlockup detector use the same global variable to prevent interleaving. This would make sure that softlockups and hardlockups can't interleave each other. That _almost_ works but has a dangerous flaw if `kernel.hardlockup_panic` is not the same as `kernel.softlockup_panic` because we might skip a call to panic() if one type of lockup was detected at the same time as another. Signed-off-by: Douglas Anderson --- kernel/watchdog.c | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/kernel/watchdog.c b/kernel/watchdog.c index bf30a6fac665..b4fd2f12137f 100644 --- a/kernel/watchdog.c +++ b/kernel/watchdog.c @@ -91,7 +91,7 @@ static DEFINE_PER_CPU(atomic_t, hrtimer_interrupts); static DEFINE_PER_CPU(int, hrtimer_interrupts_saved); static DEFINE_PER_CPU(bool, watchdog_hardlockup_warned); static DEFINE_PER_CPU(bool, watchdog_hardlockup_touched); -static unsigned long watchdog_hardlockup_all_cpu_dumped; +static unsigned long hard_lockup_nmi_warn; notrace void arch_touch_nmi_watchdog(void) { @@ -156,6 +156,15 @@ void watchdog_hardlockup_check(unsigned int cpu, struct pt_regs *regs) if (per_cpu(watchdog_hardlockup_warned, cpu)) return; + /* + * Prevent multiple hard-lockup reports if one cpu is already + * engaged in dumping all cpu back traces. + */ + if (sysctl_hardlockup_all_cpu_backtrace) { + if (test_and_set_bit_lock(0, &hard_lockup_nmi_warn)) + return; + } + pr_emerg("Watchdog detected hard LOCKUP on cpu %d\n", cpu); print_modules(); print_irqtrace_events(current); @@ -168,13 +177,10 @@ void watchdog_hardlockup_check(unsigned int cpu, struct pt_regs *regs) trigger_single_cpu_backtrace(cpu); } - /* - * Perform multi-CPU dump only once to avoid multiple - * hardlockups generating interleaving traces - */ - if (sysctl_hardlockup_all_cpu_backtrace && - !test_and_set_bit(0, &watchdog_hardlockup_all_cpu_dumped)) + if (sysctl_hardlockup_all_cpu_backtrace) { trigger_allbutcpu_cpu_backtrace(cpu); + clear_bit_unlock(0, &hard_lockup_nmi_warn); + } if (hardlockup_panic) nmi_panic(regs, "Hard LOCKUP"); From patchwork Wed Dec 20 21:15:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Doug Anderson X-Patchwork-Id: 181761 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:2483:b0:fb:cd0c:d3e with SMTP id q3csp15952dyi; Wed, 20 Dec 2023 13:17:51 -0800 (PST) X-Google-Smtp-Source: AGHT+IEjJlH91L6xjLRIiMz2TzWaW5rzw5ItGZ/bvwnfpjQzIIFDK444ho0jtBVi5GiFQoVveswv X-Received: by 2002:a05:6359:4c24:b0:170:17eb:1e5 with SMTP id kj36-20020a0563594c2400b0017017eb01e5mr271926rwc.40.1703107070871; Wed, 20 Dec 2023 13:17:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1703107070; cv=none; d=google.com; s=arc-20160816; b=qs07fFSISoFl8vfCwjHWVwFG9+pKLgC6ZiK5C4woXbf+g4JH41jLgFSdk8lxIwRyjo 6fcO9KuM8gzsbGYZZGBySC/AYvb5GjHCWlfVinKMlDpkgNLcbxKGUgeQq+5B2/nvYoxZ SwI+HlXhbu5uhI+R9M74SARVOo1Z5+QlthtLwUUg5Iv9mV9SLBZP8s1vevstSOTJaASl 1jIcwPpF2r6BG2ESWJLdN4d13V0AkG0JrKKqngGvbiw/iMBO9VwQQA52ped9UlsmTK06 Wsqps7BJdK9hF8rwhw5ldbPVIp17xmEf0uTDsMIg/Ct1NRqh3t+Ng6beX3hYHFUnL/Sl Tclg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=Pmb1hFIok+9eMKNNtq9Gk+/VDYxh0leTwTc9coy3jGk=; fh=xd3xzeCwEWmmWDqIPUTcLHu9oRFfvngZDI2DlEXKYEU=; b=fSVwonin9CwVyLTJcgTI7W8ZYyZaqK49Bv14JAozebaPkBAH6v/5ep1Ju4fqmu1njo jSyxix1XkdYpBblitEZu/KDqvqS1tnQvBjsUaN/VFqGbak4RQitf7l4pGH5c8Kttavg/ bH+I00Oxpcfrgk7RQnT0ADNMhD8gtCfuI9Ix6Xf3pj1Tk2hOrXA0wWu0g1waY9ZJeY/1 8uSHwAlcNmxUe76cNvurQQt+3EzBaXCFE2SA+7s1suF5W486lMzEinGWvWq0f0/4wFI4 9frn/7HIQHQpqZyBY6KdZXez7QpRgMIrueQ5SStj9jkw36gyzb6C9VasJsbImOYkHgPy TAlw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@chromium.org header.s=google header.b=OyV7ZQac; spf=pass (google.com: domain of linux-kernel+bounces-7437-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-7437-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=chromium.org Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id bn3-20020a056a02030300b005c661efffd0si340521pgb.754.2023.12.20.13.17.50 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Dec 2023 13:17:50 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-7437-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@chromium.org header.s=google header.b=OyV7ZQac; spf=pass (google.com: domain of linux-kernel+bounces-7437-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-7437-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=chromium.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 9D760285D87 for ; Wed, 20 Dec 2023 21:17:50 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 239814A9B8; Wed, 20 Dec 2023 21:17:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="OyV7ZQac" X-Original-To: linux-kernel@vger.kernel.org Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4A12C498AE for ; Wed, 20 Dec 2023 21:17:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-1d3fc184b6dso1202545ad.2 for ; Wed, 20 Dec 2023 13:17:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1703107028; x=1703711828; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Pmb1hFIok+9eMKNNtq9Gk+/VDYxh0leTwTc9coy3jGk=; b=OyV7ZQacnPHW04yoE+2yXbC1IJY3TUmM2UOiubF3/YOK6NtOyys99/puF1N5+Lo6L8 FVgDqJSlH78X5mJS7H4taC0IFH35E68V3+fz6/C7SggokjrkGG7RCvyhsLLTHy4A6jYW rI5/fjae+yg7LRiPRSNzMWLyeLUNuvjgaozZQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703107028; x=1703711828; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Pmb1hFIok+9eMKNNtq9Gk+/VDYxh0leTwTc9coy3jGk=; b=p9b4DmEsshz35mLue9a3FE2dUPzMBKlLd6uAnoQ1WQOkVzMSgbtE2wgL2LOF9ajEdo xoSz8KDUdSBsxrBB0FDP7JV392cKs4Ii4mE+mNEVrbF4XUDcFqrHik6gjTd69iheYQIx VadGWkWAPmHW2OzGeeVfr4o4hYqFcQoDs+xa7n8L3z1s8DZlhvwCfhlqRy/F72OlV5al yHXgYfkr17WuIE8E0XSYt3XNVaEEmsNoXRn7ksJsJ7B4szF64ssxDmDxgirZ2WfWx+8L 4PNzXV5y5mgae+5j9h56m9PEQbi46JXp+wLeOyES5mj5zaeVDvjWz7BEWjOVLjULVTD5 wd5A== X-Gm-Message-State: AOJu0Ywr3TTgoWaMjLu0QZa9GXFCnvyyQ/Yvp9qbLSYV8cbOVMu22w8i qKiqGEOzUMFHQojO5DJuAUW6MQ== X-Received: by 2002:a17:902:b28b:b0:1d3:f43a:a2d5 with SMTP id u11-20020a170902b28b00b001d3f43aa2d5mr1056238plr.120.1703107028565; Wed, 20 Dec 2023 13:17:08 -0800 (PST) Received: from tictac2.mtv.corp.google.com ([2620:15c:9d:2:5a22:d46c:eec1:e5d4]) by smtp.gmail.com with ESMTPSA id u10-20020a170902b28a00b001d3dfebc05esm175023plr.21.2023.12.20.13.17.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Dec 2023 13:17:07 -0800 (PST) From: Douglas Anderson To: Andrew Morton Cc: Petr Mladek , Li Zhe , Pingfan Liu , John Ogness , Lecopzer Chen , Douglas Anderson , linux-kernel@vger.kernel.org Subject: [PATCH 2/4] watchdog/softlockup: Use printk_cpu_sync_get_irqsave() to serialize reporting Date: Wed, 20 Dec 2023 13:15:35 -0800 Message-ID: <20231220131534.2.Ia5906525d440d8e8383cde31b7c61c2aadc8f907@changeid> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog In-Reply-To: <20231220211640.2023645-1-dianders@chromium.org> References: <20231220211640.2023645-1-dianders@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785837199523086806 X-GMAIL-MSGID: 1785837199523086806 Instead of introducing a spinlock, use printk_cpu_sync_get_irqsave() and printk_cpu_sync_put_irqrestore() to serialize softlockup reporting. Alone this doesn't have any real advantage over the spinlock, but this will allow us to use the same function in a future change to also serialize hardlockup crawls. NOTE: for the most part this serialization is important because we often end up in the show_regs() path and that has no built-in serialization if there are multiple callers at once. However, even in the case where we end up in the dump_stack() path this still has some advantages because the stack will be guaranteed to be together in the logs with the lockup message with no interleaving. NOTE: the fact that printk_cpu_sync_get_irqsave() is allowed to be called multiple times on the same CPU is important here. Specifically we hold the "lock" while calling dump_stack() which also gets the same "lock". This is explicitly documented to be OK and means we don't need to introduce a variant of dump_stack() that doesn't grab the lock. Signed-off-by: Douglas Anderson Reviewed-by: Li Zhe Reviewed-by: John Ogness --- kernel/watchdog.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/kernel/watchdog.c b/kernel/watchdog.c index b4fd2f12137f..526041a1100a 100644 --- a/kernel/watchdog.c +++ b/kernel/watchdog.c @@ -454,7 +454,7 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer) struct pt_regs *regs = get_irq_regs(); int duration; int softlockup_all_cpu_backtrace = sysctl_softlockup_all_cpu_backtrace; - static DEFINE_SPINLOCK(watchdog_output_lock); + unsigned long flags; if (!watchdog_enabled) return HRTIMER_NORESTART; @@ -521,7 +521,7 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer) /* Start period for the next softlockup warning. */ update_report_ts(); - spin_lock(&watchdog_output_lock); + printk_cpu_sync_get_irqsave(flags); pr_emerg("BUG: soft lockup - CPU#%d stuck for %us! [%s:%d]\n", smp_processor_id(), duration, current->comm, task_pid_nr(current)); @@ -531,7 +531,7 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer) show_regs(regs); else dump_stack(); - spin_unlock(&watchdog_output_lock); + printk_cpu_sync_put_irqrestore(flags); if (softlockup_all_cpu_backtrace) { trigger_allbutcpu_cpu_backtrace(smp_processor_id()); From patchwork Wed Dec 20 21:15:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Doug Anderson X-Patchwork-Id: 181763 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:2483:b0:fb:cd0c:d3e with SMTP id q3csp16460dyi; Wed, 20 Dec 2023 13:19:00 -0800 (PST) X-Google-Smtp-Source: AGHT+IG3Dd698bUvgpv/gmYSVVSVrY3Yuz/FHluG4v4ZEN4U19hKwlAyFTZOenuZCYoAH5klsDBW X-Received: by 2002:a05:6e02:1806:b0:35f:9f66:574a with SMTP id a6-20020a056e02180600b0035f9f66574amr10488971ilv.57.1703107140755; Wed, 20 Dec 2023 13:19:00 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1703107140; cv=none; d=google.com; s=arc-20160816; b=u/wjhwsUO/lceNftZaz5YcSb50mrj6rtSqtrsQ2Q/kVgWCSpgbmgVlPgMBj2Pb7WkL SXEIWHLZUwU+kVbhLiUmkHHTRRGT/b4Sho1PbGM+6pwBqLJU9oGw8mM3ydVqcwmRfKuj InqADsqhyztLCUHQsSjZ1cNfuwWutKMHXYd3KBGbk0wdk8iKW3hWpn2+cRb+6vs6DwhK uGZ2bENq6H/rC3frH3Fd8dtnpy/g5EPzCWawd3tOWihTGEol0EiMEorlTgLIU4w3+/j9 eAblMeSSq+aqtaF7HcXrJQCT5kyIlLOwN82UpAkWc0BAi5SGzuIkRXMabhoOHGmtOYiP CkTQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=nqMAcmRjFD5lYv9eaz6Li/2mY5/nRWhBk3z0a51JdoE=; fh=xd3xzeCwEWmmWDqIPUTcLHu9oRFfvngZDI2DlEXKYEU=; b=PEHeGSzbfFrqAZf65o7bkGAREz/Wo1iTAjbC1/bA5OzJS6UmkXwZzvh/ueY1xH8Dvw C2mKCKECmEor3OUYzQ6hTUzKMNh9njpkgaod2bYb8FG4Aapic60SUXUlzBjdlFM0uqTy uR6HVHY1FseO6sAEQS8XRzogD57kry6RlcMLKml6D9ElgRZr3hY8kkA6je8N9nTZIgnl JHcs9uRqb0fZxbdkdHkHnchKShfJUxUHpmlxQD3sukkhy25xennkCzFvArzA8uai6pET 5Bri6RvD072ARNB4WEAZNj9VtBnALbtrsxtozu3TdFysD/hD8+IBX9pCjH8Y56PVwrH9 1hHw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@chromium.org header.s=google header.b="Rm/li2ew"; spf=pass (google.com: domain of linux-kernel+bounces-7438-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-7438-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=chromium.org Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id e19-20020a630f13000000b005bd27be66e1si336929pgl.719.2023.12.20.13.19.00 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Dec 2023 13:19:00 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-7438-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@chromium.org header.s=google header.b="Rm/li2ew"; spf=pass (google.com: domain of linux-kernel+bounces-7438-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-7438-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=chromium.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id F1B75B25133 for ; Wed, 20 Dec 2023 21:18:12 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id B07784B12D; Wed, 20 Dec 2023 21:17:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="Rm/li2ew" X-Original-To: linux-kernel@vger.kernel.org Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CBDA34A9B3 for ; Wed, 20 Dec 2023 21:17:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-1d3aa0321b5so1291455ad.2 for ; Wed, 20 Dec 2023 13:17:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1703107031; x=1703711831; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nqMAcmRjFD5lYv9eaz6Li/2mY5/nRWhBk3z0a51JdoE=; b=Rm/li2ewUQ2jK1rUJEjKu5kv4mMWZiKi+s0VsGzL71uSuDmezamhntuxbQUmlSoHCQ q6kPeX2kxU8+X4UF1/rg9Hi3nUpMPDGzMk1lfG5D8h1BIXxw17Uu166+p5aXFxkBQfmm ktpEbQicnMv5ns1or4ClMoLB/uaGq3beS9jcE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703107031; x=1703711831; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nqMAcmRjFD5lYv9eaz6Li/2mY5/nRWhBk3z0a51JdoE=; b=YUWmzYnxHLVFRV/EAoYhA5329UD9Ju5Pin4CDFnvAL7DgJheQGjwqWTUoYyCe4vCIq EeuKL6JR5K3BjtoHQr1MQbzv1XOtqgIWsQMiOOj9MvsC7b1w+6hl6rpec2mKuyLY7M+6 ev3CpE9CkLLaPkmVKS/4CfkXsHKpIH7sQh2EiEFs2UWmDW62c3UTtN+5RpnLurz80egc rSRbgnKoIi/k8CLWJ4leUpKDUBqGIGJF/OelPFtBA4Cdjuu+LZlmkqRxIwQOBm0GzCQ0 fZZ+zuA/T+vvnJ8o9Tj9sNGId8u2vQFBQGsCW46qTh/5WQNbq7Y8wdswQuEWWkVoiWjA Jzsw== X-Gm-Message-State: AOJu0YzXHFZeHHEcsYYIbX9JpylXhCm/l4y20u2kB8e3p1JB/HhzCsOO f3LKU53JDPlnYZnitsGQpTBs0VY2ulpxUweYi34= X-Received: by 2002:a17:903:496:b0:1d4:445:ba7b with SMTP id jj22-20020a170903049600b001d40445ba7bmr24892plb.36.1703107031029; Wed, 20 Dec 2023 13:17:11 -0800 (PST) Received: from tictac2.mtv.corp.google.com ([2620:15c:9d:2:5a22:d46c:eec1:e5d4]) by smtp.gmail.com with ESMTPSA id u10-20020a170902b28a00b001d3dfebc05esm175023plr.21.2023.12.20.13.17.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Dec 2023 13:17:10 -0800 (PST) From: Douglas Anderson To: Andrew Morton Cc: Petr Mladek , Li Zhe , Pingfan Liu , John Ogness , Lecopzer Chen , Douglas Anderson , linux-kernel@vger.kernel.org Subject: [PATCH 3/4] watchdog/hardlockup: Use printk_cpu_sync_get_irqsave() to serialize reporting Date: Wed, 20 Dec 2023 13:15:36 -0800 Message-ID: <20231220131534.3.I6ff691b3b40f0379bc860f80c6e729a0485b5247@changeid> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog In-Reply-To: <20231220211640.2023645-1-dianders@chromium.org> References: <20231220211640.2023645-1-dianders@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785837273078118276 X-GMAIL-MSGID: 1785837273078118276 If two CPUs end up reporting a hardlockup at the same time then their logs could get interleaved which is hard to read. The interleaving problem was especially bad with the "perf" hardlockup detector where the locked up CPU is always the same as the running CPU and we end up in show_regs(). show_regs() has no inherent serialization so we could mix together two crawls if two hardlockups happened at the same time (and if we didn't have `sysctl_hardlockup_all_cpu_backtrace` set). With this change we'll fully serialize hardlockups when using the "perf" hardlockup detector. The interleaving problem was less bad with the "buddy" hardlockup detector. With "buddy" we always end up calling `trigger_single_cpu_backtrace(cpu)` on some CPU other than the running one. trigger_single_cpu_backtrace() always at least serializes the individual stack crawls because it eventually uses printk_cpu_sync_get_irqsave(). Unfortunately the fact that trigger_single_cpu_backtrace() eventually calls printk_cpu_sync_get_irqsave() (on a different CPU) means that we have to drop the "lock" before calling it and we can't fully serialize all printouts associated with a given hardlockup. However, we still do get the advantage of serializing the output of print_modules() and print_irqtrace_events(). Aside from serializing hardlockups from each other, this change also has the advantage of serializing hardlockups and softlockups from each other if they happen to happen at the same time since they are both using the same "lock". Even though nobody is expected to hang while holding the lock associated with printk_cpu_sync_get_irqsave(), out of an abundance of caution, we don't call printk_cpu_sync_get_irqsave() until after we print out about the hardlockup. This makes extra sure that, even if printk_cpu_sync_get_irqsave() somehow never runs we at least print that we saw the hardlockup. This is different than the choice made for softlockup because hardlockup is really our last resort. Signed-off-by: Douglas Anderson Reviewed-by: John Ogness --- kernel/watchdog.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/kernel/watchdog.c b/kernel/watchdog.c index 526041a1100a..11f9577accca 100644 --- a/kernel/watchdog.c +++ b/kernel/watchdog.c @@ -151,6 +151,7 @@ void watchdog_hardlockup_check(unsigned int cpu, struct pt_regs *regs) */ if (is_hardlockup(cpu)) { unsigned int this_cpu = smp_processor_id(); + unsigned long flags; /* Only print hardlockups once. */ if (per_cpu(watchdog_hardlockup_warned, cpu)) @@ -165,7 +166,17 @@ void watchdog_hardlockup_check(unsigned int cpu, struct pt_regs *regs) return; } + /* + * NOTE: we call printk_cpu_sync_get_irqsave() after printing + * the lockup message. While it would be nice to serialize + * that printout, we really want to make sure that if some + * other CPU somehow locked up while holding the lock associated + * with printk_cpu_sync_get_irqsave() that we can still at least + * get the message about the lockup out. + */ pr_emerg("Watchdog detected hard LOCKUP on cpu %d\n", cpu); + printk_cpu_sync_get_irqsave(flags); + print_modules(); print_irqtrace_events(current); if (cpu == this_cpu) { @@ -173,7 +184,9 @@ void watchdog_hardlockup_check(unsigned int cpu, struct pt_regs *regs) show_regs(regs); else dump_stack(); + printk_cpu_sync_put_irqrestore(flags); } else { + printk_cpu_sync_put_irqrestore(flags); trigger_single_cpu_backtrace(cpu); } From patchwork Wed Dec 20 21:15:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Doug Anderson X-Patchwork-Id: 181762 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:2483:b0:fb:cd0c:d3e with SMTP id q3csp16147dyi; Wed, 20 Dec 2023 13:18:20 -0800 (PST) X-Google-Smtp-Source: AGHT+IHGabPkAJpKy2wBl4DaMOTEF8R0qAtzdslaXVgAJOc1D3186GeASs13Cxd7r3Y+1Q5K0V1q X-Received: by 2002:a17:902:e74e:b0:1cc:5e1b:98b5 with SMTP id p14-20020a170902e74e00b001cc5e1b98b5mr23517728plf.66.1703107099814; Wed, 20 Dec 2023 13:18:19 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1703107099; cv=none; d=google.com; s=arc-20160816; b=tHja2Pke+4l27Uk241IdHnfRAVE3AuHXTEhuNVlcztTGMxXHZiwoc71MkNshO9OY+g HpJWLbkyJnZ4KEte4aETNmZT0l8pADfFcDhrAr+rs7PxuqmpUUdT1jCu61b/6/eY/IXy 9ihhpFZ1ImDqhGkRM5423coULYVDbNXzAG6ymHvoHMyGKCbx5awU/97+ekBrWbgmkeRE Cx7oMxfp9jjVBlCr0Alt5N82LcMH/i9x7Z0U1RN5j5XaN156ha9pEnRD3AXGGMpt9xj8 cAHXQTAPnTYEDM6n4l3MlZhnasyYEUZb8Js5RB01neVkvQ2qD/UMVvoer44oGw/cHMKz NWyg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=sGegIddggKFG/zutIaei4eHzxrpz6/dja4W8/BctOAU=; fh=xd3xzeCwEWmmWDqIPUTcLHu9oRFfvngZDI2DlEXKYEU=; b=G1suL/soy2SD8WAF1qoz+Mg1V94JOwsTl3eyd0/NGV6pbPdq5Nf0TT59B7Yn6kmQfF nQsHx59MMiyQYqZKZ9paroroEzHdKjRDwQdZwMOq8fUEYJ31lSOytJcwvjfpim8g4+eN wFRWag/4dHtl59gJANxPiD6lyvB4jlN2yi5wpgaXm31hVlzPeGg64g4Lu7GDh5Cr544u Xq3+gVlN2piGfecN1fPSJQ6IpeLvgJO7+GPEytNEvf0YIoF/p/us6TxwRHxJplTqsVYP X/N5dMO+wI0Gt9lH9HTj4wgKZj2GW5cyiQo6fsg03bSJB5J3iSeRKQ9b9M1lMGwCT/ft b9dA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@chromium.org header.s=google header.b=WMsXp5la; spf=pass (google.com: domain of linux-kernel+bounces-7439-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-7439-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=chromium.org Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id e9-20020a170902b78900b001d3e4207386si257933pls.317.2023.12.20.13.18.19 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Dec 2023 13:18:19 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-7439-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@chromium.org header.s=google header.b=WMsXp5la; spf=pass (google.com: domain of linux-kernel+bounces-7439-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-7439-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=chromium.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 35446286B37 for ; Wed, 20 Dec 2023 21:18:19 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id B344A4B153; Wed, 20 Dec 2023 21:17:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="WMsXp5la" X-Original-To: linux-kernel@vger.kernel.org Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DA6634AF63 for ; Wed, 20 Dec 2023 21:17:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-1d3ddccfaffso1323145ad.1 for ; Wed, 20 Dec 2023 13:17:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1703107033; x=1703711833; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sGegIddggKFG/zutIaei4eHzxrpz6/dja4W8/BctOAU=; b=WMsXp5lad1ptNXHEjUl8VA87VrrCWANP2+oOSskHvW08BCjLlInH3WWbGorhzR5IKG Q3t95JdFtKdYbMRYuP6lCmOSzzjDOtGuQwzIr0F2QKrF6hsQv/uRi3IUn7p5uYwqrV4x u2e1M/EpEU/NXfkzSPDrGyiLFplp2ZLpYLOJA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703107033; x=1703711833; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sGegIddggKFG/zutIaei4eHzxrpz6/dja4W8/BctOAU=; b=NHZN/j2xGHI8SbJe/k5aOVmelUqpvEv7hjTXXjriSUThpciRRGFBSlrdjuDcwtVIUV 4aBmjcux0OBrqlhe9TofySHhSSM7uKwBY75MKScWcM5uMMJDf1MDfZaky35sm6GaP0Sh hUO1vOcziL/VoekAjJJo9eEGD3Arl2y4s/0o4KA4Jcr4oXEUM8tPaONIHP/lKflrw/Sv MpxR5n+mC5V6yZl8ohgNglLYvrmKt6FKAJB+gZNWyUT5D7VWngmuRkGTLgj19IUwFcpP diBRLtEty5VI7BNlvbQJbnIr6I3IYBsPBq+epuFtRcFdZ6qGZr2UGbTKPl9R42YKH4tZ 0Q+w== X-Gm-Message-State: AOJu0Yy/ax01Ab5qBh6KgRqkg6F7Z3XK0sGkvrvXGn4N0q/Vgk0fK5AV SjFmpyKKaN1i/gGW2jsYml9lyw== X-Received: by 2002:a17:902:e847:b0:1d0:acfc:a653 with SMTP id t7-20020a170902e84700b001d0acfca653mr23942838plg.84.1703107033190; Wed, 20 Dec 2023 13:17:13 -0800 (PST) Received: from tictac2.mtv.corp.google.com ([2620:15c:9d:2:5a22:d46c:eec1:e5d4]) by smtp.gmail.com with ESMTPSA id u10-20020a170902b28a00b001d3dfebc05esm175023plr.21.2023.12.20.13.17.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Dec 2023 13:17:12 -0800 (PST) From: Douglas Anderson To: Andrew Morton Cc: Petr Mladek , Li Zhe , Pingfan Liu , John Ogness , Lecopzer Chen , Douglas Anderson , linux-kernel@vger.kernel.org Subject: [PATCH 4/4] watchdog: If panicking and we dumped everything, don't re-enable dumping Date: Wed, 20 Dec 2023 13:15:37 -0800 Message-ID: <20231220131534.4.Id3a9c7ec2d7d83e4080da6f8662ba2226b40543f@changeid> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog In-Reply-To: <20231220211640.2023645-1-dianders@chromium.org> References: <20231220211640.2023645-1-dianders@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785837229988657073 X-GMAIL-MSGID: 1785837229988657073 If, as part of handling a hardlockup or softlockup, we've already dumped all CPUs and we're just about to panic, don't reenable dumping and give some other CPU a chance to hop in there and add some confusing logs right as the panic is happening. Signed-off-by: Douglas Anderson --- kernel/watchdog.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/kernel/watchdog.c b/kernel/watchdog.c index 11f9577accca..81a8862295d6 100644 --- a/kernel/watchdog.c +++ b/kernel/watchdog.c @@ -192,7 +192,8 @@ void watchdog_hardlockup_check(unsigned int cpu, struct pt_regs *regs) if (sysctl_hardlockup_all_cpu_backtrace) { trigger_allbutcpu_cpu_backtrace(cpu); - clear_bit_unlock(0, &hard_lockup_nmi_warn); + if (!hardlockup_panic) + clear_bit_unlock(0, &hard_lockup_nmi_warn); } if (hardlockup_panic) @@ -548,7 +549,8 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer) if (softlockup_all_cpu_backtrace) { trigger_allbutcpu_cpu_backtrace(smp_processor_id()); - clear_bit_unlock(0, &soft_lockup_nmi_warn); + if (!softlockup_panic) + clear_bit_unlock(0, &soft_lockup_nmi_warn); } add_taint(TAINT_SOFTLOCKUP, LOCKDEP_STILL_OK);