From patchwork Sat Oct 28 19:24:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: David Woodhouse X-Patchwork-Id: 159287 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:d641:0:b0:403:3b70:6f57 with SMTP id cy1csp1367039vqb; Sat, 28 Oct 2023 12:25:43 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGgTkTwLr1wJxkiJGGZ3SmMbZgtou0CuowSSsMBFT0FfR8uHTlY4PopO1mHvWIVMRtA3eDd X-Received: by 2002:a17:902:c402:b0:1c3:e2eb:f79d with SMTP id k2-20020a170902c40200b001c3e2ebf79dmr11479041plk.8.1698521143195; Sat, 28 Oct 2023 12:25:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698521143; cv=none; d=google.com; s=arc-20160816; b=zhI8hYg+WwmKEPLS8qxw5LPwat5SLJs7GmvnR9lme2uq9nS8M5rp+IryuC1vzre7ge pEQ11PGvdM/x62YuATV5TSspS7NtgjwFElFrjqRJI2ryNNHulLva7xYiAxhg4YENlVge Gf7kH9MLFxFA1gbhS+efTBMQ1qqIFkitZhMs+QFQszsYTygQLrV6buJPuitFaqPsqB/J bc51ttDinLEXDbhUbYU2f+LK9+VPLbtuarOGmfNIfvvFj+k3ERT64Z3LWODNa9+vaneA 7/VsADR8VEA9g+80xUpYL4TZidiV5eHBCYgjGXt4Fvr9RXhWvzt37/sD6w5iRvAA81sf 3fLQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:user-agent:references:in-reply-to :date:cc:to:from:subject:message-id:dkim-signature; bh=luGzykmJCpBS80Q5OEUL47rw7m5niq2MC66f0mnvRGI=; fh=UpxAT3Jx/+h0Tm9JxI68OeoY/rdpNdBnBkZyiyuj664=; b=vDKUu4rO81Avkn7cg4Jo8pLFl4npgMhFLmajTz967RH2usyJVpJILYXEcIe3NpGqsa YfwQQ9xOp4PzySIvqX3MrEYq2Rj+K2DztlsT3SUS/2YnSptR0+o3H9kv1Efp+8Vgxewr 1mj5nHDaZaegL8sdzlVwWF27PhwCbJ57pyeKiUYMRNevOA+BO9k6JwimcebQGNwfccNK OqJjlQoRS2lW8NLTqKlTQ4goYuEjnRt2hePLU1C0G/KiybtOhsUavsIj5VrCCVAgeBLU nmNHWV2nQ1ISCollYJxks3nhSH/w5mJ3sSBHYDXNhXRqLS6xHkVQV4bMNLube3eGSMP0 Eonw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=PjO10Q1G; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from pete.vger.email (pete.vger.email. [23.128.96.36]) by mx.google.com with ESMTPS id s18-20020a17090330d200b001c5fa46f0e6si2863269plc.190.2023.10.28.12.25.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 12:25:43 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=PjO10Q1G; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 8587A8050015; Sat, 28 Oct 2023 12:25:40 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229491AbjJ1TZI (ORCPT + 28 others); Sat, 28 Oct 2023 15:25:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58114 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229446AbjJ1TZG (ORCPT ); Sat, 28 Oct 2023 15:25:06 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B074BB8 for ; Sat, 28 Oct 2023 12:25:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=MIME-Version:Content-Type:References: In-Reply-To:Date:Cc:To:From:Subject:Message-ID:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=luGzykmJCpBS80Q5OEUL47rw7m5niq2MC66f0mnvRGI=; b=PjO10Q1Gkjx/KlqXQmJ3gBVfQa FOXPCfX1Q0wNyaXvX9poBa2jFRebQFBxiD7eBVW7Vpboa5BdPyPTvNAHaQDzdR+rB6w/fIZN1lYAm CnZfmqaECao23/JxNlMm4wk1tse4+3Yq2ikkp70csxFIep06vvoKz0EMxN+PZv6NKK+zuX2Ep9Cme qbOaluf3UxPVjuwQtnqlroD4Aq7o65VRnsbhtkdTDGIMZa5g4xvfx8zSfqAnSEByI64X13hotKMzj VD7F1b+GY8rwINM6V/c5IPlq1+aMuZQ2GazgsyierzPFw1730fXug5RcdVsx+n/0atB5EXnDUmNVW x6UtPAvg==; Received: from [2001:8b0:10b:5:b7dd:c749:f0d9:3970] (helo=u3832b3a9db3152.ant.amazon.com) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1qwovO-00B7vm-4Q; Sat, 28 Oct 2023 19:24:46 +0000 Message-ID: <635fa006e8f3816b4a36b964d6281f0d8efa789b.camel@infradead.org> Subject: [PATCH v2] lockdep: add lockdep_cleanup_dead_cpu() From: David Woodhouse To: Thomas Gleixner , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , linux-kernel Cc: Juergen Gross Date: Sat, 28 Oct 2023 20:24:45 +0100 In-Reply-To: References: User-Agent: Evolution 3.44.4-0ubuntu2 MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by casper.infradead.org. See http://www.infradead.org/rpr.html X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Sat, 28 Oct 2023 12:25:40 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780997627545022762 X-GMAIL-MSGID: 1781028505866216973 From: David Woodhouse Add a function to check that an offline CPU left the tracing infrastructure in a sane state. The acpi_idle_play_dead() function was recently observed¹ calling safe_halt() instead of raw_safe_halt(), which had the side-effect of setting the hardirqs_enabled flag for the offline CPU. On x86 this triggered lockdep warnings when the CPU came back online, but too early for the exception to be handled correctly, leading to a triple-fault. Add lockdep_cleanup_dead_cpu() to check for this kind of failure mode, print the events leading up to it, and correct it so that the CPU can come online again correctly. [ 61.556652] smpboot: CPU 1 is now offline [ 61.556769] CPU 1 left hardirqs enabled! [ 61.556915] irq event stamp: 128149 [ 61.556965] hardirqs last enabled at (128149): [] acpi_idle_play_dead+0x46/0x70 [ 61.557055] hardirqs last disabled at (128148): [] do_idle+0x90/0xe0 [ 61.557117] softirqs last enabled at (128078): [] __do_softirq+0x31c/0x423 [ 61.557199] softirqs last disabled at (128065): [] __irq_exit_rcu+0x91/0x100 ¹ https://lore.kernel.org/lkml/a079bba5a0e47d6534b307553fc3772d26ce911b.camel@infradead.org/ Signed-off-by: David Woodhouse Reviewed-by: Thomas Gleixner --- v2: Fix spelling. 'Offlone' wasn't quite what I meant to type. Add reference to ACPI patch. Fix kerneldoc args for lockdep_cleanup_dead_cpu() (thanks lkp) Closes: https://lore.kernel.org/oe-kbuild-all/202310290041.L5ndwcQ9-lkp@intel.com/ include/linux/irqflags.h | 4 ++++ kernel/cpu.c | 1 + kernel/locking/lockdep.c | 24 ++++++++++++++++++++++++ 3 files changed, 29 insertions(+) diff --git a/include/linux/irqflags.h b/include/linux/irqflags.h index 2b665c32f5fe..547ed55fc7ff 100644 --- a/include/linux/irqflags.h +++ b/include/linux/irqflags.h @@ -24,12 +24,16 @@ extern void lockdep_hardirqs_on_prepare(void); extern void lockdep_hardirqs_on(unsigned long ip); extern void lockdep_hardirqs_off(unsigned long ip); + extern void lockdep_cleanup_dead_cpu(unsigned int cpu, + struct task_struct *idle); #else static inline void lockdep_softirqs_on(unsigned long ip) { } static inline void lockdep_softirqs_off(unsigned long ip) { } static inline void lockdep_hardirqs_on_prepare(void) { } static inline void lockdep_hardirqs_on(unsigned long ip) { } static inline void lockdep_hardirqs_off(unsigned long ip) { } + static inline void lockdep_cleanup_dead_cpu(unsigned int cpu, + struct task_struct *idle) {} #endif #ifdef CONFIG_TRACE_IRQFLAGS diff --git a/kernel/cpu.c b/kernel/cpu.c index 6de7c6bb74ee..225f5bc3708f 100644 --- a/kernel/cpu.c +++ b/kernel/cpu.c @@ -1371,6 +1371,7 @@ static int takedown_cpu(unsigned int cpu) cpuhp_bp_sync_dead(cpu); + lockdep_cleanup_dead_cpu(cpu, idle_thread_get(cpu)); tick_cleanup_dead_cpu(cpu); rcutree_migrate_callbacks(cpu); return 0; diff --git a/kernel/locking/lockdep.c b/kernel/locking/lockdep.c index e85b5ad3e206..62bfda8991b8 100644 --- a/kernel/locking/lockdep.c +++ b/kernel/locking/lockdep.c @@ -4538,6 +4538,30 @@ void lockdep_softirqs_off(unsigned long ip) debug_atomic_inc(redundant_softirqs_off); } +/** + * lockdep_cleanup_dead_cpu - Ensure CPU lockdep state is cleanly stopped + * + * @cpu: index of offlined CPU + * @idle: task pointer for offlined CPU's idle thread + * + * Invoked after the CPU is dead. Ensures that the tracing infrastructure + * is left in a suitable state for the CPU to be subsequently brought + * online again. + */ +void lockdep_cleanup_dead_cpu(unsigned int cpu, struct task_struct *idle) +{ + if (unlikely(!debug_locks)) + return; + + if (unlikely(per_cpu(hardirqs_enabled, cpu))) { + pr_warn("CPU %u left hardirqs enabled!", cpu); + if (idle) + print_irqtrace_events(idle); + /* Clean it up for when the CPU comes online again. */ + per_cpu(hardirqs_enabled, cpu) = 0; + } +} + static int mark_usage(struct task_struct *curr, struct held_lock *hlock, int check) {