From patchwork Wed Feb 7 13:40:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Ogness X-Patchwork-Id: 20056 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp2234410dyb; Wed, 7 Feb 2024 05:41:59 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWAqx6hKo8L95xlTrhT9B6LYYtcB2HRQsOIIJjD2nc0NMTihbiVJpWzvYt9pfxStrvsgHN6/KiTabbGqlfTKwANJGyqZA== X-Google-Smtp-Source: AGHT+IGx4+WRgdbsaObvcxyrzscDume+qbdrFC2yg/hxgVZeN3bjM0ribEBDZuXb3Aas9B762yFH X-Received: by 2002:a17:906:714d:b0:a38:45ec:d128 with SMTP id z13-20020a170906714d00b00a3845ecd128mr3001323ejj.69.1707313319445; Wed, 07 Feb 2024 05:41:59 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707313319; cv=pass; d=google.com; s=arc-20160816; b=zgqkcdoTlYSx0zF8vAXzWrBZS3ELFZSNIs4Z2gP9VYnr8/OV28kjAjVSa33TLuGXCr ssIPZY/++IsAZwRfjXhbxmbPGcVVNQDGdMTQss+2P/k0SNkAQQSTpomQpfme31aqHCxM cTqg3sFaaTtjEQaH3MCuxVwS+z+qK+5qQ06HGT7Pl8uqYasmNnJthfhcQaUxbTgIl7eG glsUCD/AUqCeKjqEFaZRl0wePvEh75iJ2RtVhXzbR7NYMrQUxMb1wF+SKtz0Rxxy8jrw bd8d2wExDEpegbmfMXHHVimHqmtrkH+bTD4F7HpZ1wJsXeVk9T67GSJ/e0fcLX1rX55V yt0w== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:message-id:date:subject:cc:to :dkim-signature:dkim-signature:from; bh=fGjdqmICuBOqdgawtrYQWGWCK/rf00yxhY3OYk6C0Gs=; fh=zr83465dfKrVAjGn/AL4MKLnp/FHK16P5xOstJtcKwc=; b=EbapsMZ2Z5taPJ6TVl3IQFB0nfGO+1ivMPRTMfnX7nxQlLH2gpfKuu9jQIcBSE0oMW 8xXE/2ytKOsu5WwMYDa/8cVl72n252JWeRNWcMqBsBtczvYxwuDY2AMqYH0tl8r7trg1 OtBUlzSx0092CVk4RX9F9mP/1FVsxaAkfJKnzxuyNu5POpE6cwhhW8E6y8ttTST+BgLY l3uRptrcyRm+bnF7RSmvHp9nBo+KDF7lBAWHKYxilhPE2SWcDn2W2Sk3V3E1UJXBFm4W k5sfMZh2LVGZXQP27ZitUWz/I01qNdCuviBQE4eP+Fk2UkgKhCEVeXWsZ4FsCsKtW5Fa xx8A==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=MYmNhCzx; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-56560-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-56560-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de X-Forwarded-Encrypted: i=2; AJvYcCUQFg9fJoT3+eme7XHzvbdZRuMOZ4llc8mpqb3GjceveRHmc6YCeicv5IpWvRwAa74y+FSdi7K3wbvfBK3FvR7fLARTyA== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id a5-20020a1709064a4500b00a367bddd8f0si897765ejv.957.2024.02.07.05.41.59 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Feb 2024 05:41:59 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-56560-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=MYmNhCzx; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-56560-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-56560-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id CF2C61F2212E for ; Wed, 7 Feb 2024 13:41:58 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 2AE607CF01; Wed, 7 Feb 2024 13:41:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="MYmNhCzx"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="DRnwLFE2" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E8C665BADD for ; Wed, 7 Feb 2024 13:41:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707313278; cv=none; b=KqP5LPdb0VmjkES7nG9hHCilLMwo+psiUIUxF37Ola+AGjCYNB7CBiS1MCSUMhdtzJ5fy2uqLBNZUsHP8wtXUGhAcRCqtXIMKniSuDbaKp/R/aFn/DvJnOxLF0C9Xi+Mj32AAH1kyxbdBJRsiw4FbWehBuho5agPzHg7YtLnt7A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707313278; c=relaxed/simple; bh=kYORAhihXQCNKA3XucZajuJW7DnSKFUFLgxbmUZsiTM=; h=From:To:Cc:Subject:Date:Message-Id:MIME-Version; b=hIWjL6udCKvpIyvGOWHTf2dujgktre359cHUFxeTgF767dV0hyKz2UYzrpuv7fEtjjepI0X5gVu73SybtuS3xuUxHtMMRrmw/VyGIP+t3U9YQuKZg3PX8/6Hn3I9CRKr+Oi2R6pFEPoW/hRtwgXNVS8eNKJEEwGrXwSn5pJuGOE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=MYmNhCzx; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=DRnwLFE2; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de From: John Ogness DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1707313274; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=fGjdqmICuBOqdgawtrYQWGWCK/rf00yxhY3OYk6C0Gs=; b=MYmNhCzxNfLhZByKKwRcyGqN9iqW0nS6WAOWXzMVugkwWTU8tlBRT9YojEG9OTzwm3Taeh lx2FVsYbUpE4dGqZ/JHal1soI4RGQws5zja4sol8v6mnl0t+KvLwD2oJFMFza1GWWxsOVw lTMqLgdCWf/HndqP9bKrDjPq3GEbJFQPlW0jxz+X6xQVXx+dc9wPTSu+cliE6f5EpmUEcv yRFDWhUOaBSFvYq8do9Y3g1wTTvtm4kcxIBYVPJxT0l1mW4oAScs/BTsb95SZOHRaj/AMW ULgQLVxqI14TKURT31WzHx4t6QJPW4EYWkcfz1UQls7MTtXbXJFYNnguByvegg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1707313274; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=fGjdqmICuBOqdgawtrYQWGWCK/rf00yxhY3OYk6C0Gs=; b=DRnwLFE2M74ctZFqoH+IFbvciwU6ARWuHt7Trh9G6dhE3ATcaPQWec08wOk2/PfnEsEbLv gEOeBdRWjf/ZQtDA== To: Petr Mladek Cc: Sergey Senozhatsky , Steven Rostedt , Thomas Gleixner , linux-kernel@vger.kernel.org, Mukesh Ojha , Andrew Morton , Josh Poimboeuf , "Peter Zijlstra (Intel)" , Ingo Molnar , Kefeng Wang , Arnd Bergmann , Uros Bizjak , "Guilherme G. Piccoli" Subject: [PATCH printk v4 00/14] fix console flushing Date: Wed, 7 Feb 2024 14:46:49 +0106 Message-Id: <20240207134103.1357162-1-john.ogness@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790247771685276262 X-GMAIL-MSGID: 1790247771685276262 Hi, While testing various flushing scenarios, I stumbled on a few issues that cause console flushing to fail. While at LPC2023 in Richmond, I sat down with Petr Mladek and we reviewed the v2 [0] series. This series is the result of that offline discussion. v3 is here [1]. This series addresses the following issues: 1. The prb_next_seq() optimization caused inconsistent return values. Fix prb_next_seq() to the originally intended behavior but keep an optimization. 2. pr_flush() might not wait until the most recently stored printk() message if non-finalized records precede it. Fix pr_flush() to wait for all records to print that are at least reserved at the time of the call. 3. In panic, the panic messages will not print if non-finalized records precede them. Add a special condition so that readers on the panic CPU will drop records that are not in a consistent state. 4. It is possible (and easy to reproduce) a scenario where the console on the panic CPU hands over to a waiter of a stopped CPU. Do not use the handover feature in panic. 5. If messages are being dropped during panic, non-panic CPUs are silenced. But by then it is already too late and most likely the panic messages have been dropped. Change the non-panic CPU silencing logic to _immediately_ silence non-panic CPUs during panic. This also leads to clean panic output when many CPUs are blasting the kernel log. 6. If a panic occurs in a context where printk() calls defer printing (NMI or printk_safe section), the printing of the final panic messages rely on irq_work. If that mechanism is not available, the final panic messages are not seen (even though they are finalized in the ringbuffer). Add one last explicit flush after all printk() calls are finished to ensure all available messages in the kernel log are printed. 7. When dumping the stacktrace from panic(), do not use the printk_cpu_sync because it can deadlock if another CPU holds and is unable to release the printk_cpu_sync. This series also performs some minor cleanups to remove open coded checks about the panic context and improve documentation language regarding data-less records. Because of multiple refactoring done in recent history, it would be helpful to provide the LTS maintainers with the proper backported patches. I am happy to do this. The changes since v3: - Drop patch 11 of v3 ("printk: ringbuffer: Consider committed as finalized in panic"). It adds complexity, may not perform as expected, and it is questionable as to whether it would help provide useful messages on panic. - Changed several comments according to the suggestions by Petr. Note that I did not include all the suggestions because IMO they were too vague in describing the related memory barriers. - Add a patch to avoid using printk_cpu_sync on panic(). This was recently discussed [2]. Feel free to drop the patch 14/14 ("dump_stack: Do not get cpu_sync for panic CPU") if you think it is not appropriate or will significantly delay this series. John Ogness [0] https://lore.kernel.org/lkml/20231106210730.115192-1-john.ogness@linutronix.de [1] https://lore.kernel.org/lkml/20231214214201.499426-1-john.ogness@linutronix.de [2] https://lore.kernel.org/lkml/ZcIGKU8sxti38Kok@alley John Ogness (12): printk: nbcon: Relocate 32bit seq macros printk: Use prb_first_seq() as base for 32bit seq macros printk: ringbuffer: Do not skip non-finalized records with prb_next_seq() printk: ringbuffer: Clarify special lpos values printk: For @suppress_panic_printk check for other CPU in panic printk: Add this_cpu_in_panic() printk: ringbuffer: Cleanup reader terminology printk: Wait for all reserved records with pr_flush() printk: ringbuffer: Skip non-finalized records in panic printk: Avoid non-panic CPUs writing to ringbuffer panic: Flush kernel log buffer at the end dump_stack: Do not get cpu_sync for panic CPU Petr Mladek (1): printk: Disable passing console lock owner completely during panic() Sebastian Andrzej Siewior (1): printk: Adjust mapping for 32bit seq macros include/linux/printk.h | 2 + kernel/panic.c | 8 + kernel/printk/nbcon.c | 41 +--- kernel/printk/printk.c | 101 +++++---- kernel/printk/printk_ringbuffer.c | 335 +++++++++++++++++++++++++----- kernel/printk/printk_ringbuffer.h | 54 ++++- lib/dump_stack.c | 16 +- 7 files changed, 419 insertions(+), 138 deletions(-) base-commit: 6c3a34e38436a2a3f7a1fa764c108ee19b05b893