From patchwork Mon Nov 6 21:07:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Ogness X-Patchwork-Id: 162194 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp2928148vqu; Mon, 6 Nov 2023 13:08:12 -0800 (PST) X-Google-Smtp-Source: AGHT+IHPAcwzV5g2kWwdBTwA8xixk/Q1GaZpNukrNYymaa996dv0APuY5pQAsKQhHGKvyN1phA+2 X-Received: by 2002:a17:902:e1cd:b0:1b8:b382:f6c3 with SMTP id t13-20020a170902e1cd00b001b8b382f6c3mr21536490pla.13.1699304892589; Mon, 06 Nov 2023 13:08:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699304892; cv=none; d=google.com; s=arc-20160816; b=a9HmrxwOs0ApWT+KzuAY/gMXftnYcne5VhHIEYekGeC2rHDVLLlTwQno3MrjP3outN F3gUvMQl+zV9h2fh7587AJjneMpN9EQSVqVGwpqvi5TjI4aOtX49xCpA77fANxJLB5r+ x7IpfbhtkY7qThueYWrTsV8ws8nETMp3vQiJldrOV+BehRTitCkzByNWUU14CU08puER VlrYh+Ave5Bfi4m1AXby8JsJvH/pyNhFgfp8Xf0igmirDZJ2gmLwTQrpza85/B8m8rxV LakY7gl55Z8VaqujfmPIlMxZ6hUJFlmpU8kZrOjjByJ3bdLdwUeLBGIPfo9WjSweG1QH VouQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:dkim-signature :dkim-signature:from; bh=vxMWmZFtd/CI8sGJ7jbK846Sk0z3M0dHtiMNHUYVwDo=; fh=pKnPKUsir0uEGSsor+4Zc2vgbu+g+ayvUgsdzkuXaoA=; b=evSPc+m5picV3xysc8Eol+E7yzjaTru3I5sX/ma7NEfMT9VlL0Uj6qVH4x8+HgrgUc OmoSFyHuUZlVliO1SFVh5i5VkImzv8aVRuNJw/LnxRBZbTvN6DPHlPTEtiwOB0AE/JPV W9TtS5UF70Wdc5pN2nhzROXF5RcdTXYo1jMA2E/E3yiAHhCM3MTQAhsqPi7MzpksN/v8 XkLuLHKpfL7AmHBhQMU0sgWMFsrlsOjpQLzF9HFS8m8YArD9umA5OiUZmJoadc/LV8TF xZDqh+b2mgewLm6KmNy6Dcsk4pZFMxoouT2zNd0MMpJshV4bvuSnT/nq2i25vRJXJSVa BV4w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=F0nq+Nls; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=iIknmtgf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id l19-20020a170902e2d300b001cc4aca5f5dsi8156892plc.636.2023.11.06.13.08.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Nov 2023 13:08:12 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=F0nq+Nls; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=iIknmtgf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id ACEB2801B308; Mon, 6 Nov 2023 13:08:11 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233197AbjKFVIA (ORCPT + 34 others); Mon, 6 Nov 2023 16:08:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57796 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233099AbjKFVHl (ORCPT ); Mon, 6 Nov 2023 16:07:41 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 50586D76 for ; Mon, 6 Nov 2023 13:07:38 -0800 (PST) From: John Ogness DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1699304856; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vxMWmZFtd/CI8sGJ7jbK846Sk0z3M0dHtiMNHUYVwDo=; b=F0nq+NlshhqBwgbmYAQfswiQz1RQAekjo/Mj/fxavpN1DopPZNz/2sjwVY+hqrvI9r1JCk +np2lvriNUf+PawEEI0BpgJmRpqxTGFldaw4+HdZPFHLhiHrDWaTn5xvr2xFwzcQEDTwoc AhNz/uO0JWi24Ms1qCAp8bkn7Yl/c/yP03rOGNVJBJmPsRd4p2OlE3PMzswZ7Rytz0NUSK m9hQGkFvrCwZ5WjTMjKd/4QOpmp27u8ZRmWAd9B58+vu0oWwSVpu8BzFTf1SuOYyJpTTgN eC4c4BiDYXkfrY+d3N97JKLFSxcEbi3MtJ8izxxdpix+zVUHLZfPI2btd3eUDg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1699304856; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vxMWmZFtd/CI8sGJ7jbK846Sk0z3M0dHtiMNHUYVwDo=; b=iIknmtgfSg2FSpw6KP51q6sFYxkf9/F1c/JoIp+Uru1hpwxLGCMX8xzDQ6iLDQTjOTKDJu ciQ+0OUj30ky+6CQ== To: Petr Mladek Cc: Sergey Senozhatsky , Steven Rostedt , Thomas Gleixner , linux-kernel@vger.kernel.org Subject: [PATCH printk v2 9/9] printk: Avoid non-panic CPUs flooding ringbuffer Date: Mon, 6 Nov 2023 22:13:30 +0106 Message-Id: <20231106210730.115192-10-john.ogness@linutronix.de> In-Reply-To: <20231106210730.115192-1-john.ogness@linutronix.de> References: <20231106210730.115192-1-john.ogness@linutronix.de> MIME-Version: 1.0 X-Spam-Status: No, score=-1.6 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,INVALID_DATE_TZ_ABSURD, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Mon, 06 Nov 2023 13:08:11 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781850327182292050 X-GMAIL-MSGID: 1781850327182292050 Commit 13fb0f74d702 ("printk: Avoid livelock with heavy printk during panic") introduced a mechanism to silence non-panic CPUs if too many messages are being dropped. Aside from trying to workaround the livelock bugs of legacy consoles, it was also intended to avoid losing panic messages. However, if non-panic CPUs are flooding the ringbuffer, then reacting to dropped messages is too late. To avoid losing panic CPU messages, the tracking needs to occur when non-panic CPUs are storing messages. If non-panic CPUs have filled approximately 1/4 the ringbuffer, they need to be silenced to ensure the ringbuffer has ample space available for the panic CPU messages. Rather than trying to come up with an accurate heuristic to measure the size used by non-panic CPUs, simply restrict them to 1/4 the possible ringbuffer descriptors. In practice this will end up being around 1/3 the ringbuffer size, which still leaves ample space for the panic CPU messages. Signed-off-by: John Ogness --- kernel/printk/printk.c | 33 ++++++++++++++++++--------------- 1 file changed, 18 insertions(+), 15 deletions(-) diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c index cb99c854a648..9ac7d50c2f18 100644 --- a/kernel/printk/printk.c +++ b/kernel/printk/printk.c @@ -2315,6 +2315,8 @@ asmlinkage int vprintk_emit(int facility, int level, const struct dev_printk_info *dev_info, const char *fmt, va_list args) { + static atomic_t panic_noise_count = ATOMIC_INIT(0); + int printed_len; bool in_sched = false; @@ -2322,8 +2324,22 @@ asmlinkage int vprintk_emit(int facility, int level, if (unlikely(suppress_printk)) return 0; - if (unlikely(suppress_panic_printk) && other_cpu_in_panic()) - return 0; + if (other_cpu_in_panic()) { + if (unlikely(suppress_panic_printk)) + return 0; + + /* + * The messages on the panic CPU are the most important. If + * non-panic CPUs are generating many messages, the panic + * messages could get lost. Limit the number of non-panic + * messages to approximately 1/4 of the ringbuffer. + */ + if (atomic_inc_return_relaxed(&panic_noise_count) > + (1 << (prb->desc_ring.count_bits - 2))) { + suppress_panic_printk = 1; + return 0; + } + } if (level == LOGLEVEL_SCHED) { level = LOGLEVEL_DEFAULT; @@ -2799,8 +2815,6 @@ void console_prepend_dropped(struct printk_message *pmsg, unsigned long dropped) bool printk_get_next_message(struct printk_message *pmsg, u64 seq, bool is_extended, bool may_suppress) { - static int panic_console_dropped; - struct printk_buffers *pbufs = pmsg->pbufs; const size_t scratchbuf_sz = sizeof(pbufs->scratchbuf); const size_t outbuf_sz = sizeof(pbufs->outbuf); @@ -2828,17 +2842,6 @@ bool printk_get_next_message(struct printk_message *pmsg, u64 seq, pmsg->seq = r.info->seq; pmsg->dropped = r.info->seq - seq; - /* - * Check for dropped messages in panic here so that printk - * suppression can occur as early as possible if necessary. - */ - if (pmsg->dropped && - panic_in_progress() && - panic_console_dropped++ > 10) { - suppress_panic_printk = 1; - pr_warn_once("Too many dropped messages. Suppress messages on non-panic CPUs to prevent livelock.\n"); - } - /* Skip record that has level above the console loglevel. */ if (may_suppress && suppress_message_printing(r.info->level)) goto out;