From patchwork Thu Nov 10 21:25:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guilherme G. Piccoli" X-Patchwork-Id: 18338 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp414539wru; Thu, 10 Nov 2022 14:44:43 -0800 (PST) X-Google-Smtp-Source: AMsMyM5Oe/PsqGGgBjzuCBvc4fsuCIugFFT94avGDhSg/0dzalFiBZ85lrcEsyEnEpBDUTTdae9v X-Received: by 2002:a63:1408:0:b0:46f:5bd0:492f with SMTP id u8-20020a631408000000b0046f5bd0492fmr3528512pgl.186.1668120282584; Thu, 10 Nov 2022 14:44:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668120282; cv=none; d=google.com; s=arc-20160816; b=R4u1hwcKWpKmAfwsqEYgKFAwpR9JMqcemSvwmljaASpk/yRFcQuims9vPRBu7vk7hT bWO0+ga+qcdDIbZa9BVENJ+iu9M2cOy1ECY9eDSHGKpUOgR462KaJYwXR2lMdmSR+DR/ KNnmt25MMVV3QTNtRT4m9DA7d9kh/JdN/YM/DE1cmKQVvpr61V3CYF9peTcdLlfAn3ms IvhuQIrmnTcExnou909EZr/LN89Yh29rcRjJPgImWM4bcwieFcXyGut7hBhdl9oSnHcZ /5Uk9JFHOY4VvjPHx83m/gmTnZl4p9LZ57MAya6RHekqxQjwIIrXGKZe8L9cGurVqAdZ 7tcA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:subject:from:cc:to:content-language:user-agent :mime-version:date:message-id:dkim-signature; bh=ehOVmf2BGJ/4XYcU1MULgCYiKF3gEtoi5LbwIzjFNf4=; b=eqe0cTbJZvNN7RtvgB7b7LcK0Gn97VkQ3H81N4xdTxgo52o1MkBSxVtxlgSbtQceJN ESZsw0/BUPzYwu4q3X+XMU+7AKBSaqGGOSwZyZngNsNLY4xVpvxWBA4UF/0GfbKWbYQG qEesk/yBEb2Dh5f5e5snRH3UnlCt1zZX9+CmQAgVkndvAGIJ1rq4zlIWBUBZNGcA25ry MLg6YLATcReCkEAr3pyxgabrsDC1B5tQJicqewA/GNtCtdrlRHCBsBwotCuy8M7e3ghE dvgTEkt9FL0W5dEgZL9ACY4sDS+XeH8A0H/zXfxSu/TdqLhT6BYMYNVo9VN8QxuFy4N5 3xmg== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@igalia.com header.s=20170329 header.b=i4gj0D5f; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id jj20-20020a170903049400b0017d2c45e8ffsi541963plb.171.2022.11.10.14.44.28; Thu, 10 Nov 2022 14:44:42 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=fail header.i=@igalia.com header.s=20170329 header.b=i4gj0D5f; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230505AbiKJWMm (ORCPT + 99 others); Thu, 10 Nov 2022 17:12:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35892 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230522AbiKJWMj (ORCPT ); Thu, 10 Nov 2022 17:12:39 -0500 Received: from fanzine2.igalia.com (fanzine2.igalia.com [213.97.179.56]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DDA30450B6 for ; Thu, 10 Nov 2022 14:12:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=igalia.com; s=20170329; h=Subject:From:Cc:To:MIME-Version:Date:Message-ID:Content-Type: Sender:Reply-To:Content-Transfer-Encoding:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: In-Reply-To:References:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=ehOVmf2BGJ/4XYcU1MULgCYiKF3gEtoi5LbwIzjFNf4=; b=i4gj0D5faVv1xOXerbvbwjpkVj r+9WoDtYzORegVRsH57rIBoGtc+Vc03xnNWODx/W3d6llbjxDiR2FkSWRrLswGuLtg6xVOBBOJw/k 12XOiEi5Q1vv7IoXULOpx/CC5GNR2O7WEhPf55vJcpAzX/nmIryxGfsc4ZYeH4TKLtzblHv7HK3VC tlbQPkCIuQCU+A2jXmAcFUjf4ejooOKCxUb3eTLXlltKYMMV7YlEue8Dy1DeMb8LnjQevP4eLLflp QVt1EJbzmwWeDQAaFNusmsZF42GfviQfoMNPA9ab+l9ZAEzTULL6jPsWXWTxliix7IBtLb40z5XIs va56zLkg==; Received: from [177.102.6.147] (helo=[192.168.1.60]) by fanzine2.igalia.com with esmtpsa (Cipher TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_128_GCM:128) (Exim) id 1otF3R-00FllT-RH; Thu, 10 Nov 2022 22:25:46 +0100 Message-ID: <1ef5fe19-a82f-835e-fda5-455e9c2b94b4@igalia.com> Date: Thu, 10 Nov 2022 18:25:41 -0300 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.3.2 Content-Language: en-US To: linux-kernel , Steven Rostedt Cc: Masami Hiramatsu , Mark Rutland , "rcu@vger.kernel.org" , "kernel-dev@igalia.com" , "Guilherme G. Piccoli" From: "Guilherme G. Piccoli" Subject: RCU stall on 6.1-rc4 (and some previous releases) related to ftrace X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749150893468282510?= X-GMAIL-MSGID: =?utf-8?q?1749150893468282510?= Hi folks, I've noticed some RCU stalls when enabling ftrace filtering in 6.1-rc4 (also 6.1-rc3 and I guess I've seen this before, but only in 6.0+). Here is the full dmesg: https://termbin.com/4xo6 , see below for a small snippet of the stall [0]. I've briefly talked to Steve on IRC, and he mentioned that my kernel seemed to have PREEMPT_VOLUNTARY=y (and indeed, this is the case - see the full config here: https://termbin.com/t48d), saying that maybe adding a cond_resched() in the loop would help. So, I've cooked a small hack (see the patch attached) and it seems to work. Steve: lemme know if you want to send it (since it's your idea and maybe my hack is not covering all cases), or if you prefer, I can work a commit message and send myself. Any other advice / tests required, please lemme me know and I'll be glad in contributing. Cheers, Guilherme [0] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 1-.... } 6 jiffies s: 1289 root: 0x2/. rcu: blocking rcu_node structures (internal RCU debug): Sending NMI from CPU 4 to CPUs 1: NMI backtrace for cpu 1 [...] RIP: 0010:find_kallsyms_symbol+0x85/0x1a0 [...] Call Trace: ? get_refcyc_per_delivery.constprop.0+0x200/0x200 [amdgpu] module_address_lookup+0x63/0xc0 ? get_refcyc_per_delivery.constprop.0+0x200/0x200 [amdgpu] kallsyms_lookup_buildid+0xb6/0x130 ftrace_match_record+0x43/0xf0 ? match_records+0x210/0x3b0 ? __kmem_cache_alloc_node+0x165/0x260 match_records+0x13a/0x3b0 ftrace_process_regex.isra.0+0x101/0x120 ftrace_filter_write+0x57/0x90 [...] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 1-.... } 25 jiffies s: 1289 root: 0x2/. rcu: blocking rcu_node structures (internal RCU debug): Sending NMI from CPU 4 to CPUs 1: [...] RIP: 0010:find_kallsyms_symbol+0x8f/0x1a0 [...] Call Trace: ? dcn21_dmcu_create+0xd0/0xd0 [amdgpu] module_address_lookup+0x63/0xc0 ? dcn21_dmcu_create+0xd0/0xd0 [amdgpu] kallsyms_lookup_buildid+0xb6/0x130 ftrace_match_record+0x43/0xf0 ? match_records+0x210/0x3b0 ? __kmem_cache_alloc_node+0x165/0x260 match_records+0x13a/0x3b0 ftrace_process_regex.isra.0+0x101/0x120 ftrace_filter_write+0x57/0x90 [...] Acked-by: Paul E. McKenney From 45fda369a49d0e59689e4a6b9babc12598265825 Mon Sep 17 00:00:00 2001 From: "Guilherme G. Piccoli" Date: Thu, 10 Nov 2022 16:42:17 -0300 Subject: [PATCH] ftrace/hack: Add cond_resched() to prevent RCU stall Suggestion from Steve on IRC --- kernel/trace/ftrace.c | 1 + 1 file changed, 1 insertion(+) diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 7dc023641bf1..8c2c04b2eb52 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -4184,6 +4184,7 @@ match_records(struct ftrace_hash *hash, char *func, int len, char *mod) if (rec->flags & FTRACE_FL_DISABLED) continue; + cond_resched(); if (ftrace_match_record(rec, &func_g, mod_match, exclude_mod)) { ret = enter_record(hash, rec, clear_filter); if (ret < 0) { -- 2.38.0