From patchwork Thu Oct 20 00:01:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Rostedt X-Patchwork-Id: 5941 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp596453wrs; Wed, 19 Oct 2022 17:06:59 -0700 (PDT) X-Google-Smtp-Source: AMsMyM5vDLU+Lj/BguP2q2sibM6gL6osTg4czUwyoYbNStlcXK1DGycH6SXm46OYFHxmPe6XZ0in X-Received: by 2002:aa7:d80b:0:b0:45f:b9ed:6e3c with SMTP id v11-20020aa7d80b000000b0045fb9ed6e3cmr1778424edq.22.1666224419324; Wed, 19 Oct 2022 17:06:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666224419; cv=none; d=google.com; s=arc-20160816; b=g/ln7htYOR6o6qSIlRzg2Rcm+cuEMNlueGEc47YeHhKjV74koIqmeBB14J70vZvhEJ RV1DoD421lasXTg4j4PwMf9sKmAfDiB+iBQvAXKt814yum720RsD3dUM6bJj+XzBwLdB BkQu6yDaw4VXXPKwA8G+fNViRf65K9l66pCls0izTr+rREPpcM1E10dhDoPN50iY9jnA axcvMX+phW9rYSNoJhPvpBIHg+S67zTTJSOieJHVWgqr3J3TBjgGu2BBHXL2BfskRBtB 1I8NmMlDOIRX12kNIWiKQMuStypeccmupYO+FOGcHyfkttUiz8Ossgzal7a4exeJevR0 ltvA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:subject:cc:to:from:date; bh=s8LxUq/bOlJlyGK3zYmQr9lK/EwcR8DTy2ruEQj9XzY=; b=sVFaKwq7tlW6+ILkPcTFSk9TayzpE6NOTrwUq8mEons1/gEx2ZPASOJOBbeyLoJ78B cwk2Cr19jGJRSav6OpodHB4fiXnbL3dq9mBv/jz6Sd8hLprQIR/WiaAiWtNzJyRk/QLw pB6Qu6pjy3JUa3ue3IjsPwHyKK0j5UgZMKsi3FthRZ22ffvteLuvAcH0G8TJMuTdrHXE 1JvwxsI5sIujKADlvwKRJaWPYVcJtjQe0nRRnWQ5y/5wNFJ5I5SApNuhclBDSzgkPVFr f+shY6/G0bgeeTtai92wS6IV0p1XeC/CqzVlNDOgJW0xm7fwBnT+kg1//+PL4ykpZQBz FFLw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id e18-20020a17090658d200b0078e0e8508fbsi17487297ejs.457.2022.10.19.17.06.30; Wed, 19 Oct 2022 17:06:59 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229722AbiJTABl (ORCPT + 99 others); Wed, 19 Oct 2022 20:01:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56312 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229930AbiJTABj (ORCPT ); Wed, 19 Oct 2022 20:01:39 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B1F6E169CC1; Wed, 19 Oct 2022 17:01:37 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 67AD9B825B8; Thu, 20 Oct 2022 00:01:36 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 37AADC433D6; Thu, 20 Oct 2022 00:01:34 +0000 (UTC) Date: Wed, 19 Oct 2022 20:01:37 -0400 From: Steven Rostedt To: LKML Cc: Linux Trace Kernel , Masami Hiramatsu , Andrew Morton , Mathieu Desnoyers , "Paul E. McKenney" , Joel Fernandes , Tom Zanussi Subject: [PATCH] tracing: Add trace_trigger kernel command line option Message-ID: <20221019200137.70343645@gandalf.local.home> X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.33; x86_64-pc-linux-gnu) MIME-Version: 1.0 X-Spam-Status: No, score=-6.7 required=5.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,RCVD_IN_DNSWL_HI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747162936262937974?= X-GMAIL-MSGID: =?utf-8?q?1747162936262937974?= From: "Steven Rostedt (Google)" Allow triggers to be enabled at kernel boot up. For example: trace_trigger="sched_switch.stacktrace if prev_state == 2" The above will enable the stacktrace trigger on top of the sched_switch event and only trigger if its prev_state is 2 (TASK_UNINTERRUPTIBLE). Then at boot up, a stacktrace will trigger and be recorded in the tracing ring buffer every time the sched_switch happens where the previous state is TASK_INTERRUPTIBLE. As this calls into tracepoint logic during very early boot (before interrupts are enabled), a check has to be done to see if early boot interrupts are still disabled, and if so, avoid any call to RCU synchronization, as that will enable interrupts and cause boot up issues. Signed-off-by: Steven Rostedt (Google) --- .../admin-guide/kernel-parameters.txt | 19 ++++++ include/linux/tracepoint.h | 4 ++ kernel/trace/trace.c | 3 +- kernel/trace/trace_events.c | 63 ++++++++++++++++++- kernel/tracepoint.c | 6 ++ 5 files changed, 92 insertions(+), 3 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index a465d5242774..ccf91a4bf113 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -6257,6 +6257,25 @@ See also Documentation/trace/ftrace.rst "trace options" section. + trace_trigger=[trigger-list] + [FTRACE] Add a event trigger on specific events. + Set a trigger on top of a specific event, with an optional + filter. + + The format is is "trace_trigger=.[ if ],..." + Where more than one trigger may be specified that are comma deliminated. + + For example: + + trace_trigger="sched_switch.stacktrace if prev_state == 2" + + The above will enable the "stacktrace" trigger on the "sched_switch" + event but only trigger it if the "prev_state" of the "sched_switch" + event is "2" (TASK_UNINTERUPTIBLE). + + See also "Event triggers" in Documentation/trace/events.rst + + traceoff_on_warning [FTRACE] enable this option to disable tracing when a warning is hit. This turns off "tracing_on". Tracing can diff --git a/include/linux/tracepoint.h b/include/linux/tracepoint.h index 4b33b95eb8be..a5c6b5772897 100644 --- a/include/linux/tracepoint.h +++ b/include/linux/tracepoint.h @@ -90,6 +90,10 @@ int unregister_tracepoint_module_notifier(struct notifier_block *nb) #ifdef CONFIG_TRACEPOINTS static inline void tracepoint_synchronize_unregister(void) { + /* Early updates do not need synchronization */ + if (early_boot_irqs_disabled) + return; + synchronize_srcu(&tracepoint_srcu); synchronize_rcu(); } diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c index 47a44b055a1d..c03fd7037add 100644 --- a/kernel/trace/trace.c +++ b/kernel/trace/trace.c @@ -2749,7 +2749,8 @@ void trace_buffered_event_disable(void) preempt_enable(); /* Wait for all current users to finish */ - synchronize_rcu(); + if (!early_boot_irqs_disabled) + synchronize_rcu(); for_each_tracing_cpu(cpu) { free_page((unsigned long)per_cpu(trace_buffered_event, cpu)); diff --git a/kernel/trace/trace_events.c b/kernel/trace/trace_events.c index 0356cae0cf74..06554939252c 100644 --- a/kernel/trace/trace_events.c +++ b/kernel/trace/trace_events.c @@ -2796,6 +2796,44 @@ trace_create_new_event(struct trace_event_call *call, return file; } +#ifdef CONFIG_HIST_TRIGGERS +#define MAX_BOOT_TRIGGERS 32 + +static struct boot_triggers { + const char *event; + char *trigger; +} bootup_triggers[MAX_BOOT_TRIGGERS]; + +static char bootup_trigger_buf[COMMAND_LINE_SIZE]; +static int nr_boot_triggers; + +static __init int setup_trace_triggers(char *str) +{ + char *trigger; + char *buf; + int i; + + strlcpy(bootup_trigger_buf, str, COMMAND_LINE_SIZE); + ring_buffer_expanded = true; + disable_tracing_selftest("running event triggers"); + + buf = bootup_trigger_buf; + for (i = 0; i < MAX_BOOT_TRIGGERS; i++) { + trigger = strsep(&buf, ","); + if (!trigger) + break; + bootup_triggers[i].event = strsep(&trigger, "."); + bootup_triggers[i].trigger = strsep(&trigger, "."); + if (!bootup_triggers[i].trigger) + break; + } + + nr_boot_triggers = i; + return 1; +} +__setup("trace_trigger=", setup_trace_triggers); +#endif + /* Add an event to a trace directory */ static int __trace_add_new_event(struct trace_event_call *call, struct trace_array *tr) @@ -2822,12 +2860,32 @@ __trace_early_add_new_event(struct trace_event_call *call, struct trace_array *tr) { struct trace_event_file *file; + int ret; + int i; file = trace_create_new_event(call, tr); if (!file) return -ENOMEM; - return event_define_fields(call); + ret = event_define_fields(call); + if (ret) + return ret; + +#ifdef CONFIG_HIST_TRIGGERS + for (i = 0; i < nr_boot_triggers; i++) { + if (strcmp(trace_event_name(call), bootup_triggers[i].event)) + continue; + mutex_lock(&event_mutex); + ret = trigger_process_regex(file, bootup_triggers[i].trigger); + mutex_unlock(&event_mutex); + if (ret) + pr_err("Failed to register trigger '%s' on event %s\n", + bootup_triggers[i].trigger, + bootup_triggers[i].event); + } +#endif + + return 0; } struct ftrace_module_file_ops; @@ -3726,6 +3784,8 @@ static __init int event_trace_enable(void) list_add(&call->list, &ftrace_events); } + register_trigger_cmds(); + /* * We need the top trace array to have a working set of trace * points at early init, before the debug files and directories @@ -3740,7 +3800,6 @@ static __init int event_trace_enable(void) register_event_cmds(); - register_trigger_cmds(); return 0; } diff --git a/kernel/tracepoint.c b/kernel/tracepoint.c index f23144af5743..f6e4ee1e40b3 100644 --- a/kernel/tracepoint.c +++ b/kernel/tracepoint.c @@ -48,6 +48,9 @@ static void tp_rcu_get_state(enum tp_transition_sync sync) { struct tp_transition_snapshot *snapshot = &tp_transition_snapshot[sync]; + if (early_boot_irqs_disabled) + return; + /* Keep the latest get_state snapshot. */ snapshot->rcu = get_state_synchronize_rcu(); snapshot->srcu = start_poll_synchronize_srcu(&tracepoint_srcu); @@ -58,6 +61,9 @@ static void tp_rcu_cond_sync(enum tp_transition_sync sync) { struct tp_transition_snapshot *snapshot = &tp_transition_snapshot[sync]; + if (early_boot_irqs_disabled) + return; + if (!snapshot->ongoing) return; cond_synchronize_rcu(snapshot->rcu);