From patchwork Tue Jan 24 14:56:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Rostedt X-Patchwork-Id: 47739 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2195267wrn; Tue, 24 Jan 2023 06:58:13 -0800 (PST) X-Google-Smtp-Source: AMrXdXs/cG3Dtoo6EluW4Oc3dPJPugIuEw3dVTnGfPXp4gTOrBfbBMh8w3xRDyN1vgtZ16S9Iqbn X-Received: by 2002:a05:6a00:4088:b0:578:119c:1c52 with SMTP id bw8-20020a056a00408800b00578119c1c52mr30704744pfb.14.1674572292799; Tue, 24 Jan 2023 06:58:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674572292; cv=none; d=google.com; s=arc-20160816; b=wvKEZOrVVxTnd0fbhfQYkCzmQwXolOlh2YeS76OburDP1IZYrBP0xsALjRRyWyGJDm PbMURFV0R1vP9xX2W8Ab4EUQtkoCwstd6N4Hr+mAAomDRsxwPkLOjgzAjc/wKT5GmIeS vy9gd0D4vIwB8ZrQhogp30YS6QQYoCY6ELWBV2y8xNaKeH1RSgKgcy+Nq0Hi5YuGRRDE 9dDue4b6gVu0aafQKbYBTf9ooOXlOssExB6QMcm8qpSUMSCH1IJqJ5dEikXbo5hLcUnP eTNpsMmS82+tsKHR2Ynp99CfYG5F/kn3Nb7epN9PeQHhG6gFjYmfEzJlFtEyi/Jvi+9J XGjw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:subject:cc:to:from:date; bh=M10qW7cvN8O43niAf7fLqk/EZvYRQgr7tOQSePZoYEE=; b=g3C3cqtLVtd8HJsGsPmfWxRggK//jzPqSs0qd6DN1L756UsHguqpd3r/OzqHmtDKg2 FSR+ljLMKLOH8S9nR3OIQ/3POPL9qD4DW7+cgcKXxHabeHQz35tdvwEQJt70mRh2gAT9 RGbF4Ril6L66ag2Ub9ae3WB5G9TLJ+MexkUK5La6rDaCouwb7x6oPILQwYlNHpiQ04AU /oNNeRTYygJk/+fVTrACePbG5rXVwlQQFjr3vSBATQ68Mm5DqBMfG2Ry2jiQTPHrDa45 GN4AFMsN8dqwo5wtkrPfTuNLOsfSTf/calKR5xOEt0hKk2UffqJ3hwuL2n7srASWpZCT fgyg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id p185-20020a62d0c2000000b005812e948545si2546766pfg.161.2023.01.24.06.57.59; Tue, 24 Jan 2023 06:58:12 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235179AbjAXO5A (ORCPT + 99 others); Tue, 24 Jan 2023 09:57:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43058 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235168AbjAXO46 (ORCPT ); Tue, 24 Jan 2023 09:56:58 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 41C9F76B3; Tue, 24 Jan 2023 06:56:57 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id D48616122D; Tue, 24 Jan 2023 14:56:56 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 88650C433EF; Tue, 24 Jan 2023 14:56:55 +0000 (UTC) Date: Tue, 24 Jan 2023 09:56:53 -0500 From: Steven Rostedt To: LKML , Linux Trace Kernel Cc: Masami Hiramatsu , Mark Rutland , Catalin Marinas , chriscli@google.com Subject: [PATCH] ftrace: Show a list of all functions that have ever been enabled Message-ID: <20230124095653.6fd1640e@gandalf.local.home> X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.33; x86_64-pc-linux-gnu) MIME-Version: 1.0 X-Spam-Status: No, score=-6.7 required=5.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,RCVD_IN_DNSWL_HI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755916316535096350?= X-GMAIL-MSGID: =?utf-8?q?1755916316535096350?= From: "Steven Rostedt (Google)" When debugging a crash that appears to be related to ftrace, but not for sure, it is useful to know if a function was ever enabled by ftrace or not. It could be that a BPF program was attached to it, or possibly a live patch. We are having crashes in the field where this information is not always known. But having ftrace set a flag if a function has ever been attached since boot up helps tremendously in trying to know if a crash had to do with something using ftrace. For analyzing crashes, the use of a kdump image can have access to the flags. When looking at issues where the kernel did not panic, the touched_functions file can simply be used. Signed-off-by: Steven Rostedt (Google) Tested-by: Mark Rutland Tested-by: Chris Li --- [ This patch will conflict with Mark's patch that is going through the ARM tree. I will hold off pulling this patch until the next merge window, and rebase it on top of the changes when the tracing tree merges with upstream that has the changes from the ARM tree. ] include/linux/ftrace.h | 5 ++++- kernel/trace/ftrace.c | 51 +++++++++++++++++++++++++++++++++++++----- 2 files changed, 50 insertions(+), 6 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 99f1146614c0..76baba9bd21b 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -563,6 +563,7 @@ bool is_ftrace_trampoline(unsigned long addr); * IPMODIFY - the record allows for the IP address to be changed. * DISABLED - the record is not ready to be touched yet * DIRECT - there is a direct function to call + * TOUCHED - A callback was added since boot up * * When a new ftrace_ops is registered and wants a function to save * pt_regs, the rec->flags REGS is set. When the function has been @@ -580,9 +581,10 @@ enum { FTRACE_FL_DISABLED = (1UL << 25), FTRACE_FL_DIRECT = (1UL << 24), FTRACE_FL_DIRECT_EN = (1UL << 23), + FTRACE_FL_TOUCHED = (1UL << 22), }; -#define FTRACE_REF_MAX_SHIFT 23 +#define FTRACE_REF_MAX_SHIFT 22 #define FTRACE_REF_MAX ((1UL << FTRACE_REF_MAX_SHIFT) - 1) #define ftrace_rec_count(rec) ((rec)->flags & FTRACE_REF_MAX) @@ -641,6 +643,7 @@ enum { FTRACE_ITER_PROBE = (1 << 4), FTRACE_ITER_MOD = (1 << 5), FTRACE_ITER_ENABLED = (1 << 6), + FTRACE_ITER_TOUCHED = (1 << 7), }; void arch_ftrace_update_code(int command); diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 442438b93fe9..7687f413ab36 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -45,6 +45,9 @@ #include "trace_output.h" #include "trace_stat.h" +/* Flags that do not get reset */ +#define FTRACE_NOCLEAR_FLAGS (FTRACE_FL_DISABLED | FTRACE_FL_TOUCHED) + #define FTRACE_INVALID_FUNCTION "__ftrace_invalid_address__" #define FTRACE_WARN_ON(cond) \ @@ -2196,7 +2199,7 @@ static int ftrace_check_record(struct dyn_ftrace *rec, bool enable, bool update) flag ^= rec->flags & FTRACE_FL_ENABLED; if (update) { - rec->flags |= FTRACE_FL_ENABLED; + rec->flags |= FTRACE_FL_ENABLED | FTRACE_FL_TOUCHED; if (flag & FTRACE_FL_REGS) { if (rec->flags & FTRACE_FL_REGS) rec->flags |= FTRACE_FL_REGS_EN; @@ -2251,7 +2254,7 @@ static int ftrace_check_record(struct dyn_ftrace *rec, bool enable, bool update) if (update) { /* If there's no more users, clear all flags */ if (!ftrace_rec_count(rec)) - rec->flags &= FTRACE_FL_DISABLED; + rec->flags &= FTRACE_NOCLEAR_FLAGS; else /* * Just disable the record, but keep the ops TRAMP @@ -3067,7 +3070,7 @@ int ftrace_shutdown(struct ftrace_ops *ops, int command) struct dyn_ftrace *rec; do_for_each_ftrace_rec(pg, rec) { - if (FTRACE_WARN_ON_ONCE(rec->flags & ~FTRACE_FL_DISABLED)) + if (FTRACE_WARN_ON_ONCE(rec->flags & ~FTRACE_NOCLEAR_FLAGS)) pr_warn(" %pS flags:%lx\n", (void *)rec->ip, rec->flags); } while_for_each_ftrace_rec(); @@ -3518,7 +3521,10 @@ t_func_next(struct seq_file *m, loff_t *pos) !ftrace_lookup_ip(iter->hash, rec->ip)) || ((iter->flags & FTRACE_ITER_ENABLED) && - !(rec->flags & FTRACE_FL_ENABLED))) { + !(rec->flags & FTRACE_FL_ENABLED)) || + + ((iter->flags & FTRACE_ITER_TOUCHED) && + !(rec->flags & FTRACE_FL_TOUCHED))) { rec = NULL; goto retry; @@ -3777,7 +3783,7 @@ static int t_show(struct seq_file *m, void *v) return 0; } - if (iter->flags & FTRACE_ITER_ENABLED) { + if (iter->flags & (FTRACE_ITER_ENABLED | FTRACE_ITER_TOUCHED)) { struct ftrace_ops *ops; seq_printf(m, " (%ld)%s%s%s", @@ -3869,6 +3875,31 @@ ftrace_enabled_open(struct inode *inode, struct file *file) return 0; } +static int +ftrace_touched_open(struct inode *inode, struct file *file) +{ + struct ftrace_iterator *iter; + + /* + * This shows us what functions have ever been enabled + * (traced, direct, patched, etc). Not sure if we want lockdown + * to hide such critical information for an admin. + * Although, perhaps it can show information we don't + * want people to see, but if something had traced + * something, we probably want to know about it. + */ + + iter = __seq_open_private(file, &show_ftrace_seq_ops, sizeof(*iter)); + if (!iter) + return -ENOMEM; + + iter->pg = ftrace_pages_start; + iter->flags = FTRACE_ITER_TOUCHED; + iter->ops = &global_ops; + + return 0; +} + /** * ftrace_regex_open - initialize function tracer filter files * @ops: The ftrace_ops that hold the hash filters @@ -6137,6 +6168,13 @@ static const struct file_operations ftrace_enabled_fops = { .release = seq_release_private, }; +static const struct file_operations ftrace_touched_fops = { + .open = ftrace_touched_open, + .read = seq_read, + .llseek = seq_lseek, + .release = seq_release_private, +}; + static const struct file_operations ftrace_filter_fops = { .open = ftrace_filter_open, .read = seq_read, @@ -6601,6 +6639,9 @@ static __init int ftrace_init_dyn_tracefs(struct dentry *d_tracer) trace_create_file("enabled_functions", TRACE_MODE_READ, d_tracer, NULL, &ftrace_enabled_fops); + trace_create_file("touched_functions", TRACE_MODE_READ, + d_tracer, NULL, &ftrace_touched_fops); + ftrace_create_filter_files(&global_ops, d_tracer); #ifdef CONFIG_FUNCTION_GRAPH_TRACER