From patchwork Wed Mar 22 18:42:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Rostedt X-Patchwork-Id: 73628 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp2508555wrt; Wed, 22 Mar 2023 11:54:10 -0700 (PDT) X-Google-Smtp-Source: AKy350ahY4rgq8R29Cwiuk5Hu9rkSCFLWJqIfCJUx6rJBLax/bhjJJsdc6wgEN20j0Kr9CGM5JPZ X-Received: by 2002:a17:902:e882:b0:1a1:f413:70b1 with SMTP id w2-20020a170902e88200b001a1f41370b1mr3016936plg.18.1679511250312; Wed, 22 Mar 2023 11:54:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679511250; cv=none; d=google.com; s=arc-20160816; b=pr26z4K062dDDMnCaAX3PvXA8WOi31nW5eel2qZ/g8z6oj6lX80FsX+zN6Cmp/o7Z+ bSxZIbHCb+wg0ReDWihBbfGWL/AojVpWIUbOCGPzmb+B04MwHfU+ApACiAJgehzWonR3 SsDf5PNwblSxYlvkV+4kzY1hwcxlp3v+p8RyCCK87Se+O0ERu5LxCRFBo08Ru06JG9LY +WV5eHP+mDSjrGmkD/IdMkxJ7uqltz50MaU4I2wnSItaZG5gZjAG5jVJUHlQDnfJzTgo IgT6tr09qUmAzEWvgMkXKdUesUtEvnnlNRIqoMs49n5Ja90NOvckp+L/fkGv6kTTQnJg tcQQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:subject:cc:to:from:date :user-agent:message-id; bh=HWHf+G/TwC0Yi0DigIsn7Kue0dAyWGPZmkV9qY39aXA=; b=JODpt8EQwSsVYnUyK2xhR5IPwBiOom5GNeiK08t4QCmmpGopriSSp/2xPdE/G9tYrD o4K7yvFnbSZNbaBKGLCz1bW2rQMXkayfG2rsZn627guRowoFpBR6hiosI5rXryhwdMn0 r+gWKmWUwAgrRFkQf8vgcTPIQ9zt2XM8YHWgbcjApJ3Xz7LzfxF2kBzQsIqM8CKXt8+E 8ywpXEbzVy6U45tcib7kONT9m7b4GX5Mgh8EEVmbAIjFvYSESDmT+oSIyU2Hzku7I7aq /LAMtyWzCc7AjRsEmPSj3YBa0w1NSGSJJxsE+emfB0Si21WTAnYszvetRt01+BAe2rUx odHQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id g12-20020a170902e38c00b0019d038765f6si15423901ple.449.2023.03.22.11.53.56; Wed, 22 Mar 2023 11:54:10 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230342AbjCVSoO (ORCPT + 99 others); Wed, 22 Mar 2023 14:44:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56244 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230214AbjCVSnk (ORCPT ); Wed, 22 Mar 2023 14:43:40 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DE39B2943F for ; Wed, 22 Mar 2023 11:43:38 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 7552962279 for ; Wed, 22 Mar 2023 18:43:38 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E8220C433D2; Wed, 22 Mar 2023 18:43:37 +0000 (UTC) Received: from rostedt by gandalf.local.home with local (Exim 4.96) (envelope-from ) id 1pf3Qt-000pbv-32; Wed, 22 Mar 2023 14:43:35 -0400 Message-ID: <20230322184335.762751069@goodmis.org> User-Agent: quilt/0.66 Date: Wed, 22 Mar 2023 14:42:50 -0400 From: Steven Rostedt To: linux-kernel@vger.kernel.org Cc: Masami Hiramatsu , Mark Rutland , Andrew Morton , Catalin Marinas , Chris Li Subject: [for-next][PATCH 11/11] ftrace: Show a list of all functions that have ever been enabled References: <20230322184239.594953818@goodmis.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, RCVD_IN_DNSWL_HI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761095188856213960?= X-GMAIL-MSGID: =?utf-8?q?1761095188856213960?= From: "Steven Rostedt (Google)" When debugging a crash that appears to be related to ftrace, but not for sure, it is useful to know if a function was ever enabled by ftrace or not. It could be that a BPF program was attached to it, or possibly a live patch. We are having crashes in the field where this information is not always known. But having ftrace set a flag if a function has ever been attached since boot up helps tremendously in trying to know if a crash had to do with something using ftrace. For analyzing crashes, the use of a kdump image can have access to the flags. When looking at issues where the kernel did not panic, the touched_functions file can simply be used. Link: https://lore.kernel.org/linux-trace-kernel/20230124095653.6fd1640e@gandalf.local.home Cc: Masami Hiramatsu Cc: Catalin Marinas Tested-by: Mark Rutland Tested-by: Chris Li Signed-off-by: Steven Rostedt (Google) --- include/linux/ftrace.h | 5 ++++- kernel/trace/ftrace.c | 51 +++++++++++++++++++++++++++++++++++++----- 2 files changed, 50 insertions(+), 6 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 931f3d904529..327046f1278d 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -548,6 +548,7 @@ bool is_ftrace_trampoline(unsigned long addr); * DIRECT - there is a direct function to call * CALL_OPS - the record can use callsite-specific ops * CALL_OPS_EN - the function is set up to use callsite-specific ops + * TOUCHED - A callback was added since boot up * * When a new ftrace_ops is registered and wants a function to save * pt_regs, the rec->flags REGS is set. When the function has been @@ -567,9 +568,10 @@ enum { FTRACE_FL_DIRECT_EN = (1UL << 23), FTRACE_FL_CALL_OPS = (1UL << 22), FTRACE_FL_CALL_OPS_EN = (1UL << 21), + FTRACE_FL_TOUCHED = (1UL << 20), }; -#define FTRACE_REF_MAX_SHIFT 21 +#define FTRACE_REF_MAX_SHIFT 20 #define FTRACE_REF_MAX ((1UL << FTRACE_REF_MAX_SHIFT) - 1) #define ftrace_rec_count(rec) ((rec)->flags & FTRACE_REF_MAX) @@ -628,6 +630,7 @@ enum { FTRACE_ITER_PROBE = (1 << 4), FTRACE_ITER_MOD = (1 << 5), FTRACE_ITER_ENABLED = (1 << 6), + FTRACE_ITER_TOUCHED = (1 << 7), }; void arch_ftrace_update_code(int command); diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 3b46dba3f69b..db8532a4d5c8 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -45,6 +45,9 @@ #include "trace_output.h" #include "trace_stat.h" +/* Flags that do not get reset */ +#define FTRACE_NOCLEAR_FLAGS (FTRACE_FL_DISABLED | FTRACE_FL_TOUCHED) + #define FTRACE_INVALID_FUNCTION "__ftrace_invalid_address__" #define FTRACE_WARN_ON(cond) \ @@ -2256,7 +2259,7 @@ static int ftrace_check_record(struct dyn_ftrace *rec, bool enable, bool update) flag ^= rec->flags & FTRACE_FL_ENABLED; if (update) { - rec->flags |= FTRACE_FL_ENABLED; + rec->flags |= FTRACE_FL_ENABLED | FTRACE_FL_TOUCHED; if (flag & FTRACE_FL_REGS) { if (rec->flags & FTRACE_FL_REGS) rec->flags |= FTRACE_FL_REGS_EN; @@ -2326,7 +2329,7 @@ static int ftrace_check_record(struct dyn_ftrace *rec, bool enable, bool update) if (update) { /* If there's no more users, clear all flags */ if (!ftrace_rec_count(rec)) - rec->flags &= FTRACE_FL_DISABLED; + rec->flags &= FTRACE_NOCLEAR_FLAGS; else /* * Just disable the record, but keep the ops TRAMP @@ -3147,7 +3150,7 @@ int ftrace_shutdown(struct ftrace_ops *ops, int command) struct dyn_ftrace *rec; do_for_each_ftrace_rec(pg, rec) { - if (FTRACE_WARN_ON_ONCE(rec->flags & ~FTRACE_FL_DISABLED)) + if (FTRACE_WARN_ON_ONCE(rec->flags & ~FTRACE_NOCLEAR_FLAGS)) pr_warn(" %pS flags:%lx\n", (void *)rec->ip, rec->flags); } while_for_each_ftrace_rec(); @@ -3598,7 +3601,10 @@ t_func_next(struct seq_file *m, loff_t *pos) !ftrace_lookup_ip(iter->hash, rec->ip)) || ((iter->flags & FTRACE_ITER_ENABLED) && - !(rec->flags & FTRACE_FL_ENABLED))) { + !(rec->flags & FTRACE_FL_ENABLED)) || + + ((iter->flags & FTRACE_ITER_TOUCHED) && + !(rec->flags & FTRACE_FL_TOUCHED))) { rec = NULL; goto retry; @@ -3857,7 +3863,7 @@ static int t_show(struct seq_file *m, void *v) return 0; } - if (iter->flags & FTRACE_ITER_ENABLED) { + if (iter->flags & (FTRACE_ITER_ENABLED | FTRACE_ITER_TOUCHED)) { struct ftrace_ops *ops; seq_printf(m, " (%ld)%s%s%s%s", @@ -3959,6 +3965,31 @@ ftrace_enabled_open(struct inode *inode, struct file *file) return 0; } +static int +ftrace_touched_open(struct inode *inode, struct file *file) +{ + struct ftrace_iterator *iter; + + /* + * This shows us what functions have ever been enabled + * (traced, direct, patched, etc). Not sure if we want lockdown + * to hide such critical information for an admin. + * Although, perhaps it can show information we don't + * want people to see, but if something had traced + * something, we probably want to know about it. + */ + + iter = __seq_open_private(file, &show_ftrace_seq_ops, sizeof(*iter)); + if (!iter) + return -ENOMEM; + + iter->pg = ftrace_pages_start; + iter->flags = FTRACE_ITER_TOUCHED; + iter->ops = &global_ops; + + return 0; +} + /** * ftrace_regex_open - initialize function tracer filter files * @ops: The ftrace_ops that hold the hash filters @@ -5872,6 +5903,13 @@ static const struct file_operations ftrace_enabled_fops = { .release = seq_release_private, }; +static const struct file_operations ftrace_touched_fops = { + .open = ftrace_touched_open, + .read = seq_read, + .llseek = seq_lseek, + .release = seq_release_private, +}; + static const struct file_operations ftrace_filter_fops = { .open = ftrace_filter_open, .read = seq_read, @@ -6336,6 +6374,9 @@ static __init int ftrace_init_dyn_tracefs(struct dentry *d_tracer) trace_create_file("enabled_functions", TRACE_MODE_READ, d_tracer, NULL, &ftrace_enabled_fops); + trace_create_file("touched_functions", TRACE_MODE_READ, + d_tracer, NULL, &ftrace_touched_fops); + ftrace_create_filter_files(&global_ops, d_tracer); #ifdef CONFIG_FUNCTION_GRAPH_TRACER