From patchwork Sun Nov 5 16:06:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 161658 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp2196062vqu; Sun, 5 Nov 2023 08:07:28 -0800 (PST) X-Google-Smtp-Source: AGHT+IEQbCBOwM4yuFbZLgFv8Zcv7nhVaiJveRxVWB6aP5BLB0VO4iMsQNplGHRvkeW6C+gBgsU5 X-Received: by 2002:a05:6a20:daa0:b0:181:b044:449e with SMTP id iy32-20020a056a20daa000b00181b044449emr9538705pzb.54.1699200448675; Sun, 05 Nov 2023 08:07:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699200448; cv=none; d=google.com; s=arc-20160816; b=d9/c2K14WGCeARNg+F15Tc7pY+ylxdbe/Op/oKKI31TiLtL/n+LVy4h+850Mq36dB3 rpar558URf9mBwxRH826VjgAwKccykSczxVR8A2/KqXN5dtCAv28glBkcAJ6rqX68y9Q rqTwphJTqN1+c6RAFNkpOnyGwz5lnb+BMY9b0Vu3p7Uo6JCIyJiNoK/rlOK6UgTPiJoZ zQwPBxPl97REkzN83QWZXC0zISAmQ5vg4+ixMx+VhAtaa11D+au3ZEOq619FM2RRfkfJ mHbeYa8y2qW9xfHjOX366D3TEcF7EmKXIkgNS9X0/XMoC2rNq1GORb4KzSarsBfbpRqt OsEg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=qfk82260fwHivQEmABc475lIDB13RydsA/UW3+hrbac=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=0wKExhy0RgdzqkK73Pt3qE1cqhQhCeePbNolpqolWUlQvLJHPLGnDLG6UjQjXdNh7F kzNoHLHIxEvf61++JWmayIHDlh778U8UiGk2HhCGvVu/ZSvs9z67qDLsIRGG76AkwZlL lg94xujKoVo6i6gSTwVAplcVsKMNBNYQCLOQhpGw+VI5m6q9jnFlMXcvYf2Cu3rMyb7w CuC9nFkAwbacTFkr/x0Bg/sqhO+0skrELMrTy5yCijHDcI2weApmOJqarzpIPQq+4bJo IGBvhmOFYvsvgnrxxwweWE/S/ZwPZE4P+IDoBSIO8LHXwrkL3UXeNElGDeL/EJ72qo1O efvg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=ef713k2m; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from agentk.vger.email (agentk.vger.email. [23.128.96.32]) by mx.google.com with ESMTPS id 126-20020a630284000000b0058b8c72c5ddsi5964047pgc.607.2023.11.05.08.07.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Nov 2023 08:07:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=ef713k2m; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id DC00B8057DAE; Sun, 5 Nov 2023 08:07:16 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229562AbjKEQGx (ORCPT + 34 others); Sun, 5 Nov 2023 11:06:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55276 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229537AbjKEQGv (ORCPT ); Sun, 5 Nov 2023 11:06:51 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D4B21191 for ; Sun, 5 Nov 2023 08:06:48 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A349EC433C8; Sun, 5 Nov 2023 16:06:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1699200408; bh=EOdwMS7vJXCXxyNePs9bFMY5mRinN5nN+9oKZfWkeWA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ef713k2m2K4eL84abS39rEQUpBZwFs43cTbJdH3llK22MC21YmzXABUiIiN7D8bjt /vgWl0JqdcWk2tEAJZ+HXoMjE7CGnZYqOB642cjq8stbyC7LpPp1bYxMOaXX78dkbC Db2kLPcdmDWjLmRgJYzy/kzuAqMyRD0LYULsnwMdYNCy7LkIFNRkxs7JF3R5Z9fj00 R2dD5noVOcxWyUK/QOIcVhhogsmo4hEryhU+RUwIMXYaXIOlqqnpUe0HSdPeS4Y1vz cKbdazyENiYlzmh38MQYqnyrGIfr+L0Z0osDfe16RP9Fy6NYil782LRyB+F0gIl//R 0jkk1vyRE0lKA== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [RFC PATCH 01/32] seq_buf: Export seq_buf_puts() Date: Mon, 6 Nov 2023 01:06:42 +0900 Message-Id: <169920040166.482486.9804422686448685654.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169920038849.482486.15796387219966662967.stgit@devnote2> References: <169920038849.482486.15796387219966662967.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Sun, 05 Nov 2023 08:07:16 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781740809507230728 X-GMAIL-MSGID: 1781740809507230728 From: Christophe JAILLET Mark seq_buf_puts() which is part of the seq_buf API to be exported to kernel loadable GPL modules. Link: https://lkml.kernel.org/r/b9e3737f66ec2450221b492048ce0d9c65c84953.1698861216.git.christophe.jaillet@wanadoo.fr Signed-off-by: Christophe JAILLET Signed-off-by: Steven Rostedt (Google) Signed-off-by: Masami Hiramatsu (Google) --- lib/seq_buf.c | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/seq_buf.c b/lib/seq_buf.c index 45c450f423fa..46a1b00c3815 100644 --- a/lib/seq_buf.c +++ b/lib/seq_buf.c @@ -189,6 +189,7 @@ int seq_buf_puts(struct seq_buf *s, const char *str) seq_buf_set_overflow(s); return -1; } +EXPORT_SYMBOL_GPL(seq_buf_puts); /** * seq_buf_putc - sequence printing of simple character From patchwork Sun Nov 5 16:06:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 161657 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp2195902vqu; Sun, 5 Nov 2023 08:07:11 -0800 (PST) X-Google-Smtp-Source: AGHT+IGrtxvYJRPYUsu3n6epithHRKuVesr8i+id0KS4G4wnfGXdH+G/CQLYqr30GvTIfGm4ehR8 X-Received: by 2002:a05:6a00:2d9e:b0:6c3:6cdc:a297 with SMTP id fb30-20020a056a002d9e00b006c36cdca297mr5747710pfb.14.1699200431349; Sun, 05 Nov 2023 08:07:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699200431; cv=none; d=google.com; s=arc-20160816; b=eEZ8rwr2WhtXtau+HOIiEw6vwMi7Fvq4s43l09sy6QmUGAP+I59zH8yLSf/Nq7Nr/N kepTY7sOY4zWqT7xKEEg5HwB+3PgZSNIBY2Y/gT1Zmjs+bpHyUGVzJY67IyXW4dbC90O J6s+b5V09ZoWjoISP+J1MTAgHDYL5dlW2NYqMEqnKRKield/hxxlp3qDrk5HZZuHX/0g dq4NWEjBhPSAv+Lx6dSnPk0gdpX9YrIeQruFmvnHHpZivwRsK2HYMfQFZX9MQMz6BCAv j5pVkD+2eNOIsk9U8YR+ubNnXSQJh+LNhQ6BNLmpU/H8nCofug1TShsifhIgLORS/Yj2 Ft0g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=q61lus1nT4A5GtGMeJ/7KxvuB+maOoUqYP+3LFK9nEU=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=nvMHvQunNq6YHPa5862PoyTbwmaS+GlhQ3dSuE/HKycsjCRBQX6wKlvxPzD9D128lM o3MPc3H6gI1VKydgF1QIdF/1lhrl55zXo71fwhzsMtcIDhj5rHAoBrPWooNgFhHlGCmp wldp2vbRzB0TneToiZGQtdQ7Dnx42lI9cAnbybKHYthffgvxQ5cqB2TqyC9ubJ7BzFom gcgoT6Fe+ocI1UyV2NOBS2eSGRNFQWSPldSa5gmsv4TKDQ0ZN6sYsekWfE7r3sbzYmkY vBM4EWCucVwG7UcIGyKtwX7iHopXNtNitnZfEAugynzv3oICUKY0r0FAjMJINKEibg4f EIUQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=n9CtgMez; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id j10-20020a056a00130a00b006c1691263cesi6048321pfu.291.2023.11.05.08.07.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Nov 2023 08:07:11 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=n9CtgMez; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id D82FE8057E36; Sun, 5 Nov 2023 08:07:09 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229593AbjKEQHH (ORCPT + 34 others); Sun, 5 Nov 2023 11:07:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47746 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229612AbjKEQHE (ORCPT ); Sun, 5 Nov 2023 11:07:04 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2A410F2 for ; Sun, 5 Nov 2023 08:07:01 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id EA0A2C433C8; Sun, 5 Nov 2023 16:06:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1699200420; bh=BaKMcq+s6VngWMd3ngPTESGkVeznx5ygK1pn7TSuURA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=n9CtgMezbPeeK/nKmkxsceKVVBcgtUz1XiJduPsHNgG/931SKatSOvZn+99oVPNMX jyYV8qeWtoNblkgoB/pa/QwSjois7amJOBGRfYBPGyktzeeRKvF+XOWiJ3N3hrvpFM 7JW2z2NB/9vUzbmeS+hUX9L1zpkqoqavle05fI5JDOHXQHrnCNSoC9kPTXLnzykRjx 2P4pqfO5jUaBknia1lT/9bTr7AJ8+1LMZy8xhBTQzkp9KEYhaETzTIhjCHUMnzWoiF H2U13LMeCjk+oBNzA7mopz0XMmA6EO+jEz8bHQumEYXvFDUNmAd+DQ+z17za8tWQGo 2E6W0a6gAP1pg== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [RFC PATCH 02/32] function_graph: Convert ret_stack to a series of longs Date: Mon, 6 Nov 2023 01:06:54 +0900 Message-Id: <169920041390.482486.12438228411786624723.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169920038849.482486.15796387219966662967.stgit@devnote2> References: <169920038849.482486.15796387219966662967.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-5.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Sun, 05 Nov 2023 08:07:10 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781740791603911357 X-GMAIL-MSGID: 1781740791603911357 From: Steven Rostedt (VMware) In order to make it possible to have multiple callbacks registered with the function_graph tracer, the retstack needs to be converted from an array of ftrace_ret_stack structures to an array of longs. This will allow to store the list of callbacks on the stack for the return side of the functions. Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- include/linux/sched.h | 2 - kernel/trace/fgraph.c | 124 ++++++++++++++++++++++++++++--------------------- 2 files changed, 71 insertions(+), 55 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 77f01ac385f7..3af00d726847 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1386,7 +1386,7 @@ struct task_struct { int curr_ret_depth; /* Stack of return addresses for return function tracing: */ - struct ftrace_ret_stack *ret_stack; + unsigned long *ret_stack; /* Timestamp for last schedule: */ unsigned long long ftrace_timestamp; diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index c83c005e654e..30edeb6d4aa9 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -25,6 +25,18 @@ #define ASSIGN_OPS_HASH(opsname, val) #endif +#define FGRAPH_RET_SIZE sizeof(struct ftrace_ret_stack) +#define FGRAPH_RET_INDEX (ALIGN(FGRAPH_RET_SIZE, sizeof(long)) / sizeof(long)) +#define SHADOW_STACK_SIZE (PAGE_SIZE) +#define SHADOW_STACK_INDEX \ + (ALIGN(SHADOW_STACK_SIZE, sizeof(long)) / sizeof(long)) +/* Leave on a buffer at the end */ +#define SHADOW_STACK_MAX_INDEX (SHADOW_STACK_INDEX - FGRAPH_RET_INDEX) + +#define RET_STACK(t, index) ((struct ftrace_ret_stack *)(&(t)->ret_stack[index])) +#define RET_STACK_INC(c) ({ c += FGRAPH_RET_INDEX; }) +#define RET_STACK_DEC(c) ({ c -= FGRAPH_RET_INDEX; }) + DEFINE_STATIC_KEY_FALSE(kill_ftrace_graph); int ftrace_graph_active; @@ -69,6 +81,7 @@ static int ftrace_push_return_trace(unsigned long ret, unsigned long func, unsigned long frame_pointer, unsigned long *retp) { + struct ftrace_ret_stack *ret_stack; unsigned long long calltime; int index; @@ -85,23 +98,25 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func, smp_rmb(); /* The return trace stack is full */ - if (current->curr_ret_stack == FTRACE_RETFUNC_DEPTH - 1) { + if (current->curr_ret_stack >= SHADOW_STACK_MAX_INDEX) { atomic_inc(¤t->trace_overrun); return -EBUSY; } calltime = trace_clock_local(); - index = ++current->curr_ret_stack; + index = current->curr_ret_stack; + RET_STACK_INC(current->curr_ret_stack); + ret_stack = RET_STACK(current, index); barrier(); - current->ret_stack[index].ret = ret; - current->ret_stack[index].func = func; - current->ret_stack[index].calltime = calltime; + ret_stack->ret = ret; + ret_stack->func = func; + ret_stack->calltime = calltime; #ifdef HAVE_FUNCTION_GRAPH_FP_TEST - current->ret_stack[index].fp = frame_pointer; + ret_stack->fp = frame_pointer; #endif #ifdef HAVE_FUNCTION_GRAPH_RET_ADDR_PTR - current->ret_stack[index].retp = retp; + ret_stack->retp = retp; #endif return 0; } @@ -148,7 +163,7 @@ int function_graph_enter(unsigned long ret, unsigned long func, return 0; out_ret: - current->curr_ret_stack--; + RET_STACK_DEC(current->curr_ret_stack); out: current->curr_ret_depth--; return -EBUSY; @@ -159,11 +174,13 @@ static void ftrace_pop_return_trace(struct ftrace_graph_ret *trace, unsigned long *ret, unsigned long frame_pointer) { + struct ftrace_ret_stack *ret_stack; int index; index = current->curr_ret_stack; + RET_STACK_DEC(index); - if (unlikely(index < 0 || index >= FTRACE_RETFUNC_DEPTH)) { + if (unlikely(index < 0 || index > SHADOW_STACK_MAX_INDEX)) { ftrace_graph_stop(); WARN_ON(1); /* Might as well panic, otherwise we have no where to go */ @@ -171,6 +188,7 @@ ftrace_pop_return_trace(struct ftrace_graph_ret *trace, unsigned long *ret, return; } + ret_stack = RET_STACK(current, index); #ifdef HAVE_FUNCTION_GRAPH_FP_TEST /* * The arch may choose to record the frame pointer used @@ -186,22 +204,22 @@ ftrace_pop_return_trace(struct ftrace_graph_ret *trace, unsigned long *ret, * Note, -mfentry does not use frame pointers, and this test * is not needed if CC_USING_FENTRY is set. */ - if (unlikely(current->ret_stack[index].fp != frame_pointer)) { + if (unlikely(ret_stack->fp != frame_pointer)) { ftrace_graph_stop(); WARN(1, "Bad frame pointer: expected %lx, received %lx\n" " from func %ps return to %lx\n", current->ret_stack[index].fp, frame_pointer, - (void *)current->ret_stack[index].func, - current->ret_stack[index].ret); + (void *)ret_stack->func, + ret_stack->ret); *ret = (unsigned long)panic; return; } #endif - *ret = current->ret_stack[index].ret; - trace->func = current->ret_stack[index].func; - trace->calltime = current->ret_stack[index].calltime; + *ret = ret_stack->ret; + trace->func = ret_stack->func; + trace->calltime = ret_stack->calltime; trace->overrun = atomic_read(¤t->trace_overrun); trace->depth = current->curr_ret_depth--; /* @@ -262,7 +280,7 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs * curr_ret_stack is after that. */ barrier(); - current->curr_ret_stack--; + RET_STACK_DEC(current->curr_ret_stack); if (unlikely(!ret)) { ftrace_graph_stop(); @@ -305,12 +323,13 @@ unsigned long ftrace_return_to_handler(unsigned long frame_pointer) struct ftrace_ret_stack * ftrace_graph_get_ret_stack(struct task_struct *task, int idx) { - idx = task->curr_ret_stack - idx; + int index = task->curr_ret_stack; - if (idx >= 0 && idx <= task->curr_ret_stack) - return &task->ret_stack[idx]; + index -= FGRAPH_RET_INDEX * (idx + 1); + if (index < 0) + return NULL; - return NULL; + return RET_STACK(task, index); } /** @@ -332,18 +351,20 @@ ftrace_graph_get_ret_stack(struct task_struct *task, int idx) unsigned long ftrace_graph_ret_addr(struct task_struct *task, int *idx, unsigned long ret, unsigned long *retp) { + struct ftrace_ret_stack *ret_stack; int index = task->curr_ret_stack; int i; if (ret != (unsigned long)dereference_kernel_function_descriptor(return_to_handler)) return ret; - if (index < 0) - return ret; + RET_STACK_DEC(index); - for (i = 0; i <= index; i++) - if (task->ret_stack[i].retp == retp) - return task->ret_stack[i].ret; + for (i = index; i >= 0; RET_STACK_DEC(i)) { + ret_stack = RET_STACK(task, i); + if (ret_stack->retp == retp) + return ret_stack->ret; + } return ret; } @@ -357,14 +378,15 @@ unsigned long ftrace_graph_ret_addr(struct task_struct *task, int *idx, return ret; task_idx = task->curr_ret_stack; + RET_STACK_DEC(task_idx); if (!task->ret_stack || task_idx < *idx) return ret; task_idx -= *idx; - (*idx)++; + RET_STACK_INC(*idx); - return task->ret_stack[task_idx].ret; + return RET_STACK(task, task_idx); } #endif /* HAVE_FUNCTION_GRAPH_RET_ADDR_PTR */ @@ -402,7 +424,7 @@ trace_func_graph_ent_t ftrace_graph_entry = ftrace_graph_entry_stub; static trace_func_graph_ent_t __ftrace_graph_entry = ftrace_graph_entry_stub; /* Try to assign a return stack array on FTRACE_RETSTACK_ALLOC_SIZE tasks. */ -static int alloc_retstack_tasklist(struct ftrace_ret_stack **ret_stack_list) +static int alloc_retstack_tasklist(unsigned long **ret_stack_list) { int i; int ret = 0; @@ -410,10 +432,7 @@ static int alloc_retstack_tasklist(struct ftrace_ret_stack **ret_stack_list) struct task_struct *g, *t; for (i = 0; i < FTRACE_RETSTACK_ALLOC_SIZE; i++) { - ret_stack_list[i] = - kmalloc_array(FTRACE_RETFUNC_DEPTH, - sizeof(struct ftrace_ret_stack), - GFP_KERNEL); + ret_stack_list[i] = kmalloc(SHADOW_STACK_SIZE, GFP_KERNEL); if (!ret_stack_list[i]) { start = 0; end = i; @@ -431,9 +450,9 @@ static int alloc_retstack_tasklist(struct ftrace_ret_stack **ret_stack_list) if (t->ret_stack == NULL) { atomic_set(&t->trace_overrun, 0); - t->curr_ret_stack = -1; + t->curr_ret_stack = 0; t->curr_ret_depth = -1; - /* Make sure the tasks see the -1 first: */ + /* Make sure the tasks see the 0 first: */ smp_wmb(); t->ret_stack = ret_stack_list[start++]; } @@ -453,6 +472,7 @@ ftrace_graph_probe_sched_switch(void *ignore, bool preempt, struct task_struct *next, unsigned int prev_state) { + struct ftrace_ret_stack *ret_stack; unsigned long long timestamp; int index; @@ -477,8 +497,11 @@ ftrace_graph_probe_sched_switch(void *ignore, bool preempt, */ timestamp -= next->ftrace_timestamp; - for (index = next->curr_ret_stack; index >= 0; index--) - next->ret_stack[index].calltime += timestamp; + for (index = next->curr_ret_stack - FGRAPH_RET_INDEX; index >= 0; ) { + ret_stack = RET_STACK(next, index); + ret_stack->calltime += timestamp; + index -= FGRAPH_RET_INDEX; + } } static int ftrace_graph_entry_test(struct ftrace_graph_ent *trace) @@ -521,10 +544,10 @@ void update_function_graph_func(void) ftrace_graph_entry = __ftrace_graph_entry; } -static DEFINE_PER_CPU(struct ftrace_ret_stack *, idle_ret_stack); +static DEFINE_PER_CPU(unsigned long *, idle_ret_stack); static void -graph_init_task(struct task_struct *t, struct ftrace_ret_stack *ret_stack) +graph_init_task(struct task_struct *t, unsigned long *ret_stack) { atomic_set(&t->trace_overrun, 0); t->ftrace_timestamp = 0; @@ -539,7 +562,7 @@ graph_init_task(struct task_struct *t, struct ftrace_ret_stack *ret_stack) */ void ftrace_graph_init_idle_task(struct task_struct *t, int cpu) { - t->curr_ret_stack = -1; + t->curr_ret_stack = 0; t->curr_ret_depth = -1; /* * The idle task has no parent, it either has its own @@ -549,14 +572,11 @@ void ftrace_graph_init_idle_task(struct task_struct *t, int cpu) WARN_ON(t->ret_stack != per_cpu(idle_ret_stack, cpu)); if (ftrace_graph_active) { - struct ftrace_ret_stack *ret_stack; + unsigned long *ret_stack; ret_stack = per_cpu(idle_ret_stack, cpu); if (!ret_stack) { - ret_stack = - kmalloc_array(FTRACE_RETFUNC_DEPTH, - sizeof(struct ftrace_ret_stack), - GFP_KERNEL); + ret_stack = kmalloc(SHADOW_STACK_SIZE, GFP_KERNEL); if (!ret_stack) return; per_cpu(idle_ret_stack, cpu) = ret_stack; @@ -570,15 +590,13 @@ void ftrace_graph_init_task(struct task_struct *t) { /* Make sure we do not use the parent ret_stack */ t->ret_stack = NULL; - t->curr_ret_stack = -1; + t->curr_ret_stack = 0; t->curr_ret_depth = -1; if (ftrace_graph_active) { - struct ftrace_ret_stack *ret_stack; + unsigned long *ret_stack; - ret_stack = kmalloc_array(FTRACE_RETFUNC_DEPTH, - sizeof(struct ftrace_ret_stack), - GFP_KERNEL); + ret_stack = kmalloc(SHADOW_STACK_SIZE, GFP_KERNEL); if (!ret_stack) return; graph_init_task(t, ret_stack); @@ -587,7 +605,7 @@ void ftrace_graph_init_task(struct task_struct *t) void ftrace_graph_exit_task(struct task_struct *t) { - struct ftrace_ret_stack *ret_stack = t->ret_stack; + unsigned long *ret_stack = t->ret_stack; t->ret_stack = NULL; /* NULL must become visible to IRQs before we free it: */ @@ -599,12 +617,10 @@ void ftrace_graph_exit_task(struct task_struct *t) /* Allocate a return stack for each task */ static int start_graph_tracing(void) { - struct ftrace_ret_stack **ret_stack_list; + unsigned long **ret_stack_list; int ret, cpu; - ret_stack_list = kmalloc_array(FTRACE_RETSTACK_ALLOC_SIZE, - sizeof(struct ftrace_ret_stack *), - GFP_KERNEL); + ret_stack_list = kmalloc(SHADOW_STACK_SIZE, GFP_KERNEL); if (!ret_stack_list) return -ENOMEM; From patchwork Sun Nov 5 16:07:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 161660 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp2196351vqu; Sun, 5 Nov 2023 08:08:00 -0800 (PST) X-Google-Smtp-Source: AGHT+IFiSTjjRSwUUtc7QFzBnZ5X9Ab2W/wU9XKRWBMBcMaV48gmpUrCHw+Mbdz93HVivxltStOo X-Received: by 2002:a05:6358:881f:b0:169:81e:b6e0 with SMTP id hv31-20020a056358881f00b00169081eb6e0mr34467704rwb.24.1699200479942; Sun, 05 Nov 2023 08:07:59 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699200479; cv=none; d=google.com; s=arc-20160816; b=IpK00y+QgzrsR3YsZn81UIH5FkAUE2/haM5gxTiXqDOkV+1eZjoOClLNXF/c4SXLh4 dUPRim8TToKBg3xUjxThUTy2SjgjmR2R5Gq4n6hY73YAXgfSfc8RWb5JBkWIgH5ScskB a4Qj2ttnVV7bnjmgP76Kjp6WlE6BDZQ1xvf6Aac3Wxz3szbKbG73b7QnFKWrlXr8BcUc w1GLW3dh9Ur3hqNK/Miu4CvkbogarNjSRaICxM+lhZp0k3ikfTQ6GwyBoGrPEK79vn/u Lf8dxsm3zgWoegeACQIyqDTJVcuiSCHsduq1Mg6nwN7cpOSMdMIOtHUoYzISCSe4j0mN BayQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=OESyYZVxbmArs5V4zMGrIL0/uPgjACK9BAVI1aKMbU0=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=PYB67YagvjiC6L5HXZuQduJ9qMIQPOEHKR/f5ZbDSYHv80ZOWhnTJowo3NEyYgQKFh Oj5i+m4XD6vPqAiRXhH4kowAf6iDRkLy5CjsKfBqSJKrkgdeu+QFVGfLslEYhp5nPG35 LoE7FWQnE0BYnks74MH2lgnX8WF5zdejdaAHzrWcSAzSAYQTi4WjQU1BGbTS+Hd+9jeh Jj3kXx1kWPhLhae2XI6+Aqoi2YOTkwufHrtjYk1R+RTHRG3Mqsbt1jWyFX6448WrFdZJ yg5G9XcROdKyyjqBAQxUwk+nxNB9sqq6rpQ2H8YokMZPpR2giH7rxWShYS6LSleWXJzn 0Mbw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=dyF6zqhC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id q184-20020a632ac1000000b005b7dd1e334csi6052344pgq.601.2023.11.05.08.07.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Nov 2023 08:07:59 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=dyF6zqhC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 1CFF180545B5; Sun, 5 Nov 2023 08:07:50 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229637AbjKEQHU (ORCPT + 34 others); Sun, 5 Nov 2023 11:07:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53886 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229698AbjKEQHQ (ORCPT ); Sun, 5 Nov 2023 11:07:16 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A27131B2 for ; Sun, 5 Nov 2023 08:07:13 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 263A0C433C8; Sun, 5 Nov 2023 16:07:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1699200433; bh=BJTb9S/48BMXgKg9JdebJFjIf4uCQaHDkO4DGbu+/d0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dyF6zqhCWxd07sdjAl8Mrzwr5pVkixS3gZKjnp0JhrnrLFhKeNzHfqhRUBW2qRj4D la81PR8h5F2034JRrNmOstiPEzOMAKF5LExKlV3vlJE2e1XFjzt6aP/4vcQSdunlm9 yxX6zJ2ylc27phnsYa7WwfcJfR3fsb58GrB8rJRnrtjRhGFH3YxJrdxkD0mKvxJXSc US3GXWMKaYMAClk0Sfy010l4pMEXaoOnyo2hUvrNlzBzZ/v+DiBEsPEKpC2blHq1hA 1uc5iuL3zDROJ4+WWfk8ccymhSQgvX+AzZaGHxWyzJZidF+0auyTegwnwwfaevPPBg KF3Fj186CGthw== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [RFC PATCH 03/32] fgraph: Use BUILD_BUG_ON() to make sure we have structures divisible by long Date: Mon, 6 Nov 2023 01:07:06 +0900 Message-Id: <169920042620.482486.2035922638334412424.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169920038849.482486.15796387219966662967.stgit@devnote2> References: <169920038849.482486.15796387219966662967.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Sun, 05 Nov 2023 08:07:50 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781740842261569774 X-GMAIL-MSGID: 1781740842261569774 From: Steven Rostedt (VMware) Instead of using "ALIGN()", use BUILD_BUG_ON() as the structures should always be divisible by sizeof(long). Link: http://lkml.kernel.org/r/20190524111144.GI2589@hirez.programming.kicks-ass.net Suggested-by: Peter Zijlstra Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- kernel/trace/fgraph.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index 30edeb6d4aa9..837daf929d2a 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -26,10 +26,9 @@ #endif #define FGRAPH_RET_SIZE sizeof(struct ftrace_ret_stack) -#define FGRAPH_RET_INDEX (ALIGN(FGRAPH_RET_SIZE, sizeof(long)) / sizeof(long)) +#define FGRAPH_RET_INDEX (FGRAPH_RET_SIZE / sizeof(long)) #define SHADOW_STACK_SIZE (PAGE_SIZE) -#define SHADOW_STACK_INDEX \ - (ALIGN(SHADOW_STACK_SIZE, sizeof(long)) / sizeof(long)) +#define SHADOW_STACK_INDEX (SHADOW_STACK_SIZE / sizeof(long)) /* Leave on a buffer at the end */ #define SHADOW_STACK_MAX_INDEX (SHADOW_STACK_INDEX - FGRAPH_RET_INDEX) @@ -91,6 +90,8 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func, if (!current->ret_stack) return -EBUSY; + BUILD_BUG_ON(SHADOW_STACK_SIZE % sizeof(long)); + /* * We must make sure the ret_stack is tested before we read * anything else. @@ -325,6 +326,8 @@ ftrace_graph_get_ret_stack(struct task_struct *task, int idx) { int index = task->curr_ret_stack; + BUILD_BUG_ON(FGRAPH_RET_SIZE % sizeof(long)); + index -= FGRAPH_RET_INDEX * (idx + 1); if (index < 0) return NULL; From patchwork Sun Nov 5 16:07:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 161663 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp2196584vqu; Sun, 5 Nov 2023 08:08:24 -0800 (PST) X-Google-Smtp-Source: AGHT+IEbee6qtsEHVIMOYA/1U9AKsJMOFXN2W9JuoQOt5l3vcCNCOtvag09PY9rw8njGPp7ozG8F X-Received: by 2002:a05:6a00:414f:b0:68f:a92a:8509 with SMTP id bv15-20020a056a00414f00b0068fa92a8509mr13044172pfb.7.1699200504632; Sun, 05 Nov 2023 08:08:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699200504; cv=none; d=google.com; s=arc-20160816; b=UquBUjnJt4smyNURLrfevB4eBBUhmEP6VXUKXu8AxRhCaFvEA6PfrVjIImb3QuFOm/ oJ1XyFOruiOoekG4f0WDb36wIBuCHO7Fyu7pXbCKWRKwqgA+Z+L4mgSQ89CD90dNC8EE f+sM5BqSU6sEbZXIefaiKpo4S91ltOyxQG1uXgZZfkE2N/djo38CuY2xx9uVL8a7mUHN tz1mJLdcsTJ4RmjOX3QBg3hGWNBNcZ0khE40RyYYGTGInQVgFOnTZmjz0MACDS2ePT9X qlOuuEo8iKhv4l4fB813YYAtQbLOGwFhoAsNp304epQwmoMweHqFGjhElKgC2VEakyOt 0ENA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=sRKxFugRG7KG7ocgZcaLlnh9N+MT1pbRHzFIwe8obIg=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=deCM+Q2Mx3w9pHXZAj76Z/7LV636t/u+frVAnqE7KjhmjjHqRn13ErDirmUCSnChpX 3sIVAtXTszN5lhqi4iNjQjQE2JDiw1phVS6UFZNTnmjYOSjfOUiK1YvxtVSrPpplZEct FiX6CrKVk3CXPYYm5iriCbxy85UUUCdFCFPxZ1caONcF6nKHfUkBWWi36buis+ammyjs PXPCPORmBvZhVZM75H2CkSE9Rd1WeXYMG1K16ZkdQ2FPQm1Ifg/ITaMszUY5s9yV/jx2 Q8eX+K+24BkvnV9rqFWqkoRg39G1c3hxJQbRmJbujJ8WLbhHadNA679sIdJddCcpga59 M0tw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=EKUOXybI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from agentk.vger.email (agentk.vger.email. [23.128.96.32]) by mx.google.com with ESMTPS id z2-20020a056a001d8200b0069342eade87si6420858pfw.18.2023.11.05.08.08.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Nov 2023 08:08:24 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=EKUOXybI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id BC8148057B0F; Sun, 5 Nov 2023 08:08:14 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229468AbjKEQHe (ORCPT + 34 others); Sun, 5 Nov 2023 11:07:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40684 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229731AbjKEQH3 (ORCPT ); Sun, 5 Nov 2023 11:07:29 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D55B1D57 for ; Sun, 5 Nov 2023 08:07:25 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 847C9C433C9; Sun, 5 Nov 2023 16:07:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1699200445; bh=B0/aSI1KB5QEqFWG8e41IiV7Rc9rgkz1NK6k4ZuIzko=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=EKUOXybIfS8LWt/VXAFtqOUAt2pZ6p0P9DovhIIDcLgYfNi7T2pZ1Z/t8/TkiDB0s 9n1B2bwzpm3U6/Rq0Y81KaMmwARZHmqckvI/0bhjX/8tQBCx5V8eSviP35qJ+lMCj2 qVJO/vCprEmOqlYEXkwFo3PBpE2towK8CL7BEe0f9P52mrrrKkcT1fxro4Fow4cB7j 4KVS01grfMjby3aUvVK6AprZw8E+M5v8WjDuUZmTUNHk77RlFqo/QuRyGG3TNljJPq 0hzCc5HEGce3OOqCOxYoPdPfwPuuKUhlFuKCh0ycIadfRTMk42zHkNdTIV9GNr/qsm IrKoC3T1QDE0Q== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [RFC PATCH 04/32] function_graph: Add an array structure that will allow multiple callbacks Date: Mon, 6 Nov 2023 01:07:18 +0900 Message-Id: <169920043839.482486.410237912533776158.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169920038849.482486.15796387219966662967.stgit@devnote2> References: <169920038849.482486.15796387219966662967.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Sun, 05 Nov 2023 08:08:14 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781740868488631704 X-GMAIL-MSGID: 1781740868488631704 From: Steven Rostedt (VMware) Add an array structure that will eventually allow the function graph tracer to have up to 16 simultaneous callbacks attached. It's an array of 16 fgraph_ops pointers, that is assigned when one is registered. On entry of a function the entry of the first item in the array is called, and if it returns zero, then the callback returns non zero if it wants the return callback to be called on exit of the function. The array will simplify the process of having more than one callback attached to the same function, as its index into the array can be stored on the shadow stack. We need to only save the index, because this will allow the fgraph_ops to be freed before the function returns (which may happen if the function call schedule for a long time). Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- kernel/trace/fgraph.c | 115 +++++++++++++++++++++++++++++++++++-------------- 1 file changed, 82 insertions(+), 33 deletions(-) diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index 837daf929d2a..b34ab7ecd7af 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -39,6 +39,11 @@ DEFINE_STATIC_KEY_FALSE(kill_ftrace_graph); int ftrace_graph_active; +static int fgraph_array_cnt; +#define FGRAPH_ARRAY_SIZE 16 + +static struct fgraph_ops *fgraph_array[FGRAPH_ARRAY_SIZE]; + /* Both enabled by default (can be cleared by function_graph tracer flags */ static bool fgraph_sleep_time = true; @@ -62,6 +67,20 @@ int __weak ftrace_disable_ftrace_graph_caller(void) } #endif +int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace) +{ + return 0; +} + +static void ftrace_graph_ret_stub(struct ftrace_graph_ret *trace) +{ +} + +static struct fgraph_ops fgraph_stub = { + .entryfunc = ftrace_graph_entry_stub, + .retfunc = ftrace_graph_ret_stub, +}; + /** * ftrace_graph_stop - set to permanently disable function graph tracing * @@ -159,7 +178,7 @@ int function_graph_enter(unsigned long ret, unsigned long func, goto out; /* Only trace if the calling function expects to */ - if (!ftrace_graph_entry(&trace)) + if (!fgraph_array[0]->entryfunc(&trace)) goto out_ret; return 0; @@ -274,7 +293,7 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs trace.retval = fgraph_ret_regs_return_value(ret_regs); #endif trace.rettime = trace_clock_local(); - ftrace_graph_return(&trace); + fgraph_array[0]->retfunc(&trace); /* * The ftrace_graph_return() may still access the current * ret_stack structure, we need to make sure the update of @@ -410,11 +429,6 @@ void ftrace_graph_sleep_time_control(bool enable) fgraph_sleep_time = enable; } -int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace) -{ - return 0; -} - /* * Simply points to ftrace_stub, but with the proper protocol. * Defined by the linker script in linux/vmlinux.lds.h @@ -652,37 +666,55 @@ static int start_graph_tracing(void) int register_ftrace_graph(struct fgraph_ops *gops) { int ret = 0; + int i; mutex_lock(&ftrace_lock); - /* we currently allow only one tracer registered at a time */ - if (ftrace_graph_active) { + if (!fgraph_array[0]) { + /* The array must always have real data on it */ + for (i = 0; i < FGRAPH_ARRAY_SIZE; i++) { + fgraph_array[i] = &fgraph_stub; + } + } + + /* Look for an available spot */ + for (i = 0; i < FGRAPH_ARRAY_SIZE; i++) { + if (fgraph_array[i] == &fgraph_stub) + break; + } + if (i >= FGRAPH_ARRAY_SIZE) { ret = -EBUSY; goto out; } - register_pm_notifier(&ftrace_suspend_notifier); + fgraph_array[i] = gops; + if (i + 1 > fgraph_array_cnt) + fgraph_array_cnt = i + 1; ftrace_graph_active++; - ret = start_graph_tracing(); - if (ret) { - ftrace_graph_active--; - goto out; - } - ftrace_graph_return = gops->retfunc; + if (ftrace_graph_active == 1) { + register_pm_notifier(&ftrace_suspend_notifier); + ret = start_graph_tracing(); + if (ret) { + ftrace_graph_active--; + goto out; + } - /* - * Update the indirect function to the entryfunc, and the - * function that gets called to the entry_test first. Then - * call the update fgraph entry function to determine if - * the entryfunc should be called directly or not. - */ - __ftrace_graph_entry = gops->entryfunc; - ftrace_graph_entry = ftrace_graph_entry_test; - update_function_graph_func(); + ftrace_graph_return = gops->retfunc; + + /* + * Update the indirect function to the entryfunc, and the + * function that gets called to the entry_test first. Then + * call the update fgraph entry function to determine if + * the entryfunc should be called directly or not. + */ + __ftrace_graph_entry = gops->entryfunc; + ftrace_graph_entry = ftrace_graph_entry_test; + update_function_graph_func(); - ret = ftrace_startup(&graph_ops, FTRACE_START_FUNC_RET); + ret = ftrace_startup(&graph_ops, FTRACE_START_FUNC_RET); + } out: mutex_unlock(&ftrace_lock); return ret; @@ -690,19 +722,36 @@ int register_ftrace_graph(struct fgraph_ops *gops) void unregister_ftrace_graph(struct fgraph_ops *gops) { + int i; + mutex_lock(&ftrace_lock); if (unlikely(!ftrace_graph_active)) goto out; - ftrace_graph_active--; - ftrace_graph_return = ftrace_stub_graph; - ftrace_graph_entry = ftrace_graph_entry_stub; - __ftrace_graph_entry = ftrace_graph_entry_stub; - ftrace_shutdown(&graph_ops, FTRACE_STOP_FUNC_RET); - unregister_pm_notifier(&ftrace_suspend_notifier); - unregister_trace_sched_switch(ftrace_graph_probe_sched_switch, NULL); + for (i = 0; i < fgraph_array_cnt; i++) + if (gops == fgraph_array[i]) + break; + if (i >= fgraph_array_cnt) + goto out; + + fgraph_array[i] = &fgraph_stub; + if (i + 1 == fgraph_array_cnt) { + for (; i >= 0; i--) + if (fgraph_array[i] != &fgraph_stub) + break; + fgraph_array_cnt = i + 1; + } + ftrace_graph_active--; + if (!ftrace_graph_active) { + ftrace_graph_return = ftrace_stub_graph; + ftrace_graph_entry = ftrace_graph_entry_stub; + __ftrace_graph_entry = ftrace_graph_entry_stub; + ftrace_shutdown(&graph_ops, FTRACE_STOP_FUNC_RET); + unregister_pm_notifier(&ftrace_suspend_notifier); + unregister_trace_sched_switch(ftrace_graph_probe_sched_switch, NULL); + } out: mutex_unlock(&ftrace_lock); } From patchwork Sun Nov 5 16:07:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 161661 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp2196360vqu; Sun, 5 Nov 2023 08:08:00 -0800 (PST) X-Google-Smtp-Source: AGHT+IEjE57b0t/o8XU5PaAeN6gUg4vbqGAD5nM1eMru9D44DqcIVhYeudCDM7VJxqUHBzCehV66 X-Received: by 2002:a05:6a20:728f:b0:177:690a:7978 with SMTP id o15-20020a056a20728f00b00177690a7978mr33089070pzk.33.1699200480573; Sun, 05 Nov 2023 08:08:00 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699200480; cv=none; d=google.com; s=arc-20160816; b=0fYSm0TBBzeT5Hf2R8bUnugND3pCSsZ/Bdzls+CDpufx6/5+IDRjg63ucuOYhR2Sr0 rv/TF+mt/JHhXY06hGhvRUi6sb3GHvzUGYxXorAQPllXLXJhTy8zHAVYLJSOHFpsKFBz /mVM8jLCnUvQIQf7zi2evyV/d0XAzY6cmJTcN91w8PcnwxeSLjZiqOynOWmT2fkl5VWy awaxEdUHDzNVHuuXUDyMRhxl+Dm+/ekFY7lwbqv2g0bscD9W7nmMI5jxlJx1K8ViVK7G DKseeawkgV+80Zoh0F879trn4usv/22n712GZKT6kiEWlVGl7BRNMZ/vcBbDCxSQrYHP 85sA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=3Xprvm4b37zyI6ZHKREXghfIBJI3kphQMOXsb6udz34=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=VVze2iwZFoSv71aEjvv4TeQpJe28C3BE/ymfjoYsLZgdmGsWAuSZ/sFDCWDe2+DTK5 C5Vz2POHsMe6nbDLQF/AIAznLTPZP8IJuNnc6RRs+RTAApn8HFcwdYqASUKoVZ9xXkpG +odIgteg04aQ0PnUlVaGkCCGFkizG9dmTJu/lJPidS6kLOsrtU3DGkIetJ93ppKzbFFL K2zCKbFJKM689DtDSMO/Cqo0Sfrcer2fjMyYDB7SfdAAIfI8lt5VNfsrAKkYtM0Y7QI1 PlnP1w2iaRHJgPxD+g73MQRpYH2zE4/Yo4+xT7o/z0GKX3xTPE6nnANK8TXYczLnVlYx I3rg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=pFeQxqWF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id pj6-20020a17090b4f4600b0027d4684804dsi6415235pjb.122.2023.11.05.08.08.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Nov 2023 08:08:00 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=pFeQxqWF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id C1BF380B1D34; Sun, 5 Nov 2023 08:07:59 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229750AbjKEQHs (ORCPT + 34 others); Sun, 5 Nov 2023 11:07:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40684 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229709AbjKEQHn (ORCPT ); Sun, 5 Nov 2023 11:07:43 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 30DCE191 for ; Sun, 5 Nov 2023 08:07:38 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9908BC433C8; Sun, 5 Nov 2023 16:07:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1699200457; bh=jOqsduSIIkCplloCCsirmYcJbL1ctguhDbc6R81kjKg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pFeQxqWFnh6amOcIYBWQdXV1YmA2Y9fvLJ90eMAoQZSOvjhBesiJlI1mMtq1HXLn+ 69sYpVJPQcWtugMLcdrjiVD+/0aAHqmWYoqwSZNAesHfv1uGaWgQVU4k0eIZo2x5Lg WMK5OoO54WYlo96L+7HLgzsVCOrhNRgREBjCxzigiq9kaktZGmP1yPjrQU90YWW/T3 DIg6GG53i7AMflYZ22y2MzCY4gKCjV5Ci+njCe2weBy31W6v266aYu4ZJ363BnkpnW 4VepMUtgy53bPB6eo8xgO0fFjiFlLyEz5/80DFqFAZym820dP4NZeECLsgeLhhJsui o0eyZrz19EGxg== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [RFC PATCH 05/32] function_graph: Allow multiple users to attach to function graph Date: Mon, 6 Nov 2023 01:07:30 +0900 Message-Id: <169920045057.482486.15121636569414129713.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169920038849.482486.15796387219966662967.stgit@devnote2> References: <169920038849.482486.15796387219966662967.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-5.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Sun, 05 Nov 2023 08:07:59 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781740843063804854 X-GMAIL-MSGID: 1781740843063804854 From: Steven Rostedt (VMware) Allow for multiple users to attach to function graph tracer at the same time. Only 16 simultaneous users can attach to the tracer. This is because there's an array that stores the pointers to the attached fgraph_ops. When a a function being traced is entered, each of the ftrace_ops entryfunc is called and if it returns non zero, its index into the array will be added to the shadow stack. On exit of the function being traced, the shadow stack will contain the indexes of the ftrace_ops on the array that want their retfunc to be called. Because a function may sleep for a long time (if a task sleeps itself), the return of the function may be literally days later. If the ftrace_ops is removed, its place on the array is replaced with a ftrace_ops that contains the stub functions and that will be called when the function finally returns. If another ftrace_ops is added that happens to get the same index into the array, its return function may be called. But that's actually the way things current work with the old function graph tracer. If one tracer is removed and another is added, the new one will get the return calls of the function traced by the previous one, thus this is not a regression. This can be fixed by adding a counter to each time the array item is updated and save that on the shadow stack as well, such that it wont be called if the index saved does not match the index on the array. Note, being able to filter functions when both are called is not completely handled yet, but that shouldn't be too hard to manage. Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- kernel/trace/fgraph.c | 332 +++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 280 insertions(+), 52 deletions(-) diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index b34ab7ecd7af..58750a395313 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -27,23 +27,144 @@ #define FGRAPH_RET_SIZE sizeof(struct ftrace_ret_stack) #define FGRAPH_RET_INDEX (FGRAPH_RET_SIZE / sizeof(long)) + +/* + * On entry to a function (via function_graph_enter()), a new ftrace_ret_stack + * is allocated on the task's ret_stack, then each fgraph_ops on the + * fgraph_array[]'s entryfunc is called and if that returns non-zero, the + * index into the fgraph_array[] for that fgraph_ops is added to the ret_stack. + * As the associated ftrace_ret_stack saved for those fgraph_ops needs to + * be found, the index to it is also added to the ret_stack along with the + * index of the fgraph_array[] to each fgraph_ops that needs their retfunc + * called. + * + * The top of the ret_stack (when not empty) will always have a reference + * to the last ftrace_ret_stack saved. All references to the + * ftrace_ret_stack has the format of: + * + * bits: 0 - 13 Index in words from the previous ftrace_ret_stack + * bits: 14 - 15 Type of storage + * 0 - reserved + * 1 - fgraph_array index + * For fgraph_array_index: + * bits: 16 - 23 The fgraph_ops fgraph_array index + * + * That is, at the end of function_graph_enter, if the first and forth + * fgraph_ops on the fgraph_array[] (index 0 and 3) needs their retfunc called + * on the return of the function being traced, this is what will be on the + * task's shadow ret_stack: (the stack grows upward) + * + * | | <- task->curr_ret_stack + * +----------------------------------+ + * | (3 << FGRAPH_ARRAY_SHIFT)|(2) | ( 3 for index of fourth fgraph_ops) + * +----------------------------------+ + * | (0 << FGRAPH_ARRAY_SHIFT)|(1) | ( 0 for index of first fgraph_ops) + * +----------------------------------+ + * | struct ftrace_ret_stack | + * | (stores the saved ret pointer) | + * +----------------------------------+ + * | (X) | (N) | ( N words away from previous ret_stack) + * | | + * + * If a backtrace is required, and the real return pointer needs to be + * fetched, then it looks at the task's curr_ret_stack index, if it + * is greater than zero, it would subtact one, and then mask the value + * on the ret_stack by FGRAPH_RET_INDEX_MASK and subtract FGRAPH_RET_INDEX + * from that, to get the index of the ftrace_ret_stack structure stored + * on the shadow stack. + */ + +#define FGRAPH_RET_INDEX_SIZE 14 +#define FGRAPH_RET_INDEX_MASK ((1 << FGRAPH_RET_INDEX_SIZE) - 1) + + +#define FGRAPH_TYPE_SIZE 2 +#define FGRAPH_TYPE_MASK ((1 << FGRAPH_TYPE_SIZE) - 1) +#define FGRAPH_TYPE_SHIFT FGRAPH_RET_INDEX_SIZE + +enum { + FGRAPH_TYPE_RESERVED = 0, + FGRAPH_TYPE_ARRAY = 1, +}; + +#define FGRAPH_ARRAY_SIZE 16 +#define FGRAPH_ARRAY_MASK ((1 << FGRAPH_ARRAY_SIZE) - 1) +#define FGRAPH_ARRAY_SHIFT (FGRAPH_TYPE_SHIFT + FGRAPH_TYPE_SIZE) + +/* Currently the max stack index can't be more than register callers */ +#define FGRAPH_MAX_INDEX FGRAPH_ARRAY_SIZE + +#define FGRAPH_FRAME_SIZE (FGRAPH_RET_SIZE + FGRAPH_ARRAY_SIZE * (sizeof(long))) +#define FGRAPH_FRAME_INDEX (ALIGN(FGRAPH_FRAME_SIZE, \ + sizeof(long)) / sizeof(long)) #define SHADOW_STACK_SIZE (PAGE_SIZE) #define SHADOW_STACK_INDEX (SHADOW_STACK_SIZE / sizeof(long)) /* Leave on a buffer at the end */ -#define SHADOW_STACK_MAX_INDEX (SHADOW_STACK_INDEX - FGRAPH_RET_INDEX) +#define SHADOW_STACK_MAX_INDEX (SHADOW_STACK_INDEX - (FGRAPH_RET_INDEX + 1)) #define RET_STACK(t, index) ((struct ftrace_ret_stack *)(&(t)->ret_stack[index])) -#define RET_STACK_INC(c) ({ c += FGRAPH_RET_INDEX; }) -#define RET_STACK_DEC(c) ({ c -= FGRAPH_RET_INDEX; }) DEFINE_STATIC_KEY_FALSE(kill_ftrace_graph); int ftrace_graph_active; static int fgraph_array_cnt; -#define FGRAPH_ARRAY_SIZE 16 static struct fgraph_ops *fgraph_array[FGRAPH_ARRAY_SIZE]; +static inline int get_ret_stack_index(struct task_struct *t, int offset) +{ + return current->ret_stack[offset] & FGRAPH_RET_INDEX_MASK; +} + +static inline int get_fgraph_type(struct task_struct *t, int offset) +{ + return (current->ret_stack[offset] >> FGRAPH_TYPE_SHIFT) & + FGRAPH_TYPE_MASK; +} + +static inline int get_fgraph_array(struct task_struct *t, int offset) +{ + return (current->ret_stack[offset] >> FGRAPH_ARRAY_SHIFT) & + FGRAPH_ARRAY_MASK; +} + +/* + * @offset: The index into @t->ret_stack to find the ret_stack entry + * @index: Where to place the index into @t->ret_stack of that entry + * + * Calling this with: + * + * offset = task->curr_ret_stack; + * do { + * ret_stack = get_ret_stack(task, offset, &offset); + * } while (ret_stack); + * + * Will iterate through all the ret_stack entries from curr_ret_stack + * down to the first one. + */ +static inline struct ftrace_ret_stack * +get_ret_stack(struct task_struct *t, int offset, int *index) +{ + int idx; + + BUILD_BUG_ON(FGRAPH_RET_SIZE % sizeof(long)); + + if (offset <= 0) + return NULL; + + idx = get_ret_stack_index(t, offset - 1); + + if (idx <= 0 || idx > FGRAPH_MAX_INDEX) + return NULL; + + offset -= idx + FGRAPH_RET_INDEX; + if (offset < 0) + return NULL; + + *index = offset; + return RET_STACK(t, offset); +} + /* Both enabled by default (can be cleared by function_graph tracer flags */ static bool fgraph_sleep_time = true; @@ -126,9 +247,34 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func, calltime = trace_clock_local(); index = current->curr_ret_stack; - RET_STACK_INC(current->curr_ret_stack); + /* ret offset = 1 ; type = reserved */ + current->ret_stack[index + FGRAPH_RET_INDEX] = 1; ret_stack = RET_STACK(current, index); + ret_stack->ret = ret; + /* + * The undwinders expect curr_ret_stack to point to either zero + * or an index where to find the next ret_stack. Even though the + * ret stack might be bogus, we want to write the ret and the + * index to find the ret_stack before we increment the stack point. + * If an interrupt comes in now before we increment the curr_ret_stack + * it may blow away what we wrote. But that's fine, because the + * index will still be correct (even though the ret wont be). + * What we worry about is the index being correct after we increment + * the curr_ret_stack and before we update that index, as if an + * interrupt comes in and does an unwind stack dump, it will need + * at least a correct index! + */ + barrier(); + current->curr_ret_stack += FGRAPH_RET_INDEX + 1; + /* + * This next barrier is to ensure that an interrupt coming in + * will not corrupt what we are about to write. + */ barrier(); + + /* Still keep it reserved even if an interrupt came in */ + current->ret_stack[index + FGRAPH_RET_INDEX] = 1; + ret_stack->ret = ret; ret_stack->func = func; ret_stack->calltime = calltime; @@ -159,6 +305,12 @@ int function_graph_enter(unsigned long ret, unsigned long func, unsigned long frame_pointer, unsigned long *retp) { struct ftrace_graph_ent trace; + int offset; + int start; + int type; + int val; + int cnt = 0; + int i; #ifndef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS /* @@ -177,38 +329,87 @@ int function_graph_enter(unsigned long ret, unsigned long func, if (ftrace_push_return_trace(ret, func, frame_pointer, retp)) goto out; - /* Only trace if the calling function expects to */ - if (!fgraph_array[0]->entryfunc(&trace)) + /* Use start for the distance to ret_stack (skipping over reserve) */ + start = offset = current->curr_ret_stack - 2; + + for (i = 0; i < fgraph_array_cnt; i++) { + struct fgraph_ops *gops = fgraph_array[i]; + + if (gops == &fgraph_stub) + continue; + + if ((offset == start) && + (current->curr_ret_stack >= SHADOW_STACK_INDEX - 1)) { + atomic_inc(¤t->trace_overrun); + break; + } + if (fgraph_array[i]->entryfunc(&trace)) { + offset = current->curr_ret_stack; + /* Check the top level stored word */ + type = get_fgraph_type(current, offset - 1); + + val = (i << FGRAPH_ARRAY_SHIFT) | + (FGRAPH_TYPE_ARRAY << FGRAPH_TYPE_SHIFT) | + ((offset - start) - 1); + + /* We can reuse the top word if it is reserved */ + if (type == FGRAPH_TYPE_RESERVED) { + current->ret_stack[offset - 1] = val; + cnt++; + continue; + } + val++; + + current->ret_stack[offset] = val; + /* + * Write the value before we increment, so that + * if an interrupt comes in after we increment + * it will still see the value and skip over + * this. + */ + barrier(); + current->curr_ret_stack++; + /* + * Have to write again, in case an interrupt + * came in before the increment and after we + * wrote the value. + */ + barrier(); + current->ret_stack[offset] = val; + cnt++; + } + } + + if (!cnt) goto out_ret; return 0; out_ret: - RET_STACK_DEC(current->curr_ret_stack); + current->curr_ret_stack -= FGRAPH_RET_INDEX + 1; out: current->curr_ret_depth--; return -EBUSY; } /* Retrieve a function return address to the trace stack on thread info.*/ -static void +static struct ftrace_ret_stack * ftrace_pop_return_trace(struct ftrace_graph_ret *trace, unsigned long *ret, unsigned long frame_pointer) { struct ftrace_ret_stack *ret_stack; int index; - index = current->curr_ret_stack; - RET_STACK_DEC(index); + ret_stack = get_ret_stack(current, current->curr_ret_stack, &index); - if (unlikely(index < 0 || index > SHADOW_STACK_MAX_INDEX)) { + if (unlikely(!ret_stack)) { ftrace_graph_stop(); - WARN_ON(1); + WARN(1, "Bad function graph ret_stack pointer: %d", + current->curr_ret_stack); /* Might as well panic, otherwise we have no where to go */ *ret = (unsigned long)panic; - return; + return NULL; } - ret_stack = RET_STACK(current, index); #ifdef HAVE_FUNCTION_GRAPH_FP_TEST /* * The arch may choose to record the frame pointer used @@ -228,12 +429,12 @@ ftrace_pop_return_trace(struct ftrace_graph_ret *trace, unsigned long *ret, ftrace_graph_stop(); WARN(1, "Bad frame pointer: expected %lx, received %lx\n" " from func %ps return to %lx\n", - current->ret_stack[index].fp, + ret_stack->fp, frame_pointer, (void *)ret_stack->func, ret_stack->ret); *ret = (unsigned long)panic; - return; + return NULL; } #endif @@ -241,13 +442,15 @@ ftrace_pop_return_trace(struct ftrace_graph_ret *trace, unsigned long *ret, trace->func = ret_stack->func; trace->calltime = ret_stack->calltime; trace->overrun = atomic_read(¤t->trace_overrun); - trace->depth = current->curr_ret_depth--; + trace->depth = current->curr_ret_depth; /* * We still want to trace interrupts coming in if * max_depth is set to 1. Make sure the decrement is * seen before ftrace_graph_return. */ barrier(); + + return ret_stack; } /* @@ -285,30 +488,47 @@ struct fgraph_ret_regs; static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs, unsigned long frame_pointer) { + struct ftrace_ret_stack *ret_stack; struct ftrace_graph_ret trace; unsigned long ret; + int offset; + int index; + int idx; + int i; + + ret_stack = ftrace_pop_return_trace(&trace, &ret, frame_pointer); + + if (unlikely(!ret)) { + ftrace_graph_stop(); + WARN_ON(1); + /* Might as well panic. What else to do? */ + return (unsigned long)panic; + } - ftrace_pop_return_trace(&trace, &ret, frame_pointer); + trace.rettime = trace_clock_local(); #ifdef CONFIG_FUNCTION_GRAPH_RETVAL trace.retval = fgraph_ret_regs_return_value(ret_regs); #endif - trace.rettime = trace_clock_local(); - fgraph_array[0]->retfunc(&trace); + + offset = current->curr_ret_stack - 1; + index = get_ret_stack_index(current, offset); + + /* index has to be at least one! Optimize for it */ + i = 0; + do { + idx = get_fgraph_array(current, offset - i); + fgraph_array[idx]->retfunc(&trace); + i++; + } while (i < index); + /* * The ftrace_graph_return() may still access the current * ret_stack structure, we need to make sure the update of * curr_ret_stack is after that. */ barrier(); - RET_STACK_DEC(current->curr_ret_stack); - - if (unlikely(!ret)) { - ftrace_graph_stop(); - WARN_ON(1); - /* Might as well panic. What else to do? */ - ret = (unsigned long)panic; - } - + current->curr_ret_stack -= index + FGRAPH_RET_INDEX; + current->curr_ret_depth--; return ret; } @@ -343,15 +563,17 @@ unsigned long ftrace_return_to_handler(unsigned long frame_pointer) struct ftrace_ret_stack * ftrace_graph_get_ret_stack(struct task_struct *task, int idx) { + struct ftrace_ret_stack *ret_stack = NULL; int index = task->curr_ret_stack; - BUILD_BUG_ON(FGRAPH_RET_SIZE % sizeof(long)); - - index -= FGRAPH_RET_INDEX * (idx + 1); if (index < 0) return NULL; - return RET_STACK(task, index); + do { + ret_stack = get_ret_stack(task, index, &index); + } while (ret_stack && --idx >= 0); + + return ret_stack; } /** @@ -374,16 +596,15 @@ unsigned long ftrace_graph_ret_addr(struct task_struct *task, int *idx, unsigned long ret, unsigned long *retp) { struct ftrace_ret_stack *ret_stack; - int index = task->curr_ret_stack; - int i; + int i = task->curr_ret_stack; if (ret != (unsigned long)dereference_kernel_function_descriptor(return_to_handler)) return ret; - RET_STACK_DEC(index); - - for (i = index; i >= 0; RET_STACK_DEC(i)) { - ret_stack = RET_STACK(task, i); + while (i > 0) { + ret_stack = get_ret_stack(current, i, &i); + if (!ret_stack) + break; if (ret_stack->retp == retp) return ret_stack->ret; } @@ -394,21 +615,26 @@ unsigned long ftrace_graph_ret_addr(struct task_struct *task, int *idx, unsigned long ftrace_graph_ret_addr(struct task_struct *task, int *idx, unsigned long ret, unsigned long *retp) { - int task_idx; + struct ftrace_ret_stack *ret_stack; + int task_idx = task->curr_ret_stack; + int i; if (ret != (unsigned long)dereference_kernel_function_descriptor(return_to_handler)) return ret; - task_idx = task->curr_ret_stack; - RET_STACK_DEC(task_idx); - - if (!task->ret_stack || task_idx < *idx) + if (!idx) return ret; - task_idx -= *idx; - RET_STACK_INC(*idx); + i = *idx; + do { + ret_stack = get_ret_stack(task, task_idx, &task_idx); + i--; + } while (i >= 0 && ret_stack); + + if (ret_stack) + return ret_stack->ret; - return RET_STACK(task, task_idx); + return ret; } #endif /* HAVE_FUNCTION_GRAPH_RET_ADDR_PTR */ @@ -514,10 +740,10 @@ ftrace_graph_probe_sched_switch(void *ignore, bool preempt, */ timestamp -= next->ftrace_timestamp; - for (index = next->curr_ret_stack - FGRAPH_RET_INDEX; index >= 0; ) { - ret_stack = RET_STACK(next, index); - ret_stack->calltime += timestamp; - index -= FGRAPH_RET_INDEX; + for (index = next->curr_ret_stack; index > 0; ) { + ret_stack = get_ret_stack(next, index, &index); + if (ret_stack) + ret_stack->calltime += timestamp; } } @@ -568,6 +794,8 @@ graph_init_task(struct task_struct *t, unsigned long *ret_stack) { atomic_set(&t->trace_overrun, 0); t->ftrace_timestamp = 0; + t->curr_ret_stack = 0; + t->curr_ret_depth = -1; /* make curr_ret_stack visible before we add the ret_stack */ smp_wmb(); t->ret_stack = ret_stack; From patchwork Sun Nov 5 16:07:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 161662 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp2196385vqu; Sun, 5 Nov 2023 08:08:03 -0800 (PST) X-Google-Smtp-Source: AGHT+IGI2zSEk76ze7Bjgex+SrUlmBzGOiN3cS33YlAqNdopTYRKoyRqz0XqA2qQiEEPxApSCqT+ X-Received: by 2002:a05:6a00:c8d:b0:6be:b92b:e5fe with SMTP id a13-20020a056a000c8d00b006beb92be5femr25206628pfv.34.1699200483125; Sun, 05 Nov 2023 08:08:03 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699200483; cv=none; d=google.com; s=arc-20160816; b=ybO7MES5B/egx1TcGvVqlDFaXP5a5YwP1C2W2cng3LN8qe6EF4/6FK7Mx5yXNzFkrr tkjhAVaU0ulDh6eO/hKQ5T7OWuhWZJtu/ADyDTD20ARCiTgMdk7ALE6y11aoynCDcVrZ Ty4cIlhMUxrCHKSCKVaTAQCbfPtMdvZbT1vRvWWVM/nF7NFmEz6KjN+g6vS0aZUA3CTi U2rUOrr9Xf5GYMEO6WPEiyGzTwFtE8QJC8t/fjXViXV0ct2DI4hmzVevHAuEWOn5lwJt FHKE+D/RoQ1fU6quufBw87MIAgfN8MYLB/ofUsYH9AUKFJ3/TqVw1cOhhwxP2q5GuS79 EPQQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=u4TIwiJPl4/ZCnewrA1SERyiHxL2bRkdnW7Qzr5Y8KY=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=DYhke9ga0TkUNALPmEqBRDQdqnkPFN3/AIwOFlaPBgmQLbBBkZl790nM7eq+k1iN6L wwM0ABBiJ/FKxMjvlvii7vDlqQEXF1BqwW1ZS/PlV1I/ODQjIVDRsTRc7oo0hRdHp5+9 gD0ugl7JrvNphKJBCpZXTTUfYcCbcWCjTRbiLcbj1q3YZQWolcEn0PaPns3PUD7KljHY odwyrw/C3CygJHvdxh1dD6llKMjMwgBts0bAddxA6VwQBtHC+Zz7AyIi6R3ovyL5tTT9 wd2S6bZSkOmq/HTcvbSdwOHGDYPNIkaeRug4NH9HY9WzTKwwPUka2qK3+1Ycioovl+6P Vmaw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=q8jwJALV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id bj20-20020a056a02019400b005b8ee4c9237si6456464pgb.236.2023.11.05.08.08.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Nov 2023 08:08:03 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=q8jwJALV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 2D8F280B1D29; Sun, 5 Nov 2023 08:08:02 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229714AbjKEQH4 (ORCPT + 34 others); Sun, 5 Nov 2023 11:07:56 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37060 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229740AbjKEQHx (ORCPT ); Sun, 5 Nov 2023 11:07:53 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E5114100 for ; Sun, 5 Nov 2023 08:07:49 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id CC075C433C7; Sun, 5 Nov 2023 16:07:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1699200469; bh=1PVngkfwoPW/M4fS6uNXVz3kvUTFG/1vtJIUy1SBl4Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=q8jwJALVpGWeikN1IfHC3g5XvrxYZu2qUeo+I7HQ/PtgN4Q68MYRGONoz//JVGAt4 4N9tfDhAjfTxqhD+7laIp63sXPIk6dw997qsI9RxhuZzxnXHmEmR8UwN/fleMptLj4 TOv7UVWWuuTg/ThKQMJGuQs4S5C3720JbGjxogqYY3KXK27rtQIGD8roJxhoNH8nmU cGWvqlJZD/JR7XyeREaYYHwKP5Bes3oOLvLXPvpQBMiji/g1144QMeX0BNS2cA6BE3 Uat0pItQzZ5N9dkbH800HgCd6kXaVeV9Cn6TlO4rNq6VhGnEdaZHR9oYHR6hJbipGr w83LPc8k+0zJQ== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [RFC PATCH 06/32] function_graph: Remove logic around ftrace_graph_entry and return Date: Mon, 6 Nov 2023 01:07:43 +0900 Message-Id: <169920046286.482486.7747735574731235728.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169920038849.482486.15796387219966662967.stgit@devnote2> References: <169920038849.482486.15796387219966662967.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-5.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Sun, 05 Nov 2023 08:08:02 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781740845592802613 X-GMAIL-MSGID: 1781740845592802613 From: Steven Rostedt (VMware) The function pointers ftrace_graph_entry and ftrace_graph_return are no longer called via the function_graph tracer. Instead, an array structure is now used that will allow for multiple users of the function_graph infrastructure. The variables are still used by the architecture code for non dynamic ftrace configs, where a test is made against them to see if they point to the default stub function or not. This is how the static function tracing knows to call into the function graph tracer infrastructure or not. Two new stub functions are made. entry_run() and return_run(). The ftrace_graph_entry and ftrace_graph_return are set to them repectively when the function graph tracer is enabled, and this will trigger the architecture specific function graph code to be executed. This also requires checking the global_ops hash for all calls into the function_graph tracer. Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- kernel/trace/fgraph.c | 72 +++++++++++----------------------------- kernel/trace/ftrace.c | 2 - kernel/trace/ftrace_internal.h | 2 - 3 files changed, 20 insertions(+), 56 deletions(-) diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index 58750a395313..78335be91007 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -128,6 +128,18 @@ static inline int get_fgraph_array(struct task_struct *t, int offset) FGRAPH_ARRAY_MASK; } +/* ftrace_graph_entry set to this to tell some archs to run function graph */ +static int entry_run(struct ftrace_graph_ent *trace) +{ + return 0; +} + +/* ftrace_graph_return set to this to tell some archs to run function graph */ +static void return_run(struct ftrace_graph_ret *trace) +{ + return; +} + /* * @offset: The index into @t->ret_stack to find the ret_stack entry * @index: Where to place the index into @t->ret_stack of that entry @@ -323,6 +335,10 @@ int function_graph_enter(unsigned long ret, unsigned long func, ftrace_find_rec_direct(ret - MCOUNT_INSN_SIZE)) return -EBUSY; #endif + + if (!ftrace_ops_test(&global_ops, func, NULL)) + return -EBUSY; + trace.func = func; trace.depth = ++current->curr_ret_depth; @@ -664,7 +680,6 @@ extern void ftrace_stub_graph(struct ftrace_graph_ret *); /* The callbacks that hook a function */ trace_func_graph_ret_t ftrace_graph_return = ftrace_stub_graph; trace_func_graph_ent_t ftrace_graph_entry = ftrace_graph_entry_stub; -static trace_func_graph_ent_t __ftrace_graph_entry = ftrace_graph_entry_stub; /* Try to assign a return stack array on FTRACE_RETSTACK_ALLOC_SIZE tasks. */ static int alloc_retstack_tasklist(unsigned long **ret_stack_list) @@ -747,46 +762,6 @@ ftrace_graph_probe_sched_switch(void *ignore, bool preempt, } } -static int ftrace_graph_entry_test(struct ftrace_graph_ent *trace) -{ - if (!ftrace_ops_test(&global_ops, trace->func, NULL)) - return 0; - return __ftrace_graph_entry(trace); -} - -/* - * The function graph tracer should only trace the functions defined - * by set_ftrace_filter and set_ftrace_notrace. If another function - * tracer ops is registered, the graph tracer requires testing the - * function against the global ops, and not just trace any function - * that any ftrace_ops registered. - */ -void update_function_graph_func(void) -{ - struct ftrace_ops *op; - bool do_test = false; - - /* - * The graph and global ops share the same set of functions - * to test. If any other ops is on the list, then - * the graph tracing needs to test if its the function - * it should call. - */ - do_for_each_ftrace_op(op, ftrace_ops_list) { - if (op != &global_ops && op != &graph_ops && - op != &ftrace_list_end) { - do_test = true; - /* in double loop, break out with goto */ - goto out; - } - } while_for_each_ftrace_op(op); - out: - if (do_test) - ftrace_graph_entry = ftrace_graph_entry_test; - else - ftrace_graph_entry = __ftrace_graph_entry; -} - static DEFINE_PER_CPU(unsigned long *, idle_ret_stack); static void @@ -928,18 +903,12 @@ int register_ftrace_graph(struct fgraph_ops *gops) ftrace_graph_active--; goto out; } - - ftrace_graph_return = gops->retfunc; - /* - * Update the indirect function to the entryfunc, and the - * function that gets called to the entry_test first. Then - * call the update fgraph entry function to determine if - * the entryfunc should be called directly or not. + * Some archs just test to see if these are not + * the default function */ - __ftrace_graph_entry = gops->entryfunc; - ftrace_graph_entry = ftrace_graph_entry_test; - update_function_graph_func(); + ftrace_graph_return = return_run; + ftrace_graph_entry = entry_run; ret = ftrace_startup(&graph_ops, FTRACE_START_FUNC_RET); } @@ -975,7 +944,6 @@ void unregister_ftrace_graph(struct fgraph_ops *gops) if (!ftrace_graph_active) { ftrace_graph_return = ftrace_stub_graph; ftrace_graph_entry = ftrace_graph_entry_stub; - __ftrace_graph_entry = ftrace_graph_entry_stub; ftrace_shutdown(&graph_ops, FTRACE_STOP_FUNC_RET); unregister_pm_notifier(&ftrace_suspend_notifier); unregister_trace_sched_switch(ftrace_graph_probe_sched_switch, NULL); diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 8de8bec5f366..fd64021ec52f 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -235,8 +235,6 @@ static void update_ftrace_function(void) func = ftrace_ops_list_func; } - update_function_graph_func(); - /* If there's no change, then do nothing more here */ if (ftrace_trace_function == func) return; diff --git a/kernel/trace/ftrace_internal.h b/kernel/trace/ftrace_internal.h index 5012c04f92c0..19eddcb91584 100644 --- a/kernel/trace/ftrace_internal.h +++ b/kernel/trace/ftrace_internal.h @@ -42,10 +42,8 @@ ftrace_ops_test(struct ftrace_ops *ops, unsigned long ip, void *regs) #ifdef CONFIG_FUNCTION_GRAPH_TRACER extern int ftrace_graph_active; -void update_function_graph_func(void); #else /* !CONFIG_FUNCTION_GRAPH_TRACER */ # define ftrace_graph_active 0 -static inline void update_function_graph_func(void) { } #endif /* CONFIG_FUNCTION_GRAPH_TRACER */ #else /* !CONFIG_FUNCTION_TRACER */ From patchwork Sun Nov 5 16:07:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 161664 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp2196608vqu; Sun, 5 Nov 2023 08:08:28 -0800 (PST) X-Google-Smtp-Source: AGHT+IESIMc5hNkBeFZmC+cqFn3xSV0Rvs+Sal/Awggwsgsg+6g8pQmtzKqmNduEjzdA/+OXB3Ht X-Received: by 2002:a05:6a00:9a6:b0:68f:d44c:22f8 with SMTP id u38-20020a056a0009a600b0068fd44c22f8mr11112646pfg.1.1699200508115; Sun, 05 Nov 2023 08:08:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699200508; cv=none; d=google.com; s=arc-20160816; b=JeXj3hrwB++Uo6NkzoQ3aGz9nqh4qQ2Z+NJX9squXyyQBrRJC57vab45BYLd1fTQ37 ho8kjzl80iu0pBGM++6pn+89hULS719si8rMOPvVC7dIwZGwzTmjAraIqaCm2inqJ6Gn 8rJcNr1ebeDnfrmO20YExddCBV2ESu/cRNhDP7q5DfhmsLizW6k43R4j3XQ2EOEpYoM9 NW0VawwZJka9YdRgGnJzDzBg373ltQb6Z4nzbvBlsIpwvmHpBfZ9n3NHbFqeTAU7L63M MHoS0JiIfCSxspDakQ+Cit151I9T66m5EblwNN5MNBoKygC3of6x8kH57LN9+2t8+oEE z5Rg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=Gp95ZUld3qvTxgIkdmeY6RXjI2S6kmqAZL1KxPVpPtA=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=meLRfLP/al5d9R3valo72lnUWJBYiSk+yXNYgbeeKR1dwIE1PP5Yf4iJZ/npS/fc89 B6LQXSUHZGiZHaU5Io59a188Vzkp+0gZquyRFS0cbfrrATMTDA1cprMGiMy0kwykHzHf db/e9tv76XCZlX2sCLzRRrmsoH7jFCG+iO+pOq6Nw3buGf6fOusZuJT88zU0STW/7bOI JPKsgFSjEl+q17984nv/Pv3hQ7Eghv1p9Yck0mTR8CwnKsfPxjewb4QtxAqqZwfIRa/j 99QTDSeeXXcmfwWcckw2Jgh1b5GUqedgA9mjZhGIxgM2nFNK6NCR8gKuELmdZlFEkXS8 ApFA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="J4yqWM/V"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id cj5-20020a056a00298500b006c332132d38si6492371pfb.268.2023.11.05.08.08.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Nov 2023 08:08:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="J4yqWM/V"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 4911F80B1D2A; Sun, 5 Nov 2023 08:08:27 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229721AbjKEQIL (ORCPT + 34 others); Sun, 5 Nov 2023 11:08:11 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37390 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229709AbjKEQIG (ORCPT ); Sun, 5 Nov 2023 11:08:06 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3CB5BD6B for ; Sun, 5 Nov 2023 08:08:02 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 054C8C433C7; Sun, 5 Nov 2023 16:07:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1699200481; bh=yvvI/fxh1WfuoHsR6rpCYSViXpG5e+1tM8mJ0EAGPvY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=J4yqWM/V5f/eNa825KVu/wz7ra7TBpeAy4ZE0HGv0iJIi8tPbR/44nWYrDIAVLqjR ebP5gOKWP3MhRviBxYh0jcJL5G2y3XAi7/c4eDN6F2YPThGeQ84bw+0dwhJXdvT8QR 4lc25v8gh01qEu5Z35iebyutzEZA7TBAnO8l36LtEYz2W/UhRR9o6537sSnssxUrLU ptfRE982DjdlPl4UsQk66kNZVwB92JUzQvhhCUB5VW1df78RzsSgBb+5TXQfNg7g6L jBfdhGKc1byj+iMjmUKYJpaxK1pv3cGyqWT+DgsVgGdbdbPTccqVjh16fBV/PdMekd gkcQSp05kIuRw== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [RFC PATCH 07/32] ftrace/function_graph: Pass fgraph_ops to function graph callbacks Date: Mon, 6 Nov 2023 01:07:55 +0900 Message-Id: <169920047497.482486.17178474677119338488.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169920038849.482486.15796387219966662967.stgit@devnote2> References: <169920038849.482486.15796387219966662967.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-5.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Sun, 05 Nov 2023 08:08:27 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781740871844280558 X-GMAIL-MSGID: 1781740871844280558 From: Steven Rostedt (VMware) Pass the fgraph_ops structure to the function graph callbacks. This will allow callbacks to add a descriptor to a fgraph_ops private field that wil be added in the future and use it for the callbacks. This will be useful when more than one callback can be registered to the function graph tracer. Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- include/linux/ftrace.h | 10 +++++++--- kernel/trace/fgraph.c | 17 ++++++++++------- kernel/trace/ftrace.c | 6 ++++-- kernel/trace/trace.h | 4 ++-- kernel/trace/trace_functions_graph.c | 11 +++++++---- kernel/trace/trace_irqsoff.c | 6 ++++-- kernel/trace/trace_sched_wakeup.c | 6 ++++-- kernel/trace/trace_selftest.c | 5 +++-- 8 files changed, 41 insertions(+), 24 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index e8921871ef9a..ff32a0ec57aa 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -1029,11 +1029,15 @@ struct ftrace_graph_ret { unsigned long long rettime; } __packed; +struct fgraph_ops; + /* Type of the callback handlers for tracing function graph*/ -typedef void (*trace_func_graph_ret_t)(struct ftrace_graph_ret *); /* return */ -typedef int (*trace_func_graph_ent_t)(struct ftrace_graph_ent *); /* entry */ +typedef void (*trace_func_graph_ret_t)(struct ftrace_graph_ret *, + struct fgraph_ops *); /* return */ +typedef int (*trace_func_graph_ent_t)(struct ftrace_graph_ent *, + struct fgraph_ops *); /* entry */ -extern int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace); +extern int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace, struct fgraph_ops *gops); #ifdef CONFIG_FUNCTION_GRAPH_TRACER diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index 78335be91007..1e8c17f70b84 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -129,13 +129,13 @@ static inline int get_fgraph_array(struct task_struct *t, int offset) } /* ftrace_graph_entry set to this to tell some archs to run function graph */ -static int entry_run(struct ftrace_graph_ent *trace) +static int entry_run(struct ftrace_graph_ent *trace, struct fgraph_ops *ops) { return 0; } /* ftrace_graph_return set to this to tell some archs to run function graph */ -static void return_run(struct ftrace_graph_ret *trace) +static void return_run(struct ftrace_graph_ret *trace, struct fgraph_ops *ops) { return; } @@ -200,12 +200,14 @@ int __weak ftrace_disable_ftrace_graph_caller(void) } #endif -int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace) +int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace, + struct fgraph_ops *gops) { return 0; } -static void ftrace_graph_ret_stub(struct ftrace_graph_ret *trace) +static void ftrace_graph_ret_stub(struct ftrace_graph_ret *trace, + struct fgraph_ops *gops) { } @@ -359,7 +361,7 @@ int function_graph_enter(unsigned long ret, unsigned long func, atomic_inc(¤t->trace_overrun); break; } - if (fgraph_array[i]->entryfunc(&trace)) { + if (fgraph_array[i]->entryfunc(&trace, fgraph_array[i])) { offset = current->curr_ret_stack; /* Check the top level stored word */ type = get_fgraph_type(current, offset - 1); @@ -533,7 +535,7 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs i = 0; do { idx = get_fgraph_array(current, offset - i); - fgraph_array[idx]->retfunc(&trace); + fgraph_array[idx]->retfunc(&trace, fgraph_array[idx]); i++; } while (i < index); @@ -675,7 +677,8 @@ void ftrace_graph_sleep_time_control(bool enable) * Simply points to ftrace_stub, but with the proper protocol. * Defined by the linker script in linux/vmlinux.lds.h */ -extern void ftrace_stub_graph(struct ftrace_graph_ret *); +extern void ftrace_stub_graph(struct ftrace_graph_ret *, + struct fgraph_ops *); /* The callbacks that hook a function */ trace_func_graph_ret_t ftrace_graph_return = ftrace_stub_graph; diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index fd64021ec52f..7ff5c454622a 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -815,7 +815,8 @@ void ftrace_graph_graph_time_control(bool enable) fgraph_graph_time = enable; } -static int profile_graph_entry(struct ftrace_graph_ent *trace) +static int profile_graph_entry(struct ftrace_graph_ent *trace, + struct fgraph_ops *gops) { struct ftrace_ret_stack *ret_stack; @@ -832,7 +833,8 @@ static int profile_graph_entry(struct ftrace_graph_ent *trace) return 1; } -static void profile_graph_return(struct ftrace_graph_ret *trace) +static void profile_graph_return(struct ftrace_graph_ret *trace, + struct fgraph_ops *gops) { struct ftrace_ret_stack *ret_stack; struct ftrace_profile_stat *stat; diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h index 77debe53f07c..ada49bf1fbc8 100644 --- a/kernel/trace/trace.h +++ b/kernel/trace/trace.h @@ -670,8 +670,8 @@ void trace_latency_header(struct seq_file *m); void trace_default_header(struct seq_file *m); void print_trace_header(struct seq_file *m, struct trace_iterator *iter); -void trace_graph_return(struct ftrace_graph_ret *trace); -int trace_graph_entry(struct ftrace_graph_ent *trace); +void trace_graph_return(struct ftrace_graph_ret *trace, struct fgraph_ops *gops); +int trace_graph_entry(struct ftrace_graph_ent *trace, struct fgraph_ops *gops); void set_graph_array(struct trace_array *tr); void tracing_start_cmdline_record(void); diff --git a/kernel/trace/trace_functions_graph.c b/kernel/trace/trace_functions_graph.c index c35fbaab2a47..b7b142b65299 100644 --- a/kernel/trace/trace_functions_graph.c +++ b/kernel/trace/trace_functions_graph.c @@ -129,7 +129,8 @@ static inline int ftrace_graph_ignore_irqs(void) return in_hardirq(); } -int trace_graph_entry(struct ftrace_graph_ent *trace) +int trace_graph_entry(struct ftrace_graph_ent *trace, + struct fgraph_ops *gops) { struct trace_array *tr = graph_array; struct trace_array_cpu *data; @@ -238,7 +239,8 @@ void __trace_graph_return(struct trace_array *tr, trace_buffer_unlock_commit_nostack(buffer, event); } -void trace_graph_return(struct ftrace_graph_ret *trace) +void trace_graph_return(struct ftrace_graph_ret *trace, + struct fgraph_ops *gops) { struct trace_array *tr = graph_array; struct trace_array_cpu *data; @@ -275,7 +277,8 @@ void set_graph_array(struct trace_array *tr) smp_mb(); } -static void trace_graph_thresh_return(struct ftrace_graph_ret *trace) +static void trace_graph_thresh_return(struct ftrace_graph_ret *trace, + struct fgraph_ops *gops) { ftrace_graph_addr_finish(trace); @@ -288,7 +291,7 @@ static void trace_graph_thresh_return(struct ftrace_graph_ret *trace) (trace->rettime - trace->calltime < tracing_thresh)) return; else - trace_graph_return(trace); + trace_graph_return(trace, gops); } static struct fgraph_ops funcgraph_thresh_ops = { diff --git a/kernel/trace/trace_irqsoff.c b/kernel/trace/trace_irqsoff.c index ba37f768e2f2..5478f4c4f708 100644 --- a/kernel/trace/trace_irqsoff.c +++ b/kernel/trace/trace_irqsoff.c @@ -175,7 +175,8 @@ static int irqsoff_display_graph(struct trace_array *tr, int set) return start_irqsoff_tracer(irqsoff_trace, set); } -static int irqsoff_graph_entry(struct ftrace_graph_ent *trace) +static int irqsoff_graph_entry(struct ftrace_graph_ent *trace, + struct fgraph_ops *gops) { struct trace_array *tr = irqsoff_trace; struct trace_array_cpu *data; @@ -205,7 +206,8 @@ static int irqsoff_graph_entry(struct ftrace_graph_ent *trace) return ret; } -static void irqsoff_graph_return(struct ftrace_graph_ret *trace) +static void irqsoff_graph_return(struct ftrace_graph_ret *trace, + struct fgraph_ops *gops) { struct trace_array *tr = irqsoff_trace; struct trace_array_cpu *data; diff --git a/kernel/trace/trace_sched_wakeup.c b/kernel/trace/trace_sched_wakeup.c index 0469a04a355f..49bcc812652c 100644 --- a/kernel/trace/trace_sched_wakeup.c +++ b/kernel/trace/trace_sched_wakeup.c @@ -112,7 +112,8 @@ static int wakeup_display_graph(struct trace_array *tr, int set) return start_func_tracer(tr, set); } -static int wakeup_graph_entry(struct ftrace_graph_ent *trace) +static int wakeup_graph_entry(struct ftrace_graph_ent *trace, + struct fgraph_ops *gops) { struct trace_array *tr = wakeup_trace; struct trace_array_cpu *data; @@ -141,7 +142,8 @@ static int wakeup_graph_entry(struct ftrace_graph_ent *trace) return ret; } -static void wakeup_graph_return(struct ftrace_graph_ret *trace) +static void wakeup_graph_return(struct ftrace_graph_ret *trace, + struct fgraph_ops *gops) { struct trace_array *tr = wakeup_trace; struct trace_array_cpu *data; diff --git a/kernel/trace/trace_selftest.c b/kernel/trace/trace_selftest.c index 529590499b1f..914331d8242c 100644 --- a/kernel/trace/trace_selftest.c +++ b/kernel/trace/trace_selftest.c @@ -762,7 +762,8 @@ trace_selftest_startup_function(struct tracer *trace, struct trace_array *tr) static unsigned int graph_hang_thresh; /* Wrap the real function entry probe to avoid possible hanging */ -static int trace_graph_entry_watchdog(struct ftrace_graph_ent *trace) +static int trace_graph_entry_watchdog(struct ftrace_graph_ent *trace, + struct fgraph_ops *gops) { /* This is harmlessly racy, we want to approximately detect a hang */ if (unlikely(++graph_hang_thresh > GRAPH_MAX_FUNC_TEST)) { @@ -776,7 +777,7 @@ static int trace_graph_entry_watchdog(struct ftrace_graph_ent *trace) return 0; } - return trace_graph_entry(trace); + return trace_graph_entry(trace, gops); } static struct fgraph_ops fgraph_ops __initdata = { From patchwork Sun Nov 5 16:08:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 161665 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp2196640vqu; Sun, 5 Nov 2023 08:08:30 -0800 (PST) X-Google-Smtp-Source: AGHT+IH1XmQua4uSSO6bkX4+SQNpch5D2dXlIuIqx7fiFqeCKkT8L+B9BFMV1Qgxgo4edWu+ajjW X-Received: by 2002:a05:6a21:3284:b0:184:26f8:4398 with SMTP id yt4-20020a056a21328400b0018426f84398mr614197pzb.16.1699200510621; Sun, 05 Nov 2023 08:08:30 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699200510; cv=none; d=google.com; s=arc-20160816; b=Oo8hF0Mx+bc2ZBed76PMrqJLbqaPO4xItWfZwF+yPY4SHOU/wEBYlljNOi3JmXQJ7A 8qNXjeo1l4jeS42KUXxpAbJkJw6zXzgOcyhHqCAkuNUVTtQLclc8/fdXuei8TaF+nvNO PoT4pD/s11+5K2Qn6fnrfwpXjNLNUPPX9cByLi25gXMGmWyeO889urblfBmr9V1knn9z UYxD0oz+nKJexbko07D2PRavP/qfhg9LyOKwiRQxT4HBaUxpVXEC0UHrh5vStPj7GeCj ewq/smEzIeqkdYr4A8cVxTbJKK7Xa2qIj7OrNZzB3Q6LOhBU+XWwk6YeqR+5CDSDFiw4 wayQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=Pi+uhDJenBEQt7g7kzWAUx9PrrJ6mdu2cGTCLK+pMCs=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=eER4ZmwOaZy7y3jayzzeEwfgLgXU/BRbd/gsASjbjjH4e+jEMc4OSdkzwy7JR17SOf cryslb+r79f94adtI1uFO/qBEqyswgBcWbA3GnHPvadKhW7IlQdRsPe1hw/fz/fxvDX/ ZkbOaFaonqH61vjq4uB/wtaCbZj7ANuiRkZL0+QL7PVAbrAtsQJbfNT/bJ+QKUquN58R 08MiKkMPtvdEgEkyjahH4Ur3AKwl9dP/O7xOgC/RrD3jCtg87XrUlCqphzw66cheAStt ieDu5eUhXr0AlSTrjD6w/CH1jTkQSpkMpXPxtukNP/uI+SugSm1bSNorF9E834bA+oxS mhiA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=vMp9EDFB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id d1-20020a170903230100b001cc4f55db71si6387070plh.343.2023.11.05.08.08.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Nov 2023 08:08:30 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=vMp9EDFB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id BE45F80B1D36; Sun, 5 Nov 2023 08:08:29 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229765AbjKEQIX (ORCPT + 34 others); Sun, 5 Nov 2023 11:08:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53056 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229707AbjKEQIS (ORCPT ); Sun, 5 Nov 2023 11:08:18 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 51E76EB for ; Sun, 5 Nov 2023 08:08:14 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 02A2DC433C8; Sun, 5 Nov 2023 16:08:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1699200494; bh=Cg6ntUmQC2rEjvbKG7mMJ57xA499xXXa/ZvQe5xjpag=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=vMp9EDFB/dc+TrbNZAz3O4HAqzYXQkw1wC1bjf7QikvwEvrcZRGeJhyRVbJ0cZ2Mv 8OjPHi0njrY+D5QU7M2LLbR7Eoo3rJFAJ52i/qExMyA6AYEYQiyR++5M9lThZDyTmy URVMLMLDa/niqzCZwrEygDjpWPv+u8qAj8IWk0ujS5q03vkKNccNQMRPQZRVs2p7u9 m7N22Ylgq5SUUwHTAEM14HwU2zaTJVX+d4qb6GQ+Qh4Qf+X68/Kte7sLRtCeTwJZHR khtg1szGP2qNWSbBLU9+90gmt82UNQqeNoldSd2zWUv94p7Bagr/W8NIUDxAaNqzlE FJAQM7r8WER1Q== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [RFC PATCH 08/32] ftrace: Allow function_graph tracer to be enabled in instances Date: Mon, 6 Nov 2023 01:08:07 +0900 Message-Id: <169920048725.482486.18063340929679228603.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169920038849.482486.15796387219966662967.stgit@devnote2> References: <169920038849.482486.15796387219966662967.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-5.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Sun, 05 Nov 2023 08:08:29 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781740874592301576 X-GMAIL-MSGID: 1781740874592301576 From: Steven Rostedt (VMware) Now that function graph tracing can handle more than one user, allow it to be enabled in the ftrace instances. Note, the filtering of the functions is still joined by the top level set_ftrace_filter and friends, as well as the graph and nograph files. Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- include/linux/ftrace.h | 1 + kernel/trace/ftrace.c | 1 + kernel/trace/trace.h | 12 ++++++ kernel/trace/trace_functions.c | 8 ++++ kernel/trace/trace_functions_graph.c | 65 +++++++++++++++++++++------------- kernel/trace/trace_selftest.c | 2 + 6 files changed, 63 insertions(+), 26 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index ff32a0ec57aa..4ad587b3525e 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -1044,6 +1044,7 @@ extern int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace, struct fgraph struct fgraph_ops { trace_func_graph_ent_t entryfunc; trace_func_graph_ret_t retfunc; + void *private; }; /* diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 7ff5c454622a..83fbfb7b48f8 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -7319,6 +7319,7 @@ __init void ftrace_init_global_array_ops(struct trace_array *tr) tr->ops = &global_ops; tr->ops->private = tr; ftrace_init_trace_array(tr); + init_array_fgraph_ops(tr); } void ftrace_init_array_ops(struct trace_array *tr, ftrace_func_t func) diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h index ada49bf1fbc8..73fe789ba143 100644 --- a/kernel/trace/trace.h +++ b/kernel/trace/trace.h @@ -395,6 +395,9 @@ struct trace_array { struct ftrace_ops *ops; struct trace_pid_list __rcu *function_pids; struct trace_pid_list __rcu *function_no_pids; +#ifdef CONFIG_FUNCTION_GRAPH_TRACER + struct fgraph_ops *gops; +#endif #ifdef CONFIG_DYNAMIC_FTRACE /* All of these are protected by the ftrace_lock */ struct list_head func_probes; @@ -883,6 +886,9 @@ extern int __trace_graph_entry(struct trace_array *tr, extern void __trace_graph_return(struct trace_array *tr, struct ftrace_graph_ret *trace, unsigned int trace_ctx); +extern void init_array_fgraph_ops(struct trace_array *tr); +extern int allocate_fgraph_ops(struct trace_array *tr); +extern void free_fgraph_ops(struct trace_array *tr); #ifdef CONFIG_DYNAMIC_FTRACE extern struct ftrace_hash __rcu *ftrace_graph_hash; @@ -995,6 +1001,12 @@ print_graph_function_flags(struct trace_iterator *iter, u32 flags) { return TRACE_TYPE_UNHANDLED; } +static inline void init_array_fgraph_ops(struct trace_array *tr) { } +static inline int allocate_fgraph_ops(struct trace_array *tr) +{ + return 0; +} +static inline void free_fgraph_ops(struct trace_array *tr) { } #endif /* CONFIG_FUNCTION_GRAPH_TRACER */ extern struct list_head ftrace_pids; diff --git a/kernel/trace/trace_functions.c b/kernel/trace/trace_functions.c index 9f1bfbe105e8..8e8da0d0ee52 100644 --- a/kernel/trace/trace_functions.c +++ b/kernel/trace/trace_functions.c @@ -80,6 +80,7 @@ void ftrace_free_ftrace_ops(struct trace_array *tr) int ftrace_create_function_files(struct trace_array *tr, struct dentry *parent) { + int ret; /* * The top level array uses the "global_ops", and the files are * created on boot up. @@ -90,6 +91,12 @@ int ftrace_create_function_files(struct trace_array *tr, if (!tr->ops) return -EINVAL; + ret = allocate_fgraph_ops(tr); + if (ret) { + kfree(tr->ops); + return ret; + } + ftrace_create_filter_files(tr->ops, parent); return 0; @@ -99,6 +106,7 @@ void ftrace_destroy_function_files(struct trace_array *tr) { ftrace_destroy_filter_files(tr->ops); ftrace_free_ftrace_ops(tr); + free_fgraph_ops(tr); } static ftrace_func_t select_trace_function(u32 flags_val) diff --git a/kernel/trace/trace_functions_graph.c b/kernel/trace/trace_functions_graph.c index b7b142b65299..9ccc904a7703 100644 --- a/kernel/trace/trace_functions_graph.c +++ b/kernel/trace/trace_functions_graph.c @@ -83,8 +83,6 @@ static struct tracer_flags tracer_flags = { .opts = trace_opts }; -static struct trace_array *graph_array; - /* * DURATION column is being also used to display IRQ signs, * following values are used by print_graph_irq and others @@ -132,7 +130,7 @@ static inline int ftrace_graph_ignore_irqs(void) int trace_graph_entry(struct ftrace_graph_ent *trace, struct fgraph_ops *gops) { - struct trace_array *tr = graph_array; + struct trace_array *tr = gops->private; struct trace_array_cpu *data; unsigned long flags; unsigned int trace_ctx; @@ -242,7 +240,7 @@ void __trace_graph_return(struct trace_array *tr, void trace_graph_return(struct ftrace_graph_ret *trace, struct fgraph_ops *gops) { - struct trace_array *tr = graph_array; + struct trace_array *tr = gops->private; struct trace_array_cpu *data; unsigned long flags; unsigned int trace_ctx; @@ -268,15 +266,6 @@ void trace_graph_return(struct ftrace_graph_ret *trace, local_irq_restore(flags); } -void set_graph_array(struct trace_array *tr) -{ - graph_array = tr; - - /* Make graph_array visible before we start tracing */ - - smp_mb(); -} - static void trace_graph_thresh_return(struct ftrace_graph_ret *trace, struct fgraph_ops *gops) { @@ -294,25 +283,53 @@ static void trace_graph_thresh_return(struct ftrace_graph_ret *trace, trace_graph_return(trace, gops); } -static struct fgraph_ops funcgraph_thresh_ops = { - .entryfunc = &trace_graph_entry, - .retfunc = &trace_graph_thresh_return, -}; - static struct fgraph_ops funcgraph_ops = { .entryfunc = &trace_graph_entry, .retfunc = &trace_graph_return, }; +int allocate_fgraph_ops(struct trace_array *tr) +{ + struct fgraph_ops *gops; + + gops = kzalloc(sizeof(*gops), GFP_KERNEL); + if (!gops) + return -ENOMEM; + + gops->entryfunc = &trace_graph_entry; + gops->retfunc = &trace_graph_return; + + tr->gops = gops; + gops->private = tr; + return 0; +} + +void free_fgraph_ops(struct trace_array *tr) +{ + kfree(tr->gops); +} + +__init void init_array_fgraph_ops(struct trace_array *tr) +{ + tr->gops = &funcgraph_ops; + funcgraph_ops.private = tr; +} + static int graph_trace_init(struct trace_array *tr) { int ret; - set_graph_array(tr); + tr->gops->entryfunc = trace_graph_entry; + if (tracing_thresh) - ret = register_ftrace_graph(&funcgraph_thresh_ops); + tr->gops->retfunc = trace_graph_thresh_return; else - ret = register_ftrace_graph(&funcgraph_ops); + tr->gops->retfunc = trace_graph_return; + + /* Make gops functions are visible before we start tracing */ + smp_mb(); + + ret = register_ftrace_graph(tr->gops); if (ret) return ret; tracing_start_cmdline_record(); @@ -323,10 +340,7 @@ static int graph_trace_init(struct trace_array *tr) static void graph_trace_reset(struct trace_array *tr) { tracing_stop_cmdline_record(); - if (tracing_thresh) - unregister_ftrace_graph(&funcgraph_thresh_ops); - else - unregister_ftrace_graph(&funcgraph_ops); + unregister_ftrace_graph(tr->gops); } static int graph_trace_update_thresh(struct trace_array *tr) @@ -1365,6 +1379,7 @@ static struct tracer graph_trace __tracer_data = { .print_header = print_graph_headers, .flags = &tracer_flags, .set_flag = func_graph_set_flag, + .allow_instances = true, #ifdef CONFIG_FTRACE_SELFTEST .selftest = trace_selftest_startup_function_graph, #endif diff --git a/kernel/trace/trace_selftest.c b/kernel/trace/trace_selftest.c index 914331d8242c..e05f7335525c 100644 --- a/kernel/trace/trace_selftest.c +++ b/kernel/trace/trace_selftest.c @@ -813,7 +813,7 @@ trace_selftest_startup_function_graph(struct tracer *trace, * to detect and recover from possible hangs */ tracing_reset_online_cpus(&tr->array_buffer); - set_graph_array(tr); + fgraph_ops.private = tr; ret = register_ftrace_graph(&fgraph_ops); if (ret) { warn_failed_init_tracer(trace, ret); From patchwork Sun Nov 5 16:08:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 161666 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp2196840vqu; Sun, 5 Nov 2023 08:08:58 -0800 (PST) X-Google-Smtp-Source: AGHT+IHjhjQc9+inY6bWyCItb5b0wspyRD8eJQMXIQekDozuVzNMLDBOe2P/ctXGZ8wNTzTwPTNb X-Received: by 2002:a05:6808:2209:b0:3ab:74a2:ab2b with SMTP id bd9-20020a056808220900b003ab74a2ab2bmr38078894oib.48.1699200537865; Sun, 05 Nov 2023 08:08:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699200537; cv=none; d=google.com; s=arc-20160816; b=DOxjiAr0j/XmW3VauQj6FPJ3kcbcQdE2ZYfswL+xKj7cEFex5ong3McYD3Np8iWyGw OYbz8QT6pWIeDhpPJWdj1Ye9WpsoXxWG95MiGmtOd/KS6Cz0RwGfEro0LdPTKSpWA4uH fEoiCcC5f4NUc+7PCa/dxvpxuOQXTFksITbj7hDCOanu9kMI4D9jxyq/i2Ioq+syocjV /h5sugbQ3K0VEvN4LXbY1U6FizixDvBdQjzLT6WrN6RePD3I92sTh8SE3OaEw5uxpo/m 0Hah8ATV/oSVvbCNoYKSK/qn1z6DY4oS9xKJZO+A9pOme7vULL3euM8YbQG6MpmDkKW3 i3eA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=CZc980L7JOlO7hBu2hN9poZ4xtzkwYDGHgaeD/20/YM=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=ez3oJF+eEUWdNLGBksBNTaapyNq5t0OhsMFK8fQ3Q+XB5qzaOCkT1f8zkg97DOR7dM L07GoK2Q5+VvdXV+2UagFVxw9OSeS8eEk1gcjmV9nQlFY58bW6pPahIpXFEjSZWk/pQM 5fTjHOjYoWJEDl83NndvClXnyNsJVX60JpujW47/St805t4CPaT7Vy+HU5SCE24WSEKb asDUpX64MJKRzl+eV2SIl/Ro2SAXC0kcScJE+ueDuhYzQPkys+i6B9mw8AoUXmO5Ndhd kofGB9TSTnQXrdN+pup0UJqjB7YKxR/FD6TCNEjtDSQ4i6xL4ktvPD0iDgt4pY7EjanK 4+MA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=HoomEPBw; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id y26-20020aa7855a000000b00690258a9766si5778793pfn.373.2023.11.05.08.08.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Nov 2023 08:08:57 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=HoomEPBw; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 741EF8057D8D; Sun, 5 Nov 2023 08:08:55 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229690AbjKEQId (ORCPT + 34 others); Sun, 5 Nov 2023 11:08:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46108 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229845AbjKEQIa (ORCPT ); Sun, 5 Nov 2023 11:08:30 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 024C1D6D for ; Sun, 5 Nov 2023 08:08:26 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 80DF3C433C7; Sun, 5 Nov 2023 16:08:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1699200506; bh=aOBVpVmxMVywZjrn0W2fI6F8TmfS5e1bc3XWd3NWy64=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HoomEPBwPk2zxl/3w8o/aqAv3SiBAK51fqRLol7AWsu31v28TcgFujagwswHGOtpZ 740Sm101TNG7Z0f+3hYU6FzUXb2lHTCMhTFxRZgPzl7Zt5cJ227n96DnAI3gJ1wTXg JLsLm1pTVpOVeswIEIVXa36y2fn1R0amZt+he7N1iNQ3unlDlBpmqzEzGVXL+1W8EN r9rg2fQz7j2yQF+zQVccfPNNKIDI3sUgHanLcW28jnu3ZSnX9DYUO5pda2IJPv9Iul NlIZddwbSMlp2z7z+Wojc0hcUAd5KPLkuCQqxTPFTkrizl69s4ytNsejtVXOk4aPlx WyryZYEsZIzgA== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [RFC PATCH 09/32] ftrace: Allow ftrace startup flags exist without dynamic ftrace Date: Mon, 6 Nov 2023 01:08:19 +0900 Message-Id: <169920049937.482486.15191811301005089130.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169920038849.482486.15796387219966662967.stgit@devnote2> References: <169920038849.482486.15796387219966662967.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Sun, 05 Nov 2023 08:08:55 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781740903135603696 X-GMAIL-MSGID: 1781740903135603696 From: Steven Rostedt (VMware) Some of the flags for ftrace_startup() may be exposed even when CONFIG_DYNAMIC_FTRACE is not configured in. This is fine as the difference between dynamic ftrace and static ftrace is done within the internals of ftrace itself. No need to have use cases fail to compile because dynamic ftrace is disabled. This change is needed to move some of the logic of what is passed to ftrace_startup() out of the parameters of ftrace_startup(). Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- include/linux/ftrace.h | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 4ad587b3525e..7fd044ae3da5 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -512,6 +512,15 @@ static inline void stack_tracer_disable(void) { } static inline void stack_tracer_enable(void) { } #endif +enum { + FTRACE_UPDATE_CALLS = (1 << 0), + FTRACE_DISABLE_CALLS = (1 << 1), + FTRACE_UPDATE_TRACE_FUNC = (1 << 2), + FTRACE_START_FUNC_RET = (1 << 3), + FTRACE_STOP_FUNC_RET = (1 << 4), + FTRACE_MAY_SLEEP = (1 << 5), +}; + #ifdef CONFIG_DYNAMIC_FTRACE void ftrace_arch_code_modify_prepare(void); @@ -606,15 +615,6 @@ void ftrace_set_global_notrace(unsigned char *buf, int len, int reset); void ftrace_free_filter(struct ftrace_ops *ops); void ftrace_ops_set_global_filter(struct ftrace_ops *ops); -enum { - FTRACE_UPDATE_CALLS = (1 << 0), - FTRACE_DISABLE_CALLS = (1 << 1), - FTRACE_UPDATE_TRACE_FUNC = (1 << 2), - FTRACE_START_FUNC_RET = (1 << 3), - FTRACE_STOP_FUNC_RET = (1 << 4), - FTRACE_MAY_SLEEP = (1 << 5), -}; - /* * The FTRACE_UPDATE_* enum is used to pass information back * from the ftrace_update_record() and ftrace_test_record() From patchwork Sun Nov 5 16:08:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 161668 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp2197149vqu; Sun, 5 Nov 2023 08:09:38 -0800 (PST) X-Google-Smtp-Source: AGHT+IHk1+h5hN6+tORPY9IGJDzIF/Hy3N9OKK1LnZfAiO2CMTJx6Bey02mfVO3CZKU79m9lF6uH X-Received: by 2002:a17:903:2450:b0:1cc:70e4:28e9 with SMTP id l16-20020a170903245000b001cc70e428e9mr14282948pls.49.1699200578404; Sun, 05 Nov 2023 08:09:38 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699200578; cv=none; d=google.com; s=arc-20160816; b=ZQBnecTYw8LVugCO2Y/PBeY1qlTmfS1eTPsPtctNNjdM3f9Pj+0I6WduBiHOH/mFt1 lF2JV9XM1N1+xKTO6G0iWqdcTVdo92RB0dVn9/KTMBm6pgVeb5GJlcqzE7ySqyvHkeUd GXFv9jg9hKBaMeCWDo+Z/L7WH+e1dZedY696NjtdN9ny/4sD5hRWJCh0iVo3SV1caIfh OXXbQXQKD8IPNx06/k8DOuiWrJ4T8Fe8wvDlf3xDl54XxN32nDtiFoNTPNrHG6MWWfGA E8+zNq93e+hD+BDVF7pLEA7YAXbl8HBeJCi4YH5MdsQlF+R3jqistOVWNneSdhwKOfYG bAvw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=9gdNbRgZwPe+WpMx41q1hM9svreHpIc9tCJhTl8chWk=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=AJ0svBjPV1LLnj5c1NKMhrVqBtHpITXdezR10NpP6VeuEwwsPugz4oJSC2EAKWcp26 JhE66oesz7FVBvj7ksLlp3dcbxWat9alU8f5qcFUoJazbwlwY4a/pFR3VPLzhb/OQkBq W1PtNesMOqDpd6s9aegzPw+By19J8kumqap7R1wPCr+fGtb2AW2VGVPy6y4PJMg6CAfM sXC94p0j8otU/sMzjwpH+1oC2SPBxYLrHHgza7bQCHrUchFQiUGuD4VbSv3R6hmus/Om pcTCJpdCtJudpZ/OYjVCRmsASENAfyppjSa4OK4WA38MJHtqmetvppPTtd5cgAOvq6vC d4tg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=p0LAesio; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id h4-20020a170902f7c400b001c9de48fc7asi6044935plw.170.2023.11.05.08.09.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Nov 2023 08:09:38 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=p0LAesio; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id A45548057DA1; Sun, 5 Nov 2023 08:09:29 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229725AbjKEQIp (ORCPT + 34 others); Sun, 5 Nov 2023 11:08:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46190 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229709AbjKEQIm (ORCPT ); Sun, 5 Nov 2023 11:08:42 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 38E53FB for ; Sun, 5 Nov 2023 08:08:39 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0B9CEC433C8; Sun, 5 Nov 2023 16:08:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1699200518; bh=osFvjJGUhbNP9wL7RPDmpl5S4Pn7UmIryjWDNJIazD4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=p0LAesiolPSvBHlEMDUci5LTxD0Q8ICVx8zS9GXRprP8GAdxY295BIGVK5ABpUj76 LFgenEkWYXi9xB/xF7ETH9+700jO8RiLeKbdwnrjYBdwFalaw4To6njLNHOP0bNrzC PuwWUWNiplwQElnpl0qLs/dsDL5/9DxQ3ePrl7ld+jDDkk3iKbeIWd+aqcyDlrymOW WD2jGSQB8qfRGHP5k/Dqf76m49hs2wJJvplamIG+PARcR0pAFMQQHA92DEjdKyQS0B SI3VQp85ybSgVNIIfjBFQ+Mn5xKNqYaO+Sw/ZyGiue2f5qeEBh6n5SF2QzE8ecECH6 zQrQdNv2DnJdQ== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [RFC PATCH 10/32] function_graph: Have the instances use their own ftrace_ops for filtering Date: Mon, 6 Nov 2023 01:08:32 +0900 Message-Id: <169920051199.482486.17674190105884047734.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169920038849.482486.15796387219966662967.stgit@devnote2> References: <169920038849.482486.15796387219966662967.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Sun, 05 Nov 2023 08:09:29 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781740945374515566 X-GMAIL-MSGID: 1781740945374515566 From: Steven Rostedt (VMware) Allow for instances to have their own ftrace_ops part of the fgraph_ops that makes the funtion_graph tracer filter on the set_ftrace_filter file of the instance and not the top instance. Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- include/linux/ftrace.h | 1 + kernel/trace/fgraph.c | 60 +++++++++++++++++++++------------- kernel/trace/ftrace.c | 6 ++- kernel/trace/trace.h | 16 +++++---- kernel/trace/trace_functions.c | 2 + kernel/trace/trace_functions_graph.c | 8 +++-- 6 files changed, 58 insertions(+), 35 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 7fd044ae3da5..9dab365c6023 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -1044,6 +1044,7 @@ extern int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace, struct fgraph struct fgraph_ops { trace_func_graph_ent_t entryfunc; trace_func_graph_ret_t retfunc; + struct ftrace_ops ops; /* for the hash lists */ void *private; }; diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index 1e8c17f70b84..0642f3281b64 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -17,14 +17,6 @@ #include "ftrace_internal.h" #include "trace.h" -#ifdef CONFIG_DYNAMIC_FTRACE -#define ASSIGN_OPS_HASH(opsname, val) \ - .func_hash = val, \ - .local_hash.regex_lock = __MUTEX_INITIALIZER(opsname.local_hash.regex_lock), -#else -#define ASSIGN_OPS_HASH(opsname, val) -#endif - #define FGRAPH_RET_SIZE sizeof(struct ftrace_ret_stack) #define FGRAPH_RET_INDEX (FGRAPH_RET_SIZE / sizeof(long)) @@ -338,9 +330,6 @@ int function_graph_enter(unsigned long ret, unsigned long func, return -EBUSY; #endif - if (!ftrace_ops_test(&global_ops, func, NULL)) - return -EBUSY; - trace.func = func; trace.depth = ++current->curr_ret_depth; @@ -361,7 +350,8 @@ int function_graph_enter(unsigned long ret, unsigned long func, atomic_inc(¤t->trace_overrun); break; } - if (fgraph_array[i]->entryfunc(&trace, fgraph_array[i])) { + if (ftrace_ops_test(&gops->ops, func, NULL) && + gops->entryfunc(&trace, gops)) { offset = current->curr_ret_stack; /* Check the top level stored word */ type = get_fgraph_type(current, offset - 1); @@ -656,17 +646,25 @@ unsigned long ftrace_graph_ret_addr(struct task_struct *task, int *idx, } #endif /* HAVE_FUNCTION_GRAPH_RET_ADDR_PTR */ -static struct ftrace_ops graph_ops = { - .func = ftrace_graph_func, - .flags = FTRACE_OPS_FL_INITIALIZED | - FTRACE_OPS_FL_PID | - FTRACE_OPS_GRAPH_STUB, +void fgraph_init_ops(struct ftrace_ops *dst_ops, + struct ftrace_ops *src_ops) +{ + dst_ops->func = ftrace_stub; + dst_ops->flags = FTRACE_OPS_FL_PID | FTRACE_OPS_FL_STUB; + #ifdef FTRACE_GRAPH_TRAMP_ADDR - .trampoline = FTRACE_GRAPH_TRAMP_ADDR, + dst_ops->trampoline = FTRACE_GRAPH_TRAMP_ADDR; /* trampoline_size is only needed for dynamically allocated tramps */ #endif - ASSIGN_OPS_HASH(graph_ops, &global_ops.local_hash) -}; + +#ifdef CONFIG_DYNAMIC_FTRACE + if (src_ops) { + dst_ops->func_hash = &src_ops->local_hash; + mutex_init(&dst_ops->local_hash.regex_lock); + dst_ops->flags |= FTRACE_OPS_FL_INITIALIZED; + } +#endif +} void ftrace_graph_sleep_time_control(bool enable) { @@ -871,11 +869,20 @@ static int start_graph_tracing(void) int register_ftrace_graph(struct fgraph_ops *gops) { + int command = 0; int ret = 0; int i; mutex_lock(&ftrace_lock); + if (!gops->ops.func) { + gops->ops.flags |= FTRACE_OPS_FL_STUB; + gops->ops.func = ftrace_stub; +#ifdef FTRACE_GRAPH_TRAMP_ADDR + gops->ops.trampoline = FTRACE_GRAPH_TRAMP_ADDR; +#endif + } + if (!fgraph_array[0]) { /* The array must always have real data on it */ for (i = 0; i < FGRAPH_ARRAY_SIZE; i++) { @@ -912,9 +919,10 @@ int register_ftrace_graph(struct fgraph_ops *gops) */ ftrace_graph_return = return_run; ftrace_graph_entry = entry_run; - - ret = ftrace_startup(&graph_ops, FTRACE_START_FUNC_RET); + command = FTRACE_START_FUNC_RET; } + + ret = ftrace_startup(&gops->ops, command); out: mutex_unlock(&ftrace_lock); return ret; @@ -922,6 +930,7 @@ int register_ftrace_graph(struct fgraph_ops *gops) void unregister_ftrace_graph(struct fgraph_ops *gops) { + int command = 0; int i; mutex_lock(&ftrace_lock); @@ -944,10 +953,15 @@ void unregister_ftrace_graph(struct fgraph_ops *gops) } ftrace_graph_active--; + + if (!ftrace_graph_active) + command = FTRACE_STOP_FUNC_RET; + + ftrace_shutdown(&gops->ops, command); + if (!ftrace_graph_active) { ftrace_graph_return = ftrace_stub_graph; ftrace_graph_entry = ftrace_graph_entry_stub; - ftrace_shutdown(&graph_ops, FTRACE_STOP_FUNC_RET); unregister_pm_notifier(&ftrace_suspend_notifier); unregister_trace_sched_switch(ftrace_graph_probe_sched_switch, NULL); } diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 83fbfb7b48f8..c4cc2a9d0047 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -3050,6 +3050,8 @@ int ftrace_startup(struct ftrace_ops *ops, int command) if (unlikely(ftrace_disabled)) return -ENODEV; + ftrace_ops_init(ops); + ret = __register_ftrace_function(ops); if (ret) return ret; @@ -7319,7 +7321,7 @@ __init void ftrace_init_global_array_ops(struct trace_array *tr) tr->ops = &global_ops; tr->ops->private = tr; ftrace_init_trace_array(tr); - init_array_fgraph_ops(tr); + init_array_fgraph_ops(tr, tr->ops); } void ftrace_init_array_ops(struct trace_array *tr, ftrace_func_t func) @@ -8051,7 +8053,7 @@ static int register_ftrace_function_nolock(struct ftrace_ops *ops) */ int register_ftrace_function(struct ftrace_ops *ops) { - int ret; + int ret = -1; lock_direct_mutex(); ret = prepare_direct_functions_for_ipmodify(ops); diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h index 73fe789ba143..531cfdb44911 100644 --- a/kernel/trace/trace.h +++ b/kernel/trace/trace.h @@ -886,8 +886,8 @@ extern int __trace_graph_entry(struct trace_array *tr, extern void __trace_graph_return(struct trace_array *tr, struct ftrace_graph_ret *trace, unsigned int trace_ctx); -extern void init_array_fgraph_ops(struct trace_array *tr); -extern int allocate_fgraph_ops(struct trace_array *tr); +extern void init_array_fgraph_ops(struct trace_array *tr, struct ftrace_ops *ops); +extern int allocate_fgraph_ops(struct trace_array *tr, struct ftrace_ops *ops); extern void free_fgraph_ops(struct trace_array *tr); #ifdef CONFIG_DYNAMIC_FTRACE @@ -970,6 +970,7 @@ static inline int ftrace_graph_notrace_addr(unsigned long addr) preempt_enable_notrace(); return ret; } + #else static inline int ftrace_graph_addr(struct ftrace_graph_ent *trace) { @@ -995,18 +996,19 @@ static inline bool ftrace_graph_ignore_func(struct ftrace_graph_ent *trace) (fgraph_max_depth && trace->depth >= fgraph_max_depth); } +void fgraph_init_ops(struct ftrace_ops *dst_ops, + struct ftrace_ops *src_ops); + #else /* CONFIG_FUNCTION_GRAPH_TRACER */ static inline enum print_line_t print_graph_function_flags(struct trace_iterator *iter, u32 flags) { return TRACE_TYPE_UNHANDLED; } -static inline void init_array_fgraph_ops(struct trace_array *tr) { } -static inline int allocate_fgraph_ops(struct trace_array *tr) -{ - return 0; -} static inline void free_fgraph_ops(struct trace_array *tr) { } +/* ftrace_ops may not be defined */ +#define init_array_fgraph_ops(tr, ops) do { } while (0) +#define allocate_fgraph_ops(tr, ops) ({ 0; }) #endif /* CONFIG_FUNCTION_GRAPH_TRACER */ extern struct list_head ftrace_pids; diff --git a/kernel/trace/trace_functions.c b/kernel/trace/trace_functions.c index 8e8da0d0ee52..13bf2415245d 100644 --- a/kernel/trace/trace_functions.c +++ b/kernel/trace/trace_functions.c @@ -91,7 +91,7 @@ int ftrace_create_function_files(struct trace_array *tr, if (!tr->ops) return -EINVAL; - ret = allocate_fgraph_ops(tr); + ret = allocate_fgraph_ops(tr, tr->ops); if (ret) { kfree(tr->ops); return ret; diff --git a/kernel/trace/trace_functions_graph.c b/kernel/trace/trace_functions_graph.c index 9ccc904a7703..7f30652f0e97 100644 --- a/kernel/trace/trace_functions_graph.c +++ b/kernel/trace/trace_functions_graph.c @@ -288,7 +288,7 @@ static struct fgraph_ops funcgraph_ops = { .retfunc = &trace_graph_return, }; -int allocate_fgraph_ops(struct trace_array *tr) +int allocate_fgraph_ops(struct trace_array *tr, struct ftrace_ops *ops) { struct fgraph_ops *gops; @@ -301,6 +301,9 @@ int allocate_fgraph_ops(struct trace_array *tr) tr->gops = gops; gops->private = tr; + + fgraph_init_ops(&gops->ops, ops); + return 0; } @@ -309,10 +312,11 @@ void free_fgraph_ops(struct trace_array *tr) kfree(tr->gops); } -__init void init_array_fgraph_ops(struct trace_array *tr) +__init void init_array_fgraph_ops(struct trace_array *tr, struct ftrace_ops *ops) { tr->gops = &funcgraph_ops; funcgraph_ops.private = tr; + fgraph_init_ops(&tr->gops->ops, ops); } static int graph_trace_init(struct trace_array *tr) From patchwork Sun Nov 5 16:08:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 161667 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp2197094vqu; Sun, 5 Nov 2023 08:09:29 -0800 (PST) X-Google-Smtp-Source: AGHT+IG8mRcRykSUjifh8BpQgvmhvTai8SGEr43mAh1s03TRMcrWpTchtgiqmKjH5hMkId1YLjYQ X-Received: by 2002:a05:6e02:1d8a:b0:359:58ee:f925 with SMTP id h10-20020a056e021d8a00b0035958eef925mr11631574ila.12.1699200569161; Sun, 05 Nov 2023 08:09:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699200569; cv=none; d=google.com; s=arc-20160816; b=Rs//1C2H1dPeZWAgASlR85f6kK0dqQzJZHAVQr8mcrKLG4n8BVbLB2Emn6ozwOrkYs dErSl6FH62d6DV0Pc7+akFAliXb7s1vaFK3/SJogPsdK76JChptJXDMAdbdmAPpTy8/v olU2Uq0aPmvtYpFmMD53otXeROeLWZmW93QMVitNp2+HhLz2EHO3ztYMxNe+NaZ0pnBo PgDKsimUdpQ5sRZnPluAHTgLBnIZj0l/QQ80n7A0CrATV51DbKct0WOTXjmD9E/ItVe4 N5ZCXHBOhQcHsreSe2nH55rEmggLZpwlcRdgu5Ue54gpcyGb8FEuKbFbV2hPvKCsZoAm s56Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=ZNejQ3rA9pnWqCHPY/4PT7WzKIUij4GVB6IrytBl1XQ=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=MYtU4RKJ43UuiXswMiq5T3RlrFw7hVezy67J1jCEwall+8Q10rdgIGurwLIbnCsO71 ODUWPCPN5t+WTR9pR0BpKacKJTVh9AoHJRQltusadfkC79hYEcDURyifhYUtSgG1C0Ls bPaS0/jIbcNt5K49odDaFvYYxQATEdtCYuG3XB+e43sMWox1W+/y405SLzuO19LsTqem S37qwgC37KP+cKJ7/rpYhzMJfEMEHZLVLUuDFf22KMh1e4HuMsliqqEPTy/cRlc2KwGK Fc8SzreTU+PZJPLxMfe/oAb8wtye2mxlN2NtklkwkIGlyEc/mKBRVqx0prW3B1RHKifK O+iA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Svb+pP2N; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id o13-20020a056a0015cd00b006c382648737si5192582pfu.115.2023.11.05.08.09.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Nov 2023 08:09:29 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Svb+pP2N; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 7647C8057DA1; Sun, 5 Nov 2023 08:09:23 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229709AbjKEQJC (ORCPT + 34 others); Sun, 5 Nov 2023 11:09:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57054 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229954AbjKEQIy (ORCPT ); Sun, 5 Nov 2023 11:08:54 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 802E0D4C for ; Sun, 5 Nov 2023 08:08:51 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 46450C433C7; Sun, 5 Nov 2023 16:08:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1699200531; bh=ppDHV4sYM4AQxjT5p82RNOMWNxUOZsOt9Rrtuz4qMjU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Svb+pP2NmlqJSS5LWHPXMG+rNPvGGXxEISxfzXxdC+/yezWosTYDxi+Spq75srogn u9KKX7Z/L16jEjsebrwXaMngM3MUsGjPm92uPRgnVHqrTIfmwT6eWFXT9X3ATdioDZ /15aSpFqshZjCNdznI/Rx97EWDHbLcdslCeBAppQimy5d/sDVOw6okhnCAibqnXWt5 ZACfxzQK/hpWPt34ZLmGI4PWisw9bjde7fX3A0hZP1lhaBDqFmWBtpjkonQdgKs+f1 eU+pQO/wNJgsmon7wa/iCy0mfbqDdrOa3lcFkE+ETbIR8Rrfu5M/TiEUeAJbbuARWi yeZyVm1R6Io5Q== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [RFC PATCH 11/32] function_graph: Add "task variables" per task for fgraph_ops Date: Mon, 6 Nov 2023 01:08:44 +0900 Message-Id: <169920052428.482486.4211809175393725858.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169920038849.482486.15796387219966662967.stgit@devnote2> References: <169920038849.482486.15796387219966662967.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Sun, 05 Nov 2023 08:09:23 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781740935992560845 X-GMAIL-MSGID: 1781740935992560845 From: Steven Rostedt (VMware) Add a "task variables" array on the tasks shadow ret_stack that is the size of longs for each possible registered fgraph_ops. That's a total of 16, taking up 8 * 16 = 128 bytes (out of a page size 4k). This will allow for fgraph_ops to do specific features on a per task basis having a way to maintain state for each task. Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- include/linux/ftrace.h | 2 + kernel/trace/fgraph.c | 73 +++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 9dab365c6023..31ad20e78522 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -1046,6 +1046,7 @@ struct fgraph_ops { trace_func_graph_ret_t retfunc; struct ftrace_ops ops; /* for the hash lists */ void *private; + int idx; }; /* @@ -1084,6 +1085,7 @@ ftrace_graph_get_ret_stack(struct task_struct *task, int idx); unsigned long ftrace_graph_ret_addr(struct task_struct *task, int *idx, unsigned long ret, unsigned long *retp); +unsigned long *fgraph_get_task_var(struct fgraph_ops *gops); /* * Sometimes we don't want to trace a function with the function diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index 0642f3281b64..748896e320e2 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -92,10 +92,18 @@ enum { #define SHADOW_STACK_SIZE (PAGE_SIZE) #define SHADOW_STACK_INDEX (SHADOW_STACK_SIZE / sizeof(long)) /* Leave on a buffer at the end */ -#define SHADOW_STACK_MAX_INDEX (SHADOW_STACK_INDEX - (FGRAPH_RET_INDEX + 1)) +#define SHADOW_STACK_MAX_INDEX \ + (SHADOW_STACK_INDEX - (FGRAPH_RET_INDEX + 1 + FGRAPH_ARRAY_SIZE)) #define RET_STACK(t, index) ((struct ftrace_ret_stack *)(&(t)->ret_stack[index])) +/* + * Each fgraph_ops has a reservered unsigned long at the end (top) of the + * ret_stack to store task specific state. + */ +#define SHADOW_STACK_TASK_VARS(ret_stack) \ + ((unsigned long *)(&(ret_stack)[SHADOW_STACK_INDEX - FGRAPH_ARRAY_SIZE])) + DEFINE_STATIC_KEY_FALSE(kill_ftrace_graph); int ftrace_graph_active; @@ -132,6 +140,44 @@ static void return_run(struct ftrace_graph_ret *trace, struct fgraph_ops *ops) return; } +static void ret_stack_set_task_var(struct task_struct *t, int idx, long val) +{ + unsigned long *gvals = SHADOW_STACK_TASK_VARS(t->ret_stack); + + gvals[idx] = val; +} + +static unsigned long * +ret_stack_get_task_var(struct task_struct *t, int idx) +{ + unsigned long *gvals = SHADOW_STACK_TASK_VARS(t->ret_stack); + + return &gvals[idx]; +} + +static void ret_stack_init_task_vars(unsigned long *ret_stack) +{ + unsigned long *gvals = SHADOW_STACK_TASK_VARS(ret_stack); + + memset(gvals, 0, sizeof(*gvals) * FGRAPH_ARRAY_SIZE); +} + +/** + * fgraph_get_task_var - retrieve a task specific state variable + * @gops: The ftrace_ops that owns the task specific variable + * + * Every registered fgraph_ops has a task state variable + * reserved on the task's ret_stack. This function returns the + * address to that variable. + * + * Returns the address to the fgraph_ops @gops tasks specific + * unsigned long variable. + */ +unsigned long *fgraph_get_task_var(struct fgraph_ops *gops) +{ + return ret_stack_get_task_var(current, gops->idx); +} + /* * @offset: The index into @t->ret_stack to find the ret_stack entry * @index: Where to place the index into @t->ret_stack of that entry @@ -709,6 +755,7 @@ static int alloc_retstack_tasklist(unsigned long **ret_stack_list) if (t->ret_stack == NULL) { atomic_set(&t->trace_overrun, 0); + ret_stack_init_task_vars(ret_stack_list[start]); t->curr_ret_stack = 0; t->curr_ret_depth = -1; /* Make sure the tasks see the 0 first: */ @@ -769,6 +816,7 @@ static void graph_init_task(struct task_struct *t, unsigned long *ret_stack) { atomic_set(&t->trace_overrun, 0); + ret_stack_init_task_vars(ret_stack); t->ftrace_timestamp = 0; t->curr_ret_stack = 0; t->curr_ret_depth = -1; @@ -867,6 +915,24 @@ static int start_graph_tracing(void) return ret; } +static void init_task_vars(int idx) +{ + struct task_struct *g, *t; + int cpu; + + for_each_online_cpu(cpu) { + if (idle_task(cpu)->ret_stack) + ret_stack_set_task_var(idle_task(cpu), idx, 0); + } + + read_lock(&tasklist_lock); + for_each_process_thread(g, t) { + if (t->ret_stack) + ret_stack_set_task_var(t, idx, 0); + } + read_unlock(&tasklist_lock); +} + int register_ftrace_graph(struct fgraph_ops *gops) { int command = 0; @@ -903,6 +969,7 @@ int register_ftrace_graph(struct fgraph_ops *gops) fgraph_array[i] = gops; if (i + 1 > fgraph_array_cnt) fgraph_array_cnt = i + 1; + gops->idx = i; ftrace_graph_active++; @@ -920,6 +987,8 @@ int register_ftrace_graph(struct fgraph_ops *gops) ftrace_graph_return = return_run; ftrace_graph_entry = entry_run; command = FTRACE_START_FUNC_RET; + } else { + init_task_vars(gops->idx); } ret = ftrace_startup(&gops->ops, command); @@ -944,6 +1013,8 @@ void unregister_ftrace_graph(struct fgraph_ops *gops) if (i >= fgraph_array_cnt) goto out; + WARN_ON_ONCE(gops->idx != i); + fgraph_array[i] = &fgraph_stub; if (i + 1 == fgraph_array_cnt) { for (; i >= 0; i--) From patchwork Sun Nov 5 16:08:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 161669 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp2197225vqu; Sun, 5 Nov 2023 08:09:48 -0800 (PST) X-Google-Smtp-Source: AGHT+IEXRvE9kOy/q5dNY+K640OMBPxQn36GjJEPSIoT75xn2LfArHSL+q3AD7ORtdcJ4lCCuXNf X-Received: by 2002:a05:6870:1ecd:b0:1e9:96c6:e040 with SMTP id pc13-20020a0568701ecd00b001e996c6e040mr32035400oab.32.1699200588692; Sun, 05 Nov 2023 08:09:48 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699200588; cv=none; d=google.com; s=arc-20160816; b=iwCBUnEj6Z8cnyzjhH1kxaV5iZUPlslNxn0eLiPBFP77/oksumMG5PXY6gjdIjFoSI CTVboyY4RjyzNBErMmMox2RtAG/P3zT+MBQZxtsDgSq2W9bf9ATdcO0AWkhXKPiky5Fv gCrcXW12ReNVJBtgra87HauHF2FQIkb7ZckN5db6ac8CvPNVDY1WlKne9/LPsocG83Xm nMIBB+iSZM3JnM3WmqcTzi3MQ4QCPlb2xviz00UmfdTdOdM3JekgMhqItGhNnJqvSRIN Y4WXCZf70tDMvJfzJE02CRe1VOVYEErUgSLwpwSK9SXkxntU+NIBhKqcrcqJi3Klx67p D7Gg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=300VZ4kbDy7VWJyGLpWIHzgXCvlOxR8XIrmEWbSERCE=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=sI+Fx58Fd9z7IZs58ahIP1ukQ+DwF939ZDoUpUd0oH/4VWGSDIIARs6xiyaj9r5pTo pxMiRIPDcblU3cVtdWtqRUeSEe3a/0em/Jz9U5Vijq9U0IbAHnnYSACW3twb6XuP1D7z TLMDLfOJrj5sFwc9dPw9jHnInFu6xLXpsK6fScWTQeEC30BTliLugF5E4Koi8PCuvWHx z+w/Oa65/82NfKfnrsolGfrEqz0Id17mHrczZyJOcSEgM7E1I2JB1w2LvkEYe4USgT0J B2ROcpRWOz8AYenNMiVxTYroE3Nn+wfs7GogLgu/nPakP0ogUIf4LWnbPYkjUOfLCTbD GumA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=jfsNrOQP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id w63-20020a638242000000b005b8aef54248si6043112pgd.411.2023.11.05.08.09.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Nov 2023 08:09:48 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=jfsNrOQP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 489658053625; Sun, 5 Nov 2023 08:09:46 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229489AbjKEQJN (ORCPT + 34 others); Sun, 5 Nov 2023 11:09:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40122 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229842AbjKEQJH (ORCPT ); Sun, 5 Nov 2023 11:09:07 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AC6DBD42 for ; Sun, 5 Nov 2023 08:09:03 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 742F0C433C8; Sun, 5 Nov 2023 16:08:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1699200543; bh=GQeSftC/Qxj4nC/0Uav+AB7wqH65R3Pkc3jJZh9Ujr4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jfsNrOQPEF88HojdUsZT50osw4YAstrLW1a203HgGfKCd3DnVqOeilwS+DFvSWAaY tD1Ic4vJ8/pgT+G+I3Le/u5GysV/2P+gOXhg4sPvLDMsxCwFQrdjviHX0Onr5kKSLp 4XD53Y0qMeTjPILkOV0F6iT1+NG8/Fse6+6OYrkYL+3JLIKcH1d3n/SRrPhS1RxJU5 fdAvFag3cGcGR0U2N6QGa6x9VPYwHhv9gxkUFW84zwo5bi/vf0pczHX2wZ1mJ1cB3P dkMWGvLXexqSHYHQjxaSWLBbB7anR+fD/lqtynlDyXF069Hv/Kg5TV9Lv21OQEWDYE HEEiARNVUkhiA== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [RFC PATCH 12/32] function_graph: Move set_graph_function tests to shadow stack global var Date: Mon, 6 Nov 2023 01:08:56 +0900 Message-Id: <169920053652.482486.2956196897000211800.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169920038849.482486.15796387219966662967.stgit@devnote2> References: <169920038849.482486.15796387219966662967.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Sun, 05 Nov 2023 08:09:46 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781740956556256606 X-GMAIL-MSGID: 1781740956556256606 From: Steven Rostedt (VMware) The use of the task->trace_recursion for the logic used for the set_graph_funnction was a bit of an abuse of that variable. Now that there exists global vars that are per stack for registered graph traces, use that instead. Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- include/linux/trace_recursion.h | 5 +---- kernel/trace/trace.h | 32 +++++++++++++++++++++----------- kernel/trace/trace_functions_graph.c | 6 +++--- kernel/trace/trace_irqsoff.c | 4 ++-- kernel/trace/trace_sched_wakeup.c | 4 ++-- 5 files changed, 29 insertions(+), 22 deletions(-) diff --git a/include/linux/trace_recursion.h b/include/linux/trace_recursion.h index d48cd92d2364..2efd5ec46d7f 100644 --- a/include/linux/trace_recursion.h +++ b/include/linux/trace_recursion.h @@ -44,9 +44,6 @@ enum { */ TRACE_IRQ_BIT, - /* Set if the function is in the set_graph_function file */ - TRACE_GRAPH_BIT, - /* * In the very unlikely case that an interrupt came in * at a start of graph tracing, and we want to trace @@ -60,7 +57,7 @@ enum { * that preempted a softirq start of a function that * preempted normal context!!!! Luckily, it can't be * greater than 3, so the next two bits are a mask - * of what the depth is when we set TRACE_GRAPH_BIT + * of what the depth is when we set TRACE_GRAPH_FL */ TRACE_GRAPH_DEPTH_START_BIT, diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h index 531cfdb44911..219520621091 100644 --- a/kernel/trace/trace.h +++ b/kernel/trace/trace.h @@ -890,11 +890,16 @@ extern void init_array_fgraph_ops(struct trace_array *tr, struct ftrace_ops *ops extern int allocate_fgraph_ops(struct trace_array *tr, struct ftrace_ops *ops); extern void free_fgraph_ops(struct trace_array *tr); +enum { + TRACE_GRAPH_FL = 1, +}; + #ifdef CONFIG_DYNAMIC_FTRACE extern struct ftrace_hash __rcu *ftrace_graph_hash; extern struct ftrace_hash __rcu *ftrace_graph_notrace_hash; -static inline int ftrace_graph_addr(struct ftrace_graph_ent *trace) +static inline int +ftrace_graph_addr(unsigned long *task_var, struct ftrace_graph_ent *trace) { unsigned long addr = trace->func; int ret = 0; @@ -916,12 +921,11 @@ static inline int ftrace_graph_addr(struct ftrace_graph_ent *trace) } if (ftrace_lookup_ip(hash, addr)) { - /* * This needs to be cleared on the return functions * when the depth is zero. */ - trace_recursion_set(TRACE_GRAPH_BIT); + *task_var |= TRACE_GRAPH_FL; trace_recursion_set_depth(trace->depth); /* @@ -941,11 +945,14 @@ static inline int ftrace_graph_addr(struct ftrace_graph_ent *trace) return ret; } -static inline void ftrace_graph_addr_finish(struct ftrace_graph_ret *trace) +static inline void +ftrace_graph_addr_finish(struct fgraph_ops *gops, struct ftrace_graph_ret *trace) { - if (trace_recursion_test(TRACE_GRAPH_BIT) && + unsigned long *task_var = fgraph_get_task_var(gops); + + if ((*task_var & TRACE_GRAPH_FL) && trace->depth == trace_recursion_depth()) - trace_recursion_clear(TRACE_GRAPH_BIT); + *task_var &= ~TRACE_GRAPH_FL; } static inline int ftrace_graph_notrace_addr(unsigned long addr) @@ -972,7 +979,7 @@ static inline int ftrace_graph_notrace_addr(unsigned long addr) } #else -static inline int ftrace_graph_addr(struct ftrace_graph_ent *trace) +static inline int ftrace_graph_addr(unsigned long *task_var, struct ftrace_graph_ent *trace) { return 1; } @@ -981,17 +988,20 @@ static inline int ftrace_graph_notrace_addr(unsigned long addr) { return 0; } -static inline void ftrace_graph_addr_finish(struct ftrace_graph_ret *trace) +static inline void ftrace_graph_addr_finish(struct fgraph_ops *gops, struct ftrace_graph_ret *trace) { } #endif /* CONFIG_DYNAMIC_FTRACE */ extern unsigned int fgraph_max_depth; -static inline bool ftrace_graph_ignore_func(struct ftrace_graph_ent *trace) +static inline bool +ftrace_graph_ignore_func(struct fgraph_ops *gops, struct ftrace_graph_ent *trace) { + unsigned long *task_var = fgraph_get_task_var(gops); + /* trace it when it is-nested-in or is a function enabled. */ - return !(trace_recursion_test(TRACE_GRAPH_BIT) || - ftrace_graph_addr(trace)) || + return !((*task_var & TRACE_GRAPH_FL) || + ftrace_graph_addr(task_var, trace)) || (trace->depth < 0) || (fgraph_max_depth && trace->depth >= fgraph_max_depth); } diff --git a/kernel/trace/trace_functions_graph.c b/kernel/trace/trace_functions_graph.c index 7f30652f0e97..66cce73e94f8 100644 --- a/kernel/trace/trace_functions_graph.c +++ b/kernel/trace/trace_functions_graph.c @@ -160,7 +160,7 @@ int trace_graph_entry(struct ftrace_graph_ent *trace, if (!ftrace_trace_task(tr)) return 0; - if (ftrace_graph_ignore_func(trace)) + if (ftrace_graph_ignore_func(gops, trace)) return 0; if (ftrace_graph_ignore_irqs()) @@ -247,7 +247,7 @@ void trace_graph_return(struct ftrace_graph_ret *trace, long disabled; int cpu; - ftrace_graph_addr_finish(trace); + ftrace_graph_addr_finish(gops, trace); if (trace_recursion_test(TRACE_GRAPH_NOTRACE_BIT)) { trace_recursion_clear(TRACE_GRAPH_NOTRACE_BIT); @@ -269,7 +269,7 @@ void trace_graph_return(struct ftrace_graph_ret *trace, static void trace_graph_thresh_return(struct ftrace_graph_ret *trace, struct fgraph_ops *gops) { - ftrace_graph_addr_finish(trace); + ftrace_graph_addr_finish(gops, trace); if (trace_recursion_test(TRACE_GRAPH_NOTRACE_BIT)) { trace_recursion_clear(TRACE_GRAPH_NOTRACE_BIT); diff --git a/kernel/trace/trace_irqsoff.c b/kernel/trace/trace_irqsoff.c index 5478f4c4f708..fce064e20570 100644 --- a/kernel/trace/trace_irqsoff.c +++ b/kernel/trace/trace_irqsoff.c @@ -184,7 +184,7 @@ static int irqsoff_graph_entry(struct ftrace_graph_ent *trace, unsigned int trace_ctx; int ret; - if (ftrace_graph_ignore_func(trace)) + if (ftrace_graph_ignore_func(gops, trace)) return 0; /* * Do not trace a function if it's filtered by set_graph_notrace. @@ -214,7 +214,7 @@ static void irqsoff_graph_return(struct ftrace_graph_ret *trace, unsigned long flags; unsigned int trace_ctx; - ftrace_graph_addr_finish(trace); + ftrace_graph_addr_finish(gops, trace); if (!func_prolog_dec(tr, &data, &flags)) return; diff --git a/kernel/trace/trace_sched_wakeup.c b/kernel/trace/trace_sched_wakeup.c index 49bcc812652c..130ca7e7787e 100644 --- a/kernel/trace/trace_sched_wakeup.c +++ b/kernel/trace/trace_sched_wakeup.c @@ -120,7 +120,7 @@ static int wakeup_graph_entry(struct ftrace_graph_ent *trace, unsigned int trace_ctx; int ret = 0; - if (ftrace_graph_ignore_func(trace)) + if (ftrace_graph_ignore_func(gops, trace)) return 0; /* * Do not trace a function if it's filtered by set_graph_notrace. @@ -149,7 +149,7 @@ static void wakeup_graph_return(struct ftrace_graph_ret *trace, struct trace_array_cpu *data; unsigned int trace_ctx; - ftrace_graph_addr_finish(trace); + ftrace_graph_addr_finish(gops, trace); if (!func_prolog_preempt_disable(tr, &data, &trace_ctx)) return; From patchwork Sun Nov 5 16:09:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 161670 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp2197232vqu; Sun, 5 Nov 2023 08:09:50 -0800 (PST) X-Google-Smtp-Source: AGHT+IEVhKdTMQ4OdSHr/+xU5nUMXFpdRCeHnQoXPPitqZUT+ZfZX76dzCM6uSu2VYLufMXKZrOv X-Received: by 2002:a05:6e02:1526:b0:357:f72d:ad1d with SMTP id i6-20020a056e02152600b00357f72dad1dmr33397055ilu.30.1699200590016; Sun, 05 Nov 2023 08:09:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699200589; cv=none; d=google.com; s=arc-20160816; b=dgjlZYA5Vy38wTVoOGAxFTo9hR7almHS+yZG0qBYIFuvi/acyTuyjFoYxwF5anFQn3 FxLZQpNWp7SWgWaBD4e1PIt+lTot49zkaR7VF1UGWTEa5JJAOvyPb6Tta1Y25YkNzMp+ MyYtQC4gHU5hThT0oHXRTR/h5dPTPgOe/u9A4Dfsu0GB1OY0YsR1pz94/YvjGWtWbx2q vYeomzJ3LXJmJRy/iqzF6shsJVfClU6/8hyMQ2Tqkl+DRQcrglG+pnetyVSsn8tqXfau YJmWMgYqC0O8H/q+bedAh9KVR1czvDVsewYhfViN2tx7FCuV8+dj3zNyIw+Kr2vR3r7f Q6Gw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=kVEWPFRz5/+fGUs7XIJwNAZNJkae3qIxVCEwhsGv+nA=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=KHCAgVArpNP0XmdsCznIlz0+XYSLYP+2PH03afcwE9Cj56AG21MbcrdIFbvlFpwiRf dYkYfW/mDEJu9ndLLisAMTT+PdgWZwr1JQ1SqdQ+BvXT71l411uqyuT8YG6d38HYApBq LOXCppJFltsMlkdQa4gCygTGP9UZyTja1Wx3jrM0rFJEFztOsRUdl70g1PyPcRv0zvrD tNyF1ZLLFz9h7Lt3hYfCI+Q09mYUjJI7xr47hqjY8ZIM53NRUXrcfMDrjyXjz/5Fkxtx j1ZenmiBHgPDH2VPxql583mUrC545F7Lnt7Y2PEDzZeSugTCWzfOuRnIvGTcWGxy4rP+ GVkQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=r4Nkok4i; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id h185-20020a636cc2000000b005b96cdc0681si6084772pgc.381.2023.11.05.08.09.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Nov 2023 08:09:49 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=r4Nkok4i; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 6D8238057D8D; Sun, 5 Nov 2023 08:09:47 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229695AbjKEQJ1 (ORCPT + 34 others); Sun, 5 Nov 2023 11:09:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47400 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229894AbjKEQJT (ORCPT ); Sun, 5 Nov 2023 11:09:19 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C0B611BC for ; Sun, 5 Nov 2023 08:09:15 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A1A18C433C7; Sun, 5 Nov 2023 16:09:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1699200555; bh=76F6UbhxbZaSAsPQvjDLsi8NvSTD+j0mXXNs1M+k06E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=r4Nkok4i0G5ndXr3QBzG86wJTNKlWrVaghBQMvUb0dp30EwQRXBgtv66TMHYP8KtW LX9+guJ59ZjiSQkoCFG7Tn8Ffrk4zRYsclIVNR8stwjQtDe79vVRm8xfblry6JgYYH dluu+Pt7wi7C1EsRIygc57ZWFVNqxAIkQZQ+2sOi/y2RtIbgiG6OzFCopHzYaMkcoM oNAgyFF+S0JdMMPz8nsbLSiF8DrJ9yriXUvAn/3NMM1nvNYnRz8i1+fbgh7wItgf9C zE+cU776CFUM4yeDN+GKOsutVAbi/0UhLTiQmUJ7Je4xCBm1DTRF+XkeGOrqPnFpvx aylPTjxWt5I5w== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [RFC PATCH 13/32] function_graph: Move graph depth stored data to shadow stack global var Date: Mon, 6 Nov 2023 01:09:09 +0900 Message-Id: <169920054871.482486.6029981940887989219.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169920038849.482486.15796387219966662967.stgit@devnote2> References: <169920038849.482486.15796387219966662967.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Sun, 05 Nov 2023 08:09:47 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781740957701326974 X-GMAIL-MSGID: 1781740957701326974 From: Steven Rostedt (VMware) The use of the task->trace_recursion for the logic used for the function graph depth was a bit of an abuse of that variable. Now that there exists global vars that are per stack for registered graph traces, use that instead. Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- include/linux/trace_recursion.h | 29 ----------------------------- kernel/trace/trace.h | 34 ++++++++++++++++++++++++++++++++-- 2 files changed, 32 insertions(+), 31 deletions(-) diff --git a/include/linux/trace_recursion.h b/include/linux/trace_recursion.h index 2efd5ec46d7f..00e792bf148d 100644 --- a/include/linux/trace_recursion.h +++ b/include/linux/trace_recursion.h @@ -44,25 +44,6 @@ enum { */ TRACE_IRQ_BIT, - /* - * In the very unlikely case that an interrupt came in - * at a start of graph tracing, and we want to trace - * the function in that interrupt, the depth can be greater - * than zero, because of the preempted start of a previous - * trace. In an even more unlikely case, depth could be 2 - * if a softirq interrupted the start of graph tracing, - * followed by an interrupt preempting a start of graph - * tracing in the softirq, and depth can even be 3 - * if an NMI came in at the start of an interrupt function - * that preempted a softirq start of a function that - * preempted normal context!!!! Luckily, it can't be - * greater than 3, so the next two bits are a mask - * of what the depth is when we set TRACE_GRAPH_FL - */ - - TRACE_GRAPH_DEPTH_START_BIT, - TRACE_GRAPH_DEPTH_END_BIT, - /* * To implement set_graph_notrace, if this bit is set, we ignore * function graph tracing of called functions, until the return @@ -78,16 +59,6 @@ enum { #define trace_recursion_clear(bit) do { (current)->trace_recursion &= ~(1<<(bit)); } while (0) #define trace_recursion_test(bit) ((current)->trace_recursion & (1<<(bit))) -#define trace_recursion_depth() \ - (((current)->trace_recursion >> TRACE_GRAPH_DEPTH_START_BIT) & 3) -#define trace_recursion_set_depth(depth) \ - do { \ - current->trace_recursion &= \ - ~(3 << TRACE_GRAPH_DEPTH_START_BIT); \ - current->trace_recursion |= \ - ((depth) & 3) << TRACE_GRAPH_DEPTH_START_BIT; \ - } while (0) - #define TRACE_CONTEXT_BITS 4 #define TRACE_FTRACE_START TRACE_FTRACE_BIT diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h index 219520621091..8e5943d3d25c 100644 --- a/kernel/trace/trace.h +++ b/kernel/trace/trace.h @@ -892,8 +892,38 @@ extern void free_fgraph_ops(struct trace_array *tr); enum { TRACE_GRAPH_FL = 1, + + /* + * In the very unlikely case that an interrupt came in + * at a start of graph tracing, and we want to trace + * the function in that interrupt, the depth can be greater + * than zero, because of the preempted start of a previous + * trace. In an even more unlikely case, depth could be 2 + * if a softirq interrupted the start of graph tracing, + * followed by an interrupt preempting a start of graph + * tracing in the softirq, and depth can even be 3 + * if an NMI came in at the start of an interrupt function + * that preempted a softirq start of a function that + * preempted normal context!!!! Luckily, it can't be + * greater than 3, so the next two bits are a mask + * of what the depth is when we set TRACE_GRAPH_FL + */ + + TRACE_GRAPH_DEPTH_START_BIT, + TRACE_GRAPH_DEPTH_END_BIT, }; +static inline unsigned long ftrace_graph_depth(unsigned long *task_var) +{ + return (*task_var >> TRACE_GRAPH_DEPTH_START_BIT) & 3; +} + +static inline void ftrace_graph_set_depth(unsigned long *task_var, int depth) +{ + *task_var &= ~(3 << TRACE_GRAPH_DEPTH_START_BIT); + *task_var |= (depth & 3) << TRACE_GRAPH_DEPTH_START_BIT; +} + #ifdef CONFIG_DYNAMIC_FTRACE extern struct ftrace_hash __rcu *ftrace_graph_hash; extern struct ftrace_hash __rcu *ftrace_graph_notrace_hash; @@ -926,7 +956,7 @@ ftrace_graph_addr(unsigned long *task_var, struct ftrace_graph_ent *trace) * when the depth is zero. */ *task_var |= TRACE_GRAPH_FL; - trace_recursion_set_depth(trace->depth); + ftrace_graph_set_depth(task_var, trace->depth); /* * If no irqs are to be traced, but a set_graph_function @@ -951,7 +981,7 @@ ftrace_graph_addr_finish(struct fgraph_ops *gops, struct ftrace_graph_ret *trace unsigned long *task_var = fgraph_get_task_var(gops); if ((*task_var & TRACE_GRAPH_FL) && - trace->depth == trace_recursion_depth()) + trace->depth == ftrace_graph_depth(task_var)) *task_var &= ~TRACE_GRAPH_FL; } From patchwork Sun Nov 5 16:09:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 161671 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp2197236vqu; Sun, 5 Nov 2023 08:09:51 -0800 (PST) X-Google-Smtp-Source: AGHT+IH6Ld03HMmxzE1HCc0pOJkNSrfTRNPWMUAKVbsirFBL35T+e4s6butc2tu2lOY/R2QME+vp X-Received: by 2002:a05:6358:5e0c:b0:168:e898:fb7d with SMTP id q12-20020a0563585e0c00b00168e898fb7dmr20883810rwn.28.1699200590884; Sun, 05 Nov 2023 08:09:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699200590; cv=none; d=google.com; s=arc-20160816; b=srGkJzmej866T6ttfyHxswIbovRMs02iTh1UFD3UZgYotKjeW0P+LlG77w3Iqmz5Ua eId6yoSovD2cxPNfidTLfGPWwn1q578ow50Dz2y7QAgI/xKgu8lRHBJNFPhVCPra1YAY 9p8MdpBdtANYI2wf0JhBYrE5xZKGfu6/civDvPLpkWzCiOT8TJ9MYID3RTCT1TsZQxiz CPmGWFCUyCBZJufoMfSMiwO9r3qRmolD2HxNYe3y4h404FXpj5cHTxVAuVW8gNIXcvYI QGzP7rJj/uHiC54gESeEjizsyb6YbFkbpaHkIRS/9p2rOs5m+AuL6TTnybJd9ccMjHHM bziQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=l4+nrejjmjt+quOSeWblSaRKiBZpudQWmKBCVtfL5xM=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=OmB9whhbmGD3W6IchyY0oVl/kC8m4FRklGZir0WSzU5Bb9mR4lNrsVDhgWcKEFimYr xTmWWjrQnQOKfLy+roCo9tilABwicUcapjAMS6ntDuU8WYKqVOlIlRxVV6Av4qkDnTeN IorSE8ulnI1Xvw9GH2QOQf34+gZNibmgzf83nTGOdbIG9DufyqaXI9QByM4yEP1oGvei BISVuHPX0ul287NQcx/KxXrUj3IuUqoDNym0tAS9odY7p+HCno3DwdMArpGVKFPw9/Si q+XNqiJW6bU2yIGvXUyUXjXygxNroCuZfzwBuWR0sqHKrbGZ6a9M79XDBnGmhDGe+US8 jo/g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=hzsj6qVn; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from lipwig.vger.email (lipwig.vger.email. [2620:137:e000::3:3]) by mx.google.com with ESMTPS id i186-20020a6387c3000000b005b8817b3cb0si5965250pge.65.2023.11.05.08.09.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Nov 2023 08:09:50 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) client-ip=2620:137:e000::3:3; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=hzsj6qVn; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 8BEF38053620; Sun, 5 Nov 2023 08:09:48 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229913AbjKEQJf (ORCPT + 34 others); Sun, 5 Nov 2023 11:09:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47566 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229633AbjKEQJa (ORCPT ); Sun, 5 Nov 2023 11:09:30 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 32FBED77 for ; Sun, 5 Nov 2023 08:09:28 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E758DC433C7; Sun, 5 Nov 2023 16:09:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1699200567; bh=yEhjs0R0Qef2/83fXa4qLGdcMfbIL29IAkz328qo8AA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hzsj6qVnQRs/v+v15sUROz1C+2AWBJYj5dYYAzCNaDUDke8KQ8Anv4ESb9v0XeTyK dvtK9bMb7OvyK06lzfrQDZuhtFghRP2nct9W7yiSmaSfIBdf8s8JeThxQPqd4LpS2O eQS1zMDhzSy7NPdcaCzAQzBFU/Vd9UT+TwUrAQPjdgeAlJy8vXiqU6u2Ht58qYNMYk +hQkAYZQtaycBwQtr3SwW6g7j18R+e+iWqdFcIgTIY6DLMLUF95M4Vhc419VV8aREF JU9gO5G/mnl4hjWzLIC+efxxmdPaVxqYGTxxLkOsrhLQyxYt3A2rhQV66+THSQJdZZ avnpAvBjktNXg== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [RFC PATCH 14/32] function_graph: Move graph notrace bit to shadow stack global var Date: Mon, 6 Nov 2023 01:09:21 +0900 Message-Id: <169920056100.482486.10749804996506200504.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169920038849.482486.15796387219966662967.stgit@devnote2> References: <169920038849.482486.15796387219966662967.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Sun, 05 Nov 2023 08:09:48 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781740958457262343 X-GMAIL-MSGID: 1781740958457262343 From: Steven Rostedt (VMware) The use of the task->trace_recursion for the logic used for the function graph no-trace was a bit of an abuse of that variable. Now that there exists global vars that are per stack for registered graph traces, use that instead. Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- include/linux/trace_recursion.h | 7 ------- kernel/trace/trace.h | 9 +++++++++ kernel/trace/trace_functions_graph.c | 10 ++++++---- 3 files changed, 15 insertions(+), 11 deletions(-) diff --git a/include/linux/trace_recursion.h b/include/linux/trace_recursion.h index 00e792bf148d..cc11b0e9d220 100644 --- a/include/linux/trace_recursion.h +++ b/include/linux/trace_recursion.h @@ -44,13 +44,6 @@ enum { */ TRACE_IRQ_BIT, - /* - * To implement set_graph_notrace, if this bit is set, we ignore - * function graph tracing of called functions, until the return - * function is called to clear it. - */ - TRACE_GRAPH_NOTRACE_BIT, - /* Used to prevent recursion recording from recursing. */ TRACE_RECORD_RECURSION_BIT, }; diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h index 8e5943d3d25c..350c889c2f1e 100644 --- a/kernel/trace/trace.h +++ b/kernel/trace/trace.h @@ -911,8 +911,17 @@ enum { TRACE_GRAPH_DEPTH_START_BIT, TRACE_GRAPH_DEPTH_END_BIT, + + /* + * To implement set_graph_notrace, if this bit is set, we ignore + * function graph tracing of called functions, until the return + * function is called to clear it. + */ + TRACE_GRAPH_NOTRACE_BIT, }; +#define TRACE_GRAPH_NOTRACE (1 << TRACE_GRAPH_NOTRACE_BIT) + static inline unsigned long ftrace_graph_depth(unsigned long *task_var) { return (*task_var >> TRACE_GRAPH_DEPTH_START_BIT) & 3; diff --git a/kernel/trace/trace_functions_graph.c b/kernel/trace/trace_functions_graph.c index 66cce73e94f8..13d0387ac6a6 100644 --- a/kernel/trace/trace_functions_graph.c +++ b/kernel/trace/trace_functions_graph.c @@ -130,6 +130,7 @@ static inline int ftrace_graph_ignore_irqs(void) int trace_graph_entry(struct ftrace_graph_ent *trace, struct fgraph_ops *gops) { + unsigned long *task_var = fgraph_get_task_var(gops); struct trace_array *tr = gops->private; struct trace_array_cpu *data; unsigned long flags; @@ -138,7 +139,7 @@ int trace_graph_entry(struct ftrace_graph_ent *trace, int ret; int cpu; - if (trace_recursion_test(TRACE_GRAPH_NOTRACE_BIT)) + if (*task_var & TRACE_GRAPH_NOTRACE) return 0; /* @@ -149,7 +150,7 @@ int trace_graph_entry(struct ftrace_graph_ent *trace, * returning from the function. */ if (ftrace_graph_notrace_addr(trace->func)) { - trace_recursion_set(TRACE_GRAPH_NOTRACE_BIT); + *task_var |= TRACE_GRAPH_NOTRACE_BIT; /* * Need to return 1 to have the return called * that will clear the NOTRACE bit. @@ -240,6 +241,7 @@ void __trace_graph_return(struct trace_array *tr, void trace_graph_return(struct ftrace_graph_ret *trace, struct fgraph_ops *gops) { + unsigned long *task_var = fgraph_get_task_var(gops); struct trace_array *tr = gops->private; struct trace_array_cpu *data; unsigned long flags; @@ -249,8 +251,8 @@ void trace_graph_return(struct ftrace_graph_ret *trace, ftrace_graph_addr_finish(gops, trace); - if (trace_recursion_test(TRACE_GRAPH_NOTRACE_BIT)) { - trace_recursion_clear(TRACE_GRAPH_NOTRACE_BIT); + if (*task_var & TRACE_GRAPH_NOTRACE) { + *task_var &= ~TRACE_GRAPH_NOTRACE; return; } From patchwork Sun Nov 5 16:09:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 161674 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp2197474vqu; Sun, 5 Nov 2023 08:10:24 -0800 (PST) X-Google-Smtp-Source: AGHT+IFko1mA99jCv3dfOwJ60OVTtP7ssPVPifOz2fN+niEa6ikHGdZqqxLsyDC2FhdXpwwcc72h X-Received: by 2002:a05:6a21:19c:b0:17e:aa00:ca62 with SMTP id le28-20020a056a21019c00b0017eaa00ca62mr13064256pzb.17.1699200623764; Sun, 05 Nov 2023 08:10:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699200623; cv=none; d=google.com; s=arc-20160816; b=LoHt5mx2kNZ0OJE8P1ib5KcusxL2M2Z6mXXG//Mfn++48muLtPLQt9lOlHbXcotg4n 3fErNi27nc5jlwaWMXkbiqfa0IvgjMDatQuByD6JhXSO4wuok5nadXH3jiknHl0I4CIJ HeROCZQc87NxwZTktv7KyzzbXfIfT3FpUxaqtkLUNwYQYRyYiKsEGzUgfjPIAFSI7deQ /WYyKBh+D7ka+peGx2OSgnLuKyXbgnOK8gQQSM4KKf3wkMQSvt+u7cp7/pv3FAkBaEvZ 7a7/e24Lwo+ClJ5okSMbs5asRYtlxUdnzY1X5OivSAiOcHPhtEJLDwEbPBx4bdvEVypO upOw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=s8BhX3V7zQvbd7De4iY+YKmWvMjvIQEMp3EsrmBV6Jw=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=Qt3HV9fOWqaA2juzM8gavxpVeg677fO+m4T4W2BUQRDLoNkCkaXKV809hoEkY3KkJ+ 1CyixUYzVSNC01SPKzse91xcklydNNA+vK2jM7dPq9TFi7bJvdXl1t2ZEjSLvS7Fd2AQ 2MLlMXb8LEmpAVsq7rirw7XT19B3nwISoBULa1XCJ1cOMZGQFqQNd4LrnIBGCuTz1H55 QTcBArnd4YmZBLkaCGyVst1tK0jdTUUMkl/nUEuhEZC902unLe7ogExsPrV1SlLBWFgj rqiglyaIIUZyOP+jumTj3C5imW/3euk6WGPrSIdABnYktmz+munl7D6WTrXqKzSnq8Qz rY8g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="T//8hR6p"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id bd33-20020a056a0027a100b006b18ecd03dasi6392415pfb.264.2023.11.05.08.10.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Nov 2023 08:10:23 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="T//8hR6p"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id E1C4A8057B03; Sun, 5 Nov 2023 08:10:19 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229887AbjKEQJu (ORCPT + 34 others); Sun, 5 Nov 2023 11:09:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53024 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229894AbjKEQJo (ORCPT ); Sun, 5 Nov 2023 11:09:44 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6E42010D4 for ; Sun, 5 Nov 2023 08:09:40 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 44870C433C7; Sun, 5 Nov 2023 16:09:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1699200580; bh=rWvyYXNgqAQXrazAdOtGXb3aSL+iaTLJvkVTxXaCuIo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=T//8hR6pATfEkkqNF92+WuWqB/m5/S6oOA9NH5YLNixkYQhd58fShbd8mXr1AF4Fv tiHThfSEt4QxOm/t07gYP9aEJr1bXxt2wxd/3bPvoRGm0p7vRImK883Ig0FPMTvFrj ZvkvzPfEhmWi5oDBlICoCH1531llMGkvkED5taleqOtrSRDFTCZHNh9LOLpCq+q//5 CagEW5w1GqccK6080wGe2BTbFM+PFl0aOdLVYlanU/w8nSgEvGqRtz9d0ayD+uxEFG j2/PAmZdAwpmdS/MwP+M/m5nzrJ0n2xjQyK6/9X5HEwwRW0eoLblo2x6Hwi8DunRgh +qA8qUNnp7VaQ== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [RFC PATCH 15/32] function_graph: Implement fgraph_reserve_data() and fgraph_retrieve_data() Date: Mon, 6 Nov 2023 01:09:33 +0900 Message-Id: <169920057328.482486.12880232375678712578.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169920038849.482486.15796387219966662967.stgit@devnote2> References: <169920038849.482486.15796387219966662967.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Sun, 05 Nov 2023 08:10:19 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781740993222993061 X-GMAIL-MSGID: 1781740993222993061 From: Steven Rostedt (VMware) Added functions that can be called by a fgraph_ops entryfunc and retfunc to store state between the entry of the function being traced to the exit of the same function. The fgraph_ops entryfunc() may call fgraph_reserve_data() to store up to 4 words onto the task's shadow ret_stack and this then can be retrived by fgraph_retrieve_data() called by the corresponding retfunc(). Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- include/linux/ftrace.h | 3 + kernel/trace/fgraph.c | 244 +++++++++++++++++++++++++++++++++++++++++------- 2 files changed, 213 insertions(+), 34 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 31ad20e78522..115758fe9fec 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -1049,6 +1049,9 @@ struct fgraph_ops { int idx; }; +void *fgraph_reserve_data(int size_bytes); +void *fgraph_retrieve_data(void); + /* * Stack of return addresses for functions * of a thread. diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index 748896e320e2..0f8e9f22e9e0 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -38,25 +38,36 @@ * bits: 14 - 15 Type of storage * 0 - reserved * 1 - fgraph_array index + * 2 - reservered data * For fgraph_array_index: * bits: 16 - 23 The fgraph_ops fgraph_array index * + * For reserved data: + * bits: 16 - 17 The size in words that is stored + * * That is, at the end of function_graph_enter, if the first and forth * fgraph_ops on the fgraph_array[] (index 0 and 3) needs their retfunc called - * on the return of the function being traced, this is what will be on the - * task's shadow ret_stack: (the stack grows upward) + * on the return of the function being traced, and the forth fgraph_ops + * stored two words of data, this is what will be on the task's shadow + * ret_stack: (the stack grows upward) + * + * | | <- task->curr_ret_stack + * +-------------------------------------+ + * | (3 << FGRAPH_ARRAY_SHIFT)|type:1|(5)| ( 3 for index of fourth fgraph_ops) + * +-------------------------------------+ + * | (3 << FGRAPH_DATA_SHIFT)|type:2|(4) | ( Data with size of 2 words) + * +-------------------------------------+ ( It is 4 words from the ret_stack) + * | STORED DATA WORD 2 | + * | STORED DATA WORD 1 | + * +-------------------------------------+ + * | (0 << FGRAPH_ARRAY_SHIFT)|type:1|(1)| ( 0 for index of first fgraph_ops) + * +-------------------------------------+ + * | struct ftrace_ret_stack | + * | (stores the saved ret pointer) | + * +-------------------------------------+ + * | (X) | (N) | ( N words away from last ret_stack) + * | | * - * | | <- task->curr_ret_stack - * +----------------------------------+ - * | (3 << FGRAPH_ARRAY_SHIFT)|(2) | ( 3 for index of fourth fgraph_ops) - * +----------------------------------+ - * | (0 << FGRAPH_ARRAY_SHIFT)|(1) | ( 0 for index of first fgraph_ops) - * +----------------------------------+ - * | struct ftrace_ret_stack | - * | (stores the saved ret pointer) | - * +----------------------------------+ - * | (X) | (N) | ( N words away from previous ret_stack) - * | | * * If a backtrace is required, and the real return pointer needs to be * fetched, then it looks at the task's curr_ret_stack index, if it @@ -77,12 +88,17 @@ enum { FGRAPH_TYPE_RESERVED = 0, FGRAPH_TYPE_ARRAY = 1, + FGRAPH_TYPE_DATA = 2, }; #define FGRAPH_ARRAY_SIZE 16 #define FGRAPH_ARRAY_MASK ((1 << FGRAPH_ARRAY_SIZE) - 1) #define FGRAPH_ARRAY_SHIFT (FGRAPH_TYPE_SHIFT + FGRAPH_TYPE_SIZE) +#define FGRAPH_DATA_SIZE 2 +#define FGRAPH_DATA_MASK ((1 << FGRAPH_DATA_SIZE) - 1) +#define FGRAPH_DATA_SHIFT (FGRAPH_TYPE_SHIFT + FGRAPH_TYPE_SIZE) + /* Currently the max stack index can't be more than register callers */ #define FGRAPH_MAX_INDEX FGRAPH_ARRAY_SIZE @@ -97,6 +113,8 @@ enum { #define RET_STACK(t, index) ((struct ftrace_ret_stack *)(&(t)->ret_stack[index])) +#define FGRAPH_MAX_DATA_SIZE (sizeof(long) * 4) + /* * Each fgraph_ops has a reservered unsigned long at the end (top) of the * ret_stack to store task specific state. @@ -111,21 +129,44 @@ static int fgraph_array_cnt; static struct fgraph_ops *fgraph_array[FGRAPH_ARRAY_SIZE]; +static inline int __get_index(unsigned long val) +{ + return val & FGRAPH_RET_INDEX_MASK; +} + +static inline int __get_type(unsigned long val) +{ + return (val >> FGRAPH_TYPE_SHIFT) & FGRAPH_TYPE_MASK; +} + +static inline int __get_array(unsigned long val) +{ + return (val >> FGRAPH_ARRAY_SHIFT) & FGRAPH_ARRAY_MASK; +} + +static inline int __get_data(unsigned long val) +{ + return (val >> FGRAPH_DATA_SHIFT) & FGRAPH_DATA_MASK; +} + static inline int get_ret_stack_index(struct task_struct *t, int offset) { - return current->ret_stack[offset] & FGRAPH_RET_INDEX_MASK; + return __get_index(current->ret_stack[offset]); } static inline int get_fgraph_type(struct task_struct *t, int offset) { - return (current->ret_stack[offset] >> FGRAPH_TYPE_SHIFT) & - FGRAPH_TYPE_MASK; + return __get_type(current->ret_stack[offset]); } static inline int get_fgraph_array(struct task_struct *t, int offset) { - return (current->ret_stack[offset] >> FGRAPH_ARRAY_SHIFT) & - FGRAPH_ARRAY_MASK; + return __get_array(current->ret_stack[offset]); +} + +static inline int get_data_idx(struct task_struct *t, int offset) +{ + return __get_data(current->ret_stack[offset]); } /* ftrace_graph_entry set to this to tell some archs to run function graph */ @@ -162,6 +203,121 @@ static void ret_stack_init_task_vars(unsigned long *ret_stack) memset(gvals, 0, sizeof(*gvals) * FGRAPH_ARRAY_SIZE); } +/** + * fgraph_reserve_data - Reserve storage on the task's ret_stack + * @size_bytes: The size in bytes to reserve (max of 4 words in size) + * + * Reserves space of up to 4 words (in word increments) on the + * task's ret_stack shadow stack, for a given fgraph_ops during + * the entryfunc() call. If entryfunc() returns zero, the storage + * is discarded. An entryfunc() can only call this once per iteration. + * The fgraph_ops retfunc() can retrieve this stored data with + * fgraph_retrieve_data(). + * + * Returns: On success, a pointer to the data on the stack. + * Otherwise, NULL if there's not enough space left on the + * ret_stack for the data, or if fgraph_reserve_data() was called + * more than once for a single entryfunc() call. + */ +void *fgraph_reserve_data(int size_bytes) +{ + unsigned long val; + void *data; + int curr_ret_stack = current->curr_ret_stack; + int data_size; + int size; + + if (size_bytes > FGRAPH_MAX_DATA_SIZE) + return NULL; + + /* Convert to number of longs + data word */ + data_size = ALIGN(size_bytes, sizeof(long)) / sizeof(long) + 1; + + /* The size to add to ret_stack (including the reserve word) */ + size = data_size + 1; + + val = current->ret_stack[curr_ret_stack - 1]; + + switch (__get_type(val)) { + case FGRAPH_TYPE_RESERVED: + /* + * A reserve word is only saved after the ret_stack + * or after a data storage, not after an fgraph_array + * entry. It's OK if its after the ret_stack in which + * case the index will be one, but if the index is + * greater than 1 it means it's a double call to + * fgraph_reserve_data() + */ + if (__get_index(val) > 1) + return NULL; + /* + * Leave the reserve in case the entryfunc() doesn't + * want to be recorded. + */ + break; + case FGRAPH_TYPE_ARRAY: + break; + default: + return NULL; + } + data = ¤t->ret_stack[curr_ret_stack]; + + curr_ret_stack += size; + if (unlikely(curr_ret_stack >= SHADOW_STACK_MAX_INDEX)) + return NULL; + + val = __get_index(val) + size; + + /* Set the last word to be reserved */ + current->ret_stack[curr_ret_stack - 1] = val; + + /* Make sure interrupts see this */ + barrier(); + current->curr_ret_stack = curr_ret_stack; + /* Again sync with interrupts, and reset reserve */ + current->ret_stack[curr_ret_stack - 1] = val; + + val = (data_size << FGRAPH_DATA_SHIFT) | + (FGRAPH_TYPE_DATA << FGRAPH_TYPE_SHIFT) | + (val - 1); + + /* Save the data header */ + current->ret_stack[curr_ret_stack - 2] = val; + + return data; +} + +/** + * fgraph_retrieve_data - Retrieve stored data from fgraph_reserve_data() + * + * This is to be called by a fgraph_ops retfunc(), to retrieve data that + * was stored by the fgraph_ops entryfunc() on the function entry. + * That is, this will retrieve the data that was reserved on the + * entry of the function that corresponds to the exit of the function + * that the fgraph_ops retfunc() is called on. + * + * Returns: The stored data from fgraph_reserve_data() called by the + * matching entryfunc() for the retfunc() this is called from. + * Or NULL if there was nothing stored. + */ +void *fgraph_retrieve_data(void) +{ + unsigned long val; + int curr_ret_stack = current->curr_ret_stack; + + /* Top of stack is the fgraph_ops */ + val = current->ret_stack[curr_ret_stack - 1]; + /* Check if there's nothing between the fgraph_ops and ret_stack */ + if (__get_index(val) == 1) + return NULL; + val = current->ret_stack[curr_ret_stack - 2]; + if (__get_type(val) != FGRAPH_TYPE_DATA) + return NULL; + + return ¤t->ret_stack[curr_ret_stack - + (__get_data(val) + 1)]; +} + /** * fgraph_get_task_var - retrieve a task specific state variable * @gops: The ftrace_ops that owns the task specific variable @@ -357,6 +513,7 @@ int function_graph_enter(unsigned long ret, unsigned long func, unsigned long frame_pointer, unsigned long *retp) { struct ftrace_graph_ent trace; + int save_curr_ret_stack; int offset; int start; int type; @@ -396,8 +553,10 @@ int function_graph_enter(unsigned long ret, unsigned long func, atomic_inc(¤t->trace_overrun); break; } + save_curr_ret_stack = current->curr_ret_stack; if (ftrace_ops_test(&gops->ops, func, NULL) && gops->entryfunc(&trace, gops)) { + /* Note, curr_ret_stack could change by enryfunc() */ offset = current->curr_ret_stack; /* Check the top level stored word */ type = get_fgraph_type(current, offset - 1); @@ -431,6 +590,9 @@ int function_graph_enter(unsigned long ret, unsigned long func, barrier(); current->ret_stack[offset] = val; cnt++; + } else { + /* Clear out any saved storage */ + current->curr_ret_stack = save_curr_ret_stack; } } @@ -545,10 +707,10 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs struct ftrace_ret_stack *ret_stack; struct ftrace_graph_ret trace; unsigned long ret; - int offset; + int curr_ret_stack; + int stop_at; int index; int idx; - int i; ret_stack = ftrace_pop_return_trace(&trace, &ret, frame_pointer); @@ -564,24 +726,38 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs trace.retval = fgraph_ret_regs_return_value(ret_regs); #endif - offset = current->curr_ret_stack - 1; - index = get_ret_stack_index(current, offset); + curr_ret_stack = current->curr_ret_stack; + index = get_ret_stack_index(current, curr_ret_stack - 1); + + stop_at = curr_ret_stack - index; /* index has to be at least one! Optimize for it */ - i = 0; do { - idx = get_fgraph_array(current, offset - i); - fgraph_array[idx]->retfunc(&trace, fgraph_array[idx]); - i++; - } while (i < index); + unsigned long val; + + val = current->ret_stack[curr_ret_stack - 1]; + switch (__get_type(val)) { + case FGRAPH_TYPE_ARRAY: + idx = __get_array(val); + fgraph_array[idx]->retfunc(&trace, fgraph_array[idx]); + fallthrough; + case FGRAPH_TYPE_RESERVED: + curr_ret_stack--; + break; + case FGRAPH_TYPE_DATA: + curr_ret_stack -= __get_data(val); + break; + default: + WARN_ONCE(1, "Bad fgraph ret_stack data type %d", + __get_type(val)); + curr_ret_stack--; + } + /* Make sure interrupts see the update after the above */ + barrier(); + current->curr_ret_stack = curr_ret_stack; + } while (curr_ret_stack > stop_at); - /* - * The ftrace_graph_return() may still access the current - * ret_stack structure, we need to make sure the update of - * curr_ret_stack is after that. - */ - barrier(); - current->curr_ret_stack -= index + FGRAPH_RET_INDEX; + current->curr_ret_stack -= FGRAPH_RET_INDEX; current->curr_ret_depth--; return ret; } From patchwork Sun Nov 5 16:09:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 161672 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp2197402vqu; Sun, 5 Nov 2023 08:10:14 -0800 (PST) X-Google-Smtp-Source: AGHT+IEg/qn9QbW0Y8sER3ngABm6Tw82xBa5MCwtBv4A/OqwNwwSwbXQ+DsmPpUODV3Oo6CuBj+B X-Received: by 2002:a17:902:e545:b0:1cc:468d:526c with SMTP id n5-20020a170902e54500b001cc468d526cmr19268745plf.9.1699200614103; Sun, 05 Nov 2023 08:10:14 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699200614; cv=none; d=google.com; s=arc-20160816; b=xLvc7+0o4aDU6bGMvKYnwQVF3sviTLzb4NqWrB7h4tqggRXbmULuF/KJ1P31piS7QW Xm8JSIIv34mrCFBnIbL5IQVe0kb+dyZS8cATJrq35zKQ3ADgA3aN/pfDInmXxQM/U4sk IsW1kVWsgaAjBeu2KpHB7GkIhl/pDCqt8Qs9Q44TVp8tJ9PpWwmv7Hji73RlzQJFRyWv yW17eeK8S6FRb1hF5k/bD17cp2ADY4KpIv09hNpZYHsEtItUmxr6wDW+9kP+dCP/RydW wOtfZPJwn25IxUze26130TDSQCT5o4mBD7Ffr/I3rebeZ1RUi6x+TdGyoeHvn6Mak+c9 oBnQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=OlxSyKjVM4y+aeDGRKyX0s+d+YhFPOWU1aAkU7qmgR0=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=GqDPXS8Ffg+6P32+NuB3rt5usq8tYurC3y/q7V9xygdrsQeaHxPpRyT1YxSf88YMjm ESu8u7RjSUAVyjjxRJ27tjAehwUsyqu1mDCBJtzeDkoBpx1j28dpe3D3276epvrfGCyz DtCUHCMRyOJyBJuR9XhQ33cmtaxvvrI2w7u/mIswJoDG5BB02SFPfyHWRHLsaluncu8M mBwmvx6GpGjVtOBnKbhhVBLGLeXZSWKJvN1UiX/xsacD1HM27Mqbr7e/Y/d0NsIMcpEd 46SzPXEbAkBub1XJw4lRsO3lNZbYRr64HXhvq7TRCSomFTF1vZAVyb3RtV+UvcqaqvEA V7pw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=s0rllwGV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id s15-20020a170902ea0f00b001b811261289si6531366plg.482.2023.11.05.08.10.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Nov 2023 08:10:14 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=s0rllwGV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 50FE580B1D39; Sun, 5 Nov 2023 08:10:13 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229514AbjKEQKA (ORCPT + 34 others); Sun, 5 Nov 2023 11:10:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53050 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229515AbjKEQJ4 (ORCPT ); Sun, 5 Nov 2023 11:09:56 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8EB1AD70 for ; Sun, 5 Nov 2023 08:09:52 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 84121C433C7; Sun, 5 Nov 2023 16:09:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1699200592; bh=iHMWoK48jd37dO5Jbonc2zMACTI9ZQjwy+qJj2dU7cc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=s0rllwGVgvS8kfem8mLBzFyyTqulUUEwwI5/CqZmswfnc7vPmaD0QsF0As/6xJiCY bla2XGcOMjXwyG50JjwihbCb9Rz3jWVbEB50399XcaNdtXw5TS00v0/Ei95/nUZMUK xuJbTeXzbcg4M4lpyOVqqWlxjCoLgKZt/RjPIK90/uBmuoRwS1zYjDQUYFwDjEKvzf VLCintK0CcfrcwfSUL9ktgqBN/rJDhglXYSTp7tXHRuC7lSYNOrg8xWqMszyE5g/Bs 6NSDvA6vCY2fFsM0f0W6NHDpvMJm0I1BJsvZ+TaueXxGYnd0Hh/ZVVkz2MaO/rqgyZ myOEFSROdm6uQ== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [RFC PATCH 16/32] function_graph: Add selftest for passing local variables Date: Mon, 6 Nov 2023 01:09:45 +0900 Message-Id: <169920058540.482486.7562979275156655655.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169920038849.482486.15796387219966662967.stgit@devnote2> References: <169920038849.482486.15796387219966662967.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Sun, 05 Nov 2023 08:10:13 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781740982926217080 X-GMAIL-MSGID: 1781740982926217080 From: Steven Rostedt (VMware) Add boot up selftest that passes variables from a function entry to a function exit, and make sure that they do get passed around. Signed-off-by: Steven Rostedt (VMware) --- kernel/trace/trace_selftest.c | 161 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 161 insertions(+) diff --git a/kernel/trace/trace_selftest.c b/kernel/trace/trace_selftest.c index e05f7335525c..14966d8509ed 100644 --- a/kernel/trace/trace_selftest.c +++ b/kernel/trace/trace_selftest.c @@ -756,6 +756,165 @@ trace_selftest_startup_function(struct tracer *trace, struct trace_array *tr) #ifdef CONFIG_FUNCTION_GRAPH_TRACER +#ifdef CONFIG_DYNAMIC_FTRACE + +#define BYTE_NUMBER 123 +#define SHORT_NUMBER 12345 +#define WORD_NUMBER 1234567890 +#define LONG_NUMBER 1234567890123456789LL + +static int fgraph_store_size __initdata; +static const char *fgraph_store_type_name __initdata; +static char *fgraph_error_str __initdata; +static char fgraph_error_str_buf[128] __initdata; + +static __init int store_entry(struct ftrace_graph_ent *trace, + struct fgraph_ops *gops) +{ + const char *type = fgraph_store_type_name; + int size = fgraph_store_size; + void *p; + + p = fgraph_reserve_data(size); + if (!p) { + snprintf(fgraph_error_str_buf, sizeof(fgraph_error_str_buf), + "Failed to reserve %s\n", type); + fgraph_error_str = fgraph_error_str_buf; + return 0; + } + + switch (fgraph_store_size) { + case 1: + *(char *)p = BYTE_NUMBER; + break; + case 2: + *(short *)p = SHORT_NUMBER; + break; + case 4: + *(int *)p = WORD_NUMBER; + break; + case 8: + *(long long *)p = LONG_NUMBER; + break; + } + + return 1; +} + +static __init void store_return(struct ftrace_graph_ret *trace, + struct fgraph_ops *gops) +{ + const char *type = fgraph_store_type_name; + long long expect = 0; + long long found = -1; + char *p; + + p = fgraph_retrieve_data(); + if (!p) { + snprintf(fgraph_error_str_buf, sizeof(fgraph_error_str_buf), + "Failed to retrieve %s\n", type); + fgraph_error_str = fgraph_error_str_buf; + return; + } + + switch (fgraph_store_size) { + case 1: + expect = BYTE_NUMBER; + found = *(char *)p; + break; + case 2: + expect = SHORT_NUMBER; + found = *(short *)p; + break; + case 4: + expect = WORD_NUMBER; + found = *(int *)p; + break; + case 8: + expect = LONG_NUMBER; + found = *(long long *)p; + break; + } + + if (found != expect) { + snprintf(fgraph_error_str_buf, sizeof(fgraph_error_str_buf), + "%s returned not %lld but %lld\n", type, expect, found); + fgraph_error_str = fgraph_error_str_buf; + return; + } + fgraph_error_str = NULL; +} + +static struct fgraph_ops store_bytes __initdata = { + .entryfunc = store_entry, + .retfunc = store_return, +}; + +static int __init test_graph_storage_type(const char *name, int size) +{ + char *func_name; + int len; + int ret; + + fgraph_store_type_name = name; + fgraph_store_size = size; + + snprintf(fgraph_error_str_buf, sizeof(fgraph_error_str_buf), + "Failed to execute storage %s\n", name); + fgraph_error_str = fgraph_error_str_buf; + + printk(KERN_CONT "PASSED\n"); + pr_info("Testing fgraph storage of %d byte%s: ", size, size > 1 ? "s" : ""); + + func_name = "*" __stringify(DYN_FTRACE_TEST_NAME); + len = strlen(func_name); + + ret = ftrace_set_filter(&store_bytes.ops, func_name, len, 1); + if (ret && ret != -ENODEV) { + pr_cont("*Could not set filter* "); + return -1; + } + + ret = register_ftrace_graph(&store_bytes); + if (ret) { + printk(KERN_WARNING "Failed to init store_bytes fgraph tracing\n"); + return -1; + } + + DYN_FTRACE_TEST_NAME(); + + unregister_ftrace_graph(&store_bytes); + + if (fgraph_error_str) { + printk(KERN_CONT "*** %s ***", fgraph_error_str); + return -1; + } + + return 0; +} +/* Test the storage passed across function_graph entry and return */ +static __init int test_graph_storage(void) +{ + int ret; + + ret = test_graph_storage_type("byte", 1); + if (ret) + return ret; + ret = test_graph_storage_type("short", 2); + if (ret) + return ret; + ret = test_graph_storage_type("word", 4); + if (ret) + return ret; + ret = test_graph_storage_type("long long", 8); + if (ret) + return ret; + return 0; +} +#else +static inline int test_graph_storage(void) { return 0; } +#endif /* CONFIG_DYNAMIC_FTRACE */ + /* Maximum number of functions to trace before diagnosing a hang */ #define GRAPH_MAX_FUNC_TEST 100000000 @@ -913,6 +1072,8 @@ trace_selftest_startup_function_graph(struct tracer *trace, ftrace_set_global_filter(NULL, 0, 1); #endif + ret = test_graph_storage(); + /* Don't test dynamic tracing, the function tracer already did */ out: /* Stop it if we failed */ From patchwork Sun Nov 5 16:09:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 161673 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp2197418vqu; Sun, 5 Nov 2023 08:10:16 -0800 (PST) X-Google-Smtp-Source: AGHT+IHtEBrEYwVPMnPUh55HxfPqpwQ7s0nO9VlO3Argrm8kgEb5n6jaItrXuOHiWn9OGw1rDeJm X-Received: by 2002:a05:6870:bb1b:b0:1f0:6565:9a37 with SMTP id nw27-20020a056870bb1b00b001f065659a37mr11771266oab.50.1699200615983; Sun, 05 Nov 2023 08:10:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699200615; cv=none; d=google.com; s=arc-20160816; b=hIZv98oYoN/MpKAWknA8CE/touwbqoM/gQodp30nb81+tOJ2U62c1e8E/znpuboNu9 aGEGtXjn+Kfzr3WeLcGchjpxsZqCAfbY90ri9uLF5jTrT88xW3/AIJHdqAHvQa7uaKB3 Uqke8MQ6bYPsJQESuliBeVXhEmL8fzHQdg7jIMTxQByIUbgELQ8RTKHhmnWEvrlX9g+W ny8QyYsZV0IVK8KcTiJMtME8DkV8L38Faf5yGkcbfC6Xez+dcOOlsGrmMlbXM89YSaZ6 +Ok8HXZWUc2JERyrBon2ymTm2UWMkFZy08XfTVJaM4ErFbKSfkAFH+2R5tbjBDAEk4CS umMA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=g2cVJrZDgiMnw2CnUCJh/woCo0rG1+gb5OVy/zc7glQ=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=jS0H/0vCu7W87ly/TVpnq7N7SqKkAVm01dkcZSvIZqN4vcY8FnJiT339ucZhmRnPX+ lda6vafLcJY43/xDmOf15bKXSGFSq8MZSpjfn9x68WjqhSEeQ+DweuUfKt/zyQE5OeV2 1p/AJRi5XlwScbI3OdzH9j0EFSjz4A6H4GXmxCPkEHRGMeDdikQheIxhhZpAwrKvfrao bfIxuyhc8JWR8zJHfdmr8cIa3j6gpzbyroz6+Ur3RXSFicbdNNSJwd/0oc3EDMg3s9mz cGilXqj0OpZU4VqKgrQ6qNMCfr5JEZ9x2st1c8eniGPI0wVdVudsNADGVhJhKRPFQ4QU Q54A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="c/CoB3R0"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id eg11-20020a056a00800b00b006bca1246564si6231515pfb.271.2023.11.05.08.10.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Nov 2023 08:10:15 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="c/CoB3R0"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 3F8EE804705A; Sun, 5 Nov 2023 08:10:15 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229593AbjKEQKK (ORCPT + 34 others); Sun, 5 Nov 2023 11:10:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44506 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229791AbjKEQKH (ORCPT ); Sun, 5 Nov 2023 11:10:07 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D1E17184 for ; Sun, 5 Nov 2023 08:10:04 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4D602C433C8; Sun, 5 Nov 2023 16:10:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1699200604; bh=6QXlcdrE6Vd0uevCRAQA5jRWA9Z4WkVDk80uAUwbJ/E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=c/CoB3R0PwHpStxpBdcBy486i6VbtYtAR1YYmvlMJgWlkvwAslfAM1MBCOHQznEuS EKB3GpxYg2osZ0mbNEFlBtDFS7IVzI8nLjnyHdFfeaBzk/uPzEYKy3lm1Bh7rSay0R qkBO9K4WnNjpxWMohFj5Bz6IizDWDIk+eYYnnzFX5DId19isCVydBj6Z1yGqVq2lxc vsiYZsNeJERJi8N1ScBOr5UoHyCG75QN3HFIMfGa2X5DeJe5Y5ak8aag8MhZQpGBcn kpy8+KCP3F7miOXW/IulJ9Y5Nh6snHUESFyYCkL2wO64KgRqRsaKfBZSyp+XcguuOu wfUKgBO98l/PA== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [RFC PATCH 17/32] function_graph: Fix to update curr_ret_stack with ARRAY entry size Date: Mon, 6 Nov 2023 01:09:57 +0900 Message-Id: <169920059747.482486.11622923727523240888.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169920038849.482486.15796387219966662967.stgit@devnote2> References: <169920038849.482486.15796387219966662967.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Sun, 05 Nov 2023 08:10:15 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781740985513097663 X-GMAIL-MSGID: 1781740985513097663 From: Masami Hiramatsu (Google) __ftrace_return_to_handler() missed to update the curr_ret_stack when it gets a FGRAPH_TYPE_ARRAY. Since that type entry will follows some data words, it must update curr_ret_stack by entry size value (__get_index(val)) instead of 1. Signed-off-by: Masami Hiramatsu (Google) --- kernel/trace/fgraph.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index 0f8e9f22e9e0..597250bd30dc 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -740,7 +740,8 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs case FGRAPH_TYPE_ARRAY: idx = __get_array(val); fgraph_array[idx]->retfunc(&trace, fgraph_array[idx]); - fallthrough; + curr_ret_stack -= __get_index(val); + break; case FGRAPH_TYPE_RESERVED: curr_ret_stack--; break; From patchwork Sun Nov 5 16:10:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 161676 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp2197851vqu; Sun, 5 Nov 2023 08:11:11 -0800 (PST) X-Google-Smtp-Source: AGHT+IEdeHr9CrMRq46V26Cv8JLAcujeaRZ831+2P9VMn5OTaZ+u0rKFSyeBqgT5kQHTnX/SvlfM X-Received: by 2002:a05:6a21:7888:b0:181:a3ae:115b with SMTP id bf8-20020a056a21788800b00181a3ae115bmr8578884pzc.61.1699200671100; Sun, 05 Nov 2023 08:11:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699200671; cv=none; d=google.com; s=arc-20160816; b=VZ0NAG7OcpK/PKbLRWA2FiRQHuUkL3KJa+pXvChtt5d5qaJnKGFIvhDYy9fXJEHyYf XReRvxN54Yk6x6Yn97+xFi0vUJNyu+jShiQuTeFmTyxobHjW02BkvGuLjsNemTIprqPn flCaNtJ5xKE5uaoDeKXF6mTXwdpSAzbgxJ0G2aHEi1Tbb0H7Vg5j1Ze8jeAk3qoihzEz wtyCp7KxWXeMc0lWrrUXqNLnhuw/4eJQdYViZAk1OcLqP79DWKaceKsZKtnCbkracmtg vP+W06A8KnWH3W8KQguFdHL12pnrBHZ7oaBA0kBitRkkls68F53jnZ8/nuUkU+QrM9Sp c/Kg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=Y/k6B6oRDzgUirIm69QYAP2gIpyR3uWTbuQWNfQzotM=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=jm60nnEI0v8ISfSsD9/GXD/QQxtvsC0hnoyUH0Y6OYjTbML9Qkme5CaeydaS4zen57 78Z1Ut3ztx5S09wEygsbJeII/sbX9b5npP77j5c5LY0QiWpN76/ouNjMw3HPv8f8cdwF XqJMhmr/v4YsUnKGV7ZHZV7XlKYAka+RilBoOLU8hIUTpFxrnI8OvQ6TVRzgPsKSjBvC igHVN9/XBKgwYcpJIyS819OXFz/Nnsf9GY5lNkzmY58M8Wk50T627xEpc9TC8sskCQ6c lc1vy9VSJw7kGIwBLpLOhRNyVX2jI/7nSgnAsuEivs/lIVJwh8cOoApQd61B05DkzqLq N3sw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=ijq8qE6a; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from fry.vger.email (fry.vger.email. [2620:137:e000::3:8]) by mx.google.com with ESMTPS id c130-20020a633588000000b005b9b68add85si6058335pga.254.2023.11.05.08.11.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Nov 2023 08:11:11 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=ijq8qE6a; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 51F258085986; Sun, 5 Nov 2023 08:10:55 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229599AbjKEQKZ (ORCPT + 34 others); Sun, 5 Nov 2023 11:10:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53290 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229956AbjKEQKT (ORCPT ); Sun, 5 Nov 2023 11:10:19 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 118BD13E for ; Sun, 5 Nov 2023 08:10:17 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C31C1C433C7; Sun, 5 Nov 2023 16:10:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1699200616; bh=/gBbJf7g1QrUiEN7ZPrz7W4ai3w02Yrm1jHvyS8561U=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ijq8qE6apP/3tEMc7xQNaT1ZgohMP5WaYkHJZJLjvfucgOnkq8r8tZKlMpnIaRzoG iAgl0QVwW8FRbNaoPA/aBIVbNAp1IW7ANhQrFRRo6ITaqIvkVGnn7cnFDbwRuAtGX7 AGAMb81eNQdq7SI/g6khmRUvWkHaqrSTyUAkslVhM09Cqa/6UbPC35PM5SJQv8T0hm 2lD1u208cjdCKzkYrE75U/ZT1OpISl8GDOQuSp9HdwT171hfKZgk3GzG6+kDoNzaP6 b+v/jsTZnxYX4auQr1LcKkM0uowq0gHJyRNe/1l6yDnU1Ng7Z+kzrq1jNiLX6UcYx/ 0BVdFt5sbwbJw== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [RFC PATCH 18/32] function_graph: Fix to initalize ftrace_ops for fgraph with ftrace_graph_func Date: Mon, 6 Nov 2023 01:10:10 +0900 Message-Id: <169920060974.482486.15664806338999944098.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169920038849.482486.15796387219966662967.stgit@devnote2> References: <169920038849.482486.15796387219966662967.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Sun, 05 Nov 2023 08:10:55 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781741042847867860 X-GMAIL-MSGID: 1781741042847867860 From: Masami Hiramatsu (Google) Fix to initialize the ftrace_ops of fgraph_ops with ftrace_graph_func instead of ftrace_stub. Signed-off-by: Masami Hiramatsu (Google) --- kernel/trace/fgraph.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index 597250bd30dc..858fb73440ec 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -872,7 +872,7 @@ unsigned long ftrace_graph_ret_addr(struct task_struct *task, int *idx, void fgraph_init_ops(struct ftrace_ops *dst_ops, struct ftrace_ops *src_ops) { - dst_ops->func = ftrace_stub; + dst_ops->func = ftrace_graph_func; dst_ops->flags = FTRACE_OPS_FL_PID | FTRACE_OPS_FL_STUB; #ifdef FTRACE_GRAPH_TRAMP_ADDR @@ -1120,7 +1120,7 @@ int register_ftrace_graph(struct fgraph_ops *gops) if (!gops->ops.func) { gops->ops.flags |= FTRACE_OPS_FL_STUB; - gops->ops.func = ftrace_stub; + gops->ops.func = ftrace_graph_func; #ifdef FTRACE_GRAPH_TRAMP_ADDR gops->ops.trampoline = FTRACE_GRAPH_TRAMP_ADDR; #endif From patchwork Sun Nov 5 16:10:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 161675 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp2197630vqu; Sun, 5 Nov 2023 08:10:47 -0800 (PST) X-Google-Smtp-Source: AGHT+IGxScUR2uw8a4nhV/bjrnRT8K1lAbFHhHAzo2cvWVM6V7q/fvGSifJZAcdqToBOTX+hUWz8 X-Received: by 2002:a17:903:32c3:b0:1cc:4828:9b07 with SMTP id i3-20020a17090332c300b001cc48289b07mr25268808plr.0.1699200646807; Sun, 05 Nov 2023 08:10:46 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699200646; cv=none; d=google.com; s=arc-20160816; b=x70PAZl8SEP7XPvMvTwwy+pcSfnwLD3p9/eYknOwLu2KVMIWYrmIExz1m5h1XQDBc+ xBv4HtLs8jS1bpACUP7Hrh2feiDGfl0TuKY9Tin/fa/EClPT5WsUSRfKfpZQiHjhOogQ o/O5tpKipnCklSgdlbSnSDos7H8+dZkvCKzSWICl4zRExfDV/24VX/QABdp8LsFaMByc miDtfIMpysO/PQCtOhFJj9hUiJWo4AGlWbgjN4oMMHHTQbcLBIAC4RlZblg6/OMLn3ex BvwwwXE2UBN8LfWTX2ubrtUlalhPQmzfOlYFDhkoabvqXYxR2aX9S4NKBELvscvlvBdj slTQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=YoYXkSa3L9Pc8/kbYQ843hZhircS1ZTYGueEwE0A+uI=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=JheLUhnsZ18uv0UK0x7Bam3kvaPk+HEeO4zxHKeIuR6KBCTpaBVIbnfjA/jyNhA3j2 cOG5S7OGs5ZrR+FSrMhw4QphPJToJckbQDHnOyD0uY6vCCAa/QDgz8GaHg/Qr3Hvo7tr EE1thPCHDY8QThkR9pfYhopymN4cxWxu+21LIiaZ6y8wYMHgtIRcCCRJCkZoK6lCuHk5 TDmcxWVWsTPQQc6fVunzUOrPQgsnzNjKMB+SkDLmNsmM477/jGAIu1dhMT7QRMI0Zwym Qt7MMVk2ykxazJPly41EqalVIbNuxGfrz17kxNw+ithQTpz5eye+9jnzQGvTekrF37Ay CK7w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=DChrFFMp; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id i10-20020a170902eb4a00b001bbbbb61c71si6022733pli.399.2023.11.05.08.10.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Nov 2023 08:10:46 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=DChrFFMp; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 1448E804705A; Sun, 5 Nov 2023 08:10:46 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229814AbjKEQKg (ORCPT + 34 others); Sun, 5 Nov 2023 11:10:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35268 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229707AbjKEQKb (ORCPT ); Sun, 5 Nov 2023 11:10:31 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 91DE1125 for ; Sun, 5 Nov 2023 08:10:28 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id DF42EC433C8; Sun, 5 Nov 2023 16:10:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1699200628; bh=owfDd6JTzvCM2mjVDdOSlbEmkdB46BrUrZqzXjclNlU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DChrFFMpRcrcTwjoa7bf2Wg2cA2IsxnVTfMWzssOMgTHcLuK6LQFafhQxAF1oGC3x rtCQ5U8qMxod4wcKgG4k5iQH2S2Xr0dV98/0HARMoKy6nSJCIJr6joQUN9Rt6jNiZx /0IC6vqqt3CJITKu1ulmp1JhYorPV5a5sq3sotVxBBwUmq3gPmgoVh8pGzoW7mw0Ut HbVWGEvrQ5A1NZbxX50j37FmJ7VacfCPPLZupCoKnU+RLTIvXNbAJjNiMHQhrrxfbK 7gY7o7pa+rcUSP0XUiMWxCmwDEnozQ4Zz81wKkPig6V+V1OOY6PYbXANoq6reaS92a RTNrzqpAWJQaw== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [RFC PATCH 19/32] function_graph: Fix to check the return value of ftrace_pop_return_trace() Date: Mon, 6 Nov 2023 01:10:22 +0900 Message-Id: <169920062203.482486.9590838723080321966.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169920038849.482486.15796387219966662967.stgit@devnote2> References: <169920038849.482486.15796387219966662967.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Sun, 05 Nov 2023 08:10:46 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781741017490031375 X-GMAIL-MSGID: 1781741017490031375 From: Masami Hiramatsu (Google) Fix to check the return value ('ret_stack') of ftrace_pop_return_trace() instead of passed storage ('ret') because ret_stack becomes NULL in error case. Signed-off-by: Masami Hiramatsu (Google) --- kernel/trace/fgraph.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index 858fb73440ec..e51695441476 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -714,7 +714,7 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs ret_stack = ftrace_pop_return_trace(&trace, &ret, frame_pointer); - if (unlikely(!ret)) { + if (unlikely(!ret_stack)) { ftrace_graph_stop(); WARN_ON(1); /* Might as well panic. What else to do? */ From patchwork Sun Nov 5 16:10:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 161678 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp2197876vqu; Sun, 5 Nov 2023 08:11:14 -0800 (PST) X-Google-Smtp-Source: AGHT+IGbIOX6P5jl8jMYkgiahSAJBqaPkJBJNbobElnsEsk/Hlq+zmDOaUWn1R+nLKlnkzj5GQDC X-Received: by 2002:a05:6a21:47ca:b0:181:bc16:48d7 with SMTP id as10-20020a056a2147ca00b00181bc1648d7mr7895004pzc.51.1699200673986; Sun, 05 Nov 2023 08:11:13 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699200673; cv=none; d=google.com; s=arc-20160816; b=EqFNxhs/lGPZwWOirjqJpy8xtqujA93yJbTf7KtSJzPObxY0J8JtCvTgULJVCECoWc b8JJfc+BQ6C/33kwShV9wSkCZVsn3iNmgWvPdnVZ/DHRwNPN+mKcScKDIIzAhLvmYmys sSKAPSqmopiznj2slJWX6Y/7DmgRSdBLtH5j2Kl1HFOXPGDjV5A5VIP7v8YNL79BooHU LG4BUE+9FItZX1kt2RwoVFs145pmvQG3EqFS8wWOQWZ2VKvC4ZuMz+llPuODbR9QfWco 9D7TrVeE7wsLfFJOdkljaVxzpFKIMu+vc2b1aSmBI2QF0Yh+OVoLQRXzb6DZcA8drBiT HvyQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=ibtVobGBbah3nTuNwHOOFIWdXfUfyT9b9e/qqj9EFas=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=NIBLkf28vHXyW+DvZBeF06QFjpryguILNmI/C6+eNMpDZyeclmUC77f56S2OYFizd/ dtXzidby3mSpBa8jJxhSf3h0iZ3SIqHyasj3Lvjq/wcBuIv7s2x6pnvUeYuwZzT5MKxo gQKo00ql9ht4EIRBO84Ysx06vX9mYqVESItU+mkOEauacG1u3iad4WievX8quivHeRQ4 x/L5NWgvacQy36//+2ekgxexQSBpDlpHBNYbRI97AAPMO23aLGeKsudCcru4Sp+NKWEe 1qWSYnO4J0hz6nJ48W5LRmYCgsQ3ZjwFyU58JF0YhiOPvdDPUhjkpmUzEEZGQI3u9/mM +dxg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=t9r39sEc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id fb30-20020a056a002d9e00b006b74a46cd3fsi6499244pfb.229.2023.11.05.08.11.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Nov 2023 08:11:13 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=t9r39sEc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 4C8A68057989; Sun, 5 Nov 2023 08:11:11 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229514AbjKEQKp (ORCPT + 34 others); Sun, 5 Nov 2023 11:10:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46538 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229593AbjKEQKm (ORCPT ); Sun, 5 Nov 2023 11:10:42 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B4DE5E0 for ; Sun, 5 Nov 2023 08:10:39 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 36D8AC433C9; Sun, 5 Nov 2023 16:10:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1699200639; bh=2WlQKdWbi99vKuQZnLhkUQgCEXy+Nfuzu2d7nEzVsTk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=t9r39sEcBlbf74NlrzR4B8Lle9WJaLJPkd7Yj8058moBUJM9BSEIitXmIzuxEEaXi xfkKEr9LFKF5//Xf6ZTafDe5ngETaGLMNpSCFrglua32coY+eWP255NG3KIJGChiJM 0mPhZ4OmmEk0xGvLSIY6RR+Kz4+MC4xpgpuqNvffIJkR7UBCwyinyVK+eafTCP3MbP hE8/xAY8TcGzILkYg8YnukhSFlP5gLxADP46/f1t6q+O1md8kd0PjUFDHEwEXUGkyI 21TvMVdjRGRPaE51PhOxqGcdeEsaN4MM+WVwTluLfRpn1M/g0G99Zs5mItY7YHZsWO 0gnyqS2oZQ8hg== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [RFC PATCH 20/32] function_graph: Pass the size of reserved data when retrieving it Date: Mon, 6 Nov 2023 01:10:34 +0900 Message-Id: <169920063363.482486.14684899342898937044.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169920038849.482486.15796387219966662967.stgit@devnote2> References: <169920038849.482486.15796387219966662967.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Sun, 05 Nov 2023 08:11:11 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781741046131810497 X-GMAIL-MSGID: 1781741046131810497 From: Masami Hiramatsu (Google) Make the fgraph_retrieve_data() returns the reverved data size via size_byte parameter. Signed-off-by: Masami Hiramatsu (Google) --- include/linux/ftrace.h | 2 +- kernel/trace/fgraph.c | 5 ++++- kernel/trace/trace_selftest.c | 10 +++++++++- 3 files changed, 14 insertions(+), 3 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 115758fe9fec..2e24a2611ca8 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -1050,7 +1050,7 @@ struct fgraph_ops { }; void *fgraph_reserve_data(int size_bytes); -void *fgraph_retrieve_data(void); +void *fgraph_retrieve_data(int *size_bytes); /* * Stack of return addresses for functions diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index e51695441476..0b8a1daef733 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -289,6 +289,7 @@ void *fgraph_reserve_data(int size_bytes) /** * fgraph_retrieve_data - Retrieve stored data from fgraph_reserve_data() + * @size_bytes: pointer to retrieved data size. * * This is to be called by a fgraph_ops retfunc(), to retrieve data that * was stored by the fgraph_ops entryfunc() on the function entry. @@ -300,7 +301,7 @@ void *fgraph_reserve_data(int size_bytes) * matching entryfunc() for the retfunc() this is called from. * Or NULL if there was nothing stored. */ -void *fgraph_retrieve_data(void) +void *fgraph_retrieve_data(int *size_bytes) { unsigned long val; int curr_ret_stack = current->curr_ret_stack; @@ -313,6 +314,8 @@ void *fgraph_retrieve_data(void) val = current->ret_stack[curr_ret_stack - 2]; if (__get_type(val) != FGRAPH_TYPE_DATA) return NULL; + if (size_bytes) + *size_bytes = (__get_data(val) - 1) * sizeof(long); return ¤t->ret_stack[curr_ret_stack - (__get_data(val) + 1)]; diff --git a/kernel/trace/trace_selftest.c b/kernel/trace/trace_selftest.c index 14966d8509ed..c656ec5b3348 100644 --- a/kernel/trace/trace_selftest.c +++ b/kernel/trace/trace_selftest.c @@ -807,15 +807,23 @@ static __init void store_return(struct ftrace_graph_ret *trace, const char *type = fgraph_store_type_name; long long expect = 0; long long found = -1; + int size; char *p; - p = fgraph_retrieve_data(); + p = fgraph_retrieve_data(&size); if (!p) { snprintf(fgraph_error_str_buf, sizeof(fgraph_error_str_buf), "Failed to retrieve %s\n", type); fgraph_error_str = fgraph_error_str_buf; return; } + if (fgraph_store_size > size) { + snprintf(fgraph_error_str_buf, sizeof(fgraph_error_str_buf), + "Retrieved size %d is smaller than expected %d\n", + size, (int)fgraph_store_size); + fgraph_error_str = fgraph_error_str_buf; + return; + } switch (fgraph_store_size) { case 1: From patchwork Sun Nov 5 16:10:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 161677 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp2197877vqu; Sun, 5 Nov 2023 08:11:14 -0800 (PST) X-Google-Smtp-Source: AGHT+IEA3FNtW91kxrdGmNJQ109Jq9TJcrpq9A2sqMDF+TduzFYt/x8YJyDJ+x+EAjYwvu6iaMlC X-Received: by 2002:a05:6a20:e114:b0:14d:abc:73dc with SMTP id kr20-20020a056a20e11400b0014d0abc73dcmr28098090pzb.32.1699200674006; Sun, 05 Nov 2023 08:11:14 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699200673; cv=none; d=google.com; s=arc-20160816; b=cgTdSl6fRRATxyD4hqJucoXUb6ARhEMOqbMQ4rFfHAr0tE2LuYHwghzwmYxVH3aJb0 tvh5DEBt8HGfSbjsS6ZDSf6XMS1FJxmOXvOnMrNhungf2Ct6OQYTytR8XH5QEdtUy6o3 l2xrz8LNeVwxKHmjSpZLvAicKGTZq2ap0OhLydyuatwROD0Ea1d+r2Ka8AXWLRE9FhnT enannUwjg+YFU5qh0RVdVsNhsmB9bdYmpf9be0eaLDFeXH/JnosDgtxBk/Qqgnum13Dl 2vlQV94qoYr4dIbAK1HcoJKUW4hHCfx6yF626SZS9pTW1EJrfUAeHfJHX5RBgDngpCSm 5bMg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=GvbSczaP3HE673qffyUF/8KrrGzAnI0iVdh21KthUrg=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=jlh1HEYgr2xjCJRv0gALtYFZoyRDxaAUIj9VP/cZzXabrix1dMsVVfPjGa8VZjsluH 2RVh+YdZCfe49FORD0D/O0/PBMnMTjrknkmed6p8hS1KRKu+KmCL5C3axTydKuCQg0cU q8onYd97CJ7GAkJmn3rcqNwaUxupxo5yS9zUe+5aVo3pNnYK/aQYDW0egiHoG1+R+0rU sCb+Uz12850QMJhfMEYzZjeebUKdNWtIHTUme+jxoWSalstgBxV1KZLoG3HthHOzAtt1 cqpc00JpQc0AnxWbUurpvZAFEukxGrVTRCXd2n14PPHw65QbSLK+5ItO7uW47CFeEUtS lomQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Elgaphgm; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from agentk.vger.email (agentk.vger.email. [23.128.96.32]) by mx.google.com with ESMTPS id nk15-20020a17090b194f00b0027da95531c9si5979860pjb.134.2023.11.05.08.11.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Nov 2023 08:11:13 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Elgaphgm; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 849CB805798D; Sun, 5 Nov 2023 08:11:11 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229978AbjKEQLA (ORCPT + 34 others); Sun, 5 Nov 2023 11:11:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46486 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229922AbjKEQKy (ORCPT ); Sun, 5 Nov 2023 11:10:54 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F31031BC for ; Sun, 5 Nov 2023 08:10:50 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3DEA8C433C8; Sun, 5 Nov 2023 16:10:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1699200650; bh=FCiUbQ7cF3hWz3W5mYkLUGazPgX9XWR3NoQQHaWsf5Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ElgaphgmZOlJVXyX6c/8ZRvVfsILDJK73lDr2cM6FuDWVRFQVi1Xl/jFjbpWMRuKl qiUTseyiLMuAGQB4ygAmSB7CeXSWOh/6CzLH7qYgVRX9Jy+jGPrVg9GVtf+h173CQ9 RV4Zke/9/N12hKsdXTVkAXIZIkd8YoxEa8A+3icYfbcDv8A5agz9eYlOWWKvZ+II4v TlFevh2JOy+CP6/Fv6AcPJ0UFGDoypzm2T99ComFygwZqQrU9IZnD8gBeWnOKNOLaG yzscPjTfm/KgcNUMQEjH92QhLUI4kznMLszTu7TS/vyRKNFwBcaWY7y36iecvK3YxB pTes6Uyn61cGg== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [RFC PATCH 21/32] function_graph: Expand the max reserved data size Date: Mon, 6 Nov 2023 01:10:45 +0900 Message-Id: <169920064470.482486.9772883857059976420.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169920038849.482486.15796387219966662967.stgit@devnote2> References: <169920038849.482486.15796387219966662967.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Sun, 05 Nov 2023 08:11:11 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781741045901699090 X-GMAIL-MSGID: 1781741045901699090 From: Masami Hiramatsu (Google) Expand the maximum reserved data size to 32 from 4. On 64bit system, this will allow reserving 256 bytes on the ret_stack in maximum but that may not happen. Signed-off-by: Masami Hiramatsu (Google) --- kernel/trace/fgraph.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index 0b8a1daef733..e7188c67356e 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -95,7 +95,7 @@ enum { #define FGRAPH_ARRAY_MASK ((1 << FGRAPH_ARRAY_SIZE) - 1) #define FGRAPH_ARRAY_SHIFT (FGRAPH_TYPE_SHIFT + FGRAPH_TYPE_SIZE) -#define FGRAPH_DATA_SIZE 2 +#define FGRAPH_DATA_SIZE 5 #define FGRAPH_DATA_MASK ((1 << FGRAPH_DATA_SIZE) - 1) #define FGRAPH_DATA_SHIFT (FGRAPH_TYPE_SHIFT + FGRAPH_TYPE_SIZE) @@ -113,7 +113,7 @@ enum { #define RET_STACK(t, index) ((struct ftrace_ret_stack *)(&(t)->ret_stack[index])) -#define FGRAPH_MAX_DATA_SIZE (sizeof(long) * 4) +#define FGRAPH_MAX_DATA_SIZE (sizeof(long) * (1 << FGRAPH_DATA_SIZE)) /* * Each fgraph_ops has a reservered unsigned long at the end (top) of the From patchwork Sun Nov 5 16:10:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 161679 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp2198166vqu; Sun, 5 Nov 2023 08:11:44 -0800 (PST) X-Google-Smtp-Source: AGHT+IErIGFFPkx0uPyCKthkhd9yqbNYHNGitOh3WPjEmgEphUy8m02JWzG2j6hiBHWjR4+ZBuga X-Received: by 2002:a05:6a21:18c:b0:136:faec:a7dc with SMTP id le12-20020a056a21018c00b00136faeca7dcmr35002650pzb.11.1699200704567; Sun, 05 Nov 2023 08:11:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699200704; cv=none; d=google.com; s=arc-20160816; b=EsVkqCz1ULxpOqUdKpuXW/ucIctV94QU48oPpq8k1snU9oNlU4h2ZfecQwnoZmuq1n FI7a8u81yyRxQpBH2fhAQyN7WoARoUvLdy/L8+I08C9ePacUNLy6p/AryzVvVbmH9IYS g4VD58r5InAh0q+Hfl5enxwkSI04QNzkrHo/e5CGEYl/lam7qQQhr9pSPCCb2rqE62X/ wxae6yPDNrqGFNmk4SfCJ7HjPsVP8ZvLI5O9FzSli4xILH0XY3c/HDFUOXhV8YDWIqhd URiP9FoA4aU6fy1aTtcvsnlSMExjjqg8ieqjt+kYcC9DKV5oTmRWStSO9HguAx7Kbin/ L6vw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=qXYWGaSpCAJD7G+RYxzNL5EqsiFeuysetLcT797rwaI=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=gSyBgnfZvXL7VMDbPWG5VGt8VTht2Hn7zGpieYeAaIasmKfziKJs3QMN7COH7iuD+4 vysIpScEVAigOEdfoffkWmhzhrzritAbVn80eJtYntvdvcKGmrv/rqUv+qmplvRIdFHK 76G2pp7UmlFUaIo4BOJOYsjY/YfF+xxf/eFPaQopj3SSCAWjNhdxQ5mL7zrkp5je98Hu K+TS03ynukRhihdu6uod5pgb8tB4VcE+dG9zsOV7wW3hlbwS/OapofUKbvioTH5XIQW9 yTxCpbOfvI3zXXuwMh0bsTIFbVX1VXO37KbpeQ9j/gbeKdHkU+3kNjO+bx/hneEZLWKX vCFw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=HPhcet3X; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id d12-20020a056a00244c00b006c128a8123bsi6066173pfj.212.2023.11.05.08.11.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Nov 2023 08:11:44 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=HPhcet3X; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id DEBF3808D28F; Sun, 5 Nov 2023 08:11:37 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229887AbjKEQLP (ORCPT + 34 others); Sun, 5 Nov 2023 11:11:15 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38170 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229889AbjKEQLG (ORCPT ); Sun, 5 Nov 2023 11:11:06 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A5479D69 for ; Sun, 5 Nov 2023 08:11:03 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1C1A4C433C8; Sun, 5 Nov 2023 16:10:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1699200663; bh=N2FbMszKmtxz9sJJDwAq7xsvDJ6ifpAiDx2oD0QYQXc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HPhcet3X6iqX6Dj3F1igf8Ms4e3nXVJOg982XUl723jNVLyYzAasXNW/EacNsBdPq OCy0SShrsPwqdP9TIu/fs8bPzBL/YbgBK25AO9OdZ11F4xLsP6yHwQpwcn/7ugMVzt amo3Im0ay5mkZXuQfcxKMJHOPZNLfGuTNKjfxgiePZLitIZhJ24Oq0CuJYdBzvb+MX UGhG0V8IDRwtuOoVvTRWtM9jKPCbJ00SnumSHyVOUS/3c6SR4ly+ywICBcsy4+8XXB YmZNa5mZvaM3zjgjqOn2LaWr1P1ssXaj+lJjaZSMV1n1zpGCQ1Wt2Y0P7+6dg25DXQ 83jV5m+auKS5A== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [RFC PATCH 22/32] function_graph: Add a new entry handler with parent_ip and ftrace_regs Date: Mon, 6 Nov 2023 01:10:56 +0900 Message-Id: <169920065596.482486.940814934361190165.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169920038849.482486.15796387219966662967.stgit@devnote2> References: <169920038849.482486.15796387219966662967.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Sun, 05 Nov 2023 08:11:38 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781741078414358555 X-GMAIL-MSGID: 1781741078414358555 From: Masami Hiramatsu (Google) Add a new entry handler to fgraph_ops as 'entryregfunc' which takes parent_ip and ftrace_regs. Note that the 'entryfunc' and 'entryregfunc' are mutual exclusive. You can set only one of them. Signed-off-by: Masami Hiramatsu (Google) --- arch/arm64/kernel/ftrace.c | 8 ++++- arch/loongarch/kernel/ftrace_dyn.c | 6 +++- arch/powerpc/kernel/trace/ftrace.c | 2 + arch/powerpc/kernel/trace/ftrace_64_pg.c | 10 ++++-- arch/x86/kernel/ftrace.c | 50 ++++++++++++++++++++---------- include/linux/ftrace.h | 17 +++++++++- kernel/trace/fgraph.c | 27 +++++++++++++--- 7 files changed, 90 insertions(+), 30 deletions(-) diff --git a/arch/arm64/kernel/ftrace.c b/arch/arm64/kernel/ftrace.c index a650f5e11fc5..e94d9dc13d89 100644 --- a/arch/arm64/kernel/ftrace.c +++ b/arch/arm64/kernel/ftrace.c @@ -481,7 +481,13 @@ void prepare_ftrace_return(unsigned long self_addr, unsigned long *parent, void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, struct ftrace_ops *op, struct ftrace_regs *fregs) { - prepare_ftrace_return(ip, &fregs->lr, fregs->fp); + if (unlikely(atomic_read(¤t->tracing_graph_pause))) + return; + + if (!function_graph_enter_regs(fregs->lr, ip, fregs->fp, + (void *)fregs->fp, fregs)) { + fregs->lr = (unsigned long)&return_to_handler; + } } #else /* diff --git a/arch/loongarch/kernel/ftrace_dyn.c b/arch/loongarch/kernel/ftrace_dyn.c index 73858c9029cc..39b3f09a5e0c 100644 --- a/arch/loongarch/kernel/ftrace_dyn.c +++ b/arch/loongarch/kernel/ftrace_dyn.c @@ -244,7 +244,11 @@ void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, struct pt_regs *regs = &fregs->regs; unsigned long *parent = (unsigned long *)®s->regs[1]; - prepare_ftrace_return(ip, (unsigned long *)parent); + if (unlikely(atomic_read(¤t->tracing_graph_pause))) + return; + + if (!function_graph_enter_regs(regs->regs[1], ip, 0, parent, fregs)) + regs->regs[1] = (unsigned long)&return_to_handler; } #else static int ftrace_modify_graph_caller(bool enable) diff --git a/arch/powerpc/kernel/trace/ftrace.c b/arch/powerpc/kernel/trace/ftrace.c index 82010629cf88..9bf1b6912116 100644 --- a/arch/powerpc/kernel/trace/ftrace.c +++ b/arch/powerpc/kernel/trace/ftrace.c @@ -422,7 +422,7 @@ void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, if (bit < 0) goto out; - if (!function_graph_enter(parent_ip, ip, 0, (unsigned long *)sp)) + if (!function_graph_enter_regs(parent_ip, ip, 0, (unsigned long *)sp, fregs)) parent_ip = ppc_function_entry(return_to_handler); ftrace_test_recursion_unlock(bit); diff --git a/arch/powerpc/kernel/trace/ftrace_64_pg.c b/arch/powerpc/kernel/trace/ftrace_64_pg.c index 7b85c3b460a3..43f6cfaaf7db 100644 --- a/arch/powerpc/kernel/trace/ftrace_64_pg.c +++ b/arch/powerpc/kernel/trace/ftrace_64_pg.c @@ -795,7 +795,8 @@ int ftrace_disable_ftrace_graph_caller(void) * in current thread info. Return the address we want to divert to. */ static unsigned long -__prepare_ftrace_return(unsigned long parent, unsigned long ip, unsigned long sp) +__prepare_ftrace_return(unsigned long parent, unsigned long ip, unsigned long sp, + struct ftrace_regs *fregs) { unsigned long return_hooker; int bit; @@ -812,7 +813,7 @@ __prepare_ftrace_return(unsigned long parent, unsigned long ip, unsigned long sp return_hooker = ppc_function_entry(return_to_handler); - if (!function_graph_enter(parent, ip, 0, (unsigned long *)sp)) + if (!function_graph_enter_regs(parent, ip, 0, (unsigned long *)sp, fregs)) parent = return_hooker; ftrace_test_recursion_unlock(bit); @@ -824,13 +825,14 @@ __prepare_ftrace_return(unsigned long parent, unsigned long ip, unsigned long sp void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, struct ftrace_ops *op, struct ftrace_regs *fregs) { - fregs->regs.link = __prepare_ftrace_return(parent_ip, ip, fregs->regs.gpr[1]); + fregs->regs.link = __prepare_ftrace_return(parent_ip, ip, + fregs->regs.gpr[1], fregs); } #else unsigned long prepare_ftrace_return(unsigned long parent, unsigned long ip, unsigned long sp) { - return __prepare_ftrace_return(parent, ip, sp); + return __prepare_ftrace_return(parent, ip, sp, NULL); } #endif #endif /* CONFIG_FUNCTION_GRAPH_TRACER */ diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c index 12df54ff0e81..85247a8f265b 100644 --- a/arch/x86/kernel/ftrace.c +++ b/arch/x86/kernel/ftrace.c @@ -614,16 +614,8 @@ int ftrace_disable_ftrace_graph_caller(void) } #endif /* CONFIG_DYNAMIC_FTRACE && !CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS */ -/* - * Hook the return address and push it in the stack of return addrs - * in current thread info. - */ -void prepare_ftrace_return(unsigned long ip, unsigned long *parent, - unsigned long frame_pointer) +static inline bool skip_ftrace_return(void) { - unsigned long return_hooker = (unsigned long)&return_to_handler; - int bit; - /* * When resuming from suspend-to-ram, this function can be indirectly * called from early CPU startup code while the CPU is in real mode, @@ -633,13 +625,28 @@ void prepare_ftrace_return(unsigned long ip, unsigned long *parent, * This check isn't as accurate as virt_addr_valid(), but it should be * good enough for this purpose, and it's fast. */ - if (unlikely((long)__builtin_frame_address(0) >= 0)) - return; + if ((long)__builtin_frame_address(0) >= 0) + return true; - if (unlikely(ftrace_graph_is_dead())) - return; + if (ftrace_graph_is_dead()) + return true; + + if (atomic_read(¤t->tracing_graph_pause)) + return true; + return false; +} + +/* + * Hook the return address and push it in the stack of return addrs + * in current thread info. + */ +void prepare_ftrace_return(unsigned long ip, unsigned long *parent, + unsigned long frame_pointer) +{ + unsigned long return_hooker = (unsigned long)&return_to_handler; + int bit; - if (unlikely(atomic_read(¤t->tracing_graph_pause))) + if (unlikely(skip_ftrace_return())) return; bit = ftrace_test_recursion_trylock(ip, *parent); @@ -657,9 +664,20 @@ void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, struct ftrace_ops *op, struct ftrace_regs *fregs) { struct pt_regs *regs = &fregs->regs; - unsigned long *stack = (unsigned long *)kernel_stack_pointer(regs); + unsigned long *parent = (unsigned long *)kernel_stack_pointer(regs); + int bit; - prepare_ftrace_return(ip, (unsigned long *)stack, 0); + if (unlikely(skip_ftrace_return())) + return; + + bit = ftrace_test_recursion_trylock(ip, *parent); + if (bit < 0) + return; + + if (!function_graph_enter_regs(*parent, ip, 0, parent, fregs)) + *parent = (unsigned long)&return_to_handler; + + ftrace_test_recursion_unlock(bit); } #endif diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 2e24a2611ca8..a6fd930d3500 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -1037,6 +1037,11 @@ typedef void (*trace_func_graph_ret_t)(struct ftrace_graph_ret *, typedef int (*trace_func_graph_ent_t)(struct ftrace_graph_ent *, struct fgraph_ops *); /* entry */ +typedef int (*trace_func_graph_regs_ent_t)(unsigned long func, + unsigned long parent_ip, + struct ftrace_regs *fregs, + struct fgraph_ops *); /* entry w/ regs */ + extern int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace, struct fgraph_ops *gops); #ifdef CONFIG_FUNCTION_GRAPH_TRACER @@ -1044,6 +1049,7 @@ extern int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace, struct fgraph struct fgraph_ops { trace_func_graph_ent_t entryfunc; trace_func_graph_ret_t retfunc; + trace_func_graph_regs_ent_t entryregfunc; struct ftrace_ops ops; /* for the hash lists */ void *private; int idx; @@ -1080,8 +1086,15 @@ struct ftrace_ret_stack { extern void return_to_handler(void); extern int -function_graph_enter(unsigned long ret, unsigned long func, - unsigned long frame_pointer, unsigned long *retp); +function_graph_enter_regs(unsigned long ret, unsigned long func, + unsigned long frame_pointer, unsigned long *retp, + struct ftrace_regs *fregs); + +static inline int function_graph_enter(unsigned long ret, unsigned long func, + unsigned long fp, unsigned long *retp) +{ + return function_graph_enter_regs(ret, func, fp, retp, NULL); +} struct ftrace_ret_stack * ftrace_graph_get_ret_stack(struct task_struct *task, int idx); diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index e7188c67356e..2623a5987f52 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -512,8 +512,20 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func, # define MCOUNT_INSN_SIZE 0 #endif -int function_graph_enter(unsigned long ret, unsigned long func, - unsigned long frame_pointer, unsigned long *retp) +static inline int call_entry_func(struct ftrace_graph_ent *trace, + unsigned long func, unsigned long ret, + struct ftrace_regs *fregs, + struct fgraph_ops *gops) +{ + if (gops->entryregfunc) + return gops->entryregfunc(func, ret, fregs, gops); + + return gops->entryfunc(trace, gops); +} + +int function_graph_enter_regs(unsigned long ret, unsigned long func, + unsigned long frame_pointer, unsigned long *retp, + struct ftrace_regs *fregs) { struct ftrace_graph_ent trace; int save_curr_ret_stack; @@ -558,7 +570,7 @@ int function_graph_enter(unsigned long ret, unsigned long func, } save_curr_ret_stack = current->curr_ret_stack; if (ftrace_ops_test(&gops->ops, func, NULL) && - gops->entryfunc(&trace, gops)) { + call_entry_func(&trace, func, ret, fregs, gops)) { /* Note, curr_ret_stack could change by enryfunc() */ offset = current->curr_ret_stack; /* Check the top level stored word */ @@ -876,7 +888,8 @@ void fgraph_init_ops(struct ftrace_ops *dst_ops, struct ftrace_ops *src_ops) { dst_ops->func = ftrace_graph_func; - dst_ops->flags = FTRACE_OPS_FL_PID | FTRACE_OPS_FL_STUB; + dst_ops->flags = FTRACE_OPS_FL_PID | FTRACE_OPS_FL_STUB | + FTRACE_OPS_FL_SAVE_ARGS; #ifdef FTRACE_GRAPH_TRAMP_ADDR dst_ops->trampoline = FTRACE_GRAPH_TRAMP_ADDR; @@ -1119,10 +1132,14 @@ int register_ftrace_graph(struct fgraph_ops *gops) int ret = 0; int i; + if (gops->entryfunc && gops->entryregfunc) + return -EINVAL; + mutex_lock(&ftrace_lock); if (!gops->ops.func) { - gops->ops.flags |= FTRACE_OPS_FL_STUB; + gops->ops.flags |= + FTRACE_OPS_FL_STUB | FTRACE_OPS_FL_SAVE_ARGS; gops->ops.func = ftrace_graph_func; #ifdef FTRACE_GRAPH_TRAMP_ADDR gops->ops.trampoline = FTRACE_GRAPH_TRAMP_ADDR; From patchwork Sun Nov 5 16:11:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 161680 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp2198186vqu; Sun, 5 Nov 2023 08:11:48 -0800 (PST) X-Google-Smtp-Source: AGHT+IFKUm8ZPkUEGo8eltp+PbWBCMrPOBZGC91IOSNaAc31mVYCjxC/LFLPx9ukuWGjV5yupzAI X-Received: by 2002:a05:6a20:7fa8:b0:184:26d8:cf47 with SMTP id d40-20020a056a207fa800b0018426d8cf47mr1605147pzj.34.1699200708327; Sun, 05 Nov 2023 08:11:48 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699200708; cv=none; d=google.com; s=arc-20160816; b=MT+wWH4L4GH7A6gtEQ6L6qXbEKU9etI/HK8A96nnM3Jc81HuBdH4FBufH0FB0M1YgP ibvBSll7z+YKPUmwmyFEXvGNCkyGIZ6KTyhsJJYKuLfTINPIiwJ+d0VkrkPn9+ZcZU9f GO7w8yTbwpoen+lSvhlzQItm/ZZa3z8XSZMwlUnWdEbpWjoE4FNj4d5nOW9ly1pmG+xi xtVvCuQ759iUGDEgY4kATONC5fRVrPq9H6brKcoCsupin2rq9RWb9Ts8yv+GYPPqz/VG cu0t8Ros66oMJDNL0dGna73sQaNiipCbvU+sOOzdXaN54egwGSOjhzUAXXygvQ+pcZ50 brcQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=VGj5uPXsoY0pVhFpzUKVjeU/JzT85wnLNTRq0ZgF/00=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=b7PZgW/HceB8QuELxaaBhe6TdbMxUrbTDTBJLAQIIXZvrXkUBhv7N5K3JyoPOjfcMQ 5GvNHuKUb9M33M512iHd2bc3mHFQ0/0FkwgQwASwYiaT4o+qbxBnx8zli8o/FgYRgTRY kkdt9E22av+QyoD4a02Axwz2r/Huj+0/uU75guanTnTiXnCGgdNz2PCYAQ5ayhfKVqcy xZbNi/xKbmEljwd2532opXY84fII+CjfT5D1jf0HjxQGuj2nKbejNOcmKQoibFGHMPvc ITD3HHkQ/oUA/6qqDlMivlydMAh6rHN1IfFT3jCXstZ/J4SjGSbIVft5mufR05YpNHYH jQmw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=r0T5omuk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id p14-20020a17090adf8e00b002800fffeb61si6262353pjv.5.2023.11.05.08.11.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Nov 2023 08:11:48 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=r0T5omuk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 056CE8057435; Sun, 5 Nov 2023 08:11:45 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229791AbjKEQL2 (ORCPT + 34 others); Sun, 5 Nov 2023 11:11:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38102 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229556AbjKEQLS (ORCPT ); Sun, 5 Nov 2023 11:11:18 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9EDF3100 for ; Sun, 5 Nov 2023 08:11:15 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 70CD2C433C7; Sun, 5 Nov 2023 16:11:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1699200675; bh=jG7ymSTJt2pOrHfOUVqQ7eTEuAUumsUaG6MXhuKCWzc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=r0T5omuklfLoUmQqMg+rXOMSSNbY/Cbw1OpcsjU6AcZb/VpvgBDSyEFREJz2FBkYF MTH20F4bneRLq+bGlm9lnSCa23FSwvi6HQLsfPDWKqPtKfJ5CadXhj+koK7qx+8g8I LdmUuo6VqLJT2VnXGeetvVCeXiYnw4NiK3ZyyKFhuXTxYf6/kmS3lh3CW6aIbdF4Di nZef1GgIj0UOTzDauZOHZYpjJbnZh1iggLXqRhaQv87tYv10Vo4SP1HEN/PHslrB5s lhdLELn6U/qbVkZXbznTJsJVxpgzQ/sUscSCRBl0aykMbqZxXOX5KmecPyL85dSi7b S8JssNdqbI0CQ== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [RFC PATCH 23/32] function_graph: Add a new exit handler with parent_ip and ftrace_regs Date: Mon, 6 Nov 2023 01:11:08 +0900 Message-Id: <169920066851.482486.17730951166172619260.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169920038849.482486.15796387219966662967.stgit@devnote2> References: <169920038849.482486.15796387219966662967.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Sun, 05 Nov 2023 08:11:45 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781741081989637746 X-GMAIL-MSGID: 1781741081989637746 From: Masami Hiramatsu (Google) Add a new return handler to fgraph_ops as 'retregfunc' which takes parent_ip and ftrace_regs instead of ftrace_graph_ret. This handler is available only if the arch support CONFIG_HAVE_FUNCTION_GRAPH_FREGS. Note that the 'retfunc' and 'reregfunc' are mutual exclusive. You can set only one of them. Signed-off-by: Masami Hiramatsu (Google) --- arch/x86/include/asm/ftrace.h | 2 + include/linux/ftrace.h | 10 +++++- kernel/trace/Kconfig | 5 ++- kernel/trace/fgraph.c | 74 +++++++++++++++++++++++++++-------------- 4 files changed, 63 insertions(+), 28 deletions(-) diff --git a/arch/x86/include/asm/ftrace.h b/arch/x86/include/asm/ftrace.h index 897cf02c20b1..74b1d245c38b 100644 --- a/arch/x86/include/asm/ftrace.h +++ b/arch/x86/include/asm/ftrace.h @@ -66,6 +66,8 @@ arch_ftrace_get_regs(struct ftrace_regs *fregs) override_function_with_return(&(fregs)->regs) #define ftrace_regs_query_register_offset(name) \ regs_query_register_offset(name) +#define ftrace_regs_get_frame_pointer(fregs) \ + frame_pointer(&(fregs)->regs) struct ftrace_ops; #define ftrace_graph_func ftrace_graph_func diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index a6fd930d3500..0c036c5d7c12 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -43,7 +43,9 @@ struct dyn_ftrace; char *arch_ftrace_match_adjust(char *str, const char *search); -#ifdef CONFIG_HAVE_FUNCTION_GRAPH_RETVAL +#ifdef CONFIG_HAVE_FUNCTION_GRAPH_FREGS +unsigned long ftrace_return_to_handler(struct ftrace_regs *fregs); +#elif defined(CONFIG_HAVE_FUNCTION_GRAPH_RETVAL) struct fgraph_ret_regs; unsigned long ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs); #else @@ -131,6 +133,7 @@ struct ftrace_regs { #define ftrace_regs_set_instruction_pointer(fregs, ip) do { } while (0) #endif /* CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS */ + static __always_inline struct pt_regs *ftrace_get_regs(struct ftrace_regs *fregs) { if (!fregs) @@ -1041,6 +1044,10 @@ typedef int (*trace_func_graph_regs_ent_t)(unsigned long func, unsigned long parent_ip, struct ftrace_regs *fregs, struct fgraph_ops *); /* entry w/ regs */ +typedef void (*trace_func_graph_regs_ret_t)(unsigned long func, + unsigned long parent_ip, + struct ftrace_regs *, + struct fgraph_ops *); /* return w/ regs */ extern int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace, struct fgraph_ops *gops); @@ -1050,6 +1057,7 @@ struct fgraph_ops { trace_func_graph_ent_t entryfunc; trace_func_graph_ret_t retfunc; trace_func_graph_regs_ent_t entryregfunc; + trace_func_graph_regs_ret_t retregfunc; struct ftrace_ops ops; /* for the hash lists */ void *private; int idx; diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig index 61c541c36596..308b3bec01b1 100644 --- a/kernel/trace/Kconfig +++ b/kernel/trace/Kconfig @@ -34,6 +34,9 @@ config HAVE_FUNCTION_GRAPH_TRACER config HAVE_FUNCTION_GRAPH_RETVAL bool +config HAVE_FUNCTION_GRAPH_FREGS + bool + config HAVE_DYNAMIC_FTRACE bool help @@ -232,7 +235,7 @@ config FUNCTION_GRAPH_TRACER config FUNCTION_GRAPH_RETVAL bool "Kernel Function Graph Return Value" - depends on HAVE_FUNCTION_GRAPH_RETVAL + depends on HAVE_FUNCTION_GRAPH_RETVAL || HAVE_FUNCTION_GRAPH_FREGS depends on FUNCTION_GRAPH_TRACER default n help diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index 2623a5987f52..c76fb2f33437 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -624,8 +624,7 @@ int function_graph_enter_regs(unsigned long ret, unsigned long func, /* Retrieve a function return address to the trace stack on thread info.*/ static struct ftrace_ret_stack * -ftrace_pop_return_trace(struct ftrace_graph_ret *trace, unsigned long *ret, - unsigned long frame_pointer) +ftrace_pop_return_trace(unsigned long *ret, unsigned long frame_pointer) { struct ftrace_ret_stack *ret_stack; int index; @@ -670,10 +669,6 @@ ftrace_pop_return_trace(struct ftrace_graph_ret *trace, unsigned long *ret, #endif *ret = ret_stack->ret; - trace->func = ret_stack->func; - trace->calltime = ret_stack->calltime; - trace->overrun = atomic_read(¤t->trace_overrun); - trace->depth = current->curr_ret_depth; /* * We still want to trace interrupts coming in if * max_depth is set to 1. Make sure the decrement is @@ -712,22 +707,43 @@ static struct notifier_block ftrace_suspend_notifier = { /* fgraph_ret_regs is not defined without CONFIG_FUNCTION_GRAPH_RETVAL */ struct fgraph_ret_regs; +static void fgraph_call_retfunc(struct ftrace_regs *fregs, + struct fgraph_ret_regs *ret_regs, + struct ftrace_ret_stack *ret_stack, + struct fgraph_ops *gops) +{ + struct ftrace_graph_ret trace; + + trace.func = ret_stack->func; + trace.calltime = ret_stack->calltime; + trace.overrun = atomic_read(¤t->trace_overrun); + trace.depth = current->curr_ret_depth; + trace.rettime = trace_clock_local(); +#ifdef CONFIG_FUNCTION_GRAPH_RETVAL + if (fregs) + trace.retval = ftrace_regs_return_value(fregs); + else + trace.retval = fgraph_ret_regs_return_value(ret_regs); +#endif + gops->retfunc(&trace, gops); +} + /* * Send the trace to the ring-buffer. * @return the original return address. */ -static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs, +static unsigned long __ftrace_return_to_handler(struct ftrace_regs *fregs, + struct fgraph_ret_regs *ret_regs, unsigned long frame_pointer) { struct ftrace_ret_stack *ret_stack; - struct ftrace_graph_ret trace; - unsigned long ret; + struct fgraph_ops *gops; int curr_ret_stack; + unsigned long ret; int stop_at; int index; - int idx; - ret_stack = ftrace_pop_return_trace(&trace, &ret, frame_pointer); + ret_stack = ftrace_pop_return_trace(&ret, frame_pointer); if (unlikely(!ret_stack)) { ftrace_graph_stop(); @@ -736,11 +752,6 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs return (unsigned long)panic; } - trace.rettime = trace_clock_local(); -#ifdef CONFIG_FUNCTION_GRAPH_RETVAL - trace.retval = fgraph_ret_regs_return_value(ret_regs); -#endif - curr_ret_stack = current->curr_ret_stack; index = get_ret_stack_index(current, curr_ret_stack - 1); @@ -753,8 +764,11 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs val = current->ret_stack[curr_ret_stack - 1]; switch (__get_type(val)) { case FGRAPH_TYPE_ARRAY: - idx = __get_array(val); - fgraph_array[idx]->retfunc(&trace, fgraph_array[idx]); + gops = fgraph_array[__get_array(val)]; + if (gops->retregfunc) + gops->retregfunc(ret_stack->func, ret, fregs, gops); + else + fgraph_call_retfunc(fregs, ret_regs, ret_stack, gops); curr_ret_stack -= __get_index(val); break; case FGRAPH_TYPE_RESERVED: @@ -778,20 +792,22 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs return ret; } -/* - * After all architecures have selected HAVE_FUNCTION_GRAPH_RETVAL, we can - * leave only ftrace_return_to_handler(ret_regs). - */ -#ifdef CONFIG_HAVE_FUNCTION_GRAPH_RETVAL +#ifdef CONFIG_HAVE_FUNCTION_GRAPH_FREGS +unsigned long ftrace_return_to_handler(struct ftrace_regs *fregs) +{ + return __ftrace_return_to_handler(fregs, NULL, + ftrace_regs_get_frame_pointer(fregs)); +} +#elif defined(CONFIG_HAVE_FUNCTION_GRAPH_RETVAL) unsigned long ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs) { - return __ftrace_return_to_handler(ret_regs, + return __ftrace_return_to_handler(NULL, ret_regs, fgraph_ret_regs_frame_pointer(ret_regs)); } #else unsigned long ftrace_return_to_handler(unsigned long frame_pointer) { - return __ftrace_return_to_handler(NULL, frame_pointer); + return __ftrace_return_to_handler(NULL, NULL, frame_pointer); } #endif @@ -1132,9 +1148,15 @@ int register_ftrace_graph(struct fgraph_ops *gops) int ret = 0; int i; - if (gops->entryfunc && gops->entryregfunc) + if ((gops->entryfunc && gops->entryregfunc) || + (gops->retfunc && gops->retregfunc)) return -EINVAL; +#ifndef CONFIG_HAVE_FUNCTION_GRAPH_FREGS + if (gops->retregfunc) + return -EOPNOTSUPP; +#endif + mutex_lock(&ftrace_lock); if (!gops->ops.func) { From patchwork Sun Nov 5 16:11:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 161681 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp2198314vqu; Sun, 5 Nov 2023 08:12:04 -0800 (PST) X-Google-Smtp-Source: AGHT+IHRgUZo5HDXxBJkMZpWni+BmsLqYjAYUqfULFBRPvETdvGz3UqKK4TW0/iBl2S/WKQ4iYcC X-Received: by 2002:a05:6602:2e0a:b0:79f:da3d:c47f with SMTP id o10-20020a0566022e0a00b0079fda3dc47fmr35412425iow.13.1699200724348; Sun, 05 Nov 2023 08:12:04 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699200724; cv=none; d=google.com; s=arc-20160816; b=PeNOiw/OuIVovyWEUJJQdxXJekKMg2ptJ7zQp1XRln2oqx/hvI6jUTIY3uUuFSII+C 1dMIN3Pjhgjh8L5hbXTpLBE9As2KQ/lor8QIMNgS3ocwtRBJ2s5phCWJykVYspb2Ao/C PAWyAveGVxv4g9g9CGkmbQ/EIICdEeyCed1ClAi5AM+SVDZBkETbpjtrChK/Dn0VESaY DLRcso/aQsgBWpDVY8a4YIsPDN4OzGf3E0uWmWqDvsNTSbzT5Q3zosAFOO2dLGL04OEy sZP0vxptdqkcdBPZ2+rTIAjHUrdKd5ZjdBdkocUgi/r2QHLiexAPKl9P4/tLa3nMIsfX ihbQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=ypeCKydyuVrJMXXG3zrJT9IZ2I/o5mVj+aArxSLXgc4=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=AsO2OjTvCcyPDnwAYQKmgWktUxTGiHUgSlv2GB/ystBcJxI1G3sffymrTZl1fgwf7I 4UTrUxjV+bbuDKCiU5xiwTIKmBWqcUPv3YmB2eMtXf+5yB1LlmQvmQpruX7fX40yP9uK UQSzVip6DW5V7INWovFlKAdf4Q4+dC9CEIMH4pBAjGzqNjgrEa+jHgs5etXHcnRx8rBx Lnk+U70aGkagTG1XMcfS4w7tZijxIs4T5+N43ALbiSpxRnhasu/W8ei67mYnj+duuIVk pjtkTCZVBZI2dM1sR3qCCwynfK0WCN+EK5FOvvU+J+OhAXpBU8ye8mCQ6ulGUwxs9FRN KlDA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Q5tYib6A; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from groat.vger.email (groat.vger.email. [2620:137:e000::3:5]) by mx.google.com with ESMTPS id i26-20020aa787da000000b00690ba709d02si5949396pfo.381.2023.11.05.08.12.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Nov 2023 08:12:04 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) client-ip=2620:137:e000::3:5; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Q5tYib6A; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 2CB29808D29E; Sun, 5 Nov 2023 08:11:57 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229549AbjKEQLi (ORCPT + 34 others); Sun, 5 Nov 2023 11:11:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38102 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229893AbjKEQLa (ORCPT ); Sun, 5 Nov 2023 11:11:30 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 32B19D61 for ; Sun, 5 Nov 2023 08:11:28 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B17FAC433C7; Sun, 5 Nov 2023 16:11:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1699200687; bh=6GBLrBVom66jXJpUAXFDmsg+lvD8A70XktPU9o8fCJA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Q5tYib6AKnSqhWsonloeJknpIvRW9fg3a+vNV71bChclAFYAYNRfWrO+MILCIcRpm 9JnCBUK/LFtRP+NJDxXpJqF71Vc7FTYssu9PmodHCpcyTaYx6srLQH7UcWrqkFtWUl R+Omc3GTxTMctRyBIwDdrCcDFNjbDpM+9p/7WFpcKHT116znYOD6sjIOLNpyfscsUl jaCntjjwMs2D2xtms6ZEAlAZjN1yPT9PwZ0AtgnooGw2PYgaCZQ3MsPZ/UOtAahxzX co+yneulf26eGDWF3flL5b/ZoPTQSdMRGjXN7tcqRqJXfZz0jinRYh+ge68PdKAOiA aJY0VKMZ5iJhg== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [RFC PATCH 24/32] x86/ftrace: Enable HAVE_FUNCTION_GRAPH_FREGS Date: Mon, 6 Nov 2023 01:11:21 +0900 Message-Id: <169920068069.482486.6540417903833579700.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169920038849.482486.15796387219966662967.stgit@devnote2> References: <169920038849.482486.15796387219966662967.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Sun, 05 Nov 2023 08:11:57 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781741098372018673 X-GMAIL-MSGID: 1781741098372018673 From: Masami Hiramatsu (Google) Support HAVE_FUNCTION_GRAPH_FREGS on x86-64, which saves ftrace_regs on the stack in ftrace_graph return trampoline so that the callbacks can access registers via ftrace_regs APIs. Note that this only recovers 'rax' and 'rdx' registers because other registers are not used anymore and recovered by caller. 'rax' and 'rdx' will be used for passing the return value. Signed-off-by: Masami Hiramatsu (Google) --- arch/x86/Kconfig | 3 ++- arch/x86/kernel/ftrace_64.S | 30 ++++++++++++++++++++++-------- 2 files changed, 24 insertions(+), 9 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 66bfabae8814..4b4c2f9d67da 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -219,7 +219,8 @@ config X86 select HAVE_FAST_GUP select HAVE_FENTRY if X86_64 || DYNAMIC_FTRACE select HAVE_FTRACE_MCOUNT_RECORD - select HAVE_FUNCTION_GRAPH_RETVAL if HAVE_FUNCTION_GRAPH_TRACER + select HAVE_FUNCTION_GRAPH_FREGS if HAVE_DYNAMIC_FTRACE_WITH_ARGS + select HAVE_FUNCTION_GRAPH_RETVAL if !HAVE_DYNAMIC_FTRACE_WITH_ARGS select HAVE_FUNCTION_GRAPH_TRACER if X86_32 || (X86_64 && DYNAMIC_FTRACE) select HAVE_FUNCTION_TRACER select HAVE_GCC_PLUGINS diff --git a/arch/x86/kernel/ftrace_64.S b/arch/x86/kernel/ftrace_64.S index 945cfa5f7239..41351a621753 100644 --- a/arch/x86/kernel/ftrace_64.S +++ b/arch/x86/kernel/ftrace_64.S @@ -348,21 +348,35 @@ STACK_FRAME_NON_STANDARD_FP(__fentry__) SYM_CODE_START(return_to_handler) UNWIND_HINT_UNDEFINED ANNOTATE_NOENDBR - subq $24, %rsp + /* + * We add enough stack to save all regs, but saves only registers + * for function params. + */ + subq $(FRAME_SIZE), %rsp + movq %rax, RAX(%rsp) + movq %rcx, RCX(%rsp) + movq %rdx, RDX(%rsp) + movq %rsi, RSI(%rsp) + movq %rdi, RDI(%rsp) + movq %r8, R8(%rsp) + movq %r9, R9(%rsp) + movq $0, ORIG_RAX(%rsp) + movq %rbp, RBP(%rsp) - /* Save the return values */ - movq %rax, (%rsp) - movq %rdx, 8(%rsp) - movq %rbp, 16(%rsp) movq %rsp, %rdi call ftrace_return_to_handler movq %rax, %rdi - movq 8(%rsp), %rdx - movq (%rsp), %rax - addq $24, %rsp + /* + * Restore only rax and rdx because other registers are not used + * for return value nor callee saved. Caller will reuse/recover it. + */ + movq RDX(%rsp), %rdx + movq RAX(%rsp), %rax + + addq $(FRAME_SIZE), %rsp /* * Jump back to the old return address. This cannot be JMP_NOSPEC rdi * since IBT would demand that contain ENDBR, which simply isn't so for From patchwork Sun Nov 5 16:11:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 161685 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp2198615vqu; Sun, 5 Nov 2023 08:12:40 -0800 (PST) X-Google-Smtp-Source: AGHT+IHzoJijym62p4kwKUgslLkkxk9kMEInFtGI1+KIZvdtCzlRHKBzU06/JGbHeCgOGiK0S1tw X-Received: by 2002:a05:6870:1191:b0:1ef:b109:b8ca with SMTP id 17-20020a056870119100b001efb109b8camr23069139oau.34.1699200760752; Sun, 05 Nov 2023 08:12:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699200760; cv=none; d=google.com; s=arc-20160816; b=iCpBD2mtnHkx3HcgM7ELtPL7E1LyHifebkvtIJ5TICQ/UqLcmyuVvGw9z1O58+8G0n MV59HpmDXoSSbf6LLZBDxWDUpYog3XGblK/Lsiz9QI4Fadfw2ZL9MRjNwrSmWQwAOXJe L1kUVPq5m3tNPmRxoMvfhiVgGJWG3Lc0RCVL84U+Cg1zKOfPsKLZrcOifYBheN66sTPg DxVXUMcHKpf0hYKURPZIGiwgA9scgvGVV16NQAoCvjus1tZXz5cObjcYfy6olmx+USJ3 FGmOQE16gk+CzONs5fZ6JNCWbiBABrWR7RgrejEhwzX+eVyryKD9+sa7wgDko76PM8EK mTLw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=+aT4nYCdwDxeICjZ/nlUFA4py1+K5KVW7e9Qgmix9rQ=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=KSHRQVxV/mVMzuZINZWVn4JRTDIAoyKIqpta1bQooMHkOggjrhvbjHftBKUisVpKKn wcZcm251l3CWeBhMJUPfKYVEqBbXPkeMoaLyD9lQ+c/Hu9phKXJPjYp7nxgSPIFLVPn+ IUbI2s/clP5Okc1hW3CKPHiMAV9zxf2eKDT3H97SlMHdHhpdgjfjHC8qn821ba0y6iPk SYPzsWlS2WqvdiZ+TO5g182OeFHnCoZq1AJBZXDr6PRoZFjbUDQww+YXGZUFlTHQuMG4 +MsPmyTJFHvvy4w5lxBWSgrN03syzYxmJN4CoFZt5hpww0SzguMQLRluYNl+7XRH3KO/ 8chQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=PfEV0Me1; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id y70-20020a638a49000000b005a1d8816535si6071192pgd.5.2023.11.05.08.12.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Nov 2023 08:12:40 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=PfEV0Me1; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id D362980615CB; Sun, 5 Nov 2023 08:12:33 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230009AbjKEQLv (ORCPT + 34 others); Sun, 5 Nov 2023 11:11:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52108 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229956AbjKEQLn (ORCPT ); Sun, 5 Nov 2023 11:11:43 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 77884D6F for ; Sun, 5 Nov 2023 08:11:40 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 03256C433C7; Sun, 5 Nov 2023 16:11:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1699200700; bh=8gPb8NMkwz/YkIjqBczF6oGA1ML3K/ChE49sxSPUXio=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PfEV0Me1rD04NbELo+EyxuF7j9Krjx7oCL22kfuNy+/30omHdeWkKd8zU6H0mI6Ll 4WKoiIWwlTjnpggGLeFgE3UTQ+KoVXAhkPWtWmDouUZ6MLobzpyl+bzzohxsDKxoHI wvIIlyHNHvroklF94XNgKeRS9hLkSwqBL66vSIvyAyNPfNFlpzirEyAWd4LdfqGdti hOloI9ABnZvSDu0X2t8sWBGSjg6lGkf52BVPeALNH8QKgAED5QRfnNlH2S1CnBzoE2 fugsMirufgv8GnygdftgalPt34ISkccS8pjcTeG4qsFGfB8NqumIKuVP1LTB8mxMnt suug144olPy5Q== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [RFC PATCH 25/32] fprobe: Use ftrace_regs in fprobe entry handler Date: Mon, 6 Nov 2023 01:11:33 +0900 Message-Id: <169920069297.482486.11530374089253107021.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169920038849.482486.15796387219966662967.stgit@devnote2> References: <169920038849.482486.15796387219966662967.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Sun, 05 Nov 2023 08:12:34 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781741136476466914 X-GMAIL-MSGID: 1781741136476466914 From: Masami Hiramatsu (Google) This allows fprobes to be available with CONFIG_DYNAMIC_FTRACE_WITH_ARGS instead of CONFIG_DYNAMIC_FTRACE_WITH_REGS, then we can enable fprobe on arm64. Signed-off-by: Masami Hiramatsu (Google) Acked-by: Florent Revest --- Changes from previous series: NOTHING, just forward ported. --- include/linux/fprobe.h | 2 +- kernel/trace/Kconfig | 3 ++- kernel/trace/bpf_trace.c | 10 +++++++--- kernel/trace/fprobe.c | 4 ++-- kernel/trace/trace_fprobe.c | 6 +++++- lib/test_fprobe.c | 4 ++-- samples/fprobe/fprobe_example.c | 2 +- 7 files changed, 20 insertions(+), 11 deletions(-) diff --git a/include/linux/fprobe.h b/include/linux/fprobe.h index 3e03758151f4..36c0595f7b93 100644 --- a/include/linux/fprobe.h +++ b/include/linux/fprobe.h @@ -35,7 +35,7 @@ struct fprobe { int nr_maxactive; int (*entry_handler)(struct fprobe *fp, unsigned long entry_ip, - unsigned long ret_ip, struct pt_regs *regs, + unsigned long ret_ip, struct ftrace_regs *regs, void *entry_data); void (*exit_handler)(struct fprobe *fp, unsigned long entry_ip, unsigned long ret_ip, struct pt_regs *regs, diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig index 308b3bec01b1..805d72ab77c6 100644 --- a/kernel/trace/Kconfig +++ b/kernel/trace/Kconfig @@ -290,7 +290,7 @@ config DYNAMIC_FTRACE_WITH_ARGS config FPROBE bool "Kernel Function Probe (fprobe)" depends on FUNCTION_TRACER - depends on DYNAMIC_FTRACE_WITH_REGS + depends on DYNAMIC_FTRACE_WITH_REGS || DYNAMIC_FTRACE_WITH_ARGS depends on HAVE_RETHOOK select RETHOOK default n @@ -675,6 +675,7 @@ config FPROBE_EVENTS select TRACING select PROBE_EVENTS select DYNAMIC_EVENTS + depends on DYNAMIC_FTRACE_WITH_REGS default y help This allows user to add tracing events on the function entry and diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index 868008f56fec..b0b98f0a0e52 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -2501,7 +2501,7 @@ static int __init bpf_event_init(void) fs_initcall(bpf_event_init); #endif /* CONFIG_MODULES */ -#ifdef CONFIG_FPROBE +#if defined(CONFIG_FPROBE) && defined(CONFIG_DYNAMIC_FTRACE_WITH_REGS) struct bpf_kprobe_multi_link { struct bpf_link link; struct fprobe fp; @@ -2729,10 +2729,14 @@ kprobe_multi_link_prog_run(struct bpf_kprobe_multi_link *link, static int kprobe_multi_link_handler(struct fprobe *fp, unsigned long fentry_ip, - unsigned long ret_ip, struct pt_regs *regs, + unsigned long ret_ip, struct ftrace_regs *fregs, void *data) { struct bpf_kprobe_multi_link *link; + struct pt_regs *regs = ftrace_get_regs(fregs); + + if (!regs) + return 0; link = container_of(fp, struct bpf_kprobe_multi_link, fp); kprobe_multi_link_prog_run(link, get_entry_ip(fentry_ip), regs); @@ -3004,7 +3008,7 @@ int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr kvfree(cookies); return err; } -#else /* !CONFIG_FPROBE */ +#else /* !CONFIG_FPROBE || !CONFIG_DYNAMIC_FTRACE_WITH_REGS */ int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog) { return -EOPNOTSUPP; diff --git a/kernel/trace/fprobe.c b/kernel/trace/fprobe.c index 881f90f0cbcf..df323cb7bed1 100644 --- a/kernel/trace/fprobe.c +++ b/kernel/trace/fprobe.c @@ -46,7 +46,7 @@ static inline void __fprobe_handler(unsigned long ip, unsigned long parent_ip, } if (fp->entry_handler) - ret = fp->entry_handler(fp, ip, parent_ip, ftrace_get_regs(fregs), entry_data); + ret = fp->entry_handler(fp, ip, parent_ip, fregs, entry_data); /* If entry_handler returns !0, nmissed is not counted. */ if (rh) { @@ -182,7 +182,7 @@ static void fprobe_init(struct fprobe *fp) fp->ops.func = fprobe_kprobe_handler; else fp->ops.func = fprobe_handler; - fp->ops.flags |= FTRACE_OPS_FL_SAVE_REGS; + fp->ops.flags |= FTRACE_OPS_FL_SAVE_ARGS; } static int fprobe_init_rethook(struct fprobe *fp, int num) diff --git a/kernel/trace/trace_fprobe.c b/kernel/trace/trace_fprobe.c index 8bfe23af9c73..71bf38d698f1 100644 --- a/kernel/trace/trace_fprobe.c +++ b/kernel/trace/trace_fprobe.c @@ -320,12 +320,16 @@ NOKPROBE_SYMBOL(fexit_perf_func); #endif /* CONFIG_PERF_EVENTS */ static int fentry_dispatcher(struct fprobe *fp, unsigned long entry_ip, - unsigned long ret_ip, struct pt_regs *regs, + unsigned long ret_ip, struct ftrace_regs *fregs, void *entry_data) { struct trace_fprobe *tf = container_of(fp, struct trace_fprobe, fp); + struct pt_regs *regs = ftrace_get_regs(fregs); int ret = 0; + if (!regs) + return 0; + if (trace_probe_test_flag(&tf->tp, TP_FLAG_TRACE)) fentry_trace_func(tf, entry_ip, regs); #ifdef CONFIG_PERF_EVENTS diff --git a/lib/test_fprobe.c b/lib/test_fprobe.c index 24de0e5ff859..ff607babba18 100644 --- a/lib/test_fprobe.c +++ b/lib/test_fprobe.c @@ -40,7 +40,7 @@ static noinline u32 fprobe_selftest_nest_target(u32 value, u32 (*nest)(u32)) static notrace int fp_entry_handler(struct fprobe *fp, unsigned long ip, unsigned long ret_ip, - struct pt_regs *regs, void *data) + struct ftrace_regs *fregs, void *data) { KUNIT_EXPECT_FALSE(current_test, preemptible()); /* This can be called on the fprobe_selftest_target and the fprobe_selftest_target2 */ @@ -81,7 +81,7 @@ static notrace void fp_exit_handler(struct fprobe *fp, unsigned long ip, static notrace int nest_entry_handler(struct fprobe *fp, unsigned long ip, unsigned long ret_ip, - struct pt_regs *regs, void *data) + struct ftrace_regs *fregs, void *data) { KUNIT_EXPECT_FALSE(current_test, preemptible()); return 0; diff --git a/samples/fprobe/fprobe_example.c b/samples/fprobe/fprobe_example.c index 64e715e7ed11..1545a1aac616 100644 --- a/samples/fprobe/fprobe_example.c +++ b/samples/fprobe/fprobe_example.c @@ -50,7 +50,7 @@ static void show_backtrace(void) static int sample_entry_handler(struct fprobe *fp, unsigned long ip, unsigned long ret_ip, - struct pt_regs *regs, void *data) + struct ftrace_regs *fregs, void *data) { if (use_trace) /* From patchwork Sun Nov 5 16:11:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 161683 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp2198483vqu; Sun, 5 Nov 2023 08:12:25 -0800 (PST) X-Google-Smtp-Source: AGHT+IEaL4fVMs0sd5UzWwn63BDUD1AIBYvYx2VcnxqTkYBF+k3FhYjzaoHcVVGDABjlx8RUkXaB X-Received: by 2002:a05:6870:36d5:b0:1e9:c28f:45ba with SMTP id u21-20020a05687036d500b001e9c28f45bamr29958780oak.16.1699200744875; Sun, 05 Nov 2023 08:12:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699200744; cv=none; d=google.com; s=arc-20160816; b=PfhjVwkx/sXtSEWdf/Qn2KIoF6RQwZUp0MTOi7Cvh+4wdzWCsjAPwn4fIQzD80LaOR FSs+NDK5PucEzo0mP+udIAap4moUjKh8+KFYdXZZc/AxRJX1areDrL2OCOKdGz6f3j1y TJRGBBlCGLy/zVjheUIvsSEeinPbNLqQQjj9pt9nxNCVhnH02XQrQxGdK6skrGBL4NGi WW7+g0U136uGQ0Rj9vc1r93gxHyA59MlgR8htcszSPW9Oioyu8JKUFCRXeoltTnhZgA+ q6kTFbI08pq1SudeEYMPQHqUlvpEDMccu+PPQADdzZs5oiTd0gy68IqvakOpIunaJ+hF 307w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=HGBI+OMroy4mVLeU09P1HW/yqDtM3NWmC/IHb5GUjVI=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=bNrP5o9piFmmfU/JCqa5ZCfVya/Mg3Awh89bZcY48aXAc58JX0947fqViwpiP0476R KCasiHRoGT50iGUL+jBHx3RA31heGTOxPLtEucruJ7bAvISA836NSkVJ+m0794UHcL6r KuIaYg4loojkjT2AsW2PJntxKHQbpznF3wSJaLaGFTUptzW1l1U+kSExRpf5wuPw8SuV YWEPz9twRS4JqFYZ21SaoGN1/Lj+eDooONJl/1ArOyLeyLbwUpXPLvfrY6FZuUw7pDqt EOqXZvxdxRW15qHNI+fizAHhbZRnim5yEXQwlisYoRAPXAVFbjVqiVvAqhrjeX3liX8J E8Tw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=rYo4Z7at; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id h129-20020a636c87000000b005898cf1c6a0si5757066pgc.324.2023.11.05.08.12.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Nov 2023 08:12:24 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=rYo4Z7at; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id E66758057435; Sun, 5 Nov 2023 08:12:19 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229811AbjKEQMG (ORCPT + 34 others); Sun, 5 Nov 2023 11:12:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46530 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229943AbjKEQL4 (ORCPT ); Sun, 5 Nov 2023 11:11:56 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8C9D5D4D for ; Sun, 5 Nov 2023 08:11:52 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5FA5EC433C7; Sun, 5 Nov 2023 16:11:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1699200712; bh=Cr9Oh59+ohU2jFV47Vs4H1il/Q3R6998uAfyDbnHEc0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rYo4Z7at845ASsIoynroABkmgczycN7RyqMbNfN6diE/z6PkrIHoWfxj4nceVVghT 3UGMTUJll68lNHwsMHhUko5yh+/S/XLbUra463xotZAgcy5W5v0SN5tKR4OzHRfyEq B+RnmPSqyuG9x7nYKwbWht8LLObrV5wUEL8jojleMI4mma/yS5yEPXILt+Qvwl0I9h maKmvDYEgN9KJ4eCLJ2knhyOVpHXaxl85VA9AFJf1G0Op/SCh/bkx0XoNLjdme9kC/ 4TL/ldtNRobde3vF5wWlMrogY+tGBrTFiaRZZ8LWKD7dfc5/IzLXQeBnwN7sCoun3V o9u01N4eV6xVA== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [RFC PATCH 26/32] fprobe: Use ftrace_regs in fprobe exit handler Date: Mon, 6 Nov 2023 01:11:45 +0900 Message-Id: <169920070544.482486.4637821890687747577.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169920038849.482486.15796387219966662967.stgit@devnote2> References: <169920038849.482486.15796387219966662967.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Sun, 05 Nov 2023 08:12:20 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781741120074269856 X-GMAIL-MSGID: 1781741120074269856 From: Masami Hiramatsu (Google) Change the fprobe exit handler to use ftrace_regs structure instead of pt_regs. This also introduce HAVE_PT_REGS_TO_FTRACE_REGS_CAST which means the ftrace_regs's memory layout is equal to the pt_regs so that those are able to cast. Fprobe introduces a new dependency with that. Signed-off-by: Masami Hiramatsu (Google) --- Changes from previous series: NOTHING, just forward ported. --- arch/loongarch/Kconfig | 1 + arch/s390/Kconfig | 1 + arch/x86/Kconfig | 1 + include/linux/fprobe.h | 2 +- include/linux/ftrace.h | 5 +++++ kernel/trace/Kconfig | 8 ++++++++ kernel/trace/bpf_trace.c | 6 +++++- kernel/trace/fprobe.c | 3 ++- kernel/trace/trace_fprobe.c | 6 +++++- lib/test_fprobe.c | 6 +++--- samples/fprobe/fprobe_example.c | 2 +- 11 files changed, 33 insertions(+), 8 deletions(-) diff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig index e14396a2ddcb..258e9bee1503 100644 --- a/arch/loongarch/Kconfig +++ b/arch/loongarch/Kconfig @@ -108,6 +108,7 @@ config LOONGARCH select HAVE_DMA_CONTIGUOUS select HAVE_DYNAMIC_FTRACE select HAVE_DYNAMIC_FTRACE_WITH_ARGS + select HAVE_PT_REGS_TO_FTRACE_REGS_CAST select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS select HAVE_DYNAMIC_FTRACE_WITH_REGS select HAVE_EBPF_JIT diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig index ae29e4392664..5aedb4320e7c 100644 --- a/arch/s390/Kconfig +++ b/arch/s390/Kconfig @@ -167,6 +167,7 @@ config S390 select HAVE_DMA_CONTIGUOUS select HAVE_DYNAMIC_FTRACE select HAVE_DYNAMIC_FTRACE_WITH_ARGS + select HAVE_PT_REGS_TO_FTRACE_REGS_CAST select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS select HAVE_DYNAMIC_FTRACE_WITH_REGS select HAVE_EBPF_JIT if HAVE_MARCH_Z196_FEATURES diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 4b4c2f9d67da..e11ba9a33afe 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -209,6 +209,7 @@ config X86 select HAVE_DYNAMIC_FTRACE select HAVE_DYNAMIC_FTRACE_WITH_REGS select HAVE_DYNAMIC_FTRACE_WITH_ARGS if X86_64 + select HAVE_PT_REGS_TO_FTRACE_REGS_CAST if X86_64 select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS select HAVE_SAMPLE_FTRACE_DIRECT if X86_64 select HAVE_SAMPLE_FTRACE_DIRECT_MULTI if X86_64 diff --git a/include/linux/fprobe.h b/include/linux/fprobe.h index 36c0595f7b93..879a30956009 100644 --- a/include/linux/fprobe.h +++ b/include/linux/fprobe.h @@ -38,7 +38,7 @@ struct fprobe { unsigned long ret_ip, struct ftrace_regs *regs, void *entry_data); void (*exit_handler)(struct fprobe *fp, unsigned long entry_ip, - unsigned long ret_ip, struct pt_regs *regs, + unsigned long ret_ip, struct ftrace_regs *fregs, void *entry_data); }; diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 0c036c5d7c12..894117e909dd 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -133,6 +133,11 @@ struct ftrace_regs { #define ftrace_regs_set_instruction_pointer(fregs, ip) do { } while (0) #endif /* CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS */ +#ifdef CONFIG_HAVE_PT_REGS_TO_FTRACE_REGS_CAST + +static_assert(sizeof(struct pt_regs) == sizeof(struct ftrace_regs)); + +#endif /* CONFIG_HAVE_PT_REGS_TO_FTRACE_REGS_CAST */ static __always_inline struct pt_regs *ftrace_get_regs(struct ftrace_regs *fregs) { diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig index 805d72ab77c6..1a2544712690 100644 --- a/kernel/trace/Kconfig +++ b/kernel/trace/Kconfig @@ -60,6 +60,13 @@ config HAVE_DYNAMIC_FTRACE_WITH_ARGS This allows for use of ftrace_regs_get_argument() and ftrace_regs_get_stack_pointer(). +config HAVE_PT_REGS_TO_FTRACE_REGS_CAST + bool + help + If this is set, the memory layout of the ftrace_regs data structure + is the same as the pt_regs. So the pt_regs is possible to be casted + to ftrace_regs. + config HAVE_DYNAMIC_FTRACE_NO_PATCHABLE bool help @@ -291,6 +298,7 @@ config FPROBE bool "Kernel Function Probe (fprobe)" depends on FUNCTION_TRACER depends on DYNAMIC_FTRACE_WITH_REGS || DYNAMIC_FTRACE_WITH_ARGS + depends on HAVE_PT_REGS_TO_FTRACE_REGS_CAST || !HAVE_DYNAMIC_FTRACE_WITH_ARGS depends on HAVE_RETHOOK select RETHOOK default n diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index b0b98f0a0e52..a4b5e34b0419 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -2745,10 +2745,14 @@ kprobe_multi_link_handler(struct fprobe *fp, unsigned long fentry_ip, static void kprobe_multi_link_exit_handler(struct fprobe *fp, unsigned long fentry_ip, - unsigned long ret_ip, struct pt_regs *regs, + unsigned long ret_ip, struct ftrace_regs *fregs, void *data) { struct bpf_kprobe_multi_link *link; + struct pt_regs *regs = ftrace_get_regs(fregs); + + if (!regs) + return; link = container_of(fp, struct bpf_kprobe_multi_link, fp); kprobe_multi_link_prog_run(link, get_entry_ip(fentry_ip), regs); diff --git a/kernel/trace/fprobe.c b/kernel/trace/fprobe.c index df323cb7bed1..38fe6a19450b 100644 --- a/kernel/trace/fprobe.c +++ b/kernel/trace/fprobe.c @@ -124,6 +124,7 @@ static void fprobe_exit_handler(struct rethook_node *rh, void *data, { struct fprobe *fp = (struct fprobe *)data; struct fprobe_rethook_node *fpr; + struct ftrace_regs *fregs = (struct ftrace_regs *)regs; int bit; if (!fp || fprobe_disabled(fp)) @@ -141,7 +142,7 @@ static void fprobe_exit_handler(struct rethook_node *rh, void *data, return; } - fp->exit_handler(fp, fpr->entry_ip, ret_ip, regs, + fp->exit_handler(fp, fpr->entry_ip, ret_ip, fregs, fp->entry_data_size ? (void *)fpr->data : NULL); ftrace_test_recursion_unlock(bit); } diff --git a/kernel/trace/trace_fprobe.c b/kernel/trace/trace_fprobe.c index 71bf38d698f1..c60d0d9f1a95 100644 --- a/kernel/trace/trace_fprobe.c +++ b/kernel/trace/trace_fprobe.c @@ -341,10 +341,14 @@ static int fentry_dispatcher(struct fprobe *fp, unsigned long entry_ip, NOKPROBE_SYMBOL(fentry_dispatcher); static void fexit_dispatcher(struct fprobe *fp, unsigned long entry_ip, - unsigned long ret_ip, struct pt_regs *regs, + unsigned long ret_ip, struct ftrace_regs *fregs, void *entry_data) { struct trace_fprobe *tf = container_of(fp, struct trace_fprobe, fp); + struct pt_regs *regs = ftrace_get_regs(fregs); + + if (!regs) + return; if (trace_probe_test_flag(&tf->tp, TP_FLAG_TRACE)) fexit_trace_func(tf, entry_ip, ret_ip, regs); diff --git a/lib/test_fprobe.c b/lib/test_fprobe.c index ff607babba18..d1e80653bf0c 100644 --- a/lib/test_fprobe.c +++ b/lib/test_fprobe.c @@ -59,9 +59,9 @@ static notrace int fp_entry_handler(struct fprobe *fp, unsigned long ip, static notrace void fp_exit_handler(struct fprobe *fp, unsigned long ip, unsigned long ret_ip, - struct pt_regs *regs, void *data) + struct ftrace_regs *fregs, void *data) { - unsigned long ret = regs_return_value(regs); + unsigned long ret = ftrace_regs_return_value(fregs); KUNIT_EXPECT_FALSE(current_test, preemptible()); if (ip != target_ip) { @@ -89,7 +89,7 @@ static notrace int nest_entry_handler(struct fprobe *fp, unsigned long ip, static notrace void nest_exit_handler(struct fprobe *fp, unsigned long ip, unsigned long ret_ip, - struct pt_regs *regs, void *data) + struct ftrace_regs *fregs, void *data) { KUNIT_EXPECT_FALSE(current_test, preemptible()); KUNIT_EXPECT_EQ(current_test, ip, target_nest_ip); diff --git a/samples/fprobe/fprobe_example.c b/samples/fprobe/fprobe_example.c index 1545a1aac616..d476d1f07538 100644 --- a/samples/fprobe/fprobe_example.c +++ b/samples/fprobe/fprobe_example.c @@ -67,7 +67,7 @@ static int sample_entry_handler(struct fprobe *fp, unsigned long ip, } static void sample_exit_handler(struct fprobe *fp, unsigned long ip, - unsigned long ret_ip, struct pt_regs *regs, + unsigned long ret_ip, struct ftrace_regs *regs, void *data) { unsigned long rip = ret_ip; From patchwork Sun Nov 5 16:11:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 161682 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp2198473vqu; Sun, 5 Nov 2023 08:12:23 -0800 (PST) X-Google-Smtp-Source: AGHT+IFZujAQELIdA3MT/09T/7j7hNWk+I9qoiqHM21ewAnNPzDIaH4Ljo8bQ14kDVZt5iFKXC6Y X-Received: by 2002:a17:902:e548:b0:1cc:4488:afba with SMTP id n8-20020a170902e54800b001cc4488afbamr11213697plf.6.1699200742590; Sun, 05 Nov 2023 08:12:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699200742; cv=none; d=google.com; s=arc-20160816; b=k4ixYL8PM+d2RS8g4RuChqMbeWjXryI7DgPYN+BGOpMou1E8TsdSrzpCJ1fJUlX2Gm B2PkqQ16NzwMNTVoQ5MRYlkAbEDM2pKxac8s7oK9EkC1qcXUpb7ynzWQMqrgKiNbaES0 U/ZGw+fk3ys4QvCDxASTI68vSkm2BO+xIQ3wt9My6uavEfI0RZSMuy/CQr1aW2wjN7yv oKL7RJzt+y+iwKBOSVXGy/mZ8ik5chG9ludDMA2iB9NQk2GHSkndhBirOae6xOSZ0gYj 48AF1oeSRNGpCOw4D+2MIwAPrIYm6oHyozLtoP6BOSMoYUCBY7xQzDavVYP8RQDxDePO 1KKg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=QjGXXbd2+S66OA9AIyzLhStxKNYnSxyrQSbswVhUyUM=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=nvXRNeA4oSAH5PyzGba5qo7T0JnwfnXJDwYQVZhvI54+980iwsj61O+RP1gECg6rsQ OLB73rYnRvRQsbhsBpDpqOQGSZBzFKVdHk/jr8m/g9RxjUzsiYoDzRucJfQgYQckf+qJ 0aGU1X4enb6Ql1AsUil/eDf/94xPw4HlVY0fL7YAMtxBL7Ih2TY65bZiMnKv8Ng47AnH Fz9KvIqVpLPQhZ6Z6lEVzm9O1yOroBhYDhrpLNU+aJ+Rd9j3qJxVLliLHkraH0gOQzZX nUQvt3hXlZpxHB12wUwKfzkHA8qx6OLPOOaz/grWG+Kk/6RDta96n6qN42eNKcUVhqBM AFlQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=bOu4qL9y; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id lh4-20020a170903290400b001cc3e75b2d2si5866781plb.446.2023.11.05.08.12.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Nov 2023 08:12:22 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=bOu4qL9y; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 5DCBC804705A; Sun, 5 Nov 2023 08:12:21 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229468AbjKEQMS (ORCPT + 34 others); Sun, 5 Nov 2023 11:12:18 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45636 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229850AbjKEQMH (ORCPT ); Sun, 5 Nov 2023 11:12:07 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AFE97123 for ; Sun, 5 Nov 2023 08:12:04 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 803E2C433C7; Sun, 5 Nov 2023 16:12:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1699200724; bh=vk7wW23Tf7Fx6d6+RxROO/tS9W9QJV09ekq203vNHlU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bOu4qL9yDmBwLqq8yLMtEshX1dvrrI23mX14nn5KTizYb62gVzWSwPBQ63E/L6wfH T/Mrt6azvN04jRkQrKAmFTJyuTLgLeBCwsDX0p6D/gKAloVK9nqSMQCXIbYZZpPFzL IVKIVLRra1NPM+Wg0wdGufy5qm2m4ml3mlYm4EVNU2+TdgCERNasGRXxOQfxYP7H/b DedDBIQe9cwh9/OvBxcXd/s3MFHFdxTXyLFE2ethoyrcaWcgrJzs6hQloqco4IFMpw MnPo9caW1dbtOi9FHePoVjuUyg2yAyNJveVhceuLRtuqe4aJ5V78xnmvHBhlsyxJAM hgE5NoBXcwIpQ== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [RFC PATCH 27/32] tracing: Add ftrace_partial_regs() for converting ftrace_regs to pt_regs Date: Mon, 6 Nov 2023 01:11:57 +0900 Message-Id: <169920071741.482486.15046208284937337125.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169920038849.482486.15796387219966662967.stgit@devnote2> References: <169920038849.482486.15796387219966662967.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Sun, 05 Nov 2023 08:12:21 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781741118157381366 X-GMAIL-MSGID: 1781741118157381366 From: Masami Hiramatsu (Google) Add ftrace_partial_regs() which converts the ftrace_regs to pt_regs. If the architecture defines its own ftrace_regs, this copies partial registers to pt_regs and returns it. If not, ftrace_regs is the same as pt_regs and ftrace_partial_regs() will return ftrace_regs::regs. Signed-off-by: Masami Hiramatsu (Google) Acked-by: Florent Revest --- Changes from previous series: NOTHING, just forward ported. --- arch/arm64/include/asm/ftrace.h | 11 +++++++++++ include/linux/ftrace.h | 17 +++++++++++++++++ 2 files changed, 28 insertions(+) diff --git a/arch/arm64/include/asm/ftrace.h b/arch/arm64/include/asm/ftrace.h index ab158196480c..5ad24f315d52 100644 --- a/arch/arm64/include/asm/ftrace.h +++ b/arch/arm64/include/asm/ftrace.h @@ -137,6 +137,17 @@ ftrace_override_function_with_return(struct ftrace_regs *fregs) fregs->pc = fregs->lr; } +static __always_inline struct pt_regs * +ftrace_partial_regs(const struct ftrace_regs *fregs, struct pt_regs *regs) +{ + memcpy(regs->regs, fregs->regs, sizeof(u64) * 9); + regs->sp = fregs->sp; + regs->pc = fregs->pc; + regs->regs[29] = fregs->fp; + regs->regs[30] = fregs->lr; + return regs; +} + int ftrace_regs_query_register_offset(const char *name); int ftrace_init_nop(struct module *mod, struct dyn_ftrace *rec); diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 894117e909dd..3d0ff6e2c4bf 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -147,6 +147,23 @@ static __always_inline struct pt_regs *ftrace_get_regs(struct ftrace_regs *fregs return arch_ftrace_get_regs(fregs); } +#if !defined(CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS) || \ + defined(CONFIG_HAVE_PT_REGS_TO_FTRACE_REGS_CAST) + +static __always_inline struct pt_regs * +ftrace_partial_regs(struct ftrace_regs *fregs, struct pt_regs *regs) +{ + /* + * If CONFIG_HAVE_PT_REGS_TO_FTRACE_REGS_CAST=y, ftrace_regs memory + * layout is the same as pt_regs. So always returns that address. + * Since arch_ftrace_get_regs() will check some members and may return + * NULL, we can not use it. + */ + return &fregs->regs; +} + +#endif /* !CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS || CONFIG_HAVE_PT_REGS_TO_FTRACE_REGS_CAST */ + /* * When true, the ftrace_regs_{get,set}_*() functions may be used on fregs. * Note: this can be true even when ftrace_get_regs() cannot provide a pt_regs. From patchwork Sun Nov 5 16:12:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 161684 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp2198591vqu; Sun, 5 Nov 2023 08:12:37 -0800 (PST) X-Google-Smtp-Source: AGHT+IE01hFL9759dZb/NSMuKUpwAx0vqmgf1If3icDQrf2u7oJMXKEtCmjPyxf6eqJQjeD2yFub X-Received: by 2002:a05:6a20:1595:b0:13f:b028:7892 with SMTP id h21-20020a056a20159500b0013fb0287892mr27874221pzj.2.1699200757524; Sun, 05 Nov 2023 08:12:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699200757; cv=none; d=google.com; s=arc-20160816; b=UWQ56N8oYADE6STafnz3OeEPLp0qyT5f1jPcGQa9LdVPOZRzJ2i4FdfpBZsxLcCtFA smUFSCIfXo4OFW8OchDUh6mrJr7UXUSThYWuswj6VVUTNX0hlHWqS7WTIwFUfXSxQZ4S TYL+d5ZoEGVeLtmJ+dP/4e2FLoYxyBZ8zSLCxj2V8TuSJL60f80kQ4VOr+UKzi26Zm8E zUwZHDMaGnq79P6a9+5dNqFcYaTepYw8upIqmNoLUpctt62m4tzvr6GiG8g0RMrHdFV9 RXMOydU71Y96jWuv4DJoc7FRmL5r19RY1UEh4u2im9dG9slcis2DFgwqqsTV6vR7yd07 BMVw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=ds1GWQtnrZuzkUdbbRabONL9CwQB5EM/SzCsNpocPFM=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=LHns7T6cFr03MWvo6U5Pxb6OC2IbgtVg3NYcrgjbOlyejoZTsfGaDoLwF1FnOI4Pe2 OIqLbjhdYpCqgHdLBhaudzV1rnkZjFXGTNMAemuKYCPeIQeZ4xisLqRv/d6VTEgdxcIt NGUINvOK8o1Hncq1XMhI4wJTqPCEScyou8PbdNoWs7DZTEpMVbSVsWDQU92bs6ghz1zc Ck2Le8TO425AU2Yf7uGxlnnjSbbBY4FvbKoHEfzlVF8fGAnTomCgGoSHeyE0YAH+bzHf IMYD0X2y5cEZym3P/yCYmxYVhGC1pmoiJk+xX2rFtfGAPgLT/B28fb931sT4WualInD4 FdQA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=UbPRS8N4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id z6-20020a170902d54600b001c9faf1439esi6907194plf.288.2023.11.05.08.12.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Nov 2023 08:12:37 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=UbPRS8N4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id CAD22804706F; Sun, 5 Nov 2023 08:12:36 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229893AbjKEQMb (ORCPT + 34 others); Sun, 5 Nov 2023 11:12:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42474 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229967AbjKEQMU (ORCPT ); Sun, 5 Nov 2023 11:12:20 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 014E4D57 for ; Sun, 5 Nov 2023 08:12:17 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C69A5C433C7; Sun, 5 Nov 2023 16:12:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1699200736; bh=P2BU0nAAbNWYgnCXzDHgoWO70BUk4qjNGqknsr0H9DE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UbPRS8N47dAHiTpXajqIuTM7JuV3ryxxsbQ3cuVeoBIaaQnzZ/DbTA6N0iD/U+Rl6 imyLtdvUb3Jo3Ll7yWLvobj2Os8ezSnjiqSPckHm1FpIo/wxi5oM9LOS61Gtc8wWdS WPo+i+sBYluqnzW0qVgRHgvQw3Mg70VVko69+O5o5pBOyftlrdN8gG+Zus5qPgE+eN xfPStKFIdTZt5+Ea1+XwdLADmMIRaPm7q5E6hK1OMdzBWZahp78sJmePl1NMD/Jd+l 4xuY26Sgbg+9hu8Y3h9qr4rbE4bbiDjPotwp+JvCgQOZuPQ2b9ndkd9G+8XCTJQNdg 9zc0ZWNFz7k9g== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [RFC PATCH 28/32] tracing: Add ftrace_fill_perf_regs() for perf event Date: Mon, 6 Nov 2023 01:12:10 +0900 Message-Id: <169920072976.482486.9502671314956022881.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169920038849.482486.15796387219966662967.stgit@devnote2> References: <169920038849.482486.15796387219966662967.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Sun, 05 Nov 2023 08:12:36 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781741133336105941 X-GMAIL-MSGID: 1781741133336105941 From: Masami Hiramatsu (Google) Add ftrace_fill_perf_regs() which should be compatible with the perf_fetch_caller_regs(). In other words, the pt_regs returned from the ftrace_fill_perf_regs() must satisfy 'user_mode(regs) == false' and can be used for stack tracing. Signed-off-by: Masami Hiramatsu (Google) --- Changes from previous series: NOTHING, just forward ported. --- arch/arm64/include/asm/ftrace.h | 7 +++++++ arch/powerpc/include/asm/ftrace.h | 7 +++++++ arch/s390/include/asm/ftrace.h | 5 +++++ arch/x86/include/asm/ftrace.h | 7 +++++++ include/linux/ftrace.h | 31 +++++++++++++++++++++++++++++++ 5 files changed, 57 insertions(+) diff --git a/arch/arm64/include/asm/ftrace.h b/arch/arm64/include/asm/ftrace.h index 5ad24f315d52..5fbe8fe3e8a5 100644 --- a/arch/arm64/include/asm/ftrace.h +++ b/arch/arm64/include/asm/ftrace.h @@ -148,6 +148,13 @@ ftrace_partial_regs(const struct ftrace_regs *fregs, struct pt_regs *regs) return regs; } +#define arch_ftrace_fill_perf_regs(fregs, _regs) do { \ + (_regs)->pc = (fregs)->pc; \ + (_regs)->regs[29] = (fregs)->fp; \ + (_regs)->sp = (fregs)->sp; \ + (_regs)->pstate = PSR_MODE_EL1h; \ + } while (0) + int ftrace_regs_query_register_offset(const char *name); int ftrace_init_nop(struct module *mod, struct dyn_ftrace *rec); diff --git a/arch/powerpc/include/asm/ftrace.h b/arch/powerpc/include/asm/ftrace.h index 9e5a39b6a311..53fa39ef476b 100644 --- a/arch/powerpc/include/asm/ftrace.h +++ b/arch/powerpc/include/asm/ftrace.h @@ -52,6 +52,13 @@ static __always_inline struct pt_regs *arch_ftrace_get_regs(struct ftrace_regs * return fregs->regs.msr ? &fregs->regs : NULL; } +#define arch_ftrace_fill_perf_regs(fregs, _regs) do { \ + (_regs)->result = 0; \ + (_regs)->nip = (fregs)->regs.nip; \ + (_regs)->gpr[1] = (fregs)->regs.gpr[1]; \ + asm volatile("mfmsr %0" : "=r" ((_regs)->msr)); \ + } while (0) + static __always_inline void ftrace_regs_set_instruction_pointer(struct ftrace_regs *fregs, unsigned long ip) diff --git a/arch/s390/include/asm/ftrace.h b/arch/s390/include/asm/ftrace.h index 5a82b08f03cd..f36ee3510af8 100644 --- a/arch/s390/include/asm/ftrace.h +++ b/arch/s390/include/asm/ftrace.h @@ -97,6 +97,11 @@ ftrace_regs_set_instruction_pointer(struct ftrace_regs *fregs, #define ftrace_regs_query_register_offset(name) \ regs_query_register_offset(name) +#define arch_ftrace_fill_perf_regs(fregs, _regs) do { \ + (_regs)->psw.addr = (fregs)->regs.psw.addr; \ + (_regs)->gprs[15] = (fregs)->regs.gprs[15]; \ + } while (0) + #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS /* * When an ftrace registered caller is tracing a function that is diff --git a/arch/x86/include/asm/ftrace.h b/arch/x86/include/asm/ftrace.h index 74b1d245c38b..66e7d2df7916 100644 --- a/arch/x86/include/asm/ftrace.h +++ b/arch/x86/include/asm/ftrace.h @@ -48,6 +48,13 @@ arch_ftrace_get_regs(struct ftrace_regs *fregs) return &fregs->regs; } +#define arch_ftrace_fill_perf_regs(fregs, _regs) do { \ + (_regs)->ip = (fregs)->regs.ip; \ + (_regs)->sp = (fregs)->regs.sp; \ + (_regs)->cs = __KERNEL_CS; \ + (_regs)->flags = 0; \ + } while (0) + #define ftrace_regs_set_instruction_pointer(fregs, _ip) \ do { (fregs)->regs.ip = (_ip); } while (0) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 3d0ff6e2c4bf..7db38341bc9e 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -164,6 +164,37 @@ ftrace_partial_regs(struct ftrace_regs *fregs, struct pt_regs *regs) #endif /* !CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS || CONFIG_HAVE_PT_REGS_TO_FTRACE_REGS_CAST */ +#ifdef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS + +/* + * Please define arch dependent pt_regs which compatible to the + * perf_arch_fetch_caller_regs() but based on ftrace_regs. + * This requires + * - user_mode(_regs) returns false (always kernel mode). + * - able to use the _regs for stack trace. + */ +#ifndef arch_ftrace_fill_perf_regs +/* As same as perf_arch_fetch_caller_regs(), do nothing by default */ +#define arch_ftrace_fill_perf_regs(fregs, _regs) do {} while (0) +#endif + +static __always_inline struct pt_regs * +ftrace_fill_perf_regs(struct ftrace_regs *fregs, struct pt_regs *regs) +{ + arch_ftrace_fill_perf_regs(fregs, regs); + return regs; +} + +#else /* !CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS */ + +static __always_inline struct pt_regs * +ftrace_fill_perf_regs(struct ftrace_regs *fregs, struct pt_regs *regs) +{ + return &fregs->regs; +} + +#endif + /* * When true, the ftrace_regs_{get,set}_*() functions may be used on fregs. * Note: this can be true even when ftrace_get_regs() cannot provide a pt_regs. From patchwork Sun Nov 5 16:12:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 161686 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp2198648vqu; Sun, 5 Nov 2023 08:12:46 -0800 (PST) X-Google-Smtp-Source: AGHT+IG14iGsfLXF/pqH6pLKTg/8aI7yobp593PW9MWPeJFOnwDQYi/YF2FxrPLJqB6g1tTOtyGA X-Received: by 2002:a17:90b:305:b0:280:2823:6612 with SMTP id ay5-20020a17090b030500b0028028236612mr16126493pjb.35.1699200766515; Sun, 05 Nov 2023 08:12:46 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699200766; cv=none; d=google.com; s=arc-20160816; b=gu7LnMVs2rGAHZvul0T6vz0SqwXGRyanE4l07JMt72Lpv9ctrVr5sxARECNv2zy9LD 8RvkjxLaPYBajRUgOegClw691CwOgpbIflGyePHF8e72Nx3mQQN2QywCmLRWrJG4vwOM zTDILxinbioeXHB3xTQdXLBKDRUjoScMlcCMJQ0XSKuhSx/NIByjrvATkzT7dt54+Fr4 fMITrdB0EoyU72lrwKLEis7PSBrwe4LcVHhi5cpLebuEnXtkt1spHGsi9GmHsxVGWbGl TPiQGFm0/j6oWJHLGW8CK75rEgF7G7g3JncrNTseTWF5nuA/PcPpp9VxSYZdjEUVwq7O kO6g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=XXDexNxvwLpofBFgrxPmuShLmGaAWRywnFAB9sfcCzk=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=XMGAs9LABayQY4WFypvrvfpE9mtngX9LeOHu540jCmkx4z8cwqkv5WWuALj86t8CA/ HFdkb/vHn3xygL8OOmE2Esj+HBFZT1pPD8WtTCwgpROevZyr/pGAiQ33QDuG0bkkEcr9 yq1R+1/YT2hnXS1PTV2gLSXDzqP6lwSEnN87qwkDZtRfTvUe6s9JMfZBjAdxwbV7XrIc nr3xxkh5Oy4lWy0K61nLsXelFCuCCavDO49dxkPal8uBdi7+KZlJlOztGgsj6nCUETRP nLcbOwhpxmC+9AMtrfNZVkCCTSip+9evRa/Fx7PLKcEuyJ50LNCBRgzy6h+J2r/jE1Vk BY9g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=ucBJ4lkU; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id y6-20020a17090ad70600b002802da109fasi6091929pju.165.2023.11.05.08.12.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Nov 2023 08:12:46 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=ucBJ4lkU; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 899E7804706D; Sun, 5 Nov 2023 08:12:45 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229992AbjKEQMj (ORCPT + 34 others); Sun, 5 Nov 2023 11:12:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36928 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229954AbjKEQMe (ORCPT ); Sun, 5 Nov 2023 11:12:34 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5BEC7123 for ; Sun, 5 Nov 2023 08:12:29 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E7860C433C7; Sun, 5 Nov 2023 16:12:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1699200749; bh=f/6EfO92DTmPO03jtbWqEihWJBI4nT7t5SWxGhuhVPI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ucBJ4lkUXtvhNTD0Ab4Oh2CC8OFb65Pms0TCmTJPBV1gcPP1Ka7At2rFR0rh88+ZY WgLf+Jn8V5Muc7cxkqonl1YnVHrYzVYF9veY+AAFwfvdlj/KUjID7+HliItU61c7Fe o/0BldoCuHGfC2VaiXkzB9g8LKggOFgGMNfpRFdMRB5ibaFqwkUMfXosCajW7GpRNS upB9A1FGkZZr/mjtn5y0ybi6Ig8X7XJYK5+UraiulJcBeklPR+hJSoBMy/Bk5PubnG Vyp8R3w7Bdq2wU4rlHuDd9nqndS2TMl0IKv8gAaUmlC29I0+9O60a2yRs7QwTq/Phk 205diapdYycEw== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [RFC PATCH 29/32] fprobe: Rewrite fprobe on function-graph tracer Date: Mon, 6 Nov 2023 01:12:22 +0900 Message-Id: <169920074205.482486.48764326873735843.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169920038849.482486.15796387219966662967.stgit@devnote2> References: <169920038849.482486.15796387219966662967.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Sun, 05 Nov 2023 08:12:45 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781741142960619351 X-GMAIL-MSGID: 1781741142960619351 From: Masami Hiramatsu (Google) Rewrite fprobe implementation on function-graph tracer. Major API changes are: - 'nr_maxactive' field is deprecated. - This depends on CONFIG_DYNAMIC_FTRACE_WITH_ARGS or !CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS, and CONFIG_HAVE_FUNCTION_GRAPH_FREGS. So currently works only on x86_64. - Currently the entry size is limited in 15 * sizeof(long). - If there is too many fprobe exit handler set on the same function, it will fail to probe. Signed-off-by: Masami Hiramatsu (Google) --- include/linux/fprobe.h | 54 +++- kernel/trace/Kconfig | 8 - kernel/trace/fprobe.c | 632 ++++++++++++++++++++++++++++++++++-------------- lib/test_fprobe.c | 45 --- 4 files changed, 487 insertions(+), 252 deletions(-) diff --git a/include/linux/fprobe.h b/include/linux/fprobe.h index 879a30956009..08b37b0d1d05 100644 --- a/include/linux/fprobe.h +++ b/include/linux/fprobe.h @@ -5,32 +5,56 @@ #include #include -#include +#include +#include +#include + +struct fprobe; + +/** + * strcut fprobe_hlist_node - address based hash list node for fprobe. + * + * @hlist: The hlist node for address search hash table. + * @addr: The address represented by this. + * @fp: The fprobe which owns this. + */ +struct fprobe_hlist_node { + struct hlist_node hlist; + unsigned long addr; + struct fprobe *fp; +}; + +/** + * struct fprobe_hlist - hash list nodes for fprobe. + * + * @hlist: The hlist node for existence checking hash table. + * @rcu: rcu_head for RCU deferred release. + * @fp: The fprobe which owns this fprobe_hlist. + * @size: The size of @array. + * @array: The fprobe_hlist_node for each address to probe. + */ +struct fprobe_hlist { + struct hlist_node hlist; + struct rcu_head rcu; + struct fprobe *fp; + int size; + struct fprobe_hlist_node array[]; +}; /** * struct fprobe - ftrace based probe. - * @ops: The ftrace_ops. + * * @nmissed: The counter for missing events. * @flags: The status flag. - * @rethook: The rethook data structure. (internal data) * @entry_data_size: The private data storage size. - * @nr_maxactive: The max number of active functions. + * @nr_maxactive: The max number of active functions. (*deprecated) * @entry_handler: The callback function for function entry. * @exit_handler: The callback function for function exit. + * @hlist_array: The fprobe_hlist for fprobe search from IP hash table. */ struct fprobe { -#ifdef CONFIG_FUNCTION_TRACER - /* - * If CONFIG_FUNCTION_TRACER is not set, CONFIG_FPROBE is disabled too. - * But user of fprobe may keep embedding the struct fprobe on their own - * code. To avoid build error, this will keep the fprobe data structure - * defined here, but remove ftrace_ops data structure. - */ - struct ftrace_ops ops; -#endif unsigned long nmissed; unsigned int flags; - struct rethook *rethook; size_t entry_data_size; int nr_maxactive; @@ -40,6 +64,8 @@ struct fprobe { void (*exit_handler)(struct fprobe *fp, unsigned long entry_ip, unsigned long ret_ip, struct ftrace_regs *fregs, void *entry_data); + + struct fprobe_hlist *hlist_array; }; /* This fprobe is soft-disabled. */ diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig index 1a2544712690..11a96275b68c 100644 --- a/kernel/trace/Kconfig +++ b/kernel/trace/Kconfig @@ -296,11 +296,9 @@ config DYNAMIC_FTRACE_WITH_ARGS config FPROBE bool "Kernel Function Probe (fprobe)" - depends on FUNCTION_TRACER - depends on DYNAMIC_FTRACE_WITH_REGS || DYNAMIC_FTRACE_WITH_ARGS - depends on HAVE_PT_REGS_TO_FTRACE_REGS_CAST || !HAVE_DYNAMIC_FTRACE_WITH_ARGS - depends on HAVE_RETHOOK - select RETHOOK + depends on FUNCTION_GRAPH_TRACER + depends on HAVE_FUNCTION_GRAPH_FREGS + depends on DYNAMIC_FTRACE_WITH_ARGS || !HAVE_DYNAMIC_FTRACE_WITH_ARGS default n help This option enables kernel function probe (fprobe) based on ftrace. diff --git a/kernel/trace/fprobe.c b/kernel/trace/fprobe.c index 38fe6a19450b..7a0c7a32bd03 100644 --- a/kernel/trace/fprobe.c +++ b/kernel/trace/fprobe.c @@ -8,98 +8,194 @@ #include #include #include -#include +#include +#include #include #include #include "trace.h" -struct fprobe_rethook_node { - struct rethook_node node; - unsigned long entry_ip; - unsigned long entry_parent_ip; - char data[]; -}; +#define FPROBE_IP_HASH_BITS 8 +#define FPROBE_IP_TABLE_SIZE (1 << FPROBE_IP_HASH_BITS) -static inline void __fprobe_handler(unsigned long ip, unsigned long parent_ip, - struct ftrace_ops *ops, struct ftrace_regs *fregs) -{ - struct fprobe_rethook_node *fpr; - struct rethook_node *rh = NULL; - struct fprobe *fp; - void *entry_data = NULL; - int ret = 0; +#define FPROBE_HASH_BITS 6 +#define FPROBE_TABLE_SIZE (1 << FPROBE_HASH_BITS) - fp = container_of(ops, struct fprobe, ops); +/* + * fprobe_table: hold 'fprobe_hlist::hlist' for checking the fprobe still + * exists. The key is the address of fprobe instance. + * fprobe_ip_table: hold 'fprobe_hlist::array[*]' for searching the fprobe + * instance related to the funciton address. The key is the ftrace IP + * address. + * + * When unregistering the fprobe, fprobe_hlist::fp and fprobe_hlist::array[*].fp + * are set NULL and delete those from both hash tables (by hlist_del_rcu). + * After an RCU grace period, the fprobe_hlist itself will be released. + * + * fprobe_table and fprobe_ip_table can be accessed from either + * - Normal hlist traversal and RCU add/del under 'fprobe_mutex' is held. + * - RCU hlist traversal under disabling preempt + */ +static struct hlist_head fprobe_table[FPROBE_TABLE_SIZE]; +static struct hlist_head fprobe_ip_table[FPROBE_IP_TABLE_SIZE]; +static DEFINE_MUTEX(fprobe_mutex); - if (fp->exit_handler) { - rh = rethook_try_get(fp->rethook); - if (!rh) { - fp->nmissed++; - return; - } - fpr = container_of(rh, struct fprobe_rethook_node, node); - fpr->entry_ip = ip; - fpr->entry_parent_ip = parent_ip; - if (fp->entry_data_size) - entry_data = fpr->data; +/* + * Find first fprobe in the hlist. It will be iterated twice in the entry + * probe, once for correcting the total required size, the second time is + * calling back the user handlers. + * Thus the hlist in the fprobe_table must be sorted and new probe needs to + * be added *before* the first fprobe. + */ +static struct fprobe_hlist_node *find_first_fprobe_node(unsigned long ip) +{ + struct fprobe_hlist_node *node; + struct hlist_head *head; + + head = &fprobe_ip_table[hash_ptr((void *)ip, FPROBE_IP_HASH_BITS)]; + hlist_for_each_entry_rcu(node, head, hlist, + lockdep_is_held(&fprobe_mutex)) { + if (node->addr == ip) + return node; } + return NULL; +} - if (fp->entry_handler) - ret = fp->entry_handler(fp, ip, parent_ip, fregs, entry_data); +/* Node insertion and deletion requires the fprobe_mutex */ +static void insert_fprobe_node(struct fprobe_hlist_node *node) +{ + unsigned long ip = node->addr; + struct fprobe_hlist_node *next; + struct hlist_head *head; - /* If entry_handler returns !0, nmissed is not counted. */ - if (rh) { - if (ret) - rethook_recycle(rh); - else - rethook_hook(rh, ftrace_get_regs(fregs), true); + lockdep_assert_held(&fprobe_mutex); + + next = find_first_fprobe_node(ip); + if (next) { + hlist_add_before_rcu(&node->hlist, &next->hlist); + return; } + head = &fprobe_ip_table[hash_ptr((void *)ip, FPROBE_IP_HASH_BITS)]; + hlist_add_head_rcu(&node->hlist, head); } -static void fprobe_handler(unsigned long ip, unsigned long parent_ip, - struct ftrace_ops *ops, struct ftrace_regs *fregs) +/* Return true if there are synonims */ +static bool delete_fprobe_node(struct fprobe_hlist_node *node) { - struct fprobe *fp; - int bit; + lockdep_assert_held(&fprobe_mutex); - fp = container_of(ops, struct fprobe, ops); - if (fprobe_disabled(fp)) - return; + node->fp = NULL; + hlist_del_rcu(&node->hlist); + return !!find_first_fprobe_node(node->addr); +} - /* recursion detection has to go before any traceable function and - * all functions before this point should be marked as notrace - */ - bit = ftrace_test_recursion_trylock(ip, parent_ip); - if (bit < 0) { - fp->nmissed++; - return; +/* Check existence of the fprobe */ +static bool is_fprobe_still_exist(struct fprobe *fp) +{ + struct hlist_head *head; + struct fprobe_hlist *fph; + + head = &fprobe_table[hash_ptr(fp, FPROBE_HASH_BITS)]; + hlist_for_each_entry_rcu(fph, head, hlist, + lockdep_is_held(&fprobe_mutex)) { + if (fph->fp == fp) + return true; } - __fprobe_handler(ip, parent_ip, ops, fregs); - ftrace_test_recursion_unlock(bit); + return false; +} + +static int add_fprobe_hash(struct fprobe *fp) +{ + struct fprobe_hlist *fph = fp->hlist_array; + struct hlist_head *head; + + if (WARN_ON_ONCE(!fph)) + return -EINVAL; + if (is_fprobe_still_exist(fp)) + return -EEXIST; + + head = &fprobe_table[hash_ptr(fp, FPROBE_HASH_BITS)]; + hlist_add_head_rcu(&fp->hlist_array->hlist, head); + return 0; } -NOKPROBE_SYMBOL(fprobe_handler); -static void fprobe_kprobe_handler(unsigned long ip, unsigned long parent_ip, - struct ftrace_ops *ops, struct ftrace_regs *fregs) +static int del_fprobe_hash(struct fprobe *fp) { - struct fprobe *fp; - int bit; + struct fprobe_hlist *fph = fp->hlist_array; - fp = container_of(ops, struct fprobe, ops); - if (fprobe_disabled(fp)) - return; + if (WARN_ON_ONCE(!fph)) + return -EINVAL; - /* recursion detection has to go before any traceable function and - * all functions called before this point should be marked as notrace - */ - bit = ftrace_test_recursion_trylock(ip, parent_ip); - if (bit < 0) { - fp->nmissed++; - return; + if (!is_fprobe_still_exist(fp)) + return -ENOENT; + + fph->fp = NULL; + hlist_del_rcu(&fph->hlist); + return 0; +} + +/* The entry data size is 4 bits (=16) * sizeof(long) in maximum */ +#define FPROBE_HEADER_SIZE_BITS 4 +#define MAX_FPROBE_DATA_SIZE_WORD ((1L << FPROBE_HEADER_SIZE_BITS) - 1) +#define MAX_FPROBE_DATA_SIZE (MAX_FPROBE_DATA_SIZE_WORD * sizeof(long)) +#define FPROBE_HEADER_PTR_BITS (BITS_PER_LONG - FPROBE_HEADER_SIZE_BITS) +#define FPROBE_HEADER_PTR_MASK GENMASK(FPROBE_HEADER_PTR_BITS - 1, 0) +#define FPROBE_HEADER_SIZE sizeof(unsigned long) + +static inline unsigned long encode_fprobe_header(struct fprobe *fp, int size_words) +{ + if (WARN_ON_ONCE(size_words > MAX_FPROBE_DATA_SIZE_WORD || + ((unsigned long)fp & ~FPROBE_HEADER_PTR_MASK) != + ~FPROBE_HEADER_PTR_MASK)) { + return 0; } + return ((unsigned long)size_words << FPROBE_HEADER_PTR_BITS) | + ((unsigned long)fp & FPROBE_HEADER_PTR_MASK); +} +/* Return reserved data size in words */ +static inline int decode_fprobe_header(unsigned long val, struct fprobe **fp) +{ + unsigned long ptr; + + ptr = (val & FPROBE_HEADER_PTR_MASK) | ~FPROBE_HEADER_PTR_MASK; + if (fp) + *fp = (struct fprobe *)ptr; + return val >> FPROBE_HEADER_PTR_BITS; +} + +/* + * fprobe shadow stack management: + * Since fprobe shares a single fgraph_ops, it needs to share the stack entry + * among the probes on the same function exit. Note that a new probe can be + * registered before a target function is returning, we can not use the hash + * table to find the corresponding probes. Thus the probe address is stored on + * the shadow stack with its entry data size. + * + */ +static inline int __fprobe_handler(unsigned long ip, unsigned long parent_ip, + struct fprobe *fp, struct ftrace_regs *fregs, + void *data) +{ + int ret = 0; + + if (fp->entry_handler) { + if (fp->exit_handler && fp->entry_data_size) + data += sizeof(unsigned long); + else + data = NULL; + ret = fp->entry_handler(fp, ip, parent_ip, fregs, data); + } + + return ret; +} + +static inline int __fprobe_kprobe_handler(unsigned long ip, unsigned long parent_ip, + struct fprobe *fp, struct ftrace_regs *fregs, + void *data) +{ + int ret; /* * This user handler is shared with other kprobes and is not expected to be * called recursively. So if any other kprobe handler is running, this will @@ -108,45 +204,173 @@ static void fprobe_kprobe_handler(unsigned long ip, unsigned long parent_ip, */ if (unlikely(kprobe_running())) { fp->nmissed++; - goto recursion_unlock; + return 0; } kprobe_busy_begin(); - __fprobe_handler(ip, parent_ip, ops, fregs); + ret = __fprobe_handler(ip, parent_ip, fp, fregs, data); kprobe_busy_end(); - -recursion_unlock: - ftrace_test_recursion_unlock(bit); + return ret; } -static void fprobe_exit_handler(struct rethook_node *rh, void *data, - unsigned long ret_ip, struct pt_regs *regs) +static int fprobe_entry(unsigned long func, unsigned long ret_ip, + struct ftrace_regs *fregs, struct fgraph_ops *gops) { - struct fprobe *fp = (struct fprobe *)data; - struct fprobe_rethook_node *fpr; - struct ftrace_regs *fregs = (struct ftrace_regs *)regs; - int bit; + struct fprobe_hlist_node *node, *first; + unsigned long header; + void *fgraph_data = NULL; + struct fprobe *fp; + int size, used, ret; - if (!fp || fprobe_disabled(fp)) - return; + if (WARN_ON_ONCE(!fregs)) + return 0; - fpr = container_of(rh, struct fprobe_rethook_node, node); + first = node = find_first_fprobe_node(func); + if (unlikely(!first)) + return 0; + + size = 0; + hlist_for_each_entry_from_rcu(node, hlist) { + if (node->addr != func) + break; + fp = node->fp; + /* + * Since fprobe can be enabled until the next loop, we ignore the + * disabled flag in this loop. + */ + if (fp && fp->exit_handler) + size += FPROBE_HEADER_SIZE + fp->entry_data_size; + } + node = first; + /* size can be 0 because fp only has entry_handler. */ + if (size) { + fgraph_data = fgraph_reserve_data(size); + if (unlikely(!fgraph_data)) { + hlist_for_each_entry_from_rcu(node, hlist) { + if (node->addr != func) + break; + fp = node->fp; + if (fp && !fprobe_disabled(fp)) + fp->nmissed++; + } + return 0; + } + } /* - * we need to assure no calls to traceable functions in-between the - * end of fprobe_handler and the beginning of fprobe_exit_handler. + * TODO: recursion detection has been done in the fgraph. Thus we need + * to add a callback to increment missed counter. */ - bit = ftrace_test_recursion_trylock(fpr->entry_ip, fpr->entry_parent_ip); - if (bit < 0) { - fp->nmissed++; + used = 0; + hlist_for_each_entry_from_rcu(node, hlist) { + if (node->addr != func) + break; + fp = node->fp; + if (!fp || fprobe_disabled(fp)) + continue; + + if (fprobe_shared_with_kprobes(fp)) + ret = __fprobe_kprobe_handler(func, ret_ip, + fp, fregs, fgraph_data + used); + else + ret = __fprobe_handler(func, ret_ip, fp, + fregs, fgraph_data + used); + /* If entry_handler returns !0, nmissed is not counted but skips exit_handler. */ + if (!ret && fp->exit_handler) { + header = encode_fprobe_header(fp, fp->entry_data_size / sizeof(long)); + if (likely(header)) { + *(unsigned long *)fgraph_data = header; + used += fp->entry_data_size + sizeof(header); + } + if (WARN_ON_ONCE(size - used < 0)) + return 0; + } + } + if (used < size) + memset(fgraph_data + used, 0, size - used); + + /* If any exit_handler is set, data must be used. */ + return used != 0; +} +NOKPROBE_SYMBOL(fprobe_entry); + +static void fprobe_return(unsigned long func, unsigned long ret_ip, + struct ftrace_regs *fregs, struct fgraph_ops *gops) +{ + unsigned long val; + struct fprobe *fp; + char *data; + int esize, size = 0; + + data = fgraph_retrieve_data(&size); + if (!data) + return; + + preempt_disable(); + while (size >= sizeof(val)) { + val = *(unsigned long *)data; + if (!val) + break; + data += sizeof(val); + size -= sizeof(val); + + esize = decode_fprobe_header(val, &fp) * sizeof(long); + if (WARN_ON_ONCE(esize > size)) + break; + + if (fp && is_fprobe_still_exist(fp) && !fprobe_disabled(fp)) { + fp->exit_handler(fp, func, ret_ip, fregs, + esize ? data : NULL); + } + data += esize; + size -= esize; + } + preempt_enable(); +} +NOKPROBE_SYMBOL(fprobe_return); + +static struct fgraph_ops fprobe_graph_ops = { + .entryregfunc = fprobe_entry, + .retregfunc = fprobe_return, +}; +static int fprobe_graph_active; + +/* Add @addrs to the ftrace filter and register fgraph if needed. */ +static int fprobe_graph_add_ips(unsigned long *addrs, int num) +{ + int ret; + + lockdep_assert_held(&fprobe_mutex); + + ret = ftrace_set_filter_ips(&fprobe_graph_ops.ops, addrs, num, 0, 0); + if (ret) + return ret; + + if (!fprobe_graph_active) { + ret = register_ftrace_graph(&fprobe_graph_ops); + if (WARN_ON_ONCE(ret)) { + ftrace_free_filter(&fprobe_graph_ops.ops); + return ret; + } + } + fprobe_graph_active++; + return 0; +} + +/* Remove @addrs from the ftrace filter and unregister fgraph if possible. */ +static void fprobe_graph_remove_ips(unsigned long *addrs, int num) +{ + lockdep_assert_held(&fprobe_mutex); + + fprobe_graph_active--; + if (!fprobe_graph_active) { + /* Q: should we unregister it ? */ + unregister_ftrace_graph(&fprobe_graph_ops); return; } - fp->exit_handler(fp, fpr->entry_ip, ret_ip, fregs, - fp->entry_data_size ? (void *)fpr->data : NULL); - ftrace_test_recursion_unlock(bit); + ftrace_set_filter_ips(&fprobe_graph_ops.ops, addrs, num, 1, 0); } -NOKPROBE_SYMBOL(fprobe_exit_handler); static int symbols_cmp(const void *a, const void *b) { @@ -176,62 +400,96 @@ static unsigned long *get_ftrace_locations(const char **syms, int num) return ERR_PTR(-ENOENT); } -static void fprobe_init(struct fprobe *fp) +struct filter_match_data { + const char *filter; + const char *notfilter; + size_t index; + size_t size; + unsigned long *addrs; +}; + +static int filter_match_callback(void *data, const char *name, unsigned long addr) { - fp->nmissed = 0; - if (fprobe_shared_with_kprobes(fp)) - fp->ops.func = fprobe_kprobe_handler; - else - fp->ops.func = fprobe_handler; - fp->ops.flags |= FTRACE_OPS_FL_SAVE_ARGS; + struct filter_match_data *match = data; + + if (!glob_match(match->filter, name) || + (match->notfilter && glob_match(match->notfilter, name))) + return 0; + + if (!ftrace_location(addr)) + return 0; + + if (match->addrs) + match->addrs[match->index] = addr; + + match->index++; + return match->index == match->size; } -static int fprobe_init_rethook(struct fprobe *fp, int num) +/* + * Make IP list from the filter/no-filter glob patterns. + * Return the number of matched symbols, or -ENOENT. + */ +static int ip_list_from_filter(const char *filter, const char *notfilter, + unsigned long *addrs, size_t size) { - int i, size; + struct filter_match_data match = { .filter = filter, .notfilter = notfilter, + .index = 0, .size = size, .addrs = addrs}; + int ret; - if (num <= 0) - return -EINVAL; + ret = kallsyms_on_each_symbol(filter_match_callback, &match); + if (ret < 0) + return ret; + ret = module_kallsyms_on_each_symbol(NULL, filter_match_callback, &match); + if (ret < 0) + return ret; - if (!fp->exit_handler) { - fp->rethook = NULL; - return 0; - } + return match.index ?: -ENOENT; +} - /* Initialize rethook if needed */ - if (fp->nr_maxactive) - size = fp->nr_maxactive; - else - size = num * num_possible_cpus() * 2; - if (size <= 0) +static void fprobe_fail_cleanup(struct fprobe *fp) +{ + kfree(fp->hlist_array); + fp->hlist_array = NULL; +} + +/* Initialize the fprobe data structure. */ +static int fprobe_init(struct fprobe *fp, unsigned long *addrs, int num) +{ + struct fprobe_hlist *hlist_array; + unsigned long addr; + int size, i; + + if (!fp || !addrs || num <= 0) return -EINVAL; - fp->rethook = rethook_alloc((void *)fp, fprobe_exit_handler); - if (!fp->rethook) + size = ALIGN(fp->entry_data_size, sizeof(long)); + if (size > MAX_FPROBE_DATA_SIZE) + return -E2BIG; + fp->entry_data_size = size; + + hlist_array = kzalloc(struct_size(hlist_array, array, num), GFP_KERNEL); + if (!hlist_array) return -ENOMEM; - for (i = 0; i < size; i++) { - struct fprobe_rethook_node *node; - - node = kzalloc(sizeof(*node) + fp->entry_data_size, GFP_KERNEL); - if (!node) { - rethook_free(fp->rethook); - fp->rethook = NULL; - return -ENOMEM; + + fp->nmissed = 0; + + hlist_array->size = num; + fp->hlist_array = hlist_array; + hlist_array->fp = fp; + for (i = 0; i < num; i++) { + hlist_array->array[i].fp = fp; + addr = ftrace_location(addrs[i]); + if (!addr) { + fprobe_fail_cleanup(fp); + return -ENOENT; } - rethook_add_node(fp->rethook, &node->node); + hlist_array->array[i].addr = addr; } return 0; } -static void fprobe_fail_cleanup(struct fprobe *fp) -{ - if (fp->rethook) { - /* Don't need to cleanup rethook->handler because this is not used. */ - rethook_free(fp->rethook); - fp->rethook = NULL; - } - ftrace_free_filter(&fp->ops); -} +#define FPROBE_IPS_MAX INT_MAX /** * register_fprobe() - Register fprobe to ftrace by pattern. @@ -246,46 +504,24 @@ static void fprobe_fail_cleanup(struct fprobe *fp) */ int register_fprobe(struct fprobe *fp, const char *filter, const char *notfilter) { - struct ftrace_hash *hash; - unsigned char *str; - int ret, len; + unsigned long *addrs; + int ret; if (!fp || !filter) return -EINVAL; - fprobe_init(fp); - - len = strlen(filter); - str = kstrdup(filter, GFP_KERNEL); - ret = ftrace_set_filter(&fp->ops, str, len, 0); - kfree(str); - if (ret) + ret = ip_list_from_filter(filter, notfilter, NULL, FPROBE_IPS_MAX); + if (ret < 0) return ret; - if (notfilter) { - len = strlen(notfilter); - str = kstrdup(notfilter, GFP_KERNEL); - ret = ftrace_set_notrace(&fp->ops, str, len, 0); - kfree(str); - if (ret) - goto out; - } - - /* TODO: - * correctly calculate the total number of filtered symbols - * from both filter and notfilter. - */ - hash = rcu_access_pointer(fp->ops.local_hash.filter_hash); - if (WARN_ON_ONCE(!hash)) - goto out; - - ret = fprobe_init_rethook(fp, (int)hash->count); - if (!ret) - ret = register_ftrace_function(&fp->ops); + addrs = kcalloc(ret, sizeof(unsigned long), GFP_KERNEL); + if (!addrs) + return -ENOMEM; + ret = ip_list_from_filter(filter, notfilter, addrs, ret); + if (ret > 0) + ret = register_fprobe_ips(fp, addrs, ret); -out: - if (ret) - fprobe_fail_cleanup(fp); + kfree(addrs); return ret; } EXPORT_SYMBOL_GPL(register_fprobe); @@ -293,7 +529,7 @@ EXPORT_SYMBOL_GPL(register_fprobe); /** * register_fprobe_ips() - Register fprobe to ftrace by address. * @fp: A fprobe data structure to be registered. - * @addrs: An array of target ftrace location addresses. + * @addrs: An array of target function address. * @num: The number of entries of @addrs. * * Register @fp to ftrace for enabling the probe on the address given by @addrs. @@ -305,23 +541,27 @@ EXPORT_SYMBOL_GPL(register_fprobe); */ int register_fprobe_ips(struct fprobe *fp, unsigned long *addrs, int num) { - int ret; - - if (!fp || !addrs || num <= 0) - return -EINVAL; + struct fprobe_hlist *hlist_array; + int ret, i; - fprobe_init(fp); - - ret = ftrace_set_filter_ips(&fp->ops, addrs, num, 0, 0); + ret = fprobe_init(fp, addrs, num); if (ret) return ret; - ret = fprobe_init_rethook(fp, num); - if (!ret) - ret = register_ftrace_function(&fp->ops); + mutex_lock(&fprobe_mutex); + + hlist_array = fp->hlist_array; + ret = fprobe_graph_add_ips(addrs, num); + if (!ret) { + add_fprobe_hash(fp); + for (i = 0; i < hlist_array->size; i++) + insert_fprobe_node(&hlist_array->array[i]); + } + mutex_unlock(&fprobe_mutex); if (ret) fprobe_fail_cleanup(fp); + return ret; } EXPORT_SYMBOL_GPL(register_fprobe_ips); @@ -359,14 +599,13 @@ EXPORT_SYMBOL_GPL(register_fprobe_syms); bool fprobe_is_registered(struct fprobe *fp) { - if (!fp || (fp->ops.saved_func != fprobe_handler && - fp->ops.saved_func != fprobe_kprobe_handler)) + if (!fp || !fp->hlist_array) return false; return true; } /** - * unregister_fprobe() - Unregister fprobe from ftrace + * unregister_fprobe() - Unregister fprobe. * @fp: A fprobe data structure to be unregistered. * * Unregister fprobe (and remove ftrace hooks from the function entries). @@ -375,23 +614,40 @@ bool fprobe_is_registered(struct fprobe *fp) */ int unregister_fprobe(struct fprobe *fp) { - int ret; + struct fprobe_hlist *hlist_array; + unsigned long *addrs = NULL; + int ret = 0, i, count; - if (!fprobe_is_registered(fp)) - return -EINVAL; + mutex_lock(&fprobe_mutex); + if (!fp || !is_fprobe_still_exist(fp)) { + ret = -EINVAL; + goto out; + } - if (fp->rethook) - rethook_stop(fp->rethook); + hlist_array = fp->hlist_array; + addrs = kcalloc(hlist_array->size, sizeof(unsigned long), GFP_KERNEL); + if (!addrs) { + ret = -ENOMEM; /* TODO: Fallback to one-by-one loop */ + goto out; + } - ret = unregister_ftrace_function(&fp->ops); - if (ret < 0) - return ret; + /* Remove non-synonim ips from table and hash */ + count = 0; + for (i = 0; i < hlist_array->size; i++) { + if (!delete_fprobe_node(&hlist_array->array[i])) + addrs[count++] = hlist_array->array[i].addr; + } + del_fprobe_hash(fp); - if (fp->rethook) - rethook_free(fp->rethook); + fprobe_graph_remove_ips(addrs, count); - ftrace_free_filter(&fp->ops); + kfree_rcu(hlist_array, rcu); + fp->hlist_array = NULL; +out: + mutex_unlock(&fprobe_mutex); + + kfree(addrs); return ret; } EXPORT_SYMBOL_GPL(unregister_fprobe); diff --git a/lib/test_fprobe.c b/lib/test_fprobe.c index d1e80653bf0c..ddffb212fba5 100644 --- a/lib/test_fprobe.c +++ b/lib/test_fprobe.c @@ -17,10 +17,8 @@ static u32 rand1, entry_val, exit_val; /* Use indirect calls to avoid inlining the target functions */ static u32 (*target)(u32 value); static u32 (*target2)(u32 value); -static u32 (*target_nest)(u32 value, u32 (*nest)(u32)); static unsigned long target_ip; static unsigned long target2_ip; -static unsigned long target_nest_ip; static int entry_return_value; static noinline u32 fprobe_selftest_target(u32 value) @@ -33,11 +31,6 @@ static noinline u32 fprobe_selftest_target2(u32 value) return (value / div_factor) + 1; } -static noinline u32 fprobe_selftest_nest_target(u32 value, u32 (*nest)(u32)) -{ - return nest(value + 2); -} - static notrace int fp_entry_handler(struct fprobe *fp, unsigned long ip, unsigned long ret_ip, struct ftrace_regs *fregs, void *data) @@ -79,22 +72,6 @@ static notrace void fp_exit_handler(struct fprobe *fp, unsigned long ip, KUNIT_EXPECT_NULL(current_test, data); } -static notrace int nest_entry_handler(struct fprobe *fp, unsigned long ip, - unsigned long ret_ip, - struct ftrace_regs *fregs, void *data) -{ - KUNIT_EXPECT_FALSE(current_test, preemptible()); - return 0; -} - -static notrace void nest_exit_handler(struct fprobe *fp, unsigned long ip, - unsigned long ret_ip, - struct ftrace_regs *fregs, void *data) -{ - KUNIT_EXPECT_FALSE(current_test, preemptible()); - KUNIT_EXPECT_EQ(current_test, ip, target_nest_ip); -} - /* Test entry only (no rethook) */ static void test_fprobe_entry(struct kunit *test) { @@ -191,25 +168,6 @@ static void test_fprobe_data(struct kunit *test) KUNIT_EXPECT_EQ(test, 0, unregister_fprobe(&fp)); } -/* Test nr_maxactive */ -static void test_fprobe_nest(struct kunit *test) -{ - static const char *syms[] = {"fprobe_selftest_target", "fprobe_selftest_nest_target"}; - struct fprobe fp = { - .entry_handler = nest_entry_handler, - .exit_handler = nest_exit_handler, - .nr_maxactive = 1, - }; - - current_test = test; - KUNIT_EXPECT_EQ(test, 0, register_fprobe_syms(&fp, syms, 2)); - - target_nest(rand1, target); - KUNIT_EXPECT_EQ(test, 1, fp.nmissed); - - KUNIT_EXPECT_EQ(test, 0, unregister_fprobe(&fp)); -} - static void test_fprobe_skip(struct kunit *test) { struct fprobe fp = { @@ -247,10 +205,8 @@ static int fprobe_test_init(struct kunit *test) rand1 = get_random_u32_above(div_factor); target = fprobe_selftest_target; target2 = fprobe_selftest_target2; - target_nest = fprobe_selftest_nest_target; target_ip = get_ftrace_location(target); target2_ip = get_ftrace_location(target2); - target_nest_ip = get_ftrace_location(target_nest); return 0; } @@ -260,7 +216,6 @@ static struct kunit_case fprobe_testcases[] = { KUNIT_CASE(test_fprobe), KUNIT_CASE(test_fprobe_syms), KUNIT_CASE(test_fprobe_data), - KUNIT_CASE(test_fprobe_nest), KUNIT_CASE(test_fprobe_skip), {} }; From patchwork Sun Nov 5 16:12:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 161689 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp2198963vqu; Sun, 5 Nov 2023 08:13:31 -0800 (PST) X-Google-Smtp-Source: AGHT+IF+f2lW1IgwJuQOxQtb1W/d/XMLOnUpeBA+arrIhpEQ/sWQRP+FFIoNY8Ij/TCVB/vH2q+A X-Received: by 2002:a05:6a21:47c1:b0:180:febc:8ecb with SMTP id as1-20020a056a2147c100b00180febc8ecbmr13735905pzc.54.1699200810780; Sun, 05 Nov 2023 08:13:30 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699200810; cv=none; d=google.com; s=arc-20160816; b=CH4FjFI4qyh0vQa+sTnLzGo/vqQuhrzWLn1A/tfIQzm7IWQ9aJRulFDC/TP3qHwzUj bQc2yB+Q2S05lpw/WC5aboiiEBAgq56LOLcmV7eFRxwR4IY/Zrilnm8aJPRgRsFw05Iu VG+sFAfia3xLeCfie71J9iF5iMyBVHKQKSmmMhWKXZYgQVXuGgMsOwyA/tlRYOEECQKp Y26OssXXxIdGF197jnhXudTGcehjShUPEYNJTOfpX7EIY1WKBrv/IuG+h0nfj8HlJn05 Nus9Kkd3Qm5n5vRzZ6ARbHAlDxKuH+QGkjngFrfoPAFOqK+Ix7tm7XxQPc6ChCx0jl19 k0Kw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=kR6cMGFBcy7J0aE3rNWxYf6v6idkPVT6x30WvknsuVg=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=mXHFy2SPHZ5snVAjD6kl+CIo3Z8zhPPpj9zBSznKPHyURHEj1/u/FozvlOj6RJpXMa TKTvk/tw/bxAEBeT3w4Ar6gba41rjPl2y2FD52er+iMov59WO+b5mXCWzjQwV6pn3SGr b8BVrRyDiZrY/4EhtKlGNwYJilFCvfGWbCCu9FsVUqA01UQWZ3Ycs6s59BPSfriVhFSG FUF7yILh6vrBDcejqFZ/XohgKSRdHgcLTDshfssasSokpLEnMXI7EgiH7pdKzl6NfJYG CJhGccLQn2FugmU618ut8sMam2ZmspRk9PWaNvT8ecy8raVLMYhknlFZPjqlaJv+Wgst F6AA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=qb5NXxkT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id j6-20020a17090a31c600b0028065b30a0dsi6192209pjf.124.2023.11.05.08.13.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Nov 2023 08:13:30 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=qb5NXxkT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id A64028056A2B; Sun, 5 Nov 2023 08:13:27 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230049AbjKEQM4 (ORCPT + 34 others); Sun, 5 Nov 2023 11:12:56 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41310 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230113AbjKEQMp (ORCPT ); Sun, 5 Nov 2023 11:12:45 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 94112D7D for ; Sun, 5 Nov 2023 08:12:41 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6BA93C433C7; Sun, 5 Nov 2023 16:12:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1699200761; bh=HjIahuTFmTXjwTrV4Enm7zMxI1fKjdaZHrz0Uldz/ns=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qb5NXxkT11vu/8YZd7+eB6gStCgUGY9wdROaGeXGD6fR/A3VYxQsI7eZfoV6OgS/j V/MZ228Qw+/W/uuTWHGgORKLLprY2VgSpHfn5Oh2aADZzAx1v0Ad/IvGAeknnothi5 l54MKEmKTWz1qbddyTeB4rlXVJxXU6UgTzrQB5rQQe6Vba6Ev5Uzv2avc/7slscTjh X2q2ru72oVk/Iih8e9n9zb29IPuo4Bx+iR4ePp6ceXAeQXXm6Q4NbbpnajTID+RgZ8 pi2GG5EGvaJCd8kLvM9C27Qrdm+Yr8p9eEOlyB6NzdMGPhxwz1zyimEWHx8Bb7AXUj vgCIje/0yX7lA== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [RFC PATCH 30/32] tracing/fprobe: Enable fprobe events with CONFIG_DYNAMIC_FTRACE_WITH_ARGS Date: Mon, 6 Nov 2023 01:12:34 +0900 Message-Id: <169920075438.482486.8569980586816970579.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169920038849.482486.15796387219966662967.stgit@devnote2> References: <169920038849.482486.15796387219966662967.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Sun, 05 Nov 2023 08:13:27 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781741189514429793 X-GMAIL-MSGID: 1781741189514429793 From: Masami Hiramatsu (Google) Allow fprobe events to be enabled with CONFIG_DYNAMIC_FTRACE_WITH_ARGS. With this change, fprobe events mostly use ftrace_regs instead of pt_regs. Note that if the arch doesn't enable HAVE_PT_REGS_COMPAT_FTRACE_REGS, fprobe events will not be able to be used from perf. Signed-off-by: Masami Hiramatsu (Google) --- Changes from previous series: Update against the new series. --- include/linux/ftrace.h | 15 ++++++++ kernel/trace/Kconfig | 1 - kernel/trace/trace_fprobe.c | 74 ++++++++++++++++++++------------------- kernel/trace/trace_probe_tmpl.h | 2 + 4 files changed, 53 insertions(+), 39 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 7db38341bc9e..34108f0610f7 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -224,6 +224,21 @@ static __always_inline bool ftrace_regs_has_args(struct ftrace_regs *fregs) regs_query_register_offset(name) #endif +#ifdef CONFIG_HAVE_REGS_AND_STACK_ACCESS_API +static __always_inline unsigned long +ftrace_regs_get_kernel_stack_nth(struct ftrace_regs *fregs, unsigned int nth) +{ + unsigned long *stackp; + + stackp = (unsigned long *)ftrace_regs_get_stack_pointer(fregs); + if (((unsigned long)(stackp + nth) & ~(THREAD_SIZE - 1)) == + ((unsigned long)stackp & ~(THREAD_SIZE - 1))) + return *(stackp + nth); + + return 0; +} +#endif /* CONFIG_HAVE_REGS_AND_STACK_ACCESS_API */ + typedef void (*ftrace_func_t)(unsigned long ip, unsigned long parent_ip, struct ftrace_ops *op, struct ftrace_regs *fregs); diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig index 11a96275b68c..169588021d90 100644 --- a/kernel/trace/Kconfig +++ b/kernel/trace/Kconfig @@ -681,7 +681,6 @@ config FPROBE_EVENTS select TRACING select PROBE_EVENTS select DYNAMIC_EVENTS - depends on DYNAMIC_FTRACE_WITH_REGS default y help This allows user to add tracing events on the function entry and diff --git a/kernel/trace/trace_fprobe.c b/kernel/trace/trace_fprobe.c index c60d0d9f1a95..882c220333cc 100644 --- a/kernel/trace/trace_fprobe.c +++ b/kernel/trace/trace_fprobe.c @@ -132,7 +132,7 @@ static int process_fetch_insn(struct fetch_insn *code, void *rec, void *dest, void *base) { - struct pt_regs *regs = rec; + struct ftrace_regs *fregs = rec; unsigned long val; int ret; @@ -140,17 +140,17 @@ process_fetch_insn(struct fetch_insn *code, void *rec, void *dest, /* 1st stage: get value from context */ switch (code->op) { case FETCH_OP_STACK: - val = regs_get_kernel_stack_nth(regs, code->param); + val = ftrace_regs_get_kernel_stack_nth(fregs, code->param); break; case FETCH_OP_STACKP: - val = kernel_stack_pointer(regs); + val = ftrace_regs_get_stack_pointer(fregs); break; case FETCH_OP_RETVAL: - val = regs_return_value(regs); + val = ftrace_regs_return_value(fregs); break; #ifdef CONFIG_HAVE_FUNCTION_ARG_ACCESS_API case FETCH_OP_ARG: - val = regs_get_kernel_argument(regs, code->param); + val = ftrace_regs_get_argument(fregs, code->param); break; #endif case FETCH_NOP_SYMBOL: /* Ignore a place holder */ @@ -170,7 +170,7 @@ NOKPROBE_SYMBOL(process_fetch_insn) /* function entry handler */ static nokprobe_inline void __fentry_trace_func(struct trace_fprobe *tf, unsigned long entry_ip, - struct pt_regs *regs, + struct ftrace_regs *fregs, struct trace_event_file *trace_file) { struct fentry_trace_entry_head *entry; @@ -184,36 +184,36 @@ __fentry_trace_func(struct trace_fprobe *tf, unsigned long entry_ip, if (trace_trigger_soft_disabled(trace_file)) return; - dsize = __get_data_size(&tf->tp, regs); + dsize = __get_data_size(&tf->tp, fregs); entry = trace_event_buffer_reserve(&fbuffer, trace_file, sizeof(*entry) + tf->tp.size + dsize); if (!entry) return; - fbuffer.regs = regs; + fbuffer.regs = ftrace_get_regs(fregs); entry = fbuffer.entry = ring_buffer_event_data(fbuffer.event); entry->ip = entry_ip; - store_trace_args(&entry[1], &tf->tp, regs, sizeof(*entry), dsize); + store_trace_args(&entry[1], &tf->tp, fregs, sizeof(*entry), dsize); trace_event_buffer_commit(&fbuffer); } static void fentry_trace_func(struct trace_fprobe *tf, unsigned long entry_ip, - struct pt_regs *regs) + struct ftrace_regs *fregs) { struct event_file_link *link; trace_probe_for_each_link_rcu(link, &tf->tp) - __fentry_trace_func(tf, entry_ip, regs, link->file); + __fentry_trace_func(tf, entry_ip, fregs, link->file); } NOKPROBE_SYMBOL(fentry_trace_func); /* Kretprobe handler */ static nokprobe_inline void __fexit_trace_func(struct trace_fprobe *tf, unsigned long entry_ip, - unsigned long ret_ip, struct pt_regs *regs, + unsigned long ret_ip, struct ftrace_regs *fregs, struct trace_event_file *trace_file) { struct fexit_trace_entry_head *entry; @@ -227,60 +227,63 @@ __fexit_trace_func(struct trace_fprobe *tf, unsigned long entry_ip, if (trace_trigger_soft_disabled(trace_file)) return; - dsize = __get_data_size(&tf->tp, regs); + dsize = __get_data_size(&tf->tp, fregs); entry = trace_event_buffer_reserve(&fbuffer, trace_file, sizeof(*entry) + tf->tp.size + dsize); if (!entry) return; - fbuffer.regs = regs; + fbuffer.regs = ftrace_get_regs(fregs); entry = fbuffer.entry = ring_buffer_event_data(fbuffer.event); entry->func = entry_ip; entry->ret_ip = ret_ip; - store_trace_args(&entry[1], &tf->tp, regs, sizeof(*entry), dsize); + store_trace_args(&entry[1], &tf->tp, fregs, sizeof(*entry), dsize); trace_event_buffer_commit(&fbuffer); } static void fexit_trace_func(struct trace_fprobe *tf, unsigned long entry_ip, - unsigned long ret_ip, struct pt_regs *regs) + unsigned long ret_ip, struct ftrace_regs *fregs) { struct event_file_link *link; trace_probe_for_each_link_rcu(link, &tf->tp) - __fexit_trace_func(tf, entry_ip, ret_ip, regs, link->file); + __fexit_trace_func(tf, entry_ip, ret_ip, fregs, link->file); } NOKPROBE_SYMBOL(fexit_trace_func); #ifdef CONFIG_PERF_EVENTS static int fentry_perf_func(struct trace_fprobe *tf, unsigned long entry_ip, - struct pt_regs *regs) + struct ftrace_regs *fregs) { struct trace_event_call *call = trace_probe_event_call(&tf->tp); struct fentry_trace_entry_head *entry; struct hlist_head *head; int size, __size, dsize; + struct pt_regs *regs; int rctx; head = this_cpu_ptr(call->perf_events); if (hlist_empty(head)) return 0; - dsize = __get_data_size(&tf->tp, regs); + dsize = __get_data_size(&tf->tp, fregs); __size = sizeof(*entry) + tf->tp.size + dsize; size = ALIGN(__size + sizeof(u32), sizeof(u64)); size -= sizeof(u32); - entry = perf_trace_buf_alloc(size, NULL, &rctx); + entry = perf_trace_buf_alloc(size, ®s, &rctx); if (!entry) return 0; + regs = ftrace_fill_perf_regs(fregs, regs); + entry->ip = entry_ip; memset(&entry[1], 0, dsize); - store_trace_args(&entry[1], &tf->tp, regs, sizeof(*entry), dsize); + store_trace_args(&entry[1], &tf->tp, fregs, sizeof(*entry), dsize); perf_trace_buf_submit(entry, size, rctx, call->event.type, 1, regs, head, NULL); return 0; @@ -289,30 +292,33 @@ NOKPROBE_SYMBOL(fentry_perf_func); static void fexit_perf_func(struct trace_fprobe *tf, unsigned long entry_ip, - unsigned long ret_ip, struct pt_regs *regs) + unsigned long ret_ip, struct ftrace_regs *fregs) { struct trace_event_call *call = trace_probe_event_call(&tf->tp); struct fexit_trace_entry_head *entry; struct hlist_head *head; int size, __size, dsize; + struct pt_regs *regs; int rctx; head = this_cpu_ptr(call->perf_events); if (hlist_empty(head)) return; - dsize = __get_data_size(&tf->tp, regs); + dsize = __get_data_size(&tf->tp, fregs); __size = sizeof(*entry) + tf->tp.size + dsize; size = ALIGN(__size + sizeof(u32), sizeof(u64)); size -= sizeof(u32); - entry = perf_trace_buf_alloc(size, NULL, &rctx); + entry = perf_trace_buf_alloc(size, ®s, &rctx); if (!entry) return; + regs = ftrace_fill_perf_regs(fregs, regs); + entry->func = entry_ip; entry->ret_ip = ret_ip; - store_trace_args(&entry[1], &tf->tp, regs, sizeof(*entry), dsize); + store_trace_args(&entry[1], &tf->tp, fregs, sizeof(*entry), dsize); perf_trace_buf_submit(entry, size, rctx, call->event.type, 1, regs, head, NULL); } @@ -324,17 +330,14 @@ static int fentry_dispatcher(struct fprobe *fp, unsigned long entry_ip, void *entry_data) { struct trace_fprobe *tf = container_of(fp, struct trace_fprobe, fp); - struct pt_regs *regs = ftrace_get_regs(fregs); int ret = 0; - if (!regs) - return 0; - if (trace_probe_test_flag(&tf->tp, TP_FLAG_TRACE)) - fentry_trace_func(tf, entry_ip, regs); + fentry_trace_func(tf, entry_ip, fregs); + #ifdef CONFIG_PERF_EVENTS if (trace_probe_test_flag(&tf->tp, TP_FLAG_PROFILE)) - ret = fentry_perf_func(tf, entry_ip, regs); + ret = fentry_perf_func(tf, entry_ip, fregs); #endif return ret; } @@ -345,16 +348,13 @@ static void fexit_dispatcher(struct fprobe *fp, unsigned long entry_ip, void *entry_data) { struct trace_fprobe *tf = container_of(fp, struct trace_fprobe, fp); - struct pt_regs *regs = ftrace_get_regs(fregs); - - if (!regs) - return; if (trace_probe_test_flag(&tf->tp, TP_FLAG_TRACE)) - fexit_trace_func(tf, entry_ip, ret_ip, regs); + fexit_trace_func(tf, entry_ip, ret_ip, fregs); + #ifdef CONFIG_PERF_EVENTS if (trace_probe_test_flag(&tf->tp, TP_FLAG_PROFILE)) - fexit_perf_func(tf, entry_ip, ret_ip, regs); + fexit_perf_func(tf, entry_ip, ret_ip, fregs); #endif } NOKPROBE_SYMBOL(fexit_dispatcher); diff --git a/kernel/trace/trace_probe_tmpl.h b/kernel/trace/trace_probe_tmpl.h index 3935b347f874..05445a745a07 100644 --- a/kernel/trace/trace_probe_tmpl.h +++ b/kernel/trace/trace_probe_tmpl.h @@ -232,7 +232,7 @@ process_fetch_insn_bottom(struct fetch_insn *code, unsigned long val, /* Sum up total data length for dynamic arrays (strings) */ static nokprobe_inline int -__get_data_size(struct trace_probe *tp, struct pt_regs *regs) +__get_data_size(struct trace_probe *tp, void *regs) { struct probe_arg *arg; int i, len, ret = 0; From patchwork Sun Nov 5 16:12:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 161687 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp2198904vqu; Sun, 5 Nov 2023 08:13:22 -0800 (PST) X-Google-Smtp-Source: AGHT+IFfTLaK2BiMyh8g0S7FE8mbC+w4E5ix7F4JxDAkUInUqEXvcIpVZCiLVbQu6fuGuUd1F1zU X-Received: by 2002:a05:6e02:1ba1:b0:358:1f0:4c5c with SMTP id n1-20020a056e021ba100b0035801f04c5cmr32562901ili.32.1699200802419; Sun, 05 Nov 2023 08:13:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699200802; cv=none; d=google.com; s=arc-20160816; b=eVA75pLmlXjuEhM0y5ef5uEenyedl0cedYH2L0QnHB+j/jb3qOMSfK4fWIKv7WLNY8 43QfHIKbWka0qosC0DaaRqSrAPnPN3NZT36dmErC53dnhVTLzke2zr2OukljHemkn16d DbbDTVTuhK/YOLX8Sm4G9xcb2Q+UQ5fvZqywujtg6+ddBdP1fEX2jdDQ6pO/CdtAcdZD nP9xsAFFhi/RazqievEQjrS4KRwOXRuQ5GuMjXPzQ63DmeMIxlqz6cE7RfaWn5/Y0xlA GF9NLfsLP4AMRYOH5snIDytBQmbdlD7TC/BOyUH93/XNX6Te4xaXMPEsDhLWekvWzuuD +UsQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=E1brWmeSwayCzbniOOCUraTFwlw+PNleAG8+mc9v1bk=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=iZxEEDOkItYMPPr9PhIg3eEc1rpXEZ6YNzim3Htohs18+z8yqeKNI3tHwUTgsgOd5h 4faJ8lUz3u9zl73NmQ6HblbeOAtUU9XTevZYUbKmNk5gifYddqwNQcxO9Q+4VZYmzfg5 yy157/itOnGf5ZqxfDRBs9xQE9ehnsyHeNADjmt7mYadqkfFqXPbJRaeuK5hBp4HRoU5 twibSYASlYgUd4dPzmD8NlOtIKEglF5kooecJvdtKx7FtUXdb+C1hk+K8XRLu6L5acrO e1LakyWjSZsEueJ0ZifPfs5CDeXMd4JIZAoxbLpc0LJfvSC9Bhaua9C7twRFn8TIyCnQ VHLw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=XX57dwXP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id l133-20020a633e8b000000b005ac86e7df6esi5894776pga.363.2023.11.05.08.13.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Nov 2023 08:13:22 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=XX57dwXP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 9E1AF8047073; Sun, 5 Nov 2023 08:13:21 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230077AbjKEQNI (ORCPT + 34 others); Sun, 5 Nov 2023 11:13:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41162 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230108AbjKEQM5 (ORCPT ); Sun, 5 Nov 2023 11:12:57 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E5A96D47 for ; Sun, 5 Nov 2023 08:12:53 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9C235C433C8; Sun, 5 Nov 2023 16:12:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1699200773; bh=x87p6LsSh0DkSyVn29mHeJbsXBvI/Rrv9TUScfOAGeQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XX57dwXPkK41NJbr00Aky+WCHF7a1NYgeAJpSkoTobyFwGlRVqre34RT1hmGSllg4 QrA7E+YRwHuf2LbSOYwOf388prr/2a/PsXq0d4pwjDEcLl4q3djQ7WG/sIZj12TDGz 1BQ/I/9+tqIm9y3k8BJzJIE0RmpHYKoFoN5Ks4LzItOJ+k4/W2iKTZSW/kbC5fF14I vYcsmWD3LXK+HC0kGrmEQac8u/LLOpV3MVIb9OFgnmdjKmTBetSUlNYg140HqZ3N6q ZesElJUgjZZ+pIJtmYFISM/N6lSUC5xP3XFoHrGw3pN3bXijOVg5jjDVjGxbnE2W8X iJCYFd+f+MwTg== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [RFC PATCH 31/32] bpf: Enable kprobe_multi feature if CONFIG_FPROBE is enabled Date: Mon, 6 Nov 2023 01:12:46 +0900 Message-Id: <169920076657.482486.11457921174678975192.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169920038849.482486.15796387219966662967.stgit@devnote2> References: <169920038849.482486.15796387219966662967.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Sun, 05 Nov 2023 08:13:21 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781741180765357081 X-GMAIL-MSGID: 1781741180765357081 From: Masami Hiramatsu (Google) Enable kprobe_multi feature if CONFIG_FPROBE is enabled. The pt_regs is converted from ftrace_regs by ftrace_partial_regs(), thus some registers may always returns 0. But it should be enough for function entry (access arguments) and exit (access return value). Signed-off-by: Masami Hiramatsu (Google) Acked-by: Florent Revest --- Changes from previous series: NOTHING, Update against the new series. --- kernel/trace/bpf_trace.c | 22 +++++++++------------- 1 file changed, 9 insertions(+), 13 deletions(-) diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index a4b5e34b0419..96d6e9993f75 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -2501,7 +2501,7 @@ static int __init bpf_event_init(void) fs_initcall(bpf_event_init); #endif /* CONFIG_MODULES */ -#if defined(CONFIG_FPROBE) && defined(CONFIG_DYNAMIC_FTRACE_WITH_REGS) +#ifdef CONFIG_FPROBE struct bpf_kprobe_multi_link { struct bpf_link link; struct fprobe fp; @@ -2524,6 +2524,8 @@ struct user_syms { char *buf; }; +static DEFINE_PER_CPU(struct pt_regs, bpf_kprobe_multi_pt_regs); + static int copy_user_syms(struct user_syms *us, unsigned long __user *usyms, u32 cnt) { unsigned long __user usymbol; @@ -2700,13 +2702,14 @@ static u64 bpf_kprobe_multi_entry_ip(struct bpf_run_ctx *ctx) static int kprobe_multi_link_prog_run(struct bpf_kprobe_multi_link *link, - unsigned long entry_ip, struct pt_regs *regs) + unsigned long entry_ip, struct ftrace_regs *fregs) { struct bpf_kprobe_multi_run_ctx run_ctx = { .link = link, .entry_ip = entry_ip, }; struct bpf_run_ctx *old_run_ctx; + struct pt_regs *regs; int err; if (unlikely(__this_cpu_inc_return(bpf_prog_active) != 1)) { @@ -2716,6 +2719,7 @@ kprobe_multi_link_prog_run(struct bpf_kprobe_multi_link *link, migrate_disable(); rcu_read_lock(); + regs = ftrace_partial_regs(fregs, this_cpu_ptr(&bpf_kprobe_multi_pt_regs)); old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx); err = bpf_prog_run(link->link.prog, regs); bpf_reset_run_ctx(old_run_ctx); @@ -2733,13 +2737,9 @@ kprobe_multi_link_handler(struct fprobe *fp, unsigned long fentry_ip, void *data) { struct bpf_kprobe_multi_link *link; - struct pt_regs *regs = ftrace_get_regs(fregs); - - if (!regs) - return 0; link = container_of(fp, struct bpf_kprobe_multi_link, fp); - kprobe_multi_link_prog_run(link, get_entry_ip(fentry_ip), regs); + kprobe_multi_link_prog_run(link, get_entry_ip(fentry_ip), fregs); return 0; } @@ -2749,13 +2749,9 @@ kprobe_multi_link_exit_handler(struct fprobe *fp, unsigned long fentry_ip, void *data) { struct bpf_kprobe_multi_link *link; - struct pt_regs *regs = ftrace_get_regs(fregs); - - if (!regs) - return; link = container_of(fp, struct bpf_kprobe_multi_link, fp); - kprobe_multi_link_prog_run(link, get_entry_ip(fentry_ip), regs); + kprobe_multi_link_prog_run(link, get_entry_ip(fentry_ip), fregs); } static int symbols_cmp_r(const void *a, const void *b, const void *priv) @@ -3012,7 +3008,7 @@ int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr kvfree(cookies); return err; } -#else /* !CONFIG_FPROBE || !CONFIG_DYNAMIC_FTRACE_WITH_REGS */ +#else /* !CONFIG_FPROBE */ int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog) { return -EOPNOTSUPP; From patchwork Sun Nov 5 16:12:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 161688 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp2198936vqu; Sun, 5 Nov 2023 08:13:27 -0800 (PST) X-Google-Smtp-Source: AGHT+IHDH5DHqQjEblztmksyQEVlQr3K84v0n1ZaJL5m1jBZdLFIM9z5c7BqmOi//eWsqmP5OMdX X-Received: by 2002:a17:903:24d:b0:1cc:9781:4782 with SMTP id j13-20020a170903024d00b001cc97814782mr7907204plh.62.1699200807397; Sun, 05 Nov 2023 08:13:27 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699200807; cv=none; d=google.com; s=arc-20160816; b=pDqFJupWVIeUnwFpPak08Z6OcR9Ea12i0POranTMqpABuqUfjykduV5UIhtW45VNEJ ujs1IPFAH27Hm7hP7AqFrmVtysnFCdam61GUuKDCg/o9RcJH3GFUgZtT58opyo1+lxo5 kI851gOekc90oJZ9/1YaoPNhrb1FoBwCHOnWcGTdEYv8fRZyQUM8RyE8LqlrjktO9ixg 8zE5OGAf0kM4SgaEVgeLWyQ4f6GoSPirAboyvoxoikp8mCih7G+1qV1Af+6I6vgc9NLZ hfi/6mbRr+KIZxzYqgRqUel2PaoVbzJDpmpAlNyvFZl7idBdDPN7LTFkh5+PJ0cvK+oC JmYQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=Gj2uNnmhVzJHjb7fQ2DkJguySdmSbMjQnu/zyX+fX9w=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=t7PVzTgqmcANLXhM/TFDYVKdL59EVfOOwfmqPV6S/qQMCQ2sc4JJhhG2gYhVU6kk1i Jln5sRxlhbipg73TiAzm/a6JE8NLit3WNcP4GoW8U5SZ4UZB0JkkWr2sMwoNAlf1YlR+ 5xF8vqSzoMkS/l8WnrmDBlksuyyNEaMlv5u+CBrw8ETXRTYgV+7vndrCuNh7zigkNd46 uSlbSqhEdyySydzEeDq0+vNQ3N/SsU85TbdbzQUgUZbPJaCvrHmiABf7Cz95D+4FOVk4 JCEtvDYxLj5NfI9IuWD8wTnSlvTpfkr6kCvD0qv5ulE5NIeMtRn+wW/5eg1civuOeMVG MxDQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=ZTesYE3H; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id n9-20020a170902d2c900b001c1e4f9c63esi6568205plc.491.2023.11.05.08.13.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Nov 2023 08:13:27 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=ZTesYE3H; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 7A47780B2312; Sun, 5 Nov 2023 08:13:26 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229879AbjKEQNT (ORCPT + 34 others); Sun, 5 Nov 2023 11:13:19 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42072 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229485AbjKEQNJ (ORCPT ); Sun, 5 Nov 2023 11:13:09 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7B31110E2 for ; Sun, 5 Nov 2023 08:13:06 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E42C5C433C7; Sun, 5 Nov 2023 16:13:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1699200786; bh=0wwZpMxwKGIQyZo3TSMiWZ/Y22+/QBwUQ7If+R6veeM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZTesYE3HpdWVCNJcBHB8i8hTSur4IzVb2plFlfyVN1bO8in5i2u6e4dC5fVOpjRLI EKf3NRwfGo7pIZ6ZdC4Ycn4M1iD9dS2yxq3ocdLZ2qWQbLtIGdUghfJpGhjWYHtmsP aBhOm6RWEyJK6YouZzWAGLRpPwjtjNpyh5IldtDMLtFMCSiWrEXO9D57AQjqBqZ5ib Lg0lU2UFXoDLxwQQkJl0HSupXNV3AjqIPCSu+aPb0c1c64x+QyJsiWXV5ILknpNLHz CGICyD9yZbV0qTU6zotdjN0ZKlajrcGiaFhSxCHsQ2hHokVxRdKMIPhN+7F6GeqEtn gYSUjmbuldTZA== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [RFC PATCH 32/32] Documentation: probes: Update fprobe on function-graph tracer Date: Mon, 6 Nov 2023 01:12:59 +0900 Message-Id: <169920077887.482486.9572304320229899702.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169920038849.482486.15796387219966662967.stgit@devnote2> References: <169920038849.482486.15796387219966662967.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Sun, 05 Nov 2023 08:13:26 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781741185678207043 X-GMAIL-MSGID: 1781741185678207043 From: Masami Hiramatsu (Google) Update fprobe documentation for the new fprobe on function-graph tracer. This includes some bahvior changes and pt_regs to ftrace_regs interface change. Signed-off-by: Masami Hiramatsu (Google) --- Documentation/trace/fprobe.rst | 42 ++++++++++++++++++++++++++-------------- 1 file changed, 27 insertions(+), 15 deletions(-) diff --git a/Documentation/trace/fprobe.rst b/Documentation/trace/fprobe.rst index 196f52386aaa..fb0446f68bcb 100644 --- a/Documentation/trace/fprobe.rst +++ b/Documentation/trace/fprobe.rst @@ -9,9 +9,10 @@ Fprobe - Function entry/exit probe Introduction ============ -Fprobe is a function entry/exit probe mechanism based on ftrace. -Instead of using ftrace full feature, if you only want to attach callbacks -on function entry and exit, similar to the kprobes and kretprobes, you can +Fprobe is a function entry/exit probe mechanism based on the function-graph +tracer. +Instead of tracing all functions, if you want to attach callbacks on specific +function entry and exit, similar to the kprobes and kretprobes, you can use fprobe. Compared with kprobes and kretprobes, fprobe gives faster instrumentation for multiple functions with single handler. This document describes how to use fprobe. @@ -91,12 +92,14 @@ The prototype of the entry/exit callback function are as follows: .. code-block:: c - int entry_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long ret_ip, struct pt_regs *regs, void *entry_data); + int entry_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long ret_ip, struct ftrace_regs *fregs, void *entry_data); - void exit_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long ret_ip, struct pt_regs *regs, void *entry_data); + void exit_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long ret_ip, struct ftrace_regs *fregs, void *entry_data); -Note that the @entry_ip is saved at function entry and passed to exit handler. -If the entry callback function returns !0, the corresponding exit callback will be cancelled. +Note that the @entry_ip is saved at function entry and passed to exit +handler. +If the entry callback function returns !0, the corresponding exit callback +will be cancelled. @fp This is the address of `fprobe` data structure related to this handler. @@ -112,12 +115,10 @@ If the entry callback function returns !0, the corresponding exit callback will This is the return address that the traced function will return to, somewhere in the caller. This can be used at both entry and exit. -@regs - This is the `pt_regs` data structure at the entry and exit. Note that - the instruction pointer of @regs may be different from the @entry_ip - in the entry_handler. If you need traced instruction pointer, you need - to use @entry_ip. On the other hand, in the exit_handler, the instruction - pointer of @regs is set to the current return address. +@fregs + This is the `ftrace_regs` data structure at the entry and exit. Note that + the instruction pointer of @fregs may be incorrect in entry handler and + exit handler, so you have to use @entry_ip and @ret_ip instead. @entry_data This is a local storage to share the data between entry and exit handlers. @@ -125,6 +126,17 @@ If the entry callback function returns !0, the corresponding exit callback will and `entry_data_size` field when registering the fprobe, the storage is allocated and passed to both `entry_handler` and `exit_handler`. +Entry data size and exit handlers on the same function +====================================================== + +Since the entry data is passed via per-task stack and it is has limited size, +the entry data size per probe is limited to `15 * sizeof(long)`. You also need +to take care that the different fprobes are probing on the same function, this +limit becomes smaller. The entry data size is aligned to `sizeof(long)` and +each fprobe which has exit handler uses a `sizeof(long)` space on the stack, +you should keep the number of fprobes on the same function as small as +possible. + Share the callbacks with kprobes ================================ @@ -165,8 +177,8 @@ This counter counts up when; - fprobe fails to take ftrace_recursion lock. This usually means that a function which is traced by other ftrace users is called from the entry_handler. - - fprobe fails to setup the function exit because of the shortage of rethook - (the shadow stack for hooking the function return.) + - fprobe fails to setup the function exit because of failing to allocate the + data buffer from the per-task shadow stack. The `fprobe::nmissed` field counts up in both cases. Therefore, the former skips both of entry and exit callback and the latter skips the exit