From patchwork Tue Feb 6 15:07:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197481 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1603150dyb; Tue, 6 Feb 2024 07:08:32 -0800 (PST) X-Google-Smtp-Source: AGHT+IFqFScJ+5jKWqRC/ZgJ2oGFKjtz+jLR3hvJD2ddhyKQ+y4M9zQOKWGz6kU3CEz6XNKuHwB9 X-Received: by 2002:a17:90b:38cb:b0:290:6170:91b5 with SMTP id nn11-20020a17090b38cb00b00290617091b5mr2568500pjb.27.1707232112630; Tue, 06 Feb 2024 07:08:32 -0800 (PST) X-Forwarded-Encrypted: i=2; AJvYcCVU/f/rG+F7GhA3nX5eDpOqJOCFyLGEz9W8KxDtQIxWy3WG+tKdo0vs4nBoKY1ojWL/FrNP5HANXPzytaJkfumUbNFuYQ== Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id w92-20020a17090a6be500b00295b08ac6d7si1296449pjj.16.2024.02.06.07.08.32 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:08:32 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55175-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@kernel.org header.s=k20201202 header.b=NaqzKEw1; arc=fail (body hash mismatch); spf=pass (google.com: domain of linux-kernel+bounces-55175-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55175-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 54F142891A1 for ; Tue, 6 Feb 2024 15:08:29 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id B90E213399C; Tue, 6 Feb 2024 15:07:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="NaqzKEw1" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CF09A13340A; Tue, 6 Feb 2024 15:07:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232066; cv=none; b=mTG8hlRk/rtE+RD8cGqh7tEvrjFJdzcye9u1OZgEGJ13FYM3vJjOPGdNvzyy3aat3vg29Wqyb5m8NVCKpJ92TLEIdc4QNjT/W7QWM4jy475BuXIsdLkjXHWggoIWoqyllCK9CgCLW4b+vmgfAqE9jq3jbbnG+30AXOeilVwJPek= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232066; c=relaxed/simple; bh=l4T04Izc/70L9UMANWu0UT3Aiu5pk/yJS8gqVOArAYs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=e2pc+VrefPDAr2NCv6gSA4ezs07CfXfxaeRs031+aMxlQpm0dStfOost5MFHy5c9bnvbuISCuip2ITvRAQCYkFiCVIezE6gSjPePETvi5HKw3OPFYLGJPKew8x1J0IOYaMOUs35Hr70zpGntfgKWRCIDegyBFum5E5o4y5jEszw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=NaqzKEw1; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1C0A5C433F1; Tue, 6 Feb 2024 15:07:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232066; bh=l4T04Izc/70L9UMANWu0UT3Aiu5pk/yJS8gqVOArAYs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NaqzKEw1xsfKMQxUEuZGZbGnKx7aPkxumq+8Er01Ga3scD14xpW2rtbO6NgOcsAjC nrVWxEMzpD9svER2A5OvqWmM5mirDKJGahKqTWQekP/dKEN3ErvcWl7IqiYwynCO6P Q7vlvMQonaQplT3SXVbIVcpLqx9El8ZQiV0lCMrMkH/fGRZSmtzXlgLL9Vwo56cf8D hck1XMn9jxFYnqbdxee4VDjbedfqffLW4f4cN5doNsrDwnhDoXPpJWIjHil1hPUF7x 5WR44h/NNgnY6TEK5BInh5y555GwSNF/dkPJn7HWJcdssuibwbRe09FtH/876F1IkO gNwjGSMMBmQ8Q== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 01/36] ftrace: Fix DIRECT_CALLS to use SAVE_REGS by default Date: Wed, 7 Feb 2024 00:07:40 +0900 Message-Id: <170723206048.502590.12669769721979471462.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790162619622147320 X-GMAIL-MSGID: 1790162619622147320 From: Masami Hiramatsu (Google) The commit 60c8971899f3 ("ftrace: Make DIRECT_CALLS work WITH_ARGS and !WITH_REGS") changed DIRECT_CALLS to use SAVE_ARGS when there are multiple ftrace_ops at the same function, but since the x86 only support to jump to direct_call from ftrace_regs_caller, when we set the function tracer on the same target function on x86, ftrace-direct does not work as below (this actually works on arm64.) At first, insmod ftrace-direct.ko to put a direct_call on 'wake_up_process()'. # insmod kernel/samples/ftrace/ftrace-direct.ko # less trace . -0 [006] ..s1. 564.686958: my_direct_func: waking up rcu_preempt-17 -0 [007] ..s1. 564.687836: my_direct_func: waking up kcompactd0-63 -0 [006] ..s1. 564.690926: my_direct_func: waking up rcu_preempt-17 -0 [006] ..s1. 564.696872: my_direct_func: waking up rcu_preempt-17 -0 [007] ..s1. 565.191982: my_direct_func: waking up kcompactd0-63 Setup a function filter to the 'wake_up_process' too, and enable it. # cd /sys/kernel/tracing/ # echo wake_up_process > set_ftrace_filter # echo function > current_tracer # less trace . -0 [006] ..s3. 686.180972: wake_up_process <-call_timer_fn -0 [006] ..s3. 686.186919: wake_up_process <-call_timer_fn -0 [002] ..s3. 686.264049: wake_up_process <-call_timer_fn -0 [002] d.h6. 686.515216: wake_up_process <-kick_pool -0 [002] d.h6. 686.691386: wake_up_process <-kick_pool Then, only function tracer is shown on x86. But if you enable 'kprobe on ftrace' event (which uses SAVE_REGS flag) on the same function, it is shown again. # echo 'p wake_up_process' >> dynamic_events # echo 1 > events/kprobes/p_wake_up_process_0/enable # echo > trace # less trace . -0 [006] ..s2. 2710.345919: p_wake_up_process_0: (wake_up_process+0x4/0x20) -0 [006] ..s3. 2710.345923: wake_up_process <-call_timer_fn -0 [006] ..s1. 2710.345928: my_direct_func: waking up rcu_preempt-17 -0 [006] ..s2. 2710.349931: p_wake_up_process_0: (wake_up_process+0x4/0x20) -0 [006] ..s3. 2710.349934: wake_up_process <-call_timer_fn -0 [006] ..s1. 2710.349937: my_direct_func: waking up rcu_preempt-17 To fix this issue, use SAVE_REGS flag for multiple ftrace_ops flag of direct_call by default. Link: https://lore.kernel.org/all/170484558617.178953.1590516949390270842.stgit@devnote2/ Fixes: 60c8971899f3 ("ftrace: Make DIRECT_CALLS work WITH_ARGS and !WITH_REGS") Cc: stable@vger.kernel.org Signed-off-by: Masami Hiramatsu (Google) Reviewed-by: Mark Rutland Tested-by: Mark Rutland [arm64] Acked-by: Jiri Olsa --- kernel/trace/ftrace.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index b01ae7d36021..c060d5b47910 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -5325,7 +5325,17 @@ static LIST_HEAD(ftrace_direct_funcs); static int register_ftrace_function_nolock(struct ftrace_ops *ops); +/* + * If there are multiple ftrace_ops, use SAVE_REGS by default, so that direct + * call will be jumped from ftrace_regs_caller. Only if the architecture does + * not support ftrace_regs_caller but direct_call, use SAVE_ARGS so that it + * jumps from ftrace_caller for multiple ftrace_ops. + */ +#ifndef HAVE_DYNAMIC_FTRACE_WITH_REGS #define MULTI_FLAGS (FTRACE_OPS_FL_DIRECT | FTRACE_OPS_FL_SAVE_ARGS) +#else +#define MULTI_FLAGS (FTRACE_OPS_FL_DIRECT | FTRACE_OPS_FL_SAVE_REGS) +#endif static int check_direct_multi(struct ftrace_ops *ops) { From patchwork Tue Feb 6 15:07:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197482 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1603341dyb; Tue, 6 Feb 2024 07:08:46 -0800 (PST) X-Google-Smtp-Source: AGHT+IHFexCT4BqSsWAjHBascwJrnhHBTXj0vJ/bYqtusY9CRQmHbKpnMFxuM3jKgVSWLJ1//kEE X-Received: by 2002:aa7:84c6:0:b0:6e0:3c34:50d8 with SMTP id x6-20020aa784c6000000b006e03c3450d8mr2853676pfn.4.1707232126493; Tue, 06 Feb 2024 07:08:46 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707232126; cv=pass; d=google.com; s=arc-20160816; b=gFzlTmROtGMLOy8lvPo5Ci8yx69dFUrjCrfDofHs8yoWenKd/eyEDGmbOS0JXPM5hy 25kxVpVVz+FiF8M4CsECkAZa77NV9HjlBsiEOrT44m1YqW6ArCRV9GwjqXb9HWsxcPYS D/l6/WkVNEdoKWDQT1M3EEiPZ9ThYGhyPouP8o9i5NYwmzNmwx+07qYc36KPLqXmhmFV rFFyXspGAgl+FcAh4sTkCWoGbOXCbLy4g6terTSAeWZLmfEoT+OvEu90HsRyROLYtIFK LT5bY8ZTZb2sAMPbijPxckRNHFHGUPkRm5CUceabx3X23yHdFS52kt11NP6rkrE5BhNo Z9NA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=pkOupataBJGsCIvcHYS0aTNEYz0mzBxc/lo+KDKHOFs=; fh=AVkoyS8Tnt8FlsxZpChuEpzHOnG/c/SzNBSpWS0t3M0=; b=hC38JM/OelVCyGbXKnYSPG5FhOahrptCy5hPcCEWpst7W+vSlgCGl6C7J7ttI5Uo9M 9rqSVPH0RxmHiKNaba8tF67YNXOj8/obgyE8ETTxPELFgU16Q9zzHKovF7CJ2GfHoUhQ FtqZP1mqtfPLD9X1WJUDv3h+P92Z7ihiovfTnB3iuSPjqpoGX/vqtl2p8hrMM1vviPQV rijlCLucZbZg+t80QcCPhH4/2wfrnYaahEK/VUvOZmAS5YNr2Pej1+bbtacoBYdxWLFJ 3/9jCeetPrLsNrycBxta/o+GbuK7P67NEgHzX5aMMAHF/mXD0pHo3odai5s+oxRLbf/W 10Uw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=MBKevHBM; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55176-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55176-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=1; AJvYcCUDWquC/RbzNmeBTHdHTHTQ0Sfq5RoscibAJftwp0juWWebJ+DkuI1A6tf+wN518y/QmBSxUnNfWzJbK2e3OP6AQccr0g== Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id k136-20020a633d8e000000b005d9252c9d05si1834223pga.103.2024.02.06.07.08.46 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:08:46 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55176-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=MBKevHBM; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55176-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55176-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 27119289D46 for ; Tue, 6 Feb 2024 15:08:46 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 9364F1332A7; Tue, 6 Feb 2024 15:08:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="MBKevHBM" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DEA831E866; Tue, 6 Feb 2024 15:07:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232078; cv=none; b=raMiHPQhgzAyhQdsQUMpBgman7AJORyf3J0c/0UdN5GLc/ZYp+TBgk/R/xdGb4UUxVqeFnFr5GIvnDsF0R3JEFH2XLQTxdtOndR4x5fQK73/M0WQR6b1MyZM1OWaUsbOnwrZSlYqNpY0qpa7md2LAvgUqivI94wPdAt0AIWJbhA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232078; c=relaxed/simple; bh=fXbFH1CnbhIn3DpSugbxQNOb5BcqjkwOHd00xmP9WY0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=DxmfUPwhfpn2YtjtGuTe+AA3nvJw44ysY9dSNNlxj+snlK/m8cABw6ne7n7chPYv/VFIlRixRByCMU8RzGVe5CkfRAJfjE1+/6s23LsAHNOzgpdRCstD+dNbQFT2vcT5raEX/7jdSR9gurh9zYm6jKpHj03apWA7gJakji+lACE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=MBKevHBM; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id C2D6BC433F1; Tue, 6 Feb 2024 15:07:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232077; bh=fXbFH1CnbhIn3DpSugbxQNOb5BcqjkwOHd00xmP9WY0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=MBKevHBMNjgsyg8h2DgvhmrkXL6rttjvpT4dSYx0BbbHzoxV08ViByyJ+iaNwJfO7 NQ6qVtAZ4US5Y+4RqlQ/YRHPu+3HF80HyDiAK8pX61xiHeyvL5C2lSk0h6zN+cvuZU qBg+R2QqnrDJ/pHur9QdtOPSkdN+iFKBtWdZoXSehew/CYUH9/17V+C79fupBwQytD RiB/v3yvh+F8yOu6pCmBrhXNj+bM4UiGYAvJSWdms/5DM1GjjztqH/DQ620e3nUJpA cjS0opgBrmrGZhTSoHQT7YdVqZIFN/2dyCge0ftYk59ybhO7sMj7rAoA1L/1KsMxiO bhLOaAdwI+9hg== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 02/36] tracing: Add a comment about ftrace_regs definition Date: Wed, 7 Feb 2024 00:07:51 +0900 Message-Id: <170723207174.502590.8310845480962046722.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790162634534375423 X-GMAIL-MSGID: 1790162634534375423 From: Masami Hiramatsu (Google) To clarify what will be expected on ftrace_regs, add a comment to the architecture independent definition of the ftrace_regs. Signed-off-by: Masami Hiramatsu (Google) Acked-by: Mark Rutland --- Changes in v3: - Add instruction pointer Changes in v2: - newly added. --- include/linux/ftrace.h | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index e8921871ef9a..8b48fc621ea0 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -118,6 +118,32 @@ extern int ftrace_enabled; #ifndef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS +/** + * ftrace_regs - ftrace partial/optimal register set + * + * ftrace_regs represents a group of registers which is used at the + * function entry and exit. There are three types of registers. + * + * - Registers for passing the parameters to callee, including the stack + * pointer. (e.g. rcx, rdx, rdi, rsi, r8, r9 and rsp on x86_64) + * - Registers for passing the return values to caller. + * (e.g. rax and rdx on x86_64) + * - Registers for hooking the function call and return including the + * frame pointer (the frame pointer is architecture/config dependent) + * (e.g. rip, rbp and rsp for x86_64) + * + * Also, architecture dependent fields can be used for internal process. + * (e.g. orig_ax on x86_64) + * + * On the function entry, those registers will be restored except for + * the stack pointer, so that user can change the function parameters + * and instruction pointer (e.g. live patching.) + * On the function exit, only registers which is used for return values + * are restored. + * + * NOTE: user *must not* access regs directly, only do it via APIs, because + * the member can be changed according to the architecture. + */ struct ftrace_regs { struct pt_regs regs; }; From patchwork Tue Feb 6 15:08:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197483 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1603568dyb; Tue, 6 Feb 2024 07:09:04 -0800 (PST) X-Google-Smtp-Source: AGHT+IE8jIlZesZVQ8puCAWeXPJB6Pf/C+RfAQcM6rs7PD23eHSy9WRDJJR7r9KpF3mmb3TzQAuG X-Received: by 2002:a05:6402:7c5:b0:560:87f:3215 with SMTP id u5-20020a05640207c500b00560087f3215mr2102969edy.12.1707232144146; Tue, 06 Feb 2024 07:09:04 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707232144; cv=pass; d=google.com; s=arc-20160816; b=cwvs/oDSPVW2FuQ1pvQ8I6LNuhEBatH5Iy/UpA/TpoaV8ueRgBOzyYzFgYxKhpq0Gf 0MgVDlNxaYJ6B6CMQZmZjXYwGZ4RrmDP8CXcKFXnI+zn24oY9D+QYks49HEm5M85Sf1J GoGoh2yMx0fSfo3r5ZCyLtApv+yHoMpy25yHVejMt+BQIRKiBpqG+TaYFHXno5Uqvrs/ 7DxNU749WBUVru37YDv9SZGWH/6iMIG1sTnZq0AljjuYYarjxKJTq35PEgxE8N4d/pBx DmaKAW9oRlnL4hlwcGrHVtgJZy1Mqnm1Po8k6pz7N7cMFrLYPTRC+eTYZNc5TWTr2S1k L4XA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=nFj3oGHTUVSCRmPr3ZQ/kGzRo141Iwf+hZWwylwbhpY=; fh=wYJa0IG+zgVu3cPvQA0cIFAHDEbOHETuEN5Qw9Q8rzI=; b=Q76/bwGj4bujBss7Y/IeQCzLYFQwttCouGHUPa5lrcYij7Q8YRoEI00I5+X2v8m1Cc f1CqhhALAsCXhto0NKNjyDACKkELAORihqWIerVz+0MazLRNSD9bxa1a/PDeyTchI0Ur LcrtDSpgDoDL9IYisNm5k2AGBgX/T49kpxdNkwuKonpl9X5ofTN4kRzCDDDRx72Ccx2I sJIef3JbIEZ+BzPzR9Lw37yO7hl7r4jbxJSrbWuU5bsEYI6h764qXMhF54HsK8aDvmil 9yq5YX5914ohS7KJpSjlXq18ESWPPUNDFKob5X2v5R1G6E5rIdu70p68BODwYCpQbRua erCQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=SnGrcVj0; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55177-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55177-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=1; AJvYcCU+CeLeeU6B3Vi6E4QdcUvzIudrMeVfSanuiQG9QbPC5QEQqQ5pk7OemdpWbRQsVyGKXEOycPxsPQbdWPAz/xXpbhJYoQ== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id t19-20020aa7d713000000b0055ee8d246e6si1254492edq.204.2024.02.06.07.09.03 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:09:04 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55177-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=SnGrcVj0; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55177-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55177-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id BA9A01F22320 for ; Tue, 6 Feb 2024 15:09:03 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 06C7A1339BB; Tue, 6 Feb 2024 15:08:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="SnGrcVj0" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 389DB131E49; Tue, 6 Feb 2024 15:08:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232089; cv=none; b=g7hYzjwqkoXHYsn4oUprwLyaUyTaYaFycPXbVjtFfut64+GUmI2mupq0pYwFaZIdIuqDAtOgoFLDChfpe73xHscBm7dcd1BcVFAiCIlwkxtrINH6I5B/JzQRxuycHjaWShErk3hIxFQoV7qly8Wty2mHMeTTV04iLLhk3XLucIM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232089; c=relaxed/simple; bh=r8dik4TbVVoqmCktPN5yl355iEZNp0JZ3yNT9goYFEk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=OUMJtGIBB4xgdHW6SOFr0aA0VH/R+oUmBGS3E8DxPq6Iw9hqJXpAFye+2yVzQ/YkL0WhiW9YTnuGxonmIvCq96I2e4J8u0Xid/7ZPCMLkIssyDa41s/ijkhF/AOKmF+E2v+a4TevXxVpz9gsygRdsiIEEBg2zmTCKIrGXnCEVhQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=SnGrcVj0; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id E2088C433F1; Tue, 6 Feb 2024 15:08:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232088; bh=r8dik4TbVVoqmCktPN5yl355iEZNp0JZ3yNT9goYFEk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=SnGrcVj0e61rmZKiC1vaToP7QwaUys7+8Esyp4/bKbCgiqcxiR8kutJeATmgvu6xD jkFEt8ijXIuQqnFjzEx7cvamSep9Gl6OxK46yw5SnBALwfTTdqGaWdy0z3kjoXidOk Nv5RMoQgbQbJWTKmCTWJ5dKz8aKI2KPKsfcZnva0ggpY72y7POZLC5OsaSbVHRaMQ0 8aLfBVe1Q7qV5JzPI2Fzn8SIP8yz38GgUWwFiUxptgs3vepRZA48UXEDFgnrWzFpaQ 43VWjPXthrjcU0fxaVYlQdrQL9ve2gSSy4wdjqOOIIZi8S3hWyweRHpfQpftcjwRcM 7qe380LmFc7sg== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 03/36] tracing: Rename ftrace_regs_return_value to ftrace_regs_get_return_value Date: Wed, 7 Feb 2024 00:08:02 +0900 Message-Id: <170723208269.502590.11563207960186717074.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790162652740118583 X-GMAIL-MSGID: 1790162652740118583 From: Masami Hiramatsu (Google) Rename ftrace_regs_return_value to ftrace_regs_get_return_value as same as other ftrace_regs_get/set_* APIs. Signed-off-by: Masami Hiramatsu (Google) Acked-by: Mark Rutland --- Changes in v6: - Moved to top of the series. Changes in v3: - Newly added. --- arch/loongarch/include/asm/ftrace.h | 2 +- arch/powerpc/include/asm/ftrace.h | 2 +- arch/s390/include/asm/ftrace.h | 2 +- arch/x86/include/asm/ftrace.h | 2 +- include/linux/ftrace.h | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/arch/loongarch/include/asm/ftrace.h b/arch/loongarch/include/asm/ftrace.h index de891c2c83d4..b43acfc5776c 100644 --- a/arch/loongarch/include/asm/ftrace.h +++ b/arch/loongarch/include/asm/ftrace.h @@ -70,7 +70,7 @@ ftrace_regs_set_instruction_pointer(struct ftrace_regs *fregs, unsigned long ip) regs_get_kernel_argument(&(fregs)->regs, n) #define ftrace_regs_get_stack_pointer(fregs) \ kernel_stack_pointer(&(fregs)->regs) -#define ftrace_regs_return_value(fregs) \ +#define ftrace_regs_get_return_value(fregs) \ regs_return_value(&(fregs)->regs) #define ftrace_regs_set_return_value(fregs, ret) \ regs_set_return_value(&(fregs)->regs, ret) diff --git a/arch/powerpc/include/asm/ftrace.h b/arch/powerpc/include/asm/ftrace.h index 1ebd2ca97f12..ce2b65cd4607 100644 --- a/arch/powerpc/include/asm/ftrace.h +++ b/arch/powerpc/include/asm/ftrace.h @@ -69,7 +69,7 @@ ftrace_regs_get_instruction_pointer(struct ftrace_regs *fregs) regs_get_kernel_argument(&(fregs)->regs, n) #define ftrace_regs_get_stack_pointer(fregs) \ kernel_stack_pointer(&(fregs)->regs) -#define ftrace_regs_return_value(fregs) \ +#define ftrace_regs_get_return_value(fregs) \ regs_return_value(&(fregs)->regs) #define ftrace_regs_set_return_value(fregs, ret) \ regs_set_return_value(&(fregs)->regs, ret) diff --git a/arch/s390/include/asm/ftrace.h b/arch/s390/include/asm/ftrace.h index 5a82b08f03cd..01e775c98425 100644 --- a/arch/s390/include/asm/ftrace.h +++ b/arch/s390/include/asm/ftrace.h @@ -88,7 +88,7 @@ ftrace_regs_set_instruction_pointer(struct ftrace_regs *fregs, regs_get_kernel_argument(&(fregs)->regs, n) #define ftrace_regs_get_stack_pointer(fregs) \ kernel_stack_pointer(&(fregs)->regs) -#define ftrace_regs_return_value(fregs) \ +#define ftrace_regs_get_return_value(fregs) \ regs_return_value(&(fregs)->regs) #define ftrace_regs_set_return_value(fregs, ret) \ regs_set_return_value(&(fregs)->regs, ret) diff --git a/arch/x86/include/asm/ftrace.h b/arch/x86/include/asm/ftrace.h index 897cf02c20b1..cf88cc8cc74d 100644 --- a/arch/x86/include/asm/ftrace.h +++ b/arch/x86/include/asm/ftrace.h @@ -58,7 +58,7 @@ arch_ftrace_get_regs(struct ftrace_regs *fregs) regs_get_kernel_argument(&(fregs)->regs, n) #define ftrace_regs_get_stack_pointer(fregs) \ kernel_stack_pointer(&(fregs)->regs) -#define ftrace_regs_return_value(fregs) \ +#define ftrace_regs_get_return_value(fregs) \ regs_return_value(&(fregs)->regs) #define ftrace_regs_set_return_value(fregs, ret) \ regs_set_return_value(&(fregs)->regs, ret) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 8b48fc621ea0..39ac1f3e8041 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -184,7 +184,7 @@ static __always_inline bool ftrace_regs_has_args(struct ftrace_regs *fregs) regs_get_kernel_argument(ftrace_get_regs(fregs), n) #define ftrace_regs_get_stack_pointer(fregs) \ kernel_stack_pointer(ftrace_get_regs(fregs)) -#define ftrace_regs_return_value(fregs) \ +#define ftrace_regs_get_return_value(fregs) \ regs_return_value(ftrace_get_regs(fregs)) #define ftrace_regs_set_return_value(fregs, ret) \ regs_set_return_value(ftrace_get_regs(fregs), ret) From patchwork Tue Feb 6 15:08:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197484 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1603815dyb; Tue, 6 Feb 2024 07:09:21 -0800 (PST) X-Google-Smtp-Source: AGHT+IEMxaxlF78B1kB2LXtpwpzNTSAgtFZEx7ziicYyzITLvPOTTgRZEEJVAh00X5jhCB5C5/AX X-Received: by 2002:a17:906:30c7:b0:a38:215c:89b with SMTP id b7-20020a17090630c700b00a38215c089bmr1233581ejb.73.1707232160937; Tue, 06 Feb 2024 07:09:20 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707232160; cv=pass; d=google.com; s=arc-20160816; b=Qq7mxUeuIgGHtSniDZGJu0twDoG1HBeUCk0cQ4QOnQgUhaBEeGDHSV9+nr69sqg2m9 wtR13cVNXuRth/mdni8BD0LeL02FD2O+8CexEaMJvaAxcmXH4oYSvjjoty2uqd/O641r YOPc0qh97cZe5hXiG7c/y4211S+6e6S+m/0T1tYM4aGColt4NJwQIHBqXUhj546aBUXt gdb2DUjeD5PFyyKQzlnH0zzuf3iANTYf1COEh0NiFxO9GKLmNF71kadWPbxF6kX/z74z wTf/ebmk9PqYIj2uIrZXO5JJfsYpzTE7Jh04+iOJ9fxRnwRzfKK1pD60RZpXn0UkULX9 aK7g== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=6UWyW+HZQB9PkPidfS9kyGvOgCmjAVd9N756+zbkz8k=; fh=HhMVAUQFNRLthITqXdnWe1mkt1D1Kh8qzda9FygJ26s=; b=rDL9A03MBdFEOPvBU5Neclj3qRY0ks9OmIbhrFWdL2PsMnLrqRzEoxVp5PTxuuG093 nA4gj79EZDu1y1hIqTRI6k5YaflJQHt3CVfqNyMIZXjCtUMQPgHlum3B7j3yqCZUnvy+ pwkUdChMWk5PVe5/L4vXL2+Z/HhVeVM2gtzcAc/9Rh9/Bsnkdb9OPpu948vfI5Jyd+11 bx/wTaOODllIjbf+X0KIByEBMoN7LruMp6V+1kDcssekTc6aqFubeb6TQUKC5oXMZnUm iMAVrWxV/u04L6C+8SJ05LUERBEGebVp+vc/G270yL3vocyA9DFSP/rYx3erDP5FjbxH oqww==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Ff66VVui; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55178-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55178-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=1; AJvYcCUk/KLkNmcGLUZXNJ1bTOv+q8rRug31zA45Kqxn/gnYV02P/XJ2zinQqXVgJTNBNRvD5u3+2hyTnVR9ETPLYgzs7nBiWg== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id s11-20020a1709066c8b00b00a316f007a47si1208076ejr.57.2024.02.06.07.09.20 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:09:20 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55178-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Ff66VVui; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55178-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55178-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 8A8C61F23EFC for ; Tue, 6 Feb 2024 15:09:20 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 46818133982; Tue, 6 Feb 2024 15:08:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Ff66VVui" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9AD7713328A; Tue, 6 Feb 2024 15:08:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232100; cv=none; b=Z/ITh0bi5My3FiY1F5+0eeSfH8BIH4W/kYSBtodo4d5i7FPQXBD73ou0GiMV1xC16egI6iaR5nX1DGroYNuKkIDDPuZ2E64gXulyY+Z0TYUCUaoEbr0vcrkPIVyDFIhB/s/NgyVXaEz7V9Le4Zdt+TttHMTy4fVRo16oDFZMBwI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232100; c=relaxed/simple; bh=Ky/M8O0rm7AMIWcFwVKGUXTVtEFqfb+EU9Ru8C9PH5M=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=skw6j7IaZJdhbz5Z2xErsIN5bhZ9ir02CTBm1ejHjhvD8raAoafYILKEzonJWy6NIIMxoMnjcrEUalu0zs5DFr0hctLsaXUECtxzylo3kBCxBVb7rbayuWwZp9Ndac7CIRgrNSVvTwUAPRSPxAo/S+pv/T3ZNCg7fDXdPcYMTSQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Ff66VVui; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 79D69C433F1; Tue, 6 Feb 2024 15:08:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232100; bh=Ky/M8O0rm7AMIWcFwVKGUXTVtEFqfb+EU9Ru8C9PH5M=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Ff66VVuimpifxG4pLQBtMTYu4BkAfVFZ/lJtfwbhKnPXzCxVyHaXJ2Q1Bl1yXh3Na zSE5JDisQTBaf+0b3qLZwh8p5guOqdB/m/I+W/SoQGwFsEbVrHLLz5qyVQteg5Ufqu qF3yqx/VLNOQGqKDcc5SblSw+571AAP6OTr0FR2sLHDDZk7beK/Q35T48DrBuFRUwC 52LKbYvoeJ1fl1sOtl413kETW6trynHTApYMMO+u3ITYbmfrUsUv/UNvVAXClRPmVd wBD/Espx+RxssAyUKAOqeJj1z0Mu1Z1YGldSuar7RL1wShyhL9tQ3bHunRnbvIE0rF nxJo7PbHfPdGw== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 04/36] x86: tracing: Add ftrace_regs definition in the header Date: Wed, 7 Feb 2024 00:08:14 +0900 Message-Id: <170723209417.502590.1912889418952674532.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790162670592075896 X-GMAIL-MSGID: 1790162670592075896 From: Masami Hiramatsu (Google) Add ftrace_regs definition for x86_64 in the ftrace header to clarify what register will be accessible from ftrace_regs. Signed-off-by: Masami Hiramatsu (Google) --- Changes in v3: - Add rip to be saved. Changes in v2: - Newly added. --- arch/x86/include/asm/ftrace.h | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/arch/x86/include/asm/ftrace.h b/arch/x86/include/asm/ftrace.h index cf88cc8cc74d..c88bf47f46da 100644 --- a/arch/x86/include/asm/ftrace.h +++ b/arch/x86/include/asm/ftrace.h @@ -36,6 +36,12 @@ static inline unsigned long ftrace_call_adjust(unsigned long addr) #ifdef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS struct ftrace_regs { + /* + * On the x86_64, the ftrace_regs saves; + * rax, rcx, rdx, rdi, rsi, r8, r9, rbp, rip and rsp. + * Also orig_ax is used for passing direct trampoline address. + * x86_32 doesn't support ftrace_regs. + */ struct pt_regs regs; }; From patchwork Tue Feb 6 15:08:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197485 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1604030dyb; Tue, 6 Feb 2024 07:09:38 -0800 (PST) X-Google-Smtp-Source: AGHT+IES+gXkDFb9QaDKqMOZrZoeuWq7ed4xIDYpBhxcGYc6xIoDkz/j6O3X93oIZZZx972VaYZg X-Received: by 2002:a05:6214:d8a:b0:68c:ae95:ddd4 with SMTP id e10-20020a0562140d8a00b0068cae95ddd4mr3337236qve.20.1707232177972; Tue, 06 Feb 2024 07:09:37 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707232177; cv=pass; d=google.com; s=arc-20160816; b=zIRHPhhlgteT0hEiLiZQ4qLhqq/UZjK74BXJMuMwae60VNQIYQgGFkJ4bTH8uQ7N8G HnwR156zKWzrkoqdMNt6nSo4Yp+ijsJ0a+mv3TtoyKRQyKj94CYcOidbqEGt37c0w9wa hkI3RLRZEhyeo6vFgQlKPcVqC+1rFZIzhbnTMKcO3k4Tt3WV2rH5gyIcsPBtdhsB3I/S GQJgQ0bOpYDCfRINcaxLB7QVjA6xVzQ6Ew3ZiMAOQlghKzOq2DAXiDQ0RV/T6NsdoiRg iidbixTFUMN0Zc9Ip8QBwjCOQhIS4LSrCR/hxLtIp84DWISsf4ItGYjCkMy4W4A+h/w6 qG+A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=hYaE7vgcX3hHzlXSQemt/OYUbCnf1bPgXh35gktpI3E=; fh=uZHM0XGh+HCepXCinnVnjaYW74Of7Jgh3ZwZEnSwrqs=; b=KM7OrRfxskGhF32wbCpHxbbxc8QxXg+zkiNA1BDgjmr+DYFkYH4JYv02dFzlHlH1+d QpKEB3CDnEws40tT6eP4GsKBh1UH7iMmKyf6SSlL71nsDVqd4AiZSQBzUuBXbK+Wpn4C fiWLH1OZYn76omu1bqgSm5G97ZMm8gox8/hUGZETjW/TyQWqMXvorknRrJuh65hmpwl1 WxDmPwomDiHKqiBS6xGPZaStrhWwCCIOVFA95iQL9S1KDV9+jwbxXDUGF9O5fI/WcTcc hE6UG1Q5Mvxz9I77lqIEzYWvCVxh4pdRRH8v7Rit72TwTMxkBX03EP48k/T5GrDJLlTo eqCg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=LlhBAIHJ; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55179-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55179-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=1; AJvYcCWE9/Adb63qNI6aX8kv/tCnZmtsVzHUufjxDX1aDR3eXuONnXmbcFwufnXoOwDDBbIs88CYaWro+hPaCIF04feTpz2Ijg== Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id 11-20020ad45bab000000b0068c78db0ea4si2861558qvq.609.2024.02.06.07.09.37 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:09:37 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55179-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=LlhBAIHJ; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55179-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55179-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id B0A0E1C23903 for ; Tue, 6 Feb 2024 15:09:37 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id E49F8133294; Tue, 6 Feb 2024 15:08:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="LlhBAIHJ" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9EE0B13328A; Tue, 6 Feb 2024 15:08:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232111; cv=none; b=TbSzuQW9P27+YKkXtIQhz2Qv4LpH1OCGU2meJ+ZgJ8VMR4BA58jC2GCm1hterHdk6JygYqGNpveM7a6NMLjWkQFkK5KQuMCdFZF4aMrnVXuea3uQFMpw1Tf6g8smd73owRH9q/4/YA8l6DY5noLtuz/zIVRPIlfLnnDYlyMJ0vI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232111; c=relaxed/simple; bh=IeyhGlrZHCNQiYvBAYbqAyUuC7XrNCZovGE/rngWoYs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=tI5UTr4h9Pz3uP3sPDaCgP/+LMRZPj3fB17CpHfYRZ6cdzob6OKlvTRmR5u3jpPlkc1k/B32M6mw5zdla6kt4VaPXf4qBfr0f1HhekBUQbHgZXge9LxmFhxsMpFMnWm9NRPtd1+KCCDv+pZnVy21BKpgsSJBYjjLVAMtDZEpYZQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=LlhBAIHJ; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id D3894C433C7; Tue, 6 Feb 2024 15:08:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232111; bh=IeyhGlrZHCNQiYvBAYbqAyUuC7XrNCZovGE/rngWoYs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=LlhBAIHJ+UQ/FsVgDQBuUvwSefK+U2ImHX1B8sifQ4ToFarJBkFepve5SIc6Af9Tw +OBk5v2Dgb9rr6K6PEfWx2upzxqHo5nykDXtS7myq9Kpaec+oMqL5LdcLkhtL1E4N0 a7hno+r0Rsj+HKGZyltK/Scjnk0nssSL1JYVeuoVOXiCkMgagxOQw62cWlm1O3JEUG 15LnrbFRrAatPcLLvt9GoSVTbXswhPUFHy0NtmIsvmO5joAwr+MM1oSMtZ048aO3LU l3mgjlulzmuCjM3f75FiaBr7W9+dZlXYEU8hk+HJ+FtrvXMngormu5fe10uUHkJSsj WEtDdHqrtWzrw== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 05/36] function_graph: Convert ret_stack to a series of longs Date: Wed, 7 Feb 2024 00:08:25 +0900 Message-Id: <170723210553.502590.11433819353474145840.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790162688131508171 X-GMAIL-MSGID: 1790162688131508171 From: Steven Rostedt (VMware) In order to make it possible to have multiple callbacks registered with the function_graph tracer, the retstack needs to be converted from an array of ftrace_ret_stack structures to an array of longs. This will allow to store the list of callbacks on the stack for the return side of the functions. Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- include/linux/sched.h | 2 - kernel/trace/fgraph.c | 124 ++++++++++++++++++++++++++++--------------------- 2 files changed, 71 insertions(+), 55 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index cdb8ea53c365..1cfabb44dcbb 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1393,7 +1393,7 @@ struct task_struct { int curr_ret_depth; /* Stack of return addresses for return function tracing: */ - struct ftrace_ret_stack *ret_stack; + unsigned long *ret_stack; /* Timestamp for last schedule: */ unsigned long long ftrace_timestamp; diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index c83c005e654e..30edeb6d4aa9 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -25,6 +25,18 @@ #define ASSIGN_OPS_HASH(opsname, val) #endif +#define FGRAPH_RET_SIZE sizeof(struct ftrace_ret_stack) +#define FGRAPH_RET_INDEX (ALIGN(FGRAPH_RET_SIZE, sizeof(long)) / sizeof(long)) +#define SHADOW_STACK_SIZE (PAGE_SIZE) +#define SHADOW_STACK_INDEX \ + (ALIGN(SHADOW_STACK_SIZE, sizeof(long)) / sizeof(long)) +/* Leave on a buffer at the end */ +#define SHADOW_STACK_MAX_INDEX (SHADOW_STACK_INDEX - FGRAPH_RET_INDEX) + +#define RET_STACK(t, index) ((struct ftrace_ret_stack *)(&(t)->ret_stack[index])) +#define RET_STACK_INC(c) ({ c += FGRAPH_RET_INDEX; }) +#define RET_STACK_DEC(c) ({ c -= FGRAPH_RET_INDEX; }) + DEFINE_STATIC_KEY_FALSE(kill_ftrace_graph); int ftrace_graph_active; @@ -69,6 +81,7 @@ static int ftrace_push_return_trace(unsigned long ret, unsigned long func, unsigned long frame_pointer, unsigned long *retp) { + struct ftrace_ret_stack *ret_stack; unsigned long long calltime; int index; @@ -85,23 +98,25 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func, smp_rmb(); /* The return trace stack is full */ - if (current->curr_ret_stack == FTRACE_RETFUNC_DEPTH - 1) { + if (current->curr_ret_stack >= SHADOW_STACK_MAX_INDEX) { atomic_inc(¤t->trace_overrun); return -EBUSY; } calltime = trace_clock_local(); - index = ++current->curr_ret_stack; + index = current->curr_ret_stack; + RET_STACK_INC(current->curr_ret_stack); + ret_stack = RET_STACK(current, index); barrier(); - current->ret_stack[index].ret = ret; - current->ret_stack[index].func = func; - current->ret_stack[index].calltime = calltime; + ret_stack->ret = ret; + ret_stack->func = func; + ret_stack->calltime = calltime; #ifdef HAVE_FUNCTION_GRAPH_FP_TEST - current->ret_stack[index].fp = frame_pointer; + ret_stack->fp = frame_pointer; #endif #ifdef HAVE_FUNCTION_GRAPH_RET_ADDR_PTR - current->ret_stack[index].retp = retp; + ret_stack->retp = retp; #endif return 0; } @@ -148,7 +163,7 @@ int function_graph_enter(unsigned long ret, unsigned long func, return 0; out_ret: - current->curr_ret_stack--; + RET_STACK_DEC(current->curr_ret_stack); out: current->curr_ret_depth--; return -EBUSY; @@ -159,11 +174,13 @@ static void ftrace_pop_return_trace(struct ftrace_graph_ret *trace, unsigned long *ret, unsigned long frame_pointer) { + struct ftrace_ret_stack *ret_stack; int index; index = current->curr_ret_stack; + RET_STACK_DEC(index); - if (unlikely(index < 0 || index >= FTRACE_RETFUNC_DEPTH)) { + if (unlikely(index < 0 || index > SHADOW_STACK_MAX_INDEX)) { ftrace_graph_stop(); WARN_ON(1); /* Might as well panic, otherwise we have no where to go */ @@ -171,6 +188,7 @@ ftrace_pop_return_trace(struct ftrace_graph_ret *trace, unsigned long *ret, return; } + ret_stack = RET_STACK(current, index); #ifdef HAVE_FUNCTION_GRAPH_FP_TEST /* * The arch may choose to record the frame pointer used @@ -186,22 +204,22 @@ ftrace_pop_return_trace(struct ftrace_graph_ret *trace, unsigned long *ret, * Note, -mfentry does not use frame pointers, and this test * is not needed if CC_USING_FENTRY is set. */ - if (unlikely(current->ret_stack[index].fp != frame_pointer)) { + if (unlikely(ret_stack->fp != frame_pointer)) { ftrace_graph_stop(); WARN(1, "Bad frame pointer: expected %lx, received %lx\n" " from func %ps return to %lx\n", current->ret_stack[index].fp, frame_pointer, - (void *)current->ret_stack[index].func, - current->ret_stack[index].ret); + (void *)ret_stack->func, + ret_stack->ret); *ret = (unsigned long)panic; return; } #endif - *ret = current->ret_stack[index].ret; - trace->func = current->ret_stack[index].func; - trace->calltime = current->ret_stack[index].calltime; + *ret = ret_stack->ret; + trace->func = ret_stack->func; + trace->calltime = ret_stack->calltime; trace->overrun = atomic_read(¤t->trace_overrun); trace->depth = current->curr_ret_depth--; /* @@ -262,7 +280,7 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs * curr_ret_stack is after that. */ barrier(); - current->curr_ret_stack--; + RET_STACK_DEC(current->curr_ret_stack); if (unlikely(!ret)) { ftrace_graph_stop(); @@ -305,12 +323,13 @@ unsigned long ftrace_return_to_handler(unsigned long frame_pointer) struct ftrace_ret_stack * ftrace_graph_get_ret_stack(struct task_struct *task, int idx) { - idx = task->curr_ret_stack - idx; + int index = task->curr_ret_stack; - if (idx >= 0 && idx <= task->curr_ret_stack) - return &task->ret_stack[idx]; + index -= FGRAPH_RET_INDEX * (idx + 1); + if (index < 0) + return NULL; - return NULL; + return RET_STACK(task, index); } /** @@ -332,18 +351,20 @@ ftrace_graph_get_ret_stack(struct task_struct *task, int idx) unsigned long ftrace_graph_ret_addr(struct task_struct *task, int *idx, unsigned long ret, unsigned long *retp) { + struct ftrace_ret_stack *ret_stack; int index = task->curr_ret_stack; int i; if (ret != (unsigned long)dereference_kernel_function_descriptor(return_to_handler)) return ret; - if (index < 0) - return ret; + RET_STACK_DEC(index); - for (i = 0; i <= index; i++) - if (task->ret_stack[i].retp == retp) - return task->ret_stack[i].ret; + for (i = index; i >= 0; RET_STACK_DEC(i)) { + ret_stack = RET_STACK(task, i); + if (ret_stack->retp == retp) + return ret_stack->ret; + } return ret; } @@ -357,14 +378,15 @@ unsigned long ftrace_graph_ret_addr(struct task_struct *task, int *idx, return ret; task_idx = task->curr_ret_stack; + RET_STACK_DEC(task_idx); if (!task->ret_stack || task_idx < *idx) return ret; task_idx -= *idx; - (*idx)++; + RET_STACK_INC(*idx); - return task->ret_stack[task_idx].ret; + return RET_STACK(task, task_idx); } #endif /* HAVE_FUNCTION_GRAPH_RET_ADDR_PTR */ @@ -402,7 +424,7 @@ trace_func_graph_ent_t ftrace_graph_entry = ftrace_graph_entry_stub; static trace_func_graph_ent_t __ftrace_graph_entry = ftrace_graph_entry_stub; /* Try to assign a return stack array on FTRACE_RETSTACK_ALLOC_SIZE tasks. */ -static int alloc_retstack_tasklist(struct ftrace_ret_stack **ret_stack_list) +static int alloc_retstack_tasklist(unsigned long **ret_stack_list) { int i; int ret = 0; @@ -410,10 +432,7 @@ static int alloc_retstack_tasklist(struct ftrace_ret_stack **ret_stack_list) struct task_struct *g, *t; for (i = 0; i < FTRACE_RETSTACK_ALLOC_SIZE; i++) { - ret_stack_list[i] = - kmalloc_array(FTRACE_RETFUNC_DEPTH, - sizeof(struct ftrace_ret_stack), - GFP_KERNEL); + ret_stack_list[i] = kmalloc(SHADOW_STACK_SIZE, GFP_KERNEL); if (!ret_stack_list[i]) { start = 0; end = i; @@ -431,9 +450,9 @@ static int alloc_retstack_tasklist(struct ftrace_ret_stack **ret_stack_list) if (t->ret_stack == NULL) { atomic_set(&t->trace_overrun, 0); - t->curr_ret_stack = -1; + t->curr_ret_stack = 0; t->curr_ret_depth = -1; - /* Make sure the tasks see the -1 first: */ + /* Make sure the tasks see the 0 first: */ smp_wmb(); t->ret_stack = ret_stack_list[start++]; } @@ -453,6 +472,7 @@ ftrace_graph_probe_sched_switch(void *ignore, bool preempt, struct task_struct *next, unsigned int prev_state) { + struct ftrace_ret_stack *ret_stack; unsigned long long timestamp; int index; @@ -477,8 +497,11 @@ ftrace_graph_probe_sched_switch(void *ignore, bool preempt, */ timestamp -= next->ftrace_timestamp; - for (index = next->curr_ret_stack; index >= 0; index--) - next->ret_stack[index].calltime += timestamp; + for (index = next->curr_ret_stack - FGRAPH_RET_INDEX; index >= 0; ) { + ret_stack = RET_STACK(next, index); + ret_stack->calltime += timestamp; + index -= FGRAPH_RET_INDEX; + } } static int ftrace_graph_entry_test(struct ftrace_graph_ent *trace) @@ -521,10 +544,10 @@ void update_function_graph_func(void) ftrace_graph_entry = __ftrace_graph_entry; } -static DEFINE_PER_CPU(struct ftrace_ret_stack *, idle_ret_stack); +static DEFINE_PER_CPU(unsigned long *, idle_ret_stack); static void -graph_init_task(struct task_struct *t, struct ftrace_ret_stack *ret_stack) +graph_init_task(struct task_struct *t, unsigned long *ret_stack) { atomic_set(&t->trace_overrun, 0); t->ftrace_timestamp = 0; @@ -539,7 +562,7 @@ graph_init_task(struct task_struct *t, struct ftrace_ret_stack *ret_stack) */ void ftrace_graph_init_idle_task(struct task_struct *t, int cpu) { - t->curr_ret_stack = -1; + t->curr_ret_stack = 0; t->curr_ret_depth = -1; /* * The idle task has no parent, it either has its own @@ -549,14 +572,11 @@ void ftrace_graph_init_idle_task(struct task_struct *t, int cpu) WARN_ON(t->ret_stack != per_cpu(idle_ret_stack, cpu)); if (ftrace_graph_active) { - struct ftrace_ret_stack *ret_stack; + unsigned long *ret_stack; ret_stack = per_cpu(idle_ret_stack, cpu); if (!ret_stack) { - ret_stack = - kmalloc_array(FTRACE_RETFUNC_DEPTH, - sizeof(struct ftrace_ret_stack), - GFP_KERNEL); + ret_stack = kmalloc(SHADOW_STACK_SIZE, GFP_KERNEL); if (!ret_stack) return; per_cpu(idle_ret_stack, cpu) = ret_stack; @@ -570,15 +590,13 @@ void ftrace_graph_init_task(struct task_struct *t) { /* Make sure we do not use the parent ret_stack */ t->ret_stack = NULL; - t->curr_ret_stack = -1; + t->curr_ret_stack = 0; t->curr_ret_depth = -1; if (ftrace_graph_active) { - struct ftrace_ret_stack *ret_stack; + unsigned long *ret_stack; - ret_stack = kmalloc_array(FTRACE_RETFUNC_DEPTH, - sizeof(struct ftrace_ret_stack), - GFP_KERNEL); + ret_stack = kmalloc(SHADOW_STACK_SIZE, GFP_KERNEL); if (!ret_stack) return; graph_init_task(t, ret_stack); @@ -587,7 +605,7 @@ void ftrace_graph_init_task(struct task_struct *t) void ftrace_graph_exit_task(struct task_struct *t) { - struct ftrace_ret_stack *ret_stack = t->ret_stack; + unsigned long *ret_stack = t->ret_stack; t->ret_stack = NULL; /* NULL must become visible to IRQs before we free it: */ @@ -599,12 +617,10 @@ void ftrace_graph_exit_task(struct task_struct *t) /* Allocate a return stack for each task */ static int start_graph_tracing(void) { - struct ftrace_ret_stack **ret_stack_list; + unsigned long **ret_stack_list; int ret, cpu; - ret_stack_list = kmalloc_array(FTRACE_RETSTACK_ALLOC_SIZE, - sizeof(struct ftrace_ret_stack *), - GFP_KERNEL); + ret_stack_list = kmalloc(SHADOW_STACK_SIZE, GFP_KERNEL); if (!ret_stack_list) return -ENOMEM; From patchwork Tue Feb 6 15:08:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197516 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1622857dyb; Tue, 6 Feb 2024 07:36:05 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCXu9TPNLFTFMAuPQx8gtrxURwO5mlJdN1ZPXO6vmo/HJIRzJTc8HBUP+xne1LPpC7iniEYlfMQXxZNXn7LXdaWKupC38Q== X-Google-Smtp-Source: AGHT+IGoWK627b7160YQpXxlCXFBkYd8elGete5Wb2tK0+E50kGEwnPuPDciJS/R6scpk9Ln0siu X-Received: by 2002:a05:6a20:8fa5:b0:19c:9b48:6982 with SMTP id k37-20020a056a208fa500b0019c9b486982mr1354678pzj.46.1707233764982; Tue, 06 Feb 2024 07:36:04 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707233764; cv=pass; d=google.com; s=arc-20160816; b=IkLLyCSOMR9pAKt5fdDx2cKp5HQ81fTlHpd3vsY19PVjHS9HhBovhU0XM5pNlbc079 9Semq6Uh3O7ePMjyOniu3ooJWdoMElUHn6vmTF2R0R0o+5OjpRz1/0MiHp8MaWCcCjv6 izNQTG64GU677svHmsqOs4C0BcBsh9BIkIMvrOr/xbwELRrZwTP9ZxgmCNhZ0iGhXApn 8Xszug+nn9G+nrGCy1TF4rVLJDiJmpTeqKLxdwPX2Hu0LgzmBvSxmha5BBckdM//nAYQ EmoFB2GM2o09b/DKC9/ZNPuNtar+SXfcKIejiv+j5d8PWQQQhAGAMq8llEz1+u4JgUOa WG/g== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=eVgZgoy+UGwGEm8Va6QmNfArgtjudfTwfrDl+ut0Pgc=; fh=s3W1i72z/3bE+tNYiebyT99UNryX/4XgPZgBtMXsGWc=; b=Da9V2+5lFcfbe/vmqydc3F4j1Bh4HLyWy+pdU3TUP7PxsykUOO7wR06S1J7FY2DJxQ ZIXKUw6EtvL4nUzgiVS2Nsr2DwQZw8GGM39kMlAGAdS+tqWRG3GN525ByhfUrCTRC4Qm 5oo/pw0eGdQit8T7xADTzVFEv5HNJcmjjFSm2ug1ALYedqWFeOJfbuVy16a1+lJIDW/g rIYA8bYkfgZFS/gHM7WBE2F+l32Wahh41YvPWVuU9Gz8GrEdnxh3TSPPnCvy99BwUKmr Ih49eljoGl3c0XDMSnM6/MwygZsk1kq5HIl1mFheSNlq78wFP7lZ6aP51iyPVo/Ey42c nRDQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=MQnVhub6; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55181-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55181-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=1; AJvYcCVVzCvaqqJyu/KiVEri4JK0rJPxzFA+XNpKXmwprvORn3TFhBbhp9isCo+ADA8lpRddU314l86IBI+RRFV0D1lMcEsL3w== Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [147.75.48.161]) by mx.google.com with ESMTPS id o125-20020a62cd83000000b006d9aa48c4e7si1727679pfg.110.2024.02.06.07.36.04 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:36:04 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55181-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=MQnVhub6; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55181-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55181-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 761D3B2AFF7 for ; Tue, 6 Feb 2024 15:10:18 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 2BDF1134CD2; Tue, 6 Feb 2024 15:08:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="MQnVhub6" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 819CE134CC2; Tue, 6 Feb 2024 15:08:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232122; cv=none; b=g7IOv0L98FYjR93FZS7NdLHeWHoOnK5irUzvrPoX4rxiZ3lgtp3pJPwaKecuyA/WcvBclx15QlyBXXmej18HbswFf98zYRk8E++Z+LhncNwyCqzPtJFl/diJP/Yar0N5fLraSTTZChshfg3pex7s9yDEM9jmhgXq9ci19H0FGn4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232122; c=relaxed/simple; bh=v6a8A4n/NQNVvl6IUBlyb4ncY6wKsPnocwxI4QmL7G4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=RxYrAgHFY/XMTShB4tEbeYzU5CCJqiRfSElOu7FEficjBzOUaSOcnRW1kITOo6Hg8BWW+4FGAIBdW4ca1q/a7hXYf+1iVhjJbj3H7pn9Yeq5E3jGIMb5axo40tE1EJTw4G9NgeCyPbFJ+Sqk3eVqkODon3fHgdURsIeITw3emk4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=MQnVhub6; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9719CC43390; Tue, 6 Feb 2024 15:08:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232122; bh=v6a8A4n/NQNVvl6IUBlyb4ncY6wKsPnocwxI4QmL7G4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=MQnVhub6V+x9VzaS4J+qqX7sfnUSbbornWCaLcO3hsFS+ImOvdng9znvsrF0imNj2 A4VKH1ng2FtNeyUZxdr822jTEUVpGZUQQfzbgMA5A8G+qJNI356PCOTZAn7wLasmub h4gGripv/GYAmh1BJrnI9JTIARJQZ9XwBx6Z3GCZf8eBxHnk1voVrESy+g6Y8Vqvdg pQe1DGQEi2UAaOKC4z4Hzsw5Ob72m2R8dTNqqlEfwKHea11gbf5hM5iArPtJKK+vfX 9kvl2G0FyEUVJYUjKRDEdBQ2a1fenhSY79OpQcp6kQUXLMlZBiRmjGUi6ztJ9xmlKd SQ30nxW1lKKqw== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 06/36] fgraph: Use BUILD_BUG_ON() to make sure we have structures divisible by long Date: Wed, 7 Feb 2024 00:08:36 +0900 Message-Id: <170723211649.502590.2604541564082439754.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790164352392655602 X-GMAIL-MSGID: 1790164352392655602 From: Steven Rostedt (VMware) Instead of using "ALIGN()", use BUILD_BUG_ON() as the structures should always be divisible by sizeof(long). Link: http://lkml.kernel.org/r/20190524111144.GI2589@hirez.programming.kicks-ass.net Suggested-by: Peter Zijlstra Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- Changes in v7: - Use DIV_ROUND_UP() to calculate FGRAPH_RET_INDEX --- kernel/trace/fgraph.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index 30edeb6d4aa9..6f8d36370994 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -26,10 +26,9 @@ #endif #define FGRAPH_RET_SIZE sizeof(struct ftrace_ret_stack) -#define FGRAPH_RET_INDEX (ALIGN(FGRAPH_RET_SIZE, sizeof(long)) / sizeof(long)) +#define FGRAPH_RET_INDEX DIV_ROUND_UP(FGRAPH_RET_SIZE, sizeof(long)) #define SHADOW_STACK_SIZE (PAGE_SIZE) -#define SHADOW_STACK_INDEX \ - (ALIGN(SHADOW_STACK_SIZE, sizeof(long)) / sizeof(long)) +#define SHADOW_STACK_INDEX (SHADOW_STACK_SIZE / sizeof(long)) /* Leave on a buffer at the end */ #define SHADOW_STACK_MAX_INDEX (SHADOW_STACK_INDEX - FGRAPH_RET_INDEX) @@ -91,6 +90,8 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func, if (!current->ret_stack) return -EBUSY; + BUILD_BUG_ON(SHADOW_STACK_SIZE % sizeof(long)); + /* * We must make sure the ret_stack is tested before we read * anything else. @@ -325,6 +326,8 @@ ftrace_graph_get_ret_stack(struct task_struct *task, int idx) { int index = task->curr_ret_stack; + BUILD_BUG_ON(FGRAPH_RET_SIZE % sizeof(long)); + index -= FGRAPH_RET_INDEX * (idx + 1); if (index < 0) return NULL; From patchwork Tue Feb 6 15:08:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197517 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1623395dyb; Tue, 6 Feb 2024 07:36:51 -0800 (PST) X-Google-Smtp-Source: AGHT+IGfsEnSAuwUzhA3D8sJQtWB3A1chg7Lfc107QxBa2bmGwASuuycRa7lBFQR7Jee/BkyYd9U X-Received: by 2002:a17:902:ec82:b0:1d9:bfde:d635 with SMTP id x2-20020a170902ec8200b001d9bfded635mr2484518plg.18.1707233811435; Tue, 06 Feb 2024 07:36:51 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707233811; cv=pass; d=google.com; s=arc-20160816; b=jF3yF61xXt6zKfJOk2+KkIXnyjBcqrIgHyZTxmi6qFemPaEbvbDft6cEWuzdE9QYSI DoJTMBL0bf8oKv8El1KyKj/1YCKZK/RMqfkVgrL9e5Sby0Z5W0Jo0UlDha95quGo2Puz ISm0AmkrdrshAV7hpIo/6RC2+h3xFtzBUldu9Q/5eAWW1De0vH2PmdtACbH7zJ+SROQa LAdkgPK7p4vlLAPz6MWSV/bz/Tp+VNci2tySUXysmfy6W+AGpnWKlmFrr/XqJROkd5wJ vQpCSqu9Mb+K/Of3SrYlY9v553DCLcWDA3vt4w/9C7Ex3R9OYLnRYKTzTt6fvqPQHP7a v/eg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=fs82T8hV0vQbKPNq2BQvLSid2V3dkDLdahl1iTwKtcg=; fh=h8I8bfuvAZAX9JKag/b2vUScFi3hBOlyiMGXhZJucv8=; b=wMZdnn0oGuJGXOS8kH8UuykLvfycJ7FhgTRksfLCf4uwQSm8Ytqlj0KQBXjLUK8lv0 rzBUBSx8twYifuIL46ZbuoUPxYZUmVZ/Jid1y8SnpyDpCrMMME0OMRqkwAWLIQdVbMQi EB12ela5vhNC5rRflGuN7RzGYHxKEmQw3JIuClixbMqdSjIF3mSiwQFWib8+sWwretc8 vTSH6LDaanjvNxdde1fLWgNIn8C3Rk6ok8cClheahuLjNCuA6acYETA+bLGGoRlcDK/T wdtf6WqdhpGaePNOInJmJjuBtQrN4p91xe02FUrnMaRXihJfeXLqs8lnWjaw6p+GiqTY Ayjw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=PfQDgU2z; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55182-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55182-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=1; AJvYcCWZXB6yOtjTxpshl9kI4yqESqjvjiSA+zSg0CsV5Ukzs62opWH3WB3SnhAM3KSEqWVEhJg0FIFKd/sqz7/nIPucUsPe/w== Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id h1-20020a170902eec100b001d8f7d2a3fbsi1743202plb.635.2024.02.06.07.36.51 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:36:51 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55182-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=PfQDgU2z; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55182-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55182-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 5E973B2E0CE for ; Tue, 6 Feb 2024 15:10:39 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id F22CD13340C; Tue, 6 Feb 2024 15:08:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="PfQDgU2z" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BD2981E866; Tue, 6 Feb 2024 15:08:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232134; cv=none; b=uaxMXKwDQlUpYgpMbPGNWD9/kHgHkMWO3ohz0bLI0Kt1D7jv3E51bXRfQnIh3qohx40KIGDi3jvkyL44Ag9or0+W4DJhLPo2eNiSZH1vugHMF6zz99bqmQcoc2R4vlN2abtSNaNlhgJAwTyLAiGqCNWKGsLEx0+CcMAMcT/gWBo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232134; c=relaxed/simple; bh=tA+xME0cqxvdm/SL6/3HnWW/4VMwGO83nJFglJrGK1Y=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=FXDQiQ6L7DRo6zYiEEv5j4WMrdJmk+mvNRVuHZFVxI77rr7gyIqKCo34i6edfRltiBo8xmjXtHew9o/Shgl7LHxmOFsiZ5ITZR0JjRnKnmXFNh7wBFxOT3857Trd0MGh8T3ihLXj5JGPPZbXdRbYt7FGePX9Oni23MWltCWltCA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=PfQDgU2z; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9206DC433F1; Tue, 6 Feb 2024 15:08:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232134; bh=tA+xME0cqxvdm/SL6/3HnWW/4VMwGO83nJFglJrGK1Y=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PfQDgU2zs3j1o4+AE+6YK5rzCbJuIiEoN9VVdriOGJ6sckJ15hwOUqUFH9NspK135 bVquqFtn2cXt3nPGo06vmKW4LkCAfSOwf2IxLgzS7TSP2dCG+ZKgQl6gBEaZRdwzZB eKI/LNC/IQbGKYEVIg5jVKriIEnf47vgOa83hXqiyzcp+iwQTLFp6/jKW8Nj/7A5ml XZEc+HYjL2FA9uYk4Hs5WIK+f328h+Zvrs9xh6QZ4IaLrsa9BiQXBD3/yZuFk2SUyi YklEQ/s1EUJZjCi/Gg6mFcTKE47wr8VbWynG/+1nRLo/+7T1KBr40e5ZiINy/U6fjN 6C9SL5oYrrfaw== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 07/36] function_graph: Add an array structure that will allow multiple callbacks Date: Wed, 7 Feb 2024 00:08:47 +0900 Message-Id: <170723212744.502590.17637620718033431909.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790164401032888721 X-GMAIL-MSGID: 1790164401032888721 From: Steven Rostedt (VMware) Add an array structure that will eventually allow the function graph tracer to have up to 16 simultaneous callbacks attached. It's an array of 16 fgraph_ops pointers, that is assigned when one is registered. On entry of a function the entry of the first item in the array is called, and if it returns zero, then the callback returns non zero if it wants the return callback to be called on exit of the function. The array will simplify the process of having more than one callback attached to the same function, as its index into the array can be stored on the shadow stack. We need to only save the index, because this will allow the fgraph_ops to be freed before the function returns (which may happen if the function call schedule for a long time). Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- Changes in v2: - Remove unneeded brace. --- kernel/trace/fgraph.c | 114 +++++++++++++++++++++++++++++++++++-------------- 1 file changed, 81 insertions(+), 33 deletions(-) diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index 6f8d36370994..3f9dd213e7d8 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -39,6 +39,11 @@ DEFINE_STATIC_KEY_FALSE(kill_ftrace_graph); int ftrace_graph_active; +static int fgraph_array_cnt; +#define FGRAPH_ARRAY_SIZE 16 + +static struct fgraph_ops *fgraph_array[FGRAPH_ARRAY_SIZE]; + /* Both enabled by default (can be cleared by function_graph tracer flags */ static bool fgraph_sleep_time = true; @@ -62,6 +67,20 @@ int __weak ftrace_disable_ftrace_graph_caller(void) } #endif +int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace) +{ + return 0; +} + +static void ftrace_graph_ret_stub(struct ftrace_graph_ret *trace) +{ +} + +static struct fgraph_ops fgraph_stub = { + .entryfunc = ftrace_graph_entry_stub, + .retfunc = ftrace_graph_ret_stub, +}; + /** * ftrace_graph_stop - set to permanently disable function graph tracing * @@ -159,7 +178,7 @@ int function_graph_enter(unsigned long ret, unsigned long func, goto out; /* Only trace if the calling function expects to */ - if (!ftrace_graph_entry(&trace)) + if (!fgraph_array[0]->entryfunc(&trace)) goto out_ret; return 0; @@ -274,7 +293,7 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs trace.retval = fgraph_ret_regs_return_value(ret_regs); #endif trace.rettime = trace_clock_local(); - ftrace_graph_return(&trace); + fgraph_array[0]->retfunc(&trace); /* * The ftrace_graph_return() may still access the current * ret_stack structure, we need to make sure the update of @@ -410,11 +429,6 @@ void ftrace_graph_sleep_time_control(bool enable) fgraph_sleep_time = enable; } -int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace) -{ - return 0; -} - /* * Simply points to ftrace_stub, but with the proper protocol. * Defined by the linker script in linux/vmlinux.lds.h @@ -652,37 +666,54 @@ static int start_graph_tracing(void) int register_ftrace_graph(struct fgraph_ops *gops) { int ret = 0; + int i; mutex_lock(&ftrace_lock); - /* we currently allow only one tracer registered at a time */ - if (ftrace_graph_active) { + if (!fgraph_array[0]) { + /* The array must always have real data on it */ + for (i = 0; i < FGRAPH_ARRAY_SIZE; i++) + fgraph_array[i] = &fgraph_stub; + } + + /* Look for an available spot */ + for (i = 0; i < FGRAPH_ARRAY_SIZE; i++) { + if (fgraph_array[i] == &fgraph_stub) + break; + } + if (i >= FGRAPH_ARRAY_SIZE) { ret = -EBUSY; goto out; } - register_pm_notifier(&ftrace_suspend_notifier); + fgraph_array[i] = gops; + if (i + 1 > fgraph_array_cnt) + fgraph_array_cnt = i + 1; ftrace_graph_active++; - ret = start_graph_tracing(); - if (ret) { - ftrace_graph_active--; - goto out; - } - ftrace_graph_return = gops->retfunc; + if (ftrace_graph_active == 1) { + register_pm_notifier(&ftrace_suspend_notifier); + ret = start_graph_tracing(); + if (ret) { + ftrace_graph_active--; + goto out; + } + + ftrace_graph_return = gops->retfunc; - /* - * Update the indirect function to the entryfunc, and the - * function that gets called to the entry_test first. Then - * call the update fgraph entry function to determine if - * the entryfunc should be called directly or not. - */ - __ftrace_graph_entry = gops->entryfunc; - ftrace_graph_entry = ftrace_graph_entry_test; - update_function_graph_func(); + /* + * Update the indirect function to the entryfunc, and the + * function that gets called to the entry_test first. Then + * call the update fgraph entry function to determine if + * the entryfunc should be called directly or not. + */ + __ftrace_graph_entry = gops->entryfunc; + ftrace_graph_entry = ftrace_graph_entry_test; + update_function_graph_func(); - ret = ftrace_startup(&graph_ops, FTRACE_START_FUNC_RET); + ret = ftrace_startup(&graph_ops, FTRACE_START_FUNC_RET); + } out: mutex_unlock(&ftrace_lock); return ret; @@ -690,19 +721,36 @@ int register_ftrace_graph(struct fgraph_ops *gops) void unregister_ftrace_graph(struct fgraph_ops *gops) { + int i; + mutex_lock(&ftrace_lock); if (unlikely(!ftrace_graph_active)) goto out; - ftrace_graph_active--; - ftrace_graph_return = ftrace_stub_graph; - ftrace_graph_entry = ftrace_graph_entry_stub; - __ftrace_graph_entry = ftrace_graph_entry_stub; - ftrace_shutdown(&graph_ops, FTRACE_STOP_FUNC_RET); - unregister_pm_notifier(&ftrace_suspend_notifier); - unregister_trace_sched_switch(ftrace_graph_probe_sched_switch, NULL); + for (i = 0; i < fgraph_array_cnt; i++) + if (gops == fgraph_array[i]) + break; + if (i >= fgraph_array_cnt) + goto out; + fgraph_array[i] = &fgraph_stub; + if (i + 1 == fgraph_array_cnt) { + for (; i >= 0; i--) + if (fgraph_array[i] != &fgraph_stub) + break; + fgraph_array_cnt = i + 1; + } + + ftrace_graph_active--; + if (!ftrace_graph_active) { + ftrace_graph_return = ftrace_stub_graph; + ftrace_graph_entry = ftrace_graph_entry_stub; + __ftrace_graph_entry = ftrace_graph_entry_stub; + ftrace_shutdown(&graph_ops, FTRACE_STOP_FUNC_RET); + unregister_pm_notifier(&ftrace_suspend_notifier); + unregister_trace_sched_switch(ftrace_graph_probe_sched_switch, NULL); + } out: mutex_unlock(&ftrace_lock); } From patchwork Tue Feb 6 15:08:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197521 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1625321dyb; Tue, 6 Feb 2024 07:39:58 -0800 (PST) X-Google-Smtp-Source: AGHT+IGUBr6HqmumTtVUG8x5FLSNgoYexORLu7bug2AiyTwN7UNNIIyyjyM2KZeWRXLWtSGtroQm X-Received: by 2002:a05:6870:a10f:b0:219:2de5:5217 with SMTP id m15-20020a056870a10f00b002192de55217mr3264878oae.35.1707233998500; Tue, 06 Feb 2024 07:39:58 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707233998; cv=pass; d=google.com; s=arc-20160816; b=wyBvKJGiQWeicKwZkQ8iqK2QgAz19pO57LIFSo7LWTdBp7jHvCu3By2NvuvrXvlvpp bZ00fPu0E7SW3BzGPidvDg4uTywgctiewElbGysqCFp2eA0whnZUHIzmS2860e5zUEVA qSM3V8vt+z1Lj7o2OXKerUdveqOqWUd4OQ5U+WpMUAii8M9Olvr+9Gw9+KCjCIDzBA2G fY75za4l7/hzKBkuu1mi6GADvDtUQ7J87BZP5G/iMTjzDwhxIZ63lK5/mQfFN1Mj76tE pNI5dCwdoM50CDnBBwH42WaEJ1d0PYtVe7gAs+t/IFysL9OGXS8SmFvZfHMsaFzp5RIK OISQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=zC0tYu77vRd+bdKXM2XLfszazihA1nTAh/Y3uuJllo0=; fh=O4fHYRJDdGp3BWSc8Gq3HMDDJ7LMlxkr9gB1Tvj33ZM=; b=rGCetk0iTEjwy1Uz7bfi62KQInBbZ1Ktze0rXFgNz+LMbYFZdeoq6xZ7xejgdQvofR nQlmBD6vjZiQcRujzxxCvqo1i0Co3ObH2RZwf8sAgVeDWpMDub+WhfYOaP5hx1HEDbqO f+cKwJkC5NuKBcG898kmpxjzRcZHoM2vKreH9qwGx1AuyL1GlYKlzAE8kIEDTkPoK3Jr DJle7fm952sRGj3zDdAzN0rkkp3GeTZ1KxkY+G90nEpznvQ0+Ll5MV3aLQTCQZJS3RLq baAEpYNwdm6EblLU0PzX/NAgN8XofxvRXi0vfJ2r11UwNhZBjr98wrppQLuNYh7Ec/hv Nc+g==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=gso4WZ3t; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55184-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55184-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=1; AJvYcCXgpiakEJb0K+3jkg1xR15F18TItLccHQRZNDKxqWRX2vRBuGU5qJIOqQsNLOFJaPpKLSk+KLDGHLWcIOLgR7+MBKWhaw== Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [147.75.48.161]) by mx.google.com with ESMTPS id i21-20020a632215000000b005c69365abc9si1847014pgi.687.2024.02.06.07.39.57 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:39:58 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55184-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=gso4WZ3t; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55184-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55184-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id D8044B2A3D8 for ; Tue, 6 Feb 2024 15:11:26 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 0D58D1350EF; Tue, 6 Feb 2024 15:09:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="gso4WZ3t" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2E88B1350DE; Tue, 6 Feb 2024 15:09:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232145; cv=none; b=IwUDE7PWpEXkq8zGBDiuOcLvmB6UeHF6VelZB9QrS3EifIkolEYok2khtzGvct+cbwDQzrATPz7oT4dMFfWBGJxeAylowUDq+qcGSt8hRwpMuOCtTqOuRSODoqW177oTJaEwIbzW8um14uGmS6BqdHNHk5Z1AplY2I6khzv9lBk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232145; c=relaxed/simple; bh=JvpjIsFGwi7NsXme8GOIZ24f5xGxw81lrlfjHeVu8uU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=hsMeoPwgywOCg17mZ0bHmpcZNGch927ySVd78FHxLgCfgIkoxNdXvOH6FqXXDvSyyq+9RivtOYEjpTbrUnOCJsYH1Iw4r3NXRKGaMgKJiMQ7p+mLgB9NhxzjdorIG0vjyJIDZxCw9oVBqc+/CJEx1Ip4DsXa1Xr1+jC6Oj0bPIQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=gso4WZ3t; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id AEA29C433F1; Tue, 6 Feb 2024 15:09:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232145; bh=JvpjIsFGwi7NsXme8GOIZ24f5xGxw81lrlfjHeVu8uU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gso4WZ3toCiFKX1/hFRhJE3WsAEZfLR266EtfxkW/x1Q2sxGK7sYg89aPcaPKnPG/ vfDSB2p2PWkthSXv5n/elBVoyMQS31oMr4boQq5i3XSMbYUg8uoQNpAcUXB7hjoQE4 3M8kfOwyff2e9iOt1Vt0UzycJLh5rVRh6dhgX3aygBxhFJEaaF7fZTVhm+5Gj2VtOT wl/WJq/1IP6MJnmy0qGz6puw3gyyuTbPP4GpIjf3zunq4MXnEH6ZSU9iNjNf+DXRJc 95fdKPTBWcF38lEe+Yp/5/XgIe0uZzb/n55EYM0AhQ49UEkMlQ9jLr5oe0ro6IHMj3 d27zukmbYwLcw== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 08/36] function_graph: Allow multiple users to attach to function graph Date: Wed, 7 Feb 2024 00:08:59 +0900 Message-Id: <170723213956.502590.13803494278920984208.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790164597064364430 X-GMAIL-MSGID: 1790164597064364430 From: Steven Rostedt (VMware) Allow for multiple users to attach to function graph tracer at the same time. Only 16 simultaneous users can attach to the tracer. This is because there's an array that stores the pointers to the attached fgraph_ops. When a function being traced is entered, each of the ftrace_ops entryfunc is called and if it returns non zero, its index into the array will be added to the shadow stack. On exit of the function being traced, the shadow stack will contain the indexes of the ftrace_ops on the array that want their retfunc to be called. Because a function may sleep for a long time (if a task sleeps itself), the return of the function may be literally days later. If the ftrace_ops is removed, its place on the array is replaced with a ftrace_ops that contains the stub functions and that will be called when the function finally returns. If another ftrace_ops is added that happens to get the same index into the array, its return function may be called. But that's actually the way things current work with the old function graph tracer. If one tracer is removed and another is added, the new one will get the return calls of the function traced by the previous one, thus this is not a regression. This can be fixed by adding a counter to each time the array item is updated and save that on the shadow stack as well, such that it won't be called if the index saved does not match the index on the array. Note, being able to filter functions when both are called is not completely handled yet, but that shouldn't be too hard to manage. Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- Changes in v7: - Fix max limitation check in ftrace_graph_push_return(). - Rewrite the shadow stack implementation using bitmap entry. This allows us to detect recursive call/tail call easier. (this implementation is moved from later patch in the series. Changes in v2: - Check return value of the ftrace_pop_return_trace() instead of 'ret' since 'ret' is set to the address of panic(). - Fix typo and make lines shorter than 76 chars in description. --- include/linux/ftrace.h | 1 kernel/trace/fgraph.c | 360 ++++++++++++++++++++++++++++++++++++++++-------- 2 files changed, 301 insertions(+), 60 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 39ac1f3e8041..b39cc6d9b4a5 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -1066,6 +1066,7 @@ extern int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace); struct fgraph_ops { trace_func_graph_ent_t entryfunc; trace_func_graph_ret_t retfunc; + int idx; }; /* diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index 3f9dd213e7d8..b9a2399b75ee 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -7,6 +7,7 @@ * * Highly modified by Steven Rostedt (VMware). */ +#include #include #include #include @@ -27,23 +28,157 @@ #define FGRAPH_RET_SIZE sizeof(struct ftrace_ret_stack) #define FGRAPH_RET_INDEX DIV_ROUND_UP(FGRAPH_RET_SIZE, sizeof(long)) + +/* + * On entry to a function (via function_graph_enter()), a new ftrace_ret_stack + * is allocated on the task's ret_stack with indexes entry, then each + * fgraph_ops on the fgraph_array[]'s entryfunc is called and if that returns + * non-zero, the index into the fgraph_array[] for that fgraph_ops is recorded + * on the indexes entry as a bit flag. + * As the associated ftrace_ret_stack saved for those fgraph_ops needs to + * be found, the index to it is also added to the ret_stack along with the + * index of the fgraph_array[] to each fgraph_ops that needs their retfunc + * called. + * + * The top of the ret_stack (when not empty) will always have a reference + * to the last ftrace_ret_stack saved. All references to the + * ftrace_ret_stack has the format of: + * + * bits: 0 - 9 offset in words from the previous ftrace_ret_stack + * (bitmap type should have FGRAPH_RET_INDEX always) + * bits: 10 - 11 Type of storage + * 0 - reserved + * 1 - bitmap of fgraph_array index + * + * For bitmap of fgraph_array index + * bits: 12 - 27 The bitmap of fgraph_ops fgraph_array index + * + * That is, at the end of function_graph_enter, if the first and forth + * fgraph_ops on the fgraph_array[] (index 0 and 3) needs their retfunc called + * on the return of the function being traced, this is what will be on the + * task's shadow ret_stack: (the stack grows upward) + * + * | | <- task->curr_ret_stack + * +--------------------------------------------+ + * | bitmap_type(bitmap:(BIT(3)|BIT(0)), | + * | offset:FGRAPH_RET_INDEX) | <- the offset is from here + * +--------------------------------------------+ + * | struct ftrace_ret_stack | + * | (stores the saved ret pointer) | <- the offset points here + * +--------------------------------------------+ + * | (X) | (N) | ( N words away from + * | | previous ret_stack) + * + * If a backtrace is required, and the real return pointer needs to be + * fetched, then it looks at the task's curr_ret_stack index, if it + * is greater than zero (reserved, or right before poped), it would mask + * the value by FGRAPH_RET_INDEX_MASK to get the offset index of the + * ftrace_ret_stack structure stored on the shadow stack. + */ + +#define FGRAPH_RET_INDEX_SIZE 10 +#define FGRAPH_RET_INDEX_MASK GENMASK(FGRAPH_RET_INDEX_SIZE - 1, 0) + +#define FGRAPH_TYPE_SIZE 2 +#define FGRAPH_TYPE_MASK GENMASK(FGRAPH_TYPE_SIZE - 1, 0) +#define FGRAPH_TYPE_SHIFT FGRAPH_RET_INDEX_SIZE + +enum { + FGRAPH_TYPE_RESERVED = 0, + FGRAPH_TYPE_BITMAP = 1, +}; + +#define FGRAPH_INDEX_SIZE 16 +#define FGRAPH_INDEX_MASK GENMASK(FGRAPH_INDEX_SIZE - 1, 0) +#define FGRAPH_INDEX_SHIFT (FGRAPH_TYPE_SHIFT + FGRAPH_TYPE_SIZE) + +/* Currently the max stack index can't be more than register callers */ +#define FGRAPH_MAX_INDEX (FGRAPH_INDEX_SIZE + FGRAPH_RET_INDEX) + +#define FGRAPH_ARRAY_SIZE FGRAPH_INDEX_SIZE + #define SHADOW_STACK_SIZE (PAGE_SIZE) #define SHADOW_STACK_INDEX (SHADOW_STACK_SIZE / sizeof(long)) /* Leave on a buffer at the end */ -#define SHADOW_STACK_MAX_INDEX (SHADOW_STACK_INDEX - FGRAPH_RET_INDEX) +#define SHADOW_STACK_MAX_INDEX (SHADOW_STACK_INDEX - (FGRAPH_RET_INDEX + 1)) #define RET_STACK(t, index) ((struct ftrace_ret_stack *)(&(t)->ret_stack[index])) -#define RET_STACK_INC(c) ({ c += FGRAPH_RET_INDEX; }) -#define RET_STACK_DEC(c) ({ c -= FGRAPH_RET_INDEX; }) DEFINE_STATIC_KEY_FALSE(kill_ftrace_graph); int ftrace_graph_active; static int fgraph_array_cnt; -#define FGRAPH_ARRAY_SIZE 16 static struct fgraph_ops *fgraph_array[FGRAPH_ARRAY_SIZE]; +static inline int get_ret_stack_index(struct task_struct *t, int offset) +{ + return t->ret_stack[offset] & FGRAPH_RET_INDEX_MASK; +} + +static inline int get_fgraph_type(struct task_struct *t, int offset) +{ + return (t->ret_stack[offset] >> FGRAPH_TYPE_SHIFT) & FGRAPH_TYPE_MASK; +} + +static inline unsigned long +get_fgraph_index_bitmap(struct task_struct *t, int offset) +{ + return (t->ret_stack[offset] >> FGRAPH_INDEX_SHIFT) & FGRAPH_INDEX_MASK; +} + +static inline void +set_fgraph_index_bitmap(struct task_struct *t, int offset, unsigned long bitmap) +{ + t->ret_stack[offset] = (bitmap << FGRAPH_INDEX_SHIFT) | + (FGRAPH_TYPE_BITMAP << FGRAPH_TYPE_SHIFT) | FGRAPH_RET_INDEX; +} + +static inline bool is_fgraph_index_set(struct task_struct *t, int offset, int idx) +{ + return !!(get_fgraph_index_bitmap(t, offset) & BIT(idx)); +} + +static inline void +add_fgraph_index_bitmap(struct task_struct *t, int offset, unsigned long bitmap) +{ + t->ret_stack[offset] |= (bitmap << FGRAPH_INDEX_SHIFT); +} + +/* + * @offset: The index into @t->ret_stack to find the ret_stack entry + * @index: Where to place the index into @t->ret_stack of that entry + * + * Calling this with: + * + * offset = task->curr_ret_stack; + * do { + * ret_stack = get_ret_stack(task, offset, &offset); + * } while (ret_stack); + * + * Will iterate through all the ret_stack entries from curr_ret_stack + * down to the first one. + */ +static inline struct ftrace_ret_stack * +get_ret_stack(struct task_struct *t, int offset, int *index) +{ + int idx; + + BUILD_BUG_ON(FGRAPH_RET_SIZE % sizeof(long)); + + if (unlikely(offset <= 0)) + return NULL; + + idx = get_ret_stack_index(t, --offset); + if (WARN_ON_ONCE(idx <= 0 || idx > offset)) + return NULL; + + offset -= idx; + + *index = offset; + return RET_STACK(t, offset); +} + /* Both enabled by default (can be cleared by function_graph tracer flags */ static bool fgraph_sleep_time = true; @@ -97,10 +232,12 @@ void ftrace_graph_stop(void) /* Add a function return address to the trace stack on thread info.*/ static int ftrace_push_return_trace(unsigned long ret, unsigned long func, - unsigned long frame_pointer, unsigned long *retp) + unsigned long frame_pointer, unsigned long *retp, + int fgraph_idx) { struct ftrace_ret_stack *ret_stack; unsigned long long calltime; + unsigned long val; int index; if (unlikely(ftrace_graph_is_dead())) @@ -109,6 +246,21 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func, if (!current->ret_stack) return -EBUSY; + /* + * At first, check whether the previous fgraph callback is pushed by + * the fgraph on the same function entry. + * But if @func is the self tail-call function, we also need to ensure + * the ret_stack is not for the previous call by checking whether the + * bit of @fgraph_idx is set or not. + */ + ret_stack = get_ret_stack(current, current->curr_ret_stack, &index); + if (ret_stack && ret_stack->func == func && + get_fgraph_type(current, index + FGRAPH_RET_INDEX) == FGRAPH_TYPE_BITMAP && + !is_fgraph_index_set(current, index + FGRAPH_RET_INDEX, fgraph_idx)) + return index + FGRAPH_RET_INDEX; + + val = (FGRAPH_TYPE_RESERVED << FGRAPH_TYPE_SHIFT) | FGRAPH_RET_INDEX; + BUILD_BUG_ON(SHADOW_STACK_SIZE % sizeof(long)); /* @@ -118,17 +270,44 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func, smp_rmb(); /* The return trace stack is full */ - if (current->curr_ret_stack >= SHADOW_STACK_MAX_INDEX) { + if (current->curr_ret_stack + FGRAPH_RET_INDEX + 1 >= SHADOW_STACK_MAX_INDEX) { atomic_inc(¤t->trace_overrun); return -EBUSY; } calltime = trace_clock_local(); - index = current->curr_ret_stack; - RET_STACK_INC(current->curr_ret_stack); + index = READ_ONCE(current->curr_ret_stack); ret_stack = RET_STACK(current, index); + index += FGRAPH_RET_INDEX; + + /* ret offset = FGRAPH_RET_INDEX ; type = reserved */ + current->ret_stack[index] = val; + ret_stack->ret = ret; + /* + * The unwinders expect curr_ret_stack to point to either zero + * or an index where to find the next ret_stack. Even though the + * ret stack might be bogus, we want to write the ret and the + * index to find the ret_stack before we increment the stack point. + * If an interrupt comes in now before we increment the curr_ret_stack + * it may blow away what we wrote. But that's fine, because the + * index will still be correct (even though the 'ret' won't be). + * What we worry about is the index being correct after we increment + * the curr_ret_stack and before we update that index, as if an + * interrupt comes in and does an unwind stack dump, it will need + * at least a correct index! + */ + barrier(); + current->curr_ret_stack = index + 1; + /* + * This next barrier is to ensure that an interrupt coming in + * will not corrupt what we are about to write. + */ barrier(); + + /* Still keep it reserved even if an interrupt came in */ + current->ret_stack[index] = val; + ret_stack->ret = ret; ret_stack->func = func; ret_stack->calltime = calltime; @@ -138,7 +317,7 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func, #ifdef HAVE_FUNCTION_GRAPH_RET_ADDR_PTR ret_stack->retp = retp; #endif - return 0; + return index; } /* @@ -155,10 +334,14 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func, # define MCOUNT_INSN_SIZE 0 #endif +/* If the caller does not use ftrace, call this function. */ int function_graph_enter(unsigned long ret, unsigned long func, unsigned long frame_pointer, unsigned long *retp) { struct ftrace_graph_ent trace; + unsigned long bitmap = 0; + int index; + int i; #ifndef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS /* @@ -171,44 +354,59 @@ int function_graph_enter(unsigned long ret, unsigned long func, ftrace_find_rec_direct(ret - MCOUNT_INSN_SIZE)) return -EBUSY; #endif + trace.func = func; trace.depth = ++current->curr_ret_depth; - if (ftrace_push_return_trace(ret, func, frame_pointer, retp)) + index = ftrace_push_return_trace(ret, func, frame_pointer, retp, 0); + if (index < 0) goto out; - /* Only trace if the calling function expects to */ - if (!fgraph_array[0]->entryfunc(&trace)) + for (i = 0; i < fgraph_array_cnt; i++) { + struct fgraph_ops *gops = fgraph_array[i]; + + if (gops == &fgraph_stub) + continue; + + if (gops->entryfunc(&trace)) + bitmap |= BIT(i); + } + + if (!bitmap) goto out_ret; + /* + * Since this function uses fgraph_idx = 0 as a tail-call checking + * flag, set that bit always. + */ + set_fgraph_index_bitmap(current, index, bitmap | BIT(0)); + return 0; out_ret: - RET_STACK_DEC(current->curr_ret_stack); + current->curr_ret_stack -= FGRAPH_RET_INDEX + 1; out: current->curr_ret_depth--; return -EBUSY; } /* Retrieve a function return address to the trace stack on thread info.*/ -static void +static struct ftrace_ret_stack * ftrace_pop_return_trace(struct ftrace_graph_ret *trace, unsigned long *ret, - unsigned long frame_pointer) + unsigned long frame_pointer, int *index) { struct ftrace_ret_stack *ret_stack; - int index; - index = current->curr_ret_stack; - RET_STACK_DEC(index); + ret_stack = get_ret_stack(current, current->curr_ret_stack, index); - if (unlikely(index < 0 || index > SHADOW_STACK_MAX_INDEX)) { + if (unlikely(!ret_stack)) { ftrace_graph_stop(); - WARN_ON(1); + WARN(1, "Bad function graph ret_stack pointer: %d", + current->curr_ret_stack); /* Might as well panic, otherwise we have no where to go */ *ret = (unsigned long)panic; - return; + return NULL; } - ret_stack = RET_STACK(current, index); #ifdef HAVE_FUNCTION_GRAPH_FP_TEST /* * The arch may choose to record the frame pointer used @@ -228,26 +426,29 @@ ftrace_pop_return_trace(struct ftrace_graph_ret *trace, unsigned long *ret, ftrace_graph_stop(); WARN(1, "Bad frame pointer: expected %lx, received %lx\n" " from func %ps return to %lx\n", - current->ret_stack[index].fp, + ret_stack->fp, frame_pointer, (void *)ret_stack->func, ret_stack->ret); *ret = (unsigned long)panic; - return; + return NULL; } #endif + *index += FGRAPH_RET_INDEX; *ret = ret_stack->ret; trace->func = ret_stack->func; trace->calltime = ret_stack->calltime; trace->overrun = atomic_read(¤t->trace_overrun); - trace->depth = current->curr_ret_depth--; + trace->depth = current->curr_ret_depth; /* * We still want to trace interrupts coming in if * max_depth is set to 1. Make sure the decrement is * seen before ftrace_graph_return. */ barrier(); + + return ret_stack; } /* @@ -285,30 +486,47 @@ struct fgraph_ret_regs; static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs, unsigned long frame_pointer) { + struct ftrace_ret_stack *ret_stack; struct ftrace_graph_ret trace; + unsigned long bitmap; unsigned long ret; + int index; + int i; - ftrace_pop_return_trace(&trace, &ret, frame_pointer); + ret_stack = ftrace_pop_return_trace(&trace, &ret, frame_pointer, &index); + + if (unlikely(!ret_stack)) { + ftrace_graph_stop(); + WARN_ON(1); + /* Might as well panic. What else to do? */ + return (unsigned long)panic; + } + + trace.rettime = trace_clock_local(); #ifdef CONFIG_FUNCTION_GRAPH_RETVAL trace.retval = fgraph_ret_regs_return_value(ret_regs); #endif - trace.rettime = trace_clock_local(); - fgraph_array[0]->retfunc(&trace); + + bitmap = get_fgraph_index_bitmap(current, index); + for (i = 0; i < FGRAPH_ARRAY_SIZE; i++) { + struct fgraph_ops *gops = fgraph_array[i]; + + if (!(bitmap & BIT(i))) + continue; + if (gops == &fgraph_stub) + continue; + + gops->retfunc(&trace); + } + /* * The ftrace_graph_return() may still access the current * ret_stack structure, we need to make sure the update of * curr_ret_stack is after that. */ barrier(); - RET_STACK_DEC(current->curr_ret_stack); - - if (unlikely(!ret)) { - ftrace_graph_stop(); - WARN_ON(1); - /* Might as well panic. What else to do? */ - ret = (unsigned long)panic; - } - + current->curr_ret_stack -= FGRAPH_RET_INDEX + 1; + current->curr_ret_depth--; return ret; } @@ -343,15 +561,17 @@ unsigned long ftrace_return_to_handler(unsigned long frame_pointer) struct ftrace_ret_stack * ftrace_graph_get_ret_stack(struct task_struct *task, int idx) { + struct ftrace_ret_stack *ret_stack = NULL; int index = task->curr_ret_stack; - BUILD_BUG_ON(FGRAPH_RET_SIZE % sizeof(long)); - - index -= FGRAPH_RET_INDEX * (idx + 1); if (index < 0) return NULL; - return RET_STACK(task, index); + do { + ret_stack = get_ret_stack(task, index, &index); + } while (ret_stack && --idx >= 0); + + return ret_stack; } /** @@ -374,17 +594,26 @@ unsigned long ftrace_graph_ret_addr(struct task_struct *task, int *idx, unsigned long ret, unsigned long *retp) { struct ftrace_ret_stack *ret_stack; - int index = task->curr_ret_stack; - int i; + int i = task->curr_ret_stack; if (ret != (unsigned long)dereference_kernel_function_descriptor(return_to_handler)) return ret; - RET_STACK_DEC(index); - - for (i = index; i >= 0; RET_STACK_DEC(i)) { - ret_stack = RET_STACK(task, i); - if (ret_stack->retp == retp) + while (i > 0) { + ret_stack = get_ret_stack(current, i, &i); + if (!ret_stack) + break; + /* + * For the tail-call, there would be 2 or more ftrace_ret_stacks on + * the ret_stack, which records "return_to_handler" as the return + * address excpt for the last one. + * But on the real stack, there should be 1 entry because tail-call + * reuses the return address on the stack and jump to the next function. + * Thus we will continue to find real return address. + */ + if (ret_stack->retp == retp && + ret_stack->ret != + (unsigned long)dereference_kernel_function_descriptor(return_to_handler)) return ret_stack->ret; } @@ -394,21 +623,29 @@ unsigned long ftrace_graph_ret_addr(struct task_struct *task, int *idx, unsigned long ftrace_graph_ret_addr(struct task_struct *task, int *idx, unsigned long ret, unsigned long *retp) { - int task_idx; + struct ftrace_ret_stack *ret_stack; + int task_idx = task->curr_ret_stack; + int i; if (ret != (unsigned long)dereference_kernel_function_descriptor(return_to_handler)) return ret; - task_idx = task->curr_ret_stack; - RET_STACK_DEC(task_idx); - - if (!task->ret_stack || task_idx < *idx) + if (!idx) return ret; - task_idx -= *idx; - RET_STACK_INC(*idx); + i = *idx; + do { + ret_stack = get_ret_stack(task, task_idx, &task_idx); + if (ret_stack && ret_stack->ret == + (unsigned long)dereference_kernel_function_descriptor(return_to_handler)) + continue; + i--; + } while (i >= 0 && ret_stack); + + if (ret_stack) + return ret_stack->ret; - return RET_STACK(task, task_idx); + return ret; } #endif /* HAVE_FUNCTION_GRAPH_RET_ADDR_PTR */ @@ -514,10 +751,10 @@ ftrace_graph_probe_sched_switch(void *ignore, bool preempt, */ timestamp -= next->ftrace_timestamp; - for (index = next->curr_ret_stack - FGRAPH_RET_INDEX; index >= 0; ) { - ret_stack = RET_STACK(next, index); - ret_stack->calltime += timestamp; - index -= FGRAPH_RET_INDEX; + for (index = next->curr_ret_stack; index > 0; ) { + ret_stack = get_ret_stack(next, index, &index); + if (ret_stack) + ret_stack->calltime += timestamp; } } @@ -568,6 +805,8 @@ graph_init_task(struct task_struct *t, unsigned long *ret_stack) { atomic_set(&t->trace_overrun, 0); t->ftrace_timestamp = 0; + t->curr_ret_stack = 0; + t->curr_ret_depth = -1; /* make curr_ret_stack visible before we add the ret_stack */ smp_wmb(); t->ret_stack = ret_stack; @@ -689,6 +928,7 @@ int register_ftrace_graph(struct fgraph_ops *gops) fgraph_array[i] = gops; if (i + 1 > fgraph_array_cnt) fgraph_array_cnt = i + 1; + gops->idx = i; ftrace_graph_active++; From patchwork Tue Feb 6 15:09:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197488 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1606102dyb; Tue, 6 Feb 2024 07:12:14 -0800 (PST) X-Google-Smtp-Source: AGHT+IFi1v5GQxf4WmVuIZ72b/X73WRnVqx6ReCdSe+5nc1bFD+pa8OrTSteHGzv/mjgb9bxy4KY X-Received: by 2002:a05:6a20:1e9f:b0:19e:959a:a0d8 with SMTP id dl31-20020a056a201e9f00b0019e959aa0d8mr1599742pzb.21.1707232334683; Tue, 06 Feb 2024 07:12:14 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707232334; cv=pass; d=google.com; s=arc-20160816; b=pUgC1KY6/fbUpOWOvkAZfchqcbUp6dbwpXoJNNTUlhBlYNh9Msdm4mOd7IqerQ9qDF vjTUukq1RfL7c66GK9DVxaMTiaCg1ABItTKw+bgM/aBqDJ8n8uGoWXESAVIQOf+QPMU2 f2csK+Q0dNJbe65hUPNXFs0JTbPoNyBR1EhcKXrjH09cDmIbF0Wog1dQ5SadIz0iR5wk /47HQkMxV3y7qAzWIvjaLu5t7Mtxe/jwa2Gqi2CZW8F7Alr5hzoa2GmacJXw4sOyTxGA uv9q4hA5iNZoZinJunqYS5cyPUR11T+RSWhxbqN10Ysls3it5rMCczNZ65yaa9S7A+Y0 URYA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=HGl8TQvYeyvgNyH0iA+AU5U9eKg8DRlnKydX0SoRpIk=; fh=1cxqcQRHGVIsRCZu8gm0oxmTLaE9lu7HFqpHsRa21N0=; b=VnYE8ZSROvKgsJWc33DOjilMxciOAF2mNFCYlPWn4ToCyRFSyJivRyI7OlzcjiUnnW M0qBWOwW/2vnIB69hvncQThRRmNJC5pPDvrfotkFoBulupqkIcy7Vp1T0CU5MxR7CEQS T11F/iq8vUpDvDzoqBw0ucyjdk833khuxt1rSr910e0I8aaoy4xzXJS9+wCI9IclTSOx bMotP6NT8ZX60clJ5O/XH7+QeMoB9wOB2Lj56LsmvbDHpt4xKfAw6OMAuTlNpFyZF99V kovmkfP4ezr3igd2DAFAmlN1Yr6YSoYK2/gU/0MK6vGcIYpKHI7NWTBExDsUz28lX4wh /jJA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=VxqVnfez; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55185-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55185-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=1; AJvYcCVwr0r8uakqebR1B15z9agp0ciD2x4MocS/ny354XMC7ZuRg+8/J0ym+epknJM2kAVsAzmXhY5B06vPTAkpRD9y18uR2A== Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id p15-20020a056a000b4f00b006e05b7bfcffsi1080011pfo.52.2024.02.06.07.12.14 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:12:14 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55185-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=VxqVnfez; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55185-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55185-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id D92E928D307 for ; Tue, 6 Feb 2024 15:11:45 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 8B46F1353E7; Tue, 6 Feb 2024 15:09:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="VxqVnfez" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5F360133434; Tue, 6 Feb 2024 15:09:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232156; cv=none; b=Q4lSSxuoV+yf4FAHXtXbDeC5s+tLW4EnV7vnkogafZTGNRDhjgZDSt3rXlf6NMJ/TTeYR2GCh3AZ/h8N7PhI6+Ayf9mwp4sqsULsTiePqEz0+Nk1imOdSGjr2ZrcQl1ajmWM8ItQq8tAtyeHnxdDrDv2oJDiWPnTADi/clOvlMo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232156; c=relaxed/simple; bh=WWUXf+6wuLAeTB0u+5WPCf6XUJQtEct3UDtGyxF5nGA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=qPlWAjoqBfV3SUfX8KkyTN4sqR6PyjNMTg3cA/odAUTRdu0YI0fQZZPO0WQ2u6SU6ZdvKT4THOv03ZMu6VvUb8HtIXSjAOzUP/wXB9chiX+54DBSJdOXII+vhe2ka0Ms7n7IoItuPmxV7CVDxqDUTILt/1+UM7mdANAbkBmJtl4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=VxqVnfez; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 91469C433C7; Tue, 6 Feb 2024 15:09:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232155; bh=WWUXf+6wuLAeTB0u+5WPCf6XUJQtEct3UDtGyxF5nGA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VxqVnfezvE1mGVD67gSrJmn+Vz8tbustPz1KfZhWslztyqdAewrAJGzyQuyPaQ8U/ 3HK3ZLYvCLQFNPXj82mH4uWSMf/0oIIjhxdiRekN0B5dVGSakYHyICv8rA69c3H5iD zDECywSWW98RnVXAjftuTniqK86DGZUQUcn7NP0TRwG8pRkSbjzwYbEjXPf/0YJEnj TO7I5d3Y53W4Nsg58rSaLF5PEs8AqjXmZO6qyTQN+v96A3Uwe81msJ/50qwXPTO4f2 o9elIV7ppsSS+UY7o+yfQfug+whBWyKl/vGhOLQitzhQMGih/BoZUr33t79K5Afbjb PHaBhHSNZk1NA== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 09/36] function_graph: Remove logic around ftrace_graph_entry and return Date: Wed, 7 Feb 2024 00:09:10 +0900 Message-Id: <170723215032.502590.2568906077275996700.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790162852286035240 X-GMAIL-MSGID: 1790162852286035240 From: Steven Rostedt (VMware) The function pointers ftrace_graph_entry and ftrace_graph_return are no longer called via the function_graph tracer. Instead, an array structure is now used that will allow for multiple users of the function_graph infrastructure. The variables are still used by the architecture code for non dynamic ftrace configs, where a test is made against them to see if they point to the default stub function or not. This is how the static function tracing knows to call into the function graph tracer infrastructure or not. Two new stub functions are made. entry_run() and return_run(). The ftrace_graph_entry and ftrace_graph_return are set to them respectively when the function graph tracer is enabled, and this will trigger the architecture specific function graph code to be executed. This also requires checking the global_ops hash for all calls into the function_graph tracer. Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- Changes in v2: - Fix typo and make lines shorter than 76 chars in the description. - Remove unneeded return from return_run() function. --- kernel/trace/fgraph.c | 67 +++++++++------------------------------- kernel/trace/ftrace.c | 2 - kernel/trace/ftrace_internal.h | 2 - 3 files changed, 15 insertions(+), 56 deletions(-) diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index b9a2399b75ee..6f3ba8e113c1 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -145,6 +145,17 @@ add_fgraph_index_bitmap(struct task_struct *t, int offset, unsigned long bitmap) t->ret_stack[offset] |= (bitmap << FGRAPH_INDEX_SHIFT); } +/* ftrace_graph_entry set to this to tell some archs to run function graph */ +static int entry_run(struct ftrace_graph_ent *trace) +{ + return 0; +} + +/* ftrace_graph_return set to this to tell some archs to run function graph */ +static void return_run(struct ftrace_graph_ret *trace) +{ +} + /* * @offset: The index into @t->ret_stack to find the ret_stack entry * @index: Where to place the index into @t->ret_stack of that entry @@ -675,7 +686,6 @@ extern void ftrace_stub_graph(struct ftrace_graph_ret *); /* The callbacks that hook a function */ trace_func_graph_ret_t ftrace_graph_return = ftrace_stub_graph; trace_func_graph_ent_t ftrace_graph_entry = ftrace_graph_entry_stub; -static trace_func_graph_ent_t __ftrace_graph_entry = ftrace_graph_entry_stub; /* Try to assign a return stack array on FTRACE_RETSTACK_ALLOC_SIZE tasks. */ static int alloc_retstack_tasklist(unsigned long **ret_stack_list) @@ -758,46 +768,6 @@ ftrace_graph_probe_sched_switch(void *ignore, bool preempt, } } -static int ftrace_graph_entry_test(struct ftrace_graph_ent *trace) -{ - if (!ftrace_ops_test(&global_ops, trace->func, NULL)) - return 0; - return __ftrace_graph_entry(trace); -} - -/* - * The function graph tracer should only trace the functions defined - * by set_ftrace_filter and set_ftrace_notrace. If another function - * tracer ops is registered, the graph tracer requires testing the - * function against the global ops, and not just trace any function - * that any ftrace_ops registered. - */ -void update_function_graph_func(void) -{ - struct ftrace_ops *op; - bool do_test = false; - - /* - * The graph and global ops share the same set of functions - * to test. If any other ops is on the list, then - * the graph tracing needs to test if its the function - * it should call. - */ - do_for_each_ftrace_op(op, ftrace_ops_list) { - if (op != &global_ops && op != &graph_ops && - op != &ftrace_list_end) { - do_test = true; - /* in double loop, break out with goto */ - goto out; - } - } while_for_each_ftrace_op(op); - out: - if (do_test) - ftrace_graph_entry = ftrace_graph_entry_test; - else - ftrace_graph_entry = __ftrace_graph_entry; -} - static DEFINE_PER_CPU(unsigned long *, idle_ret_stack); static void @@ -939,18 +909,12 @@ int register_ftrace_graph(struct fgraph_ops *gops) ftrace_graph_active--; goto out; } - - ftrace_graph_return = gops->retfunc; - /* - * Update the indirect function to the entryfunc, and the - * function that gets called to the entry_test first. Then - * call the update fgraph entry function to determine if - * the entryfunc should be called directly or not. + * Some archs just test to see if these are not + * the default function */ - __ftrace_graph_entry = gops->entryfunc; - ftrace_graph_entry = ftrace_graph_entry_test; - update_function_graph_func(); + ftrace_graph_return = return_run; + ftrace_graph_entry = entry_run; ret = ftrace_startup(&graph_ops, FTRACE_START_FUNC_RET); } @@ -986,7 +950,6 @@ void unregister_ftrace_graph(struct fgraph_ops *gops) if (!ftrace_graph_active) { ftrace_graph_return = ftrace_stub_graph; ftrace_graph_entry = ftrace_graph_entry_stub; - __ftrace_graph_entry = ftrace_graph_entry_stub; ftrace_shutdown(&graph_ops, FTRACE_STOP_FUNC_RET); unregister_pm_notifier(&ftrace_suspend_notifier); unregister_trace_sched_switch(ftrace_graph_probe_sched_switch, NULL); diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index c060d5b47910..11aac697d40f 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -235,8 +235,6 @@ static void update_ftrace_function(void) func = ftrace_ops_list_func; } - update_function_graph_func(); - /* If there's no change, then do nothing more here */ if (ftrace_trace_function == func) return; diff --git a/kernel/trace/ftrace_internal.h b/kernel/trace/ftrace_internal.h index 5012c04f92c0..19eddcb91584 100644 --- a/kernel/trace/ftrace_internal.h +++ b/kernel/trace/ftrace_internal.h @@ -42,10 +42,8 @@ ftrace_ops_test(struct ftrace_ops *ops, unsigned long ip, void *regs) #ifdef CONFIG_FUNCTION_GRAPH_TRACER extern int ftrace_graph_active; -void update_function_graph_func(void); #else /* !CONFIG_FUNCTION_GRAPH_TRACER */ # define ftrace_graph_active 0 -static inline void update_function_graph_func(void) { } #endif /* CONFIG_FUNCTION_GRAPH_TRACER */ #else /* !CONFIG_FUNCTION_TRACER */ From patchwork Tue Feb 6 15:09:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197513 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1621884dyb; Tue, 6 Feb 2024 07:34:33 -0800 (PST) X-Google-Smtp-Source: AGHT+IG6VwRsse1g3bzfySCsqIFSL764f2ZgGkw2FIPUSxy4HsvAW6pNPZjOQWw7xa8FwJz4txHS X-Received: by 2002:a05:6a20:9394:b0:19e:3c7f:3d2b with SMTP id x20-20020a056a20939400b0019e3c7f3d2bmr2393646pzh.9.1707233672959; Tue, 06 Feb 2024 07:34:32 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707233672; cv=pass; d=google.com; s=arc-20160816; b=NGXCS9bGPW/HqOeC+HcX+JBzSdeollgJStWx9LOUQs29Wz9mMm7wYbLnZ/UeupsMeu +10Nq4DA2ca5jpptAtJnuZtEaLb6Xj9du2ZO/8ypbPwonTTETqcoqmpKA/JidwD41dxi sA78onP6GAsgQbuzZo4GOHFUlS/rqOXLoGKzDqgcr+Jm8gXomfYuogzTT3iRswLoQ6+0 IwpnSQ6xuvt6/T4u9Ny/DJ0OsjXa1kHhpnlrj+UdiUWD353IrKQjXCGBWqTJTbfXDI/z Mxfo+hM73ufEiBE/6OJdnPqTmsqa7kRf/LuGMaarsRS04p/fhw6hGylc9Ax7c6nZ6Hmm VAxw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=2ITYv885o7xRq50/NsKzq+xM1MLpDzb+5IuIObgnOFc=; fh=ZXC9SPFc50mTHaBGp84U+81fOtM5Uyx1UuwCtkmcHOs=; b=locFpupzBHRIe4t4HE0wwTzLj6v+7M2/cQT53S2SHuTj5fWzsW2+RQ+Y2uptMKTwCx KEDEcFtE5SxbzG6iqZAOl4FgvwzCdI+s2RJYWIAfT0ommLUV9YrjwjHoxSaLEne8cnmV dwKiiNKr3rPlHlRd/KKNurgfZwsU4VWZh98KjKunnXteQvNXdJRdFrJE+MM83C9dhOKI YmROwwL79YXnJNmJYsJD5XOHtZfFmfsxPvE5kkb0VLS7e8K7Lgo7uVkXfWYYEk7wl7Gu jYnyeVqPelkP9KalV1WwFMCN9FdGRMfblzAQmtG+rc8cHPvPQOvMWy2akLpzyCB5hB2O eaDQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=qoNueHyO; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55187-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55187-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=1; AJvYcCVLe67GleddUXtp2m0DuSyds9LGiBi5ffoFJZsZ2sBLAO6Nsci1AN3fr8gC7HglbcXoN45T+BFLggRj/THM7rJrLTdM8g== Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [147.75.48.161]) by mx.google.com with ESMTPS id j15-20020a63cf0f000000b005d6fe985541si1801751pgg.293.2024.02.06.07.34.32 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:34:32 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55187-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=qoNueHyO; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55187-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55187-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 935EBB2FC60 for ; Tue, 6 Feb 2024 15:12:25 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 9F7D1135418; Tue, 6 Feb 2024 15:09:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="qoNueHyO" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BE6681353F1; Tue, 6 Feb 2024 15:09:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232167; cv=none; b=EtI61rwC0FJJihwV3tUMRzI78tcsKFJIVdfBeSLxqdoZO9HLiHnZhjfA/w7YFCLNuT5ZhthYirDik2U8g8TV/l2p5oxZcWMTw1jPWcJxybEiQU6B5xRA/nbAEPi+kkJ7rfYK5IvHX0pgpon9feQL+/lU6VTwtQaLyE1fa2IUv3g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232167; c=relaxed/simple; bh=mGpu0FYagab+fL3gEkACTbdo83UceSOodXCGxyuL0hk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=fYgaUk5pmECQYrlDj69T8MDDVkBkBsf/QiZV/5Dg9tsh08LTUgz49opAhbZ0GM4TxS/+sz2NOTpN7NijDtHylAQLND0//8llyq4STOyEJts1KBY0sVvaDWFHjeBBqBTIoA8KPZ/agHsiGwRwxW8wuklg1V0uKudgfkHFzq+BCLM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=qoNueHyO; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id B6B47C433C7; Tue, 6 Feb 2024 15:09:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232167; bh=mGpu0FYagab+fL3gEkACTbdo83UceSOodXCGxyuL0hk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qoNueHyO8HaxDicM0zq+FSyiMuk6bswbZak9hzDMn5tmQJMJmoBkMTdaG+M1T5xJW 4WbpmEeQebONRJr0hESf2HM2XFMI8hGSHsemuNDbg8LVrzZ6k5bTY6aSJpzddoS9DK 9gB+aiRthXfXLbE6i9rsieE1lqvHf+10yu8gz6/dkEwpvJ6d1eftCQKUgKhjt6HJO4 bHkT1wMTwMtCbXCw88HimqAqmWJ+JrFnq6IpqgWzfjX5d4VB7hTm/Pyk+cBwBqkUeU l6csstI0vpDV7+SRRfLmWkJRaJdnnQWni6A9VZ+w5bNykHyTd33tp2fhgqaPu3WFoe 9RmU7gNMWz0ug== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 10/36] ftrace/function_graph: Pass fgraph_ops to function graph callbacks Date: Wed, 7 Feb 2024 00:09:21 +0900 Message-Id: <170723216124.502590.13855631208872523552.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790164256031041648 X-GMAIL-MSGID: 1790164256031041648 From: Steven Rostedt (VMware) Pass the fgraph_ops structure to the function graph callbacks. This will allow callbacks to add a descriptor to a fgraph_ops private field that wil be added in the future and use it for the callbacks. This will be useful when more than one callback can be registered to the function graph tracer. Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- Changes in v2: - cleanup to set argument name on function prototype. --- include/linux/ftrace.h | 10 +++++++--- kernel/trace/fgraph.c | 14 ++++++++------ kernel/trace/ftrace.c | 6 ++++-- kernel/trace/trace.h | 4 ++-- kernel/trace/trace_functions_graph.c | 11 +++++++---- kernel/trace/trace_irqsoff.c | 6 ++++-- kernel/trace/trace_sched_wakeup.c | 6 ++++-- kernel/trace/trace_selftest.c | 5 +++-- 8 files changed, 39 insertions(+), 23 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index b39cc6d9b4a5..a71870183f1b 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -1055,11 +1055,15 @@ struct ftrace_graph_ret { unsigned long long rettime; } __packed; +struct fgraph_ops; + /* Type of the callback handlers for tracing function graph*/ -typedef void (*trace_func_graph_ret_t)(struct ftrace_graph_ret *); /* return */ -typedef int (*trace_func_graph_ent_t)(struct ftrace_graph_ent *); /* entry */ +typedef void (*trace_func_graph_ret_t)(struct ftrace_graph_ret *, + struct fgraph_ops *); /* return */ +typedef int (*trace_func_graph_ent_t)(struct ftrace_graph_ent *, + struct fgraph_ops *); /* entry */ -extern int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace); +extern int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace, struct fgraph_ops *gops); #ifdef CONFIG_FUNCTION_GRAPH_TRACER diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index 6f3ba8e113c1..e35a941a5af3 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -146,13 +146,13 @@ add_fgraph_index_bitmap(struct task_struct *t, int offset, unsigned long bitmap) } /* ftrace_graph_entry set to this to tell some archs to run function graph */ -static int entry_run(struct ftrace_graph_ent *trace) +static int entry_run(struct ftrace_graph_ent *trace, struct fgraph_ops *ops) { return 0; } /* ftrace_graph_return set to this to tell some archs to run function graph */ -static void return_run(struct ftrace_graph_ret *trace) +static void return_run(struct ftrace_graph_ret *trace, struct fgraph_ops *ops) { } @@ -213,12 +213,14 @@ int __weak ftrace_disable_ftrace_graph_caller(void) } #endif -int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace) +int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace, + struct fgraph_ops *gops) { return 0; } -static void ftrace_graph_ret_stub(struct ftrace_graph_ret *trace) +static void ftrace_graph_ret_stub(struct ftrace_graph_ret *trace, + struct fgraph_ops *gops) { } @@ -527,7 +529,7 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs if (gops == &fgraph_stub) continue; - gops->retfunc(&trace); + gops->retfunc(&trace, gops); } /* @@ -681,7 +683,7 @@ void ftrace_graph_sleep_time_control(bool enable) * Simply points to ftrace_stub, but with the proper protocol. * Defined by the linker script in linux/vmlinux.lds.h */ -extern void ftrace_stub_graph(struct ftrace_graph_ret *); +void ftrace_stub_graph(struct ftrace_graph_ret *trace, struct fgraph_ops *gops); /* The callbacks that hook a function */ trace_func_graph_ret_t ftrace_graph_return = ftrace_stub_graph; diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 11aac697d40f..b063ab2d2b1f 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -815,7 +815,8 @@ void ftrace_graph_graph_time_control(bool enable) fgraph_graph_time = enable; } -static int profile_graph_entry(struct ftrace_graph_ent *trace) +static int profile_graph_entry(struct ftrace_graph_ent *trace, + struct fgraph_ops *gops) { struct ftrace_ret_stack *ret_stack; @@ -832,7 +833,8 @@ static int profile_graph_entry(struct ftrace_graph_ent *trace) return 1; } -static void profile_graph_return(struct ftrace_graph_ret *trace) +static void profile_graph_return(struct ftrace_graph_ret *trace, + struct fgraph_ops *gops) { struct ftrace_ret_stack *ret_stack; struct ftrace_profile_stat *stat; diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h index 00f873910c5d..7d838a32bce5 100644 --- a/kernel/trace/trace.h +++ b/kernel/trace/trace.h @@ -678,8 +678,8 @@ void trace_latency_header(struct seq_file *m); void trace_default_header(struct seq_file *m); void print_trace_header(struct seq_file *m, struct trace_iterator *iter); -void trace_graph_return(struct ftrace_graph_ret *trace); -int trace_graph_entry(struct ftrace_graph_ent *trace); +void trace_graph_return(struct ftrace_graph_ret *trace, struct fgraph_ops *gops); +int trace_graph_entry(struct ftrace_graph_ent *trace, struct fgraph_ops *gops); void set_graph_array(struct trace_array *tr); void tracing_start_cmdline_record(void); diff --git a/kernel/trace/trace_functions_graph.c b/kernel/trace/trace_functions_graph.c index c35fbaab2a47..b7b142b65299 100644 --- a/kernel/trace/trace_functions_graph.c +++ b/kernel/trace/trace_functions_graph.c @@ -129,7 +129,8 @@ static inline int ftrace_graph_ignore_irqs(void) return in_hardirq(); } -int trace_graph_entry(struct ftrace_graph_ent *trace) +int trace_graph_entry(struct ftrace_graph_ent *trace, + struct fgraph_ops *gops) { struct trace_array *tr = graph_array; struct trace_array_cpu *data; @@ -238,7 +239,8 @@ void __trace_graph_return(struct trace_array *tr, trace_buffer_unlock_commit_nostack(buffer, event); } -void trace_graph_return(struct ftrace_graph_ret *trace) +void trace_graph_return(struct ftrace_graph_ret *trace, + struct fgraph_ops *gops) { struct trace_array *tr = graph_array; struct trace_array_cpu *data; @@ -275,7 +277,8 @@ void set_graph_array(struct trace_array *tr) smp_mb(); } -static void trace_graph_thresh_return(struct ftrace_graph_ret *trace) +static void trace_graph_thresh_return(struct ftrace_graph_ret *trace, + struct fgraph_ops *gops) { ftrace_graph_addr_finish(trace); @@ -288,7 +291,7 @@ static void trace_graph_thresh_return(struct ftrace_graph_ret *trace) (trace->rettime - trace->calltime < tracing_thresh)) return; else - trace_graph_return(trace); + trace_graph_return(trace, gops); } static struct fgraph_ops funcgraph_thresh_ops = { diff --git a/kernel/trace/trace_irqsoff.c b/kernel/trace/trace_irqsoff.c index ba37f768e2f2..5478f4c4f708 100644 --- a/kernel/trace/trace_irqsoff.c +++ b/kernel/trace/trace_irqsoff.c @@ -175,7 +175,8 @@ static int irqsoff_display_graph(struct trace_array *tr, int set) return start_irqsoff_tracer(irqsoff_trace, set); } -static int irqsoff_graph_entry(struct ftrace_graph_ent *trace) +static int irqsoff_graph_entry(struct ftrace_graph_ent *trace, + struct fgraph_ops *gops) { struct trace_array *tr = irqsoff_trace; struct trace_array_cpu *data; @@ -205,7 +206,8 @@ static int irqsoff_graph_entry(struct ftrace_graph_ent *trace) return ret; } -static void irqsoff_graph_return(struct ftrace_graph_ret *trace) +static void irqsoff_graph_return(struct ftrace_graph_ret *trace, + struct fgraph_ops *gops) { struct trace_array *tr = irqsoff_trace; struct trace_array_cpu *data; diff --git a/kernel/trace/trace_sched_wakeup.c b/kernel/trace/trace_sched_wakeup.c index 0469a04a355f..49bcc812652c 100644 --- a/kernel/trace/trace_sched_wakeup.c +++ b/kernel/trace/trace_sched_wakeup.c @@ -112,7 +112,8 @@ static int wakeup_display_graph(struct trace_array *tr, int set) return start_func_tracer(tr, set); } -static int wakeup_graph_entry(struct ftrace_graph_ent *trace) +static int wakeup_graph_entry(struct ftrace_graph_ent *trace, + struct fgraph_ops *gops) { struct trace_array *tr = wakeup_trace; struct trace_array_cpu *data; @@ -141,7 +142,8 @@ static int wakeup_graph_entry(struct ftrace_graph_ent *trace) return ret; } -static void wakeup_graph_return(struct ftrace_graph_ret *trace) +static void wakeup_graph_return(struct ftrace_graph_ret *trace, + struct fgraph_ops *gops) { struct trace_array *tr = wakeup_trace; struct trace_array_cpu *data; diff --git a/kernel/trace/trace_selftest.c b/kernel/trace/trace_selftest.c index 529590499b1f..914331d8242c 100644 --- a/kernel/trace/trace_selftest.c +++ b/kernel/trace/trace_selftest.c @@ -762,7 +762,8 @@ trace_selftest_startup_function(struct tracer *trace, struct trace_array *tr) static unsigned int graph_hang_thresh; /* Wrap the real function entry probe to avoid possible hanging */ -static int trace_graph_entry_watchdog(struct ftrace_graph_ent *trace) +static int trace_graph_entry_watchdog(struct ftrace_graph_ent *trace, + struct fgraph_ops *gops) { /* This is harmlessly racy, we want to approximately detect a hang */ if (unlikely(++graph_hang_thresh > GRAPH_MAX_FUNC_TEST)) { @@ -776,7 +777,7 @@ static int trace_graph_entry_watchdog(struct ftrace_graph_ent *trace) return 0; } - return trace_graph_entry(trace); + return trace_graph_entry(trace, gops); } static struct fgraph_ops fgraph_ops __initdata = { From patchwork Tue Feb 6 15:09:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197489 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1606511dyb; Tue, 6 Feb 2024 07:12:45 -0800 (PST) X-Google-Smtp-Source: AGHT+IGd6H7CI3hliBvnfUNiymfvLMyNmagu5wqsOdpOaPFL/ZkQmCpUX0x1HX0+Ubeh8JPqTn3S X-Received: by 2002:a05:6214:2621:b0:68c:5cd7:a7b4 with SMTP id gv1-20020a056214262100b0068c5cd7a7b4mr3026265qvb.14.1707232365284; Tue, 06 Feb 2024 07:12:45 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707232365; cv=pass; d=google.com; s=arc-20160816; b=pmRa9Pd9vkF2bS62kXbbV739w3ZpOZg9MZ035d6yU1E+4UO/Ia0Zy4YXnbMv2pvE8S 9fn8R/GNm+yMWhJt7xDHiSRh9/vUpPQtMfFeLKo403VHtc/S0c/2F+BUKXElgjk7Nf3l irpkBR+Vj7ByR4b8VNvKHNZqyOjYKt7hPbKXoNLRVM4B0bKliBwt47xnqYXDFAzHButf Te/PQ4KRprvYKCJKSUYo6fKtq4ASKei6eYQ3fjxc2HgLkI55t8VwVc/rwn+/+zM7jMVn 1LI96c5Arn98uw+Vjp3inRGfJhNA1iFQnx/siNHEl3n5R+vGKiVt2nUmJrt0wL1lsP5/ kJ0A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=DuJgSyfIvQRfH00f3d6fi4wsYab7MJ8xi7KLzknRnso=; fh=hDM6vSVaiXlLwtIAQSXIG/bfTYVSSfEXNO/ynjsU+iY=; b=EATib7WCldnSFC4r+CTjwFOvoBELLLUSAS6i/3if35INKtSwLkL5DLsF5WIOjkcb/p 09OR+5dQF0Km60+Aud6K1RPz5HT2BHXB/qY5hyCh3KVc4i2KJ36+NHa9OL7HrSiyruSw 1gD8W0X8LOOEz4RUs2wTSPZVOb5FlnBNpGYdMamR2DD9w3gxj7J5FRcAujdtWGBXu7aZ abX18ppQ9FDlt6ienlhGhW8c1k/Uc9JuY5xcuG7VZA6Cyu+sqY86bBgCRe5SQTE+DOeb xn4kay59s6Um16z8cUqM+VxZ85Ax0zdgptX/oa8mSi+uBskpHNiLYU6+hdJ4aoKIukXj r7yQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Is1leery; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55188-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55188-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=1; AJvYcCVC5klYn1+8UzJW1U+hzoZWH++bJUR8N9B5YeAAP+Uth6ww3V7pliCZRnGP1zcZHgaEVIdm+8Uvz4ZdvM10kCFB6WS52A== Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id g6-20020a0562140ac600b0068cb4beb93esi1369401qvi.137.2024.02.06.07.12.45 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:12:45 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55188-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Is1leery; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55188-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55188-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id DB94A1C24160 for ; Tue, 6 Feb 2024 15:12:44 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 53A61135A4F; Tue, 6 Feb 2024 15:09:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Is1leery" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9513A38D; Tue, 6 Feb 2024 15:09:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232178; cv=none; b=LkbakcQcxH5WbtRZjNIEDTH/1DQnbaN/sCIte08HWYm+3WfYivJp+rFFWGM1n1fBezAtYihdfW6MiQ1AH3L7MgC02cMwaGVfa9tibWm0kOYh2vL96bpXrbgVsKUb5wODslXVy89WiFOSiUUf1JLwwJlxJzeADuoXN9oT7SbNM14= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232178; c=relaxed/simple; bh=KRFwJgemBKBAZna1O62GVWExZPY7dLVRXMQTuT5NVSk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=WaG5iVt121Q752LSUuQUXCnkrsyYKRtXQyv2lCrakLQes6KBReWFsHnCOepWrYM1ve/B3DvDcdG76HVGa+8JvE+kZBW5AgCCovsVn8Qz5QedSO9f4mDql1y30sNDQ2RgAsk9MLGRaxKTRc1Sxdd2KJuTYMwY2H04iWnfv79kDK4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Is1leery; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 11A2DC433C7; Tue, 6 Feb 2024 15:09:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232178; bh=KRFwJgemBKBAZna1O62GVWExZPY7dLVRXMQTuT5NVSk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Is1leery1x42ytg7jK1gRwurrtpUqRu4bzxlm7wGXMmk4Kgt7UMfJIpIcU1oE9k8K NdVW58xU7yKGyvOd/BdOHTjTL23v4SnBICBNreNOXXFJQAQYMc2XhkXPbXtXIeMXqK FADB9bV+FHyNtKM8BX035eSOWSUNRsqf1MvqYw1K3HpCjpGCPsQdAhhoJsyB2hcSiN AIOT/DfqKAUDSsWmQpMhrjjDiV7W8Ep3fgReKtJm0lkARjDvIdIu//gpIoNo80qPC/ +O5sgrlRZ1oIEJ1s2I66lhMgvU1wRSVNDun3r9GEjkwmOvBYFZKYw09lkAB0gkQdW+ 1Lxtf8K7/MGDQ== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 11/36] ftrace: Allow function_graph tracer to be enabled in instances Date: Wed, 7 Feb 2024 00:09:32 +0900 Message-Id: <170723217268.502590.12074936644663843373.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790162884918536960 X-GMAIL-MSGID: 1790162884918536960 From: Steven Rostedt (VMware) Now that function graph tracing can handle more than one user, allow it to be enabled in the ftrace instances. Note, the filtering of the functions is still joined by the top level set_ftrace_filter and friends, as well as the graph and nograph files. Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- Changes in v2: - Fix to remove set_graph_array() completely. --- include/linux/ftrace.h | 1 + kernel/trace/ftrace.c | 1 + kernel/trace/trace.h | 13 ++++++- kernel/trace/trace_functions.c | 8 ++++ kernel/trace/trace_functions_graph.c | 65 +++++++++++++++++++++------------- kernel/trace/trace_selftest.c | 4 +- 6 files changed, 64 insertions(+), 28 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index a71870183f1b..17aa123d134e 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -1070,6 +1070,7 @@ extern int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace, struct fgraph struct fgraph_ops { trace_func_graph_ent_t entryfunc; trace_func_graph_ret_t retfunc; + void *private; int idx; }; diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index b063ab2d2b1f..a720dd7cf290 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -7323,6 +7323,7 @@ __init void ftrace_init_global_array_ops(struct trace_array *tr) tr->ops = &global_ops; tr->ops->private = tr; ftrace_init_trace_array(tr); + init_array_fgraph_ops(tr); } void ftrace_init_array_ops(struct trace_array *tr, ftrace_func_t func) diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h index 7d838a32bce5..3b39936c9375 100644 --- a/kernel/trace/trace.h +++ b/kernel/trace/trace.h @@ -396,6 +396,9 @@ struct trace_array { struct ftrace_ops *ops; struct trace_pid_list __rcu *function_pids; struct trace_pid_list __rcu *function_no_pids; +#ifdef CONFIG_FUNCTION_GRAPH_TRACER + struct fgraph_ops *gops; +#endif #ifdef CONFIG_DYNAMIC_FTRACE /* All of these are protected by the ftrace_lock */ struct list_head func_probes; @@ -680,7 +683,6 @@ void print_trace_header(struct seq_file *m, struct trace_iterator *iter); void trace_graph_return(struct ftrace_graph_ret *trace, struct fgraph_ops *gops); int trace_graph_entry(struct ftrace_graph_ent *trace, struct fgraph_ops *gops); -void set_graph_array(struct trace_array *tr); void tracing_start_cmdline_record(void); void tracing_stop_cmdline_record(void); @@ -891,6 +893,9 @@ extern int __trace_graph_entry(struct trace_array *tr, extern void __trace_graph_return(struct trace_array *tr, struct ftrace_graph_ret *trace, unsigned int trace_ctx); +extern void init_array_fgraph_ops(struct trace_array *tr); +extern int allocate_fgraph_ops(struct trace_array *tr); +extern void free_fgraph_ops(struct trace_array *tr); #ifdef CONFIG_DYNAMIC_FTRACE extern struct ftrace_hash __rcu *ftrace_graph_hash; @@ -1003,6 +1008,12 @@ print_graph_function_flags(struct trace_iterator *iter, u32 flags) { return TRACE_TYPE_UNHANDLED; } +static inline void init_array_fgraph_ops(struct trace_array *tr) { } +static inline int allocate_fgraph_ops(struct trace_array *tr) +{ + return 0; +} +static inline void free_fgraph_ops(struct trace_array *tr) { } #endif /* CONFIG_FUNCTION_GRAPH_TRACER */ extern struct list_head ftrace_pids; diff --git a/kernel/trace/trace_functions.c b/kernel/trace/trace_functions.c index 9f1bfbe105e8..8e8da0d0ee52 100644 --- a/kernel/trace/trace_functions.c +++ b/kernel/trace/trace_functions.c @@ -80,6 +80,7 @@ void ftrace_free_ftrace_ops(struct trace_array *tr) int ftrace_create_function_files(struct trace_array *tr, struct dentry *parent) { + int ret; /* * The top level array uses the "global_ops", and the files are * created on boot up. @@ -90,6 +91,12 @@ int ftrace_create_function_files(struct trace_array *tr, if (!tr->ops) return -EINVAL; + ret = allocate_fgraph_ops(tr); + if (ret) { + kfree(tr->ops); + return ret; + } + ftrace_create_filter_files(tr->ops, parent); return 0; @@ -99,6 +106,7 @@ void ftrace_destroy_function_files(struct trace_array *tr) { ftrace_destroy_filter_files(tr->ops); ftrace_free_ftrace_ops(tr); + free_fgraph_ops(tr); } static ftrace_func_t select_trace_function(u32 flags_val) diff --git a/kernel/trace/trace_functions_graph.c b/kernel/trace/trace_functions_graph.c index b7b142b65299..9ccc904a7703 100644 --- a/kernel/trace/trace_functions_graph.c +++ b/kernel/trace/trace_functions_graph.c @@ -83,8 +83,6 @@ static struct tracer_flags tracer_flags = { .opts = trace_opts }; -static struct trace_array *graph_array; - /* * DURATION column is being also used to display IRQ signs, * following values are used by print_graph_irq and others @@ -132,7 +130,7 @@ static inline int ftrace_graph_ignore_irqs(void) int trace_graph_entry(struct ftrace_graph_ent *trace, struct fgraph_ops *gops) { - struct trace_array *tr = graph_array; + struct trace_array *tr = gops->private; struct trace_array_cpu *data; unsigned long flags; unsigned int trace_ctx; @@ -242,7 +240,7 @@ void __trace_graph_return(struct trace_array *tr, void trace_graph_return(struct ftrace_graph_ret *trace, struct fgraph_ops *gops) { - struct trace_array *tr = graph_array; + struct trace_array *tr = gops->private; struct trace_array_cpu *data; unsigned long flags; unsigned int trace_ctx; @@ -268,15 +266,6 @@ void trace_graph_return(struct ftrace_graph_ret *trace, local_irq_restore(flags); } -void set_graph_array(struct trace_array *tr) -{ - graph_array = tr; - - /* Make graph_array visible before we start tracing */ - - smp_mb(); -} - static void trace_graph_thresh_return(struct ftrace_graph_ret *trace, struct fgraph_ops *gops) { @@ -294,25 +283,53 @@ static void trace_graph_thresh_return(struct ftrace_graph_ret *trace, trace_graph_return(trace, gops); } -static struct fgraph_ops funcgraph_thresh_ops = { - .entryfunc = &trace_graph_entry, - .retfunc = &trace_graph_thresh_return, -}; - static struct fgraph_ops funcgraph_ops = { .entryfunc = &trace_graph_entry, .retfunc = &trace_graph_return, }; +int allocate_fgraph_ops(struct trace_array *tr) +{ + struct fgraph_ops *gops; + + gops = kzalloc(sizeof(*gops), GFP_KERNEL); + if (!gops) + return -ENOMEM; + + gops->entryfunc = &trace_graph_entry; + gops->retfunc = &trace_graph_return; + + tr->gops = gops; + gops->private = tr; + return 0; +} + +void free_fgraph_ops(struct trace_array *tr) +{ + kfree(tr->gops); +} + +__init void init_array_fgraph_ops(struct trace_array *tr) +{ + tr->gops = &funcgraph_ops; + funcgraph_ops.private = tr; +} + static int graph_trace_init(struct trace_array *tr) { int ret; - set_graph_array(tr); + tr->gops->entryfunc = trace_graph_entry; + if (tracing_thresh) - ret = register_ftrace_graph(&funcgraph_thresh_ops); + tr->gops->retfunc = trace_graph_thresh_return; else - ret = register_ftrace_graph(&funcgraph_ops); + tr->gops->retfunc = trace_graph_return; + + /* Make gops functions are visible before we start tracing */ + smp_mb(); + + ret = register_ftrace_graph(tr->gops); if (ret) return ret; tracing_start_cmdline_record(); @@ -323,10 +340,7 @@ static int graph_trace_init(struct trace_array *tr) static void graph_trace_reset(struct trace_array *tr) { tracing_stop_cmdline_record(); - if (tracing_thresh) - unregister_ftrace_graph(&funcgraph_thresh_ops); - else - unregister_ftrace_graph(&funcgraph_ops); + unregister_ftrace_graph(tr->gops); } static int graph_trace_update_thresh(struct trace_array *tr) @@ -1365,6 +1379,7 @@ static struct tracer graph_trace __tracer_data = { .print_header = print_graph_headers, .flags = &tracer_flags, .set_flag = func_graph_set_flag, + .allow_instances = true, #ifdef CONFIG_FTRACE_SELFTEST .selftest = trace_selftest_startup_function_graph, #endif diff --git a/kernel/trace/trace_selftest.c b/kernel/trace/trace_selftest.c index 914331d8242c..f0758afa2f7d 100644 --- a/kernel/trace/trace_selftest.c +++ b/kernel/trace/trace_selftest.c @@ -813,7 +813,7 @@ trace_selftest_startup_function_graph(struct tracer *trace, * to detect and recover from possible hangs */ tracing_reset_online_cpus(&tr->array_buffer); - set_graph_array(tr); + fgraph_ops.private = tr; ret = register_ftrace_graph(&fgraph_ops); if (ret) { warn_failed_init_tracer(trace, ret); @@ -856,7 +856,7 @@ trace_selftest_startup_function_graph(struct tracer *trace, cond_resched(); tracing_reset_online_cpus(&tr->array_buffer); - set_graph_array(tr); + fgraph_ops.private = tr; /* * Some archs *cough*PowerPC*cough* add characters to the From patchwork Tue Feb 6 15:09:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197491 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1607002dyb; Tue, 6 Feb 2024 07:13:20 -0800 (PST) X-Google-Smtp-Source: AGHT+IFcfJgWlxH+kR5wwwTYvhLVPqhZT/bfUjtYXuSYt02ZfJaKi3K5LWlbsZbezJrYHOr3OT+f X-Received: by 2002:a17:902:f68e:b0:1d9:63ac:f54a with SMTP id l14-20020a170902f68e00b001d963acf54amr2398315plg.36.1707232399892; Tue, 06 Feb 2024 07:13:19 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707232399; cv=pass; d=google.com; s=arc-20160816; b=Fl4N8Ywnpsctbs5Z5wjM/YjXW4LC+KvmDzG0goCxANewpZmcmOMNNg8LlrJDcKSpLK XEqWaIoZ3v1ZH1UnsOg7FR0qwEpWEtONyaqUB+S9zMZDzAHd/Uz4xt1gZPsEvZu1KEeQ rSOUqgbR70QY6Rb5gVbzGqJHVLBd3ooNjrB9E6qScVSPcnLOiFPK8yopdKZY5U9dS2mk 5O+UB/AnwRqKXsRtSWmOZwL+8WfHDL+3OKtOQvSBjTko8zW39EQKya/XuSpoHpShsTer SMJ1X+FqYsur5RTDExBUFvL12Jf0+/fMCBfbjuI2dbNXZGKtYSgyfrvj+k3YZSgS1dyD 1H1g== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=rZyrIN0YHDigS9kpRkpOdAZzAkkfxVufaPnk5YTtw+A=; fh=gvvLH1P1dWPpFk1pXpEhQ8v88Vbnc5XIhLAG6Q4Q7fo=; b=GV8Ww8t5cJczeZPVsbl+TUHNZZySp0rBbQH502YCLBmDYQDdxeX9mQONIN//1ol7CK 3sph7nJHSguR6ni5FqwvpeVej7yfttrkFm//b4SA9AmluU84xGT0JMKxHAChrbDMMJmK shIW4Jj26KhhsyXdfI3eIoRwViHzAbTXTZY6Eo9uywYMlP1hMPtnUD631WrA4yvUge8y Dwxi9HTmPdcj3frWDwRDQoB3KlmUJKQ2eFAsw5fk2LsZEWyRxTJGa/BDD94fTfylA8EQ uvrmRoj5hXYcLKe/FUEJiXSztMdvyESOgBQDF5hZf3J1wcXoyTstN7IP3XN7774IF197 Tjrw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=EiuE64g8; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55190-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55190-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=1; AJvYcCUn8W8CMMK1kftmUozIRCnil/Z+FHO+inT2aM3VaCf0iM8QKh8/jMkKcBFI7yi9vnDfk79zbAU3E+ihmvH8Tg24C+xGPw== Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id t3-20020a170902e84300b001d556872df5si1863675plg.335.2024.02.06.07.13.19 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:13:19 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55190-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=EiuE64g8; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55190-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55190-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 7F22B284421 for ; Tue, 6 Feb 2024 15:13:19 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id DBB75136643; Tue, 6 Feb 2024 15:09:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="EiuE64g8" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 37687133299; Tue, 6 Feb 2024 15:09:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232189; cv=none; b=aI8fAYxuIkfiQCX/UFquFcJx3jB7VPUB634qThwTTuEnfXYd4ZNQOYKElwd4yV2vxk5iAqV3LHLku6iFBNWKohgm9upBqfap+oEIjrE7iYRU9nx83kQeKamyxtwEO4VOWcrTNCupF5vjc5mVheSFqkTrFlbw6vDYXQVueCHs+Xg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232189; c=relaxed/simple; bh=Lr0pIkw9vuGNLQkbuw0r2byq1uReNXl+ksUhuI+VOdA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=leJKDoFAvc91uF6PRqU0HkDVyVSyf3qk6zdK0NV6jkwtci1HBLruTXVWzPlw8K6KlPAcWKCKYoLLAvfehUxi12pjr1OPbocMoFR9rMDLKFoqCY11Rf4kJG0WKVMMZuzBtg0O6LLsCEZQ/jrJt6nRWq9SQ7GUiJ/YUr3FTAc+TJc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=EiuE64g8; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 96D83C433F1; Tue, 6 Feb 2024 15:09:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232189; bh=Lr0pIkw9vuGNLQkbuw0r2byq1uReNXl+ksUhuI+VOdA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=EiuE64g88WU790pDeY11J1U7S4TOQd1rwE+YPpEWX8RjqhIzS7v7bRdqNJ7hJsq4T f5b3GV/+QJ0ZpRWkaosolCkzj4Rmqut2mlfRJ9JC/pm7fFBe9f1AK8U06rLfmoa4yN TRWobJRY+tmleWsWpnvf0eWbEocMUmQPBqlWla99Q52STLCwizkN54uvW11uabj4Ve SEDFVp6wZpYEkcsK1cW22Kiq3G1bTgE44lUyyKHmXJH7e2eA/1H3RyUQACV2m3W4AA 0EdS2qyVRzMByBnRWf0OiWJDM3mmBrwgCR+XC12JH8O9ccT1x8TE95D2SoFzXPQsny c3J+FkNseL02Q== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 12/36] ftrace: Allow ftrace startup flags exist without dynamic ftrace Date: Wed, 7 Feb 2024 00:09:43 +0900 Message-Id: <170723218353.502590.15434076519096691266.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790162921152018522 X-GMAIL-MSGID: 1790162921152018522 From: Steven Rostedt (VMware) Some of the flags for ftrace_startup() may be exposed even when CONFIG_DYNAMIC_FTRACE is not configured in. This is fine as the difference between dynamic ftrace and static ftrace is done within the internals of ftrace itself. No need to have use cases fail to compile because dynamic ftrace is disabled. This change is needed to move some of the logic of what is passed to ftrace_startup() out of the parameters of ftrace_startup(). Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- include/linux/ftrace.h | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 17aa123d134e..b87f9676f5ce 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -538,6 +538,15 @@ static inline void stack_tracer_disable(void) { } static inline void stack_tracer_enable(void) { } #endif +enum { + FTRACE_UPDATE_CALLS = (1 << 0), + FTRACE_DISABLE_CALLS = (1 << 1), + FTRACE_UPDATE_TRACE_FUNC = (1 << 2), + FTRACE_START_FUNC_RET = (1 << 3), + FTRACE_STOP_FUNC_RET = (1 << 4), + FTRACE_MAY_SLEEP = (1 << 5), +}; + #ifdef CONFIG_DYNAMIC_FTRACE void ftrace_arch_code_modify_prepare(void); @@ -632,15 +641,6 @@ void ftrace_set_global_notrace(unsigned char *buf, int len, int reset); void ftrace_free_filter(struct ftrace_ops *ops); void ftrace_ops_set_global_filter(struct ftrace_ops *ops); -enum { - FTRACE_UPDATE_CALLS = (1 << 0), - FTRACE_DISABLE_CALLS = (1 << 1), - FTRACE_UPDATE_TRACE_FUNC = (1 << 2), - FTRACE_START_FUNC_RET = (1 << 3), - FTRACE_STOP_FUNC_RET = (1 << 4), - FTRACE_MAY_SLEEP = (1 << 5), -}; - /* * The FTRACE_UPDATE_* enum is used to pass information back * from the ftrace_update_record() and ftrace_test_record() From patchwork Tue Feb 6 15:09:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197520 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1624819dyb; Tue, 6 Feb 2024 07:39:08 -0800 (PST) X-Google-Smtp-Source: AGHT+IHZ11pPyTjeIVjaKhlAbKYm5zB3v2u2LmX0qrb0n+3RrHCLfGjkS5NhjomFqFG/FsS6Gybg X-Received: by 2002:a05:6358:706:b0:175:f406:a0b7 with SMTP id e6-20020a056358070600b00175f406a0b7mr3540984rwj.16.1707233947736; Tue, 06 Feb 2024 07:39:07 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707233947; cv=pass; d=google.com; s=arc-20160816; b=U/4q2eAQhiSOy1dYcUoRic/2bq74qJykaXZ3SJqYeo5HSXovLlTgF2V1CJaHrpz+aq lY1aWD3MISmBBpRF5fYTlrvpxvgu6G5acxPdLjiMpZ1OtGeVT1/uP6ES9ce9qkeZuo7H /u2lS/JhkvqoM9cgnuRUj/fcV9q3wnByondOM/yPC1v+6zuRP1YcV3nh5tcvgaN/grAJ 4h6es+Q6iRsl08skDOzLo91pQX1m/iYggkwWjCbeQ0mwfaBhVHWBTtzqonDNlmuvv11B Tz4mza6Up8jhlLqXdn/3wJzaiUaPHe+7Rm6jBt9hNdVT6uCi45sfOnFurI9r8An2p+pB TRfw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=88TKoBf0O5PYhUZ/m/aEgqkXR6DFh71BtVDNhzrmz4U=; fh=XFs7hmhsFibisDxuDboUlX1IQYI0S2xZWT7c7yPcXps=; b=YZWcqHnDV3B11I0X59UPo3RJ1EXVO50qaBFCzP5Hc41Lr37npWcM2ld3ccPTmWYzuJ HLEluuIhLW7IsdYhyKU+11aHI0sU8Sy5FggVBilaJ/gOyPK8O2vU3nw29yxlVnZZwm0P y+oQhpJXowvH9h0alYDhG2IckABy8g7kwDD5xOIDiB9bG15gI3H7bGrXoSggm3Pd4B2w J6qT/yPJnm/JWdxZ31VefM4kixNtEgmzSqkrpwZppnCNpJOhVTubJw7lmP8WBAv1VDgx 3vZ3P+eVFT55BlZzCoh178QN8mrcOO/loFOFbzVRat0srpI8VE1RMAQA/Fk7dFkeYX2y rphw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Vbf6bGhh; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55191-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55191-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=1; AJvYcCVNBwLQxpnhUyZcbmIR1IVfG4behBKZJeYZB6DPeuFd1XihKuBuQ20M0pmff48QZD/1/aYwyw1pdzAIS4kXoze0IEiYFg== Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [147.75.48.161]) by mx.google.com with ESMTPS id y12-20020a63e24c000000b005c1b30b1b65si1854389pgj.648.2024.02.06.07.39.07 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:39:07 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55191-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Vbf6bGhh; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55191-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55191-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 36621B2D952 for ; Tue, 6 Feb 2024 15:13:40 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id BC64613664F; Tue, 6 Feb 2024 15:10:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Vbf6bGhh" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 957E4133299; Tue, 6 Feb 2024 15:09:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232199; cv=none; b=YZKCbGypjJqy/hg7Ch0kpFkQ7bECHXQgWEAHciM0rbygGgMnd8sUTc6sonrZFWWU0wajGu1lpiZySI04NaT4wsdTA1b91pfyMTJTjRd/ixUGCvCz0fepqh9E9USIjaZwFDoJW7yBs389p7mTalgbvT/SsdHzLfOue11FTqrbCWg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232199; c=relaxed/simple; bh=rZAefjW1mTUGuRujEC9+cIcOsDMlm3gFzcjy+WktVMQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=apFH0KixRGmlgGv6qf4Z5Rb1z4qpGHGO+hWkDHb/ptMvd8qUiAsuSn8xa5wf1YsXrOA0pF+CaE466Kvdgc1/kwZM9r5bPcsgF1naxINImEgSqk1AZfTQsrY6vMLC9qSBtho/Cfpy08unEbZ4uvxj6dOUI/hc+k+TKxr9IxwuD+k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Vbf6bGhh; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 83A7FC433C7; Tue, 6 Feb 2024 15:09:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232199; bh=rZAefjW1mTUGuRujEC9+cIcOsDMlm3gFzcjy+WktVMQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Vbf6bGhhy7IXCmrPdNECwX0G41hD3YMNk3jqd/4lIaYJ/98PqaLOA2ePRWR1yOr2b y4fWKtBtFd3rKdVMG1q91p6L8k5aFtfj1W9QpTK/cxqMMivtVdWOoVmtik3c5myahj 1MI1j21R08oFPR340NsqfX4qObVYJJBYclsTrFWgpDcmaYFH20zbB+EQ5e5ZHngGnN W8cKkaT3AHz5085YhpJqvnGahTn7rRhaAQpcPttoRc+PMqoBcFCF6PMgU0+BnM/54X HznAmFnM2UsIR0zMG9fXRJePgdLwlpukhmiPa+8IvjWG3MROR/jgCvq+JD2aoBxklS rnJxSKILP7rNg== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 13/36] function_graph: Have the instances use their own ftrace_ops for filtering Date: Wed, 7 Feb 2024 00:09:54 +0900 Message-Id: <170723219437.502590.17981699514070908579.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790164544296981847 X-GMAIL-MSGID: 1790164544296981847 From: Steven Rostedt (VMware) Allow for instances to have their own ftrace_ops part of the fgraph_ops that makes the funtion_graph tracer filter on the set_ftrace_filter file of the instance and not the top instance. Note that this also requires to update ftrace_graph_func() to call new function_graph_enter_ops() instead of function_graph_enter() so that it avoid pushing on shadow stack multiple times on the same function. Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- Changes in v7: - Move FGRAPH_TYPE_BITMAP type implementation to earlier patch ( which implements FGRAPH_TYPE_ARRAY) so that it does not need to replace the FGRAPH_TYPE_ARRAY type. - Update loongarch and powerpc implementation of ftrace_graph_func(). - Update description. Changes in v6: - Fix to check whether the fgraph_ops is already unregistered in function_graph_enter_ops(). - Fix stack unwinder error on arm64 because of passing wrong value as retp. Thanks Mark! Changes in v4: - Simplify get_ret_stack() sanity check and use WARN_ON_ONCE() for obviously wrong value. - Do not check ret == return_to_handler but always read the previous ret_stack in ftrace_push_return_trace() to check it is reusable. - Set the bit 0 of the bitmap entry always in function_graph_enter() because it uses bit 0 to check re-usability. - Fix to ensure the ret_stack entry is bitmap type when checking the bitmap. Changes in v3: - Pass current fgraph_ops to the new entry handler (function_graph_enter_ops) if fgraph use ftrace. - Add fgraph_ops::idx in this patch. - Replace the array type with the bitmap type so that it can record which fgraph is called. - Fix some helper function to use passed task_struct instead of current. - Reduce the ret-index size to 1024 words. - Make the ret-index directly points the ret_stack. - Fix ftrace_graph_ret_addr() to handle tail-call case correctly. Changes in v2: - Use ftrace_graph_func and FTRACE_OPS_GRAPH_STUB instead of ftrace_stub and FTRACE_OPS_FL_STUB for new ftrace based fgraph. --- arch/arm64/kernel/ftrace.c | 21 ++++++ arch/loongarch/kernel/ftrace_dyn.c | 9 ++- arch/powerpc/kernel/trace/ftrace.c | 3 + arch/x86/kernel/ftrace.c | 19 +++++- include/linux/ftrace.h | 6 ++ kernel/trace/fgraph.c | 114 +++++++++++++++++++++++++--------- kernel/trace/ftrace.c | 4 + kernel/trace/trace.h | 16 +++-- kernel/trace/trace_functions.c | 2 - kernel/trace/trace_functions_graph.c | 8 ++ 10 files changed, 157 insertions(+), 45 deletions(-) diff --git a/arch/arm64/kernel/ftrace.c b/arch/arm64/kernel/ftrace.c index a650f5e11fc5..b96740829798 100644 --- a/arch/arm64/kernel/ftrace.c +++ b/arch/arm64/kernel/ftrace.c @@ -481,7 +481,26 @@ void prepare_ftrace_return(unsigned long self_addr, unsigned long *parent, void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, struct ftrace_ops *op, struct ftrace_regs *fregs) { - prepare_ftrace_return(ip, &fregs->lr, fregs->fp); + struct fgraph_ops *gops = container_of(op, struct fgraph_ops, ops); + unsigned long frame_pointer = fregs->fp; + unsigned long *parent = &fregs->lr; + int bit; + + if (unlikely(ftrace_graph_is_dead())) + return; + + if (unlikely(atomic_read(¤t->tracing_graph_pause))) + return; + + bit = ftrace_test_recursion_trylock(ip, *parent); + if (bit < 0) + return; + + if (!function_graph_enter_ops(*parent, ip, frame_pointer, + (void *)frame_pointer, gops)) + *parent = (unsigned long)&return_to_handler; + + ftrace_test_recursion_unlock(bit); } #else /* diff --git a/arch/loongarch/kernel/ftrace_dyn.c b/arch/loongarch/kernel/ftrace_dyn.c index 73858c9029cc..81d18b911cc1 100644 --- a/arch/loongarch/kernel/ftrace_dyn.c +++ b/arch/loongarch/kernel/ftrace_dyn.c @@ -241,10 +241,17 @@ void prepare_ftrace_return(unsigned long self_addr, unsigned long *parent) void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, struct ftrace_ops *op, struct ftrace_regs *fregs) { + struct fgraph_ops *gops = container_of(op, struct fgraph_ops, ops); struct pt_regs *regs = &fregs->regs; unsigned long *parent = (unsigned long *)®s->regs[1]; - prepare_ftrace_return(ip, (unsigned long *)parent); + if (unlikely(atomic_read(¤t->tracing_graph_pause))) + return; + + old = *parent; + + if (!function_graph_enter_ops(old, ip, 0, parent, gops)) + *parent = return_hooker; } #else static int ftrace_modify_graph_caller(bool enable) diff --git a/arch/powerpc/kernel/trace/ftrace.c b/arch/powerpc/kernel/trace/ftrace.c index 82010629cf88..4ef8bf480279 100644 --- a/arch/powerpc/kernel/trace/ftrace.c +++ b/arch/powerpc/kernel/trace/ftrace.c @@ -409,6 +409,7 @@ int __init ftrace_dyn_arch_init(void) void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, struct ftrace_ops *op, struct ftrace_regs *fregs) { + struct fgraph_ops *gops = container_of(op, struct fgraph_ops, ops); unsigned long sp = fregs->regs.gpr[1]; int bit; @@ -422,7 +423,7 @@ void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, if (bit < 0) goto out; - if (!function_graph_enter(parent_ip, ip, 0, (unsigned long *)sp)) + if (!function_graph_enter_ops(parent_ip, ip, 0, (unsigned long *)sp, gops)) parent_ip = ppc_function_entry(return_to_handler); ftrace_test_recursion_unlock(bit); diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c index 12df54ff0e81..845e29b4254f 100644 --- a/arch/x86/kernel/ftrace.c +++ b/arch/x86/kernel/ftrace.c @@ -657,9 +657,24 @@ void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, struct ftrace_ops *op, struct ftrace_regs *fregs) { struct pt_regs *regs = &fregs->regs; - unsigned long *stack = (unsigned long *)kernel_stack_pointer(regs); + unsigned long *parent = (unsigned long *)kernel_stack_pointer(regs); + struct fgraph_ops *gops = container_of(op, struct fgraph_ops, ops); + int bit; + + if (unlikely(ftrace_graph_is_dead())) + return; + + if (unlikely(atomic_read(¤t->tracing_graph_pause))) + return; - prepare_ftrace_return(ip, (unsigned long *)stack, 0); + bit = ftrace_test_recursion_trylock(ip, *parent); + if (bit < 0) + return; + + if (!function_graph_enter_ops(*parent, ip, 0, parent, gops)) + *parent = (unsigned long)&return_to_handler; + + ftrace_test_recursion_unlock(bit); } #endif diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index b87f9676f5ce..3d9e74ea6065 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -1070,6 +1070,7 @@ extern int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace, struct fgraph struct fgraph_ops { trace_func_graph_ent_t entryfunc; trace_func_graph_ret_t retfunc; + struct ftrace_ops ops; /* for the hash lists */ void *private; int idx; }; @@ -1105,6 +1106,11 @@ extern int function_graph_enter(unsigned long ret, unsigned long func, unsigned long frame_pointer, unsigned long *retp); +extern int +function_graph_enter_ops(unsigned long ret, unsigned long func, + unsigned long frame_pointer, unsigned long *retp, + struct fgraph_ops *gops); + struct ftrace_ret_stack * ftrace_graph_get_ret_stack(struct task_struct *task, int idx); diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index e35a941a5af3..ae42de909845 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -18,14 +18,6 @@ #include "ftrace_internal.h" #include "trace.h" -#ifdef CONFIG_DYNAMIC_FTRACE -#define ASSIGN_OPS_HASH(opsname, val) \ - .func_hash = val, \ - .local_hash.regex_lock = __MUTEX_INITIALIZER(opsname.local_hash.regex_lock), -#else -#define ASSIGN_OPS_HASH(opsname, val) -#endif - #define FGRAPH_RET_SIZE sizeof(struct ftrace_ret_stack) #define FGRAPH_RET_INDEX DIV_ROUND_UP(FGRAPH_RET_SIZE, sizeof(long)) @@ -381,7 +373,8 @@ int function_graph_enter(unsigned long ret, unsigned long func, if (gops == &fgraph_stub) continue; - if (gops->entryfunc(&trace)) + if (ftrace_ops_test(&gops->ops, func, NULL) && + gops->entryfunc(&trace, gops)) bitmap |= BIT(i); } @@ -402,6 +395,46 @@ int function_graph_enter(unsigned long ret, unsigned long func, return -EBUSY; } +/* This is called from ftrace_graph_func() via ftrace */ +int function_graph_enter_ops(unsigned long ret, unsigned long func, + unsigned long frame_pointer, unsigned long *retp, + struct fgraph_ops *gops) +{ + struct ftrace_graph_ent trace; + int index; + int type; + + /* Check whether the fgraph_ops is unregistered. */ + if (unlikely(fgraph_array[gops->idx] == &fgraph_stub)) + return -ENODEV; + + /* Use start for the distance to ret_stack (skipping over reserve) */ + index = ftrace_push_return_trace(ret, func, frame_pointer, retp, gops->idx); + if (index < 0) + return index; + type = get_fgraph_type(current, index); + + /* This is the first ret_stack for this fentry */ + if (type == FGRAPH_TYPE_RESERVED) + ++current->curr_ret_depth; + + trace.func = func; + trace.depth = current->curr_ret_depth; + if (gops->entryfunc(&trace, gops)) { + if (type == FGRAPH_TYPE_RESERVED) + set_fgraph_index_bitmap(current, index, BIT(gops->idx)); + else + add_fgraph_index_bitmap(current, index, BIT(gops->idx)); + return 0; + } + + if (type == FGRAPH_TYPE_RESERVED) { + current->curr_ret_stack -= FGRAPH_RET_INDEX + 1; + current->curr_ret_depth--; + } + return -EBUSY; +} + /* Retrieve a function return address to the trace stack on thread info.*/ static struct ftrace_ret_stack * ftrace_pop_return_trace(struct ftrace_graph_ret *trace, unsigned long *ret, @@ -662,17 +695,25 @@ unsigned long ftrace_graph_ret_addr(struct task_struct *task, int *idx, } #endif /* HAVE_FUNCTION_GRAPH_RET_ADDR_PTR */ -static struct ftrace_ops graph_ops = { - .func = ftrace_graph_func, - .flags = FTRACE_OPS_FL_INITIALIZED | - FTRACE_OPS_FL_PID | - FTRACE_OPS_GRAPH_STUB, +void fgraph_init_ops(struct ftrace_ops *dst_ops, + struct ftrace_ops *src_ops) +{ + dst_ops->func = ftrace_graph_func; + dst_ops->flags = FTRACE_OPS_FL_PID | FTRACE_OPS_GRAPH_STUB; + #ifdef FTRACE_GRAPH_TRAMP_ADDR - .trampoline = FTRACE_GRAPH_TRAMP_ADDR, + dst_ops->trampoline = FTRACE_GRAPH_TRAMP_ADDR; /* trampoline_size is only needed for dynamically allocated tramps */ #endif - ASSIGN_OPS_HASH(graph_ops, &global_ops.local_hash) -}; + +#ifdef CONFIG_DYNAMIC_FTRACE + if (src_ops) { + dst_ops->func_hash = &src_ops->local_hash; + mutex_init(&dst_ops->local_hash.regex_lock); + dst_ops->flags |= FTRACE_OPS_FL_INITIALIZED; + } +#endif +} void ftrace_graph_sleep_time_control(bool enable) { @@ -876,11 +917,20 @@ static int start_graph_tracing(void) int register_ftrace_graph(struct fgraph_ops *gops) { + int command = 0; int ret = 0; int i; mutex_lock(&ftrace_lock); + if (!gops->ops.func) { + gops->ops.flags |= FTRACE_OPS_GRAPH_STUB; + gops->ops.func = ftrace_graph_func; +#ifdef FTRACE_GRAPH_TRAMP_ADDR + gops->ops.trampoline = FTRACE_GRAPH_TRAMP_ADDR; +#endif + } + if (!fgraph_array[0]) { /* The array must always have real data on it */ for (i = 0; i < FGRAPH_ARRAY_SIZE; i++) @@ -917,9 +967,10 @@ int register_ftrace_graph(struct fgraph_ops *gops) */ ftrace_graph_return = return_run; ftrace_graph_entry = entry_run; - - ret = ftrace_startup(&graph_ops, FTRACE_START_FUNC_RET); + command = FTRACE_START_FUNC_RET; } + + ret = ftrace_startup(&gops->ops, command); out: mutex_unlock(&ftrace_lock); return ret; @@ -927,6 +978,7 @@ int register_ftrace_graph(struct fgraph_ops *gops) void unregister_ftrace_graph(struct fgraph_ops *gops) { + int command = 0; int i; mutex_lock(&ftrace_lock); @@ -934,25 +986,29 @@ void unregister_ftrace_graph(struct fgraph_ops *gops) if (unlikely(!ftrace_graph_active)) goto out; - for (i = 0; i < fgraph_array_cnt; i++) - if (gops == fgraph_array[i]) - break; - if (i >= fgraph_array_cnt) + if (unlikely(gops->idx < 0 || gops->idx >= fgraph_array_cnt)) goto out; - fgraph_array[i] = &fgraph_stub; - if (i + 1 == fgraph_array_cnt) { - for (; i >= 0; i--) - if (fgraph_array[i] != &fgraph_stub) - break; + WARN_ON_ONCE(fgraph_array[gops->idx] != gops); + + fgraph_array[gops->idx] = &fgraph_stub; + if (gops->idx + 1 == fgraph_array_cnt) { + i = gops->idx; + while (i >= 0 && fgraph_array[i] == &fgraph_stub) + i--; fgraph_array_cnt = i + 1; } ftrace_graph_active--; + + if (!ftrace_graph_active) + command = FTRACE_STOP_FUNC_RET; + + ftrace_shutdown(&gops->ops, command); + if (!ftrace_graph_active) { ftrace_graph_return = ftrace_stub_graph; ftrace_graph_entry = ftrace_graph_entry_stub; - ftrace_shutdown(&graph_ops, FTRACE_STOP_FUNC_RET); unregister_pm_notifier(&ftrace_suspend_notifier); unregister_trace_sched_switch(ftrace_graph_probe_sched_switch, NULL); } diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index a720dd7cf290..eeb736582e46 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -3016,6 +3016,8 @@ int ftrace_startup(struct ftrace_ops *ops, int command) if (unlikely(ftrace_disabled)) return -ENODEV; + ftrace_ops_init(ops); + ret = __register_ftrace_function(ops); if (ret) return ret; @@ -7323,7 +7325,7 @@ __init void ftrace_init_global_array_ops(struct trace_array *tr) tr->ops = &global_ops; tr->ops->private = tr; ftrace_init_trace_array(tr); - init_array_fgraph_ops(tr); + init_array_fgraph_ops(tr, tr->ops); } void ftrace_init_array_ops(struct trace_array *tr, ftrace_func_t func) diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h index 3b39936c9375..c70af6dc6485 100644 --- a/kernel/trace/trace.h +++ b/kernel/trace/trace.h @@ -893,8 +893,8 @@ extern int __trace_graph_entry(struct trace_array *tr, extern void __trace_graph_return(struct trace_array *tr, struct ftrace_graph_ret *trace, unsigned int trace_ctx); -extern void init_array_fgraph_ops(struct trace_array *tr); -extern int allocate_fgraph_ops(struct trace_array *tr); +extern void init_array_fgraph_ops(struct trace_array *tr, struct ftrace_ops *ops); +extern int allocate_fgraph_ops(struct trace_array *tr, struct ftrace_ops *ops); extern void free_fgraph_ops(struct trace_array *tr); #ifdef CONFIG_DYNAMIC_FTRACE @@ -977,6 +977,7 @@ static inline int ftrace_graph_notrace_addr(unsigned long addr) preempt_enable_notrace(); return ret; } + #else static inline int ftrace_graph_addr(struct ftrace_graph_ent *trace) { @@ -1002,18 +1003,19 @@ static inline bool ftrace_graph_ignore_func(struct ftrace_graph_ent *trace) (fgraph_max_depth && trace->depth >= fgraph_max_depth); } +void fgraph_init_ops(struct ftrace_ops *dst_ops, + struct ftrace_ops *src_ops); + #else /* CONFIG_FUNCTION_GRAPH_TRACER */ static inline enum print_line_t print_graph_function_flags(struct trace_iterator *iter, u32 flags) { return TRACE_TYPE_UNHANDLED; } -static inline void init_array_fgraph_ops(struct trace_array *tr) { } -static inline int allocate_fgraph_ops(struct trace_array *tr) -{ - return 0; -} static inline void free_fgraph_ops(struct trace_array *tr) { } +/* ftrace_ops may not be defined */ +#define init_array_fgraph_ops(tr, ops) do { } while (0) +#define allocate_fgraph_ops(tr, ops) ({ 0; }) #endif /* CONFIG_FUNCTION_GRAPH_TRACER */ extern struct list_head ftrace_pids; diff --git a/kernel/trace/trace_functions.c b/kernel/trace/trace_functions.c index 8e8da0d0ee52..13bf2415245d 100644 --- a/kernel/trace/trace_functions.c +++ b/kernel/trace/trace_functions.c @@ -91,7 +91,7 @@ int ftrace_create_function_files(struct trace_array *tr, if (!tr->ops) return -EINVAL; - ret = allocate_fgraph_ops(tr); + ret = allocate_fgraph_ops(tr, tr->ops); if (ret) { kfree(tr->ops); return ret; diff --git a/kernel/trace/trace_functions_graph.c b/kernel/trace/trace_functions_graph.c index 9ccc904a7703..7f30652f0e97 100644 --- a/kernel/trace/trace_functions_graph.c +++ b/kernel/trace/trace_functions_graph.c @@ -288,7 +288,7 @@ static struct fgraph_ops funcgraph_ops = { .retfunc = &trace_graph_return, }; -int allocate_fgraph_ops(struct trace_array *tr) +int allocate_fgraph_ops(struct trace_array *tr, struct ftrace_ops *ops) { struct fgraph_ops *gops; @@ -301,6 +301,9 @@ int allocate_fgraph_ops(struct trace_array *tr) tr->gops = gops; gops->private = tr; + + fgraph_init_ops(&gops->ops, ops); + return 0; } @@ -309,10 +312,11 @@ void free_fgraph_ops(struct trace_array *tr) kfree(tr->gops); } -__init void init_array_fgraph_ops(struct trace_array *tr) +__init void init_array_fgraph_ops(struct trace_array *tr, struct ftrace_ops *ops) { tr->gops = &funcgraph_ops; funcgraph_ops.private = tr; + fgraph_init_ops(&tr->gops->ops, ops); } static int graph_trace_init(struct trace_array *tr) From patchwork Tue Feb 6 15:10:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197519 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1624496dyb; Tue, 6 Feb 2024 07:38:38 -0800 (PST) X-Google-Smtp-Source: AGHT+IH+yNb6gHk92RDAe2mEx3Uq6PlFY99XHFhaqUVn4vlzPFxABFBxk0FVfBZcbBHRAPIG/jti X-Received: by 2002:a17:903:120f:b0:1d9:4919:ed10 with SMTP id l15-20020a170903120f00b001d94919ed10mr2386101plh.22.1707233917913; Tue, 06 Feb 2024 07:38:37 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707233917; cv=pass; d=google.com; s=arc-20160816; b=WhmIqo7NPXMBx7fKYmbKRlpBs6m2BjMWTcqtabA59nczCl6r+DlBBNj/8Ug8Rdrr+4 6ZHlakgCeDOwhYUUc2shiCYDDj/+VVRAGHVuu4dRVQ9uYx3wZe9z75X4hGDVKhN9M9bn SKLlUrO+Ib1OaVSvdMfOxyYNAtgvVSAoaKB0nYCKHupMx/6mW+Q5kxtn/c00AbPGtOhr 3ED9avRfaOLiiWBAic+rR+MrUHiahRTyVlI+maYW1vxKRU/eic7LZmFtNFy20CE5SuVu Jp2zeiS0lxif6kODqMMZVrGE696EyeJmVaDp/EuEbeqTDkLLaJrkKy1js9iK+xPEbo1Y yT0w== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=GFHomQBA5xBJD3fD0Fs8tP6D3i6iXmvE6Ql/kTKOIJA=; fh=w7O2MFcCEKnS/WmQLNi/J+AkfOUmrJ53q5rRW84OhLs=; b=oSjMWIQwI7FrpCEO4PYDk7jwubGwTd1j3YorvPoVAnY5srjKQlychbkUPWkFYgnXpf N1jLXZHuPCS49cFtocIDE+c0K923VlcGe0SLW6UZNY08iafwOq5E/hk3qFyYP63yMzT8 tMpOSJ324ZWSjm3DqBrKwQ7eWlSSXne9tzrJEj9LQJRP6VVKK60YBy7YhkkuJZfACfyL 5eRuPEMIpMW908VjclKKgfgI92QydAYYIGnANRboT0eEkPK5AfdmrIvl8HSm0KWVxMlh Ph59FMihVlDKvJ/bLanJ+niE2U2oAdwy5UYO6YHqW5c5DZmLIppNkPF9XNADuLOhugrI n28w==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="g4P0vbF/"; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55192-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55192-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=1; AJvYcCUx48yRZnEDdkkBRbjamWaLry92PDSvuw2eOtX27CKQytBj80XQD/DVpGb/2t7Scl8o8J2HpcD8pjy5W73Reb6ksxDGCQ== Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [147.75.48.161]) by mx.google.com with ESMTPS id h11-20020a170902f7cb00b001d7195d8a0fsi1776573plw.117.2024.02.06.07.38.37 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:38:37 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55192-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="g4P0vbF/"; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55192-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55192-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 9DD9DB27F9A for ; Tue, 6 Feb 2024 15:13:59 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 9C0EC1339A8; Tue, 6 Feb 2024 15:10:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="g4P0vbF/" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D3AAC13328A; Tue, 6 Feb 2024 15:10:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232210; cv=none; b=obSeae4A+MGHVFZvmSeXLgkWTp60y3Vb+t3yGw8vYGpPoSquQ7qJGXEhscxKi3jD+Ka3ZOiDZZ5ERXY91FgHhU8ybW/uXlrwHLuRVvw2wK/YTWo3TRCKMysXyvsnPygIzVpxV48uoVHr1wzoC1+eH3pAXYZiNY4CFrHiwwsCuqc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232210; c=relaxed/simple; bh=FOu7YqSOLWC5YAYXAHa1AHM2WwjkEtKXf4zI3+QaxKc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=k89Ps37mR8q2HTxjEpItpLAZPVmJcgQWRfymZugsdxvZ3H6yWBx9IjE6AERu/hdkzpAHs4P4Ztr7uCrIrCogh4CXLG/JORfakT8FMiJuJ4t9wgiI59FTz/ZnXU+TphPIVe35NndyoYgIY65GmH//d1Gmafbi6sWSqjmUzPgOCEg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=g4P0vbF/; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0F927C433F1; Tue, 6 Feb 2024 15:10:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232210; bh=FOu7YqSOLWC5YAYXAHa1AHM2WwjkEtKXf4zI3+QaxKc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=g4P0vbF/d5giNFPKpulD6m9kxqKCKQri3erw44CiBzN6J391JkRXviVOn6w09UTRo xGwLuXW7XcMfUOq63hHh4B5aY/B+dgyhpuhh7ClsYaK5WZaq58XlwykPWsDmhT/Ps1 jwzwpsHCldO5//3HkOYYfRPCJ+TVS4RQSwOLJ/ru/VHN4wAAaYUTfF5q68V08kR+dJ Dc0FcBkRrcA8wSaZnv2DagjUPqu+dman7WfeFqzt0WaEA7lypKdQQ3s6yMFxroaOEC IsU6Vce8oqRueQe58lTAgEuKEgUFa89Or/liSj8a0OcAuKOoMJwlzI0HWnssjW97I4 /PZalWNMp1TXg== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 14/36] function_graph: Use a simple LRU for fgraph_array index number Date: Wed, 7 Feb 2024 00:10:04 +0900 Message-Id: <170723220474.502590.7646977373091779892.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790164512566267889 X-GMAIL-MSGID: 1790164512566267889 From: Masami Hiramatsu (Google) Since the fgraph_array index is used for the bitmap on the shadow stack, it may leave some entries after a function_graph instance is removed. Thus if another instance reuses the fgraph_array index soon after releasing it, the fgraph may confuse to call the newer callback for the entries which are pushed by the older instance. To avoid reusing the fgraph_array index soon after releasing, introduce a simple LRU table for managing the index number. This will reduce the possibility of this confusion. Signed-off-by: Masami Hiramatsu (Google) --- Changes in v5: - Fix the underflow bug in fgraph_lru_release_index() and return 0 if the release is succeded. Changes in v4: - Newly added. --- kernel/trace/fgraph.c | 67 ++++++++++++++++++++++++++++++++++--------------- 1 file changed, 47 insertions(+), 20 deletions(-) diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index ae42de909845..323a74623543 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -99,10 +99,44 @@ enum { DEFINE_STATIC_KEY_FALSE(kill_ftrace_graph); int ftrace_graph_active; -static int fgraph_array_cnt; - static struct fgraph_ops *fgraph_array[FGRAPH_ARRAY_SIZE]; +/* LRU index table for fgraph_array */ +static int fgraph_lru_table[FGRAPH_ARRAY_SIZE]; +static int fgraph_lru_next; +static int fgraph_lru_last; + +static void fgraph_lru_init(void) +{ + int i; + + for (i = 0; i < FGRAPH_ARRAY_SIZE; i++) + fgraph_lru_table[i] = i; +} + +static int fgraph_lru_release_index(int idx) +{ + if (idx < 0 || idx >= FGRAPH_ARRAY_SIZE || + fgraph_lru_table[fgraph_lru_last] != -1) + return -1; + + fgraph_lru_table[fgraph_lru_last] = idx; + fgraph_lru_last = (fgraph_lru_last + 1) % FGRAPH_ARRAY_SIZE; + return 0; +} + +static int fgraph_lru_alloc_index(void) +{ + int idx = fgraph_lru_table[fgraph_lru_next]; + + if (idx == -1) + return -1; + + fgraph_lru_table[fgraph_lru_next] = -1; + fgraph_lru_next = (fgraph_lru_next + 1) % FGRAPH_ARRAY_SIZE; + return idx; +} + static inline int get_ret_stack_index(struct task_struct *t, int offset) { return t->ret_stack[offset] & FGRAPH_RET_INDEX_MASK; @@ -367,7 +401,7 @@ int function_graph_enter(unsigned long ret, unsigned long func, if (index < 0) goto out; - for (i = 0; i < fgraph_array_cnt; i++) { + for (i = 0; i < FGRAPH_ARRAY_SIZE; i++) { struct fgraph_ops *gops = fgraph_array[i]; if (gops == &fgraph_stub) @@ -935,21 +969,17 @@ int register_ftrace_graph(struct fgraph_ops *gops) /* The array must always have real data on it */ for (i = 0; i < FGRAPH_ARRAY_SIZE; i++) fgraph_array[i] = &fgraph_stub; + fgraph_lru_init(); } - /* Look for an available spot */ - for (i = 0; i < FGRAPH_ARRAY_SIZE; i++) { - if (fgraph_array[i] == &fgraph_stub) - break; - } - if (i >= FGRAPH_ARRAY_SIZE) { + i = fgraph_lru_alloc_index(); + if (i < 0 || + WARN_ON_ONCE(fgraph_array[i] != &fgraph_stub)) { ret = -EBUSY; goto out; } fgraph_array[i] = gops; - if (i + 1 > fgraph_array_cnt) - fgraph_array_cnt = i + 1; gops->idx = i; ftrace_graph_active++; @@ -979,25 +1009,22 @@ int register_ftrace_graph(struct fgraph_ops *gops) void unregister_ftrace_graph(struct fgraph_ops *gops) { int command = 0; - int i; mutex_lock(&ftrace_lock); if (unlikely(!ftrace_graph_active)) goto out; - if (unlikely(gops->idx < 0 || gops->idx >= fgraph_array_cnt)) + if (unlikely(gops->idx < 0 || gops->idx >= FGRAPH_ARRAY_SIZE)) + goto out; + + if (WARN_ON_ONCE(fgraph_array[gops->idx] != gops)) goto out; - WARN_ON_ONCE(fgraph_array[gops->idx] != gops); + if (fgraph_lru_release_index(gops->idx) < 0) + goto out; fgraph_array[gops->idx] = &fgraph_stub; - if (gops->idx + 1 == fgraph_array_cnt) { - i = gops->idx; - while (i >= 0 && fgraph_array[i] == &fgraph_stub) - i--; - fgraph_array_cnt = i + 1; - } ftrace_graph_active--; From patchwork Tue Feb 6 15:10:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197522 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1626323dyb; Tue, 6 Feb 2024 07:41:33 -0800 (PST) X-Google-Smtp-Source: AGHT+IH6YC8xUAGwuSIvSlF0DFPXoiuok9yUeUpvO5MGMDeHOMMYdEiVi5sTdHN61mu9fvMPZF3V X-Received: by 2002:aa7:8b91:0:b0:6e0:395f:d8aa with SMTP id r17-20020aa78b91000000b006e0395fd8aamr2285316pfd.12.1707234092955; Tue, 06 Feb 2024 07:41:32 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707234092; cv=pass; d=google.com; s=arc-20160816; b=rHY2S8b8asZKRCYtlxIfGADt8qyw6kFHWLBfrUm5mJ05o3uzgLg6LDuMYQ/HVjgXmQ cwSJh7aTshTuyf5DQJjU3l5ed6gI61BSGeyVWv9wnPhJRXLi2/t5OHtte1/iCen9gFxO i+2pNvhCxhElgiA8m5SFzwGy5iTKKJyzSt7f3zu2FMgR3YOlBv/T7iBLmXcw5X+sVNeC 8mzEDJiydqzmgHVomSt1/YIHqF2PFrHgY4QagOKumZDUyO4Jq/79ZKlZjcYnbGYhaWUi duZaDqV0a2bAj9RWJG14jMK+oCaR1QCDvskgQXGNEd+uxYTuUmJ+kEL/9efw9MIWqisF jhKA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=eq8JLCXeQ3c7UTs5Ah9ph1WbxcRSqoDw11nekl/kEz0=; fh=z2vW3fSDCBOQcVgw+R2jeirybWDzTjwNMK+zCeoPQ+k=; b=tkE876EF8ujAmNkejKmw+y29r9hCg5D/GNrkUZrhTdR6J/w3p+5v02u0JhSx0MjUcs O6gTa+8Jb21NsrLduCumUFNXlzII4Hf12j7MbaSBXIdyhSM4dzDQSTiDAca//Z5Igh9i idLy5KzXwuXTlpzc47jwrxd58XlB9DVd1fIuTK1bvLlqykeDAkvRNGD0jy9OA2RjGh8o 1QMNiv12Y2/YuxYeUDAm14QbQj6dJaFUIa9NOL7ri1ZQiiarNzW6Qv6pGxtsYUFGHV5D SicqHYkiObkFI02/81laYWDwOOBULK3n+Qzjk2COdOpONzsGIk/SQ3blDtbKSdQ+fGuh 947w==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=oTPcZnML; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55193-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55193-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=1; AJvYcCUk2ZhuAvVU+k3r2r/HmpuR4rjJXR4PP7vgN+tT+mMlXN3+JhUXJ06ErVNUNX+jufbAA4ILt0aV/fp4tRI8LNimYKInAg== Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id jo23-20020a056a00909700b006e04f0080f4si1843885pfb.335.2024.02.06.07.41.32 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:41:32 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55193-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=oTPcZnML; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55193-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55193-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 44107B2CCCA for ; Tue, 6 Feb 2024 15:14:19 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 5C4281332BC; Tue, 6 Feb 2024 15:10:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="oTPcZnML" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6C65F132C3F; Tue, 6 Feb 2024 15:10:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232222; cv=none; b=YHmlANjB/uXwiC/UMnkJLj0ry1fVCDZ5QjIFYg+gfgqe8DGuPuyoqAUHJl5tAx76L6P2eCQjfMQiIm2GMIhq4VMUI6dJrVcijZpI7KllDEkZyY50+ofXJ1+vET1JFM84m/KCJcC+r/omr8rFkbMxvwqJZ35x9Ie7ZM/TdiTqyzE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232222; c=relaxed/simple; bh=P2BCaLHD3WJJXWFdYDNVArinhnc1JMb+mPWQZqnSqls=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=PetvOT8qYIWCuaMj9QAehQNe0p2al8oPA1hrHLBepHmUeFB0P6QG62QPRinFaaaE3flYSrTg3ZhVCnY/kjFH15jbMq29qBn1tykP8Gy7Q9ipthFxYNnQr3bRvBMLO1Z/ycz7JZZtNeba7aXB6dG6Dr3UjQ/GGlnaHbRLyyz/TFI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=oTPcZnML; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4F721C433C7; Tue, 6 Feb 2024 15:10:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232222; bh=P2BCaLHD3WJJXWFdYDNVArinhnc1JMb+mPWQZqnSqls=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oTPcZnMLbAqb4UgMsnFRT06QxVi+DgCqfr5UguCEOK4RJcP4ebRf7FJ21Gn3XV5jz rxW7QNd3C0hHbDOM2VYk60FoMTYkYQ3hiTNiO9Z8trKcEdYH62g153g7zOGBffxKFN ZqVtu2UswBu5gLNndjBfQv55CY/FxZp81ZjqhoO97QUEujOAx2RSHDfD6uUebSprwE le8saDEMJCErml/Q9MZ8gJQsSRc+DAJZeZTIWWqADr1ulCN/FIaw29FcUp4Y17DoiV zc8pNlzkd/8VmZlYdbOw/erWFxCo/4rJwpuxF8rjMGHuK/fvZv5y6tqT41G14Z/Y36 QptC41XmN0brA== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 15/36] function_graph: Add "task variables" per task for fgraph_ops Date: Wed, 7 Feb 2024 00:10:15 +0900 Message-Id: <170723221579.502590.1810414801014910989.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790164696240568174 X-GMAIL-MSGID: 1790164696240568174 From: Steven Rostedt (VMware) Add a "task variables" array on the tasks shadow ret_stack that is the size of longs for each possible registered fgraph_ops. That's a total of 16, taking up 8 * 16 = 128 bytes (out of a page size 4k). This will allow for fgraph_ops to do specific features on a per task basis having a way to maintain state for each task. Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- Changes in v3: - Move fgraph_ops::idx to previous patch in the series. Changes in v2: - Make description lines shorter than 76 chars. --- include/linux/ftrace.h | 1 + kernel/trace/fgraph.c | 70 +++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 70 insertions(+), 1 deletion(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 3d9e74ea6065..737f84104577 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -1116,6 +1116,7 @@ ftrace_graph_get_ret_stack(struct task_struct *task, int idx); unsigned long ftrace_graph_ret_addr(struct task_struct *task, int *idx, unsigned long ret, unsigned long *retp); +unsigned long *fgraph_get_task_var(struct fgraph_ops *gops); /* * Sometimes we don't want to trace a function with the function diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index 323a74623543..4705681e65fd 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -92,10 +92,18 @@ enum { #define SHADOW_STACK_SIZE (PAGE_SIZE) #define SHADOW_STACK_INDEX (SHADOW_STACK_SIZE / sizeof(long)) /* Leave on a buffer at the end */ -#define SHADOW_STACK_MAX_INDEX (SHADOW_STACK_INDEX - (FGRAPH_RET_INDEX + 1)) +#define SHADOW_STACK_MAX_INDEX \ + (SHADOW_STACK_INDEX - (FGRAPH_RET_INDEX + 1 + FGRAPH_ARRAY_SIZE)) #define RET_STACK(t, index) ((struct ftrace_ret_stack *)(&(t)->ret_stack[index])) +/* + * Each fgraph_ops has a reservered unsigned long at the end (top) of the + * ret_stack to store task specific state. + */ +#define SHADOW_STACK_TASK_VARS(ret_stack) \ + ((unsigned long *)(&(ret_stack)[SHADOW_STACK_INDEX - FGRAPH_ARRAY_SIZE])) + DEFINE_STATIC_KEY_FALSE(kill_ftrace_graph); int ftrace_graph_active; @@ -182,6 +190,44 @@ static void return_run(struct ftrace_graph_ret *trace, struct fgraph_ops *ops) { } +static void ret_stack_set_task_var(struct task_struct *t, int idx, long val) +{ + unsigned long *gvals = SHADOW_STACK_TASK_VARS(t->ret_stack); + + gvals[idx] = val; +} + +static unsigned long * +ret_stack_get_task_var(struct task_struct *t, int idx) +{ + unsigned long *gvals = SHADOW_STACK_TASK_VARS(t->ret_stack); + + return &gvals[idx]; +} + +static void ret_stack_init_task_vars(unsigned long *ret_stack) +{ + unsigned long *gvals = SHADOW_STACK_TASK_VARS(ret_stack); + + memset(gvals, 0, sizeof(*gvals) * FGRAPH_ARRAY_SIZE); +} + +/** + * fgraph_get_task_var - retrieve a task specific state variable + * @gops: The ftrace_ops that owns the task specific variable + * + * Every registered fgraph_ops has a task state variable + * reserved on the task's ret_stack. This function returns the + * address to that variable. + * + * Returns the address to the fgraph_ops @gops tasks specific + * unsigned long variable. + */ +unsigned long *fgraph_get_task_var(struct fgraph_ops *gops) +{ + return ret_stack_get_task_var(current, gops->idx); +} + /* * @offset: The index into @t->ret_stack to find the ret_stack entry * @index: Where to place the index into @t->ret_stack of that entry @@ -791,6 +837,7 @@ static int alloc_retstack_tasklist(unsigned long **ret_stack_list) if (t->ret_stack == NULL) { atomic_set(&t->trace_overrun, 0); + ret_stack_init_task_vars(ret_stack_list[start]); t->curr_ret_stack = 0; t->curr_ret_depth = -1; /* Make sure the tasks see the 0 first: */ @@ -851,6 +898,7 @@ static void graph_init_task(struct task_struct *t, unsigned long *ret_stack) { atomic_set(&t->trace_overrun, 0); + ret_stack_init_task_vars(ret_stack); t->ftrace_timestamp = 0; t->curr_ret_stack = 0; t->curr_ret_depth = -1; @@ -949,6 +997,24 @@ static int start_graph_tracing(void) return ret; } +static void init_task_vars(int idx) +{ + struct task_struct *g, *t; + int cpu; + + for_each_online_cpu(cpu) { + if (idle_task(cpu)->ret_stack) + ret_stack_set_task_var(idle_task(cpu), idx, 0); + } + + read_lock(&tasklist_lock); + for_each_process_thread(g, t) { + if (t->ret_stack) + ret_stack_set_task_var(t, idx, 0); + } + read_unlock(&tasklist_lock); +} + int register_ftrace_graph(struct fgraph_ops *gops) { int command = 0; @@ -998,6 +1064,8 @@ int register_ftrace_graph(struct fgraph_ops *gops) ftrace_graph_return = return_run; ftrace_graph_entry = entry_run; command = FTRACE_START_FUNC_RET; + } else { + init_task_vars(gops->idx); } ret = ftrace_startup(&gops->ops, command); From patchwork Tue Feb 6 15:10:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197492 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1608012dyb; Tue, 6 Feb 2024 07:14:40 -0800 (PST) X-Google-Smtp-Source: AGHT+IHkeYIvlEQWRVtWGjlEyLygpXY0DLkU6jk5MJBcmy4kJBZQFGX6fvveu1kNJiRxt+o8VlYZ X-Received: by 2002:a05:651c:508:b0:2d0:be97:9ff8 with SMTP id o8-20020a05651c050800b002d0be979ff8mr1794847ljp.46.1707232480146; Tue, 06 Feb 2024 07:14:40 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707232480; cv=pass; d=google.com; s=arc-20160816; b=Pn0CCX8Vjv2amVDdRnqTY4KqNCFHzLFu6GaJzdNf3h3A81OhQYzWv2SlEN2nksI7wU Q/XLMqcOYgr97HjeBGBWSbC9BHazjfC3YMCs60/g+QdJ775d/2W899QDOgOnSjbk3n9s rP4U4hBDaFEB0mMpb8KVnXYDIBcPdcKpG7JgX5p5v8KAejsSiScgwL1CZMHaj4mTz+Sz W07HwIclRoGvyizWN9FBCH2flC2fWor5szudyDP0PSlovLZD+E+mP10G7Kgi/C9m4yH2 DP1Nt3DT9Tggp6MoDB0qH2RAy+jen4BWDqjAzOiByyutFPJO+XhDycJIsOFQ37BTrL4V qT/g== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=zZNcPt2Kw9GPylQ66Co41GHBTcCZ3fsaVWRsPDJ+4HE=; fh=ndOzjWlpzzrkJS7uVEeNT7dB7DH45nuDKQN9pHnZ1uQ=; b=uiud9Veso1xl4hwpEh60jjq6fO+0SLkXVCXB4LEf7VsnmSa/s3+xWGJLcYJrN/WXGK 3925ZsmYSqNqlNJSrRU4xPHcx/73R/UhXcvRA/W/bmgdfoRpP4vnWjGEwlaQj/kJBUfR 0BqGkYLyGKGPs8xtU61pIaz3J/DbEe6xjUVa6oNSzZx7SxPXSkGVJhyeMjfu76bjg9RN J9LHv4X+uarEpZyZX/7564REsldeg1od0ruMG9Ni6C9KofqKN7YzmHvv38tAjyRo36HN OJVnJFo2jiLzMaEmUj0A/n+yU6ThNd3akjjqp1B8nf0OFx1jJAtaU812p1TJTURpT3Ib 92kQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=bDoozujb; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55194-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55194-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=1; AJvYcCVKBond8kHPDKlEJ0F1jtxlYLYFLGfdavp9brlWBHEvxAPKHZSqF42g3iSs5sc/2JXCKELptvthpsu1o0nEZeqLzoKKpw== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id d19-20020aa7d693000000b0056054177f7fsi1227347edr.635.2024.02.06.07.14.39 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:14:40 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55194-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=bDoozujb; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55194-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55194-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id BA60A1F268A4 for ; Tue, 6 Feb 2024 15:14:39 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 580CF136987; Tue, 6 Feb 2024 15:10:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="bDoozujb" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EEFC2132C3F; Tue, 6 Feb 2024 15:10:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232234; cv=none; b=nVTjeKZtDOiQW6C5FPlJLvIXPNNqYHcRNB9O/b2TEtxI04uMXqiZa61sfCwAaTc3jAvxLVoj0HlvxQq18B79P5huPOEKNY1i7Gy1BUap3HkvTIuCcwtTbT7zacjqLVxNjFuWlc5coIP9QykWGAUrJ23OZsHB2oYW/T0OkY+39Cg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232234; c=relaxed/simple; bh=KCZGqgIeYVvFZmOBhB5ewytf0GjVMwpXASjcmqLghIg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=ctdfEzKkBl3KsWXHUD4lBDSc8wlM5gzl9I4fsQnU/eQFRb/wJUJZiGCfLOYqKRHjVRW+LOMNjsjFW+mCiFBIB9jFIY7vNptrjjvlNI/VkukpFq3xEUAMvVZGhnxb/zfyv9AJr4V64JIwSKEJ4GfQP6bz8XOv/ms2ztdVYO6+VG0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=bDoozujb; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2A3A0C433F1; Tue, 6 Feb 2024 15:10:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232233; bh=KCZGqgIeYVvFZmOBhB5ewytf0GjVMwpXASjcmqLghIg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bDoozujbX6pBeMatbvgFMxc9aeZGLMSJHLTv/xLq3DDAbyg2f4V1YYt79lEpz2HTV r8pOyZbKBixvtdMBv2OMbKuGNEv2i/zX02WNyMsw8sNWJbTg1tcKfqILHQHPyy+Bw3 Z0cTGCh+80Viuxjr/YtXNJRTcylKN92fG6OkcVplaGCkaKbRRF3uBy/YhNEN4eYkDb eYlyU8Yqme5zlxrKqx6pvzj2M6mE+MUUMt4xOljPXNi1l2UtZSVWTCvNI8LP4Leaa2 X891WeTl9WDG6mS+fzGk8XSj14s3NIC2+pp0B1QjMENUGKlTqGXMko8UUJalIaifp6 eWONbT89oKX7Q== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 16/36] function_graph: Move set_graph_function tests to shadow stack global var Date: Wed, 7 Feb 2024 00:10:27 +0900 Message-Id: <170723222764.502590.5691178889682424992.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790163004852593918 X-GMAIL-MSGID: 1790163004852593918 From: Steven Rostedt (VMware) The use of the task->trace_recursion for the logic used for the set_graph_funnction was a bit of an abuse of that variable. Now that there exists global vars that are per stack for registered graph traces, use that instead. Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- include/linux/trace_recursion.h | 5 +---- kernel/trace/trace.h | 32 +++++++++++++++++++++----------- kernel/trace/trace_functions_graph.c | 6 +++--- kernel/trace/trace_irqsoff.c | 4 ++-- kernel/trace/trace_sched_wakeup.c | 4 ++-- 5 files changed, 29 insertions(+), 22 deletions(-) diff --git a/include/linux/trace_recursion.h b/include/linux/trace_recursion.h index d48cd92d2364..2efd5ec46d7f 100644 --- a/include/linux/trace_recursion.h +++ b/include/linux/trace_recursion.h @@ -44,9 +44,6 @@ enum { */ TRACE_IRQ_BIT, - /* Set if the function is in the set_graph_function file */ - TRACE_GRAPH_BIT, - /* * In the very unlikely case that an interrupt came in * at a start of graph tracing, and we want to trace @@ -60,7 +57,7 @@ enum { * that preempted a softirq start of a function that * preempted normal context!!!! Luckily, it can't be * greater than 3, so the next two bits are a mask - * of what the depth is when we set TRACE_GRAPH_BIT + * of what the depth is when we set TRACE_GRAPH_FL */ TRACE_GRAPH_DEPTH_START_BIT, diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h index c70af6dc6485..d5501b1a10a6 100644 --- a/kernel/trace/trace.h +++ b/kernel/trace/trace.h @@ -897,11 +897,16 @@ extern void init_array_fgraph_ops(struct trace_array *tr, struct ftrace_ops *ops extern int allocate_fgraph_ops(struct trace_array *tr, struct ftrace_ops *ops); extern void free_fgraph_ops(struct trace_array *tr); +enum { + TRACE_GRAPH_FL = 1, +}; + #ifdef CONFIG_DYNAMIC_FTRACE extern struct ftrace_hash __rcu *ftrace_graph_hash; extern struct ftrace_hash __rcu *ftrace_graph_notrace_hash; -static inline int ftrace_graph_addr(struct ftrace_graph_ent *trace) +static inline int +ftrace_graph_addr(unsigned long *task_var, struct ftrace_graph_ent *trace) { unsigned long addr = trace->func; int ret = 0; @@ -923,12 +928,11 @@ static inline int ftrace_graph_addr(struct ftrace_graph_ent *trace) } if (ftrace_lookup_ip(hash, addr)) { - /* * This needs to be cleared on the return functions * when the depth is zero. */ - trace_recursion_set(TRACE_GRAPH_BIT); + *task_var |= TRACE_GRAPH_FL; trace_recursion_set_depth(trace->depth); /* @@ -948,11 +952,14 @@ static inline int ftrace_graph_addr(struct ftrace_graph_ent *trace) return ret; } -static inline void ftrace_graph_addr_finish(struct ftrace_graph_ret *trace) +static inline void +ftrace_graph_addr_finish(struct fgraph_ops *gops, struct ftrace_graph_ret *trace) { - if (trace_recursion_test(TRACE_GRAPH_BIT) && + unsigned long *task_var = fgraph_get_task_var(gops); + + if ((*task_var & TRACE_GRAPH_FL) && trace->depth == trace_recursion_depth()) - trace_recursion_clear(TRACE_GRAPH_BIT); + *task_var &= ~TRACE_GRAPH_FL; } static inline int ftrace_graph_notrace_addr(unsigned long addr) @@ -979,7 +986,7 @@ static inline int ftrace_graph_notrace_addr(unsigned long addr) } #else -static inline int ftrace_graph_addr(struct ftrace_graph_ent *trace) +static inline int ftrace_graph_addr(unsigned long *task_var, struct ftrace_graph_ent *trace) { return 1; } @@ -988,17 +995,20 @@ static inline int ftrace_graph_notrace_addr(unsigned long addr) { return 0; } -static inline void ftrace_graph_addr_finish(struct ftrace_graph_ret *trace) +static inline void ftrace_graph_addr_finish(struct fgraph_ops *gops, struct ftrace_graph_ret *trace) { } #endif /* CONFIG_DYNAMIC_FTRACE */ extern unsigned int fgraph_max_depth; -static inline bool ftrace_graph_ignore_func(struct ftrace_graph_ent *trace) +static inline bool +ftrace_graph_ignore_func(struct fgraph_ops *gops, struct ftrace_graph_ent *trace) { + unsigned long *task_var = fgraph_get_task_var(gops); + /* trace it when it is-nested-in or is a function enabled. */ - return !(trace_recursion_test(TRACE_GRAPH_BIT) || - ftrace_graph_addr(trace)) || + return !((*task_var & TRACE_GRAPH_FL) || + ftrace_graph_addr(task_var, trace)) || (trace->depth < 0) || (fgraph_max_depth && trace->depth >= fgraph_max_depth); } diff --git a/kernel/trace/trace_functions_graph.c b/kernel/trace/trace_functions_graph.c index 7f30652f0e97..66cce73e94f8 100644 --- a/kernel/trace/trace_functions_graph.c +++ b/kernel/trace/trace_functions_graph.c @@ -160,7 +160,7 @@ int trace_graph_entry(struct ftrace_graph_ent *trace, if (!ftrace_trace_task(tr)) return 0; - if (ftrace_graph_ignore_func(trace)) + if (ftrace_graph_ignore_func(gops, trace)) return 0; if (ftrace_graph_ignore_irqs()) @@ -247,7 +247,7 @@ void trace_graph_return(struct ftrace_graph_ret *trace, long disabled; int cpu; - ftrace_graph_addr_finish(trace); + ftrace_graph_addr_finish(gops, trace); if (trace_recursion_test(TRACE_GRAPH_NOTRACE_BIT)) { trace_recursion_clear(TRACE_GRAPH_NOTRACE_BIT); @@ -269,7 +269,7 @@ void trace_graph_return(struct ftrace_graph_ret *trace, static void trace_graph_thresh_return(struct ftrace_graph_ret *trace, struct fgraph_ops *gops) { - ftrace_graph_addr_finish(trace); + ftrace_graph_addr_finish(gops, trace); if (trace_recursion_test(TRACE_GRAPH_NOTRACE_BIT)) { trace_recursion_clear(TRACE_GRAPH_NOTRACE_BIT); diff --git a/kernel/trace/trace_irqsoff.c b/kernel/trace/trace_irqsoff.c index 5478f4c4f708..fce064e20570 100644 --- a/kernel/trace/trace_irqsoff.c +++ b/kernel/trace/trace_irqsoff.c @@ -184,7 +184,7 @@ static int irqsoff_graph_entry(struct ftrace_graph_ent *trace, unsigned int trace_ctx; int ret; - if (ftrace_graph_ignore_func(trace)) + if (ftrace_graph_ignore_func(gops, trace)) return 0; /* * Do not trace a function if it's filtered by set_graph_notrace. @@ -214,7 +214,7 @@ static void irqsoff_graph_return(struct ftrace_graph_ret *trace, unsigned long flags; unsigned int trace_ctx; - ftrace_graph_addr_finish(trace); + ftrace_graph_addr_finish(gops, trace); if (!func_prolog_dec(tr, &data, &flags)) return; diff --git a/kernel/trace/trace_sched_wakeup.c b/kernel/trace/trace_sched_wakeup.c index 49bcc812652c..130ca7e7787e 100644 --- a/kernel/trace/trace_sched_wakeup.c +++ b/kernel/trace/trace_sched_wakeup.c @@ -120,7 +120,7 @@ static int wakeup_graph_entry(struct ftrace_graph_ent *trace, unsigned int trace_ctx; int ret = 0; - if (ftrace_graph_ignore_func(trace)) + if (ftrace_graph_ignore_func(gops, trace)) return 0; /* * Do not trace a function if it's filtered by set_graph_notrace. @@ -149,7 +149,7 @@ static void wakeup_graph_return(struct ftrace_graph_ret *trace, struct trace_array_cpu *data; unsigned int trace_ctx; - ftrace_graph_addr_finish(trace); + ftrace_graph_addr_finish(gops, trace); if (!func_prolog_preempt_disable(tr, &data, &trace_ctx)) return; From patchwork Tue Feb 6 15:10:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197493 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1608272dyb; Tue, 6 Feb 2024 07:14:59 -0800 (PST) X-Google-Smtp-Source: AGHT+IEelgSUSWak1ukkEvDu1RbHgVHbXk3V4d5E515xD8zM/jmrSujhvdfGqBq7twEyiJdff2Es X-Received: by 2002:ac8:7342:0:b0:42c:c5d:430c with SMTP id q2-20020ac87342000000b0042c0c5d430cmr2328387qtp.18.1707232499201; Tue, 06 Feb 2024 07:14:59 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707232499; cv=pass; d=google.com; s=arc-20160816; b=jwGVfgDUaBTX1GXDlcBLtMcmOgAeHIAPvpI1F83pdcrgEwhAqodPJasY6RhFNzI/WN HGLIcBmTwBeN+pkcqvjTib3/J8JCydKNljK48L38jGHFCMEo8+72GiDbalKTpCw8LcCi feDGIhgdHOZzkDtFQXRo2SelO1BURtPU3qm1KC2wnMJgXjV/rYxwCysMFb7G6TlrNEh8 6ZIK1UmO6pK4VW88UX+PdtOZbpMxiR6a2CpeuOQXirBzBS/ISY+thFql4Ki4jFRIzGh8 dUg/A0x3JqDOe1xGtsWmfuedqxY8KHcxo0tRpZqXU/VQJ//dxBzvinC30yLGfGo10A/T a23A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=GoKuTuktY4fmTQ/r/V34b+qK7JltcK60u1i+qHry0y8=; fh=oF1z8GU3axiC2gNsoX3qa+litLHVgj4/YVTSWFYww5c=; b=cY5lvL7PgSQKXg56XWfeO1acLRZNkc3Aupj/yr5wCfwJV5TawoSwexQNyu2fNwEDPy 6buqfsFCZlag8pF1xPH/7/+XftdB3WLvLtGs/GvB879oj0qdjXV7nu8YQ4LOfBjIiKHN mtbi39q67G64p94b5+geDi7fRQM9sPnY4IejtxQY1Ngkvhcgt+ctXQrz618olO4eWLz7 YDSgdMys+IfUzJ5volC8jjvHyEPhNOj0pG/2iVErXN/nEqs6KbZ32vC1OMOT0PjSmK7v 2+01xsJQPPI0aDZAp0/XnVkUqEDd6SMbsF0nCoBWifosa5X53coJmhV3qBlcTzUUbyve RfsQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="XV40T/7C"; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55195-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55195-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=1; AJvYcCUJGwCUOkUl+oM2LTloQ3sKesn4IRxVKKe6UI/R/t5AWuOn74EDYZuba3fseoSoMN0oBN3w/Z75GtyNeDL/XTOZi+DM+A== Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id t7-20020a05622a148700b0042bdc3f52b2si2566940qtx.386.2024.02.06.07.14.59 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:14:59 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55195-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="XV40T/7C"; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55195-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55195-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 006EC1C2424B for ; Tue, 6 Feb 2024 15:14:59 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 54B8413699A; Tue, 6 Feb 2024 15:10:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="XV40T/7C" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6879A132C3F; Tue, 6 Feb 2024 15:10:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232245; cv=none; b=KW0YjhAa6YzJzFyHg7ktIXmaAz5rZEnFSX4GBNHrGmm0fYNB68F1rZGoUn5hYDpKGkrdERpfAXon+Gse30A9dPo6DfpUrYI0qZOt0BkVB6RqB+ZTaobhcoALewApY4rBF1K3KShiYCCOs4TXcskUIEp4iTeFX3AFkiJwtbQe9eQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232245; c=relaxed/simple; bh=5ZU6O1yZKo2WgktQxSz1C2P+0dzB3kSfvhbv8qkb5cU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=E3CJsTow5M8XwrUlS/UMisXWNDAzPBVQrX9spr8p9O0n+zgC5nMFZOsx4IpCmw2PTDB+I6Z8bvt4Qb/FJhVAxauKGnTNk0dQf+BaP5rDDUKM8nfXNfy3qQD6J3GK0uHHkvcsElePvAaG9w53UGx0fDMYYVNgvSxslb2gb5ddzFk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=XV40T/7C; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 840DAC433C7; Tue, 6 Feb 2024 15:10:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232244; bh=5ZU6O1yZKo2WgktQxSz1C2P+0dzB3kSfvhbv8qkb5cU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XV40T/7CnAxRJQsW8vXS7t9DXl1V6JIvtGD8FFlFiOpq1uVWn5pVm1hb8lRCnk3IO nshKQci+JbzOSZwO0Z7uYoH/3LWBavV/1MS/WpnjsxMhAh32SFSO3x/7UWLS8bsR7t wtgbWSNxtuZEeQl3GbPkGX34mo0A/T11Pke1GZ5qeT43q9QWErZFTnqGUtGzFeoOsv ukCV/hGZb6lgVIUDWfVsB5YJ35iLgp5HJRNgeOMi8BskVrsfnUnvSFcvsOm4rhDbml dF5LKNyja0AtxB3i67C/owfjTWK+LCawyy7i13exn7ZiYaflt11DG/7H9KIxMyOC7v BTHjtHyoQUlag== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 17/36] function_graph: Move graph depth stored data to shadow stack global var Date: Wed, 7 Feb 2024 00:10:39 +0900 Message-Id: <170723223911.502590.4125309234469241884.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790163024957853428 X-GMAIL-MSGID: 1790163024957853428 From: Steven Rostedt (VMware) The use of the task->trace_recursion for the logic used for the function graph depth was a bit of an abuse of that variable. Now that there exists global vars that are per stack for registered graph traces, use that instead. Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- include/linux/trace_recursion.h | 29 ----------------------------- kernel/trace/trace.h | 34 ++++++++++++++++++++++++++++++++-- 2 files changed, 32 insertions(+), 31 deletions(-) diff --git a/include/linux/trace_recursion.h b/include/linux/trace_recursion.h index 2efd5ec46d7f..00e792bf148d 100644 --- a/include/linux/trace_recursion.h +++ b/include/linux/trace_recursion.h @@ -44,25 +44,6 @@ enum { */ TRACE_IRQ_BIT, - /* - * In the very unlikely case that an interrupt came in - * at a start of graph tracing, and we want to trace - * the function in that interrupt, the depth can be greater - * than zero, because of the preempted start of a previous - * trace. In an even more unlikely case, depth could be 2 - * if a softirq interrupted the start of graph tracing, - * followed by an interrupt preempting a start of graph - * tracing in the softirq, and depth can even be 3 - * if an NMI came in at the start of an interrupt function - * that preempted a softirq start of a function that - * preempted normal context!!!! Luckily, it can't be - * greater than 3, so the next two bits are a mask - * of what the depth is when we set TRACE_GRAPH_FL - */ - - TRACE_GRAPH_DEPTH_START_BIT, - TRACE_GRAPH_DEPTH_END_BIT, - /* * To implement set_graph_notrace, if this bit is set, we ignore * function graph tracing of called functions, until the return @@ -78,16 +59,6 @@ enum { #define trace_recursion_clear(bit) do { (current)->trace_recursion &= ~(1<<(bit)); } while (0) #define trace_recursion_test(bit) ((current)->trace_recursion & (1<<(bit))) -#define trace_recursion_depth() \ - (((current)->trace_recursion >> TRACE_GRAPH_DEPTH_START_BIT) & 3) -#define trace_recursion_set_depth(depth) \ - do { \ - current->trace_recursion &= \ - ~(3 << TRACE_GRAPH_DEPTH_START_BIT); \ - current->trace_recursion |= \ - ((depth) & 3) << TRACE_GRAPH_DEPTH_START_BIT; \ - } while (0) - #define TRACE_CONTEXT_BITS 4 #define TRACE_FTRACE_START TRACE_FTRACE_BIT diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h index d5501b1a10a6..d8cdcbb4e5a6 100644 --- a/kernel/trace/trace.h +++ b/kernel/trace/trace.h @@ -899,8 +899,38 @@ extern void free_fgraph_ops(struct trace_array *tr); enum { TRACE_GRAPH_FL = 1, + + /* + * In the very unlikely case that an interrupt came in + * at a start of graph tracing, and we want to trace + * the function in that interrupt, the depth can be greater + * than zero, because of the preempted start of a previous + * trace. In an even more unlikely case, depth could be 2 + * if a softirq interrupted the start of graph tracing, + * followed by an interrupt preempting a start of graph + * tracing in the softirq, and depth can even be 3 + * if an NMI came in at the start of an interrupt function + * that preempted a softirq start of a function that + * preempted normal context!!!! Luckily, it can't be + * greater than 3, so the next two bits are a mask + * of what the depth is when we set TRACE_GRAPH_FL + */ + + TRACE_GRAPH_DEPTH_START_BIT, + TRACE_GRAPH_DEPTH_END_BIT, }; +static inline unsigned long ftrace_graph_depth(unsigned long *task_var) +{ + return (*task_var >> TRACE_GRAPH_DEPTH_START_BIT) & 3; +} + +static inline void ftrace_graph_set_depth(unsigned long *task_var, int depth) +{ + *task_var &= ~(3 << TRACE_GRAPH_DEPTH_START_BIT); + *task_var |= (depth & 3) << TRACE_GRAPH_DEPTH_START_BIT; +} + #ifdef CONFIG_DYNAMIC_FTRACE extern struct ftrace_hash __rcu *ftrace_graph_hash; extern struct ftrace_hash __rcu *ftrace_graph_notrace_hash; @@ -933,7 +963,7 @@ ftrace_graph_addr(unsigned long *task_var, struct ftrace_graph_ent *trace) * when the depth is zero. */ *task_var |= TRACE_GRAPH_FL; - trace_recursion_set_depth(trace->depth); + ftrace_graph_set_depth(task_var, trace->depth); /* * If no irqs are to be traced, but a set_graph_function @@ -958,7 +988,7 @@ ftrace_graph_addr_finish(struct fgraph_ops *gops, struct ftrace_graph_ret *trace unsigned long *task_var = fgraph_get_task_var(gops); if ((*task_var & TRACE_GRAPH_FL) && - trace->depth == trace_recursion_depth()) + trace->depth == ftrace_graph_depth(task_var)) *task_var &= ~TRACE_GRAPH_FL; } From patchwork Tue Feb 6 15:10:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197494 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1608574dyb; Tue, 6 Feb 2024 07:15:23 -0800 (PST) X-Google-Smtp-Source: AGHT+IGxxHhZZlqmYIg8Seb4O1yR0vGHVJMDUxk1Jx6jmazsUiiz8H4mXK0e8TV+4sCodhNrrCBe X-Received: by 2002:a17:906:31c6:b0:a38:801:6770 with SMTP id f6-20020a17090631c600b00a3808016770mr2756239ejf.24.1707232523376; Tue, 06 Feb 2024 07:15:23 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707232523; cv=pass; d=google.com; s=arc-20160816; b=OHusW3XYWhSGjCqD0Kofsl1dn0kDBYw2Mv0FgnbLPyM/1eCUlQH/796nQylYsZaqOc tjF8U902F3o1Br5KyhHoYTqVDcDT2nt0KcsLH0kDnuSQ4+cLQNKa/vaNxj7TIJTfYP6x Kxt9P1PFv8Ek8V5uuzzsWPbt18qq4YO17Zb1t8fB0V1e1dmqwkgeytOHallWm61Ujs0d 6xijhPKkMm44LoH4OgWq/gvNQz+gsYR1CCoAzCransWxLkKUtrNQDKPHyXEJx16mxIg4 M8qtUKMir8YF5hT6XVc4GD8KkG2KUTVIUR5O51cuOZs/Gg0N3XRcAvODr98Yfvcffdja 5Ciw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=CgjbyiPfUVr56PkrTBGflIyaHfPHVrAirnWqGZAFJy8=; fh=cFg/BCY0leMmooSxAw6Eo7J7ceKE0gF7bsd5ektg0jQ=; b=P2X6n9xxckryuNX452HY53RmjrsWfm/j5hqZaFFWnr992b3UIMRpHIOQXFwKbKssRD wbsD/5CDPH5YVoWKdO8iqWgjy7U48GHeFvdLez5NTcHZLszW7CqBNP1UsAdp6msw2A8l duauc7juMXDMdbwOA7XUzvwS1P3u86vkD904jDHRHQ64eDI/Q7ySzzd8lnuk+S5bbyV/ /t04h3JkXlZ4kx1hKJciBMqd/PbR7dqGS0ohxOHsh/Skk1LJD6zFmnZWFDTKYseUANuK 3a4yeYhqvprQitBhfmZ6wVKR7vm6iK3ZXPHN0qt7mDnlsnS9dIItMEebAvmkAnQNp5y+ bH0A==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="T/4N5YOC"; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55196-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55196-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=1; AJvYcCWBUvzNofQKveaeLabkWo5Q4qKcIASdJBdf8imq6Mw4IpRYfHmv3fuNcvFE6kKo5T36QbcY+b3FliObSfXhTrzjXpOM1g== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id t25-20020a1709066bd900b00a384f537a74si143278ejs.860.2024.02.06.07.15.23 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:15:23 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55196-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="T/4N5YOC"; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55196-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55196-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id F1A391F22885 for ; Tue, 6 Feb 2024 15:15:22 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id D369EA59; Tue, 6 Feb 2024 15:10:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="T/4N5YOC" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 19951134CFC; Tue, 6 Feb 2024 15:10:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232256; cv=none; b=R/R9BpCVKrfSv+OzCTjsqCjweRSzldzyRpylR7oQCHExiqvthbwsQzUPT6hvJLhmcBbuWNV0CE8RSAJhqW4D3sgYDjCitBba241Sve1+yZeVGS9uL1Ml9m/cceEIdJnJ0mT1boeW7iUqFgHVdMmA18FP1R4i7sHdVMUJpghGvuo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232256; c=relaxed/simple; bh=XT8mAO5U8kYBF5nHzre3NP4S7gGsaqfcEiXyQTVrS+w=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=ZTxsAHiVgw/dEPExONl1Qxts0eVsK3ptKWTs9QObO20H6ftGZyHYO2/Wcwfrls9qS5PWg6esTg7za4nOvu33bw6N6GmVCgWH7IIWi2J80buEBbkaw4+FPMuZmvVX7MBSWvA5W8spfrUS7QZdvS3KiErABF9suRw+Yad5RnpDWtU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=T/4N5YOC; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id ABAC7C433C7; Tue, 6 Feb 2024 15:10:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232255; bh=XT8mAO5U8kYBF5nHzre3NP4S7gGsaqfcEiXyQTVrS+w=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=T/4N5YOCnRDcLX/JjZRuT1ZrzMUhmlwFugMIeQXUi+cmemubEX/Ty40c0JklpMR9O a9M3+BBrmqjP6scM2QbsIaxkeCHi1p1lQuTgROaAPgE+ld3TN/ZnV+AD7zngU+RegR dv//Y3DkIRvv7zIgigLFMI1YWuJs0WLOgQWw3Hr8U5LmxphN/Dlkne9z2Ae7UMPh4a FMKJQItEN5BEAAnu4cWsNyhgqQvXa64IEXypv0uQI7tAMnHQt8rjR7pRqJz8Rl2Eh7 b1NnkwdKvwvZ+kMz+DbL/+42tOXXsXQMXAnSzL667XKN84o15Gw8AjjJ3PJp/mlyLK z5TeeA2Is6aqg== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 18/36] function_graph: Move graph notrace bit to shadow stack global var Date: Wed, 7 Feb 2024 00:10:50 +0900 Message-Id: <170723225020.502590.2063430645806603064.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790163050419715895 X-GMAIL-MSGID: 1790163050419715895 From: Steven Rostedt (VMware) The use of the task->trace_recursion for the logic used for the function graph no-trace was a bit of an abuse of that variable. Now that there exists global vars that are per stack for registered graph traces, use that instead. Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- Changes in v2: - Make description lines shorter than 76 chars. --- include/linux/trace_recursion.h | 7 ------- kernel/trace/trace.h | 9 +++++++++ kernel/trace/trace_functions_graph.c | 10 ++++++---- 3 files changed, 15 insertions(+), 11 deletions(-) diff --git a/include/linux/trace_recursion.h b/include/linux/trace_recursion.h index 00e792bf148d..cc11b0e9d220 100644 --- a/include/linux/trace_recursion.h +++ b/include/linux/trace_recursion.h @@ -44,13 +44,6 @@ enum { */ TRACE_IRQ_BIT, - /* - * To implement set_graph_notrace, if this bit is set, we ignore - * function graph tracing of called functions, until the return - * function is called to clear it. - */ - TRACE_GRAPH_NOTRACE_BIT, - /* Used to prevent recursion recording from recursing. */ TRACE_RECORD_RECURSION_BIT, }; diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h index d8cdcbb4e5a6..029a80a38c23 100644 --- a/kernel/trace/trace.h +++ b/kernel/trace/trace.h @@ -918,8 +918,17 @@ enum { TRACE_GRAPH_DEPTH_START_BIT, TRACE_GRAPH_DEPTH_END_BIT, + + /* + * To implement set_graph_notrace, if this bit is set, we ignore + * function graph tracing of called functions, until the return + * function is called to clear it. + */ + TRACE_GRAPH_NOTRACE_BIT, }; +#define TRACE_GRAPH_NOTRACE (1 << TRACE_GRAPH_NOTRACE_BIT) + static inline unsigned long ftrace_graph_depth(unsigned long *task_var) { return (*task_var >> TRACE_GRAPH_DEPTH_START_BIT) & 3; diff --git a/kernel/trace/trace_functions_graph.c b/kernel/trace/trace_functions_graph.c index 66cce73e94f8..13d0387ac6a6 100644 --- a/kernel/trace/trace_functions_graph.c +++ b/kernel/trace/trace_functions_graph.c @@ -130,6 +130,7 @@ static inline int ftrace_graph_ignore_irqs(void) int trace_graph_entry(struct ftrace_graph_ent *trace, struct fgraph_ops *gops) { + unsigned long *task_var = fgraph_get_task_var(gops); struct trace_array *tr = gops->private; struct trace_array_cpu *data; unsigned long flags; @@ -138,7 +139,7 @@ int trace_graph_entry(struct ftrace_graph_ent *trace, int ret; int cpu; - if (trace_recursion_test(TRACE_GRAPH_NOTRACE_BIT)) + if (*task_var & TRACE_GRAPH_NOTRACE) return 0; /* @@ -149,7 +150,7 @@ int trace_graph_entry(struct ftrace_graph_ent *trace, * returning from the function. */ if (ftrace_graph_notrace_addr(trace->func)) { - trace_recursion_set(TRACE_GRAPH_NOTRACE_BIT); + *task_var |= TRACE_GRAPH_NOTRACE_BIT; /* * Need to return 1 to have the return called * that will clear the NOTRACE bit. @@ -240,6 +241,7 @@ void __trace_graph_return(struct trace_array *tr, void trace_graph_return(struct ftrace_graph_ret *trace, struct fgraph_ops *gops) { + unsigned long *task_var = fgraph_get_task_var(gops); struct trace_array *tr = gops->private; struct trace_array_cpu *data; unsigned long flags; @@ -249,8 +251,8 @@ void trace_graph_return(struct ftrace_graph_ret *trace, ftrace_graph_addr_finish(gops, trace); - if (trace_recursion_test(TRACE_GRAPH_NOTRACE_BIT)) { - trace_recursion_clear(TRACE_GRAPH_NOTRACE_BIT); + if (*task_var & TRACE_GRAPH_NOTRACE) { + *task_var &= ~TRACE_GRAPH_NOTRACE; return; } From patchwork Tue Feb 6 15:11:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197526 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1630291dyb; Tue, 6 Feb 2024 07:48:10 -0800 (PST) X-Google-Smtp-Source: AGHT+IGYriz4Eve+T4hqfv20jrV5noi30J8FX0dU2ZQejZjOlnortxkj0XQJVWNAlE7KGsTMTRtR X-Received: by 2002:a17:90a:f490:b0:296:2057:28c with SMTP id bx16-20020a17090af49000b002962057028cmr2500384pjb.31.1707234489804; Tue, 06 Feb 2024 07:48:09 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707234489; cv=pass; d=google.com; s=arc-20160816; b=ajyd17SdfBMk7EQc9o7hR5snH6iVJMC5/ACIoFNGzPvhQoI7UQjkMowVbWjHTr0YZu 4E0AKrzbsxlKurY0dZi7x0SVg+ekoBQKBm1s41frzRwfRrn9PXp5lbCJPMmorR/LALOr pBvI/iKOkrxI1fQ5krprLD8Qh9lvCkq1gKcMUUWBmmGDg2ja1JTRuVKpxefUruVnar6p 7spodumBjhwtz8FfQ363iaYMNTUvioPfZ+M62DTN2f9/Xc+rf3AkW+1jZX6/9gW9v6kJ EBpPgN+vg9xeRxoZb8Mt/GqBdF0OMAX+EdJM6l95MQlDX2koxeYgQvZ2MGjMoebt+1cD h5Ug== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=WYM881lLf6av2LR2Qdtvu6W+jYpYSblKpOgtRNFRqro=; fh=7TZvCmMqdijD3og+La3x1k6NfvwQSBI2mSrhHWHrUhU=; b=Fh+d+r8xM70PQQPuJKUJddTtKposfBoP+LtSmlYhBL8yf8iFboRv/9C59refiYp5WO eHwdhoHTjQGlIBcZbtBsB3AwM526mzU8OFvDntRXJ448TxQVlFF3SbjOMjjCJVUDW4Lf LHrIf4YP2AHOzgr0TpM1DNV1E+GSHpLSvmD4JHNmOuyyNtEKi7/x7ovPa7wh5NrXUgoc nDWd5ZtTBdR5H/50Pjft64hg6PNm3Nq3qoL/Hu9m0P8jsv3wkvhx3c7FYLCJ5OZmwHMc 3SM2+bRjGBTXBdyKp7LCDNp8GweONyBrcMxoi7jQ5E0vsR86bPzI9heItcjDQ7p1vR+V 7rxw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=AwYQ1uPI; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55198-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55198-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=2; AJvYcCVDGDlqb2R2bal7+YCf8d2YIINadVVmHw2UBCzWYYelbO2ZSiDcn0yxDrGOrBt7lAh6+UvzUw8XnqrR6kqNdu0DjPNjdw== Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id d20-20020a17090ab31400b002962f429837si1325375pjr.145.2024.02.06.07.48.09 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:48:09 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55198-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=AwYQ1uPI; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55198-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55198-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 13996B31700 for ; Tue, 6 Feb 2024 15:15:55 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id E4015137C43; Tue, 6 Feb 2024 15:11:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="AwYQ1uPI" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D1F7F1339B6; Tue, 6 Feb 2024 15:11:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232266; cv=none; b=KUYOpfVzjqyIlS1iJfIb66XBpWq1ekpw8JX47VtGFDs1DKyKMZnLqmHp4KthmpOEY3ZaHNP+7j59BWsarEexmMJmpBbuaKInWgvaIk9fpHu0lwEzYgY95fZiyO5sPXNE6YguHzacWlebEjBaDAwrqO3wECa+C2FpVuFwhni5SE0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232266; c=relaxed/simple; bh=mCsnjbx0olUYMnP8Idg8RaURtW9wE/3l9D3uy0XJrV8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=Fbrq1t378vFZvvDcHhGvisAIfkJf1LiYzjvJLfgw4V7pMkkUItG0UY6u9ioMlPCMHIFVA26BOBLlqRqmP1+i+wXZA5yZuiq+c9TreCLIF4r4TC7NrIKlzMgjPZz6CnKDF2fJBuDmU20ct2elgL7CFEjd9Tyz5cEG0rRTHYjNz34= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=AwYQ1uPI; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id A1133C433C7; Tue, 6 Feb 2024 15:11:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232266; bh=mCsnjbx0olUYMnP8Idg8RaURtW9wE/3l9D3uy0XJrV8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=AwYQ1uPItdeS+ImEPRutmA0TbW+BF1DYRSN3yVszthOAZlHg5RtXJSvHDZd8/v0I/ eu0JqBULwdLt4KxyXQVwJF6AfKy0W5y1BRTMgEs+74xm7Eiwm6ZWjaG50JeDRpn38b EcZuDn9c8Yho7ScLB7ufh0oOhv+753MmAUF6n52BU9XmRMj4q2+M529L7Mm98SPRo1 omfg1BDcFSKvogrl2lBhRwiE9zdnQv2D4ChbfyV/ZLUCFeX2v3Bh1dVVRpTnYksu5y uH1Y8CtC+vYFoIIiit0sbdcFMzD/R+7aktXNFtt8e4oQRWz6yYVNVWwvFGAyIN4No5 YrtvGY2h9seBQ== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 19/36] function_graph: Implement fgraph_reserve_data() and fgraph_retrieve_data() Date: Wed, 7 Feb 2024 00:11:01 +0900 Message-Id: <170723226123.502590.4924916690354403889.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790165112346898604 X-GMAIL-MSGID: 1790165112346898604 From: Steven Rostedt (VMware) Added functions that can be called by a fgraph_ops entryfunc and retfunc to store state between the entry of the function being traced to the exit of the same function. The fgraph_ops entryfunc() may call fgraph_reserve_data() to store up to 32 words onto the task's shadow ret_stack and this then can be retrieved by fgraph_retrieve_data() called by the corresponding retfunc(). Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- Changes in v3: - Store fgraph_array index to the data entry. - Both function requires fgraph_array index to store/retrieve data. - Reserve correct size of the data. - Return correct data area. Changes in v2: - Retrieve the reserved size by fgraph_retrieve_data(). - Expand the maximum data size to 32 words. - Update stack index with __get_index(val) if FGRAPH_TYPE_ARRAY entry. - fix typos and make description lines shorter than 76 chars. --- include/linux/ftrace.h | 3 + kernel/trace/fgraph.c | 175 ++++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 170 insertions(+), 8 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 737f84104577..815e865f46c9 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -1075,6 +1075,9 @@ struct fgraph_ops { int idx; }; +void *fgraph_reserve_data(int idx, int size_bytes); +void *fgraph_retrieve_data(int idx, int *size_bytes); + /* * Stack of return addresses for functions * of a thread. diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index 4705681e65fd..5cc7f44037cd 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -41,17 +41,29 @@ * bits: 10 - 11 Type of storage * 0 - reserved * 1 - bitmap of fgraph_array index + * 2 - reserved data * * For bitmap of fgraph_array index * bits: 12 - 27 The bitmap of fgraph_ops fgraph_array index * + * For reserved data: + * bits: 12 - 17 The size in words that is stored + * bits: 18 - 23 The index of fgraph_array, which shows who is stored + * * That is, at the end of function_graph_enter, if the first and forth * fgraph_ops on the fgraph_array[] (index 0 and 3) needs their retfunc called - * on the return of the function being traced, this is what will be on the - * task's shadow ret_stack: (the stack grows upward) + * on the return of the function being traced, and the forth fgraph_ops + * stored two words of data, this is what will be on the task's shadow + * ret_stack: (the stack grows upward) * * | | <- task->curr_ret_stack * +--------------------------------------------+ + * | data_type(idx:3, size:2, | + * | offset:FGRAPH_RET_INDEX+3) | ( Data with size of 2 words) + * +--------------------------------------------+ ( It is 4 words from the ret_stack) + * | STORED DATA WORD 2 | + * | STORED DATA WORD 1 | + * +------i-------------------------------------+ * | bitmap_type(bitmap:(BIT(3)|BIT(0)), | * | offset:FGRAPH_RET_INDEX) | <- the offset is from here * +--------------------------------------------+ @@ -78,14 +90,23 @@ enum { FGRAPH_TYPE_RESERVED = 0, FGRAPH_TYPE_BITMAP = 1, + FGRAPH_TYPE_DATA = 2, }; #define FGRAPH_INDEX_SIZE 16 #define FGRAPH_INDEX_MASK GENMASK(FGRAPH_INDEX_SIZE - 1, 0) #define FGRAPH_INDEX_SHIFT (FGRAPH_TYPE_SHIFT + FGRAPH_TYPE_SIZE) -/* Currently the max stack index can't be more than register callers */ -#define FGRAPH_MAX_INDEX (FGRAPH_INDEX_SIZE + FGRAPH_RET_INDEX) +#define FGRAPH_DATA_SIZE 5 +#define FGRAPH_DATA_MASK ((1 << FGRAPH_DATA_SIZE) - 1) +#define FGRAPH_DATA_SHIFT (FGRAPH_TYPE_SHIFT + FGRAPH_TYPE_SIZE) + +#define FGRAPH_DATA_INDEX_SIZE 4 +#define FGRAPH_DATA_INDEX_MASK ((1 << FGRAPH_DATA_INDEX_SIZE) - 1) +#define FGRAPH_DATA_INDEX_SHIFT (FGRAPH_DATA_SHIFT + FGRAPH_DATA_SIZE) + +#define FGRAPH_MAX_INDEX \ + ((FGRAPH_INDEX_SIZE << FGRAPH_DATA_SIZE) + FGRAPH_RET_INDEX) #define FGRAPH_ARRAY_SIZE FGRAPH_INDEX_SIZE @@ -97,6 +118,8 @@ enum { #define RET_STACK(t, index) ((struct ftrace_ret_stack *)(&(t)->ret_stack[index])) +#define FGRAPH_MAX_DATA_SIZE (sizeof(long) * (1 << FGRAPH_DATA_SIZE)) + /* * Each fgraph_ops has a reservered unsigned long at the end (top) of the * ret_stack to store task specific state. @@ -145,14 +168,39 @@ static int fgraph_lru_alloc_index(void) return idx; } +static inline int __get_index(unsigned long val) +{ + return val & FGRAPH_RET_INDEX_MASK; +} + +static inline int __get_type(unsigned long val) +{ + return (val >> FGRAPH_TYPE_SHIFT) & FGRAPH_TYPE_MASK; +} + +static inline int __get_data_index(unsigned long val) +{ + return (val >> FGRAPH_DATA_INDEX_SHIFT) & FGRAPH_DATA_INDEX_MASK; +} + +static inline int __get_data_size(unsigned long val) +{ + return (val >> FGRAPH_DATA_SHIFT) & FGRAPH_DATA_MASK; +} + +static inline unsigned long get_fgraph_entry(struct task_struct *t, int index) +{ + return t->ret_stack[index]; +} + static inline int get_ret_stack_index(struct task_struct *t, int offset) { - return t->ret_stack[offset] & FGRAPH_RET_INDEX_MASK; + return __get_index(t->ret_stack[offset]); } static inline int get_fgraph_type(struct task_struct *t, int offset) { - return (t->ret_stack[offset] >> FGRAPH_TYPE_SHIFT) & FGRAPH_TYPE_MASK; + return __get_type(t->ret_stack[offset]); } static inline unsigned long @@ -179,6 +227,22 @@ add_fgraph_index_bitmap(struct task_struct *t, int offset, unsigned long bitmap) t->ret_stack[offset] |= (bitmap << FGRAPH_INDEX_SHIFT); } +static inline void *get_fgraph_data(struct task_struct *t, int index) +{ + unsigned long val = t->ret_stack[index]; + + if (__get_type(val) != FGRAPH_TYPE_DATA) + return NULL; + index -= __get_data_size(val); + return (void *)&t->ret_stack[index]; +} + +static inline unsigned long make_fgraph_data(int idx, int size, int offset) +{ + return (idx << FGRAPH_DATA_INDEX_SHIFT) | (size << FGRAPH_DATA_SHIFT) | + (FGRAPH_TYPE_DATA << FGRAPH_TYPE_SHIFT) | offset; +} + /* ftrace_graph_entry set to this to tell some archs to run function graph */ static int entry_run(struct ftrace_graph_ent *trace, struct fgraph_ops *ops) { @@ -212,6 +276,92 @@ static void ret_stack_init_task_vars(unsigned long *ret_stack) memset(gvals, 0, sizeof(*gvals) * FGRAPH_ARRAY_SIZE); } +/** + * fgraph_reserve_data - Reserve storage on the task's ret_stack + * @idx: The index of fgraph_array + * @size_bytes: The size in bytes to reserve + * + * Reserves space of up to FGRAPH_MAX_DATA_SIZE bytes on the + * task's ret_stack shadow stack, for a given fgraph_ops during + * the entryfunc() call. If entryfunc() returns zero, the storage + * is discarded. An entryfunc() can only call this once per iteration. + * The fgraph_ops retfunc() can retrieve this stored data with + * fgraph_retrieve_data(). + * + * Returns: On success, a pointer to the data on the stack. + * Otherwise, NULL if there's not enough space left on the + * ret_stack for the data, or if fgraph_reserve_data() was called + * more than once for a single entryfunc() call. + */ +void *fgraph_reserve_data(int idx, int size_bytes) +{ + unsigned long val; + void *data; + int curr_ret_stack = current->curr_ret_stack; + int data_size; + + if (size_bytes > FGRAPH_MAX_DATA_SIZE) + return NULL; + + /* Convert to number of longs + data word */ + data_size = DIV_ROUND_UP(size_bytes, sizeof(long)); + + val = get_fgraph_entry(current, curr_ret_stack - 1); + data = ¤t->ret_stack[curr_ret_stack]; + + curr_ret_stack += data_size + 1; + if (unlikely(curr_ret_stack >= SHADOW_STACK_MAX_INDEX)) + return NULL; + + val = make_fgraph_data(idx, data_size, __get_index(val) + data_size + 1); + + /* Set the last word to be reserved */ + current->ret_stack[curr_ret_stack - 1] = val; + + /* Make sure interrupts see this */ + barrier(); + current->curr_ret_stack = curr_ret_stack; + /* Again sync with interrupts, and reset reserve */ + current->ret_stack[curr_ret_stack - 1] = val; + + return data; +} + +/** + * fgraph_retrieve_data - Retrieve stored data from fgraph_reserve_data() + * @idx: the index of fgraph_array (fgraph_ops::idx) + * @size_bytes: pointer to retrieved data size. + * + * This is to be called by a fgraph_ops retfunc(), to retrieve data that + * was stored by the fgraph_ops entryfunc() on the function entry. + * That is, this will retrieve the data that was reserved on the + * entry of the function that corresponds to the exit of the function + * that the fgraph_ops retfunc() is called on. + * + * Returns: The stored data from fgraph_reserve_data() called by the + * matching entryfunc() for the retfunc() this is called from. + * Or NULL if there was nothing stored. + */ +void *fgraph_retrieve_data(int idx, int *size_bytes) +{ + int index = current->curr_ret_stack - 1; + unsigned long val; + + val = get_fgraph_entry(current, index); + while (__get_type(val) == FGRAPH_TYPE_DATA) { + if (__get_data_index(val) == idx) + goto found; + index -= __get_data_size(val) + 1; + val = get_fgraph_entry(current, index); + } + return NULL; +found: + if (size_bytes) + *size_bytes = __get_data_size(val) * + sizeof(long); + return get_fgraph_data(current, index); +} + /** * fgraph_get_task_var - retrieve a task specific state variable * @gops: The ftrace_ops that owns the task specific variable @@ -449,13 +599,18 @@ int function_graph_enter(unsigned long ret, unsigned long func, for (i = 0; i < FGRAPH_ARRAY_SIZE; i++) { struct fgraph_ops *gops = fgraph_array[i]; + int save_curr_ret_stack; if (gops == &fgraph_stub) continue; + save_curr_ret_stack = current->curr_ret_stack; if (ftrace_ops_test(&gops->ops, func, NULL) && gops->entryfunc(&trace, gops)) bitmap |= BIT(i); + else + /* Clear out any saved storage */ + current->curr_ret_stack = save_curr_ret_stack; } if (!bitmap) @@ -481,6 +636,7 @@ int function_graph_enter_ops(unsigned long ret, unsigned long func, struct fgraph_ops *gops) { struct ftrace_graph_ent trace; + int save_curr_ret_stack; int index; int type; @@ -500,13 +656,15 @@ int function_graph_enter_ops(unsigned long ret, unsigned long func, trace.func = func; trace.depth = current->curr_ret_depth; + save_curr_ret_stack = current->curr_ret_stack; if (gops->entryfunc(&trace, gops)) { if (type == FGRAPH_TYPE_RESERVED) set_fgraph_index_bitmap(current, index, BIT(gops->idx)); else add_fgraph_index_bitmap(current, index, BIT(gops->idx)); return 0; - } + } else + current->curr_ret_stack = save_curr_ret_stack; if (type == FGRAPH_TYPE_RESERVED) { current->curr_ret_stack -= FGRAPH_RET_INDEX + 1; @@ -651,7 +809,8 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs * curr_ret_stack is after that. */ barrier(); - current->curr_ret_stack -= FGRAPH_RET_INDEX + 1; + current->curr_ret_stack = index - FGRAPH_RET_INDEX; + current->curr_ret_depth--; return ret; } From patchwork Tue Feb 6 15:11:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197495 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1609903dyb; Tue, 6 Feb 2024 07:16:59 -0800 (PST) X-Google-Smtp-Source: AGHT+IF803qKcvbao8Hzx7zYdYE/1Qk+q/kc+WGWlPzrFhIYhK8hufGqM3VE6d3/UMH01Ij/5Cmm X-Received: by 2002:a05:6870:2181:b0:219:b163:70db with SMTP id l1-20020a056870218100b00219b16370dbmr2167130oae.1.1707232619487; Tue, 06 Feb 2024 07:16:59 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707232619; cv=pass; d=google.com; s=arc-20160816; b=V0BxEHM6REFfzxM9JfRxBG/T2EpKzr3NS8VaGe+i+8IdPMeBCJZOIJAgQlTj0nYyBT DAeHGQr3pTEFAheyuRhd77zVYl+c4iVJ1D8cZjHkCuFIP3KpWuWTGFNRyv4gk6meVnvB K33wn8PcJccng6+NJZg1IwNZWLuH85cUA3WHbUEk9ph7jxZ+e2+NO+nnrwYCfRWBeE6A ow3CNUmWKjG4zfgoBQ4snC2vHrOPw9gt+1HyjrNFhqzAw0LCsdapkzxpDrLaYbWNi0WD SqcYCnCeIz635aYwofJJpUAhkIJK6TpLc6t5fVKUCG8nN3Am3HI89Cr/BGYbxbShttjr Yn3A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=WehbdJJlNWnrUSLEUqyk8hVOeMhs2HprMZUHNe4aQ1E=; fh=4l3ELT18BrOBBPHlR8TY1F4ZzX7qckfq5rikFVt+Utg=; b=0DwWnkThrCTXl/D9d02ABXv/Wv0ZiLNMK+khgpq9nhk4UOZrvVQFItExqaeeD7J6Db Y8CxH+7JMfN/VKLZKV1Ggf25HO/VC6fxeVnqNBVYg2NtazHd7OegvdVbjFumIb3aVZMJ CCfG5phrsDSkkDOjF8M5Uz3AZuoPAYw+0cUcYsq+spLOpRt1Z+J14ZWlpn3+7d2pf9Li gv+WkOLXfkHBVSGl9Ab3+E15vQPV+SdIPOiPgY8o6h5i2JJWCS9OXgwhZtlWIxKQuDjl rSTMvGbDCSZpj8/Pv1EcdDNmYPNYLiqGbLbuXq04uQ+An0dlisSVabX1gZ34OqPaiYj3 v9iw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=RxhC6RWD; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55201-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55201-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=1; AJvYcCXI2g6zualTw2v1HTDx9yjYNuAtuIx/5S8flTMpNp3matyJor00pRDcdRnQXI6G0QKnttK4OsT3uOV1VMttgDb0MCbtng== Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id d9-20020ac85d89000000b0042be4e05e20si2524360qtx.28.2024.02.06.07.16.59 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:16:59 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55201-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=RxhC6RWD; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55201-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55201-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 9E5FD1C23EAC for ; Tue, 6 Feb 2024 15:16:49 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 99BBA1384BB; Tue, 6 Feb 2024 15:11:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="RxhC6RWD" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 07E15138485; Tue, 6 Feb 2024 15:11:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232277; cv=none; b=fOkg8UOgILzQKZcsSF70s9OLVmF6vwLOrcvAl0NNOqmqKdiGWbc/6ptSZB8dpT0/Z9Hl+RQHJqo5o3EwaWCTmuJ71uFzt4FDtWkFxdxLToLpUprZagTlVrjM0hLEp21yToVVuI/qDF5IEvpIWuniOTitL3ZaS0aOxOCLl7ySV5I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232277; c=relaxed/simple; bh=1uNXOkEFaClbT5CkblJsYzFpCC7+iCA37jndjn78mY0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=dey8dRg+lvL0BDJyCcIJbfg111TJWgd/lUHbb/6nU+sbKMvNihnUPV4Ek4cLRk/LHtQjE8ijtUXu/kSwYJS0ZT8xKNzggFs76704moCNd4W5v8gHuq3Ul6GHmaCy2P60ul8hHjdX9CQv8jCTziLx2DPLsX3JtPF3roSqQxhRQVs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=RxhC6RWD; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 50236C43390; Tue, 6 Feb 2024 15:11:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232276; bh=1uNXOkEFaClbT5CkblJsYzFpCC7+iCA37jndjn78mY0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RxhC6RWDPPrgl/x17Eod24up9sOf87T7J7nVxw8FNzw8wV31olhxwTIh/FCfR9/Ad qOrQsHGSdNDD0/f41Fkz3Q6kz1wBdyvSdIvowEgFAN1kTRqAo8riEDCsJSNDa/Jgv1 mAiw9ePJz/cJo34sl4VYvuDT4GaaNPP++iptX33hF65fgfqdTa8N5nbApHxW2FBAlw RinuCfBnJ/3QmMhakAqgWPdYshJiBy6dpp75IokkZ/peBsR0CHZfjnJ4kk6PS2fo7M PnlIOnV/6mzOL0+yIQ1Ix2El5QcLGukf6Q5Wfr9HiEESk7Kaj+zRqS/Ao35yVzV/fw V8HApTmO4XXOw== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 20/36] function_graph: Improve push operation for several interrupts Date: Wed, 7 Feb 2024 00:11:12 +0900 Message-Id: <170723227198.502590.10431025573751489041.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790163151032708938 X-GMAIL-MSGID: 1790163151032708938 From: Masami Hiramatsu (Google) Improve push and data reserve operation on the shadow stack for several sequencial interrupts. To push a ret_stack or data entry on the shadow stack, we need to prepare an index (offset) entry before updating the stack pointer (curr_ret_stack) so that unwinder from interrupts can find the next return address from the shadow stack. Currently we do write index, update the curr_ret_stack, and rewrite it again. But that is not enough for the case if two interrupts happens and the first one breaks it. For example, 1. write reserved index entry at ret_stack[new_index - 1] and ret addr. 2. interrupt comes. 2.1. push new index and ret addr on ret_stack. 2.2. pop it. (corrupt entries on new_index - 1) 3. return from interrupt. 4. update curr_ret_stack = new_index 5. interrupt comes again. 5.1. unwind <------ may not work. To avoid this issue, this introduces a new rsrv_ret_stack stack reservation pointer and a new push code (slow path) to commit previous reserved code forcibly. 0. update rsrv_ret_stack = new_index. 1. write reserved index entry at ret_stack[new_index - 1] and ret addr. 2. interrupt comes. 2.0. if rsrv_ret_stack != curr_ret_stack, add reserved index entry on ret_stack[rsrv_ret_stack - 1] to point the previous ret_stack pointed by ret_stack[curr_ret_stack - 1]. and update curr_ret_stack = rsrv_ret_stack. 2.1. push new index and ret addr on ret_stack. 2.2. pop it. (corrupt entries on new_index - 1) 3. return from interrupt. 4. update curr_ret_stack = new_index 5. interrupt comes again. 5.1. unwind works, because curr_ret_stack points the previously saved ret_stack. 5.2. this can do push/pop operations too. 6. return from interrupt. 7. rewrite reserved index entry at ret_stack[new_index] again. This maybe a bit heavier but safer. Signed-off-by: Masami Hiramatsu (Google) --- Changes in v7: - Handle interrupting pop case correctly not only push. - Always update rsrv_ret_stack before curr_ret_stack. Changes in v6: - Newly added. --- include/linux/sched.h | 1 kernel/trace/fgraph.c | 166 +++++++++++++++++++++++++++++++++++-------------- 2 files changed, 118 insertions(+), 49 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 1cfabb44dcbb..12666b640c22 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1390,6 +1390,7 @@ struct task_struct { #ifdef CONFIG_FUNCTION_GRAPH_TRACER /* Index of current stored address in ret_stack: */ int curr_ret_stack; + int rsrv_ret_stack; int curr_ret_depth; /* Stack of return addresses for return function tracing: */ diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index 5cc7f44037cd..269ea458c57a 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -298,31 +298,47 @@ void *fgraph_reserve_data(int idx, int size_bytes) unsigned long val; void *data; int curr_ret_stack = current->curr_ret_stack; + int rsrv_ret_stack = current->rsrv_ret_stack; int data_size; if (size_bytes > FGRAPH_MAX_DATA_SIZE) return NULL; + /* + * Since this API is used after pushing ret_stack, curr_ret_stack + * should be synchronized with rsrv_ret_stack. + */ + if (WARN_ON_ONCE(curr_ret_stack != rsrv_ret_stack)) + return NULL; + /* Convert to number of longs + data word */ data_size = DIV_ROUND_UP(size_bytes, sizeof(long)); val = get_fgraph_entry(current, curr_ret_stack - 1); data = ¤t->ret_stack[curr_ret_stack]; - curr_ret_stack += data_size + 1; - if (unlikely(curr_ret_stack >= SHADOW_STACK_MAX_INDEX)) + rsrv_ret_stack += data_size + 1; + if (unlikely(rsrv_ret_stack >= SHADOW_STACK_MAX_INDEX)) return NULL; val = make_fgraph_data(idx, data_size, __get_index(val) + data_size + 1); - /* Set the last word to be reserved */ - current->ret_stack[curr_ret_stack - 1] = val; - - /* Make sure interrupts see this */ + /* Extend the reserved-ret_stack at first */ + current->rsrv_ret_stack = rsrv_ret_stack; + /* And sync with interrupts, to see the new rsrv_ret_stack */ barrier(); - current->curr_ret_stack = curr_ret_stack; - /* Again sync with interrupts, and reset reserve */ - current->ret_stack[curr_ret_stack - 1] = val; + /* + * The same reason as the push, this entry must be here before updating + * the curr_ret_stack. But any interrupt comes before updating + * curr_ret_stack, it may commit it with different reserve entry. + * Thus we need to write the data entry after update the curr_ret_stack + * again. And these operations must be ordered. + */ + current->ret_stack[rsrv_ret_stack - 1] = val; + barrier(); + current->curr_ret_stack = rsrv_ret_stack; + barrier(); + current->ret_stack[rsrv_ret_stack - 1] = val; return data; } @@ -403,7 +419,16 @@ get_ret_stack(struct task_struct *t, int offset, int *index) return NULL; idx = get_ret_stack_index(t, --offset); - if (WARN_ON_ONCE(idx <= 0 || idx > offset)) + /* + * This can happen if an interrupt comes just before the first push + * increments the curr_ret_stack, and that interrupt pushes another + * entry. In that case, the frist push is forcibly committed with a + * reserved entry which points -1 stack index. + */ + if (unlikely(idx > offset)) + return NULL; + + if (WARN_ON_ONCE(idx <= 0)) return NULL; offset -= idx; @@ -473,7 +498,7 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func, struct ftrace_ret_stack *ret_stack; unsigned long long calltime; unsigned long val; - int index; + int index, rindex; if (unlikely(ftrace_graph_is_dead())) return -EBUSY; @@ -481,18 +506,42 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func, if (!current->ret_stack) return -EBUSY; - /* - * At first, check whether the previous fgraph callback is pushed by - * the fgraph on the same function entry. - * But if @func is the self tail-call function, we also need to ensure - * the ret_stack is not for the previous call by checking whether the - * bit of @fgraph_idx is set or not. - */ - ret_stack = get_ret_stack(current, current->curr_ret_stack, &index); - if (ret_stack && ret_stack->func == func && - get_fgraph_type(current, index + FGRAPH_RET_INDEX) == FGRAPH_TYPE_BITMAP && - !is_fgraph_index_set(current, index + FGRAPH_RET_INDEX, fgraph_idx)) - return index + FGRAPH_RET_INDEX; + index = READ_ONCE(current->curr_ret_stack); + rindex = READ_ONCE(current->rsrv_ret_stack); + if (unlikely(index < rindex)) { + /* + * This interrupts the push operation. Commit previous push + * temporarily with reserved entry. + */ + if (unlikely(index <= 0)) + /* This will make ret_stack[index - 1] points -1 */ + val = rindex - index; + else + val = get_ret_stack_index(current, index - 1) + + rindex - index; + current->ret_stack[rindex - 1] = val; + /* Forcibly commit it */ + current->curr_ret_stack = index = rindex; + } else { + /* + * This is normal path OR interrupting in the pop commit operation. + * In both way, @rindex should point correct next index. So use it. + * Check whether the previous fgraph callback is pushed by the fgraph + * on the same function entry. + * But if @func is the self tail-call function, we also need to ensure + * the ret_stack is not for the previous call by checking whether the + * bit of @fgraph_idx is set or not. + */ + if (likely(rindex == index)) { + ret_stack = get_ret_stack(current, index, &index); + if (ret_stack && ret_stack->func == func && + get_fgraph_type(current, index + FGRAPH_RET_INDEX) == FGRAPH_TYPE_BITMAP && + !is_fgraph_index_set(current, index + FGRAPH_RET_INDEX, fgraph_idx)) + return index + FGRAPH_RET_INDEX; + } + /* Since get_ret_stack() overwrites 'index', recover it. */ + index = rindex; + } val = (FGRAPH_TYPE_RESERVED << FGRAPH_TYPE_SHIFT) | FGRAPH_RET_INDEX; @@ -504,46 +553,53 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func, */ smp_rmb(); + ret_stack = RET_STACK(current, index); + index += FGRAPH_RET_INDEX + 1; + /* The return trace stack is full */ - if (current->curr_ret_stack + FGRAPH_RET_INDEX + 1 >= SHADOW_STACK_MAX_INDEX) { + if (index >= SHADOW_STACK_MAX_INDEX) { atomic_inc(¤t->trace_overrun); return -EBUSY; } calltime = trace_clock_local(); - index = READ_ONCE(current->curr_ret_stack); - ret_stack = RET_STACK(current, index); - index += FGRAPH_RET_INDEX; + /* + * At first, reserve the ret_stack. Beyond this point, any interrupt + * will only overwrite ret_stack[index - 1] by a reserved entry which + * points the previous ret_stack or -1. + */ + current->rsrv_ret_stack = index; + /* And ensure that the following happens after reserved */ + barrier(); - /* ret offset = FGRAPH_RET_INDEX ; type = reserved */ - current->ret_stack[index] = val; + current->ret_stack[index - 1] = val; ret_stack->ret = ret; /* * The unwinders expect curr_ret_stack to point to either zero - * or an index where to find the next ret_stack. Even though the - * ret stack might be bogus, we want to write the ret and the - * index to find the ret_stack before we increment the stack point. - * If an interrupt comes in now before we increment the curr_ret_stack - * it may blow away what we wrote. But that's fine, because the - * index will still be correct (even though the 'ret' won't be). - * What we worry about is the index being correct after we increment - * the curr_ret_stack and before we update that index, as if an - * interrupt comes in and does an unwind stack dump, it will need - * at least a correct index! + * or an index where to find the next ret_stack which has actual ret + * address. Thus we want to write the ret and the index to find the + * ret_stack before we increment the curr_ret_stack. */ barrier(); - current->curr_ret_stack = index + 1; + current->curr_ret_stack = index; /* + * There are two possibilities here. + * - More than one interrupts push/pop their entry between update + * rsrv_ret_stack and curr_ret_stack. In this case, curr_ret_stack + * is already equal to the rsrv_ret_stack and + * current->ret_stack[index] is overwritten by reserved entry which + * points the previous ret_stack. But ret_stack->ret is not. + * - Or, no interrupts push/pop. So current->ret_stack[index - 1] keeps + * its value. * This next barrier is to ensure that an interrupt coming in - * will not corrupt what we are about to write. + * will not overwrite what we are about to write anymore. */ barrier(); - /* Still keep it reserved even if an interrupt came in */ - current->ret_stack[index] = val; + /* Rewrite the entry again in case it was overwritten. */ + current->ret_stack[index - 1] = val; - ret_stack->ret = ret; ret_stack->func = func; ret_stack->calltime = calltime; #ifdef HAVE_FUNCTION_GRAPH_FP_TEST @@ -555,6 +611,14 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func, return index; } +static void __ftrace_commit_pop(int new_index) +{ + current->rsrv_ret_stack = new_index; + /* Ensure the rsrv_ret_stack always updated first */ + barrier(); + current->curr_ret_stack = current->rsrv_ret_stack; +} + /* * Not all archs define MCOUNT_INSN_SIZE which is used to look for direct * functions. But those archs currently don't support direct functions @@ -624,7 +688,7 @@ int function_graph_enter(unsigned long ret, unsigned long func, return 0; out_ret: - current->curr_ret_stack -= FGRAPH_RET_INDEX + 1; + __ftrace_commit_pop(current->rsrv_ret_stack - FGRAPH_RET_INDEX + 1); out: current->curr_ret_depth--; return -EBUSY; @@ -667,7 +731,7 @@ int function_graph_enter_ops(unsigned long ret, unsigned long func, current->curr_ret_stack = save_curr_ret_stack; if (type == FGRAPH_TYPE_RESERVED) { - current->curr_ret_stack -= FGRAPH_RET_INDEX + 1; + __ftrace_commit_pop(current->rsrv_ret_stack - FGRAPH_RET_INDEX + 1); current->curr_ret_depth--; } return -EBUSY; @@ -806,10 +870,10 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs /* * The ftrace_graph_return() may still access the current * ret_stack structure, we need to make sure the update of - * curr_ret_stack is after that. + * curr_ret_stack is after that. __ftrace_commit_pop() does + * barrier() before updating curr_ret_stack. */ - barrier(); - current->curr_ret_stack = index - FGRAPH_RET_INDEX; + __ftrace_commit_pop(index - FGRAPH_RET_INDEX); current->curr_ret_depth--; return ret; @@ -997,6 +1061,7 @@ static int alloc_retstack_tasklist(unsigned long **ret_stack_list) if (t->ret_stack == NULL) { atomic_set(&t->trace_overrun, 0); ret_stack_init_task_vars(ret_stack_list[start]); + t->rsrv_ret_stack = 0; t->curr_ret_stack = 0; t->curr_ret_depth = -1; /* Make sure the tasks see the 0 first: */ @@ -1059,6 +1124,7 @@ graph_init_task(struct task_struct *t, unsigned long *ret_stack) atomic_set(&t->trace_overrun, 0); ret_stack_init_task_vars(ret_stack); t->ftrace_timestamp = 0; + t->rsrv_ret_stack = 0; t->curr_ret_stack = 0; t->curr_ret_depth = -1; /* make curr_ret_stack visible before we add the ret_stack */ @@ -1072,6 +1138,7 @@ graph_init_task(struct task_struct *t, unsigned long *ret_stack) */ void ftrace_graph_init_idle_task(struct task_struct *t, int cpu) { + t->rsrv_ret_stack = 0; t->curr_ret_stack = 0; t->curr_ret_depth = -1; /* @@ -1100,6 +1167,7 @@ void ftrace_graph_init_task(struct task_struct *t) { /* Make sure we do not use the parent ret_stack */ t->ret_stack = NULL; + t->rsrv_ret_stack = 0; t->curr_ret_stack = 0; t->curr_ret_depth = -1; From patchwork Tue Feb 6 15:11:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197496 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1610288dyb; Tue, 6 Feb 2024 07:17:28 -0800 (PST) X-Google-Smtp-Source: AGHT+IHSvHMeUJYqWJri21MYbJEhF5z979M0Nv0AoSnQyUf++XNJQ3BbTx8lQ09tS+E/Glu2jDu7 X-Received: by 2002:ac8:44da:0:b0:42c:37f2:796e with SMTP id b26-20020ac844da000000b0042c37f2796emr1881027qto.23.1707232648172; Tue, 06 Feb 2024 07:17:28 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707232648; cv=pass; d=google.com; s=arc-20160816; b=eRcdEv4aheSiXf7OQn14N+AxJOYP7oEbRLG9whpR8iksuLQAEnsZoym7NeUl3ZT7pR GwXtcX95zokBZU16DkC5WlIWcLCTExpYusTVDdl4HoliLlQgDne5IJJ6uzOjRsnXaWVt e3EXcjbImAgJ1XqqIR9CHL4Kg/3hVv5n2GPECfe/6yimt33zthCFHPH8wsAyJr4kgvqm J9a4AN8G0u1dEPRIYYBkWAYf7VJboKXpyPmBKMbIpln4dJcThKNS52MYUU492OCO/0V1 iCos+iYNGDM4QaxXMcpgzeJQgo46sHkcxhsjrtXpYo17DtTs++vguJpwR8K4KG1+H1aX H0Wg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=gNjysZumaRGj7wRl3wCUGBAp98rdVBpztEtFKkRLpNA=; fh=GdgJWnbBSAVFfGHPZJXVEBgMhj8i6V/EFtQhpTrXkQA=; b=eVUxcXEJZqIacqRVWVEQSxitirP9GEd9TLPYWGYZkPIp7dqp3brmJTNG676EWpdcvx GuMdmyaCkZ3CgIDjGg8cLOp/8yYFU545BidYsqYnjGdae89MG8yxPvS5zmEg/b9czMcV IeqKc/3PfHBPN93Z/GqFgjEJ9DzXfe5KCdXtyetqsysQuRA2IfaHCIY6v5NxqHISFS+J RiCGzUerQ2aU/+LJ2/GvGIa8O0doP9YTw34YTbOhhhkb5EJCPPF3j9lRxZGyNjPp4enX XTEqsW5ejmrt2ExmtjubV3Qtk3t1MrPxljVZzgGZfVLgq/qmBUBMzd6EWt3lmvKUHfdd lM3w==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=lcStqYFq; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55203-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55203-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=1; AJvYcCVQMq5TExD5SedUAD3bhlfQJP+Ar0Ht437opIAQWkYY0eyOsCcT3UViSFZ4s3EmQe5VF73wmskTRp6hQL84irI96RYUHg== Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id d9-20020ac85d89000000b0042be4e05e20si2524360qtx.28.2024.02.06.07.17.28 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:17:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55203-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=lcStqYFq; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55203-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55203-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id E1BEF1C20F0E for ; Tue, 6 Feb 2024 15:17:27 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id B6D1D1386D2; Tue, 6 Feb 2024 15:11:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="lcStqYFq" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A78791386A3; Tue, 6 Feb 2024 15:11:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232288; cv=none; b=Vgg2YpzAd9D8neyn7yQwhPIaAV2vv9IHnTuuFz01APHEf+z00IcoqYEBxgA42T7bkR/m10V1IfAJmaHujmmkP4tBi7qL10P/5ZSh2qctTAgYrf0CM1WrwGRVTPXpV+nR84KVOmwuxQhiPdu2RyszSJWuSIf9WlpV91DlmDZdZpo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232288; c=relaxed/simple; bh=loWWqL6DZVWbVMu4FDyOcIsVywT7pv1to3y8Tk+anuc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=Svs2A5FP07Sb8hyip7GBGOkClPqHjgbCptfcI3UM3t7vVf3QeN0sO2isKDUnGvKHJd8Ne3zJ652GSFOGYdTh3W13jKbYmx7UMPd6yPCDifb6dRI3MZB3fX5COkSk48Oga+hGX7z4MJinOenZ+hZPb6CXPShdpae0okxRACen5k0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=lcStqYFq; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9FAE2C433C7; Tue, 6 Feb 2024 15:11:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232288; bh=loWWqL6DZVWbVMu4FDyOcIsVywT7pv1to3y8Tk+anuc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lcStqYFqWnRNyGZ4LGLXFbCe+hJv3jp2JtZBKeHB5vi6lXWaf5/Ct2knrOwHb/KQy MdlS8Bsl/jz/Cyxl/AyhcsfdfSsF2ijzzXYfMX6cil9eZOlM0cSwx+9wN6wonaRgcg H1LuNGdT7y/2ss4csI06P2kD3RHqbMs7Yk+LTS+eiG4ea8KDbl7yPfqDRZL4yp7eon AlZ4+tfFkrH0cXYb+mLX9tTCCtdefTR3SabfiLb1q/Nzj2adC3aJydbqNksOqLgTnx C6rMu5p9pStpuQsNG/LsmnTTfESNzuZ/gwXmDAEDkDtHWGXVqAB9hWlFH85gypQOB7 dZ+lb1GdJ9YsA== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 21/36] function_graph: Add selftest for passing local variables Date: Wed, 7 Feb 2024 00:11:22 +0900 Message-Id: <170723228217.502590.6615001674278328094.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790163181239114429 X-GMAIL-MSGID: 1790163181239114429 From: Steven Rostedt (VMware) Add boot up selftest that passes variables from a function entry to a function exit, and make sure that they do get passed around. Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- Changes in v2: - Add reserved size test. - Use pr_*() instead of printk(KERN_*). --- kernel/trace/trace_selftest.c | 169 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 169 insertions(+) diff --git a/kernel/trace/trace_selftest.c b/kernel/trace/trace_selftest.c index f0758afa2f7d..4d86cd4c8c8c 100644 --- a/kernel/trace/trace_selftest.c +++ b/kernel/trace/trace_selftest.c @@ -756,6 +756,173 @@ trace_selftest_startup_function(struct tracer *trace, struct trace_array *tr) #ifdef CONFIG_FUNCTION_GRAPH_TRACER +#ifdef CONFIG_DYNAMIC_FTRACE + +#define BYTE_NUMBER 123 +#define SHORT_NUMBER 12345 +#define WORD_NUMBER 1234567890 +#define LONG_NUMBER 1234567890123456789LL + +static int fgraph_store_size __initdata; +static const char *fgraph_store_type_name __initdata; +static char *fgraph_error_str __initdata; +static char fgraph_error_str_buf[128] __initdata; + +static __init int store_entry(struct ftrace_graph_ent *trace, + struct fgraph_ops *gops) +{ + const char *type = fgraph_store_type_name; + int size = fgraph_store_size; + void *p; + + p = fgraph_reserve_data(gops->idx, size); + if (!p) { + snprintf(fgraph_error_str_buf, sizeof(fgraph_error_str_buf), + "Failed to reserve %s\n", type); + fgraph_error_str = fgraph_error_str_buf; + return 0; + } + + switch (fgraph_store_size) { + case 1: + *(char *)p = BYTE_NUMBER; + break; + case 2: + *(short *)p = SHORT_NUMBER; + break; + case 4: + *(int *)p = WORD_NUMBER; + break; + case 8: + *(long long *)p = LONG_NUMBER; + break; + } + + return 1; +} + +static __init void store_return(struct ftrace_graph_ret *trace, + struct fgraph_ops *gops) +{ + const char *type = fgraph_store_type_name; + long long expect = 0; + long long found = -1; + int size; + char *p; + + p = fgraph_retrieve_data(gops->idx, &size); + if (!p) { + snprintf(fgraph_error_str_buf, sizeof(fgraph_error_str_buf), + "Failed to retrieve %s\n", type); + fgraph_error_str = fgraph_error_str_buf; + return; + } + if (fgraph_store_size > size) { + snprintf(fgraph_error_str_buf, sizeof(fgraph_error_str_buf), + "Retrieved size %d is smaller than expected %d\n", + size, (int)fgraph_store_size); + fgraph_error_str = fgraph_error_str_buf; + return; + } + + switch (fgraph_store_size) { + case 1: + expect = BYTE_NUMBER; + found = *(char *)p; + break; + case 2: + expect = SHORT_NUMBER; + found = *(short *)p; + break; + case 4: + expect = WORD_NUMBER; + found = *(int *)p; + break; + case 8: + expect = LONG_NUMBER; + found = *(long long *)p; + break; + } + + if (found != expect) { + snprintf(fgraph_error_str_buf, sizeof(fgraph_error_str_buf), + "%s returned not %lld but %lld\n", type, expect, found); + fgraph_error_str = fgraph_error_str_buf; + return; + } + fgraph_error_str = NULL; +} + +static struct fgraph_ops store_bytes __initdata = { + .entryfunc = store_entry, + .retfunc = store_return, +}; + +static int __init test_graph_storage_type(const char *name, int size) +{ + char *func_name; + int len; + int ret; + + fgraph_store_type_name = name; + fgraph_store_size = size; + + snprintf(fgraph_error_str_buf, sizeof(fgraph_error_str_buf), + "Failed to execute storage %s\n", name); + fgraph_error_str = fgraph_error_str_buf; + + pr_cont("PASSED\n"); + pr_info("Testing fgraph storage of %d byte%s: ", size, size > 1 ? "s" : ""); + + func_name = "*" __stringify(DYN_FTRACE_TEST_NAME); + len = strlen(func_name); + + ret = ftrace_set_filter(&store_bytes.ops, func_name, len, 1); + if (ret && ret != -ENODEV) { + pr_cont("*Could not set filter* "); + return -1; + } + + ret = register_ftrace_graph(&store_bytes); + if (ret) { + pr_warn("Failed to init store_bytes fgraph tracing\n"); + return -1; + } + + DYN_FTRACE_TEST_NAME(); + + unregister_ftrace_graph(&store_bytes); + + if (fgraph_error_str) { + pr_cont("*** %s ***", fgraph_error_str); + return -1; + } + + return 0; +} +/* Test the storage passed across function_graph entry and return */ +static __init int test_graph_storage(void) +{ + int ret; + + ret = test_graph_storage_type("byte", 1); + if (ret) + return ret; + ret = test_graph_storage_type("short", 2); + if (ret) + return ret; + ret = test_graph_storage_type("word", 4); + if (ret) + return ret; + ret = test_graph_storage_type("long long", 8); + if (ret) + return ret; + return 0; +} +#else +static inline int test_graph_storage(void) { return 0; } +#endif /* CONFIG_DYNAMIC_FTRACE */ + /* Maximum number of functions to trace before diagnosing a hang */ #define GRAPH_MAX_FUNC_TEST 100000000 @@ -913,6 +1080,8 @@ trace_selftest_startup_function_graph(struct tracer *trace, ftrace_set_global_filter(NULL, 0, 1); #endif + ret = test_graph_storage(); + /* Don't test dynamic tracing, the function tracer already did */ out: /* Stop it if we failed */ From patchwork Tue Feb 6 15:11:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197509 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1615485dyb; Tue, 6 Feb 2024 07:25:05 -0800 (PST) X-Google-Smtp-Source: AGHT+IG3NBZDlZo1Ri1Fm7SBjfL8kZmvXL2z5e6+NFpli3zXwp+ATCbhCjqtlUXKvbLXwayuhoAI X-Received: by 2002:a17:902:7842:b0:1d9:e1c4:c426 with SMTP id e2-20020a170902784200b001d9e1c4c426mr1050728pln.22.1707233105202; Tue, 06 Feb 2024 07:25:05 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707233105; cv=pass; d=google.com; s=arc-20160816; b=ZvLnS2JjFNvLRp2/46FB+7821GzgERT8Hng5oTl9OsUUCNaZy0/hPCm4mxpG5frPQo RaVxG9Vs2/gEYVfEl1AxjUgyTsfO70O7OfY2cPRdILv7ql/W8rnzewlLCA+aI0jhSyRb tk8hr3CVG0pnUq6MsLsz05iRwqXYoLCoATEiZpYw7FQPl0ymkh3B1mNLxUuj4P95l1cX FaMxgxmxLwlS/Ci6Zav00E3cU3onRFnWZ25zpEYL12docEAbMexQue2h5lLFK1T6PK8e Vt/hvr/spWY2poa/ftlMgy6V+GWeVjk27Wea9wGG5oPAvioOYo4fXLtUYc8EJ59TUPzU 71kA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=QHUJxe+Z50YFcOgYCCbShXfS84KzSYy/sVtgKoPmQK4=; fh=d7/mBMmFtd3ufplltxceUV7qU9T4ilCCXTt+RvtGZzI=; b=ZzXDYmc7H5BZ9ZQFGl6PXOSzkI+5EYWgtOsIADdwMyEZ/WCg70jm1xi3QHMXjTaakL I1mzBWIZxXPAqOkd8J2s+IldXGFVG61fiIGs3V7Zv2ghjFNXTbw/4qBJSlToTEyvXO72 aHgIQCuC0d8pwuMUDJxWC+HeGl5rNhkiDA4oma4DCgEWTx9QcbxD3LBCMJ5cGT1Gnd6X wLUnOQDKE8bn0Q1P2m6ijH6xODiI2BDItoO1GzMdTtaZ7E0zJEQsed6eXoV12N0TJwz0 Dn4a9GdEaqANbyIAFmBCFz9zoqYBNGWye9sSFCIXCEMNkUYrOZ02Z4pYo8qjHqn1/5L4 HK+A==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=QEUbj9N3; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55204-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55204-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=1; AJvYcCWMUPYpjSBrKtIXLmdp9P0WC2I5YVhAzehCy/5Y9ahzn4KI/UoNjPtsGX7ZYhye6q0ZJ/ClsBHEg2y8sqI5YA+AWj3+Ig== Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id s20-20020a17090330d400b001d7852b985fsi1806724plc.121.2024.02.06.07.25.05 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:25:05 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55204-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=QEUbj9N3; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55204-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55204-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id F368528FA81 for ; Tue, 6 Feb 2024 15:18:03 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 0D17D1350FD; Tue, 6 Feb 2024 15:11:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="QEUbj9N3" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E01BF133408; Tue, 6 Feb 2024 15:11:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232300; cv=none; b=E6kf/Qh/0DUjS9TV5SO+HURhNDE7BTUz+obUSjhWewy3gGK5qnPJLupk+oIGhqr4/KzkWUjx1TKbEKUpKxzZ4Or1HZNDJsyGmxbiehyEP2TUyepkUTpYsI22wxalWyQi5l9C0zzg/O747kvA3H8lC9ePJwj/AYHMLgkoJ/+H9ao= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232300; c=relaxed/simple; bh=3hC4rhP5p1jK+ExsNO3fqp2CwByR21P2ZvF5A6MdLzc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=T7AalnkvH+Kz5GZ93JsOm8RvY5pKRa7ozMQzAxoXYf+4ao+2NHWjh6WlyP11mi/Vhh42Ojs7waY/upL2KuT0Pof+QUZO2IEaC+L4LDlCwqCGOeSIJpfcuQ4MF68BU/r+/jCbOUd1+HuiXXvE1JL4KOOewEBzV5ODjKZjQbI8b4U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=QEUbj9N3; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 21BEBC433C7; Tue, 6 Feb 2024 15:11:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232299; bh=3hC4rhP5p1jK+ExsNO3fqp2CwByR21P2ZvF5A6MdLzc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QEUbj9N30XjDGT9vIDQGvGCLkhQbFjItejfHPkLKjVAA/MVSb5opq7IyWHCoWu2fK V7w3cUmw5A48wtcr0Yj2is9IzyBmg7TGInAkn5gDdM7vLUDk/UfdkD9PrmW20XmUcM u3b+J0VjdA4pLLRafQ1TcZV6kCyiwzfpJdzg65HMmPP3z84tq4Oty/BLg9pY8eMCDd EP+iw99JOyrkL2BCbxBKurpXro9fr76I97+0Ip6R53HxcdE0vZT5FHlEogIn+Q1y/n pE8srt1FaEpTTZ8axfXlu9dzYu3a9nupzQtlq4OmyKnU0Rbs7Kbeuxw3ZapjbmBbcE cHJaKvuuin/oQ== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 22/36] function_graph: Add a new entry handler with parent_ip and ftrace_regs Date: Wed, 7 Feb 2024 00:11:34 +0900 Message-Id: <170723229401.502590.8644663781359457778.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790163660721940466 X-GMAIL-MSGID: 1790163660721940466 From: Masami Hiramatsu (Google) Add a new entry handler to fgraph_ops as 'entryregfunc' which takes parent_ip and ftrace_regs. Note that the 'entryfunc' and 'entryregfunc' are mutual exclusive. You can set only one of them. Signed-off-by: Masami Hiramatsu (Google) --- Changes in v3: - Update for new multiple fgraph. --- arch/arm64/kernel/ftrace.c | 2 + arch/loongarch/kernel/ftrace_dyn.c | 2 + arch/powerpc/kernel/trace/ftrace.c | 2 + arch/powerpc/kernel/trace/ftrace_64_pg.c | 10 ++++--- arch/x86/kernel/ftrace.c | 42 ++++++++++++++++-------------- include/linux/ftrace.h | 19 +++++++++++--- kernel/trace/fgraph.c | 30 +++++++++++++++++---- 7 files changed, 72 insertions(+), 35 deletions(-) diff --git a/arch/arm64/kernel/ftrace.c b/arch/arm64/kernel/ftrace.c index b96740829798..779b975f03f5 100644 --- a/arch/arm64/kernel/ftrace.c +++ b/arch/arm64/kernel/ftrace.c @@ -497,7 +497,7 @@ void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, return; if (!function_graph_enter_ops(*parent, ip, frame_pointer, - (void *)frame_pointer, gops)) + (void *)frame_pointer, fregs, gops)) *parent = (unsigned long)&return_to_handler; ftrace_test_recursion_unlock(bit); diff --git a/arch/loongarch/kernel/ftrace_dyn.c b/arch/loongarch/kernel/ftrace_dyn.c index 81d18b911cc1..45d26c6e6564 100644 --- a/arch/loongarch/kernel/ftrace_dyn.c +++ b/arch/loongarch/kernel/ftrace_dyn.c @@ -250,7 +250,7 @@ void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, old = *parent; - if (!function_graph_enter_ops(old, ip, 0, parent, gops)) + if (!function_graph_enter_ops(old, ip, 0, parent, fregs, gops)) *parent = return_hooker; } #else diff --git a/arch/powerpc/kernel/trace/ftrace.c b/arch/powerpc/kernel/trace/ftrace.c index 4ef8bf480279..eeaaa798f4f9 100644 --- a/arch/powerpc/kernel/trace/ftrace.c +++ b/arch/powerpc/kernel/trace/ftrace.c @@ -423,7 +423,7 @@ void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, if (bit < 0) goto out; - if (!function_graph_enter_ops(parent_ip, ip, 0, (unsigned long *)sp, gops)) + if (!function_graph_enter_ops(parent_ip, ip, 0, (unsigned long *)sp, fregs, gops)) parent_ip = ppc_function_entry(return_to_handler); ftrace_test_recursion_unlock(bit); diff --git a/arch/powerpc/kernel/trace/ftrace_64_pg.c b/arch/powerpc/kernel/trace/ftrace_64_pg.c index 7b85c3b460a3..43f6cfaaf7db 100644 --- a/arch/powerpc/kernel/trace/ftrace_64_pg.c +++ b/arch/powerpc/kernel/trace/ftrace_64_pg.c @@ -795,7 +795,8 @@ int ftrace_disable_ftrace_graph_caller(void) * in current thread info. Return the address we want to divert to. */ static unsigned long -__prepare_ftrace_return(unsigned long parent, unsigned long ip, unsigned long sp) +__prepare_ftrace_return(unsigned long parent, unsigned long ip, unsigned long sp, + struct ftrace_regs *fregs) { unsigned long return_hooker; int bit; @@ -812,7 +813,7 @@ __prepare_ftrace_return(unsigned long parent, unsigned long ip, unsigned long sp return_hooker = ppc_function_entry(return_to_handler); - if (!function_graph_enter(parent, ip, 0, (unsigned long *)sp)) + if (!function_graph_enter_regs(parent, ip, 0, (unsigned long *)sp, fregs)) parent = return_hooker; ftrace_test_recursion_unlock(bit); @@ -824,13 +825,14 @@ __prepare_ftrace_return(unsigned long parent, unsigned long ip, unsigned long sp void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, struct ftrace_ops *op, struct ftrace_regs *fregs) { - fregs->regs.link = __prepare_ftrace_return(parent_ip, ip, fregs->regs.gpr[1]); + fregs->regs.link = __prepare_ftrace_return(parent_ip, ip, + fregs->regs.gpr[1], fregs); } #else unsigned long prepare_ftrace_return(unsigned long parent, unsigned long ip, unsigned long sp) { - return __prepare_ftrace_return(parent, ip, sp); + return __prepare_ftrace_return(parent, ip, sp, NULL); } #endif #endif /* CONFIG_FUNCTION_GRAPH_TRACER */ diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c index 845e29b4254f..0f757e399a96 100644 --- a/arch/x86/kernel/ftrace.c +++ b/arch/x86/kernel/ftrace.c @@ -614,16 +614,8 @@ int ftrace_disable_ftrace_graph_caller(void) } #endif /* CONFIG_DYNAMIC_FTRACE && !CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS */ -/* - * Hook the return address and push it in the stack of return addrs - * in current thread info. - */ -void prepare_ftrace_return(unsigned long ip, unsigned long *parent, - unsigned long frame_pointer) +static inline bool skip_ftrace_return(void) { - unsigned long return_hooker = (unsigned long)&return_to_handler; - int bit; - /* * When resuming from suspend-to-ram, this function can be indirectly * called from early CPU startup code while the CPU is in real mode, @@ -633,13 +625,28 @@ void prepare_ftrace_return(unsigned long ip, unsigned long *parent, * This check isn't as accurate as virt_addr_valid(), but it should be * good enough for this purpose, and it's fast. */ - if (unlikely((long)__builtin_frame_address(0) >= 0)) - return; + if ((long)__builtin_frame_address(0) >= 0) + return true; - if (unlikely(ftrace_graph_is_dead())) - return; + if (ftrace_graph_is_dead()) + return true; + + if (atomic_read(¤t->tracing_graph_pause)) + return true; + return false; +} - if (unlikely(atomic_read(¤t->tracing_graph_pause))) +/* + * Hook the return address and push it in the stack of return addrs + * in current thread info. + */ +void prepare_ftrace_return(unsigned long ip, unsigned long *parent, + unsigned long frame_pointer) +{ + unsigned long return_hooker = (unsigned long)&return_to_handler; + int bit; + + if (unlikely(skip_ftrace_return())) return; bit = ftrace_test_recursion_trylock(ip, *parent); @@ -661,17 +668,14 @@ void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, struct fgraph_ops *gops = container_of(op, struct fgraph_ops, ops); int bit; - if (unlikely(ftrace_graph_is_dead())) - return; - - if (unlikely(atomic_read(¤t->tracing_graph_pause))) + if (unlikely(skip_ftrace_return())) return; bit = ftrace_test_recursion_trylock(ip, *parent); if (bit < 0) return; - if (!function_graph_enter_ops(*parent, ip, 0, parent, gops)) + if (!function_graph_enter_ops(*parent, ip, 0, parent, fregs, gops)) *parent = (unsigned long)&return_to_handler; ftrace_test_recursion_unlock(bit); diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 815e865f46c9..65d4d4b68768 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -1063,6 +1063,11 @@ typedef void (*trace_func_graph_ret_t)(struct ftrace_graph_ret *, typedef int (*trace_func_graph_ent_t)(struct ftrace_graph_ent *, struct fgraph_ops *); /* entry */ +typedef int (*trace_func_graph_regs_ent_t)(unsigned long func, + unsigned long parent_ip, + struct ftrace_regs *fregs, + struct fgraph_ops *); /* entry w/ regs */ + extern int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace, struct fgraph_ops *gops); #ifdef CONFIG_FUNCTION_GRAPH_TRACER @@ -1070,6 +1075,7 @@ extern int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace, struct fgraph struct fgraph_ops { trace_func_graph_ent_t entryfunc; trace_func_graph_ret_t retfunc; + trace_func_graph_regs_ent_t entryregfunc; struct ftrace_ops ops; /* for the hash lists */ void *private; int idx; @@ -1106,13 +1112,20 @@ struct ftrace_ret_stack { extern void return_to_handler(void); extern int -function_graph_enter(unsigned long ret, unsigned long func, - unsigned long frame_pointer, unsigned long *retp); +function_graph_enter_regs(unsigned long ret, unsigned long func, + unsigned long frame_pointer, unsigned long *retp, + struct ftrace_regs *fregs); + +static inline int function_graph_enter(unsigned long ret, unsigned long func, + unsigned long fp, unsigned long *retp) +{ + return function_graph_enter_regs(ret, func, fp, retp, NULL); +} extern int function_graph_enter_ops(unsigned long ret, unsigned long func, unsigned long frame_pointer, unsigned long *retp, - struct fgraph_ops *gops); + struct ftrace_regs *fregs, struct fgraph_ops *gops); struct ftrace_ret_stack * ftrace_graph_get_ret_stack(struct task_struct *task, int idx); diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index 269ea458c57a..459912ca72e0 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -633,9 +633,21 @@ static void __ftrace_commit_pop(int new_index) # define MCOUNT_INSN_SIZE 0 #endif +static inline int call_entry_func(struct ftrace_graph_ent *trace, + unsigned long func, unsigned long ret, + struct ftrace_regs *fregs, + struct fgraph_ops *gops) +{ + if (gops->entryregfunc) + return gops->entryregfunc(func, ret, fregs, gops); + + return gops->entryfunc(trace, gops); +} + /* If the caller does not use ftrace, call this function. */ -int function_graph_enter(unsigned long ret, unsigned long func, - unsigned long frame_pointer, unsigned long *retp) +int function_graph_enter_regs(unsigned long ret, unsigned long func, + unsigned long frame_pointer, unsigned long *retp, + struct ftrace_regs *fregs) { struct ftrace_graph_ent trace; unsigned long bitmap = 0; @@ -670,7 +682,7 @@ int function_graph_enter(unsigned long ret, unsigned long func, save_curr_ret_stack = current->curr_ret_stack; if (ftrace_ops_test(&gops->ops, func, NULL) && - gops->entryfunc(&trace, gops)) + call_entry_func(&trace, func, ret, fregs, gops)) bitmap |= BIT(i); else /* Clear out any saved storage */ @@ -697,6 +709,7 @@ int function_graph_enter(unsigned long ret, unsigned long func, /* This is called from ftrace_graph_func() via ftrace */ int function_graph_enter_ops(unsigned long ret, unsigned long func, unsigned long frame_pointer, unsigned long *retp, + struct ftrace_regs *fregs, struct fgraph_ops *gops) { struct ftrace_graph_ent trace; @@ -721,7 +734,7 @@ int function_graph_enter_ops(unsigned long ret, unsigned long func, trace.func = func; trace.depth = current->curr_ret_depth; save_curr_ret_stack = current->curr_ret_stack; - if (gops->entryfunc(&trace, gops)) { + if (call_entry_func(&trace, func, ret, fregs, gops)) { if (type == FGRAPH_TYPE_RESERVED) set_fgraph_index_bitmap(current, index, BIT(gops->idx)); else @@ -1002,7 +1015,8 @@ void fgraph_init_ops(struct ftrace_ops *dst_ops, struct ftrace_ops *src_ops) { dst_ops->func = ftrace_graph_func; - dst_ops->flags = FTRACE_OPS_FL_PID | FTRACE_OPS_GRAPH_STUB; + dst_ops->flags = FTRACE_OPS_FL_PID | FTRACE_OPS_GRAPH_STUB | + FTRACE_OPS_FL_SAVE_ARGS; #ifdef FTRACE_GRAPH_TRAMP_ADDR dst_ops->trampoline = FTRACE_GRAPH_TRAMP_ADDR; @@ -1248,10 +1262,14 @@ int register_ftrace_graph(struct fgraph_ops *gops) int ret = 0; int i; + if (gops->entryfunc && gops->entryregfunc) + return -EINVAL; + mutex_lock(&ftrace_lock); if (!gops->ops.func) { - gops->ops.flags |= FTRACE_OPS_GRAPH_STUB; + gops->ops.flags |= FTRACE_OPS_GRAPH_STUB | + FTRACE_OPS_FL_SAVE_ARGS; gops->ops.func = ftrace_graph_func; #ifdef FTRACE_GRAPH_TRAMP_ADDR gops->ops.trampoline = FTRACE_GRAPH_TRAMP_ADDR; From patchwork Tue Feb 6 15:11:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197497 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1610999dyb; Tue, 6 Feb 2024 07:18:28 -0800 (PST) X-Google-Smtp-Source: AGHT+IFDI7BEJZUMk2kecWDiTJZON1q7o2gPKmIkjo4JC3MPGQoYV/jE1ijj6aCj82lQUC6lOCqj X-Received: by 2002:a05:6830:14f:b0:6e1:4848:a419 with SMTP id j15-20020a056830014f00b006e14848a419mr2750592otp.35.1707232708743; Tue, 06 Feb 2024 07:18:28 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707232708; cv=pass; d=google.com; s=arc-20160816; b=UoTdxDr9cfotV5dm7T20dRn2hWp+sJcaRU33TGziweNkH+1pa2WnYGf8+WzEUpoj6c FJzrNW6jroIXb9thmLiZhYoIHzj7L5zs05jAfTmXUWXkstVEuoqYUItdq0rF4wG025L4 VeBsUCyV9S/sly+k+Q5Fh8hKD5nd0BsrvuuzyYVATVEK6szbWe6PQPmADPN7LiUkcVHQ NWTl4WkXmf6TM6MEhZDObIz8Z4jxv0nHSqdxJumQRV5wd6Iggm4Q6HqrHOxhUavwOIeB uF3UECmZyXb+zx+3ogB2kwW1def+yKfsDYLzaBnS8o2IM4gmnTcR62fzNlrfgkyAPH6l jHkQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=TDqyn0aoFqAcp449XDwKLDAuvvbLq96saRVGNqCKQzY=; fh=69LbbfigaTpQCQPam9N6gInpLo5u7lnxrnznTaUd/sg=; b=VfCS1ySEpVcdwaoEF4+FmdPkn9NTi790duVA2Gg1Ax1qnxXMUPiFnoGrRCfKDDbO8T EkDsqvAYSpxk9BTfc6usoc5MwGLdrUnSi7PoyZBBnfy8oVBw27OBetqFaV7KZ9YxlQH3 0n7/yLEJzoe/sqY7WsOQxRiix+lnKeWmkKMkmc1rjja/YUqFodA605dNFX9Kt7NWLvSi tKQ7FUcJmSAQ79HX21n0YSn4XO/seaKWNrwa3ooNuAuL4f5QIubxVKnoCKcYitiJtU++ 3ZUIsJFFl8yaeJ5ONPba/bLmdne9fNw1Rc0fA5R9KhSa5ZG1FoK0D2a7lmWyaz4ZmAy3 55vg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="HfLlcw/v"; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55206-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55206-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=1; AJvYcCUFLXWRdcA9F9FhqQ9JSpsrcTCSLKRLqBmmRAciYPm7KdmxA2OLjJS4DGH1o6CLDIqzZ3mivp5V0zspXAVUR8Wfrd1jFw== Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id n12-20020a05622a040c00b0042c36a96268si1322225qtx.473.2024.02.06.07.18.28 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:18:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55206-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="HfLlcw/v"; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55206-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55206-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 77E251C23954 for ; Tue, 6 Feb 2024 15:18:28 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 3954E13A860; Tue, 6 Feb 2024 15:11:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="HfLlcw/v" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A914D13A26F; Tue, 6 Feb 2024 15:11:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232310; cv=none; b=q6oOfE0Go7kwiqFKOxzBfJZodzqXo10dnq+BcVuE0542A5uNRAPmRvYaTCRJvJDW5u7Pugv58GlkqlGLCBCZB83UxgBrjt1f2EMsTMXbeuZRK6Fgp0xNGqEqJHz83Gfvpyaw5N8W/dUGIeSHjvTlgfXHr5hDuYiZFCJ3Qxa429o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232310; c=relaxed/simple; bh=u1Sp9EDLtJjdUxY7aOcPZzg2Alw+OmlWu8uSNJvvkPE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=R6y+eyBXMRTvslrXPFHH4juGSAjgwaaGut77GLNUw8gzZa1zAzBJtqqNsL9oB4LHZYQBptfkXbGWonUVJPaTEMHRn52ovvtzUAaCo8d831Uy6TR1cnxskYjOHBzV4OCHuw9frEofSjH5u3agarLtgEBkI4O8x5v/F0OtSDTFKN0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=HfLlcw/v; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id EACA1C433F1; Tue, 6 Feb 2024 15:11:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232310; bh=u1Sp9EDLtJjdUxY7aOcPZzg2Alw+OmlWu8uSNJvvkPE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HfLlcw/v8IKgVgBYWYpRq8Fg20EtApyFESL8y+YtfOK/QmeWmRA132OtXqU+wCnga N/RhFalCNaTlRaC7L0uPd/EG6Sjb9NKdNFnd/9KHQs95x53VpqOEuDYd4r6aroV89U m1gE4A4Ye6ahnNN9azCBKEMDGEDUKppppAS3uibPdW/hscDTCcTzMnSjsotLhY2Yl4 Twnbo5mLxfXFvMrtN0/9f5RmTx3O6eFRb8Bs6Wk0OrhNLNOEk7SUX/IzULz7pQVn1Q gjboLnUirFA+qHQzQn3ZoNgq/6NGzdrvUlrmVq2iqHPgFSCyg44kw9EplRUnSpjPzN oDsM9p1yg/jtQ== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 23/36] function_graph: Add a new exit handler with parent_ip and ftrace_regs Date: Wed, 7 Feb 2024 00:11:44 +0900 Message-Id: <170723230476.502590.16817817024423790038.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790163244826598897 X-GMAIL-MSGID: 1790163244826598897 From: Masami Hiramatsu (Google) Add a new return handler to fgraph_ops as 'retregfunc' which takes parent_ip and ftrace_regs instead of ftrace_graph_ret. This handler is available only if the arch support CONFIG_HAVE_FUNCTION_GRAPH_FREGS. Note that the 'retfunc' and 'reregfunc' are mutual exclusive. You can set only one of them. Signed-off-by: Masami Hiramatsu (Google) --- Changes in v6: - update to use ftrace_regs_get_return_value() because of reordering patches. Changes in v3: - Update for new multiple fgraph. - Save the return address to instruction pointer in ftrace_regs. --- arch/x86/include/asm/ftrace.h | 2 + include/linux/ftrace.h | 10 +++++- kernel/trace/Kconfig | 5 ++- kernel/trace/fgraph.c | 70 ++++++++++++++++++++++++++++------------- 4 files changed, 63 insertions(+), 24 deletions(-) diff --git a/arch/x86/include/asm/ftrace.h b/arch/x86/include/asm/ftrace.h index c88bf47f46da..a061f8832b20 100644 --- a/arch/x86/include/asm/ftrace.h +++ b/arch/x86/include/asm/ftrace.h @@ -72,6 +72,8 @@ arch_ftrace_get_regs(struct ftrace_regs *fregs) override_function_with_return(&(fregs)->regs) #define ftrace_regs_query_register_offset(name) \ regs_query_register_offset(name) +#define ftrace_regs_get_frame_pointer(fregs) \ + frame_pointer(&(fregs)->regs) struct ftrace_ops; #define ftrace_graph_func ftrace_graph_func diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 65d4d4b68768..da2a23f5a9ed 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -43,7 +43,9 @@ struct dyn_ftrace; char *arch_ftrace_match_adjust(char *str, const char *search); -#ifdef CONFIG_HAVE_FUNCTION_GRAPH_RETVAL +#ifdef CONFIG_HAVE_FUNCTION_GRAPH_FREGS +unsigned long ftrace_return_to_handler(struct ftrace_regs *fregs); +#elif defined(CONFIG_HAVE_FUNCTION_GRAPH_RETVAL) struct fgraph_ret_regs; unsigned long ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs); #else @@ -157,6 +159,7 @@ struct ftrace_regs { #define ftrace_regs_set_instruction_pointer(fregs, ip) do { } while (0) #endif /* CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS */ + static __always_inline struct pt_regs *ftrace_get_regs(struct ftrace_regs *fregs) { if (!fregs) @@ -1067,6 +1070,10 @@ typedef int (*trace_func_graph_regs_ent_t)(unsigned long func, unsigned long parent_ip, struct ftrace_regs *fregs, struct fgraph_ops *); /* entry w/ regs */ +typedef void (*trace_func_graph_regs_ret_t)(unsigned long func, + unsigned long parent_ip, + struct ftrace_regs *, + struct fgraph_ops *); /* return w/ regs */ extern int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace, struct fgraph_ops *gops); @@ -1076,6 +1083,7 @@ struct fgraph_ops { trace_func_graph_ent_t entryfunc; trace_func_graph_ret_t retfunc; trace_func_graph_regs_ent_t entryregfunc; + trace_func_graph_regs_ret_t retregfunc; struct ftrace_ops ops; /* for the hash lists */ void *private; int idx; diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig index 61c541c36596..308b3bec01b1 100644 --- a/kernel/trace/Kconfig +++ b/kernel/trace/Kconfig @@ -34,6 +34,9 @@ config HAVE_FUNCTION_GRAPH_TRACER config HAVE_FUNCTION_GRAPH_RETVAL bool +config HAVE_FUNCTION_GRAPH_FREGS + bool + config HAVE_DYNAMIC_FTRACE bool help @@ -232,7 +235,7 @@ config FUNCTION_GRAPH_TRACER config FUNCTION_GRAPH_RETVAL bool "Kernel Function Graph Return Value" - depends on HAVE_FUNCTION_GRAPH_RETVAL + depends on HAVE_FUNCTION_GRAPH_RETVAL || HAVE_FUNCTION_GRAPH_FREGS depends on FUNCTION_GRAPH_TRACER default n help diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index 459912ca72e0..12e5f108e242 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -752,8 +752,8 @@ int function_graph_enter_ops(unsigned long ret, unsigned long func, /* Retrieve a function return address to the trace stack on thread info.*/ static struct ftrace_ret_stack * -ftrace_pop_return_trace(struct ftrace_graph_ret *trace, unsigned long *ret, - unsigned long frame_pointer, int *index) +ftrace_pop_return_trace(unsigned long *ret, unsigned long frame_pointer, + int *index) { struct ftrace_ret_stack *ret_stack; @@ -798,10 +798,6 @@ ftrace_pop_return_trace(struct ftrace_graph_ret *trace, unsigned long *ret, *index += FGRAPH_RET_INDEX; *ret = ret_stack->ret; - trace->func = ret_stack->func; - trace->calltime = ret_stack->calltime; - trace->overrun = atomic_read(¤t->trace_overrun); - trace->depth = current->curr_ret_depth; /* * We still want to trace interrupts coming in if * max_depth is set to 1. Make sure the decrement is @@ -840,21 +836,42 @@ static struct notifier_block ftrace_suspend_notifier = { /* fgraph_ret_regs is not defined without CONFIG_FUNCTION_GRAPH_RETVAL */ struct fgraph_ret_regs; +static void fgraph_call_retfunc(struct ftrace_regs *fregs, + struct fgraph_ret_regs *ret_regs, + struct ftrace_ret_stack *ret_stack, + struct fgraph_ops *gops) +{ + struct ftrace_graph_ret trace; + + trace.func = ret_stack->func; + trace.calltime = ret_stack->calltime; + trace.overrun = atomic_read(¤t->trace_overrun); + trace.depth = current->curr_ret_depth; + trace.rettime = trace_clock_local(); +#ifdef CONFIG_FUNCTION_GRAPH_RETVAL + if (fregs) + trace.retval = ftrace_regs_get_return_value(fregs); + else + trace.retval = fgraph_ret_regs_return_value(ret_regs); +#endif + gops->retfunc(&trace, gops); +} + /* * Send the trace to the ring-buffer. * @return the original return address. */ -static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs, +static unsigned long __ftrace_return_to_handler(struct ftrace_regs *fregs, + struct fgraph_ret_regs *ret_regs, unsigned long frame_pointer) { struct ftrace_ret_stack *ret_stack; - struct ftrace_graph_ret trace; unsigned long bitmap; unsigned long ret; int index; int i; - ret_stack = ftrace_pop_return_trace(&trace, &ret, frame_pointer, &index); + ret_stack = ftrace_pop_return_trace(&ret, frame_pointer, &index); if (unlikely(!ret_stack)) { ftrace_graph_stop(); @@ -863,10 +880,8 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs return (unsigned long)panic; } - trace.rettime = trace_clock_local(); -#ifdef CONFIG_FUNCTION_GRAPH_RETVAL - trace.retval = fgraph_ret_regs_return_value(ret_regs); -#endif + if (fregs) + ftrace_regs_set_instruction_pointer(fregs, ret); bitmap = get_fgraph_index_bitmap(current, index); for (i = 0; i < FGRAPH_ARRAY_SIZE; i++) { @@ -877,7 +892,10 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs if (gops == &fgraph_stub) continue; - gops->retfunc(&trace, gops); + if (gops->retregfunc) + gops->retregfunc(ret_stack->func, ret, fregs, gops); + else + fgraph_call_retfunc(fregs, ret_regs, ret_stack, gops); } /* @@ -892,20 +910,22 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs return ret; } -/* - * After all architecures have selected HAVE_FUNCTION_GRAPH_RETVAL, we can - * leave only ftrace_return_to_handler(ret_regs). - */ -#ifdef CONFIG_HAVE_FUNCTION_GRAPH_RETVAL +#ifdef CONFIG_HAVE_FUNCTION_GRAPH_FREGS +unsigned long ftrace_return_to_handler(struct ftrace_regs *fregs) +{ + return __ftrace_return_to_handler(fregs, NULL, + ftrace_regs_get_frame_pointer(fregs)); +} +#elif defined(CONFIG_HAVE_FUNCTION_GRAPH_RETVAL) unsigned long ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs) { - return __ftrace_return_to_handler(ret_regs, + return __ftrace_return_to_handler(NULL, ret_regs, fgraph_ret_regs_frame_pointer(ret_regs)); } #else unsigned long ftrace_return_to_handler(unsigned long frame_pointer) { - return __ftrace_return_to_handler(NULL, frame_pointer); + return __ftrace_return_to_handler(NULL, NULL, frame_pointer); } #endif @@ -1262,9 +1282,15 @@ int register_ftrace_graph(struct fgraph_ops *gops) int ret = 0; int i; - if (gops->entryfunc && gops->entryregfunc) + if ((gops->entryfunc && gops->entryregfunc) || + (gops->retfunc && gops->retregfunc)) return -EINVAL; +#ifndef CONFIG_HAVE_FUNCTION_GRAPH_FREGS + if (gops->retregfunc) + return -EOPNOTSUPP; +#endif + mutex_lock(&ftrace_lock); if (!gops->ops.func) { From patchwork Tue Feb 6 15:11:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197518 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1624245dyb; Tue, 6 Feb 2024 07:38:14 -0800 (PST) X-Google-Smtp-Source: AGHT+IHfvnoyJ4aWIvuhUoz8JdPxJF0wWQbhUYfoxzs4uxC/W75qgClIckcIZcjRkMSTF7FFYaYB X-Received: by 2002:a05:6a20:9586:b0:19e:99fd:38c8 with SMTP id iu6-20020a056a20958600b0019e99fd38c8mr2070174pzb.27.1707233894132; Tue, 06 Feb 2024 07:38:14 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707233894; cv=pass; d=google.com; s=arc-20160816; b=xqV44YzJphl4rz6D4mec4m9onzE55/zVqupOsde3ZujtWsIY+p2ICIV49Ik+VkZyWf W0KbG8ORZRx0AIMM0mrxVev8fZ5cmo/GtKNQ3Uvtb4K5vwFNsxXluuertWmftA7NdhuI bbINP53Ft7IVGuGbTBPca4IafJPapsQ9Dce2ypy39xDXqBOUQtvXvzCiGgbWOGgSpo4Q xnq7K15S0wiJ/0+SvFlOtlGrZA353haSxaxaGLtsN0MWQkdMyrRZ7jaWyeg4Btn04r2Y bvB8PifevvuQFo+zUCmG53V/s8ZyM6KWPIlfX6NN3ctlcFPGml/oUxcG4En8pqLIzvGq ic2A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=Tt9CuJVUqBOxHNUV/MUjZNddIcOyjajyY4hbmGAF7II=; fh=MpmRd2ePJoHfkhLjJgCEmOi6R9ya6PVIEVdpekSberA=; b=HNQnkw3f51yjlEjfyrFUKqq+7Htz7Oc+rW8w4MtE134duFyO+C4Ce/x7gveSBSilbX 3lWbie0R5aBykLo4ZVz2+yI57ebwD14YI6IA3cTTHpgDVOm6x+S58HHw5s8z7ZNuF/G+ EvR80qRYjZxXTA0b6F1bxJ1PrJ33fZjDpOm11clqk+xOXuJnCa2blAsMlCgheaDRF4Vi SzNVrcfDm7W05z4P54ov4SjIXVXCpCmfCIr561jVS0Ubxvs/llnnN3nUUbOOG/jZVuYW 2HpuDuUt6LCxvxrdUhP6NPkNPlXIVglInauWsdyTDDpb5M/UmpSpihVK73Jyv0rW1UIF ydqw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=aS27veMw; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55207-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55207-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=1; AJvYcCXN5ePeOAfurMwHH0BOXF0WeDrbuzm6ZM+Pv1SZboPvpZDlKA2OG/nJbu4xOvbztQhoBstf1zPRHrpdG/a97pQq7aDbUg== Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id m6-20020a62f206000000b006e04553a4ccsi1795229pfh.223.2024.02.06.07.38.14 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:38:14 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55207-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=aS27veMw; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55207-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55207-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 85DA6290102 for ; Tue, 6 Feb 2024 15:18:47 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id DEAF813A871; Tue, 6 Feb 2024 15:12:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="aS27veMw" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3D34113398C; Tue, 6 Feb 2024 15:12:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232321; cv=none; b=KmppwIiTOJgbC1lNSAoKfT63NFU3mhb6ndSJhkAUyI4Q9q3hqCc+06y0YD+xdeTeyzPFaaiXW6IQzMiiDhbuxn05Tmw2lrPATlfyTPjcZG1OkiFoYdN0xlqw05dJdsQqeI6fz+Ds139dXVR7JoMkxWHl0c2Xb89GPyW7Mrdw4xY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232321; c=relaxed/simple; bh=9YSxRXR2HSbZYZbflQVAgOR2is0PYbfum3hhAgIy/jA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=DZEK0HtRTebFSpP8olD/sPlkzLTTzIz16VHHnWDqjLl9hJUmolYtijoPpKsQWES+mk+J41SfvvcKDTJrVN2pFpSp8DE3+j1wjLemQEgUeAAWDxNtw7UCEp1wEvn1bt66FR+0ls3MQZ6QGPWPu8M55Mk9RyHwlJNWyNdPKzBiGv8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=aS27veMw; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id D6F1EC433F1; Tue, 6 Feb 2024 15:11:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232321; bh=9YSxRXR2HSbZYZbflQVAgOR2is0PYbfum3hhAgIy/jA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=aS27veMwGRZG8E880jNoZMHZEjVF2UWAuNu7Wr7Nnwu0kBYaVKxyo8YyQcreyAFJ+ crxhYGzmhyfqHSMiXC2PzQfPCLyrDwqo72jfcHlVoHdAgDRMtCq2cpvIFgJ9rhDdFW Eyr/fyJDDLZtKxRckOz4W8fbRdfzMHqI/QYGLU4XY/26FA6LpylexMrReD6xGZ6usC ajSEeTXQxCmgbKxxSS9DTGCrmwAQeaRl7xM8hJe3Llb7cwXemi1qLVT/oF0S7B4GFx FAx4IhwZBl3PZCOOoq4YRmErnR4vn9WYo+fsqqIbjCeH32ncQ9tZ+PXdUHEDRBf0jN BSeHT2kaOAyyQ== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 24/36] x86/ftrace: Enable HAVE_FUNCTION_GRAPH_FREGS Date: Wed, 7 Feb 2024 00:11:56 +0900 Message-Id: <170723231592.502590.12367006830540525214.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790164487953116663 X-GMAIL-MSGID: 1790164487953116663 From: Masami Hiramatsu (Google) Support HAVE_FUNCTION_GRAPH_FREGS on x86-64, which saves ftrace_regs on the stack in ftrace_graph return trampoline so that the callbacks can access registers via ftrace_regs APIs. Note that this only recovers 'rax' and 'rdx' registers because other registers are not used anymore and recovered by caller. 'rax' and 'rdx' will be used for passing the return value. Signed-off-by: Masami Hiramatsu (Google) --- Changes in v3: - Add a comment about rip. Changes in v2: - Save rsp register and drop clearing orig_ax. --- arch/x86/Kconfig | 3 ++- arch/x86/kernel/ftrace_64.S | 37 +++++++++++++++++++++++++++++-------- 2 files changed, 31 insertions(+), 9 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 5edec175b9bf..ccf17d8b6f5f 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -223,7 +223,8 @@ config X86 select HAVE_FAST_GUP select HAVE_FENTRY if X86_64 || DYNAMIC_FTRACE select HAVE_FTRACE_MCOUNT_RECORD - select HAVE_FUNCTION_GRAPH_RETVAL if HAVE_FUNCTION_GRAPH_TRACER + select HAVE_FUNCTION_GRAPH_FREGS if HAVE_DYNAMIC_FTRACE_WITH_ARGS + select HAVE_FUNCTION_GRAPH_RETVAL if !HAVE_DYNAMIC_FTRACE_WITH_ARGS select HAVE_FUNCTION_GRAPH_TRACER if X86_32 || (X86_64 && DYNAMIC_FTRACE) select HAVE_FUNCTION_TRACER select HAVE_GCC_PLUGINS diff --git a/arch/x86/kernel/ftrace_64.S b/arch/x86/kernel/ftrace_64.S index 214f30e9f0c0..8a16f774604e 100644 --- a/arch/x86/kernel/ftrace_64.S +++ b/arch/x86/kernel/ftrace_64.S @@ -348,21 +348,42 @@ STACK_FRAME_NON_STANDARD_FP(__fentry__) SYM_CODE_START(return_to_handler) UNWIND_HINT_UNDEFINED ANNOTATE_NOENDBR - subq $24, %rsp + /* + * Save the registers requires for ftrace_regs; + * rax, rcx, rdx, rdi, rsi, r8, r9 and rbp + */ + subq $(FRAME_SIZE), %rsp + movq %rax, RAX(%rsp) + movq %rcx, RCX(%rsp) + movq %rdx, RDX(%rsp) + movq %rsi, RSI(%rsp) + movq %rdi, RDI(%rsp) + movq %r8, R8(%rsp) + movq %r9, R9(%rsp) + movq %rbp, RBP(%rsp) + /* + * orig_ax is not cleared because it is used for indicating the direct + * trampoline in the fentry. And rip is not set because we don't know + * the correct return address here. + */ + + leaq FRAME_SIZE(%rsp), %rcx + movq %rcx, RSP(%rsp) - /* Save the return values */ - movq %rax, (%rsp) - movq %rdx, 8(%rsp) - movq %rbp, 16(%rsp) movq %rsp, %rdi call ftrace_return_to_handler movq %rax, %rdi - movq 8(%rsp), %rdx - movq (%rsp), %rax - addq $24, %rsp + /* + * Restore only rax and rdx because other registers are not used + * for return value nor callee saved. Caller will reuse/recover it. + */ + movq RDX(%rsp), %rdx + movq RAX(%rsp), %rax + + addq $(FRAME_SIZE), %rsp /* * Jump back to the old return address. This cannot be JMP_NOSPEC rdi * since IBT would demand that contain ENDBR, which simply isn't so for From patchwork Tue Feb 6 15:12:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197527 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1632896dyb; Tue, 6 Feb 2024 07:53:00 -0800 (PST) X-Google-Smtp-Source: AGHT+IEKhvMuyy3U+GmZYttIcsj78WWR9ei9rwr8rvlGchdEnqIFPoBakTiXj2ORo7xA7v2qH4t0 X-Received: by 2002:a17:90a:ac14:b0:296:3cde:98a with SMTP id o20-20020a17090aac1400b002963cde098amr2693492pjq.6.1707234780304; Tue, 06 Feb 2024 07:53:00 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707234780; cv=pass; d=google.com; s=arc-20160816; b=Z/4f7Rw0GD8si/NUPRqgplf9nP6pUMSNaoTadxaVPL1pneVy4yWBaOXEGtOXE0E0gT MSct3rI17vuBO7YnPYx+oaWYM/18zbUt/f3bCgjQ6jfGqOzF1+GBiz25EC+w4QlBuPsX X9kotX1y/kaIAtDmDK3f+XLB9nk8lPQbzAYz+saAoLf/YLqbTyFnPGgXtBOxCZKLJQPA 4GdYMnwgPKwQ/Ox3sVb06iKpkSxJRh/Fb/WLBEfQMMJFUnX2cAx2oN5SAMno2602hvmw FoabE1wA0w3swEwMx/5rx+mmblV4YQ2ErDyLaAjw0Q57f0p2ox9eA9FjlMqdanulxML6 uUNw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=8Mu2TsuozS9r1CBqeCbeYrOfMyk5kgHm+82edYNpeu8=; fh=+ZyRWL0TfKKqkn9fMFe86cFdhxJanqACen++w6OH3nI=; b=aodERzG1VVoQU/lwk199Ze1f8zDGhH5PBpp/fpdujyA/eQwQ8jQtA+WzDMH/VZOBP8 Hj0JKeD+AufNuqt3GeA53MUTl+/eSd30nr9hkIHEKdyyy62J2htvYkex9Coj3xUM9Ah3 zPbYT3AGGfZDuzcDHYDC07REcTlLWTTQL5nY6eLpmpEOyNIlkjj1lqXw0T8tC7fORbuu 4/G724Peha45PeuyZXU+QkM3wJ4zJYtrDEE4tfURwWbM3HwzO/q1dEOh4WCdUM8PGRjV jKRlU/R2FmZrIqeH184+bQYRHs7csTSAp9Yag6Wl0DaM3C63CoE3GriYE5ejzx7ZVlZj Y6sg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=P6btYQVu; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55208-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55208-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=2; AJvYcCVfVBqUjNEcvDrX+mv6uDtDGmVYhw5o0vxr8gFPQXq3J5g284ARfHIxdSRCrFwHDjnhefUwR2Hl7RZQG4d5G/MUjsXCmw== Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [147.75.48.161]) by mx.google.com with ESMTPS id s20-20020a17090ad49400b0029692a93e67si1288120pju.64.2024.02.06.07.52.59 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:53:00 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55208-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=P6btYQVu; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55208-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55208-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 3F6FBB2C51D for ; Tue, 6 Feb 2024 15:19:08 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 6D8B913540D; Tue, 6 Feb 2024 15:12:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="P6btYQVu" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C637213398C; Tue, 6 Feb 2024 15:12:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232332; cv=none; b=igSzdeHqprSiwZaeEPc9ZtQr1Cv5PTKLeKFef3yFxl0VBVOuX8+923XXlPU5PMJPQZRppFLio/lHfPXD08QG8Bp4DmPIJDa6VPfSxiFeW/r3b/gxmDjM7iiEFsTiowUSX8Qu4wJE560NLadvlC6trpG5sZy5EslaeSQORROAFBc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232332; c=relaxed/simple; bh=pj/1MH5zsktfbr6xicMFNOz3SlAoTZ/Owd+OXE2OlP0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=Xr6Vcw3mMpB8+BRb3qNdbwN9W6ea/4SIb3wonqvw61ptsziXf4+4uYDRRK+HRJM5ycFrJnkS0D5fot10MXK4tCgPSGAue9oJFk/Pp0nH4JOS89RIjESlPdjgQBMZSAcfjG2pno+fF3hiZO/dNtHaTA86Oh4hxrvDSr7ijeZVBPo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=P6btYQVu; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id ABCE7C433C7; Tue, 6 Feb 2024 15:12:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232332; bh=pj/1MH5zsktfbr6xicMFNOz3SlAoTZ/Owd+OXE2OlP0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=P6btYQVuzAw+Diz0qciumZ63I7JGd+Yi2YlES8sdS9WQZBn/3UbR6lM86f5wW4jdM INwIK4rTkjc0u09iR0WOU5jgKzNJrx8CYxmIiqZRc7iYI9dSob/HcfBO6/I/7UwpaT HaGMXS6BTVXv3ABk1QpuIlk+FH4SVE2tLW2dl7k/nxhehZu5xMZELFGEWmOBVePmjY NrEd3VgvsD1fO81QIm5KolAuIDVtFu4qYNBRuxYf/U9uqXMXZzHNJw7zwOhMq4akA1 IaubWYcb8uXYYSFmGjTesg//kkkaZARyapMAlRZed/kKh2GDq9t+wfkT5UhAXF75mE +BIQFbgBxBXDQ== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 25/36] arm64: ftrace: Enable HAVE_FUNCTION_GRAPH_FREGS Date: Wed, 7 Feb 2024 00:12:06 +0900 Message-Id: <170723232647.502590.10808588686838195094.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790165416880798756 X-GMAIL-MSGID: 1790165416880798756 From: Masami Hiramatsu (Google) Enable CONFIG_HAVE_FUNCTION_GRAPH_FREGS on arm64. Note that this depends on HAVE_DYNAMIC_FTRACE_WITH_ARGS which is enabled if the compiler supports "-fpatchable-function-entry=2". If not, it continue to use ftrace_ret_regs. Signed-off-by: Masami Hiramatsu (Google) --- Changes in v3: - Newly added. --- arch/arm64/Kconfig | 2 ++ arch/arm64/include/asm/ftrace.h | 6 ++++++ arch/arm64/kernel/entry-ftrace.S | 28 ++++++++++++++++++++++++++++ 3 files changed, 36 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index aa7c1d435139..34becd41ae66 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -194,6 +194,8 @@ config ARM64 select HAVE_DYNAMIC_FTRACE select HAVE_DYNAMIC_FTRACE_WITH_ARGS \ if $(cc-option,-fpatchable-function-entry=2) + select HAVE_FUNCTION_GRAPH_FREGS \ + if HAVE_DYNAMIC_FTRACE_WITH_ARGS select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS \ if DYNAMIC_FTRACE_WITH_ARGS && DYNAMIC_FTRACE_WITH_CALL_OPS select HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS \ diff --git a/arch/arm64/include/asm/ftrace.h b/arch/arm64/include/asm/ftrace.h index ab158196480c..efd5dbf74dd6 100644 --- a/arch/arm64/include/asm/ftrace.h +++ b/arch/arm64/include/asm/ftrace.h @@ -131,6 +131,12 @@ ftrace_regs_set_return_value(struct ftrace_regs *fregs, fregs->regs[0] = ret; } +static __always_inline unsigned long +ftrace_regs_get_frame_pointer(struct ftrace_regs *fregs) +{ + return fregs->fp; +} + static __always_inline void ftrace_override_function_with_return(struct ftrace_regs *fregs) { diff --git a/arch/arm64/kernel/entry-ftrace.S b/arch/arm64/kernel/entry-ftrace.S index f0c16640ef21..d87ccdb9e678 100644 --- a/arch/arm64/kernel/entry-ftrace.S +++ b/arch/arm64/kernel/entry-ftrace.S @@ -328,6 +328,33 @@ SYM_FUNC_END(ftrace_stub_graph) * Run ftrace_return_to_handler() before going back to parent. * @fp is checked against the value passed by ftrace_graph_caller(). */ +#ifdef CONFIG_HAVE_FUNCTION_GRAPH_FREGS +SYM_CODE_START(return_to_handler) + /* save ftrace_regs except for PC */ + sub sp, sp, #FREGS_SIZE + stp x0, x1, [sp, #FREGS_X0] + stp x2, x3, [sp, #FREGS_X2] + stp x4, x5, [sp, #FREGS_X4] + stp x6, x7, [sp, #FREGS_X6] + str x8, [sp, #FREGS_X8] + str x29, [sp, #FREGS_FP] + str x9, [sp, #FREGS_LR] + str x10, [sp, #FREGS_SP] + + mov x0, sp + bl ftrace_return_to_handler // addr = ftrace_return_to_hander(fregs); + mov x30, x0 // restore the original return address + + /* restore return value regs */ + ldp x0, x1, [sp, #FREGS_X0] + ldp x2, x3, [sp, #FREGS_X2] + ldp x4, x5, [sp, #FREGS_X4] + ldp x6, x7, [sp, #FREGS_X6] + add sp, sp, #FREGS_SIZE + + ret +SYM_CODE_END(return_to_handler) +#else /* !CONFIG_HAVE_FUNCTION_GRAPH_FREGS */ SYM_CODE_START(return_to_handler) /* save return value regs */ sub sp, sp, #FGRET_REGS_SIZE @@ -350,4 +377,5 @@ SYM_CODE_START(return_to_handler) ret SYM_CODE_END(return_to_handler) +#endif /* CONFIG_HAVE_FUNCTION_GRAPH_FREGS */ #endif /* CONFIG_FUNCTION_GRAPH_TRACER */ From patchwork Tue Feb 6 15:12:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197498 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1611890dyb; Tue, 6 Feb 2024 07:19:48 -0800 (PST) X-Google-Smtp-Source: AGHT+IHu1mXN7XmFn2DCdJgQFu4IzLcUnVo7/gdb25VCvrvO+Y0ztlxisX4C1Tg9WPvhUUOTTn20 X-Received: by 2002:ac8:65cc:0:b0:42b:e4a1:b993 with SMTP id t12-20020ac865cc000000b0042be4a1b993mr1990392qto.30.1707232788076; Tue, 06 Feb 2024 07:19:48 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707232788; cv=pass; d=google.com; s=arc-20160816; b=RmXpHX0gMYR5vmM1rHgwJrdTYzQ2eKZlpBtgGqQiwYnCh6dzXQX1bOSZTqfKVgKHrx tzoM/fjW+s3XLw7av25gRj01f2zPNL/vNeu8bxxUBh6QVybqUlHyiJne3ogtqNwknFkT SG7d/4gdcxUUaM4qhU6pWk/Po/mFXbEbw68GHsV6N9Oo1VRLCvNJxXZABwYdCLx/TCTB s4Mzp2JWD4soCKtBCwo9gXlYbU4GZRJLmWzay/pK8Ktt7cTf4b63lIgPDAvhUGJyywhU XbLCZ6ZPzhK4QXld9+cgktjIQKRDfbRkt8ZxQhnHbMZw1ZCHxd0LKcWoMLRTAHvY7AEz 3i3A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=8+gs1SB8pT3WAOvPrgb5gbUY1pIfdVfQ9Afbww1iVtQ=; fh=CDyhs54sbql36ZibxD4W2bNdKhZ3Gm4ypTt/SJRgj84=; b=mr5tgzR4qrUHlt+FTIrRMd6O0ymNuKV3WYvtjq8b/Fh9+Eoi1UajEKM2AJinZffo8J R/hyk82D79/niB2zohGPIUrFMiUCJVmBtEGPwq4sHBCg26W6WXyLzrJM/lHfqKcWdBIu fpQsd/W3jwpa5Tv9Qxvi4mdkrsV6cowsVIDKY788E4Bd+L8oDHFccA0vCBi8BggmZJFc k1MMccD2YIrnAbiLaH+gh2Kt7EV+CaZC4CmPcdAx5Ftct+IuGUW9E50RVsImp9Gqn4BF p8zmUvyautStXCY6b+ZU8i14/o3LRKQPcyQ83PDoYkwDRiDgl3VEYf0AiWOd8e6oF7PI aBAw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=qqQlrqOc; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55210-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55210-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=1; AJvYcCVkCYyYpeIbXbsPGXjGVIhaEVCjxI8t+w2IQVoJiBHQo8tWM/qs82YE/t+xnOth2fnuQjjStD7c+zKVWoduzyBfUPtdYA== Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id d14-20020ac85ace000000b0042c2e652de8si2622543qtd.335.2024.02.06.07.19.47 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:19:48 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55210-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=qqQlrqOc; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55210-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55210-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id C786F1C242C7 for ; Tue, 6 Feb 2024 15:19:47 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 7C36213AA3F; Tue, 6 Feb 2024 15:12:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="qqQlrqOc" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 52457133998; Tue, 6 Feb 2024 15:12:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232344; cv=none; b=OzPYczuBEDhWGWMq25AP07YLyMHeM93cNVMtmyYXdrL/0xwsckGKUd/v64j/83EK19ssbUWTAyRdnm/HxBKIA5puC7MO2ZECmCeA4ie4XNo8q93GS3vxQYOdk5uWJpU4K0nOVhxbMwjW0SWPHJXtC1EuOsr06gf+K+n7jQUtu7M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232344; c=relaxed/simple; bh=ZyaGBADclB4HCEQB+TOYvV5WEXyDzVOHprlRpBN2is0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=M09bEkIFwzjC/je3i+Dh0q3FuU/I8oQrP5X8zTDt7Hb1vRvvdDHwZw0z+Syvx9EXnDHkDLzKFmpXhgEv3qhVDxoJu9AnIMO8GNiODkSIsq+GA/baT5y5NdSyly2xg4gI/k+3AWHI6eV77CZVwi1/BJ6gEh++PRGYzT56I8BeHpU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=qqQlrqOc; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 353EDC433C7; Tue, 6 Feb 2024 15:12:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232344; bh=ZyaGBADclB4HCEQB+TOYvV5WEXyDzVOHprlRpBN2is0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qqQlrqOcG8LD4faQWfExTdWWhcE/MAZAej4vWEu208f6U55Gn/TFlJtSNYDAIyOiF w73Wp0GUg0470UjO/boL0YNfCRTpv5FmFFOJt3NSotcduGzAQa5qofdMZ59AGYOVRu mAPChT6rfF5pPverCMNPpFHgUSb7V0gV7ZYIYgebPpIUJuxj7suw66jF9V/uQK/Rbz Yy59tKcCjI+mASGMGrNVj82c1gFrBuEdP5LMROkrTAr9UOw2Si2F0K4ODDryeqjIXj TSsnrRnN+ze5dPC+pvDbcKdEoniP6SEjn3IZaNwWfmW+ZyrvqQg3ghUduziu+8yWLf jZ4cCTtms2Pkg== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 26/36] fprobe: Use ftrace_regs in fprobe entry handler Date: Wed, 7 Feb 2024 00:12:18 +0900 Message-Id: <170723233804.502590.2340080684397116910.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790163327935708559 X-GMAIL-MSGID: 1790163327935708559 From: Masami Hiramatsu (Google) This allows fprobes to be available with CONFIG_DYNAMIC_FTRACE_WITH_ARGS instead of CONFIG_DYNAMIC_FTRACE_WITH_REGS, then we can enable fprobe on arm64. Signed-off-by: Masami Hiramatsu (Google) Acked-by: Florent Revest --- Changes in v6: - Keep using SAVE_REGS flag to avoid breaking bpf kprobe-multi test. --- include/linux/fprobe.h | 2 +- kernel/trace/Kconfig | 3 ++- kernel/trace/bpf_trace.c | 10 +++++++--- kernel/trace/fprobe.c | 3 ++- kernel/trace/trace_fprobe.c | 6 +++++- lib/test_fprobe.c | 4 ++-- samples/fprobe/fprobe_example.c | 2 +- 7 files changed, 20 insertions(+), 10 deletions(-) diff --git a/include/linux/fprobe.h b/include/linux/fprobe.h index 3e03758151f4..36c0595f7b93 100644 --- a/include/linux/fprobe.h +++ b/include/linux/fprobe.h @@ -35,7 +35,7 @@ struct fprobe { int nr_maxactive; int (*entry_handler)(struct fprobe *fp, unsigned long entry_ip, - unsigned long ret_ip, struct pt_regs *regs, + unsigned long ret_ip, struct ftrace_regs *regs, void *entry_data); void (*exit_handler)(struct fprobe *fp, unsigned long entry_ip, unsigned long ret_ip, struct pt_regs *regs, diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig index 308b3bec01b1..805d72ab77c6 100644 --- a/kernel/trace/Kconfig +++ b/kernel/trace/Kconfig @@ -290,7 +290,7 @@ config DYNAMIC_FTRACE_WITH_ARGS config FPROBE bool "Kernel Function Probe (fprobe)" depends on FUNCTION_TRACER - depends on DYNAMIC_FTRACE_WITH_REGS + depends on DYNAMIC_FTRACE_WITH_REGS || DYNAMIC_FTRACE_WITH_ARGS depends on HAVE_RETHOOK select RETHOOK default n @@ -675,6 +675,7 @@ config FPROBE_EVENTS select TRACING select PROBE_EVENTS select DYNAMIC_EVENTS + depends on DYNAMIC_FTRACE_WITH_REGS default y help This allows user to add tracing events on the function entry and diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index 7ac6c52b25eb..2f04777330e0 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -2577,7 +2577,7 @@ static int __init bpf_event_init(void) fs_initcall(bpf_event_init); #endif /* CONFIG_MODULES */ -#ifdef CONFIG_FPROBE +#if defined(CONFIG_FPROBE) && defined(CONFIG_DYNAMIC_FTRACE_WITH_REGS) struct bpf_kprobe_multi_link { struct bpf_link link; struct fprobe fp; @@ -2807,10 +2807,14 @@ kprobe_multi_link_prog_run(struct bpf_kprobe_multi_link *link, static int kprobe_multi_link_handler(struct fprobe *fp, unsigned long fentry_ip, - unsigned long ret_ip, struct pt_regs *regs, + unsigned long ret_ip, struct ftrace_regs *fregs, void *data) { struct bpf_kprobe_multi_link *link; + struct pt_regs *regs = ftrace_get_regs(fregs); + + if (!regs) + return 0; link = container_of(fp, struct bpf_kprobe_multi_link, fp); kprobe_multi_link_prog_run(link, get_entry_ip(fentry_ip), regs); @@ -3084,7 +3088,7 @@ int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr kvfree(cookies); return err; } -#else /* !CONFIG_FPROBE */ +#else /* !CONFIG_FPROBE || !CONFIG_DYNAMIC_FTRACE_WITH_REGS */ int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog) { return -EOPNOTSUPP; diff --git a/kernel/trace/fprobe.c b/kernel/trace/fprobe.c index 6cd2a4e3afb8..a932c3a79e8f 100644 --- a/kernel/trace/fprobe.c +++ b/kernel/trace/fprobe.c @@ -46,7 +46,7 @@ static inline void __fprobe_handler(unsigned long ip, unsigned long parent_ip, } if (fp->entry_handler) - ret = fp->entry_handler(fp, ip, parent_ip, ftrace_get_regs(fregs), entry_data); + ret = fp->entry_handler(fp, ip, parent_ip, fregs, entry_data); /* If entry_handler returns !0, nmissed is not counted. */ if (rh) { @@ -182,6 +182,7 @@ static void fprobe_init(struct fprobe *fp) fp->ops.func = fprobe_kprobe_handler; else fp->ops.func = fprobe_handler; + fp->ops.flags |= FTRACE_OPS_FL_SAVE_REGS; } diff --git a/kernel/trace/trace_fprobe.c b/kernel/trace/trace_fprobe.c index 7d2ddbcfa377..ef6b36fd05ae 100644 --- a/kernel/trace/trace_fprobe.c +++ b/kernel/trace/trace_fprobe.c @@ -320,12 +320,16 @@ NOKPROBE_SYMBOL(fexit_perf_func); #endif /* CONFIG_PERF_EVENTS */ static int fentry_dispatcher(struct fprobe *fp, unsigned long entry_ip, - unsigned long ret_ip, struct pt_regs *regs, + unsigned long ret_ip, struct ftrace_regs *fregs, void *entry_data) { struct trace_fprobe *tf = container_of(fp, struct trace_fprobe, fp); + struct pt_regs *regs = ftrace_get_regs(fregs); int ret = 0; + if (!regs) + return 0; + if (trace_probe_test_flag(&tf->tp, TP_FLAG_TRACE)) fentry_trace_func(tf, entry_ip, regs); #ifdef CONFIG_PERF_EVENTS diff --git a/lib/test_fprobe.c b/lib/test_fprobe.c index 24de0e5ff859..ff607babba18 100644 --- a/lib/test_fprobe.c +++ b/lib/test_fprobe.c @@ -40,7 +40,7 @@ static noinline u32 fprobe_selftest_nest_target(u32 value, u32 (*nest)(u32)) static notrace int fp_entry_handler(struct fprobe *fp, unsigned long ip, unsigned long ret_ip, - struct pt_regs *regs, void *data) + struct ftrace_regs *fregs, void *data) { KUNIT_EXPECT_FALSE(current_test, preemptible()); /* This can be called on the fprobe_selftest_target and the fprobe_selftest_target2 */ @@ -81,7 +81,7 @@ static notrace void fp_exit_handler(struct fprobe *fp, unsigned long ip, static notrace int nest_entry_handler(struct fprobe *fp, unsigned long ip, unsigned long ret_ip, - struct pt_regs *regs, void *data) + struct ftrace_regs *fregs, void *data) { KUNIT_EXPECT_FALSE(current_test, preemptible()); return 0; diff --git a/samples/fprobe/fprobe_example.c b/samples/fprobe/fprobe_example.c index 64e715e7ed11..1545a1aac616 100644 --- a/samples/fprobe/fprobe_example.c +++ b/samples/fprobe/fprobe_example.c @@ -50,7 +50,7 @@ static void show_backtrace(void) static int sample_entry_handler(struct fprobe *fp, unsigned long ip, unsigned long ret_ip, - struct pt_regs *regs, void *data) + struct ftrace_regs *fregs, void *data) { if (use_trace) /* From patchwork Tue Feb 6 15:12:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197499 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1612320dyb; Tue, 6 Feb 2024 07:20:28 -0800 (PST) X-Google-Smtp-Source: AGHT+IECPmS9wz5CwIKtqXh2pzfVITWcsuKPJNlqoeMIQJemlTShmYMIPdTo+MoliZYzJ0Wu+lHo X-Received: by 2002:a05:6808:f93:b0:3bd:3e96:a8ab with SMTP id o19-20020a0568080f9300b003bd3e96a8abmr3662466oiw.53.1707232828547; Tue, 06 Feb 2024 07:20:28 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707232828; cv=pass; d=google.com; s=arc-20160816; b=rXCoHcf3DcLJiYcnr93yyezjG2C0YAF3G+lZh4Cc5mbc88/nnk/7VqWoehupqe88b6 kE3JZXgymF+Ch0dzxp3C4Z5ieKeuDptU2bbMKNPUdbNbfqNRxCYQ/005b5Gl4n0hpV8d Oe3WMvYE0oT4HBFROCpA+6M9jxLKrPm64+QFV5n1u1b251vtZzNeKYW5DnUR6udHwLnZ LyoBQEuwfxpphbdQYyzfl/HMIRMXsFBha3+64ElHs/7dlAJMpJ4Qa51U/VMQhb13Ihmo MIVB64GiUYAOhXweOmCUtibmXr7DfbA67UJEiSUis2JKiZdfGDAtAvc/9kFDN+LybtPt wDJA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=n0pu7KeQlLAxDmTWkRzKw8SR2m4kU1CaRwkge/kYNaA=; fh=zV3ShwKq9+2GPmDRiPtsRN/1JQWEd5a4G/JXKxf8FkU=; b=KAott+NXi4EeOvw1CrardDqaCXWC6g5SVXqrYr+BkTJ+KUKLdUn0yZ8+A5HNnUR5jS LMaPt9CjnRoxZO4WMMsg/9cV89MQwZF41W+Y3CXrSIdAldEQL4oBILzLG5T0CXzvzQtC tH7urX0YtprPnyYwI3l+4I/xh1mIG29+Hth8U+wnoF02tqRNmJSD8XMYfIMyYxzE8ssp ApHx4MXZk4EM5htpfB+K7TeTqKWR4UPG4g3iK7ng8ANqvLRxgWJ4A657c2v6bzXOt3Iq DrwdquNugoU6/EEHC0I6//Qk6T7HMXcplS6nLj2GUFWf928hlFVlkVJlSHSrWUTKG8kq l4JQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=YQ2J4bzr; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55212-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55212-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=1; AJvYcCWC9rAAvENk9uXWSzAPkCHrV9Dw4h8AaKOepSXX47i+/HeVY6mOrYhRMLqTh0MqsZPniaCuuBJmgOHTAhP3pybxmHI0KQ== Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id s10-20020a05622a178a00b0042c2b2aae04si2565834qtk.522.2024.02.06.07.20.28 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:20:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55212-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=YQ2J4bzr; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55212-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55212-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 558481C210F4 for ; Tue, 6 Feb 2024 15:20:28 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 330D713B29A; Tue, 6 Feb 2024 15:12:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="YQ2J4bzr" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7D8DA13AA3D; Tue, 6 Feb 2024 15:12:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232355; cv=none; b=O+43MoFTgRcEKfN49Hivefm/QZO+TvIeM7/02WTgFrvuMZQexVioug/06AVOIevv5/LAP/XKYpLtExUa6xyw1NKOb4DlK6e+6j5G4hBxbjL9hng0TXuBkcAjyKAZJo2ymq/5/KI0tXey3vmDio3U4Yod1TH6sRFBMQkDAnNH1W0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232355; c=relaxed/simple; bh=oE2TjKaWjcozECbXfuGZyZSDSxjmUqSI3Zmamt6z4EY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=KkXmNiGSHeFuQqCDrZQWk5OC0MNxud+S2doDd12Ehg1pNpYt2uedApximFhI2qm531V8bq9LR/MtYwgyqir4vpjcfdhzZ57CVKbP7EGtmccwcNvXgm/oV1R0OtHVzDnhLOCY8BJLhHez+j3G+fJdV6bc5+37ft2Nty6HZkHT/S8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=YQ2J4bzr; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0B455C433F1; Tue, 6 Feb 2024 15:12:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232355; bh=oE2TjKaWjcozECbXfuGZyZSDSxjmUqSI3Zmamt6z4EY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YQ2J4bzr/QQehkg6hwUY+riW3WryYhMYYX++x7V05fnqRp7ZPhkhANcVz3snrt7F1 MfeWdh5wDdQTOc6zLCKjTMu+S2ikLFMApZrXp+nlhylkRreqJRc+U1W0kP8tSp7vrJ O/Axe70MrqUQlARjVloNMEfwf0LkvSIZPuJz8AM/HOTwKOxIzquyhG5h1TQCu2cfIa gzT6oNAuoRVXbZq0SMIBXA1dyG67JJ/J40zOf9jb7XtV+UEMHgO9hwjRaZVC3T32fu Ek13szHY4+KOQjNi/SjgpWieFU2ep1N636DI4nZuexW2Fr6eUGhJQ2dS9+p41zCcXq yvwX66lBq1iHw== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 27/36] fprobe: Use ftrace_regs in fprobe exit handler Date: Wed, 7 Feb 2024 00:12:29 +0900 Message-Id: <170723234951.502590.1183221396077184077.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790163370368434684 X-GMAIL-MSGID: 1790163370368434684 From: Masami Hiramatsu (Google) Change the fprobe exit handler to use ftrace_regs structure instead of pt_regs. This also introduce HAVE_PT_REGS_TO_FTRACE_REGS_CAST which means the ftrace_regs's memory layout is equal to the pt_regs so that those are able to cast. Fprobe introduces a new dependency with that. Signed-off-by: Masami Hiramatsu (Google) --- Changes in v3: - Use ftrace_regs_get_return_value() Changes from previous series: NOTHING, just forward ported. --- arch/loongarch/Kconfig | 1 + arch/s390/Kconfig | 1 + arch/x86/Kconfig | 1 + include/linux/fprobe.h | 2 +- include/linux/ftrace.h | 5 +++++ kernel/trace/Kconfig | 8 ++++++++ kernel/trace/bpf_trace.c | 6 +++++- kernel/trace/fprobe.c | 3 ++- kernel/trace/trace_fprobe.c | 6 +++++- lib/test_fprobe.c | 6 +++--- samples/fprobe/fprobe_example.c | 2 +- 11 files changed, 33 insertions(+), 8 deletions(-) diff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig index 10959e6c3583..23e249254f9a 100644 --- a/arch/loongarch/Kconfig +++ b/arch/loongarch/Kconfig @@ -110,6 +110,7 @@ config LOONGARCH select HAVE_DMA_CONTIGUOUS select HAVE_DYNAMIC_FTRACE select HAVE_DYNAMIC_FTRACE_WITH_ARGS + select HAVE_PT_REGS_TO_FTRACE_REGS_CAST select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS select HAVE_DYNAMIC_FTRACE_WITH_REGS select HAVE_EBPF_JIT diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig index fe565f3a3a91..2501fc293a21 100644 --- a/arch/s390/Kconfig +++ b/arch/s390/Kconfig @@ -167,6 +167,7 @@ config S390 select HAVE_DMA_CONTIGUOUS select HAVE_DYNAMIC_FTRACE select HAVE_DYNAMIC_FTRACE_WITH_ARGS + select HAVE_PT_REGS_TO_FTRACE_REGS_CAST select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS select HAVE_DYNAMIC_FTRACE_WITH_REGS select HAVE_EBPF_JIT if HAVE_MARCH_Z196_FEATURES diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index ccf17d8b6f5f..358f2da6c4f8 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -213,6 +213,7 @@ config X86 select HAVE_DYNAMIC_FTRACE select HAVE_DYNAMIC_FTRACE_WITH_REGS select HAVE_DYNAMIC_FTRACE_WITH_ARGS if X86_64 + select HAVE_PT_REGS_TO_FTRACE_REGS_CAST if X86_64 select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS select HAVE_SAMPLE_FTRACE_DIRECT if X86_64 select HAVE_SAMPLE_FTRACE_DIRECT_MULTI if X86_64 diff --git a/include/linux/fprobe.h b/include/linux/fprobe.h index 36c0595f7b93..879a30956009 100644 --- a/include/linux/fprobe.h +++ b/include/linux/fprobe.h @@ -38,7 +38,7 @@ struct fprobe { unsigned long ret_ip, struct ftrace_regs *regs, void *entry_data); void (*exit_handler)(struct fprobe *fp, unsigned long entry_ip, - unsigned long ret_ip, struct pt_regs *regs, + unsigned long ret_ip, struct ftrace_regs *fregs, void *entry_data); }; diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index da2a23f5a9ed..a72a2eaec576 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -159,6 +159,11 @@ struct ftrace_regs { #define ftrace_regs_set_instruction_pointer(fregs, ip) do { } while (0) #endif /* CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS */ +#ifdef CONFIG_HAVE_PT_REGS_TO_FTRACE_REGS_CAST + +static_assert(sizeof(struct pt_regs) == sizeof(struct ftrace_regs)); + +#endif /* CONFIG_HAVE_PT_REGS_TO_FTRACE_REGS_CAST */ static __always_inline struct pt_regs *ftrace_get_regs(struct ftrace_regs *fregs) { diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig index 805d72ab77c6..1a2544712690 100644 --- a/kernel/trace/Kconfig +++ b/kernel/trace/Kconfig @@ -60,6 +60,13 @@ config HAVE_DYNAMIC_FTRACE_WITH_ARGS This allows for use of ftrace_regs_get_argument() and ftrace_regs_get_stack_pointer(). +config HAVE_PT_REGS_TO_FTRACE_REGS_CAST + bool + help + If this is set, the memory layout of the ftrace_regs data structure + is the same as the pt_regs. So the pt_regs is possible to be casted + to ftrace_regs. + config HAVE_DYNAMIC_FTRACE_NO_PATCHABLE bool help @@ -291,6 +298,7 @@ config FPROBE bool "Kernel Function Probe (fprobe)" depends on FUNCTION_TRACER depends on DYNAMIC_FTRACE_WITH_REGS || DYNAMIC_FTRACE_WITH_ARGS + depends on HAVE_PT_REGS_TO_FTRACE_REGS_CAST || !HAVE_DYNAMIC_FTRACE_WITH_ARGS depends on HAVE_RETHOOK select RETHOOK default n diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index 2f04777330e0..115c4166ac00 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -2823,10 +2823,14 @@ kprobe_multi_link_handler(struct fprobe *fp, unsigned long fentry_ip, static void kprobe_multi_link_exit_handler(struct fprobe *fp, unsigned long fentry_ip, - unsigned long ret_ip, struct pt_regs *regs, + unsigned long ret_ip, struct ftrace_regs *fregs, void *data) { struct bpf_kprobe_multi_link *link; + struct pt_regs *regs = ftrace_get_regs(fregs); + + if (!regs) + return; link = container_of(fp, struct bpf_kprobe_multi_link, fp); kprobe_multi_link_prog_run(link, get_entry_ip(fentry_ip), regs); diff --git a/kernel/trace/fprobe.c b/kernel/trace/fprobe.c index a932c3a79e8f..31210423efc3 100644 --- a/kernel/trace/fprobe.c +++ b/kernel/trace/fprobe.c @@ -124,6 +124,7 @@ static void fprobe_exit_handler(struct rethook_node *rh, void *data, { struct fprobe *fp = (struct fprobe *)data; struct fprobe_rethook_node *fpr; + struct ftrace_regs *fregs = (struct ftrace_regs *)regs; int bit; if (!fp || fprobe_disabled(fp)) @@ -141,7 +142,7 @@ static void fprobe_exit_handler(struct rethook_node *rh, void *data, return; } - fp->exit_handler(fp, fpr->entry_ip, ret_ip, regs, + fp->exit_handler(fp, fpr->entry_ip, ret_ip, fregs, fp->entry_data_size ? (void *)fpr->data : NULL); ftrace_test_recursion_unlock(bit); } diff --git a/kernel/trace/trace_fprobe.c b/kernel/trace/trace_fprobe.c index ef6b36fd05ae..3982626c82e6 100644 --- a/kernel/trace/trace_fprobe.c +++ b/kernel/trace/trace_fprobe.c @@ -341,10 +341,14 @@ static int fentry_dispatcher(struct fprobe *fp, unsigned long entry_ip, NOKPROBE_SYMBOL(fentry_dispatcher); static void fexit_dispatcher(struct fprobe *fp, unsigned long entry_ip, - unsigned long ret_ip, struct pt_regs *regs, + unsigned long ret_ip, struct ftrace_regs *fregs, void *entry_data) { struct trace_fprobe *tf = container_of(fp, struct trace_fprobe, fp); + struct pt_regs *regs = ftrace_get_regs(fregs); + + if (!regs) + return; if (trace_probe_test_flag(&tf->tp, TP_FLAG_TRACE)) fexit_trace_func(tf, entry_ip, ret_ip, regs); diff --git a/lib/test_fprobe.c b/lib/test_fprobe.c index ff607babba18..271ce0caeec0 100644 --- a/lib/test_fprobe.c +++ b/lib/test_fprobe.c @@ -59,9 +59,9 @@ static notrace int fp_entry_handler(struct fprobe *fp, unsigned long ip, static notrace void fp_exit_handler(struct fprobe *fp, unsigned long ip, unsigned long ret_ip, - struct pt_regs *regs, void *data) + struct ftrace_regs *fregs, void *data) { - unsigned long ret = regs_return_value(regs); + unsigned long ret = ftrace_regs_get_return_value(fregs); KUNIT_EXPECT_FALSE(current_test, preemptible()); if (ip != target_ip) { @@ -89,7 +89,7 @@ static notrace int nest_entry_handler(struct fprobe *fp, unsigned long ip, static notrace void nest_exit_handler(struct fprobe *fp, unsigned long ip, unsigned long ret_ip, - struct pt_regs *regs, void *data) + struct ftrace_regs *fregs, void *data) { KUNIT_EXPECT_FALSE(current_test, preemptible()); KUNIT_EXPECT_EQ(current_test, ip, target_nest_ip); diff --git a/samples/fprobe/fprobe_example.c b/samples/fprobe/fprobe_example.c index 1545a1aac616..d476d1f07538 100644 --- a/samples/fprobe/fprobe_example.c +++ b/samples/fprobe/fprobe_example.c @@ -67,7 +67,7 @@ static int sample_entry_handler(struct fprobe *fp, unsigned long ip, } static void sample_exit_handler(struct fprobe *fp, unsigned long ip, - unsigned long ret_ip, struct pt_regs *regs, + unsigned long ret_ip, struct ftrace_regs *regs, void *data) { unsigned long rip = ret_ip; From patchwork Tue Feb 6 15:12:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197500 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1612587dyb; Tue, 6 Feb 2024 07:20:48 -0800 (PST) X-Google-Smtp-Source: AGHT+IHdICB6aZwKeUkCcG5ReqS2tXbWQTuPy0RPr80r2lyZNfWDXoSTdO6y/rNcQLTUEggTHZk/ X-Received: by 2002:aa7:c909:0:b0:55e:f51c:8cde with SMTP id b9-20020aa7c909000000b0055ef51c8cdemr2248906edt.39.1707232848305; Tue, 06 Feb 2024 07:20:48 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707232848; cv=pass; d=google.com; s=arc-20160816; b=DSADVaQkMWqfTocf084V0eHpCoKElN1GiP+pq5qU+P0s3vRtm+n+DF02zC/uEf+IA8 egYPwvais733jsvYY9NcxFb2E07HjIPsVfdDmUROKyNaaTiurqOZUG8/4woSN1BqznYm lpCW7NWnHv2S9zzCCx2slTY/NrnOXoWxcLbH5F2/IZMwTBDVKGFsLHdX+CSLGxP4gnqP Bm6DnnlyYyg9EKKcMmn8QmHext0h/BPYx8C78C2o+GdpxsXicYwTFV9pndOkLGU8EA9u q66uspDagqPMUuIXnkZl5qW4btsInAKmH4tW5NH5jxQAR2awJfTTUBx3ckT8TAX0Zwhs 6FvA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=6CvyDT2/9dgk4JdmyncNzCvoFxaDuIj66p5rM2LDAl8=; fh=mIrsM//rfcPHIwg1VLg5BOvbmTpCK3t+e3X9uEuWrZ8=; b=sLLKPGZdjkCA/AHtoB/8wZTDEH2DhnBLfSnZPMGiEKNWBIV7Iz6Qc5JusuNR/s8BAB T/ne+q2Zqd1sLlXsAYdGq8qzk+iJdAZcjuWwdwN/lbJLUYZD7ANl9Us1uMYRHszO8SwE 9s4vcrcVBXvVjyn0VLV5MM+XlDxuLeBPYewwGJiY3JXMEXhg7eu62FbnFbQSqWvZjb6j 5/m9w5MVTh7PMQOHhXBOSk67GQcW6WoNYtNtm3wvc0ukqo42oYYacWd85UZfu5pQ1n8r 8dpVUzh+XJ7AL+HgbTmPtRJJwsC7usGXAXRa/3CfPZnvgJTgPw/SAZ8CdNGnJlqSGbON J9BA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=kCkrcUek; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55213-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55213-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=1; AJvYcCU+4bMDM5Z16sx5yo+LtWSnk2c4t4jIf/cXe+kmpppcj9Pp8smy64/LqMl2Jn6ZO8rtBYuu/Z5U8vJIw7EIaFcV/VUXGg== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id w12-20020a056402070c00b0055feb285760si1160627edx.331.2024.02.06.07.20.48 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:20:48 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55213-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=kCkrcUek; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55213-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55213-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id E35C31F24E39 for ; Tue, 6 Feb 2024 15:20:47 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id AF2E013B2AB; Tue, 6 Feb 2024 15:12:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="kCkrcUek" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 13432135A46; Tue, 6 Feb 2024 15:12:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232367; cv=none; b=ASvLUZi2/8qRmYa0B9bTMoed13OIpB6IjJGLl0xALKLMT+EjCyQoddq92wGi3oHNX7LO21kGfFvFPufViagAIIc2kY35hrVHwPRXnoouNeqZHrbvfZQvFhoaI/LvpO4/PbGP+Y/kCcEEXvE7S0FuTJIb4mEG80EzyOdeQs227jY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232367; c=relaxed/simple; bh=JC1iyaIU7mWHHBTGXNCiN27YHUuUtROsifUJHNlxJlc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=ZtaymrTQwyQljcdkOL1+oxplOi1MBhzAHrxNCSUWke+v2HTYhD6fF7vRFPF13kItb8zPLxbpR7H2gepT6O1lbZX4ug46/W+Y2nPtOO/9j6e/ONNNF1apZyUtjzJusWGU42pnLtMjXyHjUcCI+jO61cJ+v1OPKLv2LWUMQgXZ/yg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=kCkrcUek; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4B91FC433F1; Tue, 6 Feb 2024 15:12:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232366; bh=JC1iyaIU7mWHHBTGXNCiN27YHUuUtROsifUJHNlxJlc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kCkrcUekzwol6q7xY4KN9mdGIpvgQZ345e7elqG4p2/q3c1f678J6xUrOjXl8/cGa 0J9eUke7xFc7qnEO+0qOm5U7vUnfjW6Cs+0t1lUeGh+UEKyNjfV7/3XjsMy2tuWkut gjq8bm7XyzBfwfjnOJOwdruv/g+i6gDAZR0YFT04LhhnlmT6LSqdwJkOCH/cu9ctXE 209VEndnyrX5qyPrPiZv/S9s7eI9m4yObjI2CwKrSzdoiou1PutWHSpXWu/P6q9fg1 8irylQvdCKrlcRr5Yy21UQEjq+6D9j8rV4WLzDUVfLiEdNWcQFhqlsZ9SRlCVWxhNN 5oJyhhvkrosvA== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 28/36] tracing: Add ftrace_partial_regs() for converting ftrace_regs to pt_regs Date: Wed, 7 Feb 2024 00:12:40 +0900 Message-Id: <170723236068.502590.9568421023325291255.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790163391050455008 X-GMAIL-MSGID: 1790163391050455008 From: Masami Hiramatsu (Google) Add ftrace_partial_regs() which converts the ftrace_regs to pt_regs. If the architecture defines its own ftrace_regs, this copies partial registers to pt_regs and returns it. If not, ftrace_regs is the same as pt_regs and ftrace_partial_regs() will return ftrace_regs::regs. Signed-off-by: Masami Hiramatsu (Google) Acked-by: Florent Revest --- Changes from previous series: NOTHING, just forward ported. --- arch/arm64/include/asm/ftrace.h | 11 +++++++++++ include/linux/ftrace.h | 17 +++++++++++++++++ 2 files changed, 28 insertions(+) diff --git a/arch/arm64/include/asm/ftrace.h b/arch/arm64/include/asm/ftrace.h index efd5dbf74dd6..31051fa2b4d9 100644 --- a/arch/arm64/include/asm/ftrace.h +++ b/arch/arm64/include/asm/ftrace.h @@ -143,6 +143,17 @@ ftrace_override_function_with_return(struct ftrace_regs *fregs) fregs->pc = fregs->lr; } +static __always_inline struct pt_regs * +ftrace_partial_regs(const struct ftrace_regs *fregs, struct pt_regs *regs) +{ + memcpy(regs->regs, fregs->regs, sizeof(u64) * 9); + regs->sp = fregs->sp; + regs->pc = fregs->pc; + regs->regs[29] = fregs->fp; + regs->regs[30] = fregs->lr; + return regs; +} + int ftrace_regs_query_register_offset(const char *name); int ftrace_init_nop(struct module *mod, struct dyn_ftrace *rec); diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index a72a2eaec576..515ec804d605 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -173,6 +173,23 @@ static __always_inline struct pt_regs *ftrace_get_regs(struct ftrace_regs *fregs return arch_ftrace_get_regs(fregs); } +#if !defined(CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS) || \ + defined(CONFIG_HAVE_PT_REGS_TO_FTRACE_REGS_CAST) + +static __always_inline struct pt_regs * +ftrace_partial_regs(struct ftrace_regs *fregs, struct pt_regs *regs) +{ + /* + * If CONFIG_HAVE_PT_REGS_TO_FTRACE_REGS_CAST=y, ftrace_regs memory + * layout is the same as pt_regs. So always returns that address. + * Since arch_ftrace_get_regs() will check some members and may return + * NULL, we can not use it. + */ + return &fregs->regs; +} + +#endif /* !CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS || CONFIG_HAVE_PT_REGS_TO_FTRACE_REGS_CAST */ + /* * When true, the ftrace_regs_{get,set}_*() functions may be used on fregs. * Note: this can be true even when ftrace_get_regs() cannot provide a pt_regs. From patchwork Tue Feb 6 15:12:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197501 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1612798dyb; Tue, 6 Feb 2024 07:21:07 -0800 (PST) X-Google-Smtp-Source: AGHT+IHKnJJaDfpR9trkyioaQ8mVYnk3ldE5TB8Mt+gKXmx5e1FYQk8MBNODfNrO+6E5OCc7RQlR X-Received: by 2002:ac2:494e:0:b0:510:206b:e94a with SMTP id o14-20020ac2494e000000b00510206be94amr1899730lfi.53.1707232867665; Tue, 06 Feb 2024 07:21:07 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707232867; cv=pass; d=google.com; s=arc-20160816; b=u7V/0jcLqXQQNWIXsu6sOUk84HK+H4/2dJWAjW7xJcGer3JibFcC8Xd9JasDQZoK16 sKp6Z6eobS8pXATyxLbmFoK6OGU7fP/oZYpBgABQkuUVeHiL5WgfXxj72zhPclQ82+Mu SIjFUim1QBrI4XiYOFY71cVgG7kfwoA7olrzRljHEypC0/Hn+T8O+7gC+k4wR4vwNFiI lm/g2wHMEPeZwNU7R5nt8cdPbnKumqZ9WoGtT4Ydie5/PXQmYiioaDL4BAYT+v9Rdx+s wAcab89em1QAAJX6aVsZU0tglY2CkoF25DwFc4RsBWsiO+spgazCHeiL+MDXMY22Bqfk 70Dw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=ir4WPKEo1uj5mAwr2FfGPsGLcN7vjjSwUuV4xMd+sFc=; fh=WQC+eTuXYScXvHDXMn6cdqKn+gbe1IQZ+fTNaq5uwgo=; b=kHfOR6TRdmSmfmXSDefhWiOeVzrbmf9+muoXksSowbVJLruvfVatWuap9uPDlJ2k7J U1EAN73jfB9ZxuaZBso7nMTaelgIViaFaYVWVW3zAVOj5JpohQXxenSIk4PkG5MYHfI9 UIe9E5R89kxOO9k7QjZ5qV9kOKQo7UAH4e7nvfdpBhtPSQSpqC17shzV7Z93/CxadCCE b+Xd91qaQGQYGBim5HuJcWqEOuZ7ezTykmULKlBTgJxSTCsFNvifJYx+zNmh5C91vV2K SVa3/O471Hr3nAY0OeGYel2i9ADjlKREW/yIX1V7/yR5uJSJ/xM/PobeBWL1SA7zraTo 18EQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=qm+N18iV; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55214-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55214-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=1; AJvYcCVJU2edcE0empXbFElX3uEC7b1HvQrArsNSbN4TbWTiTP5bJz1Md7ulS49J3bCIazwCR5iTFuQNrPG8Ma0EAQ0OFGXbZA== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id d15-20020a170906040f00b00a384e9cf2ebsi157141eja.442.2024.02.06.07.21.07 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:21:07 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55214-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=qm+N18iV; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55214-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55214-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 1D2741F25DD4 for ; Tue, 6 Feb 2024 15:21:07 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id ED04913B2BA; Tue, 6 Feb 2024 15:13:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="qm+N18iV" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EFED538D; Tue, 6 Feb 2024 15:12:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232378; cv=none; b=tzttVLvAuYuGnbERYXDHqdrt4+DBalss4+nrTcNvixsCNdMTm4gE+eIxgAdzysnR55IRYIY3MZrvfIx+PoU32Ohk9dLZ2XfSTMS9FbuYrYHoeYnsQjr5rbDY4GGxp0guMZWxx5geuM140qNaIiPrl1eIhsgHs6wGn4sAVt/BQe8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232378; c=relaxed/simple; bh=qJEnuaEF85mMscwISTh2bfAlWJQw7icaTrOTI/yWDOc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=PtZEEvO3FgGm7aGDaHrnVxmuMNxxs1Dfy3JtxMGN/Cvb0FbF0pSSlNjoOHnzcmBnqccXkshmVcSg2Uww2zoKtFI8Es4rf1fc6d42Lk6QvlLwDxk76aUI2QoYCXEvFV6/MYM9k16vxd/4O3UFdSHxafzfiaEEDLZ6ayuGngE6Kko= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=qm+N18iV; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 738FDC433C7; Tue, 6 Feb 2024 15:12:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232377; bh=qJEnuaEF85mMscwISTh2bfAlWJQw7icaTrOTI/yWDOc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qm+N18iVBoU6jIJL0VFunI/YrUl/m+ZazApAhSi10Mf/9L+pK4IVo8+5+78GcAIOQ CNfadROb9YNdMqCk5A5+KBT6HWo7E2M995GvB8xSUugNrtUCos5qGNAzD4cr1fKQsp oovVgdGJ6aoutvpmB/tEC62kBjhbjV4ZMnb/AVIx6ojud5LY7PKYXAx98huni2lCCr 0GFEv6IK5V6agMJsTSrVrGB0ztJyNZXgDtXZvoxwZNNNkYPoY4B09TWp7kwhFSFNRC Wxl8uliejRoO9D2ZcG5oSpjrB1acKwqxANz57iwNu8Ndq5PhXIiqyiK2LylkXFQMIX ACCzm0sL1k4Iw== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 29/36] tracing: Add ftrace_fill_perf_regs() for perf event Date: Wed, 7 Feb 2024 00:12:52 +0900 Message-Id: <170723237204.502590.7676714109964389100.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790163411569354424 X-GMAIL-MSGID: 1790163411569354424 From: Masami Hiramatsu (Google) Add ftrace_fill_perf_regs() which should be compatible with the perf_fetch_caller_regs(). In other words, the pt_regs returned from the ftrace_fill_perf_regs() must satisfy 'user_mode(regs) == false' and can be used for stack tracing. Signed-off-by: Masami Hiramatsu (Google) --- Changes from previous series: NOTHING, just forward ported. --- arch/arm64/include/asm/ftrace.h | 7 +++++++ arch/powerpc/include/asm/ftrace.h | 7 +++++++ arch/s390/include/asm/ftrace.h | 5 +++++ arch/x86/include/asm/ftrace.h | 7 +++++++ include/linux/ftrace.h | 31 +++++++++++++++++++++++++++++++ 5 files changed, 57 insertions(+) diff --git a/arch/arm64/include/asm/ftrace.h b/arch/arm64/include/asm/ftrace.h index 31051fa2b4d9..c1921bdf760b 100644 --- a/arch/arm64/include/asm/ftrace.h +++ b/arch/arm64/include/asm/ftrace.h @@ -154,6 +154,13 @@ ftrace_partial_regs(const struct ftrace_regs *fregs, struct pt_regs *regs) return regs; } +#define arch_ftrace_fill_perf_regs(fregs, _regs) do { \ + (_regs)->pc = (fregs)->pc; \ + (_regs)->regs[29] = (fregs)->fp; \ + (_regs)->sp = (fregs)->sp; \ + (_regs)->pstate = PSR_MODE_EL1h; \ + } while (0) + int ftrace_regs_query_register_offset(const char *name); int ftrace_init_nop(struct module *mod, struct dyn_ftrace *rec); diff --git a/arch/powerpc/include/asm/ftrace.h b/arch/powerpc/include/asm/ftrace.h index ce2b65cd4607..773e9011cff1 100644 --- a/arch/powerpc/include/asm/ftrace.h +++ b/arch/powerpc/include/asm/ftrace.h @@ -52,6 +52,13 @@ static __always_inline struct pt_regs *arch_ftrace_get_regs(struct ftrace_regs * return fregs->regs.msr ? &fregs->regs : NULL; } +#define arch_ftrace_fill_perf_regs(fregs, _regs) do { \ + (_regs)->result = 0; \ + (_regs)->nip = (fregs)->regs.nip; \ + (_regs)->gpr[1] = (fregs)->regs.gpr[1]; \ + asm volatile("mfmsr %0" : "=r" ((_regs)->msr)); \ + } while (0) + static __always_inline void ftrace_regs_set_instruction_pointer(struct ftrace_regs *fregs, unsigned long ip) diff --git a/arch/s390/include/asm/ftrace.h b/arch/s390/include/asm/ftrace.h index 01e775c98425..c2a269c1617c 100644 --- a/arch/s390/include/asm/ftrace.h +++ b/arch/s390/include/asm/ftrace.h @@ -97,6 +97,11 @@ ftrace_regs_set_instruction_pointer(struct ftrace_regs *fregs, #define ftrace_regs_query_register_offset(name) \ regs_query_register_offset(name) +#define arch_ftrace_fill_perf_regs(fregs, _regs) do { \ + (_regs)->psw.addr = (fregs)->regs.psw.addr; \ + (_regs)->gprs[15] = (fregs)->regs.gprs[15]; \ + } while (0) + #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS /* * When an ftrace registered caller is tracing a function that is diff --git a/arch/x86/include/asm/ftrace.h b/arch/x86/include/asm/ftrace.h index a061f8832b20..2e3de45e9746 100644 --- a/arch/x86/include/asm/ftrace.h +++ b/arch/x86/include/asm/ftrace.h @@ -54,6 +54,13 @@ arch_ftrace_get_regs(struct ftrace_regs *fregs) return &fregs->regs; } +#define arch_ftrace_fill_perf_regs(fregs, _regs) do { \ + (_regs)->ip = (fregs)->regs.ip; \ + (_regs)->sp = (fregs)->regs.sp; \ + (_regs)->cs = __KERNEL_CS; \ + (_regs)->flags = 0; \ + } while (0) + #define ftrace_regs_set_instruction_pointer(fregs, _ip) \ do { (fregs)->regs.ip = (_ip); } while (0) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 515ec804d605..8150edcf8496 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -190,6 +190,37 @@ ftrace_partial_regs(struct ftrace_regs *fregs, struct pt_regs *regs) #endif /* !CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS || CONFIG_HAVE_PT_REGS_TO_FTRACE_REGS_CAST */ +#ifdef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS + +/* + * Please define arch dependent pt_regs which compatible to the + * perf_arch_fetch_caller_regs() but based on ftrace_regs. + * This requires + * - user_mode(_regs) returns false (always kernel mode). + * - able to use the _regs for stack trace. + */ +#ifndef arch_ftrace_fill_perf_regs +/* As same as perf_arch_fetch_caller_regs(), do nothing by default */ +#define arch_ftrace_fill_perf_regs(fregs, _regs) do {} while (0) +#endif + +static __always_inline struct pt_regs * +ftrace_fill_perf_regs(struct ftrace_regs *fregs, struct pt_regs *regs) +{ + arch_ftrace_fill_perf_regs(fregs, regs); + return regs; +} + +#else /* !CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS */ + +static __always_inline struct pt_regs * +ftrace_fill_perf_regs(struct ftrace_regs *fregs, struct pt_regs *regs) +{ + return &fregs->regs; +} + +#endif + /* * When true, the ftrace_regs_{get,set}_*() functions may be used on fregs. * Note: this can be true even when ftrace_get_regs() cannot provide a pt_regs. From patchwork Tue Feb 6 15:13:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197502 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1613028dyb; Tue, 6 Feb 2024 07:21:28 -0800 (PST) X-Google-Smtp-Source: AGHT+IEJwhLzwwoHO5kJMTgnWB1sY/kY+vyHpvvZaI8sdGSZwZRcHfSWqSezu7boSAlDJuyS0+dP X-Received: by 2002:ac8:461a:0:b0:42b:f207:fccc with SMTP id p26-20020ac8461a000000b0042bf207fcccmr4021547qtn.19.1707232888192; Tue, 06 Feb 2024 07:21:28 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707232888; cv=pass; d=google.com; s=arc-20160816; b=A/TENlWwA7DXlJbo0OzZIZNZjHpMQ188PfreyDTwS8O2Xe7yLlAAc+i3jHnL11rIVo B55C9bEplm/amt1ESXRdy4mHMW4i5UGpULy5AR2Dt5FzR89dj8Kqe0pSTCR5XHNnKOkj g0favw8WYT2A+XqSJkJDExS4rVQ5MUcpDbIL6PsQ3FdtXrpIYOpdLQkzB0EkWRLMX4lc oKfS2kKE3N7Clk9YUTFUjkJrwtKy662YXYSpBXgzEn+0p1vmEZFwnDTwatljVQRkfhWz yXkQR6Jknm+7j41OkMu22SNkDM4WcCvWGAo1nXw6zQPP9a1vHMjifaVn7VJzjlizSJeW 4pzg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=u1pK4TVBFLCoIZuRdbut2mSBBsi8+UK2X7ay+IVeSUY=; fh=PA6AqB8MC3rQtQ+TSTYTNyOn+gWCOS1dhZstsQ65LAs=; b=MiBb69Sh8DuLfzZe0msVCEa30+maHIPrxX2JoHV5Ol1ysht/utMyVb2KE4AD3V9jSI unLIkPumZozZWR7lw4UJITNjS+l+EM0H+WNQ4cIXxGbDrlbc9PoJ9izQRFbb6R1EQphn pChP9r+EicMDf8Ecpt64KtG+9o8CAj6vW4yKu0SP+y+7XrlQScy2Pae2MORMPEZ6TDyv KRfMjUQeuvq0wRb6GsIPK+lWsNOB1i1U99T7QaA5o0ro2rpHB6IMTJ6Cal4CdrR4ni64 rFCpE1oNk3EX1DbAYZUKD0nZFPMYGZ7pbe1q31yWHkpH/sSnJgJXPZs1qb8uA/VXCHjc WMrg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=oYM96I8i; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55215-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55215-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=1; AJvYcCWOPMBKKBVCGgTgtOLQnY2qzWPSlSVRCOGbpRCks1pxDLjIhzHL1BCQUKucz8Rnx0Ktzx3MMoOXKC+SFGhW7QB+PKfVwA== Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id s2-20020a05622a018200b0042c322f563esi2443421qtw.750.2024.02.06.07.21.28 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:21:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55215-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=oYM96I8i; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55215-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55215-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id E7E7F1C23887 for ; Tue, 6 Feb 2024 15:21:27 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 0B45413B78D; Tue, 6 Feb 2024 15:13:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="oYM96I8i" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 003BF135A67; Tue, 6 Feb 2024 15:13:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232389; cv=none; b=VcFFwlN42d19abJtOxI5RwMsb4MokEgjoTgxvz/ySbcv6KfX4K8R8aKtwfDAADUk11KyZDi/YBAINCrfpZutzQ5pAE6xqDGxJ7iDqQ4BBAI3+U3oY9S9WqQa2cjKMcUho4ZMFij6tNFM0i0AvMJvWVTF7dKnwWsnNgVBuuXOZu8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232389; c=relaxed/simple; bh=o+I60+HLnGs6KfeJTnOl8ndkevTEMUwwEElHTkUskfo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=MeX2PAaB1+kFGnRIVISzUiezWHvRk3x22D83aLpWeb1t+W9drDPuiYyftdFIfccVJ6TKBtxAcRkO2o+dRumWlRWuaKZP+cpjk7/K3IG+WIbz257+Y71VEZDPvse7MdGVMgWxLN8KoKwNn5MeMG4VvfAIpSBs7gew5CWBDdsOr94= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=oYM96I8i; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3FC26C433C7; Tue, 6 Feb 2024 15:13:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232388; bh=o+I60+HLnGs6KfeJTnOl8ndkevTEMUwwEElHTkUskfo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oYM96I8i3ODw8Cv1CucLs9mHDdqE1e7BdS0tHDO8thEOnXQP4yvpX2goiIoExz0Je RUC0W48jfownKphvuLRllKuC/GJa7JW7JmWURzE0EXyLirKdFB2V9Xzlhk+tnWj5eP ax7wx5f/Vrjiq5jItMkkiMpU39yP0NwgHJrvCLo+hTmVa4DhaExAUUVthH8sgPfyN8 dnt1Lgk0WzQPTf4qyJ1NVIuXHAqf28v8TJacDN6Jh3mvMe7FuWhggE0sQH2atDZbxa ThsLc7z6ZsB3s7Q+vj12FZ10nY1wQ7LB1VyqN6B1E3jWhE21q+/xWDSU8MonbCv8DP BosJrM+GFBSYA== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 30/36] tracing/fprobe: Enable fprobe events with CONFIG_DYNAMIC_FTRACE_WITH_ARGS Date: Wed, 7 Feb 2024 00:13:03 +0900 Message-Id: <170723238312.502590.16193570885461043902.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790163433059615254 X-GMAIL-MSGID: 1790163433059615254 From: Masami Hiramatsu (Google) Allow fprobe events to be enabled with CONFIG_DYNAMIC_FTRACE_WITH_ARGS. With this change, fprobe events mostly use ftrace_regs instead of pt_regs. Note that if the arch doesn't enable HAVE_PT_REGS_COMPAT_FTRACE_REGS, fprobe events will not be able to be used from perf. Signed-off-by: Masami Hiramatsu (Google) --- Chagnes in v3: - Use ftrace_regs_get_return_value(). Changes in v2: - Define ftrace_regs_get_kernel_stack_nth() for !CONFIG_HAVE_REGS_AND_STACK_ACCESS_API. Changes from previous series: Update against the new series. --- include/linux/ftrace.h | 17 +++++++++ kernel/trace/Kconfig | 1 - kernel/trace/trace_fprobe.c | 74 ++++++++++++++++++++------------------- kernel/trace/trace_probe_tmpl.h | 2 + 4 files changed, 55 insertions(+), 39 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 8150edcf8496..ad28daa507f7 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -250,6 +250,23 @@ static __always_inline bool ftrace_regs_has_args(struct ftrace_regs *fregs) regs_query_register_offset(name) #endif +#ifdef CONFIG_HAVE_REGS_AND_STACK_ACCESS_API +static __always_inline unsigned long +ftrace_regs_get_kernel_stack_nth(struct ftrace_regs *fregs, unsigned int nth) +{ + unsigned long *stackp; + + stackp = (unsigned long *)ftrace_regs_get_stack_pointer(fregs); + if (((unsigned long)(stackp + nth) & ~(THREAD_SIZE - 1)) == + ((unsigned long)stackp & ~(THREAD_SIZE - 1))) + return *(stackp + nth); + + return 0; +} +#else /* !CONFIG_HAVE_REGS_AND_STACK_ACCESS_API */ +#define ftrace_regs_get_kernel_stack_nth(fregs, nth) (0L) +#endif /* CONFIG_HAVE_REGS_AND_STACK_ACCESS_API */ + typedef void (*ftrace_func_t)(unsigned long ip, unsigned long parent_ip, struct ftrace_ops *op, struct ftrace_regs *fregs); diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig index 1a2544712690..8b15adde1d8f 100644 --- a/kernel/trace/Kconfig +++ b/kernel/trace/Kconfig @@ -683,7 +683,6 @@ config FPROBE_EVENTS select TRACING select PROBE_EVENTS select DYNAMIC_EVENTS - depends on DYNAMIC_FTRACE_WITH_REGS default y help This allows user to add tracing events on the function entry and diff --git a/kernel/trace/trace_fprobe.c b/kernel/trace/trace_fprobe.c index 3982626c82e6..7d2a66135f83 100644 --- a/kernel/trace/trace_fprobe.c +++ b/kernel/trace/trace_fprobe.c @@ -132,7 +132,7 @@ static int process_fetch_insn(struct fetch_insn *code, void *rec, void *dest, void *base) { - struct pt_regs *regs = rec; + struct ftrace_regs *fregs = rec; unsigned long val; int ret; @@ -140,17 +140,17 @@ process_fetch_insn(struct fetch_insn *code, void *rec, void *dest, /* 1st stage: get value from context */ switch (code->op) { case FETCH_OP_STACK: - val = regs_get_kernel_stack_nth(regs, code->param); + val = ftrace_regs_get_kernel_stack_nth(fregs, code->param); break; case FETCH_OP_STACKP: - val = kernel_stack_pointer(regs); + val = ftrace_regs_get_stack_pointer(fregs); break; case FETCH_OP_RETVAL: - val = regs_return_value(regs); + val = ftrace_regs_get_return_value(fregs); break; #ifdef CONFIG_HAVE_FUNCTION_ARG_ACCESS_API case FETCH_OP_ARG: - val = regs_get_kernel_argument(regs, code->param); + val = ftrace_regs_get_argument(fregs, code->param); break; #endif case FETCH_NOP_SYMBOL: /* Ignore a place holder */ @@ -170,7 +170,7 @@ NOKPROBE_SYMBOL(process_fetch_insn) /* function entry handler */ static nokprobe_inline void __fentry_trace_func(struct trace_fprobe *tf, unsigned long entry_ip, - struct pt_regs *regs, + struct ftrace_regs *fregs, struct trace_event_file *trace_file) { struct fentry_trace_entry_head *entry; @@ -184,36 +184,36 @@ __fentry_trace_func(struct trace_fprobe *tf, unsigned long entry_ip, if (trace_trigger_soft_disabled(trace_file)) return; - dsize = __get_data_size(&tf->tp, regs); + dsize = __get_data_size(&tf->tp, fregs); entry = trace_event_buffer_reserve(&fbuffer, trace_file, sizeof(*entry) + tf->tp.size + dsize); if (!entry) return; - fbuffer.regs = regs; + fbuffer.regs = ftrace_get_regs(fregs); entry = fbuffer.entry = ring_buffer_event_data(fbuffer.event); entry->ip = entry_ip; - store_trace_args(&entry[1], &tf->tp, regs, sizeof(*entry), dsize); + store_trace_args(&entry[1], &tf->tp, fregs, sizeof(*entry), dsize); trace_event_buffer_commit(&fbuffer); } static void fentry_trace_func(struct trace_fprobe *tf, unsigned long entry_ip, - struct pt_regs *regs) + struct ftrace_regs *fregs) { struct event_file_link *link; trace_probe_for_each_link_rcu(link, &tf->tp) - __fentry_trace_func(tf, entry_ip, regs, link->file); + __fentry_trace_func(tf, entry_ip, fregs, link->file); } NOKPROBE_SYMBOL(fentry_trace_func); /* Kretprobe handler */ static nokprobe_inline void __fexit_trace_func(struct trace_fprobe *tf, unsigned long entry_ip, - unsigned long ret_ip, struct pt_regs *regs, + unsigned long ret_ip, struct ftrace_regs *fregs, struct trace_event_file *trace_file) { struct fexit_trace_entry_head *entry; @@ -227,60 +227,63 @@ __fexit_trace_func(struct trace_fprobe *tf, unsigned long entry_ip, if (trace_trigger_soft_disabled(trace_file)) return; - dsize = __get_data_size(&tf->tp, regs); + dsize = __get_data_size(&tf->tp, fregs); entry = trace_event_buffer_reserve(&fbuffer, trace_file, sizeof(*entry) + tf->tp.size + dsize); if (!entry) return; - fbuffer.regs = regs; + fbuffer.regs = ftrace_get_regs(fregs); entry = fbuffer.entry = ring_buffer_event_data(fbuffer.event); entry->func = entry_ip; entry->ret_ip = ret_ip; - store_trace_args(&entry[1], &tf->tp, regs, sizeof(*entry), dsize); + store_trace_args(&entry[1], &tf->tp, fregs, sizeof(*entry), dsize); trace_event_buffer_commit(&fbuffer); } static void fexit_trace_func(struct trace_fprobe *tf, unsigned long entry_ip, - unsigned long ret_ip, struct pt_regs *regs) + unsigned long ret_ip, struct ftrace_regs *fregs) { struct event_file_link *link; trace_probe_for_each_link_rcu(link, &tf->tp) - __fexit_trace_func(tf, entry_ip, ret_ip, regs, link->file); + __fexit_trace_func(tf, entry_ip, ret_ip, fregs, link->file); } NOKPROBE_SYMBOL(fexit_trace_func); #ifdef CONFIG_PERF_EVENTS static int fentry_perf_func(struct trace_fprobe *tf, unsigned long entry_ip, - struct pt_regs *regs) + struct ftrace_regs *fregs) { struct trace_event_call *call = trace_probe_event_call(&tf->tp); struct fentry_trace_entry_head *entry; struct hlist_head *head; int size, __size, dsize; + struct pt_regs *regs; int rctx; head = this_cpu_ptr(call->perf_events); if (hlist_empty(head)) return 0; - dsize = __get_data_size(&tf->tp, regs); + dsize = __get_data_size(&tf->tp, fregs); __size = sizeof(*entry) + tf->tp.size + dsize; size = ALIGN(__size + sizeof(u32), sizeof(u64)); size -= sizeof(u32); - entry = perf_trace_buf_alloc(size, NULL, &rctx); + entry = perf_trace_buf_alloc(size, ®s, &rctx); if (!entry) return 0; + regs = ftrace_fill_perf_regs(fregs, regs); + entry->ip = entry_ip; memset(&entry[1], 0, dsize); - store_trace_args(&entry[1], &tf->tp, regs, sizeof(*entry), dsize); + store_trace_args(&entry[1], &tf->tp, fregs, sizeof(*entry), dsize); perf_trace_buf_submit(entry, size, rctx, call->event.type, 1, regs, head, NULL); return 0; @@ -289,30 +292,33 @@ NOKPROBE_SYMBOL(fentry_perf_func); static void fexit_perf_func(struct trace_fprobe *tf, unsigned long entry_ip, - unsigned long ret_ip, struct pt_regs *regs) + unsigned long ret_ip, struct ftrace_regs *fregs) { struct trace_event_call *call = trace_probe_event_call(&tf->tp); struct fexit_trace_entry_head *entry; struct hlist_head *head; int size, __size, dsize; + struct pt_regs *regs; int rctx; head = this_cpu_ptr(call->perf_events); if (hlist_empty(head)) return; - dsize = __get_data_size(&tf->tp, regs); + dsize = __get_data_size(&tf->tp, fregs); __size = sizeof(*entry) + tf->tp.size + dsize; size = ALIGN(__size + sizeof(u32), sizeof(u64)); size -= sizeof(u32); - entry = perf_trace_buf_alloc(size, NULL, &rctx); + entry = perf_trace_buf_alloc(size, ®s, &rctx); if (!entry) return; + regs = ftrace_fill_perf_regs(fregs, regs); + entry->func = entry_ip; entry->ret_ip = ret_ip; - store_trace_args(&entry[1], &tf->tp, regs, sizeof(*entry), dsize); + store_trace_args(&entry[1], &tf->tp, fregs, sizeof(*entry), dsize); perf_trace_buf_submit(entry, size, rctx, call->event.type, 1, regs, head, NULL); } @@ -324,17 +330,14 @@ static int fentry_dispatcher(struct fprobe *fp, unsigned long entry_ip, void *entry_data) { struct trace_fprobe *tf = container_of(fp, struct trace_fprobe, fp); - struct pt_regs *regs = ftrace_get_regs(fregs); int ret = 0; - if (!regs) - return 0; - if (trace_probe_test_flag(&tf->tp, TP_FLAG_TRACE)) - fentry_trace_func(tf, entry_ip, regs); + fentry_trace_func(tf, entry_ip, fregs); + #ifdef CONFIG_PERF_EVENTS if (trace_probe_test_flag(&tf->tp, TP_FLAG_PROFILE)) - ret = fentry_perf_func(tf, entry_ip, regs); + ret = fentry_perf_func(tf, entry_ip, fregs); #endif return ret; } @@ -345,16 +348,13 @@ static void fexit_dispatcher(struct fprobe *fp, unsigned long entry_ip, void *entry_data) { struct trace_fprobe *tf = container_of(fp, struct trace_fprobe, fp); - struct pt_regs *regs = ftrace_get_regs(fregs); - - if (!regs) - return; if (trace_probe_test_flag(&tf->tp, TP_FLAG_TRACE)) - fexit_trace_func(tf, entry_ip, ret_ip, regs); + fexit_trace_func(tf, entry_ip, ret_ip, fregs); + #ifdef CONFIG_PERF_EVENTS if (trace_probe_test_flag(&tf->tp, TP_FLAG_PROFILE)) - fexit_perf_func(tf, entry_ip, ret_ip, regs); + fexit_perf_func(tf, entry_ip, ret_ip, fregs); #endif } NOKPROBE_SYMBOL(fexit_dispatcher); diff --git a/kernel/trace/trace_probe_tmpl.h b/kernel/trace/trace_probe_tmpl.h index 3935b347f874..05445a745a07 100644 --- a/kernel/trace/trace_probe_tmpl.h +++ b/kernel/trace/trace_probe_tmpl.h @@ -232,7 +232,7 @@ process_fetch_insn_bottom(struct fetch_insn *code, unsigned long val, /* Sum up total data length for dynamic arrays (strings) */ static nokprobe_inline int -__get_data_size(struct trace_probe *tp, struct pt_regs *regs) +__get_data_size(struct trace_probe *tp, void *regs) { struct probe_arg *arg; int i, len, ret = 0; From patchwork Tue Feb 6 15:13:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197503 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1613312dyb; Tue, 6 Feb 2024 07:21:50 -0800 (PST) X-Google-Smtp-Source: AGHT+IG1QCHNnwLQPag66BxWrQf5mq4H2S9+cVLPDOPI89GGsYPtt5UgRBVTkefshaPVGMIV/kVa X-Received: by 2002:a17:906:3e4e:b0:a26:c9e6:d5e0 with SMTP id t14-20020a1709063e4e00b00a26c9e6d5e0mr2112752eji.74.1707232910055; Tue, 06 Feb 2024 07:21:50 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707232910; cv=pass; d=google.com; s=arc-20160816; b=FvpJ9G+8KVtBcQ9pJyke79gFyTJ5qItZounYlRBvg4g3pzTnAyBbQfa03v+qFwRlT5 51UjlFfJ1U4oXX2uZL+c/kMf9OIBy7ThkKEZkrsem4SYo4LX2r1zF1EdvF2lzL8TsSFh Y/tteZXPZEcI/bKEJknBGddZMmm76DNUaUAtVCIquwONPayjUpad+mlN4mbafwTsBIuM CMVaMnuOxX9wlF7O5mG4uvJN4bleogafAB31fewjbOf3BmhOFHKRI8TmX3xwUT639kN9 D++nVhrLCdTa0B4W9+kHExD9UjsG1RjNOaIoCGopvlDQnYgSMUSObzd9+YaIYO368iIf 0lUg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=RgldrXg20BQRXBgt27vuOVDsx+75i2iXTYL5CbDce7Y=; fh=vOofjzkWQTysI5NNbUuA9ZzaXLoroZ+VwSjnD+UbKn0=; b=HDJ1i47ZdvFH5SpTsE/PjncLNi5ydBbxj+hNBDRSro1HivnGQ52OALZOWAOe50jeBe ol4L1/+RL8y3lYH12Fi/aeQvz5PbJxAm/6Pudnzu2lwiRQ1GMeRHmZCxq+2ZWYk73rsJ eqGBSj3DcMS+0tMzT3OOAbaoDyB8yODfXLNXpeLU6aBuZBJtx2OZlgT7RK4X3Ff+24ej xupr3TxRtJPeROBrWz4wY1W8BdfrYYeFVt4HlLyJGqzEv+c2w2lu6d8kyG1zJdrwGwQM 2tca/o5eyn+bXe/6G7WE87JTDGeUwDCFoWeYoShGC3EcxpWS/UdlNIe4xmtbj+gqnQ0R Td5Q==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=IMcuttnA; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55216-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55216-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=1; AJvYcCXTIi1QH7WZ+zSaxpzWc+VvU5PuhczVh+ivlJwOAMkYuaktzBMuImGuD1WWWxWco69U80m+NHOILsOXsiZQKYUau6dnvQ== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id h14-20020a170906260e00b00a385325ff78si65300ejc.772.2024.02.06.07.21.49 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:21:50 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55216-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=IMcuttnA; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55216-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55216-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id A525D1F2322B for ; Tue, 6 Feb 2024 15:21:49 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 427DB13BE8A; Tue, 6 Feb 2024 15:13:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="IMcuttnA" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6DB2D131E56; Tue, 6 Feb 2024 15:13:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232400; cv=none; b=qJIAy5e78FzRIGT0ow1Wtj2jpnWx7jv0wG91ymi+4mZB1/3G3rzmtbt7TNV3ehT7jVXQWODGIQK+cVYMcbRtpmmwhEQMURD2S1KQEBwxupe8chbLzvL64u/HN9UAFSw48jRMdOqvpCp+hQbbm6aJdtsDcKgCsRBij1P0JJynyS0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232400; c=relaxed/simple; bh=zgVjHsibf80Ba7/uhgykpeSact93x0IdExGXz3s/bRQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=lumOfqvs0b97SDYGqeTN+9M/17u5vVZf4CmhqHaOaMUfcLXgvGYB5ZK6l9cyltC/RmVFVXzEplzJkARq0mK5LrvlkinF7wDhXro4z1ptVkjGRwptzkQLPPHh3IQChNu4Z3khJbp/C7ibONhJK8yZ6O3ujwJ3nKaCZD7hij3KSUQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=IMcuttnA; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 44FB5C433C7; Tue, 6 Feb 2024 15:13:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232400; bh=zgVjHsibf80Ba7/uhgykpeSact93x0IdExGXz3s/bRQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IMcuttnAwPajr0EAax7Mf2KawlZ7VzkM6WET9CCxRo5RK+nwSxnojiIv8vWCePp+1 QnpcUAjdz5MlOUILH98Q18C4BvpScsR54n3qRk7an+r/nfXQJpjpZyveZ5fcA0NTkk FVAFEDhUNuHRxMHMBfVo44Z5neKS2weXZ1ZhtA6R4IagBgH6JthoWypbas2lUwyG0z dWKvyaHY/aazwVhZ6fWbGmUI5c6caQZgrgPDIXaAHSEaXptjt32pohSTAE/bwqUiua wdwu1yM2l3LYlIw/Nhulrz3ZsURO0XBnJCzvOP+UkV3bachBN20BVYKmaHclt9oCEf o0N5fUoH7vgng== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 31/36] bpf: Enable kprobe_multi feature if CONFIG_FPROBE is enabled Date: Wed, 7 Feb 2024 00:13:13 +0900 Message-Id: <170723239387.502590.7145597542791935266.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790163455765845051 X-GMAIL-MSGID: 1790163455765845051 From: Masami Hiramatsu (Google) Enable kprobe_multi feature if CONFIG_FPROBE is enabled. The pt_regs is converted from ftrace_regs by ftrace_partial_regs(), thus some registers may always returns 0. But it should be enough for function entry (access arguments) and exit (access return value). Signed-off-by: Masami Hiramatsu (Google) Acked-by: Florent Revest --- Changes from previous series: NOTHING, Update against the new series. --- kernel/trace/bpf_trace.c | 22 +++++++++------------- 1 file changed, 9 insertions(+), 13 deletions(-) diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index 115c4166ac00..5ad3c32b1c40 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -2577,7 +2577,7 @@ static int __init bpf_event_init(void) fs_initcall(bpf_event_init); #endif /* CONFIG_MODULES */ -#if defined(CONFIG_FPROBE) && defined(CONFIG_DYNAMIC_FTRACE_WITH_REGS) +#ifdef CONFIG_FPROBE struct bpf_kprobe_multi_link { struct bpf_link link; struct fprobe fp; @@ -2600,6 +2600,8 @@ struct user_syms { char *buf; }; +static DEFINE_PER_CPU(struct pt_regs, bpf_kprobe_multi_pt_regs); + static int copy_user_syms(struct user_syms *us, unsigned long __user *usyms, u32 cnt) { unsigned long __user usymbol; @@ -2777,13 +2779,14 @@ static u64 bpf_kprobe_multi_entry_ip(struct bpf_run_ctx *ctx) static int kprobe_multi_link_prog_run(struct bpf_kprobe_multi_link *link, - unsigned long entry_ip, struct pt_regs *regs) + unsigned long entry_ip, struct ftrace_regs *fregs) { struct bpf_kprobe_multi_run_ctx run_ctx = { .link = link, .entry_ip = entry_ip, }; struct bpf_run_ctx *old_run_ctx; + struct pt_regs *regs; int err; if (unlikely(__this_cpu_inc_return(bpf_prog_active) != 1)) { @@ -2794,6 +2797,7 @@ kprobe_multi_link_prog_run(struct bpf_kprobe_multi_link *link, migrate_disable(); rcu_read_lock(); + regs = ftrace_partial_regs(fregs, this_cpu_ptr(&bpf_kprobe_multi_pt_regs)); old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx); err = bpf_prog_run(link->link.prog, regs); bpf_reset_run_ctx(old_run_ctx); @@ -2811,13 +2815,9 @@ kprobe_multi_link_handler(struct fprobe *fp, unsigned long fentry_ip, void *data) { struct bpf_kprobe_multi_link *link; - struct pt_regs *regs = ftrace_get_regs(fregs); - - if (!regs) - return 0; link = container_of(fp, struct bpf_kprobe_multi_link, fp); - kprobe_multi_link_prog_run(link, get_entry_ip(fentry_ip), regs); + kprobe_multi_link_prog_run(link, get_entry_ip(fentry_ip), fregs); return 0; } @@ -2827,13 +2827,9 @@ kprobe_multi_link_exit_handler(struct fprobe *fp, unsigned long fentry_ip, void *data) { struct bpf_kprobe_multi_link *link; - struct pt_regs *regs = ftrace_get_regs(fregs); - - if (!regs) - return; link = container_of(fp, struct bpf_kprobe_multi_link, fp); - kprobe_multi_link_prog_run(link, get_entry_ip(fentry_ip), regs); + kprobe_multi_link_prog_run(link, get_entry_ip(fentry_ip), fregs); } static int symbols_cmp_r(const void *a, const void *b, const void *priv) @@ -3092,7 +3088,7 @@ int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr kvfree(cookies); return err; } -#else /* !CONFIG_FPROBE || !CONFIG_DYNAMIC_FTRACE_WITH_REGS */ +#else /* !CONFIG_FPROBE */ int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog) { return -EOPNOTSUPP; From patchwork Tue Feb 6 15:13:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197524 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1627962dyb; Tue, 6 Feb 2024 07:44:19 -0800 (PST) X-Google-Smtp-Source: AGHT+IEeOaSho19051J+wvXB/L7xUNwFrQQM9BBmFqXJ2fvmYo0zzNjT8Hwjb81aBu23GSLScXHB X-Received: by 2002:aa7:8c5a:0:b0:6e0:3f63:ed64 with SMTP id e26-20020aa78c5a000000b006e03f63ed64mr2875609pfd.34.1707234259111; Tue, 06 Feb 2024 07:44:19 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707234259; cv=pass; d=google.com; s=arc-20160816; b=eC3i6NqYgkylLOHQoIZcuhmESvkAYGACkwh8eEiwwtwQ069zkfYEGmCj7DzktiHFcF jqYr4WBq/dRGSswtfKkINGJOSDuWY8YpplCS2q+R+NibZnwNdfv0sEHi71yAVuz7bh+w KUNTxCUSxW7L9HNJ7e4zZMBYt0/joUBBTf51KH9Le4IuJkuMH8OXIQlRyjn9OQF5GdR4 USZMjDNQk7xIuMILnHTmSM4mVPx4N+12SNVu99KqxgCnVJ16fgYlwSRWxVZNdtkLEn+k Q/3A0l19inlcN+mhxqrNbncGKG2ddw0AGCKoUJmCoXnTeAlDeCm9at/+N6W1A78AsUFF DKsA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=B5mAt8W8azn47LK2kphRwjqH60uu+XqgzfbWzrBC480=; fh=NA+vx86rq5nNT1FRzyPVjtGcfYwgltAEH+upuA2zREc=; b=q351DEan6oFwxh43gNuReq11PxU8DFZkrOvdyGL95VUvRRrFmMWjL9XHpD2hXkK2cD lLlRvxydgW6S9q31hqdNBsWbqpM6x1PG2QDVXCqzOh8+1K1WEsJNNy1Q/rUaampU/BXb T9aMovCqpG1gT9SifjEVP6jdAZl4Cw2Z8wqrn7hj2VCmdplKJ+jt9uLXCKp3l9RETsjP 4YbP3yher1kRzI7F6JVLJae77walYqKjUmoZQ4y8tIE/7FLmAPJj1PhRq4nZI73EjFSW yt5eMinDp70n0WRY+agyRVyULLfmmUBQURlBFG19JcoaJlbfDVlKRV4a9WJqoP2yuOGp tIMA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=LUQAqlLm; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55217-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55217-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=1; AJvYcCVD8UqdJAYh5nQa6S26sO5XLvRc+//hDA5Ab09IgWA3ZQ5FX3T8oHrkIwcx9Vz66gn8oo3DMGMvpG96sBUIeWJG/3xq5A== Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id 138-20020a630090000000b005d8b8c4cc37si1799787pga.295.2024.02.06.07.44.18 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:44:19 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55217-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=LUQAqlLm; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55217-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55217-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 1C144B2AD6F for ; Tue, 6 Feb 2024 15:22:09 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 7BEB713BEA0; Tue, 6 Feb 2024 15:13:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="LUQAqlLm" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 54259134751; Tue, 6 Feb 2024 15:13:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232412; cv=none; b=DXH4DtddQguhPfY0fYBOlxWXH8OPcjGeCHsy1Mcer/sFIgqMvs28tGgnslYoS+YLtyWUtDSY6cZWUddkQ58CMhdrtmFypQkOZAi0klUIzy1TXfYOuG8u6evGi29OVmbPev2TMxkFRHmp5r5jogwOv4+4WsFdKCBewGpVET0SEEA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232412; c=relaxed/simple; bh=xiqizROhD4kUvIgeKz/lwPBlsbppoAx8BUL+Q3/3Du4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=rfRTeW1g5xqea3Ud/lzZgy336SiXXEDQaWptw/p155pZkcRB0y8tGh0KtR/MCNii5VM12HDEHCPFlQ5g2WZ/uaVCpiMccRZHx7G1511kRU/ROVpykOSqatsolnUSzlb2ZH72lnx+7fQf4gNldJKmCDxkllLXcnGy0o1o4WgQDZ8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=LUQAqlLm; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id CB207C43394; Tue, 6 Feb 2024 15:13:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232411; bh=xiqizROhD4kUvIgeKz/lwPBlsbppoAx8BUL+Q3/3Du4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=LUQAqlLmI/j+nBAuMzXx3UmN3Ip1ea9onaXKxLwDH9AYGDfM/Ms/oyZVofHN/jkud x5E6ywzPm92tLB1WR+WVF8FSf1buc2oBtA9yrN+prtkPS3F6vP6ODpzIfw+ZkiGJuK O7Ljha24APpqSQaZUk1bqdiJisJccYCmqKa0HU3/GFoIiFV+0+azKGhjAzsCo06Cee bhuKAsldjwSHHPUpAGQDS5M+z5oeqsfA6MH6NgtVmWIqnP5HoFzFvfvFkrnM7OmalK uf55jQwTvqT7gGLUh8Dw0S5/1iOUGBB1vGk61A2UWPU1b3us8F76lZ4imht3e/5cuQ JzwAto/ekshAA== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 32/36] fprobe: Rewrite fprobe on function-graph tracer Date: Wed, 7 Feb 2024 00:13:25 +0900 Message-Id: <170723240525.502590.13978658964175623938.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790164870354853246 X-GMAIL-MSGID: 1790164870354853246 From: Masami Hiramatsu (Google) Rewrite fprobe implementation on function-graph tracer. Major API changes are: - 'nr_maxactive' field is deprecated. - This depends on CONFIG_DYNAMIC_FTRACE_WITH_ARGS or !CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS, and CONFIG_HAVE_FUNCTION_GRAPH_FREGS. So currently works only on x86_64. - Currently the entry size is limited in 15 * sizeof(long). - If there is too many fprobe exit handler set on the same function, it will fail to probe. Signed-off-by: Masami Hiramatsu (Google) --- Changes in v3: - Update for new reserve_data/retrieve_data API. - Fix internal push/pop on fgraph data logic so that it can correctly save/restore the returning fprobes. Changes in v2: - Add more lockdep_assert_held(fprobe_mutex) - Use READ_ONCE() and WRITE_ONCE() for fprobe_hlist_node::fp. - Add NOKPROBE_SYMBOL() for the functions which is called from entry/exit callback. --- include/linux/fprobe.h | 54 +++- kernel/trace/Kconfig | 8 - kernel/trace/fprobe.c | 633 ++++++++++++++++++++++++++++++++++-------------- lib/test_fprobe.c | 45 --- 4 files changed, 494 insertions(+), 246 deletions(-) diff --git a/include/linux/fprobe.h b/include/linux/fprobe.h index 879a30956009..08b37b0d1d05 100644 --- a/include/linux/fprobe.h +++ b/include/linux/fprobe.h @@ -5,32 +5,56 @@ #include #include -#include +#include +#include +#include + +struct fprobe; + +/** + * strcut fprobe_hlist_node - address based hash list node for fprobe. + * + * @hlist: The hlist node for address search hash table. + * @addr: The address represented by this. + * @fp: The fprobe which owns this. + */ +struct fprobe_hlist_node { + struct hlist_node hlist; + unsigned long addr; + struct fprobe *fp; +}; + +/** + * struct fprobe_hlist - hash list nodes for fprobe. + * + * @hlist: The hlist node for existence checking hash table. + * @rcu: rcu_head for RCU deferred release. + * @fp: The fprobe which owns this fprobe_hlist. + * @size: The size of @array. + * @array: The fprobe_hlist_node for each address to probe. + */ +struct fprobe_hlist { + struct hlist_node hlist; + struct rcu_head rcu; + struct fprobe *fp; + int size; + struct fprobe_hlist_node array[]; +}; /** * struct fprobe - ftrace based probe. - * @ops: The ftrace_ops. + * * @nmissed: The counter for missing events. * @flags: The status flag. - * @rethook: The rethook data structure. (internal data) * @entry_data_size: The private data storage size. - * @nr_maxactive: The max number of active functions. + * @nr_maxactive: The max number of active functions. (*deprecated) * @entry_handler: The callback function for function entry. * @exit_handler: The callback function for function exit. + * @hlist_array: The fprobe_hlist for fprobe search from IP hash table. */ struct fprobe { -#ifdef CONFIG_FUNCTION_TRACER - /* - * If CONFIG_FUNCTION_TRACER is not set, CONFIG_FPROBE is disabled too. - * But user of fprobe may keep embedding the struct fprobe on their own - * code. To avoid build error, this will keep the fprobe data structure - * defined here, but remove ftrace_ops data structure. - */ - struct ftrace_ops ops; -#endif unsigned long nmissed; unsigned int flags; - struct rethook *rethook; size_t entry_data_size; int nr_maxactive; @@ -40,6 +64,8 @@ struct fprobe { void (*exit_handler)(struct fprobe *fp, unsigned long entry_ip, unsigned long ret_ip, struct ftrace_regs *fregs, void *entry_data); + + struct fprobe_hlist *hlist_array; }; /* This fprobe is soft-disabled. */ diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig index 8b15adde1d8f..169588021d90 100644 --- a/kernel/trace/Kconfig +++ b/kernel/trace/Kconfig @@ -296,11 +296,9 @@ config DYNAMIC_FTRACE_WITH_ARGS config FPROBE bool "Kernel Function Probe (fprobe)" - depends on FUNCTION_TRACER - depends on DYNAMIC_FTRACE_WITH_REGS || DYNAMIC_FTRACE_WITH_ARGS - depends on HAVE_PT_REGS_TO_FTRACE_REGS_CAST || !HAVE_DYNAMIC_FTRACE_WITH_ARGS - depends on HAVE_RETHOOK - select RETHOOK + depends on FUNCTION_GRAPH_TRACER + depends on HAVE_FUNCTION_GRAPH_FREGS + depends on DYNAMIC_FTRACE_WITH_ARGS || !HAVE_DYNAMIC_FTRACE_WITH_ARGS default n help This option enables kernel function probe (fprobe) based on ftrace. diff --git a/kernel/trace/fprobe.c b/kernel/trace/fprobe.c index 31210423efc3..53e681c2458b 100644 --- a/kernel/trace/fprobe.c +++ b/kernel/trace/fprobe.c @@ -8,98 +8,193 @@ #include #include #include -#include +#include +#include #include #include #include "trace.h" -struct fprobe_rethook_node { - struct rethook_node node; - unsigned long entry_ip; - unsigned long entry_parent_ip; - char data[]; -}; +#define FPROBE_IP_HASH_BITS 8 +#define FPROBE_IP_TABLE_SIZE (1 << FPROBE_IP_HASH_BITS) -static inline void __fprobe_handler(unsigned long ip, unsigned long parent_ip, - struct ftrace_ops *ops, struct ftrace_regs *fregs) -{ - struct fprobe_rethook_node *fpr; - struct rethook_node *rh = NULL; - struct fprobe *fp; - void *entry_data = NULL; - int ret = 0; +#define FPROBE_HASH_BITS 6 +#define FPROBE_TABLE_SIZE (1 << FPROBE_HASH_BITS) - fp = container_of(ops, struct fprobe, ops); +/* + * fprobe_table: hold 'fprobe_hlist::hlist' for checking the fprobe still + * exists. The key is the address of fprobe instance. + * fprobe_ip_table: hold 'fprobe_hlist::array[*]' for searching the fprobe + * instance related to the funciton address. The key is the ftrace IP + * address. + * + * When unregistering the fprobe, fprobe_hlist::fp and fprobe_hlist::array[*].fp + * are set NULL and delete those from both hash tables (by hlist_del_rcu). + * After an RCU grace period, the fprobe_hlist itself will be released. + * + * fprobe_table and fprobe_ip_table can be accessed from either + * - Normal hlist traversal and RCU add/del under 'fprobe_mutex' is held. + * - RCU hlist traversal under disabling preempt + */ +static struct hlist_head fprobe_table[FPROBE_TABLE_SIZE]; +static struct hlist_head fprobe_ip_table[FPROBE_IP_TABLE_SIZE]; +static DEFINE_MUTEX(fprobe_mutex); - if (fp->exit_handler) { - rh = rethook_try_get(fp->rethook); - if (!rh) { - fp->nmissed++; - return; - } - fpr = container_of(rh, struct fprobe_rethook_node, node); - fpr->entry_ip = ip; - fpr->entry_parent_ip = parent_ip; - if (fp->entry_data_size) - entry_data = fpr->data; +/* + * Find first fprobe in the hlist. It will be iterated twice in the entry + * probe, once for correcting the total required size, the second time is + * calling back the user handlers. + * Thus the hlist in the fprobe_table must be sorted and new probe needs to + * be added *before* the first fprobe. + */ +static struct fprobe_hlist_node *find_first_fprobe_node(unsigned long ip) +{ + struct fprobe_hlist_node *node; + struct hlist_head *head; + + head = &fprobe_ip_table[hash_ptr((void *)ip, FPROBE_IP_HASH_BITS)]; + hlist_for_each_entry_rcu(node, head, hlist, + lockdep_is_held(&fprobe_mutex)) { + if (node->addr == ip) + return node; } + return NULL; +} +NOKPROBE_SYMBOL(find_first_fprobe_node); - if (fp->entry_handler) - ret = fp->entry_handler(fp, ip, parent_ip, fregs, entry_data); +/* Node insertion and deletion requires the fprobe_mutex */ +static void insert_fprobe_node(struct fprobe_hlist_node *node) +{ + unsigned long ip = node->addr; + struct fprobe_hlist_node *next; + struct hlist_head *head; - /* If entry_handler returns !0, nmissed is not counted. */ - if (rh) { - if (ret) - rethook_recycle(rh); - else - rethook_hook(rh, ftrace_get_regs(fregs), true); + lockdep_assert_held(&fprobe_mutex); + + next = find_first_fprobe_node(ip); + if (next) { + hlist_add_before_rcu(&node->hlist, &next->hlist); + return; } + head = &fprobe_ip_table[hash_ptr((void *)ip, FPROBE_IP_HASH_BITS)]; + hlist_add_head_rcu(&node->hlist, head); } -static void fprobe_handler(unsigned long ip, unsigned long parent_ip, - struct ftrace_ops *ops, struct ftrace_regs *fregs) +/* Return true if there are synonims */ +static bool delete_fprobe_node(struct fprobe_hlist_node *node) { - struct fprobe *fp; - int bit; + lockdep_assert_held(&fprobe_mutex); - fp = container_of(ops, struct fprobe, ops); - if (fprobe_disabled(fp)) - return; + WRITE_ONCE(node->fp, NULL); + hlist_del_rcu(&node->hlist); + return !!find_first_fprobe_node(node->addr); +} - /* recursion detection has to go before any traceable function and - * all functions before this point should be marked as notrace - */ - bit = ftrace_test_recursion_trylock(ip, parent_ip); - if (bit < 0) { - fp->nmissed++; - return; +/* Check existence of the fprobe */ +static bool is_fprobe_still_exist(struct fprobe *fp) +{ + struct hlist_head *head; + struct fprobe_hlist *fph; + + head = &fprobe_table[hash_ptr(fp, FPROBE_HASH_BITS)]; + hlist_for_each_entry_rcu(fph, head, hlist, + lockdep_is_held(&fprobe_mutex)) { + if (fph->fp == fp) + return true; } - __fprobe_handler(ip, parent_ip, ops, fregs); - ftrace_test_recursion_unlock(bit); + return false; +} +NOKPROBE_SYMBOL(is_fprobe_still_exist); + +static int add_fprobe_hash(struct fprobe *fp) +{ + struct fprobe_hlist *fph = fp->hlist_array; + struct hlist_head *head; + + lockdep_assert_held(&fprobe_mutex); + if (WARN_ON_ONCE(!fph)) + return -EINVAL; + + if (is_fprobe_still_exist(fp)) + return -EEXIST; + + head = &fprobe_table[hash_ptr(fp, FPROBE_HASH_BITS)]; + hlist_add_head_rcu(&fp->hlist_array->hlist, head); + return 0; } -NOKPROBE_SYMBOL(fprobe_handler); -static void fprobe_kprobe_handler(unsigned long ip, unsigned long parent_ip, - struct ftrace_ops *ops, struct ftrace_regs *fregs) +static int del_fprobe_hash(struct fprobe *fp) { - struct fprobe *fp; - int bit; + struct fprobe_hlist *fph = fp->hlist_array; - fp = container_of(ops, struct fprobe, ops); - if (fprobe_disabled(fp)) - return; + lockdep_assert_held(&fprobe_mutex); - /* recursion detection has to go before any traceable function and - * all functions called before this point should be marked as notrace - */ - bit = ftrace_test_recursion_trylock(ip, parent_ip); - if (bit < 0) { - fp->nmissed++; - return; + if (WARN_ON_ONCE(!fph)) + return -EINVAL; + + if (!is_fprobe_still_exist(fp)) + return -ENOENT; + + fph->fp = NULL; + hlist_del_rcu(&fph->hlist); + return 0; +} + +/* The entry data size is 4 bits (=16) * sizeof(long) in maximum */ +#define FPROBE_HEADER_SIZE_BITS 4 +#define MAX_FPROBE_DATA_SIZE_WORD ((1L << FPROBE_HEADER_SIZE_BITS) - 1) +#define MAX_FPROBE_DATA_SIZE (MAX_FPROBE_DATA_SIZE_WORD * sizeof(long)) +#define FPROBE_HEADER_PTR_BITS (BITS_PER_LONG - FPROBE_HEADER_SIZE_BITS) +#define FPROBE_HEADER_PTR_MASK GENMASK(FPROBE_HEADER_PTR_BITS - 1, 0) +#define FPROBE_HEADER_SIZE sizeof(unsigned long) + +static inline unsigned long encode_fprobe_header(struct fprobe *fp, int size_words) +{ + if (WARN_ON_ONCE(size_words > MAX_FPROBE_DATA_SIZE_WORD || + ((unsigned long)fp & ~FPROBE_HEADER_PTR_MASK) != + ~FPROBE_HEADER_PTR_MASK)) { + return 0; } + return ((unsigned long)size_words << FPROBE_HEADER_PTR_BITS) | + ((unsigned long)fp & FPROBE_HEADER_PTR_MASK); +} + +/* Return reserved data size in words */ +static inline int decode_fprobe_header(unsigned long val, struct fprobe **fp) +{ + unsigned long ptr; + + ptr = (val & FPROBE_HEADER_PTR_MASK) | ~FPROBE_HEADER_PTR_MASK; + if (fp) + *fp = (struct fprobe *)ptr; + return val >> FPROBE_HEADER_PTR_BITS; +} + +/* + * fprobe shadow stack management: + * Since fprobe shares a single fgraph_ops, it needs to share the stack entry + * among the probes on the same function exit. Note that a new probe can be + * registered before a target function is returning, we can not use the hash + * table to find the corresponding probes. Thus the probe address is stored on + * the shadow stack with its entry data size. + * + */ +static inline int __fprobe_handler(unsigned long ip, unsigned long parent_ip, + struct fprobe *fp, struct ftrace_regs *fregs, + void *data) +{ + if (!fp->entry_handler) + return 0; + + return fp->entry_handler(fp, ip, parent_ip, fregs, data); +} +static inline int __fprobe_kprobe_handler(unsigned long ip, unsigned long parent_ip, + struct fprobe *fp, struct ftrace_regs *fregs, + void *data) +{ + int ret; /* * This user handler is shared with other kprobes and is not expected to be * called recursively. So if any other kprobe handler is running, this will @@ -108,45 +203,180 @@ static void fprobe_kprobe_handler(unsigned long ip, unsigned long parent_ip, */ if (unlikely(kprobe_running())) { fp->nmissed++; - goto recursion_unlock; + return 0; } kprobe_busy_begin(); - __fprobe_handler(ip, parent_ip, ops, fregs); + ret = __fprobe_handler(ip, parent_ip, fp, fregs, data); kprobe_busy_end(); - -recursion_unlock: - ftrace_test_recursion_unlock(bit); + return ret; } -static void fprobe_exit_handler(struct rethook_node *rh, void *data, - unsigned long ret_ip, struct pt_regs *regs) +static int fprobe_entry(unsigned long func, unsigned long ret_ip, + struct ftrace_regs *fregs, struct fgraph_ops *gops) { - struct fprobe *fp = (struct fprobe *)data; - struct fprobe_rethook_node *fpr; - struct ftrace_regs *fregs = (struct ftrace_regs *)regs; - int bit; + struct fprobe_hlist_node *node, *first; + unsigned long *fgraph_data = NULL; + unsigned long header; + int reserved_words; + struct fprobe *fp; + int used, ret; - if (!fp || fprobe_disabled(fp)) - return; + if (WARN_ON_ONCE(!fregs)) + return 0; - fpr = container_of(rh, struct fprobe_rethook_node, node); + first = node = find_first_fprobe_node(func); + if (unlikely(!first)) + return 0; + + reserved_words = 0; + hlist_for_each_entry_from_rcu(node, hlist) { + if (node->addr != func) + break; + fp = READ_ONCE(node->fp); + if (!fp || !fp->exit_handler) + continue; + /* + * Since fprobe can be enabled until the next loop, we ignore the + * fprobe's disabled flag in this loop. + */ + reserved_words += + DIV_ROUND_UP(fp->entry_data_size, sizeof(long)) + 1; + } + node = first; + if (reserved_words) { + fgraph_data = fgraph_reserve_data(gops->idx, reserved_words * sizeof(long)); + if (unlikely(!fgraph_data)) { + hlist_for_each_entry_from_rcu(node, hlist) { + if (node->addr != func) + break; + fp = READ_ONCE(node->fp); + if (fp && !fprobe_disabled(fp)) + fp->nmissed++; + } + return 0; + } + } /* - * we need to assure no calls to traceable functions in-between the - * end of fprobe_handler and the beginning of fprobe_exit_handler. + * TODO: recursion detection has been done in the fgraph. Thus we need + * to add a callback to increment missed counter. */ - bit = ftrace_test_recursion_trylock(fpr->entry_ip, fpr->entry_parent_ip); - if (bit < 0) { - fp->nmissed++; + used = 0; + hlist_for_each_entry_from_rcu(node, hlist) { + void *data; + + if (node->addr != func) + break; + fp = READ_ONCE(node->fp); + if (!fp || fprobe_disabled(fp)) + continue; + + if (fp->entry_data_size && fp->exit_handler) + data = fgraph_data + used + 1; + else + data = NULL; + + if (fprobe_shared_with_kprobes(fp)) + ret = __fprobe_kprobe_handler(func, ret_ip, fp, fregs, data); + else + ret = __fprobe_handler(func, ret_ip, fp, fregs, data); + /* If entry_handler returns !0, nmissed is not counted but skips exit_handler. */ + if (!ret && fp->exit_handler) { + int size_words = DIV_ROUND_UP(fp->entry_data_size, sizeof(long)); + + header = encode_fprobe_header(fp, size_words); + if (likely(header)) { + fgraph_data[used] = header; + used += size_words + 1; + } + } + } + if (used < reserved_words) + memset(fgraph_data + used, 0, reserved_words - used); + + /* If any exit_handler is set, data must be used. */ + return used != 0; +} +NOKPROBE_SYMBOL(fprobe_entry); + +static void fprobe_return(unsigned long func, unsigned long ret_ip, + struct ftrace_regs *fregs, struct fgraph_ops *gops) +{ + unsigned long *fgraph_data = NULL; + unsigned long val; + struct fprobe *fp; + int size, curr; + int size_words; + + fgraph_data = (unsigned long *)fgraph_retrieve_data(gops->idx, &size); + if (!fgraph_data) return; + size_words = DIV_ROUND_UP(size, sizeof(long)); + + preempt_disable(); + + curr = 0; + while (size_words > curr) { + val = fgraph_data[curr++]; + if (!val) + break; + + size = decode_fprobe_header(val, &fp); + if (fp && is_fprobe_still_exist(fp) && !fprobe_disabled(fp)) { + if (WARN_ON_ONCE(curr + size > size_words)) + break; + fp->exit_handler(fp, func, ret_ip, fregs, + size ? fgraph_data + curr : NULL); + } + curr += size + 1; } + preempt_enable(); +} +NOKPROBE_SYMBOL(fprobe_return); - fp->exit_handler(fp, fpr->entry_ip, ret_ip, fregs, - fp->entry_data_size ? (void *)fpr->data : NULL); - ftrace_test_recursion_unlock(bit); +static struct fgraph_ops fprobe_graph_ops = { + .entryregfunc = fprobe_entry, + .retregfunc = fprobe_return, +}; +static int fprobe_graph_active; + +/* Add @addrs to the ftrace filter and register fgraph if needed. */ +static int fprobe_graph_add_ips(unsigned long *addrs, int num) +{ + int ret; + + lockdep_assert_held(&fprobe_mutex); + + ret = ftrace_set_filter_ips(&fprobe_graph_ops.ops, addrs, num, 0, 0); + if (ret) + return ret; + + if (!fprobe_graph_active) { + ret = register_ftrace_graph(&fprobe_graph_ops); + if (WARN_ON_ONCE(ret)) { + ftrace_free_filter(&fprobe_graph_ops.ops); + return ret; + } + } + fprobe_graph_active++; + return 0; +} + +/* Remove @addrs from the ftrace filter and unregister fgraph if possible. */ +static void fprobe_graph_remove_ips(unsigned long *addrs, int num) +{ + lockdep_assert_held(&fprobe_mutex); + + fprobe_graph_active--; + if (!fprobe_graph_active) { + /* Q: should we unregister it ? */ + unregister_ftrace_graph(&fprobe_graph_ops); + return; + } + + ftrace_set_filter_ips(&fprobe_graph_ops.ops, addrs, num, 1, 0); } -NOKPROBE_SYMBOL(fprobe_exit_handler); static int symbols_cmp(const void *a, const void *b) { @@ -176,56 +406,97 @@ static unsigned long *get_ftrace_locations(const char **syms, int num) return ERR_PTR(-ENOENT); } -static void fprobe_init(struct fprobe *fp) -{ - fp->nmissed = 0; - if (fprobe_shared_with_kprobes(fp)) - fp->ops.func = fprobe_kprobe_handler; - else - fp->ops.func = fprobe_handler; - - fp->ops.flags |= FTRACE_OPS_FL_SAVE_REGS; -} +struct filter_match_data { + const char *filter; + const char *notfilter; + size_t index; + size_t size; + unsigned long *addrs; +}; -static int fprobe_init_rethook(struct fprobe *fp, int num) +static int filter_match_callback(void *data, const char *name, unsigned long addr) { - int size; + struct filter_match_data *match = data; - if (num <= 0) - return -EINVAL; + if (!glob_match(match->filter, name) || + (match->notfilter && glob_match(match->notfilter, name))) + return 0; - if (!fp->exit_handler) { - fp->rethook = NULL; + if (!ftrace_location(addr)) return 0; - } - /* Initialize rethook if needed */ - if (fp->nr_maxactive) - size = fp->nr_maxactive; - else - size = num * num_possible_cpus() * 2; - if (size <= 0) - return -EINVAL; + if (match->addrs) + match->addrs[match->index] = addr; - /* Initialize rethook */ - fp->rethook = rethook_alloc((void *)fp, fprobe_exit_handler, - sizeof(struct fprobe_rethook_node), size); - if (IS_ERR(fp->rethook)) - return PTR_ERR(fp->rethook); + match->index++; + return match->index == match->size; +} - return 0; +/* + * Make IP list from the filter/no-filter glob patterns. + * Return the number of matched symbols, or -ENOENT. + */ +static int ip_list_from_filter(const char *filter, const char *notfilter, + unsigned long *addrs, size_t size) +{ + struct filter_match_data match = { .filter = filter, .notfilter = notfilter, + .index = 0, .size = size, .addrs = addrs}; + int ret; + + ret = kallsyms_on_each_symbol(filter_match_callback, &match); + if (ret < 0) + return ret; + ret = module_kallsyms_on_each_symbol(NULL, filter_match_callback, &match); + if (ret < 0) + return ret; + + return match.index ?: -ENOENT; } static void fprobe_fail_cleanup(struct fprobe *fp) { - if (!IS_ERR_OR_NULL(fp->rethook)) { - /* Don't need to cleanup rethook->handler because this is not used. */ - rethook_free(fp->rethook); - fp->rethook = NULL; + kfree(fp->hlist_array); + fp->hlist_array = NULL; +} + +/* Initialize the fprobe data structure. */ +static int fprobe_init(struct fprobe *fp, unsigned long *addrs, int num) +{ + struct fprobe_hlist *hlist_array; + unsigned long addr; + int size, i; + + if (!fp || !addrs || num <= 0) + return -EINVAL; + + size = ALIGN(fp->entry_data_size, sizeof(long)); + if (size > MAX_FPROBE_DATA_SIZE) + return -E2BIG; + fp->entry_data_size = size; + + hlist_array = kzalloc(struct_size(hlist_array, array, num), GFP_KERNEL); + if (!hlist_array) + return -ENOMEM; + + fp->nmissed = 0; + + hlist_array->size = num; + fp->hlist_array = hlist_array; + hlist_array->fp = fp; + for (i = 0; i < num; i++) { + hlist_array->array[i].fp = fp; + addr = ftrace_location(addrs[i]); + if (!addr) { + fprobe_fail_cleanup(fp); + return -ENOENT; + } + hlist_array->array[i].addr = addr; } - ftrace_free_filter(&fp->ops); + return 0; } +#define FPROBE_IPS_MAX INT_MAX + /** * register_fprobe() - Register fprobe to ftrace by pattern. * @fp: A fprobe data structure to be registered. @@ -239,46 +510,24 @@ static void fprobe_fail_cleanup(struct fprobe *fp) */ int register_fprobe(struct fprobe *fp, const char *filter, const char *notfilter) { - struct ftrace_hash *hash; - unsigned char *str; - int ret, len; + unsigned long *addrs; + int ret; if (!fp || !filter) return -EINVAL; - fprobe_init(fp); - - len = strlen(filter); - str = kstrdup(filter, GFP_KERNEL); - ret = ftrace_set_filter(&fp->ops, str, len, 0); - kfree(str); - if (ret) + ret = ip_list_from_filter(filter, notfilter, NULL, FPROBE_IPS_MAX); + if (ret < 0) return ret; - if (notfilter) { - len = strlen(notfilter); - str = kstrdup(notfilter, GFP_KERNEL); - ret = ftrace_set_notrace(&fp->ops, str, len, 0); - kfree(str); - if (ret) - goto out; - } - - /* TODO: - * correctly calculate the total number of filtered symbols - * from both filter and notfilter. - */ - hash = rcu_access_pointer(fp->ops.local_hash.filter_hash); - if (WARN_ON_ONCE(!hash)) - goto out; - - ret = fprobe_init_rethook(fp, (int)hash->count); - if (!ret) - ret = register_ftrace_function(&fp->ops); + addrs = kcalloc(ret, sizeof(unsigned long), GFP_KERNEL); + if (!addrs) + return -ENOMEM; + ret = ip_list_from_filter(filter, notfilter, addrs, ret); + if (ret > 0) + ret = register_fprobe_ips(fp, addrs, ret); -out: - if (ret) - fprobe_fail_cleanup(fp); + kfree(addrs); return ret; } EXPORT_SYMBOL_GPL(register_fprobe); @@ -286,7 +535,7 @@ EXPORT_SYMBOL_GPL(register_fprobe); /** * register_fprobe_ips() - Register fprobe to ftrace by address. * @fp: A fprobe data structure to be registered. - * @addrs: An array of target ftrace location addresses. + * @addrs: An array of target function address. * @num: The number of entries of @addrs. * * Register @fp to ftrace for enabling the probe on the address given by @addrs. @@ -298,23 +547,27 @@ EXPORT_SYMBOL_GPL(register_fprobe); */ int register_fprobe_ips(struct fprobe *fp, unsigned long *addrs, int num) { - int ret; - - if (!fp || !addrs || num <= 0) - return -EINVAL; - - fprobe_init(fp); + struct fprobe_hlist *hlist_array; + int ret, i; - ret = ftrace_set_filter_ips(&fp->ops, addrs, num, 0, 0); + ret = fprobe_init(fp, addrs, num); if (ret) return ret; - ret = fprobe_init_rethook(fp, num); - if (!ret) - ret = register_ftrace_function(&fp->ops); + mutex_lock(&fprobe_mutex); + + hlist_array = fp->hlist_array; + ret = fprobe_graph_add_ips(addrs, num); + if (!ret) { + add_fprobe_hash(fp); + for (i = 0; i < hlist_array->size; i++) + insert_fprobe_node(&hlist_array->array[i]); + } + mutex_unlock(&fprobe_mutex); if (ret) fprobe_fail_cleanup(fp); + return ret; } EXPORT_SYMBOL_GPL(register_fprobe_ips); @@ -352,14 +605,13 @@ EXPORT_SYMBOL_GPL(register_fprobe_syms); bool fprobe_is_registered(struct fprobe *fp) { - if (!fp || (fp->ops.saved_func != fprobe_handler && - fp->ops.saved_func != fprobe_kprobe_handler)) + if (!fp || !fp->hlist_array) return false; return true; } /** - * unregister_fprobe() - Unregister fprobe from ftrace + * unregister_fprobe() - Unregister fprobe. * @fp: A fprobe data structure to be unregistered. * * Unregister fprobe (and remove ftrace hooks from the function entries). @@ -368,23 +620,40 @@ bool fprobe_is_registered(struct fprobe *fp) */ int unregister_fprobe(struct fprobe *fp) { - int ret; + struct fprobe_hlist *hlist_array; + unsigned long *addrs = NULL; + int ret = 0, i, count; - if (!fprobe_is_registered(fp)) - return -EINVAL; + mutex_lock(&fprobe_mutex); + if (!fp || !is_fprobe_still_exist(fp)) { + ret = -EINVAL; + goto out; + } - if (!IS_ERR_OR_NULL(fp->rethook)) - rethook_stop(fp->rethook); + hlist_array = fp->hlist_array; + addrs = kcalloc(hlist_array->size, sizeof(unsigned long), GFP_KERNEL); + if (!addrs) { + ret = -ENOMEM; /* TODO: Fallback to one-by-one loop */ + goto out; + } - ret = unregister_ftrace_function(&fp->ops); - if (ret < 0) - return ret; + /* Remove non-synonim ips from table and hash */ + count = 0; + for (i = 0; i < hlist_array->size; i++) { + if (!delete_fprobe_node(&hlist_array->array[i])) + addrs[count++] = hlist_array->array[i].addr; + } + del_fprobe_hash(fp); - if (!IS_ERR_OR_NULL(fp->rethook)) - rethook_free(fp->rethook); + fprobe_graph_remove_ips(addrs, count); - ftrace_free_filter(&fp->ops); + kfree_rcu(hlist_array, rcu); + fp->hlist_array = NULL; +out: + mutex_unlock(&fprobe_mutex); + + kfree(addrs); return ret; } EXPORT_SYMBOL_GPL(unregister_fprobe); diff --git a/lib/test_fprobe.c b/lib/test_fprobe.c index 271ce0caeec0..cf92111b5c79 100644 --- a/lib/test_fprobe.c +++ b/lib/test_fprobe.c @@ -17,10 +17,8 @@ static u32 rand1, entry_val, exit_val; /* Use indirect calls to avoid inlining the target functions */ static u32 (*target)(u32 value); static u32 (*target2)(u32 value); -static u32 (*target_nest)(u32 value, u32 (*nest)(u32)); static unsigned long target_ip; static unsigned long target2_ip; -static unsigned long target_nest_ip; static int entry_return_value; static noinline u32 fprobe_selftest_target(u32 value) @@ -33,11 +31,6 @@ static noinline u32 fprobe_selftest_target2(u32 value) return (value / div_factor) + 1; } -static noinline u32 fprobe_selftest_nest_target(u32 value, u32 (*nest)(u32)) -{ - return nest(value + 2); -} - static notrace int fp_entry_handler(struct fprobe *fp, unsigned long ip, unsigned long ret_ip, struct ftrace_regs *fregs, void *data) @@ -79,22 +72,6 @@ static notrace void fp_exit_handler(struct fprobe *fp, unsigned long ip, KUNIT_EXPECT_NULL(current_test, data); } -static notrace int nest_entry_handler(struct fprobe *fp, unsigned long ip, - unsigned long ret_ip, - struct ftrace_regs *fregs, void *data) -{ - KUNIT_EXPECT_FALSE(current_test, preemptible()); - return 0; -} - -static notrace void nest_exit_handler(struct fprobe *fp, unsigned long ip, - unsigned long ret_ip, - struct ftrace_regs *fregs, void *data) -{ - KUNIT_EXPECT_FALSE(current_test, preemptible()); - KUNIT_EXPECT_EQ(current_test, ip, target_nest_ip); -} - /* Test entry only (no rethook) */ static void test_fprobe_entry(struct kunit *test) { @@ -191,25 +168,6 @@ static void test_fprobe_data(struct kunit *test) KUNIT_EXPECT_EQ(test, 0, unregister_fprobe(&fp)); } -/* Test nr_maxactive */ -static void test_fprobe_nest(struct kunit *test) -{ - static const char *syms[] = {"fprobe_selftest_target", "fprobe_selftest_nest_target"}; - struct fprobe fp = { - .entry_handler = nest_entry_handler, - .exit_handler = nest_exit_handler, - .nr_maxactive = 1, - }; - - current_test = test; - KUNIT_EXPECT_EQ(test, 0, register_fprobe_syms(&fp, syms, 2)); - - target_nest(rand1, target); - KUNIT_EXPECT_EQ(test, 1, fp.nmissed); - - KUNIT_EXPECT_EQ(test, 0, unregister_fprobe(&fp)); -} - static void test_fprobe_skip(struct kunit *test) { struct fprobe fp = { @@ -247,10 +205,8 @@ static int fprobe_test_init(struct kunit *test) rand1 = get_random_u32_above(div_factor); target = fprobe_selftest_target; target2 = fprobe_selftest_target2; - target_nest = fprobe_selftest_nest_target; target_ip = get_ftrace_location(target); target2_ip = get_ftrace_location(target2); - target_nest_ip = get_ftrace_location(target_nest); return 0; } @@ -260,7 +216,6 @@ static struct kunit_case fprobe_testcases[] = { KUNIT_CASE(test_fprobe), KUNIT_CASE(test_fprobe_syms), KUNIT_CASE(test_fprobe_data), - KUNIT_CASE(test_fprobe_nest), KUNIT_CASE(test_fprobe_skip), {} }; From patchwork Tue Feb 6 15:13:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197505 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1613796dyb; Tue, 6 Feb 2024 07:22:31 -0800 (PST) X-Google-Smtp-Source: AGHT+IFtI9di59VYYB7gjs3MyA6HVi149py1VR5eJ/X+SbKcvZmmm0R03NWt1asurjXFoYTY6kil X-Received: by 2002:a05:6402:7c5:b0:560:5c58:dfc4 with SMTP id u5-20020a05640207c500b005605c58dfc4mr2064964edy.1.1707232950832; Tue, 06 Feb 2024 07:22:30 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707232950; cv=pass; d=google.com; s=arc-20160816; b=mK44lNrT5IY4OGiw4qIGU8oruHmdf64dthj2QoF1fuhtOsh6gzBql/xA1oCEBvwde0 QxoXv2VysO4TiicLgrmdcKUXFxSGO8lvxMK7ZW14kMXoZIzWu7hoBjy4KxfuSmBpAJhZ VkaQ+WrmMB3GwSqe8smjmXmO9JfdgjxUZ0/Dd7bwSGt8njrVcgz4x4zUAaieoCYCacAJ JyQ479si6CS7wFnbu3t/FOnqNdr5d0Ak5SYNgpqS7yJ9UAUeqSPPBWnukVkx/n/m75Ga DlmFFYzVRQP7MV5ZJqLVz+Ry8duAwRp1pFCDI9zIyof4dM2EOoMaIuucfZZi0jf0ZC1m 8hKg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=j926EWqfo0g9vIUVygZM8OTFYJPm6KP/dXgYh+R7ao4=; fh=hYZ9odOhlbS4Lt5xSxaJ6GBbixZepTApesR4vQ9sRdM=; b=KSD5cOq5eJv6El++xXFu7bGf6tRsStSpvWxcCbqEfauTwqAZdZihPpvVYPekH7GIl3 otcArnIF1FxGXbo0C8uZiEloHYUnF0d42Y92AIm4ZTprdIu8EMnRohKzLmU9sfo6Hpfm URQPTdGM5OO/aV3kFYjQJKNQs2E4gwrZtqsgOwwx/C9nYTWNGGeGS0KwNxZcAV9C+Ds3 i5dOvsq8eTy5fNh74Nk12PFffN2NWRAaplvd2IbcQG89UVX3TbyGFnFZ2v2r3CwVCwqA PWa5IDSNvnMqNPTXdzQpf4k+zl4ZYaAgWGCSHtSvZ2S6DlKAMpLULyMW+Ok+api5sUwR E06g==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=olbKQznT; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55218-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55218-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=1; AJvYcCWcY6YeyElU43I1Yq3oUZBRN+Yj46PK1uSROCSSJp+FwUOMWBV8879/HSFvwSbn3OaobPZDdSzztL0fgJIxyIyiLlozFQ== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id bo15-20020a0564020b2f00b0055ef4fb4170si1232509edb.56.2024.02.06.07.22.30 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:22:30 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55218-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=olbKQznT; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55218-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55218-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 4258E1F22414 for ; Tue, 6 Feb 2024 15:22:30 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id E105113C1CC; Tue, 6 Feb 2024 15:13:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="olbKQznT" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 07EF8134751; Tue, 6 Feb 2024 15:13:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232423; cv=none; b=QwFXtlz99IQgt8CDUMaNnfSus0O9IDnjafl5Fh92tS7qxAgrgXkZIzMTQ27jI6jUgMMh/BgmdFJhB0ANgBO+UDXtEtRbYRpf9Uqczn1S8IkNE7NaMrkElo7vYrJLtKUF3zGlogm6NNlj58Km5Vuj5Yhdq7MMqNlhQjX3NFIIRMo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232423; c=relaxed/simple; bh=V7BE8q0MmiFvaddSn9SJAfAslRoLC7k0iroDoA8t4/U=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=hxrF5Fjz7zCN9L8wWZ1o3PbpRIbLo2y/M0TL1WEB+5Z5NBxoapcYreO4zakOq8OQRcEWnWMi6n+BVdGh3PcMbJON5VefpcdFkebnGXuuspDs5HYP/i/xagw9ZfYzh2B9H9KFM11PkhLus7XJ4Olnp0YjcUgmIJpWwy1+6ojshdE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=olbKQznT; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 67007C433C7; Tue, 6 Feb 2024 15:13:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232422; bh=V7BE8q0MmiFvaddSn9SJAfAslRoLC7k0iroDoA8t4/U=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=olbKQznTE5mHYwrXPgaz/iJTvcrAClApN17NqE5mKbxyOlzRZIHq/madhzYg3ljEO D8jQIYwYtCmeTlwE5gTg5y84ZbjgW0gxCXk65wNgzJM+KqOvlOeE44vc5slYfrL/xW m13XUqvlwFUYfO1+B8RHZ0BiOmmO/v4H8kOL3WtDMghaCK9ED8mmBeVCBhT7vMwTTk 9636QXyBzXsjCBO5XehVtc1CPeHK0OZI2wCDA/Zlb9l4cFNsUqRrPW/VhyjMawA0qG u0T6rWLRZOIFd3/PZ9Nfzt9JkgT7bORHUBVAkACjOsftoQaJZC+LZWUF3xih4pRk6T lLM3hK+AlqMiQ== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 33/36] tracing/fprobe: Remove nr_maxactive from fprobe Date: Wed, 7 Feb 2024 00:13:37 +0900 Message-Id: <170723241719.502590.13823632735930550091.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790163498991396671 X-GMAIL-MSGID: 1790163498991396671 From: Masami Hiramatsu (Google) Remove depercated fprobe::nr_maxactive. This involves fprobe events to rejects the maxactive number. Signed-off-by: Masami Hiramatsu (Google) --- Changes in v2: - Newly added. --- include/linux/fprobe.h | 2 -- kernel/trace/trace_fprobe.c | 44 ++++++------------------------------------- 2 files changed, 6 insertions(+), 40 deletions(-) diff --git a/include/linux/fprobe.h b/include/linux/fprobe.h index 08b37b0d1d05..c28d06ddfb8e 100644 --- a/include/linux/fprobe.h +++ b/include/linux/fprobe.h @@ -47,7 +47,6 @@ struct fprobe_hlist { * @nmissed: The counter for missing events. * @flags: The status flag. * @entry_data_size: The private data storage size. - * @nr_maxactive: The max number of active functions. (*deprecated) * @entry_handler: The callback function for function entry. * @exit_handler: The callback function for function exit. * @hlist_array: The fprobe_hlist for fprobe search from IP hash table. @@ -56,7 +55,6 @@ struct fprobe { unsigned long nmissed; unsigned int flags; size_t entry_data_size; - int nr_maxactive; int (*entry_handler)(struct fprobe *fp, unsigned long entry_ip, unsigned long ret_ip, struct ftrace_regs *regs, diff --git a/kernel/trace/trace_fprobe.c b/kernel/trace/trace_fprobe.c index 7d2a66135f83..d96de0dbc0cb 100644 --- a/kernel/trace/trace_fprobe.c +++ b/kernel/trace/trace_fprobe.c @@ -375,7 +375,6 @@ static struct trace_fprobe *alloc_trace_fprobe(const char *group, const char *event, const char *symbol, struct tracepoint *tpoint, - int maxactive, int nargs, bool is_return) { struct trace_fprobe *tf; @@ -395,7 +394,6 @@ static struct trace_fprobe *alloc_trace_fprobe(const char *group, tf->fp.entry_handler = fentry_dispatcher; tf->tpoint = tpoint; - tf->fp.nr_maxactive = maxactive; ret = trace_probe_init(&tf->tp, event, group, false); if (ret < 0) @@ -974,12 +972,11 @@ static int __trace_fprobe_create(int argc, const char *argv[]) * FETCHARG:TYPE : use TYPE instead of unsigned long. */ struct trace_fprobe *tf = NULL; - int i, len, new_argc = 0, ret = 0; + int i, new_argc = 0, ret = 0; bool is_return = false; char *symbol = NULL; const char *event = NULL, *group = FPROBE_EVENT_SYSTEM; const char **new_argv = NULL; - int maxactive = 0; char buf[MAX_EVENT_NAME_LEN]; char gbuf[MAX_EVENT_NAME_LEN]; char sbuf[KSYM_NAME_LEN]; @@ -1000,33 +997,13 @@ static int __trace_fprobe_create(int argc, const char *argv[]) trace_probe_log_init("trace_fprobe", argc, argv); - event = strchr(&argv[0][1], ':'); - if (event) - event++; - - if (isdigit(argv[0][1])) { - if (event) - len = event - &argv[0][1] - 1; - else - len = strlen(&argv[0][1]); - if (len > MAX_EVENT_NAME_LEN - 1) { - trace_probe_log_err(1, BAD_MAXACT); - goto parse_error; - } - memcpy(buf, &argv[0][1], len); - buf[len] = '\0'; - ret = kstrtouint(buf, 0, &maxactive); - if (ret || !maxactive) { + if (argv[0][1] != '\0') { + if (argv[0][1] != ':') { + trace_probe_log_set_index(0); trace_probe_log_err(1, BAD_MAXACT); goto parse_error; } - /* fprobe rethook instances are iterated over via a list. The - * maximum should stay reasonable. - */ - if (maxactive > RETHOOK_MAXACTIVE_MAX) { - trace_probe_log_err(1, MAXACT_TOO_BIG); - goto parse_error; - } + event = &argv[0][2]; } trace_probe_log_set_index(1); @@ -1036,12 +1013,6 @@ static int __trace_fprobe_create(int argc, const char *argv[]) if (ret < 0) goto parse_error; - if (!is_return && maxactive) { - trace_probe_log_set_index(0); - trace_probe_log_err(1, BAD_MAXACT_TYPE); - goto parse_error; - } - trace_probe_log_set_index(0); if (event) { ret = traceprobe_parse_event_name(&event, &group, gbuf, @@ -1095,8 +1066,7 @@ static int __trace_fprobe_create(int argc, const char *argv[]) } /* setup a probe */ - tf = alloc_trace_fprobe(group, event, symbol, tpoint, maxactive, - argc, is_return); + tf = alloc_trace_fprobe(group, event, symbol, tpoint, argc, is_return); if (IS_ERR(tf)) { ret = PTR_ERR(tf); /* This must return -ENOMEM, else there is a bug */ @@ -1172,8 +1142,6 @@ static int trace_fprobe_show(struct seq_file *m, struct dyn_event *ev) seq_putc(m, 't'); else seq_putc(m, 'f'); - if (trace_fprobe_is_return(tf) && tf->fp.nr_maxactive) - seq_printf(m, "%d", tf->fp.nr_maxactive); seq_printf(m, ":%s/%s", trace_probe_group_name(&tf->tp), trace_probe_name(&tf->tp)); From patchwork Tue Feb 6 15:13:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197506 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1614097dyb; Tue, 6 Feb 2024 07:22:58 -0800 (PST) X-Google-Smtp-Source: AGHT+IGaiTBDCB/+EcsX47Xd0MKodXVMtFuVHJjQUc7Jb331r3Wta6zfOcRNt1FxIEjBLcE0nXCK X-Received: by 2002:ac8:5449:0:b0:42c:69a:61ca with SMTP id d9-20020ac85449000000b0042c069a61camr4309892qtq.32.1707232978438; Tue, 06 Feb 2024 07:22:58 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707232978; cv=pass; d=google.com; s=arc-20160816; b=AYkDkjIUjcd6JPNLpnAL3fshnGuEbvGuCg/0YEwWr/C5ysvVTqRRnCksAPOZa47//z Ko4irABMcxBtgjfZ5pGMhF13oKzBr9gjalY+w2tIKwfaBkosIM4FoJnQBAopOzBQpHG7 V7f4/OLYYNMk8p6TA+x6wSpRzS/2Y4+zQ+p7h+DhuW8UGWU2D286iKMlPecnrs8dfFJ0 4IlteKAhl47DEXbHw3Q9hT+aznRkpT4vlemKLTk/FEe0iOBz95DIq91wcLAz9jKYrk5G 25t7qvq1AJntuaVQQy5dVcggbMvBpf0nw0x+Ycbhcn9QOGxOcs6Gol2WhfxDHwuLJS1Z eMEw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=NRt0pRDVMb2HoTOOgHKbTFmgFt039oUBjnBj47jnGM0=; fh=jpBaUj5enYPKAC576ICCja7DdTTyck6DHVmYCNFPf20=; b=ZMbjNxjuL98Ifric3tmAvBYwjMY4JvCCxEkowcwZzDlx1F0AqmgAq3anrADxZVBCQA FsJvKARuz1SoA04KuI3bfBTxwrZe95Z2IDdBpG5yivrRA1UU8WoDjQqGkw9KCYbaK2Ns xqejZyLoamD4LWZ2ipbRSWOD9t9El+sOcJI0DiUwOVFYoYWciJ4YkOnz8N3p9MlEdSyE Mping4bCPA9rXIEWbaN0oVpxFLuW+3KU14OK4WEp6NBqvd3+Nuq58v3AlwmAaAoY1dRS BQhiy738GJgvyYs+IeLewDpVL8X+LIzWoB5GtfTY7vAOydKI8v0gwABSES9aNcNRqdmI Lv7Q==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=D9RTYE2r; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55220-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55220-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=1; AJvYcCV3zxwjywrKpbUIg48UY9F5/iVaUBS09lBH9a/s1KsUo/SGa6zgyzV3pKLAi7zBqlIsuTfoP7cDQiZYmYB7+lwZBG/FHA== Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id e22-20020a05622a111600b0042beb2fe398si2497981qty.454.2024.02.06.07.22.58 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:22:58 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55220-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=D9RTYE2r; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55220-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55220-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 39AB41C241CE for ; Tue, 6 Feb 2024 15:22:58 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 1B20B13C1F8; Tue, 6 Feb 2024 15:13:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="D9RTYE2r" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 75098133432; Tue, 6 Feb 2024 15:13:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232433; cv=none; b=V3TOQug9Ug5HdB7vdJO1qWcBG348rrWvVKVdRFPWHF6kghrSkir1kGn4N74sg7gdgicZPkiyJkjkqWT7UDWGLlsHTau7Xezr4wNu5fgqbr5B1pIETH5+ov7N9oRCxtKc3WbzGMFovEQ1DAVlPgMHQz1TkIDwtci0k33jQHHa8Mc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232433; c=relaxed/simple; bh=AhqKrJz5NqAxQZJTxzvS35/o9mTQRDXLSjKOCvX5P6o=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=NwIsWq2obpNmKei22Aww1/3qB2gPtZ7TYLqG3qqQKa4ImKWxQV78DNfebXHvSrE9Zjb5389Ionm7WPFtqCbIdk+qUfUeyEsGz2Ctyui3jyGLvf4HHTPeXyYhUYJDo0GOM80XIzez2erHeSoybSg5hL9RwTXVT92kr89pCaNwt/M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=D9RTYE2r; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 432A2C433C7; Tue, 6 Feb 2024 15:13:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232433; bh=AhqKrJz5NqAxQZJTxzvS35/o9mTQRDXLSjKOCvX5P6o=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=D9RTYE2rucHhwjfXu5vYc7a/lKZrG1OEFTu3RxUQe9O1kV1sEL7ZTgjAUtjlkJXlI SAQi6Q0dpZAFRuGIna2o4T9/63f9sz6b1kLXqNUv3CUgCKD+rz8ntl38VXFBTwuQvq Uot4bN+NSUdc3127XooI+id/bYzOUZM/6TJHDMZ/2qD5wkc9hWEvv1MDDHM57uhrK1 ZR5XTXBkUpitJ9lprIqDB9dnE90FXUQLveEthaVMm1Es/yEl95zYLez21/wmFfDb/m mh4wOdFMAH26Y+dTZgaCul1kDuZZHuZPEmHg56R08bNZ5qy04udC6xWGTlLN9FWaNX cLq/yltWbMwUA== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 34/36] selftests: ftrace: Remove obsolate maxactive syntax check Date: Wed, 7 Feb 2024 00:13:48 +0900 Message-Id: <170723242825.502590.11677076282887486945.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790163527093836749 X-GMAIL-MSGID: 1790163527093836749 From: Masami Hiramatsu (Google) Since the fprobe event does not support maxactive anymore, stop testing the maxactive syntax error checking. Signed-off-by: Masami Hiramatsu (Google) --- .../ftrace/test.d/dynevent/fprobe_syntax_errors.tc | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/tools/testing/selftests/ftrace/test.d/dynevent/fprobe_syntax_errors.tc b/tools/testing/selftests/ftrace/test.d/dynevent/fprobe_syntax_errors.tc index 20e42c030095..66516073ff27 100644 --- a/tools/testing/selftests/ftrace/test.d/dynevent/fprobe_syntax_errors.tc +++ b/tools/testing/selftests/ftrace/test.d/dynevent/fprobe_syntax_errors.tc @@ -16,9 +16,7 @@ aarch64) REG=%r0 ;; esac -check_error 'f^100 vfs_read' # MAXACT_NO_KPROBE -check_error 'f^1a111 vfs_read' # BAD_MAXACT -check_error 'f^100000 vfs_read' # MAXACT_TOO_BIG +check_error 'f^100 vfs_read' # BAD_MAXACT check_error 'f ^non_exist_func' # BAD_PROBE_ADDR (enoent) check_error 'f ^vfs_read+10' # BAD_PROBE_ADDR From patchwork Tue Feb 6 15:13:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197507 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1614263dyb; Tue, 6 Feb 2024 07:23:16 -0800 (PST) X-Google-Smtp-Source: AGHT+IGal3TKSYvmWujoPCBeSDUmB2rc7kkuNfRliKArXcTIXdmhLl7dkNhBzbG/Ej6BFyPoFPr/ X-Received: by 2002:a17:902:c115:b0:1d9:5934:ad7f with SMTP id 21-20020a170902c11500b001d95934ad7fmr1753237pli.29.1707232996101; Tue, 06 Feb 2024 07:23:16 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707232996; cv=pass; d=google.com; s=arc-20160816; b=uesNxu9ykeN4F27GrRV8tEHvM51JdG0Mga857DVAo4IfLOL0zn9NKVDYHcqfo8skpR W2G7zCZCAWaIetGO00UIEFlTz4dUBDciQaswwskMs5ByyOEr4gr/GZ+JORk0pNHAcu0Q i6bG4vXjgYoMmrhd3Lpd33R8EpdvbrDHuQAaUTgyBJDN+3batUCBhUe3DiKxAYKtHv6b GJrfqoDwhipE1/kthRha8LH7BKg+B+fVRDoJqZ1exBYQrR2Hy6JdHKwEMnnlqT57ZM1L 6CFp08COEG10nCwCAf7GBW1wroUPND9tuXQ9YUXfAgfwCx9AVgtFyAh+O2FVWLOEIkmH Fp4Q== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=mV5TwTTFofo7Hr0MUgPsm9fSGisdc/rZSPC/dKdQApI=; fh=XPGcKBVPmC7SNeDie9JiXZCU+nF81dTFuLYSohFAvSQ=; b=Mbsm5f8hcL9Bt21hb8FVX1BkcQJP2ImmcmzRlfh8rMpTPAaNYWAWVMKP7NEpbOlwyv rrVApzIaqojgfavOGRORzAYPxRUZgjeXTrpNhB2h1EZz2VC2pNYWWOsrJLO6dfGUiK47 3V6pvJqXK3NL5NO3XAoto2mAhvJ0STxNlXUPibPUVrJvIg2J6oR66zMkM/kVLce3K8t6 WsbTdQ3IjuHoZXUV/CYYOQ/xpeijrkzAsqQ20fHp0cOyU3pTMHk0mpi0IV4f5FfS/sYz Q79er4ytn4qv5cd8Ei5m3q7dvJLvrCdyMAA6/LtGWG2HXnDLpV8k0cl7xovJDqnjSBY2 RklA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=u5wpmdya; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55221-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55221-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=1; AJvYcCVFvqf86C7JyxzdOA/bFu2yyK2QMX7buGf4IaGVEslyvoZWmyBLNMRqwE1Af0W3JmXE+8j2TLfWf4adI8a+IWZ8ag9AnQ== Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id h6-20020a170902f7c600b001d8dba2d29asi1809001plw.22.2024.02.06.07.23.15 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:23:16 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55221-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=u5wpmdya; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55221-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55221-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id DAE532914D8 for ; Tue, 6 Feb 2024 15:23:15 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id A65BC13D4F1; Tue, 6 Feb 2024 15:14:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="u5wpmdya" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0A31F13665D; Tue, 6 Feb 2024 15:14:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232445; cv=none; b=g78jd08lTn+sf2JkubVWD22v2HW3Cv/nOmBvUINNJjWqxdJ4EXsmZlA8+H+DAmge36wR4PKlUWma5q0e8gDfvbpWHu53AH5b+zAhIFPkrtCqt/1jjoVA5SEGJwcsm1VYrP4H3tIaZhn+3CQIkzStAGCOMqNdhv+hC9bjr3DJITU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232445; c=relaxed/simple; bh=MTl3WOEWmkU7N1ZYT3IdZA3wW+cclx8nbkw1UKe91tQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=outArACtuycZSlEvzRAQaCTaEjhycXhZMUyB5Yji50PqX7FXuG31Qy/Cv+gndK0Ssjohgcpcc9PPw3kseLqgpn4LpQxgV7ooCRalrxyIxw1AbBdhXGuZ8dPAn/U2ceBFLK5FwxSSOjTtV5ytXGmPeUcnlIzm91j0AiLqPDVJ0b8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=u5wpmdya; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 20024C433F1; Tue, 6 Feb 2024 15:14:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232444; bh=MTl3WOEWmkU7N1ZYT3IdZA3wW+cclx8nbkw1UKe91tQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=u5wpmdyaGXwrtkPli2doQMGJemTKJgn1aQaqSl3zgtbMeQ6KbP7eRwMcaIMLfHCWO ObIksb/42fVqgUWuTTeUGnGXhYau30D7aL9PR9T++ff/AXL6lueUA5oiN8ymWd/eQt P8DdIIfygMlg8hfJbYEf9HKAPSmIKQkC5lRUcynLM81GDgWxKP43ZBCTkIMZ3id+P+ M/sVgnr3rwlzFFUHK3/ZeYHucAxRoKem0ZQaIGkauYfjHlvY/ZvEdviR8vuMuvmKej DC/DsOVqvGd/5yFon0ur91iFEnFMxquijASYt0eNAX9TTDBIalIbrKpsNbQXbYORhp z+qsTm/tZdwrw== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 35/36] selftests/ftrace: Add a test case for repeating register/unregister fprobe Date: Wed, 7 Feb 2024 00:13:58 +0900 Message-Id: <170723243869.502590.8138799732080228760.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790163546173663220 X-GMAIL-MSGID: 1790163546173663220 From: Masami Hiramatsu (Google) This test case repeats define and undefine the fprobe dynamic event to ensure that the fprobe does not cause any issue with such operations. Signed-off-by: Masami Hiramatsu (Google) --- .../test.d/dynevent/add_remove_fprobe_repeat.tc | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 tools/testing/selftests/ftrace/test.d/dynevent/add_remove_fprobe_repeat.tc diff --git a/tools/testing/selftests/ftrace/test.d/dynevent/add_remove_fprobe_repeat.tc b/tools/testing/selftests/ftrace/test.d/dynevent/add_remove_fprobe_repeat.tc new file mode 100644 index 000000000000..b4ad09237e2a --- /dev/null +++ b/tools/testing/selftests/ftrace/test.d/dynevent/add_remove_fprobe_repeat.tc @@ -0,0 +1,19 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0 +# description: Generic dynamic event - Repeating add/remove fprobe events +# requires: dynamic_events "f[:[/][]] [%return] []":README + +echo 0 > events/enable +echo > dynamic_events + +PLACE=$FUNCTION_FORK +REPEAT_TIMES=64 + +for i in `seq 1 $REPEAT_TIMES`; do + echo "f:myevent $PLACE" >> dynamic_events + grep -q myevent dynamic_events + test -d events/fprobes/myevent + echo > dynamic_events +done + +clear_trace From patchwork Tue Feb 6 15:14:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 197508 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1614476dyb; Tue, 6 Feb 2024 07:23:34 -0800 (PST) X-Google-Smtp-Source: AGHT+IFyZnDKpfD7Kv1Kg/eUoICr0qaYdUHrKO5K95cHX71hG5925YAe/kOHR5nwt6HdoHKSdUIC X-Received: by 2002:a05:6358:719:b0:175:cfa7:953d with SMTP id e25-20020a056358071900b00175cfa7953dmr3727555rwj.2.1707233013748; Tue, 06 Feb 2024 07:23:33 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707233013; cv=pass; d=google.com; s=arc-20160816; b=RurO7NG8bo+3QUIPzoj9SSWerl/aw4PA4TP1Wfb5nrOOBDCFgvSVypII2K1jE8uR1N 3lDcs4v7Fry2b+5IO5wfMKqdk5eNryGxUmrP9akAR1rmwHrVWUYRvFaUz4XloKO0ycyV +YozMycCfLJ7/lvIQICGiLGZYsTF0UJOwTr/OTnY6RQE0Ps8WFDYPH3BKU93KqO7Sx3G R6mwJbm+750yiIADr9Hp5XoMxlH7YHX8jqCVz22psMnVXdI0r0/s9gUJHjSMAOmrptJm 6nB9XgkUKTHzb0Lgm3ujuAhI8S1AAbAQYxIBNhy5y1HDZRhwaO0mUJGOuB4yYYMiboKW j2HA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=g/RVfnmZcQdfRZB+YRyiXpHH94CCdK1xBgexCLFpl6s=; fh=CJ0w+OWXIlL10XkSs6TfkAjFNTOpc0739n+NYnym4nM=; b=gHZSRhAUpSAdHVGlOKEbCo9rJkHgZ/Hq36g4KQ9KFkfR/eTWgMvi6NmMOyAmjSqn+V sQQXtGBViqmQ9YWH96Eaj9lDkvm2tkIRdCKEih9LnXIeKdqg1TzWq2a39k2ze18yL9eL gjMsLGZv9vyWojFZW+nt5gR05eCFgTm8EPoSNgjwlUW1loDAnc1/1WTUIS4nzsNjj2Y7 vWmuAScrLWEbGDGqG9ID4I5hKQXfNYoriZHx8XkEr0LltQcAbZD5lCPIeqei6ylC+X9G DfPq0JkLVWFnTsMCMNdPQ704nPmYiVt7VUxlMkCzXJUObXI5HhGtINaD01HoNMzuxaIq I2xQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=u4p6i6QU; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55222-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55222-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=1; AJvYcCVaDZzKhsP4aLB/hNsw4V0rJHhkF5TMr6iAnFjfXFIuQOTiNV+Dq+1MeR1np0pWwabjQWUt/M4JOdcTW7OFAAZfCA/3cQ== Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id h18-20020a63e152000000b005d68eec351esi1793004pgk.740.2024.02.06.07.23.33 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 07:23:33 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55222-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=u4p6i6QU; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55222-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55222-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 7EF292914A3 for ; Tue, 6 Feb 2024 15:23:33 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id B9A3B13D517; Tue, 6 Feb 2024 15:14:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="u4p6i6QU" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C21B7132C3A; Tue, 6 Feb 2024 15:14:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232456; cv=none; b=rHLCQvhxsdGJSicx9ijvNNi5Ar9ERCrF/GnmaeLI6JmXGraFCpY+Xj5vREhPT7CHHuujP3vjKZXvEStPKh7eo14efuasu+wOPN99oyxaQCcEWhzXyB52Ul+r5ebX4TL0eCz5N89BAguo361m9ShrpZK3miLyXVeqgI6GsA24DT8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707232456; c=relaxed/simple; bh=8t7YVZmoVUR7dCk/2nFP4SmQbrkuMwJVZH30PnGw0sc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=RqczFtWVWjYdsgC5HXyEqfuU+GZ+NprY+5a1lgnYmzsjETEAVI13Yb5sFEgFql2wEhMw9AK2k7mzNTh3Fgrp+sryuE6El7ALc/rqWEbD6QQYWWBO9976v2TTYwm8LSx4FgBvG9UwHs46TYx/tXkLHPkZUmzt02ao0LBT3MDSAfc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=u4p6i6QU; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 887CEC433C7; Tue, 6 Feb 2024 15:14:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707232456; bh=8t7YVZmoVUR7dCk/2nFP4SmQbrkuMwJVZH30PnGw0sc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=u4p6i6QUvB1lmToWncDmKrM6GaNvi/jEZ/g+nhNx5D/L2J/hZDX24E165LUBcvK7b SuLgtaQOhtHCEFucuFRU/AcvJX9LnLf8Gmy8CgHXhfwiTyFnUt0TuTVVFSQYS4P4+5 j+tePW7xuLh/z+zRLINZ5ABrYo6cBj3XJAmS8M09yGZ/PZn+AJUg6xbIitupolq5iz RimZa7DqIYXUxaDpdO19uwrpgJ3aJRsTIvQ2ySxZFoNvI9gDhS/KMhWO402qVfPGB/ Gp7rduXQy/5pqORw3eVvt2GswExfdtoOubIk9oD+NwQZzvXfN8uLGUQWqwZs3rB+4p SARF3quPrel1w== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v7 36/36] Documentation: probes: Update fprobe on function-graph tracer Date: Wed, 7 Feb 2024 00:14:10 +0900 Message-Id: <170723245029.502590.2313180068078569074.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170723204881.502590.11906735097521170661.stgit@devnote2> References: <170723204881.502590.11906735097521170661.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790163564210769771 X-GMAIL-MSGID: 1790163564210769771 From: Masami Hiramatsu (Google) Update fprobe documentation for the new fprobe on function-graph tracer. This includes some bahvior changes and pt_regs to ftrace_regs interface change. Signed-off-by: Masami Hiramatsu (Google) --- Changes in v2: - Update @fregs parameter explanation. --- Documentation/trace/fprobe.rst | 42 ++++++++++++++++++++++++++-------------- 1 file changed, 27 insertions(+), 15 deletions(-) diff --git a/Documentation/trace/fprobe.rst b/Documentation/trace/fprobe.rst index 196f52386aaa..f58bdc64504f 100644 --- a/Documentation/trace/fprobe.rst +++ b/Documentation/trace/fprobe.rst @@ -9,9 +9,10 @@ Fprobe - Function entry/exit probe Introduction ============ -Fprobe is a function entry/exit probe mechanism based on ftrace. -Instead of using ftrace full feature, if you only want to attach callbacks -on function entry and exit, similar to the kprobes and kretprobes, you can +Fprobe is a function entry/exit probe mechanism based on the function-graph +tracer. +Instead of tracing all functions, if you want to attach callbacks on specific +function entry and exit, similar to the kprobes and kretprobes, you can use fprobe. Compared with kprobes and kretprobes, fprobe gives faster instrumentation for multiple functions with single handler. This document describes how to use fprobe. @@ -91,12 +92,14 @@ The prototype of the entry/exit callback function are as follows: .. code-block:: c - int entry_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long ret_ip, struct pt_regs *regs, void *entry_data); + int entry_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long ret_ip, struct ftrace_regs *fregs, void *entry_data); - void exit_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long ret_ip, struct pt_regs *regs, void *entry_data); + void exit_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long ret_ip, struct ftrace_regs *fregs, void *entry_data); -Note that the @entry_ip is saved at function entry and passed to exit handler. -If the entry callback function returns !0, the corresponding exit callback will be cancelled. +Note that the @entry_ip is saved at function entry and passed to exit +handler. +If the entry callback function returns !0, the corresponding exit callback +will be cancelled. @fp This is the address of `fprobe` data structure related to this handler. @@ -112,12 +115,10 @@ If the entry callback function returns !0, the corresponding exit callback will This is the return address that the traced function will return to, somewhere in the caller. This can be used at both entry and exit. -@regs - This is the `pt_regs` data structure at the entry and exit. Note that - the instruction pointer of @regs may be different from the @entry_ip - in the entry_handler. If you need traced instruction pointer, you need - to use @entry_ip. On the other hand, in the exit_handler, the instruction - pointer of @regs is set to the current return address. +@fregs + This is the `ftrace_regs` data structure at the entry and exit. This + includes the function parameters, or the return values. So user can + access thos values via appropriate `ftrace_regs_*` APIs. @entry_data This is a local storage to share the data between entry and exit handlers. @@ -125,6 +126,17 @@ If the entry callback function returns !0, the corresponding exit callback will and `entry_data_size` field when registering the fprobe, the storage is allocated and passed to both `entry_handler` and `exit_handler`. +Entry data size and exit handlers on the same function +====================================================== + +Since the entry data is passed via per-task stack and it is has limited size, +the entry data size per probe is limited to `15 * sizeof(long)`. You also need +to take care that the different fprobes are probing on the same function, this +limit becomes smaller. The entry data size is aligned to `sizeof(long)` and +each fprobe which has exit handler uses a `sizeof(long)` space on the stack, +you should keep the number of fprobes on the same function as small as +possible. + Share the callbacks with kprobes ================================ @@ -165,8 +177,8 @@ This counter counts up when; - fprobe fails to take ftrace_recursion lock. This usually means that a function which is traced by other ftrace users is called from the entry_handler. - - fprobe fails to setup the function exit because of the shortage of rethook - (the shadow stack for hooking the function return.) + - fprobe fails to setup the function exit because of failing to allocate the + data buffer from the per-task shadow stack. The `fprobe::nmissed` field counts up in both cases. Therefore, the former skips both of entry and exit callback and the latter skips the exit