From patchwork Fri Dec 8 10:24:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 175729 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5362025vqy; Fri, 8 Dec 2023 02:24:47 -0800 (PST) X-Google-Smtp-Source: AGHT+IGNOhT7iu2WgkO9Js3q/iubxhvn/y5FNlUxaFtib5lWM5OqHNrXE523s6Dd2Zcgb26bh8JD X-Received: by 2002:a17:90a:e293:b0:286:6cc1:2cc2 with SMTP id d19-20020a17090ae29300b002866cc12cc2mr3293311pjz.76.1702031087487; Fri, 08 Dec 2023 02:24:47 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702031087; cv=none; d=google.com; s=arc-20160816; b=eCC05BHV1Mnhh9hct3VvrlCJ8xG9vQoamPCJ+crK/EG/QTzj6S3qHiLbMgBKVXT4vD dYLC/CryZsG11kXq+9EdtaoklIdHq2jgsTnTldRg1FkMj3Xz66VQ7mJNg8N6dISGZM2b 56GEQy2hJOE+zgctRr0Hkf99sBLk64EUQsOknP8ZIzukoq3JLcqJUHjIZ1fW+K1J1Qjx UfagGPbuiD89It/9eZaVMbPuqzPd5zm7+W4rNiX5EVlIUwLfuUX0UT728ck3TQ+c3ebA rMZibYuG28s4mXizCbD9baF5Ygo2TfTa4GRq+fcr+An5VUXA+w06Wt6K4L2+TOy3rlTm fsFQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=u/kik4VLGZC2GW07eIANh56Vc+hGBXm7uOb3XXHxgl4=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=v9ORnIAs9AHIjOgPwxWIbQVREcPPGV4CfLw1D+y+SzcW/gaOUl10jGemScsj6PxjeP SdC2LDkv7xjYzQJN3xpHpKNy7pu/792ds5DVKpCkBCYnBTgFL6VF3C3hQLxr237F2vzb PTIXsSAv67ZPhhaeDsbVQJU6Pt4ZiSA46Zd4F086GZYpTLwStHveDimudXPwiak5lQXa y0Gw0XZoKY43vSdlinHNYnACbKgNvCj8wXNFSwEitBDKQhnXjn5YS5TQw+NWgaNCpLBg chXl0ycqQ8AYzbvRV0NyDBCzjfjuxs0pkMepEIP8YcFJpuocKfVnYnwlrp2tH1bB3NAB tICA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=GKgxwFmt; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from agentk.vger.email (agentk.vger.email. [23.128.96.32]) by mx.google.com with ESMTPS id z3-20020a170903018300b001cfb4ceb73bsi1422884plg.569.2023.12.08.02.24.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 02:24:47 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=GKgxwFmt; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id C46DA80F6979; Fri, 8 Dec 2023 02:24:44 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1573518AbjLHKYb (ORCPT + 99 others); Fri, 8 Dec 2023 05:24:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35548 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1573512AbjLHKY2 (ORCPT ); Fri, 8 Dec 2023 05:24:28 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A4DE41727 for ; Fri, 8 Dec 2023 02:24:34 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8BB76C433B7; Fri, 8 Dec 2023 10:24:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702031074; bh=HAB8fcsYhkztOqkqxJN8HQ+Use/Y/czjonQXwZoidcQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GKgxwFmt6nupNw1bNem/yFo57h5McuNWSuBP6GQxThVplbHcJuHWnlU6QzOH+QjSN TMllc+YsCJffIAtbl4ZDabzD9ROAuUiXeKWEYH9h1+9ZqeuKouqD6pW91hbLbnjcRB t5JNbMzYMhefnjEfPpqVc00ezORgFZ1FqQ2xq6NWPnjrQLCqKRghDoG8V5v2K6Vu0j 4vpEWbmsG0cccQ10CqgSNfl5fX3tomBQCEtgsJFjFI1P5hcMPYLrm8yBKmoxd4VLRN whyKOrWZkLM9vEgzYZm4VTBwBeyeZKXIlSjfKdtVFXldsqkuZa91d/se7yxKvpTPTc JaFpjLo6agHbw== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v4 01/33] tracing: Add a comment about ftrace_regs definition Date: Fri, 8 Dec 2023 19:24:28 +0900 Message-Id: <170203106761.579004.12087604586094043124.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170203105427.579004.8033550792660734570.stgit@devnote2> References: <170203105427.579004.8033550792660734570.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Fri, 08 Dec 2023 02:24:44 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784708949298532536 X-GMAIL-MSGID: 1784708949298532536 From: Masami Hiramatsu (Google) To clarify what will be expected on ftrace_regs, add a comment to the architecture independent definition of the ftrace_regs. Signed-off-by: Masami Hiramatsu (Google) --- Changes in v3: - Add instruction pointer Changes in v2: - newly added. --- include/linux/ftrace.h | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index e8921871ef9a..8b48fc621ea0 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -118,6 +118,32 @@ extern int ftrace_enabled; #ifndef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS +/** + * ftrace_regs - ftrace partial/optimal register set + * + * ftrace_regs represents a group of registers which is used at the + * function entry and exit. There are three types of registers. + * + * - Registers for passing the parameters to callee, including the stack + * pointer. (e.g. rcx, rdx, rdi, rsi, r8, r9 and rsp on x86_64) + * - Registers for passing the return values to caller. + * (e.g. rax and rdx on x86_64) + * - Registers for hooking the function call and return including the + * frame pointer (the frame pointer is architecture/config dependent) + * (e.g. rip, rbp and rsp for x86_64) + * + * Also, architecture dependent fields can be used for internal process. + * (e.g. orig_ax on x86_64) + * + * On the function entry, those registers will be restored except for + * the stack pointer, so that user can change the function parameters + * and instruction pointer (e.g. live patching.) + * On the function exit, only registers which is used for return values + * are restored. + * + * NOTE: user *must not* access regs directly, only do it via APIs, because + * the member can be changed according to the architecture. + */ struct ftrace_regs { struct pt_regs regs; }; From patchwork Fri Dec 8 10:24:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 175731 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5362099vqy; Fri, 8 Dec 2023 02:24:59 -0800 (PST) X-Google-Smtp-Source: AGHT+IGWoNTcSzxhCQw15DP+RWKF0qbzTBuk4fwXcr+lMCY9OlV4+1j0WC94CFg2ymtyZkam3IEj X-Received: by 2002:a05:6a21:3102:b0:18c:90b1:7bdf with SMTP id yz2-20020a056a21310200b0018c90b17bdfmr5875131pzb.53.1702031099358; Fri, 08 Dec 2023 02:24:59 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702031099; cv=none; d=google.com; s=arc-20160816; b=Oi2sXZlZJSnbY9Xf5h6fbayHKRDbqbOTQAliIYRXAuHqeGSBz8bduT5fhkRlIv2l4a iEAo8XV3NmaG6F2PXit+8VrXNDMnI/OT2Xs5M+lpvQDjpY7BIaBOqP/ikjIFNLNE1l0I JS32qVj9XjvGLZGUUl/NJ3/8nSiAluklh909+y4cNHluX01FB+VqLLEMwn3anHy9/5zW L/pQCxmNFmd35z4qcFbWVIxyjunhYWe3GVQdOd6/G2jZCC6xRMWT80upr45V5FGTDTS6 93roIAPwoJaB8d4bqDU8u6XjfwDupA77ND0XzjRkSTDA0g31JgAM9rWeN8g4CXgYJqw0 G6BQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=6u8M/yEtWeoJVCEoLcNsgpoGnmlGQaiNnoviOD6fD5k=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=ghGDhYONe2oTWhrjt/LfU2+eVkswxONKMA5XYrEhYuaAjtauq2SHbABiV9q6qRDHkO ukIG+OhtG072N+HvaYk4eISg/JNzetbYqRC6vRUXktTEZwIEJiee6us0jrK+HSGU8wUE OqrPUIdz+kuUmrgG3L6EPng2k4S/oSI5WsW/pJAacxjVn2SN2MhbWzU3IliJGMaa+Xhs Lnwplrf6nhn/ab2RNVhkLxsuibwu8WbP6656qluXpKjP6dcoVsFTuI/cFZMhJpVSTxoN +sgGmdCER+HAJLGEZX+onHct06vQsls7jbkMmCl6wMYIwnpvaUbATKaDZr6Q/hgA4sNZ wuaQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Y1qtLnmk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from lipwig.vger.email (lipwig.vger.email. [2620:137:e000::3:3]) by mx.google.com with ESMTPS id mj3-20020a17090b368300b00262e5a82047si1456396pjb.44.2023.12.08.02.24.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 02:24:59 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) client-ip=2620:137:e000::3:3; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Y1qtLnmk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 8E3A381945B2; Fri, 8 Dec 2023 02:24:56 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1573515AbjLHKYp (ORCPT + 99 others); Fri, 8 Dec 2023 05:24:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57940 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233315AbjLHKYl (ORCPT ); Fri, 8 Dec 2023 05:24:41 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 703EE1735 for ; Fri, 8 Dec 2023 02:24:47 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9F860C433B6; Fri, 8 Dec 2023 10:24:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702031086; bh=dL9JJoim7F8avDTUtPFPrZxdt2I1oZt3h3FTr1Er5xQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Y1qtLnmkzVzJGf0osX0F12HxdVVw+NsXi2ETuaEW+Vz/DfKMewGmRL7Fvb2Ri+Fjv ciQRh6HAg2Osg+ZER/FXXcvdgZqoaU7nNFurOKXbs7nXWH9wy9s54C/+NY0gGHD+Be aEir0/6kjg3bry8iScjBIT5Ug9+bI1Jq5Zz2lSpf+bm7pNZswA7dUI1sPmBrQWYvs1 53FaxmB8citwzqOC/gB9fOw9dXAPp6usb07ti03TlfE1Azzq6QJUoAWzTDudPw2vnh 8GFYigF5sX+lLqyIDVNjQCODLbrJ0lUTJfsysbIeOdy8yniMQjPxpMcwsYr9UpTzYZ yYeTcddH76ZpQ== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v4 02/33] x86: tracing: Add ftrace_regs definition in the header Date: Fri, 8 Dec 2023 19:24:40 +0900 Message-Id: <170203107969.579004.12737392554145840949.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170203105427.579004.8033550792660734570.stgit@devnote2> References: <170203105427.579004.8033550792660734570.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Fri, 08 Dec 2023 02:24:56 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784708961907929918 X-GMAIL-MSGID: 1784708961907929918 From: Masami Hiramatsu (Google) Add ftrace_regs definition for x86_64 in the ftrace header to clarify what register will be accessible from ftrace_regs. Signed-off-by: Masami Hiramatsu (Google) --- Changes in v3: - Add rip to be saved. Changes in v2: - Newly added. --- arch/x86/include/asm/ftrace.h | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/arch/x86/include/asm/ftrace.h b/arch/x86/include/asm/ftrace.h index 897cf02c20b1..415cf7a2ec2c 100644 --- a/arch/x86/include/asm/ftrace.h +++ b/arch/x86/include/asm/ftrace.h @@ -36,6 +36,12 @@ static inline unsigned long ftrace_call_adjust(unsigned long addr) #ifdef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS struct ftrace_regs { + /* + * On the x86_64, the ftrace_regs saves; + * rax, rcx, rdx, rdi, rsi, r8, r9, rbp, rip and rsp. + * Also orig_ax is used for passing direct trampoline address. + * x86_32 doesn't support ftrace_regs. + */ struct pt_regs regs; }; From patchwork Fri Dec 8 10:24:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 175732 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5362163vqy; Fri, 8 Dec 2023 02:25:10 -0800 (PST) X-Google-Smtp-Source: AGHT+IFDjmEnEQML2LJ54E//xaBiY+l9KYqu5rThhF+jS3baScul655RsjedCT7iHzhIVXAF9So5 X-Received: by 2002:a17:903:11c5:b0:1d0:9416:efb7 with SMTP id q5-20020a17090311c500b001d09416efb7mr4372330plh.108.1702031110023; Fri, 08 Dec 2023 02:25:10 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702031110; cv=none; d=google.com; s=arc-20160816; b=EJtNWRkEZnH0BnL5hFHRhX9e+WXlrYwNBClXc4ihRRWwYW/CXR5/wYbpto4yQcTt15 K/Wyhg4K1GglS2TVpzy80zs8v0X/1lNQ7qlcWMfwjnklIPAaxJk3W9xDC92S7PJWkeY8 PLoyr4heD0laFwxqxaxVU8ROh7VDUvzgclkRsM88e5+V24tFcapOigXO98hrHl3UC2+p k0y+Fhn+/8epd9WHADvej1YzdXpsZ8wHoMkHJ6wiXQF9Q+JMwrc/dWv4znbhP9I9D1eP 38gNsETiphu2FGtu2tKCuiHrxom+z7xtMYcbMll+3Xfsd5FXgSB4KofqUpLBWLvB8a59 nQHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=QLbaEG1HkAbKkwHOtmfbGYNHbzuZtxri/SOYjg2nfyE=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=RAt1UHblcJZ6auCm/ao+bz7I3zwlFvO0j9JL8ZaZf+kMaifaIntSpzt0lp7ypOenXp FrkZ7mAAJpMqxW7RAof5tYbUair9yTDcq1ITBQ1pkL2Ul0s9r0Hw/e0gfJTr+yXRb34b hD3rhmvMDwlfs0rQ8CxfbSjG9Lgq4QdG2T/aKKdKHMyFqksnIbSZm9MTgbgBKyDpWfpW h9be+BrYXwXfj9cM0zIPk4CbjAo8rk9PEoRiKB9tpFmRwiZehgimu+Qf+Hst/m7tSAq/ GbCB1PbLinXL1AGj5Ftr64ZZWQfQa71iuC6usZRJMkOFnH/Wlqd4I9b/HjW0AbNyAUjx TDGg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=DQBUXpWi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id v12-20020a170902b7cc00b001d08d307f89si1353334plz.496.2023.12.08.02.25.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 02:25:10 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=DQBUXpWi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 2126A801B89A; Fri, 8 Dec 2023 02:25:06 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233311AbjLHKY4 (ORCPT + 99 others); Fri, 8 Dec 2023 05:24:56 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60180 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230457AbjLHKYy (ORCPT ); Fri, 8 Dec 2023 05:24:54 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 34803170F for ; Fri, 8 Dec 2023 02:24:59 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0ABC0C433C7; Fri, 8 Dec 2023 10:24:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702031098; bh=z+ddj5bbKkMJhDBQHhx2JzowUtNrB5GUOB5ij9v/Xkk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DQBUXpWi2O9QKN6ZL3IcTz/sP/TfWevL7tQZXSSPR2dxQW8zewHapWj9W0EaIKPNf 14zFHI1cwhCZ1t/BQQvb95EL250ywrKiPtD7imej98DUcM7DVWZm2ndQgA2d6KkVZv fouxu0kGYt1G6LJngiWRZUcxS8cfZRXo0OlhvDjVWHkMjDkpYVtM1zmj6N1Es0v8C3 EP9Ewr6bj04t3yk26HjJVvTo7/DZt+aDh188vrb9ycJP8jNW4V3DzLy6/npNb2/vE1 UyEvIyFMBAaITPEWEbkCOF7Cz3sAg89H2arfYMqhKFzLPJCrPljc/SwKcdy9gM/qck gfvPm9j5+InLA== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v4 03/33] function_graph: Convert ret_stack to a series of longs Date: Fri, 8 Dec 2023 19:24:52 +0900 Message-Id: <170203109188.579004.874167567953087112.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170203105427.579004.8033550792660734570.stgit@devnote2> References: <170203105427.579004.8033550792660734570.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Fri, 08 Dec 2023 02:25:06 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784708972915218920 X-GMAIL-MSGID: 1784708972915218920 From: Steven Rostedt (VMware) In order to make it possible to have multiple callbacks registered with the function_graph tracer, the retstack needs to be converted from an array of ftrace_ret_stack structures to an array of longs. This will allow to store the list of callbacks on the stack for the return side of the functions. Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- include/linux/sched.h | 2 - kernel/trace/fgraph.c | 124 ++++++++++++++++++++++++++++--------------------- 2 files changed, 71 insertions(+), 55 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 292c31697248..4dab30f00211 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1390,7 +1390,7 @@ struct task_struct { int curr_ret_depth; /* Stack of return addresses for return function tracing: */ - struct ftrace_ret_stack *ret_stack; + unsigned long *ret_stack; /* Timestamp for last schedule: */ unsigned long long ftrace_timestamp; diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index c83c005e654e..30edeb6d4aa9 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -25,6 +25,18 @@ #define ASSIGN_OPS_HASH(opsname, val) #endif +#define FGRAPH_RET_SIZE sizeof(struct ftrace_ret_stack) +#define FGRAPH_RET_INDEX (ALIGN(FGRAPH_RET_SIZE, sizeof(long)) / sizeof(long)) +#define SHADOW_STACK_SIZE (PAGE_SIZE) +#define SHADOW_STACK_INDEX \ + (ALIGN(SHADOW_STACK_SIZE, sizeof(long)) / sizeof(long)) +/* Leave on a buffer at the end */ +#define SHADOW_STACK_MAX_INDEX (SHADOW_STACK_INDEX - FGRAPH_RET_INDEX) + +#define RET_STACK(t, index) ((struct ftrace_ret_stack *)(&(t)->ret_stack[index])) +#define RET_STACK_INC(c) ({ c += FGRAPH_RET_INDEX; }) +#define RET_STACK_DEC(c) ({ c -= FGRAPH_RET_INDEX; }) + DEFINE_STATIC_KEY_FALSE(kill_ftrace_graph); int ftrace_graph_active; @@ -69,6 +81,7 @@ static int ftrace_push_return_trace(unsigned long ret, unsigned long func, unsigned long frame_pointer, unsigned long *retp) { + struct ftrace_ret_stack *ret_stack; unsigned long long calltime; int index; @@ -85,23 +98,25 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func, smp_rmb(); /* The return trace stack is full */ - if (current->curr_ret_stack == FTRACE_RETFUNC_DEPTH - 1) { + if (current->curr_ret_stack >= SHADOW_STACK_MAX_INDEX) { atomic_inc(¤t->trace_overrun); return -EBUSY; } calltime = trace_clock_local(); - index = ++current->curr_ret_stack; + index = current->curr_ret_stack; + RET_STACK_INC(current->curr_ret_stack); + ret_stack = RET_STACK(current, index); barrier(); - current->ret_stack[index].ret = ret; - current->ret_stack[index].func = func; - current->ret_stack[index].calltime = calltime; + ret_stack->ret = ret; + ret_stack->func = func; + ret_stack->calltime = calltime; #ifdef HAVE_FUNCTION_GRAPH_FP_TEST - current->ret_stack[index].fp = frame_pointer; + ret_stack->fp = frame_pointer; #endif #ifdef HAVE_FUNCTION_GRAPH_RET_ADDR_PTR - current->ret_stack[index].retp = retp; + ret_stack->retp = retp; #endif return 0; } @@ -148,7 +163,7 @@ int function_graph_enter(unsigned long ret, unsigned long func, return 0; out_ret: - current->curr_ret_stack--; + RET_STACK_DEC(current->curr_ret_stack); out: current->curr_ret_depth--; return -EBUSY; @@ -159,11 +174,13 @@ static void ftrace_pop_return_trace(struct ftrace_graph_ret *trace, unsigned long *ret, unsigned long frame_pointer) { + struct ftrace_ret_stack *ret_stack; int index; index = current->curr_ret_stack; + RET_STACK_DEC(index); - if (unlikely(index < 0 || index >= FTRACE_RETFUNC_DEPTH)) { + if (unlikely(index < 0 || index > SHADOW_STACK_MAX_INDEX)) { ftrace_graph_stop(); WARN_ON(1); /* Might as well panic, otherwise we have no where to go */ @@ -171,6 +188,7 @@ ftrace_pop_return_trace(struct ftrace_graph_ret *trace, unsigned long *ret, return; } + ret_stack = RET_STACK(current, index); #ifdef HAVE_FUNCTION_GRAPH_FP_TEST /* * The arch may choose to record the frame pointer used @@ -186,22 +204,22 @@ ftrace_pop_return_trace(struct ftrace_graph_ret *trace, unsigned long *ret, * Note, -mfentry does not use frame pointers, and this test * is not needed if CC_USING_FENTRY is set. */ - if (unlikely(current->ret_stack[index].fp != frame_pointer)) { + if (unlikely(ret_stack->fp != frame_pointer)) { ftrace_graph_stop(); WARN(1, "Bad frame pointer: expected %lx, received %lx\n" " from func %ps return to %lx\n", current->ret_stack[index].fp, frame_pointer, - (void *)current->ret_stack[index].func, - current->ret_stack[index].ret); + (void *)ret_stack->func, + ret_stack->ret); *ret = (unsigned long)panic; return; } #endif - *ret = current->ret_stack[index].ret; - trace->func = current->ret_stack[index].func; - trace->calltime = current->ret_stack[index].calltime; + *ret = ret_stack->ret; + trace->func = ret_stack->func; + trace->calltime = ret_stack->calltime; trace->overrun = atomic_read(¤t->trace_overrun); trace->depth = current->curr_ret_depth--; /* @@ -262,7 +280,7 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs * curr_ret_stack is after that. */ barrier(); - current->curr_ret_stack--; + RET_STACK_DEC(current->curr_ret_stack); if (unlikely(!ret)) { ftrace_graph_stop(); @@ -305,12 +323,13 @@ unsigned long ftrace_return_to_handler(unsigned long frame_pointer) struct ftrace_ret_stack * ftrace_graph_get_ret_stack(struct task_struct *task, int idx) { - idx = task->curr_ret_stack - idx; + int index = task->curr_ret_stack; - if (idx >= 0 && idx <= task->curr_ret_stack) - return &task->ret_stack[idx]; + index -= FGRAPH_RET_INDEX * (idx + 1); + if (index < 0) + return NULL; - return NULL; + return RET_STACK(task, index); } /** @@ -332,18 +351,20 @@ ftrace_graph_get_ret_stack(struct task_struct *task, int idx) unsigned long ftrace_graph_ret_addr(struct task_struct *task, int *idx, unsigned long ret, unsigned long *retp) { + struct ftrace_ret_stack *ret_stack; int index = task->curr_ret_stack; int i; if (ret != (unsigned long)dereference_kernel_function_descriptor(return_to_handler)) return ret; - if (index < 0) - return ret; + RET_STACK_DEC(index); - for (i = 0; i <= index; i++) - if (task->ret_stack[i].retp == retp) - return task->ret_stack[i].ret; + for (i = index; i >= 0; RET_STACK_DEC(i)) { + ret_stack = RET_STACK(task, i); + if (ret_stack->retp == retp) + return ret_stack->ret; + } return ret; } @@ -357,14 +378,15 @@ unsigned long ftrace_graph_ret_addr(struct task_struct *task, int *idx, return ret; task_idx = task->curr_ret_stack; + RET_STACK_DEC(task_idx); if (!task->ret_stack || task_idx < *idx) return ret; task_idx -= *idx; - (*idx)++; + RET_STACK_INC(*idx); - return task->ret_stack[task_idx].ret; + return RET_STACK(task, task_idx); } #endif /* HAVE_FUNCTION_GRAPH_RET_ADDR_PTR */ @@ -402,7 +424,7 @@ trace_func_graph_ent_t ftrace_graph_entry = ftrace_graph_entry_stub; static trace_func_graph_ent_t __ftrace_graph_entry = ftrace_graph_entry_stub; /* Try to assign a return stack array on FTRACE_RETSTACK_ALLOC_SIZE tasks. */ -static int alloc_retstack_tasklist(struct ftrace_ret_stack **ret_stack_list) +static int alloc_retstack_tasklist(unsigned long **ret_stack_list) { int i; int ret = 0; @@ -410,10 +432,7 @@ static int alloc_retstack_tasklist(struct ftrace_ret_stack **ret_stack_list) struct task_struct *g, *t; for (i = 0; i < FTRACE_RETSTACK_ALLOC_SIZE; i++) { - ret_stack_list[i] = - kmalloc_array(FTRACE_RETFUNC_DEPTH, - sizeof(struct ftrace_ret_stack), - GFP_KERNEL); + ret_stack_list[i] = kmalloc(SHADOW_STACK_SIZE, GFP_KERNEL); if (!ret_stack_list[i]) { start = 0; end = i; @@ -431,9 +450,9 @@ static int alloc_retstack_tasklist(struct ftrace_ret_stack **ret_stack_list) if (t->ret_stack == NULL) { atomic_set(&t->trace_overrun, 0); - t->curr_ret_stack = -1; + t->curr_ret_stack = 0; t->curr_ret_depth = -1; - /* Make sure the tasks see the -1 first: */ + /* Make sure the tasks see the 0 first: */ smp_wmb(); t->ret_stack = ret_stack_list[start++]; } @@ -453,6 +472,7 @@ ftrace_graph_probe_sched_switch(void *ignore, bool preempt, struct task_struct *next, unsigned int prev_state) { + struct ftrace_ret_stack *ret_stack; unsigned long long timestamp; int index; @@ -477,8 +497,11 @@ ftrace_graph_probe_sched_switch(void *ignore, bool preempt, */ timestamp -= next->ftrace_timestamp; - for (index = next->curr_ret_stack; index >= 0; index--) - next->ret_stack[index].calltime += timestamp; + for (index = next->curr_ret_stack - FGRAPH_RET_INDEX; index >= 0; ) { + ret_stack = RET_STACK(next, index); + ret_stack->calltime += timestamp; + index -= FGRAPH_RET_INDEX; + } } static int ftrace_graph_entry_test(struct ftrace_graph_ent *trace) @@ -521,10 +544,10 @@ void update_function_graph_func(void) ftrace_graph_entry = __ftrace_graph_entry; } -static DEFINE_PER_CPU(struct ftrace_ret_stack *, idle_ret_stack); +static DEFINE_PER_CPU(unsigned long *, idle_ret_stack); static void -graph_init_task(struct task_struct *t, struct ftrace_ret_stack *ret_stack) +graph_init_task(struct task_struct *t, unsigned long *ret_stack) { atomic_set(&t->trace_overrun, 0); t->ftrace_timestamp = 0; @@ -539,7 +562,7 @@ graph_init_task(struct task_struct *t, struct ftrace_ret_stack *ret_stack) */ void ftrace_graph_init_idle_task(struct task_struct *t, int cpu) { - t->curr_ret_stack = -1; + t->curr_ret_stack = 0; t->curr_ret_depth = -1; /* * The idle task has no parent, it either has its own @@ -549,14 +572,11 @@ void ftrace_graph_init_idle_task(struct task_struct *t, int cpu) WARN_ON(t->ret_stack != per_cpu(idle_ret_stack, cpu)); if (ftrace_graph_active) { - struct ftrace_ret_stack *ret_stack; + unsigned long *ret_stack; ret_stack = per_cpu(idle_ret_stack, cpu); if (!ret_stack) { - ret_stack = - kmalloc_array(FTRACE_RETFUNC_DEPTH, - sizeof(struct ftrace_ret_stack), - GFP_KERNEL); + ret_stack = kmalloc(SHADOW_STACK_SIZE, GFP_KERNEL); if (!ret_stack) return; per_cpu(idle_ret_stack, cpu) = ret_stack; @@ -570,15 +590,13 @@ void ftrace_graph_init_task(struct task_struct *t) { /* Make sure we do not use the parent ret_stack */ t->ret_stack = NULL; - t->curr_ret_stack = -1; + t->curr_ret_stack = 0; t->curr_ret_depth = -1; if (ftrace_graph_active) { - struct ftrace_ret_stack *ret_stack; + unsigned long *ret_stack; - ret_stack = kmalloc_array(FTRACE_RETFUNC_DEPTH, - sizeof(struct ftrace_ret_stack), - GFP_KERNEL); + ret_stack = kmalloc(SHADOW_STACK_SIZE, GFP_KERNEL); if (!ret_stack) return; graph_init_task(t, ret_stack); @@ -587,7 +605,7 @@ void ftrace_graph_init_task(struct task_struct *t) void ftrace_graph_exit_task(struct task_struct *t) { - struct ftrace_ret_stack *ret_stack = t->ret_stack; + unsigned long *ret_stack = t->ret_stack; t->ret_stack = NULL; /* NULL must become visible to IRQs before we free it: */ @@ -599,12 +617,10 @@ void ftrace_graph_exit_task(struct task_struct *t) /* Allocate a return stack for each task */ static int start_graph_tracing(void) { - struct ftrace_ret_stack **ret_stack_list; + unsigned long **ret_stack_list; int ret, cpu; - ret_stack_list = kmalloc_array(FTRACE_RETSTACK_ALLOC_SIZE, - sizeof(struct ftrace_ret_stack *), - GFP_KERNEL); + ret_stack_list = kmalloc(SHADOW_STACK_SIZE, GFP_KERNEL); if (!ret_stack_list) return -ENOMEM; From patchwork Fri Dec 8 10:25:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 175733 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5362237vqy; Fri, 8 Dec 2023 02:25:23 -0800 (PST) X-Google-Smtp-Source: AGHT+IGMOao0dBoX+FHy6AXXSJ5XIKEnwVvxNDnJbrW6WnU7U2tUV6/wYzytsC6NAh/AqYk+IsU1 X-Received: by 2002:a17:90b:3912:b0:286:8672:5199 with SMTP id ob18-20020a17090b391200b0028686725199mr3146588pjb.35.1702031123646; Fri, 08 Dec 2023 02:25:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702031123; cv=none; d=google.com; s=arc-20160816; b=Upp3kPDdtadS9RF3G69gTQ5x8T0eX5/ZQzshejfiPkRvMpO5ItWuZcS86jydrRo+0J 9u75Bhg0RPtiJLt/gaotrfUWidTR27zazQ57F44c6RH+7EnuMaQjq9aI0C5Mngb4LRvB CwGAcDZI05wUdf8UywVsQ4eNf662WmuFvp+l4/qlG3Np+RrClwGAXvcZFVra9wgFhpU1 DAhUiQEL58/BrAP28eewW7CalXY7oqJOYYcXatsN1KrN6YDJ1iBgGnosaPhTc+pAyIk1 7PGNTmqbmjIf+91KKDcSh85xuQZp2pUCX+JDZ0SSP5sFv+DCH/48/rx3wiDlj/U3Nngg OXIQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=OESyYZVxbmArs5V4zMGrIL0/uPgjACK9BAVI1aKMbU0=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=rNVJPsaWm6yXwkthqGGBkZrqfIuEA1NJWWx31HnqkX1oy0xnarCBVz6zVwcwwMyoWs mDpH/QpbAjaTaz/LWIHVZUO8asOoE5duTp1oHYqhPN0gad9B2YHo8bE6LDR5VP0e5QuX toT37mGBe8up6axgf9sN/p/QL5P5YLWIccaEUFbQySVxaKt+Zlpqcw8ozymuVwihuaHD MPc9nzpm1sfVf9c2hJaR1utLs3eBiQeOgvRZdBAYoledyNFusNFI15Km4+Eb99kliQq+ cKb8bX65kV7908V38WnXGuMqM+DGPeP0nQ63H5itZq55NwNY/uVA2xwfjELqxi/NZgu1 LEvw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Rjbs+hxY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from agentk.vger.email (agentk.vger.email. [23.128.96.32]) by mx.google.com with ESMTPS id x18-20020a17090aca1200b00285a62cc148si2726408pjt.134.2023.12.08.02.25.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 02:25:23 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Rjbs+hxY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id C26C880F7E55; Fri, 8 Dec 2023 02:25:20 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1573498AbjLHKZK (ORCPT + 99 others); Fri, 8 Dec 2023 05:25:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42372 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235889AbjLHKZF (ORCPT ); Fri, 8 Dec 2023 05:25:05 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AFB001984 for ; Fri, 8 Dec 2023 02:25:11 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1F007C433C8; Fri, 8 Dec 2023 10:25:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702031111; bh=BJTb9S/48BMXgKg9JdebJFjIf4uCQaHDkO4DGbu+/d0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Rjbs+hxYEcpPGC8MLSK/pRCVRAlHDbE4xyI26zR8oOJ2neJs8NZElyzvWcuAglWyZ 6t7b6lQ+IBpA5CRFFwnlgLfpPBc87ApEPbidq42KRdPkgztPGUlHJeR8VcN/YjwEVm lh+tvQXFVncZc60f9RGHknM4JkopKP1bepIBXGJlr/hnzjytvBohTu+OjXV1AfuSMq KCqVjaiOmFqtAvkAvf+9Tf9odx/Njr1mOhSdQUlj8hok1i9Ud4RlwfHLlxL/nAaTAp i2C3LSiTdAPbqSgF05bRGR+HZIZ6XC1fjKavyeFRHbZB8850iHQ7d1RjfVCDV4zlJJ 51T6M60msuGJA== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v4 04/33] fgraph: Use BUILD_BUG_ON() to make sure we have structures divisible by long Date: Fri, 8 Dec 2023 19:25:04 +0900 Message-Id: <170203110417.579004.1203622490067947084.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170203105427.579004.8033550792660734570.stgit@devnote2> References: <170203105427.579004.8033550792660734570.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Fri, 08 Dec 2023 02:25:21 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784708987602219950 X-GMAIL-MSGID: 1784708987602219950 From: Steven Rostedt (VMware) Instead of using "ALIGN()", use BUILD_BUG_ON() as the structures should always be divisible by sizeof(long). Link: http://lkml.kernel.org/r/20190524111144.GI2589@hirez.programming.kicks-ass.net Suggested-by: Peter Zijlstra Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- kernel/trace/fgraph.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index 30edeb6d4aa9..837daf929d2a 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -26,10 +26,9 @@ #endif #define FGRAPH_RET_SIZE sizeof(struct ftrace_ret_stack) -#define FGRAPH_RET_INDEX (ALIGN(FGRAPH_RET_SIZE, sizeof(long)) / sizeof(long)) +#define FGRAPH_RET_INDEX (FGRAPH_RET_SIZE / sizeof(long)) #define SHADOW_STACK_SIZE (PAGE_SIZE) -#define SHADOW_STACK_INDEX \ - (ALIGN(SHADOW_STACK_SIZE, sizeof(long)) / sizeof(long)) +#define SHADOW_STACK_INDEX (SHADOW_STACK_SIZE / sizeof(long)) /* Leave on a buffer at the end */ #define SHADOW_STACK_MAX_INDEX (SHADOW_STACK_INDEX - FGRAPH_RET_INDEX) @@ -91,6 +90,8 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func, if (!current->ret_stack) return -EBUSY; + BUILD_BUG_ON(SHADOW_STACK_SIZE % sizeof(long)); + /* * We must make sure the ret_stack is tested before we read * anything else. @@ -325,6 +326,8 @@ ftrace_graph_get_ret_stack(struct task_struct *task, int idx) { int index = task->curr_ret_stack; + BUILD_BUG_ON(FGRAPH_RET_SIZE % sizeof(long)); + index -= FGRAPH_RET_INDEX * (idx + 1); if (index < 0) return NULL; From patchwork Fri Dec 8 10:25:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 175734 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5362364vqy; Fri, 8 Dec 2023 02:25:42 -0800 (PST) X-Google-Smtp-Source: AGHT+IFgTV2u8LYGGkb4jtLliUGbQlw9URYbNSmP5ET7H6awEt2Xm46lR1i1zQgpztl2OIEBKaoD X-Received: by 2002:a17:902:f684:b0:1d0:b1a0:8a18 with SMTP id l4-20020a170902f68400b001d0b1a08a18mr3232906plg.3.1702031142617; Fri, 08 Dec 2023 02:25:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702031142; cv=none; d=google.com; s=arc-20160816; b=qIPjY/CaYSR7Zyv2b4FotmIarSdV5UFnx5mmOIzltIMjnul87YG2vdXOHZZV1m6nOa E+hHPDxz2ZWpI3gcxPj7KeSRGssP15eYt1QH59YNXsUg1/kKyujVYOzHOJMQetRe7Y5Q xAtq0pgiP8ol8fCpCNncbZk6EfHBEpxCCzYaNULvcBpbZsXRftlXc7H388ywUHty/O41 JHrJXpa7E0jVhstY3kSrjuUCyLJ4oJGf6tuyXmUKjmvcZA1Vs/LcMA1m+zSwaMYWyhFS 1v6/mMMzX0hqXvE8OcngLJGzRgRB/vuOLTv3WwSGYt2WS7BoPuGmY2ENeqyVc0D2G1AB eQ7A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=USeyGjolvInMzVJ66tLimsGzxCbXQfhEBdfvr5j4pqU=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=Z99zZFETCARdYsGl7If9mEyxpYxaSVfL3rTjPo8yy5I4f/y9+8TInbklJoz1UPe9+Z rc+RyDbVggqXJUT2z+HX0ubmdZWS5LKRtTBUNhpuw47gnrKoFDIG2Wpn6C/mpWEP26eK KzZaV3sv2k7KV59L4gxRe6HP5MBCovGDFmtDkJ0+W8hpor+BzsLCyTGutUGuixqP8pFE MLpA0kGOtZ+ywlGVxKjzbJH1Xwc6XInIYrffoXWK9WD1xSl5dvVJAIV5D22rrY+t28hg foZrjXl8S42H728QXXTwB41HCy4ykm1RcNzNEES6HaX7AGeJPHgq3asNygey3U8LrvVX Wdww== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=VjWnz4q1; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id ju17-20020a170903429100b001cfa70fd337si1321100plb.301.2023.12.08.02.25.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 02:25:42 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=VjWnz4q1; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 743BE80F7E53; Fri, 8 Dec 2023 02:25:38 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1573511AbjLHKZ3 (ORCPT + 99 others); Fri, 8 Dec 2023 05:25:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58284 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233311AbjLHKZV (ORCPT ); Fri, 8 Dec 2023 05:25:21 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C63E719AB for ; Fri, 8 Dec 2023 02:25:23 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 65D1AC433C7; Fri, 8 Dec 2023 10:25:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702031123; bh=51Cq89Jg5fqfgwiJmEALH7euiYhrlJJ1/IPjogh9mj0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VjWnz4q1No83H7npTE+mPCxFndexlMPVDXAV5CWxySZt1gneQwCSONY3FbHqcGL8U CGTt3z6zZ7vvBEpCybVpB5rOF8JiOlawwhO0nHMrhtE+2VBZhEt5g6v1QPIfLGoTHd GiG2jJB5OxemmW53PWDVYwDkJmjz7WwxdKZfr9Kxc4EeFigbNJnv2GNOa4gDV2FNTx BUItdpKeENON5PIDOP/bZrYoVgngfl5z8w4FJgNNpXh2CEkHHwhMcaAaCeS1NsELJ5 5H5FRBr1/8bJURVAvHvGYBGKyGJ8HGu1fJX2Ot0+Ij9+2orXwxIAPS0V4mMCBQktxN pjNBm8isbYIzA== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v4 05/33] function_graph: Add an array structure that will allow multiple callbacks Date: Fri, 8 Dec 2023 19:25:16 +0900 Message-Id: <170203111635.579004.14650732702062825676.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170203105427.579004.8033550792660734570.stgit@devnote2> References: <170203105427.579004.8033550792660734570.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Fri, 08 Dec 2023 02:25:38 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784709007211476187 X-GMAIL-MSGID: 1784709007211476187 From: Steven Rostedt (VMware) Add an array structure that will eventually allow the function graph tracer to have up to 16 simultaneous callbacks attached. It's an array of 16 fgraph_ops pointers, that is assigned when one is registered. On entry of a function the entry of the first item in the array is called, and if it returns zero, then the callback returns non zero if it wants the return callback to be called on exit of the function. The array will simplify the process of having more than one callback attached to the same function, as its index into the array can be stored on the shadow stack. We need to only save the index, because this will allow the fgraph_ops to be freed before the function returns (which may happen if the function call schedule for a long time). Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- Changes in v2: - Remove unneeded brace. --- kernel/trace/fgraph.c | 114 +++++++++++++++++++++++++++++++++++-------------- 1 file changed, 81 insertions(+), 33 deletions(-) diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index 837daf929d2a..86df3ca6964f 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -39,6 +39,11 @@ DEFINE_STATIC_KEY_FALSE(kill_ftrace_graph); int ftrace_graph_active; +static int fgraph_array_cnt; +#define FGRAPH_ARRAY_SIZE 16 + +static struct fgraph_ops *fgraph_array[FGRAPH_ARRAY_SIZE]; + /* Both enabled by default (can be cleared by function_graph tracer flags */ static bool fgraph_sleep_time = true; @@ -62,6 +67,20 @@ int __weak ftrace_disable_ftrace_graph_caller(void) } #endif +int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace) +{ + return 0; +} + +static void ftrace_graph_ret_stub(struct ftrace_graph_ret *trace) +{ +} + +static struct fgraph_ops fgraph_stub = { + .entryfunc = ftrace_graph_entry_stub, + .retfunc = ftrace_graph_ret_stub, +}; + /** * ftrace_graph_stop - set to permanently disable function graph tracing * @@ -159,7 +178,7 @@ int function_graph_enter(unsigned long ret, unsigned long func, goto out; /* Only trace if the calling function expects to */ - if (!ftrace_graph_entry(&trace)) + if (!fgraph_array[0]->entryfunc(&trace)) goto out_ret; return 0; @@ -274,7 +293,7 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs trace.retval = fgraph_ret_regs_return_value(ret_regs); #endif trace.rettime = trace_clock_local(); - ftrace_graph_return(&trace); + fgraph_array[0]->retfunc(&trace); /* * The ftrace_graph_return() may still access the current * ret_stack structure, we need to make sure the update of @@ -410,11 +429,6 @@ void ftrace_graph_sleep_time_control(bool enable) fgraph_sleep_time = enable; } -int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace) -{ - return 0; -} - /* * Simply points to ftrace_stub, but with the proper protocol. * Defined by the linker script in linux/vmlinux.lds.h @@ -652,37 +666,54 @@ static int start_graph_tracing(void) int register_ftrace_graph(struct fgraph_ops *gops) { int ret = 0; + int i; mutex_lock(&ftrace_lock); - /* we currently allow only one tracer registered at a time */ - if (ftrace_graph_active) { + if (!fgraph_array[0]) { + /* The array must always have real data on it */ + for (i = 0; i < FGRAPH_ARRAY_SIZE; i++) + fgraph_array[i] = &fgraph_stub; + } + + /* Look for an available spot */ + for (i = 0; i < FGRAPH_ARRAY_SIZE; i++) { + if (fgraph_array[i] == &fgraph_stub) + break; + } + if (i >= FGRAPH_ARRAY_SIZE) { ret = -EBUSY; goto out; } - register_pm_notifier(&ftrace_suspend_notifier); + fgraph_array[i] = gops; + if (i + 1 > fgraph_array_cnt) + fgraph_array_cnt = i + 1; ftrace_graph_active++; - ret = start_graph_tracing(); - if (ret) { - ftrace_graph_active--; - goto out; - } - ftrace_graph_return = gops->retfunc; + if (ftrace_graph_active == 1) { + register_pm_notifier(&ftrace_suspend_notifier); + ret = start_graph_tracing(); + if (ret) { + ftrace_graph_active--; + goto out; + } + + ftrace_graph_return = gops->retfunc; - /* - * Update the indirect function to the entryfunc, and the - * function that gets called to the entry_test first. Then - * call the update fgraph entry function to determine if - * the entryfunc should be called directly or not. - */ - __ftrace_graph_entry = gops->entryfunc; - ftrace_graph_entry = ftrace_graph_entry_test; - update_function_graph_func(); + /* + * Update the indirect function to the entryfunc, and the + * function that gets called to the entry_test first. Then + * call the update fgraph entry function to determine if + * the entryfunc should be called directly or not. + */ + __ftrace_graph_entry = gops->entryfunc; + ftrace_graph_entry = ftrace_graph_entry_test; + update_function_graph_func(); - ret = ftrace_startup(&graph_ops, FTRACE_START_FUNC_RET); + ret = ftrace_startup(&graph_ops, FTRACE_START_FUNC_RET); + } out: mutex_unlock(&ftrace_lock); return ret; @@ -690,19 +721,36 @@ int register_ftrace_graph(struct fgraph_ops *gops) void unregister_ftrace_graph(struct fgraph_ops *gops) { + int i; + mutex_lock(&ftrace_lock); if (unlikely(!ftrace_graph_active)) goto out; - ftrace_graph_active--; - ftrace_graph_return = ftrace_stub_graph; - ftrace_graph_entry = ftrace_graph_entry_stub; - __ftrace_graph_entry = ftrace_graph_entry_stub; - ftrace_shutdown(&graph_ops, FTRACE_STOP_FUNC_RET); - unregister_pm_notifier(&ftrace_suspend_notifier); - unregister_trace_sched_switch(ftrace_graph_probe_sched_switch, NULL); + for (i = 0; i < fgraph_array_cnt; i++) + if (gops == fgraph_array[i]) + break; + if (i >= fgraph_array_cnt) + goto out; + fgraph_array[i] = &fgraph_stub; + if (i + 1 == fgraph_array_cnt) { + for (; i >= 0; i--) + if (fgraph_array[i] != &fgraph_stub) + break; + fgraph_array_cnt = i + 1; + } + + ftrace_graph_active--; + if (!ftrace_graph_active) { + ftrace_graph_return = ftrace_stub_graph; + ftrace_graph_entry = ftrace_graph_entry_stub; + __ftrace_graph_entry = ftrace_graph_entry_stub; + ftrace_shutdown(&graph_ops, FTRACE_STOP_FUNC_RET); + unregister_pm_notifier(&ftrace_suspend_notifier); + unregister_trace_sched_switch(ftrace_graph_probe_sched_switch, NULL); + } out: mutex_unlock(&ftrace_lock); } From patchwork Fri Dec 8 10:25:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 175735 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5362516vqy; Fri, 8 Dec 2023 02:26:08 -0800 (PST) X-Google-Smtp-Source: AGHT+IGPMCmK8QsCFtS+LjqHCx0wqZH2Nbt2CnIOIUrknz7lNKmCc6rhDa6ixCwqhhU283xpde8j X-Received: by 2002:a05:6830:2a15:b0:6d9:a1a2:f76e with SMTP id y21-20020a0568302a1500b006d9a1a2f76emr4862339otu.42.1702031168627; Fri, 08 Dec 2023 02:26:08 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702031168; cv=none; d=google.com; s=arc-20160816; b=NCbAlPr12L1xbAExAAW5CTRb+IQTev6xi9uujWxsMrs3CEKBUwjRFE8U4RqCd/Hrch 4oZnPCCxis0lB54RR/aIJG90Yjb1zk7wRtjQ56KsjWgg4rjRc+w8ZLE3QDg2OMDfdm6F insTzhuQWecXbZEAEtGCxqrZd1tEM8SGJwc3Yj4Gf2+/1X6hxEpEmTd+hJs69J9wOzB4 khhfIPZMsSq4GIIeYskEWp13EezY8kXK9wlXRhmhqit1+cWT2if+ggwiry1ss7XQOcTU iOHTIVhQA7JFu+boCjUWySGBrUx+C4t+UVWt+cIci/9mpkLT5FngZrwOv9s05zPn+Lzw Lwxw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=VhRYc1ru/7AODdorQn5peZB/+EM5J4hdBj5iHP1ncXY=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=WgCwufZ20blovL4GNrAOEFLPY8B3oc5lwaz39s1Cn6DPKqjcWXtaEvZmaC9kh1mqh0 HywerfPFxwMeADPaHnU4L4g4LLbDYWn2jwI0M4lsxVuYB7/i2WDjQ/Y683QdJ2R9g3sB 7RmobXsNaiftG26W/818D0CWUFqlYpgVFf57skhlmmJKLyr+aeKCcsFy1dLn8s41sOIk i6SefBQERLbKDUbR05iNTG+6VGu4iHcLVnv5cfVO1cYaRMmXXTaQMzDhocI7vATs4H9y Wmd+6i5y+lAn1RB2h+jMTq/UrAcjqSdK6znXd/f0PgWfsHTLEfSfdha3hVG3uTrAnsMb 4f5g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=U0yu+9PR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id s12-20020a65644c000000b005c660acad7asi1303945pgv.4.2023.12.08.02.26.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 02:26:08 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=U0yu+9PR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 0613C80FA852; Fri, 8 Dec 2023 02:26:07 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230353AbjLHKZ4 (ORCPT + 99 others); Fri, 8 Dec 2023 05:25:56 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33894 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1573566AbjLHKZc (ORCPT ); Fri, 8 Dec 2023 05:25:32 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EFB131996 for ; Fri, 8 Dec 2023 02:25:35 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id AB73FC433C7; Fri, 8 Dec 2023 10:25:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702031135; bh=APQJdEZ6pt+ekGviwAieGT/uClX2gyf+ahHvsmx9rEQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=U0yu+9PRCBBENa49LtZ1nKokhrixy77CiGZ+7LuP7Q3w25PrxhXy0zSfzpR2isuRt OiQoh/LKG/tqqeL31QMxyVmLa+Co8F7oAj5JhQJvqNVf4+70z4YxJ1cmaj65BifJrK TnCe0wKPNkWRloU/vFxT4jdYuDkicsM7emYMBnGjSu9Sxme7C0l9neK3OFVxjQV8E/ jxfoEh3/c00gQS9BC95TO2mxMRCVvG2YSFyUOXa+9XvwYdyVE482wBAkNLAzVV64+y CFKAET1I64qi03HfV3QZctTyTYMZE/Wapys+TyfgUtbPPLfV60EhKxCJ9ICnGVvYRi +yEhKV8St29jw== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v4 06/33] function_graph: Allow multiple users to attach to function graph Date: Fri, 8 Dec 2023 19:25:29 +0900 Message-Id: <170203112864.579004.15793024386654348775.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170203105427.579004.8033550792660734570.stgit@devnote2> References: <170203105427.579004.8033550792660734570.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Fri, 08 Dec 2023 02:26:07 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784709034565684526 X-GMAIL-MSGID: 1784709034565684526 From: Steven Rostedt (VMware) Allow for multiple users to attach to function graph tracer at the same time. Only 16 simultaneous users can attach to the tracer. This is because there's an array that stores the pointers to the attached fgraph_ops. When a function being traced is entered, each of the ftrace_ops entryfunc is called and if it returns non zero, its index into the array will be added to the shadow stack. On exit of the function being traced, the shadow stack will contain the indexes of the ftrace_ops on the array that want their retfunc to be called. Because a function may sleep for a long time (if a task sleeps itself), the return of the function may be literally days later. If the ftrace_ops is removed, its place on the array is replaced with a ftrace_ops that contains the stub functions and that will be called when the function finally returns. If another ftrace_ops is added that happens to get the same index into the array, its return function may be called. But that's actually the way things current work with the old function graph tracer. If one tracer is removed and another is added, the new one will get the return calls of the function traced by the previous one, thus this is not a regression. This can be fixed by adding a counter to each time the array item is updated and save that on the shadow stack as well, such that it won't be called if the index saved does not match the index on the array. Note, being able to filter functions when both are called is not completely handled yet, but that shouldn't be too hard to manage. Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- Changes in v2: - Check return value of the ftrace_pop_return_trace() instead of 'ret' since 'ret' is set to the address of panic(). - Fix typo and make lines shorter than 76 chars in description. --- kernel/trace/fgraph.c | 332 +++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 280 insertions(+), 52 deletions(-) diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index 86df3ca6964f..8aba93be11b2 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -27,23 +27,144 @@ #define FGRAPH_RET_SIZE sizeof(struct ftrace_ret_stack) #define FGRAPH_RET_INDEX (FGRAPH_RET_SIZE / sizeof(long)) + +/* + * On entry to a function (via function_graph_enter()), a new ftrace_ret_stack + * is allocated on the task's ret_stack, then each fgraph_ops on the + * fgraph_array[]'s entryfunc is called and if that returns non-zero, the + * index into the fgraph_array[] for that fgraph_ops is added to the ret_stack. + * As the associated ftrace_ret_stack saved for those fgraph_ops needs to + * be found, the index to it is also added to the ret_stack along with the + * index of the fgraph_array[] to each fgraph_ops that needs their retfunc + * called. + * + * The top of the ret_stack (when not empty) will always have a reference + * to the last ftrace_ret_stack saved. All references to the + * ftrace_ret_stack has the format of: + * + * bits: 0 - 13 Index in words from the previous ftrace_ret_stack + * bits: 14 - 15 Type of storage + * 0 - reserved + * 1 - fgraph_array index + * For fgraph_array_index: + * bits: 16 - 23 The fgraph_ops fgraph_array index + * + * That is, at the end of function_graph_enter, if the first and forth + * fgraph_ops on the fgraph_array[] (index 0 and 3) needs their retfunc called + * on the return of the function being traced, this is what will be on the + * task's shadow ret_stack: (the stack grows upward) + * + * | | <- task->curr_ret_stack + * +----------------------------------+ + * | (3 << FGRAPH_ARRAY_SHIFT)|(2) | ( 3 for index of fourth fgraph_ops) + * +----------------------------------+ + * | (0 << FGRAPH_ARRAY_SHIFT)|(1) | ( 0 for index of first fgraph_ops) + * +----------------------------------+ + * | struct ftrace_ret_stack | + * | (stores the saved ret pointer) | + * +----------------------------------+ + * | (X) | (N) | ( N words away from previous ret_stack) + * | | + * + * If a backtrace is required, and the real return pointer needs to be + * fetched, then it looks at the task's curr_ret_stack index, if it + * is greater than zero, it would subtact one, and then mask the value + * on the ret_stack by FGRAPH_RET_INDEX_MASK and subtract FGRAPH_RET_INDEX + * from that, to get the index of the ftrace_ret_stack structure stored + * on the shadow stack. + */ + +#define FGRAPH_RET_INDEX_SIZE 14 +#define FGRAPH_RET_INDEX_MASK ((1 << FGRAPH_RET_INDEX_SIZE) - 1) + + +#define FGRAPH_TYPE_SIZE 2 +#define FGRAPH_TYPE_MASK ((1 << FGRAPH_TYPE_SIZE) - 1) +#define FGRAPH_TYPE_SHIFT FGRAPH_RET_INDEX_SIZE + +enum { + FGRAPH_TYPE_RESERVED = 0, + FGRAPH_TYPE_ARRAY = 1, +}; + +#define FGRAPH_ARRAY_SIZE 16 +#define FGRAPH_ARRAY_MASK ((1 << FGRAPH_ARRAY_SIZE) - 1) +#define FGRAPH_ARRAY_SHIFT (FGRAPH_TYPE_SHIFT + FGRAPH_TYPE_SIZE) + +/* Currently the max stack index can't be more than register callers */ +#define FGRAPH_MAX_INDEX FGRAPH_ARRAY_SIZE + +#define FGRAPH_FRAME_SIZE (FGRAPH_RET_SIZE + FGRAPH_ARRAY_SIZE * (sizeof(long))) +#define FGRAPH_FRAME_INDEX (ALIGN(FGRAPH_FRAME_SIZE, \ + sizeof(long)) / sizeof(long)) #define SHADOW_STACK_SIZE (PAGE_SIZE) #define SHADOW_STACK_INDEX (SHADOW_STACK_SIZE / sizeof(long)) /* Leave on a buffer at the end */ -#define SHADOW_STACK_MAX_INDEX (SHADOW_STACK_INDEX - FGRAPH_RET_INDEX) +#define SHADOW_STACK_MAX_INDEX (SHADOW_STACK_INDEX - (FGRAPH_RET_INDEX + 1)) #define RET_STACK(t, index) ((struct ftrace_ret_stack *)(&(t)->ret_stack[index])) -#define RET_STACK_INC(c) ({ c += FGRAPH_RET_INDEX; }) -#define RET_STACK_DEC(c) ({ c -= FGRAPH_RET_INDEX; }) DEFINE_STATIC_KEY_FALSE(kill_ftrace_graph); int ftrace_graph_active; static int fgraph_array_cnt; -#define FGRAPH_ARRAY_SIZE 16 static struct fgraph_ops *fgraph_array[FGRAPH_ARRAY_SIZE]; +static inline int get_ret_stack_index(struct task_struct *t, int offset) +{ + return current->ret_stack[offset] & FGRAPH_RET_INDEX_MASK; +} + +static inline int get_fgraph_type(struct task_struct *t, int offset) +{ + return (current->ret_stack[offset] >> FGRAPH_TYPE_SHIFT) & + FGRAPH_TYPE_MASK; +} + +static inline int get_fgraph_array(struct task_struct *t, int offset) +{ + return (current->ret_stack[offset] >> FGRAPH_ARRAY_SHIFT) & + FGRAPH_ARRAY_MASK; +} + +/* + * @offset: The index into @t->ret_stack to find the ret_stack entry + * @index: Where to place the index into @t->ret_stack of that entry + * + * Calling this with: + * + * offset = task->curr_ret_stack; + * do { + * ret_stack = get_ret_stack(task, offset, &offset); + * } while (ret_stack); + * + * Will iterate through all the ret_stack entries from curr_ret_stack + * down to the first one. + */ +static inline struct ftrace_ret_stack * +get_ret_stack(struct task_struct *t, int offset, int *index) +{ + int idx; + + BUILD_BUG_ON(FGRAPH_RET_SIZE % sizeof(long)); + + if (offset <= 0) + return NULL; + + idx = get_ret_stack_index(t, offset - 1); + + if (idx <= 0 || idx > FGRAPH_MAX_INDEX) + return NULL; + + offset -= idx + FGRAPH_RET_INDEX; + if (offset < 0) + return NULL; + + *index = offset; + return RET_STACK(t, offset); +} + /* Both enabled by default (can be cleared by function_graph tracer flags */ static bool fgraph_sleep_time = true; @@ -126,9 +247,34 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func, calltime = trace_clock_local(); index = current->curr_ret_stack; - RET_STACK_INC(current->curr_ret_stack); + /* ret offset = 1 ; type = reserved */ + current->ret_stack[index + FGRAPH_RET_INDEX] = 1; ret_stack = RET_STACK(current, index); + ret_stack->ret = ret; + /* + * The unwinders expect curr_ret_stack to point to either zero + * or an index where to find the next ret_stack. Even though the + * ret stack might be bogus, we want to write the ret and the + * index to find the ret_stack before we increment the stack point. + * If an interrupt comes in now before we increment the curr_ret_stack + * it may blow away what we wrote. But that's fine, because the + * index will still be correct (even though the 'ret' won't be). + * What we worry about is the index being correct after we increment + * the curr_ret_stack and before we update that index, as if an + * interrupt comes in and does an unwind stack dump, it will need + * at least a correct index! + */ barrier(); + current->curr_ret_stack += FGRAPH_RET_INDEX + 1; + /* + * This next barrier is to ensure that an interrupt coming in + * will not corrupt what we are about to write. + */ + barrier(); + + /* Still keep it reserved even if an interrupt came in */ + current->ret_stack[index + FGRAPH_RET_INDEX] = 1; + ret_stack->ret = ret; ret_stack->func = func; ret_stack->calltime = calltime; @@ -159,6 +305,12 @@ int function_graph_enter(unsigned long ret, unsigned long func, unsigned long frame_pointer, unsigned long *retp) { struct ftrace_graph_ent trace; + int offset; + int start; + int type; + int val; + int cnt = 0; + int i; #ifndef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS /* @@ -177,38 +329,87 @@ int function_graph_enter(unsigned long ret, unsigned long func, if (ftrace_push_return_trace(ret, func, frame_pointer, retp)) goto out; - /* Only trace if the calling function expects to */ - if (!fgraph_array[0]->entryfunc(&trace)) + /* Use start for the distance to ret_stack (skipping over reserve) */ + start = offset = current->curr_ret_stack - 2; + + for (i = 0; i < fgraph_array_cnt; i++) { + struct fgraph_ops *gops = fgraph_array[i]; + + if (gops == &fgraph_stub) + continue; + + if ((offset == start) && + (current->curr_ret_stack >= SHADOW_STACK_INDEX - 1)) { + atomic_inc(¤t->trace_overrun); + break; + } + if (fgraph_array[i]->entryfunc(&trace)) { + offset = current->curr_ret_stack; + /* Check the top level stored word */ + type = get_fgraph_type(current, offset - 1); + + val = (i << FGRAPH_ARRAY_SHIFT) | + (FGRAPH_TYPE_ARRAY << FGRAPH_TYPE_SHIFT) | + ((offset - start) - 1); + + /* We can reuse the top word if it is reserved */ + if (type == FGRAPH_TYPE_RESERVED) { + current->ret_stack[offset - 1] = val; + cnt++; + continue; + } + val++; + + current->ret_stack[offset] = val; + /* + * Write the value before we increment, so that + * if an interrupt comes in after we increment + * it will still see the value and skip over + * this. + */ + barrier(); + current->curr_ret_stack++; + /* + * Have to write again, in case an interrupt + * came in before the increment and after we + * wrote the value. + */ + barrier(); + current->ret_stack[offset] = val; + cnt++; + } + } + + if (!cnt) goto out_ret; return 0; out_ret: - RET_STACK_DEC(current->curr_ret_stack); + current->curr_ret_stack -= FGRAPH_RET_INDEX + 1; out: current->curr_ret_depth--; return -EBUSY; } /* Retrieve a function return address to the trace stack on thread info.*/ -static void +static struct ftrace_ret_stack * ftrace_pop_return_trace(struct ftrace_graph_ret *trace, unsigned long *ret, unsigned long frame_pointer) { struct ftrace_ret_stack *ret_stack; int index; - index = current->curr_ret_stack; - RET_STACK_DEC(index); + ret_stack = get_ret_stack(current, current->curr_ret_stack, &index); - if (unlikely(index < 0 || index > SHADOW_STACK_MAX_INDEX)) { + if (unlikely(!ret_stack)) { ftrace_graph_stop(); - WARN_ON(1); + WARN(1, "Bad function graph ret_stack pointer: %d", + current->curr_ret_stack); /* Might as well panic, otherwise we have no where to go */ *ret = (unsigned long)panic; - return; + return NULL; } - ret_stack = RET_STACK(current, index); #ifdef HAVE_FUNCTION_GRAPH_FP_TEST /* * The arch may choose to record the frame pointer used @@ -228,12 +429,12 @@ ftrace_pop_return_trace(struct ftrace_graph_ret *trace, unsigned long *ret, ftrace_graph_stop(); WARN(1, "Bad frame pointer: expected %lx, received %lx\n" " from func %ps return to %lx\n", - current->ret_stack[index].fp, + ret_stack->fp, frame_pointer, (void *)ret_stack->func, ret_stack->ret); *ret = (unsigned long)panic; - return; + return NULL; } #endif @@ -241,13 +442,15 @@ ftrace_pop_return_trace(struct ftrace_graph_ret *trace, unsigned long *ret, trace->func = ret_stack->func; trace->calltime = ret_stack->calltime; trace->overrun = atomic_read(¤t->trace_overrun); - trace->depth = current->curr_ret_depth--; + trace->depth = current->curr_ret_depth; /* * We still want to trace interrupts coming in if * max_depth is set to 1. Make sure the decrement is * seen before ftrace_graph_return. */ barrier(); + + return ret_stack; } /* @@ -285,30 +488,47 @@ struct fgraph_ret_regs; static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs, unsigned long frame_pointer) { + struct ftrace_ret_stack *ret_stack; struct ftrace_graph_ret trace; unsigned long ret; + int offset; + int index; + int idx; + int i; + + ret_stack = ftrace_pop_return_trace(&trace, &ret, frame_pointer); + + if (unlikely(!ret_stack)) { + ftrace_graph_stop(); + WARN_ON(1); + /* Might as well panic. What else to do? */ + return (unsigned long)panic; + } - ftrace_pop_return_trace(&trace, &ret, frame_pointer); + trace.rettime = trace_clock_local(); #ifdef CONFIG_FUNCTION_GRAPH_RETVAL trace.retval = fgraph_ret_regs_return_value(ret_regs); #endif - trace.rettime = trace_clock_local(); - fgraph_array[0]->retfunc(&trace); + + offset = current->curr_ret_stack - 1; + index = get_ret_stack_index(current, offset); + + /* index has to be at least one! Optimize for it */ + i = 0; + do { + idx = get_fgraph_array(current, offset - i); + fgraph_array[idx]->retfunc(&trace); + i++; + } while (i < index); + /* * The ftrace_graph_return() may still access the current * ret_stack structure, we need to make sure the update of * curr_ret_stack is after that. */ barrier(); - RET_STACK_DEC(current->curr_ret_stack); - - if (unlikely(!ret)) { - ftrace_graph_stop(); - WARN_ON(1); - /* Might as well panic. What else to do? */ - ret = (unsigned long)panic; - } - + current->curr_ret_stack -= index + FGRAPH_RET_INDEX; + current->curr_ret_depth--; return ret; } @@ -343,15 +563,17 @@ unsigned long ftrace_return_to_handler(unsigned long frame_pointer) struct ftrace_ret_stack * ftrace_graph_get_ret_stack(struct task_struct *task, int idx) { + struct ftrace_ret_stack *ret_stack = NULL; int index = task->curr_ret_stack; - BUILD_BUG_ON(FGRAPH_RET_SIZE % sizeof(long)); - - index -= FGRAPH_RET_INDEX * (idx + 1); if (index < 0) return NULL; - return RET_STACK(task, index); + do { + ret_stack = get_ret_stack(task, index, &index); + } while (ret_stack && --idx >= 0); + + return ret_stack; } /** @@ -374,16 +596,15 @@ unsigned long ftrace_graph_ret_addr(struct task_struct *task, int *idx, unsigned long ret, unsigned long *retp) { struct ftrace_ret_stack *ret_stack; - int index = task->curr_ret_stack; - int i; + int i = task->curr_ret_stack; if (ret != (unsigned long)dereference_kernel_function_descriptor(return_to_handler)) return ret; - RET_STACK_DEC(index); - - for (i = index; i >= 0; RET_STACK_DEC(i)) { - ret_stack = RET_STACK(task, i); + while (i > 0) { + ret_stack = get_ret_stack(current, i, &i); + if (!ret_stack) + break; if (ret_stack->retp == retp) return ret_stack->ret; } @@ -394,21 +615,26 @@ unsigned long ftrace_graph_ret_addr(struct task_struct *task, int *idx, unsigned long ftrace_graph_ret_addr(struct task_struct *task, int *idx, unsigned long ret, unsigned long *retp) { - int task_idx; + struct ftrace_ret_stack *ret_stack; + int task_idx = task->curr_ret_stack; + int i; if (ret != (unsigned long)dereference_kernel_function_descriptor(return_to_handler)) return ret; - task_idx = task->curr_ret_stack; - RET_STACK_DEC(task_idx); - - if (!task->ret_stack || task_idx < *idx) + if (!idx) return ret; - task_idx -= *idx; - RET_STACK_INC(*idx); + i = *idx; + do { + ret_stack = get_ret_stack(task, task_idx, &task_idx); + i--; + } while (i >= 0 && ret_stack); + + if (ret_stack) + return ret_stack->ret; - return RET_STACK(task, task_idx); + return ret; } #endif /* HAVE_FUNCTION_GRAPH_RET_ADDR_PTR */ @@ -514,10 +740,10 @@ ftrace_graph_probe_sched_switch(void *ignore, bool preempt, */ timestamp -= next->ftrace_timestamp; - for (index = next->curr_ret_stack - FGRAPH_RET_INDEX; index >= 0; ) { - ret_stack = RET_STACK(next, index); - ret_stack->calltime += timestamp; - index -= FGRAPH_RET_INDEX; + for (index = next->curr_ret_stack; index > 0; ) { + ret_stack = get_ret_stack(next, index, &index); + if (ret_stack) + ret_stack->calltime += timestamp; } } @@ -568,6 +794,8 @@ graph_init_task(struct task_struct *t, unsigned long *ret_stack) { atomic_set(&t->trace_overrun, 0); t->ftrace_timestamp = 0; + t->curr_ret_stack = 0; + t->curr_ret_depth = -1; /* make curr_ret_stack visible before we add the ret_stack */ smp_wmb(); t->ret_stack = ret_stack; From patchwork Fri Dec 8 10:25:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 175738 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5362874vqy; Fri, 8 Dec 2023 02:27:14 -0800 (PST) X-Google-Smtp-Source: AGHT+IEkx/ia9/aftLBuusIXEDiiUg6bx5dLVIurJG1ScJjey7+23ebMiukzY6TuxDakp9X1e2dR X-Received: by 2002:a17:902:cec4:b0:1d0:83bc:5649 with SMTP id d4-20020a170902cec400b001d083bc5649mr3838523plg.33.1702031234230; Fri, 08 Dec 2023 02:27:14 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702031234; cv=none; d=google.com; s=arc-20160816; b=vhUqaJx43gBI8I9c3sHKNXtVbrDewKfk52lmxPm06b88jYuPou0Xs7Zlk50hps+xd6 A+Wn/d8jXyvJjmxpk5+7XP8jSWWZkvWF/9q/vbjJIMI6x86H7PDTE/NMTyVD99hPdtQ4 9eCkbNuT1SuxnBGQzr8NuWX9abkVwVCMYbzydgiWUlQgx6A04Q1jWdnEDIM1q/N9qvmw xA8YEWMGWo2gj4neGLNOdkkxTAQLr/Fs4j+vMK5wbAGWTycXGvkptvn0Vh1XuQ0WnM04 2/PKtRoJsucO0c/osJmNOsYmF0UQnhF2RUaBIz/17RnYNil/2EnzHkRKm/2wV/OaZfX3 lPgQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=BKS9fUT4VtivkcruO+8vIPrD/eLeaRjrcB7hbsjRBQs=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=andVMOCGOynlo5wDuuCLEfMsn4ehhqKDAXpuVUsnwClZXFszBjP+zovHhlluwI7WJf GHSCBLMJbyQw5LiDtEA2azLrIFJVUlTz6vqSAb08V2WaDiAWR0HCxA9m0Z0GBm0GQK9G 1w1lTotzjB4mf0o5JJ2C9uV+EQFVfczAH64UWwe0YGR07cXW11rKBWG2i8zQUl5lQM7Q RuP6WVeukkAI5I6LHYnBaZWY5+Yp2uxL7y+KepGiw0s2w0ME1zWCwavX+RWLCB/H+Kao SQsL7RSKC7GkLUUUKXFNxhMFhQlPPncx94oRfjBtx2Gnsp2fK1UIrIPFu0WU2JTIRMz/ xqrw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=r6fTkIsh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from morse.vger.email (morse.vger.email. [2620:137:e000::3:1]) by mx.google.com with ESMTPS id o1-20020a170902d4c100b001d0cc15ffddsi1382690plg.154.2023.12.08.02.27.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 02:27:14 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=r6fTkIsh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id A8BF18078664; Fri, 8 Dec 2023 02:26:11 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1573558AbjLHK0A (ORCPT + 99 others); Fri, 8 Dec 2023 05:26:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42350 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1573659AbjLHKZs (ORCPT ); Fri, 8 Dec 2023 05:25:48 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6B31D1995 for ; Fri, 8 Dec 2023 02:25:48 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id F295EC433C8; Fri, 8 Dec 2023 10:25:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702031147; bh=nRRPhz4LGfnAyoZaXqAlhWt3IaLLSNsOo0cAG6Q6RJE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=r6fTkIshaiF5d/mvYgwQhmd5bm1XLKpd0RaLBqOy16be6r0uUiVjwxTww8pdmYl0x YCh0lpNKImo1GXoGuzqOL6m6gJlQfxAD6DAKIDqouiIkUuZ/jKKBf6SxIm2w7LD8ds nj+PW3VcFYmhCmsM1UN5mwpSfc2QoEXLEi97kJ+T+3pu3jyrovPLw8ILVONiow1Vgb 1eLqIElynqf8zAw/WyzUUXblFhHTDhKxeOiD+rVdey56ZfHE38uyzDvK82v/x7b2zo W2R4XxX35FPotd7+fk7lRJr6Nx9sLMuAVTBDspDv4CykNqf67pALI8TAUc32IvQ8Wg 3s9Mq0A+oO9Iw== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v4 07/33] function_graph: Remove logic around ftrace_graph_entry and return Date: Fri, 8 Dec 2023 19:25:41 +0900 Message-Id: <170203114093.579004.15550281331450934218.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170203105427.579004.8033550792660734570.stgit@devnote2> References: <170203105427.579004.8033550792660734570.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Fri, 08 Dec 2023 02:26:11 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784709103768259066 X-GMAIL-MSGID: 1784709103768259066 From: Steven Rostedt (VMware) The function pointers ftrace_graph_entry and ftrace_graph_return are no longer called via the function_graph tracer. Instead, an array structure is now used that will allow for multiple users of the function_graph infrastructure. The variables are still used by the architecture code for non dynamic ftrace configs, where a test is made against them to see if they point to the default stub function or not. This is how the static function tracing knows to call into the function graph tracer infrastructure or not. Two new stub functions are made. entry_run() and return_run(). The ftrace_graph_entry and ftrace_graph_return are set to them respectively when the function graph tracer is enabled, and this will trigger the architecture specific function graph code to be executed. This also requires checking the global_ops hash for all calls into the function_graph tracer. Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- Changes in v2: - Fix typo and make lines shorter than 76 chars in the description. - Remove unneeded return from return_run() function. --- kernel/trace/fgraph.c | 71 +++++++++++----------------------------- kernel/trace/ftrace.c | 2 - kernel/trace/ftrace_internal.h | 2 - 3 files changed, 19 insertions(+), 56 deletions(-) diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index 8aba93be11b2..97a9ffb8bb4c 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -128,6 +128,17 @@ static inline int get_fgraph_array(struct task_struct *t, int offset) FGRAPH_ARRAY_MASK; } +/* ftrace_graph_entry set to this to tell some archs to run function graph */ +static int entry_run(struct ftrace_graph_ent *trace) +{ + return 0; +} + +/* ftrace_graph_return set to this to tell some archs to run function graph */ +static void return_run(struct ftrace_graph_ret *trace) +{ +} + /* * @offset: The index into @t->ret_stack to find the ret_stack entry * @index: Where to place the index into @t->ret_stack of that entry @@ -323,6 +334,10 @@ int function_graph_enter(unsigned long ret, unsigned long func, ftrace_find_rec_direct(ret - MCOUNT_INSN_SIZE)) return -EBUSY; #endif + + if (!ftrace_ops_test(&global_ops, func, NULL)) + return -EBUSY; + trace.func = func; trace.depth = ++current->curr_ret_depth; @@ -664,7 +679,6 @@ extern void ftrace_stub_graph(struct ftrace_graph_ret *); /* The callbacks that hook a function */ trace_func_graph_ret_t ftrace_graph_return = ftrace_stub_graph; trace_func_graph_ent_t ftrace_graph_entry = ftrace_graph_entry_stub; -static trace_func_graph_ent_t __ftrace_graph_entry = ftrace_graph_entry_stub; /* Try to assign a return stack array on FTRACE_RETSTACK_ALLOC_SIZE tasks. */ static int alloc_retstack_tasklist(unsigned long **ret_stack_list) @@ -747,46 +761,6 @@ ftrace_graph_probe_sched_switch(void *ignore, bool preempt, } } -static int ftrace_graph_entry_test(struct ftrace_graph_ent *trace) -{ - if (!ftrace_ops_test(&global_ops, trace->func, NULL)) - return 0; - return __ftrace_graph_entry(trace); -} - -/* - * The function graph tracer should only trace the functions defined - * by set_ftrace_filter and set_ftrace_notrace. If another function - * tracer ops is registered, the graph tracer requires testing the - * function against the global ops, and not just trace any function - * that any ftrace_ops registered. - */ -void update_function_graph_func(void) -{ - struct ftrace_ops *op; - bool do_test = false; - - /* - * The graph and global ops share the same set of functions - * to test. If any other ops is on the list, then - * the graph tracing needs to test if its the function - * it should call. - */ - do_for_each_ftrace_op(op, ftrace_ops_list) { - if (op != &global_ops && op != &graph_ops && - op != &ftrace_list_end) { - do_test = true; - /* in double loop, break out with goto */ - goto out; - } - } while_for_each_ftrace_op(op); - out: - if (do_test) - ftrace_graph_entry = ftrace_graph_entry_test; - else - ftrace_graph_entry = __ftrace_graph_entry; -} - static DEFINE_PER_CPU(unsigned long *, idle_ret_stack); static void @@ -927,18 +901,12 @@ int register_ftrace_graph(struct fgraph_ops *gops) ftrace_graph_active--; goto out; } - - ftrace_graph_return = gops->retfunc; - /* - * Update the indirect function to the entryfunc, and the - * function that gets called to the entry_test first. Then - * call the update fgraph entry function to determine if - * the entryfunc should be called directly or not. + * Some archs just test to see if these are not + * the default function */ - __ftrace_graph_entry = gops->entryfunc; - ftrace_graph_entry = ftrace_graph_entry_test; - update_function_graph_func(); + ftrace_graph_return = return_run; + ftrace_graph_entry = entry_run; ret = ftrace_startup(&graph_ops, FTRACE_START_FUNC_RET); } @@ -974,7 +942,6 @@ void unregister_ftrace_graph(struct fgraph_ops *gops) if (!ftrace_graph_active) { ftrace_graph_return = ftrace_stub_graph; ftrace_graph_entry = ftrace_graph_entry_stub; - __ftrace_graph_entry = ftrace_graph_entry_stub; ftrace_shutdown(&graph_ops, FTRACE_STOP_FUNC_RET); unregister_pm_notifier(&ftrace_suspend_notifier); unregister_trace_sched_switch(ftrace_graph_probe_sched_switch, NULL); diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 8de8bec5f366..fd64021ec52f 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -235,8 +235,6 @@ static void update_ftrace_function(void) func = ftrace_ops_list_func; } - update_function_graph_func(); - /* If there's no change, then do nothing more here */ if (ftrace_trace_function == func) return; diff --git a/kernel/trace/ftrace_internal.h b/kernel/trace/ftrace_internal.h index 5012c04f92c0..19eddcb91584 100644 --- a/kernel/trace/ftrace_internal.h +++ b/kernel/trace/ftrace_internal.h @@ -42,10 +42,8 @@ ftrace_ops_test(struct ftrace_ops *ops, unsigned long ip, void *regs) #ifdef CONFIG_FUNCTION_GRAPH_TRACER extern int ftrace_graph_active; -void update_function_graph_func(void); #else /* !CONFIG_FUNCTION_GRAPH_TRACER */ # define ftrace_graph_active 0 -static inline void update_function_graph_func(void) { } #endif /* CONFIG_FUNCTION_GRAPH_TRACER */ #else /* !CONFIG_FUNCTION_TRACER */ From patchwork Fri Dec 8 10:25:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 175736 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5362596vqy; Fri, 8 Dec 2023 02:26:26 -0800 (PST) X-Google-Smtp-Source: AGHT+IFlsxBcFju1U8CiHT1+3FXrCGRiM2RYPD/o4SZE8m9h7L8Kno0dk0mvv735FyfEIyfxhmrU X-Received: by 2002:a05:6358:71c3:b0:170:17ea:f4dc with SMTP id u3-20020a05635871c300b0017017eaf4dcmr4707682rwu.41.1702031186129; Fri, 08 Dec 2023 02:26:26 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702031186; cv=none; d=google.com; s=arc-20160816; b=BSj7CGAAESyhZjijPGqElBHFntB2Dti0sHtVY+qUCVEoWTKZFy1nXQkJ5ESkzLnuVF 2J780FajhZsKqjkPyLvyd9wegElxLUyXZvGhOMZLBTuclfeqY/B6JH99IglwlY5RYPNq /7lfVDBh6pqUm0OYeugKY15DersVEQx/1yJubmmsU8zYG0UWCJ8Ix/1bJHKGBaXi1v8M TI/O36Jzw82ITv3i88rjoNdAqmfft5iJ0py0gIDs4/JMUI4tu8Lc7DcXSTJQoc1k5LlM fkASWXsYa87+gQKdAaXj77R4wfmMwJ2oZdh8KokB7Bqv8drlOMGy8ZbMEwnDaushN+EE ORvA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=4krdVOe01YeqoXNjOc2D28F20ROfg0wXmnjT1Av++xw=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=X8JDfutcoiMhF6RMaXUn1JIItZ57qUXqSpF6o74L/CMyPAsuOCOOHtb3MAbzdf0yMD +/R28cShIBBRQgSgmAZzKJ8WLtuV98NEea3pLF9/0VVZJiduBH+9BdeX0v8e2iOMbP5u 5Y0EMTnVQ7pNv16oFHQzSEY+PoV4VaQzyOG5hQtNQy3xqEMV0gh1OvL1B9UWXmte27B/ 0V3QYxB9AHw+SKYy/FKf6Pjnvd30MAKTfGcZW0C5zFw7hzTUMjjWNlv1A6nmHLwjTLQ1 opp9isdQb4n+zucqrLL7hZsy/hFJ5Mt032W4WQA4Cu+FAq2BKq4+nnDYGdv/QKPZ7WBa L0Rg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=FvN1qA1g; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from pete.vger.email (pete.vger.email. [2620:137:e000::3:6]) by mx.google.com with ESMTPS id j18-20020a632312000000b005c676beba09si1279884pgj.74.2023.12.08.02.26.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 02:26:26 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) client-ip=2620:137:e000::3:6; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=FvN1qA1g; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 1881682A9BBD; Fri, 8 Dec 2023 02:26:17 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1573620AbjLHK0F (ORCPT + 99 others); Fri, 8 Dec 2023 05:26:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49260 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1573717AbjLHKZ4 (ORCPT ); Fri, 8 Dec 2023 05:25:56 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6E75A1733 for ; Fri, 8 Dec 2023 02:26:00 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 446AAC433C8; Fri, 8 Dec 2023 10:25:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702031160; bh=DxvrCuVm5CFgY/QPDT747/WL1/q6+2sScxvA4J8z3ws=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=FvN1qA1g9bMTceyP/xWNC4TmjWqbd7cr1y+E7F5PWHczrl5kE1XaAYOerS+8oRicU 5cp8Uh9xzqXGMF086trTMad7/yXjSupFu84I1xooMuh8QT14fk1kRtjWz7ddoFY0IU HEvBZoNZH8tyGZoj5xYBxb5KtCGkzLkXBNjS/6YEn867zbq6ixxFp0QxQwwr5rPRrV wo40IAnH31+NF4GUqIyMAhKkYayeh4ooZx747Zsb3Qwip4MiEHb5tQC9Wjdqh/uZNr wTMHPCnReNiJp/x8hmEOgD3swsYMgtGeQ+M2so+1zJv4FCcTOhuwQ7hHP29bNXeYxD bf+/2p7ufkHUQ== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v4 08/33] ftrace/function_graph: Pass fgraph_ops to function graph callbacks Date: Fri, 8 Dec 2023 19:25:53 +0900 Message-Id: <170203115321.579004.17389638059690297680.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170203105427.579004.8033550792660734570.stgit@devnote2> References: <170203105427.579004.8033550792660734570.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Fri, 08 Dec 2023 02:26:17 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784709053256437584 X-GMAIL-MSGID: 1784709053256437584 From: Steven Rostedt (VMware) Pass the fgraph_ops structure to the function graph callbacks. This will allow callbacks to add a descriptor to a fgraph_ops private field that wil be added in the future and use it for the callbacks. This will be useful when more than one callback can be registered to the function graph tracer. Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- Changes in v2: - cleanup to set argument name on function prototype. --- include/linux/ftrace.h | 10 +++++++--- kernel/trace/fgraph.c | 16 +++++++++------- kernel/trace/ftrace.c | 6 ++++-- kernel/trace/trace.h | 4 ++-- kernel/trace/trace_functions_graph.c | 11 +++++++---- kernel/trace/trace_irqsoff.c | 6 ++++-- kernel/trace/trace_sched_wakeup.c | 6 ++++-- kernel/trace/trace_selftest.c | 5 +++-- 8 files changed, 40 insertions(+), 24 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 8b48fc621ea0..bc216cdc742d 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -1055,11 +1055,15 @@ struct ftrace_graph_ret { unsigned long long rettime; } __packed; +struct fgraph_ops; + /* Type of the callback handlers for tracing function graph*/ -typedef void (*trace_func_graph_ret_t)(struct ftrace_graph_ret *); /* return */ -typedef int (*trace_func_graph_ent_t)(struct ftrace_graph_ent *); /* entry */ +typedef void (*trace_func_graph_ret_t)(struct ftrace_graph_ret *, + struct fgraph_ops *); /* return */ +typedef int (*trace_func_graph_ent_t)(struct ftrace_graph_ent *, + struct fgraph_ops *); /* entry */ -extern int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace); +extern int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace, struct fgraph_ops *gops); #ifdef CONFIG_FUNCTION_GRAPH_TRACER diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index 97a9ffb8bb4c..62c35d6d95f9 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -129,13 +129,13 @@ static inline int get_fgraph_array(struct task_struct *t, int offset) } /* ftrace_graph_entry set to this to tell some archs to run function graph */ -static int entry_run(struct ftrace_graph_ent *trace) +static int entry_run(struct ftrace_graph_ent *trace, struct fgraph_ops *ops) { return 0; } /* ftrace_graph_return set to this to tell some archs to run function graph */ -static void return_run(struct ftrace_graph_ret *trace) +static void return_run(struct ftrace_graph_ret *trace, struct fgraph_ops *ops) { } @@ -199,12 +199,14 @@ int __weak ftrace_disable_ftrace_graph_caller(void) } #endif -int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace) +int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace, + struct fgraph_ops *gops) { return 0; } -static void ftrace_graph_ret_stub(struct ftrace_graph_ret *trace) +static void ftrace_graph_ret_stub(struct ftrace_graph_ret *trace, + struct fgraph_ops *gops) { } @@ -358,7 +360,7 @@ int function_graph_enter(unsigned long ret, unsigned long func, atomic_inc(¤t->trace_overrun); break; } - if (fgraph_array[i]->entryfunc(&trace)) { + if (fgraph_array[i]->entryfunc(&trace, fgraph_array[i])) { offset = current->curr_ret_stack; /* Check the top level stored word */ type = get_fgraph_type(current, offset - 1); @@ -532,7 +534,7 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs i = 0; do { idx = get_fgraph_array(current, offset - i); - fgraph_array[idx]->retfunc(&trace); + fgraph_array[idx]->retfunc(&trace, fgraph_array[idx]); i++; } while (i < index); @@ -674,7 +676,7 @@ void ftrace_graph_sleep_time_control(bool enable) * Simply points to ftrace_stub, but with the proper protocol. * Defined by the linker script in linux/vmlinux.lds.h */ -extern void ftrace_stub_graph(struct ftrace_graph_ret *); +void ftrace_stub_graph(struct ftrace_graph_ret *trace, struct fgraph_ops *gops); /* The callbacks that hook a function */ trace_func_graph_ret_t ftrace_graph_return = ftrace_stub_graph; diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index fd64021ec52f..7ff5c454622a 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -815,7 +815,8 @@ void ftrace_graph_graph_time_control(bool enable) fgraph_graph_time = enable; } -static int profile_graph_entry(struct ftrace_graph_ent *trace) +static int profile_graph_entry(struct ftrace_graph_ent *trace, + struct fgraph_ops *gops) { struct ftrace_ret_stack *ret_stack; @@ -832,7 +833,8 @@ static int profile_graph_entry(struct ftrace_graph_ent *trace) return 1; } -static void profile_graph_return(struct ftrace_graph_ret *trace) +static void profile_graph_return(struct ftrace_graph_ret *trace, + struct fgraph_ops *gops) { struct ftrace_ret_stack *ret_stack; struct ftrace_profile_stat *stat; diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h index b7f4ea25a194..d72d5e81f48a 100644 --- a/kernel/trace/trace.h +++ b/kernel/trace/trace.h @@ -675,8 +675,8 @@ void trace_latency_header(struct seq_file *m); void trace_default_header(struct seq_file *m); void print_trace_header(struct seq_file *m, struct trace_iterator *iter); -void trace_graph_return(struct ftrace_graph_ret *trace); -int trace_graph_entry(struct ftrace_graph_ent *trace); +void trace_graph_return(struct ftrace_graph_ret *trace, struct fgraph_ops *gops); +int trace_graph_entry(struct ftrace_graph_ent *trace, struct fgraph_ops *gops); void set_graph_array(struct trace_array *tr); void tracing_start_cmdline_record(void); diff --git a/kernel/trace/trace_functions_graph.c b/kernel/trace/trace_functions_graph.c index c35fbaab2a47..b7b142b65299 100644 --- a/kernel/trace/trace_functions_graph.c +++ b/kernel/trace/trace_functions_graph.c @@ -129,7 +129,8 @@ static inline int ftrace_graph_ignore_irqs(void) return in_hardirq(); } -int trace_graph_entry(struct ftrace_graph_ent *trace) +int trace_graph_entry(struct ftrace_graph_ent *trace, + struct fgraph_ops *gops) { struct trace_array *tr = graph_array; struct trace_array_cpu *data; @@ -238,7 +239,8 @@ void __trace_graph_return(struct trace_array *tr, trace_buffer_unlock_commit_nostack(buffer, event); } -void trace_graph_return(struct ftrace_graph_ret *trace) +void trace_graph_return(struct ftrace_graph_ret *trace, + struct fgraph_ops *gops) { struct trace_array *tr = graph_array; struct trace_array_cpu *data; @@ -275,7 +277,8 @@ void set_graph_array(struct trace_array *tr) smp_mb(); } -static void trace_graph_thresh_return(struct ftrace_graph_ret *trace) +static void trace_graph_thresh_return(struct ftrace_graph_ret *trace, + struct fgraph_ops *gops) { ftrace_graph_addr_finish(trace); @@ -288,7 +291,7 @@ static void trace_graph_thresh_return(struct ftrace_graph_ret *trace) (trace->rettime - trace->calltime < tracing_thresh)) return; else - trace_graph_return(trace); + trace_graph_return(trace, gops); } static struct fgraph_ops funcgraph_thresh_ops = { diff --git a/kernel/trace/trace_irqsoff.c b/kernel/trace/trace_irqsoff.c index ba37f768e2f2..5478f4c4f708 100644 --- a/kernel/trace/trace_irqsoff.c +++ b/kernel/trace/trace_irqsoff.c @@ -175,7 +175,8 @@ static int irqsoff_display_graph(struct trace_array *tr, int set) return start_irqsoff_tracer(irqsoff_trace, set); } -static int irqsoff_graph_entry(struct ftrace_graph_ent *trace) +static int irqsoff_graph_entry(struct ftrace_graph_ent *trace, + struct fgraph_ops *gops) { struct trace_array *tr = irqsoff_trace; struct trace_array_cpu *data; @@ -205,7 +206,8 @@ static int irqsoff_graph_entry(struct ftrace_graph_ent *trace) return ret; } -static void irqsoff_graph_return(struct ftrace_graph_ret *trace) +static void irqsoff_graph_return(struct ftrace_graph_ret *trace, + struct fgraph_ops *gops) { struct trace_array *tr = irqsoff_trace; struct trace_array_cpu *data; diff --git a/kernel/trace/trace_sched_wakeup.c b/kernel/trace/trace_sched_wakeup.c index 0469a04a355f..49bcc812652c 100644 --- a/kernel/trace/trace_sched_wakeup.c +++ b/kernel/trace/trace_sched_wakeup.c @@ -112,7 +112,8 @@ static int wakeup_display_graph(struct trace_array *tr, int set) return start_func_tracer(tr, set); } -static int wakeup_graph_entry(struct ftrace_graph_ent *trace) +static int wakeup_graph_entry(struct ftrace_graph_ent *trace, + struct fgraph_ops *gops) { struct trace_array *tr = wakeup_trace; struct trace_array_cpu *data; @@ -141,7 +142,8 @@ static int wakeup_graph_entry(struct ftrace_graph_ent *trace) return ret; } -static void wakeup_graph_return(struct ftrace_graph_ret *trace) +static void wakeup_graph_return(struct ftrace_graph_ret *trace, + struct fgraph_ops *gops) { struct trace_array *tr = wakeup_trace; struct trace_array_cpu *data; diff --git a/kernel/trace/trace_selftest.c b/kernel/trace/trace_selftest.c index 529590499b1f..914331d8242c 100644 --- a/kernel/trace/trace_selftest.c +++ b/kernel/trace/trace_selftest.c @@ -762,7 +762,8 @@ trace_selftest_startup_function(struct tracer *trace, struct trace_array *tr) static unsigned int graph_hang_thresh; /* Wrap the real function entry probe to avoid possible hanging */ -static int trace_graph_entry_watchdog(struct ftrace_graph_ent *trace) +static int trace_graph_entry_watchdog(struct ftrace_graph_ent *trace, + struct fgraph_ops *gops) { /* This is harmlessly racy, we want to approximately detect a hang */ if (unlikely(++graph_hang_thresh > GRAPH_MAX_FUNC_TEST)) { @@ -776,7 +777,7 @@ static int trace_graph_entry_watchdog(struct ftrace_graph_ent *trace) return 0; } - return trace_graph_entry(trace); + return trace_graph_entry(trace, gops); } static struct fgraph_ops fgraph_ops __initdata = { From patchwork Fri Dec 8 10:26:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 175740 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5363082vqy; Fri, 8 Dec 2023 02:27:50 -0800 (PST) X-Google-Smtp-Source: AGHT+IFp2QaSZgEtId+nXCnakzqBtRV2VKECkgZiOAtPQBBdGQZKmKQFp9HWo3eL3me8gfhicVU4 X-Received: by 2002:a05:6a21:1ca6:b0:18f:97c:9274 with SMTP id sf38-20020a056a211ca600b0018f097c9274mr2865467pzb.89.1702031269699; Fri, 08 Dec 2023 02:27:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702031269; cv=none; d=google.com; s=arc-20160816; b=vLxkoidwtrl1EcGUJV0fZH0MT9E4rMtydl8SRY80EZcyziXB4zvTOLbOCMXDpQqzA/ iWVsEuo1Rhcm+IiV5T7F9QUuy9j7TJZeLxLzO5SXxJ9J/9w58cnPseCaliHEUCB2s9ss VIvjtk2pxsEGtYRRaotMlPO2p+Ri1qizLNzk77mn1Ce561RwF8Lf0NYTcCXRgfX2ApBe IZVwSoWV7Jb/WJznmImmhZsws0riitVML1FBgDc1CxhsbQ1jqbkeFQCFd/jD52phiF4a js0PteoxsfxaWpRguLcx9B9yjoLleZvS9d80GZ7eM57I3x1snBdaC/kXlHxZBBmWs/V9 mTww== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=+vsZdVPrSeRUr5pm9f6V63X5HTaA1bN6gzZcl2YEsc8=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=HwN6+htOGPt6wQRo6NZi9YH1aCv7+k6XuMdi0VpnzQ+iLlaklsoklD0BdgwQdj59K5 18FPXzv5wSN7iYkfo3eLoy0NHLWUk8iBB5zn4T5oxQdxVz4cZuOrh1Nr1cq8DoRXTprk W75MU3bznq4lT5/uS1XwPzggY5AsnMl7pJWlgRK7N9qdWW4pF6V2caXrpbXJlToY6q0t TEHeiCDcJdiaxhn3DqPAsUQk/nTjYAXAhL1SySNn0HHfQqwvWL2DFn521T63Y3IukJ0Q kg2eecOQpyWt9QmVTufdYYE0M1BbdYpxtBljU5cQPpgtCBT3jWEyZtg23ZZmNrVMu/Zx 1/iA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Zq37NIK4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id m8-20020a63ed48000000b005c6673e2c72si1310104pgk.213.2023.12.08.02.27.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 02:27:49 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Zq37NIK4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id F27028076641; Fri, 8 Dec 2023 02:26:50 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1573727AbjLHK0b (ORCPT + 99 others); Fri, 8 Dec 2023 05:26:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49292 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1573555AbjLHK0L (ORCPT ); Fri, 8 Dec 2023 05:26:11 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BB23F1727 for ; Fri, 8 Dec 2023 02:26:12 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 71833C433C9; Fri, 8 Dec 2023 10:26:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702031172; bh=+F71If/fo1fuj2rgJryv0so80Ngm+JpCFXk3KdHKmWA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Zq37NIK45tT01CYntyGpBuThqrJBgke5fOtd3+/+evsJIHAlgReY/KFObJ1LorzGD e3/Q01IP3IU8Pe2YKO8nCr3bQ5avWTEHdgS4VjVsVF9nY2Na6qYmggqu2YE/7Nbpeo VU3WemCOliAvuTDQDf3ITYM7efT+1UfzkWDOH3/qOVhz+5irnMjOU925VSY17Q5GRz EnP6G2tw1CjDZ+A/BO+YfozSEyKzt87NmFgxbJRgI8lbHQQuRGBAYLpdQluRPoH5pX iKSrjtCdoNWJPcGNgzs4UgiqTc+EI0hBR8zPeVtzGT4ZgjRf+1zXOFALeaJem+/qQA EM72x6jIiYJeA== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v4 09/33] ftrace: Allow function_graph tracer to be enabled in instances Date: Fri, 8 Dec 2023 19:26:05 +0900 Message-Id: <170203116540.579004.14459772322137978884.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170203105427.579004.8033550792660734570.stgit@devnote2> References: <170203105427.579004.8033550792660734570.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Fri, 08 Dec 2023 02:26:51 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784709140351809591 X-GMAIL-MSGID: 1784709140351809591 From: Steven Rostedt (VMware) Now that function graph tracing can handle more than one user, allow it to be enabled in the ftrace instances. Note, the filtering of the functions is still joined by the top level set_ftrace_filter and friends, as well as the graph and nograph files. Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- Changes in v2: - Fix to remove set_graph_array() completely. --- include/linux/ftrace.h | 1 + kernel/trace/ftrace.c | 1 + kernel/trace/trace.h | 13 ++++++- kernel/trace/trace_functions.c | 8 ++++ kernel/trace/trace_functions_graph.c | 65 +++++++++++++++++++++------------- kernel/trace/trace_selftest.c | 4 +- 6 files changed, 64 insertions(+), 28 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index bc216cdc742d..0955baccbb87 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -1070,6 +1070,7 @@ extern int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace, struct fgraph struct fgraph_ops { trace_func_graph_ent_t entryfunc; trace_func_graph_ret_t retfunc; + void *private; }; /* diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 7ff5c454622a..83fbfb7b48f8 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -7319,6 +7319,7 @@ __init void ftrace_init_global_array_ops(struct trace_array *tr) tr->ops = &global_ops; tr->ops->private = tr; ftrace_init_trace_array(tr); + init_array_fgraph_ops(tr); } void ftrace_init_array_ops(struct trace_array *tr, ftrace_func_t func) diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h index d72d5e81f48a..16948c0ed00a 100644 --- a/kernel/trace/trace.h +++ b/kernel/trace/trace.h @@ -395,6 +395,9 @@ struct trace_array { struct ftrace_ops *ops; struct trace_pid_list __rcu *function_pids; struct trace_pid_list __rcu *function_no_pids; +#ifdef CONFIG_FUNCTION_GRAPH_TRACER + struct fgraph_ops *gops; +#endif #ifdef CONFIG_DYNAMIC_FTRACE /* All of these are protected by the ftrace_lock */ struct list_head func_probes; @@ -677,7 +680,6 @@ void print_trace_header(struct seq_file *m, struct trace_iterator *iter); void trace_graph_return(struct ftrace_graph_ret *trace, struct fgraph_ops *gops); int trace_graph_entry(struct ftrace_graph_ent *trace, struct fgraph_ops *gops); -void set_graph_array(struct trace_array *tr); void tracing_start_cmdline_record(void); void tracing_stop_cmdline_record(void); @@ -888,6 +890,9 @@ extern int __trace_graph_entry(struct trace_array *tr, extern void __trace_graph_return(struct trace_array *tr, struct ftrace_graph_ret *trace, unsigned int trace_ctx); +extern void init_array_fgraph_ops(struct trace_array *tr); +extern int allocate_fgraph_ops(struct trace_array *tr); +extern void free_fgraph_ops(struct trace_array *tr); #ifdef CONFIG_DYNAMIC_FTRACE extern struct ftrace_hash __rcu *ftrace_graph_hash; @@ -1000,6 +1005,12 @@ print_graph_function_flags(struct trace_iterator *iter, u32 flags) { return TRACE_TYPE_UNHANDLED; } +static inline void init_array_fgraph_ops(struct trace_array *tr) { } +static inline int allocate_fgraph_ops(struct trace_array *tr) +{ + return 0; +} +static inline void free_fgraph_ops(struct trace_array *tr) { } #endif /* CONFIG_FUNCTION_GRAPH_TRACER */ extern struct list_head ftrace_pids; diff --git a/kernel/trace/trace_functions.c b/kernel/trace/trace_functions.c index 9f1bfbe105e8..8e8da0d0ee52 100644 --- a/kernel/trace/trace_functions.c +++ b/kernel/trace/trace_functions.c @@ -80,6 +80,7 @@ void ftrace_free_ftrace_ops(struct trace_array *tr) int ftrace_create_function_files(struct trace_array *tr, struct dentry *parent) { + int ret; /* * The top level array uses the "global_ops", and the files are * created on boot up. @@ -90,6 +91,12 @@ int ftrace_create_function_files(struct trace_array *tr, if (!tr->ops) return -EINVAL; + ret = allocate_fgraph_ops(tr); + if (ret) { + kfree(tr->ops); + return ret; + } + ftrace_create_filter_files(tr->ops, parent); return 0; @@ -99,6 +106,7 @@ void ftrace_destroy_function_files(struct trace_array *tr) { ftrace_destroy_filter_files(tr->ops); ftrace_free_ftrace_ops(tr); + free_fgraph_ops(tr); } static ftrace_func_t select_trace_function(u32 flags_val) diff --git a/kernel/trace/trace_functions_graph.c b/kernel/trace/trace_functions_graph.c index b7b142b65299..9ccc904a7703 100644 --- a/kernel/trace/trace_functions_graph.c +++ b/kernel/trace/trace_functions_graph.c @@ -83,8 +83,6 @@ static struct tracer_flags tracer_flags = { .opts = trace_opts }; -static struct trace_array *graph_array; - /* * DURATION column is being also used to display IRQ signs, * following values are used by print_graph_irq and others @@ -132,7 +130,7 @@ static inline int ftrace_graph_ignore_irqs(void) int trace_graph_entry(struct ftrace_graph_ent *trace, struct fgraph_ops *gops) { - struct trace_array *tr = graph_array; + struct trace_array *tr = gops->private; struct trace_array_cpu *data; unsigned long flags; unsigned int trace_ctx; @@ -242,7 +240,7 @@ void __trace_graph_return(struct trace_array *tr, void trace_graph_return(struct ftrace_graph_ret *trace, struct fgraph_ops *gops) { - struct trace_array *tr = graph_array; + struct trace_array *tr = gops->private; struct trace_array_cpu *data; unsigned long flags; unsigned int trace_ctx; @@ -268,15 +266,6 @@ void trace_graph_return(struct ftrace_graph_ret *trace, local_irq_restore(flags); } -void set_graph_array(struct trace_array *tr) -{ - graph_array = tr; - - /* Make graph_array visible before we start tracing */ - - smp_mb(); -} - static void trace_graph_thresh_return(struct ftrace_graph_ret *trace, struct fgraph_ops *gops) { @@ -294,25 +283,53 @@ static void trace_graph_thresh_return(struct ftrace_graph_ret *trace, trace_graph_return(trace, gops); } -static struct fgraph_ops funcgraph_thresh_ops = { - .entryfunc = &trace_graph_entry, - .retfunc = &trace_graph_thresh_return, -}; - static struct fgraph_ops funcgraph_ops = { .entryfunc = &trace_graph_entry, .retfunc = &trace_graph_return, }; +int allocate_fgraph_ops(struct trace_array *tr) +{ + struct fgraph_ops *gops; + + gops = kzalloc(sizeof(*gops), GFP_KERNEL); + if (!gops) + return -ENOMEM; + + gops->entryfunc = &trace_graph_entry; + gops->retfunc = &trace_graph_return; + + tr->gops = gops; + gops->private = tr; + return 0; +} + +void free_fgraph_ops(struct trace_array *tr) +{ + kfree(tr->gops); +} + +__init void init_array_fgraph_ops(struct trace_array *tr) +{ + tr->gops = &funcgraph_ops; + funcgraph_ops.private = tr; +} + static int graph_trace_init(struct trace_array *tr) { int ret; - set_graph_array(tr); + tr->gops->entryfunc = trace_graph_entry; + if (tracing_thresh) - ret = register_ftrace_graph(&funcgraph_thresh_ops); + tr->gops->retfunc = trace_graph_thresh_return; else - ret = register_ftrace_graph(&funcgraph_ops); + tr->gops->retfunc = trace_graph_return; + + /* Make gops functions are visible before we start tracing */ + smp_mb(); + + ret = register_ftrace_graph(tr->gops); if (ret) return ret; tracing_start_cmdline_record(); @@ -323,10 +340,7 @@ static int graph_trace_init(struct trace_array *tr) static void graph_trace_reset(struct trace_array *tr) { tracing_stop_cmdline_record(); - if (tracing_thresh) - unregister_ftrace_graph(&funcgraph_thresh_ops); - else - unregister_ftrace_graph(&funcgraph_ops); + unregister_ftrace_graph(tr->gops); } static int graph_trace_update_thresh(struct trace_array *tr) @@ -1365,6 +1379,7 @@ static struct tracer graph_trace __tracer_data = { .print_header = print_graph_headers, .flags = &tracer_flags, .set_flag = func_graph_set_flag, + .allow_instances = true, #ifdef CONFIG_FTRACE_SELFTEST .selftest = trace_selftest_startup_function_graph, #endif diff --git a/kernel/trace/trace_selftest.c b/kernel/trace/trace_selftest.c index 914331d8242c..f0758afa2f7d 100644 --- a/kernel/trace/trace_selftest.c +++ b/kernel/trace/trace_selftest.c @@ -813,7 +813,7 @@ trace_selftest_startup_function_graph(struct tracer *trace, * to detect and recover from possible hangs */ tracing_reset_online_cpus(&tr->array_buffer); - set_graph_array(tr); + fgraph_ops.private = tr; ret = register_ftrace_graph(&fgraph_ops); if (ret) { warn_failed_init_tracer(trace, ret); @@ -856,7 +856,7 @@ trace_selftest_startup_function_graph(struct tracer *trace, cond_resched(); tracing_reset_online_cpus(&tr->array_buffer); - set_graph_array(tr); + fgraph_ops.private = tr; /* * Some archs *cough*PowerPC*cough* add characters to the From patchwork Fri Dec 8 10:26:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 175737 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5362872vqy; Fri, 8 Dec 2023 02:27:14 -0800 (PST) X-Google-Smtp-Source: AGHT+IHv4pxntse73Z70EKjl/T8AvOp7BRkFCGbjVPvA2m1hYToPl8A5Z5uCyVw1rtrX1qN+OjGK X-Received: by 2002:a17:902:e746:b0:1d0:6ffe:a09 with SMTP id p6-20020a170902e74600b001d06ffe0a09mr4788273plf.103.1702031234066; Fri, 08 Dec 2023 02:27:14 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702031234; cv=none; d=google.com; s=arc-20160816; b=k+qzf26CFPch+ncVADNa1wUOINlvzMgrxt3ryks3ciEXssKY9VzVlHlvqpN1FDuXih 7jhp6HQbfdoUoBEukq3qxzfQ5z3FNyqGZGk9evmqZsKbx9hIRJT/gNuOeDIFGBHa0Dl7 v2D8aiRSkgV7jHCMrL0Aj2g+jT0jU9dBY8cXLTFKB0EuPGHwdXRCkS1z1jIzj8KltY/V baQSRg674iFZQd3YZjOV7J2OqVpTP2aB1n20GRR7Yx6YYbCtrN6l1OH+fMAjII3Vkv8T r6Mm9KfaSGHTCJvCRtxxKVr5BZf38sHQM8yC2VNzhG7iNlCtAs+uHpi+gB6EgwGg6m+q CEDQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=8FG1EfTvjghHx0E3OCkHobbbNVNqEcUAtmh9UlmGtc4=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=mJPXuuaKFGhWqTCted9A8vA8RQNSGe1I6O129nG1Wx6HsIcZ8v2q6kUJaGWnNmjyqG 3NYimwDnwfWBxGxm0VfUoi2Rphx7jPc4XNPt3bQDqkzxHI4PwpiXn9YQUQhDbFRq3Krr eVb2sFL+I3UcCcNIcdUhfgjCFRDQxp4rAwCqyTA9bjkiXbXZyiiWCY3T170IL8Wd2/Jc Gh5iX4qxTbN4b8QJiJo23/BfUW//TGutv8BR8YBft3Fd1eAAOl88UIle7pGK4ZKn+jg2 Dl/H9gl3BkJLiOq0ctyGGek2wKSNRPQgMMxAyF1UPZINnq+iczOoPMCQz1JL9L25936/ 7W/w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=WGQ5K1W9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id l4-20020a170903244400b001d09278b856si1376159pls.347.2023.12.08.02.27.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 02:27:14 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=WGQ5K1W9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 8654D80944EC; Fri, 8 Dec 2023 02:27:11 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1573473AbjLHK05 (ORCPT + 99 others); Fri, 8 Dec 2023 05:26:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60988 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1573726AbjLHK0b (ORCPT ); Fri, 8 Dec 2023 05:26:31 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 31BF01FE4 for ; Fri, 8 Dec 2023 02:26:23 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6D4EAC433C8; Fri, 8 Dec 2023 10:26:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702031183; bh=6ERBwQiaxIIDNTVWPQkxDhl1p/zo1tlsKma+ZLC6cPI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WGQ5K1W971s+MC8xP9/LxQdkaHOiHnfgk/icKi5ZJGXIe9pv0HE2mtuVnmgTTf0LK +Mg5P/tMiJCUNfxZHEFB3rLHRTNZEf0XY1Fg9nur6vtPB0N0kHTU+Ez2xvqL9CGaVa F0AeU2oqMCZKXGaQlaTmHYeHTFjdFvBXFfCWSJG+pgaovzd5cYoVQ8YyBlqeg8L0cI 5U0vE53REDc/LnaWQkH4TgiQEom4WsQBG3GybRytkNUigmV7+TX8kMI3BpYCwYvRtU j2UXOH44qRsk6bTWbv2bzLmL1uWHwrS2mNipDxui3epimDdQcKqSaPXxz5pLKDLh82 ZyK6Es3/iqzDw== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v4 10/33] ftrace: Allow ftrace startup flags exist without dynamic ftrace Date: Fri, 8 Dec 2023 19:26:18 +0900 Message-Id: <170203117769.579004.5659514449783521216.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170203105427.579004.8033550792660734570.stgit@devnote2> References: <170203105427.579004.8033550792660734570.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Fri, 08 Dec 2023 02:27:11 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784709102954522464 X-GMAIL-MSGID: 1784709102954522464 From: Steven Rostedt (VMware) Some of the flags for ftrace_startup() may be exposed even when CONFIG_DYNAMIC_FTRACE is not configured in. This is fine as the difference between dynamic ftrace and static ftrace is done within the internals of ftrace itself. No need to have use cases fail to compile because dynamic ftrace is disabled. This change is needed to move some of the logic of what is passed to ftrace_startup() out of the parameters of ftrace_startup(). Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- include/linux/ftrace.h | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 0955baccbb87..7b08169aa51d 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -538,6 +538,15 @@ static inline void stack_tracer_disable(void) { } static inline void stack_tracer_enable(void) { } #endif +enum { + FTRACE_UPDATE_CALLS = (1 << 0), + FTRACE_DISABLE_CALLS = (1 << 1), + FTRACE_UPDATE_TRACE_FUNC = (1 << 2), + FTRACE_START_FUNC_RET = (1 << 3), + FTRACE_STOP_FUNC_RET = (1 << 4), + FTRACE_MAY_SLEEP = (1 << 5), +}; + #ifdef CONFIG_DYNAMIC_FTRACE void ftrace_arch_code_modify_prepare(void); @@ -632,15 +641,6 @@ void ftrace_set_global_notrace(unsigned char *buf, int len, int reset); void ftrace_free_filter(struct ftrace_ops *ops); void ftrace_ops_set_global_filter(struct ftrace_ops *ops); -enum { - FTRACE_UPDATE_CALLS = (1 << 0), - FTRACE_DISABLE_CALLS = (1 << 1), - FTRACE_UPDATE_TRACE_FUNC = (1 << 2), - FTRACE_START_FUNC_RET = (1 << 3), - FTRACE_STOP_FUNC_RET = (1 << 4), - FTRACE_MAY_SLEEP = (1 << 5), -}; - /* * The FTRACE_UPDATE_* enum is used to pass information back * from the ftrace_update_record() and ftrace_test_record() From patchwork Fri Dec 8 10:26:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 175739 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5363055vqy; Fri, 8 Dec 2023 02:27:43 -0800 (PST) X-Google-Smtp-Source: AGHT+IFC6EdjdKHLP9Kg2f5Q2C3695UxIRzeX4bX8QzDIPYPCinucAGB1MnXQ/NCHkA+FDyBkvuH X-Received: by 2002:a17:902:f541:b0:1cf:a70b:39c8 with SMTP id h1-20020a170902f54100b001cfa70b39c8mr4034576plf.3.1702031263118; Fri, 08 Dec 2023 02:27:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702031263; cv=none; d=google.com; s=arc-20160816; b=QSqFnlO5W+Io2ZXTSjqvSS1Tmz/8g82bFko7h0v/F7CFurwIQTZHZDGxBvZtfnU2Bq CtjJ8rKBPQ+o/yoiLCnFoDIw/o0i0IsVqD51DUieUhgTBfNyB3Z/zohUnR1Isrq77Dk2 z2NT1lsFILkiGqMRAKpaF/8JrqNV5vfo9hILEnhhFEqNVSPlNQ/Q6CAOgjGnybGFsasx HSX2DahysFt9YJnj6Xfm5yP8yJ0KNJEnpO2BaNkufi6gUTb6BNKcEdTkazqz4qia/IE6 hTzoQUOApi0Kacinl/qfSO3OiNVTB4uL06r/9ZQ9OWqJqUWTqY70VSSBb25rJwx5zMVS y7lQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=QoKeCExjhZDctz5r3O40iZviQk3QP4UszfmFetvjyq0=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=P2EMyzd9oHe/hNEKHFINOdncvlAR/2wbB+sK3movX5eEqFHrTq8AZMppG9JktzuJFa 6bym1HMI1klIJGjrSY9Bj1edEE+lzu3I2/jXzIbzVU0IkiMTYQs9yVhDw+64/drE84yW jepHjffxjBhpFMA+BMfDG/N/mIzi4Or/Fx5EPAEmpLScrJaC4KOctzjB7urJJsjPnKqC QdHrQZLUEceu6lJGRQ95nT7dvUWRw6u6myWhCscyGLHegW7mkH2om6l2Ev0FfwgUrK+s sY1zASy12DCCo2NnWDRrj3dhwhpEPP67lkVHwnd0hhxTLjHD6gsK/DUDnYTFpKmHicro bBng== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=KiT6MTUn; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id be9-20020a170902aa0900b001cfc9a1915bsi1357246plb.234.2023.12.08.02.27.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 02:27:43 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=KiT6MTUn; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 9CDFB8521372; Fri, 8 Dec 2023 02:27:33 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1573617AbjLHK1Q (ORCPT + 99 others); Fri, 8 Dec 2023 05:27:16 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60836 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235941AbjLHK0s (ORCPT ); Fri, 8 Dec 2023 05:26:48 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 43C191708 for ; Fri, 8 Dec 2023 02:26:35 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 58557C433C7; Fri, 8 Dec 2023 10:26:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702031194; bh=TJdlXgVoOs1czXw3FLraoNo4e1qHiUrA0KJRNqTzl28=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KiT6MTUnXIitKHAslmL3c4I/TUZVj2ZMsl5+jO3PSwYBrNwn5BwJH7/KaGJ+lAr5j vxwKshOagbzby3sf1XurV3iMo11YGqeiYn0pgvJk9OKGiyvq5o9X0k7fj0eBTKzbDN uDTRHOzo9f3N6eO0tfZDYMHwzwGEpTJpJ3U3F6WOHarH3bYjYrarCCb/pH84YzmjY9 1ij1KDpjGcBb+TncXlunSE1Qdk5/uwOXkc/4GOxb+kbIlxjJp/ZINWIYNeVq10d9il b38sl5R/0hcn1eu5AKT6yx47avaDCRJR7VeHA31PEZQNz9cuxIqROBr42ITAzKTkzM rbgYMcgP3jXYw== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v4 11/33] function_graph: Have the instances use their own ftrace_ops for filtering Date: Fri, 8 Dec 2023 19:26:29 +0900 Message-Id: <170203118875.579004.6433626598470720315.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170203105427.579004.8033550792660734570.stgit@devnote2> References: <170203105427.579004.8033550792660734570.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Fri, 08 Dec 2023 02:27:34 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784709133546209570 X-GMAIL-MSGID: 1784709133546209570 From: Steven Rostedt (VMware) Allow for instances to have their own ftrace_ops part of the fgraph_ops that makes the funtion_graph tracer filter on the set_ftrace_filter file of the instance and not the top instance. This also change how the function_graph handles multiple instances on the shadow stack. Previously we use ARRAY type entries to record which one is enabled, and this makes it a bitmap of the fgraph_array's indexes. Previous function_graph_enter() expects calling back from prepare_ftrace_return() function which is called back only once if it is enabled. But this introduces different ftrace_ops for each fgraph instance and those are called from ftrace_graph_func() one by one. Thus we can not loop on the fgraph_array(), and need to reuse the ret_stack pushed by the previous instance. Finding the ret_stack is easy because we can check the ret_stack->func. But that is not enough for the self- recursive tail-call case. Thus fgraph uses the bitmap entry to find it is already set (this means that entry is for previous tail call). Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- Changes in v4: - Simplify get_ret_stack() sanity check and use WARN_ON_ONCE() for obviously wrong value. - Do not check ret == return_to_handler but always read the previous ret_stack in ftrace_push_return_trace() to check it is reusable. - Set the bit 0 of the bitmap entry always in function_graph_enter() because it uses bit 0 to check re-usability. - Fix to ensure the ret_stack entry is bitmap type when checking the bitmap. Changes in v3: - Pass current fgraph_ops to the new entry handler (function_graph_enter_ops) if fgraph use ftrace. - Add fgraph_ops::idx in this patch. - Replace the array type with the bitmap type so that it can record which fgraph is called. - Fix some helper function to use passed task_struct instead of current. - Reduce the ret-index size to 1024 words. - Make the ret-index directly points the ret_stack. - Fix ftrace_graph_ret_addr() to handle tail-call case correctly. Changes in v2: - Use ftrace_graph_func and FTRACE_OPS_GRAPH_STUB instead of ftrace_stub and FTRACE_OPS_FL_STUB for new ftrace based fgraph. --- arch/arm64/kernel/ftrace.c | 19 ++ arch/x86/kernel/ftrace.c | 19 ++ include/linux/ftrace.h | 7 + kernel/trace/fgraph.c | 369 ++++++++++++++++++++-------------- kernel/trace/ftrace.c | 6 - kernel/trace/trace.h | 16 + kernel/trace/trace_functions.c | 2 kernel/trace/trace_functions_graph.c | 8 + 8 files changed, 277 insertions(+), 169 deletions(-) diff --git a/arch/arm64/kernel/ftrace.c b/arch/arm64/kernel/ftrace.c index a650f5e11fc5..205937e04ece 100644 --- a/arch/arm64/kernel/ftrace.c +++ b/arch/arm64/kernel/ftrace.c @@ -481,7 +481,24 @@ void prepare_ftrace_return(unsigned long self_addr, unsigned long *parent, void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, struct ftrace_ops *op, struct ftrace_regs *fregs) { - prepare_ftrace_return(ip, &fregs->lr, fregs->fp); + unsigned long *parent = &fregs->lr; + struct fgraph_ops *gops = container_of(op, struct fgraph_ops, ops); + int bit; + + if (unlikely(ftrace_graph_is_dead())) + return; + + if (unlikely(atomic_read(¤t->tracing_graph_pause))) + return; + + bit = ftrace_test_recursion_trylock(ip, *parent); + if (bit < 0) + return; + + if (!function_graph_enter_ops(*parent, ip, fregs->fp, parent, gops)) + *parent = (unsigned long)&return_to_handler; + + ftrace_test_recursion_unlock(bit); } #else /* diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c index 12df54ff0e81..845e29b4254f 100644 --- a/arch/x86/kernel/ftrace.c +++ b/arch/x86/kernel/ftrace.c @@ -657,9 +657,24 @@ void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, struct ftrace_ops *op, struct ftrace_regs *fregs) { struct pt_regs *regs = &fregs->regs; - unsigned long *stack = (unsigned long *)kernel_stack_pointer(regs); + unsigned long *parent = (unsigned long *)kernel_stack_pointer(regs); + struct fgraph_ops *gops = container_of(op, struct fgraph_ops, ops); + int bit; + + if (unlikely(ftrace_graph_is_dead())) + return; + + if (unlikely(atomic_read(¤t->tracing_graph_pause))) + return; - prepare_ftrace_return(ip, (unsigned long *)stack, 0); + bit = ftrace_test_recursion_trylock(ip, *parent); + if (bit < 0) + return; + + if (!function_graph_enter_ops(*parent, ip, 0, parent, gops)) + *parent = (unsigned long)&return_to_handler; + + ftrace_test_recursion_unlock(bit); } #endif diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 7b08169aa51d..c431a33fe789 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -1070,7 +1070,9 @@ extern int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace, struct fgraph struct fgraph_ops { trace_func_graph_ent_t entryfunc; trace_func_graph_ret_t retfunc; + struct ftrace_ops ops; /* for the hash lists */ void *private; + int idx; }; /* @@ -1104,6 +1106,11 @@ extern int function_graph_enter(unsigned long ret, unsigned long func, unsigned long frame_pointer, unsigned long *retp); +extern int +function_graph_enter_ops(unsigned long ret, unsigned long func, + unsigned long frame_pointer, unsigned long *retp, + struct fgraph_ops *gops); + struct ftrace_ret_stack * ftrace_graph_get_ret_stack(struct task_struct *task, int idx); diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index 62c35d6d95f9..6f537ebd3ed7 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -7,6 +7,7 @@ * * Highly modified by Steven Rostedt (VMware). */ +#include #include #include #include @@ -17,22 +18,15 @@ #include "ftrace_internal.h" #include "trace.h" -#ifdef CONFIG_DYNAMIC_FTRACE -#define ASSIGN_OPS_HASH(opsname, val) \ - .func_hash = val, \ - .local_hash.regex_lock = __MUTEX_INITIALIZER(opsname.local_hash.regex_lock), -#else -#define ASSIGN_OPS_HASH(opsname, val) -#endif - #define FGRAPH_RET_SIZE sizeof(struct ftrace_ret_stack) #define FGRAPH_RET_INDEX (FGRAPH_RET_SIZE / sizeof(long)) /* * On entry to a function (via function_graph_enter()), a new ftrace_ret_stack - * is allocated on the task's ret_stack, then each fgraph_ops on the - * fgraph_array[]'s entryfunc is called and if that returns non-zero, the - * index into the fgraph_array[] for that fgraph_ops is added to the ret_stack. + * is allocated on the task's ret_stack with indexes entry, then each + * fgraph_ops on the fgraph_array[]'s entryfunc is called and if that returns + * non-zero, the index into the fgraph_array[] for that fgraph_ops is recorded + * on the indexes entry as a bit flag. * As the associated ftrace_ret_stack saved for those fgraph_ops needs to * be found, the index to it is also added to the ret_stack along with the * index of the fgraph_array[] to each fgraph_ops that needs their retfunc @@ -42,61 +36,59 @@ * to the last ftrace_ret_stack saved. All references to the * ftrace_ret_stack has the format of: * - * bits: 0 - 13 Index in words from the previous ftrace_ret_stack - * bits: 14 - 15 Type of storage + * bits: 0 - 9 offset in words from the previous ftrace_ret_stack + * (bitmap type should have FGRAPH_RET_INDEX always) + * bits: 10 - 11 Type of storage * 0 - reserved - * 1 - fgraph_array index - * For fgraph_array_index: - * bits: 16 - 23 The fgraph_ops fgraph_array index + * 1 - bitmap of fgraph_array index + * + * For bitmap of fgraph_array index + * bits: 12 - 27 The bitmap of fgraph_ops fgraph_array index * * That is, at the end of function_graph_enter, if the first and forth * fgraph_ops on the fgraph_array[] (index 0 and 3) needs their retfunc called * on the return of the function being traced, this is what will be on the * task's shadow ret_stack: (the stack grows upward) * - * | | <- task->curr_ret_stack - * +----------------------------------+ - * | (3 << FGRAPH_ARRAY_SHIFT)|(2) | ( 3 for index of fourth fgraph_ops) - * +----------------------------------+ - * | (0 << FGRAPH_ARRAY_SHIFT)|(1) | ( 0 for index of first fgraph_ops) - * +----------------------------------+ - * | struct ftrace_ret_stack | - * | (stores the saved ret pointer) | - * +----------------------------------+ - * | (X) | (N) | ( N words away from previous ret_stack) - * | | + * | | <- task->curr_ret_stack + * +--------------------------------------------+ + * | bitmap_type(bitmap:(BIT(3)|BIT(0)), | + * | offset:FGRAPH_RET_INDEX) | <- the offset is from here + * +--------------------------------------------+ + * | struct ftrace_ret_stack | + * | (stores the saved ret pointer) | <- the offset points here + * +--------------------------------------------+ + * | (X) | (N) | ( N words away from + * | | previous ret_stack) * * If a backtrace is required, and the real return pointer needs to be * fetched, then it looks at the task's curr_ret_stack index, if it - * is greater than zero, it would subtact one, and then mask the value - * on the ret_stack by FGRAPH_RET_INDEX_MASK and subtract FGRAPH_RET_INDEX - * from that, to get the index of the ftrace_ret_stack structure stored - * on the shadow stack. + * is greater than zero (reserved, or right before poped), it would mask + * the value by FGRAPH_RET_INDEX_MASK to get the offset index of the + * ftrace_ret_stack structure stored on the shadow stack. */ -#define FGRAPH_RET_INDEX_SIZE 14 -#define FGRAPH_RET_INDEX_MASK ((1 << FGRAPH_RET_INDEX_SIZE) - 1) - +#define FGRAPH_RET_INDEX_SIZE 10 +#define FGRAPH_RET_INDEX_MASK GENMASK(FGRAPH_RET_INDEX_SIZE - 1, 0) #define FGRAPH_TYPE_SIZE 2 -#define FGRAPH_TYPE_MASK ((1 << FGRAPH_TYPE_SIZE) - 1) +#define FGRAPH_TYPE_MASK GENMASK(FGRAPH_TYPE_SIZE - 1, 0) #define FGRAPH_TYPE_SHIFT FGRAPH_RET_INDEX_SIZE enum { FGRAPH_TYPE_RESERVED = 0, - FGRAPH_TYPE_ARRAY = 1, + FGRAPH_TYPE_BITMAP = 1, }; -#define FGRAPH_ARRAY_SIZE 16 -#define FGRAPH_ARRAY_MASK ((1 << FGRAPH_ARRAY_SIZE) - 1) -#define FGRAPH_ARRAY_SHIFT (FGRAPH_TYPE_SHIFT + FGRAPH_TYPE_SIZE) +#define FGRAPH_INDEX_SIZE 16 +#define FGRAPH_INDEX_MASK GENMASK(FGRAPH_INDEX_SIZE - 1, 0) +#define FGRAPH_INDEX_SHIFT (FGRAPH_TYPE_SHIFT + FGRAPH_TYPE_SIZE) /* Currently the max stack index can't be more than register callers */ -#define FGRAPH_MAX_INDEX FGRAPH_ARRAY_SIZE +#define FGRAPH_MAX_INDEX (FGRAPH_INDEX_SIZE + FGRAPH_RET_INDEX) + +#define FGRAPH_ARRAY_SIZE FGRAPH_INDEX_SIZE -#define FGRAPH_FRAME_SIZE (FGRAPH_RET_SIZE + FGRAPH_ARRAY_SIZE * (sizeof(long))) -#define FGRAPH_FRAME_INDEX (ALIGN(FGRAPH_FRAME_SIZE, \ - sizeof(long)) / sizeof(long)) #define SHADOW_STACK_SIZE (PAGE_SIZE) #define SHADOW_STACK_INDEX (SHADOW_STACK_SIZE / sizeof(long)) /* Leave on a buffer at the end */ @@ -113,19 +105,36 @@ static struct fgraph_ops *fgraph_array[FGRAPH_ARRAY_SIZE]; static inline int get_ret_stack_index(struct task_struct *t, int offset) { - return current->ret_stack[offset] & FGRAPH_RET_INDEX_MASK; + return t->ret_stack[offset] & FGRAPH_RET_INDEX_MASK; } static inline int get_fgraph_type(struct task_struct *t, int offset) { - return (current->ret_stack[offset] >> FGRAPH_TYPE_SHIFT) & - FGRAPH_TYPE_MASK; + return (t->ret_stack[offset] >> FGRAPH_TYPE_SHIFT) & FGRAPH_TYPE_MASK; +} + +static inline unsigned long +get_fgraph_index_bitmap(struct task_struct *t, int offset) +{ + return (t->ret_stack[offset] >> FGRAPH_INDEX_SHIFT) & FGRAPH_INDEX_MASK; } -static inline int get_fgraph_array(struct task_struct *t, int offset) +static inline void +set_fgraph_index_bitmap(struct task_struct *t, int offset, unsigned long bitmap) { - return (current->ret_stack[offset] >> FGRAPH_ARRAY_SHIFT) & - FGRAPH_ARRAY_MASK; + t->ret_stack[offset] = (bitmap << FGRAPH_INDEX_SHIFT) | + (FGRAPH_TYPE_BITMAP << FGRAPH_TYPE_SHIFT) | FGRAPH_RET_INDEX; +} + +static inline bool is_fgraph_index_set(struct task_struct *t, int offset, int idx) +{ + return !!(get_fgraph_index_bitmap(t, offset) & BIT(idx)); +} + +static inline void +add_fgraph_index_bitmap(struct task_struct *t, int offset, unsigned long bitmap) +{ + t->ret_stack[offset] |= (bitmap << FGRAPH_INDEX_SHIFT); } /* ftrace_graph_entry set to this to tell some archs to run function graph */ @@ -160,17 +169,14 @@ get_ret_stack(struct task_struct *t, int offset, int *index) BUILD_BUG_ON(FGRAPH_RET_SIZE % sizeof(long)); - if (offset <= 0) + if (unlikely(offset <= 0)) return NULL; - idx = get_ret_stack_index(t, offset - 1); - - if (idx <= 0 || idx > FGRAPH_MAX_INDEX) + idx = get_ret_stack_index(t, --offset); + if (WARN_ON_ONCE(idx <= 0 || idx > offset)) return NULL; - offset -= idx + FGRAPH_RET_INDEX; - if (offset < 0) - return NULL; + offset -= idx; *index = offset; return RET_STACK(t, offset); @@ -231,10 +237,12 @@ void ftrace_graph_stop(void) /* Add a function return address to the trace stack on thread info.*/ static int ftrace_push_return_trace(unsigned long ret, unsigned long func, - unsigned long frame_pointer, unsigned long *retp) + unsigned long frame_pointer, unsigned long *retp, + int fgraph_idx) { struct ftrace_ret_stack *ret_stack; unsigned long long calltime; + unsigned long val; int index; if (unlikely(ftrace_graph_is_dead())) @@ -243,6 +251,21 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func, if (!current->ret_stack) return -EBUSY; + /* + * At first, check whether the previous fgraph callback is pushed by + * the fgraph on the same function entry. + * But if @func is the self tail-call function, we also need to ensure + * the ret_stack is not for the previous call by checking whether the + * bit of @fgraph_idx is set or not. + */ + ret_stack = get_ret_stack(current, current->curr_ret_stack, &index); + if (ret_stack && ret_stack->func == func && + get_fgraph_type(current, index + FGRAPH_RET_INDEX) == FGRAPH_TYPE_BITMAP && + !is_fgraph_index_set(current, index + FGRAPH_RET_INDEX, fgraph_idx)) + return index + FGRAPH_RET_INDEX; + + val = (FGRAPH_TYPE_RESERVED << FGRAPH_TYPE_SHIFT) | FGRAPH_RET_INDEX; + BUILD_BUG_ON(SHADOW_STACK_SIZE % sizeof(long)); /* @@ -252,17 +275,19 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func, smp_rmb(); /* The return trace stack is full */ - if (current->curr_ret_stack >= SHADOW_STACK_MAX_INDEX) { + if (current->curr_ret_stack + FGRAPH_RET_INDEX >= SHADOW_STACK_MAX_INDEX) { atomic_inc(¤t->trace_overrun); return -EBUSY; } calltime = trace_clock_local(); - index = current->curr_ret_stack; - /* ret offset = 1 ; type = reserved */ - current->ret_stack[index + FGRAPH_RET_INDEX] = 1; + index = READ_ONCE(current->curr_ret_stack); ret_stack = RET_STACK(current, index); + index += FGRAPH_RET_INDEX; + + /* ret offset = FGRAPH_RET_INDEX ; type = reserved */ + current->ret_stack[index] = val; ret_stack->ret = ret; /* * The unwinders expect curr_ret_stack to point to either zero @@ -278,7 +303,7 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func, * at least a correct index! */ barrier(); - current->curr_ret_stack += FGRAPH_RET_INDEX + 1; + current->curr_ret_stack = index + 1; /* * This next barrier is to ensure that an interrupt coming in * will not corrupt what we are about to write. @@ -286,7 +311,7 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func, barrier(); /* Still keep it reserved even if an interrupt came in */ - current->ret_stack[index + FGRAPH_RET_INDEX] = 1; + current->ret_stack[index] = val; ret_stack->ret = ret; ret_stack->func = func; @@ -297,7 +322,7 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func, #ifdef HAVE_FUNCTION_GRAPH_RET_ADDR_PTR ret_stack->retp = retp; #endif - return 0; + return index; } /* @@ -314,15 +339,13 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func, # define MCOUNT_INSN_SIZE 0 #endif +/* If the caller does not use ftrace, call this function. */ int function_graph_enter(unsigned long ret, unsigned long func, unsigned long frame_pointer, unsigned long *retp) { struct ftrace_graph_ent trace; - int offset; - int start; - int type; - int val; - int cnt = 0; + unsigned long bitmap = 0; + int index; int i; #ifndef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS @@ -337,69 +360,33 @@ int function_graph_enter(unsigned long ret, unsigned long func, return -EBUSY; #endif - if (!ftrace_ops_test(&global_ops, func, NULL)) - return -EBUSY; - trace.func = func; trace.depth = ++current->curr_ret_depth; - if (ftrace_push_return_trace(ret, func, frame_pointer, retp)) + index = ftrace_push_return_trace(ret, func, frame_pointer, retp, 0); + if (index < 0) goto out; - /* Use start for the distance to ret_stack (skipping over reserve) */ - start = offset = current->curr_ret_stack - 2; - for (i = 0; i < fgraph_array_cnt; i++) { struct fgraph_ops *gops = fgraph_array[i]; if (gops == &fgraph_stub) continue; - if ((offset == start) && - (current->curr_ret_stack >= SHADOW_STACK_INDEX - 1)) { - atomic_inc(¤t->trace_overrun); - break; - } - if (fgraph_array[i]->entryfunc(&trace, fgraph_array[i])) { - offset = current->curr_ret_stack; - /* Check the top level stored word */ - type = get_fgraph_type(current, offset - 1); - - val = (i << FGRAPH_ARRAY_SHIFT) | - (FGRAPH_TYPE_ARRAY << FGRAPH_TYPE_SHIFT) | - ((offset - start) - 1); - - /* We can reuse the top word if it is reserved */ - if (type == FGRAPH_TYPE_RESERVED) { - current->ret_stack[offset - 1] = val; - cnt++; - continue; - } - val++; - - current->ret_stack[offset] = val; - /* - * Write the value before we increment, so that - * if an interrupt comes in after we increment - * it will still see the value and skip over - * this. - */ - barrier(); - current->curr_ret_stack++; - /* - * Have to write again, in case an interrupt - * came in before the increment and after we - * wrote the value. - */ - barrier(); - current->ret_stack[offset] = val; - cnt++; - } + if (ftrace_ops_test(&gops->ops, func, NULL) && + gops->entryfunc(&trace, gops)) + bitmap |= BIT(i); } - if (!cnt) + if (!bitmap) goto out_ret; + /* + * Since this function uses fgraph_idx = 0 as a tail-call checking + * flag, set that bit always. + */ + set_fgraph_index_bitmap(current, index, bitmap | BIT(0)); + return 0; out_ret: current->curr_ret_stack -= FGRAPH_RET_INDEX + 1; @@ -408,15 +395,51 @@ int function_graph_enter(unsigned long ret, unsigned long func, return -EBUSY; } +/* This is called from ftrace_graph_func() via ftrace */ +int function_graph_enter_ops(unsigned long ret, unsigned long func, + unsigned long frame_pointer, unsigned long *retp, + struct fgraph_ops *gops) +{ + struct ftrace_graph_ent trace; + int index; + int type; + + + /* Use start for the distance to ret_stack (skipping over reserve) */ + index = ftrace_push_return_trace(ret, func, frame_pointer, retp, gops->idx); + if (index < 0) + return index; + type = get_fgraph_type(current, index); + + /* This is the first ret_stack for this fentry */ + if (type == FGRAPH_TYPE_RESERVED) + ++current->curr_ret_depth; + + trace.func = func; + trace.depth = current->curr_ret_depth; + if (gops->entryfunc(&trace, gops)) { + if (type == FGRAPH_TYPE_RESERVED) + set_fgraph_index_bitmap(current, index, BIT(gops->idx)); + else + add_fgraph_index_bitmap(current, index, BIT(gops->idx)); + return 0; + } + + if (type == FGRAPH_TYPE_RESERVED) { + current->curr_ret_stack -= FGRAPH_RET_INDEX + 1; + current->curr_ret_depth--; + } + return -EBUSY; +} + /* Retrieve a function return address to the trace stack on thread info.*/ static struct ftrace_ret_stack * ftrace_pop_return_trace(struct ftrace_graph_ret *trace, unsigned long *ret, - unsigned long frame_pointer) + unsigned long frame_pointer, int *index) { struct ftrace_ret_stack *ret_stack; - int index; - ret_stack = get_ret_stack(current, current->curr_ret_stack, &index); + ret_stack = get_ret_stack(current, current->curr_ret_stack, index); if (unlikely(!ret_stack)) { ftrace_graph_stop(); @@ -455,6 +478,7 @@ ftrace_pop_return_trace(struct ftrace_graph_ret *trace, unsigned long *ret, } #endif + *index += FGRAPH_RET_INDEX; *ret = ret_stack->ret; trace->func = ret_stack->func; trace->calltime = ret_stack->calltime; @@ -507,13 +531,12 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs { struct ftrace_ret_stack *ret_stack; struct ftrace_graph_ret trace; + unsigned long bitmap; unsigned long ret; - int offset; int index; - int idx; int i; - ret_stack = ftrace_pop_return_trace(&trace, &ret, frame_pointer); + ret_stack = ftrace_pop_return_trace(&trace, &ret, frame_pointer, &index); if (unlikely(!ret_stack)) { ftrace_graph_stop(); @@ -527,16 +550,17 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs trace.retval = fgraph_ret_regs_return_value(ret_regs); #endif - offset = current->curr_ret_stack - 1; - index = get_ret_stack_index(current, offset); + bitmap = get_fgraph_index_bitmap(current, index); + for (i = 0; i < FGRAPH_ARRAY_SIZE; i++) { + struct fgraph_ops *gops = fgraph_array[i]; - /* index has to be at least one! Optimize for it */ - i = 0; - do { - idx = get_fgraph_array(current, offset - i); - fgraph_array[idx]->retfunc(&trace, fgraph_array[idx]); - i++; - } while (i < index); + if (!(bitmap & BIT(i))) + continue; + if (gops == &fgraph_stub) + continue; + + gops->retfunc(&trace, gops); + } /* * The ftrace_graph_return() may still access the current @@ -544,7 +568,7 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs * curr_ret_stack is after that. */ barrier(); - current->curr_ret_stack -= index + FGRAPH_RET_INDEX; + current->curr_ret_stack -= FGRAPH_RET_INDEX + 1; current->curr_ret_depth--; return ret; } @@ -622,7 +646,17 @@ unsigned long ftrace_graph_ret_addr(struct task_struct *task, int *idx, ret_stack = get_ret_stack(current, i, &i); if (!ret_stack) break; - if (ret_stack->retp == retp) + /* + * For the tail-call, there would be 2 or more ftrace_ret_stacks on + * the ret_stack, which records "return_to_handler" as the return + * address excpt for the last one. + * But on the real stack, there should be 1 entry because tail-call + * reuses the return address on the stack and jump to the next function. + * Thus we will continue to find real return address. + */ + if (ret_stack->retp == retp && + ret_stack->ret != + (unsigned long)dereference_kernel_function_descriptor(return_to_handler)) return ret_stack->ret; } @@ -645,6 +679,9 @@ unsigned long ftrace_graph_ret_addr(struct task_struct *task, int *idx, i = *idx; do { ret_stack = get_ret_stack(task, task_idx, &task_idx); + if (ret_stack && ret_stack->ret == + (unsigned long)dereference_kernel_function_descriptor(return_to_handler)) + continue; i--; } while (i >= 0 && ret_stack); @@ -655,17 +692,25 @@ unsigned long ftrace_graph_ret_addr(struct task_struct *task, int *idx, } #endif /* HAVE_FUNCTION_GRAPH_RET_ADDR_PTR */ -static struct ftrace_ops graph_ops = { - .func = ftrace_graph_func, - .flags = FTRACE_OPS_FL_INITIALIZED | - FTRACE_OPS_FL_PID | - FTRACE_OPS_GRAPH_STUB, +void fgraph_init_ops(struct ftrace_ops *dst_ops, + struct ftrace_ops *src_ops) +{ + dst_ops->func = ftrace_graph_func; + dst_ops->flags = FTRACE_OPS_FL_PID | FTRACE_OPS_GRAPH_STUB; + #ifdef FTRACE_GRAPH_TRAMP_ADDR - .trampoline = FTRACE_GRAPH_TRAMP_ADDR, + dst_ops->trampoline = FTRACE_GRAPH_TRAMP_ADDR; /* trampoline_size is only needed for dynamically allocated tramps */ #endif - ASSIGN_OPS_HASH(graph_ops, &global_ops.local_hash) -}; + +#ifdef CONFIG_DYNAMIC_FTRACE + if (src_ops) { + dst_ops->func_hash = &src_ops->local_hash; + mutex_init(&dst_ops->local_hash.regex_lock); + dst_ops->flags |= FTRACE_OPS_FL_INITIALIZED; + } +#endif +} void ftrace_graph_sleep_time_control(bool enable) { @@ -869,11 +914,20 @@ static int start_graph_tracing(void) int register_ftrace_graph(struct fgraph_ops *gops) { + int command = 0; int ret = 0; int i; mutex_lock(&ftrace_lock); + if (!gops->ops.func) { + gops->ops.flags |= FTRACE_OPS_GRAPH_STUB; + gops->ops.func = ftrace_graph_func; +#ifdef FTRACE_GRAPH_TRAMP_ADDR + gops->ops.trampoline = FTRACE_GRAPH_TRAMP_ADDR; +#endif + } + if (!fgraph_array[0]) { /* The array must always have real data on it */ for (i = 0; i < FGRAPH_ARRAY_SIZE; i++) @@ -893,6 +947,7 @@ int register_ftrace_graph(struct fgraph_ops *gops) fgraph_array[i] = gops; if (i + 1 > fgraph_array_cnt) fgraph_array_cnt = i + 1; + gops->idx = i; ftrace_graph_active++; @@ -909,9 +964,10 @@ int register_ftrace_graph(struct fgraph_ops *gops) */ ftrace_graph_return = return_run; ftrace_graph_entry = entry_run; - - ret = ftrace_startup(&graph_ops, FTRACE_START_FUNC_RET); + command = FTRACE_START_FUNC_RET; } + + ret = ftrace_startup(&gops->ops, command); out: mutex_unlock(&ftrace_lock); return ret; @@ -919,6 +975,7 @@ int register_ftrace_graph(struct fgraph_ops *gops) void unregister_ftrace_graph(struct fgraph_ops *gops) { + int command = 0; int i; mutex_lock(&ftrace_lock); @@ -926,25 +983,29 @@ void unregister_ftrace_graph(struct fgraph_ops *gops) if (unlikely(!ftrace_graph_active)) goto out; - for (i = 0; i < fgraph_array_cnt; i++) - if (gops == fgraph_array[i]) - break; - if (i >= fgraph_array_cnt) + if (unlikely(gops->idx < 0 || gops->idx >= fgraph_array_cnt)) goto out; - fgraph_array[i] = &fgraph_stub; - if (i + 1 == fgraph_array_cnt) { - for (; i >= 0; i--) - if (fgraph_array[i] != &fgraph_stub) - break; + WARN_ON_ONCE(fgraph_array[gops->idx] != gops); + + fgraph_array[gops->idx] = &fgraph_stub; + if (gops->idx + 1 == fgraph_array_cnt) { + i = gops->idx; + while (i >= 0 && fgraph_array[i] == &fgraph_stub) + i--; fgraph_array_cnt = i + 1; } ftrace_graph_active--; + + if (!ftrace_graph_active) + command = FTRACE_STOP_FUNC_RET; + + ftrace_shutdown(&gops->ops, command); + if (!ftrace_graph_active) { ftrace_graph_return = ftrace_stub_graph; ftrace_graph_entry = ftrace_graph_entry_stub; - ftrace_shutdown(&graph_ops, FTRACE_STOP_FUNC_RET); unregister_pm_notifier(&ftrace_suspend_notifier); unregister_trace_sched_switch(ftrace_graph_probe_sched_switch, NULL); } diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 83fbfb7b48f8..c4cc2a9d0047 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -3050,6 +3050,8 @@ int ftrace_startup(struct ftrace_ops *ops, int command) if (unlikely(ftrace_disabled)) return -ENODEV; + ftrace_ops_init(ops); + ret = __register_ftrace_function(ops); if (ret) return ret; @@ -7319,7 +7321,7 @@ __init void ftrace_init_global_array_ops(struct trace_array *tr) tr->ops = &global_ops; tr->ops->private = tr; ftrace_init_trace_array(tr); - init_array_fgraph_ops(tr); + init_array_fgraph_ops(tr, tr->ops); } void ftrace_init_array_ops(struct trace_array *tr, ftrace_func_t func) @@ -8051,7 +8053,7 @@ static int register_ftrace_function_nolock(struct ftrace_ops *ops) */ int register_ftrace_function(struct ftrace_ops *ops) { - int ret; + int ret = -1; lock_direct_mutex(); ret = prepare_direct_functions_for_ipmodify(ops); diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h index 16948c0ed00a..02edfdb68933 100644 --- a/kernel/trace/trace.h +++ b/kernel/trace/trace.h @@ -890,8 +890,8 @@ extern int __trace_graph_entry(struct trace_array *tr, extern void __trace_graph_return(struct trace_array *tr, struct ftrace_graph_ret *trace, unsigned int trace_ctx); -extern void init_array_fgraph_ops(struct trace_array *tr); -extern int allocate_fgraph_ops(struct trace_array *tr); +extern void init_array_fgraph_ops(struct trace_array *tr, struct ftrace_ops *ops); +extern int allocate_fgraph_ops(struct trace_array *tr, struct ftrace_ops *ops); extern void free_fgraph_ops(struct trace_array *tr); #ifdef CONFIG_DYNAMIC_FTRACE @@ -974,6 +974,7 @@ static inline int ftrace_graph_notrace_addr(unsigned long addr) preempt_enable_notrace(); return ret; } + #else static inline int ftrace_graph_addr(struct ftrace_graph_ent *trace) { @@ -999,18 +1000,19 @@ static inline bool ftrace_graph_ignore_func(struct ftrace_graph_ent *trace) (fgraph_max_depth && trace->depth >= fgraph_max_depth); } +void fgraph_init_ops(struct ftrace_ops *dst_ops, + struct ftrace_ops *src_ops); + #else /* CONFIG_FUNCTION_GRAPH_TRACER */ static inline enum print_line_t print_graph_function_flags(struct trace_iterator *iter, u32 flags) { return TRACE_TYPE_UNHANDLED; } -static inline void init_array_fgraph_ops(struct trace_array *tr) { } -static inline int allocate_fgraph_ops(struct trace_array *tr) -{ - return 0; -} static inline void free_fgraph_ops(struct trace_array *tr) { } +/* ftrace_ops may not be defined */ +#define init_array_fgraph_ops(tr, ops) do { } while (0) +#define allocate_fgraph_ops(tr, ops) ({ 0; }) #endif /* CONFIG_FUNCTION_GRAPH_TRACER */ extern struct list_head ftrace_pids; diff --git a/kernel/trace/trace_functions.c b/kernel/trace/trace_functions.c index 8e8da0d0ee52..13bf2415245d 100644 --- a/kernel/trace/trace_functions.c +++ b/kernel/trace/trace_functions.c @@ -91,7 +91,7 @@ int ftrace_create_function_files(struct trace_array *tr, if (!tr->ops) return -EINVAL; - ret = allocate_fgraph_ops(tr); + ret = allocate_fgraph_ops(tr, tr->ops); if (ret) { kfree(tr->ops); return ret; diff --git a/kernel/trace/trace_functions_graph.c b/kernel/trace/trace_functions_graph.c index 9ccc904a7703..7f30652f0e97 100644 --- a/kernel/trace/trace_functions_graph.c +++ b/kernel/trace/trace_functions_graph.c @@ -288,7 +288,7 @@ static struct fgraph_ops funcgraph_ops = { .retfunc = &trace_graph_return, }; -int allocate_fgraph_ops(struct trace_array *tr) +int allocate_fgraph_ops(struct trace_array *tr, struct ftrace_ops *ops) { struct fgraph_ops *gops; @@ -301,6 +301,9 @@ int allocate_fgraph_ops(struct trace_array *tr) tr->gops = gops; gops->private = tr; + + fgraph_init_ops(&gops->ops, ops); + return 0; } @@ -309,10 +312,11 @@ void free_fgraph_ops(struct trace_array *tr) kfree(tr->gops); } -__init void init_array_fgraph_ops(struct trace_array *tr) +__init void init_array_fgraph_ops(struct trace_array *tr, struct ftrace_ops *ops) { tr->gops = &funcgraph_ops; funcgraph_ops.private = tr; + fgraph_init_ops(&tr->gops->ops, ops); } static int graph_trace_init(struct trace_array *tr) From patchwork Fri Dec 8 10:26:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 175741 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5363180vqy; Fri, 8 Dec 2023 02:28:06 -0800 (PST) X-Google-Smtp-Source: AGHT+IHP6B+S825A+ufVfcUIZd/g74AP3aZ9riaF1qe13nqzsb0R3TmpZwKGxYz5iH3Tp0NfxmyQ X-Received: by 2002:a17:903:2310:b0:1d0:ca40:15a2 with SMTP id d16-20020a170903231000b001d0ca4015a2mr3809810plh.17.1702031286225; Fri, 08 Dec 2023 02:28:06 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702031286; cv=none; d=google.com; s=arc-20160816; b=AOwB/7wky09Hwsq2yWnIoShXORI3pyMOyFDG2qLmYAZ5VVX77Dm2DjOjqdSUwr5ANl +xuTG7DEMT8VMQeFisRNU7x/ziEffeblsOlrl2AMaQbixoyhCVb8ushBGWWbfkNvZk5W ooITpi/cVHjKvfwnImKlTjRqSvbR1BhF+uLwfopW7+Old/yLveNflontXO9CfDhkRTDZ Rr1COQHvUG1wBSecL9vn5Xg+OyQ3ZdrcPuGALZlOwxChbOiiasHoRhlKlbdTqHuf4b9O Fok+7OFgACLfO3L7u6LchEnpRX6CRKwYSTWaLtU6kuFSEKFMo0/npXClmIGpPBOKybC0 LQog== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=YNA1VX2mDjfQXIz74erT/WzxRw7KR9PXLakkeVYi8E8=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=b2SuoDk3Bzip6SAz1IxUeN7P4LFklO80iWNIgBz2eadzU9QVncfiG3ckD4WpA7D7HD naxnixrsqZ2MhQ7Ur5l3eQqPg+k/f7fsq4UCukOGkjuY0QeJvfzYlybk4PIDWW3FavzL KRYRJPZ+TOepfJSTb983a5r1KwwkWCD0F7Ofg1BgxT0YETgLpqrlAY35jcgtkKnQmZBN 2FO3xgkRkFDFrJcEoF5tSV6dOk8hrVlmX1wPQbwjSO+1TAuL21Vi4O0uesnCUeZwtRE/ pqmXO9VLBjtY0+YYhNHDDeermaQGO45R+1niy3Z5evRlLyD7n2yF1lNFVuA62Nm3z+3p auKw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="Igya/mQ+"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id c10-20020a170902c1ca00b001cffcc110c4si1419115plc.54.2023.12.08.02.28.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 02:28:06 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="Igya/mQ+"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 7B9038236260; Fri, 8 Dec 2023 02:27:54 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231947AbjLHK12 (ORCPT + 99 others); Fri, 8 Dec 2023 05:27:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40172 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1573605AbjLHK07 (ORCPT ); Fri, 8 Dec 2023 05:26:59 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5A6151BC3 for ; Fri, 8 Dec 2023 02:26:47 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D4C1CC433C8; Fri, 8 Dec 2023 10:26:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702031206; bh=cuVa4qIlG+/ZLno0EmFW6if4LFY8jZFXE8ZWIFfPmHg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Igya/mQ+ywbmAkJR9GvJobt6tOa9toUGUERjH95OL6YYsdJgcBaMfdysnIcCIYJLq XK8eD3Hw7rQJzOlPstXQdl4z1eK1b0xXDSr66+hcbRxqTYxmMKQ6yr4z2W6Y8ZR3Fr vr+6lMN+f5Hh6yj1cMImOWvesjL2nXCpXdK8PvL5svmeGilyM9CdyBzgjl5e27cKg0 /F1g9Z8CAk3hd+VnIA7J6kHj9iIWOh85qUsjDNEwrAZjdk/QmfmUVRVurbmI9AGEKo f/gMiO/+L2R/ovvVGVAQjY6m7vcQ87Xv8sLYrh9AFENuMJmmKR5nk52SF1HcW9KiFS QtB+XUBud20og== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v4 12/33] function_graph: Use a simple LRU for fgraph_array index number Date: Fri, 8 Dec 2023 19:26:40 +0900 Message-Id: <170203119964.579004.13369344037111885313.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170203105427.579004.8033550792660734570.stgit@devnote2> References: <170203105427.579004.8033550792660734570.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Fri, 08 Dec 2023 02:27:54 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784709157555642660 X-GMAIL-MSGID: 1784709157555642660 From: Masami Hiramatsu (Google) Since the fgraph_array index is used for the bitmap on the shadow stack, it may leave some entries after a function_graph instance is removed. Thus if another instance reuses the fgraph_array index soon after releasing it, the fgraph may confuse to call the newer callback for the entries which are pushed by the older instance. To avoid reusing the fgraph_array index soon after releasing, introduce a simple LRU table for managing the index number. This will reduce the possibility of this confusion. Signed-off-by: Masami Hiramatsu (Google) --- Changes in v4: - Newly added. --- kernel/trace/fgraph.c | 67 ++++++++++++++++++++++++++++++++++--------------- 1 file changed, 47 insertions(+), 20 deletions(-) diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index 6f537ebd3ed7..d8b139959b62 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -99,10 +99,44 @@ enum { DEFINE_STATIC_KEY_FALSE(kill_ftrace_graph); int ftrace_graph_active; -static int fgraph_array_cnt; - static struct fgraph_ops *fgraph_array[FGRAPH_ARRAY_SIZE]; +/* LRU index table for fgraph_array */ +static int fgraph_lru_table[FGRAPH_ARRAY_SIZE]; +static int fgraph_lru_next; +static int fgraph_lru_last; + +static void fgraph_lru_init(void) +{ + int i; + + for (i = 0; i < FGRAPH_ARRAY_SIZE; i++) + fgraph_lru_table[i] = i; +} + +static int fgraph_lru_release_index(int idx) +{ + if (idx < 0 || idx >= FGRAPH_ARRAY_SIZE || + fgraph_lru_table[fgraph_lru_last] != -1) + return -1; + + fgraph_lru_table[fgraph_lru_last] = idx; + fgraph_lru_last = (fgraph_lru_last + 1) % FGRAPH_ARRAY_SIZE; + return fgraph_lru_last - 1; +} + +static int fgraph_lru_alloc_index(void) +{ + int idx = fgraph_lru_table[fgraph_lru_next]; + + if (idx == -1) + return -1; + + fgraph_lru_table[fgraph_lru_next] = -1; + fgraph_lru_next = (fgraph_lru_next + 1) % FGRAPH_ARRAY_SIZE; + return idx; +} + static inline int get_ret_stack_index(struct task_struct *t, int offset) { return t->ret_stack[offset] & FGRAPH_RET_INDEX_MASK; @@ -367,7 +401,7 @@ int function_graph_enter(unsigned long ret, unsigned long func, if (index < 0) goto out; - for (i = 0; i < fgraph_array_cnt; i++) { + for (i = 0; i < FGRAPH_ARRAY_SIZE; i++) { struct fgraph_ops *gops = fgraph_array[i]; if (gops == &fgraph_stub) @@ -932,21 +966,17 @@ int register_ftrace_graph(struct fgraph_ops *gops) /* The array must always have real data on it */ for (i = 0; i < FGRAPH_ARRAY_SIZE; i++) fgraph_array[i] = &fgraph_stub; + fgraph_lru_init(); } - /* Look for an available spot */ - for (i = 0; i < FGRAPH_ARRAY_SIZE; i++) { - if (fgraph_array[i] == &fgraph_stub) - break; - } - if (i >= FGRAPH_ARRAY_SIZE) { + i = fgraph_lru_alloc_index(); + if (i < 0 || + WARN_ON_ONCE(fgraph_array[i] != &fgraph_stub)) { ret = -EBUSY; goto out; } fgraph_array[i] = gops; - if (i + 1 > fgraph_array_cnt) - fgraph_array_cnt = i + 1; gops->idx = i; ftrace_graph_active++; @@ -976,25 +1006,22 @@ int register_ftrace_graph(struct fgraph_ops *gops) void unregister_ftrace_graph(struct fgraph_ops *gops) { int command = 0; - int i; mutex_lock(&ftrace_lock); if (unlikely(!ftrace_graph_active)) goto out; - if (unlikely(gops->idx < 0 || gops->idx >= fgraph_array_cnt)) + if (unlikely(gops->idx < 0 || gops->idx >= FGRAPH_ARRAY_SIZE)) + goto out; + + if (WARN_ON_ONCE(fgraph_array[gops->idx] != gops)) goto out; - WARN_ON_ONCE(fgraph_array[gops->idx] != gops); + if (fgraph_lru_release_index(gops->idx) < 0) + goto out; fgraph_array[gops->idx] = &fgraph_stub; - if (gops->idx + 1 == fgraph_array_cnt) { - i = gops->idx; - while (i >= 0 && fgraph_array[i] == &fgraph_stub) - i--; - fgraph_array_cnt = i + 1; - } ftrace_graph_active--; From patchwork Fri Dec 8 10:26:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 175742 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5363197vqy; Fri, 8 Dec 2023 02:28:08 -0800 (PST) X-Google-Smtp-Source: AGHT+IEASShy0mZXPcNAACON7C4Hf8t7gZz7DIQZHftXmx3QrXYA3iOMoIIw3ZbRff8pmLbuZIvH X-Received: by 2002:a05:6a20:2c88:b0:18f:97c:616c with SMTP id g8-20020a056a202c8800b0018f097c616cmr3310331pzj.105.1702031288346; Fri, 08 Dec 2023 02:28:08 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702031288; cv=none; d=google.com; s=arc-20160816; b=EjzqMMcWcdjyZf8p4I6Q/JXX3+kkaeRp8i7p3W6Bfx7YkKf5ujTSnJ6EYfYWUXPpSI l25bWTaWGVkvHAT8iWUuFApvkbLlCx0vpzUOB4eNMAUnpO0NW1rjUSR9Xy5u7Iz8eUwS eXLBE9ORS26FNITrszhDUzd9yGFUk5xw/jDc2rDwat8oFTiZGZPf79yiC+AbFB1GCEZe PK95F0ml/zyVs4pWePn9KIN6cz9asa0wok8Se4m3tbvc3K//ZscIGODxYlLBXiNWatYR ocKG3DZErT/FEZXw9i34t1FL5gNQEJeCXZJZ1cFDga8BY9LXer4uyjGRdIaZJWfR1ATh P8FA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=PWXWaFepqAVVW9L+6TX7cSF1LPi5uy45FaydPPcFuhA=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=JDQ6rLy78iAILGErm2ygeestKlOFLu4VP7n7rWO9IU8eaAN9yTQsNkDJgDMUPYVNPM H/s/h7kEuagCcv5pyZCzLGI9x16vxy0WMhr6Rm3snWCeSULzv9AwaD/DSdsHn0PqLU6t 3sqcfT+rT546/4zr36yer66CChJZgc2NQZc927ForlU7u7u312SXA5muOfo6qOsS8dH0 +B+D1no7luTMivZMJ7Oc0KJiV/dVLACLJCxBZgYqhqI+yh3kdkQ54RXDNpcA5bs86B90 8hrujmwlLUl9kEu4SbuJgnZO2/36/qlEN/SALN1QBG3tTIJd/wm2V37rsc83IoXB/0la 3/tw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=AE3ZrLBO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from pete.vger.email (pete.vger.email. [23.128.96.36]) by mx.google.com with ESMTPS id y6-20020a17090322c600b001d1db5e39b0si1399325plg.222.2023.12.08.02.28.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 02:28:08 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=AE3ZrLBO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 7731E80239B9; Fri, 8 Dec 2023 02:28:03 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1573622AbjLHK1n (ORCPT + 99 others); Fri, 8 Dec 2023 05:27:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33838 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1573614AbjLHK1Q (ORCPT ); Fri, 8 Dec 2023 05:27:16 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 600901BFD for ; Fri, 8 Dec 2023 02:26:59 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 26FFDC433C7; Fri, 8 Dec 2023 10:26:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702031219; bh=frjvUNWsEXAN4Oj6+E7AclD8c5w7rL2KNggyyZAqoiM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=AE3ZrLBO7Q2bgSACGTdfH/xFH7UqbP2GrsNUXwgLD+yakLcP+mCuAJXpOB4Wy34cq C/Y1+SgmyVzKOoarBTgZEAlfNmqpwlseUWID0U0rNXW8OvHMSZUWOzHY9Iw8Dsa1mP GwGHJm/uaDV6aKql32r7fAM0g/kP8lFjMjDvWSv+pA+t7i83UYL8iGgvPlVCPaLDX7 CPKprmRiq5xltxVBs3z2hrvNU6p2lcpGYbJYLCr+RM6ivz5eOz1DRv+zkJNjfCHrFO JjvC5nNCJOOLu8G7JgFPpWfQELX743bYW+TWsCW0BiJvScpJKjEJF2ZibMmpxK3RJG QoVnFSZIBg+yw== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v4 13/33] function_graph: Add "task variables" per task for fgraph_ops Date: Fri, 8 Dec 2023 19:26:52 +0900 Message-Id: <170203121210.579004.5854708719973629833.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170203105427.579004.8033550792660734570.stgit@devnote2> References: <170203105427.579004.8033550792660734570.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Fri, 08 Dec 2023 02:28:03 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784709160425393636 X-GMAIL-MSGID: 1784709160425393636 From: Steven Rostedt (VMware) Add a "task variables" array on the tasks shadow ret_stack that is the size of longs for each possible registered fgraph_ops. That's a total of 16, taking up 8 * 16 = 128 bytes (out of a page size 4k). This will allow for fgraph_ops to do specific features on a per task basis having a way to maintain state for each task. Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- Changes in v3: - Move fgraph_ops::idx to previous patch in the series. Changes in v2: - Make description lines shorter than 76 chars. --- include/linux/ftrace.h | 1 + kernel/trace/fgraph.c | 70 +++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 70 insertions(+), 1 deletion(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index c431a33fe789..09ca4bba63f2 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -1116,6 +1116,7 @@ ftrace_graph_get_ret_stack(struct task_struct *task, int idx); unsigned long ftrace_graph_ret_addr(struct task_struct *task, int *idx, unsigned long ret, unsigned long *retp); +unsigned long *fgraph_get_task_var(struct fgraph_ops *gops); /* * Sometimes we don't want to trace a function with the function diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index d8b139959b62..5ac6477bcf20 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -92,10 +92,18 @@ enum { #define SHADOW_STACK_SIZE (PAGE_SIZE) #define SHADOW_STACK_INDEX (SHADOW_STACK_SIZE / sizeof(long)) /* Leave on a buffer at the end */ -#define SHADOW_STACK_MAX_INDEX (SHADOW_STACK_INDEX - (FGRAPH_RET_INDEX + 1)) +#define SHADOW_STACK_MAX_INDEX \ + (SHADOW_STACK_INDEX - (FGRAPH_RET_INDEX + 1 + FGRAPH_ARRAY_SIZE)) #define RET_STACK(t, index) ((struct ftrace_ret_stack *)(&(t)->ret_stack[index])) +/* + * Each fgraph_ops has a reservered unsigned long at the end (top) of the + * ret_stack to store task specific state. + */ +#define SHADOW_STACK_TASK_VARS(ret_stack) \ + ((unsigned long *)(&(ret_stack)[SHADOW_STACK_INDEX - FGRAPH_ARRAY_SIZE])) + DEFINE_STATIC_KEY_FALSE(kill_ftrace_graph); int ftrace_graph_active; @@ -182,6 +190,44 @@ static void return_run(struct ftrace_graph_ret *trace, struct fgraph_ops *ops) { } +static void ret_stack_set_task_var(struct task_struct *t, int idx, long val) +{ + unsigned long *gvals = SHADOW_STACK_TASK_VARS(t->ret_stack); + + gvals[idx] = val; +} + +static unsigned long * +ret_stack_get_task_var(struct task_struct *t, int idx) +{ + unsigned long *gvals = SHADOW_STACK_TASK_VARS(t->ret_stack); + + return &gvals[idx]; +} + +static void ret_stack_init_task_vars(unsigned long *ret_stack) +{ + unsigned long *gvals = SHADOW_STACK_TASK_VARS(ret_stack); + + memset(gvals, 0, sizeof(*gvals) * FGRAPH_ARRAY_SIZE); +} + +/** + * fgraph_get_task_var - retrieve a task specific state variable + * @gops: The ftrace_ops that owns the task specific variable + * + * Every registered fgraph_ops has a task state variable + * reserved on the task's ret_stack. This function returns the + * address to that variable. + * + * Returns the address to the fgraph_ops @gops tasks specific + * unsigned long variable. + */ +unsigned long *fgraph_get_task_var(struct fgraph_ops *gops) +{ + return ret_stack_get_task_var(current, gops->idx); +} + /* * @offset: The index into @t->ret_stack to find the ret_stack entry * @index: Where to place the index into @t->ret_stack of that entry @@ -788,6 +834,7 @@ static int alloc_retstack_tasklist(unsigned long **ret_stack_list) if (t->ret_stack == NULL) { atomic_set(&t->trace_overrun, 0); + ret_stack_init_task_vars(ret_stack_list[start]); t->curr_ret_stack = 0; t->curr_ret_depth = -1; /* Make sure the tasks see the 0 first: */ @@ -848,6 +895,7 @@ static void graph_init_task(struct task_struct *t, unsigned long *ret_stack) { atomic_set(&t->trace_overrun, 0); + ret_stack_init_task_vars(ret_stack); t->ftrace_timestamp = 0; t->curr_ret_stack = 0; t->curr_ret_depth = -1; @@ -946,6 +994,24 @@ static int start_graph_tracing(void) return ret; } +static void init_task_vars(int idx) +{ + struct task_struct *g, *t; + int cpu; + + for_each_online_cpu(cpu) { + if (idle_task(cpu)->ret_stack) + ret_stack_set_task_var(idle_task(cpu), idx, 0); + } + + read_lock(&tasklist_lock); + for_each_process_thread(g, t) { + if (t->ret_stack) + ret_stack_set_task_var(t, idx, 0); + } + read_unlock(&tasklist_lock); +} + int register_ftrace_graph(struct fgraph_ops *gops) { int command = 0; @@ -995,6 +1061,8 @@ int register_ftrace_graph(struct fgraph_ops *gops) ftrace_graph_return = return_run; ftrace_graph_entry = entry_run; command = FTRACE_START_FUNC_RET; + } else { + init_task_vars(gops->idx); } ret = ftrace_startup(&gops->ops, command); From patchwork Fri Dec 8 10:27:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 175744 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5363428vqy; Fri, 8 Dec 2023 02:28:40 -0800 (PST) X-Google-Smtp-Source: AGHT+IHU7ZS7doqS3BI2XpIj02BDFO3gcUgzR3DkF9GHSVgzEz2wusA8Ev6NvILi9rBgs5UFMJeA X-Received: by 2002:a17:90a:ce02:b0:286:e264:999b with SMTP id f2-20020a17090ace0200b00286e264999bmr3295038pju.59.1702031319785; Fri, 08 Dec 2023 02:28:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702031319; cv=none; d=google.com; s=arc-20160816; b=fwSjupSaygOAEGvGOODdaZf+1ZlMHsJe7ExxMn555NqymIYpUhKbOburrVOJvSRrsS pWAbo4mva8q9fS9Sjgied5hThAu94RajXnpS5L2SBkji7w22UMRliwS3iabqDnszF9bZ FJEJ0lbLb3d4q6SynOUMTErY+Yy+FWhhngPiLWTpuFNhi8fv+B9WKPUZY/PZrI+58GXr 6Cmlfmp3o1qAzvUGs3CgiMppSznodhwAwR1dyz+JljFQqy6kArqYbQFNLCNvZW+YPepz KSOjllgm0CNtKoEjFtElrfbbyiXI/mNiiq9YB0HGi2XWBi3sUlCV8p83ULqWzLdENkdg GF5Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=9U/KqGMOqrg0xoGTJe7+mvfqdxSl3TOL4RHNtdZdsUw=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=hsFcVA4iSo6iT9W7EAWmRrxRhEs62IqkcvHvrJX3MeSVOep8GnjOARYneTAWuQPyHW Gp0h7O3lzipAmiPXCsiH6UNdA4uDUfCU1NLSWp6Y9o+fp1nxkp5XnezAFLAkIbpe5mLx s6NW3Ampl6w7dO5WtWg5Gx/YO/IW1r6amUItvly5slZecesI8BMS0AJabmODQT2pzQ+y M1KW8XpzdKJX0qH+mthGTLGXArHxoLTrpFxT1Ugw80drc7FIStDjHPxqyeANG59kjNmC X9QmgoRXIdZtoILiL745ROByzfSjqpsHnZv7ekfESe9rB9ycITMd53YKxVFq/OJyQ0cc IkYQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=lSZaHsCB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id oa18-20020a17090b1bd200b00286bd949d17si1494477pjb.81.2023.12.08.02.28.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 02:28:39 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=lSZaHsCB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 780128236265; Fri, 8 Dec 2023 02:28:36 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1573708AbjLHK1y (ORCPT + 99 others); Fri, 8 Dec 2023 05:27:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60296 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235950AbjLHK10 (ORCPT ); Fri, 8 Dec 2023 05:27:26 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AD3C8199A for ; Fri, 8 Dec 2023 02:27:11 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 545BBC433C8; Fri, 8 Dec 2023 10:27:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702031231; bh=1xFk1PI8w6iqmqfcLM9Z1JDlLC9jWBDIZWbfYmgwqf8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lSZaHsCBNxhsJo8XpWSnj8DWx3tBlOpTZlzeJ4KPvTFuL9741m+MnRdAjd7sG46TW uGrFz1IxfyUFQsdd8pCICmxigWQKvu5b4ucbhg/2q5b5l696EaRXz+wl3Zzm1xwZVY 3uizIRG/+2EwjjluhcKWrRD2+LAx9RjQ4A+NVwEVgx/mctzztiYwF91OWICsY0qXX4 wcGcb/UsP2LB2bk/x4uGLuw6DsZAkF2cRjtEhn3yS2CmpXoMp6lJfZ8GfnbYValfgj Wa6VuCKvpp15WLXw8B0OwU3BovmZW6HZkBIc+JzflVpcrjCaT47IhoQfwL54y08FRt 45S7BePo2c+pQ== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v4 14/33] function_graph: Move set_graph_function tests to shadow stack global var Date: Fri, 8 Dec 2023 19:27:04 +0900 Message-Id: <170203122438.579004.8204078413161144185.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170203105427.579004.8033550792660734570.stgit@devnote2> References: <170203105427.579004.8033550792660734570.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Fri, 08 Dec 2023 02:28:36 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784709193243584936 X-GMAIL-MSGID: 1784709193243584936 From: Steven Rostedt (VMware) The use of the task->trace_recursion for the logic used for the set_graph_funnction was a bit of an abuse of that variable. Now that there exists global vars that are per stack for registered graph traces, use that instead. Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- include/linux/trace_recursion.h | 5 +---- kernel/trace/trace.h | 32 +++++++++++++++++++++----------- kernel/trace/trace_functions_graph.c | 6 +++--- kernel/trace/trace_irqsoff.c | 4 ++-- kernel/trace/trace_sched_wakeup.c | 4 ++-- 5 files changed, 29 insertions(+), 22 deletions(-) diff --git a/include/linux/trace_recursion.h b/include/linux/trace_recursion.h index d48cd92d2364..2efd5ec46d7f 100644 --- a/include/linux/trace_recursion.h +++ b/include/linux/trace_recursion.h @@ -44,9 +44,6 @@ enum { */ TRACE_IRQ_BIT, - /* Set if the function is in the set_graph_function file */ - TRACE_GRAPH_BIT, - /* * In the very unlikely case that an interrupt came in * at a start of graph tracing, and we want to trace @@ -60,7 +57,7 @@ enum { * that preempted a softirq start of a function that * preempted normal context!!!! Luckily, it can't be * greater than 3, so the next two bits are a mask - * of what the depth is when we set TRACE_GRAPH_BIT + * of what the depth is when we set TRACE_GRAPH_FL */ TRACE_GRAPH_DEPTH_START_BIT, diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h index 02edfdb68933..3b9266aeb43b 100644 --- a/kernel/trace/trace.h +++ b/kernel/trace/trace.h @@ -894,11 +894,16 @@ extern void init_array_fgraph_ops(struct trace_array *tr, struct ftrace_ops *ops extern int allocate_fgraph_ops(struct trace_array *tr, struct ftrace_ops *ops); extern void free_fgraph_ops(struct trace_array *tr); +enum { + TRACE_GRAPH_FL = 1, +}; + #ifdef CONFIG_DYNAMIC_FTRACE extern struct ftrace_hash __rcu *ftrace_graph_hash; extern struct ftrace_hash __rcu *ftrace_graph_notrace_hash; -static inline int ftrace_graph_addr(struct ftrace_graph_ent *trace) +static inline int +ftrace_graph_addr(unsigned long *task_var, struct ftrace_graph_ent *trace) { unsigned long addr = trace->func; int ret = 0; @@ -920,12 +925,11 @@ static inline int ftrace_graph_addr(struct ftrace_graph_ent *trace) } if (ftrace_lookup_ip(hash, addr)) { - /* * This needs to be cleared on the return functions * when the depth is zero. */ - trace_recursion_set(TRACE_GRAPH_BIT); + *task_var |= TRACE_GRAPH_FL; trace_recursion_set_depth(trace->depth); /* @@ -945,11 +949,14 @@ static inline int ftrace_graph_addr(struct ftrace_graph_ent *trace) return ret; } -static inline void ftrace_graph_addr_finish(struct ftrace_graph_ret *trace) +static inline void +ftrace_graph_addr_finish(struct fgraph_ops *gops, struct ftrace_graph_ret *trace) { - if (trace_recursion_test(TRACE_GRAPH_BIT) && + unsigned long *task_var = fgraph_get_task_var(gops); + + if ((*task_var & TRACE_GRAPH_FL) && trace->depth == trace_recursion_depth()) - trace_recursion_clear(TRACE_GRAPH_BIT); + *task_var &= ~TRACE_GRAPH_FL; } static inline int ftrace_graph_notrace_addr(unsigned long addr) @@ -976,7 +983,7 @@ static inline int ftrace_graph_notrace_addr(unsigned long addr) } #else -static inline int ftrace_graph_addr(struct ftrace_graph_ent *trace) +static inline int ftrace_graph_addr(unsigned long *task_var, struct ftrace_graph_ent *trace) { return 1; } @@ -985,17 +992,20 @@ static inline int ftrace_graph_notrace_addr(unsigned long addr) { return 0; } -static inline void ftrace_graph_addr_finish(struct ftrace_graph_ret *trace) +static inline void ftrace_graph_addr_finish(struct fgraph_ops *gops, struct ftrace_graph_ret *trace) { } #endif /* CONFIG_DYNAMIC_FTRACE */ extern unsigned int fgraph_max_depth; -static inline bool ftrace_graph_ignore_func(struct ftrace_graph_ent *trace) +static inline bool +ftrace_graph_ignore_func(struct fgraph_ops *gops, struct ftrace_graph_ent *trace) { + unsigned long *task_var = fgraph_get_task_var(gops); + /* trace it when it is-nested-in or is a function enabled. */ - return !(trace_recursion_test(TRACE_GRAPH_BIT) || - ftrace_graph_addr(trace)) || + return !((*task_var & TRACE_GRAPH_FL) || + ftrace_graph_addr(task_var, trace)) || (trace->depth < 0) || (fgraph_max_depth && trace->depth >= fgraph_max_depth); } diff --git a/kernel/trace/trace_functions_graph.c b/kernel/trace/trace_functions_graph.c index 7f30652f0e97..66cce73e94f8 100644 --- a/kernel/trace/trace_functions_graph.c +++ b/kernel/trace/trace_functions_graph.c @@ -160,7 +160,7 @@ int trace_graph_entry(struct ftrace_graph_ent *trace, if (!ftrace_trace_task(tr)) return 0; - if (ftrace_graph_ignore_func(trace)) + if (ftrace_graph_ignore_func(gops, trace)) return 0; if (ftrace_graph_ignore_irqs()) @@ -247,7 +247,7 @@ void trace_graph_return(struct ftrace_graph_ret *trace, long disabled; int cpu; - ftrace_graph_addr_finish(trace); + ftrace_graph_addr_finish(gops, trace); if (trace_recursion_test(TRACE_GRAPH_NOTRACE_BIT)) { trace_recursion_clear(TRACE_GRAPH_NOTRACE_BIT); @@ -269,7 +269,7 @@ void trace_graph_return(struct ftrace_graph_ret *trace, static void trace_graph_thresh_return(struct ftrace_graph_ret *trace, struct fgraph_ops *gops) { - ftrace_graph_addr_finish(trace); + ftrace_graph_addr_finish(gops, trace); if (trace_recursion_test(TRACE_GRAPH_NOTRACE_BIT)) { trace_recursion_clear(TRACE_GRAPH_NOTRACE_BIT); diff --git a/kernel/trace/trace_irqsoff.c b/kernel/trace/trace_irqsoff.c index 5478f4c4f708..fce064e20570 100644 --- a/kernel/trace/trace_irqsoff.c +++ b/kernel/trace/trace_irqsoff.c @@ -184,7 +184,7 @@ static int irqsoff_graph_entry(struct ftrace_graph_ent *trace, unsigned int trace_ctx; int ret; - if (ftrace_graph_ignore_func(trace)) + if (ftrace_graph_ignore_func(gops, trace)) return 0; /* * Do not trace a function if it's filtered by set_graph_notrace. @@ -214,7 +214,7 @@ static void irqsoff_graph_return(struct ftrace_graph_ret *trace, unsigned long flags; unsigned int trace_ctx; - ftrace_graph_addr_finish(trace); + ftrace_graph_addr_finish(gops, trace); if (!func_prolog_dec(tr, &data, &flags)) return; diff --git a/kernel/trace/trace_sched_wakeup.c b/kernel/trace/trace_sched_wakeup.c index 49bcc812652c..130ca7e7787e 100644 --- a/kernel/trace/trace_sched_wakeup.c +++ b/kernel/trace/trace_sched_wakeup.c @@ -120,7 +120,7 @@ static int wakeup_graph_entry(struct ftrace_graph_ent *trace, unsigned int trace_ctx; int ret = 0; - if (ftrace_graph_ignore_func(trace)) + if (ftrace_graph_ignore_func(gops, trace)) return 0; /* * Do not trace a function if it's filtered by set_graph_notrace. @@ -149,7 +149,7 @@ static void wakeup_graph_return(struct ftrace_graph_ret *trace, struct trace_array_cpu *data; unsigned int trace_ctx; - ftrace_graph_addr_finish(trace); + ftrace_graph_addr_finish(gops, trace); if (!func_prolog_preempt_disable(tr, &data, &trace_ctx)) return; From patchwork Fri Dec 8 10:27:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 175745 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5363438vqy; Fri, 8 Dec 2023 02:28:42 -0800 (PST) X-Google-Smtp-Source: AGHT+IEDXPZWWRWdqqAT6NBEIyTk/QFHlHuQAv1rMQzgYGpOwsPITNykshhg2no0ar7vAFktbzlp X-Received: by 2002:a05:6a20:a199:b0:18f:97c:928a with SMTP id r25-20020a056a20a19900b0018f097c928amr2591528pzk.111.1702031321815; Fri, 08 Dec 2023 02:28:41 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702031321; cv=none; d=google.com; s=arc-20160816; b=wMxb2w80tkkzAH+kKM14fpCSQWsRTUYFYxVVWnqD0gmyg8sib5ZEEzBJIc+/tyjDJL fq3gW6RHAiHzUvCFsnHaWB4yYT4XwKD7Mk9llPoyB2BlARqprXKxO0PUAeme4o0vu3XF OGHzg88c4CctMDrZIr/eshHzkrdS7xs3JiUkQ1dVdl+eMaeergEGg3Irbocf7ya0M0RP kg74ljNsK3aSE5X1qUSu9X4APcNaB8G6YXKBdqtf8jq1T9IqQWSRLT7YmDLRomwddtc1 cYxE0gzEi6/ayABWbAWsm5BugucLgfH+H2Ug637OiNkflsWWUaX8t69oCy5l5UsGB6s8 9B/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=Oq9Fo56RGVbai0FVad+d2OOgEML+m+c+a5p19CzL7Jg=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=QeLgR+7zm98H9kRRkI+hD4nNW9JftZfhT3wsqoL5zDg5KB7huPeeI0IvX8YNZJh73J SINoL/sIQWDh6rlLDRzhU66p2Cx0uoYQ2WGuDZbCD3C2pX83UI0YhDOOOBm7gwLUPHyK NdV3n+aqm7tRBzuzD0NOMcX+XlIHcOiolV0ILV/b5qL1OE7SOuSXWW7BNA1wFiIKK2vs Bm2b1At4TRzFarqts+DOzNOviv5v2zaaShtArnotnain2byomVvHcoW93uHL4/qoq4MA pGkbiLlelyWE5Yb+YzSZnpgN/mHcQSArZu/rLoUNyroIwkC7bgbUY8Knj345zMU7NSyK kgEA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=C2t7RaJQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id p22-20020a056a000b5600b006ce6da9d524si1301922pfo.326.2023.12.08.02.28.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 02:28:41 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=C2t7RaJQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 503BB887B748; Fri, 8 Dec 2023 02:28:37 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1573661AbjLHK2B (ORCPT + 99 others); Fri, 8 Dec 2023 05:28:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60850 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1573663AbjLHK1h (ORCPT ); Fri, 8 Dec 2023 05:27:37 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C68C619AD for ; Fri, 8 Dec 2023 02:27:23 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 928C1C433C9; Fri, 8 Dec 2023 10:27:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702031243; bh=4gnOPh72MBp2/UPzOPml+kmyjzVijWhiPGlAlFNB6DE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=C2t7RaJQSrT7iFJnVP9fCJZd6lBh0JXumou5ITkx3xjoSL4I97zMftxsjTZ7AEhJ2 reTSrGBdgDX/cGHVnHlD1B6h6uq1pRZYEypdqf1yQSFu2520p15BD49riBz1GAgeAF wPr8/wogs2mMi9jC6zCTu8YuIlqS6NtqUezMH60slaHpyOltvRtAzwKBzWiN+nW0pE MmPMAVJU9PRoyUgu0aM4NQwUAcevT5Yg+E7JjY7/jtikEBgsTjjuUK61EfA6eIYKaW OGk6BPrKeT2ypowDVpR345n+A0ZmGS9hV7y0wMtqb+yv+92EF09/kflptVaJos1KUS AFKkHXquXzkRQ== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v4 15/33] function_graph: Move graph depth stored data to shadow stack global var Date: Fri, 8 Dec 2023 19:27:17 +0900 Message-Id: <170203123663.579004.7455942203810529166.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170203105427.579004.8033550792660734570.stgit@devnote2> References: <170203105427.579004.8033550792660734570.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Fri, 08 Dec 2023 02:28:37 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784709195241719375 X-GMAIL-MSGID: 1784709195241719375 From: Steven Rostedt (VMware) The use of the task->trace_recursion for the logic used for the function graph depth was a bit of an abuse of that variable. Now that there exists global vars that are per stack for registered graph traces, use that instead. Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- include/linux/trace_recursion.h | 29 ----------------------------- kernel/trace/trace.h | 34 ++++++++++++++++++++++++++++++++-- 2 files changed, 32 insertions(+), 31 deletions(-) diff --git a/include/linux/trace_recursion.h b/include/linux/trace_recursion.h index 2efd5ec46d7f..00e792bf148d 100644 --- a/include/linux/trace_recursion.h +++ b/include/linux/trace_recursion.h @@ -44,25 +44,6 @@ enum { */ TRACE_IRQ_BIT, - /* - * In the very unlikely case that an interrupt came in - * at a start of graph tracing, and we want to trace - * the function in that interrupt, the depth can be greater - * than zero, because of the preempted start of a previous - * trace. In an even more unlikely case, depth could be 2 - * if a softirq interrupted the start of graph tracing, - * followed by an interrupt preempting a start of graph - * tracing in the softirq, and depth can even be 3 - * if an NMI came in at the start of an interrupt function - * that preempted a softirq start of a function that - * preempted normal context!!!! Luckily, it can't be - * greater than 3, so the next two bits are a mask - * of what the depth is when we set TRACE_GRAPH_FL - */ - - TRACE_GRAPH_DEPTH_START_BIT, - TRACE_GRAPH_DEPTH_END_BIT, - /* * To implement set_graph_notrace, if this bit is set, we ignore * function graph tracing of called functions, until the return @@ -78,16 +59,6 @@ enum { #define trace_recursion_clear(bit) do { (current)->trace_recursion &= ~(1<<(bit)); } while (0) #define trace_recursion_test(bit) ((current)->trace_recursion & (1<<(bit))) -#define trace_recursion_depth() \ - (((current)->trace_recursion >> TRACE_GRAPH_DEPTH_START_BIT) & 3) -#define trace_recursion_set_depth(depth) \ - do { \ - current->trace_recursion &= \ - ~(3 << TRACE_GRAPH_DEPTH_START_BIT); \ - current->trace_recursion |= \ - ((depth) & 3) << TRACE_GRAPH_DEPTH_START_BIT; \ - } while (0) - #define TRACE_CONTEXT_BITS 4 #define TRACE_FTRACE_START TRACE_FTRACE_BIT diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h index 3b9266aeb43b..e3f452eda0e3 100644 --- a/kernel/trace/trace.h +++ b/kernel/trace/trace.h @@ -896,8 +896,38 @@ extern void free_fgraph_ops(struct trace_array *tr); enum { TRACE_GRAPH_FL = 1, + + /* + * In the very unlikely case that an interrupt came in + * at a start of graph tracing, and we want to trace + * the function in that interrupt, the depth can be greater + * than zero, because of the preempted start of a previous + * trace. In an even more unlikely case, depth could be 2 + * if a softirq interrupted the start of graph tracing, + * followed by an interrupt preempting a start of graph + * tracing in the softirq, and depth can even be 3 + * if an NMI came in at the start of an interrupt function + * that preempted a softirq start of a function that + * preempted normal context!!!! Luckily, it can't be + * greater than 3, so the next two bits are a mask + * of what the depth is when we set TRACE_GRAPH_FL + */ + + TRACE_GRAPH_DEPTH_START_BIT, + TRACE_GRAPH_DEPTH_END_BIT, }; +static inline unsigned long ftrace_graph_depth(unsigned long *task_var) +{ + return (*task_var >> TRACE_GRAPH_DEPTH_START_BIT) & 3; +} + +static inline void ftrace_graph_set_depth(unsigned long *task_var, int depth) +{ + *task_var &= ~(3 << TRACE_GRAPH_DEPTH_START_BIT); + *task_var |= (depth & 3) << TRACE_GRAPH_DEPTH_START_BIT; +} + #ifdef CONFIG_DYNAMIC_FTRACE extern struct ftrace_hash __rcu *ftrace_graph_hash; extern struct ftrace_hash __rcu *ftrace_graph_notrace_hash; @@ -930,7 +960,7 @@ ftrace_graph_addr(unsigned long *task_var, struct ftrace_graph_ent *trace) * when the depth is zero. */ *task_var |= TRACE_GRAPH_FL; - trace_recursion_set_depth(trace->depth); + ftrace_graph_set_depth(task_var, trace->depth); /* * If no irqs are to be traced, but a set_graph_function @@ -955,7 +985,7 @@ ftrace_graph_addr_finish(struct fgraph_ops *gops, struct ftrace_graph_ret *trace unsigned long *task_var = fgraph_get_task_var(gops); if ((*task_var & TRACE_GRAPH_FL) && - trace->depth == trace_recursion_depth()) + trace->depth == ftrace_graph_depth(task_var)) *task_var &= ~TRACE_GRAPH_FL; } From patchwork Fri Dec 8 10:27:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 175743 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5363353vqy; Fri, 8 Dec 2023 02:28:31 -0800 (PST) X-Google-Smtp-Source: AGHT+IGkDMlZHQceX+7oco/BLIQUId2NWnFFwH0soaEYmEKeVgMWisBHbaJc45mz5VcZ5gjdp5Sl X-Received: by 2002:a17:90b:23c3:b0:286:f625:d42d with SMTP id md3-20020a17090b23c300b00286f625d42dmr920609pjb.21.1702031310761; Fri, 08 Dec 2023 02:28:30 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702031310; cv=none; d=google.com; s=arc-20160816; b=x5eBzo6qdXJI7t9vEkm6l5Y0PzrZblDe2bJelJLqd5C9v5kLqCWT9hGoB4qfxcJgMG aCnea7/6YzdJ3KKYAnsPECwJCs7+Ih8U+nFWn5CsGJv5GG7MO+Wo42/n0QQ3vg8xXgrY yIh9kfuFtPgJldwfsM6uE5y9XNQ+BK2UX1SbWa+Q9SzG3Kfl3jk2sYFWo+gLA506AJof zLoA2r2Q/cB78207FuW7svQ5H0v2OkP159oiXJMmrzmSeO9iK2r63MQOyaBL/59RH968 VTdFJ2etN4/CbCQvKJj1p1YX/HAOvneTlOCIOiJyQbKQDnTw+r99+V/2ZbKDnS/Vkdae TneA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=OYBJw1NfeATJxeGscVzw9H+t8LsucyyBedZ6vHChRco=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=i4r6tEMs0zVyBFACu8ekrRSlKLezBNfRvVZ8LnQTY7/QzS13tBgmSmPsnteV4S+aFJ gtL6DAO58t7hZQPR3nogm8svMPA3d0BMIB+QAS+cb68vX4BndQAoXWQhAoJcsp8KLZ5O VGQHOWTkn+bPxSw3Hv/8sZo4EqfMZxjMNYEG3HfUuUtUzu2VkmUMHg3eKzYDvZ/iJ7CD 3hp2D2aUDzqdAuALxYd1B/NsBfpAenmed1/6NJvTpNGQ3MMm+P1lXPdkhI9C5OAH4Ixh zF5qrGs5SVEGJld9yMK7IS+LFtMhntbHQ85a7l9EgnngbCcic3WmXc4IxMY2uIBq+964 UgOw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=jdCIWqT+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from pete.vger.email (pete.vger.email. [2620:137:e000::3:6]) by mx.google.com with ESMTPS id h13-20020a17090ac38d00b0027d04b581dcsi2745700pjt.72.2023.12.08.02.28.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 02:28:30 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) client-ip=2620:137:e000::3:6; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=jdCIWqT+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 69F3382C4008; Fri, 8 Dec 2023 02:28:25 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1573514AbjLHK2H (ORCPT + 99 others); Fri, 8 Dec 2023 05:28:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60906 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1573683AbjLHK1n (ORCPT ); Fri, 8 Dec 2023 05:27:43 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 02F4819BB for ; Fri, 8 Dec 2023 02:27:36 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C06C3C433C8; Fri, 8 Dec 2023 10:27:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702031255; bh=x44g32Pd9O3zsRrhngWEUNn/+TG68nwVi1AhDspvG5g=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jdCIWqT+IlDbMobTp0pglIWjDIs9sbMpmSUFclPYuQhTBQLfXiWyU5S+MxQyU1kaw 9s7BxlgrH7oiR+At2KTIUfz7lsQecdDiJwHYpiEptbTa01+ely2ZmE1bEVVVwQO7ES KYxKI5zPUY7GI57pUopaLJqGnvwZ3zu8qNoP9uQla78LB52sFAv/6KQxX2kry/MzHW 6arZJpdUIHVqIJ4VID2228+Qyt5G/3rdOwFJi6Mit5Qr8mHvwVA+VL7/gigY7vqfdv QvvM2GpBanMvGmxx7dypN00qSjF2Vm2n6l/aqP+CXJB4Rcbs8qBfMzd8M/UW344Enr zpDDKL+FAnCwg== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v4 16/33] function_graph: Move graph notrace bit to shadow stack global var Date: Fri, 8 Dec 2023 19:27:29 +0900 Message-Id: <170203124882.579004.5300962282313539942.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170203105427.579004.8033550792660734570.stgit@devnote2> References: <170203105427.579004.8033550792660734570.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Fri, 08 Dec 2023 02:28:25 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784709183385686835 X-GMAIL-MSGID: 1784709183385686835 From: Steven Rostedt (VMware) The use of the task->trace_recursion for the logic used for the function graph no-trace was a bit of an abuse of that variable. Now that there exists global vars that are per stack for registered graph traces, use that instead. Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- Changes in v2: - Make description lines shorter than 76 chars. --- include/linux/trace_recursion.h | 7 ------- kernel/trace/trace.h | 9 +++++++++ kernel/trace/trace_functions_graph.c | 10 ++++++---- 3 files changed, 15 insertions(+), 11 deletions(-) diff --git a/include/linux/trace_recursion.h b/include/linux/trace_recursion.h index 00e792bf148d..cc11b0e9d220 100644 --- a/include/linux/trace_recursion.h +++ b/include/linux/trace_recursion.h @@ -44,13 +44,6 @@ enum { */ TRACE_IRQ_BIT, - /* - * To implement set_graph_notrace, if this bit is set, we ignore - * function graph tracing of called functions, until the return - * function is called to clear it. - */ - TRACE_GRAPH_NOTRACE_BIT, - /* Used to prevent recursion recording from recursing. */ TRACE_RECORD_RECURSION_BIT, }; diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h index e3f452eda0e3..fac29f08d5d8 100644 --- a/kernel/trace/trace.h +++ b/kernel/trace/trace.h @@ -915,8 +915,17 @@ enum { TRACE_GRAPH_DEPTH_START_BIT, TRACE_GRAPH_DEPTH_END_BIT, + + /* + * To implement set_graph_notrace, if this bit is set, we ignore + * function graph tracing of called functions, until the return + * function is called to clear it. + */ + TRACE_GRAPH_NOTRACE_BIT, }; +#define TRACE_GRAPH_NOTRACE (1 << TRACE_GRAPH_NOTRACE_BIT) + static inline unsigned long ftrace_graph_depth(unsigned long *task_var) { return (*task_var >> TRACE_GRAPH_DEPTH_START_BIT) & 3; diff --git a/kernel/trace/trace_functions_graph.c b/kernel/trace/trace_functions_graph.c index 66cce73e94f8..13d0387ac6a6 100644 --- a/kernel/trace/trace_functions_graph.c +++ b/kernel/trace/trace_functions_graph.c @@ -130,6 +130,7 @@ static inline int ftrace_graph_ignore_irqs(void) int trace_graph_entry(struct ftrace_graph_ent *trace, struct fgraph_ops *gops) { + unsigned long *task_var = fgraph_get_task_var(gops); struct trace_array *tr = gops->private; struct trace_array_cpu *data; unsigned long flags; @@ -138,7 +139,7 @@ int trace_graph_entry(struct ftrace_graph_ent *trace, int ret; int cpu; - if (trace_recursion_test(TRACE_GRAPH_NOTRACE_BIT)) + if (*task_var & TRACE_GRAPH_NOTRACE) return 0; /* @@ -149,7 +150,7 @@ int trace_graph_entry(struct ftrace_graph_ent *trace, * returning from the function. */ if (ftrace_graph_notrace_addr(trace->func)) { - trace_recursion_set(TRACE_GRAPH_NOTRACE_BIT); + *task_var |= TRACE_GRAPH_NOTRACE_BIT; /* * Need to return 1 to have the return called * that will clear the NOTRACE bit. @@ -240,6 +241,7 @@ void __trace_graph_return(struct trace_array *tr, void trace_graph_return(struct ftrace_graph_ret *trace, struct fgraph_ops *gops) { + unsigned long *task_var = fgraph_get_task_var(gops); struct trace_array *tr = gops->private; struct trace_array_cpu *data; unsigned long flags; @@ -249,8 +251,8 @@ void trace_graph_return(struct ftrace_graph_ret *trace, ftrace_graph_addr_finish(gops, trace); - if (trace_recursion_test(TRACE_GRAPH_NOTRACE_BIT)) { - trace_recursion_clear(TRACE_GRAPH_NOTRACE_BIT); + if (*task_var & TRACE_GRAPH_NOTRACE) { + *task_var &= ~TRACE_GRAPH_NOTRACE; return; } From patchwork Fri Dec 8 10:27:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 175747 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5363556vqy; Fri, 8 Dec 2023 02:29:08 -0800 (PST) X-Google-Smtp-Source: AGHT+IGiOuU3xoCJ7sW5L8vOyH96ZH13E51qsm6+Oi1jXLZ3WMHuRzAq3s7DhGtWd75zcDHFnP9x X-Received: by 2002:a17:902:d4ce:b0:1d0:53f2:9be with SMTP id o14-20020a170902d4ce00b001d053f209bemr3489507plg.46.1702031348221; Fri, 08 Dec 2023 02:29:08 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702031348; cv=none; d=google.com; s=arc-20160816; b=KYSF4iQUAqgPuHD6v6PzHsSUWuthwptauei2YM99vDAXnZDHmNZX1noBmMEOgiInro iE+WslGEqrV/ttn0+TfGVoqwT/FF4jDw9PURV9XpIRf1cO2PLk/HaWqpl7WKQp79ASmI prAXCdLNROfqM68B/6dbLD51nna0cS17SZgamBmg2CjvqJD4DrUHHNiCXMH0JHTvyg1h mFn8w5ADMhINJptKPYPO2hNdiWS75qLU8EahFHQswM0O+AuySfpRYWtqS0q7Yu3nM9bb 4Uwq+OLYgJ0bx3fR6ynwbDRqcBG8LH920v/C9xjsldiXWvZu1YVhxjB59Esw7cIpyTql D2MA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=NpAKra6drgA2/Aw7VEHlbUUgzBJkdBXl4xZRgEVhtDw=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=I/MOsgyIjL7DZXsKdgo9ayVaoM2YmbPOSggQaLDKBVl7U67P+/qsUhpunDWvLfyZLy GJmdcOQVuTMaF4E2auMgcgoweUNenk7+sKdefX3dWiZ5WTzEyANDoqTekH+5aqTxPkto 7E4QeVWJyKoc6xNHkpsBWD2a2BpPSFFAL4bmaohoQ+IpvfYb8beZ8a44fAZ60pyBU3zg 2F2vpP6ZkBtevt5ckcKej0tL1krTn4dGFYCE43WCogp1yt8b0pl6EklzsgLPUZXqSUIp MWdKuIbmb+IqQvyNZif0Zgq2EwGV2AbFUubclMs4hbmIS2dO8urwcULL8GEhz9dOqJUH S8rA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=pdk3PJVU; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id b6-20020a170902d88600b001d050ef0078si1350800plz.21.2023.12.08.02.29.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 02:29:08 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=pdk3PJVU; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 8A5C3807745F; Fri, 8 Dec 2023 02:29:04 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1573681AbjLHK2Z (ORCPT + 99 others); Fri, 8 Dec 2023 05:28:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60370 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1573703AbjLHK1x (ORCPT ); Fri, 8 Dec 2023 05:27:53 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2A24E1736 for ; Fri, 8 Dec 2023 02:27:48 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id ED61BC433C8; Fri, 8 Dec 2023 10:27:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702031267; bh=0rGICJqPtQykeqyrGjZ9554NSwsncuF9D5VP0MB90U0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pdk3PJVUJGRqlauIccOSmCOgzL90eAlKvkhQDb3WwdutOoShqBrA8s4szrgs9CK+4 MJZrL5UVuyIL6gkcAOwXxNJtdPZmXsMDmylS30S4dkEoWQqKQ/ZQ7E5fcRScxwt85D vR1PDKK32bGX+9jhU44exLqtl0vrbrdMvfjRhfqF97yfR00nRJQUhEiDTCfF5GTtU1 fUuNFHVDWovkbk7Ei0yisr5mfvdce3vRHnY1vF42uvLr2nRcfShbJgVP7exeAyS0Bd kd8X4Nx6gYMn+jhVxZiCYo5wD6cVMawL9Bwr2t5KDy8+RqbUDUjI87KPeC/WGMWoYZ 5JFinG3l6V7Fg== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v4 17/33] function_graph: Implement fgraph_reserve_data() and fgraph_retrieve_data() Date: Fri, 8 Dec 2023 19:27:41 +0900 Message-Id: <170203126100.579004.4253156536060507449.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170203105427.579004.8033550792660734570.stgit@devnote2> References: <170203105427.579004.8033550792660734570.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Fri, 08 Dec 2023 02:29:05 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784709222957103393 X-GMAIL-MSGID: 1784709222957103393 From: Steven Rostedt (VMware) Added functions that can be called by a fgraph_ops entryfunc and retfunc to store state between the entry of the function being traced to the exit of the same function. The fgraph_ops entryfunc() may call fgraph_reserve_data() to store up to 32 words onto the task's shadow ret_stack and this then can be retrieved by fgraph_retrieve_data() called by the corresponding retfunc(). Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- Changes in v3: - Store fgraph_array index to the data entry. - Both function requires fgraph_array index to store/retrieve data. - Reserve correct size of the data. - Return correct data area. Changes in v2: - Retrieve the reserved size by fgraph_retrieve_data(). - Expand the maximum data size to 32 words. - Update stack index with __get_index(val) if FGRAPH_TYPE_ARRAY entry. - fix typos and make description lines shorter than 76 chars. --- include/linux/ftrace.h | 3 + kernel/trace/fgraph.c | 175 ++++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 170 insertions(+), 8 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 09ca4bba63f2..1c121237016e 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -1075,6 +1075,9 @@ struct fgraph_ops { int idx; }; +void *fgraph_reserve_data(int idx, int size_bytes); +void *fgraph_retrieve_data(int idx, int *size_bytes); + /* * Stack of return addresses for functions * of a thread. diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index 5ac6477bcf20..d4d7007bf619 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -41,17 +41,29 @@ * bits: 10 - 11 Type of storage * 0 - reserved * 1 - bitmap of fgraph_array index + * 2 - reserved data * * For bitmap of fgraph_array index * bits: 12 - 27 The bitmap of fgraph_ops fgraph_array index * + * For reserved data: + * bits: 12 - 17 The size in words that is stored + * bits: 18 - 23 The index of fgraph_array, which shows who is stored + * * That is, at the end of function_graph_enter, if the first and forth * fgraph_ops on the fgraph_array[] (index 0 and 3) needs their retfunc called - * on the return of the function being traced, this is what will be on the - * task's shadow ret_stack: (the stack grows upward) + * on the return of the function being traced, and the forth fgraph_ops + * stored two words of data, this is what will be on the task's shadow + * ret_stack: (the stack grows upward) * * | | <- task->curr_ret_stack * +--------------------------------------------+ + * | data_type(idx:3, size:2, | + * | offset:FGRAPH_RET_INDEX+3) | ( Data with size of 2 words) + * +--------------------------------------------+ ( It is 4 words from the ret_stack) + * | STORED DATA WORD 2 | + * | STORED DATA WORD 1 | + * +------i-------------------------------------+ * | bitmap_type(bitmap:(BIT(3)|BIT(0)), | * | offset:FGRAPH_RET_INDEX) | <- the offset is from here * +--------------------------------------------+ @@ -78,14 +90,23 @@ enum { FGRAPH_TYPE_RESERVED = 0, FGRAPH_TYPE_BITMAP = 1, + FGRAPH_TYPE_DATA = 2, }; #define FGRAPH_INDEX_SIZE 16 #define FGRAPH_INDEX_MASK GENMASK(FGRAPH_INDEX_SIZE - 1, 0) #define FGRAPH_INDEX_SHIFT (FGRAPH_TYPE_SHIFT + FGRAPH_TYPE_SIZE) -/* Currently the max stack index can't be more than register callers */ -#define FGRAPH_MAX_INDEX (FGRAPH_INDEX_SIZE + FGRAPH_RET_INDEX) +#define FGRAPH_DATA_SIZE 5 +#define FGRAPH_DATA_MASK ((1 << FGRAPH_DATA_SIZE) - 1) +#define FGRAPH_DATA_SHIFT (FGRAPH_TYPE_SHIFT + FGRAPH_TYPE_SIZE) + +#define FGRAPH_DATA_INDEX_SIZE 4 +#define FGRAPH_DATA_INDEX_MASK ((1 << FGRAPH_DATA_INDEX_SIZE) - 1) +#define FGRAPH_DATA_INDEX_SHIFT (FGRAPH_DATA_SHIFT + FGRAPH_DATA_SIZE) + +#define FGRAPH_MAX_INDEX \ + ((FGRAPH_INDEX_SIZE << FGRAPH_DATA_SIZE) + FGRAPH_RET_INDEX) #define FGRAPH_ARRAY_SIZE FGRAPH_INDEX_SIZE @@ -97,6 +118,8 @@ enum { #define RET_STACK(t, index) ((struct ftrace_ret_stack *)(&(t)->ret_stack[index])) +#define FGRAPH_MAX_DATA_SIZE (sizeof(long) * (1 << FGRAPH_DATA_SIZE)) + /* * Each fgraph_ops has a reservered unsigned long at the end (top) of the * ret_stack to store task specific state. @@ -145,14 +168,39 @@ static int fgraph_lru_alloc_index(void) return idx; } +static inline int __get_index(unsigned long val) +{ + return val & FGRAPH_RET_INDEX_MASK; +} + +static inline int __get_type(unsigned long val) +{ + return (val >> FGRAPH_TYPE_SHIFT) & FGRAPH_TYPE_MASK; +} + +static inline int __get_data_index(unsigned long val) +{ + return (val >> FGRAPH_DATA_INDEX_SHIFT) & FGRAPH_DATA_INDEX_MASK; +} + +static inline int __get_data_size(unsigned long val) +{ + return (val >> FGRAPH_DATA_SHIFT) & FGRAPH_DATA_MASK; +} + +static inline unsigned long get_fgraph_entry(struct task_struct *t, int index) +{ + return t->ret_stack[index]; +} + static inline int get_ret_stack_index(struct task_struct *t, int offset) { - return t->ret_stack[offset] & FGRAPH_RET_INDEX_MASK; + return __get_index(t->ret_stack[offset]); } static inline int get_fgraph_type(struct task_struct *t, int offset) { - return (t->ret_stack[offset] >> FGRAPH_TYPE_SHIFT) & FGRAPH_TYPE_MASK; + return __get_type(t->ret_stack[offset]); } static inline unsigned long @@ -179,6 +227,22 @@ add_fgraph_index_bitmap(struct task_struct *t, int offset, unsigned long bitmap) t->ret_stack[offset] |= (bitmap << FGRAPH_INDEX_SHIFT); } +static inline void *get_fgraph_data(struct task_struct *t, int index) +{ + unsigned long val = t->ret_stack[index]; + + if (__get_type(val) != FGRAPH_TYPE_DATA) + return NULL; + index -= __get_data_size(val); + return (void *)&t->ret_stack[index]; +} + +static inline unsigned long make_fgraph_data(int idx, int size, int offset) +{ + return (idx << FGRAPH_DATA_INDEX_SHIFT) | (size << FGRAPH_DATA_SHIFT) | + (FGRAPH_TYPE_DATA << FGRAPH_TYPE_SHIFT) | offset; +} + /* ftrace_graph_entry set to this to tell some archs to run function graph */ static int entry_run(struct ftrace_graph_ent *trace, struct fgraph_ops *ops) { @@ -212,6 +276,92 @@ static void ret_stack_init_task_vars(unsigned long *ret_stack) memset(gvals, 0, sizeof(*gvals) * FGRAPH_ARRAY_SIZE); } +/** + * fgraph_reserve_data - Reserve storage on the task's ret_stack + * @idx: The index of fgraph_array + * @size_bytes: The size in bytes to reserve + * + * Reserves space of up to FGRAPH_MAX_DATA_SIZE bytes on the + * task's ret_stack shadow stack, for a given fgraph_ops during + * the entryfunc() call. If entryfunc() returns zero, the storage + * is discarded. An entryfunc() can only call this once per iteration. + * The fgraph_ops retfunc() can retrieve this stored data with + * fgraph_retrieve_data(). + * + * Returns: On success, a pointer to the data on the stack. + * Otherwise, NULL if there's not enough space left on the + * ret_stack for the data, or if fgraph_reserve_data() was called + * more than once for a single entryfunc() call. + */ +void *fgraph_reserve_data(int idx, int size_bytes) +{ + unsigned long val; + void *data; + int curr_ret_stack = current->curr_ret_stack; + int data_size; + + if (size_bytes > FGRAPH_MAX_DATA_SIZE) + return NULL; + + /* Convert to number of longs + data word */ + data_size = DIV_ROUND_UP(size_bytes, sizeof(long)); + + val = get_fgraph_entry(current, curr_ret_stack - 1); + data = ¤t->ret_stack[curr_ret_stack]; + + curr_ret_stack += data_size + 1; + if (unlikely(curr_ret_stack >= SHADOW_STACK_MAX_INDEX)) + return NULL; + + val = make_fgraph_data(idx, data_size, __get_index(val) + data_size + 1); + + /* Set the last word to be reserved */ + current->ret_stack[curr_ret_stack - 1] = val; + + /* Make sure interrupts see this */ + barrier(); + current->curr_ret_stack = curr_ret_stack; + /* Again sync with interrupts, and reset reserve */ + current->ret_stack[curr_ret_stack - 1] = val; + + return data; +} + +/** + * fgraph_retrieve_data - Retrieve stored data from fgraph_reserve_data() + * @idx: the index of fgraph_array (fgraph_ops::idx) + * @size_bytes: pointer to retrieved data size. + * + * This is to be called by a fgraph_ops retfunc(), to retrieve data that + * was stored by the fgraph_ops entryfunc() on the function entry. + * That is, this will retrieve the data that was reserved on the + * entry of the function that corresponds to the exit of the function + * that the fgraph_ops retfunc() is called on. + * + * Returns: The stored data from fgraph_reserve_data() called by the + * matching entryfunc() for the retfunc() this is called from. + * Or NULL if there was nothing stored. + */ +void *fgraph_retrieve_data(int idx, int *size_bytes) +{ + int index = current->curr_ret_stack - 1; + unsigned long val; + + val = get_fgraph_entry(current, index); + while (__get_type(val) == FGRAPH_TYPE_DATA) { + if (__get_data_index(val) == idx) + goto found; + index -= __get_data_size(val) + 1; + val = get_fgraph_entry(current, index); + } + return NULL; +found: + if (size_bytes) + *size_bytes = __get_data_size(val) * + sizeof(long); + return get_fgraph_data(current, index); +} + /** * fgraph_get_task_var - retrieve a task specific state variable * @gops: The ftrace_ops that owns the task specific variable @@ -449,13 +599,18 @@ int function_graph_enter(unsigned long ret, unsigned long func, for (i = 0; i < FGRAPH_ARRAY_SIZE; i++) { struct fgraph_ops *gops = fgraph_array[i]; + int save_curr_ret_stack; if (gops == &fgraph_stub) continue; + save_curr_ret_stack = current->curr_ret_stack; if (ftrace_ops_test(&gops->ops, func, NULL) && gops->entryfunc(&trace, gops)) bitmap |= BIT(i); + else + /* Clear out any saved storage */ + current->curr_ret_stack = save_curr_ret_stack; } if (!bitmap) @@ -481,6 +636,7 @@ int function_graph_enter_ops(unsigned long ret, unsigned long func, struct fgraph_ops *gops) { struct ftrace_graph_ent trace; + int save_curr_ret_stack; int index; int type; @@ -497,13 +653,15 @@ int function_graph_enter_ops(unsigned long ret, unsigned long func, trace.func = func; trace.depth = current->curr_ret_depth; + save_curr_ret_stack = current->curr_ret_stack; if (gops->entryfunc(&trace, gops)) { if (type == FGRAPH_TYPE_RESERVED) set_fgraph_index_bitmap(current, index, BIT(gops->idx)); else add_fgraph_index_bitmap(current, index, BIT(gops->idx)); return 0; - } + } else + current->curr_ret_stack = save_curr_ret_stack; if (type == FGRAPH_TYPE_RESERVED) { current->curr_ret_stack -= FGRAPH_RET_INDEX + 1; @@ -648,7 +806,8 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs * curr_ret_stack is after that. */ barrier(); - current->curr_ret_stack -= FGRAPH_RET_INDEX + 1; + current->curr_ret_stack = index - FGRAPH_RET_INDEX; + current->curr_ret_depth--; return ret; } From patchwork Fri Dec 8 10:27:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 175746 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5363483vqy; Fri, 8 Dec 2023 02:28:53 -0800 (PST) X-Google-Smtp-Source: AGHT+IHvHgKnaWsxyd7nseuBvcgCx/R6eRzAfFpBbE5HNd1pep9JvioHYb2hD7sxN5rQpxhBULNj X-Received: by 2002:a17:903:1251:b0:1d0:6ffd:e2d4 with SMTP id u17-20020a170903125100b001d06ffde2d4mr4520402plh.110.1702031333406; Fri, 08 Dec 2023 02:28:53 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702031333; cv=none; d=google.com; s=arc-20160816; b=lsn7GNLlSMccbGzBJAsFOGbFZG/IuKr6ZXmKQkMG33mAK2EU0dtSNp3GkVZvg7FYiW mlJ601BNDf1Q3tawnMl451qX/F+apxhUDkSmAAn2Uo6cI4vTe0YZSXNKu0m03Tk/hlCm NYYNV355RzNTR7475Zi2lTkUbtf3WC4uh5W3YkTVob9fZgorla66u0qnMbumhZ6gfKix KAWvf5an8od/5JjbybBPThuzNGja4/ghwmO2vCax6+2AycGZ15zleODGb+LaFevZ4nZH OM5WSwZenysLSK9Myry9HxhPNAryoKfgRMdLabLrD/UjSLSjvDGOro0ETCQYE4LeVokE xBGA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=gNjysZumaRGj7wRl3wCUGBAp98rdVBpztEtFKkRLpNA=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=KM6LBQLxGY010HFKz6jGrhNnve56C0Io13AkiCNoTTsQIpQt82N8sbLY+xqqIKyILj mPXOpuTHAXF+4tpjw1pUOwsOe+8vENg40AEW1psI86gE26rqvHfdeSk5EOBJoBo9N5ij oihJwm7UVMSuuYzlBXzivcrvCCKxhUFcJ/eesSH+BiKA66Mh0ayEz6PQEhJgPTkhTxdH iXByvMDwzXXkjXoWgwC8TyBnwp7c8UgG2q2AMfNmrBSo1jYuwvWbOTBY8o+4PquLax2T r/y99cO6k0ViWI49dazcED3K7MwljI0H0njq1D7rI70X2TJPnJ5ZYcjzeuHnmY3g2G3A 0Akg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=b0nZarMs; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from fry.vger.email (fry.vger.email. [23.128.96.38]) by mx.google.com with ESMTPS id a15-20020a170902eccf00b001cfbf025320si1399704plh.346.2023.12.08.02.28.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 02:28:53 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) client-ip=23.128.96.38; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=b0nZarMs; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id A21908051CBE; Fri, 8 Dec 2023 02:28:50 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1573699AbjLHK2l (ORCPT + 99 others); Fri, 8 Dec 2023 05:28:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49312 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235888AbjLHK2E (ORCPT ); Fri, 8 Dec 2023 05:28:04 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9EE731997 for ; Fri, 8 Dec 2023 02:28:00 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 25DE3C433C8; Fri, 8 Dec 2023 10:27:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702031279; bh=loWWqL6DZVWbVMu4FDyOcIsVywT7pv1to3y8Tk+anuc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=b0nZarMsOxKkcIF4K9IU3QDpAJ2NmzHx+uaCqLm2QbyIJjwuIlf6C8K8IKVcqsOib bmwP1rWcrkR1UboplUT+gli6YtlrvFAuTgC0gM+hM5z6xGvCsIaaxXFc8NphhuiAYL CIhzdvwoTp4brE5QQMTFtC2fWSB95urrh+QJPvWsm6SbE+7MqBdRAAIFbyB4OWWnD8 PVrD0xfoqTfRnlpHuMFHHvromup4zDbKHGaR9q0cflbbEuaCQGkqhhI23enF51VGRZ As0PJI0A8itih++lKAei5Ed8KbV7ECCUpD8jM6tX7reisO3Y0bSZ1JfD5GqvcKCv14 caIvCAoOYrSWw== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v4 18/33] function_graph: Add selftest for passing local variables Date: Fri, 8 Dec 2023 19:27:53 +0900 Message-Id: <170203127319.579004.15764067848691660601.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170203105427.579004.8033550792660734570.stgit@devnote2> References: <170203105427.579004.8033550792660734570.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Fri, 08 Dec 2023 02:28:50 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784709207684285440 X-GMAIL-MSGID: 1784709207684285440 From: Steven Rostedt (VMware) Add boot up selftest that passes variables from a function entry to a function exit, and make sure that they do get passed around. Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Masami Hiramatsu (Google) --- Changes in v2: - Add reserved size test. - Use pr_*() instead of printk(KERN_*). --- kernel/trace/trace_selftest.c | 169 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 169 insertions(+) diff --git a/kernel/trace/trace_selftest.c b/kernel/trace/trace_selftest.c index f0758afa2f7d..4d86cd4c8c8c 100644 --- a/kernel/trace/trace_selftest.c +++ b/kernel/trace/trace_selftest.c @@ -756,6 +756,173 @@ trace_selftest_startup_function(struct tracer *trace, struct trace_array *tr) #ifdef CONFIG_FUNCTION_GRAPH_TRACER +#ifdef CONFIG_DYNAMIC_FTRACE + +#define BYTE_NUMBER 123 +#define SHORT_NUMBER 12345 +#define WORD_NUMBER 1234567890 +#define LONG_NUMBER 1234567890123456789LL + +static int fgraph_store_size __initdata; +static const char *fgraph_store_type_name __initdata; +static char *fgraph_error_str __initdata; +static char fgraph_error_str_buf[128] __initdata; + +static __init int store_entry(struct ftrace_graph_ent *trace, + struct fgraph_ops *gops) +{ + const char *type = fgraph_store_type_name; + int size = fgraph_store_size; + void *p; + + p = fgraph_reserve_data(gops->idx, size); + if (!p) { + snprintf(fgraph_error_str_buf, sizeof(fgraph_error_str_buf), + "Failed to reserve %s\n", type); + fgraph_error_str = fgraph_error_str_buf; + return 0; + } + + switch (fgraph_store_size) { + case 1: + *(char *)p = BYTE_NUMBER; + break; + case 2: + *(short *)p = SHORT_NUMBER; + break; + case 4: + *(int *)p = WORD_NUMBER; + break; + case 8: + *(long long *)p = LONG_NUMBER; + break; + } + + return 1; +} + +static __init void store_return(struct ftrace_graph_ret *trace, + struct fgraph_ops *gops) +{ + const char *type = fgraph_store_type_name; + long long expect = 0; + long long found = -1; + int size; + char *p; + + p = fgraph_retrieve_data(gops->idx, &size); + if (!p) { + snprintf(fgraph_error_str_buf, sizeof(fgraph_error_str_buf), + "Failed to retrieve %s\n", type); + fgraph_error_str = fgraph_error_str_buf; + return; + } + if (fgraph_store_size > size) { + snprintf(fgraph_error_str_buf, sizeof(fgraph_error_str_buf), + "Retrieved size %d is smaller than expected %d\n", + size, (int)fgraph_store_size); + fgraph_error_str = fgraph_error_str_buf; + return; + } + + switch (fgraph_store_size) { + case 1: + expect = BYTE_NUMBER; + found = *(char *)p; + break; + case 2: + expect = SHORT_NUMBER; + found = *(short *)p; + break; + case 4: + expect = WORD_NUMBER; + found = *(int *)p; + break; + case 8: + expect = LONG_NUMBER; + found = *(long long *)p; + break; + } + + if (found != expect) { + snprintf(fgraph_error_str_buf, sizeof(fgraph_error_str_buf), + "%s returned not %lld but %lld\n", type, expect, found); + fgraph_error_str = fgraph_error_str_buf; + return; + } + fgraph_error_str = NULL; +} + +static struct fgraph_ops store_bytes __initdata = { + .entryfunc = store_entry, + .retfunc = store_return, +}; + +static int __init test_graph_storage_type(const char *name, int size) +{ + char *func_name; + int len; + int ret; + + fgraph_store_type_name = name; + fgraph_store_size = size; + + snprintf(fgraph_error_str_buf, sizeof(fgraph_error_str_buf), + "Failed to execute storage %s\n", name); + fgraph_error_str = fgraph_error_str_buf; + + pr_cont("PASSED\n"); + pr_info("Testing fgraph storage of %d byte%s: ", size, size > 1 ? "s" : ""); + + func_name = "*" __stringify(DYN_FTRACE_TEST_NAME); + len = strlen(func_name); + + ret = ftrace_set_filter(&store_bytes.ops, func_name, len, 1); + if (ret && ret != -ENODEV) { + pr_cont("*Could not set filter* "); + return -1; + } + + ret = register_ftrace_graph(&store_bytes); + if (ret) { + pr_warn("Failed to init store_bytes fgraph tracing\n"); + return -1; + } + + DYN_FTRACE_TEST_NAME(); + + unregister_ftrace_graph(&store_bytes); + + if (fgraph_error_str) { + pr_cont("*** %s ***", fgraph_error_str); + return -1; + } + + return 0; +} +/* Test the storage passed across function_graph entry and return */ +static __init int test_graph_storage(void) +{ + int ret; + + ret = test_graph_storage_type("byte", 1); + if (ret) + return ret; + ret = test_graph_storage_type("short", 2); + if (ret) + return ret; + ret = test_graph_storage_type("word", 4); + if (ret) + return ret; + ret = test_graph_storage_type("long long", 8); + if (ret) + return ret; + return 0; +} +#else +static inline int test_graph_storage(void) { return 0; } +#endif /* CONFIG_DYNAMIC_FTRACE */ + /* Maximum number of functions to trace before diagnosing a hang */ #define GRAPH_MAX_FUNC_TEST 100000000 @@ -913,6 +1080,8 @@ trace_selftest_startup_function_graph(struct tracer *trace, ftrace_set_global_filter(NULL, 0, 1); #endif + ret = test_graph_storage(); + /* Don't test dynamic tracing, the function tracer already did */ out: /* Stop it if we failed */ From patchwork Fri Dec 8 10:28:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 175748 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5363624vqy; Fri, 8 Dec 2023 02:29:19 -0800 (PST) X-Google-Smtp-Source: AGHT+IFoNqYJC5lfFTAJY7RzkE0sfJm5ZJ+KWsP89VUkXm5qiVS1Wrw6Fo4zWqCygcyaUcMtQndy X-Received: by 2002:a92:b0f:0:b0:35d:5543:45dd with SMTP id b15-20020a920b0f000000b0035d554345ddmr4184148ilf.22.1702031358944; Fri, 08 Dec 2023 02:29:18 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702031358; cv=none; d=google.com; s=arc-20160816; b=wbjSTEpjbz4dOTd1DuoCEK0WXOtiUQHwjt/l6/SN9QuEXfnB7KruJtgywY4P9rnuUX 1N3hZueuAzeOZVrfQ8cc2Cy4EQxbn8aWd6Rtc8MfWEhnemC332QnGhfGPyqgsk2hg6t/ RYQFeXI8ya1ftkE1kS0RMcIyhaJeumprh1+X0dtP2N+OfofTghKATIr3gWrPEC2LBaLX Yq27iL/BpV/2xFUVPkYhMj6Y0y73wKQBd/hU8OoMHEpQQnTCc8IJJsIflYadPQMkL+Ia o4f6et5gEaq3ta1Hz+UK3VvX/BEhv2TnXGk/amTIYhnhIvGDqHdQKlh82WeL+poUCioM 0XKQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=43t4kxj71380c+ObQIJLqiQozpCfPd8/9uRxbRkIaTw=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=JNfmsKeXWNhsigCgLM7rGGfkUfd/eKuXkVhK2juz8OmKYQMGH3hi2tA19DlRVtAquX 008/kA6A/P1jbwwtMNpwI3BZbJKMg01eaC6BdJcj1hcFsmvvNUZkhfXVd8gGi+1TRiHB LocWYdull5Mnoc9RCFh66h8j1dax6egiebHDLOpqtbmFqDRQw3LHXk8RNmF1pPcvJmhy l2N28j3g4X5CeSGFZrHDM7hRFUpQ/tduQ3DNtwMTJMGF7kg81phFLTGK1ctPyHsF0H7e 1uUvdgjJtIQCdVWYqXQKUeL3M/lmbOBu3YaiGk/zK4svN61olaNladtyIfvTZPQ8HLkC qJPg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=nlyneYz2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from pete.vger.email (pete.vger.email. [2620:137:e000::3:6]) by mx.google.com with ESMTPS id b15-20020a65668f000000b005a9debd7854si1287350pgw.828.2023.12.08.02.29.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 02:29:18 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) client-ip=2620:137:e000::3:6; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=nlyneYz2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id CEB9F82C400F; Fri, 8 Dec 2023 02:29:13 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235876AbjLHK3D (ORCPT + 99 others); Fri, 8 Dec 2023 05:29:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60782 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1573685AbjLHK20 (ORCPT ); Fri, 8 Dec 2023 05:28:26 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4CF5019AE for ; Fri, 8 Dec 2023 02:28:12 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 80077C433C8; Fri, 8 Dec 2023 10:28:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702031292; bh=aYAYKkfhGpkgr2q3l9uDTnHCGnI+MN7bnV2DsJUzAAQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nlyneYz21A5Rah2UeHwTGsVY8iQCHY5KmPAHNoV/g2Oj/yzMq8T+j9W3JkjjI+mT6 osI2hrefqYvxa0dxWp1qgIzeTtqJx/mqVZY3f+cBmH8djjd53cvlBe+pJCYKIVdXnp UoIzDazfOJ5pnmwbJsVSetZTIPBUyuxxdSq5ep/uNFGxRCroquspLbiygGalaiQ50O DQCgd3VfGlcn3QqKppjEk6jzIpl7Yym24QoaTd/7xgHpeC0zwmzA8mDgT0iyWWBlEM D/eRNezZ98Ki0bV73WhAHuPQQrof2sHEaFEW2DjCoGh9KH40nxNYpSU/dCZXHPyYnR S1/YtIwL7KIxw== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v4 19/33] function_graph: Add a new entry handler with parent_ip and ftrace_regs Date: Fri, 8 Dec 2023 19:28:05 +0900 Message-Id: <170203128527.579004.1459428944015956463.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170203105427.579004.8033550792660734570.stgit@devnote2> References: <170203105427.579004.8033550792660734570.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Fri, 08 Dec 2023 02:29:14 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784709234055255177 X-GMAIL-MSGID: 1784709234055255177 From: Masami Hiramatsu (Google) Add a new entry handler to fgraph_ops as 'entryregfunc' which takes parent_ip and ftrace_regs. Note that the 'entryfunc' and 'entryregfunc' are mutual exclusive. You can set only one of them. Signed-off-by: Masami Hiramatsu (Google) --- Changes in v3: - Update for new multiple fgraph. --- arch/arm64/kernel/ftrace.c | 2 + arch/loongarch/kernel/ftrace_dyn.c | 6 ++++ arch/powerpc/kernel/trace/ftrace.c | 2 + arch/powerpc/kernel/trace/ftrace_64_pg.c | 10 ++++--- arch/x86/kernel/ftrace.c | 42 ++++++++++++++++-------------- include/linux/ftrace.h | 19 +++++++++++--- kernel/trace/fgraph.c | 30 +++++++++++++++++---- 7 files changed, 76 insertions(+), 35 deletions(-) diff --git a/arch/arm64/kernel/ftrace.c b/arch/arm64/kernel/ftrace.c index 205937e04ece..9d71a475d4c5 100644 --- a/arch/arm64/kernel/ftrace.c +++ b/arch/arm64/kernel/ftrace.c @@ -495,7 +495,7 @@ void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, if (bit < 0) return; - if (!function_graph_enter_ops(*parent, ip, fregs->fp, parent, gops)) + if (!function_graph_enter_ops(*parent, ip, fregs->fp, parent, fregs, gops)) *parent = (unsigned long)&return_to_handler; ftrace_test_recursion_unlock(bit); diff --git a/arch/loongarch/kernel/ftrace_dyn.c b/arch/loongarch/kernel/ftrace_dyn.c index 73858c9029cc..39b3f09a5e0c 100644 --- a/arch/loongarch/kernel/ftrace_dyn.c +++ b/arch/loongarch/kernel/ftrace_dyn.c @@ -244,7 +244,11 @@ void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, struct pt_regs *regs = &fregs->regs; unsigned long *parent = (unsigned long *)®s->regs[1]; - prepare_ftrace_return(ip, (unsigned long *)parent); + if (unlikely(atomic_read(¤t->tracing_graph_pause))) + return; + + if (!function_graph_enter_regs(regs->regs[1], ip, 0, parent, fregs)) + regs->regs[1] = (unsigned long)&return_to_handler; } #else static int ftrace_modify_graph_caller(bool enable) diff --git a/arch/powerpc/kernel/trace/ftrace.c b/arch/powerpc/kernel/trace/ftrace.c index 82010629cf88..9bf1b6912116 100644 --- a/arch/powerpc/kernel/trace/ftrace.c +++ b/arch/powerpc/kernel/trace/ftrace.c @@ -422,7 +422,7 @@ void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, if (bit < 0) goto out; - if (!function_graph_enter(parent_ip, ip, 0, (unsigned long *)sp)) + if (!function_graph_enter_regs(parent_ip, ip, 0, (unsigned long *)sp, fregs)) parent_ip = ppc_function_entry(return_to_handler); ftrace_test_recursion_unlock(bit); diff --git a/arch/powerpc/kernel/trace/ftrace_64_pg.c b/arch/powerpc/kernel/trace/ftrace_64_pg.c index 7b85c3b460a3..43f6cfaaf7db 100644 --- a/arch/powerpc/kernel/trace/ftrace_64_pg.c +++ b/arch/powerpc/kernel/trace/ftrace_64_pg.c @@ -795,7 +795,8 @@ int ftrace_disable_ftrace_graph_caller(void) * in current thread info. Return the address we want to divert to. */ static unsigned long -__prepare_ftrace_return(unsigned long parent, unsigned long ip, unsigned long sp) +__prepare_ftrace_return(unsigned long parent, unsigned long ip, unsigned long sp, + struct ftrace_regs *fregs) { unsigned long return_hooker; int bit; @@ -812,7 +813,7 @@ __prepare_ftrace_return(unsigned long parent, unsigned long ip, unsigned long sp return_hooker = ppc_function_entry(return_to_handler); - if (!function_graph_enter(parent, ip, 0, (unsigned long *)sp)) + if (!function_graph_enter_regs(parent, ip, 0, (unsigned long *)sp, fregs)) parent = return_hooker; ftrace_test_recursion_unlock(bit); @@ -824,13 +825,14 @@ __prepare_ftrace_return(unsigned long parent, unsigned long ip, unsigned long sp void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, struct ftrace_ops *op, struct ftrace_regs *fregs) { - fregs->regs.link = __prepare_ftrace_return(parent_ip, ip, fregs->regs.gpr[1]); + fregs->regs.link = __prepare_ftrace_return(parent_ip, ip, + fregs->regs.gpr[1], fregs); } #else unsigned long prepare_ftrace_return(unsigned long parent, unsigned long ip, unsigned long sp) { - return __prepare_ftrace_return(parent, ip, sp); + return __prepare_ftrace_return(parent, ip, sp, NULL); } #endif #endif /* CONFIG_FUNCTION_GRAPH_TRACER */ diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c index 845e29b4254f..0f757e399a96 100644 --- a/arch/x86/kernel/ftrace.c +++ b/arch/x86/kernel/ftrace.c @@ -614,16 +614,8 @@ int ftrace_disable_ftrace_graph_caller(void) } #endif /* CONFIG_DYNAMIC_FTRACE && !CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS */ -/* - * Hook the return address and push it in the stack of return addrs - * in current thread info. - */ -void prepare_ftrace_return(unsigned long ip, unsigned long *parent, - unsigned long frame_pointer) +static inline bool skip_ftrace_return(void) { - unsigned long return_hooker = (unsigned long)&return_to_handler; - int bit; - /* * When resuming from suspend-to-ram, this function can be indirectly * called from early CPU startup code while the CPU is in real mode, @@ -633,13 +625,28 @@ void prepare_ftrace_return(unsigned long ip, unsigned long *parent, * This check isn't as accurate as virt_addr_valid(), but it should be * good enough for this purpose, and it's fast. */ - if (unlikely((long)__builtin_frame_address(0) >= 0)) - return; + if ((long)__builtin_frame_address(0) >= 0) + return true; - if (unlikely(ftrace_graph_is_dead())) - return; + if (ftrace_graph_is_dead()) + return true; + + if (atomic_read(¤t->tracing_graph_pause)) + return true; + return false; +} - if (unlikely(atomic_read(¤t->tracing_graph_pause))) +/* + * Hook the return address and push it in the stack of return addrs + * in current thread info. + */ +void prepare_ftrace_return(unsigned long ip, unsigned long *parent, + unsigned long frame_pointer) +{ + unsigned long return_hooker = (unsigned long)&return_to_handler; + int bit; + + if (unlikely(skip_ftrace_return())) return; bit = ftrace_test_recursion_trylock(ip, *parent); @@ -661,17 +668,14 @@ void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, struct fgraph_ops *gops = container_of(op, struct fgraph_ops, ops); int bit; - if (unlikely(ftrace_graph_is_dead())) - return; - - if (unlikely(atomic_read(¤t->tracing_graph_pause))) + if (unlikely(skip_ftrace_return())) return; bit = ftrace_test_recursion_trylock(ip, *parent); if (bit < 0) return; - if (!function_graph_enter_ops(*parent, ip, 0, parent, gops)) + if (!function_graph_enter_ops(*parent, ip, 0, parent, fregs, gops)) *parent = (unsigned long)&return_to_handler; ftrace_test_recursion_unlock(bit); diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 1c121237016e..6da6cc9aaeaf 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -1063,6 +1063,11 @@ typedef void (*trace_func_graph_ret_t)(struct ftrace_graph_ret *, typedef int (*trace_func_graph_ent_t)(struct ftrace_graph_ent *, struct fgraph_ops *); /* entry */ +typedef int (*trace_func_graph_regs_ent_t)(unsigned long func, + unsigned long parent_ip, + struct ftrace_regs *fregs, + struct fgraph_ops *); /* entry w/ regs */ + extern int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace, struct fgraph_ops *gops); #ifdef CONFIG_FUNCTION_GRAPH_TRACER @@ -1070,6 +1075,7 @@ extern int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace, struct fgraph struct fgraph_ops { trace_func_graph_ent_t entryfunc; trace_func_graph_ret_t retfunc; + trace_func_graph_regs_ent_t entryregfunc; struct ftrace_ops ops; /* for the hash lists */ void *private; int idx; @@ -1106,13 +1112,20 @@ struct ftrace_ret_stack { extern void return_to_handler(void); extern int -function_graph_enter(unsigned long ret, unsigned long func, - unsigned long frame_pointer, unsigned long *retp); +function_graph_enter_regs(unsigned long ret, unsigned long func, + unsigned long frame_pointer, unsigned long *retp, + struct ftrace_regs *fregs); + +static inline int function_graph_enter(unsigned long ret, unsigned long func, + unsigned long fp, unsigned long *retp) +{ + return function_graph_enter_regs(ret, func, fp, retp, NULL); +} extern int function_graph_enter_ops(unsigned long ret, unsigned long func, unsigned long frame_pointer, unsigned long *retp, - struct fgraph_ops *gops); + struct ftrace_regs *fregs, struct fgraph_ops *gops); struct ftrace_ret_stack * ftrace_graph_get_ret_stack(struct task_struct *task, int idx); diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index d4d7007bf619..95b3eb4e8e23 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -569,9 +569,21 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func, # define MCOUNT_INSN_SIZE 0 #endif +static inline int call_entry_func(struct ftrace_graph_ent *trace, + unsigned long func, unsigned long ret, + struct ftrace_regs *fregs, + struct fgraph_ops *gops) +{ + if (gops->entryregfunc) + return gops->entryregfunc(func, ret, fregs, gops); + + return gops->entryfunc(trace, gops); +} + /* If the caller does not use ftrace, call this function. */ -int function_graph_enter(unsigned long ret, unsigned long func, - unsigned long frame_pointer, unsigned long *retp) +int function_graph_enter_regs(unsigned long ret, unsigned long func, + unsigned long frame_pointer, unsigned long *retp, + struct ftrace_regs *fregs) { struct ftrace_graph_ent trace; unsigned long bitmap = 0; @@ -606,7 +618,7 @@ int function_graph_enter(unsigned long ret, unsigned long func, save_curr_ret_stack = current->curr_ret_stack; if (ftrace_ops_test(&gops->ops, func, NULL) && - gops->entryfunc(&trace, gops)) + call_entry_func(&trace, func, ret, fregs, gops)) bitmap |= BIT(i); else /* Clear out any saved storage */ @@ -633,6 +645,7 @@ int function_graph_enter(unsigned long ret, unsigned long func, /* This is called from ftrace_graph_func() via ftrace */ int function_graph_enter_ops(unsigned long ret, unsigned long func, unsigned long frame_pointer, unsigned long *retp, + struct ftrace_regs *fregs, struct fgraph_ops *gops) { struct ftrace_graph_ent trace; @@ -654,7 +667,7 @@ int function_graph_enter_ops(unsigned long ret, unsigned long func, trace.func = func; trace.depth = current->curr_ret_depth; save_curr_ret_stack = current->curr_ret_stack; - if (gops->entryfunc(&trace, gops)) { + if (call_entry_func(&trace, func, ret, fregs, gops)) { if (type == FGRAPH_TYPE_RESERVED) set_fgraph_index_bitmap(current, index, BIT(gops->idx)); else @@ -935,7 +948,8 @@ void fgraph_init_ops(struct ftrace_ops *dst_ops, struct ftrace_ops *src_ops) { dst_ops->func = ftrace_graph_func; - dst_ops->flags = FTRACE_OPS_FL_PID | FTRACE_OPS_GRAPH_STUB; + dst_ops->flags = FTRACE_OPS_FL_PID | FTRACE_OPS_GRAPH_STUB | + FTRACE_OPS_FL_SAVE_ARGS; #ifdef FTRACE_GRAPH_TRAMP_ADDR dst_ops->trampoline = FTRACE_GRAPH_TRAMP_ADDR; @@ -1177,10 +1191,14 @@ int register_ftrace_graph(struct fgraph_ops *gops) int ret = 0; int i; + if (gops->entryfunc && gops->entryregfunc) + return -EINVAL; + mutex_lock(&ftrace_lock); if (!gops->ops.func) { - gops->ops.flags |= FTRACE_OPS_GRAPH_STUB; + gops->ops.flags |= FTRACE_OPS_GRAPH_STUB | + FTRACE_OPS_FL_SAVE_ARGS; gops->ops.func = ftrace_graph_func; #ifdef FTRACE_GRAPH_TRAMP_ADDR gops->ops.trampoline = FTRACE_GRAPH_TRAMP_ADDR; From patchwork Fri Dec 8 10:28:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 175749 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5363657vqy; Fri, 8 Dec 2023 02:29:25 -0800 (PST) X-Google-Smtp-Source: AGHT+IGEbtU4WPT9oZK48B27PrIorc+Td0HSzlmpBwimxim6iGFbZm0kjiXx4GQmcjb3Xe5fysdJ X-Received: by 2002:a05:6a20:b7a4:b0:18f:4779:6781 with SMTP id fh36-20020a056a20b7a400b0018f47796781mr3225233pzb.105.1702031364765; Fri, 08 Dec 2023 02:29:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702031364; cv=none; d=google.com; s=arc-20160816; b=DnQys7e2Ld8FK3ik1xTP31JMyzXgSUJ3kdTGwHRKazY5r8zOuDnJCBLW+3IEolQiNL DWCleFdIfbZ4W+3C/GNqegSwjmRqyKyzyGcAzWHafCKcLrtCWFcwymMAK9b5Njyu8AH9 HUKbRW8B5bCWFPbxdePj4m8Y+bQyqdEwaebr8zcJCquGJGRuMGz2euGlZ0v/tmfOV0gW gfrzxW3RjDNzv05qaBVWuDDixX6V/iOn77EuScITXiDt4Bf46JuYZbDM1MNqayFijQWk 0M80m9/4eJhjivDnH0otSp66y88LjeqFaVXpZQVhPP8m2WnG7ETsKYqwvNg4V8YpeTQo Px8Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=IOSqfeYI3+QjuHNy92o6BsvYc99DdBnlsCXAChYNRCc=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=R84PEQb+FqWajfc6kIejMal0KB5+hTl/zERZ+w8voM2kFAY428rjzw6sNoTvu3lLQ8 FST+WNiFQX9z98hs/sqmJiGBpv7ts8nAOtdD1dLoHzSj2YRXGICZW6P/NTJxpLqJM5Dl bq80lpT90sBFx8Gs0+8HYN/VyvRPv2LWOrNRsGT57iqXOJGhf2MnXkWhM8Ih+VE7DtFB UKee+/Dfr/ZFhpszI+nSrJJoYjedEo8MtYc9801b9aHzBfvUHDtkOCN4TFiw9esVToER 8SsrVR8iHQra2u3a9wvRjqG8f3IVSnY14cFl48OKuEYU8C4vqGw1M9vbGUKY96ABDf/9 tC2A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=LELoD5+M; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id m9-20020a170902db0900b001ce5b8081a5si1370767plx.382.2023.12.08.02.29.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 02:29:24 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=LELoD5+M; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id AF1D2887B756; Fri, 8 Dec 2023 02:29:20 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1573512AbjLHK3H (ORCPT + 99 others); Fri, 8 Dec 2023 05:29:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42432 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1573606AbjLHK2i (ORCPT ); Fri, 8 Dec 2023 05:28:38 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CACF11BCB for ; Fri, 8 Dec 2023 02:28:24 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C59A2C433C8; Fri, 8 Dec 2023 10:28:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702031304; bh=GdHQ069T+AoIplKcM5E+5FNSoRMc79P1ef7Z465uBBM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=LELoD5+MUlDEUFPOclNqC3VO2iP6y2l6DgTsDxhuJNoqcC9EcaUaLGHxHNczuZOoI 0ZfINqzzolXAUkwGGMH/v9pel8eUxuQFK2xf7j9jeXt5TR8QSA2hVWTxFh+vXf+YQH Y0TjfcC63w8Qb12D4yR5F/acfcI5IYdlSm+PtpX8sQo+fkNBYXktp6Y4CNOQ1JIaTv D2fvTRKFAk//nb9D8pFlxZw7ZUmJ1RPfnXPhOy28O9ckvc/KbzukOaiyOHN8MMP6Pu v+126y9otuxWsGfg8smt6rcfDmeFGR1X00NFOeZU33LlUZ22IDhWRRjZYfZqEdVYRO dVZJd0pWqKQvg== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v4 20/33] function_graph: Add a new exit handler with parent_ip and ftrace_regs Date: Fri, 8 Dec 2023 19:28:18 +0900 Message-Id: <170203129774.579004.3666795448412962010.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170203105427.579004.8033550792660734570.stgit@devnote2> References: <170203105427.579004.8033550792660734570.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Fri, 08 Dec 2023 02:29:20 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784709240582692595 X-GMAIL-MSGID: 1784709240582692595 From: Masami Hiramatsu (Google) Add a new return handler to fgraph_ops as 'retregfunc' which takes parent_ip and ftrace_regs instead of ftrace_graph_ret. This handler is available only if the arch support CONFIG_HAVE_FUNCTION_GRAPH_FREGS. Note that the 'retfunc' and 'reregfunc' are mutual exclusive. You can set only one of them. Signed-off-by: Masami Hiramatsu (Google) --- Changes in v3: - Update for new multiple fgraph. - Save the return address to instruction pointer in ftrace_regs. --- arch/x86/include/asm/ftrace.h | 2 + include/linux/ftrace.h | 10 +++++- kernel/trace/Kconfig | 5 ++- kernel/trace/fgraph.c | 70 ++++++++++++++++++++++++++++------------- 4 files changed, 63 insertions(+), 24 deletions(-) diff --git a/arch/x86/include/asm/ftrace.h b/arch/x86/include/asm/ftrace.h index 415cf7a2ec2c..0b306c82855d 100644 --- a/arch/x86/include/asm/ftrace.h +++ b/arch/x86/include/asm/ftrace.h @@ -72,6 +72,8 @@ arch_ftrace_get_regs(struct ftrace_regs *fregs) override_function_with_return(&(fregs)->regs) #define ftrace_regs_query_register_offset(name) \ regs_query_register_offset(name) +#define ftrace_regs_get_frame_pointer(fregs) \ + frame_pointer(&(fregs)->regs) struct ftrace_ops; #define ftrace_graph_func ftrace_graph_func diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 6da6cc9aaeaf..79875a00c02b 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -43,7 +43,9 @@ struct dyn_ftrace; char *arch_ftrace_match_adjust(char *str, const char *search); -#ifdef CONFIG_HAVE_FUNCTION_GRAPH_RETVAL +#ifdef CONFIG_HAVE_FUNCTION_GRAPH_FREGS +unsigned long ftrace_return_to_handler(struct ftrace_regs *fregs); +#elif defined(CONFIG_HAVE_FUNCTION_GRAPH_RETVAL) struct fgraph_ret_regs; unsigned long ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs); #else @@ -157,6 +159,7 @@ struct ftrace_regs { #define ftrace_regs_set_instruction_pointer(fregs, ip) do { } while (0) #endif /* CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS */ + static __always_inline struct pt_regs *ftrace_get_regs(struct ftrace_regs *fregs) { if (!fregs) @@ -1067,6 +1070,10 @@ typedef int (*trace_func_graph_regs_ent_t)(unsigned long func, unsigned long parent_ip, struct ftrace_regs *fregs, struct fgraph_ops *); /* entry w/ regs */ +typedef void (*trace_func_graph_regs_ret_t)(unsigned long func, + unsigned long parent_ip, + struct ftrace_regs *, + struct fgraph_ops *); /* return w/ regs */ extern int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace, struct fgraph_ops *gops); @@ -1076,6 +1083,7 @@ struct fgraph_ops { trace_func_graph_ent_t entryfunc; trace_func_graph_ret_t retfunc; trace_func_graph_regs_ent_t entryregfunc; + trace_func_graph_regs_ret_t retregfunc; struct ftrace_ops ops; /* for the hash lists */ void *private; int idx; diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig index 61c541c36596..308b3bec01b1 100644 --- a/kernel/trace/Kconfig +++ b/kernel/trace/Kconfig @@ -34,6 +34,9 @@ config HAVE_FUNCTION_GRAPH_TRACER config HAVE_FUNCTION_GRAPH_RETVAL bool +config HAVE_FUNCTION_GRAPH_FREGS + bool + config HAVE_DYNAMIC_FTRACE bool help @@ -232,7 +235,7 @@ config FUNCTION_GRAPH_TRACER config FUNCTION_GRAPH_RETVAL bool "Kernel Function Graph Return Value" - depends on HAVE_FUNCTION_GRAPH_RETVAL + depends on HAVE_FUNCTION_GRAPH_RETVAL || HAVE_FUNCTION_GRAPH_FREGS depends on FUNCTION_GRAPH_TRACER default n help diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index 95b3eb4e8e23..0ac242d22724 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -685,8 +685,8 @@ int function_graph_enter_ops(unsigned long ret, unsigned long func, /* Retrieve a function return address to the trace stack on thread info.*/ static struct ftrace_ret_stack * -ftrace_pop_return_trace(struct ftrace_graph_ret *trace, unsigned long *ret, - unsigned long frame_pointer, int *index) +ftrace_pop_return_trace(unsigned long *ret, unsigned long frame_pointer, + int *index) { struct ftrace_ret_stack *ret_stack; @@ -731,10 +731,6 @@ ftrace_pop_return_trace(struct ftrace_graph_ret *trace, unsigned long *ret, *index += FGRAPH_RET_INDEX; *ret = ret_stack->ret; - trace->func = ret_stack->func; - trace->calltime = ret_stack->calltime; - trace->overrun = atomic_read(¤t->trace_overrun); - trace->depth = current->curr_ret_depth; /* * We still want to trace interrupts coming in if * max_depth is set to 1. Make sure the decrement is @@ -773,21 +769,42 @@ static struct notifier_block ftrace_suspend_notifier = { /* fgraph_ret_regs is not defined without CONFIG_FUNCTION_GRAPH_RETVAL */ struct fgraph_ret_regs; +static void fgraph_call_retfunc(struct ftrace_regs *fregs, + struct fgraph_ret_regs *ret_regs, + struct ftrace_ret_stack *ret_stack, + struct fgraph_ops *gops) +{ + struct ftrace_graph_ret trace; + + trace.func = ret_stack->func; + trace.calltime = ret_stack->calltime; + trace.overrun = atomic_read(¤t->trace_overrun); + trace.depth = current->curr_ret_depth; + trace.rettime = trace_clock_local(); +#ifdef CONFIG_FUNCTION_GRAPH_RETVAL + if (fregs) + trace.retval = ftrace_regs_return_value(fregs); + else + trace.retval = fgraph_ret_regs_return_value(ret_regs); +#endif + gops->retfunc(&trace, gops); +} + /* * Send the trace to the ring-buffer. * @return the original return address. */ -static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs, +static unsigned long __ftrace_return_to_handler(struct ftrace_regs *fregs, + struct fgraph_ret_regs *ret_regs, unsigned long frame_pointer) { struct ftrace_ret_stack *ret_stack; - struct ftrace_graph_ret trace; unsigned long bitmap; unsigned long ret; int index; int i; - ret_stack = ftrace_pop_return_trace(&trace, &ret, frame_pointer, &index); + ret_stack = ftrace_pop_return_trace(&ret, frame_pointer, &index); if (unlikely(!ret_stack)) { ftrace_graph_stop(); @@ -796,10 +813,8 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs return (unsigned long)panic; } - trace.rettime = trace_clock_local(); -#ifdef CONFIG_FUNCTION_GRAPH_RETVAL - trace.retval = fgraph_ret_regs_return_value(ret_regs); -#endif + if (fregs) + ftrace_regs_set_instruction_pointer(fregs, ret); bitmap = get_fgraph_index_bitmap(current, index); for (i = 0; i < FGRAPH_ARRAY_SIZE; i++) { @@ -810,7 +825,10 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs if (gops == &fgraph_stub) continue; - gops->retfunc(&trace, gops); + if (gops->retregfunc) + gops->retregfunc(ret_stack->func, ret, fregs, gops); + else + fgraph_call_retfunc(fregs, ret_regs, ret_stack, gops); } /* @@ -825,20 +843,22 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs return ret; } -/* - * After all architecures have selected HAVE_FUNCTION_GRAPH_RETVAL, we can - * leave only ftrace_return_to_handler(ret_regs). - */ -#ifdef CONFIG_HAVE_FUNCTION_GRAPH_RETVAL +#ifdef CONFIG_HAVE_FUNCTION_GRAPH_FREGS +unsigned long ftrace_return_to_handler(struct ftrace_regs *fregs) +{ + return __ftrace_return_to_handler(fregs, NULL, + ftrace_regs_get_frame_pointer(fregs)); +} +#elif defined(CONFIG_HAVE_FUNCTION_GRAPH_RETVAL) unsigned long ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs) { - return __ftrace_return_to_handler(ret_regs, + return __ftrace_return_to_handler(NULL, ret_regs, fgraph_ret_regs_frame_pointer(ret_regs)); } #else unsigned long ftrace_return_to_handler(unsigned long frame_pointer) { - return __ftrace_return_to_handler(NULL, frame_pointer); + return __ftrace_return_to_handler(NULL, NULL, frame_pointer); } #endif @@ -1191,9 +1211,15 @@ int register_ftrace_graph(struct fgraph_ops *gops) int ret = 0; int i; - if (gops->entryfunc && gops->entryregfunc) + if ((gops->entryfunc && gops->entryregfunc) || + (gops->retfunc && gops->retregfunc)) return -EINVAL; +#ifndef CONFIG_HAVE_FUNCTION_GRAPH_FREGS + if (gops->retregfunc) + return -EOPNOTSUPP; +#endif + mutex_lock(&ftrace_lock); if (!gops->ops.func) { From patchwork Fri Dec 8 10:28:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 175750 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5363823vqy; Fri, 8 Dec 2023 02:29:47 -0800 (PST) X-Google-Smtp-Source: AGHT+IEz+FYkESbHi1yTFTcRDvxYct6gTw4sMvaHsfgSMeMMuGco6E1Wo/Ct0UpGFhgGN8KyrQQF X-Received: by 2002:a17:902:d511:b0:1d0:6ffd:9e13 with SMTP id b17-20020a170902d51100b001d06ffd9e13mr4836710plg.101.1702031387614; Fri, 08 Dec 2023 02:29:47 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702031387; cv=none; d=google.com; s=arc-20160816; b=kdfNRtAI7cxfeGQ1R89b/HA2uPielFBjepDSZtFYpVjNdRmm6MlFK3RPgZiMpfzKRx YYlVF+yyZPzg5am5voULaow6q0euUBgTwDzwABn/6zgrAbN0xZqffgQLJYiQ2ZvV5XIY GjfuHUbMHUfihoCqL6MEFJAQ3DUpXr/IGDTZqlpXbIu/BeXb1ZbNc0gaLjXTVrDaaebe s4R7h4srunOusPd7nk2ko4AGjalHHDCfu12EfT87lsk/gPk5Xjb6l0IEWzG9t3vLFUwU UzI1KlQy3ScuPDa+9U8xGlhmpE9+xMEleiAs+Kl6qEHGfMG5E4okKlEj8O/Hoqr4o+Hd 0aSg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=DTVHGVwQZZBvqCqPcw35Ltjz0zNvoABeQNqXyujJvmA=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=hfee85baSQdFgeofCfK8VjP55uvGKQQpwZRJ1PvVn74mkWt/EWvkXBKXE8HmDNDH5z txCkSwMTC9LV24XksGnwt8jVHoC0FdhALiHtEhD4PCccVUuAmXWDIid4iSi1kz/2XEdg Qo56/jUUCv0LPKWi+ubP/GFq8ExgjP0rjldAe6cITT4t8EvHt/2xDE34BHZ2KicaqR5+ xD81aDbrIsDFISvYNT1DFMPHYV85nTiIrhqpdEIq9Hl119RHq7q8WEgGMmXKTjMOysfo hvgmkri5mEYzuUCajCfWNy9nvQqnQ90IOxeqfThZ3FLe1tSjo7fI8f6YEA/QbaROJpdA eZHg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=M4lND6uY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id x10-20020a170902ec8a00b001d1ccbe983asi1388532plg.383.2023.12.08.02.29.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 02:29:47 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=M4lND6uY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 3DACC80ABFC8; Fri, 8 Dec 2023 02:29:45 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235986AbjLHK3f (ORCPT + 99 others); Fri, 8 Dec 2023 05:29:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60296 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233360AbjLHK3R (ORCPT ); Fri, 8 Dec 2023 05:29:17 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 265AE1FC4 for ; Fri, 8 Dec 2023 02:28:37 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C2033C433C7; Fri, 8 Dec 2023 10:28:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702031316; bh=pPNJXjygt/V4vNYQ4X8NJkqM9twm0EdHZQuhceTJqsQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=M4lND6uY3SW66V0a116zBlvW7YKyDx2JWvKdATv15tvxVs1GXs5ibzMrf3AouWWlm VZWOWCEUTOmagWiyUuRHNRrYwpEFM2i8rqFUGPXtxhe9R8vuP4FuPk6DGFskiMSYdZ CGvTIJGqE2DFalwb9Qh8MJVZc6au4hduQkmMIEdF78R+E3QMXePZ5uhXgXJsVUnumd +4UTBzNiyf6godknzWaq4txBMIOe0FUd4rgCF8GlQCE+UaxYGZ3CeF/qJd4u+6Wie5 vE2dtpOLCaYuJQSww8ynTzuiHDYocyqtb4P2ePNFWFpvGkerx7xxLW/kEF3wQRV8ps bug+7C7g9HEXA== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v4 21/33] x86/ftrace: Enable HAVE_FUNCTION_GRAPH_FREGS Date: Fri, 8 Dec 2023 19:28:30 +0900 Message-Id: <170203130982.579004.8630034805609766591.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170203105427.579004.8033550792660734570.stgit@devnote2> References: <170203105427.579004.8033550792660734570.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Fri, 08 Dec 2023 02:29:45 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784709264206676767 X-GMAIL-MSGID: 1784709264206676767 From: Masami Hiramatsu (Google) Support HAVE_FUNCTION_GRAPH_FREGS on x86-64, which saves ftrace_regs on the stack in ftrace_graph return trampoline so that the callbacks can access registers via ftrace_regs APIs. Note that this only recovers 'rax' and 'rdx' registers because other registers are not used anymore and recovered by caller. 'rax' and 'rdx' will be used for passing the return value. Signed-off-by: Masami Hiramatsu (Google) --- Changes in v3: - Add a comment about rip. Changes in v2: - Save rsp register and drop clearing orig_ax. --- arch/x86/Kconfig | 3 ++- arch/x86/kernel/ftrace_64.S | 37 +++++++++++++++++++++++++++++-------- 2 files changed, 31 insertions(+), 9 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 3762f41bb092..3b955c9e4eb6 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -219,7 +219,8 @@ config X86 select HAVE_FAST_GUP select HAVE_FENTRY if X86_64 || DYNAMIC_FTRACE select HAVE_FTRACE_MCOUNT_RECORD - select HAVE_FUNCTION_GRAPH_RETVAL if HAVE_FUNCTION_GRAPH_TRACER + select HAVE_FUNCTION_GRAPH_FREGS if HAVE_DYNAMIC_FTRACE_WITH_ARGS + select HAVE_FUNCTION_GRAPH_RETVAL if !HAVE_DYNAMIC_FTRACE_WITH_ARGS select HAVE_FUNCTION_GRAPH_TRACER if X86_32 || (X86_64 && DYNAMIC_FTRACE) select HAVE_FUNCTION_TRACER select HAVE_GCC_PLUGINS diff --git a/arch/x86/kernel/ftrace_64.S b/arch/x86/kernel/ftrace_64.S index 214f30e9f0c0..8a16f774604e 100644 --- a/arch/x86/kernel/ftrace_64.S +++ b/arch/x86/kernel/ftrace_64.S @@ -348,21 +348,42 @@ STACK_FRAME_NON_STANDARD_FP(__fentry__) SYM_CODE_START(return_to_handler) UNWIND_HINT_UNDEFINED ANNOTATE_NOENDBR - subq $24, %rsp + /* + * Save the registers requires for ftrace_regs; + * rax, rcx, rdx, rdi, rsi, r8, r9 and rbp + */ + subq $(FRAME_SIZE), %rsp + movq %rax, RAX(%rsp) + movq %rcx, RCX(%rsp) + movq %rdx, RDX(%rsp) + movq %rsi, RSI(%rsp) + movq %rdi, RDI(%rsp) + movq %r8, R8(%rsp) + movq %r9, R9(%rsp) + movq %rbp, RBP(%rsp) + /* + * orig_ax is not cleared because it is used for indicating the direct + * trampoline in the fentry. And rip is not set because we don't know + * the correct return address here. + */ + + leaq FRAME_SIZE(%rsp), %rcx + movq %rcx, RSP(%rsp) - /* Save the return values */ - movq %rax, (%rsp) - movq %rdx, 8(%rsp) - movq %rbp, 16(%rsp) movq %rsp, %rdi call ftrace_return_to_handler movq %rax, %rdi - movq 8(%rsp), %rdx - movq (%rsp), %rax - addq $24, %rsp + /* + * Restore only rax and rdx because other registers are not used + * for return value nor callee saved. Caller will reuse/recover it. + */ + movq RDX(%rsp), %rdx + movq RAX(%rsp), %rax + + addq $(FRAME_SIZE), %rsp /* * Jump back to the old return address. This cannot be JMP_NOSPEC rdi * since IBT would demand that contain ENDBR, which simply isn't so for From patchwork Fri Dec 8 10:28:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 175751 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5364410vqy; Fri, 8 Dec 2023 02:31:07 -0800 (PST) X-Google-Smtp-Source: AGHT+IGp6S+dOHjmMnlWKyjTLNctyqTROPjMoC+lyPSuwZugdrBEuz9zP+B1KfgzbBJ4AOPtZBst X-Received: by 2002:a05:6358:63a1:b0:16e:292:2af2 with SMTP id k33-20020a05635863a100b0016e02922af2mr5103659rwh.21.1702031467054; Fri, 08 Dec 2023 02:31:07 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702031467; cv=none; d=google.com; s=arc-20160816; b=xz2M3FdwDhPBV66NNqD7DzJEv3289qk2bfzHy3L5zsKK6WVZmJz087rQjp+ApRDynC KMFQsnWNN/sBBLJejTyaF/uEBX3+GSCt1cQqWsN7CpXEtChOR6Ff4xlXO935z6dJp8AL V1kCc5rYW8HYEbE3aXUfVK0niyAutDd78+HJSijGRswXxjl4zReQYBjUj1jTPEyu9GVF TPViEGF5XL1fCc4kaZpXlz7DZ7PvrbLij4nqHGn6pLbjfBFnKX257j92yyf+tltx2bV1 MlVfJZoYLtwUb4SAsWXCAevgo5t6yXJI5sa7IX435+/PB2P97/ONLSrs1S/aYRzNJcyD s3Rw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=qJUudBxt3sssuiP5nsf8PvhfIqz19gr4FDYkYD1YrQ4=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=MqJMft5j1c1uWc0LtMmREHTsgcTHSClNCHsGkEJZMovlOfzLhQGgrTE/O07YGhEZ/G lokhA51slJz3bL8em92AFRB49eOKeKOrpYsEOUWncHVe+GSvq3rBxMNzoewjOBfaaS4+ rJWNLS2OTqVEWwK3ZVtDLl+s6KFAzvUr7jz80Jv9YnHlSyHlCf28SegUwG/Z9vqFrhhv xpjEwNK2VmXFmWJ2BDYGNJl56UtU2Zyyly5XCl2OZfXCv38+Bb0RnWkjspmFdYgbn3LW zbTagLwQsi7uv/e41w3cHSKF1LYh0AcAyZNvO9E/hU3hPIrLBsvabwtPYMEh7Cnzjqhw BZ2w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=AbpVdnfp; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from pete.vger.email (pete.vger.email. [2620:137:e000::3:6]) by mx.google.com with ESMTPS id fj5-20020a056a003a0500b006cdaac97b57si1347137pfb.213.2023.12.08.02.31.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 02:31:07 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) client-ip=2620:137:e000::3:6; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=AbpVdnfp; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 8FD9682C402F; Fri, 8 Dec 2023 02:31:00 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1573453AbjLHKaZ (ORCPT + 99 others); Fri, 8 Dec 2023 05:30:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60996 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1573661AbjLHKaE (ORCPT ); Fri, 8 Dec 2023 05:30:04 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 963421BFD for ; Fri, 8 Dec 2023 02:28:49 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 13E3BC433C9; Fri, 8 Dec 2023 10:28:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702031328; bh=c9wlW1E1FDo8FxkgpVxLOzRtzdXB3eV3Ja8e79iiw3U=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=AbpVdnfpsJ/aC6SSScCHBqjgOPMvcknjQnTyvOvG2u5HaGeryQI5Y1Smm5aigBEcE WrquRlTN5rpbzFrzfUQ1/ws42QWTJ49u4E90ZO0hq/4zGMsR6PJE05JlMApnXj7vBS iGa4a3CWHOAurdpZEt60Dg0+6tZJpZyq/Ee9kO6CIGzrtalyv4QMhM2zgfL6VkvFee IfTIdNGk7iwqzc6rs8wgUszJHC9YxyivKgz1pkC7QBS8Qaj3QtNR/aPZ17kzgGiiRo 7H82TIjfsFxnWF2Q0961VMu6ODGkjoCxVkGj0zec+p+6GvtnuSJdkNSDqk6xXmPR7d IoY9UQ+Pe8Mvg== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v4 22/33] tracing: Rename ftrace_regs_return_value to ftrace_regs_get_return_value Date: Fri, 8 Dec 2023 19:28:42 +0900 Message-Id: <170203132201.579004.3148503140553102819.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170203105427.579004.8033550792660734570.stgit@devnote2> References: <170203105427.579004.8033550792660734570.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Fri, 08 Dec 2023 02:31:00 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784709347852895553 X-GMAIL-MSGID: 1784709347852895553 From: Masami Hiramatsu (Google) Rename ftrace_regs_return_value to ftrace_regs_get_return_value as same as other ftrace_regs_get/set_* APIs. Signed-off-by: Masami Hiramatsu (Google) --- Changes in v3: - Newly added. --- arch/loongarch/include/asm/ftrace.h | 2 +- arch/powerpc/include/asm/ftrace.h | 2 +- arch/s390/include/asm/ftrace.h | 2 +- arch/x86/include/asm/ftrace.h | 2 +- include/linux/ftrace.h | 2 +- kernel/trace/fgraph.c | 2 +- 6 files changed, 6 insertions(+), 6 deletions(-) diff --git a/arch/loongarch/include/asm/ftrace.h b/arch/loongarch/include/asm/ftrace.h index a11996eb5892..a9c3d0f2f941 100644 --- a/arch/loongarch/include/asm/ftrace.h +++ b/arch/loongarch/include/asm/ftrace.h @@ -70,7 +70,7 @@ ftrace_regs_set_instruction_pointer(struct ftrace_regs *fregs, unsigned long ip) regs_get_kernel_argument(&(fregs)->regs, n) #define ftrace_regs_get_stack_pointer(fregs) \ kernel_stack_pointer(&(fregs)->regs) -#define ftrace_regs_return_value(fregs) \ +#define ftrace_regs_get_return_value(fregs) \ regs_return_value(&(fregs)->regs) #define ftrace_regs_set_return_value(fregs, ret) \ regs_set_return_value(&(fregs)->regs, ret) diff --git a/arch/powerpc/include/asm/ftrace.h b/arch/powerpc/include/asm/ftrace.h index 9e5a39b6a311..7e138e0e3baf 100644 --- a/arch/powerpc/include/asm/ftrace.h +++ b/arch/powerpc/include/asm/ftrace.h @@ -69,7 +69,7 @@ ftrace_regs_get_instruction_pointer(struct ftrace_regs *fregs) regs_get_kernel_argument(&(fregs)->regs, n) #define ftrace_regs_get_stack_pointer(fregs) \ kernel_stack_pointer(&(fregs)->regs) -#define ftrace_regs_return_value(fregs) \ +#define ftrace_regs_get_return_value(fregs) \ regs_return_value(&(fregs)->regs) #define ftrace_regs_set_return_value(fregs, ret) \ regs_set_return_value(&(fregs)->regs, ret) diff --git a/arch/s390/include/asm/ftrace.h b/arch/s390/include/asm/ftrace.h index 5a82b08f03cd..01e775c98425 100644 --- a/arch/s390/include/asm/ftrace.h +++ b/arch/s390/include/asm/ftrace.h @@ -88,7 +88,7 @@ ftrace_regs_set_instruction_pointer(struct ftrace_regs *fregs, regs_get_kernel_argument(&(fregs)->regs, n) #define ftrace_regs_get_stack_pointer(fregs) \ kernel_stack_pointer(&(fregs)->regs) -#define ftrace_regs_return_value(fregs) \ +#define ftrace_regs_get_return_value(fregs) \ regs_return_value(&(fregs)->regs) #define ftrace_regs_set_return_value(fregs, ret) \ regs_set_return_value(&(fregs)->regs, ret) diff --git a/arch/x86/include/asm/ftrace.h b/arch/x86/include/asm/ftrace.h index 0b306c82855d..a061f8832b20 100644 --- a/arch/x86/include/asm/ftrace.h +++ b/arch/x86/include/asm/ftrace.h @@ -64,7 +64,7 @@ arch_ftrace_get_regs(struct ftrace_regs *fregs) regs_get_kernel_argument(&(fregs)->regs, n) #define ftrace_regs_get_stack_pointer(fregs) \ kernel_stack_pointer(&(fregs)->regs) -#define ftrace_regs_return_value(fregs) \ +#define ftrace_regs_get_return_value(fregs) \ regs_return_value(&(fregs)->regs) #define ftrace_regs_set_return_value(fregs, ret) \ regs_set_return_value(&(fregs)->regs, ret) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 79875a00c02b..da2a23f5a9ed 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -187,7 +187,7 @@ static __always_inline bool ftrace_regs_has_args(struct ftrace_regs *fregs) regs_get_kernel_argument(ftrace_get_regs(fregs), n) #define ftrace_regs_get_stack_pointer(fregs) \ kernel_stack_pointer(ftrace_get_regs(fregs)) -#define ftrace_regs_return_value(fregs) \ +#define ftrace_regs_get_return_value(fregs) \ regs_return_value(ftrace_get_regs(fregs)) #define ftrace_regs_set_return_value(fregs, ret) \ regs_set_return_value(ftrace_get_regs(fregs), ret) diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index 0ac242d22724..ff830e7f8966 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -783,7 +783,7 @@ static void fgraph_call_retfunc(struct ftrace_regs *fregs, trace.rettime = trace_clock_local(); #ifdef CONFIG_FUNCTION_GRAPH_RETVAL if (fregs) - trace.retval = ftrace_regs_return_value(fregs); + trace.retval = ftrace_regs_get_return_value(fregs); else trace.retval = fgraph_ret_regs_return_value(ret_regs); #endif From patchwork Fri Dec 8 10:28:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 175752 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5364476vqy; Fri, 8 Dec 2023 02:31:16 -0800 (PST) X-Google-Smtp-Source: AGHT+IFbAALgpFdwcedyxC6OT8SLzLllw9nKwjRxa3fN095ysqUd+J/cem59I3H/SA3AvrT68Y4P X-Received: by 2002:a05:6a21:9712:b0:187:4e8c:ac5c with SMTP id ub18-20020a056a21971200b001874e8cac5cmr757005pzb.1.1702031476435; Fri, 08 Dec 2023 02:31:16 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702031476; cv=none; d=google.com; s=arc-20160816; b=O6BzUWbP2cvJ/bfZqPvtU6yhKqptY9GszlUUy5Peim2dH6cx8M96IyK9/WPDg0AOHI inoo6plDnvKNu0dylfa7a57zaGW8fJtn+RLw2uLk7kXjMQH9zcn0qjIuyvUpn/3rrXsu e/TjAM9EsTpgs2u6xgXQoGWoDC4CVGejJopTQmB7SkNKgADLxcgatGOpTTrBnhkhziyN lCC0AzWkrDEPZeNXPLDZKU7BcT8qlDP3d0OfbkQeIh+f5Nz8IJhoDEJlnSTs9DADcfuP rYqNPYi1Yznk18LBmJnPHiynEH9kVLnn38r6RWb+0JC4nUwiafvtCQHXVvEw73SjkTUv NFnw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=gyhDufMlqWYyit6UVacrM1UC8KltlsDNU7U2cyRBaIc=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=rGCOV0XWDB2Tf0A/OjyoV3tJ3QHX4ZH2vJs4ck8/ar3L2wan2xDE6mwW4PmbEL02RH 1vmYFCVz+JbcPEwQAhNN+Oezl6mKWszrASdiDlfBoHvaA1vxnfB9rTyVGyP6ZckoqvOG VJvX0UdcYmkEtOitYkFWYrdeKzUIMTlYwWZiYqPjARMwIETuvW7Mjn+Aqw7FNETOpfJY Iq0BRCPa+gO6RDPSQJvnVdVuVTnqdYd1xtY3XmyPsifs9bEBH2aVuKmJtfiz3E+hMT3r iWGu1x+hmZNpb9FxK9FH1tklK8Y1onjsshO2mJB5WWYsVh85QNCLZXY6fKoSOE4t2Agc P2VA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Wxrv3Qui; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from morse.vger.email (morse.vger.email. [2620:137:e000::3:1]) by mx.google.com with ESMTPS id s14-20020a65644e000000b005c1b59032b5si1301466pgv.453.2023.12.08.02.31.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 02:31:16 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Wxrv3Qui; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 8354C80A537F; Fri, 8 Dec 2023 02:30:19 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1573611AbjLHKaJ (ORCPT + 99 others); Fri, 8 Dec 2023 05:30:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60300 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235940AbjLHK3w (ORCPT ); Fri, 8 Dec 2023 05:29:52 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 761F319A3 for ; Fri, 8 Dec 2023 02:29:01 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4189FC433CA; Fri, 8 Dec 2023 10:28:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702031340; bh=Tv6V79hvaGG+hzXn9p0p+FTonUg+dTMFhQaVZNAc9e0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Wxrv3Qui0F6Sp5RExiFduF25BOMaAx6zNCR9WT/VL2AtmH+nhDGTedlPosbIrhZgJ Q0fm5Vi1kGn/6E15KjGehNP3bJzn0BlFlwsljxiFVIUryDHGxYaNwdRHbkz1VF4Lng riwsQ0ykEL6UdpbHlRDkWtE40AUUxuVqjqGoM/UJzvxGDZh+neIvkW8tZOHb4hr5Q1 eyi+EcXkbs+ZkiH0cmgZ5ds8/pRKzdLXnbpC6a7hhbgQJbKeuWt29G1D1q6yYypasF PpLcatFgWcw7JtPFvVNu/ncCXNlDECwptUctVQ5phqfmzPjwjoc10hgNIhwUYFtiyJ St1m5kNXfKJ+g== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v4 23/33] arm64: ftrace: Enable HAVE_FUNCTION_GRAPH_FREGS Date: Fri, 8 Dec 2023 19:28:54 +0900 Message-Id: <170203133430.579004.13001036830710360658.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170203105427.579004.8033550792660734570.stgit@devnote2> References: <170203105427.579004.8033550792660734570.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Fri, 08 Dec 2023 02:30:19 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784709357281827669 X-GMAIL-MSGID: 1784709357281827669 From: Masami Hiramatsu (Google) Enable CONFIG_HAVE_FUNCTION_GRAPH_FREGS on arm64. Note that this depends on HAVE_DYNAMIC_FTRACE_WITH_ARGS which is enabled if the compiler supports "-fpatchable-function-entry=2". If not, it continue to use ftrace_ret_regs. Signed-off-by: Masami Hiramatsu (Google) --- Changes in v3: - Newly added. --- arch/arm64/Kconfig | 2 ++ arch/arm64/include/asm/ftrace.h | 6 ++++++ arch/arm64/kernel/entry-ftrace.S | 28 ++++++++++++++++++++++++++++ 3 files changed, 36 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 7b071a00425d..beebc724dcae 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -192,6 +192,8 @@ config ARM64 select HAVE_DYNAMIC_FTRACE select HAVE_DYNAMIC_FTRACE_WITH_ARGS \ if $(cc-option,-fpatchable-function-entry=2) + select HAVE_FUNCTION_GRAPH_FREGS \ + if HAVE_DYNAMIC_FTRACE_WITH_ARGS select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS \ if DYNAMIC_FTRACE_WITH_ARGS && DYNAMIC_FTRACE_WITH_CALL_OPS select HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS \ diff --git a/arch/arm64/include/asm/ftrace.h b/arch/arm64/include/asm/ftrace.h index ab158196480c..efd5dbf74dd6 100644 --- a/arch/arm64/include/asm/ftrace.h +++ b/arch/arm64/include/asm/ftrace.h @@ -131,6 +131,12 @@ ftrace_regs_set_return_value(struct ftrace_regs *fregs, fregs->regs[0] = ret; } +static __always_inline unsigned long +ftrace_regs_get_frame_pointer(struct ftrace_regs *fregs) +{ + return fregs->fp; +} + static __always_inline void ftrace_override_function_with_return(struct ftrace_regs *fregs) { diff --git a/arch/arm64/kernel/entry-ftrace.S b/arch/arm64/kernel/entry-ftrace.S index f0c16640ef21..d87ccdb9e678 100644 --- a/arch/arm64/kernel/entry-ftrace.S +++ b/arch/arm64/kernel/entry-ftrace.S @@ -328,6 +328,33 @@ SYM_FUNC_END(ftrace_stub_graph) * Run ftrace_return_to_handler() before going back to parent. * @fp is checked against the value passed by ftrace_graph_caller(). */ +#ifdef CONFIG_HAVE_FUNCTION_GRAPH_FREGS +SYM_CODE_START(return_to_handler) + /* save ftrace_regs except for PC */ + sub sp, sp, #FREGS_SIZE + stp x0, x1, [sp, #FREGS_X0] + stp x2, x3, [sp, #FREGS_X2] + stp x4, x5, [sp, #FREGS_X4] + stp x6, x7, [sp, #FREGS_X6] + str x8, [sp, #FREGS_X8] + str x29, [sp, #FREGS_FP] + str x9, [sp, #FREGS_LR] + str x10, [sp, #FREGS_SP] + + mov x0, sp + bl ftrace_return_to_handler // addr = ftrace_return_to_hander(fregs); + mov x30, x0 // restore the original return address + + /* restore return value regs */ + ldp x0, x1, [sp, #FREGS_X0] + ldp x2, x3, [sp, #FREGS_X2] + ldp x4, x5, [sp, #FREGS_X4] + ldp x6, x7, [sp, #FREGS_X6] + add sp, sp, #FREGS_SIZE + + ret +SYM_CODE_END(return_to_handler) +#else /* !CONFIG_HAVE_FUNCTION_GRAPH_FREGS */ SYM_CODE_START(return_to_handler) /* save return value regs */ sub sp, sp, #FGRET_REGS_SIZE @@ -350,4 +377,5 @@ SYM_CODE_START(return_to_handler) ret SYM_CODE_END(return_to_handler) +#endif /* CONFIG_HAVE_FUNCTION_GRAPH_FREGS */ #endif /* CONFIG_FUNCTION_GRAPH_TRACER */ From patchwork Fri Dec 8 10:29:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 175753 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5364498vqy; Fri, 8 Dec 2023 02:31:19 -0800 (PST) X-Google-Smtp-Source: AGHT+IFWuYzimy/etLioDIoN8E55KR95r/iGbm1qnCNUTbEmh76AJTUqgaLNq+3FBe+lQQGdnjxy X-Received: by 2002:a05:6358:5918:b0:170:6675:a50a with SMTP id g24-20020a056358591800b001706675a50amr4651065rwf.36.1702031479136; Fri, 08 Dec 2023 02:31:19 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702031479; cv=none; d=google.com; s=arc-20160816; b=URcjlPjoPa7e+gVh5aNteRQWlUCKJ8nJOGj/a3Ltfsf2hCanz9B8pwMWQntOX0+XgD HIOrivPqeAVvcEULYpV3aJt5rtl55Y0ld2J17lwcs3Lws1jo0w0xanvhayeZQJ7uBuhO Hpp4F0S7qWUs24Lg4uvUgtBhunF+oVhlkKIGFZsZHX5PEJt/0ZxohbqtFGiAtjO3ckZJ kRtIQlHJDadSCbYH2I5lr8YdZ8msRzNN4ejXXBNoIyTnLAPohgnmQPCmnVTaA/E30uh2 FC3I/OQHSa59hee3r9UWnPdxZS9Rz5ANMsLlSWR+5j2q7OZ8ijorrApjeJvWjF69rUB9 vrSA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=b4g/hBIPEB0vqDLWH8qUko0XbjTYu/wJMCrVQAytV/A=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=PZETUs63mUfq9zDuOx37YStl4xzX0kNRd4kz9AK1zzYPTbLlFd9WYs4fXjpBalpgX4 k7TyBVdva99HDzcSALheX95xi5/9UkmN/Q4tM1badZEhKwlo5+k59rt9sp9N4HGUSvVG ziU7SPdc8r36jHGr83F4lVFU9Y9eX0mWGpO/h3aI5Zc6/TL82fWEFOl6oSdaeTfKBOIu BmwiAiBDJLat9YPZ0bcEEmoww4AbN6JuQQxBINQhrwFU1J37Lti2zPNWvBxhyJ/44K3U InNAo03iaX6/D/prz4ngi+ycjEhQ50Se/MLdMeSwvGn9Wl/ham+LAsr6Z91xBRILCXBl ++BA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Y4IMOGrZ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id a16-20020a656410000000b005b106cd44casi1376191pgv.145.2023.12.08.02.31.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 02:31:19 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Y4IMOGrZ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 6908C836E44D; Fri, 8 Dec 2023 02:31:03 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1573671AbjLHKaN (ORCPT + 99 others); Fri, 8 Dec 2023 05:30:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60408 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235889AbjLHK3z (ORCPT ); Fri, 8 Dec 2023 05:29:55 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 708721FDF for ; Fri, 8 Dec 2023 02:29:13 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3BFFFC433C8; Fri, 8 Dec 2023 10:29:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702031353; bh=zNyqbmxAtDH2ZV7+/rv/xziDgvyQgyv2PF9GtYfDqbk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Y4IMOGrZBvKBitZnGECxREOUZuc2cMyD93FDsRcqfDEevCMkugDgpFOyvAhPCvgmn OOIVszIS0Rj2fQSTO3F3nSGPfsjclAJTJ/66qCRGQVvbQwVVyU8NRbztTA+u955MNz JGPkdjPrg/hiZfvXR4p3P50Owm2zaHGY1EdXVd1yKuOK4zj17NeZe9tzYqzSgCvrzU px6/5rnY6EVcgzxOa8EnRhY8q+KdeEWDEU6g8K09e0C3GGuLd1brYw99prXs40TyAb Os8cTMR+H7vsrgdfsucsYcuiz5HVxglD3IeT4HAj5UlCo78AeVRp4/4DaTrCLX1UUW VJj+RLb6VANDg== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v4 24/33] fprobe: Use ftrace_regs in fprobe entry handler Date: Fri, 8 Dec 2023 19:29:06 +0900 Message-Id: <170203134628.579004.2373463093946605380.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170203105427.579004.8033550792660734570.stgit@devnote2> References: <170203105427.579004.8033550792660734570.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Fri, 08 Dec 2023 02:31:03 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784709360153636212 X-GMAIL-MSGID: 1784709360153636212 From: Masami Hiramatsu (Google) This allows fprobes to be available with CONFIG_DYNAMIC_FTRACE_WITH_ARGS instead of CONFIG_DYNAMIC_FTRACE_WITH_REGS, then we can enable fprobe on arm64. Signed-off-by: Masami Hiramatsu (Google) Acked-by: Florent Revest --- Changes from previous series: NOTHING, just forward ported. --- include/linux/fprobe.h | 2 +- kernel/trace/Kconfig | 3 ++- kernel/trace/bpf_trace.c | 10 +++++++--- kernel/trace/fprobe.c | 4 ++-- kernel/trace/trace_fprobe.c | 6 +++++- lib/test_fprobe.c | 4 ++-- samples/fprobe/fprobe_example.c | 2 +- 7 files changed, 20 insertions(+), 11 deletions(-) diff --git a/include/linux/fprobe.h b/include/linux/fprobe.h index 3e03758151f4..36c0595f7b93 100644 --- a/include/linux/fprobe.h +++ b/include/linux/fprobe.h @@ -35,7 +35,7 @@ struct fprobe { int nr_maxactive; int (*entry_handler)(struct fprobe *fp, unsigned long entry_ip, - unsigned long ret_ip, struct pt_regs *regs, + unsigned long ret_ip, struct ftrace_regs *regs, void *entry_data); void (*exit_handler)(struct fprobe *fp, unsigned long entry_ip, unsigned long ret_ip, struct pt_regs *regs, diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig index 308b3bec01b1..805d72ab77c6 100644 --- a/kernel/trace/Kconfig +++ b/kernel/trace/Kconfig @@ -290,7 +290,7 @@ config DYNAMIC_FTRACE_WITH_ARGS config FPROBE bool "Kernel Function Probe (fprobe)" depends on FUNCTION_TRACER - depends on DYNAMIC_FTRACE_WITH_REGS + depends on DYNAMIC_FTRACE_WITH_REGS || DYNAMIC_FTRACE_WITH_ARGS depends on HAVE_RETHOOK select RETHOOK default n @@ -675,6 +675,7 @@ config FPROBE_EVENTS select TRACING select PROBE_EVENTS select DYNAMIC_EVENTS + depends on DYNAMIC_FTRACE_WITH_REGS default y help This allows user to add tracing events on the function entry and diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index 84e8a0f6e4e0..d3f8745d8ead 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -2503,7 +2503,7 @@ static int __init bpf_event_init(void) fs_initcall(bpf_event_init); #endif /* CONFIG_MODULES */ -#ifdef CONFIG_FPROBE +#if defined(CONFIG_FPROBE) && defined(CONFIG_DYNAMIC_FTRACE_WITH_REGS) struct bpf_kprobe_multi_link { struct bpf_link link; struct fprobe fp; @@ -2733,10 +2733,14 @@ kprobe_multi_link_prog_run(struct bpf_kprobe_multi_link *link, static int kprobe_multi_link_handler(struct fprobe *fp, unsigned long fentry_ip, - unsigned long ret_ip, struct pt_regs *regs, + unsigned long ret_ip, struct ftrace_regs *fregs, void *data) { struct bpf_kprobe_multi_link *link; + struct pt_regs *regs = ftrace_get_regs(fregs); + + if (!regs) + return 0; link = container_of(fp, struct bpf_kprobe_multi_link, fp); kprobe_multi_link_prog_run(link, get_entry_ip(fentry_ip), regs); @@ -3008,7 +3012,7 @@ int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr kvfree(cookies); return err; } -#else /* !CONFIG_FPROBE */ +#else /* !CONFIG_FPROBE || !CONFIG_DYNAMIC_FTRACE_WITH_REGS */ int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog) { return -EOPNOTSUPP; diff --git a/kernel/trace/fprobe.c b/kernel/trace/fprobe.c index 6cd2a4e3afb8..f12569494d8a 100644 --- a/kernel/trace/fprobe.c +++ b/kernel/trace/fprobe.c @@ -46,7 +46,7 @@ static inline void __fprobe_handler(unsigned long ip, unsigned long parent_ip, } if (fp->entry_handler) - ret = fp->entry_handler(fp, ip, parent_ip, ftrace_get_regs(fregs), entry_data); + ret = fp->entry_handler(fp, ip, parent_ip, fregs, entry_data); /* If entry_handler returns !0, nmissed is not counted. */ if (rh) { @@ -182,7 +182,7 @@ static void fprobe_init(struct fprobe *fp) fp->ops.func = fprobe_kprobe_handler; else fp->ops.func = fprobe_handler; - fp->ops.flags |= FTRACE_OPS_FL_SAVE_REGS; + fp->ops.flags |= FTRACE_OPS_FL_SAVE_ARGS; } static int fprobe_init_rethook(struct fprobe *fp, int num) diff --git a/kernel/trace/trace_fprobe.c b/kernel/trace/trace_fprobe.c index 7d2ddbcfa377..ef6b36fd05ae 100644 --- a/kernel/trace/trace_fprobe.c +++ b/kernel/trace/trace_fprobe.c @@ -320,12 +320,16 @@ NOKPROBE_SYMBOL(fexit_perf_func); #endif /* CONFIG_PERF_EVENTS */ static int fentry_dispatcher(struct fprobe *fp, unsigned long entry_ip, - unsigned long ret_ip, struct pt_regs *regs, + unsigned long ret_ip, struct ftrace_regs *fregs, void *entry_data) { struct trace_fprobe *tf = container_of(fp, struct trace_fprobe, fp); + struct pt_regs *regs = ftrace_get_regs(fregs); int ret = 0; + if (!regs) + return 0; + if (trace_probe_test_flag(&tf->tp, TP_FLAG_TRACE)) fentry_trace_func(tf, entry_ip, regs); #ifdef CONFIG_PERF_EVENTS diff --git a/lib/test_fprobe.c b/lib/test_fprobe.c index 24de0e5ff859..ff607babba18 100644 --- a/lib/test_fprobe.c +++ b/lib/test_fprobe.c @@ -40,7 +40,7 @@ static noinline u32 fprobe_selftest_nest_target(u32 value, u32 (*nest)(u32)) static notrace int fp_entry_handler(struct fprobe *fp, unsigned long ip, unsigned long ret_ip, - struct pt_regs *regs, void *data) + struct ftrace_regs *fregs, void *data) { KUNIT_EXPECT_FALSE(current_test, preemptible()); /* This can be called on the fprobe_selftest_target and the fprobe_selftest_target2 */ @@ -81,7 +81,7 @@ static notrace void fp_exit_handler(struct fprobe *fp, unsigned long ip, static notrace int nest_entry_handler(struct fprobe *fp, unsigned long ip, unsigned long ret_ip, - struct pt_regs *regs, void *data) + struct ftrace_regs *fregs, void *data) { KUNIT_EXPECT_FALSE(current_test, preemptible()); return 0; diff --git a/samples/fprobe/fprobe_example.c b/samples/fprobe/fprobe_example.c index 64e715e7ed11..1545a1aac616 100644 --- a/samples/fprobe/fprobe_example.c +++ b/samples/fprobe/fprobe_example.c @@ -50,7 +50,7 @@ static void show_backtrace(void) static int sample_entry_handler(struct fprobe *fp, unsigned long ip, unsigned long ret_ip, - struct pt_regs *regs, void *data) + struct ftrace_regs *fregs, void *data) { if (use_trace) /* From patchwork Fri Dec 8 10:29:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 175755 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5364606vqy; Fri, 8 Dec 2023 02:31:33 -0800 (PST) X-Google-Smtp-Source: AGHT+IEfbG/19PkaS/TKLcNsOWqk8UyjsdONkMizFY4ZvUZMy5cRRBTT56+TyM8k+/FCXyaMwBWk X-Received: by 2002:a05:6a20:8605:b0:18c:1af0:df70 with SMTP id l5-20020a056a20860500b0018c1af0df70mr3638398pze.6.1702031492852; Fri, 08 Dec 2023 02:31:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702031492; cv=none; d=google.com; s=arc-20160816; b=FtChEOVRmWr38gzCziC7oDOL6QT35EzcN7cNBQFsYZ6MQ7X1DR9qvF0qkatxsiv5ir OHZFgJX+zrDk/uWqhYsL+2DO/x6ZEjNuvHHM6CRSz/NQ2j0NLl6lZhueWN0lFi40rXS+ B+D0cY50cS1PNW5bz4E6IpxENEpKkjde40iQA0J2vM42Y02uZs3QgwWTpBRy5Z9/iQEA BHkfYoxafDbiBy2QXCf4TNOHEnZ7eN3q0WJghY0LtXw645iSwqsrDOP+7ivfowVEDP69 JrN0DOZ+x4xM8d5AsguqJdq7cNsqmht6PkeESyp5K0lEezdiWKjKGucCVPrGC4SFn8so 8eQQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=lJXzaxm+7zhJL/RYv7PkJwMgsuRC1tWTTBNwqSpYIIs=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=e3kTXYF8jBpp0TzFOZ7D8rww5PHaBNcnSNebS6OP0vZSGsD/+bTaIk8gsUFrXpXwzp 8PqJ8ydoaibvTNhAqDOc04/emsqEiuM7+AYVstbuYl8bW8D6HSu8rivV8sFsjWHZ3zbE Ai7m5keD4vvhlEk2uDxjXI+xgiMlYEa0Q/QITTr8GdOQnjA3IOJfJAQHvsfmeeD1+0JE wSGJ8tF4sYczA29o8X5ZQctF0YlFVQAh2xKaSHF0IyWpYCBjeOQZwBuS8ipXa3IIePqy /QAPN4yHsycZ0V3U8BuZ+BrGPLpWIe1iLtRVsr4PK0wxiETmhCF8j5JlAo067zp9AIL+ xllA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=DR8YD1eE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id m12-20020a056a00080c00b006cb63c86cc2si1377967pfk.100.2023.12.08.02.31.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 02:31:32 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=DR8YD1eE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id C05E98089E47; Fri, 8 Dec 2023 02:30:46 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1573643AbjLHKa2 (ORCPT + 99 others); Fri, 8 Dec 2023 05:30:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49322 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1573663AbjLHKaF (ORCPT ); Fri, 8 Dec 2023 05:30:05 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D93D519BF for ; Fri, 8 Dec 2023 02:29:25 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 82C0EC433C7; Fri, 8 Dec 2023 10:29:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702031365; bh=oA88YoOQlagGtzf0XLpmZbrBwGQLOglr7IhXnROpFY0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DR8YD1eE18ip1VP88KiC1uM8UrMRddV3qpJMsh9wM72Mp+RkMbdvb2aS7BPVNsELl snUMFTKWziGy82Hs1IGTfZSH9VNVKNHPAl0eS3OHZ+JqGf1wkCgKUyKC/JzCj/yZdR +yrELu9Z3XANiWTQuX1VoS8uIWFsK4fKYmF4tN3r2ahjTJ9/g6j+1VPZ603oH/SebM DoJRwaMRQ2SD/OEWmu3Wpto5QLHNiQAjsgpiKQXMDHlOdnMusM1tF/wTG5yHF4Ry2x jcg8povqI2LZOiXSWD4x0rPv3LtWnirWDtGOK6UJP+RATSXhQK6H9kBId7sQUxMv1Y lH39CTCXAKp0Q== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v4 25/33] fprobe: Use ftrace_regs in fprobe exit handler Date: Fri, 8 Dec 2023 19:29:18 +0900 Message-Id: <170203135846.579004.17734254494569718868.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170203105427.579004.8033550792660734570.stgit@devnote2> References: <170203105427.579004.8033550792660734570.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Fri, 08 Dec 2023 02:30:47 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784709374761417235 X-GMAIL-MSGID: 1784709374761417235 From: Masami Hiramatsu (Google) Change the fprobe exit handler to use ftrace_regs structure instead of pt_regs. This also introduce HAVE_PT_REGS_TO_FTRACE_REGS_CAST which means the ftrace_regs's memory layout is equal to the pt_regs so that those are able to cast. Fprobe introduces a new dependency with that. Signed-off-by: Masami Hiramatsu (Google) --- Changes in v3: - Use ftrace_regs_get_return_value() Changes from previous series: NOTHING, just forward ported. --- arch/loongarch/Kconfig | 1 + arch/s390/Kconfig | 1 + arch/x86/Kconfig | 1 + include/linux/fprobe.h | 2 +- include/linux/ftrace.h | 5 +++++ kernel/trace/Kconfig | 8 ++++++++ kernel/trace/bpf_trace.c | 6 +++++- kernel/trace/fprobe.c | 3 ++- kernel/trace/trace_fprobe.c | 6 +++++- lib/test_fprobe.c | 6 +++--- samples/fprobe/fprobe_example.c | 2 +- 11 files changed, 33 insertions(+), 8 deletions(-) diff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig index ee123820a476..b0bd252aefe8 100644 --- a/arch/loongarch/Kconfig +++ b/arch/loongarch/Kconfig @@ -108,6 +108,7 @@ config LOONGARCH select HAVE_DMA_CONTIGUOUS select HAVE_DYNAMIC_FTRACE select HAVE_DYNAMIC_FTRACE_WITH_ARGS + select HAVE_PT_REGS_TO_FTRACE_REGS_CAST select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS select HAVE_DYNAMIC_FTRACE_WITH_REGS select HAVE_EBPF_JIT diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig index 3bec98d20283..122e9d6e3ad3 100644 --- a/arch/s390/Kconfig +++ b/arch/s390/Kconfig @@ -168,6 +168,7 @@ config S390 select HAVE_DMA_CONTIGUOUS select HAVE_DYNAMIC_FTRACE select HAVE_DYNAMIC_FTRACE_WITH_ARGS + select HAVE_PT_REGS_TO_FTRACE_REGS_CAST select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS select HAVE_DYNAMIC_FTRACE_WITH_REGS select HAVE_EBPF_JIT if HAVE_MARCH_Z196_FEATURES diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 3b955c9e4eb6..1d1da801da7f 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -209,6 +209,7 @@ config X86 select HAVE_DYNAMIC_FTRACE select HAVE_DYNAMIC_FTRACE_WITH_REGS select HAVE_DYNAMIC_FTRACE_WITH_ARGS if X86_64 + select HAVE_PT_REGS_TO_FTRACE_REGS_CAST if X86_64 select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS select HAVE_SAMPLE_FTRACE_DIRECT if X86_64 select HAVE_SAMPLE_FTRACE_DIRECT_MULTI if X86_64 diff --git a/include/linux/fprobe.h b/include/linux/fprobe.h index 36c0595f7b93..879a30956009 100644 --- a/include/linux/fprobe.h +++ b/include/linux/fprobe.h @@ -38,7 +38,7 @@ struct fprobe { unsigned long ret_ip, struct ftrace_regs *regs, void *entry_data); void (*exit_handler)(struct fprobe *fp, unsigned long entry_ip, - unsigned long ret_ip, struct pt_regs *regs, + unsigned long ret_ip, struct ftrace_regs *fregs, void *entry_data); }; diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index da2a23f5a9ed..a72a2eaec576 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -159,6 +159,11 @@ struct ftrace_regs { #define ftrace_regs_set_instruction_pointer(fregs, ip) do { } while (0) #endif /* CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS */ +#ifdef CONFIG_HAVE_PT_REGS_TO_FTRACE_REGS_CAST + +static_assert(sizeof(struct pt_regs) == sizeof(struct ftrace_regs)); + +#endif /* CONFIG_HAVE_PT_REGS_TO_FTRACE_REGS_CAST */ static __always_inline struct pt_regs *ftrace_get_regs(struct ftrace_regs *fregs) { diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig index 805d72ab77c6..1a2544712690 100644 --- a/kernel/trace/Kconfig +++ b/kernel/trace/Kconfig @@ -60,6 +60,13 @@ config HAVE_DYNAMIC_FTRACE_WITH_ARGS This allows for use of ftrace_regs_get_argument() and ftrace_regs_get_stack_pointer(). +config HAVE_PT_REGS_TO_FTRACE_REGS_CAST + bool + help + If this is set, the memory layout of the ftrace_regs data structure + is the same as the pt_regs. So the pt_regs is possible to be casted + to ftrace_regs. + config HAVE_DYNAMIC_FTRACE_NO_PATCHABLE bool help @@ -291,6 +298,7 @@ config FPROBE bool "Kernel Function Probe (fprobe)" depends on FUNCTION_TRACER depends on DYNAMIC_FTRACE_WITH_REGS || DYNAMIC_FTRACE_WITH_ARGS + depends on HAVE_PT_REGS_TO_FTRACE_REGS_CAST || !HAVE_DYNAMIC_FTRACE_WITH_ARGS depends on HAVE_RETHOOK select RETHOOK default n diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index d3f8745d8ead..efb792f8f2ea 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -2749,10 +2749,14 @@ kprobe_multi_link_handler(struct fprobe *fp, unsigned long fentry_ip, static void kprobe_multi_link_exit_handler(struct fprobe *fp, unsigned long fentry_ip, - unsigned long ret_ip, struct pt_regs *regs, + unsigned long ret_ip, struct ftrace_regs *fregs, void *data) { struct bpf_kprobe_multi_link *link; + struct pt_regs *regs = ftrace_get_regs(fregs); + + if (!regs) + return; link = container_of(fp, struct bpf_kprobe_multi_link, fp); kprobe_multi_link_prog_run(link, get_entry_ip(fentry_ip), regs); diff --git a/kernel/trace/fprobe.c b/kernel/trace/fprobe.c index f12569494d8a..688b897626b4 100644 --- a/kernel/trace/fprobe.c +++ b/kernel/trace/fprobe.c @@ -124,6 +124,7 @@ static void fprobe_exit_handler(struct rethook_node *rh, void *data, { struct fprobe *fp = (struct fprobe *)data; struct fprobe_rethook_node *fpr; + struct ftrace_regs *fregs = (struct ftrace_regs *)regs; int bit; if (!fp || fprobe_disabled(fp)) @@ -141,7 +142,7 @@ static void fprobe_exit_handler(struct rethook_node *rh, void *data, return; } - fp->exit_handler(fp, fpr->entry_ip, ret_ip, regs, + fp->exit_handler(fp, fpr->entry_ip, ret_ip, fregs, fp->entry_data_size ? (void *)fpr->data : NULL); ftrace_test_recursion_unlock(bit); } diff --git a/kernel/trace/trace_fprobe.c b/kernel/trace/trace_fprobe.c index ef6b36fd05ae..3982626c82e6 100644 --- a/kernel/trace/trace_fprobe.c +++ b/kernel/trace/trace_fprobe.c @@ -341,10 +341,14 @@ static int fentry_dispatcher(struct fprobe *fp, unsigned long entry_ip, NOKPROBE_SYMBOL(fentry_dispatcher); static void fexit_dispatcher(struct fprobe *fp, unsigned long entry_ip, - unsigned long ret_ip, struct pt_regs *regs, + unsigned long ret_ip, struct ftrace_regs *fregs, void *entry_data) { struct trace_fprobe *tf = container_of(fp, struct trace_fprobe, fp); + struct pt_regs *regs = ftrace_get_regs(fregs); + + if (!regs) + return; if (trace_probe_test_flag(&tf->tp, TP_FLAG_TRACE)) fexit_trace_func(tf, entry_ip, ret_ip, regs); diff --git a/lib/test_fprobe.c b/lib/test_fprobe.c index ff607babba18..271ce0caeec0 100644 --- a/lib/test_fprobe.c +++ b/lib/test_fprobe.c @@ -59,9 +59,9 @@ static notrace int fp_entry_handler(struct fprobe *fp, unsigned long ip, static notrace void fp_exit_handler(struct fprobe *fp, unsigned long ip, unsigned long ret_ip, - struct pt_regs *regs, void *data) + struct ftrace_regs *fregs, void *data) { - unsigned long ret = regs_return_value(regs); + unsigned long ret = ftrace_regs_get_return_value(fregs); KUNIT_EXPECT_FALSE(current_test, preemptible()); if (ip != target_ip) { @@ -89,7 +89,7 @@ static notrace int nest_entry_handler(struct fprobe *fp, unsigned long ip, static notrace void nest_exit_handler(struct fprobe *fp, unsigned long ip, unsigned long ret_ip, - struct pt_regs *regs, void *data) + struct ftrace_regs *fregs, void *data) { KUNIT_EXPECT_FALSE(current_test, preemptible()); KUNIT_EXPECT_EQ(current_test, ip, target_nest_ip); diff --git a/samples/fprobe/fprobe_example.c b/samples/fprobe/fprobe_example.c index 1545a1aac616..d476d1f07538 100644 --- a/samples/fprobe/fprobe_example.c +++ b/samples/fprobe/fprobe_example.c @@ -67,7 +67,7 @@ static int sample_entry_handler(struct fprobe *fp, unsigned long ip, } static void sample_exit_handler(struct fprobe *fp, unsigned long ip, - unsigned long ret_ip, struct pt_regs *regs, + unsigned long ret_ip, struct ftrace_regs *regs, void *data) { unsigned long rip = ret_ip; From patchwork Fri Dec 8 10:29:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 175759 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5364818vqy; Fri, 8 Dec 2023 02:31:55 -0800 (PST) X-Google-Smtp-Source: AGHT+IFnpe68QvpN1zYuOUDUp/7Lt53vgVcn7j0ITUPy1XYBo9bQegYhRe617of9sI5FRY8Uspil X-Received: by 2002:a05:6a20:daaa:b0:190:3d8d:a0bc with SMTP id iy42-20020a056a20daaa00b001903d8da0bcmr567358pzb.13.1702031515103; Fri, 08 Dec 2023 02:31:55 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702031515; cv=none; d=google.com; s=arc-20160816; b=A1OvdNduBgm5b2e0cd+gGnjApIJFMPcHmx2uy3qfimrUq13CcA3dCICB05CPlQy/u3 1j0JwhVT2imSvxn9cEU1lR98ABGIHcT1LuW3595BLlfEX7L9lc8P/oTx3XpWIMW1OKpi cc5FccNtzIN6oayy4n9YZ7/R0OZaIQpF4RbIGpU2WodqzI1ce0gu/dPiVhHgjbNZXG7P PaOQk3/dZO0+NG0a3EnIqTL8Sb5kyykKZoXf35IeJ3GmHe7N6Yey+0sRwOEAv3ugdNVS vAfsQf6w5OidVJ+Xgi5PR/58oDBr8ln+JdjTg7VgLU2QSKpe4teGwe7ORGboJ1xfD3bw T85A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=6CvyDT2/9dgk4JdmyncNzCvoFxaDuIj66p5rM2LDAl8=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=DVs9oqOktQK8VdkuWvMxKXjdWDaZc3LzFv4fFVIyVBfBWYcG7zRQnWRYBSz6C868mD wOzAEzsvJzw0pmtZr6A9mKNE952iRaOksJLqX3T30gakznqeDy5DebkaG5tGH79IMnG/ FR5eNvISiEDQ2w8VkYqggaqiu5NhSahx0xKB9RNxtTFV4S9/A05FD/ki1ksHmrmH6IHW 6vwFZFPA/xSjwEBt84gMujWyGCWawtTu0TtUV9nl+kTKf6D/VTRswE6DLJh9LYoOk10O d7kLAUFnlkqEIrEB08HfdeZHn5kFHkGU5IBJuh7dc/bm4Klzel2OngS08EL2I9sqWESQ Gf6A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=n7Re6rUO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from pete.vger.email (pete.vger.email. [23.128.96.36]) by mx.google.com with ESMTPS id mi9-20020a17090b4b4900b0028a09a28496si1517550pjb.66.2023.12.08.02.31.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 02:31:55 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=n7Re6rUO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 0DFE9805A7EA; Fri, 8 Dec 2023 02:31:50 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1573620AbjLHKat (ORCPT + 99 others); Fri, 8 Dec 2023 05:30:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60390 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235867AbjLHKaL (ORCPT ); Fri, 8 Dec 2023 05:30:11 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2E3CD1BE3 for ; Fri, 8 Dec 2023 02:29:38 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id AFA15C43391; Fri, 8 Dec 2023 10:29:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702031377; bh=JC1iyaIU7mWHHBTGXNCiN27YHUuUtROsifUJHNlxJlc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=n7Re6rUO6fn9k4hVJEvPaax3RsF8sF/iz/MGcYJm/ESScLZGLdvsKZR2NaUCzQT+h 1UddnQcN9e739IL0Z2KqPLqwGbyw31Ehmveu9YxnVQKhbGW48hngQbxSahDp513zAQ BMQ5TltayWO0lhiaX0mViMkSvHtBi5VmMPbiDqiwnL3lwg7nI17p0knTYGzHbS/y0d IbGdz//IeqHF05UCiL9XP1t9ucFHAx0g9GmEKJFi5FnilNuHrvVu1fMNUjECaWDUKh xa2NX2RCk/vX2Eimf9ghf22I8XdIRPBQ+yWS70ASxa6vkNm9CIdec8pLh5aIx4x1qp FpQrsmLSV0biA== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v4 26/33] tracing: Add ftrace_partial_regs() for converting ftrace_regs to pt_regs Date: Fri, 8 Dec 2023 19:29:31 +0900 Message-Id: <170203137075.579004.6040641469988010653.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170203105427.579004.8033550792660734570.stgit@devnote2> References: <170203105427.579004.8033550792660734570.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Fri, 08 Dec 2023 02:31:50 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784709397617837922 X-GMAIL-MSGID: 1784709397617837922 From: Masami Hiramatsu (Google) Add ftrace_partial_regs() which converts the ftrace_regs to pt_regs. If the architecture defines its own ftrace_regs, this copies partial registers to pt_regs and returns it. If not, ftrace_regs is the same as pt_regs and ftrace_partial_regs() will return ftrace_regs::regs. Signed-off-by: Masami Hiramatsu (Google) Acked-by: Florent Revest --- Changes from previous series: NOTHING, just forward ported. --- arch/arm64/include/asm/ftrace.h | 11 +++++++++++ include/linux/ftrace.h | 17 +++++++++++++++++ 2 files changed, 28 insertions(+) diff --git a/arch/arm64/include/asm/ftrace.h b/arch/arm64/include/asm/ftrace.h index efd5dbf74dd6..31051fa2b4d9 100644 --- a/arch/arm64/include/asm/ftrace.h +++ b/arch/arm64/include/asm/ftrace.h @@ -143,6 +143,17 @@ ftrace_override_function_with_return(struct ftrace_regs *fregs) fregs->pc = fregs->lr; } +static __always_inline struct pt_regs * +ftrace_partial_regs(const struct ftrace_regs *fregs, struct pt_regs *regs) +{ + memcpy(regs->regs, fregs->regs, sizeof(u64) * 9); + regs->sp = fregs->sp; + regs->pc = fregs->pc; + regs->regs[29] = fregs->fp; + regs->regs[30] = fregs->lr; + return regs; +} + int ftrace_regs_query_register_offset(const char *name); int ftrace_init_nop(struct module *mod, struct dyn_ftrace *rec); diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index a72a2eaec576..515ec804d605 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -173,6 +173,23 @@ static __always_inline struct pt_regs *ftrace_get_regs(struct ftrace_regs *fregs return arch_ftrace_get_regs(fregs); } +#if !defined(CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS) || \ + defined(CONFIG_HAVE_PT_REGS_TO_FTRACE_REGS_CAST) + +static __always_inline struct pt_regs * +ftrace_partial_regs(struct ftrace_regs *fregs, struct pt_regs *regs) +{ + /* + * If CONFIG_HAVE_PT_REGS_TO_FTRACE_REGS_CAST=y, ftrace_regs memory + * layout is the same as pt_regs. So always returns that address. + * Since arch_ftrace_get_regs() will check some members and may return + * NULL, we can not use it. + */ + return &fregs->regs; +} + +#endif /* !CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS || CONFIG_HAVE_PT_REGS_TO_FTRACE_REGS_CAST */ + /* * When true, the ftrace_regs_{get,set}_*() functions may be used on fregs. * Note: this can be true even when ftrace_get_regs() cannot provide a pt_regs. From patchwork Fri Dec 8 10:29:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 175764 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5367293vqy; Fri, 8 Dec 2023 02:38:29 -0800 (PST) X-Google-Smtp-Source: AGHT+IHh80EbFKYz3lpE4R+PXtoGIs/V1hBWyyk1P5WDPG8XwFCQaPhTVP8hZL4pVno7yaQxJBed X-Received: by 2002:a05:6a00:1399:b0:6ce:866:7bef with SMTP id t25-20020a056a00139900b006ce08667befmr4945321pfg.5.1702031909519; Fri, 08 Dec 2023 02:38:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702031909; cv=none; d=google.com; s=arc-20160816; b=aoIwtUlVmvGtPenNHmhT1mMVpi3G6HJY+NYSuj3o+uvyFlUT6ZOVSbUhUDdn8mNO0u c22zcN8aFnCKJxLYpC7My4v/kCCQU8FOJFybzk65ycn2OcXxujKvczvw41sBanguGW6h dms+NQYN1aWaFc80VA4n3AJqSuUvphdfhGrKwHZssl4/COPAI0ZQjzKtw2f6DflbNqLp cOowdR0/bYekONIzYTl7ejsoUR7crJ+lJM0YfVKSsc2wha/lHoktHMtPImjizkQZE9Bz +96VyXjq3OWNMHbS51m5PQvlYpa6jGqk2CiAJ4DWLQgU9A7bIL+JyyBMwy4nFUtS1k5t dimA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=I/NOdqZd4vjz2DBRERgllFY0C5eXmhFfbexyugXtquw=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=Itsji8BPDMOU/6rWFP6lqP2/xXgdDObtXuLHTQ5OJbF4qUDNr6X7dIEN/LropXQ0br kk7Q0s92HcK+p7M9fD2oD9xVy6NazzvIpcBMPQbR0r9YMuEEEUIEfYaI+kEILvOyu6B7 vhCttY32bGMqRkLHpG3y7oIebhkHPw9/l53qvWYP9bFH4RtcXXIq94JpchjJmFD5FU16 FQYuJUDWoZxyMAj5MENyYHUoPw+MRCVxajSgFKCwQckS4g5nIHY93LIQbWvMXnDFSp8i UTz1IEAE9+rxkIu70KzAJOBfZINjE5npmw/wz+t6726yuD8+RHFlCuAU39A6SN5S1hML sv4w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=oOvWvE8T; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id t3-20020a63eb03000000b005c6183ccab2si1315277pgh.630.2023.12.08.02.38.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 02:38:29 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=oOvWvE8T; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 3A55F82372CA; Fri, 8 Dec 2023 02:38:25 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1573724AbjLHKiL (ORCPT + 99 others); Fri, 8 Dec 2023 05:38:11 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39782 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1573654AbjLHKiI (ORCPT ); Fri, 8 Dec 2023 05:38:08 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1A5D82114 for ; Fri, 8 Dec 2023 02:29:50 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id DD247C433C9; Fri, 8 Dec 2023 10:29:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702031389; bh=qNmjHVAcUhvkoBgfYIyluKuCUP9AzOWE8BbeAF90uB0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oOvWvE8TkF6/u8bgZIdhZQWWiUj1p3OGXQYmyUFwxiu0wsbjUyP7i2CzK5diCXNWf PWrUGxmSpeHo7fFG8aszT1HXDNIH/IYwlB8FV19hhwRlazuYsWV95X2miUjdYyrt2n nBQvLquyQZKg9AMydnA5S9a3CidSY9pvfksOSBtAn4TOyMfwHaxJN7lHRS7w5TpwzQ 0dXtC84iM/tkNGVNXGXIopkm1xwWzyWlrOLj1fF40R6PL+fBz9dThGzoCj9IugPDPX 3YSNqplCzX6DuOtWdqJPVLtXB91AVM6OP8pLDi4jvKNtwC9Fqel+j0K7ENoIBdk+mM 4djFLqETUUqMg== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v4 27/33] tracing: Add ftrace_fill_perf_regs() for perf event Date: Fri, 8 Dec 2023 19:29:43 +0900 Message-Id: <170203138294.579004.12112201392035747803.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170203105427.579004.8033550792660734570.stgit@devnote2> References: <170203105427.579004.8033550792660734570.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Fri, 08 Dec 2023 02:38:25 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784709811445008077 X-GMAIL-MSGID: 1784709811445008077 From: Masami Hiramatsu (Google) Add ftrace_fill_perf_regs() which should be compatible with the perf_fetch_caller_regs(). In other words, the pt_regs returned from the ftrace_fill_perf_regs() must satisfy 'user_mode(regs) == false' and can be used for stack tracing. Signed-off-by: Masami Hiramatsu (Google) --- Changes from previous series: NOTHING, just forward ported. --- arch/arm64/include/asm/ftrace.h | 7 +++++++ arch/powerpc/include/asm/ftrace.h | 7 +++++++ arch/s390/include/asm/ftrace.h | 5 +++++ arch/x86/include/asm/ftrace.h | 7 +++++++ include/linux/ftrace.h | 31 +++++++++++++++++++++++++++++++ 5 files changed, 57 insertions(+) diff --git a/arch/arm64/include/asm/ftrace.h b/arch/arm64/include/asm/ftrace.h index 31051fa2b4d9..c1921bdf760b 100644 --- a/arch/arm64/include/asm/ftrace.h +++ b/arch/arm64/include/asm/ftrace.h @@ -154,6 +154,13 @@ ftrace_partial_regs(const struct ftrace_regs *fregs, struct pt_regs *regs) return regs; } +#define arch_ftrace_fill_perf_regs(fregs, _regs) do { \ + (_regs)->pc = (fregs)->pc; \ + (_regs)->regs[29] = (fregs)->fp; \ + (_regs)->sp = (fregs)->sp; \ + (_regs)->pstate = PSR_MODE_EL1h; \ + } while (0) + int ftrace_regs_query_register_offset(const char *name); int ftrace_init_nop(struct module *mod, struct dyn_ftrace *rec); diff --git a/arch/powerpc/include/asm/ftrace.h b/arch/powerpc/include/asm/ftrace.h index 7e138e0e3baf..8737a794c764 100644 --- a/arch/powerpc/include/asm/ftrace.h +++ b/arch/powerpc/include/asm/ftrace.h @@ -52,6 +52,13 @@ static __always_inline struct pt_regs *arch_ftrace_get_regs(struct ftrace_regs * return fregs->regs.msr ? &fregs->regs : NULL; } +#define arch_ftrace_fill_perf_regs(fregs, _regs) do { \ + (_regs)->result = 0; \ + (_regs)->nip = (fregs)->regs.nip; \ + (_regs)->gpr[1] = (fregs)->regs.gpr[1]; \ + asm volatile("mfmsr %0" : "=r" ((_regs)->msr)); \ + } while (0) + static __always_inline void ftrace_regs_set_instruction_pointer(struct ftrace_regs *fregs, unsigned long ip) diff --git a/arch/s390/include/asm/ftrace.h b/arch/s390/include/asm/ftrace.h index 01e775c98425..c2a269c1617c 100644 --- a/arch/s390/include/asm/ftrace.h +++ b/arch/s390/include/asm/ftrace.h @@ -97,6 +97,11 @@ ftrace_regs_set_instruction_pointer(struct ftrace_regs *fregs, #define ftrace_regs_query_register_offset(name) \ regs_query_register_offset(name) +#define arch_ftrace_fill_perf_regs(fregs, _regs) do { \ + (_regs)->psw.addr = (fregs)->regs.psw.addr; \ + (_regs)->gprs[15] = (fregs)->regs.gprs[15]; \ + } while (0) + #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS /* * When an ftrace registered caller is tracing a function that is diff --git a/arch/x86/include/asm/ftrace.h b/arch/x86/include/asm/ftrace.h index a061f8832b20..2e3de45e9746 100644 --- a/arch/x86/include/asm/ftrace.h +++ b/arch/x86/include/asm/ftrace.h @@ -54,6 +54,13 @@ arch_ftrace_get_regs(struct ftrace_regs *fregs) return &fregs->regs; } +#define arch_ftrace_fill_perf_regs(fregs, _regs) do { \ + (_regs)->ip = (fregs)->regs.ip; \ + (_regs)->sp = (fregs)->regs.sp; \ + (_regs)->cs = __KERNEL_CS; \ + (_regs)->flags = 0; \ + } while (0) + #define ftrace_regs_set_instruction_pointer(fregs, _ip) \ do { (fregs)->regs.ip = (_ip); } while (0) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 515ec804d605..8150edcf8496 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -190,6 +190,37 @@ ftrace_partial_regs(struct ftrace_regs *fregs, struct pt_regs *regs) #endif /* !CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS || CONFIG_HAVE_PT_REGS_TO_FTRACE_REGS_CAST */ +#ifdef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS + +/* + * Please define arch dependent pt_regs which compatible to the + * perf_arch_fetch_caller_regs() but based on ftrace_regs. + * This requires + * - user_mode(_regs) returns false (always kernel mode). + * - able to use the _regs for stack trace. + */ +#ifndef arch_ftrace_fill_perf_regs +/* As same as perf_arch_fetch_caller_regs(), do nothing by default */ +#define arch_ftrace_fill_perf_regs(fregs, _regs) do {} while (0) +#endif + +static __always_inline struct pt_regs * +ftrace_fill_perf_regs(struct ftrace_regs *fregs, struct pt_regs *regs) +{ + arch_ftrace_fill_perf_regs(fregs, regs); + return regs; +} + +#else /* !CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS */ + +static __always_inline struct pt_regs * +ftrace_fill_perf_regs(struct ftrace_regs *fregs, struct pt_regs *regs) +{ + return &fregs->regs; +} + +#endif + /* * When true, the ftrace_regs_{get,set}_*() functions may be used on fregs. * Note: this can be true even when ftrace_get_regs() cannot provide a pt_regs. From patchwork Fri Dec 8 10:29:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 175754 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5364577vqy; Fri, 8 Dec 2023 02:31:29 -0800 (PST) X-Google-Smtp-Source: AGHT+IEv0W2FJXYdBOmFbPqtyfn6OMl6wf76Amz6KHF71xPdtyjdhv3GvhOkw2myc5HPsTGVgQoO X-Received: by 2002:a17:90b:3504:b0:27d:3439:c141 with SMTP id ls4-20020a17090b350400b0027d3439c141mr3339636pjb.39.1702031489318; Fri, 08 Dec 2023 02:31:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702031489; cv=none; d=google.com; s=arc-20160816; b=HgrsRRnL/+SM/uEoXBqcD/au/aS9XF3tDBIPDgvfQde4PjeHO96ivMeh0dhcy6083E 6Ol/y6Z6gqHpWCW2VOW/Z9Bp38coUKmHU3jGN3dq4opYu5cZQB7Z87swwE6d2vqpWe0J +yJi/IbZppBuj8FhzneXruVFosEA/KUAZdA10bc0PWye3OHAosrOKnA8kYsBiCMLdClP 9/ppHT6hlzeqXvxR/1+mf5Ru52MbsL7ncLtJOG72KjphMSmmdJnXNhCWqkFDY8PsjjnU 3engq5EAJxEY2MTlVe+tSWChwNojx7QMX8R9nENKAHiOR85eMSrAum3KmhrTQ7963P90 Gr8g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=P+HfD/5oqs7vb9coeTqed46Okv8wAv5jqp8r/h0pgs8=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=ckTb9Mu9ynuk9An1xbAvYroNS3odNBd1UGGhh0ikpjlHzf1auYlCqR7gu4h+2i0pxb Qf6ba9Lq5Askm7g2V48klyXoSX3IN4z+PnK0uWGpByGw0VNrH9QKAaqTvb5SR/fo0CT5 +WeZm5ttzs+yWq8nCD1mf65LP4tSZ4ljQs5QxzaE/Tv5KHM463TcRB7wB0vRhGvnUDLn 6SPCDLHT+TADy4EZFxsplu8NgcEQ6ZNQ1eRFO2E7tZV3VaZy65JQHVBnmZdJIl74bIZS TpNsk84zIS4qBKdlTPqOrOhWmy66y3PXQS9Z5BAJfrhSROYAPU31s2cVb2BSTcnmQjD3 2X+w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=GtYIxLJI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from fry.vger.email (fry.vger.email. [23.128.96.38]) by mx.google.com with ESMTPS id 30-20020a17090a09a100b00286926f5b20si2745145pjo.148.2023.12.08.02.31.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 02:31:29 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) client-ip=23.128.96.38; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=GtYIxLJI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 82E1681113AB; Fri, 8 Dec 2023 02:31:20 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1573641AbjLHKaz (ORCPT + 99 others); Fri, 8 Dec 2023 05:30:55 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60300 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1573642AbjLHKa2 (ORCPT ); Fri, 8 Dec 2023 05:30:28 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9E1511BD2 for ; Fri, 8 Dec 2023 02:30:02 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2FBC1C433C7; Fri, 8 Dec 2023 10:29:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702031402; bh=j7cOJZgtZsX3oBL0hR7rnKNtOuVVKnmGxoHA7ISMNXU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GtYIxLJIfnq1wRcWETJqY9y/St8HOS5wMK30bTiSI+LioCG0fOBxDW5eRMUwIAqFq XzLH7Rtr2AwbQbYfbPeEmzDd5NS3z/hpCOjZFIcwxdS8YfhgGaQ5ZH/8MTAJhEJfUf uJuqwPl+dQpExJmFGh1/Wr5rupxOa398JIUMiYt/cdKYlUpYXdL+5gIynFMrqMtsnm Iy4fSrndcZbMK2p7MrE/BUGDos+p1Jvbna208M25vSPtns68scug8W+UgEuamG6Pyd pz6kMYyiPJ5X0SWdOkS6aN3I6+ilm46KMUlQvf5ExkqOA4l6DgT+qoE2SwJL0k/Ffh pbpKDQ4FbjFHg== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v4 28/33] fprobe: Rewrite fprobe on function-graph tracer Date: Fri, 8 Dec 2023 19:29:55 +0900 Message-Id: <170203139512.579004.1763946749526872058.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170203105427.579004.8033550792660734570.stgit@devnote2> References: <170203105427.579004.8033550792660734570.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Fri, 08 Dec 2023 02:31:20 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784709371031008479 X-GMAIL-MSGID: 1784709371031008479 From: Masami Hiramatsu (Google) Rewrite fprobe implementation on function-graph tracer. Major API changes are: - 'nr_maxactive' field is deprecated. - This depends on CONFIG_DYNAMIC_FTRACE_WITH_ARGS or !CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS, and CONFIG_HAVE_FUNCTION_GRAPH_FREGS. So currently works only on x86_64. - Currently the entry size is limited in 15 * sizeof(long). - If there is too many fprobe exit handler set on the same function, it will fail to probe. Signed-off-by: Masami Hiramatsu (Google) --- Changes in v3: - Update for new reserve_data/retrieve_data API. - Fix internal push/pop on fgraph data logic so that it can correctly save/restore the returning fprobes. Changes in v2: - Add more lockdep_assert_held(fprobe_mutex) - Use READ_ONCE() and WRITE_ONCE() for fprobe_hlist_node::fp. - Add NOKPROBE_SYMBOL() for the functions which is called from entry/exit callback. --- include/linux/fprobe.h | 54 +++- kernel/trace/Kconfig | 8 - kernel/trace/fprobe.c | 632 ++++++++++++++++++++++++++++++++++-------------- lib/test_fprobe.c | 45 --- 4 files changed, 494 insertions(+), 245 deletions(-) diff --git a/include/linux/fprobe.h b/include/linux/fprobe.h index 879a30956009..08b37b0d1d05 100644 --- a/include/linux/fprobe.h +++ b/include/linux/fprobe.h @@ -5,32 +5,56 @@ #include #include -#include +#include +#include +#include + +struct fprobe; + +/** + * strcut fprobe_hlist_node - address based hash list node for fprobe. + * + * @hlist: The hlist node for address search hash table. + * @addr: The address represented by this. + * @fp: The fprobe which owns this. + */ +struct fprobe_hlist_node { + struct hlist_node hlist; + unsigned long addr; + struct fprobe *fp; +}; + +/** + * struct fprobe_hlist - hash list nodes for fprobe. + * + * @hlist: The hlist node for existence checking hash table. + * @rcu: rcu_head for RCU deferred release. + * @fp: The fprobe which owns this fprobe_hlist. + * @size: The size of @array. + * @array: The fprobe_hlist_node for each address to probe. + */ +struct fprobe_hlist { + struct hlist_node hlist; + struct rcu_head rcu; + struct fprobe *fp; + int size; + struct fprobe_hlist_node array[]; +}; /** * struct fprobe - ftrace based probe. - * @ops: The ftrace_ops. + * * @nmissed: The counter for missing events. * @flags: The status flag. - * @rethook: The rethook data structure. (internal data) * @entry_data_size: The private data storage size. - * @nr_maxactive: The max number of active functions. + * @nr_maxactive: The max number of active functions. (*deprecated) * @entry_handler: The callback function for function entry. * @exit_handler: The callback function for function exit. + * @hlist_array: The fprobe_hlist for fprobe search from IP hash table. */ struct fprobe { -#ifdef CONFIG_FUNCTION_TRACER - /* - * If CONFIG_FUNCTION_TRACER is not set, CONFIG_FPROBE is disabled too. - * But user of fprobe may keep embedding the struct fprobe on their own - * code. To avoid build error, this will keep the fprobe data structure - * defined here, but remove ftrace_ops data structure. - */ - struct ftrace_ops ops; -#endif unsigned long nmissed; unsigned int flags; - struct rethook *rethook; size_t entry_data_size; int nr_maxactive; @@ -40,6 +64,8 @@ struct fprobe { void (*exit_handler)(struct fprobe *fp, unsigned long entry_ip, unsigned long ret_ip, struct ftrace_regs *fregs, void *entry_data); + + struct fprobe_hlist *hlist_array; }; /* This fprobe is soft-disabled. */ diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig index 1a2544712690..11a96275b68c 100644 --- a/kernel/trace/Kconfig +++ b/kernel/trace/Kconfig @@ -296,11 +296,9 @@ config DYNAMIC_FTRACE_WITH_ARGS config FPROBE bool "Kernel Function Probe (fprobe)" - depends on FUNCTION_TRACER - depends on DYNAMIC_FTRACE_WITH_REGS || DYNAMIC_FTRACE_WITH_ARGS - depends on HAVE_PT_REGS_TO_FTRACE_REGS_CAST || !HAVE_DYNAMIC_FTRACE_WITH_ARGS - depends on HAVE_RETHOOK - select RETHOOK + depends on FUNCTION_GRAPH_TRACER + depends on HAVE_FUNCTION_GRAPH_FREGS + depends on DYNAMIC_FTRACE_WITH_ARGS || !HAVE_DYNAMIC_FTRACE_WITH_ARGS default n help This option enables kernel function probe (fprobe) based on ftrace. diff --git a/kernel/trace/fprobe.c b/kernel/trace/fprobe.c index 688b897626b4..53e681c2458b 100644 --- a/kernel/trace/fprobe.c +++ b/kernel/trace/fprobe.c @@ -8,98 +8,193 @@ #include #include #include -#include +#include +#include #include #include #include "trace.h" -struct fprobe_rethook_node { - struct rethook_node node; - unsigned long entry_ip; - unsigned long entry_parent_ip; - char data[]; -}; +#define FPROBE_IP_HASH_BITS 8 +#define FPROBE_IP_TABLE_SIZE (1 << FPROBE_IP_HASH_BITS) -static inline void __fprobe_handler(unsigned long ip, unsigned long parent_ip, - struct ftrace_ops *ops, struct ftrace_regs *fregs) -{ - struct fprobe_rethook_node *fpr; - struct rethook_node *rh = NULL; - struct fprobe *fp; - void *entry_data = NULL; - int ret = 0; +#define FPROBE_HASH_BITS 6 +#define FPROBE_TABLE_SIZE (1 << FPROBE_HASH_BITS) - fp = container_of(ops, struct fprobe, ops); +/* + * fprobe_table: hold 'fprobe_hlist::hlist' for checking the fprobe still + * exists. The key is the address of fprobe instance. + * fprobe_ip_table: hold 'fprobe_hlist::array[*]' for searching the fprobe + * instance related to the funciton address. The key is the ftrace IP + * address. + * + * When unregistering the fprobe, fprobe_hlist::fp and fprobe_hlist::array[*].fp + * are set NULL and delete those from both hash tables (by hlist_del_rcu). + * After an RCU grace period, the fprobe_hlist itself will be released. + * + * fprobe_table and fprobe_ip_table can be accessed from either + * - Normal hlist traversal and RCU add/del under 'fprobe_mutex' is held. + * - RCU hlist traversal under disabling preempt + */ +static struct hlist_head fprobe_table[FPROBE_TABLE_SIZE]; +static struct hlist_head fprobe_ip_table[FPROBE_IP_TABLE_SIZE]; +static DEFINE_MUTEX(fprobe_mutex); - if (fp->exit_handler) { - rh = rethook_try_get(fp->rethook); - if (!rh) { - fp->nmissed++; - return; - } - fpr = container_of(rh, struct fprobe_rethook_node, node); - fpr->entry_ip = ip; - fpr->entry_parent_ip = parent_ip; - if (fp->entry_data_size) - entry_data = fpr->data; +/* + * Find first fprobe in the hlist. It will be iterated twice in the entry + * probe, once for correcting the total required size, the second time is + * calling back the user handlers. + * Thus the hlist in the fprobe_table must be sorted and new probe needs to + * be added *before* the first fprobe. + */ +static struct fprobe_hlist_node *find_first_fprobe_node(unsigned long ip) +{ + struct fprobe_hlist_node *node; + struct hlist_head *head; + + head = &fprobe_ip_table[hash_ptr((void *)ip, FPROBE_IP_HASH_BITS)]; + hlist_for_each_entry_rcu(node, head, hlist, + lockdep_is_held(&fprobe_mutex)) { + if (node->addr == ip) + return node; } + return NULL; +} +NOKPROBE_SYMBOL(find_first_fprobe_node); - if (fp->entry_handler) - ret = fp->entry_handler(fp, ip, parent_ip, fregs, entry_data); +/* Node insertion and deletion requires the fprobe_mutex */ +static void insert_fprobe_node(struct fprobe_hlist_node *node) +{ + unsigned long ip = node->addr; + struct fprobe_hlist_node *next; + struct hlist_head *head; - /* If entry_handler returns !0, nmissed is not counted. */ - if (rh) { - if (ret) - rethook_recycle(rh); - else - rethook_hook(rh, ftrace_get_regs(fregs), true); + lockdep_assert_held(&fprobe_mutex); + + next = find_first_fprobe_node(ip); + if (next) { + hlist_add_before_rcu(&node->hlist, &next->hlist); + return; } + head = &fprobe_ip_table[hash_ptr((void *)ip, FPROBE_IP_HASH_BITS)]; + hlist_add_head_rcu(&node->hlist, head); } -static void fprobe_handler(unsigned long ip, unsigned long parent_ip, - struct ftrace_ops *ops, struct ftrace_regs *fregs) +/* Return true if there are synonims */ +static bool delete_fprobe_node(struct fprobe_hlist_node *node) { - struct fprobe *fp; - int bit; + lockdep_assert_held(&fprobe_mutex); - fp = container_of(ops, struct fprobe, ops); - if (fprobe_disabled(fp)) - return; + WRITE_ONCE(node->fp, NULL); + hlist_del_rcu(&node->hlist); + return !!find_first_fprobe_node(node->addr); +} - /* recursion detection has to go before any traceable function and - * all functions before this point should be marked as notrace - */ - bit = ftrace_test_recursion_trylock(ip, parent_ip); - if (bit < 0) { - fp->nmissed++; - return; +/* Check existence of the fprobe */ +static bool is_fprobe_still_exist(struct fprobe *fp) +{ + struct hlist_head *head; + struct fprobe_hlist *fph; + + head = &fprobe_table[hash_ptr(fp, FPROBE_HASH_BITS)]; + hlist_for_each_entry_rcu(fph, head, hlist, + lockdep_is_held(&fprobe_mutex)) { + if (fph->fp == fp) + return true; } - __fprobe_handler(ip, parent_ip, ops, fregs); - ftrace_test_recursion_unlock(bit); + return false; +} +NOKPROBE_SYMBOL(is_fprobe_still_exist); + +static int add_fprobe_hash(struct fprobe *fp) +{ + struct fprobe_hlist *fph = fp->hlist_array; + struct hlist_head *head; + + lockdep_assert_held(&fprobe_mutex); + if (WARN_ON_ONCE(!fph)) + return -EINVAL; + + if (is_fprobe_still_exist(fp)) + return -EEXIST; + + head = &fprobe_table[hash_ptr(fp, FPROBE_HASH_BITS)]; + hlist_add_head_rcu(&fp->hlist_array->hlist, head); + return 0; } -NOKPROBE_SYMBOL(fprobe_handler); -static void fprobe_kprobe_handler(unsigned long ip, unsigned long parent_ip, - struct ftrace_ops *ops, struct ftrace_regs *fregs) +static int del_fprobe_hash(struct fprobe *fp) { - struct fprobe *fp; - int bit; + struct fprobe_hlist *fph = fp->hlist_array; - fp = container_of(ops, struct fprobe, ops); - if (fprobe_disabled(fp)) - return; + lockdep_assert_held(&fprobe_mutex); - /* recursion detection has to go before any traceable function and - * all functions called before this point should be marked as notrace - */ - bit = ftrace_test_recursion_trylock(ip, parent_ip); - if (bit < 0) { - fp->nmissed++; - return; + if (WARN_ON_ONCE(!fph)) + return -EINVAL; + + if (!is_fprobe_still_exist(fp)) + return -ENOENT; + + fph->fp = NULL; + hlist_del_rcu(&fph->hlist); + return 0; +} + +/* The entry data size is 4 bits (=16) * sizeof(long) in maximum */ +#define FPROBE_HEADER_SIZE_BITS 4 +#define MAX_FPROBE_DATA_SIZE_WORD ((1L << FPROBE_HEADER_SIZE_BITS) - 1) +#define MAX_FPROBE_DATA_SIZE (MAX_FPROBE_DATA_SIZE_WORD * sizeof(long)) +#define FPROBE_HEADER_PTR_BITS (BITS_PER_LONG - FPROBE_HEADER_SIZE_BITS) +#define FPROBE_HEADER_PTR_MASK GENMASK(FPROBE_HEADER_PTR_BITS - 1, 0) +#define FPROBE_HEADER_SIZE sizeof(unsigned long) + +static inline unsigned long encode_fprobe_header(struct fprobe *fp, int size_words) +{ + if (WARN_ON_ONCE(size_words > MAX_FPROBE_DATA_SIZE_WORD || + ((unsigned long)fp & ~FPROBE_HEADER_PTR_MASK) != + ~FPROBE_HEADER_PTR_MASK)) { + return 0; } + return ((unsigned long)size_words << FPROBE_HEADER_PTR_BITS) | + ((unsigned long)fp & FPROBE_HEADER_PTR_MASK); +} + +/* Return reserved data size in words */ +static inline int decode_fprobe_header(unsigned long val, struct fprobe **fp) +{ + unsigned long ptr; + + ptr = (val & FPROBE_HEADER_PTR_MASK) | ~FPROBE_HEADER_PTR_MASK; + if (fp) + *fp = (struct fprobe *)ptr; + return val >> FPROBE_HEADER_PTR_BITS; +} + +/* + * fprobe shadow stack management: + * Since fprobe shares a single fgraph_ops, it needs to share the stack entry + * among the probes on the same function exit. Note that a new probe can be + * registered before a target function is returning, we can not use the hash + * table to find the corresponding probes. Thus the probe address is stored on + * the shadow stack with its entry data size. + * + */ +static inline int __fprobe_handler(unsigned long ip, unsigned long parent_ip, + struct fprobe *fp, struct ftrace_regs *fregs, + void *data) +{ + if (!fp->entry_handler) + return 0; + + return fp->entry_handler(fp, ip, parent_ip, fregs, data); +} +static inline int __fprobe_kprobe_handler(unsigned long ip, unsigned long parent_ip, + struct fprobe *fp, struct ftrace_regs *fregs, + void *data) +{ + int ret; /* * This user handler is shared with other kprobes and is not expected to be * called recursively. So if any other kprobe handler is running, this will @@ -108,45 +203,180 @@ static void fprobe_kprobe_handler(unsigned long ip, unsigned long parent_ip, */ if (unlikely(kprobe_running())) { fp->nmissed++; - goto recursion_unlock; + return 0; } kprobe_busy_begin(); - __fprobe_handler(ip, parent_ip, ops, fregs); + ret = __fprobe_handler(ip, parent_ip, fp, fregs, data); kprobe_busy_end(); - -recursion_unlock: - ftrace_test_recursion_unlock(bit); + return ret; } -static void fprobe_exit_handler(struct rethook_node *rh, void *data, - unsigned long ret_ip, struct pt_regs *regs) +static int fprobe_entry(unsigned long func, unsigned long ret_ip, + struct ftrace_regs *fregs, struct fgraph_ops *gops) { - struct fprobe *fp = (struct fprobe *)data; - struct fprobe_rethook_node *fpr; - struct ftrace_regs *fregs = (struct ftrace_regs *)regs; - int bit; + struct fprobe_hlist_node *node, *first; + unsigned long *fgraph_data = NULL; + unsigned long header; + int reserved_words; + struct fprobe *fp; + int used, ret; - if (!fp || fprobe_disabled(fp)) - return; + if (WARN_ON_ONCE(!fregs)) + return 0; - fpr = container_of(rh, struct fprobe_rethook_node, node); + first = node = find_first_fprobe_node(func); + if (unlikely(!first)) + return 0; + + reserved_words = 0; + hlist_for_each_entry_from_rcu(node, hlist) { + if (node->addr != func) + break; + fp = READ_ONCE(node->fp); + if (!fp || !fp->exit_handler) + continue; + /* + * Since fprobe can be enabled until the next loop, we ignore the + * fprobe's disabled flag in this loop. + */ + reserved_words += + DIV_ROUND_UP(fp->entry_data_size, sizeof(long)) + 1; + } + node = first; + if (reserved_words) { + fgraph_data = fgraph_reserve_data(gops->idx, reserved_words * sizeof(long)); + if (unlikely(!fgraph_data)) { + hlist_for_each_entry_from_rcu(node, hlist) { + if (node->addr != func) + break; + fp = READ_ONCE(node->fp); + if (fp && !fprobe_disabled(fp)) + fp->nmissed++; + } + return 0; + } + } /* - * we need to assure no calls to traceable functions in-between the - * end of fprobe_handler and the beginning of fprobe_exit_handler. + * TODO: recursion detection has been done in the fgraph. Thus we need + * to add a callback to increment missed counter. */ - bit = ftrace_test_recursion_trylock(fpr->entry_ip, fpr->entry_parent_ip); - if (bit < 0) { - fp->nmissed++; + used = 0; + hlist_for_each_entry_from_rcu(node, hlist) { + void *data; + + if (node->addr != func) + break; + fp = READ_ONCE(node->fp); + if (!fp || fprobe_disabled(fp)) + continue; + + if (fp->entry_data_size && fp->exit_handler) + data = fgraph_data + used + 1; + else + data = NULL; + + if (fprobe_shared_with_kprobes(fp)) + ret = __fprobe_kprobe_handler(func, ret_ip, fp, fregs, data); + else + ret = __fprobe_handler(func, ret_ip, fp, fregs, data); + /* If entry_handler returns !0, nmissed is not counted but skips exit_handler. */ + if (!ret && fp->exit_handler) { + int size_words = DIV_ROUND_UP(fp->entry_data_size, sizeof(long)); + + header = encode_fprobe_header(fp, size_words); + if (likely(header)) { + fgraph_data[used] = header; + used += size_words + 1; + } + } + } + if (used < reserved_words) + memset(fgraph_data + used, 0, reserved_words - used); + + /* If any exit_handler is set, data must be used. */ + return used != 0; +} +NOKPROBE_SYMBOL(fprobe_entry); + +static void fprobe_return(unsigned long func, unsigned long ret_ip, + struct ftrace_regs *fregs, struct fgraph_ops *gops) +{ + unsigned long *fgraph_data = NULL; + unsigned long val; + struct fprobe *fp; + int size, curr; + int size_words; + + fgraph_data = (unsigned long *)fgraph_retrieve_data(gops->idx, &size); + if (!fgraph_data) + return; + size_words = DIV_ROUND_UP(size, sizeof(long)); + + preempt_disable(); + + curr = 0; + while (size_words > curr) { + val = fgraph_data[curr++]; + if (!val) + break; + + size = decode_fprobe_header(val, &fp); + if (fp && is_fprobe_still_exist(fp) && !fprobe_disabled(fp)) { + if (WARN_ON_ONCE(curr + size > size_words)) + break; + fp->exit_handler(fp, func, ret_ip, fregs, + size ? fgraph_data + curr : NULL); + } + curr += size + 1; + } + preempt_enable(); +} +NOKPROBE_SYMBOL(fprobe_return); + +static struct fgraph_ops fprobe_graph_ops = { + .entryregfunc = fprobe_entry, + .retregfunc = fprobe_return, +}; +static int fprobe_graph_active; + +/* Add @addrs to the ftrace filter and register fgraph if needed. */ +static int fprobe_graph_add_ips(unsigned long *addrs, int num) +{ + int ret; + + lockdep_assert_held(&fprobe_mutex); + + ret = ftrace_set_filter_ips(&fprobe_graph_ops.ops, addrs, num, 0, 0); + if (ret) + return ret; + + if (!fprobe_graph_active) { + ret = register_ftrace_graph(&fprobe_graph_ops); + if (WARN_ON_ONCE(ret)) { + ftrace_free_filter(&fprobe_graph_ops.ops); + return ret; + } + } + fprobe_graph_active++; + return 0; +} + +/* Remove @addrs from the ftrace filter and unregister fgraph if possible. */ +static void fprobe_graph_remove_ips(unsigned long *addrs, int num) +{ + lockdep_assert_held(&fprobe_mutex); + + fprobe_graph_active--; + if (!fprobe_graph_active) { + /* Q: should we unregister it ? */ + unregister_ftrace_graph(&fprobe_graph_ops); return; } - fp->exit_handler(fp, fpr->entry_ip, ret_ip, fregs, - fp->entry_data_size ? (void *)fpr->data : NULL); - ftrace_test_recursion_unlock(bit); + ftrace_set_filter_ips(&fprobe_graph_ops.ops, addrs, num, 1, 0); } -NOKPROBE_SYMBOL(fprobe_exit_handler); static int symbols_cmp(const void *a, const void *b) { @@ -176,55 +406,97 @@ static unsigned long *get_ftrace_locations(const char **syms, int num) return ERR_PTR(-ENOENT); } -static void fprobe_init(struct fprobe *fp) -{ - fp->nmissed = 0; - if (fprobe_shared_with_kprobes(fp)) - fp->ops.func = fprobe_kprobe_handler; - else - fp->ops.func = fprobe_handler; - fp->ops.flags |= FTRACE_OPS_FL_SAVE_ARGS; -} +struct filter_match_data { + const char *filter; + const char *notfilter; + size_t index; + size_t size; + unsigned long *addrs; +}; -static int fprobe_init_rethook(struct fprobe *fp, int num) +static int filter_match_callback(void *data, const char *name, unsigned long addr) { - int size; + struct filter_match_data *match = data; - if (num <= 0) - return -EINVAL; + if (!glob_match(match->filter, name) || + (match->notfilter && glob_match(match->notfilter, name))) + return 0; - if (!fp->exit_handler) { - fp->rethook = NULL; + if (!ftrace_location(addr)) return 0; - } - /* Initialize rethook if needed */ - if (fp->nr_maxactive) - size = fp->nr_maxactive; - else - size = num * num_possible_cpus() * 2; - if (size <= 0) - return -EINVAL; + if (match->addrs) + match->addrs[match->index] = addr; - /* Initialize rethook */ - fp->rethook = rethook_alloc((void *)fp, fprobe_exit_handler, - sizeof(struct fprobe_rethook_node), size); - if (IS_ERR(fp->rethook)) - return PTR_ERR(fp->rethook); + match->index++; + return match->index == match->size; +} - return 0; +/* + * Make IP list from the filter/no-filter glob patterns. + * Return the number of matched symbols, or -ENOENT. + */ +static int ip_list_from_filter(const char *filter, const char *notfilter, + unsigned long *addrs, size_t size) +{ + struct filter_match_data match = { .filter = filter, .notfilter = notfilter, + .index = 0, .size = size, .addrs = addrs}; + int ret; + + ret = kallsyms_on_each_symbol(filter_match_callback, &match); + if (ret < 0) + return ret; + ret = module_kallsyms_on_each_symbol(NULL, filter_match_callback, &match); + if (ret < 0) + return ret; + + return match.index ?: -ENOENT; } static void fprobe_fail_cleanup(struct fprobe *fp) { - if (!IS_ERR_OR_NULL(fp->rethook)) { - /* Don't need to cleanup rethook->handler because this is not used. */ - rethook_free(fp->rethook); - fp->rethook = NULL; + kfree(fp->hlist_array); + fp->hlist_array = NULL; +} + +/* Initialize the fprobe data structure. */ +static int fprobe_init(struct fprobe *fp, unsigned long *addrs, int num) +{ + struct fprobe_hlist *hlist_array; + unsigned long addr; + int size, i; + + if (!fp || !addrs || num <= 0) + return -EINVAL; + + size = ALIGN(fp->entry_data_size, sizeof(long)); + if (size > MAX_FPROBE_DATA_SIZE) + return -E2BIG; + fp->entry_data_size = size; + + hlist_array = kzalloc(struct_size(hlist_array, array, num), GFP_KERNEL); + if (!hlist_array) + return -ENOMEM; + + fp->nmissed = 0; + + hlist_array->size = num; + fp->hlist_array = hlist_array; + hlist_array->fp = fp; + for (i = 0; i < num; i++) { + hlist_array->array[i].fp = fp; + addr = ftrace_location(addrs[i]); + if (!addr) { + fprobe_fail_cleanup(fp); + return -ENOENT; + } + hlist_array->array[i].addr = addr; } - ftrace_free_filter(&fp->ops); + return 0; } +#define FPROBE_IPS_MAX INT_MAX + /** * register_fprobe() - Register fprobe to ftrace by pattern. * @fp: A fprobe data structure to be registered. @@ -238,46 +510,24 @@ static void fprobe_fail_cleanup(struct fprobe *fp) */ int register_fprobe(struct fprobe *fp, const char *filter, const char *notfilter) { - struct ftrace_hash *hash; - unsigned char *str; - int ret, len; + unsigned long *addrs; + int ret; if (!fp || !filter) return -EINVAL; - fprobe_init(fp); - - len = strlen(filter); - str = kstrdup(filter, GFP_KERNEL); - ret = ftrace_set_filter(&fp->ops, str, len, 0); - kfree(str); - if (ret) + ret = ip_list_from_filter(filter, notfilter, NULL, FPROBE_IPS_MAX); + if (ret < 0) return ret; - if (notfilter) { - len = strlen(notfilter); - str = kstrdup(notfilter, GFP_KERNEL); - ret = ftrace_set_notrace(&fp->ops, str, len, 0); - kfree(str); - if (ret) - goto out; - } - - /* TODO: - * correctly calculate the total number of filtered symbols - * from both filter and notfilter. - */ - hash = rcu_access_pointer(fp->ops.local_hash.filter_hash); - if (WARN_ON_ONCE(!hash)) - goto out; - - ret = fprobe_init_rethook(fp, (int)hash->count); - if (!ret) - ret = register_ftrace_function(&fp->ops); + addrs = kcalloc(ret, sizeof(unsigned long), GFP_KERNEL); + if (!addrs) + return -ENOMEM; + ret = ip_list_from_filter(filter, notfilter, addrs, ret); + if (ret > 0) + ret = register_fprobe_ips(fp, addrs, ret); -out: - if (ret) - fprobe_fail_cleanup(fp); + kfree(addrs); return ret; } EXPORT_SYMBOL_GPL(register_fprobe); @@ -285,7 +535,7 @@ EXPORT_SYMBOL_GPL(register_fprobe); /** * register_fprobe_ips() - Register fprobe to ftrace by address. * @fp: A fprobe data structure to be registered. - * @addrs: An array of target ftrace location addresses. + * @addrs: An array of target function address. * @num: The number of entries of @addrs. * * Register @fp to ftrace for enabling the probe on the address given by @addrs. @@ -297,23 +547,27 @@ EXPORT_SYMBOL_GPL(register_fprobe); */ int register_fprobe_ips(struct fprobe *fp, unsigned long *addrs, int num) { - int ret; - - if (!fp || !addrs || num <= 0) - return -EINVAL; - - fprobe_init(fp); + struct fprobe_hlist *hlist_array; + int ret, i; - ret = ftrace_set_filter_ips(&fp->ops, addrs, num, 0, 0); + ret = fprobe_init(fp, addrs, num); if (ret) return ret; - ret = fprobe_init_rethook(fp, num); - if (!ret) - ret = register_ftrace_function(&fp->ops); + mutex_lock(&fprobe_mutex); + + hlist_array = fp->hlist_array; + ret = fprobe_graph_add_ips(addrs, num); + if (!ret) { + add_fprobe_hash(fp); + for (i = 0; i < hlist_array->size; i++) + insert_fprobe_node(&hlist_array->array[i]); + } + mutex_unlock(&fprobe_mutex); if (ret) fprobe_fail_cleanup(fp); + return ret; } EXPORT_SYMBOL_GPL(register_fprobe_ips); @@ -351,14 +605,13 @@ EXPORT_SYMBOL_GPL(register_fprobe_syms); bool fprobe_is_registered(struct fprobe *fp) { - if (!fp || (fp->ops.saved_func != fprobe_handler && - fp->ops.saved_func != fprobe_kprobe_handler)) + if (!fp || !fp->hlist_array) return false; return true; } /** - * unregister_fprobe() - Unregister fprobe from ftrace + * unregister_fprobe() - Unregister fprobe. * @fp: A fprobe data structure to be unregistered. * * Unregister fprobe (and remove ftrace hooks from the function entries). @@ -367,23 +620,40 @@ bool fprobe_is_registered(struct fprobe *fp) */ int unregister_fprobe(struct fprobe *fp) { - int ret; + struct fprobe_hlist *hlist_array; + unsigned long *addrs = NULL; + int ret = 0, i, count; - if (!fprobe_is_registered(fp)) - return -EINVAL; + mutex_lock(&fprobe_mutex); + if (!fp || !is_fprobe_still_exist(fp)) { + ret = -EINVAL; + goto out; + } - if (!IS_ERR_OR_NULL(fp->rethook)) - rethook_stop(fp->rethook); + hlist_array = fp->hlist_array; + addrs = kcalloc(hlist_array->size, sizeof(unsigned long), GFP_KERNEL); + if (!addrs) { + ret = -ENOMEM; /* TODO: Fallback to one-by-one loop */ + goto out; + } - ret = unregister_ftrace_function(&fp->ops); - if (ret < 0) - return ret; + /* Remove non-synonim ips from table and hash */ + count = 0; + for (i = 0; i < hlist_array->size; i++) { + if (!delete_fprobe_node(&hlist_array->array[i])) + addrs[count++] = hlist_array->array[i].addr; + } + del_fprobe_hash(fp); - if (!IS_ERR_OR_NULL(fp->rethook)) - rethook_free(fp->rethook); + fprobe_graph_remove_ips(addrs, count); - ftrace_free_filter(&fp->ops); + kfree_rcu(hlist_array, rcu); + fp->hlist_array = NULL; +out: + mutex_unlock(&fprobe_mutex); + + kfree(addrs); return ret; } EXPORT_SYMBOL_GPL(unregister_fprobe); diff --git a/lib/test_fprobe.c b/lib/test_fprobe.c index 271ce0caeec0..cf92111b5c79 100644 --- a/lib/test_fprobe.c +++ b/lib/test_fprobe.c @@ -17,10 +17,8 @@ static u32 rand1, entry_val, exit_val; /* Use indirect calls to avoid inlining the target functions */ static u32 (*target)(u32 value); static u32 (*target2)(u32 value); -static u32 (*target_nest)(u32 value, u32 (*nest)(u32)); static unsigned long target_ip; static unsigned long target2_ip; -static unsigned long target_nest_ip; static int entry_return_value; static noinline u32 fprobe_selftest_target(u32 value) @@ -33,11 +31,6 @@ static noinline u32 fprobe_selftest_target2(u32 value) return (value / div_factor) + 1; } -static noinline u32 fprobe_selftest_nest_target(u32 value, u32 (*nest)(u32)) -{ - return nest(value + 2); -} - static notrace int fp_entry_handler(struct fprobe *fp, unsigned long ip, unsigned long ret_ip, struct ftrace_regs *fregs, void *data) @@ -79,22 +72,6 @@ static notrace void fp_exit_handler(struct fprobe *fp, unsigned long ip, KUNIT_EXPECT_NULL(current_test, data); } -static notrace int nest_entry_handler(struct fprobe *fp, unsigned long ip, - unsigned long ret_ip, - struct ftrace_regs *fregs, void *data) -{ - KUNIT_EXPECT_FALSE(current_test, preemptible()); - return 0; -} - -static notrace void nest_exit_handler(struct fprobe *fp, unsigned long ip, - unsigned long ret_ip, - struct ftrace_regs *fregs, void *data) -{ - KUNIT_EXPECT_FALSE(current_test, preemptible()); - KUNIT_EXPECT_EQ(current_test, ip, target_nest_ip); -} - /* Test entry only (no rethook) */ static void test_fprobe_entry(struct kunit *test) { @@ -191,25 +168,6 @@ static void test_fprobe_data(struct kunit *test) KUNIT_EXPECT_EQ(test, 0, unregister_fprobe(&fp)); } -/* Test nr_maxactive */ -static void test_fprobe_nest(struct kunit *test) -{ - static const char *syms[] = {"fprobe_selftest_target", "fprobe_selftest_nest_target"}; - struct fprobe fp = { - .entry_handler = nest_entry_handler, - .exit_handler = nest_exit_handler, - .nr_maxactive = 1, - }; - - current_test = test; - KUNIT_EXPECT_EQ(test, 0, register_fprobe_syms(&fp, syms, 2)); - - target_nest(rand1, target); - KUNIT_EXPECT_EQ(test, 1, fp.nmissed); - - KUNIT_EXPECT_EQ(test, 0, unregister_fprobe(&fp)); -} - static void test_fprobe_skip(struct kunit *test) { struct fprobe fp = { @@ -247,10 +205,8 @@ static int fprobe_test_init(struct kunit *test) rand1 = get_random_u32_above(div_factor); target = fprobe_selftest_target; target2 = fprobe_selftest_target2; - target_nest = fprobe_selftest_nest_target; target_ip = get_ftrace_location(target); target2_ip = get_ftrace_location(target2); - target_nest_ip = get_ftrace_location(target_nest); return 0; } @@ -260,7 +216,6 @@ static struct kunit_case fprobe_testcases[] = { KUNIT_CASE(test_fprobe), KUNIT_CASE(test_fprobe_syms), KUNIT_CASE(test_fprobe_data), - KUNIT_CASE(test_fprobe_nest), KUNIT_CASE(test_fprobe_skip), {} }; From patchwork Fri Dec 8 10:30:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 175757 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5364676vqy; Fri, 8 Dec 2023 02:31:41 -0800 (PST) X-Google-Smtp-Source: AGHT+IFMmnp8ldvnnTlKdKPBpjFrYwm4SeoDI8IOb8gwvSi0iHjYo1EQf2pmnCnHAxn3wH03ax0U X-Received: by 2002:a05:6e02:485:b0:35d:59a2:2e1 with SMTP id b5-20020a056e02048500b0035d59a202e1mr4164078ils.129.1702031501002; Fri, 08 Dec 2023 02:31:41 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702031500; cv=none; d=google.com; s=arc-20160816; b=nvBLbfLATUkZJIPNX9pq26OufSkd2CZholoMe+5KO9BKR0QlkVz/4bv+0VGIZxQFmu c6HCD7hvIReAAubAy87beBV7nGBft2MhsxniIBoCAwuMnKn0kWoeJfk+btTzYCP5ZYjP 3rUGJV4e5SW3LsjoW1/DYjLTqxgCkGPYZWC37/3VQGY9CXzp5DzyTJShY6qj9rpe8Cma UnBoxFwP9nYHR9xA+atdQD2chtq50hpvp8IuR1NJMjz1Q/bOJjhhHxlsys+D5IvKQAJS xmiWuZwteMQd4JWzX6s7FQqtBw6HbJ9n9G/A2Pn3QsoCdgGDRpv4heEkcavWz3VKvJe3 rDCA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=TfMRFkzBB7vh9il89fw23mjYo+w5nH5084NawZeD+xo=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=mhZdtwLJ+5cLZy/CVoK703jpPxMpxYblgYb587fIc/TXUl1uMi6jfkMErXZv9nGIjS tHV/lLNk2T63bwO99FqtIHXlw9LKJ6TzkX3k3U+ls0rKQV4i8bK8Z0+as3P23ISMcez7 Lw6ZV4nFqbKDYGPmI9FOcFm7Z90MFZvY1qxu3xbiWBROJqihAGHF9i45pELuTp/FiDNb EYppRMgNRTIXBTP17J3/Xjpdt96WvFVta5CftGgCyMol8PLxsbvC1cQySbtPaXXtwNsK KYD0j6a1ZUIyhFoUqclhW3KRzxFd8JrCRI7hauNz2XNDEGaubwsQlknvIJ1B0ORNoyGV N5GA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=N84DsEjf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from morse.vger.email (morse.vger.email. [2620:137:e000::3:1]) by mx.google.com with ESMTPS id bv129-20020a632e87000000b0056a36ac322fsi1273845pgb.514.2023.12.08.02.31.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 02:31:40 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=N84DsEjf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 4B43A8374BA6; Fri, 8 Dec 2023 02:31:38 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1573607AbjLHKbJ (ORCPT + 99 others); Fri, 8 Dec 2023 05:31:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40234 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1573533AbjLHKak (ORCPT ); Fri, 8 Dec 2023 05:30:40 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D72751703 for ; Fri, 8 Dec 2023 02:30:14 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8EB17C433C7; Fri, 8 Dec 2023 10:30:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702031414; bh=VVauZgXIlWYIcHDkYwErITV9yot6GwNuKuwbq9iJmVQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=N84DsEjfrrBzs3l6Oi0bpHtxyHQtNf4qlTHzhhtRsboAvSsJmaqL7T0VEaA/2laCl UTuuLLpNKIm6tJkAriMKZwKIdaf3C8yCYeTPaWSvRhwMd++3CTo23rAtE3DXhJIiu9 h2ZGUcPJZY92ijdmvkAyVOmMhDnfyRL6sMPTm/6ireSVOZeE8D+cswuw/GzVDQGQHN kWyKyea1Zv7YfgeQuyIb4PvB8MTiZhya+X0RpOPfR2pR1wieYcV31XEILTz5UjH/Fc QCFCJw4sFRb1ErM0xU7epMM2Tlu3mhjzHBnzjrk0NjaPE6UWsOLtN9npU/sTnMns2O ThXOO/Y2zfakQ== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v4 29/33] tracing/fprobe: Remove nr_maxactive from fprobe Date: Fri, 8 Dec 2023 19:30:07 +0900 Message-Id: <170203140741.579004.12565529293650029395.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170203105427.579004.8033550792660734570.stgit@devnote2> References: <170203105427.579004.8033550792660734570.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Fri, 08 Dec 2023 02:31:38 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784709383155011873 X-GMAIL-MSGID: 1784709383155011873 From: Masami Hiramatsu (Google) Remove depercated fprobe::nr_maxactive. This involves fprobe events to rejects the maxactive number. Signed-off-by: Masami Hiramatsu (Google) --- Changes in v2: - Newly added. --- include/linux/fprobe.h | 2 -- kernel/trace/trace_fprobe.c | 44 ++++++------------------------------------- 2 files changed, 6 insertions(+), 40 deletions(-) diff --git a/include/linux/fprobe.h b/include/linux/fprobe.h index 08b37b0d1d05..c28d06ddfb8e 100644 --- a/include/linux/fprobe.h +++ b/include/linux/fprobe.h @@ -47,7 +47,6 @@ struct fprobe_hlist { * @nmissed: The counter for missing events. * @flags: The status flag. * @entry_data_size: The private data storage size. - * @nr_maxactive: The max number of active functions. (*deprecated) * @entry_handler: The callback function for function entry. * @exit_handler: The callback function for function exit. * @hlist_array: The fprobe_hlist for fprobe search from IP hash table. @@ -56,7 +55,6 @@ struct fprobe { unsigned long nmissed; unsigned int flags; size_t entry_data_size; - int nr_maxactive; int (*entry_handler)(struct fprobe *fp, unsigned long entry_ip, unsigned long ret_ip, struct ftrace_regs *regs, diff --git a/kernel/trace/trace_fprobe.c b/kernel/trace/trace_fprobe.c index 3982626c82e6..22cf51217b81 100644 --- a/kernel/trace/trace_fprobe.c +++ b/kernel/trace/trace_fprobe.c @@ -375,7 +375,6 @@ static struct trace_fprobe *alloc_trace_fprobe(const char *group, const char *event, const char *symbol, struct tracepoint *tpoint, - int maxactive, int nargs, bool is_return) { struct trace_fprobe *tf; @@ -395,7 +394,6 @@ static struct trace_fprobe *alloc_trace_fprobe(const char *group, tf->fp.entry_handler = fentry_dispatcher; tf->tpoint = tpoint; - tf->fp.nr_maxactive = maxactive; ret = trace_probe_init(&tf->tp, event, group, false); if (ret < 0) @@ -974,12 +972,11 @@ static int __trace_fprobe_create(int argc, const char *argv[]) * FETCHARG:TYPE : use TYPE instead of unsigned long. */ struct trace_fprobe *tf = NULL; - int i, len, new_argc = 0, ret = 0; + int i, new_argc = 0, ret = 0; bool is_return = false; char *symbol = NULL; const char *event = NULL, *group = FPROBE_EVENT_SYSTEM; const char **new_argv = NULL; - int maxactive = 0; char buf[MAX_EVENT_NAME_LEN]; char gbuf[MAX_EVENT_NAME_LEN]; char sbuf[KSYM_NAME_LEN]; @@ -1000,33 +997,13 @@ static int __trace_fprobe_create(int argc, const char *argv[]) trace_probe_log_init("trace_fprobe", argc, argv); - event = strchr(&argv[0][1], ':'); - if (event) - event++; - - if (isdigit(argv[0][1])) { - if (event) - len = event - &argv[0][1] - 1; - else - len = strlen(&argv[0][1]); - if (len > MAX_EVENT_NAME_LEN - 1) { - trace_probe_log_err(1, BAD_MAXACT); - goto parse_error; - } - memcpy(buf, &argv[0][1], len); - buf[len] = '\0'; - ret = kstrtouint(buf, 0, &maxactive); - if (ret || !maxactive) { + if (argv[0][1] != '\0') { + if (argv[0][1] != ':') { + trace_probe_log_set_index(0); trace_probe_log_err(1, BAD_MAXACT); goto parse_error; } - /* fprobe rethook instances are iterated over via a list. The - * maximum should stay reasonable. - */ - if (maxactive > RETHOOK_MAXACTIVE_MAX) { - trace_probe_log_err(1, MAXACT_TOO_BIG); - goto parse_error; - } + event = &argv[0][2]; } trace_probe_log_set_index(1); @@ -1036,12 +1013,6 @@ static int __trace_fprobe_create(int argc, const char *argv[]) if (ret < 0) goto parse_error; - if (!is_return && maxactive) { - trace_probe_log_set_index(0); - trace_probe_log_err(1, BAD_MAXACT_TYPE); - goto parse_error; - } - trace_probe_log_set_index(0); if (event) { ret = traceprobe_parse_event_name(&event, &group, gbuf, @@ -1095,8 +1066,7 @@ static int __trace_fprobe_create(int argc, const char *argv[]) } /* setup a probe */ - tf = alloc_trace_fprobe(group, event, symbol, tpoint, maxactive, - argc, is_return); + tf = alloc_trace_fprobe(group, event, symbol, tpoint, argc, is_return); if (IS_ERR(tf)) { ret = PTR_ERR(tf); /* This must return -ENOMEM, else there is a bug */ @@ -1172,8 +1142,6 @@ static int trace_fprobe_show(struct seq_file *m, struct dyn_event *ev) seq_putc(m, 't'); else seq_putc(m, 'f'); - if (trace_fprobe_is_return(tf) && tf->fp.nr_maxactive) - seq_printf(m, "%d", tf->fp.nr_maxactive); seq_printf(m, ":%s/%s", trace_probe_group_name(&tf->tp), trace_probe_name(&tf->tp)); From patchwork Fri Dec 8 10:30:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 175765 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5367331vqy; Fri, 8 Dec 2023 02:38:38 -0800 (PST) X-Google-Smtp-Source: AGHT+IHQR4iGUcFykLU/3L/AfLtX32X06uOjMJ08aU0Yv4szn3QzdT9Hlq6IDwdr0in+ZVtWNOcF X-Received: by 2002:a17:902:8491:b0:1d0:6ffd:f224 with SMTP id c17-20020a170902849100b001d06ffdf224mr3135447plo.122.1702031917868; Fri, 08 Dec 2023 02:38:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702031917; cv=none; d=google.com; s=arc-20160816; b=wjBIhg2Ob747866hqbGuL/7SpZ3h6QOAX1bWyaVpzZrCtmDXJVUXHQq6/GcCPmnrdS 3vntlHBwRdvBngch94OX1mHFFbBkNJma2M0auW8JwAWjahMBMwC5tO8WrFfRr6ucr+f8 dHQdgaQKbuC3Jsmpqss4hmovIrpBL3L4y7LQ57SzHOFIJFaezeei/hirK/QuRve6euez XPouBeXrfvKk+hXVc360Ctc6K6/EWXHLSjoJc0F8zI/Zq0Vj+UDA8nQvS9VUNtsDvdz7 P8afThaO+N0vLkUxwYYVglRIsuZ6OzKNkAUvWtFoWhkS7sePZBq+i+QTb9KKr/UDV4pq rhvA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=dK+uN1oqrOBiEn0XHVEujMNGy4oyAVuAKVLv2qWEM7I=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=P4n962J7NfG6ym3dB/T7c2MzVMmJwvXHKt5gMPsmjQMJmOuna5H/H6AHipQPpmZSI5 u2F0lks35Yyq5ut+IsfuEGfTSHQKS/krKfDZRnxKl/o+DHn/G5hRPyPLc1DCgUhrngtR J7oVNszkalmJ5cesIxq5lpdBRYB0iu86xPfae2TDXlBVqNOPTjF7QTMuv5DUT8IKWza8 BgXVzE5l5MW7F6DoDI1MbLslxeJ3+v5wCqzD7ncgN1m9ZGgJ/W//Gu6FJ4DNGOZjmC4C dMhnCifMPX2TzaoOzCWaLA0nLg1Xc5lAixJguGvJQ0mTkxndg0rRJO9arfQvFT4GKHf5 M9Tg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=gO7ms5ah; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from groat.vger.email (groat.vger.email. [2620:137:e000::3:5]) by mx.google.com with ESMTPS id x17-20020a170902ea9100b001d04c097d28si1365878plb.518.2023.12.08.02.38.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 02:38:37 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) client-ip=2620:137:e000::3:5; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=gO7ms5ah; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 75E5E82372C7; Fri, 8 Dec 2023 02:38:30 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1573738AbjLHKiQ (ORCPT + 99 others); Fri, 8 Dec 2023 05:38:16 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39806 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1573667AbjLHKiI (ORCPT ); Fri, 8 Dec 2023 05:38:08 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1B1C7213B for ; Fri, 8 Dec 2023 02:30:27 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D4E75C433C7; Fri, 8 Dec 2023 10:30:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702031426; bh=E6nu5gsbftq0MW8we9W5Y+vMqoODRszc8WZZUgWBgbk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gO7ms5ahGRlRQun4j8LDWsK5RJkD8ioWRysOiRukBnoxwH6F5gCM66iuCL+g1j+h3 DZTcp2r9YtjkP44gk1YNMa7WL2qC1KXg82dTqadjY9NVvWku3r41XhIMdbzfbC1TGl dHbAVcnMvnj33Mf9b29oo6JihCkYmiqBeMvD08g+7zdR5eWayVgNfUUUxMiupm6q7Y 9Mmh2NdDPX/11M/V4jYUp+L/WYCzKsItB30cb3ax8B/QxycOBW4IK5/ITIEUvRr42C 43xZEubqx3LVM60BzbJ7KEocR5nZhGs093sIHGtFemHNf9y3OgeKUIEDvx+yepqHH2 kOAexL9qbLYyg== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v4 30/33] tracing/fprobe: Enable fprobe events with CONFIG_DYNAMIC_FTRACE_WITH_ARGS Date: Fri, 8 Dec 2023 19:30:20 +0900 Message-Id: <170203141980.579004.5047004830423702367.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170203105427.579004.8033550792660734570.stgit@devnote2> References: <170203105427.579004.8033550792660734570.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Fri, 08 Dec 2023 02:38:30 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784709820431866793 X-GMAIL-MSGID: 1784709820431866793 From: Masami Hiramatsu (Google) Allow fprobe events to be enabled with CONFIG_DYNAMIC_FTRACE_WITH_ARGS. With this change, fprobe events mostly use ftrace_regs instead of pt_regs. Note that if the arch doesn't enable HAVE_PT_REGS_COMPAT_FTRACE_REGS, fprobe events will not be able to be used from perf. Signed-off-by: Masami Hiramatsu (Google) --- Chagnes in v3: - Use ftrace_regs_get_return_value(). Changes in v2: - Define ftrace_regs_get_kernel_stack_nth() for !CONFIG_HAVE_REGS_AND_STACK_ACCESS_API. Changes from previous series: Update against the new series. --- include/linux/ftrace.h | 17 +++++++++ kernel/trace/Kconfig | 1 - kernel/trace/trace_fprobe.c | 74 ++++++++++++++++++++------------------- kernel/trace/trace_probe_tmpl.h | 2 + 4 files changed, 55 insertions(+), 39 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 8150edcf8496..ad28daa507f7 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -250,6 +250,23 @@ static __always_inline bool ftrace_regs_has_args(struct ftrace_regs *fregs) regs_query_register_offset(name) #endif +#ifdef CONFIG_HAVE_REGS_AND_STACK_ACCESS_API +static __always_inline unsigned long +ftrace_regs_get_kernel_stack_nth(struct ftrace_regs *fregs, unsigned int nth) +{ + unsigned long *stackp; + + stackp = (unsigned long *)ftrace_regs_get_stack_pointer(fregs); + if (((unsigned long)(stackp + nth) & ~(THREAD_SIZE - 1)) == + ((unsigned long)stackp & ~(THREAD_SIZE - 1))) + return *(stackp + nth); + + return 0; +} +#else /* !CONFIG_HAVE_REGS_AND_STACK_ACCESS_API */ +#define ftrace_regs_get_kernel_stack_nth(fregs, nth) (0L) +#endif /* CONFIG_HAVE_REGS_AND_STACK_ACCESS_API */ + typedef void (*ftrace_func_t)(unsigned long ip, unsigned long parent_ip, struct ftrace_ops *op, struct ftrace_regs *fregs); diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig index 11a96275b68c..169588021d90 100644 --- a/kernel/trace/Kconfig +++ b/kernel/trace/Kconfig @@ -681,7 +681,6 @@ config FPROBE_EVENTS select TRACING select PROBE_EVENTS select DYNAMIC_EVENTS - depends on DYNAMIC_FTRACE_WITH_REGS default y help This allows user to add tracing events on the function entry and diff --git a/kernel/trace/trace_fprobe.c b/kernel/trace/trace_fprobe.c index 22cf51217b81..d96de0dbc0cb 100644 --- a/kernel/trace/trace_fprobe.c +++ b/kernel/trace/trace_fprobe.c @@ -132,7 +132,7 @@ static int process_fetch_insn(struct fetch_insn *code, void *rec, void *dest, void *base) { - struct pt_regs *regs = rec; + struct ftrace_regs *fregs = rec; unsigned long val; int ret; @@ -140,17 +140,17 @@ process_fetch_insn(struct fetch_insn *code, void *rec, void *dest, /* 1st stage: get value from context */ switch (code->op) { case FETCH_OP_STACK: - val = regs_get_kernel_stack_nth(regs, code->param); + val = ftrace_regs_get_kernel_stack_nth(fregs, code->param); break; case FETCH_OP_STACKP: - val = kernel_stack_pointer(regs); + val = ftrace_regs_get_stack_pointer(fregs); break; case FETCH_OP_RETVAL: - val = regs_return_value(regs); + val = ftrace_regs_get_return_value(fregs); break; #ifdef CONFIG_HAVE_FUNCTION_ARG_ACCESS_API case FETCH_OP_ARG: - val = regs_get_kernel_argument(regs, code->param); + val = ftrace_regs_get_argument(fregs, code->param); break; #endif case FETCH_NOP_SYMBOL: /* Ignore a place holder */ @@ -170,7 +170,7 @@ NOKPROBE_SYMBOL(process_fetch_insn) /* function entry handler */ static nokprobe_inline void __fentry_trace_func(struct trace_fprobe *tf, unsigned long entry_ip, - struct pt_regs *regs, + struct ftrace_regs *fregs, struct trace_event_file *trace_file) { struct fentry_trace_entry_head *entry; @@ -184,36 +184,36 @@ __fentry_trace_func(struct trace_fprobe *tf, unsigned long entry_ip, if (trace_trigger_soft_disabled(trace_file)) return; - dsize = __get_data_size(&tf->tp, regs); + dsize = __get_data_size(&tf->tp, fregs); entry = trace_event_buffer_reserve(&fbuffer, trace_file, sizeof(*entry) + tf->tp.size + dsize); if (!entry) return; - fbuffer.regs = regs; + fbuffer.regs = ftrace_get_regs(fregs); entry = fbuffer.entry = ring_buffer_event_data(fbuffer.event); entry->ip = entry_ip; - store_trace_args(&entry[1], &tf->tp, regs, sizeof(*entry), dsize); + store_trace_args(&entry[1], &tf->tp, fregs, sizeof(*entry), dsize); trace_event_buffer_commit(&fbuffer); } static void fentry_trace_func(struct trace_fprobe *tf, unsigned long entry_ip, - struct pt_regs *regs) + struct ftrace_regs *fregs) { struct event_file_link *link; trace_probe_for_each_link_rcu(link, &tf->tp) - __fentry_trace_func(tf, entry_ip, regs, link->file); + __fentry_trace_func(tf, entry_ip, fregs, link->file); } NOKPROBE_SYMBOL(fentry_trace_func); /* Kretprobe handler */ static nokprobe_inline void __fexit_trace_func(struct trace_fprobe *tf, unsigned long entry_ip, - unsigned long ret_ip, struct pt_regs *regs, + unsigned long ret_ip, struct ftrace_regs *fregs, struct trace_event_file *trace_file) { struct fexit_trace_entry_head *entry; @@ -227,60 +227,63 @@ __fexit_trace_func(struct trace_fprobe *tf, unsigned long entry_ip, if (trace_trigger_soft_disabled(trace_file)) return; - dsize = __get_data_size(&tf->tp, regs); + dsize = __get_data_size(&tf->tp, fregs); entry = trace_event_buffer_reserve(&fbuffer, trace_file, sizeof(*entry) + tf->tp.size + dsize); if (!entry) return; - fbuffer.regs = regs; + fbuffer.regs = ftrace_get_regs(fregs); entry = fbuffer.entry = ring_buffer_event_data(fbuffer.event); entry->func = entry_ip; entry->ret_ip = ret_ip; - store_trace_args(&entry[1], &tf->tp, regs, sizeof(*entry), dsize); + store_trace_args(&entry[1], &tf->tp, fregs, sizeof(*entry), dsize); trace_event_buffer_commit(&fbuffer); } static void fexit_trace_func(struct trace_fprobe *tf, unsigned long entry_ip, - unsigned long ret_ip, struct pt_regs *regs) + unsigned long ret_ip, struct ftrace_regs *fregs) { struct event_file_link *link; trace_probe_for_each_link_rcu(link, &tf->tp) - __fexit_trace_func(tf, entry_ip, ret_ip, regs, link->file); + __fexit_trace_func(tf, entry_ip, ret_ip, fregs, link->file); } NOKPROBE_SYMBOL(fexit_trace_func); #ifdef CONFIG_PERF_EVENTS static int fentry_perf_func(struct trace_fprobe *tf, unsigned long entry_ip, - struct pt_regs *regs) + struct ftrace_regs *fregs) { struct trace_event_call *call = trace_probe_event_call(&tf->tp); struct fentry_trace_entry_head *entry; struct hlist_head *head; int size, __size, dsize; + struct pt_regs *regs; int rctx; head = this_cpu_ptr(call->perf_events); if (hlist_empty(head)) return 0; - dsize = __get_data_size(&tf->tp, regs); + dsize = __get_data_size(&tf->tp, fregs); __size = sizeof(*entry) + tf->tp.size + dsize; size = ALIGN(__size + sizeof(u32), sizeof(u64)); size -= sizeof(u32); - entry = perf_trace_buf_alloc(size, NULL, &rctx); + entry = perf_trace_buf_alloc(size, ®s, &rctx); if (!entry) return 0; + regs = ftrace_fill_perf_regs(fregs, regs); + entry->ip = entry_ip; memset(&entry[1], 0, dsize); - store_trace_args(&entry[1], &tf->tp, regs, sizeof(*entry), dsize); + store_trace_args(&entry[1], &tf->tp, fregs, sizeof(*entry), dsize); perf_trace_buf_submit(entry, size, rctx, call->event.type, 1, regs, head, NULL); return 0; @@ -289,30 +292,33 @@ NOKPROBE_SYMBOL(fentry_perf_func); static void fexit_perf_func(struct trace_fprobe *tf, unsigned long entry_ip, - unsigned long ret_ip, struct pt_regs *regs) + unsigned long ret_ip, struct ftrace_regs *fregs) { struct trace_event_call *call = trace_probe_event_call(&tf->tp); struct fexit_trace_entry_head *entry; struct hlist_head *head; int size, __size, dsize; + struct pt_regs *regs; int rctx; head = this_cpu_ptr(call->perf_events); if (hlist_empty(head)) return; - dsize = __get_data_size(&tf->tp, regs); + dsize = __get_data_size(&tf->tp, fregs); __size = sizeof(*entry) + tf->tp.size + dsize; size = ALIGN(__size + sizeof(u32), sizeof(u64)); size -= sizeof(u32); - entry = perf_trace_buf_alloc(size, NULL, &rctx); + entry = perf_trace_buf_alloc(size, ®s, &rctx); if (!entry) return; + regs = ftrace_fill_perf_regs(fregs, regs); + entry->func = entry_ip; entry->ret_ip = ret_ip; - store_trace_args(&entry[1], &tf->tp, regs, sizeof(*entry), dsize); + store_trace_args(&entry[1], &tf->tp, fregs, sizeof(*entry), dsize); perf_trace_buf_submit(entry, size, rctx, call->event.type, 1, regs, head, NULL); } @@ -324,17 +330,14 @@ static int fentry_dispatcher(struct fprobe *fp, unsigned long entry_ip, void *entry_data) { struct trace_fprobe *tf = container_of(fp, struct trace_fprobe, fp); - struct pt_regs *regs = ftrace_get_regs(fregs); int ret = 0; - if (!regs) - return 0; - if (trace_probe_test_flag(&tf->tp, TP_FLAG_TRACE)) - fentry_trace_func(tf, entry_ip, regs); + fentry_trace_func(tf, entry_ip, fregs); + #ifdef CONFIG_PERF_EVENTS if (trace_probe_test_flag(&tf->tp, TP_FLAG_PROFILE)) - ret = fentry_perf_func(tf, entry_ip, regs); + ret = fentry_perf_func(tf, entry_ip, fregs); #endif return ret; } @@ -345,16 +348,13 @@ static void fexit_dispatcher(struct fprobe *fp, unsigned long entry_ip, void *entry_data) { struct trace_fprobe *tf = container_of(fp, struct trace_fprobe, fp); - struct pt_regs *regs = ftrace_get_regs(fregs); - - if (!regs) - return; if (trace_probe_test_flag(&tf->tp, TP_FLAG_TRACE)) - fexit_trace_func(tf, entry_ip, ret_ip, regs); + fexit_trace_func(tf, entry_ip, ret_ip, fregs); + #ifdef CONFIG_PERF_EVENTS if (trace_probe_test_flag(&tf->tp, TP_FLAG_PROFILE)) - fexit_perf_func(tf, entry_ip, ret_ip, regs); + fexit_perf_func(tf, entry_ip, ret_ip, fregs); #endif } NOKPROBE_SYMBOL(fexit_dispatcher); diff --git a/kernel/trace/trace_probe_tmpl.h b/kernel/trace/trace_probe_tmpl.h index 3935b347f874..05445a745a07 100644 --- a/kernel/trace/trace_probe_tmpl.h +++ b/kernel/trace/trace_probe_tmpl.h @@ -232,7 +232,7 @@ process_fetch_insn_bottom(struct fetch_insn *code, unsigned long val, /* Sum up total data length for dynamic arrays (strings) */ static nokprobe_inline int -__get_data_size(struct trace_probe *tp, struct pt_regs *regs) +__get_data_size(struct trace_probe *tp, void *regs) { struct probe_arg *arg; int i, len, ret = 0; From patchwork Fri Dec 8 10:30:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 175766 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5367491vqy; Fri, 8 Dec 2023 02:39:06 -0800 (PST) X-Google-Smtp-Source: AGHT+IHnAcyMuxaIAi8X1dEAUJtLuUBy1rfeuqweumERyGlcjEie8EfEkl8LtzyjKIhxHpyFIYV3 X-Received: by 2002:a17:902:ea03:b0:1d0:6ffd:9df4 with SMTP id s3-20020a170902ea0300b001d06ffd9df4mr5067890plg.70.1702031946313; Fri, 08 Dec 2023 02:39:06 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702031946; cv=none; d=google.com; s=arc-20160816; b=uHu5ZP0kqDZdHIe9YOSYfYZkxGZFiQ6TevFeURpBtQpghATymV1LfLGcW2la8ky6xX nbSIoEk4fdHvVXqPzxvJxIaDZIgnZ/Kxk/oDuoVuorsSOCPEH/dA4MqBNDdGzvLdmo/w iCcTBo/Uih8sGvAsaJ2ZJjAjGLPiNoT8w/dFVitw63OoxamqX/x+UNUjsPNylMZ5Xrif xHEUaw4csEtCUkkqU/QeXZKYnz2+CcFs/oOZGTqGDBTihdWvmY8Iaa870qeW92+Spvhh RH2KMWrWlxHzOjag9OAL/n5hzaAqEOW3ziVtLvVh6BabQxKi1fI6/E5AU3bpp/pW8RY8 zORA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=B+IimFdLV6G8hCJEMsVcQeurP5etXlmAnEIlOhkzF3g=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=UMR0zwzbsm1vjcmFh93hZCYi4ReaGq4Q/BzQ9qdYiq8u/Od1IfI1eSucoyFGYpGgTm 80qjSV4+PN31sqaTjGkQgxO4Sm1ttvX4Q3pt2dbG12KXqHOnUl9D/hmBLgwSAyxcgLdB DZDpSqas0JefuRI2lszNM72SsZYOObZu/Rs9PHbTHyF3AW9hQeZh9Liu2r/O0sC9ycbG xv48w0+nIlxHIpJsIODIIN6DJwDYDvMk4g2KpEyNi/Fa2P/6v4Duhi/UqgmEvvalsYTM jWLJ3PHQrPq6T1W52j+71qoplGxV5SUq9dZN6DzTMvymZJ/flOIQa2dwiBhl2mg5NOr1 jgdQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=gogPUb0m; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id d20-20020a170902729400b001cc282684c6si1348840pll.278.2023.12.08.02.39.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 02:39:06 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=gogPUb0m; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id BB6FF80F7E66; Fri, 8 Dec 2023 02:39:02 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1573670AbjLHKiv (ORCPT + 99 others); Fri, 8 Dec 2023 05:38:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39736 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1573792AbjLHKid (ORCPT ); Fri, 8 Dec 2023 05:38:33 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B46A61FFE for ; Fri, 8 Dec 2023 02:30:39 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 27A66C433C8; Fri, 8 Dec 2023 10:30:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702031439; bh=uaiQ0NlnGDpVimzHf7hc4bouD6LM9m4Fjq8sHDW6itc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gogPUb0mk7pCFhGy1iFlvqkZhP+m5OexrJM8E4/R2bkrr4FQvpNE+5GJoEQjM5DVt AouFvrTgRvLuqrH0FxpB6UGnYRkg/YpJHpSnwqNZgHhc0+WFBTRXRwPu6O1FGSUFws LKONy3N9DzVlj8tFyO04HP/KRu+JCVmSmnt0WpXOj+dIytAB/Yq0d/EQn13rXIATKO haVg9jgwE2EyEmUIXkPRD81Dtg76Rf/1FGXlwUgaBwyoQCVTNaQaPRivgYbIoppE0c px97dCdf9ajDlb06XQCD2NWJenwJlSoIsB4HdOKAQ39taEr4C0o3pjwFZ/tWJpwGJt rYWfnft5RdjVw== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v4 31/33] bpf: Enable kprobe_multi feature if CONFIG_FPROBE is enabled Date: Fri, 8 Dec 2023 19:30:32 +0900 Message-Id: <170203143209.579004.15628231153884760718.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170203105427.579004.8033550792660734570.stgit@devnote2> References: <170203105427.579004.8033550792660734570.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Fri, 08 Dec 2023 02:39:02 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784709850509568837 X-GMAIL-MSGID: 1784709850509568837 From: Masami Hiramatsu (Google) Enable kprobe_multi feature if CONFIG_FPROBE is enabled. The pt_regs is converted from ftrace_regs by ftrace_partial_regs(), thus some registers may always returns 0. But it should be enough for function entry (access arguments) and exit (access return value). Signed-off-by: Masami Hiramatsu (Google) Acked-by: Florent Revest --- Changes from previous series: NOTHING, Update against the new series. --- kernel/trace/bpf_trace.c | 22 +++++++++------------- 1 file changed, 9 insertions(+), 13 deletions(-) diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index efb792f8f2ea..24ee4e960f1d 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -2503,7 +2503,7 @@ static int __init bpf_event_init(void) fs_initcall(bpf_event_init); #endif /* CONFIG_MODULES */ -#if defined(CONFIG_FPROBE) && defined(CONFIG_DYNAMIC_FTRACE_WITH_REGS) +#ifdef CONFIG_FPROBE struct bpf_kprobe_multi_link { struct bpf_link link; struct fprobe fp; @@ -2526,6 +2526,8 @@ struct user_syms { char *buf; }; +static DEFINE_PER_CPU(struct pt_regs, bpf_kprobe_multi_pt_regs); + static int copy_user_syms(struct user_syms *us, unsigned long __user *usyms, u32 cnt) { unsigned long __user usymbol; @@ -2703,13 +2705,14 @@ static u64 bpf_kprobe_multi_entry_ip(struct bpf_run_ctx *ctx) static int kprobe_multi_link_prog_run(struct bpf_kprobe_multi_link *link, - unsigned long entry_ip, struct pt_regs *regs) + unsigned long entry_ip, struct ftrace_regs *fregs) { struct bpf_kprobe_multi_run_ctx run_ctx = { .link = link, .entry_ip = entry_ip, }; struct bpf_run_ctx *old_run_ctx; + struct pt_regs *regs; int err; if (unlikely(__this_cpu_inc_return(bpf_prog_active) != 1)) { @@ -2720,6 +2723,7 @@ kprobe_multi_link_prog_run(struct bpf_kprobe_multi_link *link, migrate_disable(); rcu_read_lock(); + regs = ftrace_partial_regs(fregs, this_cpu_ptr(&bpf_kprobe_multi_pt_regs)); old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx); err = bpf_prog_run(link->link.prog, regs); bpf_reset_run_ctx(old_run_ctx); @@ -2737,13 +2741,9 @@ kprobe_multi_link_handler(struct fprobe *fp, unsigned long fentry_ip, void *data) { struct bpf_kprobe_multi_link *link; - struct pt_regs *regs = ftrace_get_regs(fregs); - - if (!regs) - return 0; link = container_of(fp, struct bpf_kprobe_multi_link, fp); - kprobe_multi_link_prog_run(link, get_entry_ip(fentry_ip), regs); + kprobe_multi_link_prog_run(link, get_entry_ip(fentry_ip), fregs); return 0; } @@ -2753,13 +2753,9 @@ kprobe_multi_link_exit_handler(struct fprobe *fp, unsigned long fentry_ip, void *data) { struct bpf_kprobe_multi_link *link; - struct pt_regs *regs = ftrace_get_regs(fregs); - - if (!regs) - return; link = container_of(fp, struct bpf_kprobe_multi_link, fp); - kprobe_multi_link_prog_run(link, get_entry_ip(fentry_ip), regs); + kprobe_multi_link_prog_run(link, get_entry_ip(fentry_ip), fregs); } static int symbols_cmp_r(const void *a, const void *b, const void *priv) @@ -3016,7 +3012,7 @@ int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr kvfree(cookies); return err; } -#else /* !CONFIG_FPROBE || !CONFIG_DYNAMIC_FTRACE_WITH_REGS */ +#else /* !CONFIG_FPROBE */ int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog) { return -EOPNOTSUPP; From patchwork Fri Dec 8 10:30:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 175758 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5364784vqy; Fri, 8 Dec 2023 02:31:51 -0800 (PST) X-Google-Smtp-Source: AGHT+IGmOuKjDJZBSq6P7GhNc9TgUj7T02fAxVM7lLNKumdq641K7nfw3lf5SJi2uKSnUsGtuiMO X-Received: by 2002:a17:903:2683:b0:1d0:6ffd:e2d1 with SMTP id jf3-20020a170903268300b001d06ffde2d1mr3744459plb.107.1702031511573; Fri, 08 Dec 2023 02:31:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702031511; cv=none; d=google.com; s=arc-20160816; b=xKd1CIH7gwvXZJ6O7ZwpkrWoe3W+4fRc/LkRk8Cn9obbOZb0oeeGFJh1Xgj4wqAaMZ wzpBe+UkZTs/BRLJ2bgcoGY6cCYi1cDVHNYm2w3uci8dr86kyDGJWxjC/Eya82hsvhP/ TlzGhdYRM+K+0CY6NEPCJAYWAjDvcIfQFvp1ooL6loqFV8xGpnX/vLE6KagLwH1QUqW6 7xNFLzjyVzMDL0SIsIz+Ts3MEB9qqrrwY3fuHMfKgK6QL0Ki/5xmS1tcJ2PWWe7+0tDx 1RNEFvcj39oIvmceETDtVIaWKCt5GhDYNLYL4qYdjT6zZY9yuNnqtQ/QeQAIt8vztAxg 6nUg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=NRt0pRDVMb2HoTOOgHKbTFmgFt039oUBjnBj47jnGM0=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=zDw6l6aT2BqouitJsf+bLP7ShlAxHZUiVGtRjwPQpPo63huGTu7C+fT/uTTdC9bZWC O1xjYT/6GejdInCNOFpt4Kbpl6BcTBpdso42HexjE7U80TaVIuV2tvxfTqaaajyZE9jO 2WxFm3QRo5d5Hkv7rjzmSE4UNzemBpMD37cnPjkzOjJ1beP5P1GIpNNQtx5LQV6wN456 a8Xzdqx6d3SBzOuKCs4RRwTQ7y5EtWG7ORUsfLgZZh0kiyhgywsoBpjkVWeXQRfGH+vy NdpW3UkufmmdzH/sAs6oUvvg+Dq39VKzU+hxsCi2O1WAWMqupH1oDJSdsfWiQydVCdYK 18eQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=RqLPmJs6; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from fry.vger.email (fry.vger.email. [23.128.96.38]) by mx.google.com with ESMTPS id h12-20020a170902f54c00b001d0ba4101e0si1422128plf.381.2023.12.08.02.31.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 02:31:51 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) client-ip=23.128.96.38; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=RqLPmJs6; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 718E681113B3; Fri, 8 Dec 2023 02:31:43 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1573606AbjLHKbb (ORCPT + 99 others); Fri, 8 Dec 2023 05:31:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40170 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1573601AbjLHKaw (ORCPT ); Fri, 8 Dec 2023 05:30:52 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DDB301BF5 for ; Fri, 8 Dec 2023 02:30:51 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6D7A8C433C8; Fri, 8 Dec 2023 10:30:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702031451; bh=AhqKrJz5NqAxQZJTxzvS35/o9mTQRDXLSjKOCvX5P6o=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RqLPmJs6TkupePJ2oGnCaudjklWh6g++k1H9atKQBrD8KabE4wb6g0uuD33T+1eFK 0nemWBH76Y3LbfpdXMkgjEygZaAidOYJe1SG1a/7EOCNl+y+48+w86GAexcyZIyGFz matj65644y6tNsVQrLQxs4AuK4aFqBZzCjf0OB8eBE4FyKh8sogPrXEC9BZbI4StB3 9rswzhCItDzN+9rv07YxsBeZEfUNbwI4lsklJC5H2CPgwS6WdIn999nP6mdqC9Ernt yo9M+vquknu2/xp+j0C4IKIgZSePQQmHEoShiXF1pHg/oShog2UZq3BVxVq3SWz1qK LfpQzJ1P2/BLA== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v4 32/33] selftests: ftrace: Remove obsolate maxactive syntax check Date: Fri, 8 Dec 2023 19:30:44 +0900 Message-Id: <170203144437.579004.4423247283109714307.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170203105427.579004.8033550792660734570.stgit@devnote2> References: <170203105427.579004.8033550792660734570.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Fri, 08 Dec 2023 02:31:43 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784709393980879212 X-GMAIL-MSGID: 1784709393980879212 From: Masami Hiramatsu (Google) Since the fprobe event does not support maxactive anymore, stop testing the maxactive syntax error checking. Signed-off-by: Masami Hiramatsu (Google) --- .../ftrace/test.d/dynevent/fprobe_syntax_errors.tc | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/tools/testing/selftests/ftrace/test.d/dynevent/fprobe_syntax_errors.tc b/tools/testing/selftests/ftrace/test.d/dynevent/fprobe_syntax_errors.tc index 20e42c030095..66516073ff27 100644 --- a/tools/testing/selftests/ftrace/test.d/dynevent/fprobe_syntax_errors.tc +++ b/tools/testing/selftests/ftrace/test.d/dynevent/fprobe_syntax_errors.tc @@ -16,9 +16,7 @@ aarch64) REG=%r0 ;; esac -check_error 'f^100 vfs_read' # MAXACT_NO_KPROBE -check_error 'f^1a111 vfs_read' # BAD_MAXACT -check_error 'f^100000 vfs_read' # MAXACT_TOO_BIG +check_error 'f^100 vfs_read' # BAD_MAXACT check_error 'f ^non_exist_func' # BAD_PROBE_ADDR (enoent) check_error 'f ^vfs_read+10' # BAD_PROBE_ADDR From patchwork Fri Dec 8 10:30:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 175760 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5364995vqy; Fri, 8 Dec 2023 02:32:19 -0800 (PST) X-Google-Smtp-Source: AGHT+IGK9bRk6xl6KzgwX3EMI3Mjd+vLgnuhAWZbpuh2nHm5Fq03ghsaONt6jf8Tw5qimXToQJ9C X-Received: by 2002:a17:90a:ce03:b0:286:6cc1:7819 with SMTP id f3-20020a17090ace0300b002866cc17819mr4042031pju.92.1702031538867; Fri, 08 Dec 2023 02:32:18 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702031538; cv=none; d=google.com; s=arc-20160816; b=PRgiHe417Rn7eghJSzDm+o91kb9X0tRBUDCW2qhgovSyvUP8Ab10GNE48nrLsyZn+V aKV/XyN1ofpycIAZ3axKb6kM38vqmDj1epCBAujmKnvmcKgIRzASRkTR3Wxfajr/MRdZ NgAh79eG3/V70Xayw6f+OJVn8ToXWOsGHvJe+0zI0teCSQjktSRo8SbjMIzRkGMXrpEC G9hu4c68RgX+NADjfUhZ32553jD3OVC1+neLwXyxpAMp0D9ram2kNc0/Yjg/TjAfuYha SCSsmvNGC23MuL2g+DXwxt+7lyI5qZiUJ9zHpmaf+IO/amWOoHbmEjQhhxC+KL9FUqZ2 Qj5Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=g/RVfnmZcQdfRZB+YRyiXpHH94CCdK1xBgexCLFpl6s=; fh=SIgps5XdV0XNwjZfT2uAI7g3mrspDldK9Qs8qQAfoa4=; b=0yQyvQuyULeJ+d5e7aMfsLcqRZOhshZ0nro1ruTEoiMiP5Nr9oR8b6D13BCs46s1tW pU/yLbt4V0+IofzSTgWev1h6T4ZtYw4c8mWT5p26Zyu4Pww4pwYOKb0BAFCKIYW0BCCN X41Iaoqx06gwZEImvRHaa4kuqD/xnfBti12xYMAMSqMHEwJfnEyECwlGv5Q8tG58JjQR yMawLh4dRMmamLK2z8MyLivZ02qSOnT1zoRsaVEy7o9E5kly6Hf5hCLox6yIgno+YR0/ 0sZVi/YHCRJHJ8aSPP0fMpQZtjfYT8hZb7l5E4G57M3v5cBHrDDQCani3MojMIazWVx+ 0xkw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=DWhZa3EO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from fry.vger.email (fry.vger.email. [2620:137:e000::3:8]) by mx.google.com with ESMTPS id a20-20020a17090ad81400b00285a910661esi2770516pjv.10.2023.12.08.02.32.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 02:32:18 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=DWhZa3EO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 3B82D81113B6; Fri, 8 Dec 2023 02:32:15 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235902AbjLHKbr (ORCPT + 99 others); Fri, 8 Dec 2023 05:31:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60906 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235908AbjLHKbW (ORCPT ); Fri, 8 Dec 2023 05:31:22 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 14C4E2719 for ; Fri, 8 Dec 2023 02:31:04 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B6762C433C8; Fri, 8 Dec 2023 10:30:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702031463; bh=8t7YVZmoVUR7dCk/2nFP4SmQbrkuMwJVZH30PnGw0sc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DWhZa3EOQnOFLbqkXnThPPDz2DVUmGjTYOjFwf2l6M56pBwfRF2DkMvTJBFMsdS62 AzmIltwCI5BHlEcV3wBKQQ3D9mDNoJ9EGh5gh2LT+nJ9+6yLK9nJtKZJBWXDlXuQlK EbKuwfNZfYFZryd7rOKcpLSA74Q9ry2xkLiDT36oozRzAmTcZ84mImUGVGhRvQBYuU EZ5yAsRvuVTJ2m4pJXhdrJKN3/V81vu0kipeaTB+o/xMoTh3erT+bvrmRU1HUyKdx6 XZb5Blz2HPNDUVJjJy8c8OoFf4th9CqYF446V3HrAbtlbLu+cWbdgiUV8r8Y4EyHaq tYOCeKdsIc94w== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v4 33/33] Documentation: probes: Update fprobe on function-graph tracer Date: Fri, 8 Dec 2023 19:30:57 +0900 Message-Id: <170203145666.579004.6151223991854866508.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <170203105427.579004.8033550792660734570.stgit@devnote2> References: <170203105427.579004.8033550792660734570.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Fri, 08 Dec 2023 02:32:15 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784709423234262260 X-GMAIL-MSGID: 1784709423234262260 From: Masami Hiramatsu (Google) Update fprobe documentation for the new fprobe on function-graph tracer. This includes some bahvior changes and pt_regs to ftrace_regs interface change. Signed-off-by: Masami Hiramatsu (Google) --- Changes in v2: - Update @fregs parameter explanation. --- Documentation/trace/fprobe.rst | 42 ++++++++++++++++++++++++++-------------- 1 file changed, 27 insertions(+), 15 deletions(-) diff --git a/Documentation/trace/fprobe.rst b/Documentation/trace/fprobe.rst index 196f52386aaa..f58bdc64504f 100644 --- a/Documentation/trace/fprobe.rst +++ b/Documentation/trace/fprobe.rst @@ -9,9 +9,10 @@ Fprobe - Function entry/exit probe Introduction ============ -Fprobe is a function entry/exit probe mechanism based on ftrace. -Instead of using ftrace full feature, if you only want to attach callbacks -on function entry and exit, similar to the kprobes and kretprobes, you can +Fprobe is a function entry/exit probe mechanism based on the function-graph +tracer. +Instead of tracing all functions, if you want to attach callbacks on specific +function entry and exit, similar to the kprobes and kretprobes, you can use fprobe. Compared with kprobes and kretprobes, fprobe gives faster instrumentation for multiple functions with single handler. This document describes how to use fprobe. @@ -91,12 +92,14 @@ The prototype of the entry/exit callback function are as follows: .. code-block:: c - int entry_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long ret_ip, struct pt_regs *regs, void *entry_data); + int entry_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long ret_ip, struct ftrace_regs *fregs, void *entry_data); - void exit_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long ret_ip, struct pt_regs *regs, void *entry_data); + void exit_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long ret_ip, struct ftrace_regs *fregs, void *entry_data); -Note that the @entry_ip is saved at function entry and passed to exit handler. -If the entry callback function returns !0, the corresponding exit callback will be cancelled. +Note that the @entry_ip is saved at function entry and passed to exit +handler. +If the entry callback function returns !0, the corresponding exit callback +will be cancelled. @fp This is the address of `fprobe` data structure related to this handler. @@ -112,12 +115,10 @@ If the entry callback function returns !0, the corresponding exit callback will This is the return address that the traced function will return to, somewhere in the caller. This can be used at both entry and exit. -@regs - This is the `pt_regs` data structure at the entry and exit. Note that - the instruction pointer of @regs may be different from the @entry_ip - in the entry_handler. If you need traced instruction pointer, you need - to use @entry_ip. On the other hand, in the exit_handler, the instruction - pointer of @regs is set to the current return address. +@fregs + This is the `ftrace_regs` data structure at the entry and exit. This + includes the function parameters, or the return values. So user can + access thos values via appropriate `ftrace_regs_*` APIs. @entry_data This is a local storage to share the data between entry and exit handlers. @@ -125,6 +126,17 @@ If the entry callback function returns !0, the corresponding exit callback will and `entry_data_size` field when registering the fprobe, the storage is allocated and passed to both `entry_handler` and `exit_handler`. +Entry data size and exit handlers on the same function +====================================================== + +Since the entry data is passed via per-task stack and it is has limited size, +the entry data size per probe is limited to `15 * sizeof(long)`. You also need +to take care that the different fprobes are probing on the same function, this +limit becomes smaller. The entry data size is aligned to `sizeof(long)` and +each fprobe which has exit handler uses a `sizeof(long)` space on the stack, +you should keep the number of fprobes on the same function as small as +possible. + Share the callbacks with kprobes ================================ @@ -165,8 +177,8 @@ This counter counts up when; - fprobe fails to take ftrace_recursion lock. This usually means that a function which is traced by other ftrace users is called from the entry_handler. - - fprobe fails to setup the function exit because of the shortage of rethook - (the shadow stack for hooking the function return.) + - fprobe fails to setup the function exit because of failing to allocate the + data buffer from the per-task shadow stack. The `fprobe::nmissed` field counts up in both cases. Therefore, the former skips both of entry and exit callback and the latter skips the exit