From patchwork Mon Nov 20 20:54:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathieu Desnoyers X-Patchwork-Id: 167380 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2b07:b0:403:3b70:6f57 with SMTP id io7csp211789vqb; Mon, 20 Nov 2023 12:57:08 -0800 (PST) X-Google-Smtp-Source: AGHT+IHvW8dPnODTrv2E4Daz//Hb3/NuktQ4rZ3imhWK7bQ5X6HlMx2CuO1PR1GBgIag+2RB/3Hi X-Received: by 2002:a05:6a00:ac3:b0:693:3d55:2bb3 with SMTP id c3-20020a056a000ac300b006933d552bb3mr7383724pfl.26.1700513828277; Mon, 20 Nov 2023 12:57:08 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700513828; cv=none; d=google.com; s=arc-20160816; b=zsL3+8pJQw7OvS5xZxGKJPGHPkanXvBDvHwJAaY3XjKHvObBlVru7gUMAXgKU/p/+0 jxO5Mu/5YKuBGmIeuoexOUS1hzgjno/UiP1ngPmkglonDwlfZ5hQZ5NHmg72qp6Y+E8Z jjfqsM/GRrA2Z8asGEyu8BmwPXvZgByDxvR7OUnLd36fwvi6T5+Rmc0r6SX2LStkRQ9b JVRop+yVx+jEi7UMz2MhekSY7NBODiv6/EHfrszDNqUt5VCn4nx2Eg8OPnBjpv5Umr0Y Gf8vS9GFu4Hh5hp4WbaVcymbyhDyImR1VauPqC+GINaJBDi72D8SRJygWJMVGbT41GbN n4Nw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=fXtR8sGL//4sG+Ijb0+AOy1FRCb2ZR+AWqeaxHQfb+A=; fh=JZL+K55lXFxcyRYngJxDU2F2dZwuWhBnvU/9dYWYrHY=; b=VPWao6lge+gQL0xT6n3ztDjTxTSX5hdDOqkezW2G5LlNHteiBFXIeyRE1FoOaXl6Fq 2t+wuA8Knp3E7fi6ImSgtcCG3PxEnru2TtqydETdKcqz4rQCgKDcCr8/Fw8+vE6NsGfS YIlxS+u+pH5gBMpux1I0LXvk9aM3s4jR9idCRrNYkjSBekZZA0LGtEM0RqAD1+I/3a3f V5/Kd0mTKUsSzYlU4g7KbmljVTSJZz//YfLgsJsPo3kH7xyREY41/INcbQGXNb0sAPIE qqy1w2G2iFXxY2OOagklvyrVkxN/vuIsxdMbw45/fkDRqVK+xk/6sVx1JoGLzQI1QBi/ 650g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@efficios.com header.s=smtpout1 header.b=HDso6e46; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=efficios.com Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id a128-20020a624d86000000b006c33c8497casi9043662pfb.222.2023.11.20.12.57.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Nov 2023 12:57:08 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@efficios.com header.s=smtpout1 header.b=HDso6e46; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=efficios.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id C5E13808A8E4; Mon, 20 Nov 2023 12:56:12 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232109AbjKTUzA (ORCPT + 27 others); Mon, 20 Nov 2023 15:55:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35724 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230429AbjKTUyt (ORCPT ); Mon, 20 Nov 2023 15:54:49 -0500 Received: from smtpout.efficios.com (unknown [IPv6:2607:5300:203:b2ee::31e5]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E5CB9D2; Mon, 20 Nov 2023 12:54:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=efficios.com; s=smtpout1; t=1700513679; bh=DlcbITmTq90KuvaXd8GCbyB/ieomNbLpvBDRX6s70iQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HDso6e46r1QxSiXLdnvx5HLqFHWrWNX8fdEKaUCKV6dElWPBBcUa0s9EomvgUYb5U Qk37UIPhNNfEXnEAEIDqnnR4WAui5Ktg/LX7gkG7niQVhWHbaaP7Mw7L3jyWc9neoG ezMT3lx7Yu0QIuNCkRXEmPdnuZrM3YfC0Kuu1zXhuZ9ycGvyr8i+J11NhBkX63yYgE mBy26duotqFvn91d+QcNyW7V8rAg2JwXI6qX5+1XpSB3IMphA4maffMempCFFHYs90 n5zie+yUWAPzNHTZgzgIqFpNJ/Mq78SmH592VEKhzt3ALxKHWdaAdflNS6DGOp7RE4 mllbCd+BcFi0g== Received: from localhost.localdomain (192-222-143-198.qc.cable.ebox.net [192.222.143.198]) by smtpout.efficios.com (Postfix) with ESMTPSA id 4SZ0B72kbkz1cX6; Mon, 20 Nov 2023 15:54:39 -0500 (EST) From: Mathieu Desnoyers To: Steven Rostedt , Masami Hiramatsu Cc: linux-kernel@vger.kernel.org, Mathieu Desnoyers , Michael Jeanson , Peter Zijlstra , Alexei Starovoitov , Yonghong Song , "Paul E . McKenney" , Ingo Molnar , Arnaldo Carvalho de Melo , Mark Rutland , Alexander Shishkin , Jiri Olsa , Namhyung Kim , bpf@vger.kernel.org, Joel Fernandes Subject: [PATCH v4 1/5] tracing: Introduce faultable tracepoints Date: Mon, 20 Nov 2023 15:54:14 -0500 Message-Id: <20231120205418.334172-2-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231120205418.334172-1-mathieu.desnoyers@efficios.com> References: <20231120205418.334172-1-mathieu.desnoyers@efficios.com> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Mon, 20 Nov 2023 12:56:12 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1783117987832798583 X-GMAIL-MSGID: 1783117987832798583 When invoked from system call enter/exit instrumentation, accessing user-space data is a common use-case for tracers. However, tracepoints currently disable preemption around iteration on the registered tracepoint probes and invocation of the probe callbacks, which prevents tracers from handling page faults. Extend the tracepoint and trace event APIs to allow defining a faultable tracepoint which invokes its callback with preemption enabled. Also extend the tracepoint API to allow tracers to request specific probes to be connected to those faultable tracepoints. When the TRACEPOINT_MAY_FAULT flag is provided on registration, the probe callback will be called with preemption enabled, and is allowed to take page faults. Faultable probes can only be registered on faultable tracepoints and non-faultable probes on non-faultable tracepoints. The tasks trace rcu mechanism is used to synchronize read-side marshalling of the registered probes with respect to faultable probes unregistration and teardown. Link: https://lore.kernel.org/lkml/20231002202531.3160-1-mathieu.desnoyers@efficios.com/ Co-developed-by: Michael Jeanson Signed-off-by: Mathieu Desnoyers Signed-off-by: Michael Jeanson Cc: Steven Rostedt Cc: Masami Hiramatsu Cc: Peter Zijlstra Cc: Alexei Starovoitov Cc: Yonghong Song Cc: Paul E. McKenney Cc: Ingo Molnar Cc: Arnaldo Carvalho de Melo Cc: Mark Rutland Cc: Alexander Shishkin Cc: Jiri Olsa Cc: Namhyung Kim Cc: bpf@vger.kernel.org Cc: Joel Fernandes --- Changes since v1: - Cleanup __DO_TRACE() implementation. - Rename "sleepable tracepoints" to "faultable tracepoints", MAYSLEEP to MAYFAULT, and use might_fault() rather than might_sleep(), to properly convey that the tracepoints are meant to be able to take a page fault, which requires to be able to sleep *and* to hold the mmap_sem. Changes since v2: - Rename MAYFAULT to MAY_FAULT. - Rebased on 6.5.5. - Introduce MAY_EXIST tracepoint flag. Changes since v3: - Rebased on 6.6.2. --- include/linux/tracepoint-defs.h | 14 ++++++ include/linux/tracepoint.h | 88 +++++++++++++++++++++++---------- include/trace/define_trace.h | 7 +++ include/trace/trace_events.h | 6 +++ init/Kconfig | 1 + kernel/trace/bpf_trace.c | 5 +- kernel/trace/trace_fprobe.c | 5 +- kernel/tracepoint.c | 58 ++++++++++++---------- 8 files changed, 129 insertions(+), 55 deletions(-) diff --git a/include/linux/tracepoint-defs.h b/include/linux/tracepoint-defs.h index 4dc4955f0fbf..67bacfaa8fd0 100644 --- a/include/linux/tracepoint-defs.h +++ b/include/linux/tracepoint-defs.h @@ -29,6 +29,19 @@ struct tracepoint_func { int prio; }; +/** + * enum tracepoint_flags - Tracepoint flags + * @TRACEPOINT_MAY_EXIST: Don't return an error if the tracepoint does not + * exist upon registration. + * @TRACEPOINT_MAY_FAULT: The tracepoint probe callback will be called with + * preemption enabled, and is allowed to take page + * faults. + */ +enum tracepoint_flags { + TRACEPOINT_MAY_EXIST = (1 << 0), + TRACEPOINT_MAY_FAULT = (1 << 1), +}; + struct tracepoint { const char *name; /* Tracepoint name */ struct static_key key; @@ -39,6 +52,7 @@ struct tracepoint { int (*regfunc)(void); void (*unregfunc)(void); struct tracepoint_func __rcu *funcs; + unsigned int flags; }; #ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS diff --git a/include/linux/tracepoint.h b/include/linux/tracepoint.h index 88c0ba623ee6..8a6b58a2bf3b 100644 --- a/include/linux/tracepoint.h +++ b/include/linux/tracepoint.h @@ -18,6 +18,7 @@ #include #include #include +#include #include #include @@ -41,17 +42,10 @@ extern int tracepoint_probe_register_prio(struct tracepoint *tp, void *probe, void *data, int prio); extern int -tracepoint_probe_register_prio_may_exist(struct tracepoint *tp, void *probe, void *data, - int prio); +tracepoint_probe_register_prio_flags(struct tracepoint *tp, void *probe, void *data, + int prio, unsigned int flags); extern int tracepoint_probe_unregister(struct tracepoint *tp, void *probe, void *data); -static inline int -tracepoint_probe_register_may_exist(struct tracepoint *tp, void *probe, - void *data) -{ - return tracepoint_probe_register_prio_may_exist(tp, probe, data, - TRACEPOINT_DEFAULT_PRIO); -} extern void for_each_kernel_tracepoint(void (*fct)(struct tracepoint *tp, void *priv), void *priv); @@ -90,6 +84,7 @@ int unregister_tracepoint_module_notifier(struct notifier_block *nb) #ifdef CONFIG_TRACEPOINTS static inline void tracepoint_synchronize_unregister(void) { + synchronize_rcu_tasks_trace(); synchronize_srcu(&tracepoint_srcu); synchronize_rcu(); } @@ -192,9 +187,10 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) * it_func[0] is never NULL because there is at least one element in the array * when the array itself is non NULL. */ -#define __DO_TRACE(name, args, cond, rcuidle) \ +#define __DO_TRACE(name, args, cond, rcuidle, tp_flags) \ do { \ int __maybe_unused __idx = 0; \ + bool mayfault = (tp_flags) & TRACEPOINT_MAY_FAULT; \ \ if (!(cond)) \ return; \ @@ -202,8 +198,12 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) if (WARN_ON_ONCE(RCUIDLE_COND(rcuidle))) \ return; \ \ - /* keep srcu and sched-rcu usage consistent */ \ - preempt_disable_notrace(); \ + if (mayfault) { \ + rcu_read_lock_trace(); \ + } else { \ + /* keep srcu and sched-rcu usage consistent */ \ + preempt_disable_notrace(); \ + } \ \ /* \ * For rcuidle callers, use srcu since sched-rcu \ @@ -221,20 +221,23 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) srcu_read_unlock_notrace(&tracepoint_srcu, __idx);\ } \ \ - preempt_enable_notrace(); \ + if (mayfault) \ + rcu_read_unlock_trace(); \ + else \ + preempt_enable_notrace(); \ } while (0) #ifndef MODULE -#define __DECLARE_TRACE_RCU(name, proto, args, cond) \ +#define __DECLARE_TRACE_RCU(name, proto, args, cond, tp_flags) \ static inline void trace_##name##_rcuidle(proto) \ { \ if (static_key_false(&__tracepoint_##name.key)) \ __DO_TRACE(name, \ TP_ARGS(args), \ - TP_CONDITION(cond), 1); \ + TP_CONDITION(cond), 1, tp_flags); \ } #else -#define __DECLARE_TRACE_RCU(name, proto, args, cond) +#define __DECLARE_TRACE_RCU(name, proto, args, cond, tp_flags) #endif /* @@ -248,7 +251,7 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) * site if it is not watching, as it will need to be active when the * tracepoint is enabled. */ -#define __DECLARE_TRACE(name, proto, args, cond, data_proto) \ +#define __DECLARE_TRACE(name, proto, args, cond, data_proto, tp_flags) \ extern int __traceiter_##name(data_proto); \ DECLARE_STATIC_CALL(tp_func_##name, __traceiter_##name); \ extern struct tracepoint __tracepoint_##name; \ @@ -257,13 +260,15 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) if (static_key_false(&__tracepoint_##name.key)) \ __DO_TRACE(name, \ TP_ARGS(args), \ - TP_CONDITION(cond), 0); \ + TP_CONDITION(cond), 0, tp_flags); \ if (IS_ENABLED(CONFIG_LOCKDEP) && (cond)) { \ WARN_ON_ONCE(!rcu_is_watching()); \ } \ + if ((tp_flags) & TRACEPOINT_MAY_FAULT) \ + might_fault(); \ } \ __DECLARE_TRACE_RCU(name, PARAMS(proto), PARAMS(args), \ - PARAMS(cond)) \ + PARAMS(cond), tp_flags) \ static inline int \ register_trace_##name(void (*probe)(data_proto), void *data) \ { \ @@ -278,6 +283,13 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) (void *)probe, data, prio); \ } \ static inline int \ + register_trace_prio_flags_##name(void (*probe)(data_proto), void *data, \ + int prio, unsigned int flags) \ + { \ + return tracepoint_probe_register_prio_flags(&__tracepoint_##name, \ + (void *)probe, data, prio, flags); \ + } \ + static inline int \ unregister_trace_##name(void (*probe)(data_proto), void *data) \ { \ return tracepoint_probe_unregister(&__tracepoint_##name,\ @@ -298,7 +310,7 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) * structures, so we create an array of pointers that will be used for iteration * on the tracepoints. */ -#define DEFINE_TRACE_FN(_name, _reg, _unreg, proto, args) \ +#define DEFINE_TRACE_FN_FLAGS(_name, _reg, _unreg, proto, args, tp_flags) \ static const char __tpstrtab_##_name[] \ __section("__tracepoints_strings") = #_name; \ extern struct static_call_key STATIC_CALL_KEY(tp_func_##_name); \ @@ -314,7 +326,9 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) .probestub = &__probestub_##_name, \ .regfunc = _reg, \ .unregfunc = _unreg, \ - .funcs = NULL }; \ + .funcs = NULL, \ + .flags = (tp_flags), \ + }; \ __TRACEPOINT_ENTRY(_name); \ int __traceiter_##_name(void *__data, proto) \ { \ @@ -337,8 +351,11 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) } \ DEFINE_STATIC_CALL(tp_func_##_name, __traceiter_##_name); +#define DEFINE_TRACE_FN(_name, _reg, _unreg, proto, args) \ + DEFINE_TRACE_FN_FLAGS(_name, _reg, _unreg, PARAMS(proto), PARAMS(args), 0) + #define DEFINE_TRACE(name, proto, args) \ - DEFINE_TRACE_FN(name, NULL, NULL, PARAMS(proto), PARAMS(args)); + DEFINE_TRACE_FN(name, NULL, NULL, PARAMS(proto), PARAMS(args)) #define EXPORT_TRACEPOINT_SYMBOL_GPL(name) \ EXPORT_SYMBOL_GPL(__tracepoint_##name); \ @@ -351,7 +368,7 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) #else /* !TRACEPOINTS_ENABLED */ -#define __DECLARE_TRACE(name, proto, args, cond, data_proto) \ +#define __DECLARE_TRACE(name, proto, args, cond, data_proto, tp_flags) \ static inline void trace_##name(proto) \ { } \ static inline void trace_##name##_rcuidle(proto) \ @@ -363,6 +380,18 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) return -ENOSYS; \ } \ static inline int \ + register_trace_prio_##name(void (*probe)(data_proto), \ + void *data, int prio) \ + { \ + return -ENOSYS; \ + } \ + static inline int \ + register_trace_prio_flags_##name(void (*probe)(data_proto), \ + void *data, int prio, unsigned int flags) \ + { \ + return -ENOSYS; \ + } \ + static inline int \ unregister_trace_##name(void (*probe)(data_proto), \ void *data) \ { \ @@ -377,6 +406,7 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) return false; \ } +#define DEFINE_TRACE_FN_FLAGS(name, reg, unreg, proto, args, tp_flags) #define DEFINE_TRACE_FN(name, reg, unreg, proto, args) #define DEFINE_TRACE(name, proto, args) #define EXPORT_TRACEPOINT_SYMBOL_GPL(name) @@ -431,12 +461,17 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) #define DECLARE_TRACE(name, proto, args) \ __DECLARE_TRACE(name, PARAMS(proto), PARAMS(args), \ cpu_online(raw_smp_processor_id()), \ - PARAMS(void *__data, proto)) + PARAMS(void *__data, proto), 0) + +#define DECLARE_TRACE_MAY_FAULT(name, proto, args) \ + __DECLARE_TRACE(name, PARAMS(proto), PARAMS(args), \ + cpu_online(raw_smp_processor_id()), \ + PARAMS(void *__data, proto), TRACEPOINT_MAY_FAULT) #define DECLARE_TRACE_CONDITION(name, proto, args, cond) \ __DECLARE_TRACE(name, PARAMS(proto), PARAMS(args), \ cpu_online(raw_smp_processor_id()) && (PARAMS(cond)), \ - PARAMS(void *__data, proto)) + PARAMS(void *__data, proto), 0) #define TRACE_EVENT_FLAGS(event, flag) @@ -567,6 +602,9 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) #define TRACE_EVENT_FN(name, proto, args, struct, \ assign, print, reg, unreg) \ DECLARE_TRACE(name, PARAMS(proto), PARAMS(args)) +#define TRACE_EVENT_FN_MAY_FAULT(name, proto, args, struct, \ + assign, print, reg, unreg) \ + DECLARE_TRACE_MAY_FAULT(name, PARAMS(proto), PARAMS(args)) #define TRACE_EVENT_FN_COND(name, proto, args, cond, struct, \ assign, print, reg, unreg) \ DECLARE_TRACE_CONDITION(name, PARAMS(proto), \ diff --git a/include/trace/define_trace.h b/include/trace/define_trace.h index 00723935dcc7..1b8ca143724a 100644 --- a/include/trace/define_trace.h +++ b/include/trace/define_trace.h @@ -41,6 +41,12 @@ assign, print, reg, unreg) \ DEFINE_TRACE_FN(name, reg, unreg, PARAMS(proto), PARAMS(args)) +#undef TRACE_EVENT_FN_MAY_FAULT +#define TRACE_EVENT_FN_MAY_FAULT(name, proto, args, tstruct, \ + assign, print, reg, unreg) \ + DEFINE_TRACE_FN_FLAGS(name, reg, unreg, PARAMS(proto), \ + PARAMS(args), TRACEPOINT_MAY_FAULT) + #undef TRACE_EVENT_FN_COND #define TRACE_EVENT_FN_COND(name, proto, args, cond, tstruct, \ assign, print, reg, unreg) \ @@ -106,6 +112,7 @@ #undef TRACE_EVENT #undef TRACE_EVENT_FN +#undef TRACE_EVENT_FN_MAY_FAULT #undef TRACE_EVENT_FN_COND #undef TRACE_EVENT_CONDITION #undef TRACE_EVENT_NOP diff --git a/include/trace/trace_events.h b/include/trace/trace_events.h index c2f9cabf154d..df590eea8ae4 100644 --- a/include/trace/trace_events.h +++ b/include/trace/trace_events.h @@ -77,6 +77,12 @@ TRACE_EVENT(name, PARAMS(proto), PARAMS(args), \ PARAMS(tstruct), PARAMS(assign), PARAMS(print)) \ +#undef TRACE_EVENT_FN_MAY_FAULT +#define TRACE_EVENT_FN_MAY_FAULT(name, proto, args, tstruct, \ + assign, print, reg, unreg) \ + TRACE_EVENT(name, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print)) \ + #undef TRACE_EVENT_FN_COND #define TRACE_EVENT_FN_COND(name, proto, args, cond, tstruct, \ assign, print, reg, unreg) \ diff --git a/init/Kconfig b/init/Kconfig index 6d35728b94b2..f6fd53822868 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -1919,6 +1919,7 @@ config BINDGEN_VERSION_TEXT # config TRACEPOINTS bool + select TASKS_TRACE_RCU source "kernel/Kconfig.kexec" diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index 868008f56fec..344a41873445 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -2368,8 +2368,9 @@ static int __bpf_probe_register(struct bpf_raw_event_map *btp, struct bpf_prog * if (prog->aux->max_tp_access > btp->writable_size) return -EINVAL; - return tracepoint_probe_register_may_exist(tp, (void *)btp->bpf_func, - prog); + return tracepoint_probe_register_prio_flags(tp, (void *)btp->bpf_func, + prog, TRACEPOINT_DEFAULT_PRIO, + TRACEPOINT_MAY_EXIST); } int bpf_probe_register(struct bpf_raw_event_map *btp, struct bpf_prog *prog) diff --git a/kernel/trace/trace_fprobe.c b/kernel/trace/trace_fprobe.c index 8bfe23af9c73..22b935c82730 100644 --- a/kernel/trace/trace_fprobe.c +++ b/kernel/trace/trace_fprobe.c @@ -687,8 +687,9 @@ static int __register_trace_fprobe(struct trace_fprobe *tf) * At first, put __probestub_##TP function on the tracepoint * and put a fprobe on the stub function. */ - ret = tracepoint_probe_register_prio_may_exist(tpoint, - tpoint->probestub, NULL, 0); + ret = tracepoint_probe_register_prio_flags(tpoint, + tpoint->probestub, NULL, 0, + TRACEPOINT_MAY_EXIST); if (ret < 0) return ret; return register_fprobe_ips(&tf->fp, &ip, 1); diff --git a/kernel/tracepoint.c b/kernel/tracepoint.c index 8d1507dd0724..1f137163bdc5 100644 --- a/kernel/tracepoint.c +++ b/kernel/tracepoint.c @@ -111,11 +111,16 @@ static inline void *allocate_probes(int count) return p == NULL ? NULL : p->probes; } -static void srcu_free_old_probes(struct rcu_head *head) +static void rcu_tasks_trace_free_old_probes(struct rcu_head *head) { kfree(container_of(head, struct tp_probes, rcu)); } +static void srcu_free_old_probes(struct rcu_head *head) +{ + call_rcu_tasks_trace(head, rcu_tasks_trace_free_old_probes); +} + static void rcu_free_old_probes(struct rcu_head *head) { call_srcu(&tracepoint_srcu, head, srcu_free_old_probes); @@ -136,7 +141,7 @@ static __init int release_early_probes(void) return 0; } -/* SRCU is initialized at core_initcall */ +/* SRCU and Tasks Trace RCU are initialized at core_initcall */ postcore_initcall(release_early_probes); static inline void release_probes(struct tracepoint_func *old) @@ -146,8 +151,9 @@ static inline void release_probes(struct tracepoint_func *old) struct tp_probes, probes[0]); /* - * We can't free probes if SRCU is not initialized yet. - * Postpone the freeing till after SRCU is initialized. + * We can't free probes if SRCU and Tasks Trace RCU are not + * initialized yet. Postpone the freeing till after both are + * initialized. */ if (unlikely(!ok_to_free_tracepoints)) { tp_probes->rcu.next = early_probes; @@ -156,10 +162,9 @@ static inline void release_probes(struct tracepoint_func *old) } /* - * Tracepoint probes are protected by both sched RCU and SRCU, - * by calling the SRCU callback in the sched RCU callback we - * cover both cases. So let us chain the SRCU and sched RCU - * callbacks to wait for both grace periods. + * Tracepoint probes are protected by sched RCU, SRCU and + * Tasks Trace RCU by chaining the callbacks we cover all three + * cases and wait for all three grace periods. */ call_rcu(&tp_probes->rcu, rcu_free_old_probes); } @@ -460,30 +465,38 @@ static int tracepoint_remove_func(struct tracepoint *tp, } /** - * tracepoint_probe_register_prio_may_exist - Connect a probe to a tracepoint with priority + * tracepoint_probe_register_prio_flags - Connect a probe to a tracepoint with priority and flags * @tp: tracepoint * @probe: probe handler * @data: tracepoint data * @prio: priority of this function over other registered functions + * @flags: tracepoint flags argument (enum tracepoint_flags bits) * - * Same as tracepoint_probe_register_prio() except that it will not warn - * if the tracepoint is already registered. + * Returns 0 if ok, error value on error. + * Note: if @tp is within a module, the caller is responsible for + * unregistering the probe before the module is gone. This can be + * performed either with a tracepoint module going notifier, or from + * within module exit functions. */ -int tracepoint_probe_register_prio_may_exist(struct tracepoint *tp, void *probe, - void *data, int prio) +int tracepoint_probe_register_prio_flags(struct tracepoint *tp, void *probe, + void *data, int prio, unsigned int flags) { struct tracepoint_func tp_func; int ret; + if (((tp->flags & TRACEPOINT_MAY_FAULT) && !(flags & TRACEPOINT_MAY_FAULT)) || + (!(tp->flags & TRACEPOINT_MAY_FAULT) && (flags & TRACEPOINT_MAY_FAULT))) + return -EINVAL; + mutex_lock(&tracepoints_mutex); tp_func.func = probe; tp_func.data = data; tp_func.prio = prio; - ret = tracepoint_add_func(tp, &tp_func, prio, false); + ret = tracepoint_add_func(tp, &tp_func, prio, flags & TRACEPOINT_MAY_EXIST); mutex_unlock(&tracepoints_mutex); return ret; } -EXPORT_SYMBOL_GPL(tracepoint_probe_register_prio_may_exist); +EXPORT_SYMBOL_GPL(tracepoint_probe_register_prio_flags); /** * tracepoint_probe_register_prio - Connect a probe to a tracepoint with priority @@ -501,16 +514,7 @@ EXPORT_SYMBOL_GPL(tracepoint_probe_register_prio_may_exist); int tracepoint_probe_register_prio(struct tracepoint *tp, void *probe, void *data, int prio) { - struct tracepoint_func tp_func; - int ret; - - mutex_lock(&tracepoints_mutex); - tp_func.func = probe; - tp_func.data = data; - tp_func.prio = prio; - ret = tracepoint_add_func(tp, &tp_func, prio, true); - mutex_unlock(&tracepoints_mutex); - return ret; + return tracepoint_probe_register_prio_flags(tp, probe, data, prio, 0); } EXPORT_SYMBOL_GPL(tracepoint_probe_register_prio); @@ -520,6 +524,8 @@ EXPORT_SYMBOL_GPL(tracepoint_probe_register_prio); * @probe: probe handler * @data: tracepoint data * + * Non-faultable probes can only be registered on non-faultable tracepoints. + * * Returns 0 if ok, error value on error. * Note: if @tp is within a module, the caller is responsible for * unregistering the probe before the module is gone. This can be @@ -528,7 +534,7 @@ EXPORT_SYMBOL_GPL(tracepoint_probe_register_prio); */ int tracepoint_probe_register(struct tracepoint *tp, void *probe, void *data) { - return tracepoint_probe_register_prio(tp, probe, data, TRACEPOINT_DEFAULT_PRIO); + return tracepoint_probe_register_prio_flags(tp, probe, data, TRACEPOINT_DEFAULT_PRIO, 0); } EXPORT_SYMBOL_GPL(tracepoint_probe_register); From patchwork Mon Nov 20 20:54:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathieu Desnoyers X-Patchwork-Id: 167376 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2b07:b0:403:3b70:6f57 with SMTP id io7csp210854vqb; Mon, 20 Nov 2023 12:54:59 -0800 (PST) X-Google-Smtp-Source: AGHT+IGAS/twtOmPYZxRbirSBR98AIcqwt5ETitu8UplTUaPP5NlcQY/XVF6dIgZLoJlwMAEgiUT X-Received: by 2002:a05:6a00:9389:b0:690:2ad9:1436 with SMTP id ka9-20020a056a00938900b006902ad91436mr7952531pfb.7.1700513699578; Mon, 20 Nov 2023 12:54:59 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700513699; cv=none; d=google.com; s=arc-20160816; b=zq3Nb/gzIqir7Q/9HNC0A8S73s306LpFclhyBTzS62sk13ddJdPK2z3ieUHoT6jYYf kjgsxZcUYMjr5BHZBCTjO+nXaETRGFoa3e9a2DI1huQzMTEIud37ip5EXdOUvZZyi++a 4DPeQHImYNpk0sLi9N0P/OtJPKZW8+VU6zA18NblRzPOkcY6EROmQpFJ9LBr3wJO+ky5 xHEysNuDMzPFTMyu8k5vISYAi4jKTKeQbjxJKTDNHPBisgdMcOpzwIT3wSJXqvPRnihq GUzNkcg+g5i0lOHEwT9JH6XNTbIXQShb1oOV7yxW199Ag9z4ErCylprPiDul/0XL7mQu bHbg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=PANhLZLHQ6rI4eYbVMe+HWgTBwujKHsBu318rhOM5PA=; fh=JZL+K55lXFxcyRYngJxDU2F2dZwuWhBnvU/9dYWYrHY=; b=p6xVs1JVVGObWq78RfpBfZBTnx1FTrBMQjNdGCqH++MkQ1SUu64Q+TDx0qjAOE7BHr f/d8zx7sO0EVPk6x1eqzdNF376BWLS/XAUnYtF/wwxLN9Ll1wtSZ/TEBJudT/eePlzfQ pOegxmmHf93ToUKp4sZk3Z5GY1jmhyucf/A5nwHZm98DeKcqDy2UYBspiuZKWV3mG6QU LvjNP+MW8Bl3TSXeSinNOYL+eX1jb7gDzxstTLxD3zrvoLC1KVrjwuiqEJZI6/DtW/ys 4oGAHvbLj+DfWY7HcZKfHmZKMteCZ2+rlKV9+K/ZfYUzDiHotSJEuddRCZC+VqFSQJv7 GFRw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@efficios.com header.s=smtpout1 header.b=iC8gJ75T; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=efficios.com Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id fa4-20020a056a002d0400b006cb916f7794si3245850pfb.183.2023.11.20.12.54.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Nov 2023 12:54:59 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@efficios.com header.s=smtpout1 header.b=iC8gJ75T; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=efficios.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id C333C801F76A; Mon, 20 Nov 2023 12:54:57 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231438AbjKTUyv (ORCPT + 27 others); Mon, 20 Nov 2023 15:54:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35718 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229476AbjKTUys (ORCPT ); Mon, 20 Nov 2023 15:54:48 -0500 Received: from smtpout.efficios.com (unknown [IPv6:2607:5300:203:b2ee::31e5]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E5DC0D8; Mon, 20 Nov 2023 12:54:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=efficios.com; s=smtpout1; t=1700513680; bh=3na0my5GwRzgfHKRZqp6LpzRwS8xt8lVMei987Vc0nU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=iC8gJ75TzLrVcTnSXyBAFagP0C4E0+TB/V6nyjfSkVXD0UScwBQyX7EMBOr+Uvm3u tkNWPlyNX8MOqlvrTCKQszcxN1KSvfJCtBzXbzn++S2o5qriAyXD3+BzePklOEbkVX DsvCQlp07ehlxAgkgOQrFcWBlN4tZyD7pRhIc/7nQfGrk+sVczOBprO2dZ/gXU5Gi3 qCoSduMVLrRIp7pFD9EcCXG2uOiRSai+9Sl5t50CKJIRNy3fdB7MT/T8J/LxfGdH7t sQBF27KRHNSIXkAoVZvUVVHY487bW3hYdU3tbGpELkPblfftAQKkSx2KaWY/+a1Yqc strOxkyemhTtg== Received: from localhost.localdomain (192-222-143-198.qc.cable.ebox.net [192.222.143.198]) by smtpout.efficios.com (Postfix) with ESMTPSA id 4SZ0B75Vttz1cX7; Mon, 20 Nov 2023 15:54:39 -0500 (EST) From: Mathieu Desnoyers To: Steven Rostedt , Masami Hiramatsu Cc: linux-kernel@vger.kernel.org, Mathieu Desnoyers , Michael Jeanson , Peter Zijlstra , Alexei Starovoitov , Yonghong Song , "Paul E . McKenney" , Ingo Molnar , Arnaldo Carvalho de Melo , Mark Rutland , Alexander Shishkin , Jiri Olsa , Namhyung Kim , bpf@vger.kernel.org, Joel Fernandes Subject: [PATCH v4 2/5] tracing/ftrace: Add support for faultable tracepoints Date: Mon, 20 Nov 2023 15:54:15 -0500 Message-Id: <20231120205418.334172-3-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231120205418.334172-1-mathieu.desnoyers@efficios.com> References: <20231120205418.334172-1-mathieu.desnoyers@efficios.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.3 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED,RDNS_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Mon, 20 Nov 2023 12:54:57 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1783117852822935137 X-GMAIL-MSGID: 1783117852822935137 In preparation for converting system call enter/exit instrumentation into faultable tracepoints, make sure that ftrace can handle registering to such tracepoints by explicitly disabling preemption within the ftrace tracepoint probes to respect the current expectations within ftrace ring buffer code. This change does not yet allow ftrace to take page faults per se within its probe, but allows its existing probes to connect to faultable tracepoints. Link: https://lore.kernel.org/lkml/20231002202531.3160-1-mathieu.desnoyers@efficios.com/ Co-developed-by: Michael Jeanson Signed-off-by: Michael Jeanson Signed-off-by: Mathieu Desnoyers Cc: Steven Rostedt Cc: Masami Hiramatsu Cc: Peter Zijlstra Cc: Alexei Starovoitov Cc: Yonghong Song Cc: Paul E. McKenney Cc: Ingo Molnar Cc: Arnaldo Carvalho de Melo Cc: Mark Rutland Cc: Alexander Shishkin Cc: Jiri Olsa Cc: Namhyung Kim Cc: bpf@vger.kernel.org Cc: Joel Fernandes --- include/trace/trace_events.h | 69 +++++++++++++++++++++++++++++++++--- kernel/trace/trace_events.c | 26 ++++++++++---- 2 files changed, 84 insertions(+), 11 deletions(-) diff --git a/include/trace/trace_events.h b/include/trace/trace_events.h index df590eea8ae4..558af4960157 100644 --- a/include/trace/trace_events.h +++ b/include/trace/trace_events.h @@ -45,6 +45,16 @@ PARAMS(print)); \ DEFINE_EVENT(name, name, PARAMS(proto), PARAMS(args)); +#undef TRACE_EVENT_MAY_FAULT +#define TRACE_EVENT_MAY_FAULT(name, proto, args, tstruct, assign, print) \ + DECLARE_EVENT_CLASS_MAY_FAULT(name, \ + PARAMS(proto), \ + PARAMS(args), \ + PARAMS(tstruct), \ + PARAMS(assign), \ + PARAMS(print)); \ + DEFINE_EVENT(name, name, PARAMS(proto), PARAMS(args)); + #include "stages/stage1_struct_define.h" #undef DECLARE_EVENT_CLASS @@ -57,6 +67,11 @@ \ static struct trace_event_class event_class_##name; +#undef DECLARE_EVENT_CLASS_MAY_FAULT +#define DECLARE_EVENT_CLASS_MAY_FAULT(name, proto, args, tstruct, assign, print) \ + DECLARE_EVENT_CLASS(name, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print)) + #undef DEFINE_EVENT #define DEFINE_EVENT(template, name, proto, args) \ static struct trace_event_call __used \ @@ -80,7 +95,7 @@ #undef TRACE_EVENT_FN_MAY_FAULT #define TRACE_EVENT_FN_MAY_FAULT(name, proto, args, tstruct, \ assign, print, reg, unreg) \ - TRACE_EVENT(name, PARAMS(proto), PARAMS(args), \ + TRACE_EVENT_MAY_FAULT(name, PARAMS(proto), PARAMS(args), \ PARAMS(tstruct), PARAMS(assign), PARAMS(print)) \ #undef TRACE_EVENT_FN_COND @@ -123,6 +138,11 @@ tstruct; \ }; +#undef DECLARE_EVENT_CLASS_MAY_FAULT +#define DECLARE_EVENT_CLASS_MAY_FAULT(call, proto, args, tstruct, assign, print) \ + DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print)) + #undef DEFINE_EVENT #define DEFINE_EVENT(template, name, proto, args) @@ -214,6 +234,11 @@ static struct trace_event_functions trace_event_type_funcs_##call = { \ .trace = trace_raw_output_##call, \ }; +#undef DECLARE_EVENT_CLASS_MAY_FAULT +#define DECLARE_EVENT_CLASS_MAY_FAULT(call, proto, args, tstruct, assign, print) \ + DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print)) + #undef DEFINE_EVENT_PRINT #define DEFINE_EVENT_PRINT(template, call, proto, args, print) \ static notrace enum print_line_t \ @@ -250,6 +275,11 @@ static struct trace_event_fields trace_event_fields_##call[] = { \ tstruct \ {} }; +#undef DECLARE_EVENT_CLASS_MAY_FAULT +#define DECLARE_EVENT_CLASS_MAY_FAULT(call, proto, args, tstruct, assign, print) \ + DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print)) + #undef DEFINE_EVENT_PRINT #define DEFINE_EVENT_PRINT(template, name, proto, args, print) @@ -271,6 +301,11 @@ static inline notrace int trace_event_get_offsets_##call( \ return __data_size; \ } +#undef DECLARE_EVENT_CLASS_MAY_FAULT +#define DECLARE_EVENT_CLASS_MAY_FAULT(call, proto, args, tstruct, assign, print) \ + DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print)) + #include TRACE_INCLUDE(TRACE_INCLUDE_FILE) /* @@ -380,8 +415,8 @@ static inline notrace int trace_event_get_offsets_##call( \ #include "stages/stage6_event_callback.h" -#undef DECLARE_EVENT_CLASS -#define DECLARE_EVENT_CLASS(call, proto, args, tstruct, assign, print) \ +#undef _DECLARE_EVENT_CLASS +#define _DECLARE_EVENT_CLASS(call, proto, args, tstruct, assign, print, tp_flags) \ \ static notrace void \ trace_event_raw_event_##call(void *__data, proto) \ @@ -392,8 +427,13 @@ trace_event_raw_event_##call(void *__data, proto) \ struct trace_event_raw_##call *entry; \ int __data_size; \ \ + if ((tp_flags) & TRACEPOINT_MAY_FAULT) { \ + might_fault(); \ + preempt_disable_notrace(); \ + } \ + \ if (trace_trigger_soft_disabled(trace_file)) \ - return; \ + goto end; \ \ __data_size = trace_event_get_offsets_##call(&__data_offsets, args); \ \ @@ -401,14 +441,28 @@ trace_event_raw_event_##call(void *__data, proto) \ sizeof(*entry) + __data_size); \ \ if (!entry) \ - return; \ + goto end; \ \ tstruct \ \ { assign; } \ \ trace_event_buffer_commit(&fbuffer); \ +end: \ + if ((tp_flags) & TRACEPOINT_MAY_FAULT) \ + preempt_enable_notrace(); \ } + +#undef DECLARE_EVENT_CLASS +#define DECLARE_EVENT_CLASS(call, proto, args, tstruct, assign, print) \ + _DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print), 0) + +#undef DECLARE_EVENT_CLASS_MAY_FAULT +#define DECLARE_EVENT_CLASS_MAY_FAULT(call, proto, args, tstruct, assign, print) \ + _DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print), TRACEPOINT_MAY_FAULT) + /* * The ftrace_test_probe is compiled out, it is only here as a build time check * to make sure that if the tracepoint handling changes, the ftrace probe will @@ -440,6 +494,11 @@ static struct trace_event_class __used __refdata event_class_##call = { \ _TRACE_PERF_INIT(call) \ }; +#undef DECLARE_EVENT_CLASS_MAY_FAULT +#define DECLARE_EVENT_CLASS_MAY_FAULT(call, proto, args, tstruct, assign, print) \ + DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print)) + #undef DEFINE_EVENT #define DEFINE_EVENT(template, call, proto, args) \ \ diff --git a/kernel/trace/trace_events.c b/kernel/trace/trace_events.c index 82cb22ad6d61..954f87515668 100644 --- a/kernel/trace/trace_events.c +++ b/kernel/trace/trace_events.c @@ -532,9 +532,16 @@ int trace_event_reg(struct trace_event_call *call, WARN_ON(!(call->flags & TRACE_EVENT_FL_TRACEPOINT)); switch (type) { case TRACE_REG_REGISTER: - return tracepoint_probe_register(call->tp, - call->class->probe, - file); + if (call->tp->flags & TRACEPOINT_MAY_FAULT) + return tracepoint_probe_register_prio_flags(call->tp, + call->class->probe, + file, + TRACEPOINT_DEFAULT_PRIO, + TRACEPOINT_MAY_FAULT); + else + return tracepoint_probe_register(call->tp, + call->class->probe, + file); case TRACE_REG_UNREGISTER: tracepoint_probe_unregister(call->tp, call->class->probe, @@ -543,9 +550,16 @@ int trace_event_reg(struct trace_event_call *call, #ifdef CONFIG_PERF_EVENTS case TRACE_REG_PERF_REGISTER: - return tracepoint_probe_register(call->tp, - call->class->perf_probe, - call); + if (call->tp->flags & TRACEPOINT_MAY_FAULT) + return tracepoint_probe_register_prio_flags(call->tp, + call->class->perf_probe, + call, + TRACEPOINT_DEFAULT_PRIO, + TRACEPOINT_MAY_FAULT); + else + return tracepoint_probe_register(call->tp, + call->class->perf_probe, + call); case TRACE_REG_PERF_UNREGISTER: tracepoint_probe_unregister(call->tp, call->class->perf_probe, From patchwork Mon Nov 20 20:54:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathieu Desnoyers X-Patchwork-Id: 167379 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2b07:b0:403:3b70:6f57 with SMTP id io7csp211437vqb; Mon, 20 Nov 2023 12:56:21 -0800 (PST) X-Google-Smtp-Source: AGHT+IHxeJoVFqcDDasMbd7m5W5fCyirjjoPITD/XpC8BNfx9YFAN3Y9EQxt7twPLNK3Pw/LWmeL X-Received: by 2002:a05:6830:11ce:b0:6cd:4358:1f02 with SMTP id v14-20020a05683011ce00b006cd43581f02mr9666203otq.34.1700513781424; Mon, 20 Nov 2023 12:56:21 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700513781; cv=none; d=google.com; s=arc-20160816; b=iTONxhzMDnxXGeZmh2b5zn5JaWK8v0x2zhdy8RrnXVZHEabT+y6Qtye0LMk2i50SIQ TnVuploMVhBttdNPTSaaY1++mLxQcbvQ40Tp7xcU7X6laftbf1zTFUMuXIRILdqXiKsK +mIoy8TlRjcqpzHnTTeOokvHd5JvIOZokaNysLOV+xQjCzGR+1kDYDZGjGG0RDY5S1zr M8QZJtpN1KUgMdilPp4EsFLsUyrI0jukOh87Ufhf6qTsjxSO8ahgqhxaUzbJu6Tojpz4 T+/OKxpUHkqZofPKPipud6wp3Ra/fJifuNif+UbAfCJ5fURtIJDmi7/LK+JP6c/VoeYZ JgnQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=hTc99U5QpZHJo4970DapfP/0dO6JpKRNReE4z6smZoY=; fh=JZL+K55lXFxcyRYngJxDU2F2dZwuWhBnvU/9dYWYrHY=; b=hldlecre/YkFIlxmuKWFPEx4XO64AaOhmQ3rPm9uqYH86u2KphCOohZbmdVahglyCv 3Maoo9PmH3+wH7dvgN2wrAqHVUpeU0hBtsites4hnx5acAGZGqRDhM6TXO+flnBszI2x td5un2McVRCcbv+7jRDA+7UF44rmH+dXhGb45tl7YwEGsdCbT1xIXdQBzCbk8l74JACj 0dX0yyG8RzoZW86iKUaOtUNJQNPNbGtnoIYauvQu90CKaZltXUhH+MgO8ufIFn63jP9D thDFp+Q+vJj9reyBobS4kKXK/JV35v0iWOlyC7x8CGEavBxmuJZKAaDgz1sYN/BK06rG WTPA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@efficios.com header.s=smtpout1 header.b=oLQ5Jv2J; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=efficios.com Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id u22-20020a631416000000b005be1955657esi8816290pgl.127.2023.11.20.12.56.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Nov 2023 12:56:21 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@efficios.com header.s=smtpout1 header.b=oLQ5Jv2J; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=efficios.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id E56BA8076661; Mon, 20 Nov 2023 12:55:47 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231204AbjKTUyu (ORCPT + 27 others); Mon, 20 Nov 2023 15:54:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35702 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230296AbjKTUys (ORCPT ); Mon, 20 Nov 2023 15:54:48 -0500 Received: from smtpout.efficios.com (unknown [IPv6:2607:5300:203:b2ee::31e5]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E51F2CF; Mon, 20 Nov 2023 12:54:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=efficios.com; s=smtpout1; t=1700513680; bh=T7Nr4r/6B9r8V/IXWfdEL5nzZ2OVp6Lt4IjgQKAxAJM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oLQ5Jv2JPcwEa+icp0BZjOHxsNfVcK1LCUYVoZE6SmDGbC4oUBOBiPOLXxmGi1tSE TTU0pgrcELQpQZNXJeXv4CTpqZFJHcp1M9apNGovdE+d3p5eYKVkUlA5EG5cyxu5UL p4gsrZ6RFVRKcyBBpvaCkTTadHxj81t2Brg5uj9sDR/R7sEBe4EPIGcIFDUV8IfDAp a70A5F4j9Fny9L/8f8YtOA7slJQRCi4yiVbOj8gbledJttrdjEBXKMSuX1j02u9uTe eMkqMQT8uVVGDjAWOv6dbzwU4Ez0kp/RrMECEjt+0iB337XZoXPL+CrnboG4fquTxp QeQa34EMelXAQ== Received: from localhost.localdomain (192-222-143-198.qc.cable.ebox.net [192.222.143.198]) by smtpout.efficios.com (Postfix) with ESMTPSA id 4SZ0B80w7Zz1cy0; Mon, 20 Nov 2023 15:54:40 -0500 (EST) From: Mathieu Desnoyers To: Steven Rostedt , Masami Hiramatsu Cc: linux-kernel@vger.kernel.org, Mathieu Desnoyers , Michael Jeanson , Peter Zijlstra , Alexei Starovoitov , Yonghong Song , "Paul E . McKenney" , Ingo Molnar , Arnaldo Carvalho de Melo , Mark Rutland , Alexander Shishkin , Jiri Olsa , Namhyung Kim , bpf@vger.kernel.org, Joel Fernandes Subject: [PATCH v4 3/5] tracing/bpf-trace: add support for faultable tracepoints Date: Mon, 20 Nov 2023 15:54:16 -0500 Message-Id: <20231120205418.334172-4-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231120205418.334172-1-mathieu.desnoyers@efficios.com> References: <20231120205418.334172-1-mathieu.desnoyers@efficios.com> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Mon, 20 Nov 2023 12:55:48 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1783117938915005285 X-GMAIL-MSGID: 1783117938915005285 In preparation for converting system call enter/exit instrumentation into faultable tracepoints, make sure that bpf can handle registering to such tracepoints by explicitly disabling preemption within the bpf tracepoint probes to respect the current expectations within bpf tracing code. This change does not yet allow bpf to take page faults per se within its probe, but allows its existing probes to connect to faultable tracepoints. Link: https://lore.kernel.org/lkml/20231002202531.3160-1-mathieu.desnoyers@efficios.com/ Co-developed-by: Michael Jeanson Signed-off-by: Mathieu Desnoyers Signed-off-by: Michael Jeanson Cc: Steven Rostedt Cc: Masami Hiramatsu Cc: Peter Zijlstra Cc: Alexei Starovoitov Cc: Yonghong Song Cc: Paul E. McKenney Cc: Ingo Molnar Cc: Arnaldo Carvalho de Melo Cc: Mark Rutland Cc: Alexander Shishkin Cc: Jiri Olsa Cc: Namhyung Kim Cc: bpf@vger.kernel.org Cc: Joel Fernandes --- include/trace/bpf_probe.h | 21 +++++++++++++++++---- kernel/trace/bpf_trace.c | 11 ++++++++--- 2 files changed, 25 insertions(+), 7 deletions(-) diff --git a/include/trace/bpf_probe.h b/include/trace/bpf_probe.h index e609cd7da47e..dd4b42715e5d 100644 --- a/include/trace/bpf_probe.h +++ b/include/trace/bpf_probe.h @@ -42,17 +42,30 @@ /* tracepoints with more than 12 arguments will hit build error */ #define CAST_TO_U64(...) CONCATENATE(__CAST, COUNT_ARGS(__VA_ARGS__))(__VA_ARGS__) -#define __BPF_DECLARE_TRACE(call, proto, args) \ +#define __BPF_DECLARE_TRACE(call, proto, args, tp_flags) \ static notrace void \ __bpf_trace_##call(void *__data, proto) \ { \ struct bpf_prog *prog = __data; \ + \ + if ((tp_flags) & TRACEPOINT_MAY_FAULT) { \ + might_fault(); \ + preempt_disable_notrace(); \ + } \ + \ CONCATENATE(bpf_trace_run, COUNT_ARGS(args))(prog, CAST_TO_U64(args)); \ + \ + if ((tp_flags) & TRACEPOINT_MAY_FAULT) \ + preempt_enable_notrace(); \ } #undef DECLARE_EVENT_CLASS #define DECLARE_EVENT_CLASS(call, proto, args, tstruct, assign, print) \ - __BPF_DECLARE_TRACE(call, PARAMS(proto), PARAMS(args)) + __BPF_DECLARE_TRACE(call, PARAMS(proto), PARAMS(args), 0) + +#undef DECLARE_EVENT_CLASS_MAY_FAULT +#define DECLARE_EVENT_CLASS_MAY_FAULT(call, proto, args, tstruct, assign, print) \ + __BPF_DECLARE_TRACE(call, PARAMS(proto), PARAMS(args), TRACEPOINT_MAY_FAULT) /* * This part is compiled out, it is only here as a build time check @@ -106,13 +119,13 @@ static inline void bpf_test_buffer_##call(void) \ #undef DECLARE_TRACE #define DECLARE_TRACE(call, proto, args) \ - __BPF_DECLARE_TRACE(call, PARAMS(proto), PARAMS(args)) \ + __BPF_DECLARE_TRACE(call, PARAMS(proto), PARAMS(args), 0) \ __DEFINE_EVENT(call, call, PARAMS(proto), PARAMS(args), 0) #undef DECLARE_TRACE_WRITABLE #define DECLARE_TRACE_WRITABLE(call, proto, args, size) \ __CHECK_WRITABLE_BUF_SIZE(call, PARAMS(proto), PARAMS(args), size) \ - __BPF_DECLARE_TRACE(call, PARAMS(proto), PARAMS(args)) \ + __BPF_DECLARE_TRACE(call, PARAMS(proto), PARAMS(args), 0) \ __DEFINE_EVENT(call, call, PARAMS(proto), PARAMS(args), size) #include TRACE_INCLUDE(TRACE_INCLUDE_FILE) diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index 344a41873445..1eb770dafa8a 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -2368,9 +2368,14 @@ static int __bpf_probe_register(struct bpf_raw_event_map *btp, struct bpf_prog * if (prog->aux->max_tp_access > btp->writable_size) return -EINVAL; - return tracepoint_probe_register_prio_flags(tp, (void *)btp->bpf_func, - prog, TRACEPOINT_DEFAULT_PRIO, - TRACEPOINT_MAY_EXIST); + if (tp->flags & TRACEPOINT_MAY_FAULT) + return tracepoint_probe_register_prio_flags(tp, (void *)btp->bpf_func, + prog, TRACEPOINT_DEFAULT_PRIO, + TRACEPOINT_MAY_EXIST | TRACEPOINT_MAY_FAULT); + else + return tracepoint_probe_register_prio_flags(tp, (void *)btp->bpf_func, + prog, TRACEPOINT_DEFAULT_PRIO, + TRACEPOINT_MAY_EXIST); } int bpf_probe_register(struct bpf_raw_event_map *btp, struct bpf_prog *prog) From patchwork Mon Nov 20 20:54:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathieu Desnoyers X-Patchwork-Id: 167377 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2b07:b0:403:3b70:6f57 with SMTP id io7csp210877vqb; Mon, 20 Nov 2023 12:55:02 -0800 (PST) X-Google-Smtp-Source: AGHT+IGKqxjNcryFAWug8Ks9Nc9fUMBS6REtkNwA3iAQ8CzBeY3thM66jKxVguiKE3c0Vp2/ifNO X-Received: by 2002:a17:902:968e:b0:1cc:361b:7b10 with SMTP id n14-20020a170902968e00b001cc361b7b10mr6935095plp.24.1700513701782; Mon, 20 Nov 2023 12:55:01 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700513701; cv=none; d=google.com; s=arc-20160816; b=xM5pYSubn7xyOnDTOx6DytaJz/yoc5mFEhp11CZz5Wf3YXuNXqa4cgcre2Z9CFfcBe sAlt8UZEVLAhoeClX9jf8KBM5WEi8mC6CH6r0SYcRaTQFT3YWYIc15x4odKEaZ8XyJxU gUUFJuJW2zqqApwBW3cBH2q0LIiaAPlOgAJCSAfc6DaQ3/KODngZoCxP5kgaKR8T9Dgy Ea4hzfwhkey3krg3HqkVJhK1VWg3uxEVVOPGG0dowabXmH1Im36AbUFnGdytSfd2AbRe qVx+CVuNFVVunBZvzHWqmoagOz1GQW3gkZnG/iI3+ug4UtssgqgTWuTB+2X48Tk85cay G2vQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=uhPGCTfPEyhleVGHSQGIWrRH/PSYsUhWP4eniAeNcOs=; fh=JZL+K55lXFxcyRYngJxDU2F2dZwuWhBnvU/9dYWYrHY=; b=HmxiIoGq10BzVHhCJtYorM9DVBeiT6HP5W2DA+oEnnCqqtuTmwcw66MSIsr/urYWX1 4m80EzJuDD6e22BnyfZlbT9p+wWwizsNxCWqGbbkitc02VnfVBcPPOR3bpp/tBVcf63U AV33SE3mgAmUJgqSC3nOb7tGRv2rncBlsQWikEIRTQUC0jfWcHUUbeiKygLTv8OLgWh5 ICqDvQpInas7Wq8T7lkJ7fmmif681zMAi1Xnxg0JhsKaE4hX8GWHrS4jzvYyNowgIphr E1UnO5v5NdzddXZDVQ61zx4d69LcqlteOZhUZCA1T+BWob/PTAsNsigPRTfSvw20aU8M xclQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@efficios.com header.s=smtpout1 header.b=UVLZmdzA; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=efficios.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id e13-20020a170902d38d00b001cf641feee3si1889226pld.494.2023.11.20.12.55.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Nov 2023 12:55:01 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@efficios.com header.s=smtpout1 header.b=UVLZmdzA; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=efficios.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id D5936801F763; Mon, 20 Nov 2023 12:54:59 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231724AbjKTUyy (ORCPT + 27 others); Mon, 20 Nov 2023 15:54:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35712 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230357AbjKTUys (ORCPT ); Mon, 20 Nov 2023 15:54:48 -0500 Received: from smtpout.efficios.com (unknown [IPv6:2607:5300:203:b2ee::31e5]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E5F2DDC; Mon, 20 Nov 2023 12:54:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=efficios.com; s=smtpout1; t=1700513680; bh=Daj+/1NhYCfniiA2tUuq6iASXun5yE8kO5F1mYx3zX4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UVLZmdzA+6t/55WMLsBXzqGOH8ntevxRZQqooyook/p6evO7BV6kRTHOkHQSjrDyh sgj8aO9h/0cs/5EmYRRmMU+kJLDaFW2SgpvGgEOIBgiEXZTnSylEfkU0GuZIOZMkNb E1JKWFH4vgz4bgrLhBUih5or6T0ROIy4GtJAoVxvnPgYdwBS0MF9VDjU2mtNI9ExGT CUUTDmJeyPtUKOrDfSkGlB/e977Z9bUpqJSVAKZSlPZnyCRPi7kX5e0cLMm/pPXN+k VfGPh9tcOOJ1RZaPQVb6xxx4DzDe7YDytJvsIjmuxUkk94Rjn+DJY/yMvpjjEqDMoZ /ebfrTLj3uixQ== Received: from localhost.localdomain (192-222-143-198.qc.cable.ebox.net [192.222.143.198]) by smtpout.efficios.com (Postfix) with ESMTPSA id 4SZ0B83q4Sz1cb0; Mon, 20 Nov 2023 15:54:40 -0500 (EST) From: Mathieu Desnoyers To: Steven Rostedt , Masami Hiramatsu Cc: linux-kernel@vger.kernel.org, Mathieu Desnoyers , Michael Jeanson , Peter Zijlstra , Alexei Starovoitov , Yonghong Song , "Paul E . McKenney" , Ingo Molnar , Arnaldo Carvalho de Melo , Mark Rutland , Alexander Shishkin , Jiri Olsa , Namhyung Kim , bpf@vger.kernel.org, Joel Fernandes Subject: [PATCH v4 4/5] tracing/perf: add support for faultable tracepoints Date: Mon, 20 Nov 2023 15:54:17 -0500 Message-Id: <20231120205418.334172-5-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231120205418.334172-1-mathieu.desnoyers@efficios.com> References: <20231120205418.334172-1-mathieu.desnoyers@efficios.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.3 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED,RDNS_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Mon, 20 Nov 2023 12:54:59 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1783117855294705079 X-GMAIL-MSGID: 1783117855294705079 In preparation for converting system call enter/exit instrumentation into faultable tracepoints, make sure that perf can handle registering to such tracepoints by explicitly disabling preemption within the perf tracepoint probes to respect the current expectations within perf ring buffer code. This change does not yet allow perf to take page faults per se within its probe, but allows its existing probes to connect to faultable tracepoints. Link: https://lore.kernel.org/lkml/20231002202531.3160-1-mathieu.desnoyers@efficios.com/ Co-developed-by: Michael Jeanson Signed-off-by: Mathieu Desnoyers Signed-off-by: Michael Jeanson Cc: Steven Rostedt Cc: Masami Hiramatsu Cc: Peter Zijlstra Cc: Alexei Starovoitov Cc: Yonghong Song Cc: Paul E. McKenney Cc: Ingo Molnar Cc: Arnaldo Carvalho de Melo Cc: Mark Rutland Cc: Alexander Shishkin Cc: Jiri Olsa Cc: Namhyung Kim Cc: bpf@vger.kernel.org Cc: Joel Fernandes --- include/trace/perf.h | 27 +++++++++++++++++++++++---- 1 file changed, 23 insertions(+), 4 deletions(-) diff --git a/include/trace/perf.h b/include/trace/perf.h index 2c11181c82e0..fb47815f6eff 100644 --- a/include/trace/perf.h +++ b/include/trace/perf.h @@ -12,8 +12,8 @@ #undef __perf_task #define __perf_task(t) (__task = (t)) -#undef DECLARE_EVENT_CLASS -#define DECLARE_EVENT_CLASS(call, proto, args, tstruct, assign, print) \ +#undef _DECLARE_EVENT_CLASS +#define _DECLARE_EVENT_CLASS(call, proto, args, tstruct, assign, print, tp_flags) \ static notrace void \ perf_trace_##call(void *__data, proto) \ { \ @@ -28,13 +28,18 @@ perf_trace_##call(void *__data, proto) \ int __data_size; \ int rctx; \ \ + if ((tp_flags) & TRACEPOINT_MAY_FAULT) { \ + might_fault(); \ + preempt_disable_notrace(); \ + } \ + \ __data_size = trace_event_get_offsets_##call(&__data_offsets, args); \ \ head = this_cpu_ptr(event_call->perf_events); \ if (!bpf_prog_array_valid(event_call) && \ __builtin_constant_p(!__task) && !__task && \ hlist_empty(head)) \ - return; \ + goto end; \ \ __entry_size = ALIGN(__data_size + sizeof(*entry) + sizeof(u32),\ sizeof(u64)); \ @@ -42,7 +47,7 @@ perf_trace_##call(void *__data, proto) \ \ entry = perf_trace_buf_alloc(__entry_size, &__regs, &rctx); \ if (!entry) \ - return; \ + goto end; \ \ perf_fetch_caller_regs(__regs); \ \ @@ -53,8 +58,22 @@ perf_trace_##call(void *__data, proto) \ perf_trace_run_bpf_submit(entry, __entry_size, rctx, \ event_call, __count, __regs, \ head, __task); \ +end: \ + if ((tp_flags) & TRACEPOINT_MAY_FAULT) \ + preempt_enable_notrace(); \ } +#undef DECLARE_EVENT_CLASS +#define DECLARE_EVENT_CLASS(call, proto, args, tstruct, assign, print) \ + _DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print), 0) + +#undef DECLARE_EVENT_CLASS_MAY_FAULT +#define DECLARE_EVENT_CLASS_MAY_FAULT(call, proto, args, tstruct, assign, print) \ + _DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print), \ + TRACEPOINT_MAY_FAULT) + /* * This part is compiled out, it is only here as a build time check * to make sure that if the tracepoint handling changes, the From patchwork Mon Nov 20 20:54:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathieu Desnoyers X-Patchwork-Id: 167378 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2b07:b0:403:3b70:6f57 with SMTP id io7csp210965vqb; Mon, 20 Nov 2023 12:55:13 -0800 (PST) X-Google-Smtp-Source: AGHT+IFBOQm6Qs/hoAFmUnOHUQhE5kCUEyGeBcTmCWdS2O2V33BkeFARteOi5YyEwEwMWH0BkiFE X-Received: by 2002:a05:6a20:42a6:b0:187:858b:8c06 with SMTP id o38-20020a056a2042a600b00187858b8c06mr11665690pzj.16.1700513713262; Mon, 20 Nov 2023 12:55:13 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700513713; cv=none; d=google.com; s=arc-20160816; b=WjHo/6HGiboh8iT9nG+DB6Wsj7i2jyNdnNqLohI36VRleYshetq8NRMn5qVbYJxEWn 0j/arEQ9vi2dMygyl0f+YMElogrDuFGoF1Zy+y/PI6URNPuvNw1psFMUpA/A0I7QgaoG VyFIH+gaFZrG91tCIxHLDhaAdLqQhRDGPpzfaEjsu68nT3h4cFzurYYAd3CNC51dAMWi zIA2XWZrczd98JeZVde3/svEWJPHmfkJ8nXfkFcFOTmSP4s0MdCC59copsdKJa7ULRHM Q7k8aGT33c5uICvBxRvo3x1U7L+q+0VHDmhSMN/jM5Xofz/i3zF6wlh4dNcsgbYvfgZK WaAw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=PpRqtij0GYXU1vzuiOLQ9bGoy7z9ZKLKKDVJWwyVTjk=; fh=JZL+K55lXFxcyRYngJxDU2F2dZwuWhBnvU/9dYWYrHY=; b=lo8ijV0cIygQqjAXcE7f5jTmQ0PMatxKodoynMNDR4IjwVr2yW1Drr8QCJv1zKG+0r 4RJqmm4ZSxc2mAn1OZ3asyFGoTA4O5N+FdpKa1eHL/jK52W27+ObI3QIOpV+t7+YuCKC 1Gnog7tuJq464uOO6Dmi6lgYkZ0oH/boRF9mVFeQc4Xud+hmbbL1y1eFo3RJFnVOMUzs NH8kr0eXltFbTklQb7gAjVY+v3oLeozOJQ/J92hv1O9rw2uFaLeTtgjYztgqOnELt18g K5mAgSE3eDrdou19tYi6bbzqZNmLCqpjyd1FVjeTutxcUzix6/+R1KWMiNe+7NF+oBnX P+fA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@efficios.com header.s=smtpout1 header.b=dQb6s5nO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=efficios.com Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id c12-20020a65420c000000b005c1cc7273besi8813396pgq.521.2023.11.20.12.55.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Nov 2023 12:55:13 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@efficios.com header.s=smtpout1 header.b=dQb6s5nO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=efficios.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 9ADD5801C7C0; Mon, 20 Nov 2023 12:55:11 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232340AbjKTUzD (ORCPT + 27 others); Mon, 20 Nov 2023 15:55:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35732 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230170AbjKTUyu (ORCPT ); Mon, 20 Nov 2023 15:54:50 -0500 Received: from smtpout.efficios.com (unknown [IPv6:2607:5300:203:b2ee::31e5]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 555C3E7; Mon, 20 Nov 2023 12:54:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=efficios.com; s=smtpout1; t=1700513681; bh=gzqVrF2HW4E6ZSO3Uvr5o3XrzMWnpKg+s2vm1djRgJ4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dQb6s5nOoxjKf21ZJDHu1gXmBPZBiQ5ybxukjIAKdlvlS0UkwvOL/5j9k6H8OeeXx 8MX3SIXzYUaaHh6H6EGNUYOzpj4pQEIyJ9K70LiUuztHl7NJh+cu0aty18lSHlN589 uhee8wdZEKFOGZ97hNkDQFls1GvpCBqH6jRnT7bouGTtyt4NTYQnjWXD4yQcrNuCEJ 4adkDFtQGEWWHvgwCdsmxcnhEIvWuqaVwYcFgHIX/oWyQ+RHACOFF2tf+r7RDkiSTB jrpS2AkTPFyaUxzxqqbY/+SLvr/ShJKATDCgTqW/ToLCmVSb+eWH7ZLmSHqLCgvELH 4K9l/kVztkkgQ== Received: from localhost.localdomain (192-222-143-198.qc.cable.ebox.net [192.222.143.198]) by smtpout.efficios.com (Postfix) with ESMTPSA id 4SZ0B86pWRz1cmG; Mon, 20 Nov 2023 15:54:40 -0500 (EST) From: Mathieu Desnoyers To: Steven Rostedt , Masami Hiramatsu Cc: linux-kernel@vger.kernel.org, Mathieu Desnoyers , Michael Jeanson , Peter Zijlstra , Alexei Starovoitov , Yonghong Song , "Paul E . McKenney" , Ingo Molnar , Arnaldo Carvalho de Melo , Mark Rutland , Alexander Shishkin , Jiri Olsa , Namhyung Kim , bpf@vger.kernel.org, Joel Fernandes Subject: [PATCH v4 5/5] tracing: convert sys_enter/exit to faultable tracepoints Date: Mon, 20 Nov 2023 15:54:18 -0500 Message-Id: <20231120205418.334172-6-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231120205418.334172-1-mathieu.desnoyers@efficios.com> References: <20231120205418.334172-1-mathieu.desnoyers@efficios.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.3 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED,RDNS_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Mon, 20 Nov 2023 12:55:11 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1783117867373056628 X-GMAIL-MSGID: 1783117867373056628 Convert the definition of the system call enter/exit tracepoints to faultable tracepoints now that all upstream tracers handle it. This allows tracers to fault-in userspace system call arguments such as path strings within their probe callbacks. Link: https://lore.kernel.org/lkml/20231002202531.3160-1-mathieu.desnoyers@efficios.com/ Co-developed-by: Michael Jeanson Signed-off-by: Mathieu Desnoyers Signed-off-by: Michael Jeanson Cc: Steven Rostedt Cc: Masami Hiramatsu Cc: Peter Zijlstra Cc: Alexei Starovoitov Cc: Yonghong Song Cc: Paul E. McKenney Cc: Ingo Molnar Cc: Arnaldo Carvalho de Melo Cc: Mark Rutland Cc: Alexander Shishkin Cc: Jiri Olsa Cc: Namhyung Kim Cc: bpf@vger.kernel.org Cc: Joel Fernandes --- include/trace/events/syscalls.h | 4 +- kernel/trace/trace_syscalls.c | 92 +++++++++++++++++++++++---------- 2 files changed, 68 insertions(+), 28 deletions(-) diff --git a/include/trace/events/syscalls.h b/include/trace/events/syscalls.h index b6e0cbc2c71f..dc30e3004818 100644 --- a/include/trace/events/syscalls.h +++ b/include/trace/events/syscalls.h @@ -15,7 +15,7 @@ #ifdef CONFIG_HAVE_SYSCALL_TRACEPOINTS -TRACE_EVENT_FN(sys_enter, +TRACE_EVENT_FN_MAY_FAULT(sys_enter, TP_PROTO(struct pt_regs *regs, long id), @@ -41,7 +41,7 @@ TRACE_EVENT_FN(sys_enter, TRACE_EVENT_FLAGS(sys_enter, TRACE_EVENT_FL_CAP_ANY) -TRACE_EVENT_FN(sys_exit, +TRACE_EVENT_FN_MAY_FAULT(sys_exit, TP_PROTO(struct pt_regs *regs, long ret), diff --git a/kernel/trace/trace_syscalls.c b/kernel/trace/trace_syscalls.c index de753403cdaf..718a0723a0bc 100644 --- a/kernel/trace/trace_syscalls.c +++ b/kernel/trace/trace_syscalls.c @@ -299,27 +299,33 @@ static void ftrace_syscall_enter(void *data, struct pt_regs *regs, long id) int syscall_nr; int size; + /* + * Probe called with preemption enabled (may_fault), but ring buffer and + * per-cpu data require preemption to be disabled. + */ + preempt_disable_notrace(); + syscall_nr = trace_get_syscall_nr(current, regs); if (syscall_nr < 0 || syscall_nr >= NR_syscalls) - return; + goto end; /* Here we're inside tp handler's rcu_read_lock_sched (__DO_TRACE) */ trace_file = rcu_dereference_sched(tr->enter_syscall_files[syscall_nr]); if (!trace_file) - return; + goto end; if (trace_trigger_soft_disabled(trace_file)) - return; + goto end; sys_data = syscall_nr_to_meta(syscall_nr); if (!sys_data) - return; + goto end; size = sizeof(*entry) + sizeof(unsigned long) * sys_data->nb_args; entry = trace_event_buffer_reserve(&fbuffer, trace_file, size); if (!entry) - return; + goto end; entry = ring_buffer_event_data(fbuffer.event); entry->nr = syscall_nr; @@ -327,6 +333,8 @@ static void ftrace_syscall_enter(void *data, struct pt_regs *regs, long id) memcpy(entry->args, args, sizeof(unsigned long) * sys_data->nb_args); trace_event_buffer_commit(&fbuffer); +end: + preempt_enable_notrace(); } static void ftrace_syscall_exit(void *data, struct pt_regs *regs, long ret) @@ -338,31 +346,39 @@ static void ftrace_syscall_exit(void *data, struct pt_regs *regs, long ret) struct trace_event_buffer fbuffer; int syscall_nr; + /* + * Probe called with preemption enabled (may_fault), but ring buffer and + * per-cpu data require preemption to be disabled. + */ + preempt_disable_notrace(); + syscall_nr = trace_get_syscall_nr(current, regs); if (syscall_nr < 0 || syscall_nr >= NR_syscalls) - return; + goto end; /* Here we're inside tp handler's rcu_read_lock_sched (__DO_TRACE()) */ trace_file = rcu_dereference_sched(tr->exit_syscall_files[syscall_nr]); if (!trace_file) - return; + goto end; if (trace_trigger_soft_disabled(trace_file)) - return; + goto end; sys_data = syscall_nr_to_meta(syscall_nr); if (!sys_data) - return; + goto end; entry = trace_event_buffer_reserve(&fbuffer, trace_file, sizeof(*entry)); if (!entry) - return; + goto end; entry = ring_buffer_event_data(fbuffer.event); entry->nr = syscall_nr; entry->ret = syscall_get_return_value(current, regs); trace_event_buffer_commit(&fbuffer); +end: + preempt_enable_notrace(); } static int reg_event_syscall_enter(struct trace_event_file *file, @@ -377,7 +393,9 @@ static int reg_event_syscall_enter(struct trace_event_file *file, return -ENOSYS; mutex_lock(&syscall_trace_lock); if (!tr->sys_refcount_enter) - ret = register_trace_sys_enter(ftrace_syscall_enter, tr); + ret = register_trace_prio_flags_sys_enter(ftrace_syscall_enter, tr, + TRACEPOINT_DEFAULT_PRIO, + TRACEPOINT_MAY_FAULT); if (!ret) { rcu_assign_pointer(tr->enter_syscall_files[num], file); tr->sys_refcount_enter++; @@ -415,7 +433,9 @@ static int reg_event_syscall_exit(struct trace_event_file *file, return -ENOSYS; mutex_lock(&syscall_trace_lock); if (!tr->sys_refcount_exit) - ret = register_trace_sys_exit(ftrace_syscall_exit, tr); + ret = register_trace_prio_flags_sys_exit(ftrace_syscall_exit, tr, + TRACEPOINT_DEFAULT_PRIO, + TRACEPOINT_MAY_FAULT); if (!ret) { rcu_assign_pointer(tr->exit_syscall_files[num], file); tr->sys_refcount_exit++; @@ -582,20 +602,26 @@ static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id) int rctx; int size; + /* + * Probe called with preemption enabled (may_fault), but ring buffer and + * per-cpu data require preemption to be disabled. + */ + preempt_disable_notrace(); + syscall_nr = trace_get_syscall_nr(current, regs); if (syscall_nr < 0 || syscall_nr >= NR_syscalls) - return; + goto end; if (!test_bit(syscall_nr, enabled_perf_enter_syscalls)) - return; + goto end; sys_data = syscall_nr_to_meta(syscall_nr); if (!sys_data) - return; + goto end; head = this_cpu_ptr(sys_data->enter_event->perf_events); valid_prog_array = bpf_prog_array_valid(sys_data->enter_event); if (!valid_prog_array && hlist_empty(head)) - return; + goto end; /* get the size after alignment with the u32 buffer size field */ size = sizeof(unsigned long) * sys_data->nb_args + sizeof(*rec); @@ -604,7 +630,7 @@ static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id) rec = perf_trace_buf_alloc(size, NULL, &rctx); if (!rec) - return; + goto end; rec->nr = syscall_nr; syscall_get_arguments(current, regs, args); @@ -614,12 +640,14 @@ static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id) !perf_call_bpf_enter(sys_data->enter_event, regs, sys_data, rec)) || hlist_empty(head)) { perf_swevent_put_recursion_context(rctx); - return; + goto end; } perf_trace_buf_submit(rec, size, rctx, sys_data->enter_event->event.type, 1, regs, head, NULL); +end: + preempt_enable_notrace(); } static int perf_sysenter_enable(struct trace_event_call *call) @@ -631,7 +659,9 @@ static int perf_sysenter_enable(struct trace_event_call *call) mutex_lock(&syscall_trace_lock); if (!sys_perf_refcount_enter) - ret = register_trace_sys_enter(perf_syscall_enter, NULL); + ret = register_trace_prio_flags_sys_enter(perf_syscall_enter, NULL, + TRACEPOINT_DEFAULT_PRIO, + TRACEPOINT_MAY_FAULT); if (ret) { pr_info("event trace: Could not activate syscall entry trace point"); } else { @@ -682,20 +712,26 @@ static void perf_syscall_exit(void *ignore, struct pt_regs *regs, long ret) int rctx; int size; + /* + * Probe called with preemption enabled (may_fault), but ring buffer and + * per-cpu data require preemption to be disabled. + */ + preempt_disable_notrace(); + syscall_nr = trace_get_syscall_nr(current, regs); if (syscall_nr < 0 || syscall_nr >= NR_syscalls) - return; + goto end; if (!test_bit(syscall_nr, enabled_perf_exit_syscalls)) - return; + goto end; sys_data = syscall_nr_to_meta(syscall_nr); if (!sys_data) - return; + goto end; head = this_cpu_ptr(sys_data->exit_event->perf_events); valid_prog_array = bpf_prog_array_valid(sys_data->exit_event); if (!valid_prog_array && hlist_empty(head)) - return; + goto end; /* We can probably do that at build time */ size = ALIGN(sizeof(*rec) + sizeof(u32), sizeof(u64)); @@ -703,7 +739,7 @@ static void perf_syscall_exit(void *ignore, struct pt_regs *regs, long ret) rec = perf_trace_buf_alloc(size, NULL, &rctx); if (!rec) - return; + goto end; rec->nr = syscall_nr; rec->ret = syscall_get_return_value(current, regs); @@ -712,11 +748,13 @@ static void perf_syscall_exit(void *ignore, struct pt_regs *regs, long ret) !perf_call_bpf_exit(sys_data->exit_event, regs, rec)) || hlist_empty(head)) { perf_swevent_put_recursion_context(rctx); - return; + goto end; } perf_trace_buf_submit(rec, size, rctx, sys_data->exit_event->event.type, 1, regs, head, NULL); +end: + preempt_enable_notrace(); } static int perf_sysexit_enable(struct trace_event_call *call) @@ -728,7 +766,9 @@ static int perf_sysexit_enable(struct trace_event_call *call) mutex_lock(&syscall_trace_lock); if (!sys_perf_refcount_exit) - ret = register_trace_sys_exit(perf_syscall_exit, NULL); + ret = register_trace_prio_flags_sys_exit(perf_syscall_exit, NULL, + TRACEPOINT_DEFAULT_PRIO, + TRACEPOINT_MAY_FAULT); if (ret) { pr_info("event trace: Could not activate syscall exit trace point"); } else {