From patchwork Wed Mar 22 23:27:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Stultz X-Patchwork-Id: 73694 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp2627050wrt; Wed, 22 Mar 2023 16:33:51 -0700 (PDT) X-Google-Smtp-Source: AK7set95Waz8ZJZe//opK7Nswt1Cb9DFm4YJJgj0SgouOh7NhctWw7VvsgOB6s/JlBINy75xlqLi X-Received: by 2002:aa7:9f45:0:b0:590:921d:5740 with SMTP id h5-20020aa79f45000000b00590921d5740mr4292669pfr.13.1679528030724; Wed, 22 Mar 2023 16:33:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679528030; cv=none; d=google.com; s=arc-20160816; b=XmTafL6rWdok56a30sBLBlXfqS4dUdYdYhqOGi4O80hCZ0QZfD9gKbH5l5b2ZCdOMs H4XxlayTetHTPtdulGmoOPwpkxa12QUj+PerMNQPlPBzs+N1FJd+dV3LRGfpBX66wIan 2oHdympUtGfrnUlqswIx5THhhc44jnFCvTTrG5Td2anp06IXbG+WYcHOSrduS1oA8p4r mvYlOEpzHDZB/dCi0nswMe/uED+xKASFs6f29IJ9FO6O5DFnpZlNUGgL0kwKGZ/XIotm 7J+XjYklxp2SSXMaTuKfQeWiCKvSOjzP3XqL3p2w+zmUs6EfiJO8n9dMLwrR3zJ4LKvS TTKQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:mime-version:date :dkim-signature; bh=F8gJn/RqHZd9pZyBGjIkjsEpwHztmYNBKyUX9XcX/sw=; b=muqxUNCpE5RH9Jt7YZNRP2CRl4R4y9/cLlWX5bteAuAZTGiwgfkWBWy04kDL7J9L3r cGzEKHBe13mXWUEOLxHoyd+5m5GMoAIE2panJgIdGcrcEVcDZADXbJ3FchlJCVNgYZvq 81KubgByhdCrxcPa9iVLyP2N7721snP/ynv6QNUab8uzIqin/eGyQTOEQEjo+ShVltw2 kMMowIe1vGn+RS0cmtyDp6E3MqMIjUgV6G8TDx5r/IH9i6aBqv/SENIrpAq97xSPsb19 uvSpPlUlHFAQkLSnbhezwT0flSzuWFhRIAYfyBNiTnzd6wpa1fdwUvGj+MapP12/rJym tAjw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=FxEe+6vQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t18-20020a62d152000000b005a910ae9a76si17007306pfl.305.2023.03.22.16.33.37; Wed, 22 Mar 2023 16:33:50 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=FxEe+6vQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229757AbjCVX17 (ORCPT + 99 others); Wed, 22 Mar 2023 19:27:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40354 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229480AbjCVX15 (ORCPT ); Wed, 22 Mar 2023 19:27:57 -0400 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C3E6C2D63 for ; Wed, 22 Mar 2023 16:27:56 -0700 (PDT) Received: by mail-pj1-x104a.google.com with SMTP id j15-20020a17090a318f00b0023fe33f8825so2017258pjb.9 for ; Wed, 22 Mar 2023 16:27:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679527676; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=F8gJn/RqHZd9pZyBGjIkjsEpwHztmYNBKyUX9XcX/sw=; b=FxEe+6vQbntvDlAA9NkLZloGQ6PHegmT23ZDI4/wuFQsobb2Hd7xUlVF+lkQgFerOp wDdH8g01ny0Q5nZZ+uJJ/TbvDqi4wtbQq7XVj8JmJhG9dOfKC11pGzUGiZAq0nz89N7F ThqUFa5qa509TFg/ejvn1G76vsbp7JDh4ZocaKSSvM49oY1EMygMACUnSs05wDlyDo+e oiJSRbrm3vjktkOMc5Z/FpU0ZJPCjPkmpE9VQ9jHD1jYfmatfHlTFYZogZkJODh7hOMn CobVaXbhwl08hMvBIghDPhHQoi/E/7AlELCnhLQ32ykTfpGEEE6o/0QJQGROetYwQZcF 50xQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679527676; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=F8gJn/RqHZd9pZyBGjIkjsEpwHztmYNBKyUX9XcX/sw=; b=eqeS8vSQ0plyrlFw1nlWNFOsTBZOnaoSgJqS+kRopW0e/KKt7IiT9iVIOpFsanLtwR pxQxTJs4I88eIrMjPD55aZdCjDPDr5ZoFeO95YBMC0XqkYa5QgZ88FLjpYX7tvx/FPu8 6ydVdDSwJo5K+sHcuXXuFAoixaVZONk0/4GM2WQ/P6zWFX//B3PX/Ed+ROml2ux6HkhC WZhigw+DDL6G3A2xUfGY3OwbhNkEw70rVqwpAqJc3cg9QvoLj5PqBzHvQ7sGovY9GWeB 26GdbidyE01AQqdSw7wsKzKT1+ZtvNBIxSsCT0sxS562dKZamTpMFf4RWr+6jFJ2BiHa rAZw== X-Gm-Message-State: AO0yUKVE5m7eg1c5Usy+/UB6IbZVsa0X8xIKZQJIg3dx3eGrDMXplH4E 3og9BHg96I9IVnFrcTbntig2RAu+IAMDq4a0X0cGTN5t+irz+IGL2aGwk7EWQYdVRAltzFo+BRX ELhOwn9cq4IrCeGOwwfxET1SITGXigUre/3u3EdwFYgoSaB+LixYtbFWTTra7RKpKAcQ5v0E= X-Received: from jstultz-noogler2.c.googlers.com ([fda3:e722:ac3:cc00:24:72f4:c0a8:600]) (user=jstultz job=sendgmr) by 2002:a05:6a00:2d0c:b0:627:df77:8303 with SMTP id fa12-20020a056a002d0c00b00627df778303mr2627788pfb.5.1679527676079; Wed, 22 Mar 2023 16:27:56 -0700 (PDT) Date: Wed, 22 Mar 2023 23:27:53 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog Message-ID: <20230322232753.2789341-1-jstultz@google.com> Subject: [RESEND x3][PATCH v3] trace: Add trace points for tasklet entry/exit From: John Stultz To: LKML Cc: Lingutla Chandrasekhar , Steven Rostedt , Masami Hiramatsu , Thomas Gleixner , "Paul E. McKenney" , kernel-team@android.com, "J . Avila" , John Stultz X-Spam-Status: No, score=-7.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761112784148501544?= X-GMAIL-MSGID: =?utf-8?q?1761112784148501544?= From: Lingutla Chandrasekhar Tasklets are supposed to finish their work quickly and should not block the current running process, but it is not guaranteed that. Currently softirq_entry/exit can be used to know total tasklets execution time, but not helpful to track individual tasklet's execution time. With that we can't find any culprit tasklet function, which is taking more time. Add tasklet_entry/exit trace point support to track individual tasklet execution. Trivial usage example: # echo 1 > /sys/kernel/debug/tracing/events/irq/tasklet_entry/enable # echo 1 > /sys/kernel/debug/tracing/events/irq/tasklet_exit/enable # cat /sys/kernel/debug/tracing/trace # tracer: nop # # entries-in-buffer/entries-written: 4/4 #P:4 # # _-----=> irqs-off/BH-disabled # / _----=> need-resched # | / _---=> hardirq/softirq # || / _--=> preempt-depth # ||| / _-=> migrate-disable # |||| / delay # TASK-PID CPU# ||||| TIMESTAMP FUNCTION # | | | ||||| | | -0 [003] ..s1. 314.011428: tasklet_entry: tasklet=0xffffa01ef8db2740 function=tcp_tasklet_func -0 [003] ..s1. 314.011432: tasklet_exit: tasklet=0xffffa01ef8db2740 function=tcp_tasklet_func -0 [003] ..s1. 314.017369: tasklet_entry: tasklet=0xffffa01ef8db2740 function=tcp_tasklet_func -0 [003] ..s1. 314.017371: tasklet_exit: tasklet=0xffffa01ef8db2740 function=tcp_tasklet_func This patch has been carried in the Android tree for awhile so I wanted to submit it for review upstream. Feedback would be appreciated! Cc: Steven Rostedt Cc: Masami Hiramatsu Cc: Thomas Gleixner Cc: "Paul E. McKenney" Cc: kernel-team@android.com Reviewed-by: Steven Rostedt (Google) Signed-off-by: Lingutla Chandrasekhar [elavila: Port to android-mainline] Signed-off-by: J. Avila [jstultz: Rebased to upstream, cut unused trace points, added comments for the tracepoints, reworded commit] Signed-off-by: John Stultz --- v2: * Added tasklet pointer to the trace event as suggested by Steven v3: * Minor tweak to commit to show usage and output example --- include/trace/events/irq.h | 47 ++++++++++++++++++++++++++++++++++++++ kernel/softirq.c | 9 ++++++-- 2 files changed, 54 insertions(+), 2 deletions(-) diff --git a/include/trace/events/irq.h b/include/trace/events/irq.h index eeceafaaea4c..a07b4607b663 100644 --- a/include/trace/events/irq.h +++ b/include/trace/events/irq.h @@ -160,6 +160,53 @@ DEFINE_EVENT(softirq, softirq_raise, TP_ARGS(vec_nr) ); +DECLARE_EVENT_CLASS(tasklet, + + TP_PROTO(struct tasklet_struct *t, void *func), + + TP_ARGS(t, func), + + TP_STRUCT__entry( + __field( void *, tasklet) + __field( void *, func) + ), + + TP_fast_assign( + __entry->tasklet = t; + __entry->func = func; + ), + + TP_printk("tasklet=%ps function=%ps", __entry->tasklet, __entry->func) +); + +/** + * tasklet_entry - called immediately before the tasklet is run + * @t: tasklet pointer + * @func: tasklet callback or function being run + * + * Used to find individual tasklet execution time + */ +DEFINE_EVENT(tasklet, tasklet_entry, + + TP_PROTO(struct tasklet_struct *t, void *func), + + TP_ARGS(t, func) +); + +/** + * tasklet_exit - called immediately after the tasklet is run + * @t: tasklet pointer + * @func: tasklet callback or function being run + * + * Used to find individual tasklet execution time + */ +DEFINE_EVENT(tasklet, tasklet_exit, + + TP_PROTO(struct tasklet_struct *t, void *func), + + TP_ARGS(t, func) +); + #endif /* _TRACE_IRQ_H */ /* This part must be outside protection */ diff --git a/kernel/softirq.c b/kernel/softirq.c index c8a6913c067d..1b725510dd0f 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -793,10 +793,15 @@ static void tasklet_action_common(struct softirq_action *a, if (tasklet_trylock(t)) { if (!atomic_read(&t->count)) { if (tasklet_clear_sched(t)) { - if (t->use_callback) + if (t->use_callback) { + trace_tasklet_entry(t, t->callback); t->callback(t); - else + trace_tasklet_exit(t, t->callback); + } else { + trace_tasklet_entry(t, t->func); t->func(t->data); + trace_tasklet_exit(t, t->func); + } } tasklet_unlock(t); continue;