From patchwork Fri Apr 7 23:05:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Stultz X-Patchwork-Id: 81078 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp572401vqo; Fri, 7 Apr 2023 16:11:12 -0700 (PDT) X-Google-Smtp-Source: AKy350ZN1dgIgis1emNOoBJ3XdxbYbucZah5YpDPx2QwaZnf/VpXW9KSPYXrzs4f7rwXxcHnRXQq X-Received: by 2002:a05:6a20:6685:b0:da:2d16:db89 with SMTP id o5-20020a056a20668500b000da2d16db89mr4223563pzh.28.1680909072089; Fri, 07 Apr 2023 16:11:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680909072; cv=none; d=google.com; s=arc-20160816; b=FNWsjmSndGvjO0Bu+mmiD5JL4IOGkMCybXLqeztH3JQitbdLYTtdwbKHIvcuVNies5 vNjS3T7ZGYKA7/nCVGmNAYs0DZwI/5YG7ZGNz4yTcTxmPoaeoXy3QNKSvwwLTJWn1+gY rzYovs9PoyyV2TT3DrOeogoF3ypXfA3qomIxMkLkHvk67hBB0QmekM/CY73WI5StXL31 PIV4dvILEkShezmUiXmROItGWtwiy7Yk7YCA1qlNEn7f8wAQTiZAFvXjFEmrItEDuh3y NsrT0UfVt/aAxKn5vPHttDaY0SPl0HN+fkdUA6B+ztoLAWcMcm7EZUu5GXpmx4DIHSO0 f1BQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:mime-version:date :dkim-signature; bh=T+Aq5EGpv0TdcyR3mJ0rG0U8bcZUTZ5BKGaHK0JrYVc=; b=oBMPCGWWgXDsDWcNnktoZI4g0LOG7wIjlxhLD4RV1//t4g6pZNvc4MzIai4mmK9qIi 8BfESTp4w40vRd0XVWY8zZN28OtylLxQq9AVO8svaUWEG4FQHIu3RWazGx2B6fUExgg7 KmEIzV1inrHpfXB7mmeTOO13Pb4llyzYmVPCaluuJr8Yqj4kIu/2KOTr7LDQWRpvYu02 S4lZyUrkqx69E/N+YtRWQc7sYI+QJpN+HjXeGCyE3iPrRRMXXRmf7/gbj3qyu8tSimx/ 5wgQCrKvmoSPcQKY74vXKyHdnK0PXVEwmMDIvV0R1/pX6UH1T8PXl3gsO9qU4dZqbRmq 9sBA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=joThEOBH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id w71-20020a63824a000000b0050780e66602si4475290pgd.734.2023.04.07.16.10.59; Fri, 07 Apr 2023 16:11:12 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=joThEOBH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229679AbjDGXGH (ORCPT + 99 others); Fri, 7 Apr 2023 19:06:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59152 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229451AbjDGXGB (ORCPT ); Fri, 7 Apr 2023 19:06:01 -0400 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A8F66E07B for ; Fri, 7 Apr 2023 16:05:32 -0700 (PDT) Received: by mail-pj1-x1049.google.com with SMTP id s93-20020a17090a2f6600b0024670ac71caso141403pjd.4 for ; Fri, 07 Apr 2023 16:05:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680908732; x=1683500732; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=T+Aq5EGpv0TdcyR3mJ0rG0U8bcZUTZ5BKGaHK0JrYVc=; b=joThEOBHVXoPtxtwqaFrm5dG382oUEuC1Z7udKCQMpKFT/BWvfI6nQPyjLU1LUL3te mBpw/CtuNM6bTwwDxWFVQdel2qstpt+P+m0KSEjKzA9E3PppJLphsjyrfJ5wk4ihmOml TvxpslPz0hEt4peTxtTYvBgcR4ns/Xv53JrIA4P71lrj4d3hUW7HgxVuTeOugl/9aWHw 5P4p+m3t/xDpCLHV/HmizJ+/zeD6yq+8GlycZd+pdDDku7Nu7TF3T8m1rGCuzJBXPBvZ TuPavi2Mt4Led/cG7CKvhPJy1AYMopuj+yb4W7zxIq9/TUAKGi5gTCFR02m4UWbKeQ+b KNmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680908732; x=1683500732; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=T+Aq5EGpv0TdcyR3mJ0rG0U8bcZUTZ5BKGaHK0JrYVc=; b=CRzXcqWfOToiR1iju53KK9GnGckRXyGBdlxi9z3TnIV0rPJnaNPttUlvWqfBqLSLJu ZyKcTh0mUinO8vS6TgKNv/yKfmDfWKVgsReEqbhI/zU6HVZu1D4EWuHdA1Xv0YfeYGyj 4jkJaSl+fRmg2+txni+pdkbn+7ZJDvuMQH8UfduBQr/SWTfof+mdUZeUEAxg2PXMl8NU JRKtkqK7pWhlGiFP5VQkitjdDHh1R2P0qC+7f8ltyNn4ffYZKq5K3tt4IlF/ATWpssUq 6+V0DfsnJ/ToH0jcAYlCQt7+5qmBoyy2/lxMO1dioAYeSZgQOWr8X3E7/NbyA/1wrZ6X LCqQ== X-Gm-Message-State: AAQBX9fkPmmIPDLdjDmHm6SXWMU1TitQWyYqSpT1bMNsvjREMBTQTBh7 cloV40ePnmvCI9UEbHXt6ZgDqjKq7rRdbXbeuoqA+2jiukzn/j2H4ywVwglxxEpKg/QtFe9rNT8 NgukkxPLa4Ob94oIVEc80roYToLaFRyRBNmTePZ0sWWQZ/vXoL+RsYwJPI0t8yneoZCxSTUk= X-Received: from jstultz-noogler2.c.googlers.com ([fda3:e722:ac3:cc00:24:72f4:c0a8:600]) (user=jstultz job=sendgmr) by 2002:a63:58e:0:b0:50b:e80f:caff with SMTP id 136-20020a63058e000000b0050be80fcaffmr3764515pgf.0.1680908731909; Fri, 07 Apr 2023 16:05:31 -0700 (PDT) Date: Fri, 7 Apr 2023 23:05:26 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.40.0.577.gac1e443424-goog Message-ID: <20230407230526.1685443-1-jstultz@google.com> Subject: [RESEND x4][PATCH v3] trace: Add trace points for tasklet entry/exit From: John Stultz To: LKML Cc: Lingutla Chandrasekhar , Steven Rostedt , Masami Hiramatsu , Thomas Gleixner , "Paul E. McKenney" , kernel-team@android.com, "J . Avila" , John Stultz X-Spam-Status: No, score=-7.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762560911266898433?= X-GMAIL-MSGID: =?utf-8?q?1762560911266898433?= From: Lingutla Chandrasekhar Tasklets are supposed to finish their work quickly and should not block the current running process, but it is not guaranteed that. Currently softirq_entry/exit can be used to know total tasklets execution time, but not helpful to track individual tasklet's execution time. With that we can't find any culprit tasklet function, which is taking more time. Add tasklet_entry/exit trace point support to track individual tasklet execution. Trivial usage example: # echo 1 > /sys/kernel/debug/tracing/events/irq/tasklet_entry/enable # echo 1 > /sys/kernel/debug/tracing/events/irq/tasklet_exit/enable # cat /sys/kernel/debug/tracing/trace # tracer: nop # # entries-in-buffer/entries-written: 4/4 #P:4 # # _-----=> irqs-off/BH-disabled # / _----=> need-resched # | / _---=> hardirq/softirq # || / _--=> preempt-depth # ||| / _-=> migrate-disable # |||| / delay # TASK-PID CPU# ||||| TIMESTAMP FUNCTION # | | | ||||| | | -0 [003] ..s1. 314.011428: tasklet_entry: tasklet=0xffffa01ef8db2740 function=tcp_tasklet_func -0 [003] ..s1. 314.011432: tasklet_exit: tasklet=0xffffa01ef8db2740 function=tcp_tasklet_func -0 [003] ..s1. 314.017369: tasklet_entry: tasklet=0xffffa01ef8db2740 function=tcp_tasklet_func -0 [003] ..s1. 314.017371: tasklet_exit: tasklet=0xffffa01ef8db2740 function=tcp_tasklet_func This patch has been carried in the Android tree for awhile so I wanted to submit it for review upstream. Feedback would be appreciated! Cc: Steven Rostedt Cc: Masami Hiramatsu Cc: Thomas Gleixner Cc: "Paul E. McKenney" Cc: kernel-team@android.com Reviewed-by: Steven Rostedt (Google) Signed-off-by: Lingutla Chandrasekhar [elavila: Port to android-mainline] Signed-off-by: J. Avila [jstultz: Rebased to upstream, cut unused trace points, added comments for the tracepoints, reworded commit] Signed-off-by: John Stultz --- "Helloooooo..." he yelled into the void. v2: * Added tasklet pointer to the trace event as suggested by Steven v3: * Minor tweak to commit to show usage and output example --- include/trace/events/irq.h | 47 ++++++++++++++++++++++++++++++++++++++ kernel/softirq.c | 9 ++++++-- 2 files changed, 54 insertions(+), 2 deletions(-) diff --git a/include/trace/events/irq.h b/include/trace/events/irq.h index eeceafaaea4c..a07b4607b663 100644 --- a/include/trace/events/irq.h +++ b/include/trace/events/irq.h @@ -160,6 +160,53 @@ DEFINE_EVENT(softirq, softirq_raise, TP_ARGS(vec_nr) ); +DECLARE_EVENT_CLASS(tasklet, + + TP_PROTO(struct tasklet_struct *t, void *func), + + TP_ARGS(t, func), + + TP_STRUCT__entry( + __field( void *, tasklet) + __field( void *, func) + ), + + TP_fast_assign( + __entry->tasklet = t; + __entry->func = func; + ), + + TP_printk("tasklet=%ps function=%ps", __entry->tasklet, __entry->func) +); + +/** + * tasklet_entry - called immediately before the tasklet is run + * @t: tasklet pointer + * @func: tasklet callback or function being run + * + * Used to find individual tasklet execution time + */ +DEFINE_EVENT(tasklet, tasklet_entry, + + TP_PROTO(struct tasklet_struct *t, void *func), + + TP_ARGS(t, func) +); + +/** + * tasklet_exit - called immediately after the tasklet is run + * @t: tasklet pointer + * @func: tasklet callback or function being run + * + * Used to find individual tasklet execution time + */ +DEFINE_EVENT(tasklet, tasklet_exit, + + TP_PROTO(struct tasklet_struct *t, void *func), + + TP_ARGS(t, func) +); + #endif /* _TRACE_IRQ_H */ /* This part must be outside protection */ diff --git a/kernel/softirq.c b/kernel/softirq.c index c8a6913c067d..1b725510dd0f 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -793,10 +793,15 @@ static void tasklet_action_common(struct softirq_action *a, if (tasklet_trylock(t)) { if (!atomic_read(&t->count)) { if (tasklet_clear_sched(t)) { - if (t->use_callback) + if (t->use_callback) { + trace_tasklet_entry(t, t->callback); t->callback(t); - else + trace_tasklet_exit(t, t->callback); + } else { + trace_tasklet_entry(t, t->func); t->func(t->data); + trace_tasklet_exit(t, t->func); + } } tasklet_unlock(t); continue;