From patchwork Thu Feb 16 20:24:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Stultz X-Patchwork-Id: 58225 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp512897wrn; Thu, 16 Feb 2023 12:27:24 -0800 (PST) X-Google-Smtp-Source: AK7set8ZgZp+ydO6SiVIM+cG+6JfRBPV4AsrGGmFK/MslmuwiOQ98HIVKisoDNF8B2Wzaz0RopJF X-Received: by 2002:aa7:8e88:0:b0:5a8:d97d:c397 with SMTP id a8-20020aa78e88000000b005a8d97dc397mr5683825pfr.24.1676579244438; Thu, 16 Feb 2023 12:27:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676579244; cv=none; d=google.com; s=arc-20160816; b=DoW7XNtniM+fXSU6mfCwksh9HEyUNwqE+1vRZYO5pEVUJ5DQT2nwz8OW5G+mlGEEeV jJT170pAiZAllDytcNknOdSB4sggofuQO0c3XbRaI7pDQLjIgWlJX3d8D0Ob6Vj0QZFG 1UnLrKuM0n/8n14l0UEBR/Ih5ej78l44JnTfz4f9B036AtYKO6VktJgsxNpEwH8sE+Mo sCAJtPOoHvGA86REaz63TEb6mUMTAG4XlBJRF3B3n8cV5BmPlfW5RqPLuXKyYjm4HTRM HM0fetqFIbLI9ziEI5NEGAOsGL8hVth1SEhFv851G9ONSB9H2SsaKRYx9DzjJv84CsIK n9dg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:mime-version:date :dkim-signature; bh=ueiGzinkCypCvrWOeR0IuYP20VC8RlC4hQfGrf/zFWw=; b=QfiyMeAWIIDFrAU36zrdAMiJbKazasGFWYTm1kfIaf5R3ayQvUvdIssmRGOUvMI1V8 PhYPTyDY7Gm2p6vF8Z9cLo+cL2NSQ0k1naEbJIbUWHe20fGU6EY/Sry4zDiF5g+pPzJV OEVSEr0REtmh74a9Kxg9ZjwpqCq8D2R7t3vaOAiTJ16S2CQwu+mb/OKP6cbAhHFfKX20 OtYK6ViTp/YPNUVHZ59RaAkjk0y+hNrapFt/3867U1PlXhp0fDos74C2uHhm3tyBcMUp HoZneRfIUuA5s+/Z+O2i4IssowYSlpnGSHiLNTJtOmX08KLeoEskOLT2WzTEr5QacUF4 NV7g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=DLhaRK8i; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id z11-20020aa7888b000000b005a816064ddasi2718719pfe.158.2023.02.16.12.27.10; Thu, 16 Feb 2023 12:27:24 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=DLhaRK8i; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229962AbjBPUYy (ORCPT + 99 others); Thu, 16 Feb 2023 15:24:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43938 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229834AbjBPUYx (ORCPT ); Thu, 16 Feb 2023 15:24:53 -0500 Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7F3323B21F for ; Thu, 16 Feb 2023 12:24:46 -0800 (PST) Received: by mail-pf1-x44a.google.com with SMTP id y28-20020a056a001c9c00b005a8c5cd5ae9so1785943pfw.22 for ; Thu, 16 Feb 2023 12:24:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=ueiGzinkCypCvrWOeR0IuYP20VC8RlC4hQfGrf/zFWw=; b=DLhaRK8iZ8L8olm6SGgsxFJxxk/urNJrFy25qWiopGlpMj+l3Ug3LCsj0tHmvXIBgY 4x2scIIy+kj5kOmi91fhM60dKNLVeqoKxKvAQx9pP4QqXz9lzveJCt63QBZvgq7btEen sAacLk/3doGXB8A9vg3s2q0tNJHGlkPiMIvLwd323PFdoj6ZP+0t97wcxJ0wiuAy6SaH Pwf0i7V0FkoZssaiEtZmkESaHbyI6Odwfzcw4shH7MSew5cFGr2+VfTFaAOYKBmgujFy eUkrVYK/Ec5ENcgrw1DtY67eKzzi3+Uvcr+sXopkkAW5SRG1u7NEaUk4b2jnbKRVtkN5 cMCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=ueiGzinkCypCvrWOeR0IuYP20VC8RlC4hQfGrf/zFWw=; b=DlunuVXcFL5R/spBgUeC98dbPR3SAlljtdnTHwIYU996fDJFIWMEKLLHGlQkaochyW yqLf/o05vExHu1EsOCStmq2hf+zVn7w2IHMEpIx0bf3BcnVUm36+7NsN1VtQzJoQTk4b LnDP82BzI9sayMteAqLcs8ermujUlusz09rsGvI92g8xz4OAz2sioT8c4eTPyV/OSVo9 48vliY+wygAXzU7ZKXyWUiA400x6KVOpQ2sjJvxO8wpOyNJw1X90AXZGhCq2h/c97EVG 1OgSz2lkSUuqwDo2doBITC0jXVqjYXiDkcAg0ab/OO/cMA6OX9NRDYXq4MFs7w6LqwWG tWaQ== X-Gm-Message-State: AO0yUKUioQYedGnwz8eqInO/LU8vEaFZvEPA5BHqzGiMbP8yIUBlHbKZ 5Prpk/LwhssDa/OWpUTXqQW2d75HQ2B5x/Y04SY1C2l3Ysqv5yqIsDXBrSvsgdiCAvuv6zNPaSv 7Py07coZDU4JrBK9p0C8DTbiwxYQgvyc6u5LPZiAJW71TBlHRHcwq1X9iG51ViFSUkZ/gtzA= X-Received: from jstultz-noogler2.c.googlers.com ([fda3:e722:ac3:cc00:24:72f4:c0a8:600]) (user=jstultz job=sendgmr) by 2002:a63:770a:0:b0:4fb:8fb6:37e3 with SMTP id s10-20020a63770a000000b004fb8fb637e3mr1080059pgc.6.1676579085861; Thu, 16 Feb 2023 12:24:45 -0800 (PST) Date: Thu, 16 Feb 2023 20:24:42 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.39.2.637.g21b0678d19-goog Message-ID: <20230216202442.698394-1-jstultz@google.com> Subject: [RESEND x2][PATCH v3] trace: Add trace points for tasklet entry/exit From: John Stultz To: LKML Cc: Lingutla Chandrasekhar , Steven Rostedt , Masami Hiramatsu , Thomas Gleixner , "Paul E. McKenney" , kernel-team@android.com, "J . Avila" , John Stultz X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758020757739831191?= X-GMAIL-MSGID: =?utf-8?q?1758020757739831191?= From: Lingutla Chandrasekhar Tasklets are supposed to finish their work quickly and should not block the current running process, but it is not guaranteed that. Currently softirq_entry/exit can be used to know total tasklets execution time, but not helpful to track individual tasklet's execution time. With that we can't find any culprit tasklet function, which is taking more time. Add tasklet_entry/exit trace point support to track individual tasklet execution. Trivial usage example: # echo 1 > /sys/kernel/debug/tracing/events/irq/tasklet_entry/enable # echo 1 > /sys/kernel/debug/tracing/events/irq/tasklet_exit/enable # cat /sys/kernel/debug/tracing/trace # tracer: nop # # entries-in-buffer/entries-written: 4/4 #P:4 # # _-----=> irqs-off/BH-disabled # / _----=> need-resched # | / _---=> hardirq/softirq # || / _--=> preempt-depth # ||| / _-=> migrate-disable # |||| / delay # TASK-PID CPU# ||||| TIMESTAMP FUNCTION # | | | ||||| | | -0 [003] ..s1. 314.011428: tasklet_entry: tasklet=0xffffa01ef8db2740 function=tcp_tasklet_func -0 [003] ..s1. 314.011432: tasklet_exit: tasklet=0xffffa01ef8db2740 function=tcp_tasklet_func -0 [003] ..s1. 314.017369: tasklet_entry: tasklet=0xffffa01ef8db2740 function=tcp_tasklet_func -0 [003] ..s1. 314.017371: tasklet_exit: tasklet=0xffffa01ef8db2740 function=tcp_tasklet_func This patch has been carried in the Android tree for awhile so I wanted to submit it for review upstream. Feedback would be appreciated! Cc: Steven Rostedt Cc: Masami Hiramatsu Cc: Thomas Gleixner Cc: "Paul E. McKenney" Cc: kernel-team@android.com Reviewed-by: Steven Rostedt (Google) Signed-off-by: Lingutla Chandrasekhar [elavila: Port to android-mainline] Signed-off-by: J. Avila [jstultz: Rebased to upstream, cut unused trace points, added comments for the tracepoints, reworded commit] Signed-off-by: John Stultz --- v2: * Added tasklet pointer to the trace event as suggested by Steven v3: * Minor tweak to commit to show usage and output example --- include/trace/events/irq.h | 47 ++++++++++++++++++++++++++++++++++++++ kernel/softirq.c | 9 ++++++-- 2 files changed, 54 insertions(+), 2 deletions(-) diff --git a/include/trace/events/irq.h b/include/trace/events/irq.h index eeceafaaea4c..a07b4607b663 100644 --- a/include/trace/events/irq.h +++ b/include/trace/events/irq.h @@ -160,6 +160,53 @@ DEFINE_EVENT(softirq, softirq_raise, TP_ARGS(vec_nr) ); +DECLARE_EVENT_CLASS(tasklet, + + TP_PROTO(struct tasklet_struct *t, void *func), + + TP_ARGS(t, func), + + TP_STRUCT__entry( + __field( void *, tasklet) + __field( void *, func) + ), + + TP_fast_assign( + __entry->tasklet = t; + __entry->func = func; + ), + + TP_printk("tasklet=%ps function=%ps", __entry->tasklet, __entry->func) +); + +/** + * tasklet_entry - called immediately before the tasklet is run + * @t: tasklet pointer + * @func: tasklet callback or function being run + * + * Used to find individual tasklet execution time + */ +DEFINE_EVENT(tasklet, tasklet_entry, + + TP_PROTO(struct tasklet_struct *t, void *func), + + TP_ARGS(t, func) +); + +/** + * tasklet_exit - called immediately after the tasklet is run + * @t: tasklet pointer + * @func: tasklet callback or function being run + * + * Used to find individual tasklet execution time + */ +DEFINE_EVENT(tasklet, tasklet_exit, + + TP_PROTO(struct tasklet_struct *t, void *func), + + TP_ARGS(t, func) +); + #endif /* _TRACE_IRQ_H */ /* This part must be outside protection */ diff --git a/kernel/softirq.c b/kernel/softirq.c index c8a6913c067d..1b725510dd0f 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -793,10 +793,15 @@ static void tasklet_action_common(struct softirq_action *a, if (tasklet_trylock(t)) { if (!atomic_read(&t->count)) { if (tasklet_clear_sched(t)) { - if (t->use_callback) + if (t->use_callback) { + trace_tasklet_entry(t, t->callback); t->callback(t); - else + trace_tasklet_exit(t, t->callback); + } else { + trace_tasklet_entry(t, t->func); t->func(t->data); + trace_tasklet_exit(t, t->func); + } } tasklet_unlock(t); continue;