From patchwork Thu May 11 18:19:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tejun Heo X-Patchwork-Id: 92804 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp4581753vqo; Thu, 11 May 2023 11:26:22 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7C76bhSSaoE2Og1E9U4xxeqQMuOYW5xWceoDNSWufCee7ppu5OqnPxuf4GrUnDMXoo9lse X-Received: by 2002:a17:90b:4ac8:b0:24e:507:7408 with SMTP id mh8-20020a17090b4ac800b0024e05077408mr21880107pjb.37.1683829581776; Thu, 11 May 2023 11:26:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683829581; cv=none; d=google.com; s=arc-20160816; b=MzMy1PRyL5pcSQPHyYmHDdz34T2HjPAyxIbzQfwXK3Ovov+gz3PAw+Bla8RLrYQEWa PAdrLSffoWuo97YNCWJKHcmOtm4I7k9HlQh2Tb7L/mDjSYNy+AHj+UrbBU6T37DnA2mX 4XDqxjZfTYBpKeVSPra+sY+Lpl+Dul+mx9DkkDgmS8MvnyWlTXqiRUjJaptXSWt7iW2I fJp2MgvlKxp3v64LAXfFWps38vS5rLHoCx9f0D/PegfA5vgyR3ghn2oWwyr6rl4La9Wj /+zzeIcTvwyM1DYmb8xnQw4dVKM6hlObSz7LrVwzS/ol1lmXaVC17jhNZTSJRSdcipaa XcDg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from:sender :dkim-signature; bh=5nDUioqnzKbpPmnws2aJ7UXESX8vQy0kJwI44IzqgUo=; b=iLzJTVhabhGy+lbhd9Io4TLwssQUPYdcSCL2/9wxB+tNCOFGHxrTOFMMZKQnuj8T6m rfCIL8bYYgrTgalSwLfRj+kymgWy6KnQ5alo9QrgQvHBYIBx/eM9ArixHmljsGg2sGbe nlepp4cybq1lRN29sa65sh0CCrlEnpKQ2YPcJbRUvekfZPRJ16n5OscdQ+VZlfCztU9G J8gfX3QnGNVcKliw3TkAVrtBWPz1IZjVxIIBZ6ujFKr27fkov8b/2tiJnO9QMf11Cuis vNWQ+jSAaeYkeCDyTGCxmC96kTEFqEqDJPMy2hT/8OKLeXXYs7RRxhtTtjbaJiwigSuu GROg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=A4Jz+dPF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id bm18-20020a656e92000000b0052c999cd20csi5135086pgb.899.2023.05.11.11.26.08; Thu, 11 May 2023 11:26:21 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=A4Jz+dPF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239200AbjEKSTo (ORCPT + 99 others); Thu, 11 May 2023 14:19:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43768 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238819AbjEKSTl (ORCPT ); Thu, 11 May 2023 14:19:41 -0400 Received: from mail-pf1-x430.google.com (mail-pf1-x430.google.com [IPv6:2607:f8b0:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2533040ED for ; Thu, 11 May 2023 11:19:39 -0700 (PDT) Received: by mail-pf1-x430.google.com with SMTP id d2e1a72fcca58-6439df6c268so5629248b3a.0 for ; Thu, 11 May 2023 11:19:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1683829178; x=1686421178; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:from:to:cc:subject:date :message-id:reply-to; bh=5nDUioqnzKbpPmnws2aJ7UXESX8vQy0kJwI44IzqgUo=; b=A4Jz+dPFoAkbSpYUNmHUl2WYuA2WEPyFnHj32rFAl2XuvsPIKUfIfGXoDoPyEJXKun hbri3vFZ9KWUsstoplUSqICpYRmt4e+a24aJP9kdFjNS09Cloca5XkMIhB2pcEp5+aiX hjbLWIuOcdbuAToGn4jpPSEZkS46c6nqS+UcScMmnac1vAJaLU1SQT3NMV6a7G7Q9tkf uU/wDKra8QjPW4DuPEOGb258niWCEQLhA3EW/DCSKZs5Q30D4JPW6EkAwUmtHvveEoTo Q6+9XexvTw617RkKDGI7fsHDmfscT8GkWd4MiQ8UnXLmkMLIn7EcTUiDezWA4Smzmbak 7OPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683829178; x=1686421178; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=5nDUioqnzKbpPmnws2aJ7UXESX8vQy0kJwI44IzqgUo=; b=b/2V8jZTZ+4Oe5ZNrWWBG/B2ts0fg8joUTPzAmSBFP86Myjq7U6Jx/MObhAo3Mm9dy BckyY6YIYxHT0HarE+5bc/Lax/Tda6M9V/Q4ktNcSN/oWmupqNRWST3N1Wvmunc8Gapk Y1OXltSZ2WKKyiBZH4gSpBq27ToeXQoG/l78f1bEbhoz3gjNSRvF0PCewlZe05IU63Ks r7NT2oi45YZueBC1zaYxG4cHSvbA9zbWyTgrtiHxi8nghrIRqlpa6kHApgfYeJWUwNYL nCS8rs4a7dia7mcIqA0Rh+MamEItpjhj8+2OdnsNXMu27u5bDEuldz2McQ8PNIdTJPhF kpvA== X-Gm-Message-State: AC+VfDwvwAHStt8P/5hCRCQ0JYBTnZDpbEew18qTX0Y5/Wu4X5ij41fN WE8lh3OVt1Qbp2zcDgrPu2M= X-Received: by 2002:a05:6a20:54a2:b0:100:8592:9a7f with SMTP id i34-20020a056a2054a200b0010085929a7fmr20152423pzk.45.1683829178247; Thu, 11 May 2023 11:19:38 -0700 (PDT) Received: from localhost (2603-800c-1a02-1bae-a7fa-157f-969a-4cde.res6.spectrum.com. [2603:800c:1a02:1bae:a7fa:157f:969a:4cde]) by smtp.gmail.com with ESMTPSA id s4-20020a62e704000000b0063b8d21be5asm5772885pfh.147.2023.05.11.11.19.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 May 2023 11:19:37 -0700 (PDT) Sender: Tejun Heo From: Tejun Heo To: jiangshanlai@gmail.com Cc: torvalds@linux-foundation.org, peterz@infradead.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, Tejun Heo Subject: [PATCH 1/7] workqueue: Add pwq->stats[] and a monitoring script Date: Thu, 11 May 2023 08:19:25 -1000 Message-Id: <20230511181931.869812-2-tj@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230511181931.869812-1-tj@kernel.org> References: <20230511181931.869812-1-tj@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765623287167125503?= X-GMAIL-MSGID: =?utf-8?q?1765623287167125503?= Currently, the only way to peer into workqueue operations is through tracing. While possible, it isn't easy or convenient to monitor per-workqueue behaviors over time this way. Let's add pwq->stats[] that track relevant events and a drgn monitoring script - tools/workqueue/wq_monitor.py. It's arguable whether this needs to be configurable. However, it currently only has several counters and the runtime overhead shouldn't be noticeable given that they're on pwq's which are per-cpu on per-cpu workqueues and per-numa-node on unbound ones. Let's keep it simple for the time being. v2: Patch reordered to earlier with fewer fields. Field will be added back gradually. Help message improved. Signed-off-by: Tejun Heo Cc: Lai Jiangshan --- Documentation/core-api/workqueue.rst | 32 ++++++ kernel/workqueue.c | 24 ++++- tools/workqueue/wq_monitor.py | 151 +++++++++++++++++++++++++++ 3 files changed, 206 insertions(+), 1 deletion(-) create mode 100644 tools/workqueue/wq_monitor.py diff --git a/Documentation/core-api/workqueue.rst b/Documentation/core-api/workqueue.rst index 8ec4d6270b24..7e5c39310bbf 100644 --- a/Documentation/core-api/workqueue.rst +++ b/Documentation/core-api/workqueue.rst @@ -348,6 +348,37 @@ Guidelines level of locality in wq operations and work item execution. +Monitoring +========== + +Use tools/workqueue/wq_monitor.py to monitor workqueue operations: :: + + $ tools/workqueue/wq_monitor.py events + total infl CMwake mayday rescued + events 18545 0 5 - - + events_highpri 8 0 0 - - + events_long 3 0 0 - - + events_unbound 38306 0 - - - + events_freezable 0 0 0 - - + events_power_efficient 29598 0 0 - - + events_freezable_power_ 10 0 0 - - + sock_diag_events 0 0 0 - - + + total infl CMwake mayday rescued + events 18548 0 5 - - + events_highpri 8 0 0 - - + events_long 3 0 0 - - + events_unbound 38322 0 - - - + events_freezable 0 0 0 - - + events_power_efficient 29603 0 0 - - + events_freezable_power_ 10 0 0 - - + sock_diag_events 0 0 0 - - + + ... + +See the command's help message for more info. + + Debugging ========= @@ -387,6 +418,7 @@ For the second type of problems it should be possible to just check The work item's function should be trivially visible in the stack trace. + Non-reentrance Conditions ========================= diff --git a/kernel/workqueue.c b/kernel/workqueue.c index 4666a1a92a31..33ddfaacdf29 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -199,6 +199,20 @@ struct worker_pool { struct rcu_head rcu; }; +/* + * Per-pool_workqueue statistics. These can be monitored using + * tools/workqueue/wq_monitor.py. + */ +enum pool_workqueue_stats { + PWQ_STAT_STARTED, /* work items started execution */ + PWQ_STAT_COMPLETED, /* work items completed execution */ + PWQ_STAT_CM_WAKEUP, /* concurrency-management worker wakeups */ + PWQ_STAT_MAYDAY, /* maydays to rescuer */ + PWQ_STAT_RESCUED, /* linked work items executed by rescuer */ + + PWQ_NR_STATS, +}; + /* * The per-pool workqueue. While queued, the lower WORK_STRUCT_FLAG_BITS * of work_struct->data are used for flags and the remaining high bits @@ -236,6 +250,8 @@ struct pool_workqueue { struct list_head pwqs_node; /* WR: node on wq->pwqs */ struct list_head mayday_node; /* MD: node on wq->maydays */ + u64 stats[PWQ_NR_STATS]; + /* * Release of unbound pwq is punted to system_wq. See put_pwq() * and pwq_unbound_release_workfn() for details. pool_workqueue @@ -929,8 +945,10 @@ void wq_worker_sleeping(struct task_struct *task) } pool->nr_running--; - if (need_more_worker(pool)) + if (need_more_worker(pool)) { + worker->current_pwq->stats[PWQ_STAT_CM_WAKEUP]++; wake_up_worker(pool); + } raw_spin_unlock_irq(&pool->lock); } @@ -2163,6 +2181,7 @@ static void send_mayday(struct work_struct *work) get_pwq(pwq); list_add_tail(&pwq->mayday_node, &wq->maydays); wake_up_process(wq->rescuer->task); + pwq->stats[PWQ_STAT_MAYDAY]++; } } @@ -2401,6 +2420,7 @@ __acquires(&pool->lock) * workqueues), so hiding them isn't a problem. */ lockdep_invariant_state(true); + pwq->stats[PWQ_STAT_STARTED]++; trace_workqueue_execute_start(work); worker->current_func(work); /* @@ -2408,6 +2428,7 @@ __acquires(&pool->lock) * point will only record its address. */ trace_workqueue_execute_end(work, worker->current_func); + pwq->stats[PWQ_STAT_COMPLETED]++; lock_map_release(&lockdep_map); lock_map_release(&pwq->wq->lockdep_map); @@ -2651,6 +2672,7 @@ static int rescuer_thread(void *__rescuer) if (first) pool->watchdog_ts = jiffies; move_linked_works(work, scheduled, &n); + pwq->stats[PWQ_STAT_RESCUED]++; } first = false; } diff --git a/tools/workqueue/wq_monitor.py b/tools/workqueue/wq_monitor.py new file mode 100644 index 000000000000..94435fd14c98 --- /dev/null +++ b/tools/workqueue/wq_monitor.py @@ -0,0 +1,151 @@ +#!/usr/bin/env drgn +# +# Copyright (C) 2023 Tejun Heo +# Copyright (C) 2023 Meta Platforms, Inc. and affiliates. + +desc = """ + +This is a drgn script to monitor workqueues. For more info on drgn, visit +https://github.com/osandov/drgn. + + total Total number of work items executed by the workqueue. + + infl The number of currently in-flight work items. + + CMwake The number of concurrency-management wake-ups while executing a + work item of the workqueue. + + mayday The number of times the rescuer was requested while waiting for + new worker creation. + + rescued The number of work items executed by the rescuer. +""" + +import sys +import signal +import os +import re +import time +import json + +import drgn +from drgn.helpers.linux.list import list_for_each_entry,list_empty +from drgn.helpers.linux.cpumask import for_each_possible_cpu + +import argparse +parser = argparse.ArgumentParser(description=desc, + formatter_class=argparse.RawTextHelpFormatter) +parser.add_argument('workqueue', metavar='REGEX', nargs='*', + help='Target workqueue name patterns (all if empty)') +parser.add_argument('-i', '--interval', metavar='SECS', type=float, default=1, + help='Monitoring interval (0 to print once and exit)') +parser.add_argument('-j', '--json', action='store_true', + help='Output in json') +args = parser.parse_args() + +def err(s): + print(s, file=sys.stderr, flush=True) + sys.exit(1) + +workqueues = prog['workqueues'] + +WQ_UNBOUND = prog['WQ_UNBOUND'] +WQ_MEM_RECLAIM = prog['WQ_MEM_RECLAIM'] + +PWQ_STAT_STARTED = prog['PWQ_STAT_STARTED'] # work items started execution +PWQ_STAT_COMPLETED = prog['PWQ_STAT_COMPLETED'] # work items completed execution +PWQ_STAT_CM_WAKEUP = prog['PWQ_STAT_CM_WAKEUP'] # concurrency-management worker wakeups +PWQ_STAT_MAYDAY = prog['PWQ_STAT_MAYDAY'] # maydays to rescuer +PWQ_STAT_RESCUED = prog['PWQ_STAT_RESCUED'] # linked work items executed by rescuer +PWQ_NR_STATS = prog['PWQ_NR_STATS'] + +class WqStats: + def __init__(self, wq): + self.name = wq.name.string_().decode() + self.unbound = wq.flags & WQ_UNBOUND != 0 + self.mem_reclaim = wq.flags & WQ_MEM_RECLAIM != 0 + self.stats = [0] * PWQ_NR_STATS + for pwq in list_for_each_entry('struct pool_workqueue', wq.pwqs.address_of_(), 'pwqs_node'): + for i in range(PWQ_NR_STATS): + self.stats[i] += int(pwq.stats[i]) + + def dict(self, now): + return { 'timestamp' : now, + 'name' : self.name, + 'unbound' : self.unbound, + 'mem_reclaim' : self.mem_reclaim, + 'started' : self.stats[PWQ_STAT_STARTED], + 'completed' : self.stats[PWQ_STAT_COMPLETED], + 'cm_wakeup' : self.stats[PWQ_STAT_CM_WAKEUP], + 'mayday' : self.stats[PWQ_STAT_MAYDAY], + 'rescued' : self.stats[PWQ_STAT_RESCUED], } + + def table_header_str(): + return f'{"":>24} {"total":>8} {"infl":>5} {"CMwake":>7} {"mayday":>7} {"rescued":>7}' + + def table_row_str(self): + cm_wakeup = '-' + mayday = '-' + rescued = '-' + + if not self.unbound: + cm_wakeup = str(self.stats[PWQ_STAT_CM_WAKEUP]) + + if self.mem_reclaim: + mayday = str(self.stats[PWQ_STAT_MAYDAY]) + rescued = str(self.stats[PWQ_STAT_RESCUED]) + + out = f'{self.name[-24:]:24} ' \ + f'{self.stats[PWQ_STAT_STARTED]:8} ' \ + f'{max(self.stats[PWQ_STAT_STARTED] - self.stats[PWQ_STAT_COMPLETED], 0):5} ' \ + f'{cm_wakeup:>7} ' \ + f'{mayday:>7} ' \ + f'{rescued:>7} ' + return out.rstrip(':') + +exit_req = False + +def sigint_handler(signr, frame): + global exit_req + exit_req = True + +def main(): + # handle args + table_fmt = not args.json + interval = args.interval + + re_str = None + if args.workqueue: + for r in args.workqueue: + if re_str is None: + re_str = r + else: + re_str += '|' + r + + filter_re = re.compile(re_str) if re_str else None + + # monitoring loop + signal.signal(signal.SIGINT, sigint_handler) + + while not exit_req: + now = time.time() + + if table_fmt: + print() + print(WqStats.table_header_str()) + + for wq in list_for_each_entry('struct workqueue_struct', workqueues.address_of_(), 'list'): + stats = WqStats(wq) + if filter_re and not filter_re.search(stats.name): + continue + if table_fmt: + print(stats.table_row_str()) + else: + print(stats.dict(now)) + + if interval == 0: + break + time.sleep(interval) + +if __name__ == "__main__": + main() From patchwork Thu May 11 18:19:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tejun Heo X-Patchwork-Id: 92805 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp4581814vqo; Thu, 11 May 2023 11:26:29 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ46yTH+4jtOyDd6CNFpMEQBh9M6C8uxbd8r3vnceOfkCiY5TvuNA126+6o7FJuGHcIxTkmp X-Received: by 2002:a05:6a20:7d81:b0:103:8f59:6ee1 with SMTP id v1-20020a056a207d8100b001038f596ee1mr7155096pzj.29.1683829589414; Thu, 11 May 2023 11:26:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683829589; cv=none; d=google.com; s=arc-20160816; b=dnWWXw9lhwwNl3LwvFAeI4FrBl+kQLi000u1HGvz3m4OkvXQABDgk2TcrmbPJ8CcPv oWkBdTugzU2vAaFV406GM7dO7eo1Xs3oht1b8hrVJe0g/7E1GqfgSZyS4kDNoVH0UkEd jVjbY9nSewCo/8Rdg31vPL7ieQMHyWh+/Sq4tNlYHxELKXkvPua+elvmLqaTPD0BOXOz UKIf/5kiETPK/F3XVmGayymZdTj2skYF7L8OItF/BTzgl4/CVWhkwejQ6al/ksK4Ixok i3Uijovg7VBfdB+/0nmgF6B7AG176lRTuTNg7y437DbeQ32zzW02G+Pdymx55w+zVPNY tDLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from:sender :dkim-signature; bh=QDANAV6Ubr5KkEqy3DIRzGm4anT8hIgcnesbBUBsaAg=; b=IyS46XJlcrMM/kGIa04K6lJCHQ+7gG5a8i+IsJGyZSGE44YWglOZIMnb7tJ9bMsfX9 +pV1OD2YGE1yvaRfhYCTF0ldEON8z8jhEAA/bb4zNk1XA6t2VdgDcsVw/f94L6fgB6ep +gn0bI8f+85qckKFI0Q/r4samgzSufEKn1KANcHhK84L+zUIMFbShnCzurm2a96bkxYU MZ92n0TYHcdPNctEuh9N9XqFT/7mpXaFNbNLEDloUmoInLO2/PBJ/ew5sJ2i1yGjmc0D LPse73mDvPkkowWyWBrkpedY/vTdj4ctuNNYplCGw8F38tsHeLNY9UIXq3i020iOsmQB H4EQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=WtIDqfn+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u187-20020a6379c4000000b005034a5a0a58si7011470pgc.434.2023.05.11.11.26.16; Thu, 11 May 2023 11:26:29 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=WtIDqfn+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239214AbjEKSTt (ORCPT + 99 others); Thu, 11 May 2023 14:19:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43820 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239195AbjEKSTn (ORCPT ); Thu, 11 May 2023 14:19:43 -0400 Received: from mail-pl1-x630.google.com (mail-pl1-x630.google.com [IPv6:2607:f8b0:4864:20::630]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 436E465A4 for ; Thu, 11 May 2023 11:19:42 -0700 (PDT) Received: by mail-pl1-x630.google.com with SMTP id d9443c01a7336-1aad6f2be8eso83130555ad.3 for ; Thu, 11 May 2023 11:19:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1683829181; x=1686421181; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:from:to:cc:subject:date :message-id:reply-to; bh=QDANAV6Ubr5KkEqy3DIRzGm4anT8hIgcnesbBUBsaAg=; b=WtIDqfn+V8MY0/g3+qsxH3AhZvgNbaX5FZiNHaef8+SdE8ZMxAVNTffmWdOgQ9bDM8 BdepHSrLnfySVcOp7N2MU5mP/S/iFFzodZtgU4UTTB9g31/2D4mp4Gs/+ea4rbqUIKAI 2+kWfcuXv9jFEAEtrYmK4xtA5qGeLUPfv3Qi7jib6l/4OTiYmWi6sDjUiVA7amKhxRIT /H0n8vSxg5J7f/K7CbVpWMVp27+zrWCrChAkK+b3IDWoOz06qOQrmrRCb2Qjq8YuaoYd lK+P4o+ymI5G+NFe+Qdw+yOZE44DqpnLkJc/sf58WWgjIwtzHkQMK/7T5Q2hJ2tetPn+ P3wg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683829181; x=1686421181; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=QDANAV6Ubr5KkEqy3DIRzGm4anT8hIgcnesbBUBsaAg=; b=HA0WJFKqlcHduSq4em1dm3wFCTcVXnUo4zCpW8ir61sB13pZOghKk5m09dYpjEI55Z REyff7YhgFl2kiEubXnuNV+iMYiT0IHxoJJk8Cx5TTaXebU4g7Pj1UJdIDs+2f+/V1Ux eVbVmXvPc94/+y0Z1aJ99dzllx8AAYDAQ+WOcOwqPkWgdebNhrhaaWpd/GLtPB1d2k/a nL3lVRGtxRwlxmFi2SXnUtQAhK9mr93mTR3IU9qn3zEy9VnMgw88J1VVbvif0IeEZFAi jo32BrJGYV9dGDirq5s3kZ1fmI3CD057vJKbJ3FSKsjbjdpUWvev9QUVmvgSbpZQgvvA YbXQ== X-Gm-Message-State: AC+VfDy7/y5fFeqk0kcHzTWMLv7aBBw2CfY1iVbvevYtG+bCoCnUDuGv oCeNcdwtjWt8U+XS9DPulpg= X-Received: by 2002:a17:902:ba84:b0:1ab:1c09:2df8 with SMTP id k4-20020a170902ba8400b001ab1c092df8mr22587871pls.50.1683829181121; Thu, 11 May 2023 11:19:41 -0700 (PDT) Received: from localhost (2603-800c-1a02-1bae-a7fa-157f-969a-4cde.res6.spectrum.com. [2603:800c:1a02:1bae:a7fa:157f:969a:4cde]) by smtp.gmail.com with ESMTPSA id s16-20020a170902989000b001acae093442sm4500255plp.82.2023.05.11.11.19.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 May 2023 11:19:40 -0700 (PDT) Sender: Tejun Heo From: Tejun Heo To: jiangshanlai@gmail.com Cc: torvalds@linux-foundation.org, peterz@infradead.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, Tejun Heo Subject: [PATCH 2/7] workqueue: Re-order struct worker fields Date: Thu, 11 May 2023 08:19:26 -1000 Message-Id: <20230511181931.869812-3-tj@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230511181931.869812-1-tj@kernel.org> References: <20230511181931.869812-1-tj@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765623295134789204?= X-GMAIL-MSGID: =?utf-8?q?1765623295134789204?= struct worker was laid out with the intent that all fields that are modified for each work item execution are in the first cacheline. However, this hasn't been true for a while with the addition of ->last_func. Let's just collect hot fields together at the top. Move ->sleeping in the hole after ->current_color and move ->lst_func right below. While at it, drop the cacheline comment which isn't useful anymore. Signed-off-by: Tejun Heo Cc: Lai Jiangshan --- kernel/workqueue_internal.h | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/kernel/workqueue_internal.h b/kernel/workqueue_internal.h index e00b1204a8e9..0600f04ceeb2 100644 --- a/kernel/workqueue_internal.h +++ b/kernel/workqueue_internal.h @@ -32,9 +32,12 @@ struct worker { work_func_t current_func; /* L: current_work's fn */ struct pool_workqueue *current_pwq; /* L: current_work's pwq */ unsigned int current_color; /* L: current_work's color */ - struct list_head scheduled; /* L: scheduled works */ + int sleeping; /* None */ + + /* used by the scheduler to determine a worker's last known identity */ + work_func_t last_func; /* L: last work's fn */ - /* 64 bytes boundary on 64bit, 32 on 32bit */ + struct list_head scheduled; /* L: scheduled works */ struct task_struct *task; /* I: worker task */ struct worker_pool *pool; /* A: the associated pool */ @@ -45,7 +48,6 @@ struct worker { unsigned long last_active; /* L: last active timestamp */ unsigned int flags; /* X: flags */ int id; /* I: worker id */ - int sleeping; /* None */ /* * Opaque string set with work_set_desc(). Printed out with task @@ -55,9 +57,6 @@ struct worker { /* used only by rescuers to point to the target workqueue */ struct workqueue_struct *rescue_wq; /* I: the workqueue to rescue */ - - /* used by the scheduler to determine a worker's last known identity */ - work_func_t last_func; }; /** From patchwork Thu May 11 18:19:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tejun Heo X-Patchwork-Id: 92814 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp4588820vqo; Thu, 11 May 2023 11:39:30 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7QBKVstEQeoNhm7GTOxpDREUPbkdP3CJoRhvb0s1y2gIDiGtoNbLaikWZ1u26ltqerps9h X-Received: by 2002:aa7:88ce:0:b0:645:6142:7f5a with SMTP id k14-20020aa788ce000000b0064561427f5amr20703444pff.3.1683830369709; Thu, 11 May 2023 11:39:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683830369; cv=none; d=google.com; s=arc-20160816; b=L5sAUYfEIPWMWNhMOlW1ZBIIPG6QbM0tTwqkSk4foXr52B694sVicelph5i40OKE+F lII3yEHLMnsAXSkTzcolZPuznbHNoFKZhnWstEXNB8FlcN0YOwNKBymbn+vKyuce3Gds 6EQNzmSLRNGUxbKrUjRKHrzOiOalJ9H2Sw4I2IXwcba3jPM1K+O/bC9EGV/Qvd3fO5HY RrTIm5tKo6LlvLGSUb8V14q1IOcefCJ/PIVVPz8igGu9m+tNzyE4b9myD1oXo+Nigj1K VrbUngGVyTY3k+ZNRnOZfnlEXLwGvljTHxG3su9281vMtIM8KSfiiZMAlq7tP8Ibrbe2 sRXg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from:sender :dkim-signature; bh=ZTWAQkbcaPX8SKv+21F9cvlNho68RajHcKaNsycJvbc=; b=OEs/p7I3Ya2AfLqMplku+PLScxHqMp+2ls41dbmvAsUngisUTrtLnJaIFCvOq8Vq/3 FPc3H29exVRGZNZb8ANH6rBbuboAERnUN2emBSCmF2ysvODhwem+BQvSKCLwR4K2Hk2B 1B0ka+XsAoQE2VMr0Ka9GVPgv/ej2VHxEvdDOIQhuU1pVrF1t2/Nh/mcCL9sm/aha93j NqQw5oaP9iSmTJWXD8H4Uwv0XxBbHQ7C3q320nHcmpXoP6H/tymwt9QSw7mvVAXX+J5k tb4tfMBAOtXacmZ6XJjFhbze2cuc8wU82ptfKQ7HL+dIyQTJf7uREO+7MJGWqPPuUoPr VGQA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=kU752pRj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id j13-20020a63cf0d000000b005139e5df68asi7182831pgg.433.2023.05.11.11.39.16; Thu, 11 May 2023 11:39:29 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=kU752pRj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239217AbjEKSUE (ORCPT + 99 others); Thu, 11 May 2023 14:20:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43902 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238819AbjEKSTq (ORCPT ); Thu, 11 May 2023 14:19:46 -0400 Received: from mail-pf1-x433.google.com (mail-pf1-x433.google.com [IPv6:2607:f8b0:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 856487696 for ; Thu, 11 May 2023 11:19:44 -0700 (PDT) Received: by mail-pf1-x433.google.com with SMTP id d2e1a72fcca58-6439bbc93b6so6175716b3a.1 for ; Thu, 11 May 2023 11:19:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1683829184; x=1686421184; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:from:to:cc:subject:date :message-id:reply-to; bh=ZTWAQkbcaPX8SKv+21F9cvlNho68RajHcKaNsycJvbc=; b=kU752pRjjFvhx4dpLtFlAxLg9JwwMVdo0h6v4yD9qyVVja8cLqBoNygPdrpEo+3P4R jR5ON10fwKyVUwRgnLPJG2HXusrClIELoHAUOs3xgtc6+Wf/xGtkFQUHfGotVgaOpzHv pUbZq4tTNYhvXnsejm0De0xAUwSJPBqnhqX0Oy6oOrzY5czkE7uY4w1NHoiOi0McTLT5 H0+w6mgVSO7wj9MszL8gdKAdU5L426BaK4ZvhFL1m2sSrVe5YObe3Ij36w3hDQeKQx0b bY2I4BMzvE5/lAPEMyfYRHeIn3/aJ7EbqTzc5dRC/WmL7Csfw3N6dvTqzvhWdgwTuljq 4Tpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683829184; x=1686421184; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=ZTWAQkbcaPX8SKv+21F9cvlNho68RajHcKaNsycJvbc=; b=bhn8W5CT8LXODcIw1b8xKGY9NXSV4brD/5eVQ8wM98BBUofw6cxbdxZ0jENTwUug7A kvUMmNFnTr48DTNVrMxM7b7binG0KVKNAjJIXFst9KdF+FWitMXZyL1gbre/7nUsVrsz TPqrJc5aIj5VoRIOebL3SkrziabitFVEVhsWSX9yqSFC64ZFujRGVTAksrELcAp+Juab Fyycijb+7/UiDtKFnInAfcYraUuXLq8fmBSM6M5s3fZdgjSeHSWUCTyVQG+D7LixmeRP zGKvQxc5DHbAfYjk83wR8XAkQXnx/yD1RBbR6sCO5W0zXNLA9uISteOj0LZchxxSpWNF eROA== X-Gm-Message-State: AC+VfDyUdaFzi4eB5ceF07S306TeQaxaXY04cYXxi1n3NbQ1R7x9n/ni 5GFSgBlnGWvjChWCTO+qjCF4HUERHrY= X-Received: by 2002:a05:6a00:2ea0:b0:648:ebb2:3d6 with SMTP id fd32-20020a056a002ea000b00648ebb203d6mr4994823pfb.26.1683829183721; Thu, 11 May 2023 11:19:43 -0700 (PDT) Received: from localhost (2603-800c-1a02-1bae-a7fa-157f-969a-4cde.res6.spectrum.com. [2603:800c:1a02:1bae:a7fa:157f:969a:4cde]) by smtp.gmail.com with ESMTPSA id k4-20020aa790c4000000b0062bc045bf4fsm5822850pfk.19.2023.05.11.11.19.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 May 2023 11:19:43 -0700 (PDT) Sender: Tejun Heo From: Tejun Heo To: jiangshanlai@gmail.com Cc: torvalds@linux-foundation.org, peterz@infradead.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, Tejun Heo Subject: [PATCH 3/7] workqueue: Move worker_set/clr_flags() upwards Date: Thu, 11 May 2023 08:19:27 -1000 Message-Id: <20230511181931.869812-4-tj@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230511181931.869812-1-tj@kernel.org> References: <20230511181931.869812-1-tj@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765624113376689583?= X-GMAIL-MSGID: =?utf-8?q?1765624113376689583?= They are going to be used in wq_worker_stopping(). Move them upwards. Signed-off-by: Tejun Heo Cc: Lai Jiangshan --- kernel/workqueue.c | 108 ++++++++++++++++++++++----------------------- 1 file changed, 54 insertions(+), 54 deletions(-) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index 33ddfaacdf29..452bcb49192a 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -879,6 +879,60 @@ static void wake_up_worker(struct worker_pool *pool) wake_up_process(worker->task); } +/** + * worker_set_flags - set worker flags and adjust nr_running accordingly + * @worker: self + * @flags: flags to set + * + * Set @flags in @worker->flags and adjust nr_running accordingly. + * + * CONTEXT: + * raw_spin_lock_irq(pool->lock) + */ +static inline void worker_set_flags(struct worker *worker, unsigned int flags) +{ + struct worker_pool *pool = worker->pool; + + WARN_ON_ONCE(worker->task != current); + + /* If transitioning into NOT_RUNNING, adjust nr_running. */ + if ((flags & WORKER_NOT_RUNNING) && + !(worker->flags & WORKER_NOT_RUNNING)) { + pool->nr_running--; + } + + worker->flags |= flags; +} + +/** + * worker_clr_flags - clear worker flags and adjust nr_running accordingly + * @worker: self + * @flags: flags to clear + * + * Clear @flags in @worker->flags and adjust nr_running accordingly. + * + * CONTEXT: + * raw_spin_lock_irq(pool->lock) + */ +static inline void worker_clr_flags(struct worker *worker, unsigned int flags) +{ + struct worker_pool *pool = worker->pool; + unsigned int oflags = worker->flags; + + WARN_ON_ONCE(worker->task != current); + + worker->flags &= ~flags; + + /* + * If transitioning out of NOT_RUNNING, increment nr_running. Note + * that the nested NOT_RUNNING is not a noop. NOT_RUNNING is mask + * of multiple flags, not a single flag. + */ + if ((flags & WORKER_NOT_RUNNING) && (oflags & WORKER_NOT_RUNNING)) + if (!(worker->flags & WORKER_NOT_RUNNING)) + pool->nr_running++; +} + /** * wq_worker_running - a worker is running again * @task: task waking up @@ -983,60 +1037,6 @@ work_func_t wq_worker_last_func(struct task_struct *task) return worker->last_func; } -/** - * worker_set_flags - set worker flags and adjust nr_running accordingly - * @worker: self - * @flags: flags to set - * - * Set @flags in @worker->flags and adjust nr_running accordingly. - * - * CONTEXT: - * raw_spin_lock_irq(pool->lock) - */ -static inline void worker_set_flags(struct worker *worker, unsigned int flags) -{ - struct worker_pool *pool = worker->pool; - - WARN_ON_ONCE(worker->task != current); - - /* If transitioning into NOT_RUNNING, adjust nr_running. */ - if ((flags & WORKER_NOT_RUNNING) && - !(worker->flags & WORKER_NOT_RUNNING)) { - pool->nr_running--; - } - - worker->flags |= flags; -} - -/** - * worker_clr_flags - clear worker flags and adjust nr_running accordingly - * @worker: self - * @flags: flags to clear - * - * Clear @flags in @worker->flags and adjust nr_running accordingly. - * - * CONTEXT: - * raw_spin_lock_irq(pool->lock) - */ -static inline void worker_clr_flags(struct worker *worker, unsigned int flags) -{ - struct worker_pool *pool = worker->pool; - unsigned int oflags = worker->flags; - - WARN_ON_ONCE(worker->task != current); - - worker->flags &= ~flags; - - /* - * If transitioning out of NOT_RUNNING, increment nr_running. Note - * that the nested NOT_RUNNING is not a noop. NOT_RUNNING is mask - * of multiple flags, not a single flag. - */ - if ((flags & WORKER_NOT_RUNNING) && (oflags & WORKER_NOT_RUNNING)) - if (!(worker->flags & WORKER_NOT_RUNNING)) - pool->nr_running++; -} - /** * find_worker_executing_work - find worker which is executing a work * @pool: pool of interest From patchwork Thu May 11 18:19:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tejun Heo X-Patchwork-Id: 92815 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp4589106vqo; Thu, 11 May 2023 11:40:06 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5uo2crDO2L3KfxgKoh1HjUSp0nbLjo2OKiLmLUyXRthm/o7QKOu7AURUuCpMsIi5YN2N8W X-Received: by 2002:a17:90a:2dcf:b0:250:ad4a:292a with SMTP id q15-20020a17090a2dcf00b00250ad4a292amr10682868pjm.28.1683830406207; Thu, 11 May 2023 11:40:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683830406; cv=none; d=google.com; s=arc-20160816; b=07f197Ptz30YN9vl+tpax45pHGJz3MlkIlhwVozN/TTYqA0iK7vEdamhrYdFafaxqO DRpvm0apISiQMZTvXF3vHudyyGC646yFgVdcQwLb4XIUxQijnYvsVF53khtnJKFVepgA vVQGvlTztsvfC8yG3CTlwFi5Pq12DlBAWLzahITQrzQEzS867PRZwueeIqScb5/Z4eFc d0ejrz2EwsaICR4W2tmlN8BFB76BYnU0P6nUloLrRtiSdRhq0mFsPfA91B81I2Sa+I9a lfW/rCe1F62iyBO5hJIS42guOINOL0G4Z9LLY8/7j7Q/KIBnU9PsDyaj3W/RSmehbHlA Y4mw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from:sender :dkim-signature; bh=QEMIUUrhmTXrgN4icuTRdubFtkqVCDLnAwLOvCI/lRk=; b=TuTfZfPOZCS7IJd268glHsTn4N4n57iZb6evoo/58PQOBjaobu8LSpBSIULo/E9TzN tOI4BK1bXMHwkBkMOjbVTAkVDB8n5JBqfQIxhqvGiw6plrSufTIFoH3E9ldlCrcqBJbr Q/ukrULOxkGR53+jXJi71twwf13qaIYjjGTNgdS0h35szG91QFfp5opHgu2Bn7XwpOgV Bu/7zJjd/SAvSNTjQ4mQhf0wXQ/b4K9+ZKKUr7k7n08XN2ToHGA0V//7qg6wDm3y+jIX sCWNINRwjPRP6INA140WnyoR7fTLTmwhKVA+6Aq0c6/gtKNqhjTgFYbYhtrjixcx++B0 vwWA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=BLt5ctJ1; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h2-20020a17090a2ec200b0024981e98175si31694583pjs.75.2023.05.11.11.39.53; Thu, 11 May 2023 11:40:06 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=BLt5ctJ1; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239236AbjEKSUH (ORCPT + 99 others); Thu, 11 May 2023 14:20:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44002 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239210AbjEKSTs (ORCPT ); Thu, 11 May 2023 14:19:48 -0400 Received: from mail-pg1-x532.google.com (mail-pg1-x532.google.com [IPv6:2607:f8b0:4864:20::532]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3A0AF9EC8 for ; Thu, 11 May 2023 11:19:47 -0700 (PDT) Received: by mail-pg1-x532.google.com with SMTP id 41be03b00d2f7-52cb78647ecso5164941a12.1 for ; Thu, 11 May 2023 11:19:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1683829186; x=1686421186; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:from:to:cc:subject:date :message-id:reply-to; bh=QEMIUUrhmTXrgN4icuTRdubFtkqVCDLnAwLOvCI/lRk=; b=BLt5ctJ1/mYsPAopbjovnpqCSOrx7+cc+Lbw+7VmmGUEo+5zlzaj9lY/7hR76f3Kn9 K4BENr45d+xLKIz1ccc0Hdci3olB+MqAJXPJ4Kw9/cqic567IFu5mcyLFDrGCGZE5PET 3E7OQZ6/M+DRmSFcoKRMLydyC2bWh5aYIlGemZYC9daNdDoMtfQ3EoiwJUdic3WCBHbo J7WiKXBSeAVjpASjFN4jD/MGOpENryd6YSK+kn+J4bMlw9VI/jF0hll4yIgQ4aMqCu3L JO6Rix14pdCLccIDNrSsjFEJqGBS/+nAOruuP2ozwGhjQpydDHzvwadN1J9MM5jF2Lbf OjGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683829186; x=1686421186; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=QEMIUUrhmTXrgN4icuTRdubFtkqVCDLnAwLOvCI/lRk=; b=he36WBHWrywjci875O7DE/L8PVqc/gBqmCyuV7GxLqGqaIihRtAZvKbsmlLZfSWlWn HKKR0bO9t8/yhvJVIvs467LWoJCXNwzcR4hkA8rIoGMaqi8CqHwoWGQQdtVZ0VDtbxhQ ++WRtWkFVeoyeM6T0WN6mkghpYcDgpIJaCh/e14GTaEGfA2Yh/a57XvqYxnHx9FyzJ2Z kT0gkzzW7Mo/jbgXGjKCLvmAGGaTEUwhXQfPJ+KZ8K8YInKLgbS3DLxy0nudtBg6lFpJ vrrNAEIhvP8WP5FB03l7p7SDZS1YteAVmyTFbbOujKJCziOgPq2WuOW15SebSnZnM/xN mpqw== X-Gm-Message-State: AC+VfDwgIUuZD82ci0TZMgYDZYC+yAPIPsmtE59ryldXQOL2VQK9uTGz elXzw58M+pImQdT/ODkqSSs= X-Received: by 2002:a17:902:d2c4:b0:1ac:71a7:a1fb with SMTP id n4-20020a170902d2c400b001ac71a7a1fbmr21200516plc.18.1683829186432; Thu, 11 May 2023 11:19:46 -0700 (PDT) Received: from localhost (2603-800c-1a02-1bae-a7fa-157f-969a-4cde.res6.spectrum.com. [2603:800c:1a02:1bae:a7fa:157f:969a:4cde]) by smtp.gmail.com with ESMTPSA id p4-20020a170902e74400b001aafe56ea70sm6275199plf.5.2023.05.11.11.19.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 May 2023 11:19:45 -0700 (PDT) Sender: Tejun Heo From: Tejun Heo To: jiangshanlai@gmail.com Cc: torvalds@linux-foundation.org, peterz@infradead.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, Tejun Heo Subject: [PATCH 4/7] workqueue: Improve locking rule description for worker fields Date: Thu, 11 May 2023 08:19:28 -1000 Message-Id: <20230511181931.869812-5-tj@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230511181931.869812-1-tj@kernel.org> References: <20230511181931.869812-1-tj@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765624152441474080?= X-GMAIL-MSGID: =?utf-8?q?1765624152441474080?= * Some worker fields are modified only by the worker itself while holding pool->lock thus making them safe to read from self, IRQ context if the CPU is running the worker or while holding pool->lock. Add 'K' locking rule for them. * worker->sleeping is currently marked "None" which isn't very descriptive. It's used only by the worker itself. Add 'S' locking rule for it. A future patch will depend on the 'K' rule to access worker->current_* from the scheduler ticks. Signed-off-by: Tejun Heo --- kernel/workqueue.c | 6 ++++++ kernel/workqueue_internal.h | 15 ++++++++------- 2 files changed, 14 insertions(+), 7 deletions(-) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index 452bcb49192a..d6bfac871965 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -126,6 +126,12 @@ enum { * cpu or grabbing pool->lock is enough for read access. If * POOL_DISASSOCIATED is set, it's identical to L. * + * K: Only modified by worker while holding pool->lock. Can be safely read by + * self, while holding pool->lock or from IRQ context if %current is the + * kworker. + * + * S: Only modified by worker self. + * * A: wq_pool_attach_mutex protected. * * PL: wq_pool_mutex protected. diff --git a/kernel/workqueue_internal.h b/kernel/workqueue_internal.h index 0600f04ceeb2..c2455be7b4c2 100644 --- a/kernel/workqueue_internal.h +++ b/kernel/workqueue_internal.h @@ -28,14 +28,15 @@ struct worker { struct hlist_node hentry; /* L: while busy */ }; - struct work_struct *current_work; /* L: work being processed */ - work_func_t current_func; /* L: current_work's fn */ - struct pool_workqueue *current_pwq; /* L: current_work's pwq */ - unsigned int current_color; /* L: current_work's color */ - int sleeping; /* None */ + struct work_struct *current_work; /* K: work being processed and its */ + work_func_t current_func; /* K: function */ + struct pool_workqueue *current_pwq; /* K: pwq */ + unsigned int current_color; /* K: color */ + + int sleeping; /* S: is worker sleeping? */ /* used by the scheduler to determine a worker's last known identity */ - work_func_t last_func; /* L: last work's fn */ + work_func_t last_func; /* K: last work's fn */ struct list_head scheduled; /* L: scheduled works */ @@ -45,7 +46,7 @@ struct worker { struct list_head node; /* A: anchored at pool->workers */ /* A: runs through worker->node */ - unsigned long last_active; /* L: last active timestamp */ + unsigned long last_active; /* K: last active timestamp */ unsigned int flags; /* X: flags */ int id; /* I: worker id */ From patchwork Thu May 11 18:19:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tejun Heo X-Patchwork-Id: 92807 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp4582772vqo; Thu, 11 May 2023 11:28:28 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5gQRiGn0vj7kqF8U6wCbaKGdYyCa3cni3I6OQFzagncTSdc7jc6CFLaWmy34PR5Ozm4T/l X-Received: by 2002:a05:6a20:5481:b0:101:7ccd:e195 with SMTP id i1-20020a056a20548100b001017ccde195mr13371889pzk.8.1683829708311; Thu, 11 May 2023 11:28:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683829708; cv=none; d=google.com; s=arc-20160816; b=mI0MmLKtN5vmFxu/eoVNXtntW5flYmRC6Sc3uk8AkWBBYcWGzALnJKNV5UN9PoI0lW ZV19w84rBGCo6bKSqDAo4/iKfjc1Cbb6HeVCAbtqPrlIiSDxkKKoaIscCUJusKeciCrw kXpZEc7XgPj89cOk32QElgvIHIk/9UrUkhTnRTjewvfzvqPrpa4/QDJh02/JtSvZd/Cn 1ff4evnQOYDxB1xx7aS0fwHlPf/FoTnDhKZ38S8SxSDiEkZiIpRtT/AS96LUQwtqD5NB 2+NNuddNn/MSZZ+8iBCfGG6JHHdQtomUJKJUzsXQYBc+0B72Vtw0b6AFe9vY5KAqNpti PyBg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from:sender :dkim-signature; bh=quiqybTUMzvK3J6v+qwshTsVN5/Bo32L2Xe+MCI29HI=; b=mfsoznHdDGyi3rqlWNi3fv+Lc+MupJbxjuMEZ9B9zSy1JfX7gJPni2A9wjFacssUOR ybSyVBUHSnUxmaxp9OjeGgw4gXtaS58BVurSvO0+AUnoqGTDzmKWnexsGcQXfJSJNdN2 MYJ+34+G9MGYr3xmNXVKJ3bpzX9YXeSBIqnIuxfr2l10eL70oKKmt28E+UJdn6HQMMaJ +qhiJrDfthRchCBgQNioZEzmr1ZW5S8GG3joKVVplcMtPoJ90vZmrYHf/SYn8k1PqsSV PNohA0d9v9tmkhSNzmXcAUKOtHif1cTOQlR7rPghNXhwRIODHRgskE+ObA7dQsyiswgf Cifg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=VTQg80mx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b191-20020a6334c8000000b005303b08eb16si7477593pga.631.2023.05.11.11.28.12; Thu, 11 May 2023 11:28:28 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=VTQg80mx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239253AbjEKSUK (ORCPT + 99 others); Thu, 11 May 2023 14:20:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44032 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239254AbjEKSTz (ORCPT ); Thu, 11 May 2023 14:19:55 -0400 Received: from mail-pj1-x1036.google.com (mail-pj1-x1036.google.com [IPv6:2607:f8b0:4864:20::1036]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 05D7F6EB5 for ; Thu, 11 May 2023 11:19:50 -0700 (PDT) Received: by mail-pj1-x1036.google.com with SMTP id 98e67ed59e1d1-24dea6d5ce8so8404809a91.2 for ; Thu, 11 May 2023 11:19:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1683829189; x=1686421189; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:from:to:cc:subject:date :message-id:reply-to; bh=quiqybTUMzvK3J6v+qwshTsVN5/Bo32L2Xe+MCI29HI=; b=VTQg80mxnr1Y21EauJUvxc7Xp6xRFN0R+MdcnB0DZxOhOqx+miMx1Ns4vPybGwzJyH 7HR0JeLtzdAbQupIc9545ySKMHMRvb1tY/Z4wSGbMj3aRgEeF76lc6lz70q2MMqWQADM a+r/7Lzkxutoy5A7+s0yvi1pLvg3KAv1tViO3EN31vSx/4jZ0q/316wNWbwbMhdRIvEc MRisjYvx6yjB7SnjGdmmyxg1yczASChWx664qLF1Fb81ITrHJ2zN9WSMOtyqv9+MR599 SOVzunAXsKWCZytTYPY0RhZ7nMKmAOcOczQAcRJ1W8a7HTPI2NTkSRqOFw9AWc6ExRwW vZWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683829189; x=1686421189; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=quiqybTUMzvK3J6v+qwshTsVN5/Bo32L2Xe+MCI29HI=; b=c1v7S3V82q5+CE0D0/ZFi4t+VJOpaDmPy6N17YE6QQM5z4oAMqms4RsMLCJIPk0z7P 0UdT4sklifigpcIi2Ll0MEj2YuqxcTzSoSz9xbYjnZ6Pmi1NzFQ8EX+bBWr2WoAfVQeA C0mhE2U0Ie+HKgsRANQsIRb6NDUf4XU1U9MaqVdfUQ+IV+hhI7K7vLjM1YDgqjLQ9l+U ADkYpMckxVY4bWknvzQaf44Ne/fg99RSuiOg4sTIPFN/s6h45wGlSqr1zwIZkpPXvebf Nde56tlu5cUCgaVBQGuf0grpDTMHnvLD3tHroH2DOQ6NcN+XVGZvzpQQSFJi6vM4vBb6 IGBw== X-Gm-Message-State: AC+VfDz8rlgBoV+VuvUDJYARwKt0Ra96iyxhH60WPCbNfulNYbpWMCjy GM2ctiYDQTR14onetKNs4MQ= X-Received: by 2002:a17:90b:1e11:b0:24b:2f97:9208 with SMTP id pg17-20020a17090b1e1100b0024b2f979208mr22132635pjb.0.1683829189078; Thu, 11 May 2023 11:19:49 -0700 (PDT) Received: from localhost (2603-800c-1a02-1bae-a7fa-157f-969a-4cde.res6.spectrum.com. [2603:800c:1a02:1bae:a7fa:157f:969a:4cde]) by smtp.gmail.com with ESMTPSA id z11-20020a17090a8b8b00b0024dfb08da87sm15502960pjn.33.2023.05.11.11.19.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 May 2023 11:19:48 -0700 (PDT) Sender: Tejun Heo From: Tejun Heo To: jiangshanlai@gmail.com Cc: torvalds@linux-foundation.org, peterz@infradead.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, Tejun Heo Subject: [PATCH 5/7] workqueue: Automatically mark CPU-hogging work items CPU_INTENSIVE Date: Thu, 11 May 2023 08:19:29 -1000 Message-Id: <20230511181931.869812-6-tj@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230511181931.869812-1-tj@kernel.org> References: <20230511181931.869812-1-tj@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765623420358684145?= X-GMAIL-MSGID: =?utf-8?q?1765623420358684145?= If a per-cpu work item hogs the CPU, it can prevent other work items from starting through concurrency management. A per-cpu workqueue which intends to host such CPU-hogging work items can choose to not participate in concurrency management by setting %WQ_CPU_INTENSIVE; however, this can be error-prone and difficult to debug when missed. This patch adds an automatic CPU usage based detection. If a concurrency-managed work item consumes more CPU time than the threshold (10ms by default) continuously without intervening sleeps, wq_worker_tick() which is called from scheduler_tick() will detect the condition and automatically mark it CPU_INTENSIVE. The mechanism isn't foolproof: * Detection depends on tick hitting the work item. Getting preempted at the right timings may allow a violating work item to evade detection at least temporarily. * nohz_full CPUs may not be running ticks and thus can fail detection. * Even when detection is working, the 10ms detection delays can add up if many CPU-hogging work items are queued at the same time. However, in vast majority of cases, this should be able to detect violations reliably and provide reasonable protection with a small increase in code complexity. If some work items trigger this condition repeatedly, the bigger problem likely is the CPU being saturated with such per-cpu work items and the solution would be making them UNBOUND. The next patch will add a debug mechanism to help spot such cases. v3: Switch to use wq_worker_tick() instead of hooking into preemptions as suggested by Peter. v2: Lai pointed out that wq_worker_stopping() also needs to be called from preemption and rtlock paths and an earlier patch was updated accordingly. This patch adds a comment describing the risk of infinte recursions and how they're avoided. Signed-off-by: Tejun Heo Cc: Peter Zijlstra Cc: Linus Torvalds Cc: Lai Jiangshan Acked-by: Peter Zijlstra (Intel) --- Documentation/core-api/workqueue.rst | 38 ++++++++-------- kernel/sched/core.c | 3 ++ kernel/workqueue.c | 68 ++++++++++++++++++++++++++-- kernel/workqueue_internal.h | 2 + tools/workqueue/wq_monitor.py | 13 +++++- 5 files changed, 99 insertions(+), 25 deletions(-) diff --git a/Documentation/core-api/workqueue.rst b/Documentation/core-api/workqueue.rst index 7e5c39310bbf..a389f31b025c 100644 --- a/Documentation/core-api/workqueue.rst +++ b/Documentation/core-api/workqueue.rst @@ -354,25 +354,25 @@ Monitoring Use tools/workqueue/wq_monitor.py to monitor workqueue operations: :: $ tools/workqueue/wq_monitor.py events - total infl CMwake mayday rescued - events 18545 0 5 - - - events_highpri 8 0 0 - - - events_long 3 0 0 - - - events_unbound 38306 0 - - - - events_freezable 0 0 0 - - - events_power_efficient 29598 0 0 - - - events_freezable_power_ 10 0 0 - - - sock_diag_events 0 0 0 - - - - total infl CMwake mayday rescued - events 18548 0 5 - - - events_highpri 8 0 0 - - - events_long 3 0 0 - - - events_unbound 38322 0 - - - - events_freezable 0 0 0 - - - events_power_efficient 29603 0 0 - - - events_freezable_power_ 10 0 0 - - - sock_diag_events 0 0 0 - - + total infl CPUitsv CMwake mayday rescued + events 18545 0 0 5 - - + events_highpri 8 0 0 0 - - + events_long 3 0 0 0 - - + events_unbound 38306 0 - - - - + events_freezable 0 0 0 0 - - + events_power_efficient 29598 0 0 0 - - + events_freezable_power_ 10 0 0 0 - - + sock_diag_events 0 0 0 0 - - + + total infl CPUitsv CMwake mayday rescued + events 18548 0 0 5 - - + events_highpri 8 0 0 0 - - + events_long 3 0 0 0 - - + events_unbound 38322 0 - - - - + events_freezable 0 0 0 0 - - + events_power_efficient 29603 0 0 0 - - + events_freezable_power_ 10 0 0 0 - - + sock_diag_events 0 0 0 0 - - ... diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 944c3ae39861..3484cada9a4a 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -5632,6 +5632,9 @@ void scheduler_tick(void) perf_event_task_tick(); + if (curr->flags & PF_WQ_WORKER) + wq_worker_tick(curr); + #ifdef CONFIG_SMP rq->idle_balance = idle_cpu(cpu); trigger_load_balance(rq); diff --git a/kernel/workqueue.c b/kernel/workqueue.c index d6bfac871965..661acc5afcfd 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -212,6 +212,7 @@ struct worker_pool { enum pool_workqueue_stats { PWQ_STAT_STARTED, /* work items started execution */ PWQ_STAT_COMPLETED, /* work items completed execution */ + PWQ_STAT_CPU_INTENSIVE, /* wq_cpu_intensive_thresh_us violations */ PWQ_STAT_CM_WAKEUP, /* concurrency-management worker wakeups */ PWQ_STAT_MAYDAY, /* maydays to rescuer */ PWQ_STAT_RESCUED, /* linked work items executed by rescuer */ @@ -332,6 +333,14 @@ static struct kmem_cache *pwq_cache; static cpumask_var_t *wq_numa_possible_cpumask; /* possible CPUs of each node */ +/* + * Per-cpu work items which run for longer than the following threshold are + * automatically considered CPU intensive and excluded from concurrency + * management to prevent them from noticeably delaying other per-cpu work items. + */ +static unsigned long wq_cpu_intensive_thresh_us = 10000; +module_param_named(cpu_intensive_thresh_us, wq_cpu_intensive_thresh_us, ulong, 0644); + static bool wq_disable_numa; module_param_named(disable_numa, wq_disable_numa, bool, 0444); @@ -962,6 +971,13 @@ void wq_worker_running(struct task_struct *task) if (!(worker->flags & WORKER_NOT_RUNNING)) worker->pool->nr_running++; preempt_enable(); + + /* + * CPU intensive auto-detection cares about how long a work item hogged + * CPU without sleeping. Reset the starting timestamp on wakeup. + */ + worker->current_at = worker->task->se.sum_exec_runtime; + worker->sleeping = 0; } @@ -1012,6 +1028,45 @@ void wq_worker_sleeping(struct task_struct *task) raw_spin_unlock_irq(&pool->lock); } +/** + * wq_worker_tick - a scheduler tick occurred while a kworker is running + * @task: task currently running + * + * Called from scheduler_tick(). We're in the IRQ context and the current + * worker's fields which follow the 'K' locking rule can be accessed safely. + */ +void wq_worker_tick(struct task_struct *task) +{ + struct worker *worker = kthread_data(task); + struct pool_workqueue *pwq = worker->current_pwq; + struct worker_pool *pool = worker->pool; + + if (!pwq) + return; + + /* + * If the current worker is concurrency managed and hogged the CPU for + * longer than wq_cpu_intensive_thresh_us, it's automatically marked + * CPU_INTENSIVE to avoid stalling other concurrency-managed work items. + */ + if ((worker->flags & WORKER_NOT_RUNNING) || + worker->task->se.sum_exec_runtime - worker->current_at < + wq_cpu_intensive_thresh_us * NSEC_PER_USEC) + return; + + raw_spin_lock(&pool->lock); + + worker_set_flags(worker, WORKER_CPU_INTENSIVE); + pwq->stats[PWQ_STAT_CPU_INTENSIVE]++; + + if (need_more_worker(pool)) { + pwq->stats[PWQ_STAT_CM_WAKEUP]++; + wake_up_worker(pool); + } + + raw_spin_unlock(&pool->lock); +} + /** * wq_worker_last_func - retrieve worker's last work function * @task: Task to retrieve last work function of. @@ -2325,7 +2380,6 @@ __acquires(&pool->lock) { struct pool_workqueue *pwq = get_work_pwq(work); struct worker_pool *pool = worker->pool; - bool cpu_intensive = pwq->wq->flags & WQ_CPU_INTENSIVE; unsigned long work_data; struct worker *collision; #ifdef CONFIG_LOCKDEP @@ -2362,6 +2416,7 @@ __acquires(&pool->lock) worker->current_work = work; worker->current_func = work->func; worker->current_pwq = pwq; + worker->current_at = worker->task->se.sum_exec_runtime; work_data = *work_data_bits(work); worker->current_color = get_work_color(work_data); @@ -2379,7 +2434,7 @@ __acquires(&pool->lock) * of concurrency management and the next code block will chain * execution of the pending work items. */ - if (unlikely(cpu_intensive)) + if (unlikely(pwq->wq->flags & WQ_CPU_INTENSIVE)) worker_set_flags(worker, WORKER_CPU_INTENSIVE); /* @@ -2459,9 +2514,12 @@ __acquires(&pool->lock) raw_spin_lock_irq(&pool->lock); - /* clear cpu intensive status */ - if (unlikely(cpu_intensive)) - worker_clr_flags(worker, WORKER_CPU_INTENSIVE); + /* + * In addition to %WQ_CPU_INTENSIVE, @worker may also have been marked + * CPU intensive by wq_worker_tick() if @work hogged CPU longer than + * wq_cpu_intensive_thresh_us. Clear it. + */ + worker_clr_flags(worker, WORKER_CPU_INTENSIVE); /* tag the worker for identification in schedule() */ worker->last_func = worker->current_func; diff --git a/kernel/workqueue_internal.h b/kernel/workqueue_internal.h index c2455be7b4c2..6b1d66e28269 100644 --- a/kernel/workqueue_internal.h +++ b/kernel/workqueue_internal.h @@ -31,6 +31,7 @@ struct worker { struct work_struct *current_work; /* K: work being processed and its */ work_func_t current_func; /* K: function */ struct pool_workqueue *current_pwq; /* K: pwq */ + u64 current_at; /* K: runtime at start or last wakeup */ unsigned int current_color; /* K: color */ int sleeping; /* S: is worker sleeping? */ @@ -76,6 +77,7 @@ static inline struct worker *current_wq_worker(void) */ void wq_worker_running(struct task_struct *task); void wq_worker_sleeping(struct task_struct *task); +void wq_worker_tick(struct task_struct *task); work_func_t wq_worker_last_func(struct task_struct *task); #endif /* _KERNEL_WORKQUEUE_INTERNAL_H */ diff --git a/tools/workqueue/wq_monitor.py b/tools/workqueue/wq_monitor.py index 94435fd14c98..5dc0b61b36e8 100644 --- a/tools/workqueue/wq_monitor.py +++ b/tools/workqueue/wq_monitor.py @@ -12,6 +12,11 @@ https://github.com/osandov/drgn. infl The number of currently in-flight work items. + CPUitsv The number of times a concurrency-managed work item hogged CPU + longer than the threshold (workqueue.cpu_intensive_thresh_us) + and got excluded from concurrency management to avoid stalling + other work items. + CMwake The number of concurrency-management wake-ups while executing a work item of the workqueue. @@ -54,6 +59,7 @@ WQ_MEM_RECLAIM = prog['WQ_MEM_RECLAIM'] PWQ_STAT_STARTED = prog['PWQ_STAT_STARTED'] # work items started execution PWQ_STAT_COMPLETED = prog['PWQ_STAT_COMPLETED'] # work items completed execution +PWQ_STAT_CPU_INTENSIVE = prog['PWQ_STAT_CPU_INTENSIVE'] # wq_cpu_intensive_thresh_us violations PWQ_STAT_CM_WAKEUP = prog['PWQ_STAT_CM_WAKEUP'] # concurrency-management worker wakeups PWQ_STAT_MAYDAY = prog['PWQ_STAT_MAYDAY'] # maydays to rescuer PWQ_STAT_RESCUED = prog['PWQ_STAT_RESCUED'] # linked work items executed by rescuer @@ -76,19 +82,23 @@ PWQ_NR_STATS = prog['PWQ_NR_STATS'] 'mem_reclaim' : self.mem_reclaim, 'started' : self.stats[PWQ_STAT_STARTED], 'completed' : self.stats[PWQ_STAT_COMPLETED], + 'cpu_intensive' : self.stats[PWQ_STAT_CPU_INTENSIVE], 'cm_wakeup' : self.stats[PWQ_STAT_CM_WAKEUP], 'mayday' : self.stats[PWQ_STAT_MAYDAY], 'rescued' : self.stats[PWQ_STAT_RESCUED], } def table_header_str(): - return f'{"":>24} {"total":>8} {"infl":>5} {"CMwake":>7} {"mayday":>7} {"rescued":>7}' + return f'{"":>24} {"total":>8} {"infl":>5} {"CPUitsv":>7} '\ + f'{"CMwake":>7} {"mayday":>7} {"rescued":>7}' def table_row_str(self): + cpu_intensive = '-' cm_wakeup = '-' mayday = '-' rescued = '-' if not self.unbound: + cpu_intensive = str(self.stats[PWQ_STAT_CPU_INTENSIVE]) cm_wakeup = str(self.stats[PWQ_STAT_CM_WAKEUP]) if self.mem_reclaim: @@ -98,6 +108,7 @@ PWQ_NR_STATS = prog['PWQ_NR_STATS'] out = f'{self.name[-24:]:24} ' \ f'{self.stats[PWQ_STAT_STARTED]:8} ' \ f'{max(self.stats[PWQ_STAT_STARTED] - self.stats[PWQ_STAT_COMPLETED], 0):5} ' \ + f'{cpu_intensive:>7} ' \ f'{cm_wakeup:>7} ' \ f'{mayday:>7} ' \ f'{rescued:>7} ' From patchwork Thu May 11 18:19:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tejun Heo X-Patchwork-Id: 92808 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp4582844vqo; Thu, 11 May 2023 11:28:38 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5dOUxsfT9u4bi/mw+xtrUW0WNE0jVaqxP+WH7/ZKfyNE+gZKk6/TugPAtBGPgneAjPpgL4 X-Received: by 2002:a05:6a00:2345:b0:63d:3981:313d with SMTP id j5-20020a056a00234500b0063d3981313dmr31687376pfj.10.1683829718459; Thu, 11 May 2023 11:28:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683829718; cv=none; d=google.com; s=arc-20160816; b=qZ2ZE59hT3jLMgyvS1P25jlO7IhdUbBgksjKeCjgXpVsCxPA7+frhOT0HQCf9YvPej NXwhV4NbL2YWrGxbvldaEkAR1CZJgSQs+dgGVQl9YWLSxCqhbIFWBW2crcsNqz7Vc5pM 6uGAC9Dz0IKHwUHj5E7ArFJnncAtRhenmKSzfgcU3SNGFv0BG3FK0mA9yJk7mJO6BQVQ 2PtYexfSsgtrSiKzywyCf3qk1VMqhApFUTy5MEjJhruohXZI08o0geLdrh65cY2v6bew Jii10GYGlLNTXgE9wvdUise5xoGR6DKU2TFQqnLmiEh5fSuznAOXm+FE2ktz4fubzusM ixew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from:sender :dkim-signature; bh=kTlE5rh9U2p4prvVBNqK5UgG3zfgJOt1FV4r5trtoPw=; b=WkN04XjsZPdn9HQPZOdQTIiUqpkOmyKPQO6m4k1S3nTpxs2OxAujN/ApnpmcnPUJBl S/6ZIHx/o4LHJtISS0H+G9M3ymyg4C3piimy9tNx8djrVJJFPcUEGZafbNK7RuxFRT6T Oh7mkdNeDVn3TrkqZYmRzWpXBRcVWaWnsNKmeoWKBQNBXKnTO4I3zDhGzPg5pYgC/oHL vsRVYk/D8tgzHIK4SfBWJ4GzFqB1Ml8IEPoGNILxvbplk/7bjY5sxj1zZaidy4IqLTdY OxOx3zAOgjAATLAjeTx9OZBFA7gYSHirCiZpecL+7wDzAwevGKLmVoT6bwUW9+QecRw8 RZKg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b="EycSu/vj"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id bm18-20020a656e92000000b0052c999cd20csi5135086pgb.899.2023.05.11.11.28.24; Thu, 11 May 2023 11:28:38 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b="EycSu/vj"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239274AbjEKSUP (ORCPT + 99 others); Thu, 11 May 2023 14:20:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44166 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239282AbjEKST6 (ORCPT ); Thu, 11 May 2023 14:19:58 -0400 Received: from mail-pf1-x432.google.com (mail-pf1-x432.google.com [IPv6:2607:f8b0:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A6487A248 for ; Thu, 11 May 2023 11:19:52 -0700 (PDT) Received: by mail-pf1-x432.google.com with SMTP id d2e1a72fcca58-643ac91c51fso5498140b3a.1 for ; Thu, 11 May 2023 11:19:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1683829192; x=1686421192; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:from:to:cc:subject:date :message-id:reply-to; bh=kTlE5rh9U2p4prvVBNqK5UgG3zfgJOt1FV4r5trtoPw=; b=EycSu/vj0nY37kNn+pr12cmZ7gAg9BJPB6B+hp3zWDwRnSBSuPwcaIC8rmfmFREueW 3THfvkyhgE0vl2pXGtYqjUnVBP+T8O8jMVlisCw9uZkog0fnbHX/zg2coob3dqP3zV0x 3QYRnjQsvW6JLOaP7cm5CqsHppg3XU1Y9jUPWi+5YQuGgCX0NSCWjXyWwny4LWFGBLd7 21qlFjbJs+zU4WTGBvZeAAvzZJ5ACICJNhDoa9g3PkzPY24UYbbjpz7CjNLUzbcgUxa6 EkRMkeOoU0fgJZLRS/gY7Lt36qOkVvOWmN3qmDMN4SW1GbRkKGgwUCvwvrQC17WdBc9f LjUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683829192; x=1686421192; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=kTlE5rh9U2p4prvVBNqK5UgG3zfgJOt1FV4r5trtoPw=; b=jp+rWaUF277cxxpg47cak+k6fiY0m0HGzQsM/Xr1Pm0uS0gm+RyKq3hJ9b/O6eQluH dcN36LCUYuHiqDWk6m2tyj3JjI3joX0ojAzxZ8Ytju0N1ztt3hlKqbk9sNU5NOtxuHyL w6ubt4EUAYPoJYrk2mgJlva+u5EnlYXX88GoWUVKUKjb6csZDDIr8NBn8ZHFT7LieZJ2 oHT/k+E6fdVf+cd+n9nR2dCxA1KzRbwzJuamRiJBGrb3rQ9sln6Mh6vkCYYPlmo5p/EG DaP7R3xT+na+P3eeNFNvJI0aZ63PUCoLJ/jtrjnaMUJhSB3Vz3CLsjcSZ5aP/X+Wih16 EdQw== X-Gm-Message-State: AC+VfDxUR50gn2gcrEd0Rwim/jNQIfYI97OttT4MmDiBtnNroWG+MRjg O3PYjVLFXzcRGIBSrF5aVfA= X-Received: by 2002:a05:6a00:2e0e:b0:63d:24c0:1223 with SMTP id fc14-20020a056a002e0e00b0063d24c01223mr32053103pfb.29.1683829191869; Thu, 11 May 2023 11:19:51 -0700 (PDT) Received: from localhost (2603-800c-1a02-1bae-a7fa-157f-969a-4cde.res6.spectrum.com. [2603:800c:1a02:1bae:a7fa:157f:969a:4cde]) by smtp.gmail.com with ESMTPSA id b20-20020aa78714000000b0062dbafced27sm3682194pfo.27.2023.05.11.11.19.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 May 2023 11:19:51 -0700 (PDT) Sender: Tejun Heo From: Tejun Heo To: jiangshanlai@gmail.com Cc: torvalds@linux-foundation.org, peterz@infradead.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, Tejun Heo Subject: [PATCH 6/7] workqueue: Report work funcs that trigger automatic CPU_INTENSIVE mechanism Date: Thu, 11 May 2023 08:19:30 -1000 Message-Id: <20230511181931.869812-7-tj@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230511181931.869812-1-tj@kernel.org> References: <20230511181931.869812-1-tj@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765623430745772902?= X-GMAIL-MSGID: =?utf-8?q?1765623430745772902?= Workqueue now automatically marks per-cpu work items that hog CPU for too long as CPU_INTENSIVE, which excludes them from concurrency management and prevents stalling other concurrency-managed work items. If a work function keeps running over the thershold, it likely needs to be switched to use an unbound workqueue. This patch adds a debug mechanism which tracks the work functions which trigger the automatic CPU_INTENSIVE mechanism and report them using pr_warn() with exponential backoff. Signed-off-by: Tejun Heo Suggested-by: Peter Zijlstra --- kernel/workqueue.c | 128 +++++++++++++++++++++++++++++++++++++++++++++ lib/Kconfig.debug | 13 +++++ 2 files changed, 141 insertions(+) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index 661acc5afcfd..b85d70f133f7 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -948,6 +948,132 @@ static inline void worker_clr_flags(struct worker *worker, unsigned int flags) pool->nr_running++; } +#ifdef CONFIG_WQ_CPU_INTENSIVE_REPORT + +/* + * Concurrency-managed per-cpu work items that hog CPU for longer than + * wq_cpu_intensive_thresh_us trigger the automatic CPU_INTENSIVE mechanism, + * which prevents them from stalling other concurrency-managed work items. If a + * work function keeps triggering this mechanism, it's likely that the work item + * should be using an unbound workqueue instead. + * + * wq_cpu_intensive_report() tracks work functions which trigger such conditions + * and report them so that they can be examined and converted to use unbound + * workqueues as appropriate. To avoid flooding the console, each violating work + * function is tracked and reported with exponential backoff. + */ +#define WCI_MAX_ENTS 128 + +struct wci_ent { + work_func_t func; + atomic64_t cnt; + struct hlist_node hash_node; + struct list_head report_node; +}; + +static struct wci_ent wci_ents[WCI_MAX_ENTS]; +static int wci_nr_ents; +static DEFINE_RAW_SPINLOCK(wci_lock); +static DEFINE_HASHTABLE(wci_hash, ilog2(WCI_MAX_ENTS)); +static LIST_HEAD(wci_report_list); +static struct kthread_worker *wci_report_worker; + +static void wci_report_workfn(struct kthread_work *work) +{ + struct wci_ent *ent, *n; + + raw_spin_lock_irq(&wci_lock); + list_for_each_entry_safe(ent, n, &wci_report_list, report_node) { + pr_warn("workqueue: %ps hogged CPU for >%luus %llu times, consider switching to WQ_UNBOUND\n", + ent->func, wq_cpu_intensive_thresh_us, + atomic64_read(&ent->cnt)); + list_del_init(&ent->report_node); + } + raw_spin_unlock_irq(&wci_lock); +} + +static DEFINE_KTHREAD_WORK(wci_report_work, wci_report_workfn); + +static struct wci_ent *wci_find_ent(work_func_t func) +{ + struct wci_ent *ent; + + hash_for_each_possible_rcu(wci_hash, ent, hash_node, + (unsigned long)func) { + if (ent->func == func) + return ent; + } + return NULL; +} + +static void wq_cpu_intensive_report(work_func_t func) +{ + struct wci_ent *ent; + +restart: + ent = wci_find_ent(func); + if (ent) { + u64 cnt; + + /* + * Start reporting from the fourth time and back off + * exponentially. + */ + cnt = atomic64_inc_return_relaxed(&ent->cnt); + if (cnt < 4 || !is_power_of_2(cnt)) + return; + + raw_spin_lock(&wci_lock); + if (list_empty(&ent->report_node)) { + list_add_tail(&ent->report_node, &wci_report_list); + if (wci_report_worker) + kthread_queue_work(wci_report_worker, + &wci_report_work); + } + raw_spin_unlock(&wci_lock); + return; + } + + /* + * @func is a new violation. Allocate a new entry for it. If wcn_ents[] + * is exhausted, something went really wrong and we probably made enough + * noise already. + */ + if (wci_nr_ents >= WCI_MAX_ENTS) + return; + + raw_spin_lock(&wci_lock); + + if (wci_nr_ents >= WCI_MAX_ENTS) { + raw_spin_unlock(&wci_lock); + return; + } + + if (wci_find_ent(func)) { + raw_spin_unlock(&wci_lock); + goto restart; + } + + ent = &wci_ents[wci_nr_ents++]; + ent->func = func; + atomic64_set(&ent->cnt, 1); + INIT_LIST_HEAD(&ent->report_node); + hash_add_rcu(wci_hash, &ent->hash_node, (unsigned long)func); + + raw_spin_unlock(&wci_lock); +} + +static void __init wq_cpu_intensive_report_init(void) +{ + wci_report_worker = kthread_create_worker(0, "wq_cpu_intensive_report"); + WARN_ON(!wci_report_worker); +} + +#else /* CONFIG_WQ_CPU_INTENSIVE_REPORT */ +static void wq_cpu_intensive_report(work_func_t func) {} +static void __init wq_cpu_intensive_report_init(void) {} +#endif /* CONFIG_WQ_CPU_INTENSIVE_REPORT */ + /** * wq_worker_running - a worker is running again * @task: task waking up @@ -1057,6 +1183,7 @@ void wq_worker_tick(struct task_struct *task) raw_spin_lock(&pool->lock); worker_set_flags(worker, WORKER_CPU_INTENSIVE); + wq_cpu_intensive_report(worker->current_func); pwq->stats[PWQ_STAT_CPU_INTENSIVE]++; if (need_more_worker(pool)) { @@ -6458,6 +6585,7 @@ void __init workqueue_init(void) wq_online = true; wq_watchdog_init(); + wq_cpu_intensive_report_init(); } /* diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index ce51d4dc6803..97e880aa48d7 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -1134,6 +1134,19 @@ config WQ_WATCHDOG state. This can be configured through kernel parameter "workqueue.watchdog_thresh" and its sysfs counterpart. +config WQ_CPU_INTENSIVE_REPORT + bool "Report per-cpu work items which hog CPU for too long" + depends on DEBUG_KERNEL + help + Say Y here to enable reporting of concurrency-managed per-cpu work + items that hog CPUs for longer than + workqueue.cpu_intensive_threshold_us. Workqueue automatically + detects and excludes them from concurrency management to prevent + them from stalling other per-cpu work items. Occassional + triggering may not necessarily indicate a problem. Repeated + triggering likely indicates that the work item should be switched + to use an unbound workqueue. + config TEST_LOCKUP tristate "Test module to generate lockups" depends on m From patchwork Thu May 11 18:19:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tejun Heo X-Patchwork-Id: 92810 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp4583059vqo; Thu, 11 May 2023 11:29:00 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ49FQ+upmOD3cq6VnwR+Z23zHihaBVghqcDLxT0dxc65eeDS2zpjQEXeJYVBZ/6lFwLMpnA X-Received: by 2002:a05:6a00:24d3:b0:647:370c:2c2a with SMTP id d19-20020a056a0024d300b00647370c2c2amr13984116pfv.6.1683829740545; Thu, 11 May 2023 11:29:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683829740; cv=none; d=google.com; s=arc-20160816; b=dq9uakANqDCjwoLlgBcz2Xz3V4JHLUG+cBDhn8WF5oMKvyyAxkDsP3j7mhcqiSNqkW L7jaMNgY2HI381IKKl4U7e9cMnsfgRbNI0bCHSC5FJMT6Q7/JSz+bxg+NkLFp+3sv+rD 9Ryuhol39oKpArVAgSTth10i0oJ97YamIFa9N84YyMYzeb7D/CfXYEtqH7FfmiWdMy7e mXsod/W3okEfEDvFStmTWioXQ7OoeWXQKRHdcGSp6ywcY0oKp5llelICT/arILNRRLi8 zVY5ZhaQ7VU3AlFWuWDjxsFjOCOR4f59je/JAptklV8orkPuK4J1c9OCR1d8cwBC3S6t KJ+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from:sender :dkim-signature; bh=JkYqgFUcqMwizBouzYCG0qlgoOAQQIMFoBJwBvsb0Vs=; b=nsOxKXaJbypz987v72WpJZTY4g6uckdUguYvsXkgHKq8gyamRh99MNGi+GTuv/HLEp 2o5Y8BKweJF2ICJgM7fS1VlsALQL8ZhD+lWNL3RH9jAdpm7cGYAQAHvuaR2WsHbn8AqR 5nIkZWcuC9C+IbI1XScW2QqNo3DjSe4uUenphq1cByXFBH35AVc/lKIDNBw45rD3ifEH uetGRnvMvgPuG/thuOuVgSj22PXI+eesXkTNs5mVBkZdVTK3Gu8QxUvH1908jV8EKMqG ivqDvHpVo9eiAWPNMzJui2D4z0jwtDjAW60bLdOIkPxCCTlYtKlN1jNwIB5Or0Z4CfPR 8tTA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=GjrHUo2E; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y1-20020aa79ae1000000b0063b8f3a4d42si8012668pfp.163.2023.05.11.11.28.47; Thu, 11 May 2023 11:29:00 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=GjrHUo2E; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239211AbjEKSU2 (ORCPT + 99 others); Thu, 11 May 2023 14:20:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44370 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239208AbjEKSUE (ORCPT ); Thu, 11 May 2023 14:20:04 -0400 Received: from mail-pf1-x433.google.com (mail-pf1-x433.google.com [IPv6:2607:f8b0:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8844740ED for ; Thu, 11 May 2023 11:19:55 -0700 (PDT) Received: by mail-pf1-x433.google.com with SMTP id d2e1a72fcca58-644d9bf05b7so4507884b3a.3 for ; Thu, 11 May 2023 11:19:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1683829195; x=1686421195; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:from:to:cc:subject:date :message-id:reply-to; bh=JkYqgFUcqMwizBouzYCG0qlgoOAQQIMFoBJwBvsb0Vs=; b=GjrHUo2ELF5xX6tAJ6WCz3klJ0+A896W4F4H3/SRCD+B3wCrensN8UKMgygN2Iixiq Hm7POiTzcSOEOK1tEgviFTB3lwTeQ3HKnbeCqSbTy0KY+U2WL/MTLnaX74v0bqkPg9/d +VfXmmg5cyILkqqAtcSlNMcFq8GkWDpTZIu6+iawBItBCddnYwF512URFvaNBcm2Okxj CUYM52k+LdvsqFo0a68PkdmPb/GarXaYafROcpzoiFbEaCoh+Zrrw0Y6ikY6VIEuDJlc Slo1ttHJBzLpPYnAWPopfKQVbdSTksiSuDMg8ipFCBHnklXp0BoPJsZIY70t7RS9ns7l 2gkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683829195; x=1686421195; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=JkYqgFUcqMwizBouzYCG0qlgoOAQQIMFoBJwBvsb0Vs=; b=Jsgz9Ftg3T+YjxynCQK7LTOj60RjwF+zKjyWjYkS5qE/Vr+q6FVqBWM50hWmZcmEg+ nbWFHhlo2Dpuro1E/GAowBCiBcMxKS9qtohEMKmRZI69bu987nyRK+/ZLIs9AkG954UK 272Ikuj0r+i+3+7sncg2p+LVjCo4Sx+BwMjzwMMkPLoefpgAMqUwHW8gO5MKL0UzVbhH Q+XWbOIfnYA8WTyU1TNZCZNLTz+R1oM5lKsnkxeyMWHAymWVLH7uoyJkOs3yn2lJJjme GigEwTBtd/gIYbC85bAiw10MsQil8hr/7sWMDprMPMzeJYQlLMIQojZEgzXEaxw1qVcH dtZQ== X-Gm-Message-State: AC+VfDw2khoybVMgHnZfN0FBGiU6fANRkcsm/Lo8tBEXRi4uTHFkMYqw vS5dAj6E6adN3hqkcWFmOZg= X-Received: by 2002:a05:6a00:1950:b0:647:2ce5:57c4 with SMTP id s16-20020a056a00195000b006472ce557c4mr12932714pfk.5.1683829194472; Thu, 11 May 2023 11:19:54 -0700 (PDT) Received: from localhost (2603-800c-1a02-1bae-a7fa-157f-969a-4cde.res6.spectrum.com. [2603:800c:1a02:1bae:a7fa:157f:969a:4cde]) by smtp.gmail.com with ESMTPSA id h5-20020a62b405000000b0062dd9a8c1b8sm5593895pfn.100.2023.05.11.11.19.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 May 2023 11:19:54 -0700 (PDT) Sender: Tejun Heo From: Tejun Heo To: jiangshanlai@gmail.com Cc: torvalds@linux-foundation.org, peterz@infradead.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, Tejun Heo Subject: [PATCH 7/7] workqueue: Track and monitor per-workqueue CPU time usage Date: Thu, 11 May 2023 08:19:31 -1000 Message-Id: <20230511181931.869812-8-tj@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230511181931.869812-1-tj@kernel.org> References: <20230511181931.869812-1-tj@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765623454008625130?= X-GMAIL-MSGID: =?utf-8?q?1765623454008625130?= Now that wq_worker_tick() is there, we can easily track the rough CPU time consumption of each workqueue by charging the whole tick whenever a tick hits an active workqueue. While not super accurate, it provides reasonable visibility into the workqueues that consume a lot of CPU cycles. wq_monitor.py is updated to report the per-workqueue CPU times. Signed-off-by: Tejun Heo --- Documentation/core-api/workqueue.rst | 38 ++++++++++++++-------------- kernel/workqueue.c | 3 +++ tools/workqueue/wq_monitor.py | 9 ++++++- 3 files changed, 30 insertions(+), 20 deletions(-) diff --git a/Documentation/core-api/workqueue.rst b/Documentation/core-api/workqueue.rst index a389f31b025c..a4c9b9d1905f 100644 --- a/Documentation/core-api/workqueue.rst +++ b/Documentation/core-api/workqueue.rst @@ -354,25 +354,25 @@ Monitoring Use tools/workqueue/wq_monitor.py to monitor workqueue operations: :: $ tools/workqueue/wq_monitor.py events - total infl CPUitsv CMwake mayday rescued - events 18545 0 0 5 - - - events_highpri 8 0 0 0 - - - events_long 3 0 0 0 - - - events_unbound 38306 0 - - - - - events_freezable 0 0 0 0 - - - events_power_efficient 29598 0 0 0 - - - events_freezable_power_ 10 0 0 0 - - - sock_diag_events 0 0 0 0 - - - - total infl CPUitsv CMwake mayday rescued - events 18548 0 0 5 - - - events_highpri 8 0 0 0 - - - events_long 3 0 0 0 - - - events_unbound 38322 0 - - - - - events_freezable 0 0 0 0 - - - events_power_efficient 29603 0 0 0 - - - events_freezable_power_ 10 0 0 0 - - - sock_diag_events 0 0 0 0 - - + total infl CPUtime CPUhog CMwake mayday rescued + events 18545 0 6.1 0 5 - - + events_highpri 8 0 0.0 0 0 - - + events_long 3 0 0.0 0 0 - - + events_unbound 38306 0 0.1 - - - - + events_freezable 0 0 0.0 0 0 - - + events_power_efficient 29598 0 0.2 0 0 - - + events_freezable_power_ 10 0 0.0 0 0 - - + sock_diag_events 0 0 0.0 0 0 - - + + total infl CPUtime CPUhog CMwake mayday rescued + events 18548 0 6.1 0 5 - - + events_highpri 8 0 0.0 0 0 - - + events_long 3 0 0.0 0 0 - - + events_unbound 38322 0 0.1 - - - - + events_freezable 0 0 0.0 0 0 - - + events_power_efficient 29603 0 0.2 0 0 - - + events_freezable_power_ 10 0 0.0 0 0 - - + sock_diag_events 0 0 0.0 0 0 - - ... diff --git a/kernel/workqueue.c b/kernel/workqueue.c index b85d70f133f7..42c41d5368dc 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -212,6 +212,7 @@ struct worker_pool { enum pool_workqueue_stats { PWQ_STAT_STARTED, /* work items started execution */ PWQ_STAT_COMPLETED, /* work items completed execution */ + PWQ_STAT_CPU_TIME, /* total CPU time consumed */ PWQ_STAT_CPU_INTENSIVE, /* wq_cpu_intensive_thresh_us violations */ PWQ_STAT_CM_WAKEUP, /* concurrency-management worker wakeups */ PWQ_STAT_MAYDAY, /* maydays to rescuer */ @@ -1170,6 +1171,8 @@ void wq_worker_tick(struct task_struct *task) if (!pwq) return; + pwq->stats[PWQ_STAT_CPU_TIME] += TICK_USEC; + /* * If the current worker is concurrency managed and hogged the CPU for * longer than wq_cpu_intensive_thresh_us, it's automatically marked diff --git a/tools/workqueue/wq_monitor.py b/tools/workqueue/wq_monitor.py index 5dc0b61b36e8..3c35239d50c4 100644 --- a/tools/workqueue/wq_monitor.py +++ b/tools/workqueue/wq_monitor.py @@ -12,6 +12,10 @@ https://github.com/osandov/drgn. infl The number of currently in-flight work items. + CPUtime Total CPU time consumed by the workqueue in seconds. This is + sampled from scheduler ticks and only provides ballpark + measurement. "nohz_full=" CPUs are excluded from measurement. + CPUitsv The number of times a concurrency-managed work item hogged CPU longer than the threshold (workqueue.cpu_intensive_thresh_us) and got excluded from concurrency management to avoid stalling @@ -59,6 +63,7 @@ WQ_MEM_RECLAIM = prog['WQ_MEM_RECLAIM'] PWQ_STAT_STARTED = prog['PWQ_STAT_STARTED'] # work items started execution PWQ_STAT_COMPLETED = prog['PWQ_STAT_COMPLETED'] # work items completed execution +PWQ_STAT_CPU_TIME = prog['PWQ_STAT_CPU_TIME'] # total CPU time consumed PWQ_STAT_CPU_INTENSIVE = prog['PWQ_STAT_CPU_INTENSIVE'] # wq_cpu_intensive_thresh_us violations PWQ_STAT_CM_WAKEUP = prog['PWQ_STAT_CM_WAKEUP'] # concurrency-management worker wakeups PWQ_STAT_MAYDAY = prog['PWQ_STAT_MAYDAY'] # maydays to rescuer @@ -82,13 +87,14 @@ PWQ_NR_STATS = prog['PWQ_NR_STATS'] 'mem_reclaim' : self.mem_reclaim, 'started' : self.stats[PWQ_STAT_STARTED], 'completed' : self.stats[PWQ_STAT_COMPLETED], + 'cputime' : self.stats[PWQ_STAT_CPU_TIME], 'cpu_intensive' : self.stats[PWQ_STAT_CPU_INTENSIVE], 'cm_wakeup' : self.stats[PWQ_STAT_CM_WAKEUP], 'mayday' : self.stats[PWQ_STAT_MAYDAY], 'rescued' : self.stats[PWQ_STAT_RESCUED], } def table_header_str(): - return f'{"":>24} {"total":>8} {"infl":>5} {"CPUitsv":>7} '\ + return f'{"":>24} {"total":>8} {"infl":>5} {"CPUtime":>8} {"CPUitsv":>7} '\ f'{"CMwake":>7} {"mayday":>7} {"rescued":>7}' def table_row_str(self): @@ -108,6 +114,7 @@ PWQ_NR_STATS = prog['PWQ_NR_STATS'] out = f'{self.name[-24:]:24} ' \ f'{self.stats[PWQ_STAT_STARTED]:8} ' \ f'{max(self.stats[PWQ_STAT_STARTED] - self.stats[PWQ_STAT_COMPLETED], 0):5} ' \ + f'{self.stats[PWQ_STAT_CPU_TIME] / 1000000:8.1f} ' \ f'{cpu_intensive:>7} ' \ f'{cm_wakeup:>7} ' \ f'{mayday:>7} ' \