From patchwork Wed Feb 7 05:05:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Namhyung Kim X-Patchwork-Id: 197735 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp2009532dyb; Tue, 6 Feb 2024 21:06:27 -0800 (PST) X-Google-Smtp-Source: AGHT+IFlvZ4fOms8KIeeytfr8tA+0F6zjlDQeEu+RXcqz0FSDS07BUTHgrhi23hzeQYTyupByD// X-Received: by 2002:ac8:580e:0:b0:42c:32cd:140c with SMTP id g14-20020ac8580e000000b0042c32cd140cmr5855923qtg.44.1707282387785; Tue, 06 Feb 2024 21:06:27 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707282387; cv=pass; d=google.com; s=arc-20160816; b=K28/tFyOSjjZ0xNMhQf/CTcVbADvC4wZG+WJrNo4NVkCsm5+45aWNlvR/bEfcMCxwQ vYEuK8/qXlY5qWXvkwYQQ/o/PLKStS+jeAI2OYj+T0RpR+TgRn0Ya8VhQbz1n7c0N8FR hq/0Jzi2xXLZ0tTXRGGOfFod8/3zbkbbxTMhVQWoNIBm6usvUD85IVoBv17QOWJx51Ww RgxXED8S7W+lnEgUziBUJoim43V7znX0GPW7H15DMJkg1xhs0yGfQwzG7V9jKXM9TMzn cPCoXnH37LRyiVVYs6SjaDYEHVG2sHSvUu2PDo+YP5leKhyWuFPgR7duVIHv4kprI3ux lV5g== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:message-id:date:subject:cc:to :from:dkim-signature; bh=52mYTXcGTBrKz+7unLR0dzMeOektUZfolF3cKBPNNhM=; fh=a9f7Fuk7sQeWoAzPVb4U2wKzpbU+HOS627e/zfEizPM=; b=Lxz4k5F5yhyrx6+AcXNTkOCQ4sJZT18XSDhIHsolnaW7AWK7YvAe+LyKJNF4jIec2W v3KzRW+enbDMr22DkNvPlqyn49pPgvmEatJM5X/xwyFYya6r+2flcRlkUNQawjspS89B njH6x3LlN6l5XT8Ad1+vtmHA22fXamqGG+mODJteMJAyJ2wBOSVX1zA8KvM8BYkLJG57 0hVOGSph7mPrceGGoo2MRNyI1NS/wZZYcaHM8sXiLwBOa2Q1pSoeNo8uLWbIc5IcWC34 tzqkvw8nGh6HitCnh6AUno80Vw+oO5IXhpWE/YVzsPVQGcIAnP5vivDnunec60y954fD 1qsA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Xsui2B8z; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55970-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55970-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=2; AJvYcCUNtE9slWxtq6SmO+i9EjR7JTEgciVjCCGl6kYSLGjf36466I86MP+QqVeDWneim9V1DSX2VrgBhjXvdd5YwHtooshgYw== Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id b7-20020a05622a020700b0042c2c06dcbcsi448619qtx.719.2024.02.06.21.06.27 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 21:06:27 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-55970-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Xsui2B8z; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-55970-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-55970-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 82C8C1C24E12 for ; Wed, 7 Feb 2024 05:06:27 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 2B62F1CF9C; Wed, 7 Feb 2024 05:05:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Xsui2B8z" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 18EF41AAA5 for ; Wed, 7 Feb 2024 05:05:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707282348; cv=none; b=dMRLh3G+JN6oD3TALcoAU7GhcCNvMDuK9L+5UypcZbN9s8vxoEoqjkw+UfgC0xoTzET+tZZ9awpMg+wJUNyNTOmO/YosK6xPoVDMZIMalDSfxzs9xWKyDq5ZaAd7T5f3hkaWHPPBrrg/UA3c0o44purvWQWgkY7BE9MjebYafY0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707282348; c=relaxed/simple; bh=KO8aCqFSDUR/kXIhzYaIU5Xdq8meb2AOBlSD7gNJRzM=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=syoTWPFqjmAz6TEg3iG93CfZpDfpLgdFFtsIcVkRVrSQgzFenT/Z8ZMBjznkFA1Ycq0oj75+cxCefvBRGRsFeiU4zmUAMaXNH0IRUNML5GJKF7RBighITNN2j9c2gx9wMRaXm2A9q6ETV+GUIr3U1Vg87NhvyzA/2Q+tszU8HOs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Xsui2B8z; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 269F6C43390; Wed, 7 Feb 2024 05:05:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707282347; bh=KO8aCqFSDUR/kXIhzYaIU5Xdq8meb2AOBlSD7gNJRzM=; h=From:To:Cc:Subject:Date:From; b=Xsui2B8zeKksF5RoXhLUKtZLmLE2+Z8YUqqUVlvDF4L+YwhQ4Ea6fGlhURSVaJKtn fm/gizNluTnfdYWj6cNKTZiqn6ASQNe+mAHqNESbzrJ0h9ei2biDUE/pywUlqcOXMf ypFx/heuXNhFdxaCmR4Ir8RMNxllWeN+SahTsczK/bS0XUol7WmNJ8QbblHZWKWd7T NSFn1mc4v4AsxClRRl4iVv7FFCYdqhAxDA3G0ctEVic6n79Ty99drb0KfsXoLrD8WC /6q5TNBYbCyAdQlHFxXe8r+QG1+Auam7Oip09Vs9tgbZDfzyCAI6rubUPjNyKV5Dt8 4w/Cm10nYt0EQ== From: Namhyung Kim To: Peter Zijlstra , Ingo Molnar Cc: Mark Rutland , Alexander Shishkin , Arnaldo Carvalho de Melo , Ian Rogers , Adrian Hunter , Kan Liang , LKML , Mingwei Zhang Subject: [PATCH v3 1/2] perf/core: Update perf_adjust_freq_unthr_context() Date: Tue, 6 Feb 2024 21:05:44 -0800 Message-ID: <20240207050545.2727923-1-namhyung@kernel.org> X-Mailer: git-send-email 2.43.0.594.gd9cf4e227d-goog Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790215336794052668 X-GMAIL-MSGID: 1790215336794052668 It was unnecessarily disabling and enabling PMUs for each event. It should be done at PMU level. Add pmu_ctx->nr_freq counter to check it at each PMU. As pmu context has separate active lists for pinned group and flexible group, factor out a new function to do the job. Another minor optimization is that it can skip PMUs w/ CAP_NO_INTERRUPT even if it needs to unthrottle sampling events. Reviewed-by: Ian Rogers Reviewed-by: Kan Liang Tested-by: Mingwei Zhang Signed-off-by: Namhyung Kim --- include/linux/perf_event.h | 6 ++++ kernel/events/core.c | 70 ++++++++++++++++++++++++-------------- 2 files changed, 50 insertions(+), 26 deletions(-) diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h index d2a15c0c6f8a..3e33b366347a 100644 --- a/include/linux/perf_event.h +++ b/include/linux/perf_event.h @@ -883,6 +883,7 @@ struct perf_event_pmu_context { unsigned int nr_events; unsigned int nr_cgroups; + unsigned int nr_freq; atomic_t refcount; /* event <-> epc */ struct rcu_head rcu_head; @@ -897,6 +898,11 @@ struct perf_event_pmu_context { int rotate_necessary; }; +static inline bool perf_pmu_ctx_is_active(struct perf_event_pmu_context *epc) +{ + return !list_empty(&epc->flexible_active) || !list_empty(&epc->pinned_active); +} + struct perf_event_groups { struct rb_root tree; u64 index; diff --git a/kernel/events/core.c b/kernel/events/core.c index 59b332cce9e7..e9ce79c8c145 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -2277,8 +2277,10 @@ event_sched_out(struct perf_event *event, struct perf_event_context *ctx) if (!is_software_event(event)) cpc->active_oncpu--; - if (event->attr.freq && event->attr.sample_freq) + if (event->attr.freq && event->attr.sample_freq) { ctx->nr_freq--; + epc->nr_freq--; + } if (event->attr.exclusive || !cpc->active_oncpu) cpc->exclusive = 0; @@ -2533,9 +2535,10 @@ event_sched_in(struct perf_event *event, struct perf_event_context *ctx) if (!is_software_event(event)) cpc->active_oncpu++; - if (event->attr.freq && event->attr.sample_freq) + if (event->attr.freq && event->attr.sample_freq) { ctx->nr_freq++; - + epc->nr_freq++; + } if (event->attr.exclusive) cpc->exclusive = 1; @@ -4098,30 +4101,14 @@ static void perf_adjust_period(struct perf_event *event, u64 nsec, u64 count, bo } } -/* - * combine freq adjustment with unthrottling to avoid two passes over the - * events. At the same time, make sure, having freq events does not change - * the rate of unthrottling as that would introduce bias. - */ -static void -perf_adjust_freq_unthr_context(struct perf_event_context *ctx, bool unthrottle) +static void perf_adjust_freq_unthr_events(struct list_head *event_list) { struct perf_event *event; struct hw_perf_event *hwc; u64 now, period = TICK_NSEC; s64 delta; - /* - * only need to iterate over all events iff: - * - context have events in frequency mode (needs freq adjust) - * - there are events to unthrottle on this cpu - */ - if (!(ctx->nr_freq || unthrottle)) - return; - - raw_spin_lock(&ctx->lock); - - list_for_each_entry_rcu(event, &ctx->event_list, event_entry) { + list_for_each_entry(event, event_list, active_list) { if (event->state != PERF_EVENT_STATE_ACTIVE) continue; @@ -4129,8 +4116,6 @@ perf_adjust_freq_unthr_context(struct perf_event_context *ctx, bool unthrottle) if (!event_filter_match(event)) continue; - perf_pmu_disable(event->pmu); - hwc = &event->hw; if (hwc->interrupts == MAX_INTERRUPTS) { @@ -4140,7 +4125,7 @@ perf_adjust_freq_unthr_context(struct perf_event_context *ctx, bool unthrottle) } if (!event->attr.freq || !event->attr.sample_freq) - goto next; + continue; /* * stop the event and update event->count @@ -4162,8 +4147,41 @@ perf_adjust_freq_unthr_context(struct perf_event_context *ctx, bool unthrottle) perf_adjust_period(event, period, delta, false); event->pmu->start(event, delta > 0 ? PERF_EF_RELOAD : 0); - next: - perf_pmu_enable(event->pmu); + } +} + +/* + * combine freq adjustment with unthrottling to avoid two passes over the + * events. At the same time, make sure, having freq events does not change + * the rate of unthrottling as that would introduce bias. + */ +static void +perf_adjust_freq_unthr_context(struct perf_event_context *ctx, bool unthrottle) +{ + struct perf_event_pmu_context *pmu_ctx; + + /* + * only need to iterate over all events iff: + * - context have events in frequency mode (needs freq adjust) + * - there are events to unthrottle on this cpu + */ + if (!(ctx->nr_freq || unthrottle)) + return; + + raw_spin_lock(&ctx->lock); + + list_for_each_entry(pmu_ctx, &ctx->pmu_ctx_list, pmu_ctx_entry) { + if (!(pmu_ctx->nr_freq || unthrottle)) + continue; + if (!perf_pmu_ctx_is_active(pmu_ctx)) + continue; + if (pmu_ctx->pmu->capabilities & PERF_PMU_CAP_NO_INTERRUPT) + continue; + + perf_pmu_disable(pmu_ctx->pmu); + perf_adjust_freq_unthr_events(&pmu_ctx->pinned_active); + perf_adjust_freq_unthr_events(&pmu_ctx->flexible_active); + perf_pmu_enable(pmu_ctx->pmu); } raw_spin_unlock(&ctx->lock);