From patchwork Wed May 31 04:04:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anshuman Khandual X-Patchwork-Id: 101221 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2630222vqr; Tue, 30 May 2023 21:26:23 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4sqKmiGtru9mK4ne+QW9ArQyPgDAOH+8KHcHXGf4VP+pfDS2DA2DQH7TAtBxlMh8jMO5IG X-Received: by 2002:a0d:f3c6:0:b0:568:a870:314f with SMTP id c189-20020a0df3c6000000b00568a870314fmr4970530ywf.30.1685507183564; Tue, 30 May 2023 21:26:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685507183; cv=none; d=google.com; s=arc-20160816; b=Y+lJyYe3sPclcl/ZAOqLp4LVsHYVdg/BpKRoT+KTsLdB6OwxuKds05PUuKoCjiZFkD z8UgdUcR0vLYp17GJMJZW9ZirX5wOj32fqV4tLpLJzNci2GZEsMYjjJ2XqyRqIQpFCKq tNFzVzRP3AlM/7/tdsoXbtnaQ/yKm3omRmloMwEsEYimeCapsAhrtEf2QQNMAXgXRkLt kzPC7gDas1iDtctgTQn1EFqcMRewk/X6UW/Ri0nyvTWUwGWli+OF3Fd1Xtzn0ie38oI5 dy/yvEJyHTZZmH72eBeQuKqObQhyU1Y13pXoyklE7wpvsSUOwG4oLZQrmgDBK+s0HZSk ym8g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=IPTSnlkPjTSGS6O7j/v1RRpRTPcMzBWksWREu64LHFQ=; b=01ptjpUrlFzp7C/aoSNKE0mTkO7tuo+Hm12sbGi28gql8tNy4ZZ55YlX9dIjsX/2E9 UmTFhq8EMnpAGWNkf3M6L/GHHCM5BnBbwUJgjRt4rhV0id+dAkWKSVdgtQTTahbdZtUg 1R7GCpgAwnbsy52whXrgL1/yEwnfSYAv/2UiOSNyB3jV5SPML6l4L4LF014VmCN0IX2Q C9AJTyL+458+FfogVcdlL+UPlLsxU7jMJ+yuCBXnUYDHOHdpfnhd2xNsP1+3c/Owfueu KrOV5oTNVtf1QC1YN6x240Wz7XHzlVIys7LpwfFpBSGmY014ennqw2jR3+tUr09s9S1l Lwgg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id w28-20020aa7955c000000b0064d3e651979si633336pfq.233.2023.05.30.21.26.11; Tue, 30 May 2023 21:26:23 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234223AbjEaEFJ (ORCPT + 99 others); Wed, 31 May 2023 00:05:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54784 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233995AbjEaEFD (ORCPT ); Wed, 31 May 2023 00:05:03 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 2C04211F; Tue, 30 May 2023 21:05:02 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 3811D15DB; Tue, 30 May 2023 21:05:47 -0700 (PDT) Received: from a077893.arm.com (unknown [10.163.73.163]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 9BF413F6C4; Tue, 30 May 2023 21:04:55 -0700 (PDT) From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, will@kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com Cc: Anshuman Khandual , Mark Brown , James Clark , Rob Herring , Marc Zyngier , Suzuki Poulose , Peter Zijlstra , Ingo Molnar , Arnaldo Carvalho de Melo , linux-perf-users@vger.kernel.org Subject: [PATCH V11 01/10] drivers: perf: arm_pmu: Add new sched_task() callback Date: Wed, 31 May 2023 09:34:19 +0530 Message-Id: <20230531040428.501523-2-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230531040428.501523-1-anshuman.khandual@arm.com> References: <20230531040428.501523-1-anshuman.khandual@arm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1767382380053104399?= X-GMAIL-MSGID: =?utf-8?q?1767382380053104399?= This adds armpmu_sched_task(), as generic pmu's sched_task() override which in turn can utilize a new arm_pmu.sched_task() callback when available from the arm_pmu instance. This new callback will be used while enabling BRBE in ARMV8 PMU. Cc: Catalin Marinas Cc: Will Deacon Cc: Mark Rutland Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Tested-by: James Clark Signed-off-by: Anshuman Khandual Acked-by: Mark Rutland --- drivers/perf/arm_pmu.c | 9 +++++++++ include/linux/perf/arm_pmu.h | 1 + 2 files changed, 10 insertions(+) diff --git a/drivers/perf/arm_pmu.c b/drivers/perf/arm_pmu.c index 15bd1e34a88e..aada47e3b126 100644 --- a/drivers/perf/arm_pmu.c +++ b/drivers/perf/arm_pmu.c @@ -517,6 +517,14 @@ static int armpmu_event_init(struct perf_event *event) return __hw_perf_event_init(event); } +static void armpmu_sched_task(struct perf_event_pmu_context *pmu_ctx, bool sched_in) +{ + struct arm_pmu *armpmu = to_arm_pmu(pmu_ctx->pmu); + + if (armpmu->sched_task) + armpmu->sched_task(pmu_ctx, sched_in); +} + static void armpmu_enable(struct pmu *pmu) { struct arm_pmu *armpmu = to_arm_pmu(pmu); @@ -858,6 +866,7 @@ struct arm_pmu *armpmu_alloc(void) } pmu->pmu = (struct pmu) { + .sched_task = armpmu_sched_task, .pmu_enable = armpmu_enable, .pmu_disable = armpmu_disable, .event_init = armpmu_event_init, diff --git a/include/linux/perf/arm_pmu.h b/include/linux/perf/arm_pmu.h index 525b5d64e394..f7fbd162ca4c 100644 --- a/include/linux/perf/arm_pmu.h +++ b/include/linux/perf/arm_pmu.h @@ -100,6 +100,7 @@ struct arm_pmu { void (*stop)(struct arm_pmu *); void (*reset)(void *); int (*map_event)(struct perf_event *event); + void (*sched_task)(struct perf_event_pmu_context *pmu_ctx, bool sched_in); int num_events; bool secure_access; /* 32-bit ARM only */ #define ARMV8_PMUV3_MAX_COMMON_EVENTS 0x40 From patchwork Wed May 31 04:04:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anshuman Khandual X-Patchwork-Id: 101223 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2630345vqr; Tue, 30 May 2023 21:26:44 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ62ZCH0wP+yrOubcknxDcFUbFZHUnrCg2HbuVmrWP5eRBpIc1Rf9L4J/XkBO9f5oAkqvZ1I X-Received: by 2002:a05:6870:90c5:b0:19f:65db:4d7b with SMTP id s5-20020a05687090c500b0019f65db4d7bmr2462929oab.18.1685507203831; Tue, 30 May 2023 21:26:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685507203; cv=none; d=google.com; s=arc-20160816; b=AIvK7bmXyMwFKvUZ41shaFD4+r7QI0u4DrvQymXTOYxBFo5HhQaMHXwr0ZDX9SIhEk 4V/qVfZoG1/BOTyyPva/8KvHzZsvSV2nxpGz8HdatHJCKpnl9uV+gk/78z4KvwgVTepl DK39nl10K07rg0AyoguMK42oIoZZFFyg42OmJLn27ONl0HxiAQOSfcODqlEycxFtSoqF CtBClWH9m/roh0ALi7c6B9vSwi22qlCM8qjANa7Kl1OuvY6rqVsqIBrlRVMrTYImMqA6 itNeV8kDv7CF3Qe9Nl9RirxfDZ1yx+vQ5gZuwUYrNk743Todw4x/04J7lPK3CT23ZO+g TBBQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=oH1wLj1jXavF9G4zfXNpqs8ubOrW8TLf4nVc4QOZ1io=; b=pXAopezkTx1fxGwOiUFBqRWaTMQ3j5QkJ0GtgYJv9FRkVyJ+DwUr+GLVA7O5Hhu5PL Oengy1K70+GNqoqNv8xg0Zfq1Lbe9D9t5ApK1OUbYI7bwA/LAX7aB46EVK2edA6E2epq 3NkqlrKr2hO7p+9mSRykSyRsKwaIY/AbIQz/vOslw+fj98R9+w20bfkhAjIzYiYx/SxK U6gzhmt5acWVZH8gidau8YJWkgWcbTF8Z1EqkA9oB6w+wH1WSQ5wHk0m3bJdBtTLoivD LP715wHxz5ugcBPQ7nRuP8WnrPcreIOLm0Mm/tx1JFnQ3opBpja4GRTDJvL6E27/oSIK Z3/Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id v23-20020a63b957000000b0053fbb2ddb4csi329013pgo.36.2023.05.30.21.26.31; Tue, 30 May 2023 21:26:43 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234230AbjEaEF3 (ORCPT + 99 others); Wed, 31 May 2023 00:05:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55106 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234229AbjEaEFT (ORCPT ); Wed, 31 May 2023 00:05:19 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 6B2BB13E; Tue, 30 May 2023 21:05:08 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7449E15DB; Tue, 30 May 2023 21:05:53 -0700 (PDT) Received: from a077893.arm.com (unknown [10.163.73.163]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id CDDB43F6C4; Tue, 30 May 2023 21:05:02 -0700 (PDT) From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, will@kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com Cc: Anshuman Khandual , Mark Brown , James Clark , Rob Herring , Marc Zyngier , Suzuki Poulose , Peter Zijlstra , Ingo Molnar , Arnaldo Carvalho de Melo , linux-perf-users@vger.kernel.org Subject: [PATCH V11 02/10] arm64/perf: Add BRBE registers and fields Date: Wed, 31 May 2023 09:34:20 +0530 Message-Id: <20230531040428.501523-3-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230531040428.501523-1-anshuman.khandual@arm.com> References: <20230531040428.501523-1-anshuman.khandual@arm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1767382402032452046?= X-GMAIL-MSGID: =?utf-8?q?1767382402032452046?= This adds BRBE related register definitions and various other related field macros there in. These will be used subsequently in a BRBE driver which is being added later on. Cc: Catalin Marinas Cc: Will Deacon Cc: Marc Zyngier Cc: Mark Rutland Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Tested-by: James Clark Reviewed-by: Mark Brown Signed-off-by: Anshuman Khandual --- arch/arm64/include/asm/sysreg.h | 103 +++++++++++++++++++++ arch/arm64/tools/sysreg | 159 ++++++++++++++++++++++++++++++++ 2 files changed, 262 insertions(+) diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index e72d9aaab6b1..12419c55d3b7 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -165,6 +165,109 @@ #define SYS_DBGDTRTX_EL0 sys_reg(2, 3, 0, 5, 0) #define SYS_DBGVCR32_EL2 sys_reg(2, 4, 0, 7, 0) +#define __SYS_BRBINFO(n) sys_reg(2, 1, 8, ((n) & 0xf), ((((n) & 0x10)) >> 2 + 0)) +#define __SYS_BRBSRC(n) sys_reg(2, 1, 8, ((n) & 0xf), ((((n) & 0x10)) >> 2 + 1)) +#define __SYS_BRBTGT(n) sys_reg(2, 1, 8, ((n) & 0xf), ((((n) & 0x10)) >> 2 + 2)) + +#define SYS_BRBINF0_EL1 __SYS_BRBINFO(0) +#define SYS_BRBINF1_EL1 __SYS_BRBINFO(1) +#define SYS_BRBINF2_EL1 __SYS_BRBINFO(2) +#define SYS_BRBINF3_EL1 __SYS_BRBINFO(3) +#define SYS_BRBINF4_EL1 __SYS_BRBINFO(4) +#define SYS_BRBINF5_EL1 __SYS_BRBINFO(5) +#define SYS_BRBINF6_EL1 __SYS_BRBINFO(6) +#define SYS_BRBINF7_EL1 __SYS_BRBINFO(7) +#define SYS_BRBINF8_EL1 __SYS_BRBINFO(8) +#define SYS_BRBINF9_EL1 __SYS_BRBINFO(9) +#define SYS_BRBINF10_EL1 __SYS_BRBINFO(10) +#define SYS_BRBINF11_EL1 __SYS_BRBINFO(11) +#define SYS_BRBINF12_EL1 __SYS_BRBINFO(12) +#define SYS_BRBINF13_EL1 __SYS_BRBINFO(13) +#define SYS_BRBINF14_EL1 __SYS_BRBINFO(14) +#define SYS_BRBINF15_EL1 __SYS_BRBINFO(15) +#define SYS_BRBINF16_EL1 __SYS_BRBINFO(16) +#define SYS_BRBINF17_EL1 __SYS_BRBINFO(17) +#define SYS_BRBINF18_EL1 __SYS_BRBINFO(18) +#define SYS_BRBINF19_EL1 __SYS_BRBINFO(19) +#define SYS_BRBINF20_EL1 __SYS_BRBINFO(20) +#define SYS_BRBINF21_EL1 __SYS_BRBINFO(21) +#define SYS_BRBINF22_EL1 __SYS_BRBINFO(22) +#define SYS_BRBINF23_EL1 __SYS_BRBINFO(23) +#define SYS_BRBINF24_EL1 __SYS_BRBINFO(24) +#define SYS_BRBINF25_EL1 __SYS_BRBINFO(25) +#define SYS_BRBINF26_EL1 __SYS_BRBINFO(26) +#define SYS_BRBINF27_EL1 __SYS_BRBINFO(27) +#define SYS_BRBINF28_EL1 __SYS_BRBINFO(28) +#define SYS_BRBINF29_EL1 __SYS_BRBINFO(29) +#define SYS_BRBINF30_EL1 __SYS_BRBINFO(30) +#define SYS_BRBINF31_EL1 __SYS_BRBINFO(31) + +#define SYS_BRBSRC0_EL1 __SYS_BRBSRC(0) +#define SYS_BRBSRC1_EL1 __SYS_BRBSRC(1) +#define SYS_BRBSRC2_EL1 __SYS_BRBSRC(2) +#define SYS_BRBSRC3_EL1 __SYS_BRBSRC(3) +#define SYS_BRBSRC4_EL1 __SYS_BRBSRC(4) +#define SYS_BRBSRC5_EL1 __SYS_BRBSRC(5) +#define SYS_BRBSRC6_EL1 __SYS_BRBSRC(6) +#define SYS_BRBSRC7_EL1 __SYS_BRBSRC(7) +#define SYS_BRBSRC8_EL1 __SYS_BRBSRC(8) +#define SYS_BRBSRC9_EL1 __SYS_BRBSRC(9) +#define SYS_BRBSRC10_EL1 __SYS_BRBSRC(10) +#define SYS_BRBSRC11_EL1 __SYS_BRBSRC(11) +#define SYS_BRBSRC12_EL1 __SYS_BRBSRC(12) +#define SYS_BRBSRC13_EL1 __SYS_BRBSRC(13) +#define SYS_BRBSRC14_EL1 __SYS_BRBSRC(14) +#define SYS_BRBSRC15_EL1 __SYS_BRBSRC(15) +#define SYS_BRBSRC16_EL1 __SYS_BRBSRC(16) +#define SYS_BRBSRC17_EL1 __SYS_BRBSRC(17) +#define SYS_BRBSRC18_EL1 __SYS_BRBSRC(18) +#define SYS_BRBSRC19_EL1 __SYS_BRBSRC(19) +#define SYS_BRBSRC20_EL1 __SYS_BRBSRC(20) +#define SYS_BRBSRC21_EL1 __SYS_BRBSRC(21) +#define SYS_BRBSRC22_EL1 __SYS_BRBSRC(22) +#define SYS_BRBSRC23_EL1 __SYS_BRBSRC(23) +#define SYS_BRBSRC24_EL1 __SYS_BRBSRC(24) +#define SYS_BRBSRC25_EL1 __SYS_BRBSRC(25) +#define SYS_BRBSRC26_EL1 __SYS_BRBSRC(26) +#define SYS_BRBSRC27_EL1 __SYS_BRBSRC(27) +#define SYS_BRBSRC28_EL1 __SYS_BRBSRC(28) +#define SYS_BRBSRC29_EL1 __SYS_BRBSRC(29) +#define SYS_BRBSRC30_EL1 __SYS_BRBSRC(30) +#define SYS_BRBSRC31_EL1 __SYS_BRBSRC(31) + +#define SYS_BRBTGT0_EL1 __SYS_BRBTGT(0) +#define SYS_BRBTGT1_EL1 __SYS_BRBTGT(1) +#define SYS_BRBTGT2_EL1 __SYS_BRBTGT(2) +#define SYS_BRBTGT3_EL1 __SYS_BRBTGT(3) +#define SYS_BRBTGT4_EL1 __SYS_BRBTGT(4) +#define SYS_BRBTGT5_EL1 __SYS_BRBTGT(5) +#define SYS_BRBTGT6_EL1 __SYS_BRBTGT(6) +#define SYS_BRBTGT7_EL1 __SYS_BRBTGT(7) +#define SYS_BRBTGT8_EL1 __SYS_BRBTGT(8) +#define SYS_BRBTGT9_EL1 __SYS_BRBTGT(9) +#define SYS_BRBTGT10_EL1 __SYS_BRBTGT(10) +#define SYS_BRBTGT11_EL1 __SYS_BRBTGT(11) +#define SYS_BRBTGT12_EL1 __SYS_BRBTGT(12) +#define SYS_BRBTGT13_EL1 __SYS_BRBTGT(13) +#define SYS_BRBTGT14_EL1 __SYS_BRBTGT(14) +#define SYS_BRBTGT15_EL1 __SYS_BRBTGT(15) +#define SYS_BRBTGT16_EL1 __SYS_BRBTGT(16) +#define SYS_BRBTGT17_EL1 __SYS_BRBTGT(17) +#define SYS_BRBTGT18_EL1 __SYS_BRBTGT(18) +#define SYS_BRBTGT19_EL1 __SYS_BRBTGT(19) +#define SYS_BRBTGT20_EL1 __SYS_BRBTGT(20) +#define SYS_BRBTGT21_EL1 __SYS_BRBTGT(21) +#define SYS_BRBTGT22_EL1 __SYS_BRBTGT(22) +#define SYS_BRBTGT23_EL1 __SYS_BRBTGT(23) +#define SYS_BRBTGT24_EL1 __SYS_BRBTGT(24) +#define SYS_BRBTGT25_EL1 __SYS_BRBTGT(25) +#define SYS_BRBTGT26_EL1 __SYS_BRBTGT(26) +#define SYS_BRBTGT27_EL1 __SYS_BRBTGT(27) +#define SYS_BRBTGT28_EL1 __SYS_BRBTGT(28) +#define SYS_BRBTGT29_EL1 __SYS_BRBTGT(29) +#define SYS_BRBTGT30_EL1 __SYS_BRBTGT(30) +#define SYS_BRBTGT31_EL1 __SYS_BRBTGT(31) + #define SYS_MIDR_EL1 sys_reg(3, 0, 0, 0, 0) #define SYS_MPIDR_EL1 sys_reg(3, 0, 0, 0, 5) #define SYS_REVIDR_EL1 sys_reg(3, 0, 0, 0, 6) diff --git a/arch/arm64/tools/sysreg b/arch/arm64/tools/sysreg index c9a0d1fa3209..44745f42262f 100644 --- a/arch/arm64/tools/sysreg +++ b/arch/arm64/tools/sysreg @@ -947,6 +947,165 @@ UnsignedEnum 3:0 BT EndEnum EndSysreg + +SysregFields BRBINFx_EL1 +Res0 63:47 +Field 46 CCU +Field 45:32 CC +Res0 31:18 +Field 17 LASTFAILED +Field 16 T +Res0 15:14 +Enum 13:8 TYPE + 0b000000 UNCOND_DIR + 0b000001 INDIR + 0b000010 DIR_LINK + 0b000011 INDIR_LINK + 0b000101 RET_SUB + 0b000111 RET_EXCPT + 0b001000 COND_DIR + 0b100001 DEBUG_HALT + 0b100010 CALL + 0b100011 TRAP + 0b100100 SERROR + 0b100110 INST_DEBUG + 0b100111 DATA_DEBUG + 0b101010 ALGN_FAULT + 0b101011 INST_FAULT + 0b101100 DATA_FAULT + 0b101110 IRQ + 0b101111 FIQ + 0b111001 DEBUG_EXIT +EndEnum +Enum 7:6 EL + 0b00 EL0 + 0b01 EL1 + 0b10 EL2 + 0b11 EL3 +EndEnum +Field 5 MPRED +Res0 4:2 +Enum 1:0 VALID + 0b00 NONE + 0b01 TARGET + 0b10 SOURCE + 0b11 FULL +EndEnum +EndSysregFields + +Sysreg BRBCR_EL1 2 1 9 0 0 +Res0 63:24 +Field 23 EXCEPTION +Field 22 ERTN +Res0 21:9 +Field 8 FZP +Res0 7 +Enum 6:5 TS + 0b01 VIRTUAL + 0b10 GST_PHYSICAL + 0b11 PHYSICAL +EndEnum +Field 4 MPRED +Field 3 CC +Res0 2 +Field 1 E1BRE +Field 0 E0BRE +EndSysreg + +Sysreg BRBFCR_EL1 2 1 9 0 1 +Res0 63:30 +Enum 29:28 BANK + 0b0 FIRST + 0b1 SECOND +EndEnum +Res0 27:23 +Field 22 CONDDIR +Field 21 DIRCALL +Field 20 INDCALL +Field 19 RTN +Field 18 INDIRECT +Field 17 DIRECT +Field 16 EnI +Res0 15:8 +Field 7 PAUSED +Field 6 LASTFAILED +Res0 5:0 +EndSysreg + +Sysreg BRBTS_EL1 2 1 9 0 2 +Field 63:0 TS +EndSysreg + +Sysreg BRBINFINJ_EL1 2 1 9 1 0 +Res0 63:47 +Field 46 CCU +Field 45:32 CC +Res0 31:18 +Field 17 LASTFAILED +Field 16 T +Res0 15:14 +Enum 13:8 TYPE + 0b000000 UNCOND_DIR + 0b000001 INDIR + 0b000010 DIR_LINK + 0b000011 INDIR_LINK + 0b000100 RET_SUB + 0b000100 RET_SUB + 0b000111 RET_EXCPT + 0b001000 COND_DIR + 0b100001 DEBUG_HALT + 0b100010 CALL + 0b100011 TRAP + 0b100100 SERROR + 0b100110 INST_DEBUG + 0b100111 DATA_DEBUG + 0b101010 ALGN_FAULT + 0b101011 INST_FAULT + 0b101100 DATA_FAULT + 0b101110 IRQ + 0b101111 FIQ + 0b111001 DEBUG_EXIT +EndEnum +Enum 7:6 EL + 0b00 EL0 + 0b01 EL1 + 0b10 EL2 + 0b11 EL3 +EndEnum +Field 5 MPRED +Res0 4:2 +Enum 1:0 VALID + 0b00 NONE + 0b01 TARGET + 0b10 SOURCE + 0b00 FULL +EndEnum +EndSysreg + +Sysreg BRBSRCINJ_EL1 2 1 9 1 1 +Field 63:0 ADDRESS +EndSysreg + +Sysreg BRBTGTINJ_EL1 2 1 9 1 2 +Field 63:0 ADDRESS +EndSysreg + +Sysreg BRBIDR0_EL1 2 1 9 2 0 +Res0 63:16 +Enum 15:12 CC + 0b101 20_BIT +EndEnum +Enum 11:8 FORMAT + 0b0 0 +EndEnum +Enum 7:0 NUMREC + 0b1000 8 + 0b10000 16 + 0b100000 32 + 0b1000000 64 +EndEnum +EndSysreg + Sysreg ID_AA64ZFR0_EL1 3 0 0 4 4 Res0 63:60 UnsignedEnum 59:56 F64MM From patchwork Wed May 31 04:04:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anshuman Khandual X-Patchwork-Id: 101219 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2629608vqr; Tue, 30 May 2023 21:24:13 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4BhI8tgHBPxe3rgt3Be3NhI1eFWRR5NOgSoFh7P1ONJJM0sL5yMX92RJYObu7OlI3dfkH0 X-Received: by 2002:a81:4857:0:b0:559:fad8:322f with SMTP id v84-20020a814857000000b00559fad8322fmr5184262ywa.24.1685507053418; Tue, 30 May 2023 21:24:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685507053; cv=none; d=google.com; s=arc-20160816; b=AU/SkjipWHL76U+86cDOboJr99UkF5MllqWtKE0AGis5UcFocVeGaTAIAGZsRXKgzz Cj00bPSjTbADJDRMoXcgsfIC88eIc7qsC3qSjpOKqoJDCS/jld+T2nEPAWVdyRGrmR9K Te2HpmnHbIyV/oTV67595akakj6OnCe7r2Fv+rrdMh0eLMyGY3gE5bODJ85Q759eVZkq grp84sNB9yv771YgKY+BxC6fXcCAwx0IKtFMxMcM2bLC2+u1VfkQhpDW/BcyXf2TTsf8 LkJvRTDTkqqRQwEuwJ+66x5oFCGtr4pC30pZ17XZ2kQxX+vYEqeGvdZzpzi4alPNpxqo f5TQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=l0HnQnNZwsWsfI4gRo2iE+eSfqSupzLQCthMtHdjT8A=; b=OoNJYDJaV01aWGuEAIkBFv9N7+rj1IePoGmsNpdKRcgSaJj8XI7tcsbBWmKDE/4v9q 6v1wxeGVPcrC14nXQx5I2sSqHkc92b4NmcV5tHz91toYCX8X7tYuvaRP0yJK5WoDfgh3 9G41mqLpAR92dBioJdrWdyT4gwiQQST7ta5Fuo3p93TODe0tFf6knIkadUSa1YJOo+DR cnho2Wkul16xaedngTIrBQf7aWwC4xl5e6ePWK6x00vp6alrtJHKVCq2gH31FJnyn37Q /nq+kzEPZdSU358+d1VG0Ya/JoG+tjeH1JB410/xo+FCSE6ogLU8pxDfCVytZlajs19M 47cw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h13-20020a056a00000d00b0064d5c5d72c5si2837204pfk.330.2023.05.30.21.24.01; Tue, 30 May 2023 21:24:13 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234238AbjEaEFj (ORCPT + 99 others); Wed, 31 May 2023 00:05:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54796 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234040AbjEaEFY (ORCPT ); Wed, 31 May 2023 00:05:24 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 6D436198; Tue, 30 May 2023 21:05:14 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 83F4515BF; Tue, 30 May 2023 21:05:59 -0700 (PDT) Received: from a077893.arm.com (unknown [10.163.73.163]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id B60A63F6C4; Tue, 30 May 2023 21:05:08 -0700 (PDT) From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, will@kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com Cc: Anshuman Khandual , Mark Brown , James Clark , Rob Herring , Marc Zyngier , Suzuki Poulose , Peter Zijlstra , Ingo Molnar , Arnaldo Carvalho de Melo , linux-perf-users@vger.kernel.org Subject: [PATCH V11 03/10] arm64/perf: Add branch stack support in struct arm_pmu Date: Wed, 31 May 2023 09:34:21 +0530 Message-Id: <20230531040428.501523-4-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230531040428.501523-1-anshuman.khandual@arm.com> References: <20230531040428.501523-1-anshuman.khandual@arm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1767382243680758804?= X-GMAIL-MSGID: =?utf-8?q?1767382243680758804?= This updates 'struct arm_pmu' for branch stack sampling support later. This adds a new 'features' element in the structure to track supported features, and another 'private' element to encapsulate implementation attributes on a given 'struct arm_pmu'. These updates here will help in tracking any branch stack sampling support, which is being added later. This also adds a helper arm_pmu_branch_stack_supported(). This also enables perf branch stack sampling event on all 'struct arm pmu', supporting the feature but after removing the current gate that blocks such events unconditionally in armpmu_event_init(). Instead a quick probe can be initiated via arm_pmu_branch_stack_supported() to ascertain the support. Cc: Catalin Marinas Cc: Will Deacon Cc: Mark Rutland Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Tested-by: James Clark Signed-off-by: Anshuman Khandual Acked-by: Mark Rutland --- drivers/perf/arm_pmu.c | 3 +-- include/linux/perf/arm_pmu.h | 12 +++++++++++- 2 files changed, 12 insertions(+), 3 deletions(-) diff --git a/drivers/perf/arm_pmu.c b/drivers/perf/arm_pmu.c index aada47e3b126..d4a4f2bd89a5 100644 --- a/drivers/perf/arm_pmu.c +++ b/drivers/perf/arm_pmu.c @@ -510,8 +510,7 @@ static int armpmu_event_init(struct perf_event *event) !cpumask_test_cpu(event->cpu, &armpmu->supported_cpus)) return -ENOENT; - /* does not support taken branch sampling */ - if (has_branch_stack(event)) + if (has_branch_stack(event) && !arm_pmu_branch_stack_supported(armpmu)) return -EOPNOTSUPP; return __hw_perf_event_init(event); diff --git a/include/linux/perf/arm_pmu.h b/include/linux/perf/arm_pmu.h index f7fbd162ca4c..0da745eaf426 100644 --- a/include/linux/perf/arm_pmu.h +++ b/include/linux/perf/arm_pmu.h @@ -102,7 +102,9 @@ struct arm_pmu { int (*map_event)(struct perf_event *event); void (*sched_task)(struct perf_event_pmu_context *pmu_ctx, bool sched_in); int num_events; - bool secure_access; /* 32-bit ARM only */ + unsigned int secure_access : 1, /* 32-bit ARM only */ + has_branch_stack: 1, /* 64-bit ARM only */ + reserved : 30; #define ARMV8_PMUV3_MAX_COMMON_EVENTS 0x40 DECLARE_BITMAP(pmceid_bitmap, ARMV8_PMUV3_MAX_COMMON_EVENTS); #define ARMV8_PMUV3_EXT_COMMON_EVENT_BASE 0x4000 @@ -118,8 +120,16 @@ struct arm_pmu { /* Only to be used by ACPI probing code */ unsigned long acpi_cpuid; + + /* Implementation specific attributes */ + void *private; }; +static inline bool arm_pmu_branch_stack_supported(struct arm_pmu *armpmu) +{ + return armpmu->has_branch_stack; +} + #define to_arm_pmu(p) (container_of(p, struct arm_pmu, pmu)) u64 armpmu_event_update(struct perf_event *event); From patchwork Wed May 31 04:04:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anshuman Khandual X-Patchwork-Id: 101220 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2630184vqr; Tue, 30 May 2023 21:26:16 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6TXr8U87kLxbNWAg7swRC7efk/TdI+LQM5gftsI3WqpFMxW2pha/6FVQzGh6AyOQPufx70 X-Received: by 2002:a17:903:2450:b0:1af:ac49:e048 with SMTP id l16-20020a170903245000b001afac49e048mr4630471pls.25.1685507176497; Tue, 30 May 2023 21:26:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685507176; cv=none; d=google.com; s=arc-20160816; b=TkVELS2x9PxhNx8usqwi0Nw5spVN4YWDMcyV30YR293YMMTocyzGbAHnLNrhrgsbFz J7Ejsf6exthnvATAwWKy7QcFN7csD9sh9R19gc6BYUCViKj4nW46UFhU6pGl/d2KhhWa 0hoNZqgnGCccgoZjp5Jq3iVFQY3FSEbODkeWoQbrveI8pInEr2aWJFtThzHMsaR4ImYw 00/hFyjye0s35jRPQ6avAwqzODgA226U2Fbw8DmerVj/fWUsonmGWcuDEK8Khvw+KRoB xAuQK08rS2BMpmc6UgnjwWwpOvjomHNPErOqlJiPA0p/Cf+xPc6HbejHx3gTUQi+Ad8r wXVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=Ywjsl8KNCiO/ijMrpusMohOs5baNdn6nwSqUjMhOk1Y=; b=D8YHJScgCcMvy16Ei+dh0WYHBLPN7HLJZoM3w/qOGgVwaWpE00G/1tjSu61utfBccg Vl3WCLbXVDDR9lv9d/S6ojsp+h1EkXG2jNYJNyS/qs8W3qUEnvUOdUrTIS4pG0So+n9V XRcFCpbGF6DkeLAIg647fYCVxjzdIOD8Hxz4fNtWxg57Xb3eRaE5Bule2qkHLO33YcYu Tkg+jkayo9EPYFBLzH7W0AYgfV9jmq1Bn7zYZN8oCvU2Hv/oGqhHjZnA7S5rLynXISfZ s7CwQ7vhvIe2CIeLDN1xPWo1TVi82ImmDEetpFu3e8z1r0vYdDaZvSxagTvH/1kAVyYe zsZQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 1-20020a170902ee4100b001b0395e10ebsi196275plo.377.2023.05.30.21.26.03; Tue, 30 May 2023 21:26:16 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234235AbjEaEF6 (ORCPT + 99 others); Wed, 31 May 2023 00:05:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55142 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234232AbjEaEFc (ORCPT ); Wed, 31 May 2023 00:05:32 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 81175E44; Tue, 30 May 2023 21:05:20 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6939015BF; Tue, 30 May 2023 21:06:05 -0700 (PDT) Received: from a077893.arm.com (unknown [10.163.73.163]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id DDE783F6C4; Tue, 30 May 2023 21:05:14 -0700 (PDT) From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, will@kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com Cc: Anshuman Khandual , Mark Brown , James Clark , Rob Herring , Marc Zyngier , Suzuki Poulose , Peter Zijlstra , Ingo Molnar , Arnaldo Carvalho de Melo , linux-perf-users@vger.kernel.org Subject: [PATCH V11 04/10] arm64/perf: Add branch stack support in struct pmu_hw_events Date: Wed, 31 May 2023 09:34:22 +0530 Message-Id: <20230531040428.501523-5-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230531040428.501523-1-anshuman.khandual@arm.com> References: <20230531040428.501523-1-anshuman.khandual@arm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1767382373210615024?= X-GMAIL-MSGID: =?utf-8?q?1767382373210615024?= This adds branch records buffer pointer in 'struct pmu_hw_events' which can be used to capture branch records during PMU interrupt. This percpu pointer here needs to be allocated first before usage. Cc: Catalin Marinas Cc: Will Deacon Cc: Mark Rutland Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Tested-by: James Clark Signed-off-by: Anshuman Khandual Acked-by: Mark Rutland --- include/linux/perf/arm_pmu.h | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/include/linux/perf/arm_pmu.h b/include/linux/perf/arm_pmu.h index 0da745eaf426..694b241e456c 100644 --- a/include/linux/perf/arm_pmu.h +++ b/include/linux/perf/arm_pmu.h @@ -44,6 +44,13 @@ static_assert((PERF_EVENT_FLAG_ARCH & ARMPMU_EVT_47BIT) == ARMPMU_EVT_47BIT); }, \ } +#define MAX_BRANCH_RECORDS 64 + +struct branch_records { + struct perf_branch_stack branch_stack; + struct perf_branch_entry branch_entries[MAX_BRANCH_RECORDS]; +}; + /* The events for a given PMU register set. */ struct pmu_hw_events { /* @@ -70,6 +77,8 @@ struct pmu_hw_events { struct arm_pmu *percpu_pmu; int irq; + + struct branch_records *branches; }; enum armpmu_attr_groups { From patchwork Wed May 31 04:04:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anshuman Khandual X-Patchwork-Id: 101217 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2629039vqr; Tue, 30 May 2023 21:22:02 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5tTM297kIKuU/FggkWxDmYv+l00nQIzGOaKLccRbcCgf9A+hP7PoDY6RUjlR6ATT7dg1YU X-Received: by 2002:a05:6a20:840f:b0:10c:29e5:941e with SMTP id c15-20020a056a20840f00b0010c29e5941emr5467540pzd.59.1685506922639; Tue, 30 May 2023 21:22:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685506922; cv=none; d=google.com; s=arc-20160816; b=hpHNAU6JSGbNHPizAIxD4naMOwUZmXevjhNnjM+kJBvl5GpLqn8O5NWzVdZpLqs85/ O+eaLy8ACoBskj1Sa1I4d2CHGNYwne29oZvx8MIrOokEAlCc7QJssNeOxmj21bczaysl 4I/Ba2LscsWQXlJgcsHvGmzSe9alu3f9v+BYuBzy1t31dQOe8bYSQODlA7P2aQwNyj6T 7gS2q4MPW7WfRQC3/LDBtfTjztmCJJdLwjZrgahNnIOR/LX5X3M0k/S7S5OLynqqMNFi gbif7/czjqic+NQyVAa0aWJfUNBvhjR0/UvrBS3MtlaWlc1wq6LF5R1OitRVybciOJn2 eT+w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=bYbF//AWc7jYJoDcbheuvtuQW1UqByX6Wz63zC4JZUI=; b=Zb1M/b08pkIK10l0w0w11PCPlOwfEZCo11hj4A6QCmSdlZttDcB8Z/gwYc4tru7A+f xcsy+pQ0HfKFJkgiPQABqTm+deBW+NWnriCWsP0NSn+a/XXSoDsQFgiwu5niQ54f5bl9 lZP8xhtUuD+Oy14HTHyXSaLl0WoQeMDKR+eI/XCiGJO3SSBHwah9nKAsn2Y+QObYfwOt j/M6wnyhtv4rH8wD9BhiBNyG79ut+vBULnlKCH2hNfrLTmUJXUbRtFVPtKMDJGsRnKLi rP8iUGR6Ex5UkmQoGgZuJSPs55JdZVCg9JpV3iD096TLWok27tgB9Z/t9HoVxDUmb2vP Xvdw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c66-20020a633545000000b005346c49e06esi238670pga.843.2023.05.30.21.21.50; Tue, 30 May 2023 21:22:02 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234291AbjEaEGJ (ORCPT + 99 others); Wed, 31 May 2023 00:06:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55434 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234078AbjEaEFl (ORCPT ); Wed, 31 May 2023 00:05:41 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 779C9133; Tue, 30 May 2023 21:05:26 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 946ED15DB; Tue, 30 May 2023 21:06:11 -0700 (PDT) Received: from a077893.arm.com (unknown [10.163.73.163]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id AB3D73F6C4; Tue, 30 May 2023 21:05:20 -0700 (PDT) From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, will@kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com Cc: Anshuman Khandual , Mark Brown , James Clark , Rob Herring , Marc Zyngier , Suzuki Poulose , Peter Zijlstra , Ingo Molnar , Arnaldo Carvalho de Melo , linux-perf-users@vger.kernel.org Subject: [PATCH V11 05/10] arm64/perf: Add branch stack support in ARMV8 PMU Date: Wed, 31 May 2023 09:34:23 +0530 Message-Id: <20230531040428.501523-6-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230531040428.501523-1-anshuman.khandual@arm.com> References: <20230531040428.501523-1-anshuman.khandual@arm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1767382107183573348?= X-GMAIL-MSGID: =?utf-8?q?1767382107183573348?= This enables support for branch stack sampling event in ARMV8 PMU, checking has_branch_stack() on the event inside 'struct arm_pmu' callbacks. Although these branch stack helpers armv8pmu_branch_XXXXX() are just dummy functions for now. While here, this also defines arm_pmu's sched_task() callback with armv8pmu_sched_task(), which resets the branch record buffer on a sched_in. Cc: Catalin Marinas Cc: Will Deacon Cc: Mark Rutland Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Tested-by: James Clark Signed-off-by: Anshuman Khandual --- arch/arm64/include/asm/perf_event.h | 33 +++++++++++++ drivers/perf/arm_pmuv3.c | 76 ++++++++++++++++++++--------- 2 files changed, 86 insertions(+), 23 deletions(-) diff --git a/arch/arm64/include/asm/perf_event.h b/arch/arm64/include/asm/perf_event.h index eb7071c9eb34..7548813783ba 100644 --- a/arch/arm64/include/asm/perf_event.h +++ b/arch/arm64/include/asm/perf_event.h @@ -24,4 +24,37 @@ extern unsigned long perf_misc_flags(struct pt_regs *regs); (regs)->pstate = PSR_MODE_EL1h; \ } +struct pmu_hw_events; +struct arm_pmu; +struct perf_event; + +#ifdef CONFIG_PERF_EVENTS +static inline bool has_branch_stack(struct perf_event *event); + +static inline void armv8pmu_branch_read(struct pmu_hw_events *cpuc, struct perf_event *event) +{ + WARN_ON_ONCE(!has_branch_stack(event)); +} + +static inline bool armv8pmu_branch_valid(struct perf_event *event) +{ + WARN_ON_ONCE(!has_branch_stack(event)); + return false; +} + +static inline void armv8pmu_branch_enable(struct perf_event *event) +{ + WARN_ON_ONCE(!has_branch_stack(event)); +} + +static inline void armv8pmu_branch_disable(struct perf_event *event) +{ + WARN_ON_ONCE(!has_branch_stack(event)); +} + +static inline void armv8pmu_branch_probe(struct arm_pmu *arm_pmu) { } +static inline void armv8pmu_branch_reset(void) { } +static inline int armv8pmu_private_alloc(struct arm_pmu *arm_pmu) { return 0; } +static inline void armv8pmu_private_free(struct arm_pmu *arm_pmu) { } +#endif #endif diff --git a/drivers/perf/arm_pmuv3.c b/drivers/perf/arm_pmuv3.c index c98e4039386d..86d803ff1ae3 100644 --- a/drivers/perf/arm_pmuv3.c +++ b/drivers/perf/arm_pmuv3.c @@ -705,38 +705,21 @@ static void armv8pmu_enable_event(struct perf_event *event) * Enable counter and interrupt, and set the counter to count * the event that we're interested in. */ - - /* - * Disable counter - */ armv8pmu_disable_event_counter(event); - - /* - * Set event. - */ armv8pmu_write_event_type(event); - - /* - * Enable interrupt for this counter - */ armv8pmu_enable_event_irq(event); - - /* - * Enable counter - */ armv8pmu_enable_event_counter(event); + + if (has_branch_stack(event)) + armv8pmu_branch_enable(event); } static void armv8pmu_disable_event(struct perf_event *event) { - /* - * Disable counter - */ - armv8pmu_disable_event_counter(event); + if (has_branch_stack(event)) + armv8pmu_branch_disable(event); - /* - * Disable interrupt for this counter - */ + armv8pmu_disable_event_counter(event); armv8pmu_disable_event_irq(event); } @@ -814,6 +797,11 @@ static irqreturn_t armv8pmu_handle_irq(struct arm_pmu *cpu_pmu) if (!armpmu_event_set_period(event)) continue; + if (has_branch_stack(event) && !WARN_ON(!cpuc->branches)) { + armv8pmu_branch_read(cpuc, event); + perf_sample_save_brstack(&data, event, &cpuc->branches->branch_stack); + } + /* * Perf event overflow will queue the processing of the event as * an irq_work which will be taken care of in the handling of @@ -912,6 +900,14 @@ static int armv8pmu_user_event_idx(struct perf_event *event) return event->hw.idx; } +static void armv8pmu_sched_task(struct perf_event_pmu_context *pmu_ctx, bool sched_in) +{ + struct arm_pmu *armpmu = to_arm_pmu(pmu_ctx->pmu); + + if (sched_in && arm_pmu_branch_stack_supported(armpmu)) + armv8pmu_branch_reset(); +} + /* * Add an event filter to a given event. */ @@ -982,6 +978,9 @@ static void armv8pmu_reset(void *info) pmcr |= ARMV8_PMU_PMCR_LP; armv8pmu_pmcr_write(pmcr); + + if (arm_pmu_branch_stack_supported(cpu_pmu)) + armv8pmu_branch_reset(); } static int __armv8_pmuv3_map_event_id(struct arm_pmu *armpmu, @@ -1019,6 +1018,9 @@ static int __armv8_pmuv3_map_event(struct perf_event *event, hw_event_id = __armv8_pmuv3_map_event_id(armpmu, event); + if (has_branch_stack(event) && !armv8pmu_branch_valid(event)) + return -EOPNOTSUPP; + /* * CHAIN events only work when paired with an adjacent counter, and it * never makes sense for a user to open one in isolation, as they'll be @@ -1135,6 +1137,21 @@ static void __armv8pmu_probe_pmu(void *info) cpu_pmu->reg_pmmir = read_pmmir(); else cpu_pmu->reg_pmmir = 0; + armv8pmu_branch_probe(cpu_pmu); +} + +static int branch_records_alloc(struct arm_pmu *armpmu) +{ + struct pmu_hw_events *events; + int cpu; + + for_each_possible_cpu(cpu) { + events = per_cpu_ptr(armpmu->hw_events, cpu); + events->branches = kzalloc(sizeof(struct branch_records), GFP_KERNEL); + if (!events->branches) + return -ENOMEM; + } + return 0; } static int armv8pmu_probe_pmu(struct arm_pmu *cpu_pmu) @@ -1145,12 +1162,24 @@ static int armv8pmu_probe_pmu(struct arm_pmu *cpu_pmu) }; int ret; + ret = armv8pmu_private_alloc(cpu_pmu); + if (ret) + return ret; + ret = smp_call_function_any(&cpu_pmu->supported_cpus, __armv8pmu_probe_pmu, &probe, 1); if (ret) return ret; + if (arm_pmu_branch_stack_supported(cpu_pmu)) { + ret = branch_records_alloc(cpu_pmu); + if (ret) + return ret; + } else { + armv8pmu_private_free(cpu_pmu); + } + return probe.present ? 0 : -ENODEV; } @@ -1214,6 +1243,7 @@ static int armv8_pmu_init(struct arm_pmu *cpu_pmu, char *name, cpu_pmu->set_event_filter = armv8pmu_set_event_filter; cpu_pmu->pmu.event_idx = armv8pmu_user_event_idx; + cpu_pmu->sched_task = armv8pmu_sched_task; cpu_pmu->name = name; cpu_pmu->map_event = map_event; From patchwork Wed May 31 04:04:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anshuman Khandual X-Patchwork-Id: 101226 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2630724vqr; Tue, 30 May 2023 21:28:14 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4vVqSwasp43oDUgqMd9iqmziKJ8bgBPqT8YEK0rEpUukNbnUJSVnEOETAf5M5DYWrrIqxB X-Received: by 2002:a05:622a:355:b0:3f6:c5cd:dc5e with SMTP id r21-20020a05622a035500b003f6c5cddc5emr5075158qtw.23.1685507293832; Tue, 30 May 2023 21:28:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685507293; cv=none; d=google.com; s=arc-20160816; b=x/F+qlc/dOp84Grq15BRiCpwxZRum30I7qBEi2VBTI47UE8D5rVWHK2HGK+Hjg7GRm Vdf49HQfKkXrwyclc4AlgiUFNfmQrpMp6Q7tHpm2pu3Xkv9vuP2ukT3+xcM24mPNcpOJ 9jlaYTT9fk8ogjDXlA1oy3l6VCANxFTHx6ARectgVnzv2xwJmFIA1vq7t9VPuNeFyysu 5SaKP0A08N/fjaIlP90HEpKuoY/fiQBFoMthidi0T+tytAiXKEuc5XxpxKaoj/dPCLM8 pFFqIXQDUhVFrqwED4eaKx2ayEctGxZVM+Qp1OiYEdBKdnysOYgfdoN7SHzFHRCpjFRe Nxdw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=kxDxdAhSKX4TCgg57QVqKvxeapLgMqbaOpprItkp0Ws=; b=bwrC5YeN1EYFnKnrKPprmV+JrBEeparAvxzTlqFhmh7dGkRshBAN+EZxGZCaVU3cRp wpeY8T/IPT1x8B3fMC33hpNYKUCDXrwZotk7f/IouFTKfGJtJfgnxiH7F70zvapl0HtZ e92j281Qq+1aE2zOIwlDs2rAXJBvdfogsJiJDmEyfV/yeDlcx4yN5u9Lkl1LPc3jtptb EmRJrpn2waEkTNf2G/83+Y3dHQ0Nx4sbn5aNhaY3x6a3VMWZ16zX86x0/o7V2CwnekdR NX/3gj1xa4KKnKZaIsZYIrb99jqU8j8mNFkxAiQDNihE1F9bua1o7dCBn7hTGN5tQoDd ipGw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i10-20020a17090a64ca00b002472cd8ded6si297032pjm.103.2023.05.30.21.28.01; Tue, 30 May 2023 21:28:13 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234208AbjEaEGT (ORCPT + 99 others); Wed, 31 May 2023 00:06:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55606 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234047AbjEaEFt (ORCPT ); Wed, 31 May 2023 00:05:49 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id B67DC188; Tue, 30 May 2023 21:05:32 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B62A9165C; Tue, 30 May 2023 21:06:17 -0700 (PDT) Received: from a077893.arm.com (unknown [10.163.73.163]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id EF1763F6C4; Tue, 30 May 2023 21:05:26 -0700 (PDT) From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, will@kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com Cc: Anshuman Khandual , Mark Brown , James Clark , Rob Herring , Marc Zyngier , Suzuki Poulose , Peter Zijlstra , Ingo Molnar , Arnaldo Carvalho de Melo , linux-perf-users@vger.kernel.org Subject: [PATCH V11 06/10] arm64/perf: Enable branch stack events via FEAT_BRBE Date: Wed, 31 May 2023 09:34:24 +0530 Message-Id: <20230531040428.501523-7-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230531040428.501523-1-anshuman.khandual@arm.com> References: <20230531040428.501523-1-anshuman.khandual@arm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1767382496076548938?= X-GMAIL-MSGID: =?utf-8?q?1767382496076548938?= This enables branch stack sampling events in ARMV8 PMU, via an architecture feature FEAT_BRBE aka branch record buffer extension. This defines required branch helper functions pmuv8pmu_branch_XXXXX() and the implementation here is wrapped with a new config option CONFIG_ARM64_BRBE. Cc: Catalin Marinas Cc: Will Deacon Cc: Mark Rutland Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Tested-by: James Clark Signed-off-by: Anshuman Khandual --- arch/arm64/include/asm/perf_event.h | 11 + drivers/perf/Kconfig | 11 + drivers/perf/Makefile | 1 + drivers/perf/arm_brbe.c | 571 ++++++++++++++++++++++++++++ drivers/perf/arm_brbe.h | 257 +++++++++++++ drivers/perf/arm_pmuv3.c | 21 +- 6 files changed, 869 insertions(+), 3 deletions(-) create mode 100644 drivers/perf/arm_brbe.c create mode 100644 drivers/perf/arm_brbe.h diff --git a/arch/arm64/include/asm/perf_event.h b/arch/arm64/include/asm/perf_event.h index 7548813783ba..f071d629c0cf 100644 --- a/arch/arm64/include/asm/perf_event.h +++ b/arch/arm64/include/asm/perf_event.h @@ -31,6 +31,16 @@ struct perf_event; #ifdef CONFIG_PERF_EVENTS static inline bool has_branch_stack(struct perf_event *event); +#ifdef CONFIG_ARM64_BRBE +void armv8pmu_branch_read(struct pmu_hw_events *cpuc, struct perf_event *event); +bool armv8pmu_branch_valid(struct perf_event *event); +void armv8pmu_branch_enable(struct perf_event *event); +void armv8pmu_branch_disable(struct perf_event *event); +void armv8pmu_branch_probe(struct arm_pmu *arm_pmu); +void armv8pmu_branch_reset(void); +int armv8pmu_private_alloc(struct arm_pmu *arm_pmu); +void armv8pmu_private_free(struct arm_pmu *arm_pmu); +#else static inline void armv8pmu_branch_read(struct pmu_hw_events *cpuc, struct perf_event *event) { WARN_ON_ONCE(!has_branch_stack(event)); @@ -58,3 +68,4 @@ static inline int armv8pmu_private_alloc(struct arm_pmu *arm_pmu) { return 0; } static inline void armv8pmu_private_free(struct arm_pmu *arm_pmu) { } #endif #endif +#endif diff --git a/drivers/perf/Kconfig b/drivers/perf/Kconfig index 711f82400086..7d07aa79e5b0 100644 --- a/drivers/perf/Kconfig +++ b/drivers/perf/Kconfig @@ -172,6 +172,17 @@ config ARM_SPE_PMU Extension, which provides periodic sampling of operations in the CPU pipeline and reports this via the perf AUX interface. +config ARM64_BRBE + bool "Enable support for Branch Record Buffer Extension (BRBE)" + depends on PERF_EVENTS && ARM64 && ARM_PMU + default y + help + Enable perf support for Branch Record Buffer Extension (BRBE) which + records all branches taken in an execution path. This supports some + branch types and privilege based filtering. It captured additional + relevant information such as cycle count, misprediction and branch + type, branch privilege level etc. + config ARM_DMC620_PMU tristate "Enable PMU support for the ARM DMC-620 memory controller" depends on (ARM64 && ACPI) || COMPILE_TEST diff --git a/drivers/perf/Makefile b/drivers/perf/Makefile index dabc859540ce..29d256f2deaa 100644 --- a/drivers/perf/Makefile +++ b/drivers/perf/Makefile @@ -17,6 +17,7 @@ obj-$(CONFIG_RISCV_PMU_SBI) += riscv_pmu_sbi.o obj-$(CONFIG_THUNDERX2_PMU) += thunderx2_pmu.o obj-$(CONFIG_XGENE_PMU) += xgene_pmu.o obj-$(CONFIG_ARM_SPE_PMU) += arm_spe_pmu.o +obj-$(CONFIG_ARM64_BRBE) += arm_brbe.o obj-$(CONFIG_ARM_DMC620_PMU) += arm_dmc620_pmu.o obj-$(CONFIG_MARVELL_CN10K_TAD_PMU) += marvell_cn10k_tad_pmu.o obj-$(CONFIG_MARVELL_CN10K_DDR_PMU) += marvell_cn10k_ddr_pmu.o diff --git a/drivers/perf/arm_brbe.c b/drivers/perf/arm_brbe.c new file mode 100644 index 000000000000..34547ad750ad --- /dev/null +++ b/drivers/perf/arm_brbe.c @@ -0,0 +1,571 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Branch Record Buffer Extension Driver. + * + * Copyright (C) 2022 ARM Limited + * + * Author: Anshuman Khandual + */ +#include "arm_brbe.h" + +static bool valid_brbe_nr(int brbe_nr) +{ + return brbe_nr == BRBIDR0_EL1_NUMREC_8 || + brbe_nr == BRBIDR0_EL1_NUMREC_16 || + brbe_nr == BRBIDR0_EL1_NUMREC_32 || + brbe_nr == BRBIDR0_EL1_NUMREC_64; +} + +static bool valid_brbe_cc(int brbe_cc) +{ + return brbe_cc == BRBIDR0_EL1_CC_20_BIT; +} + +static bool valid_brbe_format(int brbe_format) +{ + return brbe_format == BRBIDR0_EL1_FORMAT_0; +} + +static bool valid_brbe_version(int brbe_version) +{ + return brbe_version == ID_AA64DFR0_EL1_BRBE_IMP || + brbe_version == ID_AA64DFR0_EL1_BRBE_BRBE_V1P1; +} + +static void select_brbe_bank(int bank) +{ + u64 brbfcr; + + WARN_ON(bank > BRBE_BANK_IDX_1); + brbfcr = read_sysreg_s(SYS_BRBFCR_EL1); + brbfcr &= ~BRBFCR_EL1_BANK_MASK; + brbfcr |= SYS_FIELD_PREP(BRBFCR_EL1, BANK, bank); + write_sysreg_s(brbfcr, SYS_BRBFCR_EL1); + isb(); +} + +/* + * Generic perf branch filters supported on BRBE + * + * New branch filters need to be evaluated whether they could be supported on + * BRBE. This ensures that such branch filters would not just be accepted, to + * fail silently. PERF_SAMPLE_BRANCH_HV is a special case that is selectively + * supported only on platforms where kernel is in hyp mode. + */ +#define BRBE_EXCLUDE_BRANCH_FILTERS (PERF_SAMPLE_BRANCH_ABORT_TX | \ + PERF_SAMPLE_BRANCH_IN_TX | \ + PERF_SAMPLE_BRANCH_NO_TX | \ + PERF_SAMPLE_BRANCH_CALL_STACK) + +#define BRBE_ALLOWED_BRANCH_FILTERS (PERF_SAMPLE_BRANCH_USER | \ + PERF_SAMPLE_BRANCH_KERNEL | \ + PERF_SAMPLE_BRANCH_HV | \ + PERF_SAMPLE_BRANCH_ANY | \ + PERF_SAMPLE_BRANCH_ANY_CALL | \ + PERF_SAMPLE_BRANCH_ANY_RETURN | \ + PERF_SAMPLE_BRANCH_IND_CALL | \ + PERF_SAMPLE_BRANCH_COND | \ + PERF_SAMPLE_BRANCH_IND_JUMP | \ + PERF_SAMPLE_BRANCH_CALL | \ + PERF_SAMPLE_BRANCH_NO_FLAGS | \ + PERF_SAMPLE_BRANCH_NO_CYCLES | \ + PERF_SAMPLE_BRANCH_TYPE_SAVE | \ + PERF_SAMPLE_BRANCH_HW_INDEX | \ + PERF_SAMPLE_BRANCH_PRIV_SAVE) + +#define BRBE_PERF_BRANCH_FILTERS (BRBE_ALLOWED_BRANCH_FILTERS | \ + BRBE_EXCLUDE_BRANCH_FILTERS) + +bool armv8pmu_branch_valid(struct perf_event *event) +{ + u64 branch_type = event->attr.branch_sample_type; + + /* + * Ensure both perf branch filter allowed and exclude + * masks are always in sync with the generic perf ABI. + */ + BUILD_BUG_ON(BRBE_PERF_BRANCH_FILTERS != (PERF_SAMPLE_BRANCH_MAX - 1)); + + if (branch_type & ~BRBE_ALLOWED_BRANCH_FILTERS) { + pr_debug_once("requested branch filter not supported 0x%llx\n", branch_type); + return false; + } + + /* + * If the event does not have at least one of the privilege + * branch filters as in PERF_SAMPLE_BRANCH_PLM_ALL, the core + * perf will adjust its value based on perf event's existing + * privilege level via attr.exclude_[user|kernel|hv]. + * + * As event->attr.branch_sample_type might have been changed + * when the event reaches here, it is not possible to figure + * out whether the event originally had HV privilege request + * or got added via the core perf. Just report this situation + * once and continue ignoring if there are other instances. + */ + if ((branch_type & PERF_SAMPLE_BRANCH_HV) && !is_kernel_in_hyp_mode()) + pr_debug_once("hypervisor privilege filter not supported 0x%llx\n", branch_type); + + return true; +} + +int armv8pmu_private_alloc(struct arm_pmu *arm_pmu) +{ + struct brbe_hw_attr *brbe_attr = kzalloc(sizeof(struct brbe_hw_attr), GFP_KERNEL); + + if (!brbe_attr) + return -ENOMEM; + + arm_pmu->private = brbe_attr; + return 0; +} + +void armv8pmu_private_free(struct arm_pmu *arm_pmu) +{ + kfree(arm_pmu->private); +} + +static int brbe_attributes_probe(struct arm_pmu *armpmu, u32 brbe) +{ + struct brbe_hw_attr *brbe_attr = (struct brbe_hw_attr *)armpmu->private; + u64 brbidr = read_sysreg_s(SYS_BRBIDR0_EL1); + + brbe_attr->brbe_version = brbe; + brbe_attr->brbe_format = brbe_get_format(brbidr); + brbe_attr->brbe_cc = brbe_get_cc_bits(brbidr); + brbe_attr->brbe_nr = brbe_get_numrec(brbidr); + + if (!valid_brbe_version(brbe_attr->brbe_version) || + !valid_brbe_format(brbe_attr->brbe_format) || + !valid_brbe_cc(brbe_attr->brbe_cc) || + !valid_brbe_nr(brbe_attr->brbe_nr)) + return -EOPNOTSUPP; + + return 0; +} + +void armv8pmu_branch_probe(struct arm_pmu *armpmu) +{ + u64 aa64dfr0 = read_sysreg_s(SYS_ID_AA64DFR0_EL1); + u32 brbe; + + brbe = cpuid_feature_extract_unsigned_field(aa64dfr0, ID_AA64DFR0_EL1_BRBE_SHIFT); + if (!brbe) + return; + + if (brbe_attributes_probe(armpmu, brbe)) + return; + + armpmu->has_branch_stack = 1; +} + +static u64 branch_type_to_brbfcr(int branch_type) +{ + u64 brbfcr = 0; + + if (branch_type & PERF_SAMPLE_BRANCH_ANY) { + brbfcr |= BRBFCR_EL1_BRANCH_FILTERS; + return brbfcr; + } + + if (branch_type & PERF_SAMPLE_BRANCH_ANY_CALL) { + brbfcr |= BRBFCR_EL1_INDCALL; + brbfcr |= BRBFCR_EL1_DIRCALL; + } + + if (branch_type & PERF_SAMPLE_BRANCH_ANY_RETURN) + brbfcr |= BRBFCR_EL1_RTN; + + if (branch_type & PERF_SAMPLE_BRANCH_IND_CALL) + brbfcr |= BRBFCR_EL1_INDCALL; + + if (branch_type & PERF_SAMPLE_BRANCH_COND) + brbfcr |= BRBFCR_EL1_CONDDIR; + + if (branch_type & PERF_SAMPLE_BRANCH_IND_JUMP) + brbfcr |= BRBFCR_EL1_INDIRECT; + + if (branch_type & PERF_SAMPLE_BRANCH_CALL) + brbfcr |= BRBFCR_EL1_DIRCALL; + + return brbfcr; +} + +static u64 branch_type_to_brbcr(int branch_type) +{ + u64 brbcr = BRBCR_EL1_DEFAULT_TS; + + /* + * BRBE need not be paused on PMU interrupt while tracing only + * the user space, bcause it will automatically be inside the + * prohibited region. But even after PMU overflow occurs, the + * interrupt could still take much more cycles, before it can + * be taken and by that time BRBE will have been overwritten. + * Let's enable pause on PMU interrupt mechanism even for user + * only traces. + */ + brbcr |= BRBCR_EL1_FZP; + + /* + * When running in the hyp mode, writing into BRBCR_EL1 + * actually writes into BRBCR_EL2 instead. Field E2BRE + * is also at the same position as E1BRE. + */ + if (branch_type & PERF_SAMPLE_BRANCH_USER) + brbcr |= BRBCR_EL1_E0BRE; + + if (branch_type & PERF_SAMPLE_BRANCH_KERNEL) + brbcr |= BRBCR_EL1_E1BRE; + + if (branch_type & PERF_SAMPLE_BRANCH_HV) { + if (is_kernel_in_hyp_mode()) + brbcr |= BRBCR_EL1_E1BRE; + } + + if (!(branch_type & PERF_SAMPLE_BRANCH_NO_CYCLES)) + brbcr |= BRBCR_EL1_CC; + + if (!(branch_type & PERF_SAMPLE_BRANCH_NO_FLAGS)) + brbcr |= BRBCR_EL1_MPRED; + + /* + * The exception and exception return branches could be + * captured, irrespective of the perf event's privilege. + * If the perf event does not have enough privilege for + * a given exception level, then addresses which falls + * under that exception level will be reported as zero + * for the captured branch record, creating source only + * or target only records. + */ + if (branch_type & PERF_SAMPLE_BRANCH_ANY) { + brbcr |= BRBCR_EL1_EXCEPTION; + brbcr |= BRBCR_EL1_ERTN; + } + + if (branch_type & PERF_SAMPLE_BRANCH_ANY_CALL) + brbcr |= BRBCR_EL1_EXCEPTION; + + if (branch_type & PERF_SAMPLE_BRANCH_ANY_RETURN) + brbcr |= BRBCR_EL1_ERTN; + + return brbcr & BRBCR_EL1_DEFAULT_CONFIG; +} + +void armv8pmu_branch_enable(struct perf_event *event) +{ + u64 branch_type = event->attr.branch_sample_type; + u64 brbfcr, brbcr; + + brbfcr = read_sysreg_s(SYS_BRBFCR_EL1); + brbfcr &= ~BRBFCR_EL1_DEFAULT_CONFIG; + brbfcr |= branch_type_to_brbfcr(branch_type); + write_sysreg_s(brbfcr, SYS_BRBFCR_EL1); + isb(); + + brbcr = read_sysreg_s(SYS_BRBCR_EL1); + brbcr &= ~BRBCR_EL1_DEFAULT_CONFIG; + brbcr |= branch_type_to_brbcr(branch_type); + write_sysreg_s(brbcr, SYS_BRBCR_EL1); + isb(); + armv8pmu_branch_reset(); +} + +void armv8pmu_branch_disable(struct perf_event *event) +{ + u64 brbfcr = read_sysreg_s(SYS_BRBFCR_EL1); + u64 brbcr = read_sysreg_s(SYS_BRBCR_EL1); + + brbcr &= ~(BRBCR_EL1_E0BRE | BRBCR_EL1_E1BRE); + brbfcr |= BRBFCR_EL1_PAUSED; + write_sysreg_s(brbcr, SYS_BRBCR_EL1); + write_sysreg_s(brbfcr, SYS_BRBFCR_EL1); + isb(); +} + +static void brbe_set_perf_entry_type(struct perf_branch_entry *entry, u64 brbinf) +{ + int brbe_type = brbe_get_type(brbinf); + + switch (brbe_type) { + case BRBINFx_EL1_TYPE_UNCOND_DIR: + entry->type = PERF_BR_UNCOND; + break; + case BRBINFx_EL1_TYPE_INDIR: + entry->type = PERF_BR_IND; + break; + case BRBINFx_EL1_TYPE_DIR_LINK: + entry->type = PERF_BR_CALL; + break; + case BRBINFx_EL1_TYPE_INDIR_LINK: + entry->type = PERF_BR_IND_CALL; + break; + case BRBINFx_EL1_TYPE_RET_SUB: + entry->type = PERF_BR_RET; + break; + case BRBINFx_EL1_TYPE_COND_DIR: + entry->type = PERF_BR_COND; + break; + case BRBINFx_EL1_TYPE_CALL: + entry->type = PERF_BR_CALL; + break; + case BRBINFx_EL1_TYPE_TRAP: + entry->type = PERF_BR_SYSCALL; + break; + case BRBINFx_EL1_TYPE_RET_EXCPT: + entry->type = PERF_BR_ERET; + break; + case BRBINFx_EL1_TYPE_IRQ: + entry->type = PERF_BR_IRQ; + break; + case BRBINFx_EL1_TYPE_DEBUG_HALT: + entry->type = PERF_BR_EXTEND_ABI; + entry->new_type = PERF_BR_ARM64_DEBUG_HALT; + break; + case BRBINFx_EL1_TYPE_SERROR: + entry->type = PERF_BR_SERROR; + break; + case BRBINFx_EL1_TYPE_INST_DEBUG: + entry->type = PERF_BR_EXTEND_ABI; + entry->new_type = PERF_BR_ARM64_DEBUG_INST; + break; + case BRBINFx_EL1_TYPE_DATA_DEBUG: + entry->type = PERF_BR_EXTEND_ABI; + entry->new_type = PERF_BR_ARM64_DEBUG_DATA; + break; + case BRBINFx_EL1_TYPE_ALGN_FAULT: + entry->type = PERF_BR_EXTEND_ABI; + entry->new_type = PERF_BR_NEW_FAULT_ALGN; + break; + case BRBINFx_EL1_TYPE_INST_FAULT: + entry->type = PERF_BR_EXTEND_ABI; + entry->new_type = PERF_BR_NEW_FAULT_INST; + break; + case BRBINFx_EL1_TYPE_DATA_FAULT: + entry->type = PERF_BR_EXTEND_ABI; + entry->new_type = PERF_BR_NEW_FAULT_DATA; + break; + case BRBINFx_EL1_TYPE_FIQ: + entry->type = PERF_BR_EXTEND_ABI; + entry->new_type = PERF_BR_ARM64_FIQ; + break; + case BRBINFx_EL1_TYPE_DEBUG_EXIT: + entry->type = PERF_BR_EXTEND_ABI; + entry->new_type = PERF_BR_ARM64_DEBUG_EXIT; + break; + default: + pr_warn_once("%d - unknown branch type captured\n", brbe_type); + entry->type = PERF_BR_UNKNOWN; + break; + } +} + +static int brbe_get_perf_priv(u64 brbinf) +{ + int brbe_el = brbe_get_el(brbinf); + + switch (brbe_el) { + case BRBINFx_EL1_EL_EL0: + return PERF_BR_PRIV_USER; + case BRBINFx_EL1_EL_EL1: + return PERF_BR_PRIV_KERNEL; + case BRBINFx_EL1_EL_EL2: + if (is_kernel_in_hyp_mode()) + return PERF_BR_PRIV_KERNEL; + return PERF_BR_PRIV_HV; + default: + pr_warn_once("%d - unknown branch privilege captured\n", brbe_el); + return PERF_BR_PRIV_UNKNOWN; + } +} + +static void capture_brbe_flags(struct perf_branch_entry *entry, struct perf_event *event, + u64 brbinf) +{ + if (branch_sample_type(event)) + brbe_set_perf_entry_type(entry, brbinf); + + if (!branch_sample_no_cycles(event)) + entry->cycles = brbe_get_cycles(brbinf); + + if (!branch_sample_no_flags(event)) { + /* + * BRBINFx_EL1.LASTFAILED indicates that a TME transaction failed (or + * was cancelled) prior to this record, and some number of records + * prior to this one, may have been generated during an attempt to + * execute the transaction. + * + * We will remove such entries later in process_branch_aborts(). + */ + entry->abort = brbe_get_lastfailed(brbinf); + + /* + * All these information (i.e transaction state and mispredicts) + * are available for source only and complete branch records. + */ + if (brbe_record_is_complete(brbinf) || + brbe_record_is_source_only(brbinf)) { + entry->mispred = brbe_get_mispredict(brbinf); + entry->predicted = !entry->mispred; + entry->in_tx = brbe_get_in_tx(brbinf); + } + } + + if (branch_sample_priv(event)) { + /* + * All these information (i.e branch privilege level) are + * available for target only and complete branch records. + */ + if (brbe_record_is_complete(brbinf) || + brbe_record_is_target_only(brbinf)) + entry->priv = brbe_get_perf_priv(brbinf); + } +} + +/* + * A branch record with BRBINFx_EL1.LASTFAILED set, implies that all + * preceding consecutive branch records, that were in a transaction + * (i.e their BRBINFx_EL1.TX set) have been aborted. + * + * Similarly BRBFCR_EL1.LASTFAILED set, indicate that all preceding + * consecutive branch records up to the last record, which were in a + * transaction (i.e their BRBINFx_EL1.TX set) have been aborted. + * + * --------------------------------- ------------------- + * | 00 | BRBSRC | BRBTGT | BRBINF | | TX = 1 | LF = 0 | [TX success] + * --------------------------------- ------------------- + * | 01 | BRBSRC | BRBTGT | BRBINF | | TX = 1 | LF = 0 | [TX success] + * --------------------------------- ------------------- + * | 02 | BRBSRC | BRBTGT | BRBINF | | TX = 0 | LF = 0 | + * --------------------------------- ------------------- + * | 03 | BRBSRC | BRBTGT | BRBINF | | TX = 1 | LF = 0 | [TX failed] + * --------------------------------- ------------------- + * | 04 | BRBSRC | BRBTGT | BRBINF | | TX = 1 | LF = 0 | [TX failed] + * --------------------------------- ------------------- + * | 05 | BRBSRC | BRBTGT | BRBINF | | TX = 0 | LF = 1 | + * --------------------------------- ------------------- + * | .. | BRBSRC | BRBTGT | BRBINF | | TX = 0 | LF = 0 | + * --------------------------------- ------------------- + * | 61 | BRBSRC | BRBTGT | BRBINF | | TX = 1 | LF = 0 | [TX failed] + * --------------------------------- ------------------- + * | 62 | BRBSRC | BRBTGT | BRBINF | | TX = 1 | LF = 0 | [TX failed] + * --------------------------------- ------------------- + * | 63 | BRBSRC | BRBTGT | BRBINF | | TX = 1 | LF = 0 | [TX failed] + * --------------------------------- ------------------- + * + * BRBFCR_EL1.LASTFAILED == 1 + * + * BRBFCR_EL1.LASTFAILED fails all those consecutive, in transaction + * branches records near the end of the BRBE buffer. + * + * Architecture does not guarantee a non transaction (TX = 0) branch + * record between two different transactions. So it is possible that + * a subsequent lastfailed record (TX = 0, LF = 1) might erroneously + * mark more than required transactions as aborted. + */ +static void process_branch_aborts(struct pmu_hw_events *cpuc) +{ + struct brbe_hw_attr *brbe_attr = (struct brbe_hw_attr *)cpuc->percpu_pmu->private; + u64 brbfcr = read_sysreg_s(SYS_BRBFCR_EL1); + bool lastfailed = !!(brbfcr & BRBFCR_EL1_LASTFAILED); + int idx = brbe_attr->brbe_nr - 1; + struct perf_branch_entry *entry; + + do { + entry = &cpuc->branches->branch_entries[idx]; + if (entry->in_tx) { + entry->abort = lastfailed; + } else { + lastfailed = entry->abort; + entry->abort = false; + } + } while (idx--, idx >= 0); +} + +void armv8pmu_branch_reset(void) +{ + asm volatile(BRB_IALL); + isb(); +} + +static bool capture_branch_entry(struct pmu_hw_events *cpuc, + struct perf_event *event, int idx) +{ + struct perf_branch_entry *entry = &cpuc->branches->branch_entries[idx]; + u64 brbinf = get_brbinf_reg(idx); + + /* + * There are no valid entries anymore on the buffer. + * Abort the branch record processing to save some + * cycles and also reduce the capture/process load + * for the user space as well. + */ + if (brbe_invalid(brbinf)) + return false; + + perf_clear_branch_entry_bitfields(entry); + if (brbe_record_is_complete(brbinf)) { + entry->from = get_brbsrc_reg(idx); + entry->to = get_brbtgt_reg(idx); + } else if (brbe_record_is_source_only(brbinf)) { + entry->from = get_brbsrc_reg(idx); + entry->to = 0; + } else if (brbe_record_is_target_only(brbinf)) { + entry->from = 0; + entry->to = get_brbtgt_reg(idx); + } + capture_brbe_flags(entry, event, brbinf); + return true; +} + +void armv8pmu_branch_read(struct pmu_hw_events *cpuc, struct perf_event *event) +{ + struct brbe_hw_attr *brbe_attr = (struct brbe_hw_attr *)cpuc->percpu_pmu->private; + u64 brbfcr, brbcr; + int idx, loop1_idx1, loop1_idx2, loop2_idx1, loop2_idx2, count; + + brbcr = read_sysreg_s(SYS_BRBCR_EL1); + brbfcr = read_sysreg_s(SYS_BRBFCR_EL1); + + /* Ensure pause on PMU interrupt is enabled */ + WARN_ON_ONCE(!(brbcr & BRBCR_EL1_FZP)); + + /* Pause the buffer */ + write_sysreg_s(brbfcr | BRBFCR_EL1_PAUSED, SYS_BRBFCR_EL1); + isb(); + + /* Determine the indices for each loop */ + loop1_idx1 = BRBE_BANK0_IDX_MIN; + if (brbe_attr->brbe_nr <= BRBE_BANK_MAX_ENTRIES) { + loop1_idx2 = brbe_attr->brbe_nr - 1; + loop2_idx1 = BRBE_BANK1_IDX_MIN; + loop2_idx2 = BRBE_BANK0_IDX_MAX; + } else { + loop1_idx2 = BRBE_BANK0_IDX_MAX; + loop2_idx1 = BRBE_BANK1_IDX_MIN; + loop2_idx2 = brbe_attr->brbe_nr - 1; + } + + /* Loop through bank 0 */ + select_brbe_bank(BRBE_BANK_IDX_0); + for (idx = 0, count = loop1_idx1; count <= loop1_idx2; idx++, count++) { + if (!capture_branch_entry(cpuc, event, idx)) + goto skip_bank_1; + } + + /* Loop through bank 1 */ + select_brbe_bank(BRBE_BANK_IDX_1); + for (count = loop2_idx1; count <= loop2_idx2; idx++, count++) { + if (!capture_branch_entry(cpuc, event, idx)) + break; + } + +skip_bank_1: + cpuc->branches->branch_stack.nr = idx; + cpuc->branches->branch_stack.hw_idx = -1ULL; + process_branch_aborts(cpuc); + + /* Unpause the buffer */ + write_sysreg_s(brbfcr & ~BRBFCR_EL1_PAUSED, SYS_BRBFCR_EL1); + isb(); + armv8pmu_branch_reset(); +} diff --git a/drivers/perf/arm_brbe.h b/drivers/perf/arm_brbe.h new file mode 100644 index 000000000000..a47480eec070 --- /dev/null +++ b/drivers/perf/arm_brbe.h @@ -0,0 +1,257 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Branch Record Buffer Extension Helpers. + * + * Copyright (C) 2022 ARM Limited + * + * Author: Anshuman Khandual + */ +#define pr_fmt(fmt) "brbe: " fmt + +#include + +#define BRBFCR_EL1_BRANCH_FILTERS (BRBFCR_EL1_DIRECT | \ + BRBFCR_EL1_INDIRECT | \ + BRBFCR_EL1_RTN | \ + BRBFCR_EL1_INDCALL | \ + BRBFCR_EL1_DIRCALL | \ + BRBFCR_EL1_CONDDIR) + +#define BRBFCR_EL1_DEFAULT_CONFIG (BRBFCR_EL1_BANK_MASK | \ + BRBFCR_EL1_PAUSED | \ + BRBFCR_EL1_EnI | \ + BRBFCR_EL1_BRANCH_FILTERS) + +/* + * BRBTS_EL1 is currently not used for branch stack implementation + * purpose but BRBCR_EL1.TS needs to have a valid value from all + * available options. BRBCR_EL1_TS_VIRTUAL is selected for this. + */ +#define BRBCR_EL1_DEFAULT_TS FIELD_PREP(BRBCR_EL1_TS_MASK, BRBCR_EL1_TS_VIRTUAL) + +#define BRBCR_EL1_DEFAULT_CONFIG (BRBCR_EL1_EXCEPTION | \ + BRBCR_EL1_ERTN | \ + BRBCR_EL1_CC | \ + BRBCR_EL1_MPRED | \ + BRBCR_EL1_E1BRE | \ + BRBCR_EL1_E0BRE | \ + BRBCR_EL1_FZP | \ + BRBCR_EL1_DEFAULT_TS) +/* + * BRBE Instructions + * + * BRB_IALL : Invalidate the entire buffer + * BRB_INJ : Inject latest branch record derived from [BRBSRCINJ, BRBTGTINJ, BRBINFINJ] + */ +#define BRB_IALL __emit_inst(0xD5000000 | sys_insn(1, 1, 7, 2, 4) | (0x1f)) +#define BRB_INJ __emit_inst(0xD5000000 | sys_insn(1, 1, 7, 2, 5) | (0x1f)) + +/* + * BRBE Buffer Organization + * + * BRBE buffer is arranged as multiple banks of 32 branch record + * entries each. An individual branch record in a given bank could + * be accessed, after selecting the bank in BRBFCR_EL1.BANK and + * accessing the registers i.e [BRBSRC, BRBTGT, BRBINF] set with + * indices [0..31]. + * + * Bank 0 + * + * --------------------------------- ------ + * | 00 | BRBSRC | BRBTGT | BRBINF | | 00 | + * --------------------------------- ------ + * | 01 | BRBSRC | BRBTGT | BRBINF | | 01 | + * --------------------------------- ------ + * | .. | BRBSRC | BRBTGT | BRBINF | | .. | + * --------------------------------- ------ + * | 31 | BRBSRC | BRBTGT | BRBINF | | 31 | + * --------------------------------- ------ + * + * Bank 1 + * + * --------------------------------- ------ + * | 32 | BRBSRC | BRBTGT | BRBINF | | 00 | + * --------------------------------- ------ + * | 33 | BRBSRC | BRBTGT | BRBINF | | 01 | + * --------------------------------- ------ + * | .. | BRBSRC | BRBTGT | BRBINF | | .. | + * --------------------------------- ------ + * | 63 | BRBSRC | BRBTGT | BRBINF | | 31 | + * --------------------------------- ------ + */ +#define BRBE_BANK_MAX_ENTRIES 32 + +#define BRBE_BANK0_IDX_MIN 0 +#define BRBE_BANK0_IDX_MAX 31 +#define BRBE_BANK1_IDX_MIN 32 +#define BRBE_BANK1_IDX_MAX 63 + +struct brbe_hw_attr { + int brbe_version; + int brbe_cc; + int brbe_nr; + int brbe_format; +}; + +enum brbe_bank_idx { + BRBE_BANK_IDX_INVALID = -1, + BRBE_BANK_IDX_0, + BRBE_BANK_IDX_1, + BRBE_BANK_IDX_MAX +}; + +#define RETURN_READ_BRBSRCN(n) \ + read_sysreg_s(SYS_BRBSRC##n##_EL1) + +#define RETURN_READ_BRBTGTN(n) \ + read_sysreg_s(SYS_BRBTGT##n##_EL1) + +#define RETURN_READ_BRBINFN(n) \ + read_sysreg_s(SYS_BRBINF##n##_EL1) + +#define BRBE_REGN_CASE(n, case_macro) \ + case n: return case_macro(n); break + +#define BRBE_REGN_SWITCH(x, case_macro) \ + do { \ + switch (x) { \ + BRBE_REGN_CASE(0, case_macro); \ + BRBE_REGN_CASE(1, case_macro); \ + BRBE_REGN_CASE(2, case_macro); \ + BRBE_REGN_CASE(3, case_macro); \ + BRBE_REGN_CASE(4, case_macro); \ + BRBE_REGN_CASE(5, case_macro); \ + BRBE_REGN_CASE(6, case_macro); \ + BRBE_REGN_CASE(7, case_macro); \ + BRBE_REGN_CASE(8, case_macro); \ + BRBE_REGN_CASE(9, case_macro); \ + BRBE_REGN_CASE(10, case_macro); \ + BRBE_REGN_CASE(11, case_macro); \ + BRBE_REGN_CASE(12, case_macro); \ + BRBE_REGN_CASE(13, case_macro); \ + BRBE_REGN_CASE(14, case_macro); \ + BRBE_REGN_CASE(15, case_macro); \ + BRBE_REGN_CASE(16, case_macro); \ + BRBE_REGN_CASE(17, case_macro); \ + BRBE_REGN_CASE(18, case_macro); \ + BRBE_REGN_CASE(19, case_macro); \ + BRBE_REGN_CASE(20, case_macro); \ + BRBE_REGN_CASE(21, case_macro); \ + BRBE_REGN_CASE(22, case_macro); \ + BRBE_REGN_CASE(23, case_macro); \ + BRBE_REGN_CASE(24, case_macro); \ + BRBE_REGN_CASE(25, case_macro); \ + BRBE_REGN_CASE(26, case_macro); \ + BRBE_REGN_CASE(27, case_macro); \ + BRBE_REGN_CASE(28, case_macro); \ + BRBE_REGN_CASE(29, case_macro); \ + BRBE_REGN_CASE(30, case_macro); \ + BRBE_REGN_CASE(31, case_macro); \ + default: \ + pr_warn("unknown register index\n"); \ + return -1; \ + } \ + } while (0) + +static inline int buffer_to_brbe_idx(int buffer_idx) +{ + return buffer_idx % BRBE_BANK_MAX_ENTRIES; +} + +static inline u64 get_brbsrc_reg(int buffer_idx) +{ + int brbe_idx = buffer_to_brbe_idx(buffer_idx); + + BRBE_REGN_SWITCH(brbe_idx, RETURN_READ_BRBSRCN); +} + +static inline u64 get_brbtgt_reg(int buffer_idx) +{ + int brbe_idx = buffer_to_brbe_idx(buffer_idx); + + BRBE_REGN_SWITCH(brbe_idx, RETURN_READ_BRBTGTN); +} + +static inline u64 get_brbinf_reg(int buffer_idx) +{ + int brbe_idx = buffer_to_brbe_idx(buffer_idx); + + BRBE_REGN_SWITCH(brbe_idx, RETURN_READ_BRBINFN); +} + +static inline u64 brbe_record_valid(u64 brbinf) +{ + return FIELD_GET(BRBINFx_EL1_VALID_MASK, brbinf); +} + +static inline bool brbe_invalid(u64 brbinf) +{ + return brbe_record_valid(brbinf) == BRBINFx_EL1_VALID_NONE; +} + +static inline bool brbe_record_is_complete(u64 brbinf) +{ + return brbe_record_valid(brbinf) == BRBINFx_EL1_VALID_FULL; +} + +static inline bool brbe_record_is_source_only(u64 brbinf) +{ + return brbe_record_valid(brbinf) == BRBINFx_EL1_VALID_SOURCE; +} + +static inline bool brbe_record_is_target_only(u64 brbinf) +{ + return brbe_record_valid(brbinf) == BRBINFx_EL1_VALID_TARGET; +} + +static inline int brbe_get_in_tx(u64 brbinf) +{ + return FIELD_GET(BRBINFx_EL1_T_MASK, brbinf); +} + +static inline int brbe_get_mispredict(u64 brbinf) +{ + return FIELD_GET(BRBINFx_EL1_MPRED_MASK, brbinf); +} + +static inline int brbe_get_lastfailed(u64 brbinf) +{ + return FIELD_GET(BRBINFx_EL1_LASTFAILED_MASK, brbinf); +} + +static inline int brbe_get_cycles(u64 brbinf) +{ + /* + * Captured cycle count is unknown and hence + * should not be passed on to the user space. + */ + if (brbinf & BRBINFx_EL1_CCU) + return 0; + + return FIELD_GET(BRBINFx_EL1_CC_MASK, brbinf); +} + +static inline int brbe_get_type(u64 brbinf) +{ + return FIELD_GET(BRBINFx_EL1_TYPE_MASK, brbinf); +} + +static inline int brbe_get_el(u64 brbinf) +{ + return FIELD_GET(BRBINFx_EL1_EL_MASK, brbinf); +} + +static inline int brbe_get_numrec(u64 brbidr) +{ + return FIELD_GET(BRBIDR0_EL1_NUMREC_MASK, brbidr); +} + +static inline int brbe_get_format(u64 brbidr) +{ + return FIELD_GET(BRBIDR0_EL1_FORMAT_MASK, brbidr); +} + +static inline int brbe_get_cc_bits(u64 brbidr) +{ + return FIELD_GET(BRBIDR0_EL1_CC_MASK, brbidr); +} diff --git a/drivers/perf/arm_pmuv3.c b/drivers/perf/arm_pmuv3.c index 86d803ff1ae3..fef1bc6067cc 100644 --- a/drivers/perf/arm_pmuv3.c +++ b/drivers/perf/arm_pmuv3.c @@ -797,6 +797,10 @@ static irqreturn_t armv8pmu_handle_irq(struct arm_pmu *cpu_pmu) if (!armpmu_event_set_period(event)) continue; + /* + * PMU IRQ should remain asserted until all branch records + * are captured and processed into struct perf_sample_data. + */ if (has_branch_stack(event) && !WARN_ON(!cpuc->branches)) { armv8pmu_branch_read(cpuc, event); perf_sample_save_brstack(&data, event, &cpuc->branches->branch_stack); @@ -1142,14 +1146,25 @@ static void __armv8pmu_probe_pmu(void *info) static int branch_records_alloc(struct arm_pmu *armpmu) { + struct branch_records __percpu *tmp_alloc_ptr; + struct branch_records *records; struct pmu_hw_events *events; int cpu; + tmp_alloc_ptr = alloc_percpu_gfp(struct branch_records, GFP_KERNEL); + if (!tmp_alloc_ptr) + return -ENOMEM; + + /* + * FIXME: Memory allocated via tmp_alloc_ptr gets completely + * consumed here, never required to be freed up later. Hence + * losing access to on stack 'tmp_alloc_ptr' is acceptible. + * Otherwise this alloc handle has to be saved some where. + */ for_each_possible_cpu(cpu) { events = per_cpu_ptr(armpmu->hw_events, cpu); - events->branches = kzalloc(sizeof(struct branch_records), GFP_KERNEL); - if (!events->branches) - return -ENOMEM; + records = per_cpu_ptr(tmp_alloc_ptr, cpu); + events->branches = records; } return 0; } From patchwork Wed May 31 04:04:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anshuman Khandual X-Patchwork-Id: 101229 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2634859vqr; Tue, 30 May 2023 21:41:54 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ69TcajxZC2x/bnzCwwkWm8mhUP8jU/wYFpbnvXN/+JndSeV6ZUlMFGLtGbCCwsH5qm7UH4 X-Received: by 2002:a05:6808:3025:b0:398:29ce:81f7 with SMTP id ay37-20020a056808302500b0039829ce81f7mr2791414oib.11.1685508113472; Tue, 30 May 2023 21:41:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685508113; cv=none; d=google.com; s=arc-20160816; b=v3ppQHBclNTbVkrMpMLfv5zHxCg9INLlLRxBafekfuN5wsOe6SWnboRq0ywmZYlgW2 3icYQ+ScfnjrITQe8+xWa8eD4VcyOTws11aXiXc2Pq835+MtJc2T08e98W+VQs4gA5Ph kI7lPJ4Wt9yQHk/TDxmHpyTMjuFSyMvWW7JF8TdN3DmpCejDhzcCWiuUYAbMrML9mXk4 1rCLJ3ZnYGEbE3wMZWd6M0JfFIqqZzy20i41tFHNtN5jsL1uG3p63+17RgYqUglx82lt DSIM9vjUhcKbOOlJxUu8er2y85CAJIAMUobaS28UpSvaWiCwf3cfm9UnXlf9zGTePwLN CvWQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=n3+m5C1RzAdusOr8unmcOZz1ysqV38JhqFjcUJ8hLjw=; b=QPSDXWE0ftOK21+84wduzND58dUvwITtc1VzAoknBKfS7e00BDrhE7IvgZMlTed87x EH+ofqTwZWjfAqzFGpFts7IB4cYwT/2TXdjzXq+MnAjgF3zaTWWnEAZa8R589uLxyfqY V2scril1a8cwZfS2luU/Dze8gA+phKCFyGh0xLlz0SQBIb4ntsleI+2xIZ8C99SNQOwU HJWiZETob9WHcnoqhIaMUC1rdZrpjjRNRfcrsAvysKaFyFy52ljT9o6fkx+8cCBPM0OJ XA8ko1tA8DtCvg3kmoABxHcbxeWqnrix60lFA03ajIPyawo8rPQx7BeRfce6I+VqB9Rd B1NQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q30-20020a638c5e000000b0053fb32e621bsi271328pgn.896.2023.05.30.21.41.39; Tue, 30 May 2023 21:41:53 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234256AbjEaEGe (ORCPT + 99 others); Wed, 31 May 2023 00:06:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55126 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234119AbjEaEFy (ORCPT ); Wed, 31 May 2023 00:05:54 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id C17A0E5D; Tue, 30 May 2023 21:05:38 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 89A9C15BF; Tue, 30 May 2023 21:06:23 -0700 (PDT) Received: from a077893.arm.com (unknown [10.163.73.163]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 05EA33F6C4; Tue, 30 May 2023 21:05:32 -0700 (PDT) From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, will@kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com Cc: Anshuman Khandual , Mark Brown , James Clark , Rob Herring , Marc Zyngier , Suzuki Poulose , Peter Zijlstra , Ingo Molnar , Arnaldo Carvalho de Melo , linux-perf-users@vger.kernel.org Subject: [PATCH V11 07/10] arm64/perf: Add PERF_ATTACH_TASK_DATA to events with has_branch_stack() Date: Wed, 31 May 2023 09:34:25 +0530 Message-Id: <20230531040428.501523-8-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230531040428.501523-1-anshuman.khandual@arm.com> References: <20230531040428.501523-1-anshuman.khandual@arm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1767383356031529464?= X-GMAIL-MSGID: =?utf-8?q?1767383356031529464?= Short running processes i.e those getting very small cpu run time each time when they get scheduled on, might not accumulate much branch records before a PMU IRQ really happens. This increases possibility, for such processes to loose much of its branch records, while being scheduled in-out of various cpus on the system. There is a need to save all occurred branch records during the cpu run time while the process gets scheduled out. It requires an event context specific buffer for such storage. This adds PERF_ATTACH_TASK_DATA flag unconditionally, for all branch stack sampling events, which would allocate task_ctx_data during its event init. This also creates a platform specific task_ctx_data kmem cache which will serve such allocation requests. This adds a new structure 'arm64_perf_task_context' which encapsulates brbe register set for maximum possible BRBE entries on the HW along with a valid records tracking element. Cc: Catalin Marinas Cc: Will Deacon Cc: Mark Rutland Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Tested-by: James Clark Signed-off-by: Anshuman Khandual --- drivers/perf/arm_brbe.c | 13 +++++++++++++ drivers/perf/arm_brbe.h | 13 +++++++++++++ drivers/perf/arm_pmuv3.c | 8 ++++++-- 3 files changed, 32 insertions(+), 2 deletions(-) diff --git a/drivers/perf/arm_brbe.c b/drivers/perf/arm_brbe.c index 34547ad750ad..484842d8cf3e 100644 --- a/drivers/perf/arm_brbe.c +++ b/drivers/perf/arm_brbe.c @@ -109,20 +109,33 @@ bool armv8pmu_branch_valid(struct perf_event *event) return true; } +static inline struct kmem_cache * +arm64_create_brbe_task_ctx_kmem_cache(size_t size) +{ + return kmem_cache_create("arm64_brbe_task_ctx", size, 0, 0, NULL); +} + int armv8pmu_private_alloc(struct arm_pmu *arm_pmu) { struct brbe_hw_attr *brbe_attr = kzalloc(sizeof(struct brbe_hw_attr), GFP_KERNEL); + size_t size = sizeof(struct arm64_perf_task_context); if (!brbe_attr) return -ENOMEM; arm_pmu->private = brbe_attr; + arm_pmu->pmu.task_ctx_cache = arm64_create_brbe_task_ctx_kmem_cache(size); + if (!arm_pmu->pmu.task_ctx_cache) { + kfree(arm_pmu->private); + return -ENOMEM; + } return 0; } void armv8pmu_private_free(struct arm_pmu *arm_pmu) { kfree(arm_pmu->private); + kmem_cache_destroy(arm_pmu->pmu.task_ctx_cache); } static int brbe_attributes_probe(struct arm_pmu *armpmu, u32 brbe) diff --git a/drivers/perf/arm_brbe.h b/drivers/perf/arm_brbe.h index a47480eec070..4a72c2ba7140 100644 --- a/drivers/perf/arm_brbe.h +++ b/drivers/perf/arm_brbe.h @@ -80,12 +80,25 @@ * --------------------------------- ------ */ #define BRBE_BANK_MAX_ENTRIES 32 +#define BRBE_MAX_BANK 2 +#define BRBE_MAX_ENTRIES (BRBE_BANK_MAX_ENTRIES * BRBE_MAX_BANK) #define BRBE_BANK0_IDX_MIN 0 #define BRBE_BANK0_IDX_MAX 31 #define BRBE_BANK1_IDX_MIN 32 #define BRBE_BANK1_IDX_MAX 63 +struct brbe_regset { + unsigned long brbsrc; + unsigned long brbtgt; + unsigned long brbinf; +}; + +struct arm64_perf_task_context { + struct brbe_regset store[BRBE_MAX_ENTRIES]; + int nr_brbe_records; +}; + struct brbe_hw_attr { int brbe_version; int brbe_cc; diff --git a/drivers/perf/arm_pmuv3.c b/drivers/perf/arm_pmuv3.c index fef1bc6067cc..29672ff20026 100644 --- a/drivers/perf/arm_pmuv3.c +++ b/drivers/perf/arm_pmuv3.c @@ -1022,8 +1022,12 @@ static int __armv8_pmuv3_map_event(struct perf_event *event, hw_event_id = __armv8_pmuv3_map_event_id(armpmu, event); - if (has_branch_stack(event) && !armv8pmu_branch_valid(event)) - return -EOPNOTSUPP; + if (has_branch_stack(event)) { + if (!armv8pmu_branch_valid(event)) + return -EOPNOTSUPP; + + event->attach_state |= PERF_ATTACH_TASK_DATA; + } /* * CHAIN events only work when paired with an adjacent counter, and it From patchwork Wed May 31 04:04:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anshuman Khandual X-Patchwork-Id: 101224 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2630395vqr; Tue, 30 May 2023 21:26:56 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5AnbrTo8A6qzIRUHu2GZ4A0YWz87rH9kGaGHwvEV1B9dz8edfsmdKBSt+hwzLB0e6kbiun X-Received: by 2002:a17:902:d2d0:b0:1ae:8fa:cd4c with SMTP id n16-20020a170902d2d000b001ae08facd4cmr14121020plc.7.1685507216192; Tue, 30 May 2023 21:26:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685507216; cv=none; d=google.com; s=arc-20160816; b=GIMGQbPfZeSK0VjJVlD1Wf6U2fPcTVCrvhpbfW8PQbCA5tRsPY9uy/3zYTW3RcmKgK rpIxkqgrsV+mgkcWeO05zTAsl9Rt6RaLqPinK8m12HcY89MBiG4PWzzn5vIx3/KlDw9+ 6X91l2L2CV399BAL8D9mvm4dkfctzC/tcAXEJq63HwwaRDVMMSinIeBn7m8wDSD6S5WB VM3YTtVJCNdmW1kVWRsMa6cFi9VHPNHbk2xMguo8ZIEh71ok4ToITjiJnwHnvRH8z2n0 rmdqxGPtcF7tE6Hi6eGkVOS1bo4tFc+RZo5HwwQ5Y9HxcgK/uUCN/6yYZJkCuN7uAklD +NPw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=c70OKjiv0Fn7XZ8+ZZ1HzDi9eCv1icTS3FLXrLUO2gc=; b=svfWWOS93udKH9RpwgTIgvbYTlap5gC9ORUOHbP+Qe323V/ET4dCDUnOQS5/DzHyFc 7lnOTODSfCyR3Qc7GGAdo+KYovlnPZVFYeAyvhxYhFN0DXVa+2sGvg6JpdtbIEKysKFP ZWZCSjQh1nP73DFhuihT88BcwMLFQXZ4580C5qIA4QIc8XiZFdp4Hi6WH1U2+Nu8yxT5 QdufwYIiMkK98E+bGswhdt67umv4zfvPRVd6orz2alf2+dP1YLsLNKadTNLBMgycKwQI lPDNtOGFJ12tQ4BJMO4u8BtWCPZ7U/lRqWrbaO6vlO7lB+29pvTHyoU3fet2icVVfZBf J0dw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h7-20020a170902680700b001b061af0599si195357plk.340.2023.05.30.21.26.44; Tue, 30 May 2023 21:26:56 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234039AbjEaEGt (ORCPT + 99 others); Wed, 31 May 2023 00:06:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55396 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234265AbjEaEGG (ORCPT ); Wed, 31 May 2023 00:06:06 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id B0C94E6D; Tue, 30 May 2023 21:05:44 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5AE1615DB; Tue, 30 May 2023 21:06:29 -0700 (PDT) Received: from a077893.arm.com (unknown [10.163.73.163]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id EA8CF3F6C4; Tue, 30 May 2023 21:05:38 -0700 (PDT) From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, will@kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com Cc: Anshuman Khandual , Mark Brown , James Clark , Rob Herring , Marc Zyngier , Suzuki Poulose , Peter Zijlstra , Ingo Molnar , Arnaldo Carvalho de Melo , linux-perf-users@vger.kernel.org Subject: [PATCH V11 08/10] arm64/perf: Add struct brbe_regset helper functions Date: Wed, 31 May 2023 09:34:26 +0530 Message-Id: <20230531040428.501523-9-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230531040428.501523-1-anshuman.khandual@arm.com> References: <20230531040428.501523-1-anshuman.khandual@arm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1767382414797224718?= X-GMAIL-MSGID: =?utf-8?q?1767382414797224718?= The primary abstraction level for fetching branch records from BRBE HW has been changed as 'struct brbe_regset', which contains storage for all three BRBE registers i.e BRBSRC, BRBTGT, BRBINF. Whether branch record processing happens in the task sched out path, or in the PMU IRQ handling path, these registers need to be extracted from the HW. Afterwards both live and stored sets need to be stitched together to create final branch records set. This adds required helper functions for such operations. Cc: Catalin Marinas Cc: Will Deacon Cc: Mark Rutland Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Tested-by: James Clark Signed-off-by: Anshuman Khandual --- drivers/perf/arm_brbe.c | 163 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 163 insertions(+) diff --git a/drivers/perf/arm_brbe.c b/drivers/perf/arm_brbe.c index 484842d8cf3e..759db681d673 100644 --- a/drivers/perf/arm_brbe.c +++ b/drivers/perf/arm_brbe.c @@ -44,6 +44,169 @@ static void select_brbe_bank(int bank) isb(); } +/* + * This scans over BRBE register banks and captures individual branch reocrds + * [BRBSRC, BRBTGT, BRBINF] into a pre-allocated 'struct brbe_regset' buffer, + * until an invalid one gets encountered. The caller for this function needs + * to ensure BRBE is an appropriate state before the records can be captured. + */ +static int capture_brbe_regset(struct brbe_hw_attr *brbe_attr, struct brbe_regset *buf) +{ + int loop1_idx1, loop1_idx2, loop2_idx1, loop2_idx2; + int idx, count; + + loop1_idx1 = BRBE_BANK0_IDX_MIN; + if (brbe_attr->brbe_nr <= BRBE_BANK_MAX_ENTRIES) { + loop1_idx2 = brbe_attr->brbe_nr - 1; + loop2_idx1 = BRBE_BANK1_IDX_MIN; + loop2_idx2 = BRBE_BANK0_IDX_MAX; + } else { + loop1_idx2 = BRBE_BANK0_IDX_MAX; + loop2_idx1 = BRBE_BANK1_IDX_MIN; + loop2_idx2 = brbe_attr->brbe_nr - 1; + } + + select_brbe_bank(BRBE_BANK_IDX_0); + for (idx = 0, count = loop1_idx1; count <= loop1_idx2; idx++, count++) { + buf[idx].brbinf = get_brbinf_reg(idx); + /* + * There are no valid entries anymore on the buffer. + * Abort the branch record processing to save some + * cycles and also reduce the capture/process load + * for the user space as well. + */ + if (brbe_invalid(buf[idx].brbinf)) + return idx; + + buf[idx].brbsrc = get_brbsrc_reg(idx); + buf[idx].brbtgt = get_brbtgt_reg(idx); + } + + select_brbe_bank(BRBE_BANK_IDX_1); + for (count = loop2_idx1; count <= loop2_idx2; idx++, count++) { + buf[idx].brbinf = get_brbinf_reg(idx); + /* + * There are no valid entries anymore on the buffer. + * Abort the branch record processing to save some + * cycles and also reduce the capture/process load + * for the user space as well. + */ + if (brbe_invalid(buf[idx].brbinf)) + return idx; + + buf[idx].brbsrc = get_brbsrc_reg(idx); + buf[idx].brbtgt = get_brbtgt_reg(idx); + } + return idx; +} + +static inline void copy_brbe_regset(struct brbe_regset *src, int src_idx, + struct brbe_regset *dst, int dst_idx) +{ + dst[dst_idx].brbinf = src[src_idx].brbinf; + dst[dst_idx].brbsrc = src[src_idx].brbsrc; + dst[dst_idx].brbtgt = src[src_idx].brbtgt; +} + +/* + * This function concatenates branch records from stored and live buffer + * up to maximum nr_max records and the stored buffer holds the resultant + * buffer. The concatenated buffer contains all the branch records from + * the live buffer but might contain some from stored buffer considering + * the maximum combined length does not exceed 'nr_max'. + * + * Stored records Live records + * ------------------------------------------------^ + * | S0 | L0 | Newest | + * --------------------------------- | + * | S1 | L1 | | + * --------------------------------- | + * | S2 | L2 | | + * --------------------------------- | + * | S3 | L3 | | + * --------------------------------- | + * | S4 | L4 | nr_max + * --------------------------------- | + * | | L5 | | + * --------------------------------- | + * | | L6 | | + * --------------------------------- | + * | | L7 | | + * --------------------------------- | + * | | | | + * --------------------------------- | + * | | | Oldest | + * ------------------------------------------------V + * + * + * S0 is the newest in the stored records, where as L7 is the oldest in + * the live reocords. Unless the live buffer is detetcted as being full + * thus potentially dropping off some older records, L7 and S0 records + * are contiguous in time for a user task context. The stitched buffer + * here represents maximum possible branch records, contiguous in time. + * + * Stored records Live records + * ------------------------------------------------^ + * | L0 | L0 | Newest | + * --------------------------------- | + * | L0 | L1 | | + * --------------------------------- | + * | L2 | L2 | | + * --------------------------------- | + * | L3 | L3 | | + * --------------------------------- | + * | L4 | L4 | nr_max + * --------------------------------- | + * | L5 | L5 | | + * --------------------------------- | + * | L6 | L6 | | + * --------------------------------- | + * | L7 | L7 | | + * --------------------------------- | + * | S0 | | | + * --------------------------------- | + * | S1 | | Oldest | + * ------------------------------------------------V + * | S2 | <----| + * ----------------- | + * | S3 | <----| Dropped off after nr_max + * ----------------- | + * | S4 | <----| + * ----------------- + */ +static int stitch_stored_live_entries(struct brbe_regset *stored, + struct brbe_regset *live, + int nr_stored, int nr_live, + int nr_max) +{ + int nr_total, nr_excess, nr_last, i; + + nr_total = nr_stored + nr_live; + nr_excess = nr_total - nr_max; + + /* Stored branch records in stitched buffer */ + if (nr_live == nr_max) + nr_stored = 0; + else if (nr_excess > 0) + nr_stored -= nr_excess; + + /* Stitched buffer branch records length */ + if (nr_total > nr_max) + nr_last = nr_max; + else + nr_last = nr_total; + + /* Move stored branch records */ + for (i = 0; i < nr_stored; i++) + copy_brbe_regset(stored, i, stored, nr_last - nr_stored - 1 + i); + + /* Copy live branch records */ + for (i = 0; i < nr_live; i++) + copy_brbe_regset(live, i, stored, i); + + return nr_last; +} + /* * Generic perf branch filters supported on BRBE * From patchwork Wed May 31 04:04:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anshuman Khandual X-Patchwork-Id: 101225 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2630694vqr; Tue, 30 May 2023 21:28:08 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6SHzN17AZFCWSKVy8Wy2NdoJC4r6UziCQ1njRAn+EG0y8T1mvqoE8U0wiSFRT9xZEWPQ6P X-Received: by 2002:a05:6808:3090:b0:395:f4fd:9fe9 with SMTP id bl16-20020a056808309000b00395f4fd9fe9mr3631235oib.50.1685507288155; Tue, 30 May 2023 21:28:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685507288; cv=none; d=google.com; s=arc-20160816; b=hHErH/CgHouMHETje8IYGylAEkSiV8cmeaLznwwb4mRT/tVOioBwhvCQ2Dkn6zyMu1 32Kt7GgGs0L0FbXGSP2T2O+RHRSIcISYcOVpCBHg2PA7a6LlfJs4dY/gi7l9JrryUUGX W9H1UwuY2QOqSMk4TD0PV4O8Td6sahm7XKhbMtQPkiTpPWdCX6Fw4Sre+LcUzBJ8D5vv wWpGqew2pVLYlWCkuMDDnetbT9PdImxPXM/4c6hLNGfPRZjDCqFJFj5GaQgkdk7mBI7d ol+ed9OCnE2KOFlcZYDql2u2c5tmsHeIGrPqRL5fcP8lXMCObghujadrJZz+2h1Xi1IU 0Irg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=qPuXWntLaW802UGQUeuE9f22fbA5mHyRl93Q7mxtCP4=; b=J8f7QnypWch1mGCmbWQ7ABdhhOVifqBvrWFTOQc4WBlHSW5xMDnpOF5+jspqnxejSk wjdL/UjuEWk8cKuHktDp7TT+r1KGhwAnpibQGOSqrYKj8OPyP+IzOGx1N8zMs/GzLCp1 OIEYSh9NdjFtzGjlY+INW11D8Lr0YJKohUmA4g7F2IMVEL5U+xJN7+GDXVH/tvFeCUCP oSFFSBq+Gaf0+hwVLDq4Bibwmmu9B4c+gDRz4826L/qqMCdDFupDdGDNh0NFw0UDp9kh wA7FDTRQbhlguzmH832XJFrmTUOB+LemdqR06Kyy4/k0Vo8gpg8e0kOIqqVlB5DmWYjf 3weQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k64-20020a638443000000b0052c6200e5e1si273624pgd.649.2023.05.30.21.27.56; Tue, 30 May 2023 21:28:08 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234277AbjEaEG5 (ORCPT + 99 others); Wed, 31 May 2023 00:06:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55598 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234243AbjEaEGQ (ORCPT ); Wed, 31 May 2023 00:06:16 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 58B891B1; Tue, 30 May 2023 21:05:50 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6C14E165C; Tue, 30 May 2023 21:06:35 -0700 (PDT) Received: from a077893.arm.com (unknown [10.163.73.163]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 9795D3F6C4; Tue, 30 May 2023 21:05:44 -0700 (PDT) From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, will@kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com Cc: Anshuman Khandual , Mark Brown , James Clark , Rob Herring , Marc Zyngier , Suzuki Poulose , Peter Zijlstra , Ingo Molnar , Arnaldo Carvalho de Melo , linux-perf-users@vger.kernel.org Subject: [PATCH V11 09/10] arm64/perf: Implement branch records save on task sched out Date: Wed, 31 May 2023 09:34:27 +0530 Message-Id: <20230531040428.501523-10-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230531040428.501523-1-anshuman.khandual@arm.com> References: <20230531040428.501523-1-anshuman.khandual@arm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1767382490195424651?= X-GMAIL-MSGID: =?utf-8?q?1767382490195424651?= This modifies current armv8pmu_sched_task(), to implement a branch records save mechanism via armv8pmu_branch_save() when a task scheds out of a cpu. BRBE is paused and disabled for all exception levels before branch records get captured, which then get concatenated with all existing stored records present in the task context maintaining the contiguity. Although the final length of the concatenated buffer does not exceed implemented BRBE length. Cc: Catalin Marinas Cc: Will Deacon Cc: Mark Rutland Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Tested-by: James Clark Signed-off-by: Anshuman Khandual --- arch/arm64/include/asm/perf_event.h | 2 ++ drivers/perf/arm_brbe.c | 30 +++++++++++++++++++++++++++++ drivers/perf/arm_pmuv3.c | 14 ++++++++++++-- 3 files changed, 44 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/perf_event.h b/arch/arm64/include/asm/perf_event.h index f071d629c0cf..c81b768cd172 100644 --- a/arch/arm64/include/asm/perf_event.h +++ b/arch/arm64/include/asm/perf_event.h @@ -40,6 +40,7 @@ void armv8pmu_branch_probe(struct arm_pmu *arm_pmu); void armv8pmu_branch_reset(void); int armv8pmu_private_alloc(struct arm_pmu *arm_pmu); void armv8pmu_private_free(struct arm_pmu *arm_pmu); +void armv8pmu_branch_save(struct arm_pmu *arm_pmu, void *ctx); #else static inline void armv8pmu_branch_read(struct pmu_hw_events *cpuc, struct perf_event *event) { @@ -66,6 +67,7 @@ static inline void armv8pmu_branch_probe(struct arm_pmu *arm_pmu) { } static inline void armv8pmu_branch_reset(void) { } static inline int armv8pmu_private_alloc(struct arm_pmu *arm_pmu) { return 0; } static inline void armv8pmu_private_free(struct arm_pmu *arm_pmu) { } +static inline void armv8pmu_branch_save(struct arm_pmu *arm_pmu, void *ctx) { } #endif #endif #endif diff --git a/drivers/perf/arm_brbe.c b/drivers/perf/arm_brbe.c index 759db681d673..0678ebf0a896 100644 --- a/drivers/perf/arm_brbe.c +++ b/drivers/perf/arm_brbe.c @@ -207,6 +207,36 @@ static int stitch_stored_live_entries(struct brbe_regset *stored, return nr_last; } +static int brbe_branch_save(struct brbe_hw_attr *brbe_attr, struct brbe_regset *live) +{ + u64 brbfcr = read_sysreg_s(SYS_BRBFCR_EL1); + int nr_live; + + write_sysreg_s(brbfcr | BRBFCR_EL1_PAUSED, SYS_BRBFCR_EL1); + isb(); + + nr_live = capture_brbe_regset(brbe_attr, live); + + write_sysreg_s(brbfcr & ~BRBFCR_EL1_PAUSED, SYS_BRBFCR_EL1); + isb(); + + return nr_live; +} + +void armv8pmu_branch_save(struct arm_pmu *arm_pmu, void *ctx) +{ + struct brbe_hw_attr *brbe_attr = (struct brbe_hw_attr *)arm_pmu->private; + struct arm64_perf_task_context *task_ctx = ctx; + struct brbe_regset live[BRBE_MAX_ENTRIES]; + int nr_live, nr_store; + + nr_live = brbe_branch_save(brbe_attr, live); + nr_store = task_ctx->nr_brbe_records; + nr_store = stitch_stored_live_entries(task_ctx->store, live, nr_store, + nr_live, brbe_attr->brbe_nr); + task_ctx->nr_brbe_records = nr_store; +} + /* * Generic perf branch filters supported on BRBE * diff --git a/drivers/perf/arm_pmuv3.c b/drivers/perf/arm_pmuv3.c index 29672ff20026..9725a53d6799 100644 --- a/drivers/perf/arm_pmuv3.c +++ b/drivers/perf/arm_pmuv3.c @@ -907,9 +907,19 @@ static int armv8pmu_user_event_idx(struct perf_event *event) static void armv8pmu_sched_task(struct perf_event_pmu_context *pmu_ctx, bool sched_in) { struct arm_pmu *armpmu = to_arm_pmu(pmu_ctx->pmu); + void *task_ctx = pmu_ctx ? pmu_ctx->task_ctx_data : NULL; - if (sched_in && arm_pmu_branch_stack_supported(armpmu)) - armv8pmu_branch_reset(); + if (arm_pmu_branch_stack_supported(armpmu)) { + /* Save branch records in task_ctx on sched out */ + if (task_ctx && !sched_in) { + armv8pmu_branch_save(armpmu, task_ctx); + return; + } + + /* Reset branch records on sched in */ + if (sched_in) + armv8pmu_branch_reset(); + } } /* From patchwork Wed May 31 04:04:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anshuman Khandual X-Patchwork-Id: 101216 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2628336vqr; Tue, 30 May 2023 21:19:41 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7SuJ/wq1kCunhE71khdJh/5xGgXmfJTNgn7eKqUf8m+BGSXsVJQ3vQLKgHIWx/8VdQNtQI X-Received: by 2002:a17:903:550:b0:1a6:84be:a08f with SMTP id jo16-20020a170903055000b001a684bea08fmr3485606plb.64.1685506781063; Tue, 30 May 2023 21:19:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685506781; cv=none; d=google.com; s=arc-20160816; b=ISmO9Z4PcqPJm1FDHuw6r0okQhZx8LdX9HCeR1WPEZFOgYLa9i2vOYK65sAmAOgno6 U2UmOcOAfwHBzQAyRdE0EUC1FPK+3LFMnlgTbJX8Ceq2EVsDdaC+LTSsbAmzoDO0ucsP LNn85fNovHJlcePDRB8Ac/t/j1FZszKprMVp9BQ0ZFf8ZRf154+wx5kb8CTJcbejuCmm ivKHAyoXls++Xt2HWmsUdYcIAMW5lfCm4L2Y9iBsEu9J1tv/Mz288ZF69rF5VU9cJLqX gQ+zy1n5J8WQlSWsMyOTkp5l6xPUW3Ku/cwW9wJ7DCslGk/5Ru3m4XtS3WXE0ayOxgxN EiVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=1eHQVmKzWRmoHBK4zIE9tdcqs1VwI15J9h6z2PpV5sA=; b=NBguwnpokQihe8VA1aiBr65IzIQQ+iRiDq7aCGvHS01oyMyI68nPNEaTcdbo8/zE+w IJZu3hmcRx7lOhW76rcp1TS488wz24AnLVSD4iUceU+yndKKf8wl2xPPsb36qHbQBFvc GBj2Y3xdph9mIJj4rtkDhh9FO7YxeGrzlbnWMMBq1pvMTyt7UJrjWnamXyANj5eWorZu eWzCkMBKgATyOybnvuqC0aMJq8ejdrPXJDU4IaFUScAz1NZLMlBC5cjh9o32FZobBlVH D4zwas2JblH6JVQp8ZFzGPLeKGCycgmCI8Magk6EjtHRwrU8g2kXJeiszJ6kIvQA6U6d unZg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id x2-20020a170902ea8200b001aafec82436si209283plb.204.2023.05.30.21.19.27; Tue, 30 May 2023 21:19:41 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234249AbjEaEHF (ORCPT + 99 others); Wed, 31 May 2023 00:07:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55648 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234228AbjEaEGa (ORCPT ); Wed, 31 May 2023 00:06:30 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 2D7D01BE; Tue, 30 May 2023 21:05:56 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 30F4B15BF; Tue, 30 May 2023 21:06:41 -0700 (PDT) Received: from a077893.arm.com (unknown [10.163.73.163]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id B980F3F6C4; Tue, 30 May 2023 21:05:50 -0700 (PDT) From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, will@kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com Cc: Anshuman Khandual , Mark Brown , James Clark , Rob Herring , Marc Zyngier , Suzuki Poulose , Peter Zijlstra , Ingo Molnar , Arnaldo Carvalho de Melo , linux-perf-users@vger.kernel.org Subject: [PATCH V11 10/10] arm64/perf: Implement branch records save on PMU IRQ Date: Wed, 31 May 2023 09:34:28 +0530 Message-Id: <20230531040428.501523-11-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230531040428.501523-1-anshuman.khandual@arm.com> References: <20230531040428.501523-1-anshuman.khandual@arm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1767381958252462948?= X-GMAIL-MSGID: =?utf-8?q?1767381958252462948?= This modifies armv8pmu_branch_read() to concatenate live entries along with task context stored entries and then process the resultant buffer to create perf branch entry array for perf_sample_data. It follows the same principle like task sched out. Cc: Catalin Marinas Cc: Will Deacon Cc: Mark Rutland Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Tested-by: James Clark Signed-off-by: Anshuman Khandual --- drivers/perf/arm_brbe.c | 75 +++++++++++++++++------------------------ 1 file changed, 30 insertions(+), 45 deletions(-) diff --git a/drivers/perf/arm_brbe.c b/drivers/perf/arm_brbe.c index 0678ebf0a896..e3efc1563111 100644 --- a/drivers/perf/arm_brbe.c +++ b/drivers/perf/arm_brbe.c @@ -693,41 +693,45 @@ void armv8pmu_branch_reset(void) isb(); } -static bool capture_branch_entry(struct pmu_hw_events *cpuc, - struct perf_event *event, int idx) +static void brbe_regset_branch_entries(struct pmu_hw_events *cpuc, struct perf_event *event, + struct brbe_regset *regset, int idx) { struct perf_branch_entry *entry = &cpuc->branches->branch_entries[idx]; - u64 brbinf = get_brbinf_reg(idx); - - /* - * There are no valid entries anymore on the buffer. - * Abort the branch record processing to save some - * cycles and also reduce the capture/process load - * for the user space as well. - */ - if (brbe_invalid(brbinf)) - return false; + u64 brbinf = regset[idx].brbinf; perf_clear_branch_entry_bitfields(entry); if (brbe_record_is_complete(brbinf)) { - entry->from = get_brbsrc_reg(idx); - entry->to = get_brbtgt_reg(idx); + entry->from = regset[idx].brbsrc; + entry->to = regset[idx].brbtgt; } else if (brbe_record_is_source_only(brbinf)) { - entry->from = get_brbsrc_reg(idx); + entry->from = regset[idx].brbsrc; entry->to = 0; } else if (brbe_record_is_target_only(brbinf)) { entry->from = 0; - entry->to = get_brbtgt_reg(idx); + entry->to = regset[idx].brbtgt; } capture_brbe_flags(entry, event, brbinf); - return true; +} + +static void process_branch_entries(struct pmu_hw_events *cpuc, struct perf_event *event, + struct brbe_regset *regset, int nr_regset) +{ + int idx; + + for (idx = 0; idx < nr_regset; idx++) + brbe_regset_branch_entries(cpuc, event, regset, idx); + + cpuc->branches->branch_stack.nr = nr_regset; + cpuc->branches->branch_stack.hw_idx = -1ULL; } void armv8pmu_branch_read(struct pmu_hw_events *cpuc, struct perf_event *event) { struct brbe_hw_attr *brbe_attr = (struct brbe_hw_attr *)cpuc->percpu_pmu->private; + struct arm64_perf_task_context *task_ctx = event->pmu_ctx->task_ctx_data; + struct brbe_regset live[BRBE_MAX_ENTRIES]; + int nr_live, nr_store; u64 brbfcr, brbcr; - int idx, loop1_idx1, loop1_idx2, loop2_idx1, loop2_idx2, count; brbcr = read_sysreg_s(SYS_BRBCR_EL1); brbfcr = read_sysreg_s(SYS_BRBFCR_EL1); @@ -739,35 +743,16 @@ void armv8pmu_branch_read(struct pmu_hw_events *cpuc, struct perf_event *event) write_sysreg_s(brbfcr | BRBFCR_EL1_PAUSED, SYS_BRBFCR_EL1); isb(); - /* Determine the indices for each loop */ - loop1_idx1 = BRBE_BANK0_IDX_MIN; - if (brbe_attr->brbe_nr <= BRBE_BANK_MAX_ENTRIES) { - loop1_idx2 = brbe_attr->brbe_nr - 1; - loop2_idx1 = BRBE_BANK1_IDX_MIN; - loop2_idx2 = BRBE_BANK0_IDX_MAX; + nr_live = capture_brbe_regset(brbe_attr, live); + if (event->ctx->task) { + nr_store = task_ctx->nr_brbe_records; + nr_store = stitch_stored_live_entries(task_ctx->store, live, nr_store, + nr_live, brbe_attr->brbe_nr); + process_branch_entries(cpuc, event, task_ctx->store, nr_store); + task_ctx->nr_brbe_records = 0; } else { - loop1_idx2 = BRBE_BANK0_IDX_MAX; - loop2_idx1 = BRBE_BANK1_IDX_MIN; - loop2_idx2 = brbe_attr->brbe_nr - 1; - } - - /* Loop through bank 0 */ - select_brbe_bank(BRBE_BANK_IDX_0); - for (idx = 0, count = loop1_idx1; count <= loop1_idx2; idx++, count++) { - if (!capture_branch_entry(cpuc, event, idx)) - goto skip_bank_1; - } - - /* Loop through bank 1 */ - select_brbe_bank(BRBE_BANK_IDX_1); - for (count = loop2_idx1; count <= loop2_idx2; idx++, count++) { - if (!capture_branch_entry(cpuc, event, idx)) - break; + process_branch_entries(cpuc, event, live, nr_live); } - -skip_bank_1: - cpuc->branches->branch_stack.nr = idx; - cpuc->branches->branch_stack.hw_idx = -1ULL; process_branch_aborts(cpuc); /* Unpause the buffer */