diff mbox series

perf/x86/rapl: Enable Core RAPL for AMD

Message ID	20230217161354.129442-1-wyes.karny@amd.com
State	New
Headers	Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C From: Wyes Karny <wyes.karny@amd.com> To: Peter Zijlstra <peterz@infradead.org>, Ingo Molnar <mingo@redhat.com>, Arnaldo Carvalho de Melo <acme@kernel.org>, Mark Rutland <mark.rutland@arm.com>, Alexander Shishkin <alexander.shishkin@linux.intel.com>, Jiri Olsa <jolsa@kernel.org>, "Namhyung Kim" <namhyung@kernel.org>, Thomas Gleixner <tglx@linutronix.de>, "Borislav Petkov" <bp@alien8.de>, Dave Hansen <dave.hansen@linux.intel.com>, <hpa@zytor.com> CC: <x86@kernel.org>, <linux-perf-users@vger.kernel.org>, <linux-kernel@vger.kernel.org>, <gautham.shenoy@amd.com>, <ananth.narayan@amd.com>, Wyes Karny <wyes.karny@amd.com> Subject: [PATCH] perf/x86/rapl: Enable Core RAPL for AMD Date: Fri, 17 Feb 2023 16:13:54 +0000 Message-ID: <20230217161354.129442-1-wyes.karny@amd.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain Precedence: bulk
Series	perf/x86/rapl: Enable Core RAPL for AMD \| perf/x86/rapl: Enable Core RAPL for AMD

Commit Message

Wyes Karny Feb. 17, 2023, 4:13 p.m. UTC

  AMD processors support per-package and per-core energy monitoring
through RAPL counters which can be accessed by users running in
supervisor mode.

Core RAPL counters gives power consumption information per core.  For
AMD processors the package level RAPL counter are already exposed to
perf. Expose the core level RAPL counters also.

sudo perf stat -a --per-core -C 0-127 -e power/energy-cores/

Output:
S0-D0-C0           2               8.73 Joules power/energy-cores/
S0-D0-C1           2               8.73 Joules power/energy-cores/
S0-D0-C2           2               8.73 Joules power/energy-cores/
S0-D0-C3           2               8.73 Joules power/energy-cores/
S0-D0-C4           2               8.73 Joules power/energy-cores/

Signed-off-by: Wyes Karny <wyes.karny@amd.com>
---
 arch/x86/events/rapl.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

Comments

Oleksandr Natalenko Feb. 19, 2023, 11:52 a.m. UTC | #1

Hello.

On pátek 17. února 2023 17:13:54 CET Wyes Karny wrote:
> AMD processors support per-package and per-core energy monitoring
> through RAPL counters which can be accessed by users running in
> supervisor mode.
> 
> Core RAPL counters gives power consumption information per core.  For
> AMD processors the package level RAPL counter are already exposed to
> perf. Expose the core level RAPL counters also.
> 
> sudo perf stat -a --per-core -C 0-127 -e power/energy-cores/
> 
> Output:
> S0-D0-C0           2               8.73 Joules power/energy-cores/
> S0-D0-C1           2               8.73 Joules power/energy-cores/
> S0-D0-C2           2               8.73 Joules power/energy-cores/
> S0-D0-C3           2               8.73 Joules power/energy-cores/
> S0-D0-C4           2               8.73 Joules power/energy-cores/
> 
> Signed-off-by: Wyes Karny <wyes.karny@amd.com>
> ---
>  arch/x86/events/rapl.c | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/x86/events/rapl.c b/arch/x86/events/rapl.c
> index 52e6e7ed4f78..d301bbbc3b93 100644
> --- a/arch/x86/events/rapl.c
> +++ b/arch/x86/events/rapl.c
> @@ -537,7 +537,7 @@ static struct perf_msr intel_rapl_spr_msrs[] = {
>   * - want to use same event codes across both architectures
>   */
>  static struct perf_msr amd_rapl_msrs[] = {
> -	[PERF_RAPL_PP0]  = { 0, &rapl_events_cores_group, 0, false, 0 },
> +	[PERF_RAPL_PP0]  = { MSR_AMD_CORE_ENERGY_STATUS, &rapl_events_cores_group, test_msr, false, RAPL_MSR_MASK },
>  	[PERF_RAPL_PKG]  = { MSR_AMD_PKG_ENERGY_STATUS,  &rapl_events_pkg_group,   test_msr, false, RAPL_MSR_MASK },
>  	[PERF_RAPL_RAM]  = { 0, &rapl_events_ram_group,   0, false, 0 },
>  	[PERF_RAPL_PP1]  = { 0, &rapl_events_gpu_group,   0, false, 0 },
> @@ -764,7 +764,8 @@ static struct rapl_model model_spr = {
>  };
>  
>  static struct rapl_model model_amd_hygon = {
> -	.events		= BIT(PERF_RAPL_PKG),
> +	.events		= BIT(PERF_RAPL_PP0) |
> +			  BIT(PERF_RAPL_PKG),
>  	.msr_power_unit = MSR_AMD_RAPL_POWER_UNIT,
>  	.rapl_msrs      = amd_rapl_msrs,
>  };
> 

With this patch:

```
$ lscpu | grep 'Model name'
Model name:                      AMD Ryzen 9 5950X 16-Core Processor

$ sudo perf stat -a --per-core -C 0-15 -e power/energy-cores/ -- dd if=/dev/zero of=/dev/null bs=1M count=100000
100000+0 records in
100000+0 records out
104857600000 bytes (105 GB, 98 GiB) copied, 1,59252 s, 65,8 GB/s

Performance counter stats for 'system wide':

S0-D0-C0           1               1,56 Joules power/energy-cores/
S0-D0-C1           1               1,56 Joules power/energy-cores/
S0-D0-C2           1               1,56 Joules power/energy-cores/
S0-D0-C3           1               1,56 Joules power/energy-cores/
S0-D0-C4           1               1,56 Joules power/energy-cores/
S0-D0-C5           1               1,56 Joules power/energy-cores/
S0-D0-C6           1               1,56 Joules power/energy-cores/
S0-D0-C7           1               1,56 Joules power/energy-cores/
S0-D0-C8           1               1,56 Joules power/energy-cores/
S0-D0-C9           1               1,56 Joules power/energy-cores/
S0-D0-C10          1               1,56 Joules power/energy-cores/
S0-D0-C11          1               1,56 Joules power/energy-cores/
S0-D0-C12          1               1,56 Joules power/energy-cores/
S0-D0-C13          1               1,56 Joules power/energy-cores/
S0-D0-C14          1               1,56 Joules power/energy-cores/
S0-D0-C15          1               1,56 Joules power/energy-cores/

1,593982452 seconds time elapsed
```

Hence,

Tested-by: Oleksandr Natalenko <oleksandr@natalenko.name>

Thank you.

Peter Zijlstra Feb. 20, 2023, 11:45 a.m. UTC | #2

On Fri, Feb 17, 2023 at 04:13:54PM +0000, Wyes Karny wrote:
> AMD processors support per-package and per-core energy monitoring
> through RAPL counters which can be accessed by users running in
> supervisor mode.
> 
> Core RAPL counters gives power consumption information per core.  For
> AMD processors the package level RAPL counter are already exposed to
> perf. Expose the core level RAPL counters also.
> 
> sudo perf stat -a --per-core -C 0-127 -e power/energy-cores/
> 
> Output:
> S0-D0-C0           2               8.73 Joules power/energy-cores/
> S0-D0-C1           2               8.73 Joules power/energy-cores/
> S0-D0-C2           2               8.73 Joules power/energy-cores/
> S0-D0-C3           2               8.73 Joules power/energy-cores/
> S0-D0-C4           2               8.73 Joules power/energy-cores/
> 
> Signed-off-by: Wyes Karny <wyes.karny@amd.com>
> ---
>  arch/x86/events/rapl.c | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/x86/events/rapl.c b/arch/x86/events/rapl.c
> index 52e6e7ed4f78..d301bbbc3b93 100644
> --- a/arch/x86/events/rapl.c
> +++ b/arch/x86/events/rapl.c
> @@ -537,7 +537,7 @@ static struct perf_msr intel_rapl_spr_msrs[] = {
>   * - want to use same event codes across both architectures
>   */
>  static struct perf_msr amd_rapl_msrs[] = {
> -	[PERF_RAPL_PP0]  = { 0, &rapl_events_cores_group, 0, false, 0 },
> +	[PERF_RAPL_PP0]  = { MSR_AMD_CORE_ENERGY_STATUS, &rapl_events_cores_group, test_msr, false, RAPL_MSR_MASK },

Stephane, this was an oversight?

>  	[PERF_RAPL_PKG]  = { MSR_AMD_PKG_ENERGY_STATUS,  &rapl_events_pkg_group,   test_msr, false, RAPL_MSR_MASK },
>  	[PERF_RAPL_RAM]  = { 0, &rapl_events_ram_group,   0, false, 0 },
>  	[PERF_RAPL_PP1]  = { 0, &rapl_events_gpu_group,   0, false, 0 },
> @@ -764,7 +764,8 @@ static struct rapl_model model_spr = {
>  };
>  
>  static struct rapl_model model_amd_hygon = {
> -	.events		= BIT(PERF_RAPL_PKG),
> +	.events		= BIT(PERF_RAPL_PP0) |
> +			  BIT(PERF_RAPL_PKG),
>  	.msr_power_unit = MSR_AMD_RAPL_POWER_UNIT,
>  	.rapl_msrs      = amd_rapl_msrs,
>  };
> -- 
> 2.34.1
>

Stephane Eranian Feb. 20, 2023, 9:29 p.m. UTC | #3

On Mon, Feb 20, 2023 at 3:45 AM Peter Zijlstra <peterz@infradead.org> wrote:
>
> On Fri, Feb 17, 2023 at 04:13:54PM +0000, Wyes Karny wrote:
> > AMD processors support per-package and per-core energy monitoring
> > through RAPL counters which can be accessed by users running in
> > supervisor mode.
> >
> > Core RAPL counters gives power consumption information per core.  For
> > AMD processors the package level RAPL counter are already exposed to
> > perf. Expose the core level RAPL counters also.
> >
> > sudo perf stat -a --per-core -C 0-127 -e power/energy-cores/
> >
> > Output:
> > S0-D0-C0           2               8.73 Joules power/energy-cores/
> > S0-D0-C1           2               8.73 Joules power/energy-cores/
> > S0-D0-C2           2               8.73 Joules power/energy-cores/
> > S0-D0-C3           2               8.73 Joules power/energy-cores/
> > S0-D0-C4           2               8.73 Joules power/energy-cores/
> >
> > Signed-off-by: Wyes Karny <wyes.karny@amd.com>
> > ---
> >  arch/x86/events/rapl.c | 5 +++--
> >  1 file changed, 3 insertions(+), 2 deletions(-)
> >
> > diff --git a/arch/x86/events/rapl.c b/arch/x86/events/rapl.c
> > index 52e6e7ed4f78..d301bbbc3b93 100644
> > --- a/arch/x86/events/rapl.c
> > +++ b/arch/x86/events/rapl.c
> > @@ -537,7 +537,7 @@ static struct perf_msr intel_rapl_spr_msrs[] = {
> >   * - want to use same event codes across both architectures
> >   */
> >  static struct perf_msr amd_rapl_msrs[] = {
> > -     [PERF_RAPL_PP0]  = { 0, &rapl_events_cores_group, 0, false, 0 },
> > +     [PERF_RAPL_PP0]  = { MSR_AMD_CORE_ENERGY_STATUS, &rapl_events_cores_group, test_msr, false, RAPL_MSR_MASK },
>
> Stephane, this was an oversight?
>
I think it may depend on the CPU model. I remember it returning either
0 or bogus values on my systems. They may have improved that.
The commit msg does not show which CPU model this is run on.

>
> >       [PERF_RAPL_PKG]  = { MSR_AMD_PKG_ENERGY_STATUS,  &rapl_events_pkg_group,   test_msr, false, RAPL_MSR_MASK },
> >       [PERF_RAPL_RAM]  = { 0, &rapl_events_ram_group,   0, false, 0 },
> >       [PERF_RAPL_PP1]  = { 0, &rapl_events_gpu_group,   0, false, 0 },
> > @@ -764,7 +764,8 @@ static struct rapl_model model_spr = {
> >  };
> >
> >  static struct rapl_model model_amd_hygon = {
> > -     .events         = BIT(PERF_RAPL_PKG),
> > +     .events         = BIT(PERF_RAPL_PP0) |
> > +                       BIT(PERF_RAPL_PKG),
> >       .msr_power_unit = MSR_AMD_RAPL_POWER_UNIT,
> >       .rapl_msrs      = amd_rapl_msrs,
> >  };
> > --
> > 2.34.1
> >

Wyes Karny Feb. 21, 2023, 4:53 a.m. UTC | #4

On 20 Feb 13:29, Stephane Eranian wrote:
> On Mon, Feb 20, 2023 at 3:45 AM Peter Zijlstra <peterz@infradead.org> wrote:
> >
> > On Fri, Feb 17, 2023 at 04:13:54PM +0000, Wyes Karny wrote:
> > > AMD processors support per-package and per-core energy monitoring
> > > through RAPL counters which can be accessed by users running in
> > > supervisor mode.
> > >
> > > Core RAPL counters gives power consumption information per core.  For
> > > AMD processors the package level RAPL counter are already exposed to
> > > perf. Expose the core level RAPL counters also.
> > >
> > > sudo perf stat -a --per-core -C 0-127 -e power/energy-cores/
> > >
> > > Output:
> > > S0-D0-C0           2               8.73 Joules power/energy-cores/
> > > S0-D0-C1           2               8.73 Joules power/energy-cores/
> > > S0-D0-C2           2               8.73 Joules power/energy-cores/
> > > S0-D0-C3           2               8.73 Joules power/energy-cores/
> > > S0-D0-C4           2               8.73 Joules power/energy-cores/
> > >
> > > Signed-off-by: Wyes Karny <wyes.karny@amd.com>
> > > ---
> > >  arch/x86/events/rapl.c | 5 +++--
> > >  1 file changed, 3 insertions(+), 2 deletions(-)
> > >
> > > diff --git a/arch/x86/events/rapl.c b/arch/x86/events/rapl.c
> > > index 52e6e7ed4f78..d301bbbc3b93 100644
> > > --- a/arch/x86/events/rapl.c
> > > +++ b/arch/x86/events/rapl.c
> > > @@ -537,7 +537,7 @@ static struct perf_msr intel_rapl_spr_msrs[] = {
> > >   * - want to use same event codes across both architectures
> > >   */
> > >  static struct perf_msr amd_rapl_msrs[] = {
> > > -     [PERF_RAPL_PP0]  = { 0, &rapl_events_cores_group, 0, false, 0 },
> > > +     [PERF_RAPL_PP0]  = { MSR_AMD_CORE_ENERGY_STATUS, &rapl_events_cores_group, test_msr, false, RAPL_MSR_MASK },
> >
> > Stephane, this was an oversight?
> >
> I think it may depend on the CPU model. I remember it returning either
> 0 or bogus values on my systems. They may have improved that.
> The commit msg does not show which CPU model this is run on.

I've tested this on Zen 2, 3 and 4 server systems.

Thanks,
Wyes
> 
> >
> > >       [PERF_RAPL_PKG]  = { MSR_AMD_PKG_ENERGY_STATUS,  &rapl_events_pkg_group,   test_msr, false, RAPL_MSR_MASK },
> > >       [PERF_RAPL_RAM]  = { 0, &rapl_events_ram_group,   0, false, 0 },
> > >       [PERF_RAPL_PP1]  = { 0, &rapl_events_gpu_group,   0, false, 0 },
> > > @@ -764,7 +764,8 @@ static struct rapl_model model_spr = {
> > >  };
> > >
> > >  static struct rapl_model model_amd_hygon = {
> > > -     .events         = BIT(PERF_RAPL_PKG),
> > > +     .events         = BIT(PERF_RAPL_PP0) |
> > > +                       BIT(PERF_RAPL_PKG),
> > >       .msr_power_unit = MSR_AMD_RAPL_POWER_UNIT,
> > >       .rapl_msrs      = amd_rapl_msrs,
> > >  };
> > > --
> > > 2.34.1
> > >

Stephane Eranian Feb. 21, 2023, 8:50 a.m. UTC | #5

Hi,

Testing Wyes' patch for energy-cores on Zen3 server loaded with triad
bench on socket0:

$ perf stat --per-core -a -C0-63 -I 1000 -e
power/energy-cores/,power/energy-pkg/
#           time core               cpus             counts   unit events
     1.001019203 S0-D0-C0              1               1.28 Joules
power/energy-cores/
     1.001019203 S0-D0-C0              1             231.38 Joules
power/energy-pkg/
     1.001019203 S0-D0-C1              1   4,294,967,130.96 Joules
power/energy-cores/
     1.001019203 S0-D0-C1              1             231.38 Joules
power/energy-pkg/
     1.001019203 S0-D0-C2              1   4,294,967,126.23 Joules
power/energy-cores/
     1.001019203 S0-D0-C2              1             231.38 Joules
power/energy-pkg/
     1.001019203 S0-D0-C3              1   4,294,967,122.50 Joules
power/energy-cores/
     1.001019203 S0-D0-C3              1             231.38 Joules
power/energy-pkg/
     1.001019203 S0-D0-C4              1   4,294,967,129.92 Joules
power/energy-cores/
     1.001019203 S0-D0-C4              1             231.38 Joules
power/energy-pkg/
     1.001019203 S0-D0-C5              1   4,294,967,121.49 Joules
power/energy-cores/
     1.001019203 S0-D0-C5              1             231.39 Joules
power/energy-pkg/

I think the result of energy-cores is not reliable and I think that is
why I did not
include it in the patch.

Could also be a problem with the kernel code, but I don't know why it would only
impact energy-cores given energy-pkg looks reasonable here.


On Mon, Feb 20, 2023 at 8:53 PM Wyes Karny <wyes.karny@amd.com> wrote:
>
> On 20 Feb 13:29, Stephane Eranian wrote:
> > On Mon, Feb 20, 2023 at 3:45 AM Peter Zijlstra <peterz@infradead.org> wrote:
> > >
> > > On Fri, Feb 17, 2023 at 04:13:54PM +0000, Wyes Karny wrote:
> > > > AMD processors support per-package and per-core energy monitoring
> > > > through RAPL counters which can be accessed by users running in
> > > > supervisor mode.
> > > >
> > > > Core RAPL counters gives power consumption information per core.  For
> > > > AMD processors the package level RAPL counter are already exposed to
> > > > perf. Expose the core level RAPL counters also.
> > > >
> > > > sudo perf stat -a --per-core -C 0-127 -e power/energy-cores/
> > > >
> > > > Output:
> > > > S0-D0-C0           2               8.73 Joules power/energy-cores/
> > > > S0-D0-C1           2               8.73 Joules power/energy-cores/
> > > > S0-D0-C2           2               8.73 Joules power/energy-cores/
> > > > S0-D0-C3           2               8.73 Joules power/energy-cores/
> > > > S0-D0-C4           2               8.73 Joules power/energy-cores/
> > > >
> > > > Signed-off-by: Wyes Karny <wyes.karny@amd.com>
> > > > ---
> > > >  arch/x86/events/rapl.c | 5 +++--
> > > >  1 file changed, 3 insertions(+), 2 deletions(-)
> > > >
> > > > diff --git a/arch/x86/events/rapl.c b/arch/x86/events/rapl.c
> > > > index 52e6e7ed4f78..d301bbbc3b93 100644
> > > > --- a/arch/x86/events/rapl.c
> > > > +++ b/arch/x86/events/rapl.c
> > > > @@ -537,7 +537,7 @@ static struct perf_msr intel_rapl_spr_msrs[] = {
> > > >   * - want to use same event codes across both architectures
> > > >   */
> > > >  static struct perf_msr amd_rapl_msrs[] = {
> > > > -     [PERF_RAPL_PP0]  = { 0, &rapl_events_cores_group, 0, false, 0 },
> > > > +     [PERF_RAPL_PP0]  = { MSR_AMD_CORE_ENERGY_STATUS, &rapl_events_cores_group, test_msr, false, RAPL_MSR_MASK },
> > >
> > > Stephane, this was an oversight?
> > >
> > I think it may depend on the CPU model. I remember it returning either
> > 0 or bogus values on my systems. They may have improved that.
> > The commit msg does not show which CPU model this is run on.
>
> I've tested this on Zen 2, 3 and 4 server systems.
>
> Thanks,
> Wyes
> >
> > >
> > > >       [PERF_RAPL_PKG]  = { MSR_AMD_PKG_ENERGY_STATUS,  &rapl_events_pkg_group,   test_msr, false, RAPL_MSR_MASK },
> > > >       [PERF_RAPL_RAM]  = { 0, &rapl_events_ram_group,   0, false, 0 },
> > > >       [PERF_RAPL_PP1]  = { 0, &rapl_events_gpu_group,   0, false, 0 },
> > > > @@ -764,7 +764,8 @@ static struct rapl_model model_spr = {
> > > >  };
> > > >
> > > >  static struct rapl_model model_amd_hygon = {
> > > > -     .events         = BIT(PERF_RAPL_PKG),
> > > > +     .events         = BIT(PERF_RAPL_PP0) |
> > > > +                       BIT(PERF_RAPL_PKG),
> > > >       .msr_power_unit = MSR_AMD_RAPL_POWER_UNIT,
> > > >       .rapl_msrs      = amd_rapl_msrs,
> > > >  };
> > > > --
> > > > 2.34.1
> > > >

Muhammad Usama Anjum Feb. 28, 2023, 2:20 p.m. UTC | #6

On 2/21/23 1:50 PM, Stephane Eranian wrote:
> Hi,
> 
> Testing Wyes' patch for energy-cores on Zen3 server loaded with triad
> bench on socket0:
> 
> $ perf stat --per-core -a -C0-63 -I 1000 -e
> power/energy-cores/,power/energy-pkg/
> #           time core               cpus             counts   unit events
>      1.001019203 S0-D0-C0              1               1.28 Joules
> power/energy-cores/
>      1.001019203 S0-D0-C0              1             231.38 Joules
> power/energy-pkg/
>      1.001019203 S0-D0-C1              1   4,294,967,130.96 Joules
> power/energy-cores/
>      1.001019203 S0-D0-C1              1             231.38 Joules
> power/energy-pkg/
>      1.001019203 S0-D0-C2              1   4,294,967,126.23 Joules
> power/energy-cores/
>      1.001019203 S0-D0-C2              1             231.38 Joules
> power/energy-pkg/
>      1.001019203 S0-D0-C3              1   4,294,967,122.50 Joules
> power/energy-cores/
>      1.001019203 S0-D0-C3              1             231.38 Joules
> power/energy-pkg/
>      1.001019203 S0-D0-C4              1   4,294,967,129.92 Joules
> power/energy-cores/
>      1.001019203 S0-D0-C4              1             231.38 Joules
> power/energy-pkg/
>      1.001019203 S0-D0-C5              1   4,294,967,121.49 Joules
> power/energy-cores/
>      1.001019203 S0-D0-C5              1             231.39 Joules
> power/energy-pkg/
> 
> I think the result of energy-cores is not reliable and I think that is
> why I did not
> include it in the patch.
> 
> Could also be a problem with the kernel code, but I don't know why it would only
> impact energy-cores given energy-pkg looks reasonable here.
> 

I'm getting consistent per core energy measurements on AMD Custom APU 405.
Probably consistency depends on the chip.

sudo perf stat -a --per-core -C 0-7 -e power/energy-cores/

 Performance counter stats for 'system wide':

 S0-D0-C0           2               0.51 Joules power/energy-cores/
 S0-D0-C1           2               0.51 Joules power/energy-cores/
 S0-D0-C2           2               0.51 Joules power/energy-cores/
 S0-D0-C3           2               0.51 Joules power/energy-cores/

What is the fate of this patch now?

[..]

>>>> Stephane, this was an oversight?
>>>>
>>> I think it may depend on the CPU model. I remember it returning either
>>> 0 or bogus values on my systems. They may have improved that.
>>> The commit msg does not show which CPU model this is run on.
>>
>> I've tested this on Zen 2, 3 and 4 server systems.
>>
>> Thanks,
>> Wyes

Wyes Karny Feb. 28, 2023, 3:55 p.m. UTC | #7

Hi Muhammad,

On 28 Feb 19:20, Muhammad Usama Anjum wrote:
> On 2/21/23 1:50 PM, Stephane Eranian wrote:
> > Hi,
> > 
> > Testing Wyes' patch for energy-cores on Zen3 server loaded with triad
> > bench on socket0:
> > 
> > $ perf stat --per-core -a -C0-63 -I 1000 -e
> > power/energy-cores/,power/energy-pkg/
> > #           time core               cpus             counts   unit events
> >      1.001019203 S0-D0-C0              1               1.28 Joules
> > power/energy-cores/
> >      1.001019203 S0-D0-C0              1             231.38 Joules
> > power/energy-pkg/
> >      1.001019203 S0-D0-C1              1   4,294,967,130.96 Joules
> > power/energy-cores/
> >      1.001019203 S0-D0-C1              1             231.38 Joules
> > power/energy-pkg/
> >      1.001019203 S0-D0-C2              1   4,294,967,126.23 Joules
> > power/energy-cores/
> >      1.001019203 S0-D0-C2              1             231.38 Joules
> > power/energy-pkg/
> >      1.001019203 S0-D0-C3              1   4,294,967,122.50 Joules
> > power/energy-cores/
> >      1.001019203 S0-D0-C3              1             231.38 Joules
> > power/energy-pkg/
> >      1.001019203 S0-D0-C4              1   4,294,967,129.92 Joules
> > power/energy-cores/
> >      1.001019203 S0-D0-C4              1             231.38 Joules
> > power/energy-pkg/
> >      1.001019203 S0-D0-C5              1   4,294,967,121.49 Joules
> > power/energy-cores/
> >      1.001019203 S0-D0-C5              1             231.39 Joules
> > power/energy-pkg/
> > 
> > I think the result of energy-cores is not reliable and I think that is
> > why I did not
> > include it in the patch.
> > 
> > Could also be a problem with the kernel code, but I don't know why it would only
> > impact energy-cores given energy-pkg looks reasonable here.
> > 
> 
> I'm getting consistent per core energy measurements on AMD Custom APU 405.
> Probably consistency depends on the chip.
> 
> sudo perf stat -a --per-core -C 0-7 -e power/energy-cores/
> 
>  Performance counter stats for 'system wide':
> 
>  S0-D0-C0           2               0.51 Joules power/energy-cores/
>  S0-D0-C1           2               0.51 Joules power/energy-cores/
>  S0-D0-C2           2               0.51 Joules power/energy-cores/
>  S0-D0-C3           2               0.51 Joules power/energy-cores/
> 
> What is the fate of this patch now?

I'm working on the issue reported by Stephane. Will send v2 very soon.

Thanks,
Wyes
> 
> [..]
> 
> >>>> Stephane, this was an oversight?
> >>>>
> >>> I think it may depend on the CPU model. I remember it returning either
> >>> 0 or bogus values on my systems. They may have improved that.
> >>> The commit msg does not show which CPU model this is run on.
> >>
> >> I've tested this on Zen 2, 3 and 4 server systems.
> >>
> >> Thanks,
> >> Wyes
> 
> -- 
> BR,
> Muhammad Usama Anjum

Wyes Karny March 1, 2023, 6:27 p.m. UTC | #8

Hi Stephane,

On 2/21/2023 2:20 PM, Stephane Eranian wrote:
> Hi,
> 
> Testing Wyes' patch for energy-cores on Zen3 server loaded with triad
> bench on socket0:
> 
> $ perf stat --per-core -a -C0-63 -I 1000 -e
> power/energy-cores/,power/energy-pkg/
> #           time core               cpus             counts   unit events
>      1.001019203 S0-D0-C0              1               1.28 Joules
> power/energy-cores/
>      1.001019203 S0-D0-C0              1             231.38 Joules
> power/energy-pkg/
>      1.001019203 S0-D0-C1              1   4,294,967,130.96 Joules
> power/energy-cores/
>      1.001019203 S0-D0-C1              1             231.38 Joules
> power/energy-pkg/
>      1.001019203 S0-D0-C2              1   4,294,967,126.23 Joules
> power/energy-cores/
>      1.001019203 S0-D0-C2              1             231.38 Joules
> power/energy-pkg/
>      1.001019203 S0-D0-C3              1   4,294,967,122.50 Joules
> power/energy-cores/
>      1.001019203 S0-D0-C3              1             231.38 Joules
> power/energy-pkg/
>      1.001019203 S0-D0-C4              1   4,294,967,129.92 Joules
> power/energy-cores/
>      1.001019203 S0-D0-C4              1             231.38 Joules
> power/energy-pkg/
>      1.001019203 S0-D0-C5              1   4,294,967,121.49 Joules
> power/energy-cores/
>      1.001019203 S0-D0-C5              1             231.39 Joules
> power/energy-pkg/
> 
> I think the result of energy-cores is not reliable and I think that is
> why I did not
> include it in the patch.
> 
> Could also be a problem with the kernel code, but I don't know why it would only
> impact energy-cores given energy-pkg looks reasonable here.

This is a kernel issue. I've addressed this in v2.

Thanks,
Wyes
> 
> 
> On Mon, Feb 20, 2023 at 8:53 PM Wyes Karny <wyes.karny@amd.com> wrote:
>>
>> On 20 Feb 13:29, Stephane Eranian wrote:
>>> On Mon, Feb 20, 2023 at 3:45 AM Peter Zijlstra <peterz@infradead.org> wrote:
>>>>
>>>> On Fri, Feb 17, 2023 at 04:13:54PM +0000, Wyes Karny wrote:
>>>>> AMD processors support per-package and per-core energy monitoring
>>>>> through RAPL counters which can be accessed by users running in
>>>>> supervisor mode.
>>>>>
>>>>> Core RAPL counters gives power consumption information per core.  For
>>>>> AMD processors the package level RAPL counter are already exposed to
>>>>> perf. Expose the core level RAPL counters also.
>>>>>
>>>>> sudo perf stat -a --per-core -C 0-127 -e power/energy-cores/
>>>>>
>>>>> Output:
>>>>> S0-D0-C0           2               8.73 Joules power/energy-cores/
>>>>> S0-D0-C1           2               8.73 Joules power/energy-cores/
>>>>> S0-D0-C2           2               8.73 Joules power/energy-cores/
>>>>> S0-D0-C3           2               8.73 Joules power/energy-cores/
>>>>> S0-D0-C4           2               8.73 Joules power/energy-cores/
>>>>>
>>>>> Signed-off-by: Wyes Karny <wyes.karny@amd.com>
>>>>> ---
>>>>>  arch/x86/events/rapl.c | 5 +++--
>>>>>  1 file changed, 3 insertions(+), 2 deletions(-)
>>>>>
>>>>> diff --git a/arch/x86/events/rapl.c b/arch/x86/events/rapl.c
>>>>> index 52e6e7ed4f78..d301bbbc3b93 100644
>>>>> --- a/arch/x86/events/rapl.c
>>>>> +++ b/arch/x86/events/rapl.c
>>>>> @@ -537,7 +537,7 @@ static struct perf_msr intel_rapl_spr_msrs[] = {
>>>>>   * - want to use same event codes across both architectures
>>>>>   */
>>>>>  static struct perf_msr amd_rapl_msrs[] = {
>>>>> -     [PERF_RAPL_PP0]  = { 0, &rapl_events_cores_group, 0, false, 0 },
>>>>> +     [PERF_RAPL_PP0]  = { MSR_AMD_CORE_ENERGY_STATUS, &rapl_events_cores_group, test_msr, false, RAPL_MSR_MASK },
>>>>
>>>> Stephane, this was an oversight?
>>>>
>>> I think it may depend on the CPU model. I remember it returning either
>>> 0 or bogus values on my systems. They may have improved that.
>>> The commit msg does not show which CPU model this is run on.
>>
>> I've tested this on Zen 2, 3 and 4 server systems.
>>
>> Thanks,
>> Wyes
>>>
>>>>
>>>>>       [PERF_RAPL_PKG]  = { MSR_AMD_PKG_ENERGY_STATUS,  &rapl_events_pkg_group,   test_msr, false, RAPL_MSR_MASK },
>>>>>       [PERF_RAPL_RAM]  = { 0, &rapl_events_ram_group,   0, false, 0 },
>>>>>       [PERF_RAPL_PP1]  = { 0, &rapl_events_gpu_group,   0, false, 0 },
>>>>> @@ -764,7 +764,8 @@ static struct rapl_model model_spr = {
>>>>>  };
>>>>>
>>>>>  static struct rapl_model model_amd_hygon = {
>>>>> -     .events         = BIT(PERF_RAPL_PKG),
>>>>> +     .events         = BIT(PERF_RAPL_PP0) |
>>>>> +                       BIT(PERF_RAPL_PKG),
>>>>>       .msr_power_unit = MSR_AMD_RAPL_POWER_UNIT,
>>>>>       .rapl_msrs      = amd_rapl_msrs,
>>>>>  };
>>>>> --
>>>>> 2.34.1
>>>>>

Wyes Karny March 1, 2023, 6:32 p.m. UTC | #9

Hi Oleksandr,

On 2/19/2023 5:22 PM, Oleksandr Natalenko wrote:
> Hello.
> 
> On pátek 17. února 2023 17:13:54 CET Wyes Karny wrote:
>> AMD processors support per-package and per-core energy monitoring
>> through RAPL counters which can be accessed by users running in
>> supervisor mode.
>>
>> Core RAPL counters gives power consumption information per core.  For
>> AMD processors the package level RAPL counter are already exposed to
>> perf. Expose the core level RAPL counters also.
>>
>> sudo perf stat -a --per-core -C 0-127 -e power/energy-cores/
>>
>> Output:
>> S0-D0-C0           2               8.73 Joules power/energy-cores/
>> S0-D0-C1           2               8.73 Joules power/energy-cores/
>> S0-D0-C2           2               8.73 Joules power/energy-cores/
>> S0-D0-C3           2               8.73 Joules power/energy-cores/
>> S0-D0-C4           2               8.73 Joules power/energy-cores/
>>
>> Signed-off-by: Wyes Karny <wyes.karny@amd.com>
>> ---
>>  arch/x86/events/rapl.c | 5 +++--
>>  1 file changed, 3 insertions(+), 2 deletions(-)
>>
>> diff --git a/arch/x86/events/rapl.c b/arch/x86/events/rapl.c
>> index 52e6e7ed4f78..d301bbbc3b93 100644
>> --- a/arch/x86/events/rapl.c
>> +++ b/arch/x86/events/rapl.c
>> @@ -537,7 +537,7 @@ static struct perf_msr intel_rapl_spr_msrs[] = {
>>   * - want to use same event codes across both architectures
>>   */
>>  static struct perf_msr amd_rapl_msrs[] = {
>> -	[PERF_RAPL_PP0]  = { 0, &rapl_events_cores_group, 0, false, 0 },
>> +	[PERF_RAPL_PP0]  = { MSR_AMD_CORE_ENERGY_STATUS, &rapl_events_cores_group, test_msr, false, RAPL_MSR_MASK },
>>  	[PERF_RAPL_PKG]  = { MSR_AMD_PKG_ENERGY_STATUS,  &rapl_events_pkg_group,   test_msr, false, RAPL_MSR_MASK },
>>  	[PERF_RAPL_RAM]  = { 0, &rapl_events_ram_group,   0, false, 0 },
>>  	[PERF_RAPL_PP1]  = { 0, &rapl_events_gpu_group,   0, false, 0 },
>> @@ -764,7 +764,8 @@ static struct rapl_model model_spr = {
>>  };
>>  
>>  static struct rapl_model model_amd_hygon = {
>> -	.events		= BIT(PERF_RAPL_PKG),
>> +	.events		= BIT(PERF_RAPL_PP0) |
>> +			  BIT(PERF_RAPL_PKG),
>>  	.msr_power_unit = MSR_AMD_RAPL_POWER_UNIT,
>>  	.rapl_msrs      = amd_rapl_msrs,
>>  };
>>
> 
> With this patch:
> 
> ```
> $ lscpu | grep 'Model name'
> Model name:                      AMD Ryzen 9 5950X 16-Core Processor
> 
> $ sudo perf stat -a --per-core -C 0-15 -e power/energy-cores/ -- dd if=/dev/zero of=/dev/null bs=1M count=100000
> 100000+0 records in
> 100000+0 records out
> 104857600000 bytes (105 GB, 98 GiB) copied, 1,59252 s, 65,8 GB/s
> 
> Performance counter stats for 'system wide':
> 
> S0-D0-C0           1               1,56 Joules power/energy-cores/
> S0-D0-C1           1               1,56 Joules power/energy-cores/
> S0-D0-C2           1               1,56 Joules power/energy-cores/
> S0-D0-C3           1               1,56 Joules power/energy-cores/
> S0-D0-C4           1               1,56 Joules power/energy-cores/
> S0-D0-C5           1               1,56 Joules power/energy-cores/
> S0-D0-C6           1               1,56 Joules power/energy-cores/
> S0-D0-C7           1               1,56 Joules power/energy-cores/
> S0-D0-C8           1               1,56 Joules power/energy-cores/
> S0-D0-C9           1               1,56 Joules power/energy-cores/
> S0-D0-C10          1               1,56 Joules power/energy-cores/
> S0-D0-C11          1               1,56 Joules power/energy-cores/
> S0-D0-C12          1               1,56 Joules power/energy-cores/
> S0-D0-C13          1               1,56 Joules power/energy-cores/
> S0-D0-C14          1               1,56 Joules power/energy-cores/
> S0-D0-C15          1               1,56 Joules power/energy-cores/
> 
> 1,593982452 seconds time elapsed
> ```
> 
> Hence,
> 
> Tested-by: Oleksandr Natalenko <oleksandr@natalenko.name>

I haven't put your tested-by tag in v2 as I've added a new patch there.
Please let me know if v2 works well for you and if I can add tested-by on v2.
 
> 
> Thank you.
>

diff mbox series

Patch

diff --git a/arch/x86/events/rapl.c b/arch/x86/events/rapl.c
index 52e6e7ed4f78..d301bbbc3b93 100644
--- a/arch/x86/events/rapl.c
+++ b/arch/x86/events/rapl.c
@@ -537,7 +537,7 @@  static struct perf_msr intel_rapl_spr_msrs[] = {
  * - want to use same event codes across both architectures
  */
 static struct perf_msr amd_rapl_msrs[] = {
-	[PERF_RAPL_PP0]  = { 0, &rapl_events_cores_group, 0, false, 0 },
+	[PERF_RAPL_PP0]  = { MSR_AMD_CORE_ENERGY_STATUS, &rapl_events_cores_group, test_msr, false, RAPL_MSR_MASK },
 	[PERF_RAPL_PKG]  = { MSR_AMD_PKG_ENERGY_STATUS,  &rapl_events_pkg_group,   test_msr, false, RAPL_MSR_MASK },
 	[PERF_RAPL_RAM]  = { 0, &rapl_events_ram_group,   0, false, 0 },
 	[PERF_RAPL_PP1]  = { 0, &rapl_events_gpu_group,   0, false, 0 },
@@ -764,7 +764,8 @@  static struct rapl_model model_spr = {
 };
 
 static struct rapl_model model_amd_hygon = {
-	.events		= BIT(PERF_RAPL_PKG),
+	.events		= BIT(PERF_RAPL_PP0) |
+			  BIT(PERF_RAPL_PKG),
 	.msr_power_unit = MSR_AMD_RAPL_POWER_UNIT,
 	.rapl_msrs      = amd_rapl_msrs,
 };