[v4,3/3] cpufreq: Return failure if fast_switch is not set and fast_switch_possible is set

Message ID 20230517162817.8538-4-wyes.karny@amd.com
State New
Headers
Series cpufreq/amd-pstate: Fix null pointer dereference when frequency invariance is disabled |

Commit Message

Wyes Karny May 17, 2023, 4:28 p.m. UTC
  If fast_switch_possible flag is set by the scaling driver, the governor
is free to select fast_switch function even if adjust_perf is set.  Some
scaling drivers which use adjust_perf don't set fast_switch thinking
that the governor would never fall back to fast_switch. But the governor
can fall back to fast_switch even in runtime if frequency invariance is
disabled due to some reason. This could crash the kernel if the driver
didn't set the fast_switch function pointer.

Therefore, return failure in cpufreq_online function if fast_switch is
not set and fast_switch_possible is set.

Signed-off-by: Wyes Karny <wyes.karny@amd.com>
---
 drivers/cpufreq/cpufreq.c | 5 +++++
 include/linux/cpufreq.h   | 4 +++-
 2 files changed, 8 insertions(+), 1 deletion(-)
  

Comments

kernel test robot May 22, 2023, 8:28 a.m. UTC | #1
Hello,

kernel test robot noticed a 302.3% regression of phoronix-test-suite.glibc-bench.sin.nanoseconds on:


commit: 278dfbd17c5d0767834471d7fea71d3127c3ecfb ("[PATCH v4 3/3] cpufreq: Return failure if fast_switch is not set and fast_switch_possible is set")
url: https://github.com/intel-lab-lkp/linux/commits/Wyes-Karny/amd_pstate-Add-fast_switch-callback/20230518-010448
base: https://git.kernel.org/cgit/linux/kernel/git/rafael/linux-pm.git linux-next
patch link: https://lore.kernel.org/all/20230517162817.8538-4-wyes.karny@amd.com/
patch subject: [PATCH v4 3/3] cpufreq: Return failure if fast_switch is not set and fast_switch_possible is set

testcase: phoronix-test-suite
test machine: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory
parameters:

	test: glibc-bench-1.6.1
	option_a: sin
	cpufreq_governor: performance

test-description: The Phoronix Test Suite is the most comprehensive testing and benchmarking platform available that provides an extensible framework for which new tests can be easily added.
test-url: http://www.phoronix-test-suite.com/

In addition to that, the commit also has significant impact on the following tests:

+------------------+-----------------------------------------------------------------------------------------------------------------------------------------+
| testcase: change | phoronix-test-suite: phoronix-test-suite.n-queens.0.seconds 313.6% regression                                                           |
| test machine     | 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory                                           |
| test parameters  | cpufreq_governor=performance                                                                                                            |
|                  | test=n-queens-1.2.1                                                                                                                     |
+------------------+-----------------------------------------------------------------------------------------------------------------------------------------+
| testcase: change | filebench: filebench.sum_operations/s -72.1% regression                                                                                 |
| test machine     | 96 threads 2 sockets (Ice Lake) with 256G memory                                                                                        |
| test parameters  | cpufreq_governor=performance                                                                                                            |
|                  | disk=1HDD                                                                                                                               |
|                  | fs=xfs                                                                                                                                  |
|                  | test=cvar_example.f                                                                                                                     |
+------------------+-----------------------------------------------------------------------------------------------------------------------------------------+
| testcase: change | phoronix-test-suite: phoronix-test-suite.glibc-bench.ffs.nanoseconds 301.2% regression                                                  |
| test machine     | 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory                                           |
| test parameters  | cpufreq_governor=performance                                                                                                            |
|                  | option_a=ffs                                                                                                                            |
|                  | test=glibc-bench-1.6.1                                                                                                                  |
+------------------+-----------------------------------------------------------------------------------------------------------------------------------------+
| testcase: change | phoronix-test-suite: phoronix-test-suite.glibc-bench.pthread_once.nanoseconds 300.8% regression                                                                                   |
| test machine     | 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory                                           |
| test parameters  | cpufreq_governor=performance                                                                                                            |
|                  | option_a=pthread_once                                                                                                                   |
|                  | test=glibc-bench-1.6.1                                                                                                                  |
+------------------+-----------------------------------------------------------------------------------------------------------------------------------------+
| testcase: change | phoronix-test-suite: phoronix-test-suite.cyclictest.clock_nanosleepTIME_ABSTIME,Interval500ms,100000Loops.ms_average 5550.0% regression |
| test machine     | 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory                                           |
| test parameters  | cpufreq_governor=performance                                                                                                            |
|                  | option_a=3                                                                                                                              |
|                  | test=cyclictest-1.0.0                                                                                                                   |
+------------------+-----------------------------------------------------------------------------------------------------------------------------------------+
| testcase: change | phoronix-test-suite: phoronix-test-suite.astcenc.Medium.seconds 106.6% regression                                                       |
| test machine     | 16 threads 1 sockets Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz (Coffee Lake) with 32G memory                                               |
| test parameters  | cpufreq_governor=performance                                                                                                            |
|                  | option_a=Medium                                                                                                                         |
|                  | test=astcenc-1.1.0                                                                                                                      |
+------------------+-----------------------------------------------------------------------------------------------------------------------------------------+
| testcase: change | phoronix-test-suite: phoronix-test-suite.pyperformance.pickle_pure_python.milliseconds -99.4% regression                                |
| test machine     | 16 threads 1 sockets Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz (Coffee Lake) with 32G memory                                               |
| test parameters  | cpufreq_governor=performance                                                                                                            |
|                  | option_a=pickle_pure_python                                                                                                             |
|                  | test=pyperformance-1.0.2                                                                                                                |
+------------------+-----------------------------------------------------------------------------------------------------------------------------------------+
| testcase: change | phoronix-test-suite: phoronix-test-suite.x11perf.500pxCopyFromWindowToWindow.operations___second -7.3% regression                       |
| test machine     | 12 threads 1 sockets Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz (Coffee Lake) with 32G memory                                              |
| test parameters  | cpufreq_governor=performance                                                                                                            |
|                  | need_x=true                                                                                                                             |
|                  | option_a=500px Copy From Window To Window                                                                                               |
|                  | test=x11perf-1.1.1                                                                                                                      |
+------------------+-----------------------------------------------------------------------------------------------------------------------------------------+


If you fix the issue, kindly add following tag
| Reported-by: kernel test robot <oliver.sang@intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202305221612.200d8fb1-oliver.sang@intel.com


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        sudo bin/lkp install job.yaml           # job file is attached in this email
        bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
        sudo bin/lkp run generated-yaml-file

        # if come across any failure that blocks the test,
        # please remove ~/.lkp and /lkp dir to run from a clean state.

=========================================================================================
compiler/cpufreq_governor/kconfig/option_a/rootfs/tbox_group/test/testcase:
  gcc-11/performance/x86_64-rhel-8.3/sin/debian-x86_64-phoronix/lkp-csl-2sp7/glibc-bench-1.6.1/phoronix-test-suite

commit: 
  a525b95417 ("cpufreq/amd-pstate: Remove fast_switch_possible flag from active driver")
  278dfbd17c ("cpufreq: Return failure if fast_switch is not set and fast_switch_possible is set")

a525b954173e74cd 278dfbd17c5d0767834471d7fea 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :6          100%           6:6     kmsg.cpufreq:fast_switch_possible_is_enabled_but_fast_switch_callback_is_not_set
         %stddev     %change         %stddev
             \          |                \  
     72.00          +302.3%     289.69        phoronix-test-suite.glibc-bench.sin.nanoseconds
     86.29           +14.5%      98.84        phoronix-test-suite.time.elapsed_time
     86.29           +14.5%      98.84        phoronix-test-suite.time.elapsed_time.max
    307.00 ± 15%    +397.0%       1525 ±  3%  phoronix-test-suite.time.involuntary_context_switches
     74.33            +5.6%      78.50        phoronix-test-suite.time.percent_of_cpu_this_job_got
     61.97           +12.2%      69.52        phoronix-test-suite.time.user_time
      8020 ±  3%     +32.1%      10593        meminfo.Active(anon)
 8.199e+09            +9.8%  9.003e+09        cpuidle..time
   8533649           +12.1%    9564836        cpuidle..usage
     80285 ±  5%     -39.2%      48852 ± 28%  numa-meminfo.node0.AnonHugePages
     14184 ± 93%    +360.9%      65375 ± 24%  numa-meminfo.node1.AnonHugePages
    144.48 ±  5%     +13.8%     164.46        uptime.boot
     12935 ±  5%     +11.6%      14439        uptime.idle
      1.00          +100.0%       2.00        vmstat.procs.r
      2346          +108.3%       4888        vmstat.system.cs
     95898            -2.3%      93730        vmstat.system.in
      0.04 ±  8%      -0.0        0.03 ±  4%  mpstat.cpu.all.iowait%
      1.64            +2.0        3.59        mpstat.cpu.all.irq%
      0.09 ±  3%      +0.1        0.21 ±  2%  mpstat.cpu.all.soft%
      0.09            +0.2        0.28        mpstat.cpu.all.sys%
      0.90            +0.8        1.68        mpstat.cpu.all.usr%
      2004 ±  3%     +32.1%       2648        proc-vmstat.nr_active_anon
    110098            -4.0%     105659        proc-vmstat.nr_anon_pages
    286491            -1.6%     281968        proc-vmstat.nr_inactive_anon
     52653            -1.1%      52049        proc-vmstat.nr_slab_unreclaimable
      2004 ±  3%     +32.1%       2648        proc-vmstat.nr_zone_active_anon
    286491            -1.6%     281968        proc-vmstat.nr_zone_inactive_anon
    513152            +8.7%     557738        proc-vmstat.numa_hit
    494207            +9.0%     538756        proc-vmstat.numa_local
    592220            +9.6%     649301        proc-vmstat.pgalloc_normal
    681984 ±  2%     +10.6%     754452        proc-vmstat.pgfault
    492147 ±  2%     +10.6%     544510        proc-vmstat.pgfree
     89356 ±  2%     +16.7%     104272        proc-vmstat.pgreuse
    835328           +13.6%     948992        proc-vmstat.unevictable_pgs_scanned
      4.17            +4.7        8.90        turbostat.Busy%
      1607           -56.4%     700.00        turbostat.Bzy_MHz
     59849 ± 12%    +264.1%     217899 ±  7%  turbostat.C1E
      0.23 ±  4%      +0.1        0.30 ±  2%  turbostat.C1E%
   8348392           +10.9%    9254503        turbostat.C6
     71.41           -21.0%      56.44        turbostat.CPU%c1
     24.42 ±  2%     +41.9%      34.65        turbostat.CPU%c6
     51.50 ±  3%     -25.9%      38.17        turbostat.CoreTmp
   8545229           +11.7%    9542613        turbostat.IRQ
     35868 ±  6%     -96.2%       1375 ± 38%  turbostat.POLL
     18.83 ±  5%     -57.2%       8.06 ±  3%  turbostat.Pkg%pc2
      2.57 ±  7%     -74.2%       0.66 ±  4%  turbostat.Pkg%pc6
     51.67 ±  2%     -25.5%      38.50        turbostat.PkgTmp
    121.11           -42.2%      70.00        turbostat.PkgWatt
     53.52 ± 18%     -16.0       37.52 ± 22%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     53.52 ± 18%     -16.0       37.52 ± 22%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.74 ±103%      -8.3        2.42 ±155%  perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.__mmput.exit_mm.do_exit
     10.74 ±103%      -8.3        2.42 ±155%  perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.__mmput.exit_mm
      8.15 ±116%      -6.9        1.26 ±145%  perf-profile.calltrace.cycles-pp.asm_exc_page_fault.fault_in_readable.fault_in_iov_iter_readable.generic_perform_write.__generic_file_write_iter
     10.74 ±103%      -6.6        4.09 ±178%  perf-profile.calltrace.cycles-pp.zap_pmd_range.unmap_page_range.unmap_vmas.exit_mmap.__mmput
     10.74 ±103%      -6.6        4.09 ±178%  perf-profile.calltrace.cycles-pp.zap_pte_range.zap_pmd_range.unmap_page_range.unmap_vmas.exit_mmap
      4.26 ±105%      -4.3        0.00        perf-profile.calltrace.cycles-pp.page_remove_rmap.zap_pte_range.zap_pmd_range.unmap_page_range.unmap_vmas
      4.26 ± 84%      -3.8        0.50 ±223%  perf-profile.calltrace.cycles-pp.__do_fault.do_read_fault.do_fault.__handle_mm_fault.handle_mm_fault
      5.19 ± 85%      -3.0        2.17 ±169%  perf-profile.calltrace.cycles-pp.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault
      5.19 ± 85%      -3.0        2.17 ±169%  perf-profile.calltrace.cycles-pp.do_read_fault.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
     53.52 ± 18%     -16.0       37.52 ± 22%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     53.52 ± 18%     -16.0       37.52 ± 22%  perf-profile.children.cycles-pp.do_syscall_64
     10.74 ±103%      -6.6        4.09 ±178%  perf-profile.children.cycles-pp.unmap_vmas
     10.74 ±103%      -6.6        4.09 ±178%  perf-profile.children.cycles-pp.unmap_page_range
     10.74 ±103%      -6.6        4.09 ±178%  perf-profile.children.cycles-pp.zap_pmd_range
     10.74 ±103%      -6.6        4.09 ±178%  perf-profile.children.cycles-pp.zap_pte_range
      4.26 ±105%      -4.3        0.00        perf-profile.children.cycles-pp.page_remove_rmap
      4.26 ± 84%      -3.8        0.50 ±223%  perf-profile.children.cycles-pp.__do_fault
      5.18 ± 85%      -3.0        2.17 ±169%  perf-profile.children.cycles-pp.do_fault
      5.18 ± 85%      -3.0        2.17 ±169%  perf-profile.children.cycles-pp.do_read_fault
      2862 ± 20%   +5653.9%     164728 ±213%  sched_debug.cfs_rq:/.load.avg
     47107 ± 16%  +32534.6%   15373401 ±219%  sched_debug.cfs_rq:/.load.max
      9468 ± 12%  +16445.2%    1566546 ±218%  sched_debug.cfs_rq:/.load.stddev
     58.57 ± 33%     -58.9%      24.09 ± 46%  sched_debug.cfs_rq:/.load_avg.avg
    203.27 ± 18%     -53.0%      95.47 ± 50%  sched_debug.cfs_rq:/.load_avg.stddev
     37231 ± 14%     +51.4%      56368 ±  5%  sched_debug.cfs_rq:/.min_vruntime.max
     39.58 ± 51%     -77.9%       8.76 ±126%  sched_debug.cfs_rq:/.removed.load_avg.avg
    187.92 ± 25%     -66.1%      63.64 ±106%  sched_debug.cfs_rq:/.removed.load_avg.stddev
     18.16 ± 57%     -77.1%       4.16 ±123%  sched_debug.cfs_rq:/.removed.runnable_avg.avg
     18.16 ± 57%     -77.1%       4.16 ±123%  sched_debug.cfs_rq:/.removed.util_avg.avg
    214.61 ± 14%     -37.7%     133.71 ±  6%  sched_debug.cfs_rq:/.runnable_avg.avg
    248.38 ±  6%     -17.4%     205.07 ±  4%  sched_debug.cfs_rq:/.runnable_avg.stddev
    214.53 ± 14%     -37.7%     133.59 ±  6%  sched_debug.cfs_rq:/.util_avg.avg
    248.36 ±  6%     -17.4%     205.03 ±  4%  sched_debug.cfs_rq:/.util_avg.stddev
    715.50 ±  6%     -41.8%     416.50 ± 25%  sched_debug.cfs_rq:/.util_est_enqueued.max
     99.21 ±  6%     -31.7%      67.78 ± 24%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
   2382316 ± 46%     -49.6%    1201480 ± 12%  sched_debug.cpu.avg_idle.max
     36009           +21.1%      43615 ±  2%  sched_debug.cpu.clock_task.min
    117475 ± 75%     -88.0%      14136 ± 82%  sched_debug.cpu.max_idle_balance_cost.stddev
      0.00 ± 20%    +130.0%       0.00 ± 63%  sched_debug.cpu.next_balance.stddev
     52309           +15.6%      60475        sched_debug.sched_clk
     25.76 ±  4%     +48.2%      38.19        perf-stat.i.MPKI
 7.905e+08           -45.2%   4.33e+08        perf-stat.i.branch-instructions
      3.18 ±  2%      +2.8        5.96        perf-stat.i.branch-miss-rate%
  15210221           +64.8%   25060061        perf-stat.i.branch-misses
   2046023           +68.1%    3438514        perf-stat.i.cache-misses
  49258879           +69.3%   83372455        perf-stat.i.cache-references
      1997          +130.4%       4601        perf-stat.i.context-switches
      2.07 ±  2%     +12.5%       2.32        perf-stat.i.cpi
 5.568e+09            -6.3%  5.218e+09        perf-stat.i.cpu-cycles
      3074           -48.9%       1571        perf-stat.i.cycles-between-cache-misses
      0.31 ±  5%      +0.2        0.52        perf-stat.i.dTLB-load-miss-rate%
   1469231           +70.0%    2497301        perf-stat.i.dTLB-load-misses
 1.117e+09           -52.2%  5.343e+08        perf-stat.i.dTLB-loads
      0.06 ±  5%      +0.0        0.11        perf-stat.i.dTLB-store-miss-rate%
    140944 ±  3%     +63.8%     230910        perf-stat.i.dTLB-store-misses
 4.405e+08           -46.6%  2.354e+08        perf-stat.i.dTLB-stores
     61.61           +12.2       73.84        perf-stat.i.iTLB-load-miss-rate%
    799474            -3.9%     768539        perf-stat.i.iTLB-load-misses
    499584 ±  2%     -45.3%     273344 ±  3%  perf-stat.i.iTLB-loads
 5.589e+09           -55.2%  2.504e+09        perf-stat.i.instructions
      7134           -54.0%       3281        perf-stat.i.instructions-per-iTLB-miss
      1.00           -52.6%       0.47        perf-stat.i.ipc
      2.07 ±  4%     -58.4%       0.86 ±  6%  perf-stat.i.major-faults
      0.06            -6.3%       0.05        perf-stat.i.metric.GHz
    618.20           +40.6%     869.20        perf-stat.i.metric.K/sec
     24.35           -48.5%      12.53        perf-stat.i.metric.M/sec
     83.55            -3.9       79.68        perf-stat.i.node-load-miss-rate%
     92330 ±  2%     +36.8%     126293 ±  3%  perf-stat.i.node-load-misses
     38082 ±  2%     +17.1%      44601 ±  3%  perf-stat.i.node-loads
     42624 ± 14%     +41.2%      60176 ±  9%  perf-stat.i.node-store-misses
      8.82          +277.4%      33.29        perf-stat.overall.MPKI
      1.92            +3.9        5.79        perf-stat.overall.branch-miss-rate%
      1.00          +109.0%       2.08        perf-stat.overall.cpi
      2722 ±  2%     -44.3%       1517        perf-stat.overall.cycles-between-cache-misses
      0.13            +0.3        0.47        perf-stat.overall.dTLB-load-miss-rate%
      0.03 ±  3%      +0.1        0.10        perf-stat.overall.dTLB-store-miss-rate%
     61.55           +12.2       73.77        perf-stat.overall.iTLB-load-miss-rate%
      6987           -53.4%       3258        perf-stat.overall.instructions-per-iTLB-miss
      1.00           -52.2%       0.48        perf-stat.overall.ipc
     70.79            +3.1       73.87        perf-stat.overall.node-load-miss-rate%
     44.13 ± 10%      +7.7       51.80 ±  8%  perf-stat.overall.node-store-miss-rate%
  7.81e+08           -45.1%  4.286e+08        perf-stat.ps.branch-instructions
  15031718           +65.0%   24799436        perf-stat.ps.branch-misses
   2022310           +68.3%    3402668        perf-stat.ps.cache-misses
  48693838           +69.4%   82502207        perf-stat.ps.cache-references
      1974          +130.7%       4554        perf-stat.ps.context-switches
 5.503e+09            -6.2%  5.164e+09        perf-stat.ps.cpu-cycles
   1452373           +70.1%    2471118        perf-stat.ps.dTLB-load-misses
 1.103e+09           -52.1%  5.288e+08        perf-stat.ps.dTLB-loads
    139332 ±  3%     +64.0%     228525        perf-stat.ps.dTLB-store-misses
 4.353e+08           -46.5%   2.33e+08        perf-stat.ps.dTLB-stores
    790174            -3.7%     760575        perf-stat.ps.iTLB-load-misses
    493698 ±  2%     -45.2%     270481 ±  3%  perf-stat.ps.iTLB-loads
 5.521e+09           -55.1%  2.479e+09        perf-stat.ps.instructions
      2.04 ±  4%     -58.2%       0.85 ±  6%  perf-stat.ps.major-faults
     91256 ±  2%     +37.0%     125023 ±  3%  perf-stat.ps.node-load-misses
     37653 ±  2%     +17.3%      44171 ±  3%  perf-stat.ps.node-loads
     42127 ± 14%     +41.4%      59560 ±  9%  perf-stat.ps.node-store-misses
 4.781e+11           -49.8%  2.401e+11        perf-stat.total.instructions


***************************************************************************************************
lkp-csl-2sp7: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/rootfs/tbox_group/test/testcase:
  gcc-11/performance/x86_64-rhel-8.3/debian-x86_64-phoronix/lkp-csl-2sp7/n-queens-1.2.1/phoronix-test-suite

commit: 
  a525b95417 ("cpufreq/amd-pstate: Remove fast_switch_possible flag from active driver")
  278dfbd17c ("cpufreq: Return failure if fast_switch is not set and fast_switch_possible is set")

a525b954173e74cd 278dfbd17c5d0767834471d7fea 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :6          100%           6:6     kmsg.cpufreq:fast_switch_possible_is_enabled_but_fast_switch_callback_is_not_set
         %stddev     %change         %stddev
             \          |                \  
      4.14          +313.6%      17.10        phoronix-test-suite.n-queens.0.seconds
     35.10 ±  2%    +128.7%      80.28        phoronix-test-suite.time.elapsed_time
     35.10 ±  2%    +128.7%      80.28        phoronix-test-suite.time.elapsed_time.max
     28850            -3.0%      27994 ±  2%  phoronix-test-suite.time.maximum_resident_set_size
  2.27e+09 ±  4%     +22.0%  2.769e+09        cpuidle..time
   2439080 ±  3%     +16.5%    2841419        cpuidle..usage
      7803 ±  2%     +30.3%      10166        meminfo.Active(anon)
   2128118           +13.6%    2416662        meminfo.Committed_AS
      3883 ±  8%     -16.6%       3240 ±  9%  numa-meminfo.node0.PageTables
      9073 ±  5%     +15.6%      10489 ±  2%  numa-meminfo.node1.KernelStack
      1929 ± 14%     +49.6%       2885 ±  9%  numa-meminfo.node1.PageTables
    971.33 ±  8%     -16.6%     810.00 ±  9%  numa-vmstat.node0.nr_page_table_pages
      9074 ±  5%     +15.7%      10494 ±  3%  numa-vmstat.node1.nr_kernel_stack
    482.00 ± 14%     +49.8%     721.83 ±  9%  numa-vmstat.node1.nr_page_table_pages
     63.98           -28.9       35.05        mpstat.cpu.all.idle%
      0.09 ±  5%      -0.1        0.04        mpstat.cpu.all.iowait%
      1.64            +1.2        2.84        mpstat.cpu.all.irq%
      0.09 ±  4%      +0.0        0.11        mpstat.cpu.all.soft%
      0.13 ±  2%      +0.2        0.29        mpstat.cpu.all.sys%
     34.08 ±  2%     +27.6       61.67        mpstat.cpu.all.usr%
     65.33           -44.9%      36.00        vmstat.cpu.id
     31.67 ±  2%     +86.8%      59.17        vmstat.cpu.us
      2345 ±  3%     -54.4%       1069 ±  3%  vmstat.io.bi
     31.00           +94.1%      60.17        vmstat.procs.r
      2937           +67.8%       4927        vmstat.system.cs
     96735            -2.6%      94264        vmstat.system.in
    980.50 ±  2%     -53.7%     454.00        turbostat.Avg_MHz
     37.06 ±  2%     +28.7       65.74        turbostat.Busy%
      2671           -73.8%     700.00        turbostat.Bzy_MHz
     64543 ± 21%     -70.0%      19387 ±  7%  turbostat.C1
      0.05 ± 22%      -0.0        0.01        turbostat.C1%
     36674 ± 10%    +197.1%     108951 ± 11%  turbostat.C1E
      0.30 ±  6%      -0.1        0.17 ±  9%  turbostat.C1E%
   2305719 ±  4%     +16.6%    2687774        turbostat.C6
     63.05           -28.7       34.39        turbostat.C6%
     43.38           -53.5%      20.16        turbostat.CPU%c1
     19.56           -27.9%      14.11        turbostat.CPU%c6
     51.00 ±  3%     -24.2%      38.67        turbostat.CoreTmp
   3714942 ±  2%    +113.2%    7921708        turbostat.IRQ
     13286 ±  2%     -96.2%     509.50 ± 24%  turbostat.POLL
     10.25           -67.9%       3.30 ± 11%  turbostat.Pkg%pc2
      5.70 ±  5%     -97.2%       0.16 ±  5%  turbostat.Pkg%pc6
     50.83 ±  4%     -23.0%      39.17        turbostat.PkgTmp
    165.02           -49.5%      83.38        turbostat.PkgWatt
    139.21            +3.3%     143.74        turbostat.RAMWatt
      1950 ±  2%     +30.2%       2539        proc-vmstat.nr_active_anon
     26828            +3.9%      27867        proc-vmstat.nr_active_file
    107483            -4.1%     103038        proc-vmstat.nr_anon_pages
    283850            -1.6%     279280        proc-vmstat.nr_inactive_anon
     20564 ±  2%      +3.8%      21344        proc-vmstat.nr_kernel_stack
      1452            +5.1%       1526        proc-vmstat.nr_page_table_pages
     25654            +1.0%      25919        proc-vmstat.nr_slab_reclaimable
      1950 ±  2%     +30.2%       2539        proc-vmstat.nr_zone_active_anon
     26828            +3.9%      27867        proc-vmstat.nr_zone_active_file
    283850            -1.6%     279280        proc-vmstat.nr_zone_inactive_anon
    390716           +25.2%     489291        proc-vmstat.numa_hit
    371740           +26.5%     470277        proc-vmstat.numa_local
      6451 ± 56%     -90.3%     628.33 ± 59%  proc-vmstat.numa_pages_migrated
     21981 ± 24%     -75.9%       5294 ± 67%  proc-vmstat.numa_pte_updates
    461444           +24.3%     573395        proc-vmstat.pgalloc_normal
    484259           +33.7%     647486        proc-vmstat.pgfault
    361816           +29.2%     467634        proc-vmstat.pgfree
      6451 ± 56%     -90.3%     628.33 ± 59%  proc-vmstat.pgmigrate_success
     58553           +50.8%      88315 ±  2%  proc-vmstat.pgreuse
    335104          +102.6%     678912        proc-vmstat.unevictable_pgs_scanned
      2379 ± 34%    +159.9%       6184 ± 32%  sched_debug.cfs_rq:/.load.avg
      8773 ± 34%    +156.3%      22488 ± 53%  sched_debug.cfs_rq:/.load.stddev
     59.18 ± 31%     -59.2%      24.17 ± 32%  sched_debug.cfs_rq:/.load_avg.avg
      1020 ±  3%     -19.2%     824.58 ± 25%  sched_debug.cfs_rq:/.load_avg.max
    201.36 ± 21%     -49.2%     102.31 ± 30%  sched_debug.cfs_rq:/.load_avg.stddev
      4306 ± 17%   +1283.4%      59568 ±128%  sched_debug.cfs_rq:/.min_vruntime.stddev
     39.17 ± 50%     -80.4%       7.68 ± 95%  sched_debug.cfs_rq:/.removed.load_avg.avg
    185.53 ± 28%     -65.2%      64.59 ± 83%  sched_debug.cfs_rq:/.removed.load_avg.stddev
     17.61 ± 45%     -78.3%       3.82 ± 96%  sched_debug.cfs_rq:/.removed.runnable_avg.avg
     86.06 ± 23%     -62.7%      32.11 ± 84%  sched_debug.cfs_rq:/.removed.runnable_avg.stddev
     17.61 ± 45%     -78.3%       3.82 ± 96%  sched_debug.cfs_rq:/.removed.util_avg.avg
     86.06 ± 23%     -62.7%      32.11 ± 84%  sched_debug.cfs_rq:/.removed.util_avg.stddev
    225.01 ± 11%     -44.8%     124.22 ± 24%  sched_debug.cfs_rq:/.runnable_avg.avg
    253.39 ±  5%     -21.5%     198.91 ± 13%  sched_debug.cfs_rq:/.runnable_avg.stddev
      4306 ± 17%   +1283.1%      59563 ±128%  sched_debug.cfs_rq:/.spread0.stddev
    224.86 ± 11%     -44.8%     124.11 ± 24%  sched_debug.cfs_rq:/.util_avg.avg
    253.32 ±  5%     -21.5%     198.83 ± 13%  sched_debug.cfs_rq:/.util_avg.stddev
    783.83 ± 14%     -42.9%     447.67 ± 49%  sched_debug.cfs_rq:/.util_est_enqueued.max
     96.58 ± 11%     -32.0%      65.67 ± 33%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
      3.45 ± 29%    +151.3%       8.67 ±  5%  sched_debug.cpu.clock.stddev
     36256           +46.5%      53120 ± 26%  sched_debug.cpu.clock_task.min
    140.27 ± 23%     +57.5%     220.89 ±  8%  sched_debug.cpu.curr->pid.avg
    525.18 ± 13%     +27.1%     667.27 ±  3%  sched_debug.cpu.curr->pid.stddev
      0.00 ±  9%     +85.4%       0.00 ± 17%  sched_debug.cpu.next_balance.stddev
      0.06 ± 21%     +59.5%       0.10 ± 13%  sched_debug.cpu.nr_running.avg
     52700           +32.7%      69957 ± 20%  sched_debug.sched_clk
     14.93 ± 37%     -14.9        0.00        perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
     14.93 ± 37%     -14.9        0.00        perf-profile.calltrace.cycles-pp.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      7.40 ± 56%      -7.4        0.00        perf-profile.calltrace.cycles-pp.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      7.40 ± 56%      -7.4        0.00        perf-profile.calltrace.cycles-pp.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64
      7.40 ± 56%      -7.4        0.00        perf-profile.calltrace.cycles-pp.search_binary_handler.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve
      7.40 ± 56%      -7.4        0.00        perf-profile.calltrace.cycles-pp.load_elf_binary.search_binary_handler.exec_binprm.bprm_execve.do_execveat_common
      5.05 ±105%      -5.0        0.00        perf-profile.calltrace.cycles-pp.__mmput.exec_mmap.begin_new_exec.load_elf_binary.search_binary_handler
      5.05 ±105%      -5.0        0.00        perf-profile.calltrace.cycles-pp.exit_mmap.__mmput.exec_mmap.begin_new_exec.load_elf_binary
      5.05 ±105%      -5.0        0.00        perf-profile.calltrace.cycles-pp.begin_new_exec.load_elf_binary.search_binary_handler.exec_binprm.bprm_execve
      5.05 ±105%      -5.0        0.00        perf-profile.calltrace.cycles-pp.exec_mmap.begin_new_exec.load_elf_binary.search_binary_handler.exec_binprm
      5.05 ±105%      -5.0        0.00        perf-profile.calltrace.cycles-pp.free_pgtables.exit_mmap.__mmput.exec_mmap.begin_new_exec
      0.00            +6.2        6.21 ± 52%  perf-profile.calltrace.cycles-pp.perf_release.__fput.task_work_run.do_exit.do_group_exit
      0.00            +6.2        6.21 ± 52%  perf-profile.calltrace.cycles-pp.perf_event_release_kernel.perf_release.__fput.task_work_run.do_exit
      0.00            +6.9        6.90 ± 46%  perf-profile.calltrace.cycles-pp.task_work_run.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart
      0.00            +6.9        6.90 ± 46%  perf-profile.calltrace.cycles-pp.__fput.task_work_run.do_exit.do_group_exit.get_signal
     14.93 ± 37%     -14.9        0.00        perf-profile.children.cycles-pp.__x64_sys_execve
     14.93 ± 37%     -14.9        0.00        perf-profile.children.cycles-pp.do_execveat_common
      7.40 ± 56%      -7.4        0.00        perf-profile.children.cycles-pp.bprm_execve
      7.40 ± 56%      -7.4        0.00        perf-profile.children.cycles-pp.exec_binprm
      7.40 ± 56%      -7.4        0.00        perf-profile.children.cycles-pp.search_binary_handler
      7.40 ± 56%      -7.4        0.00        perf-profile.children.cycles-pp.load_elf_binary
      5.05 ±105%      -5.0        0.00        perf-profile.children.cycles-pp.begin_new_exec
      5.05 ±105%      -5.0        0.00        perf-profile.children.cycles-pp.exec_mmap
      5.88 ± 82%      -4.8        1.11 ±223%  perf-profile.children.cycles-pp.free_pgtables
      0.00            +6.2        6.21 ± 52%  perf-profile.children.cycles-pp.perf_release
      0.00            +6.2        6.21 ± 52%  perf-profile.children.cycles-pp.perf_event_release_kernel
      0.83 ±223%      +6.9        7.74 ± 41%  perf-profile.children.cycles-pp.task_work_run
      0.83 ±223%      +6.9        7.74 ± 41%  perf-profile.children.cycles-pp.__fput
      6.71 ± 75%      -6.0        0.70 ±223%  perf-profile.self.cycles-pp.zap_pte_range
     43.58 ±  5%     -71.0%      12.62 ±  5%  perf-stat.i.MPKI
 8.111e+09 ±  2%     -54.1%  3.721e+09        perf-stat.i.branch-instructions
     14.40            +1.9       16.33        perf-stat.i.branch-miss-rate%
 1.658e+09 ±  2%     -55.4%  7.401e+08        perf-stat.i.branch-misses
      7.61            +2.2        9.79 ±  4%  perf-stat.i.cache-miss-rate%
   2374185           -18.4%    1936172 ±  2%  perf-stat.i.cache-misses
  45065640           -25.1%   33764433        perf-stat.i.cache-references
      2358           +97.5%       4657        perf-stat.i.context-switches
      3.44 ±  3%     -58.6%       1.42 ±  2%  perf-stat.i.cpi
 9.928e+10           -54.8%  4.488e+10        perf-stat.i.cpu-cycles
    125.68 ±  2%     +12.2%     141.01        perf-stat.i.cpu-migrations
    157340 ±  3%     -56.5%      68500 ±  5%  perf-stat.i.cycles-between-cache-misses
      0.47 ±  5%      -0.3        0.17 ±  4%  perf-stat.i.dTLB-load-miss-rate%
   1300850           -25.3%     972044        perf-stat.i.dTLB-load-misses
 1.072e+10 ±  2%     -54.6%  4.867e+09        perf-stat.i.dTLB-loads
      0.08 ±  5%      -0.0        0.04 ±  2%  perf-stat.i.dTLB-store-miss-rate%
    200676            +9.5%     219785        perf-stat.i.dTLB-store-misses
 1.048e+10 ±  2%     -55.3%  4.684e+09        perf-stat.i.dTLB-stores
     77.21            -6.7       70.51        perf-stat.i.iTLB-load-miss-rate%
    835438 ±  7%     -34.9%     544154 ±  3%  perf-stat.i.iTLB-load-misses
    281802           -22.1%     219530        perf-stat.i.iTLB-loads
 1.023e+11 ±  2%     -55.3%  4.576e+10        perf-stat.i.instructions
      0.70 ±  2%     +21.8%       0.85        perf-stat.i.ipc
      5.05 ±  3%     -78.3%       1.10 ±  5%  perf-stat.i.major-faults
      1.03           -54.9%       0.47        perf-stat.i.metric.GHz
    667.47 ±  3%     -46.4%     357.69        perf-stat.i.metric.K/sec
    305.23 ±  2%     -54.7%     138.14        perf-stat.i.metric.M/sec
      6910           -28.6%       4935        perf-stat.i.minor-faults
    143333 ±  4%     -11.2%     127226 ±  3%  perf-stat.i.node-load-misses
     70783 ±  6%     -34.1%      46661 ±  4%  perf-stat.i.node-loads
     59.90 ±  4%      +7.5       67.39        perf-stat.i.node-store-miss-rate%
     78522 ± 14%     -46.0%      42369 ±  3%  perf-stat.i.node-stores
      6915           -28.6%       4936        perf-stat.i.page-faults
      0.44 ±  3%     +60.7%       0.71        perf-stat.overall.MPKI
     20.44            -0.5       19.92        perf-stat.overall.branch-miss-rate%
      5.27 ±  2%      +0.5        5.76 ±  2%  perf-stat.overall.cache-miss-rate%
      0.97            +1.0%       0.98        perf-stat.overall.cpi
     41833 ±  2%     -42.6%      24012 ±  3%  perf-stat.overall.cycles-between-cache-misses
      0.01 ±  3%      +0.0        0.02        perf-stat.overall.dTLB-load-miss-rate%
      0.00 ±  2%      +0.0        0.00        perf-stat.overall.dTLB-store-miss-rate%
     74.71            -3.5       71.17        perf-stat.overall.iTLB-load-miss-rate%
    123027 ±  5%     -29.9%      86229 ±  3%  perf-stat.overall.instructions-per-iTLB-miss
      1.03            -1.0%       1.02        perf-stat.overall.ipc
     66.94 ±  2%      +6.4       73.36        perf-stat.overall.node-load-miss-rate%
     51.71 ±  8%     +11.2       62.86        perf-stat.overall.node-store-miss-rate%
 7.879e+09           -52.8%  3.722e+09        perf-stat.ps.branch-instructions
  1.61e+09           -54.0%  7.415e+08        perf-stat.ps.branch-misses
   2305884           -18.8%    1872089 ±  2%  perf-stat.ps.cache-misses
  43800806           -25.8%   32479193        perf-stat.ps.cache-references
      2291          +100.8%       4600        perf-stat.ps.context-switches
     93316            +1.6%      94785        perf-stat.ps.cpu-clock
 9.645e+10           -53.4%  4.491e+10        perf-stat.ps.cpu-cycles
    122.10           +14.4%     139.72        perf-stat.ps.cpu-migrations
   1264505           -26.1%     934112        perf-stat.ps.dTLB-load-misses
 1.041e+10           -53.2%  4.871e+09        perf-stat.ps.dTLB-loads
    194993           +11.1%     216552        perf-stat.ps.dTLB-store-misses
 1.018e+10           -53.9%  4.692e+09        perf-stat.ps.dTLB-stores
    811464 ±  7%     -34.4%     532304 ±  3%  perf-stat.ps.iTLB-load-misses
    273639           -21.3%     215398        perf-stat.ps.iTLB-loads
  9.94e+10           -53.9%  4.583e+10        perf-stat.ps.instructions
      4.88 ±  3%     -78.6%       1.05 ±  4%  perf-stat.ps.major-faults
      6706           -28.0%       4827        perf-stat.ps.minor-faults
    139125 ±  4%     -10.6%     124398 ±  3%  perf-stat.ps.node-load-misses
     68666 ±  6%     -34.2%      45167 ±  4%  perf-stat.ps.node-loads
     76159 ± 14%     -46.0%      41093 ±  3%  perf-stat.ps.node-stores
      6711           -28.1%       4828        perf-stat.ps.page-faults
     93316            +1.6%      94785        perf-stat.ps.task-clock
 3.538e+12            +1.6%  3.593e+12        perf-stat.total.instructions



***************************************************************************************************
lkp-icl-2sp1: 96 threads 2 sockets (Ice Lake) with 256G memory
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/rootfs/tbox_group/test/testcase:
  gcc-11/performance/1HDD/xfs/x86_64-rhel-8.3/debian-11.1-x86_64-20220510.cgz/lkp-icl-2sp1/cvar_example.f/filebench

commit: 
  a525b95417 ("cpufreq/amd-pstate: Remove fast_switch_possible flag from active driver")
  278dfbd17c ("cpufreq: Return failure if fast_switch is not set and fast_switch_possible is set")

a525b954173e74cd 278dfbd17c5d0767834471d7fea 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :6          100%           6:6     kmsg.cpufreq:fast_switch_possible_is_enabled_but_fast_switch_callback_is_not_set
         %stddev     %change         %stddev
             \          |                \  
      2831           -72.1%     788.78        filebench.sum_bytes_mb/s
  87006971           -72.1%   24234692        filebench.sum_operations
   1450022           -72.1%     403880        filebench.sum_operations/s
    483340           -72.1%     134626        filebench.sum_reads/s
      0.00          +100.0%       0.00        filebench.sum_time_ms/op
    283.33 ± 97%   +2342.1%       6919 ± 13%  filebench.time.involuntary_context_switches
     14383            +2.1%      14680        filebench.time.minor_page_faults
     93.00            -1.1%      92.00        filebench.time.percent_of_cpu_this_job_got
     72.15 ± 13%    +120.2%     158.88 ± 40%  boot-time.boot
      6397 ± 13%    +119.9%      14065 ± 39%  boot-time.idle
     96.77            -1.7%      95.12        iostat.cpu.idle
      2.79           +49.2%       4.17        iostat.cpu.system
     31818 ± 69%    +201.5%      95941 ±  8%  numa-meminfo.node1.AnonHugePages
    196985 ± 43%     +62.8%     320714 ± 20%  numa-meminfo.node1.Inactive
    141.31 ±  6%     +61.8%     228.60 ± 28%  uptime.boot
     12823 ±  6%     +59.4%      20437 ± 27%  uptime.idle
      2294 ±  2%    +142.4%       5562        vmstat.system.cs
     94416            -1.8%      92687        vmstat.system.in
      1697           +88.1%       3193        meminfo.Active
      1688           +88.7%       3185        meminfo.Active(anon)
     74749 ± 10%     +45.0%     108373 ±  4%  meminfo.AnonHugePages
   8341845 ± 11%     +28.7%   10738346 ± 19%  meminfo.DirectMap2M
      2.05            +1.2        3.27        mpstat.cpu.all.irq%
      0.12 ±  3%      +0.0        0.17 ±  6%  mpstat.cpu.all.soft%
      0.63            +0.1        0.76        mpstat.cpu.all.sys%
      0.42            +0.3        0.70        mpstat.cpu.all.usr%
      2.52 ± 51%      +2.9        5.39 ± 26%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      1.86 ± 76%      +3.5        5.39 ± 26%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      1.62 ± 74%      +3.8        5.39 ± 26%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      1.62 ± 74%      +3.8        5.39 ± 26%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      0.14 ±223%      +6.1        6.24 ± 46%  perf-profile.calltrace.cycles-pp.show_interrupts.seq_read_iter.proc_reg_read_iter.vfs_read.ksys_read
      0.38 ±164%      +6.8        7.16 ± 60%  perf-profile.calltrace.cycles-pp.proc_reg_read_iter.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.24 ±144%      +6.9        7.16 ± 60%  perf-profile.calltrace.cycles-pp.seq_read_iter.proc_reg_read_iter.vfs_read.ksys_read.do_syscall_64
      2.52 ± 51%      +2.9        5.39 ± 26%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      1.86 ± 76%      +3.5        5.39 ± 26%  perf-profile.children.cycles-pp.tick_sched_timer
      1.62 ± 74%      +3.8        5.39 ± 26%  perf-profile.children.cycles-pp.tick_sched_handle
      1.62 ± 74%      +3.8        5.39 ± 26%  perf-profile.children.cycles-pp.update_process_times
      0.14 ±223%      +5.7        5.83 ± 54%  perf-profile.children.cycles-pp.show_interrupts
      0.38 ±164%      +6.8        7.16 ± 60%  perf-profile.children.cycles-pp.proc_reg_read_iter
    421.67           +88.9%     796.33        proc-vmstat.nr_active_anon
     18437            +2.3%      18864        proc-vmstat.nr_kernel_stack
     11616            +3.4%      12017        proc-vmstat.nr_shmem
     55935            -2.0%      54811        proc-vmstat.nr_slab_unreclaimable
    421.67           +88.9%     796.33        proc-vmstat.nr_zone_active_anon
      2418           +12.1%       2710 ±  9%  proc-vmstat.numa_hint_faults_local
   1116500 ±  4%     -54.4%     509544 ±  2%  proc-vmstat.numa_hit
   1029620 ±  4%     -59.0%     422532 ±  2%  proc-vmstat.numa_local
   1837630 ±  5%     -66.1%     623575 ±  3%  proc-vmstat.pgalloc_normal
    308419            +4.3%     321665        proc-vmstat.pgfault
   1648136 ±  6%     -74.1%     427598 ±  5%  proc-vmstat.pgfree
     13210           +10.0%      14531        proc-vmstat.pgreuse
    510506           +11.8%     570880        proc-vmstat.unevictable_pgs_scanned
     85.17           -26.8%      62.33        turbostat.Avg_MHz
      4.09            +2.2        6.29        turbostat.Busy%
      2077           -51.9%       1000        turbostat.Bzy_MHz
      7434 ±  5%    +986.2%      80757 ±  2%  turbostat.C1
     29350 ±  3%    +127.1%      66640 ±  2%  turbostat.C1E
      3.51 ±  7%     +51.0%       5.30 ±  5%  turbostat.CPU%c1
     53.50           -15.3%      45.33 ±  5%  turbostat.CoreTmp
     16996 ±  5%     -96.8%     545.33 ± 30%  turbostat.POLL
     36.83           -16.8%      30.63 ±  3%  turbostat.Pkg%pc2
      1.58 ±  8%     -25.8%       1.17 ±  2%  turbostat.Pkg%pc6
     54.17           -13.5%      46.83 ±  4%  turbostat.PkgTmp
    209.90           -41.0%     123.86        turbostat.PkgWatt
    162.53           -10.2%     145.96        turbostat.RAMWatt
    100.43 ± 20%     -71.1%      28.98 ± 21%  sched_debug.cfs_rq:/.load_avg.avg
    272.22 ± 10%     -55.7%     120.65 ± 19%  sched_debug.cfs_rq:/.load_avg.stddev
     21531 ± 21%     -40.9%      12715 ± 19%  sched_debug.cfs_rq:/.min_vruntime.avg
     61.41 ± 33%     -83.9%       9.91 ± 54%  sched_debug.cfs_rq:/.removed.load_avg.avg
    237.67 ± 15%     -63.1%      87.63 ± 46%  sched_debug.cfs_rq:/.removed.load_avg.stddev
     24.37 ± 40%     -83.6%       3.99 ± 73%  sched_debug.cfs_rq:/.removed.runnable_avg.avg
    528.83           -41.3%     310.17 ± 56%  sched_debug.cfs_rq:/.removed.runnable_avg.max
     98.29 ± 21%     -65.0%      34.42 ± 61%  sched_debug.cfs_rq:/.removed.runnable_avg.stddev
     24.37 ± 40%     -83.6%       3.99 ± 73%  sched_debug.cfs_rq:/.removed.util_avg.avg
    528.83           -41.3%     310.17 ± 56%  sched_debug.cfs_rq:/.removed.util_avg.max
     98.29 ± 21%     -65.0%      34.42 ± 61%  sched_debug.cfs_rq:/.removed.util_avg.stddev
    311.92 ±  4%     -29.3%     220.52 ±  4%  sched_debug.cfs_rq:/.runnable_avg.avg
      1445 ± 11%     -29.1%       1024        sched_debug.cfs_rq:/.runnable_avg.max
    337.24 ±  7%     -26.1%     249.36 ±  3%  sched_debug.cfs_rq:/.runnable_avg.stddev
    311.31 ±  4%     -29.2%     220.40 ±  4%  sched_debug.cfs_rq:/.util_avg.avg
      1444 ± 11%     -29.1%       1024        sched_debug.cfs_rq:/.util_avg.max
    336.63 ±  7%     -25.9%     249.28 ±  3%  sched_debug.cfs_rq:/.util_avg.stddev
    502.50 ±  4%     +57.5%     791.33        sched_debug.cfs_rq:/.util_est_enqueued.max
      2321 ± 18%    +196.4%       6880 ±  6%  sched_debug.cpu.avg_idle.min
     78576 ± 12%    +111.1%     165880 ± 38%  sched_debug.cpu.clock.avg
     78581 ± 12%    +111.1%     165890 ± 38%  sched_debug.cpu.clock.max
     78570 ± 12%    +111.1%     165870 ± 38%  sched_debug.cpu.clock.min
      2.89 ± 26%     +96.7%       5.68 ±  5%  sched_debug.cpu.clock.stddev
     77698 ± 11%    +109.1%     162504 ± 38%  sched_debug.cpu.clock_task.avg
     78440 ± 12%    +110.4%     165076 ± 38%  sched_debug.cpu.clock_task.max
     48018           +54.5%      74192 ± 10%  sched_debug.cpu.clock_task.min
     78574 ± 12%    +111.1%     165870 ± 38%  sched_debug.cpu_clk
     77364 ± 12%    +112.8%     164658 ± 39%  sched_debug.ktime
     64011           +42.5%      91223 ±  9%  sched_debug.sched_clk
     12.22 ±  2%    +157.3%      31.43        perf-stat.i.MPKI
 1.623e+09           -60.9%   6.35e+08        perf-stat.i.branch-instructions
      1.03 ±  4%      +1.4        2.47 ±  2%  perf-stat.i.branch-miss-rate%
      0.58 ±  2%      +0.1        0.68 ±  5%  perf-stat.i.cache-miss-rate%
    562035 ±  2%     +11.1%     624178 ±  4%  perf-stat.i.cache-misses
  94757587 ±  2%      -5.8%   89223187        perf-stat.i.cache-references
      1864 ±  2%    +184.1%       5298        perf-stat.i.context-switches
      0.87           +79.6%       1.57        perf-stat.i.cpi
 6.768e+09           -33.0%  4.536e+09        perf-stat.i.cpu-cycles
    105.39            -2.3%     102.97        perf-stat.i.cpu-migrations
     17826 ±  4%     -45.7%       9673 ±  6%  perf-stat.i.cycles-between-cache-misses
      0.13            +0.2        0.35        perf-stat.i.dTLB-load-miss-rate%
   2476399            +3.2%    2556155        perf-stat.i.dTLB-load-misses
 1.953e+09           -60.6%  7.691e+08        perf-stat.i.dTLB-loads
      0.02 ±  2%      +0.0        0.06 ±  5%  perf-stat.i.dTLB-store-miss-rate%
  1.16e+09           -63.3%  4.259e+08        perf-stat.i.dTLB-stores
  7.77e+09           -60.4%   3.08e+09        perf-stat.i.instructions
      1.15           -42.5%       0.66        perf-stat.i.ipc
      0.09 ± 68%     -93.8%       0.01 ±141%  perf-stat.i.major-faults
      0.07           -32.9%       0.05        perf-stat.i.metric.GHz
     49.68           -61.6%      19.08        perf-stat.i.metric.M/sec
    107291 ±  7%     +31.6%     141162 ±  5%  perf-stat.i.node-load-misses
     12.20 ±  2%    +137.6%      28.98        perf-stat.overall.MPKI
      1.02 ±  4%      +1.4        2.47 ±  2%  perf-stat.overall.branch-miss-rate%
      0.59 ±  2%      +0.1        0.70 ±  4%  perf-stat.overall.cache-miss-rate%
      0.87           +69.1%       1.47        perf-stat.overall.cpi
     12075 ±  2%     -39.7%       7285 ±  4%  perf-stat.overall.cycles-between-cache-misses
      0.13            +0.2        0.33 ±  2%  perf-stat.overall.dTLB-load-miss-rate%
      0.02 ±  2%      +0.0        0.06 ±  5%  perf-stat.overall.dTLB-store-miss-rate%
      1.15           -40.9%       0.68        perf-stat.overall.ipc
 1.597e+09           -60.9%  6.242e+08        perf-stat.ps.branch-instructions
    551866 ±  2%     +11.2%     613416 ±  4%  perf-stat.ps.cache-misses
  93226966 ±  2%      -5.9%   87736015        perf-stat.ps.cache-references
      1834 ±  2%    +184.1%       5212        perf-stat.ps.context-switches
 6.659e+09           -33.0%   4.46e+09        perf-stat.ps.cpu-cycles
    103.65            -2.4%     101.20        perf-stat.ps.cpu-migrations
   2436431            +3.2%    2513516        perf-stat.ps.dTLB-load-misses
 1.922e+09           -60.7%   7.56e+08        perf-stat.ps.dTLB-loads
 1.141e+09           -63.3%  4.188e+08        perf-stat.ps.dTLB-stores
 7.644e+09           -60.4%  3.028e+09        perf-stat.ps.instructions
      0.09 ± 68%     -93.8%       0.01 ±141%  perf-stat.ps.major-faults
    105526 ±  7%     +31.5%     138766 ±  5%  perf-stat.ps.node-load-misses
 4.798e+11           -61.5%   1.85e+11        perf-stat.total.instructions



***************************************************************************************************
lkp-csl-2sp7: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/option_a/rootfs/tbox_group/test/testcase:
  gcc-11/performance/x86_64-rhel-8.3/ffs/debian-x86_64-phoronix/lkp-csl-2sp7/glibc-bench-1.6.1/phoronix-test-suite

commit: 
  a525b95417 ("cpufreq/amd-pstate: Remove fast_switch_possible flag from active driver")
  278dfbd17c ("cpufreq: Return failure if fast_switch is not set and fast_switch_possible is set")

a525b954173e74cd 278dfbd17c5d0767834471d7fea 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :6          100%           6:6     kmsg.cpufreq:fast_switch_possible_is_enabled_but_fast_switch_callback_is_not_set
         %stddev     %change         %stddev
             \          |                \  
      2.57          +301.2%      10.31        phoronix-test-suite.glibc-bench.ffs.nanoseconds
     55.10           +23.5%      68.06        phoronix-test-suite.time.elapsed_time
     55.10           +23.5%      68.06        phoronix-test-suite.time.elapsed_time.max
    300.00 ± 14%    +398.7%       1496 ±  2%  phoronix-test-suite.time.involuntary_context_switches
     62.33           +12.0%      69.83        phoronix-test-suite.time.percent_of_cpu_this_job_got
      5.40          +110.3%      11.36        phoronix-test-suite.time.system_time
     29.22           +24.6%      36.42        phoronix-test-suite.time.user_time
 5.284e+09           +17.9%  6.228e+09        cpuidle..time
   5539887           +18.5%    6564240        cpuidle..usage
      7874           +26.6%       9970        meminfo.Active(anon)
    525422 ±  7%     -16.3%     439748 ± 11%  meminfo.DirectMap4k
    114.59 ±  4%     +19.2%     136.55 ±  7%  uptime.boot
     10125 ±  4%     +16.8%      11822 ±  6%  uptime.idle
      1414 ±  5%     -16.7%       1177 ±  6%  vmstat.io.bi
      1.00          +100.0%       2.00        vmstat.procs.r
      2577           +93.3%       4983        vmstat.system.cs
     95670            -3.3%      92477        vmstat.system.in
      0.06 ± 13%      -0.0        0.05 ± 18%  mpstat.cpu.all.iowait%
      1.73            +1.8        3.56        mpstat.cpu.all.irq%
      0.10 ±  5%      +0.1        0.21        mpstat.cpu.all.soft%
      0.18            +0.2        0.40        mpstat.cpu.all.sys%
      0.79            +1.1        1.87        mpstat.cpu.all.usr%
    155.96 ± 32%     -28.9%     110.84 ± 34%  sched_debug.cfs_rq:/.load_avg.stddev
     32174 ± 20%     -24.5%      24289 ±  9%  sched_debug.cfs_rq:/.min_vruntime.avg
     45718 ± 11%     +25.8%      57491 ±  4%  sched_debug.cfs_rq:/.min_vruntime.max
    244.32 ± 11%     -18.3%     199.54 ±  8%  sched_debug.cfs_rq:/.runnable_avg.stddev
     18560 ± 27%     +88.6%      34998 ±  7%  sched_debug.cfs_rq:/.spread0.max
    244.33 ± 11%     -18.3%     199.53 ±  8%  sched_debug.cfs_rq:/.util_avg.stddev
      3.37 ± 26%    +161.7%       8.82 ±  4%  sched_debug.cpu.clock.stddev
     35302           +20.5%      42534        sched_debug.cpu.clock_task.min
    149.18 ± 36%     +74.1%     259.73 ± 31%  sched_debug.cpu.curr->pid.avg
    530.50 ± 18%     +29.1%     684.71 ± 13%  sched_debug.cpu.curr->pid.stddev
     51821           +14.8%      59506        sched_debug.sched_clk
      1968           +26.6%       2492        proc-vmstat.nr_active_anon
    108779            -6.0%     102231        proc-vmstat.nr_anon_pages
    285189            -2.4%     278339        proc-vmstat.nr_inactive_anon
     52438            -1.6%      51613        proc-vmstat.nr_slab_unreclaimable
      1968           +26.6%       2492        proc-vmstat.nr_zone_active_anon
    285189            -2.4%     278339        proc-vmstat.nr_zone_inactive_anon
    445853            +9.3%     487223        proc-vmstat.numa_hit
    426867            +9.7%     468279        proc-vmstat.numa_local
    513203            +9.1%     560135        proc-vmstat.pgalloc_normal
    570191           +12.7%     642729        proc-vmstat.pgfault
    419473            +9.4%     458954        proc-vmstat.pgfree
     69487           +20.8%      83936        proc-vmstat.pgreuse
    595200           +21.5%     722944        proc-vmstat.unevictable_pgs_scanned
      4.38            +4.7        9.07        turbostat.Busy%
      1570           -55.4%     700.00        turbostat.Bzy_MHz
     75921 ± 10%     -37.6%      47362 ±  2%  turbostat.C1
      0.04 ± 21%      -0.0        0.02        turbostat.C1%
     56181 ± 20%    +172.3%     153009 ±  9%  turbostat.C1E
   5377888           +18.0%    6343412        turbostat.C6
     70.60           -20.2%      56.35        turbostat.CPU%c1
     25.02 ±  2%     +38.2%      34.58        turbostat.CPU%c6
     47.67           -21.3%      37.50        turbostat.CoreTmp
   5554932           +18.2%    6565797        turbostat.IRQ
     25793           -96.1%       1008 ± 14%  turbostat.POLL
     17.91 ±  4%     -70.0%       5.38 ±  4%  turbostat.Pkg%pc2
      3.66 ±  4%     -94.8%       0.19 ±  3%  turbostat.Pkg%pc6
     47.83 ±  2%     -20.6%      38.00        turbostat.PkgTmp
    116.41           -40.0%      69.85        turbostat.PkgWatt
    141.47            +1.4%     143.42        turbostat.RAMWatt
     18.20 ± 15%     -11.1        7.12 ± 56%  perf-profile.calltrace.cycles-pp.generic_file_write_iter.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     18.20 ± 15%     -11.1        7.12 ± 56%  perf-profile.calltrace.cycles-pp.__generic_file_write_iter.generic_file_write_iter.vfs_write.ksys_write.do_syscall_64
     18.20 ± 15%     -11.1        7.12 ± 56%  perf-profile.calltrace.cycles-pp.generic_perform_write.__generic_file_write_iter.generic_file_write_iter.vfs_write.ksys_write
     18.20 ± 15%     -10.3        7.91 ± 60%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     18.20 ± 15%     -10.3        7.91 ± 60%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.24 ±107%      -4.4        0.79 ±223%  perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.24 ±107%      -4.4        0.79 ±223%  perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.__x64_sys_openat.do_syscall_64
      3.86 ±100%      -3.9        0.00        perf-profile.calltrace.cycles-pp.__do_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.24 ±107%      -3.8        1.39 ±143%  perf-profile.calltrace.cycles-pp.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.24 ±107%      -3.8        1.39 ±143%  perf-profile.calltrace.cycles-pp.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.53 ± 72%      -2.8        2.75 ±159%  perf-profile.calltrace.cycles-pp.shmem_write_begin.generic_perform_write.__generic_file_write_iter.generic_file_write_iter.vfs_write
      4.14 ±102%      -2.2        1.96 ±223%  perf-profile.calltrace.cycles-pp.shmem_alloc_and_acct_folio.shmem_get_folio_gfp.shmem_write_begin.generic_perform_write.__generic_file_write_iter
      4.14 ±102%      -1.4        2.75 ±159%  perf-profile.calltrace.cycles-pp.shmem_get_folio_gfp.shmem_write_begin.generic_perform_write.__generic_file_write_iter.generic_file_write_iter
     18.20 ± 15%     -11.1        7.12 ± 56%  perf-profile.children.cycles-pp.generic_file_write_iter
     18.20 ± 15%     -11.1        7.12 ± 56%  perf-profile.children.cycles-pp.__generic_file_write_iter
     18.20 ± 15%     -11.1        7.12 ± 56%  perf-profile.children.cycles-pp.generic_perform_write
     18.20 ± 15%     -10.3        7.91 ± 60%  perf-profile.children.cycles-pp.ksys_write
     18.20 ± 15%     -10.3        7.91 ± 60%  perf-profile.children.cycles-pp.vfs_write
      5.24 ±107%      -4.4        0.79 ±223%  perf-profile.children.cycles-pp.do_filp_open
      5.24 ±107%      -4.4        0.79 ±223%  perf-profile.children.cycles-pp.path_openat
      4.34 ±101%      -4.3        0.00        perf-profile.children.cycles-pp.do_vmi_align_munmap
      3.86 ±100%      -3.9        0.00        perf-profile.children.cycles-pp.__do_sys_brk
      5.24 ±107%      -3.8        1.39 ±143%  perf-profile.children.cycles-pp.__x64_sys_openat
      5.24 ±107%      -3.8        1.39 ±143%  perf-profile.children.cycles-pp.do_sys_openat2
      4.14 ±102%      -3.3        0.79 ±223%  perf-profile.children.cycles-pp.__alloc_pages
      5.53 ± 72%      -2.8        2.75 ±159%  perf-profile.children.cycles-pp.shmem_write_begin
      4.14 ±102%      -2.2        1.96 ±223%  perf-profile.children.cycles-pp.shmem_alloc_and_acct_folio
      4.14 ±102%      -1.4        2.75 ±159%  perf-profile.children.cycles-pp.shmem_get_folio_gfp
      4.76 ±111%      -4.0        0.79 ±223%  perf-profile.self.cycles-pp.fault_in_readable
      3.77 ±100%      -3.2        0.60 ±223%  perf-profile.self.cycles-pp.zap_pte_range
     35.29            +6.2%      37.47        perf-stat.i.MPKI
 1.103e+09           -49.5%  5.566e+08        perf-stat.i.branch-instructions
      3.99            +1.8        5.80        perf-stat.i.branch-miss-rate%
  19937132           +51.2%   30154000        perf-stat.i.branch-misses
      4.48            -0.2        4.25        perf-stat.i.cache-miss-rate%
   2311967 ±  2%     +57.9%    3651013        perf-stat.i.cache-misses
  52356349           +63.0%   85336312        perf-stat.i.cache-references
      2132 ±  3%    +120.5%       4703        perf-stat.i.context-switches
      2.80           -17.2%       2.31        perf-stat.i.cpi
 5.652e+09            -5.3%  5.353e+09        perf-stat.i.cpu-cycles
      2808           -45.3%       1536        perf-stat.i.cycles-between-cache-misses
      0.39 ±  2%      +0.1        0.48        perf-stat.i.dTLB-load-miss-rate%
   1543223 ±  3%     +63.8%    2527374        perf-stat.i.dTLB-load-misses
 1.017e+09           -44.3%  5.669e+08        perf-stat.i.dTLB-loads
      0.07 ±  3%      +0.0        0.09        perf-stat.i.dTLB-store-miss-rate%
    152204 ±  4%     +54.3%     234914        perf-stat.i.dTLB-store-misses
 5.998e+08           -51.9%  2.885e+08        perf-stat.i.dTLB-stores
     72.30            +4.0       76.32        perf-stat.i.iTLB-load-miss-rate%
   1341052 ±  2%     -32.8%     901087        perf-stat.i.iTLB-load-misses
    487115 ±  2%     -42.8%     278743 ±  2%  perf-stat.i.iTLB-loads
 4.089e+09           -39.9%  2.455e+09        perf-stat.i.instructions
      3035           -10.0%       2730        perf-stat.i.instructions-per-iTLB-miss
      0.71           -36.1%       0.45        perf-stat.i.ipc
      3.27 ±  4%     -62.7%       1.22        perf-stat.i.major-faults
      0.06            -5.3%       0.06        perf-stat.i.metric.GHz
    690.67 ±  2%     +27.4%     879.69        perf-stat.i.metric.K/sec
     28.18           -47.8%      14.72        perf-stat.i.metric.M/sec
      6125            -6.1%       5749        perf-stat.i.minor-faults
     82.46            -6.4       76.09        perf-stat.i.node-load-miss-rate%
    116280 ±  5%     +25.1%     145514 ±  4%  perf-stat.i.node-load-misses
      6128            -6.2%       5751        perf-stat.i.page-faults
     12.82          +171.2%      34.76        perf-stat.overall.MPKI
      1.81            +3.6        5.42        perf-stat.overall.branch-miss-rate%
      4.41            -0.1        4.28        perf-stat.overall.cache-miss-rate%
      1.38           +57.6%       2.18        perf-stat.overall.cpi
      2445 ±  2%     -40.0%       1466        perf-stat.overall.cycles-between-cache-misses
      0.15 ±  3%      +0.3        0.44        perf-stat.overall.dTLB-load-miss-rate%
      0.03 ±  4%      +0.1        0.08        perf-stat.overall.dTLB-store-miss-rate%
     73.35            +3.0       76.38        perf-stat.overall.iTLB-load-miss-rate%
      3049 ±  2%     -10.6%       2725        perf-stat.overall.instructions-per-iTLB-miss
      0.72           -36.6%       0.46        perf-stat.overall.ipc
 1.082e+09           -49.4%   5.48e+08        perf-stat.ps.branch-instructions
  19556338           +51.8%   29688966        perf-stat.ps.branch-misses
   2269777 ±  2%     +58.4%    3594809        perf-stat.ps.cache-misses
  51414505           +63.4%   84023854        perf-stat.ps.cache-references
      2094 ±  3%    +121.1%       4630        perf-stat.ps.context-switches
 5.549e+09            -5.0%  5.271e+09        perf-stat.ps.cpu-cycles
   1515443 ±  3%     +64.2%    2488466        perf-stat.ps.dTLB-load-misses
 9.979e+08           -44.1%  5.582e+08        perf-stat.ps.dTLB-loads
    149490 ±  4%     +54.7%     231314        perf-stat.ps.dTLB-store-misses
 5.886e+08           -51.7%  2.841e+08        perf-stat.ps.dTLB-stores
   1316324 ±  2%     -32.6%     887291        perf-stat.ps.iTLB-load-misses
    478150 ±  2%     -42.6%     274431 ±  2%  perf-stat.ps.iTLB-loads
 4.012e+09           -39.7%  2.418e+09        perf-stat.ps.instructions
      3.20 ±  4%     -62.6%       1.20        perf-stat.ps.major-faults
      6015            -5.9%       5661        perf-stat.ps.minor-faults
    114133 ±  5%     +25.6%     143299 ±  4%  perf-stat.ps.node-load-misses
      6018            -5.9%       5662        perf-stat.ps.page-faults
 2.216e+11           -28.0%  1.596e+11        perf-stat.total.instructions



***************************************************************************************************
lkp-csl-2sp7: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/option_a/rootfs/tbox_group/test/testcase:
  gcc-11/performance/x86_64-rhel-8.3/pthread_once/debian-x86_64-phoronix/lkp-csl-2sp7/glibc-bench-1.6.1/phoronix-test-suite

commit: 
  a525b95417 ("cpufreq/amd-pstate: Remove fast_switch_possible flag from active driver")
  278dfbd17c ("cpufreq: Return failure if fast_switch is not set and fast_switch_possible is set")

a525b954173e74cd 278dfbd17c5d0767834471d7fea 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :6          100%           6:6     kmsg.cpufreq:fast_switch_possible_is_enabled_but_fast_switch_callback_is_not_set
         %stddev     %change         %stddev
             \          |                \  
      2.59          +300.8%      10.40        phoronix-test-suite.glibc-bench.pthread_once.nanoseconds
     55.71           +22.1%      68.01        phoronix-test-suite.time.elapsed_time
     55.71           +22.1%      68.01        phoronix-test-suite.time.elapsed_time.max
    317.17 ± 10%    +366.7%       1480 ±  2%  phoronix-test-suite.time.involuntary_context_switches
     61.67           +13.0%      69.67        phoronix-test-suite.time.percent_of_cpu_this_job_got
      5.29          +112.4%      11.24        phoronix-test-suite.time.system_time
     29.32           +24.3%      36.46        phoronix-test-suite.time.user_time
      8014 ±  3%     +24.5%       9980        meminfo.Active(anon)
     14023 ± 28%     -33.3%       9356 ± 44%  numa-vmstat.node1.nr_slab_reclaimable
 5.345e+09           +16.4%  6.223e+09        cpuidle..time
   5596240           +17.4%    6571074        cpuidle..usage
     56094 ± 28%     -33.3%      37426 ± 44%  numa-meminfo.node1.KReclaimable
     56094 ± 28%     -33.3%      37426 ± 44%  numa-meminfo.node1.SReclaimable
    125.08 ± 12%     +91.5%     239.55 ± 50%  uptime.boot
     11056 ± 11%     +86.0%      20560 ± 49%  uptime.idle
     66.42 ± 22%    +152.7%     167.84 ± 72%  boot-time.boot
     46.41 ± 31%    +206.2%     142.14 ± 85%  boot-time.dhcp
      5588 ± 23%    +152.3%      14100 ± 72%  boot-time.idle
      2.58 ± 45%    +307.1%      10.52 ± 98%  boot-time.smp_boot
      1511           -20.4%       1202 ±  5%  vmstat.io.bi
      1.00          +100.0%       2.00        vmstat.procs.r
      2580 ±  2%     +94.1%       5008        vmstat.system.cs
     95728            -3.3%      92603        vmstat.system.in
      0.06 ±  3%      -0.0        0.04 ±  3%  mpstat.cpu.all.iowait%
      1.70            +1.8        3.55        mpstat.cpu.all.irq%
      0.10 ±  4%      +0.1        0.21        mpstat.cpu.all.soft%
      0.18            +0.2        0.40        mpstat.cpu.all.sys%
      0.79            +1.1        1.89        mpstat.cpu.all.usr%
      2003 ±  3%     +24.6%       2495        proc-vmstat.nr_active_anon
    108638            -5.6%     102534        proc-vmstat.nr_anon_pages
    285043            -2.2%     278915        proc-vmstat.nr_inactive_anon
     52421            -1.5%      51635        proc-vmstat.nr_slab_unreclaimable
      2003 ±  3%     +24.6%       2495        proc-vmstat.nr_zone_active_anon
    285043            -2.2%     278915        proc-vmstat.nr_zone_inactive_anon
    459003            +6.8%     490091        proc-vmstat.numa_hit
    440043            +7.3%     472049        proc-vmstat.numa_local
    529679            +6.8%     565786        proc-vmstat.pgalloc_normal
    592060 ±  2%      +9.0%     645487        proc-vmstat.pgfault
    431573            +7.1%     462298        proc-vmstat.pgfree
     72771 ±  3%     +16.0%      84416        proc-vmstat.pgreuse
    609792 ±  2%     +18.5%     722688        proc-vmstat.unevictable_pgs_scanned
      4.31            +4.8        9.10        turbostat.Busy%
      1576           -55.6%     700.00        turbostat.Bzy_MHz
     61354 ± 14%     -22.8%      47384 ±  2%  turbostat.C1
     49691 ± 12%    +226.7%     162332 ±  3%  turbostat.C1E
      0.28 ±  2%      +0.0        0.32        turbostat.C1E%
   5437863           +16.4%    6329781        turbostat.C6
     70.92           -20.8%      56.16        turbostat.CPU%c1
     24.77           +40.3%      34.74        turbostat.CPU%c6
     47.00           -19.5%      37.83        turbostat.CoreTmp
   5596785           +17.2%    6561866        turbostat.IRQ
     25649           -96.9%     792.83 ±  7%  turbostat.POLL
     17.44 ±  4%     -64.5%       6.19 ± 27%  turbostat.Pkg%pc2
      3.73 ±  5%     -94.8%       0.20 ±  2%  turbostat.Pkg%pc6
     47.00           -18.1%      38.50        turbostat.PkgTmp
    116.45           -40.0%      69.92        turbostat.PkgWatt
    141.45            +1.4%     143.45        turbostat.RAMWatt
      0.07 ± 32%     +74.4%       0.13 ±  7%  sched_debug.cfs_rq:/.h_nr_running.avg
      0.26 ± 13%     +29.7%       0.34 ±  3%  sched_debug.cfs_rq:/.h_nr_running.stddev
     34989 ± 16%     -29.9%      24543 ± 12%  sched_debug.cfs_rq:/.min_vruntime.avg
      0.07 ± 34%     +76.2%       0.13 ± 10%  sched_debug.cfs_rq:/.nr_running.avg
      0.26 ± 14%     +30.5%       0.33 ±  4%  sched_debug.cfs_rq:/.nr_running.stddev
     38.84 ± 25%     -83.8%       6.29 ±160%  sched_debug.cfs_rq:/.removed.load_avg.avg
    193.00 ± 13%     -78.0%      42.53 ±145%  sched_debug.cfs_rq:/.removed.load_avg.stddev
     16.82 ± 21%     -84.4%       2.62 ±190%  sched_debug.cfs_rq:/.removed.runnable_avg.avg
    514.17           -78.2%     112.33 ±157%  sched_debug.cfs_rq:/.removed.runnable_avg.max
     85.74 ± 10%     -81.2%      16.12 ±173%  sched_debug.cfs_rq:/.removed.runnable_avg.stddev
     16.82 ± 21%     -84.4%       2.62 ±190%  sched_debug.cfs_rq:/.removed.util_avg.avg
    514.00           -78.2%     112.17 ±157%  sched_debug.cfs_rq:/.removed.util_avg.max
     85.73 ± 10%     -81.2%      16.11 ±173%  sched_debug.cfs_rq:/.removed.util_avg.stddev
    205.16 ±  6%     -32.6%     138.25 ±  7%  sched_debug.cfs_rq:/.runnable_avg.avg
    250.68 ±  5%     -15.7%     211.41 ±  5%  sched_debug.cfs_rq:/.runnable_avg.stddev
     12358 ± 66%     -78.7%       2634 ±154%  sched_debug.cfs_rq:/.spread0.avg
    205.08 ±  6%     -32.7%     138.10 ±  7%  sched_debug.cfs_rq:/.util_avg.avg
    250.67 ±  5%     -15.7%     211.37 ±  5%  sched_debug.cfs_rq:/.util_avg.stddev
    760.17 ±  5%     -38.4%     468.33 ± 11%  sched_debug.cfs_rq:/.util_est_enqueued.max
    899715 ±  3%     +17.3%    1055048 ± 12%  sched_debug.cpu.avg_idle.avg
     68244 ± 21%    +149.7%     170398 ± 71%  sched_debug.cpu.clock.avg
     68250 ± 21%    +149.7%     170415 ± 71%  sched_debug.cpu.clock.max
     68235 ± 21%    +149.7%     170385 ± 71%  sched_debug.cpu.clock.min
      3.51 ± 13%    +140.4%       8.43 ±  9%  sched_debug.cpu.clock.stddev
     67132 ± 21%    +144.5%     164149 ± 70%  sched_debug.cpu.clock_task.avg
     67835 ± 21%    +147.7%     168044 ± 71%  sched_debug.cpu.clock_task.max
     35817           +19.1%      42655 ±  2%  sched_debug.cpu.clock_task.min
      3228 ± 46%    +287.0%      12494 ± 95%  sched_debug.cpu.clock_task.stddev
    133.56 ± 44%    +108.6%     278.63 ± 10%  sched_debug.cpu.curr->pid.avg
    502.90 ± 20%     +42.3%     715.86 ±  4%  sched_debug.cpu.curr->pid.stddev
      0.00 ± 13%    +135.2%       0.00 ± 29%  sched_debug.cpu.next_balance.stddev
      0.06 ± 41%    +111.1%       0.13 ± 11%  sched_debug.cpu.nr_running.avg
      0.24 ± 18%     +42.8%       0.34 ±  4%  sched_debug.cpu.nr_running.stddev
     68240 ± 21%    +149.7%     170386 ± 71%  sched_debug.cpu_clk
     67525 ± 21%    +151.3%     169670 ± 71%  sched_debug.ktime
     52331           +14.7%      60008        sched_debug.sched_clk
 1.263e+09           -52.8%  5.957e+08        perf-stat.i.branch-instructions
      3.93 ±  2%      +1.6        5.57        perf-stat.i.branch-miss-rate%
  19844846           +52.5%   30265786        perf-stat.i.branch-misses
      4.49            -0.2        4.27        perf-stat.i.cache-miss-rate%
   2283572           +59.1%    3633877        perf-stat.i.cache-misses
  52070252           +62.4%   84585057        perf-stat.i.cache-references
      2159 ±  2%    +117.9%       4705        perf-stat.i.context-switches
      2.70 ±  2%     -16.7%       2.25        perf-stat.i.cpi
 5.606e+09            -4.1%  5.374e+09        perf-stat.i.cpu-cycles
    101.91            -2.3%      99.54        perf-stat.i.cpu-migrations
      2804           -44.9%       1545        perf-stat.i.cycles-between-cache-misses
      0.37 ±  5%      +0.1        0.43        perf-stat.i.dTLB-load-miss-rate%
   1542705           +65.2%    2549066 ±  2%  perf-stat.i.dTLB-load-misses
 1.535e+09           -55.7%  6.799e+08        perf-stat.i.dTLB-loads
      0.07 ±  5%      +0.0        0.10 ±  3%  perf-stat.i.dTLB-store-miss-rate%
    152237 ±  3%     +59.7%     243190 ±  3%  perf-stat.i.dTLB-store-misses
 4.136e+08           -39.1%  2.517e+08        perf-stat.i.dTLB-stores
     71.73            +4.6       76.38        perf-stat.i.iTLB-load-miss-rate%
   1314924 ±  6%     -32.7%     884328        perf-stat.i.iTLB-load-misses
    493184           -44.6%     273276 ±  2%  perf-stat.i.iTLB-loads
 4.402e+09           -42.3%  2.542e+09        perf-stat.i.instructions
      3289 ±  4%     -12.6%       2873        perf-stat.i.instructions-per-iTLB-miss
      0.77           -39.2%       0.47        perf-stat.i.ipc
      3.16 ±  2%     -61.4%       1.22        perf-stat.i.major-faults
      0.06            -4.2%       0.06        perf-stat.i.metric.GHz
    697.29 ±  2%     +26.2%     880.12        perf-stat.i.metric.K/sec
     33.29           -52.2%      15.91        perf-stat.i.metric.M/sec
      6215 ±  3%      -7.1%       5771        perf-stat.i.minor-faults
     82.62            -6.8       75.78        perf-stat.i.node-load-miss-rate%
    117468 ±  6%     +24.2%     145903 ±  2%  perf-stat.i.node-load-misses
     49000 ±  6%     +22.0%      59761 ±  6%  perf-stat.i.node-loads
      6218 ±  3%      -7.2%       5772        perf-stat.i.page-faults
     11.84 ±  2%    +181.1%      33.29        perf-stat.overall.MPKI
      1.57            +3.5        5.08        perf-stat.overall.branch-miss-rate%
      1.27           +65.9%       2.11        perf-stat.overall.cpi
      2455           -39.8%       1479        perf-stat.overall.cycles-between-cache-misses
      0.10 ±  3%      +0.3        0.37        perf-stat.overall.dTLB-load-miss-rate%
      0.04 ±  4%      +0.1        0.10 ±  3%  perf-stat.overall.dTLB-store-miss-rate%
     72.66            +3.7       76.40        perf-stat.overall.iTLB-load-miss-rate%
      3360 ±  6%     -14.5%       2874        perf-stat.overall.instructions-per-iTLB-miss
      0.78           -39.7%       0.47        perf-stat.overall.ipc
  1.24e+09           -52.7%  5.865e+08        perf-stat.ps.branch-instructions
  19487700           +52.9%   29789495        perf-stat.ps.branch-misses
   2242413           +59.5%    3577319        perf-stat.ps.cache-misses
  51143893           +62.8%   83278509        perf-stat.ps.cache-references
      2120 ±  2%    +118.4%       4632        perf-stat.ps.context-switches
 5.505e+09            -3.9%  5.291e+09        perf-stat.ps.cpu-cycles
    100.07            -2.1%      98.01        perf-stat.ps.cpu-migrations
   1515252           +65.6%    2509672 ±  2%  perf-stat.ps.dTLB-load-misses
 1.506e+09           -55.6%  6.693e+08        perf-stat.ps.dTLB-loads
    149535 ±  3%     +60.1%     239445 ±  3%  perf-stat.ps.dTLB-store-misses
  4.06e+08           -39.0%  2.478e+08        perf-stat.ps.dTLB-stores
   1290945 ±  6%     -32.5%     870773        perf-stat.ps.iTLB-load-misses
    484246           -44.4%     269019 ±  2%  perf-stat.ps.iTLB-loads
  4.32e+09           -42.1%  2.502e+09        perf-stat.ps.instructions
      3.10 ±  2%     -61.2%       1.20        perf-stat.ps.major-faults
      6104 ±  3%      -6.9%       5681        perf-stat.ps.minor-faults
    115360 ±  6%     +24.5%     143643 ±  2%  perf-stat.ps.node-load-misses
     48119 ±  7%     +22.3%      58833 ±  6%  perf-stat.ps.node-loads
      6107 ±  3%      -6.9%       5683        perf-stat.ps.page-faults
 2.422e+11           -32.0%  1.646e+11        perf-stat.total.instructions
     12.90 ± 35%      -9.8        3.08 ± 72%  perf-profile.calltrace.cycles-pp.exit_mmap.__mmput.exit_mm.do_exit.do_group_exit
     11.79 ± 43%      -9.6        2.16 ±100%  perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.__mmput.exit_mm.do_exit
     11.79 ± 43%      -9.6        2.16 ±100%  perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.__mmput.exit_mm
      8.19 ± 58%      -6.6        1.59 ±144%  perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      6.66 ±133%      -6.0        0.70 ±223%  perf-profile.calltrace.cycles-pp.__do_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.62 ±121%      -4.9        0.70 ±223%  perf-profile.calltrace.cycles-pp.do_vmi_align_munmap.__do_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.62 ±121%      -4.9        0.70 ±223%  perf-profile.calltrace.cycles-pp.unmap_region.do_vmi_align_munmap.__do_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.38 ±109%      -4.7        0.67 ±223%  perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.38 ±109%      -4.7        0.67 ±223%  perf-profile.calltrace.cycles-pp.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.38 ±109%      -4.7        0.67 ±223%  perf-profile.calltrace.cycles-pp.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.58 ±108%      -4.6        0.00        perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault.fault_in_readable
      5.84 ± 88%      -4.3        1.59 ±144%  perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      4.24 ±115%      -4.2        0.00        perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.24 ±115%      -4.2        0.00        perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.24 ±115%      -4.2        0.00        perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.11 ±108%      -4.1        0.00        perf-profile.calltrace.cycles-pp.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault
      4.11 ±108%      -4.1        0.00        perf-profile.calltrace.cycles-pp.do_read_fault.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
      4.11 ±108%      -4.1        0.00        perf-profile.calltrace.cycles-pp.__do_fault.do_read_fault.do_fault.__handle_mm_fault.handle_mm_fault
      4.58 ±108%      -3.8        0.79 ±223%  perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault.fault_in_readable.fault_in_iov_iter_readable.generic_perform_write
      4.58 ±108%      -3.8        0.79 ±223%  perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault.fault_in_readable.fault_in_iov_iter_readable
      3.13 ±100%      -3.1        0.00        perf-profile.calltrace.cycles-pp.exit_mm.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      3.13 ±100%      -3.1        0.00        perf-profile.calltrace.cycles-pp.__mmput.exit_mm.do_exit.do_group_exit.__x64_sys_exit_group
      3.13 ±100%      -3.1        0.00        perf-profile.calltrace.cycles-pp.page_remove_rmap.zap_pte_range.zap_pmd_range.unmap_page_range.unmap_vmas
      3.21 ±100%      -2.5        0.67 ±223%  perf-profile.calltrace.cycles-pp.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64
      3.21 ±100%      -2.5        0.67 ±223%  perf-profile.calltrace.cycles-pp.search_binary_handler.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve
      3.82 ±101%      -2.4        1.46 ±142%  perf-profile.calltrace.cycles-pp.rep_movs_alternative.copyin.copy_page_from_iter_atomic.generic_perform_write.__generic_file_write_iter
      3.82 ±101%      -2.4        1.46 ±142%  perf-profile.calltrace.cycles-pp.copyin.copy_page_from_iter_atomic.generic_perform_write.__generic_file_write_iter.generic_file_write_iter
      0.00            +8.4        8.42 ± 42%  perf-profile.calltrace.cycles-pp.proc_reg_read_iter.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +8.4        8.42 ± 42%  perf-profile.calltrace.cycles-pp.seq_read_iter.proc_reg_read_iter.vfs_read.ksys_read.do_syscall_64
      0.00           +10.3       10.30 ± 57%  perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00           +10.3       10.30 ± 57%  perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
     14.85 ± 40%     -12.7        2.16 ±100%  perf-profile.children.cycles-pp.unmap_vmas
     13.94 ± 25%     -10.9        3.08 ± 72%  perf-profile.children.cycles-pp.__mmput
     13.94 ± 25%     -10.9        3.08 ± 72%  perf-profile.children.cycles-pp.exit_mmap
     12.83 ± 34%     -10.7        2.16 ±100%  perf-profile.children.cycles-pp.unmap_page_range
     12.90 ± 35%      -9.8        3.08 ± 72%  perf-profile.children.cycles-pp.exit_mm
     10.42 ± 29%      -8.8        1.59 ±144%  perf-profile.children.cycles-pp.handle_mm_fault
     11.71 ± 47%      -8.4        3.31 ± 72%  perf-profile.children.cycles-pp.exc_page_fault
     11.71 ± 47%      -8.4        3.31 ± 72%  perf-profile.children.cycles-pp.do_user_addr_fault
      8.19 ± 58%      -6.6        1.59 ±144%  perf-profile.children.cycles-pp.__handle_mm_fault
      6.66 ±133%      -6.0        0.70 ±223%  perf-profile.children.cycles-pp.__do_sys_brk
      6.66 ±133%      -6.0        0.70 ±223%  perf-profile.children.cycles-pp.do_vmi_align_munmap
      6.66 ±133%      -6.0        0.70 ±223%  perf-profile.children.cycles-pp.unmap_region
      5.38 ±109%      -4.7        0.67 ±223%  perf-profile.children.cycles-pp.__x64_sys_execve
      5.38 ±109%      -4.7        0.67 ±223%  perf-profile.children.cycles-pp.do_execveat_common
      5.38 ±109%      -4.7        0.67 ±223%  perf-profile.children.cycles-pp.bprm_execve
      4.24 ±115%      -4.2        0.00        perf-profile.children.cycles-pp.__x64_sys_exit_group
      4.11 ±108%      -4.1        0.00        perf-profile.children.cycles-pp.do_fault
      4.11 ±108%      -4.1        0.00        perf-profile.children.cycles-pp.do_read_fault
      4.11 ±108%      -4.1        0.00        perf-profile.children.cycles-pp.__do_fault
      3.13 ±100%      -3.1        0.00        perf-profile.children.cycles-pp.page_remove_rmap
      4.89 ± 73%      -2.7        2.18 ±149%  perf-profile.children.cycles-pp.__alloc_pages
      3.21 ±100%      -2.5        0.67 ±223%  perf-profile.children.cycles-pp.exec_binprm
      3.21 ±100%      -2.5        0.67 ±223%  perf-profile.children.cycles-pp.search_binary_handler
      3.82 ±101%      -2.4        1.46 ±142%  perf-profile.children.cycles-pp.rep_movs_alternative
      3.82 ±101%      -2.4        1.46 ±142%  perf-profile.children.cycles-pp.copyin
      0.00            +8.4        8.42 ± 42%  perf-profile.children.cycles-pp.proc_reg_read_iter
      0.00            +8.4        8.42 ± 42%  perf-profile.children.cycles-pp.seq_read_iter
      0.00           +10.3       10.30 ± 57%  perf-profile.children.cycles-pp.ksys_read
      0.00           +10.3       10.30 ± 57%  perf-profile.children.cycles-pp.vfs_read
      3.54 ±104%      -2.8        0.70 ±223%  perf-profile.self.cycles-pp.zap_pte_range
      3.82 ±101%      -2.4        1.46 ±142%  perf-profile.self.cycles-pp.rep_movs_alternative



***************************************************************************************************
lkp-csl-2sp7: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/option_a/rootfs/tbox_group/test/testcase:
  gcc-11/performance/x86_64-rhel-8.3/3/debian-x86_64-phoronix/lkp-csl-2sp7/cyclictest-1.0.0/phoronix-test-suite

commit: 
  a525b95417 ("cpufreq/amd-pstate: Remove fast_switch_possible flag from active driver")
  278dfbd17c ("cpufreq: Return failure if fast_switch is not set and fast_switch_possible is set")

a525b954173e74cd 278dfbd17c5d0767834471d7fea 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :6          100%           6:6     kmsg.cpufreq:fast_switch_possible_is_enabled_but_fast_switch_callback_is_not_set
         %stddev     %change         %stddev
             \          |                \  
      1.00         +5550.0%      56.50 ± 90%  phoronix-test-suite.cyclictest.clock_nanosleepTIME_ABSTIME,Interval500ms,100000Loops.ms_average
     67.47           +12.4%      75.83        phoronix-test-suite.time.elapsed_time
     67.47           +12.4%      75.83        phoronix-test-suite.time.elapsed_time.max
    228.83 ± 13%    +555.6%       1500 ±  3%  phoronix-test-suite.time.involuntary_context_switches
      5.00          +270.0%      18.50 ±  2%  phoronix-test-suite.time.percent_of_cpu_this_job_got
 2.229e+08           -28.7%  1.588e+08        cpuidle..usage
      8121 ±  3%     +24.7%      10129        meminfo.Active(anon)
      1244           -14.1%       1069 ±  3%  vmstat.io.bi
      0.00       +1.3e+102%       1.33 ± 35%  vmstat.procs.r
      5430           +39.5%       7576        vmstat.system.cs
     97212            -3.0%      94332        vmstat.system.in
      0.05 ±  2%      -0.0        0.04 ±  5%  mpstat.cpu.all.iowait%
      0.90 ±  2%      +1.6        2.47        mpstat.cpu.all.irq%
      0.05 ±  3%      +0.1        0.16 ±  2%  mpstat.cpu.all.soft%
      0.10            +0.2        0.34 ±  2%  mpstat.cpu.all.sys%
      0.23            +1.1        1.36        mpstat.cpu.all.usr%
      2030 ±  3%     +24.7%       2531        proc-vmstat.nr_active_anon
    108456            -4.6%     103460        proc-vmstat.nr_anon_pages
    284847            -1.8%     279599        proc-vmstat.nr_inactive_anon
     52453            -1.2%      51832        proc-vmstat.nr_slab_unreclaimable
      2030 ±  3%     +24.7%       2531        proc-vmstat.nr_zone_active_anon
    284847            -1.8%     279599        proc-vmstat.nr_zone_inactive_anon
    474897            +6.5%     505695        proc-vmstat.numa_hit
    457187            +6.5%     486800        proc-vmstat.numa_local
    551902 ±  2%      +4.8%     578226        proc-vmstat.pgalloc_normal
    615780            +9.0%     671086        proc-vmstat.pgfault
    456513 ±  2%      +4.3%     476256        proc-vmstat.pgfree
     77912           +13.6%      88541 ±  2%  proc-vmstat.pgreuse
    667136           +12.5%     750208        proc-vmstat.unevictable_pgs_scanned
      2033           -77.0%     467.83        turbostat.Avg_MHz
      2778           -74.8%     700.00        turbostat.Bzy_MHz
     65709 ± 14%     -61.7%      25182 ±  8%  turbostat.C1
      0.02 ± 20%      -0.0        0.01        turbostat.C1%
     31391 ± 20%    +190.6%      91233 ±  7%  turbostat.C1E
   1797823 ±  4%     +31.5%    2364347        turbostat.C6
     26.41 ±  3%      +5.9       32.32        turbostat.C6%
      7.08 ±  6%     +87.7%      13.30        turbostat.CPU%c6
     58.00 ±  2%     -31.3%      39.83        turbostat.CoreTmp
  2.21e+08           -29.3%  1.563e+08        turbostat.POLL
     71.21           -10.2       60.97        turbostat.POLL%
      3.92 ±  4%     -45.7%       2.12 ±  7%  turbostat.Pkg%pc2
      1.58 ± 10%     -88.7%       0.18 ±  6%  turbostat.Pkg%pc6
     57.83 ±  3%     -30.5%      40.17        turbostat.PkgTmp
    223.68           -63.4%      81.80        turbostat.PkgWatt
      3459 ± 34%     +45.1%       5020 ± 11%  sched_debug.cfs_rq:/.load.avg
     10426 ± 20%     +36.6%      14243 ±  5%  sched_debug.cfs_rq:/.load.stddev
     62.38 ± 31%     -59.0%      25.56 ± 30%  sched_debug.cfs_rq:/.load_avg.avg
    212.19 ± 22%     -50.5%     104.97 ± 34%  sched_debug.cfs_rq:/.load_avg.stddev
     42608 ± 18%     +36.6%      58197 ±  2%  sched_debug.cfs_rq:/.min_vruntime.max
      4070 ± 12%     +28.5%       5231 ±  3%  sched_debug.cfs_rq:/.min_vruntime.stddev
     45.18 ± 45%     -77.5%      10.16 ± 58%  sched_debug.cfs_rq:/.removed.load_avg.avg
    200.73 ± 25%     -55.6%      89.17 ± 48%  sched_debug.cfs_rq:/.removed.load_avg.stddev
     19.50 ± 44%     -74.2%       5.03 ± 58%  sched_debug.cfs_rq:/.removed.runnable_avg.avg
     88.18 ± 24%     -50.0%      44.09 ± 48%  sched_debug.cfs_rq:/.removed.runnable_avg.stddev
     19.50 ± 44%     -74.2%       5.02 ± 58%  sched_debug.cfs_rq:/.removed.util_avg.avg
     88.18 ± 24%     -50.0%      44.07 ± 48%  sched_debug.cfs_rq:/.removed.util_avg.stddev
    199.68 ±  9%     -32.3%     135.14 ± 10%  sched_debug.cfs_rq:/.runnable_avg.avg
    246.87 ±  5%     -16.2%     206.81 ±  7%  sched_debug.cfs_rq:/.runnable_avg.stddev
      4075 ± 13%     +28.6%       5239 ±  3%  sched_debug.cfs_rq:/.spread0.stddev
    199.62 ±  9%     -32.4%     135.03 ± 10%  sched_debug.cfs_rq:/.util_avg.avg
    246.86 ±  5%     -16.2%     206.77 ±  7%  sched_debug.cfs_rq:/.util_avg.stddev
    812.67 ± 12%     -45.7%     441.00 ± 17%  sched_debug.cfs_rq:/.util_est_enqueued.max
    101.76 ± 19%     -31.7%      69.51 ± 16%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
      4205 ± 45%    +167.9%      11264 ± 12%  sched_debug.cpu.avg_idle.min
      3.54 ± 29%    +152.9%       8.96 ± 17%  sched_debug.cpu.clock.stddev
     35726           +19.2%      42576 ±  2%  sched_debug.cpu.clock_task.min
     52210           +14.9%      60013        sched_debug.sched_clk
      5.54 ± 76%      -5.5        0.00        perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.54 ± 76%      -5.5        0.00        perf-profile.calltrace.cycles-pp.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.54 ± 76%      -5.5        0.00        perf-profile.calltrace.cycles-pp.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.54 ± 76%      -5.5        0.00        perf-profile.calltrace.cycles-pp.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64
      5.54 ± 76%      -5.5        0.00        perf-profile.calltrace.cycles-pp.search_binary_handler.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve
      4.43 ±105%      -4.4        0.00        perf-profile.calltrace.cycles-pp.load_elf_binary.search_binary_handler.exec_binprm.bprm_execve.do_execveat_common
      4.80 ± 73%      -3.2        1.55 ±141%  perf-profile.calltrace.cycles-pp.page_remove_rmap.zap_pte_range.zap_pmd_range.unmap_page_range.unmap_vmas
      0.00            +5.6        5.63 ± 28%  perf-profile.calltrace.cycles-pp.__fput.task_work_run.do_exit.do_group_exit.get_signal
      0.00            +5.6        5.63 ± 28%  perf-profile.calltrace.cycles-pp.task_work_run.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart
      0.00            +5.6        5.63 ± 28%  perf-profile.calltrace.cycles-pp.perf_release.__fput.task_work_run.do_exit.do_group_exit
      0.00            +5.6        5.63 ± 28%  perf-profile.calltrace.cycles-pp.perf_event_release_kernel.perf_release.__fput.task_work_run.do_exit
      4.80 ± 73%      +5.7       10.50 ± 15%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.80 ± 73%      +5.7       10.50 ± 15%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.80 ± 73%      +5.7       10.50 ± 15%  perf-profile.calltrace.cycles-pp.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      4.80 ± 73%      +5.7       10.50 ± 15%  perf-profile.calltrace.cycles-pp.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode
      4.80 ± 73%      +5.7       10.50 ± 15%  perf-profile.calltrace.cycles-pp.do_group_exit.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare
      4.80 ± 73%      +5.7       10.50 ± 15%  perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop
      6.08 ± 47%      +7.5       13.60 ± 27%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
     33.77 ± 15%     +11.1       44.92 ±  8%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
     32.84 ± 17%     +12.1       44.92 ±  8%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
     32.84 ± 17%     +13.0       45.84 ±  5%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     33.77 ± 15%     +13.9       47.65 ±  3%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     32.84 ± 17%     +14.8       47.65 ±  3%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     32.84 ± 17%     +14.8       47.65 ±  3%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     32.84 ± 17%     +14.8       47.65 ±  3%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      5.98 ±108%      -6.0        0.00        perf-profile.children.cycles-pp.inode_permission
      5.54 ± 76%      -5.5        0.00        perf-profile.children.cycles-pp.__x64_sys_execve
      5.54 ± 76%      -5.5        0.00        perf-profile.children.cycles-pp.do_execveat_common
      5.54 ± 76%      -5.5        0.00        perf-profile.children.cycles-pp.bprm_execve
      5.54 ± 76%      -5.5        0.00        perf-profile.children.cycles-pp.exec_binprm
      5.54 ± 76%      -5.5        0.00        perf-profile.children.cycles-pp.search_binary_handler
      4.86 ±115%      -4.9        0.00        perf-profile.children.cycles-pp.override_creds
      4.43 ±105%      -4.4        0.00        perf-profile.children.cycles-pp.load_elf_binary
      4.80 ± 73%      -3.2        1.55 ±141%  perf-profile.children.cycles-pp.page_remove_rmap
      0.00            +5.6        5.63 ± 28%  perf-profile.children.cycles-pp.perf_release
      0.00            +5.6        5.63 ± 28%  perf-profile.children.cycles-pp.perf_event_release_kernel
      4.80 ± 73%      +5.7       10.50 ± 15%  perf-profile.children.cycles-pp.arch_do_signal_or_restart
      4.80 ± 73%      +5.7       10.50 ± 15%  perf-profile.children.cycles-pp.get_signal
      4.80 ± 73%      +6.6       11.42 ± 24%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      4.80 ± 73%      +6.6       11.42 ± 24%  perf-profile.children.cycles-pp.exit_to_user_mode_loop
      6.08 ± 47%      +7.5       13.60 ± 27%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
     33.77 ± 15%     +11.2       44.92 ±  8%  perf-profile.children.cycles-pp.cpuidle_enter
     33.77 ± 15%     +11.2       44.92 ±  8%  perf-profile.children.cycles-pp.cpuidle_enter_state
     33.77 ± 15%     +12.1       45.84 ±  5%  perf-profile.children.cycles-pp.cpuidle_idle_call
     33.77 ± 15%     +13.9       47.65 ±  3%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     33.77 ± 15%     +13.9       47.65 ±  3%  perf-profile.children.cycles-pp.cpu_startup_entry
     33.77 ± 15%     +13.9       47.65 ±  3%  perf-profile.children.cycles-pp.do_idle
     32.84 ± 17%     +14.8       47.65 ±  3%  perf-profile.children.cycles-pp.start_secondary
      7.64 ±120%      -6.9        0.76 ±223%  perf-profile.self.cycles-pp.fault_in_readable
      4.86 ±115%      -4.9        0.00        perf-profile.self.cycles-pp.override_creds
      4.94 ±130%      -3.4        1.52 ±223%  perf-profile.self.cycles-pp.show_interrupts
 9.155e+09           -71.5%   2.61e+09        perf-stat.i.branch-instructions
      1.97 ±  7%      +0.5        2.48        perf-stat.i.branch-miss-rate%
  37281940           -28.1%   26816122        perf-stat.i.branch-misses
      9.37 ±  2%      -1.2        8.20 ±  3%  perf-stat.i.cache-miss-rate%
   1168532 ±  3%     +56.3%    1826809        perf-stat.i.cache-misses
  18204626 ±  6%     +74.1%   31696003        perf-stat.i.cache-references
      5294           +41.2%       7477        perf-stat.i.context-switches
      6.21           -29.6%       4.37        perf-stat.i.cpi
 2.021e+11           -76.6%  4.725e+10        perf-stat.i.cpu-cycles
    576582 ±  3%     -84.5%      89226 ±  3%  perf-stat.i.cycles-between-cache-misses
    462899 ±  6%     +85.2%     857410 ±  2%  perf-stat.i.dTLB-load-misses
 5.539e+09           -64.5%  1.966e+09        perf-stat.i.dTLB-loads
     58476 ±  5%     +58.1%      92464 ±  3%  perf-stat.i.dTLB-store-misses
 8.442e+08           -29.3%  5.971e+08        perf-stat.i.dTLB-stores
     88.97            -6.7       82.29        perf-stat.i.iTLB-load-miss-rate%
    151482 ±  2%     +37.7%     208592 ±  2%  perf-stat.i.iTLB-loads
 2.974e+10           -68.2%  9.452e+09        perf-stat.i.instructions
     30509 ±  2%     -67.6%       9882 ±  2%  perf-stat.i.instructions-per-iTLB-miss
      0.23           +17.4%       0.28        perf-stat.i.ipc
      2.72 ±  5%     -55.9%       1.20 ±  4%  perf-stat.i.major-faults
      2.11           -76.6%       0.49        perf-stat.i.metric.GHz
    243.40 ± 10%     +37.5%     334.64        perf-stat.i.metric.K/sec
    161.80           -66.7%      53.88        perf-stat.i.metric.M/sec
     83.31            -3.2       80.11        perf-stat.i.node-load-miss-rate%
    106552 ±  5%     +32.3%     140953 ±  7%  perf-stat.i.node-load-misses
     43996 ± 11%     +16.0%      51045 ±  2%  perf-stat.i.node-loads
     59564 ±  5%     +16.5%      69365 ±  8%  perf-stat.i.node-store-misses
      0.62 ±  7%    +447.4%       3.37 ±  2%  perf-stat.overall.MPKI
      0.41            +0.6        1.03        perf-stat.overall.branch-miss-rate%
      6.43 ±  4%      -0.7        5.76 ±  2%  perf-stat.overall.cache-miss-rate%
      6.80           -26.5%       5.00        perf-stat.overall.cpi
    172717 ±  4%     -85.1%      25771        perf-stat.overall.cycles-between-cache-misses
      0.01 ±  7%      +0.0        0.04        perf-stat.overall.dTLB-load-miss-rate%
      0.01 ±  6%      +0.0        0.02 ±  2%  perf-stat.overall.dTLB-store-miss-rate%
     86.59            -4.6       82.02        perf-stat.overall.iTLB-load-miss-rate%
     30298           -67.3%       9918        perf-stat.overall.instructions-per-iTLB-miss
      0.15           +36.0%       0.20        perf-stat.overall.ipc
     56.74 ±  5%      +6.0       62.77 ±  4%  perf-stat.overall.node-store-miss-rate%
 9.009e+09           -71.5%  2.571e+09        perf-stat.ps.branch-instructions
  36711771           -27.9%   26457021        perf-stat.ps.branch-misses
   1153423 ±  3%     +56.6%    1806281        perf-stat.ps.cache-misses
  17988261 ±  6%     +74.3%   31357377        perf-stat.ps.cache-references
      5212           +41.4%       7371        perf-stat.ps.context-switches
 1.989e+11           -76.6%  4.653e+10        perf-stat.ps.cpu-cycles
    457693 ±  6%     +85.4%     848579 ±  2%  perf-stat.ps.dTLB-load-misses
 5.451e+09           -64.5%  1.937e+09        perf-stat.ps.dTLB-loads
     57782 ±  5%     +58.3%      91486 ±  3%  perf-stat.ps.dTLB-store-misses
  8.31e+08           -29.2%  5.884e+08        perf-stat.ps.dTLB-stores
    149554 ±  2%     +37.7%     205884 ±  2%  perf-stat.ps.iTLB-loads
 2.926e+10           -68.2%   9.31e+09        perf-stat.ps.instructions
      2.68 ±  5%     -55.8%       1.19 ±  4%  perf-stat.ps.major-faults
    105070 ±  5%     +32.5%     139219 ±  7%  perf-stat.ps.node-load-misses
     43441 ± 11%     +16.2%      50497 ±  2%  perf-stat.ps.node-loads
     58710 ±  5%     +16.6%      68472 ±  8%  perf-stat.ps.node-store-misses
 1.984e+12           -65.3%  6.875e+11        perf-stat.total.instructions



***************************************************************************************************
lkp-cfl-e1: 16 threads 1 sockets Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz (Coffee Lake) with 32G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/option_a/rootfs/tbox_group/test/testcase:
  gcc-11/performance/x86_64-rhel-8.3/Medium/debian-x86_64-phoronix/lkp-cfl-e1/astcenc-1.1.0/phoronix-test-suite

commit: 
  a525b95417 ("cpufreq/amd-pstate: Remove fast_switch_possible flag from active driver")
  278dfbd17c ("cpufreq: Return failure if fast_switch is not set and fast_switch_possible is set")

a525b954173e74cd 278dfbd17c5d0767834471d7fea 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :6          100%           6:6     kmsg.cpufreq:fast_switch_possible_is_enabled_but_fast_switch_callback_is_not_set
         %stddev     %change         %stddev
             \          |                \  
      6.51          +106.6%      13.45        phoronix-test-suite.astcenc.Medium.seconds
     41.10          +223.2%     132.82        phoronix-test-suite.time.elapsed_time
     41.10          +223.2%     132.82        phoronix-test-suite.time.elapsed_time.max
      1674         +1056.2%      19360 ±  6%  phoronix-test-suite.time.involuntary_context_switches
    236.67           +56.9%     371.33        phoronix-test-suite.time.percent_of_cpu_this_job_got
     96.53          +406.6%     489.06        phoronix-test-suite.time.user_time
 5.719e+08          +161.8%  1.497e+09        cpuidle..time
    609347          +172.7%    1661621        cpuidle..usage
      6795           +16.3%       7902        meminfo.Active(anon)
    197921 ±  7%     +73.9%     344256 ±  2%  meminfo.AnonHugePages
    469519           +19.9%     562889        meminfo.AnonPages
     82.86           -12.8       70.05        mpstat.cpu.all.idle%
      0.72 ± 76%      -0.6        0.14 ±  4%  mpstat.cpu.all.iowait%
      0.63            +2.3        2.94        mpstat.cpu.all.irq%
      0.04 ± 11%      +0.1        0.16 ±  3%  mpstat.cpu.all.soft%
      0.43 ±  2%      +0.3        0.70        mpstat.cpu.all.sys%
     15.32           +10.7       26.00        mpstat.cpu.all.usr%
     82.83           -15.5%      70.00        vmstat.cpu.id
     14.00           +78.6%      25.00        vmstat.cpu.us
      2578 ± 12%     -70.2%     769.17 ±  2%  vmstat.io.bi
      2.00          +100.0%       4.00        vmstat.procs.r
      1565           +26.7%       1984        vmstat.system.cs
     15803            +4.2%      16461        vmstat.system.in
      1698           +16.3%       1976        proc-vmstat.nr_active_anon
     28204            +6.8%      30122        proc-vmstat.nr_active_file
    117478           +19.9%     140853        proc-vmstat.nr_anon_pages
     96.33 ±  7%     +74.2%     167.83 ±  2%  proc-vmstat.nr_anon_transparent_hugepages
    296875            +8.1%     320871        proc-vmstat.nr_inactive_anon
      6598            +2.5%       6764        proc-vmstat.nr_kernel_stack
     18110            +1.2%      18332        proc-vmstat.nr_mapped
      1365            +7.0%       1460        proc-vmstat.nr_page_table_pages
     15776            -1.9%      15472        proc-vmstat.nr_slab_unreclaimable
      1698           +16.3%       1976        proc-vmstat.nr_zone_active_anon
     28204            +6.8%      30122        proc-vmstat.nr_zone_active_file
    296875            +8.1%     320871        proc-vmstat.nr_zone_inactive_anon
    290802 ±  2%     +40.6%     408922        proc-vmstat.numa_hit
    290802 ±  2%     +40.7%     409089        proc-vmstat.numa_local
     18530            +4.7%      19409        proc-vmstat.pgactivate
    828771           +17.9%     976790        proc-vmstat.pgalloc_normal
    338161 ±  3%     +56.3%     528417        proc-vmstat.pgfault
    770641           +19.0%     917408        proc-vmstat.pgfree
     31372 ±  3%    +108.5%      65408        proc-vmstat.pgreuse
    465920          +150.6%    1167488        proc-vmstat.unevictable_pgs_scanned
     60.29           +56.9%      94.59        turbostat.Any%C0
    679.67           -63.6%     247.50        turbostat.Avg_MHz
     16.46           +14.5       30.92        turbostat.Busy%
      4144           -80.7%     800.00        turbostat.Bzy_MHz
      1700 ± 20%   +1986.3%      35467 ±  4%  turbostat.C1
      0.01            +0.0        0.02 ± 20%  turbostat.C1%
      0.06 ± 11%      -0.0        0.04 ± 23%  turbostat.C10%
      5370 ± 23%    +492.7%      31828 ±  2%  turbostat.C1E
      0.24 ± 52%      -0.1        0.13 ±  3%  turbostat.C1E%
      0.55 ± 72%      -0.4        0.13 ±  2%  turbostat.C3%
    327658          +127.0%     743779        turbostat.C6
     45.83           -13.8       32.04        turbostat.C6%
    253362          +229.7%     835400        turbostat.C8
     15.34           -15.9%      12.89        turbostat.CPU%c1
      0.83 ± 66%     -77.9%       0.18 ±  4%  turbostat.CPU%c3
     26.89           -37.2%      16.88        turbostat.CPU%c7
     28.37           -90.5%       2.69        turbostat.CorWatt
     55.00 ±  2%     -44.8%      30.33        turbostat.CoreTmp
    697284          +220.7%    2236189        turbostat.IRQ
     11573 ±  3%     -88.6%       1320 ± 14%  turbostat.POLL
      6.93           -92.2%       0.54 ± 13%  turbostat.Pkg%pc2
     21.26 ±  2%     -78.8%       4.51        turbostat.Pkg%pc3
     54.50 ±  2%     -42.8%      31.17        turbostat.PkgTmp
     28.82           -89.0%       3.17        turbostat.PkgWatt
    166.20           +92.8%     320.39        turbostat.Totl%C0
     33915 ± 16%    +104.7%      69438 ± 21%  sched_debug.cfs_rq:/.load.avg
    127082 ±  5%    +189.1%     367447 ± 47%  sched_debug.cfs_rq:/.load.max
     43893 ±  8%    +167.8%     117549 ± 33%  sched_debug.cfs_rq:/.load.stddev
    210.50 ± 31%     -53.3%      98.31 ± 17%  sched_debug.cfs_rq:/.load_avg.avg
      1208 ± 20%     -52.4%     575.11 ± 40%  sched_debug.cfs_rq:/.load_avg.max
     19.17 ± 50%     -77.8%       4.25 ± 67%  sched_debug.cfs_rq:/.load_avg.min
    321.23 ± 29%     -48.4%     165.60 ± 34%  sched_debug.cfs_rq:/.load_avg.stddev
      9895 ± 32%   +1532.2%     161504 ± 38%  sched_debug.cfs_rq:/.min_vruntime.avg
     16191 ± 22%   +1115.3%     196783 ± 35%  sched_debug.cfs_rq:/.min_vruntime.max
      5611 ± 19%   +2491.5%     145426 ± 40%  sched_debug.cfs_rq:/.min_vruntime.min
      2882 ± 24%    +363.4%      13357 ± 29%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.49           -14.9%       0.42 ±  8%  sched_debug.cfs_rq:/.nr_running.stddev
    735.49 ±  9%     -53.0%     345.89 ± 11%  sched_debug.cfs_rq:/.runnable_avg.avg
    214.50 ± 66%     -71.7%      60.78 ± 31%  sched_debug.cfs_rq:/.runnable_avg.min
      3969 ± 83%    +985.0%      43070 ± 53%  sched_debug.cfs_rq:/.spread0.max
      2884 ± 24%    +363.2%      13358 ± 29%  sched_debug.cfs_rq:/.spread0.stddev
    705.35 ±  9%     -52.8%     333.26 ± 11%  sched_debug.cfs_rq:/.util_avg.avg
     70.43 ± 11%     -27.1%      51.31 ± 29%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    790.50           -57.0%     340.19 ± 44%  sched_debug.cfs_rq:/.util_est_enqueued.max
    195.01           -50.4%      96.73 ± 35%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    550906 ± 19%     +47.6%     813196 ±  9%  sched_debug.cpu.avg_idle.avg
     35231          +137.2%      83557 ± 17%  sched_debug.cpu.clock_task.min
    485.08 ± 14%     +27.4%     617.87 ±  9%  sched_debug.cpu.curr->pid.avg
      1171           +97.3%       2311 ± 12%  sched_debug.cpu.curr->pid.max
    526.49 ±  2%     +54.7%     814.24 ± 12%  sched_debug.cpu.curr->pid.stddev
      4635 ±  2%    +128.9%      10611 ± 17%  sched_debug.cpu.nr_switches.avg
     10915 ± 18%     +90.5%      20797 ± 13%  sched_debug.cpu.nr_switches.max
      1339 ± 19%    +195.5%       3959 ± 27%  sched_debug.cpu.nr_switches.min
      2524 ± 10%     +89.8%       4792 ± 22%  sched_debug.cpu.nr_switches.stddev
     44326          +112.3%      94104 ± 16%  sched_debug.sched_clk
     87.63 ±  5%     -56.1%      38.49        perf-stat.i.MPKI
 1.616e+09           -64.6%  5.725e+08        perf-stat.i.branch-instructions
      5.71 ±  3%      -1.1        4.59        perf-stat.i.branch-miss-rate%
  64667069           -61.2%   25072928        perf-stat.i.branch-misses
      7.54            -1.1        6.47        perf-stat.i.cache-miss-rate%
   6852594           -46.7%    3654271        perf-stat.i.cache-misses
 4.577e+08           -63.0%  1.691e+08        perf-stat.i.cache-references
      1350           +40.7%       1899        perf-stat.i.context-switches
      2.07 ±  4%     -53.8%       0.96        perf-stat.i.cpi
 1.103e+10           -65.4%   3.82e+09        perf-stat.i.cpu-cycles
     29.77 ±  2%     +10.2%      32.81        perf-stat.i.cpu-migrations
      2461 ±  3%     -31.9%       1675        perf-stat.i.cycles-between-cache-misses
      0.39 ±  8%      -0.2        0.15 ±  8%  perf-stat.i.dTLB-load-miss-rate%
    804093 ± 10%     -37.4%     503607 ±  8%  perf-stat.i.dTLB-load-misses
 3.753e+09           -67.1%  1.236e+09        perf-stat.i.dTLB-loads
      0.07 ±  4%      -0.0        0.05 ± 13%  perf-stat.i.dTLB-store-miss-rate%
     74383 ±  2%     -25.9%      55137 ±  9%  perf-stat.i.dTLB-store-misses
 1.026e+09           -65.9%  3.495e+08        perf-stat.i.dTLB-stores
     73.52            -3.8       69.74        perf-stat.i.iTLB-load-miss-rate%
    190642 ±  3%     -15.4%     161312        perf-stat.i.iTLB-load-misses
  1.33e+10           -66.3%  4.476e+09        perf-stat.i.instructions
    107118 ± 17%     -67.2%      35109 ±  5%  perf-stat.i.instructions-per-iTLB-miss
      1.01           +18.4%       1.19        perf-stat.i.ipc
      4.27 ±  2%     -70.3%       1.27        perf-stat.i.major-faults
      0.14 ±  8%    -100.0%       0.00        perf-stat.i.metric.G/sec
      0.69           -65.3%       0.24        perf-stat.i.metric.GHz
    151.41 ± 21%     -73.9%      39.59 ±  2%  perf-stat.i.metric.K/sec
    284.19 ±  4%     -48.8%     145.40        perf-stat.i.metric.M/sec
      6118 ±  4%     -42.6%       3513        perf-stat.i.minor-faults
      1.09 ± 21%     -58.2%       0.46 ± 33%  perf-stat.i.node-load-misses
    243872 ±  2%     -60.2%      97050        perf-stat.i.node-loads
      1.36 ± 20%     -70.9%       0.39 ± 38%  perf-stat.i.node-store-misses
   1154861 ±  5%     -64.6%     409216 ±  3%  perf-stat.i.node-stores
      6123 ±  4%     -42.6%       3514        perf-stat.i.page-faults
     34.42            +9.6%      37.74        perf-stat.overall.MPKI
      4.00            +0.4        4.37        perf-stat.overall.branch-miss-rate%
      1.50            +0.7        2.17        perf-stat.overall.cache-miss-rate%
      0.83            +2.9%       0.85        perf-stat.overall.cpi
      1608           -35.1%       1043        perf-stat.overall.cycles-between-cache-misses
      0.02 ± 10%      +0.0        0.04 ±  9%  perf-stat.overall.dTLB-load-miss-rate%
      0.01 ±  2%      +0.0        0.02 ±  9%  perf-stat.overall.dTLB-store-miss-rate%
     74.02            -4.4       69.67        perf-stat.overall.iTLB-load-miss-rate%
     69782 ±  3%     -60.3%      27693        perf-stat.overall.instructions-per-iTLB-miss
      1.21            -2.8%       1.17        perf-stat.overall.ipc
 1.577e+09           -64.1%  5.665e+08        perf-stat.ps.branch-instructions
  63097630           -60.7%   24776524        perf-stat.ps.branch-misses
   6689008           -45.9%    3620492        perf-stat.ps.cache-misses
 4.465e+08           -62.6%  1.671e+08        perf-stat.ps.cache-references
      1318           +42.7%       1881        perf-stat.ps.context-switches
     15621            +1.7%      15880        perf-stat.ps.cpu-clock
 1.076e+10           -64.9%  3.778e+09        perf-stat.ps.cpu-cycles
     29.06 ±  2%     +11.9%      32.52        perf-stat.ps.cpu-migrations
    784798 ± 10%     -36.4%     499506 ±  8%  perf-stat.ps.dTLB-load-misses
 3.661e+09           -66.6%  1.223e+09        perf-stat.ps.dTLB-loads
     72623 ±  2%     -24.7%      54670 ±  9%  perf-stat.ps.dTLB-store-misses
 1.001e+09           -65.4%  3.463e+08        perf-stat.ps.dTLB-stores
    186142 ±  3%     -14.1%     159974        perf-stat.ps.iTLB-load-misses
 1.297e+10           -65.9%  4.429e+09        perf-stat.ps.instructions
      4.17 ±  2%     -70.0%       1.25        perf-stat.ps.major-faults
      5975 ±  4%     -41.8%       3478        perf-stat.ps.minor-faults
      1.07 ± 21%     -57.7%       0.45 ± 33%  perf-stat.ps.node-load-misses
    238147 ±  2%     -59.6%      96125        perf-stat.ps.node-loads
      1.32 ± 20%     -70.6%       0.39 ± 38%  perf-stat.ps.node-store-misses
   1127035 ±  5%     -64.0%     405385 ±  4%  perf-stat.ps.node-stores
      5979 ±  4%     -41.8%       3479        perf-stat.ps.page-faults
     15621            +1.7%      15880        perf-stat.ps.task-clock
  5.48e+11            +8.0%  5.918e+11        perf-stat.total.instructions
     65.13 ± 17%     -58.5        6.60 ± 24%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     65.13 ± 17%     -58.4        6.70 ± 25%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     16.88 ± 61%     -16.7        0.23 ±141%  perf-profile.calltrace.cycles-pp.exit_mmap.__mmput.exit_mm.do_exit.do_group_exit
     16.60 ± 46%     -16.2        0.45 ± 72%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
     13.65 ± 75%     -13.6        0.00        perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
     13.65 ± 75%     -13.6        0.00        perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
     11.62 ± 77%     -11.4        0.24 ±142%  perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
     11.62 ± 77%     -11.4        0.24 ±142%  perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
     11.62 ± 77%     -11.4        0.24 ±142%  perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
     11.33 ± 78%     -11.3        0.00        perf-profile.calltrace.cycles-pp.zap_pmd_range.unmap_page_range.unmap_vmas.exit_mmap.__mmput
     11.33 ± 78%     -11.3        0.00        perf-profile.calltrace.cycles-pp.zap_pte_range.zap_pmd_range.unmap_page_range.unmap_vmas.exit_mmap
     10.87 ± 53%     -10.9        0.00        perf-profile.calltrace.cycles-pp.do_group_exit.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare
     10.87 ± 53%     -10.9        0.00        perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop
     10.87 ± 53%     -10.9        0.00        perf-profile.calltrace.cycles-pp.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
     10.87 ± 53%     -10.9        0.00        perf-profile.calltrace.cycles-pp.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode
     10.30 ± 61%     -10.3        0.00        perf-profile.calltrace.cycles-pp.exit_mm.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart
     10.30 ± 61%     -10.3        0.00        perf-profile.calltrace.cycles-pp.__mmput.exit_mm.do_exit.do_group_exit.get_signal
      9.48 ±128%      -9.5        0.00        perf-profile.calltrace.cycles-pp.__generic_file_write_iter.generic_file_write_iter.vfs_write.ksys_write.do_syscall_64
      9.48 ±128%      -9.5        0.00        perf-profile.calltrace.cycles-pp.generic_file_write_iter.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      9.48 ±128%      -9.5        0.00        perf-profile.calltrace.cycles-pp.generic_perform_write.__generic_file_write_iter.generic_file_write_iter.vfs_write.ksys_write
      8.10 ± 78%      -8.1        0.00        perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.__mmput.exit_mm.do_exit
      8.10 ± 78%      -8.1        0.00        perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.__mmput.exit_mm
      6.58 ±103%      -6.4        0.23 ±141%  perf-profile.calltrace.cycles-pp.__mmput.exit_mm.do_exit.do_group_exit.__x64_sys_exit_group
      6.58 ±103%      -6.4        0.23 ±141%  perf-profile.calltrace.cycles-pp.exit_mm.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      6.01 ±110%      -6.0        0.00        perf-profile.calltrace.cycles-pp.page_remove_rmap.zap_pte_range.zap_pmd_range.unmap_page_range.unmap_vmas
      5.32 ±105%      -5.3        0.00        perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.__mmput.exec_mmap.begin_new_exec
      5.32 ±105%      -5.1        0.20 ±142%  perf-profile.calltrace.cycles-pp.__mmput.exec_mmap.begin_new_exec.load_elf_binary.search_binary_handler
      5.32 ±105%      -5.1        0.20 ±142%  perf-profile.calltrace.cycles-pp.exit_mmap.__mmput.exec_mmap.begin_new_exec.load_elf_binary
      0.00            +0.8        0.83 ± 34%  perf-profile.calltrace.cycles-pp.lapic_next_deadline.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      0.00            +1.0        0.97 ± 30%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry
      0.00            +1.0        0.99 ± 20%  perf-profile.calltrace.cycles-pp.update_rq_clock_task.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
      0.00            +1.1        1.10 ± 26%  perf-profile.calltrace.cycles-pp.account_user_time.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      0.00            +1.2        1.17 ± 34%  perf-profile.calltrace.cycles-pp.update_curr.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle
      0.00            +1.3        1.26 ± 35%  perf-profile.calltrace.cycles-pp.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.00            +1.3        1.28 ± 36%  perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00            +1.3        1.34 ± 35%  perf-profile.calltrace.cycles-pp.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.00            +1.4        1.41 ± 32%  perf-profile.calltrace.cycles-pp.perf_rotate_context.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      0.00            +1.4        1.45 ± 38%  perf-profile.calltrace.cycles-pp.sync_regs.asm_sysvec_apic_timer_interrupt
      0.00            +1.5        1.52 ± 48%  perf-profile.calltrace.cycles-pp.drm_fb_helper_damage_work.process_one_work.worker_thread.kthread.ret_from_fork
      0.00            +1.5        1.52 ± 48%  perf-profile.calltrace.cycles-pp.drm_fbdev_generic_helper_fb_dirty.drm_fb_helper_damage_work.process_one_work.worker_thread.kthread
      0.00            +1.6        1.60 ± 43%  perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
      0.00            +1.6        1.63 ± 25%  perf-profile.calltrace.cycles-pp.update_load_avg.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle
      0.00            +1.7        1.65 ± 41%  perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
      0.00            +1.9        1.86 ± 26%  perf-profile.calltrace.cycles-pp.asm_sysvec_irq_work
      0.00            +1.9        1.86 ± 26%  perf-profile.calltrace.cycles-pp.sysvec_irq_work.asm_sysvec_irq_work
      0.00            +1.9        1.86 ± 26%  perf-profile.calltrace.cycles-pp.__sysvec_irq_work.sysvec_irq_work.asm_sysvec_irq_work
      0.00            +1.9        1.86 ± 26%  perf-profile.calltrace.cycles-pp.irq_work_run.__sysvec_irq_work.sysvec_irq_work.asm_sysvec_irq_work
      0.00            +2.2        2.21 ± 16%  perf-profile.calltrace.cycles-pp.asm_sysvec_irq_work.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_irq_work
      0.00            +2.2        2.21 ± 16%  perf-profile.calltrace.cycles-pp.sysvec_irq_work.asm_sysvec_irq_work.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode
      0.00            +2.2        2.21 ± 16%  perf-profile.calltrace.cycles-pp.__sysvec_irq_work.sysvec_irq_work.asm_sysvec_irq_work.exit_to_user_mode_loop.exit_to_user_mode_prepare
      0.00            +2.2        2.21 ± 16%  perf-profile.calltrace.cycles-pp.irq_work_run.__sysvec_irq_work.sysvec_irq_work.asm_sysvec_irq_work.exit_to_user_mode_loop
      0.00            +2.2        2.21 ± 16%  perf-profile.calltrace.cycles-pp.irqentry_exit_to_user_mode.asm_sysvec_irq_work
      0.00            +2.2        2.21 ± 16%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_irq_work
      0.00            +2.2        2.21 ± 16%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_irq_work
      0.00            +2.6        2.60 ± 17%  perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      0.00            +3.0        2.96 ± 19%  perf-profile.calltrace.cycles-pp.io_serial_in.wait_for_lsr.serial8250_console_write.console_flush_all.console_unlock
      0.00            +3.6        3.62 ± 19%  perf-profile.calltrace.cycles-pp.wait_for_lsr.serial8250_console_write.console_flush_all.console_unlock.vprintk_emit
      0.00            +4.2        4.19 ± 20%  perf-profile.calltrace.cycles-pp.serial8250_console_write.console_flush_all.console_unlock.vprintk_emit._printk
      0.00            +4.8        4.76 ± 20%  perf-profile.calltrace.cycles-pp.irq_work_run_list.irq_work_run.__sysvec_irq_work.sysvec_irq_work.asm_sysvec_irq_work
      0.00            +4.8        4.76 ± 20%  perf-profile.calltrace.cycles-pp.irq_work_single.irq_work_run_list.irq_work_run.__sysvec_irq_work.sysvec_irq_work
      0.00            +4.8        4.76 ± 20%  perf-profile.calltrace.cycles-pp._printk.irq_work_single.irq_work_run_list.irq_work_run.__sysvec_irq_work
      0.00            +4.8        4.76 ± 20%  perf-profile.calltrace.cycles-pp.vprintk_emit._printk.irq_work_single.irq_work_run_list.irq_work_run
      0.00            +4.8        4.76 ± 20%  perf-profile.calltrace.cycles-pp.console_unlock.vprintk_emit._printk.irq_work_single.irq_work_run_list
      0.00            +4.8        4.76 ± 20%  perf-profile.calltrace.cycles-pp.console_flush_all.console_unlock.vprintk_emit._printk.irq_work_single
      0.00            +6.4        6.43 ±  9%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.00            +6.7        6.70 ±  7%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.00            +9.6        9.56 ± 11%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
      4.86 ±143%     +10.7       15.57 ± 23%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      0.00           +10.9       10.91 ± 13%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      4.86 ±143%     +17.1       21.96 ± 21%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00           +20.0       20.04 ± 25%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00           +20.6       20.64 ± 25%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
     19.32 ± 61%     +22.1       41.42 ± 23%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      0.58 ±223%     +22.2       22.76 ± 26%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.58 ±223%     +25.6       26.14 ± 27%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt
     16.94 ± 66%     +31.3       48.20 ± 17%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
     17.52 ± 65%     +33.4       50.92 ± 17%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     19.32 ± 61%     +33.5       52.83 ± 18%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
     17.52 ± 65%     +34.0       51.54 ± 16%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     17.52 ± 65%     +34.0       51.55 ± 16%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     17.52 ± 65%     +34.0       51.55 ± 16%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     19.90 ± 60%     +37.5       57.38 ± 16%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     65.13 ± 17%     -58.5        6.60 ± 24%  perf-profile.children.cycles-pp.do_syscall_64
     65.13 ± 17%     -58.4        6.70 ± 25%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     22.50 ± 42%     -22.0        0.50 ± 33%  perf-profile.children.cycles-pp.do_group_exit
     22.50 ± 42%     -22.0        0.50 ± 33%  perf-profile.children.cycles-pp.do_exit
     16.60 ± 46%     -16.1        0.50 ± 49%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
     10.87 ± 53%     -10.9        0.00        perf-profile.children.cycles-pp.arch_do_signal_or_restart
     10.87 ± 53%     -10.9        0.00        perf-profile.children.cycles-pp.get_signal
      0.00            +0.1        0.08 ± 11%  perf-profile.children.cycles-pp.perf_event_mmap
      0.00            +0.1        0.10 ± 38%  perf-profile.children.cycles-pp.do_anonymous_page
      0.00            +0.1        0.10 ± 41%  perf-profile.children.cycles-pp.run_timer_softirq
      0.00            +0.1        0.12 ± 61%  perf-profile.children.cycles-pp.__run_timers
      0.00            +0.1        0.12 ± 39%  perf-profile.children.cycles-pp.mas_alloc_nodes
      0.00            +0.1        0.13 ± 46%  perf-profile.children.cycles-pp.rcu_gp_fqs_loop
      0.00            +0.1        0.13 ± 69%  perf-profile.children.cycles-pp.flush_smp_call_function_queue
      0.00            +0.1        0.14 ± 28%  perf-profile.children.cycles-pp.irq_work_needs_cpu
      0.00            +0.1        0.14 ± 49%  perf-profile.children.cycles-pp.schedule_timeout
      0.00            +0.1        0.14 ± 46%  perf-profile.children.cycles-pp.perf_adjust_freq_unthr_context
      0.00            +0.1        0.14 ± 68%  perf-profile.children.cycles-pp.select_task_rq
      0.00            +0.1        0.14 ± 61%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.00            +0.2        0.15 ± 39%  perf-profile.children.cycles-pp.profile_tick
      0.00            +0.2        0.16 ± 94%  perf-profile.children.cycles-pp.enqueue_entity
      0.00            +0.2        0.16 ± 15%  perf-profile.children.cycles-pp.io_serial_out
      0.00            +0.2        0.16 ± 45%  perf-profile.children.cycles-pp.rcu_gp_kthread
      0.00            +0.2        0.16 ± 73%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      0.00            +0.2        0.16 ± 73%  perf-profile.children.cycles-pp.do_vmi_munmap
      0.00            +0.2        0.16 ± 73%  perf-profile.children.cycles-pp.do_vmi_align_munmap
      0.00            +0.2        0.16 ± 43%  perf-profile.children.cycles-pp.futex_wait
      0.00            +0.2        0.17 ± 50%  perf-profile.children.cycles-pp.hrtimer_forward
      0.00            +0.2        0.17 ± 52%  perf-profile.children.cycles-pp.drm_gem_vunmap_unlocked
      0.00            +0.2        0.17 ± 52%  perf-profile.children.cycles-pp.drm_gem_vunmap
      0.00            +0.2        0.18 ± 38%  perf-profile.children.cycles-pp.hrtimer_wakeup
      0.00            +0.2        0.18 ± 50%  perf-profile.children.cycles-pp.__x64_sys_futex
      0.00            +0.2        0.18 ± 50%  perf-profile.children.cycles-pp.do_futex
      0.00            +0.2        0.20 ± 47%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.00            +0.2        0.20 ± 18%  perf-profile.children.cycles-pp.cgroup_rstat_updated
      0.00            +0.2        0.21 ± 53%  perf-profile.children.cycles-pp.drm_client_buffer_vmap
      0.00            +0.2        0.21 ± 41%  perf-profile.children.cycles-pp.number
      0.00            +0.2        0.22 ± 49%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.00            +0.2        0.22 ± 41%  perf-profile.children.cycles-pp.tick_check_oneshot_broadcast_this_cpu
      0.00            +0.2        0.22 ± 68%  perf-profile.children.cycles-pp.error_entry
      0.00            +0.2        0.22 ± 50%  perf-profile.children.cycles-pp.autoremove_wake_function
      0.00            +0.2        0.23 ± 36%  perf-profile.children.cycles-pp.__irqentry_text_end
      0.00            +0.2        0.24 ± 41%  perf-profile.children.cycles-pp.native_apic_mem_write
      0.00            +0.2        0.24 ± 47%  perf-profile.children.cycles-pp.nohz_balancer_kick
      0.00            +0.2        0.25 ± 40%  perf-profile.children.cycles-pp.irqentry_enter
      0.00            +0.3        0.25 ± 46%  perf-profile.children.cycles-pp.__wake_up_common
      0.00            +0.3        0.26 ± 49%  perf-profile.children.cycles-pp.hrtimer_active
      0.00            +0.3        0.26 ± 28%  perf-profile.children.cycles-pp.cpuidle_governor_latency_req
      0.00            +0.3        0.26 ± 38%  perf-profile.children.cycles-pp.task_tick_mm_cid
      0.00            +0.3        0.27 ± 37%  perf-profile.children.cycles-pp.acct_account_cputime
      0.00            +0.3        0.27 ± 64%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.00            +0.3        0.28 ± 48%  perf-profile.children.cycles-pp.rb_insert_color
      0.00            +0.3        0.28 ± 55%  perf-profile.children.cycles-pp.activate_task
      0.00            +0.3        0.28 ± 39%  perf-profile.children.cycles-pp.__wake_up_common_lock
      0.00            +0.3        0.28 ± 64%  perf-profile.children.cycles-pp.trigger_load_balance
      0.00            +0.3        0.28 ± 51%  perf-profile.children.cycles-pp.timekeeping_advance
      0.00            +0.3        0.28 ± 51%  perf-profile.children.cycles-pp.update_wall_time
      0.00            +0.3        0.29 ± 53%  perf-profile.children.cycles-pp.irqtime_account_process_tick
      0.00            +0.3        0.29 ± 97%  perf-profile.children.cycles-pp.drm_atomic_helper_prepare_planes
      0.00            +0.3        0.29 ± 97%  perf-profile.children.cycles-pp.drm_gem_fb_vmap
      0.00            +0.3        0.30 ± 52%  perf-profile.children.cycles-pp.__do_huge_pmd_anonymous_page
      0.00            +0.3        0.30 ± 52%  perf-profile.children.cycles-pp.clear_huge_page
      0.00            +0.3        0.30 ± 48%  perf-profile.children.cycles-pp.account_process_tick
      0.00            +0.3        0.30 ± 50%  perf-profile.children.cycles-pp.update_rq_clock
      0.00            +0.3        0.31 ± 29%  perf-profile.children.cycles-pp.__calc_delta
      0.00            +0.3        0.31 ± 49%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.00            +0.3        0.32 ± 66%  perf-profile.children.cycles-pp.ct_idle_exit
      0.00            +0.3        0.33 ± 62%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.00            +0.3        0.33 ± 72%  perf-profile.children.cycles-pp.sprintf
      0.00            +0.3        0.34 ± 50%  perf-profile.children.cycles-pp.clear_page_erms
      0.00            +0.4        0.37 ± 28%  perf-profile.children.cycles-pp.pipe_write
      0.00            +0.4        0.37 ± 34%  perf-profile.children.cycles-pp.irq_work_tick
      0.00            +0.4        0.37 ± 52%  perf-profile.children.cycles-pp.check_cpu_stall
      0.00            +0.4        0.38 ± 50%  perf-profile.children.cycles-pp.rb_erase
      0.00            +0.4        0.38 ± 62%  perf-profile.children.cycles-pp.show_time_in_state
      0.00            +0.4        0.38 ± 62%  perf-profile.children.cycles-pp.show
      0.00            +0.4        0.38 ± 23%  perf-profile.children.cycles-pp.__cgroup_account_cputime_field
      0.00            +0.4        0.39 ± 60%  perf-profile.children.cycles-pp.vsnprintf
      0.00            +0.4        0.40 ± 42%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.00            +0.4        0.41 ± 59%  perf-profile.children.cycles-pp.perf_event_task_tick
      0.00            +0.4        0.41 ± 42%  perf-profile.children.cycles-pp.hrtimer_update_next_event
      0.00            +0.4        0.42 ± 27%  perf-profile.children.cycles-pp.wait_for_xmitr
      0.00            +0.4        0.42 ± 62%  perf-profile.children.cycles-pp.sysfs_kf_seq_show
      0.00            +0.4        0.42 ± 75%  perf-profile.children.cycles-pp.drm_gem_get_pages
      0.00            +0.4        0.42 ± 75%  perf-profile.children.cycles-pp.shmem_read_mapping_page_gfp
      0.00            +0.4        0.43 ± 25%  perf-profile.children.cycles-pp.dup_mmap
      0.00            +0.5        0.45 ± 23%  perf-profile.children.cycles-pp.dup_mm
      0.00            +0.5        0.46 ± 40%  perf-profile.children.cycles-pp.rb_next
      0.00            +0.5        0.47 ± 27%  perf-profile.children.cycles-pp.try_to_wake_up
      0.00            +0.5        0.47 ± 57%  perf-profile.children.cycles-pp.traverse
      0.00            +0.5        0.48 ± 11%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.00            +0.5        0.49 ± 63%  perf-profile.children.cycles-pp.run_posix_cpu_timers
      0.00            +0.5        0.49 ± 80%  perf-profile.children.cycles-pp.delay_tsc
      0.00            +0.5        0.50 ± 62%  perf-profile.children.cycles-pp.drm_gem_vmap_unlocked
      0.00            +0.5        0.50 ± 62%  perf-profile.children.cycles-pp.drm_gem_vmap
      0.00            +0.5        0.52 ± 31%  perf-profile.children.cycles-pp.lf
      0.00            +0.5        0.52 ± 31%  perf-profile.children.cycles-pp.con_scroll
      0.00            +0.5        0.52 ± 31%  perf-profile.children.cycles-pp.fbcon_scroll
      0.00            +0.5        0.52 ± 31%  perf-profile.children.cycles-pp.fbcon_redraw
      0.00            +0.5        0.52 ± 40%  perf-profile.children.cycles-pp.fbcon_putcs
      0.00            +0.5        0.54 ± 50%  perf-profile.children.cycles-pp.rcu_core
      0.00            +0.6        0.57 ± 30%  perf-profile.children.cycles-pp.vt_console_print
      0.00            +0.6        0.58 ± 38%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.00            +0.6        0.60 ± 45%  perf-profile.children.cycles-pp.update_cfs_group
      0.00            +0.6        0.62 ± 28%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.00            +0.6        0.63 ± 31%  perf-profile.children.cycles-pp.copy_process
      0.00            +0.6        0.64 ± 29%  perf-profile.children.cycles-pp.__do_sys_clone
      0.00            +0.6        0.64 ± 29%  perf-profile.children.cycles-pp.kernel_clone
      0.00            +0.7        0.66 ± 47%  perf-profile.children.cycles-pp.tick_sched_do_timer
      0.00            +0.7        0.67 ± 47%  perf-profile.children.cycles-pp.seq_read_iter
      0.00            +0.7        0.73 ± 32%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.00            +0.7        0.73 ± 37%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.00            +0.7        0.75 ± 32%  perf-profile.children.cycles-pp.native_sched_clock
      0.00            +0.8        0.76 ± 67%  perf-profile.children.cycles-pp.drm_atomic_helper_commit_tail_rpm
      0.00            +0.8        0.76 ± 67%  perf-profile.children.cycles-pp.drm_atomic_helper_commit_planes
      0.00            +0.8        0.76 ± 67%  perf-profile.children.cycles-pp.ast_primary_plane_helper_atomic_update
      0.00            +0.8        0.76 ± 67%  perf-profile.children.cycles-pp.drm_fb_memcpy
      0.00            +0.8        0.76 ± 67%  perf-profile.children.cycles-pp.memcpy_toio
      0.00            +0.8        0.77 ± 27%  perf-profile.children.cycles-pp.rebalance_domains
      0.00            +0.8        0.82 ± 36%  perf-profile.children.cycles-pp._raw_spin_lock
      0.00            +0.9        0.85 ± 61%  perf-profile.children.cycles-pp.commit_tail
      0.00            +0.9        0.88 ± 23%  perf-profile.children.cycles-pp.timerqueue_add
      0.00            +0.9        0.88 ± 53%  perf-profile.children.cycles-pp.rcu_pending
      0.00            +0.9        0.94 ± 39%  perf-profile.children.cycles-pp.tick_irq_enter
      0.00            +1.0        0.96 ± 35%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.00            +1.0        0.96 ± 22%  perf-profile.children.cycles-pp.read_tsc
      0.00            +1.0        0.98 ± 30%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.00            +1.0        1.00 ± 36%  perf-profile.children.cycles-pp.timerqueue_del
      0.00            +1.0        1.00 ± 27%  perf-profile.children.cycles-pp.enqueue_hrtimer
      0.00            +1.0        1.02 ± 15%  perf-profile.children.cycles-pp.update_irq_load_avg
      0.00            +1.0        1.03 ± 31%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.00            +1.1        1.06 ± 29%  perf-profile.children.cycles-pp.arch_scale_freq_tick
      0.00            +1.1        1.10 ± 26%  perf-profile.children.cycles-pp.account_user_time
      0.00            +1.1        1.14 ± 64%  perf-profile.children.cycles-pp.drm_atomic_helper_commit
      0.00            +1.2        1.15 ± 62%  perf-profile.children.cycles-pp.drm_atomic_helper_dirtyfb
      0.00            +1.2        1.16 ± 28%  perf-profile.children.cycles-pp.irq_enter_rcu
      0.00            +1.2        1.16 ± 23%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.00            +1.3        1.33 ± 17%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.00            +1.4        1.36 ± 45%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.00            +1.5        1.52 ± 48%  perf-profile.children.cycles-pp.drm_fb_helper_damage_work
      0.00            +1.5        1.52 ± 48%  perf-profile.children.cycles-pp.drm_fbdev_generic_helper_fb_dirty
      0.00            +1.6        1.57 ± 22%  perf-profile.children.cycles-pp.perf_rotate_context
      0.00            +1.6        1.58 ± 25%  perf-profile.children.cycles-pp.clockevents_program_event
      0.00            +1.6        1.60 ± 43%  perf-profile.children.cycles-pp.process_one_work
      0.00            +1.7        1.65 ± 41%  perf-profile.children.cycles-pp.worker_thread
      0.00            +1.7        1.66 ± 39%  perf-profile.children.cycles-pp.sync_regs
      0.00            +1.9        1.88 ± 23%  perf-profile.children.cycles-pp.update_load_avg
      0.00            +2.7        2.74 ± 20%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.00            +3.6        3.56 ± 16%  perf-profile.children.cycles-pp.io_serial_in
      0.00            +3.9        3.90 ± 32%  perf-profile.children.cycles-pp.irqentry_exit_to_user_mode
      0.00            +4.0        4.04 ± 20%  perf-profile.children.cycles-pp.wait_for_lsr
      0.00            +4.2        4.19 ± 20%  perf-profile.children.cycles-pp.serial8250_console_write
      0.00            +4.8        4.76 ± 20%  perf-profile.children.cycles-pp.asm_sysvec_irq_work
      0.00            +4.8        4.76 ± 20%  perf-profile.children.cycles-pp.sysvec_irq_work
      0.00            +4.8        4.76 ± 20%  perf-profile.children.cycles-pp.__sysvec_irq_work
      0.00            +4.8        4.76 ± 20%  perf-profile.children.cycles-pp.irq_work_run
      0.00            +4.8        4.76 ± 20%  perf-profile.children.cycles-pp.irq_work_single
      0.00            +4.8        4.76 ± 20%  perf-profile.children.cycles-pp._printk
      0.00            +4.8        4.76 ± 20%  perf-profile.children.cycles-pp.vprintk_emit
      0.00            +4.8        4.78 ± 20%  perf-profile.children.cycles-pp.console_unlock
      0.00            +4.8        4.78 ± 20%  perf-profile.children.cycles-pp.console_flush_all
      0.00            +4.9        4.89 ± 20%  perf-profile.children.cycles-pp.irq_work_run_list
      4.86 ±143%     +10.9       15.80 ± 23%  perf-profile.children.cycles-pp.tick_sched_timer
      4.86 ±143%     +17.5       22.32 ± 21%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      4.86 ±143%     +21.9       26.72 ± 20%  perf-profile.children.cycles-pp.hrtimer_interrupt
     19.32 ± 61%     +22.1       41.41 ± 23%  perf-profile.children.cycles-pp.intel_idle
      4.86 ±143%     +22.7       27.60 ± 19%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      5.44 ±123%     +27.2       32.68 ± 17%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      5.44 ±123%     +31.4       36.88 ± 18%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
     17.52 ± 65%     +34.0       51.55 ± 16%  perf-profile.children.cycles-pp.start_secondary
     19.32 ± 61%     +34.4       53.68 ± 18%  perf-profile.children.cycles-pp.cpuidle_enter_state
     19.32 ± 61%     +34.5       53.82 ± 18%  perf-profile.children.cycles-pp.cpuidle_enter
     19.90 ± 60%     +36.8       56.72 ± 17%  perf-profile.children.cycles-pp.cpuidle_idle_call
     19.90 ± 60%     +37.5       57.37 ± 16%  perf-profile.children.cycles-pp.do_idle
     19.90 ± 60%     +37.5       57.38 ± 16%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     19.90 ± 60%     +37.5       57.38 ± 16%  perf-profile.children.cycles-pp.cpu_startup_entry
      6.01 ±110%      -6.0        0.02 ±223%  perf-profile.self.cycles-pp.page_remove_rmap
      0.00            +0.1        0.11 ± 69%  perf-profile.self.cycles-pp.timekeeping_advance
      0.00            +0.1        0.11 ± 40%  perf-profile.self.cycles-pp.irq_work_needs_cpu
      0.00            +0.1        0.13 ± 40%  perf-profile.self.cycles-pp.irq_work_run_list
      0.00            +0.1        0.14 ± 68%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.00            +0.1        0.14 ± 46%  perf-profile.self.cycles-pp.perf_adjust_freq_unthr_context
      0.00            +0.1        0.14 ± 20%  perf-profile.self.cycles-pp.tick_irq_enter
      0.00            +0.1        0.14 ± 71%  perf-profile.self.cycles-pp.nohz_balancer_kick
      0.00            +0.1        0.15 ± 56%  perf-profile.self.cycles-pp.number
      0.00            +0.2        0.15 ± 39%  perf-profile.self.cycles-pp.profile_tick
      0.00            +0.2        0.16 ± 15%  perf-profile.self.cycles-pp.io_serial_out
      0.00            +0.2        0.17 ± 50%  perf-profile.self.cycles-pp.hrtimer_forward
      0.00            +0.2        0.20 ± 51%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.00            +0.2        0.20 ± 18%  perf-profile.self.cycles-pp.cgroup_rstat_updated
      0.00            +0.2        0.21 ± 38%  perf-profile.self.cycles-pp.timerqueue_del
      0.00            +0.2        0.22 ± 41%  perf-profile.self.cycles-pp.tick_check_oneshot_broadcast_this_cpu
      0.00            +0.2        0.22 ± 68%  perf-profile.self.cycles-pp.error_entry
      0.00            +0.2        0.23 ± 45%  perf-profile.self.cycles-pp.reweight_entity
      0.00            +0.2        0.23 ± 36%  perf-profile.self.cycles-pp.__irqentry_text_end
      0.00            +0.2        0.24 ± 41%  perf-profile.self.cycles-pp.native_apic_mem_write
      0.00            +0.2        0.25 ± 47%  perf-profile.self.cycles-pp.hrtimer_active
      0.00            +0.2        0.25 ± 62%  perf-profile.self.cycles-pp.rb_insert_color
      0.00            +0.3        0.25 ± 38%  perf-profile.self.cycles-pp.task_tick_mm_cid
      0.00            +0.3        0.27 ± 37%  perf-profile.self.cycles-pp.acct_account_cputime
      0.00            +0.3        0.27 ± 69%  perf-profile.self.cycles-pp.trigger_load_balance
      0.00            +0.3        0.28 ± 28%  perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
      0.00            +0.3        0.29 ± 53%  perf-profile.self.cycles-pp.irqtime_account_process_tick
      0.00            +0.3        0.30 ± 48%  perf-profile.self.cycles-pp.account_process_tick
      0.00            +0.3        0.31 ± 29%  perf-profile.self.cycles-pp.__calc_delta
      0.00            +0.3        0.31 ± 37%  perf-profile.self.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.00            +0.3        0.32 ± 62%  perf-profile.self.cycles-pp.cpuidle_idle_call
      0.00            +0.3        0.32 ± 39%  perf-profile.self.cycles-pp.irq_work_tick
      0.00            +0.3        0.34 ± 50%  perf-profile.self.cycles-pp.clear_page_erms
      0.00            +0.4        0.36 ± 52%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.00            +0.4        0.37 ± 52%  perf-profile.self.cycles-pp.check_cpu_stall
      0.00            +0.4        0.38 ± 50%  perf-profile.self.cycles-pp.rb_erase
      0.00            +0.4        0.39 ± 54%  perf-profile.self.cycles-pp.rcu_sched_clock_irq
      0.00            +0.4        0.41 ± 59%  perf-profile.self.cycles-pp.rcu_pending
      0.00            +0.4        0.42 ± 13%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.00            +0.4        0.44 ± 70%  perf-profile.self.cycles-pp.task_tick_fair
      0.00            +0.4        0.44 ± 33%  perf-profile.self.cycles-pp.memcpy_toio
      0.00            +0.5        0.46 ± 40%  perf-profile.self.cycles-pp.rb_next
      0.00            +0.5        0.46 ± 26%  perf-profile.self.cycles-pp.irqtime_account_irq
      0.00            +0.5        0.49 ± 63%  perf-profile.self.cycles-pp.run_posix_cpu_timers
      0.00            +0.5        0.49 ± 80%  perf-profile.self.cycles-pp.delay_tsc
      0.00            +0.5        0.52 ± 56%  perf-profile.self.cycles-pp.update_process_times
      0.00            +0.5        0.55 ± 41%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.00            +0.6        0.57 ± 40%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.00            +0.6        0.60 ± 24%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.00            +0.6        0.60 ± 45%  perf-profile.self.cycles-pp.update_cfs_group
      0.00            +0.6        0.63 ± 26%  perf-profile.self.cycles-pp.timerqueue_add
      0.00            +0.6        0.63 ± 26%  perf-profile.self.cycles-pp.account_user_time
      0.00            +0.6        0.65 ± 21%  perf-profile.self.cycles-pp.__sysvec_apic_timer_interrupt
      0.00            +0.7        0.67 ± 17%  perf-profile.self.cycles-pp.perf_rotate_context
      0.00            +0.7        0.73 ± 33%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.00            +0.7        0.73 ± 30%  perf-profile.self.cycles-pp.native_sched_clock
      0.00            +0.8        0.75 ± 35%  perf-profile.self.cycles-pp.update_load_avg
      0.00            +0.8        0.81 ± 35%  perf-profile.self.cycles-pp._raw_spin_lock
      0.00            +0.8        0.84 ± 40%  perf-profile.self.cycles-pp.scheduler_tick
      0.00            +0.9        0.89 ± 28%  perf-profile.self.cycles-pp.hrtimer_interrupt
      0.00            +0.9        0.94 ± 25%  perf-profile.self.cycles-pp.read_tsc
      0.00            +1.0        1.02 ± 15%  perf-profile.self.cycles-pp.update_irq_load_avg
      0.00            +1.1        1.06 ± 29%  perf-profile.self.cycles-pp.arch_scale_freq_tick
      0.00            +1.2        1.16 ± 23%  perf-profile.self.cycles-pp.lapic_next_deadline
      0.00            +1.2        1.25 ± 25%  perf-profile.self.cycles-pp.__hrtimer_run_queues
      0.00            +1.3        1.27 ± 23%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.00            +1.3        1.33 ± 17%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.00            +1.7        1.66 ± 39%  perf-profile.self.cycles-pp.sync_regs
      0.00            +3.6        3.56 ± 16%  perf-profile.self.cycles-pp.io_serial_in
     19.32 ± 61%     +22.1       41.41 ± 23%  perf-profile.self.cycles-pp.intel_idle



***************************************************************************************************
lkp-cfl-e1: 16 threads 1 sockets Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz (Coffee Lake) with 32G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/option_a/rootfs/tbox_group/test/testcase:
  gcc-11/performance/x86_64-rhel-8.3/pickle_pure_python/debian-x86_64-phoronix/lkp-cfl-e1/pyperformance-1.0.2/phoronix-test-suite

commit: 
  a525b95417 ("cpufreq/amd-pstate: Remove fast_switch_possible flag from active driver")
  278dfbd17c ("cpufreq: Return failure if fast_switch is not set and fast_switch_possible is set")

a525b954173e74cd 278dfbd17c5d0767834471d7fea 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :6          100%           6:6     kmsg.cpufreq:fast_switch_possible_is_enabled_but_fast_switch_callback_is_not_set
         %stddev     %change         %stddev
             \          |                \  
    391.00           -99.4%       2.22        phoronix-test-suite.pyperformance.pickle_pure_python.milliseconds
     91.97           +70.0%     156.34        phoronix-test-suite.time.elapsed_time
     91.97           +70.0%     156.34        phoronix-test-suite.time.elapsed_time.max
     75.83           +10.8%      84.00        phoronix-test-suite.time.percent_of_cpu_this_job_got
     68.51           +78.7%     122.45        phoronix-test-suite.time.user_time
     44.67 ±  3%      +9.1%      48.74        boot-time.boot
    578.11 ±  2%      +6.0%     612.59        boot-time.idle
 1.408e+09           +57.1%  2.211e+09        cpuidle..time
   1492421           +64.8%    2460130        cpuidle..usage
      6896           +18.6%       8177        meminfo.Active(anon)
    121274 ±  9%     +14.4%     138735 ±  2%  meminfo.AnonHugePages
    139.63           +49.3%     208.46        uptime.boot
      1999           +43.0%       2859        uptime.idle
     93.00            -5.7%      87.67        vmstat.cpu.id
      1112 ±  4%     -40.6%     660.50 ±  2%  vmstat.io.bi
      1468           +38.0%       2026        vmstat.system.cs
      0.30 ±  5%      -0.1        0.17 ±  2%  mpstat.cpu.all.iowait%
      0.59            +2.5        3.06        mpstat.cpu.all.irq%
      0.04 ±  7%      +0.2        0.20        mpstat.cpu.all.soft%
      0.29            +0.5        0.83        mpstat.cpu.all.sys%
      5.09            +2.5        7.59        mpstat.cpu.all.usr%
      1723           +18.6%       2044        proc-vmstat.nr_active_anon
     92126            -1.7%      90534        proc-vmstat.nr_anon_pages
      6684            +1.6%       6789        proc-vmstat.nr_kernel_stack
     15983            -2.6%      15569        proc-vmstat.nr_slab_unreclaimable
      1723           +18.6%       2044        proc-vmstat.nr_zone_active_anon
    691105           +15.3%     796970        proc-vmstat.numa_hit
    691096           +15.3%     796942        proc-vmstat.numa_local
     16545            +4.3%      17256        proc-vmstat.pgactivate
    748098           +17.2%     876972        proc-vmstat.pgalloc_normal
   1113450           +14.9%    1279681        proc-vmstat.pgfault
    696774           +18.2%     823764        proc-vmstat.pgfree
    347697            +8.8%     378312        proc-vmstat.pgreuse
    869760           +57.1%    1366784        proc-vmstat.unevictable_pgs_scanned
     78.73           +21.3%      95.53        turbostat.Any%C0
    279.33           -62.2%     105.50        turbostat.Avg_MHz
      6.12            +7.1       13.20        turbostat.Busy%
      4569           -82.5%     800.00        turbostat.Bzy_MHz
      2894 ±  6%   +1657.0%      50860        turbostat.C1
      0.01            +0.0        0.03        turbostat.C1%
     11256 ±  3%    +332.0%      48630        turbostat.C1E
     14141 ±  2%     +61.5%      22838 ±  3%  turbostat.C3
      0.22 ±  3%      -0.1        0.15        turbostat.C3%
    499577          +124.0%    1118904        turbostat.C6
     31.51            +9.4       40.94        turbostat.C6%
    937249           +30.0%    1218418        turbostat.C8
     61.61           -15.6       46.02        turbostat.C8%
      0.28 ± 12%     -28.7%       0.20 ±  2%  turbostat.CPU%c3
     31.36           +58.4%      49.69        turbostat.CPU%c6
     46.27           -54.4%      21.11        turbostat.CPU%c7
     24.55           -92.1%       1.95        turbostat.CorWatt
     52.50 ±  7%     -42.9%      30.00        turbostat.CoreTmp
   1519070           +68.8%    2564565        turbostat.IRQ
     26254           -93.4%       1721 ± 24%  turbostat.POLL
      3.21 ±  4%     -86.1%       0.44 ±  4%  turbostat.Pkg%pc2
     10.04 ±  2%     -62.6%       3.76 ±  2%  turbostat.Pkg%pc3
     51.83 ±  9%     -40.8%      30.67        turbostat.PkgTmp
     25.03           -90.3%       2.43        turbostat.PkgWatt
     97.03           +99.1%     193.14        turbostat.Totl%C0
      0.52 ± 23%     -18.7%       0.42 ±  8%  sched_debug.cfs_rq:/.h_nr_running.stddev
      9607 ± 15%    +152.8%      24290 ±  8%  sched_debug.cfs_rq:/.min_vruntime.avg
     16594 ± 20%    +249.3%      57957 ± 15%  sched_debug.cfs_rq:/.min_vruntime.max
      6011 ± 18%     +79.8%      10806 ±  6%  sched_debug.cfs_rq:/.min_vruntime.min
      2916 ± 29%    +387.4%      14213 ± 19%  sched_debug.cfs_rq:/.min_vruntime.stddev
    666.91 ± 29%     -47.1%     352.68 ± 21%  sched_debug.cfs_rq:/.runnable_avg.avg
    229.00 ± 93%     -78.1%      50.08 ± 41%  sched_debug.cfs_rq:/.runnable_avg.min
      3691 ± 73%    +741.5%      31063 ± 41%  sched_debug.cfs_rq:/.spread0.max
     -6899          +133.6%     -16116        sched_debug.cfs_rq:/.spread0.min
      2917 ± 29%    +387.4%      14220 ± 19%  sched_debug.cfs_rq:/.spread0.stddev
    637.35 ± 27%     -46.9%     338.25 ± 21%  sched_debug.cfs_rq:/.util_avg.avg
    212.83 ± 90%     -79.1%      44.39 ± 50%  sched_debug.cfs_rq:/.util_avg.min
    730.42 ± 16%     -73.8%     191.67 ± 23%  sched_debug.cfs_rq:/.util_est_enqueued.max
    186.21 ± 19%     -66.5%      62.29 ± 21%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    264982 ± 37%     +89.7%     502726 ± 12%  sched_debug.cpu.avg_idle.min
     51644 ± 24%     +76.5%      91142 ± 16%  sched_debug.cpu.clock.avg
     51645 ± 24%     +76.5%      91144 ± 16%  sched_debug.cpu.clock.max
     51643 ± 24%     +76.5%      91140 ± 16%  sched_debug.cpu.clock.min
     50245 ± 24%     +76.1%      88500 ± 15%  sched_debug.cpu.clock_task.avg
     51446 ± 23%     +74.5%      89762 ± 15%  sched_debug.cpu.clock_task.max
     40424 ± 26%     +95.8%      79135 ± 17%  sched_debug.cpu.clock_task.min
      1323 ± 25%     +73.1%       2290 ± 12%  sched_debug.cpu.curr->pid.max
    515.04 ± 26%     +53.7%     791.50 ± 13%  sched_debug.cpu.curr->pid.stddev
      4983 ± 22%    +104.8%      10204 ± 16%  sched_debug.cpu.nr_switches.avg
     11553 ± 42%    +110.0%      24267 ± 14%  sched_debug.cpu.nr_switches.max
      1502 ± 29%    +147.1%       3713 ± 17%  sched_debug.cpu.nr_switches.min
      2809 ± 35%     +93.1%       5425 ± 15%  sched_debug.cpu.nr_switches.stddev
     51643 ± 24%     +76.5%      91140 ± 16%  sched_debug.cpu_clk
     50769 ± 24%     +77.8%      90265 ± 16%  sched_debug.ktime
     49225 ± 22%     +81.8%      89499 ± 15%  sched_debug.sched_clk
     44.71 ±  4%     -14.6%      38.20        perf-stat.i.MPKI
 1.929e+09           -76.9%  4.446e+08        perf-stat.i.branch-instructions
      2.33            +0.4        2.78        perf-stat.i.branch-miss-rate%
  17917862           -33.9%   11845681        perf-stat.i.branch-misses
      3.39 ±  3%      +1.6        5.04        perf-stat.i.cache-miss-rate%
   3279255 ±  3%      -6.9%    3053819        perf-stat.i.cache-misses
 1.045e+08           -41.7%   60884324        perf-stat.i.cache-references
      1373           +43.8%       1974        perf-stat.i.context-switches
      1.15 ±  2%     -27.1%       0.84        perf-stat.i.cpi
  4.39e+09           -65.5%  1.515e+09        perf-stat.i.cpu-cycles
     24.91            -5.4%      23.57 ±  2%  perf-stat.i.cpu-migrations
      1930 ±  5%     -66.8%     640.60        perf-stat.i.cycles-between-cache-misses
      0.19 ±  4%      -0.1        0.13 ±  2%  perf-stat.i.dTLB-load-miss-rate%
    567898           -11.4%     503424        perf-stat.i.dTLB-load-misses
 2.756e+09           -79.0%  5.784e+08        perf-stat.i.dTLB-loads
      0.04 ±  5%      -0.0        0.03        perf-stat.i.dTLB-store-miss-rate%
     81667           -21.4%      64183        perf-stat.i.dTLB-store-misses
 1.267e+09           -79.2%  2.639e+08        perf-stat.i.dTLB-stores
     44.98            +1.2       46.14        perf-stat.i.iTLB-load-miss-rate%
    548983 ±  2%     -52.0%     263472        perf-stat.i.iTLB-load-misses
    900745           -62.6%     336446        perf-stat.i.iTLB-loads
 8.909e+09           -76.8%   2.07e+09        perf-stat.i.instructions
     15290           -48.3%       7907        perf-stat.i.instructions-per-iTLB-miss
      1.81           -25.5%       1.35        perf-stat.i.ipc
      2.37           -46.6%       1.26 ±  2%  perf-stat.i.major-faults
      0.27           -65.5%       0.09        perf-stat.i.metric.GHz
    117.79 ±  5%     -59.4%      47.79        perf-stat.i.metric.K/sec
    378.49           -77.7%      84.21        perf-stat.i.metric.M/sec
     11121           -30.1%       7775        perf-stat.i.minor-faults
      7.42 ± 10%     -36.3%       4.73 ±  4%  perf-stat.i.node-load-misses
    177094 ±  3%     -34.2%     116568        perf-stat.i.node-loads
      8.67 ± 19%     -40.8%       5.13 ± 12%  perf-stat.i.node-store-misses
    364565 ±  4%     -32.5%     246258        perf-stat.i.node-stores
     11123           -30.1%       7776        perf-stat.i.page-faults
     11.73          +151.2%      29.48        perf-stat.overall.MPKI
      0.93            +1.7        2.66        perf-stat.overall.branch-miss-rate%
      3.14 ±  2%      +1.9        5.01        perf-stat.overall.cache-miss-rate%
      0.49           +48.6%       0.73        perf-stat.overall.cpi
      1339 ±  3%     -63.0%     496.35        perf-stat.overall.cycles-between-cache-misses
      0.02            +0.1        0.09        perf-stat.overall.dTLB-load-miss-rate%
      0.01            +0.0        0.02        perf-stat.overall.dTLB-store-miss-rate%
     37.87            +6.0       43.87        perf-stat.overall.iTLB-load-miss-rate%
     16234 ±  2%     -51.7%       7839        perf-stat.overall.instructions-per-iTLB-miss
      2.03           -32.7%       1.37        perf-stat.overall.ipc
 1.907e+09           -76.9%  4.412e+08        perf-stat.ps.branch-instructions
  17720496           -33.7%   11756414        perf-stat.ps.branch-misses
   3243482 ±  3%      -6.5%    3031172        perf-stat.ps.cache-misses
 1.034e+08           -41.4%   60556271        perf-stat.ps.cache-references
      1358           +44.3%       1959        perf-stat.ps.context-switches
 4.341e+09           -65.3%  1.504e+09        perf-stat.ps.cpu-cycles
     24.65            -4.8%      23.46 ±  2%  perf-stat.ps.cpu-migrations
    561792           -11.0%     500237        perf-stat.ps.dTLB-load-misses
 2.726e+09           -78.9%   5.74e+08        perf-stat.ps.dTLB-loads
     80786           -21.0%      63822        perf-stat.ps.dTLB-store-misses
 1.253e+09           -79.1%   2.62e+08        perf-stat.ps.dTLB-stores
    542990 ±  2%     -51.7%     262035        perf-stat.ps.iTLB-load-misses
    890835           -62.4%     335305        perf-stat.ps.iTLB-loads
 8.811e+09           -76.7%  2.054e+09        perf-stat.ps.instructions
      2.34           -46.6%       1.25 ±  2%  perf-stat.ps.major-faults
     11000           -29.7%       7730        perf-stat.ps.minor-faults
      7.34 ± 10%     -36.0%       4.70 ±  5%  perf-stat.ps.node-load-misses
    175176 ±  3%     -33.9%     115833        perf-stat.ps.node-loads
      8.57 ± 19%     -40.4%       5.11 ± 13%  perf-stat.ps.node-store-misses
    360620 ±  4%     -31.9%     245454        perf-stat.ps.node-stores
     11002           -29.7%       7731        perf-stat.ps.page-faults
 8.234e+11           -60.7%  3.233e+11        perf-stat.total.instructions
     51.62 ± 14%     -47.4        4.20 ± 10%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     51.62 ± 14%     -47.4        4.22 ± 10%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     12.88 ± 77%     -12.9        0.00        perf-profile.calltrace.cycles-pp.exit_mmap.__mmput.exit_mm.do_exit.do_group_exit
     11.55 ± 64%     -11.6        0.00        perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
     12.09 ± 63%     -10.5        1.56 ±  8%  perf-profile.calltrace.cycles-pp.asm_exc_page_fault
     10.37 ± 94%     -10.4        0.00        perf-profile.calltrace.cycles-pp.zap_pte_range.zap_pmd_range.unmap_page_range.unmap_vmas.exit_mmap
     10.36 ± 92%     -10.4        0.00        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.36 ± 92%     -10.4        0.00        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.04 ± 66%     -10.0        0.00        perf-profile.calltrace.cycles-pp.do_group_exit.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare
     10.04 ± 66%     -10.0        0.00        perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop
      8.93 ± 82%      -8.9        0.00        perf-profile.calltrace.cycles-pp.zap_pmd_range.unmap_page_range.unmap_vmas.exit_mmap.__mmput
      9.15 ± 58%      -8.7        0.47 ± 71%  perf-profile.calltrace.cycles-pp.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.53 ± 91%      -8.5        0.00        perf-profile.calltrace.cycles-pp.exit_mm.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart
      8.53 ± 91%      -8.5        0.00        perf-profile.calltrace.cycles-pp.__mmput.exit_mm.do_exit.do_group_exit.get_signal
      9.15 ± 58%      -8.4        0.73 ± 18%  perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      9.15 ± 58%      -8.4        0.73 ± 18%  perf-profile.calltrace.cycles-pp.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      7.70 ± 84%      -7.7        0.00        perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      7.70 ± 84%      -7.7        0.00        perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      7.18 ± 84%      -7.2        0.00        perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      7.18 ± 84%      -7.2        0.00        perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      7.18 ± 84%      -7.2        0.00        perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      7.01 ± 90%      -7.0        0.00        perf-profile.calltrace.cycles-pp.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      7.01 ± 90%      -7.0        0.00        perf-profile.calltrace.cycles-pp.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode
      5.99 ± 92%      -6.0        0.00        perf-profile.calltrace.cycles-pp.generic_file_write_iter.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.99 ± 92%      -6.0        0.00        perf-profile.calltrace.cycles-pp.__generic_file_write_iter.generic_file_write_iter.vfs_write.ksys_write.do_syscall_64
      5.99 ± 92%      -6.0        0.00        perf-profile.calltrace.cycles-pp.generic_perform_write.__generic_file_write_iter.generic_file_write_iter.vfs_write.ksys_write
      5.90 ± 68%      -5.9        0.00        perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.__mmput.exit_mm.do_exit
      5.90 ± 68%      -5.9        0.00        perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.__mmput.exit_mm
      0.00            +0.7        0.70 ±  6%  perf-profile.calltrace.cycles-pp.timerqueue_del.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      0.00            +0.7        0.71 ± 20%  perf-profile.calltrace.cycles-pp.rcu_sched_clock_irq.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      0.00            +0.7        0.71 ± 19%  perf-profile.calltrace.cycles-pp.enqueue_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      0.00            +0.8        0.77 ± 16%  perf-profile.calltrace.cycles-pp.load_balance.rebalance_domains.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt
      0.00            +0.8        0.79 ± 24%  perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle
      0.00            +0.8        0.80 ± 16%  perf-profile.calltrace.cycles-pp.update_blocked_averages.run_rebalance_domains.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt
      0.00            +0.8        0.80 ± 23%  perf-profile.calltrace.cycles-pp.perf_rotate_context.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      0.00            +0.8        0.85 ± 16%  perf-profile.calltrace.cycles-pp.run_rebalance_domains.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00            +1.0        0.97 ± 16%  perf-profile.calltrace.cycles-pp.rebalance_domains.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00            +1.2        1.20 ± 25%  perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      0.00            +1.4        1.41 ± 36%  perf-profile.calltrace.cycles-pp.serial8250_console_write.console_flush_all.console_unlock.vprintk_emit._printk
      0.00            +1.4        1.42 ± 26%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry
      0.00            +1.5        1.46 ± 20%  perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.00            +1.5        1.50 ± 18%  perf-profile.calltrace.cycles-pp.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.00            +1.7        1.71 ± 23%  perf-profile.calltrace.cycles-pp.vprintk_emit._printk.irq_work_single.irq_work_run_list.irq_work_run
      0.00            +1.7        1.71 ± 23%  perf-profile.calltrace.cycles-pp.console_unlock.vprintk_emit._printk.irq_work_single.irq_work_run_list
      0.00            +1.7        1.71 ± 23%  perf-profile.calltrace.cycles-pp.console_flush_all.console_unlock.vprintk_emit._printk.irq_work_single
      0.00            +1.7        1.71 ± 23%  perf-profile.calltrace.cycles-pp.irq_work_run_list.irq_work_run.__sysvec_irq_work.sysvec_irq_work.asm_sysvec_irq_work
      0.00            +1.7        1.71 ± 23%  perf-profile.calltrace.cycles-pp.irq_work_single.irq_work_run_list.irq_work_run.__sysvec_irq_work.sysvec_irq_work
      0.00            +1.7        1.71 ± 23%  perf-profile.calltrace.cycles-pp._printk.irq_work_single.irq_work_run_list.irq_work_run.__sysvec_irq_work
      0.00            +2.3        2.32 ± 10%  perf-profile.calltrace.cycles-pp.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.00            +2.4        2.37 ± 19%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      0.00            +2.8        2.85 ± 10%  perf-profile.calltrace.cycles-pp.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.00            +3.5        3.52 ± 20%  perf-profile.calltrace.cycles-pp.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
      0.00            +4.3        4.27 ±  9%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      0.00            +4.8        4.76 ±  8%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      0.00            +5.4        5.42 ±  6%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      0.00            +5.6        5.62 ± 82%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.rest_init
      0.00            +5.8        5.83 ± 79%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.rest_init.arch_call_rest_init
      0.00            +5.9        5.90 ± 79%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.rest_init.arch_call_rest_init.start_kernel.x86_64_start_reservations
      0.00            +5.9        5.90 ± 79%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.rest_init.arch_call_rest_init.start_kernel
      0.00            +5.9        5.90 ± 79%  perf-profile.calltrace.cycles-pp.x86_64_start_kernel.secondary_startup_64_no_verify
      0.00            +5.9        5.90 ± 79%  perf-profile.calltrace.cycles-pp.x86_64_start_reservations.x86_64_start_kernel.secondary_startup_64_no_verify
      0.00            +5.9        5.90 ± 79%  perf-profile.calltrace.cycles-pp.start_kernel.x86_64_start_reservations.x86_64_start_kernel.secondary_startup_64_no_verify
      0.00            +5.9        5.90 ± 79%  perf-profile.calltrace.cycles-pp.arch_call_rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel.secondary_startup_64_no_verify
      0.00            +5.9        5.90 ± 79%  perf-profile.calltrace.cycles-pp.rest_init.arch_call_rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel
      0.00            +9.2        9.21 ±  2%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00           +11.2       11.19 ±  3%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.00           +11.7       11.74 ±  3%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.00           +18.1       18.08        perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
      0.00           +20.5       20.55 ±  3%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     21.44 ± 35%     +40.0       61.39 ±  2%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     25.50 ± 46%     +53.7       79.24 ±  5%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
     25.50 ± 46%     +57.7       83.24        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
     25.50 ± 46%     +58.2       83.74 ±  5%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     25.58 ± 45%     +59.2       84.78 ±  4%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     25.58 ± 45%     +59.2       84.82 ±  4%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     25.58 ± 45%     +59.2       84.82 ±  4%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     25.66 ± 45%     +65.0       90.72        perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     51.62 ± 14%     -47.4        4.21 ± 10%  perf-profile.children.cycles-pp.do_syscall_64
     51.62 ± 14%     -47.4        4.22 ± 10%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     17.21 ± 61%     -16.9        0.28 ± 20%  perf-profile.children.cycles-pp.do_group_exit
     17.21 ± 61%     -16.9        0.28 ± 20%  perf-profile.children.cycles-pp.do_exit
     17.42 ± 68%     -16.9        0.51 ± 10%  perf-profile.children.cycles-pp.__mmput
     17.34 ± 69%     -16.8        0.51 ± 10%  perf-profile.children.cycles-pp.exit_mmap
     13.04 ± 75%     -12.8        0.22 ± 30%  perf-profile.children.cycles-pp.exit_mm
     11.55 ± 64%     -11.2        0.39 ± 30%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
     12.90 ± 53%     -11.1        1.78 ± 12%  perf-profile.children.cycles-pp.asm_exc_page_fault
     10.04 ± 66%     -10.0        0.00        perf-profile.children.cycles-pp.get_signal
     10.04 ± 66%     -10.0        0.01 ±223%  perf-profile.children.cycles-pp.arch_do_signal_or_restart
      9.15 ± 58%      -8.5        0.62 ± 21%  perf-profile.children.cycles-pp.bprm_execve
      9.15 ± 58%      -8.4        0.73 ± 18%  perf-profile.children.cycles-pp.__x64_sys_execve
      9.15 ± 58%      -8.4        0.73 ± 18%  perf-profile.children.cycles-pp.do_execveat_common
      5.99 ± 92%      -6.0        0.00        perf-profile.children.cycles-pp.generic_perform_write
      5.99 ± 92%      -6.0        0.01 ±223%  perf-profile.children.cycles-pp.__generic_file_write_iter
      5.99 ± 92%      -6.0        0.01 ±223%  perf-profile.children.cycles-pp.generic_file_write_iter
      0.00            +0.1        0.09 ± 24%  perf-profile.children.cycles-pp.mas_store_prealloc
      0.00            +0.1        0.10 ± 38%  perf-profile.children.cycles-pp.hrtimer_get_next_event
      0.00            +0.1        0.10 ± 19%  perf-profile.children.cycles-pp.sprintf
      0.00            +0.1        0.11 ± 23%  perf-profile.children.cycles-pp.show
      0.00            +0.1        0.11 ± 23%  perf-profile.children.cycles-pp.show_time_in_state
      0.00            +0.1        0.12 ± 24%  perf-profile.children.cycles-pp.traverse
      0.00            +0.1        0.12 ± 28%  perf-profile.children.cycles-pp.hrtimer_wakeup
      0.00            +0.1        0.13 ± 17%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.00            +0.1        0.13 ± 19%  perf-profile.children.cycles-pp.irq_work_needs_cpu
      0.00            +0.1        0.13 ± 28%  perf-profile.children.cycles-pp.calc_global_load_tick
      0.00            +0.1        0.14 ± 36%  perf-profile.children.cycles-pp.pm_qos_read_value
      0.00            +0.1        0.14 ± 48%  perf-profile.children.cycles-pp.cpuidle_not_available
      0.00            +0.1        0.14 ± 33%  perf-profile.children.cycles-pp.schedule_idle
      0.00            +0.1        0.14 ± 34%  perf-profile.children.cycles-pp.sysfs_kf_seq_show
      0.00            +0.1        0.14 ± 45%  perf-profile.children.cycles-pp.rcu_nocb_flush_deferred_wakeup
      0.00            +0.1        0.15 ± 45%  perf-profile.children.cycles-pp.trigger_load_balance
      0.00            +0.2        0.16 ± 43%  perf-profile.children.cycles-pp.enqueue_entity
      0.00            +0.2        0.16 ± 38%  perf-profile.children.cycles-pp.call_cpuidle
      0.00            +0.2        0.17 ± 53%  perf-profile.children.cycles-pp.ct_irq_enter
      0.00            +0.2        0.17 ± 39%  perf-profile.children.cycles-pp.x86_pmu_disable
      0.00            +0.2        0.18 ± 48%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.00            +0.2        0.18 ± 43%  perf-profile.children.cycles-pp.update_load_avg
      0.00            +0.2        0.19 ± 33%  perf-profile.children.cycles-pp.tick_nohz_tick_stopped
      0.00            +0.2        0.19 ± 22%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.00            +0.2        0.20 ± 47%  perf-profile.children.cycles-pp.task_tick_mm_cid
      0.00            +0.2        0.20 ± 39%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.00            +0.2        0.21 ± 36%  perf-profile.children.cycles-pp.activate_task
      0.00            +0.2        0.22 ± 54%  perf-profile.children.cycles-pp.update_wall_time
      0.00            +0.2        0.22 ± 54%  perf-profile.children.cycles-pp.timekeeping_advance
      0.00            +0.2        0.22 ± 27%  perf-profile.children.cycles-pp.rb_insert_color
      0.00            +0.2        0.22 ± 30%  perf-profile.children.cycles-pp.ct_kernel_exit
      0.00            +0.2        0.22 ± 27%  perf-profile.children.cycles-pp.ct_nmi_enter
      0.00            +0.2        0.23 ± 24%  perf-profile.children.cycles-pp.rb_next
      0.00            +0.2        0.23 ± 27%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.00            +0.2        0.24 ± 76%  perf-profile.children.cycles-pp.copy_page_to_iter
      0.00            +0.2        0.24 ± 76%  perf-profile.children.cycles-pp._copy_to_iter
      0.00            +0.2        0.24 ± 76%  perf-profile.children.cycles-pp.copyout
      0.00            +0.2        0.25 ± 32%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      0.00            +0.2        0.25 ± 32%  perf-profile.children.cycles-pp.rcu_core
      0.00            +0.3        0.26 ± 30%  perf-profile.children.cycles-pp.irq_work_tick
      0.00            +0.3        0.27 ± 29%  perf-profile.children.cycles-pp.try_to_wake_up
      0.00            +0.3        0.27 ± 47%  perf-profile.children.cycles-pp.get_cpu_device
      0.00            +0.3        0.28 ± 27%  perf-profile.children.cycles-pp.check_cpu_stall
      0.00            +0.3        0.30 ± 50%  perf-profile.children.cycles-pp.irqtime_account_process_tick
      0.00            +0.3        0.31 ± 73%  perf-profile.children.cycles-pp.perf_event_task_tick
      0.00            +0.3        0.31 ± 27%  perf-profile.children.cycles-pp.__intel_pmu_enable_all
      0.00            +0.3        0.31 ± 42%  perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
      0.00            +0.3        0.32 ± 33%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.00            +0.3        0.32 ± 38%  perf-profile.children.cycles-pp.arch_cpu_idle_enter
      0.00            +0.3        0.34 ± 36%  perf-profile.children.cycles-pp.tick_sched_do_timer
      0.00            +0.4        0.35 ±  9%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.00            +0.4        0.36 ± 42%  perf-profile.children.cycles-pp.ct_kernel_enter
      0.00            +0.4        0.37 ± 43%  perf-profile.children.cycles-pp.native_apic_mem_write
      0.00            +0.4        0.39 ± 23%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.00            +0.4        0.39 ± 17%  perf-profile.children.cycles-pp.nr_iowait_cpu
      0.00            +0.4        0.40 ± 23%  perf-profile.children.cycles-pp.rb_erase
      0.00            +0.4        0.40 ± 31%  perf-profile.children.cycles-pp.local_clock
      0.00            +0.4        0.41 ± 30%  perf-profile.children.cycles-pp.tick_check_oneshot_broadcast_this_cpu
      0.08 ±223%      +0.4        0.50 ± 14%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.00            +0.5        0.46 ± 21%  perf-profile.children.cycles-pp.run_posix_cpu_timers
      0.08 ±223%      +0.5        0.55 ± 15%  perf-profile.children.cycles-pp.find_busiest_group
      0.00            +0.5        0.47 ± 20%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.00            +0.5        0.48 ± 26%  perf-profile.children.cycles-pp.__update_blocked_fair
      0.00            +0.5        0.49 ± 36%  perf-profile.children.cycles-pp.hrtimer_update_next_event
      0.00            +0.5        0.50 ± 25%  perf-profile.children.cycles-pp.irqentry_enter
      0.00            +0.5        0.51 ± 22%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.00            +0.5        0.52 ± 33%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.00            +0.5        0.52 ± 29%  perf-profile.children.cycles-pp.update_irq_load_avg
      0.00            +0.5        0.54 ± 26%  perf-profile.children.cycles-pp.cpuidle_governor_latency_req
      0.00            +0.6        0.55 ± 11%  perf-profile.children.cycles-pp.arch_scale_freq_tick
      0.00            +0.6        0.56 ± 29%  perf-profile.children.cycles-pp.ct_idle_exit
      0.00            +0.6        0.57 ± 18%  perf-profile.children.cycles-pp.tick_nohz_stop_idle
      0.00            +0.6        0.57 ± 22%  perf-profile.children.cycles-pp.read_tsc
      0.00            +0.6        0.58 ± 28%  perf-profile.children.cycles-pp.rcu_pending
      0.00            +0.6        0.59 ± 20%  perf-profile.children.cycles-pp.native_sched_clock
      0.00            +0.6        0.61 ±  6%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.00            +0.7        0.68 ± 25%  perf-profile.children.cycles-pp.timerqueue_add
      0.00            +0.7        0.74 ±  5%  perf-profile.children.cycles-pp.timerqueue_del
      0.00            +0.8        0.75 ± 13%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.00            +0.8        0.76 ± 25%  perf-profile.children.cycles-pp.update_rq_clock_task
      0.08 ±223%      +0.8        0.84 ± 15%  perf-profile.children.cycles-pp.load_balance
      0.00            +0.8        0.76 ± 25%  perf-profile.children.cycles-pp.clockevents_program_event
      0.00            +0.8        0.78 ± 17%  perf-profile.children.cycles-pp.enqueue_hrtimer
      0.00            +0.8        0.80 ± 22%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.00            +0.9        0.89 ± 17%  perf-profile.children.cycles-pp.update_blocked_averages
      0.00            +0.9        0.89 ± 18%  perf-profile.children.cycles-pp.perf_rotate_context
      0.00            +0.9        0.92 ± 24%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.00            +0.9        0.95 ± 19%  perf-profile.children.cycles-pp.run_rebalance_domains
      0.08 ±223%      +1.0        1.04 ± 14%  perf-profile.children.cycles-pp.rebalance_domains
      0.00            +1.1        1.05 ± 17%  perf-profile.children.cycles-pp.ktime_get
      0.00            +1.3        1.34 ± 18%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.00            +1.6        1.55 ± 22%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.00            +1.6        1.60 ± 18%  perf-profile.children.cycles-pp.tick_irq_enter
      0.00            +1.6        1.64 ± 16%  perf-profile.children.cycles-pp.irq_enter_rcu
      0.08 ±223%      +2.5        2.54 ± 10%  perf-profile.children.cycles-pp.__do_softirq
      0.00            +2.5        2.46 ±  7%  perf-profile.children.cycles-pp.asm_sysvec_irq_work
      0.00            +2.5        2.46 ±  7%  perf-profile.children.cycles-pp.sysvec_irq_work
      0.00            +2.5        2.46 ±  7%  perf-profile.children.cycles-pp.__sysvec_irq_work
      0.00            +2.5        2.46 ±  7%  perf-profile.children.cycles-pp.irq_work_run
      0.00            +2.5        2.46 ±  7%  perf-profile.children.cycles-pp.irq_work_single
      0.00            +2.5        2.46 ±  7%  perf-profile.children.cycles-pp._printk
      0.00            +2.5        2.52 ±  6%  perf-profile.children.cycles-pp.irq_work_run_list
      0.00            +2.8        2.76 ± 20%  perf-profile.children.cycles-pp.scheduler_tick
      0.08 ±223%      +3.0        3.13 ± 11%  perf-profile.children.cycles-pp.__irq_exit_rcu
      0.00            +3.8        3.76 ± 20%  perf-profile.children.cycles-pp.menu_select
      0.08 ±223%      +4.8        4.92 ±  8%  perf-profile.children.cycles-pp.update_process_times
      0.08 ±223%      +5.3        5.41 ±  7%  perf-profile.children.cycles-pp.tick_sched_handle
      0.08 ±223%      +5.8        5.90 ± 79%  perf-profile.children.cycles-pp.x86_64_start_kernel
      0.08 ±223%      +5.8        5.90 ± 79%  perf-profile.children.cycles-pp.x86_64_start_reservations
      0.08 ±223%      +5.8        5.90 ± 79%  perf-profile.children.cycles-pp.start_kernel
      0.08 ±223%      +5.8        5.90 ± 79%  perf-profile.children.cycles-pp.arch_call_rest_init
      0.08 ±223%      +5.8        5.90 ± 79%  perf-profile.children.cycles-pp.rest_init
      0.08 ±223%      +6.0        6.13 ±  4%  perf-profile.children.cycles-pp.tick_sched_timer
      0.08 ±223%      +9.8        9.85 ±  3%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.08 ±223%     +11.8       11.92 ±  3%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.08 ±223%     +12.4       12.48 ±  2%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      0.16 ±223%     +18.6       18.80        perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      0.16 ±223%     +20.1       20.24 ±  2%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
     21.44 ± 35%     +40.0       61.40 ±  2%  perf-profile.children.cycles-pp.intel_idle
     25.58 ± 45%     +59.1       84.64        perf-profile.children.cycles-pp.cpuidle_enter_state
     25.58 ± 45%     +59.2       84.82 ±  4%  perf-profile.children.cycles-pp.start_secondary
     25.58 ± 45%     +59.3       84.86        perf-profile.children.cycles-pp.cpuidle_enter
     25.58 ± 45%     +64.1       89.64        perf-profile.children.cycles-pp.cpuidle_idle_call
     25.66 ± 45%     +65.0       90.72        perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     25.66 ± 45%     +65.0       90.72        perf-profile.children.cycles-pp.cpu_startup_entry
     25.66 ± 45%     +65.0       90.72        perf-profile.children.cycles-pp.do_idle
      0.00            +0.1        0.13 ± 36%  perf-profile.self.cycles-pp.pm_qos_read_value
      0.00            +0.1        0.14 ± 48%  perf-profile.self.cycles-pp.cpuidle_not_available
      0.00            +0.1        0.14 ± 37%  perf-profile.self.cycles-pp.trigger_load_balance
      0.00            +0.1        0.14 ± 33%  perf-profile.self.cycles-pp.cpuidle_governor_latency_req
      0.00            +0.1        0.14 ± 45%  perf-profile.self.cycles-pp.rcu_nocb_flush_deferred_wakeup
      0.00            +0.1        0.15 ± 27%  perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
      0.00            +0.2        0.15 ± 42%  perf-profile.self.cycles-pp.call_cpuidle
      0.00            +0.2        0.15 ± 32%  perf-profile.self.cycles-pp.timerqueue_del
      0.00            +0.2        0.16 ± 33%  perf-profile.self.cycles-pp.irqentry_enter
      0.00            +0.2        0.16 ± 40%  perf-profile.self.cycles-pp.x86_pmu_disable
      0.00            +0.2        0.18 ± 48%  perf-profile.self.cycles-pp.task_tick_mm_cid
      0.00            +0.2        0.18 ± 48%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.00            +0.2        0.19 ± 22%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.00            +0.2        0.20 ± 50%  perf-profile.self.cycles-pp.rcu_sched_clock_irq
      0.00            +0.2        0.20 ± 52%  perf-profile.self.cycles-pp.tsc_verify_tsc_adjust
      0.00            +0.2        0.20 ± 42%  perf-profile.self.cycles-pp.do_idle
      0.00            +0.2        0.20 ± 46%  perf-profile.self.cycles-pp.tick_nohz_stop_idle
      0.00            +0.2        0.21 ± 47%  perf-profile.self.cycles-pp.ct_kernel_enter
      0.00            +0.2        0.21 ± 32%  perf-profile.self.cycles-pp.ct_nmi_enter
      0.00            +0.2        0.22 ± 27%  perf-profile.self.cycles-pp.rb_insert_color
      0.00            +0.2        0.22 ± 20%  perf-profile.self.cycles-pp.cpuidle_enter
      0.00            +0.2        0.22 ± 34%  perf-profile.self.cycles-pp.tick_irq_enter
      0.00            +0.2        0.22 ± 29%  perf-profile.self.cycles-pp.rb_next
      0.00            +0.2        0.22 ± 37%  perf-profile.self.cycles-pp.load_balance
      0.00            +0.2        0.23 ± 25%  perf-profile.self.cycles-pp.irq_work_tick
      0.00            +0.2        0.24 ± 30%  perf-profile.self.cycles-pp.__irq_exit_rcu
      0.00            +0.2        0.24 ±  9%  perf-profile.self.cycles-pp.rebalance_domains
      0.00            +0.2        0.24 ± 25%  perf-profile.self.cycles-pp.update_rq_clock_task
      0.00            +0.2        0.24 ± 28%  perf-profile.self.cycles-pp.get_next_timer_interrupt
      0.00            +0.3        0.26 ± 23%  perf-profile.self.cycles-pp.hrtimer_interrupt
      0.00            +0.3        0.27 ± 52%  perf-profile.self.cycles-pp.get_cpu_device
      0.00            +0.3        0.28 ± 27%  perf-profile.self.cycles-pp.check_cpu_stall
      0.00            +0.3        0.28 ± 40%  perf-profile.self.cycles-pp.perf_rotate_context
      0.00            +0.3        0.28 ± 64%  perf-profile.self.cycles-pp.rcu_pending
      0.00            +0.3        0.30 ± 50%  perf-profile.self.cycles-pp.irqtime_account_process_tick
      0.00            +0.3        0.31 ± 27%  perf-profile.self.cycles-pp.__intel_pmu_enable_all
      0.00            +0.3        0.31 ± 37%  perf-profile.self.cycles-pp.__update_blocked_fair
      0.00            +0.3        0.34 ± 26%  perf-profile.self.cycles-pp.sysvec_apic_timer_interrupt
      0.00            +0.3        0.34 ±  6%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.00            +0.3        0.35 ± 39%  perf-profile.self.cycles-pp.scheduler_tick
      0.00            +0.4        0.35 ± 37%  perf-profile.self.cycles-pp.update_process_times
      0.00            +0.4        0.36 ± 44%  perf-profile.self.cycles-pp.native_apic_mem_write
      0.00            +0.4        0.38 ± 20%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.00            +0.4        0.39 ± 27%  perf-profile.self.cycles-pp.cpuidle_idle_call
      0.00            +0.4        0.40 ± 23%  perf-profile.self.cycles-pp.rb_erase
      0.00            +0.4        0.41 ± 30%  perf-profile.self.cycles-pp.tick_check_oneshot_broadcast_this_cpu
      0.00            +0.4        0.44 ± 12%  perf-profile.self.cycles-pp.irqtime_account_irq
      0.00            +0.5        0.46 ± 31%  perf-profile.self.cycles-pp.timerqueue_add
      0.00            +0.5        0.46 ± 21%  perf-profile.self.cycles-pp.run_posix_cpu_timers
      0.00            +0.5        0.51 ± 22%  perf-profile.self.cycles-pp.lapic_next_deadline
      0.00            +0.5        0.52 ± 29%  perf-profile.self.cycles-pp.update_irq_load_avg
      0.00            +0.5        0.54 ± 44%  perf-profile.self.cycles-pp.__hrtimer_run_queues
      0.00            +0.6        0.55 ± 11%  perf-profile.self.cycles-pp.arch_scale_freq_tick
      0.00            +0.6        0.55 ± 21%  perf-profile.self.cycles-pp.read_tsc
      0.00            +0.6        0.56 ± 12%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.00            +0.6        0.58 ± 19%  perf-profile.self.cycles-pp.native_sched_clock
      0.00            +0.6        0.60 ± 25%  perf-profile.self.cycles-pp.ktime_get
      0.00            +1.6        1.57 ± 29%  perf-profile.self.cycles-pp.menu_select
      0.00            +2.0        2.02 ± 18%  perf-profile.self.cycles-pp.cpuidle_enter_state
     21.44 ± 35%     +40.0       61.40 ±  2%  perf-profile.self.cycles-pp.intel_idle



***************************************************************************************************
lkp-cfl-d2: 12 threads 1 sockets Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz (Coffee Lake) with 32G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/need_x/option_a/rootfs/tbox_group/test/testcase:
  gcc-11/performance/x86_64-rhel-8.3/true/500px Copy From Window To Window/debian-x86_64-phoronix/lkp-cfl-d2/x11perf-1.1.1/phoronix-test-suite

commit: 
  a525b95417 ("cpufreq/amd-pstate: Remove fast_switch_possible flag from active driver")
  278dfbd17c ("cpufreq: Return failure if fast_switch is not set and fast_switch_possible is set")

a525b954173e74cd 278dfbd17c5d0767834471d7fea 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :6          100%           6:6     kmsg.cpufreq:fast_switch_possible_is_enabled_but_fast_switch_callback_is_not_set
         %stddev     %change         %stddev
             \          |                \  
    711.17 ± 13%    +756.8%       6093 ± 45%  phoronix-test-suite.time.involuntary_context_switches
    125556            -4.0%     120523        phoronix-test-suite.time.maximum_resident_set_size
    193366           -23.6%     147783        phoronix-test-suite.time.minor_page_faults
     35.17          +136.0%      83.00        phoronix-test-suite.time.percent_of_cpu_this_job_got
     28.44          +251.3%      99.90        phoronix-test-suite.time.user_time
   1411873 ±  7%     -79.4%     290969 ± 10%  phoronix-test-suite.time.voluntary_context_switches
     11105            -7.3%      10294        phoronix-test-suite.x11perf.500pxCopyFromWindowToWindow.operations___second
 1.596e+09           -10.0%  1.436e+09        cpuidle..time
   3803740 ±  2%     -31.3%    2612320        cpuidle..usage
      4.28            -4.0        0.32        mpstat.cpu.all.iowait%
      2.26 ±  2%      +1.4        3.70        mpstat.cpu.all.irq%
      0.20 ±  3%      +0.1        0.26        mpstat.cpu.all.soft%
      1.61 ±  2%      +0.6        2.23        mpstat.cpu.all.sys%
      2.87            +9.7       12.59        mpstat.cpu.all.usr%
     88.00            -9.1%      80.00        vmstat.cpu.id
      2.50 ± 20%    +380.0%      12.00        vmstat.cpu.us
      2093            -2.3%       2045        vmstat.io.bi
  10868294           -76.8%    2519224        vmstat.memory.cache
  21085319           +40.1%   29541029        vmstat.memory.free
      0.00         +1e+102%       1.00        vmstat.procs.r
     32259 ±  3%     -52.1%      15452 ±  2%  vmstat.system.cs
     18239            -7.5%      16870        vmstat.system.in
    459850            -9.1%     417876        meminfo.AnonPages
  10854060           -77.5%    2446909        meminfo.Cached
  10787077           -78.3%    2346035        meminfo.Committed_AS
   1211148 ±  2%      -7.4%    1121115        meminfo.Inactive(anon)
     99909           -24.9%      75030        meminfo.KReclaimable
    150648           -23.1%     115779        meminfo.Mapped
  22114323           +38.6%   30641210        meminfo.MemAvailable
  20999037           +40.7%   29538155        meminfo.MemFree
  11664521           -73.2%    3125406        meminfo.Memused
     22823           -71.3%       6547        meminfo.PageTables
     99909           -24.9%      75030        meminfo.SReclaimable
     96043           -35.6%      61884        meminfo.SUnreclaim
   9422747           -89.2%    1015386        meminfo.Shmem
    195953           -30.1%     136915        meminfo.Slab
   8667525           -96.4%     308501        meminfo.Unevictable
  12972755           -75.7%    3154603        meminfo.max_used_kB
      0.42 ± 21%     +77.8%       0.75 ± 21%  sched_debug.cfs_rq:/.h_nr_running.avg
     13959 ±  5%    +127.0%      31685 ± 12%  sched_debug.cfs_rq:/.min_vruntime.avg
     30077 ±  8%    +127.7%      68497 ± 16%  sched_debug.cfs_rq:/.min_vruntime.max
      7885 ±  9%    +118.6%      17233 ± 12%  sched_debug.cfs_rq:/.min_vruntime.min
      6353 ± 13%    +143.0%      15442 ± 18%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.41 ± 21%     +45.6%       0.60 ± 11%  sched_debug.cfs_rq:/.nr_running.avg
    370.63 ± 14%     +45.2%     537.98 ± 11%  sched_debug.cfs_rq:/.runnable_avg.avg
    961.25 ±  8%     +45.3%       1397 ±  6%  sched_debug.cfs_rq:/.runnable_avg.max
    259.86 ±  9%     +43.6%     373.22 ± 12%  sched_debug.cfs_rq:/.runnable_avg.stddev
      6354 ± 13%    +143.1%      15448 ± 18%  sched_debug.cfs_rq:/.spread0.stddev
    352.22 ± 13%     +32.7%     467.44 ± 11%  sched_debug.cfs_rq:/.util_avg.avg
    935.44 ±  7%     +32.4%       1238 ±  6%  sched_debug.cfs_rq:/.util_avg.max
    256.68 ±  9%     +34.4%     345.04 ± 10%  sched_debug.cfs_rq:/.util_avg.stddev
     76.34 ± 22%     +78.9%     136.54 ± 21%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    753033 ±  6%     +18.8%     894877 ± 10%  sched_debug.cpu.avg_idle.avg
      0.54 ± 42%    +243.8%       1.85 ± 13%  sched_debug.cpu.clock.stddev
      0.00 ± 12%     +78.0%       0.00 ± 13%  sched_debug.cpu.next_balance.stddev
      0.43 ± 23%     +74.1%       0.75 ± 18%  sched_debug.cpu.nr_running.avg
    199704 ± 17%     -53.6%      92716 ± 19%  sched_debug.cpu.nr_switches.avg
    449460 ± 20%     -53.4%     209625 ± 18%  sched_debug.cpu.nr_switches.max
     58141 ± 39%     -71.6%      16484 ± 29%  sched_debug.cpu.nr_switches.min
    117521 ± 17%     -51.4%      57142 ± 25%  sched_debug.cpu.nr_switches.stddev
    114963            -9.1%     104454        proc-vmstat.nr_anon_pages
    553299           +38.5%     766481        proc-vmstat.nr_dirty_background_threshold
   1107953           +38.5%    1534837        proc-vmstat.nr_dirty_threshold
   2714060           -77.4%     612214        proc-vmstat.nr_file_pages
   5249754           +40.7%    7384644        proc-vmstat.nr_free_pages
    302779 ±  2%      -7.4%     280420        proc-vmstat.nr_inactive_anon
      6151            +1.4%       6236        proc-vmstat.nr_kernel_stack
     37671           -23.2%      28939        proc-vmstat.nr_mapped
      5705           -71.3%       1638        proc-vmstat.nr_page_table_pages
   2355704           -89.2%     253805        proc-vmstat.nr_shmem
     24977           -24.9%      18757        proc-vmstat.nr_slab_reclaimable
     24010           -35.6%      15470        proc-vmstat.nr_slab_unreclaimable
   2166906           -96.4%      76929        proc-vmstat.nr_unevictable
    302779 ±  2%      -7.4%     280420        proc-vmstat.nr_zone_inactive_anon
   2166906           -96.4%      76929        proc-vmstat.nr_zone_unevictable
  10747274 ±  3%     -92.9%     765514        proc-vmstat.numa_hit
  10756248 ±  3%     -92.9%     765422        proc-vmstat.numa_local
  10997298 ±  3%     -91.8%     902283        proc-vmstat.pgalloc_normal
    501839            +6.5%     534236 ±  2%  proc-vmstat.pgfault
  10872643 ±  3%     -92.9%     775875        proc-vmstat.pgfree
     55315           +28.2%      70928 ±  2%  proc-vmstat.pgreuse
  10027217 ±  3%     -97.0%     305188        proc-vmstat.unevictable_pgs_culled
  10027217 ±  3%     -97.0%     305188        proc-vmstat.unevictable_pgs_rescued
  10027229 ±  3%     -97.0%     305217        proc-vmstat.unevictable_pgs_scanned
     51.17           +88.2%      96.32        turbostat.Any%C0
    340.83           -51.4%     165.67        turbostat.Avg_MHz
      8.10           +12.6       20.71        turbostat.Busy%
      4202           -81.0%     800.00        turbostat.Bzy_MHz
    849019           -12.5%     743127        turbostat.C1E
      3.53            -1.0        2.48        turbostat.C1E%
     72327 ±  2%     -66.7%      24104 ±  2%  turbostat.C3
      2.99 ±  3%      -2.7        0.31 ±  2%  turbostat.C3%
    723419           +14.3%     827187        turbostat.C6
    848543           -20.8%     671679        turbostat.C8
     47.43           -12.0       35.46        turbostat.C8%
     19.64           +24.4%      24.45        turbostat.CPU%c1
      5.18 ±  3%     -92.3%       0.40 ±  4%  turbostat.CPU%c3
     27.31           -50.3%      13.58        turbostat.CPU%c7
     32.05           +32.8       64.90        turbostat.CPUGFX%
     18.82           -77.2%       4.28        turbostat.CorWatt
     69.50           -44.1%      38.83        turbostat.CoreTmp
     73.77           -12.0%      64.94        turbostat.GFX%C0
     26.07 ±  2%     +32.0%      34.42        turbostat.GFX%rc6
     11.01           -13.8%       9.48        turbostat.GFXWatt
   1003094 ± 11%     -99.4%       5641 ± 53%  turbostat.POLL
      0.10 ± 10%      -0.1        0.01 ± 35%  turbostat.POLL%
     68.83           -43.1%      39.17        turbostat.PkgTmp
     30.81           -52.5%      14.65        turbostat.PkgWatt
      2.72            -4.7%       2.59        turbostat.RAMWatt
     84.82          +168.2%     227.52        turbostat.Totl%C0
    100.09           -21.4%      78.69        perf-stat.i.MPKI
 4.224e+08           -21.1%  3.333e+08        perf-stat.i.branch-instructions
      2.16 ±  3%      +1.8        3.93        perf-stat.i.branch-miss-rate%
  12591395           +10.0%   13851976        perf-stat.i.branch-misses
  21154943           -12.8%   18457066 ±  2%  perf-stat.i.cache-misses
 1.278e+08           -21.3%  1.006e+08        perf-stat.i.cache-references
     32972 ±  3%     -52.4%      15699 ±  2%  perf-stat.i.context-switches
      2.39           -46.7%       1.27        perf-stat.i.cpi
 3.498e+09           -47.9%  1.823e+09        perf-stat.i.cpu-cycles
     26.99 ±  2%      +7.8%      29.10 ±  2%  perf-stat.i.cpu-migrations
    647.57 ± 10%     -67.8%     208.52 ±  6%  perf-stat.i.cycles-between-cache-misses
      0.21 ±  4%      -0.0        0.17 ±  3%  perf-stat.i.dTLB-load-miss-rate%
    616008 ±  3%     -22.4%     477839 ±  4%  perf-stat.i.dTLB-load-misses
 4.915e+08           -26.4%  3.616e+08        perf-stat.i.dTLB-loads
 2.558e+08           -33.0%  1.714e+08        perf-stat.i.dTLB-stores
    458573 ± 27%     -40.5%     272629 ± 14%  perf-stat.i.iTLB-load-misses
   4257436 ±  2%     -16.5%    3554676 ±  2%  perf-stat.i.iTLB-loads
 2.038e+09           -22.3%  1.584e+09        perf-stat.i.instructions
      0.50           +67.1%       0.83        perf-stat.i.ipc
      5.03 ±  7%     -58.7%       2.08 ± 10%  perf-stat.i.major-faults
      0.29           -47.9%       0.15        perf-stat.i.metric.GHz
    659.63 ±  2%     -21.7%     516.58        perf-stat.i.metric.K/sec
    108.42           -25.7%      80.56        perf-stat.i.metric.M/sec
      3039            +5.6%       3211 ±  2%  perf-stat.i.minor-faults
      0.00 ± 17%      +0.0        0.00 ± 12%  perf-stat.i.node-load-miss-rate%
   1450700            -6.0%    1363538        perf-stat.i.node-loads
      0.00 ± 28%      +0.0        0.00 ± 19%  perf-stat.i.node-store-miss-rate%
   6054128 ±  3%     -79.8%    1225203 ±  2%  perf-stat.i.node-stores
      3044            +5.5%       3213 ±  2%  perf-stat.i.page-faults
      2.98            +1.2        4.15        perf-stat.overall.branch-miss-rate%
     16.55            +1.8       18.35 ±  2%  perf-stat.overall.cache-miss-rate%
      1.72           -32.9%       1.15        perf-stat.overall.cpi
    165.40           -40.3%      98.75        perf-stat.overall.cycles-between-cache-misses
      0.58           +49.1%       0.87        perf-stat.overall.ipc
      0.00 ± 40%      +0.0        0.00 ±  9%  perf-stat.overall.node-store-miss-rate%
 4.195e+08           -21.1%   3.31e+08        perf-stat.ps.branch-instructions
  12510468            +9.9%   13751528        perf-stat.ps.branch-misses
  21007097           -12.7%   18338973 ±  2%  perf-stat.ps.cache-misses
 1.269e+08           -21.2%   99965770        perf-stat.ps.cache-references
     32738 ±  3%     -52.4%      15579 ±  2%  perf-stat.ps.context-switches
 3.474e+09           -47.9%   1.81e+09        perf-stat.ps.cpu-cycles
     26.81 ±  2%      +7.8%      28.90 ±  2%  perf-stat.ps.cpu-migrations
    611716 ±  3%     -22.4%     474612 ±  4%  perf-stat.ps.dTLB-load-misses
 4.882e+08           -26.4%  3.591e+08        perf-stat.ps.dTLB-loads
  2.54e+08           -33.0%  1.702e+08        perf-stat.ps.dTLB-stores
    455336 ± 27%     -40.5%     270782 ± 14%  perf-stat.ps.iTLB-load-misses
   4227298 ±  2%     -16.5%    3531717 ±  2%  perf-stat.ps.iTLB-loads
 2.024e+09           -22.3%  1.573e+09        perf-stat.ps.instructions
      5.00 ±  7%     -58.8%       2.06 ± 10%  perf-stat.ps.major-faults
      3019            +5.6%       3187 ±  2%  perf-stat.ps.minor-faults
   1440523            -5.9%    1354982        perf-stat.ps.node-loads
   6011497 ±  3%     -79.8%    1217164 ±  2%  perf-stat.ps.node-stores
      3024            +5.5%       3189 ±  2%  perf-stat.ps.page-faults
 2.908e+11           -19.6%  2.338e+11        perf-stat.total.instructions
     42.60 ±  8%     -23.2       19.38 ±  4%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     40.68 ±  8%     -22.8       17.90 ±  5%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     20.00 ± 15%     -16.6        3.44 ± 12%  perf-profile.calltrace.cycles-pp.drm_ioctl_kernel.drm_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
     20.45 ± 15%     -16.4        4.07 ±  9%  perf-profile.calltrace.cycles-pp.drm_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
     20.70 ± 14%     -16.3        4.44 ±  8%  perf-profile.calltrace.cycles-pp.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.64 ± 17%      -5.3        0.30 ±101%  perf-profile.calltrace.cycles-pp.evict.__dentry_kill.dentry_kill.dput.__fput
      5.64 ± 17%      -5.3        0.30 ±101%  perf-profile.calltrace.cycles-pp.shmem_evict_inode.evict.__dentry_kill.dentry_kill.dput
      5.62 ± 17%      -5.3        0.30 ±101%  perf-profile.calltrace.cycles-pp.shmem_undo_range.shmem_evict_inode.evict.__dentry_kill.dentry_kill
      7.24 ± 39%      -4.2        3.05 ± 15%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.08 ± 18%      -1.5        2.60 ±  9%  perf-profile.calltrace.cycles-pp.do_writev.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.89 ± 18%      -1.4        2.47 ± 10%  perf-profile.calltrace.cycles-pp.vfs_writev.do_writev.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.71 ± 18%      -1.3        2.37 ± 10%  perf-profile.calltrace.cycles-pp.do_iter_write.vfs_writev.do_writev.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.55 ± 18%      -1.3        2.30 ± 10%  perf-profile.calltrace.cycles-pp.do_iter_readv_writev.do_iter_write.vfs_writev.do_writev.do_syscall_64
      3.49 ± 17%      -1.2        2.27 ± 11%  perf-profile.calltrace.cycles-pp.sock_write_iter.do_iter_readv_writev.do_iter_write.vfs_writev.do_writev
      3.29 ± 18%      -1.1        2.14 ± 10%  perf-profile.calltrace.cycles-pp.unix_stream_sendmsg.sock_write_iter.do_iter_readv_writev.do_iter_write.vfs_writev
      2.48 ± 23%      -1.0        1.48 ± 13%  perf-profile.calltrace.cycles-pp.__sys_recvmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.70 ± 18%      -1.0        2.72 ±  9%  perf-profile.calltrace.cycles-pp.syscall_return_via_sysret
      2.26 ± 23%      -0.9        1.37 ± 12%  perf-profile.calltrace.cycles-pp.___sys_recvmsg.__sys_recvmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.67 ± 39%      -0.8        0.90 ± 26%  perf-profile.calltrace.cycles-pp.__x64_sys_epoll_wait.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.00 ± 23%      -0.8        1.24 ± 11%  perf-profile.calltrace.cycles-pp.____sys_recvmsg.___sys_recvmsg.__sys_recvmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.06 ± 41%      -0.7        0.33 ±101%  perf-profile.calltrace.cycles-pp.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait.do_syscall_64
      1.57 ± 39%      -0.7        0.86 ± 25%  perf-profile.calltrace.cycles-pp.do_epoll_wait.__x64_sys_epoll_wait.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.89 ± 24%      -0.7        1.19 ± 14%  perf-profile.calltrace.cycles-pp.sock_recvmsg.____sys_recvmsg.___sys_recvmsg.__sys_recvmsg.do_syscall_64
      1.48 ± 40%      -0.7        0.80 ± 27%  perf-profile.calltrace.cycles-pp.ep_poll.do_epoll_wait.__x64_sys_epoll_wait.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.75 ± 23%      -0.7        1.09 ± 17%  perf-profile.calltrace.cycles-pp.unix_stream_recvmsg.sock_recvmsg.____sys_recvmsg.___sys_recvmsg.__sys_recvmsg
      1.70 ± 23%      -0.7        1.05 ± 16%  perf-profile.calltrace.cycles-pp.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg.____sys_recvmsg.___sys_recvmsg
      1.09 ± 18%      -0.6        0.44 ± 72%  perf-profile.calltrace.cycles-pp.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg.sock_write_iter
      1.12 ± 19%      -0.6        0.53 ± 47%  perf-profile.calltrace.cycles-pp.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg.sock_write_iter.do_iter_readv_writev
      1.26 ± 18%      -0.5        0.72 ± 16%  perf-profile.calltrace.cycles-pp.sock_alloc_send_pskb.unix_stream_sendmsg.sock_write_iter.do_iter_readv_writev.do_iter_write
      0.75 ± 16%      -0.5        0.28 ±100%  perf-profile.calltrace.cycles-pp.consume_skb.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg.____sys_recvmsg
      1.14 ± 16%      -0.4        0.69 ± 49%  perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.96 ± 17%      -0.4        0.53 ± 48%  perf-profile.calltrace.cycles-pp.unix_stream_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
      1.01 ± 16%      -0.4        0.59 ± 48%  perf-profile.calltrace.cycles-pp.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.92 ± 16%      -0.4        0.52 ± 47%  perf-profile.calltrace.cycles-pp.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
      1.34 ± 16%      -0.4        0.95 ± 20%  perf-profile.calltrace.cycles-pp.__entry_text_start
      1.05 ± 14%      +0.4        1.44 ± 15%  perf-profile.calltrace.cycles-pp.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.76 ± 18%      +0.5        1.22 ± 12%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      1.14 ± 15%      +0.5        1.64 ± 12%  perf-profile.calltrace.cycles-pp.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.10 ±223%      +0.5        0.65 ± 11%  perf-profile.calltrace.cycles-pp.asm_exc_page_fault
      0.09 ±223%      +0.6        0.64 ± 18%  perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.09 ±223%      +0.6        0.66 ± 19%  perf-profile.calltrace.cycles-pp.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.08 ±223%      +0.6        0.73 ± 16%  perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      1.40 ± 20%      +0.8        2.22 ±  9%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      0.09 ±223%      +0.8        0.94 ± 23%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry
      1.68 ± 20%      +0.9        2.58 ±  8%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      1.42 ± 10%      +1.0        2.40 ± 14%  perf-profile.calltrace.cycles-pp.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
      1.14 ± 20%      +1.0        2.16 ± 10%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      1.89 ± 22%      +1.0        2.92 ± 10%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      3.14 ± 26%      +1.7        4.86 ±  9%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      3.81 ± 26%      +2.1        5.95 ±  7%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      3.93 ± 26%      +2.3        6.25 ±  7%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      6.30 ± 18%      +3.2        9.46 ±  9%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
      6.93 ± 20%      +3.5       10.46 ± 11%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     37.49 ±  5%     +21.5       59.02 ±  8%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
     42.24 ±  4%     +22.4       64.61 ±  8%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     42.29 ±  4%     +22.4       64.67 ±  8%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     42.28 ±  4%     +22.4       64.66 ±  8%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     39.44 ±  4%     +22.7       62.12 ±  8%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     27.18 ±  6%     +23.8       50.99 ±  2%  perf-profile.calltrace.cycles-pp.intel_idle_ibrs.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     40.48 ±  6%     +25.5       65.93 ±  2%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
     45.98 ±  4%     +26.8       72.75 ±  2%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     42.67 ±  8%     -23.3       19.40 ±  4%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     40.70 ±  8%     -22.8       17.93 ±  5%  perf-profile.children.cycles-pp.do_syscall_64
     20.00 ± 15%     -16.6        3.45 ± 12%  perf-profile.children.cycles-pp.drm_ioctl_kernel
     20.45 ± 15%     -16.4        4.08 ± 10%  perf-profile.children.cycles-pp.drm_ioctl
     20.71 ± 14%     -16.3        4.44 ±  8%  perf-profile.children.cycles-pp.__x64_sys_ioctl
     15.06 ± 17%     -13.4        1.71 ± 15%  perf-profile.children.cycles-pp.shmem_read_mapping_page_gfp
     14.79 ± 18%     -13.1        1.68 ± 17%  perf-profile.children.cycles-pp.shmem_get_folio_gfp
      5.78 ± 16%      -5.0        0.78 ± 14%  perf-profile.children.cycles-pp.dentry_kill
      5.80 ± 17%      -5.0        0.80 ± 13%  perf-profile.children.cycles-pp.__fput
      5.78 ± 16%      -5.0        0.78 ± 14%  perf-profile.children.cycles-pp.__dentry_kill
      5.77 ± 16%      -5.0        0.77 ± 14%  perf-profile.children.cycles-pp.evict
      5.77 ± 16%      -5.0        0.77 ± 14%  perf-profile.children.cycles-pp.shmem_evict_inode
      5.76 ± 16%      -5.0        0.77 ± 14%  perf-profile.children.cycles-pp.shmem_undo_range
      5.79 ± 16%      -5.0        0.80 ± 12%  perf-profile.children.cycles-pp.dput
      7.28 ± 39%      -4.2        3.08 ± 14%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      3.96 ± 23%      -3.9        0.09 ± 43%  perf-profile.children.cycles-pp.poll_idle
      3.36 ± 18%      -2.9        0.45 ± 15%  perf-profile.children.cycles-pp.shmem_add_to_page_cache
      3.07 ± 17%      -2.7        0.32 ± 50%  perf-profile.children.cycles-pp.clear_page_erms
      3.00 ± 23%      -2.5        0.47 ± 30%  perf-profile.children.cycles-pp.shmem_alloc_and_acct_folio
      2.90 ± 16%      -2.5        0.41 ±  9%  perf-profile.children.cycles-pp.release_pages
      2.76 ± 16%      -2.4        0.34 ± 12%  perf-profile.children.cycles-pp.__pagevec_release
      2.76 ± 48%      -2.3        0.42 ± 35%  perf-profile.children.cycles-pp.check_move_unevictable_pages
      2.67 ± 48%      -2.3        0.41 ± 39%  perf-profile.children.cycles-pp.check_move_unevictable_folios
      2.52 ± 17%      -2.2        0.36 ± 15%  perf-profile.children.cycles-pp.truncate_inode_folio
      2.55 ± 24%      -2.1        0.42 ± 35%  perf-profile.children.cycles-pp.shmem_alloc_folio
      2.32 ± 17%      -2.0        0.33 ± 16%  perf-profile.children.cycles-pp.filemap_remove_folio
      2.17 ± 22%      -1.8        0.38 ± 33%  perf-profile.children.cycles-pp.vma_alloc_folio
      1.95 ± 17%      -1.7        0.30 ± 11%  perf-profile.children.cycles-pp.__filemap_remove_folio
      4.09 ± 18%      -1.5        2.60 ±  9%  perf-profile.children.cycles-pp.do_writev
      3.89 ± 18%      -1.4        2.47 ± 10%  perf-profile.children.cycles-pp.vfs_writev
      1.72 ± 19%      -1.4        0.32 ± 33%  perf-profile.children.cycles-pp.__folio_alloc
      1.75 ± 20%      -1.4        0.36 ± 31%  perf-profile.children.cycles-pp.__alloc_pages
      3.71 ± 18%      -1.3        2.37 ± 10%  perf-profile.children.cycles-pp.do_iter_write
      3.56 ± 18%      -1.2        2.32 ±  9%  perf-profile.children.cycles-pp.do_iter_readv_writev
      3.50 ± 17%      -1.2        2.27 ± 11%  perf-profile.children.cycles-pp.sock_write_iter
      1.32 ± 15%      -1.2        0.15 ± 19%  perf-profile.children.cycles-pp.free_unref_page_list
      3.29 ± 18%      -1.1        2.14 ± 10%  perf-profile.children.cycles-pp.unix_stream_sendmsg
      1.40 ± 23%      -1.1        0.29 ± 36%  perf-profile.children.cycles-pp.get_page_from_freelist
      1.26 ±  8%      -1.1        0.16 ± 30%  perf-profile.children.cycles-pp.xas_store
      2.90 ± 20%      -1.0        1.85 ± 14%  perf-profile.children.cycles-pp.sock_recvmsg
      1.24 ± 14%      -1.0        0.22 ± 21%  perf-profile.children.cycles-pp.__mod_lruvec_page_state
      2.70 ± 20%      -1.0        1.68 ± 17%  perf-profile.children.cycles-pp.unix_stream_recvmsg
      2.64 ± 20%      -1.0        1.64 ± 17%  perf-profile.children.cycles-pp.unix_stream_read_generic
      2.48 ± 23%      -1.0        1.49 ± 13%  perf-profile.children.cycles-pp.__sys_recvmsg
      1.24 ± 15%      -1.0        0.26 ± 49%  perf-profile.children.cycles-pp.folio_add_lru
      3.74 ± 17%      -1.0        2.76 ±  8%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      1.18 ± 25%      -1.0        0.23 ± 41%  perf-profile.children.cycles-pp.rmqueue
      2.26 ± 23%      -0.9        1.37 ± 12%  perf-profile.children.cycles-pp.___sys_recvmsg
      1.08 ± 16%      -0.8        0.24 ± 50%  perf-profile.children.cycles-pp.folio_batch_move_lru
      1.17 ± 66%      -0.8        0.33 ± 25%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      1.67 ± 39%      -0.8        0.90 ± 26%  perf-profile.children.cycles-pp.__x64_sys_epoll_wait
      2.00 ± 23%      -0.8        1.24 ± 11%  perf-profile.children.cycles-pp.____sys_recvmsg
      0.85 ± 28%      -0.7        0.12 ± 50%  perf-profile.children.cycles-pp.__mem_cgroup_charge
      1.57 ± 39%      -0.7        0.86 ± 25%  perf-profile.children.cycles-pp.do_epoll_wait
      0.87 ± 29%      -0.7        0.17 ± 31%  perf-profile.children.cycles-pp.__rmqueue_pcplist
      1.48 ± 40%      -0.7        0.80 ± 27%  perf-profile.children.cycles-pp.ep_poll
      0.76 ±  9%      -0.6        0.12 ± 21%  perf-profile.children.cycles-pp.__mod_lruvec_state
      0.74 ± 13%      -0.6        0.10 ± 22%  perf-profile.children.cycles-pp.free_pcppages_bulk
      0.73 ± 15%      -0.6        0.10 ± 50%  perf-profile.children.cycles-pp.lru_add_fn
      0.73 ± 15%      -0.6        0.12 ± 60%  perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
      0.72 ± 29%      -0.6        0.10 ± 58%  perf-profile.children.cycles-pp.rmqueue_bulk
      1.26 ± 19%      -0.5        0.72 ± 16%  perf-profile.children.cycles-pp.sock_alloc_send_pskb
      1.12 ± 19%      -0.5        0.61 ± 18%  perf-profile.children.cycles-pp.alloc_skb_with_frags
      0.57 ± 11%      -0.5        0.08 ± 21%  perf-profile.children.cycles-pp.__mod_node_page_state
      1.09 ± 19%      -0.5        0.60 ± 19%  perf-profile.children.cycles-pp.__alloc_skb
      0.57 ± 37%      -0.5        0.08 ± 81%  perf-profile.children.cycles-pp.charge_memcg
      1.18 ± 14%      -0.5        0.69 ± 13%  perf-profile.children.cycles-pp.consume_skb
      0.60 ± 19%      -0.5        0.12 ± 21%  perf-profile.children.cycles-pp.filemap_unaccount_folio
      1.16 ± 20%      -0.5        0.70 ± 14%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.54 ± 15%      -0.5        0.08 ± 51%  perf-profile.children.cycles-pp.__free_one_page
      0.43 ± 26%      -0.4        0.04 ±101%  perf-profile.children.cycles-pp.find_lock_entries
      0.43 ± 17%      -0.4        0.05 ± 81%  perf-profile.children.cycles-pp.xas_load
      1.14 ± 16%      -0.4        0.76 ± 26%  perf-profile.children.cycles-pp.__sys_recvfrom
      0.54 ± 13%      -0.3        0.22 ± 30%  perf-profile.children.cycles-pp.vm_mmap_pgoff
      0.42 ± 18%      -0.3        0.10 ± 53%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.52 ± 15%      -0.3        0.20 ± 25%  perf-profile.children.cycles-pp.do_mmap
      0.60 ± 16%      -0.3        0.29 ± 22%  perf-profile.children.cycles-pp.__kmem_cache_alloc_node
      0.75 ± 19%      -0.3        0.45 ± 26%  perf-profile.children.cycles-pp.ep_poll_callback
      0.33 ± 21%      -0.3        0.04 ±104%  perf-profile.children.cycles-pp.xas_clear_mark
      0.62 ± 11%      -0.3        0.34 ± 19%  perf-profile.children.cycles-pp.unix_destruct_scm
      0.60 ± 12%      -0.3        0.34 ± 18%  perf-profile.children.cycles-pp.sock_wfree
      0.63 ± 11%      -0.3        0.37 ± 27%  perf-profile.children.cycles-pp.skb_release_head_state
      0.53 ± 20%      -0.3        0.27 ± 33%  perf-profile.children.cycles-pp.kmalloc_reserve
      0.30 ± 20%      -0.2        0.05 ± 72%  perf-profile.children.cycles-pp.xas_find_conflict
      0.30 ±  9%      -0.2        0.06 ± 80%  perf-profile.children.cycles-pp.rcu_do_batch
      0.28 ± 34%      -0.2        0.05 ±107%  perf-profile.children.cycles-pp.cgroup_rstat_updated
      0.52 ± 23%      -0.2        0.29 ± 17%  perf-profile.children.cycles-pp.skb_release_data
      0.41 ± 19%      -0.2        0.18 ± 34%  perf-profile.children.cycles-pp.mmap_region
      0.48 ± 20%      -0.2        0.25 ± 33%  perf-profile.children.cycles-pp.__kmalloc_node_track_caller
      0.78 ± 16%      -0.2        0.57 ± 18%  perf-profile.children.cycles-pp._raw_spin_lock
      0.24 ± 26%      -0.2        0.03 ±101%  perf-profile.children.cycles-pp.folio_unlock
      0.35 ±  9%      -0.2        0.14 ± 22%  perf-profile.children.cycles-pp.rcu_core
      0.48 ± 20%      -0.2        0.30 ± 21%  perf-profile.children.cycles-pp.kmem_cache_alloc_node
      0.42 ± 10%      -0.2        0.24 ± 21%  perf-profile.children.cycles-pp.unix_write_space
      0.39 ± 25%      -0.2        0.21 ± 34%  perf-profile.children.cycles-pp.ep_autoremove_wake_function
      0.49 ± 19%      -0.2        0.31 ± 28%  perf-profile.children.cycles-pp.memcg_slab_post_alloc_hook
      0.22 ± 21%      -0.2        0.05 ± 72%  perf-profile.children.cycles-pp.__list_add_valid
      0.21 ± 19%      -0.2        0.04 ±105%  perf-profile.children.cycles-pp.try_charge_memcg
      0.23 ± 24%      -0.2        0.06 ± 58%  perf-profile.children.cycles-pp.__folio_throttle_swaprate
      0.44 ± 23%      -0.2        0.27 ± 22%  perf-profile.children.cycles-pp.sock_poll
      0.22 ± 24%      -0.2        0.06 ±110%  perf-profile.children.cycles-pp.do_vmi_munmap
      0.32 ± 26%      -0.2        0.17 ± 33%  perf-profile.children.cycles-pp.__kmem_cache_free
      0.19 ± 25%      -0.2        0.04 ±115%  perf-profile.children.cycles-pp.blk_cgroup_congested
      0.28 ± 28%      -0.2        0.13 ± 17%  perf-profile.children.cycles-pp.unix_poll
      0.35 ± 29%      -0.1        0.21 ± 30%  perf-profile.children.cycles-pp.skb_copy_datagram_from_iter
      0.18 ± 20%      -0.1        0.04 ±110%  perf-profile.children.cycles-pp.memset_orig
      0.35 ± 27%      -0.1        0.22 ± 39%  perf-profile.children.cycles-pp.rep_movs_alternative
      0.30 ± 15%      -0.1        0.18 ± 27%  perf-profile.children.cycles-pp.__slab_free
      0.23 ± 24%      -0.1        0.11 ± 36%  perf-profile.children.cycles-pp.copy_msghdr_from_user
      0.16 ± 37%      -0.1        0.05 ± 77%  perf-profile.children.cycles-pp._copy_from_iter
      0.20 ± 28%      -0.1        0.10 ± 67%  perf-profile.children.cycles-pp.mod_objcg_state
      0.19 ± 25%      -0.1        0.08 ± 65%  perf-profile.children.cycles-pp.import_iovec
      0.18 ± 23%      -0.1        0.07 ± 80%  perf-profile.children.cycles-pp.__import_iovec
      0.30 ±  9%      -0.1        0.19 ± 33%  perf-profile.children.cycles-pp.kmem_cache_free
      0.15 ± 29%      -0.1        0.06 ± 64%  perf-profile.children.cycles-pp.mas_store_prealloc
      0.15 ± 20%      -0.1        0.08 ± 30%  perf-profile.children.cycles-pp.dma_resv_iter_walk_unlocked
      0.15 ± 19%      -0.1        0.09 ± 37%  perf-profile.children.cycles-pp.obj_cgroup_charge
      0.06 ± 50%      +0.1        0.12 ± 16%  perf-profile.children.cycles-pp.cpuidle_not_available
      0.02 ±141%      +0.1        0.08 ± 27%  perf-profile.children.cycles-pp.account_process_tick
      0.02 ±141%      +0.1        0.10 ± 15%  perf-profile.children.cycles-pp.link_path_walk
      0.06 ± 59%      +0.1        0.14 ± 20%  perf-profile.children.cycles-pp.rb_insert_color
      0.02 ±141%      +0.1        0.10 ± 46%  perf-profile.children.cycles-pp.tick_nohz_tick_stopped
      0.02 ±146%      +0.1        0.11 ± 30%  perf-profile.children.cycles-pp.cpuidle_reflect
      0.00            +0.1        0.10 ± 36%  perf-profile.children.cycles-pp.pipe_write
      0.01 ±223%      +0.1        0.11 ± 28%  perf-profile.children.cycles-pp.task_tick_idle
      0.05 ± 76%      +0.1        0.15 ± 24%  perf-profile.children.cycles-pp.dup_mmap
      0.20 ± 21%      +0.1        0.30 ± 24%  perf-profile.children.cycles-pp.ct_idle_exit
      0.00            +0.1        0.10 ± 36%  perf-profile.children.cycles-pp.show
      0.00            +0.1        0.10 ± 36%  perf-profile.children.cycles-pp.show_time_in_state
      0.09 ± 49%      +0.1        0.19 ± 18%  perf-profile.children.cycles-pp.copy_process
      0.06 ± 75%      +0.1        0.17 ± 21%  perf-profile.children.cycles-pp.dup_mm
      0.00            +0.1        0.11 ± 34%  perf-profile.children.cycles-pp.sysfs_kf_seq_show
      0.02 ±144%      +0.1        0.13 ± 28%  perf-profile.children.cycles-pp.do_futex
      0.01 ±223%      +0.1        0.12 ± 30%  perf-profile.children.cycles-pp.futex_wait
      0.05 ± 49%      +0.1        0.16 ± 44%  perf-profile.children.cycles-pp.ct_kernel_exit_state
      0.02 ±149%      +0.1        0.14 ± 35%  perf-profile.children.cycles-pp.ksys_mmap_pgoff
      0.05 ± 47%      +0.1        0.16 ± 20%  perf-profile.children.cycles-pp.error_entry
      0.02 ±141%      +0.1        0.13 ± 34%  perf-profile.children.cycles-pp.ct_kernel_exit
      0.19 ± 17%      +0.1        0.31 ± 24%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.02 ±146%      +0.1        0.14 ± 31%  perf-profile.children.cycles-pp.__x64_sys_futex
      0.09 ± 49%      +0.1        0.21 ± 22%  perf-profile.children.cycles-pp.__do_sys_clone
      0.09 ± 49%      +0.1        0.21 ± 22%  perf-profile.children.cycles-pp.kernel_clone
      0.01 ±223%      +0.1        0.13 ± 58%  perf-profile.children.cycles-pp.__irqentry_text_end
      0.04 ±101%      +0.1        0.16 ± 41%  perf-profile.children.cycles-pp.irqentry_exit_to_user_mode
      0.00            +0.1        0.12 ± 45%  perf-profile.children.cycles-pp.traverse
      0.03 ±102%      +0.1        0.15 ± 45%  perf-profile.children.cycles-pp.vfs_write
      0.11 ± 15%      +0.1        0.24 ± 17%  perf-profile.children.cycles-pp.__intel_pmu_enable_all
      0.18 ± 22%      +0.1        0.31 ± 26%  perf-profile.children.cycles-pp.find_busiest_group
      0.04 ± 73%      +0.1        0.17 ± 45%  perf-profile.children.cycles-pp.ksys_write
      0.00            +0.1        0.13 ± 41%  perf-profile.children.cycles-pp.vsnprintf
      0.00            +0.1        0.14 ± 27%  perf-profile.children.cycles-pp.pipe_read
      0.05 ± 45%      +0.1        0.19 ± 18%  perf-profile.children.cycles-pp.hrtimer_update_next_event
      0.42 ± 19%      +0.2        0.58 ± 13%  perf-profile.children.cycles-pp.perf_rotate_context
      0.18 ± 29%      +0.2        0.34 ± 18%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.04 ± 71%      +0.2        0.20 ± 27%  perf-profile.children.cycles-pp.do_filp_open
      0.04 ± 71%      +0.2        0.20 ± 27%  perf-profile.children.cycles-pp.path_openat
      0.04 ± 75%      +0.2        0.22 ± 50%  perf-profile.children.cycles-pp.do_read_fault
      0.10 ± 32%      +0.2        0.28 ± 45%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.05 ± 45%      +0.2        0.24 ± 22%  perf-profile.children.cycles-pp.native_apic_msr_eoi_write
      0.02 ±142%      +0.2        0.21 ± 56%  perf-profile.children.cycles-pp.filemap_map_pages
      0.14 ± 31%      +0.2        0.33 ± 11%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.04 ± 72%      +0.2        0.23 ± 18%  perf-profile.children.cycles-pp.__x64_sys_openat
      0.12 ± 25%      +0.2        0.31 ± 33%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.04 ± 71%      +0.2        0.23 ± 18%  perf-profile.children.cycles-pp.do_sys_openat2
      0.07 ± 58%      +0.2        0.26 ± 26%  perf-profile.children.cycles-pp.__mmput
      0.07 ± 58%      +0.2        0.26 ± 26%  perf-profile.children.cycles-pp.exit_mmap
      0.27 ± 27%      +0.2        0.46 ± 23%  perf-profile.children.cycles-pp.__handle_mm_fault
      0.14 ± 31%      +0.2        0.34 ± 29%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.06 ± 25%      +0.2        0.25 ± 15%  perf-profile.children.cycles-pp.wait_for_xmitr
      0.29 ± 23%      +0.2        0.48 ± 25%  perf-profile.children.cycles-pp.handle_mm_fault
      0.10 ± 17%      +0.2        0.31 ± 39%  perf-profile.children.cycles-pp.arch_scale_freq_tick
      0.09 ± 23%      +0.2        0.31 ± 31%  perf-profile.children.cycles-pp.load_elf_binary
      0.35 ± 24%      +0.2        0.57 ± 20%  perf-profile.children.cycles-pp.exc_page_fault
      0.34 ± 24%      +0.2        0.57 ± 19%  perf-profile.children.cycles-pp.do_user_addr_fault
      0.39 ± 11%      +0.2        0.62 ± 34%  perf-profile.children.cycles-pp.rebalance_domains
      0.09 ± 21%      +0.2        0.32 ± 33%  perf-profile.children.cycles-pp.search_binary_handler
      0.10 ± 13%      +0.2        0.33 ± 31%  perf-profile.children.cycles-pp.bprm_execve
      0.09 ± 21%      +0.2        0.33 ± 32%  perf-profile.children.cycles-pp.exec_binprm
      0.01 ±223%      +0.3        0.26 ± 49%  perf-profile.children.cycles-pp.seq_read_iter
      0.47 ± 14%      +0.3        0.72 ± 14%  perf-profile.children.cycles-pp.irq_enter_rcu
      0.46 ± 14%      +0.3        0.71 ± 14%  perf-profile.children.cycles-pp.tick_irq_enter
      0.24 ± 28%      +0.3        0.51 ± 15%  perf-profile.children.cycles-pp.read_tsc
      0.12 ± 15%      +0.3        0.41 ± 27%  perf-profile.children.cycles-pp.__x64_sys_execve
      0.12 ± 17%      +0.3        0.41 ± 27%  perf-profile.children.cycles-pp.do_execveat_common
      0.41 ± 25%      +0.3        0.70 ± 11%  perf-profile.children.cycles-pp.asm_exc_page_fault
      0.14 ± 23%      +0.3        0.44 ± 30%  perf-profile.children.cycles-pp.clockevents_program_event
      0.50 ± 21%      +0.3        0.83 ± 18%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.09 ± 33%      +0.4        0.48 ± 35%  perf-profile.children.cycles-pp.vfs_read
      0.09 ± 31%      +0.4        0.51 ± 32%  perf-profile.children.cycles-pp.ksys_read
      0.40 ± 13%      +0.4        0.82 ± 15%  perf-profile.children.cycles-pp.ktime_get
      1.54 ± 11%      +0.5        2.00 ± 10%  perf-profile.children.cycles-pp.__irq_exit_rcu
      0.98 ± 17%      +0.5        1.47 ± 11%  perf-profile.children.cycles-pp.scheduler_tick
      0.47 ± 19%      +0.6        1.03 ± 27%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      1.78 ± 19%      +0.9        2.69 ± 10%  perf-profile.children.cycles-pp.update_process_times
      2.09 ± 19%      +0.9        3.03 ±  9%  perf-profile.children.cycles-pp.tick_sched_handle
      1.22 ± 15%      +1.0        2.22 ± 10%  perf-profile.children.cycles-pp.intel_idle
      1.50 ±  9%      +1.1        2.56 ± 17%  perf-profile.children.cycles-pp.menu_select
      2.34 ± 20%      +1.1        3.43 ±  7%  perf-profile.children.cycles-pp.tick_sched_timer
      3.76 ± 21%      +1.8        5.56 ±  7%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      4.39 ± 21%      +2.2        6.63 ±  6%  perf-profile.children.cycles-pp.hrtimer_interrupt
      4.53 ± 21%      +2.5        7.00 ±  6%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      6.93 ± 17%      +3.5       10.44 ±  7%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      7.50 ± 17%      +3.8       11.31 ±  7%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
     42.29 ±  4%     +22.4       64.67 ±  8%  perf-profile.children.cycles-pp.start_secondary
     27.19 ±  6%     +23.8       51.00 ±  2%  perf-profile.children.cycles-pp.intel_idle_ibrs
     40.99 ±  6%     +25.6       66.55 ±  2%  perf-profile.children.cycles-pp.cpuidle_enter_state
     41.02 ±  6%     +25.6       66.63 ±  2%  perf-profile.children.cycles-pp.cpuidle_enter
     45.98 ±  4%     +26.8       72.75 ±  2%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     45.98 ±  4%     +26.8       72.75 ±  2%  perf-profile.children.cycles-pp.cpu_startup_entry
     45.95 ±  4%     +26.8       72.73 ±  2%  perf-profile.children.cycles-pp.do_idle
     43.10 ±  5%     +26.9       70.00 ±  2%  perf-profile.children.cycles-pp.cpuidle_idle_call
      3.86 ± 23%      -3.8        0.08 ± 59%  perf-profile.self.cycles-pp.poll_idle
      3.56 ± 19%      -3.4        0.15 ± 37%  perf-profile.self.cycles-pp.shmem_get_folio_gfp
      3.03 ± 17%      -2.7        0.28 ± 40%  perf-profile.self.cycles-pp.clear_page_erms
      3.81 ± 18%      -1.3        2.49 ± 15%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode
      3.73 ± 17%      -1.0        2.76 ±  8%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      1.08 ± 26%      -0.9        0.15 ± 38%  perf-profile.self.cycles-pp.check_move_unevictable_folios
      0.64 ±  9%      -0.6        0.07 ± 71%  perf-profile.self.cycles-pp.xas_store
      0.66 ± 14%      -0.6        0.09 ± 62%  perf-profile.self.cycles-pp.release_pages
      2.03 ± 18%      -0.5        1.50 ±  3%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.61 ± 11%      -0.5        0.09 ± 83%  perf-profile.self.cycles-pp.shmem_add_to_page_cache
      0.54 ± 13%      -0.5        0.08 ± 21%  perf-profile.self.cycles-pp.__mod_node_page_state
      0.54 ± 15%      -0.5        0.09 ± 44%  perf-profile.self.cycles-pp.lru_add_fn
      0.52 ± 16%      -0.4        0.08 ± 82%  perf-profile.self.cycles-pp.rmqueue_bulk
      0.52 ± 11%      -0.4        0.09 ± 66%  perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
      0.40 ± 15%      -0.3        0.06 ± 83%  perf-profile.self.cycles-pp.__mod_lruvec_page_state
      0.40 ± 17%      -0.3        0.06 ± 48%  perf-profile.self.cycles-pp.__free_one_page
      0.41 ± 18%      -0.3        0.10 ± 50%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.33 ± 29%      -0.3        0.04 ±105%  perf-profile.self.cycles-pp.shmem_alloc_folio
      0.30 ± 26%      -0.3        0.03 ±100%  perf-profile.self.cycles-pp.__filemap_remove_folio
      0.28 ± 21%      -0.2        0.03 ±101%  perf-profile.self.cycles-pp.vma_alloc_folio
      0.28 ± 16%      -0.2        0.04 ±112%  perf-profile.self.cycles-pp.__alloc_pages
      0.40 ±  3%      -0.2        0.16 ± 35%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.88 ± 10%      -0.2        0.64 ± 13%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.28 ± 35%      -0.2        0.05 ±107%  perf-profile.self.cycles-pp.cgroup_rstat_updated
      0.77 ± 16%      -0.2        0.56 ± 19%  perf-profile.self.cycles-pp._raw_spin_lock
      0.20 ± 23%      -0.2        0.05 ± 72%  perf-profile.self.cycles-pp.__list_add_valid
      0.39 ± 17%      -0.1        0.26 ± 29%  perf-profile.self.cycles-pp.memcg_slab_post_alloc_hook
      0.17 ± 21%      -0.1        0.04 ±110%  perf-profile.self.cycles-pp.memset_orig
      0.29 ± 17%      -0.1        0.16 ± 20%  perf-profile.self.cycles-pp.__slab_free
      0.15 ± 15%      -0.1        0.03 ±101%  perf-profile.self.cycles-pp.try_charge_memcg
      0.20 ±  9%      -0.1        0.10 ± 50%  perf-profile.self.cycles-pp.kmem_cache_free
      0.14 ± 23%      -0.1        0.04 ±107%  perf-profile.self.cycles-pp.__import_iovec
      0.18 ± 24%      -0.1        0.10 ± 30%  perf-profile.self.cycles-pp.sock_wfree
      0.12 ± 12%      -0.1        0.04 ±112%  perf-profile.self.cycles-pp.tick_nohz_stop_idle
      0.11 ± 27%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.filemap_unaccount_folio
      0.02 ±141%      +0.1        0.08 ± 27%  perf-profile.self.cycles-pp.account_process_tick
      0.06 ± 50%      +0.1        0.12 ± 16%  perf-profile.self.cycles-pp.cpuidle_not_available
      0.14 ± 29%      +0.1        0.22 ± 15%  perf-profile.self.cycles-pp.sysvec_apic_timer_interrupt
      0.01 ±223%      +0.1        0.09 ± 48%  perf-profile.self.cycles-pp.tick_nohz_tick_stopped
      0.02 ±144%      +0.1        0.10 ± 36%  perf-profile.self.cycles-pp.cpuidle_reflect
      0.06 ± 73%      +0.1        0.15 ± 29%  perf-profile.self.cycles-pp.update_sg_lb_stats
      0.05 ± 52%      +0.1        0.14 ± 20%  perf-profile.self.cycles-pp.rb_insert_color
      0.01 ±223%      +0.1        0.11 ± 28%  perf-profile.self.cycles-pp.task_tick_idle
      0.00            +0.1        0.10 ± 20%  perf-profile.self.cycles-pp.tick_sched_timer
      0.04 ± 47%      +0.1        0.16 ± 45%  perf-profile.self.cycles-pp.ct_kernel_exit_state
      0.05 ± 46%      +0.1        0.16 ± 20%  perf-profile.self.cycles-pp.error_entry
      0.01 ±223%      +0.1        0.13 ± 58%  perf-profile.self.cycles-pp.__irqentry_text_end
      0.11 ± 15%      +0.1        0.24 ± 17%  perf-profile.self.cycles-pp.__intel_pmu_enable_all
      0.05 ± 72%      +0.1        0.18 ± 47%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.10 ± 25%      +0.2        0.28 ± 38%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.05 ± 45%      +0.2        0.23 ± 21%  perf-profile.self.cycles-pp.native_apic_msr_eoi_write
      0.10 ± 32%      +0.2        0.28 ± 45%  perf-profile.self.cycles-pp.lapic_next_deadline
      0.21 ± 13%      +0.2        0.40 ± 33%  perf-profile.self.cycles-pp.ktime_get
      0.14 ± 29%      +0.2        0.33 ± 11%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.10 ± 17%      +0.2        0.31 ± 39%  perf-profile.self.cycles-pp.arch_scale_freq_tick
      0.24 ± 27%      +0.3        0.50 ± 16%  perf-profile.self.cycles-pp.read_tsc
      0.72 ± 19%      +0.3        1.03 ± 14%  perf-profile.self.cycles-pp.cpuidle_enter_state
      1.22 ± 15%      +1.0        2.22 ± 10%  perf-profile.self.cycles-pp.intel_idle
     27.18 ±  6%     +23.8       50.98 ±  2%  perf-profile.self.cycles-pp.intel_idle_ibrs





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
  
Rafael J. Wysocki May 24, 2023, 5:44 p.m. UTC | #2
On Wed, May 17, 2023 at 6:30 PM Wyes Karny <wyes.karny@amd.com> wrote:
>
> If fast_switch_possible flag is set by the scaling driver, the governor
> is free to select fast_switch function even if adjust_perf is set.  Some
> scaling drivers which use adjust_perf don't set fast_switch thinking
> that the governor would never fall back to fast_switch. But the governor
> can fall back to fast_switch even in runtime if frequency invariance is
> disabled due to some reason. This could crash the kernel if the driver
> didn't set the fast_switch function pointer.
>
> Therefore, return failure in cpufreq_online function if fast_switch is
> not set and fast_switch_possible is set.
>
> Signed-off-by: Wyes Karny <wyes.karny@amd.com>
> ---
>  drivers/cpufreq/cpufreq.c | 5 +++++
>  include/linux/cpufreq.h   | 4 +++-
>  2 files changed, 8 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/cpufreq/cpufreq.c b/drivers/cpufreq/cpufreq.c
> index 6b52ebe5a890..7835ba4fa34c 100644
> --- a/drivers/cpufreq/cpufreq.c
> +++ b/drivers/cpufreq/cpufreq.c
> @@ -1376,6 +1376,11 @@ static int cpufreq_online(unsigned int cpu)
>                         goto out_free_policy;
>                 }
>
> +               if (policy->fast_switch_possible && !cpufreq_driver->fast_switch) {
> +                       pr_err("fast_switch_possible is enabled but fast_switch callback is not set\n");
> +                       ret = -EINVAL;
> +                       goto out_destroy_policy;
> +               }

The driver registration can fail if the driver has ->adjust_perf
without ->fast_switch.  Then the check above would not be necessary
any more.

>                 /*
>                  * The initialization has succeeded and the policy is online.
>                  * If there is a problem with its frequency table, take it
> diff --git a/include/linux/cpufreq.h b/include/linux/cpufreq.h
> index 26e2eb399484..8cdf77bb3bc1 100644
> --- a/include/linux/cpufreq.h
> +++ b/include/linux/cpufreq.h
> @@ -340,7 +340,9 @@ struct cpufreq_driver {
>         /*
>          * ->fast_switch() replacement for drivers that use an internal
>          * representation of performance levels and can pass hints other than
> -        * the target performance level to the hardware.
> +        * the target performance level to the hardware. If driver is setting this,
> +        * then it needs to set fast_switch also. Because in certain scenario scale
> +        * invariance could be disabled and governor can switch back to fast_switch.

I would say something like "This can only be set if ->fast_switch is
set too, because in those cases (under specific conditions) scale
invariance can be disabled, which causes the schedutil governor to
fall back to the latter."

>          */
>         void            (*adjust_perf)(unsigned int cpu,
>                                        unsigned long min_perf,
> --
> 2.34.1
>
  
Wyes Karny May 29, 2023, 2:08 p.m. UTC | #3
Hi Rafael,

On 24 May 19:44, Rafael J. Wysocki wrote:
> On Wed, May 17, 2023 at 6:30 PM Wyes Karny <wyes.karny@amd.com> wrote:
> >
> > If fast_switch_possible flag is set by the scaling driver, the governor
> > is free to select fast_switch function even if adjust_perf is set.  Some
> > scaling drivers which use adjust_perf don't set fast_switch thinking
> > that the governor would never fall back to fast_switch. But the governor
> > can fall back to fast_switch even in runtime if frequency invariance is
> > disabled due to some reason. This could crash the kernel if the driver
> > didn't set the fast_switch function pointer.
> >
> > Therefore, return failure in cpufreq_online function if fast_switch is
> > not set and fast_switch_possible is set.
> >
> > Signed-off-by: Wyes Karny <wyes.karny@amd.com>
> > ---
> >  drivers/cpufreq/cpufreq.c | 5 +++++
> >  include/linux/cpufreq.h   | 4 +++-
> >  2 files changed, 8 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/cpufreq/cpufreq.c b/drivers/cpufreq/cpufreq.c
> > index 6b52ebe5a890..7835ba4fa34c 100644
> > --- a/drivers/cpufreq/cpufreq.c
> > +++ b/drivers/cpufreq/cpufreq.c
> > @@ -1376,6 +1376,11 @@ static int cpufreq_online(unsigned int cpu)
> >                         goto out_free_policy;
> >                 }
> >
> > +               if (policy->fast_switch_possible && !cpufreq_driver->fast_switch) {
> > +                       pr_err("fast_switch_possible is enabled but fast_switch callback is not set\n");
> > +                       ret = -EINVAL;
> > +                       goto out_destroy_policy;
> > +               }
> 
> The driver registration can fail if the driver has ->adjust_perf
> without ->fast_switch.  Then the check above would not be necessary
> any more.

Sure. Will do that.

> 
> >                 /*
> >                  * The initialization has succeeded and the policy is online.
> >                  * If there is a problem with its frequency table, take it
> > diff --git a/include/linux/cpufreq.h b/include/linux/cpufreq.h
> > index 26e2eb399484..8cdf77bb3bc1 100644
> > --- a/include/linux/cpufreq.h
> > +++ b/include/linux/cpufreq.h
> > @@ -340,7 +340,9 @@ struct cpufreq_driver {
> >         /*
> >          * ->fast_switch() replacement for drivers that use an internal
> >          * representation of performance levels and can pass hints other than
> > -        * the target performance level to the hardware.
> > +        * the target performance level to the hardware. If driver is setting this,
> > +        * then it needs to set fast_switch also. Because in certain scenario scale
> > +        * invariance could be disabled and governor can switch back to fast_switch.
> 
> I would say something like "This can only be set if ->fast_switch is
> set too, because in those cases (under specific conditions) scale
> invariance can be disabled, which causes the schedutil governor to
> fall back to the latter."

Sure. Will update and send the updated patch in-reply-to this.

Thanks & Regards,
Wyes

> 
> >          */
> >         void            (*adjust_perf)(unsigned int cpu,
> >                                        unsigned long min_perf,
> > --
> > 2.34.1
> >
  

Patch

diff --git a/drivers/cpufreq/cpufreq.c b/drivers/cpufreq/cpufreq.c
index 6b52ebe5a890..7835ba4fa34c 100644
--- a/drivers/cpufreq/cpufreq.c
+++ b/drivers/cpufreq/cpufreq.c
@@ -1376,6 +1376,11 @@  static int cpufreq_online(unsigned int cpu)
 			goto out_free_policy;
 		}
 
+		if (policy->fast_switch_possible && !cpufreq_driver->fast_switch) {
+			pr_err("fast_switch_possible is enabled but fast_switch callback is not set\n");
+			ret = -EINVAL;
+			goto out_destroy_policy;
+		}
 		/*
 		 * The initialization has succeeded and the policy is online.
 		 * If there is a problem with its frequency table, take it
diff --git a/include/linux/cpufreq.h b/include/linux/cpufreq.h
index 26e2eb399484..8cdf77bb3bc1 100644
--- a/include/linux/cpufreq.h
+++ b/include/linux/cpufreq.h
@@ -340,7 +340,9 @@  struct cpufreq_driver {
 	/*
 	 * ->fast_switch() replacement for drivers that use an internal
 	 * representation of performance levels and can pass hints other than
-	 * the target performance level to the hardware.
+	 * the target performance level to the hardware. If driver is setting this,
+	 * then it needs to set fast_switch also. Because in certain scenario scale
+	 * invariance could be disabled and governor can switch back to fast_switch.
 	 */
 	void		(*adjust_perf)(unsigned int cpu,
 				       unsigned long min_perf,