diff mbox series

sched: consider WF_SYNC to find idle siblings

Message ID 20221027202603.670616-1-avagin@google.com
State New
Headers
Series sched: consider WF_SYNC to find idle siblings |

Commit Message

Andrei Vagin Oct. 27, 2022, 8:26 p.m. UTC 
  From: Andrei Vagin <avagin@gmail.com>

WF_SYNC means that the waker goes to sleep after wakeup, so the current
cpu can be considered idle if the waker is the only process that is
running on it.

The perf pipe benchmark shows that this change reduces the average time
per operation from 8.8 usecs/op to 3.7 usecs/op.

Before:
 $ ./tools/perf/perf bench sched pipe
 # Running 'sched/pipe' benchmark:
 # Executed 1000000 pipe operations between two processes

     Total time: 8.813 [sec]

       8.813985 usecs/op
         113456 ops/sec

After:
 $ ./tools/perf/perf bench sched pipe
 # Running 'sched/pipe' benchmark:
 # Executed 1000000 pipe operations between two processes

     Total time: 3.743 [sec]

       3.743971 usecs/op
         267096 ops/sec

Cc: Ingo Molnar <mingo@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Vincent Guittot <vincent.guittot@linaro.org>
Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Ben Segall <bsegall@google.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Daniel Bristot de Oliveira <bristot@redhat.com>
Cc: Valentin Schneider <vschneid@redhat.com>
Signed-off-by: Andrei Vagin <avagin@gmail.com>
---
 kernel/sched/fair.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Comments

Peter Zijlstra Oct. 31, 2022, 12:57 p.m. UTC | #1 
On Thu, Oct 27, 2022 at 01:26:03PM -0700, Andrei Vagin wrote:
> From: Andrei Vagin <avagin@gmail.com>
> 
> WF_SYNC means that the waker goes to sleep after wakeup, so the current
> cpu can be considered idle if the waker is the only process that is
> running on it.
> 
> The perf pipe benchmark shows that this change reduces the average time
> per operation from 8.8 usecs/op to 3.7 usecs/op.
> 
> Before:
>  $ ./tools/perf/perf bench sched pipe
>  # Running 'sched/pipe' benchmark:
>  # Executed 1000000 pipe operations between two processes
> 
>      Total time: 8.813 [sec]
> 
>        8.813985 usecs/op
>          113456 ops/sec
> 
> After:
>  $ ./tools/perf/perf bench sched pipe
>  # Running 'sched/pipe' benchmark:
>  # Executed 1000000 pipe operations between two processes
> 
>      Total time: 3.743 [sec]
> 
>        3.743971 usecs/op
>          267096 ops/sec

But what; if anything, does it do for the myrad of other benchmarks we
run?
Andrei Vagin Oct. 31, 2022, 10:37 p.m. UTC | #2 
On Mon, Oct 31, 2022 at 5:57 AM Peter Zijlstra <peterz@infradead.org> wrote:
>
> On Thu, Oct 27, 2022 at 01:26:03PM -0700, Andrei Vagin wrote:
> > From: Andrei Vagin <avagin@gmail.com>
> >
> > WF_SYNC means that the waker goes to sleep after wakeup, so the current
> > cpu can be considered idle if the waker is the only process that is
> > running on it.
> >
> > The perf pipe benchmark shows that this change reduces the average time
> > per operation from 8.8 usecs/op to 3.7 usecs/op.
> >
> > Before:
> >  $ ./tools/perf/perf bench sched pipe
> >  # Running 'sched/pipe' benchmark:
> >  # Executed 1000000 pipe operations between two processes
> >
> >      Total time: 8.813 [sec]
> >
> >        8.813985 usecs/op
> >          113456 ops/sec
> >
> > After:
> >  $ ./tools/perf/perf bench sched pipe
> >  # Running 'sched/pipe' benchmark:
> >  # Executed 1000000 pipe operations between two processes
> >
> >      Total time: 3.743 [sec]
> >
> >        3.743971 usecs/op
> >          267096 ops/sec
>
> But what; if anything, does it do for the myrad of other benchmarks we
> run?

I've run these set of benchmarks:
* perf bench sched messaging
* perf bench epoll all
* perf bench futex all
* schbench
* tbench
*  kernel compilation

Results look the same with and without this change for all benchmarks except
tbench.  tbench shows improvements when a number of processes is less than a
number of cpu-s.

Here are results from my test host with 8 cpu-s.

$ tbench_srv &  "tbench" "-t" "15" "1" "127.0.0.1"
Before: Throughput 260.498 MB/sec  1 clients  1 procs  max_latency=1.301 ms
After:  Throughput 462.047 MB/sec  1 clients  1 procs  max_latency=1.066 ms

$ tbench_srv &  "tbench" "-t" "15" "4" "127.0.0.1"
Before: Throughput 733.44 MB/sec  4 clients  4 procs  max_latency=0.935 ms
After:  Throughput 1778.94 MB/sec  4 clients  4 procs  max_latency=0.882 ms

$ tbench_srv &  "tbench" "-t" "15" "8" "127.0.0.1"
Before: Throughput 1965.41 MB/sec  8 clients  8 procs  max_latency=2.145 ms
After:  Throughput 2002.96 MB/sec  8 clients  8 procs  max_latency=1.881 ms

$ tbench_srv &  "tbench" "-t" "15" "32" "127.0.0.1"
Before: Throughput 1881.79 MB/sec  32 clients  32 procs  max_latency=16.365 ms
After:  Throughput 1891.87 MB/sec  32 clients  32 procs  max_latency=4.050 ms

Let me know if you want to see results for any other specific benchmark.

Thanks,
Andrei
Mel Gorman Nov. 1, 2022, 9:41 a.m. UTC | #3 
On Thu, Oct 27, 2022 at 01:26:03PM -0700, Andrei Vagin wrote:
> From: Andrei Vagin <avagin@gmail.com>
> 
> WF_SYNC means that the waker goes to sleep after wakeup, so the current
> cpu can be considered idle if the waker is the only process that is
> running on it.
> 
> The perf pipe benchmark shows that this change reduces the average time
> per operation from 8.8 usecs/op to 3.7 usecs/op.
> 
> Before:
>  $ ./tools/perf/perf bench sched pipe
>  # Running 'sched/pipe' benchmark:
>  # Executed 1000000 pipe operations between two processes
> 
>      Total time: 8.813 [sec]
> 
>        8.813985 usecs/op
>          113456 ops/sec
> 
> After:
>  $ ./tools/perf/perf bench sched pipe
>  # Running 'sched/pipe' benchmark:
>  # Executed 1000000 pipe operations between two processes
> 
>      Total time: 3.743 [sec]
> 
>        3.743971 usecs/op
>          267096 ops/sec
> 

The WF_SYNC hint in unreliable as the waking process does not always
go to sleep immediately. While it's great for a benchmark like a pipe
benchmark as the relationship is strictly synchronous, it does not work
out as well for networking which can use WF_SYNC for wakeups but either
multiple tasks are being woken up or the waker does not go to sleep as
there is sufficient inbound traffic to keep it awake. There used to be
an attempt to track how accurate WF_SYNC was, using avg_overlap I think,
but it was ultimately removed.
Chen Yu Nov. 1, 2022, 12:58 p.m. UTC | #4 
Hi Mel,
On 2022-11-01 at 09:41:57 +0000, Mel Gorman wrote:
> On Thu, Oct 27, 2022 at 01:26:03PM -0700, Andrei Vagin wrote:
> > From: Andrei Vagin <avagin@gmail.com>
> > 
> > WF_SYNC means that the waker goes to sleep after wakeup, so the current
> > cpu can be considered idle if the waker is the only process that is
> > running on it.
> > 
> > The perf pipe benchmark shows that this change reduces the average time
> > per operation from 8.8 usecs/op to 3.7 usecs/op.
> > 
> > Before:
> >  $ ./tools/perf/perf bench sched pipe
> >  # Running 'sched/pipe' benchmark:
> >  # Executed 1000000 pipe operations between two processes
> > 
> >      Total time: 8.813 [sec]
> > 
> >        8.813985 usecs/op
> >          113456 ops/sec
> > 
> > After:
> >  $ ./tools/perf/perf bench sched pipe
> >  # Running 'sched/pipe' benchmark:
> >  # Executed 1000000 pipe operations between two processes
> > 
> >      Total time: 3.743 [sec]
> > 
> >        3.743971 usecs/op
> >          267096 ops/sec
> > 
> 
> The WF_SYNC hint in unreliable as the waking process does not always
> go to sleep immediately. While it's great for a benchmark like a pipe
> benchmark as the relationship is strictly synchronous, it does not work
> out as well for networking which can use WF_SYNC for wakeups but either
> multiple tasks are being woken up or the waker does not go to sleep as
> there is sufficient inbound traffic to keep it awake. There used to be
> an attempt to track how accurate WF_SYNC was, using avg_overlap I think,
> but it was ultimately removed.
avg_overlap was removed 10 years ago because of accuracy problem that
"we are missing the necessary call to update_curr()" according to
commit e12f31d3e5d3 ("sched: Remove avg_overlap"). But in current code
I think this issue described in above commit does not exist anymore because
in current code the put_prev_task() would invoke update_curr() for each
entity, then calculating the avg_overlap is always using the update-to-date
runtime? If it is true, is it applicable to bring avg_overlap back?

Some benchmarks suffer from cross-CPU wakeup which introduces rq lock
contention. Similar to this patch, I tracked the average duration of the
task and place the wakee to a CPU where only 1 short-running task is running,
which is another direction to mitigate cross-CPU wakeup[1]. Not sure if we
could deal with more accurately?

[1] https://lore.kernel.org/lkml/6b81eea9a8cafb7634f36586f1744b8d4ac49da5.1666531576.git.yu.c.chen@intel.com/

thanks,
Chenyu
Andrei Vagin Nov. 2, 2022, 12:18 a.m. UTC | #5 
On Tue, Nov 1, 2022 at 2:42 AM Mel Gorman <mgorman@suse.de> wrote:
>
> On Thu, Oct 27, 2022 at 01:26:03PM -0700, Andrei Vagin wrote:
> > From: Andrei Vagin <avagin@gmail.com>
> >
> > WF_SYNC means that the waker goes to sleep after wakeup, so the current
> > cpu can be considered idle if the waker is the only process that is
> > running on it.
> >
> > The perf pipe benchmark shows that this change reduces the average time
> > per operation from 8.8 usecs/op to 3.7 usecs/op.
> >
> > Before:
> >  $ ./tools/perf/perf bench sched pipe
> >  # Running 'sched/pipe' benchmark:
> >  # Executed 1000000 pipe operations between two processes
> >
> >      Total time: 8.813 [sec]
> >
> >        8.813985 usecs/op
> >          113456 ops/sec
> >
> > After:
> >  $ ./tools/perf/perf bench sched pipe
> >  # Running 'sched/pipe' benchmark:
> >  # Executed 1000000 pipe operations between two processes
> >
> >      Total time: 3.743 [sec]
> >
> >        3.743971 usecs/op
> >          267096 ops/sec
> >
>
> The WF_SYNC hint in unreliable as the waking process does not always
> go to sleep immediately. While it's great for a benchmark like a pipe
> benchmark as the relationship is strictly synchronous, it does not work
> out as well for networking which can use WF_SYNC for wakeups but either
> multiple tasks are being woken up or the waker does not go to sleep as
> there is sufficient inbound traffic to keep it awake.

This change should work fine when we wake up multiple tasks. If the waker
doesn't go to sleep, it sounds like a misuse of WF_SYNC. For example,
wake_affine_idle contains the same check like introduced in this
patch. At the first
glance, wake_affine_weight handles WF_SYNC incorrectly in this case too.

As for benchmarks, tbench shows much better numbers with this change:

$ tbench_srv &  "tbench" "-t" "15" "4" "127.0.0.1"
Before: Throughput 733.44 MB/sec  4 clients  4 procs  max_latency=0.935 ms
After:  Throughput 1778.94 MB/sec  4 clients  4 procs  max_latency=0.882 ms

I know it is just another synchronous benchmark...

I am working on the synchronous mode of seccom user notifies[1]. In the
first two versions, I used the WF_CURRENT_CPU [2] flag that has been borrowed
from the umcg patchset [3].  But when I was preparing the third version of the
patchset, I wondered why WF_SYNC didn't work in this case and ended up with this
patch. For the seccom patchset, fast synchronous context switches are the most
critical part, so any advice on how to do that properly are welcome.

[1] https://lore.kernel.org/lkml/20221020011048.156415-1-avagin@gmail.com/T/
[2] https://lore.kernel.org/lkml/20221020011048.156415-1-avagin@gmail.com/T/#m8a597d43764aa8ded2788ea7ce4276f9045668d1
[3] https://lkml.iu.edu/hypermail/linux/kernel/2111.0/04473.html

Thanks,
Andrei
> There used to be
> an attempt to track how accurate WF_SYNC was, using avg_overlap I think,
> but it was ultimately removed.
>
> --
> Mel Gorman
> SUSE Labs
K Prateek Nayak Nov. 14, 2022, 9:56 a.m. UTC | #6 
Hello Andrei,

I've tested this patch on the a dual socket Zen3 system
(2 x 64C/128T)

tl;dr

o I observe consistent regression for hackbench running
  smaller number of groups.
o tbench shows improvements for smaller number of clients
  but regresses for larger client counts.

I'll leave the detailed results below:

On 10/28/2022 1:56 AM, Andrei Vagin wrote:
> From: Andrei Vagin <avagin@gmail.com>
> 
> WF_SYNC means that the waker goes to sleep after wakeup, so the current
> cpu can be considered idle if the waker is the only process that is
> running on it.
> 
> The perf pipe benchmark shows that this change reduces the average time
> per operation from 8.8 usecs/op to 3.7 usecs/op.
> 
> Before:
>  $ ./tools/perf/perf bench sched pipe
>  # Running 'sched/pipe' benchmark:
>  # Executed 1000000 pipe operations between two processes
> 
>      Total time: 8.813 [sec]
> 
>        8.813985 usecs/op
>          113456 ops/sec
> 
> After:
>  $ ./tools/perf/perf bench sched pipe
>  # Running 'sched/pipe' benchmark:
>  # Executed 1000000 pipe operations between two processes
> 
>      Total time: 3.743 [sec]
> 
>        3.743971 usecs/op
>          267096 ops/sec
>

Following are the results from running standard benchmarks on a
dual socket Zen3 (2 x 64C/128T) machine configured in different
NPS modes.

NPS Modes are used to logically divide single socket into
multiple NUMA region.
Following is the NUMA configuration for each NPS mode on the system:

NPS1: Each socket is a NUMA node.
    Total 2 NUMA nodes in the dual socket machine.

    Node 0: 0-63,   128-191
    Node 1: 64-127, 192-255

NPS2: Each socket is further logically divided into 2 NUMA regions.
    Total 4 NUMA nodes exist over 2 socket.
   
    Node 0: 0-31,   128-159
    Node 1: 32-63,  160-191
    Node 2: 64-95,  192-223
    Node 3: 96-127, 223-255

NPS4: Each socket is logically divided into 4 NUMA regions.
    Total 8 NUMA nodes exist over 2 socket.
   
    Node 0: 0-15,    128-143
    Node 1: 16-31,   144-159
    Node 2: 32-47,   160-175
    Node 3: 48-63,   176-191
    Node 4: 64-79,   192-207
    Node 5: 80-95,   208-223
    Node 6: 96-111,  223-231
    Node 7: 112-127, 232-255

Benchmark Results:

Kernel versions:
- tip:          5.19.0 tip sched/core
- sync: 	5.19.0 tip sched/core + this patch

When we started testing, the tip was at:
commit fdf756f71271 ("sched: Fix more TASK_state comparisons")

~~~~~~~~~~~~~
~ hackbench ~
~~~~~~~~~~~~~

o NPS1

Test:			tip			sync
 1-groups:	   4.06 (0.00 pct)	   4.38 (-7.88 pct)     *
 2-groups:	   4.76 (0.00 pct)	   4.91 (-3.15 pct)
 4-groups:	   5.22 (0.00 pct)	   5.03 (3.63 pct)
 8-groups:	   5.35 (0.00 pct)	   5.23 (2.24 pct)
16-groups:	   7.21 (0.00 pct)	   6.86 (4.85 pct)

o NPS2

Test:			tip			sync
 1-groups:	   4.09 (0.00 pct)	   4.39 (-7.33 pct)     *
 2-groups:	   4.70 (0.00 pct)	   4.82 (-2.55 pct)
 4-groups:	   5.05 (0.00 pct)	   4.94 (2.17 pct)
 8-groups:	   5.35 (0.00 pct)	   5.15 (3.73 pct)
16-groups:	   6.37 (0.00 pct)	   6.55 (-2.82 pct)

o NPS4

Test:			tip			sync
 1-groups:	   4.07 (0.00 pct)	   4.31 (-5.89 pct)     *
 2-groups:	   4.65 (0.00 pct)	   4.79 (-3.01 pct)
 4-groups:	   5.13 (0.00 pct)	   4.99 (2.72 pct)
 8-groups:	   5.47 (0.00 pct)	   5.51 (-0.73 pct)
16-groups:	   6.82 (0.00 pct)	   7.07 (-3.66 pct)

~~~~~~~~~~~~
~ schbench ~
~~~~~~~~~~~~

o NPS1

#workers:	tip			sync
  1:	  33.00 (0.00 pct)	  32.00 (3.03 pct)
  2:	  35.00 (0.00 pct)	  36.00 (-2.85 pct)
  4:	  39.00 (0.00 pct)	  36.00 (7.69 pct)
  8:	  49.00 (0.00 pct)	  48.00 (2.04 pct)
 16:	  63.00 (0.00 pct)	  67.00 (-6.34 pct)
 32:	 109.00 (0.00 pct)	 107.00 (1.83 pct)
 64:	 208.00 (0.00 pct)	 220.00 (-5.76 pct)
128:	 559.00 (0.00 pct)	 551.00 (1.43 pct)
256:	 45888.00 (0.00 pct)	 40512.00 (11.71 pct)
512:	 80000.00 (0.00 pct)	 79744.00 (0.32 pct)

o NPS2

#workers:	tip			sync
  1:	  30.00 (0.00 pct)	  31.00 (-3.33 pct)
  2:	  37.00 (0.00 pct)	  36.00 (2.70 pct)
  4:	  39.00 (0.00 pct)	  42.00 (-7.69 pct)
  8:	  51.00 (0.00 pct)	  47.00 (7.84 pct)
 16:	  67.00 (0.00 pct)	  67.00 (0.00 pct)
 32:	 117.00 (0.00 pct)	 113.00 (3.41 pct)
 64:	 216.00 (0.00 pct)	 228.00 (-5.55 pct)
128:	 529.00 (0.00 pct)	 531.00 (-0.37 pct)
256:	 47040.00 (0.00 pct)	 42688.00 (9.25 pct)
512:	 84864.00 (0.00 pct)	 81280.00 (4.22 pct)

o NPS4

#workers:	tip			sync
  1:	  23.00 (0.00 pct)	  34.00 (-47.82 pct)
  2:	  28.00 (0.00 pct)	  35.00 (-25.00 pct)
  4:	  41.00 (0.00 pct)	  42.00 (-2.43 pct)
  8:	  60.00 (0.00 pct)	  55.00 (8.33 pct)
 16:	  71.00 (0.00 pct)	  67.00 (5.63 pct)
 32:	 117.00 (0.00 pct)	 116.00 (0.85 pct)
 64:	 227.00 (0.00 pct)	 221.00 (2.64 pct)
128:	 545.00 (0.00 pct)	 599.00 (-9.90 pct)
256:	 45632.00 (0.00 pct)	 45760.00 (-0.28 pct)
512:	 81024.00 (0.00 pct)	 79744.00 (1.57 pct)

Note: schebench at low worker count can show large run
to run variation. Unless the regressions are unusually
large, these data points can be ignored.

~~~~~~~~~~
~ tbench ~
~~~~~~~~~~

o NPS1

Clients:	tip			sync
    1	 578.37 (0.00 pct)	 652.14 (12.75 pct)
    2	 1062.09 (0.00 pct)	 1179.10 (11.01 pct)
    4	 1800.62 (0.00 pct)	 2160.13 (19.96 pct)
    8	 3211.02 (0.00 pct)	 3705.97 (15.41 pct)
   16	 4848.92 (0.00 pct)	 5906.04 (21.80 pct)
   32	 9091.36 (0.00 pct)	 10622.56 (16.84 pct)
   64	 15454.01 (0.00 pct)	 20319.16 (31.48 pct)
  128	 3511.33 (0.00 pct)	 31631.81 (800.84 pct)  *
  128    19910.99 (0.00pct)      31631.81 (58.86 pct)   [Verification Run]
  256	 50019.32 (0.00 pct)	 39234.55 (-21.56 pct)  *
  512	 44317.68 (0.00 pct)	 38788.24 (-12.47 pct)  *
 1024	 41200.85 (0.00 pct)	 37231.35 (-9.63 pct)   *

o NPS2

Clients:	tip			sync
    1	 576.05 (0.00 pct)	 648.53 (12.58 pct)
    2	 1037.68 (0.00 pct)	 1231.59 (18.68 pct)
    4	 1818.13 (0.00 pct)	 2173.43 (19.54 pct)
    8	 3004.16 (0.00 pct)	 3636.79 (21.05 pct)
   16	 4520.11 (0.00 pct)	 5786.93 (28.02 pct)
   32	 8624.23 (0.00 pct)	 10927.48 (26.70 pct)
   64	 14886.75 (0.00 pct)	 18573.28 (24.76 pct)
  128	 20602.00 (0.00 pct)	 28635.03 (38.99 pct)
  256	 45566.83 (0.00 pct)	 36262.90 (-20.41 pct)  *
  512	 42717.49 (0.00 pct)	 35884.09 (-15.99 pct)  *
 1024	 40936.61 (0.00 pct)	 37045.24 (-9.50 pct)   *

o NPS4

Clients:	tip			sync
    1	 576.36 (0.00 pct)	 658.78 (14.30 pct)
    2	 1044.26 (0.00 pct)	 1220.65 (16.89 pct)
    4	 1839.77 (0.00 pct)	 2190.02 (19.03 pct)
    8	 3043.53 (0.00 pct)	 3582.88 (17.72 pct)
   16	 5207.54 (0.00 pct)	 5349.74 (2.73 pct)
   32	 9263.86 (0.00 pct)	 10608.17 (14.51 pct)
   64	 14959.66 (0.00 pct)	 18186.46 (21.57 pct)
  128	 20698.65 (0.00 pct)	 31209.19 (50.77 pct)
  256	 46666.21 (0.00 pct)	 38551.07 (-17.38 pct)  *
  512	 41532.80 (0.00 pct)	 37525.65 (-9.64 pct)   *
 1024	 39459.49 (0.00 pct)	 36075.96 (-8.57 pct)   *

Note: On the tested kernel, with 128 clients, tbench can
run into a bottleneck during C-state exit. More details
can be found at
https://lore.kernel.org/lkml/20220921063638.2489-1-kprateek.nayak@amd.com/
This issue has been fixed in v6.0 but was not a part of
the tip kernel when we began testing. This data point has
been rerun with C2 disabled to get representative results.

~~~~~~~~~~
~ stream ~
~~~~~~~~~~

o NPS1

-> 10 Runs:

Test:		tip			sync
 Copy:	 328419.14 (0.00 pct)	 331174.37 (0.83 pct)
Scale:	 206071.21 (0.00 pct)	 211655.02 (2.70 pct)
  Add:	 235271.48 (0.00 pct)	 240925.76 (2.40 pct)
Triad:	 253175.80 (0.00 pct)	 250029.15 (-1.24 pct)

-> 100 Runs:

Test:		tip			sync
 Copy:	 328209.61 (0.00 pct)	 316634.10 (-3.52 pct)
Scale:	 216310.13 (0.00 pct)	 211496.10 (-2.22 pct)
  Add:	 244417.83 (0.00 pct)	 237258.24 (-2.92 pct)
Triad:	 237508.83 (0.00 pct)	 247541.91 (4.22 pct)

o NPS2

-> 10 Runs:

Test:		tip			sync
 Copy:	 336503.88 (0.00 pct)	 333502.90 (-0.89 pct)
Scale:	 218035.23 (0.00 pct)	 217009.06 (-0.47 pct)
  Add:	 257677.42 (0.00 pct)	 253882.69 (-1.47 pct)
Triad:	 268872.37 (0.00 pct)	 263099.47 (-2.14 pct)

-> 100 Runs:

Test:		tip			sync
 Copy:	 332304.34 (0.00 pct)	 336798.10 (1.35 pct)
Scale:	 223421.60 (0.00 pct)	 217501.94 (-2.64 pct)
  Add:	 252363.56 (0.00 pct)	 255571.69 (1.27 pct)
Triad:	 266687.56 (0.00 pct)	 262833.28 (-1.44 pct)

o NPS4

-> 10 Runs:

Test:		tip			sync
 Copy:	 353515.62 (0.00 pct)	 335743.68 (-5.02 pct)
Scale:	 228854.37 (0.00 pct)	 237557.44 (3.80 pct)
  Add:	 254942.12 (0.00 pct)	 259415.35 (1.75 pct)
Triad:	 270521.87 (0.00 pct)	 273002.56 (0.91 pct)

-> 100 Runs:

Test:		tip			sync
 Copy:	 374520.81 (0.00 pct)	 374736.48 (0.05 pct)
Scale:	 246280.23 (0.00 pct)	 237696.80 (-3.48 pct)
  Add:	 262772.72 (0.00 pct)	 259964.95 (-1.06 pct)
Triad:	 283740.92 (0.00 pct)	 279790.28 (-1.39 pct)

~~~~~~~~~~~~~~~~~~
~ Schedstat Data ~
~~~~~~~~~~~~~~~~~~

-> Following are the schedstat data from hackbench 1-group
   and tbench 64-clients and tbench 256-clients

-> Legend for per CPU stats:

rq->yld_count:                  sched_yield count
rq->sched_count:                schedule called
rq->sched_goidle:               schedule left the processor idle
rq->ttwu_count:                 try_to_wake_up was called
rq->ttwu_local:                 try_to_wake_up was called to wake up the local cpu
rq->rq_cpu_time:                total runtime by tasks on this processor (in jiffies)
rq->rq_sched_info.run_delay:    total waittime by tasks on this processor (in jiffies)
rq->rq_sched_info.pcount:       total timeslices run on this cpu

o Hackbench - NPS1

tip:    4.069s
sync:   4.525s

------------------------------------------------------------------------------------------------------------------------------------------------------
cpu:  all_cpus (avg) vs cpu:  all_cpus (avg)
------------------------------------------------------------------------------------------------------------------------------------------------------
kernel                                                     :           tip,         sync
sched_yield count                                          :             0,            0
Legacy counter can be ignored                              :             0,            0
schedule called                                            :         27633,        25474  |  -7.81|
schedule left the processor idle                           :         11609,        10587  |  -8.80|  (   42.01,  41.56  )
try_to_wake_up was called                                  :         15991,        14807  |  -7.40|
try_to_wake_up was called to wake up the local cpu         :           473,         1630  | 244.61|  (    2.96% of total,  11.01% of total  ) <--- More wakeup on local CPU
total runtime by tasks on this processor (in jiffies)      :     252079468,    316798504  |  25.67|
total waittime by tasks on this processor (in jiffies)     :     204693750,    207418931             (   81.20,  65.47  )
total timeslices run on this cpu                           :         16020,        14884  |  -7.09|  <------------------------ The increase in runtime has a
                                                                                                                               strong correlation with
                                                                                                                               rq->rq_sched_info.pcount
------------------------------------------------------------------------------------------------------------------------------------------------------

< -----------------------------------------------------------------  Wakeup info:  ----------------------------------------------------------------- >
kernel                                                  :           tip,         sync
Wakeups on same         SMT cpus = all_cpus (avg)       :           854,          556  | -34.89|
Wakeups on same         MC cpus = all_cpus (avg)        :         12855,         8624  | -32.91|
Wakeups on same         DIE cpus = all_cpus (avg)       :          1270,         2496  |  96.54|
Wakeups on same         NUMA cpus = all_cpus (avg)      :           538,         1500  | 178.81|
Affine wakeups on same  SMT cpus = all_cpus (avg)       :           590,          512  | -13.22|
Affine wakeups on same  MC cpus = all_cpus (avg)        :          8048,         6244  | -22.42|
Affine wakeups on same  DIE cpus = all_cpus (avg)       :           641,         1712  | 167.08|
Affine wakeups on same  NUMA cpus = all_cpus (avg)      :           256,          800  | 212.50|
------------------------------------------------------------------------------------------------------------------------------------------------------

o tbench - NPS1 (64 Clients)

tip:    15674.9 MB/sec
sync:   19510.4 MB/sec (+24.46%)

------------------------------------------------------------------------------------------------------------------------------------------------------
cpu:  all_cpus (avg) vs cpu:  all_cpus (avg)
------------------------------------------------------------------------------------------------------------------------------------------------------
kernel                                                     :           tip,         sync
sched_yield count                                          :             0,            0
Legacy counter can be ignored                              :             0,            0
schedule called                                            :       3245409,      2088248  | -35.66|
schedule left the processor idle                           :       1621656,         5675  | -99.65|  (   49.97% of total,   0.27% of total)
try_to_wake_up was called                                  :       1622705,      1373295  | -15.37|
try_to_wake_up was called to wake up the local cpu         :          1075,      1369101  |127258.23|  (    0.07% of total,  99.69% of total  ) <---- In case of modified kernel
total runtime by tasks on this processor (in jiffies)      :   18612280720,  17991066483                                                              all wakeup are on the same CPU
total waittime by tasks on this processor (in jiffies)     :       7698505,   7046293108  |91428.07|  (    0.04% of total,  39.17% of total  )
total timeslices run on this cpu                           :       1623752,      2082438  |  28.25|  <----------------------------------------------- Total rq->rq_sched_info.pcount is
                                                                                                                                                      larger on the modified kernel. Strong
                                                                                                                                                      correlation with improvements in BW
------------------------------------------------------------------------------------------------------------------------------------------------------

< -----------------------------------------------------------------  Wakeup info:  ----------------------------------------------------------------- >
kernel                                                  :           tip,         sync
Wakeups on same         SMT cpus = all_cpus (avg)       :         64021,         3757  | -94.13|
Wakeups on same         MC cpus = all_cpus (avg)        :       1557597,          392  | -99.97| <-- In most case, the affine wakeup
Wakeups on same         DIE cpus = all_cpus (avg)       :             4,           18  | 350.00|     is on another CPU is same MC doamin
Wakeups on same         NUMA cpus = all_cpus (avg)      :             5,           25  | 400.00|     in case of tip kernel
Affine wakeups on same  SMT cpus = all_cpus (avg)       :         64018,         1374  | -97.85|        |
Affine wakeups on same  MC cpus = all_cpus (avg)        :       1557431,          129  | -99.99| <-------
Affine wakeups on same  DIE cpus = all_cpus (avg)       :             3,           10  | 233.33|
Affine wakeups on same  NUMA cpus = all_cpus (avg)      :             2,           14  | 600.00|
------------------------------------------------------------------------------------------------------------------------------------------------------

o tbench - NPS1 (256 Clients)

tip:    44792.6 MB/sec
sync:   36050.4 MB/sec (-19.51%)

------------------------------------------------------------------------------------------------------------------------------------------------------
cpu:  all_cpus (avg) vs cpu:  all_cpus (avg)
------------------------------------------------------------------------------------------------------------------------------------------------------
kernel                                                     :           tip,         sync
sched_yield count                                          :             3,            0  |-100.00|
Legacy counter can be ignored                              :             0,            0
schedule called                                            :       4795945,      3839616  | -19.94|
schedule left the processor idle                           :         21549,           63  | -99.71|  (    0.45,      0  )
try_to_wake_up was called                                  :       3077285,      2526474  | -17.90|
try_to_wake_up was called to wake up the local cpu         :       3055451,      2526380  | -17.32|  (   99.29,    100  ) <------ More wakeup count for tip almost all of which
total runtime by tasks on this processor (in jiffies)      :   71776758037,  71864382378                                          is on the local CPU
total waittime by tasks on this processor (in jiffies)     :   29064423457,  27994939439             (   40.49,  38.96  )
total timeslices run on this cpu                           :       4774388,      3839547  | -19.58| <---------------------------- rq->rq_sched_info.pcount is lower on newer
                                                                                                                                  kernel which has strong correlation with the B/W drop
------------------------------------------------------------------------------------------------------------------------------------------------------

< -----------------------------------------------------------------  Wakeup info:  ----------------------------------------------------------------- >
kernel                                                  :           tip,         sync
Wakeups on same         SMT cpus = all_cpus (avg)       :         19979,           78  | -99.61|
Wakeups on same         MC cpus = all_cpus (avg)        :          1848,            9  | -99.51|
Wakeups on same         DIE cpus = all_cpus (avg)       :             3,            2  | -33.33|
Wakeups on same         NUMA cpus = all_cpus (avg)      :             3,            3           
Affine wakeups on same  SMT cpus = all_cpus (avg)       :         19860,           36  | -99.82|
Affine wakeups on same  MC cpus = all_cpus (avg)        :          1758,            4  | -99.77|
Affine wakeups on same  DIE cpus = all_cpus (avg)       :             2,            1  | -50.00|
Affine wakeups on same  NUMA cpus = all_cpus (avg)      :             2,            2           
------------------------------------------------------------------------------------------------------------------------------------------------------


> Cc: Ingo Molnar <mingo@redhat.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Juri Lelli <juri.lelli@redhat.com>
> Cc: Vincent Guittot <vincent.guittot@linaro.org>
> Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
> Cc: Steven Rostedt <rostedt@goodmis.org>
> Cc: Ben Segall <bsegall@google.com>
> Cc: Mel Gorman <mgorman@suse.de>
> Cc: Daniel Bristot de Oliveira <bristot@redhat.com>
> Cc: Valentin Schneider <vschneid@redhat.com>
> Signed-off-by: Andrei Vagin <avagin@gmail.com>
> ---
>  kernel/sched/fair.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index e4a0b8bd941c..40ac3cc68f5b 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -7245,7 +7245,8 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
>  		new_cpu = find_idlest_cpu(sd, p, cpu, prev_cpu, sd_flag);
>  	} else if (wake_flags & WF_TTWU) { /* XXX always ? */
>  		/* Fast path */
> -		new_cpu = select_idle_sibling(p, prev_cpu, new_cpu);
> +		if (!sync || cpu != new_cpu || this_rq()->nr_running != 1)
> +			new_cpu = select_idle_sibling(p, prev_cpu, new_cpu);

Adding perf stat data below which shows a larger dip in IPC for the patched
kernel as the system gets busy.

~~~~~~~~~~~~~
~ perf stat ~
~~~~~~~~~~~~~

Command: perf stat -a -e cycles -e instructions -- ./tbench_runner.sh

- tbench (NPS1)

-> 64 clients

o tip (15182 MB/sec):

18,054,464,226,798      cycles            
14,634,257,521,310      instructions #    0.81  insn per cycle      

o sync (19597.7 MB/sec [+29.08%]):

14,355,896,738,265      cycles
13,331,402,605,112      instructions #    0.93 (+14.81%)  insn per cycle  <-- Patched kernel has higher IPC
									      probably due to lesser stalls
									      with data warm at L1 and L2 cache.

-> 256 clients

o tip (51581 MB/sec):

51,719,263,738,848      cycles
34,387,747,050,053      instructions #    0.66  insn per cycle

o sync (42409 MB/sec [-17.78%]):

55,236,537,108,392      cycles
28,406,928,952,272      instructions #    0.51 (-22.72%)  insn per cycle  <-- Patched kernel has lower IPC when
									      system is busy.

>  	}
>  	rcu_read_unlock();
>  
If you would like me to run any specific workload on the
test system or gather any specific data, please let me
know.
--
Thanks and Regards,
Prateek
Mel Gorman Nov. 16, 2022, 10:57 a.m. UTC | #7 
On Mon, Oct 31, 2022 at 01:57:31PM +0100, Peter Zijlstra wrote:
> On Thu, Oct 27, 2022 at 01:26:03PM -0700, Andrei Vagin wrote:
> > From: Andrei Vagin <avagin@gmail.com>
> > 
> > WF_SYNC means that the waker goes to sleep after wakeup, so the current
> > cpu can be considered idle if the waker is the only process that is
> > running on it.
> > 
> > The perf pipe benchmark shows that this change reduces the average time
> > per operation from 8.8 usecs/op to 3.7 usecs/op.
> > 
> > Before:
> >  $ ./tools/perf/perf bench sched pipe
> >  # Running 'sched/pipe' benchmark:
> >  # Executed 1000000 pipe operations between two processes
> > 
> >      Total time: 8.813 [sec]
> > 
> >        8.813985 usecs/op
> >          113456 ops/sec
> > 
> > After:
> >  $ ./tools/perf/perf bench sched pipe
> >  # Running 'sched/pipe' benchmark:
> >  # Executed 1000000 pipe operations between two processes
> > 
> >      Total time: 3.743 [sec]
> > 
> >        3.743971 usecs/op
> >          267096 ops/sec
> 
> But what; if anything, does it do for the myrad of other benchmarks we
> run?

Varies as expected.

For a basic IO benchmark like dbench, the headline difference is small
although on occasion bad for high thread counts. Variability tends to be
higher.

Tbench4 tended to look great for lower thread counts as it's quite
synchronous but regresses for larger thread counts.

perf pipe tends to look great as it's strictly synchronous. On one machine
(1 socket Skylake), it showed a regression of 27% but sometimes it was
the opposite with 70-80% gains depending on the machine.

Then something like netperf gets punished severely across all machines.
TCP_STREAM is variable but UDP_STREAM gets punished severely for all
machines. TCP_STREAM sometimes shows gains and losses but mostly losses
of around 5% except for 1 machine. UDP_STREAM consistently shows losses
in the 40-60% mark even for the simple case of running on a UMA machine

netperf-udp
                                  6.1.0-rc3              6.1.0-rc3
                                    vanilla      sched-wfsync-v1r1
Hmean     send-64         235.21 (   0.00%)      112.75 * -52.06%*
Hmean     send-128        475.87 (   0.00%)      227.27 * -52.24%*
Hmean     send-256        968.82 (   0.00%)      452.43 * -53.30%*
Hmean     send-1024      3859.30 (   0.00%)     1792.63 * -53.55%*
Hmean     send-2048      7720.07 (   0.00%)     3525.27 * -54.34%*
Hmean     send-3312     12095.78 (   0.00%)     5587.11 * -53.81%*
Hmean     send-4096     14498.47 (   0.00%)     6824.96 * -52.93%*
Hmean     send-8192     25713.27 (   0.00%)    12474.87 * -51.48%*
Hmean     send-16384    43537.08 (   0.00%)    23080.04 * -46.99%*

This is not too surprising as UDP_STREAM is blasting packets so there are
wakeups but the waker is not going to sleep immediately. So yeah, there are
cases where the patch helps but when it hurts, it can hurt a lot. The patch
certainly demonstrates that there is room for improvement on how WF_SYNC is
treated but as it stands, it would start a game of apply/revert ping-pong
as different bisections showed the patch caused one set of problems and
the revert caused another.
Andrei Vagin Nov. 16, 2022, 6:50 p.m. UTC | #8 
Hello Mel and Prateek,

Thank you both for running tests and publishing results here.

On Wed, Nov 16, 2022 at 2:57 AM Mel Gorman <mgorman@suse.de> wrote:
<snip>
>
> This is not too surprising as UDP_STREAM is blasting packets so there are
> wakeups but the waker is not going to sleep immediately. So yeah, there are
> cases where the patch helps but when it hurts, it can hurt a lot. The patch
> certainly demonstrates that there is room for improvement on how WF_SYNC is
> treated but as it stands, it would start a game of apply/revert ping-pong
> as different bisections showed the patch caused one set of problems and
> the revert caused another.

I agree with these conclusions. The situation is more complex than how
I saw it initially.

Thanks,
Andrei
kernel test robot Nov. 21, 2022, 1:59 p.m. UTC | #9 
Greeting,

FYI, we noticed a 53.3% improvement of netperf.Throughput_tps due to commit:


commit: b6aabb01e3004b0285d953761e1592974b4b50fa ("[PATCH] sched: consider WF_SYNC to find idle siblings")
url: https://github.com/intel-lab-lkp/linux/commits/Andrei-Vagin/sched-consider-WF_SYNC-to-find-idle-siblings/20221028-042823
base: https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git c36eae5a86d15a694968be35d7ff056854997a72
patch link: https://lore.kernel.org/all/20221027202603.670616-1-avagin@google.com/
patch subject: [PATCH] sched: consider WF_SYNC to find idle siblings

in testcase: netperf
on test machine: 144 threads 4 sockets Intel(R) Xeon(R) Gold 5318H CPU @ 2.50GHz (Cooper Lake) with 128G memory
with following parameters:

	ip: ipv4
	runtime: 300s
	nr_threads: 25%
	cluster: cs-localhost
	test: TCP_RR
	cpufreq_governor: performance

test-description: Netperf is a benchmark that can be use to measure various aspect of networking performance.
test-url: http://www.netperf.org/netperf/

In addition to that, the commit also has significant impact on the following tests:

+------------------+-------------------------------------------------------------------------------------------+
| testcase: change | unixbench: unixbench.score 36.2% improvement                                              |
| test machine     | 16 threads 1 sockets Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz (Coffee Lake) with 32G memory |
| test parameters  | cpufreq_governor=performance                                                              |
|                  | nr_task=30%                                                                               |
|                  | runtime=300s                                                                              |
|                  | test=context1                                                                             |
+------------------+-------------------------------------------------------------------------------------------+




Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        sudo bin/lkp install job.yaml           # job file is attached in this email
        bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
        sudo bin/lkp run generated-yaml-file

        # if come across any failure that blocks the test,
        # please remove ~/.lkp and /lkp dir to run from a clean state.

=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase:
  cs-localhost/gcc-11/performance/ipv4/x86_64-rhel-8.3/25%/debian-11.1-x86_64-20220510.cgz/300s/lkp-cpl-4sp1/TCP_RR/netperf

commit: 
  c36eae5a86 ("sched/psi: Fix possible missing or delayed pending event")
  b6aabb01e3 ("sched: consider WF_SYNC to find idle siblings")

c36eae5a86d15a69 b6aabb01e3004b0285d953761e1 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
   3611595           +53.3%    5537882        netperf.Throughput_total_tps
    100322           +53.3%     153830        netperf.Throughput_tps
    130329        +8.7e+05%  1.128e+09 ± 19%  netperf.time.involuntary_context_switches
     16743 ±  2%     -22.6%      12962        netperf.time.minor_page_faults
      1202           +16.4%       1399 ±  2%  netperf.time.percent_of_cpu_this_job_got
      3432           +12.6%       3865 ±  2%  netperf.time.system_time
    188.48           +85.8%     350.16 ±  4%  netperf.time.user_time
 1.083e+09           -50.8%  5.334e+08 ± 39%  netperf.time.voluntary_context_switches
 1.083e+09           +53.3%  1.661e+09        netperf.workload
 2.212e+09           -97.0%   65981036        cpuidle..usage
     84551 ±  5%     -18.5%      68943 ±  3%  meminfo.Active
     84433 ±  5%     -18.5%      68833 ±  3%  meminfo.Active(anon)
     79312 ±  6%     -21.0%      62677 ±  5%  numa-meminfo.node3.Active
     79280 ±  6%     -21.0%      62645 ±  5%  numa-meminfo.node3.Active(anon)
     19806 ±  6%     -20.8%      15687 ±  5%  numa-vmstat.node3.nr_active_anon
     19806 ±  6%     -20.8%      15687 ±  5%  numa-vmstat.node3.nr_zone_active_anon
      8.60            -2.4        6.16        mpstat.cpu.all.soft%
     18.97            -1.9       17.04        mpstat.cpu.all.sys%
      1.30            +0.5        1.85 ±  3%  mpstat.cpu.all.usr%
     37.00           +47.7%      54.67        vmstat.procs.r
  14204693           -23.0%   10930950        vmstat.system.cs
    295062            +6.5%     314162        vmstat.system.in
     21070 ±  5%     -18.5%      17172 ±  3%  proc-vmstat.nr_active_anon
    106997            -2.5%     104321        proc-vmstat.nr_anon_pages
    409349 ±  4%     +10.0%     450452 ±  4%  proc-vmstat.nr_mapped
     83608            -1.5%      82352        proc-vmstat.nr_slab_unreclaimable
     21070 ±  5%     -18.5%      17172 ±  3%  proc-vmstat.nr_zone_active_anon
    531984 ±  9%     -15.9%     447458 ±  6%  proc-vmstat.numa_hint_faults_local
     24368 ±  7%     -29.6%      17161 ±  4%  proc-vmstat.pgactivate
   7844373            -3.4%    7576615        proc-vmstat.pgalloc_normal
   1450970 ±  2%     -18.0%    1189301        proc-vmstat.pgfree
      1358           -36.3%     864.67        turbostat.Avg_MHz
     41.28           -15.0       26.27        turbostat.Busy%
 1.281e+09          -100.0%      15898 ± 95%  turbostat.C1
     12.60           -12.6        0.00        turbostat.C1%
     58.66           +25.5%      73.65        turbostat.CPU%c1
      0.21           +77.0%       0.37        turbostat.IPC
     14.47 ± 21%     -14.5        0.00        turbostat.PKG_%
 8.861e+08          -100.0%       4493 ± 14%  turbostat.POLL
      6.47            -6.5        0.00        turbostat.POLL%
    571.65            -8.5%     522.90        turbostat.PkgWatt
      8.27            -4.0%       7.94        turbostat.RAMWatt
     27233 ± 32%     -95.4%       1255 ±136%  sched_debug.cfs_rq:/.MIN_vruntime.avg
    750764 ±  9%     -76.0%     180343 ±136%  sched_debug.cfs_rq:/.MIN_vruntime.max
    133981 ± 19%     -88.8%      14993 ±136%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
      0.26           +23.0%       0.32 ±  6%  sched_debug.cfs_rq:/.h_nr_running.avg
      1.03 ±  6%     +88.1%       1.93 ±  4%  sched_debug.cfs_rq:/.h_nr_running.max
      0.42           +46.9%       0.62 ±  5%  sched_debug.cfs_rq:/.h_nr_running.stddev
      9751 ± 10%     -32.8%       6552 ± 13%  sched_debug.cfs_rq:/.load.avg
     27233 ± 32%     -95.4%       1255 ±136%  sched_debug.cfs_rq:/.max_vruntime.avg
    750764 ±  9%     -76.0%     180343 ±136%  sched_debug.cfs_rq:/.max_vruntime.max
    133981 ± 19%     -88.8%      14993 ±136%  sched_debug.cfs_rq:/.max_vruntime.stddev
    724533 ±  8%     +35.8%     983881 ±  8%  sched_debug.cfs_rq:/.min_vruntime.avg
    845404 ± 10%     +54.1%    1302855 ±  8%  sched_debug.cfs_rq:/.min_vruntime.max
     27934 ± 13%    +297.8%     111130 ± 11%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.26           -18.8%       0.21 ±  4%  sched_debug.cfs_rq:/.nr_running.avg
    197.05           +90.2%     374.73 ±  4%  sched_debug.cfs_rq:/.runnable_avg.avg
    849.80 ±  4%     +77.3%       1506 ±  2%  sched_debug.cfs_rq:/.runnable_avg.max
    209.48          +161.4%     547.51        sched_debug.cfs_rq:/.runnable_avg.stddev
    114133 ± 29%    +172.2%     310722 ± 39%  sched_debug.cfs_rq:/.spread0.max
    -65385          +469.6%    -372412        sched_debug.cfs_rq:/.spread0.min
     27949 ± 13%    +297.7%     111154 ± 11%  sched_debug.cfs_rq:/.spread0.stddev
    196.98           +35.1%     266.15 ±  3%  sched_debug.cfs_rq:/.util_avg.avg
    849.66 ±  4%     +35.1%       1147 ±  3%  sched_debug.cfs_rq:/.util_avg.max
    209.42           +86.8%     391.19        sched_debug.cfs_rq:/.util_avg.stddev
     93.07 ±  4%     +76.1%     163.93 ±  4%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    664.98 ±  8%     +48.2%     985.29 ±  3%  sched_debug.cfs_rq:/.util_est_enqueued.max
    162.79           +91.6%     311.99        sched_debug.cfs_rq:/.util_est_enqueued.stddev
    250834 ±  9%     +84.0%     461429 ±  5%  sched_debug.cpu.avg_idle.avg
      3157 ±  6%    +939.5%      32825 ± 45%  sched_debug.cpu.avg_idle.min
    282325 ±  5%     +16.5%     328926        sched_debug.cpu.avg_idle.stddev
      7.58 ±  3%     +34.3%      10.18 ±  2%  sched_debug.cpu.clock.stddev
     10236 ±  4%     -13.5%       8854 ±  5%  sched_debug.cpu.curr->pid.max
      0.00 ± 13%     +55.3%       0.00 ±  4%  sched_debug.cpu.next_balance.stddev
      0.23 ±  3%     +41.6%       0.32 ±  5%  sched_debug.cpu.nr_running.avg
      1.03 ±  6%     +81.6%       1.87 ±  5%  sched_debug.cpu.nr_running.max
      0.40 ±  2%     +53.6%       0.62 ±  4%  sched_debug.cpu.nr_running.stddev
  13964454 ± 10%     -31.5%    9562330 ±  9%  sched_debug.cpu.nr_switches.avg
  15141338 ± 10%     -19.0%   12257576 ±  7%  sched_debug.cpu.nr_switches.max
  12639092 ±  9%     -54.5%    5746871 ± 11%  sched_debug.cpu.nr_switches.min
    444327 ± 13%    +150.9%    1114942 ± 10%  sched_debug.cpu.nr_switches.stddev
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.avg
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.max
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.min
     12.33           -96.1%       0.48 ±  7%  perf-stat.i.MPKI
 2.876e+10           +11.9%  3.217e+10        perf-stat.i.branch-instructions
      1.21            +0.1        1.26        perf-stat.i.branch-miss-rate%
 3.406e+08           +17.8%  4.012e+08        perf-stat.i.branch-misses
      1.90 ± 15%     +34.8       36.70 ± 16%  perf-stat.i.cache-miss-rate%
 1.739e+09           -95.6%   76617839 ±  7%  perf-stat.i.cache-references
  14360622           -23.1%   11042127        perf-stat.i.context-switches
      1.43           -46.7%       0.76        perf-stat.i.cpi
 2.023e+11           -38.7%  1.239e+11        perf-stat.i.cpu-cycles
      1322         +1109.9%      15995 ±  2%  perf-stat.i.cpu-migrations
      7928 ± 19%     -35.8%       5089 ± 19%  perf-stat.i.cycles-between-cache-misses
      0.02 ±  6%      -0.0        0.01 ±  4%  perf-stat.i.dTLB-load-miss-rate%
   6843508 ±  6%     -32.1%    4644002 ±  3%  perf-stat.i.dTLB-load-misses
 4.213e+10           +15.6%   4.87e+10        perf-stat.i.dTLB-loads
      0.00 ±  5%      -0.0        0.00 ±  4%  perf-stat.i.dTLB-store-miss-rate%
 2.439e+10           +14.5%  2.793e+10        perf-stat.i.dTLB-stores
     62.06            +2.8       64.90        perf-stat.i.iTLB-load-miss-rate%
 1.596e+08 ±  2%     +44.2%  2.302e+08        perf-stat.i.iTLB-load-misses
  97319700           +27.8%  1.244e+08        perf-stat.i.iTLB-loads
 1.429e+11           +14.0%  1.629e+11        perf-stat.i.instructions
    918.99 ±  2%     -19.1%     743.46        perf-stat.i.instructions-per-iTLB-miss
      0.71           +85.9%       1.31        perf-stat.i.ipc
      1.40           -38.7%       0.86        perf-stat.i.metric.GHz
    873.16           +76.2%       1538 ±  3%  perf-stat.i.metric.K/sec
    673.72           +12.2%     755.60        perf-stat.i.metric.M/sec
     82.06 ±  3%      +9.3       91.32        perf-stat.i.node-store-miss-rate%
   1049204 ± 16%     +53.8%    1613542 ± 18%  perf-stat.i.node-store-misses
    443904 ±  6%     -30.0%     310869 ±  6%  perf-stat.i.node-stores
     12.17           -96.1%       0.47 ±  7%  perf-stat.overall.MPKI
      1.18            +0.1        1.25        perf-stat.overall.branch-miss-rate%
      1.75 ± 17%     +36.3       38.07 ± 15%  perf-stat.overall.cache-miss-rate%
      1.42           -46.3%       0.76        perf-stat.overall.cpi
      6820 ± 15%     -35.7%       4383 ± 17%  perf-stat.overall.cycles-between-cache-misses
      0.02 ±  6%      -0.0        0.01 ±  4%  perf-stat.overall.dTLB-load-miss-rate%
      0.00 ±  5%      -0.0        0.00 ±  3%  perf-stat.overall.dTLB-store-miss-rate%
     62.11            +2.8       64.92        perf-stat.overall.iTLB-load-miss-rate%
    895.64 ±  2%     -21.0%     707.76        perf-stat.overall.instructions-per-iTLB-miss
      0.71           +86.1%       1.31        perf-stat.overall.ipc
     69.85 ±  6%     +13.7       83.54 ±  2%  perf-stat.overall.node-store-miss-rate%
     39789           -25.6%      29606        perf-stat.overall.path-length
 2.866e+10           +11.9%  3.206e+10        perf-stat.ps.branch-instructions
 3.395e+08           +17.8%  3.999e+08        perf-stat.ps.branch-misses
 1.733e+09           -95.6%   76361868 ±  7%  perf-stat.ps.cache-references
  14312085           -23.1%   11004648        perf-stat.ps.context-switches
 2.016e+11           -38.7%  1.235e+11        perf-stat.ps.cpu-cycles
      1317         +1109.8%      15942 ±  2%  perf-stat.ps.cpu-migrations
   6808921 ±  6%     -32.1%    4621256 ±  3%  perf-stat.ps.dTLB-load-misses
 4.199e+10           +15.6%  4.854e+10        perf-stat.ps.dTLB-loads
 2.431e+10           +14.5%  2.784e+10        perf-stat.ps.dTLB-stores
 1.591e+08 ±  2%     +44.2%  2.294e+08        perf-stat.ps.iTLB-load-misses
  96991959           +27.8%  1.239e+08        perf-stat.ps.iTLB-loads
 1.424e+11           +14.0%  1.624e+11        perf-stat.ps.instructions
      0.01 ± 50%     -72.0%       0.00 ±115%  perf-stat.ps.major-faults
   1046642 ± 16%     +53.7%    1608395 ± 18%  perf-stat.ps.node-store-misses
    442450 ±  6%     -29.9%     309961 ±  7%  perf-stat.ps.node-stores
 4.311e+13           +14.1%  4.919e+13        perf-stat.total.instructions
     55.82 ±  5%     -21.0       34.84 ± 14%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     55.50 ±  5%     -20.9       34.61 ± 14%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     55.49 ±  5%     -20.9       34.61 ± 14%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     55.40 ±  5%     -20.8       34.61 ± 14%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     48.23 ±  7%     -13.7       34.56 ± 14%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     46.09 ±  7%     -11.7       34.40 ± 14%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
     46.05 ±  7%     -11.7       34.36 ± 14%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
     10.29 ±  7%     -10.3        0.00        perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     10.00 ±  6%     -10.0        0.00        perf-profile.calltrace.cycles-pp.intel_idle_irq.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      9.36 ±  6%      -9.4        0.00        perf-profile.calltrace.cycles-pp.mwait_idle_with_hints.intel_idle_irq.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
      1.58 ±  7%      -1.2        0.40 ± 72%  perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.schedule_timeout
      6.53 ±  6%      -1.1        5.42 ± 13%  perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv.send_omni_inner
      0.00            +0.8        0.82 ±  8%  perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare
      0.00            +0.9        0.88 ± 42%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.recv_omni
      0.00            +0.9        0.93 ±  9%  perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__schedule.schedule.schedule_timeout.wait_woken
      0.00            +1.0        0.98 ±  7%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send
      0.00            +1.0        0.98 ±  7%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send
      0.00            +1.0        0.98 ±  7%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__send
      0.00            +1.0        1.03 ± 22%  perf-profile.calltrace.cycles-pp.skb_release_data.__kfree_skb.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established
      0.00            +1.1        1.13 ±  9%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.recv
      0.00            +1.1        1.13 ±  9%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv
      0.00            +1.1        1.13 ±  9%  perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv
      0.00            +1.2        1.15 ±  8%  perf-profile.calltrace.cycles-pp.__kfree_skb.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv
      1.54 ±  6%      +1.2        2.76 ±  8%  perf-profile.calltrace.cycles-pp.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
      0.00            +1.3        1.32 ±  7%  perf-profile.calltrace.cycles-pp.skb_do_copy_data_nocache.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
      5.02 ±  7%      +1.3        6.35 ±  8%  perf-profile.calltrace.cycles-pp.wait_woken.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
      1.33 ±  7%      +1.4        2.74 ±  7%  perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit
     10.51 ±  5%      +1.4       11.96 ±  8%  perf-profile.calltrace.cycles-pp.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog
      8.66 ±  7%      +1.5       10.14 ±  8%  perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
     14.38 ±  6%      +1.5       15.87 ±  7%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq.__local_bh_enable_ip.ip_finish_output2.__ip_queue_xmit
      8.75 ±  7%      +1.5       10.27 ±  8%  perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core
      1.06 ± 22%      +1.5        2.60 ±  8%  perf-profile.calltrace.cycles-pp.tcp_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
     14.55 ±  6%      +1.5       16.09 ±  7%  perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb
     14.67 ±  6%      +1.6       16.26 ±  7%  perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit
     13.28 ±  6%      +1.7       14.93 ±  8%  perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv
      0.00            +1.7        1.67 ±  8%  perf-profile.calltrace.cycles-pp.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit
     10.67 ±  5%      +1.7       12.38 ±  7%  perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll
      0.00            +1.7        1.75 ± 14%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.__wake_up_common.__wake_up_common_lock
      2.39 ±  6%      +1.8        4.16 ±  8%  perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
     10.70 ±  5%      +1.8       12.47 ±  7%  perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action
      0.00            +1.8        1.78 ±  8%  perf-profile.calltrace.cycles-pp.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb
      0.00            +1.8        1.79 ± 13%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable
      0.00            +1.8        1.84 ±  8%  perf-profile.calltrace.cycles-pp.__alloc_skb.tcp_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
      3.91 ±  7%      +1.9        5.78 ±  8%  perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked
      3.68 ±  7%      +1.9        5.58 ±  8%  perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.wait_woken.sk_wait_data
      4.03 ±  7%      +1.9        5.97 ±  8%  perf-profile.calltrace.cycles-pp.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg
      0.00            +2.0        1.99 ±  8%  perf-profile.calltrace.cycles-pp.__send
     11.19 ±  6%      +2.2       13.36 ±  7%  perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action.__softirqentry_text_start
      0.00            +2.3        2.26 ± 21%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.send_omni_inner
      0.00            +2.3        2.29 ±  9%  perf-profile.calltrace.cycles-pp.recv
     11.56 ±  6%      +2.5       14.09 ±  7%  perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.__softirqentry_text_start.do_softirq
      0.00            +2.6        2.59 ±  8%  perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode
     11.58 ±  6%      +2.6       14.18 ±  7%  perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.__softirqentry_text_start.do_softirq.__local_bh_enable_ip
      0.00            +2.7        2.72 ±  8%  perf-profile.calltrace.cycles-pp.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      0.00            +2.9        2.90 ±  8%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
     16.38 ±  6%      +3.4       19.76 ±  7%  perf-profile.calltrace.cycles-pp.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames
      0.00            +3.9        3.86 ±  3%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send
     17.18 ±  6%      +4.3       21.44 ±  6%  perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked
     12.91 ±  6%      +4.6       17.50 ±  7%  perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.recv_omni
     18.67 ±  6%      +5.3       24.00 ±  6%  perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg
     13.02 ±  6%      +5.5       18.56 ±  7%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.recv_omni.process_requests
     13.10 ±  6%      +5.7       18.77 ±  7%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__send.recv_omni.process_requests.spawn_child
     13.24 ±  6%      +5.8       19.06 ±  7%  perf-profile.calltrace.cycles-pp.__send.recv_omni.process_requests.spawn_child.accept_connection
     20.18 ±  6%      +6.3       26.50 ±  6%  perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
     20.27 ±  6%      +6.5       26.78 ±  6%  perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
     12.70 ±  6%      +6.5       19.22 ±  9%  perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.send_omni_inner
     21.45 ±  6%      +8.3       29.75 ±  7%  perf-profile.calltrace.cycles-pp.recv_omni.process_requests.spawn_child.accept_connection.accept_connections
     12.80 ±  6%      +8.8       21.60 ± 10%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.send_omni_inner.send_tcp_rr
     19.60 ±  7%      +8.9       28.49 ±  7%  perf-profile.calltrace.cycles-pp.send_omni_inner.send_tcp_rr.main.__libc_start_main
     24.83 ± 19%      +8.9       33.77 ± 15%  perf-profile.calltrace.cycles-pp.mwait_idle_with_hints.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
     24.83 ± 19%      +8.9       33.77 ± 15%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     19.76 ±  7%      +9.0       28.80 ±  7%  perf-profile.calltrace.cycles-pp.send_tcp_rr.main.__libc_start_main
     11.96 ±  7%      +9.4       21.31 ±  9%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__send.send_omni_inner.send_tcp_rr.main
     24.07 ±  6%      +9.4       33.44 ±  7%  perf-profile.calltrace.cycles-pp.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
     12.09 ±  7%      +9.4       21.47 ±  9%  perf-profile.calltrace.cycles-pp.__send.send_omni_inner.send_tcp_rr.main.__libc_start_main
     24.63 ±  6%      +9.8       34.41 ±  7%  perf-profile.calltrace.cycles-pp.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64
     25.07 ±  6%     +10.3       35.41 ±  7%  perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
     25.49 ±  6%     +11.0       36.46 ±  7%  perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send
     55.82 ±  5%     -21.0       34.84 ± 14%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     55.82 ±  5%     -21.0       34.84 ± 14%  perf-profile.children.cycles-pp.cpu_startup_entry
     55.50 ±  5%     -20.9       34.61 ± 14%  perf-profile.children.cycles-pp.start_secondary
     55.74 ±  5%     -20.9       34.84 ± 14%  perf-profile.children.cycles-pp.do_idle
     48.52 ±  7%     -13.7       34.79 ± 14%  perf-profile.children.cycles-pp.cpuidle_idle_call
     46.36 ±  7%     -11.7       34.63 ± 14%  perf-profile.children.cycles-pp.cpuidle_enter
     46.34 ±  7%     -11.7       34.63 ± 14%  perf-profile.children.cycles-pp.cpuidle_enter_state
     10.35 ±  7%     -10.4        0.00        perf-profile.children.cycles-pp.poll_idle
     10.06 ±  6%     -10.1        0.00        perf-profile.children.cycles-pp.intel_idle_irq
      3.04 ±  7%      -2.3        0.72 ± 10%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      1.70 ±  5%      -1.6        0.15 ± 16%  perf-profile.children.cycles-pp.menu_select
      1.33 ±  6%      -1.3        0.03 ± 70%  perf-profile.children.cycles-pp.ttwu_queue_wakelist
      1.68 ±  7%      -0.9        0.75 ±  8%  perf-profile.children.cycles-pp.dequeue_entity
      0.92 ±  7%      -0.6        0.29 ± 18%  perf-profile.children.cycles-pp.select_task_rq
      0.73 ±  4%      -0.6        0.10 ± 18%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.90 ±  8%      -0.6        0.28 ±  7%  perf-profile.children.cycles-pp.skb_attempt_defer_free
      0.79 ±  7%      -0.6        0.24 ± 19%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.66 ±  7%      -0.5        0.19 ± 10%  perf-profile.children.cycles-pp.sock_rfree
      0.48 ±  3%      -0.4        0.09 ± 20%  perf-profile.children.cycles-pp.tick_nohz_next_event
      1.11 ± 10%      -0.4        0.74 ± 18%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.64 ±  5%      -0.4        0.28 ± 11%  perf-profile.children.cycles-pp.remove_wait_queue
      0.96 ±  7%      -0.3        0.62 ± 11%  perf-profile.children.cycles-pp.__inet_lookup_established
      0.81 ±  5%      -0.3        0.47 ± 13%  perf-profile.children.cycles-pp.prepare_task_switch
      0.82 ±  7%      -0.3        0.52 ±  7%  perf-profile.children.cycles-pp._raw_spin_lock
      0.57 ±  9%      -0.2        0.33 ± 11%  perf-profile.children.cycles-pp.tcp_check_space
      1.10 ±  7%      -0.2        0.88 ±  8%  perf-profile.children.cycles-pp.enqueue_entity
      0.56 ±  8%      -0.2        0.35 ±  9%  perf-profile.children.cycles-pp.copyout
      0.55 ±  9%      -0.2        0.34 ±  7%  perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
      0.35 ±  8%      -0.2        0.16 ±  8%  perf-profile.children.cycles-pp.kfree_skbmem
      0.58 ±  7%      -0.2        0.41 ±  7%  perf-profile.children.cycles-pp.copy_user_short_string
      0.31 ±  8%      -0.2        0.15 ± 11%  perf-profile.children.cycles-pp.perf_tp_event
      0.20 ±  8%      -0.1        0.05 ±  7%  perf-profile.children.cycles-pp.skb_release_head_state
      0.58 ±  7%      -0.1        0.47 ±  8%  perf-profile.children.cycles-pp.tcp_rcv_space_adjust
      0.09 ± 15%      -0.0        0.04 ± 45%  perf-profile.children.cycles-pp.rebalance_domains
      0.07 ± 11%      -0.0        0.03 ± 70%  perf-profile.children.cycles-pp.tracing_gen_ctx_irq_test
      0.07 ±  7%      +0.0        0.09 ±  7%  perf-profile.children.cycles-pp.tcp_rate_skb_delivered
      0.15 ±  7%      +0.0        0.18 ±  9%  perf-profile.children.cycles-pp.add_wait_queue
      0.07 ±  7%      +0.0        0.10 ± 10%  perf-profile.children.cycles-pp.tcp_event_data_recv
      0.04 ± 44%      +0.0        0.08 ± 10%  perf-profile.children.cycles-pp.place_entity
      0.11 ± 11%      +0.0        0.15 ± 15%  perf-profile.children.cycles-pp.tcp_rearm_rto
      0.06 ± 14%      +0.0        0.11 ± 12%  perf-profile.children.cycles-pp.ip_skb_dst_mtu
      0.02 ±141%      +0.0        0.06 ± 21%  perf-profile.children.cycles-pp.timekeeping_max_deferment
      0.05 ± 47%      +0.0        0.09 ±  7%  perf-profile.children.cycles-pp.nf_hook_slow
      0.02 ±141%      +0.0        0.06 ± 11%  perf-profile.children.cycles-pp.__cgroup_account_cputime
      0.22 ± 10%      +0.1        0.27 ±  9%  perf-profile.children.cycles-pp.__mod_timer
      0.09 ± 14%      +0.1        0.14 ± 19%  perf-profile.children.cycles-pp.scheduler_tick
      0.00            +0.1        0.05 ±  7%  perf-profile.children.cycles-pp.minmax_running_min
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.ip_local_deliver
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.tcp_options_write
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.inet_sendmsg
      0.01 ±223%      +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.skb_clone
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.tcp_small_queue_check
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.cubictcp_cong_avoid
      0.00            +0.1        0.06 ± 18%  perf-profile.children.cycles-pp.security_sock_rcv_skb
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.tcp_chrono_stop
      0.00            +0.1        0.06 ± 15%  perf-profile.children.cycles-pp.eth_type_trans
      0.00            +0.1        0.06 ± 14%  perf-profile.children.cycles-pp.kmalloc_slab
      0.00            +0.1        0.06 ± 17%  perf-profile.children.cycles-pp.__raise_softirq_irqoff
      0.06 ±  6%      +0.1        0.12 ±  7%  perf-profile.children.cycles-pp.sk_filter_trim_cap
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.switch_ldt
      0.00            +0.1        0.06 ± 14%  perf-profile.children.cycles-pp.rb_insert_color
      0.27 ±  9%      +0.1        0.33 ±  7%  perf-profile.children.cycles-pp.sk_reset_timer
      0.10 ± 15%      +0.1        0.17 ± 11%  perf-profile.children.cycles-pp.perf_trace_sched_switch
      0.10 ±  8%      +0.1        0.16 ±  9%  perf-profile.children.cycles-pp.__ip_finish_output
      0.10 ± 14%      +0.1        0.17 ± 11%  perf-profile.children.cycles-pp.__tcp_cleanup_rbuf
      0.08 ±  8%      +0.1        0.15 ± 11%  perf-profile.children.cycles-pp.update_min_vruntime
      0.06 ±  9%      +0.1        0.13 ±  9%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
      0.09 ±  6%      +0.1        0.16 ±  9%  perf-profile.children.cycles-pp.try_charge_memcg
      0.00            +0.1        0.07 ± 10%  perf-profile.children.cycles-pp.check_cfs_rq_runtime
      0.00            +0.1        0.07 ± 10%  perf-profile.children.cycles-pp.clear_buddies
      0.04 ± 44%      +0.1        0.11 ±  6%  perf-profile.children.cycles-pp.tcp_established_options
      0.00            +0.1        0.07 ± 14%  perf-profile.children.cycles-pp.apparmor_socket_recvmsg
      0.61 ±  7%      +0.1        0.68 ±  6%  perf-profile.children.cycles-pp.__switch_to
      0.08 ±  6%      +0.1        0.15 ±  9%  perf-profile.children.cycles-pp.__copy_skb_header
      0.00            +0.1        0.07 ± 20%  perf-profile.children.cycles-pp.cubictcp_acked
      0.06 ±  9%      +0.1        0.13 ±  5%  perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
      0.00            +0.1        0.07 ± 12%  perf-profile.children.cycles-pp.memcg_slab_post_alloc_hook
      0.06 ± 11%      +0.1        0.14 ±  9%  perf-profile.children.cycles-pp.tcp_rate_skb_sent
      0.08 ± 13%      +0.1        0.16 ±  8%  perf-profile.children.cycles-pp.rcu_all_qs
      0.12 ± 12%      +0.1        0.20 ±  8%  perf-profile.children.cycles-pp.ip_send_check
      0.06 ±  9%      +0.1        0.13 ±  7%  perf-profile.children.cycles-pp.inet_ehashfn
      0.03 ± 70%      +0.1        0.11 ±  8%  perf-profile.children.cycles-pp.sk_free
      0.06 ± 11%      +0.1        0.14 ±  8%  perf-profile.children.cycles-pp.raw_v4_input
      0.00            +0.1        0.08 ± 12%  perf-profile.children.cycles-pp.tcp_inbound_md5_hash
      0.00            +0.1        0.08 ± 10%  perf-profile.children.cycles-pp.neigh_hh_output
      0.19 ± 17%      +0.1        0.28 ± 18%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.04 ± 45%      +0.1        0.13 ±  8%  perf-profile.children.cycles-pp.tcp_rate_gen
      0.00            +0.1        0.08 ±  5%  perf-profile.children.cycles-pp.syscall_enter_from_user_mode
      0.00            +0.1        0.09 ± 10%  perf-profile.children.cycles-pp.apparmor_socket_sendmsg
      0.58 ±  8%      +0.1        0.67 ±  8%  perf-profile.children.cycles-pp.tcp_event_new_data_sent
      0.00            +0.1        0.09 ±  6%  perf-profile.children.cycles-pp.check_stack_object
      0.00            +0.1        0.09 ±  6%  perf-profile.children.cycles-pp.tcp_v4_fill_cb
      0.00            +0.1        0.09 ± 11%  perf-profile.children.cycles-pp.tcp_newly_delivered
      0.07 ± 13%      +0.1        0.16 ±  5%  perf-profile.children.cycles-pp.sock_put
      0.10 ±  8%      +0.1        0.19 ±  7%  perf-profile.children.cycles-pp.__ksize
      0.05 ± 45%      +0.1        0.14 ± 13%  perf-profile.children.cycles-pp.netif_skb_features
      0.10 ±  9%      +0.1        0.20 ± 14%  perf-profile.children.cycles-pp.sk_forced_mem_schedule
      0.04 ± 71%      +0.1        0.13 ±  5%  perf-profile.children.cycles-pp.__usecs_to_jiffies
      0.09 ± 14%      +0.1        0.19 ± 14%  perf-profile.children.cycles-pp.tcp_rtt_estimator
      0.00            +0.1        0.10 ±  4%  perf-profile.children.cycles-pp.tcp_push
      0.10 ± 10%      +0.1        0.21 ±  7%  perf-profile.children.cycles-pp.__tcp_select_window
      0.63 ±  6%      +0.1        0.74 ±  7%  perf-profile.children.cycles-pp.simple_copy_to_iter
      0.00            +0.1        0.11 ± 11%  perf-profile.children.cycles-pp.resched_curr
      0.07 ± 12%      +0.1        0.19 ± 22%  perf-profile.children.cycles-pp.cgroup_rstat_updated
      0.43 ±  6%      +0.1        0.54 ±  9%  perf-profile.children.cycles-pp.update_rq_clock
      0.12 ±  9%      +0.1        0.24 ±  7%  perf-profile.children.cycles-pp.ip_output
      0.18 ±  9%      +0.1        0.29 ± 14%  perf-profile.children.cycles-pp.__calc_delta
      0.50 ±  6%      +0.1        0.63 ±  9%  perf-profile.children.cycles-pp.set_next_entity
      0.09 ±  6%      +0.1        0.21 ±  7%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.28 ±  7%      +0.1        0.41 ±  8%  perf-profile.children.cycles-pp.recv_data
      0.00            +0.1        0.13 ±  8%  perf-profile.children.cycles-pp.cubictcp_cwnd_event
      0.04 ± 71%      +0.1        0.17 ± 13%  perf-profile.children.cycles-pp.skb_page_frag_refill
      0.11 ±  8%      +0.1        0.24 ± 10%  perf-profile.children.cycles-pp.tcp_tso_segs
      0.07 ± 14%      +0.1        0.20 ± 10%  perf-profile.children.cycles-pp.sk_page_frag_refill
      0.11 ±  6%      +0.1        0.25 ±  6%  perf-profile.children.cycles-pp.__netif_receive_skb_core
      0.30 ±  6%      +0.1        0.44 ±  8%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.08 ± 18%      +0.1        0.23 ± 19%  perf-profile.children.cycles-pp.cpuacct_charge
      0.29 ±  9%      +0.1        0.44 ±  8%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.08 ±  8%      +0.1        0.23 ±  9%  perf-profile.children.cycles-pp.ip_rcv_core
      0.16 ±  9%      +0.2        0.32 ±  8%  perf-profile.children.cycles-pp.__ip_local_out
      0.29 ±  6%      +0.2        0.46 ±  6%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.12 ±  6%      +0.2        0.28 ± 10%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.35 ±  4%      +0.2        0.52 ±  8%  perf-profile.children.cycles-pp.update_cfs_group
      0.14 ±  5%      +0.2        0.31 ±  7%  perf-profile.children.cycles-pp.copyin
      0.11 ±  6%      +0.2        0.28 ±  8%  perf-profile.children.cycles-pp.refill_stock
      0.18 ±  7%      +0.2        0.35 ±  8%  perf-profile.children.cycles-pp.__cond_resched
      0.14 ±  7%      +0.2        0.32 ±  8%  perf-profile.children.cycles-pp.tcp_update_pacing_rate
      0.11 ±  9%      +0.2        0.28 ±  8%  perf-profile.children.cycles-pp.tcp_update_skb_after_send
      0.15 ± 10%      +0.2        0.32 ± 15%  perf-profile.children.cycles-pp.mem_cgroup_uncharge_skmem
      0.17 ±  6%      +0.2        0.34 ±  7%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.12 ±  9%      +0.2        0.30 ± 14%  perf-profile.children.cycles-pp.validate_xmit_skb
      0.15 ± 20%      +0.2        0.33 ± 17%  perf-profile.children.cycles-pp.find_busiest_group
      0.07 ±  9%      +0.2        0.26 ±  8%  perf-profile.children.cycles-pp.__list_add_valid
      0.18 ±  8%      +0.2        0.36 ± 10%  perf-profile.children.cycles-pp.ip_local_out
      0.17 ±  6%      +0.2        0.36 ± 17%  perf-profile.children.cycles-pp.__mod_memcg_state
      0.16 ±  4%      +0.2        0.35 ±  9%  perf-profile.children.cycles-pp.tcp_wfree
      0.46 ±  6%      +0.2        0.66 ±  7%  perf-profile.children.cycles-pp.read_tsc
      0.11 ±  9%      +0.2        0.31 ±  8%  perf-profile.children.cycles-pp.__might_fault
      0.00            +0.2        0.20 ± 10%  perf-profile.children.cycles-pp.set_next_buddy
      0.12 ± 21%      +0.2        0.33 ± 17%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.19 ± 17%      +0.2        0.40 ± 17%  perf-profile.children.cycles-pp.load_balance
      0.06 ±  7%      +0.2        0.27 ±  8%  perf-profile.children.cycles-pp.__kmem_cache_free
      0.10 ± 23%      +0.2        0.32 ± 17%  perf-profile.children.cycles-pp.update_sg_lb_stats
      0.18 ±  8%      +0.2        0.39 ±  6%  perf-profile.children.cycles-pp.__might_sleep
      0.23 ±  9%      +0.2        0.45 ±  7%  perf-profile.children.cycles-pp.send_data
      0.35 ±  8%      +0.2        0.58 ±  8%  perf-profile.children.cycles-pp.__skb_clone
      0.67 ±  6%      +0.2        0.90 ±  9%  perf-profile.children.cycles-pp.__switch_to_asm
      0.07 ± 10%      +0.2        0.31 ±  8%  perf-profile.children.cycles-pp.kmem_cache_free
      0.07 ± 12%      +0.2        0.32 ±  6%  perf-profile.children.cycles-pp.put_prev_entity
      0.12 ± 10%      +0.2        0.37 ±  8%  perf-profile.children.cycles-pp.tcp_ack_update_rtt
      0.24 ±  7%      +0.3        0.50 ±  9%  perf-profile.children.cycles-pp.mem_cgroup_charge_skmem
      0.27 ± 12%      +0.3        0.54 ± 12%  perf-profile.children.cycles-pp.ip_rcv
      0.48 ±  9%      +0.3        0.75 ±  7%  perf-profile.children.cycles-pp.release_sock
      0.22 ±  8%      +0.3        0.52 ±  6%  perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.25 ±  7%      +0.3        0.56 ±  8%  perf-profile.children.cycles-pp.os_xsave
      1.01 ±  6%      +0.3        1.33 ±  8%  perf-profile.children.cycles-pp.skb_release_data
      0.56 ±  8%      +0.3        0.87 ±  7%  perf-profile.children.cycles-pp._raw_spin_lock_bh
      0.24 ±  8%      +0.3        0.56 ±  9%  perf-profile.children.cycles-pp.__kmem_cache_alloc_node
      0.14 ±  3%      +0.3        0.47 ±  7%  perf-profile.children.cycles-pp.__virt_addr_valid
      0.20 ±  8%      +0.3        0.54 ±  8%  perf-profile.children.cycles-pp.tcp_schedule_loss_probe
      0.83 ±  5%      +0.3        1.17 ±  7%  perf-profile.children.cycles-pp.__check_object_size
      0.06 ±  9%      +0.4        0.41 ±  9%  perf-profile.children.cycles-pp.import_single_range
      0.35 ±  7%      +0.4        0.71 ±  7%  perf-profile.children.cycles-pp.lock_sock_nested
      0.13 ± 10%      +0.4        0.50 ± 11%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.10 ± 13%      +0.4        0.47 ± 11%  perf-profile.children.cycles-pp.check_preempt_curr
      0.27 ±  6%      +0.4        0.66 ±  9%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.20 ±  9%      +0.4        0.59 ±  7%  perf-profile.children.cycles-pp.kmem_cache_alloc_node
      0.84 ±  2%      +0.4        1.23 ± 11%  perf-profile.children.cycles-pp.ktime_get
      0.30 ±  5%      +0.4        0.70 ±  8%  perf-profile.children.cycles-pp.security_socket_sendmsg
      0.28 ±  7%      +0.4        0.68 ±  9%  perf-profile.children.cycles-pp.__kmalloc_node_track_caller
      0.19 ±  5%      +0.4        0.60 ±  9%  perf-profile.children.cycles-pp.enqueue_to_backlog
      0.94 ±  6%      +0.4        1.34 ±  7%  perf-profile.children.cycles-pp.update_load_avg
      0.00            +0.4        0.41 ± 12%  perf-profile.children.cycles-pp.check_preempt_wakeup
      0.21 ±  6%      +0.4        0.63 ±  8%  perf-profile.children.cycles-pp.netif_rx_internal
      0.21 ±  5%      +0.4        0.64 ±  9%  perf-profile.children.cycles-pp.__netif_rx
      0.30 ±  8%      +0.4        0.74 ±  9%  perf-profile.children.cycles-pp.kmalloc_reserve
      0.34 ±  5%      +0.4        0.78 ±  7%  perf-profile.children.cycles-pp.tcp_mstamp_refresh
      0.26 ±  8%      +0.4        0.71 ± 10%  perf-profile.children.cycles-pp.reweight_entity
      0.27 ± 11%      +0.5        0.75 ±  9%  perf-profile.children.cycles-pp.__fdget
      0.41 ±  5%      +0.5        0.89 ±  8%  perf-profile.children.cycles-pp.__sk_mem_reduce_allocated
      0.00            +0.5        0.50 ± 10%  perf-profile.children.cycles-pp.kfree
      0.22 ±  7%      +0.5        0.76 ±  8%  perf-profile.children.cycles-pp.security_socket_recvmsg
      0.29 ±  6%      +0.5        0.84 ±  7%  perf-profile.children.cycles-pp._copy_from_iter
      0.77 ±  8%      +0.5        1.31 ±  9%  perf-profile.children.cycles-pp.update_curr
      0.35 ±  6%      +0.6        0.91 ±  7%  perf-profile.children.cycles-pp.aa_sk_perm
      0.24 ±  8%      +0.6        0.80 ±  8%  perf-profile.children.cycles-pp.sock_recvmsg
      0.43 ±  7%      +0.6        0.99 ±  7%  perf-profile.children.cycles-pp.__entry_text_start
      0.33 ± 10%      +0.6        0.91 ±  9%  perf-profile.children.cycles-pp.sockfd_lookup_light
      1.23 ±  8%      +0.6        1.82 ±  7%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.54 ±  7%      +0.6        1.18 ±  8%  perf-profile.children.cycles-pp.__kfree_skb
      1.28 ±  7%      +0.8        2.04 ±  8%  perf-profile.children.cycles-pp.enqueue_task_fair
      1.32 ±  7%      +0.8        2.09 ±  8%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.54 ±  6%      +0.8        1.36 ±  7%  perf-profile.children.cycles-pp.skb_do_copy_data_nocache
      0.00            +1.0        1.01 ±  8%  perf-profile.children.cycles-pp.restore_fpregs_from_fpstate
      0.63 ±  6%      +1.1        1.70 ±  8%  perf-profile.children.cycles-pp.loopback_xmit
      0.82 ±  7%      +1.1        1.91 ±  8%  perf-profile.children.cycles-pp.__alloc_skb
      5.25 ±  7%      +1.1        6.39 ±  8%  perf-profile.children.cycles-pp.wait_woken
      0.68 ±  5%      +1.1        1.82 ±  8%  perf-profile.children.cycles-pp.dev_hard_start_xmit
      0.15 ±  8%      +1.2        1.33 ±  8%  perf-profile.children.cycles-pp.switch_fpu_return
      1.65 ±  6%      +1.2        2.87 ±  8%  perf-profile.children.cycles-pp.tcp_clean_rtx_queue
      9.07 ±  7%      +1.4       10.45 ±  8%  perf-profile.children.cycles-pp.tcp_rcv_established
     10.89 ±  6%      +1.4       12.30 ±  8%  perf-profile.children.cycles-pp.tcp_v4_rcv
      1.40 ±  6%      +1.4        2.82 ±  8%  perf-profile.children.cycles-pp.__dev_queue_xmit
      9.14 ±  6%      +1.4       10.56 ±  8%  perf-profile.children.cycles-pp.tcp_v4_do_rcv
      1.20 ±  7%      +1.5        2.68 ±  8%  perf-profile.children.cycles-pp.tcp_stream_alloc_skb
     13.56 ±  7%      +1.6       15.16 ±  8%  perf-profile.children.cycles-pp.__sys_recvfrom
     11.05 ±  6%      +1.7       12.71 ±  8%  perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
     13.66 ±  7%      +1.7       15.36 ±  8%  perf-profile.children.cycles-pp.__x64_sys_recvfrom
     11.08 ±  6%      +1.7       12.80 ±  8%  perf-profile.children.cycles-pp.ip_local_deliver_finish
      2.51 ±  6%      +1.8        4.28 ±  8%  perf-profile.children.cycles-pp.tcp_ack
      4.22 ±  7%      +1.8        6.02 ±  7%  perf-profile.children.cycles-pp.schedule_timeout
      0.10 ±  9%      +2.1        2.19 ±  8%  perf-profile.children.cycles-pp.switch_mm_irqs_off
     11.50 ±  6%      +2.2       13.70 ±  7%  perf-profile.children.cycles-pp.__netif_receive_skb_one_core
     11.89 ±  6%      +2.5       14.38 ±  7%  perf-profile.children.cycles-pp.process_backlog
     11.90 ±  6%      +2.6       14.46 ±  7%  perf-profile.children.cycles-pp.__napi_poll
      6.04 ±  7%      +2.6        8.61 ±  7%  perf-profile.children.cycles-pp.__schedule
      0.00            +3.2        3.24 ±  7%  perf-profile.children.cycles-pp.exit_to_user_mode_loop
     16.72 ±  6%      +3.3       20.05 ±  7%  perf-profile.children.cycles-pp.ip_finish_output2
     15.81 ±  7%      +3.7       19.46 ±  8%  perf-profile.children.cycles-pp.recv
     17.54 ±  6%      +4.2       21.74 ±  7%  perf-profile.children.cycles-pp.__ip_queue_xmit
      0.20 ±  8%      +4.7        4.87 ±  7%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      4.10 ±  7%      +4.8        8.90 ±  7%  perf-profile.children.cycles-pp.schedule
      0.32 ±  6%      +4.8        5.12 ±  7%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
     18.90 ±  6%      +5.5       24.36 ±  7%  perf-profile.children.cycles-pp.__tcp_transmit_skb
     20.44 ±  6%      +6.5       26.93 ±  7%  perf-profile.children.cycles-pp.tcp_write_xmit
     20.51 ±  6%      +6.6       27.14 ±  7%  perf-profile.children.cycles-pp.__tcp_push_pending_frames
     21.50 ±  6%      +8.3       29.79 ±  6%  perf-profile.children.cycles-pp.send_tcp_rr
     21.57 ±  6%      +8.5       30.03 ±  7%  perf-profile.children.cycles-pp.send_omni_inner
     21.58 ±  6%      +8.5       30.06 ±  7%  perf-profile.children.cycles-pp.recv_omni
     21.59 ±  6%      +8.5       30.08 ±  7%  perf-profile.children.cycles-pp.accept_connections
     21.59 ±  6%      +8.5       30.08 ±  7%  perf-profile.children.cycles-pp.accept_connection
     21.59 ±  6%      +8.5       30.08 ±  7%  perf-profile.children.cycles-pp.spawn_child
     21.59 ±  6%      +8.5       30.08 ±  7%  perf-profile.children.cycles-pp.process_requests
     24.97 ± 19%      +9.0       34.00 ± 15%  perf-profile.children.cycles-pp.intel_idle
     24.26 ±  6%      +9.3       33.55 ±  7%  perf-profile.children.cycles-pp.tcp_sendmsg_locked
     24.84 ±  6%      +9.7       34.54 ±  7%  perf-profile.children.cycles-pp.tcp_sendmsg
     25.26 ±  6%     +10.2       35.49 ±  7%  perf-profile.children.cycles-pp.sock_sendmsg
     25.68 ±  6%     +10.9       36.56 ±  7%  perf-profile.children.cycles-pp.__sys_sendto
     25.80 ±  6%     +11.0       36.82 ±  7%  perf-profile.children.cycles-pp.__x64_sys_sendto
     26.86 ±  6%     +17.5       44.40 ±  7%  perf-profile.children.cycles-pp.__send
     40.08 ±  6%     +17.8       57.92 ±  7%  perf-profile.children.cycles-pp.do_syscall_64
     40.47 ±  6%     +18.4       58.88 ±  7%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     10.22 ±  7%     -10.2        0.00        perf-profile.self.cycles-pp.poll_idle
      3.03 ±  7%      -2.3        0.70 ± 10%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      1.03 ±  7%      -0.7        0.37 ±  9%  perf-profile.self.cycles-pp.wait_woken
      0.66 ±  7%      -0.5        0.19 ± 10%  perf-profile.self.cycles-pp.sock_rfree
      0.91 ±  7%      -0.4        0.49 ± 12%  perf-profile.self.cycles-pp.__inet_lookup_established
      0.64 ±  8%      -0.4        0.26 ±  8%  perf-profile.self.cycles-pp.tcp_rcv_established
      0.92 ±  6%      -0.4        0.55 ±  6%  perf-profile.self.cycles-pp.skb_release_data
      0.40 ±  9%      -0.3        0.06 ±  6%  perf-profile.self.cycles-pp.skb_attempt_defer_free
      0.40 ±  8%      -0.3        0.08 ±  9%  perf-profile.self.cycles-pp.tcp_rcv_space_adjust
      0.82 ±  7%      -0.3        0.50 ±  8%  perf-profile.self.cycles-pp._raw_spin_lock
      0.54 ±  5%      -0.3        0.24 ±  9%  perf-profile.self.cycles-pp.check_heap_object
      0.43 ±  6%      -0.3        0.16 ± 14%  perf-profile.self.cycles-pp.prepare_task_switch
      0.57 ±  6%      -0.3        0.32 ± 11%  perf-profile.self.cycles-pp.__skb_datagram_iter
      0.56 ±  9%      -0.2        0.32 ± 10%  perf-profile.self.cycles-pp.tcp_check_space
      0.39 ±  7%      -0.2        0.19 ± 11%  perf-profile.self.cycles-pp.enqueue_entity
      0.35 ±  8%      -0.2        0.16 ±  8%  perf-profile.self.cycles-pp.kfree_skbmem
      0.57 ±  7%      -0.2        0.40 ±  8%  perf-profile.self.cycles-pp.copy_user_short_string
      0.26 ±  9%      -0.2        0.09 ±  7%  perf-profile.self.cycles-pp.__wake_up_common
      0.19 ±  7%      -0.2        0.03 ± 70%  perf-profile.self.cycles-pp.skb_release_head_state
      0.22 ±  8%      -0.2        0.07 ± 20%  perf-profile.self.cycles-pp.perf_tp_event
      0.21 ±  4%      -0.2        0.06 ± 19%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.67 ±  7%      -0.2        0.52 ±  8%  perf-profile.self.cycles-pp.tcp_recvmsg_locked
      0.25 ±  9%      -0.1        0.14 ± 12%  perf-profile.self.cycles-pp.dequeue_entity
      0.14 ±  9%      -0.1        0.05 ± 46%  perf-profile.self.cycles-pp.select_task_rq
      0.34 ±  3%      -0.1        0.27 ± 14%  perf-profile.self.cycles-pp.___perf_sw_event
      0.16 ± 12%      -0.1        0.10 ± 16%  perf-profile.self.cycles-pp.perf_trace_sched_wakeup_template
      0.20 ±  7%      -0.0        0.16 ±  7%  perf-profile.self.cycles-pp.finish_task_switch
      0.09 ± 11%      -0.0        0.06 ±  9%  perf-profile.self.cycles-pp.check_preempt_curr
      0.06 ±  7%      -0.0        0.03 ± 70%  perf-profile.self.cycles-pp.tracing_gen_ctx_irq_test
      0.06 ± 11%      +0.0        0.09 ±  9%  perf-profile.self.cycles-pp.tcp_event_data_recv
      0.05 ± 45%      +0.0        0.08 ±  7%  perf-profile.self.cycles-pp.put_prev_entity
      0.22 ±  7%      +0.0        0.25 ±  7%  perf-profile.self.cycles-pp.__wrgsbase_inactive
      0.10 ± 10%      +0.0        0.14 ± 15%  perf-profile.self.cycles-pp.tcp_rearm_rto
      0.22 ±  6%      +0.0        0.26 ±  8%  perf-profile.self.cycles-pp.dequeue_task_fair
      0.13 ±  6%      +0.0        0.17 ± 11%  perf-profile.self.cycles-pp.update_rq_clock
      0.04 ± 47%      +0.0        0.09 ± 18%  perf-profile.self.cycles-pp.tcp_v4_do_rcv
      0.06 ± 11%      +0.0        0.10 ± 13%  perf-profile.self.cycles-pp.ip_skb_dst_mtu
      0.02 ±142%      +0.0        0.06 ± 21%  perf-profile.self.cycles-pp.timekeeping_max_deferment
      0.02 ±141%      +0.0        0.06 ± 11%  perf-profile.self.cycles-pp.place_entity
      0.01 ±223%      +0.0        0.06 ± 13%  perf-profile.self.cycles-pp.skb_clone
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.minmax_running_min
      0.06 ± 13%      +0.1        0.11 ±  9%  perf-profile.self.cycles-pp.rcu_all_qs
      0.00            +0.1        0.05 ±  7%  perf-profile.self.cycles-pp.__netif_receive_skb_one_core
      0.00            +0.1        0.05 ±  7%  perf-profile.self.cycles-pp.clear_buddies
      0.00            +0.1        0.05 ±  7%  perf-profile.self.cycles-pp.tcp_small_queue_check
      0.18 ±  7%      +0.1        0.23 ± 10%  perf-profile.self.cycles-pp.do_softirq
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.ip_local_deliver
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.__kmalloc_node_track_caller
      0.00            +0.1        0.06 ± 13%  perf-profile.self.cycles-pp.skb_copy_datagram_iter
      0.00            +0.1        0.06 ± 13%  perf-profile.self.cycles-pp.cubictcp_cong_avoid
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.sk_reset_timer
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.switch_ldt
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.rb_insert_color
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.remove_wait_queue
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.__ip_finish_output
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.nf_hook_slow
      0.02 ±141%      +0.1        0.08 ± 10%  perf-profile.self.cycles-pp.tcp_send_mss
      0.00            +0.1        0.06 ± 15%  perf-profile.self.cycles-pp.eth_type_trans
      0.13 ±  8%      +0.1        0.19 ± 10%  perf-profile.self.cycles-pp.schedule_timeout
      0.00            +0.1        0.06 ± 13%  perf-profile.self.cycles-pp.memcg_slab_post_alloc_hook
      0.00            +0.1        0.06 ± 16%  perf-profile.self.cycles-pp.kmalloc_reserve
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.sk_filter_trim_cap
      0.00            +0.1        0.06 ± 16%  perf-profile.self.cycles-pp.kmalloc_slab
      0.06 ± 13%      +0.1        0.12 ±  8%  perf-profile.self.cycles-pp.tcp_recvmsg
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.tcp_inbound_md5_hash
      0.00            +0.1        0.06 ± 17%  perf-profile.self.cycles-pp.apparmor_socket_recvmsg
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.__wake_up_common_lock
      0.07 ± 12%      +0.1        0.14 ± 14%  perf-profile.self.cycles-pp.update_min_vruntime
      0.06 ± 11%      +0.1        0.12 ± 10%  perf-profile.self.cycles-pp.tcp_rate_skb_sent
      0.04 ± 45%      +0.1        0.11 ±  9%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode_prepare
      0.08 ±  6%      +0.1        0.14 ±  6%  perf-profile.self.cycles-pp.ip_output
      0.04 ± 44%      +0.1        0.11 ±  5%  perf-profile.self.cycles-pp.tcp_established_options
      0.01 ±223%      +0.1        0.08 ±  9%  perf-profile.self.cycles-pp.lock_sock_nested
      0.10 ± 16%      +0.1        0.17 ± 11%  perf-profile.self.cycles-pp.perf_trace_sched_switch
      0.08 ±  8%      +0.1        0.16 ± 10%  perf-profile.self.cycles-pp.try_charge_memcg
      0.00            +0.1        0.07 ± 14%  perf-profile.self.cycles-pp.tcp_stream_alloc_skb
      0.00            +0.1        0.07 ± 20%  perf-profile.self.cycles-pp.cubictcp_acked
      0.59 ±  6%      +0.1        0.66 ±  6%  perf-profile.self.cycles-pp.__switch_to
      0.07 ±  7%      +0.1        0.14 ± 11%  perf-profile.self.cycles-pp.irqtime_account_irq
      0.07 ±  8%      +0.1        0.14 ± 11%  perf-profile.self.cycles-pp.__copy_skb_header
      0.06 ±  9%      +0.1        0.13 ±  5%  perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
      0.00            +0.1        0.07 ± 10%  perf-profile.self.cycles-pp.skb_do_copy_data_nocache
      0.04 ± 44%      +0.1        0.12 ± 10%  perf-profile.self.cycles-pp.dev_hard_start_xmit
      0.11 ±  6%      +0.1        0.19 ± 11%  perf-profile.self.cycles-pp._copy_to_iter
      0.04 ± 71%      +0.1        0.11 ±  8%  perf-profile.self.cycles-pp.mem_cgroup_uncharge_skmem
      0.03 ± 70%      +0.1        0.11 ±  9%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode
      0.06 ±  8%      +0.1        0.14 ±  7%  perf-profile.self.cycles-pp.raw_v4_input
      0.03 ± 70%      +0.1        0.11 ±  8%  perf-profile.self.cycles-pp.sk_free
      0.00            +0.1        0.08 ±  8%  perf-profile.self.cycles-pp.neigh_hh_output
      0.26 ±  7%      +0.1        0.34 ±  9%  perf-profile.self.cycles-pp.recv_data
      0.12 ± 11%      +0.1        0.20 ±  8%  perf-profile.self.cycles-pp.ip_send_check
      0.00            +0.1        0.08 ±  7%  perf-profile.self.cycles-pp.syscall_enter_from_user_mode
      0.06 ± 11%      +0.1        0.15 ± 18%  perf-profile.self.cycles-pp.validate_xmit_skb
      0.00            +0.1        0.08 ±  5%  perf-profile.self.cycles-pp.__napi_poll
      0.00            +0.1        0.08 ±  5%  perf-profile.self.cycles-pp.tcp_v4_fill_cb
      0.00            +0.1        0.08 ± 11%  perf-profile.self.cycles-pp.apparmor_socket_sendmsg
      0.04 ± 45%      +0.1        0.13 ±  8%  perf-profile.self.cycles-pp.inet_ehashfn
      0.04 ± 45%      +0.1        0.13 ±  8%  perf-profile.self.cycles-pp.tcp_rate_gen
      0.02 ±141%      +0.1        0.10 ±  8%  perf-profile.self.cycles-pp.tcp_mstamp_refresh
      0.00            +0.1        0.08 ± 14%  perf-profile.self.cycles-pp.netif_skb_features
      0.00            +0.1        0.09 ± 15%  perf-profile.self.cycles-pp.__might_fault
      0.00            +0.1        0.09 ±  7%  perf-profile.self.cycles-pp.tcp_newly_delivered
      0.10 ±  6%      +0.1        0.19 ± 14%  perf-profile.self.cycles-pp.sk_forced_mem_schedule
      0.09 ±  7%      +0.1        0.18 ±  9%  perf-profile.self.cycles-pp.__ksize
      0.00            +0.1        0.09 ±  6%  perf-profile.self.cycles-pp.ip_local_deliver_finish
      0.00            +0.1        0.09 ±  6%  perf-profile.self.cycles-pp.check_stack_object
      0.08 ± 14%      +0.1        0.18 ± 16%  perf-profile.self.cycles-pp.tcp_rtt_estimator
      0.06 ± 11%      +0.1        0.16 ±  5%  perf-profile.self.cycles-pp.sock_put
      0.10 ± 10%      +0.1        0.20 ± 10%  perf-profile.self.cycles-pp.__cond_resched
      0.22 ±  8%      +0.1        0.32 ±  9%  perf-profile.self.cycles-pp.schedule
      0.06 ±  9%      +0.1        0.16 ±  9%  perf-profile.self.cycles-pp.sockfd_lookup_light
      0.00            +0.1        0.10 ± 12%  perf-profile.self.cycles-pp.resched_curr
      0.11 ±  6%      +0.1        0.21 ± 20%  perf-profile.self.cycles-pp.__mod_memcg_state
      0.13 ±  9%      +0.1        0.23 ± 19%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.10 ± 10%      +0.1        0.21 ±  7%  perf-profile.self.cycles-pp.__tcp_select_window
      0.07 ±  5%      +0.1        0.17 ±  8%  perf-profile.self.cycles-pp.enqueue_to_backlog
      0.09 ±  5%      +0.1        0.20 ± 12%  perf-profile.self.cycles-pp.__x64_sys_recvfrom
      0.06 ± 14%      +0.1        0.17 ± 25%  perf-profile.self.cycles-pp.cgroup_rstat_updated
      0.00            +0.1        0.10 ±  4%  perf-profile.self.cycles-pp.tcp_ack_update_rtt
      0.00            +0.1        0.10 ±  4%  perf-profile.self.cycles-pp.tcp_push
      0.01 ±223%      +0.1        0.11 ±  6%  perf-profile.self.cycles-pp.__usecs_to_jiffies
      0.35 ±  7%      +0.1        0.45 ±  7%  perf-profile.self.cycles-pp.update_load_avg
      0.09 ±  7%      +0.1        0.20 ±  8%  perf-profile.self.cycles-pp.sock_sendmsg
      0.00            +0.1        0.11 ±  9%  perf-profile.self.cycles-pp.ip_rcv
      0.12 ±  6%      +0.1        0.23 ±  9%  perf-profile.self.cycles-pp.security_socket_sendmsg
      0.16 ± 10%      +0.1        0.27 ±  8%  perf-profile.self.cycles-pp.sk_wait_data
      0.00            +0.1        0.11 ±  8%  perf-profile.self.cycles-pp.__ip_local_out
      0.09 ± 14%      +0.1        0.20 ±  8%  perf-profile.self.cycles-pp.mem_cgroup_charge_skmem
      0.18 ± 10%      +0.1        0.29 ± 14%  perf-profile.self.cycles-pp.__calc_delta
      0.09 ±  6%      +0.1        0.21 ±  8%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.24 ±  6%      +0.1        0.36 ± 11%  perf-profile.self.cycles-pp.__local_bh_enable_ip
      0.00            +0.1        0.13 ±  8%  perf-profile.self.cycles-pp.cubictcp_cwnd_event
      0.14 ±  7%      +0.1        0.27 ±  5%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.04 ± 71%      +0.1        0.17 ± 11%  perf-profile.self.cycles-pp.skb_page_frag_refill
      0.15 ±  7%      +0.1        0.28 ±  8%  perf-profile.self.cycles-pp.switch_fpu_return
      0.11 ±  6%      +0.1        0.24 ±  9%  perf-profile.self.cycles-pp.tcp_sendmsg
      0.07 ± 14%      +0.1        0.20 ±  8%  perf-profile.self.cycles-pp.tcp_update_skb_after_send
      0.15 ±  4%      +0.1        0.29 ±  9%  perf-profile.self.cycles-pp.__sk_mem_reduce_allocated
      0.29 ±  6%      +0.1        0.42 ±  8%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.29 ±  9%      +0.1        0.42 ±  8%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.10 ±  7%      +0.1        0.24 ±  9%  perf-profile.self.cycles-pp.tcp_tso_segs
      0.11 ±  6%      +0.1        0.25 ±  7%  perf-profile.self.cycles-pp.__netif_receive_skb_core
      0.07 ± 12%      +0.1        0.21 ±  5%  perf-profile.self.cycles-pp.__tcp_push_pending_frames
      0.34 ±  9%      +0.1        0.48 ±  8%  perf-profile.self.cycles-pp.update_curr
      0.18 ±  9%      +0.1        0.32 ±  7%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.08 ± 19%      +0.1        0.22 ± 18%  perf-profile.self.cycles-pp.cpuacct_charge
      0.08 ±  8%      +0.1        0.23 ±  9%  perf-profile.self.cycles-pp.ip_rcv_core
      0.01 ±223%      +0.1        0.16 ± 12%  perf-profile.self.cycles-pp.inet_recvmsg
      0.29 ±  5%      +0.2        0.44 ±  6%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.10 ±  5%      +0.2        0.25 ± 10%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.10 ±  6%      +0.2        0.26 ±  8%  perf-profile.self.cycles-pp.process_backlog
      0.12 ±  8%      +0.2        0.27 ±  7%  perf-profile.self.cycles-pp.__x64_sys_sendto
      0.00            +0.2        0.16 ± 14%  perf-profile.self.cycles-pp.check_preempt_wakeup
      0.28 ±  8%      +0.2        0.44 ±  6%  perf-profile.self.cycles-pp.__skb_clone
      0.12 ±  8%      +0.2        0.28 ±  7%  perf-profile.self.cycles-pp.tcp_schedule_loss_probe
      0.34 ±  4%      +0.2        0.51 ±  8%  perf-profile.self.cycles-pp.update_cfs_group
      0.16 ±  6%      +0.2        0.33 ±  7%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.14 ±  7%      +0.2        0.31 ±  8%  perf-profile.self.cycles-pp.tcp_update_pacing_rate
      0.11 ±  6%      +0.2        0.28 ±  8%  perf-profile.self.cycles-pp.refill_stock
      0.09 ± 13%      +0.2        0.26 ±  8%  perf-profile.self.cycles-pp.ip_protocol_deliver_rcu
      0.00            +0.2        0.17 ± 11%  perf-profile.self.cycles-pp.security_socket_recvmsg
      0.17 ±  9%      +0.2        0.34 ±  8%  perf-profile.self.cycles-pp.__alloc_skb
      0.08 ± 25%      +0.2        0.26 ± 17%  perf-profile.self.cycles-pp.update_sg_lb_stats
      0.27 ±  5%      +0.2        0.46 ±  7%  perf-profile.self.cycles-pp.send_omni_inner
      0.06 ±  7%      +0.2        0.25 ±  8%  perf-profile.self.cycles-pp.__list_add_valid
      0.22 ±  8%      +0.2        0.40 ± 11%  perf-profile.self.cycles-pp.recv
      0.00            +0.2        0.19 ±  8%  perf-profile.self.cycles-pp.exit_to_user_mode_loop
      0.20 ±  8%      +0.2        0.39 ±  8%  perf-profile.self.cycles-pp.send_data
      0.38 ±  8%      +0.2        0.57 ± 23%  perf-profile.self.cycles-pp.ktime_get
      0.45 ±  7%      +0.2        0.65 ±  7%  perf-profile.self.cycles-pp.read_tsc
      0.16 ±  4%      +0.2        0.35 ±  9%  perf-profile.self.cycles-pp.tcp_wfree
      0.00            +0.2        0.20 ±  9%  perf-profile.self.cycles-pp.set_next_buddy
      0.16 ±  9%      +0.2        0.36 ±  6%  perf-profile.self.cycles-pp.__might_sleep
      0.06 ±  7%      +0.2        0.27 ±  8%  perf-profile.self.cycles-pp.__kmem_cache_free
      0.32 ±  5%      +0.2        0.53 ±  9%  perf-profile.self.cycles-pp.__sys_recvfrom
      0.25 ± 14%      +0.2        0.46 ±  9%  perf-profile.self.cycles-pp.recv_omni
      0.62 ±  7%      +0.2        0.84 ±  8%  perf-profile.self.cycles-pp.__schedule
      0.16 ±  7%      +0.2        0.38 ±  9%  perf-profile.self.cycles-pp.__kmem_cache_alloc_node
      0.19 ±  8%      +0.2        0.42 ±  7%  perf-profile.self.cycles-pp.__softirqentry_text_start
      0.67 ±  6%      +0.2        0.89 ±  8%  perf-profile.self.cycles-pp.__switch_to_asm
      0.09 ±  7%      +0.2        0.32 ±  8%  perf-profile.self.cycles-pp._copy_from_iter
      0.18 ±  6%      +0.2        0.42 ±  5%  perf-profile.self.cycles-pp.__send
      0.10 ± 11%      +0.2        0.34 ± 10%  perf-profile.self.cycles-pp.__check_object_size
      0.07 ± 10%      +0.2        0.31 ±  8%  perf-profile.self.cycles-pp.kmem_cache_free
      0.12 ±  7%      +0.3        0.38 ±  8%  perf-profile.self.cycles-pp.kmem_cache_alloc_node
      0.26 ±  6%      +0.3        0.52 ±  7%  perf-profile.self.cycles-pp.do_syscall_64
      0.21 ±  7%      +0.3        0.48 ±  7%  perf-profile.self.cycles-pp.__entry_text_start
      0.22 ±  7%      +0.3        0.50 ±  8%  perf-profile.self.cycles-pp.__sys_sendto
      0.22 ±  8%      +0.3        0.50 ±  6%  perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.53 ±  7%      +0.3        0.84 ±  7%  perf-profile.self.cycles-pp._raw_spin_lock_bh
      0.25 ±  7%      +0.3        0.56 ±  8%  perf-profile.self.cycles-pp.os_xsave
      0.13 ±  4%      +0.3        0.46 ±  7%  perf-profile.self.cycles-pp.__virt_addr_valid
      0.16 ±  7%      +0.3        0.49 ±  7%  perf-profile.self.cycles-pp.loopback_xmit
      0.05 ±  7%      +0.3        0.40 ±  9%  perf-profile.self.cycles-pp.import_single_range
      0.32 ±  6%      +0.4        0.67 ±  9%  perf-profile.self.cycles-pp.ip_finish_output2
      0.01 ±223%      +0.4        0.36 ±  9%  perf-profile.self.cycles-pp.exit_to_user_mode_prepare
      0.21 ±  8%      +0.4        0.59 ±  8%  perf-profile.self.cycles-pp.aa_sk_perm
      0.27 ±  6%      +0.4        0.69 ±  6%  perf-profile.self.cycles-pp.__ip_queue_xmit
      0.29 ±  6%      +0.4        0.73 ±  8%  perf-profile.self.cycles-pp.tcp_write_xmit
      0.27 ± 10%      +0.5        0.74 ± 10%  perf-profile.self.cycles-pp.__fdget
      0.00            +0.5        0.49 ± 10%  perf-profile.self.cycles-pp.kfree
      0.66 ±  6%      +0.5        1.16 ±  8%  perf-profile.self.cycles-pp.__tcp_transmit_skb
      0.38 ±  6%      +0.6        0.96 ±  8%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.00            +1.0        1.01 ±  8%  perf-profile.self.cycles-pp.restore_fpregs_from_fpstate
      0.10 ±  9%      +2.1        2.17 ±  7%  perf-profile.self.cycles-pp.switch_mm_irqs_off


***************************************************************************************************
lkp-cfl-e1: 16 threads 1 sockets Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz (Coffee Lake) with 32G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase:
  gcc-11/performance/x86_64-rhel-8.3/30%/debian-11.1-x86_64-20220510.cgz/300s/lkp-cfl-e1/context1/unixbench

commit: 
  c36eae5a86 ("sched/psi: Fix possible missing or delayed pending event")
  b6aabb01e3 ("sched: consider WF_SYNC to find idle siblings")

c36eae5a86d15a69 b6aabb01e3004b0285d953761e1 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      3547           +36.2%       4834        unixbench.score
      1202 ±  4%  +3.2e+05%    3838646 ± 17%  unixbench.time.involuntary_context_switches
    146.17            +4.7%     153.00        unixbench.time.percent_of_cpu_this_job_got
    533.06            +1.7%     542.03        unixbench.time.system_time
     40.02           +40.3%      56.14        unixbench.time.user_time
 4.156e+08           +34.9%  5.604e+08        unixbench.time.voluntary_context_switches
 5.537e+08           +36.3%  7.548e+08        unixbench.workload
      1.53            +0.5        2.01        mpstat.cpu.all.usr%
 4.434e+09           +13.2%  5.022e+09        cpuidle..time
 8.352e+08           -98.7%   11131648 ± 12%  cpuidle..usage
      3.00           +33.3%       4.00        vmstat.procs.r
   4212747           -32.1%    2861762        vmstat.system.cs
     40960           -29.9%      28703        vmstat.system.in
    161422           -32.1%     109641        meminfo.Active
    161345           -32.1%     109569        meminfo.Active(anon)
    530075           -10.9%     472307        meminfo.Committed_AS
     80181           -27.2%      58384        meminfo.Mapped
    162280           -32.2%     110057        meminfo.Shmem
     40427           -31.8%      27576        proc-vmstat.nr_active_anon
    978441            -1.3%     965308        proc-vmstat.nr_file_pages
     20137           -26.7%      14769        proc-vmstat.nr_mapped
     40661           -31.9%      27701        proc-vmstat.nr_shmem
     13329            -0.9%      13206        proc-vmstat.nr_slab_unreclaimable
     40427           -31.8%      27576        proc-vmstat.nr_zone_active_anon
   1751058           -14.2%    1502883        proc-vmstat.numa_hit
   1751425           -14.1%    1504002        proc-vmstat.numa_local
     92726           -17.3%      76685        proc-vmstat.pgactivate
   1819543           -13.7%    1570329        proc-vmstat.pgalloc_normal
    684646            -2.4%     668006        proc-vmstat.pgfault
   1807696           -13.7%    1559418        proc-vmstat.pgfree
     40680            -1.8%      39968        proc-vmstat.pgreuse
      1740           -47.5%     912.83        turbostat.Avg_MHz
     38.54           -18.6       19.90        turbostat.Busy%
      4515            +1.6%       4586        turbostat.Bzy_MHz
   3885351 ±  9%     -82.8%     667034 ± 73%  turbostat.C1
     26.41 ±  6%     +50.7%      39.80 ± 19%  turbostat.CPU%c6
     74.05           -31.1%      50.99 ±  2%  turbostat.CorWatt
     60.17           -18.3%      49.17 ±  2%  turbostat.CoreTmp
      0.22 ±  2%     +83.6%       0.41        turbostat.IPC
  16161117           -30.0%   11319624        turbostat.IRQ
 8.241e+08          -100.0%       3881 ± 26%  turbostat.POLL
      9.50            -9.5        0.00        turbostat.POLL%
     58.67 ±  2%     -17.0%      48.67 ±  2%  turbostat.PkgTmp
     74.52           -30.9%      51.46 ±  2%  turbostat.PkgWatt
    489.94           -35.7%     315.24        turbostat.Totl%C0
      4851 ± 56%    -100.0%       0.00        sched_debug.cfs_rq:/.MIN_vruntime.avg
     56902 ± 44%    -100.0%       0.00        sched_debug.cfs_rq:/.MIN_vruntime.max
     15406 ± 47%    -100.0%       0.00        sched_debug.cfs_rq:/.MIN_vruntime.stddev
      0.44 ±  9%     -14.4%       0.38 ±  2%  sched_debug.cfs_rq:/.h_nr_running.avg
    127353           +10.5%     140708 ±  3%  sched_debug.cfs_rq:/.load.stddev
    120.76 ±  8%     +25.4%     151.41 ± 17%  sched_debug.cfs_rq:/.load_avg.avg
      4851 ± 56%    -100.0%       0.00        sched_debug.cfs_rq:/.max_vruntime.avg
     56902 ± 44%    -100.0%       0.00        sched_debug.cfs_rq:/.max_vruntime.max
     15406 ± 47%    -100.0%       0.00        sched_debug.cfs_rq:/.max_vruntime.stddev
    197189            -9.1%     179171 ±  2%  sched_debug.cfs_rq:/.min_vruntime.max
    116754 ±  6%     +12.9%     131851 ±  3%  sched_debug.cfs_rq:/.min_vruntime.min
     21217 ± 10%     -39.9%      12759 ± 18%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.44 ±  7%     -18.5%       0.36 ±  3%  sched_debug.cfs_rq:/.nr_running.avg
    366.23           +21.1%     443.60 ±  3%  sched_debug.cfs_rq:/.runnable_avg.avg
    714.74 ±  4%     +81.8%       1299 ±  3%  sched_debug.cfs_rq:/.runnable_avg.max
    246.75 ±  5%    +101.3%     496.81 ±  3%  sched_debug.cfs_rq:/.runnable_avg.stddev
     51496 ± 44%     -66.9%      17048 ± 49%  sched_debug.cfs_rq:/.spread0.max
     21217 ± 10%     -39.9%      12760 ± 18%  sched_debug.cfs_rq:/.spread0.stddev
    708.50 ±  4%     +49.8%       1061 ±  4%  sched_debug.cfs_rq:/.util_avg.max
    245.43 ±  5%     +63.7%     401.71 ±  3%  sched_debug.cfs_rq:/.util_avg.stddev
    107.96 ± 11%     +42.0%     153.34 ±  4%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    499.12 ±  2%     +65.3%     825.00 ±  4%  sched_debug.cfs_rq:/.util_est_enqueued.max
    181.05 ±  4%     +58.9%     287.67 ±  4%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    407511 ±  4%    +110.5%     857930        sched_debug.cpu.avg_idle.avg
    104621          +462.1%     588066 ± 11%  sched_debug.cpu.avg_idle.min
    350206 ±  2%     -68.5%     110282 ± 21%  sched_debug.cpu.avg_idle.stddev
      0.35 ± 18%     +89.2%       0.67 ±  3%  sched_debug.cpu.clock.stddev
      0.00 ± 37%     +68.1%       0.00 ±  9%  sched_debug.cpu.next_balance.stddev
      1.07 ±  6%     +55.6%       1.67 ±  9%  sched_debug.cpu.nr_running.max
      0.46 ±  3%     +24.0%       0.57 ±  4%  sched_debug.cpu.nr_running.stddev
  47178388           -32.4%   31875847        sched_debug.cpu.nr_switches.avg
  56953632 ±  2%     -36.6%   36093819 ±  2%  sched_debug.cpu.nr_switches.max
  31935712 ±  9%     -13.3%   27680720 ±  3%  sched_debug.cpu.nr_switches.min
   6579393 ± 14%     -64.6%    2330670 ± 21%  sched_debug.cpu.nr_switches.stddev
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.avg
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.max
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.min
     65.64           -30.8%      45.42        perf-stat.i.MPKI
 4.603e+09            -7.6%  4.252e+09        perf-stat.i.branch-instructions
      2.81            +0.1        2.89        perf-stat.i.branch-miss-rate%
  49172917            -1.4%   48506007        perf-stat.i.branch-misses
      0.79            +2.6        3.39 ±  7%  perf-stat.i.cache-miss-rate%
   1887466            -4.2%    1808731 ±  2%  perf-stat.i.cache-misses
 5.404e+08           -94.1%   32037290 ±  3%  perf-stat.i.cache-references
   4247507           -32.1%    2884302        perf-stat.i.context-switches
      1.99           -24.4%       1.50        perf-stat.i.cpi
 2.784e+10           -47.7%  1.457e+10        perf-stat.i.cpu-cycles
     52.17 ±  2%    +634.1%     383.01 ±  4%  perf-stat.i.cpu-migrations
     56513 ±  2%     -59.7%      22750 ±  9%  perf-stat.i.cycles-between-cache-misses
    410439 ±  2%     +23.7%     507559 ±  7%  perf-stat.i.dTLB-load-misses
 6.418e+09            -3.1%   6.22e+09        perf-stat.i.dTLB-loads
 3.826e+09            -4.0%  3.674e+09        perf-stat.i.dTLB-stores
     95.43           -38.9       56.58        perf-stat.i.iTLB-load-miss-rate%
  16498563            -5.9%   15523524        perf-stat.i.iTLB-load-misses
    239550 ±  3%   +5782.0%   14090457        perf-stat.i.iTLB-loads
 2.197e+10            -4.8%  2.092e+10        perf-stat.i.instructions
      0.70           +78.1%       1.25        perf-stat.i.ipc
      1.74           -47.7%       0.91        perf-stat.i.metric.GHz
    404.88 ±  2%      -4.8%     385.49 ±  2%  perf-stat.i.metric.K/sec
    961.54            -7.8%     886.77        perf-stat.i.metric.M/sec
      1575            -2.7%       1533        perf-stat.i.minor-faults
      0.40 ±  4%      +0.1        0.47 ±  7%  perf-stat.i.node-load-miss-rate%
    261859           -14.5%     223787        perf-stat.i.node-stores
      1575            -2.7%       1533        perf-stat.i.page-faults
     24.60           -93.8%       1.53 ±  3%  perf-stat.overall.MPKI
      1.07            +0.1        1.14        perf-stat.overall.branch-miss-rate%
      0.35            +5.3        5.67 ±  5%  perf-stat.overall.cache-miss-rate%
      1.27           -45.0%       0.70        perf-stat.overall.cpi
     14726           -45.3%       8048 ±  2%  perf-stat.overall.cycles-between-cache-misses
      0.01 ±  2%      +0.0        0.01 ±  7%  perf-stat.overall.dTLB-load-miss-rate%
     98.57           -46.1       52.42        perf-stat.overall.iTLB-load-miss-rate%
      1331            +1.2%       1347        perf-stat.overall.instructions-per-iTLB-miss
      0.79           +81.9%       1.44        perf-stat.overall.ipc
      0.00 ± 11%      +0.0        0.01 ± 10%  perf-stat.overall.node-store-miss-rate%
     15528           -30.1%      10847        perf-stat.overall.path-length
 4.591e+09            -7.6%   4.24e+09        perf-stat.ps.branch-instructions
  49052787            -1.4%   48388408        perf-stat.ps.branch-misses
   1885579            -4.2%    1807271 ±  2%  perf-stat.ps.cache-misses
 5.389e+08           -94.1%   31961586 ±  3%  perf-stat.ps.cache-references
   4235546           -32.1%    2876157        perf-stat.ps.context-switches
 2.777e+10           -47.6%  1.454e+10        perf-stat.ps.cpu-cycles
     52.05 ±  2%    +633.9%     381.95 ±  4%  perf-stat.ps.cpu-migrations
    409410 ±  2%     +23.7%     506263 ±  7%  perf-stat.ps.dTLB-load-misses
   6.4e+09            -3.1%  6.203e+09        perf-stat.ps.dTLB-loads
 3.815e+09            -4.0%  3.664e+09        perf-stat.ps.dTLB-stores
  16452159            -5.9%   15479733        perf-stat.ps.iTLB-load-misses
    238902 ±  3%   +5781.4%   14050699        perf-stat.ps.iTLB-loads
  2.19e+10            -4.8%  2.086e+10        perf-stat.ps.instructions
      1571            -2.7%       1529        perf-stat.ps.minor-faults
    261165           -14.5%     223199        perf-stat.ps.node-stores
      1571            -2.7%       1529        perf-stat.ps.page-faults
 8.598e+12            -4.8%  8.188e+12        perf-stat.total.instructions
     57.34           -35.5       21.87        perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     53.70 ±  2%     -32.8       20.91        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     53.66 ±  2%     -32.8       20.91        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     53.51 ±  2%     -32.6       20.91        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     21.60           -21.6        0.00        perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     38.36 ±  2%     -17.5       20.86        perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     35.56 ±  2%     -14.1       21.44        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
     33.37 ±  2%     -12.8       20.57        perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
      7.88 ±  2%      -7.9        0.00        perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      5.64            -5.6        0.00        perf-profile.calltrace.cycles-pp.sched_ttwu_pending.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary
      3.63 ± 24%      -2.7        0.96 ± 31%  perf-profile.calltrace.cycles-pp.start_kernel.secondary_startup_64_no_verify
      3.63 ± 24%      -2.7        0.96 ± 31%  perf-profile.calltrace.cycles-pp.arch_call_rest_init.start_kernel.secondary_startup_64_no_verify
      3.63 ± 24%      -2.7        0.96 ± 31%  perf-profile.calltrace.cycles-pp.rest_init.arch_call_rest_init.start_kernel.secondary_startup_64_no_verify
      3.63 ± 24%      -2.7        0.96 ± 31%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.rest_init.arch_call_rest_init.start_kernel.secondary_startup_64_no_verify
      3.62 ± 24%      -2.7        0.96 ± 31%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.rest_init.arch_call_rest_init.start_kernel
      2.64 ± 18%      -1.7        0.96 ± 31%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.rest_init.arch_call_rest_init
      2.31 ± 15%      -1.4        0.94 ± 31%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.rest_init
      4.08 ±  2%      -1.4        2.73        perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.pipe_read
      1.90 ±  3%      -0.8        1.11 ±  5%  perf-profile.calltrace.cycles-pp.select_task_rq_fair.select_task_rq.try_to_wake_up.autoremove_wake_function.__wake_up_common
      2.12 ±  2%      -0.8        1.32 ±  3%  perf-profile.calltrace.cycles-pp.select_task_rq.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      0.86 ±  4%      -0.6        0.26 ±100%  perf-profile.calltrace.cycles-pp.copy_user_short_string.copyout._copy_to_iter.copy_page_to_iter.pipe_read
      1.60 ±  2%      -0.4        1.16 ±  3%  perf-profile.calltrace.cycles-pp.update_curr.dequeue_entity.dequeue_task_fair.__schedule.schedule
      1.12 ±  2%      -0.4        0.70 ±  3%  perf-profile.calltrace.cycles-pp.mutex_lock.pipe_write.vfs_write.ksys_write.do_syscall_64
      0.97 ±  4%      -0.2        0.75 ±  5%  perf-profile.calltrace.cycles-pp.update_load_avg.dequeue_entity.dequeue_task_fair.__schedule.schedule
      1.20 ±  4%      -0.2        1.00 ±  5%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.prepare_to_wait_event.pipe_read.vfs_read.ksys_read
      1.04 ±  4%      -0.1        0.90 ±  5%  perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.copy_page_to_iter.pipe_read.vfs_read
      1.31 ±  3%      +0.1        1.40 ±  2%  perf-profile.calltrace.cycles-pp._copy_to_iter.copy_page_to_iter.pipe_read.vfs_read.ksys_read
      1.41            +0.1        1.53        perf-profile.calltrace.cycles-pp.mutex_lock.pipe_read.vfs_read.ksys_read.do_syscall_64
      1.04            +0.2        1.21 ±  4%  perf-profile.calltrace.cycles-pp.perf_trace_sched_wakeup_template.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      2.68 ±  2%      +0.2        2.85        perf-profile.calltrace.cycles-pp.prepare_to_wait_event.pipe_read.vfs_read.ksys_read.do_syscall_64
      0.56 ±  4%      +0.2        0.80 ±  7%  perf-profile.calltrace.cycles-pp.security_file_permission.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.42 ±  2%      +0.3        1.70        perf-profile.calltrace.cycles-pp.copy_page_to_iter.pipe_read.vfs_read.ksys_read.do_syscall_64
      0.26 ±100%      +0.4        0.69 ±  6%  perf-profile.calltrace.cycles-pp.apparmor_file_permission.security_file_permission.vfs_write.ksys_write.do_syscall_64
      0.56 ±  3%      +0.5        1.07 ±  8%  perf-profile.calltrace.cycles-pp.prepare_task_switch.__schedule.schedule.pipe_read.vfs_read
      0.00            +0.6        0.56 ± 11%  perf-profile.calltrace.cycles-pp.__fdget_pos.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      0.00            +0.6        0.56 ±  7%  perf-profile.calltrace.cycles-pp.__might_resched.__might_fault._copy_from_iter.copy_page_from_iter.pipe_write
      0.00            +0.6        0.58 ±  3%  perf-profile.calltrace.cycles-pp.apparmor_file_permission.security_file_permission.vfs_read.ksys_read.do_syscall_64
      0.00            +0.6        0.58 ±  5%  perf-profile.calltrace.cycles-pp.native_sched_clock.sched_clock_cpu.update_rq_clock.__schedule.schedule
      0.00            +0.6        0.59 ±  3%  perf-profile.calltrace.cycles-pp.native_sched_clock.sched_clock_cpu.update_rq_clock.try_to_wake_up.autoremove_wake_function
      0.00            +0.6        0.62 ±  6%  perf-profile.calltrace.cycles-pp.sched_clock_cpu.update_rq_clock.__schedule.schedule.pipe_read
      0.00            +0.6        0.63 ±  2%  perf-profile.calltrace.cycles-pp.sched_clock_cpu.update_rq_clock.try_to_wake_up.autoremove_wake_function.__wake_up_common
      0.00            +0.6        0.63 ±  4%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.finish_wait.pipe_read.vfs_read.ksys_read
      0.00            +0.7        0.66 ± 10%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
      0.00            +0.7        0.66 ±  4%  perf-profile.calltrace.cycles-pp.__wrgsbase_inactive.read
      0.00            +0.7        0.68 ±  8%  perf-profile.calltrace.cycles-pp.___perf_sw_event.prepare_task_switch.__schedule.schedule.pipe_read
      0.00            +0.7        0.69 ±  4%  perf-profile.calltrace.cycles-pp.security_file_permission.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.7        0.70 ±  6%  perf-profile.calltrace.cycles-pp.update_load_avg.dequeue_task_fair.__schedule.schedule.pipe_read
      0.00            +0.7        0.70 ±  8%  perf-profile.calltrace.cycles-pp.mutex_unlock.pipe_read.vfs_read.ksys_read.do_syscall_64
      0.65 ±  5%      +0.7        1.37 ±  3%  perf-profile.calltrace.cycles-pp.os_xsave.read
      1.26 ±  4%      +0.7        1.98 ±  2%  perf-profile.calltrace.cycles-pp._copy_from_iter.copy_page_from_iter.pipe_write.vfs_write.ksys_write
      0.00            +0.7        0.72 ± 10%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      0.00            +0.7        0.74 ±  5%  perf-profile.calltrace.cycles-pp.copyin._copy_from_iter.copy_page_from_iter.pipe_write.vfs_write
      0.00            +0.8        0.76 ±  3%  perf-profile.calltrace.cycles-pp.update_load_avg.set_next_entity.pick_next_task_fair.__schedule.schedule
      0.00            +0.8        0.76 ±  6%  perf-profile.calltrace.cycles-pp.__calc_delta.update_curr.reweight_entity.dequeue_task_fair.__schedule
      0.00            +0.8        0.80 ±  4%  perf-profile.calltrace.cycles-pp.__calc_delta.update_curr.reweight_entity.enqueue_task_fair.ttwu_do_activate
      0.00            +0.8        0.81 ±  4%  perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up
      0.00            +0.8        0.82 ±  9%  perf-profile.calltrace.cycles-pp.__switch_to.read
      0.00            +0.8        0.82 ±  4%  perf-profile.calltrace.cycles-pp.finish_wait.pipe_read.vfs_read.ksys_read.do_syscall_64
      0.00            +0.9        0.87 ±  3%  perf-profile.calltrace.cycles-pp.update_rq_clock.__schedule.schedule.pipe_read.vfs_read
      0.00            +0.9        0.88 ±  4%  perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      0.00            +0.9        0.89 ±  3%  perf-profile.calltrace.cycles-pp.main
      0.00            +0.9        0.91 ±  5%  perf-profile.calltrace.cycles-pp.update_cfs_group.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      0.00            +0.9        0.91 ±  5%  perf-profile.calltrace.cycles-pp.update_rq_clock.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      0.00            +0.9        0.94 ±  2%  perf-profile.calltrace.cycles-pp.__switch_to.__schedule.schedule.pipe_read.vfs_read
      1.43 ±  3%      +1.0        2.38 ±  2%  perf-profile.calltrace.cycles-pp.copy_page_from_iter.pipe_write.vfs_write.ksys_write.do_syscall_64
      0.00            +1.0        0.95 ±  2%  perf-profile.calltrace.cycles-pp.update_cfs_group.dequeue_task_fair.__schedule.schedule.pipe_read
      0.00            +1.0        1.01 ±  5%  perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.dequeue_task_fair.__schedule.schedule
      0.00            +1.0        1.01 ±  4%  perf-profile.calltrace.cycles-pp.check_preempt_wakeup.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function
      0.91 ±  2%      +1.0        1.93 ±  3%  perf-profile.calltrace.cycles-pp.__switch_to_asm.read
      0.00            +1.0        1.04 ±  4%  perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up
      0.00            +1.2        1.18 ±  3%  perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule.pipe_read
      0.00            +1.3        1.27 ±  4%  perf-profile.calltrace.cycles-pp.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common
      0.00            +1.3        1.30 ±  4%  perf-profile.calltrace.cycles-pp.update_curr.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up
      0.00            +1.4        1.38 ±  3%  perf-profile.calltrace.cycles-pp.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      0.54 ±  3%      +1.4        1.94 ±  3%  perf-profile.calltrace.cycles-pp.__entry_text_start.read
      0.00            +1.6        1.58 ±  3%  perf-profile.calltrace.cycles-pp.restore_fpregs_from_fpstate.switch_fpu_return.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      0.17 ±141%      +1.7        1.84 ±  3%  perf-profile.calltrace.cycles-pp.__entry_text_start.write
      0.00            +1.7        1.67 ±  3%  perf-profile.calltrace.cycles-pp.reweight_entity.dequeue_task_fair.__schedule.schedule.pipe_read
      0.00            +1.7        1.68 ±  2%  perf-profile.calltrace.cycles-pp.reweight_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      0.60 ±  5%      +2.1        2.66 ±  2%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      0.00            +2.2        2.21 ±  2%  perf-profile.calltrace.cycles-pp.switch_fpu_return.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.58 ±  6%      +2.2        2.82        perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.pipe_read.vfs_read
      0.00            +2.3        2.30 ±  3%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      4.70            +2.4        7.11        perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.pipe_read.vfs_read
      0.00            +3.1        3.15 ±  3%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      0.00            +5.1        5.05 ±  2%  perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__schedule.schedule.pipe_read.vfs_read
      0.00            +7.3        7.32        perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
      7.83            +7.3       15.18        perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.vfs_write.ksys_write
      0.00            +7.5        7.52        perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      6.88 ±  2%      +7.6       14.49        perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write
      8.24            +7.8       16.01        perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.vfs_write.ksys_write.do_syscall_64
      7.11            +7.8       14.94        perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.vfs_write
     11.86 ±  6%      +8.6       20.50 ±  2%  perf-profile.calltrace.cycles-pp.mwait_idle_with_hints.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
     11.86 ±  6%      +8.6       20.51 ±  2%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     12.29            +9.5       21.75        perf-profile.calltrace.cycles-pp.pipe_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     13.40           +10.0       23.36        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     13.83           +10.4       24.21        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     14.17           +10.9       25.10        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     14.33           +11.1       25.46        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
      9.16           +11.3       20.49        perf-profile.calltrace.cycles-pp.__schedule.schedule.pipe_read.vfs_read.ksys_read
      9.44           +11.9       21.39        perf-profile.calltrace.cycles-pp.schedule.pipe_read.vfs_read.ksys_read.do_syscall_64
     15.53           +13.3       28.86        perf-profile.calltrace.cycles-pp.write
     18.03           +14.7       32.74        perf-profile.calltrace.cycles-pp.pipe_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
     19.37           +15.7       35.10        perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
     19.90           +16.3       36.21        perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
     20.88           +18.6       39.51        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
     21.30           +19.5       40.84        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
     24.96           +24.5       49.46        perf-profile.calltrace.cycles-pp.read
     57.34           -35.5       21.87        perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     57.34           -35.5       21.87        perf-profile.children.cycles-pp.cpu_startup_entry
     57.17           -35.3       21.87        perf-profile.children.cycles-pp.do_idle
     53.70 ±  2%     -32.8       20.91        perf-profile.children.cycles-pp.start_secondary
     21.62           -21.6        0.00        perf-profile.children.cycles-pp.poll_idle
     41.05           -19.2       21.83        perf-profile.children.cycles-pp.cpuidle_idle_call
     35.70 ±  2%     -14.2       21.51        perf-profile.children.cycles-pp.cpuidle_enter
     35.64 ±  2%     -14.1       21.51        perf-profile.children.cycles-pp.cpuidle_enter_state
      8.40            -8.4        0.00        perf-profile.children.cycles-pp.flush_smp_call_function_queue
      6.02            -6.0        0.00        perf-profile.children.cycles-pp.sched_ttwu_pending
      5.20 ±  2%      -5.2        0.00        perf-profile.children.cycles-pp.schedule_idle
      4.01            -3.8        0.25 ± 11%  perf-profile.children.cycles-pp.menu_select
      3.63 ± 24%      -2.7        0.96 ± 31%  perf-profile.children.cycles-pp.start_kernel
      3.63 ± 24%      -2.7        0.96 ± 31%  perf-profile.children.cycles-pp.arch_call_rest_init
      3.63 ± 24%      -2.7        0.96 ± 31%  perf-profile.children.cycles-pp.rest_init
      2.59 ±  3%      -2.3        0.32 ± 10%  perf-profile.children.cycles-pp.ttwu_queue_wakelist
      1.83 ±  2%      -1.7        0.09 ± 20%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      4.18 ±  2%      -1.4        2.81        perf-profile.children.cycles-pp.dequeue_entity
      1.13 ±  2%      -1.1        0.04 ± 72%  perf-profile.children.cycles-pp.tick_nohz_next_event
      1.92 ±  3%      -0.8        1.11 ±  5%  perf-profile.children.cycles-pp.select_task_rq_fair
      2.12 ±  2%      -0.8        1.34 ±  4%  perf-profile.children.cycles-pp.select_task_rq
      0.72 ±  3%      -0.7        0.06 ± 47%  perf-profile.children.cycles-pp.ktime_get
      1.53 ±  2%      -0.6        0.90 ±  4%  perf-profile.children.cycles-pp._raw_spin_lock
      1.86            -0.5        1.33        perf-profile.children.cycles-pp.sched_clock_cpu
      1.63 ±  2%      -0.4        1.20        perf-profile.children.cycles-pp.native_sched_clock
      3.03 ±  3%      -0.4        2.65 ±  3%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      1.45 ±  3%      -0.4        1.09 ±  8%  perf-profile.children.cycles-pp.prepare_task_switch
      2.58            -0.3        2.30        perf-profile.children.cycles-pp.mutex_lock
      0.53 ±  4%      -0.2        0.34 ±  6%  perf-profile.children.cycles-pp.finish_task_switch
      1.05 ±  4%      -0.1        0.90 ±  5%  perf-profile.children.cycles-pp.copyout
      1.38 ±  5%      -0.1        1.26 ±  3%  perf-profile.children.cycles-pp.set_next_entity
      0.17 ± 10%      -0.1        0.09 ±  4%  perf-profile.children.cycles-pp.rcu_note_context_switch
      0.18 ±  9%      -0.0        0.13 ± 16%  perf-profile.children.cycles-pp.place_entity
      0.10 ±  6%      -0.0        0.05 ± 48%  perf-profile.children.cycles-pp.rb_next
      0.08 ± 11%      -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.sched_clock
      0.06 ± 11%      +0.0        0.10 ± 13%  perf-profile.children.cycles-pp.rw_verify_area
      0.08 ± 11%      +0.0        0.12 ± 11%  perf-profile.children.cycles-pp.__x64_sys_read
      0.30 ±  8%      +0.0        0.34 ±  8%  perf-profile.children.cycles-pp.perf_trace_buf_update
      0.02 ±141%      +0.0        0.06 ± 11%  perf-profile.children.cycles-pp.clockevents_program_event
      0.06 ± 45%      +0.0        0.10 ± 13%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.04 ± 71%      +0.0        0.08 ±  8%  perf-profile.children.cycles-pp.perf_swevent_get_recursion_context
      0.04 ± 71%      +0.1        0.09 ± 14%  perf-profile.children.cycles-pp.inode_needs_update_time
      0.03 ±102%      +0.1        0.08 ± 19%  perf-profile.children.cycles-pp.save_fpregs_to_fpstate
      0.22 ±  6%      +0.1        0.28 ± 11%  perf-profile.children.cycles-pp.put_prev_entity
      0.01 ±223%      +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.08 ± 14%      +0.1        0.13 ± 16%  perf-profile.children.cycles-pp.__irq_exit_rcu
      0.12 ±  6%      +0.1        0.17 ±  4%  perf-profile.children.cycles-pp.perf_trace_buf_alloc
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.default_wake_function
      0.31 ±  5%      +0.1        0.38 ±  6%  perf-profile.children.cycles-pp.__rdgsbase_inactive
      0.09 ±  9%      +0.1        0.16 ± 10%  perf-profile.children.cycles-pp.__x64_sys_write
      0.09 ± 10%      +0.1        0.16 ±  8%  perf-profile.children.cycles-pp.rcu_all_qs
      0.04 ± 44%      +0.1        0.12 ± 16%  perf-profile.children.cycles-pp.page_copy_sane
      0.04 ± 71%      +0.1        0.11 ± 18%  perf-profile.children.cycles-pp.scheduler_tick
      0.30 ±  8%      +0.1        0.38 ±  4%  perf-profile.children.cycles-pp.pick_next_entity
      0.11 ± 15%      +0.1        0.20 ±  7%  perf-profile.children.cycles-pp.syscall_enter_from_user_mode
      0.09 ± 15%      +0.1        0.18 ± 17%  perf-profile.children.cycles-pp.update_process_times
      0.05 ± 46%      +0.1        0.14 ± 14%  perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64
      0.07 ± 11%      +0.1        0.15 ± 15%  perf-profile.children.cycles-pp.perf_trace_run_bpf_submit
      0.06 ± 11%      +0.1        0.15 ± 12%  perf-profile.children.cycles-pp.kill_fasync
      0.10 ± 16%      +0.1        0.20 ± 17%  perf-profile.children.cycles-pp.tick_sched_handle
      0.04 ± 72%      +0.1        0.14 ± 34%  perf-profile.children.cycles-pp.write@plt
      0.12 ± 19%      +0.1        0.23 ± 18%  perf-profile.children.cycles-pp.tick_sched_timer
      0.00            +0.1        0.12 ± 36%  perf-profile.children.cycles-pp.exit_to_user_mode_loop
      0.19 ±  9%      +0.1        0.31 ±  9%  perf-profile.children.cycles-pp.__get_task_ioprio
      0.57 ±  6%      +0.1        0.68 ±  4%  perf-profile.children.cycles-pp.__wrgsbase_inactive
      0.13 ± 15%      +0.1        0.25 ±  7%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
      1.32 ±  3%      +0.1        1.44 ±  2%  perf-profile.children.cycles-pp._copy_to_iter
      1.54 ±  2%      +0.1        1.67        perf-profile.children.cycles-pp.__might_resched
      0.00            +0.1        0.14 ± 11%  perf-profile.children.cycles-pp.cgroup_rstat_updated
      0.14 ±  9%      +0.2        0.30 ±  6%  perf-profile.children.cycles-pp.aa_file_perm
      0.09 ± 20%      +0.2        0.26 ±  5%  perf-profile.children.cycles-pp.__cgroup_account_cputime
      0.14 ± 13%      +0.2        0.31 ± 12%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.14 ± 11%      +0.2        0.31 ±  5%  perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
      0.20 ± 14%      +0.2        0.37 ± 12%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.21 ±  9%      +0.2        0.38 ±  5%  perf-profile.children.cycles-pp.__cond_resched
      1.06            +0.2        1.23 ±  4%  perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
      0.06 ± 19%      +0.2        0.24 ±  8%  perf-profile.children.cycles-pp.check_cfs_rq_runtime
      0.22 ±  7%      +0.2        0.41 ±  6%  perf-profile.children.cycles-pp.atime_needs_update
      0.00            +0.2        0.22 ±  6%  perf-profile.children.cycles-pp.set_next_buddy
      2.70 ±  2%      +0.2        2.92        perf-profile.children.cycles-pp.prepare_to_wait_event
      0.28 ± 11%      +0.2        0.51 ±  9%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.88 ±  5%      +0.2        1.12 ±  4%  perf-profile.children.cycles-pp.__might_fault
      0.22 ±  8%      +0.2        0.46 ±  6%  perf-profile.children.cycles-pp.current_time
      0.28 ±  7%      +0.2        0.52 ±  5%  perf-profile.children.cycles-pp.touch_atime
      0.30 ± 11%      +0.2        0.55 ±  8%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      0.21 ±  9%      +0.2        0.46 ±  8%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.22 ±  8%      +0.2        0.47 ±  6%  perf-profile.children.cycles-pp.file_update_time
      0.62 ±  2%      +0.3        0.88 ±  3%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      2.96 ±  2%      +0.3        3.23 ±  3%  perf-profile.children.cycles-pp.enqueue_entity
      1.44 ±  2%      +0.3        1.72        perf-profile.children.cycles-pp.copy_page_to_iter
      1.53 ±  3%      +0.3        1.81 ±  3%  perf-profile.children.cycles-pp.__switch_to
      0.26 ±  6%      +0.3        0.57 ±  5%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.21 ±  7%      +0.3        0.53 ±  7%  perf-profile.children.cycles-pp.update_min_vruntime
      2.68 ±  5%      +0.3        3.02        perf-profile.children.cycles-pp.pick_next_task_fair
      0.12 ±  8%      +0.3        0.46 ±  4%  perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
      0.42 ±  8%      +0.4        0.78 ±  9%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      0.73 ±  6%      +0.4        1.11 ±  5%  perf-profile.children.cycles-pp.mutex_unlock
      0.48 ±  7%      +0.4        0.86 ±  8%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.87 ±  2%      +0.4        1.27 ±  4%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.90 ±  3%      +0.4        1.30 ±  4%  perf-profile.children.cycles-pp.apparmor_file_permission
      0.14 ± 15%      +0.4        0.54 ±  2%  perf-profile.children.cycles-pp.__list_add_valid
      0.89 ±  4%      +0.4        1.30 ±  3%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.37 ±  7%      +0.4        0.78 ±  8%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.33 ±  4%      +0.4        0.75 ±  5%  perf-profile.children.cycles-pp.copyin
      0.36 ±  4%      +0.5        0.82 ±  4%  perf-profile.children.cycles-pp.finish_wait
      1.03 ±  3%      +0.5        1.50 ±  4%  perf-profile.children.cycles-pp.security_file_permission
      0.18 ± 16%      +0.5        0.68 ±  7%  perf-profile.children.cycles-pp.cpuacct_charge
      0.40 ±  6%      +0.5        0.93 ±  3%  perf-profile.children.cycles-pp.main
      0.53 ±  2%      +0.6        1.12 ±  7%  perf-profile.children.cycles-pp.__fdget_pos
      0.65 ±  5%      +0.7        1.38 ±  3%  perf-profile.children.cycles-pp.os_xsave
      1.30 ±  4%      +0.7        2.04 ±  2%  perf-profile.children.cycles-pp._copy_from_iter
      0.65 ±  3%      +0.8        1.41 ±  2%  perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
      1.02 ±  2%      +0.8        1.82 ±  2%  perf-profile.children.cycles-pp.update_rq_clock
      1.44 ±  3%      +1.0        2.42 ±  2%  perf-profile.children.cycles-pp.copy_page_from_iter
      0.35 ±  5%      +1.0        1.38 ±  3%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.00            +1.0        1.04 ±  5%  perf-profile.children.cycles-pp.check_preempt_wakeup
      0.21 ±  7%      +1.1        1.28 ±  4%  perf-profile.children.cycles-pp.check_preempt_curr
      0.46 ±  5%      +1.1        1.56 ±  2%  perf-profile.children.cycles-pp.__calc_delta
      0.83 ±  5%      +1.1        1.96 ±  2%  perf-profile.children.cycles-pp.update_cfs_group
      2.68 ±  4%      +1.3        3.98 ±  3%  perf-profile.children.cycles-pp.update_load_avg
      1.02            +1.4        2.47 ±  2%  perf-profile.children.cycles-pp.__entry_text_start
      0.00            +1.6        1.59 ±  3%  perf-profile.children.cycles-pp.restore_fpregs_from_fpstate
      0.36 ±  6%      +1.9        2.23 ±  3%  perf-profile.children.cycles-pp.switch_fpu_return
      0.49 ±  4%      +2.1        2.61 ±  4%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      4.72            +2.4        7.12        perf-profile.children.cycles-pp.dequeue_task_fair
      0.74 ±  6%      +2.4        3.15 ±  3%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.63 ±  4%      +2.7        3.35        perf-profile.children.cycles-pp.reweight_entity
      1.74 ±  3%      +3.2        4.91 ±  2%  perf-profile.children.cycles-pp.update_curr
      3.34 ±  2%      +4.0        7.34        perf-profile.children.cycles-pp.enqueue_task_fair
      3.44 ±  2%      +4.1        7.54        perf-profile.children.cycles-pp.ttwu_do_activate
      0.26 ±  6%      +4.9        5.12 ±  3%  perf-profile.children.cycles-pp.switch_mm_irqs_off
     14.17 ±  2%      +6.6       20.77        perf-profile.children.cycles-pp.__schedule
      7.84            +7.4       15.19        perf-profile.children.cycles-pp.__wake_up_common
      6.90            +7.7       14.56        perf-profile.children.cycles-pp.try_to_wake_up
      8.26            +7.8       16.03        perf-profile.children.cycles-pp.__wake_up_common_lock
      7.12            +7.9       14.98        perf-profile.children.cycles-pp.autoremove_wake_function
     11.94 ±  6%      +8.6       20.59 ±  2%  perf-profile.children.cycles-pp.mwait_idle_with_hints
     11.95 ±  6%      +8.6       20.59 ±  2%  perf-profile.children.cycles-pp.intel_idle
     12.32            +9.5       21.81        perf-profile.children.cycles-pp.pipe_write
     13.43           +10.0       23.43        perf-profile.children.cycles-pp.vfs_write
     13.85           +10.4       24.25        perf-profile.children.cycles-pp.ksys_write
      9.45           +12.1       21.52        perf-profile.children.cycles-pp.schedule
     15.82           +13.1       28.92        perf-profile.children.cycles-pp.write
     18.16           +14.9       33.02        perf-profile.children.cycles-pp.pipe_read
     19.41           +15.7       35.11        perf-profile.children.cycles-pp.vfs_read
     19.92           +16.3       36.24        perf-profile.children.cycles-pp.ksys_read
     25.25           +24.2       49.48        perf-profile.children.cycles-pp.read
     35.17           +29.7       64.91        perf-profile.children.cycles-pp.do_syscall_64
     35.72           +30.8       66.50        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     21.21           -21.2        0.00        perf-profile.self.cycles-pp.poll_idle
      1.76 ±  2%      -1.6        0.14 ± 11%  perf-profile.self.cycles-pp.menu_select
      1.52 ±  2%      -0.6        0.90 ±  4%  perf-profile.self.cycles-pp._raw_spin_lock
      0.72 ±  5%      -0.5        0.23 ± 11%  perf-profile.self.cycles-pp.__wake_up_common
      2.98 ±  3%      -0.4        2.55 ±  2%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      1.58 ±  2%      -0.4        1.16        perf-profile.self.cycles-pp.native_sched_clock
      0.78 ±  2%      -0.4        0.40 ± 14%  perf-profile.self.cycles-pp.prepare_task_switch
      0.32 ±  8%      -0.2        0.11 ± 13%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.47 ±  5%      -0.2        0.28 ±  8%  perf-profile.self.cycles-pp.finish_task_switch
      1.24            -0.2        1.06 ±  3%  perf-profile.self.cycles-pp.mutex_lock
      1.03 ±  3%      -0.2        0.86 ±  2%  perf-profile.self.cycles-pp.enqueue_entity
      0.46 ± 13%      -0.2        0.31 ± 12%  perf-profile.self.cycles-pp.ttwu_queue_wakelist
      0.20 ±  5%      -0.1        0.10 ± 11%  perf-profile.self.cycles-pp.sched_clock_cpu
      0.16 ±  8%      -0.1        0.08 ±  6%  perf-profile.self.cycles-pp.rcu_note_context_switch
      0.38 ±  9%      -0.1        0.31 ± 10%  perf-profile.self.cycles-pp.set_next_entity
      0.35 ±  6%      -0.1        0.29 ±  7%  perf-profile.self.cycles-pp.perf_tp_event
      0.16 ±  6%      -0.0        0.11 ± 13%  perf-profile.self.cycles-pp.put_prev_entity
      0.16 ±  7%      -0.0        0.12 ± 14%  perf-profile.self.cycles-pp.place_entity
      0.09 ±  7%      -0.0        0.05 ± 47%  perf-profile.self.cycles-pp.rb_next
      0.07 ± 13%      +0.0        0.09 ± 11%  perf-profile.self.cycles-pp.perf_trace_buf_alloc
      0.08 ± 12%      +0.0        0.11 ± 13%  perf-profile.self.cycles-pp.__x64_sys_read
      0.14 ±  3%      +0.0        0.17 ± 10%  perf-profile.self.cycles-pp.__might_fault
      0.20 ±  9%      +0.0        0.24 ±  8%  perf-profile.self.cycles-pp.select_task_rq
      0.02 ±141%      +0.0        0.06 ±  8%  perf-profile.self.cycles-pp.perf_swevent_event
      0.03 ± 70%      +0.0        0.08 ± 14%  perf-profile.self.cycles-pp.inode_needs_update_time
      0.07 ± 11%      +0.0        0.12 ± 14%  perf-profile.self.cycles-pp.perf_trace_buf_update
      0.06 ± 17%      +0.1        0.12 ±  9%  perf-profile.self.cycles-pp.rcu_all_qs
      0.23 ±  7%      +0.1        0.28 ±  3%  perf-profile.self.cycles-pp.perf_trace_sched_wakeup_template
      0.06 ±  7%      +0.1        0.12 ± 12%  perf-profile.self.cycles-pp.finish_wait
      0.08 ± 11%      +0.1        0.14 ± 14%  perf-profile.self.cycles-pp.__x64_sys_write
      0.11 ± 11%      +0.1        0.18 ±  5%  perf-profile.self.cycles-pp.atime_needs_update
      0.31 ±  4%      +0.1        0.38 ±  6%  perf-profile.self.cycles-pp.__rdgsbase_inactive
      0.05 ± 51%      +0.1        0.12 ± 14%  perf-profile.self.cycles-pp.__cgroup_account_cputime
      0.01 ±223%      +0.1        0.08 ±  8%  perf-profile.self.cycles-pp.perf_swevent_get_recursion_context
      0.05 ± 46%      +0.1        0.12 ± 14%  perf-profile.self.cycles-pp.touch_atime
      0.01 ±223%      +0.1        0.08 ± 17%  perf-profile.self.cycles-pp.rw_verify_area
      0.15 ±  8%      +0.1        0.23 ±  6%  perf-profile.self.cycles-pp.check_preempt_curr
      0.13 ± 11%      +0.1        0.20 ±  3%  perf-profile.self.cycles-pp.security_file_permission
      0.06 ± 11%      +0.1        0.14 ± 13%  perf-profile.self.cycles-pp.perf_trace_run_bpf_submit
      0.04 ± 71%      +0.1        0.12 ± 34%  perf-profile.self.cycles-pp.write@plt
      0.10 ±  6%      +0.1        0.19 ± 13%  perf-profile.self.cycles-pp.ttwu_do_activate
      0.05 ± 46%      +0.1        0.14 ± 18%  perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64
      0.42 ±  7%      +0.1        0.50 ±  7%  perf-profile.self.cycles-pp.update_rq_clock
      0.10 ± 11%      +0.1        0.20 ±  7%  perf-profile.self.cycles-pp.syscall_enter_from_user_mode
      0.02 ± 99%      +0.1        0.12 ± 15%  perf-profile.self.cycles-pp.kill_fasync
      0.11 ± 12%      +0.1        0.20 ±  9%  perf-profile.self.cycles-pp.__cond_resched
      0.10 ±  3%      +0.1        0.20 ±  9%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode
      0.06 ± 11%      +0.1        0.16 ± 14%  perf-profile.self.cycles-pp.file_update_time
      0.00            +0.1        0.10 ± 15%  perf-profile.self.cycles-pp.page_copy_sane
      0.18 ±  7%      +0.1        0.28 ±  8%  perf-profile.self.cycles-pp.__get_task_ioprio
      0.11 ±  8%      +0.1        0.22 ± 20%  perf-profile.self.cycles-pp.__wake_up_common_lock
      0.12 ± 17%      +0.1        0.23 ±  5%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode_prepare
      0.57 ±  6%      +0.1        0.68 ±  4%  perf-profile.self.cycles-pp.__wrgsbase_inactive
      0.12 ±  6%      +0.1        0.24 ±  9%  perf-profile.self.cycles-pp.exit_to_user_mode_prepare
      0.10 ±  8%      +0.1        0.22 ±  4%  perf-profile.self.cycles-pp.copy_page_to_iter
      0.00            +0.1        0.13 ± 10%  perf-profile.self.cycles-pp.cgroup_rstat_updated
      0.12 ± 10%      +0.1        0.25 ±  8%  perf-profile.self.cycles-pp.ksys_write
      0.17 ±  9%      +0.1        0.30 ±  7%  perf-profile.self.cycles-pp.autoremove_wake_function
      1.54 ±  2%      +0.1        1.67        perf-profile.self.cycles-pp.__might_resched
      0.14 ± 14%      +0.1        0.28 ± 13%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.13 ± 12%      +0.1        0.27 ±  6%  perf-profile.self.cycles-pp.aa_file_perm
      0.16 ± 13%      +0.1        0.30 ±  4%  perf-profile.self.cycles-pp.current_time
      0.14 ±  3%      +0.2        0.30 ±  8%  perf-profile.self.cycles-pp._copy_to_iter
      0.14 ± 11%      +0.2        0.31 ±  5%  perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
      0.03 ±101%      +0.2        0.21 ± 10%  perf-profile.self.cycles-pp.check_cfs_rq_runtime
      0.12 ± 10%      +0.2        0.30 ±  7%  perf-profile.self.cycles-pp.copy_page_from_iter
      0.41 ±  4%      +0.2        0.60 ±  5%  perf-profile.self.cycles-pp.vfs_write
      0.54 ±  4%      +0.2        0.76        perf-profile.self.cycles-pp.dequeue_task_fair
      0.00            +0.2        0.22 ±  6%  perf-profile.self.cycles-pp.set_next_buddy
      0.38 ±  9%      +0.2        0.62 ±  5%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.25 ±  5%      +0.2        0.50        perf-profile.self.cycles-pp._copy_from_iter
      0.21 ±  7%      +0.3        0.46 ±  8%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.61            +0.3        0.87 ±  3%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.76 ±  5%      +0.3        1.02 ±  4%  perf-profile.self.cycles-pp.apparmor_file_permission
      0.35 ±  6%      +0.3        0.62 ±  4%  perf-profile.self.cycles-pp.switch_fpu_return
      1.48 ±  3%      +0.3        1.76 ±  2%  perf-profile.self.cycles-pp.__switch_to
      0.18 ±  7%      +0.3        0.46 ±  6%  perf-profile.self.cycles-pp.update_min_vruntime
      0.44 ±  7%      +0.3        0.73 ±  6%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.22 ±  7%      +0.3        0.54 ±  5%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.28 ±  7%      +0.3        0.61 ±  7%  perf-profile.self.cycles-pp.ksys_read
      0.12 ± 11%      +0.3        0.46 ±  4%  perf-profile.self.cycles-pp.perf_trace_sched_stat_runtime
      0.73 ±  6%      +0.4        1.09 ±  6%  perf-profile.self.cycles-pp.mutex_unlock
      0.36 ±  8%      +0.4        0.74 ±  8%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.85            +0.4        1.24 ±  4%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.35 ±  6%      +0.4        0.74 ±  4%  perf-profile.self.cycles-pp.try_to_wake_up
      0.14 ± 16%      +0.4        0.54 ±  2%  perf-profile.self.cycles-pp.__list_add_valid
      0.84 ±  4%      +0.4        1.27 ±  3%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.38 ±  3%      +0.5        0.84 ±  4%  perf-profile.self.cycles-pp.do_syscall_64
      0.39 ±  8%      +0.5        0.86 ±  5%  perf-profile.self.cycles-pp.read
      0.40 ±  6%      +0.5        0.88 ±  4%  perf-profile.self.cycles-pp.write
      0.94 ±  5%      +0.5        1.42 ±  5%  perf-profile.self.cycles-pp.update_load_avg
      0.36 ±  6%      +0.5        0.86 ±  4%  perf-profile.self.cycles-pp.main
      0.17 ± 16%      +0.5        0.67 ±  7%  perf-profile.self.cycles-pp.cpuacct_charge
      0.22 ±  4%      +0.5        0.76 ±  3%  perf-profile.self.cycles-pp.schedule
      0.52 ±  3%      +0.6        1.08 ±  7%  perf-profile.self.cycles-pp.__fdget_pos
      0.58 ±  5%      +0.6        1.16 ±  6%  perf-profile.self.cycles-pp.vfs_read
      0.63 ±  4%      +0.6        1.26 ±  3%  perf-profile.self.cycles-pp.reweight_entity
      0.64 ±  6%      +0.7        1.31        perf-profile.self.cycles-pp.pipe_write
      0.46 ±  4%      +0.7        1.15 ±  4%  perf-profile.self.cycles-pp.__entry_text_start
      0.65 ±  5%      +0.7        1.37 ±  2%  perf-profile.self.cycles-pp.os_xsave
      0.62 ±  3%      +0.7        1.35 ±  3%  perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.35 ±  7%      +0.7        1.09 ±  4%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.77 ±  6%      +0.8        1.52 ±  3%  perf-profile.self.cycles-pp.update_curr
      0.00            +0.9        0.90 ±  6%  perf-profile.self.cycles-pp.check_preempt_wakeup
      0.94 ±  5%      +1.0        1.94        perf-profile.self.cycles-pp.pipe_read
      0.55 ±  5%      +1.1        1.60 ±  4%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.45 ±  5%      +1.1        1.56 ±  2%  perf-profile.self.cycles-pp.__calc_delta
      0.81 ±  5%      +1.1        1.93 ±  2%  perf-profile.self.cycles-pp.update_cfs_group
      0.00            +1.6        1.59 ±  3%  perf-profile.self.cycles-pp.restore_fpregs_from_fpstate
      0.25 ±  7%      +4.8        5.07 ±  2%  perf-profile.self.cycles-pp.switch_mm_irqs_off
     11.94 ±  6%      +8.6       20.59 ±  2%  perf-profile.self.cycles-pp.mwait_idle_with_hints





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
Liu, Yujie Nov. 22, 2022, 5:52 a.m. UTC | #10 
Please be noted that we observed various performance changes in
different benchmarks. The impact is well discussed in the mail thread,
e.g. "strictly synchronous jobs may benefit, but netperf will get
punished". We are just sending this report to provide some additional
test data for your reference. Thanks.

Greeting,

FYI, we noticed a -66.0% regression of netperf.Throughput_Mbps due to commit:

commit: b6aabb01e3004b0285d953761e1592974b4b50fa ("[PATCH] sched: consider WF_SYNC to find idle siblings")
url: https://github.com/intel-lab-lkp/linux/commits/Andrei-Vagin/sched-consider-WF_SYNC-to-find-idle-siblings/20221028-042823
base: https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git c36eae5a86d15a694968be35d7ff056854997a72
patch link: https://lore.kernel.org/all/20221027202603.670616-1-avagin@google.com/
patch subject: [PATCH] sched: consider WF_SYNC to find idle siblings

in testcase: netperf
on test machine: 128 threads 2 sockets Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz (Ice Lake) with 128G memory
with following parameters:

	ip: ipv4
	runtime: 300s
	nr_threads: 50%
	cluster: cs-localhost
	send_size: 10K
	test: SCTP_STREAM_MANY
	cpufreq_governor: performance

test-description: Netperf is a benchmark that can be use to measure various aspect of networking performance.
test-url: http://www.netperf.org/netperf/

In addition to that, the commit also has significant impact on the following tests:

+------------------+------------------------------------------------------------------------------------------------+
| testcase: change | stress-ng: stress-ng.sctp.ops_per_sec -48.4% regression                                        |
| test machine     | 128 threads 2 sockets Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz (Ice Lake) with 128G memory |
| test parameters  | class=network                                                                                  |
|                  | cpufreq_governor=performance                                                                   |
|                  | nr_threads=100%                                                                                |
|                  | test=sctp                                                                                      |
|                  | testtime=60s                                                                                   |
+------------------+------------------------------------------------------------------------------------------------+
| testcase: change | unixbench: unixbench.score 36.2% improvement                                                   |
| test machine     | 16 threads 1 sockets Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz (Coffee Lake) with 32G memory      |
| test parameters  | cpufreq_governor=performance                                                                   |
|                  | nr_task=30%                                                                                    |
|                  | runtime=300s                                                                                   |
|                  | test=context1                                                                                  |
+------------------+------------------------------------------------------------------------------------------------+
| testcase: change | netperf: netperf.Throughput_tps 53.3% improvement                                              |
| test machine     | 144 threads 4 sockets Intel(R) Xeon(R) Gold 5318H CPU @ 2.50GHz (Cooper Lake) with 128G memory |
| test parameters  | cluster=cs-localhost                                                                           |
|                  | cpufreq_governor=performance                                                                   |
|                  | ip=ipv4                                                                                        |
|                  | nr_threads=25%                                                                                 |
|                  | runtime=300s                                                                                   |
|                  | test=TCP_RR                                                                                    |
+------------------+------------------------------------------------------------------------------------------------+


Details are as below:

=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/send_size/tbox_group/test/testcase:
  cs-localhost/gcc-11/performance/ipv4/x86_64-rhel-8.3/50%/debian-11.1-x86_64-20220510.cgz/300s/10K/lkp-icl-2sp4/SCTP_STREAM_MANY/netperf

commit: 
  c36eae5a86 ("sched/psi: Fix possible missing or delayed pending event")
  b6aabb01e3 ("sched: consider WF_SYNC to find idle siblings")

c36eae5a86d15a69 b6aabb01e3004b0285d953761e1 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      7844           -66.0%       2669 ± 23%  netperf.Throughput_Mbps
    502046           -66.0%     170845 ± 23%  netperf.Throughput_total_Mbps
     96866        +87957.6%   85298188 ± 23%  netperf.time.involuntary_context_switches
     22262            -5.6%      21022        netperf.time.minor_page_faults
      1674           -79.6%     341.00 ± 23%  netperf.time.percent_of_cpu_this_job_got
      4913           -79.7%     995.08 ± 23%  netperf.time.system_time
    133.78 ±  2%     -74.6%      34.03 ± 23%  netperf.time.user_time
    345136          +237.6%    1165282 ±  9%  netperf.time.voluntary_context_switches
 1.839e+09           -66.0%  6.257e+08 ± 23%  netperf.workload
     34569           +22.0%      42159        uptime.idle
 2.841e+10           +27.0%  3.608e+10        cpuidle..time
 5.644e+08           -86.9%   73779743        cpuidle..usage
 1.201e+09 ±  2%     -66.7%  4.002e+08 ± 21%  numa-numastat.node0.local_node
 1.201e+09 ±  2%     -66.7%  4.003e+08 ± 21%  numa-numastat.node0.numa_hit
 1.171e+09 ±  3%     -65.3%  4.064e+08 ± 25%  numa-numastat.node1.local_node
 1.171e+09 ±  3%     -65.3%  4.064e+08 ± 25%  numa-numastat.node1.numa_hit
     74.00           +25.0%      92.50        vmstat.cpu.id
   4716732           -15.7%    3974601        vmstat.memory.cache
     31.83           -65.4%      11.00 ± 21%  vmstat.procs.r
   3346244           -82.4%     589168 ± 23%  vmstat.system.cs
     74.13           +19.0       93.17        mpstat.cpu.all.idle%
      0.60 ±  2%      +0.3        0.95 ± 24%  mpstat.cpu.all.irq%
      3.85            -2.9        0.93 ± 22%  mpstat.cpu.all.soft%
     20.66           -15.9        4.72 ± 23%  mpstat.cpu.all.sys%
      0.77            -0.5        0.24 ± 20%  mpstat.cpu.all.usr%
    147463 ± 10%     +15.6%     170513 ±  6%  numa-meminfo.node0.SUnreclaim
    803425 ±  5%     -95.6%      35689 ± 56%  numa-meminfo.node1.Active
    803380 ±  5%     -95.6%      35646 ± 56%  numa-meminfo.node1.Active(anon)
    330592 ±  9%     -90.1%      32819 ± 45%  numa-meminfo.node1.Mapped
    815993 ±  5%     -90.1%      80396 ± 35%  numa-meminfo.node1.Shmem
    832708 ±  2%     -95.0%      41879 ± 52%  meminfo.Active
    832610 ±  2%     -95.0%      41793 ± 52%  meminfo.Active(anon)
   4598130           -16.3%    3847596        meminfo.Cached
   1436261           -53.0%     674663 ±  3%  meminfo.Committed_AS
    381642 ±  3%     -81.9%      69247 ± 10%  meminfo.Mapped
   6510432           -10.7%    5812535        meminfo.Memused
    846123 ±  2%     -88.6%      96290 ± 24%  meminfo.Shmem
  10463627           -32.4%    7073752 ±  3%  meminfo.max_used_kB
     36887 ± 10%     +15.4%      42580 ±  6%  numa-vmstat.node0.nr_slab_unreclaimable
 1.201e+09 ±  2%     -66.7%  4.003e+08 ± 21%  numa-vmstat.node0.numa_hit
 1.201e+09 ±  2%     -66.7%  4.002e+08 ± 21%  numa-vmstat.node0.numa_local
    200557 ±  5%     -95.5%       8951 ± 56%  numa-vmstat.node1.nr_active_anon
     83108 ±  9%     -90.0%       8342 ± 44%  numa-vmstat.node1.nr_mapped
    203712 ±  5%     -90.1%      20228 ± 35%  numa-vmstat.node1.nr_shmem
    200557 ±  5%     -95.5%       8951 ± 56%  numa-vmstat.node1.nr_zone_active_anon
 1.171e+09 ±  3%     -65.3%  4.064e+08 ± 25%  numa-vmstat.node1.numa_hit
 1.171e+09 ±  3%     -65.3%  4.064e+08 ± 25%  numa-vmstat.node1.numa_local
    957.83           -73.0%     258.33 ± 21%  turbostat.Avg_MHz
     29.11           -21.2        7.89 ± 20%  turbostat.Busy%
 4.303e+08          -100.0%     107318 ± 26%  turbostat.C1
      5.52            -5.5        0.03 ±100%  turbostat.C1%
 1.317e+08 ±  2%     -62.3%   49626627 ± 35%  turbostat.C1E
     74.33           -24.7%      56.00 ±  4%  turbostat.CoreTmp
      0.30           +20.0%       0.36 ±  4%  turbostat.IPC
   1105923           -99.4%       6441 ± 46%  turbostat.POLL
      0.03            -0.0        0.00        turbostat.POLL%
     74.33 ±  2%     -24.4%      56.17 ±  4%  turbostat.PkgTmp
    455.30           -38.7%     279.16 ±  5%  turbostat.PkgWatt
     46.51            -6.1%      43.69        turbostat.RAMWatt
    208073 ±  2%     -95.0%      10499 ± 52%  proc-vmstat.nr_active_anon
    101503            -1.2%     100305        proc-vmstat.nr_anon_pages
   1149456           -16.3%     962060        proc-vmstat.nr_file_pages
    106787            +8.5%     115818        proc-vmstat.nr_inactive_anon
     95661 ±  3%     -81.7%      17510 ± 10%  proc-vmstat.nr_mapped
      2428 ±  2%      -6.0%       2282        proc-vmstat.nr_page_table_pages
    211454 ±  2%     -88.5%      24233 ± 24%  proc-vmstat.nr_shmem
     31087            -2.0%      30451        proc-vmstat.nr_slab_reclaimable
     74377            +7.9%      80233        proc-vmstat.nr_slab_unreclaimable
    208073 ±  2%     -95.0%      10499 ± 52%  proc-vmstat.nr_zone_active_anon
    106787            +8.5%     115818        proc-vmstat.nr_zone_inactive_anon
 2.372e+09           -66.0%  8.068e+08 ± 23%  proc-vmstat.numa_hit
 2.372e+09           -66.0%  8.066e+08 ± 23%  proc-vmstat.numa_local
    242358 ±  2%     +95.5%     473925 ±  7%  proc-vmstat.pgactivate
 1.366e+10           -66.0%  4.643e+09 ± 23%  proc-vmstat.pgalloc_normal
   1461389           -16.3%    1222803 ±  4%  proc-vmstat.pgfault
 1.366e+10           -66.0%  4.643e+09 ± 23%  proc-vmstat.pgfree
     12556 ± 50%     -99.9%      17.47 ±159%  sched_debug.cfs_rq:/.MIN_vruntime.avg
    766470 ± 38%     -99.7%       2236 ±159%  sched_debug.cfs_rq:/.MIN_vruntime.max
     93742 ± 44%     -99.8%     196.88 ±159%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
      0.24 ±  6%     -58.4%       0.10 ± 37%  sched_debug.cfs_rq:/.h_nr_running.avg
      1.06 ±  7%     +66.3%       1.76 ±  5%  sched_debug.cfs_rq:/.h_nr_running.max
    218.12 ±  7%    +153.6%     553.21 ± 76%  sched_debug.cfs_rq:/.load_avg.max
     12556 ± 50%     -99.9%      17.47 ±159%  sched_debug.cfs_rq:/.max_vruntime.avg
    766470 ± 38%     -99.7%       2236 ±159%  sched_debug.cfs_rq:/.max_vruntime.max
     93742 ± 44%     -99.8%     196.88 ±159%  sched_debug.cfs_rq:/.max_vruntime.stddev
    998821 ± 10%     -90.9%      90606 ± 16%  sched_debug.cfs_rq:/.min_vruntime.avg
   1154960 ± 10%     -88.8%     128907 ± 11%  sched_debug.cfs_rq:/.min_vruntime.max
    874022 ± 13%     -92.2%      67934 ± 21%  sched_debug.cfs_rq:/.min_vruntime.min
     49122 ± 17%     -80.4%       9626 ± 13%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.24 ±  6%     -70.0%       0.07 ± 32%  sched_debug.cfs_rq:/.nr_running.avg
      0.42 ±  2%     -40.3%       0.25 ± 14%  sched_debug.cfs_rq:/.nr_running.stddev
    269.97 ±  2%     -53.3%     126.18 ± 19%  sched_debug.cfs_rq:/.runnable_avg.avg
      1166 ±  4%     +24.6%       1453 ±  4%  sched_debug.cfs_rq:/.runnable_avg.max
    116926 ± 27%     -84.1%      18583 ± 50%  sched_debug.cfs_rq:/.spread0.max
   -164017           -74.1%     -42414        sched_debug.cfs_rq:/.spread0.min
     49129 ± 17%     -80.4%       9632 ± 13%  sched_debug.cfs_rq:/.spread0.stddev
    269.72 ±  2%     -61.9%     102.78 ± 15%  sched_debug.cfs_rq:/.util_avg.avg
      1166 ±  4%      -7.0%       1084 ±  3%  sched_debug.cfs_rq:/.util_avg.max
    323.32 ±  2%     -32.6%     217.82 ± 13%  sched_debug.cfs_rq:/.util_avg.stddev
    146.06 ±  6%     -77.0%      33.66 ± 44%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    989.44           -13.3%     857.96 ±  8%  sched_debug.cfs_rq:/.util_est_enqueued.max
    276.89 ±  2%     -50.8%     136.28 ± 23%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    574938 ±  3%     +26.2%     725536 ±  3%  sched_debug.cpu.avg_idle.avg
      3945 ±  3%    +867.5%      38177 ± 80%  sched_debug.cpu.avg_idle.min
    406249 ±  2%     -21.9%     317312 ±  6%  sched_debug.cpu.avg_idle.stddev
      3.57 ±  5%     +54.4%       5.51 ±  6%  sched_debug.cpu.clock.stddev
      1612 ±  2%     -11.4%       1429        sched_debug.cpu.clock_task.stddev
      1391 ±  5%     -69.5%     424.83 ± 33%  sched_debug.cpu.curr->pid.avg
      8941 ±  4%     -17.6%       7367 ±  4%  sched_debug.cpu.curr->pid.max
      2392 ±  2%     -39.0%       1459 ± 14%  sched_debug.cpu.curr->pid.stddev
      0.00 ±  6%     -27.8%       0.00 ±  7%  sched_debug.cpu.next_balance.stddev
      0.24 ±  5%     -56.8%       0.10 ± 37%  sched_debug.cpu.nr_running.avg
      1.06 ±  7%     +86.8%       1.97 ±  3%  sched_debug.cpu.nr_running.max
   3581710 ± 11%     -84.7%     549192 ± 15%  sched_debug.cpu.nr_switches.avg
   4987997 ± 13%     -85.8%     708118 ± 15%  sched_debug.cpu.nr_switches.max
   2372208 ± 19%     -82.6%     411758 ± 18%  sched_debug.cpu.nr_switches.min
    497412 ± 10%     -88.1%      59277 ± 14%  sched_debug.cpu.nr_switches.stddev
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.avg
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.max
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.min
     25.06           -64.2%       8.97 ± 18%  perf-stat.i.MPKI
 2.221e+10           -67.9%   7.14e+09 ± 22%  perf-stat.i.branch-instructions
 1.197e+08           -59.5%   48489409 ± 32%  perf-stat.i.branch-misses
      1.48 ±  3%      +1.5        2.96 ± 20%  perf-stat.i.cache-miss-rate%
  39815902 ±  3%     -78.9%    8385204 ± 16%  perf-stat.i.cache-misses
 2.787e+09           -88.6%  3.168e+08 ± 30%  perf-stat.i.cache-references
   3386965           -82.4%     596733 ± 23%  perf-stat.i.context-switches
      1.17           -19.7%       0.94 ±  4%  perf-stat.i.cpi
 1.291e+11           -74.7%  3.267e+10 ± 21%  perf-stat.i.cpu-cycles
    744.79          +888.7%       7363 ± 16%  perf-stat.i.cpu-migrations
      3317 ±  2%     +18.6%       3933 ±  5%  perf-stat.i.cycles-between-cache-misses
 3.178e+10           -67.6%  1.029e+10 ± 22%  perf-stat.i.dTLB-loads
 1.882e+10           -67.8%  6.057e+09 ± 22%  perf-stat.i.dTLB-stores
  1.11e+11           -67.7%  3.585e+10 ± 22%  perf-stat.i.instructions
      0.86           +24.8%       1.07 ±  4%  perf-stat.i.ipc
      1.01           -74.7%       0.26 ± 21%  perf-stat.i.metric.GHz
    590.56           -68.5%     185.89 ± 22%  perf-stat.i.metric.M/sec
      4085           -19.2%       3300 ±  6%  perf-stat.i.minor-faults
     91.59            -1.9       89.65        perf-stat.i.node-load-miss-rate%
   3843758 ±  8%     -56.4%    1676424 ± 17%  perf-stat.i.node-load-misses
    345198 ±  4%     -42.8%     197478 ± 17%  perf-stat.i.node-loads
     70.70           -31.5       39.23 ±  6%  perf-stat.i.node-store-miss-rate%
   3113677 ±  4%     -64.0%    1121141 ± 17%  perf-stat.i.node-store-misses
      4085           -19.2%       3300 ±  6%  perf-stat.i.page-faults
     25.10           -65.2%       8.73 ± 16%  perf-stat.overall.MPKI
      1.43 ±  3%      +1.4        2.79 ± 17%  perf-stat.overall.cache-miss-rate%
      1.16           -21.4%       0.91 ±  4%  perf-stat.overall.cpi
      3244 ±  2%     +19.3%       3871 ±  5%  perf-stat.overall.cycles-between-cache-misses
      0.86           +27.4%       1.10 ±  4%  perf-stat.overall.ipc
     91.72            -2.3       89.42        perf-stat.overall.node-load-miss-rate%
     68.26           -29.8       38.51 ±  6%  perf-stat.overall.node-store-miss-rate%
     18276            -5.2%      17333        perf-stat.overall.path-length
 2.214e+10           -67.8%  7.118e+09 ± 22%  perf-stat.ps.branch-instructions
 1.192e+08           -59.5%   48334433 ± 32%  perf-stat.ps.branch-misses
  39676032 ±  3%     -78.9%    8357755 ± 16%  perf-stat.ps.cache-misses
 2.777e+09           -88.6%  3.158e+08 ± 30%  perf-stat.ps.cache-references
   3374855           -82.4%     594908 ± 23%  perf-stat.ps.context-switches
 1.286e+11           -74.7%  3.257e+10 ± 21%  perf-stat.ps.cpu-cycles
    742.42          +888.6%       7339 ± 16%  perf-stat.ps.cpu-migrations
 3.166e+10           -67.6%  1.026e+10 ± 22%  perf-stat.ps.dTLB-loads
 1.876e+10           -67.8%  6.038e+09 ± 22%  perf-stat.ps.dTLB-stores
 1.106e+11           -67.7%  3.574e+10 ± 22%  perf-stat.ps.instructions
      4069           -19.2%       3290 ±  6%  perf-stat.ps.minor-faults
   3830257 ±  8%     -56.4%    1670802 ± 17%  perf-stat.ps.node-load-misses
    344097 ±  4%     -42.8%     196864 ± 17%  perf-stat.ps.node-loads
   3102660 ±  4%     -64.0%    1117149 ± 17%  perf-stat.ps.node-store-misses
      4069           -19.2%       3290 ±  6%  perf-stat.ps.page-faults
  3.36e+13           -67.8%  1.082e+13 ± 22%  perf-stat.total.instructions
     22.54            -9.4       13.14 ±  5%  perf-profile.calltrace.cycles-pp.sctp_recvmsg.inet_recvmsg.____sys_recvmsg.___sys_recvmsg.__sys_recvmsg
     22.62            -9.4       13.26 ±  5%  perf-profile.calltrace.cycles-pp.inet_recvmsg.____sys_recvmsg.___sys_recvmsg.__sys_recvmsg.do_syscall_64
     23.55            -9.2       14.36 ±  5%  perf-profile.calltrace.cycles-pp.____sys_recvmsg.___sys_recvmsg.__sys_recvmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe
     24.92            -9.0       15.96 ±  4%  perf-profile.calltrace.cycles-pp.___sys_recvmsg.__sys_recvmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvmsg
     25.25            -8.9       16.31 ±  4%  perf-profile.calltrace.cycles-pp.__sys_recvmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvmsg
     26.64            -8.9       17.72 ±  4%  perf-profile.calltrace.cycles-pp.recvmsg
     25.74            -8.9       16.87 ±  4%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.recvmsg
     25.56            -8.8       16.74 ±  4%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvmsg
      8.60            -8.6        0.00        perf-profile.calltrace.cycles-pp.intel_idle_irq.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      8.45            -8.5        0.00        perf-profile.calltrace.cycles-pp.mwait_idle_with_hints.intel_idle_irq.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
     10.10            -4.9        5.19 ±  3%  perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.sctp_recvmsg.inet_recvmsg.____sys_recvmsg
     10.13            -4.9        5.24 ±  3%  perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.sctp_recvmsg.inet_recvmsg.____sys_recvmsg.___sys_recvmsg
      8.62            -4.3        4.35 ±  3%  perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.sctp_recvmsg
      8.58            -4.3        4.30 ±  3%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter
      8.87            -4.2        4.66 ±  3%  perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.sctp_recvmsg.inet_recvmsg
      4.71            -2.5        2.22 ±  4%  perf-profile.calltrace.cycles-pp.kfree_skb_reason.sctp_recvmsg.inet_recvmsg.____sys_recvmsg.___sys_recvmsg
      3.72 ±  2%      -1.9        1.83 ±  4%  perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages.__kmalloc_large_node.__kmalloc_node_track_caller.kmalloc_reserve
      4.19 ±  2%      -1.8        2.36 ±  4%  perf-profile.calltrace.cycles-pp.__alloc_pages.__kmalloc_large_node.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb
      2.89 ±  2%      -1.8        1.09 ±  5%  perf-profile.calltrace.cycles-pp.skb_release_data.kfree_skb_reason.sctp_recvmsg.inet_recvmsg.____sys_recvmsg
     10.94            -1.8        9.18 ±  4%  perf-profile.calltrace.cycles-pp.sctp_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog
     11.10            -1.7        9.37 ±  4%  perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll
     11.08            -1.7        9.35 ±  3%  perf-profile.calltrace.cycles-pp.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND
     11.12            -1.7        9.44 ±  4%  perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action
     13.03            -1.6       11.41 ±  3%  perf-profile.calltrace.cycles-pp.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND.sctp_sendmsg_to_asoc
     11.43            -1.6        9.85 ±  4%  perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action.__softirqentry_text_start
     11.76            -1.5       10.22 ±  4%  perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq.__local_bh_enable_ip.ip_finish_output2
     11.65            -1.5       10.11 ±  4%  perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.__softirqentry_text_start.do_softirq.__local_bh_enable_ip
     11.62            -1.5       10.10 ±  4%  perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.__softirqentry_text_start.do_softirq
     12.02            -1.5       10.50 ±  4%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq.__local_bh_enable_ip.ip_finish_output2.__ip_queue_xmit
     12.10            -1.5       10.59 ±  4%  perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.__ip_queue_xmit.sctp_packet_transmit
     12.16            -1.5       10.67 ±  4%  perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.__ip_queue_xmit.sctp_packet_transmit.sctp_outq_flush
      2.44 ±  2%      -1.4        1.00 ± 17%  perf-profile.calltrace.cycles-pp.__alloc_skb.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm
     12.44            -1.4       11.00 ±  4%  perf-profile.calltrace.cycles-pp.ip_finish_output2.__ip_queue_xmit.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter
     14.88            -1.4       13.45 ±  3%  perf-profile.calltrace.cycles-pp.sctp_primitive_SEND.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg.____sys_sendmsg
     12.62            -1.4       11.21 ±  4%  perf-profile.calltrace.cycles-pp.__ip_queue_xmit.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm
      8.62            -1.4        7.21 ±  4%  perf-profile.calltrace.cycles-pp.sctp_do_sm.sctp_assoc_bh_rcv.sctp_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
      2.17 ±  2%      -1.4        0.79 ±  4%  perf-profile.calltrace.cycles-pp.kmalloc_reserve.__alloc_skb.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter
      2.16 ±  2%      -1.4        0.78 ±  4%  perf-profile.calltrace.cycles-pp.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb.sctp_packet_transmit.sctp_outq_flush
      2.15 ±  2%      -1.4        0.77 ±  4%  perf-profile.calltrace.cycles-pp.__kmalloc_large_node.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb.sctp_packet_transmit
      9.20            -1.4        7.83 ±  4%  perf-profile.calltrace.cycles-pp.sctp_assoc_bh_rcv.sctp_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core
      5.66            -1.3        4.33 ±  5%  perf-profile.calltrace.cycles-pp.sctp_ulpq_tail_data.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_rcv
      2.18 ±  2%      -1.3        0.87 ± 24%  perf-profile.calltrace.cycles-pp.rmqueue.get_page_from_freelist.__alloc_pages.__kmalloc_large_node.__kmalloc_node_track_caller
      7.72            -1.3        6.46 ±  5%  perf-profile.calltrace.cycles-pp.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_rcv.ip_protocol_deliver_rcu
     10.04            -1.2        8.80 ±  3%  perf-profile.calltrace.cycles-pp.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND.sctp_sendmsg_to_asoc.sctp_sendmsg
      2.33            -1.2        1.14 ±  5%  perf-profile.calltrace.cycles-pp.sctp_ulpq_tail_event.sctp_ulpq_tail_data.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv
      9.78            -1.1        8.67 ±  3%  perf-profile.calltrace.cycles-pp.sctp_do_sm.sctp_primitive_SEND.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg
      2.14            -1.1        1.08 ±  5%  perf-profile.calltrace.cycles-pp.sctp_data_ready.sctp_ulpq_tail_event.sctp_ulpq_tail_data.sctp_cmd_interpreter.sctp_do_sm
      4.22 ±  2%      -0.9        3.28 ±  4%  perf-profile.calltrace.cycles-pp.sctp_chunk_put.sctp_outq_sack.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv
     19.14            -0.9       18.21 ±  3%  perf-profile.calltrace.cycles-pp.sctp_backlog_rcv.__release_sock.release_sock.sctp_sendmsg.sock_sendmsg
      3.72 ±  2%      -0.8        2.90 ±  4%  perf-profile.calltrace.cycles-pp.consume_skb.sctp_chunk_put.sctp_outq_sack.sctp_cmd_interpreter.sctp_do_sm
      1.77            -0.8        1.00 ±  4%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sctp_data_ready.sctp_ulpq_tail_event.sctp_ulpq_tail_data.sctp_cmd_interpreter
      1.69            -0.7        0.96 ±  4%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sctp_data_ready.sctp_ulpq_tail_event.sctp_ulpq_tail_data
      5.43 ±  2%      -0.7        4.72 ±  3%  perf-profile.calltrace.cycles-pp.sctp_outq_sack.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_backlog_rcv
      1.35            -0.7        0.65 ±  9%  perf-profile.calltrace.cycles-pp.__sk_mem_raise_allocated.__sk_mem_schedule.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg
      1.80            -0.7        1.10 ±  4%  perf-profile.calltrace.cycles-pp.skb_release_head_state.kfree_skb_reason.sctp_recvmsg.inet_recvmsg.____sys_recvmsg
      1.38            -0.7        0.68 ±  9%  perf-profile.calltrace.cycles-pp.__sk_mem_schedule.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg.____sys_sendmsg
      0.98 ±  2%      -0.6        0.35 ± 70%  perf-profile.calltrace.cycles-pp.mem_cgroup_charge_skmem.__sk_mem_raise_allocated.__sk_mem_schedule.sctp_sendmsg_to_asoc.sctp_sendmsg
      1.29 ±  2%      -0.6        0.69 ±  5%  perf-profile.calltrace.cycles-pp.__sk_mem_reduce_allocated.skb_release_head_state.kfree_skb_reason.sctp_recvmsg.inet_recvmsg
      2.42 ±  2%      -0.6        1.84 ±  4%  perf-profile.calltrace.cycles-pp.__kmalloc_large_node.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb._sctp_make_chunk
      2.48 ±  2%      -0.6        1.92 ±  4%  perf-profile.calltrace.cycles-pp.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb._sctp_make_chunk.sctp_make_datafrag_empty
      2.52 ±  2%      -0.6        1.97 ±  4%  perf-profile.calltrace.cycles-pp.kmalloc_reserve.__alloc_skb._sctp_make_chunk.sctp_make_datafrag_empty.sctp_datamsg_from_user
     13.06            -0.5       12.53 ±  3%  perf-profile.calltrace.cycles-pp.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv
      1.04 ±  2%      -0.5        0.54 ±  5%  perf-profile.calltrace.cycles-pp.__sk_mem_reduce_allocated.sctp_wfree.skb_release_head_state.consume_skb.sctp_chunk_put
      1.42            -0.5        0.94 ±  4%  perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sctp_data_ready.sctp_ulpq_tail_event
      1.40            -0.5        0.92 ±  4%  perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sctp_data_ready
      1.58 ±  2%      -0.4        1.14 ±  5%  perf-profile.calltrace.cycles-pp.sctp_wfree.skb_release_head_state.consume_skb.sctp_chunk_put.sctp_outq_sack
      1.74 ±  2%      -0.4        1.32 ±  4%  perf-profile.calltrace.cycles-pp.skb_release_head_state.consume_skb.sctp_chunk_put.sctp_outq_sack.sctp_cmd_interpreter
      3.02 ±  2%      -0.4        2.61 ±  4%  perf-profile.calltrace.cycles-pp.sctp_ulpevent_make_rcvmsg.sctp_ulpq_tail_data.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv
      3.27            -0.4        2.87 ±  4%  perf-profile.calltrace.cycles-pp.__alloc_skb._sctp_make_chunk.sctp_make_datafrag_empty.sctp_datamsg_from_user.sctp_sendmsg_to_asoc
      1.42 ±  2%      -0.4        1.04 ±  5%  perf-profile.calltrace.cycles-pp.free_unref_page.skb_release_data.consume_skb.sctp_chunk_put.sctp_outq_sack
      1.92 ±  2%      -0.4        1.55 ±  5%  perf-profile.calltrace.cycles-pp.skb_release_data.consume_skb.sctp_chunk_put.sctp_outq_sack.sctp_cmd_interpreter
      0.91            -0.3        0.60 ±  4%  perf-profile.calltrace.cycles-pp.sctp_outq_select_transport.sctp_outq_flush_data.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm
      0.75            -0.3        0.47 ± 44%  perf-profile.calltrace.cycles-pp.sctp_chunk_put.sctp_ulpevent_free.sctp_recvmsg.inet_recvmsg.____sys_recvmsg
      3.92            -0.2        3.70 ±  4%  perf-profile.calltrace.cycles-pp._sctp_make_chunk.sctp_make_datafrag_empty.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg
      1.38            -0.2        1.23 ±  3%  perf-profile.calltrace.cycles-pp.sctp_ulpevent_free.sctp_recvmsg.inet_recvmsg.____sys_recvmsg.___sys_recvmsg
      0.52 ±  2%      +0.1        0.64 ±  4%  perf-profile.calltrace.cycles-pp.sock_kmalloc.____sys_sendmsg.___sys_sendmsg.__sys_sendmsg.do_syscall_64
      0.59 ±  3%      +0.1        0.74 ±  5%  perf-profile.calltrace.cycles-pp.__import_iovec.import_iovec.copy_msghdr_from_user.___sys_sendmsg.__sys_sendmsg
      0.58 ±  4%      +0.2        0.73 ±  5%  perf-profile.calltrace.cycles-pp.__import_iovec.import_iovec.copy_msghdr_from_user.___sys_recvmsg.__sys_recvmsg
      0.61 ±  2%      +0.2        0.77 ±  4%  perf-profile.calltrace.cycles-pp.import_iovec.copy_msghdr_from_user.___sys_sendmsg.__sys_sendmsg.do_syscall_64
      0.62 ±  3%      +0.2        0.78 ±  5%  perf-profile.calltrace.cycles-pp.__copy_msghdr.copy_msghdr_from_user.___sys_sendmsg.__sys_sendmsg.do_syscall_64
      0.60 ±  4%      +0.2        0.76 ±  5%  perf-profile.calltrace.cycles-pp.import_iovec.copy_msghdr_from_user.___sys_recvmsg.__sys_recvmsg.do_syscall_64
      0.62 ±  5%      +0.2        0.78 ±  9%  perf-profile.calltrace.cycles-pp.__rhashtable_lookup.sctp_epaddr_lookup_transport.sctp_endpoint_lookup_assoc.sctp_sendmsg.sock_sendmsg
      0.79            +0.2        0.97 ±  4%  perf-profile.calltrace.cycles-pp.sctp_outq_flush_data.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv
      0.68 ±  4%      +0.2        0.86 ±  8%  perf-profile.calltrace.cycles-pp.sctp_epaddr_lookup_transport.sctp_endpoint_lookup_assoc.sctp_sendmsg.sock_sendmsg.____sys_sendmsg
      0.72 ±  4%      +0.2        0.92 ±  8%  perf-profile.calltrace.cycles-pp.sctp_endpoint_lookup_assoc.sctp_sendmsg.sock_sendmsg.____sys_sendmsg.___sys_sendmsg
      0.45 ± 44%      +0.2        0.67 ±  4%  perf-profile.calltrace.cycles-pp.move_addr_to_kernel.__copy_msghdr.copy_msghdr_from_user.___sys_sendmsg.__sys_sendmsg
      1.20            +0.2        1.43 ±  3%  perf-profile.calltrace.cycles-pp.copy_msghdr_from_user.___sys_recvmsg.__sys_recvmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.34 ± 70%      +0.3        0.64 ±  7%  perf-profile.calltrace.cycles-pp.sctp_chunkify._sctp_make_chunk.sctp_make_datafrag_empty.sctp_datamsg_from_user.sctp_sendmsg_to_asoc
      1.82 ±  2%      +0.4        2.18 ±  4%  perf-profile.calltrace.cycles-pp.schedule_timeout.sctp_skb_recv_datagram.sctp_recvmsg.inet_recvmsg.____sys_recvmsg
      0.26 ±100%      +0.4        0.64 ±  6%  perf-profile.calltrace.cycles-pp.iovec_from_user.__import_iovec.import_iovec.copy_msghdr_from_user.___sys_sendmsg
      1.69 ±  2%      +0.4        2.14 ±  3%  perf-profile.calltrace.cycles-pp.copy_msghdr_from_user.___sys_sendmsg.__sys_sendmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.09 ±223%      +0.5        0.62 ±  5%  perf-profile.calltrace.cycles-pp.iovec_from_user.__import_iovec.import_iovec.copy_msghdr_from_user.___sys_recvmsg
      0.00            +0.6        0.56 ±  6%  perf-profile.calltrace.cycles-pp.release_sock.sctp_recvmsg.inet_recvmsg.____sys_recvmsg.___sys_recvmsg
      0.00            +0.6        0.59 ±  5%  perf-profile.calltrace.cycles-pp.sctp_chunk_free.sctp_outq_sack.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv
      0.00            +0.6        0.62 ±  3%  perf-profile.calltrace.cycles-pp.__schedule.__cond_resched.__release_sock.release_sock.sctp_sendmsg
      0.00            +0.6        0.62 ±  3%  perf-profile.calltrace.cycles-pp.update_sg_lb_stats.update_sd_lb_stats.find_busiest_group.load_balance.newidle_balance
      0.00            +0.6        0.62 ±  5%  perf-profile.calltrace.cycles-pp.sctp_addto_chunk.sctp_make_datafrag_empty.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg
      1.50 ±  2%      +0.6        2.13 ±  3%  perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.sctp_skb_recv_datagram.sctp_recvmsg.inet_recvmsg
      1.44 ±  2%      +0.6        2.08 ±  3%  perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.sctp_skb_recv_datagram.sctp_recvmsg
      0.00            +0.7        0.67 ±  3%  perf-profile.calltrace.cycles-pp.update_sd_lb_stats.find_busiest_group.load_balance.newidle_balance.pick_next_task_fair
      0.00            +0.7        0.67 ±  4%  perf-profile.calltrace.cycles-pp.find_busiest_group.load_balance.newidle_balance.pick_next_task_fair.__schedule
      0.00            +0.7        0.68 ± 14%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry
      0.00            +0.7        0.68 ±  3%  perf-profile.calltrace.cycles-pp.__cond_resched.__release_sock.release_sock.sctp_sendmsg.sock_sendmsg
      0.00            +0.7        0.73 ± 18%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      0.00            +0.7        0.74 ±  4%  perf-profile.calltrace.cycles-pp.load_balance.newidle_balance.pick_next_task_fair.__schedule.schedule
      0.00            +0.8        0.79 ±  4%  perf-profile.calltrace.cycles-pp.newidle_balance.pick_next_task_fair.__schedule.schedule.schedule_timeout
      3.36            +0.9        4.22 ±  3%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin._copy_from_iter.sctp_user_addto_chunk.sctp_datamsg_from_user
      3.40            +0.9        4.27 ±  3%  perf-profile.calltrace.cycles-pp.copyin._copy_from_iter.sctp_user_addto_chunk.sctp_datamsg_from_user.sctp_sendmsg_to_asoc
      0.68 ±  2%      +0.9        1.55 ± 19%  perf-profile.calltrace.cycles-pp.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
      0.00            +0.9        0.90 ± 15%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.00            +0.9        0.93 ±  3%  perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.schedule_timeout.sctp_skb_recv_datagram
      3.65            +0.9        4.59 ±  3%  perf-profile.calltrace.cycles-pp._copy_from_iter.sctp_user_addto_chunk.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg
      4.08            +1.1        5.13 ±  3%  perf-profile.calltrace.cycles-pp.sctp_user_addto_chunk.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg
      0.00            +1.1        1.14 ± 14%  perf-profile.calltrace.cycles-pp.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.00            +1.4        1.36 ± 26%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      9.56            +1.4       10.94 ±  3%  perf-profile.calltrace.cycles-pp.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg.____sys_sendmsg
      0.00            +1.5        1.49 ± 30%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      0.00            +1.7        1.66 ± 27%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      0.00            +2.5        2.46 ± 26%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00            +3.5        3.48 ± 19%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.00            +3.6        3.57 ± 20%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.66 ± 10%      +4.9        5.60 ± 18%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
      0.75 ±  9%      +5.9        6.66 ± 17%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     17.20            +6.5       23.70 ± 13%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     17.19            +6.5       23.70 ± 13%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     17.15            +6.5       23.67 ± 13%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     17.32            +6.6       23.88 ± 13%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     13.07            +7.7       20.79 ± 12%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
     13.17            +8.2       21.38 ± 12%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
     14.06            +9.2       23.24 ± 13%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      3.18 ±  4%     +10.6       13.78 ± 12%  perf-profile.calltrace.cycles-pp.mwait_idle_with_hints.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
      3.20 ±  4%     +10.6       13.84 ± 12%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     22.86            -9.4       13.47 ±  5%  perf-profile.children.cycles-pp.sctp_recvmsg
     22.63            -9.4       13.27 ±  5%  perf-profile.children.cycles-pp.inet_recvmsg
     23.57            -9.2       14.38 ±  5%  perf-profile.children.cycles-pp.____sys_recvmsg
     24.96            -9.0       16.00 ±  4%  perf-profile.children.cycles-pp.___sys_recvmsg
     25.27            -8.9       16.34 ±  4%  perf-profile.children.cycles-pp.__sys_recvmsg
     26.87            -8.9       18.01 ±  4%  perf-profile.children.cycles-pp.recvmsg
      8.67            -8.7        0.00        perf-profile.children.cycles-pp.intel_idle_irq
     79.11            -7.0       72.07 ±  4%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     78.72            -7.0       71.69 ±  4%  perf-profile.children.cycles-pp.do_syscall_64
     10.11            -4.9        5.20 ±  3%  perf-profile.children.cycles-pp.__skb_datagram_iter
     10.14            -4.9        5.25 ±  3%  perf-profile.children.cycles-pp.skb_copy_datagram_iter
      8.64            -4.3        4.36 ±  3%  perf-profile.children.cycles-pp.copyout
      8.88            -4.2        4.68 ±  3%  perf-profile.children.cycles-pp._copy_to_iter
     13.72            -2.9       10.78 ±  3%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      4.73            -2.5        2.25 ±  4%  perf-profile.children.cycles-pp.kfree_skb_reason
     33.66            -2.3       31.40 ±  3%  perf-profile.children.cycles-pp.sctp_cmd_interpreter
     33.91            -2.2       31.71 ±  3%  perf-profile.children.cycles-pp.sctp_do_sm
      4.98            -2.1        2.88 ±  4%  perf-profile.children.cycles-pp.skb_release_data
      2.42 ±  2%      -2.0        0.45 ± 71%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     24.48            -1.9       22.58 ±  3%  perf-profile.children.cycles-pp.sctp_packet_transmit
     27.60            -1.6       25.95 ±  3%  perf-profile.children.cycles-pp.sctp_outq_flush
      3.79            -1.6        2.16 ±  4%  perf-profile.children.cycles-pp.get_page_from_freelist
      4.66            -1.6        3.04 ±  3%  perf-profile.children.cycles-pp.__kmalloc_large_node
      4.81            -1.6        3.25 ±  3%  perf-profile.children.cycles-pp.__kmalloc_node_track_caller
      4.87            -1.5        3.33 ±  3%  perf-profile.children.cycles-pp.kmalloc_reserve
      4.28 ±  2%      -1.5        2.76 ±  3%  perf-profile.children.cycles-pp.__alloc_pages
     15.05            -1.5       13.56 ±  4%  perf-profile.children.cycles-pp.sctp_primitive_SEND
      2.22 ±  2%      -1.5        0.75 ±  8%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
     11.15            -1.4        9.72 ±  4%  perf-profile.children.cycles-pp.sctp_rcv
      2.74 ±  2%      -1.4        1.34 ±  5%  perf-profile.children.cycles-pp.free_unref_page
     11.28            -1.4        9.89 ±  4%  perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
      6.14            -1.4        4.76 ±  3%  perf-profile.children.cycles-pp.__alloc_skb
     11.30            -1.4        9.96 ±  4%  perf-profile.children.cycles-pp.ip_local_deliver_finish
      5.81            -1.3        4.49 ±  5%  perf-profile.children.cycles-pp.sctp_ulpq_tail_data
      1.36 ±  2%      -1.3        0.06 ±  6%  perf-profile.children.cycles-pp.__slab_free
      2.67 ±  2%      -1.3        1.40 ±  4%  perf-profile.children.cycles-pp.rmqueue
     11.61            -1.2       10.38 ±  4%  perf-profile.children.cycles-pp.__netif_receive_skb_one_core
     12.28            -1.2       11.06 ±  4%  perf-profile.children.cycles-pp.do_softirq
     11.95            -1.2       10.76 ±  4%  perf-profile.children.cycles-pp.net_rx_action
     11.83            -1.2       10.64 ±  4%  perf-profile.children.cycles-pp.__napi_poll
     11.81            -1.2       10.63 ±  4%  perf-profile.children.cycles-pp.process_backlog
      5.74            -1.2        4.56 ±  4%  perf-profile.children.cycles-pp.sctp_chunk_put
     12.49            -1.2       11.32 ±  4%  perf-profile.children.cycles-pp.__local_bh_enable_ip
      1.62 ±  2%      -1.2        0.45 ±  8%  perf-profile.children.cycles-pp._raw_spin_lock
      2.60            -1.2        1.43 ±  7%  perf-profile.children.cycles-pp.__sk_mem_raise_allocated
      3.83            -1.2        2.66 ±  4%  perf-profile.children.cycles-pp.skb_release_head_state
      2.66            -1.2        1.51 ±  6%  perf-profile.children.cycles-pp.__sk_mem_schedule
     24.83            -1.1       23.72 ±  4%  perf-profile.children.cycles-pp.sctp_assoc_bh_rcv
      2.37            -1.1        1.27 ±  5%  perf-profile.children.cycles-pp.__sk_mem_reduce_allocated
     12.86            -1.1       11.80 ±  4%  perf-profile.children.cycles-pp.ip_finish_output2
      1.10            -1.1        0.04 ± 44%  perf-profile.children.cycles-pp.sched_ttwu_pending
     13.14            -1.0       12.13 ±  4%  perf-profile.children.cycles-pp.__ip_queue_xmit
      2.38            -1.0        1.41 ±  5%  perf-profile.children.cycles-pp.sctp_ulpq_tail_event
     19.52            -1.0       18.56 ±  3%  perf-profile.children.cycles-pp.sctp_backlog_rcv
      2.02            -0.9        1.09 ±  5%  perf-profile.children.cycles-pp.mem_cgroup_charge_skmem
      2.17            -0.9        1.25 ±  5%  perf-profile.children.cycles-pp.sctp_data_ready
      4.27            -0.9        3.40 ±  4%  perf-profile.children.cycles-pp.consume_skb
      0.94 ±  2%      -0.8        0.12 ± 17%  perf-profile.children.cycles-pp.schedule_idle
      1.10 ±  3%      -0.8        0.34 ±  7%  perf-profile.children.cycles-pp.try_charge_memcg
      1.80            -0.7        1.07 ±  4%  perf-profile.children.cycles-pp.__wake_up_common_lock
      1.72            -0.7        1.02 ±  4%  perf-profile.children.cycles-pp.__wake_up_common
      5.54            -0.6        4.92 ±  3%  perf-profile.children.cycles-pp.sctp_outq_sack
      1.21 ±  3%      -0.6        0.65 ±  7%  perf-profile.children.cycles-pp.__mod_memcg_state
      1.06 ±  2%      -0.5        0.51 ±  9%  perf-profile.children.cycles-pp.__free_pages_ok
      0.90 ±  2%      -0.5        0.44 ±  6%  perf-profile.children.cycles-pp.refill_stock
     12.34            -0.5       11.88 ±  2%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.62 ±  3%      -0.5        0.17 ± 20%  perf-profile.children.cycles-pp.ipv4_dst_check
      1.45            -0.5        1.00 ±  3%  perf-profile.children.cycles-pp.autoremove_wake_function
      1.09            -0.4        0.65 ±  6%  perf-profile.children.cycles-pp.sctp_packet_config
      1.43            -0.4        0.99 ±  3%  perf-profile.children.cycles-pp.try_to_wake_up
      0.77 ±  2%      -0.4        0.33 ±  7%  perf-profile.children.cycles-pp.free_unref_page_commit
      1.24            -0.4        0.82 ±  5%  perf-profile.children.cycles-pp.sctp_outq_select_transport
      1.33            -0.4        0.92 ±  5%  perf-profile.children.cycles-pp.kmem_cache_free
      1.62            -0.4        1.22 ±  4%  perf-profile.children.cycles-pp.sctp_wfree
      3.15            -0.4        2.76 ±  5%  perf-profile.children.cycles-pp.sctp_ulpevent_make_rcvmsg
      0.40            -0.3        0.07 ±  6%  perf-profile.children.cycles-pp.select_task_rq
      0.46 ±  2%      -0.3        0.16 ±  7%  perf-profile.children.cycles-pp.sctp_inet_skb_msgname
      0.57 ±  2%      -0.3        0.29 ±  8%  perf-profile.children.cycles-pp.dequeue_entity
      0.33            -0.3        0.06 ±  9%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.49 ±  2%      -0.3        0.23 ±  8%  perf-profile.children.cycles-pp.__mod_node_page_state
      0.76 ±  3%      -0.2        0.51 ±  7%  perf-profile.children.cycles-pp.mem_cgroup_uncharge_skmem
      0.55 ±  3%      -0.2        0.33 ± 16%  perf-profile.children.cycles-pp.dst_release
      0.49            -0.2        0.28 ±  6%  perf-profile.children.cycles-pp.enqueue_entity
      0.38 ±  3%      -0.2        0.18 ±  9%  perf-profile.children.cycles-pp.__free_one_page
      4.26            -0.2        4.07 ±  4%  perf-profile.children.cycles-pp._sctp_make_chunk
      0.70 ±  2%      -0.2        0.51 ±  5%  perf-profile.children.cycles-pp.sctp_eat_data
      0.79 ±  2%      -0.2        0.61 ±  4%  perf-profile.children.cycles-pp.sctp_sf_eat_data_6_2
      0.25 ±  3%      -0.2        0.07 ± 11%  perf-profile.children.cycles-pp.prepare_to_wait_exclusive
      0.60 ±  2%      -0.2        0.42 ±  8%  perf-profile.children.cycles-pp.simple_copy_to_iter
      0.58 ±  4%      -0.2        0.42 ±  9%  perf-profile.children.cycles-pp.__copy_skb_header
      0.79 ±  3%      -0.2        0.64 ±  7%  perf-profile.children.cycles-pp.__skb_clone
      1.40            -0.1        1.26 ±  3%  perf-profile.children.cycles-pp.sctp_ulpevent_free
      0.76 ±  2%      -0.1        0.63 ±  5%  perf-profile.children.cycles-pp.sctp_addrs_lookup_transport
      0.79 ±  2%      -0.1        0.66 ±  5%  perf-profile.children.cycles-pp.__sctp_rcv_lookup
      0.45 ±  2%      -0.1        0.32 ±  4%  perf-profile.children.cycles-pp.free_large_kmalloc
      0.53 ±  2%      -0.1        0.40 ±  4%  perf-profile.children.cycles-pp.__zone_watermark_ok
      0.28            -0.1        0.16 ±  8%  perf-profile.children.cycles-pp.prepare_task_switch
      0.60            -0.1        0.49 ±  3%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.84            -0.1        0.72 ±  6%  perf-profile.children.cycles-pp.check_heap_object
      0.58            -0.1        0.49 ±  3%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.62 ±  2%      -0.1        0.52 ±  6%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.33            -0.1        0.24 ±  5%  perf-profile.children.cycles-pp.__switch_to
      0.43            -0.1        0.34 ±  4%  perf-profile.children.cycles-pp.skb_set_owner_w
      0.20            -0.1        0.12 ± 12%  perf-profile.children.cycles-pp.___perf_sw_event
      0.20 ±  2%      -0.1        0.13 ±  7%  perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
      0.13            -0.1        0.06 ±  9%  perf-profile.children.cycles-pp.perf_tp_event
      0.60            -0.1        0.53 ±  5%  perf-profile.children.cycles-pp.sctp_inq_pop
      0.12 ±  3%      -0.0        0.08 ±  9%  perf-profile.children.cycles-pp.__wrgsbase_inactive
      0.20 ±  3%      -0.0        0.16 ±  6%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.15 ±  3%      -0.0        0.11 ±  8%  perf-profile.children.cycles-pp.__build_skb_around
      0.10 ±  3%      -0.0        0.08 ±  6%  perf-profile.children.cycles-pp.check_new_pages
      0.05            +0.0        0.07 ±  7%  perf-profile.children.cycles-pp.sctp_sendmsg_check_sflags
      0.06 ±  7%      +0.0        0.08 ±  7%  perf-profile.children.cycles-pp.sctp_assoc_rwnd_decrease
      0.05 ±  7%      +0.0        0.07 ± 10%  perf-profile.children.cycles-pp.skb_pull
      0.10 ±  3%      +0.0        0.12 ±  8%  perf-profile.children.cycles-pp.validate_xmit_skb
      0.07 ±  5%      +0.0        0.09 ±  7%  perf-profile.children.cycles-pp.ip_send_check
      0.05            +0.0        0.07 ±  9%  perf-profile.children.cycles-pp.sctp_tsnmap_check
      0.08 ±  5%      +0.0        0.10 ±  4%  perf-profile.children.cycles-pp.sctp_validate_data
      0.14            +0.0        0.16 ±  3%  perf-profile.children.cycles-pp.__mod_lruvec_page_state
      0.09 ±  5%      +0.0        0.11 ±  9%  perf-profile.children.cycles-pp.syscall_enter_from_user_mode
      0.07 ±  7%      +0.0        0.09 ± 11%  perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
      0.09 ±  5%      +0.0        0.12 ±  8%  perf-profile.children.cycles-pp.__ip_local_out
      0.12 ±  4%      +0.0        0.14 ±  5%  perf-profile.children.cycles-pp.sctp_sock_rfree
      0.17 ±  2%      +0.0        0.20 ±  5%  perf-profile.children.cycles-pp.sctp_ulpq_order
      0.06 ± 11%      +0.0        0.09 ± 19%  perf-profile.children.cycles-pp.sctp_get_af_specific
      0.08 ±  4%      +0.0        0.11 ±  8%  perf-profile.children.cycles-pp.ip_rcv_core
      0.11 ±  6%      +0.0        0.14 ±  9%  perf-profile.children.cycles-pp.ip_local_out
      0.14 ±  5%      +0.0        0.16 ±  7%  perf-profile.children.cycles-pp.check_stack_object
      0.03 ± 70%      +0.0        0.06 ±  7%  perf-profile.children.cycles-pp.sctp_auth_send_cid
      0.10            +0.0        0.13 ±  9%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
      0.11 ±  3%      +0.0        0.14 ±  9%  perf-profile.children.cycles-pp.__netif_receive_skb_core
      0.30 ±  2%      +0.0        0.33 ±  6%  perf-profile.children.cycles-pp.sctp_transport_put
      0.07            +0.0        0.10 ±  7%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.16 ±  2%      +0.0        0.20 ±  5%  perf-profile.children.cycles-pp.__put_user_nocheck_8
      0.12 ±  5%      +0.0        0.16 ±  7%  perf-profile.children.cycles-pp.os_xsave
      0.04 ± 44%      +0.0        0.08 ± 41%  perf-profile.children.cycles-pp.inet_sendmsg
      0.12 ±  6%      +0.0        0.16 ±  7%  perf-profile.children.cycles-pp.sctp_cmp_addr_exact
      0.27 ±  2%      +0.0        0.31 ±  6%  perf-profile.children.cycles-pp.sctp_make_control
      0.25 ±  3%      +0.0        0.29 ±  6%  perf-profile.children.cycles-pp.sctp_outq_tail
      0.12 ±  4%      +0.0        0.16 ±  8%  perf-profile.children.cycles-pp.sctp_chunk_assign_ssn
      0.02 ±141%      +0.0        0.06 ± 11%  perf-profile.children.cycles-pp.update_min_vruntime
      0.05 ±  8%      +0.0        0.10 ±  8%  perf-profile.children.cycles-pp.check_preempt_curr
      0.11 ±  5%      +0.0        0.15 ± 22%  perf-profile.children.cycles-pp.sctp_hash_key
      0.08 ±  5%      +0.0        0.13 ±  9%  perf-profile.children.cycles-pp.__sctp_write_space
      0.24 ±  4%      +0.0        0.29 ±  3%  perf-profile.children.cycles-pp.skb_clone
      0.01 ±223%      +0.0        0.06 ±  9%  perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
      0.18 ±  2%      +0.0        0.23 ±  7%  perf-profile.children.cycles-pp.rcu_all_qs
      0.16 ±  3%      +0.0        0.20 ±  3%  perf-profile.children.cycles-pp.ip_rcv
      0.20            +0.0        0.25 ±  9%  perf-profile.children.cycles-pp.sctp_datamsg_destroy
      0.23 ±  3%      +0.1        0.28 ±  7%  perf-profile.children.cycles-pp.sctp_association_hold
      0.18 ±  5%      +0.1        0.24 ±  7%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.01 ±223%      +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.netif_skb_features
      0.34 ±  3%      +0.1        0.39 ±  5%  perf-profile.children.cycles-pp.sctp_sched_fcfs_dequeue
      0.00            +0.1        0.05 ± 13%  perf-profile.children.cycles-pp.sctp_ulpq_retrieve_ordered
      0.23 ±  4%      +0.1        0.29 ±  5%  perf-profile.children.cycles-pp.sctp_chunk_hold
      0.41 ±  4%      +0.1        0.46 ±  5%  perf-profile.children.cycles-pp.sock_recvmsg
      0.39            +0.1        0.45 ±  5%  perf-profile.children.cycles-pp.send_sctp_stream_1toMany
      0.06            +0.1        0.12 ± 10%  perf-profile.children.cycles-pp.reweight_entity
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.sctp_transport_burst_reset
      0.45            +0.1        0.51 ±  7%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.15 ±  2%      +0.1        0.21 ±  9%  perf-profile.children.cycles-pp.__mod_timer
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.asm_sysvec_call_function_single
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.error_entry
      0.17 ±  2%      +0.1        0.24 ±  6%  perf-profile.children.cycles-pp.__netif_rx
      0.15 ±  4%      +0.1        0.21 ±  6%  perf-profile.children.cycles-pp.enqueue_to_backlog
      0.16 ±  4%      +0.1        0.22 ±  6%  perf-profile.children.cycles-pp.netif_rx_internal
      0.27 ±  3%      +0.1        0.34 ±  8%  perf-profile.children.cycles-pp.sctp_sendmsg_parse
      0.00            +0.1        0.07 ± 16%  perf-profile.children.cycles-pp.perf_trace_sched_switch
      0.46 ±  2%      +0.1        0.53 ±  6%  perf-profile.children.cycles-pp.sctp_check_transmitted
      0.21 ±  2%      +0.1        0.28 ±  8%  perf-profile.children.cycles-pp.__put_user_nocheck_4
      0.12 ± 36%      +0.1        0.19 ± 25%  perf-profile.children.cycles-pp.start_kernel
      0.12 ± 36%      +0.1        0.19 ± 25%  perf-profile.children.cycles-pp.arch_call_rest_init
      0.12 ± 36%      +0.1        0.19 ± 25%  perf-profile.children.cycles-pp.rest_init
      0.24 ±  3%      +0.1        0.31 ±  4%  perf-profile.children.cycles-pp.kfree
      0.25 ±  2%      +0.1        0.33 ±  6%  perf-profile.children.cycles-pp.sctp_datamsg_put
      0.08 ±  4%      +0.1        0.16 ±  9%  perf-profile.children.cycles-pp.sctp_transport_reset_t3_rtx
      0.59            +0.1        0.66 ±  3%  perf-profile.children.cycles-pp.kmem_cache_alloc_node
      0.28 ±  3%      +0.1        0.36 ±  6%  perf-profile.children.cycles-pp.__genradix_ptr
      0.00            +0.1        0.08 ±  6%  perf-profile.children.cycles-pp.check_preempt_wakeup
      0.48 ±  2%      +0.1        0.56 ±  3%  perf-profile.children.cycles-pp.free_pcp_prepare
      0.33            +0.1        0.41 ±  6%  perf-profile.children.cycles-pp.kmalloc_trace
      0.30            +0.1        0.38 ±  5%  perf-profile.children.cycles-pp.sctp_association_put
      0.00            +0.1        0.09 ±  7%  perf-profile.children.cycles-pp.put_prev_entity
      0.01 ±223%      +0.1        0.10 ± 17%  perf-profile.children.cycles-pp.ret_from_fork
      0.01 ±223%      +0.1        0.10 ± 17%  perf-profile.children.cycles-pp.kthread
      0.03 ±100%      +0.1        0.12 ± 28%  perf-profile.children.cycles-pp.arch_cpu_idle_enter
      0.40            +0.1        0.49 ±  6%  perf-profile.children.cycles-pp.sctp_ulpevent_receive_data
      0.30            +0.1        0.40 ±  8%  perf-profile.children.cycles-pp.__virt_addr_valid
      0.00            +0.1        0.10 ± 17%  perf-profile.children.cycles-pp.tick_sched_do_timer
      0.42 ±  2%      +0.1        0.52 ±  5%  perf-profile.children.cycles-pp.__kmalloc
      0.33 ±  2%      +0.1        0.44 ±  8%  perf-profile.children.cycles-pp.__sctp_packet_append_chunk
      0.33 ±  2%      +0.1        0.43 ±  7%  perf-profile.children.cycles-pp.loopback_xmit
      0.27 ±  4%      +0.1        0.38 ±  7%  perf-profile.children.cycles-pp.update_curr
      0.00            +0.1        0.10 ±  7%  perf-profile.children.cycles-pp._find_next_and_bit
      0.01 ±223%      +0.1        0.11 ± 27%  perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
      0.26 ±  3%      +0.1        0.37 ±  7%  perf-profile.children.cycles-pp.sock_kfree_s
      0.04 ± 45%      +0.1        0.15 ± 35%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      0.00            +0.1        0.11 ± 21%  perf-profile.children.cycles-pp.__intel_pmu_enable_all
      0.11            +0.1        0.22 ± 29%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.28 ±  3%      +0.1        0.39 ±  6%  perf-profile.children.cycles-pp.__kmem_cache_free
      0.29 ±  2%      +0.1        0.40 ± 48%  perf-profile.children.cycles-pp.security_socket_sendmsg
      0.79 ±  2%      +0.1        0.91 ±  4%  perf-profile.children.cycles-pp.__copy_msghdr
      0.00            +0.1        0.11 ± 17%  perf-profile.children.cycles-pp.update_irq_load_avg
      0.36 ±  2%      +0.1        0.48 ±  4%  perf-profile.children.cycles-pp.__fdget
      0.34 ±  2%      +0.1        0.45 ±  8%  perf-profile.children.cycles-pp._raw_spin_trylock
      0.36 ±  2%      +0.1        0.48 ±  8%  perf-profile.children.cycles-pp.dev_hard_start_xmit
      0.43 ±  3%      +0.1        0.55 ±  6%  perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.02 ± 99%      +0.1        0.15 ± 33%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.54            +0.1        0.66 ±  5%  perf-profile.children.cycles-pp.sock_kmalloc
      0.33            +0.1        0.46 ±  3%  perf-profile.children.cycles-pp.read_tsc
      0.00            +0.1        0.12 ± 40%  perf-profile.children.cycles-pp.hrtimer_update_next_event
      0.44 ±  2%      +0.1        0.57 ±  4%  perf-profile.children.cycles-pp.sockfd_lookup_light
      0.00            +0.1        0.13 ± 42%  perf-profile.children.cycles-pp.wait_for_lsr
      0.50            +0.1        0.63 ±  4%  perf-profile.children.cycles-pp.sctp_chunk_free
      0.00            +0.1        0.13 ± 27%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.00            +0.1        0.13 ± 43%  perf-profile.children.cycles-pp.serial8250_console_write
      0.00            +0.1        0.14 ± 43%  perf-profile.children.cycles-pp.irq_work_single
      0.00            +0.1        0.14 ± 43%  perf-profile.children.cycles-pp.console_unlock
      0.00            +0.1        0.14 ± 43%  perf-profile.children.cycles-pp.console_emit_next_record
      0.04 ± 71%      +0.1        0.17 ± 67%  perf-profile.children.cycles-pp.timekeeping_max_deferment
      0.00            +0.1        0.14 ± 40%  perf-profile.children.cycles-pp.irq_work_run_list
      0.56 ±  2%      +0.2        0.71 ±  4%  perf-profile.children.cycles-pp.move_addr_to_kernel
      0.48 ±  2%      +0.2        0.65 ± 19%  perf-profile.children.cycles-pp.aa_sk_perm
      0.65            +0.2        0.81 ±  3%  perf-profile.children.cycles-pp.__entry_text_start
      0.54            +0.2        0.70        perf-profile.children.cycles-pp.__might_sleep
      0.78            +0.2        0.94 ±  5%  perf-profile.children.cycles-pp.sctp_chunkify
      0.60 ±  2%      +0.2        0.76 ±  5%  perf-profile.children.cycles-pp.sctp_addto_chunk
      0.66            +0.2        0.82 ±  3%  perf-profile.children.cycles-pp.__might_fault
      0.50            +0.2        0.66 ±  6%  perf-profile.children.cycles-pp.memset_erms
      0.60 ±  2%      +0.2        0.77 ±  4%  perf-profile.children.cycles-pp.__dev_queue_xmit
      0.00            +0.2        0.17 ± 52%  perf-profile.children.cycles-pp.rcu_pending
      0.00            +0.2        0.17 ± 29%  perf-profile.children.cycles-pp.__update_blocked_fair
      0.57 ±  2%      +0.2        0.75 ±  6%  perf-profile.children.cycles-pp.sctp_packet_append_chunk
      0.18 ±  3%      +0.2        0.36 ±  5%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.05 ±  8%      +0.2        0.24 ± 21%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.89            +0.2        1.08 ±  4%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.70 ±  3%      +0.2        0.89 ±  8%  perf-profile.children.cycles-pp.sctp_epaddr_lookup_transport
      0.00            +0.2        0.20 ± 26%  perf-profile.children.cycles-pp.arch_scale_freq_tick
      0.73 ±  3%      +0.2        0.93 ±  9%  perf-profile.children.cycles-pp.sctp_endpoint_lookup_assoc
      0.74            +0.2        0.94 ±  4%  perf-profile.children.cycles-pp.__kmem_cache_alloc_node
      0.06 ± 14%      +0.2        0.28 ± 30%  perf-profile.children.cycles-pp.update_blocked_averages
      0.00            +0.2        0.21 ± 51%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.07            +0.2        0.30 ± 15%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.00            +0.2        0.24 ±  8%  perf-profile.children.cycles-pp.idle_cpu
      0.06 ±  6%      +0.2        0.31 ±  8%  perf-profile.children.cycles-pp.sctp_assoc_rwnd_increase
      0.09 ±  6%      +0.3        0.34 ±  8%  perf-profile.children.cycles-pp.switch_fpu_return
      0.00            +0.3        0.26 ± 32%  perf-profile.children.cycles-pp.run_rebalance_domains
      1.04 ±  2%      +0.3        1.31 ±  4%  perf-profile.children.cycles-pp.iovec_from_user
      0.05 ±  7%      +0.3        0.32 ± 15%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.00            +0.3        0.27 ±  7%  perf-profile.children.cycles-pp.restore_fpregs_from_fpstate
      0.24 ±  3%      +0.3        0.52 ± 18%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.06 ±  7%      +0.3        0.34 ± 15%  perf-profile.children.cycles-pp.sctp_wait_for_sndbuf
      0.18 ±  5%      +0.3        0.46 ±  6%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      1.20 ±  2%      +0.3        1.51 ±  4%  perf-profile.children.cycles-pp.__import_iovec
      1.24 ±  2%      +0.3        1.56 ±  4%  perf-profile.children.cycles-pp.import_iovec
      0.36 ±  4%      +0.3        0.69 ± 14%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.35 ±  2%      +0.4        0.71 ±  5%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.09 ±  6%      +0.4        0.45 ± 10%  perf-profile.children.cycles-pp.rebalance_domains
      0.09 ± 30%      +0.4        0.48 ± 32%  perf-profile.children.cycles-pp.tick_irq_enter
      0.10 ± 27%      +0.4        0.49 ± 31%  perf-profile.children.cycles-pp.irq_enter_rcu
      0.65 ±  2%      +0.4        1.08 ±  6%  perf-profile.children.cycles-pp.sctp_packet_transmit_chunk
      2.88            +0.4        3.31 ±  4%  perf-profile.children.cycles-pp.sctp_outq_flush_data
      0.18 ± 12%      +0.5        0.63 ± 21%  perf-profile.children.cycles-pp.clockevents_program_event
      0.71 ±  7%      +0.5        1.20 ± 27%  perf-profile.children.cycles-pp.ktime_get
      0.01 ±223%      +0.5        0.51 ±  4%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      1.84 ±  2%      +0.5        2.37 ±  2%  perf-profile.children.cycles-pp.schedule_timeout
      0.17 ±  4%      +0.6        0.78 ± 16%  perf-profile.children.cycles-pp.scheduler_tick
      2.44            +0.7        3.10 ±  3%  perf-profile.children.cycles-pp._copy_from_user
      2.95            +0.7        3.64 ±  3%  perf-profile.children.cycles-pp.copy_msghdr_from_user
      0.23 ± 10%      +0.7        0.96 ±  2%  perf-profile.children.cycles-pp.newidle_balance
      0.12 ±  9%      +0.8        0.89 ±  2%  perf-profile.children.cycles-pp.update_sg_lb_stats
      1.53 ±  2%      +0.8        2.35 ±  2%  perf-profile.children.cycles-pp.schedule
      0.13 ±  9%      +0.9        0.98 ±  2%  perf-profile.children.cycles-pp.update_sd_lb_stats
      3.45            +0.9        4.31 ±  3%  perf-profile.children.cycles-pp.copyin
      0.14 ± 10%      +0.9        1.00 ±  2%  perf-profile.children.cycles-pp.find_busiest_group
      0.70 ±  2%      +0.9        1.58 ± 19%  perf-profile.children.cycles-pp.menu_select
      0.47 ±  4%      +0.9        1.36 ±  2%  perf-profile.children.cycles-pp.pick_next_task_fair
      2.38            +0.9        3.28 ±  3%  perf-profile.children.cycles-pp.__schedule
      3.70            +0.9        4.65 ±  2%  perf-profile.children.cycles-pp._copy_from_iter
      0.38            +1.0        1.36 ±  4%  perf-profile.children.cycles-pp.__cond_resched
      0.18 ±  4%      +1.0        1.16 ± 14%  perf-profile.children.cycles-pp.__irq_exit_rcu
      0.16 ±  8%      +1.0        1.20 ±  3%  perf-profile.children.cycles-pp.load_balance
      4.16            +1.0        5.20 ±  2%  perf-profile.children.cycles-pp.sctp_user_addto_chunk
      0.24 ±  5%      +1.2        1.44 ± 25%  perf-profile.children.cycles-pp.update_process_times
      0.25 ±  6%      +1.3        1.56 ± 28%  perf-profile.children.cycles-pp.tick_sched_handle
      9.70            +1.3       11.05 ±  3%  perf-profile.children.cycles-pp.sctp_datamsg_from_user
      0.29 ±  5%      +1.4        1.74 ± 25%  perf-profile.children.cycles-pp.tick_sched_timer
      0.38 ±  4%      +2.2        2.57 ± 25%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.63 ±  5%      +3.0        3.63 ± 18%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.64 ±  5%      +3.1        3.72 ± 19%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      0.99 ±  7%      +4.8        5.79 ± 18%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      1.24 ±  5%      +5.3        6.52 ± 16%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
     17.20            +6.5       23.70 ± 13%  perf-profile.children.cycles-pp.start_secondary
     17.32            +6.6       23.88 ± 13%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     17.32            +6.6       23.88 ± 13%  perf-profile.children.cycles-pp.cpu_startup_entry
     17.29            +6.6       23.88 ± 13%  perf-profile.children.cycles-pp.do_idle
     13.26            +8.3       21.54 ± 12%  perf-profile.children.cycles-pp.cpuidle_enter_state
     13.27            +8.3       21.56 ± 12%  perf-profile.children.cycles-pp.cpuidle_enter
     14.17            +9.3       23.44 ± 13%  perf-profile.children.cycles-pp.cpuidle_idle_call
      3.22 ±  4%     +10.7       13.95 ± 12%  perf-profile.children.cycles-pp.intel_idle
     13.62            -2.9       10.68 ±  3%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      2.42 ±  2%      -2.0        0.45 ± 71%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      1.36 ±  2%      -1.3        0.06 ±  8%  perf-profile.self.cycles-pp.__slab_free
      1.03 ±  3%      -0.6        0.43 ± 10%  perf-profile.self.cycles-pp.__mod_memcg_state
      0.63 ±  2%      -0.5        0.11 ± 11%  perf-profile.self.cycles-pp.__skb_datagram_iter
      0.61 ±  2%      -0.5        0.16 ± 23%  perf-profile.self.cycles-pp.ipv4_dst_check
      1.11 ±  2%      -0.4        0.66 ±  6%  perf-profile.self.cycles-pp.rmqueue
      0.74 ±  2%      -0.4        0.30 ±  8%  perf-profile.self.cycles-pp.free_unref_page_commit
      1.09            -0.4        0.66 ±  7%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      1.30            -0.4        0.90 ±  5%  perf-profile.self.cycles-pp.kmem_cache_free
      0.73            -0.4        0.34 ±  7%  perf-profile.self.cycles-pp.__sk_mem_reduce_allocated
      0.79            -0.4        0.42 ±  7%  perf-profile.self.cycles-pp._raw_spin_lock
      0.44 ±  2%      -0.3        0.14 ±  7%  perf-profile.self.cycles-pp.sctp_inet_skb_msgname
      0.45            -0.3        0.16 ±  5%  perf-profile.self.cycles-pp.sctp_skb_recv_datagram
      0.86            -0.3        0.59 ±  8%  perf-profile.self.cycles-pp.sctp_recvmsg
      0.47            -0.3        0.21 ±  9%  perf-profile.self.cycles-pp.__mod_node_page_state
      0.32            -0.3        0.07 ± 11%  perf-profile.self.cycles-pp.schedule_timeout
      0.56 ±  2%      -0.2        0.31 ±  5%  perf-profile.self.cycles-pp.sctp_eat_data
      0.58 ±  2%      -0.2        0.34 ± 13%  perf-profile.self.cycles-pp.__sk_mem_raise_allocated
      0.60            -0.2        0.37 ±  6%  perf-profile.self.cycles-pp.get_page_from_freelist
      0.53 ±  2%      -0.2        0.30 ±  7%  perf-profile.self.cycles-pp.try_charge_memcg
      0.55 ±  3%      -0.2        0.32 ± 17%  perf-profile.self.cycles-pp.dst_release
      0.50            -0.2        0.30 ±  5%  perf-profile.self.cycles-pp.check_heap_object
      0.31            -0.2        0.13 ± 12%  perf-profile.self.cycles-pp.sctp_hash_cmp
      0.36            -0.2        0.19 ± 11%  perf-profile.self.cycles-pp.sctp_data_ready
      0.57 ±  4%      -0.2        0.41 ±  9%  perf-profile.self.cycles-pp.__copy_skb_header
      0.52 ±  2%      -0.1        0.40 ±  4%  perf-profile.self.cycles-pp.__zone_watermark_ok
      0.31 ±  2%      -0.1        0.20 ±  5%  perf-profile.self.cycles-pp.sctp_rcv
      0.17 ±  2%      -0.1        0.06 ± 14%  perf-profile.self.cycles-pp.enqueue_entity
      0.18 ±  4%      -0.1        0.08 ±  8%  perf-profile.self.cycles-pp.consume_skb
      0.13 ±  5%      -0.1        0.04 ± 71%  perf-profile.self.cycles-pp.__free_one_page
      0.24 ±  2%      -0.1        0.15 ± 22%  perf-profile.self.cycles-pp.update_rq_clock
      0.32            -0.1        0.24 ±  4%  perf-profile.self.cycles-pp.__switch_to
      0.18            -0.1        0.09 ± 10%  perf-profile.self.cycles-pp.___perf_sw_event
      0.42            -0.1        0.33 ±  4%  perf-profile.self.cycles-pp.skb_set_owner_w
      0.38 ±  2%      -0.1        0.30 ±  4%  perf-profile.self.cycles-pp.skb_release_data
      0.15 ±  4%      -0.1        0.07 ± 24%  perf-profile.self.cycles-pp.do_idle
      0.20 ±  4%      -0.1        0.14 ±  8%  perf-profile.self.cycles-pp.sctp_ulpq_tail_event
      0.29 ±  3%      -0.1        0.24 ±  7%  perf-profile.self.cycles-pp.__schedule
      0.11 ±  6%      -0.0        0.06 ±  7%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.14 ±  3%      -0.0        0.10 ±  7%  perf-profile.self.cycles-pp.__build_skb_around
      0.12 ±  4%      -0.0        0.07 ± 10%  perf-profile.self.cycles-pp.__wrgsbase_inactive
      0.09 ±  5%      -0.0        0.04 ± 44%  perf-profile.self.cycles-pp.try_to_wake_up
      0.19 ±  3%      -0.0        0.15 ±  7%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.14 ±  3%      -0.0        0.10 ± 10%  perf-profile.self.cycles-pp.__sys_recvmsg
      0.26 ±  2%      -0.0        0.22 ±  3%  perf-profile.self.cycles-pp.sctp_assoc_bh_rcv
      0.23 ±  4%      -0.0        0.19 ±  7%  perf-profile.self.cycles-pp.__copy_msghdr
      0.08 ±  6%      -0.0        0.04 ± 45%  perf-profile.self.cycles-pp.prepare_task_switch
      0.10 ±  5%      -0.0        0.07 ± 10%  perf-profile.self.cycles-pp.security_socket_recvmsg
      0.10 ±  3%      -0.0        0.07 ±  9%  perf-profile.self.cycles-pp.check_new_pages
      0.08 ±  6%      -0.0        0.06 ±  9%  perf-profile.self.cycles-pp.finish_task_switch
      0.06 ±  6%      +0.0        0.08 ±  9%  perf-profile.self.cycles-pp.kmalloc_reserve
      0.06 ±  6%      +0.0        0.08 ±  6%  perf-profile.self.cycles-pp.__sk_mem_schedule
      0.16 ±  2%      +0.0        0.18 ±  2%  perf-profile.self.cycles-pp.sctp_inq_pop
      0.06 ±  7%      +0.0        0.08 ±  5%  perf-profile.self.cycles-pp.sctp_epaddr_lookup_transport
      0.12 ±  3%      +0.0        0.14 ±  5%  perf-profile.self.cycles-pp.sock_kmalloc
      0.09 ±  6%      +0.0        0.11 ± 10%  perf-profile.self.cycles-pp.__sys_sendmsg
      0.07 ±  6%      +0.0        0.09 ± 10%  perf-profile.self.cycles-pp.syscall_enter_from_user_mode
      0.07 ±  8%      +0.0        0.09 ± 11%  perf-profile.self.cycles-pp.__sctp_write_space
      0.09 ±  5%      +0.0        0.12 ±  6%  perf-profile.self.cycles-pp.___sys_sendmsg
      0.08 ±  6%      +0.0        0.10 ±  6%  perf-profile.self.cycles-pp.exit_to_user_mode_prepare
      0.11 ±  3%      +0.0        0.13 ±  4%  perf-profile.self.cycles-pp.sctp_sock_rfree
      0.08 ±  5%      +0.0        0.10 ±  9%  perf-profile.self.cycles-pp.lock_sock_nested
      0.08 ±  4%      +0.0        0.10 ±  9%  perf-profile.self.cycles-pp.move_addr_to_kernel
      0.07 ±  5%      +0.0        0.09 ± 10%  perf-profile.self.cycles-pp.sctp_outq_flush_ctrl
      0.11 ±  5%      +0.0        0.13 ±  5%  perf-profile.self.cycles-pp.check_stack_object
      0.08            +0.0        0.10 ±  7%  perf-profile.self.cycles-pp.sctp_packet_transmit_chunk
      0.07 ±  5%      +0.0        0.09 ±  7%  perf-profile.self.cycles-pp.__kmalloc_node_track_caller
      0.05 ±  8%      +0.0        0.08 ± 13%  perf-profile.self.cycles-pp.sctp_assoc_rwnd_increase
      0.12            +0.0        0.14 ±  8%  perf-profile.self.cycles-pp.__softirqentry_text_start
      0.08 ±  6%      +0.0        0.10 ±  8%  perf-profile.self.cycles-pp.ip_rcv_core
      0.07 ±  5%      +0.0        0.10 ± 25%  perf-profile.self.cycles-pp.sock_sendmsg
      0.12 ±  3%      +0.0        0.15 ±  4%  perf-profile.self.cycles-pp.__mod_lruvec_page_state
      0.15 ±  3%      +0.0        0.18 ±  5%  perf-profile.self.cycles-pp.update_load_avg
      0.10 ±  4%      +0.0        0.13 ±  7%  perf-profile.self.cycles-pp.process_backlog
      0.06 ± 11%      +0.0        0.09 ± 11%  perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
      0.11 ±  4%      +0.0        0.14 ±  8%  perf-profile.self.cycles-pp.__netif_receive_skb_core
      0.08 ±  4%      +0.0        0.11 ±  7%  perf-profile.self.cycles-pp.sctp_chunk_assign_ssn
      0.20 ±  3%      +0.0        0.23 ±  7%  perf-profile.self.cycles-pp.sctp_check_transmitted
      0.05 ±  8%      +0.0        0.08 ± 11%  perf-profile.self.cycles-pp.sctp_sched_fcfs_dequeue
      0.05            +0.0        0.08 ± 12%  perf-profile.self.cycles-pp.inet_recvmsg
      0.05            +0.0        0.08 ± 10%  perf-profile.self.cycles-pp.enqueue_to_backlog
      0.12            +0.0        0.15 ±  3%  perf-profile.self.cycles-pp.copy_msghdr_from_user
      0.11 ±  4%      +0.0        0.14 ±  9%  perf-profile.self.cycles-pp._sctp_make_chunk
      0.19 ±  3%      +0.0        0.22 ±  8%  perf-profile.self.cycles-pp.free_unref_page
      0.14 ±  3%      +0.0        0.18 ±  5%  perf-profile.self.cycles-pp.sctp_user_addto_chunk
      0.08 ±  6%      +0.0        0.11 ±  9%  perf-profile.self.cycles-pp.ip_protocol_deliver_rcu
      0.08 ±  8%      +0.0        0.11 ±  6%  perf-profile.self.cycles-pp.sctp_cmp_addr_exact
      0.13 ±  3%      +0.0        0.16 ±  6%  perf-profile.self.cycles-pp.__ip_queue_xmit
      0.12 ±  4%      +0.0        0.15 ±  5%  perf-profile.self.cycles-pp.__might_fault
      0.16 ±  3%      +0.0        0.19 ±  6%  perf-profile.self.cycles-pp.__put_user_nocheck_8
      0.05            +0.0        0.08 ± 11%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.13 ±  4%      +0.0        0.16 ±  7%  perf-profile.self.cycles-pp.skb_put
      0.12 ±  5%      +0.0        0.16 ±  7%  perf-profile.self.cycles-pp.os_xsave
      0.36            +0.0        0.40 ±  6%  perf-profile.self.cycles-pp.native_sched_clock
      0.27 ±  2%      +0.0        0.31 ±  3%  perf-profile.self.cycles-pp.recvmsg
      0.18 ±  2%      +0.0        0.21 ±  6%  perf-profile.self.cycles-pp.__import_iovec
      0.08 ±  5%      +0.0        0.12 ±  9%  perf-profile.self.cycles-pp.sctp_make_datafrag_empty
      0.18 ±  2%      +0.0        0.21 ±  6%  perf-profile.self.cycles-pp._copy_to_iter
      0.22 ±  4%      +0.0        0.26 ±  7%  perf-profile.self.cycles-pp.do_syscall_64
      0.10 ±  4%      +0.0        0.14 ± 24%  perf-profile.self.cycles-pp.sctp_hash_key
      0.13 ±  3%      +0.0        0.17 ±  6%  perf-profile.self.cycles-pp.sctp_ulpevent_free
      0.11 ±  3%      +0.0        0.16 ± 10%  perf-profile.self.cycles-pp.sctp_outq_sack
      0.11 ±  6%      +0.0        0.16 ±  6%  perf-profile.self.cycles-pp.update_curr
      0.12 ±  4%      +0.0        0.17 ±  4%  perf-profile.self.cycles-pp.ip_finish_output2
      0.09 ±  7%      +0.0        0.14 ± 11%  perf-profile.self.cycles-pp.sock_kfree_s
      0.19 ±  3%      +0.0        0.24 ±  5%  perf-profile.self.cycles-pp.__local_bh_enable_ip
      0.32 ±  2%      +0.0        0.36 ±  5%  perf-profile.self.cycles-pp.__alloc_pages
      0.23 ±  2%      +0.0        0.27 ±  5%  perf-profile.self.cycles-pp.__entry_text_start
      0.02 ±141%      +0.0        0.06 ± 11%  perf-profile.self.cycles-pp.sctp_tsnmap_check
      0.01 ±223%      +0.0        0.06 ±  9%  perf-profile.self.cycles-pp.perf_trace_sched_stat_runtime
      0.16 ±  2%      +0.0        0.21 ± 10%  perf-profile.self.cycles-pp.__sctp_packet_append_chunk
      0.16            +0.0        0.21 ±  5%  perf-profile.self.cycles-pp._copy_from_iter
      0.14 ±  5%      +0.0        0.19 ±  5%  perf-profile.self.cycles-pp.iovec_from_user
      0.22 ±  4%      +0.1        0.27 ±  5%  perf-profile.self.cycles-pp.sctp_chunk_hold
      0.31 ±  5%      +0.1        0.36 ±  8%  perf-profile.self.cycles-pp.sctp_transport_hold
      0.34 ±  3%      +0.1        0.40 ±  4%  perf-profile.self.cycles-pp.sctp_packet_pack
      0.21 ±  2%      +0.1        0.26 ±  5%  perf-profile.self.cycles-pp.____sys_sendmsg
      0.21 ±  3%      +0.1        0.26 ±  6%  perf-profile.self.cycles-pp.sctp_association_hold
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.skb_pull
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.sctp_sendmsg_check_sflags
      0.20 ±  3%      +0.1        0.25 ±  5%  perf-profile.self.cycles-pp.kfree
      0.00            +0.1        0.06 ± 13%  perf-profile.self.cycles-pp.error_entry
      0.17 ±  2%      +0.1        0.23 ±  8%  perf-profile.self.cycles-pp.mem_cgroup_uncharge_skmem
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.loopback_xmit
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.ip_local_deliver_finish
      0.12 ±  4%      +0.1        0.18 ±  9%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.24            +0.1        0.30 ±  6%  perf-profile.self.cycles-pp.sctp_ulpevent_receive_data
      0.23 ±  3%      +0.1        0.29 ±  6%  perf-profile.self.cycles-pp.sctp_datamsg_put
      0.38            +0.1        0.44 ±  6%  perf-profile.self.cycles-pp.send_sctp_stream_1toMany
      0.35 ±  2%      +0.1        0.41 ±  4%  perf-profile.self.cycles-pp.kmem_cache_alloc_node
      0.06 ±  7%      +0.1        0.13 ±  7%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode
      0.23 ±  3%      +0.1        0.29 ±  7%  perf-profile.self.cycles-pp.__genradix_ptr
      0.21 ±  2%      +0.1        0.27 ±  8%  perf-profile.self.cycles-pp.__put_user_nocheck_4
      0.00            +0.1        0.06 ± 19%  perf-profile.self.cycles-pp.perf_trace_sched_switch
      0.00            +0.1        0.06 ± 17%  perf-profile.self.cycles-pp.update_sd_lb_stats
      0.00            +0.1        0.07 ± 11%  perf-profile.self.cycles-pp.__netif_receive_skb_one_core
      0.26 ±  3%      +0.1        0.33 ±  8%  perf-profile.self.cycles-pp.sctp_sendmsg_parse
      0.00            +0.1        0.07 ± 25%  perf-profile.self.cycles-pp.intel_idle
      0.01 ±223%      +0.1        0.08 ± 34%  perf-profile.self.cycles-pp.tsc_verify_tsc_adjust
      0.00            +0.1        0.07 ± 18%  perf-profile.self.cycles-pp.tick_sched_do_timer
      0.23 ±  5%      +0.1        0.30 ±  3%  perf-profile.self.cycles-pp.sctp_packet_append_chunk
      0.29 ±  3%      +0.1        0.36 ±  4%  perf-profile.self.cycles-pp.sctp_do_sm
      0.00            +0.1        0.07 ± 15%  perf-profile.self.cycles-pp.load_balance
      0.22 ±  2%      +0.1        0.30 ±  4%  perf-profile.self.cycles-pp.sctp_outq_flush
      0.29            +0.1        0.37 ±  5%  perf-profile.self.cycles-pp.sctp_association_put
      0.06 ± 11%      +0.1        0.14 ± 15%  perf-profile.self.cycles-pp.irqtime_account_irq
      0.47 ±  2%      +0.1        0.55 ±  4%  perf-profile.self.cycles-pp.free_pcp_prepare
      0.00            +0.1        0.09 ± 40%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.30 ±  2%      +0.1        0.40 ±  4%  perf-profile.self.cycles-pp._copy_from_user
      0.29            +0.1        0.38 ±  9%  perf-profile.self.cycles-pp.__virt_addr_valid
      0.00            +0.1        0.09 ± 32%  perf-profile.self.cycles-pp.hrtimer_interrupt
      0.26 ±  2%      +0.1        0.36 ±  8%  perf-profile.self.cycles-pp.sctp_wfree
      0.00            +0.1        0.10 ±  8%  perf-profile.self.cycles-pp._find_next_and_bit
      0.34            +0.1        0.44 ±  7%  perf-profile.self.cycles-pp.__check_object_size
      0.29 ±  3%      +0.1        0.39 ±  5%  perf-profile.self.cycles-pp.sendmsg
      0.00            +0.1        0.10 ±  7%  perf-profile.self.cycles-pp.__release_sock
      0.00            +0.1        0.10 ± 12%  perf-profile.self.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.29 ±  5%      +0.1        0.39 ± 25%  perf-profile.self.cycles-pp.aa_sk_perm
      0.56 ±  2%      +0.1        0.66 ±  4%  perf-profile.self.cycles-pp.sctp_sendmsg
      0.32 ±  2%      +0.1        0.42 ±  4%  perf-profile.self.cycles-pp.mem_cgroup_charge_skmem
      0.27 ±  3%      +0.1        0.37 ±  6%  perf-profile.self.cycles-pp.__kmem_cache_free
      0.11 ±  3%      +0.1        0.21 ± 31%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.32 ±  2%      +0.1        0.43 ±  8%  perf-profile.self.cycles-pp._raw_spin_trylock
      0.00            +0.1        0.10 ± 33%  perf-profile.self.cycles-pp.__update_blocked_fair
      0.46            +0.1        0.56 ±  4%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.23 ±  2%      +0.1        0.34 ±  4%  perf-profile.self.cycles-pp.__cond_resched
      0.00            +0.1        0.11 ± 32%  perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
      0.00            +0.1        0.11 ± 21%  perf-profile.self.cycles-pp.__intel_pmu_enable_all
      0.40 ±  3%      +0.1        0.51 ±  9%  perf-profile.self.cycles-pp.sctp_cmd_interpreter
      0.00            +0.1        0.11 ± 19%  perf-profile.self.cycles-pp.update_irq_load_avg
      0.35 ±  3%      +0.1        0.46 ±  4%  perf-profile.self.cycles-pp.__fdget
      0.32            +0.1        0.44 ±  3%  perf-profile.self.cycles-pp.read_tsc
      0.42            +0.1        0.54 ±  4%  perf-profile.self.cycles-pp.__alloc_skb
      0.41 ±  2%      +0.1        0.52 ±  3%  perf-profile.self.cycles-pp.sctp_sendmsg_to_asoc
      0.42 ±  2%      +0.1        0.54 ±  5%  perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.01 ±223%      +0.1        0.13 ± 23%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.01 ±223%      +0.1        0.13 ± 32%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.04 ± 71%      +0.1        0.16 ± 71%  perf-profile.self.cycles-pp.timekeeping_max_deferment
      0.44            +0.1        0.58 ±  4%  perf-profile.self.cycles-pp.__kmem_cache_alloc_node
      0.55            +0.1        0.69 ±  3%  perf-profile.self.cycles-pp.sctp_chunk_put
      0.44 ±  2%      +0.1        0.59        perf-profile.self.cycles-pp.__might_sleep
      0.48 ±  2%      +0.2        0.64 ±  5%  perf-profile.self.cycles-pp.memset_erms
      0.47            +0.2        0.64 ±  6%  perf-profile.self.cycles-pp.sctp_outq_flush_data
      0.57            +0.2        0.75 ±  6%  perf-profile.self.cycles-pp._raw_spin_lock_bh
      0.05            +0.2        0.24 ± 21%  perf-profile.self.cycles-pp.lapic_next_deadline
      0.00            +0.2        0.20 ± 26%  perf-profile.self.cycles-pp.arch_scale_freq_tick
      0.00            +0.2        0.23 ±  7%  perf-profile.self.cycles-pp.idle_cpu
      0.07            +0.2        0.30 ± 15%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.63            +0.2        0.88 ±  3%  perf-profile.self.cycles-pp.sctp_datamsg_from_user
      0.00            +0.3        0.27 ±  7%  perf-profile.self.cycles-pp.restore_fpregs_from_fpstate
      0.38 ± 13%      +0.4        0.78 ± 42%  perf-profile.self.cycles-pp.ktime_get
      0.24 ±  3%      +0.5        0.71 ± 30%  perf-profile.self.cycles-pp.menu_select
      0.00            +0.5        0.50 ±  4%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.10 ± 11%      +0.6        0.70        perf-profile.self.cycles-pp.update_sg_lb_stats
      0.28 ±  3%      +0.7        0.97 ± 14%  perf-profile.self.cycles-pp.cpuidle_enter_state


***************************************************************************************************
lkp-icl-2sp6: 128 threads 2 sockets Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz (Ice Lake) with 128G memory
=========================================================================================
class/compiler/cpufreq_governor/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime:
  network/gcc-11/performance/x86_64-rhel-8.3/100%/debian-11.1-x86_64-20220510.cgz/lkp-icl-2sp6/sctp/stress-ng/60s

commit: 
  c36eae5a86 ("sched/psi: Fix possible missing or delayed pending event")
  b6aabb01e3 ("sched: consider WF_SYNC to find idle siblings")

c36eae5a86d15a69 b6aabb01e3004b0285d953761e1 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
   9399721           -48.4%    4850253 ±  5%  stress-ng.sctp.ops
    156661           -48.4%      80837 ±  5%  stress-ng.sctp.ops_per_sec
    514.00 ±  3%  +41004.5%     211277 ± 10%  stress-ng.time.involuntary_context_switches
     59.67 ±  2%     -51.1%      29.17 ± 10%  stress-ng.time.percent_of_cpu_this_job_got
     36.02 ±  2%     -50.6%      17.80 ± 10%  stress-ng.time.system_time
    970641           -63.3%     356134 ±  5%  stress-ng.time.voluntary_context_switches
     31732           -60.7%      12482 ±  3%  vmstat.system.cs
      0.43            -0.2        0.22 ±  9%  mpstat.cpu.all.sys%
      0.10 ±  3%      -0.0        0.09 ±  9%  mpstat.cpu.all.usr%
      1544 ±  2%     +16.4%       1797 ± 11%  numa-meminfo.node0.Active
      3914 ±  5%     -16.9%       3253 ± 12%  numa-meminfo.node1.Active
      3834 ±  5%     -15.6%       3237 ± 12%  numa-meminfo.node1.Active(anon)
   1154373 ±  2%     -31.7%     788904 ±  8%  numa-numastat.node0.local_node
   1239775 ±  2%     -32.6%     835766 ±  7%  numa-numastat.node0.numa_hit
   1300505 ±  2%     -40.6%     772809 ± 12%  numa-numastat.node1.local_node
   1330886 ±  2%     -36.8%     841742 ± 10%  numa-numastat.node1.numa_hit
    213291 ± 17%     -99.3%       1496 ± 33%  turbostat.C1
      0.35 ± 78%      -0.3        0.00 ±141%  turbostat.C1%
    389879 ±129%   +1006.6%    4314548 ± 91%  turbostat.C6
     16517 ±  3%     -97.4%     425.50 ± 54%  turbostat.POLL
   1239630 ±  2%     -32.6%     835988 ±  7%  numa-vmstat.node0.numa_hit
   1154228 ±  2%     -31.6%     789126 ±  8%  numa-vmstat.node0.numa_local
    958.33 ±  5%     -15.6%     809.17 ± 12%  numa-vmstat.node1.nr_active_anon
    958.33 ±  5%     -15.6%     809.17 ± 12%  numa-vmstat.node1.nr_zone_active_anon
   1330563 ±  2%     -36.7%     841771 ± 10%  numa-vmstat.node1.numa_hit
   1300182 ±  2%     -40.6%     772838 ± 12%  numa-vmstat.node1.numa_local
      2.67 ± 35%  +59431.2%       1587 ±116%  proc-vmstat.numa_hint_faults_local
   2580484           -34.9%    1679518 ±  2%  proc-vmstat.numa_hit
   2464701           -36.6%    1563724 ±  2%  proc-vmstat.numa_local
     14670 ± 55%     +64.3%      24106 ± 21%  proc-vmstat.numa_pte_updates
  18610165           -48.5%    9577564 ±  4%  proc-vmstat.pgalloc_normal
  18478395           -48.9%    9435687 ±  4%  proc-vmstat.pgfree
     18554            -3.3%      17937        proc-vmstat.pgreuse
      6103 ± 43%     -72.4%       1682 ± 36%  sched_debug.cfs_rq:/.load.avg
     50624 ± 58%     -81.4%       9439 ± 83%  sched_debug.cfs_rq:/.load.stddev
    853.50 ± 14%   +3161.6%      27837 ±136%  sched_debug.cfs_rq:/.load_avg.max
    965.00 ±  5%     +52.0%       1466 ± 18%  sched_debug.cfs_rq:/.runnable_avg.max
    223.02 ± 23%     +36.3%     303.95 ± 15%  sched_debug.cfs_rq:/.runnable_avg.stddev
    964.33 ±  5%     +51.4%       1460 ± 18%  sched_debug.cfs_rq:/.util_avg.max
    222.86 ± 23%     +35.4%     301.73 ± 15%  sched_debug.cfs_rq:/.util_avg.stddev
    351971 ± 38%     -41.6%     205390 ± 25%  sched_debug.cpu.avg_idle.stddev
      4.02 ±  8%     +18.1%       4.75 ±  5%  sched_debug.cpu.clock.stddev
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.avg
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.max
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.min
 7.447e+08           -27.7%  5.387e+08 ±  3%  perf-stat.i.branch-instructions
      0.79 ± 20%      +2.6        3.35 ± 75%  perf-stat.i.branch-miss-rate%
   4421153 ±  3%     -29.9%    3100372 ±  5%  perf-stat.i.cache-misses
     32967           -61.7%      12612 ±  4%  perf-stat.i.context-switches
      1.67           +50.8%       2.52 ± 21%  perf-stat.i.cpi
    313.73 ±  3%    +253.6%       1109 ±  4%  perf-stat.i.cpu-migrations
      1477 ±  4%     +55.0%       2289 ± 19%  perf-stat.i.cycles-between-cache-misses
      0.06 ± 42%      +0.3        0.36 ± 77%  perf-stat.i.dTLB-load-miss-rate%
 1.027e+09           -28.5%   7.34e+08 ±  2%  perf-stat.i.dTLB-loads
 5.512e+08           -31.7%  3.762e+08 ±  2%  perf-stat.i.dTLB-stores
 3.762e+09           -27.1%  2.742e+09 ±  3%  perf-stat.i.instructions
      0.61           -29.2%       0.43 ± 21%  perf-stat.i.ipc
    393.08 ±  7%     -73.0%     106.27 ± 76%  perf-stat.i.metric.K/sec
     18.00           -26.4%      13.25 ±  5%  perf-stat.i.metric.M/sec
     51.73 ±  4%     +22.8       74.57 ±  6%  perf-stat.i.node-load-miss-rate%
    225775 ±  5%     +85.3%     418438 ±  3%  perf-stat.i.node-load-misses
     14.54 ±  2%     +17.0       31.53 ± 11%  perf-stat.i.node-store-miss-rate%
    213471 ±  5%     +19.7%     255432 ±  7%  perf-stat.i.node-store-misses
   1489600           -47.6%     780285 ±  7%  perf-stat.i.node-stores
      1.03 ± 14%      +2.2        3.19 ± 64%  perf-stat.overall.branch-miss-rate%
      1.61           +42.0%       2.29 ± 19%  perf-stat.overall.cpi
      1370           +46.6%       2008 ± 17%  perf-stat.overall.cycles-between-cache-misses
      0.06 ± 42%      +0.3        0.31 ± 76%  perf-stat.overall.dTLB-load-miss-rate%
      0.62           -26.7%       0.46 ± 20%  perf-stat.overall.ipc
     53.37 ±  3%     +20.7       74.08 ±  5%  perf-stat.overall.node-load-miss-rate%
     12.54 ±  4%     +12.2       24.74 ±  9%  perf-stat.overall.node-store-miss-rate%
 7.319e+08           -27.8%  5.286e+08 ±  3%  perf-stat.ps.branch-instructions
   4344607 ±  3%     -29.8%    3051326 ±  5%  perf-stat.ps.cache-misses
     32395           -61.7%      12403 ±  3%  perf-stat.ps.context-switches
    308.45 ±  4%    +253.7%       1091 ±  4%  perf-stat.ps.cpu-migrations
 1.009e+09           -28.6%  7.207e+08 ±  2%  perf-stat.ps.dTLB-loads
 5.417e+08           -31.8%  3.695e+08 ±  2%  perf-stat.ps.dTLB-stores
 3.697e+09           -27.2%  2.691e+09 ±  3%  perf-stat.ps.instructions
    222119 ±  5%     +85.2%     411273 ±  3%  perf-stat.ps.node-load-misses
    209895 ±  5%     +19.8%     251498 ±  7%  perf-stat.ps.node-store-misses
   1463189           -47.5%     768638 ±  7%  perf-stat.ps.node-stores
 2.374e+11           -26.9%  1.736e+11 ±  2%  perf-stat.total.instructions
     25.09 ±  2%     -14.3       10.78 ± 29%  perf-profile.calltrace.cycles-pp.stress_sctp
     22.22 ±  2%     -12.6        9.64 ± 30%  perf-profile.calltrace.cycles-pp.sendmsg.stress_sctp
     21.90 ±  2%     -12.4        9.50 ± 29%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.sendmsg.stress_sctp
     21.80 ±  2%     -12.3        9.47 ± 30%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendmsg.stress_sctp
     21.67 ±  2%     -12.3        9.40 ± 30%  perf-profile.calltrace.cycles-pp.__sys_sendmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendmsg.stress_sctp
     21.51 ±  2%     -12.2        9.32 ± 30%  perf-profile.calltrace.cycles-pp.___sys_sendmsg.__sys_sendmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendmsg
     20.62 ±  2%     -11.7        8.96 ± 30%  perf-profile.calltrace.cycles-pp.____sys_sendmsg.___sys_sendmsg.__sys_sendmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe
     19.56 ±  2%     -11.1        8.47 ± 30%  perf-profile.calltrace.cycles-pp.sock_sendmsg.____sys_sendmsg.___sys_sendmsg.__sys_sendmsg.do_syscall_64
     19.33 ±  2%     -11.0        8.37 ± 30%  perf-profile.calltrace.cycles-pp.sctp_sendmsg.sock_sendmsg.____sys_sendmsg.___sys_sendmsg.__sys_sendmsg
     11.82 ±  4%      -8.0        3.85 ± 29%  perf-profile.calltrace.cycles-pp.recvmsg
     11.46 ±  4%      -7.8        3.68 ± 29%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.recvmsg
     11.40 ±  4%      -7.8        3.65 ± 29%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvmsg
     11.23 ±  5%      -7.7        3.58 ± 29%  perf-profile.calltrace.cycles-pp.__sys_recvmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvmsg
     11.04 ±  4%      -7.5        3.50 ± 29%  perf-profile.calltrace.cycles-pp.___sys_recvmsg.__sys_recvmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvmsg
     13.15 ±  3%      -7.2        5.97 ± 32%  perf-profile.calltrace.cycles-pp.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg.____sys_sendmsg.___sys_sendmsg
     10.28 ±  4%      -7.2        3.13 ± 29%  perf-profile.calltrace.cycles-pp.____sys_recvmsg.___sys_recvmsg.__sys_recvmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe
      9.78 ±  4%      -6.9        2.89 ± 29%  perf-profile.calltrace.cycles-pp.inet_recvmsg.____sys_recvmsg.___sys_recvmsg.__sys_recvmsg.do_syscall_64
      9.74 ±  4%      -6.9        2.85 ± 29%  perf-profile.calltrace.cycles-pp.sctp_recvmsg.inet_recvmsg.____sys_recvmsg.___sys_recvmsg.__sys_recvmsg
      6.57 ±  5%      -4.1        2.45 ± 30%  perf-profile.calltrace.cycles-pp.sctp_primitive_SEND.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg.____sys_sendmsg
      5.83 ±  5%      -3.7        2.14 ± 31%  perf-profile.calltrace.cycles-pp.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND.sctp_sendmsg_to_asoc
      5.48 ±  2%      -3.5        2.00 ± 26%  perf-profile.calltrace.cycles-pp.sctp_backlog_rcv.__release_sock.release_sock.sctp_sendmsg.sock_sendmsg
      5.59 ±  2%      -3.5        2.13 ± 26%  perf-profile.calltrace.cycles-pp.release_sock.sctp_sendmsg.sock_sendmsg.____sys_sendmsg.___sys_sendmsg
      5.49 ±  2%      -3.4        2.09 ± 26%  perf-profile.calltrace.cycles-pp.__release_sock.release_sock.sctp_sendmsg.sock_sendmsg.____sys_sendmsg
      4.61            -3.0        1.66 ± 27%  perf-profile.calltrace.cycles-pp.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_backlog_rcv.__release_sock
      4.57            -2.9        1.62 ± 27%  perf-profile.calltrace.cycles-pp.sctp_assoc_bh_rcv.sctp_backlog_rcv.__release_sock.release_sock.sctp_sendmsg
      4.49            -2.9        1.61 ± 27%  perf-profile.calltrace.cycles-pp.sctp_do_sm.sctp_assoc_bh_rcv.sctp_backlog_rcv.__release_sock.release_sock
      4.22 ±  2%      -2.9        1.34 ± 53%  perf-profile.calltrace.cycles-pp.sctp_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog
      5.33            -2.8        2.50 ± 35%  perf-profile.calltrace.cycles-pp.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg.____sys_sendmsg
      4.28 ±  2%      -2.8        1.45 ± 54%  perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action
      4.27 ±  2%      -2.8        1.44 ± 54%  perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll
      4.36 ±  2%      -2.8        1.57 ± 46%  perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action.__softirqentry_text_start
      4.32 ±  4%      -2.7        1.58 ± 30%  perf-profile.calltrace.cycles-pp.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND.sctp_sendmsg_to_asoc.sctp_sendmsg
      3.96 ±  2%      -2.7        1.24 ± 49%  perf-profile.calltrace.cycles-pp.__ip_queue_xmit.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm
      3.92 ±  2%      -2.7        1.23 ± 50%  perf-profile.calltrace.cycles-pp.ip_finish_output2.__ip_queue_xmit.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter
      3.67 ±  2%      -2.7        0.98 ± 67%  perf-profile.calltrace.cycles-pp.sctp_assoc_bh_rcv.sctp_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core
      3.72 ±  3%      -2.7        1.06 ± 52%  perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.__softirqentry_text_start.do_softirq.__local_bh_enable_ip
      3.72 ±  3%      -2.7        1.06 ± 53%  perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.__softirqentry_text_start.do_softirq
      4.00 ±  3%      -2.7        1.34 ± 30%  perf-profile.calltrace.cycles-pp.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND
      3.83 ±  3%      -2.6        1.19 ± 51%  perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.__ip_queue_xmit.sctp_packet_transmit.sctp_outq_flush
      3.81 ±  3%      -2.6        1.19 ± 50%  perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.__ip_queue_xmit.sctp_packet_transmit
      3.47            -2.6        0.84 ± 83%  perf-profile.calltrace.cycles-pp.sctp_do_sm.sctp_assoc_bh_rcv.sctp_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
      3.80 ±  3%      -2.6        1.18 ± 50%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq.__local_bh_enable_ip.ip_finish_output2.__ip_queue_xmit
      3.21 ±  2%      -2.6        0.61 ± 55%  perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.sctp_recvmsg.inet_recvmsg.____sys_recvmsg
      4.13 ±  4%      -2.6        1.54 ± 30%  perf-profile.calltrace.cycles-pp.sctp_do_sm.sctp_primitive_SEND.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg
      3.74 ±  3%      -2.6        1.15 ± 50%  perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq.__local_bh_enable_ip.ip_finish_output2
      3.23 ±  2%      -2.5        0.71 ± 29%  perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.sctp_recvmsg.inet_recvmsg.____sys_recvmsg.___sys_recvmsg
      3.09 ±  2%      -2.4        0.71 ±100%  perf-profile.calltrace.cycles-pp.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_rcv.ip_protocol_deliver_rcu
      2.51 ±  2%      -2.2        0.36 ±103%  perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.sctp_recvmsg.inet_recvmsg
      2.38 ±  2%      -2.1        0.32 ±101%  perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.sctp_recvmsg
      2.35 ±  2%      -2.0        0.31 ±102%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter
      2.85            -1.8        1.02 ± 23%  perf-profile.calltrace.cycles-pp.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_backlog_rcv
      2.56            -1.7        0.87 ± 25%  perf-profile.calltrace.cycles-pp.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv
      2.24 ±  8%      -1.7        0.57 ± 52%  perf-profile.calltrace.cycles-pp.sctp_skb_recv_datagram.sctp_recvmsg.inet_recvmsg.____sys_recvmsg.___sys_recvmsg
      2.40 ±  3%      -1.5        0.89 ± 31%  perf-profile.calltrace.cycles-pp.sctp_outq_sack.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_backlog_rcv
      1.72 ±  4%      -1.4        0.37 ±104%  perf-profile.calltrace.cycles-pp.sctp_chunk_put.sctp_outq_sack.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv
      2.57 ±  2%      -1.3        1.24 ± 42%  perf-profile.calltrace.cycles-pp.sctp_make_datafrag_empty.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg
      1.77 ±  9%      -1.2        0.60 ± 75%  perf-profile.calltrace.cycles-pp.sctp_outq_flush_data.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND
      2.05 ±  3%      -1.1        0.95 ± 38%  perf-profile.calltrace.cycles-pp._sctp_make_chunk.sctp_make_datafrag_empty.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg
      1.47 ±  4%      -1.0        0.48 ±105%  perf-profile.calltrace.cycles-pp.__alloc_skb._sctp_make_chunk.sctp_make_datafrag_empty.sctp_datamsg_from_user.sctp_sendmsg_to_asoc
      1.54 ±  3%      -0.9        0.60 ± 53%  perf-profile.calltrace.cycles-pp.sctp_user_addto_chunk.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg
      1.19 ±  4%      -0.9        0.32 ±102%  perf-profile.calltrace.cycles-pp._copy_from_iter.sctp_user_addto_chunk.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg
      1.22 ±  3%      +0.2        1.39 ±  7%  perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      1.02 ±  5%      +0.5        1.47 ±  6%  perf-profile.calltrace.cycles-pp.rebalance_domains.__softirqentry_text_start.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.19 ±141%      +0.5        0.67 ± 14%  perf-profile.calltrace.cycles-pp.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call
      0.17 ±141%      +0.5        0.69 ± 19%  perf-profile.calltrace.cycles-pp.find_busiest_group.load_balance.rebalance_domains.__softirqentry_text_start.__irq_exit_rcu
      0.40 ± 71%      +0.6        0.95 ± 18%  perf-profile.calltrace.cycles-pp.load_balance.rebalance_domains.__softirqentry_text_start.__irq_exit_rcu.sysvec_apic_timer_interrupt
      2.27 ± 14%      +0.6        2.82 ± 10%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry
      0.00            +0.6        0.57 ±  3%  perf-profile.calltrace.cycles-pp.ktime_get_update_offsets_now.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00            +0.6        0.60 ±  7%  perf-profile.calltrace.cycles-pp.update_rq_clock.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
      0.00            +0.6        0.64 ± 18%  perf-profile.calltrace.cycles-pp.update_sd_lb_stats.find_busiest_group.load_balance.rebalance_domains.__softirqentry_text_start
      0.00            +0.6        0.64 ±  6%  perf-profile.calltrace.cycles-pp.ret_from_fork
      0.00            +0.6        0.64 ±  6%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      0.44 ± 72%      +0.8        1.22 ± 39%  perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      1.48 ±  3%      +0.8        2.27 ± 11%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      3.20 ±  3%      +0.8        4.00 ±  7%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      3.70 ±  2%      +1.1        4.80 ±  7%  perf-profile.calltrace.cycles-pp.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      2.36 ±  4%      +1.9        4.25 ± 23%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      2.45 ±  5%      +2.2        4.70 ± 28%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      3.99 ±  8%      +2.4        6.41 ± 15%  perf-profile.calltrace.cycles-pp.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
      2.99 ±  2%      +2.4        5.42 ± 24%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      4.96 ±  2%      +3.6        8.54 ± 25%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      9.02 ±  2%      +3.7       12.76 ± 12%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      9.12 ±  2%      +3.9       13.04 ± 12%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
     14.90            +6.0       20.88 ± 12%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
     20.77            +6.3       27.04 ±  7%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     30.56           +12.3       42.87 ±  5%  perf-profile.calltrace.cycles-pp.mwait_idle_with_hints.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
     30.80           +12.4       43.18 ±  5%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     54.38           +19.1       73.50 ±  4%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
     53.23           +19.3       72.57 ±  4%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
     60.54           +21.9       82.44 ±  5%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     60.62           +21.9       82.56 ±  5%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     60.62           +21.9       82.56 ±  5%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     59.19           +22.2       81.34 ±  5%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     61.07           +22.3       83.39 ±  5%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     36.24 ±  2%     -21.4       14.80 ± 28%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     36.09 ±  2%     -21.4       14.74 ± 28%  perf-profile.children.cycles-pp.do_syscall_64
     25.09 ±  2%     -14.3       10.78 ± 29%  perf-profile.children.cycles-pp.stress_sctp
     22.40 ±  2%     -12.6        9.77 ± 30%  perf-profile.children.cycles-pp.sendmsg
     21.69 ±  2%     -12.3        9.44 ± 30%  perf-profile.children.cycles-pp.__sys_sendmsg
     21.54 ±  2%     -12.2        9.36 ± 30%  perf-profile.children.cycles-pp.___sys_sendmsg
     20.65 ±  2%     -11.7        9.00 ± 30%  perf-profile.children.cycles-pp.____sys_sendmsg
     19.57 ±  2%     -11.1        8.50 ± 30%  perf-profile.children.cycles-pp.sctp_sendmsg
     19.56 ±  2%     -11.1        8.50 ± 30%  perf-profile.children.cycles-pp.sock_sendmsg
     14.14            -8.2        5.89 ± 26%  perf-profile.children.cycles-pp.sctp_do_sm
     13.90            -8.1        5.79 ± 26%  perf-profile.children.cycles-pp.sctp_cmd_interpreter
     11.97 ±  4%      -8.1        3.92 ± 29%  perf-profile.children.cycles-pp.recvmsg
     11.24 ±  4%      -7.7        3.58 ± 29%  perf-profile.children.cycles-pp.__sys_recvmsg
     11.06 ±  4%      -7.6        3.51 ± 29%  perf-profile.children.cycles-pp.___sys_recvmsg
     13.19 ±  3%      -7.2        6.00 ± 32%  perf-profile.children.cycles-pp.sctp_sendmsg_to_asoc
     10.29 ±  4%      -7.2        3.13 ± 29%  perf-profile.children.cycles-pp.____sys_recvmsg
      9.97 ±  4%      -7.0        2.95 ± 29%  perf-profile.children.cycles-pp.sctp_recvmsg
      9.79 ±  4%      -6.9        2.89 ± 29%  perf-profile.children.cycles-pp.inet_recvmsg
     10.33 ±  2%      -6.1        4.23 ± 27%  perf-profile.children.cycles-pp.sctp_outq_flush
      9.94            -5.7        4.22 ± 26%  perf-profile.children.cycles-pp.sctp_assoc_bh_rcv
      8.54            -5.0        3.51 ± 27%  perf-profile.children.cycles-pp.sctp_packet_transmit
      7.06 ±  2%      -4.3        2.76 ± 27%  perf-profile.children.cycles-pp.release_sock
      6.76 ±  2%      -4.2        2.52 ± 27%  perf-profile.children.cycles-pp.sctp_backlog_rcv
      6.79 ±  2%      -4.2        2.64 ± 27%  perf-profile.children.cycles-pp.__release_sock
      6.58 ±  5%      -4.1        2.45 ± 31%  perf-profile.children.cycles-pp.sctp_primitive_SEND
      4.28 ±  2%      -3.0        1.32 ± 29%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      5.24 ±  2%      -2.9        2.37 ± 29%  perf-profile.children.cycles-pp.__ip_queue_xmit
      5.36            -2.8        2.51 ± 35%  perf-profile.children.cycles-pp.sctp_datamsg_from_user
      5.15 ±  3%      -2.8        2.32 ± 29%  perf-profile.children.cycles-pp.ip_finish_output2
      5.03 ±  3%      -2.8        2.25 ± 31%  perf-profile.children.cycles-pp.__local_bh_enable_ip
      5.57 ±  3%      -2.8        2.80 ± 26%  perf-profile.children.cycles-pp.net_rx_action
      5.53 ±  3%      -2.8        2.78 ± 26%  perf-profile.children.cycles-pp.process_backlog
      5.54 ±  3%      -2.8        2.78 ± 26%  perf-profile.children.cycles-pp.__napi_poll
      4.93 ±  3%      -2.7        2.20 ± 30%  perf-profile.children.cycles-pp.do_softirq
      5.47 ±  3%      -2.7        2.74 ± 26%  perf-profile.children.cycles-pp.__netif_receive_skb_one_core
      5.35 ±  3%      -2.7        2.68 ± 26%  perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
      5.35 ±  3%      -2.7        2.68 ± 26%  perf-profile.children.cycles-pp.ip_local_deliver_finish
      5.30 ±  3%      -2.6        2.65 ± 26%  perf-profile.children.cycles-pp.sctp_rcv
      3.24 ±  2%      -2.5        0.71 ± 29%  perf-profile.children.cycles-pp.skb_copy_datagram_iter
      3.22 ±  2%      -2.5        0.70 ± 30%  perf-profile.children.cycles-pp.__skb_datagram_iter
      2.36 ±  7%      -2.0        0.35 ± 36%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      2.54 ±  2%      -2.0        0.57 ± 33%  perf-profile.children.cycles-pp._copy_to_iter
      8.20            -1.9        6.30 ±  7%  perf-profile.children.cycles-pp.__softirqentry_text_start
      2.40 ±  2%      -1.9        0.49 ± 34%  perf-profile.children.cycles-pp.copyout
      2.14 ±  6%      -1.7        0.46 ± 30%  perf-profile.children.cycles-pp._raw_spin_lock_bh
      2.25 ±  8%      -1.6        0.65 ± 25%  perf-profile.children.cycles-pp.sctp_skb_recv_datagram
      2.01 ±  5%      -1.5        0.46 ± 30%  perf-profile.children.cycles-pp.lock_sock_nested
      2.60 ±  3%      -1.5        1.05 ± 27%  perf-profile.children.cycles-pp.sctp_ulpq_tail_data
      2.52 ±  3%      -1.5        1.00 ± 25%  perf-profile.children.cycles-pp.sctp_chunk_put
      2.70 ±  4%      -1.5        1.23 ± 24%  perf-profile.children.cycles-pp.sctp_outq_sack
      2.44 ±  3%      -1.4        1.00 ± 28%  perf-profile.children.cycles-pp.sctp_packet_pack
      2.33 ±  6%      -1.4        0.97 ± 34%  perf-profile.children.cycles-pp.__alloc_skb
      2.58 ±  2%      -1.3        1.25 ± 42%  perf-profile.children.cycles-pp.sctp_make_datafrag_empty
      2.10 ±  4%      -1.2        0.93 ± 32%  perf-profile.children.cycles-pp.memcpy_erms
      2.19 ±  4%      -1.2        1.03 ± 38%  perf-profile.children.cycles-pp._sctp_make_chunk
      1.76 ±  5%      -1.1        0.68 ± 27%  perf-profile.children.cycles-pp.sctp_ulpevent_make_rcvmsg
      2.17 ±  7%      -1.1        1.11 ± 27%  perf-profile.children.cycles-pp.sctp_outq_flush_data
      1.74 ±  3%      -1.1        0.67 ± 26%  perf-profile.children.cycles-pp.skb_release_head_state
      1.75 ±  3%      -1.0        0.74 ± 24%  perf-profile.children.cycles-pp.consume_skb
      1.38 ±  7%      -1.0        0.39 ± 34%  perf-profile.children.cycles-pp.kfree_skb_reason
      1.60 ±  8%      -1.0        0.62 ± 41%  perf-profile.children.cycles-pp.kmalloc_reserve
      1.59 ±  9%      -1.0        0.60 ± 41%  perf-profile.children.cycles-pp.__kmalloc_node_track_caller
      1.57 ±  3%      -0.9        0.68 ± 29%  perf-profile.children.cycles-pp.sctp_user_addto_chunk
      1.27 ±  2%      -0.8        0.43 ± 31%  perf-profile.children.cycles-pp.skb_release_data
      1.49 ±  4%      -0.8        0.67 ± 26%  perf-profile.children.cycles-pp.copy_msghdr_from_user
      1.30 ±  9%      -0.7        0.60 ± 28%  perf-profile.children.cycles-pp._copy_from_user
      1.20 ±  5%      -0.7        0.52 ± 27%  perf-profile.children.cycles-pp._copy_from_iter
      1.32 ±  5%      -0.7        0.65 ± 39%  perf-profile.children.cycles-pp.__kmem_cache_alloc_node
      1.52 ±  9%      -0.7        0.86 ± 22%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      1.05 ±  5%      -0.6        0.40 ± 25%  perf-profile.children.cycles-pp.copyin
      0.96 ±  5%      -0.6        0.34 ± 32%  perf-profile.children.cycles-pp.__sk_mem_schedule
      0.92 ±  5%      -0.6        0.32 ± 33%  perf-profile.children.cycles-pp.__sk_mem_raise_allocated
      0.90 ±  8%      -0.6        0.34 ± 23%  perf-profile.children.cycles-pp.__check_object_size
      0.83 ±  3%      -0.5        0.28 ± 28%  perf-profile.children.cycles-pp.__sk_mem_reduce_allocated
      0.72 ± 13%      -0.5        0.19 ± 28%  perf-profile.children.cycles-pp.__alloc_pages
      0.68 ± 12%      -0.5        0.16 ± 26%  perf-profile.children.cycles-pp.__kmalloc_large_node
      0.90 ±  6%      -0.5        0.38 ± 28%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.77 ±  5%      -0.5        0.27 ± 38%  perf-profile.children.cycles-pp.__close
      0.72 ±  4%      -0.5        0.22 ± 33%  perf-profile.children.cycles-pp.kmem_cache_free
      0.75 ±  5%      -0.5        0.26 ± 38%  perf-profile.children.cycles-pp.task_work_run
      0.65 ± 16%      -0.5        0.16 ± 33%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.74 ±  6%      -0.5        0.26 ± 39%  perf-profile.children.cycles-pp.__fput
      0.83 ±  5%      -0.5        0.35 ± 29%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      0.75 ±  6%      -0.5        0.27 ± 36%  perf-profile.children.cycles-pp.exit_to_user_mode_loop
      0.71 ±  5%      -0.5        0.23 ± 34%  perf-profile.children.cycles-pp.__slab_free
      0.64 ± 10%      -0.5        0.17 ± 23%  perf-profile.children.cycles-pp.sctp_outq_select_transport
      0.68 ±  5%      -0.5        0.22 ± 46%  perf-profile.children.cycles-pp.sock_close
      0.68 ±  5%      -0.5        0.22 ± 46%  perf-profile.children.cycles-pp.__sock_release
      0.68 ±  5%      -0.5        0.22 ± 46%  perf-profile.children.cycles-pp.inet_release
      0.82 ± 10%      -0.5        0.36 ± 30%  perf-profile.children.cycles-pp.connect
      0.81 ±  9%      -0.5        0.36 ± 30%  perf-profile.children.cycles-pp.__x64_sys_connect
      0.81 ±  9%      -0.5        0.36 ± 30%  perf-profile.children.cycles-pp.__sys_connect
      0.68 ±  5%      -0.5        0.22 ± 46%  perf-profile.children.cycles-pp.sctp_close
      0.81 ±  9%      -0.5        0.36 ± 30%  perf-profile.children.cycles-pp.sctp_inet_connect
      0.81 ±  9%      -0.5        0.36 ± 30%  perf-profile.children.cycles-pp.__sctp_connect
      0.81 ±  3%      -0.4        0.37 ± 21%  perf-profile.children.cycles-pp.sctp_wfree
      0.55 ± 17%      -0.4        0.12 ± 38%  perf-profile.children.cycles-pp.rmqueue
      0.55 ± 15%      -0.4        0.13 ± 29%  perf-profile.children.cycles-pp.sctp_packet_config
      0.78 ±  2%      -0.4        0.36 ± 25%  perf-profile.children.cycles-pp.import_iovec
      0.66 ± 11%      -0.4        0.25 ± 34%  perf-profile.children.cycles-pp.mem_cgroup_charge_skmem
      0.75 ±  3%      -0.4        0.35 ± 24%  perf-profile.children.cycles-pp.__import_iovec
      0.69 ±  6%      -0.4        0.29 ± 27%  perf-profile.children.cycles-pp.sctp_ulpq_tail_event
      0.69 ±  5%      -0.4        0.30 ± 32%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.64 ± 10%      -0.4        0.27 ± 28%  perf-profile.children.cycles-pp.__might_resched
      0.60 ±  6%      -0.4        0.25 ± 36%  perf-profile.children.cycles-pp.sctp_chunkify
      0.38 ±  4%      -0.4        0.03 ±102%  perf-profile.children.cycles-pp.flush_smp_call_function_queue
      0.63 ±  3%      -0.3        0.29 ± 25%  perf-profile.children.cycles-pp.iovec_from_user
      0.50 ±  7%      -0.3        0.16 ± 30%  perf-profile.children.cycles-pp.check_heap_object
      0.56 ±  6%      -0.3        0.25 ± 44%  perf-profile.children.cycles-pp.___slab_alloc
      0.51 ± 14%      -0.3        0.20 ± 28%  perf-profile.children.cycles-pp.__skb_clone
      0.94 ±  8%      -0.3        0.65 ± 29%  perf-profile.children.cycles-pp.sctp_packet_transmit_chunk
      0.41 ±  3%      -0.3        0.13 ± 23%  perf-profile.children.cycles-pp.sctp_sf_eat_data_6_2
      0.53 ±  5%      -0.3        0.25 ± 33%  perf-profile.children.cycles-pp.sctp_data_ready
      0.33 ±  5%      -0.3        0.05 ± 59%  perf-profile.children.cycles-pp.__free_pages_ok
      0.40 ±  2%      -0.3        0.13 ± 37%  perf-profile.children.cycles-pp.__mod_memcg_state
      0.35 ±  4%      -0.3        0.10 ± 21%  perf-profile.children.cycles-pp.sctp_eat_data
      0.51 ± 12%      -0.2        0.26 ± 48%  perf-profile.children.cycles-pp.sctp_outq_flush_ctrl
      0.43 ± 12%      -0.2        0.19 ± 28%  perf-profile.children.cycles-pp.__entry_text_start
      0.43 ±  5%      -0.2        0.19 ± 27%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.33 ±  7%      -0.2        0.10 ± 19%  perf-profile.children.cycles-pp.simple_copy_to_iter
      0.32 ± 30%      -0.2        0.09 ± 36%  perf-profile.children.cycles-pp.try_charge_memcg
      0.39 ± 16%      -0.2        0.16 ± 24%  perf-profile.children.cycles-pp.__might_sleep
      0.32 ± 11%      -0.2        0.10 ± 16%  perf-profile.children.cycles-pp.schedule_idle
      0.34 ± 20%      -0.2        0.13 ± 22%  perf-profile.children.cycles-pp.__copy_skb_header
      0.44 ±  4%      -0.2        0.24 ± 22%  perf-profile.children.cycles-pp.__wake_up_common
      0.29 ±  7%      -0.2        0.08 ± 21%  perf-profile.children.cycles-pp.sctp_inq_pop
      0.37 ±  9%      -0.2        0.16 ± 38%  perf-profile.children.cycles-pp.memset_erms
      0.38 ± 12%      -0.2        0.18 ± 29%  perf-profile.children.cycles-pp.__might_fault
      0.25 ±  8%      -0.2        0.05 ± 72%  perf-profile.children.cycles-pp.dst_release
      0.46 ±  5%      -0.2        0.26 ± 20%  perf-profile.children.cycles-pp.__wake_up_common_lock
      0.36 ±  3%      -0.2        0.16 ± 25%  perf-profile.children.cycles-pp.sock_kmalloc
      0.29 ± 14%      -0.2        0.10 ± 42%  perf-profile.children.cycles-pp.mem_cgroup_uncharge_skmem
      0.33 ±  5%      -0.2        0.15 ± 41%  perf-profile.children.cycles-pp.sctp_wait_for_connect
      0.29 ± 12%      -0.2        0.12 ± 33%  perf-profile.children.cycles-pp.sctp_ulpevent_receive_data
      0.28 ± 16%      -0.2        0.11 ± 31%  perf-profile.children.cycles-pp.refill_stock
      0.39 ±  3%      -0.2        0.22 ± 20%  perf-profile.children.cycles-pp.autoremove_wake_function
      0.32 ±  9%      -0.2        0.15 ± 22%  perf-profile.children.cycles-pp.kmem_cache_alloc_node
      0.34 ±  4%      -0.2        0.18 ± 31%  perf-profile.children.cycles-pp.sctp_packet_append_chunk
      0.42 ±  3%      -0.2        0.26 ± 17%  perf-profile.children.cycles-pp.try_to_wake_up
      0.29 ±  8%      -0.2        0.13 ± 31%  perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.35 ±  4%      -0.2        0.19 ± 47%  perf-profile.children.cycles-pp.sctp_endpoint_bh_rcv
      0.18 ± 10%      -0.1        0.04 ±101%  perf-profile.children.cycles-pp.free_unref_page
      0.23 ±  4%      -0.1        0.09 ± 36%  perf-profile.children.cycles-pp.stress_sctp_server
      0.28 ±  8%      -0.1        0.13 ± 31%  perf-profile.children.cycles-pp.__kmalloc
      0.26 ±  9%      -0.1        0.12 ± 33%  perf-profile.children.cycles-pp.sockfd_lookup_light
      0.25 ±  3%      -0.1        0.12 ± 42%  perf-profile.children.cycles-pp.kmalloc_trace
      0.31 ± 12%      -0.1        0.18 ± 19%  perf-profile.children.cycles-pp.sctp_chunk_free
      0.20 ± 16%      -0.1        0.07 ± 69%  perf-profile.children.cycles-pp.skb_set_owner_w
      0.27 ±  4%      -0.1        0.14 ± 33%  perf-profile.children.cycles-pp.aa_sk_perm
      0.24 ±  3%      -0.1        0.12 ± 19%  perf-profile.children.cycles-pp.__kmem_cache_free
      0.18 ± 11%      -0.1        0.06 ± 84%  perf-profile.children.cycles-pp.__genradix_ptr
      0.20 ± 12%      -0.1        0.08 ± 25%  perf-profile.children.cycles-pp.skb_clone
      0.28 ± 10%      -0.1        0.16 ± 27%  perf-profile.children.cycles-pp.sctp_ulpevent_read_sndrcvinfo
      0.21 ±  5%      -0.1        0.10 ± 39%  perf-profile.children.cycles-pp.__fdget
      0.19 ±  9%      -0.1        0.07 ± 21%  perf-profile.children.cycles-pp.setsockopt
      0.25 ±  8%      -0.1        0.14 ± 28%  perf-profile.children.cycles-pp.sctp_check_transmitted
      0.15 ± 14%      -0.1        0.04 ±101%  perf-profile.children.cycles-pp.do_accept
      0.18 ± 26%      -0.1        0.06 ± 79%  perf-profile.children.cycles-pp.sctp_sendmsg_parse
      0.16 ± 13%      -0.1        0.05 ± 76%  perf-profile.children.cycles-pp.accept
      0.16 ± 16%      -0.1        0.05 ± 74%  perf-profile.children.cycles-pp.__x64_sys_accept
      0.16 ± 16%      -0.1        0.05 ± 74%  perf-profile.children.cycles-pp.__sys_accept4
      0.16 ± 16%      -0.1        0.05 ± 85%  perf-profile.children.cycles-pp.get_partial_node
      0.20 ±  6%      -0.1        0.09 ± 29%  perf-profile.children.cycles-pp.__sctp_packet_append_chunk
      0.15 ± 14%      -0.1        0.04 ± 73%  perf-profile.children.cycles-pp.__x64_sys_setsockopt
      0.13 ± 12%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.__copy_msghdr
      0.18 ± 16%      -0.1        0.08 ± 29%  perf-profile.children.cycles-pp.sock_kfree_s
      0.15 ± 17%      -0.1        0.04 ± 73%  perf-profile.children.cycles-pp.__sys_setsockopt
      0.21 ±  2%      -0.1        0.10 ± 24%  perf-profile.children.cycles-pp.sock_recvmsg
      0.19 ±  5%      -0.1        0.08 ± 14%  perf-profile.children.cycles-pp.enqueue_entity
      0.24 ± 12%      -0.1        0.14 ± 29%  perf-profile.children.cycles-pp.put_cmsg
      0.22 ± 12%      -0.1        0.12 ± 14%  perf-profile.children.cycles-pp.__dev_queue_xmit
      0.16 ±  6%      -0.1        0.06 ± 47%  perf-profile.children.cycles-pp.sock_wfree
      0.19 ± 10%      -0.1        0.09 ± 43%  perf-profile.children.cycles-pp.__virt_addr_valid
      0.18 ±  5%      -0.1        0.09 ± 26%  perf-profile.children.cycles-pp.sctp_sched_fcfs_dequeue
      0.20 ± 15%      -0.1        0.10 ± 32%  perf-profile.children.cycles-pp.dequeue_entity
      0.21 ±  8%      -0.1        0.11 ± 18%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.21 ±  9%      -0.1        0.11 ± 37%  perf-profile.children.cycles-pp.sctp_association_put
      0.19 ±  7%      -0.1        0.10 ± 29%  perf-profile.children.cycles-pp.skb_put
      0.17 ±  8%      -0.1        0.08 ± 31%  perf-profile.children.cycles-pp.kfree
      0.16 ±  5%      -0.1        0.07 ± 56%  perf-profile.children.cycles-pp.security_socket_sendmsg
      0.15 ±  5%      -0.1        0.06 ± 51%  perf-profile.children.cycles-pp.sctp_sched_dequeue_common
      0.18 ±  2%      -0.1        0.09 ± 22%  perf-profile.children.cycles-pp.security_socket_recvmsg
      0.12 ± 11%      -0.1        0.03 ±101%  perf-profile.children.cycles-pp.inet_accept
      0.13 ± 16%      -0.1        0.04 ± 73%  perf-profile.children.cycles-pp.sctp_setsockopt
      0.13 ± 18%      -0.1        0.04 ± 77%  perf-profile.children.cycles-pp.sctp_ulpq_order
      0.12 ± 16%      -0.1        0.03 ±106%  perf-profile.children.cycles-pp.sctp_ulpevent_init
      0.14 ± 17%      -0.1        0.05 ± 55%  perf-profile.children.cycles-pp.sctp_connect_new_asoc
      0.14 ±  3%      -0.1        0.05 ± 73%  perf-profile.children.cycles-pp.sctp_outq_tail
      0.11 ± 12%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.sctp_accept
      0.21 ±  8%      -0.1        0.13 ± 17%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.15 ± 10%      -0.1        0.08 ± 33%  perf-profile.children.cycles-pp.sctp_association_hold
      0.13 ±  3%      -0.1        0.06 ± 53%  perf-profile.children.cycles-pp.__ksize
      0.17 ± 12%      -0.1        0.10 ± 34%  perf-profile.children.cycles-pp.memcg_slab_post_alloc_hook
      0.12 ± 27%      -0.1        0.05 ± 46%  perf-profile.children.cycles-pp.__check_heap_object
      0.14 ± 21%      -0.1        0.07 ± 18%  perf-profile.children.cycles-pp.dev_hard_start_xmit
      0.13 ± 16%      -0.1        0.06 ± 78%  perf-profile.children.cycles-pp.sctp_chunk_hold
      0.12 ±  3%      -0.1        0.05 ± 49%  perf-profile.children.cycles-pp.__x64_sys_getsockopt
      0.11 ± 11%      -0.1        0.04 ±109%  perf-profile.children.cycles-pp.__put_user_nocheck_8
      0.15 ±  8%      -0.1        0.08 ± 17%  perf-profile.children.cycles-pp.getsockopt
      0.13 ± 26%      -0.1        0.06 ± 20%  perf-profile.children.cycles-pp.loopback_xmit
      0.10 ±  8%      -0.1        0.03 ±108%  perf-profile.children.cycles-pp.cgroup_rstat_updated
      0.15 ± 14%      -0.1        0.08 ± 24%  perf-profile.children.cycles-pp.sctp_gen_sack
      0.09 ± 13%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.__socket
      0.09 ± 13%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.__sys_socket
      0.09 ± 13%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.__x64_sys_socket
      0.21 ±  2%      -0.1        0.15 ±  8%  perf-profile.children.cycles-pp.update_load_avg
      0.12 ± 14%      -0.1        0.05 ± 76%  perf-profile.children.cycles-pp.rcu_all_qs
      0.11 ±  8%      -0.1        0.05 ± 49%  perf-profile.children.cycles-pp.__sys_getsockopt
      0.15 ± 11%      -0.1        0.09 ± 29%  perf-profile.children.cycles-pp.sctp_make_sack
      0.09 ±  9%      -0.1        0.03 ±103%  perf-profile.children.cycles-pp.sctp_chunk_abandoned
      0.09 ± 30%      -0.1        0.04 ± 75%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.09 ± 10%      -0.1        0.04 ± 75%  perf-profile.children.cycles-pp.sctp_getsockopt
      0.10 ±  9%      -0.0        0.05 ± 45%  perf-profile.children.cycles-pp.sctp_primitive_SHUTDOWN
      0.08 ±  5%      -0.0        0.03 ± 70%  perf-profile.children.cycles-pp.prepare_task_switch
      0.13 ± 12%      +0.0        0.16 ± 12%  perf-profile.children.cycles-pp.calc_global_load_tick
      0.04 ± 71%      +0.1        0.11 ± 17%  perf-profile.children.cycles-pp.smpboot_thread_fn
      0.02 ±141%      +0.1        0.09 ± 14%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.23 ±  5%      +0.1        0.31 ± 11%  perf-profile.children.cycles-pp.ct_kernel_exit_state
      0.02 ±141%      +0.1        0.10 ± 33%  perf-profile.children.cycles-pp.update_rt_rq_load_avg
      0.07 ± 18%      +0.1        0.16 ± 52%  perf-profile.children.cycles-pp.ct_nmi_enter
      0.25 ± 13%      +0.1        0.35 ± 14%  perf-profile.children.cycles-pp.memcpy_toio
      0.25 ± 13%      +0.1        0.35 ± 15%  perf-profile.children.cycles-pp.drm_fb_helper_damage_work
      0.25 ± 13%      +0.1        0.35 ± 15%  perf-profile.children.cycles-pp.drm_fb_helper_damage_blit_real
      0.11 ±  4%      +0.1        0.21 ± 33%  perf-profile.children.cycles-pp.sctp_assoc_rwnd_increase
      0.30 ± 13%      +0.1        0.40 ± 13%  perf-profile.children.cycles-pp.process_one_work
      0.09 ± 15%      +0.1        0.20 ± 28%  perf-profile.children.cycles-pp._find_next_and_bit
      0.32 ±  8%      +0.1        0.44 ± 10%  perf-profile.children.cycles-pp.worker_thread
      0.25            +0.1        0.38 ± 22%  perf-profile.children.cycles-pp.ct_kernel_enter
      0.32 ± 19%      +0.1        0.45 ±  9%  perf-profile.children.cycles-pp.io_serial_in
      0.46 ± 16%      +0.1        0.60 ± 10%  perf-profile.children.cycles-pp.rcu_core
      0.23 ±  7%      +0.1        0.38 ± 14%  perf-profile.children.cycles-pp.arch_cpu_idle_enter
      0.18 ±  9%      +0.2        0.33 ± 20%  perf-profile.children.cycles-pp.idle_cpu
      0.26            +0.2        0.42 ± 21%  perf-profile.children.cycles-pp.update_irq_load_avg
      0.25 ± 10%      +0.2        0.41 ± 13%  perf-profile.children.cycles-pp.note_gp_changes
      0.20 ± 10%      +0.2        0.36 ± 17%  perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
      0.43 ±  3%      +0.2        0.60 ±  4%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      1.27 ±  3%      +0.2        1.44 ±  8%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.58 ±  4%      +0.2        0.76 ±  8%  perf-profile.children.cycles-pp.update_rq_clock
      0.34            +0.2        0.52 ± 26%  perf-profile.children.cycles-pp.ct_idle_exit
      0.44 ± 18%      +0.2        0.63 ± 12%  perf-profile.children.cycles-pp.wait_for_lsr
      0.45 ±  9%      +0.2        0.64 ±  6%  perf-profile.children.cycles-pp.ret_from_fork
      0.44 ± 10%      +0.2        0.64 ±  6%  perf-profile.children.cycles-pp.kthread
      0.45 ± 16%      +0.2        0.65 ± 13%  perf-profile.children.cycles-pp.serial8250_console_write
      0.50 ± 12%      +0.2        0.71 ± 13%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.46 ± 17%      +0.2        0.66 ± 11%  perf-profile.children.cycles-pp.asm_sysvec_irq_work
      0.46 ± 17%      +0.2        0.66 ± 11%  perf-profile.children.cycles-pp.sysvec_irq_work
      0.46 ± 17%      +0.2        0.66 ± 11%  perf-profile.children.cycles-pp.__sysvec_irq_work
      0.46 ± 17%      +0.2        0.66 ± 11%  perf-profile.children.cycles-pp.irq_work_run
      0.46 ± 17%      +0.2        0.66 ± 11%  perf-profile.children.cycles-pp.irq_work_single
      0.46 ± 17%      +0.2        0.66 ± 11%  perf-profile.children.cycles-pp._printk
      0.46 ± 17%      +0.2        0.66 ± 11%  perf-profile.children.cycles-pp.vprintk_emit
      0.46 ± 17%      +0.2        0.66 ± 11%  perf-profile.children.cycles-pp.console_unlock
      0.46 ± 17%      +0.2        0.66 ± 11%  perf-profile.children.cycles-pp.console_emit_next_record
      0.40 ± 10%      +0.2        0.62 ± 23%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      1.05 ±  3%      +0.2        1.28 ±  7%  perf-profile.children.cycles-pp.native_sched_clock
      0.48 ± 19%      +0.2        0.71 ± 11%  perf-profile.children.cycles-pp.irq_work_run_list
      1.29 ±  2%      +0.2        1.54 ±  9%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.08 ± 20%      +0.3        0.37 ± 63%  perf-profile.children.cycles-pp.check_cpu_stall
      0.71 ±  4%      +0.3        1.01 ± 11%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.74 ±  4%      +0.3        1.06 ± 24%  perf-profile.children.cycles-pp.schedule
      0.34 ±  2%      +0.3        0.65 ± 39%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.46 ± 34%      +0.4        0.83 ± 33%  perf-profile.children.cycles-pp.start_kernel
      0.46 ± 34%      +0.4        0.83 ± 33%  perf-profile.children.cycles-pp.arch_call_rest_init
      0.46 ± 34%      +0.4        0.83 ± 33%  perf-profile.children.cycles-pp.rest_init
      0.37 ±  8%      +0.4        0.76 ± 25%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.19 ± 16%      +0.4        0.58 ± 26%  perf-profile.children.cycles-pp.sctp_wait_for_sndbuf
      0.31 ±  9%      +0.4        0.75 ± 27%  perf-profile.children.cycles-pp.newidle_balance
      0.26 ±  6%      +0.4        0.70 ± 40%  perf-profile.children.cycles-pp.rcu_pending
      1.04 ±  4%      +0.5        1.50 ±  6%  perf-profile.children.cycles-pp.rebalance_domains
      0.51 ± 15%      +0.5        1.03 ± 14%  perf-profile.children.cycles-pp.update_sg_lb_stats
      2.33 ± 14%      +0.6        2.89 ± 10%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.34 ±  2%      +0.6        0.91 ± 41%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.62 ± 12%      +0.6        1.26 ±  9%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.64 ± 18%      +0.6        1.27 ± 37%  perf-profile.children.cycles-pp.tick_irq_enter
      0.65 ± 10%      +0.6        1.30 ±  8%  perf-profile.children.cycles-pp.find_busiest_group
      1.59 ±  3%      +0.8        2.36 ± 10%  perf-profile.children.cycles-pp.scheduler_tick
      0.85 ±  8%      +0.8        1.66 ±  7%  perf-profile.children.cycles-pp.load_balance
      3.82 ±  3%      +1.1        4.90 ±  7%  perf-profile.children.cycles-pp.__irq_exit_rcu
      2.48 ±  5%      +1.9        4.36 ± 23%  perf-profile.children.cycles-pp.update_process_times
      2.56 ±  5%      +2.2        4.76 ± 28%  perf-profile.children.cycles-pp.tick_sched_handle
      3.13 ±  3%      +2.4        5.52 ± 24%  perf-profile.children.cycles-pp.tick_sched_timer
      4.07 ±  8%      +2.5        6.53 ± 15%  perf-profile.children.cycles-pp.menu_select
      5.17 ±  3%      +3.5        8.70 ± 25%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      9.32 ±  2%      +3.7       12.99 ± 11%  perf-profile.children.cycles-pp.hrtimer_interrupt
      9.38 ±  2%      +3.8       13.22 ± 12%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
     15.33 ±  2%      +5.9       21.22 ± 12%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
     18.70            +6.0       24.66 ±  9%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
     31.40           +11.9       43.30 ±  5%  perf-profile.children.cycles-pp.mwait_idle_with_hints
     31.04           +12.6       43.59 ±  5%  perf-profile.children.cycles-pp.intel_idle
     54.74           +19.4       74.17 ±  4%  perf-profile.children.cycles-pp.cpuidle_enter_state
     54.81           +19.5       74.31 ±  4%  perf-profile.children.cycles-pp.cpuidle_enter
     60.62           +21.9       82.56 ±  5%  perf-profile.children.cycles-pp.start_secondary
     61.07           +22.3       83.39 ±  5%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     61.07           +22.3       83.39 ±  5%  perf-profile.children.cycles-pp.cpu_startup_entry
     61.07           +22.3       83.38 ±  5%  perf-profile.children.cycles-pp.do_idle
     59.68           +22.6       82.26 ±  5%  perf-profile.children.cycles-pp.cpuidle_idle_call
      4.21 ±  2%      -2.9        1.30 ± 29%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      2.35 ±  7%      -2.0        0.35 ± 36%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      2.09 ±  4%      -1.2        0.90 ± 29%  perf-profile.self.cycles-pp.memcpy_erms
      0.99 ±  6%      -0.5        0.47 ± 28%  perf-profile.self.cycles-pp.sctp_datamsg_from_user
      0.68 ±  7%      -0.5        0.21 ± 33%  perf-profile.self.cycles-pp.kmem_cache_free
      0.69 ±  5%      -0.5        0.22 ± 34%  perf-profile.self.cycles-pp.__slab_free
      0.58 ± 10%      -0.3        0.24 ± 29%  perf-profile.self.cycles-pp.__might_resched
      0.45 ±  6%      -0.3        0.14 ± 39%  perf-profile.self.cycles-pp.sctp_recvmsg
      0.50 ±  4%      -0.3        0.24 ± 28%  perf-profile.self.cycles-pp.__kmem_cache_alloc_node
      0.30 ± 10%      -0.2        0.07 ± 47%  perf-profile.self.cycles-pp.check_heap_object
      0.38 ± 10%      -0.2        0.16 ± 32%  perf-profile.self.cycles-pp._raw_spin_lock_bh
      0.34 ± 20%      -0.2        0.12 ± 23%  perf-profile.self.cycles-pp.__copy_skb_header
      0.38 ±  3%      -0.2        0.17 ± 28%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.37 ±  9%      -0.2        0.16 ± 40%  perf-profile.self.cycles-pp.memset_erms
      0.25 ±  6%      -0.2        0.05 ± 72%  perf-profile.self.cycles-pp.dst_release
      0.32 ±  7%      -0.2        0.13 ± 25%  perf-profile.self.cycles-pp.sctp_chunk_put
      0.26 ± 15%      -0.2        0.07 ± 31%  perf-profile.self.cycles-pp.__sk_mem_raise_allocated
      0.27 ±  6%      -0.2        0.08 ± 36%  perf-profile.self.cycles-pp.__mod_memcg_state
      0.27 ±  3%      -0.2        0.08 ± 30%  perf-profile.self.cycles-pp.sctp_packet_pack
      0.30 ±  8%      -0.2        0.12 ± 38%  perf-profile.self.cycles-pp.sctp_sendmsg
      0.25 ± 11%      -0.2        0.08 ± 27%  perf-profile.self.cycles-pp.__sk_mem_reduce_allocated
      0.22 ± 11%      -0.2        0.04 ± 72%  perf-profile.self.cycles-pp.sctp_eat_data
      0.24 ± 13%      -0.2        0.06 ± 47%  perf-profile.self.cycles-pp.sctp_packet_config
      0.31 ±  3%      -0.2        0.14 ± 31%  perf-profile.self.cycles-pp.sctp_outq_flush_data
      0.28 ±  7%      -0.2        0.13 ± 29%  perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.30 ±  4%      -0.2        0.15 ± 52%  perf-profile.self.cycles-pp.___slab_alloc
      0.28            -0.2        0.13 ± 28%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.29 ±  2%      -0.2        0.14 ± 21%  perf-profile.self.cycles-pp.__might_sleep
      0.22 ±  3%      -0.1        0.08 ± 54%  perf-profile.self.cycles-pp.stress_sctp_server
      0.22 ±  4%      -0.1        0.08 ± 75%  perf-profile.self.cycles-pp.mem_cgroup_charge_skmem
      0.27 ± 10%      -0.1        0.13 ± 26%  perf-profile.self.cycles-pp.sctp_sendmsg_to_asoc
      0.20 ± 16%      -0.1        0.06 ± 88%  perf-profile.self.cycles-pp.skb_set_owner_w
      0.17 ± 12%      -0.1        0.03 ±102%  perf-profile.self.cycles-pp.sctp_skb_recv_datagram
      0.25 ± 11%      -0.1        0.12 ± 34%  perf-profile.self.cycles-pp.__alloc_skb
      0.24 ±  5%      -0.1        0.11 ± 20%  perf-profile.self.cycles-pp.__kmem_cache_free
      0.20 ± 12%      -0.1        0.08 ± 41%  perf-profile.self.cycles-pp.sendmsg
      0.18 ±  9%      -0.1        0.06 ± 54%  perf-profile.self.cycles-pp.sctp_ulpevent_receive_data
      0.20 ± 12%      -0.1        0.08 ± 52%  perf-profile.self.cycles-pp._copy_from_user
      0.16 ±  6%      -0.1        0.04 ± 73%  perf-profile.self.cycles-pp.sock_wfree
      0.15 ± 21%      -0.1        0.04 ±102%  perf-profile.self.cycles-pp.sctp_chunkify
      0.21 ±  6%      -0.1        0.10 ± 41%  perf-profile.self.cycles-pp.__fdget
      0.24 ± 13%      -0.1        0.13 ± 24%  perf-profile.self.cycles-pp.sctp_cmd_interpreter
      0.19 ± 18%      -0.1        0.08 ± 39%  perf-profile.self.cycles-pp.__list_add_valid
      0.17 ± 17%      -0.1        0.06 ± 68%  perf-profile.self.cycles-pp.try_charge_memcg
      0.17 ± 24%      -0.1        0.06 ± 79%  perf-profile.self.cycles-pp.sctp_sendmsg_parse
      0.24 ± 12%      -0.1        0.13 ± 26%  perf-profile.self.cycles-pp.put_cmsg
      0.16 ±  5%      -0.1        0.06 ± 47%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.19 ±  2%      -0.1        0.08 ± 22%  perf-profile.self.cycles-pp.kmem_cache_alloc_node
      0.15 ± 17%      -0.1        0.04 ±101%  perf-profile.self.cycles-pp.sctp_data_ready
      0.19 ± 17%      -0.1        0.09 ± 23%  perf-profile.self.cycles-pp.__check_object_size
      0.13 ±  9%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.__copy_msghdr
      0.19 ± 22%      -0.1        0.09 ± 27%  perf-profile.self.cycles-pp.refill_stock
      0.17 ±  9%      -0.1        0.08 ± 60%  perf-profile.self.cycles-pp.__virt_addr_valid
      0.16 ± 17%      -0.1        0.06 ± 60%  perf-profile.self.cycles-pp.sctp_do_sm
      0.16 ±  7%      -0.1        0.07 ± 58%  perf-profile.self.cycles-pp.__skb_clone
      0.19 ±  6%      -0.1        0.09 ± 38%  perf-profile.self.cycles-pp.sctp_wfree
      0.17 ±  4%      -0.1        0.08 ± 28%  perf-profile.self.cycles-pp.recvmsg
      0.15 ±  3%      -0.1        0.06 ± 82%  perf-profile.self.cycles-pp.__genradix_ptr
      0.16 ± 10%      -0.1        0.07 ± 62%  perf-profile.self.cycles-pp.skb_release_data
      0.16 ± 10%      -0.1        0.07 ± 52%  perf-profile.self.cycles-pp.__cond_resched
      0.11 ± 14%      -0.1        0.03 ±102%  perf-profile.self.cycles-pp.do_syscall_64
      0.16 ± 10%      -0.1        0.08 ± 51%  perf-profile.self.cycles-pp.skb_put
      0.12 ± 26%      -0.1        0.04 ± 72%  perf-profile.self.cycles-pp.__check_heap_object
      0.13 ± 22%      -0.1        0.05 ± 56%  perf-profile.self.cycles-pp.__entry_text_start
      0.14 ±  8%      -0.1        0.07 ± 31%  perf-profile.self.cycles-pp.kfree
      0.12 ± 20%      -0.1        0.05 ± 78%  perf-profile.self.cycles-pp.sctp_chunk_hold
      0.13 ±  6%      -0.1        0.06 ± 51%  perf-profile.self.cycles-pp.__import_iovec
      0.10 ±  9%      -0.1        0.03 ±111%  perf-profile.self.cycles-pp.__put_user_nocheck_8
      0.13 ±  7%      -0.1        0.06 ± 53%  perf-profile.self.cycles-pp.__ksize
      0.14 ±  9%      -0.1        0.07 ± 52%  perf-profile.self.cycles-pp.memcg_slab_post_alloc_hook
      0.09 ± 26%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.__sctp_packet_append_chunk
      0.14 ±  6%      -0.1        0.08 ± 39%  perf-profile.self.cycles-pp.aa_sk_perm
      0.13 ± 18%      -0.1        0.07 ± 35%  perf-profile.self.cycles-pp.sctp_association_hold
      0.16 ± 13%      -0.1        0.10 ± 34%  perf-profile.self.cycles-pp.sctp_association_put
      0.10 ±  9%      -0.1        0.04 ± 75%  perf-profile.self.cycles-pp.sctp_outq_tail
      0.09 ± 14%      -0.1        0.03 ±105%  perf-profile.self.cycles-pp._sctp_make_chunk
      0.10 ± 21%      -0.1        0.04 ± 76%  perf-profile.self.cycles-pp.__local_bh_enable_ip
      0.09 ± 30%      -0.1        0.04 ± 75%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.12 ±  6%      -0.1        0.06 ± 24%  perf-profile.self.cycles-pp.sctp_check_transmitted
      0.10 ± 12%      -0.0        0.06 ± 49%  perf-profile.self.cycles-pp.____sys_sendmsg
      0.12 ± 13%      +0.0        0.15 ± 11%  perf-profile.self.cycles-pp.calc_global_load_tick
      0.09 ± 24%      +0.0        0.12 ± 22%  perf-profile.self.cycles-pp.tick_sched_timer
      0.07 ± 23%      +0.0        0.11 ± 17%  perf-profile.self.cycles-pp.trigger_load_balance
      0.02 ±141%      +0.0        0.06 ± 11%  perf-profile.self.cycles-pp.hrtimer_forward
      0.22 ±  5%      +0.0        0.27 ±  9%  perf-profile.self.cycles-pp.ct_kernel_exit_state
      0.15 ± 17%      +0.1        0.22 ± 19%  perf-profile.self.cycles-pp.update_rq_clock
      0.02 ±141%      +0.1        0.09 ± 14%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.02 ±141%      +0.1        0.10 ± 35%  perf-profile.self.cycles-pp.update_rt_rq_load_avg
      0.14 ± 11%      +0.1        0.22 ± 23%  perf-profile.self.cycles-pp.ct_kernel_enter
      0.07 ± 11%      +0.1        0.15 ± 42%  perf-profile.self.cycles-pp.update_sd_lb_stats
      0.09 ± 14%      +0.1        0.17 ± 10%  perf-profile.self.cycles-pp.note_gp_changes
      0.08 ± 14%      +0.1        0.17 ± 43%  perf-profile.self.cycles-pp.tick_irq_enter
      0.00            +0.1        0.09 ± 37%  perf-profile.self.cycles-pp.rcu_core
      0.11 ± 25%      +0.1        0.20 ± 22%  perf-profile.self.cycles-pp.rebalance_domains
      0.18 ±  9%      +0.1        0.28 ± 11%  perf-profile.self.cycles-pp.tsc_verify_tsc_adjust
      0.02 ±141%      +0.1        0.12 ± 28%  perf-profile.self.cycles-pp.cpuidle_enter
      0.24 ±  7%      +0.1        0.35 ±  4%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.09 ± 15%      +0.1        0.20 ± 14%  perf-profile.self.cycles-pp.load_balance
      0.08 ± 22%      +0.1        0.19 ± 31%  perf-profile.self.cycles-pp._find_next_and_bit
      0.17 ± 12%      +0.1        0.29 ± 20%  perf-profile.self.cycles-pp.irq_work_tick
      0.29 ±  4%      +0.1        0.42 ± 16%  perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
      0.17 ±  7%      +0.1        0.30 ± 17%  perf-profile.self.cycles-pp.idle_cpu
      0.32 ± 19%      +0.1        0.45 ±  9%  perf-profile.self.cycles-pp.io_serial_in
      0.26            +0.2        0.41 ± 22%  perf-profile.self.cycles-pp.update_irq_load_avg
      0.12 ± 17%      +0.2        0.27 ± 40%  perf-profile.self.cycles-pp.get_next_timer_interrupt
      0.16 ±  5%      +0.2        0.32 ± 18%  perf-profile.self.cycles-pp.rcu_pending
      0.10 ± 12%      +0.2        0.28 ± 54%  perf-profile.self.cycles-pp.update_process_times
      1.01 ±  3%      +0.2        1.21 ±  6%  perf-profile.self.cycles-pp.native_sched_clock
      0.26 ± 12%      +0.2        0.46 ± 14%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.09 ± 18%      +0.3        0.34 ± 67%  perf-profile.self.cycles-pp.timerqueue_add
      0.08 ± 20%      +0.3        0.36 ± 65%  perf-profile.self.cycles-pp.check_cpu_stall
      0.58            +0.3        0.87 ±  7%  perf-profile.self.cycles-pp.cpuidle_idle_call
      0.35 ±  8%      +0.4        0.78 ± 16%  perf-profile.self.cycles-pp.update_sg_lb_stats
      4.33            +0.4        4.78 ±  9%  perf-profile.self.cycles-pp.cpuidle_enter_state
      1.43 ± 34%      +1.6        3.07 ± 33%  perf-profile.self.cycles-pp.menu_select
     31.31           +12.0       43.29 ±  5%  perf-profile.self.cycles-pp.mwait_idle_with_hints



***************************************************************************************************
lkp-cfl-e1: 16 threads 1 sockets Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz (Coffee Lake) with 32G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase:
  gcc-11/performance/x86_64-rhel-8.3/30%/debian-11.1-x86_64-20220510.cgz/300s/lkp-cfl-e1/context1/unixbench

commit: 
  c36eae5a86 ("sched/psi: Fix possible missing or delayed pending event")
  b6aabb01e3 ("sched: consider WF_SYNC to find idle siblings")

c36eae5a86d15a69 b6aabb01e3004b0285d953761e1 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      3547           +36.2%       4834        unixbench.score
      1202 ±  4%  +3.2e+05%    3838646 ± 17%  unixbench.time.involuntary_context_switches
    146.17            +4.7%     153.00        unixbench.time.percent_of_cpu_this_job_got
    533.06            +1.7%     542.03        unixbench.time.system_time
     40.02           +40.3%      56.14        unixbench.time.user_time
 4.156e+08           +34.9%  5.604e+08        unixbench.time.voluntary_context_switches
 5.537e+08           +36.3%  7.548e+08        unixbench.workload
      1.53            +0.5        2.01        mpstat.cpu.all.usr%
 4.434e+09           +13.2%  5.022e+09        cpuidle..time
 8.352e+08           -98.7%   11131648 ± 12%  cpuidle..usage
      3.00           +33.3%       4.00        vmstat.procs.r
   4212747           -32.1%    2861762        vmstat.system.cs
     40960           -29.9%      28703        vmstat.system.in
    161422           -32.1%     109641        meminfo.Active
    161345           -32.1%     109569        meminfo.Active(anon)
    530075           -10.9%     472307        meminfo.Committed_AS
     80181           -27.2%      58384        meminfo.Mapped
    162280           -32.2%     110057        meminfo.Shmem
     40427           -31.8%      27576        proc-vmstat.nr_active_anon
    978441            -1.3%     965308        proc-vmstat.nr_file_pages
     20137           -26.7%      14769        proc-vmstat.nr_mapped
     40661           -31.9%      27701        proc-vmstat.nr_shmem
     13329            -0.9%      13206        proc-vmstat.nr_slab_unreclaimable
     40427           -31.8%      27576        proc-vmstat.nr_zone_active_anon
   1751058           -14.2%    1502883        proc-vmstat.numa_hit
   1751425           -14.1%    1504002        proc-vmstat.numa_local
     92726           -17.3%      76685        proc-vmstat.pgactivate
   1819543           -13.7%    1570329        proc-vmstat.pgalloc_normal
    684646            -2.4%     668006        proc-vmstat.pgfault
   1807696           -13.7%    1559418        proc-vmstat.pgfree
     40680            -1.8%      39968        proc-vmstat.pgreuse
      1740           -47.5%     912.83        turbostat.Avg_MHz
     38.54           -18.6       19.90        turbostat.Busy%
      4515            +1.6%       4586        turbostat.Bzy_MHz
   3885351 ±  9%     -82.8%     667034 ± 73%  turbostat.C1
     26.41 ±  6%     +50.7%      39.80 ± 19%  turbostat.CPU%c6
     74.05           -31.1%      50.99 ±  2%  turbostat.CorWatt
     60.17           -18.3%      49.17 ±  2%  turbostat.CoreTmp
      0.22 ±  2%     +83.6%       0.41        turbostat.IPC
  16161117           -30.0%   11319624        turbostat.IRQ
 8.241e+08          -100.0%       3881 ± 26%  turbostat.POLL
      9.50            -9.5        0.00        turbostat.POLL%
     58.67 ±  2%     -17.0%      48.67 ±  2%  turbostat.PkgTmp
     74.52           -30.9%      51.46 ±  2%  turbostat.PkgWatt
    489.94           -35.7%     315.24        turbostat.Totl%C0
      4851 ± 56%    -100.0%       0.00        sched_debug.cfs_rq:/.MIN_vruntime.avg
     56902 ± 44%    -100.0%       0.00        sched_debug.cfs_rq:/.MIN_vruntime.max
     15406 ± 47%    -100.0%       0.00        sched_debug.cfs_rq:/.MIN_vruntime.stddev
      0.44 ±  9%     -14.4%       0.38 ±  2%  sched_debug.cfs_rq:/.h_nr_running.avg
    127353           +10.5%     140708 ±  3%  sched_debug.cfs_rq:/.load.stddev
    120.76 ±  8%     +25.4%     151.41 ± 17%  sched_debug.cfs_rq:/.load_avg.avg
      4851 ± 56%    -100.0%       0.00        sched_debug.cfs_rq:/.max_vruntime.avg
     56902 ± 44%    -100.0%       0.00        sched_debug.cfs_rq:/.max_vruntime.max
     15406 ± 47%    -100.0%       0.00        sched_debug.cfs_rq:/.max_vruntime.stddev
    197189            -9.1%     179171 ±  2%  sched_debug.cfs_rq:/.min_vruntime.max
    116754 ±  6%     +12.9%     131851 ±  3%  sched_debug.cfs_rq:/.min_vruntime.min
     21217 ± 10%     -39.9%      12759 ± 18%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.44 ±  7%     -18.5%       0.36 ±  3%  sched_debug.cfs_rq:/.nr_running.avg
    366.23           +21.1%     443.60 ±  3%  sched_debug.cfs_rq:/.runnable_avg.avg
    714.74 ±  4%     +81.8%       1299 ±  3%  sched_debug.cfs_rq:/.runnable_avg.max
    246.75 ±  5%    +101.3%     496.81 ±  3%  sched_debug.cfs_rq:/.runnable_avg.stddev
     51496 ± 44%     -66.9%      17048 ± 49%  sched_debug.cfs_rq:/.spread0.max
     21217 ± 10%     -39.9%      12760 ± 18%  sched_debug.cfs_rq:/.spread0.stddev
    708.50 ±  4%     +49.8%       1061 ±  4%  sched_debug.cfs_rq:/.util_avg.max
    245.43 ±  5%     +63.7%     401.71 ±  3%  sched_debug.cfs_rq:/.util_avg.stddev
    107.96 ± 11%     +42.0%     153.34 ±  4%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    499.12 ±  2%     +65.3%     825.00 ±  4%  sched_debug.cfs_rq:/.util_est_enqueued.max
    181.05 ±  4%     +58.9%     287.67 ±  4%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    407511 ±  4%    +110.5%     857930        sched_debug.cpu.avg_idle.avg
    104621          +462.1%     588066 ± 11%  sched_debug.cpu.avg_idle.min
    350206 ±  2%     -68.5%     110282 ± 21%  sched_debug.cpu.avg_idle.stddev
      0.35 ± 18%     +89.2%       0.67 ±  3%  sched_debug.cpu.clock.stddev
      0.00 ± 37%     +68.1%       0.00 ±  9%  sched_debug.cpu.next_balance.stddev
      1.07 ±  6%     +55.6%       1.67 ±  9%  sched_debug.cpu.nr_running.max
      0.46 ±  3%     +24.0%       0.57 ±  4%  sched_debug.cpu.nr_running.stddev
  47178388           -32.4%   31875847        sched_debug.cpu.nr_switches.avg
  56953632 ±  2%     -36.6%   36093819 ±  2%  sched_debug.cpu.nr_switches.max
  31935712 ±  9%     -13.3%   27680720 ±  3%  sched_debug.cpu.nr_switches.min
   6579393 ± 14%     -64.6%    2330670 ± 21%  sched_debug.cpu.nr_switches.stddev
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.avg
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.max
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.min
     65.64           -30.8%      45.42        perf-stat.i.MPKI
 4.603e+09            -7.6%  4.252e+09        perf-stat.i.branch-instructions
      2.81            +0.1        2.89        perf-stat.i.branch-miss-rate%
  49172917            -1.4%   48506007        perf-stat.i.branch-misses
      0.79            +2.6        3.39 ±  7%  perf-stat.i.cache-miss-rate%
   1887466            -4.2%    1808731 ±  2%  perf-stat.i.cache-misses
 5.404e+08           -94.1%   32037290 ±  3%  perf-stat.i.cache-references
   4247507           -32.1%    2884302        perf-stat.i.context-switches
      1.99           -24.4%       1.50        perf-stat.i.cpi
 2.784e+10           -47.7%  1.457e+10        perf-stat.i.cpu-cycles
     52.17 ±  2%    +634.1%     383.01 ±  4%  perf-stat.i.cpu-migrations
     56513 ±  2%     -59.7%      22750 ±  9%  perf-stat.i.cycles-between-cache-misses
    410439 ±  2%     +23.7%     507559 ±  7%  perf-stat.i.dTLB-load-misses
 6.418e+09            -3.1%   6.22e+09        perf-stat.i.dTLB-loads
 3.826e+09            -4.0%  3.674e+09        perf-stat.i.dTLB-stores
     95.43           -38.9       56.58        perf-stat.i.iTLB-load-miss-rate%
  16498563            -5.9%   15523524        perf-stat.i.iTLB-load-misses
    239550 ±  3%   +5782.0%   14090457        perf-stat.i.iTLB-loads
 2.197e+10            -4.8%  2.092e+10        perf-stat.i.instructions
      0.70           +78.1%       1.25        perf-stat.i.ipc
      1.74           -47.7%       0.91        perf-stat.i.metric.GHz
    404.88 ±  2%      -4.8%     385.49 ±  2%  perf-stat.i.metric.K/sec
    961.54            -7.8%     886.77        perf-stat.i.metric.M/sec
      1575            -2.7%       1533        perf-stat.i.minor-faults
      0.40 ±  4%      +0.1        0.47 ±  7%  perf-stat.i.node-load-miss-rate%
    261859           -14.5%     223787        perf-stat.i.node-stores
      1575            -2.7%       1533        perf-stat.i.page-faults
     24.60           -93.8%       1.53 ±  3%  perf-stat.overall.MPKI
      1.07            +0.1        1.14        perf-stat.overall.branch-miss-rate%
      0.35            +5.3        5.67 ±  5%  perf-stat.overall.cache-miss-rate%
      1.27           -45.0%       0.70        perf-stat.overall.cpi
     14726           -45.3%       8048 ±  2%  perf-stat.overall.cycles-between-cache-misses
      0.01 ±  2%      +0.0        0.01 ±  7%  perf-stat.overall.dTLB-load-miss-rate%
     98.57           -46.1       52.42        perf-stat.overall.iTLB-load-miss-rate%
      1331            +1.2%       1347        perf-stat.overall.instructions-per-iTLB-miss
      0.79           +81.9%       1.44        perf-stat.overall.ipc
      0.00 ± 11%      +0.0        0.01 ± 10%  perf-stat.overall.node-store-miss-rate%
     15528           -30.1%      10847        perf-stat.overall.path-length
 4.591e+09            -7.6%   4.24e+09        perf-stat.ps.branch-instructions
  49052787            -1.4%   48388408        perf-stat.ps.branch-misses
   1885579            -4.2%    1807271 ±  2%  perf-stat.ps.cache-misses
 5.389e+08           -94.1%   31961586 ±  3%  perf-stat.ps.cache-references
   4235546           -32.1%    2876157        perf-stat.ps.context-switches
 2.777e+10           -47.6%  1.454e+10        perf-stat.ps.cpu-cycles
     52.05 ±  2%    +633.9%     381.95 ±  4%  perf-stat.ps.cpu-migrations
    409410 ±  2%     +23.7%     506263 ±  7%  perf-stat.ps.dTLB-load-misses
   6.4e+09            -3.1%  6.203e+09        perf-stat.ps.dTLB-loads
 3.815e+09            -4.0%  3.664e+09        perf-stat.ps.dTLB-stores
  16452159            -5.9%   15479733        perf-stat.ps.iTLB-load-misses
    238902 ±  3%   +5781.4%   14050699        perf-stat.ps.iTLB-loads
  2.19e+10            -4.8%  2.086e+10        perf-stat.ps.instructions
      1571            -2.7%       1529        perf-stat.ps.minor-faults
    261165           -14.5%     223199        perf-stat.ps.node-stores
      1571            -2.7%       1529        perf-stat.ps.page-faults
 8.598e+12            -4.8%  8.188e+12        perf-stat.total.instructions
     57.34           -35.5       21.87        perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     53.70 ±  2%     -32.8       20.91        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     53.66 ±  2%     -32.8       20.91        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     53.51 ±  2%     -32.6       20.91        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     21.60           -21.6        0.00        perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     38.36 ±  2%     -17.5       20.86        perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     35.56 ±  2%     -14.1       21.44        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
     33.37 ±  2%     -12.8       20.57        perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
      7.88 ±  2%      -7.9        0.00        perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      5.64            -5.6        0.00        perf-profile.calltrace.cycles-pp.sched_ttwu_pending.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary
      3.63 ± 24%      -2.7        0.96 ± 31%  perf-profile.calltrace.cycles-pp.start_kernel.secondary_startup_64_no_verify
      3.63 ± 24%      -2.7        0.96 ± 31%  perf-profile.calltrace.cycles-pp.arch_call_rest_init.start_kernel.secondary_startup_64_no_verify
      3.63 ± 24%      -2.7        0.96 ± 31%  perf-profile.calltrace.cycles-pp.rest_init.arch_call_rest_init.start_kernel.secondary_startup_64_no_verify
      3.63 ± 24%      -2.7        0.96 ± 31%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.rest_init.arch_call_rest_init.start_kernel.secondary_startup_64_no_verify
      3.62 ± 24%      -2.7        0.96 ± 31%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.rest_init.arch_call_rest_init.start_kernel
      2.64 ± 18%      -1.7        0.96 ± 31%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.rest_init.arch_call_rest_init
      2.31 ± 15%      -1.4        0.94 ± 31%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.rest_init
      4.08 ±  2%      -1.4        2.73        perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.pipe_read
      1.90 ±  3%      -0.8        1.11 ±  5%  perf-profile.calltrace.cycles-pp.select_task_rq_fair.select_task_rq.try_to_wake_up.autoremove_wake_function.__wake_up_common
      2.12 ±  2%      -0.8        1.32 ±  3%  perf-profile.calltrace.cycles-pp.select_task_rq.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      0.86 ±  4%      -0.6        0.26 ±100%  perf-profile.calltrace.cycles-pp.copy_user_short_string.copyout._copy_to_iter.copy_page_to_iter.pipe_read
      1.60 ±  2%      -0.4        1.16 ±  3%  perf-profile.calltrace.cycles-pp.update_curr.dequeue_entity.dequeue_task_fair.__schedule.schedule
      1.12 ±  2%      -0.4        0.70 ±  3%  perf-profile.calltrace.cycles-pp.mutex_lock.pipe_write.vfs_write.ksys_write.do_syscall_64
      0.97 ±  4%      -0.2        0.75 ±  5%  perf-profile.calltrace.cycles-pp.update_load_avg.dequeue_entity.dequeue_task_fair.__schedule.schedule
      1.20 ±  4%      -0.2        1.00 ±  5%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.prepare_to_wait_event.pipe_read.vfs_read.ksys_read
      1.04 ±  4%      -0.1        0.90 ±  5%  perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.copy_page_to_iter.pipe_read.vfs_read
      1.31 ±  3%      +0.1        1.40 ±  2%  perf-profile.calltrace.cycles-pp._copy_to_iter.copy_page_to_iter.pipe_read.vfs_read.ksys_read
      1.41            +0.1        1.53        perf-profile.calltrace.cycles-pp.mutex_lock.pipe_read.vfs_read.ksys_read.do_syscall_64
      1.04            +0.2        1.21 ±  4%  perf-profile.calltrace.cycles-pp.perf_trace_sched_wakeup_template.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      2.68 ±  2%      +0.2        2.85        perf-profile.calltrace.cycles-pp.prepare_to_wait_event.pipe_read.vfs_read.ksys_read.do_syscall_64
      0.56 ±  4%      +0.2        0.80 ±  7%  perf-profile.calltrace.cycles-pp.security_file_permission.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.42 ±  2%      +0.3        1.70        perf-profile.calltrace.cycles-pp.copy_page_to_iter.pipe_read.vfs_read.ksys_read.do_syscall_64
      0.26 ±100%      +0.4        0.69 ±  6%  perf-profile.calltrace.cycles-pp.apparmor_file_permission.security_file_permission.vfs_write.ksys_write.do_syscall_64
      0.56 ±  3%      +0.5        1.07 ±  8%  perf-profile.calltrace.cycles-pp.prepare_task_switch.__schedule.schedule.pipe_read.vfs_read
      0.00            +0.6        0.56 ± 11%  perf-profile.calltrace.cycles-pp.__fdget_pos.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      0.00            +0.6        0.56 ±  7%  perf-profile.calltrace.cycles-pp.__might_resched.__might_fault._copy_from_iter.copy_page_from_iter.pipe_write
      0.00            +0.6        0.58 ±  3%  perf-profile.calltrace.cycles-pp.apparmor_file_permission.security_file_permission.vfs_read.ksys_read.do_syscall_64
      0.00            +0.6        0.58 ±  5%  perf-profile.calltrace.cycles-pp.native_sched_clock.sched_clock_cpu.update_rq_clock.__schedule.schedule
      0.00            +0.6        0.59 ±  3%  perf-profile.calltrace.cycles-pp.native_sched_clock.sched_clock_cpu.update_rq_clock.try_to_wake_up.autoremove_wake_function
      0.00            +0.6        0.62 ±  6%  perf-profile.calltrace.cycles-pp.sched_clock_cpu.update_rq_clock.__schedule.schedule.pipe_read
      0.00            +0.6        0.63 ±  2%  perf-profile.calltrace.cycles-pp.sched_clock_cpu.update_rq_clock.try_to_wake_up.autoremove_wake_function.__wake_up_common
      0.00            +0.6        0.63 ±  4%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.finish_wait.pipe_read.vfs_read.ksys_read
      0.00            +0.7        0.66 ± 10%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
      0.00            +0.7        0.66 ±  4%  perf-profile.calltrace.cycles-pp.__wrgsbase_inactive.read
      0.00            +0.7        0.68 ±  8%  perf-profile.calltrace.cycles-pp.___perf_sw_event.prepare_task_switch.__schedule.schedule.pipe_read
      0.00            +0.7        0.69 ±  4%  perf-profile.calltrace.cycles-pp.security_file_permission.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.7        0.70 ±  6%  perf-profile.calltrace.cycles-pp.update_load_avg.dequeue_task_fair.__schedule.schedule.pipe_read
      0.00            +0.7        0.70 ±  8%  perf-profile.calltrace.cycles-pp.mutex_unlock.pipe_read.vfs_read.ksys_read.do_syscall_64
      0.65 ±  5%      +0.7        1.37 ±  3%  perf-profile.calltrace.cycles-pp.os_xsave.read
      1.26 ±  4%      +0.7        1.98 ±  2%  perf-profile.calltrace.cycles-pp._copy_from_iter.copy_page_from_iter.pipe_write.vfs_write.ksys_write
      0.00            +0.7        0.72 ± 10%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      0.00            +0.7        0.74 ±  5%  perf-profile.calltrace.cycles-pp.copyin._copy_from_iter.copy_page_from_iter.pipe_write.vfs_write
      0.00            +0.8        0.76 ±  3%  perf-profile.calltrace.cycles-pp.update_load_avg.set_next_entity.pick_next_task_fair.__schedule.schedule
      0.00            +0.8        0.76 ±  6%  perf-profile.calltrace.cycles-pp.__calc_delta.update_curr.reweight_entity.dequeue_task_fair.__schedule
      0.00            +0.8        0.80 ±  4%  perf-profile.calltrace.cycles-pp.__calc_delta.update_curr.reweight_entity.enqueue_task_fair.ttwu_do_activate
      0.00            +0.8        0.81 ±  4%  perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up
      0.00            +0.8        0.82 ±  9%  perf-profile.calltrace.cycles-pp.__switch_to.read
      0.00            +0.8        0.82 ±  4%  perf-profile.calltrace.cycles-pp.finish_wait.pipe_read.vfs_read.ksys_read.do_syscall_64
      0.00            +0.9        0.87 ±  3%  perf-profile.calltrace.cycles-pp.update_rq_clock.__schedule.schedule.pipe_read.vfs_read
      0.00            +0.9        0.88 ±  4%  perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      0.00            +0.9        0.89 ±  3%  perf-profile.calltrace.cycles-pp.main
      0.00            +0.9        0.91 ±  5%  perf-profile.calltrace.cycles-pp.update_cfs_group.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      0.00            +0.9        0.91 ±  5%  perf-profile.calltrace.cycles-pp.update_rq_clock.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      0.00            +0.9        0.94 ±  2%  perf-profile.calltrace.cycles-pp.__switch_to.__schedule.schedule.pipe_read.vfs_read
      1.43 ±  3%      +1.0        2.38 ±  2%  perf-profile.calltrace.cycles-pp.copy_page_from_iter.pipe_write.vfs_write.ksys_write.do_syscall_64
      0.00            +1.0        0.95 ±  2%  perf-profile.calltrace.cycles-pp.update_cfs_group.dequeue_task_fair.__schedule.schedule.pipe_read
      0.00            +1.0        1.01 ±  5%  perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.dequeue_task_fair.__schedule.schedule
      0.00            +1.0        1.01 ±  4%  perf-profile.calltrace.cycles-pp.check_preempt_wakeup.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function
      0.91 ±  2%      +1.0        1.93 ±  3%  perf-profile.calltrace.cycles-pp.__switch_to_asm.read
      0.00            +1.0        1.04 ±  4%  perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up
      0.00            +1.2        1.18 ±  3%  perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule.pipe_read
      0.00            +1.3        1.27 ±  4%  perf-profile.calltrace.cycles-pp.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common
      0.00            +1.3        1.30 ±  4%  perf-profile.calltrace.cycles-pp.update_curr.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up
      0.00            +1.4        1.38 ±  3%  perf-profile.calltrace.cycles-pp.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      0.54 ±  3%      +1.4        1.94 ±  3%  perf-profile.calltrace.cycles-pp.__entry_text_start.read
      0.00            +1.6        1.58 ±  3%  perf-profile.calltrace.cycles-pp.restore_fpregs_from_fpstate.switch_fpu_return.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      0.17 ±141%      +1.7        1.84 ±  3%  perf-profile.calltrace.cycles-pp.__entry_text_start.write
      0.00            +1.7        1.67 ±  3%  perf-profile.calltrace.cycles-pp.reweight_entity.dequeue_task_fair.__schedule.schedule.pipe_read
      0.00            +1.7        1.68 ±  2%  perf-profile.calltrace.cycles-pp.reweight_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      0.60 ±  5%      +2.1        2.66 ±  2%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      0.00            +2.2        2.21 ±  2%  perf-profile.calltrace.cycles-pp.switch_fpu_return.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.58 ±  6%      +2.2        2.82        perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.pipe_read.vfs_read
      0.00            +2.3        2.30 ±  3%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      4.70            +2.4        7.11        perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.pipe_read.vfs_read
      0.00            +3.1        3.15 ±  3%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      0.00            +5.1        5.05 ±  2%  perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__schedule.schedule.pipe_read.vfs_read
      0.00            +7.3        7.32        perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
      7.83            +7.3       15.18        perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.vfs_write.ksys_write
      0.00            +7.5        7.52        perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      6.88 ±  2%      +7.6       14.49        perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write
      8.24            +7.8       16.01        perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.vfs_write.ksys_write.do_syscall_64
      7.11            +7.8       14.94        perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.vfs_write
     11.86 ±  6%      +8.6       20.50 ±  2%  perf-profile.calltrace.cycles-pp.mwait_idle_with_hints.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
     11.86 ±  6%      +8.6       20.51 ±  2%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     12.29            +9.5       21.75        perf-profile.calltrace.cycles-pp.pipe_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     13.40           +10.0       23.36        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     13.83           +10.4       24.21        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     14.17           +10.9       25.10        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     14.33           +11.1       25.46        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
      9.16           +11.3       20.49        perf-profile.calltrace.cycles-pp.__schedule.schedule.pipe_read.vfs_read.ksys_read
      9.44           +11.9       21.39        perf-profile.calltrace.cycles-pp.schedule.pipe_read.vfs_read.ksys_read.do_syscall_64
     15.53           +13.3       28.86        perf-profile.calltrace.cycles-pp.write
     18.03           +14.7       32.74        perf-profile.calltrace.cycles-pp.pipe_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
     19.37           +15.7       35.10        perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
     19.90           +16.3       36.21        perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
     20.88           +18.6       39.51        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
     21.30           +19.5       40.84        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
     24.96           +24.5       49.46        perf-profile.calltrace.cycles-pp.read
     57.34           -35.5       21.87        perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     57.34           -35.5       21.87        perf-profile.children.cycles-pp.cpu_startup_entry
     57.17           -35.3       21.87        perf-profile.children.cycles-pp.do_idle
     53.70 ±  2%     -32.8       20.91        perf-profile.children.cycles-pp.start_secondary
     21.62           -21.6        0.00        perf-profile.children.cycles-pp.poll_idle
     41.05           -19.2       21.83        perf-profile.children.cycles-pp.cpuidle_idle_call
     35.70 ±  2%     -14.2       21.51        perf-profile.children.cycles-pp.cpuidle_enter
     35.64 ±  2%     -14.1       21.51        perf-profile.children.cycles-pp.cpuidle_enter_state
      8.40            -8.4        0.00        perf-profile.children.cycles-pp.flush_smp_call_function_queue
      6.02            -6.0        0.00        perf-profile.children.cycles-pp.sched_ttwu_pending
      5.20 ±  2%      -5.2        0.00        perf-profile.children.cycles-pp.schedule_idle
      4.01            -3.8        0.25 ± 11%  perf-profile.children.cycles-pp.menu_select
      3.63 ± 24%      -2.7        0.96 ± 31%  perf-profile.children.cycles-pp.start_kernel
      3.63 ± 24%      -2.7        0.96 ± 31%  perf-profile.children.cycles-pp.arch_call_rest_init
      3.63 ± 24%      -2.7        0.96 ± 31%  perf-profile.children.cycles-pp.rest_init
      2.59 ±  3%      -2.3        0.32 ± 10%  perf-profile.children.cycles-pp.ttwu_queue_wakelist
      1.83 ±  2%      -1.7        0.09 ± 20%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      4.18 ±  2%      -1.4        2.81        perf-profile.children.cycles-pp.dequeue_entity
      1.13 ±  2%      -1.1        0.04 ± 72%  perf-profile.children.cycles-pp.tick_nohz_next_event
      1.92 ±  3%      -0.8        1.11 ±  5%  perf-profile.children.cycles-pp.select_task_rq_fair
      2.12 ±  2%      -0.8        1.34 ±  4%  perf-profile.children.cycles-pp.select_task_rq
      0.72 ±  3%      -0.7        0.06 ± 47%  perf-profile.children.cycles-pp.ktime_get
      1.53 ±  2%      -0.6        0.90 ±  4%  perf-profile.children.cycles-pp._raw_spin_lock
      1.86            -0.5        1.33        perf-profile.children.cycles-pp.sched_clock_cpu
      1.63 ±  2%      -0.4        1.20        perf-profile.children.cycles-pp.native_sched_clock
      3.03 ±  3%      -0.4        2.65 ±  3%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      1.45 ±  3%      -0.4        1.09 ±  8%  perf-profile.children.cycles-pp.prepare_task_switch
      2.58            -0.3        2.30        perf-profile.children.cycles-pp.mutex_lock
      0.53 ±  4%      -0.2        0.34 ±  6%  perf-profile.children.cycles-pp.finish_task_switch
      1.05 ±  4%      -0.1        0.90 ±  5%  perf-profile.children.cycles-pp.copyout
      1.38 ±  5%      -0.1        1.26 ±  3%  perf-profile.children.cycles-pp.set_next_entity
      0.17 ± 10%      -0.1        0.09 ±  4%  perf-profile.children.cycles-pp.rcu_note_context_switch
      0.18 ±  9%      -0.0        0.13 ± 16%  perf-profile.children.cycles-pp.place_entity
      0.10 ±  6%      -0.0        0.05 ± 48%  perf-profile.children.cycles-pp.rb_next
      0.08 ± 11%      -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.sched_clock
      0.06 ± 11%      +0.0        0.10 ± 13%  perf-profile.children.cycles-pp.rw_verify_area
      0.08 ± 11%      +0.0        0.12 ± 11%  perf-profile.children.cycles-pp.__x64_sys_read
      0.30 ±  8%      +0.0        0.34 ±  8%  perf-profile.children.cycles-pp.perf_trace_buf_update
      0.02 ±141%      +0.0        0.06 ± 11%  perf-profile.children.cycles-pp.clockevents_program_event
      0.06 ± 45%      +0.0        0.10 ± 13%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.04 ± 71%      +0.0        0.08 ±  8%  perf-profile.children.cycles-pp.perf_swevent_get_recursion_context
      0.04 ± 71%      +0.1        0.09 ± 14%  perf-profile.children.cycles-pp.inode_needs_update_time
      0.03 ±102%      +0.1        0.08 ± 19%  perf-profile.children.cycles-pp.save_fpregs_to_fpstate
      0.22 ±  6%      +0.1        0.28 ± 11%  perf-profile.children.cycles-pp.put_prev_entity
      0.01 ±223%      +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.08 ± 14%      +0.1        0.13 ± 16%  perf-profile.children.cycles-pp.__irq_exit_rcu
      0.12 ±  6%      +0.1        0.17 ±  4%  perf-profile.children.cycles-pp.perf_trace_buf_alloc
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.default_wake_function
      0.31 ±  5%      +0.1        0.38 ±  6%  perf-profile.children.cycles-pp.__rdgsbase_inactive
      0.09 ±  9%      +0.1        0.16 ± 10%  perf-profile.children.cycles-pp.__x64_sys_write
      0.09 ± 10%      +0.1        0.16 ±  8%  perf-profile.children.cycles-pp.rcu_all_qs
      0.04 ± 44%      +0.1        0.12 ± 16%  perf-profile.children.cycles-pp.page_copy_sane
      0.04 ± 71%      +0.1        0.11 ± 18%  perf-profile.children.cycles-pp.scheduler_tick
      0.30 ±  8%      +0.1        0.38 ±  4%  perf-profile.children.cycles-pp.pick_next_entity
      0.11 ± 15%      +0.1        0.20 ±  7%  perf-profile.children.cycles-pp.syscall_enter_from_user_mode
      0.09 ± 15%      +0.1        0.18 ± 17%  perf-profile.children.cycles-pp.update_process_times
      0.05 ± 46%      +0.1        0.14 ± 14%  perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64
      0.07 ± 11%      +0.1        0.15 ± 15%  perf-profile.children.cycles-pp.perf_trace_run_bpf_submit
      0.06 ± 11%      +0.1        0.15 ± 12%  perf-profile.children.cycles-pp.kill_fasync
      0.10 ± 16%      +0.1        0.20 ± 17%  perf-profile.children.cycles-pp.tick_sched_handle
      0.04 ± 72%      +0.1        0.14 ± 34%  perf-profile.children.cycles-pp.write@plt
      0.12 ± 19%      +0.1        0.23 ± 18%  perf-profile.children.cycles-pp.tick_sched_timer
      0.00            +0.1        0.12 ± 36%  perf-profile.children.cycles-pp.exit_to_user_mode_loop
      0.19 ±  9%      +0.1        0.31 ±  9%  perf-profile.children.cycles-pp.__get_task_ioprio
      0.57 ±  6%      +0.1        0.68 ±  4%  perf-profile.children.cycles-pp.__wrgsbase_inactive
      0.13 ± 15%      +0.1        0.25 ±  7%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
      1.32 ±  3%      +0.1        1.44 ±  2%  perf-profile.children.cycles-pp._copy_to_iter
      1.54 ±  2%      +0.1        1.67        perf-profile.children.cycles-pp.__might_resched
      0.00            +0.1        0.14 ± 11%  perf-profile.children.cycles-pp.cgroup_rstat_updated
      0.14 ±  9%      +0.2        0.30 ±  6%  perf-profile.children.cycles-pp.aa_file_perm
      0.09 ± 20%      +0.2        0.26 ±  5%  perf-profile.children.cycles-pp.__cgroup_account_cputime
      0.14 ± 13%      +0.2        0.31 ± 12%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.14 ± 11%      +0.2        0.31 ±  5%  perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
      0.20 ± 14%      +0.2        0.37 ± 12%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.21 ±  9%      +0.2        0.38 ±  5%  perf-profile.children.cycles-pp.__cond_resched
      1.06            +0.2        1.23 ±  4%  perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
      0.06 ± 19%      +0.2        0.24 ±  8%  perf-profile.children.cycles-pp.check_cfs_rq_runtime
      0.22 ±  7%      +0.2        0.41 ±  6%  perf-profile.children.cycles-pp.atime_needs_update
      0.00            +0.2        0.22 ±  6%  perf-profile.children.cycles-pp.set_next_buddy
      2.70 ±  2%      +0.2        2.92        perf-profile.children.cycles-pp.prepare_to_wait_event
      0.28 ± 11%      +0.2        0.51 ±  9%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.88 ±  5%      +0.2        1.12 ±  4%  perf-profile.children.cycles-pp.__might_fault
      0.22 ±  8%      +0.2        0.46 ±  6%  perf-profile.children.cycles-pp.current_time
      0.28 ±  7%      +0.2        0.52 ±  5%  perf-profile.children.cycles-pp.touch_atime
      0.30 ± 11%      +0.2        0.55 ±  8%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      0.21 ±  9%      +0.2        0.46 ±  8%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.22 ±  8%      +0.2        0.47 ±  6%  perf-profile.children.cycles-pp.file_update_time
      0.62 ±  2%      +0.3        0.88 ±  3%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      2.96 ±  2%      +0.3        3.23 ±  3%  perf-profile.children.cycles-pp.enqueue_entity
      1.44 ±  2%      +0.3        1.72        perf-profile.children.cycles-pp.copy_page_to_iter
      1.53 ±  3%      +0.3        1.81 ±  3%  perf-profile.children.cycles-pp.__switch_to
      0.26 ±  6%      +0.3        0.57 ±  5%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.21 ±  7%      +0.3        0.53 ±  7%  perf-profile.children.cycles-pp.update_min_vruntime
      2.68 ±  5%      +0.3        3.02        perf-profile.children.cycles-pp.pick_next_task_fair
      0.12 ±  8%      +0.3        0.46 ±  4%  perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
      0.42 ±  8%      +0.4        0.78 ±  9%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      0.73 ±  6%      +0.4        1.11 ±  5%  perf-profile.children.cycles-pp.mutex_unlock
      0.48 ±  7%      +0.4        0.86 ±  8%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.87 ±  2%      +0.4        1.27 ±  4%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.90 ±  3%      +0.4        1.30 ±  4%  perf-profile.children.cycles-pp.apparmor_file_permission
      0.14 ± 15%      +0.4        0.54 ±  2%  perf-profile.children.cycles-pp.__list_add_valid
      0.89 ±  4%      +0.4        1.30 ±  3%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.37 ±  7%      +0.4        0.78 ±  8%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.33 ±  4%      +0.4        0.75 ±  5%  perf-profile.children.cycles-pp.copyin
      0.36 ±  4%      +0.5        0.82 ±  4%  perf-profile.children.cycles-pp.finish_wait
      1.03 ±  3%      +0.5        1.50 ±  4%  perf-profile.children.cycles-pp.security_file_permission
      0.18 ± 16%      +0.5        0.68 ±  7%  perf-profile.children.cycles-pp.cpuacct_charge
      0.40 ±  6%      +0.5        0.93 ±  3%  perf-profile.children.cycles-pp.main
      0.53 ±  2%      +0.6        1.12 ±  7%  perf-profile.children.cycles-pp.__fdget_pos
      0.65 ±  5%      +0.7        1.38 ±  3%  perf-profile.children.cycles-pp.os_xsave
      1.30 ±  4%      +0.7        2.04 ±  2%  perf-profile.children.cycles-pp._copy_from_iter
      0.65 ±  3%      +0.8        1.41 ±  2%  perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
      1.02 ±  2%      +0.8        1.82 ±  2%  perf-profile.children.cycles-pp.update_rq_clock
      1.44 ±  3%      +1.0        2.42 ±  2%  perf-profile.children.cycles-pp.copy_page_from_iter
      0.35 ±  5%      +1.0        1.38 ±  3%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.00            +1.0        1.04 ±  5%  perf-profile.children.cycles-pp.check_preempt_wakeup
      0.21 ±  7%      +1.1        1.28 ±  4%  perf-profile.children.cycles-pp.check_preempt_curr
      0.46 ±  5%      +1.1        1.56 ±  2%  perf-profile.children.cycles-pp.__calc_delta
      0.83 ±  5%      +1.1        1.96 ±  2%  perf-profile.children.cycles-pp.update_cfs_group
      2.68 ±  4%      +1.3        3.98 ±  3%  perf-profile.children.cycles-pp.update_load_avg
      1.02            +1.4        2.47 ±  2%  perf-profile.children.cycles-pp.__entry_text_start
      0.00            +1.6        1.59 ±  3%  perf-profile.children.cycles-pp.restore_fpregs_from_fpstate
      0.36 ±  6%      +1.9        2.23 ±  3%  perf-profile.children.cycles-pp.switch_fpu_return
      0.49 ±  4%      +2.1        2.61 ±  4%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      4.72            +2.4        7.12        perf-profile.children.cycles-pp.dequeue_task_fair
      0.74 ±  6%      +2.4        3.15 ±  3%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.63 ±  4%      +2.7        3.35        perf-profile.children.cycles-pp.reweight_entity
      1.74 ±  3%      +3.2        4.91 ±  2%  perf-profile.children.cycles-pp.update_curr
      3.34 ±  2%      +4.0        7.34        perf-profile.children.cycles-pp.enqueue_task_fair
      3.44 ±  2%      +4.1        7.54        perf-profile.children.cycles-pp.ttwu_do_activate
      0.26 ±  6%      +4.9        5.12 ±  3%  perf-profile.children.cycles-pp.switch_mm_irqs_off
     14.17 ±  2%      +6.6       20.77        perf-profile.children.cycles-pp.__schedule
      7.84            +7.4       15.19        perf-profile.children.cycles-pp.__wake_up_common
      6.90            +7.7       14.56        perf-profile.children.cycles-pp.try_to_wake_up
      8.26            +7.8       16.03        perf-profile.children.cycles-pp.__wake_up_common_lock
      7.12            +7.9       14.98        perf-profile.children.cycles-pp.autoremove_wake_function
     11.94 ±  6%      +8.6       20.59 ±  2%  perf-profile.children.cycles-pp.mwait_idle_with_hints
     11.95 ±  6%      +8.6       20.59 ±  2%  perf-profile.children.cycles-pp.intel_idle
     12.32            +9.5       21.81        perf-profile.children.cycles-pp.pipe_write
     13.43           +10.0       23.43        perf-profile.children.cycles-pp.vfs_write
     13.85           +10.4       24.25        perf-profile.children.cycles-pp.ksys_write
      9.45           +12.1       21.52        perf-profile.children.cycles-pp.schedule
     15.82           +13.1       28.92        perf-profile.children.cycles-pp.write
     18.16           +14.9       33.02        perf-profile.children.cycles-pp.pipe_read
     19.41           +15.7       35.11        perf-profile.children.cycles-pp.vfs_read
     19.92           +16.3       36.24        perf-profile.children.cycles-pp.ksys_read
     25.25           +24.2       49.48        perf-profile.children.cycles-pp.read
     35.17           +29.7       64.91        perf-profile.children.cycles-pp.do_syscall_64
     35.72           +30.8       66.50        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     21.21           -21.2        0.00        perf-profile.self.cycles-pp.poll_idle
      1.76 ±  2%      -1.6        0.14 ± 11%  perf-profile.self.cycles-pp.menu_select
      1.52 ±  2%      -0.6        0.90 ±  4%  perf-profile.self.cycles-pp._raw_spin_lock
      0.72 ±  5%      -0.5        0.23 ± 11%  perf-profile.self.cycles-pp.__wake_up_common
      2.98 ±  3%      -0.4        2.55 ±  2%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      1.58 ±  2%      -0.4        1.16        perf-profile.self.cycles-pp.native_sched_clock
      0.78 ±  2%      -0.4        0.40 ± 14%  perf-profile.self.cycles-pp.prepare_task_switch
      0.32 ±  8%      -0.2        0.11 ± 13%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.47 ±  5%      -0.2        0.28 ±  8%  perf-profile.self.cycles-pp.finish_task_switch
      1.24            -0.2        1.06 ±  3%  perf-profile.self.cycles-pp.mutex_lock
      1.03 ±  3%      -0.2        0.86 ±  2%  perf-profile.self.cycles-pp.enqueue_entity
      0.46 ± 13%      -0.2        0.31 ± 12%  perf-profile.self.cycles-pp.ttwu_queue_wakelist
      0.20 ±  5%      -0.1        0.10 ± 11%  perf-profile.self.cycles-pp.sched_clock_cpu
      0.16 ±  8%      -0.1        0.08 ±  6%  perf-profile.self.cycles-pp.rcu_note_context_switch
      0.38 ±  9%      -0.1        0.31 ± 10%  perf-profile.self.cycles-pp.set_next_entity
      0.35 ±  6%      -0.1        0.29 ±  7%  perf-profile.self.cycles-pp.perf_tp_event
      0.16 ±  6%      -0.0        0.11 ± 13%  perf-profile.self.cycles-pp.put_prev_entity
      0.16 ±  7%      -0.0        0.12 ± 14%  perf-profile.self.cycles-pp.place_entity
      0.09 ±  7%      -0.0        0.05 ± 47%  perf-profile.self.cycles-pp.rb_next
      0.07 ± 13%      +0.0        0.09 ± 11%  perf-profile.self.cycles-pp.perf_trace_buf_alloc
      0.08 ± 12%      +0.0        0.11 ± 13%  perf-profile.self.cycles-pp.__x64_sys_read
      0.14 ±  3%      +0.0        0.17 ± 10%  perf-profile.self.cycles-pp.__might_fault
      0.20 ±  9%      +0.0        0.24 ±  8%  perf-profile.self.cycles-pp.select_task_rq
      0.02 ±141%      +0.0        0.06 ±  8%  perf-profile.self.cycles-pp.perf_swevent_event
      0.03 ± 70%      +0.0        0.08 ± 14%  perf-profile.self.cycles-pp.inode_needs_update_time
      0.07 ± 11%      +0.0        0.12 ± 14%  perf-profile.self.cycles-pp.perf_trace_buf_update
      0.06 ± 17%      +0.1        0.12 ±  9%  perf-profile.self.cycles-pp.rcu_all_qs
      0.23 ±  7%      +0.1        0.28 ±  3%  perf-profile.self.cycles-pp.perf_trace_sched_wakeup_template
      0.06 ±  7%      +0.1        0.12 ± 12%  perf-profile.self.cycles-pp.finish_wait
      0.08 ± 11%      +0.1        0.14 ± 14%  perf-profile.self.cycles-pp.__x64_sys_write
      0.11 ± 11%      +0.1        0.18 ±  5%  perf-profile.self.cycles-pp.atime_needs_update
      0.31 ±  4%      +0.1        0.38 ±  6%  perf-profile.self.cycles-pp.__rdgsbase_inactive
      0.05 ± 51%      +0.1        0.12 ± 14%  perf-profile.self.cycles-pp.__cgroup_account_cputime
      0.01 ±223%      +0.1        0.08 ±  8%  perf-profile.self.cycles-pp.perf_swevent_get_recursion_context
      0.05 ± 46%      +0.1        0.12 ± 14%  perf-profile.self.cycles-pp.touch_atime
      0.01 ±223%      +0.1        0.08 ± 17%  perf-profile.self.cycles-pp.rw_verify_area
      0.15 ±  8%      +0.1        0.23 ±  6%  perf-profile.self.cycles-pp.check_preempt_curr
      0.13 ± 11%      +0.1        0.20 ±  3%  perf-profile.self.cycles-pp.security_file_permission
      0.06 ± 11%      +0.1        0.14 ± 13%  perf-profile.self.cycles-pp.perf_trace_run_bpf_submit
      0.04 ± 71%      +0.1        0.12 ± 34%  perf-profile.self.cycles-pp.write@plt
      0.10 ±  6%      +0.1        0.19 ± 13%  perf-profile.self.cycles-pp.ttwu_do_activate
      0.05 ± 46%      +0.1        0.14 ± 18%  perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64
      0.42 ±  7%      +0.1        0.50 ±  7%  perf-profile.self.cycles-pp.update_rq_clock
      0.10 ± 11%      +0.1        0.20 ±  7%  perf-profile.self.cycles-pp.syscall_enter_from_user_mode
      0.02 ± 99%      +0.1        0.12 ± 15%  perf-profile.self.cycles-pp.kill_fasync
      0.11 ± 12%      +0.1        0.20 ±  9%  perf-profile.self.cycles-pp.__cond_resched
      0.10 ±  3%      +0.1        0.20 ±  9%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode
      0.06 ± 11%      +0.1        0.16 ± 14%  perf-profile.self.cycles-pp.file_update_time
      0.00            +0.1        0.10 ± 15%  perf-profile.self.cycles-pp.page_copy_sane
      0.18 ±  7%      +0.1        0.28 ±  8%  perf-profile.self.cycles-pp.__get_task_ioprio
      0.11 ±  8%      +0.1        0.22 ± 20%  perf-profile.self.cycles-pp.__wake_up_common_lock
      0.12 ± 17%      +0.1        0.23 ±  5%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode_prepare
      0.57 ±  6%      +0.1        0.68 ±  4%  perf-profile.self.cycles-pp.__wrgsbase_inactive
      0.12 ±  6%      +0.1        0.24 ±  9%  perf-profile.self.cycles-pp.exit_to_user_mode_prepare
      0.10 ±  8%      +0.1        0.22 ±  4%  perf-profile.self.cycles-pp.copy_page_to_iter
      0.00            +0.1        0.13 ± 10%  perf-profile.self.cycles-pp.cgroup_rstat_updated
      0.12 ± 10%      +0.1        0.25 ±  8%  perf-profile.self.cycles-pp.ksys_write
      0.17 ±  9%      +0.1        0.30 ±  7%  perf-profile.self.cycles-pp.autoremove_wake_function
      1.54 ±  2%      +0.1        1.67        perf-profile.self.cycles-pp.__might_resched
      0.14 ± 14%      +0.1        0.28 ± 13%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.13 ± 12%      +0.1        0.27 ±  6%  perf-profile.self.cycles-pp.aa_file_perm
      0.16 ± 13%      +0.1        0.30 ±  4%  perf-profile.self.cycles-pp.current_time
      0.14 ±  3%      +0.2        0.30 ±  8%  perf-profile.self.cycles-pp._copy_to_iter
      0.14 ± 11%      +0.2        0.31 ±  5%  perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
      0.03 ±101%      +0.2        0.21 ± 10%  perf-profile.self.cycles-pp.check_cfs_rq_runtime
      0.12 ± 10%      +0.2        0.30 ±  7%  perf-profile.self.cycles-pp.copy_page_from_iter
      0.41 ±  4%      +0.2        0.60 ±  5%  perf-profile.self.cycles-pp.vfs_write
      0.54 ±  4%      +0.2        0.76        perf-profile.self.cycles-pp.dequeue_task_fair
      0.00            +0.2        0.22 ±  6%  perf-profile.self.cycles-pp.set_next_buddy
      0.38 ±  9%      +0.2        0.62 ±  5%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.25 ±  5%      +0.2        0.50        perf-profile.self.cycles-pp._copy_from_iter
      0.21 ±  7%      +0.3        0.46 ±  8%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.61            +0.3        0.87 ±  3%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.76 ±  5%      +0.3        1.02 ±  4%  perf-profile.self.cycles-pp.apparmor_file_permission
      0.35 ±  6%      +0.3        0.62 ±  4%  perf-profile.self.cycles-pp.switch_fpu_return
      1.48 ±  3%      +0.3        1.76 ±  2%  perf-profile.self.cycles-pp.__switch_to
      0.18 ±  7%      +0.3        0.46 ±  6%  perf-profile.self.cycles-pp.update_min_vruntime
      0.44 ±  7%      +0.3        0.73 ±  6%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.22 ±  7%      +0.3        0.54 ±  5%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.28 ±  7%      +0.3        0.61 ±  7%  perf-profile.self.cycles-pp.ksys_read
      0.12 ± 11%      +0.3        0.46 ±  4%  perf-profile.self.cycles-pp.perf_trace_sched_stat_runtime
      0.73 ±  6%      +0.4        1.09 ±  6%  perf-profile.self.cycles-pp.mutex_unlock
      0.36 ±  8%      +0.4        0.74 ±  8%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.85            +0.4        1.24 ±  4%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.35 ±  6%      +0.4        0.74 ±  4%  perf-profile.self.cycles-pp.try_to_wake_up
      0.14 ± 16%      +0.4        0.54 ±  2%  perf-profile.self.cycles-pp.__list_add_valid
      0.84 ±  4%      +0.4        1.27 ±  3%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.38 ±  3%      +0.5        0.84 ±  4%  perf-profile.self.cycles-pp.do_syscall_64
      0.39 ±  8%      +0.5        0.86 ±  5%  perf-profile.self.cycles-pp.read
      0.40 ±  6%      +0.5        0.88 ±  4%  perf-profile.self.cycles-pp.write
      0.94 ±  5%      +0.5        1.42 ±  5%  perf-profile.self.cycles-pp.update_load_avg
      0.36 ±  6%      +0.5        0.86 ±  4%  perf-profile.self.cycles-pp.main
      0.17 ± 16%      +0.5        0.67 ±  7%  perf-profile.self.cycles-pp.cpuacct_charge
      0.22 ±  4%      +0.5        0.76 ±  3%  perf-profile.self.cycles-pp.schedule
      0.52 ±  3%      +0.6        1.08 ±  7%  perf-profile.self.cycles-pp.__fdget_pos
      0.58 ±  5%      +0.6        1.16 ±  6%  perf-profile.self.cycles-pp.vfs_read
      0.63 ±  4%      +0.6        1.26 ±  3%  perf-profile.self.cycles-pp.reweight_entity
      0.64 ±  6%      +0.7        1.31        perf-profile.self.cycles-pp.pipe_write
      0.46 ±  4%      +0.7        1.15 ±  4%  perf-profile.self.cycles-pp.__entry_text_start
      0.65 ±  5%      +0.7        1.37 ±  2%  perf-profile.self.cycles-pp.os_xsave
      0.62 ±  3%      +0.7        1.35 ±  3%  perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.35 ±  7%      +0.7        1.09 ±  4%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.77 ±  6%      +0.8        1.52 ±  3%  perf-profile.self.cycles-pp.update_curr
      0.00            +0.9        0.90 ±  6%  perf-profile.self.cycles-pp.check_preempt_wakeup
      0.94 ±  5%      +1.0        1.94        perf-profile.self.cycles-pp.pipe_read
      0.55 ±  5%      +1.1        1.60 ±  4%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.45 ±  5%      +1.1        1.56 ±  2%  perf-profile.self.cycles-pp.__calc_delta
      0.81 ±  5%      +1.1        1.93 ±  2%  perf-profile.self.cycles-pp.update_cfs_group
      0.00            +1.6        1.59 ±  3%  perf-profile.self.cycles-pp.restore_fpregs_from_fpstate
      0.25 ±  7%      +4.8        5.07 ±  2%  perf-profile.self.cycles-pp.switch_mm_irqs_off
     11.94 ±  6%      +8.6       20.59 ±  2%  perf-profile.self.cycles-pp.mwait_idle_with_hints



***************************************************************************************************
lkp-cpl-4sp1: 144 threads 4 sockets Intel(R) Xeon(R) Gold 5318H CPU @ 2.50GHz (Cooper Lake) with 128G memory
=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase:
  cs-localhost/gcc-11/performance/ipv4/x86_64-rhel-8.3/25%/debian-11.1-x86_64-20220510.cgz/300s/lkp-cpl-4sp1/TCP_RR/netperf

commit: 
  c36eae5a86 ("sched/psi: Fix possible missing or delayed pending event")
  b6aabb01e3 ("sched: consider WF_SYNC to find idle siblings")

c36eae5a86d15a69 b6aabb01e3004b0285d953761e1 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
   3611595           +53.3%    5537882        netperf.Throughput_total_tps
    100322           +53.3%     153830        netperf.Throughput_tps
    130329        +8.7e+05%  1.128e+09 ± 19%  netperf.time.involuntary_context_switches
     16743 ±  2%     -22.6%      12962        netperf.time.minor_page_faults
      1202           +16.4%       1399 ±  2%  netperf.time.percent_of_cpu_this_job_got
      3432           +12.6%       3865 ±  2%  netperf.time.system_time
    188.48           +85.8%     350.16 ±  4%  netperf.time.user_time
 1.083e+09           -50.8%  5.334e+08 ± 39%  netperf.time.voluntary_context_switches
 1.083e+09           +53.3%  1.661e+09        netperf.workload
 2.212e+09           -97.0%   65981036        cpuidle..usage
     84551 ±  5%     -18.5%      68943 ±  3%  meminfo.Active
     84433 ±  5%     -18.5%      68833 ±  3%  meminfo.Active(anon)
     79312 ±  6%     -21.0%      62677 ±  5%  numa-meminfo.node3.Active
     79280 ±  6%     -21.0%      62645 ±  5%  numa-meminfo.node3.Active(anon)
     19806 ±  6%     -20.8%      15687 ±  5%  numa-vmstat.node3.nr_active_anon
     19806 ±  6%     -20.8%      15687 ±  5%  numa-vmstat.node3.nr_zone_active_anon
      8.60            -2.4        6.16        mpstat.cpu.all.soft%
     18.97            -1.9       17.04        mpstat.cpu.all.sys%
      1.30            +0.5        1.85 ±  3%  mpstat.cpu.all.usr%
     37.00           +47.7%      54.67        vmstat.procs.r
  14204693           -23.0%   10930950        vmstat.system.cs
    295062            +6.5%     314162        vmstat.system.in
     21070 ±  5%     -18.5%      17172 ±  3%  proc-vmstat.nr_active_anon
    106997            -2.5%     104321        proc-vmstat.nr_anon_pages
    409349 ±  4%     +10.0%     450452 ±  4%  proc-vmstat.nr_mapped
     83608            -1.5%      82352        proc-vmstat.nr_slab_unreclaimable
     21070 ±  5%     -18.5%      17172 ±  3%  proc-vmstat.nr_zone_active_anon
    531984 ±  9%     -15.9%     447458 ±  6%  proc-vmstat.numa_hint_faults_local
     24368 ±  7%     -29.6%      17161 ±  4%  proc-vmstat.pgactivate
   7844373            -3.4%    7576615        proc-vmstat.pgalloc_normal
   1450970 ±  2%     -18.0%    1189301        proc-vmstat.pgfree
      1358           -36.3%     864.67        turbostat.Avg_MHz
     41.28           -15.0       26.27        turbostat.Busy%
 1.281e+09          -100.0%      15898 ± 95%  turbostat.C1
     12.60           -12.6        0.00        turbostat.C1%
     58.66           +25.5%      73.65        turbostat.CPU%c1
      0.21           +77.0%       0.37        turbostat.IPC
     14.47 ± 21%     -14.5        0.00        turbostat.PKG_%
 8.861e+08          -100.0%       4493 ± 14%  turbostat.POLL
      6.47            -6.5        0.00        turbostat.POLL%
    571.65            -8.5%     522.90        turbostat.PkgWatt
      8.27            -4.0%       7.94        turbostat.RAMWatt
     27233 ± 32%     -95.4%       1255 ±136%  sched_debug.cfs_rq:/.MIN_vruntime.avg
    750764 ±  9%     -76.0%     180343 ±136%  sched_debug.cfs_rq:/.MIN_vruntime.max
    133981 ± 19%     -88.8%      14993 ±136%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
      0.26           +23.0%       0.32 ±  6%  sched_debug.cfs_rq:/.h_nr_running.avg
      1.03 ±  6%     +88.1%       1.93 ±  4%  sched_debug.cfs_rq:/.h_nr_running.max
      0.42           +46.9%       0.62 ±  5%  sched_debug.cfs_rq:/.h_nr_running.stddev
      9751 ± 10%     -32.8%       6552 ± 13%  sched_debug.cfs_rq:/.load.avg
     27233 ± 32%     -95.4%       1255 ±136%  sched_debug.cfs_rq:/.max_vruntime.avg
    750764 ±  9%     -76.0%     180343 ±136%  sched_debug.cfs_rq:/.max_vruntime.max
    133981 ± 19%     -88.8%      14993 ±136%  sched_debug.cfs_rq:/.max_vruntime.stddev
    724533 ±  8%     +35.8%     983881 ±  8%  sched_debug.cfs_rq:/.min_vruntime.avg
    845404 ± 10%     +54.1%    1302855 ±  8%  sched_debug.cfs_rq:/.min_vruntime.max
     27934 ± 13%    +297.8%     111130 ± 11%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.26           -18.8%       0.21 ±  4%  sched_debug.cfs_rq:/.nr_running.avg
    197.05           +90.2%     374.73 ±  4%  sched_debug.cfs_rq:/.runnable_avg.avg
    849.80 ±  4%     +77.3%       1506 ±  2%  sched_debug.cfs_rq:/.runnable_avg.max
    209.48          +161.4%     547.51        sched_debug.cfs_rq:/.runnable_avg.stddev
    114133 ± 29%    +172.2%     310722 ± 39%  sched_debug.cfs_rq:/.spread0.max
    -65385          +469.6%    -372412        sched_debug.cfs_rq:/.spread0.min
     27949 ± 13%    +297.7%     111154 ± 11%  sched_debug.cfs_rq:/.spread0.stddev
    196.98           +35.1%     266.15 ±  3%  sched_debug.cfs_rq:/.util_avg.avg
    849.66 ±  4%     +35.1%       1147 ±  3%  sched_debug.cfs_rq:/.util_avg.max
    209.42           +86.8%     391.19        sched_debug.cfs_rq:/.util_avg.stddev
     93.07 ±  4%     +76.1%     163.93 ±  4%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    664.98 ±  8%     +48.2%     985.29 ±  3%  sched_debug.cfs_rq:/.util_est_enqueued.max
    162.79           +91.6%     311.99        sched_debug.cfs_rq:/.util_est_enqueued.stddev
    250834 ±  9%     +84.0%     461429 ±  5%  sched_debug.cpu.avg_idle.avg
      3157 ±  6%    +939.5%      32825 ± 45%  sched_debug.cpu.avg_idle.min
    282325 ±  5%     +16.5%     328926        sched_debug.cpu.avg_idle.stddev
      7.58 ±  3%     +34.3%      10.18 ±  2%  sched_debug.cpu.clock.stddev
     10236 ±  4%     -13.5%       8854 ±  5%  sched_debug.cpu.curr->pid.max
      0.00 ± 13%     +55.3%       0.00 ±  4%  sched_debug.cpu.next_balance.stddev
      0.23 ±  3%     +41.6%       0.32 ±  5%  sched_debug.cpu.nr_running.avg
      1.03 ±  6%     +81.6%       1.87 ±  5%  sched_debug.cpu.nr_running.max
      0.40 ±  2%     +53.6%       0.62 ±  4%  sched_debug.cpu.nr_running.stddev
  13964454 ± 10%     -31.5%    9562330 ±  9%  sched_debug.cpu.nr_switches.avg
  15141338 ± 10%     -19.0%   12257576 ±  7%  sched_debug.cpu.nr_switches.max
  12639092 ±  9%     -54.5%    5746871 ± 11%  sched_debug.cpu.nr_switches.min
    444327 ± 13%    +150.9%    1114942 ± 10%  sched_debug.cpu.nr_switches.stddev
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.avg
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.max
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.min
     12.33           -96.1%       0.48 ±  7%  perf-stat.i.MPKI
 2.876e+10           +11.9%  3.217e+10        perf-stat.i.branch-instructions
      1.21            +0.1        1.26        perf-stat.i.branch-miss-rate%
 3.406e+08           +17.8%  4.012e+08        perf-stat.i.branch-misses
      1.90 ± 15%     +34.8       36.70 ± 16%  perf-stat.i.cache-miss-rate%
 1.739e+09           -95.6%   76617839 ±  7%  perf-stat.i.cache-references
  14360622           -23.1%   11042127        perf-stat.i.context-switches
      1.43           -46.7%       0.76        perf-stat.i.cpi
 2.023e+11           -38.7%  1.239e+11        perf-stat.i.cpu-cycles
      1322         +1109.9%      15995 ±  2%  perf-stat.i.cpu-migrations
      7928 ± 19%     -35.8%       5089 ± 19%  perf-stat.i.cycles-between-cache-misses
      0.02 ±  6%      -0.0        0.01 ±  4%  perf-stat.i.dTLB-load-miss-rate%
   6843508 ±  6%     -32.1%    4644002 ±  3%  perf-stat.i.dTLB-load-misses
 4.213e+10           +15.6%   4.87e+10        perf-stat.i.dTLB-loads
      0.00 ±  5%      -0.0        0.00 ±  4%  perf-stat.i.dTLB-store-miss-rate%
 2.439e+10           +14.5%  2.793e+10        perf-stat.i.dTLB-stores
     62.06            +2.8       64.90        perf-stat.i.iTLB-load-miss-rate%
 1.596e+08 ±  2%     +44.2%  2.302e+08        perf-stat.i.iTLB-load-misses
  97319700           +27.8%  1.244e+08        perf-stat.i.iTLB-loads
 1.429e+11           +14.0%  1.629e+11        perf-stat.i.instructions
    918.99 ±  2%     -19.1%     743.46        perf-stat.i.instructions-per-iTLB-miss
      0.71           +85.9%       1.31        perf-stat.i.ipc
      1.40           -38.7%       0.86        perf-stat.i.metric.GHz
    873.16           +76.2%       1538 ±  3%  perf-stat.i.metric.K/sec
    673.72           +12.2%     755.60        perf-stat.i.metric.M/sec
     82.06 ±  3%      +9.3       91.32        perf-stat.i.node-store-miss-rate%
   1049204 ± 16%     +53.8%    1613542 ± 18%  perf-stat.i.node-store-misses
    443904 ±  6%     -30.0%     310869 ±  6%  perf-stat.i.node-stores
     12.17           -96.1%       0.47 ±  7%  perf-stat.overall.MPKI
      1.18            +0.1        1.25        perf-stat.overall.branch-miss-rate%
      1.75 ± 17%     +36.3       38.07 ± 15%  perf-stat.overall.cache-miss-rate%
      1.42           -46.3%       0.76        perf-stat.overall.cpi
      6820 ± 15%     -35.7%       4383 ± 17%  perf-stat.overall.cycles-between-cache-misses
      0.02 ±  6%      -0.0        0.01 ±  4%  perf-stat.overall.dTLB-load-miss-rate%
      0.00 ±  5%      -0.0        0.00 ±  3%  perf-stat.overall.dTLB-store-miss-rate%
     62.11            +2.8       64.92        perf-stat.overall.iTLB-load-miss-rate%
    895.64 ±  2%     -21.0%     707.76        perf-stat.overall.instructions-per-iTLB-miss
      0.71           +86.1%       1.31        perf-stat.overall.ipc
     69.85 ±  6%     +13.7       83.54 ±  2%  perf-stat.overall.node-store-miss-rate%
     39789           -25.6%      29606        perf-stat.overall.path-length
 2.866e+10           +11.9%  3.206e+10        perf-stat.ps.branch-instructions
 3.395e+08           +17.8%  3.999e+08        perf-stat.ps.branch-misses
 1.733e+09           -95.6%   76361868 ±  7%  perf-stat.ps.cache-references
  14312085           -23.1%   11004648        perf-stat.ps.context-switches
 2.016e+11           -38.7%  1.235e+11        perf-stat.ps.cpu-cycles
      1317         +1109.8%      15942 ±  2%  perf-stat.ps.cpu-migrations
   6808921 ±  6%     -32.1%    4621256 ±  3%  perf-stat.ps.dTLB-load-misses
 4.199e+10           +15.6%  4.854e+10        perf-stat.ps.dTLB-loads
 2.431e+10           +14.5%  2.784e+10        perf-stat.ps.dTLB-stores
 1.591e+08 ±  2%     +44.2%  2.294e+08        perf-stat.ps.iTLB-load-misses
  96991959           +27.8%  1.239e+08        perf-stat.ps.iTLB-loads
 1.424e+11           +14.0%  1.624e+11        perf-stat.ps.instructions
      0.01 ± 50%     -72.0%       0.00 ±115%  perf-stat.ps.major-faults
   1046642 ± 16%     +53.7%    1608395 ± 18%  perf-stat.ps.node-store-misses
    442450 ±  6%     -29.9%     309961 ±  7%  perf-stat.ps.node-stores
 4.311e+13           +14.1%  4.919e+13        perf-stat.total.instructions
     55.82 ±  5%     -21.0       34.84 ± 14%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     55.50 ±  5%     -20.9       34.61 ± 14%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     55.49 ±  5%     -20.9       34.61 ± 14%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     55.40 ±  5%     -20.8       34.61 ± 14%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     48.23 ±  7%     -13.7       34.56 ± 14%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     46.09 ±  7%     -11.7       34.40 ± 14%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
     46.05 ±  7%     -11.7       34.36 ± 14%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
     10.29 ±  7%     -10.3        0.00        perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     10.00 ±  6%     -10.0        0.00        perf-profile.calltrace.cycles-pp.intel_idle_irq.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      9.36 ±  6%      -9.4        0.00        perf-profile.calltrace.cycles-pp.mwait_idle_with_hints.intel_idle_irq.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
      1.58 ±  7%      -1.2        0.40 ± 72%  perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.schedule_timeout
      6.53 ±  6%      -1.1        5.42 ± 13%  perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv.send_omni_inner
      0.00            +0.8        0.82 ±  8%  perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare
      0.00            +0.9        0.88 ± 42%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.recv_omni
      0.00            +0.9        0.93 ±  9%  perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__schedule.schedule.schedule_timeout.wait_woken
      0.00            +1.0        0.98 ±  7%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send
      0.00            +1.0        0.98 ±  7%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send
      0.00            +1.0        0.98 ±  7%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__send
      0.00            +1.0        1.03 ± 22%  perf-profile.calltrace.cycles-pp.skb_release_data.__kfree_skb.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established
      0.00            +1.1        1.13 ±  9%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.recv
      0.00            +1.1        1.13 ±  9%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv
      0.00            +1.1        1.13 ±  9%  perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv
      0.00            +1.2        1.15 ±  8%  perf-profile.calltrace.cycles-pp.__kfree_skb.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv
      1.54 ±  6%      +1.2        2.76 ±  8%  perf-profile.calltrace.cycles-pp.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
      0.00            +1.3        1.32 ±  7%  perf-profile.calltrace.cycles-pp.skb_do_copy_data_nocache.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
      5.02 ±  7%      +1.3        6.35 ±  8%  perf-profile.calltrace.cycles-pp.wait_woken.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
      1.33 ±  7%      +1.4        2.74 ±  7%  perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit
     10.51 ±  5%      +1.4       11.96 ±  8%  perf-profile.calltrace.cycles-pp.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog
      8.66 ±  7%      +1.5       10.14 ±  8%  perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
     14.38 ±  6%      +1.5       15.87 ±  7%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq.__local_bh_enable_ip.ip_finish_output2.__ip_queue_xmit
      8.75 ±  7%      +1.5       10.27 ±  8%  perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core
      1.06 ± 22%      +1.5        2.60 ±  8%  perf-profile.calltrace.cycles-pp.tcp_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
     14.55 ±  6%      +1.5       16.09 ±  7%  perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb
     14.67 ±  6%      +1.6       16.26 ±  7%  perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit
     13.28 ±  6%      +1.7       14.93 ±  8%  perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv
      0.00            +1.7        1.67 ±  8%  perf-profile.calltrace.cycles-pp.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit
     10.67 ±  5%      +1.7       12.38 ±  7%  perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll
      0.00            +1.7        1.75 ± 14%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.__wake_up_common.__wake_up_common_lock
      2.39 ±  6%      +1.8        4.16 ±  8%  perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
     10.70 ±  5%      +1.8       12.47 ±  7%  perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action
      0.00            +1.8        1.78 ±  8%  perf-profile.calltrace.cycles-pp.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb
      0.00            +1.8        1.79 ± 13%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable
      0.00            +1.8        1.84 ±  8%  perf-profile.calltrace.cycles-pp.__alloc_skb.tcp_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
      3.91 ±  7%      +1.9        5.78 ±  8%  perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked
      3.68 ±  7%      +1.9        5.58 ±  8%  perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.wait_woken.sk_wait_data
      4.03 ±  7%      +1.9        5.97 ±  8%  perf-profile.calltrace.cycles-pp.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg
      0.00            +2.0        1.99 ±  8%  perf-profile.calltrace.cycles-pp.__send
     11.19 ±  6%      +2.2       13.36 ±  7%  perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action.__softirqentry_text_start
      0.00            +2.3        2.26 ± 21%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.send_omni_inner
      0.00            +2.3        2.29 ±  9%  perf-profile.calltrace.cycles-pp.recv
     11.56 ±  6%      +2.5       14.09 ±  7%  perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.__softirqentry_text_start.do_softirq
      0.00            +2.6        2.59 ±  8%  perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode
     11.58 ±  6%      +2.6       14.18 ±  7%  perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.__softirqentry_text_start.do_softirq.__local_bh_enable_ip
      0.00            +2.7        2.72 ±  8%  perf-profile.calltrace.cycles-pp.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      0.00            +2.9        2.90 ±  8%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
     16.38 ±  6%      +3.4       19.76 ±  7%  perf-profile.calltrace.cycles-pp.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames
      0.00            +3.9        3.86 ±  3%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send
     17.18 ±  6%      +4.3       21.44 ±  6%  perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked
     12.91 ±  6%      +4.6       17.50 ±  7%  perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.recv_omni
     18.67 ±  6%      +5.3       24.00 ±  6%  perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg
     13.02 ±  6%      +5.5       18.56 ±  7%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.recv_omni.process_requests
     13.10 ±  6%      +5.7       18.77 ±  7%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__send.recv_omni.process_requests.spawn_child
     13.24 ±  6%      +5.8       19.06 ±  7%  perf-profile.calltrace.cycles-pp.__send.recv_omni.process_requests.spawn_child.accept_connection
     20.18 ±  6%      +6.3       26.50 ±  6%  perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
     20.27 ±  6%      +6.5       26.78 ±  6%  perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
     12.70 ±  6%      +6.5       19.22 ±  9%  perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.send_omni_inner
     21.45 ±  6%      +8.3       29.75 ±  7%  perf-profile.calltrace.cycles-pp.recv_omni.process_requests.spawn_child.accept_connection.accept_connections
     12.80 ±  6%      +8.8       21.60 ± 10%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.send_omni_inner.send_tcp_rr
     19.60 ±  7%      +8.9       28.49 ±  7%  perf-profile.calltrace.cycles-pp.send_omni_inner.send_tcp_rr.main.__libc_start_main
     24.83 ± 19%      +8.9       33.77 ± 15%  perf-profile.calltrace.cycles-pp.mwait_idle_with_hints.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
     24.83 ± 19%      +8.9       33.77 ± 15%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     19.76 ±  7%      +9.0       28.80 ±  7%  perf-profile.calltrace.cycles-pp.send_tcp_rr.main.__libc_start_main
     11.96 ±  7%      +9.4       21.31 ±  9%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__send.send_omni_inner.send_tcp_rr.main
     24.07 ±  6%      +9.4       33.44 ±  7%  perf-profile.calltrace.cycles-pp.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
     12.09 ±  7%      +9.4       21.47 ±  9%  perf-profile.calltrace.cycles-pp.__send.send_omni_inner.send_tcp_rr.main.__libc_start_main
     24.63 ±  6%      +9.8       34.41 ±  7%  perf-profile.calltrace.cycles-pp.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64
     25.07 ±  6%     +10.3       35.41 ±  7%  perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
     25.49 ±  6%     +11.0       36.46 ±  7%  perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send
     55.82 ±  5%     -21.0       34.84 ± 14%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     55.82 ±  5%     -21.0       34.84 ± 14%  perf-profile.children.cycles-pp.cpu_startup_entry
     55.50 ±  5%     -20.9       34.61 ± 14%  perf-profile.children.cycles-pp.start_secondary
     55.74 ±  5%     -20.9       34.84 ± 14%  perf-profile.children.cycles-pp.do_idle
     48.52 ±  7%     -13.7       34.79 ± 14%  perf-profile.children.cycles-pp.cpuidle_idle_call
     46.36 ±  7%     -11.7       34.63 ± 14%  perf-profile.children.cycles-pp.cpuidle_enter
     46.34 ±  7%     -11.7       34.63 ± 14%  perf-profile.children.cycles-pp.cpuidle_enter_state
     10.35 ±  7%     -10.4        0.00        perf-profile.children.cycles-pp.poll_idle
     10.06 ±  6%     -10.1        0.00        perf-profile.children.cycles-pp.intel_idle_irq
      3.04 ±  7%      -2.3        0.72 ± 10%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      1.70 ±  5%      -1.6        0.15 ± 16%  perf-profile.children.cycles-pp.menu_select
      1.33 ±  6%      -1.3        0.03 ± 70%  perf-profile.children.cycles-pp.ttwu_queue_wakelist
      1.68 ±  7%      -0.9        0.75 ±  8%  perf-profile.children.cycles-pp.dequeue_entity
      0.92 ±  7%      -0.6        0.29 ± 18%  perf-profile.children.cycles-pp.select_task_rq
      0.73 ±  4%      -0.6        0.10 ± 18%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.90 ±  8%      -0.6        0.28 ±  7%  perf-profile.children.cycles-pp.skb_attempt_defer_free
      0.79 ±  7%      -0.6        0.24 ± 19%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.66 ±  7%      -0.5        0.19 ± 10%  perf-profile.children.cycles-pp.sock_rfree
      0.48 ±  3%      -0.4        0.09 ± 20%  perf-profile.children.cycles-pp.tick_nohz_next_event
      1.11 ± 10%      -0.4        0.74 ± 18%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.64 ±  5%      -0.4        0.28 ± 11%  perf-profile.children.cycles-pp.remove_wait_queue
      0.96 ±  7%      -0.3        0.62 ± 11%  perf-profile.children.cycles-pp.__inet_lookup_established
      0.81 ±  5%      -0.3        0.47 ± 13%  perf-profile.children.cycles-pp.prepare_task_switch
      0.82 ±  7%      -0.3        0.52 ±  7%  perf-profile.children.cycles-pp._raw_spin_lock
      0.57 ±  9%      -0.2        0.33 ± 11%  perf-profile.children.cycles-pp.tcp_check_space
      1.10 ±  7%      -0.2        0.88 ±  8%  perf-profile.children.cycles-pp.enqueue_entity
      0.56 ±  8%      -0.2        0.35 ±  9%  perf-profile.children.cycles-pp.copyout
      0.55 ±  9%      -0.2        0.34 ±  7%  perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
      0.35 ±  8%      -0.2        0.16 ±  8%  perf-profile.children.cycles-pp.kfree_skbmem
      0.58 ±  7%      -0.2        0.41 ±  7%  perf-profile.children.cycles-pp.copy_user_short_string
      0.31 ±  8%      -0.2        0.15 ± 11%  perf-profile.children.cycles-pp.perf_tp_event
      0.20 ±  8%      -0.1        0.05 ±  7%  perf-profile.children.cycles-pp.skb_release_head_state
      0.58 ±  7%      -0.1        0.47 ±  8%  perf-profile.children.cycles-pp.tcp_rcv_space_adjust
      0.09 ± 15%      -0.0        0.04 ± 45%  perf-profile.children.cycles-pp.rebalance_domains
      0.07 ± 11%      -0.0        0.03 ± 70%  perf-profile.children.cycles-pp.tracing_gen_ctx_irq_test
      0.07 ±  7%      +0.0        0.09 ±  7%  perf-profile.children.cycles-pp.tcp_rate_skb_delivered
      0.15 ±  7%      +0.0        0.18 ±  9%  perf-profile.children.cycles-pp.add_wait_queue
      0.07 ±  7%      +0.0        0.10 ± 10%  perf-profile.children.cycles-pp.tcp_event_data_recv
      0.04 ± 44%      +0.0        0.08 ± 10%  perf-profile.children.cycles-pp.place_entity
      0.11 ± 11%      +0.0        0.15 ± 15%  perf-profile.children.cycles-pp.tcp_rearm_rto
      0.06 ± 14%      +0.0        0.11 ± 12%  perf-profile.children.cycles-pp.ip_skb_dst_mtu
      0.02 ±141%      +0.0        0.06 ± 21%  perf-profile.children.cycles-pp.timekeeping_max_deferment
      0.05 ± 47%      +0.0        0.09 ±  7%  perf-profile.children.cycles-pp.nf_hook_slow
      0.02 ±141%      +0.0        0.06 ± 11%  perf-profile.children.cycles-pp.__cgroup_account_cputime
      0.22 ± 10%      +0.1        0.27 ±  9%  perf-profile.children.cycles-pp.__mod_timer
      0.09 ± 14%      +0.1        0.14 ± 19%  perf-profile.children.cycles-pp.scheduler_tick
      0.00            +0.1        0.05 ±  7%  perf-profile.children.cycles-pp.minmax_running_min
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.ip_local_deliver
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.tcp_options_write
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.inet_sendmsg
      0.01 ±223%      +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.skb_clone
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.tcp_small_queue_check
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.cubictcp_cong_avoid
      0.00            +0.1        0.06 ± 18%  perf-profile.children.cycles-pp.security_sock_rcv_skb
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.tcp_chrono_stop
      0.00            +0.1        0.06 ± 15%  perf-profile.children.cycles-pp.eth_type_trans
      0.00            +0.1        0.06 ± 14%  perf-profile.children.cycles-pp.kmalloc_slab
      0.00            +0.1        0.06 ± 17%  perf-profile.children.cycles-pp.__raise_softirq_irqoff
      0.06 ±  6%      +0.1        0.12 ±  7%  perf-profile.children.cycles-pp.sk_filter_trim_cap
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.switch_ldt
      0.00            +0.1        0.06 ± 14%  perf-profile.children.cycles-pp.rb_insert_color
      0.27 ±  9%      +0.1        0.33 ±  7%  perf-profile.children.cycles-pp.sk_reset_timer
      0.10 ± 15%      +0.1        0.17 ± 11%  perf-profile.children.cycles-pp.perf_trace_sched_switch
      0.10 ±  8%      +0.1        0.16 ±  9%  perf-profile.children.cycles-pp.__ip_finish_output
      0.10 ± 14%      +0.1        0.17 ± 11%  perf-profile.children.cycles-pp.__tcp_cleanup_rbuf
      0.08 ±  8%      +0.1        0.15 ± 11%  perf-profile.children.cycles-pp.update_min_vruntime
      0.06 ±  9%      +0.1        0.13 ±  9%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
      0.09 ±  6%      +0.1        0.16 ±  9%  perf-profile.children.cycles-pp.try_charge_memcg
      0.00            +0.1        0.07 ± 10%  perf-profile.children.cycles-pp.check_cfs_rq_runtime
      0.00            +0.1        0.07 ± 10%  perf-profile.children.cycles-pp.clear_buddies
      0.04 ± 44%      +0.1        0.11 ±  6%  perf-profile.children.cycles-pp.tcp_established_options
      0.00            +0.1        0.07 ± 14%  perf-profile.children.cycles-pp.apparmor_socket_recvmsg
      0.61 ±  7%      +0.1        0.68 ±  6%  perf-profile.children.cycles-pp.__switch_to
      0.08 ±  6%      +0.1        0.15 ±  9%  perf-profile.children.cycles-pp.__copy_skb_header
      0.00            +0.1        0.07 ± 20%  perf-profile.children.cycles-pp.cubictcp_acked
      0.06 ±  9%      +0.1        0.13 ±  5%  perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
      0.00            +0.1        0.07 ± 12%  perf-profile.children.cycles-pp.memcg_slab_post_alloc_hook
      0.06 ± 11%      +0.1        0.14 ±  9%  perf-profile.children.cycles-pp.tcp_rate_skb_sent
      0.08 ± 13%      +0.1        0.16 ±  8%  perf-profile.children.cycles-pp.rcu_all_qs
      0.12 ± 12%      +0.1        0.20 ±  8%  perf-profile.children.cycles-pp.ip_send_check
      0.06 ±  9%      +0.1        0.13 ±  7%  perf-profile.children.cycles-pp.inet_ehashfn
      0.03 ± 70%      +0.1        0.11 ±  8%  perf-profile.children.cycles-pp.sk_free
      0.06 ± 11%      +0.1        0.14 ±  8%  perf-profile.children.cycles-pp.raw_v4_input
      0.00            +0.1        0.08 ± 12%  perf-profile.children.cycles-pp.tcp_inbound_md5_hash
      0.00            +0.1        0.08 ± 10%  perf-profile.children.cycles-pp.neigh_hh_output
      0.19 ± 17%      +0.1        0.28 ± 18%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.04 ± 45%      +0.1        0.13 ±  8%  perf-profile.children.cycles-pp.tcp_rate_gen
      0.00            +0.1        0.08 ±  5%  perf-profile.children.cycles-pp.syscall_enter_from_user_mode
      0.00            +0.1        0.09 ± 10%  perf-profile.children.cycles-pp.apparmor_socket_sendmsg
      0.58 ±  8%      +0.1        0.67 ±  8%  perf-profile.children.cycles-pp.tcp_event_new_data_sent
      0.00            +0.1        0.09 ±  6%  perf-profile.children.cycles-pp.check_stack_object
      0.00            +0.1        0.09 ±  6%  perf-profile.children.cycles-pp.tcp_v4_fill_cb
      0.00            +0.1        0.09 ± 11%  perf-profile.children.cycles-pp.tcp_newly_delivered
      0.07 ± 13%      +0.1        0.16 ±  5%  perf-profile.children.cycles-pp.sock_put
      0.10 ±  8%      +0.1        0.19 ±  7%  perf-profile.children.cycles-pp.__ksize
      0.05 ± 45%      +0.1        0.14 ± 13%  perf-profile.children.cycles-pp.netif_skb_features
      0.10 ±  9%      +0.1        0.20 ± 14%  perf-profile.children.cycles-pp.sk_forced_mem_schedule
      0.04 ± 71%      +0.1        0.13 ±  5%  perf-profile.children.cycles-pp.__usecs_to_jiffies
      0.09 ± 14%      +0.1        0.19 ± 14%  perf-profile.children.cycles-pp.tcp_rtt_estimator
      0.00            +0.1        0.10 ±  4%  perf-profile.children.cycles-pp.tcp_push
      0.10 ± 10%      +0.1        0.21 ±  7%  perf-profile.children.cycles-pp.__tcp_select_window
      0.63 ±  6%      +0.1        0.74 ±  7%  perf-profile.children.cycles-pp.simple_copy_to_iter
      0.00            +0.1        0.11 ± 11%  perf-profile.children.cycles-pp.resched_curr
      0.07 ± 12%      +0.1        0.19 ± 22%  perf-profile.children.cycles-pp.cgroup_rstat_updated
      0.43 ±  6%      +0.1        0.54 ±  9%  perf-profile.children.cycles-pp.update_rq_clock
      0.12 ±  9%      +0.1        0.24 ±  7%  perf-profile.children.cycles-pp.ip_output
      0.18 ±  9%      +0.1        0.29 ± 14%  perf-profile.children.cycles-pp.__calc_delta
      0.50 ±  6%      +0.1        0.63 ±  9%  perf-profile.children.cycles-pp.set_next_entity
      0.09 ±  6%      +0.1        0.21 ±  7%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.28 ±  7%      +0.1        0.41 ±  8%  perf-profile.children.cycles-pp.recv_data
      0.00            +0.1        0.13 ±  8%  perf-profile.children.cycles-pp.cubictcp_cwnd_event
      0.04 ± 71%      +0.1        0.17 ± 13%  perf-profile.children.cycles-pp.skb_page_frag_refill
      0.11 ±  8%      +0.1        0.24 ± 10%  perf-profile.children.cycles-pp.tcp_tso_segs
      0.07 ± 14%      +0.1        0.20 ± 10%  perf-profile.children.cycles-pp.sk_page_frag_refill
      0.11 ±  6%      +0.1        0.25 ±  6%  perf-profile.children.cycles-pp.__netif_receive_skb_core
      0.30 ±  6%      +0.1        0.44 ±  8%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.08 ± 18%      +0.1        0.23 ± 19%  perf-profile.children.cycles-pp.cpuacct_charge
      0.29 ±  9%      +0.1        0.44 ±  8%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.08 ±  8%      +0.1        0.23 ±  9%  perf-profile.children.cycles-pp.ip_rcv_core
      0.16 ±  9%      +0.2        0.32 ±  8%  perf-profile.children.cycles-pp.__ip_local_out
      0.29 ±  6%      +0.2        0.46 ±  6%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.12 ±  6%      +0.2        0.28 ± 10%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.35 ±  4%      +0.2        0.52 ±  8%  perf-profile.children.cycles-pp.update_cfs_group
      0.14 ±  5%      +0.2        0.31 ±  7%  perf-profile.children.cycles-pp.copyin
      0.11 ±  6%      +0.2        0.28 ±  8%  perf-profile.children.cycles-pp.refill_stock
      0.18 ±  7%      +0.2        0.35 ±  8%  perf-profile.children.cycles-pp.__cond_resched
      0.14 ±  7%      +0.2        0.32 ±  8%  perf-profile.children.cycles-pp.tcp_update_pacing_rate
      0.11 ±  9%      +0.2        0.28 ±  8%  perf-profile.children.cycles-pp.tcp_update_skb_after_send
      0.15 ± 10%      +0.2        0.32 ± 15%  perf-profile.children.cycles-pp.mem_cgroup_uncharge_skmem
      0.17 ±  6%      +0.2        0.34 ±  7%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.12 ±  9%      +0.2        0.30 ± 14%  perf-profile.children.cycles-pp.validate_xmit_skb
      0.15 ± 20%      +0.2        0.33 ± 17%  perf-profile.children.cycles-pp.find_busiest_group
      0.07 ±  9%      +0.2        0.26 ±  8%  perf-profile.children.cycles-pp.__list_add_valid
      0.18 ±  8%      +0.2        0.36 ± 10%  perf-profile.children.cycles-pp.ip_local_out
      0.17 ±  6%      +0.2        0.36 ± 17%  perf-profile.children.cycles-pp.__mod_memcg_state
      0.16 ±  4%      +0.2        0.35 ±  9%  perf-profile.children.cycles-pp.tcp_wfree
      0.46 ±  6%      +0.2        0.66 ±  7%  perf-profile.children.cycles-pp.read_tsc
      0.11 ±  9%      +0.2        0.31 ±  8%  perf-profile.children.cycles-pp.__might_fault
      0.00            +0.2        0.20 ± 10%  perf-profile.children.cycles-pp.set_next_buddy
      0.12 ± 21%      +0.2        0.33 ± 17%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.19 ± 17%      +0.2        0.40 ± 17%  perf-profile.children.cycles-pp.load_balance
      0.06 ±  7%      +0.2        0.27 ±  8%  perf-profile.children.cycles-pp.__kmem_cache_free
      0.10 ± 23%      +0.2        0.32 ± 17%  perf-profile.children.cycles-pp.update_sg_lb_stats
      0.18 ±  8%      +0.2        0.39 ±  6%  perf-profile.children.cycles-pp.__might_sleep
      0.23 ±  9%      +0.2        0.45 ±  7%  perf-profile.children.cycles-pp.send_data
      0.35 ±  8%      +0.2        0.58 ±  8%  perf-profile.children.cycles-pp.__skb_clone
      0.67 ±  6%      +0.2        0.90 ±  9%  perf-profile.children.cycles-pp.__switch_to_asm
      0.07 ± 10%      +0.2        0.31 ±  8%  perf-profile.children.cycles-pp.kmem_cache_free
      0.07 ± 12%      +0.2        0.32 ±  6%  perf-profile.children.cycles-pp.put_prev_entity
      0.12 ± 10%      +0.2        0.37 ±  8%  perf-profile.children.cycles-pp.tcp_ack_update_rtt
      0.24 ±  7%      +0.3        0.50 ±  9%  perf-profile.children.cycles-pp.mem_cgroup_charge_skmem
      0.27 ± 12%      +0.3        0.54 ± 12%  perf-profile.children.cycles-pp.ip_rcv
      0.48 ±  9%      +0.3        0.75 ±  7%  perf-profile.children.cycles-pp.release_sock
      0.22 ±  8%      +0.3        0.52 ±  6%  perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.25 ±  7%      +0.3        0.56 ±  8%  perf-profile.children.cycles-pp.os_xsave
      1.01 ±  6%      +0.3        1.33 ±  8%  perf-profile.children.cycles-pp.skb_release_data
      0.56 ±  8%      +0.3        0.87 ±  7%  perf-profile.children.cycles-pp._raw_spin_lock_bh
      0.24 ±  8%      +0.3        0.56 ±  9%  perf-profile.children.cycles-pp.__kmem_cache_alloc_node
      0.14 ±  3%      +0.3        0.47 ±  7%  perf-profile.children.cycles-pp.__virt_addr_valid
      0.20 ±  8%      +0.3        0.54 ±  8%  perf-profile.children.cycles-pp.tcp_schedule_loss_probe
      0.83 ±  5%      +0.3        1.17 ±  7%  perf-profile.children.cycles-pp.__check_object_size
      0.06 ±  9%      +0.4        0.41 ±  9%  perf-profile.children.cycles-pp.import_single_range
      0.35 ±  7%      +0.4        0.71 ±  7%  perf-profile.children.cycles-pp.lock_sock_nested
      0.13 ± 10%      +0.4        0.50 ± 11%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.10 ± 13%      +0.4        0.47 ± 11%  perf-profile.children.cycles-pp.check_preempt_curr
      0.27 ±  6%      +0.4        0.66 ±  9%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.20 ±  9%      +0.4        0.59 ±  7%  perf-profile.children.cycles-pp.kmem_cache_alloc_node
      0.84 ±  2%      +0.4        1.23 ± 11%  perf-profile.children.cycles-pp.ktime_get
      0.30 ±  5%      +0.4        0.70 ±  8%  perf-profile.children.cycles-pp.security_socket_sendmsg
      0.28 ±  7%      +0.4        0.68 ±  9%  perf-profile.children.cycles-pp.__kmalloc_node_track_caller
      0.19 ±  5%      +0.4        0.60 ±  9%  perf-profile.children.cycles-pp.enqueue_to_backlog
      0.94 ±  6%      +0.4        1.34 ±  7%  perf-profile.children.cycles-pp.update_load_avg
      0.00            +0.4        0.41 ± 12%  perf-profile.children.cycles-pp.check_preempt_wakeup
      0.21 ±  6%      +0.4        0.63 ±  8%  perf-profile.children.cycles-pp.netif_rx_internal
      0.21 ±  5%      +0.4        0.64 ±  9%  perf-profile.children.cycles-pp.__netif_rx
      0.30 ±  8%      +0.4        0.74 ±  9%  perf-profile.children.cycles-pp.kmalloc_reserve
      0.34 ±  5%      +0.4        0.78 ±  7%  perf-profile.children.cycles-pp.tcp_mstamp_refresh
      0.26 ±  8%      +0.4        0.71 ± 10%  perf-profile.children.cycles-pp.reweight_entity
      0.27 ± 11%      +0.5        0.75 ±  9%  perf-profile.children.cycles-pp.__fdget
      0.41 ±  5%      +0.5        0.89 ±  8%  perf-profile.children.cycles-pp.__sk_mem_reduce_allocated
      0.00            +0.5        0.50 ± 10%  perf-profile.children.cycles-pp.kfree
      0.22 ±  7%      +0.5        0.76 ±  8%  perf-profile.children.cycles-pp.security_socket_recvmsg
      0.29 ±  6%      +0.5        0.84 ±  7%  perf-profile.children.cycles-pp._copy_from_iter
      0.77 ±  8%      +0.5        1.31 ±  9%  perf-profile.children.cycles-pp.update_curr
      0.35 ±  6%      +0.6        0.91 ±  7%  perf-profile.children.cycles-pp.aa_sk_perm
      0.24 ±  8%      +0.6        0.80 ±  8%  perf-profile.children.cycles-pp.sock_recvmsg
      0.43 ±  7%      +0.6        0.99 ±  7%  perf-profile.children.cycles-pp.__entry_text_start
      0.33 ± 10%      +0.6        0.91 ±  9%  perf-profile.children.cycles-pp.sockfd_lookup_light
      1.23 ±  8%      +0.6        1.82 ±  7%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.54 ±  7%      +0.6        1.18 ±  8%  perf-profile.children.cycles-pp.__kfree_skb
      1.28 ±  7%      +0.8        2.04 ±  8%  perf-profile.children.cycles-pp.enqueue_task_fair
      1.32 ±  7%      +0.8        2.09 ±  8%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.54 ±  6%      +0.8        1.36 ±  7%  perf-profile.children.cycles-pp.skb_do_copy_data_nocache
      0.00            +1.0        1.01 ±  8%  perf-profile.children.cycles-pp.restore_fpregs_from_fpstate
      0.63 ±  6%      +1.1        1.70 ±  8%  perf-profile.children.cycles-pp.loopback_xmit
      0.82 ±  7%      +1.1        1.91 ±  8%  perf-profile.children.cycles-pp.__alloc_skb
      5.25 ±  7%      +1.1        6.39 ±  8%  perf-profile.children.cycles-pp.wait_woken
      0.68 ±  5%      +1.1        1.82 ±  8%  perf-profile.children.cycles-pp.dev_hard_start_xmit
      0.15 ±  8%      +1.2        1.33 ±  8%  perf-profile.children.cycles-pp.switch_fpu_return
      1.65 ±  6%      +1.2        2.87 ±  8%  perf-profile.children.cycles-pp.tcp_clean_rtx_queue
      9.07 ±  7%      +1.4       10.45 ±  8%  perf-profile.children.cycles-pp.tcp_rcv_established
     10.89 ±  6%      +1.4       12.30 ±  8%  perf-profile.children.cycles-pp.tcp_v4_rcv
      1.40 ±  6%      +1.4        2.82 ±  8%  perf-profile.children.cycles-pp.__dev_queue_xmit
      9.14 ±  6%      +1.4       10.56 ±  8%  perf-profile.children.cycles-pp.tcp_v4_do_rcv
      1.20 ±  7%      +1.5        2.68 ±  8%  perf-profile.children.cycles-pp.tcp_stream_alloc_skb
     13.56 ±  7%      +1.6       15.16 ±  8%  perf-profile.children.cycles-pp.__sys_recvfrom
     11.05 ±  6%      +1.7       12.71 ±  8%  perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
     13.66 ±  7%      +1.7       15.36 ±  8%  perf-profile.children.cycles-pp.__x64_sys_recvfrom
     11.08 ±  6%      +1.7       12.80 ±  8%  perf-profile.children.cycles-pp.ip_local_deliver_finish
      2.51 ±  6%      +1.8        4.28 ±  8%  perf-profile.children.cycles-pp.tcp_ack
      4.22 ±  7%      +1.8        6.02 ±  7%  perf-profile.children.cycles-pp.schedule_timeout
      0.10 ±  9%      +2.1        2.19 ±  8%  perf-profile.children.cycles-pp.switch_mm_irqs_off
     11.50 ±  6%      +2.2       13.70 ±  7%  perf-profile.children.cycles-pp.__netif_receive_skb_one_core
     11.89 ±  6%      +2.5       14.38 ±  7%  perf-profile.children.cycles-pp.process_backlog
     11.90 ±  6%      +2.6       14.46 ±  7%  perf-profile.children.cycles-pp.__napi_poll
      6.04 ±  7%      +2.6        8.61 ±  7%  perf-profile.children.cycles-pp.__schedule
      0.00            +3.2        3.24 ±  7%  perf-profile.children.cycles-pp.exit_to_user_mode_loop
     16.72 ±  6%      +3.3       20.05 ±  7%  perf-profile.children.cycles-pp.ip_finish_output2
     15.81 ±  7%      +3.7       19.46 ±  8%  perf-profile.children.cycles-pp.recv
     17.54 ±  6%      +4.2       21.74 ±  7%  perf-profile.children.cycles-pp.__ip_queue_xmit
      0.20 ±  8%      +4.7        4.87 ±  7%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      4.10 ±  7%      +4.8        8.90 ±  7%  perf-profile.children.cycles-pp.schedule
      0.32 ±  6%      +4.8        5.12 ±  7%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
     18.90 ±  6%      +5.5       24.36 ±  7%  perf-profile.children.cycles-pp.__tcp_transmit_skb
     20.44 ±  6%      +6.5       26.93 ±  7%  perf-profile.children.cycles-pp.tcp_write_xmit
     20.51 ±  6%      +6.6       27.14 ±  7%  perf-profile.children.cycles-pp.__tcp_push_pending_frames
     21.50 ±  6%      +8.3       29.79 ±  6%  perf-profile.children.cycles-pp.send_tcp_rr
     21.57 ±  6%      +8.5       30.03 ±  7%  perf-profile.children.cycles-pp.send_omni_inner
     21.58 ±  6%      +8.5       30.06 ±  7%  perf-profile.children.cycles-pp.recv_omni
     21.59 ±  6%      +8.5       30.08 ±  7%  perf-profile.children.cycles-pp.accept_connections
     21.59 ±  6%      +8.5       30.08 ±  7%  perf-profile.children.cycles-pp.accept_connection
     21.59 ±  6%      +8.5       30.08 ±  7%  perf-profile.children.cycles-pp.spawn_child
     21.59 ±  6%      +8.5       30.08 ±  7%  perf-profile.children.cycles-pp.process_requests
     24.97 ± 19%      +9.0       34.00 ± 15%  perf-profile.children.cycles-pp.intel_idle
     24.26 ±  6%      +9.3       33.55 ±  7%  perf-profile.children.cycles-pp.tcp_sendmsg_locked
     24.84 ±  6%      +9.7       34.54 ±  7%  perf-profile.children.cycles-pp.tcp_sendmsg
     25.26 ±  6%     +10.2       35.49 ±  7%  perf-profile.children.cycles-pp.sock_sendmsg
     25.68 ±  6%     +10.9       36.56 ±  7%  perf-profile.children.cycles-pp.__sys_sendto
     25.80 ±  6%     +11.0       36.82 ±  7%  perf-profile.children.cycles-pp.__x64_sys_sendto
     26.86 ±  6%     +17.5       44.40 ±  7%  perf-profile.children.cycles-pp.__send
     40.08 ±  6%     +17.8       57.92 ±  7%  perf-profile.children.cycles-pp.do_syscall_64
     40.47 ±  6%     +18.4       58.88 ±  7%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     10.22 ±  7%     -10.2        0.00        perf-profile.self.cycles-pp.poll_idle
      3.03 ±  7%      -2.3        0.70 ± 10%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      1.03 ±  7%      -0.7        0.37 ±  9%  perf-profile.self.cycles-pp.wait_woken
      0.66 ±  7%      -0.5        0.19 ± 10%  perf-profile.self.cycles-pp.sock_rfree
      0.91 ±  7%      -0.4        0.49 ± 12%  perf-profile.self.cycles-pp.__inet_lookup_established
      0.64 ±  8%      -0.4        0.26 ±  8%  perf-profile.self.cycles-pp.tcp_rcv_established
      0.92 ±  6%      -0.4        0.55 ±  6%  perf-profile.self.cycles-pp.skb_release_data
      0.40 ±  9%      -0.3        0.06 ±  6%  perf-profile.self.cycles-pp.skb_attempt_defer_free
      0.40 ±  8%      -0.3        0.08 ±  9%  perf-profile.self.cycles-pp.tcp_rcv_space_adjust
      0.82 ±  7%      -0.3        0.50 ±  8%  perf-profile.self.cycles-pp._raw_spin_lock
      0.54 ±  5%      -0.3        0.24 ±  9%  perf-profile.self.cycles-pp.check_heap_object
      0.43 ±  6%      -0.3        0.16 ± 14%  perf-profile.self.cycles-pp.prepare_task_switch
      0.57 ±  6%      -0.3        0.32 ± 11%  perf-profile.self.cycles-pp.__skb_datagram_iter
      0.56 ±  9%      -0.2        0.32 ± 10%  perf-profile.self.cycles-pp.tcp_check_space
      0.39 ±  7%      -0.2        0.19 ± 11%  perf-profile.self.cycles-pp.enqueue_entity
      0.35 ±  8%      -0.2        0.16 ±  8%  perf-profile.self.cycles-pp.kfree_skbmem
      0.57 ±  7%      -0.2        0.40 ±  8%  perf-profile.self.cycles-pp.copy_user_short_string
      0.26 ±  9%      -0.2        0.09 ±  7%  perf-profile.self.cycles-pp.__wake_up_common
      0.19 ±  7%      -0.2        0.03 ± 70%  perf-profile.self.cycles-pp.skb_release_head_state
      0.22 ±  8%      -0.2        0.07 ± 20%  perf-profile.self.cycles-pp.perf_tp_event
      0.21 ±  4%      -0.2        0.06 ± 19%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.67 ±  7%      -0.2        0.52 ±  8%  perf-profile.self.cycles-pp.tcp_recvmsg_locked
      0.25 ±  9%      -0.1        0.14 ± 12%  perf-profile.self.cycles-pp.dequeue_entity
      0.14 ±  9%      -0.1        0.05 ± 46%  perf-profile.self.cycles-pp.select_task_rq
      0.34 ±  3%      -0.1        0.27 ± 14%  perf-profile.self.cycles-pp.___perf_sw_event
      0.16 ± 12%      -0.1        0.10 ± 16%  perf-profile.self.cycles-pp.perf_trace_sched_wakeup_template
      0.20 ±  7%      -0.0        0.16 ±  7%  perf-profile.self.cycles-pp.finish_task_switch
      0.09 ± 11%      -0.0        0.06 ±  9%  perf-profile.self.cycles-pp.check_preempt_curr
      0.06 ±  7%      -0.0        0.03 ± 70%  perf-profile.self.cycles-pp.tracing_gen_ctx_irq_test
      0.06 ± 11%      +0.0        0.09 ±  9%  perf-profile.self.cycles-pp.tcp_event_data_recv
      0.05 ± 45%      +0.0        0.08 ±  7%  perf-profile.self.cycles-pp.put_prev_entity
      0.22 ±  7%      +0.0        0.25 ±  7%  perf-profile.self.cycles-pp.__wrgsbase_inactive
      0.10 ± 10%      +0.0        0.14 ± 15%  perf-profile.self.cycles-pp.tcp_rearm_rto
      0.22 ±  6%      +0.0        0.26 ±  8%  perf-profile.self.cycles-pp.dequeue_task_fair
      0.13 ±  6%      +0.0        0.17 ± 11%  perf-profile.self.cycles-pp.update_rq_clock
      0.04 ± 47%      +0.0        0.09 ± 18%  perf-profile.self.cycles-pp.tcp_v4_do_rcv
      0.06 ± 11%      +0.0        0.10 ± 13%  perf-profile.self.cycles-pp.ip_skb_dst_mtu
      0.02 ±142%      +0.0        0.06 ± 21%  perf-profile.self.cycles-pp.timekeeping_max_deferment
      0.02 ±141%      +0.0        0.06 ± 11%  perf-profile.self.cycles-pp.place_entity
      0.01 ±223%      +0.0        0.06 ± 13%  perf-profile.self.cycles-pp.skb_clone
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.minmax_running_min
      0.06 ± 13%      +0.1        0.11 ±  9%  perf-profile.self.cycles-pp.rcu_all_qs
      0.00            +0.1        0.05 ±  7%  perf-profile.self.cycles-pp.__netif_receive_skb_one_core
      0.00            +0.1        0.05 ±  7%  perf-profile.self.cycles-pp.clear_buddies
      0.00            +0.1        0.05 ±  7%  perf-profile.self.cycles-pp.tcp_small_queue_check
      0.18 ±  7%      +0.1        0.23 ± 10%  perf-profile.self.cycles-pp.do_softirq
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.ip_local_deliver
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.__kmalloc_node_track_caller
      0.00            +0.1        0.06 ± 13%  perf-profile.self.cycles-pp.skb_copy_datagram_iter
      0.00            +0.1        0.06 ± 13%  perf-profile.self.cycles-pp.cubictcp_cong_avoid
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.sk_reset_timer
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.switch_ldt
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.rb_insert_color
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.remove_wait_queue
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.__ip_finish_output
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.nf_hook_slow
      0.02 ±141%      +0.1        0.08 ± 10%  perf-profile.self.cycles-pp.tcp_send_mss
      0.00            +0.1        0.06 ± 15%  perf-profile.self.cycles-pp.eth_type_trans
      0.13 ±  8%      +0.1        0.19 ± 10%  perf-profile.self.cycles-pp.schedule_timeout
      0.00            +0.1        0.06 ± 13%  perf-profile.self.cycles-pp.memcg_slab_post_alloc_hook
      0.00            +0.1        0.06 ± 16%  perf-profile.self.cycles-pp.kmalloc_reserve
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.sk_filter_trim_cap
      0.00            +0.1        0.06 ± 16%  perf-profile.self.cycles-pp.kmalloc_slab
      0.06 ± 13%      +0.1        0.12 ±  8%  perf-profile.self.cycles-pp.tcp_recvmsg
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.tcp_inbound_md5_hash
      0.00            +0.1        0.06 ± 17%  perf-profile.self.cycles-pp.apparmor_socket_recvmsg
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.__wake_up_common_lock
      0.07 ± 12%      +0.1        0.14 ± 14%  perf-profile.self.cycles-pp.update_min_vruntime
      0.06 ± 11%      +0.1        0.12 ± 10%  perf-profile.self.cycles-pp.tcp_rate_skb_sent
      0.04 ± 45%      +0.1        0.11 ±  9%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode_prepare
      0.08 ±  6%      +0.1        0.14 ±  6%  perf-profile.self.cycles-pp.ip_output
      0.04 ± 44%      +0.1        0.11 ±  5%  perf-profile.self.cycles-pp.tcp_established_options
      0.01 ±223%      +0.1        0.08 ±  9%  perf-profile.self.cycles-pp.lock_sock_nested
      0.10 ± 16%      +0.1        0.17 ± 11%  perf-profile.self.cycles-pp.perf_trace_sched_switch
      0.08 ±  8%      +0.1        0.16 ± 10%  perf-profile.self.cycles-pp.try_charge_memcg
      0.00            +0.1        0.07 ± 14%  perf-profile.self.cycles-pp.tcp_stream_alloc_skb
      0.00            +0.1        0.07 ± 20%  perf-profile.self.cycles-pp.cubictcp_acked
      0.59 ±  6%      +0.1        0.66 ±  6%  perf-profile.self.cycles-pp.__switch_to
      0.07 ±  7%      +0.1        0.14 ± 11%  perf-profile.self.cycles-pp.irqtime_account_irq
      0.07 ±  8%      +0.1        0.14 ± 11%  perf-profile.self.cycles-pp.__copy_skb_header
      0.06 ±  9%      +0.1        0.13 ±  5%  perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
      0.00            +0.1        0.07 ± 10%  perf-profile.self.cycles-pp.skb_do_copy_data_nocache
      0.04 ± 44%      +0.1        0.12 ± 10%  perf-profile.self.cycles-pp.dev_hard_start_xmit
      0.11 ±  6%      +0.1        0.19 ± 11%  perf-profile.self.cycles-pp._copy_to_iter
      0.04 ± 71%      +0.1        0.11 ±  8%  perf-profile.self.cycles-pp.mem_cgroup_uncharge_skmem
      0.03 ± 70%      +0.1        0.11 ±  9%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode
      0.06 ±  8%      +0.1        0.14 ±  7%  perf-profile.self.cycles-pp.raw_v4_input
      0.03 ± 70%      +0.1        0.11 ±  8%  perf-profile.self.cycles-pp.sk_free
      0.00            +0.1        0.08 ±  8%  perf-profile.self.cycles-pp.neigh_hh_output
      0.26 ±  7%      +0.1        0.34 ±  9%  perf-profile.self.cycles-pp.recv_data
      0.12 ± 11%      +0.1        0.20 ±  8%  perf-profile.self.cycles-pp.ip_send_check
      0.00            +0.1        0.08 ±  7%  perf-profile.self.cycles-pp.syscall_enter_from_user_mode
      0.06 ± 11%      +0.1        0.15 ± 18%  perf-profile.self.cycles-pp.validate_xmit_skb
      0.00            +0.1        0.08 ±  5%  perf-profile.self.cycles-pp.__napi_poll
      0.00            +0.1        0.08 ±  5%  perf-profile.self.cycles-pp.tcp_v4_fill_cb
      0.00            +0.1        0.08 ± 11%  perf-profile.self.cycles-pp.apparmor_socket_sendmsg
      0.04 ± 45%      +0.1        0.13 ±  8%  perf-profile.self.cycles-pp.inet_ehashfn
      0.04 ± 45%      +0.1        0.13 ±  8%  perf-profile.self.cycles-pp.tcp_rate_gen
      0.02 ±141%      +0.1        0.10 ±  8%  perf-profile.self.cycles-pp.tcp_mstamp_refresh
      0.00            +0.1        0.08 ± 14%  perf-profile.self.cycles-pp.netif_skb_features
      0.00            +0.1        0.09 ± 15%  perf-profile.self.cycles-pp.__might_fault
      0.00            +0.1        0.09 ±  7%  perf-profile.self.cycles-pp.tcp_newly_delivered
      0.10 ±  6%      +0.1        0.19 ± 14%  perf-profile.self.cycles-pp.sk_forced_mem_schedule
      0.09 ±  7%      +0.1        0.18 ±  9%  perf-profile.self.cycles-pp.__ksize
      0.00            +0.1        0.09 ±  6%  perf-profile.self.cycles-pp.ip_local_deliver_finish
      0.00            +0.1        0.09 ±  6%  perf-profile.self.cycles-pp.check_stack_object
      0.08 ± 14%      +0.1        0.18 ± 16%  perf-profile.self.cycles-pp.tcp_rtt_estimator
      0.06 ± 11%      +0.1        0.16 ±  5%  perf-profile.self.cycles-pp.sock_put
      0.10 ± 10%      +0.1        0.20 ± 10%  perf-profile.self.cycles-pp.__cond_resched
      0.22 ±  8%      +0.1        0.32 ±  9%  perf-profile.self.cycles-pp.schedule
      0.06 ±  9%      +0.1        0.16 ±  9%  perf-profile.self.cycles-pp.sockfd_lookup_light
      0.00            +0.1        0.10 ± 12%  perf-profile.self.cycles-pp.resched_curr
      0.11 ±  6%      +0.1        0.21 ± 20%  perf-profile.self.cycles-pp.__mod_memcg_state
      0.13 ±  9%      +0.1        0.23 ± 19%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.10 ± 10%      +0.1        0.21 ±  7%  perf-profile.self.cycles-pp.__tcp_select_window
      0.07 ±  5%      +0.1        0.17 ±  8%  perf-profile.self.cycles-pp.enqueue_to_backlog
      0.09 ±  5%      +0.1        0.20 ± 12%  perf-profile.self.cycles-pp.__x64_sys_recvfrom
      0.06 ± 14%      +0.1        0.17 ± 25%  perf-profile.self.cycles-pp.cgroup_rstat_updated
      0.00            +0.1        0.10 ±  4%  perf-profile.self.cycles-pp.tcp_ack_update_rtt
      0.00            +0.1        0.10 ±  4%  perf-profile.self.cycles-pp.tcp_push
      0.01 ±223%      +0.1        0.11 ±  6%  perf-profile.self.cycles-pp.__usecs_to_jiffies
      0.35 ±  7%      +0.1        0.45 ±  7%  perf-profile.self.cycles-pp.update_load_avg
      0.09 ±  7%      +0.1        0.20 ±  8%  perf-profile.self.cycles-pp.sock_sendmsg
      0.00            +0.1        0.11 ±  9%  perf-profile.self.cycles-pp.ip_rcv
      0.12 ±  6%      +0.1        0.23 ±  9%  perf-profile.self.cycles-pp.security_socket_sendmsg
      0.16 ± 10%      +0.1        0.27 ±  8%  perf-profile.self.cycles-pp.sk_wait_data
      0.00            +0.1        0.11 ±  8%  perf-profile.self.cycles-pp.__ip_local_out
      0.09 ± 14%      +0.1        0.20 ±  8%  perf-profile.self.cycles-pp.mem_cgroup_charge_skmem
      0.18 ± 10%      +0.1        0.29 ± 14%  perf-profile.self.cycles-pp.__calc_delta
      0.09 ±  6%      +0.1        0.21 ±  8%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.24 ±  6%      +0.1        0.36 ± 11%  perf-profile.self.cycles-pp.__local_bh_enable_ip
      0.00            +0.1        0.13 ±  8%  perf-profile.self.cycles-pp.cubictcp_cwnd_event
      0.14 ±  7%      +0.1        0.27 ±  5%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.04 ± 71%      +0.1        0.17 ± 11%  perf-profile.self.cycles-pp.skb_page_frag_refill
      0.15 ±  7%      +0.1        0.28 ±  8%  perf-profile.self.cycles-pp.switch_fpu_return
      0.11 ±  6%      +0.1        0.24 ±  9%  perf-profile.self.cycles-pp.tcp_sendmsg
      0.07 ± 14%      +0.1        0.20 ±  8%  perf-profile.self.cycles-pp.tcp_update_skb_after_send
      0.15 ±  4%      +0.1        0.29 ±  9%  perf-profile.self.cycles-pp.__sk_mem_reduce_allocated
      0.29 ±  6%      +0.1        0.42 ±  8%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.29 ±  9%      +0.1        0.42 ±  8%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.10 ±  7%      +0.1        0.24 ±  9%  perf-profile.self.cycles-pp.tcp_tso_segs
      0.11 ±  6%      +0.1        0.25 ±  7%  perf-profile.self.cycles-pp.__netif_receive_skb_core
      0.07 ± 12%      +0.1        0.21 ±  5%  perf-profile.self.cycles-pp.__tcp_push_pending_frames
      0.34 ±  9%      +0.1        0.48 ±  8%  perf-profile.self.cycles-pp.update_curr
      0.18 ±  9%      +0.1        0.32 ±  7%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.08 ± 19%      +0.1        0.22 ± 18%  perf-profile.self.cycles-pp.cpuacct_charge
      0.08 ±  8%      +0.1        0.23 ±  9%  perf-profile.self.cycles-pp.ip_rcv_core
      0.01 ±223%      +0.1        0.16 ± 12%  perf-profile.self.cycles-pp.inet_recvmsg
      0.29 ±  5%      +0.2        0.44 ±  6%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.10 ±  5%      +0.2        0.25 ± 10%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.10 ±  6%      +0.2        0.26 ±  8%  perf-profile.self.cycles-pp.process_backlog
      0.12 ±  8%      +0.2        0.27 ±  7%  perf-profile.self.cycles-pp.__x64_sys_sendto
      0.00            +0.2        0.16 ± 14%  perf-profile.self.cycles-pp.check_preempt_wakeup
      0.28 ±  8%      +0.2        0.44 ±  6%  perf-profile.self.cycles-pp.__skb_clone
      0.12 ±  8%      +0.2        0.28 ±  7%  perf-profile.self.cycles-pp.tcp_schedule_loss_probe
      0.34 ±  4%      +0.2        0.51 ±  8%  perf-profile.self.cycles-pp.update_cfs_group
      0.16 ±  6%      +0.2        0.33 ±  7%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.14 ±  7%      +0.2        0.31 ±  8%  perf-profile.self.cycles-pp.tcp_update_pacing_rate
      0.11 ±  6%      +0.2        0.28 ±  8%  perf-profile.self.cycles-pp.refill_stock
      0.09 ± 13%      +0.2        0.26 ±  8%  perf-profile.self.cycles-pp.ip_protocol_deliver_rcu
      0.00            +0.2        0.17 ± 11%  perf-profile.self.cycles-pp.security_socket_recvmsg
      0.17 ±  9%      +0.2        0.34 ±  8%  perf-profile.self.cycles-pp.__alloc_skb
      0.08 ± 25%      +0.2        0.26 ± 17%  perf-profile.self.cycles-pp.update_sg_lb_stats
      0.27 ±  5%      +0.2        0.46 ±  7%  perf-profile.self.cycles-pp.send_omni_inner
      0.06 ±  7%      +0.2        0.25 ±  8%  perf-profile.self.cycles-pp.__list_add_valid
      0.22 ±  8%      +0.2        0.40 ± 11%  perf-profile.self.cycles-pp.recv
      0.00            +0.2        0.19 ±  8%  perf-profile.self.cycles-pp.exit_to_user_mode_loop
      0.20 ±  8%      +0.2        0.39 ±  8%  perf-profile.self.cycles-pp.send_data
      0.38 ±  8%      +0.2        0.57 ± 23%  perf-profile.self.cycles-pp.ktime_get
      0.45 ±  7%      +0.2        0.65 ±  7%  perf-profile.self.cycles-pp.read_tsc
      0.16 ±  4%      +0.2        0.35 ±  9%  perf-profile.self.cycles-pp.tcp_wfree
      0.00            +0.2        0.20 ±  9%  perf-profile.self.cycles-pp.set_next_buddy
      0.16 ±  9%      +0.2        0.36 ±  6%  perf-profile.self.cycles-pp.__might_sleep
      0.06 ±  7%      +0.2        0.27 ±  8%  perf-profile.self.cycles-pp.__kmem_cache_free
      0.32 ±  5%      +0.2        0.53 ±  9%  perf-profile.self.cycles-pp.__sys_recvfrom
      0.25 ± 14%      +0.2        0.46 ±  9%  perf-profile.self.cycles-pp.recv_omni
      0.62 ±  7%      +0.2        0.84 ±  8%  perf-profile.self.cycles-pp.__schedule
      0.16 ±  7%      +0.2        0.38 ±  9%  perf-profile.self.cycles-pp.__kmem_cache_alloc_node
      0.19 ±  8%      +0.2        0.42 ±  7%  perf-profile.self.cycles-pp.__softirqentry_text_start
      0.67 ±  6%      +0.2        0.89 ±  8%  perf-profile.self.cycles-pp.__switch_to_asm
      0.09 ±  7%      +0.2        0.32 ±  8%  perf-profile.self.cycles-pp._copy_from_iter
      0.18 ±  6%      +0.2        0.42 ±  5%  perf-profile.self.cycles-pp.__send
      0.10 ± 11%      +0.2        0.34 ± 10%  perf-profile.self.cycles-pp.__check_object_size
      0.07 ± 10%      +0.2        0.31 ±  8%  perf-profile.self.cycles-pp.kmem_cache_free
      0.12 ±  7%      +0.3        0.38 ±  8%  perf-profile.self.cycles-pp.kmem_cache_alloc_node
      0.26 ±  6%      +0.3        0.52 ±  7%  perf-profile.self.cycles-pp.do_syscall_64
      0.21 ±  7%      +0.3        0.48 ±  7%  perf-profile.self.cycles-pp.__entry_text_start
      0.22 ±  7%      +0.3        0.50 ±  8%  perf-profile.self.cycles-pp.__sys_sendto
      0.22 ±  8%      +0.3        0.50 ±  6%  perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.53 ±  7%      +0.3        0.84 ±  7%  perf-profile.self.cycles-pp._raw_spin_lock_bh
      0.25 ±  7%      +0.3        0.56 ±  8%  perf-profile.self.cycles-pp.os_xsave
      0.13 ±  4%      +0.3        0.46 ±  7%  perf-profile.self.cycles-pp.__virt_addr_valid
      0.16 ±  7%      +0.3        0.49 ±  7%  perf-profile.self.cycles-pp.loopback_xmit
      0.05 ±  7%      +0.3        0.40 ±  9%  perf-profile.self.cycles-pp.import_single_range
      0.32 ±  6%      +0.4        0.67 ±  9%  perf-profile.self.cycles-pp.ip_finish_output2
      0.01 ±223%      +0.4        0.36 ±  9%  perf-profile.self.cycles-pp.exit_to_user_mode_prepare
      0.21 ±  8%      +0.4        0.59 ±  8%  perf-profile.self.cycles-pp.aa_sk_perm
      0.27 ±  6%      +0.4        0.69 ±  6%  perf-profile.self.cycles-pp.__ip_queue_xmit
      0.29 ±  6%      +0.4        0.73 ±  8%  perf-profile.self.cycles-pp.tcp_write_xmit
      0.27 ± 10%      +0.5        0.74 ± 10%  perf-profile.self.cycles-pp.__fdget
      0.00            +0.5        0.49 ± 10%  perf-profile.self.cycles-pp.kfree
      0.66 ±  6%      +0.5        1.16 ±  8%  perf-profile.self.cycles-pp.__tcp_transmit_skb
      0.38 ±  6%      +0.6        0.96 ±  8%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.00            +1.0        1.01 ±  8%  perf-profile.self.cycles-pp.restore_fpregs_from_fpstate
      0.10 ±  9%      +2.1        2.17 ±  7%  perf-profile.self.cycles-pp.switch_mm_irqs_off



If you fix the issue, kindly add following tag
| Reported-by: kernel test robot <yujie.liu@intel.com>
| Link: https://lore.kernel.org/oe-lkp/202211221125.64e73ddf-yujie.liu@intel.com


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        sudo bin/lkp install job.yaml           # job file is attached in this email
        bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
        sudo bin/lkp run generated-yaml-file

        # if come across any failure that blocks the test,
        # please remove ~/.lkp and /lkp dir to run from a clean state.


Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
diff mbox series

Patch

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index e4a0b8bd941c..40ac3cc68f5b 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -7245,7 +7245,8 @@  select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
 		new_cpu = find_idlest_cpu(sd, p, cpu, prev_cpu, sd_flag);
 	} else if (wake_flags & WF_TTWU) { /* XXX always ? */
 		/* Fast path */
-		new_cpu = select_idle_sibling(p, prev_cpu, new_cpu);
+		if (!sync || cpu != new_cpu || this_rq()->nr_running != 1)
+			new_cpu = select_idle_sibling(p, prev_cpu, new_cpu);
 	}
 	rcu_read_unlock();