[v2,21/24] selftests/resctrl: Read in less obvious order to defeat prefetch optimizations

  When reading memory in order, HW prefetching optimizations will
interfere with measuring how caches and memory are being accessed. This
adds noise into the results.

Change the fill_buf reading loop to not use an obvious in-order
access using multiply by a prime and modulo.

Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
---
 tools/testing/selftests/resctrl/fill_buf.c | 17 ++++++++++-------
 1 file changed, 10 insertions(+), 7 deletions(-)
  

Hi Ilpo,

> When reading memory in order, HW prefetching optimizations will interfere
> with measuring how caches and memory are being accessed. This adds noise
> into the results.
> 
> Change the fill_buf reading loop to not use an obvious in-order access using
> multiply by a prime and modulo.
> 
> Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
> ---
>  tools/testing/selftests/resctrl/fill_buf.c | 17 ++++++++++-------
>  1 file changed, 10 insertions(+), 7 deletions(-)
> 
> diff --git a/tools/testing/selftests/resctrl/fill_buf.c
> b/tools/testing/selftests/resctrl/fill_buf.c
> index 7e0d3a1ea555..049a520498a9 100644
> --- a/tools/testing/selftests/resctrl/fill_buf.c
> +++ b/tools/testing/selftests/resctrl/fill_buf.c
> @@ -88,14 +88,17 @@ static void *malloc_and_init_memory(size_t s)
> 
>  static int fill_one_span_read(unsigned char *start_ptr, unsigned char
> *end_ptr)  {
> -	unsigned char sum, *p;
> -
> +	unsigned int size = (end_ptr - start_ptr) / (CL_SIZE / 2);
> +	unsigned int count = size;
> +	unsigned char sum;
> +
> +	/*
> +	 * Read the buffer in an order that is unexpected by HW prefetching
> +	 * optimizations to prevent them interfering with the caching pattern.
> +	 */
>  	sum = 0;
> -	p = start_ptr;
> -	while (p < end_ptr) {
> -		sum += *p;
> -		p += (CL_SIZE / 2);
> -	}
> +	while (count--)
> +		sum += start_ptr[((count * 59) % size) * CL_SIZE / 2];
Could you please elaborate why 59 is used?

Best regards,
Shaopeng TAN
  

On Wed, 31 May 2023, Shaopeng Tan (Fujitsu) wrote:

> Hi Ilpo,
> 
> > When reading memory in order, HW prefetching optimizations will interfere
> > with measuring how caches and memory are being accessed. This adds noise
> > into the results.
> > 
> > Change the fill_buf reading loop to not use an obvious in-order access using
> > multiply by a prime and modulo.
> > 
> > Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
> > ---
> >  tools/testing/selftests/resctrl/fill_buf.c | 17 ++++++++++-------
> >  1 file changed, 10 insertions(+), 7 deletions(-)
> > 
> > diff --git a/tools/testing/selftests/resctrl/fill_buf.c
> > b/tools/testing/selftests/resctrl/fill_buf.c
> > index 7e0d3a1ea555..049a520498a9 100644
> > --- a/tools/testing/selftests/resctrl/fill_buf.c
> > +++ b/tools/testing/selftests/resctrl/fill_buf.c
> > @@ -88,14 +88,17 @@ static void *malloc_and_init_memory(size_t s)
> > 
> >  static int fill_one_span_read(unsigned char *start_ptr, unsigned char
> > *end_ptr)  {
> > -	unsigned char sum, *p;
> > -
> > +	unsigned int size = (end_ptr - start_ptr) / (CL_SIZE / 2);
> > +	unsigned int count = size;
> > +	unsigned char sum;
> > +
> > +	/*
> > +	 * Read the buffer in an order that is unexpected by HW prefetching
> > +	 * optimizations to prevent them interfering with the caching pattern.
> > +	 */
> >  	sum = 0;
> > -	p = start_ptr;
> > -	while (p < end_ptr) {
> > -		sum += *p;
> > -		p += (CL_SIZE / 2);
> > -	}
> > +	while (count--)
> > +		sum += start_ptr[((count * 59) % size) * CL_SIZE / 2];
>
> Could you please elaborate why 59 is used?

The main reason is that it's a prime number ensuring the whole buffer 
gets read. I picked something that doesn't make it to wrap on almost 
every iteration.
  

Hi Ilpo,

> > > When reading memory in order, HW prefetching optimizations will
> > > interfere with measuring how caches and memory are being accessed.
> > > This adds noise into the results.
> > >
> > > Change the fill_buf reading loop to not use an obvious in-order
> > > access using multiply by a prime and modulo.
> > >
> > > Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
> > > ---
> > >  tools/testing/selftests/resctrl/fill_buf.c | 17 ++++++++++-------
> > >  1 file changed, 10 insertions(+), 7 deletions(-)
> > >
> > > diff --git a/tools/testing/selftests/resctrl/fill_buf.c
> > > b/tools/testing/selftests/resctrl/fill_buf.c
> > > index 7e0d3a1ea555..049a520498a9 100644
> > > --- a/tools/testing/selftests/resctrl/fill_buf.c
> > > +++ b/tools/testing/selftests/resctrl/fill_buf.c
> > > @@ -88,14 +88,17 @@ static void *malloc_and_init_memory(size_t s)
> > >
> > >  static int fill_one_span_read(unsigned char *start_ptr, unsigned
> > > char
> > > *end_ptr)  {
> > > -	unsigned char sum, *p;
> > > -
> > > +	unsigned int size = (end_ptr - start_ptr) / (CL_SIZE / 2);
> > > +	unsigned int count = size;
> > > +	unsigned char sum;
> > > +
> > > +	/*
> > > +	 * Read the buffer in an order that is unexpected by HW prefetching
> > > +	 * optimizations to prevent them interfering with the caching pattern.
> > > +	 */
> > >  	sum = 0;
> > > -	p = start_ptr;
> > > -	while (p < end_ptr) {
> > > -		sum += *p;
> > > -		p += (CL_SIZE / 2);
> > > -	}
> > > +	while (count--)
> > > +		sum += start_ptr[((count * 59) % size) * CL_SIZE / 2];
> >
> > Could you please elaborate why 59 is used?
> 
> The main reason is that it's a prime number ensuring the whole buffer gets read.
> I picked something that doesn't make it to wrap on almost every iteration.

Thanks for your explanation. It seems there is no problem.

Perhaps you have already tested this patch in your environment and got a test result of "ok". 
Because HW prefetching does not work well,
the IMC counter fluctuates a lot in my environment,
and the test result is "not ok". 

In order to ensure this test set runs in any environments and gets "ok",
would you consider changing the value of MAX_DIFF_PERCENT of each test?
or changing something else?

```
Environment:
 Kernel: 6.4.0-rc2
 CPU: Intel(R) Xeon(R) Gold 6254 CPU @ 3.10GHz

Test result(MBM as an example):
# # Starting MBM BW change ...
# # Mounting resctrl to "/sys/fs/resctrl"
# # Benchmark PID: 8671
# # Writing benchmark parameters to resctrl FS
# # Write schema "MB:0=100" to resctrl FS
# # Checking for pass/fail
# # Fail: Check MBM diff within 5%
# # avg_diff_per: 9%
# # Span in bytes: 262144000
# # avg_bw_imc: 6202
# # avg_bw_resc: 5585
# not ok 1 MBM: bw change
```

Best regards,
Shaopeng TAN
  

On Thu, 1 Jun 2023, Shaopeng Tan (Fujitsu) wrote:
>
> > > > When reading memory in order, HW prefetching optimizations will
> > > > interfere with measuring how caches and memory are being accessed.
> > > > This adds noise into the results.
> > > >
> > > > Change the fill_buf reading loop to not use an obvious in-order
> > > > access using multiply by a prime and modulo.
> > > >
> > > > Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
> > > > ---
> > > >  tools/testing/selftests/resctrl/fill_buf.c | 17 ++++++++++-------
> > > >  1 file changed, 10 insertions(+), 7 deletions(-)
> > > >
> > > > diff --git a/tools/testing/selftests/resctrl/fill_buf.c
> > > > b/tools/testing/selftests/resctrl/fill_buf.c
> > > > index 7e0d3a1ea555..049a520498a9 100644
> > > > --- a/tools/testing/selftests/resctrl/fill_buf.c
> > > > +++ b/tools/testing/selftests/resctrl/fill_buf.c
> > > > @@ -88,14 +88,17 @@ static void *malloc_and_init_memory(size_t s)
> > > >
> > > >  static int fill_one_span_read(unsigned char *start_ptr, unsigned
> > > > char
> > > > *end_ptr)  {
> > > > -	unsigned char sum, *p;
> > > > -
> > > > +	unsigned int size = (end_ptr - start_ptr) / (CL_SIZE / 2);
> > > > +	unsigned int count = size;
> > > > +	unsigned char sum;
> > > > +
> > > > +	/*
> > > > +	 * Read the buffer in an order that is unexpected by HW prefetching
> > > > +	 * optimizations to prevent them interfering with the caching pattern.
> > > > +	 */
> > > >  	sum = 0;
> > > > -	p = start_ptr;
> > > > -	while (p < end_ptr) {
> > > > -		sum += *p;
> > > > -		p += (CL_SIZE / 2);
> > > > -	}
> > > > +	while (count--)
> > > > +		sum += start_ptr[((count * 59) % size) * CL_SIZE / 2];
> > >
> > > Could you please elaborate why 59 is used?
> > 
> > The main reason is that it's a prime number ensuring the whole buffer gets read.
> > I picked something that doesn't make it to wrap on almost every iteration.
> 
> Thanks for your explanation. It seems there is no problem.
> 
> Perhaps you have already tested this patch in your environment and got a 
> test result of "ok".  

Yes, it was tested :-) and all looked fine here. But my testing was more 
focused on the systems which come with CAT and on all those, this change 
clearly improved MBA/MBM results (they became almost always diff=0 except 
for the smallest ones in the MBA test).

> Because HW prefetching does not work well,
> the IMC counter fluctuates a lot in my environment,
> and the test result is "not ok". 
>
> In order to ensure this test set runs in any environments and gets "ok",
> would you consider changing the value of MAX_DIFF_PERCENT of each test?
> or changing something else?
>
> ```
> Environment:
>  Kernel: 6.4.0-rc2
>  CPU: Intel(R) Xeon(R) Gold 6254 CPU @ 3.10GHz
> 
> Test result(MBM as an example):
> # # Starting MBM BW change ...
> # # Mounting resctrl to "/sys/fs/resctrl"
> # # Benchmark PID: 8671
> # # Writing benchmark parameters to resctrl FS
> # # Write schema "MB:0=100" to resctrl FS
> # # Checking for pass/fail
> # # Fail: Check MBM diff within 5%
> # # avg_diff_per: 9%
> # # Span in bytes: 262144000
> # # avg_bw_imc: 6202
> # # avg_bw_resc: 5585
> # not ok 1 MBM: bw change

Oh, I see. It seems that these CPUs break the trend and get much worse 
and more unstable for some reason. It might be that some i9 I recently 
got a lkp report from could have the same problem. I'll look more into 
this, thanks a lot for testing and bringing it up.

So to answer your question above, I've no intention to tweak 
MAX_DIFF_PERCENT because of this issue but I'll instead try to improve the 
approach to defeat the HW prefetcher.

If HW prefetcher is not defeated, the CAT test LLC misses have a slowly 
converging ramp which is not very useful unless number of runs is 
increased by much (and perhaps the first samples dropped entirely). So
it is kinda needed and it would be nice if an approach that is non-HW 
specific could be used for this.

It will probably take some time... Should I send a v3 with only the fixes 
and useful refactors at the head of this series while I try to sort these 
problems with the test changes out?
  

Hi Ilpo,

On 6/2/2023 6:51 AM, Ilpo Järvinen wrote:
> It will probably take some time... Should I send a v3 with only the fixes 
> and useful refactors at the head of this series while I try to sort these 
> problems with the test changes out?

This sounds good to me. This series is already at 24 patches so I think that
splitting the redesign of the CAT test from the other fixes would indeed make
this work easier to parse.

Thank you

Reinette
  

On Thu, 1 Jun 2023, Shaopeng Tan (Fujitsu) wrote:
> > > > When reading memory in order, HW prefetching optimizations will
> > > > interfere with measuring how caches and memory are being accessed.
> > > > This adds noise into the results.
> > > >
> > > > Change the fill_buf reading loop to not use an obvious in-order
> > > > access using multiply by a prime and modulo.
> > > >
> > > > Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
> > > > ---
> > > >  tools/testing/selftests/resctrl/fill_buf.c | 17 ++++++++++-------
> > > >  1 file changed, 10 insertions(+), 7 deletions(-)
> > > >
> > > > diff --git a/tools/testing/selftests/resctrl/fill_buf.c
> > > > b/tools/testing/selftests/resctrl/fill_buf.c
> > > > index 7e0d3a1ea555..049a520498a9 100644
> > > > --- a/tools/testing/selftests/resctrl/fill_buf.c
> > > > +++ b/tools/testing/selftests/resctrl/fill_buf.c
> > > > @@ -88,14 +88,17 @@ static void *malloc_and_init_memory(size_t s)
> > > >
> > > >  static int fill_one_span_read(unsigned char *start_ptr, unsigned
> > > > char
> > > > *end_ptr)  {
> > > > -	unsigned char sum, *p;
> > > > -
> > > > +	unsigned int size = (end_ptr - start_ptr) / (CL_SIZE / 2);
> > > > +	unsigned int count = size;
> > > > +	unsigned char sum;
> > > > +
> > > > +	/*
> > > > +	 * Read the buffer in an order that is unexpected by HW prefetching
> > > > +	 * optimizations to prevent them interfering with the caching pattern.
> > > > +	 */
> > > >  	sum = 0;
> > > > -	p = start_ptr;
> > > > -	while (p < end_ptr) {
> > > > -		sum += *p;
> > > > -		p += (CL_SIZE / 2);
> > > > -	}
> > > > +	while (count--)
> > > > +		sum += start_ptr[((count * 59) % size) * CL_SIZE / 2];
> > >
> > > Could you please elaborate why 59 is used?
> > 
> > The main reason is that it's a prime number ensuring the whole buffer gets read.
> > I picked something that doesn't make it to wrap on almost every iteration.
> 
> Thanks for your explanation. It seems there is no problem.
> 
> Perhaps you have already tested this patch in your environment and got a test result of "ok". 
> Because HW prefetching does not work well,
> the IMC counter fluctuates a lot in my environment,
> and the test result is "not ok". 
> 
> In order to ensure this test set runs in any environments and gets "ok",
> would you consider changing the value of MAX_DIFF_PERCENT of each test?
> or changing something else?
> 
> ```
> Environment:
>  Kernel: 6.4.0-rc2
>  CPU: Intel(R) Xeon(R) Gold 6254 CPU @ 3.10GHz
> 
> Test result(MBM as an example):
> # # Starting MBM BW change ...
> # # Mounting resctrl to "/sys/fs/resctrl"
> # # Benchmark PID: 8671
> # # Writing benchmark parameters to resctrl FS
> # # Write schema "MB:0=100" to resctrl FS
> # # Checking for pass/fail
> # # Fail: Check MBM diff within 5%
> # # avg_diff_per: 9%
> # # Span in bytes: 262144000
> # # avg_bw_imc: 6202
> # # avg_bw_resc: 5585
> # not ok 1 MBM: bw change
> ```

Could you try if the approach below works better (I think it should apply 
cleanly on top of the fixes+cleanups v3 series which you recently tested, 
no need to have the other CAT test changes).

The biggest difference in terms of result stability my tests come from 
these factors:
- Removed reversed index order.
- Open-coded the modulo in the loop to subtraction.

In addition, I changed the prime to one which works slightly better than 
59. The MBM/MBA results were already <5% with 59 too due to the other two 
changes, but using 23 lowered them further in my tests (with Platinum 
8260L).

---
From: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
[PATCH] selftests/resctrl: Read in less obvious order to defeat prefetch optimizations

When reading memory in order, HW prefetching optimizations will
interfere with measuring how caches and memory are being accessed. This
adds noise into the results.

Change the fill_buf reading loop to not use an obvious in-order access
using multiply by a prime and modulo.

Using a prime multiplier with modulo ensures the entire buffer is
eventually read. 23 is small enough that the reads are spread out but
wrapping does not occur very frequently (wrapping too often can trigger
L2 hits more frequently which causes noise to the test because getting
the data from LLC is not required).

It was discovered that not all primes work equally well and some can
cause wildly unstable results (e.g., in an earlier version of this
patch, the reads were done in reversed order and 59 was used as the
prime resulting in unacceptably high and unstable results in MBA and
MBM test on some architectures).

Link: https://lore.kernel.org/linux-kselftest/TYAPR01MB6330025B5E6537F94DA49ACB8B499@TYAPR01MB6330.jpnprd01.prod.outlook.com/
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>

---
 tools/testing/selftests/resctrl/fill_buf.c | 38 +++++++++++++++++++++++-------
 1 file changed, 30 insertions(+), 8 deletions(-)

diff --git a/tools/testing/selftests/resctrl/fill_buf.c b/tools/testing/selftests/resctrl/fill_buf.c
index f9893edda869..afde37d3fca0 100644
--- a/tools/testing/selftests/resctrl/fill_buf.c
+++ b/tools/testing/selftests/resctrl/fill_buf.c
@@ -74,16 +74,38 @@ static void *malloc_and_init_memory(size_t buf_size)
 	return p;
 }
 
+/*
+ * Buffer index step advance to workaround HW prefetching interfering with
+ * the measurements.
+ *
+ * Must be a prime to step through all indexes of the buffer.
+ *
+ * Some primes work better than others on some architectures (from MBA/MBM
+ * result stability point of view).
+ */
+#define FILL_IDX_MULT	23
+
 static int fill_one_span_read(unsigned char *buf, size_t buf_size)
 {
-	unsigned char *end_ptr = buf + buf_size;
-	unsigned char sum, *p;
-
-	sum = 0;
-	p = buf;
-	while (p < end_ptr) {
-		sum += *p;
-		p += (CL_SIZE / 2);
+	unsigned int size = buf_size / (CL_SIZE / 2);
+	unsigned int i, idx = 0;
+	unsigned char sum = 0;
+
+	/*
+	 * Read the buffer in an order that is unexpected by HW prefetching
+	 * optimizations to prevent them interfering with the caching pattern.
+	 *
+	 * The read order is (in terms of halves of cachelines):
+	 *	i * FILL_IDX_MULT % size
+	 * The formula is open-coded below to avoiding modulo inside the loop
+	 * as it improves MBA/MBM result stability on some architectures.
+	 */
+	for (i = 0; i < size; i++) {
+		sum += buf[idx * (CL_SIZE / 2)];
+
+		idx += FILL_IDX_MULT;
+		while (idx >= size)
+			idx -= size;
 	}
 
 	return sum;
  

Hi Ilpo,

> On Thu, 1 Jun 2023, Shaopeng Tan (Fujitsu) wrote:
> > > > > When reading memory in order, HW prefetching optimizations will
> > > > > interfere with measuring how caches and memory are being accessed.
> > > > > This adds noise into the results.
> > > > >
> > > > > Change the fill_buf reading loop to not use an obvious in-order
> > > > > access using multiply by a prime and modulo.
> > > > >
> > > > > Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
> > > > > ---
> > > > >  tools/testing/selftests/resctrl/fill_buf.c | 17
> > > > > ++++++++++-------
> > > > >  1 file changed, 10 insertions(+), 7 deletions(-)
> > > > >
> > > > > diff --git a/tools/testing/selftests/resctrl/fill_buf.c
> > > > > b/tools/testing/selftests/resctrl/fill_buf.c
> > > > > index 7e0d3a1ea555..049a520498a9 100644
> > > > > --- a/tools/testing/selftests/resctrl/fill_buf.c
> > > > > +++ b/tools/testing/selftests/resctrl/fill_buf.c
> > > > > @@ -88,14 +88,17 @@ static void *malloc_and_init_memory(size_t
> > > > > s)
> > > > >
> > > > >  static int fill_one_span_read(unsigned char *start_ptr,
> > > > > unsigned char
> > > > > *end_ptr)  {
> > > > > -	unsigned char sum, *p;
> > > > > -
> > > > > +	unsigned int size = (end_ptr - start_ptr) / (CL_SIZE / 2);
> > > > > +	unsigned int count = size;
> > > > > +	unsigned char sum;
> > > > > +
> > > > > +	/*
> > > > > +	 * Read the buffer in an order that is unexpected by HW
> prefetching
> > > > > +	 * optimizations to prevent them interfering with the caching
> pattern.
> > > > > +	 */
> > > > >  	sum = 0;
> > > > > -	p = start_ptr;
> > > > > -	while (p < end_ptr) {
> > > > > -		sum += *p;
> > > > > -		p += (CL_SIZE / 2);
> > > > > -	}
> > > > > +	while (count--)
> > > > > +		sum += start_ptr[((count * 59) % size) * CL_SIZE / 2];
> > > >
> > > > Could you please elaborate why 59 is used?
> > >
> > > The main reason is that it's a prime number ensuring the whole buffer gets
> read.
> > > I picked something that doesn't make it to wrap on almost every iteration.
> >
> > Thanks for your explanation. It seems there is no problem.
> >
> > Perhaps you have already tested this patch in your environment and got a test
> result of "ok".
> > Because HW prefetching does not work well, the IMC counter fluctuates
> > a lot in my environment, and the test result is "not ok".
> >
> > In order to ensure this test set runs in any environments and gets
> > "ok", would you consider changing the value of MAX_DIFF_PERCENT of
> each test?
> > or changing something else?
> >
> > ```
> > Environment:
> >  Kernel: 6.4.0-rc2
> >  CPU: Intel(R) Xeon(R) Gold 6254 CPU @ 3.10GHz
> >
> > Test result(MBM as an example):
> > # # Starting MBM BW change ...
> > # # Mounting resctrl to "/sys/fs/resctrl"
> > # # Benchmark PID: 8671
> > # # Writing benchmark parameters to resctrl FS # # Write schema
> > "MB:0=100" to resctrl FS # # Checking for pass/fail # # Fail: Check
> > MBM diff within 5% # # avg_diff_per: 9% # # Span in bytes: 262144000 #
> > # avg_bw_imc: 6202 # # avg_bw_resc: 5585 # not ok 1 MBM: bw change ```
> 
> Could you try if the approach below works better (I think it should apply cleanly
> on top of the fixes+cleanups v3 series which you recently tested, no need to
> have the other CAT test changes).

I ran the test set several times. 
MBA and MBM seem fine, but CAT is always "not ok".
The result is below.

---
$ sudo make -C tools/testing/selftests/resctrl run_tests
make: Entering directory '/**/tools/testing/selftests/resctrl'
TAP version 13
1..1
# selftests: resctrl: resctrl_tests
# TAP version 13
# # Pass: Check kernel supports resctrl filesystem
# # Pass: Check resctrl mountpoint "/sys/fs/resctrl" exists
# # resctrl filesystem not mounted
# # dmesg: [    3.658029] resctrl: L3 allocation detected
# # dmesg: [    3.658420] resctrl: MB allocation detected
# # dmesg: [    3.658604] resctrl: L3 monitoring detected
# 1..4
# # Starting MBM BW change ...
# # Mounting resctrl to "/sys/fs/resctrl"
# # Benchmark PID: 9555
# # Writing benchmark parameters to resctrl FS
# # Write schema "MB:0=100" to resctrl FS
# # Checking for pass/fail
# # Pass: Check MBM diff within 5%
# # avg_diff_per: 0%
# # Span (MB): 250
# # avg_bw_imc: 6880
# # avg_bw_resc: 6895
# ok 1 MBM: bw change
# # Starting MBA Schemata change ...
# # Mounting resctrl to "/sys/fs/resctrl"
# # Benchmark PID: 9561
# # Writing benchmark parameters to resctrl FS
# # Write schema "MB:0=100" to resctrl FS
# # Write schema "MB:0=90" to resctrl FS
# # Write schema "MB:0=80" to resctrl FS
# # Write schema "MB:0=70" to resctrl FS
# # Write schema "MB:0=60" to resctrl FS
# # Write schema "MB:0=50" to resctrl FS
# # Write schema "MB:0=40" to resctrl FS
# # Write schema "MB:0=30" to resctrl FS
# # Write schema "MB:0=20" to resctrl FS
# # Write schema "MB:0=10" to resctrl FS
# # Results are displayed in (MB)
# # Pass: Check MBA diff within 5% for schemata 100
# # avg_diff_per: 0%
# # avg_bw_imc: 6874
# # avg_bw_resc: 6904
# # Pass: Check MBA diff within 5% for schemata 90
# # avg_diff_per: 1%
# # avg_bw_imc: 6738
# # avg_bw_resc: 6807
# # Pass: Check MBA diff within 5% for schemata 80
# # avg_diff_per: 1%
# # avg_bw_imc: 6735
# # avg_bw_resc: 6803
# # Pass: Check MBA diff within 5% for schemata 70
# # avg_diff_per: 1%
# # avg_bw_imc: 6702
# # avg_bw_resc: 6770
# # Pass: Check MBA diff within 5% for schemata 60
# # avg_diff_per: 1%
# # avg_bw_imc: 6632
# # avg_bw_resc: 6725
# # Pass: Check MBA diff within 5% for schemata 50
# # avg_diff_per: 1%
# # avg_bw_imc: 6510
# # avg_bw_resc: 6635
# # Pass: Check MBA diff within 5% for schemata 40
# # avg_diff_per: 2%
# # avg_bw_imc: 6206
# # avg_bw_resc: 6342
# # Pass: Check MBA diff within 5% for schemata 30
# # avg_diff_per: 1%
# # avg_bw_imc: 3826
# # avg_bw_resc: 3896
# # Pass: Check MBA diff within 5% for schemata 20
# # avg_diff_per: 1%
# # avg_bw_imc: 2820
# # avg_bw_resc: 2862
# # Pass: Check MBA diff within 5% for schemata 10
# # avg_diff_per: 1%
# # avg_bw_imc: 1876
# # avg_bw_resc: 1898
# # Pass: Check schemata change using MBA
# ok 2 MBA: schemata change
# # Starting CMT test ...
# # Mounting resctrl to "/sys/fs/resctrl"
# # Cache size :25952256
# # Benchmark PID: 9573
# # Writing benchmark parameters to resctrl FS
# # Checking for pass/fail
# # Pass: Check cache miss rate within 15%
# # Percent diff=10
# # Number of bits: 5
# # Average LLC val: 12994560
# # Cache span (bytes): 11796480
# ok 3 CMT: test
# # Starting CAT test ...
# # Mounting resctrl to "/sys/fs/resctrl"
# # Cache size :25952256
# # Writing benchmark parameters to resctrl FS
# # Write schema "L3:0=3f" to resctrl FS
# # Checking for pass/fail
# # Fail: Check cache miss rate within 4%
# # Percent diff=24
# # Number of bits: 6
# # Average LLC val: 275418
# # Cache span (lines): 221184
# not ok 4 CAT: test
# # Totals: pass:3 fail:1 xfail:0 xpass:0 skip:0 error:0
not ok 1 selftests: resctrl: resctrl_tests # exit=1
make: Leaving directory '/**/tools/testing/selftests/resctrl'
---

Best regards,
Shaopeng TAN
  

On Fri, 16 Jun 2023, Shaopeng Tan (Fujitsu) wrote:

> Hi Ilpo,
> 
> > On Thu, 1 Jun 2023, Shaopeng Tan (Fujitsu) wrote:
> > > > > > When reading memory in order, HW prefetching optimizations will
> > > > > > interfere with measuring how caches and memory are being accessed.
> > > > > > This adds noise into the results.
> > > > > >
> > > > > > Change the fill_buf reading loop to not use an obvious in-order
> > > > > > access using multiply by a prime and modulo.
> > > > > >
> > > > > > Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
> > > > > > ---
> > > > > >  tools/testing/selftests/resctrl/fill_buf.c | 17
> > > > > > ++++++++++-------
> > > > > >  1 file changed, 10 insertions(+), 7 deletions(-)
> > > > > >
> > > > > > diff --git a/tools/testing/selftests/resctrl/fill_buf.c
> > > > > > b/tools/testing/selftests/resctrl/fill_buf.c
> > > > > > index 7e0d3a1ea555..049a520498a9 100644
> > > > > > --- a/tools/testing/selftests/resctrl/fill_buf.c
> > > > > > +++ b/tools/testing/selftests/resctrl/fill_buf.c
> > > > > > @@ -88,14 +88,17 @@ static void *malloc_and_init_memory(size_t
> > > > > > s)
> > > > > >
> > > > > >  static int fill_one_span_read(unsigned char *start_ptr,
> > > > > > unsigned char
> > > > > > *end_ptr)  {
> > > > > > -	unsigned char sum, *p;
> > > > > > -
> > > > > > +	unsigned int size = (end_ptr - start_ptr) / (CL_SIZE / 2);
> > > > > > +	unsigned int count = size;
> > > > > > +	unsigned char sum;
> > > > > > +
> > > > > > +	/*
> > > > > > +	 * Read the buffer in an order that is unexpected by HW
> > prefetching
> > > > > > +	 * optimizations to prevent them interfering with the caching
> > pattern.
> > > > > > +	 */
> > > > > >  	sum = 0;
> > > > > > -	p = start_ptr;
> > > > > > -	while (p < end_ptr) {
> > > > > > -		sum += *p;
> > > > > > -		p += (CL_SIZE / 2);
> > > > > > -	}
> > > > > > +	while (count--)
> > > > > > +		sum += start_ptr[((count * 59) % size) * CL_SIZE / 2];
> > > > >
> > > > > Could you please elaborate why 59 is used?
> > > >
> > > > The main reason is that it's a prime number ensuring the whole buffer gets
> > read.
> > > > I picked something that doesn't make it to wrap on almost every iteration.
> > >
> > > Thanks for your explanation. It seems there is no problem.
> > >
> > > Perhaps you have already tested this patch in your environment and got a test
> > result of "ok".
> > > Because HW prefetching does not work well, the IMC counter fluctuates
> > > a lot in my environment, and the test result is "not ok".
> > >
> > > In order to ensure this test set runs in any environments and gets
> > > "ok", would you consider changing the value of MAX_DIFF_PERCENT of
> > each test?
> > > or changing something else?
> > >
> > > ```
> > > Environment:
> > >  Kernel: 6.4.0-rc2
> > >  CPU: Intel(R) Xeon(R) Gold 6254 CPU @ 3.10GHz
> > >
> > > Test result(MBM as an example):
> > > # # Starting MBM BW change ...
> > > # # Mounting resctrl to "/sys/fs/resctrl"
> > > # # Benchmark PID: 8671
> > > # # Writing benchmark parameters to resctrl FS # # Write schema
> > > "MB:0=100" to resctrl FS # # Checking for pass/fail # # Fail: Check
> > > MBM diff within 5% # # avg_diff_per: 9% # # Span in bytes: 262144000 #
> > > # avg_bw_imc: 6202 # # avg_bw_resc: 5585 # not ok 1 MBM: bw change ```
> > 
> > Could you try if the approach below works better (I think it should apply cleanly
> > on top of the fixes+cleanups v3 series which you recently tested, no need to
> > have the other CAT test changes).
> 
> I ran the test set several times. 
> MBA and MBM seem fine, but CAT is always "not ok".
> The result is below.

Ok, thanks a lot for confirming. I was just interested to see MBA/MBM test 
results.

I'm not surprised the old CAT test is failing with "not ok". I see it 
occurring quite often with the old CAT test. It is one of the reasons why 
it is being rewritten, although the main motivator is that the old one 
doesn't really even test CAT because it flushes LLC and reads the 
buffer only once after the flush.

The rewritten CAT test should work better in this regard but it was not 
among fixes+cleanups series (v3) + this patch.