diff mbox series

tracing: Ensure visibility when inserting an element into tracing_map

Message ID 20240122150928.27725-1-petr.pavlu@suse.com
State New
Headers
Series tracing: Ensure visibility when inserting an element into tracing_map |

Commit Message

Petr Pavlu Jan. 22, 2024, 3:09 p.m. UTC 
  Running the following two commands in parallel on a multi-processor
AArch64 machine can sporadically produce an unexpected warning about
duplicate histogram entries:

 $ while true; do
     echo hist:key=id.syscall:val=hitcount > \
       /sys/kernel/debug/tracing/events/raw_syscalls/sys_enter/trigger
     cat /sys/kernel/debug/tracing/events/raw_syscalls/sys_enter/hist
     sleep 0.001
   done
 $ stress-ng --sysbadaddr $(nproc)

The warning looks as follows:

[ 2911.172474] ------------[ cut here ]------------
[ 2911.173111] Duplicates detected: 1
[ 2911.173574] WARNING: CPU: 2 PID: 12247 at kernel/trace/tracing_map.c:983 tracing_map_sort_entries+0x3e0/0x408
[ 2911.174702] Modules linked in: iscsi_ibft(E) iscsi_boot_sysfs(E) rfkill(E) af_packet(E) nls_iso8859_1(E) nls_cp437(E) vfat(E) fat(E) ena(E) tiny_power_button(E) qemu_fw_cfg(E) button(E) fuse(E) efi_pstore(E) ip_tables(E) x_tables(E) xfs(E) libcrc32c(E) aes_ce_blk(E) aes_ce_cipher(E) crct10dif_ce(E) polyval_ce(E) polyval_generic(E) ghash_ce(E) gf128mul(E) sm4_ce_gcm(E) sm4_ce_ccm(E) sm4_ce(E) sm4_ce_cipher(E) sm4(E) sm3_ce(E) sm3(E) sha3_ce(E) sha512_ce(E) sha512_arm64(E) sha2_ce(E) sha256_arm64(E) nvme(E) sha1_ce(E) nvme_core(E) nvme_auth(E) t10_pi(E) sg(E) scsi_mod(E) scsi_common(E) efivarfs(E)
[ 2911.174738] Unloaded tainted modules: cppc_cpufreq(E):1
[ 2911.180985] CPU: 2 PID: 12247 Comm: cat Kdump: loaded Tainted: G            E      6.7.0-default #2 1b58bbb22c97e4399dc09f92d309344f69c44a01
[ 2911.182398] Hardware name: Amazon EC2 c7g.8xlarge/, BIOS 1.0 11/1/2018
[ 2911.183208] pstate: 61400005 (nZCv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
[ 2911.184038] pc : tracing_map_sort_entries+0x3e0/0x408
[ 2911.184667] lr : tracing_map_sort_entries+0x3e0/0x408
[ 2911.185310] sp : ffff8000a1513900
[ 2911.185750] x29: ffff8000a1513900 x28: ffff0003f272fe80 x27: 0000000000000001
[ 2911.186600] x26: ffff0003f272fe80 x25: 0000000000000030 x24: 0000000000000008
[ 2911.187458] x23: ffff0003c5788000 x22: ffff0003c16710c8 x21: ffff80008017f180
[ 2911.188310] x20: ffff80008017f000 x19: ffff80008017f180 x18: ffffffffffffffff
[ 2911.189160] x17: 0000000000000000 x16: 0000000000000000 x15: ffff8000a15134b8
[ 2911.190015] x14: 0000000000000000 x13: 205d373432323154 x12: 5b5d313131333731
[ 2911.190844] x11: 00000000fffeffff x10: 00000000fffeffff x9 : ffffd1b78274a13c
[ 2911.191716] x8 : 000000000017ffe8 x7 : c0000000fffeffff x6 : 000000000057ffa8
[ 2911.192554] x5 : ffff0012f6c24ec0 x4 : 0000000000000000 x3 : ffff2e5b72b5d000
[ 2911.193404] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff0003ff254480
[ 2911.194259] Call trace:
[ 2911.194626]  tracing_map_sort_entries+0x3e0/0x408
[ 2911.195220]  hist_show+0x124/0x800
[ 2911.195692]  seq_read_iter+0x1d4/0x4e8
[ 2911.196193]  seq_read+0xe8/0x138
[ 2911.196638]  vfs_read+0xc8/0x300
[ 2911.197078]  ksys_read+0x70/0x108
[ 2911.197534]  __arm64_sys_read+0x24/0x38
[ 2911.198046]  invoke_syscall+0x78/0x108
[ 2911.198553]  el0_svc_common.constprop.0+0xd0/0xf8
[ 2911.199157]  do_el0_svc+0x28/0x40
[ 2911.199613]  el0_svc+0x40/0x178
[ 2911.200048]  el0t_64_sync_handler+0x13c/0x158
[ 2911.200621]  el0t_64_sync+0x1a8/0x1b0
[ 2911.201115] ---[ end trace 0000000000000000 ]---

The problem appears to be caused by CPU reordering of writes issued from
__tracing_map_insert().

The check for the presence of an element with a given key in this
function is:

 val = READ_ONCE(entry->val);
 if (val && keys_match(key, val->key, map->key_size)) ...

The write of a new entry is:

 elt = get_free_elt(map);
 memcpy(elt->key, key, map->key_size);
 entry->val = elt;

The "memcpy(elt->key, key, map->key_size);" and "entry->val = elt;"
stores may become visible in the reversed order on another CPU. This
second CPU might then incorrectly determine that a new key doesn't match
an already present val->key and subsequently insert a new element,
resulting in a duplicate.

Fix the problem by adding a write barrier between
"memcpy(elt->key, key, map->key_size);" and "entry->val = elt;", and for
good measure, also use WRITE_ONCE(entry->val, elt) for publishing the
element. The sequence pairs with the mentioned "READ_ONCE(entry->val);"
and the "val->key" check which has an address dependency.

The barrier is placed on a path executed when adding an element for
a new key. Subsequent updates targeting the same key remain unaffected.

From the user's perspective, the issue was introduced by commit
c193707dde77 ("tracing: Remove code which merges duplicates"), which
followed commit cbf4100efb8f ("tracing: Add support to detect and avoid
duplicates"). The previous code operated differently; it inherently
expected potential races which result in duplicates but merged them
later when they occurred.

Fixes: c193707dde77 ("tracing: Remove code which merges duplicates")
Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>
---
 kernel/trace/tracing_map.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)


base-commit: 9d1694dc91ce7b80bc96d6d8eaf1a1eca668d847

Comments

Tom Zanussi Jan. 22, 2024, 9:55 p.m. UTC | #1 
Hi Petr,

On Mon, 2024-01-22 at 16:09 +0100, Petr Pavlu wrote:
> Running the following two commands in parallel on a multi-processor
> AArch64 machine can sporadically produce an unexpected warning about
> duplicate histogram entries:
> 
>  $ while true; do
>      echo hist:key=id.syscall:val=hitcount > \
>        /sys/kernel/debug/tracing/events/raw_syscalls/sys_enter/trigger
>      cat /sys/kernel/debug/tracing/events/raw_syscalls/sys_enter/hist
>      sleep 0.001
>    done
>  $ stress-ng --sysbadaddr $(nproc)
> 
> The warning looks as follows:
> 
> [ 2911.172474] ------------[ cut here ]------------
> [ 2911.173111] Duplicates detected: 1
> [ 2911.173574] WARNING: CPU: 2 PID: 12247 at kernel/trace/tracing_map.c:983 tracing_map_sort_entries+0x3e0/0x408
> [ 2911.174702] Modules linked in: iscsi_ibft(E) iscsi_boot_sysfs(E) rfkill(E) af_packet(E) nls_iso8859_1(E) nls_cp437(E) vfat(E) fat(E) ena(E) tiny_power_button(E) qemu_fw_cfg(E) button(E) fuse(E) efi_pstore(E) ip_tables(E) x_tables(E) xfs(E) libcrc32c(E) aes_ce_blk(E) aes_ce_cipher(E) crct10dif_ce(E) polyval_ce(E) polyval_generic(E) ghash_ce(E) gf128mul(E) sm4_ce_gcm(E) sm4_ce_ccm(E) sm4_ce(E) sm4_ce_cipher(E) sm4(E) sm3_ce(E) sm3(E) sha3_ce(E) sha512_ce(E) sha512_arm64(E) sha2_ce(E) sha256_arm64(E) nvme(E) sha1_ce(E) nvme_core(E) nvme_auth(E) t10_pi(E) sg(E) scsi_mod(E) scsi_common(E) efivarfs(E)
> [ 2911.174738] Unloaded tainted modules: cppc_cpufreq(E):1
> [ 2911.180985] CPU: 2 PID: 12247 Comm: cat Kdump: loaded Tainted: G            E      6.7.0-default #2 1b58bbb22c97e4399dc09f92d309344f69c44a01
> [ 2911.182398] Hardware name: Amazon EC2 c7g.8xlarge/, BIOS 1.0 11/1/2018
> [ 2911.183208] pstate: 61400005 (nZCv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
> [ 2911.184038] pc : tracing_map_sort_entries+0x3e0/0x408
> [ 2911.184667] lr : tracing_map_sort_entries+0x3e0/0x408
> [ 2911.185310] sp : ffff8000a1513900
> [ 2911.185750] x29: ffff8000a1513900 x28: ffff0003f272fe80 x27: 0000000000000001
> [ 2911.186600] x26: ffff0003f272fe80 x25: 0000000000000030 x24: 0000000000000008
> [ 2911.187458] x23: ffff0003c5788000 x22: ffff0003c16710c8 x21: ffff80008017f180
> [ 2911.188310] x20: ffff80008017f000 x19: ffff80008017f180 x18: ffffffffffffffff
> [ 2911.189160] x17: 0000000000000000 x16: 0000000000000000 x15: ffff8000a15134b8
> [ 2911.190015] x14: 0000000000000000 x13: 205d373432323154 x12: 5b5d313131333731
> [ 2911.190844] x11: 00000000fffeffff x10: 00000000fffeffff x9 : ffffd1b78274a13c
> [ 2911.191716] x8 : 000000000017ffe8 x7 : c0000000fffeffff x6 : 000000000057ffa8
> [ 2911.192554] x5 : ffff0012f6c24ec0 x4 : 0000000000000000 x3 : ffff2e5b72b5d000
> [ 2911.193404] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff0003ff254480
> [ 2911.194259] Call trace:
> [ 2911.194626]  tracing_map_sort_entries+0x3e0/0x408
> [ 2911.195220]  hist_show+0x124/0x800
> [ 2911.195692]  seq_read_iter+0x1d4/0x4e8
> [ 2911.196193]  seq_read+0xe8/0x138
> [ 2911.196638]  vfs_read+0xc8/0x300
> [ 2911.197078]  ksys_read+0x70/0x108
> [ 2911.197534]  __arm64_sys_read+0x24/0x38
> [ 2911.198046]  invoke_syscall+0x78/0x108
> [ 2911.198553]  el0_svc_common.constprop.0+0xd0/0xf8
> [ 2911.199157]  do_el0_svc+0x28/0x40
> [ 2911.199613]  el0_svc+0x40/0x178
> [ 2911.200048]  el0t_64_sync_handler+0x13c/0x158
> [ 2911.200621]  el0t_64_sync+0x1a8/0x1b0
> [ 2911.201115] ---[ end trace 0000000000000000 ]---
> 
> The problem appears to be caused by CPU reordering of writes issued from
> __tracing_map_insert().
> 
> The check for the presence of an element with a given key in this
> function is:
> 
>  val = READ_ONCE(entry->val);
>  if (val && keys_match(key, val->key, map->key_size)) ...
> 
> The write of a new entry is:
> 
>  elt = get_free_elt(map);
>  memcpy(elt->key, key, map->key_size);
>  entry->val = elt;
> 
> The "memcpy(elt->key, key, map->key_size);" and "entry->val = elt;"
> stores may become visible in the reversed order on another CPU. This
> second CPU might then incorrectly determine that a new key doesn't match
> an already present val->key and subsequently insert a new element,
> resulting in a duplicate.
> 
> Fix the problem by adding a write barrier between
> "memcpy(elt->key, key, map->key_size);" and "entry->val = elt;", and for
> good measure, also use WRITE_ONCE(entry->val, elt) for publishing the
> element. The sequence pairs with the mentioned "READ_ONCE(entry->val);"
> and the "val->key" check which has an address dependency.
> 
> The barrier is placed on a path executed when adding an element for
> a new key. Subsequent updates targeting the same key remain unaffected.
> 
> From the user's perspective, the issue was introduced by commit
> c193707dde77 ("tracing: Remove code which merges duplicates"), which
> followed commit cbf4100efb8f ("tracing: Add support to detect and avoid
> duplicates"). The previous code operated differently; it inherently
> expected potential races which result in duplicates but merged them
> later when they occurred.
> 
> Fixes: c193707dde77 ("tracing: Remove code which merges duplicates")
> Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>
> ---
>  kernel/trace/tracing_map.c | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/kernel/trace/tracing_map.c b/kernel/trace/tracing_map.c
> index c774e560f2f9..a4dcf0f24352 100644
> --- a/kernel/trace/tracing_map.c
> +++ b/kernel/trace/tracing_map.c
> @@ -574,7 +574,12 @@ __tracing_map_insert(struct tracing_map *map, void *key, bool lookup_only)
>                                 }
>  
>                                 memcpy(elt->key, key, map->key_size);
> -                               entry->val = elt;
> +                               /*
> +                                * Ensure the initialization is visible and
> +                                * publish the elt.
> +                                */
> +                               smp_wmb();
> +                               WRITE_ONCE(entry->val, elt);
>                                 atomic64_inc(&map->hits);
>  
>                                 return entry->val;
> 
> base-commit: 9d1694dc91ce7b80bc96d6d8eaf1a1eca668d847

Makes sense, thanks for fixing this!

Acked-by: Tom Zanussi <tom.zanussi@linux.intel.com>

Tom
diff mbox series

Patch

diff --git a/kernel/trace/tracing_map.c b/kernel/trace/tracing_map.c
index c774e560f2f9..a4dcf0f24352 100644
--- a/kernel/trace/tracing_map.c
+++ b/kernel/trace/tracing_map.c
@@ -574,7 +574,12 @@  __tracing_map_insert(struct tracing_map *map, void *key, bool lookup_only)
 				}
 
 				memcpy(elt->key, key, map->key_size);
-				entry->val = elt;
+				/*
+				 * Ensure the initialization is visible and
+				 * publish the elt.
+				 */
+				smp_wmb();
+				WRITE_ONCE(entry->val, elt);
 				atomic64_inc(&map->hits);
 
 				return entry->val;