[v3,4/4] selftests: kmod: Add tests for merging same-name module load requests

  Add two tests to check that loading the same module multiple times in
parallel results only in one real attempt to initialize it.
Synchronization of the loads is done by waiting 1000 ms in the init
function of a sample module kmod_test_0014. The tests measure time
needed to perform all inserts to verify that the loads get merged by the
module loader and are not serialized.

* Case 0014 checks a situation when the load is successful. It should
  result in one insert returning 0 and remaining inserts returning
  EEXIST.
* Case 0015 checks a situation when the load is failing because the
  module init function returns ENODEV. It should result in one insert
  returning this error code and remaining inserts returning EBUSY.

The tests use a simple init_module program to load kmod_test_0014.ko. It
enables to obtain directly a return code from the finit_module syscall.

Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>
---
 tools/testing/selftests/kmod/.gitignore       |   1 +
 tools/testing/selftests/kmod/Makefile         |  17 ++-
 tools/testing/selftests/kmod/init_module.c    |  49 ++++++
 tools/testing/selftests/kmod/kmod.sh          | 139 ++++++++++++++++++
 .../selftests/kmod/kmod_test_0014/Makefile    |  14 ++
 .../kmod/kmod_test_0014/kmod_test_0014.c      |  29 ++++
 6 files changed, 244 insertions(+), 5 deletions(-)
 create mode 100644 tools/testing/selftests/kmod/.gitignore
 create mode 100644 tools/testing/selftests/kmod/init_module.c
 create mode 100644 tools/testing/selftests/kmod/kmod_test_0014/Makefile
 create mode 100644 tools/testing/selftests/kmod/kmod_test_0014/kmod_test_0014.c
  

On Sun 2022-10-16 14:30:31, Petr Pavlu wrote:
> Add two tests to check that loading the same module multiple times in
> parallel results only in one real attempt to initialize it.
> Synchronization of the loads is done by waiting 1000 ms in the init

I do not have a good experience with this kind of synchronization.
It usually is not reliable. The test might be very slow especially when
false positives are solved by prolonging the delay.

Alternative solution would be to have two modules:

1st module would provide a counter, for example:

int modB_load_cnt;
module_param(modB_load_cnt, int, 0444);
EXPORT_SYMBOL(modB_load_cnt);

EXPORT_SYMBOL() should allow to directly increment the counter
from the 2nd module.

module_param() should make the value readable via
/sys/module/modA/parameters/modB_load_cnt. It can be
checked by kmod_sh.

> function of a sample module kmod_test_0014. The tests measure time
> needed to perform all inserts to verify that the loads get merged by the
> module loader and are not serialized.
> 
> * Case 0014 checks a situation when the load is successful. It should
>   result in one insert returning 0 and remaining inserts returning
>   EEXIST.
> * Case 0015 checks a situation when the load is failing because the
>   module init function returns ENODEV. It should result in one insert
>   returning this error code and remaining inserts returning EBUSY.
> 
> The tests use a simple init_module program to load kmod_test_0014.ko. It
> enables to obtain directly a return code from the finit_module syscall.

The tested scenarios look good.

Best Regards,
Petr
  

On Sun, Oct 16, 2022 at 02:30:31PM +0200, Petr Pavlu wrote:
> diff --git a/tools/testing/selftests/kmod/init_module.c b/tools/testing/selftests/kmod/init_module.c
> @@ -525,6 +662,8 @@ list_tests()
>  	echo "0011 x $(get_test_count 0011) - test completely disabling module autoloading"
>  	echo "0012 x $(get_test_count 0012) - test /proc/modules address visibility under CAP_SYSLOG"
>  	echo "0013 x $(get_test_count 0013) - test /sys/module/*/sections/* visibility under CAP_SYSLOG"
> +	echo "0014 x $(get_test_count 0014) - test handling of parallel loads, success case"
> +	echo "0015 x $(get_test_count 0015) - test handling of parallel loads, init returning error"

Good stuff!

So test 0015 mimics the error reported by Prarit Bhargava through
commit 6e6de3dee51a ("kernel/module.c: Only return -EEXIST for modules
that have finished loading")? If so it would be good to document that
here.

Also, this patch should go first, with the ALL_TESTS variable set to
disable the tests which are known to fail, so to demonstrate the *new*
issues as they are failing and then your fix first and then you enable
that test afterwards.

If 6e6de3dee51a introduced another regression which is not covered
by the tests we'd add it, and fix on top of it as a stepping stone
to prove / send to stable.

  Luis
  

On Tue, Oct 25, 2022 at 10:51:14AM -0700, Luis Chamberlain wrote:
> On Sun, Oct 16, 2022 at 02:30:31PM +0200, Petr Pavlu wrote:
> > diff --git a/tools/testing/selftests/kmod/init_module.c b/tools/testing/selftests/kmod/init_module.c
> > @@ -525,6 +662,8 @@ list_tests()
> >  	echo "0011 x $(get_test_count 0011) - test completely disabling module autoloading"
> >  	echo "0012 x $(get_test_count 0012) - test /proc/modules address visibility under CAP_SYSLOG"
> >  	echo "0013 x $(get_test_count 0013) - test /sys/module/*/sections/* visibility under CAP_SYSLOG"
> > +	echo "0014 x $(get_test_count 0014) - test handling of parallel loads, success case"
> > +	echo "0015 x $(get_test_count 0015) - test handling of parallel loads, init returning error"
> 
> Good stuff!
> 
> So test 0015 mimics the error reported by Prarit Bhargava through
> commit 6e6de3dee51a ("kernel/module.c: Only return -EEXIST for modules
> that have finished loading")? If so it would be good to document that
> here.
> 
> Also, this patch should go first, with the ALL_TESTS variable set to
> disable the tests which are known to fail, so to demonstrate the *new*
> issues as they are failing and then your fix first and then you enable
> that test afterwards.
> 
> If 6e6de3dee51a introduced another regression which is not covered
> by the tests we'd add it, and fix on top of it as a stepping stone
> to prove / send to stable.

On second though, this could also be made into a self-contained C code
test within lib/test_kmod.c:

  * You'd extend kmod_test_case with a new enum, say TEST_KMOD_INIT_RACES
  * You could extend run_request() to run the series of tests required.
  * request_module() does not allow you to specify module parameters,
    but we could have optional support added, and make the existing
    request_module() always pass that empty / NULL.

Kind of like lib/test_vmalloc.c which has an array of tests, you can peg
new tests to the TEST_KMOD_INIT_RACES for it.

Let me know what folks think.

  Luis
  

[A different fix that the one from this thread was selected but it is still
useful to discuss these test cases and if they should be added in some form.]

On 10/17/22 15:51, Petr Mladek wrote:
> On Sun 2022-10-16 14:30:31, Petr Pavlu wrote:
>> Add two tests to check that loading the same module multiple times in
>> parallel results only in one real attempt to initialize it.
>> Synchronization of the loads is done by waiting 1000 ms in the init
> 
> I do not have a good experience with this kind of synchronization.
> It usually is not reliable. The test might be very slow especially when
> false positives are solved by prolonging the delay.
> 
> Alternative solution would be to have two modules:
> 
> 1st module would provide a counter, for example:
> 
> int modB_load_cnt;
> module_param(modB_load_cnt, int, 0444);
> EXPORT_SYMBOL(modB_load_cnt);
> 
> EXPORT_SYMBOL() should allow to directly increment the counter
> from the 2nd module.
> 
> module_param() should make the value readable via
> /sys/module/modA/parameters/modB_load_cnt. It can be
> checked by kmod_sh.

I agree that it would be best to avoid any synchronization based on timeouts
in these tests.

My reading is that your idea should allow the tests to remove measuring how
long it took in total to process all module inserts. It was possible for me to
implement this change.

It unfortunately doesn't help with the 1 second timeout that the
kmod_test_0014 module (modB in your description) has in its init function. Its
purpose is to make sure that any parallel loads of the same module which were
started by kmod.sh manage to reach add_unformed_module(), sleep there and
therefore hit the updated logic.

One option how to avoid this timeout is to extend modA to register a kprobe on
finished_loading() and export via a parameter which loads started by kmod.sh
reached this point. This approach works ok on my system and a prototype is
pasted at the end of this mail. Two shortcomings are that it relies on
internal knowledge of the module loader code and function finished_loading()
might not be always available for probing as it could get inlined in some
configurations.

To summarize, I see the following options for these tests:
* Use a timeout to synchronize the loads.
* Use the outlined kprobe approach.
* Don't add these tests at all.

Any opinions what would be preferred? I'm leaning towards not adding these
tests as they look fragile to me.

Thanks,
Petr

---

diff --git a/tools/testing/selftests/kmod/.gitignore b/tools/testing/selftests/kmod/.gitignore
new file mode 100644
index 000000000000..ea3afcaccc79
--- /dev/null
+++ b/tools/testing/selftests/kmod/.gitignore
@@ -0,0 +1 @@
+init_module
diff --git a/tools/testing/selftests/kmod/Makefile b/tools/testing/selftests/kmod/Makefile
index 5b3e746a0bee..1c684ac700b5 100644
--- a/tools/testing/selftests/kmod/Makefile
+++ b/tools/testing/selftests/kmod/Makefile
@@ -1,12 +1,20 @@
 # SPDX-License-Identifier: GPL-2.0-only
 # Makefile for kmod loading selftests
 
-# No binaries, but make sure arg-less "make" doesn't trigger "run_tests"
-all:
-
 TEST_PROGS := kmod.sh
 
+# compile but not part of 'make run_tests'
+TEST_GEN_PROGS_EXTENDED := \
+	init_module test_kmod_parallel_count.ko test_kmod_parallel_sleep.ko
+
+# override lib.mk's default rule
+override define CLEAN
+	$(RM) $(TEST_GEN_PROGS_EXTENDED)
+	$(MAKE) -C test_kmod_parallel clean
+endef
+
 include ../lib.mk
 
-# Nothing to clean up.
-clean:
+$(OUTPUT)/test_kmod_parallel_count.ko $(OUTPUT)/test_kmod_parallel_sleep.ko: $(wildcard test_kmod_parallel/Makefile test_kmod_parallel/*.[ch])
+	$(MAKE) -C test_kmod_parallel
+	cp test_kmod_parallel/$(notdir $@) $@
diff --git a/tools/testing/selftests/kmod/config b/tools/testing/selftests/kmod/config
index 259f4fd6b5e2..704c4735686c 100644
--- a/tools/testing/selftests/kmod/config
+++ b/tools/testing/selftests/kmod/config
@@ -5,3 +5,7 @@ CONFIG_XFS_FS=m
 # For the module parameter force_init_test is used
 CONFIG_TUN=m
 CONFIG_BTRFS_FS=m
+
+# For the module test_kmod_parallel
+CONFIG_KALLSYMS=y
+CONFIG_KPROBES=y
diff --git a/tools/testing/selftests/kmod/init_module.c b/tools/testing/selftests/kmod/init_module.c
new file mode 100644
index 000000000000..529abd0a8357
--- /dev/null
+++ b/tools/testing/selftests/kmod/init_module.c
@@ -0,0 +1,49 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+/*
+ * Simple program to insert a module, similar to insmod but tailored
+ * specifically for needs of module loader tests.
+ */
+
+#define _GNU_SOURCE
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+#include <stdio.h>
+#include <string.h>
+#include <unistd.h>
+#include <sys/syscall.h>
+#include <errno.h>
+
+int main(int argc, char *argv[])
+{
+	const char *filename, *args;
+	int fd, ret;
+
+	if (argc < 2 || argc > 3) {
+		fprintf(stderr, "usage: %s filename [args]\n", argv[0]);
+		return EINVAL;
+	}
+
+	filename = argv[1];
+	args = argc > 2 ? argv[2] : "";
+
+	fd = open(filename, O_RDONLY);
+	if (fd == -1) {
+		ret = errno;
+		fprintf(stderr, "init_module: could not open file %s: %s\n",
+			filename, strerror(ret));
+		return ret;
+	}
+
+	ret = syscall(SYS_finit_module, fd, args, 0);
+	if (ret != 0) {
+		ret = errno;
+		fprintf(stderr, "init_module: could not load module %s: %s\n",
+			filename, strerror(ret));
+	}
+
+	close(fd);
+
+	return ret;
+}
diff --git a/tools/testing/selftests/kmod/kmod.sh b/tools/testing/selftests/kmod/kmod.sh
index afd42387e8b2..8ca82b16f165 100755
--- a/tools/testing/selftests/kmod/kmod.sh
+++ b/tools/testing/selftests/kmod/kmod.sh
@@ -65,6 +65,8 @@ ALL_TESTS="$ALL_TESTS 0010:1:1"
 ALL_TESTS="$ALL_TESTS 0011:1:1"
 ALL_TESTS="$ALL_TESTS 0012:1:1"
 ALL_TESTS="$ALL_TESTS 0013:1:1"
+ALL_TESTS="$ALL_TESTS 0014:1:1"
+ALL_TESTS="$ALL_TESTS 0015:1:1"
 
 # Kselftest framework requirement - SKIP code is 4.
 ksft_skip=4
@@ -171,6 +173,12 @@ errno_name_to_val()
 		echo -1;;
 	-ENOENT)
 		echo -2;;
+	-EBUSY)
+		echo -16;;
+	-EEXIST)
+		echo -17;;
+	-ENODEV)
+		echo -19;;
 	-EINVAL)
 		echo -22;;
 	-ERR_ANY)
@@ -190,6 +198,12 @@ errno_val_to_name()
 		echo -EPERM;;
 	-2)
 		echo -ENOENT;;
+	-16)
+		echo -EBUSY;;
+	-17)
+		echo -EEXIST;;
+	-19)
+		echo -ENODEV;;
 	-22)
 		echo -EINVAL;;
 	-123456)
@@ -504,6 +518,135 @@ kmod_test_0013()
 		"cat /sys/module/${DEFAULT_KMOD_DRIVER}/sections/.*text | head -n1"
 }
 
+# Check that loading the same module multiple times in parallel results only in
+# one real attempt to initialize it. The function utilizes two sample modules
+# test_kmod_parallel_count and test_kmod_parallel_sleep:
+# TODO Describe.
+#
+# Cases:
+# * Test 0014 checks a situation when the load is successful. It should result
+#   in one insert returning 0 and remaining inserts returning EEXIST.
+# * Test 0015 checks a situation when the load is failing because the module
+#   init function returns ENODEV. It should result in one insert returning this
+#   error code and remaining inserts returning EBUSY.
+kmod_check_parallel_loads()
+{
+	local test_name="$1"
+	local parallel_loads="$2"
+	local insmod_params="$3"
+	local success_exp="$4"
+	local ebusy_exp="$5"
+	local eexist_exp="$6"
+	local enodev_exp="$7"
+	local other_exp="$8"
+
+	local EBUSY=$(errno_name_to_val -EBUSY)
+	local EEXIST=$(errno_name_to_val -EEXIST)
+	local ENODEV=$(errno_name_to_val -ENODEV)
+	local init_cnt pids hits rc
+	local success_cnt=0 ebusy_cnt=0 eexist_cnt=0 enodev_cnt=0 other_cnt=0
+	local test_ok=true
+
+	# Load the counter module
+	./init_module test_kmod_parallel_count.ko
+
+	# Run the parallel loads.
+	for i in $(seq 0 $(($parallel_loads - 1))); do
+		(echo $BASHPID > /sys/module/test_kmod_parallel_count/parameters/pids_watched;
+		 exec ./init_module test_kmod_parallel_sleep.ko "$insmod_params") &
+		pids[$i]=$!
+	done
+
+	# Wait for all secondary loads to block in add_unformed_module().
+	while true; do
+		hits=( $(cat /sys/module/test_kmod_parallel_count/parameters/pids_hit) )
+		(( ${#hits[@]} >= $parallel_loads - 1 )) && break
+		sleep 0.1
+	done
+	echo > /sys/module/test_kmod_parallel_count/parameters/pids_watched
+
+	# Set the state to 1 which unblocks the init function.
+	echo 1 > /sys/module/test_kmod_parallel_count/parameters/state
+
+	# Collect all results.
+	for pid in ${pids[*]}; do
+		{ wait $pid; let rc=-$?; } || true
+
+		case $rc in
+		0)
+			let success_cnt=$success_cnt+1;;
+		$EBUSY)
+			let ebusy_cnt=$ebusy_cnt+1;;
+		$EEXIST)
+			let eexist_cnt=$eexist_cnt+1;;
+		$ENODEV)
+			let enodev_cnt=$enodev_cnt+1;;
+		*)
+			let other_cnt=$other_cnt+1;;
+		esac
+	done
+
+	if (( $success_cnt > 0 )); then
+		rmmod test_kmod_parallel_sleep
+	fi
+
+	init_cnt=$(cat /sys/module/test_kmod_parallel_count/parameters/init_count)
+	rmmod test_kmod_parallel_count
+
+	# Check the results.
+	if (( $init_cnt != 1 )); then
+		echo "$test_name: FAIL, number of init calls, test expects '1' - got '$init_cnt'" >&2
+		test_ok=false
+	else
+		echo "$test_name: OK! - number of init calls, test expects '1' - got '$init_cnt'" >&2
+	fi
+	if (( $success_cnt != $success_exp )); then
+		echo "$test_name: FAIL, number of successful loads, test expects '$success_exp' - got '$success_cnt'" >&2
+		test_ok=false
+	else
+		echo "$test_name: OK! - number of successful loads, test expects '$success_exp' - got '$success_cnt'" >&2
+	fi
+	if (( $ebusy_cnt != $ebusy_exp )); then
+		echo "$test_name: FAIL, number of loads returning EBUSY, test expects '$ebusy_exp' - got '$ebusy_cnt'" >&2
+		test_ok=false
+	else
+		echo "$test_name: OK! - number of loads returning EBUSY, test expects '$ebusy_exp' - got '$ebusy_cnt'" >&2
+	fi
+	if (( $eexist_cnt != $eexist_exp )); then
+		echo "$test_name: FAIL, number of loads returning EEXIST, test expects '$eexist_exp' - got '$eexist_cnt'" >&2
+		test_ok=false
+	else
+		echo "$test_name: OK! - number of loads returning EEXIST, test expects '$eexist_exp' - got '$eexist_cnt'" >&2
+	fi
+	if (( $enodev_cnt != $enodev_exp )); then
+		echo "$test_name: FAIL, number of loads returning ENODEV, test expects '$enodev_exp' - got '$enodev_cnt'" >&2
+		test_ok=false
+	else
+		echo "$test_name: OK! - number of loads returning ENODEV, test expects '$enodev_exp' - got '$enodev_cnt'" >&2
+	fi
+	if (( $other_cnt != $other_exp )); then
+		echo "$test_name: FAIL, number of loads returning other values, test expects '$other_exp' - got '$other_cnt'" >&2
+		test_ok=false
+	else
+		echo "$test_name: OK! - number of loads returning other values, test expects '$other_exp' - got '$other_cnt'" >&2
+	fi
+
+	[ $test_ok = true ] || exit 1
+}
+
+kmod_test_0014()
+{
+	kmod_check_parallel_loads \
+		"${FUNCNAME[0]}" 4 "sleep_msecs=1000 init_retval=0" 1 0 3 0 0
+}
+
+kmod_test_0015()
+{
+	local ENODEV=$(errno_name_to_val -ENODEV)
+	kmod_check_parallel_loads \
+		"${FUNCNAME[0]}" 4 "sleep_msecs=1000 init_retval=$ENODEV" 0 3 0 1 0
+}
+
 list_tests()
 {
 	echo "Test ID list:"
@@ -525,6 +668,8 @@ list_tests()
 	echo "0011 x $(get_test_count 0011) - test completely disabling module autoloading"
 	echo "0012 x $(get_test_count 0012) - test /proc/modules address visibility under CAP_SYSLOG"
 	echo "0013 x $(get_test_count 0013) - test /sys/module/*/sections/* visibility under CAP_SYSLOG"
+	echo "0014 x $(get_test_count 0014) - test handling of parallel loads, success case"
+	echo "0015 x $(get_test_count 0015) - test handling of parallel loads, init returning error"
 }
 
 usage()
diff --git a/tools/testing/selftests/kmod/settings b/tools/testing/selftests/kmod/settings
new file mode 100644
index 000000000000..ba4d85f74cd6
--- /dev/null
+++ b/tools/testing/selftests/kmod/settings
@@ -0,0 +1 @@
+timeout=90
diff --git a/tools/testing/selftests/kmod/test_kmod_parallel/Makefile b/tools/testing/selftests/kmod/test_kmod_parallel/Makefile
new file mode 100644
index 000000000000..8d2b24aa868f
--- /dev/null
+++ b/tools/testing/selftests/kmod/test_kmod_parallel/Makefile
@@ -0,0 +1,12 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+KMOD_TEST_PARALLEL_DIR := $(realpath $(dir $(abspath $(lastword $(MAKEFILE_LIST)))))
+KDIR ?= $(abspath $(KMOD_TEST_PARALLEL_DIR)/../../../../..)
+
+obj-m := test_kmod_parallel_count.o test_kmod_parallel_sleep.o
+
+all:
+	$(MAKE) -C $(KDIR) M=$(KMOD_TEST_PARALLEL_DIR) modules
+
+clean:
+	$(MAKE) -C $(KDIR) M=$(KMOD_TEST_PARALLEL_DIR) clean
diff --git a/tools/testing/selftests/kmod/test_kmod_parallel/test_kmod_parallel_count.c b/tools/testing/selftests/kmod/test_kmod_parallel/test_kmod_parallel_count.c
new file mode 100644
index 000000000000..8de9351d6df9
--- /dev/null
+++ b/tools/testing/selftests/kmod/test_kmod_parallel/test_kmod_parallel_count.c
@@ -0,0 +1,219 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+#include <linux/init.h>
+#include <linux/kernel.h>
+#include <linux/kprobes.h>
+#include <linux/kstrtox.h>
+#include <linux/module.h>
+#include <linux/moduleparam.h>
+#include <linux/string.h>
+#include <linux/wait.h>
+
+/*
+ * State parameter: value 0 blocks the init function of test_kmod_parallel_sleep
+ * when it invokes test_kmod_parallel_count_wait(), value 1 makes it to proceed.
+ */
+static unsigned int state;
+static DECLARE_WAIT_QUEUE_HEAD(state_wq);
+
+static int state_set(const char *val, const struct kernel_param *kp)
+{
+	int err;
+
+	err = kstrtoint(val, 0, &state);
+	if (err != 0)
+		return err;
+
+	wake_up_interruptible(&state_wq);
+	return 0;
+}
+
+static const struct kernel_param_ops state_ops = {
+	.set = state_set,
+	.get = param_get_uint,
+};
+
+module_param_cb(state, &state_ops, &state, 0644);
+MODULE_PARM_DESC(state, "module state switch, default 0");
+
+/*
+ * Counter to record how many times the init function of
+ * test_kmod_parallel_sleep was invoked.
+ */
+static unsigned int init_count;
+module_param(init_count, uint, 0444);
+MODULE_PARM_DESC(init_count, "counter for test_kmod_parallel_sleep_init()");
+
+int test_kmod_parallel_count_wait(void)
+{
+	kernel_param_lock(THIS_MODULE);
+	init_count++;
+	kernel_param_unlock(THIS_MODULE);
+
+	/* Wait for an external actor to transfer the state from 0 to 1. */
+	return wait_event_interruptible(state_wq, state == 1);
+}
+EXPORT_SYMBOL(test_kmod_parallel_count_wait);
+
+/*
+ * Tasks watched for invocation of finished_loading() and which ones actually
+ * executed the function.
+ */
+static DEFINE_SPINLOCK(pids_lock);
+static pid_t pids_watched[4] = { -1, -1, -1, -1 };
+static pid_t pids_hit[4] = { -1, -1, -1, -1 };
+
+static int format_pids(char *buffer, const pid_t *pids, size_t pids_size)
+{
+	int ret = 0;
+	size_t i;
+
+	for (i = 0; i < pids_size; i++) {
+		if (pids[i] == -1)
+			break;
+		ret += sprintf(buffer + ret, "%s%d", ret != 0 ? " " : "",
+			       pids[i]);
+	}
+
+	ret += sprintf(buffer + ret, "\n");
+	return ret;
+}
+
+static int pids_watched_set(const char *val, const struct kernel_param *kp)
+{
+	int err;
+	size_t i;
+	pid_t pid;
+
+	spin_lock(&pids_lock);
+
+	/* Handle the reset case. */
+	if (sysfs_streq(val, "")) {
+		for (i = 0; i < ARRAY_SIZE(pids_watched); i++) {
+			pids_watched[i] = -1;
+			pids_hit[i] = -1;
+		}
+		goto success;
+	}
+
+	/* Add one PID to the watch set. */
+	err = kstrtoint(val, 0, &pid);
+	if (err != 0)
+		goto err;
+	if (pid < 0) {
+		err = -EINVAL;
+		goto err;
+	}
+
+	for (i = 0; i < ARRAY_SIZE(pids_watched); i++)
+		if (pids_watched[i] == -1) {
+			pids_watched[i] = pid;
+			break;
+		}
+
+	if (i == ARRAY_SIZE(pids_watched)) {
+		err = -EINVAL;
+		goto err;
+	}
+
+success:
+	err = 0;
+err:
+	spin_unlock(&pids_lock);
+	return err;
+}
+
+static int pids_watched_get(char *buffer, const struct kernel_param *kp)
+{
+	int ret;
+
+	spin_lock(&pids_lock);
+	ret = format_pids(buffer, pids_watched, ARRAY_SIZE(pids_watched));
+	spin_unlock(&pids_lock);
+
+	return ret;
+}
+
+static const struct kernel_param_ops pids_watched_ops = {
+	.set = pids_watched_set,
+	.get = pids_watched_get,
+};
+
+module_param_cb(pids_watched, &pids_watched_ops, NULL, 0644);
+MODULE_PARM_DESC(state, "PIDs watched for finished_loading()");
+
+static int pids_hit_get(char *buffer, const struct kernel_param *kp)
+{
+	int ret;
+
+	spin_lock(&pids_lock);
+	ret = format_pids(buffer, pids_hit, ARRAY_SIZE(pids_hit));
+	spin_unlock(&pids_lock);
+
+	return ret;
+}
+
+static const struct kernel_param_ops pids_hit_ops = {
+	.get = pids_hit_get,
+};
+
+module_param_cb(pids_hit, &pids_hit_ops, NULL, 0444);
+MODULE_PARM_DESC(state, "PIDs which hit finished_loading()");
+
+static int __kprobes finished_loading_pre(struct kprobe *p,
+					  struct pt_regs *regs)
+{
+	size_t i;
+
+	spin_lock(&pids_lock);
+
+	/* Check if this task is watched. */
+	for (i = 0; i < ARRAY_SIZE(pids_watched); i++)
+		if (current->pid == pids_watched[i])
+			break;
+	if (i == ARRAY_SIZE(pids_watched))
+		goto out;
+
+	/* Record the hit. */
+	for (i = 0; i < ARRAY_SIZE(pids_hit); i++) {
+		if (current->pid == pids_hit[i])
+			break;
+		if (pids_hit[i] != -1)
+			continue;
+		pids_hit[i] = current->pid;
+		break;
+	}
+
+out:
+	spin_unlock(&pids_lock);
+	return 0;
+}
+
+static struct kprobe finished_loading_kp = {
+	.symbol_name = "finished_loading",
+	.pre_handler = finished_loading_pre,
+};
+
+static int __init test_kmod_parallel_count_init(void)
+{
+	int err;
+
+	err = register_kprobe(&finished_loading_kp);
+	if (err < 0) {
+		pr_err("register_kprobe failed, returned %d\n", err);
+		return err;
+	}
+	return 0;
+}
+
+static void __exit test_kmod_parallel_count_exit(void)
+{
+	unregister_kprobe(&finished_loading_kp);
+}
+
+module_init(test_kmod_parallel_count_init);
+module_exit(test_kmod_parallel_count_exit);
+
+MODULE_AUTHOR("Petr Pavlu <petr.pavlu@suse.com>");
+MODULE_LICENSE("GPL v2");
+MODULE_INFO(test, "Y");
diff --git a/tools/testing/selftests/kmod/test_kmod_parallel/test_kmod_parallel_sleep.c b/tools/testing/selftests/kmod/test_kmod_parallel/test_kmod_parallel_sleep.c
new file mode 100644
index 000000000000..cd1b8184af2a
--- /dev/null
+++ b/tools/testing/selftests/kmod/test_kmod_parallel/test_kmod_parallel_sleep.c
@@ -0,0 +1,33 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+#include <linux/init.h>
+#include <linux/module.h>
+#include <linux/moduleparam.h>
+
+int test_kmod_parallel_count_wait(void);
+
+static int init_retval;
+module_param(init_retval, int, 0644);
+MODULE_PARM_DESC(init_retval, "init return value, default 0");
+
+static int __init test_kmod_parallel_sleep_init(void)
+{
+	int err;
+
+	err = test_kmod_parallel_count_wait();
+	if (err != 0)
+		return err;
+
+	return init_retval;
+}
+
+static void __exit test_kmod_parallel_sleep_exit(void)
+{
+}
+
+module_init(test_kmod_parallel_sleep_init);
+module_exit(test_kmod_parallel_sleep_exit);
+
+MODULE_AUTHOR("Petr Pavlu <petr.pavlu@suse.com>");
+MODULE_LICENSE("GPL v2");
+MODULE_INFO(test, "Y");
  

On Thu 2023-01-12 10:03:10, Petr Pavlu wrote:
> [A different fix that the one from this thread was selected but it is still
> useful to discuss these test cases and if they should be added in some form.]
> 
> On 10/17/22 15:51, Petr Mladek wrote:
> > On Sun 2022-10-16 14:30:31, Petr Pavlu wrote:
> >> Add two tests to check that loading the same module multiple times in
> >> parallel results only in one real attempt to initialize it.
> >> Synchronization of the loads is done by waiting 1000 ms in the init
> > 
> > I do not have a good experience with this kind of synchronization.
> > It usually is not reliable. The test might be very slow especially when
> > false positives are solved by prolonging the delay.
> > 
> > Alternative solution would be to have two modules:
> > 
> > 1st module would provide a counter, for example:
> > 
> > int modB_load_cnt;
> > module_param(modB_load_cnt, int, 0444);
> > EXPORT_SYMBOL(modB_load_cnt);
> > 
> > EXPORT_SYMBOL() should allow to directly increment the counter
> > from the 2nd module.
> > 
> > module_param() should make the value readable via
> > /sys/module/modA/parameters/modB_load_cnt. It can be
> > checked by kmod_sh.
> 
> I agree that it would be best to avoid any synchronization based on timeouts
> in these tests.
> 
> My reading is that your idea should allow the tests to remove measuring how
> long it took in total to process all module inserts. It was possible for me to
> implement this change.
> 
> It unfortunately doesn't help with the 1 second timeout that the
> kmod_test_0014 module (modB in your description) has in its init function. Its
> purpose is to make sure that any parallel loads of the same module which were
> started by kmod.sh manage to reach add_unformed_module(), sleep there and
> therefore hit the updated logic.

I see.

> One option how to avoid this timeout is to extend modA to register a kprobe on
> finished_loading() and export via a parameter which loads started by kmod.sh
> reached this point. This approach works ok on my system and a prototype is
> pasted at the end of this mail. Two shortcomings are that it relies on
> internal knowledge of the module loader code and function finished_loading()
> might not be always available for probing as it could get inlined in some
> configurations.

Yeah, it is a bit fragile as well.

> To summarize, I see the following options for these tests:
> * Use a timeout to synchronize the loads.
> * Use the outlined kprobe approach.
> * Don't add these tests at all.

Yet another solution would be to add a support for this test into
the module loaded code. I mean that add_unformed_module() might
increment a global counter when a certain module is waiting there.
The global counter then might be used to unblock the init()
callback.

The test module might be distinguished, for example, by a test
specific module info string. For example, see
check_modinfo_livepatch(). It looks for the info string defined
in the livepatch modules, see MODULE_INFO(livepatch, "Y"); in
samples/livepatch/livepatch-sample.c.

That said, I do not like this much either. I am not sure if it is
more or less crazy than the kprobe approach.


> Any opinions what would be preferred? I'm leaning towards not adding these
> tests as they look fragile to me.

I do not have strong opinion.

My experience is that some tests are not worth the effort. The
maintenance or false positives might add more harm than good.

My feeling is that this one belongs into this category.
We could keep the timeout and make it error prone.
Or we could use some hacks and make it hard to maintain.

Best Regards,
Petr