selftests: allow runners to override the timeout
Commit Message
The default timeout for selftests tests is 45 seconds. Although
we already have 13 settings for tests of about 96 sefltests which
use a timeout greater than this, we want to try to avoid encouraging
more tests to forcing a higher test timeout as selftests strives to
run all tests quickly. Selftests also uses the timeout as a non-fatal
error. Only tests runners which have control over a system would know
if to treat a timeout as fatal or not.
To help with all this:
o Enhance documentation to avoid future increases of insane timeouts
o Add the option to allow overriding the default timeout with test
runners with a command line option
Suggested-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
Documentation/dev-tools/kselftest.rst | 22 +++++++++++++++++++++
tools/testing/selftests/kselftest/runner.sh | 11 ++++++++++-
tools/testing/selftests/run_kselftest.sh | 5 +++++
3 files changed, 37 insertions(+), 1 deletion(-)
Comments
On 4/15/23 12:38 AM, Luis Chamberlain wrote:
> The default timeout for selftests tests is 45 seconds. Although
> we already have 13 settings for tests of about 96 sefltests which
> use a timeout greater than this, we want to try to avoid encouraging
> more tests to forcing a higher test timeout as selftests strives to
> run all tests quickly. Selftests also uses the timeout as a non-fatal
> error. Only tests runners which have control over a system would know
> if to treat a timeout as fatal or not.
>
> To help with all this:
>
> o Enhance documentation to avoid future increases of insane timeouts
> o Add the option to allow overriding the default timeout with test
> runners with a command line option
>
> Suggested-by: Shuah Khan <skhan@linuxfoundation.org>
> Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
Reviewed-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
Tested-by:Muhammad Usama Anjum <usama.anjum@collabora.com>
> ---
> Documentation/dev-tools/kselftest.rst | 22 +++++++++++++++++++++
> tools/testing/selftests/kselftest/runner.sh | 11 ++++++++++-
> tools/testing/selftests/run_kselftest.sh | 5 +++++
> 3 files changed, 37 insertions(+), 1 deletion(-)
>
> diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst
> index 12b575b76b20..dd214af7b7ff 100644
> --- a/Documentation/dev-tools/kselftest.rst
> +++ b/Documentation/dev-tools/kselftest.rst
> @@ -168,6 +168,28 @@ the `-t` option for specific single tests. Either can be used multiple times::
>
> For other features see the script usage output, seen with the `-h` option.
>
> +Timeout for selftests
> +=====================
> +
> +Selftests are designed to be quick and so a default timeout is used of 45
> +seconds for each test. Tests can override the default timeout by adding
> +a settings file in their directory and set a timeout variable there to the
> +configured a desired upper timeout for the test. Only a few tests override
> +the timeout with a value higher than 45 seconds, selftests strives to keep
> +it that way. Timeouts in selftests are not considered fatal because the
> +system under which a test runs may change and this can also modify the
> +expected time it takes to run a test. If you have control over the systems
> +which will run the tests you can configure a test runner on those systems to
> +use a greater or lower timeout on the command line as with the `-o` or
> +the `--override-timeout` argument. For example to use 165 seconds instead
> +one would use:
> +
> + $ ./run_kselftest.sh --override-timeout 165
> +
> +You can look at the TAP output to see if you ran into the timeout. Test
> +runners which know a test must run under a specific time can then optionally
> +treat these timeouts then as fatal.
> +
> Packaging selftests
> ===================
>
> diff --git a/tools/testing/selftests/kselftest/runner.sh b/tools/testing/selftests/kselftest/runner.sh
> index 294619ade49f..1c952d1401d4 100644
> --- a/tools/testing/selftests/kselftest/runner.sh
> +++ b/tools/testing/selftests/kselftest/runner.sh
> @@ -8,7 +8,8 @@ export logfile=/dev/stdout
> export per_test_logging=
>
> # Defaults for "settings" file fields:
> -# "timeout" how many seconds to let each test run before failing.
> +# "timeout" how many seconds to let each test run before running
> +# over our soft timeout limit.
> export kselftest_default_timeout=45
>
> # There isn't a shell-agnostic way to find the path of a sourced file,
> @@ -90,6 +91,14 @@ run_one()
> done < "$settings"
> fi
>
> + # Command line timeout overrides the settings file
> + if [ -n "$kselftest_override_timeout" ]; then
> + kselftest_timeout="$kselftest_override_timeout"
> + echo "# overriding timeout to $kselftest_timeout" >> "$logfile"
> + else
> + echo "# timeout set to $kselftest_timeout" >> "$logfile"
> + fi
> +
> TEST_HDR_MSG="selftests: $DIR: $BASENAME_TEST"
> echo "# $TEST_HDR_MSG"
> if [ ! -e "$TEST" ]; then
> diff --git a/tools/testing/selftests/run_kselftest.sh b/tools/testing/selftests/run_kselftest.sh
> index 97165a83df63..9a981b36bd7f 100755
> --- a/tools/testing/selftests/run_kselftest.sh
> +++ b/tools/testing/selftests/run_kselftest.sh
> @@ -26,6 +26,7 @@ Usage: $0 [OPTIONS]
> -l | --list List the available collection:test entries
> -d | --dry-run Don't actually run any tests
> -h | --help Show this usage info
> + -o | --override-timeout Number of seconds after which we timeout
> EOF
> exit $1
> }
> @@ -33,6 +34,7 @@ EOF
> COLLECTIONS=""
> TESTS=""
> dryrun=""
> +kselftest_override_timeout=""
> while true; do
> case "$1" in
> -s | --summary)
> @@ -51,6 +53,9 @@ while true; do
> -d | --dry-run)
> dryrun="echo"
> shift ;;
> + -o | --override-timeout)
> + kselftest_override_timeout="$2"
> + shift 2 ;;
> -h | --help)
> usage 0 ;;
> "")
On Fri, Apr 28, 2023 at 1:34 AM Muhammad Usama Anjum
<usama.anjum@collabora.com> wrote:
>
> On 4/15/23 12:38 AM, Luis Chamberlain wrote:
> > The default timeout for selftests tests is 45 seconds. Although
> > we already have 13 settings for tests of about 96 sefltests which
> > use a timeout greater than this, we want to try to avoid encouraging
> > more tests to forcing a higher test timeout as selftests strives to
> > run all tests quickly. Selftests also uses the timeout as a non-fatal
> > error. Only tests runners which have control over a system would know
> > if to treat a timeout as fatal or not.
> >
> > To help with all this:
> >
> > o Enhance documentation to avoid future increases of insane timeouts
> > o Add the option to allow overriding the default timeout with test
> > runners with a command line option
> >
> > Suggested-by: Shuah Khan <skhan@linuxfoundation.org>
> > Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
> Reviewed-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
> Tested-by:Muhammad Usama Anjum <usama.anjum@collabora.com>
Shuah, just a friendly poke! This is needed to allow me to enable full
automation for kdevops for selftests.
Luis
On Thu, May 11, 2023 at 08:26:42AM -0700, Luis Chamberlain wrote:
> On Fri, Apr 28, 2023 at 1:34 AM Muhammad Usama Anjum
> <usama.anjum@collabora.com> wrote:
> >
> > On 4/15/23 12:38 AM, Luis Chamberlain wrote:
> > > The default timeout for selftests tests is 45 seconds. Although
> > > we already have 13 settings for tests of about 96 sefltests which
> > > use a timeout greater than this, we want to try to avoid encouraging
> > > more tests to forcing a higher test timeout as selftests strives to
> > > run all tests quickly. Selftests also uses the timeout as a non-fatal
> > > error. Only tests runners which have control over a system would know
> > > if to treat a timeout as fatal or not.
> > >
> > > To help with all this:
> > >
> > > o Enhance documentation to avoid future increases of insane timeouts
> > > o Add the option to allow overriding the default timeout with test
> > > runners with a command line option
> > >
> > > Suggested-by: Shuah Khan <skhan@linuxfoundation.org>
> > > Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
> > Reviewed-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
> > Tested-by:Muhammad Usama Anjum <usama.anjum@collabora.com>
>
> Shuah, just a friendly poke! This is needed to allow me to enable full
> automation for kdevops for selftests.
Shuah, friendly re-poke.
Luis
On 5/24/23 16:58, Luis Chamberlain wrote:
> On Thu, May 11, 2023 at 08:26:42AM -0700, Luis Chamberlain wrote:
>> On Fri, Apr 28, 2023 at 1:34 AM Muhammad Usama Anjum
>> <usama.anjum@collabora.com> wrote:
>>>
>>> On 4/15/23 12:38 AM, Luis Chamberlain wrote:
>>>> The default timeout for selftests tests is 45 seconds. Although
>>>> we already have 13 settings for tests of about 96 sefltests which
>>>> use a timeout greater than this, we want to try to avoid encouraging
>>>> more tests to forcing a higher test timeout as selftests strives to
>>>> run all tests quickly. Selftests also uses the timeout as a non-fatal
>>>> error. Only tests runners which have control over a system would know
>>>> if to treat a timeout as fatal or not.
>>>>
>>>> To help with all this:
>>>>
>>>> o Enhance documentation to avoid future increases of insane timeouts
>>>> o Add the option to allow overriding the default timeout with test
>>>> runners with a command line option
>>>>
>>>> Suggested-by: Shuah Khan <skhan@linuxfoundation.org>
>>>> Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
>>> Reviewed-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
>>> Tested-by:Muhammad Usama Anjum <usama.anjum@collabora.com>
>>
>> Shuah, just a friendly poke! This is needed to allow me to enable full
>> automation for kdevops for selftests.
>
> Shuah, friendly re-poke.
>
> Luis
Thanks. I will apply this for next.
thanks,
-- Shuah
@@ -168,6 +168,28 @@ the `-t` option for specific single tests. Either can be used multiple times::
For other features see the script usage output, seen with the `-h` option.
+Timeout for selftests
+=====================
+
+Selftests are designed to be quick and so a default timeout is used of 45
+seconds for each test. Tests can override the default timeout by adding
+a settings file in their directory and set a timeout variable there to the
+configured a desired upper timeout for the test. Only a few tests override
+the timeout with a value higher than 45 seconds, selftests strives to keep
+it that way. Timeouts in selftests are not considered fatal because the
+system under which a test runs may change and this can also modify the
+expected time it takes to run a test. If you have control over the systems
+which will run the tests you can configure a test runner on those systems to
+use a greater or lower timeout on the command line as with the `-o` or
+the `--override-timeout` argument. For example to use 165 seconds instead
+one would use:
+
+ $ ./run_kselftest.sh --override-timeout 165
+
+You can look at the TAP output to see if you ran into the timeout. Test
+runners which know a test must run under a specific time can then optionally
+treat these timeouts then as fatal.
+
Packaging selftests
===================
@@ -8,7 +8,8 @@ export logfile=/dev/stdout
export per_test_logging=
# Defaults for "settings" file fields:
-# "timeout" how many seconds to let each test run before failing.
+# "timeout" how many seconds to let each test run before running
+# over our soft timeout limit.
export kselftest_default_timeout=45
# There isn't a shell-agnostic way to find the path of a sourced file,
@@ -90,6 +91,14 @@ run_one()
done < "$settings"
fi
+ # Command line timeout overrides the settings file
+ if [ -n "$kselftest_override_timeout" ]; then
+ kselftest_timeout="$kselftest_override_timeout"
+ echo "# overriding timeout to $kselftest_timeout" >> "$logfile"
+ else
+ echo "# timeout set to $kselftest_timeout" >> "$logfile"
+ fi
+
TEST_HDR_MSG="selftests: $DIR: $BASENAME_TEST"
echo "# $TEST_HDR_MSG"
if [ ! -e "$TEST" ]; then
@@ -26,6 +26,7 @@ Usage: $0 [OPTIONS]
-l | --list List the available collection:test entries
-d | --dry-run Don't actually run any tests
-h | --help Show this usage info
+ -o | --override-timeout Number of seconds after which we timeout
EOF
exit $1
}
@@ -33,6 +34,7 @@ EOF
COLLECTIONS=""
TESTS=""
dryrun=""
+kselftest_override_timeout=""
while true; do
case "$1" in
-s | --summary)
@@ -51,6 +53,9 @@ while true; do
-d | --dry-run)
dryrun="echo"
shift ;;
+ -o | --override-timeout)
+ kselftest_override_timeout="$2"
+ shift 2 ;;
-h | --help)
usage 0 ;;
"")