diff mbox series

[1/2] kunit: tool: make unit test not print parsed testdata to stdout

Message ID 20221028210256.3776835-1-dlatypov@google.com
State New
Headers
Series [1/2] kunit: tool: make unit test not print parsed testdata to stdout |

Commit Message

Daniel Latypov Oct. 28, 2022, 9:02 p.m. UTC 
  Currently, if you run
$ ./tools/testing/kunit/kunit_tool_test.py
you'll see a lot of output from the parser as we feed it testdata.

This makes the output hard to read and fairly confusing, esp. since our
testdata includes example failures, which get printed out in red.

Silence that output so real failures are easier to see.

Signed-off-by: Daniel Latypov <dlatypov@google.com>
---
 tools/testing/kunit/kunit_tool_test.py | 6 ++++++
 1 file changed, 6 insertions(+)


base-commit: 8f8b51f7d5c8bd3a89e7ea87aed2cdaa52ca5ba4

Comments

David Gow Oct. 30, 2022, 3:31 a.m. UTC | #1 
On Sat, Oct 29, 2022 at 5:03 AM Daniel Latypov <dlatypov@google.com> wrote:
>
> Currently, if you run
> $ ./tools/testing/kunit/kunit_tool_test.py
> you'll see a lot of output from the parser as we feed it testdata.
>
> This makes the output hard to read and fairly confusing, esp. since our
> testdata includes example failures, which get printed out in red.
>
> Silence that output so real failures are easier to see.
>
> Signed-off-by: Daniel Latypov <dlatypov@google.com>
> ---

Thanks -- this has been annoying me for ages.

That being said, this isn't a perfect fix, the "usage" text and
"Reconfiguring .config"  still show up for me:
---
davidgow@slicestar:~/Development/linux-kselftest$
./tools/testing/kunit/kunit_tool_test.py
..............................usage: kunit_tool_test.py run [-h]
[--build_dir DIR] [--make_options X=Y] [--alltests] [--kunitconfig
PATHS] [--kconfig_add CONFIG_X=Y] [--arch ARCH] [--cross_compile
PREFIX] [--qemu_config FILE] [--qemu_ar
gs] [--jobs N]
                             [--timeout SECONDS] [--kernel_args]
[--run_isolated {suite,test}] [--raw_output [{all,kunit}]] [--json
[FILE]]
                             [filter_glob]
kunit_tool_test.py run: error: argument --raw_output: invalid choice:
'invalid' (choose from 'all', 'kunit')
..............................Generating .config ...
.Regenerating .config ...
.........
----------------------------------------------------------------------
Ran 70 tests in 0.232s

OK
---

That's still a significant improvement on what we had before, though, so:

Reviewed-by: David Gow <davidgow@google.com>

Cheers,
-- David

>  tools/testing/kunit/kunit_tool_test.py | 6 ++++++
>  1 file changed, 6 insertions(+)
>
> diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py
> index e2cd2cc2e98f..a6e53945656e 100755
> --- a/tools/testing/kunit/kunit_tool_test.py
> +++ b/tools/testing/kunit/kunit_tool_test.py
> @@ -80,6 +80,9 @@ class KconfigTest(unittest.TestCase):
>                 self.assertEqual(actual_kconfig, expected_kconfig)
>
>  class KUnitParserTest(unittest.TestCase):
> +       def setUp(self):
> +               self.print_mock = mock.patch('kunit_printer.Printer.print').start()
> +               self.addCleanup(mock.patch.stopall)
>
>         def assertContains(self, needle: str, haystack: kunit_parser.LineStream):
>                 # Clone the iterator so we can print the contents on failure.
> @@ -485,6 +488,9 @@ class LinuxSourceTreeTest(unittest.TestCase):
>
>
>  class KUnitJsonTest(unittest.TestCase):
> +       def setUp(self):
> +               self.print_mock = mock.patch('kunit_printer.Printer.print').start()
> +               self.addCleanup(mock.patch.stopall)
>
>         def _json_for(self, log_file):
>                 with open(test_data_path(log_file)) as file:
>
> base-commit: 8f8b51f7d5c8bd3a89e7ea87aed2cdaa52ca5ba4
> --
> 2.38.1.273.g43a17bfeac-goog
>
Daniel Latypov Oct. 31, 2022, 4:37 p.m. UTC | #2 
On Sat, Oct 29, 2022 at 8:31 PM David Gow <davidgow@google.com> wrote:
>
> On Sat, Oct 29, 2022 at 5:03 AM Daniel Latypov <dlatypov@google.com> wrote:
> >
> > Currently, if you run
> > $ ./tools/testing/kunit/kunit_tool_test.py
> > you'll see a lot of output from the parser as we feed it testdata.
> >
> > This makes the output hard to read and fairly confusing, esp. since our
> > testdata includes example failures, which get printed out in red.
> >
> > Silence that output so real failures are easier to see.
> >
> > Signed-off-by: Daniel Latypov <dlatypov@google.com>
> > ---
>
> Thanks -- this has been annoying me for ages.
>
> That being said, this isn't a perfect fix, the "usage" text and
> "Reconfiguring .config"  still show up for me:
> ---
> davidgow@slicestar:~/Development/linux-kselftest$
> ./tools/testing/kunit/kunit_tool_test.py
> ..............................usage: kunit_tool_test.py run [-h]
> [--build_dir DIR] [--make_options X=Y] [--alltests] [--kunitconfig
> PATHS] [--kconfig_add CONFIG_X=Y] [--arch ARCH] [--cross_compile
> PREFIX] [--qemu_config FILE] [--qemu_ar
> gs] [--jobs N]
>                              [--timeout SECONDS] [--kernel_args]
> [--run_isolated {suite,test}] [--raw_output [{all,kunit}]] [--json
> [FILE]]
>                              [filter_glob]
> kunit_tool_test.py run: error: argument --raw_output: invalid choice:
> 'invalid' (choose from 'all', 'kunit')
> ..............................Generating .config ...
> .Regenerating .config ...
> .........
> ----------------------------------------------------------------------
> Ran 70 tests in 0.232s
>
> OK
> ---
>
> That's still a significant improvement on what we had before, though, so:

Yeah, I was originally going to make this patch silence all the output.
But I figured I would focus this patch on just the code using our
kunit_printer abstraction, aka the parser.

We could address the other ones, but we'd have to mock out the
built-in print function.
As someone who is guilty of print(f) debugging, doing that means I no
longer have an easy of way of adding debugging code to those test
cases :P
diff mbox series

Patch

diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py
index e2cd2cc2e98f..a6e53945656e 100755
--- a/tools/testing/kunit/kunit_tool_test.py
+++ b/tools/testing/kunit/kunit_tool_test.py
@@ -80,6 +80,9 @@  class KconfigTest(unittest.TestCase):
 		self.assertEqual(actual_kconfig, expected_kconfig)
 
 class KUnitParserTest(unittest.TestCase):
+	def setUp(self):
+		self.print_mock = mock.patch('kunit_printer.Printer.print').start()
+		self.addCleanup(mock.patch.stopall)
 
 	def assertContains(self, needle: str, haystack: kunit_parser.LineStream):
 		# Clone the iterator so we can print the contents on failure.
@@ -485,6 +488,9 @@  class LinuxSourceTreeTest(unittest.TestCase):
 
 
 class KUnitJsonTest(unittest.TestCase):
+	def setUp(self):
+		self.print_mock = mock.patch('kunit_printer.Printer.print').start()
+		self.addCleanup(mock.patch.stopall)
 
 	def _json_for(self, log_file):
 		with open(test_data_path(log_file)) as file: