From patchwork Wed Nov 2 16:40:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Latypov X-Patchwork-Id: 14393 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp18953wru; Wed, 2 Nov 2022 09:46:25 -0700 (PDT) X-Google-Smtp-Source: AMsMyM7cQQIaMyOK0kPUdMq/Jq29opQ4QChWB3Khgz7fElhDHG08lSCgnZNFewKZ4rczk+zltD6B X-Received: by 2002:a17:906:1f48:b0:7ae:77d:bac with SMTP id d8-20020a1709061f4800b007ae077d0bacmr3306951ejk.708.1667407585555; Wed, 02 Nov 2022 09:46:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667407585; cv=none; d=google.com; s=arc-20160816; b=Cnd/DaZCOOqFIquDBgWtCFS5sZKjFWVNCiB9HXVmPcb6Md3ibSJDbGLvRkHFUWjTfq Hj7BwbMwE3dJkaJicIUL8T2VwXI6IfDsGe9r0vSkKILjG6hcFoAfamTMhutogvlljSZP vTdR3fUgzanvpWvQieftBCk9NTD2zIncAkBN9vWtxy0Szt4bIbrxKCABXESsv/eGyXps TyGnSTqOOo1HaeYFBMLiVUYIpd2hYn/p2myeH5ZGQBsnbX83d4tkShevq7t6Qxr92DLw tl02envBqC/lNGX629jgHBZxZAr2JS9d3824PCfgBUZia6SRbYqSBRy/67Gy2ykGb/IG 32Ng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=KIew2SL18zFJ6u7nCVq8c3JgfzFW65AAtUvUlav4Acg=; b=OFfudOMYgmEXZHjikycX8wPn3DfOk3VQ3UK8m3r3LdyeuP/BJaopbIoXDfzu0gGsKw xFt0MbiYQIQz/4DGbsHuMLumLzo4pj8jLhNEEuDDRCN/p8PdtKuU/5XoY07H0qnS9EKg ZOrLRj4qto8psuDtrryiZIqAQcmJpNgDTU2J5fq5m+Ws3XXsKE1HLC2WHrKF/u7wVQvD +D3nUl3n0Rm47HuqtvHh/rldUhlnZX+yOr3CLzFZMvS1SP4HR03Jk/NTeuXUAvLNtq9Y UzNc3zqlDgOlYbMrkFL2wxypiD2Ncp7mbgmuGtcscMcDUWAJeP0258j3yCP6tWux3NTC fbcw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=airvtcUY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id go44-20020a1709070dac00b007313312730esi19178790ejc.85.2022.11.02.09.45.49; Wed, 02 Nov 2022 09:46:25 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=airvtcUY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232081AbiKBQpH (ORCPT + 99 others); Wed, 2 Nov 2022 12:45:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48404 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232069AbiKBQoj (ORCPT ); Wed, 2 Nov 2022 12:44:39 -0400 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9CEE7111D for ; Wed, 2 Nov 2022 09:40:43 -0700 (PDT) Received: by mail-pj1-x1049.google.com with SMTP id g6-20020a17090a300600b00212f609f6aeso1484265pjb.9 for ; Wed, 02 Nov 2022 09:40:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=KIew2SL18zFJ6u7nCVq8c3JgfzFW65AAtUvUlav4Acg=; b=airvtcUYcZZ8MHdLIhi3pbgBZi6gh+e7/dZtX9JFrnv3hRtBujUKQyyGPN5qk0Y8nt phFxU+70nNgIiGjKiWMhXn7b5yoptQzXEk/ouwpXGh75dSrNYJZOiKAWOP5BemNzSyOb Tdm7veavPLEGXo+SKLYqgOFO2HTT3xiaufEW51KFiU/3PveV9Ab3/jCb/8ZXNnG8ZvPO fqZoPNkoOLVgF7ooqKPcI7pY/L+91nIxxuC+u6uuAIZT3uXaxqRhc1/TtxEUpufDFc7D SdR6pl37Hb2iD4VJQ/Xc/ZkU02fVDASFIoCgeLGeG0hmO67+MmID9x2/2hIDUMztta0u 2K6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=KIew2SL18zFJ6u7nCVq8c3JgfzFW65AAtUvUlav4Acg=; b=ziuKwIaL199Bsw/fEzIYr3ntwh872iENILEdloFiUuLtw/T61NH5THygxJAvrNq8SN 6+A9EzxA/js6w8pxaDqv/i+Q5f2bWOoB2HkLX+3D6K7Osp/awJfjcig+ukWQAqJzc5tc pjlq/EH9+l3NRO7fjoZEggR9zr1BsHTFKEnsBk/DszI2fbKfKrnN6XJ1KsGh26feCRzs GV4atOnZsbk1gjXMAklM+C1HfrJ4KxSBuIOjXkw+W7tSLZCQRKVwU28ZypH4qcqlx5Gm X1VvxsVWX7ZBqvGm2bmRcv8orwV6+ny7puh0JoE+QyrES5whuKtwmp4z3lwIzYn8UBwj x7SQ== X-Gm-Message-State: ACrzQf0+JNFbZ1IxubQYFncMd3Iqj86DtfEaGbXfN3LxFmKEaH70JMMJ t7wNM32EdFVn2E0PzplV0L3yVpC682tTlA== X-Received: from dlatypov-spec.c.googlers.com ([fda3:e722:ac3:cc00:24:72f4:c0a8:3f35]) (user=dlatypov job=sendgmr) by 2002:a17:90b:1bc1:b0:213:e2af:b1f4 with SMTP id oa1-20020a17090b1bc100b00213e2afb1f4mr17299469pjb.47.1667407243096; Wed, 02 Nov 2022 09:40:43 -0700 (PDT) Date: Wed, 2 Nov 2022 09:40:04 -0700 In-Reply-To: <20221102164005.2516646-1-dlatypov@google.com> Mime-Version: 1.0 References: <20221102164005.2516646-1-dlatypov@google.com> X-Mailer: git-send-email 2.38.1.273.g43a17bfeac-goog Message-ID: <20221102164005.2516646-2-dlatypov@google.com> Subject: [PATCH 2/3] kunit: tool: unit tests all check parser errors, standardize formatting a bit From: Daniel Latypov To: brendanhiggins@google.com, davidgow@google.com Cc: rmoar@google.com, linux-kernel@vger.kernel.org, kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org, skhan@linuxfoundation.org, Daniel Latypov X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748403576607148391?= X-GMAIL-MSGID: =?utf-8?q?1748403576607148391?= Let's verify that the parser isn't reporting any errors for valid inputs. This change also * does result.status checking on one line * makes sure we consistently do it outside of the `with` block Signed-off-by: Daniel Latypov Reviewed-by: David Gow --- tools/testing/kunit/kunit_tool_test.py | 93 +++++++++++--------------- 1 file changed, 38 insertions(+), 55 deletions(-) diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index 9fa4babb2506..0063773c0fc4 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -133,33 +133,29 @@ class KUnitParserTest(unittest.TestCase): all_passed_log = test_data_path('test_is_test_passed-all_passed.log') with open(all_passed_log) as file: result = kunit_parser.parse_run_tests(file.readlines()) - self.assertEqual( - kunit_parser.TestStatus.SUCCESS, - result.status) + self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status) + self.assertEqual(result.counts.errors, 0) def test_parse_successful_nested_tests_log(self): all_passed_log = test_data_path('test_is_test_passed-all_passed_nested.log') with open(all_passed_log) as file: result = kunit_parser.parse_run_tests(file.readlines()) - self.assertEqual( - kunit_parser.TestStatus.SUCCESS, - result.status) + self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status) + self.assertEqual(result.counts.errors, 0) def test_kselftest_nested(self): kselftest_log = test_data_path('test_is_test_passed-kselftest.log') with open(kselftest_log) as file: result = kunit_parser.parse_run_tests(file.readlines()) - self.assertEqual( - kunit_parser.TestStatus.SUCCESS, - result.status) + self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status) + self.assertEqual(result.counts.errors, 0) def test_parse_failed_test_log(self): failed_log = test_data_path('test_is_test_passed-failure.log') with open(failed_log) as file: result = kunit_parser.parse_run_tests(file.readlines()) - self.assertEqual( - kunit_parser.TestStatus.FAILURE, - result.status) + self.assertEqual(kunit_parser.TestStatus.FAILURE, result.status) + self.assertEqual(result.counts.errors, 0) def test_no_header(self): empty_log = test_data_path('test_is_test_passed-no_tests_run_no_header.log') @@ -167,9 +163,8 @@ class KUnitParserTest(unittest.TestCase): result = kunit_parser.parse_run_tests( kunit_parser.extract_tap_lines(file.readlines())) self.assertEqual(0, len(result.subtests)) - self.assertEqual( - kunit_parser.TestStatus.FAILURE_TO_PARSE_TESTS, - result.status) + self.assertEqual(kunit_parser.TestStatus.FAILURE_TO_PARSE_TESTS, result.status) + self.assertEqual(result.counts.errors, 1) def test_missing_test_plan(self): missing_plan_log = test_data_path('test_is_test_passed-' @@ -180,9 +175,7 @@ class KUnitParserTest(unittest.TestCase): file.readlines())) # A missing test plan is not an error. self.assertEqual(result.counts, kunit_parser.TestCounts(passed=10, errors=0)) - self.assertEqual( - kunit_parser.TestStatus.SUCCESS, - result.status) + self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status) def test_no_tests(self): header_log = test_data_path('test_is_test_passed-no_tests_run_with_header.log') @@ -190,9 +183,8 @@ class KUnitParserTest(unittest.TestCase): result = kunit_parser.parse_run_tests( kunit_parser.extract_tap_lines(file.readlines())) self.assertEqual(0, len(result.subtests)) - self.assertEqual( - kunit_parser.TestStatus.NO_TESTS, - result.status) + self.assertEqual(kunit_parser.TestStatus.NO_TESTS, result.status) + self.assertEqual(result.counts.errors, 1) def test_no_tests_no_plan(self): no_plan_log = test_data_path('test_is_test_passed-no_tests_no_plan.log') @@ -203,7 +195,7 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.NO_TESTS, result.subtests[0].subtests[0].status) - self.assertEqual(1, result.counts.errors) + self.assertEqual(result.counts, kunit_parser.TestCounts(passed=1, errors=1)) def test_no_kunit_output(self): @@ -215,6 +207,7 @@ class KUnitParserTest(unittest.TestCase): print_mock.assert_any_call(StrContains('could not find any KTAP output!')) print_mock.stop() self.assertEqual(0, len(result.subtests)) + self.assertEqual(result.counts.errors, 1) def test_skipped_test(self): skipped_log = test_data_path('test_skip_tests.log') @@ -222,18 +215,16 @@ class KUnitParserTest(unittest.TestCase): result = kunit_parser.parse_run_tests(file.readlines()) # A skipped test does not fail the whole suite. - self.assertEqual( - kunit_parser.TestStatus.SUCCESS, - result.status) + self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status) + self.assertEqual(result.counts, kunit_parser.TestCounts(passed=4, skipped=1)) def test_skipped_all_tests(self): skipped_log = test_data_path('test_skip_all_tests.log') with open(skipped_log) as file: result = kunit_parser.parse_run_tests(file.readlines()) - self.assertEqual( - kunit_parser.TestStatus.SKIPPED, - result.status) + self.assertEqual(kunit_parser.TestStatus.SKIPPED, result.status) + self.assertEqual(result.counts, kunit_parser.TestCounts(skipped=5)) def test_ignores_hyphen(self): hyphen_log = test_data_path('test_strip_hyphen.log') @@ -241,9 +232,7 @@ class KUnitParserTest(unittest.TestCase): result = kunit_parser.parse_run_tests(file.readlines()) # A skipped test does not fail the whole suite. - self.assertEqual( - kunit_parser.TestStatus.SUCCESS, - result.status) + self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status) self.assertEqual( "sysctl_test", result.subtests[0].name) @@ -257,55 +246,49 @@ class KUnitParserTest(unittest.TestCase): prefix_log = test_data_path('test_config_printk_time.log') with open(prefix_log) as file: result = kunit_parser.parse_run_tests(file.readlines()) - self.assertEqual( - kunit_parser.TestStatus.SUCCESS, - result.status) - self.assertEqual('kunit-resource-test', result.subtests[0].name) + self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status) + self.assertEqual('kunit-resource-test', result.subtests[0].name) + self.assertEqual(result.counts.errors, 0) def test_ignores_multiple_prefixes(self): prefix_log = test_data_path('test_multiple_prefixes.log') with open(prefix_log) as file: result = kunit_parser.parse_run_tests(file.readlines()) - self.assertEqual( - kunit_parser.TestStatus.SUCCESS, - result.status) - self.assertEqual('kunit-resource-test', result.subtests[0].name) + self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status) + self.assertEqual('kunit-resource-test', result.subtests[0].name) + self.assertEqual(result.counts.errors, 0) def test_prefix_mixed_kernel_output(self): mixed_prefix_log = test_data_path('test_interrupted_tap_output.log') with open(mixed_prefix_log) as file: result = kunit_parser.parse_run_tests(file.readlines()) - self.assertEqual( - kunit_parser.TestStatus.SUCCESS, - result.status) - self.assertEqual('kunit-resource-test', result.subtests[0].name) + self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status) + self.assertEqual('kunit-resource-test', result.subtests[0].name) + self.assertEqual(result.counts.errors, 0) def test_prefix_poundsign(self): pound_log = test_data_path('test_pound_sign.log') with open(pound_log) as file: result = kunit_parser.parse_run_tests(file.readlines()) - self.assertEqual( - kunit_parser.TestStatus.SUCCESS, - result.status) - self.assertEqual('kunit-resource-test', result.subtests[0].name) + self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status) + self.assertEqual('kunit-resource-test', result.subtests[0].name) + self.assertEqual(result.counts.errors, 0) def test_kernel_panic_end(self): panic_log = test_data_path('test_kernel_panic_interrupt.log') with open(panic_log) as file: result = kunit_parser.parse_run_tests(file.readlines()) - self.assertEqual( - kunit_parser.TestStatus.TEST_CRASHED, - result.status) - self.assertEqual('kunit-resource-test', result.subtests[0].name) + self.assertEqual(kunit_parser.TestStatus.TEST_CRASHED, result.status) + self.assertEqual('kunit-resource-test', result.subtests[0].name) + self.assertGreaterEqual(result.counts.errors, 1) def test_pound_no_prefix(self): pound_log = test_data_path('test_pound_no_prefix.log') with open(pound_log) as file: result = kunit_parser.parse_run_tests(file.readlines()) - self.assertEqual( - kunit_parser.TestStatus.SUCCESS, - result.status) - self.assertEqual('kunit-resource-test', result.subtests[0].name) + self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.status) + self.assertEqual('kunit-resource-test', result.subtests[0].name) + self.assertEqual(result.counts.errors, 0) def line_stream_from_strs(strs: Iterable[str]) -> kunit_parser.LineStream: return kunit_parser.LineStream(enumerate(strs, start=1))