From patchwork Wed Nov 30 18:54:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Latypov X-Patchwork-Id: 27958 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1095278wrr; Wed, 30 Nov 2022 10:55:20 -0800 (PST) X-Google-Smtp-Source: AA0mqf4f9ijQ5dPmxzFRy2gnQG7RmzmEMo1y4Q2kHaDZ04LFwkqOm0KT94ar4khmNQtBsxjluaq9 X-Received: by 2002:a17:906:8c4:b0:7ae:fbe6:e7ca with SMTP id o4-20020a17090608c400b007aefbe6e7camr53769424eje.408.1669834520385; Wed, 30 Nov 2022 10:55:20 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669834520; cv=none; d=google.com; s=arc-20160816; b=ZGECLA+I/myKV1OnXO1Rd60+fI0IqNz7PQpW6w2EbX9PCsbeBvKB0KIivPK8x0qTXC JkkQrYXrg4xrdLAnp9G7hz+K/Mxv1k5q36gsZY1L4pX3ZoHlyNFhKpY1d0Fo9pMcHs37 XHw8bYuQr7w2L4K5rZL+IUds7h2sSecMy1r3dsS+Wyq4KyOJkLs4pYAWix/p5APK9BBY egjvGTV/YkCr83meipZguFKRCtAUrF4JsqOPVQ6je2JVT2jwSNmTyyx0sGHaecVsukLH U0AFvphAN6bXIVXs3q9sAApP4qEdE/h8YnwKNYZuLYw+qGDsgtC/xScxf1pidfsPEAHn NHOw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:mime-version:date :dkim-signature; bh=kT7qAueBeGRnOJjj9r+kURoSnmaGiK5XiYcd6AOaEjs=; b=027+3pa9/P7IkGjcu8EPEzkAuItcniFYoLqC7xH66vitNQhEvD1hDkHXzBreC3yv7y H+jJwfhZ/00viMlQSq5PWdxMVpmO8+gR8M9h9zUXofKL9XA3GGqG93omMvfdeUSGJ/T+ ze063QmoPBRf/NCbGl8v39NipVUkROddw6MhCBPqgPYfMe2fy9nz4B3mXPLUCqBR1+5M 8svkL/V2BsiHIQaNGL5suyGCa1pYbD+Tc6dl+687Ztx5QpwzDB114XePm/zA0u5zYsEc hL5FBx3ccsQCzTybz1ZKpAxkamKT/17wf9y0JEq72XETEt0ITywfKHtV80nfZ2wFW4Qw iCBA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=KBjf0M5m; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h14-20020a170906718e00b0078d1faeb619si1614688ejk.777.2022.11.30.10.54.56; Wed, 30 Nov 2022 10:55:20 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=KBjf0M5m; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231555AbiK3Syf (ORCPT + 99 others); Wed, 30 Nov 2022 13:54:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45202 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231557AbiK3Syc (ORCPT ); Wed, 30 Nov 2022 13:54:32 -0500 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BBE1063D73 for ; Wed, 30 Nov 2022 10:54:24 -0800 (PST) Received: by mail-pj1-x1049.google.com with SMTP id mh8-20020a17090b4ac800b0021348e084a0so3175557pjb.8 for ; Wed, 30 Nov 2022 10:54:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=kT7qAueBeGRnOJjj9r+kURoSnmaGiK5XiYcd6AOaEjs=; b=KBjf0M5mFzj8Y8FdDNyMasuByqyi/Ln81ZfaaVTpRJq/hZs/smb8OyyKtMCLJL0OwX QgXO+vSHteLh44HdAGy/i8wLHg/HRv4y2qoiHTYyOcjNr1EWKAw5Owvbx/5bHNUo9zTy JtGMBflHJwIcXi8DBAJA0E93yEw9GlcUC0rymXaPPJ4NyaFOvepzaDwCzySIngnZCgRO CpD+41c5Zbj053rMiRV7gpj/XYK6XEVps/+LFqpFQx+/KXcXWgkUuw45RBru0eZP2DYS zPkKYOSLVcqjT+oBa879XNwgfc/kvDsBu7qyH4MQmUMmWIquX7bBLYLZ4eA8H17U0i86 14dQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=kT7qAueBeGRnOJjj9r+kURoSnmaGiK5XiYcd6AOaEjs=; b=W1c/LONRrangIEEnFiVjehUJE+L3P4iWx0qSffTRqcKa93e/egEEMsiFlRaCZ9RnkO SR18FIgvwySEvBtYDxTPKf7Fp/tciNEHaksqE+b+jr1xPPh0ToJitdGJSVHqvE88rhcd SEZLfP4wFJpKC8xPLxY77mscbACdsRgi6D4iqiFRcDlqrcmSXek46+d7ISWo23YRX+gD 3ImS8AFoV4haLBWLjiXwB9C3RzqqKFgeCiMoTl3vxAn3Hsj+9EA6yzNaW0lCMkbnGlhq 90mEhlMFVa1Q84bKpPzvFlknaRGUo1uuzamI4eIRqXqeA7rflSpfR3NPbMlVkBeRMCgp 281A== X-Gm-Message-State: ANoB5pl3PoXSsy42if2SRtJs2lcQkqsUPZqAANLK3mj8dU94niRaQ5lo KcQYIvG0cMDpDaUaKXFRsWS8jkb00gaHCA== X-Received: from dlatypov-spec.c.googlers.com ([fda3:e722:ac3:cc00:24:72f4:c0a8:3f35]) (user=dlatypov job=sendgmr) by 2002:a17:90a:43a4:b0:219:1d0a:34a6 with SMTP id r33-20020a17090a43a400b002191d0a34a6mr2435533pjg.1.1669834463839; Wed, 30 Nov 2022 10:54:23 -0800 (PST) Date: Wed, 30 Nov 2022 10:54:19 -0800 Mime-Version: 1.0 X-Mailer: git-send-email 2.38.1.584.g0f3c55d4c2-goog Message-ID: <20221130185419.2552673-1-dlatypov@google.com> Subject: [PATCH] kunit: tool: make parser preserve whitespace when printing test log From: Daniel Latypov To: brendanhiggins@google.com, davidgow@google.com Cc: rmoar@google.com, linux-kernel@vger.kernel.org, kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org, skhan@linuxfoundation.org, Daniel Latypov X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750948402075252674?= X-GMAIL-MSGID: =?utf-8?q?1750948402075252674?= Currently, kunit_parser.py is stripping all leading whitespace to make parsing easier. But this means we can't accurately show kernel output for failing tests or when the kernel crashes. Embarassingly, this affects even KUnit's own output, e.g. [13:40:46] Expected 2 + 1 == 2, but [13:40:46] 2 + 1 == 3 (0x3) [13:40:46] not ok 1 example_simple_test [13:40:46] [FAILED] example_simple_test After this change, here's what the output in context would look like [13:40:46] =================== example (4 subtests) =================== [13:40:46] # example_simple_test: initializing [13:40:46] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29 [13:40:46] Expected 2 + 1 == 2, but [13:40:46] 2 + 1 == 3 (0x3) [13:40:46] [FAILED] example_simple_test [13:40:46] [SKIPPED] example_skip_test [13:40:46] [SKIPPED] example_mark_skipped_test [13:40:46] [PASSED] example_all_expect_macros_test [13:40:46] # example: initializing suite [13:40:46] # example: pass:1 fail:1 skip:2 total:4 [13:40:46] # Totals: pass:1 fail:1 skip:2 total:4 [13:40:46] ===================== [FAILED] example ===================== This example shows one minor cosmetic defect this approach has. The test counts lines prevent us from dedenting the suite-level output. But at the same time, any form of non-KUnit output would do the same unless it happened to be indented as well. Signed-off-by: Daniel Latypov Reviewed-by: David Gow --- tools/testing/kunit/kunit.py | 2 +- tools/testing/kunit/kunit_parser.py | 27 +++++++++++++------------- tools/testing/kunit/kunit_tool_test.py | 2 ++ 3 files changed, 16 insertions(+), 15 deletions(-) base-commit: 0f08f3e2a0186dfb8e33cb46105228eb18448a0e diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py index e7b6549712d6..43fbe96318fe 100755 --- a/tools/testing/kunit/kunit.py +++ b/tools/testing/kunit/kunit.py @@ -202,7 +202,7 @@ def parse_tests(request: KunitParseRequest, metadata: kunit_json.Metadata, input if request.raw_output == 'all': pass elif request.raw_output == 'kunit': - output = kunit_parser.extract_tap_lines(output, lstrip=False) + output = kunit_parser.extract_tap_lines(output) for line in output: print(line.rstrip()) parse_time = time.time() - parse_start diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py index 99b8f058db40..a225799f6b1b 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -13,6 +13,7 @@ from __future__ import annotations from dataclasses import dataclass import re import sys +import textwrap from enum import Enum, auto from typing import Iterable, Iterator, List, Optional, Tuple @@ -208,12 +209,12 @@ class LineStream: # Parsing helper methods: -KTAP_START = re.compile(r'KTAP version ([0-9]+)$') -TAP_START = re.compile(r'TAP version ([0-9]+)$') -KTAP_END = re.compile('(List of all partitions:|' +KTAP_START = re.compile(r'\s*KTAP version ([0-9]+)$') +TAP_START = re.compile(r'\s*TAP version ([0-9]+)$') +KTAP_END = re.compile(r'\s*(List of all partitions:|' 'Kernel panic - not syncing: VFS:|reboot: System halted)') -def extract_tap_lines(kernel_output: Iterable[str], lstrip=True) -> LineStream: +def extract_tap_lines(kernel_output: Iterable[str]) -> LineStream: """Extracts KTAP lines from the kernel output.""" def isolate_ktap_output(kernel_output: Iterable[str]) \ -> Iterator[Tuple[int, str]]: @@ -239,11 +240,8 @@ def extract_tap_lines(kernel_output: Iterable[str], lstrip=True) -> LineStream: # stop extracting KTAP lines break elif started: - # remove the prefix and optionally any leading - # whitespace. Our parsing logic relies on this. + # remove the prefix, if any. line = line[prefix_len:] - if lstrip: - line = line.lstrip() yield line_num, line return LineStream(lines=isolate_ktap_output(kernel_output)) @@ -298,7 +296,7 @@ def parse_ktap_header(lines: LineStream, test: Test) -> bool: lines.pop() return True -TEST_HEADER = re.compile(r'^# Subtest: (.*)$') +TEST_HEADER = re.compile(r'^\s*# Subtest: (.*)$') def parse_test_header(lines: LineStream, test: Test) -> bool: """ @@ -322,7 +320,7 @@ def parse_test_header(lines: LineStream, test: Test) -> bool: lines.pop() return True -TEST_PLAN = re.compile(r'1\.\.([0-9]+)') +TEST_PLAN = re.compile(r'^\s*1\.\.([0-9]+)') def parse_test_plan(lines: LineStream, test: Test) -> bool: """ @@ -350,9 +348,9 @@ def parse_test_plan(lines: LineStream, test: Test) -> bool: lines.pop() return True -TEST_RESULT = re.compile(r'^(ok|not ok) ([0-9]+) (- )?([^#]*)( # .*)?$') +TEST_RESULT = re.compile(r'^\s*(ok|not ok) ([0-9]+) (- )?([^#]*)( # .*)?$') -TEST_RESULT_SKIP = re.compile(r'^(ok|not ok) ([0-9]+) (- )?(.*) # SKIP(.*)$') +TEST_RESULT_SKIP = re.compile(r'^\s*(ok|not ok) ([0-9]+) (- )?(.*) # SKIP(.*)$') def peek_test_name_match(lines: LineStream, test: Test) -> bool: """ @@ -511,8 +509,9 @@ def print_test_header(test: Test) -> None: def print_log(log: Iterable[str]) -> None: """Prints all strings in saved log for test in yellow.""" - for m in log: - stdout.print_with_timestamp(stdout.yellow(m)) + formatted = textwrap.dedent('\n'.join(log)) + for line in formatted.splitlines(): + stdout.print_with_timestamp(stdout.yellow(line)) def format_test_result(test: Test) -> str: """ diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index 1ef921ac4331..0c2190514103 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -336,12 +336,14 @@ class KUnitParserTest(unittest.TestCase): KTAP version 1 1..1 Test output. + Indented more. not ok 1 test1 """ result = kunit_parser.parse_run_tests(output.splitlines()) self.assertEqual(kunit_parser.TestStatus.FAILURE, result.status) self.print_mock.assert_any_call(StrContains('Test output.')) + self.print_mock.assert_any_call(StrContains(' Indented more.')) self.noPrintCallContains('not ok 1 test1') def line_stream_from_strs(strs: Iterable[str]) -> kunit_parser.LineStream: