From patchwork Fri Oct 28 21:02:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Latypov X-Patchwork-Id: 12561 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp1044691wru; Fri, 28 Oct 2022 14:10:06 -0700 (PDT) X-Google-Smtp-Source: AMsMyM75Ooo6FvFmj2KjRVLhszd3er0pPt8QNnvjQnqmG+fxvfaQUKeFH6dLlEF3+bc4eER4kPDj X-Received: by 2002:a05:6a00:230d:b0:53d:c198:6ad7 with SMTP id h13-20020a056a00230d00b0053dc1986ad7mr1085788pfh.67.1666991405721; Fri, 28 Oct 2022 14:10:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666991405; cv=none; d=google.com; s=arc-20160816; b=g+csH0BWm4iFc9ifbg8dtCZP3nzMtRQxsrqeGkVb7m+B1Nx1EVwuUPLCGUCuEp0I+M OXRIHiYmtj9cH4w8nJ5jAxeb8Mmok5vHSiHdZyaCdAq8Zlzi34BjzUxSJie/RMSiJDkm h2mXUJnyNVQpao7TMv3Y7vidfz0YkZzTpI4Xdro39f+lliY7gjrouAuhtS5+ImbtGvm/ 1YsNK/wyl+tdKcZnGgRD7fmocDSI1+qbcrQ8Q74fUIxRIlBBICRhF3rRC26cc8nPvOkw 9OTcsIdiTMkJSxf8EOaOXE/xl1Bt1kkfuA8VPkGmARPoGeBG+YALo5P9VZXEFdt/wSBZ loIg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:mime-version:date :dkim-signature; bh=XV7iaO5xhJsUUi6v6TQz064kMq8CSbatm+UbozZNDY0=; b=wlAUBoJsqX2D4FeNo1m5tBmx17nXUcX1xtCJa+OPs2gYjZb/Ql+bDkD+6yGQCPScIr /PuCSJdRp9BucXwCvwwYZ6d8uTF6VoeiWnHE3sE/trd20hpZlnkcRbWXw3X0su+vL1Gh AzKzw6U/4girgklXRA2s9xFamR0y7ABNJJVXhDgBR0Pc8nmPrtz48G+8iliCmUHYid6b 3rRHKUpDVUFbShtUQgYZb2WH2XOhDr4PbCD1uhIleVsMtUDzZ3RehuI9GFkbqmlNMiWf LQOZt42LnhaXImgrptDFuT/zUCtaBiTW55vNCsB7fSq2gE7jY/ntVkDW91GxKXIk4tXQ io8g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=WdEBUnTb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q18-20020a17090311d200b00186b1bb149fsi7255685plh.474.2022.10.28.14.09.52; Fri, 28 Oct 2022 14:10:05 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=WdEBUnTb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229553AbiJ1VDO (ORCPT + 99 others); Fri, 28 Oct 2022 17:03:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46164 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229457AbiJ1VDM (ORCPT ); Fri, 28 Oct 2022 17:03:12 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5695A244C7D for ; Fri, 28 Oct 2022 14:03:07 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id b14-20020a056902030e00b006a827d81fd8so5403358ybs.17 for ; Fri, 28 Oct 2022 14:03:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=XV7iaO5xhJsUUi6v6TQz064kMq8CSbatm+UbozZNDY0=; b=WdEBUnTbhxJLMg0NrpL18rCOOI7uGGfwy1OMESfUixvQGK+UZ2TPM+1YJGLk5ibNVw EXecXCN/Yv+UYJ3wSew4oqQF8QxgF+5fAR00RCwkwOFkXt+WCmCpoHN0C/FeKzXJ0tnl xaFJhvqw4/JXMeuzBaHnIzmq9RlxXf2co0A9Pv4sJ3TsTi7lVx+PJGcEuODK1lCQCf/p Zmsgax6qqXV8Jul+miO9w7bK8zvDTkwcVMPJuqVb8kQ01h+0wuwgHnEc0768xPh3kegr V2KCT4zllQc+IT8uUA5H/xPVoPZLUdGM/ngcaySaSQsppTHAnkL2vwK8qJEXmNl3EWPW 9Yiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=XV7iaO5xhJsUUi6v6TQz064kMq8CSbatm+UbozZNDY0=; b=SkCRRR/KzlCvwgyyt08LiVkCbJIdt2LH8XIUYtCECdbNMtxafkEGt4HiHItCLi7l1D gNkLzPPD1dahIoVooDmEzzOepjEpwrvokcdcu51gb+/SPyXlmsyEEAkCvzKrxm2LJESU Nw5Z0w/dGv6ZqRsPE7uK01XSwpXwzsbWhbdgMMcjc+rf7wiafBurK/tjqqsHZ0KEWgl9 yOOVr8/okAVNmURJulwZ0DCD5ploUEFFZGmbLMfhN28ODMpWGTiDclR83aV7L2CrqVDH H2sPZnlS7A5Gm6RRMjMeEEybXCrcv1lZeHWbFzZiWJjNihjvmiwCRaBtalg/gYOSxqhA 4ADA== X-Gm-Message-State: ACrzQf1LN7MQeopYDYdhRTz97WYcexzeK81/fe2rYDDcR51xpJImhmLL 5GdujEVSeovZSuNaWDT35jTn5UfPiW3U6A== X-Received: from dlatypov-spec.c.googlers.com ([fda3:e722:ac3:cc00:24:72f4:c0a8:3f35]) (user=dlatypov job=sendgmr) by 2002:a81:bc5:0:b0:36f:fd68:b56a with SMTP id 188-20020a810bc5000000b0036ffd68b56amr1ywl.41.1666990986198; Fri, 28 Oct 2022 14:03:06 -0700 (PDT) Date: Fri, 28 Oct 2022 14:02:55 -0700 Mime-Version: 1.0 X-Mailer: git-send-email 2.38.1.273.g43a17bfeac-goog Message-ID: <20221028210256.3776835-1-dlatypov@google.com> Subject: [PATCH 1/2] kunit: tool: make unit test not print parsed testdata to stdout From: Daniel Latypov To: brendanhiggins@google.com, davidgow@google.com Cc: rmoar@google.com, linux-kernel@vger.kernel.org, kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org, skhan@linuxfoundation.org, Daniel Latypov X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747967180265947024?= X-GMAIL-MSGID: =?utf-8?q?1747967180265947024?= Currently, if you run $ ./tools/testing/kunit/kunit_tool_test.py you'll see a lot of output from the parser as we feed it testdata. This makes the output hard to read and fairly confusing, esp. since our testdata includes example failures, which get printed out in red. Silence that output so real failures are easier to see. Signed-off-by: Daniel Latypov Reviewed-by: David Gow --- tools/testing/kunit/kunit_tool_test.py | 6 ++++++ 1 file changed, 6 insertions(+) base-commit: 8f8b51f7d5c8bd3a89e7ea87aed2cdaa52ca5ba4 diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index e2cd2cc2e98f..a6e53945656e 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -80,6 +80,9 @@ class KconfigTest(unittest.TestCase): self.assertEqual(actual_kconfig, expected_kconfig) class KUnitParserTest(unittest.TestCase): + def setUp(self): + self.print_mock = mock.patch('kunit_printer.Printer.print').start() + self.addCleanup(mock.patch.stopall) def assertContains(self, needle: str, haystack: kunit_parser.LineStream): # Clone the iterator so we can print the contents on failure. @@ -485,6 +488,9 @@ class LinuxSourceTreeTest(unittest.TestCase): class KUnitJsonTest(unittest.TestCase): + def setUp(self): + self.print_mock = mock.patch('kunit_printer.Printer.print').start() + self.addCleanup(mock.patch.stopall) def _json_for(self, log_file): with open(test_data_path(log_file)) as file: From patchwork Fri Oct 28 21:02:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Latypov X-Patchwork-Id: 12557 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp1043975wru; Fri, 28 Oct 2022 14:08:34 -0700 (PDT) X-Google-Smtp-Source: AMsMyM61Q228mV922SOl3K3woZFyypwJlzpej//jav0NSrlW/TDBQGmXRiOSpm+hLnMsqBimGx6W X-Received: by 2002:a17:906:dcc:b0:7ad:a454:5642 with SMTP id p12-20020a1709060dcc00b007ada4545642mr1106282eji.521.1666991313943; Fri, 28 Oct 2022 14:08:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666991313; cv=none; d=google.com; s=arc-20160816; b=JWsXd8U0kK/8SZy/eqQ2cbL667zF/OaJf3LMjcFAaynAY5f2oHJYwNAN/xSp4YVH25 H738N6VUijCKQWhpreg65c2uGYM2tn561dqJBf6mz2uBFngCqhoZznZ4RGH+EMbiAtVH Cat9UnD7ML0IBix0lSiRMelJenpCL11DUng4fN472cJhpigRlqbWIFDaC814aJLDE99a ipDp0IDq6EECWo4DH/eM7y6eo9wvW+clx1W8pkMxhsOxbdv4VQ+DHAAqyiRkyt5+MJfp QM6fSpCxi5HjgGEi5do9HTBgyA8ZO2Og7fHUtXrRqavK0gao1dcH9jdboQpeuBzwEKOM +iog== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=7SXh0okIwiV6Ma7sPYTqdOlim1w0WkxuN7ZrR2Or+Zk=; b=u1A+rG+9ThKIUuUi9h6/g/eeNkbjGye+6Mxp6WFHF6jbQ/N+91GNE8avCEEvcWGihl QdkmG2YuowA96Y857q7Oj+n8hBJ9AVu9QNxUTY7+UydlNiumn8PeOa3B/icogS+v9p8V S8WJP2MyslIsvk6RSYs92exUdtFvSCCuxOs30burIyzgni1RT/Suhie8qkd1bsGPoIhD P7spqWmGmkwsM+uWp37CmJMaXNXnSl3FbSygWf22Gt6gW6zB8pKhmQg2eNEqOrepIlkU 9zHK7QuSxcL6qQc/VtIA3g1jrPyEek4XbXOXaeLIAhB02yYgUbGIsKcxcgaTXmjX2FUx 0/6w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=sMoW1Vjs; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id p15-20020aa7d30f000000b00462d53d6de6si2100840edq.10.2022.10.28.14.08.07; Fri, 28 Oct 2022 14:08:33 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=sMoW1Vjs; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229494AbiJ1VDR (ORCPT + 99 others); Fri, 28 Oct 2022 17:03:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46166 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229544AbiJ1VDO (ORCPT ); Fri, 28 Oct 2022 17:03:14 -0400 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 75AC7244C7E for ; Fri, 28 Oct 2022 14:03:08 -0700 (PDT) Received: by mail-pf1-x449.google.com with SMTP id s11-20020a056a00178b00b0056cb4545c3fso2483397pfg.5 for ; Fri, 28 Oct 2022 14:03:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7SXh0okIwiV6Ma7sPYTqdOlim1w0WkxuN7ZrR2Or+Zk=; b=sMoW1VjscveV6hpN+gf0TGyCtjqBhB6rkQLUDWkT6TJIQDVnicRXENiLc1lewMZJar 4GsUxtoNkn2H4lBoXwTCmtzEU04exdCF8/06lItVqpwlz752bPH55HheTJ21XOwXImpS T/xe6vXJy2QfEWHvCUP9lI/0D43YEST8/0U81nBabiErdYqKTiO3x74U6LMXJtwl2GU+ sc59v2GoeaSjNA7PALOGSYFz4XB61mbu+DiQBTHtZsynxQwaJDBxR8KwZyve3O7O8D59 jtNP11toD85lQCkSgLSMoOxP3W0zczBS+ERHlFHxyFywpwoZoWxFEH8lSfDE6wUA6wc6 94SA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7SXh0okIwiV6Ma7sPYTqdOlim1w0WkxuN7ZrR2Or+Zk=; b=79cP/I3EYaMoVEkmqC5UEYQjK/MNJ43Tg3X1H/VnkW05unAUTlrJzz3TVRGTEoGy4N hqx4LXvrpguSuvTp/nWpnmHthfWeLIGLZkqvzH1jkdQQKOGOE7aQV6LVy3YsaS2F8GMY OC/pSWd7Bhj/Lj7R980t3dinRpmzk3dJOmTAKJoWGhZh4Za85HnMqFaAUcZxgNJMnt7R UHw9SNGvxvuPYVUfrD68POPt7OkMK6qwAf/X8+L0E58F827bhg0zaNoBjCdGtLE78Ery rHCzHxsN7Q5H2soaWjX7g80j38fFzDIYXnmqGXSSPNhZxwSkujsQ84VHmdJrnj1N00bI AwUA== X-Gm-Message-State: ACrzQf1FbmstdYN4OjjdTKfLs16+EaVWDIkVUl57ZA18qzc8Y4SKI/ba 7wcAyeiY1pVCfEzPAhegN41XdEVOixDoUw== X-Received: from dlatypov-spec.c.googlers.com ([fda3:e722:ac3:cc00:24:72f4:c0a8:3f35]) (user=dlatypov job=sendgmr) by 2002:a05:6a00:1707:b0:566:15a1:8b07 with SMTP id h7-20020a056a00170700b0056615a18b07mr1023804pfc.34.1666990987970; Fri, 28 Oct 2022 14:03:07 -0700 (PDT) Date: Fri, 28 Oct 2022 14:02:56 -0700 In-Reply-To: <20221028210256.3776835-1-dlatypov@google.com> Mime-Version: 1.0 References: <20221028210256.3776835-1-dlatypov@google.com> X-Mailer: git-send-email 2.38.1.273.g43a17bfeac-goog Message-ID: <20221028210256.3776835-2-dlatypov@google.com> Subject: [PATCH 2/2] kunit: tool: print summary of failed tests if a few failed out of a lot From: Daniel Latypov To: brendanhiggins@google.com, davidgow@google.com Cc: rmoar@google.com, linux-kernel@vger.kernel.org, kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org, skhan@linuxfoundation.org, Daniel Latypov X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747967084150816803?= X-GMAIL-MSGID: =?utf-8?q?1747967084150816803?= E.g. all the hw_breakpoint tests are failing right now. So if I run `kunit.py run --altests --arch=x86_64`, then I see > Testing complete. Ran 408 tests: passed: 392, failed: 9, skipped: 7 Seeing which 9 tests failed out of the hundreds is annoying. If my terminal doesn't have scrollback support, I have to resort to looking at `.kunit/test.log` for the `not ok` lines. Teach kunit.py to print a summarized list of failures if the # of tests reachs an arbitrary threshold (>=100 tests). To try and keep the output from being too long/noisy, this new logic a) just reports "parent_test failed" if every child test failed b) won't print anything if there are >10 failures (also arbitrary). With this patch, we get an extra line of output showing: > Testing complete. Ran 408 tests: passed: 392, failed: 9, skipped: 7 > Failures: hw_breakpoint This also works with parameterized tests, e.g. if I add a fake failure > Failures: kcsan.test_atomic_builtins_missing_barrier.threads=6 Note: we didn't have enough tests for this to be a problem before. But with commit 980ac3ad0512 ("kunit: tool: rename all_test_uml.config, use it for --alltests"), --alltests works and thus running >100 tests will probably become more common. Signed-off-by: Daniel Latypov Reviewed-by: David Gow --- tools/testing/kunit/kunit_parser.py | 47 ++++++++++++++++++++++++++ tools/testing/kunit/kunit_tool_test.py | 22 ++++++++++++ 2 files changed, 69 insertions(+) diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py index 1ae873e3e341..94dba66feec5 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -58,6 +58,10 @@ class Test: self.counts.errors += 1 stdout.print_with_timestamp(stdout.red('[ERROR]') + f' Test: {self.name}: {error_message}') + def ok_status(self) -> bool: + """Returns true if the status was ok, i.e. passed or skipped.""" + return self.status in (TestStatus.SUCCESS, TestStatus.SKIPPED) + class TestStatus(Enum): """An enumeration class to represent the status of a test.""" SUCCESS = auto() @@ -565,6 +569,40 @@ def print_test_footer(test: Test) -> None: stdout.print_with_timestamp(format_test_divider(message, len(message) - stdout.color_len())) + + +def _summarize_failed_tests(test: Test) -> str: + """Tries to summarize all the failing subtests in `test`.""" + + def failed_names(test: Test, parent_name: str) -> List[str]: + # Note: we use 'main' internally for the top-level test. + if not parent_name or parent_name == 'main': + full_name = test.name + else: + full_name = parent_name + '.' + test.name + + if not test.subtests: # this is a leaf node + return [full_name] + + # If all the children failed, just say this subtest failed. + # Don't summarize it down "the top-level test failed", though. + failed_subtests = [sub for sub in test.subtests if not sub.ok_status()] + if parent_name and len(failed_subtests) == len(test.subtests): + return [full_name] + + all_failures = [] # type: List[str] + for t in failed_subtests: + all_failures.extend(failed_names(t, full_name)) + return all_failures + + failures = failed_names(test, '') + # If there are too many failures, printing them out will just be noisy. + if len(failures) > 10: # this is an arbitrary limit + return '' + + return 'Failures: ' + ', '.join(failures) + + def print_summary_line(test: Test) -> None: """ Prints summary line of test object. Color of line is dependent on @@ -587,6 +625,15 @@ def print_summary_line(test: Test) -> None: color = stdout.red stdout.print_with_timestamp(color(f'Testing complete. {test.counts}')) + # Summarize failures that might have gone off-screen since we had a lot + # of tests (arbitrarily defined as >=100 for now). + if test.ok_status() or test.counts.total() < 100: + return + summarized = _summarize_failed_tests(test) + if not summarized: + return + stdout.print_with_timestamp(color(summarized)) + # Other methods: def bubble_up_test_results(test: Test) -> None: diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index a6e53945656e..7dcd67003b23 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -312,6 +312,28 @@ class KUnitParserTest(unittest.TestCase): result.status) self.assertEqual('kunit-resource-test', result.subtests[0].name) + def test_summarize_failures(self): + output = """ + KTAP version 1 + 1..2 + # Subtest: all_failed_suite + 1..2 + not ok 1 - test1 + not ok 2 - test2 + not ok 1 - all_failed_suite + # Subtest: some_failed_suite + 1..2 + ok 1 - test1 + not ok 2 - test2 + not ok 1 - some_failed_suite + """ + result = kunit_parser.parse_run_tests(output.splitlines()) + self.assertEqual(kunit_parser.TestStatus.FAILURE, result.status) + + self.assertEqual(kunit_parser._summarize_failed_tests(result), + 'Failures: all_failed_suite, some_failed_suite.test2') + + def line_stream_from_strs(strs: Iterable[str]) -> kunit_parser.LineStream: return kunit_parser.LineStream(enumerate(strs, start=1))