From patchwork Fri Nov 4 19:47:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rae Moar X-Patchwork-Id: 15763 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp604051wru; Fri, 4 Nov 2022 12:48:31 -0700 (PDT) X-Google-Smtp-Source: AMsMyM4GuE1/Zn8lQuuzE/I8BbHMjm8XwfUt1hb3AW/Rw6z43bO2HgWE2T3VTVXzYZ/YKWR7PWRk X-Received: by 2002:a05:6a00:1706:b0:56d:3028:23f0 with SMTP id h6-20020a056a00170600b0056d302823f0mr33596975pfc.59.1667591311016; Fri, 04 Nov 2022 12:48:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667591311; cv=none; d=google.com; s=arc-20160816; b=pNrkU1gHlJzSGTD7rK8iYp2fQ5gg1Qwz4JcK8/n7UxvTeKRN71s2MtjQ2FrmyAnDXT CKBtftoo8nG0KN/jdaH1qPD8RQGl1Lhlc2XgQSMXbSI7Bgez6FUcWUnF50TbeDyg9Gw8 hNAa5betXi1RluC/TGAL6Ot+dJYXi93gbWE4a7Gz5PZbGRit5V9VyNjs2G7lcQ24sNSz VpfU3yqbxHsR58uLy6GC8OU4L2TcaoIZW4k7npYVpbwAtwcsoi4r2SVsdaVsq67LGSbw Tu3KYA1xdeYLB7VE2T3yPe5kiK+XHkwcfEZrORE0iw9zMOpSDNfadayNR5OC4Mc8TDtP fRkg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:mime-version:date :dkim-signature; bh=2bUeD5ryefkaAP3EviNA21y9Ow7pZ1M96u/IoQ0qxoI=; b=Yn+rnf/KYAwk3jWof0feGIWxt6QLB4zzNFRhbnSVNYYWDgPlsGt3IYVUBg0TRjr26G zWnUm9jcYrooOe46PKF/rkV9r/T16taydEGgx/AWL4HMYoH2WXCb3QZnywLDPOGa89PS 7HReR0YlW9ruJlnGZdhEhhnfzlxkvdbrxh2Qmsj0Sw5tGQYYTjcclJ4gsRwDg7UOuC3g FUnRYm1Q9MpQfmRUgNLq9NQYn+6FXsqZwu6iZB0SUlKi/myHGK47zk1dDHLw88oIxyCI k3hm8dPy3+XdC5XH8pLkIGp5X5k2xdz8MtOHPC3HyokKgpw3lRcNyK1JoCGJcIiPHDft jNOg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=YaMKPHlx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id az8-20020a170902a58800b00174f3a4935fsi389371plb.249.2022.11.04.12.48.16; Fri, 04 Nov 2022 12:48:31 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=YaMKPHlx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229648AbiKDTsN (ORCPT + 99 others); Fri, 4 Nov 2022 15:48:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41704 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229619AbiKDTsL (ORCPT ); Fri, 4 Nov 2022 15:48:11 -0400 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A74402C665 for ; Fri, 4 Nov 2022 12:48:09 -0700 (PDT) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-373582569edso54735697b3.2 for ; Fri, 04 Nov 2022 12:48:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=2bUeD5ryefkaAP3EviNA21y9Ow7pZ1M96u/IoQ0qxoI=; b=YaMKPHlxllqPZ2vZAbyCAJhIsE2fCVjETNrDBwTKTaMYWSnJY/XMvkmxwORj69K0sq 0BtGbu/xP2uIqvv+taFyl3bA6mjZ3G7OGW3dg8QBDW2Gni38+3BviMEaJBpPrZ5Ubf6H Qc8YNfqdAp9oCJHMvyPZYcU99U4EGRi7twWVQ/jSsbGdJEGVUyeC07T0eau1msopD2D5 wngbzRBrXgoxvgt+6TbATOp6wukwI2wy0Kz5lq2iFy4B3Snv9aKONfSGDYuVvAP8To4w Mp1/45dn8qs2+VwioIE9vxGzTL/J9iNtLFgxmZBGAQuQvQNggXRs1ezIzoixl7V18vBk 1A2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=2bUeD5ryefkaAP3EviNA21y9Ow7pZ1M96u/IoQ0qxoI=; b=AIoUTZ6Guhov5Mkncqx6G/cxiyJWRAxtsIraLtJz4Al5aXEfhhK+k4/HnsMAyRfyIL FM4noshGiaBvwniCSG0h7g2/mE1JmNBrgAkcYtzo8q/9D8xbKku/L/jEZBAsx82hGp/1 b5sd4F45gole03+8dMnPBCASldld0FyelxP6bwPVdhSzJFXqJC7kQZ9kJIsoaKgX8g8+ D+2yWgGtLoSj1dLbBEk8ccP8unexzXvCficArCSd+VHTIDjy4Vc+VWk89/hV22VMfKwf zYQWuzay90gV0kY8lv5Yr2LVdbi2bfoqPvSg/4bcCDi5QeNKT7IXaVaiKu7TCCHTj+YR 6A9A== X-Gm-Message-State: ACrzQf2EoF8relBNd1m9GY/y9fiZfXImy+KJPheAdhWCyUQCX8aJxdTG 9IpnhSrng16Qdfv88UCYvXzzXKqjvg== X-Received: from rmoar.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:4259]) (user=rmoar job=sendgmr) by 2002:a81:17d1:0:b0:36a:6185:fb06 with SMTP id 200-20020a8117d1000000b0036a6185fb06mr35749150ywx.351.1667591289013; Fri, 04 Nov 2022 12:48:09 -0700 (PDT) Date: Fri, 4 Nov 2022 19:47:04 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.38.1.431.g37b22c650d-goog Message-ID: <20221104194705.3245738-1-rmoar@google.com> Subject: [PATCH v1 1/2] kunit: improve KTAP compliance of KUnit test output From: Rae Moar To: brendanhiggins@google.com, davidgow@google.com, dlatypov@google.com Cc: skhan@linuxfoundation.org, mauro.chehab@linux.intel.com, kunit-dev@googlegroups.com, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Rae Moar X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748596226860107652?= X-GMAIL-MSGID: =?utf-8?q?1748596226860107652?= Change KUnit test output to comply with KTAP version 1 specifications found here: https://kernel.org/doc/html/latest/dev-tools/ktap.html. 1) Use "KTAP version 1" instead of "TAP version 14" as test output header 2) Remove '-' between test number and test name on test result lines 2) Add KTAP version lines to each subtest header as well Original output: TAP version 14 1..1 # Subtest: kunit-test-suite 1..3 ok 1 - kunit_test_1 ok 2 - kunit_test_2 ok 3 - kunit_test_3 # kunit-test-suite: pass:3 fail:0 skip:0 total:3 # Totals: pass:3 fail:0 skip:0 total:3 ok 1 - kunit-test-suite New output: KTAP version 1 1..1 # Subtest: kunit-test-suite KTAP version 1 1..3 ok 1 kunit_test_1 ok 2 kunit_test_2 ok 3 kunit_test_3 # kunit-test-suite: pass:3 fail:0 skip:0 total:3 # Totals: pass:3 fail:0 skip:0 total:3 ok 1 kunit-test-suite Signed-off-by: Rae Moar Reviewed-by: Daniel Latypov Reviewed-by: David Gow --- lib/kunit/executor.c | 6 +++--- lib/kunit/test.c | 5 +++-- 2 files changed, 6 insertions(+), 5 deletions(-) base-commit: 6fe1ad4a156095859721fef85073df3ed43081d4 diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c index 9bbc422c284b..74982b83707c 100644 --- a/lib/kunit/executor.c +++ b/lib/kunit/executor.c @@ -166,7 +166,7 @@ static void kunit_exec_run_tests(struct suite_set *suite_set) { size_t num_suites = suite_set->end - suite_set->start; - pr_info("TAP version 14\n"); + pr_info("KTAP version 1\n"); pr_info("1..%zu\n", num_suites); __kunit_test_suites_init(suite_set->start, num_suites); @@ -177,8 +177,8 @@ static void kunit_exec_list_tests(struct suite_set *suite_set) struct kunit_suite * const *suites; struct kunit_case *test_case; - /* Hack: print a tap header so kunit.py can find the start of KUnit output. */ - pr_info("TAP version 14\n"); + /* Hack: print a ktap header so kunit.py can find the start of KUnit output. */ + pr_info("KTAP version 1\n"); for (suites = suite_set->start; suites < suite_set->end; suites++) kunit_suite_for_each_test_case((*suites), test_case) { diff --git a/lib/kunit/test.c b/lib/kunit/test.c index 90640a43cf62..b541d59a05c3 100644 --- a/lib/kunit/test.c +++ b/lib/kunit/test.c @@ -151,6 +151,7 @@ static void kunit_print_suite_start(struct kunit_suite *suite) { kunit_log(KERN_INFO, suite, KUNIT_SUBTEST_INDENT "# Subtest: %s", suite->name); + kunit_log(KERN_INFO, suite, KUNIT_SUBTEST_INDENT "KTAP version 1\n"); kunit_log(KERN_INFO, suite, KUNIT_SUBTEST_INDENT "1..%zd", kunit_suite_num_test_cases(suite)); } @@ -175,13 +176,13 @@ static void kunit_print_ok_not_ok(void *test_or_suite, * representation. */ if (suite) - pr_info("%s %zd - %s%s%s\n", + pr_info("%s %zd %s%s%s\n", kunit_status_to_ok_not_ok(status), test_number, description, directive_header, (status == KUNIT_SKIPPED) ? directive : ""); else kunit_log(KERN_INFO, test, - KUNIT_SUBTEST_INDENT "%s %zd - %s%s%s", + KUNIT_SUBTEST_INDENT "%s %zd %s%s%s", kunit_status_to_ok_not_ok(status), test_number, description, directive_header, (status == KUNIT_SKIPPED) ? directive : ""); From patchwork Fri Nov 4 19:47:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rae Moar X-Patchwork-Id: 15764 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp604570wru; Fri, 4 Nov 2022 12:50:02 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6hltx7M8wAEzv/mQOqCnLQrAU/IvlcZy9WDwQyFcYTUIE5Q6HEyra6WqlOc9r8D5LhPFiB X-Received: by 2002:a17:902:cf03:b0:179:b7fe:dc90 with SMTP id i3-20020a170902cf0300b00179b7fedc90mr38164821plg.112.1667591401937; Fri, 04 Nov 2022 12:50:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667591401; cv=none; d=google.com; s=arc-20160816; b=YklDLnNVedBo9/ynpJeobEjco/rK/kBQ7iIRZu0GzGQAW5yTjfSnpuiT3lbVWSSeGt 1in0DsaHP80zuMDmL4x1eMoJ8FP4socxm+trEWDaIx+i0rtpLjYYYbwct8VNukyuokKc V7O09EABpi4i92rYV0M1ZUgvhHoXAbgN4kZ7ufIeEzw63sorNDH3eOlTXH17yudpNP8F SeEoNAC8SASoQzCvAjJbBIbzqyz3vXC2bD6FTdy+eWxLj9sjeAGvzl1dkFLv4PfACC7d Wm+2T3JxOWiW+Vm7w90fGTabo/gq8/cN6NSirMFt87KPBUehGKHxe5Olg15/lKLwUeZf 14IA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=YmVRWe+ZTvwocJ5amMSffx0ArVbnRwoqLEK5+eLfveU=; b=wT78yuyuujC23wW16sFOtZI6ehymT8BElWu3Y3k9xLbPi8Ti3y2KcdNQGsE5CUTxRr 70jvwtoEp7Jw7M3q/RXO4ufwb/XAF2GL09/pdsSkK8x1prdS1zml5EOnK2HbMlenNK4i Qv8Ldo4/C5hX79s8UgrxcoY5AnLSCSjY75gBV2zXgNpmlm3SwOQICb6q5ZP3WsFIvmNH 7pUuMbcwK7hEZSZCgphx+lvYzkbL5MBnBI/V5AOwgUjBHHQiztv0KZyvTlsdbgDMblhk bZDCWa+6FHOMWIYGHhV7q2zQIVdj9P5YkPTTAVLWI9muSJYNidI5sCKqhKCubzOqRCOs djQA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=jlNwOGLx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i19-20020a635413000000b00457f4e8e21esi353176pgb.731.2022.11.04.12.49.48; Fri, 04 Nov 2022 12:50:01 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=jlNwOGLx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229733AbiKDTsP (ORCPT + 99 others); Fri, 4 Nov 2022 15:48:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41714 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229632AbiKDTsN (ORCPT ); Fri, 4 Nov 2022 15:48:13 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CBE922CDC0 for ; Fri, 4 Nov 2022 12:48:11 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id h4-20020a5b02c4000000b006bc192d672bso5810983ybp.22 for ; Fri, 04 Nov 2022 12:48:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=YmVRWe+ZTvwocJ5amMSffx0ArVbnRwoqLEK5+eLfveU=; b=jlNwOGLxD0ZiiRzt9L5CK1CIgnMFD1Ckz7GE1zr0Y8clOhfuxHMOrSYu5cOtErUNaB jBNAE3PPRVGCDl6ZjOfWBOdfslMVmsgQThSWS3UvVdKAACpTPrkVfy2kXngFwX3nlvup beg1Lt2JWfvIvE580th65ST33V46vAqANvBijnpUQn2NU3WsT95Xkm1wQ531nUtVS/aQ 6mOnqGjD30XiJsQduYOHGHJTdHVJtnyxCP0G64OIG7E6z0m1ogoI/AMTX8RfBKu9jj8g dISoDNGZPsezZ675elUOR3Kril2TtHvqU6V5A2/gs/PHQpG8BdegcgTnV3TpovWrt/HF aq7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=YmVRWe+ZTvwocJ5amMSffx0ArVbnRwoqLEK5+eLfveU=; b=wvT0oR5lICLF/KnICDxYHyGuk9tx/RqUGTKjWr3eIHhyBTpt78uWtRrw4yoPC6POrg 9B5GZ8WZvZjGqPyLzKP3TMhy++3Ruo2HH/mCWwV90mQ6PXRfyf7II6ul6DF8wLxEzY77 pQobxTsU1p1gR92P4uzA37BsnvXPQ6qKP5UnwC+uXYP8xaq1I+uPe8zipwALvoYDMe6Q ikaA0BkLU0nWyyhUILdoWKYGdExFMiFDbbqDROyTmsWKlBli9rvlrJ50WRueDpGbT5q9 3xIBjieIPzDUJcSH1VNZ/TpUli+OGC2vpPCj0nN080QzQJBSOrb3g/a7wQOUXvA4EEiW P1hA== X-Gm-Message-State: ACrzQf1bxTzdq/J9pazqNCzbytqoChzk/pMPffuKdPalP9nBKD+UBvJF ukCouVepyCkn2LDA0Nl1cgjtI3RTcA== X-Received: from rmoar.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:4259]) (user=rmoar job=sendgmr) by 2002:a25:4fc2:0:b0:6be:afb4:d392 with SMTP id d185-20020a254fc2000000b006beafb4d392mr33870585ybb.604.1667591291143; Fri, 04 Nov 2022 12:48:11 -0700 (PDT) Date: Fri, 4 Nov 2022 19:47:05 +0000 In-Reply-To: <20221104194705.3245738-1-rmoar@google.com> Mime-Version: 1.0 References: <20221104194705.3245738-1-rmoar@google.com> X-Mailer: git-send-email 2.38.1.431.g37b22c650d-goog Message-ID: <20221104194705.3245738-2-rmoar@google.com> Subject: [PATCH v1 2/2] kunit: tool: parse KTAP compliant test output From: Rae Moar To: brendanhiggins@google.com, davidgow@google.com, dlatypov@google.com Cc: skhan@linuxfoundation.org, mauro.chehab@linux.intel.com, kunit-dev@googlegroups.com, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Rae Moar X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748596321828117312?= X-GMAIL-MSGID: =?utf-8?q?1748596321828117312?= Change the KUnit parser to be able to parse test output that complies with the KTAP version 1 specification format found here: https://kernel.org/doc/html/latest/dev-tools/ktap.html. Ensure the parser is able to parse tests with the original KUnit test output format as well. KUnit parser now accepts any of the following test output formats: Original KUnit test output format: TAP version 14 1..1 # Subtest: kunit-test-suite 1..3 ok 1 - kunit_test_1 ok 2 - kunit_test_2 ok 3 - kunit_test_3 # kunit-test-suite: pass:3 fail:0 skip:0 total:3 # Totals: pass:3 fail:0 skip:0 total:3 ok 1 - kunit-test-suite KTAP version 1 test output format: KTAP version 1 1..1 KTAP version 1 1..3 ok 1 kunit_test_1 ok 2 kunit_test_2 ok 3 kunit_test_3 ok 1 kunit-test-suite New KUnit test output format (preferred for KUnit tests): KTAP version 1 1..1 # Subtest: kunit-test-suite KTAP version 1 1..3 ok 1 kunit_test_1 ok 2 kunit_test_2 ok 3 kunit_test_3 # kunit-test-suite: pass:3 fail:0 skip:0 total:3 # Totals: pass:3 fail:0 skip:0 total:3 ok 1 kunit-test-suite Signed-off-by: Rae Moar Reviewed-by: Daniel Latypov Reviewed-by: David Gow --- Note: this patch is based on the linux-kselftest/kunit branch. --- tools/testing/kunit/kunit_parser.py | 69 ++++++++++++------- tools/testing/kunit/kunit_tool_test.py | 8 +++ .../test_data/test_parse_ktap_output.log | 8 +++ 3 files changed, 60 insertions(+), 25 deletions(-) create mode 100644 tools/testing/kunit/test_data/test_parse_ktap_output.log diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py index a56c75a973b5..abb69f898263 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -441,6 +441,7 @@ def parse_diagnostic(lines: LineStream) -> List[str]: - '# Subtest: [test name]' - '[ok|not ok] [test number] [-] [test name] [optional skip directive]' + - 'KTAP version [version number]' Parameters: lines - LineStream of KTAP output to parse @@ -449,8 +450,9 @@ def parse_diagnostic(lines: LineStream) -> List[str]: Log of diagnostic lines """ log = [] # type: List[str] - while lines and not TEST_RESULT.match(lines.peek()) and not \ - TEST_HEADER.match(lines.peek()): + non_diagnostic_lines = [TEST_RESULT, TEST_HEADER, KTAP_START] + while lines and not any(re.match(lines.peek()) + for re in non_diagnostic_lines): log.append(lines.pop()) return log @@ -496,6 +498,12 @@ def print_test_header(test: Test) -> None: test - Test object representing current test being printed """ message = test.name + if message == "": + # KUnit tests print a Subtest header line that provides the name + # of the test suite. But the subtest header line isn't required + # by the KTAP spec, so use a placeholder name "Test suite" in that + # case. + message = "Test suite" if test.expected_count: if test.expected_count == 1: message += ' (1 subtest)' @@ -647,13 +655,13 @@ def bubble_up_test_results(test: Test) -> None: elif test.counts.get_status() == TestStatus.TEST_CRASHED: test.status = TestStatus.TEST_CRASHED -def parse_test(lines: LineStream, expected_num: int, log: List[str]) -> Test: +def parse_test(lines: LineStream, expected_num: int, log: List[str], is_subtest: bool) -> Test: """ Finds next test to parse in LineStream, creates new Test object, parses any subtests of the test, populates Test object with all information (status, name) about the test and the Test objects for any subtests, and then returns the Test object. The method accepts - three formats of tests: + four formats of tests: Accepted test formats: @@ -674,6 +682,16 @@ def parse_test(lines: LineStream, expected_num: int, log: List[str]) -> Test: [subtests] ok 1 name + - KTAP subtest header (in compliance with KTAP specification) + + Example: + + # May include subtest header line here + KTAP version 1 + 1..3 + [subtests] + ok 1 name + - Test result line Example: @@ -685,6 +703,7 @@ def parse_test(lines: LineStream, expected_num: int, log: List[str]) -> Test: expected_num - expected test number for test to be parsed log - list of strings containing any preceding diagnostic lines corresponding to the current test + is_subtest - boolean indicating whether test is a subtest Return: Test object populated with characteristics and any subtests @@ -692,21 +711,22 @@ def parse_test(lines: LineStream, expected_num: int, log: List[str]) -> Test: test = Test() test.log.extend(log) parent_test = False - main = parse_ktap_header(lines, test) - if main: - # If KTAP/TAP header is found, attempt to parse - # test plan + if not is_subtest: + # If parsing the main test, attempt to parse KTAP/TAP header + # and test plan test.name = "main" + parse_ktap_header(lines, test) parse_test_plan(lines, test) parent_test = True else: - # If KTAP/TAP header is not found, test must be subtest - # header or test result line so parse attempt to parser - # subtest header - parent_test = parse_test_header(lines, test) + # If test is a subtest, attempt to parse test suite header + # (either subtest line and/or KTAP/TAP version line) + subtest_line = parse_test_header(lines, test) + ktap_line = parse_ktap_header(lines, test) + parent_test = subtest_line or ktap_line if parent_test: - # If subtest header is found, attempt to parse - # test plan and print header + # If subtest header and/or KTAP/version line is found, attempt + # to parse test plan and print header parse_test_plan(lines, test) print_test_header(test) expected_count = test.expected_count @@ -721,7 +741,7 @@ def parse_test(lines: LineStream, expected_num: int, log: List[str]) -> Test: sub_log = parse_diagnostic(lines) sub_test = Test() if not lines or (peek_test_name_match(lines, test) and - not main): + is_subtest): if expected_count and test_num <= expected_count: # If parser reaches end of test before # parsing expected number of subtests, print @@ -735,20 +755,19 @@ def parse_test(lines: LineStream, expected_num: int, log: List[str]) -> Test: test.log.extend(sub_log) break else: - sub_test = parse_test(lines, test_num, sub_log) + sub_test = parse_test(lines, test_num, sub_log, True) subtests.append(sub_test) test_num += 1 test.subtests = subtests - if not main: + if is_subtest: # If not main test, look for test result line test.log.extend(parse_diagnostic(lines)) - if (parent_test and peek_test_name_match(lines, test)) or \ - not parent_test: - parse_test_result(lines, test, expected_num) - else: + if subtest_line and not peek_test_name_match(lines, test): test.add_error('missing subtest result line!') + else: + parse_test_result(lines, test, expected_num) - # Check for there being no tests + # Check for there being no subtests within parent test if parent_test and len(subtests) == 0: # Don't override a bad status if this test had one reported. # Assumption: no subtests means CRASHED is from Test.__init__() @@ -758,11 +777,11 @@ def parse_test(lines: LineStream, expected_num: int, log: List[str]) -> Test: # Add statuses to TestCounts attribute in Test object bubble_up_test_results(test) - if parent_test and not main: + if parent_test and is_subtest: # If test has subtests and is not the main test object, print # footer. print_test_footer(test) - elif not main: + elif is_subtest: print_test_result(test) return test @@ -785,7 +804,7 @@ def parse_run_tests(kernel_output: Iterable[str]) -> Test: test.add_error('could not find any KTAP output!') test.status = TestStatus.FAILURE_TO_PARSE_TESTS else: - test = parse_test(lines, 0, []) + test = parse_test(lines, 0, [], False) if test.status != TestStatus.NO_TESTS: test.status = test.counts.get_status() stdout.print_with_timestamp(DIVIDER) diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index 90c65b072be9..7c2e2a45f330 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -312,6 +312,14 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual(kunit_parser._summarize_failed_tests(result), 'Failures: all_failed_suite, some_failed_suite.test2') + def test_ktap_format(self): + ktap_log = test_data_path('test_parse_ktap_output.log') + with open(ktap_log) as file: + result = kunit_parser.parse_run_tests(file.readlines()) + self.assertEqual(result.counts, kunit_parser.TestCounts(passed=3)) + self.assertEqual('suite', result.subtests[0].name) + self.assertEqual('case_1', result.subtests[0].subtests[0].name) + self.assertEqual('case_2', result.subtests[0].subtests[1].name) def line_stream_from_strs(strs: Iterable[str]) -> kunit_parser.LineStream: return kunit_parser.LineStream(enumerate(strs, start=1)) diff --git a/tools/testing/kunit/test_data/test_parse_ktap_output.log b/tools/testing/kunit/test_data/test_parse_ktap_output.log new file mode 100644 index 000000000000..ccdf244e5303 --- /dev/null +++ b/tools/testing/kunit/test_data/test_parse_ktap_output.log @@ -0,0 +1,8 @@ +KTAP version 1 +1..1 + KTAP version 1 + 1..3 + ok 1 case_1 + ok 2 case_2 + ok 3 case_3 +ok 1 suite