Message ID | 20230731141021.2854827-6-janusz.krzysztofik@linux.intel.com |
---|---|
State | New |
Headers |
Return-Path: <linux-kernel-owner@vger.kernel.org> Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp2071887vqg; Mon, 31 Jul 2023 08:01:24 -0700 (PDT) X-Google-Smtp-Source: APBJJlFtwdwfWw0P87nW3iAqipNyHe3gjeVfHwARCIYpF+IGk3crLSNWi+laJ4aQcVeYdJQxBbWj X-Received: by 2002:a05:6a20:a104:b0:118:e70:6f7d with SMTP id q4-20020a056a20a10400b001180e706f7dmr10601846pzk.10.1690815684285; Mon, 31 Jul 2023 08:01:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690815684; cv=none; d=google.com; s=arc-20160816; b=hL8gzc7SNAI/fqhxzvIMmB5ATLDuq2G2X5Jgvz5Y4RbHqWhnFGJ2uE8B4vKX9l7FG5 92b75PjDwXdyC9OUczI9f0x+dMi1fK+/VRu+HknQO7oywRu1Czn2B1kIJ9xoTqrsw9C0 xiUswhANog7CabylBmJHG8IL7QI+ZJsLjCMoDpJBRztlGtFTVjwd5wVknLvqp4x/Q1g4 wRe3YQJxyS2rFsxgraDI6cY46PVpm5cx8Vg+LqOAuk4Dz+u27E6499BRhohPYnnJBU86 EzP/AD0PR6WAvQeOE1hTEwytrjG+q87GWxuy1krXJd2LKGMkashotoUr0dPwS/LTarZA gVhw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=molosIvnLrvnltEaGVKzvi1qIz4BKcQV3nndhcgulbY=; fh=2DascdEigk5YKkI7xsPhzXnEwnq6+gHGf5jSQyJPeBg=; b=Hk6D9uoSYh9OeqMTgvT2HDrAzVV0MipujYZzqqHZahzKr4P168TfwgG0OidulSG/I+ B9Vk1jdofsYuMCOt6NTaUjVxMpOGVHPjEOeoN/tEpp8X0aPvNb9ro+s7syqOD0lp1OeQ vVUscCFagINoZoPkmLGJtM8EU4etISzG4XAlW1PVqoPYu9YdI9WQEdUgtfEwqklxCsH/ TOQmhMcUThj5o/kodSpCXN8956PA2W1RHrUoiSpuYuJQ3PYJHrR48HmfevQvwxlQNFZ9 MR7NNHiZCwNl7Gstkou1xRE2F5+rTn0H1GJOwKOpzVx9CSpmL+43zOjp0W4t5Czuj+rg wB7g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=D7RoLEGS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i5-20020a636d05000000b005538c82b70fsi7356977pgc.166.2023.07.31.08.01.08; Mon, 31 Jul 2023 08:01:24 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=D7RoLEGS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231773AbjGaONH (ORCPT <rfc822;dengxinlin2429@gmail.com> + 99 others); Mon, 31 Jul 2023 10:13:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60112 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231867AbjGaOMr (ORCPT <rfc822;linux-kernel@vger.kernel.org>); Mon, 31 Jul 2023 10:12:47 -0400 Received: from mgamail.intel.com (unknown [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 71F8AD8; Mon, 31 Jul 2023 07:12:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690812764; x=1722348764; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=xfWKTT2Wn3/IWe//m/LpBpFN0F6nm+3nkhbl/ibfElU=; b=D7RoLEGSSRXBJYSsWpoOe1BhF5oPVLCNW1iv8RkHovivM1L2XEWwT7J1 d67jcxsUMgYYZjJarVfsgD0cgFeA5VpPnNMSIfn16IGQd/SaCORIiWkAP ketO89XEp/tZy+ACCLfvti754Nt1YTVgS7C/rGcqNjhxXjMRjerTPPtlL dAJA6VK9Oi21pV671DsNsPAXJhAF7yAZge8pp+kJlp17Dc1OhlFA+3AsI JEqdGvdRBsc7JBJgvIG3AJ43A8gP8lFnHMcEH6Gpe+EKZxTggnf3QVyZp Kfla1iFrZfVbxNhtFEKTMWptSKG1fz7MyoNFrXrYK8wmzf9NB6QDkcGom A==; X-IronPort-AV: E=McAfee;i="6600,9927,10788"; a="455403626" X-IronPort-AV: E=Sophos;i="6.01,244,1684825200"; d="scan'208";a="455403626" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 31 Jul 2023 07:12:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10788"; a="728324347" X-IronPort-AV: E=Sophos;i="6.01,244,1684825200"; d="scan'208";a="728324347" Received: from jkrzyszt-mobl2.ger.corp.intel.com (HELO jkrzyszt-mobl2.intranet) ([10.213.1.128]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 31 Jul 2023 07:12:41 -0700 From: Janusz Krzysztofik <janusz.krzysztofik@linux.intel.com> To: Brendan Higgins <brendan.higgins@linux.dev>, David Gow <davidgow@google.com> Cc: linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, Mauro Carvalho Chehab <mchehab@kernel.org>, igt-dev@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, Janusz Krzysztofik <janusz.krzysztofik@linux.intel.com> Subject: [PATCH v3 1/3] kunit: Report the count of test suites in a module Date: Mon, 31 Jul 2023 16:10:23 +0200 Message-ID: <20230731141021.2854827-6-janusz.krzysztofik@linux.intel.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230731141021.2854827-5-janusz.krzysztofik@linux.intel.com> References: <20230731141021.2854827-5-janusz.krzysztofik@linux.intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: <linux-kernel.vger.kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772948747209634542 X-GMAIL-MSGID: 1772948747209634542 |
Series |
kunit: Expose some built-in features to modules
|
|
Commit Message
Janusz Krzysztofik
July 31, 2023, 2:10 p.m. UTC
According to KTAP specification[1], results should always start from a
header that provides a TAP protocol version, followed by a test plan with
a count of items to be executed. That pattern should be followed at each
nesting level. In the current implementation of the top-most, i.e., test
suite level, those rules apply only for test suites built into the kernel,
executed and reported on boot. Results submitted to dmesg from kunit test
modules loaded later are missing those top-level headers.
As a consequence, if a kunit test module provides more than one test suite
then, without the top level test plan, external tools that are parsing
dmesg for kunit test output are not able to tell how many test suites
should be expected and whether to continue parsing after complete output
from the first test suite is collected.
Submit the top-level headers also from the kunit test module notifier
initialization callback.
[1] https://docs.kernel.org/dev-tools/ktap.html#
Signed-off-by: Janusz Krzysztofik <janusz.krzysztofik@linux.intel.com>
---
lib/kunit/test.c | 5 +++++
1 file changed, 5 insertions(+)
Comments
Em Mon, 31 Jul 2023 16:10:23 +0200 Janusz Krzysztofik <janusz.krzysztofik@linux.intel.com> escreveu: > According to KTAP specification[1], results should always start from a > header that provides a TAP protocol version, followed by a test plan with > a count of items to be executed. That pattern should be followed at each > nesting level. In the current implementation of the top-most, i.e., test > suite level, those rules apply only for test suites built into the kernel, > executed and reported on boot. Results submitted to dmesg from kunit test > modules loaded later are missing those top-level headers. > > As a consequence, if a kunit test module provides more than one test suite > then, without the top level test plan, external tools that are parsing > dmesg for kunit test output are not able to tell how many test suites > should be expected and whether to continue parsing after complete output > from the first test suite is collected. > > Submit the top-level headers also from the kunit test module notifier > initialization callback. > > [1] https://docs.kernel.org/dev-tools/ktap.html# > > Signed-off-by: Janusz Krzysztofik <janusz.krzysztofik@linux.intel.com> > --- > lib/kunit/test.c | 5 +++++ > 1 file changed, 5 insertions(+) > > diff --git a/lib/kunit/test.c b/lib/kunit/test.c > index 84e4666555c94..a29ca1acc4d81 100644 > --- a/lib/kunit/test.c > +++ b/lib/kunit/test.c > @@ -729,6 +729,11 @@ EXPORT_SYMBOL_GPL(__kunit_test_suites_exit); > #ifdef CONFIG_MODULES > static void kunit_module_init(struct module *mod) > { > + if (mod->num_kunit_suites > 0) { > + pr_info("KTAP version 1\n"); > + pr_info("1..%d\n", mod->num_kunit_suites); > + } > + > __kunit_test_suites_init(mod->kunit_suites, mod->num_kunit_suites); > } IMO, the best would be instead to export kunit_exec_run_tests() and use it here too. Except for the nit, LGTM. Thanks, Mauro
Hi Mauro, Thanks for review. On Tuesday, 1 August 2023 15:17:11 CEST Mauro Carvalho Chehab wrote: > Em Mon, 31 Jul 2023 16:10:23 +0200 > Janusz Krzysztofik <janusz.krzysztofik@linux.intel.com> escreveu: > > > According to KTAP specification[1], results should always start from a > > header that provides a TAP protocol version, followed by a test plan with > > a count of items to be executed. That pattern should be followed at each > > nesting level. In the current implementation of the top-most, i.e., test > > suite level, those rules apply only for test suites built into the kernel, > > executed and reported on boot. Results submitted to dmesg from kunit test > > modules loaded later are missing those top-level headers. > > > > As a consequence, if a kunit test module provides more than one test suite > > then, without the top level test plan, external tools that are parsing > > dmesg for kunit test output are not able to tell how many test suites > > should be expected and whether to continue parsing after complete output > > from the first test suite is collected. > > > > Submit the top-level headers also from the kunit test module notifier > > initialization callback. > > > > [1] https://docs.kernel.org/dev-tools/ktap.html# > > > > Signed-off-by: Janusz Krzysztofik <janusz.krzysztofik@linux.intel.com> > > --- > > lib/kunit/test.c | 5 +++++ > > 1 file changed, 5 insertions(+) > > > > diff --git a/lib/kunit/test.c b/lib/kunit/test.c > > index 84e4666555c94..a29ca1acc4d81 100644 > > --- a/lib/kunit/test.c > > +++ b/lib/kunit/test.c > > @@ -729,6 +729,11 @@ EXPORT_SYMBOL_GPL(__kunit_test_suites_exit); > > #ifdef CONFIG_MODULES > > static void kunit_module_init(struct module *mod) > > { > > + if (mod->num_kunit_suites > 0) { > > + pr_info("KTAP version 1\n"); > > + pr_info("1..%d\n", mod->num_kunit_suites); > > + } > > + > > __kunit_test_suites_init(mod->kunit_suites, mod->num_kunit_suites); > > } > > IMO, the best would be instead to export kunit_exec_run_tests() and > use it here too. I was considering a similar approach, i.e., moving those two pr_info() lines from built-in only kunit_exec_run_tests() to __kunit_test_suites_init() which is common to both built-in and modular paths, but please note that with kunit built in, an empty test plan "1..0" is now reported on boot, while we don't want similar reports to appear on loading modules that don't provide any kunit tests. Then, inside either your exported kunit_exec_run_tests() or my __kunit_test_suites_init(), we would have to check somehow if it has been called from a module notifier initialization callback, and that seemed to me too much complicated and less clean than what I've proposed: keep using unmodified kunit_exec_run_tests() in built-in and updated kunit_module_init() in modular processing path. Dropping the empty "1..0" test plan from boot messages would mean an ABI change, I believe, which I'd rather avoid adding to the scope of this patch as not required. Thanks, Janusz > > Except for the nit, LGTM. > > > Thanks, > Mauro >
On Mon, Jul 31, 2023 at 10:12 AM Janusz Krzysztofik <janusz.krzysztofik@linux.intel.com> wrote: > > According to KTAP specification[1], results should always start from a > header that provides a TAP protocol version, followed by a test plan with > a count of items to be executed. That pattern should be followed at each > nesting level. In the current implementation of the top-most, i.e., test > suite level, those rules apply only for test suites built into the kernel, > executed and reported on boot. Results submitted to dmesg from kunit test > modules loaded later are missing those top-level headers. > > As a consequence, if a kunit test module provides more than one test suite > then, without the top level test plan, external tools that are parsing > dmesg for kunit test output are not able to tell how many test suites > should be expected and whether to continue parsing after complete output > from the first test suite is collected. > > Submit the top-level headers also from the kunit test module notifier > initialization callback. > > [1] https://docs.kernel.org/dev-tools/ktap.html# > > Signed-off-by: Janusz Krzysztofik <janusz.krzysztofik@linux.intel.com> > --- Hi! I think this is a really great idea to improve the KTAP compatibility for module output. I do agree with Mauro and I wonder if this could be replaced with using kunit_exec_run_tests. However, if the output of 1..0 for a module with no KUnit tests run is not wanted, I am ok with this change as is. LGTM. Tested-by: Rae Moar <rmoar@google.com> -Rae > lib/kunit/test.c | 5 +++++ > 1 file changed, 5 insertions(+) > > diff --git a/lib/kunit/test.c b/lib/kunit/test.c > index 84e4666555c94..a29ca1acc4d81 100644 > --- a/lib/kunit/test.c > +++ b/lib/kunit/test.c > @@ -729,6 +729,11 @@ EXPORT_SYMBOL_GPL(__kunit_test_suites_exit); > #ifdef CONFIG_MODULES > static void kunit_module_init(struct module *mod) > { > + if (mod->num_kunit_suites > 0) { > + pr_info("KTAP version 1\n"); > + pr_info("1..%d\n", mod->num_kunit_suites); > + } > + > __kunit_test_suites_init(mod->kunit_suites, mod->num_kunit_suites); > } > > -- > 2.41.0 > > -- > You received this message because you are subscribed to the Google Groups "KUnit Development" group. > To unsubscribe from this group and stop receiving emails from it, send an email to kunit-dev+unsubscribe@googlegroups.com. > To view this discussion on the web visit https://groups.google.com/d/msgid/kunit-dev/20230731141021.2854827-6-janusz.krzysztofik%40linux.intel.com.
Hi Rae, On Thursday, 3 August 2023 22:57:43 CEST Rae Moar wrote: > On Mon, Jul 31, 2023 at 10:12 AM Janusz Krzysztofik > <janusz.krzysztofik@linux.intel.com> wrote: > > > > According to KTAP specification[1], results should always start from a > > header that provides a TAP protocol version, followed by a test plan with > > a count of items to be executed. That pattern should be followed at each > > nesting level. In the current implementation of the top-most, i.e., test > > suite level, those rules apply only for test suites built into the kernel, > > executed and reported on boot. Results submitted to dmesg from kunit test > > modules loaded later are missing those top-level headers. > > > > As a consequence, if a kunit test module provides more than one test suite > > then, without the top level test plan, external tools that are parsing > > dmesg for kunit test output are not able to tell how many test suites > > should be expected and whether to continue parsing after complete output > > from the first test suite is collected. > > > > Submit the top-level headers also from the kunit test module notifier > > initialization callback. > > > > [1] https://docs.kernel.org/dev-tools/ktap.html# > > > > Signed-off-by: Janusz Krzysztofik <janusz.krzysztofik@linux.intel.com> > > --- > > Hi! > > I think this is a really great idea to improve the KTAP compatibility > for module output. I do agree with Mauro and I wonder if this could be > replaced with using kunit_exec_run_tests. However, if the output of > 1..0 for a module with no KUnit tests run is not wanted, I do believe we really don't want that. As soon as kunit framework registers its notifier callbacks, those callbacks are executed by generic module handling code on load / unload of every module, not only those providing kunit tests. If our module initialization callback called unmodified kunit_exec_run_tests() that deliberately prints these two lines unconditionally: KTAP version 1 1..n then there would be a lot of unnecessary noise. To avoid that noise, I decided to teach the callback itself to display the header with the number of test suits provided by the module before processing them if there is at least one, and be silent otherwise. But since both you and Mauro think that kunit_exec_run_tests() should be reused, I can do that by moving that logic to kunit_exec_run_tests() and passing an additional flag that controls that logic from kunit_module_init() to kunit_exec_run_tests(). Would that approach be more acceptable? > I am ok with > this change as is. > > LGTM. > > Tested-by: Rae Moar <rmoar@google.com> Thank you for testing. Janusz > > -Rae > > > lib/kunit/test.c | 5 +++++ > > 1 file changed, 5 insertions(+) > > > > diff --git a/lib/kunit/test.c b/lib/kunit/test.c > > index 84e4666555c94..a29ca1acc4d81 100644 > > --- a/lib/kunit/test.c > > +++ b/lib/kunit/test.c > > @@ -729,6 +729,11 @@ EXPORT_SYMBOL_GPL(__kunit_test_suites_exit); > > #ifdef CONFIG_MODULES > > static void kunit_module_init(struct module *mod) > > { > > + if (mod->num_kunit_suites > 0) { > > + pr_info("KTAP version 1\n"); > > + pr_info("1..%d\n", mod->num_kunit_suites); > > + } > > + > > __kunit_test_suites_init(mod->kunit_suites, mod- >num_kunit_suites); > > } > > > > -- > > 2.41.0 > > > > -- > > You received this message because you are subscribed to the Google Groups "KUnit Development" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to kunit-dev+unsubscribe@googlegroups.com. > > To view this discussion on the web visit https://groups.google.com/d/ msgid/kunit-dev/20230731141021.2854827-6-janusz.krzysztofik%40linux.intel.com. >
diff --git a/lib/kunit/test.c b/lib/kunit/test.c index 84e4666555c94..a29ca1acc4d81 100644 --- a/lib/kunit/test.c +++ b/lib/kunit/test.c @@ -729,6 +729,11 @@ EXPORT_SYMBOL_GPL(__kunit_test_suites_exit); #ifdef CONFIG_MODULES static void kunit_module_init(struct module *mod) { + if (mod->num_kunit_suites > 0) { + pr_info("KTAP version 1\n"); + pr_info("1..%d\n", mod->num_kunit_suites); + } + __kunit_test_suites_init(mod->kunit_suites, mod->num_kunit_suites); }