Message ID | 20230610005149.1145665-1-rmoar@google.com |
---|---|
Headers |
Return-Path: <linux-kernel-owner@vger.kernel.org> Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp1277246vqr; Fri, 9 Jun 2023 17:52:50 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6wdlQNs786aq0xIRJQaNez89X6vo/QCvohx+V38pFMZhGM79JKubNGTGcvoejttI5Wlxyb X-Received: by 2002:a05:6a00:14c2:b0:658:12ca:385b with SMTP id w2-20020a056a0014c200b0065812ca385bmr3101765pfu.22.1686358370656; Fri, 09 Jun 2023 17:52:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1686358370; cv=none; d=google.com; s=arc-20160816; b=VHaCwdkVklxjmX34rCKDoScDSh+s/3RsvAkpr+Z27qGzUwF7sJcr1bVoPNgWAxVGHA begmWAVH8rdtP8NHH/MY5FXiKNudXX0FhMhZXmxJsTjrDpkZlaE4JhWN+YdtGTaezGFu UK+M2XirhQsKKiikJyGeNhiGhZYM46WlM9umCZ54dYXwxzzAs8zYNw3pHyL0yktyIWTM i45kAaFwmHNuN9mQDeuJbSuGG4Ae5Ov/66L8DR8YQU3w6ZxSA3EnIXPJNH385DN7ugyW 8TvSJ5Yln0AQV/FRBoReTJ8WxZqXHCmTbXM2sQJqPysyBTAq8bqXPU5yj3+803yIczLw BrGg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:mime-version:date :dkim-signature; bh=Wg++2gE8/jht9e+KkYaoylKtJ1FY+uAMq5VBMWkg2vI=; b=T4GEqCrq6U3P6X3C23lXrJYMmUt9BARxOUxWAw9AFtK9IqtYUJh5EeLNkzCU/2W/Fx KWaQJCXcHxu5sj4ZilmKXn72Q4LZPTiNAf9cr8x4r0Bx1pDtXgaOgpciqPadtrp98IdT Bj4L7uIJxh8NwQYs4S5xFBCztv2B5f3NVVMygnblREjn34iXmyjZGQ/mv/RdTZLMzLcO 4Jaw8ydEaXEai/sVckAhu7WFXoybCzU4k87KLUOgAzqQgEP1W13Kdqz+ukt94OPQ/n4I Kjja4jkejLFZJ9+iI2VD2htilUTbom/ZVSxF/kcKoRvDkgpPoUlaigPcoJjQ9xT3Mapq jydQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=vZa0uQJu; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id bs8-20020a632808000000b0054b4670f5e0si460063pgb.39.2023.06.09.17.52.32; Fri, 09 Jun 2023 17:52:50 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=vZa0uQJu; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230295AbjFJAwK (ORCPT <rfc822;liningstudo@gmail.com> + 99 others); Fri, 9 Jun 2023 20:52:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52222 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231154AbjFJAwI (ORCPT <rfc822;linux-kernel@vger.kernel.org>); Fri, 9 Jun 2023 20:52:08 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 77C1C19D for <linux-kernel@vger.kernel.org>; Fri, 9 Jun 2023 17:52:07 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-ba8338f20bdso3214190276.0 for <linux-kernel@vger.kernel.org>; Fri, 09 Jun 2023 17:52:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686358326; x=1688950326; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=Wg++2gE8/jht9e+KkYaoylKtJ1FY+uAMq5VBMWkg2vI=; b=vZa0uQJub1/SthRtIpRIUdBcECAOCf4iTZtfveOCEH2qhZXN6OXSLQU4rxA4XAuWzc rrF2wOzq4rqLr6cV80saM9nYxWLvo2dzC2QqLpKrmAn1QFps3fMUVaLZpX+hs6doupWf +pcjaEgSTskPgaJ+dIkjcli6C7/q6N210FLnQScBFO/2EXIWNa4dYsshbKQ4hJK7OqOp K3eLXaGxkOgCzqdEkUJPZgzhf8YRFJ4BmCoXVPCvTSlou9mIQf835BE68v9uc1K8AR9K naQ/WX8z5MCpfalOCjg4+ro4EyFf4vZOzWQX2gmnoKO08v9Kvq1jPYFGpDGQW6QQZ9aL 30zw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686358326; x=1688950326; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=Wg++2gE8/jht9e+KkYaoylKtJ1FY+uAMq5VBMWkg2vI=; b=GlCtfm5ux38ZcYdM5jt9kPErNc6ZUEPbTyNDCHmUDMaTmfp169i895LFbKDI2qLL3w elvcSJvBzYUXEecItpHIx1q+sW2E8oFFR1NNYIs5qc/z2bTTwEEy7dOUNQpTwZfk0v+t 1/XR9KRVlxQE7DCl616xEAkWXvjeKwooDvav6wYP701GKKmTu2BfDX05/Ga+YGxhBRZ8 Dfb+i/xZ4X43qlUTrGy3ba70qiWvSmN9m4chg/0eLmDLYEAjJge6Gp3yRLJ/YiA/xqF8 rfFpAFEa2QJYQ7GOaHycXneJszhoic6cJTxXauep3uyS8jQEQepPiBOA1KB3wR6cR84F yppw== X-Gm-Message-State: AC+VfDxXyx2lDlvQLHXjDdEticbVQU2/bW6J9DZNJ2T62k+PfEtyCFYq nWwna+rA9UqJEomXeLgcjcsWoT4pfA== X-Received: from rmoar-specialist.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:45d3]) (user=rmoar job=sendgmr) by 2002:a5b:b4a:0:b0:bac:f608:7113 with SMTP id b10-20020a5b0b4a000000b00bacf6087113mr1547448ybr.4.1686358326518; Fri, 09 Jun 2023 17:52:06 -0700 (PDT) Date: Sat, 10 Jun 2023 00:51:43 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230610005149.1145665-1-rmoar@google.com> Subject: [RFC v1 0/6] kunit: Add test attributes API From: Rae Moar <rmoar@google.com> To: shuah@kernel.org, davidgow@google.com, dlatypov@google.com, brendan.higgins@linux.dev Cc: linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, linux-kernel@vger.kernel.org, keescook@chromium.org, linux-hardening@vger.kernel.org, jstultz@google.com, tglx@linutronix.de, sboyd@kernel.org, Rae Moar <rmoar@google.com> Content-Type: text/plain; charset="UTF-8" X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED, USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: <linux-kernel.vger.kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1768274914721160179?= X-GMAIL-MSGID: =?utf-8?q?1768274914721160179?= |
Series |
kunit: Add test attributes API
|
|
Message
Rae Moar
June 10, 2023, 12:51 a.m. UTC
Hello everyone, This is an RFC patch series to propose the addition of a test attributes framework to KUnit. There has been interest in filtering out "slow" KUnit tests. Most notably, a new config, CONFIG_MEMCPY_SLOW_KUNIT_TEST, has been added to exclude particularly slow memcpy tests (https://lore.kernel.org/all/20230118200653.give.574-kees@kernel.org/). This proposed attributes framework would be used to save and access test associated data, including whether a test is slow. These attributes would be reportable (via KTAP and command line output) and some will be filterable. This framework is designed to allow for the addition of other attributes in the future. These attributes could include whether the test is flaky, associated test files, etc. Note that this could intersect with the discussions on how to format test-associated data in KTAP v2 that I am also involved in (https://lore.kernel.org/all/20230420205734.1288498-1-rmoar@google.com/). If the overall idea seems good, I'll make sure to add tests/documentation, and more patches marking existing tests as slow to the patch series. Thanks! Rae Rae Moar (6): kunit: Add test attributes API structure kunit: Add speed attribute kunit: Add ability to filter attributes kunit: tool: Add command line interface to filter and report attributes kunit: memcpy: Mark tests as slow using test attributes kunit: time: Mark test as slow using test attributes include/kunit/attributes.h | 41 ++++ include/kunit/test.h | 62 ++++++ kernel/time/time_test.c | 2 +- lib/kunit/Makefile | 3 +- lib/kunit/attributes.c | 280 +++++++++++++++++++++++++ lib/kunit/executor.c | 89 ++++++-- lib/kunit/executor_test.c | 8 +- lib/kunit/kunit-example-test.c | 9 + lib/kunit/test.c | 17 +- lib/memcpy_kunit.c | 8 +- tools/testing/kunit/kunit.py | 34 ++- tools/testing/kunit/kunit_kernel.py | 6 +- tools/testing/kunit/kunit_tool_test.py | 41 ++-- 13 files changed, 536 insertions(+), 64 deletions(-) create mode 100644 include/kunit/attributes.h create mode 100644 lib/kunit/attributes.c base-commit: fefdb43943c1a0d87e1b43ae4d03e5f9a1d058f4
Comments
On Sat, 10 Jun 2023 at 08:52, Rae Moar <rmoar@google.com> wrote: > > Hello everyone, > > This is an RFC patch series to propose the addition of a test attributes > framework to KUnit. > > There has been interest in filtering out "slow" KUnit tests. Most notably, > a new config, CONFIG_MEMCPY_SLOW_KUNIT_TEST, has been added to exclude > particularly slow memcpy tests > (https://lore.kernel.org/all/20230118200653.give.574-kees@kernel.org/). Awesome: this is a long overdue feature. > This proposed attributes framework would be used to save and access test > associated data, including whether a test is slow. These attributes would > be reportable (via KTAP and command line output) and some will be > filterable. Why wouldn't they all be filterable? I guess I can imagine some where filtering wouldn't be useful, but I can't think of a technical reason why the filter shouldn't work. Also, as I understand it, I think this could also work with data which is not "saved" in this kunit_attributes struct you define. So we could have attributes which are generated automatically from other information about the test. I could definitely see value in being able to filter on things like "is_parameterised" or "runs_at_init" or similar. Finally, it'd be really great if these attributes could apply to individual parameters in parameterised tests, in which case we could have and filter on the parameter value. That seems like it could be incredibly useful. > > This framework is designed to allow for the addition of other attributes in > the future. These attributes could include whether the test is flaky, > associated test files, etc. A small part of me is hesitant to add this much framework code for only one attribute, so it'd be nice to look into at least having an RFC for some of these. Hopefully we don't actually have flaky tests, but "is_deterministic" would be nice (alongside a future ability to inject a random seed, or similar). Other properties like "is_threadsafe", "is_reentrant", etc could be useful for future features. And I'm sure there could be some more subsystem-specific things which would be useful to filter on, too. Some of these could probably replace the need for custom code to make the test skip itself if dependencies aren't met, too, which would be fun. I'm not sure "associated test files" quite gels perfectly with this system as-is (assuming I understand what that refers to). If it's the ability to "attach" extra data (logs, etc) to the KTAP output, that's possibly best injected at runtime, or added by the separate parser, rather than hardcoded in the kernel. > > Note that this could intersect with the discussions on how to format > test-associated data in KTAP v2 that I am also involved in > (https://lore.kernel.org/all/20230420205734.1288498-1-rmoar@google.com/). > I definitely need to re-read and respond to that. I'm not 100% thrilled with the output format here, and I think the goal with KTAP "test associated data" is, as you say, related, but not identical to this. There'd definitely be data which doesn't make sense as a KUnit attribute which we might want to add to the output (e.g., data only calcuated while the test runs), and attributes which we might not want to always print out with the results. > If the overall idea seems good, I'll make sure to add tests/documentation, > and more patches marking existing tests as slow to the patch series. > I think the series is good overall. If no-one else objects, let's move forward with it. I'd definitely prefer to see a few more tests and some documentation. Having another attribute would be great, too, though I can certainly live with that being a separate series. > Thanks! > Rae > > Rae Moar (6): > kunit: Add test attributes API structure > kunit: Add speed attribute > kunit: Add ability to filter attributes > kunit: tool: Add command line interface to filter and report > attributes > kunit: memcpy: Mark tests as slow using test attributes > kunit: time: Mark test as slow using test attributes > > include/kunit/attributes.h | 41 ++++ > include/kunit/test.h | 62 ++++++ > kernel/time/time_test.c | 2 +- > lib/kunit/Makefile | 3 +- > lib/kunit/attributes.c | 280 +++++++++++++++++++++++++ > lib/kunit/executor.c | 89 ++++++-- > lib/kunit/executor_test.c | 8 +- > lib/kunit/kunit-example-test.c | 9 + > lib/kunit/test.c | 17 +- > lib/memcpy_kunit.c | 8 +- > tools/testing/kunit/kunit.py | 34 ++- > tools/testing/kunit/kunit_kernel.py | 6 +- > tools/testing/kunit/kunit_tool_test.py | 41 ++-- > 13 files changed, 536 insertions(+), 64 deletions(-) > create mode 100644 include/kunit/attributes.h > create mode 100644 lib/kunit/attributes.c > > > base-commit: fefdb43943c1a0d87e1b43ae4d03e5f9a1d058f4 > -- > 2.41.0.162.gfafddb0af9-goog >
On Sat, Jun 10, 2023 at 4:29 AM David Gow <davidgow@google.com> wrote: > > On Sat, 10 Jun 2023 at 08:52, Rae Moar <rmoar@google.com> wrote: > > > > Hello everyone, > > > > This is an RFC patch series to propose the addition of a test attributes > > framework to KUnit. > > > > There has been interest in filtering out "slow" KUnit tests. Most notably, > > a new config, CONFIG_MEMCPY_SLOW_KUNIT_TEST, has been added to exclude > > particularly slow memcpy tests > > (https://lore.kernel.org/all/20230118200653.give.574-kees@kernel.org/). > > Awesome: this is a long overdue feature. > Hi David! Thank you for all the comments! > > This proposed attributes framework would be used to save and access test > > associated data, including whether a test is slow. These attributes would > > be reportable (via KTAP and command line output) and some will be > > filterable. > > Why wouldn't they all be filterable? I guess I can imagine some where > filtering wouldn't be useful, but I can't think of a technical reason > why the filter shouldn't work. I am definitely open to all attributes being filterable. My reservation is that I can imagine an attribute with a complex data type that would cause the filtering method to be difficult to implement. If the attribute does not benefit much from being filterable, I wonder if it is worth requiring the filtering method to be implemented in that case. Perhaps for now all attributes are filterable and then if this becomes the case, this is addressed then? > > Also, as I understand it, I think this could also work with data which > is not "saved" in this kunit_attributes struct you define. So we could > have attributes which are generated automatically from other > information about the test. > I could definitely see value in being able to filter on things like > "is_parameterised" or "runs_at_init" or similar. > Yes! This is a great benefit of this flexible structure for attributes. I would definitely be interested in implementing "is_parameterised" and "runs_at_init" in future patches. > Finally, it'd be really great if these attributes could apply to > individual parameters in parameterised tests, in which case we could > have and filter on the parameter value. That seems like it could be > incredibly useful. > Yes, this would be an exciting extension for this project. I have started thinking about this as potentially a follow up project. > > > > This framework is designed to allow for the addition of other attributes in > > the future. These attributes could include whether the test is flaky, > > associated test files, etc. > > A small part of me is hesitant to add this much framework code for > only one attribute, so it'd be nice to look into at least having an > RFC for some of these. Hopefully we don't actually have flaky tests, > but "is_deterministic" would be nice (alongside a future ability to > inject a random seed, or similar). Other properties like > "is_threadsafe", "is_reentrant", etc could be useful for future > features. And I'm sure there could be some more subsystem-specific > things which would be useful to filter on, too. > I understand the reservations to add this large framework for one attribute. I would definitely consider adding an additional attribute to this RFC or creating a separate RFC. I would be happy to go ahead and add "is_deterministic" if there is interest. As well as potentially "is_threadsafe", "is_reentrant", etc in future patches. > Some of these could probably replace the need for custom code to make > the test skip itself if dependencies aren't met, too, which would be > fun. This would be great! > > I'm not sure "associated test files" quite gels perfectly with this > system as-is (assuming I understand what that refers to). If it's the > ability to "attach" extra data (logs, etc) to the KTAP output, that's > possibly best injected at runtime, or added by the separate parser, > rather than hardcoded in the kernel. > Hmm I see what you are saying here. If the associated test files of interest are best injected at runtime I am happy to scrap this idea for now. > > > > Note that this could intersect with the discussions on how to format > > test-associated data in KTAP v2 that I am also involved in > > (https://lore.kernel.org/all/20230420205734.1288498-1-rmoar@google.com/). > > > I definitely need to re-read and respond to that. I'm not 100% > thrilled with the output format here, and I think the goal with KTAP > "test associated data" is, as you say, related, but not identical to > this. There'd definitely be data which doesn't make sense as a KUnit > attribute which we might want to add to the output (e.g., data only > calcuated while the test runs), and attributes which we might not want > to always print out with the results. > I have thought much about the differences between the two concepts. My current understanding with KTAP metadata and KUnit attributes is that they are not going to be perfect mirrors of each other but the KUnit attributes framework can help to save and output some of the KTAP metadata. I am happy to be flexible on the output format or see more discussion on KTAP metadata in general. What part of the output is most concerning? > > If the overall idea seems good, I'll make sure to add tests/documentation, > > and more patches marking existing tests as slow to the patch series. > > > > I think the series is good overall. If no-one else objects, let's move > forward with it. > I'd definitely prefer to see a few more tests and some documentation. > Having another attribute would be great, too, though I can certainly > live with that being a separate series. Great! Yes I will add documentation and more tests in the next versions. I will also work on the implementation of another attribute. > > > > Thanks! > > Rae > > > > Rae Moar (6): > > kunit: Add test attributes API structure > > kunit: Add speed attribute > > kunit: Add ability to filter attributes > > kunit: tool: Add command line interface to filter and report > > attributes > > kunit: memcpy: Mark tests as slow using test attributes > > kunit: time: Mark test as slow using test attributes > > > > include/kunit/attributes.h | 41 ++++ > > include/kunit/test.h | 62 ++++++ > > kernel/time/time_test.c | 2 +- > > lib/kunit/Makefile | 3 +- > > lib/kunit/attributes.c | 280 +++++++++++++++++++++++++ > > lib/kunit/executor.c | 89 ++++++-- > > lib/kunit/executor_test.c | 8 +- > > lib/kunit/kunit-example-test.c | 9 + > > lib/kunit/test.c | 17 +- > > lib/memcpy_kunit.c | 8 +- > > tools/testing/kunit/kunit.py | 34 ++- > > tools/testing/kunit/kunit_kernel.py | 6 +- > > tools/testing/kunit/kunit_tool_test.py | 41 ++-- > > 13 files changed, 536 insertions(+), 64 deletions(-) > > create mode 100644 include/kunit/attributes.h > > create mode 100644 lib/kunit/attributes.c > > > > > > base-commit: fefdb43943c1a0d87e1b43ae4d03e5f9a1d058f4 > > -- > > 2.41.0.162.gfafddb0af9-goog > >