From patchwork Fri Jul 7 21:09:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rae Moar X-Patchwork-Id: 11755 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f45:0:b0:3ea:f831:8777 with SMTP id v5csp3553953vqx; Fri, 7 Jul 2023 14:41:31 -0700 (PDT) X-Google-Smtp-Source: APBJJlHSf99q+pw4Y94kJbMlmZE5IZLuiajMpAvm7624yHrLBgQyOt7Azk5B6BmCy/RlSZpf8SJ3 X-Received: by 2002:a05:6402:68e:b0:51d:a2d9:85a3 with SMTP id f14-20020a056402068e00b0051da2d985a3mr4946770edy.8.1688766090884; Fri, 07 Jul 2023 14:41:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1688766090; cv=none; d=google.com; s=arc-20160816; b=xoOSvcD1k9/OzhP2k0WE6FY7m5jY/PjmTH0pNwG9h/EdP3yp1MKm58mzlPpw4ZKZkR 6xC6U8Lx5IrMhpJYuoVzMHmWKFdMho1gokVa43jSg8XeG6T0fRbDbrSHCzeSqFQ/K7Sv LvKvahgaNP8K+rCr0fugpIufXKjsNfFU3nuVrEnXDEeNeYFCaPgrmK1TPkP33MUDMnVy 0Wy5sQN7thekRnF8FA6GXu96uyVF68Lu+DuUwlOt27nV/riVy9ZvusBzNxAzhfzfDiYt Fk+CLE8E++g9FnBsbKuP+QJ1d7LWnLyMJ+gLIwkbYHKCzMFgj5NwEbe9unNR7a94XYeM b0RA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:mime-version:date :dkim-signature; bh=ZTrWAIUayaBHwDRCQHgGm0qH9ucv7JkIRMB8GVQtyc8=; fh=eSOGplXNKm0TQlVMiAPjma3ZxjxuDQ0l9LD5ETEQPUY=; b=yE7Cz7xhkO3+zPg9K1Ftb6grDWHQkmz3dM21h8hmarpSSDPHk+aVJ4yOS4pwA/PY08 imFVYVHDAQ7ai9m6bUW/I1OhAbmj5MNNtmcWPC7YRO83eJhaDg9X4D5n+HwcmZnMjrdY q2CGaMTy0jnXsD4D2QO0BQuuWrPiTw1vP++HGir8gxBSo9VpAFGClMedTu5GCFeX13iV B8uvW/3WawwFoU+bInTkKlLmp4td0Wi7UHfotRHjKdW6kX/srO2kfYZkIS4/KawxqkN1 58ASt8BBI+fWsWZhiBA6bMuqAeE5Gs++AqHBok/TwCeL2vSWh2pDGeCdzOFJ6hoUJASM Ql5w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=pmsOikxb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id bf6-20020a0564021a4600b0051de2ee33fdsi2706468edb.433.2023.07.07.14.41.03; Fri, 07 Jul 2023 14:41:30 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=pmsOikxb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229515AbjGGVJz (ORCPT + 99 others); Fri, 7 Jul 2023 17:09:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59892 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229471AbjGGVJy (ORCPT ); Fri, 7 Jul 2023 17:09:54 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D56CC1BD2 for ; Fri, 7 Jul 2023 14:09:52 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-c5cea5773e8so2626574276.1 for ; Fri, 07 Jul 2023 14:09:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1688764192; x=1691356192; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=ZTrWAIUayaBHwDRCQHgGm0qH9ucv7JkIRMB8GVQtyc8=; b=pmsOikxbpPPKi6SHT/ZT9CcQCmKqH1TPbAL1YPRN+/EvfSYqZWCf7863xGKzcQw3sc nGY1ZvpN6V73iMT5yLIJSRHdqLgxPw7aJYWkMGrfhzlOyvibq/HcsR9w/z4z/VyaZgfp n91L4Uj89hsFLFnir4XCaN/Z43f4e72guBYeLJrmAbIU1n4ebxgu/frUIpm4/JM2GMSR ac/7WzQi4sFjz0Kq6KC8XUhUlg+jcGnjcx5QuqG/cGR+5WgczLyULdXlRnx//5JSW+Hl ZEmlEvSZcKJg4Loxr8JjWRoh0Iz1sAtEdqzc6s3cCGH4jMGc1xAdc2VL8/SEYbnBCp8K Rppw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688764192; x=1691356192; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=ZTrWAIUayaBHwDRCQHgGm0qH9ucv7JkIRMB8GVQtyc8=; b=XqP0tKXqsv0OxO9eMcAXjbcdyXodipkLGnIjQ9YkNmeCZw3+1Ax+SyaltWEXAxjU+u KG64ocAufdYFJurPmYt6omHtIWmPXR8i/69J7vW5bkaBP89ZRZTlhXNkLxn82p3ubv5u kqwEsceUVxW59XL1b7TLJyQWoeX7Bx0u4sPOIXtO1G99TTpL6jPSwc0ZkjZ6SzlsvS0R cDwYpWBlfVn94/mY8p76d0J3RV+bHG58iz91C+R5Smmxg0LFb/ypuzfHhG0y5cfWWi1P 2S8+v0uyAWZrnYGTwhZH/wE/FK7BSpGulwjQPLqW7oktN2C9vPduEzPG82zCmysjv/DP zZ1w== X-Gm-Message-State: ABy/qLbsQcJg4Ml/BWdvHsMOiSCAs2TL/h5eR63Y6mOiaxN8PFM6RKVL oDySo8H3yIhwygqebsUc2m1zKOzwHg== X-Received: from rmoar-specialist.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:45d3]) (user=rmoar job=sendgmr) by 2002:a25:ae8a:0:b0:c5f:85f5:a0e5 with SMTP id b10-20020a25ae8a000000b00c5f85f5a0e5mr56453ybj.5.1688764192066; Fri, 07 Jul 2023 14:09:52 -0700 (PDT) Date: Fri, 7 Jul 2023 21:09:38 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230707210947.1208717-1-rmoar@google.com> Subject: [RFC v2 0/9] kunit: Add test attributes API From: Rae Moar To: shuah@kernel.org, davidgow@google.com, dlatypov@google.com, brendan.higgins@linux.dev Cc: linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, linux-kernel@vger.kernel.org, keescook@chromium.org, linux-hardening@vger.kernel.org, jstultz@google.com, tglx@linutronix.de, sboyd@kernel.org, Rae Moar X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1770799592742237227?= X-GMAIL-MSGID: =?utf-8?q?1770799592742237227?= Hello everyone, This is an RFC patch series to propose the addition of a test attributes framework to KUnit. There has been interest in filtering out "slow" KUnit tests. Most notably, a new config, CONFIG_MEMCPY_SLOW_KUNIT_TEST, has been added to exclude a particularly slow memcpy test (https://lore.kernel.org/all/20230118200653.give.574-kees@kernel.org/). This proposed attributes framework would be used to save and access test associated data, including whether a test is slow. These attributes would be reportable (via KTAP and command line output) and some will be filterable. This framework is designed to allow for the addition of other attributes in the future. These attributes could include whether the test is flaky, associated test files, etc. This is the second version of the RFC I have added a few big changes: - Change method for inputting filters to allow for spaces in filtering values - Add option to skip filtered tests instead of not run or show them with the --filter_skip flag - Separate the new feature to list tests and their attributes into both --list_tests (lists just tests) and --list_tests_attr (lists all) - Add new attribute to store module name associated with test - Add Tests to executor_test.c - Add Documentation - A few small changes to code commented on previously I would love to hear about the new features. If the series seems overall good I will send out the next version as an official patch series. Thanks! Rae Rae Moar (9): kunit: Add test attributes API structure kunit: Add speed attribute kunit: Add module attribute kunit: Add ability to filter attributes kunit: tool: Add command line interface to filter and report attributes kunit: memcpy: Mark tests as slow using test attributes kunit: time: Mark test as slow using test attributes kunit: add tests for filtering attributes kunit: Add documentation of KUnit test attributes .../dev-tools/kunit/running_tips.rst | 163 +++++++ include/kunit/attributes.h | 50 +++ include/kunit/test.h | 68 ++- kernel/time/time_test.c | 2 +- lib/Kconfig.debug | 3 + lib/kunit/Makefile | 3 +- lib/kunit/attributes.c | 406 ++++++++++++++++++ lib/kunit/executor.c | 115 ++++- lib/kunit/executor_test.c | 119 ++++- lib/kunit/kunit-example-test.c | 9 + lib/kunit/test.c | 27 +- lib/memcpy_kunit.c | 8 +- tools/testing/kunit/kunit.py | 80 +++- tools/testing/kunit/kunit_kernel.py | 6 +- tools/testing/kunit/kunit_tool_test.py | 39 +- 15 files changed, 1022 insertions(+), 76 deletions(-) create mode 100644 include/kunit/attributes.h create mode 100644 lib/kunit/attributes.c base-commit: 2e66833579ed759d7b7da1a8f07eb727ec6e80db