From patchwork Mon May 8 13:15:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 91136 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2149531vqo; Mon, 8 May 2023 06:17:38 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ64yMvWcjTxIBvp6kgcUZFdXr+M1N3Acv9hTnFAI083C6+4QE/uS4tWX6PlexsV/qmUx2Xt X-Received: by 2002:a05:6a21:33a7:b0:ff:b84d:f1be with SMTP id yy39-20020a056a2133a700b000ffb84df1bemr8236412pzb.56.1683551857815; Mon, 08 May 2023 06:17:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683551857; cv=none; d=google.com; s=arc-20160816; b=wJJ31d4zCR5+LiDGzEEjke3Sf6c3YTy6A+G4AFW/E1JfDilfO/3969znckzOv+eAeW +C7SN2RQtHLgRtY5U9Db5IZr3WRkOPxZkHmxd4kqre3YW5a7lWe+xJcTvCnlK068O0Yd Hy8gj6dFxLAyOxUTIfPklt3Cub+UzWYhq0ol3pTLw3S3pCE0zVZ5+fkP8seMR5Bgty7X b79S5dJxA5vyd6Haya13+fGjaO4gi/3l29wwxo5C/i9k329rg+Zkcf9bzlOyXD9KlBOH wqNaYfvPguR06WJLqVdH6c9VQwKsnLLPL8EFEsIbhdDplz0HxFdgmwwnwtXea1PnjyPd DL4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:message-id:content-transfer-encoding :mime-version:subject:date:from:dkim-signature; bh=2U/yLGNXIBFNbB9nNfn+OVlSiVwxxquu5nz6SGDPtl4=; b=aBe7mwHZ967KmrHr6ipn2jS9v8KVfUd5hJ6vp6zV+J0v5bWcMhcCmdNG3jaDYEqP6C HwUPGW1XOW8m0otdA5PAwZgzsfF+g1aN+l+37J7QE4pKYMhVe4XVQyaTQ2b3AFFe4u0+ Lc0nIDag3iQppN2m8y8oiwS3Vfm64e6z+aCKjvOm9w11QkTtXYViG8mQUx/BH7fZ/3V/ DMHxvN7ptTyqAHGrv/1VpROu40LMQKNZ7V/OIIaDKg0qgnwHDRG1/P8cPUF4v1pwseNa 9s0OD7MmcHnM1e1rRZ+4UMvutIpbnV9a7deOcOd6LAb48SB4pOikbSVa63dh2VNCvki/ ze2g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=iewibZCA; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id s15-20020a63af4f000000b0052864b23dbasi8390473pgo.41.2023.05.08.06.17.25; Mon, 08 May 2023 06:17:37 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=iewibZCA; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233841AbjEHNP5 (ORCPT + 99 others); Mon, 8 May 2023 09:15:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33346 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233267AbjEHNPu (ORCPT ); Mon, 8 May 2023 09:15:50 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 636ED2D797; Mon, 8 May 2023 06:15:48 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id DE25B62D42; Mon, 8 May 2023 13:15:47 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9469AC433D2; Mon, 8 May 2023 13:15:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1683551747; bh=Yx/d/vlSfUg1A+0BfdmAMW65S4eMBXSw23P4c1Qdsuo=; h=From:Date:Subject:To:Cc:From; b=iewibZCA5YkX7wJ83jSl3W8R0qv8D7QY5v6Foef7NOoPbVys+aKiLmQsFIi057WBF vPXV8WD+H2T0/7H8XVXLqFOU0UOcCLb2HYNvJFyqJSbTWVVXleKZ0JmWZyE9AkYImI 42UuVnOVAavP2zUUR7BPEtF43uzD9CD08LZ3NmEpQDpbHpsCAkigTU6IS2hT6bOoK6 r79yUENbru7DXGzkUA5POfFRRvo88oQRoLjXlwdi67FUuxTQdC8IEM8qEfRUp9AK4q RE6XKQf7BRUYUiseog1RHrNwGvjjy8v90DzZnSdLPH2I+Zpt5nJl1by12t8LoGEGxI UCrusKl1xFVQA== From: Mark Brown Date: Mon, 08 May 2023 22:15:42 +0900 Subject: [PATCH v2] selftests/ftrace: Improve integration with kselftest runner MIME-Version: 1.0 Message-Id: <20230302-ftrace-kselftest-ktap-v2-1-ecc482212729@kernel.org> X-B4-Tracking: v=1; b=H4sIAP31WGQC/4WNQQqDMBAAvyI5d0sSq1FP/UfxsOqqQRtlE6RF/ HujH+hxBobZhSe25EWV7IJps94uLoK+JaId0Q0EtosstNSpTKWGPjC2BJOnuQ/kA0wBVyg7U5g iL1WWGRHbBj1Bw+ja8azDez3tytTbz3V71ZFH68PC32u+qdP++2wKFGDxQGnyrDHYPSdiR/N94 UHUx3H8AMmk61HRAAAA To: Steven Rostedt , Masami Hiramatsu , Shuah Khan Cc: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Mark Brown X-Mailer: b4 0.13-dev-bfdf5 X-Developer-Signature: v=1; a=openpgp-sha256; l=7774; i=broonie@kernel.org; h=from:subject:message-id; bh=Yx/d/vlSfUg1A+0BfdmAMW65S4eMBXSw23P4c1Qdsuo=; b=owEBbAGT/pANAwAKASTWi3JdVIfQAcsmYgBkWPYAg3dhyoXZ+y4LAlxspmifvOKuzUu+/DSKz CpmYMVjlNqJATIEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZFj2AAAKCRAk1otyXVSH 0OHwB/joun9Ln9Cw6ohgcgBT0CTtYOUHOJPFeqT96+uRYUcmXVaxZ2rpeXD/oBkxxqJvoHfNd/V Iqb+iWkk1Yy9lc1N9TAzxntQCNQZEnok1EkpyDVxlk7x25Z/qOBF6HdM22LlPdAzN0Nbs9Y64gL 05PtmxL10DbgCr8uwaL9NBz3d3EXKvWlK1jFj6UqhUNuAidS9zne5AGoV7OzAqAWgBuJ3prCLXV YwsqmO36bqWD1zc2dwZui4XK4KJrDbC/0h14cAbB1R+2tR3DgDE9YgfLQ0STE3e0jf+oYfzYRPb Y9vQ3lxujIiIscn2VVz6bew0Dide6jL/9OS6uLSySQ0KUrs= X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765332073071735309?= X-GMAIL-MSGID: =?utf-8?q?1765332073071735309?= The ftrace selftests do not currently produce KTAP output, they produce a custom format much nicer for human consumption. This means that when run in automated test systems we just get a single result for the suite as a whole rather than recording results for individual test cases, making it harder to look at the test data and masking things like inappropriate skips. Address this by adding support for KTAP output to the ftracetest script and providing a trivial wrapper which will be invoked by the kselftest runner to generate output in this format by default, users using ftracetest directly will continue to get the existing output. This is not the most elegant solution but it is simple and effective. I did consider implementing this by post processing the existing output format but that felt more complex and likely to result in all output being lost if something goes seriously wrong during the run which would not be helpful. I did also consider just writing a separate runner script but there's enough going on with things like the signal handling for that to seem like it would be duplicating too much. Acked-by: Steven Rostedt (Google) Acked-by: Masami Hiramatsu (Google) Tested-by: Masami Hiramatsu (Google) Signed-off-by: Mark Brown --- It might make sense to merge this via the ftrace tree - kselftests often get merged with the code they test. Changes in v2: - Rebase onto v6.4-rc1. - Link to v1: https://lore.kernel.org/r/20230302-ftrace-kselftest-ktap-v1-1-a84a0765b7ad@kernel.org --- tools/testing/selftests/ftrace/Makefile | 3 +- tools/testing/selftests/ftrace/ftracetest | 63 ++++++++++++++++++++++++-- tools/testing/selftests/ftrace/ftracetest-ktap | 8 ++++ 3 files changed, 70 insertions(+), 4 deletions(-) --- base-commit: ac9a78681b921877518763ba0e89202254349d1b change-id: 20230302-ftrace-kselftest-ktap-9d7878691557 Best regards, diff --git a/tools/testing/selftests/ftrace/Makefile b/tools/testing/selftests/ftrace/Makefile index d6e106fbce11..a1e955d2de4c 100644 --- a/tools/testing/selftests/ftrace/Makefile +++ b/tools/testing/selftests/ftrace/Makefile @@ -1,7 +1,8 @@ # SPDX-License-Identifier: GPL-2.0 all: -TEST_PROGS := ftracetest +TEST_PROGS_EXTENDED := ftracetest +TEST_PROGS := ftracetest-ktap TEST_FILES := test.d settings EXTRA_CLEAN := $(OUTPUT)/logs/* diff --git a/tools/testing/selftests/ftrace/ftracetest b/tools/testing/selftests/ftrace/ftracetest index c3311c8c4089..2506621e75df 100755 --- a/tools/testing/selftests/ftrace/ftracetest +++ b/tools/testing/selftests/ftrace/ftracetest @@ -13,6 +13,7 @@ echo "Usage: ftracetest [options] [testcase(s)] [testcase-directory(s)]" echo " Options:" echo " -h|--help Show help message" echo " -k|--keep Keep passed test logs" +echo " -K|--ktap Output in KTAP format" echo " -v|--verbose Increase verbosity of test messages" echo " -vv Alias of -v -v (Show all results in stdout)" echo " -vvv Alias of -v -v -v (Show all commands immediately)" @@ -85,6 +86,10 @@ parse_opts() { # opts KEEP_LOG=1 shift 1 ;; + --ktap|-K) + KTAP=1 + shift 1 + ;; --verbose|-v|-vv|-vvv) if [ $VERBOSE -eq -1 ]; then usage "--console can not use with --verbose" @@ -178,6 +183,7 @@ TEST_DIR=$TOP_DIR/test.d TEST_CASES=`find_testcases $TEST_DIR` LOG_DIR=$TOP_DIR/logs/`date +%Y%m%d-%H%M%S`/ KEEP_LOG=0 +KTAP=0 DEBUG=0 VERBOSE=0 UNSUPPORTED_RESULT=0 @@ -229,7 +235,7 @@ prlog() { # messages newline= shift fi - printf "$*$newline" + [ "$KTAP" != "1" ] && printf "$*$newline" [ "$LOG_FILE" ] && printf "$*$newline" | strip_esc >> $LOG_FILE } catlog() { #file @@ -260,11 +266,11 @@ TOTAL_RESULT=0 INSTANCE= CASENO=0 +CASENAME= testcase() { # testfile CASENO=$((CASENO+1)) - desc=`grep "^#[ \t]*description:" $1 | cut -f2- -d:` - prlog -n "[$CASENO]$INSTANCE$desc" + CASENAME=`grep "^#[ \t]*description:" $1 | cut -f2- -d:` } checkreq() { # testfile @@ -277,40 +283,68 @@ test_on_instance() { # testfile grep -q "^#[ \t]*flags:.*instance" $1 } +ktaptest() { # result comment + if [ "$KTAP" != "1" ]; then + return + fi + + local result= + if [ "$1" = "1" ]; then + result="ok" + else + result="not ok" + fi + shift + + local comment=$* + if [ "$comment" != "" ]; then + comment="# $comment" + fi + + echo $CASENO $result $INSTANCE$CASENAME $comment +} + eval_result() { # sigval case $1 in $PASS) prlog " [${color_green}PASS${color_reset}]" + ktaptest 1 PASSED_CASES="$PASSED_CASES $CASENO" return 0 ;; $FAIL) prlog " [${color_red}FAIL${color_reset}]" + ktaptest 0 FAILED_CASES="$FAILED_CASES $CASENO" return 1 # this is a bug. ;; $UNRESOLVED) prlog " [${color_blue}UNRESOLVED${color_reset}]" + ktaptest 0 UNRESOLVED UNRESOLVED_CASES="$UNRESOLVED_CASES $CASENO" return $UNRESOLVED_RESULT # depends on use case ;; $UNTESTED) prlog " [${color_blue}UNTESTED${color_reset}]" + ktaptest 1 SKIP UNTESTED_CASES="$UNTESTED_CASES $CASENO" return 0 ;; $UNSUPPORTED) prlog " [${color_blue}UNSUPPORTED${color_reset}]" + ktaptest 1 SKIP UNSUPPORTED_CASES="$UNSUPPORTED_CASES $CASENO" return $UNSUPPORTED_RESULT # depends on use case ;; $XFAIL) prlog " [${color_green}XFAIL${color_reset}]" + ktaptest 1 XFAIL XFAILED_CASES="$XFAILED_CASES $CASENO" return 0 ;; *) prlog " [${color_blue}UNDEFINED${color_reset}]" + ktaptest 0 error UNDEFINED_CASES="$UNDEFINED_CASES $CASENO" return 1 # this must be a test bug ;; @@ -371,6 +405,7 @@ __run_test() { # testfile run_test() { # testfile local testname=`basename $1` testcase $1 + prlog -n "[$CASENO]$INSTANCE$CASENAME" if [ ! -z "$LOG_FILE" ] ; then local testlog=`mktemp $LOG_DIR/${CASENO}-${testname}-log.XXXXXX` else @@ -405,6 +440,17 @@ run_test() { # testfile # load in the helper functions . $TEST_DIR/functions +if [ "$KTAP" = "1" ]; then + echo "TAP version 13" + + casecount=`echo $TEST_CASES | wc -w` + for t in $TEST_CASES; do + test_on_instance $t || continue + casecount=$((casecount+1)) + done + echo "1..${casecount}" +fi + # Main loop for t in $TEST_CASES; do run_test $t @@ -439,6 +485,17 @@ prlog "# of unsupported: " `echo $UNSUPPORTED_CASES | wc -w` prlog "# of xfailed: " `echo $XFAILED_CASES | wc -w` prlog "# of undefined(test bug): " `echo $UNDEFINED_CASES | wc -w` +if [ "$KTAP" = "1" ]; then + echo -n "# Totals:" + echo -n " pass:"`echo $PASSED_CASES | wc -w` + echo -n " faii:"`echo $FAILED_CASES | wc -w` + echo -n " xfail:"`echo $XFAILED_CASES | wc -w` + echo -n " xpass:0" + echo -n " skip:"`echo $UNTESTED_CASES $UNSUPPORTED_CASES | wc -w` + echo -n " error:"`echo $UNRESOLVED_CASES $UNDEFINED_CASES | wc -w` + echo +fi + cleanup # if no error, return 0 diff --git a/tools/testing/selftests/ftrace/ftracetest-ktap b/tools/testing/selftests/ftrace/ftracetest-ktap new file mode 100755 index 000000000000..b3284679ef3a --- /dev/null +++ b/tools/testing/selftests/ftrace/ftracetest-ktap @@ -0,0 +1,8 @@ +#!/bin/sh -e +# SPDX-License-Identifier: GPL-2.0-only +# +# ftracetest-ktap: Wrapper to integrate ftracetest with the kselftest runner +# +# Copyright (C) Arm Ltd., 2023 + +./ftracetest -K