From patchwork Wed Dec 20 06:31:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Oliva X-Patchwork-Id: 181481 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:24d3:b0:fb:cd0c:d3e with SMTP id r19csp2451622dyi; Tue, 19 Dec 2023 22:32:07 -0800 (PST) X-Google-Smtp-Source: AGHT+IHF4aLoRT8aVx//o6vAgfZwHUL5IB1rFWysNoqzqlbR0UxY01xrKLwJ7MVOsxehiYKh6yNt X-Received: by 2002:a05:620a:9ce:b0:781:e12:1376 with SMTP id y14-20020a05620a09ce00b007810e121376mr1057071qky.85.1703053927150; Tue, 19 Dec 2023 22:32:07 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1703053927; cv=pass; d=google.com; s=arc-20160816; b=ppNBqKojZlPHA+Jc2FgIRAWg754BU1hMVcxOHFTdfkdMBJvJ2de7Vuvm0rNDL68n6E 2U4uOomQy/KjX/mJtxNZjYT199g+8JlqQIpPai9aZ2IdSD/jX/ulizZxH0LRGmd1SVpd BLLIl1SWUiJQG4KBr7ICJYPPHIhVQ5MAQrUJzBzQ1fOHOdj6//6gY48LU0gs0QgeYPbe J9dpXFd6xJe0NokNkAGfWI+zuL2jKl2tBnMZPC1P2bttQDRoZRtRfmcTHzLDR0H49Bhi 1hvgEtaK/6IxT7wiDxDa3zGptbFp/gUYm4lVPCg2Co0aeVRjEr5bVbAV0WOmkFQw/jBF LeRw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:mime-version:user-agent :message-id:in-reply-to:date:references:organization:subject:cc:to :from:arc-filter:dmarc-filter:delivered-to; bh=HduHcnWn+VA/qfS8t98PVd4vOjH6ayOnzDURgIAXxxg=; fh=oP2seqwYM9kCmF0wCmpEffREX9w/v3iZnOsMAdpu/zY=; b=dAFMdxKcmJK+gibogtG65QYkoDugHOvDjDjmLEMMGP8o231SvdvHGSCklUAiZS+q5x /zr7fPJ/nSrYuxRN4MLvyR2PihGFfqWxuE4yZaBlLDcUm+nWykjORI1DThyvAJvxxm7T MmR+9+lEFRcxW3ayiuO1runeut64d21pw0wndDhAa24ff7jqzRA7OjjhlzTEq4Ci9e5G b9aNV5XW9h3rYasAgDuddF9GOK43vFwDV6mmtLgjfU2TU9nXDWIsjnjStOfFdDT/jjvP udKzCbZ0UILv7ND0z01YE/o9MaDbo5AmsOOHaX0496nI1HFsjdnN9R9o0BeWRt0Qzfmb eqJg== ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gnu.org Received: from server2.sourceware.org (server2.sourceware.org. [2620:52:3:1:0:246e:9693:128c]) by mx.google.com with ESMTPS id dt16-20020a05620a479000b0077efefae26asi2135945qkb.410.2023.12.19.22.32.07 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Dec 2023 22:32:07 -0800 (PST) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) client-ip=2620:52:3:1:0:246e:9693:128c; Authentication-Results: mx.google.com; arc=pass (i=1); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gnu.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id DA82F3860769 for ; Wed, 20 Dec 2023 06:32:06 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from eggs.gnu.org (eggs.gnu.org [IPv6:2001:470:142:3::10]) by sourceware.org (Postfix) with ESMTPS id C496A385E019 for ; Wed, 20 Dec 2023 06:31:38 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org C496A385E019 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gnu.org Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gnu.org ARC-Filter: OpenARC Filter v1.0.0 sourceware.org C496A385E019 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2001:470:142:3::10 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1703053901; cv=none; b=gw8PVqbjRnXILYehZjHhWx9Wso4MsHP4tklo9SU8QJtgGPQVUU1SfYO0LRGtZAUQiC+MdmHxZBc6vW2jcxHShsvQmYDN9FexCdC1Bc5/jqCfr+Tive+CINU5UKyVOLrckBpfnUUKGyZ70Dtd7VOh5l5EZdTjJqN0W25WegkQo4I= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1703053901; c=relaxed/simple; bh=ihSbM1Z99V/INq/1IMcFoEpAXkTZ+epvXtOBfPPnFGc=; h=From:To:Subject:Date:Message-ID:MIME-Version; b=k/+0x/7P3fWQO60DECtXaNSUCYehgLLDot6+LBQMJn8C1PsVlGnKABol0oFMxrJFEOZqNp5v2LDj+SobepBVNRndlact7y56f48eMobf6H9XbsZepLkWP+Mt6oayo5XnkI5TRCKznMrGZFOmEcn2y5TJ/vE+gOyAznC4+yzCxdw= ARC-Authentication-Results: i=1; server2.sourceware.org Received: from linux-libre.fsfla.org ([2001:470:142:5::54] helo=free.home) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rFq7G-0005iM-30; Wed, 20 Dec 2023 01:31:38 -0500 Received: from livre (livre.home [172.31.160.2]) by free.home (8.15.2/8.15.2) with ESMTPS id 3BK6VQKd680382 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NOT); Wed, 20 Dec 2023 03:31:27 -0300 From: Alexandre Oliva To: Jeff Law Cc: "gcc-patches@gcc.gnu.org" Subject: [PATCH] compare_tests: distinguish c-c++-common results by tool Organization: Free thinker, not speaking for the GNU Project References: <76a02442-290a-4929-b566-035753afd859@gmail.com> Date: Wed, 20 Dec 2023 03:31:26 -0300 In-Reply-To: <76a02442-290a-4929-b566-035753afd859@gmail.com> (Jeff Law's message of "Tue, 19 Dec 2023 21:40:32 -0700") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 X-Spam-Status: No, score=-8.5 required=5.0 tests=BAYES_00, GIT_PATCH_0, KAM_DMARC_STATUS, SPF_HELO_PASS, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785781474797417556 X-GMAIL-MSGID: 1785781474797417556 On Dec 20, 2023, Jeff Law wrote: > So the strub tests in c-c++-common are problematical. They get run > twice, once for C, once for C++. Yet the name of the test is the same > in both runs. (by the name, I mean the name emitted into the dejagnu > summary and log files). > Thus if you have a test in there which passes in one context, but > fails in the other, comparison tools like contrib/compare_tests may > erroneously report the tests as both a test which now fails, but > passed before and a test which now passes but failed before. > It looks like some of the strub tests are currently known to fail with > C++ and are triggering this problem Yeah, type warnings/errors are different between C and C++, and this is noticeable with permissible conversions between strub types. > A third option would be to change the compare_tests tool to somehow > distinguish between the C and C++ tests. Not sure how feasible that > is. Most feasible among the possibilities ;-) I've tested the following by comparing my obj-x86_64-linux-gnu test tree with itself. Ok to install? When compare_tests compares both C and C++ tests in c-c++-common, they get the same identifier, so expected differences in results across languages become undesirably noisy. This patch adds tool identifiers to tests, so that runs by different tools are not confused by the compare logic. It also fixes a bug in reporting differences, that would attempt to print an undefined fname (the definitions are in subshell loops), and adjusts the target insertion to match tabs in addition to blanks after colons. for contrib/ChangeLog * compare_tests: Add tool to test lines. Match tabs besides blanks to insert tool and target. Don't print undefined fname. --- contrib/compare_tests | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/contrib/compare_tests b/contrib/compare_tests index 2dfa8640756a0..e09fc4f113a3f 100755 --- a/contrib/compare_tests +++ b/contrib/compare_tests @@ -96,7 +96,7 @@ if [ -d "$1" -a -d "$2" ] ; then ret=$? if [ $ret -ne 0 ]; then exit_status=`expr $exit_status + 1` - echo "## Differences found: $fname" + echo "## Differences found" fi if [ $exit_status -ne 0 ]; then echo "# $exit_status differences in $cmnsums common sum files found" @@ -108,8 +108,8 @@ elif [ -d "$1" -o -d "$2" ] ; then usage "Must specify either two directories or two files" fi -sed 's/^XFAIL/FAIL/; s/^ERROR/FAIL/; s/^XPASS/PASS/' < "$1" | awk '/^Running target / {target = $3} { if (target != "unix") { sub(/: /, "&"target": " ); }; print $0; }' | cut -c1-2000 >$tmp1 -sed 's/^XFAIL/FAIL/; s/^ERROR/FAIL/; s/^XPASS/PASS/' < "$2" | awk '/^Running target / {target = $3} { if (target != "unix") { sub(/: /, "&"target": " ); }; print $0; }' | cut -c1-2000 >$tmp2 +sed 's/^XFAIL/FAIL/; s/^ERROR/FAIL/; s/^XPASS/PASS/' < "$1" | awk '/^[ ]*=== [^ ]* tests ===$/ {tool = $2} /^Running target / {target = $3} { if (tool != "") { sub(/:[ ]/, "&"tool": " ); }; if (target != "unix") { sub(/:[ ]/, "&"target": " ); }; print $0; }' | cut -c1-2000 >$tmp1 +sed 's/^XFAIL/FAIL/; s/^ERROR/FAIL/; s/^XPASS/PASS/' < "$2" | awk '/^[ ]*=== [^ ]* tests ===$/ {tool = $2} /^Running target / {target = $3} { if (tool != "") { sub(/:[ ]/, "&"tool": " ); }; if (target != "unix") { sub(/:[ ]/, "&"target": " ); }; print $0; }' | cut -c1-2000 >$tmp2 before=$tmp1 now=$tmp2