Message ID | 20220714215325.GA18923@ldh-imac.local |
---|---|
State | New, archived |
Headers |
Return-Path: <gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org> Delivered-To: ouuuleilei@gmail.com Received: by 2002:a98:d5ce:0:b0:178:cc93:bf7d with SMTP id g14csp556676eik; Thu, 14 Jul 2022 14:54:14 -0700 (PDT) X-Google-Smtp-Source: AGRyM1svGjendUdm601fSL1fC1+JFl+xLp8zI4veTJ4D8LdRkY2a2N0WsRM8Kewn5yJ6cMlNJF2V X-Received: by 2002:a17:906:4fc4:b0:6da:b4c6:fadb with SMTP id i4-20020a1709064fc400b006dab4c6fadbmr10716518ejw.282.1657835654130; Thu, 14 Jul 2022 14:54:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1657835654; cv=none; d=google.com; s=arc-20160816; b=s5NoDY4xJWYIUQkrHWVrT/5H0N7/m0f7NB9LfHQo/QY1ja2ZSteYAGTltlV9XJuMgB l98ZNmeYW2BqQQWnQ7MRCZpc/tE4arXAa/XxgVswTs8PyI3Q8p/pU5wGWRm6S6vBo7LS WpJ8ePme2XoltQLWz26p3qQMTF6DH50G/ns0bdRLTTgc5w3qBoF4lYo+Ggt2smt3R7eE d5mBUX8nenhalGjAWlNNwWK69Wpk8DBeJtTar7IhPG6zi2hIb5NwU9MBbJSLeczlzWUR oRxGm4TTOgrR7B/4sulVw1h33LlqQljbUiq48LXaEp6vU58iFKYJE9CR9OiSLmIBrhj1 UdyA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:reply-to:from:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence :content-disposition:mime-version:message-id:subject:to:date :dmarc-filter:delivered-to:dkim-signature:dkim-filter; bh=E/iN3BZNT1aNyuQsuL9+ussYtFnVTQAkqG4PGqmuyNo=; b=RjLcnUQHcgr2Xh0+pJVqc4frh4sXbjoZJX8pXfFuL6H0pqAoYHDk29xSKUtfgw8y5m KXwOhZUQpapVqj6RRaERcZIy7mv2klIL93rwPdNqmQwnw3KpPRmAiRA8WCRaanYsJXTB c7G0hFr23DiUDABX57Jc1Kkmdx70jr6bFw9+NCC/++2vSWOaI+oipow3eodlO8khu11v Mx2Sjv7iZ+K8cXrf66TyaG+nmWdBtvCWankaqjQH14MIS+lKvygKgk4KGzDGiEr80Jyv 9aFttwHCk92/vBqu+YuwhBm0O935bEhBqZf7Fgycf1URd2WvYfoWjIMRkKhFc+8xe0AY 6Kcw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=rdoDLGhl; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gnu.org Received: from sourceware.org (server2.sourceware.org. [2620:52:3:1:0:246e:9693:128c]) by mx.google.com with ESMTPS id hs2-20020a1709073e8200b0072b81296609si4465909ejc.64.2022.07.14.14.54.13 for <ouuuleilei@gmail.com> (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Jul 2022 14:54:14 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) client-ip=2620:52:3:1:0:246e:9693:128c; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=rdoDLGhl; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gnu.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id E54ED3858C55 for <ouuuleilei@gmail.com>; Thu, 14 Jul 2022 21:54:12 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org E54ED3858C55 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1657835652; bh=E/iN3BZNT1aNyuQsuL9+ussYtFnVTQAkqG4PGqmuyNo=; h=Date:To:Subject:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=rdoDLGhlDWOj7p11U96bHEJMzKKhk+P7wqzT1X8ZV39nRLZhaIvla3ovX6N6o/SmC IY8yRy1luz8cE288SuOWnvsOrVvadvWwa2URVS56UfmNy5keztz/kMg2q7qpC3VDa/ aVrTqthkAYWB63Vi7JpfgvBgf+ui4JyqCFTPCRps= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mail-qk1-x72e.google.com (mail-qk1-x72e.google.com [IPv6:2607:f8b0:4864:20::72e]) by sourceware.org (Postfix) with ESMTPS id 7EBA63858D37 for <gcc-patches@gcc.gnu.org>; Thu, 14 Jul 2022 21:53:29 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 7EBA63858D37 Received: by mail-qk1-x72e.google.com with SMTP id b25so2305558qkl.1 for <gcc-patches@gcc.gnu.org>; Thu, 14 Jul 2022 14:53:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:subject:message-id:mime-version :content-disposition; bh=E/iN3BZNT1aNyuQsuL9+ussYtFnVTQAkqG4PGqmuyNo=; b=7+YqUeh4Y/DAfFrKl50lxp4VWtwrZEGIw73RZLMw1eIO6S8+bTNoAvDGZql0N5FYXt v+HcQV+Kntl2PsmhKXq1uiFpeKSGvtun9dIP3+j99Ru2KZoRHHobZpbGrzHvj1vW9AAx +UdEM6hyO1CloWI/1duEN4Eb7nqQEFVtRjhrEw2wU0spLhQUV2GOKWFXbJneNNxJCj86 GTEyoIpGFY+4VrJHpleDaiDHgAjaGxgRpgzC+Z06HLcFXb/URq4wMIF7Gf9UTnmiUQAH Bvgs1ZgCoU1sThsfAn+80COnT3FoKfiR/E23SSUicicZkFrCjfdGFY6TTLYO9x9lVfHR TVjg== X-Gm-Message-State: AJIora/RvEZYL6qNV60vHZgage/W3/SLVFMqXE0gS8UD7/GBpdYFNVwG ebNqDxAW0Hw84ZY44bu7I1fsSRz2Pqc= X-Received: by 2002:a05:620a:44c9:b0:6b2:1f49:f0f5 with SMTP id y9-20020a05620a44c900b006b21f49f0f5mr7486961qkp.327.1657835608492; Thu, 14 Jul 2022 14:53:28 -0700 (PDT) Received: from ldh-imac.local (96-67-140-173-static.hfc.comcastbusiness.net. [96.67.140.173]) by smtp.gmail.com with ESMTPSA id k10-20020ac8474a000000b00304bc2acc25sm2384383qtp.6.2022.07.14.14.53.27 for <gcc-patches@gcc.gnu.org> (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Jul 2022 14:53:27 -0700 (PDT) Date: Thu, 14 Jul 2022 17:53:25 -0400 To: gcc-patches@gcc.gnu.org Subject: [PATCH] libphobos: Fix instability in the parallelized testsuite Message-ID: <20220714215325.GA18923@ldh-imac.local> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="jRHKVT23PllUwdXP" Content-Disposition: inline X-Spam-Status: No, score=-3039.1 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list <gcc-patches.gcc.gnu.org> List-Unsubscribe: <https://gcc.gnu.org/mailman/options/gcc-patches>, <mailto:gcc-patches-request@gcc.gnu.org?subject=unsubscribe> List-Archive: <https://gcc.gnu.org/pipermail/gcc-patches/> List-Post: <mailto:gcc-patches@gcc.gnu.org> List-Help: <mailto:gcc-patches-request@gcc.gnu.org?subject=help> List-Subscribe: <https://gcc.gnu.org/mailman/listinfo/gcc-patches>, <mailto:gcc-patches-request@gcc.gnu.org?subject=subscribe> From: Lewis Hyatt via Gcc-patches <gcc-patches@gcc.gnu.org> Reply-To: Lewis Hyatt <lhyatt@gmail.com> Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org Sender: "Gcc-patches" <gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org> X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1738366678845852100?= X-GMAIL-MSGID: =?utf-8?q?1738366678845852100?= |
Series |
libphobos: Fix instability in the parallelized testsuite
|
|
Commit Message
Li, Pan2 via Gcc-patches
July 14, 2022, 9:53 p.m. UTC
Hello- I get a different number of test results from libphobos.unittest/unittest.exp, depending on server load. I believe it's because this testsuite doesn't check runtest_file_p: $ make -j 1 RUNTESTFLAGS='unittest.exp' check-target-libphobos | grep '^#' # of expected passes 10 $ make -j 2 RUNTESTFLAGS='unittest.exp' check-target-libphobos | grep '^#' # of expected passes 10 # of expected passes 10 $ make -j 4 RUNTESTFLAGS='unittest.exp' check-target-libphobos | grep '^#' # of expected passes 10 # of expected passes 10 # of expected passes 10 # of expected passes 10 When running in parallel along with other tests, even at a fixed argument for -j, the number of tests that actually execute will depend on how many of the parallel sub-makes happened to start prior to the first one finishing, hence it changes from run to run. The attached patch fixes it for me, if it looks OK? Thanks, this would remove some noise from before/after test comparisons. -Lewis libphobos: Fix instability in the parallelized testsuite libphobos.unittest/unittest.exp calls bare dg-test rather than dg-runtest, and so it should call runtest_file_p to determine whether to run each test or not. Without that call, the tests run too many times in parallel mode (they will run as many times, as the argument to make -j). libphobos/ChangeLog: * testsuite/libphobos.unittest/unittest.exp: Call runtest_file_p prior to running each test.
Comments
Excerpts from Lewis Hyatt via Gcc-patches's message of Juli 14, 2022 11:53 pm: > Hello- > > I get a different number of test results from libphobos.unittest/unittest.exp, > depending on server load. I believe it's because this testsuite doesn't check > runtest_file_p: > > $ make -j 1 RUNTESTFLAGS='unittest.exp' check-target-libphobos | grep '^#' > # of expected passes 10 > > $ make -j 2 RUNTESTFLAGS='unittest.exp' check-target-libphobos | grep '^#' > # of expected passes 10 > # of expected passes 10 > > $ make -j 4 RUNTESTFLAGS='unittest.exp' check-target-libphobos | grep '^#' > # of expected passes 10 > # of expected passes 10 > # of expected passes 10 > # of expected passes 10 > > When running in parallel along with other tests, even at a fixed argument > for -j, the number of tests that actually execute will depend on how many of the > parallel sub-makes happened to start prior to the first one finishing, hence > it changes from run to run. > > The attached patch fixes it for me, if it looks OK? Thanks, this would remove > some noise from before/after test comparisons. > > -Lewis > libphobos: Fix instability in the parallelized testsuite > > libphobos.unittest/unittest.exp calls bare dg-test rather than dg-runtest, and > so it should call runtest_file_p to determine whether to run each test or > not. Without that call, the tests run too many times in parallel mode (they will > run as many times, as the argument to make -j). Hi Lewis, Thanks! Good spot. I think it should be calling dg-runtest however, same as what libphobos.cycles/cycles.exp is doing. Could also fix the test name so each one is unique, just to hit two birds in one - something like the following would suffice (haven't had time to check). Kind Regards, Iain. --- --- a/libphobos/testsuite/libphobos.unittest/unittest.exp +++ b/libphobos/testsuite/libphobos.unittest/unittest.exp @@ -42,8 +42,10 @@ foreach unit_test $unit_test_list { set expected_fail [lindex $unit_test 1] foreach test $tests { - set shouldfail $expected_fail - dg-test $test "" $test_flags + set libphobos_test_name "[dg-trim-dirname $srcdir $test] $test_flags" + set shouldfail $expected_fail + dg-runtest $test "" $test_flags + set libphobos_test_name "" } set shouldfail 0
> Hi Lewis, > > Thanks! Good spot. I think it should be calling dg-runtest however, > same as what libphobos.cycles/cycles.exp is doing. Could also fix the > test name so each one is unique, just to hit two birds in one - > something like the following would suffice (haven't had time to check). > > Kind Regards, > Iain. > > --- > > --- a/libphobos/testsuite/libphobos.unittest/unittest.exp > +++ b/libphobos/testsuite/libphobos.unittest/unittest.exp > @@ -42,8 +42,10 @@ foreach unit_test $unit_test_list { > set expected_fail [lindex $unit_test 1] > > foreach test $tests { > - set shouldfail $expected_fail > - dg-test $test "" $test_flags > + set libphobos_test_name "[dg-trim-dirname $srcdir $test] $test_flags" > + set shouldfail $expected_fail > + dg-runtest $test "" $test_flags > + set libphobos_test_name "" > } > > set shouldfail 0 > Thanks for the followup. I tested and can confirm your version works fine: PASS: libphobos.unittest/customhandler.d -fversion=FailNoPrintout (test for excess errors) PASS: libphobos.unittest/customhandler.d -fversion=FailNoPrintout execution test PASS: libphobos.unittest/customhandler.d -fversion=FailedTests (test for excess errors) PASS: libphobos.unittest/customhandler.d -fversion=FailedTests execution test PASS: libphobos.unittest/customhandler.d -fversion=GoodTests (test for excess errors) PASS: libphobos.unittest/customhandler.d -fversion=GoodTests execution test PASS: libphobos.unittest/customhandler.d -fversion=NoTests (test for excess errors) PASS: libphobos.unittest/customhandler.d -fversion=NoTests execution test PASS: libphobos.unittest/customhandler.d -fversion=PassNoPrintout (test for excess errors) PASS: libphobos.unittest/customhandler.d -fversion=PassNoPrintout execution test Let me know if you want me to do anything from there please? By the way, there are a few other tests that cause some minor glitches with comparing results: libphobos.sum:PASS: libphobos.shared/link.d -I/home/lewis/gccdev/base/src/libphobos/testsuite/libphobos.shared lib.so -shared-libphobos (test for excess errors) libphobos.sum:PASS: libphobos.shared/link.d -I/home/lewis/gccdev/base/src/libphobos/testsuite/libphobos.shared lib.so -shared-libphobos execution test libphobos.sum:PASS: libphobos.shared/link_linkdep.d -I/home/lewis/gccdev/base/src/libphobos/testsuite/libphobos.shared liblinkdep.so lib.so -shared-libphobos (test for excess errors) libphobos.sum:PASS: libphobos.shared/link_linkdep.d -I/home/lewis/gccdev/base/src/libphobos/testsuite/libphobos.shared liblinkdep.so lib.so -shared-libphobos execution test libphobos.sum:PASS: libphobos.shared/link_loaddep.d -I/home/lewis/gccdev/base/src/libphobos/testsuite/libphobos.shared libloaddep.so -shared-libphobos (test for excess errors) libphobos.sum:PASS: libphobos.shared/link_loaddep.d -I/home/lewis/gccdev/base/src/libphobos/testsuite/libphobos.shared libloaddep.so -shared-libphobos execution test The problem here is that the absolute path to the test dir ends up in the results summary, since it appears in the options string that is part of the test name. It's not so hard to work around when doing the comparisons, but it seems to be the only case where this happens in the whole testsuite, other than one other similar case from libgo. Is there a standard way to handle it I take it? Thanks... -Lewis
diff --git a/libphobos/testsuite/libphobos.unittest/unittest.exp b/libphobos/testsuite/libphobos.unittest/unittest.exp index 2a019caca8c..175decdc333 100644 --- a/libphobos/testsuite/libphobos.unittest/unittest.exp +++ b/libphobos/testsuite/libphobos.unittest/unittest.exp @@ -42,6 +42,9 @@ foreach unit_test $unit_test_list { set expected_fail [lindex $unit_test 1] foreach test $tests { + if {![runtest_file_p $runtests $test]} { + continue + } set shouldfail $expected_fail dg-test $test "" $test_flags }