testsuite: Fix mistransformed gcov

  How was r13-2619-g34b9a03353d3fd "gcov: Respect triplet when looking
for gcov" tested?  I'm having a hard time believing it was tested with
a *cross-compiler* *in-build-tree*.  I think it was only tested for
the special-case of an installed cross-compiler; not even with a
native build exercising the false branch (see patch), considering that
a breaking typo ("}" vs "]") snuck by in the first revision, before
commit.

I guess reviewers forgot to ask that, but it's really on the
submitter; it's a general requirement for patches to say how it was
tested.  Usually no more is needed than "tested with a cross-compiler
for ..., no regressions" (implying running twice and comparing results
before and after the patch).

In this case, when adjusting test-framework bits, a little more care
and mentioned details about how it was tested, would have been in
order.  It's likely it'd have shown that an uninstalled in-tree cross
(an IMHO more expected case) wasn't tested, which that patch broke for
the one gcov invocation that the testsuite does for cross-builds
(IIUC).

It can be said like this: I tested *this* patch as follows (all
directory names below manually redacted), showing no regressions and
fixing the regression for the in-tree cross build;

For a native build (x86_64, Debian 11):
- In the gcc build directory:
 make check RUNTESTFLAGS=gcov.exp

- In an empty new directory after native installation in /pre.
/gccsrc/top/contrib/test_installed --prefix=/pre gcov.exp

- Also, for good measure (mentioned somewhere in the installation
documentation) native:
for tool in gcc g++ ; do env PATH=/pre/bin:$PATH runtest \
 --tool $tool --srcdir=/gccsrc/top/gcc/testsuite gcov.exp; done

For a cris-elf cross:
- In the gcc build directory:
make check 'RUNTESTFLAGS=--target_board=cris-sim gcov.exp'

- In an empty new directory after installation in /pre.
/gccsrc/top/contrib/test_installed --with-gcc=/pre/bin/cris-elf-gcc \
 --with-g++=/pre/bin/cris-elf-g++ --with-gfortran=/pre/bin/cris-elf-gfortran \
 --target=cris-elf --target_board=cris-sim gcov.exp

Ok to commit?

brgds, H-P
PS. Beware that the proc name may be up for bikeshedding.

---- 8< -------- 8< -------- 8< -------- 8< ----

In commit r13-2619-g34b9a03353d3fd, [transform] was applied to all
invocations of gcov, for both out-of-tree and in-tree testing.
For in-tree cross builds, this means gcov was called as
"/path/to/gccobj/gcc/target-tuple-gcov" gcov-pr94029.c which is
incorrect, as it's there "/path/to/gccobj/gcc/gcov" until it's
installed.  This caused a testsuite failure, like:

Running /x/gcc/gcc/testsuite/gcc.misc-tests/gcov.exp ...
FAIL: gcc.misc-tests/gcov-pr94029.c gcov failed: spawn failed

To avoid cumbersome conditionals, use a dedicated new helper function.

gcc/testsuite:
	* lib/gcc-dg.exp (gcc-transform-out-of-tree): New proc.
	* g++.dg/gcov/gcov.exp, gcc.misc-tests/gcov.exp: Call
	gcc-transform-out-of-tree instead of transform.
---
 gcc/testsuite/g++.dg/gcov/gcov.exp    |  4 ++--
 gcc/testsuite/gcc.misc-tests/gcov.exp |  4 ++--
 gcc/testsuite/lib/gcc-dg.exp          | 13 +++++++++++++
 3 files changed, 17 insertions(+), 4 deletions(-)
  

Hi,

On 2022-11-16 03:11, Hans-Peter Nilsson wrote:
> How was r13-2619-g34b9a03353d3fd "gcov: Respect triplet when looking
> for gcov" tested?  I'm having a hard time believing it was tested with
> a *cross-compiler* *in-build-tree*.  I think it was only tested for
> the special-case of an installed cross-compiler; not even with a
> native build exercising the false branch (see patch), considering that
> a breaking typo ("}" vs "]") snuck by in the first revision, before
> commit.

I was testing this in out-of-tree test with a cross-compiler (built for 
arm-none-eabi) running on Windows and Linux and just noticed the failure 
and (wrongly) assumed that all of the cases needed the transformation 
call as that's how other tools were handled.
The test systems used is are hosts that does not have any internet 
connection, so that's why I got a typo in the first patch when moving 
the fix to a system that did have an internet connection.
Sorry for the mess I caused!

Regarding the patch you propose; it looks good to me, but I'm not able 
to approve it.

Kind regards,
Torbjörn

> I guess reviewers forgot to ask that, but it's really on the
> submitter; it's a general requirement for patches to say how it was
> tested.  Usually no more is needed than "tested with a cross-compiler
> for ..., no regressions" (implying running twice and comparing results
> before and after the patch).
> 
> In this case, when adjusting test-framework bits, a little more care
> and mentioned details about how it was tested, would have been in
> order.  It's likely it'd have shown that an uninstalled in-tree cross
> (an IMHO more expected case) wasn't tested, which that patch broke for
> the one gcov invocation that the testsuite does for cross-builds
> (IIUC).
> 
> It can be said like this: I tested *this* patch as follows (all
> directory names below manually redacted), showing no regressions and
> fixing the regression for the in-tree cross build;
> 
> For a native build (x86_64, Debian 11):
> - In the gcc build directory:
>   make check RUNTESTFLAGS=gcov.exp
> 
> - In an empty new directory after native installation in /pre.
> /gccsrc/top/contrib/test_installed --prefix=/pre gcov.exp
> 
> - Also, for good measure (mentioned somewhere in the installation
> documentation) native:
> for tool in gcc g++ ; do env PATH=/pre/bin:$PATH runtest \
>   --tool $tool --srcdir=/gccsrc/top/gcc/testsuite gcov.exp; done
> 
> For a cris-elf cross:
> - In the gcc build directory:
> make check 'RUNTESTFLAGS=--target_board=cris-sim gcov.exp'
> 
> - In an empty new directory after installation in /pre.
> /gccsrc/top/contrib/test_installed --with-gcc=/pre/bin/cris-elf-gcc \
>   --with-g++=/pre/bin/cris-elf-g++ --with-gfortran=/pre/bin/cris-elf-gfortran \
>   --target=cris-elf --target_board=cris-sim gcov.exp
> 
> Ok to commit?
> 
> brgds, H-P
> PS. Beware that the proc name may be up for bikeshedding.
> 
> ---- 8< -------- 8< -------- 8< -------- 8< ----
> 
> In commit r13-2619-g34b9a03353d3fd, [transform] was applied to all
> invocations of gcov, for both out-of-tree and in-tree testing.
> For in-tree cross builds, this means gcov was called as
> "/path/to/gccobj/gcc/target-tuple-gcov" gcov-pr94029.c which is
> incorrect, as it's there "/path/to/gccobj/gcc/gcov" until it's
> installed.  This caused a testsuite failure, like:
> 
> Running /x/gcc/gcc/testsuite/gcc.misc-tests/gcov.exp ...
> FAIL: gcc.misc-tests/gcov-pr94029.c gcov failed: spawn failed
> 
> To avoid cumbersome conditionals, use a dedicated new helper function.
> 
> gcc/testsuite:
> 	* lib/gcc-dg.exp (gcc-transform-out-of-tree): New proc.
> 	* g++.dg/gcov/gcov.exp, gcc.misc-tests/gcov.exp: Call
> 	gcc-transform-out-of-tree instead of transform.
> ---
>   gcc/testsuite/g++.dg/gcov/gcov.exp    |  4 ++--
>   gcc/testsuite/gcc.misc-tests/gcov.exp |  4 ++--
>   gcc/testsuite/lib/gcc-dg.exp          | 13 +++++++++++++
>   3 files changed, 17 insertions(+), 4 deletions(-)
> 
> diff --git a/gcc/testsuite/g++.dg/gcov/gcov.exp b/gcc/testsuite/g++.dg/gcov/gcov.exp
> index 04e7a0164865..c9f20958836b 100644
> --- a/gcc/testsuite/g++.dg/gcov/gcov.exp
> +++ b/gcc/testsuite/g++.dg/gcov/gcov.exp
> @@ -24,9 +24,9 @@ global GXX_UNDER_TEST
>   
>   # Find gcov in the same directory as $GXX_UNDER_TEST.
>   if { ![is_remote host] && [string match "*/*" [lindex $GXX_UNDER_TEST 0]] } {
> -    set GCOV [file dirname [lindex $GXX_UNDER_TEST 0]]/[transform gcov]
> +    set GCOV [file dirname [lindex $GXX_UNDER_TEST 0]]/[gcc-transform-out-of-tree gcov]
>   } else {
> -    set GCOV [transform gcov]
> +    set GCOV [gcc-transform-out-of-tree gcov]
>   }
>   
>   # Initialize harness.
> diff --git a/gcc/testsuite/gcc.misc-tests/gcov.exp b/gcc/testsuite/gcc.misc-tests/gcov.exp
> index b8e9661aa537..90ceec46e0f0 100644
> --- a/gcc/testsuite/gcc.misc-tests/gcov.exp
> +++ b/gcc/testsuite/gcc.misc-tests/gcov.exp
> @@ -24,9 +24,9 @@ global GCC_UNDER_TEST
>   
>   # For now find gcov in the same directory as $GCC_UNDER_TEST.
>   if { ![is_remote host] && [string match "*/*" [lindex $GCC_UNDER_TEST 0]] } {
> -    set GCOV [file dirname [lindex $GCC_UNDER_TEST 0]]/[transform gcov]
> +    set GCOV [file dirname [lindex $GCC_UNDER_TEST 0]]/[gcc-transform-out-of-tree gcov]
>   } else {
> -    set GCOV [transform gcov]
> +    set GCOV [gcc-transform-out-of-tree gcov]
>   }
>   
>   # Initialize harness.
> diff --git a/gcc/testsuite/lib/gcc-dg.exp b/gcc/testsuite/lib/gcc-dg.exp
> index 23ec038f41eb..2910c9ce0998 100644
> --- a/gcc/testsuite/lib/gcc-dg.exp
> +++ b/gcc/testsuite/lib/gcc-dg.exp
> @@ -1429,5 +1429,18 @@ proc scan-symbol-not { args } {
>       }
>   }
>   
> +# Transform a tool-name to its canonical-target-name by "transform"
> +# (which may return the original name for native targets) but only if
> +# testing out-of-tree.  When in-tree, the tool is expected to be found
> +# by its original name, typically with some build-directory prefix
> +# prepended by the caller.
> +proc gcc-transform-out-of-tree { args } {
> +    global TESTING_IN_BUILD_TREE
> +    if { [info exists TESTING_IN_BUILD_TREE] } {
> +	return $args;
> +    }
> +    return [transform $args]
> +}
> +
>   set additional_prunes ""
>   set dg_runtest_extra_prunes ""
  

On Wed, Nov 16, 2022 at 9:00 AM Torbjorn SVENSSON via Gcc-patches
<gcc-patches@gcc.gnu.org> wrote:
>
> Hi,
>
> On 2022-11-16 03:11, Hans-Peter Nilsson wrote:
> > How was r13-2619-g34b9a03353d3fd "gcov: Respect triplet when looking
> > for gcov" tested?  I'm having a hard time believing it was tested with
> > a *cross-compiler* *in-build-tree*.  I think it was only tested for
> > the special-case of an installed cross-compiler; not even with a
> > native build exercising the false branch (see patch), considering that
> > a breaking typo ("}" vs "]") snuck by in the first revision, before
> > commit.
>
> I was testing this in out-of-tree test with a cross-compiler (built for
> arm-none-eabi) running on Windows and Linux and just noticed the failure
> and (wrongly) assumed that all of the cases needed the transformation
> call as that's how other tools were handled.
> The test systems used is are hosts that does not have any internet
> connection, so that's why I got a typo in the first patch when moving
> the fix to a system that did have an internet connection.
> Sorry for the mess I caused!
>
> Regarding the patch you propose; it looks good to me, but I'm not able
> to approve it.

The patch is OK.

Richard.

> Kind regards,
> Torbjörn
>
> > I guess reviewers forgot to ask that, but it's really on the
> > submitter; it's a general requirement for patches to say how it was
> > tested.  Usually no more is needed than "tested with a cross-compiler
> > for ..., no regressions" (implying running twice and comparing results
> > before and after the patch).
> >
> > In this case, when adjusting test-framework bits, a little more care
> > and mentioned details about how it was tested, would have been in
> > order.  It's likely it'd have shown that an uninstalled in-tree cross
> > (an IMHO more expected case) wasn't tested, which that patch broke for
> > the one gcov invocation that the testsuite does for cross-builds
> > (IIUC).
> >
> > It can be said like this: I tested *this* patch as follows (all
> > directory names below manually redacted), showing no regressions and
> > fixing the regression for the in-tree cross build;
> >
> > For a native build (x86_64, Debian 11):
> > - In the gcc build directory:
> >   make check RUNTESTFLAGS=gcov.exp
> >
> > - In an empty new directory after native installation in /pre.
> > /gccsrc/top/contrib/test_installed --prefix=/pre gcov.exp
> >
> > - Also, for good measure (mentioned somewhere in the installation
> > documentation) native:
> > for tool in gcc g++ ; do env PATH=/pre/bin:$PATH runtest \
> >   --tool $tool --srcdir=/gccsrc/top/gcc/testsuite gcov.exp; done
> >
> > For a cris-elf cross:
> > - In the gcc build directory:
> > make check 'RUNTESTFLAGS=--target_board=cris-sim gcov.exp'
> >
> > - In an empty new directory after installation in /pre.
> > /gccsrc/top/contrib/test_installed --with-gcc=/pre/bin/cris-elf-gcc \
> >   --with-g++=/pre/bin/cris-elf-g++ --with-gfortran=/pre/bin/cris-elf-gfortran \
> >   --target=cris-elf --target_board=cris-sim gcov.exp
> >
> > Ok to commit?
> >
> > brgds, H-P
> > PS. Beware that the proc name may be up for bikeshedding.
> >
> > ---- 8< -------- 8< -------- 8< -------- 8< ----
> >
> > In commit r13-2619-g34b9a03353d3fd, [transform] was applied to all
> > invocations of gcov, for both out-of-tree and in-tree testing.
> > For in-tree cross builds, this means gcov was called as
> > "/path/to/gccobj/gcc/target-tuple-gcov" gcov-pr94029.c which is
> > incorrect, as it's there "/path/to/gccobj/gcc/gcov" until it's
> > installed.  This caused a testsuite failure, like:
> >
> > Running /x/gcc/gcc/testsuite/gcc.misc-tests/gcov.exp ...
> > FAIL: gcc.misc-tests/gcov-pr94029.c gcov failed: spawn failed
> >
> > To avoid cumbersome conditionals, use a dedicated new helper function.
> >
> > gcc/testsuite:
> >       * lib/gcc-dg.exp (gcc-transform-out-of-tree): New proc.
> >       * g++.dg/gcov/gcov.exp, gcc.misc-tests/gcov.exp: Call
> >       gcc-transform-out-of-tree instead of transform.
> > ---
> >   gcc/testsuite/g++.dg/gcov/gcov.exp    |  4 ++--
> >   gcc/testsuite/gcc.misc-tests/gcov.exp |  4 ++--
> >   gcc/testsuite/lib/gcc-dg.exp          | 13 +++++++++++++
> >   3 files changed, 17 insertions(+), 4 deletions(-)
> >
> > diff --git a/gcc/testsuite/g++.dg/gcov/gcov.exp b/gcc/testsuite/g++.dg/gcov/gcov.exp
> > index 04e7a0164865..c9f20958836b 100644
> > --- a/gcc/testsuite/g++.dg/gcov/gcov.exp
> > +++ b/gcc/testsuite/g++.dg/gcov/gcov.exp
> > @@ -24,9 +24,9 @@ global GXX_UNDER_TEST
> >
> >   # Find gcov in the same directory as $GXX_UNDER_TEST.
> >   if { ![is_remote host] && [string match "*/*" [lindex $GXX_UNDER_TEST 0]] } {
> > -    set GCOV [file dirname [lindex $GXX_UNDER_TEST 0]]/[transform gcov]
> > +    set GCOV [file dirname [lindex $GXX_UNDER_TEST 0]]/[gcc-transform-out-of-tree gcov]
> >   } else {
> > -    set GCOV [transform gcov]
> > +    set GCOV [gcc-transform-out-of-tree gcov]
> >   }
> >
> >   # Initialize harness.
> > diff --git a/gcc/testsuite/gcc.misc-tests/gcov.exp b/gcc/testsuite/gcc.misc-tests/gcov.exp
> > index b8e9661aa537..90ceec46e0f0 100644
> > --- a/gcc/testsuite/gcc.misc-tests/gcov.exp
> > +++ b/gcc/testsuite/gcc.misc-tests/gcov.exp
> > @@ -24,9 +24,9 @@ global GCC_UNDER_TEST
> >
> >   # For now find gcov in the same directory as $GCC_UNDER_TEST.
> >   if { ![is_remote host] && [string match "*/*" [lindex $GCC_UNDER_TEST 0]] } {
> > -    set GCOV [file dirname [lindex $GCC_UNDER_TEST 0]]/[transform gcov]
> > +    set GCOV [file dirname [lindex $GCC_UNDER_TEST 0]]/[gcc-transform-out-of-tree gcov]
> >   } else {
> > -    set GCOV [transform gcov]
> > +    set GCOV [gcc-transform-out-of-tree gcov]
> >   }
> >
> >   # Initialize harness.
> > diff --git a/gcc/testsuite/lib/gcc-dg.exp b/gcc/testsuite/lib/gcc-dg.exp
> > index 23ec038f41eb..2910c9ce0998 100644
> > --- a/gcc/testsuite/lib/gcc-dg.exp
> > +++ b/gcc/testsuite/lib/gcc-dg.exp
> > @@ -1429,5 +1429,18 @@ proc scan-symbol-not { args } {
> >       }
> >   }
> >
> > +# Transform a tool-name to its canonical-target-name by "transform"
> > +# (which may return the original name for native targets) but only if
> > +# testing out-of-tree.  When in-tree, the tool is expected to be found
> > +# by its original name, typically with some build-directory prefix
> > +# prepended by the caller.
> > +proc gcc-transform-out-of-tree { args } {
> > +    global TESTING_IN_BUILD_TREE
> > +    if { [info exists TESTING_IN_BUILD_TREE] } {
> > +     return $args;
> > +    }
> > +    return [transform $args]
> > +}
> > +
> >   set additional_prunes ""
> >   set dg_runtest_extra_prunes ""