[0/4] v2 of Option handling: add documentation URLs

  On Wed, 2023-11-15 at 23:40 +0000, Joseph Myers wrote:
> On Wed, 15 Nov 2023, David Malcolm wrote:
>
> > As mentioned, I'm currently investigating capturing per-language
> > option
> > URLs (to address Iain's and Marc's comments about D and Ada); if I
> > get
> > that working, I may need to add a similar note for adding a new
> > frontend.
> > 
> > Hope the overall approach seems reasonable.
> 
> > Yes, the approach seems reasonable.

Thanks.

> > I suppose a difficulty with per-language URLs is that a given option
> > has a 
> > single OPT_* enumeration value; the diagnostic calls don't say
> > whether 
> > it's being used from a front end or the middle end (though maybe
> > there's 
> > not much overlap between the two) - though some option handling
> > already 
> > distinguishes based on what language is being compiled (e.g. 
> > LangEnabledBy).  For per-architecture URLs you don't have this issue 
> > because only one architecture is built into GCC at a time.

The urlifier does "know" the lang_mask, so I had a go at using that.

Here's an updated version of the patch kit.

In this one, rather than just parsing 'gcc/Option-Index.html',
regenerate-opt-urls.py now parses all of:

 'gcc/Option-Index.html' for generic options
 'gdc/Option-Index.html' with language='D'
 'gfortran/Option-Index.html' with language='Fortran'

As before, it generates .opt.urls files, but now they can contain
multiple directives: as well as the UrlSuffix directive, each
language now has a LangUrlSuffix_NAME_OF_LANG directive, such as
  LangUrlSuffix_D
and
  LangUrlSuffix_Fortran

I initially tried to have a single LangUrlSuffix(LANG, URL) directive,
but having them as a family of separate directives turned out to be much
easier to handle from the .awk scripts.

Hence the optionlist contains options for all configured targets
and for all frontends, and can contain multiple URLs.  For example,
Walloca has:

Walloca UrlSuffix(gcc/Warning-Options.html#index-Walloca) LangUrlSuffix_D(gdc/Warnings.html#index-Walloca)

The options-urls.cc generated by options-urls-cc-gen.awk previously
contained a big array of string literals.  To handle lang-specific
options, options-urls.cc now contains a generated switch statement
of the form:

const char *
get_opt_url_suffix (int option_index, unsigned lang_mask)
{
  switch (option_index)
    {
     [...snip...]
     case OPT_Walloca:
       if (lang_mask & CL_D)
         return "gdc/Warnings.html#index-Walloca";
       return "gcc/Warning-Options.html#index-Walloca";
     [...snip...]
  return nullptr;
}

I tested this via compiling .c and .d files with gcc and gdc to generate
a -Walloca warning.  Each compiler emitted the "correct" documentation
URL for -Walloca, in that gcc's URL pointed at the generic documentation,
and gdc's at the D-specific documentation.  There is also automated
selftest coverage for lang-specific URLs (in
selftest::test_get_option_url_suffix).

As before, with:

   warning: ‘#pragma pack’ has no effect with ‘-fpack-struct’ - ignored [-Wpragmas]

it successfully provides the user with three URLs (in a suitable
terminal): the documentation of the pragma, the -fpack-struct option,
and the -Wpragmas warning, respectively.

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu, with all
languages enabled, using gawk-5.0.1

I also smoke-tested the build with the patch on a minimal configuration
with just --enable-languages=c,c++ (and, in particular, with D *not*
enabled), and it built successfully.


How is this looking for trunk?

Thanks
Dave


David Malcolm (4):
  options: add gcc/regenerate-opt-urls.py
  Add generated .opt.urls files
  opts: add logic to generate options-urls.cc
  options: wire up options-urls.cc into gcc_urlifier

 gcc/Makefile.in                              |   29 +-
 gcc/ada/gcc-interface/lang.opt.urls          |   30 +
 gcc/analyzer/analyzer.opt.urls               |  206 ++
 gcc/c-family/c.opt.urls                      | 1409 ++++++++++++++
 gcc/common.opt.urls                          | 1832 ++++++++++++++++++
 gcc/config/aarch64/aarch64.opt.urls          |   84 +
 gcc/config/alpha/alpha.opt.urls              |   76 +
 gcc/config/alpha/elf.opt.urls                |    2 +
 gcc/config/arc/arc-tables.opt.urls           |    2 +
 gcc/config/arc/arc.opt.urls                  |  260 +++
 gcc/config/arm/arm-tables.opt.urls           |    2 +
 gcc/config/arm/arm.opt.urls                  |  149 ++
 gcc/config/arm/vxworks.opt.urls              |    2 +
 gcc/config/avr/avr.opt.urls                  |   71 +
 gcc/config/bfin/bfin.opt.urls                |   61 +
 gcc/config/bpf/bpf.opt.urls                  |   35 +
 gcc/config/c6x/c6x-tables.opt.urls           |    2 +
 gcc/config/c6x/c6x.opt.urls                  |   18 +
 gcc/config/cris/cris.opt.urls                |   65 +
 gcc/config/cris/elf.opt.urls                 |    8 +
 gcc/config/csky/csky.opt.urls                |  104 +
 gcc/config/csky/csky_tables.opt.urls         |    2 +
 gcc/config/darwin.opt.urls                   |  224 +++
 gcc/config/dragonfly.opt.urls                |    9 +
 gcc/config/epiphany/epiphany.opt.urls        |   52 +
 gcc/config/fr30/fr30.opt.urls                |    8 +
 gcc/config/freebsd.opt.urls                  |    9 +
 gcc/config/frv/frv.opt.urls                  |  111 ++
 gcc/config/ft32/ft32.opt.urls                |   20 +
 gcc/config/fused-madd.opt.urls               |    4 +
 gcc/config/g.opt.urls                        |    5 +
 gcc/config/gcn/gcn.opt.urls                  |   23 +
 gcc/config/gnu-user.opt.urls                 |    9 +
 gcc/config/h8300/h8300.opt.urls              |   29 +
 gcc/config/hpux11.opt.urls                   |    6 +
 gcc/config/i386/cygming.opt.urls             |   30 +
 gcc/config/i386/cygwin.opt.urls              |    6 +
 gcc/config/i386/djgpp.opt.urls               |    2 +
 gcc/config/i386/i386.opt.urls                |  602 ++++++
 gcc/config/i386/mingw-w64.opt.urls           |    5 +
 gcc/config/i386/mingw.opt.urls               |   12 +
 gcc/config/i386/nto.opt.urls                 |    5 +
 gcc/config/ia64/ia64.opt.urls                |  122 ++
 gcc/config/ia64/ilp32.opt.urls               |    8 +
 gcc/config/ia64/vms.opt.urls                 |    2 +
 gcc/config/iq2000/iq2000.opt.urls            |   14 +
 gcc/config/linux-android.opt.urls            |   11 +
 gcc/config/linux.opt.urls                    |   14 +
 gcc/config/lm32/lm32.opt.urls                |   14 +
 gcc/config/loongarch/loongarch.opt.urls      |   60 +
 gcc/config/lynx.opt.urls                     |    5 +
 gcc/config/m32c/m32c.opt.urls                |    8 +
 gcc/config/m32r/m32r.opt.urls                |   27 +
 gcc/config/m68k/ieee.opt.urls                |    4 +
 gcc/config/m68k/m68k-tables.opt.urls         |    2 +
 gcc/config/m68k/m68k.opt.urls                |  107 +
 gcc/config/m68k/uclinux.opt.urls             |    2 +
 gcc/config/mcore/mcore.opt.urls              |   38 +
 gcc/config/microblaze/microblaze.opt.urls    |   59 +
 gcc/config/mips/mips-tables.opt.urls         |    2 +
 gcc/config/mips/mips.opt.urls                |  269 +++
 gcc/config/mips/sde.opt.urls                 |    2 +
 gcc/config/mmix/mmix.opt.urls                |   44 +
 gcc/config/mn10300/mn10300.opt.urls          |   32 +
 gcc/config/moxie/moxie.opt.urls              |   14 +
 gcc/config/msp430/msp430.opt.urls            |   53 +
 gcc/config/nds32/nds32-elf.opt.urls          |    5 +
 gcc/config/nds32/nds32-linux.opt.urls        |    5 +
 gcc/config/nds32/nds32.opt.urls              |   57 +
 gcc/config/netbsd-elf.opt.urls               |    5 +
 gcc/config/netbsd.opt.urls                   |    6 +
 gcc/config/nios2/elf.opt.urls                |   14 +
 gcc/config/nios2/nios2.opt.urls              |   50 +
 gcc/config/nvptx/nvptx-gen.opt.urls          |    2 +
 gcc/config/nvptx/nvptx.opt.urls              |   29 +
 gcc/config/openbsd.opt.urls                  |    6 +
 gcc/config/or1k/elf.opt.urls                 |    8 +
 gcc/config/or1k/or1k.opt.urls                |   46 +
 gcc/config/pa/pa-hpux.opt.urls               |   11 +
 gcc/config/pa/pa-hpux1010.opt.urls           |    2 +
 gcc/config/pa/pa-hpux1111.opt.urls           |    2 +
 gcc/config/pa/pa-hpux1131.opt.urls           |    2 +
 gcc/config/pa/pa.opt.urls                    |   71 +
 gcc/config/pa/pa64-hpux.opt.urls             |    8 +
 gcc/config/pdp11/pdp11.opt.urls              |   41 +
 gcc/config/pru/pru.opt.urls                  |   17 +
 gcc/config/riscv/riscv.opt.urls              |   88 +
 gcc/config/rl78/rl78.opt.urls                |   31 +
 gcc/config/rpath.opt.urls                    |    2 +
 gcc/config/rs6000/476.opt.urls               |    2 +
 gcc/config/rs6000/aix64.opt.urls             |   23 +
 gcc/config/rs6000/darwin.opt.urls            |   14 +
 gcc/config/rs6000/linux64.opt.urls           |    4 +
 gcc/config/rs6000/rs6000-tables.opt.urls     |    2 +
 gcc/config/rs6000/rs6000.opt.urls            |  214 ++
 gcc/config/rs6000/sysv4.opt.urls             |   87 +
 gcc/config/rtems.opt.urls                    |    6 +
 gcc/config/rx/elf.opt.urls                   |   14 +
 gcc/config/rx/rx.opt.urls                    |   54 +
 gcc/config/s390/s390.opt.urls                |   92 +
 gcc/config/s390/tpf.opt.urls                 |    8 +
 gcc/config/sh/sh.opt.urls                    |  174 ++
 gcc/config/sh/superh.opt.urls                |    4 +
 gcc/config/sol2.opt.urls                     |   21 +
 gcc/config/sparc/long-double-switch.opt.urls |    6 +
 gcc/config/sparc/sparc.opt.urls              |  108 ++
 gcc/config/stormy16/stormy16.opt.urls        |    5 +
 gcc/config/v850/v850.opt.urls                |   60 +
 gcc/config/vax/elf.opt.urls                  |    2 +
 gcc/config/vax/vax.opt.urls                  |   10 +
 gcc/config/visium/visium.opt.urls            |   29 +
 gcc/config/vms/vms.opt.urls                  |    8 +
 gcc/config/vxworks-smp.opt.urls              |    5 +
 gcc/config/vxworks.opt.urls                  |   20 +
 gcc/config/xtensa/elf.opt.urls               |    5 +
 gcc/config/xtensa/uclinux.opt.urls           |    2 +
 gcc/config/xtensa/xtensa.opt.urls            |   37 +
 gcc/d/lang.opt.urls                          |  223 +++
 gcc/diagnostic.h                             |    6 +-
 gcc/doc/options.texi                         |   26 +
 gcc/doc/sourcebuild.texi                     |    4 +
 gcc/fortran/lang.opt.urls                    |  161 ++
 gcc/gcc-urlifier.cc                          |  106 +-
 gcc/gcc-urlifier.def                         |    2 -
 gcc/gcc-urlifier.h                           |    2 +-
 gcc/gcc.cc                                   |    2 +-
 gcc/go/lang.opt.urls                         |   17 +
 gcc/lto/lang.opt.urls                        |    8 +
 gcc/m2/lang.opt.urls                         |  115 ++
 gcc/opt-functions.awk                        |   15 +
 gcc/options-urls-cc-gen.awk                  |  105 +
 gcc/opts-diagnostic.h                        |    3 +-
 gcc/opts.cc                                  |   95 +-
 gcc/opts.h                                   |    7 +
 gcc/params.opt.urls                          |    2 +
 gcc/regenerate-opt-urls.py                   |  408 ++++
 gcc/rust/lang.opt.urls                       |   29 +
 gcc/testsuite/lib/gcc-dg.exp                 |    6 +
 gcc/toplev.cc                                |    5 +-
 139 files changed, 9338 insertions(+), 66 deletions(-)
 create mode 100644 gcc/ada/gcc-interface/lang.opt.urls
 create mode 100644 gcc/analyzer/analyzer.opt.urls
 create mode 100644 gcc/c-family/c.opt.urls
 create mode 100644 gcc/common.opt.urls
 create mode 100644 gcc/config/aarch64/aarch64.opt.urls
 create mode 100644 gcc/config/alpha/alpha.opt.urls
 create mode 100644 gcc/config/alpha/elf.opt.urls
 create mode 100644 gcc/config/arc/arc-tables.opt.urls
 create mode 100644 gcc/config/arc/arc.opt.urls
 create mode 100644 gcc/config/arm/arm-tables.opt.urls
 create mode 100644 gcc/config/arm/arm.opt.urls
 create mode 100644 gcc/config/arm/vxworks.opt.urls
 create mode 100644 gcc/config/avr/avr.opt.urls
 create mode 100644 gcc/config/bfin/bfin.opt.urls
 create mode 100644 gcc/config/bpf/bpf.opt.urls
 create mode 100644 gcc/config/c6x/c6x-tables.opt.urls
 create mode 100644 gcc/config/c6x/c6x.opt.urls
 create mode 100644 gcc/config/cris/cris.opt.urls
 create mode 100644 gcc/config/cris/elf.opt.urls
 create mode 100644 gcc/config/csky/csky.opt.urls
 create mode 100644 gcc/config/csky/csky_tables.opt.urls
 create mode 100644 gcc/config/darwin.opt.urls
 create mode 100644 gcc/config/dragonfly.opt.urls
 create mode 100644 gcc/config/epiphany/epiphany.opt.urls
 create mode 100644 gcc/config/fr30/fr30.opt.urls
 create mode 100644 gcc/config/freebsd.opt.urls
 create mode 100644 gcc/config/frv/frv.opt.urls
 create mode 100644 gcc/config/ft32/ft32.opt.urls
 create mode 100644 gcc/config/fused-madd.opt.urls
 create mode 100644 gcc/config/g.opt.urls
 create mode 100644 gcc/config/gcn/gcn.opt.urls
 create mode 100644 gcc/config/gnu-user.opt.urls
 create mode 100644 gcc/config/h8300/h8300.opt.urls
 create mode 100644 gcc/config/hpux11.opt.urls
 create mode 100644 gcc/config/i386/cygming.opt.urls
 create mode 100644 gcc/config/i386/cygwin.opt.urls
 create mode 100644 gcc/config/i386/djgpp.opt.urls
 create mode 100644 gcc/config/i386/i386.opt.urls
 create mode 100644 gcc/config/i386/mingw-w64.opt.urls
 create mode 100644 gcc/config/i386/mingw.opt.urls
 create mode 100644 gcc/config/i386/nto.opt.urls
 create mode 100644 gcc/config/ia64/ia64.opt.urls
 create mode 100644 gcc/config/ia64/ilp32.opt.urls
 create mode 100644 gcc/config/ia64/vms.opt.urls
 create mode 100644 gcc/config/iq2000/iq2000.opt.urls
 create mode 100644 gcc/config/linux-android.opt.urls
 create mode 100644 gcc/config/linux.opt.urls
 create mode 100644 gcc/config/lm32/lm32.opt.urls
 create mode 100644 gcc/config/loongarch/loongarch.opt.urls
 create mode 100644 gcc/config/lynx.opt.urls
 create mode 100644 gcc/config/m32c/m32c.opt.urls
 create mode 100644 gcc/config/m32r/m32r.opt.urls
 create mode 100644 gcc/config/m68k/ieee.opt.urls
 create mode 100644 gcc/config/m68k/m68k-tables.opt.urls
 create mode 100644 gcc/config/m68k/m68k.opt.urls
 create mode 100644 gcc/config/m68k/uclinux.opt.urls
 create mode 100644 gcc/config/mcore/mcore.opt.urls
 create mode 100644 gcc/config/microblaze/microblaze.opt.urls
 create mode 100644 gcc/config/mips/mips-tables.opt.urls
 create mode 100644 gcc/config/mips/mips.opt.urls
 create mode 100644 gcc/config/mips/sde.opt.urls
 create mode 100644 gcc/config/mmix/mmix.opt.urls
 create mode 100644 gcc/config/mn10300/mn10300.opt.urls
 create mode 100644 gcc/config/moxie/moxie.opt.urls
 create mode 100644 gcc/config/msp430/msp430.opt.urls
 create mode 100644 gcc/config/nds32/nds32-elf.opt.urls
 create mode 100644 gcc/config/nds32/nds32-linux.opt.urls
 create mode 100644 gcc/config/nds32/nds32.opt.urls
 create mode 100644 gcc/config/netbsd-elf.opt.urls
 create mode 100644 gcc/config/netbsd.opt.urls
 create mode 100644 gcc/config/nios2/elf.opt.urls
 create mode 100644 gcc/config/nios2/nios2.opt.urls
 create mode 100644 gcc/config/nvptx/nvptx-gen.opt.urls
 create mode 100644 gcc/config/nvptx/nvptx.opt.urls
 create mode 100644 gcc/config/openbsd.opt.urls
 create mode 100644 gcc/config/or1k/elf.opt.urls
 create mode 100644 gcc/config/or1k/or1k.opt.urls
 create mode 100644 gcc/config/pa/pa-hpux.opt.urls
 create mode 100644 gcc/config/pa/pa-hpux1010.opt.urls
 create mode 100644 gcc/config/pa/pa-hpux1111.opt.urls
 create mode 100644 gcc/config/pa/pa-hpux1131.opt.urls
 create mode 100644 gcc/config/pa/pa.opt.urls
 create mode 100644 gcc/config/pa/pa64-hpux.opt.urls
 create mode 100644 gcc/config/pdp11/pdp11.opt.urls
 create mode 100644 gcc/config/pru/pru.opt.urls
 create mode 100644 gcc/config/riscv/riscv.opt.urls
 create mode 100644 gcc/config/rl78/rl78.opt.urls
 create mode 100644 gcc/config/rpath.opt.urls
 create mode 100644 gcc/config/rs6000/476.opt.urls
 create mode 100644 gcc/config/rs6000/aix64.opt.urls
 create mode 100644 gcc/config/rs6000/darwin.opt.urls
 create mode 100644 gcc/config/rs6000/linux64.opt.urls
 create mode 100644 gcc/config/rs6000/rs6000-tables.opt.urls
 create mode 100644 gcc/config/rs6000/rs6000.opt.urls
 create mode 100644 gcc/config/rs6000/sysv4.opt.urls
 create mode 100644 gcc/config/rtems.opt.urls
 create mode 100644 gcc/config/rx/elf.opt.urls
 create mode 100644 gcc/config/rx/rx.opt.urls
 create mode 100644 gcc/config/s390/s390.opt.urls
 create mode 100644 gcc/config/s390/tpf.opt.urls
 create mode 100644 gcc/config/sh/sh.opt.urls
 create mode 100644 gcc/config/sh/superh.opt.urls
 create mode 100644 gcc/config/sol2.opt.urls
 create mode 100644 gcc/config/sparc/long-double-switch.opt.urls
 create mode 100644 gcc/config/sparc/sparc.opt.urls
 create mode 100644 gcc/config/stormy16/stormy16.opt.urls
 create mode 100644 gcc/config/v850/v850.opt.urls
 create mode 100644 gcc/config/vax/elf.opt.urls
 create mode 100644 gcc/config/vax/vax.opt.urls
 create mode 100644 gcc/config/visium/visium.opt.urls
 create mode 100644 gcc/config/vms/vms.opt.urls
 create mode 100644 gcc/config/vxworks-smp.opt.urls
 create mode 100644 gcc/config/vxworks.opt.urls
 create mode 100644 gcc/config/xtensa/elf.opt.urls
 create mode 100644 gcc/config/xtensa/uclinux.opt.urls
 create mode 100644 gcc/config/xtensa/xtensa.opt.urls
 create mode 100644 gcc/d/lang.opt.urls
 create mode 100644 gcc/fortran/lang.opt.urls
 create mode 100644 gcc/go/lang.opt.urls
 create mode 100644 gcc/lto/lang.opt.urls
 create mode 100644 gcc/m2/lang.opt.urls
 create mode 100644 gcc/options-urls-cc-gen.awk
 create mode 100644 gcc/params.opt.urls
 create mode 100755 gcc/regenerate-opt-urls.py
 create mode 100644 gcc/rust/lang.opt.urls
  

> From: David Malcolm <dmalcolm@redhat.com>
> Date: Thu, 16 Nov 2023 09:28:54 -0500

> How is this looking for trunk?
> 
> Thanks
> Dave
> 
> 
> David Malcolm (4):
>   options: add gcc/regenerate-opt-urls.py
>   Add generated .opt.urls files
>   opts: add logic to generate options-urls.cc
>   options: wire up options-urls.cc into gcc_urlifier
> 
>  gcc/Makefile.in                              |   29 +-
>  gcc/ada/gcc-interface/lang.opt.urls          |   30 +
>  gcc/analyzer/analyzer.opt.urls               |  206 ++
>  gcc/c-family/c.opt.urls                      | 1409 ++++++++++++++
>  gcc/common.opt.urls                          | 1832 ++++++++++++++++++
>  gcc/config/aarch64/aarch64.opt.urls          |   84 +
>  gcc/config/alpha/alpha.opt.urls              |   76 +
>  gcc/config/alpha/elf.opt.urls                |    2 +
>  gcc/config/arc/arc-tables.opt.urls           |    2 +

[... etc .opt.urls particularly in gcc/config/*
autogenerated for each *.opt ...]

Sorry for barging in though I did try finding the relevant
discussion, but is committing this generated stuff necessary?
Is it because we don't want to depend on Python being
present at build time?

If nothing else, can you add a few lines to the commit
message why this can't be/is preferably not done at build
time?

brgds, H-P
  

On Tue, 2023-11-21 at 02:09 +0100, Hans-Peter Nilsson wrote:
> > From: David Malcolm <dmalcolm@redhat.com>
> > Date: Thu, 16 Nov 2023 09:28:54 -0500
> 
> > How is this looking for trunk?
> > 
> > Thanks
> > Dave
> > 
> > 
> > David Malcolm (4):
> >   options: add gcc/regenerate-opt-urls.py
> >   Add generated .opt.urls files
> >   opts: add logic to generate options-urls.cc
> >   options: wire up options-urls.cc into gcc_urlifier
> > 
> >  gcc/Makefile.in                              |   29 +-
> >  gcc/ada/gcc-interface/lang.opt.urls          |   30 +
> >  gcc/analyzer/analyzer.opt.urls               |  206 ++
> >  gcc/c-family/c.opt.urls                      | 1409 ++++++++++++++
> >  gcc/common.opt.urls                          | 1832
> > ++++++++++++++++++
> >  gcc/config/aarch64/aarch64.opt.urls          |   84 +
> >  gcc/config/alpha/alpha.opt.urls              |   76 +
> >  gcc/config/alpha/elf.opt.urls                |    2 +
> >  gcc/config/arc/arc-tables.opt.urls           |    2 +
> 
> [... etc .opt.urls particularly in gcc/config/*
> autogenerated for each *.opt ...]
> 
> Sorry for barging in though I did try finding the relevant
> discussion, but is committing this generated stuff necessary?
> Is it because we don't want to depend on Python being
> present at build time?

Partly, yes, but also because this information is generated from the
generated HTML for the user manuals: gcc, gdc, and gfortran, and it
gets used to generate "optionlist", which is a dependency for all of
the rest of the "gcc" subdirectory.  Doing it automatically would make
the generation of the HTML docs for gcc, d and gfortran be a hard
dependency early on in the build, which seems to me to be a bad idea -
better to have this rarely-changing and non-critical information be
regenerated when it's needed, and not impose the requirement to build
the HTML docs for all these langs on all builds of gcc.

> 
> If nothing else, can you add a few lines to the commit
> message why this can't be/is preferably not done at build
> time?

The wording here:
https://gcc.gnu.org/pipermail/gcc-patches/2023-November/636060.html
may be what you're looking for.

Maybe that text (or something like it) should go in as a big comment at
the top of regenerate-opt-urls.py ?

Hope this clarifies things
Dave
  

On 21.11.23 14:57, David Malcolm wrote:
> On Tue, 2023-11-21 at 02:09 +0100, Hans-Peter Nilsson wrote:
>> Sorry for barging in though I did try finding the relevant
>> discussion, but is committing this generated stuff necessary?
>> Is it because we don't want to depend on Python being
>> present at build time?
> Partly, yes, [...]

I wonder how to ensure that this remains up to date. Should there be an
item at

https://gcc.gnu.org/branching.html and/or
https://gcc.gnu.org/releasing.html similar to the .pot generation?

Tobias

-----------------
Siemens Electronic Design Automation GmbH; Anschrift: Arnulfstraße 201, 80634 München; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Thomas Heurung, Frank Thürauf; Sitz der Gesellschaft: München; Registergericht München, HRB 106955
  

On Tue, 2023-11-21 at 15:12 +0100, Tobias Burnus wrote:
> On 21.11.23 14:57, David Malcolm wrote:
> > On Tue, 2023-11-21 at 02:09 +0100, Hans-Peter Nilsson wrote:
> > > Sorry for barging in though I did try finding the relevant
> > > discussion, but is committing this generated stuff necessary?
> > > Is it because we don't want to depend on Python being
> > > present at build time?
> > Partly, yes, [...]
> 
> I wonder how to ensure that this remains up to date. Should there be
> an
> item at
> 
> https://gcc.gnu.org/branching.html and/or
> https://gcc.gnu.org/releasing.html similar to the .pot generation?

Good point; the releasing.html's Preparations could have as point 6:

  Regenerate the .opt.urls files by running "make regenerate-opt-urls"
in the build/gcc directory.

or similar.

We should also probably recommend that people do that when adding a new
option; is there a documentation page for that?

Dave
  

On Tue, 21 Nov 2023, Tobias Burnus wrote:

> On 21.11.23 14:57, David Malcolm wrote:
> > On Tue, 2023-11-21 at 02:09 +0100, Hans-Peter Nilsson wrote:
> > > Sorry for barging in though I did try finding the relevant
> > > discussion, but is committing this generated stuff necessary?
> > > Is it because we don't want to depend on Python being
> > > present at build time?
> > Partly, yes, [...]
> 
> I wonder how to ensure that this remains up to date. Should there be an
> item at
> 
> https://gcc.gnu.org/branching.html and/or
> https://gcc.gnu.org/releasing.html similar to the .pot generation?

My suggestion earlier in the discussion was that it should be added to the 
post-commit CI discussed starting at 
<https://gcc.gnu.org/pipermail/gcc/2023-November/242835.html> (I think 
that CI is now in operation).  These are generated files that ought to be 
kept up to date with each commit that affects .opt files, unlike the .pot 
files where the expectation is that they should be up to date for releases 
and updated from time to time at other times for submission to the TP.
  

On Tue, 2023-11-21 at 23:43 +0000, Joseph Myers wrote:
> On Tue, 21 Nov 2023, Tobias Burnus wrote:
> 
> > On 21.11.23 14:57, David Malcolm wrote:
> > > On Tue, 2023-11-21 at 02:09 +0100, Hans-Peter Nilsson wrote:
> > > > Sorry for barging in though I did try finding the relevant
> > > > discussion, but is committing this generated stuff necessary?
> > > > Is it because we don't want to depend on Python being
> > > > present at build time?
> > > Partly, yes, [...]
> > 
> > I wonder how to ensure that this remains up to date. Should there
> > be an
> > item at
> > 
> > https://gcc.gnu.org/branching.html and/or
> > https://gcc.gnu.org/releasing.html similar to the .pot generation?
> 
> My suggestion earlier in the discussion was that it should be added
> to the 
> post-commit CI discussed starting at 
> <https://gcc.gnu.org/pipermail/gcc/2023-November/242835.html> (I
> think 
> that CI is now in operation).  These are generated files that ought
> to be 
> kept up to date with each commit that affects .opt files, unlike the
> .pot 
> files where the expectation is that they should be up to date for
> releases 
> and updated from time to time at other times for submission to the
> TP.

I had a go at scripting the testing of this, but I am terrible at shell
scripts (maybe I should use Python?).  Here's what I have so far:

$ cat contrib/regenerate-index-urls.sh

set -x

SRC_DIR=$1
BUILD_DIR=$2
NUM_JOBS=$3

# FIXME: error-checking!

mkdir -p $BUILD_DIR || exit 1
cd $BUILD_DIR
$SRC_DIR/configure --disable-bootstrap --enable-languages=c,d,fortran || exit 2
make html-gcc -j$NUM_JOBS || exit 3
cd gcc || exit 4
make regenerate-opt-urls || exit 5
cd $SRC_DIR
(git diff $1 > /dev/null ) && echo "regenerate-opt-urls needs to be run and the results committed" || exit 6

# e.g.
#  time bash contrib/regenerate-index-urls.sh $(pwd) $(pwd)/../build-ci 64

This takes about 100 seconds of wallclock on my 64-core box (mostly
configuring gcc, which as well as the usual sequence of unparallelized
tests seems to require building libiberty and lto-plugin).  Is that
something we want to do on every commit?  Is implementing the CI a
blocker for getting the patches in? (if so, I'll likely need some help)

As it turned out, I hadn't regenerated the .opt.urls in my working copy
for a couple of weeks, leading to a correct-looking patch containing
things like:

@@ -154,8 +157,8 @@ UrlSuffix(gcc/Warning-Options.html#index-Wbuiltin-declaration-mismatch) LangUrlS
 Wbuiltin-macro-redefined
 UrlSuffix(gcc/Warning-Options.html#index-Wbuiltin-macro-redefined)
 
-Wc11-c2x-compat
-UrlSuffix(gcc/Warning-Options.html#index-Wc11-c2x-compat)
+Wc11-c23-compat
+UrlSuffix(gcc/Warning-Options.html#index-Wc11-c23-compat)
 
 Wc90-c99-compat
 UrlSuffix(gcc/Warning-Options.html#index-Wc90-c99-compat)

so I think the idea works; and the only issue for not regenerating was
some missing/out-of-date URLs.

Dave
  

Hi David,

On Fri, Dec 08, 2023 at 06:35:56PM -0500, David Malcolm wrote:
> On Tue, 2023-11-21 at 23:43 +0000, Joseph Myers wrote:
> > On Tue, 21 Nov 2023, Tobias Burnus wrote:
> > 
> > > On 21.11.23 14:57, David Malcolm wrote:
> > > > On Tue, 2023-11-21 at 02:09 +0100, Hans-Peter Nilsson wrote:
> > > > > Sorry for barging in though I did try finding the relevant
> > > > > discussion, but is committing this generated stuff necessary?
> > > > > Is it because we don't want to depend on Python being
> > > > > present at build time?
> > > > Partly, yes, [...]
> > > 
> > > I wonder how to ensure that this remains up to date. Should there
> > > be an item at
> > > 
> > > https://gcc.gnu.org/branching.html and/or
> > > https://gcc.gnu.org/releasing.html similar to the .pot generation?
> >
> > My suggestion earlier in the discussion was that it should be
> > added to the post-commit CI discussed starting at
> > <https://gcc.gnu.org/pipermail/gcc/2023-November/242835.html> (I
> > think that CI is now in operation).  These are generated files
> > that ought to be kept up to date with each commit that affects
> > .opt files, unlike the .pot files where the expectation is that
> > they should be up to date for releases and updated from time to
> > time at other times for submission to the TP.
> > 
> I had a go at scripting the testing of this, but I am terrible at shell
> scripts (maybe I should use Python?).  Here's what I have so far:
> 
> $ cat contrib/regenerate-index-urls.sh
> 
> set -x
> 
> SRC_DIR=$1
> BUILD_DIR=$2
> NUM_JOBS=$3
> 
> # FIXME: error-checking!
> 
> mkdir -p $BUILD_DIR || exit 1
> cd $BUILD_DIR
> $SRC_DIR/configure --disable-bootstrap --enable-languages=c,d,fortran || exit 2
> make html-gcc -j$NUM_JOBS || exit 3
> cd gcc || exit 4
> make regenerate-opt-urls || exit 5
> cd $SRC_DIR
> (git diff $1 > /dev/null ) && echo "regenerate-opt-urls needs to be run and the results committed" || exit 6
> 
> # e.g.
> #  time bash contrib/regenerate-index-urls.sh $(pwd) $(pwd)/../build-ci 64
> 
> This takes about 100 seconds of wallclock on my 64-core box (mostly
> configuring gcc, which as well as the usual sequence of unparallelized
> tests seems to require building libiberty and lto-plugin).  Is that
> something we want to do on every commit?  Is implementing the CI a
> blocker for getting the patches in? (if so, I'll likely need some help)

The CI builers don't have 64-cores, but a couple of hundred seconds
shouldn't be an issue to do on each commit (OSUOSL just got us a
second x86_64 container builder for larger jobs). The above can easily
be added to the existing gcc-autoregen builder:
https://builder.sourceware.org/buildbot/#/builders/gcc-autoregen
https://sourceware.org/cgit/builder/tree/builder/master.cfg#n3453

Once your patch is in please feel free to sent an email to
buildbot@sourceware.org
https://sourceware.org/mailman/listinfo/buildbot
And we'll add the above build steps and update the autotools
Containerfile to include the fortran (gfortran?) and d (gdc?) build
dependencies.

Cheers,

Mark