[v2,00/14] Fix perf tools/lib includes

  The previous build would add -Itools/lib and get dependencies for
libtraceevent, libsubcmd, libsymbol, libapi and libbpf meaning that
overriding these libraries would change the link time dependency but
the headers would erroneously come from tools/lib. Fix the build to
install headers and then depend on these. To reduce exposing internal
headers/APIs some clean up is performed. tools/lib/symbol has a
Makefile added, while tools/lib/api and tools/lib/subcmd have install
targets added. The pattern used for the dependencies in Makefile.perf
is modelled on libbpf.

The problem and solution were motivated by this issue and discussion:
https://lore.kernel.org/lkml/CAEf4BzbbOHQZUAe6iWaehKCPQAf3VC=hq657buqe2_yRKxaK-A@mail.gmail.com/

v2. Fix a MANIFEST issue for the source tar ball. Add dependencies for
    the installed header files so that the build doesn't overtake
    building these dependencies. Both issues reported by Arnaldo
    Carvalho de Melo <acme@kernel.org>.

Ian Rogers (14):
  tools lib api: Add install target
  tools lib subcmd: Add install target
  perf build: Install libsubcmd locally when building
  perf build: Install libapi locally when building
  perf build: Install libperf locally when building
  perf build: Install libtraceevent locally when building
  tools lib api: Add missing install headers
  tools lib perf: Add missing install headers
  tool lib symbol: Add Makefile/Build
  perf build: Install libsymbol locally when building
  perf expr: Tidy hashmap dependency
  perf thread_map: Reduce exposure of libperf internal API
  perf cpumap: Tidy libperf includes
  perf build: Use tools/lib headers from install path

 tools/lib/api/Makefile                        |  52 +++++
 tools/lib/perf/Makefile                       |  10 +-
 tools/lib/subcmd/Makefile                     |  49 +++++
 tools/lib/symbol/Build                        |   1 +
 tools/lib/symbol/Makefile                     | 115 +++++++++++
 tools/perf/.gitignore                         |   7 +-
 tools/perf/MANIFEST                           |   3 +-
 tools/perf/Makefile.config                    |   2 -
 tools/perf/Makefile.perf                      | 192 ++++++++++++++----
 tools/perf/builtin-stat.c                     |   1 +
 tools/perf/builtin-trace.c                    |   4 +-
 tools/perf/tests/cpumap.c                     |   2 +-
 tools/perf/tests/expr.c                       |   1 +
 tools/perf/tests/openat-syscall.c             |   1 +
 tools/perf/tests/pmu-events.c                 |   1 +
 tools/perf/tests/thread-map.c                 |   1 +
 tools/perf/util/Build                         |   5 -
 tools/perf/util/auxtrace.h                    |   2 +-
 tools/perf/util/bpf-loader.c                  |   4 -
 tools/perf/util/bpf_counter.c                 |   2 +-
 tools/perf/util/cpumap.c                      |   1 +
 tools/perf/util/cpumap.h                      |   2 +-
 tools/perf/util/evsel.c                       |   5 +-
 tools/perf/util/evsel.h                       |   2 -
 tools/perf/util/expr.c                        |   1 +
 tools/perf/util/expr.h                        |   7 +-
 tools/perf/util/metricgroup.c                 |   1 +
 tools/perf/util/python.c                      |   6 +-
 .../scripting-engines/trace-event-python.c    |   2 +-
 tools/perf/util/stat-shadow.c                 |   1 +
 tools/perf/util/stat.c                        |   4 -
 tools/perf/util/thread_map.c                  |   1 +
 tools/perf/util/thread_map.h                  |   2 -
 33 files changed, 402 insertions(+), 88 deletions(-)
 create mode 100644 tools/lib/symbol/Build
 create mode 100644 tools/lib/symbol/Makefile
  

Hi Ian,

On Wed, Nov 9, 2022 at 10:49 AM Ian Rogers <irogers@google.com> wrote:
>
> The previous build would add -Itools/lib and get dependencies for
> libtraceevent, libsubcmd, libsymbol, libapi and libbpf meaning that
> overriding these libraries would change the link time dependency but
> the headers would erroneously come from tools/lib. Fix the build to
> install headers and then depend on these. To reduce exposing internal
> headers/APIs some clean up is performed. tools/lib/symbol has a
> Makefile added, while tools/lib/api and tools/lib/subcmd have install
> targets added. The pattern used for the dependencies in Makefile.perf
> is modelled on libbpf.
>
> The problem and solution were motivated by this issue and discussion:
> https://lore.kernel.org/lkml/CAEf4BzbbOHQZUAe6iWaehKCPQAf3VC=hq657buqe2_yRKxaK-A@mail.gmail.com/
>
> v2. Fix a MANIFEST issue for the source tar ball. Add dependencies for
>     the installed header files so that the build doesn't overtake
>     building these dependencies. Both issues reported by Arnaldo
>     Carvalho de Melo <acme@kernel.org>.
>
> Ian Rogers (14):
>   tools lib api: Add install target
>   tools lib subcmd: Add install target
>   perf build: Install libsubcmd locally when building
>   perf build: Install libapi locally when building
>   perf build: Install libperf locally when building
>   perf build: Install libtraceevent locally when building
>   tools lib api: Add missing install headers
>   tools lib perf: Add missing install headers
>   tool lib symbol: Add Makefile/Build
>   perf build: Install libsymbol locally when building
>   perf expr: Tidy hashmap dependency
>   perf thread_map: Reduce exposure of libperf internal API
>   perf cpumap: Tidy libperf includes
>   perf build: Use tools/lib headers from install path

Acked-by: Namhyung Kim <namhyung@kernel.org>

Thanks,
Namhyung


>
>  tools/lib/api/Makefile                        |  52 +++++
>  tools/lib/perf/Makefile                       |  10 +-
>  tools/lib/subcmd/Makefile                     |  49 +++++
>  tools/lib/symbol/Build                        |   1 +
>  tools/lib/symbol/Makefile                     | 115 +++++++++++
>  tools/perf/.gitignore                         |   7 +-
>  tools/perf/MANIFEST                           |   3 +-
>  tools/perf/Makefile.config                    |   2 -
>  tools/perf/Makefile.perf                      | 192 ++++++++++++++----
>  tools/perf/builtin-stat.c                     |   1 +
>  tools/perf/builtin-trace.c                    |   4 +-
>  tools/perf/tests/cpumap.c                     |   2 +-
>  tools/perf/tests/expr.c                       |   1 +
>  tools/perf/tests/openat-syscall.c             |   1 +
>  tools/perf/tests/pmu-events.c                 |   1 +
>  tools/perf/tests/thread-map.c                 |   1 +
>  tools/perf/util/Build                         |   5 -
>  tools/perf/util/auxtrace.h                    |   2 +-
>  tools/perf/util/bpf-loader.c                  |   4 -
>  tools/perf/util/bpf_counter.c                 |   2 +-
>  tools/perf/util/cpumap.c                      |   1 +
>  tools/perf/util/cpumap.h                      |   2 +-
>  tools/perf/util/evsel.c                       |   5 +-
>  tools/perf/util/evsel.h                       |   2 -
>  tools/perf/util/expr.c                        |   1 +
>  tools/perf/util/expr.h                        |   7 +-
>  tools/perf/util/metricgroup.c                 |   1 +
>  tools/perf/util/python.c                      |   6 +-
>  .../scripting-engines/trace-event-python.c    |   2 +-
>  tools/perf/util/stat-shadow.c                 |   1 +
>  tools/perf/util/stat.c                        |   4 -
>  tools/perf/util/thread_map.c                  |   1 +
>  tools/perf/util/thread_map.h                  |   2 -
>  33 files changed, 402 insertions(+), 88 deletions(-)
>  create mode 100644 tools/lib/symbol/Build
>  create mode 100644 tools/lib/symbol/Makefile
>
> --
> 2.38.1.431.g37b22c650d-goog
>
  

Em Thu, Nov 10, 2022 at 10:10:18AM -0800, Namhyung Kim escreveu:
> Hi Ian,
> 
> On Wed, Nov 9, 2022 at 10:49 AM Ian Rogers <irogers@google.com> wrote:
> >
> > The previous build would add -Itools/lib and get dependencies for
> > libtraceevent, libsubcmd, libsymbol, libapi and libbpf meaning that
> > overriding these libraries would change the link time dependency but
> > the headers would erroneously come from tools/lib. Fix the build to
> > install headers and then depend on these. To reduce exposing internal
> > headers/APIs some clean up is performed. tools/lib/symbol has a
> > Makefile added, while tools/lib/api and tools/lib/subcmd have install
> > targets added. The pattern used for the dependencies in Makefile.perf
> > is modelled on libbpf.
> >
> > The problem and solution were motivated by this issue and discussion:
> > https://lore.kernel.org/lkml/CAEf4BzbbOHQZUAe6iWaehKCPQAf3VC=hq657buqe2_yRKxaK-A@mail.gmail.com/
> >
> > v2. Fix a MANIFEST issue for the source tar ball. Add dependencies for
> >     the installed header files so that the build doesn't overtake
> >     building these dependencies. Both issues reported by Arnaldo
> >     Carvalho de Melo <acme@kernel.org>.
> >
> > Ian Rogers (14):
> >   tools lib api: Add install target
> >   tools lib subcmd: Add install target
> >   perf build: Install libsubcmd locally when building
> >   perf build: Install libapi locally when building
> >   perf build: Install libperf locally when building
> >   perf build: Install libtraceevent locally when building
> >   tools lib api: Add missing install headers
> >   tools lib perf: Add missing install headers
> >   tool lib symbol: Add Makefile/Build
> >   perf build: Install libsymbol locally when building
> >   perf expr: Tidy hashmap dependency
> >   perf thread_map: Reduce exposure of libperf internal API
> >   perf cpumap: Tidy libperf includes
> >   perf build: Use tools/lib headers from install path
> 
> Acked-by: Namhyung Kim <namhyung@kernel.org>

'make -C tools/perf build-test' is failing, looks like parallelization
woes again:

⬢[acme@toolbox perf]$ getconf _NPROCESSORS_ONLN
32
⬢[acme@toolbox perf]$

⬢[acme@toolbox perf]$ git log --oneline -1 ; time make -C tools/perf build-test
9f22d36a1dd0297a (HEAD -> perf/core) perf build: Use tools/lib headers from install path
make: Entering directory '/var/home/acme/git/perf/tools/perf'
- tarpkg: ./tests/perf-targz-src-pkg .
                 make_static: cd . && make LDFLAGS=-static NO_PERF_READ_VDSO32=1 NO_PERF_READ_VDSOX32=1 NO_JVMTI=1 -j32  DESTDIR=/tmp/tmp.i0kUDsiQbV
              make_with_gtk2: cd . && make GTK2=1 -j32  DESTDIR=/tmp/tmp.MNcgfeG07c
cd . && make GTK2=1 -j32 DESTDIR=/tmp/tmp.MNcgfeG07c
  BUILD:   Doing 'make -j32' parallel build
  HOSTCC  fixdep.o
  HOSTLD  fixdep-in.o
  LINK    fixdep
Warning: Kernel ABI header at 'tools/arch/arm64/include/uapi/asm/perf_regs.h' differs from latest version at 'arch/arm64/include/uapi/asm/perf_regs.h'
diff -u tools/arch/arm64/include/uapi/asm/perf_regs.h arch/arm64/include/uapi/asm/perf_regs.h
Warning: Kernel ABI header at 'tools/arch/arm64/include/asm/cputype.h' differs from latest version at 'arch/arm64/include/asm/cputype.h'
diff -u tools/arch/arm64/include/asm/cputype.h arch/arm64/include/asm/cputype.h
Warning: Kernel ABI header at 'tools/perf/arch/powerpc/entry/syscalls/syscall.tbl' differs from latest version at 'arch/powerpc/kernel/syscalls/syscall.tbl'
diff -u tools/perf/arch/powerpc/entry/syscalls/syscall.tbl arch/powerpc/kernel/syscalls/syscall.tbl

Auto-detecting system features:
...                                   dwarf: [ on  ]
...                      dwarf_getlocations: [ on  ]
...                                   glibc: [ on  ]
...                                  libbfd: [ on  ]
...                          libbfd-buildid: [ on  ]
...                                  libcap: [ on  ]
...                                  libelf: [ on  ]
...                                 libnuma: [ on  ]
...                  numa_num_possible_cpus: [ on  ]
...                                 libperl: [ on  ]
...                               libpython: [ on  ]
...                               libcrypto: [ on  ]
...                               libunwind: [ on  ]
...                      libdw-dwarf-unwind: [ on  ]
...                                    zlib: [ on  ]
...                                    lzma: [ on  ]
...                               get_cpuid: [ on  ]
...                                     bpf: [ on  ]
...                                  libaio: [ on  ]
...                                 libzstd: [ on  ]

  GEN     common-cmds.h
  CC      perf-read-vdso32
  CC      dlfilters/dlfilter-test-api-v0.o
  CC      dlfilters/dlfilter-show-cycles.o
  CC      jvmti/libjvmti.o
  CC      jvmti/jvmti_agent.o
  CC      jvmti/libstring.o
  CC      jvmti/libctype.o
  CC      ui/gtk/browser.o
  CC      ui/gtk/hists.o
  CC      ui/gtk/setup.o
  CC      ui/gtk/util.o
  GEN     pmu-events/pmu-events.c
  CC      ui/gtk/helpline.o
  LINK    dlfilters/dlfilter-show-cycles.so
  CC      ui/gtk/progress.o
  CC      ui/gtk/annotate.o
  CC      ui/gtk/zalloc.o
  LINK    dlfilters/dlfilter-test-api-v0.so
  INSTALL headers
  CC      /var/home/acme/git/perf/tools/perf/libsubcmd/exec-cmd.o
  CC      /var/home/acme/git/perf/tools/perf/libsubcmd/help.o
  CC      /var/home/acme/git/perf/tools/perf/libsubcmd/pager.o
  CC      /var/home/acme/git/perf/tools/perf/libsubcmd/parse-options.o
  CC      /var/home/acme/git/perf/tools/perf/libsubcmd/run-command.o
In file included from ui/gtk/hists.c:2:
/var/home/acme/git/perf/tools/perf/util/include/../evlist.h:9:10: fatal error: api/fd/array.h: No such file or directory
    9 | #include <api/fd/array.h>
      |          ^~~~~~~~~~~~~~~~
compilation terminated.
make[6]: *** [/var/home/acme/git/perf/tools/build/Makefile.build:97: ui/gtk/hists.o] Error 1
make[6]: *** Waiting for unfinished jobs....
  CC      /var/home/acme/git/perf/tools/perf/libsubcmd/sigchain.o
make -f /var/home/acme/git/perf/tools/build/Makefile.build dir=. obj=libsymbol V=1
In file included from ui/gtk/browser.c:2:
/var/home/acme/git/perf/tools/perf/util/include/../evsel.h:10:10: fatal error: internal/evsel.h: No such file or directory
   10 | #include <internal/evsel.h>
      |          ^~~~~~~~~~~~~~~~~~
compilation terminated.
make[6]: *** [/var/home/acme/git/perf/tools/build/Makefile.build:97: ui/gtk/browser.o] Error 1
  INSTALL headers
  CC      /var/home/acme/git/perf/tools/perf/libtraceevent/event-parse.o
  

Em Mon, Nov 14, 2022 at 05:16:57PM -0300, Arnaldo Carvalho de Melo escreveu:
> Em Thu, Nov 10, 2022 at 10:10:18AM -0800, Namhyung Kim escreveu:
> > Hi Ian,
> > 
> > On Wed, Nov 9, 2022 at 10:49 AM Ian Rogers <irogers@google.com> wrote:
> > >
> > > The previous build would add -Itools/lib and get dependencies for
> > > libtraceevent, libsubcmd, libsymbol, libapi and libbpf meaning that
> > > overriding these libraries would change the link time dependency but
> > > the headers would erroneously come from tools/lib. Fix the build to
> > > install headers and then depend on these. To reduce exposing internal
> > > headers/APIs some clean up is performed. tools/lib/symbol has a
> > > Makefile added, while tools/lib/api and tools/lib/subcmd have install
> > > targets added. The pattern used for the dependencies in Makefile.perf
> > > is modelled on libbpf.
> > >
> > > The problem and solution were motivated by this issue and discussion:
> > > https://lore.kernel.org/lkml/CAEf4BzbbOHQZUAe6iWaehKCPQAf3VC=hq657buqe2_yRKxaK-A@mail.gmail.com/
> > >
> > > v2. Fix a MANIFEST issue for the source tar ball. Add dependencies for
> > >     the installed header files so that the build doesn't overtake
> > >     building these dependencies. Both issues reported by Arnaldo
> > >     Carvalho de Melo <acme@kernel.org>.
> > >
> > > Ian Rogers (14):
> > >   tools lib api: Add install target
> > >   tools lib subcmd: Add install target
> > >   perf build: Install libsubcmd locally when building
> > >   perf build: Install libapi locally when building
> > >   perf build: Install libperf locally when building
> > >   perf build: Install libtraceevent locally when building
> > >   tools lib api: Add missing install headers
> > >   tools lib perf: Add missing install headers
> > >   tool lib symbol: Add Makefile/Build
> > >   perf build: Install libsymbol locally when building
> > >   perf expr: Tidy hashmap dependency
> > >   perf thread_map: Reduce exposure of libperf internal API
> > >   perf cpumap: Tidy libperf includes
> > >   perf build: Use tools/lib headers from install path
> > 
> > Acked-by: Namhyung Kim <namhyung@kernel.org>
> 
> 'make -C tools/perf build-test' is failing, looks like parallelization
> woes again:
 
It's all at the perf/tools-libs-includes branch in my tree.

- Arnaldo

> ⬢[acme@toolbox perf]$ getconf _NPROCESSORS_ONLN
> 32
> ⬢[acme@toolbox perf]$
> 
> ⬢[acme@toolbox perf]$ git log --oneline -1 ; time make -C tools/perf build-test
> 9f22d36a1dd0297a (HEAD -> perf/core) perf build: Use tools/lib headers from install path
> make: Entering directory '/var/home/acme/git/perf/tools/perf'
> - tarpkg: ./tests/perf-targz-src-pkg .
>                  make_static: cd . && make LDFLAGS=-static NO_PERF_READ_VDSO32=1 NO_PERF_READ_VDSOX32=1 NO_JVMTI=1 -j32  DESTDIR=/tmp/tmp.i0kUDsiQbV
>               make_with_gtk2: cd . && make GTK2=1 -j32  DESTDIR=/tmp/tmp.MNcgfeG07c
> cd . && make GTK2=1 -j32 DESTDIR=/tmp/tmp.MNcgfeG07c
>   BUILD:   Doing 'make -j32' parallel build
>   HOSTCC  fixdep.o
>   HOSTLD  fixdep-in.o
>   LINK    fixdep
> Warning: Kernel ABI header at 'tools/arch/arm64/include/uapi/asm/perf_regs.h' differs from latest version at 'arch/arm64/include/uapi/asm/perf_regs.h'
> diff -u tools/arch/arm64/include/uapi/asm/perf_regs.h arch/arm64/include/uapi/asm/perf_regs.h
> Warning: Kernel ABI header at 'tools/arch/arm64/include/asm/cputype.h' differs from latest version at 'arch/arm64/include/asm/cputype.h'
> diff -u tools/arch/arm64/include/asm/cputype.h arch/arm64/include/asm/cputype.h
> Warning: Kernel ABI header at 'tools/perf/arch/powerpc/entry/syscalls/syscall.tbl' differs from latest version at 'arch/powerpc/kernel/syscalls/syscall.tbl'
> diff -u tools/perf/arch/powerpc/entry/syscalls/syscall.tbl arch/powerpc/kernel/syscalls/syscall.tbl
> 
> Auto-detecting system features:
> ...                                   dwarf: [ on  ]
> ...                      dwarf_getlocations: [ on  ]
> ...                                   glibc: [ on  ]
> ...                                  libbfd: [ on  ]
> ...                          libbfd-buildid: [ on  ]
> ...                                  libcap: [ on  ]
> ...                                  libelf: [ on  ]
> ...                                 libnuma: [ on  ]
> ...                  numa_num_possible_cpus: [ on  ]
> ...                                 libperl: [ on  ]
> ...                               libpython: [ on  ]
> ...                               libcrypto: [ on  ]
> ...                               libunwind: [ on  ]
> ...                      libdw-dwarf-unwind: [ on  ]
> ...                                    zlib: [ on  ]
> ...                                    lzma: [ on  ]
> ...                               get_cpuid: [ on  ]
> ...                                     bpf: [ on  ]
> ...                                  libaio: [ on  ]
> ...                                 libzstd: [ on  ]
> 
>   GEN     common-cmds.h
>   CC      perf-read-vdso32
>   CC      dlfilters/dlfilter-test-api-v0.o
>   CC      dlfilters/dlfilter-show-cycles.o
>   CC      jvmti/libjvmti.o
>   CC      jvmti/jvmti_agent.o
>   CC      jvmti/libstring.o
>   CC      jvmti/libctype.o
>   CC      ui/gtk/browser.o
>   CC      ui/gtk/hists.o
>   CC      ui/gtk/setup.o
>   CC      ui/gtk/util.o
>   GEN     pmu-events/pmu-events.c
>   CC      ui/gtk/helpline.o
>   LINK    dlfilters/dlfilter-show-cycles.so
>   CC      ui/gtk/progress.o
>   CC      ui/gtk/annotate.o
>   CC      ui/gtk/zalloc.o
>   LINK    dlfilters/dlfilter-test-api-v0.so
>   INSTALL headers
>   CC      /var/home/acme/git/perf/tools/perf/libsubcmd/exec-cmd.o
>   CC      /var/home/acme/git/perf/tools/perf/libsubcmd/help.o
>   CC      /var/home/acme/git/perf/tools/perf/libsubcmd/pager.o
>   CC      /var/home/acme/git/perf/tools/perf/libsubcmd/parse-options.o
>   CC      /var/home/acme/git/perf/tools/perf/libsubcmd/run-command.o
> In file included from ui/gtk/hists.c:2:
> /var/home/acme/git/perf/tools/perf/util/include/../evlist.h:9:10: fatal error: api/fd/array.h: No such file or directory
>     9 | #include <api/fd/array.h>
>       |          ^~~~~~~~~~~~~~~~
> compilation terminated.
> make[6]: *** [/var/home/acme/git/perf/tools/build/Makefile.build:97: ui/gtk/hists.o] Error 1
> make[6]: *** Waiting for unfinished jobs....
>   CC      /var/home/acme/git/perf/tools/perf/libsubcmd/sigchain.o
> make -f /var/home/acme/git/perf/tools/build/Makefile.build dir=. obj=libsymbol V=1
> In file included from ui/gtk/browser.c:2:
> /var/home/acme/git/perf/tools/perf/util/include/../evsel.h:10:10: fatal error: internal/evsel.h: No such file or directory
>    10 | #include <internal/evsel.h>
>       |          ^~~~~~~~~~~~~~~~~~
> compilation terminated.
> make[6]: *** [/var/home/acme/git/perf/tools/build/Makefile.build:97: ui/gtk/browser.o] Error 1
>   INSTALL headers
>   CC      /var/home/acme/git/perf/tools/perf/libtraceevent/event-parse.o