vmlinux.lds.h: Force-align ELF notes section to four bytes

  On Tue, Feb 14, 2023 at 02:33:02PM +0800, Tianyi Liu wrote:
> > LLVM_OBJCOPY=objcopy pahole -J --btf_gen_floats -j
> > --skip_encoding_btf_inconsistent_proto --btf_gen_optimized
> > .tmp_vmlinux.btf
> > btf_encoder__encode: btf__dedup failed!
> > Failed to encode BTF
> >
> > Thanks,
> >
> 
> I encountered the same problem when building a new kernel and I found some
> reasons for the error.
> 
> In short, enabling CONFIG_X86_KERNEL_IBT will change the order of records in
> .notes section. In addition, due to historical problems, the alignment of
> records in the .notes section is not unified, which leads to the inability of
> gelf_getnote() to read the records after the wrong one.

Alexandre, Tianyi, are you still seeing this issue with the latest
dwarves?  If so can you confirm the below patch fixes it?

Apparently the latest dwarves release fixes it on Fedora Rawhide [1],
does anybody know if there a specific dwarves and/or libbpf change for
this?

[1] https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2346#note_1348057786

---8<---

From: Josh Poimboeuf <jpoimboe@kernel.org>
Subject: [PATCH] vmlinux.lds.h: Force-align ELF notes section to four bytes

When tooling reads ELF notes, it assumes each note entry is aligned to
the value listed in the .note section header's sh_addralign field.

The kernel-created ELF notes in the .note.Linux and .note.Xen sections
are aligned to 4 bytes.  This causes the toolchain to set those
sections' sh_addralign values to 4.

On the other hand, the GCC-created .note.gnu.property section has an
sh_addralign value of 8 for some reason, despite being based on struct
Elf32_Nhdr which only needs 4-byte alignment.

When the mismatched input sections get linked together into the vmlinux
.notes output section, the higher alignment "wins", resulting in an
sh_addralign of 8, which confuses tooling.  For example:

  $ readelf -n .tmp_vmlinux.btf
  ...
  readelf: .tmp_vmlinux.btf: Warning: note with invalid namesz and/or descsz found at offset 0x170
  readelf: .tmp_vmlinux.btf: Warning:  type: 0x4, namesize: 0x006e6558, descsize: 0x00008801, alignment: 8

In this case readelf thinks there's alignment padding where there is
none, so it starts reading an ELF note in the middle.

With newer toolchains (e.g., latest Fedora Rawhide), a similar mismatch
triggers a build failure when combined with CONFIG_X86_KERNEL_IBT:

  btf_encoder__encode: btf__dedup failed!
  Failed to encode BTF
  libbpf: failed to find '.BTF' ELF section in vmlinux
  FAILED: load BTF from vmlinux: No data available
  make[1]: *** [scripts/Makefile.vmlinux:35: vmlinux] Error 255

Fix it by forcing the .notes section input and output alignments to 4 to
match the kernel's note entry alignments.

Note this doesn't break the 8-byte-aligned .note.gnu.property entries
because their internal data representations fill the entire 8-byte
alignment boundary, so there's no padding between entries to be
misinterpreted.  And there's only a single entry in that section anyway.

Reported-by: Daniel Xu <dxu@dxuuu.xyz>
Debugged-by: Tianyi Liu <i.pear@outlook.com>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
---
 include/asm-generic/vmlinux.lds.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
  

On Tue, Apr 11, 2023 at 17:00 , Josh Poimboeuf wrote:
> On Tue, Feb 14, 2023 at 02:33:02PM +0800, Tianyi Liu wrote:
> > > LLVM_OBJCOPY=objcopy pahole -J --btf_gen_floats -j
> > > --skip_encoding_btf_inconsistent_proto --btf_gen_optimized
> > > .tmp_vmlinux.btf
> > > btf_encoder__encode: btf__dedup failed!
> > > Failed to encode BTF
> > >
> > > Thanks,
> > >
> >
> > I encountered the same problem when building a new kernel and I found some
> > reasons for the error.
> >
> > In short, enabling CONFIG_X86_KERNEL_IBT will change the order of records in
> > .notes section. In addition, due to historical problems, the alignment of
> > records in the .notes section is not unified, which leads to the inability of
> > gelf_getnote() to read the records after the wrong one.
> 
> Alexandre, Tianyi, are you still seeing this issue with the latest
> dwarves?  If so can you confirm the below patch fixes it?
> 

Josh, first of all, thank you very much for your help. However, this patch
doesn't work in my environment. I am using gcc 12.2.1 20230201.
After compiling, when I use readelf -S to view ELF sections,
the align of .notes section is still 8:

$ readelf -S .tmp_vmlinux.btf
[20] .notes            NOTE             ffffffff8250b570  0170b570
     0000000000000084  0000000000000000   A       0     0     8

> Apparently the latest dwarves release fixes it on Fedora Rawhide [1],
> does anybody know if there a specific dwarves and/or libbpf change for
> this?
> 

It has been fixed in dwarves[1], but it may just be a coincidence.

> [1] https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2346#note_1348057786
> 
> ---8<---
> 
> From: Josh Poimboeuf <jpoimboe@kernel.org>
> Subject: [PATCH] vmlinux.lds.h: Force-align ELF notes section to four bytes
> 
> When tooling reads ELF notes, it assumes each note entry is aligned to
> the value listed in the .note section header's sh_addralign field.
> 
> The kernel-created ELF notes in the .note.Linux and .note.Xen sections
> are aligned to 4 bytes.  This causes the toolchain to set those
> sections' sh_addralign values to 4.
> 
> On the other hand, the GCC-created .note.gnu.property section has an
> sh_addralign value of 8 for some reason, despite being based on struct
> Elf32_Nhdr which only needs 4-byte alignment.
> 
> When the mismatched input sections get linked together into the vmlinux
> .notes output section, the higher alignment "wins", resulting in an
> sh_addralign of 8, which confuses tooling.  For example:
> 
>   $ readelf -n .tmp_vmlinux.btf
>   ...
>   readelf: .tmp_vmlinux.btf: Warning: note with invalid namesz and/or descsz found at offset 0x170
>   readelf: .tmp_vmlinux.btf: Warning:  type: 0x4, namesize: 0x006e6558, descsize: 0x00008801, alignment: 8
> 
> In this case readelf thinks there's alignment padding where there is
> none, so it starts reading an ELF note in the middle.
> 
> With newer toolchains (e.g., latest Fedora Rawhide), a similar mismatch
> triggers a build failure when combined with CONFIG_X86_KERNEL_IBT:
> 
>   btf_encoder__encode: btf__dedup failed!
>   Failed to encode BTF
>   libbpf: failed to find '.BTF' ELF section in vmlinux
>   FAILED: load BTF from vmlinux: No data available
>   make[1]: *** [scripts/Makefile.vmlinux:35: vmlinux] Error 255
> 
> Fix it by forcing the .notes section input and output alignments to 4 to
> match the kernel's note entry alignments.
> 
> Note this doesn't break the 8-byte-aligned .note.gnu.property entries
> because their internal data representations fill the entire 8-byte
> alignment boundary, so there's no padding between entries to be
> misinterpreted.  And there's only a single entry in that section anyway.
> 
> Reported-by: Daniel Xu <dxu@dxuuu.xyz>
> Debugged-by: Tianyi Liu <i.pear@outlook.com>
> Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
> ---
>  include/asm-generic/vmlinux.lds.h | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
> index d1f57e4868ed..1c7c87c9ae71 100644
> --- a/include/asm-generic/vmlinux.lds.h
> +++ b/include/asm-generic/vmlinux.lds.h
> @@ -894,7 +894,7 @@
>   */
>  #define NOTES                                                           \
>          /DISCARD/ : { *(.note.GNU-stack) }                              \
> -       .notes : AT(ADDR(.notes) - LOAD_OFFSET) {                       \
> +       .notes ALIGN(4) : AT(ADDR(.notes) - LOAD_OFFSET) SUBALIGN(4) {  \
>                  BOUNDED_SECTION_BY(.note.*, _notes)                     \
>          } NOTES_HEADERS                                                 \
>          NOTES_HEADERS_RESTORE
> --
> 2.39.2

[1] https://github.com/acmel/dwarves/commit/a9498899109d3be14f17abbc322a8f55a1067bee
  

On Wed, Apr 12, 2023 at 03:10:14PM +0800, Tianyi Liu wrote:
> On Tue, Apr 11, 2023 at 17:00 , Josh Poimboeuf wrote:
> > On Tue, Feb 14, 2023 at 02:33:02PM +0800, Tianyi Liu wrote:
> > > > LLVM_OBJCOPY=objcopy pahole -J --btf_gen_floats -j
> > > > --skip_encoding_btf_inconsistent_proto --btf_gen_optimized
> > > > .tmp_vmlinux.btf
> > > > btf_encoder__encode: btf__dedup failed!
> > > > Failed to encode BTF
> > > >
> > > > Thanks,
> > > >
> > >
> > > I encountered the same problem when building a new kernel and I found some
> > > reasons for the error.
> > >
> > > In short, enabling CONFIG_X86_KERNEL_IBT will change the order of records in
> > > .notes section. In addition, due to historical problems, the alignment of
> > > records in the .notes section is not unified, which leads to the inability of
> > > gelf_getnote() to read the records after the wrong one.
> > 
> > Alexandre, Tianyi, are you still seeing this issue with the latest
> > dwarves?  If so can you confirm the below patch fixes it?
> > 
> 
> Josh, first of all, thank you very much for your help. However, this patch
> doesn't work in my environment. I am using gcc 12.2.1 20230201.
> After compiling, when I use readelf -S to view ELF sections,
> the align of .notes section is still 8:
> 
> $ readelf -S .tmp_vmlinux.btf
> [20] .notes            NOTE             ffffffff8250b570  0170b570
>      0000000000000084  0000000000000000   A       0     0     8

Hm, weird.

> > Apparently the latest dwarves release fixes it on Fedora Rawhide [1],
> > does anybody know if there a specific dwarves and/or libbpf change for
> > this?
> > 
> 
> It has been fixed in dwarves[1], but it may just be a coincidence.

So just to confirm, the btf__dedup error is gone for you with the latest
dwarves?
  

I have done some tests with Arch Linux which is also affected by this issue:

Base scenario: Running Arch Linux updated as of 2023-04-13.
               Building the linux-mainline 6.3rc6-1 from AUR
               (https://aur.archlinux.org/packages/linux-mainline)
               with CONFIG_X86_KERNEL_IBT changed to 'y'
Result:        Build fails with the "btf__dedup failed!" error

Test 1: Update dwarves (=pahole package on Arch Linux) from the current version
        (1:1.24+r29+g02d67c5-1) to the staging version (1:1.25-1).
Result: The build works correctly.

        However, the notes section is still not parsed correctly, as confirmed
        by `readelf -n` or adding a printf near the code on `cus__merging_cu`
        (on `dwarf_loader.c`) pointed to by Tianyi.

        A bisect shows the commit that fixes the build is
        a9498899109d3be14f17abbc322a8f55a1067bee
        "dwarf_loader: Fix for BTF id drift caused by adding unspecified types"
        I don't know why though.

Test 2: Applying Josh's patch to force-align the ELF notes section to 4 bytes
        (clarification: using the base pahole version again here)
Result: The build works correctly.
        I can read the notes correctly using `readelf -n` as well,
        and they are aligned to 4 bytes instead of 8.

Test 3: Similar to Josh's patch, but instead of force-aligning the sections,
        I added `.note.gnu.property` to the `/DISCARD/` list in the line above.
Result: The build works correctly.
        I can read the notes correctly using `readelf -n` as well
        (of course, the GNU notes which made the alignment be 8 bytes are gone;
         I don't know if this has any negative effect).

So, in summary, either upgrading dwarves/pahole from 1.24 to (not yet fully
released?) 1.25, or applying Josh's patch fixes the issue for me.
Despite the new dwarves/pahole version fixing the build, Josh's patch or
something else to fix the notes alignment is needed.
  

On Wed, Apr 12, 2023 at 16:30PM UTC, Josh Poimboeuf wrote:
> On Wed, Apr 12, 2023 at 03:10:14PM +0800, Tianyi Liu wrote:
> > On Tue, Apr 11, 2023 at 17:00 , Josh Poimboeuf wrote:
> > > On Tue, Feb 14, 2023 at 02:33:02PM +0800, Tianyi Liu wrote:
> > > > > LLVM_OBJCOPY=objcopy pahole -J --btf_gen_floats -j
> > > > > --skip_encoding_btf_inconsistent_proto --btf_gen_optimized
> > > > > .tmp_vmlinux.btf
> > > > > btf_encoder__encode: btf__dedup failed!
> > > > > Failed to encode BTF
> > > > >
> > > > > Thanks,
> > > > >
> > > >
> > > > I encountered the same problem when building a new kernel and I found some
> > > > reasons for the error.
> > > >
> > > > In short, enabling CONFIG_X86_KERNEL_IBT will change the order of records in
> > > > .notes section. In addition, due to historical problems, the alignment of
> > > > records in the .notes section is not unified, which leads to the inability of
> > > > gelf_getnote() to read the records after the wrong one.
> > >
> > > Alexandre, Tianyi, are you still seeing this issue with the latest
> > > dwarves?  If so can you confirm the below patch fixes it?
> > >
> >
> > Josh, first of all, thank you very much for your help. However, this patch
> > doesn't work in my environment. I am using gcc 12.2.1 20230201.
> > After compiling, when I use readelf -S to view ELF sections,
> > the align of .notes section is still 8:
> >
> > $ readelf -S .tmp_vmlinux.btf
> > [20] .notes            NOTE             ffffffff8250b570  0170b570
> >      0000000000000084  0000000000000000   A       0     0     8
> 
> Hm, weird.

I have consulted some materials and found that using ALIGN() in linker
scripts can only "increase" alignment, not decrease it.

Perhaps could you try modifying your patch to use ALIGN(2) and SUBALIGN(2)
and see if the .notes section in the output file is aligned to 2?
In my tests, this had no effect.

[1] https://sourceware.org/binutils/docs/ld/Forced-Output-Alignment.html

> 
> > > Apparently the latest dwarves release fixes it on Fedora Rawhide [1],
> > > does anybody know if there a specific dwarves and/or libbpf change for
> > > this?
> > >
> >
> > It has been fixed in dwarves[1], but it may just be a coincidence.
> 
> So just to confirm, the btf__dedup error is gone for you with the latest
> dwarves?

Yes, this issue was fixed after a9498899109d3be14f17abbc322a8f55a1067bee
("dwarf_loader: Fix for BTF id drift caused by adding unspecified types"),
which will be included in version 1.25.
  

On Thu, Apr 13, 2023 at 02:21:49 +0000, Joan Bruguera Micó wrote:
> I have done some tests with Arch Linux which is also affected by this issue:
> 
> Base scenario: Running Arch Linux updated as of 2023-04-13.
>                Building the linux-mainline 6.3rc6-1 from AUR
>                (https://aur.archlinux.org/packages/linux-mainline)
>                with CONFIG_X86_KERNEL_IBT changed to 'y'
> Result:        Build fails with the "btf__dedup failed!" error

I use MANJARO with kernel version 6.2.10-1. We should be using the same kernel.

> Test 1: Update dwarves (=pahole package on Arch Linux) from the current version
>         (1:1.24+r29+g02d67c5-1) to the staging version (1:1.25-1).
> Result: The build works correctly.
> 
>         However, the notes section is still not parsed correctly, as confirmed
>         by `readelf -n` or adding a printf near the code on `cus__merging_cu`
>         (on `dwarf_loader.c`) pointed to by Tianyi.
> 
>         A bisect shows the commit that fixes the build is
>         a9498899109d3be14f17abbc322a8f55a1067bee
>         "dwarf_loader: Fix for BTF id drift caused by adding unspecified types"
>         I don't know why though.

Pahole reads .notes to look for LINUX_ELFNOTE_BUILD_LTO. When LTO is
enabled, pahole needs to call cus__merge_and_process_cu to merge compile
units, at which point there should only be one unspecified type (used to
represent some compilation information) in the global context.

However, when the kernel is compiled without LTO, if pahole calls
cus__merge_and_process_cu due to alignment issues with notes, multiple
unspecified types may appear after merging the cus, and older versions of
pahole only support up to one. This is why pahole 1.24 crashes, while
newer versions support multiple. However, the latest version of pahole
still does not solve the problem of incorrect LTO recognition, so
compiling the kernel may be slower than normal.

If the only objective is to fix the bug of incorrect LTO recognition,
perhaps this naive patch[1] could be sufficient.

[1]: https://lore.kernel.org/bpf/SY4P282MB1084A0E31D4228DF89FC42639DA29@SY4P282MB1084.AUSP282.PROD.OUTLOOK.COM/

> Test 2: Applying Josh's patch to force-align the ELF notes section to 4 bytes
>         (clarification: using the base pahole version again here)
> Result: The build works correctly.
>         I can read the notes correctly using `readelf -n` as well,
>         and they are aligned to 4 bytes instead of 8.

I'm still currently unable to reproduce this fix, but it may just be an
issue with my environment.

> Test 3: Similar to Josh's patch, but instead of force-aligning the sections,
>         I added `.note.gnu.property` to the `/DISCARD/` list in the line above.
> Result: The build works correctly.
>         I can read the notes correctly using `readelf -n` as well
>         (of course, the GNU notes which made the alignment be 8 bytes are gone;
>          I don't know if this has any negative effect).

I think discarding .note.gnu.property will not have side effects. For some
reason, this note does not exist in my final vmlinux. I also did not find
any usage of this property in the later steps of compiling the kernel.

> So, in summary, either upgrading dwarves/pahole from 1.24 to (not yet fully
> released?) 1.25, or applying Josh's patch fixes the issue for me.
> Despite the new dwarves/pahole version fixing the build, Josh's patch or
> something else to fix the notes alignment is needed.
  

On Thu, Apr 13, 2023 at 12:54:19PM +0800, Tianyi Liu wrote:
> On Wed, Apr 12, 2023 at 16:30PM UTC, Josh Poimboeuf wrote:
> > On Wed, Apr 12, 2023 at 03:10:14PM +0800, Tianyi Liu wrote:
> > > On Tue, Apr 11, 2023 at 17:00 , Josh Poimboeuf wrote:
> > > > On Tue, Feb 14, 2023 at 02:33:02PM +0800, Tianyi Liu wrote:
> > > > > > LLVM_OBJCOPY=objcopy pahole -J --btf_gen_floats -j
> > > > > > --skip_encoding_btf_inconsistent_proto --btf_gen_optimized
> > > > > > .tmp_vmlinux.btf
> > > > > > btf_encoder__encode: btf__dedup failed!
> > > > > > Failed to encode BTF
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > >
> > > > > I encountered the same problem when building a new kernel and I found some
> > > > > reasons for the error.
> > > > >
> > > > > In short, enabling CONFIG_X86_KERNEL_IBT will change the order of records in
> > > > > .notes section. In addition, due to historical problems, the alignment of
> > > > > records in the .notes section is not unified, which leads to the inability of
> > > > > gelf_getnote() to read the records after the wrong one.
> > > >
> > > > Alexandre, Tianyi, are you still seeing this issue with the latest
> > > > dwarves?  If so can you confirm the below patch fixes it?
> > > >
> > >
> > > Josh, first of all, thank you very much for your help. However, this patch
> > > doesn't work in my environment. I am using gcc 12.2.1 20230201.
> > > After compiling, when I use readelf -S to view ELF sections,
> > > the align of .notes section is still 8:
> > >
> > > $ readelf -S .tmp_vmlinux.btf
> > > [20] .notes            NOTE             ffffffff8250b570  0170b570
> > >      0000000000000084  0000000000000000   A       0     0     8
> > 
> > Hm, weird.
> 
> I have consulted some materials and found that using ALIGN() in linker
> scripts can only "increase" alignment, not decrease it.
> 
> Perhaps could you try modifying your patch to use ALIGN(2) and SUBALIGN(2)
> and see if the .notes section in the output file is aligned to 2?
> In my tests, this had no effect.
> 
> [1] https://sourceware.org/binutils/docs/ld/Forced-Output-Alignment.html

Not sure about the ld documentation (it may just be ambiguous wording)
but while doing some tests, I have found that Josh's ALIGN/SUBALIGN(4)
patch only works for certain kernel configs and fails for others.
This likely explains why Josh's patch worked for me but not for Tianyi.

I haven't investigated why or when it works though, but here's a config
where Josh's ALIGN(4) patch works and or where it doesn't (for me):
https://zealcharm.com/20230416-btf-dedup-bug-sample-configs/align-4-working
https://zealcharm.com/20230416-btf-dedup-bug-sample-configs/align-4-not-working

OTOH the new patch to discard .note.gnu.property works in both cases.
  

PS: As additional information for posterity, there are various conditions that
explain why this is/was only a problem on some specific distributions:

* In addition to dwarves/pahole 1.24, binutils 2.40 seems to also be required
  for the problem to occur (in Arch, downgrading to binutils 2.39 fixes it).
* Debian (and thus derived distributions) configures binutils with
  `--disable-x86-used-note`, which by default disables emitting the GNU notes
  which ultimately cause the .notes section to be aligned to 8 bytes.

  On Debian Bookworm & Ubuntu 23.04, the problem reproduces when building like:
  ```
  export KCFLAGS="-Xassembler -mx86-used-note=yes"
  export KAFLAGS="-Xassembler -mx86-used-note=yes"
  make
  ```

Finally, here's a smaller .config to reproduce the problem on affected systems:
https://zealcharm.com/20230416-btf-dedup-bug-sample-configs/minimal-repro