[v2,0/8] reduce maximum memory usage

  The processing of vmlinux.o with objtool is the most memory-intensive step
of a kernel build. By reducing the maximum memory usage here we can reduce
the maximum memory usage of the whole kernel build.
Therefore memory pressure on memory starved machines is relieved during
kernel builds and the build is faster as less swapping has to occur.

To: Josh Poimboeuf <jpoimboe@kernel.org>
To: Peter Zijlstra <peterz@infradead.org>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>

---
Changes in v2:
- Warn on out of range values for reloc->type
- Also reduce size of struct special_alt
- Note: v1 did not make it to the lists, only to individual recipients

---
Thomas Weißschuh (8):
      objtool: make struct entries[] static and const
      objtool: make struct check_options static
      objtool: allocate multiple structures with calloc()
      objtool: introduce function elf_reloc_set_type
      objtool: reduce memory usage of struct reloc
      objtool: optimize layout of struct symbol
      objtool: optimize layout of struct special_alt
      objtool: explicitly cleanup resources on success

 tools/objtool/builtin-check.c           |  4 ++-
 tools/objtool/check.c                   |  6 ++--
 tools/objtool/elf.c                     | 56 +++++++++++++++++++--------------
 tools/objtool/include/objtool/builtin.h |  2 --
 tools/objtool/include/objtool/elf.h     | 13 +++++---
 tools/objtool/include/objtool/special.h |  2 +-
 tools/objtool/special.c                 |  6 ++--
 7 files changed, 51 insertions(+), 38 deletions(-)
---
base-commit: 1b929c02afd37871d5afb9d498426f83432e71c2
change-id: 20221216-objtool-memory-06db3b8bf111

Best regards,
  

On Tue, Dec 27, 2022 at 04:00:57PM +0000, Thomas Weißschuh wrote:
> The processing of vmlinux.o with objtool is the most memory-intensive step
> of a kernel build. By reducing the maximum memory usage here we can reduce
> the maximum memory usage of the whole kernel build.
> Therefore memory pressure on memory starved machines is relieved during
> kernel builds and the build is faster as less swapping has to occur.

Friendly ping.

These patches can also applied one by one, the only dependency is from
patch 5 to patch 4.

Thanks,
Thomas

> To: Josh Poimboeuf <jpoimboe@kernel.org>
> To: Peter Zijlstra <peterz@infradead.org>
> Cc: linux-kernel@vger.kernel.org
> Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
> 
> ---
> Changes in v2:
> - Warn on out of range values for reloc->type
> - Also reduce size of struct special_alt
> - Note: v1 did not make it to the lists, only to individual recipients
> 
> ---
> Thomas Weißschuh (8):
>       objtool: make struct entries[] static and const
>       objtool: make struct check_options static
>       objtool: allocate multiple structures with calloc()
>       objtool: introduce function elf_reloc_set_type
>       objtool: reduce memory usage of struct reloc
>       objtool: optimize layout of struct symbol
>       objtool: optimize layout of struct special_alt
>       objtool: explicitly cleanup resources on success
> 
>  tools/objtool/builtin-check.c           |  4 ++-
>  tools/objtool/check.c                   |  6 ++--
>  tools/objtool/elf.c                     | 56 +++++++++++++++++++--------------
>  tools/objtool/include/objtool/builtin.h |  2 --
>  tools/objtool/include/objtool/elf.h     | 13 +++++---
>  tools/objtool/include/objtool/special.h |  2 +-
>  tools/objtool/special.c                 |  6 ++--
>  7 files changed, 51 insertions(+), 38 deletions(-)
> ---
> base-commit: 1b929c02afd37871d5afb9d498426f83432e71c2
> change-id: 20221216-objtool-memory-06db3b8bf111
> 
> Best regards,
> -- 
> Thomas Weißschuh <linux@weissschuh.net>
  

On Sun, Jan 29, 2023 at 09:43:39PM +0000, Thomas Weißschuh wrote:
> On Tue, Dec 27, 2022 at 04:00:57PM +0000, Thomas Weißschuh wrote:
> > The processing of vmlinux.o with objtool is the most memory-intensive step
> > of a kernel build. By reducing the maximum memory usage here we can reduce
> > the maximum memory usage of the whole kernel build.
> > Therefore memory pressure on memory starved machines is relieved during
> > kernel builds and the build is faster as less swapping has to occur.
> 
> Friendly ping.
> 
> These patches can also applied one by one, the only dependency is from
> patch 5 to patch 4.

Thanks, I'll go ahead and take five of them now.
  

On Mon, Jan 30, 2023 at 04:03:56PM -0800, Josh Poimboeuf wrote:
> On Sun, Jan 29, 2023 at 09:43:39PM +0000, Thomas Weißschuh wrote:
> > On Tue, Dec 27, 2022 at 04:00:57PM +0000, Thomas Weißschuh wrote:
> > > The processing of vmlinux.o with objtool is the most memory-intensive step
> > > of a kernel build. By reducing the maximum memory usage here we can reduce
> > > the maximum memory usage of the whole kernel build.
> > > Therefore memory pressure on memory starved machines is relieved during
> > > kernel builds and the build is faster as less swapping has to occur.
> > 
> > Friendly ping.
> > 
> > These patches can also applied one by one, the only dependency is from
> > patch 5 to patch 4.
> 
> Thanks, I'll go ahead and take five of them now.

Thanks.

I have another half-finished series that replaces the doubly-linked
list_heads used by elf.h with a custom singly-linked list.
This would save a few pointers per struct.

Do you think this is worth it?

Thomas
  

On Tue, Jan 31, 2023 at 03:54:42AM +0000, Thomas Weißschuh wrote:
> On Mon, Jan 30, 2023 at 04:03:56PM -0800, Josh Poimboeuf wrote:
> > On Sun, Jan 29, 2023 at 09:43:39PM +0000, Thomas Weißschuh wrote:
> > > On Tue, Dec 27, 2022 at 04:00:57PM +0000, Thomas Weißschuh wrote:
> > > > The processing of vmlinux.o with objtool is the most memory-intensive step
> > > > of a kernel build. By reducing the maximum memory usage here we can reduce
> > > > the maximum memory usage of the whole kernel build.
> > > > Therefore memory pressure on memory starved machines is relieved during
> > > > kernel builds and the build is faster as less swapping has to occur.
> > > 
> > > Friendly ping.
> > > 
> > > These patches can also applied one by one, the only dependency is from
> > > patch 5 to patch 4.
> > 
> > Thanks, I'll go ahead and take five of them now.
> 
> Thanks.
> 
> I have another half-finished series that replaces the doubly-linked
> list_heads used by elf.h with a custom singly-linked list.
> This would save a few pointers per struct.
> 
> Do you think this is worth it?

Maybe, depending on the memory savings.
  

From: Thomas Weißschuh.
> Sent: 31 January 2023 03:55
...
> I have another half-finished series that replaces the doubly-linked
> list_heads used by elf.h with a custom singly-linked list.
> This would save a few pointers per struct.
> 
> Do you think this is worth it?

If you allocate the structures in blocks of (say) 256 you
can use an array of pointers to the blocks and then a
32bit index instead of a 64bit pointer.

For real space-saving you might decide that the index can
never exceed 2^^24 and use a bitfield!

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)
  

On Tue, Jan 31, 2023 at 09:27:15AM -0800, Josh Poimboeuf wrote:
> On Tue, Jan 31, 2023 at 03:54:42AM +0000, Thomas Weißschuh wrote:
> > On Mon, Jan 30, 2023 at 04:03:56PM -0800, Josh Poimboeuf wrote:
> > > On Sun, Jan 29, 2023 at 09:43:39PM +0000, Thomas Weißschuh wrote:
> > > > On Tue, Dec 27, 2022 at 04:00:57PM +0000, Thomas Weißschuh wrote:
> > > > > The processing of vmlinux.o with objtool is the most memory-intensive step
> > > > > of a kernel build. By reducing the maximum memory usage here we can reduce
> > > > > the maximum memory usage of the whole kernel build.
> > > > > Therefore memory pressure on memory starved machines is relieved during
> > > > > kernel builds and the build is faster as less swapping has to occur.
> > > > 
> > > > Friendly ping.
> > > > 
> > > > These patches can also applied one by one, the only dependency is from
> > > > patch 5 to patch 4.
> > > 
> > > Thanks, I'll go ahead and take five of them now.
> > 
> > Thanks.
> > 
> > I have another half-finished series that replaces the doubly-linked
> > list_heads used by elf.h with a custom singly-linked list.
> > This would save a few pointers per struct.
> > 
> > Do you think this is worth it?
> 
> Maybe, depending on the memory savings.

FYI, there were more memory usage complaints, so Peter worked up a nice
series to do this and more:

  https://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git/log/?h=objtool/core