[v1,00/22] selftests/nolibc: add minimal kernel config support

  Willy, Thomas

We just sent the 'selftests/nolibc: allow run with minimal kernel
config' series [1], Here is the 'minimal' kernel config support, with
both of them, it is possible to run nolibc-test for all architectures
with oneline command and in less than ~30 minutes - 1 hour (not fullly
measured yet):

    // run with tiny config + qemu-system
    // Note: rv32 and loongarch require to download the bios at first
    $ time make run-tiny-all QUIET_RUN=1

    // run with default config + qemu-system
    $ time make run-default-all QUIET_RUN=1

    // run with qemu-user
    $ time make run-user-all QUIET_RUN=1

Besides the 'tinyconfig' suggestion from Thomas, this patch also merge
the generic part of my local powerpc porting (the extconfig to add
additional console support).

This is applied after the test report patchset [2] and the rv32 compile
patchset [3], because all of them touched the same Makefile.

Even without the 'selftests/nolibc: allow run with minimal kernel
config' series [1], all of the tests can pass except the /proc/self/net
related ones (We haven't enable CONFIG_NET in this patchset), the
chmod_net one will be removed by Thomas from this patchset [4] for the
wrong chmodable attribute issue of /proc/self/net, the link_cross one
can be simply fixed up by using another /proc/self interface (like
/proc/self/cmdline), which will be covered in our revision of the [1]
series.

Beside the core 'minimal' config support, some generic patch are added
together to avoid patch conflicts.

* selftests/nolibc: add test for -include /path/to/nolibc.h

  Add a test switch to allow run nolibc-test with nolibc.h

* selftests/nolibc: print result to the screen too

  Let the run targets print results by default, allow disable by
  QUIET_RUN=1

* selftests/nolibc: allow use x86_64 toolchain for i386

  Allow use x86_64 toolchains for i386

* selftests/nolibc: add menuconfig target for manual config

  a new 'menuconfig' target added for development and debugging

* selftests/nolibc: add tinyconfig target

  a new 'tinyconfig' compare to 'defconfig', smaller and faster, but not
  enough for boot and print, require following 'extconfig' target

* selftests/nolibc: allow customize extra kernel config options

  a new 'extconfig' allows to add extra config options for 'defconfig'
  and 'tinyconfig'

* selftests/nolibc: add common extra config options
  selftests/nolibc: add power reset control support
  selftests/nolibc: add procfs, shmem and tmpfs

  Add common extra configs, the 3rd one (procfs, shmem and tmpfs) can be
  completely reverted after [1] series, but as discuss with Thomas,
  procfs may be still a hard requirement.

* selftests/nolibc: add extra configs for i386
  selftests/nolibc: add extra configs for x86_64
  selftests/nolibc: add extra configs for arm64
  selftests/nolibc: add extra configs for arm
  selftests/nolibc: add extra configs for mips
  selftests/nolibc: add extra configs for riscv32
  selftests/nolibc: add extra configs for riscv64
  selftests/nolibc: add extra configs for s390x
  selftests/nolibc: add extra configs for loongarch

  Add architecture specific extra configs to let kernel boot and
  nolibc-test print. The rv32 added here is only for test, it should not
  be merged before the missing 64bit syscalls are added (still wait for
  the merging of the __sysret and -ENOSYS patches).

* selftests/nolibc: config default CROSS_COMPILE
  selftests/nolibc: add run-tiny and run-default

  both run-tiny and run-default are added to do config and run together,
  this easier test a log.

* selftests/nolibc: allow run tests on all targets
  selftests/nolibc: detect bios existing to avoid hang

  Further allow do run-user, run-tiny and run-default for all
  architectures at once, the -all suffix is added to do so.

Since some generic patches are still in review, before sending the left
rv32 patches, I'm will send more generic patches later, the coming one
is arch-xxx.h cleanup, and then, the 32bit powerpc porting support.

For the compile speedup, the next step may be add architecture specific
'O' support, which may allow us rerun across architectures without
mrproper, for a single architecture development, this 'minimal' config
should be enough ;-)

Thanks.

Best regards,
Zhangjin
---
[1]: https://lore.kernel.org/lkml/cover.1687344643.git.falcon@tinylab.org/
[2]: https://lore.kernel.org/lkml/cover.1687156559.git.falcon@tinylab.org/
[3]: https://lore.kernel.org/linux-riscv/cover.1687176996.git.falcon@tinylab.org/
[4]: https://lore.kernel.org/lkml/20230624-proc-net-setattr-v1-0-73176812adee@weissschuh.net/

Zhangjin Wu (22):
  selftests/nolibc: add test for -include /path/to/nolibc.h
  selftests/nolibc: print result to the screen too
  selftests/nolibc: allow use x86_64 toolchain for i386
  selftests/nolibc: add menuconfig target for manual config
  selftests/nolibc: add tinyconfig target
  selftests/nolibc: allow customize extra kernel config options
  selftests/nolibc: add common extra config options
  selftests/nolibc: add power reset control support
  selftests/nolibc: add procfs, shmem and tmpfs
  selftests/nolibc: add extra configs for i386
  selftests/nolibc: add extra configs for x86_64
  selftests/nolibc: add extra configs for arm64
  selftests/nolibc: add extra configs for arm
  selftests/nolibc: add extra configs for mips
  selftests/nolibc: add extra configs for riscv32
  selftests/nolibc: add extra configs for riscv64
  selftests/nolibc: add extra configs for s390x
  selftests/nolibc: add extra configs for loongarch
  selftests/nolibc: config default CROSS_COMPILE
  selftests/nolibc: add run-tiny and run-default
  selftests/nolibc: allow run tests on all targets
  selftests/nolibc: detect bios existing to avoid hang

 tools/testing/selftests/nolibc/Makefile | 125 ++++++++++++++++++++++--
 1 file changed, 119 insertions(+), 6 deletions(-)
  

Hi, Willy and Thomas

I'm planning to renew this patchset, so, require more discuss.

The main goal of this minimal kernel config support here is faster build
and therefore faster test for the run target, but this old patchset is
not enough, especially for the 'mrproper' operation during every config:

    defconfig:
     	$(Q)$(MAKE) -C $(srctree) ARCH=$(KARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE) mrproper $(DEFCONFIG) prepare
     
    +tinyconfig:
    +	$(Q)$(MAKE) -C $(srctree) ARCH=$(KARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE) mrproper tinyconfig prepare
    +

In order to avoid this 'mrproper', one solution is create an arch
specicifc output directory for all of the kernel targets to avoid they
share the same build output (by default, it is the same as the source
code directory and therefore pollute the source code tree), it is
required for both tools/nolibc and selftests/nolibc:

    // tools/nolibc

    headers_standalone: headers
    	$(Q)$(MAKE) -C $(srctree) headers
    	$(Q)$(MAKE) -C $(srctree) headers_install INSTALL_HDR_PATH=$(OUTPUT)sysroot
    

    // selftests/nolibc

    defconfig:
    	$(Q)$(MAKE) -C $(srctree) ARCH=$(ARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE) mrproper $(DEFCONFIG) prepare
    
    kernel: initramfs
    	$(Q)$(MAKE) -C $(srctree) ARCH=$(ARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE) $(IMAGE_NAME) CONFIG_INITRAMFS_SOURCE=$(CURDIR)/initramfs
    
    # run the tests after building the kernel
    run: kernel
    	$(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out"
    	$(Q)$(REPORT) $(CURDIR)/run.out


The "O" variable could be used to pass an arch specific output for every
arch:

    # OUTPUT is only set when run from the main makefile, otherwise
    # it defaults to this nolibc directory.
    OUTPUT ?= $(CURDIR)/

    # Architecture specific directory
    ARCH_OUTPUT ?= $(CURDIR)/build/$(ARCH)/


With "O":

    // tools/nolibc

    KERNEL_BUILD   ?= $(ARCH_OUTPUT)linux/
    MAKE_KERNEL     = $(MAKE) -C "$(srctree)" O="$(KERNEL_BUILD)" ARCH=$(ARCH)

    headers_standalone: headers
    	$(Q)$(MAKE_KERNEL) headers
    	$(Q)$(MAKE_KERNEL) headers_install INSTALL_HDR_PATH=$(ARCH_OUTPUT)sysroot
    

    // selftests/nolibc
    RUN_OUT         = $(ARCH_OUTPUT)run.out
    ...
    NOLIBC_INITRAMFS  = $(ARCH_OUTPUT)initramfs/
    ...
    KERNEL_BUILD   ?= $(ARCH_OUTPUT)linux/
    MAKE_KERNEL     = $(MAKE) -C "$(srctree)" O="$(KERNEL_BUILD)" ARCH=$(ARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE)
    KERNEL_IMAGE    = $(KERNEL_BUILD)$(IMAGE)
    KERNEL_CONFIG   = $(KERNEL_BUILD).config

    defconfig:
    	$(Q)$(MAKE_KERNEL) $(DEFCONFIG) prepare
    
    kernel: initramfs
    	$(Q)$(MAKE_KERNEL) $(IMAGE_NAME) CONFIG_INITRAMFS_SOURCE=$(NOLIBC_INITRAMFS)
    
    # run the tests after building the kernel
    run: kernel
    	$(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(KERNEL_IMAGE)" -serial stdio $(QEMU_ARGS) > "$(RUN_OUT)"
    	$(Q)$(REPORT) $(RUN_OUT)

And further, the output of the other targets should be put in the
$(ARCH_OUTPUT) directory too, include the sysroot, nolibc-test, initramfs,
run.out and even libc-test, the whole Makefile of selftests/nolibc will be
touched.

I have prepared and verified such a solution locally, before send the patches,
welcome your suggestions.

My local patchset now provides such features:

  - Allow build everything for a target arch in the arch specific directory
  
  - Allow clean everything of a target arch
  
  - Allow trigger a config target automatically for the run target, use the new
    tinyconfig by default, be able to choose defconfig as before
  
  - Allow configure additional config options for both defconfig and tinyconfig
  
  - Allow configure the default CROSS_COMPILE by default, only the ARCH variable is required
  
  - Allow download the missing cross toolchains automatically (from mirrrors.edge.kernel.org)
  
  - Allow download the missing bios automatically
  
  - Allow download and build musl automatically
  
  - Allow run tests for all of the supported architectures automatically with one command
    with minimal preparation, may be only the installation of qemu-user and qemu-system

With the new patchset, it is able to rebuild and rerun everything in several
minutes, it may make people very happy to develop nolibc itself and also make
people to work on linux with nolibc, especially for developing and testing a
new kernel feature ;-)

Without this reorg of the output directory, it is very hard to verify this
tinyconfig support patchset itself, it always require 'mrproper' to clean the
whole build while config and test another ARCH (because the source code tree is
polluted). for one ARCH, it is not that time-costed, but for all, it waste too
much time. That means, without the arch specific output support, our minimal
config support is really not that helpful, it also breaks the possibility to
share the already built binaries for rebuild after a reconfig.

If it is hard to make decision on this description, I'm ok to send the whole
new patchset for more discussion.

The change is 'huge' and the review work may be 'huge' too ;-)

Thanks,
Zhangjin

> Willy, Thomas
> 
> We just sent the 'selftests/nolibc: allow run with minimal kernel
> config' series [1], Here is the 'minimal' kernel config support, with
> both of them, it is possible to run nolibc-test for all architectures
> with oneline command and in less than ~30 minutes - 1 hour (not fullly
> measured yet):
> 
>     // run with tiny config + qemu-system
>     // Note: rv32 and loongarch require to download the bios at first
>     $ time make run-tiny-all QUIET_RUN=1
> 
>     // run with default config + qemu-system
>     $ time make run-default-all QUIET_RUN=1
> 
>     // run with qemu-user
>     $ time make run-user-all QUIET_RUN=1
> 
> Besides the 'tinyconfig' suggestion from Thomas, this patch also merge
> the generic part of my local powerpc porting (the extconfig to add
> additional console support).
> 
> This is applied after the test report patchset [2] and the rv32 compile
> patchset [3], because all of them touched the same Makefile.
> 
> Even without the 'selftests/nolibc: allow run with minimal kernel
> config' series [1], all of the tests can pass except the /proc/self/net
> related ones (We haven't enable CONFIG_NET in this patchset), the
> chmod_net one will be removed by Thomas from this patchset [4] for the
> wrong chmodable attribute issue of /proc/self/net, the link_cross one
> can be simply fixed up by using another /proc/self interface (like
> /proc/self/cmdline), which will be covered in our revision of the [1]
> series.
> 
> Beside the core 'minimal' config support, some generic patch are added
> together to avoid patch conflicts.
> 
> * selftests/nolibc: add test for -include /path/to/nolibc.h
> 
>   Add a test switch to allow run nolibc-test with nolibc.h
> 
> * selftests/nolibc: print result to the screen too
> 
>   Let the run targets print results by default, allow disable by
>   QUIET_RUN=1
> 
> * selftests/nolibc: allow use x86_64 toolchain for i386
> 
>   Allow use x86_64 toolchains for i386
> 
> * selftests/nolibc: add menuconfig target for manual config
> 
>   a new 'menuconfig' target added for development and debugging
> 
> * selftests/nolibc: add tinyconfig target
> 
>   a new 'tinyconfig' compare to 'defconfig', smaller and faster, but not
>   enough for boot and print, require following 'extconfig' target
> 
> * selftests/nolibc: allow customize extra kernel config options
> 
>   a new 'extconfig' allows to add extra config options for 'defconfig'
>   and 'tinyconfig'
> 
> * selftests/nolibc: add common extra config options
>   selftests/nolibc: add power reset control support
>   selftests/nolibc: add procfs, shmem and tmpfs
> 
>   Add common extra configs, the 3rd one (procfs, shmem and tmpfs) can be
>   completely reverted after [1] series, but as discuss with Thomas,
>   procfs may be still a hard requirement.
> 
> * selftests/nolibc: add extra configs for i386
>   selftests/nolibc: add extra configs for x86_64
>   selftests/nolibc: add extra configs for arm64
>   selftests/nolibc: add extra configs for arm
>   selftests/nolibc: add extra configs for mips
>   selftests/nolibc: add extra configs for riscv32
>   selftests/nolibc: add extra configs for riscv64
>   selftests/nolibc: add extra configs for s390x
>   selftests/nolibc: add extra configs for loongarch
> 
>   Add architecture specific extra configs to let kernel boot and
>   nolibc-test print. The rv32 added here is only for test, it should not
>   be merged before the missing 64bit syscalls are added (still wait for
>   the merging of the __sysret and -ENOSYS patches).
> 
> * selftests/nolibc: config default CROSS_COMPILE
>   selftests/nolibc: add run-tiny and run-default
> 
>   both run-tiny and run-default are added to do config and run together,
>   this easier test a log.
> 
> * selftests/nolibc: allow run tests on all targets
>   selftests/nolibc: detect bios existing to avoid hang
> 
>   Further allow do run-user, run-tiny and run-default for all
>   architectures at once, the -all suffix is added to do so.
> 
> Since some generic patches are still in review, before sending the left
> rv32 patches, I'm will send more generic patches later, the coming one
> is arch-xxx.h cleanup, and then, the 32bit powerpc porting support.
> 
> For the compile speedup, the next step may be add architecture specific
> 'O' support, which may allow us rerun across architectures without
> mrproper, for a single architecture development, this 'minimal' config
> should be enough ;-)
> 
> Thanks.
> 
> Best regards,
> Zhangjin
> ---
> [1]: https://lore.kernel.org/lkml/cover.1687344643.git.falcon@tinylab.org/
> [2]: https://lore.kernel.org/lkml/cover.1687156559.git.falcon@tinylab.org/
> [3]: https://lore.kernel.org/linux-riscv/cover.1687176996.git.falcon@tinylab.org/
> [4]: https://lore.kernel.org/lkml/20230624-proc-net-setattr-v1-0-73176812adee@weissschuh.net/
> 
> Zhangjin Wu (22):
>   selftests/nolibc: add test for -include /path/to/nolibc.h
>   selftests/nolibc: print result to the screen too
>   selftests/nolibc: allow use x86_64 toolchain for i386
>   selftests/nolibc: add menuconfig target for manual config
>   selftests/nolibc: add tinyconfig target
>   selftests/nolibc: allow customize extra kernel config options
>   selftests/nolibc: add common extra config options
>   selftests/nolibc: add power reset control support
>   selftests/nolibc: add procfs, shmem and tmpfs
>   selftests/nolibc: add extra configs for i386
>   selftests/nolibc: add extra configs for x86_64
>   selftests/nolibc: add extra configs for arm64
>   selftests/nolibc: add extra configs for arm
>   selftests/nolibc: add extra configs for mips
>   selftests/nolibc: add extra configs for riscv32
>   selftests/nolibc: add extra configs for riscv64
>   selftests/nolibc: add extra configs for s390x
>   selftests/nolibc: add extra configs for loongarch
>   selftests/nolibc: config default CROSS_COMPILE
>   selftests/nolibc: add run-tiny and run-default
>   selftests/nolibc: allow run tests on all targets
>   selftests/nolibc: detect bios existing to avoid hang
> 
>  tools/testing/selftests/nolibc/Makefile | 125 ++++++++++++++++++++++--
>  1 file changed, 119 insertions(+), 6 deletions(-)
> 
> -- 
> 2.25.1
  

Hi Zhangjin,

On Tue, Jul 11, 2023 at 11:55:08AM +0800, Zhangjin Wu wrote:
> Hi, Willy and Thomas
> 
> I'm planning to renew this patchset, so, require more discuss.
> 
> The main goal of this minimal kernel config support here is faster build
> and therefore faster test for the run target, but this old patchset is
> not enough, especially for the 'mrproper' operation during every config:
> 
>     defconfig:
>      	$(Q)$(MAKE) -C $(srctree) ARCH=$(KARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE) mrproper $(DEFCONFIG) prepare
>      
>     +tinyconfig:
>     +	$(Q)$(MAKE) -C $(srctree) ARCH=$(KARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE) mrproper tinyconfig prepare
>     +

It will depend on the number of runs, because the mrproper will
only be run before recreating a config. Logically once you have
your kernel you're not going to reconfigure it.

> In order to avoid this 'mrproper', one solution is create an arch
> specicifc output directory for all of the kernel targets to avoid they
> share the same build output (by default, it is the same as the source
> code directory and therefore pollute the source code tree), it is
> required for both tools/nolibc and selftests/nolibc:

This could provide an improvement when building for multiple archs
in a row, indeed. But keep in mind that an important aspect is also
to be able to test the kernel you're currently working on, and not
to have to rebuild it again for nolibc into a subdir.

And given that the makefile doesn't know how to iterate over multiple
archs, it would make sense to permit to pass the build output as an
argument to the makefile so that the external script calling make for
each arch would just pass the build directory (as is sometimes done
for the normal kernel).

> The "O" variable could be used to pass an arch specific output for every
> arch:
> 
>     # OUTPUT is only set when run from the main makefile, otherwise
>     # it defaults to this nolibc directory.
>     OUTPUT ?= $(CURDIR)/
> 
>     # Architecture specific directory
>     ARCH_OUTPUT ?= $(CURDIR)/build/$(ARCH)/
> 
> 
> With "O":
> 
>     // tools/nolibc
> 
>     KERNEL_BUILD   ?= $(ARCH_OUTPUT)linux/
>     MAKE_KERNEL     = $(MAKE) -C "$(srctree)" O="$(KERNEL_BUILD)" ARCH=$(ARCH)
> 
>     headers_standalone: headers
>     	$(Q)$(MAKE_KERNEL) headers
>     	$(Q)$(MAKE_KERNEL) headers_install INSTALL_HDR_PATH=$(ARCH_OUTPUT)sysroot
>     
> 
>     // selftests/nolibc
>     RUN_OUT         = $(ARCH_OUTPUT)run.out
>     ...
>     NOLIBC_INITRAMFS  = $(ARCH_OUTPUT)initramfs/
>     ...
>     KERNEL_BUILD   ?= $(ARCH_OUTPUT)linux/
>     MAKE_KERNEL     = $(MAKE) -C "$(srctree)" O="$(KERNEL_BUILD)" ARCH=$(ARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE)

Yes but this one will break reuse of the already built kernel. Also
for those trying all archs for a single test it will eat a lot of
disk space (when enabling certain debugging symbols you can end up
with extremely large build dirs, I already ran out of space a few
times).

I think that instead we should just let the user pass this "O=" to the
make command and let the lower ones inherit it. That makes it closer
to what it usually done. I'm also fine with using another variable if
there's a good reason for this, but given that it's already called "O"
it's not something unexpected to the user. Thus we could imagine just
running:

    for arch in "${ARCHS[@]}"; do
        make -j$(nproc) O=$PWD/kernel-$arch ARCH=$arch \
             CROSS_COMPILE=/path/to/cc/$arch/bin/$arch- \
             run
    done
             
Which is simple enough without requiring a user to have to figure what
build option to hack on for just a regular build.

> And further, the output of the other targets should be put in the
> $(ARCH_OUTPUT) directory too, include the sysroot, nolibc-test, initramfs,
> run.out and even libc-test, the whole Makefile of selftests/nolibc will be
> touched.

It's a matter of taste but there are pros and cons. I personally spend
my time using grep -r on the sysroots, having gigabytes of .o there would
quickly made me regret that approach. I would also argue that when you
pass "O=" to the kernel build, it serves as a prefix and the directory
hierarchy is preserved, so if you would want to put the sysroots etc
under $O then it would be more logical that it appears in
$O/tools/testing/selftests/nolibc/sysroot/$arch, which would then remain
compatible with today when O corresponds to the kernel's CURDIR. Then
maybe that would be the solution to making O per-arch for those who want
to re-run over many kernels, while keeping it easy for those who want to
test their own kernel or want to grep in sysroot.

> I have prepared and verified such a solution locally, before send the patches,
> welcome your suggestions.
> 
> My local patchset now provides such features:
> 
>   - Allow build everything for a target arch in the arch specific directory
>   
>   - Allow clean everything of a target arch
>   
>   - Allow trigger a config target automatically for the run target, use the new
>     tinyconfig by default, be able to choose defconfig as before
>   
>   - Allow configure additional config options for both defconfig and tinyconfig
>   
>   - Allow configure the default CROSS_COMPILE by default, only the ARCH variable is required
>   
>   - Allow download the missing cross toolchains automatically (from mirrrors.edge.kernel.org)
>   
>   - Allow download the missing bios automatically
>   
>   - Allow download and build musl automatically
>   
>   - Allow run tests for all of the supported architectures automatically with one command
>     with minimal preparation, may be only the installation of qemu-user and qemu-system

There can be good things there, but I'm seeing a lot of "by default" and
"automatically" which worry me, because when you're debugging something,
there's nothing more annoying than all the stuff happening in your back
and fooling your analysis. On the opposite I do value the ability to
proceed through steps that I don't need to revalidate, and these ones
can be done automatically just by chaining a few commands from a script.

And no download should ever be done automatically. Any download must
always be deliberately and consciously initiated by the user. How many
times we've seen some scripts stall with no information, just to discover
a SYN_SENT socket in netstat! You can have a "download-toolchain",
"download-bios", "download-musl" etc if you want, but quite frankly, I'm
seriously doubting it will be *that* useful to anyone working on the
kernel since by definition they already have these toolchains, and
sometimes different ones (e.g. linaro, bootlin etc) and in different
versions. In fact if some expresses the need for having a simplified
command to download a toolchain, then it wouldn't be specific to
nolibc-test but would be used as a general tool for the kernel since
it would be shared with all the rest of the code. You could also imagine
downloading and building qemu, but it also takes a while and we're very
far from the simple build script whose purpose was to validate that a
few syscalls appeared to work fine.

> With the new patchset, it is able to rebuild and rerun everything in several
> minutes, it may make people very happy to develop nolibc itself and also make
> people to work on linux with nolibc, especially for developing and testing a
> new kernel feature ;-)

I doubt it. Rebuilding everything is not what most developers need. What
they need is to just fix a missing parenthesis that broke the build and
rebuild the strict minimum, restarting from where they left. And they
need to be able to figure what caused that "FAILED" output, disassemble
the output, re-run it manually under qemu-user, rebuild the .c manually
after copy-pasting the whole command line via "V=1", etc.

Keep in mind that the purpose of a selftest is not to optimize the case
where it works fine, but to optimize the developer's work when it fails.
This is something that a lot of people tend to get wrong. They're proud
of a "make world" that does everything including delivering them pizzas
to wait for the builds to complete, and they say "hey, impressed?". But
when someone else reports "I'm getting this strange error I don't
understand", they can hardly suggest more than "hmmm run make clean, try
again, or use the same distro as me because it works for me".

What I've tried to focus on from the beginning with this tiny test suite
precisely is debugging. That's why I wanted to support alternate libcs,
to help compare, making it easy to use a different toolchain and other
cflags, running in userland, building just the binary so that I can scp
it to a remote native machine etc. All of these are important, and your
approach above makes me feel like I would lose most of it, or still be
able to do that but in a very complicated way (i.e. constantly have to
reverse-engineer a complicated makefile to figure where I can inject an
option).

However one thing I mentioned in the past is the usefulness of some
batching scripts (I even shared one). And I think that helping the user
prepare certain steps or iterate over architectures *is* useful. When
you do it in two layers (the script replacing the user, the makefile
doing the build job), it remains easy and convenient to use, and you
can pick only what you need (e.g. "please build musl for me"). And if
something goes wrong, it's simple for the user to takeover and complete
that failed task by changing an arch name, a directory or anything, and
have their tools ready. Let's just never make that automatic for the
sake of everyone's health!

Thanks,
Willy
  

Hi, Willy

> 
> Hi Zhangjin,
> 
> On Tue, Jul 11, 2023 at 11:55:08AM +0800, Zhangjin Wu wrote:
> > Hi, Willy and Thomas
> > 
> > I'm planning to renew this patchset, so, require more discuss.
> > 
> > The main goal of this minimal kernel config support here is faster build
> > and therefore faster test for the run target, but this old patchset is
> > not enough, especially for the 'mrproper' operation during every config:
> > 
> >     defconfig:
> >      	$(Q)$(MAKE) -C $(srctree) ARCH=$(KARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE) mrproper $(DEFCONFIG) prepare
> >      
> >     +tinyconfig:
> >     +	$(Q)$(MAKE) -C $(srctree) ARCH=$(KARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE) mrproper tinyconfig prepare
> >     +
> 
> It will depend on the number of runs, because the mrproper will
> only be run before recreating a config. Logically once you have
> your kernel you're not going to reconfigure it.
>

Yes, for one arch it is once.

> > In order to avoid this 'mrproper', one solution is create an arch
> > specicifc output directory for all of the kernel targets to avoid they
> > share the same build output (by default, it is the same as the source
> > code directory and therefore pollute the source code tree), it is
> > required for both tools/nolibc and selftests/nolibc:
> 
> This could provide an improvement when building for multiple archs
> in a row, indeed. But keep in mind that an important aspect is also
> to be able to test the kernel you're currently working on, and not
> to have to rebuild it again for nolibc into a subdir.
> 
> And given that the makefile doesn't know how to iterate over multiple
> archs, it would make sense to permit to pass the build output as an
> argument to the makefile so that the external script calling make for
> each arch would just pass the build directory (as is sometimes done
> for the normal kernel).
> 
> > The "O" variable could be used to pass an arch specific output for every
> > arch:
> > 
> >     # OUTPUT is only set when run from the main makefile, otherwise
> >     # it defaults to this nolibc directory.
> >     OUTPUT ?= $(CURDIR)/
> > 
> >     # Architecture specific directory
> >     ARCH_OUTPUT ?= $(CURDIR)/build/$(ARCH)/
> > 
> > 
> > With "O":
> > 
> >     // tools/nolibc
> > 
> >     KERNEL_BUILD   ?= $(ARCH_OUTPUT)linux/
> >     MAKE_KERNEL     = $(MAKE) -C "$(srctree)" O="$(KERNEL_BUILD)" ARCH=$(ARCH)
> > 
> >     headers_standalone: headers
> >     	$(Q)$(MAKE_KERNEL) headers
> >     	$(Q)$(MAKE_KERNEL) headers_install INSTALL_HDR_PATH=$(ARCH_OUTPUT)sysroot
> >     
> > 
> >     // selftests/nolibc
> >     RUN_OUT         = $(ARCH_OUTPUT)run.out
> >     ...
> >     NOLIBC_INITRAMFS  = $(ARCH_OUTPUT)initramfs/
> >     ...
> >     KERNEL_BUILD   ?= $(ARCH_OUTPUT)linux/
> >     MAKE_KERNEL     = $(MAKE) -C "$(srctree)" O="$(KERNEL_BUILD)" ARCH=$(ARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE)
> 
> Yes but this one will break reuse of the already built kernel. Also
> for those trying all archs for a single test it will eat a lot of
> disk space (when enabling certain debugging symbols you can end up
> with extremely large build dirs, I already ran out of space a few
> times).

For defconfig, it is, but for tinyconfig, it is smaller ;-)

Of course, with debugging symbols, it is bigger.

> 
> I think that instead we should just let the user pass this "O=" to the
> make command and let the lower ones inherit it. That makes it closer
> to what it usually done. I'm also fine with using another variable if
> there's a good reason for this, but given that it's already called "O"
> it's not something unexpected to the user. Thus we could imagine just
> running:
> 
>     for arch in "${ARCHS[@]}"; do
>         make -j$(nproc) O=$PWD/kernel-$arch ARCH=$arch \
>              CROSS_COMPILE=/path/to/cc/$arch/bin/$arch- \
>              run
>     done
>              
> Which is simple enough without requiring a user to have to figure what
> build option to hack on for just a regular build.
>

Yeah, passing it from a script is a substitute.

> > And further, the output of the other targets should be put in the
> > $(ARCH_OUTPUT) directory too, include the sysroot, nolibc-test, initramfs,
> > run.out and even libc-test, the whole Makefile of selftests/nolibc will be
> > touched.
> 
> It's a matter of taste but there are pros and cons. I personally spend
> my time using grep -r on the sysroots, having gigabytes of .o there would
> quickly made me regret that approach. I would also argue that when you
> pass "O=" to the kernel build, it serves as a prefix and the directory
> hierarchy is preserved, so if you would want to put the sysroots etc
> under $O then it would be more logical that it appears in
> $O/tools/testing/selftests/nolibc/sysroot/$arch, which would then remain
> compatible with today when O corresponds to the kernel's CURDIR. Then
> maybe that would be the solution to making O per-arch for those who want
> to re-run over many kernels, while keeping it easy for those who want to
> test their own kernel or want to grep in sysroot.
>

Agree, using a $(ARCH_OUTPUT) does changes directory hierarchy.

> > I have prepared and verified such a solution locally, before send the patches,
> > welcome your suggestions.
> > 
> > My local patchset now provides such features:
> > 
> >   - Allow build everything for a target arch in the arch specific directory
> >   
> >   - Allow clean everything of a target arch
> >   
> >   - Allow trigger a config target automatically for the run target, use the new
> >     tinyconfig by default, be able to choose defconfig as before
> >   
> >   - Allow configure additional config options for both defconfig and tinyconfig
> >   
> >   - Allow configure the default CROSS_COMPILE by default, only the ARCH variable is required
> >   
> >   - Allow download the missing cross toolchains automatically (from mirrrors.edge.kernel.org)
> >   
> >   - Allow download the missing bios automatically
> >   
> >   - Allow download and build musl automatically
> >   
> >   - Allow run tests for all of the supported architectures automatically with one command
> >     with minimal preparation, may be only the installation of qemu-user and qemu-system
> 
> There can be good things there, but I'm seeing a lot of "by default" and
> "automatically" which worry me, because when you're debugging something,
> there's nothing more annoying than all the stuff happening in your back
> and fooling your analysis. On the opposite I do value the ability to
> proceed through steps that I don't need to revalidate, and these ones
> can be done automatically just by chaining a few commands from a script.
>

Ok, let these work in a standalone script.

> And no download should ever be done automatically. Any download must
> always be deliberately and consciously initiated by the user. How many
> times we've seen some scripts stall with no information, just to discover
> a SYN_SENT socket in netstat! You can have a "download-toolchain",
> "download-bios", "download-musl" etc if you want, but quite frankly, I'm
> seriously doubting it will be *that* useful to anyone working on the
> kernel since by definition they already have these toolchains, and
> sometimes different ones (e.g. linaro, bootlin etc) and in different
> versions. In fact if some expresses the need for having a simplified
> command to download a toolchain, then it wouldn't be specific to
> nolibc-test but would be used as a general tool for the kernel since
> it would be shared with all the rest of the code. You could also imagine
> downloading and building qemu, but it also takes a while and we're very
> far from the simple build script whose purpose was to validate that a
> few syscalls appeared to work fine.
>

More work may in reality add extra unexpected work outside of our core
target.

> > With the new patchset, it is able to rebuild and rerun everything in several
> > minutes, it may make people very happy to develop nolibc itself and also make
> > people to work on linux with nolibc, especially for developing and testing a
> > new kernel feature ;-)
> 
> I doubt it. Rebuilding everything is not what most developers need. What
> they need is to just fix a missing parenthesis that broke the build and
> rebuild the strict minimum, restarting from where they left. And they
> need to be able to figure what caused that "FAILED" output, disassemble
> the output, re-run it manually under qemu-user, rebuild the .c manually
> after copy-pasting the whole command line via "V=1", etc.
>

It is mainly for a cross arch change, like the _start_c() patchset we
just discuss. This may also happen for every release (not that helpful,
for a new release, a mrproper or distclean may be required).

> Keep in mind that the purpose of a selftest is not to optimize the case
> where it works fine, but to optimize the developer's work when it fails.
> This is something that a lot of people tend to get wrong. They're proud
> of a "make world" that does everything including delivering them pizzas
> to wait for the builds to complete, and they say "hey, impressed?". But
> when someone else reports "I'm getting this strange error I don't
> understand", they can hardly suggest more than "hmmm run make clean, try
> again, or use the same distro as me because it works for me".
>

Parts of them do want to meet the 'optimize the developer's work when it
fails', for example, new developers may be hard to find a loongarch or
riscv bios, or find a toolchain for the not frequently used
architecture, to avoid introduce many bug reports about "strange errors"
outside of our core functions, perhaps we'd better do these in a nolibc
doc under Documentation/, tell people how to prepare the develop and
test environment of nolibc and how to use nolibc there.

> What I've tried to focus on from the beginning with this tiny test suite
> precisely is debugging. That's why I wanted to support alternate libcs,
> to help compare, making it easy to use a different toolchain and other
> cflags, running in userland, building just the binary so that I can scp
> it to a remote native machine etc. All of these are important, and your
> approach above makes me feel like I would lose most of it, or still be
> able to do that but in a very complicated way (i.e. constantly have to
> reverse-engineer a complicated makefile to figure where I can inject an
> option).
>

Let's move most of them to a nolibc doc.

> However one thing I mentioned in the past is the usefulness of some
> batching scripts (I even shared one).

Yeah, I remember, Thanks. I have used something similar in my Linux Lab
project.

> And I think that helping the user
> prepare certain steps or iterate over architectures *is* useful. When
> you do it in two layers (the script replacing the user, the makefile
> doing the build job), it remains easy and convenient to use, and you
> can pick only what you need (e.g. "please build musl for me"). And if
> something goes wrong, it's simple for the user to takeover and complete
> that failed task by changing an arch name, a directory or anything, and
> have their tools ready. Let's just never make that automatic for the
> sake of everyone's health!

Ok, the revision will align with the original Makefile and remove the automatic
parts and no change about the OUTPUT.

Best regards,
Zhangjin

> 
> Thanks,
> Willy
  

On Wed, Jul 12, 2023 at 01:18:26AM +0800, Zhangjin Wu wrote:
> > > With the new patchset, it is able to rebuild and rerun everything in several
> > > minutes, it may make people very happy to develop nolibc itself and also make
> > > people to work on linux with nolibc, especially for developing and testing a
> > > new kernel feature ;-)
> > 
> > I doubt it. Rebuilding everything is not what most developers need. What
> > they need is to just fix a missing parenthesis that broke the build and
> > rebuild the strict minimum, restarting from where they left. And they
> > need to be able to figure what caused that "FAILED" output, disassemble
> > the output, re-run it manually under qemu-user, rebuild the .c manually
> > after copy-pasting the whole command line via "V=1", etc.
> >
> 
> It is mainly for a cross arch change, like the _start_c() patchset we
> just discuss. This may also happen for every release (not that helpful,
> for a new release, a mrproper or distclean may be required).

Maybe, but let's focus on fixing what's really annoying before trying
to improve things that work. If you find yourself failing to do certain
things, annoyed with some flags that are forced on you etc, not finding
a way to work with multiple output dirs, these are good reasons for
applying changes. But thinking that it will likely be better organized
this or that way tends to make us forget what we're relying on and that
we might lose by accident.

> > Keep in mind that the purpose of a selftest is not to optimize the case
> > where it works fine, but to optimize the developer's work when it fails.
> > This is something that a lot of people tend to get wrong. They're proud
> > of a "make world" that does everything including delivering them pizzas
> > to wait for the builds to complete, and they say "hey, impressed?". But
> > when someone else reports "I'm getting this strange error I don't
> > understand", they can hardly suggest more than "hmmm run make clean, try
> > again, or use the same distro as me because it works for me".
> >
> 
> Parts of them do want to meet the 'optimize the developer's work when it
> fails', for example, new developers may be hard to find a loongarch or
> riscv bios, or find a toolchain for the not frequently used
> architecture, to avoid introduce many bug reports about "strange errors"
> outside of our core functions, perhaps we'd better do these in a nolibc
> doc under Documentation/, tell people how to prepare the develop and
> test environment of nolibc and how to use nolibc there.

Reading the beginning of the sentence made me immediately think that it's
what doc is for. You know, if you give a fish to a hungry man he eats one
day, if you teach him to fish he eats every day. Knowing what to download
from where is much more instructive than running "make download" or even
worse, "make" and having the downloads secretly succeed (or fail). If you
think the doc is hard to find I'm also fine with a patch for makefile
and/or nolibc-test passing a pointer to its location as a reminding
comment for example.

> > And I think that helping the user
> > prepare certain steps or iterate over architectures *is* useful. When
> > you do it in two layers (the script replacing the user, the makefile
> > doing the build job), it remains easy and convenient to use, and you
> > can pick only what you need (e.g. "please build musl for me"). And if
> > something goes wrong, it's simple for the user to takeover and complete
> > that failed task by changing an arch name, a directory or anything, and
> > have their tools ready. Let's just never make that automatic for the
> > sake of everyone's health!
> 
> Ok, the revision will align with the original Makefile and remove the automatic
> parts and no change about the OUTPUT.

Just check that you can force it from your script on the make command
line. If you see that it's not possible, we should do something because
I don't want to force you to make distclean all the time if not needed.
But if you find that passing certain options (O=, OUTPUT= or anything
else) does the job, it only needs to be documented.

Thanks,
Willy
  

Hi, Willy, Thomas

I have prepared the powerpc + powerpc64 support together with the
tinyconfig for them, still have some questions for more discussion.

> On Wed, Jul 12, 2023 at 01:18:26AM +0800, Zhangjin Wu wrote:
[...]
> 
> Reading the beginning of the sentence made me immediately think that it's
> what doc is for. You know, if you give a fish to a hungry man he eats one
> day, if you teach him to fish he eats every day. Knowing what to download
> from where is much more instructive than running "make download" or even
> worse, "make" and having the downloads secretly succeed (or fail). If you
> think the doc is hard to find I'm also fine with a patch for makefile
> and/or nolibc-test passing a pointer to its location as a reminding
> comment for example.
>

The whole tinyconfig support for every architecture is 'huge', I plan to
send them one by one, if required, will document them with the required
bios and/or toolchain.

The first architectures plan to support are powerpc + powerpc64, powerpc does
require extra kernel config options even with defconfig, so, it is a very good
first example, and the extconfig target will be added together.

The left question from me is that if is it ok to just use tinyconfig instead of
defconfig after we enable tinyconfig for a new architecture, I mean just add a
new DEFCONFIG_<ARCH>=tinyconfig line for the new architecture, don't use the
'defconfig' any more, let's take a look at the powerpc/powerpc64 lines below:

    # default kernel configurations that appear to be usable
    DEFCONFIG_i386       = defconfig
    DEFCONFIG_x86_64     = defconfig
    DEFCONFIG_x86        = defconfig
    DEFCONFIG_arm64      = defconfig
    DEFCONFIG_arm        = multi_v7_defconfig
    DEFCONFIG_mips       = malta_defconfig
    DEFCONFIG_powerpc    = tinyconfig
    DEFCONFIG_powerpc64  = tinyconfig
    DEFCONFIG_riscv      = defconfig
    DEFCONFIG_s390       = defconfig
    DEFCONFIG_loongarch  = defconfig
    DEFCONFIG            = $(DEFCONFIG_$(XARCH))

Of course, we need to customize the EXTCONFIG for them (about ~5 options for
all of the architectures):

    # extra kernel configs by architecture
    EXTCONFIG_powerpc    = $(addprefix -e ,COMPAT_32BIT_TIME PPC_PMAC PPC_OF_BOOT_TRAMPOLINE SERIAL_PMACZILOG SERIAL_PMACZILOG_TTYS SERIAL_PMACZILOG_CONSOLE)
    EXTCONFIG_powerpc64  = $(addprefix -e ,PPC64 CPU_LITTLE_ENDIAN PPC_POWERNV HVC_OPAL)
    EXTCONFIG_XARCH      = $(EXTCONFIG_$(XARCH))

The extra common options (based on default kernel tinyconfig) are also required
to make nolibc-test.c passes without failures (~2 skips are procfs related, so,
procfs is not added) are minimal:

    # extra kernel configs shared among architectures
    EXTCONFIG_COMMON     = -e BLK_DEV_INITRD --set-str INITRAMFS_SOURCE $(CURDIR)/initramfs
    EXTCONFIG_COMMON    += -e BINFMT_ELF
    EXTCONFIG_COMMON    += -e PRINTK -e TTY

Compare to defconfig, tinyconfig not only allows test all of the nolibc
functions, but also is faster (around ~1-2 minutes, defconfig may cost ~30
minutes and even more) and brings us with smaller image size.

To only test nolibc itself, I do think tinyconfig with the above
extconfig support is enough, even if we need more, we can update the
EXTCONFIG_COMMON and EXTCONFIG_<ARCH> in the future.

I have prepared tinyconfig for all of the supported architectures
locally, If you agree with only reserve the DEFCONFIG_<ARCH>=tinyconfig
line, I will send a series of patchset to add tinyconfig for every
architecture with it, at last, it will become:

    # default kernel configurations that appear to be usable
    DEFCONFIG_i386       = tinyconfig
    DEFCONFIG_x86_64     = tinyconfig
    DEFCONFIG_x86        = tinyconfig
    DEFCONFIG_arm64      = tinyconfig
    DEFCONFIG_arm        = tinyconfig
    DEFCONFIG_mips       = tinyconfig
    DEFCONFIG_powerpc    = tinyconfig
    DEFCONFIG_powerpc64  = tinyconfig
    DEFCONFIG_riscv      = tinyconfig
    DEFCONFIG_s390       = tinyconfig
    DEFCONFIG_loongarch  = tinyconfig
    DEFCONFIG            = $(DEFCONFIG_$(XARCH))

So, perhaps it is better to simply use tinyconfig as the default DEFCONFIG, and
therefore there is no need to add powerpc and powerpc64 specific lines:

    # default kernel configurations that appear to be usable
    DEFCONFIG_i386       = defconfig
    DEFCONFIG_x86_64     = defconfig
    DEFCONFIG_x86        = defconfig
    DEFCONFIG_arm64      = defconfig
    DEFCONFIG_arm        = multi_v7_defconfig
    DEFCONFIG_mips       = malta_defconfig
    DEFCONFIG_riscv      = defconfig
    DEFCONFIG_s390       = defconfig
    DEFCONFIG_loongarch  = defconfig
    DEFCONFIG            = $(or $(DEFCONFIG_$(XARCH)),tinyconfig)

To support tinyconfig for a new architecture, we can simply remove the
'DEFCONFIG_<ARCH> = defconfig' line and get the core options from
defconfig to customize the EXTCONFIG_ARCH, with tinyconfig, it is very
fast and easy to verify the run target for a new architecture.

At last, we will have many EXTCONFIG_<ARCH> lines and only a DEFCONFIG line:

    # default kernel configurations that appear to be usable
    DEFCONFIG            = $(or $(DEFCONFIG_$(XARCH)),tinyconfig)

Or at last, we remove the above line and the defconfig target and only reserve
a tinyconfig target:

    tinyconfig:
	$(Q)$(MAKE_KERNEL) tinyconfig prepare

Welcome your suggestion.

> > > And I think that helping the user
> > > prepare certain steps or iterate over architectures *is* useful. When
> > > you do it in two layers (the script replacing the user, the makefile
> > > doing the build job), it remains easy and convenient to use, and you
> > > can pick only what you need (e.g. "please build musl for me"). And if
> > > something goes wrong, it's simple for the user to takeover and complete
> > > that failed task by changing an arch name, a directory or anything, and
> > > have their tools ready. Let's just never make that automatic for the
> > > sake of everyone's health!
> > 
> > Ok, the revision will align with the original Makefile and remove the automatic
> > parts and no change about the OUTPUT.
> 
> Just check that you can force it from your script on the make command
> line. If you see that it's not possible, we should do something because
> I don't want to force you to make distclean all the time if not needed.
> But if you find that passing certain options (O=, OUTPUT= or anything
> else) does the job, it only needs to be documented.

Yeah, I have used objtree intead of srctree to fix up the O= argument
support, it fills my requirement to build kernel for every architecture
in their own output directory.

Best regards,
Zhangjin

> 
> Thanks,
> Willy
  

On 2023-07-18 21:43:23+0800, Zhangjin Wu wrote:
> Hi, Willy, Thomas
> 
> I have prepared the powerpc + powerpc64 support together with the
> tinyconfig for them, still have some questions for more discussion.
> 
> > On Wed, Jul 12, 2023 at 01:18:26AM +0800, Zhangjin Wu wrote:
> [...]
> > 
> > Reading the beginning of the sentence made me immediately think that it's
> > what doc is for. You know, if you give a fish to a hungry man he eats one
> > day, if you teach him to fish he eats every day. Knowing what to download
> > from where is much more instructive than running "make download" or even
> > worse, "make" and having the downloads secretly succeed (or fail). If you
> > think the doc is hard to find I'm also fine with a patch for makefile
> > and/or nolibc-test passing a pointer to its location as a reminding
> > comment for example.
> >
> 
> The whole tinyconfig support for every architecture is 'huge', I plan to
> send them one by one, if required, will document them with the required
> bios and/or toolchain.

Which part is huge?
This is surprising.

> The first architectures plan to support are powerpc + powerpc64, powerpc does
> require extra kernel config options even with defconfig, so, it is a very good
> first example, and the extconfig target will be added together.

Are you planning to do powerpc and tinyconfig support in one series?
Splitting it would be better in my opinion.

> The left question from me is that if is it ok to just use tinyconfig instead of
> defconfig after we enable tinyconfig for a new architecture, I mean just add a
> new DEFCONFIG_<ARCH>=tinyconfig line for the new architecture, don't use the
> 'defconfig' any more, let's take a look at the powerpc/powerpc64 lines below:
> 
>     # default kernel configurations that appear to be usable
>     DEFCONFIG_i386       = defconfig
>     DEFCONFIG_x86_64     = defconfig
>     DEFCONFIG_x86        = defconfig
>     DEFCONFIG_arm64      = defconfig
>     DEFCONFIG_arm        = multi_v7_defconfig
>     DEFCONFIG_mips       = malta_defconfig
>     DEFCONFIG_powerpc    = tinyconfig
>     DEFCONFIG_powerpc64  = tinyconfig
>     DEFCONFIG_riscv      = defconfig
>     DEFCONFIG_s390       = defconfig
>     DEFCONFIG_loongarch  = defconfig
>     DEFCONFIG            = $(DEFCONFIG_$(XARCH))
> 
> Of course, we need to customize the EXTCONFIG for them (about ~5 options for
> all of the architectures):
> 
>     # extra kernel configs by architecture
>     EXTCONFIG_powerpc    = $(addprefix -e ,COMPAT_32BIT_TIME PPC_PMAC PPC_OF_BOOT_TRAMPOLINE SERIAL_PMACZILOG SERIAL_PMACZILOG_TTYS SERIAL_PMACZILOG_CONSOLE)
>     EXTCONFIG_powerpc64  = $(addprefix -e ,PPC64 CPU_LITTLE_ENDIAN PPC_POWERNV HVC_OPAL)
>     EXTCONFIG_XARCH      = $(EXTCONFIG_$(XARCH))

These could also be put into dedicated config files. Then they don't
clutter the makefile and it's easier to maintain.

nolibc.config:       Generic configs on top of tinyconfig
nolibc.arm64.config: Arch-specific configs for arm64

Also the extra parameter could also be passed via command line arguments to make.
This way we don't have to modify the configuration options at all.
The user can provide a config (or we use a tinyconfig) and everything
required by nolibc-test is enabled on top when building.

make CONFIG_COMPAT_32BIT_TIME=y ...

tools/testing/selftests/wireguard/qemu/ seems to be doing something
similar as nolibc-test.

> The extra common options (based on default kernel tinyconfig) are also required
> to make nolibc-test.c passes without failures (~2 skips are procfs related, so,
> procfs is not added) are minimal:
> 
>     # extra kernel configs shared among architectures
>     EXTCONFIG_COMMON     = -e BLK_DEV_INITRD --set-str INITRAMFS_SOURCE $(CURDIR)/initramfs
>     EXTCONFIG_COMMON    += -e BINFMT_ELF
>     EXTCONFIG_COMMON    += -e PRINTK -e TTY
> 
> Compare to defconfig, tinyconfig not only allows test all of the nolibc
> functions, but also is faster (around ~1-2 minutes, defconfig may cost ~30
> minutes and even more) and brings us with smaller image size.
> 
> To only test nolibc itself, I do think tinyconfig with the above
> extconfig support is enough, even if we need more, we can update the
> EXTCONFIG_COMMON and EXTCONFIG_<ARCH> in the future.

IMO tinyconfig is enough, defconfig doesn't seem to be necessary then.

> I have prepared tinyconfig for all of the supported architectures
> locally, If you agree with only reserve the DEFCONFIG_<ARCH>=tinyconfig
> line, I will send a series of patchset to add tinyconfig for every
> architecture with it, at last, it will become:
> 
>     # default kernel configurations that appear to be usable
>     DEFCONFIG_i386       = tinyconfig
>     DEFCONFIG_x86_64     = tinyconfig
>     DEFCONFIG_x86        = tinyconfig
>     DEFCONFIG_arm64      = tinyconfig
>     DEFCONFIG_arm        = tinyconfig
>     DEFCONFIG_mips       = tinyconfig
>     DEFCONFIG_powerpc    = tinyconfig
>     DEFCONFIG_powerpc64  = tinyconfig
>     DEFCONFIG_riscv      = tinyconfig
>     DEFCONFIG_s390       = tinyconfig
>     DEFCONFIG_loongarch  = tinyconfig
>     DEFCONFIG            = $(DEFCONFIG_$(XARCH))
> 
> So, perhaps it is better to simply use tinyconfig as the default DEFCONFIG, and
> therefore there is no need to add powerpc and powerpc64 specific lines:
> 
>     # default kernel configurations that appear to be usable
>     DEFCONFIG_i386       = defconfig
>     DEFCONFIG_x86_64     = defconfig
>     DEFCONFIG_x86        = defconfig
>     DEFCONFIG_arm64      = defconfig
>     DEFCONFIG_arm        = multi_v7_defconfig
>     DEFCONFIG_mips       = malta_defconfig
>     DEFCONFIG_riscv      = defconfig
>     DEFCONFIG_s390       = defconfig
>     DEFCONFIG_loongarch  = defconfig
>     DEFCONFIG            = $(or $(DEFCONFIG_$(XARCH)),tinyconfig)
> 
> To support tinyconfig for a new architecture, we can simply remove the
> 'DEFCONFIG_<ARCH> = defconfig' line and get the core options from
> defconfig to customize the EXTCONFIG_ARCH, with tinyconfig, it is very
> fast and easy to verify the run target for a new architecture.
> 
> At last, we will have many EXTCONFIG_<ARCH> lines and only a DEFCONFIG line:
> 
>     # default kernel configurations that appear to be usable
>     DEFCONFIG            = $(or $(DEFCONFIG_$(XARCH)),tinyconfig)
> 
> Or at last, we remove the above line and the defconfig target and only reserve
> a tinyconfig target:
> 
>     tinyconfig:
> 	$(Q)$(MAKE_KERNEL) tinyconfig prepare
> 
> Welcome your suggestion.

Looks fine to me either way.

> > > > And I think that helping the user
> > > > prepare certain steps or iterate over architectures *is* useful. When
> > > > you do it in two layers (the script replacing the user, the makefile
> > > > doing the build job), it remains easy and convenient to use, and you
> > > > can pick only what you need (e.g. "please build musl for me"). And if
> > > > something goes wrong, it's simple for the user to takeover and complete
> > > > that failed task by changing an arch name, a directory or anything, and
> > > > have their tools ready. Let's just never make that automatic for the
> > > > sake of everyone's health!
> > > 
> > > Ok, the revision will align with the original Makefile and remove the automatic
> > > parts and no change about the OUTPUT.
> > 
> > Just check that you can force it from your script on the make command
> > line. If you see that it's not possible, we should do something because
> > I don't want to force you to make distclean all the time if not needed.
> > But if you find that passing certain options (O=, OUTPUT= or anything
> > else) does the job, it only needs to be documented.
> 
> Yeah, I have used objtree intead of srctree to fix up the O= argument
> support, it fills my requirement to build kernel for every architecture
> in their own output directory.
> 
> Best regards,
> Zhangjin
> 
> > 
> > Thanks,
> > Willy
  

Hi Zhangjin, Thomas,

Just a quick response below on one point.

On Tue, Jul 18, 2023 at 05:19:50PM +0200, Thomas Weißschuh wrote:

> > The first architectures plan to support are powerpc + powerpc64, powerpc does
> > require extra kernel config options even with defconfig, so, it is a very good
> > first example, and the extconfig target will be added together.
> 
> Are you planning to do powerpc and tinyconfig support in one series?
> Splitting it would be better in my opinion.

I agree for splitting.

(...)
> > To only test nolibc itself, I do think tinyconfig with the above
> > extconfig support is enough, even if we need more, we can update the
> > EXTCONFIG_COMMON and EXTCONFIG_<ARCH> in the future.
> 
> IMO tinyconfig is enough, defconfig doesn't seem to be necessary then.

This is a point where I think I disagree. In fact the important feature
of defconfig, for any arch, is that it's supposed to work out of the box.
If we need to add extra options on top of tiny config, someone will
regularly have to maintain this set of options up to date when more are
added or when they're split. It could even start to fire back to a few
developers who want to make the config more modular and are forced to
update these ones without knowing well how to proceed with nolibc. Thus
I would suggest that defconfig remains available (as a make target) for
when we want to rely on a safe config, and that tinyconfig + local tuning
is available for those like us who tend to spend more time on nolibc and
who don't care much about having to tweak some options once in a while.

Just my two cents,
Willy
  

Hi, Thomas

> On 2023-07-18 21:43:23+0800, Zhangjin Wu wrote:
> > Hi, Willy, Thomas
> > 
> > I have prepared the powerpc + powerpc64 support together with the
> > tinyconfig for them, still have some questions for more discussion.
> > 
> > > On Wed, Jul 12, 2023 at 01:18:26AM +0800, Zhangjin Wu wrote:
> > [...]
> > > 
> > > Reading the beginning of the sentence made me immediately think that it's
> > > what doc is for. You know, if you give a fish to a hungry man he eats one
> > > day, if you teach him to fish he eats every day. Knowing what to download
> > > from where is much more instructive than running "make download" or even
> > > worse, "make" and having the downloads secretly succeed (or fail). If you
> > > think the doc is hard to find I'm also fine with a patch for makefile
> > > and/or nolibc-test passing a pointer to its location as a reminding
> > > comment for example.
> > >
> > 
> > The whole tinyconfig support for every architecture is 'huge', I plan to
> > send them one by one, if required, will document them with the required
> > bios and/or toolchain.
> 
> Which part is huge?

Actually, we only need to add ~30 lines for all of them ;-)

But some architectures require extra operations, for example,

- x86 requires extra "tsc=unstable noapic" kernel cmdline to save the
  CONFIG_ACPI options, if we only reserve tinyconfig target, it is ok to
  add it directly to the original QEMU_ARGS_<ARCH>

- x86 and some other architectures may fail to poweroff due to save some
  time-cost options or due to missing kernel poweroff support, they
  require qemu-poweroff timeout support

- some architectures have no built-in firmware for qemu, we need to
  document them carefully

- some architectures have no available toolchain in any Linux
  distributions, we need to document them carefully

...

> This is surprising.
>

For one architecture, it is not that hard, but for all of the supported
architectures, weird bugs came out one by one ;-)

> > The first architectures plan to support are powerpc + powerpc64, powerpc does
> > require extra kernel config options even with defconfig, so, it is a very good
> > first example, and the extconfig target will be added together.
> 
> Are you planning to do powerpc and tinyconfig support in one series?
> Splitting it would be better in my opinion.
>

Ok, as Willy replied in another thread, let's add powerpc support with
defconfig as the first patchset.

Since, even with defconfig, we still need extra console options to make
the test print normally, so, the extra options support code will be
added anyway, adding tinyconfig support is a fastforward step.

> > The left question from me is that if is it ok to just use tinyconfig instead of
> > defconfig after we enable tinyconfig for a new architecture, I mean just add a
> > new DEFCONFIG_<ARCH>=tinyconfig line for the new architecture, don't use the
> > 'defconfig' any more, let's take a look at the powerpc/powerpc64 lines below:
> > 
> >     # default kernel configurations that appear to be usable
> >     DEFCONFIG_i386       = defconfig
> >     DEFCONFIG_x86_64     = defconfig
> >     DEFCONFIG_x86        = defconfig
> >     DEFCONFIG_arm64      = defconfig
> >     DEFCONFIG_arm        = multi_v7_defconfig
> >     DEFCONFIG_mips       = malta_defconfig
> >     DEFCONFIG_powerpc    = tinyconfig
> >     DEFCONFIG_powerpc64  = tinyconfig
> >     DEFCONFIG_riscv      = defconfig
> >     DEFCONFIG_s390       = defconfig
> >     DEFCONFIG_loongarch  = defconfig
> >     DEFCONFIG            = $(DEFCONFIG_$(XARCH))
> > 
> > Of course, we need to customize the EXTCONFIG for them (about ~5 options for
> > all of the architectures):
> > 
> >     # extra kernel configs by architecture
> >     EXTCONFIG_powerpc    = $(addprefix -e ,COMPAT_32BIT_TIME PPC_PMAC PPC_OF_BOOT_TRAMPOLINE SERIAL_PMACZILOG SERIAL_PMACZILOG_TTYS SERIAL_PMACZILOG_CONSOLE)
> >     EXTCONFIG_powerpc64  = $(addprefix -e ,PPC64 CPU_LITTLE_ENDIAN PPC_POWERNV HVC_OPAL)
> >     EXTCONFIG_XARCH      = $(EXTCONFIG_$(XARCH))
> 
> These could also be put into dedicated config files. Then they don't
> clutter the makefile and it's easier to maintain.
> 
> nolibc.config:       Generic configs on top of tinyconfig
> nolibc.arm64.config: Arch-specific configs for arm64
>

Thanks, xxx.config files should work well with the merge tool:
scripts/kconfig/merge_config.sh.

Since in all of the supported architectures, the maximum extra options
required by tinyconfig is 8, the others is around 5 options (some even
less), all of them can be put in one line, at the same time, the
initramfs source can use the same EXTCONFIG_COMMON logic, so, the
oneline EXTCONFIG_<ARCH> is used without any new files.

Will take a look at the xxx.config files method.

But unfortunately, powerpc defconfig doesn't print by default, it also
require extra console options, to apply the xxx.config files method, we
need wait more time to restruct the powerpc support.

> Also the extra parameter could also be passed via command line arguments to make.
> This way we don't have to modify the configuration options at all.
> The user can provide a config (or we use a tinyconfig) and everything
> required by nolibc-test is enabled on top when building.
> 
> make CONFIG_COMPAT_32BIT_TIME=y ...
>

This doesn't work sometimes especially when we want to disable an
option, that's why we eventually use scripts/config instead.

> tools/testing/selftests/wireguard/qemu/ seems to be doing something
> similar as nolibc-test.
> 
> > The extra common options (based on default kernel tinyconfig) are also required
> > to make nolibc-test.c passes without failures (~2 skips are procfs related, so,
> > procfs is not added) are minimal:
> > 
> >     # extra kernel configs shared among architectures
> >     EXTCONFIG_COMMON     = -e BLK_DEV_INITRD --set-str INITRAMFS_SOURCE $(CURDIR)/initramfs
> >     EXTCONFIG_COMMON    += -e BINFMT_ELF
> >     EXTCONFIG_COMMON    += -e PRINTK -e TTY
> > 
> > Compare to defconfig, tinyconfig not only allows test all of the nolibc
> > functions, but also is faster (around ~1-2 minutes, defconfig may cost ~30
> > minutes and even more) and brings us with smaller image size.
> > 
> > To only test nolibc itself, I do think tinyconfig with the above
> > extconfig support is enough, even if we need more, we can update the
> > EXTCONFIG_COMMON and EXTCONFIG_<ARCH> in the future.
> 
> IMO tinyconfig is enough, defconfig doesn't seem to be necessary then.
>

As Willy replied in another thread, tinyconfig may fail sometimes
because many developers only use defconfig for tests, I did find
tinyconfig fail on powerpc [1] and riscv64 [2], although later, both of
them have been fixed.

I reported the 2nd one, the 1st one has just been fixed up by somebody
else before I plan to send the patch, so, our nolibc tinyconfig support
may function as a very good coverage test for kernel ;-)

[1]: https://lore.kernel.org/lkml/20230629124500.1.I55e2f4e7903d686c4484cb23c033c6a9e1a9d4c4@changeid/
[2]: https://lore.kernel.org/linux-riscv/20230716001506.3506041-1-guoren@kernel.org/

> > I have prepared tinyconfig for all of the supported architectures
> > locally, If you agree with only reserve the DEFCONFIG_<ARCH>=tinyconfig
> > line, I will send a series of patchset to add tinyconfig for every
> > architecture with it, at last, it will become:
> > 
> >     # default kernel configurations that appear to be usable
> >     DEFCONFIG_i386       = tinyconfig
> >     DEFCONFIG_x86_64     = tinyconfig
> >     DEFCONFIG_x86        = tinyconfig
> >     DEFCONFIG_arm64      = tinyconfig
> >     DEFCONFIG_arm        = tinyconfig
> >     DEFCONFIG_mips       = tinyconfig
> >     DEFCONFIG_powerpc    = tinyconfig
> >     DEFCONFIG_powerpc64  = tinyconfig
> >     DEFCONFIG_riscv      = tinyconfig
> >     DEFCONFIG_s390       = tinyconfig
> >     DEFCONFIG_loongarch  = tinyconfig
> >     DEFCONFIG            = $(DEFCONFIG_$(XARCH))
> > 
> > So, perhaps it is better to simply use tinyconfig as the default DEFCONFIG, and
> > therefore there is no need to add powerpc and powerpc64 specific lines:
> > 
> >     # default kernel configurations that appear to be usable
> >     DEFCONFIG_i386       = defconfig
> >     DEFCONFIG_x86_64     = defconfig
> >     DEFCONFIG_x86        = defconfig
> >     DEFCONFIG_arm64      = defconfig
> >     DEFCONFIG_arm        = multi_v7_defconfig
> >     DEFCONFIG_mips       = malta_defconfig
> >     DEFCONFIG_riscv      = defconfig
> >     DEFCONFIG_s390       = defconfig
> >     DEFCONFIG_loongarch  = defconfig
> >     DEFCONFIG            = $(or $(DEFCONFIG_$(XARCH)),tinyconfig)
> > 
> > To support tinyconfig for a new architecture, we can simply remove the
> > 'DEFCONFIG_<ARCH> = defconfig' line and get the core options from
> > defconfig to customize the EXTCONFIG_ARCH, with tinyconfig, it is very
> > fast and easy to verify the run target for a new architecture.
> > 
> > At last, we will have many EXTCONFIG_<ARCH> lines and only a DEFCONFIG line:
> > 
> >     # default kernel configurations that appear to be usable
> >     DEFCONFIG            = $(or $(DEFCONFIG_$(XARCH)),tinyconfig)
> > 
> > Or at last, we remove the above line and the defconfig target and only reserve
> > a tinyconfig target:
> > 
> >     tinyconfig:
> > 	$(Q)$(MAKE_KERNEL) tinyconfig prepare
> > 
> > Welcome your suggestion.
> 
> Looks fine to me either way.

Ok, so, let's reserve defconfig as a backup although tinyconfig should
work at most of the time.

Best regards,
Zhangjin