[1/2] package: add tar development package for 3rd party modules
Commit Message
Most, if not all, Linux distributions provides a Linux development
package which purpose is to support the building of out-of-tree modules
without providing the entire source tree.
What ends up in this development directory is a mixture of source
files (mainly headers) and generated ones (headers, and tools produced
by `make modules_prepare`).
This patch is an attempt to generate a tarball archive containing all
required files to build external modules. It could be than reused by
packagers.
Signed-off-by: Federico Vaga <federico.vaga@cern.ch>
---
Makefile | 2 +-
scripts/Makefile.package | 13 +++
scripts/package/buildtar-devel | 207 +++++++++++++++++++++++++++++++++
3 files changed, 221 insertions(+), 1 deletion(-)
create mode 100644 scripts/package/buildtar-devel
Comments
Hi--
On 10/21/22 03:14, Federico Vaga wrote:
> Most, if not all, Linux distributions provides a Linux development
> package which purpose is to support the building of out-of-tree modules
> without providing the entire source tree.
>
> What ends up in this development directory is a mixture of source
> files (mainly headers) and generated ones (headers, and tools produced
> by `make modules_prepare`).
>
> This patch is an attempt to generate a tarball archive containing all
> required files to build external modules. It could be than reused by
> packagers.
>
> Signed-off-by: Federico Vaga <federico.vaga@cern.ch>
> ---
> Makefile | 2 +-
> scripts/Makefile.package | 13 +++
> scripts/package/buildtar-devel | 207 +++++++++++++++++++++++++++++++++
> 3 files changed, 221 insertions(+), 1 deletion(-)
> create mode 100644 scripts/package/buildtar-devel
Is there a patch 2/2? I don't see it anywhere.
thanks.
On Fri, Oct 21, 2022 at 01:11:18PM -0700, Randy Dunlap wrote:
>Hi--
>
>On 10/21/22 03:14, Federico Vaga wrote:
>> Most, if not all, Linux distributions provides a Linux development
>> package which purpose is to support the building of out-of-tree modules
>> without providing the entire source tree.
>>
>> What ends up in this development directory is a mixture of source
>> files (mainly headers) and generated ones (headers, and tools produced
>> by `make modules_prepare`).
>>
>> This patch is an attempt to generate a tarball archive containing all
>> required files to build external modules. It could be than reused by
>> packagers.
>>
>> Signed-off-by: Federico Vaga <federico.vaga@cern.ch>
>> ---
>> Makefile | 2 +-
>> scripts/Makefile.package | 13 +++
>> scripts/package/buildtar-devel | 207 +++++++++++++++++++++++++++++++++
>> 3 files changed, 221 insertions(+), 1 deletion(-)
>> create mode 100644 scripts/package/buildtar-devel
>
>Is there a patch 2/2? I don't see it anywhere.
My mistake.
Yes there is a second one but I did not want to send it becuase it is about
generalizing buildtar to build 3 type of tarballs: the linux binaries to be
placed in /boot, the header files for user-space, and the development headers
and tools for out-of-tree modules (this patch).
The second one makes sense, only if this one makes sense. That's why I wrote few
lines in the RFC cover letter. I should have used the format-patch option to not
enumerate patches :)
>thanks.
>--
>~Randy
On 10/21/22 14:48, Federico Vaga wrote:
> On Fri, Oct 21, 2022 at 01:11:18PM -0700, Randy Dunlap wrote:
>> Hi--
>>
>> On 10/21/22 03:14, Federico Vaga wrote:
>>> Most, if not all, Linux distributions provides a Linux development
>>> package which purpose is to support the building of out-of-tree modules
>>> without providing the entire source tree.
>>>
>>> What ends up in this development directory is a mixture of source
>>> files (mainly headers) and generated ones (headers, and tools produced
>>> by `make modules_prepare`).
>>>
>>> This patch is an attempt to generate a tarball archive containing all
>>> required files to build external modules. It could be than reused by
>>> packagers.
>>>
>>> Signed-off-by: Federico Vaga <federico.vaga@cern.ch>
>>> ---
>>> Makefile | 2 +-
>>> scripts/Makefile.package | 13 +++
>>> scripts/package/buildtar-devel | 207 +++++++++++++++++++++++++++++++++
>>> 3 files changed, 221 insertions(+), 1 deletion(-)
>>> create mode 100644 scripts/package/buildtar-devel
>>
>> Is there a patch 2/2? I don't see it anywhere.
>
> My mistake.
>
> Yes there is a second one but I did not want to send it becuase it is about
> generalizing buildtar to build 3 type of tarballs: the linux binaries to be
> placed in /boot, the header files for user-space, and the development headers
> and tools for out-of-tree modules (this patch).
Ah, I was wondering why this change was included:
diff --git a/Makefile b/Makefile
index cfbe6a7de640..36a58394ce16 100644
--- a/Makefile
+++ b/Makefile
@@ -1578,7 +1578,7 @@ CLEAN_FILES += include/ksym vmlinux.symvers modules-only.symvers \
# Directories & files removed with 'make mrproper'
MRPROPER_FILES += include/config include/generated \
arch/$(SRCARCH)/include/generated .objdiff \
- debian snap tar-install \
+ debian snap tar-install* \
.config .config.old .version \
Module.symvers \
certs/signing_key.pem \
so I think that you just explained it.
> The second one makes sense, only if this one makes sense. That's why I wrote few
> lines in the RFC cover letter. I should have used the format-patch option to not
> enumerate patches :)
>
>> thanks.
>> --
>> ~Randy
>
@@ -1578,7 +1578,7 @@ CLEAN_FILES += include/ksym vmlinux.symvers modules-only.symvers \
# Directories & files removed with 'make mrproper'
MRPROPER_FILES += include/config include/generated \
arch/$(SRCARCH)/include/generated .objdiff \
- debian snap tar-install \
+ debian snap tar-install* \
.config .config.old .version \
Module.symvers \
certs/signing_key.pem \
@@ -112,6 +112,13 @@ $(tar-pkgs):
$(MAKE) -f $(srctree)/Makefile
+$(CONFIG_SHELL) $(srctree)/scripts/package/buildtar $@
+tar-dev-pkgs := dir-dev-pkg tar-dev-pkg targz-dev-pkg tarbz2-dev-pkg
+tar-dev-pkgs += tarxz-dev-pkg tarzst-dev-pkg
+PHONY += $(tar-dev-pkgs)
+$(tar-dev-pkgs):
+ $(MAKE) -f $(srctree)/Makefile
+ +$(CONFIG_SHELL) $(srctree)/scripts/package/buildtar-devel $@
+
# perf-pkg - generate a source tarball with perf source
# ---------------------------------------------------------------------------
@@ -159,6 +166,12 @@ help:
@echo ' tarbz2-pkg - Build the kernel as a bzip2 compressed tarball'
@echo ' tarxz-pkg - Build the kernel as a xz compressed tarball'
@echo ' tarzst-pkg - Build the kernel as a zstd compressed tarball'
+ @echo ' dir-dev-pkg - Module development as a plain directory structure'
+ @echo ' tar-dev-pkg - Module development as an uncompressed tarball'
+ @echo ' targz-dev-pkg - Module development as a gzip compressed tarball'
+ @echo ' tarbz2-dev-pkg - Module development as a bzip2 compressed tarball'
+ @echo ' tarxz-dev-pkg - Module development as a xz compressed tarball'
+ @echo ' tarzst-dev-pkg - Module development as a zstd compressed tarball'
@echo ' perf-tar-src-pkg - Build $(perf-tar).tar source tarball'
@echo ' perf-targz-src-pkg - Build $(perf-tar).tar.gz source tarball'
@echo ' perf-tarbz2-src-pkg - Build $(perf-tar).tar.bz2 source tarball'
new file mode 100644
@@ -0,0 +1,207 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+
+#
+# (C) 2022 CERN (home.cern)
+# Author Federico Vaga <federico.vaga@cern.ch>
+#
+# This script is used to build a 3rd party kernel module development tarball
+# from the currently prepared kernel.
+#
+
+set -ex
+
+#
+# Some variables and settings used throughout the script
+#
+tmpdir="${abs_objtree}/tar-install-dev"
+tarball="${abs_objtree}/linux-${KERNELRELEASE}-${ARCH}-dev.tar"
+
+#
+# Figure out how to compress, if requested at all
+#
+case "${1}" in
+ dir-pkg-dev|tar-pkg-dev)
+ opts=
+ ;;
+ targz-pkg-dev)
+ opts="-I ${KGZIP}"
+ tarball=${tarball}.gz
+ ;;
+ tarbz2-pkg-dev)
+ opts="-I ${KBZIP2}"
+ tarball=${tarball}.bz2
+ ;;
+ tarxz-pkg-dev)
+ opts="-I ${XZ}"
+ tarball=${tarball}.xz
+ ;;
+ tarzst-pkg-dev)
+ opts="-I ${ZSTD}"
+ tarball=${tarball}.zst
+ ;;
+ *)
+ echo "Unknown tarball target \"${1}\" requested, please add it to ${0}." >&2
+ exit 1
+ ;;
+esac
+
+#
+# Clean-up and re-create the temporary directory
+#
+
+rm -rf -- "${tmpdir}"
+mkdir -p -- "${tmpdir}"
+
+#
+# Copy required files
+#
+FILTER_FILE=$(mktemp)
+cat <<EOF > ${FILTER_FILE}
+
+#
+# Include the following files and directories required to build external modules
+#
+
++ /arch/
++ /arch/${ARCH}/
++ /arch/${ARCH}/**.h
++ /arch/${ARCH}/**/Makefile*
++ /.config
++ /drivers/
++ /drivers/**/Kbuild*
++ /drivers/**/Kconfig*
++ /drivers/**/Makefile*
++ /include/*
++ /include/**/*.h
++ /Kbuild
++ /Kconfig
++ /Makefile
++ /Module.symvers
++ /scripts/Makefile.*
++ /scripts/basic
++ /scripts/basic/fixdep
++ /scripts/bin2c
++ /scripts/checkincludes.pl
++ /scripts/checkstack.pl
++ /scripts/checkversion.pl
++ /scripts/check-local-export
++ /scripts/depmod.sh
++ /scripts/extract-cert
++ /scripts/gcc-goto.sh
++ /scripts/gcc-version.sh
++ /scripts/gcc-x86_32-has-stack-protector.sh
++ /scripts/gcc-x86_64-has-stack-protector.sh
++ /scripts/genksyms
++ /scripts/genksyms/genksyms
++ /scripts/headers_install.sh
++ /scripts/kallsyms
++ /scripts/Kbuild.include
++ /scripts/kconfig
++ /scripts/kconfig/conf
++ /scripts/kernel-doc
++ /scripts/ld-version.sh
++ /scripts/Lindent
++ /scripts/makelst
++ /scripts/mksysmap
++ /scripts/mkuboot.sh
++ /scripts/mod/
++ /scripts/mod/modpost
++ /scripts/module.lds
++ /scripts/patch-kernel
++ /scripts/pahole-flags.sh
++ /scripts/recordmcount
++ /scripts/recordmcount.pl
++ /scripts/setlocalversion
++ /scripts/sign-file
++ /scripts/subarch.include
++ /scripts/unifdef
++ /scripts/ver_linux
++ /tools
++ /tools/objtool/
++ /tools/objtool/objtool
++ /.version
+
+#
+# Completly ignore the following directories
+#
+- /arch/*
+- /Documentation
+- /LICENSES
+- /block
+- /certs
+- /crypto
+- /drivers/*
+- /fs
+- /init
+- /io_uring
+- /ipc
+- /kernel
+- /lib
+- /MAINTAINERS
+- /mm
+- /modules.*
+- /net
+- /README
+- /rust
+- /samples
+- /scripts/*
+- /scripts/basic/Makefile
+- /scripts/genksyms/*
+- /scripts/kconfig/*
+- /scripts/mod/*
+- /security
+- /sound
+- /System.map
+- /tools/*
+- /tools/objtool/*
+- /usr
+- /virt
+
+#
+# Completly ignore intermediate directories
+#
+- /source
+- /tar-install*
+- /${O}
+#
+# Completly ignore the following files
+#
+
+- .*
+- *.a
+- *.c
+- *.o
+- *.S
+- *vmlinux*
+EOF
+
+#
+# Copy from object directory as well when it is not the same as the source one
+#
+if [ "${abs_objtree}" != "${abs_srctree}" ]
+then
+ rsync -a --filter="merge ${FILTER_FILE}" ${abs_objtree}/ ${tmpdir}
+fi
+rsync -a --filter="merge ${FILTER_FILE}" ${abs_srctree}/ ${tmpdir}
+rm ${FILTER_FILE}
+
+if [ "${1}" = dir-pkg-dev ]; then
+ echo "Kernel tree successfully created in $tmpdir"
+ exit 0
+fi
+
+#
+# Create the tarball
+#
+if tar --owner=root --group=root --help >/dev/null 2>&1; then
+ opts="$opts --owner=root --group=root"
+fi
+
+opts="$opts"
+
+tar cf $tarball -C $tmpdir $opts .
+
+echo "Tarball successfully created in $tarball"
+
+exit 0