[2/3] kbuild: do not create intermediate *.tar for source tarballs
Commit Message
Since commit 05e96e96a315 ("kbuild: use git-archive for source package
creation"), source tarballs are created in two steps; create *.tar file
then compress it. I split the compression as a separate rule because I
just thought 'git archive' supported only gzip for compression. I admit
the unneeded *.tar file is annoying.
For other compression algorithms, I could pipe the two commands:
$ git archive HEAD | xz > linux.tar.xz
I read git-archive(1) carefully, and I realized GIT had provided a
more elegant way:
$ git -c tar.tar.xz.command=xz archive -o linux.tar.xz HEAD
This commit uses 'tar.tar.*.command' configuration to specify the
compression backend so we can create a compressed tarball directly.
GIT commit 767cf4579f0e ("archive: implement configurable tar filters")
is more than a decade old, so it should be available on almost all build
environments.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
---
scripts/Makefile.package | 24 +++++++++++++++++-------
1 file changed, 17 insertions(+), 7 deletions(-)
Comments
On Fri, Apr 07, 2023 at 07:16:28PM +0900, Masahiro Yamada wrote:
> Since commit 05e96e96a315 ("kbuild: use git-archive for source package
> creation"), source tarballs are created in two steps; create *.tar file
> then compress it. I split the compression as a separate rule because I
> just thought 'git archive' supported only gzip for compression. I admit
> the unneeded *.tar file is annoying.
>
> For other compression algorithms, I could pipe the two commands:
>
> $ git archive HEAD | xz > linux.tar.xz
>
> I read git-archive(1) carefully, and I realized GIT had provided a
> more elegant way:
Hooray for documentation :)
> $ git -c tar.tar.xz.command=xz archive -o linux.tar.xz HEAD
>
> This commit uses 'tar.tar.*.command' configuration to specify the
> compression backend so we can create a compressed tarball directly.
>
> GIT commit 767cf4579f0e ("archive: implement configurable tar filters")
> is more than a decade old, so it should be available on almost all build
> environments.
git 1.7.7 it seems, certainly ancientware in my opinion. If people have
issues with this, they can just upgrade git; even RHEL7 has git 1.8.x.
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Nathan Chancellor <nathan@kernel.org>
> ---
>
> scripts/Makefile.package | 24 +++++++++++++++++-------
> 1 file changed, 17 insertions(+), 7 deletions(-)
>
> diff --git a/scripts/Makefile.package b/scripts/Makefile.package
> index a205617730c6..7707975f729b 100644
> --- a/scripts/Makefile.package
> +++ b/scripts/Makefile.package
> @@ -57,16 +57,23 @@ check-git:
> false; \
> fi
>
> +archive-config-tar.gz = -c tar.tar.gz.command="$(KGZIP)"
> +archive-config-tar.bz2 = -c tar.tar.bz2.command="$(KBZIP2)"
> +archive-config-tar.xz = -c tar.tar.xz.command="$(XZ)"
> +archive-config-tar.zst = -c tar.tar.zst.command="$(ZSTD)"
> +
> quiet_cmd_archive = ARCHIVE $@
> - cmd_archive = git -C $(srctree) archive \
> + cmd_archive = git -C $(srctree) $(archive-config-tar$(suffix $@)) archive \
> --output=$$(realpath $@) --prefix=$(basename $@)/ $(archive-args)
>
> # Linux source tarball
> # ---------------------------------------------------------------------------
>
> -targets += linux.tar
> -linux.tar: archive-args = $$(cat $<)
> -linux.tar: .tmp_HEAD FORCE
> +linux-tarballs := $(addprefix linux, .tar.gz)
Is there any reason not to allow other compression formats for linux
like you do for perf?
> +
> +targets += $(linux-tarballs)
> +$(linux-tarballs): archive-args = $$(cat $<)
> +$(linux-tarballs): .tmp_HEAD FORCE
> $(call if_changed,archive)
>
> # rpm-pkg
> @@ -185,9 +192,12 @@ perf-archive-args = --add-file=$$(realpath $(word 2, $^)) \
> --add-file=$$(realpath $(word 3, $^)) \
> $$(cat $(word 2, $^))^{tree} $$(cat $<)
>
> -targets += perf-$(KERNELVERSION).tar
> -perf-$(KERNELVERSION).tar: archive-args = $(perf-archive-args)
> -perf-$(KERNELVERSION).tar: tools/perf/MANIFEST .tmp_perf/HEAD .tmp_perf/PERF-VERSION-FILE FORCE
> +
> +perf-tarballs := $(addprefix perf-$(KERNELVERSION), .tar .tar.gz .tar.bz2 .tar.xz .tar.zst)
> +
> +targets += $(perf-tarballs)
> +$(perf-tarballs): archive-args = $(perf-archive-args)
> +$(perf-tarballs): tools/perf/MANIFEST .tmp_perf/HEAD .tmp_perf/PERF-VERSION-FILE FORCE
> $(call if_changed,archive)
>
> PHONY += perf-tar-src-pkg
> --
> 2.37.2
>
On Sat, Apr 8, 2023 at 3:11 AM Nathan Chancellor <nathan@kernel.org> wrote:
>
> On Fri, Apr 07, 2023 at 07:16:28PM +0900, Masahiro Yamada wrote:
> > Since commit 05e96e96a315 ("kbuild: use git-archive for source package
> > creation"), source tarballs are created in two steps; create *.tar file
> > then compress it. I split the compression as a separate rule because I
> > just thought 'git archive' supported only gzip for compression. I admit
> > the unneeded *.tar file is annoying.
> >
> > For other compression algorithms, I could pipe the two commands:
> >
> > $ git archive HEAD | xz > linux.tar.xz
> >
> > I read git-archive(1) carefully, and I realized GIT had provided a
> > more elegant way:
>
> Hooray for documentation :)
>
> > $ git -c tar.tar.xz.command=xz archive -o linux.tar.xz HEAD
> >
> > This commit uses 'tar.tar.*.command' configuration to specify the
> > compression backend so we can create a compressed tarball directly.
> >
> > GIT commit 767cf4579f0e ("archive: implement configurable tar filters")
> > is more than a decade old, so it should be available on almost all build
> > environments.
>
> git 1.7.7 it seems, certainly ancientware in my opinion. If people have
> issues with this, they can just upgrade git; even RHEL7 has git 1.8.x.
>
> > Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
>
> Reviewed-by: Nathan Chancellor <nathan@kernel.org>
>
> > ---
> >
> > scripts/Makefile.package | 24 +++++++++++++++++-------
> > 1 file changed, 17 insertions(+), 7 deletions(-)
> >
> > diff --git a/scripts/Makefile.package b/scripts/Makefile.package
> > index a205617730c6..7707975f729b 100644
> > --- a/scripts/Makefile.package
> > +++ b/scripts/Makefile.package
> > @@ -57,16 +57,23 @@ check-git:
> > false; \
> > fi
> >
> > +archive-config-tar.gz = -c tar.tar.gz.command="$(KGZIP)"
> > +archive-config-tar.bz2 = -c tar.tar.bz2.command="$(KBZIP2)"
> > +archive-config-tar.xz = -c tar.tar.xz.command="$(XZ)"
> > +archive-config-tar.zst = -c tar.tar.zst.command="$(ZSTD)"
> > +
> > quiet_cmd_archive = ARCHIVE $@
> > - cmd_archive = git -C $(srctree) archive \
> > + cmd_archive = git -C $(srctree) $(archive-config-tar$(suffix $@)) archive \
> > --output=$$(realpath $@) --prefix=$(basename $@)/ $(archive-args)
> >
> > # Linux source tarball
> > # ---------------------------------------------------------------------------
> >
> > -targets += linux.tar
> > -linux.tar: archive-args = $$(cat $<)
> > -linux.tar: .tmp_HEAD FORCE
> > +linux-tarballs := $(addprefix linux, .tar.gz)
>
> Is there any reason not to allow other compression formats for linux
> like you do for perf?
Currently, gzip is only allowed because there is no way to specify
the compression algorithm.
I already have a patch locally, but not submitted yet.
I prioritize groundwork, then add new features later.
@@ -57,16 +57,23 @@ check-git:
false; \
fi
+archive-config-tar.gz = -c tar.tar.gz.command="$(KGZIP)"
+archive-config-tar.bz2 = -c tar.tar.bz2.command="$(KBZIP2)"
+archive-config-tar.xz = -c tar.tar.xz.command="$(XZ)"
+archive-config-tar.zst = -c tar.tar.zst.command="$(ZSTD)"
+
quiet_cmd_archive = ARCHIVE $@
- cmd_archive = git -C $(srctree) archive \
+ cmd_archive = git -C $(srctree) $(archive-config-tar$(suffix $@)) archive \
--output=$$(realpath $@) --prefix=$(basename $@)/ $(archive-args)
# Linux source tarball
# ---------------------------------------------------------------------------
-targets += linux.tar
-linux.tar: archive-args = $$(cat $<)
-linux.tar: .tmp_HEAD FORCE
+linux-tarballs := $(addprefix linux, .tar.gz)
+
+targets += $(linux-tarballs)
+$(linux-tarballs): archive-args = $$(cat $<)
+$(linux-tarballs): .tmp_HEAD FORCE
$(call if_changed,archive)
# rpm-pkg
@@ -185,9 +192,12 @@ perf-archive-args = --add-file=$$(realpath $(word 2, $^)) \
--add-file=$$(realpath $(word 3, $^)) \
$$(cat $(word 2, $^))^{tree} $$(cat $<)
-targets += perf-$(KERNELVERSION).tar
-perf-$(KERNELVERSION).tar: archive-args = $(perf-archive-args)
-perf-$(KERNELVERSION).tar: tools/perf/MANIFEST .tmp_perf/HEAD .tmp_perf/PERF-VERSION-FILE FORCE
+
+perf-tarballs := $(addprefix perf-$(KERNELVERSION), .tar .tar.gz .tar.bz2 .tar.xz .tar.zst)
+
+targets += $(perf-tarballs)
+$(perf-tarballs): archive-args = $(perf-archive-args)
+$(perf-tarballs): tools/perf/MANIFEST .tmp_perf/HEAD .tmp_perf/PERF-VERSION-FILE FORCE
$(call if_changed,archive)
PHONY += perf-tar-src-pkg