| Message ID | 20231017103742.130927-2-masahiroy@kernel.org |
|---|---|
| State | New |
| Headers |
Return-Path: <linux-kernel-owner@vger.kernel.org>
Delivered-To: ouuuleilei@gmail.com
Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id
ib8csp4039938vqb;
Tue, 17 Oct 2023 03:38:35 -0700 (PDT)
X-Google-Smtp-Source:
AGHT+IGjui678Z6i3iMzW1/2+B0KCX6ORumoVta2HXjOGAODDaYE0VZnLkjLUziS2mjqDmmS8Fhe
X-Received: by 2002:a05:6359:5e20:b0:166:d93e:339 with SMTP id
pw32-20020a0563595e2000b00166d93e0339mr1791408rwb.0.1697539114961;
Tue, 17 Oct 2023 03:38:34 -0700 (PDT)
ARC-Seal: i=1; a=rsa-sha256; t=1697539114; cv=none;
d=google.com; s=arc-20160816;
b=uRrTd5M8Del/NuyUmr3SyNQz8R3IgFPNz/EegZMraZGf0B/TxJVuQqNsL8FEg8MuLw
26d/Qf5hqIS93qaQTPRBab6hX60S7CT6XEGQjEc9bxVb7U9ry/4R8Tgtc/kqZSszDmcF
Amw6Qhrfo//WDgH2p14uP+uGUy/mzWQhGSt+lc3loDtFuP5krLpHmJQucgevWxzHX50u
xdbmcDYQglE0Eb9eA8qpMrmJLRslW9jfjvayRjR9lOJFelto2E0UafMJ4lLdh988Os+p
ovee+S/B1Ps7M1COkEZ8Ps0tvfWqLq7nXuMilISqISWTS/nQnFEBGnWFT5iiw6bYI8uJ
okug==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com;
s=arc-20160816;
h=list-id:precedence:content-transfer-encoding:mime-version
:references:in-reply-to:message-id:date:subject:cc:to:from
:dkim-signature;
bh=ajzPYkPzS+JKnfhwdhQyXEGy3tVX8Us7yT8vlF3e3qc=;
fh=fDkx6bVHh3l/rA0E4892gjuP6gNEs3Z88QlOHrZLqpA=;
b=auctRIuwrW6/dKL21n7W++EsBF6VvxlDDGzKQOUv2dObg5oqSJiHZFwE+nW8v0Ru83
i64vEVNcdlrdVmtoB7vCKuOZ/HZUeNyvYHp8cnIoLMZFKvCJCXhkwetMe7JuLEHTXB5J
Z/CtR/Q4UhGEbsJF4C6MAFBSEhpR9BQVU2IZ6jOTfbHHZmwZ21NqLW8Qw/rt7JEC47D2
SqG90nw8twyEin6BOQV4qefGEao9iJ6XN8g84ILwNvrx4ZcsW+kM5J8mzaLpJnsUFCv/
AiENrHMCZ3od/dZMFguYWYJrJYdFsUrA8X1ofxrZyztR3Y0hyvr7nZNtDVaQ3DoOSf6R
oJNQ==
ARC-Authentication-Results: i=1; mx.google.com;
dkim=pass header.i=@kernel.org header.s=k20201202 header.b=k2ytQAQi;
spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org
designates 2620:137:e000::3:2 as permitted sender)
smtp.mailfrom=linux-kernel-owner@vger.kernel.org;
dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org
Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2])
by mx.google.com with ESMTPS id
u191-20020a6385c8000000b005b2d044af30si1421947pgd.480.2023.10.17.03.38.34
(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
Tue, 17 Oct 2023 03:38:34 -0700 (PDT)
Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org
designates 2620:137:e000::3:2 as permitted sender)
client-ip=2620:137:e000::3:2;
Authentication-Results: mx.google.com;
dkim=pass header.i=@kernel.org header.s=k20201202 header.b=k2ytQAQi;
spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org
designates 2620:137:e000::3:2 as permitted sender)
smtp.mailfrom=linux-kernel-owner@vger.kernel.org;
dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org
Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0])
by agentk.vger.email (Postfix) with ESMTP id A18C78036846;
Tue, 17 Oct 2023 03:38:30 -0700 (PDT)
X-Virus-Status: Clean
X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
id S1343652AbjJQKiS (ORCPT <rfc822;hjfbswb@gmail.com> + 19 others);
Tue, 17 Oct 2023 06:38:18 -0400
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54068 "EHLO
lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
with ESMTP id S234594AbjJQKiQ (ORCPT
<rfc822;linux-kernel@vger.kernel.org>);
Tue, 17 Oct 2023 06:38:16 -0400
Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140])
by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1F6EBA2;
Tue, 17 Oct 2023 03:38:15 -0700 (PDT)
Received: by smtp.kernel.org (Postfix) with ESMTPSA id 71C4EC433CA;
Tue, 17 Oct 2023 10:38:08 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org;
s=k20201202; t=1697539094;
bh=A6qLcCYxkwgWjCL+CxEKfQPWGv0e7JNIKXIZW89MeuU=;
h=From:To:Cc:Subject:Date:In-Reply-To:References:From;
b=k2ytQAQi1GJKjYWyS2S00/+UNBIvCCyZOAFkCE9TFt+3ZueDC4OLAjybipTBO++p7
lSznrtrnBtPgpvlmBNt6OM2fAUeOwQxj4dQNjqNvWKTX+DMCwS4qNiSkTiXcgF7iXX
IVWyV/5vgO1OM9YzlILfH8VzLtz/pCA9hqXZrvwDjhxZ+F6gUUJP9/OIEhYH1ElMue
dFHIhx2GPCHJTtAQeuqceZ02h0gP8+H7GgB8vFa1SINkFD1W4N0B3HEMOSRPXrmsYA
6ACuJiALmbS0dKkNEiRZKjV1i4jhjjf/Y7/5bJXdH8bcAESPGvcIcVX73YsDJgg5gO
Wsuoqa5OLtQBQ==
From: Masahiro Yamada <masahiroy@kernel.org>
To: linux-kbuild@vger.kernel.org
Cc: linux-kernel@vger.kernel.org, Masahiro Yamada <masahiroy@kernel.org>,
Alex Gaynor <alex.gaynor@gmail.com>, Alexei Starovoitov <ast@kernel.org>,
Alice Ryhl <aliceryhl@google.com>, Andreas Hindborg <a.hindborg@samsung.com>,
Andrii Nakryiko <andrii@kernel.org>, Benno Lossin <benno.lossin@proton.me>,
=?utf-8?q?Bj=C3=B6rn_Roy_Baron?= <bjorn3_gh@protonmail.com>,
Boqun Feng <boqun.feng@gmail.com>, Daniel Borkmann <daniel@iogearbox.net>,
Gary Guo <gary@garyguo.net>, Hao Luo <haoluo@google.com>,
Jiri Olsa <jolsa@kernel.org>, John Fastabend <john.fastabend@gmail.com>,
KP Singh <kpsingh@kernel.org>, Martin KaFai Lau <martin.lau@linux.dev>,
Miguel Ojeda <ojeda@kernel.org>, Nathan Chancellor <nathan@kernel.org>,
Nick Desaulniers <ndesaulniers@google.com>,
Nicolas Schier <nicolas@fjasle.eu>, Song Liu <song@kernel.org>,
Stanislav Fomichev <sdf@google.com>,
Wedson Almeida Filho <wedsonaf@gmail.com>,
Yonghong Song <yonghong.song@linux.dev>, bpf@vger.kernel.org,
rust-for-linux@vger.kernel.org
Subject: [PATCH 2/4] kbuild: avoid too many execution of
scripts/pahole-flags.sh
Date: Tue, 17 Oct 2023 19:37:40 +0900
Message-Id: <20231017103742.130927-2-masahiroy@kernel.org>
X-Mailer: git-send-email 2.40.1
In-Reply-To: <20231017103742.130927-1-masahiroy@kernel.org>
References: <20231017103742.130927-1-masahiroy@kernel.org>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI,
SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no
version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test,
not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]);
Tue, 17 Oct 2023 03:38:30 -0700 (PDT)
X-getmail-retrieved-from-mailbox: INBOX
X-GMAIL-THRID: 1779998775013847069
X-GMAIL-MSGID: 1779998775013847069
|
| Series |
[1/4] kbuild: remove ARCH_POSTLINK from module builds
|
|
Commit Message
Masahiro Yamada
Oct. 17, 2023, 10:37 a.m. UTC
scripts/pahole-flags.sh is executed so many times. You can check how many times it is invoked during the build, as follows: $ cat <<EOF >> scripts/pahole-flags.sh > echo "scripts/pahole-flags.sh was executed" >&2 > EOF $ make -s scripts/pahole-flags.sh was executed scripts/pahole-flags.sh was executed scripts/pahole-flags.sh was executed scripts/pahole-flags.sh was executed scripts/pahole-flags.sh was executed [ lots of repeated lines suppressed... ] This scripts is exectuted more than 20 times during the kernel build because PAHOLE_FLAGS is a recursively expanded variable and exported to sub-processes. With the GNU Make >= 4.4, it is executed more than 60 times because exported variables are also passed to other $(shell ) invocations. Without careful coding, it is known to cause an exponential fork explosion. [1] The use of $(shell ) in an exported recursive variable is likely wrong because $(shell ) is always evaluated due to the 'export' keyword, and the evaluation can occur multiple times by the nature of recursive variables. Convert the shell script to a Makefile, which is included only when CONFIG_DEBUG_INFO_BTF=y. [1]: https://savannah.gnu.org/bugs/index.php?64746 Signed-off-by: Masahiro Yamada <masahiroy@kernel.org> --- Makefile | 4 +--- scripts/Makefile.btf | 19 +++++++++++++++++++ scripts/pahole-flags.sh | 30 ------------------------------ 3 files changed, 20 insertions(+), 33 deletions(-) create mode 100644 scripts/Makefile.btf delete mode 100755 scripts/pahole-flags.sh
Comments
On 17/10/2023 11:37, Masahiro Yamada wrote: > scripts/pahole-flags.sh is executed so many times. > > You can check how many times it is invoked during the build, as follows: > > $ cat <<EOF >> scripts/pahole-flags.sh > > echo "scripts/pahole-flags.sh was executed" >&2 > > EOF > > $ make -s > scripts/pahole-flags.sh was executed > scripts/pahole-flags.sh was executed > scripts/pahole-flags.sh was executed > scripts/pahole-flags.sh was executed > scripts/pahole-flags.sh was executed > [ lots of repeated lines suppressed... ] > > This scripts is exectuted more than 20 times during the kernel build > because PAHOLE_FLAGS is a recursively expanded variable and exported > to sub-processes. > > With the GNU Make >= 4.4, it is executed more than 60 times because > exported variables are also passed to other $(shell ) invocations. > Without careful coding, it is known to cause an exponential fork > explosion. [1] > > The use of $(shell ) in an exported recursive variable is likely wrong > because $(shell ) is always evaluated due to the 'export' keyword, and > the evaluation can occur multiple times by the nature of recursive > variables. > > Convert the shell script to a Makefile, which is included only when > CONFIG_DEBUG_INFO_BTF=y. > > [1]: https://savannah.gnu.org/bugs/index.php?64746 > > Signed-off-by: Masahiro Yamada <masahiroy@kernel.org> Looks like a neat change to me. I tested with CONFIG_DEBUG_INFO_BTF=y builds for full kernel build and 'make M=modpath' module-only builds, and both work well with correct pahole flags. Feel free to add either/both of Reviewed-by: Alan Maguire <alan.maguire@oracle.com> Tested-by: Alan Maguire <alan.maguire@oracle.com> > --- > > Makefile | 4 +--- > scripts/Makefile.btf | 19 +++++++++++++++++++ > scripts/pahole-flags.sh | 30 ------------------------------ > 3 files changed, 20 insertions(+), 33 deletions(-) > create mode 100644 scripts/Makefile.btf > delete mode 100755 scripts/pahole-flags.sh > > diff --git a/Makefile b/Makefile > index fed9a6cc3665..eaddec67e5e1 100644 > --- a/Makefile > +++ b/Makefile > @@ -513,8 +513,6 @@ LZ4 = lz4c > XZ = xz > ZSTD = zstd > > -PAHOLE_FLAGS = $(shell PAHOLE=$(PAHOLE) $(srctree)/scripts/pahole-flags.sh) > - > CHECKFLAGS := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ \ > -Wbitwise -Wno-return-void -Wno-unknown-attribute $(CF) > NOSTDINC_FLAGS := > @@ -605,7 +603,6 @@ export KBUILD_RUSTFLAGS RUSTFLAGS_KERNEL RUSTFLAGS_MODULE > export KBUILD_AFLAGS AFLAGS_KERNEL AFLAGS_MODULE > export KBUILD_AFLAGS_MODULE KBUILD_CFLAGS_MODULE KBUILD_RUSTFLAGS_MODULE KBUILD_LDFLAGS_MODULE > export KBUILD_AFLAGS_KERNEL KBUILD_CFLAGS_KERNEL KBUILD_RUSTFLAGS_KERNEL > -export PAHOLE_FLAGS > > # Files to ignore in find ... statements > > @@ -1002,6 +999,7 @@ KBUILD_CPPFLAGS += $(call cc-option,-fmacro-prefix-map=$(srctree)/=) > # include additional Makefiles when needed > include-y := scripts/Makefile.extrawarn > include-$(CONFIG_DEBUG_INFO) += scripts/Makefile.debug > +include-$(CONFIG_DEBUG_INFO_BTF)+= scripts/Makefile.btf > include-$(CONFIG_KASAN) += scripts/Makefile.kasan > include-$(CONFIG_KCSAN) += scripts/Makefile.kcsan > include-$(CONFIG_KMSAN) += scripts/Makefile.kmsan > diff --git a/scripts/Makefile.btf b/scripts/Makefile.btf > new file mode 100644 > index 000000000000..82377e470aed > --- /dev/null > +++ b/scripts/Makefile.btf > @@ -0,0 +1,19 @@ > +# SPDX-License-Identifier: GPL-2.0 > + > +pahole-ver := $(CONFIG_PAHOLE_VERSION) > +pahole-flags-y := > + > +# pahole 1.18 through 1.21 can't handle zero-sized per-CPU vars > +ifeq ($(call test-le, $(pahole-ver), 121),y) > +pahole-flags-$(call test-ge, $(pahole-ver), 118) += --skip_encoding_btf_vars > +endif > + > +pahole-flags-$(call test-ge, $(pahole-ver), 121) += --btf_gen_floats > + > +pahole-flags-$(call test-ge, $(pahole-ver), 122) += -j > + > +pahole-flags-$(CONFIG_PAHOLE_HAS_LANG_EXCLUDE) += --lang_exclude=rust > + > +pahole-flags-$(call test-ge, $(pahole-ver), 125) += --skip_encoding_btf_inconsistent_proto --btf_gen_optimized > + > +export PAHOLE_FLAGS := $(pahole-flags-y) > diff --git a/scripts/pahole-flags.sh b/scripts/pahole-flags.sh > deleted file mode 100755 > index 728d55190d97..000000000000 > --- a/scripts/pahole-flags.sh > +++ /dev/null > @@ -1,30 +0,0 @@ > -#!/bin/sh > -# SPDX-License-Identifier: GPL-2.0 > - > -extra_paholeopt= > - > -if ! [ -x "$(command -v ${PAHOLE})" ]; then > - exit 0 > -fi > - > -pahole_ver=$($(dirname $0)/pahole-version.sh ${PAHOLE}) > - > -if [ "${pahole_ver}" -ge "118" ] && [ "${pahole_ver}" -le "121" ]; then > - # pahole 1.18 through 1.21 can't handle zero-sized per-CPU vars > - extra_paholeopt="${extra_paholeopt} --skip_encoding_btf_vars" > -fi > -if [ "${pahole_ver}" -ge "121" ]; then > - extra_paholeopt="${extra_paholeopt} --btf_gen_floats" > -fi > -if [ "${pahole_ver}" -ge "122" ]; then > - extra_paholeopt="${extra_paholeopt} -j" > -fi > -if [ "${pahole_ver}" -ge "124" ]; then > - # see PAHOLE_HAS_LANG_EXCLUDE > - extra_paholeopt="${extra_paholeopt} --lang_exclude=rust" > -fi > -if [ "${pahole_ver}" -ge "125" ]; then > - extra_paholeopt="${extra_paholeopt} --skip_encoding_btf_inconsistent_proto --btf_gen_optimized" > -fi > - > -echo ${extra_paholeopt}
On Tue, Oct 17, 2023 at 07:37:40PM +0900, Masahiro Yamada wrote: > scripts/pahole-flags.sh is executed so many times. > > You can check how many times it is invoked during the build, as follows: > > $ cat <<EOF >> scripts/pahole-flags.sh > > echo "scripts/pahole-flags.sh was executed" >&2 > > EOF > > $ make -s > scripts/pahole-flags.sh was executed > scripts/pahole-flags.sh was executed > scripts/pahole-flags.sh was executed > scripts/pahole-flags.sh was executed > scripts/pahole-flags.sh was executed > [ lots of repeated lines suppressed... ] > > This scripts is exectuted more than 20 times during the kernel build > because PAHOLE_FLAGS is a recursively expanded variable and exported > to sub-processes. > > With the GNU Make >= 4.4, it is executed more than 60 times because > exported variables are also passed to other $(shell ) invocations. > Without careful coding, it is known to cause an exponential fork > explosion. [1] nice :-\ > > The use of $(shell ) in an exported recursive variable is likely wrong > because $(shell ) is always evaluated due to the 'export' keyword, and > the evaluation can occur multiple times by the nature of recursive > variables. > > Convert the shell script to a Makefile, which is included only when > CONFIG_DEBUG_INFO_BTF=y. looks good.. could you please resend this patch with bpf-next in subject so CI tests would trigger for it? thanks, jirka > > [1]: https://savannah.gnu.org/bugs/index.php?64746 > > Signed-off-by: Masahiro Yamada <masahiroy@kernel.org> > --- > > Makefile | 4 +--- > scripts/Makefile.btf | 19 +++++++++++++++++++ > scripts/pahole-flags.sh | 30 ------------------------------ > 3 files changed, 20 insertions(+), 33 deletions(-) > create mode 100644 scripts/Makefile.btf > delete mode 100755 scripts/pahole-flags.sh > > diff --git a/Makefile b/Makefile > index fed9a6cc3665..eaddec67e5e1 100644 > --- a/Makefile > +++ b/Makefile > @@ -513,8 +513,6 @@ LZ4 = lz4c > XZ = xz > ZSTD = zstd > > -PAHOLE_FLAGS = $(shell PAHOLE=$(PAHOLE) $(srctree)/scripts/pahole-flags.sh) > - > CHECKFLAGS := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ \ > -Wbitwise -Wno-return-void -Wno-unknown-attribute $(CF) > NOSTDINC_FLAGS := > @@ -605,7 +603,6 @@ export KBUILD_RUSTFLAGS RUSTFLAGS_KERNEL RUSTFLAGS_MODULE > export KBUILD_AFLAGS AFLAGS_KERNEL AFLAGS_MODULE > export KBUILD_AFLAGS_MODULE KBUILD_CFLAGS_MODULE KBUILD_RUSTFLAGS_MODULE KBUILD_LDFLAGS_MODULE > export KBUILD_AFLAGS_KERNEL KBUILD_CFLAGS_KERNEL KBUILD_RUSTFLAGS_KERNEL > -export PAHOLE_FLAGS > > # Files to ignore in find ... statements > > @@ -1002,6 +999,7 @@ KBUILD_CPPFLAGS += $(call cc-option,-fmacro-prefix-map=$(srctree)/=) > # include additional Makefiles when needed > include-y := scripts/Makefile.extrawarn > include-$(CONFIG_DEBUG_INFO) += scripts/Makefile.debug > +include-$(CONFIG_DEBUG_INFO_BTF)+= scripts/Makefile.btf > include-$(CONFIG_KASAN) += scripts/Makefile.kasan > include-$(CONFIG_KCSAN) += scripts/Makefile.kcsan > include-$(CONFIG_KMSAN) += scripts/Makefile.kmsan > diff --git a/scripts/Makefile.btf b/scripts/Makefile.btf > new file mode 100644 > index 000000000000..82377e470aed > --- /dev/null > +++ b/scripts/Makefile.btf > @@ -0,0 +1,19 @@ > +# SPDX-License-Identifier: GPL-2.0 > + > +pahole-ver := $(CONFIG_PAHOLE_VERSION) > +pahole-flags-y := > + > +# pahole 1.18 through 1.21 can't handle zero-sized per-CPU vars > +ifeq ($(call test-le, $(pahole-ver), 121),y) > +pahole-flags-$(call test-ge, $(pahole-ver), 118) += --skip_encoding_btf_vars > +endif > + > +pahole-flags-$(call test-ge, $(pahole-ver), 121) += --btf_gen_floats > + > +pahole-flags-$(call test-ge, $(pahole-ver), 122) += -j > + > +pahole-flags-$(CONFIG_PAHOLE_HAS_LANG_EXCLUDE) += --lang_exclude=rust > + > +pahole-flags-$(call test-ge, $(pahole-ver), 125) += --skip_encoding_btf_inconsistent_proto --btf_gen_optimized > + > +export PAHOLE_FLAGS := $(pahole-flags-y) > diff --git a/scripts/pahole-flags.sh b/scripts/pahole-flags.sh > deleted file mode 100755 > index 728d55190d97..000000000000 > --- a/scripts/pahole-flags.sh > +++ /dev/null > @@ -1,30 +0,0 @@ > -#!/bin/sh > -# SPDX-License-Identifier: GPL-2.0 > - > -extra_paholeopt= > - > -if ! [ -x "$(command -v ${PAHOLE})" ]; then > - exit 0 > -fi > - > -pahole_ver=$($(dirname $0)/pahole-version.sh ${PAHOLE}) > - > -if [ "${pahole_ver}" -ge "118" ] && [ "${pahole_ver}" -le "121" ]; then > - # pahole 1.18 through 1.21 can't handle zero-sized per-CPU vars > - extra_paholeopt="${extra_paholeopt} --skip_encoding_btf_vars" > -fi > -if [ "${pahole_ver}" -ge "121" ]; then > - extra_paholeopt="${extra_paholeopt} --btf_gen_floats" > -fi > -if [ "${pahole_ver}" -ge "122" ]; then > - extra_paholeopt="${extra_paholeopt} -j" > -fi > -if [ "${pahole_ver}" -ge "124" ]; then > - # see PAHOLE_HAS_LANG_EXCLUDE > - extra_paholeopt="${extra_paholeopt} --lang_exclude=rust" > -fi > -if [ "${pahole_ver}" -ge "125" ]; then > - extra_paholeopt="${extra_paholeopt} --skip_encoding_btf_inconsistent_proto --btf_gen_optimized" > -fi > - > -echo ${extra_paholeopt} > -- > 2.40.1 >
> Subject: kbuild: avoid too many execution of scripts/pahole-flags.sh executions? On Tue, Oct 17, 2023 at 07:37:40PM +0900, Masahiro Yamada wrote: > scripts/pahole-flags.sh is executed so many times. > > You can check how many times it is invoked during the build, as follows: > > $ cat <<EOF >> scripts/pahole-flags.sh > > echo "scripts/pahole-flags.sh was executed" >&2 > > EOF > > $ make -s > scripts/pahole-flags.sh was executed > scripts/pahole-flags.sh was executed > scripts/pahole-flags.sh was executed > scripts/pahole-flags.sh was executed > scripts/pahole-flags.sh was executed > [ lots of repeated lines suppressed... ] > > This scripts is exectuted more than 20 times during the kernel build executed > because PAHOLE_FLAGS is a recursively expanded variable and exported > to sub-processes. > > With the GNU Make >= 4.4, it is executed more than 60 times because > exported variables are also passed to other $(shell ) invocations. > Without careful coding, it is known to cause an exponential fork > explosion. [1] > > The use of $(shell ) in an exported recursive variable is likely wrong > because $(shell ) is always evaluated due to the 'export' keyword, and > the evaluation can occur multiple times by the nature of recursive > variables. > > Convert the shell script to a Makefile, which is included only when > CONFIG_DEBUG_INFO_BTF=y. > > [1]: https://savannah.gnu.org/bugs/index.php?64746 > > Signed-off-by: Masahiro Yamada <masahiroy@kernel.org> > --- Thanks, looks good to me! Reviewed-by: Nicolas Schier <n.schier@avm.de>
On Tue, Oct 17, 2023 at 12:38 PM Masahiro Yamada <masahiroy@kernel.org> wrote: > > Convert the shell script to a Makefile, which is included only when > CONFIG_DEBUG_INFO_BTF=y. > > [1]: https://savannah.gnu.org/bugs/index.php?64746 > > Signed-off-by: Masahiro Yamada <masahiroy@kernel.org> The field in `MAINTAINERS` should be removed: F: scripts/pahole-flags.sh But other than that, it looks good to me! I tried it for a given config and it does call `pahole` with the same flags. Tested-by: Miguel Ojeda <ojeda@kernel.org> Acked-by: Miguel Ojeda <ojeda@kernel.org> Cheers, Miguel
On Tue, Oct 17, 2023 at 3:38 AM Masahiro Yamada <masahiroy@kernel.org> wrote: > > scripts/pahole-flags.sh is executed so many times. > > You can check how many times it is invoked during the build, as follows: > > $ cat <<EOF >> scripts/pahole-flags.sh > > echo "scripts/pahole-flags.sh was executed" >&2 > > EOF > > $ make -s > scripts/pahole-flags.sh was executed > scripts/pahole-flags.sh was executed > scripts/pahole-flags.sh was executed > scripts/pahole-flags.sh was executed > scripts/pahole-flags.sh was executed > [ lots of repeated lines suppressed... ] > > This scripts is exectuted more than 20 times during the kernel build > because PAHOLE_FLAGS is a recursively expanded variable and exported > to sub-processes. > > With the GNU Make >= 4.4, it is executed more than 60 times because > exported variables are also passed to other $(shell ) invocations. > Without careful coding, it is known to cause an exponential fork > explosion. [1] > > The use of $(shell ) in an exported recursive variable is likely wrong > because $(shell ) is always evaluated due to the 'export' keyword, and > the evaluation can occur multiple times by the nature of recursive > variables. > > Convert the shell script to a Makefile, which is included only when > CONFIG_DEBUG_INFO_BTF=y. > > [1]: https://savannah.gnu.org/bugs/index.php?64746 > > Signed-off-by: Masahiro Yamada <masahiroy@kernel.org> > --- > > Makefile | 4 +--- > scripts/Makefile.btf | 19 +++++++++++++++++++ > scripts/pahole-flags.sh | 30 ------------------------------ > 3 files changed, 20 insertions(+), 33 deletions(-) > create mode 100644 scripts/Makefile.btf > delete mode 100755 scripts/pahole-flags.sh > > diff --git a/Makefile b/Makefile > index fed9a6cc3665..eaddec67e5e1 100644 > --- a/Makefile > +++ b/Makefile > @@ -513,8 +513,6 @@ LZ4 = lz4c > XZ = xz > ZSTD = zstd > > -PAHOLE_FLAGS = $(shell PAHOLE=$(PAHOLE) $(srctree)/scripts/pahole-flags.sh) What if we just used := here? Wouldn't it avoid unnecessary multiple executions? I don't make Makefile.btf approach, just curious why := doesn't work, if it doesn't. > - > CHECKFLAGS := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ \ > -Wbitwise -Wno-return-void -Wno-unknown-attribute $(CF) > NOSTDINC_FLAGS := > @@ -605,7 +603,6 @@ export KBUILD_RUSTFLAGS RUSTFLAGS_KERNEL RUSTFLAGS_MODULE > export KBUILD_AFLAGS AFLAGS_KERNEL AFLAGS_MODULE > export KBUILD_AFLAGS_MODULE KBUILD_CFLAGS_MODULE KBUILD_RUSTFLAGS_MODULE KBUILD_LDFLAGS_MODULE > export KBUILD_AFLAGS_KERNEL KBUILD_CFLAGS_KERNEL KBUILD_RUSTFLAGS_KERNEL > -export PAHOLE_FLAGS > > # Files to ignore in find ... statements > > @@ -1002,6 +999,7 @@ KBUILD_CPPFLAGS += $(call cc-option,-fmacro-prefix-map=$(srctree)/=) > # include additional Makefiles when needed > include-y := scripts/Makefile.extrawarn > include-$(CONFIG_DEBUG_INFO) += scripts/Makefile.debug > +include-$(CONFIG_DEBUG_INFO_BTF)+= scripts/Makefile.btf > include-$(CONFIG_KASAN) += scripts/Makefile.kasan > include-$(CONFIG_KCSAN) += scripts/Makefile.kcsan > include-$(CONFIG_KMSAN) += scripts/Makefile.kmsan > diff --git a/scripts/Makefile.btf b/scripts/Makefile.btf > new file mode 100644 > index 000000000000..82377e470aed > --- /dev/null > +++ b/scripts/Makefile.btf > @@ -0,0 +1,19 @@ > +# SPDX-License-Identifier: GPL-2.0 > + > +pahole-ver := $(CONFIG_PAHOLE_VERSION) > +pahole-flags-y := > + > +# pahole 1.18 through 1.21 can't handle zero-sized per-CPU vars > +ifeq ($(call test-le, $(pahole-ver), 121),y) > +pahole-flags-$(call test-ge, $(pahole-ver), 118) += --skip_encoding_btf_vars > +endif > + > +pahole-flags-$(call test-ge, $(pahole-ver), 121) += --btf_gen_floats > + > +pahole-flags-$(call test-ge, $(pahole-ver), 122) += -j > + > +pahole-flags-$(CONFIG_PAHOLE_HAS_LANG_EXCLUDE) += --lang_exclude=rust > + > +pahole-flags-$(call test-ge, $(pahole-ver), 125) += --skip_encoding_btf_inconsistent_proto --btf_gen_optimized > + > +export PAHOLE_FLAGS := $(pahole-flags-y) > diff --git a/scripts/pahole-flags.sh b/scripts/pahole-flags.sh > deleted file mode 100755 > index 728d55190d97..000000000000 > --- a/scripts/pahole-flags.sh > +++ /dev/null > @@ -1,30 +0,0 @@ > -#!/bin/sh > -# SPDX-License-Identifier: GPL-2.0 > - > -extra_paholeopt= > - > -if ! [ -x "$(command -v ${PAHOLE})" ]; then > - exit 0 > -fi > - > -pahole_ver=$($(dirname $0)/pahole-version.sh ${PAHOLE}) > - > -if [ "${pahole_ver}" -ge "118" ] && [ "${pahole_ver}" -le "121" ]; then > - # pahole 1.18 through 1.21 can't handle zero-sized per-CPU vars > - extra_paholeopt="${extra_paholeopt} --skip_encoding_btf_vars" > -fi > -if [ "${pahole_ver}" -ge "121" ]; then > - extra_paholeopt="${extra_paholeopt} --btf_gen_floats" > -fi > -if [ "${pahole_ver}" -ge "122" ]; then > - extra_paholeopt="${extra_paholeopt} -j" > -fi > -if [ "${pahole_ver}" -ge "124" ]; then > - # see PAHOLE_HAS_LANG_EXCLUDE > - extra_paholeopt="${extra_paholeopt} --lang_exclude=rust" > -fi > -if [ "${pahole_ver}" -ge "125" ]; then > - extra_paholeopt="${extra_paholeopt} --skip_encoding_btf_inconsistent_proto --btf_gen_optimized" > -fi > - > -echo ${extra_paholeopt} > -- > 2.40.1 >
On Wed, Oct 18, 2023 at 4:57 AM Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote: > > On Tue, Oct 17, 2023 at 3:38 AM Masahiro Yamada <masahiroy@kernel.org> wrote: > > > > scripts/pahole-flags.sh is executed so many times. > > > > You can check how many times it is invoked during the build, as follows: > > > > $ cat <<EOF >> scripts/pahole-flags.sh > > > echo "scripts/pahole-flags.sh was executed" >&2 > > > EOF > > > > $ make -s > > scripts/pahole-flags.sh was executed > > scripts/pahole-flags.sh was executed > > scripts/pahole-flags.sh was executed > > scripts/pahole-flags.sh was executed > > scripts/pahole-flags.sh was executed > > [ lots of repeated lines suppressed... ] > > > > This scripts is exectuted more than 20 times during the kernel build > > because PAHOLE_FLAGS is a recursively expanded variable and exported > > to sub-processes. > > > > With the GNU Make >= 4.4, it is executed more than 60 times because > > exported variables are also passed to other $(shell ) invocations. > > Without careful coding, it is known to cause an exponential fork > > explosion. [1] > > > > The use of $(shell ) in an exported recursive variable is likely wrong > > because $(shell ) is always evaluated due to the 'export' keyword, and > > the evaluation can occur multiple times by the nature of recursive > > variables. > > > > Convert the shell script to a Makefile, which is included only when > > CONFIG_DEBUG_INFO_BTF=y. > > > > [1]: https://savannah.gnu.org/bugs/index.php?64746 > > > > Signed-off-by: Masahiro Yamada <masahiroy@kernel.org> > > --- > > > > Makefile | 4 +--- > > scripts/Makefile.btf | 19 +++++++++++++++++++ > > scripts/pahole-flags.sh | 30 ------------------------------ > > 3 files changed, 20 insertions(+), 33 deletions(-) > > create mode 100644 scripts/Makefile.btf > > delete mode 100755 scripts/pahole-flags.sh > > > > diff --git a/Makefile b/Makefile > > index fed9a6cc3665..eaddec67e5e1 100644 > > --- a/Makefile > > +++ b/Makefile > > @@ -513,8 +513,6 @@ LZ4 = lz4c > > XZ = xz > > ZSTD = zstd > > > > -PAHOLE_FLAGS = $(shell PAHOLE=$(PAHOLE) $(srctree)/scripts/pahole-flags.sh) > > What if we just used := here? Wouldn't it avoid unnecessary multiple executions? Yeah, := is less silly than =. But, I do not like to run the script for non-build targets such as 'make clean', 'make help', etc. Also, when building with CONFIG_DEBUG_INFO_BTF=n, the shell is forked to compute PAHOLE_FLAGS, which we know are unnecessary. -- Best Regards Masahiro Yamada
On Wed, Oct 18, 2023 at 12:21 AM Miguel Ojeda <miguel.ojeda.sandonis@gmail.com> wrote: > > On Tue, Oct 17, 2023 at 12:38 PM Masahiro Yamada <masahiroy@kernel.org> wrote: > > > > Convert the shell script to a Makefile, which is included only when > > CONFIG_DEBUG_INFO_BTF=y. > > > > [1]: https://savannah.gnu.org/bugs/index.php?64746 > > > > Signed-off-by: Masahiro Yamada <masahiroy@kernel.org> > > The field in `MAINTAINERS` should be removed: > > F: scripts/pahole-flags.sh I will replace it with F: scripts/Makefile.btf Thanks. > > But other than that, it looks good to me! I tried it for a given > config and it does call `pahole` with the same flags. > > Tested-by: Miguel Ojeda <ojeda@kernel.org> > Acked-by: Miguel Ojeda <ojeda@kernel.org> > > Cheers, > Miguel
On 10/17/23 07:37, Masahiro Yamada wrote: > scripts/pahole-flags.sh is executed so many times. > > You can check how many times it is invoked during the build, as follows: > > $ cat <<EOF >> scripts/pahole-flags.sh > > echo "scripts/pahole-flags.sh was executed" >&2 > > EOF > > $ make -s > scripts/pahole-flags.sh was executed > scripts/pahole-flags.sh was executed > scripts/pahole-flags.sh was executed > scripts/pahole-flags.sh was executed > scripts/pahole-flags.sh was executed > [ lots of repeated lines suppressed... ] > > This scripts is exectuted more than 20 times during the kernel build > because PAHOLE_FLAGS is a recursively expanded variable and exported > to sub-processes. > > With the GNU Make >= 4.4, it is executed more than 60 times because > exported variables are also passed to other $(shell ) invocations. > Without careful coding, it is known to cause an exponential fork > explosion. [1] > > The use of $(shell ) in an exported recursive variable is likely wrong > because $(shell ) is always evaluated due to the 'export' keyword, and > the evaluation can occur multiple times by the nature of recursive > variables. > > Convert the shell script to a Makefile, which is included only when > CONFIG_DEBUG_INFO_BTF=y. > > [1]: https://savannah.gnu.org/bugs/index.php?64746 > > Signed-off-by: Masahiro Yamada <masahiroy@kernel.org> > --- > [...] > +include-$(CONFIG_DEBUG_INFO_BTF)+= scripts/Makefile.btf Would have used a tab. > [...] Reviewed-by: Martin Rodriguez Reboredo <yakoyoku@gmail.com>
diff --git a/Makefile b/Makefile index fed9a6cc3665..eaddec67e5e1 100644 --- a/Makefile +++ b/Makefile @@ -513,8 +513,6 @@ LZ4 = lz4c XZ = xz ZSTD = zstd -PAHOLE_FLAGS = $(shell PAHOLE=$(PAHOLE) $(srctree)/scripts/pahole-flags.sh) - CHECKFLAGS := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ \ -Wbitwise -Wno-return-void -Wno-unknown-attribute $(CF) NOSTDINC_FLAGS := @@ -605,7 +603,6 @@ export KBUILD_RUSTFLAGS RUSTFLAGS_KERNEL RUSTFLAGS_MODULE export KBUILD_AFLAGS AFLAGS_KERNEL AFLAGS_MODULE export KBUILD_AFLAGS_MODULE KBUILD_CFLAGS_MODULE KBUILD_RUSTFLAGS_MODULE KBUILD_LDFLAGS_MODULE export KBUILD_AFLAGS_KERNEL KBUILD_CFLAGS_KERNEL KBUILD_RUSTFLAGS_KERNEL -export PAHOLE_FLAGS # Files to ignore in find ... statements @@ -1002,6 +999,7 @@ KBUILD_CPPFLAGS += $(call cc-option,-fmacro-prefix-map=$(srctree)/=) # include additional Makefiles when needed include-y := scripts/Makefile.extrawarn include-$(CONFIG_DEBUG_INFO) += scripts/Makefile.debug +include-$(CONFIG_DEBUG_INFO_BTF)+= scripts/Makefile.btf include-$(CONFIG_KASAN) += scripts/Makefile.kasan include-$(CONFIG_KCSAN) += scripts/Makefile.kcsan include-$(CONFIG_KMSAN) += scripts/Makefile.kmsan diff --git a/scripts/Makefile.btf b/scripts/Makefile.btf new file mode 100644 index 000000000000..82377e470aed --- /dev/null +++ b/scripts/Makefile.btf @@ -0,0 +1,19 @@ +# SPDX-License-Identifier: GPL-2.0 + +pahole-ver := $(CONFIG_PAHOLE_VERSION) +pahole-flags-y := + +# pahole 1.18 through 1.21 can't handle zero-sized per-CPU vars +ifeq ($(call test-le, $(pahole-ver), 121),y) +pahole-flags-$(call test-ge, $(pahole-ver), 118) += --skip_encoding_btf_vars +endif + +pahole-flags-$(call test-ge, $(pahole-ver), 121) += --btf_gen_floats + +pahole-flags-$(call test-ge, $(pahole-ver), 122) += -j + +pahole-flags-$(CONFIG_PAHOLE_HAS_LANG_EXCLUDE) += --lang_exclude=rust + +pahole-flags-$(call test-ge, $(pahole-ver), 125) += --skip_encoding_btf_inconsistent_proto --btf_gen_optimized + +export PAHOLE_FLAGS := $(pahole-flags-y) diff --git a/scripts/pahole-flags.sh b/scripts/pahole-flags.sh deleted file mode 100755 index 728d55190d97..000000000000 --- a/scripts/pahole-flags.sh +++ /dev/null @@ -1,30 +0,0 @@ -#!/bin/sh -# SPDX-License-Identifier: GPL-2.0 - -extra_paholeopt= - -if ! [ -x "$(command -v ${PAHOLE})" ]; then - exit 0 -fi - -pahole_ver=$($(dirname $0)/pahole-version.sh ${PAHOLE}) - -if [ "${pahole_ver}" -ge "118" ] && [ "${pahole_ver}" -le "121" ]; then - # pahole 1.18 through 1.21 can't handle zero-sized per-CPU vars - extra_paholeopt="${extra_paholeopt} --skip_encoding_btf_vars" -fi -if [ "${pahole_ver}" -ge "121" ]; then - extra_paholeopt="${extra_paholeopt} --btf_gen_floats" -fi -if [ "${pahole_ver}" -ge "122" ]; then - extra_paholeopt="${extra_paholeopt} -j" -fi -if [ "${pahole_ver}" -ge "124" ]; then - # see PAHOLE_HAS_LANG_EXCLUDE - extra_paholeopt="${extra_paholeopt} --lang_exclude=rust" -fi -if [ "${pahole_ver}" -ge "125" ]; then - extra_paholeopt="${extra_paholeopt} --skip_encoding_btf_inconsistent_proto --btf_gen_optimized" -fi - -echo ${extra_paholeopt}