Message ID | 20230617153025.1653851-1-masahiroy@kernel.org |
---|---|
State | New |
Headers |
Return-Path: <linux-kernel-owner@vger.kernel.org> Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2041292vqr; Sat, 17 Jun 2023 08:36:14 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6zUU7gzMTQtUGBF+/QHlVK4ZbPKBQgSvPEFSvsBcZlWVEkf4QF/J+uwRrfXrgbJU334K/w X-Received: by 2002:a05:6a00:a28:b0:668:70f2:50f9 with SMTP id p40-20020a056a000a2800b0066870f250f9mr647179pfh.14.1687016174435; Sat, 17 Jun 2023 08:36:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1687016174; cv=none; d=google.com; s=arc-20160816; b=nwVzCYAtGgX5smlTrFtU1AG6OuJIqzkZzPmidFQy9IOabN90RbuFLhMnMIuAlJM5mR ZvaHmxiUTWJpC3Hd4niJneS4fFFDQ5TVmV03ljPiDNmENzcYJ8me0FPFcQG5ugx3YOlb TYYIy0bP9cbjq2h4OMgtPcp9Cy5YfPU9SL69TtWwW7crRK3pAVp3dBTIl8Z4pYdFFqsS FpWiiq9E/WfDG/QenCJHtCpq+Lkvkap7gcmn/EuHMEn7witTpLLM7IulJXGRqsS8FLU7 g/2gBv5BaywlX6UyHe9hSFCU/0iOWMz/6FwrLRrqNbynfGXTQcHnyXXR8Yz6fP4anXXv LUfw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=gaVWtv8uEvP/T7qhS8cF1xXBGVNT3qUpqCTpUt+MowQ=; b=klPIS89U3rEVc9EaL2+G4EosB2gei2sAEpCfKhvN2NraZHu1YnCXQOFxzbSsSHhWIC JwwPaHWSH3mIfNJyuqNLz0NMrTt74G/iBkfmZJSyHW+ddhICckeu3iMdM31Fi6O7lAdx ACycee1qRNwrZ/nL1A9H+I6Fv6Oa/i4jTOSZegaXAA1VHzAPu0mM5TiBofbkTgwAw4bJ N7gwRsDYX8qUbSOrLUDHxO0O8Evps05RjBJfrRs4d1ciGiJ272cproyvq81313uxJ9vX D3pZjt35aXaV/vu7VOwyWqHBVNt0GNujsNjCR6quMEK5mehhB5I2oP5nZl+1VZGDNdwv dSmw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=JKwE4ZuW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id s19-20020a639253000000b0051f74e18927si17355291pgn.184.2023.06.17.08.35.59; Sat, 17 Jun 2023 08:36:14 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=JKwE4ZuW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233419AbjFQPai (ORCPT <rfc822;duw91626@gmail.com> + 99 others); Sat, 17 Jun 2023 11:30:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33666 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231618AbjFQPaf (ORCPT <rfc822;linux-kernel@vger.kernel.org>); Sat, 17 Jun 2023 11:30:35 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1E25F1FD7; Sat, 17 Jun 2023 08:30:34 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 9991560FA0; Sat, 17 Jun 2023 15:30:33 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 62639C433C8; Sat, 17 Jun 2023 15:30:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1687015833; bh=5iL1FxKNMlL/BSu7pumyfvfaweoKYzZMLOet3Mrh+B8=; h=From:To:Cc:Subject:Date:From; b=JKwE4ZuWdpep3jsmBWR6/OStT781HZs+JeLHr6Gj8JdPqnFoNuC88C/wjDniaqWor Eu0EylBuoewFFPFcFhp1Z1bK5RoL067rBj1l0mQ+ORGhXqokEnNR8fmzdIAb6q3H5e 9ksyD+6gPr5i8SBz18Qfeb2tUCK4ZEqgwaeizqTtFCua3U2g5Ql04lLwP3EQkigWZy idl7nfcsjpXO54AZ8fjAuBMtqx7Xd5DucsjdiWX2Vrt1YmdsI5bKgbKqOx5VhMhL2h u33MKHY2esIj/rms6ACTtJkkJC4S4nLFi1KYY22ptQmdbO1BldHJbhqeqOrcvSH3yZ jQe+bji+zDKcw== From: Masahiro Yamada <masahiroy@kernel.org> To: linux-kbuild@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Nathan Chancellor <nathan@kernel.org>, Nick Desaulniers <ndesaulniers@google.com>, Nicolas Schier <nicolas@fjasle.eu>, Masahiro Yamada <masahiroy@kernel.org>, Eugeniu Rosca <erosca@de.adit-jv.com> Subject: [PATCH] kbuild: make clean rule robust against too long argument error Date: Sun, 18 Jun 2023 00:30:25 +0900 Message-Id: <20230617153025.1653851-1-masahiroy@kernel.org> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: <linux-kernel.vger.kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1768964672277725509?= X-GMAIL-MSGID: =?utf-8?q?1768964672277725509?= |
Series |
kbuild: make clean rule robust against too long argument error
|
|
Commit Message
Masahiro Yamada
June 17, 2023, 3:30 p.m. UTC
Commit cd968b97c492 ("kbuild: make built-in.a rule robust against too
long argument error") made a build rule robust against "Argument list
too long" error.
Eugeniu Rosca reported the same error occurred when cleaning an external
module.
The $(obj)/ prefix can be a very long path for external modules.
Apply a similar solution to 'make clean'.
Reported-by: Eugeniu Rosca <erosca@de.adit-jv.com>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
---
scripts/Makefile.clean | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
Comments
Hello Yamada-san, Many thanks for your feedback and for your patch. It indeed addresses the issue I have reported in [1]. On Sun, Jun 18, 2023 at 12:30:25AM +0900, Masahiro Yamada wrote: > Commit cd968b97c492 ("kbuild: make built-in.a rule robust against too > long argument error") made a build rule robust against "Argument list > too long" error. > > Eugeniu Rosca reported the same error occurred when cleaning an external > module. > > The $(obj)/ prefix can be a very long path for external modules. Confirmed. I am seeing an instance of $(obj) being 150 characters long, due to an out-of-tree module deeply buried in a specific Yocto build. In the current vanilla version of 'make clean' (w/o this patch), the $(obj) prefix is applied to each and every file being removed, dramatically increasing the strlen of arguments passed to 'rm -rf'. > > Apply a similar solution to 'make clean'. > > Reported-by: Eugeniu Rosca <erosca@de.adit-jv.com> > Signed-off-by: Masahiro Yamada <masahiroy@kernel.org> > --- > > scripts/Makefile.clean | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/scripts/Makefile.clean b/scripts/Makefile.clean > index 3649900696dd..235408a44f90 100644 > --- a/scripts/Makefile.clean > +++ b/scripts/Makefile.clean > @@ -37,8 +37,9 @@ __clean-files := $(wildcard $(addprefix $(obj)/, $(__clean-files))) > > # ========================================================================== > > +# Use xargs to make this robust against "Argument list too long" error Please, correct me if I am wrong, but it looks like the magic/brilliance is in the 'patsubst' function, since below version also fails for me: NOK: cmd_clean = printf '%s ' $(__clean-files) | xargs rm -rf When running 'make clean' for my particular out-of-tree kernel module, 'patsubst' is able to decrease the total number of characters passed to the shell's argument list from ~204k to ~47k, which elegantly prevents the "Argument list too long" error. > quiet_cmd_clean = CLEAN $(obj) > - cmd_clean = rm -rf $(__clean-files) > + cmd_clean = printf '$(obj)/%s ' $(patsubst $(obj)/%,%,$(__clean-files)) | xargs rm -rf > > __clean: $(subdir-ymn) > ifneq ($(strip $(__clean-files)),) > -- > 2.39.2 > Since it fixes the problem reported in [1] (much appreciated): Reviewed-by: Eugeniu Rosca <erosca@de.adit-jv.com> Tested-by: Eugeniu Rosca <erosca@de.adit-jv.com> [1] https://lore.kernel.org/linux-kbuild/20230616194505.GA27753@lxhi-065/
On Sun, Jun 18, 2023 at 5:07 AM Eugeniu Rosca <erosca@de.adit-jv.com> wrote: > > Hello Yamada-san, > > Many thanks for your feedback and for your patch. > It indeed addresses the issue I have reported in [1]. > > On Sun, Jun 18, 2023 at 12:30:25AM +0900, Masahiro Yamada wrote: > > Commit cd968b97c492 ("kbuild: make built-in.a rule robust against too > > long argument error") made a build rule robust against "Argument list > > too long" error. > > > > Eugeniu Rosca reported the same error occurred when cleaning an external > > module. > > > > The $(obj)/ prefix can be a very long path for external modules. > > Confirmed. I am seeing an instance of $(obj) being 150 characters long, > due to an out-of-tree module deeply buried in a specific Yocto build. > > In the current vanilla version of 'make clean' (w/o this patch), the > $(obj) prefix is applied to each and every file being removed, > dramatically increasing the strlen of arguments passed to 'rm -rf'. > > > > > Apply a similar solution to 'make clean'. > > > > Reported-by: Eugeniu Rosca <erosca@de.adit-jv.com> > > Signed-off-by: Masahiro Yamada <masahiroy@kernel.org> > > --- > > > > scripts/Makefile.clean | 3 ++- > > 1 file changed, 2 insertions(+), 1 deletion(-) > > > > diff --git a/scripts/Makefile.clean b/scripts/Makefile.clean > > index 3649900696dd..235408a44f90 100644 > > --- a/scripts/Makefile.clean > > +++ b/scripts/Makefile.clean > > @@ -37,8 +37,9 @@ __clean-files := $(wildcard $(addprefix $(obj)/, $(__clean-files))) > > > > # ========================================================================== > > > > +# Use xargs to make this robust against "Argument list too long" error > > Please, correct me if I am wrong, but it looks like the magic/brilliance > is in the 'patsubst' function, since below version also fails for me: > > NOK: cmd_clean = printf '%s ' $(__clean-files) | xargs rm -rf Right. Now, 'printf' (instead of 'rm') failed with the too long argument list. GNU Make does not have the length limit, but shell does. So, the full-path list must be passed via stdout instead of the command line. The comment might be confusing. I will repeat the same comment written in scripts/Makefile.build: # To make this rule robust against "Argument list too long" error, # remove $(obj)/ prefix, and restore it by a shell command.
Hello Yamada-san, On Mon, Jun 19, 2023 at 12:50:48AM +0900, Masahiro Yamada wrote: > On Sun, Jun 18, 2023 at 5:07 AM Eugeniu Rosca <erosca@de.adit-jv.com> wrote: [snip] > > Please, correct me if I am wrong, but it looks like the magic/brilliance > > is in the 'patsubst' function, since below version also fails for me: > > > > NOK: cmd_clean = printf '%s ' $(__clean-files) | xargs rm -rf > > Right. > Now, 'printf' (instead of 'rm') failed with the too long argument list. > > GNU Make does not have the length limit, but shell does. > So, the full-path list must be passed via stdout > instead of the command line. > > > The comment might be confusing. > > I will repeat the same comment written in scripts/Makefile.build: > > # To make this rule robust against "Argument list too long" error, > # remove $(obj)/ prefix, and restore it by a shell command. Sounds good to me. No open questions from my end :)
diff --git a/scripts/Makefile.clean b/scripts/Makefile.clean index 3649900696dd..235408a44f90 100644 --- a/scripts/Makefile.clean +++ b/scripts/Makefile.clean @@ -37,8 +37,9 @@ __clean-files := $(wildcard $(addprefix $(obj)/, $(__clean-files))) # ========================================================================== +# Use xargs to make this robust against "Argument list too long" error quiet_cmd_clean = CLEAN $(obj) - cmd_clean = rm -rf $(__clean-files) + cmd_clean = printf '$(obj)/%s ' $(patsubst $(obj)/%,%,$(__clean-files)) | xargs rm -rf __clean: $(subdir-ymn) ifneq ($(strip $(__clean-files)),)