From patchwork Thu Jan 12 02:30:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Masahiro Yamada X-Patchwork-Id: 42223 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp3653202wrt; Wed, 11 Jan 2023 18:33:35 -0800 (PST) X-Google-Smtp-Source: AMrXdXtlTFG3USYbynexlwNcv1t8NK59v5kxVknQoVTpUbgW444YRiHIzb7VwXBUNKxNg/JxbJ+r X-Received: by 2002:a05:6402:501c:b0:48f:a9a2:29fa with SMTP id p28-20020a056402501c00b0048fa9a229famr28401271eda.2.1673490815721; Wed, 11 Jan 2023 18:33:35 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673490815; cv=none; d=google.com; s=arc-20160816; b=nMxN/qmK+IzOgsQfwOvdm30d91IEyB7X6Qtb8X97hwJHr6mpo1XQ+sOGwP1JP0QgNA BCcDTtPVDivKlNccMQkjNLLFMIeDpb/Prxa725C+e4o2GtFrhVihnHUZFRpCjGmDcrCs +Onso05/W7PqR/FF++RASV262A3MGssBdvYjzsKwHg3yPVFh3I+BmNEH1lDRQd5py7GC kHBDdt5YBB2WNZ0j+yN/gi1sFnSWgW+3+dqTz1I8sUEkEQsU9n0IrXak3LMIhCHh32Ys pvedDoSnPPQU1RgM2k8zCnE7eZSghOe0Va0M7Iifv0dM4cLX2UOP3ulBFgOKCtAWQcX7 efUQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=mQUBeK7v2oJLNuUH5ttIzK6k4kb8JAPg00ASQMnWQi8=; b=m910/BO6fxcP3VdAbSklRzYNbvBcvusOH/46RZGEp7ChuAPzB/cd/sL+hV4rD6TA3g uMZr6TWWeIX0FaWmdAOIaBTBtbPkLRKhv2VKqbl4EybIwz3i2/HnZWNhmSmpQM4hbhnL /7ZIEh/umIpIidkovPO8rJ0ZeKYvkyGLh7IWabYO68OMWTba8HRr03b6VInX3t4ve4hH w3CUKQPPRkPo4jxEJYvc/FcI+Dmp7m4LkGGZgio02POxjbHIllFcd1C+a+r6A5LuCmyS Pwd+ybZcujeMQrRLygYY61M8QxUeQHRA+y9x24ezlK18wu9WpDv5jiVQwnLpq4keQH/4 T73A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="dxygjh/v"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ds5-20020a170907724500b0084514612c2bsi18097883ejc.610.2023.01.11.18.33.12; Wed, 11 Jan 2023 18:33:35 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="dxygjh/v"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235111AbjALCaS (ORCPT + 99 others); Wed, 11 Jan 2023 21:30:18 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43534 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232849AbjALCaQ (ORCPT ); Wed, 11 Jan 2023 21:30:16 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B8375BC93; Wed, 11 Jan 2023 18:30:15 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 42A4D61F26; Thu, 12 Jan 2023 02:30:15 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E681FC433D2; Thu, 12 Jan 2023 02:30:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1673490614; bh=+mXHR/+rGW0aCgJwKXrXqkMMXhnGOb1bifJkJNWqx5c=; h=From:To:Cc:Subject:Date:From; b=dxygjh/vD0wnwbL8avrJWBV66sRuZ3j2IUIoJ9d7woImIR7P+EzZG7hgdN197ns7D 1jJ5Vx3Pi/FvFNWAVrb82MYT80AdnociIKU9LTYFjnJVu8/cYRxRFlGJkUEvfYyAi5 /H1RbOdfzNo1FK0gmjCOUmpAnOWe+dBv60btqS0UzrWobbz3aMYvj0zfMZRGmYsCPk fedawfi0kWZ9ztlPbBcOrUmIqEiS53fGtuPODyf4xzcfVB2sJqsp4zztGlwtoI1kGZ Y6kvzSjdZ0lt1+STKN/qOxbMfpOqcDxnxTmTdBlHrhv74FUdH5DJqJTBPvoHjCL/lx D1IeNxBp/6U1w== From: Masahiro Yamada To: linux-kbuild@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Masahiro Yamada , Kees Cook , Nathan Chancellor , Nick Desaulniers , Tom Rix , llvm@lists.linux.dev Subject: [PATCH] scripts: handle BrokenPipeError for python scripts Date: Thu, 12 Jan 2023 11:30:06 +0900 Message-Id: <20230112023006.1873859-1-masahiroy@kernel.org> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754782305871785300?= X-GMAIL-MSGID: =?utf-8?q?1754782305871785300?= In the follow-up of commit fb3041d61f68 ("kbuild: fix SIGPIPE error message for AR=gcc-ar and AR=llvm-ar"), Kees Cook pointed out that tools should _not_ catch their own SIGPIPEs [1] [2]. Based on his feedback, LLVM was fixed [3]. However, Python's default behavior is to show noisy bracktrace when SIGPIPE is sent. So, scripts written in Python are basically in the same situation as the buggy llvm tools. Example: $ make -s allnoconfig $ make -s allmodconfig $ scripts/diffconfig .config.old .config | head -n1 -ALIX n Traceback (most recent call last): File "/home/masahiro/linux/scripts/diffconfig", line 132, in main() File "/home/masahiro/linux/scripts/diffconfig", line 130, in main print_config("+", config, None, b[config]) File "/home/masahiro/linux/scripts/diffconfig", line 64, in print_config print("+%s %s" % (config, new_value)) BrokenPipeError: [Errno 32] Broken pipe Python documentatin [4] notes how to make scripts die immediately and silently: """ Piping output of your program to tools like head(1) will cause a SIGPIPE signal to be sent to your process when the receiver of its standard output closes early. This results in an exception like BrokenPipeError: [Errno 32] Broken pipe. To handle this case, wrap your entry point to catch this exception as follows: import os import sys def main(): try: # simulate large output (your code replaces this loop) for x in range(10000): print("y") # flush output here to force SIGPIPE to be triggered # while inside this try block. sys.stdout.flush() except BrokenPipeError: # Python flushes standard streams on exit; redirect remaining output # to devnull to avoid another BrokenPipeError at shutdown devnull = os.open(os.devnull, os.O_WRONLY) os.dup2(devnull, sys.stdout.fileno()) sys.exit(1) # Python exits with error code 1 on EPIPE if __name__ == '__main__': main() Do not set SIGPIPE’s disposition to SIG_DFL in order to avoid BrokenPipeError. Doing that would cause your program to exit unexpectedly whenever any socket connection is interrupted while your program is still writing to it. """ Currently, tools/perf/scripts/python/intel-pt-events.py seems the only script that fixes the issue that way. tools/perf/scripts/python/compaction-times.py uses another approach signal.signal(signal.SIGPIPE, signal.SIG_DFL) but the Python documentation clearly says "Don't do it". I cannot fix all Python scripts since there are so many. I fixed some in the scripts/ directory. [1]: https://lore.kernel.org/all/202211161056.1B9611A@keescook/ [2]: https://github.com/llvm/llvm-project/issues/59037 [3]: https://github.com/llvm/llvm-project/commit/4787efa38066adb51e2c049499d25b3610c0877b [4]: https://docs.python.org/3/library/signal.html#note-on-sigpipe Cc: Kees Cook Signed-off-by: Masahiro Yamada Reviewed-by: Nick Desaulniers Reviewed-by: Nicolas Schier --- scripts/checkkconfigsymbols.py | 13 ++++++++++++- scripts/clang-tools/run-clang-tools.py | 21 ++++++++++++++------- scripts/diffconfig | 16 ++++++++++++++-- 3 files changed, 40 insertions(+), 10 deletions(-) diff --git a/scripts/checkkconfigsymbols.py b/scripts/checkkconfigsymbols.py index 217d21abc86e..36c920e71313 100755 --- a/scripts/checkkconfigsymbols.py +++ b/scripts/checkkconfigsymbols.py @@ -115,7 +115,7 @@ def parse_options(): return args -def main(): +def print_undefined_symbols(): """Main function of this module.""" args = parse_options() @@ -467,5 +467,16 @@ def parse_kconfig_file(kfile): return defined, references +def main(): + try: + print_undefined_symbols() + except BrokenPipeError: + # Python flushes standard streams on exit; redirect remaining output + # to devnull to avoid another BrokenPipeError at shutdown + devnull = os.open(os.devnull, os.O_WRONLY) + os.dup2(devnull, sys.stdout.fileno()) + sys.exit(1) # Python exits with error code 1 on EPIPE + + if __name__ == "__main__": main() diff --git a/scripts/clang-tools/run-clang-tools.py b/scripts/clang-tools/run-clang-tools.py index 56f2ec8f0f40..3266708a8658 100755 --- a/scripts/clang-tools/run-clang-tools.py +++ b/scripts/clang-tools/run-clang-tools.py @@ -61,14 +61,21 @@ def run_analysis(entry): def main(): - args = parse_arguments() + try: + args = parse_arguments() - lock = multiprocessing.Lock() - pool = multiprocessing.Pool(initializer=init, initargs=(lock, args)) - # Read JSON data into the datastore variable - with open(args.path, "r") as f: - datastore = json.load(f) - pool.map(run_analysis, datastore) + lock = multiprocessing.Lock() + pool = multiprocessing.Pool(initializer=init, initargs=(lock, args)) + # Read JSON data into the datastore variable + with open(args.path, "r") as f: + datastore = json.load(f) + pool.map(run_analysis, datastore) + except BrokenPipeError: + # Python flushes standard streams on exit; redirect remaining output + # to devnull to avoid another BrokenPipeError at shutdown + devnull = os.open(os.devnull, os.O_WRONLY) + os.dup2(devnull, sys.stdout.fileno()) + sys.exit(1) # Python exits with error code 1 on EPIPE if __name__ == "__main__": diff --git a/scripts/diffconfig b/scripts/diffconfig index d5da5fa05d1d..43f0f3d273ae 100755 --- a/scripts/diffconfig +++ b/scripts/diffconfig @@ -65,7 +65,7 @@ def print_config(op, config, value, new_value): else: print(" %s %s -> %s" % (config, value, new_value)) -def main(): +def show_diff(): global merge_style # parse command line args @@ -129,4 +129,16 @@ def main(): for config in new: print_config("+", config, None, b[config]) -main() +def main(): + try: + show_diff() + except BrokenPipeError: + # Python flushes standard streams on exit; redirect remaining output + # to devnull to avoid another BrokenPipeError at shutdown + devnull = os.open(os.devnull, os.O_WRONLY) + os.dup2(devnull, sys.stdout.fileno()) + sys.exit(1) # Python exits with error code 1 on EPIPE + + +if __name__ == '__main__': + main()