From patchwork Tue Aug 9 12:00:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Martin_Li=C5=A1ka?= X-Patchwork-Id: 445 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:6a10:20da:b0:2d3:3019:e567 with SMTP id n26csp2466533pxc; Tue, 9 Aug 2022 05:02:19 -0700 (PDT) X-Google-Smtp-Source: AA6agR4l7Oqrt923s86T7R20tA6I6Z7YnGgFce3qAUw+fUUb/SRyCauCRZKogYmDWWZkjnCXZI/n X-Received: by 2002:a05:6402:3595:b0:43d:710a:3f3f with SMTP id y21-20020a056402359500b0043d710a3f3fmr21531680edc.375.1660046539777; Tue, 09 Aug 2022 05:02:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1660046539; cv=none; d=google.com; s=arc-20160816; b=ljZGt8bYKcYCsn2Qx3MuOJ78vdGQvo5T/cI3YU/OaMFzhyDaWWrArMzHDkZzefGZS9 zPS/kP2X8zg/obUfvpcGmXB24o0NtZQc9ruaGhSlvRNSnlj5N9np6TyldIX4smZsAwLE wKkzLjrsiPZO+1cizsfKbF+Lns0uflQM1tQltFy2zljpGxMRSZ27UFSoHynUgUOjPKOH 8/1ewqkSqvzVwXY1pxCIvkW0mZB9oVldiASWKdPqrQE8DkdrIrmU6ajZVGZDN5jVEbau amIBynT47JOSfDpHorCVPCK5Tk07okR0pWwTQFuHHto/3jlEizH91Jsoi5eUotPtYAG9 /nsg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :in-reply-to:content-language:references:to:subject:from:user-agent :mime-version:date:message-id:dkim-signature:dkim-signature :dmarc-filter:delivered-to; bh=8KzaMSGjc5p6+pNghtiwFKaJVf/6DC9367rxYvckkG0=; b=pc+vX6WfVSljHZWxrjHRWxws5jBNzWKMI/tiPXuByvd2S5SMs0OWqFlnMrBdnFEe5a bKnbAma7LtfosohES5qDLeWotML6zkrxeNX8g/mXCqlPDoseVK5rDmCAh8VaQb5UglE9 D1evonOeP6xtdn8OtlqVamhvBeRFeIQBdULicHS7RcuQrvrTSvOCyOSv+hWCidU1rJgw mZ0grBq4oIdBSf+ZaIEeC4Jin3vLc7bO+vkryX7ncR4jbkLAvC+ry1Ex5LsZ+huJbQcr a2FoNizzwfiFkw4003mTL9H/g4YMRF6qKJEot29aiG7ED553GVpZ1Gpg4AEmJes9ubUv aOKA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.cz header.s=susede2_rsa header.b=zXOuTTLY; dkim=neutral (no key) header.i=@suse.cz header.s=susede2_ed25519 header.b=YalzVI1i; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from sourceware.org (server2.sourceware.org. [8.43.85.97]) by mx.google.com with ESMTPS id g8-20020a056402090800b0043cd07bb8c3si9479871edz.380.2022.08.09.05.02.19 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Aug 2022 05:02:19 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) client-ip=8.43.85.97; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.cz header.s=susede2_rsa header.b=zXOuTTLY; dkim=neutral (no key) header.i=@suse.cz header.s=susede2_ed25519 header.b=YalzVI1i; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 8172D3851168 for ; Tue, 9 Aug 2022 12:01:09 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by sourceware.org (Postfix) with ESMTPS id B0FD5385AC1D for ; Tue, 9 Aug 2022 12:00:44 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org B0FD5385AC1D Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=suse.cz Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 573881FDB6 for ; Tue, 9 Aug 2022 12:00:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1660046443; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8KzaMSGjc5p6+pNghtiwFKaJVf/6DC9367rxYvckkG0=; b=zXOuTTLYl4PUY+aCgw7sPKis0J2Yf4r1KngDVD/Vt70NlpVrTQlvQVVSeZjpnb9xQA1ywe +JUg3MY91v43yIG3uSpVDHWLFHgMCCmJ0OOkOjqrTmPCPWb/qUm2PZ/9tCplNnYLAvA+bi QNJNScczU3BN8g0gj/Qu3ijkEOpZcpg= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1660046443; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8KzaMSGjc5p6+pNghtiwFKaJVf/6DC9367rxYvckkG0=; b=YalzVI1igiAioDh8IYAFtCUdNMTb9b1qvhx2QNZerH3vcYuYHlIp27SqKohCr6CuWvcFij 2z/IctDqLl98ssBw== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 49D5D13A9D for ; Tue, 9 Aug 2022 12:00:43 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id gsSJEGtM8mL0AwAAMHmgww (envelope-from ) for ; Tue, 09 Aug 2022 12:00:43 +0000 Message-ID: Date: Tue, 9 Aug 2022 14:00:42 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.1.0 From: =?utf-8?q?Martin_Li=C5=A1ka?= Subject: [PATCH 3/3] lto: respect jobserver in parallel WPA streaming To: gcc-patches@gcc.gnu.org References: <95d2ec7222d50c8ca1e6535cb8c272d47abbb08d.1660046383.git.mliska@suse.cz> <7ad7af4bb978bffdbc220c754002c667df1b0209.1660046383.git.mliska@suse.cz> Content-Language: en-US In-Reply-To: <7ad7af4bb978bffdbc220c754002c667df1b0209.1660046383.git.mliska@suse.cz> X-Spam-Status: No, score=-11.9 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org Sender: "Gcc-patches" X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1740684960424385408?= X-GMAIL-MSGID: =?utf-8?q?1740684960424385408?= Patch can bootstrap on x86_64-linux-gnu and survives regression tests. Ready to be installed? Thanks, Martin PR lto/106328 gcc/ChangeLog: * jobserver.h (struct jobserver_info): Add pipefd. (jobserver_info::connect): New. (jobserver_info::disconnect): Likewise. (jobserver_info::get_token): Likewise. (jobserver_info::return_token): Likewise. gcc/lto/ChangeLog: * lto.cc (wait_for_child): Decrement nruns once a process finishes. (stream_out_partitions): Use job server if active. (do_whole_program_analysis): Likewise. --- gcc/jobserver.h | 54 ++++++++++++++++++++++++++++++++++++++++++++++++ gcc/lto/lto.cc | 55 +++++++++++++++++++++++++++++++++++++------------ 2 files changed, 96 insertions(+), 13 deletions(-) diff --git a/gcc/jobserver.h b/gcc/jobserver.h index 856e326ddfc..2a7dc9f4113 100644 --- a/gcc/jobserver.h +++ b/gcc/jobserver.h @@ -31,6 +31,18 @@ struct jobserver_info /* Default constructor. */ jobserver_info (); + /* Connect to the server. */ + void connect (); + + /* Disconnect from the server. */ + void disconnect (); + + /* Get token from the server. */ + bool get_token (); + + /* Return token to the server. */ + void return_token (); + /* Error message if there is a problem. */ string error_msg = ""; /* Skipped MAKEFLAGS where --jobserver-auth is skipped. */ @@ -41,6 +53,8 @@ struct jobserver_info int wfd = -1; /* Named pipe path. */ string pipe_path = ""; + /* Pipe file descriptor. */ + int pipefd = -1; /* Return true if jobserver is active. */ bool is_active = false; }; @@ -97,4 +111,44 @@ jobserver_info::jobserver_info () error_msg = "jobserver is not available: " + error_msg; } +void +jobserver_info::connect () +{ + if (!pipe_path.empty ()) + pipefd = open (pipe_path.c_str (), O_RDWR); +} + +void +jobserver_info::disconnect () +{ + if (!pipe_path.empty ()) + { + gcc_assert (close (pipefd) == 0); + pipefd = -1; + } +} + +bool +jobserver_info::get_token () +{ + int fd = pipe_path.empty () ? rfd : pipefd; + char c; + unsigned n = read (fd, &c, 1); + if (n != 1) + { + gcc_assert (errno == EAGAIN); + return false; + } + else + return true; +} + +void +jobserver_info::return_token () +{ + int fd = pipe_path.empty () ? wfd : pipefd; + char c = 'G'; + gcc_assert (write (fd, &c, 1) == 1); +} + #endif /* GCC_JOBSERVER_H */ diff --git a/gcc/lto/lto.cc b/gcc/lto/lto.cc index 31b0c1862f7..56266195ead 100644 --- a/gcc/lto/lto.cc +++ b/gcc/lto/lto.cc @@ -54,11 +54,17 @@ along with GCC; see the file COPYING3. If not see #include "attribs.h" #include "builtins.h" #include "lto-common.h" - +#include "jobserver.h" /* Number of parallel tasks to run, -1 if we want to use GNU Make jobserver. */ static int lto_parallelism; +/* Number of active WPA streaming processes. */ +static int nruns = 0; + +/* GNU make's jobserver info. */ +static jobserver_info *jinfo = NULL; + /* Return true when NODE has a clone that is analyzed (i.e. we need to load its body even if the node itself is not needed). */ @@ -205,6 +211,12 @@ wait_for_child () "streaming subprocess was killed by signal"); } while (!WIFEXITED (status) && !WIFSIGNALED (status)); + + --nruns; + + /* Return token to the jobserver if active. */ + if (jinfo != NULL && jinfo->is_active) + jinfo->return_token (); } #endif @@ -228,25 +240,35 @@ stream_out_partitions (char *temp_filename, int blen, int min, int max, bool ARG_UNUSED (last)) { #ifdef HAVE_WORKING_FORK - static int nruns; - if (lto_parallelism <= 1) { stream_out_partitions_1 (temp_filename, blen, min, max); return; } - /* Do not run more than LTO_PARALLELISM streamings - FIXME: we ignore limits on jobserver. */ if (lto_parallelism > 0 && nruns >= lto_parallelism) - { - wait_for_child (); - nruns --; - } + wait_for_child (); + /* If this is not the last parallel partition, execute new streaming process. */ if (!last) { + if (jinfo != NULL && jinfo->is_active) + while (true) + { + if (jinfo->get_token ()) + break; + if (nruns > 0) + wait_for_child (); + else + { + /* There are no free tokens, lets do the job outselves. */ + stream_out_partitions_1 (temp_filename, blen, min, max); + asm_nodes_output = true; + return; + } + } + pid_t cpid = fork (); if (!cpid) @@ -264,10 +286,12 @@ stream_out_partitions (char *temp_filename, int blen, int min, int max, /* Last partition; stream it and wait for all children to die. */ else { - int i; stream_out_partitions_1 (temp_filename, blen, min, max); - for (i = 0; i < nruns; i++) + while (nruns > 0) wait_for_child (); + + if (jinfo != NULL && jinfo->is_active) + jinfo->disconnect (); } asm_nodes_output = true; #else @@ -460,9 +484,14 @@ do_whole_program_analysis (void) lto_parallelism = 1; - /* TODO: jobserver communication is not supported, yet. */ if (!strcmp (flag_wpa, "jobserver")) - lto_parallelism = param_max_lto_streaming_parallelism; + { + jinfo = new jobserver_info (); + if (jinfo->is_active) + jinfo->connect (); + + lto_parallelism = param_max_lto_streaming_parallelism; + } else { lto_parallelism = atoi (flag_wpa);