From patchwork Fri Nov 17 20:16:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michal Jires X-Patchwork-Id: 166315 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9910:0:b0:403:3b70:6f57 with SMTP id i16csp788937vqn; Fri, 17 Nov 2023 12:17:07 -0800 (PST) X-Google-Smtp-Source: AGHT+IH5iOUzZ2G5YV/exWso1L1JFy6OWOuU1z6qjB/iLr0rugj1AtAItlMQ/Ml5IQuWsy3gtO69 X-Received: by 2002:a05:620a:4502:b0:77b:aa22:9b6c with SMTP id t2-20020a05620a450200b0077baa229b6cmr712756qkp.61.1700252227052; Fri, 17 Nov 2023 12:17:07 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1700252227; cv=pass; d=google.com; s=arc-20160816; b=fM7gu1+lxZvH90ovWPUzm7azjR0kQNsWtU/P/ZWMs6/qJoT2fecMXCwrVDCx+OHQLr Yr88QCw/Vhdx0yo6TydlAGf4K+NAR+g8RVli4/MU1CAWtLlQx0ya0JetZ5BBaApnePPa TCPwmDHcmMa5psXtjLA8rypE5QDbj4N5W4l0/s1z4Jsyw2Znkm7fquwoEkb0kRQyL7o3 pva54afzobz8zb7Fbjsr597FeZBhMZ05oF5wqbHjd9SYMkgCzyqR4ipdfIIJK+EAIL3U k108BrAIQvbuLfkqHYqqWC75seeLziTpn/P/58z39dUKNkpYWebo0bft+IcZGE+JNfIq n5pA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:in-reply-to:content-disposition :mime-version:references:message-id:subject:to:from:date :dkim-signature:dkim-signature:arc-filter:dmarc-filter:delivered-to; bh=VkJjbGfRjEkMIIdK4apKbrTTQsEnILwWa2zS3AnAUNo=; fh=hPrbWPhweUx4V0GV9uXJqbyAzg2ABmTz7kczrAQqMmM=; b=nmfwcyC6DbbPF7wB9KOd4Ci0oEr7SkKLwmw5gpqRyry9n0bwfgrIwX9jhk387mwbp3 iZHCKmFe77/D7iAjoErRs9Z9vWEtOxnmhC9Zrq6f056OWaKgLmVPpx3zo3ntX5uN8ydq +OOw3Uic4sJdfLrkbiEMhH7bnUT+6iwTlY/IlUbf5xfkmxs1zWvgcZ80al/3AHpLvL5/ HQdjZh6rF0ypD954K0qIQ+s0rseb22prHigK0M04FVUT9/bIXQV/sYP8Vp8OWxUiTyEp pFx/h67wSLn0fxqUxHLTx76m3HBnSd4rmrtYMUYFKLwdzn+LDNsfgFoi5WOf14vZcOJK j32w== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@suse.cz header.s=susede2_rsa header.b=XJ4daz2F; dkim=neutral (no key) header.i=@suse.cz header.s=susede2_ed25519 header.b=nMX+KMME; arc=pass (i=1); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (server2.sourceware.org. [8.43.85.97]) by mx.google.com with ESMTPS id bp31-20020a05620a459f00b0077a1c623987si2359671qkb.293.2023.11.17.12.17.06 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 17 Nov 2023 12:17:07 -0800 (PST) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) client-ip=8.43.85.97; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.cz header.s=susede2_rsa header.b=XJ4daz2F; dkim=neutral (no key) header.i=@suse.cz header.s=susede2_ed25519 header.b=nMX+KMME; arc=pass (i=1); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id A12343858C5F for ; Fri, 17 Nov 2023 20:17:04 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by sourceware.org (Postfix) with ESMTPS id 223EB3858CD1 for ; Fri, 17 Nov 2023 20:16:40 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 223EB3858CD1 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=suse.cz ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 223EB3858CD1 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=195.135.220.29 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1700252201; cv=none; b=Tbqd4Ipmgfr3LWhzXqIB+fJg4mS1uAuwnuAVUcTCDsSt9uncT+82Q/DuYYAVhtxkwNG6AEbd3Ny+AhFp5Bst1talKtsGI+VsseAgJhcZ5R8vwyGmhbKdLXr/nVPXq7UsaLugcawnVw34mc8wSP6UFPT2X9fxvTBFfLQ1xQH6VWI= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1700252201; c=relaxed/simple; bh=xbb7Bi8DwypeT5HMSjaN3AWcUh2egJ24vfC9IfZ5cDs=; h=DKIM-Signature:DKIM-Signature:Date:From:To:Subject:Message-ID: MIME-Version; b=r/0M1TeAFAQda8XcCyaLhXeC+1XMmLajeblQBQzkc4n/v+uuo5R/TfbDShMZcfYno+3RIb41p1mlN6xKIx8RauuPA7C316GQuI8uLNwqdCnVFZKeWg0H/0F24+MfiPHh1EIPytiVyUddn3kCtmCJeRZ8qM4/xIMeSmaa/4czKlk= ARC-Authentication-Results: i=1; server2.sourceware.org Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 170181F388 for ; Fri, 17 Nov 2023 20:16:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1700252199; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=VkJjbGfRjEkMIIdK4apKbrTTQsEnILwWa2zS3AnAUNo=; b=XJ4daz2FKmEQZ+5QH6JMsPgC9vVsB5se5piolYAYc6Gat3qAQoCxKtiQGt46xJuu/it2Wp bo+gRqUZrsx+NUKRuHSVxfoPFVKmnVbNn6SRicAtzydOVa/xT74XAJ6qrEPiHGy8O+zxAF UsXzNdL7lxta0kzXu9ZL69q6ziKjb3g= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1700252199; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=VkJjbGfRjEkMIIdK4apKbrTTQsEnILwWa2zS3AnAUNo=; b=nMX+KMMEiuJ87RtX9sLkfX/rw3jkVyvAx+gcGKUDIdrChyM9iG5hlai540tueuSAjZX3+W puqBtqAQ1J8GK1BQ== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id F1A0C1341F for ; Fri, 17 Nov 2023 20:16:38 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id /x8XOSbKV2XuGQAAMHmgww (envelope-from ) for ; Fri, 17 Nov 2023 20:16:38 +0000 Date: Fri, 17 Nov 2023 21:16:37 +0100 From: Michal Jires To: gcc-patches@gcc.gnu.org Subject: [PATCH 1/7] lto: Skip flag OPT_fltrans_output_list_. Message-ID: <81dd3d9d61f5b1907f91ca35209167049e5bee54.1700222403.git.mjires@suse.cz> References: <18cc1c3980551ac1881eea6e78811a629c7baa82.1700222403.git.mjires@suse.cz> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <18cc1c3980551ac1881eea6e78811a629c7baa82.1700222403.git.mjires@suse.cz> Authentication-Results: smtp-out2.suse.de; none X-Spam-Level: X-Spam-Score: -3.29 X-Spamd-Result: default: False [-3.29 / 50.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[gcc-patches@gcc.gnu.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; DKIM_SIGNED(0.00)[suse.cz:s=susede2_rsa,suse.cz:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.19)[-0.928]; MID_CONTAINS_FROM(1.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; BAYES_HAM(-3.00)[99.99%] X-Spam-Status: No, score=-11.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1782843679118365414 X-GMAIL-MSGID: 1782843679118365414 Bootstrapped/regtested on x86_64-pc-linux-gnu gcc/ChangeLog: * lto-opts.cc (lto_write_options): Skip OPT_fltrans_output_list_. --- gcc/lto-opts.cc | 1 + 1 file changed, 1 insertion(+) diff --git a/gcc/lto-opts.cc b/gcc/lto-opts.cc index c9bee9d4197..0451e290c75 100644 --- a/gcc/lto-opts.cc +++ b/gcc/lto-opts.cc @@ -152,6 +152,7 @@ lto_write_options (void) case OPT_fprofile_prefix_map_: case OPT_fcanon_prefix_map: case OPT_fwhole_program: + case OPT_fltrans_output_list_: continue; default: From patchwork Fri Nov 17 20:16:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michal Jires X-Patchwork-Id: 166316 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9910:0:b0:403:3b70:6f57 with SMTP id i16csp789061vqn; Fri, 17 Nov 2023 12:17:20 -0800 (PST) X-Google-Smtp-Source: AGHT+IHDdQjDXKJ2kArkI94PjVCgF5pwU8TCvuld5rR9yQIo5S1c1jrMfxKrmSpnYQ8ECj/+a8qm X-Received: by 2002:a05:620a:258d:b0:774:c82:7536 with SMTP id x13-20020a05620a258d00b007740c827536mr681160qko.71.1700252240126; Fri, 17 Nov 2023 12:17:20 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1700252240; cv=pass; d=google.com; s=arc-20160816; b=R7nilm66dp0OCpjSYkw2D+aQ4jiSf4xYTWe8PdoHZQ8Fv6OV5433q34AKiAR3gH2/x SqQH3eJV7hieVZ50ugKlN97fJ5MS5NmQPVsGplusxBEDTmyzCYOfPhUqc4gnfnkaHGCB cIS9LDbbH3u0x0txC3mWfcP1AhP4YbTA94jliDBhTPS+A7Vq+8Y7Ts7hFca0z4Nof7Ne IKDQZzNWTL8fKCJsWq9pYU2pH79pyhfMmb1JBoOFG4Jr2xmpEWzTnfwU6KfAB24ong/q PDcq9pwPsyRN69eIY1CrVDHUJXs8azo8rKsYyYeGc6HAPF0Z2EX560hOlDA1aFq2rDrR uD7Q== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:in-reply-to:content-disposition :mime-version:references:message-id:subject:to:from:date :dkim-signature:dkim-signature:arc-filter:dmarc-filter:delivered-to; bh=kE0N19nzY/cS0AuyPWRvzNzDpaxfMwxOh4gT5ooCYIU=; fh=hPrbWPhweUx4V0GV9uXJqbyAzg2ABmTz7kczrAQqMmM=; b=vdBULujV12LAkJeDhLRu7EiBGMrnvti/ykT3pRk9EnZgQ/wtbzHDwDsoZLJagpoA6m 3tBlksEoP9qREGeuThh7I5coJ+zQ2QrrEQUGwi3FTGksiiEOcFnSanKEg5WxCGgsX5bt kWlXZ6AyuUxC2Xq/2VrjvSnVhK1Eyj6+7ylzWseVzpy/yJi85/yxPfBmfMXvrTeAgWKY mFH08zC9rTOEwJiGhH+m8YQ+YKT01+SivkMIlP+oNR7iIDAhay8U9ez3SoofkJUv3cSP /tP1fdmqrbiWNciCUEq4FwCWtY30nLzq0+Qn07nwx+Zx7Z+xbu7MC/76PsHX9YfuLPxX 2g0Q== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@suse.cz header.s=susede2_rsa header.b=nTE13SSG; dkim=neutral (no key) header.i=@suse.cz; arc=pass (i=1); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (server2.sourceware.org. [2620:52:3:1:0:246e:9693:128c]) by mx.google.com with ESMTPS id a27-20020a05620a16db00b00775cab41f33si2120772qkn.552.2023.11.17.12.17.20 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 17 Nov 2023 12:17:20 -0800 (PST) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) client-ip=2620:52:3:1:0:246e:9693:128c; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.cz header.s=susede2_rsa header.b=nTE13SSG; dkim=neutral (no key) header.i=@suse.cz; arc=pass (i=1); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 1905438582A6 for ; Fri, 17 Nov 2023 20:17:19 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtp-out2.suse.de (smtp-out2.suse.de [IPv6:2001:67c:2178:6::1d]) by sourceware.org (Postfix) with ESMTPS id 11DBB3857B8A for ; Fri, 17 Nov 2023 20:16:55 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 11DBB3857B8A Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: sourceware.org; spf=fail smtp.mailfrom=suse.cz ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 11DBB3857B8A Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2001:67c:2178:6::1d ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1700252216; cv=none; b=C0SiLsSNow5rJzd4inDShNwEiCpdSHfHufOomyy6vxR7kkhdwbNnIwFb+ugCJVSGzQtg+0fkUzdffX9ck0aEqmVgAM2gKyvdWVTbZKXsjsST0BmU0ehB4sd4Ae97jQ3lxTCqd8yqHpPU+eK0fRfSobmv0LD+PgZKpza+DxZ3dkE= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1700252216; c=relaxed/simple; bh=jTltF1SKnS67wrHeAcxxoNnDy4Q7Wd5UYtcG4hFfg8s=; h=DKIM-Signature:DKIM-Signature:Date:From:To:Subject:Message-ID: MIME-Version; b=p/HjXbD8nlqVMHm8KGuM3qQvdiJ41iOyupeQWVLTOYLP6PqyeTeD24o31ZEj0yeHPbjGe6z49ViwVblw0TPjj97yyKbOrlXZpkf+h3Re5K0CuDMvR9QUsCULWX6lzVoh8bmxWdq4+sQ+FvsQCvR6ovo1Tu+pCOW4EZf7Q14Tk8Y= ARC-Authentication-Results: i=1; server2.sourceware.org Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 031DB1F37E for ; Fri, 17 Nov 2023 20:16:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1700252214; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=kE0N19nzY/cS0AuyPWRvzNzDpaxfMwxOh4gT5ooCYIU=; b=nTE13SSGfB7aA4JH3kJAGHfrt8ltDJPtsujkhQoL4irr2o4ThMnzHoP7jywdoTNsUah2il 0ED9BKBe0XNCqtp19jwrtVu5CdILloBUjqr/BN0GjCAvuXohMt985TNFXCU0cUR7K8CAjA MAF0A2aiCLRA2A8nkor0DeMtv7MZu9o= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1700252214; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=kE0N19nzY/cS0AuyPWRvzNzDpaxfMwxOh4gT5ooCYIU=; b=n/LxS6TnIrH79AHCBSa4UJYu62ya15GmDG4TzipB5EMB3UhdupU4tQBiYj+KGSbSiMGEdD QDk+nNYUAPHi0RDA== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id DEB541341F for ; Fri, 17 Nov 2023 20:16:53 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id Pt9zNDXKV2UFGgAAMHmgww (envelope-from ) for ; Fri, 17 Nov 2023 20:16:53 +0000 Date: Fri, 17 Nov 2023 21:16:52 +0100 From: Michal Jires To: gcc-patches@gcc.gnu.org Subject: [PATCH 2/7] lto: Remove random_seed from section name. Message-ID: <1ab98391f1f12962c48d1dd4d7309fc219454c96.1700222403.git.mjires@suse.cz> References: <18cc1c3980551ac1881eea6e78811a629c7baa82.1700222403.git.mjires@suse.cz> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <18cc1c3980551ac1881eea6e78811a629c7baa82.1700222403.git.mjires@suse.cz> Authentication-Results: smtp-out2.suse.de; none X-Spam-Level: X-Spam-Score: -3.27 X-Spamd-Result: default: False [-3.27 / 50.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[gcc-patches@gcc.gnu.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; DKIM_SIGNED(0.00)[suse.cz:s=susede2_rsa,suse.cz:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.19)[-0.929]; MID_CONTAINS_FROM(1.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; BAYES_HAM(-2.99)[99.94%] X-Spam-Status: No, score=-11.5 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, SPF_HELO_NONE, SPF_SOFTFAIL, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1782843693017873047 X-GMAIL-MSGID: 1782843693017873047 Bootstrapped/regtested on x86_64-pc-linux-gnu gcc/ChangeLog: * lto-streamer.cc (lto_get_section_name): Remove random_seed in WPA. --- gcc/lto-streamer.cc | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/gcc/lto-streamer.cc b/gcc/lto-streamer.cc index 4968fd13413..53275e32618 100644 --- a/gcc/lto-streamer.cc +++ b/gcc/lto-streamer.cc @@ -132,11 +132,17 @@ lto_get_section_name (int section_type, const char *name, doesn't confuse the reader with merged sections. For options don't add a ID, the option reader cannot deal with them - and merging should be ok here. */ + and merging should be ok here. + + WPA output is sent to LTRANS directly inside of lto-wrapper, so name + uniqueness for external tools is not needed. + Randomness would inhibit incremental LTO. */ if (section_type == LTO_section_opts) strcpy (post, ""); else if (f != NULL) sprintf (post, "." HOST_WIDE_INT_PRINT_HEX_PURE, f->id); + else if (flag_wpa) + strcpy (post, ".0"); else sprintf (post, "." HOST_WIDE_INT_PRINT_HEX_PURE, get_random_seed (false)); char *res = concat (section_name_prefix, sep, add, post, NULL); From patchwork Fri Nov 17 20:17:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michal Jires X-Patchwork-Id: 166318 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9910:0:b0:403:3b70:6f57 with SMTP id i16csp789346vqn; Fri, 17 Nov 2023 12:17:53 -0800 (PST) X-Google-Smtp-Source: AGHT+IEzF5rfd7qgN+QLjX0i6lRwbcJMxOk8A43R19PnhDcmUo3eb34vVWLRkmj2SNxYyYgkuQRB X-Received: by 2002:a05:6870:8184:b0:1e9:cb91:1b05 with SMTP id k4-20020a056870818400b001e9cb911b05mr313649oae.53.1700252273261; Fri, 17 Nov 2023 12:17:53 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1700252273; cv=pass; d=google.com; s=arc-20160816; b=w7ttJue+2m8gNEbNMzktPS3vAjNG8YsHShTX5sbXUlPHJ+oHqMwknlXPidxj+ShOd3 HJyigPBiBJ13rfJZbOqbrJ6zg6v6YVgdAdh6ABYd8/WOR69hKr7JvRIqEXsuwJ27N7+m JxIPIzV/DQj+YskNzLVHSaSaIxViZ9SKbWfwEWxc6hHsZl0RVEnMCxAh0rRn9Vl1qXqY 8OUp76WbN8a1lTpAXXV1qNq5CfSmdUv+MR/LYVgkQj3giJ45jqSsrWZhkQwyRks0at59 12glcc13fGFLtmCwwPf/+yt9z+btsEjjNr+hTMfpUT8/+6qy/7epFfaVAXxbuv8odPcm ePCA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:in-reply-to:content-disposition :mime-version:references:message-id:subject:to:from:date :dkim-signature:dkim-signature:arc-filter:dmarc-filter:delivered-to; bh=CJ0SS3mli0cQQ4cGzfn3Vow56RLldVPq+z/mLh/ntdw=; fh=hPrbWPhweUx4V0GV9uXJqbyAzg2ABmTz7kczrAQqMmM=; b=DlSJPPPVETM9Kpel7WqnTuAulVef5l6lUVy6K+jYKJ5f1ags+eYZDMGUmCYsHtDoQM xsE0RN2pipPd6YBaXTcyeZ5l1z+aEEle5foK8aY+5VKeYtKx7lc3mAq4YcWMDSrMjWou WVNASJteeW4WD15WidGdzF+dkqxc9iN3RUHgFeG+ZQNnwkWEr1MnNoCO24oAm6lk5MVA R9H9KP0zslftfkWpo8irp8sRFmyxjC6RpprYi5NIPt+MRXzMjEAM42ViznkOer3hXdZV GYAVNfZTZIJa77L2Veqv5ScEBbY1KlzkxI2mIQPwZbyFno+VnuS4qGAddOo9GpOUSkku R+2g== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@suse.cz header.s=susede2_rsa header.b=nizzB5XO; dkim=neutral (no key) header.i=@suse.cz header.s=susede2_ed25519 header.b=nRb18Xd4; arc=pass (i=1); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (server2.sourceware.org. [8.43.85.97]) by mx.google.com with ESMTPS id v8-20020a05622a188800b0041993fce7aasi2347932qtc.94.2023.11.17.12.17.53 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 17 Nov 2023 12:17:53 -0800 (PST) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) client-ip=8.43.85.97; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.cz header.s=susede2_rsa header.b=nizzB5XO; dkim=neutral (no key) header.i=@suse.cz header.s=susede2_ed25519 header.b=nRb18Xd4; arc=pass (i=1); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 3BA54385B504 for ; Fri, 17 Nov 2023 20:17:50 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtp-out1.suse.de (smtp-out1.suse.de [IPv6:2001:67c:2178:6::1c]) by sourceware.org (Postfix) with ESMTPS id 4DA2B385828F for ; Fri, 17 Nov 2023 20:17:04 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 4DA2B385828F Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: sourceware.org; spf=fail smtp.mailfrom=suse.cz ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 4DA2B385828F Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2001:67c:2178:6::1c ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1700252227; cv=none; b=EyYlPC6bzZVJ7YOSe/GCopgvFAFHA8lJRNs1OGl1su8c3Un/kHnBvjQzNyeuI4Olm1G+0JwR9dCA+dv93jhf+qU1wCQkM0mZa8RxqM0JryXZrPtlFih2VahZSukGseCLQG10IDJY0VDTeRaqpF55aNLWOrthVJQcTnN953cYnPg= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1700252227; c=relaxed/simple; bh=N5JWao+n+5bILwHxzEpvWa/msIM8pHKGmLMIihGGAP0=; h=DKIM-Signature:DKIM-Signature:Date:From:To:Subject:Message-ID: MIME-Version; b=UnZ2WcUc7u/ZixVNa5z2BEhDYfClEX0zgGZ1QKcBvY6BegtS6BLsMVQy8eEowBHIiJGvPkF/1DEXXH7DogpL7LvERXoztUoF7Dw8lKTO3C8jKz2mtThnjAe1UFmRncy9hWwTapqEEjUFGSUjG3Ek36rdSFhCR/L9BNR2kl+mMpA= ARC-Authentication-Results: i=1; server2.sourceware.org Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 2C162228FA for ; Fri, 17 Nov 2023 20:17:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1700252223; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=CJ0SS3mli0cQQ4cGzfn3Vow56RLldVPq+z/mLh/ntdw=; b=nizzB5XO7+U5Kv3kXzRHF/LdUJv+GZstXzyHb1nMUcFikI6eGB8GvKPV8ejgSOZSA+y/VR ZJRd4+r69FoLDXLOOAeS0pRQ9MEMapcz5d3mtFwilxMxQ5JjBk+icd/Pe3y4WaCtu01jeL Kzoc09XGxwsDRBuZWKSWlNNcrvNR7yU= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1700252223; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=CJ0SS3mli0cQQ4cGzfn3Vow56RLldVPq+z/mLh/ntdw=; b=nRb18Xd4jUUOVb5Djpm4gNu+gWDPQmfjNowthhHXzC3GHGuDaduaNjrqopGfpg8zdpyG1k 8hTeJMgCE8c0GCDA== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 0BCB81341F for ; Fri, 17 Nov 2023 20:17:03 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id TTEcAT/KV2UcGgAAMHmgww (envelope-from ) for ; Fri, 17 Nov 2023 20:17:03 +0000 Date: Fri, 17 Nov 2023 21:17:01 +0100 From: Michal Jires To: gcc-patches@gcc.gnu.org Subject: [PATCH 3/7] Lockfile. Message-ID: References: <18cc1c3980551ac1881eea6e78811a629c7baa82.1700222403.git.mjires@suse.cz> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <18cc1c3980551ac1881eea6e78811a629c7baa82.1700222403.git.mjires@suse.cz> Authentication-Results: smtp-out1.suse.de; none X-Spam-Level: X-Spam-Score: -3.29 X-Spamd-Result: default: False [-3.29 / 50.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[gcc-patches@gcc.gnu.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; DKIM_SIGNED(0.00)[suse.cz:s=susede2_rsa,suse.cz:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.19)[-0.930]; MID_CONTAINS_FROM(1.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; BAYES_HAM(-3.00)[100.00%] X-Spam-Status: No, score=-11.7 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_SHORT, SPF_HELO_NONE, SPF_SOFTFAIL, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1782843728035130760 X-GMAIL-MSGID: 1782843728035130760 This patch implements lockfile used for incremental LTO. Bootstrapped/regtested on x86_64-pc-linux-gnu gcc/ChangeLog: * Makefile.in: Add lockfile.o. * lockfile.cc: New file. * lockfile.h: New file. --- gcc/Makefile.in | 5 +- gcc/lockfile.cc | 136 ++++++++++++++++++++++++++++++++++++++++++++++++ gcc/lockfile.h | 85 ++++++++++++++++++++++++++++++ 3 files changed, 224 insertions(+), 2 deletions(-) create mode 100644 gcc/lockfile.cc create mode 100644 gcc/lockfile.h diff --git a/gcc/Makefile.in b/gcc/Makefile.in index 7b7a4ff789a..2c527245c81 100644 --- a/gcc/Makefile.in +++ b/gcc/Makefile.in @@ -1831,7 +1831,7 @@ ALL_HOST_BACKEND_OBJS = $(GCC_OBJS) $(OBJS) $(OBJS-libcommon) \ $(OBJS-libcommon-target) main.o c-family/cppspec.o \ $(COLLECT2_OBJS) $(EXTRA_GCC_OBJS) $(GCOV_OBJS) $(GCOV_DUMP_OBJS) \ $(GCOV_TOOL_OBJS) $(GENGTYPE_OBJS) gcc-ar.o gcc-nm.o gcc-ranlib.o \ - lto-wrapper.o collect-utils.o + lto-wrapper.o collect-utils.o lockfile.o # for anything that is shared use the cc1plus profile data, as that # is likely the most exercised during the build @@ -2359,7 +2359,8 @@ collect2$(exeext): $(COLLECT2_OBJS) $(LIBDEPS) CFLAGS-collect2.o += -DTARGET_MACHINE=\"$(target_noncanonical)\" \ @TARGET_SYSTEM_ROOT_DEFINE@ -LTO_WRAPPER_OBJS = lto-wrapper.o collect-utils.o ggc-none.o +LTO_WRAPPER_OBJS = lto-wrapper.o collect-utils.o ggc-none.o lockfile.o + lto-wrapper$(exeext): $(LTO_WRAPPER_OBJS) libcommon-target.a $(LIBDEPS) +$(LINKER) $(ALL_LINKERFLAGS) $(LDFLAGS) -o T$@ \ $(LTO_WRAPPER_OBJS) libcommon-target.a $(LIBS) diff --git a/gcc/lockfile.cc b/gcc/lockfile.cc new file mode 100644 index 00000000000..9440e8938f3 --- /dev/null +++ b/gcc/lockfile.cc @@ -0,0 +1,136 @@ +/* File locking. + Copyright (C) 2009-2023 Free Software Foundation, Inc. + +This file is part of GCC. + +GCC is free software; you can redistribute it and/or modify it under +the terms of the GNU General Public License as published by the Free +Software Foundation; either version 3, or (at your option) any later +version. + +GCC is distributed in the hope that it will be useful, but WITHOUT ANY +WARRANTY; without even the implied warranty of MERCHANTABILITY or +FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License +for more details. + +You should have received a copy of the GNU General Public License +along with GCC; see the file COPYING3. If not see +. */ + +#include "config.h" +#include "system.h" + +#include "lockfile.h" + + +/* Unique write lock. No other lock can be held on this lockfile. + Blocking call. */ +int +lockfile::lock_write () +{ + fd = open (filename.c_str (), O_RDWR | O_CREAT, 0666); + if (fd < 0) + return -1; + +#if HAVE_FCNTL_H + struct flock s_flock; + + s_flock.l_whence = SEEK_SET; + s_flock.l_start = 0; + s_flock.l_len = 0; + s_flock.l_pid = getpid (); + s_flock.l_type = F_WRLCK; + + while (fcntl (fd, F_SETLKW, &s_flock) && errno == EINTR) + continue; +#endif + return 0; +} + +/* Unique write lock. No other lock can be held on this lockfile. + Only locks if this filelock is not locked by any other process. + Return whether locking was successful. */ +int +lockfile::try_lock_write () +{ + fd = open (filename.c_str (), O_RDWR | O_CREAT, 0666); + if (fd < 0) + return -1; + +#if HAVE_FCNTL_H + struct flock s_flock; + + s_flock.l_whence = SEEK_SET; + s_flock.l_start = 0; + s_flock.l_len = 0; + s_flock.l_pid = getpid (); + s_flock.l_type = F_WRLCK; + + if (fcntl (fd, F_SETLK, &s_flock) == -1) + { + close (fd); + fd = -1; + return 1; + } +#endif + return 0; +} + +/* Shared read lock. Only read lock can be held concurrently. + If write lock is already held by this process, it will be + changed to read lock. + Blocking call. */ +int +lockfile::lock_read () +{ + fd = open (filename.c_str (), O_RDWR | O_CREAT, 0666); + if (fd < 0) + return -1; + +#if HAVE_FCNTL_H + struct flock s_flock; + + s_flock.l_whence = SEEK_SET; + s_flock.l_start = 0; + s_flock.l_len = 0; + s_flock.l_pid = getpid (); + s_flock.l_type = F_RDLCK; + + while (fcntl (fd, F_SETLKW, &s_flock) && errno == EINTR) + continue; +#endif + return 0; +} + +/* Unlock all previously placed locks. */ +void +lockfile::unlock () +{ + if (fd < 0) + { +#if HAVE_FCNTL_H + struct flock s_flock; + + s_flock.l_whence = SEEK_SET; + s_flock.l_start = 0; + s_flock.l_len = 0; + s_flock.l_pid = getpid (); + s_flock.l_type = F_UNLCK; + + fcntl (fd, F_SETLK, &s_flock); +#endif + close (fd); + fd = -1; + } +} + +/* Are lockfiles supported? */ +bool +lockfile::lockfile_supported () +{ +#if HAVE_FCNTL_H + return true; +#else + return false; +#endif +} diff --git a/gcc/lockfile.h b/gcc/lockfile.h new file mode 100644 index 00000000000..afcbaf599c1 --- /dev/null +++ b/gcc/lockfile.h @@ -0,0 +1,85 @@ +/* File locking. + Copyright (C) 2009-2023 Free Software Foundation, Inc. + +This file is part of GCC. + +GCC is free software; you can redistribute it and/or modify it under +the terms of the GNU General Public License as published by the Free +Software Foundation; either version 3, or (at your option) any later +version. + +GCC is distributed in the hope that it will be useful, but WITHOUT ANY +WARRANTY; without even the implied warranty of MERCHANTABILITY or +FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License +for more details. + +You should have received a copy of the GNU General Public License +along with GCC; see the file COPYING3. If not see +. */ + +#ifndef LOCKFILE_H +#define LOCKFILE_H + +#include + +/* Used to synchronize across multiple processes. */ +class lockfile { +public: + /* Default constructor. */ + lockfile (): fd (-1) + {} + /* Intended constructor for use. Filename should not be used for anything + other than locking to prevent unintentional unlock. */ + lockfile (std::string filename): lockfile () + { + this->filename = std::move (filename); + } + lockfile (lockfile const& o): lockfile (o.filename) + {} + + void operator=(lockfile o) + { + unlock (); + this->filename = o.filename; + this->fd = o.fd; + o.fd = -1; + } + + /* Unique write lock. No other lock can be held on this lockfile. + Blocking call. */ + int + lock_write (); + + /* Unique write lock. No other lock can be held on this lockfile. + Only locks if this filelock is not locked by any other process. + Return whether locking was successful. */ + int + try_lock_write (); + + /* Shared read lock. Only read lock can be held concurrently. + If write lock is already held by this process, it will be + changed to read lock. + Blocking call. */ + int + lock_read (); + + /* Unlock all previously placed locks. */ + void + unlock (); + + /* Returns whether any lock is held. */ + bool + locked () + { + return fd < 0; + } + + /* Are lockfiles supported? */ + static bool + lockfile_supported (); +private: + std::string filename; + int fd; +}; + +#endif From patchwork Fri Nov 17 20:17:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michal Jires X-Patchwork-Id: 166317 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9910:0:b0:403:3b70:6f57 with SMTP id i16csp789249vqn; Fri, 17 Nov 2023 12:17:42 -0800 (PST) X-Google-Smtp-Source: AGHT+IGt8To0qsXYoB/yOpHGLKwjlIMF3FQfSQOxTmhUgBwi+WG9Xg6297wK+t5GsIWSKbCa7yTN X-Received: by 2002:a05:6830:1da4:b0:6b8:807b:b50 with SMTP id z4-20020a0568301da400b006b8807b0b50mr324489oti.22.1700252262518; Fri, 17 Nov 2023 12:17:42 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1700252262; cv=pass; d=google.com; s=arc-20160816; b=HuDPS1hZTsiW/9uBW+6OOEsFboFQjcI5u4VhCFnJpVkrcjfxenGqrQxpNT3EFrATzr bRSbRQD2P1pgCqu2FIAnrzyRZkKN/0yOEz3JTD8ZiufSrndUPM1QuSFyBeLIy4bxAo8p 3atuFZFjU3PJrK/afIE5xBGh11ay1AXb3qej6cTMytD2omz4vCa3LEEcXY2KDMfkx8lh wtfIGz4u1KI1OWCfnLWwVkLsallsqnvs+NIyLXZFvdN7bBSNThQ+j6AHMfcr4QdvxFhd xMF7Fdf142ahUJx9Sh9X2XlXCHGs7GYPDvqrjHqpBp9hwiEJKcsMoZ4dko445gFNxhBS Qvxw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:in-reply-to:content-disposition :mime-version:references:message-id:subject:to:from:date :dkim-signature:dkim-signature:arc-filter:dmarc-filter:delivered-to; bh=XsBmjZEwUiQ2z2tqsVn7uKN+61p6JBz3Jjuio+NxHDI=; fh=hPrbWPhweUx4V0GV9uXJqbyAzg2ABmTz7kczrAQqMmM=; b=DexlKMuJZCXsg1A7d7z77TxP50FhsmS+yzmZZgFlK9vSDZDxmMKyBhqu07xljlEc4L ywdnWAejzjgg+Olbji9x+0dEAYFpcR/9QpRkMozv3y+EuDObbrFWhwFa3SWmjcxMq9Fn 44glXmTSsIenjJ/dFdjSqvl3B4ZbKMhTYvHOlPsgVHhvWKBl4O3niT9cC5ro6uNZLrnF CWJGWAhBODYxMppH7iuyLLPxuKfONAaZnvxV1KNGr4S7H1ruGkGh+HxI6BRvWbxOMJu+ X4gd6IIQq07cnbHH/xRnC+kKvhfqz10/vV8aHpcCR3v79WUUCAZiN2Y+rn6O99DVAMf+ y5gQ== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@suse.cz header.s=susede2_rsa header.b=Q0wL04W0; dkim=neutral (no key) header.i=@suse.cz header.s=susede2_ed25519 header.b="t/piFz1o"; arc=pass (i=1); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (server2.sourceware.org. [8.43.85.97]) by mx.google.com with ESMTPS id n8-20020a0cfbc8000000b00670b3a8241bsi2293603qvp.350.2023.11.17.12.17.42 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 17 Nov 2023 12:17:42 -0800 (PST) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) client-ip=8.43.85.97; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.cz header.s=susede2_rsa header.b=Q0wL04W0; dkim=neutral (no key) header.i=@suse.cz header.s=susede2_ed25519 header.b="t/piFz1o"; arc=pass (i=1); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 930093857C41 for ; Fri, 17 Nov 2023 20:17:41 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by sourceware.org (Postfix) with ESMTPS id 425E438582B1 for ; Fri, 17 Nov 2023 20:17:14 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 425E438582B1 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=suse.cz ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 425E438582B1 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=195.135.220.29 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1700252237; cv=none; b=t1YXTf9+pz4mnvU4iPQuLO3c+IrLA+SbL8a1hjym3deJAkL6Fq6/ieRFCYrnPEXDUR1inDw0uWFWOw2whYVNLahaoKPBRz0WHnvdnyfRaAlJjRp+/A+ObtNjksihAQ75Bc5yetItI5vORnwzcE8PnTMIawCrlvq3BxbmL8VIdI8= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1700252237; c=relaxed/simple; bh=YxB5uc9+Oq3mieuQ/GCSRk+B+Ri5Oq8QlhG4osN76zE=; h=DKIM-Signature:DKIM-Signature:Date:From:To:Subject:Message-ID: MIME-Version; b=Dk7DBvIwIKdK+E9fmrdsYwLkKxIW1qDo5QKJG8g+jfl4hplpB76H8U/KJxjZHXtFiQfjOJUqlwq+OkTWT6J29YwSHCt7oN3skL+ideM+DaV1Kj9lYSTizsatimq3EYgV5wSlchYIZik9l2zF/sp8c5eGCxtvJH0kJhyw+bhjCtM= ARC-Authentication-Results: i=1; server2.sourceware.org Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 5F7251F37E for ; Fri, 17 Nov 2023 20:17:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1700252233; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=XsBmjZEwUiQ2z2tqsVn7uKN+61p6JBz3Jjuio+NxHDI=; b=Q0wL04W0dfQSclz2MPeTjqTFIbLmWa5iNoeaAWj23shKZeufdBqIDkyAC+4yJ5NBW4Vs23 jolOoBiVq8T77bUgyGbY/dr7g8QTnkGjcGSRaqyVK55k3ZxL6VkGGFEeHLmC+XWewAdkSh AsCPDXN6ujIrkIP12a7sHIkfx5VG+xU= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1700252233; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=XsBmjZEwUiQ2z2tqsVn7uKN+61p6JBz3Jjuio+NxHDI=; b=t/piFz1oRDen5b5iKKIRuZw5n9QZxi95QqayAlIvHMEFNVws6k5tTS4LyuNchj1mHmxOoo DVjJLkFrLMIwXiDg== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 4231F1341F for ; Fri, 17 Nov 2023 20:17:13 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id J6F4DknKV2UvGgAAMHmgww (envelope-from ) for ; Fri, 17 Nov 2023 20:17:13 +0000 Date: Fri, 17 Nov 2023 21:17:11 +0100 From: Michal Jires To: gcc-patches@gcc.gnu.org Subject: [PATCH 4/7] lto: Implement ltrans cache Message-ID: <788aa123a8fd4bbfa8a80eda37fbacf38ec78c9b.1700222403.git.mjires@suse.cz> References: <18cc1c3980551ac1881eea6e78811a629c7baa82.1700222403.git.mjires@suse.cz> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <18cc1c3980551ac1881eea6e78811a629c7baa82.1700222403.git.mjires@suse.cz> Authentication-Results: smtp-out2.suse.de; none X-Spam-Level: X-Spam-Score: -3.29 X-Spamd-Result: default: False [-3.29 / 50.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[gcc-patches@gcc.gnu.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; DKIM_SIGNED(0.00)[suse.cz:s=susede2_rsa,suse.cz:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.19)[-0.937]; MID_CONTAINS_FROM(1.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; BAYES_HAM(-3.00)[100.00%] X-Spam-Status: No, score=-12.3 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_SHORT, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1782843716750708078 X-GMAIL-MSGID: 1782843716750708078 This patch implements Incremental LTO as ltrans cache. The cache is active when directory $GCC_LTRANS_CACHE is specified and exists. Stored are pairs of ltrans input/output files and input file hash. File locking is used to allow multiple GCC instances to use to same cache. Bootstrapped/regtested on x86_64-pc-linux-gnu gcc/ChangeLog: * Makefile.in: Add lto-ltrans-cache.o. * lto-wrapper.cc: Use ltrans cache. * lto-ltrans-cache.cc: New file. * lto-ltrans-cache.h: New file. --- gcc/Makefile.in | 5 +- gcc/lto-ltrans-cache.cc | 407 ++++++++++++++++++++++++++++++++++++++++ gcc/lto-ltrans-cache.h | 164 ++++++++++++++++ gcc/lto-wrapper.cc | 150 +++++++++++++-- 4 files changed, 711 insertions(+), 15 deletions(-) create mode 100644 gcc/lto-ltrans-cache.cc create mode 100644 gcc/lto-ltrans-cache.h diff --git a/gcc/Makefile.in b/gcc/Makefile.in index 2c527245c81..495e5f3d069 100644 --- a/gcc/Makefile.in +++ b/gcc/Makefile.in @@ -1831,7 +1831,7 @@ ALL_HOST_BACKEND_OBJS = $(GCC_OBJS) $(OBJS) $(OBJS-libcommon) \ $(OBJS-libcommon-target) main.o c-family/cppspec.o \ $(COLLECT2_OBJS) $(EXTRA_GCC_OBJS) $(GCOV_OBJS) $(GCOV_DUMP_OBJS) \ $(GCOV_TOOL_OBJS) $(GENGTYPE_OBJS) gcc-ar.o gcc-nm.o gcc-ranlib.o \ - lto-wrapper.o collect-utils.o lockfile.o + lto-wrapper.o collect-utils.o lockfile.o lto-ltrans-cache.o # for anything that is shared use the cc1plus profile data, as that # is likely the most exercised during the build @@ -2359,7 +2359,8 @@ collect2$(exeext): $(COLLECT2_OBJS) $(LIBDEPS) CFLAGS-collect2.o += -DTARGET_MACHINE=\"$(target_noncanonical)\" \ @TARGET_SYSTEM_ROOT_DEFINE@ -LTO_WRAPPER_OBJS = lto-wrapper.o collect-utils.o ggc-none.o lockfile.o +LTO_WRAPPER_OBJS = lto-wrapper.o collect-utils.o ggc-none.o lockfile.o \ + lto-ltrans-cache.o lto-wrapper$(exeext): $(LTO_WRAPPER_OBJS) libcommon-target.a $(LIBDEPS) +$(LINKER) $(ALL_LINKERFLAGS) $(LDFLAGS) -o T$@ \ diff --git a/gcc/lto-ltrans-cache.cc b/gcc/lto-ltrans-cache.cc new file mode 100644 index 00000000000..0d43e548fb3 --- /dev/null +++ b/gcc/lto-ltrans-cache.cc @@ -0,0 +1,407 @@ +/* File caching. + Copyright (C) 2009-2023 Free Software Foundation, Inc. + +This file is part of GCC. + +GCC is free software; you can redistribute it and/or modify it under +the terms of the GNU General Public License as published by the Free +Software Foundation; either version 3, or (at your option) any later +version. + +GCC is distributed in the hope that it will be useful, but WITHOUT ANY +WARRANTY; without even the implied warranty of MERCHANTABILITY or +FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License +for more details. + +You should have received a copy of the GNU General Public License +along with GCC; see the file COPYING3. If not see +. */ + +#include "config.h" +#include "system.h" +#include "md5.h" +#include "lto-ltrans-cache.h" + +#include +#include +#include + +const md5_checksum_t INVALID_CHECKSUM = { + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, +}; + +/* Computes checksum for given file, returns INVALID_CHECKSUM if not possible. + */ +static md5_checksum_t +file_checksum (char const *filename) +{ + FILE *file = fopen (filename, "rb"); + + if (!file) + return INVALID_CHECKSUM; + + md5_checksum_t result; + + int ret = md5_stream (file, &result); + + if (ret) + result = INVALID_CHECKSUM; + + fclose (file); + + return result; +} + +/* Checks identity of two files byte by byte. */ +static bool +files_identical (char const *first_filename, char const *second_filename) +{ + FILE *f_first = fopen (first_filename, "rb"); + if (!f_first) + return false; + + FILE *f_second = fopen (second_filename, "rb"); + if (!f_second) + { + fclose (f_first); + return false; + } + + bool ret = true; + + for (;;) + { + int c1, c2; + c1 = fgetc (f_first); + c2 = fgetc (f_second); + + if (c1 != c2) + { + ret = false; + break; + } + + if (c1 == EOF) + break; + } + + fclose (f_first); + fclose (f_second); + return ret; +} + +/* Contructor of cache item. */ +ltrans_file_cache::item::item (std::string input, std::string output, + md5_checksum_t input_checksum, uint32_t last_used): + input (std::move (input)), output (std::move (output)), + input_checksum (input_checksum), last_used (last_used) +{ + lock = lockfile (this->input + ".lock"); +} +/* Destructor of cache item. */ +ltrans_file_cache::item::~item () +{ + lock.unlock (); +} + +/* Reads next cache item from cachedata file. + Adds `dir/` prefix to filenames. */ +static ltrans_file_cache::item* +read_cache_item (FILE* f, const char* dir) +{ + md5_checksum_t checksum; + uint32_t last_used; + + if (fread (&checksum, 1, checksum.size (), f) != checksum.size ()) + return NULL; + if (fread (&last_used, sizeof (last_used), 1, f) != 1) + return NULL; + + std::vector input (strlen (dir)); + memcpy (&input[0], dir, input.size ()); + input.push_back ('/'); + std::vector output = input; /* Copy. */ + + int c; + while ((c = getc (f))) + { + if (c == EOF) + return NULL; + input.push_back (c); + } + input.push_back (0); + while ((c = getc (f))) + { + if (c == EOF) + return NULL; + output.push_back (c); + } + output.push_back (0); + + return new ltrans_file_cache::item (&input[0], &output[0], checksum, + last_used); +} + +/* Writes cache item to cachedata file. + Removes `dir/` prefix from filenames. */ +static void +write_cache_item (FILE* f, ltrans_file_cache::item *item, const char* dir) +{ + fwrite (&item->input_checksum, 1, item->input_checksum.size (), f); + fwrite (&item->last_used, sizeof (item->last_used), 1, f); + + gcc_assert (item->input.size () > strlen (dir)); + fputs (item->input.c_str () + strlen (dir) + 1, f); + fputc (0, f); + + gcc_assert (item->output.size () > strlen (dir)); + fputs (item->output.c_str () + strlen (dir) + 1, f); + fputc (0, f); +} + +/* Constructor. Resulting cache item filenames will be + in format `prefix%d[.ltrans]suffix`. */ +ltrans_file_cache::ltrans_file_cache (const char* dir, const char* prefix, + const char* suffix) +{ + this->dir = dir; + if (!dir) return; + + creation_lock = lockfile (std::string (dir) + "/lockfile_creation"); + deletion_lock = lockfile (std::string (dir) + "/lockfile_deletion"); + + soft_cache_size = 2048; + + cache_prefix = std::string (dir) + "/" + prefix; + cache_free_idx = 0; + + this->prefix = prefix; + this->suffix = suffix; + + str_buffer = (char *)xmalloc (cache_prefix.size () + + sizeof ("4294967295.ltrans") + + strlen (suffix)); +} + +/* Destructor. */ +ltrans_file_cache::~ltrans_file_cache () +{ + if (!*this) + return; + + cleanup (); + free (str_buffer); +} + +/* Adds given cache item to all relevant datastructures. */ +void +ltrans_file_cache::add_item (ltrans_file_cache::item* item) +{ + items.push_back (item); + map_checksum[item->input_checksum] = item; + map_input[item->input] = item; + + usage_counter = std::max (usage_counter, item->last_used); +} + +/* Creates cachedata filename for save/load. */ +std::string +ltrans_file_cache::filename_cachedata () +{ + return std::string (dir) + "/cachedata"; +} + +/* Loads data about previously cached items from cachedata file. + Sorts items by last_used and remaps last_used to small integers. + + Must be called with creation_lock or deletion_lock held to + prevent data race. */ +void +ltrans_file_cache::load_cache () +{ + cleanup (); + + std::string filename = filename_cachedata (); + FILE *file = fopen (filename.c_str (), "rb"); + + if (!file) + return; + + ltrans_file_cache::item* _item; + while ((_item = read_cache_item (file, dir))) + add_item (_item); + + fclose (file); + + std::sort (items.begin (), items.end (), + [](item* a, item* b) + {return a->last_used < b->last_used;}); + + for (size_t i = 0; i < items.size (); ++i) + items[i]->last_used = (uint32_t) i; + usage_counter = (uint32_t) items.size (); +} + +/* Rewrites data about cache items into cachedata file. + + Must be only called when creation_lock or deletion_lock was held since last + call to load_cache. */ +void +ltrans_file_cache::save_cache () +{ + std::string filename = filename_cachedata (); + FILE *file = fopen (filename.c_str (), "wb"); + + if (!file) + return; + + for (item* _item: items) + write_cache_item (file, _item, dir); + + fclose (file); +} + +/* Creates new cache item with given checksum. + New input/output files are chosen to not collide with other items. + + Must be called with creation_lock held to prevent data race. */ +ltrans_file_cache::item* +ltrans_file_cache::create_item (md5_checksum_t checksum) +{ + size_t prefix_len = cache_prefix.size (); + + strcpy (str_buffer, cache_prefix.c_str ()); + + for (;;) + { + sprintf (str_buffer + prefix_len, "%04u%s", cache_free_idx, suffix); + + if (map_input.find (str_buffer) == map_input.end ()) + break; + cache_free_idx++; + } + + std::string input = str_buffer; + + sprintf (str_buffer + prefix_len, "%04u.ltrans%s", cache_free_idx, suffix); + + return new item (std::move (input), str_buffer, checksum, usage_counter++); +} + +/* Adds input file into cache. Cache item with input file identical to + added input file will be returned as _item. + If the file was already cached, `true` is returned, `false` otherwise. + The added input file is deleted (or moved). + + Must be called with creation_lock held to prevent data race. */ +bool +ltrans_file_cache::add_to_cache (const char* filename, item*& _item) +{ + md5_checksum_t checksum = file_checksum (filename); + + auto it = map_checksum.find (checksum); + + if (it != map_checksum.end () + && files_identical (filename, it->second->input.c_str ())) + { + unlink (filename); + _item = it->second; + _item->last_used = usage_counter++; + return true; + } + else + { + /* Cache miss. Move into cache dir. */ + _item = create_item (checksum); + add_item (_item); + + rename (filename, _item->input.c_str ()); + return false; + } +} + +/* If exists, returns cache item corresponding to cached input file. */ +ltrans_file_cache::item* +ltrans_file_cache::get_item (const char* input) +{ + auto it = map_input.find (input); + if (it == map_input.end ()) + return NULL; + return it->second; +} + +/* If no other process holds the deletion_lock, prunes oldest unused cache + items over limit. */ +void +ltrans_file_cache::try_prune () +{ + if (deletion_lock.try_lock_write () == 0) + { + prune (); + deletion_lock.unlock (); + } +} + +/* Returns true if file exists. */ +static int +file_exists (char const *name) +{ + return access (name, R_OK) == 0; +} + +/* Prunes oldest unused cache items over limit. + Must be called with deletion_lock held to prevent data race. */ +void +ltrans_file_cache::prune () +{ + load_cache (); + if (items.size () > soft_cache_size) + { + std::vector sorted_items = std::move (items); + + cleanup (); + + for (size_t i = 0; i < sorted_items.size (); ++i) + { + ltrans_file_cache::item* item = sorted_items[i]; + if ((i < sorted_items.size () - soft_cache_size) + || !file_exists (item->input.c_str ()) + || !file_exists (item->output.c_str ())) + { + unlink (item->input.c_str ()); + unlink (item->output.c_str ()); + delete item; + } + else + add_item (item); + } + } + save_cache (); +} + +/* Clears cache class, as if only constructor was called. */ +void +ltrans_file_cache::cleanup () +{ + map_checksum.clear (); + map_input.clear (); + + for (ltrans_file_cache::item* item: items) + delete item; + items.clear (); + + usage_counter = 0; +} + + +/* Returns ltrans cache dir. + Returns NULL if ltrans cache is disabled. */ +const char* +get_ltrans_cache_dir () +{ + const char *ltrans_cache_dir = getenv ("GCC_LTRANS_CACHE"); + if (!ltrans_cache_dir || ltrans_cache_dir[0] == '\0' + || !file_exists (ltrans_cache_dir)) + ltrans_cache_dir = NULL; + return ltrans_cache_dir; +} diff --git a/gcc/lto-ltrans-cache.h b/gcc/lto-ltrans-cache.h new file mode 100644 index 00000000000..763a23635f7 --- /dev/null +++ b/gcc/lto-ltrans-cache.h @@ -0,0 +1,164 @@ +/* File caching. + Copyright (C) 2009-2023 Free Software Foundation, Inc. + +This file is part of GCC. + +GCC is free software; you can redistribute it and/or modify it under +the terms of the GNU General Public License as published by the Free +Software Foundation; either version 3, or (at your option) any later +version. + +GCC is distributed in the hope that it will be useful, but WITHOUT ANY +WARRANTY; without even the implied warranty of MERCHANTABILITY or +FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License +for more details. + +You should have received a copy of the GNU General Public License +along with GCC; see the file COPYING3. If not see +. */ + +#ifndef GCC_LTO_LTRANS_CACHE_H +#define GCC_LTO_LTRANS_CACHE_H + +#include +#include +#include +#include +#include + +#include "lockfile.h" + +using md5_checksum_t = std::array; + +class ltrans_file_cache +{ +public: + /* Cache item representing input/output filename pair. */ + struct item + { + item (std::string input, std::string output, + md5_checksum_t input_checksum, uint32_t last_used); + ~item (); + + /* Full path to input filename. */ + const std::string input; + /* Full path to output filename. */ + const std::string output; + /* Checksum of input file. */ + const md5_checksum_t input_checksum; + + /* Larger last_used corresponds to later usage. */ + uint32_t last_used; + + /* Lockfile so that output file can be created later than input file. */ + lockfile lock; + }; + + /* Constructor. Resulting cache item filenames will be + in format `prefix%d[.ltrans]suffix`. */ + ltrans_file_cache (const char* dir, const char* prefix, const char* suffix); + /* Destructor. */ + ~ltrans_file_cache (); + + /* Loads data about previously cached items from cachedata file. + + Must be called with creation_lock or deletion_lock held to + prevent data race. */ + void + load_cache (); + + /* Rewrites data about cache items into cachedata file. + + Must be only called when creation_lock or deletion_lock was held since last + call to load_cache. */ + void + save_cache (); + + + /* Adds input file into cache. Cache item with input file identical to + added input file will be returned as _item. + If the file was already cached, `true` is returned, `false` otherwise. + The added input file is deleted (or moved). + + Must be called with creation_lock held to prevent data race. */ + bool + add_to_cache (const char* filename, item*& _item); + + /* If exists, returns cache item corresponding to cached input file. */ + item* + get_item (const char* input); + + /* If no other process holds the deletion_lock, prunes oldest unused cache + items over limit. */ + void + try_prune (); + + /* Clears cache class, as if only constructor was called. */ + void + cleanup (); + + /* Cache is enabled if true. */ + operator bool () + { + return dir; + } + + /* Access to already created items can be concurrent with item creation. */ + lockfile creation_lock; + /* Access to already created items cannot be concurrent + with item deletion. */ + lockfile deletion_lock; + + /* Directory of cache. NULL if cache is disabled. */ + const char* dir; +private: + /* Adds given cache item to all relevant datastructures. */ + void + add_item (item* item); + + /* Creates new cache item with given checksum. + New input/output files are chosen to not collide with other items. + + Must be called with creation_lock held to prevent data race. */ + item* + create_item (md5_checksum_t checksum); + + /* Prunes oldest unused cache items over limit. + Must be called with deletion_lock held to prevent data race. */ + void + prune (); + + /* Creates cachedata filename for save/load. */ + std::string + filename_cachedata (); + + /* All cache items in current cache. */ + std::vector items; + std::map map_checksum; + std::map map_input; + + /* Cached filenames are in format "prefix%d[.ltrans]suffix". */ + const char* prefix; + const char* suffix; + + /* If cache items count is larger, prune deletes old items. */ + size_t soft_cache_size; + + /* Counter used to populate last_used of items. */ + uint32_t usage_counter; + + /* String in format "dir/prefix". */ + std::string cache_prefix; + /* Lower indices are occupied. */ + uint32_t cache_free_idx; + + /* Buffer for sprintf. */ + char* str_buffer; +}; + +/* Returns ltrans cache dir. + Returns NULL if ltrans cache is disabled. */ +const char* +get_ltrans_cache_dir (); + +#endif diff --git a/gcc/lto-wrapper.cc b/gcc/lto-wrapper.cc index 5186d040ce0..4b98539067e 100644 --- a/gcc/lto-wrapper.cc +++ b/gcc/lto-wrapper.cc @@ -52,6 +52,7 @@ along with GCC; see the file COPYING3. If not see #include "opts-diagnostic.h" #include "opt-suggestions.h" #include "opts-jobserver.h" +#include "lto-ltrans-cache.h" /* Environment variable, used for passing the names of offload targets from GCC driver to lto-wrapper. */ @@ -79,7 +80,7 @@ static char *flto_out; static unsigned int nr; static int *ltrans_priorities; static char **input_names; -static char **output_names; +static char const**output_names; static char **offload_names; static char *offload_objects_file_name; static char *makefile; @@ -1802,6 +1803,8 @@ cont1: } } + const char *ltrans_cache_dir = get_ltrans_cache_dir (); + /* If object files contain offload sections, but do not contain LTO sections, then there is no need to perform a link-time recompilation, i.e. lto-wrapper is used only for a compilation of offload images. */ @@ -1827,7 +1830,17 @@ cont1: char *dumpbase = concat (dumppfx, "wpa", NULL); obstack_ptr_grow (&argv_obstack, dumpbase); - if (save_temps) + if (ltrans_cache_dir) + { + /* Results of wpa phase must be on the same disk partition as + cache. */ + char* file = concat (ltrans_cache_dir, "/ccXXXXXX.ltrans.out", NULL); + int fd = mkstemps (file, strlen (".ltrans.out")); + gcc_assert (fd != -1 && !close (fd)); + + ltrans_output_file = file; + } + else if (save_temps) ltrans_output_file = concat (dumppfx, "ltrans.out", NULL); else ltrans_output_file = make_temp_file (".ltrans.out"); @@ -1953,7 +1966,8 @@ cont: ltrans_priorities = (int *)xrealloc (ltrans_priorities, nr * sizeof (int) * 2); input_names = (char **)xrealloc (input_names, nr * sizeof (char *)); - output_names = (char **)xrealloc (output_names, nr * sizeof (char *)); + output_names = (char const**) + xrealloc (output_names, nr * sizeof (char const*)); ltrans_priorities[(nr-1)*2] = priority; ltrans_priorities[(nr-1)*2+1] = nr-1; input_names[nr-1] = input_name; @@ -1985,21 +1999,79 @@ cont: qsort (ltrans_priorities, nr, sizeof (int) * 2, cmp_priority); } + ltrans_file_cache ltrans_cache (ltrans_cache_dir, "ltrans", ".o"); + + if (ltrans_cache) + { + if (!lockfile::lockfile_supported ()) + { + warning (0, "using ltrans cache without file locking support," + " do not use in parallel"); + } + ltrans_cache.deletion_lock.lock_read (); + ltrans_cache.creation_lock.lock_write (); + + ltrans_cache.load_cache (); + + int recompiling = 0; + + for (i = 0; i < nr; ++i) + { + /* If it's a pass-through file do nothing. */ + if (output_names[i]) + continue; + + ltrans_file_cache::item* item; + bool existed = ltrans_cache.add_to_cache (input_names[i], item); + free (input_names[i]); + input_names[i] = xstrdup (item->input.c_str ()); + + if (existed) + { + /* Fill the output_name to skip compilation. */ + output_names[i] = item->output.c_str (); + recompiling++; + } + else + { + /* Lock so no other process can access until the file is + compiled. */ + item->lock.lock_write (); + } + } + if (verbose) + fprintf (stderr, "LTRANS: recompiling %d/%d\n", recompiling, nr); + + ltrans_cache.save_cache (); + ltrans_cache.creation_lock.unlock (); + } + /* Execute the LTRANS stage for each input file (or prepare a makefile to invoke this in parallel). */ for (i = 0; i < nr; ++i) { - char *output_name; + char const* output_name; char *input_name = input_names[i]; - /* If it's a pass-through file do nothing. */ + /* If it's a pass-through or cached file do nothing. */ if (output_names[i]) continue; - /* Replace the .o suffix with a .ltrans.o suffix and write - the resulting name to the LTRANS output list. */ - obstack_grow (&env_obstack, input_name, strlen (input_name) - 2); - obstack_grow (&env_obstack, ".ltrans.o", sizeof (".ltrans.o")); - output_name = XOBFINISH (&env_obstack, char *); + if (ltrans_cache) + { + ltrans_file_cache::item* item; + item = ltrans_cache.get_item (input_name); + gcc_assert (item); + + output_name = item->output.c_str (); + } + else + { + /* Replace the .o suffix with a .ltrans.o suffix and write + the resulting name to the LTRANS output list. */ + obstack_grow (&env_obstack, input_name, strlen (input_name) - 2); + obstack_grow (&env_obstack, ".ltrans.o", sizeof (".ltrans.o")); + output_name = XOBFINISH (&env_obstack, char const*); + } /* Adjust the dumpbase if the linker output file was seen. */ int dumpbase_len = (strlen (dumppfx) @@ -2023,7 +2095,7 @@ cont: /* If we are not preserving the ltrans input files then truncate them as soon as we have processed it. This reduces temporary disk-space usage. */ - if (! save_temps) + if (!ltrans_cache && !save_temps) fprintf (mstream, "\t@-touch -r \"%s\" \"%s.tem\" > /dev/null " "2>&1 && mv \"%s.tem\" \"%s\"\n", input_name, input_name, input_name, input_name); @@ -2038,7 +2110,8 @@ cont: "ltrans%u.ltrans_args", i); fork_execute (new_argv[0], CONST_CAST (char **, new_argv), true, save_temps ? argsuffix : NULL); - maybe_unlink (input_name); + if (!ltrans_cache) + maybe_unlink (input_names[i]); } output_names[i] = output_name; @@ -2093,15 +2166,66 @@ cont: freeargv (make_argv); maybe_unlink (makefile); makefile = NULL; + + if (!ltrans_cache) + for (i = 0; i < nr; ++i) + maybe_unlink (input_names[i]); + } + + if (ltrans_cache) + { for (i = 0; i < nr; ++i) - maybe_unlink (input_names[i]); + { + char *input_name = input_names[i]; + char const *output_name = output_names[i]; + + ltrans_file_cache::item* item; + item = ltrans_cache.get_item (input_name); + + if (item && !save_temps) + { + item->lock.lock_read (); + /* Ensure that cached compiled file is not deleted. + Create copy. */ + + obstack_grow (&env_obstack, output_name, + strlen (output_name) - 2); + obstack_grow (&env_obstack, ".cache_copy.XXX.o", + sizeof (".cache_copy.XXX.o")); + + char* output_name_link = XOBFINISH (&env_obstack, char *); + char* name_idx = output_name_link + strlen (output_name_link) + - strlen ("XXX.o"); + + /* lto-wrapper can run in parallel and access + the same partition. */ + for (int j = 0; ; j++) + { + gcc_assert (j < 1000); + sprintf (name_idx, "%03d.o", j); + + if (link (output_name, output_name_link) != EEXIST) + break; + } + + output_names[i] = output_name_link; + item->lock.unlock (); + } + } + + ltrans_cache.deletion_lock.unlock (); } + for (i = 0; i < nr; ++i) { fputs (output_names[i], stdout); putc ('\n', stdout); free (input_names[i]); } + + if (ltrans_cache && !save_temps) + ltrans_cache.try_prune (); + if (!skip_debug) { for (i = 0; i < ltoobj_argc; ++i) From patchwork Fri Nov 17 20:17:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michal Jires X-Patchwork-Id: 166321 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9910:0:b0:403:3b70:6f57 with SMTP id i16csp789500vqn; Fri, 17 Nov 2023 12:18:12 -0800 (PST) X-Google-Smtp-Source: AGHT+IFXfyE3ladujf1NRIw3HFdnLMZ71YTh+hjFS72X6PHYfOrUE+f3b54pGtDCxnY6Lpyh9fv6 X-Received: by 2002:a05:622a:1487:b0:421:c7a8:d376 with SMTP id t7-20020a05622a148700b00421c7a8d376mr989550qtx.6.1700252291967; Fri, 17 Nov 2023 12:18:11 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1700252291; cv=pass; d=google.com; s=arc-20160816; b=Uabo8s1CQRQmL4MQM/Et0FLt3W67E5/YieUyX9uNoShwZ3E8mgf73/Iy+ABoYeSkUH XP95xx57ujHnPfc1r0ArSNNVLhXe44XGQkOBRzSNp50CcMtV5/yuKtOtmpLZv4PY5rxQ OyqthuaVSFR0VKU0ExMr4zhaiLHj5dd4/MdgNXL6ilnwLKO0yJJ6Ogr96eb0FOJru+IW yEoQOvvkO8KKCvVRGiM8THv8Xirz7BKUxqXWTOyYPpwVesgjQa5HgnYPFEgIgJdE/giy U13UFaUj3sGMORClTzvefx2AGWC+SnaWLe1qxR23mZjg9CBN4bdGvjkN3VVGDcRORs+h lOwg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:in-reply-to:content-disposition :mime-version:references:message-id:subject:to:from:date :dkim-signature:dkim-signature:arc-filter:dmarc-filter:delivered-to; bh=XSsGdXHRdaqaF6CdLjvyecWulau8qQsK5IAdmEuwIrU=; fh=hPrbWPhweUx4V0GV9uXJqbyAzg2ABmTz7kczrAQqMmM=; b=QgknoPuaVZAtBXZPHhn+7gVp+4tdrV1VFQORvK71ZdJd+yHLjEeFpaBWNIQV3ahO71 JBM72yew8PX+8k0qaFYfuofVdjE911t5LrvjU532leZw2zKKdFlhF4aEgJUn8ppI7nD3 /XTUWmO0YNBakPYPwlnoSAoG0q8yZ2LtRjIrfg+79lGhCa4YwWv5+o7Xi1X4zKspQhsk Vldc/ZWpKMykOXMvUvflzTzQGtde0SDEF3y5a8me8jgSewO5aQFl8eLBcdIWIBHSnFfG VoHq04jGEqFDVQvPcBpYQ1mjcddB/7Br28bz0eAr9ZNrwFFQ1AHaslq+dZOlSkSR3BG2 SGWw== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@suse.cz header.s=susede2_rsa header.b=hWcTk2XG; dkim=neutral (no key) header.i=@suse.cz header.s=susede2_ed25519 header.b=rUsxnIc4; arc=pass (i=1); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (server2.sourceware.org. [8.43.85.97]) by mx.google.com with ESMTPS id e6-20020ac85986000000b0041cba17e38fsi2296349qte.276.2023.11.17.12.18.11 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 17 Nov 2023 12:18:11 -0800 (PST) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) client-ip=8.43.85.97; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.cz header.s=susede2_rsa header.b=hWcTk2XG; dkim=neutral (no key) header.i=@suse.cz header.s=susede2_ed25519 header.b=rUsxnIc4; arc=pass (i=1); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id B9176385841F for ; Fri, 17 Nov 2023 20:18:11 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by sourceware.org (Postfix) with ESMTPS id 8EF973858C39 for ; Fri, 17 Nov 2023 20:17:21 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 8EF973858C39 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=suse.cz ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 8EF973858C39 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=195.135.220.28 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1700252244; cv=none; b=vtJJKv+NGBOkk3QBn2WX5CGE8VK/mHqK/nwyvQ74wL2QGdYO1zuJ34uLLagrd9jJvkXlFhymnqoy7Qy1ALBgP81k7+Lpt0YPtC+4XerHiqq52/4HNzFr+/GDHaH+hnwcJKvbTDlW540zd3PmHZoaYyjVQ5j4Ltiga8p3eiJdYSA= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1700252244; c=relaxed/simple; bh=Keg7CIsJeGmZean8ELawaO5oEK83rIxCB1P1mNTrlwE=; h=DKIM-Signature:DKIM-Signature:Date:From:To:Subject:Message-ID: MIME-Version; b=bt4jZlBcjZcGcyCzKfOj5yQMfSjrJ1Tdg0o7ZCz6xGXB3kE9sOR9yjnQM8zVc1SPMzmE9qvhx3mHxdC/R3ELMMrd37Zo9fvV5oFAarlR1iucJDRnLnh6XmE/AeswmTZkJsaEXy8uvz7CY6nd2zgYGUEvBHtRDPMN7X/MkBOmBYc= ARC-Authentication-Results: i=1; server2.sourceware.org Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 7D214228FA for ; Fri, 17 Nov 2023 20:17:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1700252240; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=XSsGdXHRdaqaF6CdLjvyecWulau8qQsK5IAdmEuwIrU=; b=hWcTk2XGs3dO+NBRmi7ZG9MiKSWAnmRTBpa6QUmx3eZI+MqShPpMlkdEwB7tOu68f1kjuP EOPRRobMzB+xpe5zWiSQSY7y229efL+Gqpcn/RSdiycdtebrv7ao0ptiu83TyTC5tNiv6L tCdKsL+d8JdrzMQ1zzq95FXvYDx85Ro= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1700252240; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=XSsGdXHRdaqaF6CdLjvyecWulau8qQsK5IAdmEuwIrU=; b=rUsxnIc4Q4OTp6bW5yuWtWoisVZp12xlSIP9XzerTGWpR5IOdqWIhjHIeWfH11TkxYbgGg c1ftXr7LK1qndpAA== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 551EB1341F for ; Fri, 17 Nov 2023 20:17:20 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id OGkGE1DKV2VFGgAAMHmgww (envelope-from ) for ; Fri, 17 Nov 2023 20:17:20 +0000 Date: Fri, 17 Nov 2023 21:17:18 +0100 From: Michal Jires To: gcc-patches@gcc.gnu.org Subject: [PATCH 5/7] lto: Implement cache partitioning Message-ID: <6d4cad01621dec0d30d7e4565469f440c87cb588.1700222403.git.mjires@suse.cz> References: <18cc1c3980551ac1881eea6e78811a629c7baa82.1700222403.git.mjires@suse.cz> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <18cc1c3980551ac1881eea6e78811a629c7baa82.1700222403.git.mjires@suse.cz> Authentication-Results: smtp-out1.suse.de; none X-Spam-Level: X-Spam-Score: -3.30 X-Spamd-Result: default: False [-3.30 / 50.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[gcc-patches@gcc.gnu.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; DKIM_SIGNED(0.00)[suse.cz:s=susede2_rsa,suse.cz:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.20)[-0.976]; MID_CONTAINS_FROM(1.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; BAYES_HAM(-3.00)[100.00%] X-Spam-Status: No, score=-11.7 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_STOCKGEN, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1782843747114861107 X-GMAIL-MSGID: 1782843747114861107 This patch implements new cache partitioning. It tries to keep symbols from single source file together to minimize propagation of divergence. It starts with symbols already grouped by source files. If reasonably possible it only either combines several files into one final partition, or, if a file is large, split the file into several final partitions. Intermediate representation is partition_set which contains set of groups of symbols (each group corresponding to original source file) and number of final partitions this partition_set should split into. First partition_fixed_split splits partition_set into constant number of partition_sets with equal number of symbols groups. If for example there are 39 source files, the resulting partition_sets will contain 10, 10, 10, and 9 source files. This splitting intentionally ignores estimated instruction counts to minimize propagation of divergence. Second partition_over_target_split separates too large files and splits them into individual symbols to be combined back into several smaller files in next step. Third partition_binary_split splits partition_set into two halves until it should be split into only one final partition, at which point the remaining symbols are joined into one final partition. Bootstrapped/regtested on x86_64-pc-linux-gnu gcc/ChangeLog: * common.opt: Add cache partitioning. * flag-types.h (enum lto_partition_model): Likewise. gcc/lto/ChangeLog: * lto-partition.cc (new_partition): Use new_partition_no_push. (new_partition_no_push): New. (free_ltrans_partition): New. (free_ltrans_partitions): Use free_ltrans_partition. (join_partitions): New. (split_partition_into_nodes): New. (is_partition_reorder): New. (class partition_set): New. (distribute_n_partitions): New. (partition_over_target_split): New. (partition_binary_split): New. (partition_fixed_split): New. (class partitioner_base): New. (class partitioner_default): New. (lto_cache_map): New. * lto-partition.h (lto_cache_map): New. * lto.cc (do_whole_program_analysis): Use lto_cache_map. gcc/testsuite/ChangeLog: * gcc.dg/completion-2.c: Add -flto-partition=cache. --- gcc/common.opt | 3 + gcc/flag-types.h | 3 +- gcc/lto/lto-partition.cc | 605 +++++++++++++++++++++++++++- gcc/lto/lto-partition.h | 1 + gcc/lto/lto.cc | 2 + gcc/testsuite/gcc.dg/completion-2.c | 1 + 6 files changed, 605 insertions(+), 10 deletions(-) diff --git a/gcc/common.opt b/gcc/common.opt index 1cf3bdd3b51..fe5cf3c0a05 100644 --- a/gcc/common.opt +++ b/gcc/common.opt @@ -2174,6 +2174,9 @@ Enum(lto_partition_model) String(1to1) Value(LTO_PARTITION_1TO1) EnumValue Enum(lto_partition_model) String(max) Value(LTO_PARTITION_MAX) +EnumValue +Enum(lto_partition_model) String(cache) Value(LTO_PARTITION_CACHE) + flto-partition= Common Joined RejectNegative Enum(lto_partition_model) Var(flag_lto_partition) Init(LTO_PARTITION_BALANCED) Specify the algorithm to partition symbols and vars at linktime. diff --git a/gcc/flag-types.h b/gcc/flag-types.h index c1852cd810c..59b3c23081b 100644 --- a/gcc/flag-types.h +++ b/gcc/flag-types.h @@ -393,7 +393,8 @@ enum lto_partition_model { LTO_PARTITION_ONE = 1, LTO_PARTITION_BALANCED = 2, LTO_PARTITION_1TO1 = 3, - LTO_PARTITION_MAX = 4 + LTO_PARTITION_MAX = 4, + LTO_PARTITION_CACHE = 5 }; /* flag_lto_linker_output initialization values. */ diff --git a/gcc/lto/lto-partition.cc b/gcc/lto/lto-partition.cc index e4c91213f4b..eb31ecba0d3 100644 --- a/gcc/lto/lto-partition.cc +++ b/gcc/lto/lto-partition.cc @@ -36,6 +36,9 @@ along with GCC; see the file COPYING3. If not see #include "lto-partition.h" #include "sreal.h" +#include +#include + vec ltrans_partitions; static void add_symbol_to_partition (ltrans_partition part, symtab_node *node); @@ -59,20 +62,41 @@ cmp_partitions_order (const void *a, const void *b) return orderb - ordera; } -/* Create new partition with name NAME. */ - +/* Create new partition with name NAME. + Does not push into ltrans_partitions. */ static ltrans_partition -new_partition (const char *name) +new_partition_no_push (const char *name) { ltrans_partition part = XCNEW (struct ltrans_partition_def); part->encoder = lto_symtab_encoder_new (false); part->name = name; part->insns = 0; part->symbols = 0; + return part; +} + +/* Create new partition with name NAME. */ + +static ltrans_partition +new_partition (const char *name) +{ + ltrans_partition part = new_partition_no_push (name); ltrans_partitions.safe_push (part); return part; } +/* Free memory used by ltrans partition. + Encoder can be kept to be freed after streaming. */ +static void +free_ltrans_partition (ltrans_partition part, bool delete_encoder) + { + if (part->initializers_visited) + delete part->initializers_visited; + if (delete_encoder) + lto_symtab_encoder_delete (part->encoder); + free (part); + } + /* Free memory used by ltrans datastructures. */ void @@ -81,12 +105,7 @@ free_ltrans_partitions (void) unsigned int idx; ltrans_partition part; for (idx = 0; ltrans_partitions.iterate (idx, &part); idx++) - { - if (part->initializers_visited) - delete part->initializers_visited; - /* Symtab encoder is freed after streaming. */ - free (part); - } + free_ltrans_partition (part, false); ltrans_partitions.release (); } @@ -427,6 +446,574 @@ account_reference_p (symtab_node *n1, symtab_node *n2) return true; } +/* Joins two partitions into one. + NULL partitions are equivalent to empty partition. + If both partition are non-null, symbols from FROM are added into INTO. */ +static ltrans_partition +join_partitions (ltrans_partition into, ltrans_partition from) +{ + if (!into) + return from; + if (!from) + return into; + + lto_symtab_encoder_iterator lsei; + lto_symtab_encoder_t encoder = from->encoder; + + /* If aux is non zero, it will not be added to the new partition. Since + adding symbols is recursive, it is safer to reduce aux of all symbols + before adding any symbols to other partition. */ + for (lsei = lsei_start (encoder); !lsei_end_p (lsei); lsei_next (&lsei)) + { + symtab_node *node = lsei_node (lsei); + node->aux = (void *)((size_t)node->aux - 1); + } + + for (lsei = lsei_start (encoder); !lsei_end_p (lsei); lsei_next (&lsei)) + { + symtab_node *node = lsei_node (lsei); + + if (symbol_partitioned_p (node)) + continue; + + add_symbol_to_partition (into, node); + } + + free_ltrans_partition (from, true); + + return into; +} + +/* Takes symbols from given partitions and splits them into N partitions where + each partitions contains one symbol and its requirements. */ +static std::vector +split_partition_into_nodes (ltrans_partition part) +{ + std::vector partitions; + + lto_symtab_encoder_iterator lsei; + lto_symtab_encoder_t encoder = part->encoder; + + for (lsei = lsei_start (encoder); !lsei_end_p (lsei); lsei_next (&lsei)) + { + symtab_node *node = lsei_node (lsei); + node->aux = (void *)((size_t)node->aux - 1); + } + + for (lsei = lsei_start (encoder); !lsei_end_p (lsei); lsei_next (&lsei)) + { + symtab_node *node = lsei_node (lsei); + + if (node->get_partitioning_class () != SYMBOL_PARTITION + || symbol_partitioned_p (node)) + continue; + + ltrans_partition new_part = new_partition_no_push (part->name); + add_symbol_to_partition (new_part, node); + partitions.push_back (new_part); + } + + return partitions; +} + +/* Returns whether partition contains symbols that cannot be reordered. */ +static bool +is_partition_reorder (ltrans_partition part) +{ + lto_symtab_encoder_iterator lsei; + lto_symtab_encoder_t encoder = part->encoder; + + for (lsei = lsei_start (encoder); !lsei_end_p (lsei); lsei_next (&lsei)) + { + symtab_node *node = lsei_node (lsei); + if (node->no_reorder) + return false; + } + return true; +} + +/* Represents groups of symbols, that should be partitioned into n_partitions + partitions. */ +class partition_set +{ +public: + /* Metadata to easily pass copy to new partition_set. */ + class metadata + { + public: + /* Partitions can be reordered. */ + bool reorder; + /* Partitions can be split into individual symbols. */ + bool splitable; + + metadata (bool reorder, bool splitable): + reorder (reorder), splitable (splitable) + {} + }; + metadata data; + + /* Symbol groups. Use push (g) to insert symbols. */ + std::vector sym_groups; + /* Number of partitions these symbols should be partitioned into. */ + size_t n_partitions; + /* Total number of instructions of all symbols. */ + int64_t insns; + + /* Constructor. Symbols and n_partitions can be added later. */ + partition_set (metadata data, std::vector sym_groups = {}, + size_t n_partitions = 0) + : data (data), sym_groups (std::move (sym_groups)), + n_partitions (n_partitions), insns (0) + { + for (ltrans_partition g: this->sym_groups) + insns += g->insns; + } + + /* Adds symbol group and updates total insns. */ + void + push (ltrans_partition g) + { + sym_groups.push_back (g); + insns += g->insns; + } + + /* Returns whether any symbols group is contained. */ + bool + empty () + { + return sym_groups.empty (); + } +}; + +/* Distributes total n_partitions among partition_sets. + Aims to be as fair as possible. */ +static void +distribute_n_partitions (std::vector& ps, size_t n_partitions) +{ + gcc_assert (ps.size ()); + gcc_assert (ps.size () <= n_partitions); + + int64_t total_size = 0; + + for (partition_set& p: ps) + { + total_size += p.insns; + p.n_partitions = 0; + } + + if (total_size <= 0) + total_size = 1; + + size_t n_partitions_allocated = 0; + + /* First we allocate largest amount of partitions so that target_sizes are + larger than target size of total (insns/total_size). + All partition_set must have n_partitions at least one. */ + for (partition_set& p: ps) + { + p.n_partitions = n_partitions * p.insns / total_size; + if (p.n_partitions == 0 && p.sym_groups.size ()) + p.n_partitions = 1; + if (!p.data.splitable) + p.n_partitions = std::min (p.n_partitions, p.sym_groups.size ()); + + n_partitions_allocated += p.n_partitions; + } + + /* Rare special case, with a lot of initially 0 sized splits. */ + while (n_partitions_allocated > n_partitions) + { + size_t idx = 0; + int64_t min = std::numeric_limits::max (); + + for (size_t i = 0; i < ps.size (); ++i) + { + if (ps[i].n_partitions <= 1) + continue; + + int64_t target_size = ps[i].insns / ps[i].n_partitions; + if (min > target_size) + { + min = target_size; + idx = i; + } + } + + ps[idx].n_partitions--; + n_partitions_allocated--; + } + + /* Typical case where with any increase of n_partitions target size will cross + total target size. We optimize for minimal: + (old_target_size - total_target_size) + - (total_target_size - new_target_size). */ + while (n_partitions_allocated < n_partitions) + { + size_t idx = 0; + int64_t max = 0; + + for (size_t i = 0; i < ps.size (); ++i) + { + if (ps[i].sym_groups.size () <= 1 && !ps[i].data.splitable) + continue; + + int64_t target_size = ps[i].insns / ps[i].n_partitions; + int64_t new_target_size = ps[i].insns / (ps[i].n_partitions + 1); + + int64_t positive_change = target_size + new_target_size; + + if (max < positive_change) + { + max = positive_change; + idx = i; + } + } + + ps[idx].n_partitions++; + n_partitions_allocated++; + } +} + +/* Splits off symbol groups that are larger than target size. + n_partitions are then distributed between individual + split off symbol groups, and everything else as a whole. + + Split off symbol groups with n_partitions > 1, are + then split into individual symbols. + + Order is not conserved. This pass is ignored if reorder is not allowed. */ +static std::vector +partition_over_target_split (partition_set& p) +{ + gcc_assert (p.n_partitions >= 1); + + std::vector all; + partition_set small (p.data); + + int64_t target_size = p.insns / p.n_partitions; + + for (ltrans_partition g: p.sym_groups) + { + if (g->insns > target_size + && (p.data.reorder || is_partition_reorder (g))) + all.push_back (partition_set (p.data, {g})); + else + small.push (g); + } + + if (all.empty ()) + return {}; + + if (small.sym_groups.size ()) + { + /* Handles special case where n_partitions might be smaller than + all.size (). Which can happen as result of interger division or with + 0 sized partition_sets. Then also prevents too small symbol group. + This should also be a special case; more common one, + but with no correctness problems. */ + if (all.size () && ( + small.insns < (int64_t) p.n_partitions + || small.insns < target_size * 0.6 + || small.insns < param_min_partition_size)) + { + size_t idx = 0; + int64_t min_insns = std::numeric_limits::max (); + for (size_t i = 0; i < all.size (); ++i) + { + if (all[i].insns < min_insns) + { + min_insns = all[i].insns; + idx = i; + } + } + + gcc_assert (all[idx].sym_groups.size () == 1); + + ltrans_partition& into = all[idx].sym_groups[0]; + for (ltrans_partition g: small.sym_groups) + into = join_partitions (into, g); + + all[idx].insns = into->insns; + } + else + { + gcc_assert (all.size () < p.n_partitions); + all.push_back (std::move (small)); + } + } + + distribute_n_partitions (all, p.n_partitions); + + for (partition_set& p: all) + { + gcc_assert (p.sym_groups.size ()); + + /* Handles large symbol groups (large files) that will be + further divided. */ + if (p.sym_groups.size () == 1 && p.n_partitions > 1) + { + p.sym_groups = split_partition_into_nodes (p.sym_groups[0]); + p.data.reorder = false; + p.data.splitable = false; + } + } + + return all; +} + +/* Splits partition_set into two partition_sets with + equal or off by one n_partitions. + Order is conserved. */ +static std::vector +partition_binary_split (partition_set& p) +{ + gcc_assert (p.n_partitions > 1); + + if (p.sym_groups.size () < 2) + return {}; + + int64_t target_size = p.insns / p.n_partitions; + + + std::vector result (2, partition_set (p.data)); + partition_set& first = result[0]; + partition_set& second = result[1]; + + first.n_partitions = p.n_partitions/2; + second.n_partitions = p.n_partitions - first.n_partitions; + + int64_t first_target_size = first.n_partitions * target_size; + + int64_t insns = 0; + for (ltrans_partition g: p.sym_groups) + { + /* We want at least one symbol in first partition. */ + if (first.empty ()) + first.push (g); + else if (insns < first_target_size) + { + if (insns + g->insns < first_target_size) + first.push (g); + else + { + /* Target splitting point is in this symbol group. */ + int64_t diff_first = first_target_size - insns; + int64_t diff_second = (insns + g->insns) - first_target_size; + + if (diff_first * second.n_partitions + > diff_second * first.n_partitions) + first.push (g); + else + second.push (g); + } + } + else + second.push (g); + + insns += g->insns; + } + + return result; +} + +/* Split partition_set into 'into' partition_sets with equal or off by one + number of symbol groups. Sizes of symbol groups are ignored for deciding + where to split. n_partitions is then distributed among new partition_sets + based on their sizes. + Order in conserved. */ +static std::vector +partition_fixed_split (partition_set& p, size_t into) +{ + gcc_assert (into < p.n_partitions); + + std::vector result; + + for (size_t i = 0; i < into; ++i) + { + size_t begin = i * p.sym_groups.size () / into; + size_t end = (i + 1) * p.sym_groups.size () / into; + + auto it = p.sym_groups.begin (); + result.push_back (partition_set (p.data, {it + begin, it + end})); + } + + distribute_n_partitions (result, p.n_partitions); + + return result; +} + + +/* Base implementation to inherit from for all Partitioners. */ +class partitioner_base { +public: + /* Partitions sym_groups into n_partitions partitions inserted into + ltrans_partitions. */ + void + apply (std::vector& sym_groups, int n_partitions) + { + partition_set p (partition_set::metadata (true, true), + std::move (sym_groups), n_partitions); + split (p, 0); + } + +protected: + partitioner_base (int64_t min_partition_size, int64_t max_partition_size): + min_partition_size (min_partition_size), + max_partition_size (max_partition_size) + { + gcc_assert (min_partition_size != 0); + gcc_assert (max_partition_size != 0); + } + virtual ~partitioner_base () + {} + + /* Joins all symbol groups into one finalized partition. */ + void + finalize (partition_set& p) + { + ltrans_partition joined = NULL; + + for (ltrans_partition g: p.sym_groups) + joined = join_partitions (joined, g); + + if (joined) + ltrans_partitions.safe_push (joined); + } + + /* Splits all partition_sets. */ + void + split_list (std::vector& ps, uintptr_t state) + { + for (partition_set& p: ps) + split (p, state); + } + + /* Handles common cases: + too large or small n_partitions, or n_partitions = 1. + And then calls split_state. */ + void + split (partition_set& p, uintptr_t state) + { + size_t min_partitions = p.insns / max_partition_size + 1; + size_t max_partitions = p.insns / min_partition_size; + if (!p.data.splitable) + max_partitions = std::min (max_partitions, p.sym_groups.size ()); + + p.n_partitions = std::max (p.n_partitions, min_partitions); + p.n_partitions = std::min (p.n_partitions, max_partitions); + + if (p.n_partitions <= 1) + return finalize (p); + + split_state (p, state); + } + + /* State machine for specific partitioner implementation. */ + virtual void + split_state (partition_set& p, uintptr_t state) = 0; + + int64_t min_partition_size, max_partition_size; +}; + + +/* Partitioner combining fixed, over_target, and binary partitionings. */ +class partitioner_default: public partitioner_base +{ +public: + partitioner_default (int64_t min_partition_size, int64_t max_partition_size): + partitioner_base (min_partition_size, max_partition_size) + {} + +private: + virtual void + split_state (partition_set& p, uintptr_t state) + { + const uintptr_t FIXED = 0; + const uintptr_t OVER_TARGET = 1; + const uintptr_t BINARY = 2; + + std::vector ps; + + switch (state) + { + case FIXED: + if (p.n_partitions > 64 && p.sym_groups.size () >= 4) + { + ps = partition_fixed_split (p, 4); + split_list (ps, OVER_TARGET); + break; + } + + /* FALLTHROUGH */ + case OVER_TARGET: + ps = partition_over_target_split (p); + if (!ps.empty ()) + { + split_list (ps, BINARY); + break; + } + + /* FALLTHROUGH */ + case BINARY: + ps = partition_binary_split (p); + if (!ps.empty ()) + { + split_list (ps, OVER_TARGET); + break; + } + + /* FALLTHROUGH */ + default: + finalize (p); + } + } +}; + +/* Group cgraph nodes into equally-sized partitions. + It tries to keep symbols from single source file together to minimize + propagation of divergence. + + It starts with symbols already grouped by source files. If reasonably + possible it only either combines several files into one final partition, + or, if a file is large, split the file into several final partitions. + + Intermediate representation is partition_set which contains set of + groups of symbols (each group corresponding to original source file) and + number of final partitions this partition_set should split into. + + First partition_fixed_split splits partition_set into constant number of + partition_sets with equal number of symbols groups. If for example there + are 39 source files, the resulting partition_sets will contain 10, 10, + 10, and 9 source files. This splitting intentionally ignores estimated + instruction counts to minimize propagation of divergence. + + Second partition_over_target_split separates too large files and splits + them into individual symbols to be combined back into several smaller + files in next step. + + Third partition_binary_split splits partition_set into two halves until + it should be split into only one final partition, at which point the + remaining symbols are joined into one final partition. +*/ + +void +lto_cache_map (int n_lto_partitions, int max_partition_size) +{ + lto_1_to_1_map (); + + std::vector partitions; + for (unsigned i = 0; i < ltrans_partitions.length (); ++i) + { + ltrans_partition part = ltrans_partitions[i]; + partitions.push_back (part); + } + ltrans_partitions.truncate (0); + + partitioner_default partitioner = partitioner_default + (param_min_partition_size, max_partition_size); + + partitioner.apply (partitions, n_lto_partitions); +} /* Group cgraph nodes into equally-sized partitions. diff --git a/gcc/lto/lto-partition.h b/gcc/lto/lto-partition.h index 4299033e665..853f456537e 100644 --- a/gcc/lto/lto-partition.h +++ b/gcc/lto/lto-partition.h @@ -35,6 +35,7 @@ extern vec ltrans_partitions; void lto_1_to_1_map (void); void lto_max_map (void); +void lto_cache_map (int, int); void lto_balanced_map (int, int); void lto_promote_cross_file_statics (void); void free_ltrans_partitions (void); diff --git a/gcc/lto/lto.cc b/gcc/lto/lto.cc index 12d4c6b95fe..f77ad0ea034 100644 --- a/gcc/lto/lto.cc +++ b/gcc/lto/lto.cc @@ -552,6 +552,8 @@ do_whole_program_analysis (void) else if (flag_lto_partition == LTO_PARTITION_BALANCED) lto_balanced_map (param_lto_partitions, param_max_partition_size); + else if (flag_lto_partition == LTO_PARTITION_CACHE) + lto_cache_map (param_lto_partitions, param_max_partition_size); else gcc_unreachable (); diff --git a/gcc/testsuite/gcc.dg/completion-2.c b/gcc/testsuite/gcc.dg/completion-2.c index 166bfdc1424..99e65312201 100644 --- a/gcc/testsuite/gcc.dg/completion-2.c +++ b/gcc/testsuite/gcc.dg/completion-2.c @@ -4,6 +4,7 @@ /* { dg-begin-multiline-output "" } -flto-partition=1to1 -flto-partition=balanced +-flto-partition=cache -flto-partition=max -flto-partition=none -flto-partition=one From patchwork Fri Nov 17 20:17:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michal Jires X-Patchwork-Id: 166327 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9910:0:b0:403:3b70:6f57 with SMTP id i16csp789748vqn; Fri, 17 Nov 2023 12:18:41 -0800 (PST) X-Google-Smtp-Source: AGHT+IE+XrpGicduDSa8x1IrisH11HYO25fhiYrwOFnysrAMhdELihmaf6oumZMxOd6278ZELvU7 X-Received: by 2002:ac8:58ce:0:b0:417:9a94:870b with SMTP id u14-20020ac858ce000000b004179a94870bmr929974qta.35.1700252321054; Fri, 17 Nov 2023 12:18:41 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1700252321; cv=pass; d=google.com; s=arc-20160816; b=ik2k0KKdbYGJrUl8rMVumYKAY3Y9lwPachaSK1VzL1XbRyedNGEDvZdGrwkVr1NX2i PoAN8lDGTPq/Nwg9+XRQEUhDnjN6mUPQ1s5RRtgarI6iMWiJRi0VnJrLfN7UNIk1hyO6 MRh5BnRrtOMfCZ2YJ1eJl15NzXdbT/ow0FPqTO9yyKTot9fRnRO3mvuFJcsUrmD0ZLce BYTgP7JoIhgacygbY3USI6XWge3cfhsK6xciiIYiSLRp3jgGIgKh56NAtAtJ7nSmNcKK SdtShzbCvQ3FAFwzMpx33ShTtAD4UQUflZFLphyi+nGSwYmjntVyKtbtTgw466WTfO/m r0Ug== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:in-reply-to:content-disposition :mime-version:references:message-id:subject:to:from:date :dkim-signature:dkim-signature:arc-filter:dmarc-filter:delivered-to; bh=TYCVoSOQPM/Sd+vhuhaE0Y76vqvgrWbiNH+fDNAXStU=; fh=hPrbWPhweUx4V0GV9uXJqbyAzg2ABmTz7kczrAQqMmM=; b=DlAlr5wTDGnxWbvcKRaIoey8AyutcYzJYPRZXfUamcThphcmugPKUHg/9fyG5aPdJG 9Oug0zvXl32oFrQm6M3MtqT2mdliJrXflqjmwwIDknmCOSMrqGwAetNImfw7uR0Qdqh8 xXqFNbq1RhNbEEvjKiKsHz3Ri7uTgN+VjPxU5ZQLd6+eKt0qBGkhzpzEzaOYGeptUunZ wdv13yQ8mN+9xGJQ1+ePZhV0mi80ggXXbxr6GSbpr7CDFn4KhIHWuaT6Du1pvuANhElv 6fezblTqKfAut3tfJma+LyQvh1jrq+7/R+hNVM3SRMGjw4m0Tu1adKDOTCUTbYEEOxKk PL1A== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@suse.cz header.s=susede2_rsa header.b=Gz3N9g39; dkim=neutral (no key) header.i=@suse.cz; arc=pass (i=1); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (server2.sourceware.org. [2620:52:3:1:0:246e:9693:128c]) by mx.google.com with ESMTPS id u22-20020ac858d6000000b00403bf9d2091si2347752qta.363.2023.11.17.12.18.40 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 17 Nov 2023 12:18:41 -0800 (PST) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) client-ip=2620:52:3:1:0:246e:9693:128c; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.cz header.s=susede2_rsa header.b=Gz3N9g39; dkim=neutral (no key) header.i=@suse.cz; arc=pass (i=1); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id CCD023858C50 for ; Fri, 17 Nov 2023 20:18:40 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtp-out1.suse.de (smtp-out1.suse.de [IPv6:2001:67c:2178:6::1c]) by sourceware.org (Postfix) with ESMTPS id 43F8A3857830 for ; Fri, 17 Nov 2023 20:17:29 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 43F8A3857830 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: sourceware.org; spf=fail smtp.mailfrom=suse.cz ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 43F8A3857830 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2001:67c:2178:6::1c ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1700252251; cv=none; b=By1+MVEGfAfb1PFv8opXWwQcJ0jmkyT8uqZ2fcAO3osLSbXTsr+If+04ZT4K48PHLvXNlNqRWiQWxh+xLUk3b99+9mc8IMgSy+3A4d+RURc++qQtyz6cdwagj/i/+5RNpulqSBIe90MzKmWkYJkESO0fwMEGvEBXP/r/oKGZnxE= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1700252251; c=relaxed/simple; bh=fU1MnP2nhBngY3mkYq70ipSFi+rAxZx1ta3lZ5U6xlU=; h=DKIM-Signature:DKIM-Signature:Date:From:To:Subject:Message-ID: MIME-Version; b=fb+HjRWABUAXnXPsVMUO9Ra/LcmHQkpH4Y7ECBkZ42SHxBaL7TxExBfzHjqriKAO1n5jGFVChIIdC7eaqXp1Nb7BC0AN4cnayoNytBadTsgeJD7sjZh6bDMtBIt3guvHKqkbFoH+AoVgRiDIzUpSvvrvMciBROaDNmAHvwzWQ5c= ARC-Authentication-Results: i=1; server2.sourceware.org Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 4839A2293E for ; Fri, 17 Nov 2023 20:17:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1700252248; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=TYCVoSOQPM/Sd+vhuhaE0Y76vqvgrWbiNH+fDNAXStU=; b=Gz3N9g39/Up65UstIpOivywrDw6cXM5hjou3Z3Mc0HX7nzT8bhE/xjmLEfMKIswguAzejn p/ieZCtodgX2VLiuxXXsu35A/UKURlA1yOwJDQKiq5F0l5q+6SBQk95StcxW9Wz/LIavDW z7q9AhwpFOSdk/wh4QFMR4IRn/dM2hQ= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1700252248; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=TYCVoSOQPM/Sd+vhuhaE0Y76vqvgrWbiNH+fDNAXStU=; b=Fb6TkshA/DBOTpRey1cY2PwQESbeWvqMlc2LERjD10ZZXGAoD4zWI5bMHT4V8i3CgCBapa sWX7rHZjtF5yE9Cg== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 278481341F for ; Fri, 17 Nov 2023 20:17:28 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id E5GrB1jKV2VWGgAAMHmgww (envelope-from ) for ; Fri, 17 Nov 2023 20:17:28 +0000 Date: Fri, 17 Nov 2023 21:17:26 +0100 From: Michal Jires To: gcc-patches@gcc.gnu.org Subject: [PATCH 6/7] lto: squash order of symbols in partitions Message-ID: <1169efeea8ca079fc9297a4f95ad292558b1bbcf.1700222403.git.mjires@suse.cz> References: <18cc1c3980551ac1881eea6e78811a629c7baa82.1700222403.git.mjires@suse.cz> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <18cc1c3980551ac1881eea6e78811a629c7baa82.1700222403.git.mjires@suse.cz> Authentication-Results: smtp-out1.suse.de; none X-Spam-Level: X-Spam-Score: -3.29 X-Spamd-Result: default: False [-3.29 / 50.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[gcc-patches@gcc.gnu.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; DKIM_SIGNED(0.00)[suse.cz:s=susede2_rsa,suse.cz:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.19)[-0.968]; MID_CONTAINS_FROM(1.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; BAYES_HAM(-3.00)[100.00%] X-Spam-Status: No, score=-11.9 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, SPF_HELO_NONE, SPF_SOFTFAIL, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1782843777913605433 X-GMAIL-MSGID: 1782843777913605433 This patch squashes order of symbols in individual partitions, so that their relative order is conserved, but is not influenced by symbols in other partitions. Order of cloned symbols is set to 0. This should be fine because order specifies order of symbols in input files, which cloned symbols are not part of. This is important for incremental LTO because if there is a new symbol, it otherwise shifts order of all symbols with higher order, which would diverge them all. Bootstrapped/regtested on x86_64-pc-linux-gnu gcc/ChangeLog: * lto-cgraph.cc (lto_output_node): Add and use order_remap. (lto_output_varpool_node): Likewise. (output_symtab): Likewise. * lto-streamer-out.cc (produce_asm): Likewise. (output_function): Likewise. (output_constructor): Likewise. (copy_function_or_variable): Likewise. (cmp_int): New. (lto_output): Generate order_remap. * lto-streamer.h (produce_asm): Add order_remap. (output_symtab): Likewise. --- gcc/lto-cgraph.cc | 20 ++++++++---- gcc/lto-streamer-out.cc | 71 +++++++++++++++++++++++++++++++++-------- gcc/lto-streamer.h | 5 +-- 3 files changed, 73 insertions(+), 23 deletions(-) diff --git a/gcc/lto-cgraph.cc b/gcc/lto-cgraph.cc index 32c0f5ac6db..a7530290fba 100644 --- a/gcc/lto-cgraph.cc +++ b/gcc/lto-cgraph.cc @@ -381,7 +381,8 @@ reachable_from_this_partition_p (struct cgraph_node *node, lto_symtab_encoder_t static void lto_output_node (struct lto_simple_output_block *ob, struct cgraph_node *node, - lto_symtab_encoder_t encoder) + lto_symtab_encoder_t encoder, + hash_map, int>* order_remap) { unsigned int tag; struct bitpack_d bp; @@ -405,7 +406,9 @@ lto_output_node (struct lto_simple_output_block *ob, struct cgraph_node *node, streamer_write_enum (ob->main_stream, LTO_symtab_tags, LTO_symtab_last_tag, tag); - streamer_write_hwi_stream (ob->main_stream, node->order); + + int order = flag_wpa ? *order_remap->get (node->order) : node->order; + streamer_write_hwi_stream (ob->main_stream, order); /* In WPA mode, we only output part of the call-graph. Also, we fake cgraph node attributes. There are two cases that we care. @@ -585,7 +588,8 @@ lto_output_node (struct lto_simple_output_block *ob, struct cgraph_node *node, static void lto_output_varpool_node (struct lto_simple_output_block *ob, varpool_node *node, - lto_symtab_encoder_t encoder) + lto_symtab_encoder_t encoder, + hash_map, int>* order_remap) { bool boundary_p = !lto_symtab_encoder_in_partition_p (encoder, node); bool encode_initializer_p @@ -602,7 +606,8 @@ lto_output_varpool_node (struct lto_simple_output_block *ob, varpool_node *node, streamer_write_enum (ob->main_stream, LTO_symtab_tags, LTO_symtab_last_tag, LTO_symtab_variable); - streamer_write_hwi_stream (ob->main_stream, node->order); + int order = flag_wpa ? *order_remap->get (node->order) : node->order; + streamer_write_hwi_stream (ob->main_stream, order); lto_output_var_decl_ref (ob->decl_state, ob->main_stream, node->decl); bp = bitpack_create (ob->main_stream); bp_pack_value (&bp, node->externally_visible, 1); @@ -967,7 +972,7 @@ compute_ltrans_boundary (lto_symtab_encoder_t in_encoder) /* Output the part of the symtab in SET and VSET. */ void -output_symtab (void) +output_symtab (hash_map, int>* order_remap) { struct cgraph_node *node; struct lto_simple_output_block *ob; @@ -994,9 +999,10 @@ output_symtab (void) { symtab_node *node = lto_symtab_encoder_deref (encoder, i); if (cgraph_node *cnode = dyn_cast (node)) - lto_output_node (ob, cnode, encoder); + lto_output_node (ob, cnode, encoder, order_remap); else - lto_output_varpool_node (ob, dyn_cast (node), encoder); + lto_output_varpool_node (ob, dyn_cast (node), encoder, + order_remap); } /* Go over the nodes in SET again to write edges. */ diff --git a/gcc/lto-streamer-out.cc b/gcc/lto-streamer-out.cc index a1bbea8fc68..9448ab195d5 100644 --- a/gcc/lto-streamer-out.cc +++ b/gcc/lto-streamer-out.cc @@ -2212,7 +2212,8 @@ output_cfg (struct output_block *ob, struct function *fn) a function, set FN to the decl for that function. */ void -produce_asm (struct output_block *ob, tree fn) +produce_asm (struct output_block *ob, tree fn, + hash_map, int>* order_remap) { enum lto_section_type section_type = ob->section_type; struct lto_function_header header; @@ -2221,9 +2222,11 @@ produce_asm (struct output_block *ob, tree fn) if (section_type == LTO_section_function_body) { const char *name = IDENTIFIER_POINTER (DECL_ASSEMBLER_NAME (fn)); - section_name = lto_get_section_name (section_type, name, - symtab_node::get (fn)->order, - NULL); + + int order = symtab_node::get (fn)->order; + if (flag_wpa && order_remap) + order = *order_remap->get (order); + section_name = lto_get_section_name (section_type, name, order, NULL); } else section_name = lto_get_section_name (section_type, NULL, 0, NULL); @@ -2405,7 +2408,8 @@ streamer_write_chain (struct output_block *ob, tree t, bool ref_p) /* Output the body of function NODE->DECL. */ static void -output_function (struct cgraph_node *node) +output_function (struct cgraph_node *node, + hash_map, int>* order_remap) { tree function; struct function *fn; @@ -2482,7 +2486,7 @@ output_function (struct cgraph_node *node) streamer_write_uhwi (ob, 0); /* Create a section to hold the pickled output of this function. */ - produce_asm (ob, function); + produce_asm (ob, function, order_remap); destroy_output_block (ob); if (streamer_dump_file) @@ -2493,7 +2497,8 @@ output_function (struct cgraph_node *node) /* Output the body of function NODE->DECL. */ static void -output_constructor (struct varpool_node *node) +output_constructor (struct varpool_node *node, + hash_map, int>* order_remap) { tree var = node->decl; struct output_block *ob; @@ -2515,7 +2520,7 @@ output_constructor (struct varpool_node *node) stream_write_tree (ob, DECL_INITIAL (var), true); /* Create a section to hold the pickled output of this function. */ - produce_asm (ob, var); + produce_asm (ob, var, order_remap); destroy_output_block (ob); if (streamer_dump_file) @@ -2576,15 +2581,18 @@ lto_output_toplevel_asms (void) /* Copy the function body or variable constructor of NODE without deserializing. */ static void -copy_function_or_variable (struct symtab_node *node) +copy_function_or_variable (struct symtab_node *node, + hash_map, int>* order_remap) { tree function = node->decl; struct lto_file_decl_data *file_data = node->lto_file_data; const char *data; size_t len; const char *name = IDENTIFIER_POINTER (DECL_ASSEMBLER_NAME (function)); + + int order = flag_wpa ? *order_remap->get (node->order) : node->order; char *section_name = - lto_get_section_name (LTO_section_function_body, name, node->order, NULL); + lto_get_section_name (LTO_section_function_body, name, order, NULL); size_t i, j; struct lto_in_decl_state *in_state; struct lto_out_decl_state *out_state = lto_get_out_decl_state (); @@ -2729,6 +2737,15 @@ cmp_symbol_files (const void *pn1, const void *pn2, void *id_map_) return n1->order - n2->order; } +/* Compare ints, callback for qsort. */ +static int +cmp_int (const void *a, const void *b) +{ + int ia = *(int const*) a; + int ib = *(int const*) b; + return ia - ib; +} + /* Main entry point from the pass manager. */ void @@ -2741,6 +2758,32 @@ lto_output (void) lto_symtab_encoder_t encoder = lto_get_out_decl_state ()->symtab_node_encoder; auto_vec symbols_to_copy; + hash_map, int> order_remap; + if (flag_wpa) + { + /* Remap order so that it does not depend on symbols outside of + partition. */ + auto_vec orders; + + n_nodes = lto_symtab_encoder_size (encoder); + for (i = 0; i < n_nodes; i++) + { + symtab_node *snode = lto_symtab_encoder_deref (encoder, i); + if (cgraph_node *cnode = dyn_cast (snode)) + { + if (cnode->clone_of) + { + order_remap.put (snode->order, 0); + continue; + } + } + orders.safe_push (snode->order); + } + orders.qsort (cmp_int); + for (i = 0; i < orders.length (); i++) + order_remap.put (orders[i], i); + } + prune_offload_funcs (); if (flag_checking) @@ -2817,14 +2860,14 @@ lto_output (void) at WPA time. */ || DECL_ARGUMENTS (cnode->decl) || cnode->declare_variant_alt)) - output_function (cnode); + output_function (cnode, &order_remap); else if ((vnode = dyn_cast (snode)) && (DECL_INITIAL (vnode->decl) != error_mark_node || (!flag_wpa && flag_incremental_link != INCREMENTAL_LINK_LTO))) - output_constructor (vnode); + output_constructor (vnode, &order_remap); else - copy_function_or_variable (snode); + copy_function_or_variable (snode, &order_remap); gcc_assert (lto_get_out_decl_state () == decl_state); lto_pop_out_decl_state (); lto_record_function_out_decl_state (snode->decl, decl_state); @@ -2834,7 +2877,7 @@ lto_output (void) be done now to make sure that all the statements in every function have been renumbered so that edges can be associated with call statements using the statement UIDs. */ - output_symtab (); + output_symtab (&order_remap); output_offload_tables (); diff --git a/gcc/lto-streamer.h b/gcc/lto-streamer.h index 0556b34c837..3363e6f9e61 100644 --- a/gcc/lto-streamer.h +++ b/gcc/lto-streamer.h @@ -888,7 +888,8 @@ extern void lto_output_fn_decl_ref (struct lto_out_decl_state *, extern tree lto_input_var_decl_ref (lto_input_block *, lto_file_decl_data *); extern tree lto_input_fn_decl_ref (lto_input_block *, lto_file_decl_data *); extern void lto_output_toplevel_asms (void); -extern void produce_asm (struct output_block *ob, tree fn); +extern void produce_asm (struct output_block *ob, tree fn, + hash_map, int>* order_remap = 0); extern void lto_output (); extern void produce_asm_for_decls (); void lto_output_decl_state_streams (struct output_block *, @@ -919,7 +920,7 @@ void lto_set_symtab_encoder_in_partition (lto_symtab_encoder_t, bool lto_symtab_encoder_encode_initializer_p (lto_symtab_encoder_t, varpool_node *); -void output_symtab (void); +void output_symtab (hash_map, int>*); void input_symtab (void); void output_offload_tables (void); void input_offload_tables (bool); From patchwork Fri Nov 17 20:17:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michal Jires X-Patchwork-Id: 166329 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9910:0:b0:403:3b70:6f57 with SMTP id i16csp789972vqn; Fri, 17 Nov 2023 12:19:10 -0800 (PST) X-Google-Smtp-Source: AGHT+IHwqHxM6oC/FAQQG1Ze5gXtKcVfzl892laoyPPoa69zB+ZiHhHW63O5SjslqxBvEB5gip4B X-Received: by 2002:a05:6214:124c:b0:670:710f:2042 with SMTP id r12-20020a056214124c00b00670710f2042mr174638qvv.15.1700252350262; Fri, 17 Nov 2023 12:19:10 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1700252350; cv=pass; d=google.com; s=arc-20160816; b=fMJ34ZRdjxlz47zC1ceF3hBb3Fr636d5z1F2N5X8TI3uq1kVz2f1cW2cPxaWtjXt+l hXaGm2uQaQvY22CPsnkxnn+1Q5SfVtAzAdQiTdDPwBBX/tk5aEN9u17q7j8ySUqh87Be i5/hTWBzK1440MwHwUTSAAyW+i2cZMyTAYxIap9+ES2KLLg+qrIIpJpuuRfTHIHsi5jC dI/xO9VWpgFM/FRi4Bzm50s4PYcNXGXZ7T4+Z1cJKmTJ9UHsPnTMZjG6OsgLb4cD4f37 XEvRqq+xqgp0acbz7atTksGZntn8Y9ZV+rTbc1zYU8Y1i9c2MGAI4v7jR0FfQpupswTk FRlA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:in-reply-to:content-disposition :mime-version:references:message-id:subject:to:from:date :dkim-signature:dkim-signature:arc-filter:dmarc-filter:delivered-to; bh=+weIGxwXo/8De0+gZFSXrhCKLRAnzFdsl4El7qME/iI=; fh=hPrbWPhweUx4V0GV9uXJqbyAzg2ABmTz7kczrAQqMmM=; b=CxrNhzHSyZ6wHlOJtbg6hYS3g0Hy/P/T29sGVEkb5+FKoQL13bXt21o4FZL101kcXb JYPAdtSIjy/yF/j27ec+JpY8UvPo5tTsICX0DBelWvG6+N9LBIEuXq1SfCMnsrE7B9EG 80Xh/X8wXquZ7JfGgKnvfZZ4AP9WQyyYPSHb1CDcj8d6Mxoew9DNcANKGFAzK7pMw6jK a58ZqABVxWU9jc4BywxejqkQry2PgJcgJxI5QZJ6JQ746LHc8lAP4+E3psFq/yOOktQe rKvn9XBmZ2u5R3/pd9L5kYfcdJyDkrvtZKXrsTFDktXqHYPlOANIAGljlFz/dzoVDEGa Vu2g== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@suse.cz header.s=susede2_rsa header.b=qY3mjI+A; dkim=neutral (no key) header.i=@suse.cz header.s=susede2_ed25519 header.b=ggdqQt7q; arc=pass (i=1); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (server2.sourceware.org. [8.43.85.97]) by mx.google.com with ESMTPS id i6-20020a056214020600b006563f18c948si2270583qvt.367.2023.11.17.12.19.10 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 17 Nov 2023 12:19:10 -0800 (PST) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) client-ip=8.43.85.97; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.cz header.s=susede2_rsa header.b=qY3mjI+A; dkim=neutral (no key) header.i=@suse.cz header.s=susede2_ed25519 header.b=ggdqQt7q; arc=pass (i=1); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 0E9AA38582AF for ; Fri, 17 Nov 2023 20:19:10 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by sourceware.org (Postfix) with ESMTPS id CA0693857B8B for ; Fri, 17 Nov 2023 20:17:36 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org CA0693857B8B Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=suse.cz ARC-Filter: OpenARC Filter v1.0.0 sourceware.org CA0693857B8B Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=195.135.220.29 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1700252258; cv=none; b=Gc0O5uUwsOPSMc+FHbnR2g+UTs8kFWiA5dh4gh6s7e5GJ+v64gNk9gvRTwcItnw+RAN6DZoQHf+i9Mc0dTSwK0nahSq9jG0xqNuslo/d6SEuPfj74cXZ2j08Gw2p0XIEj8CggS83T3xM+J6wAaRSftdqvzTDTpD2IJ4I3zDrhAM= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1700252258; c=relaxed/simple; bh=XqZfn1Klj0awjdeMxfSv9n8ID2VjvGYG3bvTQoL4tWo=; h=DKIM-Signature:DKIM-Signature:Date:From:To:Subject:Message-ID: MIME-Version; b=B1mYqTdJNOMQVeKPhcRu8yFasO7Ut7s8264YWKL6iqgd84l00xsa4D2xkP9WtP8tX+7yT5PiBuQWRga1CXKzBvU5Yv/Q+ZEhg4Al9trWAeqA+xCkPMudFnnC7m1CS0EgLCPubt46v0Irhfnv4gLQN+A5k8wkpXit40i8N+4UwnY= ARC-Authentication-Results: i=1; server2.sourceware.org Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id E71AA1F37E for ; Fri, 17 Nov 2023 20:17:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1700252255; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=+weIGxwXo/8De0+gZFSXrhCKLRAnzFdsl4El7qME/iI=; b=qY3mjI+A4vx83qdUCF/JMNvfDoR0+U+XGEmvST0iR6GaeQm7nsLhG7MD+QQaVePdc9iLTe 7JJKLbEGpU7s2ruCu8bWHQ/ChwZsMdcXFal/0iqf4EwQYtx5/VQTvw/w+Zf7Ss4WUDP1tM 6Ex0m9fpNPifqGBp8tMZi5tYWID9vEw= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1700252255; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=+weIGxwXo/8De0+gZFSXrhCKLRAnzFdsl4El7qME/iI=; b=ggdqQt7qkeSjrNeWHw6XEfu7mluJPT3abtHlPhApAClR8zfSyqQDudovumgOFdJAQbVzd7 tOwDjp13MQ1mzdAg== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id D02E41341F for ; Fri, 17 Nov 2023 20:17:35 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id qJEGMV/KV2VnGgAAMHmgww (envelope-from ) for ; Fri, 17 Nov 2023 20:17:35 +0000 Date: Fri, 17 Nov 2023 21:17:34 +0100 From: Michal Jires To: gcc-patches@gcc.gnu.org Subject: [PATCH 7/7] lto: partition specific lto_clone_numbers Message-ID: <7c76f258995c20eb0b44eb021f44039718dc45ce.1700222403.git.mjires@suse.cz> References: <18cc1c3980551ac1881eea6e78811a629c7baa82.1700222403.git.mjires@suse.cz> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <18cc1c3980551ac1881eea6e78811a629c7baa82.1700222403.git.mjires@suse.cz> Authentication-Results: smtp-out2.suse.de; none X-Spam-Level: X-Spam-Score: -3.02 X-Spamd-Result: default: False [-3.02 / 50.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[gcc-patches@gcc.gnu.org]; TO_DN_NONE(0.00)[]; CLAM_VIRUS_FAIL(0.00)[failed to scan and retransmits exceed]; RCPT_COUNT_ONE(0.00)[1]; DKIM_SIGNED(0.00)[suse.cz:s=susede2_rsa,suse.cz:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.17)[-0.859]; MID_CONTAINS_FROM(1.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; BAYES_HAM(-2.75)[98.89%] X-Spam-Status: No, score=-12.5 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1782843808116600994 X-GMAIL-MSGID: 1782843808116600994 Replaces "lto_priv.$clone_number" by "lto_priv.$partition_hash.$partition_specific_clone_number". To reduce divergence for incremental LTO. Bootstrapped/regtested on x86_64-pc-linux-gnu gcc/lto/ChangeLog: * lto-partition.cc (set_clone_partition_name_checksum): New. (CHECKSUM_STRING): New. (privatize_symbol_name_1): Use partition hash for lto_priv. (lto_promote_cross_file_statics): Use set_clone_partition_name_checksum. (lto_promote_statics_nonwpa): Changed clone_map type. --- gcc/lto/lto-partition.cc | 49 +++++++++++++++++++++++++++++++++++----- 1 file changed, 43 insertions(+), 6 deletions(-) diff --git a/gcc/lto/lto-partition.cc b/gcc/lto/lto-partition.cc index eb31ecba0d3..a2ce24eea23 100644 --- a/gcc/lto/lto-partition.cc +++ b/gcc/lto/lto-partition.cc @@ -35,6 +35,7 @@ along with GCC; see the file COPYING3. If not see #include "ipa-fnsummary.h" #include "lto-partition.h" #include "sreal.h" +#include "md5.h" #include #include @@ -1516,8 +1517,36 @@ validize_symbol_for_target (symtab_node *node) } } -/* Maps symbol names to unique lto clone counters. */ -static hash_map *lto_clone_numbers; +/* Maps symbol names with partition checksum to unique lto clone counters. */ +using clone_map = hash_map>, unsigned>; +static clone_map *lto_clone_numbers; +uint64_t current_partition_checksum = 0; + +/* Computes a quick checksum to distinguish partitions of clone numbers. */ +void +set_clone_partition_name_checksum (ltrans_partition part) +{ +#define CHECKSUM_STRING(FOO) md5_process_bytes ((FOO), strlen (FOO), &ctx) + struct md5_ctx ctx; + md5_init_ctx (&ctx); + + CHECKSUM_STRING (part->name); + + lto_symtab_encoder_iterator lsei; + lto_symtab_encoder_t encoder = part->encoder; + + for (lsei = lsei_start (encoder); !lsei_end_p (lsei); lsei_next (&lsei)) + { + symtab_node *node = lsei_node (lsei); + CHECKSUM_STRING (node->name ()); + } + + uint64_t checksum[2]; + md5_finish_ctx (&ctx, checksum); + current_partition_checksum = checksum[0]; +#undef CHECKSUM_STRING +} /* Helper for privatize_symbol_name. Mangle NODE symbol name represented by DECL. */ @@ -1531,10 +1560,16 @@ privatize_symbol_name_1 (symtab_node *node, tree decl) return false; const char *name = maybe_rewrite_identifier (name0); - unsigned &clone_number = lto_clone_numbers->get_or_insert (name); + + unsigned &clone_number = lto_clone_numbers->get_or_insert ( + std::pair {name, current_partition_checksum}); + + char lto_priv[32]; + sprintf (lto_priv, "lto_priv.%lu", current_partition_checksum); + symtab->change_decl_assembler_name (decl, clone_function_name ( - name, "lto_priv", clone_number)); + name, lto_priv, clone_number)); clone_number++; if (node->lto_file_data) @@ -1735,11 +1770,13 @@ lto_promote_cross_file_statics (void) part->encoder = compute_ltrans_boundary (part->encoder); } - lto_clone_numbers = new hash_map; + lto_clone_numbers = new clone_map; /* Look at boundaries and promote symbols as needed. */ for (i = 0; i < n_sets; i++) { + set_clone_partition_name_checksum (ltrans_partitions[i]); + lto_symtab_encoder_iterator lsei; lto_symtab_encoder_t encoder = ltrans_partitions[i]->encoder; @@ -1778,7 +1815,7 @@ lto_promote_statics_nonwpa (void) { symtab_node *node; - lto_clone_numbers = new hash_map; + lto_clone_numbers = new clone_map; FOR_EACH_SYMBOL (node) { rename_statics (NULL, node);