From patchwork Fri Aug 25 11:15:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Biener X-Patchwork-Id: 136913 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a7d1:0:b0:3f2:4152:657d with SMTP id p17csp1723438vqm; Fri, 25 Aug 2023 04:16:21 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH+qgGucsvO6Ag4kvS68xy+Ienk1c6k0BjxmnXikjTLlTnmfxtNedM4LOB1nyDup+S/BtZC X-Received: by 2002:a17:906:5187:b0:9a1:b5fc:8c56 with SMTP id y7-20020a170906518700b009a1b5fc8c56mr8618822ejk.55.1692962181185; Fri, 25 Aug 2023 04:16:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1692962181; cv=none; d=google.com; s=arc-20160816; b=wl8CT71fJMG5S6oCcdPl3n2q862yhV/aw7lzb/KSddvovVkjN9F7saM/Egv2yKEdvE 1FQj47ko+ghKgRdyg4jE2ILmWtyc7JUdfhjbS+RfRzmW9TXxLBMIWGF5Ayjxhf1ZvcH/ m1tIbiZjDCrKRXoXLOdy+NO90AJDp337fZjMMaWy5N/p5eWZWDuoinXGVoYxIESexrPl eLSylQHlfiOmtxgnMGRpNC2/BsemGEShayG9LgmLxhB/UUPSxs9mRew2XeanLHUgU7do ozVmjb3Gt8f5tj7Q5nMCGomRQ0Xo8cZhhjM9sbvuQXlJX1CjK1vri0f02JK6nMAvmevM KGuQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:reply-to:from:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence:message-id :mime-version:subject:to:date:dmarc-filter:delivered-to :dkim-signature:dkim-filter; bh=FYIvpk7zTDWMnIB0oFcToO/ryXHS9kxyB18NvEIXu14=; fh=hPrbWPhweUx4V0GV9uXJqbyAzg2ABmTz7kczrAQqMmM=; b=a26mVWHy3P+/NNg1t36+c9bKzRh8z9pZwB6rQ/a6XmyiA4Yp7tbvmpmAy0wWd6g2vh 3R4bInXPAKlR6ENf0KXBaskpxUZktyTbDL8OVmLShFLLzJSKKwBKoHXE7YmDwVgXo+V5 TmvzRaClUvF/VcMvBTWSBowIdqsuTy9hRDhDPsYshwV0FgeDjnlGTFRsvkm3xyjbJ8cJ 7MaLpWmCi8BI+GV6H0Y8Mh27PE2eqPOo6EXABtqLgnERT81iRtQ2Wt6NReMnF9jnowtO dFX4AGdK+/w+MzY7dldsl2EEx5B+yy74KkX9RbVEj04IFxo4X/gS6nSMzPVlREmtC4Cw be/g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=AtrzhjIH; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gnu.org Received: from server2.sourceware.org (ip-8-43-85-97.sourceware.org. [8.43.85.97]) by mx.google.com with ESMTPS id n1-20020a170906378100b009931472bdb8si791353ejc.965.2023.08.25.04.16.20 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Aug 2023 04:16:21 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) client-ip=8.43.85.97; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=AtrzhjIH; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gnu.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id DD729385842E for ; Fri, 25 Aug 2023 11:16:19 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org DD729385842E DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1692962179; bh=FYIvpk7zTDWMnIB0oFcToO/ryXHS9kxyB18NvEIXu14=; h=Date:To:Subject:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=AtrzhjIHnHD8u11EkFjgjvF+i/jz2DjPKaSQ1neARl7cvX2w7FKvpuW4B5wNJDzrF CNDfSk1PFPp9Ut22xajJsqAzGlNEwBorxUWOk9hMBEAUC/saGicxQigONWuU57A1+N ViNaBWjLe8PMW/d0QINwb2ztBUik8rbT/iy11uF0= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by sourceware.org (Postfix) with ESMTPS id 05D413858C53 for ; Fri, 25 Aug 2023 11:15:33 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 05D413858C53 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 11BA72185D for ; Fri, 25 Aug 2023 11:15:33 +0000 (UTC) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id F27E6138F9 for ; Fri, 25 Aug 2023 11:15:32 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id DI3wOVSN6GSMNgAAMHmgww (envelope-from ) for ; Fri, 25 Aug 2023 11:15:32 +0000 Date: Fri, 25 Aug 2023 13:15:32 +0200 (CEST) To: gcc-patches@gcc.gnu.org Subject: [PATCH] Apply some TLC to vect_slp_analyze_instance_dependence MIME-Version: 1.0 Message-Id: <20230825111532.F27E6138F9@imap2.suse-dmz.suse.de> X-Spam-Status: No, score=-11.5 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Richard Biener via Gcc-patches From: Richard Biener Reply-To: Richard Biener Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org Sender: "Gcc-patches" X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1775199512372921089 X-GMAIL-MSGID: 1775199512372921089 This refactors things, separating load and store handing, adjusting comments to reflect reality and removing some dead code. Bootstrapped and tested on x86_64-unknown-linux-gnu, pushed. This is in preparation for a fix for PR111137. * tree-vect-data-refs.cc (vect_slp_analyze_store_dependences): Split out from vect_slp_analyze_node_dependences, remove dead code. (vect_slp_analyze_load_dependences): Split out from vect_slp_analyze_node_dependences, adjust comments. Process queued stores before any disambiguation. (vect_slp_analyze_node_dependences): Remove. (vect_slp_analyze_instance_dependence): Adjust. --- gcc/tree-vect-data-refs.cc | 238 +++++++++++++++++-------------------- 1 file changed, 108 insertions(+), 130 deletions(-) diff --git a/gcc/tree-vect-data-refs.cc b/gcc/tree-vect-data-refs.cc index a2caf6cb1c7..0295e256a44 100644 --- a/gcc/tree-vect-data-refs.cc +++ b/gcc/tree-vect-data-refs.cc @@ -670,160 +670,138 @@ vect_slp_analyze_data_ref_dependence (vec_info *vinfo, } -/* Analyze dependences involved in the transform of SLP NODE. STORES - contain the vector of scalar stores of this instance if we are - disambiguating the loads. */ +/* Analyze dependences involved in the transform of a store SLP NODE. */ static bool -vect_slp_analyze_node_dependences (vec_info *vinfo, slp_tree node, - vec stores, - stmt_vec_info last_store_info) +vect_slp_analyze_store_dependences (vec_info *vinfo, slp_tree node) { - /* This walks over all stmts involved in the SLP load/store done + /* This walks over all stmts involved in the SLP store done in NODE verifying we can sink them up to the last stmt in the group. */ - if (DR_IS_WRITE (STMT_VINFO_DATA_REF (SLP_TREE_REPRESENTATIVE (node)))) + stmt_vec_info last_access_info = vect_find_last_scalar_stmt_in_slp (node); + gcc_assert (DR_IS_WRITE (STMT_VINFO_DATA_REF (last_access_info))); + + for (unsigned k = 0; k < SLP_TREE_SCALAR_STMTS (node).length (); ++k) { - stmt_vec_info last_access_info = vect_find_last_scalar_stmt_in_slp (node); - for (unsigned k = 0; k < SLP_TREE_SCALAR_STMTS (node).length (); ++k) + stmt_vec_info access_info + = vect_orig_stmt (SLP_TREE_SCALAR_STMTS (node)[k]); + if (access_info == last_access_info) + continue; + data_reference *dr_a = STMT_VINFO_DATA_REF (access_info); + ao_ref ref; + bool ref_initialized_p = false; + for (gimple_stmt_iterator gsi = gsi_for_stmt (access_info->stmt); + gsi_stmt (gsi) != last_access_info->stmt; gsi_next (&gsi)) { - stmt_vec_info access_info - = vect_orig_stmt (SLP_TREE_SCALAR_STMTS (node)[k]); - if (access_info == last_access_info) + gimple *stmt = gsi_stmt (gsi); + if (! gimple_vuse (stmt)) continue; - data_reference *dr_a = STMT_VINFO_DATA_REF (access_info); - ao_ref ref; - bool ref_initialized_p = false; - for (gimple_stmt_iterator gsi = gsi_for_stmt (access_info->stmt); - gsi_stmt (gsi) != last_access_info->stmt; gsi_next (&gsi)) - { - gimple *stmt = gsi_stmt (gsi); - if (! gimple_vuse (stmt)) - continue; - - /* If we couldn't record a (single) data reference for this - stmt we have to resort to the alias oracle. */ - stmt_vec_info stmt_info = vinfo->lookup_stmt (stmt); - data_reference *dr_b = STMT_VINFO_DATA_REF (stmt_info); - if (!dr_b) - { - /* We are moving a store - this means - we cannot use TBAA for disambiguation. */ - if (!ref_initialized_p) - ao_ref_init (&ref, DR_REF (dr_a)); - if (stmt_may_clobber_ref_p_1 (stmt, &ref, false) - || ref_maybe_used_by_stmt_p (stmt, &ref, false)) - return false; - continue; - } - - bool dependent = false; - /* If we run into a store of this same instance (we've just - marked those) then delay dependence checking until we run - into the last store because this is where it will have - been sunk to (and we verify if we can do that as well). */ - if (gimple_visited_p (stmt)) - { - if (stmt_info != last_store_info) - continue; - for (stmt_vec_info &store_info : stores) - { - data_reference *store_dr - = STMT_VINFO_DATA_REF (store_info); - ddr_p ddr = initialize_data_dependence_relation - (dr_a, store_dr, vNULL); - dependent - = vect_slp_analyze_data_ref_dependence (vinfo, ddr); - free_dependence_relation (ddr); - if (dependent) - break; - } - } - else - { - ddr_p ddr = initialize_data_dependence_relation (dr_a, - dr_b, vNULL); - dependent = vect_slp_analyze_data_ref_dependence (vinfo, ddr); - free_dependence_relation (ddr); - } - if (dependent) + /* If we couldn't record a (single) data reference for this + stmt we have to resort to the alias oracle. */ + stmt_vec_info stmt_info = vinfo->lookup_stmt (stmt); + data_reference *dr_b = STMT_VINFO_DATA_REF (stmt_info); + if (!dr_b) + { + /* We are moving a store - this means + we cannot use TBAA for disambiguation. */ + if (!ref_initialized_p) + ao_ref_init (&ref, DR_REF (dr_a)); + if (stmt_may_clobber_ref_p_1 (stmt, &ref, false) + || ref_maybe_used_by_stmt_p (stmt, &ref, false)) return false; + continue; } + + gcc_assert (!gimple_visited_p (stmt)); + + ddr_p ddr = initialize_data_dependence_relation (dr_a, + dr_b, vNULL); + bool dependent = vect_slp_analyze_data_ref_dependence (vinfo, ddr); + free_dependence_relation (ddr); + if (dependent) + return false; } } - else /* DR_IS_READ */ + return true; +} + +/* Analyze dependences involved in the transform of a load SLP NODE. STORES + contain the vector of scalar stores of this instance if we are + disambiguating the loads. */ + +static bool +vect_slp_analyze_load_dependences (vec_info *vinfo, slp_tree node, + vec stores, + stmt_vec_info last_store_info) +{ + /* This walks over all stmts involved in the SLP load done + in NODE verifying we can hoist them up to the first stmt in the + group. */ + stmt_vec_info first_access_info = vect_find_first_scalar_stmt_in_slp (node); + gcc_assert (DR_IS_READ (STMT_VINFO_DATA_REF (first_access_info))); + + for (unsigned k = 0; k < SLP_TREE_SCALAR_STMTS (node).length (); ++k) { - stmt_vec_info first_access_info - = vect_find_first_scalar_stmt_in_slp (node); - for (unsigned k = 0; k < SLP_TREE_SCALAR_STMTS (node).length (); ++k) + stmt_vec_info access_info + = vect_orig_stmt (SLP_TREE_SCALAR_STMTS (node)[k]); + if (access_info == first_access_info) + continue; + data_reference *dr_a = STMT_VINFO_DATA_REF (access_info); + ao_ref ref; + bool ref_initialized_p = false; + for (gimple_stmt_iterator gsi = gsi_for_stmt (access_info->stmt); + gsi_stmt (gsi) != first_access_info->stmt; gsi_prev (&gsi)) { - stmt_vec_info access_info - = vect_orig_stmt (SLP_TREE_SCALAR_STMTS (node)[k]); - if (access_info == first_access_info) + gimple *stmt = gsi_stmt (gsi); + if (! gimple_vdef (stmt)) continue; - data_reference *dr_a = STMT_VINFO_DATA_REF (access_info); - ao_ref ref; - bool ref_initialized_p = false; - for (gimple_stmt_iterator gsi = gsi_for_stmt (access_info->stmt); - gsi_stmt (gsi) != first_access_info->stmt; gsi_prev (&gsi)) - { - gimple *stmt = gsi_stmt (gsi); - if (! gimple_vdef (stmt)) - continue; - /* If we couldn't record a (single) data reference for this - stmt we have to resort to the alias oracle. */ - stmt_vec_info stmt_info = vinfo->lookup_stmt (stmt); - data_reference *dr_b = STMT_VINFO_DATA_REF (stmt_info); + stmt_vec_info stmt_info = vinfo->lookup_stmt (stmt); - /* We are hoisting a load - this means we can use - TBAA for disambiguation. */ - if (!ref_initialized_p) - ao_ref_init (&ref, DR_REF (dr_a)); - if (stmt_may_clobber_ref_p_1 (stmt, &ref, true)) - { - if (!dr_b) - return false; - /* Resort to dependence checking below. */ - } - else - /* No dependence. */ + /* If we run into a store of this same instance (we've just + marked those) then delay dependence checking until we run + into the last store because this is where it will have + been sunk to (and we verified that we can do that already). */ + if (gimple_visited_p (stmt)) + { + if (stmt_info != last_store_info) continue; - bool dependent = false; - /* If we run into a store of this same instance (we've just - marked those) then delay dependence checking until we run - into the last store because this is where it will have - been sunk to (and we verify if we can do that as well). */ - if (gimple_visited_p (stmt)) - { - if (stmt_info != last_store_info) - continue; - - for (stmt_vec_info &store_info : stores) - { - data_reference *store_dr - = STMT_VINFO_DATA_REF (store_info); - ddr_p ddr = initialize_data_dependence_relation - (dr_a, store_dr, vNULL); - dependent - = vect_slp_analyze_data_ref_dependence (vinfo, ddr); - free_dependence_relation (ddr); - if (dependent) - break; - } - } - else + for (stmt_vec_info &store_info : stores) { - ddr_p ddr = initialize_data_dependence_relation (dr_a, - dr_b, vNULL); - dependent = vect_slp_analyze_data_ref_dependence (vinfo, ddr); + data_reference *store_dr = STMT_VINFO_DATA_REF (store_info); + ddr_p ddr = initialize_data_dependence_relation + (dr_a, store_dr, vNULL); + bool dependent + = vect_slp_analyze_data_ref_dependence (vinfo, ddr); free_dependence_relation (ddr); + if (dependent) + return false; } + continue; + } + + /* We are hoisting a load - this means we can use TBAA for + disambiguation. */ + if (!ref_initialized_p) + ao_ref_init (&ref, DR_REF (dr_a)); + if (stmt_may_clobber_ref_p_1 (stmt, &ref, true)) + { + /* If we couldn't record a (single) data reference for this + stmt we have to give up now. */ + data_reference *dr_b = STMT_VINFO_DATA_REF (stmt_info); + if (!dr_b) + return false; + ddr_p ddr = initialize_data_dependence_relation (dr_a, + dr_b, vNULL); + bool dependent + = vect_slp_analyze_data_ref_dependence (vinfo, ddr); + free_dependence_relation (ddr); if (dependent) return false; } + /* No dependence. */ } } return true; @@ -850,7 +828,7 @@ vect_slp_analyze_instance_dependence (vec_info *vinfo, slp_instance instance) stmt_vec_info last_store_info = NULL; if (store) { - if (! vect_slp_analyze_node_dependences (vinfo, store, vNULL, NULL)) + if (! vect_slp_analyze_store_dependences (vinfo, store)) return false; /* Mark stores in this instance and remember the last one. */ @@ -864,7 +842,7 @@ vect_slp_analyze_instance_dependence (vec_info *vinfo, slp_instance instance) /* Verify we can sink loads to the vectorized stmt insert location, special-casing stores of this instance. */ for (slp_tree &load : SLP_INSTANCE_LOADS (instance)) - if (! vect_slp_analyze_node_dependences (vinfo, load, + if (! vect_slp_analyze_load_dependences (vinfo, load, store ? SLP_TREE_SCALAR_STMTS (store) : vNULL, last_store_info))