From patchwork Tue Jan 16 14:43:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Biener X-Patchwork-Id: 188540 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:42cf:b0:101:a8e8:374 with SMTP id q15csp304350dye; Tue, 16 Jan 2024 06:49:17 -0800 (PST) X-Google-Smtp-Source: AGHT+IEFGKWhB1R0kxiWe7obegtdfYRW/CXR+SiE1+ktnLmRI5/xE0cDxu/Yos9FLlEnkHZdTHFP X-Received: by 2002:a05:622a:a08b:b0:42a:7e2:ae65 with SMTP id jv11-20020a05622aa08b00b0042a07e2ae65mr658293qtb.59.1705416556821; Tue, 16 Jan 2024 06:49:16 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1705416556; cv=pass; d=google.com; s=arc-20160816; b=LZsGcjGG3kz9j/VNmN+pH8DFTAgV7+uQYG/Th/hPfwiIBGK6pKqrUhHnxK16bRKM6d QCibb0q6ARIxsaF98JnUH1a4/g+dyvMVptgCGl7BewXOBm73hoI2dJ7Vm1J5zmG7V7a5 M4N8660s94M8XI5FCRL3XahSYCdphvxzV8WlxV7M06elN/TCvTKdwpMQtIZGZcIRFEd7 XymQ+hpLiSxnmbZsr0gzjqLifMGup2bFt8uJ1Xu2FH61NS55Stnn5J60uWtGX4EqIwT0 AhBuD5OqAjO0/9QhPZAnHv9hSv0pQzUs3wibQ0d6UqpBv/x5oJ5gtEZnlyNAqGM76C8b 673w== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:errors-to:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence:mime-version :subject:cc:to:from:date:dkim-signature:dkim-signature :dkim-signature:dkim-signature:arc-filter:dmarc-filter:delivered-to; bh=foL8l7KptvhL4oGeNYMA1RddlSRZr77zExteAKXaW8Q=; fh=iwBcjOQuJQeC9NGqXlwJbGDynxCKlwwYbxsdwpHBsEc=; b=oJyDnZpxG3x0rU+YvmnaBS45+8TMDbjh4pb1XZnhn97fzOO2w8hlGOJYkGhQgppmf1 r6+TMGghF9xBE74GwyYeJFlfULcliirPknF8valnTtdOmNVh/Pz/BJbucOWpbW+gFZte E4tDi5fyKFwD4hYapW1vW2jKx+izubItZZafQ6luw85gujFvA8XdJmztWyGOXpD+YaXP kTNmQq40nbr4Xgh+9hKU3s5f0G8+93j2NocjFxIUfMAMytqMpeCBNlDPInMObBOCywXG QSXS739nDdP0XS1tMAIhh1MCabbkvqftQSorkDJvR0mH1bDF+tIkMHN/zRkbHNvgJidm 6nyw== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b="ROk3r/iU"; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519 header.b=xqN1MQLK; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=gMZI7T0R; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519; arc=pass (i=1); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: from server2.sourceware.org (server2.sourceware.org. [8.43.85.97]) by mx.google.com with ESMTPS id f6-20020ac85d06000000b00429be28d592si10534423qtx.750.2024.01.16.06.49.16 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jan 2024 06:49:16 -0800 (PST) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) client-ip=8.43.85.97; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b="ROk3r/iU"; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519 header.b=xqN1MQLK; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=gMZI7T0R; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519; arc=pass (i=1); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 8449F386180D for ; Tue, 16 Jan 2024 14:49:16 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by sourceware.org (Postfix) with ESMTPS id 2995F385773D for ; Tue, 16 Jan 2024 14:48:24 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 2995F385773D Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=suse.de ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 2995F385773D Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1705416514; cv=none; b=wOQfj+b5pVEP2wkdjfiw5A5WuOySjtUO6wuynlt/QCA/kjvWu1VqxfTxICrFC2R10VGAR9ZwRAsE6lZxXJP5jNFrezqF7uEXmjIhb2b/U23QioF1KeOzbfAv8PeLAT8WthJurU07pIhm1Zw5ikCmWhz80gSu5jMuIUWx4PX62G8= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1705416514; c=relaxed/simple; bh=Rl/ulHEVA9H3uRDcoQVfLOZo446TIuamPqNp0xJba4w=; h=DKIM-Signature:DKIM-Signature:DKIM-Signature:DKIM-Signature:Date: From:To:Subject:MIME-Version; b=f3kOuPH6cblZ0ln2TXDxT5lLhT7QCSTt7fpO4GKIJKTtvxwCemMTWRI9ndDoS+qEZRduMEyHtNCYNNfMEV7R4eBkRwMDCa66FZlM/BA3aKRugoFwwcEmiCf7LhdNNR8+e2768ViJosuxZrXibjttz1elxSfZo+sMjQkCRvltRcM= ARC-Authentication-Results: i=1; server2.sourceware.org Received: from [10.168.4.150] (unknown [10.168.4.150]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id C1FC81FBAF; Tue, 16 Jan 2024 14:48:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1705416502; h=from:from:reply-to:date:date:to:to:cc:cc:mime-version:mime-version: content-type:content-type; bh=foL8l7KptvhL4oGeNYMA1RddlSRZr77zExteAKXaW8Q=; b=ROk3r/iU6EPpAQ2b3RSgjcrOrGV34W++PWWFpLuicIoKoemodu//RBxBXh+c8Ttzy9Z6mW x5eP0Bve5XPVkPxYorr14tlubJiadV5AheRbBSE10aCwQKHMd3Zpzk62vMtH63uY4VXzpG efE9div5AhI4YrXdTyOUuq80qyOJhHI= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1705416502; h=from:from:reply-to:date:date:to:to:cc:cc:mime-version:mime-version: content-type:content-type; bh=foL8l7KptvhL4oGeNYMA1RddlSRZr77zExteAKXaW8Q=; b=xqN1MQLKZotOQHSmVborf5uh0ecY/4q23ZlqCi9JgeyCE3SUsYveMJfg1KyAmMLNxSiYoD sUKqsPaoBkxuikAg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1705416501; h=from:from:reply-to:date:date:to:to:cc:cc:mime-version:mime-version: content-type:content-type; bh=foL8l7KptvhL4oGeNYMA1RddlSRZr77zExteAKXaW8Q=; b=gMZI7T0RX27VxbRZ8GWzxEV1LcLPIsdU7Noneenyi6lX1LK5Hr19jV8pEvJe1fR93gL+Im POGGRwjFyhuLURwRQGpUgo8UrfF2zoC6QCse8q+ot7dcAFD7DHWZeHp7M9PebdrW3+QxqH qq2XnGneQ59KdqIU4kFIWFTBxQj9054= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1705416501; h=from:from:reply-to:date:date:to:to:cc:cc:mime-version:mime-version: content-type:content-type; bh=foL8l7KptvhL4oGeNYMA1RddlSRZr77zExteAKXaW8Q=; b=XhxaszfljLOtHW1WUublgEGzLy8aobdk6O1+WoqqFvcqNBfbXgPmfBMXEgxNEe8bY9Rls3 Fm62ocA9afl68tAw== Date: Tue, 16 Jan 2024 15:43:15 +0100 (CET) From: Richard Biener To: gcc-patches@gcc.gnu.org cc: tamar.christina@arm.com Subject: [PATCH] tree-optimization/113373 - work around early exit vect missing LC PHI MIME-Version: 1.0 Authentication-Results: smtp-out2.suse.de; none X-Spamd-Result: default: False [-0.60 / 50.00]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; MISSING_MID(2.50)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; RCPT_COUNT_TWO(0.00)[2]; DBL_BLOCKED_OPENRESOLVER(0.00)[tree-vect-loop.cc:url]; FUZZY_BLOCKED(0.00)[rspamd.com]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; BAYES_HAM(-3.00)[100.00%] X-Spam-Level: X-Spam-Score: -0.60 X-Spam-Status: No, score=-10.9 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, MISSING_MID, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org Message-Id: <20240116144916.8449F386180D@sourceware.org> X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788258871616983028 X-GMAIL-MSGID: 1788258871616983028 The following makes reduction epilogue code generation handle missing LC PHIs for the reductions, in particular also avoid replacing results on one exit with epilogue result from another one (but not avoiding generation of dead epilogues yet). The proper fix should be eventually to create the missing LC PHIs of course. Bootstrapped and tested on x86_64-unknown-linux-gnu, waiting for the pre-checkin CI. PR tree-optimization/113373 * tree-vect-loop.cc (vect_create_epilog_for_reduction): Only replace LC PHI results. Handle merge PHIs as seen for early breaks which miss the LC PHIs for the reduction results. * gcc.dg/vect/vect-early-break_104-pr113373.c: New testcase. --- .../vect/vect-early-break_104-pr113373.c | 19 ++++++ gcc/tree-vect-loop.cc | 67 +++++++++++-------- 2 files changed, 59 insertions(+), 27 deletions(-) create mode 100644 gcc/testsuite/gcc.dg/vect/vect-early-break_104-pr113373.c diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break_104-pr113373.c b/gcc/testsuite/gcc.dg/vect/vect-early-break_104-pr113373.c new file mode 100644 index 00000000000..1601aafb3e6 --- /dev/null +++ b/gcc/testsuite/gcc.dg/vect/vect-early-break_104-pr113373.c @@ -0,0 +1,19 @@ +/* { dg-do compile } */ +/* { dg-add-options vect_early_break } */ +/* { dg-require-effective-target vect_early_break } */ + +struct asCArray { + unsigned *array; + int length; +}; +unsigned asCReaderTranslateFunction(struct asCArray b, unsigned t) +{ + int size = 0; + for (unsigned num; num < t; num++) + { + if (num >= b.length) + __builtin_abort(); + size += b.array[num]; + } + return size; +} diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc index fedecd620e9..1c1b59dc5db 100644 --- a/gcc/tree-vect-loop.cc +++ b/gcc/tree-vect-loop.cc @@ -6013,7 +6013,6 @@ vect_create_epilog_for_reduction (loop_vec_info loop_vinfo, gimple_stmt_iterator exit_gsi; tree new_temp = NULL_TREE, new_name, new_scalar_dest; gimple *epilog_stmt = NULL; - gimple *exit_phi; tree bitsize; tree def; tree orig_name, scalar_result; @@ -6024,7 +6023,6 @@ vect_create_epilog_for_reduction (loop_vec_info loop_vinfo, int j, i; vec &scalar_results = reduc_info->reduc_scalar_results; unsigned int group_size = 1, k; - auto_vec phis; /* SLP reduction without reduction chain, e.g., # a1 = phi # b1 = phi @@ -6937,44 +6935,59 @@ vect_create_epilog_for_reduction (loop_vec_info loop_vinfo, use */ gcc_assert (live_out_stmts.size () == scalar_results.length ()); + auto_vec phis; for (k = 0; k < live_out_stmts.size (); k++) { stmt_vec_info scalar_stmt_info = vect_orig_stmt (live_out_stmts[k]); scalar_dest = gimple_get_lhs (scalar_stmt_info->stmt); - phis.create (3); /* Find the loop-closed-use at the loop exit of the original scalar result. (The reduction result is expected to have two immediate uses, one at the latch block, and one at the loop exit). For double - reductions we are looking for exit phis of the outer loop. */ - FOR_EACH_IMM_USE_FAST (use_p, imm_iter, scalar_dest) - { - if (!flow_bb_inside_loop_p (loop, gimple_bb (USE_STMT (use_p)))) + reductions we are looking for exit phis of the outer loop. + ??? For early-breaks we mess up scalar LC SSA form so also match + a PHI in a merge block and replace that PHIs use on the correct + edge. */ + FOR_EACH_IMM_USE_STMT (use_stmt, imm_iter, scalar_dest) + { + if (is_gimple_debug (use_stmt)) + continue; + else if (!flow_bb_inside_loop_p (loop, gimple_bb (use_stmt))) { - if (!is_gimple_debug (USE_STMT (use_p))) - phis.safe_push (USE_STMT (use_p)); + if (gimple_bb (use_stmt) == loop_exit->dest) + phis.safe_push (as_a (use_stmt)); + else if (gimple_code (use_stmt) == GIMPLE_PHI) + { + /* ??? This is the case running into a merge PHI with + a missing LC PHI for early exit vectorization. Replace + only the merge argument corresponding to loop_exit. */ + FOR_EACH_IMM_USE_ON_STMT (use_p, imm_iter) + if (gimple_phi_arg_edge (as_a (use_stmt), + phi_arg_index_from_use (use_p)) + ->src == loop_exit->dest) + SET_USE (use_p, scalar_results[k]); + update_stmt (use_stmt); + } } - else - { - if (double_reduc && gimple_code (USE_STMT (use_p)) == GIMPLE_PHI) - { - tree phi_res = PHI_RESULT (USE_STMT (use_p)); + else if (double_reduc && gimple_code (use_stmt) == GIMPLE_PHI) + { + tree phi_res = PHI_RESULT (use_stmt); - FOR_EACH_IMM_USE_FAST (phi_use_p, phi_imm_iter, phi_res) - { - if (!flow_bb_inside_loop_p (loop, - gimple_bb (USE_STMT (phi_use_p))) - && !is_gimple_debug (USE_STMT (phi_use_p))) - phis.safe_push (USE_STMT (phi_use_p)); - } - } - } - } + FOR_EACH_IMM_USE_FAST (phi_use_p, phi_imm_iter, phi_res) + { + if (!flow_bb_inside_loop_p (outer_loop, + gimple_bb (USE_STMT (phi_use_p))) + && !is_gimple_debug (USE_STMT (phi_use_p))) + phis.safe_push (as_a (USE_STMT (phi_use_p))); + } + } + } + gphi *exit_phi; FOR_EACH_VEC_ELT (phis, i, exit_phi) { - /* Replace the uses: */ - orig_name = PHI_RESULT (exit_phi); + /* Replace the uses: */ + orig_name = gimple_phi_result (exit_phi); /* Look for a single use at the target of the skip edge. */ if (unify_with_main_loop_p) @@ -6995,7 +7008,7 @@ vect_create_epilog_for_reduction (loop_vec_info loop_vinfo, } } - phis.release (); + phis.truncate (0); } }