From patchwork Tue Jan 31 13:06:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Sandiford X-Patchwork-Id: 50824 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2740097wrn; Tue, 31 Jan 2023 05:08:15 -0800 (PST) X-Google-Smtp-Source: AMrXdXuMVUeZB49M+wNXkAflV15wpp7CAYTgABnmaQwy7ESuRBlv6rtEFCj8xeZvgGGrD7FKYqmG X-Received: by 2002:a17:906:a182:b0:872:14ea:1a7b with SMTP id s2-20020a170906a18200b0087214ea1a7bmr54483441ejy.13.1675170495368; Tue, 31 Jan 2023 05:08:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675170495; cv=none; d=google.com; s=arc-20160816; b=N+mGHQ30uWYETU/YVqjWLdUW2Z5emrn4kZd4Lf4yYz4LAqfDrwn3gqrAHQ2fV9Z+Sz 5d+9v/hYGwhF3VMnz2gsya+csL3GJumYpIrUEreOekcRh0Hl5JmS6rrPLHYdkuu3DiT0 vZXEYZVJcf/BtngYwgVtbrMhFnlLaDw3S0FkS5gXR9Gn6y5hf0chcJAaupny+Eu0+ALr r3MXfaFAIwl0eluYGwDpxw0y498raEKeaIMS+2/4/cut5lBF8ygU+JI33Bkm2N2/h6xX 1nPJx/pRGxiRG3FTHUcgdNBbuOSjBp46Zyj40d1b4x+XdZcWp1obUT2Ipd+9aiuUAxld lBRg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:reply-to:from:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence:mime-version :user-agent:message-id:date:subject:mail-followup-to:to:dmarc-filter :delivered-to:dkim-signature:dkim-filter; bh=/Xp3Tzf0MVBXy8YFmvJ7joMW2ejhLSofTNfJgqIRhZ0=; b=Dp48vf3ob1oiNlxrfz8PfvsGVFLChTipi9tHLb6CQIfFFe3ZAwpOtEeWIio6ekhLfn richmZLpQ7tlkG/Zd9lQQcftKmFamYALnchfYyVEXAlA0bbXLhibX5YfEHzuwCtznasl UuyuQgIw1JAbl/HLcdvi1AOYayAurBQdlj8e15Mfkg13F3JsVzJUQQ+rVPLSa+SRLD5S 5oerN5Qhk/SL2wNLpjs0xlEnKQ/DcZPZkDmAClstau5f0QrqXMQuI+mes/AGUurCS9BT GZTSEAVusj7HL/XTo8mK+to1tFiw82ipuKAMvdcNe/zeeU5GsnGNuMvyIqQL/MxdgweG qvKA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=c00e9l8Q; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gnu.org Received: from sourceware.org (server2.sourceware.org. [2620:52:3:1:0:246e:9693:128c]) by mx.google.com with ESMTPS id bp13-20020a170907918d00b0087b12541eb1si13123023ejb.74.2023.01.31.05.08.15 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 31 Jan 2023 05:08:15 -0800 (PST) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) client-ip=2620:52:3:1:0:246e:9693:128c; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=c00e9l8Q; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gnu.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id DEC583858404 for ; Tue, 31 Jan 2023 13:07:47 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org DEC583858404 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1675170467; bh=/Xp3Tzf0MVBXy8YFmvJ7joMW2ejhLSofTNfJgqIRhZ0=; h=To:Subject:Date:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=c00e9l8QYOz2HH7m99MDYnjsyPv5AwwUyb3V/WH+LOogGhqgiyCjhvbFPtHE8Lnkq efKmPVXZsxKznZBMw32Oc2JCcO1QGlLQLqzS9+IJgQt/dK82TlXPFYK7X+m7oqWKE/ VDIvcrhOoUgtiz2RkBHGgOBGD1xrOyhH7cPLmN9I= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by sourceware.org (Postfix) with ESMTP id 3491E3858D28 for ; Tue, 31 Jan 2023 13:06:58 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 3491E3858D28 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A14F32F4 for ; Tue, 31 Jan 2023 05:07:39 -0800 (PST) Received: from localhost (e121540-lin.manchester.arm.com [10.32.99.50]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 3C4243F71E for ; Tue, 31 Jan 2023 05:06:57 -0800 (PST) To: gcc-patches@gcc.gnu.org Mail-Followup-To: gcc-patches@gcc.gnu.org, richard.sandiford@arm.com Subject: [PATCH] vect: Fix single def-use cycle for ifn reductions [PR108608] Date: Tue, 31 Jan 2023 13:06:56 +0000 Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 X-Spam-Status: No, score=-36.3 required=5.0 tests=BAYES_00, GIT_PATCH_0, KAM_DMARC_NONE, KAM_DMARC_STATUS, KAM_LAZY_DOMAIN_SECURITY, KAM_SHORT, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Richard Sandiford via Gcc-patches From: Richard Sandiford Reply-To: Richard Sandiford Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org Sender: "Gcc-patches" X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756543577187230527?= X-GMAIL-MSGID: =?utf-8?q?1756543577187230527?= The patch that added support for fmin/fmax reductions didn't handle single def-use cycles. In some ways, this seems like going out of our way to make things slower, but that's a discussion for another day. Tested on aarch64-linux-gnu & x86_64-linux-gnu. OK for trunk and the GCC 12 branch? Richard gcc/ PR tree-optimization/108608 * tree-vect-loop.cc (vect_transform_reduction): Handle single def-use cycles that involve function calls rather than tree codes. gcc/testsuite/ PR tree-optimization/108608 * gcc.dg/vect/pr108608.c: New test. * gcc.target/aarch64/sve/pr108608-1.c: Likewise. --- gcc/testsuite/gcc.dg/vect/pr108608.c | 24 +++++++++++++++++++ .../gcc.target/aarch64/sve/pr108608-1.c | 9 +++++++ gcc/tree-vect-loop.cc | 22 ++++++++++------- 3 files changed, 46 insertions(+), 9 deletions(-) create mode 100644 gcc/testsuite/gcc.dg/vect/pr108608.c create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/pr108608-1.c diff --git a/gcc/testsuite/gcc.dg/vect/pr108608.c b/gcc/testsuite/gcc.dg/vect/pr108608.c new file mode 100644 index 00000000000..e968141ba03 --- /dev/null +++ b/gcc/testsuite/gcc.dg/vect/pr108608.c @@ -0,0 +1,24 @@ +#include "tree-vect.h" + +double __attribute__((noipa)) +foo (double m, float *ptr) +{ + for (int i = 0; i < 256; i++) + m = __builtin_fmax (m, ptr[i]); + return m; +} + +int +main (void) +{ + check_vect (); + float ptr[256]; + for (int j = 0; j < 16; ++j) + { + for (int i = 0; i < 256; ++i) + ptr[i] = i == 128 + j ? 2 + j : i == 161 ? 1 : 0; + if (foo (0, ptr) != 2 + j) + __builtin_abort (); + } + return 0; +} diff --git a/gcc/testsuite/gcc.target/aarch64/sve/pr108608-1.c b/gcc/testsuite/gcc.target/aarch64/sve/pr108608-1.c new file mode 100644 index 00000000000..0a7d485e047 --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sve/pr108608-1.c @@ -0,0 +1,9 @@ +/* { dg-options "-O3" } */ + +double __attribute__((noipa)) +foo (double m, float *ptr) +{ + for (int i = 0; i < 256; i++) + m = __builtin_fmax (m, ptr[i]); + return m; +} diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc index f0801c23671..f03af1efd0f 100644 --- a/gcc/tree-vect-loop.cc +++ b/gcc/tree-vect-loop.cc @@ -7755,8 +7755,6 @@ vect_transform_reduction (loop_vec_info loop_vinfo, gimple_match_op op; if (!gimple_extract_op (stmt_info->stmt, &op)) gcc_unreachable (); - gcc_assert (op.code.is_tree_code ()); - auto code = tree_code (op.code); /* All uses but the last are expected to be defined in the loop. The last use is the reduction variable. In case of nested cycle this @@ -7778,7 +7776,8 @@ vect_transform_reduction (loop_vec_info loop_vinfo, vec_num = 1; } - internal_fn cond_fn = get_conditional_internal_fn (code); + code_helper code = canonicalize_code (op.code, op.type); + internal_fn cond_fn = get_conditional_internal_fn (code, op.type); vec_loop_masks *masks = &LOOP_VINFO_MASKS (loop_vinfo); bool mask_by_cond_expr = use_mask_by_cond_expr_p (code, cond_fn, vectype_in); @@ -7802,9 +7801,10 @@ vect_transform_reduction (loop_vec_info loop_vinfo, if (reduction_type == FOLD_LEFT_REDUCTION) { internal_fn reduc_fn = STMT_VINFO_REDUC_FN (reduc_info); + gcc_assert (code.is_tree_code ()); return vectorize_fold_left_reduction - (loop_vinfo, stmt_info, gsi, vec_stmt, slp_node, reduc_def_phi, code, - reduc_fn, op.ops, vectype_in, reduc_index, masks); + (loop_vinfo, stmt_info, gsi, vec_stmt, slp_node, reduc_def_phi, + tree_code (code), reduc_fn, op.ops, vectype_in, reduc_index, masks); } bool single_defuse_cycle = STMT_VINFO_FORCE_SINGLE_CYCLE (reduc_info); @@ -7814,7 +7814,7 @@ vect_transform_reduction (loop_vec_info loop_vinfo, || code == SAD_EXPR); /* Create the destination vector */ - tree scalar_dest = gimple_assign_lhs (stmt_info->stmt); + tree scalar_dest = gimple_get_lhs (stmt_info->stmt); tree vec_dest = vect_create_destination_var (scalar_dest, vectype_out); vect_get_vec_defs (loop_vinfo, stmt_info, slp_node, ncopies, @@ -7849,7 +7849,7 @@ vect_transform_reduction (loop_vec_info loop_vinfo, /* Make sure that the reduction accumulator is vop[0]. */ if (reduc_index == 1) { - gcc_assert (commutative_tree_code (code)); + gcc_assert (commutative_binary_op_p (code, op.type)); std::swap (vop[0], vop[1]); } tree mask = vect_get_loop_mask (gsi, masks, vec_num * ncopies, @@ -7877,11 +7877,15 @@ vect_transform_reduction (loop_vec_info loop_vinfo, if (emulated_mixed_dot_prod) new_stmt = vect_emulate_mixed_dot_prod (loop_vinfo, stmt_info, gsi, vec_dest, vop); + else if (code.is_internal_fn ()) + new_stmt = gimple_build_call_internal (internal_fn (code), + op.num_ops, + vop[0], vop[1], vop[2]); else - new_stmt = gimple_build_assign (vec_dest, code, + new_stmt = gimple_build_assign (vec_dest, tree_code (op.code), vop[0], vop[1], vop[2]); new_temp = make_ssa_name (vec_dest, new_stmt); - gimple_assign_set_lhs (new_stmt, new_temp); + gimple_set_lhs (new_stmt, new_temp); vect_finish_stmt_generation (loop_vinfo, stmt_info, new_stmt, gsi); }