From patchwork Mon Sep 18 11:37:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lehua Ding X-Patchwork-Id: 141365 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp2586439vqi; Mon, 18 Sep 2023 04:38:26 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFD6pqPFyAP/wEjYfz0BgOyUORL0LKhdgE6CWR1wWbZVhSLbNAawCUZbvam4xH/g+9q6B3W X-Received: by 2002:a17:907:2716:b0:9a5:c5a8:a1a0 with SMTP id w22-20020a170907271600b009a5c5a8a1a0mr7485926ejk.49.1695037105728; Mon, 18 Sep 2023 04:38:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695037105; cv=none; d=google.com; s=arc-20160816; b=XP5A56sAiqlz2hTzkZQIP4dN31yIM/DoMT57G5lRqMMSe0AzBgJqbLwDppFd9Nqjxu 75ND1g7anXhDXOK3oFwIk1logii8BaOvChk4olpGmla96u4AsjUpycSzIyaLe8a/VkxE cjJG+u3PaBxxgdx7NMna5XmiT5e6dT1rvVpy2MlT6YkfOgesx/wqrSMz5Fz3fe9sQqeT X7iqhhTlohYVdyYnCAkSp2JseOQ/G9acjKsrwDmYN6nh49/0ztxpWUNjTKexjq9PyG7f sXOcvdVg5/HlvUd7RfxeAThEnScXkq1pgw/snzZSaGsy5/ceGz0/rbID8HvUKnNFmYd7 9Smw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:feedback-id :content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:dmarc-filter:delivered-to; bh=eyU0vkO2k7WirjE7uBBeBWyIn4e3pvPmE7QwbduwOAI=; fh=x/Q0OlwHuvCZ3FpkiZPiUSvevOYVxUAi4aNnf76mUPQ=; b=GwAkbXGN7Bk/x2RKGw+2js6hV3LvMrIMUt0gJoRGf/heOFJhQoSX3U+NzuqBH/eKcn XasUQ6MRMmvEbP0AVvwpgRq9wAcXe3CVRY+9CiKEAF/3Pjdw5Cz16oqInbhW+5Ck0arr LPJzxcH030al2pi6uzGU4461RCrrgAIYRf+kCIv16eWVHwWh8vODrtoH7Fq6xJ7G1nSy ELENJpXWhkoCw0QvRRs+TAhHVOCw+QasSt1uN267U7gUXAPrX0HLzrL4p7kSrAS2Q1xU ubeoSLXE1fOx5ttjpnO3MES/joZOaqLiYhQOWzNMVfk8NzF6cqLxquL8qD2moUHqvkOb /JXg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (server2.sourceware.org. [2620:52:3:1:0:246e:9693:128c]) by mx.google.com with ESMTPS id lg13-20020a170906f88d00b009a2235ed499si8342229ejb.686.2023.09.18.04.38.25 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Sep 2023 04:38:25 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) client-ip=2620:52:3:1:0:246e:9693:128c; Authentication-Results: mx.google.com; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id E570E385CCA4 for ; Mon, 18 Sep 2023 11:38:06 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtpbgeu2.qq.com (smtpbgeu2.qq.com [18.194.254.142]) by sourceware.org (Postfix) with ESMTPS id 634C23858C60 for ; Mon, 18 Sep 2023 11:37:25 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 634C23858C60 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivai.ai Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivai.ai X-QQ-mid: bizesmtp74t1695037038talenjvv Received: from rios-cad121.hadoop.rioslab.org ( [58.60.1.9]) by bizesmtp.qq.com (ESMTP) with id ; Mon, 18 Sep 2023 19:37:17 +0800 (CST) X-QQ-SSF: 01400000000000C0F000000A0000000 X-QQ-FEAT: v/I4QfEc9z0wpnQUkaooAOz01ZOmSLM0tDwviBM/22jO/eaR2uLwjjYAC5Q7g 89WhfuQcF2A3p2Lx+XcooGbA/oSXLRWo+ROZ54oKowod1rhXIybwliUJqA4euh+pC4XRGh5 lsrVq8HZWejhhcvbbTx5yvv61s15WHnc6/nV8OglxGfX/4SJnJKeKkMksDBSb1exbx7hMXi 3/zYQImuA62OqeFpdCf+gIAn6A+pWcKBshYPAPHF5FC9QZ7CdMTr0JkYuwIrFlqMO9xPXfU /PyrdDakp5KASTBoBxGt7NnT6s5TzFlvubROynoFJhGdzbphaQhl4gFJLYhfjtYLkQCR2lM eMTs5vzQpXzKkMfuc5MIUd95DJN5oQviReJ2LJLGLFUXatIruPz3xlhxAxYx7DIqNdb/iag X-QQ-GoodBg: 2 X-BIZMAIL-ID: 8830431632993973990 From: Lehua Ding To: gcc-patches@gcc.gnu.org Cc: juzhe.zhong@rivai.ai, kito.cheng@gmail.com, rdapp.gcc@gmail.com, palmer@rivosinc.com, jeffreyalaw@gmail.com, lehua.ding@rivai.ai Subject: [PATCH] RISC-V: Refactor and cleanup fma patterns Date: Mon, 18 Sep 2023 19:37:17 +0800 Message-Id: <20230918113717.3538058-1-lehua.ding@rivai.ai> X-Mailer: git-send-email 2.36.3 MIME-Version: 1.0 X-QQ-SENDSIZE: 520 Feedback-ID: bizesmtp:rivai.ai:qybglogicsvrgz:qybglogicsvrgz6a-0 X-Spam-Status: No, score=-11.0 required=5.0 tests=BAYES_00, GIT_PATCH_0, KAM_ASCII_DIVIDERS, KAM_DMARC_STATUS, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, SPF_PASS, TXREP, T_SPF_HELO_TEMPERROR autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org Sender: "Gcc-patches" X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777375228181594124 X-GMAIL-MSGID: 1777375228181594124 At present, FMA autovec's patterns do not fully use the corresponding pattern in vector.md. The previous reason is that the merge operand of pattern in vector.md cannot be VUNDEF. Now allowing it to be VUNDEF, reunify insn used for reload pass into vector.md, and the corresponding vlmax pattern in autovec.md is used for combine. This patch also refactors the corresponding combine pattern inside autovec-opt.md and removes the unused ones. gcc/ChangeLog: * config/riscv/autovec-opt.md (*_fma): Removed old combine patterns. (*single_mult_plus): Ditto. (*double_mult_plus): Ditto. (*sign_zero_extend_fma): Ditto. (*zero_sign_extend_fma): Ditto. (*double_widen_fma): Ditto. (*single_widen_fma): Ditto. (*double_widen_fnma): Ditto. (*single_widen_fnma): Ditto. (*double_widen_fms): Ditto. (*single_widen_fms): Ditto. (*double_widen_fnms): Ditto. (*single_widen_fnms): Ditto. (*reduc_plus_scal_): Adjust name. (*widen_reduc_plus_scal_): Adjust name. (*dual_widen_fma): New combine pattern. (*dual_widen_fmasu): Ditto. (*dual_widen_fmaus): Ditto. (*dual_fma): Ditto. (*single_fma): Ditto. (*dual_fnma): Ditto. (*single_fnma): Ditto. (*dual_fms): Ditto. (*single_fms): Ditto. (*dual_fnms): Ditto. (*single_fnms): Ditto. * config/riscv/autovec.md (fma4): Reafctor fma pattern. (*fma): Removed. (fnma4): Reafctor. (*fnma): Removed. (*fma): Removed. (*fnma): Removed. (fms4): Reafctor. (*fms): Removed. (fnms4): Reafctor. (*fnms): Removed. * config/riscv/riscv-protos.h (prepare_ternary_operands): Adjust prototype. * config/riscv/riscv-v.cc (prepare_ternary_operands): Refactor. * config/riscv/vector.md (*pred_mul_plus_undef): New pattern. (*pred_mul_plus): Removed. (*pred_mul_plus_scalar): Removed. (*pred_mul_plus_extended_scalar): Removed. (*pred_minus_mul_undef): New pattern. (*pred_minus_mul): Removed. (*pred_minus_mul_scalar): Removed. (*pred_minus_mul_extended_scalar): Removed. (*pred_mul__undef): New pattern. (*pred_mul_): Removed. (*pred_mul__scalar): Removed. (*pred_mul_neg__undef): New pattern. (*pred_mul_neg_): Removed. (*pred_mul_neg__scalar): Removed. --- gcc/config/riscv/autovec-opt.md | 736 ++++++++++++++------------------ gcc/config/riscv/autovec.md | 301 ++++--------- gcc/config/riscv/riscv-protos.h | 2 +- gcc/config/riscv/riscv-v.cc | 14 +- gcc/config/riscv/vector.md | 439 ++++++------------- 5 files changed, 528 insertions(+), 964 deletions(-) -- 2.36.3 diff --git a/gcc/config/riscv/autovec-opt.md b/gcc/config/riscv/autovec-opt.md index df516849527..c94cd0ae087 100644 --- a/gcc/config/riscv/autovec-opt.md +++ b/gcc/config/riscv/autovec-opt.md @@ -110,166 +110,6 @@ [(set_attr "type" "vmalu") (set_attr "mode" "")]) -;; ========================================================================= -;; == Widening Ternary arithmetic -;; ========================================================================= - -;; ------------------------------------------------------------------------- -;; ---- [INT] VWMACC -;; ------------------------------------------------------------------------- -;; Includes: -;; - vwmacc.vv -;; - vwmaccu.vv -;; ------------------------------------------------------------------------- - -;; Combine ext + ext + fma ===> widen fma. -;; Most of circumstantces, LoopVectorizer will generate the following IR: -;; vect__8.64_40 = (vector([4,4]) int) vect__7.63_41; -;; vect__11.68_35 = (vector([4,4]) int) vect__10.67_36; -;; vect__13.70_33 = .FMA (vect__11.68_35, vect__8.64_40, vect__4.60_45); -(define_insn_and_split "*_fma" - [(set (match_operand:VWEXTI 0 "register_operand") - (plus:VWEXTI - (mult:VWEXTI - (any_extend:VWEXTI - (match_operand: 2 "register_operand")) - (any_extend:VWEXTI - (match_operand: 3 "register_operand"))) - (match_operand:VWEXTI 1 "register_operand")))] - "TARGET_VECTOR && can_create_pseudo_p ()" - "#" - "&& 1" - [(const_int 0)] - { - riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_plus (, mode), - riscv_vector::WIDEN_TERNARY_OP, operands); - DONE; - } - [(set_attr "type" "viwmuladd") - (set_attr "mode" "")]) - -;; This helps to match ext + fma. -(define_insn_and_split "*single_mult_plus" - [(set (match_operand:VWEXTI 0 "register_operand") - (plus:VWEXTI - (mult:VWEXTI - (any_extend:VWEXTI - (match_operand: 2 "register_operand")) - (match_operand:VWEXTI 3 "register_operand")) - (match_operand:VWEXTI 1 "register_operand")))] - "TARGET_VECTOR && can_create_pseudo_p ()" - "#" - "&& 1" - [(const_int 0)] - { - insn_code icode = code_for_pred_vf2 (, mode); - rtx tmp = gen_reg_rtx (mode); - rtx ext_ops[] = {tmp, operands[2]}; - riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP, ext_ops); - - rtx dst = expand_ternary_op (mode, fma_optab, tmp, operands[3], - operands[1], operands[0], 0); - emit_move_insn (operands[0], dst); - DONE; - } - [(set_attr "type" "viwmuladd") - (set_attr "mode" "")]) - -;; Combine ext + ext + mult + plus ===> widen fma. -;; We have some special cases generated by LoopVectorizer: -;; vect__8.18_46 = (vector([8,8]) signed short) vect__7.17_47; -;; vect__11.22_41 = (vector([8,8]) signed short) vect__10.21_42; -;; vect__12.23_40 = vect__11.22_41 * vect__8.18_46; -;; vect__14.25_38 = vect__13.24_39 + vect__5.14_51; -;; This situation doesn't generate FMA IR. -(define_insn_and_split "*double_mult_plus" - [(set (match_operand:VWEXTI 0 "register_operand") - (if_then_else:VWEXTI - (unspec: - [(match_operand: 1 "vector_mask_operand") - (match_operand 6 "vector_length_operand") - (match_operand 7 "const_int_operand") - (match_operand 8 "const_int_operand") - (match_operand 9 "const_int_operand") - (reg:SI VL_REGNUM) - (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE) - (plus:VWEXTI - (if_then_else:VWEXTI - (unspec: - [(match_dup 1) - (match_dup 6) - (match_dup 7) - (match_dup 8) - (match_dup 9) - (reg:SI VL_REGNUM) - (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE) - (mult:VWEXTI - (any_extend:VWEXTI - (match_operand: 4 "register_operand")) - (any_extend:VWEXTI - (match_operand: 5 "register_operand"))) - (match_operand:VWEXTI 2 "vector_undef_operand")) - (match_operand:VWEXTI 3 "register_operand")) - (match_dup 2)))] - "TARGET_VECTOR && can_create_pseudo_p ()" - "#" - "&& 1" - [(const_int 0)] - { - emit_insn (gen_pred_widen_mul_plus (, mode, operands[0], - operands[1], operands[3], operands[4], - operands[5], operands[6], operands[7], - operands[8], operands[9])); - DONE; - } - [(set_attr "type" "viwmuladd") - (set_attr "mode" "")]) - -;; Combine sign_extend + zero_extend + fma ===> widen fma (su). -(define_insn_and_split "*sign_zero_extend_fma" - [(set (match_operand:VWEXTI 0 "register_operand") - (plus:VWEXTI - (mult:VWEXTI - (sign_extend:VWEXTI - (match_operand: 2 "register_operand")) - (zero_extend:VWEXTI - (match_operand: 3 "register_operand"))) - (match_operand:VWEXTI 1 "register_operand")))] - "TARGET_VECTOR && can_create_pseudo_p ()" - "#" - "&& 1" - [(const_int 0)] - { - riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_plussu (mode), - riscv_vector::WIDEN_TERNARY_OP, operands); - DONE; - } - [(set_attr "type" "viwmuladd") - (set_attr "mode" "")]) - -;; This helps to match zero_extend + sign_extend + fma. -(define_insn_and_split "*zero_sign_extend_fma" - [(set (match_operand:VWEXTI 0 "register_operand") - (plus:VWEXTI - (mult:VWEXTI - (zero_extend:VWEXTI - (match_operand: 2 "register_operand")) - (sign_extend:VWEXTI - (match_operand: 3 "register_operand"))) - (match_operand:VWEXTI 1 "register_operand")))] - "TARGET_VECTOR && can_create_pseudo_p ()" - "#" - "&& 1" - [(const_int 0)] - { - rtx ops[] = {operands[0], operands[1], operands[3], operands[2]}; - riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_plussu (mode), - riscv_vector::WIDEN_TERNARY_OP, ops); - DONE; - } - [(set_attr "type" "viwmuladd") - (set_attr "mode" "")]) - ;; ------------------------------------------------------------------------- ;; ---- [INT] Binary narrow shifts. ;; ------------------------------------------------------------------------- @@ -345,269 +185,9 @@ [(set_attr "type" "vimovvx") (set_attr "mode" "")]) -;; ------------------------------------------------------------------------- -;; ---- [FP] VFWMACC -;; ------------------------------------------------------------------------- -;; Includes: -;; - vfwmacc.vv -;; ------------------------------------------------------------------------- - -;; Combine ext + ext + fma ===> widen fma. -;; Most of circumstantces, LoopVectorizer will generate the following IR: -;; vect__8.176_40 = (vector([2,2]) double) vect__7.175_41; -;; vect__11.180_35 = (vector([2,2]) double) vect__10.179_36; -;; vect__13.182_33 = .FMA (vect__11.180_35, vect__8.176_40, vect__4.172_45); -(define_insn_and_split "*double_widen_fma" - [(set (match_operand:VWEXTF 0 "register_operand") - (unspec:VWEXTF - [(fma:VWEXTF - (float_extend:VWEXTF - (match_operand: 2 "register_operand")) - (float_extend:VWEXTF - (match_operand: 3 "register_operand")) - (match_operand:VWEXTF 1 "register_operand")) - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))] - "TARGET_VECTOR && can_create_pseudo_p ()" - "#" - "&& 1" - [(const_int 0)] - { - riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul (PLUS, mode), - riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, operands); - DONE; - } - [(set_attr "type" "vfwmuladd") - (set_attr "mode" "") - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))]) - -;; This helps to match ext + fma. -(define_insn_and_split "*single_widen_fma" - [(set (match_operand:VWEXTF 0 "register_operand") - (unspec:VWEXTF - [(fma:VWEXTF - (float_extend:VWEXTF - (match_operand: 2 "register_operand")) - (match_operand:VWEXTF 3 "register_operand") - (match_operand:VWEXTF 1 "register_operand")) - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))] - "TARGET_VECTOR && can_create_pseudo_p ()" - "#" - "&& 1" - [(const_int 0)] - { - insn_code icode = code_for_pred_extend (mode); - rtx tmp = gen_reg_rtx (mode); - rtx ext_ops[] = {tmp, operands[2]}; - riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP, ext_ops); - - rtx dst = expand_ternary_op (mode, fma_optab, tmp, operands[3], - operands[1], operands[0], 0); - emit_move_insn (operands[0], dst); - DONE; - } - [(set_attr "type" "vfwmuladd") - (set_attr "mode" "") - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))]) - -;; ------------------------------------------------------------------------- -;; ---- [FP] VFWNMSAC -;; ------------------------------------------------------------------------- -;; Includes: -;; - vfwnmsac.vv -;; ------------------------------------------------------------------------- - -;; Combine ext + ext + fnma ===> widen fnma. -;; Most of circumstantces, LoopVectorizer will generate the following IR: -;; vect__8.176_40 = (vector([2,2]) double) vect__7.175_41; -;; vect__11.180_35 = (vector([2,2]) double) vect__10.179_36; -;; vect__13.182_33 = .FNMA (vect__11.180_35, vect__8.176_40, vect__4.172_45); -(define_insn_and_split "*double_widen_fnma" - [(set (match_operand:VWEXTF 0 "register_operand") - (unspec:VWEXTF - [(fma:VWEXTF - (neg:VWEXTF - (float_extend:VWEXTF - (match_operand: 2 "register_operand"))) - (float_extend:VWEXTF - (match_operand: 3 "register_operand")) - (match_operand:VWEXTF 1 "register_operand")) - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))] - "TARGET_VECTOR && can_create_pseudo_p ()" - "#" - "&& 1" - [(const_int 0)] - { - riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_neg (PLUS, mode), - riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, operands); - DONE; - } - [(set_attr "type" "vfwmuladd") - (set_attr "mode" "") - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))]) - -;; This helps to match ext + fnma. -(define_insn_and_split "*single_widen_fnma" - [(set (match_operand:VWEXTF 0 "register_operand") - (unspec:VWEXTF - [(fma:VWEXTF - (neg:VWEXTF - (float_extend:VWEXTF - (match_operand: 2 "register_operand"))) - (match_operand:VWEXTF 3 "register_operand") - (match_operand:VWEXTF 1 "register_operand")) - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))] - "TARGET_VECTOR && can_create_pseudo_p ()" - "#" - "&& 1" - [(const_int 0)] - { - insn_code icode = code_for_pred_extend (mode); - rtx tmp = gen_reg_rtx (mode); - rtx ext_ops[] = {tmp, operands[2]}; - riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP, ext_ops); - - rtx dst = expand_ternary_op (mode, fnma_optab, tmp, operands[3], - operands[1], operands[0], 0); - emit_move_insn (operands[0], dst); - DONE; - } - [(set_attr "type" "vfwmuladd") - (set_attr "mode" "") - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))]) - -;; ------------------------------------------------------------------------- -;; ---- [FP] VFWMSAC -;; ------------------------------------------------------------------------- -;; Includes: -;; - vfwmsac.vv -;; ------------------------------------------------------------------------- - -;; Combine ext + ext + fms ===> widen fms. -;; Most of circumstantces, LoopVectorizer will generate the following IR: -;; vect__8.176_40 = (vector([2,2]) double) vect__7.175_41; -;; vect__11.180_35 = (vector([2,2]) double) vect__10.179_36; -;; vect__13.182_33 = .FMS (vect__11.180_35, vect__8.176_40, vect__4.172_45); -(define_insn_and_split "*double_widen_fms" - [(set (match_operand:VWEXTF 0 "register_operand") - (unspec:VWEXTF - [(fma:VWEXTF - (float_extend:VWEXTF - (match_operand: 2 "register_operand")) - (float_extend:VWEXTF - (match_operand: 3 "register_operand")) - (neg:VWEXTF - (match_operand:VWEXTF 1 "register_operand"))) - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))] - "TARGET_VECTOR && can_create_pseudo_p ()" - "#" - "&& 1" - [(const_int 0)] - { - riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul (MINUS, mode), - riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, operands); - DONE; - } - [(set_attr "type" "vfwmuladd") - (set_attr "mode" "") - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))]) - -;; This helps to match ext + fms. -(define_insn_and_split "*single_widen_fms" - [(set (match_operand:VWEXTF 0 "register_operand") - (unspec:VWEXTF - [(fma:VWEXTF - (float_extend:VWEXTF - (match_operand: 2 "register_operand")) - (match_operand:VWEXTF 3 "register_operand") - (neg:VWEXTF - (match_operand:VWEXTF 1 "register_operand"))) - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))] - "TARGET_VECTOR && can_create_pseudo_p ()" - "#" - "&& 1" - [(const_int 0)] - { - insn_code icode = code_for_pred_extend (mode); - rtx tmp = gen_reg_rtx (mode); - rtx ext_ops[] = {tmp, operands[2]}; - riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP, ext_ops); - - rtx dst = expand_ternary_op (mode, fms_optab, tmp, operands[3], - operands[1], operands[0], 0); - emit_move_insn (operands[0], dst); - DONE; - } - [(set_attr "type" "vfwmuladd") - (set_attr "mode" "") - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))]) - -;; ------------------------------------------------------------------------- -;; ---- [FP] VFWNMACC -;; ------------------------------------------------------------------------- -;; Includes: -;; - vfwnmacc.vv -;; ------------------------------------------------------------------------- - -;; Combine ext + ext + fnms ===> widen fnms. -;; Most of circumstantces, LoopVectorizer will generate the following IR: -;; vect__8.176_40 = (vector([2,2]) double) vect__7.175_41; -;; vect__11.180_35 = (vector([2,2]) double) vect__10.179_36; -;; vect__13.182_33 = .FNMS (vect__11.180_35, vect__8.176_40, vect__4.172_45); -(define_insn_and_split "*double_widen_fnms" - [(set (match_operand:VWEXTF 0 "register_operand") - (unspec:VWEXTF - [(fma:VWEXTF - (neg:VWEXTF - (float_extend:VWEXTF - (match_operand: 2 "register_operand"))) - (float_extend:VWEXTF - (match_operand: 3 "register_operand")) - (neg:VWEXTF - (match_operand:VWEXTF 1 "register_operand"))) - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))] - "TARGET_VECTOR && can_create_pseudo_p ()" - "#" - "&& 1" - [(const_int 0)] - { - riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_neg (MINUS, mode), - riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, operands); - DONE; - } - [(set_attr "type" "vfwmuladd") - (set_attr "mode" "") - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))]) - -;; This helps to match ext + fnms. -(define_insn_and_split "*single_widen_fnms" - [(set (match_operand:VWEXTF 0 "register_operand") - (unspec:VWEXTF - [(fma:VWEXTF - (neg:VWEXTF - (float_extend:VWEXTF - (match_operand: 2 "register_operand"))) - (match_operand:VWEXTF 3 "register_operand") - (neg:VWEXTF - (match_operand:VWEXTF 1 "register_operand"))) - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA))] - "TARGET_VECTOR && can_create_pseudo_p ()" - "#" - "&& 1" - [(const_int 0)] - { - insn_code icode = code_for_pred_extend (mode); - rtx tmp = gen_reg_rtx (mode); - rtx ext_ops[] = {tmp, operands[2]}; - riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP, ext_ops); - - rtx dst = expand_ternary_op (mode, fnms_optab, tmp, operands[3], - operands[1], operands[0], 0); - emit_move_insn (operands[0], dst); - DONE; - } - [(set_attr "type" "vfwmuladd") - (set_attr "mode" "") - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))]) +;; ============================================================================= +;; All combine patterns for combine pass. +;; ============================================================================= ;; ============================================================================= ;; Combine op + vmerge to cond_op @@ -1197,7 +777,7 @@ [(set_attr "type" "vfwmul")]) ;; Combine extend + vredsum to vwredsum[u] -(define_insn_and_split "*reduc_plus_scal_" +(define_insn_and_split "*widen_reduc_plus_scal_" [(set (match_operand: 0 "register_operand") (unspec: [ (any_extend: @@ -1215,7 +795,7 @@ [(set_attr "type" "vector")]) ;; Combine extend + vfredusum to vfwredusum -(define_insn_and_split "*reduc_plus_scal_" +(define_insn_and_split "*widen_reduc_plus_scal_" [(set (match_operand: 0 "register_operand") (unspec: [ (float_extend: @@ -1278,6 +858,312 @@ } [(set_attr "type" "vector")]) +;; ============================================================================= +;; Combine extend + ternop to widen_ternop +;; ============================================================================= + +;; Combine ext + fma(vmacc,vmadd) to widen_fma (vwmacc) +(define_insn_and_split "*dual_widen_fma" + [(set (match_operand:VWEXTI 0 "register_operand") + (plus:VWEXTI + (mult:VWEXTI + (any_extend:VWEXTI + (match_operand: 2 "register_operand")) + (any_extend:VWEXTI + (match_operand: 3 "register_operand"))) + (match_operand:VWEXTI 1 "register_operand")))] + "TARGET_VECTOR && can_create_pseudo_p ()" + "#" + "&& 1" + [(const_int 0)] + { + rtx ops[] = {operands[0], operands[1], operands[2], operands[3]}; + riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_plus (, mode), + riscv_vector::WIDEN_TERNARY_OP, ops); + DONE; + } + [(set_attr "type" "viwmuladd")]) + +;; Combine sign_extend + zero_extend + fma to widen_fma (vwmaccsu) +(define_insn_and_split "*dual_widen_fmasu" + [(set (match_operand:VWEXTI 0 "register_operand") + (plus:VWEXTI + (mult:VWEXTI + (sign_extend:VWEXTI + (match_operand: 2 "register_operand")) + (zero_extend:VWEXTI + (match_operand: 3 "register_operand"))) + (match_operand:VWEXTI 1 "register_operand")))] + "TARGET_VECTOR && can_create_pseudo_p ()" + "#" + "&& 1" + [(const_int 0)] + { + rtx ops[] = {operands[0], operands[1], operands[2], operands[3]}; + riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_plussu (mode), + riscv_vector::WIDEN_TERNARY_OP, ops); + DONE; + } + [(set_attr "type" "viwmuladd")]) + +;; Combine zero_extend + sign_extend + fma to widen_fma (vwmaccsu) +(define_insn_and_split "*dual_widen_fmaus" + [(set (match_operand:VWEXTI 0 "register_operand") + (plus:VWEXTI + (mult:VWEXTI + (zero_extend:VWEXTI + (match_operand: 3 "register_operand")) + (sign_extend:VWEXTI + (match_operand: 2 "register_operand"))) + (match_operand:VWEXTI 1 "register_operand")))] + "TARGET_VECTOR && can_create_pseudo_p ()" + "#" + "&& 1" + [(const_int 0)] + { + rtx ops[] = {operands[0], operands[1], operands[2], operands[3]}; + riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_plussu (mode), + riscv_vector::WIDEN_TERNARY_OP, ops); + DONE; + } + [(set_attr "type" "viwmuladd")]) + +;; This combine pattern does not correspond to an single instruction. +;; This is a temporary pattern produced by a combine pass and if there +;; is no further combine into widen pattern, then fall back to extend +;; pattern and non-widen fma pattern. +(define_insn_and_split "*single_widen_fma" + [(set (match_operand:VWEXTI 0 "register_operand") + (plus:VWEXTI + (mult:VWEXTI + (any_extend:VWEXTI + (match_operand: 2 "register_operand")) + (match_operand:VWEXTI 3 "register_operand")) + (match_operand:VWEXTI 1 "register_operand")))] + "TARGET_VECTOR && can_create_pseudo_p ()" + "#" + "&& 1" + [(const_int 0)] + { + insn_code extend_icode = code_for_pred_vf2 (, mode); + rtx tmp = gen_reg_rtx (mode); + rtx extend_ops[] = {tmp, operands[2]}; + riscv_vector::emit_vlmax_insn (extend_icode, riscv_vector::UNARY_OP, + extend_ops); + + rtx ops[] = {operands[0], tmp, operands[3], operands[1]}; + riscv_vector::emit_vlmax_insn (code_for_pred_mul_plus (mode), + riscv_vector::TERNARY_OP, ops); + DONE; + } + [(set_attr "type" "viwmuladd")]) + +;; Combine extend + fma to widen_fma (vfwmacc) +(define_insn_and_split "*dual_fma" + [(set (match_operand:VWEXTF 0 "register_operand") + (plus:VWEXTF + (mult:VWEXTF + (float_extend:VWEXTF + (match_operand: 2 "register_operand")) + (float_extend:VWEXTF + (match_operand: 3 "register_operand"))) + (match_operand:VWEXTF 1 "register_operand")))] + "TARGET_VECTOR && can_create_pseudo_p ()" + "#" + "&& 1" + [(const_int 0)] + { + rtx ops[] = {operands[0], operands[1], operands[2], operands[3]}; + riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul (PLUS, mode), + riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, ops); + DONE; + } + [(set_attr "type" "vfwmuladd")]) + +;; This combine pattern does not correspond to an single instruction. +;; This is a temporary pattern produced by a combine pass and if there +;; is no further combine into widen pattern, then fall back to extend +;; pattern and non-widen fma pattern. +(define_insn_and_split "*single_fma" + [(set (match_operand:VWEXTF 0 "register_operand") + (plus:VWEXTF + (mult:VWEXTF + (float_extend:VWEXTF + (match_operand: 2 "register_operand")) + (match_operand:VWEXTF 3 "register_operand")) + (match_operand:VWEXTF 1 "register_operand")))] + "TARGET_VECTOR && can_create_pseudo_p ()" + "#" + "&& 1" + [(const_int 0)] + { + insn_code icode = code_for_pred_extend (mode); + rtx tmp = gen_reg_rtx (mode); + rtx ext_ops[] = {tmp, operands[2]}; + riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP, ext_ops); + + rtx ops[] = {operands[0], tmp, operands[3], operands[1]}; + riscv_vector::emit_vlmax_insn (code_for_pred_mul (PLUS, mode), + riscv_vector::TERNARY_OP_FRM_DYN, ops); + DONE; + } + [(set_attr "type" "vfwmuladd")]) + +;; Combine extend + fnma to widen_fnma (vfwnmsac) +(define_insn_and_split "*dual_fnma" + [(set (match_operand:VWEXTF 0 "register_operand") + (minus:VWEXTF + (match_operand:VWEXTF 1 "register_operand") + (mult:VWEXTF + (float_extend:VWEXTF + (match_operand: 2 "register_operand")) + (float_extend:VWEXTF + (match_operand: 3 "register_operand")))))] + "TARGET_VECTOR && can_create_pseudo_p ()" + "#" + "&& 1" + [(const_int 0)] + { + rtx ops[] = {operands[0], operands[1], operands[2], operands[3]}; + riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_neg (PLUS, mode), + riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, ops); + DONE; + } + [(set_attr "type" "vfwmuladd")]) + +;; This combine pattern does not correspond to an single instruction. +;; This is a temporary pattern produced by a combine pass and if there +;; is no further combine into widen pattern, then fall back to extend +;; pattern and non-widen fnma pattern. +(define_insn_and_split "*single_fnma" + [(set (match_operand:VWEXTF 0 "register_operand") + (minus:VWEXTF + (match_operand:VWEXTF 1 "register_operand") + (mult:VWEXTF + (float_extend:VWEXTF + (match_operand: 2 "register_operand")) + (match_operand:VWEXTF 3 "register_operand"))))] + "TARGET_VECTOR && can_create_pseudo_p ()" + "#" + "&& 1" + [(const_int 0)] + { + insn_code icode = code_for_pred_extend (mode); + rtx tmp = gen_reg_rtx (mode); + rtx ext_ops[] = {tmp, operands[2]}; + riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP, ext_ops); + + rtx ops[] = {operands[0], tmp, operands[3], operands[1]}; + riscv_vector::emit_vlmax_insn (code_for_pred_mul_neg (PLUS, mode), + riscv_vector::TERNARY_OP_FRM_DYN, ops); + DONE; + } + [(set_attr "type" "vfwmuladd")]) + +;; Combine extend + fms to widen_fms (vfwmsac) +(define_insn_and_split "*dual_fms" + [(set (match_operand:VWEXTF 0 "register_operand") + (minus:VWEXTF + (mult:VWEXTF + (float_extend:VWEXTF + (match_operand: 2 "register_operand")) + (float_extend:VWEXTF + (match_operand: 3 "register_operand"))) + (match_operand:VWEXTF 1 "register_operand")))] + "TARGET_VECTOR && can_create_pseudo_p ()" + "#" + "&& 1" + [(const_int 0)] + { + rtx ops[] = {operands[0], operands[1], operands[2], operands[3]}; + riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul (MINUS, mode), + riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, ops); + DONE; + } + [(set_attr "type" "vfwmuladd")]) + +;; This combine pattern does not correspond to an single instruction. +;; This is a temporary pattern produced by a combine pass and if there +;; is no further combine into widen pattern, then fall back to extend +;; pattern and non-widen fms pattern. +(define_insn_and_split "*single_fms" + [(set (match_operand:VWEXTF 0 "register_operand") + (minus:VWEXTF + (mult:VWEXTF + (float_extend:VWEXTF + (match_operand: 2 "register_operand")) + (match_operand:VWEXTF 3 "register_operand")) + (match_operand:VWEXTF 1 "register_operand")))] + "TARGET_VECTOR && can_create_pseudo_p ()" + "#" + "&& 1" + [(const_int 0)] + { + insn_code icode = code_for_pred_extend (mode); + rtx tmp = gen_reg_rtx (mode); + rtx ext_ops[] = {tmp, operands[2]}; + riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP, ext_ops); + + rtx ops[] = {operands[0], tmp, operands[3], operands[1]}; + riscv_vector::emit_vlmax_insn (code_for_pred_mul (MINUS, mode), + riscv_vector::TERNARY_OP_FRM_DYN, ops); + DONE; + } + [(set_attr "type" "vfwmuladd")]) + +;; Combine extend + fnms to widen_fnms (vfwnmacc) +(define_insn_and_split "*dual_fnms" + [(set (match_operand:VWEXTF 0 "register_operand") + (minus:VWEXTF + (mult:VWEXTF + (neg:VWEXTF + (float_extend:VWEXTF + (match_operand: 2 "register_operand"))) + (float_extend:VWEXTF + (match_operand: 3 "register_operand"))) + (match_operand:VWEXTF 1 "register_operand")))] + "TARGET_VECTOR && can_create_pseudo_p ()" + "#" + "&& 1" + [(const_int 0)] + { + rtx ops[] = {operands[0], operands[1], operands[2], operands[3]}; + riscv_vector::emit_vlmax_insn (code_for_pred_widen_mul_neg (MINUS, mode), + riscv_vector::WIDEN_TERNARY_OP_FRM_DYN, ops); + DONE; + } + [(set_attr "type" "vfwmuladd")]) + +;; This combine pattern does not correspond to an single instruction. +;; This is a temporary pattern produced by a combine pass and if there +;; is no further combine into widen pattern, then fall back to extend +;; pattern and non-widen fnms pattern. +(define_insn_and_split "*single_fnms" + [(set (match_operand:VWEXTF 0 "register_operand") + (minus:VWEXTF + (mult:VWEXTF + (neg:VWEXTF + (match_operand:VWEXTF 3 "register_operand")) + (float_extend:VWEXTF + (match_operand: 2 "register_operand"))) + (match_operand:VWEXTF 1 "register_operand")))] + "TARGET_VECTOR && can_create_pseudo_p ()" + "#" + "&& 1" + [(const_int 0)] + { + insn_code icode = code_for_pred_extend (mode); + rtx tmp = gen_reg_rtx (mode); + rtx ext_ops[] = {tmp, operands[2]}; + riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP, ext_ops); + + rtx ops[] = {operands[0], tmp, operands[3], operands[1]}; + riscv_vector::emit_vlmax_insn (code_for_pred_mul_neg (MINUS, mode), + riscv_vector::TERNARY_OP_FRM_DYN, ops); + DONE; + } + [(set_attr "type" "vfwmuladd")]) + ;; ============================================================================= ;; Misc combine patterns ;; ============================================================================= diff --git a/gcc/config/riscv/autovec.md b/gcc/config/riscv/autovec.md index 8537b9d41f6..96aaf869f88 100644 --- a/gcc/config/riscv/autovec.md +++ b/gcc/config/riscv/autovec.md @@ -1078,57 +1078,25 @@ ;; - vmadd ;; ------------------------------------------------------------------------- -;; We can't expand FMA for the following reasons: -;; 1. Before RA, we don't know which multiply-add instruction is the ideal one. -;; The vmacc is the ideal instruction when operands[3] overlaps operands[0]. -;; The vmadd is the ideal instruction when operands[1|2] overlaps operands[0]. -;; 2. According to vector.md, the multiply-add patterns has 'merge' operand which -;; is the operands[5]. Since operands[5] should overlap operands[0], this operand -;; should be allocated the same regno as operands[1|2|3]. -;; 3. The 'merge' operand is always a real merge operand and we don't allow undefined -;; operand. -;; 4. The operation of FMA pattern needs VLMAX vsetlvi which needs a VL operand. -;; -;; In this situation, we design the codegen of FMA as follows: -;; 1. clobber a scratch in the expand pattern of FMA. -;; 2. Let's RA decide which input operand (operands[1|2|3]) overlap operands[0]. -;; 3. Generate instructions (vmacc or vmadd) according to the register allocation -;; result after reload_completed. -(define_expand "fma4" - [(parallel - [(set (match_operand:VI 0 "register_operand") - (plus:VI - (mult:VI - (match_operand:VI 1 "register_operand") - (match_operand:VI 2 "register_operand")) - (match_operand:VI 3 "register_operand"))) - (clobber (match_dup 4))])] - "TARGET_VECTOR" - { - operands[4] = gen_reg_rtx (Pmode); - }) - -(define_insn_and_split "*fma" - [(set (match_operand:VI 0 "register_operand" "=vr, vr, ?&vr") +(define_insn_and_split "fma4" + [(set (match_operand:VI 0 "register_operand") (plus:VI (mult:VI - (match_operand:VI 1 "register_operand" " %0, vr, vr") - (match_operand:VI 2 "register_operand" " vr, vr, vr")) - (match_operand:VI 3 "register_operand" " vr, 0, vr"))) - (clobber (match_operand:P 4 "register_operand" "=r,r,r"))] - "TARGET_VECTOR" + (match_operand:VI 1 "register_operand") + (match_operand:VI 2 "register_operand")) + (match_operand:VI 3 "register_operand")))] + "TARGET_VECTOR && can_create_pseudo_p ()" "#" - "&& reload_completed" + "&& 1" [(const_int 0)] { - riscv_vector::emit_vlmax_vsetvl (mode, operands[4]); - rtx ops[] = {operands[0], operands[1], operands[2], operands[3], operands[0]}; - riscv_vector::emit_vlmax_insn_lra (code_for_pred_mul_plus (mode), - riscv_vector::TERNARY_OP, ops, operands[4]); + rtx ops[] = {operands[0], operands[1], operands[2], operands[3], + RVV_VUNDEF(mode)}; + riscv_vector::emit_vlmax_insn (code_for_pred_mul_plus (mode), + riscv_vector::TERNARY_OP, ops); DONE; } - [(set_attr "type" "vimuladd") - (set_attr "mode" "")]) + [(set_attr "type" "vector")]) ;; ------------------------------------------------------------------------- ;; ---- [INT] VNMSAC and VNMSUB @@ -1138,41 +1106,25 @@ ;; - vnmsub ;; ------------------------------------------------------------------------- -(define_expand "fnma4" - [(parallel - [(set (match_operand:VI 0 "register_operand") - (minus:VI - (match_operand:VI 3 "register_operand") - (mult:VI - (match_operand:VI 1 "register_operand") - (match_operand:VI 2 "register_operand")))) - (clobber (match_dup 4))])] - "TARGET_VECTOR" - { - operands[4] = gen_reg_rtx (Pmode); - }) - -(define_insn_and_split "*fnma" - [(set (match_operand:VI 0 "register_operand" "=vr, vr, ?&vr") - (minus:VI - (match_operand:VI 3 "register_operand" " vr, 0, vr") - (mult:VI - (match_operand:VI 1 "register_operand" " %0, vr, vr") - (match_operand:VI 2 "register_operand" " vr, vr, vr")))) - (clobber (match_operand:P 4 "register_operand" "=r,r,r"))] - "TARGET_VECTOR" +(define_insn_and_split "fnma4" + [(set (match_operand:VI 0 "register_operand") + (minus:VI + (match_operand:VI 3 "register_operand") + (mult:VI + (match_operand:VI 1 "register_operand") + (match_operand:VI 2 "register_operand"))))] + "TARGET_VECTOR && can_create_pseudo_p ()" "#" - "&& reload_completed" + "&& 1" [(const_int 0)] { - riscv_vector::emit_vlmax_vsetvl (mode, operands[4]); - rtx ops[] = {operands[0], operands[1], operands[2], operands[3], operands[0]}; - riscv_vector::emit_vlmax_insn_lra (code_for_pred_minus_mul (mode), - riscv_vector::TERNARY_OP, ops, operands[4]); + rtx ops[] = {operands[0], operands[1], operands[2], operands[3], + RVV_VUNDEF(mode)}; + riscv_vector::emit_vlmax_insn (code_for_pred_minus_mul (mode), + riscv_vector::TERNARY_OP, ops); DONE; } - [(set_attr "type" "vimuladd") - (set_attr "mode" "")]) + [(set_attr "type" "vector")]) ;; ------------------------------------------------------------------------- ;; ---- [FP] VFMACC and VFMADD @@ -1182,45 +1134,25 @@ ;; - vfmadd ;; ------------------------------------------------------------------------- -(define_expand "fma4" - [(parallel - [(set (match_operand:VF 0 "register_operand") - (unspec:VF - [(fma:VF - (match_operand:VF 1 "register_operand") - (match_operand:VF 2 "register_operand") - (match_operand:VF 3 "register_operand")) - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA)) - (clobber (match_dup 4))])] - "TARGET_VECTOR" - { - operands[4] = gen_reg_rtx (Pmode); - } - [(set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))]) - -(define_insn_and_split "*fma" - [(set (match_operand:VF 0 "register_operand" "=vr, vr, ?&vr") - (unspec:VF - [(fma:VF - (match_operand:VF 1 "register_operand" " %0, vr, vr") - (match_operand:VF 2 "register_operand" " vr, vr, vr") - (match_operand:VF 3 "register_operand" " vr, 0, vr")) - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA)) - (clobber (match_operand:P 4 "register_operand" "=r,r,r"))] - "TARGET_VECTOR" +(define_insn_and_split "fma4" + [(set (match_operand:VF 0 "register_operand") + (plus:VF + (mult:VF + (match_operand:VF 1 "register_operand") + (match_operand:VF 2 "register_operand")) + (match_operand:VF 3 "register_operand")))] + "TARGET_VECTOR && can_create_pseudo_p ()" "#" - "&& reload_completed" + "&& 1" [(const_int 0)] { - riscv_vector::emit_vlmax_vsetvl (mode, operands[4]); - rtx ops[] = {operands[0], operands[1], operands[2], operands[3], operands[0]}; - riscv_vector::emit_vlmax_insn_lra (code_for_pred_mul (PLUS, mode), - riscv_vector::TERNARY_OP_FRM_DYN, ops, operands[4]); + rtx ops[] = {operands[0], operands[1], operands[2], operands[3], + RVV_VUNDEF(mode)}; + riscv_vector::emit_vlmax_insn (code_for_pred_mul (PLUS, mode), + riscv_vector::TERNARY_OP_FRM_DYN, ops); DONE; } - [(set_attr "type" "vfmuladd") - (set_attr "mode" "") - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))]) + [(set_attr "type" "vector")]) ;; ------------------------------------------------------------------------- ;; ---- [FP] VFNMSAC and VFNMSUB @@ -1230,47 +1162,25 @@ ;; - vfnmsub ;; ------------------------------------------------------------------------- -(define_expand "fnma4" - [(parallel - [(set (match_operand:VF 0 "register_operand") - (unspec:VF - [(fma:VF - (neg:VF - (match_operand:VF 1 "register_operand")) - (match_operand:VF 2 "register_operand") - (match_operand:VF 3 "register_operand")) - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA)) - (clobber (match_dup 4))])] - "TARGET_VECTOR" - { - operands[4] = gen_reg_rtx (Pmode); - } - [(set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))]) - -(define_insn_and_split "*fnma" - [(set (match_operand:VF 0 "register_operand" "=vr, vr, ?&vr") - (unspec:VF - [(fma:VF - (neg:VF - (match_operand:VF 1 "register_operand" " %0, vr, vr")) - (match_operand:VF 2 "register_operand" " vr, vr, vr") - (match_operand:VF 3 "register_operand" " vr, 0, vr")) - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA)) - (clobber (match_operand:P 4 "register_operand" "=r,r,r"))] - "TARGET_VECTOR" +(define_insn_and_split "fnma4" + [(set (match_operand:VF 0 "register_operand") + (minus:VF + (match_operand:VF 3 "register_operand") + (mult:VF + (match_operand:VF 1 "register_operand") + (match_operand:VF 2 "register_operand"))))] + "TARGET_VECTOR && can_create_pseudo_p ()" "#" - "&& reload_completed" + "&& 1" [(const_int 0)] { - riscv_vector::emit_vlmax_vsetvl (mode, operands[4]); - rtx ops[] = {operands[0], operands[1], operands[2], operands[3], operands[0]}; - riscv_vector::emit_vlmax_insn_lra (code_for_pred_mul_neg (PLUS, mode), - riscv_vector::TERNARY_OP_FRM_DYN, ops, operands[4]); + rtx ops[] = {operands[0], operands[1], operands[2], operands[3], + RVV_VUNDEF(mode)}; + riscv_vector::emit_vlmax_insn (code_for_pred_mul_neg (PLUS, mode), + riscv_vector::TERNARY_OP_FRM_DYN, ops); DONE; } - [(set_attr "type" "vfmuladd") - (set_attr "mode" "") - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))]) + [(set_attr "type" "vector")]) ;; ------------------------------------------------------------------------- ;; ---- [FP] VFMSAC and VFMSUB @@ -1280,47 +1190,25 @@ ;; - vfmsub ;; ------------------------------------------------------------------------- -(define_expand "fms4" - [(parallel - [(set (match_operand:VF 0 "register_operand") - (unspec:VF - [(fma:VF - (match_operand:VF 1 "register_operand") - (match_operand:VF 2 "register_operand") - (neg:VF - (match_operand:VF 3 "register_operand"))) - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA)) - (clobber (match_dup 4))])] - "TARGET_VECTOR" - { - operands[4] = gen_reg_rtx (Pmode); - } - [(set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))]) - -(define_insn_and_split "*fms" - [(set (match_operand:VF 0 "register_operand" "=vr, vr, ?&vr") - (unspec:VF - [(fma:VF - (match_operand:VF 1 "register_operand" " %0, vr, vr") - (match_operand:VF 2 "register_operand" " vr, vr, vr") - (neg:VF - (match_operand:VF 3 "register_operand" " vr, 0, vr"))) - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA)) - (clobber (match_operand:P 4 "register_operand" "=r,r,r"))] - "TARGET_VECTOR" +(define_insn_and_split "fms4" + [(set (match_operand:VF 0 "register_operand") + (minus:VF + (mult:VF + (match_operand:VF 1 "register_operand") + (match_operand:VF 2 "register_operand")) + (match_operand:VF 3 "register_operand")))] + "TARGET_VECTOR && can_create_pseudo_p ()" "#" - "&& reload_completed" + "&& 1" [(const_int 0)] { - riscv_vector::emit_vlmax_vsetvl (mode, operands[4]); - rtx ops[] = {operands[0], operands[1], operands[2], operands[3], operands[0]}; - riscv_vector::emit_vlmax_insn_lra (code_for_pred_mul (MINUS, mode), - riscv_vector::TERNARY_OP_FRM_DYN, ops, operands[4]); + rtx ops[] = {operands[0], operands[1], operands[2], operands[3], + RVV_VUNDEF(mode)}; + riscv_vector::emit_vlmax_insn (code_for_pred_mul (MINUS, mode), + riscv_vector::TERNARY_OP_FRM_DYN, ops); DONE; } - [(set_attr "type" "vfmuladd") - (set_attr "mode" "") - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))]) + [(set_attr "type" "vector")]) ;; ------------------------------------------------------------------------- ;; ---- [FP] VFNMACC and VFNMADD @@ -1330,49 +1218,26 @@ ;; - vfnmadd ;; ------------------------------------------------------------------------- -(define_expand "fnms4" - [(parallel - [(set (match_operand:VF 0 "register_operand") - (unspec:VF - [(fma:VF - (neg:VF - (match_operand:VF 1 "register_operand")) - (match_operand:VF 2 "register_operand") - (neg:VF - (match_operand:VF 3 "register_operand"))) - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA)) - (clobber (match_dup 4))])] - "TARGET_VECTOR" - { - operands[4] = gen_reg_rtx (Pmode); - } - [(set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))]) - -(define_insn_and_split "*fnms" - [(set (match_operand:VF 0 "register_operand" "=vr, vr, ?&vr") - (unspec:VF - [(fma:VF - (neg:VF - (match_operand:VF 1 "register_operand" " %0, vr, vr")) - (match_operand:VF 2 "register_operand" " vr, vr, vr") - (neg:VF - (match_operand:VF 3 "register_operand" " vr, 0, vr"))) - (reg:SI FRM_REGNUM)] UNSPEC_VFFMA)) - (clobber (match_operand:P 4 "register_operand" "=r,r,r"))] - "TARGET_VECTOR" +(define_insn_and_split "fnms4" + [(set (match_operand:VF 0 "register_operand") + (minus:VF + (neg:VF + (mult:VF + (match_operand:VF 1 "register_operand") + (match_operand:VF 2 "register_operand"))) + (match_operand:VF 3 "register_operand")))] + "TARGET_VECTOR && can_create_pseudo_p ()" "#" - "&& reload_completed" + "&& 1" [(const_int 0)] { - riscv_vector::emit_vlmax_vsetvl (mode, operands[4]); - rtx ops[] = {operands[0], operands[1], operands[2], operands[3], operands[0]}; - riscv_vector::emit_vlmax_insn_lra (code_for_pred_mul_neg (MINUS, mode), - riscv_vector::TERNARY_OP_FRM_DYN, ops, operands[4]); + rtx ops[] = {operands[0], operands[1], operands[2], operands[3], + RVV_VUNDEF(mode)}; + riscv_vector::emit_vlmax_insn (code_for_pred_mul_neg (MINUS, mode), + riscv_vector::TERNARY_OP_FRM_DYN, ops); DONE; } - [(set_attr "type" "vfmuladd") - (set_attr "mode" "") - (set (attr "frm_mode") (symbol_ref "riscv_vector::FRM_DYN"))]) + [(set_attr "type" "vector")]) ;; ========================================================================= ;; == SELECT_VL diff --git a/gcc/config/riscv/riscv-protos.h b/gcc/config/riscv/riscv-protos.h index 44fa36c32ab..46fc8f39974 100644 --- a/gcc/config/riscv/riscv-protos.h +++ b/gcc/config/riscv/riscv-protos.h @@ -462,7 +462,7 @@ void expand_select_vl (rtx *); void expand_load_store (rtx *, bool); void expand_gather_scatter (rtx *, bool); void expand_cond_len_ternop (unsigned, rtx *); -void prepare_ternary_operands (rtx *, bool = false); +void prepare_ternary_operands (rtx *); void expand_lanes_load_store (rtx *, bool); void expand_fold_extract_last (rtx *); diff --git a/gcc/config/riscv/riscv-v.cc b/gcc/config/riscv/riscv-v.cc index 668594b65ed..f51986aebde 100644 --- a/gcc/config/riscv/riscv-v.cc +++ b/gcc/config/riscv/riscv-v.cc @@ -3253,15 +3253,15 @@ expand_reduction (unsigned unspec, rtx *ops, rtx init, reduction_type type) /* Prepare ops for ternary operations. It can be called before or after RA. */ void -prepare_ternary_operands (rtx *ops, bool split_p) +prepare_ternary_operands (rtx *ops) { machine_mode mode = GET_MODE (ops[0]); - if (split_p - || (!rtx_equal_p (ops[2], ops[5]) - && !rtx_equal_p (ops[3], ops[5]) - && !rtx_equal_p (ops[4], ops[5]) - && riscv_get_v_regno_alignment (mode) == 8)) + if (!rtx_equal_p (ops[5], RVV_VUNDEF (mode)) + && (VECTOR_MODE_P (GET_MODE (ops[2])) + && !rtx_equal_p (ops[2], ops[5])) + && !rtx_equal_p (ops[3], ops[5]) + && !rtx_equal_p (ops[4], ops[5])) { /* RA will fail to find vector REG and report ICE, so we pre-merge the ops for LMUL = 8. */ @@ -3286,6 +3286,8 @@ prepare_ternary_operands (rtx *ops, bool split_p) /* TODO: ??? Maybe we could support splitting FMA (a, 4, b) into PLUS (ASHIFT (a, 2), b) according to uarchs. */ } + gcc_assert (rtx_equal_p (ops[5], RVV_VUNDEF (mode)) + || rtx_equal_p (ops[5], ops[2]) || rtx_equal_p (ops[5], ops[4])); } /* Expand VEC_MASK_LEN_{LOAD_LANES,STORE_LANES}. */ diff --git a/gcc/config/riscv/vector.md b/gcc/config/riscv/vector.md index fb0c8abd995..e174322a459 100644 --- a/gcc/config/riscv/vector.md +++ b/gcc/config/riscv/vector.md @@ -5002,12 +5002,40 @@ (match_operand:VI 2 "register_operand") (match_operand:VI 3 "register_operand")) (match_operand:VI 4 "register_operand")) - (match_operand:VI 5 "register_operand")))] + (match_operand:VI 5 "vector_merge_operand")))] "TARGET_VECTOR" { riscv_vector::prepare_ternary_operands (operands); }) +(define_insn "*pred_mul_plus_undef" + [(set (match_operand:VI 0 "register_operand" "=vd, vd,?&vd, vr, vr,?&vr") + (if_then_else:VI + (unspec: + [(match_operand: 1 "vector_mask_operand" " vm, vm, vm,Wc1,Wc1, Wc1") + (match_operand 6 "vector_length_operand" " rK, rK, rK, rK, rK, rK") + (match_operand 7 "const_int_operand" " i, i, i, i, i, i") + (match_operand 8 "const_int_operand" " i, i, i, i, i, i") + (match_operand 9 "const_int_operand" " i, i, i, i, i, i") + (reg:SI VL_REGNUM) + (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE) + (plus:VI + (mult:VI + (match_operand:VI 3 "register_operand" " 0, vr, vr, 0, vr, vr") + (match_operand:VI 4 "register_operand" " vr, vr, vr, vr, vr, vr")) + (match_operand:VI 5 "register_operand" " vr, 0, vr, vr, 0, vr")) + (match_operand:VI 2 "vector_undef_operand")))] + "TARGET_VECTOR" + "@ + vmadd.vv\t%0,%4,%5%p1 + vmacc.vv\t%0,%3,%4%p1 + vmv.v.v\t%0,%4\;vmacc.vv\t%0,%3,%4%p1 + vmadd.vv\t%0,%4,%5%p1 + vmacc.vv\t%0,%3,%4%p1 + vmv.v.v\t%0,%5\;vmacc.vv\t%0,%3,%4%p1" + [(set_attr "type" "vimuladd") + (set_attr "mode" "")]) + (define_insn "*pred_madd" [(set (match_operand:VI 0 "register_operand" "=vd,?&vd, vr,?&vr") (if_then_else:VI @@ -5070,35 +5098,6 @@ (set (attr "ma") (symbol_ref "riscv_vector::get_ma(operands[7])")) (set (attr "avl_type") (symbol_ref "INTVAL (operands[8])"))]) -(define_insn_and_rewrite "*pred_mul_plus" - [(set (match_operand:VI 0 "register_operand" "=&vr") - (if_then_else:VI - (unspec: - [(match_operand: 1 "vector_mask_operand" "vmWc1") - (match_operand 6 "vector_length_operand" " rK") - (match_operand 7 "const_int_operand" " i") - (match_operand 8 "const_int_operand" " i") - (match_operand 9 "const_int_operand" " i") - (reg:SI VL_REGNUM) - (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE) - (plus:VI - (mult:VI - (match_operand:VI 2 "register_operand" " vr") - (match_operand:VI 3 "register_operand" " vr")) - (match_operand:VI 4 "register_operand" " vr")) - (match_operand:VI 5 "register_operand" " vr")))] - "TARGET_VECTOR - && !rtx_equal_p (operands[2], operands[5]) - && !rtx_equal_p (operands[3], operands[5]) - && !rtx_equal_p (operands[4], operands[5])" - "#" - "&& reload_completed" - { - riscv_vector::prepare_ternary_operands (operands, true); - } - [(set_attr "type" "vimuladd") - (set_attr "mode" "")]) - (define_expand "@pred_mul_plus_scalar" [(set (match_operand:VI_QHS 0 "register_operand") (if_then_else:VI_QHS @@ -5184,35 +5183,6 @@ (set (attr "ma") (symbol_ref "riscv_vector::get_ma(operands[7])")) (set (attr "avl_type") (symbol_ref "INTVAL (operands[8])"))]) -(define_insn_and_rewrite "*pred_mul_plus_scalar" - [(set (match_operand:VI 0 "register_operand" "=&vr") - (if_then_else:VI - (unspec: - [(match_operand: 1 "vector_mask_operand" "vmWc1") - (match_operand 6 "vector_length_operand" " rK") - (match_operand 7 "const_int_operand" " i") - (match_operand 8 "const_int_operand" " i") - (match_operand 9 "const_int_operand" " i") - (reg:SI VL_REGNUM) - (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE) - (plus:VI - (mult:VI - (vec_duplicate:VI - (match_operand: 2 "register_operand" " r")) - (match_operand:VI 3 "register_operand" " vr")) - (match_operand:VI 4 "vector_arith_operand" " vr")) - (match_operand:VI 5 "register_operand" " vr")))] - "TARGET_VECTOR - && !rtx_equal_p (operands[3], operands[5]) - && !rtx_equal_p (operands[4], operands[5])" - "#" - "&& reload_completed" - { - riscv_vector::prepare_ternary_operands (operands, true); - } - [(set_attr "type" "vimuladd") - (set_attr "mode" "")]) - (define_expand "@pred_mul_plus_scalar" [(set (match_operand:VI_D 0 "register_operand") (if_then_else:VI_D @@ -5313,36 +5283,6 @@ (set (attr "ma") (symbol_ref "riscv_vector::get_ma(operands[7])")) (set (attr "avl_type") (symbol_ref "INTVAL (operands[8])"))]) -(define_insn_and_rewrite "*pred_mul_plus_extended_scalar" - [(set (match_operand:VI_D 0 "register_operand" "=&vr") - (if_then_else:VI_D - (unspec: - [(match_operand: 1 "vector_mask_operand" "vmWc1") - (match_operand 6 "vector_length_operand" " rK") - (match_operand 7 "const_int_operand" " i") - (match_operand 8 "const_int_operand" " i") - (match_operand 9 "const_int_operand" " i") - (reg:SI VL_REGNUM) - (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE) - (plus:VI_D - (mult:VI_D - (vec_duplicate:VI_D - (sign_extend: - (match_operand: 2 "register_operand" " r"))) - (match_operand:VI_D 3 "register_operand" " vr")) - (match_operand:VI_D 4 "register_operand" " vr")) - (match_operand:VI_D 5 "register_operand" " vr")))] - "TARGET_VECTOR - && !rtx_equal_p (operands[3], operands[5]) - && !rtx_equal_p (operands[4], operands[5])" - "#" - "&& reload_completed" - { - riscv_vector::prepare_ternary_operands (operands, true); - } - [(set_attr "type" "vimuladd") - (set_attr "mode" "")]) - (define_expand "@pred_minus_mul" [(set (match_operand:VI 0 "register_operand") (if_then_else:VI @@ -5359,12 +5299,40 @@ (mult:VI (match_operand:VI 2 "register_operand") (match_operand:VI 3 "register_operand"))) - (match_operand:VI 5 "register_operand")))] + (match_operand:VI 5 "vector_merge_operand")))] "TARGET_VECTOR" { riscv_vector::prepare_ternary_operands (operands); }) +(define_insn "*pred_minus_mul_undef" + [(set (match_operand:VI 0 "register_operand" "=vd, vd,?&vd, vr, vr,?&vr") + (if_then_else:VI + (unspec: + [(match_operand: 1 "vector_mask_operand" " vm, vm, vm,Wc1,Wc1, Wc1") + (match_operand 6 "vector_length_operand" " rK, rK, rK, rK, rK, rK") + (match_operand 7 "const_int_operand" " i, i, i, i, i, i") + (match_operand 8 "const_int_operand" " i, i, i, i, i, i") + (match_operand 9 "const_int_operand" " i, i, i, i, i, i") + (reg:SI VL_REGNUM) + (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE) + (minus:VI + (match_operand:VI 5 "register_operand" " vr, 0, vr, vr, 0, vr") + (mult:VI + (match_operand:VI 3 "register_operand" " 0, vr, vr, 0, vr, vr") + (match_operand:VI 4 "register_operand" " vr, vr, vr, vr, vr, vr"))) + (match_operand:VI 2 "vector_undef_operand")))] + "TARGET_VECTOR" + "@ + vnmsub.vv\t%0,%4,%5%p1 + vnmsac.vv\t%0,%3,%4%p1 + vmv.v.v\t%0,%3\;vnmsub.vv\t%0,%4,%5%p1 + vnmsub.vv\t%0,%4,%5%p1 + vnmsac.vv\t%0,%3,%4%p1 + vmv.v.v\t%0,%3\;vnmsub.vv\t%0,%4,%5%p1" + [(set_attr "type" "vimuladd") + (set_attr "mode" "")]) + (define_insn "*pred_nmsub" [(set (match_operand:VI 0 "register_operand" "=vd,?&vd, vr,?&vr") (if_then_else:VI @@ -5427,35 +5395,6 @@ (set (attr "ma") (symbol_ref "riscv_vector::get_ma(operands[7])")) (set (attr "avl_type") (symbol_ref "INTVAL (operands[8])"))]) -(define_insn_and_rewrite "*pred_minus_mul" - [(set (match_operand:VI 0 "register_operand" "=&vr") - (if_then_else:VI - (unspec: - [(match_operand: 1 "vector_mask_operand" "vmWc1") - (match_operand 6 "vector_length_operand" " rK") - (match_operand 7 "const_int_operand" " i") - (match_operand 8 "const_int_operand" " i") - (match_operand 9 "const_int_operand" " i") - (reg:SI VL_REGNUM) - (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE) - (minus:VI - (match_operand:VI 4 "vector_arith_operand" " vr") - (mult:VI - (match_operand:VI 2 "register_operand" " vr") - (match_operand:VI 3 "register_operand" " vr"))) - (match_operand:VI 5 "register_operand" " vr")))] - "TARGET_VECTOR - && !rtx_equal_p (operands[2], operands[5]) - && !rtx_equal_p (operands[3], operands[5]) - && !rtx_equal_p (operands[4], operands[5])" - "#" - "&& reload_completed" - { - riscv_vector::prepare_ternary_operands (operands, true); - } - [(set_attr "type" "vimuladd") - (set_attr "mode" "")]) - (define_expand "@pred_minus_mul_scalar" [(set (match_operand:VI_QHS 0 "register_operand") (if_then_else:VI_QHS @@ -5541,35 +5480,6 @@ (set (attr "ma") (symbol_ref "riscv_vector::get_ma(operands[7])")) (set (attr "avl_type") (symbol_ref "INTVAL (operands[8])"))]) -(define_insn_and_rewrite "*pred_minus_mul_scalar" - [(set (match_operand:VI 0 "register_operand" "=&vr") - (if_then_else:VI - (unspec: - [(match_operand: 1 "vector_mask_operand" "vmWc1") - (match_operand 6 "vector_length_operand" " rK") - (match_operand 7 "const_int_operand" " i") - (match_operand 8 "const_int_operand" " i") - (match_operand 9 "const_int_operand" " i") - (reg:SI VL_REGNUM) - (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE) - (minus:VI - (match_operand:VI 4 "vector_arith_operand" " vr") - (mult:VI - (vec_duplicate:VI - (match_operand: 2 "register_operand" " r")) - (match_operand:VI 3 "register_operand" " vr"))) - (match_operand:VI 5 "register_operand" " vr")))] - "TARGET_VECTOR - && !rtx_equal_p (operands[3], operands[5]) - && !rtx_equal_p (operands[4], operands[5])" - "#" - "&& reload_completed" - { - riscv_vector::prepare_ternary_operands (operands, true); - } - [(set_attr "type" "vimuladd") - (set_attr "mode" "")]) - (define_expand "@pred_minus_mul_scalar" [(set (match_operand:VI_D 0 "register_operand") (if_then_else:VI_D @@ -5670,36 +5580,6 @@ (set (attr "ma") (symbol_ref "riscv_vector::get_ma(operands[7])")) (set (attr "avl_type") (symbol_ref "INTVAL (operands[8])"))]) -(define_insn_and_rewrite "*pred_minus_mul_extended_scalar" - [(set (match_operand:VI_D 0 "register_operand" "=&vr") - (if_then_else:VI_D - (unspec: - [(match_operand: 1 "vector_mask_operand" "vmWc1") - (match_operand 6 "vector_length_operand" " rK") - (match_operand 7 "const_int_operand" " i") - (match_operand 8 "const_int_operand" " i") - (match_operand 9 "const_int_operand" " i") - (reg:SI VL_REGNUM) - (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE) - (minus:VI_D - (match_operand:VI_D 4 "vector_arith_operand" " vr") - (mult:VI_D - (vec_duplicate:VI_D - (sign_extend: - (match_operand: 2 "register_operand" " r"))) - (match_operand:VI_D 3 "register_operand" " vr"))) - (match_operand:VI_D 5 "register_operand" " vr")))] - "TARGET_VECTOR - && !rtx_equal_p (operands[3], operands[5]) - && !rtx_equal_p (operands[4], operands[5])" - "#" - "&& reload_completed" - { - riscv_vector::prepare_ternary_operands (operands, true); - } - [(set_attr "type" "vimuladd") - (set_attr "mode" "")]) - ;; ------------------------------------------------------------------------------- ;; ---- Predicated widen integer ternary operations ;; ------------------------------------------------------------------------------- @@ -6276,12 +6156,44 @@ (match_operand:VF 2 "register_operand") (match_operand:VF 3 "register_operand")) (match_operand:VF 4 "register_operand")) - (match_operand:VF 5 "register_operand")))] + (match_operand:VF 5 "vector_merge_operand")))] "TARGET_VECTOR" { riscv_vector::prepare_ternary_operands (operands); }) +(define_insn "*pred_mul__undef" + [(set (match_operand:VF 0 "register_operand" "=vd,vd,?&vd, vr, vr,?&vr") + (if_then_else:VF + (unspec: + [(match_operand: 1 "vector_mask_operand" " vm,vm, vm,Wc1,Wc1, Wc1") + (match_operand 6 "vector_length_operand" " rK,rK, rK, rK, rK, rK") + (match_operand 7 "const_int_operand" " i, i, i, i, i, i") + (match_operand 8 "const_int_operand" " i, i, i, i, i, i") + (match_operand 9 "const_int_operand" " i, i, i, i, i, i") + (match_operand 10 "const_int_operand" " i, i, i, i, i, i") + (reg:SI VL_REGNUM) + (reg:SI VTYPE_REGNUM) + (reg:SI FRM_REGNUM)] UNSPEC_VPREDICATE) + (plus_minus:VF + (mult:VF + (match_operand:VF 3 "register_operand" " 0,vr, vr, 0, vr, vr") + (match_operand:VF 4 "register_operand" " vr,vr, vr, vr, vr, vr")) + (match_operand:VF 5 "register_operand" " vr, 0, vr, vr, 0, vr")) + (match_operand:VF 2 "vector_undef_operand")))] + "TARGET_VECTOR" + "@ + vf.vv\t%0,%4,%5%p1 + vf.vv\t%0,%3,%4%p1 + vmv.v.v\t%0,%3\;vf.vv\t%0,%4,%5%p1 + vf.vv\t%0,%4,%5%p1 + vf.vv\t%0,%3,%4%p1 + vmv.v.v\t%0,%3\;vf.vv\t%0,%4,%5%p1" + [(set_attr "type" "vfmuladd") + (set_attr "mode" "") + (set (attr "frm_mode") + (symbol_ref "riscv_vector::get_frm_mode (operands[10])"))]) + (define_insn "*pred_" [(set (match_operand:VF 0 "register_operand" "=vd, ?&vd, vr, ?&vr") (if_then_else:VF @@ -6352,39 +6264,6 @@ (set (attr "frm_mode") (symbol_ref "riscv_vector::get_frm_mode (operands[9])"))]) -(define_insn_and_rewrite "*pred_mul_" - [(set (match_operand:VF 0 "register_operand" "=&vr") - (if_then_else:VF - (unspec: - [(match_operand: 1 "vector_mask_operand" "vmWc1") - (match_operand 6 "vector_length_operand" " rK") - (match_operand 7 "const_int_operand" " i") - (match_operand 8 "const_int_operand" " i") - (match_operand 9 "const_int_operand" " i") - (match_operand 10 "const_int_operand" " i") - (reg:SI VL_REGNUM) - (reg:SI VTYPE_REGNUM) - (reg:SI FRM_REGNUM)] UNSPEC_VPREDICATE) - (plus_minus:VF - (mult:VF - (match_operand:VF 2 "register_operand" " vr") - (match_operand:VF 3 "register_operand" " vr")) - (match_operand:VF 4 "register_operand" " vr")) - (match_operand:VF 5 "register_operand" " vr")))] - "TARGET_VECTOR - && !rtx_equal_p (operands[2], operands[5]) - && !rtx_equal_p (operands[3], operands[5]) - && !rtx_equal_p (operands[4], operands[5])" - "#" - "&& reload_completed" - { - riscv_vector::prepare_ternary_operands (operands, true); - } - [(set_attr "type" "vfmuladd") - (set_attr "mode" "") - (set (attr "frm_mode") - (symbol_ref "riscv_vector::get_frm_mode (operands[10])"))]) - (define_expand "@pred_mul__scalar" [(set (match_operand:VF 0 "register_operand") (if_then_else:VF @@ -6480,39 +6359,6 @@ (set (attr "frm_mode") (symbol_ref "riscv_vector::get_frm_mode (operands[9])"))]) -(define_insn_and_rewrite "*pred_mul__scalar" - [(set (match_operand:VF 0 "register_operand" "=&vr") - (if_then_else:VF - (unspec: - [(match_operand: 1 "vector_mask_operand" "vmWc1") - (match_operand 6 "vector_length_operand" " rK") - (match_operand 7 "const_int_operand" " i") - (match_operand 8 "const_int_operand" " i") - (match_operand 9 "const_int_operand" " i") - (match_operand 10 "const_int_operand" " i") - (reg:SI VL_REGNUM) - (reg:SI VTYPE_REGNUM) - (reg:SI FRM_REGNUM)] UNSPEC_VPREDICATE) - (plus_minus:VF - (mult:VF - (vec_duplicate:VF - (match_operand: 2 "register_operand" " f")) - (match_operand:VF 3 "register_operand" " vr")) - (match_operand:VF 4 "vector_arith_operand" " vr")) - (match_operand:VF 5 "register_operand" " vr")))] - "TARGET_VECTOR - && !rtx_equal_p (operands[3], operands[5]) - && !rtx_equal_p (operands[4], operands[5])" - "#" - "&& reload_completed" - { - riscv_vector::prepare_ternary_operands (operands, true); - } - [(set_attr "type" "vfmuladd") - (set_attr "mode" "") - (set (attr "frm_mode") - (symbol_ref "riscv_vector::get_frm_mode (operands[10])"))]) - (define_expand "@pred_mul_neg_" [(set (match_operand:VF 0 "register_operand") (if_then_else:VF @@ -6532,12 +6378,45 @@ (match_operand:VF 2 "register_operand") (match_operand:VF 3 "register_operand"))) (match_operand:VF 4 "register_operand")) - (match_operand:VF 5 "register_operand")))] + (match_operand:VF 5 "vector_merge_operand")))] "TARGET_VECTOR" { riscv_vector::prepare_ternary_operands (operands); }) +(define_insn "*pred_mul_neg__undef" + [(set (match_operand:VF 0 "register_operand" "=vd,vd,?&vd, vr, vr,?&vr") + (if_then_else:VF + (unspec: + [(match_operand: 1 "vector_mask_operand" " vm,vm, vm,Wc1,Wc1, Wc1") + (match_operand 6 "vector_length_operand" " rK,rK, rK, rK, rK, rK") + (match_operand 7 "const_int_operand" " i, i, i, i, i, i") + (match_operand 8 "const_int_operand" " i, i, i, i, i, i") + (match_operand 9 "const_int_operand" " i, i, i, i, i, i") + (match_operand 10 "const_int_operand" " i, i, i, i, i, i") + (reg:SI VL_REGNUM) + (reg:SI VTYPE_REGNUM) + (reg:SI FRM_REGNUM)] UNSPEC_VPREDICATE) + (plus_minus:VF + (neg:VF + (mult:VF + (match_operand:VF 3 "register_operand" " 0,vr, vr, 0, vr, vr") + (match_operand:VF 4 "register_operand" " vr,vr, vr, vr, vr, vr"))) + (match_operand:VF 5 "register_operand" " vr, 0, vr, vr, 0, vr")) + (match_operand:VF 2 "vector_undef_operand")))] + "TARGET_VECTOR" + "@ + vf.vv\t%0,%4,%5%p1 + vf.vv\t%0,%3,%4%p1 + vmv.v.v\t%0,%3\;vf.vv\t%0,%4,%5%p1 + vf.vv\t%0,%4,%5%p1 + vf.vv\t%0,%3,%4%p1 + vmv.v.v\t%0,%3\;vf.vv\t%0,%4,%5%p1" + [(set_attr "type" "vfmuladd") + (set_attr "mode" "") + (set (attr "frm_mode") + (symbol_ref "riscv_vector::get_frm_mode (operands[10])"))]) + (define_insn "*pred_" [(set (match_operand:VF 0 "register_operand" "=vd, ?&vd, vr, ?&vr") (if_then_else:VF @@ -6610,40 +6489,6 @@ (set (attr "frm_mode") (symbol_ref "riscv_vector::get_frm_mode (operands[9])"))]) -(define_insn_and_rewrite "*pred_mul_neg_" - [(set (match_operand:VF 0 "register_operand" "=&vr") - (if_then_else:VF - (unspec: - [(match_operand: 1 "vector_mask_operand" "vmWc1") - (match_operand 6 "vector_length_operand" " rK") - (match_operand 7 "const_int_operand" " i") - (match_operand 8 "const_int_operand" " i") - (match_operand 9 "const_int_operand" " i") - (match_operand 10 "const_int_operand" " i") - (reg:SI VL_REGNUM) - (reg:SI VTYPE_REGNUM) - (reg:SI FRM_REGNUM)] UNSPEC_VPREDICATE) - (plus_minus:VF - (neg:VF - (mult:VF - (match_operand:VF 2 "register_operand" " vr") - (match_operand:VF 3 "register_operand" " vr"))) - (match_operand:VF 4 "vector_arith_operand" " vr")) - (match_operand:VF 5 "register_operand" " vr")))] - "TARGET_VECTOR - && !rtx_equal_p (operands[2], operands[5]) - && !rtx_equal_p (operands[3], operands[5]) - && !rtx_equal_p (operands[4], operands[5])" - "#" - "&& reload_completed" - { - riscv_vector::prepare_ternary_operands (operands, true); - } - [(set_attr "type" "vfmuladd") - (set_attr "mode" "") - (set (attr "frm_mode") - (symbol_ref "riscv_vector::get_frm_mode (operands[10])"))]) - (define_expand "@pred_mul_neg__scalar" [(set (match_operand:VF 0 "register_operand") (if_then_else:VF @@ -6742,40 +6587,6 @@ (set (attr "frm_mode") (symbol_ref "riscv_vector::get_frm_mode (operands[9])"))]) -(define_insn_and_rewrite "*pred_mul_neg__scalar" - [(set (match_operand:VF 0 "register_operand" "=&vr") - (if_then_else:VF - (unspec: - [(match_operand: 1 "vector_mask_operand" "vmWc1") - (match_operand 6 "vector_length_operand" " rK") - (match_operand 7 "const_int_operand" " i") - (match_operand 8 "const_int_operand" " i") - (match_operand 9 "const_int_operand" " i") - (match_operand 10 "const_int_operand" " i") - (reg:SI VL_REGNUM) - (reg:SI VTYPE_REGNUM) - (reg:SI FRM_REGNUM)] UNSPEC_VPREDICATE) - (plus_minus:VF - (neg:VF - (mult:VF - (vec_duplicate:VF - (match_operand: 2 "register_operand" " f")) - (match_operand:VF 3 "register_operand" " vr"))) - (match_operand:VF 4 "vector_arith_operand" " vr")) - (match_operand:VF 5 "register_operand" " vr")))] - "TARGET_VECTOR - && !rtx_equal_p (operands[3], operands[5]) - && !rtx_equal_p (operands[4], operands[5])" - "#" - "&& reload_completed" - { - riscv_vector::prepare_ternary_operands (operands, true); - } - [(set_attr "type" "vfmuladd") - (set_attr "mode" "") - (set (attr "frm_mode") - (symbol_ref "riscv_vector::get_frm_mode (operands[10])"))]) - ;; ------------------------------------------------------------------------------- ;; ---- Predicated floating-point unary operations ;; -------------------------------------------------------------------------------