RISC-V: Keep vlmax vector operators in simple form until split1 pass
Checks
Commit Message
This patch keep vlmax vector pattern in simple before split1 pass which
will allow more optimization (e.g. combine) before split1 pass.
This patch changes the vlmax pattern in autovec.md to define_insn_and_split
as much as possible and clean up some combine patterns that are no longer needed.
This patch also fixed PR111232 bug which was caused by a combined failed.
PR target/111232
gcc/ChangeLog:
* config/riscv/autovec-opt.md (@pred_single_widen_mul<any_extend:su><mode>):
Delete.
(*pred_widen_mulsu<mode>): Delete.
(*pred_single_widen_mul<mode>): Delete.
(*dual_widen_<any_widen_binop:optab><any_extend:su><mode>):
Add new combine patterns.
(*single_widen_sub<any_extend:su><mode>): Ditto.
(*single_widen_add<any_extend:su><mode>): Ditto.
(*single_widen_mult<any_extend:su><mode>): Ditto.
(*dual_widen_mulsu<mode>): Ditto.
(*dual_widen_mulus<mode>): Ditto.
(*dual_widen_<optab><mode>): Ditto.
(*single_widen_add<mode>): Ditto.
(*single_widen_sub<mode>): Ditto.
(*single_widen_mult<mode>): Ditto.
* config/riscv/autovec.md (<optab><mode>3):
Change define_expand to define_insn_and_split.
(<optab><mode>2): Ditto.
(abs<mode>2): Ditto.
(smul<mode>3_highpart): Ditto.
(umul<mode>3_highpart): Ditto.
gcc/testsuite/ChangeLog:
* gcc.target/riscv/rvv/autovec/widen/widen-4.c: Add more testcases.
* gcc.target/riscv/rvv/autovec/widen/widen-complicate-4.c: Ditto.
* gcc.target/riscv/rvv/autovec/pr111232.c: New test.
---
gcc/config/riscv/autovec-opt.md | 294 ++++++++++++------
gcc/config/riscv/autovec.md | 82 +++--
.../gcc.target/riscv/rvv/autovec/pr111232.c | 18 ++
.../riscv/rvv/autovec/widen/widen-4.c | 7 +-
.../rvv/autovec/widen/widen-complicate-4.c | 11 +-
5 files changed, 276 insertions(+), 136 deletions(-)
create mode 100644 gcc/testsuite/gcc.target/riscv/rvv/autovec/pr111232.c
Comments
Can those intermediate patterns be used for intrinsic? I would prefer
to keep those stuff *IF* possible used for intrinsics.
On Mon, Sep 4, 2023 at 7:14 PM Lehua Ding <lehua.ding@rivai.ai> wrote:
>
> This patch keep vlmax vector pattern in simple before split1 pass which
> will allow more optimization (e.g. combine) before split1 pass.
> This patch changes the vlmax pattern in autovec.md to define_insn_and_split
> as much as possible and clean up some combine patterns that are no longer needed.
> This patch also fixed PR111232 bug which was caused by a combined failed.
>
> PR target/111232
>
> gcc/ChangeLog:
>
> * config/riscv/autovec-opt.md (@pred_single_widen_mul<any_extend:su><mode>):
> Delete.
> (*pred_widen_mulsu<mode>): Delete.
> (*pred_single_widen_mul<mode>): Delete.
> (*dual_widen_<any_widen_binop:optab><any_extend:su><mode>):
> Add new combine patterns.
> (*single_widen_sub<any_extend:su><mode>): Ditto.
> (*single_widen_add<any_extend:su><mode>): Ditto.
> (*single_widen_mult<any_extend:su><mode>): Ditto.
> (*dual_widen_mulsu<mode>): Ditto.
> (*dual_widen_mulus<mode>): Ditto.
> (*dual_widen_<optab><mode>): Ditto.
> (*single_widen_add<mode>): Ditto.
> (*single_widen_sub<mode>): Ditto.
> (*single_widen_mult<mode>): Ditto.
> * config/riscv/autovec.md (<optab><mode>3):
> Change define_expand to define_insn_and_split.
> (<optab><mode>2): Ditto.
> (abs<mode>2): Ditto.
> (smul<mode>3_highpart): Ditto.
> (umul<mode>3_highpart): Ditto.
>
> gcc/testsuite/ChangeLog:
>
> * gcc.target/riscv/rvv/autovec/widen/widen-4.c: Add more testcases.
> * gcc.target/riscv/rvv/autovec/widen/widen-complicate-4.c: Ditto.
> * gcc.target/riscv/rvv/autovec/pr111232.c: New test.
>
> ---
> gcc/config/riscv/autovec-opt.md | 294 ++++++++++++------
> gcc/config/riscv/autovec.md | 82 +++--
> .../gcc.target/riscv/rvv/autovec/pr111232.c | 18 ++
> .../riscv/rvv/autovec/widen/widen-4.c | 7 +-
> .../rvv/autovec/widen/widen-complicate-4.c | 11 +-
> 5 files changed, 276 insertions(+), 136 deletions(-)
> create mode 100644 gcc/testsuite/gcc.target/riscv/rvv/autovec/pr111232.c
>
> diff --git a/gcc/config/riscv/autovec-opt.md b/gcc/config/riscv/autovec-opt.md
> index d9863c76654..3aaee54f02a 100644
> --- a/gcc/config/riscv/autovec-opt.md
> +++ b/gcc/config/riscv/autovec-opt.md
> @@ -18,67 +18,6 @@
> ;; along with GCC; see the file COPYING3. If not see
> ;; <http://www.gnu.org/licenses/>.
>
> -;; We don't have vwmul.wv instruction like vwadd.wv in RVV.
> -;; This pattern is an intermediate RTL IR as a pseudo vwmul.wv to enhance
> -;; optimization of instructions combine.
> -(define_insn_and_split "@pred_single_widen_mul<any_extend:su><mode>"
> - [(set (match_operand:VWEXTI 0 "register_operand" "=&vr,&vr")
> - (if_then_else:VWEXTI
> - (unspec:<VM>
> - [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1,vmWc1")
> - (match_operand 5 "vector_length_operand" " rK, rK")
> - (match_operand 6 "const_int_operand" " i, i")
> - (match_operand 7 "const_int_operand" " i, i")
> - (match_operand 8 "const_int_operand" " i, i")
> - (reg:SI VL_REGNUM)
> - (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
> - (mult:VWEXTI
> - (any_extend:VWEXTI
> - (match_operand:<V_DOUBLE_TRUNC> 4 "register_operand" " vr, vr"))
> - (match_operand:VWEXTI 3 "register_operand" " vr, vr"))
> - (match_operand:VWEXTI 2 "vector_merge_operand" " vu, 0")))]
> - "TARGET_VECTOR && can_create_pseudo_p ()"
> - "#"
> - "&& 1"
> - [(const_int 0)]
> - {
> - insn_code icode = code_for_pred_vf2 (<CODE>, <MODE>mode);
> - rtx tmp = gen_reg_rtx (<MODE>mode);
> - rtx ops[] = {tmp, operands[4]};
> - riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP, ops);
> -
> - emit_insn (gen_pred (MULT, <MODE>mode, operands[0], operands[1], operands[2],
> - operands[3], tmp, operands[5], operands[6],
> - operands[7], operands[8]));
> - DONE;
> - }
> - [(set_attr "type" "viwmul")
> - (set_attr "mode" "<MODE>")])
> -
> -;; This pattern it to enchance the instruction combine optimizations for complicate
> -;; sign and unsigned widening multiplication operations.
> -(define_insn "*pred_widen_mulsu<mode>"
> - [(set (match_operand:VWEXTI 0 "register_operand" "=&vr,&vr")
> - (if_then_else:VWEXTI
> - (unspec:<VM>
> - [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1,vmWc1")
> - (match_operand 5 "vector_length_operand" " rK, rK")
> - (match_operand 6 "const_int_operand" " i, i")
> - (match_operand 7 "const_int_operand" " i, i")
> - (match_operand 8 "const_int_operand" " i, i")
> - (reg:SI VL_REGNUM)
> - (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
> - (mult:VWEXTI
> - (zero_extend:VWEXTI
> - (match_operand:<V_DOUBLE_TRUNC> 4 "register_operand" " vr, vr"))
> - (sign_extend:VWEXTI
> - (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand" " vr, vr")))
> - (match_operand:VWEXTI 2 "vector_merge_operand" " vu, 0")))]
> - "TARGET_VECTOR"
> - "vwmulsu.vv\t%0,%3,%4%p1"
> - [(set_attr "type" "viwmul")
> - (set_attr "mode" "<V_DOUBLE_TRUNC>")])
> -
> ;; -----------------------------------------------------------------------------
> ;; ---- Integer Compare Instructions Simplification
> ;; -----------------------------------------------------------------------------
> @@ -406,45 +345,6 @@
> [(set_attr "type" "vimovvx")
> (set_attr "mode" "<MODE>")])
>
> -;; We don't have vfwmul.wv instruction like vfwadd.wv in RVV.
> -;; This pattern is an intermediate RTL IR as a pseudo vfwmul.wv to enhance
> -;; optimization of instructions combine.
> -(define_insn_and_split "*pred_single_widen_mul<mode>"
> - [(set (match_operand:VWEXTF 0 "register_operand" "=&vr, &vr")
> - (if_then_else:VWEXTF
> - (unspec:<VM>
> - [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1,vmWc1")
> - (match_operand 5 "vector_length_operand" " rK, rK")
> - (match_operand 6 "const_int_operand" " i, i")
> - (match_operand 7 "const_int_operand" " i, i")
> - (match_operand 8 "const_int_operand" " i, i")
> - (match_operand 9 "const_int_operand" " i, i")
> - (reg:SI VL_REGNUM)
> - (reg:SI VTYPE_REGNUM)
> - (reg:SI FRM_REGNUM)] UNSPEC_VPREDICATE)
> - (mult:VWEXTF
> - (float_extend:VWEXTF
> - (match_operand:<V_DOUBLE_TRUNC> 4 "register_operand" " vr, vr"))
> - (match_operand:VWEXTF 3 "register_operand" " vr, vr"))
> - (match_operand:VWEXTF 2 "vector_merge_operand" " vu, 0")))]
> - "TARGET_VECTOR && can_create_pseudo_p ()"
> - "#"
> - "&& 1"
> - [(const_int 0)]
> - {
> - insn_code icode = code_for_pred_extend (<MODE>mode);
> - rtx tmp = gen_reg_rtx (<MODE>mode);
> - rtx ops[] = {tmp, operands[4]};
> - riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP, ops);
> -
> - emit_insn (gen_pred (MULT, <MODE>mode, operands[0], operands[1], operands[2],
> - operands[3], tmp, operands[5], operands[6],
> - operands[7], operands[8], operands[9]));
> - DONE;
> - }
> - [(set_attr "type" "vfwmul")
> - (set_attr "mode" "<MODE>")])
> -
> ;; -------------------------------------------------------------------------
> ;; ---- [FP] VFWMACC
> ;; -------------------------------------------------------------------------
> @@ -845,7 +745,7 @@
> DONE;
> })
>
> -;; Combine FP sign_extend/zero_extend(vf2) and vcond_mask
> +;; Combine FP extend(vf2) and vcond_mask
> (define_insn_and_split "*cond_extend<v_double_trunc><mode>"
> [(set (match_operand:VWEXTF_ZVFHMIN 0 "register_operand")
> (if_then_else:VWEXTF_ZVFHMIN
> @@ -1003,3 +903,195 @@
> riscv_vector::expand_cond_len_unop (icode, ops);
> DONE;
> })
> +
> +;; =============================================================================
> +;; Combine extend + binop to widen_binop
> +;; =============================================================================
> +
> +(define_insn_and_split "*dual_widen_<any_widen_binop:optab><any_extend:su><mode>"
> + [(set (match_operand:VWEXTI 0 "register_operand")
> + (any_widen_binop:VWEXTI
> + (any_extend:VWEXTI
> + (match_operand:<V_DOUBLE_TRUNC> 1 "register_operand"))
> + (any_extend:VWEXTI
> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))))]
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> + "#"
> + "&& 1"
> + [(const_int 0)]
> +{
> + insn_code icode = code_for_pred_dual_widen (<any_widen_binop:CODE>,
> + <any_extend:CODE>,
> + <MODE>mode);
> + riscv_vector::emit_vlmax_insn (icode, riscv_vector::BINARY_OP, operands);
> + DONE;
> +})
> +
> +(define_insn_and_split "*single_widen_sub<any_extend:su><mode>"
> + [(set (match_operand:VWEXTI 0 "register_operand")
> + (minus:VWEXTI
> + (match_operand:VWEXTI 1 "register_operand")
> + (any_extend:VWEXTI
> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))))]
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> + "#"
> + "&& 1"
> + [(const_int 0)]
> +{
> + insn_code icode = code_for_pred_single_widen_sub (<any_extend:CODE>,
> + <MODE>mode);
> + riscv_vector::emit_vlmax_insn (icode, riscv_vector::BINARY_OP, operands);
> + DONE;
> +})
> +
> +(define_insn_and_split "*single_widen_add<any_extend:su><mode>"
> + [(set (match_operand:VWEXTI 0 "register_operand")
> + (plus:VWEXTI
> + (any_extend:VWEXTI
> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
> + (match_operand:VWEXTI 1 "register_operand")))]
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> + "#"
> + "&& 1"
> + [(const_int 0)]
> +{
> + insn_code icode = code_for_pred_single_widen_add (<any_extend:CODE>,
> + <MODE>mode);
> + riscv_vector::emit_vlmax_insn (icode, riscv_vector::BINARY_OP, operands);
> + DONE;
> +})
> +
> +;; This combine pattern does not correspond to an single instruction,
> +;; i.e. there is no vwmul.wv instruction. This is a temporary pattern
> +;; produced by a combine pass and if there is no further combine into
> +;; vwmul.vv pattern, then fall back to extend pattern and vmul.vv pattern.
> +(define_insn_and_split "*single_widen_mult<any_extend:su><mode>"
> + [(set (match_operand:VWEXTI 0 "register_operand")
> + (mult:VWEXTI
> + (any_extend:VWEXTI
> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
> + (match_operand:VWEXTI 1 "register_operand")))]
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> + "#"
> + "&& 1"
> + [(const_int 0)]
> +{
> + insn_code extend_icode = code_for_pred_vf2 (<any_extend:CODE>, <MODE>mode);
> + rtx tmp = gen_reg_rtx (<MODE>mode);
> + rtx extend_ops[] = {tmp, operands[2]};
> + riscv_vector::emit_vlmax_insn (extend_icode, riscv_vector::UNARY_OP, extend_ops);
> +
> + rtx ops[] = {operands[0], operands[1], tmp};
> + insn_code icode = code_for_pred (MULT, <MODE>mode);
> + riscv_vector::emit_vlmax_insn (icode, riscv_vector::BINARY_OP, ops);
> + DONE;
> +})
> +
> +(define_insn_and_split "*dual_widen_mulsu<mode>"
> + [(set (match_operand:VWEXTI 0 "register_operand")
> + (mult:VWEXTI
> + (sign_extend:VWEXTI
> + (match_operand:<V_DOUBLE_TRUNC> 1 "register_operand"))
> + (zero_extend:VWEXTI
> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))))]
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> + "#"
> + "&& 1"
> + [(const_int 0)]
> +{
> + insn_code icode = code_for_pred_widen_mulsu (<MODE>mode);
> + riscv_vector::emit_vlmax_insn (icode, riscv_vector::BINARY_OP, operands);
> + DONE;
> +})
> +
> +(define_insn_and_split "*dual_widen_mulus<mode>"
> + [(set (match_operand:VWEXTI 0 "register_operand")
> + (mult:VWEXTI
> + (zero_extend:VWEXTI
> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
> + (sign_extend:VWEXTI
> + (match_operand:<V_DOUBLE_TRUNC> 1 "register_operand"))))]
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> + "#"
> + "&& 1"
> + [(const_int 0)]
> +{
> + insn_code icode = code_for_pred_widen_mulsu (<MODE>mode);
> + riscv_vector::emit_vlmax_insn (icode, riscv_vector::BINARY_OP, operands);
> + DONE;
> +})
> +
> +(define_insn_and_split "*dual_widen_<optab><mode>"
> + [(set (match_operand:VWEXTF 0 "register_operand")
> + (any_widen_binop:VWEXTF
> + (float_extend:VWEXTF
> + (match_operand:<V_DOUBLE_TRUNC> 1 "register_operand"))
> + (float_extend:VWEXTF
> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))))]
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> + "#"
> + "&& 1"
> + [(const_int 0)]
> +{
> + insn_code icode = code_for_pred_dual_widen (<CODE>, <MODE>mode);
> + riscv_vector::emit_vlmax_insn (icode, riscv_vector::BINARY_OP_FRM_DYN, operands);
> + DONE;
> +})
> +
> +(define_insn_and_split "*single_widen_add<mode>"
> + [(set (match_operand:VWEXTF 0 "register_operand")
> + (plus:VWEXTF
> + (float_extend:VWEXTF
> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
> + (match_operand:VWEXTF 1 "register_operand")))]
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> + "#"
> + "&& 1"
> + [(const_int 0)]
> +{
> + insn_code icode = code_for_pred_single_widen_add (<MODE>mode);
> + riscv_vector::emit_vlmax_insn (icode, riscv_vector::BINARY_OP_FRM_DYN, operands);
> + DONE;
> +})
> +
> +(define_insn_and_split "*single_widen_sub<mode>"
> + [(set (match_operand:VWEXTF 0 "register_operand")
> + (minus:VWEXTF
> + (match_operand:VWEXTF 1 "register_operand")
> + (float_extend:VWEXTF
> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))))]
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> + "#"
> + "&& 1"
> + [(const_int 0)]
> +{
> + insn_code icode = code_for_pred_single_widen_sub (<MODE>mode);
> + riscv_vector::emit_vlmax_insn (icode, riscv_vector::BINARY_OP_FRM_DYN, operands);
> + DONE;
> +})
> +
> +;; This combine pattern does not correspond to an single instruction,
> +;; i.e. there is no vfwmul.wv instruction. This is a temporary pattern
> +;; produced by a combine pass and if there is no further combine into
> +;; vfwmul.vv pattern, then fall back to extend pattern and vfmul.vv pattern.
> +(define_insn_and_split "*single_widen_mult<mode>"
> + [(set (match_operand:VWEXTF 0 "register_operand")
> + (mult:VWEXTF
> + (float_extend:VWEXTF
> + (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
> + (match_operand:VWEXTF 1 "register_operand")))]
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> + "#"
> + "&& 1"
> + [(const_int 0)]
> +{
> + insn_code extend_icode = code_for_pred_extend (<MODE>mode);
> + rtx tmp = gen_reg_rtx (<MODE>mode);
> + rtx extend_ops[] = {tmp, operands[2]};
> + riscv_vector::emit_vlmax_insn (extend_icode, riscv_vector::UNARY_OP, extend_ops);
> +
> + rtx ops[] = {operands[0], operands[1], tmp};
> + riscv_vector::emit_vlmax_insn (code_for_pred (MULT, <MODE>mode),
> + riscv_vector::BINARY_OP_FRM_DYN, ops);
> + DONE;
> +})
> diff --git a/gcc/config/riscv/autovec.md b/gcc/config/riscv/autovec.md
> index c220fda312e..98cd0c07625 100644
> --- a/gcc/config/riscv/autovec.md
> +++ b/gcc/config/riscv/autovec.md
> @@ -419,12 +419,15 @@
> ;; - vadd.vi/vsub.vi/...
> ;; -------------------------------------------------------------------------
>
> -(define_expand "<optab><mode>3"
> +(define_insn_and_split "<optab><mode>3"
> [(set (match_operand:VI 0 "register_operand")
> (any_int_binop_no_shift:VI
> (match_operand:VI 1 "<binop_rhs1_predicate>")
> (match_operand:VI 2 "<binop_rhs2_predicate>")))]
> - "TARGET_VECTOR"
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> + "#"
> + "&& 1"
> + [(const_int 0)]
> {
> riscv_vector::emit_vlmax_insn (code_for_pred (<CODE>, <MODE>mode),
> riscv_vector::BINARY_OP, operands);
> @@ -937,11 +940,14 @@
> ;; Includes:
> ;; - vneg.v/vnot.v
> ;; -------------------------------------------------------------------------------
> -(define_expand "<optab><mode>2"
> +(define_insn_and_split "<optab><mode>2"
> [(set (match_operand:VI 0 "register_operand")
> (any_int_unop:VI
> (match_operand:VI 1 "register_operand")))]
> - "TARGET_VECTOR"
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> + "#"
> + "&& 1"
> + [(const_int 0)]
> {
> insn_code icode = code_for_pred (<CODE>, <MODE>mode);
> riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP, operands);
> @@ -952,10 +958,14 @@
> ;; - [INT] ABS expansion to vmslt and vneg.
> ;; -------------------------------------------------------------------------------
>
> -(define_expand "abs<mode>2"
> +(define_insn_and_split "abs<mode>2"
> [(set (match_operand:VI 0 "register_operand")
> - (match_operand:VI 1 "register_operand"))]
> - "TARGET_VECTOR"
> + (abs:VI
> + (match_operand:VI 1 "register_operand")))]
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> + "#"
> + "&& 1"
> + [(const_int 0)]
> {
> rtx zero = gen_const_vec_duplicate (<MODE>mode, GEN_INT (0));
> machine_mode mask_mode = riscv_vector::get_mask_mode (<MODE>mode);
> @@ -1457,12 +1467,15 @@
> ;; - vfadd.vv/vfsub.vv/...
> ;; - vfadd.vf/vfsub.vf/...
> ;; -------------------------------------------------------------------------
> -(define_expand "<optab><mode>3"
> - [(match_operand:VF 0 "register_operand")
> - (any_float_binop:VF
> - (match_operand:VF 1 "register_operand")
> - (match_operand:VF 2 "register_operand"))]
> - "TARGET_VECTOR"
> +(define_insn_and_split "<optab><mode>3"
> + [(set (match_operand:VF 0 "register_operand")
> + (any_float_binop:VF
> + (match_operand:VF 1 "register_operand")
> + (match_operand:VF 2 "register_operand")))]
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> + "#"
> + "&& 1"
> + [(const_int 0)]
> {
> riscv_vector::emit_vlmax_insn (code_for_pred (<CODE>, <MODE>mode),
> riscv_vector::BINARY_OP_FRM_DYN, operands);
> @@ -1474,12 +1487,15 @@
> ;; - vfmin.vv/vfmax.vv
> ;; - vfmin.vf/vfmax.vf
> ;; -------------------------------------------------------------------------
> -(define_expand "<optab><mode>3"
> - [(match_operand:VF 0 "register_operand")
> - (any_float_binop_nofrm:VF
> - (match_operand:VF 1 "register_operand")
> - (match_operand:VF 2 "register_operand"))]
> - "TARGET_VECTOR"
> +(define_insn_and_split "<optab><mode>3"
> + [(set (match_operand:VF 0 "register_operand")
> + (any_float_binop_nofrm:VF
> + (match_operand:VF 1 "register_operand")
> + (match_operand:VF 2 "register_operand")))]
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> + "#"
> + "&& 1"
> + [(const_int 0)]
> {
> riscv_vector::emit_vlmax_insn (code_for_pred (<CODE>, <MODE>mode),
> riscv_vector::BINARY_OP, operands);
> @@ -1537,22 +1553,30 @@
> ;; - vmulhu.vv
> ;; -------------------------------------------------------------------------
>
> -(define_expand "smul<mode>3_highpart"
> - [(match_operand:VFULLI 0 "register_operand")
> - (match_operand:VFULLI 1 "register_operand")
> - (match_operand:VFULLI 2 "register_operand")]
> - "TARGET_VECTOR"
> +(define_insn_and_split "smul<mode>3_highpart"
> + [(set (match_operand:VFULLI 0 "register_operand")
> + (smul_highpart:VFULLI
> + (match_operand:VFULLI 1 "register_operand")
> + (match_operand:VFULLI 2 "register_operand")))]
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> + "#"
> + "&& 1"
> + [(const_int 0)]
> {
> insn_code icode = code_for_pred_mulh (UNSPEC_VMULHS, <MODE>mode);
> riscv_vector::emit_vlmax_insn (icode, riscv_vector::BINARY_OP, operands);
> DONE;
> })
>
> -(define_expand "umul<mode>3_highpart"
> - [(match_operand:VFULLI 0 "register_operand")
> - (match_operand:VFULLI 1 "register_operand")
> - (match_operand:VFULLI 2 "register_operand")]
> - "TARGET_VECTOR"
> +(define_insn_and_split "umul<mode>3_highpart"
> + [(set (match_operand:VFULLI 0 "register_operand")
> + (umul_highpart:VFULLI
> + (match_operand:VFULLI 1 "register_operand")
> + (match_operand:VFULLI 2 "register_operand")))]
> + "TARGET_VECTOR && can_create_pseudo_p ()"
> + "#"
> + "&& 1"
> + [(const_int 0)]
> {
> insn_code icode = code_for_pred_mulh (UNSPEC_VMULHU, <MODE>mode);
> riscv_vector::emit_vlmax_insn (icode, riscv_vector::BINARY_OP, operands);
> diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr111232.c b/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr111232.c
> new file mode 100644
> index 00000000000..de815c5fac9
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr111232.c
> @@ -0,0 +1,18 @@
> +/* { dg-do compile } */
> +/* { dg-options "-march=rv64gcv -mabi=lp64d --param=riscv-autovec-preference=scalable -Ofast -fno-schedule-insns -fno-schedule-insns2" } */
> +
> +#include <stdint.h>
> +
> +int16_t
> +foo (int8_t *restrict x, int8_t *restrict y, int n)
> +{
> + int16_t result = 0;
> +
> + for (int i = 0; i < n; i++)
> + {
> + result += (x[i] * y[i]);
> + }
> + return result;
> +}
> +
> +/* { dg-final { scan-assembler {\tvwmacc\.vv\tv[0-9]+,v[0-9]+,v[0-9]+} } } */
> diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/widen/widen-4.c b/gcc/testsuite/gcc.target/riscv/rvv/autovec/widen/widen-4.c
> index c29a74c4f8b..26f27ea6283 100644
> --- a/gcc/testsuite/gcc.target/riscv/rvv/autovec/widen/widen-4.c
> +++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/widen/widen-4.c
> @@ -16,8 +16,11 @@
> #define TEST_ALL() \
> TEST_TYPE (int16_t, int8_t, uint8_t) \
> TEST_TYPE (int32_t, int16_t, uint16_t) \
> - TEST_TYPE (int64_t, int32_t, uint32_t)
> + TEST_TYPE (int64_t, int32_t, uint32_t) \
> + TEST_TYPE (int16_t, uint8_t, int8_t) \
> + TEST_TYPE (int32_t, uint16_t, int16_t) \
> + TEST_TYPE (int64_t, uint32_t, int32_t)
>
> TEST_ALL ()
>
> -/* { dg-final { scan-assembler-times {\tvwmulsu\.vv} 3 } } */
> +/* { dg-final { scan-assembler-times {\tvwmulsu\.vv} 6 } } */
> diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/widen/widen-complicate-4.c b/gcc/testsuite/gcc.target/riscv/rvv/autovec/widen/widen-complicate-4.c
> index 15fdefc550b..aeac4cb79c2 100644
> --- a/gcc/testsuite/gcc.target/riscv/rvv/autovec/widen/widen-complicate-4.c
> +++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/widen/widen-complicate-4.c
> @@ -21,11 +21,14 @@
> #define TEST_ALL() \
> TEST_TYPE (int16_t, int8_t, uint8_t) \
> TEST_TYPE (int32_t, int16_t, uint16_t) \
> - TEST_TYPE (int64_t, int32_t, uint32_t)
> + TEST_TYPE (int64_t, int32_t, uint32_t) \
> + TEST_TYPE (int16_t, uint8_t, int8_t) \
> + TEST_TYPE (int32_t, uint16_t, int16_t) \
> + TEST_TYPE (int64_t, uint32_t, int32_t)
>
> TEST_ALL ()
>
> -/* { dg-final { scan-assembler-times {\tvwmulsu\.vv} 6 } } */
> -/* { dg-final { scan-assembler-times {\tvwmul\.vv} 3 } } */
> -/* { dg-final { scan-assembler-times {\tvwmulu\.vv} 3 } } */
> +/* { dg-final { scan-assembler-times {\tvwmulsu\.vv} 12 } } */
> +/* { dg-final { scan-assembler-times {\tvwmul\.vv} 6 } } */
> +/* { dg-final { scan-assembler-times {\tvwmulu\.vv} 6 } } */
> /* { dg-final { scan-assembler-not {\tvmul} } } */
> --
> 2.36.3
>
Hi Kito,
> Can those intermediate patterns be used for intrinsic? I would prefer
> to keep those stuff *IF* possible used for intrinsics.
I think we don't need those patterns for intrinisc. First, the deleted
pattern does not directly correspond to intrinisc. Second, if you want
to use these patterns to optimize the following intrinisc program (e.g.
combine to a single vwadd.vv instruction), we'd better fold them
directly into the internal function of autovec in Gimple IR.
vint32m1_t foo(vint16mf2_t va1, vint16mf2_t vb1) {
vint32m1_t va2 = __riscv_vsext_vf2_i32m1(va1,
__riscv_vsetvlmax_e32m1());
vint32m1_t vb2 = __riscv_vsext_vf2_i32m1(vb1,
__riscv_vsetvlmax_e32m1());
vint32m1_t vc = __riscv_vadd_vv_i32m1(va2, vb2,
__riscv_vsetvlmax_e32m1());
return vc;
}
Thanks for explaining, LGTM :)
On Mon, Sep 4, 2023 at 11:39 PM Lehua Ding <lehua.ding@rivai.ai> wrote:
>
> Hi Kito,
>
> > Can those intermediate patterns be used for intrinsic? I would prefer
> > to keep those stuff *IF* possible used for intrinsics.
>
> I think we don't need those patterns for intrinisc. First, the deleted
> pattern does not directly correspond to intrinisc. Second, if you want
> to use these patterns to optimize the following intrinisc program (e.g.
> combine to a single vwadd.vv instruction), we'd better fold them
> directly into the internal function of autovec in Gimple IR.
>
> vint32m1_t foo(vint16mf2_t va1, vint16mf2_t vb1) {
> vint32m1_t va2 = __riscv_vsext_vf2_i32m1(va1,
> __riscv_vsetvlmax_e32m1());
> vint32m1_t vb2 = __riscv_vsext_vf2_i32m1(vb1,
> __riscv_vsetvlmax_e32m1());
> vint32m1_t vc = __riscv_vadd_vv_i32m1(va2, vb2,
> __riscv_vsetvlmax_e32m1());
> return vc;
> }
> --
> Best,
> Lehua
>
>
@@ -18,67 +18,6 @@
;; along with GCC; see the file COPYING3. If not see
;; <http://www.gnu.org/licenses/>.
-;; We don't have vwmul.wv instruction like vwadd.wv in RVV.
-;; This pattern is an intermediate RTL IR as a pseudo vwmul.wv to enhance
-;; optimization of instructions combine.
-(define_insn_and_split "@pred_single_widen_mul<any_extend:su><mode>"
- [(set (match_operand:VWEXTI 0 "register_operand" "=&vr,&vr")
- (if_then_else:VWEXTI
- (unspec:<VM>
- [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1,vmWc1")
- (match_operand 5 "vector_length_operand" " rK, rK")
- (match_operand 6 "const_int_operand" " i, i")
- (match_operand 7 "const_int_operand" " i, i")
- (match_operand 8 "const_int_operand" " i, i")
- (reg:SI VL_REGNUM)
- (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
- (mult:VWEXTI
- (any_extend:VWEXTI
- (match_operand:<V_DOUBLE_TRUNC> 4 "register_operand" " vr, vr"))
- (match_operand:VWEXTI 3 "register_operand" " vr, vr"))
- (match_operand:VWEXTI 2 "vector_merge_operand" " vu, 0")))]
- "TARGET_VECTOR && can_create_pseudo_p ()"
- "#"
- "&& 1"
- [(const_int 0)]
- {
- insn_code icode = code_for_pred_vf2 (<CODE>, <MODE>mode);
- rtx tmp = gen_reg_rtx (<MODE>mode);
- rtx ops[] = {tmp, operands[4]};
- riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP, ops);
-
- emit_insn (gen_pred (MULT, <MODE>mode, operands[0], operands[1], operands[2],
- operands[3], tmp, operands[5], operands[6],
- operands[7], operands[8]));
- DONE;
- }
- [(set_attr "type" "viwmul")
- (set_attr "mode" "<MODE>")])
-
-;; This pattern it to enchance the instruction combine optimizations for complicate
-;; sign and unsigned widening multiplication operations.
-(define_insn "*pred_widen_mulsu<mode>"
- [(set (match_operand:VWEXTI 0 "register_operand" "=&vr,&vr")
- (if_then_else:VWEXTI
- (unspec:<VM>
- [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1,vmWc1")
- (match_operand 5 "vector_length_operand" " rK, rK")
- (match_operand 6 "const_int_operand" " i, i")
- (match_operand 7 "const_int_operand" " i, i")
- (match_operand 8 "const_int_operand" " i, i")
- (reg:SI VL_REGNUM)
- (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)
- (mult:VWEXTI
- (zero_extend:VWEXTI
- (match_operand:<V_DOUBLE_TRUNC> 4 "register_operand" " vr, vr"))
- (sign_extend:VWEXTI
- (match_operand:<V_DOUBLE_TRUNC> 3 "register_operand" " vr, vr")))
- (match_operand:VWEXTI 2 "vector_merge_operand" " vu, 0")))]
- "TARGET_VECTOR"
- "vwmulsu.vv\t%0,%3,%4%p1"
- [(set_attr "type" "viwmul")
- (set_attr "mode" "<V_DOUBLE_TRUNC>")])
-
;; -----------------------------------------------------------------------------
;; ---- Integer Compare Instructions Simplification
;; -----------------------------------------------------------------------------
@@ -406,45 +345,6 @@
[(set_attr "type" "vimovvx")
(set_attr "mode" "<MODE>")])
-;; We don't have vfwmul.wv instruction like vfwadd.wv in RVV.
-;; This pattern is an intermediate RTL IR as a pseudo vfwmul.wv to enhance
-;; optimization of instructions combine.
-(define_insn_and_split "*pred_single_widen_mul<mode>"
- [(set (match_operand:VWEXTF 0 "register_operand" "=&vr, &vr")
- (if_then_else:VWEXTF
- (unspec:<VM>
- [(match_operand:<VM> 1 "vector_mask_operand" "vmWc1,vmWc1")
- (match_operand 5 "vector_length_operand" " rK, rK")
- (match_operand 6 "const_int_operand" " i, i")
- (match_operand 7 "const_int_operand" " i, i")
- (match_operand 8 "const_int_operand" " i, i")
- (match_operand 9 "const_int_operand" " i, i")
- (reg:SI VL_REGNUM)
- (reg:SI VTYPE_REGNUM)
- (reg:SI FRM_REGNUM)] UNSPEC_VPREDICATE)
- (mult:VWEXTF
- (float_extend:VWEXTF
- (match_operand:<V_DOUBLE_TRUNC> 4 "register_operand" " vr, vr"))
- (match_operand:VWEXTF 3 "register_operand" " vr, vr"))
- (match_operand:VWEXTF 2 "vector_merge_operand" " vu, 0")))]
- "TARGET_VECTOR && can_create_pseudo_p ()"
- "#"
- "&& 1"
- [(const_int 0)]
- {
- insn_code icode = code_for_pred_extend (<MODE>mode);
- rtx tmp = gen_reg_rtx (<MODE>mode);
- rtx ops[] = {tmp, operands[4]};
- riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP, ops);
-
- emit_insn (gen_pred (MULT, <MODE>mode, operands[0], operands[1], operands[2],
- operands[3], tmp, operands[5], operands[6],
- operands[7], operands[8], operands[9]));
- DONE;
- }
- [(set_attr "type" "vfwmul")
- (set_attr "mode" "<MODE>")])
-
;; -------------------------------------------------------------------------
;; ---- [FP] VFWMACC
;; -------------------------------------------------------------------------
@@ -845,7 +745,7 @@
DONE;
})
-;; Combine FP sign_extend/zero_extend(vf2) and vcond_mask
+;; Combine FP extend(vf2) and vcond_mask
(define_insn_and_split "*cond_extend<v_double_trunc><mode>"
[(set (match_operand:VWEXTF_ZVFHMIN 0 "register_operand")
(if_then_else:VWEXTF_ZVFHMIN
@@ -1003,3 +903,195 @@
riscv_vector::expand_cond_len_unop (icode, ops);
DONE;
})
+
+;; =============================================================================
+;; Combine extend + binop to widen_binop
+;; =============================================================================
+
+(define_insn_and_split "*dual_widen_<any_widen_binop:optab><any_extend:su><mode>"
+ [(set (match_operand:VWEXTI 0 "register_operand")
+ (any_widen_binop:VWEXTI
+ (any_extend:VWEXTI
+ (match_operand:<V_DOUBLE_TRUNC> 1 "register_operand"))
+ (any_extend:VWEXTI
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+{
+ insn_code icode = code_for_pred_dual_widen (<any_widen_binop:CODE>,
+ <any_extend:CODE>,
+ <MODE>mode);
+ riscv_vector::emit_vlmax_insn (icode, riscv_vector::BINARY_OP, operands);
+ DONE;
+})
+
+(define_insn_and_split "*single_widen_sub<any_extend:su><mode>"
+ [(set (match_operand:VWEXTI 0 "register_operand")
+ (minus:VWEXTI
+ (match_operand:VWEXTI 1 "register_operand")
+ (any_extend:VWEXTI
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+{
+ insn_code icode = code_for_pred_single_widen_sub (<any_extend:CODE>,
+ <MODE>mode);
+ riscv_vector::emit_vlmax_insn (icode, riscv_vector::BINARY_OP, operands);
+ DONE;
+})
+
+(define_insn_and_split "*single_widen_add<any_extend:su><mode>"
+ [(set (match_operand:VWEXTI 0 "register_operand")
+ (plus:VWEXTI
+ (any_extend:VWEXTI
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
+ (match_operand:VWEXTI 1 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+{
+ insn_code icode = code_for_pred_single_widen_add (<any_extend:CODE>,
+ <MODE>mode);
+ riscv_vector::emit_vlmax_insn (icode, riscv_vector::BINARY_OP, operands);
+ DONE;
+})
+
+;; This combine pattern does not correspond to an single instruction,
+;; i.e. there is no vwmul.wv instruction. This is a temporary pattern
+;; produced by a combine pass and if there is no further combine into
+;; vwmul.vv pattern, then fall back to extend pattern and vmul.vv pattern.
+(define_insn_and_split "*single_widen_mult<any_extend:su><mode>"
+ [(set (match_operand:VWEXTI 0 "register_operand")
+ (mult:VWEXTI
+ (any_extend:VWEXTI
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
+ (match_operand:VWEXTI 1 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+{
+ insn_code extend_icode = code_for_pred_vf2 (<any_extend:CODE>, <MODE>mode);
+ rtx tmp = gen_reg_rtx (<MODE>mode);
+ rtx extend_ops[] = {tmp, operands[2]};
+ riscv_vector::emit_vlmax_insn (extend_icode, riscv_vector::UNARY_OP, extend_ops);
+
+ rtx ops[] = {operands[0], operands[1], tmp};
+ insn_code icode = code_for_pred (MULT, <MODE>mode);
+ riscv_vector::emit_vlmax_insn (icode, riscv_vector::BINARY_OP, ops);
+ DONE;
+})
+
+(define_insn_and_split "*dual_widen_mulsu<mode>"
+ [(set (match_operand:VWEXTI 0 "register_operand")
+ (mult:VWEXTI
+ (sign_extend:VWEXTI
+ (match_operand:<V_DOUBLE_TRUNC> 1 "register_operand"))
+ (zero_extend:VWEXTI
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+{
+ insn_code icode = code_for_pred_widen_mulsu (<MODE>mode);
+ riscv_vector::emit_vlmax_insn (icode, riscv_vector::BINARY_OP, operands);
+ DONE;
+})
+
+(define_insn_and_split "*dual_widen_mulus<mode>"
+ [(set (match_operand:VWEXTI 0 "register_operand")
+ (mult:VWEXTI
+ (zero_extend:VWEXTI
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
+ (sign_extend:VWEXTI
+ (match_operand:<V_DOUBLE_TRUNC> 1 "register_operand"))))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+{
+ insn_code icode = code_for_pred_widen_mulsu (<MODE>mode);
+ riscv_vector::emit_vlmax_insn (icode, riscv_vector::BINARY_OP, operands);
+ DONE;
+})
+
+(define_insn_and_split "*dual_widen_<optab><mode>"
+ [(set (match_operand:VWEXTF 0 "register_operand")
+ (any_widen_binop:VWEXTF
+ (float_extend:VWEXTF
+ (match_operand:<V_DOUBLE_TRUNC> 1 "register_operand"))
+ (float_extend:VWEXTF
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+{
+ insn_code icode = code_for_pred_dual_widen (<CODE>, <MODE>mode);
+ riscv_vector::emit_vlmax_insn (icode, riscv_vector::BINARY_OP_FRM_DYN, operands);
+ DONE;
+})
+
+(define_insn_and_split "*single_widen_add<mode>"
+ [(set (match_operand:VWEXTF 0 "register_operand")
+ (plus:VWEXTF
+ (float_extend:VWEXTF
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
+ (match_operand:VWEXTF 1 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+{
+ insn_code icode = code_for_pred_single_widen_add (<MODE>mode);
+ riscv_vector::emit_vlmax_insn (icode, riscv_vector::BINARY_OP_FRM_DYN, operands);
+ DONE;
+})
+
+(define_insn_and_split "*single_widen_sub<mode>"
+ [(set (match_operand:VWEXTF 0 "register_operand")
+ (minus:VWEXTF
+ (match_operand:VWEXTF 1 "register_operand")
+ (float_extend:VWEXTF
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+{
+ insn_code icode = code_for_pred_single_widen_sub (<MODE>mode);
+ riscv_vector::emit_vlmax_insn (icode, riscv_vector::BINARY_OP_FRM_DYN, operands);
+ DONE;
+})
+
+;; This combine pattern does not correspond to an single instruction,
+;; i.e. there is no vfwmul.wv instruction. This is a temporary pattern
+;; produced by a combine pass and if there is no further combine into
+;; vfwmul.vv pattern, then fall back to extend pattern and vfmul.vv pattern.
+(define_insn_and_split "*single_widen_mult<mode>"
+ [(set (match_operand:VWEXTF 0 "register_operand")
+ (mult:VWEXTF
+ (float_extend:VWEXTF
+ (match_operand:<V_DOUBLE_TRUNC> 2 "register_operand"))
+ (match_operand:VWEXTF 1 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
+{
+ insn_code extend_icode = code_for_pred_extend (<MODE>mode);
+ rtx tmp = gen_reg_rtx (<MODE>mode);
+ rtx extend_ops[] = {tmp, operands[2]};
+ riscv_vector::emit_vlmax_insn (extend_icode, riscv_vector::UNARY_OP, extend_ops);
+
+ rtx ops[] = {operands[0], operands[1], tmp};
+ riscv_vector::emit_vlmax_insn (code_for_pred (MULT, <MODE>mode),
+ riscv_vector::BINARY_OP_FRM_DYN, ops);
+ DONE;
+})
@@ -419,12 +419,15 @@
;; - vadd.vi/vsub.vi/...
;; -------------------------------------------------------------------------
-(define_expand "<optab><mode>3"
+(define_insn_and_split "<optab><mode>3"
[(set (match_operand:VI 0 "register_operand")
(any_int_binop_no_shift:VI
(match_operand:VI 1 "<binop_rhs1_predicate>")
(match_operand:VI 2 "<binop_rhs2_predicate>")))]
- "TARGET_VECTOR"
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
{
riscv_vector::emit_vlmax_insn (code_for_pred (<CODE>, <MODE>mode),
riscv_vector::BINARY_OP, operands);
@@ -937,11 +940,14 @@
;; Includes:
;; - vneg.v/vnot.v
;; -------------------------------------------------------------------------------
-(define_expand "<optab><mode>2"
+(define_insn_and_split "<optab><mode>2"
[(set (match_operand:VI 0 "register_operand")
(any_int_unop:VI
(match_operand:VI 1 "register_operand")))]
- "TARGET_VECTOR"
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
{
insn_code icode = code_for_pred (<CODE>, <MODE>mode);
riscv_vector::emit_vlmax_insn (icode, riscv_vector::UNARY_OP, operands);
@@ -952,10 +958,14 @@
;; - [INT] ABS expansion to vmslt and vneg.
;; -------------------------------------------------------------------------------
-(define_expand "abs<mode>2"
+(define_insn_and_split "abs<mode>2"
[(set (match_operand:VI 0 "register_operand")
- (match_operand:VI 1 "register_operand"))]
- "TARGET_VECTOR"
+ (abs:VI
+ (match_operand:VI 1 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
{
rtx zero = gen_const_vec_duplicate (<MODE>mode, GEN_INT (0));
machine_mode mask_mode = riscv_vector::get_mask_mode (<MODE>mode);
@@ -1457,12 +1467,15 @@
;; - vfadd.vv/vfsub.vv/...
;; - vfadd.vf/vfsub.vf/...
;; -------------------------------------------------------------------------
-(define_expand "<optab><mode>3"
- [(match_operand:VF 0 "register_operand")
- (any_float_binop:VF
- (match_operand:VF 1 "register_operand")
- (match_operand:VF 2 "register_operand"))]
- "TARGET_VECTOR"
+(define_insn_and_split "<optab><mode>3"
+ [(set (match_operand:VF 0 "register_operand")
+ (any_float_binop:VF
+ (match_operand:VF 1 "register_operand")
+ (match_operand:VF 2 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
{
riscv_vector::emit_vlmax_insn (code_for_pred (<CODE>, <MODE>mode),
riscv_vector::BINARY_OP_FRM_DYN, operands);
@@ -1474,12 +1487,15 @@
;; - vfmin.vv/vfmax.vv
;; - vfmin.vf/vfmax.vf
;; -------------------------------------------------------------------------
-(define_expand "<optab><mode>3"
- [(match_operand:VF 0 "register_operand")
- (any_float_binop_nofrm:VF
- (match_operand:VF 1 "register_operand")
- (match_operand:VF 2 "register_operand"))]
- "TARGET_VECTOR"
+(define_insn_and_split "<optab><mode>3"
+ [(set (match_operand:VF 0 "register_operand")
+ (any_float_binop_nofrm:VF
+ (match_operand:VF 1 "register_operand")
+ (match_operand:VF 2 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
{
riscv_vector::emit_vlmax_insn (code_for_pred (<CODE>, <MODE>mode),
riscv_vector::BINARY_OP, operands);
@@ -1537,22 +1553,30 @@
;; - vmulhu.vv
;; -------------------------------------------------------------------------
-(define_expand "smul<mode>3_highpart"
- [(match_operand:VFULLI 0 "register_operand")
- (match_operand:VFULLI 1 "register_operand")
- (match_operand:VFULLI 2 "register_operand")]
- "TARGET_VECTOR"
+(define_insn_and_split "smul<mode>3_highpart"
+ [(set (match_operand:VFULLI 0 "register_operand")
+ (smul_highpart:VFULLI
+ (match_operand:VFULLI 1 "register_operand")
+ (match_operand:VFULLI 2 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
{
insn_code icode = code_for_pred_mulh (UNSPEC_VMULHS, <MODE>mode);
riscv_vector::emit_vlmax_insn (icode, riscv_vector::BINARY_OP, operands);
DONE;
})
-(define_expand "umul<mode>3_highpart"
- [(match_operand:VFULLI 0 "register_operand")
- (match_operand:VFULLI 1 "register_operand")
- (match_operand:VFULLI 2 "register_operand")]
- "TARGET_VECTOR"
+(define_insn_and_split "umul<mode>3_highpart"
+ [(set (match_operand:VFULLI 0 "register_operand")
+ (umul_highpart:VFULLI
+ (match_operand:VFULLI 1 "register_operand")
+ (match_operand:VFULLI 2 "register_operand")))]
+ "TARGET_VECTOR && can_create_pseudo_p ()"
+ "#"
+ "&& 1"
+ [(const_int 0)]
{
insn_code icode = code_for_pred_mulh (UNSPEC_VMULHU, <MODE>mode);
riscv_vector::emit_vlmax_insn (icode, riscv_vector::BINARY_OP, operands);
new file mode 100644
@@ -0,0 +1,18 @@
+/* { dg-do compile } */
+/* { dg-options "-march=rv64gcv -mabi=lp64d --param=riscv-autovec-preference=scalable -Ofast -fno-schedule-insns -fno-schedule-insns2" } */
+
+#include <stdint.h>
+
+int16_t
+foo (int8_t *restrict x, int8_t *restrict y, int n)
+{
+ int16_t result = 0;
+
+ for (int i = 0; i < n; i++)
+ {
+ result += (x[i] * y[i]);
+ }
+ return result;
+}
+
+/* { dg-final { scan-assembler {\tvwmacc\.vv\tv[0-9]+,v[0-9]+,v[0-9]+} } } */
@@ -16,8 +16,11 @@
#define TEST_ALL() \
TEST_TYPE (int16_t, int8_t, uint8_t) \
TEST_TYPE (int32_t, int16_t, uint16_t) \
- TEST_TYPE (int64_t, int32_t, uint32_t)
+ TEST_TYPE (int64_t, int32_t, uint32_t) \
+ TEST_TYPE (int16_t, uint8_t, int8_t) \
+ TEST_TYPE (int32_t, uint16_t, int16_t) \
+ TEST_TYPE (int64_t, uint32_t, int32_t)
TEST_ALL ()
-/* { dg-final { scan-assembler-times {\tvwmulsu\.vv} 3 } } */
+/* { dg-final { scan-assembler-times {\tvwmulsu\.vv} 6 } } */
@@ -21,11 +21,14 @@
#define TEST_ALL() \
TEST_TYPE (int16_t, int8_t, uint8_t) \
TEST_TYPE (int32_t, int16_t, uint16_t) \
- TEST_TYPE (int64_t, int32_t, uint32_t)
+ TEST_TYPE (int64_t, int32_t, uint32_t) \
+ TEST_TYPE (int16_t, uint8_t, int8_t) \
+ TEST_TYPE (int32_t, uint16_t, int16_t) \
+ TEST_TYPE (int64_t, uint32_t, int32_t)
TEST_ALL ()
-/* { dg-final { scan-assembler-times {\tvwmulsu\.vv} 6 } } */
-/* { dg-final { scan-assembler-times {\tvwmul\.vv} 3 } } */
-/* { dg-final { scan-assembler-times {\tvwmulu\.vv} 3 } } */
+/* { dg-final { scan-assembler-times {\tvwmulsu\.vv} 12 } } */
+/* { dg-final { scan-assembler-times {\tvwmul\.vv} 6 } } */
+/* { dg-final { scan-assembler-times {\tvwmulu\.vv} 6 } } */
/* { dg-final { scan-assembler-not {\tvmul} } } */