From patchwork Mon Jul 3 09:10:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "juzhe.zhong@rivai.ai" X-Patchwork-Id: 115236 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f45:0:b0:3ea:f831:8777 with SMTP id v5csp384709vqx; Mon, 3 Jul 2023 02:11:13 -0700 (PDT) X-Google-Smtp-Source: APBJJlGCNvF/QgejAMCFfz0SRZjnPCQ5XRug0HFgbcEfbFpE1cb3cUfP+lAUp5Wo7DaVd7KX64NM X-Received: by 2002:a17:906:5f9a:b0:98a:29ca:c58e with SMTP id a26-20020a1709065f9a00b0098a29cac58emr7822711eju.27.1688375473572; Mon, 03 Jul 2023 02:11:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1688375473; cv=none; d=google.com; s=arc-20160816; b=gSCWSB/gi5vODvpCzmu+arN4CvWOgAl4h//HQk+5QcYevwHMABUXuVJ2HhsqfQu/l+ RoU4VYfekZNK5X8DHQak5BRiXKwGn/ed3kuevwhLvNyG4zh7JmkSFxRWZ12X3cb5HANJ uAoiX6UJYcBjVlr0qAfB3z4Qo9OxcaGFaEuCFbJ91J6ugCclpfVrgaE517LB8rFpiMTy G2TGWd1/sjca7/a/OfVz/VaIdCHjW74C4tz04MvOs3TU2nXbjzD1hVE7CgJcvtI2E4Zd V4PvX7jyXxIjAeq7kQFfSGpNoIbd+ckkN+vl/YDvigfYy52Pm8yME9RbUCjj9IB63quu UvJw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:feedback-id :content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:dmarc-filter:delivered-to; bh=8GpvvLm+x3TGHpEsWnXrP7vVNtnJ5kLroJEN3g3l8IE=; fh=h6UG6PIw2FD/e8JL+9n/VDDoO75bXx4B3jGj5okR2Rw=; b=KJt6Uj02H5I+ohA6f4QEC68PPG4IcqKhGQjjCbI5hIQdqgj6Z2Zk27nwr/bQfRsL5x FYu7SipFIlo24cDUWPLsqPT+k7faOftC45oYDVpXYg6WmTMfg7NhLnWTjb4/FBbfkf3b pPqONmyWnzeaCdFeA2IuE4D9o1WqaM6esApMDmgOh8gVZX2u+bGw9jHEWeiWZ1riKd4b B1KZ6+fNgSACZfEqJvztNUccMvYnrpyoSTJKGQylgwTjDflW90h65VzPmHiSPYhLOEFU 7iNgkVnTENQhSFQjszVasr4gS0N+ka+xcNAITMwRsuIAoQbWCA2g1dXBYOPfknj0CS/M VHJA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (server2.sourceware.org. [2620:52:3:1:0:246e:9693:128c]) by mx.google.com with ESMTPS id qx9-20020a170906fcc900b00988e89079fesi11093211ejb.167.2023.07.03.02.11.13 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Jul 2023 02:11:13 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) client-ip=2620:52:3:1:0:246e:9693:128c; Authentication-Results: mx.google.com; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 05F233857C71 for ; Mon, 3 Jul 2023 09:11:11 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtpbgau2.qq.com (smtpbgau2.qq.com [54.206.34.216]) by sourceware.org (Postfix) with ESMTPS id 8223B3858D28 for ; Mon, 3 Jul 2023 09:10:35 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 8223B3858D28 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivai.ai Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivai.ai X-QQ-mid: bizesmtp82t1688375429t5uaofne Received: from server1.localdomain ( [58.60.1.22]) by bizesmtp.qq.com (ESMTP) with id ; Mon, 03 Jul 2023 17:10:28 +0800 (CST) X-QQ-SSF: 01400000000000G0T000000A0000000 X-QQ-FEAT: ZkxZBO9qcf4mUzQVrJ9ZKRzgtQXZ8cW1J3/CrnmC1pLyZZEKIZq6rNe02l9bv 6hPIM5vXuWfaZvYdej76y0v/tsc7vq3NWCg4Udlv/9mx/qtzP9zXcJPISmpeeyYphjIzyKe E4nAnmsD8hhrhA4LfnSkXu2ob0xjy7QwTdvo4lrYM2wfP5bWATZNDskmAAj4QHuvi5EspYW mwQF3DCptv4FZnLTSqrOu+NrtzUMV2isEXASG6X/Wzc9EjuIiPnfeOys99/n1Z80Age9MmB jGQQ87nWJZexl99qNwpfK3Y8sAkFQ2yQ+PCLVMJWWweOJ0Dk/j7eid1LD7v4pJAqbGYRLCc Mseaa1VPinsIasICCEkbjtvDPelmiRhrbCufBlrbr8jIM/1rtNY3zkB8Plz851zshVzpyMn rlgJ0cMu2ZA= X-QQ-GoodBg: 2 X-BIZMAIL-ID: 17265230086901185706 From: juzhe.zhong@rivai.ai To: gcc-patches@gcc.gnu.org Cc: richard.sandiford@arm.com, rguenther@suse.de, Ju-Zhe Zhong Subject: [PATCH V2] Middle-end: Change order of LEN_MASK_LOAD/LEN_MASK_STORE arguments Date: Mon, 3 Jul 2023 17:10:26 +0800 Message-Id: <20230703091026.305277-1-juzhe.zhong@rivai.ai> X-Mailer: git-send-email 2.36.1 MIME-Version: 1.0 X-QQ-SENDSIZE: 520 Feedback-ID: bizesmtp:rivai.ai:qybglogicsvrgz:qybglogicsvrgz7a-one-0 X-Spam-Status: No, score=-10.3 required=5.0 tests=BAYES_00, GIT_PATCH_0, KAM_DMARC_STATUS, RCVD_IN_BARRACUDACENTRAL, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H4, RCVD_IN_MSPIKE_WL, SPF_HELO_PASS, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org Sender: "Gcc-patches" X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1770390000532209333?= X-GMAIL-MSGID: =?utf-8?q?1770390000532209333?= From: Ju-Zhe Zhong Hi, Richard. I fix the order as you suggeted. Before this patch, the order is {len,mask,bias}. Now, after this patch, the order becomes {len,bias,mask}. Since you said we should not need 'internal_fn_bias_index', the bias index should always be the len index + 1. I notice LEN_STORE order is {len,vector,bias}, to make them consistent, I reorder into LEN_STORE {len,bias,vector}. Just like MASK_STORE {mask,vector}. Ok for trunk ? gcc/ChangeLog: * config/riscv/autovec.md: Change order of LEN_MASK_LOAD/LEN_MASK_STORE/LEN_LOAD/LEN_STORE arguments. * config/riscv/riscv-v.cc (expand_load_store): Ditto. * doc/md.texi: Ditto. * gimple-fold.cc (gimple_fold_partial_load_store_mem_ref): Ditto. * internal-fn.cc (len_maskload_direct): Ditto. (len_maskstore_direct): Ditto. (add_len_and_mask_args): New function. (expand_partial_load_optab_fn): Change order of LEN_MASK_LOAD/LEN_MASK_STORE/LEN_LOAD/LEN_STORE arguments. (expand_partial_store_optab_fn): Ditto. (internal_fn_len_index): New function. (internal_fn_mask_index): Change order of LEN_MASK_LOAD/LEN_MASK_STORE/LEN_LOAD/LEN_STORE arguments. (internal_fn_stored_value_index): Ditto. (internal_len_load_store_bias): Ditto. * internal-fn.h (internal_fn_len_index): New function. * tree-ssa-dse.cc (initialize_ao_ref_for_dse): Change order of LEN_MASK_LOAD/LEN_MASK_STORE/LEN_LOAD/LEN_STORE arguments. * tree-vect-stmts.cc (vectorizable_store): Ditto. (vectorizable_load): Ditto. --- gcc/config/riscv/autovec.md | 8 +- gcc/config/riscv/riscv-v.cc | 2 +- gcc/doc/md.texi | 16 ++-- gcc/gimple-fold.cc | 8 +- gcc/internal-fn.cc | 156 ++++++++++++++++++------------------ gcc/internal-fn.h | 1 + gcc/tree-ssa-dse.cc | 11 +-- gcc/tree-vect-stmts.cc | 11 +-- 8 files changed, 107 insertions(+), 106 deletions(-) diff --git a/gcc/config/riscv/autovec.md b/gcc/config/riscv/autovec.md index 1488f2be1be..4ab0e9f99eb 100644 --- a/gcc/config/riscv/autovec.md +++ b/gcc/config/riscv/autovec.md @@ -26,8 +26,8 @@ [(match_operand:V 0 "register_operand") (match_operand:V 1 "memory_operand") (match_operand 2 "autovec_length_operand") - (match_operand: 3 "vector_mask_operand") - (match_operand 4 "const_0_operand")] + (match_operand 3 "const_0_operand") + (match_operand: 4 "vector_mask_operand")] "TARGET_VECTOR" { riscv_vector::expand_load_store (operands, true); @@ -38,8 +38,8 @@ [(match_operand:V 0 "memory_operand") (match_operand:V 1 "register_operand") (match_operand 2 "autovec_length_operand") - (match_operand: 3 "vector_mask_operand") - (match_operand 4 "const_0_operand")] + (match_operand 3 "const_0_operand") + (match_operand: 4 "vector_mask_operand")] "TARGET_VECTOR" { riscv_vector::expand_load_store (operands, false); diff --git a/gcc/config/riscv/riscv-v.cc b/gcc/config/riscv/riscv-v.cc index adb8d7d36a5..8d5bed7ebe4 100644 --- a/gcc/config/riscv/riscv-v.cc +++ b/gcc/config/riscv/riscv-v.cc @@ -2777,7 +2777,7 @@ expand_load_store (rtx *ops, bool is_load) { poly_int64 value; rtx len = ops[2]; - rtx mask = ops[3]; + rtx mask = ops[4]; machine_mode mode = GET_MODE (ops[0]); if (poly_int_rtx_p (len, &value) && known_eq (value, GET_MODE_NUNITS (mode))) diff --git a/gcc/doc/md.texi b/gcc/doc/md.texi index cefdee84821..5e5482265cd 100644 --- a/gcc/doc/md.texi +++ b/gcc/doc/md.texi @@ -5302,15 +5302,15 @@ This pattern is not allowed to @code{FAIL}. @cindex @code{len_maskload@var{m}@var{n}} instruction pattern @item @samp{len_maskload@var{m}@var{n}} Perform a masked load from the memory location pointed to by operand 1 -into register operand 0. (operand 2 + operand 4) elements are loaded from +into register operand 0. (operand 2 + operand 3) elements are loaded from memory and other elements in operand 0 are set to undefined values. This is a combination of len_load and maskload. Operands 0 and 1 have mode @var{m}, which must be a vector mode. Operand 2 has whichever integer mode the target prefers. A mask is specified in -operand 3 which must be of type @var{n}. The mask has lower precedence than +operand 4 which must be of type @var{n}. The mask has lower precedence than the length and is itself subject to length masking, -i.e. only mask indices < (operand 2 + operand 4) are used. -Operand 4 conceptually has mode @code{QI}. +i.e. only mask indices < (operand 2 + operand 3) are used. +Operand 3 conceptually has mode @code{QI}. Operand 2 can be a variable or a constant amount. Operand 4 specifies a constant bias: it is either a constant 0 or a constant -1. The predicate on @@ -5329,14 +5329,14 @@ This pattern is not allowed to @code{FAIL}. @cindex @code{len_maskstore@var{m}@var{n}} instruction pattern @item @samp{len_maskstore@var{m}@var{n}} Perform a masked store from vector register operand 1 into memory operand 0. -(operand 2 + operand 4) elements are stored to memory +(operand 2 + operand 3) elements are stored to memory and leave the other elements of operand 0 unchanged. This is a combination of len_store and maskstore. Operands 0 and 1 have mode @var{m}, which must be a vector mode. Operand 2 has whichever -integer mode the target prefers. A mask is specified in operand 3 which must be +integer mode the target prefers. A mask is specified in operand 4 which must be of type @var{n}. The mask has lower precedence than the length and is itself subject to -length masking, i.e. only mask indices < (operand 2 + operand 4) are used. -Operand 4 conceptually has mode @code{QI}. +length masking, i.e. only mask indices < (operand 2 + operand 3) are used. +Operand 3 conceptually has mode @code{QI}. Operand 2 can be a variable or a constant amount. Operand 3 specifies a constant bias: it is either a constant 0 or a constant -1. The predicate on diff --git a/gcc/gimple-fold.cc b/gcc/gimple-fold.cc index 8434274f69d..4027ff71e10 100644 --- a/gcc/gimple-fold.cc +++ b/gcc/gimple-fold.cc @@ -5391,11 +5391,12 @@ gimple_fold_partial_load_store_mem_ref (gcall *call, tree vectype, bool mask_p) } else { - tree basic_len = gimple_call_arg (call, 2); + internal_fn ifn = gimple_call_internal_fn (call); + int len_index = internal_fn_len_index (ifn); + tree basic_len = gimple_call_arg (call, len_index); if (!poly_int_tree_p (basic_len)) return NULL_TREE; - unsigned int nargs = gimple_call_num_args (call); - tree bias = gimple_call_arg (call, nargs - 1); + tree bias = gimple_call_arg (call, len_index + 1); gcc_assert (TREE_CODE (bias) == INTEGER_CST); /* For LEN_LOAD/LEN_STORE/LEN_MASK_LOAD/LEN_MASK_STORE, we don't fold when (bias + len) != VF. */ @@ -5405,7 +5406,6 @@ gimple_fold_partial_load_store_mem_ref (gcall *call, tree vectype, bool mask_p) /* For LEN_MASK_{LOAD,STORE}, we should also check whether the mask is all ones mask. */ - internal_fn ifn = gimple_call_internal_fn (call); if (ifn == IFN_LEN_MASK_LOAD || ifn == IFN_LEN_MASK_STORE) { tree mask = gimple_call_arg (call, internal_fn_mask_index (ifn)); diff --git a/gcc/internal-fn.cc b/gcc/internal-fn.cc index 9017176dc7a..c1fcb38b17b 100644 --- a/gcc/internal-fn.cc +++ b/gcc/internal-fn.cc @@ -165,7 +165,7 @@ init_internal_fns () #define mask_load_lanes_direct { -1, -1, false } #define gather_load_direct { 3, 1, false } #define len_load_direct { -1, -1, false } -#define len_maskload_direct { -1, 3, false } +#define len_maskload_direct { -1, 4, false } #define mask_store_direct { 3, 2, false } #define store_lanes_direct { 0, 0, false } #define mask_store_lanes_direct { 0, 0, false } @@ -173,7 +173,7 @@ init_internal_fns () #define vec_cond_direct { 2, 0, false } #define scatter_store_direct { 3, 1, false } #define len_store_direct { 3, 3, false } -#define len_maskstore_direct { 4, 3, false } +#define len_maskstore_direct { 4, 5, false } #define vec_set_direct { 3, 3, false } #define unary_direct { 0, 0, true } #define unary_convert_direct { -1, 0, true } @@ -293,6 +293,38 @@ get_multi_vector_move (tree array_type, convert_optab optab) return convert_optab_handler (optab, imode, vmode); } +/* Add len and mask arguments according to the STMT. */ + +static unsigned int +add_len_and_mask_args (expand_operand *ops, unsigned int opno, gcall *stmt) +{ + internal_fn ifn = gimple_call_internal_fn (stmt); + int len_index = internal_fn_len_index (ifn); + /* BIAS is always consecutive next of LEN. */ + int bias_index = len_index + 1; + int mask_index = internal_fn_mask_index (ifn); + /* The order of arguments are always {len,bias,mask}. */ + if (len_index >= 0) + { + tree len = gimple_call_arg (stmt, len_index); + rtx len_rtx = expand_normal (len); + create_convert_operand_from (&ops[opno++], len_rtx, + TYPE_MODE (TREE_TYPE (len)), + TYPE_UNSIGNED (TREE_TYPE (len))); + tree biast = gimple_call_arg (stmt, bias_index); + rtx bias = expand_normal (biast); + create_input_operand (&ops[opno++], bias, QImode); + } + if (mask_index >= 0) + { + tree mask = gimple_call_arg (stmt, mask_index); + rtx mask_rtx = expand_normal (mask); + create_input_operand (&ops[opno++], mask_rtx, + TYPE_MODE (TREE_TYPE (mask))); + } + return opno; +} + /* Expand LOAD_LANES call STMT using optab OPTAB. */ static void @@ -2879,14 +2911,15 @@ expand_call_mem_ref (tree type, gcall *stmt, int index) * OPTAB. */ static void -expand_partial_load_optab_fn (internal_fn, gcall *stmt, convert_optab optab) +expand_partial_load_optab_fn (internal_fn ifn, gcall *stmt, convert_optab optab) { + int i = 0; class expand_operand ops[5]; - tree type, lhs, rhs, maskt, biast; - rtx mem, target, mask, bias; + tree type, lhs, rhs, maskt; + rtx mem, target; insn_code icode; - maskt = gimple_call_arg (stmt, 2); + maskt = gimple_call_arg (stmt, internal_fn_mask_index (ifn)); lhs = gimple_call_lhs (stmt); if (lhs == NULL_TREE) return; @@ -2903,38 +2936,11 @@ expand_partial_load_optab_fn (internal_fn, gcall *stmt, convert_optab optab) mem = expand_expr (rhs, NULL_RTX, VOIDmode, EXPAND_WRITE); gcc_assert (MEM_P (mem)); - mask = expand_normal (maskt); target = expand_expr (lhs, NULL_RTX, VOIDmode, EXPAND_WRITE); - create_output_operand (&ops[0], target, TYPE_MODE (type)); - create_fixed_operand (&ops[1], mem); - if (optab == len_load_optab) - { - create_convert_operand_from (&ops[2], mask, TYPE_MODE (TREE_TYPE (maskt)), - TYPE_UNSIGNED (TREE_TYPE (maskt))); - biast = gimple_call_arg (stmt, 3); - bias = expand_normal (biast); - create_input_operand (&ops[3], bias, QImode); - expand_insn (icode, 4, ops); - } - else if (optab == len_maskload_optab) - { - create_convert_operand_from (&ops[2], mask, TYPE_MODE (TREE_TYPE (maskt)), - TYPE_UNSIGNED (TREE_TYPE (maskt))); - maskt = gimple_call_arg (stmt, 3); - mask = expand_normal (maskt); - create_input_operand (&ops[3], mask, TYPE_MODE (TREE_TYPE (maskt))); - icode = convert_optab_handler (optab, TYPE_MODE (type), - TYPE_MODE (TREE_TYPE (maskt))); - biast = gimple_call_arg (stmt, 4); - bias = expand_normal (biast); - create_input_operand (&ops[4], bias, QImode); - expand_insn (icode, 5, ops); - } - else - { - create_input_operand (&ops[2], mask, TYPE_MODE (TREE_TYPE (maskt))); - expand_insn (icode, 3, ops); - } + create_output_operand (&ops[i++], target, TYPE_MODE (type)); + create_fixed_operand (&ops[i++], mem); + i = add_len_and_mask_args (ops, i, stmt); + expand_insn (icode, i, ops); if (!rtx_equal_p (target, ops[0].value)) emit_move_insn (target, ops[0].value); @@ -2951,12 +2957,13 @@ expand_partial_load_optab_fn (internal_fn, gcall *stmt, convert_optab optab) static void expand_partial_store_optab_fn (internal_fn ifn, gcall *stmt, convert_optab optab) { + int i = 0; class expand_operand ops[5]; - tree type, lhs, rhs, maskt, biast; - rtx mem, reg, mask, bias; + tree type, lhs, rhs, maskt; + rtx mem, reg; insn_code icode; - maskt = gimple_call_arg (stmt, 2); + maskt = gimple_call_arg (stmt, internal_fn_mask_index (ifn)); rhs = gimple_call_arg (stmt, internal_fn_stored_value_index (ifn)); type = TREE_TYPE (rhs); lhs = expand_call_mem_ref (type, stmt, 0); @@ -2971,37 +2978,11 @@ expand_partial_store_optab_fn (internal_fn ifn, gcall *stmt, convert_optab optab mem = expand_expr (lhs, NULL_RTX, VOIDmode, EXPAND_WRITE); gcc_assert (MEM_P (mem)); - mask = expand_normal (maskt); reg = expand_normal (rhs); - create_fixed_operand (&ops[0], mem); - create_input_operand (&ops[1], reg, TYPE_MODE (type)); - if (optab == len_store_optab) - { - create_convert_operand_from (&ops[2], mask, TYPE_MODE (TREE_TYPE (maskt)), - TYPE_UNSIGNED (TREE_TYPE (maskt))); - biast = gimple_call_arg (stmt, 4); - bias = expand_normal (biast); - create_input_operand (&ops[3], bias, QImode); - expand_insn (icode, 4, ops); - } - else if (optab == len_maskstore_optab) - { - create_convert_operand_from (&ops[2], mask, TYPE_MODE (TREE_TYPE (maskt)), - TYPE_UNSIGNED (TREE_TYPE (maskt))); - maskt = gimple_call_arg (stmt, 3); - mask = expand_normal (maskt); - create_input_operand (&ops[3], mask, TYPE_MODE (TREE_TYPE (maskt))); - biast = gimple_call_arg (stmt, 5); - bias = expand_normal (biast); - create_input_operand (&ops[4], bias, QImode); - icode = convert_optab_handler (optab, TYPE_MODE (type), GET_MODE (mask)); - expand_insn (icode, 5, ops); - } - else - { - create_input_operand (&ops[2], mask, TYPE_MODE (TREE_TYPE (maskt))); - expand_insn (icode, 3, ops); - } + create_fixed_operand (&ops[i++], mem); + create_input_operand (&ops[i++], reg, TYPE_MODE (type)); + i = add_len_and_mask_args (ops, i, stmt); + expand_insn (icode, i, ops); } #define expand_mask_store_optab_fn expand_partial_store_optab_fn @@ -4482,6 +4463,25 @@ internal_gather_scatter_fn_p (internal_fn fn) } } +/* If FN takes a vector len argument, return the index of that argument, + otherwise return -1. */ + +int +internal_fn_len_index (internal_fn fn) +{ + switch (fn) + { + case IFN_LEN_LOAD: + case IFN_LEN_STORE: + case IFN_LEN_MASK_LOAD: + case IFN_LEN_MASK_STORE: + return 2; + + default: + return -1; + } +} + /* If FN takes a vector mask argument, return the index of that argument, otherwise return -1. */ @@ -4498,11 +4498,9 @@ internal_fn_mask_index (internal_fn fn) case IFN_MASK_GATHER_LOAD: case IFN_MASK_SCATTER_STORE: - return 4; - case IFN_LEN_MASK_LOAD: case IFN_LEN_MASK_STORE: - return 3; + return 4; default: return (conditional_internal_fn_code (fn) != ERROR_MARK @@ -4522,12 +4520,14 @@ internal_fn_stored_value_index (internal_fn fn) case IFN_MASK_STORE_LANES: case IFN_SCATTER_STORE: case IFN_MASK_SCATTER_STORE: - case IFN_LEN_STORE: return 3; - case IFN_LEN_MASK_STORE: + case IFN_LEN_STORE: return 4; + case IFN_LEN_MASK_STORE: + return 5; + default: return -1; } @@ -4592,7 +4592,6 @@ internal_len_load_store_bias (internal_fn ifn, machine_mode mode) { optab optab = direct_internal_fn_optab (ifn); insn_code icode = direct_optab_handler (optab, mode); - int bias_opno = 3; if (icode == CODE_FOR_nothing) { @@ -4610,15 +4609,14 @@ internal_len_load_store_bias (internal_fn ifn, machine_mode mode) optab = direct_internal_fn_optab (IFN_LEN_MASK_STORE); } icode = convert_optab_handler (optab, mode, mask_mode); - bias_opno = 4; } if (icode != CODE_FOR_nothing) { /* For now we only support biases of 0 or -1. Try both of them. */ - if (insn_operand_matches (icode, bias_opno, GEN_INT (0))) + if (insn_operand_matches (icode, 3, GEN_INT (0))) return 0; - if (insn_operand_matches (icode, bias_opno, GEN_INT (-1))) + if (insn_operand_matches (icode, 3, GEN_INT (-1))) return -1; } diff --git a/gcc/internal-fn.h b/gcc/internal-fn.h index 8f21068e300..4234bbfed87 100644 --- a/gcc/internal-fn.h +++ b/gcc/internal-fn.h @@ -234,6 +234,7 @@ extern bool internal_load_fn_p (internal_fn); extern bool internal_store_fn_p (internal_fn); extern bool internal_gather_scatter_fn_p (internal_fn); extern int internal_fn_mask_index (internal_fn); +extern int internal_fn_len_index (internal_fn); extern int internal_fn_stored_value_index (internal_fn); extern bool internal_gather_scatter_fn_supported_p (internal_fn, tree, tree, tree, int); diff --git a/gcc/tree-ssa-dse.cc b/gcc/tree-ssa-dse.cc index f8338037a61..9c6004cdce8 100644 --- a/gcc/tree-ssa-dse.cc +++ b/gcc/tree-ssa-dse.cc @@ -161,12 +161,13 @@ initialize_ao_ref_for_dse (gimple *stmt, ao_ref *write, bool may_def_ok = false) case IFN_MASK_STORE: case IFN_LEN_MASK_STORE: { - int stored_value_index - = internal_fn_stored_value_index (gimple_call_internal_fn (stmt)); - if (gimple_call_internal_fn (stmt) == IFN_LEN_STORE) + internal_fn ifn = gimple_call_internal_fn (stmt); + int stored_value_index = internal_fn_stored_value_index (ifn); + int len_index = internal_fn_len_index (ifn); + if (ifn == IFN_LEN_STORE) { - tree len = gimple_call_arg (stmt, 2); - tree bias = gimple_call_arg (stmt, 4); + tree len = gimple_call_arg (stmt, len_index); + tree bias = gimple_call_arg (stmt, len_index + 1); if (tree_fits_uhwi_p (len)) { ao_ref_init_from_ptr_and_size (write, diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc index 68faa8ead39..a0c39268bf0 100644 --- a/gcc/tree-vect-stmts.cc +++ b/gcc/tree-vect-stmts.cc @@ -9122,13 +9122,14 @@ vectorizable_store (vec_info *vinfo, if (partial_ifn == IFN_LEN_MASK_STORE) call = gimple_build_call_internal (IFN_LEN_MASK_STORE, 6, dataref_ptr, ptr, - final_len, final_mask, - vec_oprnd, bias); + final_len, bias, + final_mask, vec_oprnd); else call = gimple_build_call_internal (IFN_LEN_STORE, 5, - dataref_ptr, ptr, final_len, - vec_oprnd, bias); + dataref_ptr, ptr, + final_len, bias, + vec_oprnd); gimple_call_set_nothrow (call, true); vect_finish_stmt_generation (vinfo, stmt_info, call, gsi); new_stmt = call; @@ -10523,7 +10524,7 @@ vectorizable_load (vec_info *vinfo, call = gimple_build_call_internal (IFN_LEN_MASK_LOAD, 5, dataref_ptr, ptr, final_len, - final_mask, bias); + bias, final_mask); else call = gimple_build_call_internal (IFN_LEN_LOAD, 4, dataref_ptr, ptr,