From patchwork Mon Jul 3 02:15:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "juzhe.zhong@rivai.ai" X-Patchwork-Id: 115166 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f45:0:b0:3ea:f831:8777 with SMTP id v5csp236926vqx; Sun, 2 Jul 2023 19:15:59 -0700 (PDT) X-Google-Smtp-Source: APBJJlHNHh0aScw3dtI3qccnnnK4U5faSihjKGqNtJUmzzqTulQFOaPInHbaQbsB2GJz7LRmWqjM X-Received: by 2002:a17:907:7679:b0:982:8c28:c50d with SMTP id kk25-20020a170907767900b009828c28c50dmr4675090ejc.58.1688350559743; Sun, 02 Jul 2023 19:15:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1688350559; cv=none; d=google.com; s=arc-20160816; b=Nc1D2ANFWaG3GUpwhK4VyAjhHzpYNJUohh3uRhlkccjGYmGuGDv9qZMXO3ZExpaUor 2P67NvLkHlwrIvW2B2e3jOHJ/CN+ZNAU+13nOHaPZkTS1qKmwRWToC+GPbDFrFa5vswl mjUG1apWZIUikwt62oK+UzBCLrKJpGOZJCyjsBR8v2tru/RH0JKy8f/Rj/XpkVx5chAr NLV9XCuHiex3YVFTiThxhfMRq/sqnw0nuK8JKS8z39Bgn1WAa50Py6dM0kKESo3HtcKM RHPhzgSD1xhIXNTIA044VxH1KCy395VEMK81yk3as85oSn4j/uS+MbX+xOhloikFeX2O 1IUA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:feedback-id :content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:dmarc-filter:delivered-to; bh=4fCVqF1xFBSI8umYUoLY1fnn2tr7j2/fTRrMT6GSGYs=; fh=LxgSMRQgUe3V4LqA1E0dccbBxNhkicg+4forc5/KnEY=; b=wsgpvVjl1cvKjCG6vDNL51+tuNT6J9o222x9ikSwt0RkJCJb0TigrSUv2QVRze7jQf MbsL83if+YXxdhvP5DZE9+Vsj5rwZaCOSsako9Gz9TXGe7SLuA+yFLPGvj/8Zi0wMae0 CQhbmHlO6584KWhmiMZN65lf+Bb3GDANda3kb7HZYS7O38qi5xtLGpa2t9ojiqCKjX8V ZRSxwLda0GtOYvRweGhTv91VM/E9gei3J2LmDI7RdHILop5doeV9KXCdStoKZ+qBeRfg DbvL/T/lV2K+uMu6/dE0Kpcg7tUUyjPguw1Rcbie7BXzWnhA3vdCqVn7sJnqeZyBJIRi VWUQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (server2.sourceware.org. [2620:52:3:1:0:246e:9693:128c]) by mx.google.com with ESMTPS id g15-20020a170906348f00b00992d6fa3b33si3691595ejb.440.2023.07.02.19.15.59 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 02 Jul 2023 19:15:59 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) client-ip=2620:52:3:1:0:246e:9693:128c; Authentication-Results: mx.google.com; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id C1DA63857721 for ; Mon, 3 Jul 2023 02:15:47 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtpbgau1.qq.com (smtpbgau1.qq.com [54.206.16.166]) by sourceware.org (Postfix) with ESMTPS id 48E183858D28 for ; Mon, 3 Jul 2023 02:15:17 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 48E183858D28 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivai.ai Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivai.ai X-QQ-mid: bizesmtp86t1688350503txy92hbc Received: from server1.localdomain ( [58.60.1.22]) by bizesmtp.qq.com (ESMTP) with id ; Mon, 03 Jul 2023 10:15:01 +0800 (CST) X-QQ-SSF: 01400000000000G0T000000A0000000 X-QQ-FEAT: k0mQ4ihyJQNrQ7pFPSe83YnouElXchchEKAcaRuNIfR/mCqRuu2Uvwlrvw1FI 5s7xWOOfI3uUcQXb+RMNrTKbnx2DYHgutY5Gnc7eznKu1Z+8dOuBjro5Tx175vbCnznXM7P OxB3IUS+bEcp5/DBv7J8WGo+rI9yvSMTbbyAsci4OIP27Vpxuc+BwE+XS6S2Uy5bBELcwY/ 1eEB1FDWvXEd8m1mt6dBAE4nxA4Vaau9m/qAx86JUBxeBPmtPgFnduQS5awpSdlUs861woG jq72jtzoCmMXWK0g2PVZzxagJyiqmVPNFLnj8Hk3tQWHxLeNER9DcjZwJkf1+k8POoiyMdy OKRjlrdklo08kFNZ/LOOpjgpX/AkIag3+tUY5gXHu4rTkFZLTE= X-QQ-GoodBg: 2 X-BIZMAIL-ID: 12071828761280748610 From: juzhe.zhong@rivai.ai To: gcc-patches@gcc.gnu.org Cc: richard.sandiford@arm.com, rguenther@suse.de, pan2.li@intel.com, Ju-Zhe Zhong Subject: [PATCH V6] Machine Description: Add LEN_MASK_{GATHER_LOAD, SCATTER_STORE} pattern Date: Mon, 3 Jul 2023 10:15:00 +0800 Message-Id: <20230703021500.165265-1-juzhe.zhong@rivai.ai> X-Mailer: git-send-email 2.36.1 MIME-Version: 1.0 X-QQ-SENDSIZE: 520 Feedback-ID: bizesmtp:rivai.ai:qybglogicsvrgz:qybglogicsvrgz7a-one-0 X-Spam-Status: No, score=-10.8 required=5.0 tests=BAYES_00, GIT_PATCH_0, KAM_DMARC_STATUS, KAM_SHORT, RCVD_IN_BARRACUDACENTRAL, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H5, RCVD_IN_MSPIKE_WL, SPF_HELO_PASS, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org Sender: "Gcc-patches" X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1770363876260769443?= X-GMAIL-MSGID: =?utf-8?q?1770363876260769443?= From: Ju-Zhe Zhong Hi, Richi and Richard. Base one the review comments from Richard: https://gcc.gnu.org/pipermail/gcc-patches/2023-July/623405.html I add the helper function: /* Add len, bias and mask arguments according to the STMT. */ static unsigned int add_len_bias_and_mask_args (expand_operand *ops, unsigned int opno, gcall *stmt) { internal_fn ifn = gimple_call_internal_fn (stmt); int len_index = internal_fn_len_index (ifn); int bias_index = internal_fn_bias_index (ifn); int mask_index = internal_fn_mask_index (ifn); /* The order of arguments are always {len,bias,mask}. */ if (len_index >= 0) { tree len = gimple_call_arg (stmt, len_index); rtx len_rtx = expand_normal (len); create_convert_operand_from (&ops[opno++], len_rtx, TYPE_MODE (TREE_TYPE (len)), TYPE_UNSIGNED (TREE_TYPE (len))); } if (bias_index >= 0) { tree biast = gimple_call_arg (stmt, bias_index); rtx bias = expand_normal (biast); create_input_operand (&ops[opno++], bias, QImode); } if (mask_index >= 0) { tree mask = gimple_call_arg (stmt, mask_index); rtx mask_rtx = expand_normal (mask); create_input_operand (&ops[opno++], mask_rtx, TYPE_MODE (TREE_TYPE (mask))); } return opno; } I think current codes look more reasonable and easier maitain now. This patch is adding LEN_MASK_{GATHER_LOAD,SCATTER_STORE} to allow targets handle flow control by mask and loop control by length on gather/scatter memory operations. Consider this following case: #include void f (uint8_t *restrict a, uint8_t *restrict b, int n, int base, int step, int *restrict cond) { for (int i = 0; i < n; ++i) { if (cond[i]) a[i * step + base] = b[i * step + base]; } } We hope RVV can vectorize such case into following IR: loop_len = SELECT_VL control_mask = comparison v = LEN_MASK_GATHER_LOAD (.., loop_len, control_mask, bias) LEN_SCATTER_STORE (... v, ..., loop_len, control_mask, bias) This patch doesn't apply such patterns into vectorizer, just add patterns and update the documents. Will send patch which apply such patterns into vectorizer soon after this patch is approved. Ok for trunk? gcc/ChangeLog: * doc/md.texi: Add len_mask_gather_load/len_mask_scatter_store. * internal-fn.cc (add_len_bias_and_mask_args): New function. (expand_scatter_store_optab_fn): Add len_mask_gather_load/len_mask_scatter_store. (expand_gather_load_optab_fn): Ditto. (internal_load_fn_p): Ditto. (internal_store_fn_p): Ditto. (internal_gather_scatter_fn_p): Ditto. (internal_fn_mask_index): Ditto. (internal_fn_len_index): New function. (internal_fn_bias_index): New function. (internal_fn_stored_value_index): Add len_mask_gather_load/len_mask_scatter_store. * internal-fn.def (LEN_MASK_GATHER_LOAD): Ditto. (LEN_MASK_SCATTER_STORE): Ditto. * internal-fn.h (internal_fn_len_index): New function. (internal_fn_bias_index): Ditto. * optabs.def (OPTAB_CD): Add len_mask_gather_load/len_mask_scatter_store. --- gcc/doc/md.texi | 17 +++++++ gcc/internal-fn.cc | 108 ++++++++++++++++++++++++++++++++++++++------ gcc/internal-fn.def | 8 +++- gcc/internal-fn.h | 2 + gcc/optabs.def | 2 + 5 files changed, 120 insertions(+), 17 deletions(-) diff --git a/gcc/doc/md.texi b/gcc/doc/md.texi index cefdee84821..b44d1ba3af9 100644 --- a/gcc/doc/md.texi +++ b/gcc/doc/md.texi @@ -5040,6 +5040,15 @@ operand 5. Bit @var{i} of the mask is set if element @var{i} of the result should be loaded from memory and clear if element @var{i} of the result should be set to zero. +@cindex @code{len_mask_gather_load@var{m}@var{n}} instruction pattern +@item @samp{len_mask_gather_load@var{m}@var{n}} +Like @samp{gather_load@var{m}@var{n}}, but takes an extra length operand (operand 5), +a bias operand (operand 6) as well as a mask operand (operand 7). Similar to len_maskload, +the instruction loads at most (operand 5 + operand 6) elements from memory. +Bit @var{i} of the mask is set if element @var{i} of the result should +be loaded from memory and clear if element @var{i} of the result should be undefined. +Mask elements @var{i} with @var{i} > (operand 5 + operand 6) are ignored. + @cindex @code{scatter_store@var{m}@var{n}} instruction pattern @item @samp{scatter_store@var{m}@var{n}} Store a vector of mode @var{m} into several distinct memory locations. @@ -5069,6 +5078,14 @@ Like @samp{scatter_store@var{m}@var{n}}, but takes an extra mask operand as operand 5. Bit @var{i} of the mask is set if element @var{i} of the result should be stored to memory. +@cindex @code{len_mask_scatter_store@var{m}@var{n}} instruction pattern +@item @samp{len_mask_scatter_store@var{m}@var{n}} +Like @samp{scatter_store@var{m}@var{n}}, but takes an extra length operand (operand 5), +a bias operand (operand 6) as well as a mask operand (operand 7). The instruction stores +at most (operand 5 + operand 6) elements of (operand 4) to memory. +Bit @var{i} of the mask is set if element @var{i} of (operand 4) should be stored. +Mask elements @var{i} with @var{i} > (operand 5 + operand 6) are ignored. + @cindex @code{vec_set@var{m}} instruction pattern @item @samp{vec_set@var{m}} Set given field in the vector value. Operand 0 is the vector to modify, diff --git a/gcc/internal-fn.cc b/gcc/internal-fn.cc index 9017176dc7a..43ac5b37cc9 100644 --- a/gcc/internal-fn.cc +++ b/gcc/internal-fn.cc @@ -3519,6 +3519,40 @@ expand_LAUNDER (internal_fn, gcall *call) expand_assignment (lhs, gimple_call_arg (call, 0), false); } +/* Add len, bias and mask arguments according to the STMT. */ + +static unsigned int +add_len_bias_and_mask_args (expand_operand *ops, unsigned int opno, gcall *stmt) +{ + internal_fn ifn = gimple_call_internal_fn (stmt); + int len_index = internal_fn_len_index (ifn); + int bias_index = internal_fn_bias_index (ifn); + int mask_index = internal_fn_mask_index (ifn); + /* The order of arguments are always {len,bias,mask}. */ + if (len_index >= 0) + { + tree len = gimple_call_arg (stmt, len_index); + rtx len_rtx = expand_normal (len); + create_convert_operand_from (&ops[opno++], len_rtx, + TYPE_MODE (TREE_TYPE (len)), + TYPE_UNSIGNED (TREE_TYPE (len))); + } + if (bias_index >= 0) + { + tree biast = gimple_call_arg (stmt, bias_index); + rtx bias = expand_normal (biast); + create_input_operand (&ops[opno++], bias, QImode); + } + if (mask_index >= 0) + { + tree mask = gimple_call_arg (stmt, mask_index); + rtx mask_rtx = expand_normal (mask); + create_input_operand (&ops[opno++], mask_rtx, + TYPE_MODE (TREE_TYPE (mask))); + } + return opno; +} + /* Expand {MASK_,}SCATTER_STORE{S,U} call CALL using optab OPTAB. */ static void @@ -3526,7 +3560,6 @@ expand_scatter_store_optab_fn (internal_fn, gcall *stmt, direct_optab optab) { internal_fn ifn = gimple_call_internal_fn (stmt); int rhs_index = internal_fn_stored_value_index (ifn); - int mask_index = internal_fn_mask_index (ifn); tree base = gimple_call_arg (stmt, 0); tree offset = gimple_call_arg (stmt, 1); tree scale = gimple_call_arg (stmt, 2); @@ -3537,19 +3570,14 @@ expand_scatter_store_optab_fn (internal_fn, gcall *stmt, direct_optab optab) HOST_WIDE_INT scale_int = tree_to_shwi (scale); rtx rhs_rtx = expand_normal (rhs); - class expand_operand ops[6]; + class expand_operand ops[8]; int i = 0; create_address_operand (&ops[i++], base_rtx); create_input_operand (&ops[i++], offset_rtx, TYPE_MODE (TREE_TYPE (offset))); create_integer_operand (&ops[i++], TYPE_UNSIGNED (TREE_TYPE (offset))); create_integer_operand (&ops[i++], scale_int); create_input_operand (&ops[i++], rhs_rtx, TYPE_MODE (TREE_TYPE (rhs))); - if (mask_index >= 0) - { - tree mask = gimple_call_arg (stmt, mask_index); - rtx mask_rtx = expand_normal (mask); - create_input_operand (&ops[i++], mask_rtx, TYPE_MODE (TREE_TYPE (mask))); - } + i = add_len_bias_and_mask_args (ops, i, stmt); insn_code icode = convert_optab_handler (optab, TYPE_MODE (TREE_TYPE (rhs)), TYPE_MODE (TREE_TYPE (offset))); @@ -3572,18 +3600,13 @@ expand_gather_load_optab_fn (internal_fn, gcall *stmt, direct_optab optab) HOST_WIDE_INT scale_int = tree_to_shwi (scale); int i = 0; - class expand_operand ops[6]; + class expand_operand ops[8]; create_output_operand (&ops[i++], lhs_rtx, TYPE_MODE (TREE_TYPE (lhs))); create_address_operand (&ops[i++], base_rtx); create_input_operand (&ops[i++], offset_rtx, TYPE_MODE (TREE_TYPE (offset))); create_integer_operand (&ops[i++], TYPE_UNSIGNED (TREE_TYPE (offset))); create_integer_operand (&ops[i++], scale_int); - if (optab == mask_gather_load_optab) - { - tree mask = gimple_call_arg (stmt, 4); - rtx mask_rtx = expand_normal (mask); - create_input_operand (&ops[i++], mask_rtx, TYPE_MODE (TREE_TYPE (mask))); - } + i = add_len_bias_and_mask_args (ops, i, stmt); insn_code icode = convert_optab_handler (optab, TYPE_MODE (TREE_TYPE (lhs)), TYPE_MODE (TREE_TYPE (offset))); expand_insn (icode, i, ops); @@ -4434,6 +4457,7 @@ internal_load_fn_p (internal_fn fn) case IFN_MASK_LOAD_LANES: case IFN_GATHER_LOAD: case IFN_MASK_GATHER_LOAD: + case IFN_LEN_MASK_GATHER_LOAD: case IFN_LEN_LOAD: case IFN_LEN_MASK_LOAD: return true; @@ -4455,6 +4479,7 @@ internal_store_fn_p (internal_fn fn) case IFN_MASK_STORE_LANES: case IFN_SCATTER_STORE: case IFN_MASK_SCATTER_STORE: + case IFN_LEN_MASK_SCATTER_STORE: case IFN_LEN_STORE: case IFN_LEN_MASK_STORE: return true; @@ -4473,8 +4498,10 @@ internal_gather_scatter_fn_p (internal_fn fn) { case IFN_GATHER_LOAD: case IFN_MASK_GATHER_LOAD: + case IFN_LEN_MASK_GATHER_LOAD: case IFN_SCATTER_STORE: case IFN_MASK_SCATTER_STORE: + case IFN_LEN_MASK_SCATTER_STORE: return true; default: @@ -4500,6 +4527,10 @@ internal_fn_mask_index (internal_fn fn) case IFN_MASK_SCATTER_STORE: return 4; + case IFN_LEN_MASK_GATHER_LOAD: + case IFN_LEN_MASK_SCATTER_STORE: + return 6; + case IFN_LEN_MASK_LOAD: case IFN_LEN_MASK_STORE: return 3; @@ -4510,6 +4541,52 @@ internal_fn_mask_index (internal_fn fn) } } +/* If FN takes a vector len argument, return the index of that argument, + otherwise return -1. */ + +int +internal_fn_len_index (internal_fn fn) +{ + switch (fn) + { + case IFN_LEN_LOAD: + case IFN_LEN_STORE: + case IFN_LEN_MASK_LOAD: + case IFN_LEN_MASK_STORE: + return 3; + + case IFN_LEN_MASK_GATHER_LOAD: + case IFN_LEN_MASK_SCATTER_STORE: + return 4; + + default: + return -1; + } +} + +/* If FN takes a vector bias argument, return the index of that argument, + otherwise return -1. */ + +int +internal_fn_bias_index (internal_fn fn) +{ + switch (fn) + { + case IFN_LEN_LOAD: + case IFN_LEN_STORE: + case IFN_LEN_MASK_LOAD: + case IFN_LEN_MASK_STORE: + return 4; + + case IFN_LEN_MASK_GATHER_LOAD: + case IFN_LEN_MASK_SCATTER_STORE: + return 5; + + default: + return -1; + } +} + /* If FN takes a value that should be stored to memory, return the index of that argument, otherwise return -1. */ @@ -4522,6 +4599,7 @@ internal_fn_stored_value_index (internal_fn fn) case IFN_MASK_STORE_LANES: case IFN_SCATTER_STORE: case IFN_MASK_SCATTER_STORE: + case IFN_LEN_MASK_SCATTER_STORE: case IFN_LEN_STORE: return 3; diff --git a/gcc/internal-fn.def b/gcc/internal-fn.def index d9fcca8430f..a1fdbcbf515 100644 --- a/gcc/internal-fn.def +++ b/gcc/internal-fn.def @@ -48,14 +48,14 @@ along with GCC; see the file COPYING3. If not see - mask_load: currently just maskload - load_lanes: currently just vec_load_lanes - mask_load_lanes: currently just vec_mask_load_lanes - - gather_load: used for {mask_,}gather_load + - gather_load: used for {mask_,len_mask_,}gather_load - len_load: currently just len_load - len_maskload: currently just len_maskload - mask_store: currently just maskstore - store_lanes: currently just vec_store_lanes - mask_store_lanes: currently just vec_mask_store_lanes - - scatter_store: used for {mask_,}scatter_store + - scatter_store: used for {mask_,len_mask_,}scatter_store - len_store: currently just len_store - len_maskstore: currently just len_maskstore @@ -157,6 +157,8 @@ DEF_INTERNAL_OPTAB_FN (MASK_LOAD_LANES, ECF_PURE, DEF_INTERNAL_OPTAB_FN (GATHER_LOAD, ECF_PURE, gather_load, gather_load) DEF_INTERNAL_OPTAB_FN (MASK_GATHER_LOAD, ECF_PURE, mask_gather_load, gather_load) +DEF_INTERNAL_OPTAB_FN (LEN_MASK_GATHER_LOAD, ECF_PURE, + len_mask_gather_load, gather_load) DEF_INTERNAL_OPTAB_FN (LEN_LOAD, ECF_PURE, len_load, len_load) DEF_INTERNAL_OPTAB_FN (LEN_MASK_LOAD, ECF_PURE, len_maskload, len_maskload) @@ -164,6 +166,8 @@ DEF_INTERNAL_OPTAB_FN (LEN_MASK_LOAD, ECF_PURE, len_maskload, len_maskload) DEF_INTERNAL_OPTAB_FN (SCATTER_STORE, 0, scatter_store, scatter_store) DEF_INTERNAL_OPTAB_FN (MASK_SCATTER_STORE, 0, mask_scatter_store, scatter_store) +DEF_INTERNAL_OPTAB_FN (LEN_MASK_SCATTER_STORE, 0, + len_mask_scatter_store, scatter_store) DEF_INTERNAL_OPTAB_FN (MASK_STORE, 0, maskstore, mask_store) DEF_INTERNAL_OPTAB_FN (STORE_LANES, ECF_CONST, vec_store_lanes, store_lanes) diff --git a/gcc/internal-fn.h b/gcc/internal-fn.h index 8f21068e300..396ad2a48bc 100644 --- a/gcc/internal-fn.h +++ b/gcc/internal-fn.h @@ -234,6 +234,8 @@ extern bool internal_load_fn_p (internal_fn); extern bool internal_store_fn_p (internal_fn); extern bool internal_gather_scatter_fn_p (internal_fn); extern int internal_fn_mask_index (internal_fn); +extern int internal_fn_len_index (internal_fn); +extern int internal_fn_bias_index (internal_fn); extern int internal_fn_stored_value_index (internal_fn); extern bool internal_gather_scatter_fn_supported_p (internal_fn, tree, tree, tree, int); diff --git a/gcc/optabs.def b/gcc/optabs.def index a901b68c538..73c9a0c760f 100644 --- a/gcc/optabs.def +++ b/gcc/optabs.def @@ -95,8 +95,10 @@ OPTAB_CD(len_maskload_optab, "len_maskload$a$b") OPTAB_CD(len_maskstore_optab, "len_maskstore$a$b") OPTAB_CD(gather_load_optab, "gather_load$a$b") OPTAB_CD(mask_gather_load_optab, "mask_gather_load$a$b") +OPTAB_CD(len_mask_gather_load_optab, "len_mask_gather_load$a$b") OPTAB_CD(scatter_store_optab, "scatter_store$a$b") OPTAB_CD(mask_scatter_store_optab, "mask_scatter_store$a$b") +OPTAB_CD(len_mask_scatter_store_optab, "len_mask_scatter_store$a$b") OPTAB_CD(vec_extract_optab, "vec_extract$a$b") OPTAB_CD(vec_init_optab, "vec_init$a$b")