From patchwork Thu Jun 15 13:14:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "juzhe.zhong@rivai.ai" X-Patchwork-Id: 108512 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp627050vqr; Thu, 15 Jun 2023 06:15:31 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7mFGLYAxTT7HJyS6mfc40yiuV4+8kf+00knnim15tAMrUTjB73nAoDNLGL6XGDQeEQe8GX X-Received: by 2002:a17:907:9306:b0:974:1e85:6a69 with SMTP id bu6-20020a170907930600b009741e856a69mr18590510ejc.16.1686834931540; Thu, 15 Jun 2023 06:15:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1686834931; cv=none; d=google.com; s=arc-20160816; b=xVtNvDO1yezKWFdxDV1F/6TJoTP1nuOtIAjnlqkuleH8d7us6zkvY234/Jv1X3W3Kq 3IYcMWN6jL8DZ813ExNuCJB/wGeIlTc2i/HaojKzw0VVtQmLsG7/4SMw6FhN4AUb6oQp XXhKLYgHySfUTgPU87sip0DPKrV8yQgzTETB/bCxHB0yiLk//SO38YvXYrvKZ3i7GM9j th6rSQF8oYT6sDneqI5K8NX6OPPWeY+RYOLteFcYxKWCDirj+ZHm+N2gne1xGwIATlDy UrJuqcrmm6pDwOtPTv0cNN7vGxjnOC3NHW0Q6ohO1TUTTfTVeCfgI/gWOEOtt0eW9sFz 2vFQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:feedback-id :content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:dmarc-filter:delivered-to; bh=z2kloMUNcW+tz15Sn637xreCr5TFqVKcE+eot6mpH/M=; b=zW2sIeazpyPfCa9R8C7kVsr8hQpvoS3g9QPSRXfbG3hlNWHOswxGLuYBDnQDNQvRUM RUq/Ql/k4x5QopiE+/eNV149rZj938A3cFKzMli2VQ2I20dq5wJ6V4vVGKit9bVUFt1P dkrmYYs8m5yNWAufTrylNsy5aC/k7MSsfKkLsdayL+maMHefGWtZVweyfYy2Lw4DGUn1 Y6hYLZIVLuKr5UXMVmM5NFJ2gzquJnN8TX7s0wuzyOKsyEVOobgw9kydT1DhCXzSXJxc VJB0zLfAJc+NjWxFlToRn75I3NvAu2x9F1j8aUKWabTWBIjZ/zy5hlUcByo8QcA4IMDG 2YIQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from sourceware.org (server2.sourceware.org. [2620:52:3:1:0:246e:9693:128c]) by mx.google.com with ESMTPS id b5-20020a170906194500b00977e10fafd3si6214966eje.1045.2023.06.15.06.15.31 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Jun 2023 06:15:31 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) client-ip=2620:52:3:1:0:246e:9693:128c; Authentication-Results: mx.google.com; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 0B1C83857013 for ; Thu, 15 Jun 2023 13:15:17 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtpbgau1.qq.com (smtpbgau1.qq.com [54.206.16.166]) by sourceware.org (Postfix) with ESMTPS id D5E8F3858C2F for ; Thu, 15 Jun 2023 13:14:46 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org D5E8F3858C2F Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivai.ai Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivai.ai X-QQ-mid: bizesmtp89t1686834879td7sp88d Received: from server1.localdomain ( [58.60.1.22]) by bizesmtp.qq.com (ESMTP) with id ; Thu, 15 Jun 2023 21:14:38 +0800 (CST) X-QQ-SSF: 01400000000000G0S000000A0001000 X-QQ-FEAT: PS/N6jJLnDYWUEfYAYmEZ4NW50CSnk+RGhIYwoYcFxIQ6+yMGrQw5NoSV7KAQ Ji30OXvXU57wiVDel/xSmc2iMMZDJ+hWH/GHx2Uay4VG9Tg0P8MKohYBckLSGlqlZ0VcxoE n5kPKfarrrZ2bTrhsvwnpIPcOd6epItj1rqDirgTEmtDVXrY+dfgQGYlZ6WWM8/dRbT90NE a87JD/MSTeKK67Kt5kigp5VPI9UDaQ2mJig849u8ZBrZQxY2xonn3xU5XPpB6hFblvJeaGB rRDIdUshWqvytVaJ3ClJufD2dEJJWYJnyMZWBUUyCgiX2Yvj4PoMxckT2tApKIirC+kQScj PViwrawWY9n+77H0UTRSbzuiYnMoainYYYwMmOb+bY2VU8tGjSVKSlr2j20bDcKDLT4KZBr LDcwMbfKFag= X-QQ-GoodBg: 2 X-BIZMAIL-ID: 13639301889871104734 From: juzhe.zhong@rivai.ai To: gcc-patches@gcc.gnu.org Cc: rguenther@suse.de, richard.sandiford@arm.com, rdapp.gcc@gmail.com, Ju-Zhe Zhong Subject: [PATCH V4] VECT: Support LEN_MASK_{LOAD,STORE} ifn && optabs Date: Thu, 15 Jun 2023 21:14:35 +0800 Message-Id: <20230615131435.10323-1-juzhe.zhong@rivai.ai> X-Mailer: git-send-email 2.36.1 MIME-Version: 1.0 X-QQ-SENDSIZE: 520 Feedback-ID: bizesmtp:rivai.ai:qybglogicsvrgz:qybglogicsvrgz7a-one-0 X-Spam-Status: No, score=-10.3 required=5.0 tests=BAYES_00, GIT_PATCH_0, KAM_DMARC_STATUS, RCVD_IN_BARRACUDACENTRAL, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H5, RCVD_IN_MSPIKE_WL, SPF_HELO_PASS, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org Sender: "Gcc-patches" X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1768762735153489045?= X-GMAIL-MSGID: =?utf-8?q?1768774625138676182?= From: Ju-Zhe Zhong This patch bootstrap pass on X86, ok for trunk ? Accoding to comments from Richi, split the first patch to add ifn && optabs of LEN_MASK_{LOAD,STORE} only, we don't apply them into vectorizer in this patch. And also add BIAS argument for possible s390's future use. The description of the patterns in doc are coming Robin. After this patch is approved, will send the second patch to apply len_mask_* patterns into vectorizer. Target like ARM SVE in GCC has an elegant way to handle both loop control and flow control simultaneously: loop_control_mask = WHILE_ULT flow_control_mask = comparison control_mask = loop_control_mask & flow_control_mask; MASK_LOAD (control_mask) MASK_STORE (control_mask) However, targets like RVV (RISC-V Vector) can not use this approach in auto-vectorization since RVV use length in loop control. This patch adds LEN_MASK_ LOAD/STORE to support flow control for targets like RISC-V that uses length in loop control. Normalize load/store into LEN_MASK_ LOAD/STORE as long as either length or mask is valid. Length is the outcome of SELECT_VL or MIN_EXPR. Mask is the outcome of comparison. LEN_MASK_ LOAD/STORE format is defined as follows: 1). LEN_MASK_LOAD (ptr, align, length, mask). 2). LEN_MASK_STORE (ptr, align, length, mask, vec). Consider these 4 following cases: VLA: Variable-length auto-vectorization VLS: Specific-length auto-vectorization Case 1 (VLS): -mrvv-vector-bits=128 IR (Does not use LEN_MASK_*): Code: v1 = MEM (...) for (int i = 0; i < 4; i++) v2 = MEM (...) a[i] = b[i] + c[i]; v3 = v1 + v2 MEM[...] = v3 Case 2 (VLS): -mrvv-vector-bits=128 IR (LEN_MASK_* with length = VF, mask = comparison): Code: mask = comparison for (int i = 0; i < 4; i++) v1 = LEN_MASK_LOAD (length = VF, mask) if (cond[i]) v2 = LEN_MASK_LOAD (length = VF, mask) a[i] = b[i] + c[i]; v3 = v1 + v2 LEN_MASK_STORE (length = VF, mask, v3) Case 3 (VLA): Code: loop_len = SELECT_VL or MIN for (int i = 0; i < n; i++) v1 = LEN_MASK_LOAD (length = loop_len, mask = {-1,-1,...}) a[i] = b[i] + c[i]; v2 = LEN_MASK_LOAD (length = loop_len, mask = {-1,-1,...}) v3 = v1 + v2 LEN_MASK_STORE (length = loop_len, mask = {-1,-1,...}, v3) Case 4 (VLA): Code: loop_len = SELECT_VL or MIN for (int i = 0; i < n; i++) mask = comparison if (cond[i]) v1 = LEN_MASK_LOAD (length = loop_len, mask) a[i] = b[i] + c[i]; v2 = LEN_MASK_LOAD (length = loop_len, mask) v3 = v1 + v2 LEN_MASK_STORE (length = loop_len, mask, v3) Co-authored-by: Robin Dapp gcc/ChangeLog: * doc/md.texi: Add len_mask{load,store}. * genopinit.cc (main): Ditto. (CMP_NAME): Ditto. * internal-fn.cc (len_maskload_direct): Ditto. (len_maskstore_direct): Ditto. (expand_call_mem_ref): Ditto. (expand_partial_load_optab_fn): Ditto. (expand_len_maskload_optab_fn): Ditto. (expand_partial_store_optab_fn): Ditto. (expand_len_maskstore_optab_fn): Ditto. (direct_len_maskload_optab_supported_p): Ditto. (direct_len_maskstore_optab_supported_p): Ditto. * internal-fn.def (LEN_MASK_LOAD): Ditto. (LEN_MASK_STORE): Ditto. * optabs.def (OPTAB_CD): Ditto. --- gcc/doc/md.texi | 46 +++++++++++++++++++++++++++++++++++++++++++++ gcc/genopinit.cc | 6 ++++-- gcc/internal-fn.cc | 43 ++++++++++++++++++++++++++++++++++++++---- gcc/internal-fn.def | 4 ++++ gcc/optabs.def | 2 ++ 5 files changed, 95 insertions(+), 6 deletions(-) diff --git a/gcc/doc/md.texi b/gcc/doc/md.texi index a43fd65a2b2..af23ec938d6 100644 --- a/gcc/doc/md.texi +++ b/gcc/doc/md.texi @@ -5136,6 +5136,52 @@ of @code{QI} elements. This pattern is not allowed to @code{FAIL}. +@cindex @code{len_maskload@var{m}@var{n}} instruction pattern +@item @samp{len_maskload@var{m}@var{n}} +Perform a masked load (operand 2 - operand 4) elements from vector memory +operand 1 into vector register operand 0, setting the other elements of +operand 0 to undefined values. This is a combination of len_load and maskload. +Operands 0 and 1 have mode @var{m}, which must be a vector mode. Operand 2 +has whichever integer mode the target prefers. A secondary mask is specified in +operand 3 which must be of type @var{n}. Operand 4 conceptually has mode @code{QI}. + +Operand 2 can be a variable or a constant amount. Operand 4 specifies a +constant bias: it is either a constant 0 or a constant -1. The predicate on +operand 4 must only accept the bias values that the target actually supports. +GCC handles a bias of 0 more efficiently than a bias of -1. + +If (operand 2 - operand 4) exceeds the number of elements in mode +@var{m}, the behavior is undefined. + +If the target prefers the length to be measured in bytes +rather than elements, it should only implement this pattern for vectors +of @code{QI} elements. + +This pattern is not allowed to @code{FAIL}. + +@cindex @code{len_maskstore@var{m}@var{n}} instruction pattern +@item @samp{len_maskstore@var{m}@var{n}} +Perform a masked store (operand 2 - operand 4) vector elements from vector register +operand 1 into memory operand 0, leaving the other elements of operand 0 unchanged. +This is a combination of len_store and maskstore. +Operands 0 and 1 have mode @var{m}, which must be a vector mode. Operand 2 has whichever +integer mode the target prefers. A secondary mask is specified in operand 3 which must be +of type @var{n}. Operand 4 conceptually has mode @code{QI}. + +Operand 2 can be a variable or a constant amount. Operand 3 specifies a +constant bias: it is either a constant 0 or a constant -1. The predicate on +operand 4 must only accept the bias values that the target actually supports. +GCC handles a bias of 0 more efficiently than a bias of -1. + +If (operand 2 - operand 4) exceeds the number of elements in mode +@var{m}, the behavior is undefined. + +If the target prefers the length to be measured in bytes +rather than elements, it should only implement this pattern for vectors +of @code{QI} elements. + +This pattern is not allowed to @code{FAIL}. + @cindex @code{vec_perm@var{m}} instruction pattern @item @samp{vec_perm@var{m}} Output a (variable) vector permutation. Operand 0 is the destination diff --git a/gcc/genopinit.cc b/gcc/genopinit.cc index 0c1b6859ca0..6bd8858a1d9 100644 --- a/gcc/genopinit.cc +++ b/gcc/genopinit.cc @@ -376,7 +376,8 @@ main (int argc, const char **argv) fprintf (s_file, "/* Returns TRUE if the target supports any of the partial vector\n" - " optabs: while_ult_optab, len_load_optab or len_store_optab,\n" + " optabs: while_ult_optab, len_load_optab, len_store_optab,\n" + " len_maskload_optab or len_maskstore_optab,\n" " for any mode. */\n" "bool\npartial_vectors_supported_p (void)\n{\n"); bool any_match = false; @@ -386,7 +387,8 @@ main (int argc, const char **argv) { #define CMP_NAME(N) !strncmp (p->name, (N), strlen ((N))) if (CMP_NAME("while_ult") || CMP_NAME ("len_load") - || CMP_NAME ("len_store")) + || CMP_NAME ("len_store")|| CMP_NAME ("len_maskload") + || CMP_NAME ("len_maskstore")) { if (first) fprintf (s_file, " HAVE_%s", p->name); diff --git a/gcc/internal-fn.cc b/gcc/internal-fn.cc index 208bdf497eb..c911ae790cb 100644 --- a/gcc/internal-fn.cc +++ b/gcc/internal-fn.cc @@ -165,6 +165,7 @@ init_internal_fns () #define mask_load_lanes_direct { -1, -1, false } #define gather_load_direct { 3, 1, false } #define len_load_direct { -1, -1, false } +#define len_maskload_direct { -1, 3, false } #define mask_store_direct { 3, 2, false } #define store_lanes_direct { 0, 0, false } #define mask_store_lanes_direct { 0, 0, false } @@ -172,6 +173,7 @@ init_internal_fns () #define vec_cond_direct { 2, 0, false } #define scatter_store_direct { 3, 1, false } #define len_store_direct { 3, 3, false } +#define len_maskstore_direct { 4, 3, false } #define vec_set_direct { 3, 3, false } #define unary_direct { 0, 0, true } #define unary_convert_direct { -1, 0, true } @@ -2873,12 +2875,13 @@ expand_call_mem_ref (tree type, gcall *stmt, int index) return fold_build2 (MEM_REF, type, addr, build_int_cst (alias_ptr_type, 0)); } -/* Expand MASK_LOAD{,_LANES} or LEN_LOAD call STMT using optab OPTAB. */ +/* Expand MASK_LOAD{,_LANES}, LEN_MASK_LOAD or LEN_LOAD call STMT using optab + * OPTAB. */ static void expand_partial_load_optab_fn (internal_fn, gcall *stmt, convert_optab optab) { - class expand_operand ops[4]; + class expand_operand ops[5]; tree type, lhs, rhs, maskt, biast; rtx mem, target, mask, bias; insn_code icode; @@ -2913,6 +2916,20 @@ expand_partial_load_optab_fn (internal_fn, gcall *stmt, convert_optab optab) create_input_operand (&ops[3], bias, QImode); expand_insn (icode, 4, ops); } + else if (optab == len_maskload_optab) + { + create_convert_operand_from (&ops[2], mask, TYPE_MODE (TREE_TYPE (maskt)), + TYPE_UNSIGNED (TREE_TYPE (maskt))); + maskt = gimple_call_arg (stmt, 3); + mask = expand_normal (maskt); + create_input_operand (&ops[3], mask, TYPE_MODE (TREE_TYPE (maskt))); + icode = convert_optab_handler (optab, TYPE_MODE (type), + TYPE_MODE (TREE_TYPE (maskt))); + biast = gimple_call_arg (stmt, 4); + bias = expand_normal (biast); + create_input_operand (&ops[4], bias, QImode); + expand_insn (icode, 5, ops); + } else { create_input_operand (&ops[2], mask, TYPE_MODE (TREE_TYPE (maskt))); @@ -2926,13 +2943,15 @@ expand_partial_load_optab_fn (internal_fn, gcall *stmt, convert_optab optab) #define expand_mask_load_optab_fn expand_partial_load_optab_fn #define expand_mask_load_lanes_optab_fn expand_mask_load_optab_fn #define expand_len_load_optab_fn expand_partial_load_optab_fn +#define expand_len_maskload_optab_fn expand_partial_load_optab_fn -/* Expand MASK_STORE{,_LANES} or LEN_STORE call STMT using optab OPTAB. */ +/* Expand MASK_STORE{,_LANES}, LEN_MASK_STORE or LEN_STORE call STMT using optab + * OPTAB. */ static void expand_partial_store_optab_fn (internal_fn, gcall *stmt, convert_optab optab) { - class expand_operand ops[4]; + class expand_operand ops[5]; tree type, lhs, rhs, maskt, biast; rtx mem, reg, mask, bias; insn_code icode; @@ -2965,6 +2984,19 @@ expand_partial_store_optab_fn (internal_fn, gcall *stmt, convert_optab optab) create_input_operand (&ops[3], bias, QImode); expand_insn (icode, 4, ops); } + else if (optab == len_maskstore_optab) + { + create_convert_operand_from (&ops[2], mask, TYPE_MODE (TREE_TYPE (maskt)), + TYPE_UNSIGNED (TREE_TYPE (maskt))); + maskt = gimple_call_arg (stmt, 3); + mask = expand_normal (maskt); + create_input_operand (&ops[3], mask, TYPE_MODE (TREE_TYPE (maskt))); + biast = gimple_call_arg (stmt, 4); + bias = expand_normal (biast); + create_input_operand (&ops[4], bias, QImode); + icode = convert_optab_handler (optab, TYPE_MODE (type), GET_MODE (mask)); + expand_insn (icode, 5, ops); + } else { create_input_operand (&ops[2], mask, TYPE_MODE (TREE_TYPE (maskt))); @@ -2975,6 +3007,7 @@ expand_partial_store_optab_fn (internal_fn, gcall *stmt, convert_optab optab) #define expand_mask_store_optab_fn expand_partial_store_optab_fn #define expand_mask_store_lanes_optab_fn expand_mask_store_optab_fn #define expand_len_store_optab_fn expand_partial_store_optab_fn +#define expand_len_maskstore_optab_fn expand_partial_store_optab_fn /* Expand VCOND, VCONDU and VCONDEQ optab internal functions. The expansion of STMT happens based on OPTAB table associated. */ @@ -3928,6 +3961,7 @@ multi_vector_optab_supported_p (convert_optab optab, tree_pair types, #define direct_mask_load_lanes_optab_supported_p multi_vector_optab_supported_p #define direct_gather_load_optab_supported_p convert_optab_supported_p #define direct_len_load_optab_supported_p direct_optab_supported_p +#define direct_len_maskload_optab_supported_p convert_optab_supported_p #define direct_mask_store_optab_supported_p convert_optab_supported_p #define direct_store_lanes_optab_supported_p multi_vector_optab_supported_p #define direct_mask_store_lanes_optab_supported_p multi_vector_optab_supported_p @@ -3935,6 +3969,7 @@ multi_vector_optab_supported_p (convert_optab optab, tree_pair types, #define direct_vec_cond_optab_supported_p convert_optab_supported_p #define direct_scatter_store_optab_supported_p convert_optab_supported_p #define direct_len_store_optab_supported_p direct_optab_supported_p +#define direct_len_maskstore_optab_supported_p convert_optab_supported_p #define direct_while_optab_supported_p convert_optab_supported_p #define direct_fold_extract_optab_supported_p direct_optab_supported_p #define direct_fold_left_optab_supported_p direct_optab_supported_p diff --git a/gcc/internal-fn.def b/gcc/internal-fn.def index 9da5f31636e..bc947c0fde7 100644 --- a/gcc/internal-fn.def +++ b/gcc/internal-fn.def @@ -50,12 +50,14 @@ along with GCC; see the file COPYING3. If not see - mask_load_lanes: currently just vec_mask_load_lanes - gather_load: used for {mask_,}gather_load - len_load: currently just len_load + - len_maskload: currently just len_maskload - mask_store: currently just maskstore - store_lanes: currently just vec_store_lanes - mask_store_lanes: currently just vec_mask_store_lanes - scatter_store: used for {mask_,}scatter_store - len_store: currently just len_store + - len_maskstore: currently just len_maskstore - unary: a normal unary optab, such as vec_reverse_ - binary: a normal binary optab, such as vec_interleave_lo_ @@ -157,6 +159,7 @@ DEF_INTERNAL_OPTAB_FN (MASK_GATHER_LOAD, ECF_PURE, mask_gather_load, gather_load) DEF_INTERNAL_OPTAB_FN (LEN_LOAD, ECF_PURE, len_load, len_load) +DEF_INTERNAL_OPTAB_FN (LEN_MASK_LOAD, ECF_PURE, len_maskload, len_maskload) DEF_INTERNAL_OPTAB_FN (SCATTER_STORE, 0, scatter_store, scatter_store) DEF_INTERNAL_OPTAB_FN (MASK_SCATTER_STORE, 0, @@ -175,6 +178,7 @@ DEF_INTERNAL_OPTAB_FN (VCOND_MASK, 0, vcond_mask, vec_cond_mask) DEF_INTERNAL_OPTAB_FN (VEC_SET, 0, vec_set, vec_set) DEF_INTERNAL_OPTAB_FN (LEN_STORE, 0, len_store, len_store) +DEF_INTERNAL_OPTAB_FN (LEN_MASK_STORE, 0, len_maskstore, len_maskstore) DEF_INTERNAL_OPTAB_FN (WHILE_ULT, ECF_CONST | ECF_NOTHROW, while_ult, while) DEF_INTERNAL_OPTAB_FN (SELECT_VL, ECF_CONST | ECF_NOTHROW, select_vl, binary) diff --git a/gcc/optabs.def b/gcc/optabs.def index 22b31be0f72..9533eb11565 100644 --- a/gcc/optabs.def +++ b/gcc/optabs.def @@ -91,6 +91,8 @@ OPTAB_CD(vec_cmpu_optab, "vec_cmpu$a$b") OPTAB_CD(vec_cmpeq_optab, "vec_cmpeq$a$b") OPTAB_CD(maskload_optab, "maskload$a$b") OPTAB_CD(maskstore_optab, "maskstore$a$b") +OPTAB_CD(len_maskload_optab, "len_maskload$a$b") +OPTAB_CD(len_maskstore_optab, "len_maskstore$a$b") OPTAB_CD(gather_load_optab, "gather_load$a$b") OPTAB_CD(mask_gather_load_optab, "mask_gather_load$a$b") OPTAB_CD(scatter_store_optab, "scatter_store$a$b")