From patchwork Mon Jul 10 11:35:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "juzhe.zhong@rivai.ai" X-Patchwork-Id: 117869 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f45:0:b0:3ea:f831:8777 with SMTP id v5csp4957363vqx; Mon, 10 Jul 2023 04:36:36 -0700 (PDT) X-Google-Smtp-Source: APBJJlFkbEbPcLRm5TfqmZrdOBJR0yCWpcAK/seNt7gooyo6HXG0oSzSSwOTGWlOk/mXHqgsXO/O X-Received: by 2002:a2e:9844:0:b0:2b7:2f2:231d with SMTP id e4-20020a2e9844000000b002b702f2231dmr9336786ljj.27.1688988995990; Mon, 10 Jul 2023 04:36:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1688988995; cv=none; d=google.com; s=arc-20160816; b=qnhCR6i2+IqbjLABuhLV7kDNsdUPVpBziy+YgN+FZbdX4/A4jR9pPamxObgpVIBi/m PtpBNil47sdkAKgt5retoECPK/d0bhKISSNS+br/ZIEtDXB5bUV1MK33llePavOZHh2C aqEHazkC6shW5x82j9ddBB+8Ykttsbny+RGt0dE9Su7MkT+OLWCJ/NC8bPWq4rsJ4afe 82I+rQbTWXyxu3mPAjbZpOj28eJsopBGYnCAiFrAkdtbEst57SdjSNcHt9YNOm6igPey eh5PiC3hhuVImHE16A0lKYhpRZv1jsh6DErGEO8PLo2ox1J7rTYLcoa2p7F7PdINvhJN U9Aw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:feedback-id :content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:dmarc-filter:delivered-to; bh=b/ZN0MC/5OgkjEuX64p2oZl1PDVFQJguIOiAovj6qr0=; fh=CHbn33ss3MXGFqXGZpS89+qfBQv2oFkoJCJQmHw6RIo=; b=PfdaozWCl7Fo58jIpig7b9rCP1qTh7gfAEkdKhrbEuVbkv9VlvUIpdWCeLFDDbw8Wu bClEFkAqHGsWm0Y12rkKWyItaPUJzmYx+D4lHlLt6HsXUky02S1gEdGmzrl9CJaxOc1h 17Vzm3yjwfZtwLqXCsyYQitf42lIwKO5di2iowCTPrTbu1mcJP/Lqn/8xTsOMvZPCw6b jMtkQ/c0JMz5DInbmI/ZdjJUu0uu+dDKARKrJlt72gtHVkB8rwrRfyxDOlUrTQfaDKn/ zWIT3vW9mV9Veots2fMscZ9zhJW4lSrdZ+lzBFarkBV8gllmss2dvwrSW4vJdmIE4bF4 fB8Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (server2.sourceware.org. [2620:52:3:1:0:246e:9693:128c]) by mx.google.com with ESMTPS id gl13-20020a170906e0cd00b00992dcae843esi7489121ejb.202.2023.07.10.04.36.35 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jul 2023 04:36:35 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) client-ip=2620:52:3:1:0:246e:9693:128c; Authentication-Results: mx.google.com; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 67CE1385802F for ; Mon, 10 Jul 2023 11:36:28 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtpbgsg1.qq.com (smtpbgsg1.qq.com [54.254.200.92]) by sourceware.org (Postfix) with ESMTPS id 89CD0385770A for ; Mon, 10 Jul 2023 11:35:56 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 89CD0385770A Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivai.ai Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivai.ai X-QQ-mid: bizesmtp68t1688988950tw6fcwxy Received: from server1.localdomain ( [58.60.1.22]) by bizesmtp.qq.com (ESMTP) with id ; Mon, 10 Jul 2023 19:35:49 +0800 (CST) X-QQ-SSF: 01400000000000G0T000000A0000000 X-QQ-FEAT: C46Rb8GPIEcJX6cjqSOD8H97RUqBDEBgOjRHdlKBAGKgWEmdX5eGdjaEiBgfJ 9Vx5uaqkGRIFUu3wHiVGebdh5IXcwdD75b4Tha/1r7sRo9MSb3BTPl4I6apgu7hFU80I4wJ rK/EoQVptfZNJLroPmI2+A4gZiYs2B//Tv89dHsqdsRYrbCrdNJK7Vfc6kJzTB3EKjo3qOJ r7L4KbuFOKYIswbxDIkwOW+OJPC3DQ0SzMA7V5bGfwYgsIWA3HdkrQSMvxHGiOTwTPv5P+H BMu9u7PXR28i8Ti/V6/8Q5pjAMfvgaiyT8NKjFMYr1ymr+/nzgeziG+niKcO8WrPYyBufA1 BFvLhXa+4nykBHcDHEtujGUJ0sdbe04Jzwb/rCw9PPHJMyihHDGAe6UXo3ltgK9wK/awlwt R7lWPLn/UDY= X-QQ-GoodBg: 2 X-BIZMAIL-ID: 5602987407085105005 From: juzhe.zhong@rivai.ai To: gcc-patches@gcc.gnu.org Cc: richard.sandiford@arm.com, rguenther@suse.de, Ju-Zhe Zhong Subject: [PATCH V2] VECT: Add COND_LEN_* operations for loop control with length targets Date: Mon, 10 Jul 2023 19:35:47 +0800 Message-Id: <20230710113547.142562-1-juzhe.zhong@rivai.ai> X-Mailer: git-send-email 2.36.1 MIME-Version: 1.0 X-QQ-SENDSIZE: 520 Feedback-ID: bizesmtp:rivai.ai:qybglogicsvrgz:qybglogicsvrgz7a-one-0 X-Spam-Status: No, score=-10.3 required=5.0 tests=BAYES_00, GIT_PATCH_0, KAM_DMARC_STATUS, RCVD_IN_BARRACUDACENTRAL, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H4, RCVD_IN_MSPIKE_WL, SPF_HELO_PASS, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org Sender: "Gcc-patches" X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771033325295339192 X-GMAIL-MSGID: 1771033325295339192 From: Ju-Zhe Zhong Hi, Richard and Richi. This patch is adding cond_len_* operations pattern for target support loop control with length. These patterns will be used in these following case: 1. Integer division: void f (int32_t *restrict a, int32_t *restrict b, int32_t *restrict c, int n) { for (int i = 0; i < n; ++i) { a[i] = b[i] / c[i]; } } ARM SVE IR: ... max_mask_36 = .WHILE_ULT (0, bnd.5_32, { 0, ... }); Loop: ... # loop_mask_29 = PHI ... vect__4.8_28 = .MASK_LOAD (_33, 32B, loop_mask_29); ... vect__6.11_25 = .MASK_LOAD (_20, 32B, loop_mask_29); vect__8.12_24 = .COND_DIV (loop_mask_29, vect__4.8_28, vect__6.11_25, vect__4.8_28); ... .MASK_STORE (_1, 32B, loop_mask_29, vect__8.12_24); ... next_mask_37 = .WHILE_ULT (_2, bnd.5_32, { 0, ... }); ... For target like RVV who support loop control with length, we want to see IR as follows: Loop: ... # loop_len_29 = SELECT_VL ... vect__4.8_28 = .LEN_MASK_LOAD (_33, 32B, loop_len_29); ... vect__6.11_25 = .LEN_MASK_LOAD (_20, 32B, loop_len_29); vect__8.12_24 = .COND_LEN_DIV (dummp_mask, vect__4.8_28, vect__6.11_25, vect__4.8_28, loop_len_29, bias); ... .LEN_MASK_STORE (_1, 32B, loop_len_29, vect__8.12_24); ... next_mask_37 = .WHILE_ULT (_2, bnd.5_32, { 0, ... }); ... Notice here, we use dummp_mask = { -1, -1, .... , -1 } 2. Integer conditional division: Similar case with (1) but with condtion: void f (int32_t *restrict a, int32_t *restrict b, int32_t *restrict c, int32_t * cond, int n) { for (int i = 0; i < n; ++i) { if (cond[i]) a[i] = b[i] / c[i]; } } ARM SVE: ... max_mask_76 = .WHILE_ULT (0, bnd.6_52, { 0, ... }); Loop: ... # loop_mask_55 = PHI ... vect__4.9_56 = .MASK_LOAD (_51, 32B, loop_mask_55); mask__29.10_58 = vect__4.9_56 != { 0, ... }; vec_mask_and_61 = loop_mask_55 & mask__29.10_58; ... vect__6.13_62 = .MASK_LOAD (_24, 32B, vec_mask_and_61); ... vect__8.16_66 = .MASK_LOAD (_1, 32B, vec_mask_and_61); vect__10.17_68 = .COND_DIV (vec_mask_and_61, vect__6.13_62, vect__8.16_66, vect__6.13_62); ... .MASK_STORE (_2, 32B, vec_mask_and_61, vect__10.17_68); ... next_mask_77 = .WHILE_ULT (_3, bnd.6_52, { 0, ... }); Here, ARM SVE use vec_mask_and_61 = loop_mask_55 & mask__29.10_58; to gurantee the correct result. However, target with length control can not perform this elegant flow, for RVV, we would expect: Loop: ... loop_len_55 = SELECT_VL ... mask__29.10_58 = vect__4.9_56 != { 0, ... }; ... vect__10.17_68 = .COND_LEN_DIV (mask__29.10_58, vect__6.13_62, vect__8.16_66, vect__6.13_62, loop_len_55, bias); ... Here we expect COND_LEN_DIV predicated by a real mask which is the outcome of comparison: mask__29.10_58 = vect__4.9_56 != { 0, ... }; and a real length which is produced by loop control : loop_len_55 = SELECT_VL 3. conditional Floating-point operations (no -ffast-math): void f (float *restrict a, float *restrict b, int32_t *restrict cond, int n) { for (int i = 0; i < n; ++i) { if (cond[i]) a[i] = b[i] + a[i]; } } ARM SVE IR: max_mask_70 = .WHILE_ULT (0, bnd.6_46, { 0, ... }); ... # loop_mask_49 = PHI ... mask__27.10_52 = vect__4.9_50 != { 0, ... }; vec_mask_and_55 = loop_mask_49 & mask__27.10_52; ... vect__9.17_62 = .COND_ADD (vec_mask_and_55, vect__6.13_56, vect__8.16_60, vect__6.13_56); ... next_mask_71 = .WHILE_ULT (_22, bnd.6_46, { 0, ... }); ... For RVV, we would expect IR: ... loop_len_49 = SELECT_VL ... mask__27.10_52 = vect__4.9_50 != { 0, ... }; ... vect__9.17_62 = .COND_LEN_ADD (mask__27.10_52, vect__6.13_56, vect__8.16_60, vect__6.13_56, loop_len_49, bias); ... 4. Conditional un-ordered reduction: int32_t f (int32_t *restrict a, int32_t *restrict cond, int n) { int32_t result = 0; for (int i = 0; i < n; ++i) { if (cond[i]) result += a[i]; } return result; } ARM SVE IR: Loop: # vect_result_18.7_37 = PHI ... # loop_mask_40 = PHI ... mask__17.11_43 = vect__4.10_41 != { 0, ... }; vec_mask_and_46 = loop_mask_40 & mask__17.11_43; ... vect__33.16_51 = .COND_ADD (vec_mask_and_46, vect_result_18.7_37, vect__7.14_47, vect_result_18.7_37); ... next_mask_58 = .WHILE_ULT (_15, bnd.6_36, { 0, ... }); ... Epilogue: _53 = .REDUC_PLUS (vect__33.16_51); [tail call] For RVV, we expect: Loop: # vect_result_18.7_37 = PHI ... loop_len_40 = SELECT_VL ... mask__17.11_43 = vect__4.10_41 != { 0, ... }; ... vect__33.16_51 = .COND_LEN_ADD (mask__17.11_43, vect_result_18.7_37, vect__7.14_47, vect_result_18.7_37, loop_len_40, bias); ... next_mask_58 = .WHILE_ULT (_15, bnd.6_36, { 0, ... }); ... Epilogue: _53 = .REDUC_PLUS (vect__33.16_51); [tail call] I name these patterns as "cond_len_*" since I want the length operand comes after mask operand and all other operands except length operand same order as "cond_*" patterns. Such order will make life easier in the following loop vectorizer support. gcc/ChangeLog: * doc/md.texi: Add COND_LEN_* operations for loop control with length. * internal-fn.cc (cond_len_unary_direct): Ditto. (cond_len_binary_direct): Ditto. (cond_len_ternary_direct): Ditto. (expand_cond_len_unary_optab_fn): Ditto. (expand_cond_len_binary_optab_fn): Ditto. (expand_cond_len_ternary_optab_fn): Ditto. (direct_cond_len_unary_optab_supported_p): Ditto. (direct_cond_len_binary_optab_supported_p): Ditto. (direct_cond_len_ternary_optab_supported_p): Ditto. * internal-fn.def (COND_LEN_ADD): Ditto. (COND_LEN_SUB): Ditto. (COND_LEN_MUL): Ditto. (COND_LEN_DIV): Ditto. (COND_LEN_MOD): Ditto. (COND_LEN_RDIV): Ditto. (COND_LEN_MIN): Ditto. (COND_LEN_MAX): Ditto. (COND_LEN_FMIN): Ditto. (COND_LEN_FMAX): Ditto. (COND_LEN_AND): Ditto. (COND_LEN_IOR): Ditto. (COND_LEN_XOR): Ditto. (COND_LEN_SHL): Ditto. (COND_LEN_SHR): Ditto. (COND_LEN_FMA): Ditto. (COND_LEN_FMS): Ditto. (COND_LEN_FNMA): Ditto. (COND_LEN_FNMS): Ditto. (COND_LEN_NEG): Ditto. * optabs.def (OPTAB_D): Ditto. --- gcc/doc/md.texi | 80 +++++++++++++++++++++++++++++++++++++++++++++ gcc/internal-fn.cc | 15 +++++++++ gcc/internal-fn.def | 38 +++++++++++++++++++++ gcc/optabs.def | 24 ++++++++++++++ 4 files changed, 157 insertions(+) diff --git a/gcc/doc/md.texi b/gcc/doc/md.texi index b30a824488b..cbcb992e5d7 100644 --- a/gcc/doc/md.texi +++ b/gcc/doc/md.texi @@ -7234,6 +7234,86 @@ for (i = 0; i < GET_MODE_NUNITS (@var{m}); i++) op0[i] = op1[i] ? fma (op2[i], op3[i], op4[i]) : op5[i]; @end smallexample +@cindex @code{cond_len_add@var{mode}} instruction pattern +@cindex @code{cond_len_sub@var{mode}} instruction pattern +@cindex @code{cond_len_mul@var{mode}} instruction pattern +@cindex @code{cond_len_div@var{mode}} instruction pattern +@cindex @code{cond_len_udiv@var{mode}} instruction pattern +@cindex @code{cond_len_mod@var{mode}} instruction pattern +@cindex @code{cond_len_umod@var{mode}} instruction pattern +@cindex @code{cond_len_and@var{mode}} instruction pattern +@cindex @code{cond_len_ior@var{mode}} instruction pattern +@cindex @code{cond_len_xor@var{mode}} instruction pattern +@cindex @code{cond_len_smin@var{mode}} instruction pattern +@cindex @code{cond_len_smax@var{mode}} instruction pattern +@cindex @code{cond_len_umin@var{mode}} instruction pattern +@cindex @code{cond_len_umax@var{mode}} instruction pattern +@cindex @code{cond_len_fmin@var{mode}} instruction pattern +@cindex @code{cond_len_fmax@var{mode}} instruction pattern +@cindex @code{cond_len_ashl@var{mode}} instruction pattern +@cindex @code{cond_len_ashr@var{mode}} instruction pattern +@cindex @code{cond_len_lshr@var{mode}} instruction pattern +@item @samp{cond_len_add@var{mode}} +@itemx @samp{cond_len_sub@var{mode}} +@itemx @samp{cond_len_mul@var{mode}} +@itemx @samp{cond_len_div@var{mode}} +@itemx @samp{cond_len_udiv@var{mode}} +@itemx @samp{cond_len_mod@var{mode}} +@itemx @samp{cond_len_umod@var{mode}} +@itemx @samp{cond_len_and@var{mode}} +@itemx @samp{cond_len_ior@var{mode}} +@itemx @samp{cond_len_xor@var{mode}} +@itemx @samp{cond_len_smin@var{mode}} +@itemx @samp{cond_len_smax@var{mode}} +@itemx @samp{cond_len_umin@var{mode}} +@itemx @samp{cond_len_umax@var{mode}} +@itemx @samp{cond_len_fmin@var{mode}} +@itemx @samp{cond_len_fmax@var{mode}} +@itemx @samp{cond_len_ashl@var{mode}} +@itemx @samp{cond_len_ashr@var{mode}} +@itemx @samp{cond_len_lshr@var{mode}} +When operand 1 is true and element index < operand 5 + operand 6, perform an operation on operands 2 and 3 and +store the result in operand 0, otherwise store operand 4 in operand 0. +The operation only works for the operands are vectors. + +@smallexample +for (i = 0; i < ops[5] + ops[6]; i++) + op0[i] = op1[i] ? op2[i] @var{op} op3[i] : op4[i]; +@end smallexample + +where, for example, @var{op} is @code{+} for @samp{cond_len_add@var{mode}}. + +When defined for floating-point modes, the contents of @samp{op3[i]} +are not interpreted if @samp{op1[i]} is false, just like they would not +be in a normal C @samp{?:} condition. + +Operands 0, 2, 3 and 4 all have mode @var{m}. Operand 1 is a scalar +integer if @var{m} is scalar, otherwise it has the mode returned by +@code{TARGET_VECTORIZE_GET_MASK_MODE}. Operand 5 has whichever +integer mode the target prefers. + +@samp{cond_@var{op}@var{mode}} generally corresponds to a conditional +form of @samp{@var{op}@var{mode}3}. As an exception, the vector forms +of shifts correspond to patterns like @code{vashl@var{mode}3} rather +than patterns like @code{ashl@var{mode}3}. + +@cindex @code{cond_len_fma@var{mode}} instruction pattern +@cindex @code{cond_len_fms@var{mode}} instruction pattern +@cindex @code{cond_len_fnma@var{mode}} instruction pattern +@cindex @code{cond_len_fnms@var{mode}} instruction pattern +@item @samp{cond_len_fma@var{mode}} +@itemx @samp{cond_len_fms@var{mode}} +@itemx @samp{cond_len_fnma@var{mode}} +@itemx @samp{cond_len_fnms@var{mode}} +Like @samp{cond_len_add@var{m}}, except that the conditional operation +takes 3 operands rather than two. For example, the vector form of +@samp{cond_len_fma@var{mode}} is equivalent to: + +@smallexample +for (i = 0; i < ops[6] + ops[7]; i++) + op0[i] = op1[i] ? fma (op2[i], op3[i], op4[i]) : op5[i]; +@end smallexample + @cindex @code{neg@var{mode}cc} instruction pattern @item @samp{neg@var{mode}cc} Similar to @samp{mov@var{mode}cc} but for conditional negation. Conditionally diff --git a/gcc/internal-fn.cc b/gcc/internal-fn.cc index 278db7b1805..f9aaf66cf2a 100644 --- a/gcc/internal-fn.cc +++ b/gcc/internal-fn.cc @@ -183,6 +183,9 @@ init_internal_fns () #define cond_unary_direct { 1, 1, true } #define cond_binary_direct { 1, 1, true } #define cond_ternary_direct { 1, 1, true } +#define cond_len_unary_direct { 1, 1, true } +#define cond_len_binary_direct { 1, 1, true } +#define cond_len_ternary_direct { 1, 1, true } #define while_direct { 0, 2, false } #define fold_extract_direct { 2, 2, false } #define fold_left_direct { 1, 1, false } @@ -3869,6 +3872,15 @@ expand_convert_optab_fn (internal_fn fn, gcall *stmt, convert_optab optab, #define expand_cond_ternary_optab_fn(FN, STMT, OPTAB) \ expand_direct_optab_fn (FN, STMT, OPTAB, 5) +#define expand_cond_len_unary_optab_fn(FN, STMT, OPTAB) \ + expand_direct_optab_fn (FN, STMT, OPTAB, 5) + +#define expand_cond_len_binary_optab_fn(FN, STMT, OPTAB) \ + expand_direct_optab_fn (FN, STMT, OPTAB, 6) + +#define expand_cond_len_ternary_optab_fn(FN, STMT, OPTAB) \ + expand_direct_optab_fn (FN, STMT, OPTAB, 7) + #define expand_fold_extract_optab_fn(FN, STMT, OPTAB) \ expand_direct_optab_fn (FN, STMT, OPTAB, 3) @@ -3964,6 +3976,9 @@ multi_vector_optab_supported_p (convert_optab optab, tree_pair types, #define direct_cond_unary_optab_supported_p direct_optab_supported_p #define direct_cond_binary_optab_supported_p direct_optab_supported_p #define direct_cond_ternary_optab_supported_p direct_optab_supported_p +#define direct_cond_len_unary_optab_supported_p direct_optab_supported_p +#define direct_cond_len_binary_optab_supported_p direct_optab_supported_p +#define direct_cond_len_ternary_optab_supported_p direct_optab_supported_p #define direct_mask_load_optab_supported_p convert_optab_supported_p #define direct_load_lanes_optab_supported_p multi_vector_optab_supported_p #define direct_mask_load_lanes_optab_supported_p multi_vector_optab_supported_p diff --git a/gcc/internal-fn.def b/gcc/internal-fn.def index 238b7ee0bc9..ea750a921ed 100644 --- a/gcc/internal-fn.def +++ b/gcc/internal-fn.def @@ -72,6 +72,10 @@ along with GCC; see the file COPYING3. If not see - fold_left: for scalar = FN (scalar, vector), keyed off the vector mode - check_ptrs: used for check_{raw,war}_ptrs + - cond_len_unary: a conditional unary optab, such as cond_len_neg + - cond_len_binary: a conditional binary optab, such as cond_len_add + - cond_len_ternary: a conditional ternary optab, such as cond_len_fma_rev + DEF_INTERNAL_SIGNED_OPTAB_FN defines an internal function that maps to one of two optabs, depending on the signedness of an input. SIGNED_OPTAB and UNSIGNED_OPTAB are the optabs for signed and @@ -248,6 +252,40 @@ DEF_INTERNAL_OPTAB_FN (COND_FNMS, ECF_CONST, cond_fnms, cond_ternary) DEF_INTERNAL_OPTAB_FN (COND_NEG, ECF_CONST, cond_neg, cond_unary) +DEF_INTERNAL_OPTAB_FN (COND_LEN_ADD, ECF_CONST, cond_len_add, cond_len_binary) +DEF_INTERNAL_OPTAB_FN (COND_LEN_SUB, ECF_CONST, cond_len_sub, cond_len_binary) +DEF_INTERNAL_OPTAB_FN (COND_LEN_MUL, ECF_CONST, cond_len_smul, cond_len_binary) +DEF_INTERNAL_SIGNED_OPTAB_FN (COND_LEN_DIV, ECF_CONST, first, cond_len_sdiv, + cond_len_udiv, cond_len_binary) +DEF_INTERNAL_SIGNED_OPTAB_FN (COND_LEN_MOD, ECF_CONST, first, cond_len_smod, + cond_len_umod, cond_len_binary) +DEF_INTERNAL_OPTAB_FN (COND_LEN_RDIV, ECF_CONST, cond_len_sdiv, cond_len_binary) +DEF_INTERNAL_SIGNED_OPTAB_FN (COND_LEN_MIN, ECF_CONST, first, cond_len_smin, + cond_len_umin, cond_len_binary) +DEF_INTERNAL_SIGNED_OPTAB_FN (COND_LEN_MAX, ECF_CONST, first, cond_len_smax, + cond_len_umax, cond_len_binary) +DEF_INTERNAL_OPTAB_FN (COND_LEN_FMIN, ECF_CONST, cond_len_fmin, cond_len_binary) +DEF_INTERNAL_OPTAB_FN (COND_LEN_FMAX, ECF_CONST, cond_len_fmax, cond_len_binary) +DEF_INTERNAL_OPTAB_FN (COND_LEN_AND, ECF_CONST | ECF_NOTHROW, cond_len_and, + cond_len_binary) +DEF_INTERNAL_OPTAB_FN (COND_LEN_IOR, ECF_CONST | ECF_NOTHROW, cond_len_ior, + cond_len_binary) +DEF_INTERNAL_OPTAB_FN (COND_LEN_XOR, ECF_CONST | ECF_NOTHROW, cond_len_xor, + cond_len_binary) +DEF_INTERNAL_OPTAB_FN (COND_LEN_SHL, ECF_CONST | ECF_NOTHROW, cond_len_ashl, + cond_len_binary) +DEF_INTERNAL_SIGNED_OPTAB_FN (COND_LEN_SHR, ECF_CONST | ECF_NOTHROW, first, + cond_len_ashr, cond_len_lshr, cond_len_binary) + +DEF_INTERNAL_OPTAB_FN (COND_LEN_FMA, ECF_CONST, cond_len_fma, cond_len_ternary) +DEF_INTERNAL_OPTAB_FN (COND_LEN_FMS, ECF_CONST, cond_len_fms, cond_len_ternary) +DEF_INTERNAL_OPTAB_FN (COND_LEN_FNMA, ECF_CONST, cond_len_fnma, + cond_len_ternary) +DEF_INTERNAL_OPTAB_FN (COND_LEN_FNMS, ECF_CONST, cond_len_fnms, + cond_len_ternary) + +DEF_INTERNAL_OPTAB_FN (COND_LEN_NEG, ECF_CONST, cond_len_neg, cond_len_unary) + DEF_INTERNAL_OPTAB_FN (RSQRT, ECF_CONST, rsqrt, unary) DEF_INTERNAL_OPTAB_FN (REDUC_PLUS, ECF_CONST | ECF_NOTHROW, diff --git a/gcc/optabs.def b/gcc/optabs.def index 73c9a0c760f..3dae228fba6 100644 --- a/gcc/optabs.def +++ b/gcc/optabs.def @@ -254,6 +254,30 @@ OPTAB_D (cond_fms_optab, "cond_fms$a") OPTAB_D (cond_fnma_optab, "cond_fnma$a") OPTAB_D (cond_fnms_optab, "cond_fnms$a") OPTAB_D (cond_neg_optab, "cond_neg$a") +OPTAB_D (cond_len_add_optab, "cond_len_add$a") +OPTAB_D (cond_len_sub_optab, "cond_len_sub$a") +OPTAB_D (cond_len_smul_optab, "cond_len_mul$a") +OPTAB_D (cond_len_sdiv_optab, "cond_len_div$a") +OPTAB_D (cond_len_smod_optab, "cond_len_mod$a") +OPTAB_D (cond_len_udiv_optab, "cond_len_udiv$a") +OPTAB_D (cond_len_umod_optab, "cond_len_umod$a") +OPTAB_D (cond_len_and_optab, "cond_len_and$a") +OPTAB_D (cond_len_ior_optab, "cond_len_ior$a") +OPTAB_D (cond_len_xor_optab, "cond_len_xor$a") +OPTAB_D (cond_len_ashl_optab, "cond_len_ashl$a") +OPTAB_D (cond_len_ashr_optab, "cond_len_ashr$a") +OPTAB_D (cond_len_lshr_optab, "cond_len_lshr$a") +OPTAB_D (cond_len_smin_optab, "cond_len_smin$a") +OPTAB_D (cond_len_smax_optab, "cond_len_smax$a") +OPTAB_D (cond_len_umin_optab, "cond_len_umin$a") +OPTAB_D (cond_len_umax_optab, "cond_len_umax$a") +OPTAB_D (cond_len_fmin_optab, "cond_len_fmin$a") +OPTAB_D (cond_len_fmax_optab, "cond_len_fmax$a") +OPTAB_D (cond_len_fma_optab, "cond_len_fma$a") +OPTAB_D (cond_len_fms_optab, "cond_len_fms$a") +OPTAB_D (cond_len_fnma_optab, "cond_len_fnma$a") +OPTAB_D (cond_len_fnms_optab, "cond_len_fnms$a") +OPTAB_D (cond_len_neg_optab, "cond_len_neg$a") OPTAB_D (cmov_optab, "cmov$a6") OPTAB_D (cstore_optab, "cstore$a4") OPTAB_D (ctrap_optab, "ctrap$a4")