From patchwork Tue Oct 31 10:07:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "juzhe.zhong@rivai.ai" X-Patchwork-Id: 160055 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b90f:0:b0:403:3b70:6f57 with SMTP id t15csp127426vqg; Tue, 31 Oct 2023 03:08:29 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFcEUi1ljysQgCMKkwOPf/tIt/UHQ4LOORasXNgJpBZ5XwNT8HrhIna6zUlivl+XocfKEox X-Received: by 2002:ac8:5d12:0:b0:41e:24aa:81a0 with SMTP id f18-20020ac85d12000000b0041e24aa81a0mr18895951qtx.62.1698746909275; Tue, 31 Oct 2023 03:08:29 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1698746909; cv=pass; d=google.com; s=arc-20160816; b=oVusas8BlTRHZswMqT7o7wCB6Z9woJvG5BZqvhgcu8O+0dU4OgJ4ktgSW+Z1Eadp39 sE9C4d/kGKOogqcJLFqhXg+JUBSw2cUVhghgjSyxEepRP5yMl2lRbCZ6Ks3+hdTkCfRt Suzy/+XvdD63lfg1rDhS0FeyUXZ23WdQ8iSbZuCUbm9ijlNiR7iTWnmsxo17bPESugjz aKqpd/zYCiTkMbvuqOA5UD8hnoGgSY4vk4mpHEoLCJ8TjD8Ne8OsI0B+VaDXZFT9o1H5 jzPGvQ/pEl7pgcRaQIGTchcfIBz29fJZ+PccAUEmPcehEErrdPLAYHhgALbUFWLlPll3 HFPw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:feedback-id :content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:arc-filter:dmarc-filter:delivered-to; bh=udkHYiqPmV4TPXV6hSx0bA8mp8otW+yNq6JhGyOXElY=; fh=mlgczqbmeDnBuudaUD0OKNtrN3YEp3C0UptLjTcyUZ8=; b=Eb4peAWuHlh9avLWnc9WHnW20AsV7pQR5vA42mwMWGXeUsm4LnKvgfRfahHA3j+b8B LoZznYxFxBacsS9Vh5XxYCy1z0c/LGUdzYCfcUhBAhxFSfMAs+oPjSRmmJs1CuigGxyT XWq09T7Vpyc6AMHDp8L6Wqin1g2qHmPClJeFtnvII+LnbHDt959Vcd/AIcaHfDQu4Vu+ B46cv8Six9nMQEF538LhbJTall4FJihrkfynL/WkmWSTm89fYnIh/W25IxE53vb37KkP RE5N3+cAtilKvuqfsTUGvIhQ7Zgj+EuugW3PK5GO2rqSAkJoZTaoEDqNV/MqGfHqHkIu KMYA== ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (server2.sourceware.org. [8.43.85.97]) by mx.google.com with ESMTPS id c17-20020a05622a059100b0041cc5e9877dsi765545qtb.746.2023.10.31.03.08.29 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 31 Oct 2023 03:08:29 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) client-ip=8.43.85.97; Authentication-Results: mx.google.com; arc=pass (i=1); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 0A6B838582A1 for ; Tue, 31 Oct 2023 10:08:29 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtpbguseast1.qq.com (smtpbguseast1.qq.com [54.204.34.129]) by sourceware.org (Postfix) with ESMTPS id EDA3F3858D20 for ; Tue, 31 Oct 2023 10:08:00 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org EDA3F3858D20 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivai.ai Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivai.ai ARC-Filter: OpenARC Filter v1.0.0 sourceware.org EDA3F3858D20 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=54.204.34.129 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1698746885; cv=none; b=wMoSQUw1C+hMZ0T9d5X8cZ8aHIypP6UDTddlBxO1lIXUv31Jufj68zEoPklmGOtjseMuEdAT9n38Chm5TjFBP/UYgPOeZErwmKwPJXbmL8/mTvwINgfbH6hZXHatvfB91tJ6eGnSwFPlmPdu8bEi95/ENKtIavntakNvY924Gds= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1698746885; c=relaxed/simple; bh=xKpAha2dchQ2tfqqQ+8KuZFzFdbQaDV2YQfjzbpXh5I=; h=From:To:Subject:Date:Message-Id:MIME-Version; b=pzjiCbWkV8dhnhNpQBdY4HXLsK+2Fw7k0OYvI5xmRmV6yilmnVAMQyKYxt9RIHqUzvPueczvmSYn6i2dzPLyungW/kkFleQ7jpjxkmyUVoG6nJ2/HU4Y52pas+JLH+UAnQrPprg0dmWlZL9ZEhLX3DRgr7oBnJxlaqORaaU5Y5s= ARC-Authentication-Results: i=1; server2.sourceware.org X-QQ-mid: bizesmtp85t1698746872t03d4pr2 Received: from rios-cad122.hadoop.rioslab.org ( [58.60.1.26]) by bizesmtp.qq.com (ESMTP) with id ; Tue, 31 Oct 2023 18:07:51 +0800 (CST) X-QQ-SSF: 01400000000000G0V000000A0000000 X-QQ-FEAT: q+EIYT+FhZr0vR7rrySoxFWgeysZTnzcbJw/YQnWMH5F6gtzKkbt+q2ygT4XH 1wO1AglV0VyHK0KKw2bhDARLUJlRGMpqk3gqZqMaZm/rVc+gW1+yCZNj+53cf4hqlpSvluL 3nMiHfa0qXu7rJT+nK5A47QAc0zYZO1rVjgtWki0EPKVmDha9FowaT5R5ZDhucxa5acunPz VaJusYQKCnC+QwplciF0sauLjvYJGkVk4XL87JEf8bkjadYS1/lPGVNEy4BcafTgS3JxuDe 1M8D7VwfyEsF7FzIHIw48hKx9HC0jYZutO4oDoSUYwf7nL7/yLR4xnM1kda/9YKgI3K/yeU JyT0N+KcQYXVmS/iqgXaDh3u7js7g49xQxH/hR7f+UGy/DFMBJrmEAmwcfuwGGvy8gCe6Yq ScUnyJoKmhEzZKOHJ0m+Ww== X-QQ-GoodBg: 2 X-BIZMAIL-ID: 4629385623853714529 From: Juzhe-Zhong To: gcc-patches@gcc.gnu.org Cc: richard.sandiford@arm.com, rguenther@suse.de, jeffreyalaw@gmail.com, Juzhe-Zhong Subject: [PATCH] VECT: Support mask_len_strided_load/mask_len_strided_store in loop vectorize Date: Tue, 31 Oct 2023 18:07:45 +0800 Message-Id: <20231031100745.2356816-1-juzhe.zhong@rivai.ai> X-Mailer: git-send-email 2.36.3 MIME-Version: 1.0 X-QQ-SENDSIZE: 520 Feedback-ID: bizesmtp:rivai.ai:qybglogicsvrgz:qybglogicsvrgz7a-one-0 X-Spam-Status: No, score=-10.4 required=5.0 tests=BAYES_00, GIT_PATCH_0, KAM_DMARC_STATUS, RCVD_IN_BARRACUDACENTRAL, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781265239217356625 X-GMAIL-MSGID: 1781265239217356625 This patch support loop vectorizer generate direct strided load/store IFN if targets enable it. Note that this patch provide the ability that target enabling strided load/store but without gather/scatter can vectorize stride memory access. gcc/ChangeLog: * optabs-query.cc (supports_vec_gather_load_p): Support strided load/store. (supports_vec_scatter_store_p): Ditto. * optabs-query.h (supports_vec_gather_load_p): Ditto. (supports_vec_scatter_store_p): Ditto. * tree-vect-data-refs.cc (vect_gather_scatter_fn_p): Ditto. (vect_check_gather_scatter): Ditto. * tree-vect-stmts.cc (check_load_store_for_partial_vectors): Ditto. (vect_truncate_gather_scatter_offset): Ditto. (vect_use_strided_gather_scatters_p): Ditto. (vect_get_strided_load_store_ops): Ditto. (vectorizable_store): Ditto. (vectorizable_load): Ditto. * tree-vectorizer.h (vect_gather_scatter_fn_p): Ditto. --- gcc/optabs-query.cc | 27 ++++++++++----- gcc/optabs-query.h | 4 +-- gcc/tree-vect-data-refs.cc | 71 ++++++++++++++++++++++++++++---------- gcc/tree-vect-stmts.cc | 46 +++++++++++++++++------- gcc/tree-vectorizer.h | 3 +- 5 files changed, 109 insertions(+), 42 deletions(-) diff --git a/gcc/optabs-query.cc b/gcc/optabs-query.cc index 947ccef218c..ea594baf15d 100644 --- a/gcc/optabs-query.cc +++ b/gcc/optabs-query.cc @@ -670,14 +670,19 @@ supports_vec_convert_optab_p (optab op, machine_mode mode) for at least one vector mode. */ bool -supports_vec_gather_load_p (machine_mode mode) +supports_vec_gather_load_p (machine_mode mode, bool strided_p) { if (!this_fn_optabs->supports_vec_gather_load[mode]) this_fn_optabs->supports_vec_gather_load[mode] = (supports_vec_convert_optab_p (gather_load_optab, mode) - || supports_vec_convert_optab_p (mask_gather_load_optab, mode) - || supports_vec_convert_optab_p (mask_len_gather_load_optab, mode) - ? 1 : -1); + || supports_vec_convert_optab_p (mask_gather_load_optab, mode) + || supports_vec_convert_optab_p (mask_len_gather_load_optab, mode) + || (strided_p + && convert_optab_handler (mask_len_strided_load_optab, mode, + Pmode) + != CODE_FOR_nothing) + ? 1 + : -1); return this_fn_optabs->supports_vec_gather_load[mode] > 0; } @@ -687,14 +692,20 @@ supports_vec_gather_load_p (machine_mode mode) for at least one vector mode. */ bool -supports_vec_scatter_store_p (machine_mode mode) +supports_vec_scatter_store_p (machine_mode mode, bool strided_p) { if (!this_fn_optabs->supports_vec_scatter_store[mode]) this_fn_optabs->supports_vec_scatter_store[mode] = (supports_vec_convert_optab_p (scatter_store_optab, mode) - || supports_vec_convert_optab_p (mask_scatter_store_optab, mode) - || supports_vec_convert_optab_p (mask_len_scatter_store_optab, mode) - ? 1 : -1); + || supports_vec_convert_optab_p (mask_scatter_store_optab, mode) + || supports_vec_convert_optab_p (mask_len_scatter_store_optab, + mode) + || (strided_p + && convert_optab_handler (mask_len_strided_store_optab, mode, + Pmode) + != CODE_FOR_nothing) + ? 1 + : -1); return this_fn_optabs->supports_vec_scatter_store[mode] > 0; } diff --git a/gcc/optabs-query.h b/gcc/optabs-query.h index 920eb6a1b67..7c22edc5a78 100644 --- a/gcc/optabs-query.h +++ b/gcc/optabs-query.h @@ -191,8 +191,8 @@ bool can_compare_and_swap_p (machine_mode, bool); bool can_atomic_exchange_p (machine_mode, bool); bool can_atomic_load_p (machine_mode); bool lshift_cheap_p (bool); -bool supports_vec_gather_load_p (machine_mode = E_VOIDmode); -bool supports_vec_scatter_store_p (machine_mode = E_VOIDmode); +bool supports_vec_gather_load_p (machine_mode = E_VOIDmode, bool = false); +bool supports_vec_scatter_store_p (machine_mode = E_VOIDmode, bool = false); bool can_vec_extract (machine_mode, machine_mode); /* Version of find_widening_optab_handler_and_mode that operates on diff --git a/gcc/tree-vect-data-refs.cc b/gcc/tree-vect-data-refs.cc index d5c9c4a11c2..d374849b0a7 100644 --- a/gcc/tree-vect-data-refs.cc +++ b/gcc/tree-vect-data-refs.cc @@ -3913,9 +3913,9 @@ vect_prune_runtime_alias_test_list (loop_vec_info loop_vinfo) *IFN_OUT and the vector type for the offset in *OFFSET_VECTYPE_OUT. */ bool -vect_gather_scatter_fn_p (vec_info *vinfo, bool read_p, bool masked_p, - tree vectype, tree memory_type, tree offset_type, - int scale, internal_fn *ifn_out, +vect_gather_scatter_fn_p (vec_info *vinfo, bool strided_p, bool read_p, + bool masked_p, tree vectype, tree memory_type, + tree offset_type, int scale, internal_fn *ifn_out, tree *offset_vectype_out) { unsigned int memory_bits = tree_to_uhwi (TYPE_SIZE (memory_type)); @@ -3926,7 +3926,7 @@ vect_gather_scatter_fn_p (vec_info *vinfo, bool read_p, bool masked_p, return false; /* Work out which function we need. */ - internal_fn ifn, alt_ifn, alt_ifn2; + internal_fn ifn, alt_ifn, alt_ifn2, alt_ifn3; if (read_p) { ifn = masked_p ? IFN_MASK_GATHER_LOAD : IFN_GATHER_LOAD; @@ -3935,6 +3935,12 @@ vect_gather_scatter_fn_p (vec_info *vinfo, bool read_p, bool masked_p, use MASK_LEN_GATHER_LOAD regardless whether len and mask are valid or not. */ alt_ifn2 = IFN_MASK_LEN_GATHER_LOAD; + /* When target supports MASK_LEN_STRIDED_LOAD, we can relax the + restrictions around the relationship of the vector offset type + to the loaded by using a gather load with strided access. + E.g. a "gather" of N bytes with a 64-bit stride would in principle + be possible without needing an Nx64-bit vector offset type. */ + alt_ifn3 = IFN_MASK_LEN_STRIDED_LOAD; } else { @@ -3944,6 +3950,12 @@ vect_gather_scatter_fn_p (vec_info *vinfo, bool read_p, bool masked_p, use MASK_LEN_SCATTER_STORE regardless whether len and mask are valid or not. */ alt_ifn2 = IFN_MASK_LEN_SCATTER_STORE; + /* When target supports MASK_LEN_STRIDED_STORE, we can relax the + restrictions around the relationship of the vector offset type + to the stored by using a scatter store with strided access. + E.g. a "scatter" of N bytes with a 64-bit stride would in principle + be possible without needing an Nx64-bit vector offset type. */ + alt_ifn3 = IFN_MASK_LEN_STRIDED_STORE; } for (;;) @@ -3953,8 +3965,20 @@ vect_gather_scatter_fn_p (vec_info *vinfo, bool read_p, bool masked_p, return false; /* Test whether the target supports this combination. */ - if (internal_gather_scatter_fn_supported_p (ifn, vectype, memory_type, - offset_vectype, scale)) + /* We don't need to check whether target supports gather/scatter IFN + with expected vector offset for gather/scatter with a strided access + when target itself support strided load/store IFN. */ + if (strided_p + && internal_strided_fn_supported_p (alt_ifn3, vectype, offset_type, + scale)) + { + *ifn_out = alt_ifn3; + *offset_vectype_out = offset_vectype; + return true; + } + else if (internal_gather_scatter_fn_supported_p (ifn, vectype, + memory_type, + offset_vectype, scale)) { *ifn_out = ifn; *offset_vectype_out = offset_vectype; @@ -4047,9 +4071,12 @@ vect_check_gather_scatter (stmt_vec_info stmt_info, loop_vec_info loop_vinfo, /* True if we should aim to use internal functions rather than built-in functions. */ - bool use_ifn_p = (DR_IS_READ (dr) - ? supports_vec_gather_load_p (TYPE_MODE (vectype)) - : supports_vec_scatter_store_p (TYPE_MODE (vectype))); + bool use_ifn_p + = (DR_IS_READ (dr) + ? supports_vec_gather_load_p (TYPE_MODE (vectype), + STMT_VINFO_STRIDED_P (stmt_info)) + : supports_vec_scatter_store_p (TYPE_MODE (vectype), + STMT_VINFO_STRIDED_P (stmt_info))); base = DR_REF (dr); /* For masked loads/stores, DR_REF (dr) is an artificial MEM_REF, @@ -4196,13 +4223,17 @@ vect_check_gather_scatter (stmt_vec_info stmt_info, loop_vec_info loop_vinfo, /* Only treat this as a scaling operation if the target supports it for at least some offset type. */ if (use_ifn_p - && !vect_gather_scatter_fn_p (loop_vinfo, DR_IS_READ (dr), - masked_p, vectype, memory_type, + && !vect_gather_scatter_fn_p (loop_vinfo, + STMT_VINFO_STRIDED_P (stmt_info), + DR_IS_READ (dr), masked_p, + vectype, memory_type, signed_char_type_node, new_scale, &ifn, &offset_vectype) - && !vect_gather_scatter_fn_p (loop_vinfo, DR_IS_READ (dr), - masked_p, vectype, memory_type, + && !vect_gather_scatter_fn_p (loop_vinfo, + STMT_VINFO_STRIDED_P (stmt_info), + DR_IS_READ (dr), masked_p, + vectype, memory_type, unsigned_char_type_node, new_scale, &ifn, &offset_vectype)) @@ -4225,8 +4256,10 @@ vect_check_gather_scatter (stmt_vec_info stmt_info, loop_vec_info loop_vinfo, if (use_ifn_p && TREE_CODE (off) == SSA_NAME && !POINTER_TYPE_P (TREE_TYPE (off)) - && vect_gather_scatter_fn_p (loop_vinfo, DR_IS_READ (dr), - masked_p, vectype, memory_type, + && vect_gather_scatter_fn_p (loop_vinfo, + STMT_VINFO_STRIDED_P (stmt_info), + DR_IS_READ (dr), masked_p, + vectype, memory_type, TREE_TYPE (off), scale, &ifn, &offset_vectype)) break; @@ -4280,9 +4313,11 @@ vect_check_gather_scatter (stmt_vec_info stmt_info, loop_vec_info loop_vinfo, if (use_ifn_p) { - if (!vect_gather_scatter_fn_p (loop_vinfo, DR_IS_READ (dr), masked_p, - vectype, memory_type, offtype, scale, - &ifn, &offset_vectype)) + if (!vect_gather_scatter_fn_p (loop_vinfo, + STMT_VINFO_STRIDED_P (stmt_info), + DR_IS_READ (dr), masked_p, vectype, + memory_type, offtype, scale, &ifn, + &offset_vectype)) ifn = IFN_LAST; decl = NULL_TREE; } diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc index a9200767f67..8ff06bd3acb 100644 --- a/gcc/tree-vect-stmts.cc +++ b/gcc/tree-vect-stmts.cc @@ -1506,10 +1506,15 @@ check_load_store_for_partial_vectors (loop_vec_info loop_vinfo, tree vectype, internal_fn len_ifn = (is_load ? IFN_MASK_LEN_GATHER_LOAD : IFN_MASK_LEN_SCATTER_STORE); - if (internal_gather_scatter_fn_supported_p (len_ifn, vectype, - gs_info->memory_type, - gs_info->offset_vectype, - gs_info->scale)) + if (internal_strided_fn_p (gs_info->ifn) + && internal_strided_fn_supported_p (gs_info->ifn, vectype, + TREE_TYPE (gs_info->offset), + gs_info->scale)) + vect_record_loop_len (loop_vinfo, lens, nvectors, vectype, 1); + else if (internal_gather_scatter_fn_supported_p (len_ifn, vectype, + gs_info->memory_type, + gs_info->offset_vectype, + gs_info->scale)) vect_record_loop_len (loop_vinfo, lens, nvectors, vectype, 1); else if (internal_gather_scatter_fn_supported_p (ifn, vectype, gs_info->memory_type, @@ -1693,8 +1698,10 @@ vect_truncate_gather_scatter_offset (stmt_vec_info stmt_info, /* See whether the target supports the operation with an offset no narrower than OFFSET_TYPE. */ tree memory_type = TREE_TYPE (DR_REF (dr)); - if (!vect_gather_scatter_fn_p (loop_vinfo, DR_IS_READ (dr), masked_p, - vectype, memory_type, offset_type, scale, + if (!vect_gather_scatter_fn_p (loop_vinfo, + STMT_VINFO_STRIDED_P (stmt_info), + DR_IS_READ (dr), masked_p, vectype, + memory_type, offset_type, scale, &gs_info->ifn, &gs_info->offset_vectype) || gs_info->ifn == IFN_LAST) continue; @@ -1734,6 +1741,15 @@ vect_use_strided_gather_scatters_p (stmt_vec_info stmt_info, || gs_info->ifn == IFN_LAST) return vect_truncate_gather_scatter_offset (stmt_info, loop_vinfo, masked_p, gs_info); + else if (internal_strided_fn_p (gs_info->ifn)) + { + if (dump_enabled_p ()) + dump_printf_loc (MSG_NOTE, vect_location, + "using strided IFN for strided/grouped access," + " scale = %d\n", + gs_info->scale); + return true; + } tree old_offset_type = TREE_TYPE (gs_info->offset); tree new_offset_type = TREE_TYPE (gs_info->offset_vectype); @@ -3012,9 +3028,13 @@ vect_get_strided_load_store_ops (stmt_vec_info stmt_info, ssize_int (gs_info->scale)); step = fold_convert (offset_type, step); - /* Create {0, X, X*2, X*3, ...}. */ - tree offset = fold_build2 (VEC_SERIES_EXPR, gs_info->offset_vectype, - build_zero_cst (offset_type), step); + tree offset; + if (internal_strided_fn_p (gs_info->ifn)) + offset = step; + else + /* Create {0, X, X*2, X*3, ...}. */ + offset = fold_build2 (VEC_SERIES_EXPR, gs_info->offset_vectype, + build_zero_cst (offset_type), step); *vec_offset = cse_and_gimplify_to_preheader (loop_vinfo, offset); } @@ -9125,7 +9145,7 @@ vectorizable_store (vec_info *vinfo, vec_offset = vec_offsets[j]; tree scale = size_int (gs_info.scale); - if (gs_info.ifn == IFN_MASK_LEN_SCATTER_STORE) + if (internal_fn_len_index (gs_info.ifn) >= 0) { if (loop_lens) final_len = vect_get_loop_len (loop_vinfo, gsi, loop_lens, @@ -9145,7 +9165,7 @@ vectorizable_store (vec_info *vinfo, gcall *call; if (final_len && final_mask) - call = gimple_build_call_internal (IFN_MASK_LEN_SCATTER_STORE, + call = gimple_build_call_internal (gs_info.ifn, 7, dataref_ptr, vec_offset, scale, vec_oprnd, final_mask, final_len, bias); @@ -10949,7 +10969,7 @@ vectorizable_load (vec_info *vinfo, tree zero = build_zero_cst (vectype); tree scale = size_int (gs_info.scale); - if (gs_info.ifn == IFN_MASK_LEN_GATHER_LOAD) + if (internal_fn_len_index (gs_info.ifn) >= 0) { if (loop_lens) final_len @@ -10973,7 +10993,7 @@ vectorizable_load (vec_info *vinfo, gcall *call; if (final_len && final_mask) call - = gimple_build_call_internal (IFN_MASK_LEN_GATHER_LOAD, 7, + = gimple_build_call_internal (gs_info.ifn, 7, dataref_ptr, vec_offset, scale, zero, final_mask, final_len, bias); diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h index a4043e4a656..76bf3aa14b4 100644 --- a/gcc/tree-vectorizer.h +++ b/gcc/tree-vectorizer.h @@ -2309,7 +2309,8 @@ extern opt_result vect_analyze_data_refs_alignment (loop_vec_info); extern bool vect_slp_analyze_instance_alignment (vec_info *, slp_instance); extern opt_result vect_analyze_data_ref_accesses (vec_info *, vec *); extern opt_result vect_prune_runtime_alias_test_list (loop_vec_info); -extern bool vect_gather_scatter_fn_p (vec_info *, bool, bool, tree, tree, +extern bool vect_gather_scatter_fn_p (vec_info *, + bool, bool, bool, tree, tree, tree, int, internal_fn *, tree *); extern bool vect_check_gather_scatter (stmt_vec_info, loop_vec_info, gather_scatter_info *);