From patchwork Thu Aug 17 06:22:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kewen.Lin" X-Patchwork-Id: 135849 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b82d:0:b0:3f2:4152:657d with SMTP id z13csp492876vqi; Wed, 16 Aug 2023 23:23:24 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFE2VSdEPhC4vzydxD9Vk/1WMPZINRI3f0/ZPVdqesW6+iHAHTr/ELCiaLUbBHCUSP7GHoX X-Received: by 2002:a17:906:74d7:b0:99c:3b4:940c with SMTP id z23-20020a17090674d700b0099c03b4940cmr2988335ejl.7.1692253403937; Wed, 16 Aug 2023 23:23:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1692253403; cv=none; d=google.com; s=arc-20160816; b=bpbVVuswAuPBjncvu2CdHbytNCyGJtIvse8iEmWV7lXQbSZS95xQSfk7bHr74O55pE SOC7gqttBzyik34s6sJ5iyz4B+cMrPx+revgn3mJiKs9xMHVT++jzWWr0J7qexadfsIT o6pXUCBX3ohsd9kuRkr4761fohw2QSn0T+oEZixa97cOBYEX+VqQqTMF/apbuiAY0zpO CMZwomSZxtFMnt+Z+UftKclu2P4HSwaXExYRE8/0xNL5wW1kal/CqA/SKLWD0AlC0miA tdm3w2BG+qpT8YKWVl21ysTq204tn+vKQbi2s0TXN/z7tGG7up66OJt5Tlx8t4yduFrj ng8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:reply-to:from:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence :content-transfer-encoding:cc:to:subject:content-language:user-agent :mime-version:date:message-id:dmarc-filter:delivered-to :dkim-signature:dkim-filter; bh=erYDYSTLI+R92G3B5X127onXMxcegXu+OSAhzUkkbR0=; fh=igBTPjHNdy5qHek4vewG2hwX+gxODq9WIce1cDBLf4Y=; b=g/6tQ2DSX5jC0nXa7x0l5aBAoh73CrqaKeyK4e53ntDIuQ4tk3+Cii6WW0FS8UwN49 ymN+V2aKAM5ez4wjgaWgcgrvpE6Xe9S0NYkpzKd/WhFzY/wdcfJdJkueiqQRzXhS2AhW unpGAqPjMg9+pgSdg+ijrKHjp1O3oCUxCZKF6PZSx/DsnignCyCfSoiOMH10gShNqTG5 x00PQXM6s1PWvxn3z2jNKfUElCb5K9S5u4n67UV4Ga4y8KBB5Ljv+/8j+A0oCAt7cxH7 deGWPrLghDvVY7fN2g6HOThMaMWnB4TZ/7bE1v98J52/b0JAY0wVx/hrIGnCp8U5BLUb NhDw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=p02kpKwy; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gnu.org Received: from server2.sourceware.org (server2.sourceware.org. [2620:52:3:1:0:246e:9693:128c]) by mx.google.com with ESMTPS id qt5-20020a170906ece500b00997e71d037csi12656167ejb.618.2023.08.16.23.23.23 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Aug 2023 23:23:23 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) client-ip=2620:52:3:1:0:246e:9693:128c; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=p02kpKwy; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gnu.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 5C3B2385C6F2 for ; Thu, 17 Aug 2023 06:23:21 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 5C3B2385C6F2 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1692253401; bh=erYDYSTLI+R92G3B5X127onXMxcegXu+OSAhzUkkbR0=; h=Date:Subject:To:Cc:List-Id:List-Unsubscribe:List-Archive: List-Post:List-Help:List-Subscribe:From:Reply-To:From; b=p02kpKwy83C8D4JaiEZDMX6cSvVEVHl54OpIDVh7McGyhVCHsymR/utAHMR6zTIay nsvkmhk5YkA/wVyz77VmG9Rzk9bgM9+IX8aGQCm7jytHKoycfNDW8E1AcVu/5qB3vI pD6MO/Uwn5jJLEQrZYbk4QWj/6e0Fy9I+7SXceuI= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by sourceware.org (Postfix) with ESMTPS id EFB773858D28 for ; Thu, 17 Aug 2023 06:22:36 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org EFB773858D28 Received: from pps.filterd (m0360083.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37H6JTWm028959; Thu, 17 Aug 2023 06:22:33 GMT Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3she2wg6a7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 17 Aug 2023 06:22:33 +0000 Received: from m0360083.ppops.net (m0360083.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 37H6D8ht005680; Thu, 17 Aug 2023 06:22:32 GMT Received: from ppma23.wdc07v.mail.ibm.com (5d.69.3da9.ip4.static.sl-reverse.com [169.61.105.93]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3she2wg69s-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 17 Aug 2023 06:22:32 +0000 Received: from pps.filterd (ppma23.wdc07v.mail.ibm.com [127.0.0.1]) by ppma23.wdc07v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 37H4pCfo007871; Thu, 17 Aug 2023 06:22:31 GMT Received: from smtprelay07.fra02v.mail.ibm.com ([9.218.2.229]) by ppma23.wdc07v.mail.ibm.com (PPS) with ESMTPS id 3senwkkjpy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 17 Aug 2023 06:22:31 +0000 Received: from smtpav06.fra02v.mail.ibm.com (smtpav06.fra02v.mail.ibm.com [10.20.54.105]) by smtprelay07.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 37H6MTpd56623508 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Aug 2023 06:22:29 GMT Received: from smtpav06.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 673E42004B; Thu, 17 Aug 2023 06:22:29 +0000 (GMT) Received: from smtpav06.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B3F7220049; Thu, 17 Aug 2023 06:22:27 +0000 (GMT) Received: from [9.197.250.153] (unknown [9.197.250.153]) by smtpav06.fra02v.mail.ibm.com (Postfix) with ESMTP; Thu, 17 Aug 2023 06:22:27 +0000 (GMT) Message-ID: Date: Thu, 17 Aug 2023 14:22:26 +0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Thunderbird/91.6.1 Content-Language: en-US Subject: [PATCH] vect: Factor out the handling on scatter store having gs_info.decl To: GCC Patches Cc: Richard Biener , Richard Sandiford X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: g5ghfoXrAvaVTcJGNXELVOweYBcW4xHk X-Proofpoint-GUID: OzqdT9eBrqKRn-qOhieNH6tUZk0Pm-8n X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-17_03,2023-08-15_02,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 suspectscore=0 impostorscore=0 clxscore=1015 mlxscore=0 lowpriorityscore=0 malwarescore=0 spamscore=0 adultscore=0 mlxlogscore=848 bulkscore=0 priorityscore=1501 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2306200000 definitions=main-2308170052 X-Spam-Status: No, score=-11.7 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_MSPIKE_H5, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: "Kewen.Lin via Gcc-patches" From: "Kewen.Lin" Reply-To: "Kewen.Lin" Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org Sender: "Gcc-patches" X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1774456305343447116 X-GMAIL-MSGID: 1774456305343447116 Hi, Similar to the existing function vect_build_gather_load_calls, this patch is to factor out the handling on scatter store having gs_info.decl to vect_build_scatter_store_calls which is a new function. It also does some minor refactoring like moving some variables' declarations close to their uses and restrict the scope for some of them etc. It's a pre-patch for upcoming vectorizable_store re-structuring for costing. Bootstrapped and regtested on x86_64-redhat-linux, aarch64-linux-gnu and powerpc64{,le}-linux-gnu. Is it ok for trunk? BR, Kewen ----- gcc/ChangeLog: * tree-vect-stmts.cc (vect_build_scatter_store_calls): New, factor out from ... (vectorizable_store): ... here. --- gcc/tree-vect-stmts.cc | 411 +++++++++++++++++++++-------------------- 1 file changed, 212 insertions(+), 199 deletions(-) -- 2.31.1 diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc index cd8e0a76374..f8a904de503 100644 --- a/gcc/tree-vect-stmts.cc +++ b/gcc/tree-vect-stmts.cc @@ -2989,6 +2989,216 @@ vect_build_gather_load_calls (vec_info *vinfo, stmt_vec_info stmt_info, *vec_stmt = STMT_VINFO_VEC_STMTS (stmt_info)[0]; } +/* Build a scatter store call while vectorizing STMT_INFO. Insert new + instructions before GSI and add them to VEC_STMT. GS_INFO describes + the scatter store operation. If the store is conditional, MASK is the + unvectorized condition, otherwise MASK is null. */ + +static void +vect_build_scatter_store_calls (vec_info *vinfo, stmt_vec_info stmt_info, + gimple_stmt_iterator *gsi, gimple **vec_stmt, + gather_scatter_info *gs_info, tree mask) +{ + loop_vec_info loop_vinfo = dyn_cast (vinfo); + tree vectype = STMT_VINFO_VECTYPE (stmt_info); + poly_uint64 nunits = TYPE_VECTOR_SUBPARTS (vectype); + int ncopies = vect_get_num_copies (loop_vinfo, vectype); + enum { NARROW, NONE, WIDEN } modifier; + poly_uint64 scatter_off_nunits + = TYPE_VECTOR_SUBPARTS (gs_info->offset_vectype); + + tree perm_mask = NULL_TREE, mask_halfvectype = NULL_TREE; + if (known_eq (nunits, scatter_off_nunits)) + modifier = NONE; + else if (known_eq (nunits * 2, scatter_off_nunits)) + { + modifier = WIDEN; + + /* Currently gathers and scatters are only supported for + fixed-length vectors. */ + unsigned int count = scatter_off_nunits.to_constant (); + vec_perm_builder sel (count, count, 1); + for (unsigned i = 0; i < (unsigned int) count; ++i) + sel.quick_push (i | (count / 2)); + + vec_perm_indices indices (sel, 1, count); + perm_mask = vect_gen_perm_mask_checked (gs_info->offset_vectype, indices); + gcc_assert (perm_mask != NULL_TREE); + } + else if (known_eq (nunits, scatter_off_nunits * 2)) + { + modifier = NARROW; + + /* Currently gathers and scatters are only supported for + fixed-length vectors. */ + unsigned int count = nunits.to_constant (); + vec_perm_builder sel (count, count, 1); + for (unsigned i = 0; i < (unsigned int) count; ++i) + sel.quick_push (i | (count / 2)); + + vec_perm_indices indices (sel, 2, count); + perm_mask = vect_gen_perm_mask_checked (vectype, indices); + gcc_assert (perm_mask != NULL_TREE); + ncopies *= 2; + + if (mask) + mask_halfvectype = truth_type_for (gs_info->offset_vectype); + } + else + gcc_unreachable (); + + tree rettype = TREE_TYPE (TREE_TYPE (gs_info->decl)); + tree arglist = TYPE_ARG_TYPES (TREE_TYPE (gs_info->decl)); + tree ptrtype = TREE_VALUE (arglist); arglist = TREE_CHAIN (arglist); + tree masktype = TREE_VALUE (arglist); arglist = TREE_CHAIN (arglist); + tree idxtype = TREE_VALUE (arglist); arglist = TREE_CHAIN (arglist); + tree srctype = TREE_VALUE (arglist); arglist = TREE_CHAIN (arglist); + tree scaletype = TREE_VALUE (arglist); + + gcc_checking_assert (TREE_CODE (masktype) == INTEGER_TYPE + && TREE_CODE (rettype) == VOID_TYPE); + + tree ptr = fold_convert (ptrtype, gs_info->base); + if (!is_gimple_min_invariant (ptr)) + { + gimple_seq seq; + ptr = force_gimple_operand (ptr, &seq, true, NULL_TREE); + class loop *loop = LOOP_VINFO_LOOP (loop_vinfo); + edge pe = loop_preheader_edge (loop); + basic_block new_bb = gsi_insert_seq_on_edge_immediate (pe, seq); + gcc_assert (!new_bb); + } + + tree mask_arg = NULL_TREE; + if (mask == NULL_TREE) + { + mask_arg = build_int_cst (masktype, -1); + mask_arg = vect_init_vector (vinfo, stmt_info, mask_arg, masktype, NULL); + } + + tree scale = build_int_cst (scaletype, gs_info->scale); + + auto_vec vec_oprnds0; + auto_vec vec_oprnds1; + auto_vec vec_masks; + if (mask) + { + tree mask_vectype = truth_type_for (vectype); + vect_get_vec_defs_for_operand (vinfo, stmt_info, + modifier == NARROW ? ncopies / 2 : ncopies, + mask, &vec_masks, mask_vectype); + } + vect_get_vec_defs_for_operand (vinfo, stmt_info, + modifier == WIDEN ? ncopies / 2 : ncopies, + gs_info->offset, &vec_oprnds0); + tree op = vect_get_store_rhs (stmt_info); + vect_get_vec_defs_for_operand (vinfo, stmt_info, + modifier == NARROW ? ncopies / 2 : ncopies, op, + &vec_oprnds1); + + tree vec_oprnd0 = NULL_TREE, vec_oprnd1 = NULL_TREE; + tree mask_op = NULL_TREE; + tree src, vec_mask; + for (int j = 0; j < ncopies; ++j) + { + if (modifier == WIDEN) + { + if (j & 1) + op = permute_vec_elements (vinfo, vec_oprnd0, vec_oprnd0, perm_mask, + stmt_info, gsi); + else + op = vec_oprnd0 = vec_oprnds0[j / 2]; + src = vec_oprnd1 = vec_oprnds1[j]; + if (mask) + mask_op = vec_mask = vec_masks[j]; + } + else if (modifier == NARROW) + { + if (j & 1) + src = permute_vec_elements (vinfo, vec_oprnd1, vec_oprnd1, + perm_mask, stmt_info, gsi); + else + src = vec_oprnd1 = vec_oprnds1[j / 2]; + op = vec_oprnd0 = vec_oprnds0[j]; + if (mask) + mask_op = vec_mask = vec_masks[j / 2]; + } + else + { + op = vec_oprnd0 = vec_oprnds0[j]; + src = vec_oprnd1 = vec_oprnds1[j]; + if (mask) + mask_op = vec_mask = vec_masks[j]; + } + + if (!useless_type_conversion_p (srctype, TREE_TYPE (src))) + { + gcc_assert (known_eq (TYPE_VECTOR_SUBPARTS (TREE_TYPE (src)), + TYPE_VECTOR_SUBPARTS (srctype))); + tree var = vect_get_new_ssa_name (srctype, vect_simple_var); + src = build1 (VIEW_CONVERT_EXPR, srctype, src); + gassign *new_stmt = gimple_build_assign (var, VIEW_CONVERT_EXPR, src); + vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi); + src = var; + } + + if (!useless_type_conversion_p (idxtype, TREE_TYPE (op))) + { + gcc_assert (known_eq (TYPE_VECTOR_SUBPARTS (TREE_TYPE (op)), + TYPE_VECTOR_SUBPARTS (idxtype))); + tree var = vect_get_new_ssa_name (idxtype, vect_simple_var); + op = build1 (VIEW_CONVERT_EXPR, idxtype, op); + gassign *new_stmt = gimple_build_assign (var, VIEW_CONVERT_EXPR, op); + vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi); + op = var; + } + + if (mask) + { + tree utype; + mask_arg = mask_op; + if (modifier == NARROW) + { + tree var + = vect_get_new_ssa_name (mask_halfvectype, vect_simple_var); + gassign *new_stmt + = gimple_build_assign (var, + (j & 1) ? VEC_UNPACK_HI_EXPR + : VEC_UNPACK_LO_EXPR, + mask_op); + vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi); + mask_arg = var; + } + tree optype = TREE_TYPE (mask_arg); + if (TYPE_MODE (masktype) == TYPE_MODE (optype)) + utype = masktype; + else + utype = lang_hooks.types.type_for_mode (TYPE_MODE (optype), 1); + tree var = vect_get_new_ssa_name (utype, vect_scalar_var); + mask_arg = build1 (VIEW_CONVERT_EXPR, utype, mask_arg); + gassign *new_stmt + = gimple_build_assign (var, VIEW_CONVERT_EXPR, mask_arg); + vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi); + mask_arg = var; + if (!useless_type_conversion_p (masktype, utype)) + { + gcc_assert (TYPE_PRECISION (utype) <= TYPE_PRECISION (masktype)); + tree var = vect_get_new_ssa_name (masktype, vect_scalar_var); + new_stmt = gimple_build_assign (var, NOP_EXPR, mask_arg); + vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi); + mask_arg = var; + } + } + + gcall *new_stmt + = gimple_build_call (gs_info->decl, 5, ptr, mask_arg, op, src, scale); + vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi); + + STMT_VINFO_VEC_STMTS (stmt_info).safe_push (new_stmt); + } + *vec_stmt = STMT_VINFO_VEC_STMTS (stmt_info)[0]; +} + /* Prepare the base and offset in GS_INFO for vectorization. Set *DATAREF_PTR to the loop-invariant base address and *VEC_OFFSET to the vectorized offset argument for the first copy of STMT_INFO. @@ -8212,205 +8422,8 @@ vectorizable_store (vec_info *vinfo, if (memory_access_type == VMAT_GATHER_SCATTER && gs_info.decl) { - tree vec_oprnd0 = NULL_TREE, vec_oprnd1 = NULL_TREE, src; - tree arglist = TYPE_ARG_TYPES (TREE_TYPE (gs_info.decl)); - tree rettype, srctype, ptrtype, idxtype, masktype, scaletype; - tree ptr, var, scale, vec_mask; - tree mask_arg = NULL_TREE, mask_op = NULL_TREE, perm_mask = NULL_TREE; - tree mask_halfvectype = mask_vectype; - edge pe = loop_preheader_edge (loop); - gimple_seq seq; - basic_block new_bb; - enum { NARROW, NONE, WIDEN } modifier; - poly_uint64 scatter_off_nunits - = TYPE_VECTOR_SUBPARTS (gs_info.offset_vectype); - - if (known_eq (nunits, scatter_off_nunits)) - modifier = NONE; - else if (known_eq (nunits * 2, scatter_off_nunits)) - { - modifier = WIDEN; - - /* Currently gathers and scatters are only supported for - fixed-length vectors. */ - unsigned int count = scatter_off_nunits.to_constant (); - vec_perm_builder sel (count, count, 1); - for (i = 0; i < (unsigned int) count; ++i) - sel.quick_push (i | (count / 2)); - - vec_perm_indices indices (sel, 1, count); - perm_mask = vect_gen_perm_mask_checked (gs_info.offset_vectype, - indices); - gcc_assert (perm_mask != NULL_TREE); - } - else if (known_eq (nunits, scatter_off_nunits * 2)) - { - modifier = NARROW; - - /* Currently gathers and scatters are only supported for - fixed-length vectors. */ - unsigned int count = nunits.to_constant (); - vec_perm_builder sel (count, count, 1); - for (i = 0; i < (unsigned int) count; ++i) - sel.quick_push (i | (count / 2)); - - vec_perm_indices indices (sel, 2, count); - perm_mask = vect_gen_perm_mask_checked (vectype, indices); - gcc_assert (perm_mask != NULL_TREE); - ncopies *= 2; - - if (mask) - mask_halfvectype = truth_type_for (gs_info.offset_vectype); - } - else - gcc_unreachable (); - - rettype = TREE_TYPE (TREE_TYPE (gs_info.decl)); - ptrtype = TREE_VALUE (arglist); arglist = TREE_CHAIN (arglist); - masktype = TREE_VALUE (arglist); arglist = TREE_CHAIN (arglist); - idxtype = TREE_VALUE (arglist); arglist = TREE_CHAIN (arglist); - srctype = TREE_VALUE (arglist); arglist = TREE_CHAIN (arglist); - scaletype = TREE_VALUE (arglist); - - gcc_checking_assert (TREE_CODE (masktype) == INTEGER_TYPE - && TREE_CODE (rettype) == VOID_TYPE); - - ptr = fold_convert (ptrtype, gs_info.base); - if (!is_gimple_min_invariant (ptr)) - { - ptr = force_gimple_operand (ptr, &seq, true, NULL_TREE); - new_bb = gsi_insert_seq_on_edge_immediate (pe, seq); - gcc_assert (!new_bb); - } - - if (mask == NULL_TREE) - { - mask_arg = build_int_cst (masktype, -1); - mask_arg = vect_init_vector (vinfo, stmt_info, - mask_arg, masktype, NULL); - } - - scale = build_int_cst (scaletype, gs_info.scale); - - auto_vec vec_oprnds0; - auto_vec vec_oprnds1; - auto_vec vec_masks; - if (mask) - { - tree mask_vectype = truth_type_for (vectype); - vect_get_vec_defs_for_operand (vinfo, stmt_info, - modifier == NARROW - ? ncopies / 2 : ncopies, - mask, &vec_masks, mask_vectype); - } - vect_get_vec_defs_for_operand (vinfo, stmt_info, - modifier == WIDEN - ? ncopies / 2 : ncopies, - gs_info.offset, &vec_oprnds0); - vect_get_vec_defs_for_operand (vinfo, stmt_info, - modifier == NARROW - ? ncopies / 2 : ncopies, - op, &vec_oprnds1); - for (j = 0; j < ncopies; ++j) - { - if (modifier == WIDEN) - { - if (j & 1) - op = permute_vec_elements (vinfo, vec_oprnd0, vec_oprnd0, - perm_mask, stmt_info, gsi); - else - op = vec_oprnd0 = vec_oprnds0[j / 2]; - src = vec_oprnd1 = vec_oprnds1[j]; - if (mask) - mask_op = vec_mask = vec_masks[j]; - } - else if (modifier == NARROW) - { - if (j & 1) - src = permute_vec_elements (vinfo, vec_oprnd1, vec_oprnd1, - perm_mask, stmt_info, gsi); - else - src = vec_oprnd1 = vec_oprnds1[j / 2]; - op = vec_oprnd0 = vec_oprnds0[j]; - if (mask) - mask_op = vec_mask = vec_masks[j / 2]; - } - else - { - op = vec_oprnd0 = vec_oprnds0[j]; - src = vec_oprnd1 = vec_oprnds1[j]; - if (mask) - mask_op = vec_mask = vec_masks[j]; - } - - if (!useless_type_conversion_p (srctype, TREE_TYPE (src))) - { - gcc_assert (known_eq (TYPE_VECTOR_SUBPARTS (TREE_TYPE (src)), - TYPE_VECTOR_SUBPARTS (srctype))); - var = vect_get_new_ssa_name (srctype, vect_simple_var); - src = build1 (VIEW_CONVERT_EXPR, srctype, src); - gassign *new_stmt - = gimple_build_assign (var, VIEW_CONVERT_EXPR, src); - vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi); - src = var; - } - - if (!useless_type_conversion_p (idxtype, TREE_TYPE (op))) - { - gcc_assert (known_eq (TYPE_VECTOR_SUBPARTS (TREE_TYPE (op)), - TYPE_VECTOR_SUBPARTS (idxtype))); - var = vect_get_new_ssa_name (idxtype, vect_simple_var); - op = build1 (VIEW_CONVERT_EXPR, idxtype, op); - gassign *new_stmt - = gimple_build_assign (var, VIEW_CONVERT_EXPR, op); - vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi); - op = var; - } - - if (mask) - { - tree utype; - mask_arg = mask_op; - if (modifier == NARROW) - { - var = vect_get_new_ssa_name (mask_halfvectype, - vect_simple_var); - gassign *new_stmt - = gimple_build_assign (var, (j & 1) ? VEC_UNPACK_HI_EXPR - : VEC_UNPACK_LO_EXPR, - mask_op); - vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi); - mask_arg = var; - } - tree optype = TREE_TYPE (mask_arg); - if (TYPE_MODE (masktype) == TYPE_MODE (optype)) - utype = masktype; - else - utype = lang_hooks.types.type_for_mode (TYPE_MODE (optype), 1); - var = vect_get_new_ssa_name (utype, vect_scalar_var); - mask_arg = build1 (VIEW_CONVERT_EXPR, utype, mask_arg); - gassign *new_stmt - = gimple_build_assign (var, VIEW_CONVERT_EXPR, mask_arg); - vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi); - mask_arg = var; - if (!useless_type_conversion_p (masktype, utype)) - { - gcc_assert (TYPE_PRECISION (utype) - <= TYPE_PRECISION (masktype)); - var = vect_get_new_ssa_name (masktype, vect_scalar_var); - new_stmt = gimple_build_assign (var, NOP_EXPR, mask_arg); - vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi); - mask_arg = var; - } - } - - gcall *new_stmt - = gimple_build_call (gs_info.decl, 5, ptr, mask_arg, op, src, scale); - vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi); - - STMT_VINFO_VEC_STMTS (stmt_info).safe_push (new_stmt); - } - *vec_stmt = STMT_VINFO_VEC_STMTS (stmt_info)[0]; + vect_build_scatter_store_calls (vinfo, stmt_info, gsi, vec_stmt, + &gs_info, mask); return true; } else if (STMT_VINFO_SIMD_LANE_ACCESS_P (stmt_info) >= 3)