From patchwork Thu Sep 14 03:11:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kewen.Lin" X-Patchwork-Id: 139278 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp81051vqi; Wed, 13 Sep 2023 20:17:42 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHZhAUlmv7VZPNuEY9nCYRIgKOBU0chIrkZ0ohLOYMsvvBZZoQEwUsNDXjLiWL6O4MciLvH X-Received: by 2002:a5d:5486:0:b0:31f:a259:733 with SMTP id h6-20020a5d5486000000b0031fa2590733mr3383847wrv.20.1694661462199; Wed, 13 Sep 2023 20:17:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694661462; cv=none; d=google.com; s=arc-20160816; b=tcb1kgGVg4CRs5Ibrn8zfcmcKY0CnAR4JxXHriaA6UHe3KkADu05XQewiCAVMEEcvx Pt7vr8TmxkRKNhJlCpgSZDsgIGqBKgT7034D1XXbs2mfDAZksIolg7CJfPp3fOBej8If Wjh4UCFDLzWs78YIUOEjubL2aHgF9IMiQviA44EyQopNAwEs/WKlP7keoZCSR4ScK9Fw DNI+i5V7XZmr3TRy8wJWgFJemKf+Q1PzKCTsXbY4OSHMBeur89SpFiXnqSFcoH8xhRH8 Z8rX9B9tBuwrnsyAyF/LKjieOSnoHZMNLBun/y9k2myCQxL+4iJHIu2BRtM7TVwkbttY JY7g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:reply-to:from:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence :content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:dmarc-filter:delivered-to :dkim-signature:dkim-filter; bh=uL6bd/ub+xytwJBUOufIlU+oclxP38BlDoWzO935S6I=; fh=se0EChbRuDzWeQLxh4ma2F8om/5/For7fEVL6Npc7+A=; b=VI9XcH+wizyKutygG9R+QpzU/zz7kmiWbpzf0C6Hx/UDoGr8QsLu/65A9sAi4plPGK sVdWmF73h6yg12cSLivaNPmXo21iLhb2N4zyu3d94gADUj1HEJhuBoC/GmNDve+FNGjq /v/eBkja6hU24eLoVllcjDDMcUNyVCWCBY3xrw/9ceZKFQGUfnHEvfhdua9EKB9zV1n6 bZR3jpHKtdBQT199XmRAeJfTzcBWeKgGvybxCe/iBwJWOPd9q7kKE6ikNMFwsBfV84zU /17bXU/GUoHknA3k93SjeNL8AR4grCwmOxOMPEemjhM4Qxgtqv6eDCCRxKfEjhBlBkAM g0OQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=rXBVObg6; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gnu.org Received: from server2.sourceware.org (server2.sourceware.org. [2620:52:3:1:0:246e:9693:128c]) by mx.google.com with ESMTPS id e6-20020a50fb86000000b0052f5f8a9db6si547619edq.91.2023.09.13.20.17.41 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 20:17:42 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) client-ip=2620:52:3:1:0:246e:9693:128c; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=rXBVObg6; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gnu.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 8AC43388202D for ; Thu, 14 Sep 2023 03:15:00 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 8AC43388202D DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1694661300; bh=uL6bd/ub+xytwJBUOufIlU+oclxP38BlDoWzO935S6I=; h=To:Cc:Subject:Date:In-Reply-To:References:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=rXBVObg67NTxbF+M62Y9nR5yE7fGGEHaCPjJf8xILzbTabiHLdgrCIyCUnt+6zhur w9buVoOP4gaXe1mW29Pcr6lPW+u6BQ2IYTrsH7i+mzIzt3VMWYFhlqlaElXuth64dn 9kMy0SIe8Q1dZLVOP6G2f5133qB6yjJwaAw/6fz8= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by sourceware.org (Postfix) with ESMTPS id 6BB023858C3A for ; Thu, 14 Sep 2023 03:12:11 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 6BB023858C3A Received: from pps.filterd (m0353726.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38E37MKY022234; Thu, 14 Sep 2023 03:12:08 GMT Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3t3ssm0c0v-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 14 Sep 2023 03:12:07 +0000 Received: from m0353726.ppops.net (m0353726.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 38E38CMv027867; Thu, 14 Sep 2023 03:12:07 GMT Received: from ppma12.dal12v.mail.ibm.com (dc.9e.1632.ip4.static.sl-reverse.com [50.22.158.220]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3t3ssm0c0m-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 14 Sep 2023 03:12:07 +0000 Received: from pps.filterd (ppma12.dal12v.mail.ibm.com [127.0.0.1]) by ppma12.dal12v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 38E198SP024021; Thu, 14 Sep 2023 03:12:06 GMT Received: from smtprelay06.fra02v.mail.ibm.com ([9.218.2.230]) by ppma12.dal12v.mail.ibm.com (PPS) with ESMTPS id 3t131tg7j8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 14 Sep 2023 03:12:06 +0000 Received: from smtpav06.fra02v.mail.ibm.com (smtpav06.fra02v.mail.ibm.com [10.20.54.105]) by smtprelay06.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 38E3C4G344368432 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 14 Sep 2023 03:12:04 GMT Received: from smtpav06.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C038220049; Thu, 14 Sep 2023 03:12:04 +0000 (GMT) Received: from smtpav06.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0546720040; Thu, 14 Sep 2023 03:12:04 +0000 (GMT) Received: from trout.aus.stglabs.ibm.com (unknown [9.40.194.100]) by smtpav06.fra02v.mail.ibm.com (Postfix) with ESMTP; Thu, 14 Sep 2023 03:12:03 +0000 (GMT) To: gcc-patches@gcc.gnu.org Cc: richard.guenther@gmail.com, richard.sandiford@arm.com Subject: [PATCH 03/10] vect: Adjust vectorizable_store costing on VMAT_GATHER_SCATTER Date: Wed, 13 Sep 2023 22:11:52 -0500 Message-Id: <8abc6ddb4683d9058ffb48eb54f3a717e655efb4.1694657494.git.linkw@linux.ibm.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: Sxgm0zRklnKuXp0jRp_qzNmrYjYLBWx2 X-Proofpoint-GUID: LCQS5XHNY2je7-0FR9EsxVbOMutU0xQs X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-13_19,2023-09-13_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 priorityscore=1501 mlxscore=0 bulkscore=0 spamscore=0 impostorscore=0 lowpriorityscore=0 suspectscore=0 clxscore=1015 mlxlogscore=999 phishscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309140025 X-Spam-Status: No, score=-12.3 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_MSPIKE_H4, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Kewen Lin via Gcc-patches From: "Kewen.Lin" Reply-To: Kewen Lin Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org Sender: "Gcc-patches" X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776981337479286747 X-GMAIL-MSGID: 1776981337479286747 This patch adjusts the cost handling on VMAT_GATHER_SCATTER in function vectorizable_store (all three cases), then we won't depend on vect_model_load_store for its costing any more. This patch shouldn't have any functional changes. gcc/ChangeLog: * tree-vect-stmts.cc (vect_model_store_cost): Assert it won't get VMAT_GATHER_SCATTER any more, remove VMAT_GATHER_SCATTER related handlings and the related parameter gs_info. (vect_build_scatter_store_calls): Add the handlings on costing with one more argument cost_vec. (vectorizable_store): Adjust the cost handling on VMAT_GATHER_SCATTER without calling vect_model_store_cost any more. --- gcc/tree-vect-stmts.cc | 188 ++++++++++++++++++++++++++--------------- 1 file changed, 118 insertions(+), 70 deletions(-) diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc index 36f7c5b9f4b..3f908242fee 100644 --- a/gcc/tree-vect-stmts.cc +++ b/gcc/tree-vect-stmts.cc @@ -959,12 +959,12 @@ cfun_returns (tree decl) static void vect_model_store_cost (vec_info *vinfo, stmt_vec_info stmt_info, int ncopies, vect_memory_access_type memory_access_type, - gather_scatter_info *gs_info, dr_alignment_support alignment_support_scheme, int misalignment, vec_load_store_type vls_type, slp_tree slp_node, stmt_vector_for_cost *cost_vec) { + gcc_assert (memory_access_type != VMAT_GATHER_SCATTER); unsigned int inside_cost = 0, prologue_cost = 0; stmt_vec_info first_stmt_info = stmt_info; bool grouped_access_p = STMT_VINFO_GROUPED_ACCESS (stmt_info); @@ -1012,18 +1012,9 @@ vect_model_store_cost (vec_info *vinfo, stmt_vec_info stmt_info, int ncopies, tree vectype = STMT_VINFO_VECTYPE (stmt_info); /* Costs of the stores. */ - if (memory_access_type == VMAT_ELEMENTWISE - || memory_access_type == VMAT_GATHER_SCATTER) + if (memory_access_type == VMAT_ELEMENTWISE) { unsigned int assumed_nunits = vect_nunits_for_cost (vectype); - if (memory_access_type == VMAT_GATHER_SCATTER - && gs_info->ifn == IFN_LAST && !gs_info->decl) - /* For emulated scatter N offset vector element extracts - (we assume the scalar scaling and ptr + offset add is consumed by - the load). */ - inside_cost += record_stmt_cost (cost_vec, ncopies * assumed_nunits, - vec_to_scalar, stmt_info, 0, - vect_body); /* N scalar stores plus extracting the elements. */ inside_cost += record_stmt_cost (cost_vec, ncopies * assumed_nunits, @@ -1034,9 +1025,7 @@ vect_model_store_cost (vec_info *vinfo, stmt_vec_info stmt_info, int ncopies, misalignment, &inside_cost, cost_vec); if (memory_access_type == VMAT_ELEMENTWISE - || memory_access_type == VMAT_STRIDED_SLP - || (memory_access_type == VMAT_GATHER_SCATTER - && gs_info->ifn == IFN_LAST && !gs_info->decl)) + || memory_access_type == VMAT_STRIDED_SLP) { /* N scalar stores plus extracting the elements. */ unsigned int assumed_nunits = vect_nunits_for_cost (vectype); @@ -2999,7 +2988,8 @@ vect_build_gather_load_calls (vec_info *vinfo, stmt_vec_info stmt_info, static void vect_build_scatter_store_calls (vec_info *vinfo, stmt_vec_info stmt_info, gimple_stmt_iterator *gsi, gimple **vec_stmt, - gather_scatter_info *gs_info, tree mask) + gather_scatter_info *gs_info, tree mask, + stmt_vector_for_cost *cost_vec) { loop_vec_info loop_vinfo = dyn_cast (vinfo); tree vectype = STMT_VINFO_VECTYPE (stmt_info); @@ -3009,6 +2999,30 @@ vect_build_scatter_store_calls (vec_info *vinfo, stmt_vec_info stmt_info, poly_uint64 scatter_off_nunits = TYPE_VECTOR_SUBPARTS (gs_info->offset_vectype); + /* FIXME: Keep the previous costing way in vect_model_store_cost by + costing N scalar stores, but it should be tweaked to use target + specific costs on related scatter store calls. */ + if (cost_vec) + { + tree op = vect_get_store_rhs (stmt_info); + enum vect_def_type dt; + gcc_assert (vect_is_simple_use (op, vinfo, &dt)); + unsigned int inside_cost, prologue_cost = 0; + if (dt == vect_constant_def || dt == vect_external_def) + prologue_cost += record_stmt_cost (cost_vec, 1, scalar_to_vec, + stmt_info, 0, vect_prologue); + unsigned int assumed_nunits = vect_nunits_for_cost (vectype); + inside_cost = record_stmt_cost (cost_vec, ncopies * assumed_nunits, + scalar_store, stmt_info, 0, vect_body); + + if (dump_enabled_p ()) + dump_printf_loc (MSG_NOTE, vect_location, + "vect_model_store_cost: inside_cost = %d, " + "prologue_cost = %d .\n", + inside_cost, prologue_cost); + return; + } + tree perm_mask = NULL_TREE, mask_halfvectype = NULL_TREE; if (known_eq (nunits, scatter_off_nunits)) modifier = NONE; @@ -8411,13 +8425,8 @@ vectorizable_store (vec_info *vinfo, if (memory_access_type == VMAT_GATHER_SCATTER && gs_info.decl) { - if (costing_p) - vect_model_store_cost (vinfo, stmt_info, ncopies, memory_access_type, - &gs_info, alignment_support_scheme, misalignment, - vls_type, slp_node, cost_vec); - else - vect_build_scatter_store_calls (vinfo, stmt_info, gsi, vec_stmt, - &gs_info, mask); + vect_build_scatter_store_calls (vinfo, stmt_info, gsi, vec_stmt, &gs_info, + mask, cost_vec); return true; } else if (STMT_VINFO_SIMD_LANE_ACCESS_P (stmt_info) >= 3) @@ -8426,8 +8435,8 @@ vectorizable_store (vec_info *vinfo, if (costing_p) { vect_model_store_cost (vinfo, stmt_info, ncopies, memory_access_type, - &gs_info, alignment_support_scheme, - misalignment, vls_type, slp_node, cost_vec); + alignment_support_scheme, misalignment, + vls_type, slp_node, cost_vec); return true; } return vectorizable_scan_store (vinfo, stmt_info, gsi, vec_stmt, ncopies); @@ -8470,8 +8479,8 @@ vectorizable_store (vec_info *vinfo, if (costing_p) { vect_model_store_cost (vinfo, stmt_info, ncopies, memory_access_type, - &gs_info, alignment_support_scheme, - misalignment, vls_type, slp_node, cost_vec); + alignment_support_scheme, misalignment, + vls_type, slp_node, cost_vec); return true; } @@ -8805,8 +8814,8 @@ vectorizable_store (vec_info *vinfo, if (costing_p) { vect_model_store_cost (vinfo, stmt_info, ncopies, memory_access_type, - &gs_info, alignment_support_scheme, - misalignment, vls_type, slp_node, cost_vec); + alignment_support_scheme, misalignment, + vls_type, slp_node, cost_vec); return true; } for (j = 0; j < ncopies; j++) @@ -8954,49 +8963,50 @@ vectorizable_store (vec_info *vinfo, if (memory_access_type == VMAT_GATHER_SCATTER) { gcc_assert (!slp && !grouped_store); - if (costing_p) - { - vect_model_store_cost (vinfo, stmt_info, ncopies, memory_access_type, - &gs_info, alignment_support_scheme, - misalignment, vls_type, slp_node, cost_vec); - return true; - } auto_vec vec_offsets; + unsigned int inside_cost = 0, prologue_cost = 0; for (j = 0; j < ncopies; j++) { gimple *new_stmt; if (j == 0) { - /* Since the store is not grouped, DR_GROUP_SIZE is 1, and - DR_CHAIN is of size 1. */ - gcc_assert (group_size == 1); - op = vect_get_store_rhs (first_stmt_info); - vect_get_vec_defs_for_operand (vinfo, first_stmt_info, ncopies, - op, gvec_oprnds[0]); - vec_oprnd = (*gvec_oprnds[0])[0]; - dr_chain.quick_push (vec_oprnd); - if (mask) + if (costing_p && vls_type == VLS_STORE_INVARIANT) + prologue_cost += record_stmt_cost (cost_vec, 1, scalar_to_vec, + stmt_info, 0, vect_prologue); + else if (!costing_p) { - vect_get_vec_defs_for_operand (vinfo, stmt_info, ncopies, - mask, &vec_masks, - mask_vectype); - vec_mask = vec_masks[0]; - } + /* Since the store is not grouped, DR_GROUP_SIZE is 1, and + DR_CHAIN is of size 1. */ + gcc_assert (group_size == 1); + op = vect_get_store_rhs (first_stmt_info); + vect_get_vec_defs_for_operand (vinfo, first_stmt_info, + ncopies, op, gvec_oprnds[0]); + vec_oprnd = (*gvec_oprnds[0])[0]; + dr_chain.quick_push (vec_oprnd); + if (mask) + { + vect_get_vec_defs_for_operand (vinfo, stmt_info, ncopies, + mask, &vec_masks, + mask_vectype); + vec_mask = vec_masks[0]; + } - /* We should have catched mismatched types earlier. */ - gcc_assert (useless_type_conversion_p (vectype, - TREE_TYPE (vec_oprnd))); - if (STMT_VINFO_GATHER_SCATTER_P (stmt_info)) - vect_get_gather_scatter_ops (loop_vinfo, loop, stmt_info, - slp_node, &gs_info, &dataref_ptr, - &vec_offsets); - else - dataref_ptr - = vect_create_data_ref_ptr (vinfo, first_stmt_info, aggr_type, - NULL, offset, &dummy, gsi, - &ptr_incr, false, bump); + /* We should have catched mismatched types earlier. */ + gcc_assert ( + useless_type_conversion_p (vectype, TREE_TYPE (vec_oprnd))); + if (STMT_VINFO_GATHER_SCATTER_P (stmt_info)) + vect_get_gather_scatter_ops (loop_vinfo, loop, stmt_info, + slp_node, &gs_info, + &dataref_ptr, &vec_offsets); + else + dataref_ptr + = vect_create_data_ref_ptr (vinfo, first_stmt_info, + aggr_type, NULL, offset, + &dummy, gsi, &ptr_incr, false, + bump); + } } - else + else if (!costing_p) { gcc_assert (!LOOP_VINFO_USING_SELECT_VL_P (loop_vinfo)); vec_oprnd = (*gvec_oprnds[0])[j]; @@ -9013,15 +9023,27 @@ vectorizable_store (vec_info *vinfo, tree final_mask = NULL_TREE; tree final_len = NULL_TREE; tree bias = NULL_TREE; - if (loop_masks) - final_mask = vect_get_loop_mask (loop_vinfo, gsi, loop_masks, - ncopies, vectype, j); - if (vec_mask) - final_mask = prepare_vec_mask (loop_vinfo, mask_vectype, final_mask, - vec_mask, gsi); + if (!costing_p) + { + if (loop_masks) + final_mask = vect_get_loop_mask (loop_vinfo, gsi, loop_masks, + ncopies, vectype, j); + if (vec_mask) + final_mask = prepare_vec_mask (loop_vinfo, mask_vectype, + final_mask, vec_mask, gsi); + } if (gs_info.ifn != IFN_LAST) { + if (costing_p) + { + unsigned int cnunits = vect_nunits_for_cost (vectype); + inside_cost + += record_stmt_cost (cost_vec, cnunits, scalar_store, + stmt_info, 0, vect_body); + continue; + } + if (STMT_VINFO_GATHER_SCATTER_P (stmt_info)) vec_offset = vec_offsets[j]; tree scale = size_int (gs_info.scale); @@ -9067,6 +9089,25 @@ vectorizable_store (vec_info *vinfo, { /* Emulated scatter. */ gcc_assert (!final_mask); + if (costing_p) + { + unsigned int cnunits = vect_nunits_for_cost (vectype); + /* For emulated scatter N offset vector element extracts + (we assume the scalar scaling and ptr + offset add is + consumed by the load). */ + inside_cost + += record_stmt_cost (cost_vec, cnunits, vec_to_scalar, + stmt_info, 0, vect_body); + /* N scalar stores plus extracting the elements. */ + inside_cost + += record_stmt_cost (cost_vec, cnunits, vec_to_scalar, + stmt_info, 0, vect_body); + inside_cost + += record_stmt_cost (cost_vec, cnunits, scalar_store, + stmt_info, 0, vect_body); + continue; + } + unsigned HOST_WIDE_INT const_nunits = nunits.to_constant (); unsigned HOST_WIDE_INT const_offset_nunits = TYPE_VECTOR_SUBPARTS (gs_info.offset_vectype).to_constant (); @@ -9117,6 +9158,13 @@ vectorizable_store (vec_info *vinfo, *vec_stmt = new_stmt; STMT_VINFO_VEC_STMTS (stmt_info).safe_push (new_stmt); } + + if (costing_p && dump_enabled_p ()) + dump_printf_loc (MSG_NOTE, vect_location, + "vect_model_store_cost: inside_cost = %d, " + "prologue_cost = %d .\n", + inside_cost, prologue_cost); + return true; } @@ -9407,8 +9455,8 @@ vectorizable_store (vec_info *vinfo, if (costing_p) vect_model_store_cost (vinfo, stmt_info, ncopies, memory_access_type, - &gs_info, alignment_support_scheme, misalignment, - vls_type, slp_node, cost_vec); + alignment_support_scheme, misalignment, vls_type, + slp_node, cost_vec); return true; }