From patchwork Thu Sep 14 03:11:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kewen.Lin" X-Patchwork-Id: 139269 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp79676vqi; Wed, 13 Sep 2023 20:14:03 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGcrFfj4W6Eapbe0U8qUuukifh9VE5MwCPEJeqacOHSQqwzQbVgWSOfLg2WxXnV1ahVebT+ X-Received: by 2002:a17:906:3282:b0:9a2:120a:5779 with SMTP id 2-20020a170906328200b009a2120a5779mr3163326ejw.60.1694661242936; Wed, 13 Sep 2023 20:14:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694661242; cv=none; d=google.com; s=arc-20160816; b=DVSJ+W9X4R3dgCMaucwrFHIBwVPJJe77JQKj36JtoAQu0tEvPf1aYAj6ILFQpLxpcy KphqFB3Nfq2dFrhzsMitMwIU9Wsx4OYt5tz8feI0IqMGJVnTK1ZJtHfia5A1eisgONUh ZsqDpaZdC+gfseJ3qXAdlsKGqxMiT1YaNVFIyTbO1deAUhKVost8K4b1j82rg1v6lsUz UkthlscIehAkor/Var9wnJ82rl9xZA73U/+0ttoGjGW2l7gqn6dj5XtTCU7eaQ4R89sL F9yKQ3h+lWXbR22uL4nfMHoEx2FYhFeUWokn3jijpOnywzCIqwqOOa3CNixXGJiawCet Gc/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:reply-to:from:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence :content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:dmarc-filter:delivered-to :dkim-signature:dkim-filter; bh=hBnjte7FFsy6RRoK9snmkszsO0pbNv4vGDg463ngIMw=; fh=se0EChbRuDzWeQLxh4ma2F8om/5/For7fEVL6Npc7+A=; b=RRPAWzdwlmf1m2ASouKV9i5AnElsI4/Zg11/peGmfsrEtZWBjR3hPsB9ejnvvhek0w 75ouo3KWj2yXK2i9UwPffJdojcZUOyNuituhhP1sVGUHku9DKxlGaRT//A9t/PL/aB3r 063EbXc8U7Fomc/gbQCkHTrkf5IEDjmOfqhJpS4qEBlgC/+xA2n3YkAo1w+ZzfOIk5er 8Oh/smLLUvVg4EmiMJGUpJ/wCyqisR4C3RxfvAWl2OjS+Al7ayzNEemUPTCui1bsFcjz 74+XLm7txPyWx5JOJwhk9YseRH4BdD1XoUuVuUX72Q48hY9p1eo8WtpcybdHf7zJ5VdX aIHQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=TJivs19w; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gnu.org Received: from server2.sourceware.org (ip-8-43-85-97.sourceware.org. [8.43.85.97]) by mx.google.com with ESMTPS id f25-20020a170906085900b0098283e90548si485917ejd.570.2023.09.13.20.14.02 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 20:14:02 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) client-ip=8.43.85.97; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=TJivs19w; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gnu.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id BC5683839DDF for ; Thu, 14 Sep 2023 03:13:00 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org BC5683839DDF DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1694661180; bh=hBnjte7FFsy6RRoK9snmkszsO0pbNv4vGDg463ngIMw=; h=To:Cc:Subject:Date:In-Reply-To:References:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=TJivs19wfyfA9I8pVlFu7nVNmJDj4Y8sfCN79g8o856xLqjZDnthVhMNH5ZVRSdSp qeMAr4v1fWiXAf6s/cC9D/X8AbnxA6SselWmquqkqZUAW3VMZ6D1zpcQIZzONF6E3a 8/cOKtbABJ4c/9Lzl6jfhJbl8Tz46V+I2gkop4zw= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by sourceware.org (Postfix) with ESMTPS id EC26C3858C2B for ; Thu, 14 Sep 2023 03:12:08 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org EC26C3858C2B Received: from pps.filterd (m0356516.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38E38tU5028540; Thu, 14 Sep 2023 03:12:07 GMT Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3t3sq3rar4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 14 Sep 2023 03:12:06 +0000 Received: from m0356516.ppops.net (m0356516.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 38E3ALtU002343; Thu, 14 Sep 2023 03:12:06 GMT Received: from ppma12.dal12v.mail.ibm.com (dc.9e.1632.ip4.static.sl-reverse.com [50.22.158.220]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3t3sq3raqy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 14 Sep 2023 03:12:06 +0000 Received: from pps.filterd (ppma12.dal12v.mail.ibm.com [127.0.0.1]) by ppma12.dal12v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 38E18D4F024088; Thu, 14 Sep 2023 03:12:05 GMT Received: from smtprelay05.fra02v.mail.ibm.com ([9.218.2.225]) by ppma12.dal12v.mail.ibm.com (PPS) with ESMTPS id 3t131tg7j6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 14 Sep 2023 03:12:05 +0000 Received: from smtpav06.fra02v.mail.ibm.com (smtpav06.fra02v.mail.ibm.com [10.20.54.105]) by smtprelay05.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 38E3C3hq16056960 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 14 Sep 2023 03:12:03 GMT Received: from smtpav06.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B0D5E20040; Thu, 14 Sep 2023 03:12:03 +0000 (GMT) Received: from smtpav06.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E89932004B; Thu, 14 Sep 2023 03:12:02 +0000 (GMT) Received: from trout.aus.stglabs.ibm.com (unknown [9.40.194.100]) by smtpav06.fra02v.mail.ibm.com (Postfix) with ESMTP; Thu, 14 Sep 2023 03:12:02 +0000 (GMT) To: gcc-patches@gcc.gnu.org Cc: richard.guenther@gmail.com, richard.sandiford@arm.com Subject: [PATCH 02/10] vect: Move vect_model_store_cost next to the transform in vectorizable_store Date: Wed, 13 Sep 2023 22:11:51 -0500 Message-Id: <1539ec7d34af4e38467420b3aed342d708a64a48.1694657494.git.linkw@linux.ibm.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: DF7KICfyTP2M0VtLkqkznXEtcQJoW0n4 X-Proofpoint-ORIG-GUID: cpsy1P73M4FF_amiWGjShHfXRIcpDQe7 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-13_19,2023-09-13_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 suspectscore=0 adultscore=0 spamscore=0 clxscore=1015 malwarescore=0 phishscore=0 bulkscore=0 lowpriorityscore=0 priorityscore=1501 impostorscore=0 mlxlogscore=973 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309140025 X-Spam-Status: No, score=-12.3 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_MSPIKE_H4, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Kewen Lin via Gcc-patches From: "Kewen.Lin" Reply-To: Kewen Lin Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org Sender: "Gcc-patches" X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776981107466135062 X-GMAIL-MSGID: 1776981107466135062 This patch is an initial patch to move costing next to the transform, it still adopts vect_model_store_cost for costing but moves and duplicates it down according to the handlings of different vect_memory_access_types or some special handling need, hope it can make the subsequent patches easy to review. This patch should not have any functional changes. gcc/ChangeLog: * tree-vect-stmts.cc (vectorizable_store): Move and duplicate the call to vect_model_store_cost down to some different transform paths according to the handlings of different vect_memory_access_types or some special handling need. --- gcc/tree-vect-stmts.cc | 79 ++++++++++++++++++++++++++++++++---------- 1 file changed, 60 insertions(+), 19 deletions(-) diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc index a5caaf0bca2..36f7c5b9f4b 100644 --- a/gcc/tree-vect-stmts.cc +++ b/gcc/tree-vect-stmts.cc @@ -8372,7 +8372,8 @@ vectorizable_store (vec_info *vinfo, return false; } - if (!vec_stmt) /* transformation not required. */ + bool costing_p = !vec_stmt; + if (costing_p) /* transformation not required. */ { STMT_VINFO_MEMORY_ACCESS_TYPE (stmt_info) = memory_access_type; @@ -8401,11 +8402,6 @@ vectorizable_store (vec_info *vinfo, "Vectorizing an unaligned access.\n"); STMT_VINFO_TYPE (stmt_info) = store_vec_info_type; - vect_model_store_cost (vinfo, stmt_info, ncopies, - memory_access_type, &gs_info, - alignment_support_scheme, - misalignment, vls_type, slp_node, cost_vec); - return true; } gcc_assert (memory_access_type == STMT_VINFO_MEMORY_ACCESS_TYPE (stmt_info)); @@ -8415,12 +8411,27 @@ vectorizable_store (vec_info *vinfo, if (memory_access_type == VMAT_GATHER_SCATTER && gs_info.decl) { - vect_build_scatter_store_calls (vinfo, stmt_info, gsi, vec_stmt, - &gs_info, mask); + if (costing_p) + vect_model_store_cost (vinfo, stmt_info, ncopies, memory_access_type, + &gs_info, alignment_support_scheme, misalignment, + vls_type, slp_node, cost_vec); + else + vect_build_scatter_store_calls (vinfo, stmt_info, gsi, vec_stmt, + &gs_info, mask); return true; } else if (STMT_VINFO_SIMD_LANE_ACCESS_P (stmt_info) >= 3) - return vectorizable_scan_store (vinfo, stmt_info, gsi, vec_stmt, ncopies); + { + gcc_assert (memory_access_type == VMAT_CONTIGUOUS); + if (costing_p) + { + vect_model_store_cost (vinfo, stmt_info, ncopies, memory_access_type, + &gs_info, alignment_support_scheme, + misalignment, vls_type, slp_node, cost_vec); + return true; + } + return vectorizable_scan_store (vinfo, stmt_info, gsi, vec_stmt, ncopies); + } if (grouped_store) { @@ -8449,13 +8460,21 @@ vectorizable_store (vec_info *vinfo, else ref_type = reference_alias_ptr_type (DR_REF (first_dr_info->dr)); - if (dump_enabled_p ()) - dump_printf_loc (MSG_NOTE, vect_location, - "transform store. ncopies = %d\n", ncopies); + if (!costing_p && dump_enabled_p ()) + dump_printf_loc (MSG_NOTE, vect_location, "transform store. ncopies = %d\n", + ncopies); if (memory_access_type == VMAT_ELEMENTWISE || memory_access_type == VMAT_STRIDED_SLP) { + if (costing_p) + { + vect_model_store_cost (vinfo, stmt_info, ncopies, memory_access_type, + &gs_info, alignment_support_scheme, + misalignment, vls_type, slp_node, cost_vec); + return true; + } + gimple_stmt_iterator incr_gsi; bool insert_after; gimple *incr; @@ -8718,8 +8737,9 @@ vectorizable_store (vec_info *vinfo, else if (memory_access_type == VMAT_GATHER_SCATTER) { aggr_type = elem_type; - vect_get_strided_load_store_ops (stmt_info, loop_vinfo, gsi, &gs_info, - &bump, &vec_offset, loop_lens); + if (!costing_p) + vect_get_strided_load_store_ops (stmt_info, loop_vinfo, gsi, &gs_info, + &bump, &vec_offset, loop_lens); } else { @@ -8731,7 +8751,7 @@ vectorizable_store (vec_info *vinfo, memory_access_type, loop_lens); } - if (mask) + if (mask && !costing_p) LOOP_VINFO_HAS_MASK_STORE (loop_vinfo) = true; /* In case the vectorization factor (VF) is bigger than the number @@ -8782,6 +8802,13 @@ vectorizable_store (vec_info *vinfo, if (memory_access_type == VMAT_LOAD_STORE_LANES) { gcc_assert (!slp && grouped_store); + if (costing_p) + { + vect_model_store_cost (vinfo, stmt_info, ncopies, memory_access_type, + &gs_info, alignment_support_scheme, + misalignment, vls_type, slp_node, cost_vec); + return true; + } for (j = 0; j < ncopies; j++) { gimple *new_stmt; @@ -8927,6 +8954,13 @@ vectorizable_store (vec_info *vinfo, if (memory_access_type == VMAT_GATHER_SCATTER) { gcc_assert (!slp && !grouped_store); + if (costing_p) + { + vect_model_store_cost (vinfo, stmt_info, ncopies, memory_access_type, + &gs_info, alignment_support_scheme, + misalignment, vls_type, slp_node, cost_vec); + return true; + } auto_vec vec_offsets; for (j = 0; j < ncopies; j++) { @@ -9091,7 +9125,7 @@ vectorizable_store (vec_info *vinfo, for (j = 0; j < ncopies; j++) { gimple *new_stmt; - if (j == 0) + if (j == 0 && !costing_p) { if (slp) { @@ -9158,7 +9192,7 @@ vectorizable_store (vec_info *vinfo, offset, &dummy, gsi, &ptr_incr, simd_lane_access_p, bump); } - else + else if (!costing_p) { gcc_assert (!LOOP_VINFO_USING_SELECT_VL_P (loop_vinfo)); /* DR_CHAIN is then used as an input to vect_permute_store_chain(). @@ -9179,7 +9213,7 @@ vectorizable_store (vec_info *vinfo, } new_stmt = NULL; - if (grouped_store) + if (!costing_p && grouped_store) /* Permute. */ vect_permute_store_chain (vinfo, dr_chain, group_size, stmt_info, gsi, &result_chain); @@ -9187,6 +9221,8 @@ vectorizable_store (vec_info *vinfo, stmt_vec_info next_stmt_info = first_stmt_info; for (i = 0; i < vec_num; i++) { + if (costing_p) + continue; unsigned misalign; unsigned HOST_WIDE_INT align; @@ -9361,7 +9397,7 @@ vectorizable_store (vec_info *vinfo, if (!next_stmt_info) break; } - if (!slp) + if (!slp && !costing_p) { if (j == 0) *vec_stmt = new_stmt; @@ -9369,6 +9405,11 @@ vectorizable_store (vec_info *vinfo, } } + if (costing_p) + vect_model_store_cost (vinfo, stmt_info, ncopies, memory_access_type, + &gs_info, alignment_support_scheme, misalignment, + vls_type, slp_node, cost_vec); + return true; }