From patchwork Tue Jun 13 02:03:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kewen.Lin" X-Patchwork-Id: 107016 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp250908vqr; Mon, 12 Jun 2023 19:05:10 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5bVRHpc9HYodZ3Vav1zximR7Ox3BGhTc9/JfTVIOnMO/tzXmtbKA9+oXCUwlmOkBgTpIst X-Received: by 2002:a17:907:785:b0:968:2bb1:f39d with SMTP id xd5-20020a170907078500b009682bb1f39dmr11787171ejb.36.1686621910141; Mon, 12 Jun 2023 19:05:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1686621910; cv=none; d=google.com; s=arc-20160816; b=buIpJfwVgJHvmthMcZ0X2oOORto0CPcp9m4puDE8616zxSx22gNM0z2PW4E0Dw8YtH LZX5CeMH9rf8x3bbz5bi5wWh88cMaHzbBzyBx2TRFZTjxBqy786AXzvaJqvvIHL2VkA2 5vNV6xMAbY5nOGkadOzbe0xAL4RzBoZloUtPSSh4PJqSbejiHS1/iFJ6dcPnWeNDC5Xk n1JZPnQGw9B9OGLz9ow6RQSqCeEFR6LReAyALYU7Yjm9jCryJdUlXAWjjTOA90fVb1Ts SLCCzooUiyhPyArT6VKqLy8AzsBitdkSiEZPKJEHU3tYl4I/4nnsptFU71DnPpjraWpx aRiQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:reply-to:from:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence :content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:dmarc-filter:delivered-to :dkim-signature:dkim-filter; bh=DfP78s4ejZX/xk6T5/bESAP04I3jfMu8OzccAcqdMNQ=; b=p6kg59pNQ4FKB3ZJSH91qDR1W6FRe19s2wxLQt+Uusy4tv37hms7lvfk6fEN3/7gH4 CHJB91elEPiVAcYZK+XIAqK2Lwu5POrUazvH1YFJJj1eFrNQplF2HFaoF1LCLLxxbhNP S/sdteRDuHeF1CV34EMVhqvCpkD4/HPYso67KmcFX0WaCUDySvZDxrnEQweubypIcjpA jpvABhH10H6OJQO3+72U7d2QlPY3XCBdJ20ugz1Xl3p1tg/xqJZzHc7kZtBCoM/JEqCj tyrkLTezIsJPz949FtjtBR4cvDbaLu7QX0T7XEPVUw35dgjIw9ktsNxvRuWwbu3OEJFl +s8g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=xdXBbjB2; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gnu.org Received: from sourceware.org (server2.sourceware.org. [2620:52:3:1:0:246e:9693:128c]) by mx.google.com with ESMTPS id r27-20020a170906281b00b009665a49fc65si6308490ejc.961.2023.06.12.19.05.09 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Jun 2023 19:05:10 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) client-ip=2620:52:3:1:0:246e:9693:128c; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=xdXBbjB2; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gnu.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id CADF63857716 for ; Tue, 13 Jun 2023 02:04:44 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org CADF63857716 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1686621884; bh=DfP78s4ejZX/xk6T5/bESAP04I3jfMu8OzccAcqdMNQ=; h=To:Cc:Subject:Date:In-Reply-To:References:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=xdXBbjB2jbmhp++ovv3KM7IbEYljP0GOAKu+88FTIYx1RUBcMfhVz5NucYcTN7Vdu Sgxm+Yny7l5JgzsMVheP3JRMuq3sIJr72nWBA3iaXKxifLfVqrXlqdjnRQnxwmw3j8 wcDsup1jJ9XmNiSchbKCdtrn67GNnZ1qM5RtIcv0= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by sourceware.org (Postfix) with ESMTPS id A9A093858D32 for ; Tue, 13 Jun 2023 02:03:52 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org A9A093858D32 Received: from pps.filterd (m0353729.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 35D2175X020248; Tue, 13 Jun 2023 02:03:49 GMT Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3r6fa2829b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 13 Jun 2023 02:03:48 +0000 Received: from m0353729.ppops.net (m0353729.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 35D22Psj024218; Tue, 13 Jun 2023 02:03:48 GMT Received: from ppma02fra.de.ibm.com (47.49.7a9f.ip4.static.sl-reverse.com [159.122.73.71]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3r6fa28281-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 13 Jun 2023 02:03:48 +0000 Received: from pps.filterd (ppma02fra.de.ibm.com [127.0.0.1]) by ppma02fra.de.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 35D1ojFw031937; Tue, 13 Jun 2023 02:03:45 GMT Received: from smtprelay02.fra02v.mail.ibm.com ([9.218.2.226]) by ppma02fra.de.ibm.com (PPS) with ESMTPS id 3r4gt51bmn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 13 Jun 2023 02:03:45 +0000 Received: from smtpav04.fra02v.mail.ibm.com (smtpav04.fra02v.mail.ibm.com [10.20.54.103]) by smtprelay02.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 35D23h0C51773790 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Jun 2023 02:03:43 GMT Received: from smtpav04.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4467C20040; Tue, 13 Jun 2023 02:03:43 +0000 (GMT) Received: from smtpav04.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5693420043; Tue, 13 Jun 2023 02:03:42 +0000 (GMT) Received: from trout.aus.stglabs.ibm.com (unknown [9.40.194.100]) by smtpav04.fra02v.mail.ibm.com (Postfix) with ESMTP; Tue, 13 Jun 2023 02:03:42 +0000 (GMT) To: gcc-patches@gcc.gnu.org Cc: richard.guenther@gmail.com, richard.sandiford@arm.com, segher@kernel.crashing.org, bergner@linux.ibm.com Subject: [PATCH 1/9] vect: Move vect_model_load_cost next to the transform in vectorizable_load Date: Mon, 12 Jun 2023 21:03:22 -0500 Message-Id: X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: 2J0PigIBd2vBtkufuH_x7eLqAvvnw3HZ X-Proofpoint-GUID: LLmldVoNeCc4-gqSznHy8e1UdHh10JHo X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.573,FMLib:17.11.176.26 definitions=2023-06-12_18,2023-06-12_02,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 impostorscore=0 suspectscore=0 spamscore=0 adultscore=0 lowpriorityscore=0 mlxscore=0 priorityscore=1501 malwarescore=0 clxscore=1015 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2306130016 X-Spam-Status: No, score=-12.4 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_MSPIKE_H5, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Kewen Lin via Gcc-patches From: "Kewen.Lin" Reply-To: Kewen Lin Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org Sender: "Gcc-patches" X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1768551256242239098?= X-GMAIL-MSGID: =?utf-8?q?1768551256242239098?= This patch is an initial patch to move costing next to the transform, it still adopts vect_model_load_cost for costing but moves and duplicates it down according to the handlings of different vect_memory_access_types, hope it can make the subsequent patches easy to review. This patch should not have any functional changes. gcc/ChangeLog: * tree-vect-stmts.cc (vectorizable_load): Move and duplicate the call to vect_model_load_cost down to some different transform paths according to the handlings of different vect_memory_access_types. --- gcc/tree-vect-stmts.cc | 86 ++++++++++++++++++++++++++++-------------- 1 file changed, 57 insertions(+), 29 deletions(-) diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc index a7acc032d47..44514658be3 100644 --- a/gcc/tree-vect-stmts.cc +++ b/gcc/tree-vect-stmts.cc @@ -9430,7 +9430,9 @@ vectorizable_load (vec_info *vinfo, } } - if (!vec_stmt) /* transformation not required. */ + bool costing_p = !vec_stmt; + + if (costing_p) /* transformation not required. */ { if (slp_node && mask @@ -9464,17 +9466,13 @@ vectorizable_load (vec_info *vinfo, vinfo->any_known_not_updated_vssa = true; STMT_VINFO_TYPE (stmt_info) = load_vec_info_type; - vect_model_load_cost (vinfo, stmt_info, ncopies, vf, memory_access_type, - alignment_support_scheme, misalignment, - &gs_info, slp_node, cost_vec); - return true; } if (!slp) gcc_assert (memory_access_type == STMT_VINFO_MEMORY_ACCESS_TYPE (stmt_info)); - if (dump_enabled_p ()) + if (dump_enabled_p () && !costing_p) dump_printf_loc (MSG_NOTE, vect_location, "transform load. ncopies = %d\n", ncopies); @@ -9485,13 +9483,26 @@ vectorizable_load (vec_info *vinfo, if (memory_access_type == VMAT_GATHER_SCATTER && gs_info.decl) { - vect_build_gather_load_calls (vinfo, - stmt_info, gsi, vec_stmt, &gs_info, mask); + if (costing_p) + vect_model_load_cost (vinfo, stmt_info, ncopies, vf, memory_access_type, + alignment_support_scheme, misalignment, &gs_info, + slp_node, cost_vec); + else + vect_build_gather_load_calls (vinfo, stmt_info, gsi, vec_stmt, &gs_info, + mask); return true; } if (memory_access_type == VMAT_INVARIANT) { + if (costing_p) + { + vect_model_load_cost (vinfo, stmt_info, ncopies, vf, + memory_access_type, alignment_support_scheme, + misalignment, &gs_info, slp_node, cost_vec); + return true; + } + gcc_assert (!grouped_load && !mask && !bb_vinfo); /* If we have versioned for aliasing or the loop doesn't have any data dependencies that would preclude this, @@ -9548,6 +9559,14 @@ vectorizable_load (vec_info *vinfo, if (memory_access_type == VMAT_ELEMENTWISE || memory_access_type == VMAT_STRIDED_SLP) { + if (costing_p) + { + vect_model_load_cost (vinfo, stmt_info, ncopies, vf, + memory_access_type, alignment_support_scheme, + misalignment, &gs_info, slp_node, cost_vec); + return true; + } + gimple_stmt_iterator incr_gsi; bool insert_after; tree offvar; @@ -9989,17 +10008,20 @@ vectorizable_load (vec_info *vinfo, here, since we can't guarantee first_stmt_info DR has been initialized yet, use first_stmt_info_for_drptr DR by bumping the distance from first_stmt_info DR instead as below. */ - if (!diff_first_stmt_info) - msq = vect_setup_realignment (vinfo, - first_stmt_info, gsi, &realignment_token, - alignment_support_scheme, NULL_TREE, - &at_loop); - if (alignment_support_scheme == dr_explicit_realign_optimized) - { - phi = as_a (SSA_NAME_DEF_STMT (msq)); - offset = size_binop (MINUS_EXPR, TYPE_SIZE_UNIT (vectype), - size_one_node); - gcc_assert (!first_stmt_info_for_drptr); + if (!costing_p) + { + if (!diff_first_stmt_info) + msq = vect_setup_realignment (vinfo, first_stmt_info, gsi, + &realignment_token, + alignment_support_scheme, NULL_TREE, + &at_loop); + if (alignment_support_scheme == dr_explicit_realign_optimized) + { + phi = as_a (SSA_NAME_DEF_STMT (msq)); + offset = size_binop (MINUS_EXPR, TYPE_SIZE_UNIT (vectype), + size_one_node); + gcc_assert (!first_stmt_info_for_drptr); + } } } else @@ -10020,8 +10042,9 @@ vectorizable_load (vec_info *vinfo, else if (memory_access_type == VMAT_GATHER_SCATTER) { aggr_type = elem_type; - vect_get_strided_load_store_ops (stmt_info, loop_vinfo, &gs_info, - &bump, &vec_offset); + if (!costing_p) + vect_get_strided_load_store_ops (stmt_info, loop_vinfo, &gs_info, &bump, + &vec_offset); } else { @@ -10035,7 +10058,7 @@ vectorizable_load (vec_info *vinfo, auto_vec vec_offsets; auto_vec vec_masks; - if (mask) + if (mask && !costing_p) { if (slp_node) vect_get_slp_defs (SLP_TREE_CHILDREN (slp_node)[mask_index], @@ -10049,7 +10072,7 @@ vectorizable_load (vec_info *vinfo, for (j = 0; j < ncopies; j++) { /* 1. Create the vector or array pointer update chain. */ - if (j == 0) + if (j == 0 && !costing_p) { bool simd_lane_access_p = STMT_VINFO_SIMD_LANE_ACCESS_P (stmt_info) != 0; @@ -10108,7 +10131,7 @@ vectorizable_load (vec_info *vinfo, if (mask) vec_mask = vec_masks[0]; } - else + else if (!costing_p) { gcc_assert (!LOOP_VINFO_USING_SELECT_VL_P (loop_vinfo)); if (dataref_offset) @@ -10125,7 +10148,7 @@ vectorizable_load (vec_info *vinfo, dr_chain.create (vec_num); gimple *new_stmt = NULL; - if (memory_access_type == VMAT_LOAD_STORE_LANES) + if (memory_access_type == VMAT_LOAD_STORE_LANES && !costing_p) { tree vec_array; @@ -10177,7 +10200,7 @@ vectorizable_load (vec_info *vinfo, /* Record that VEC_ARRAY is now dead. */ vect_clobber_variable (vinfo, stmt_info, gsi, vec_array); } - else + else if (!costing_p) { for (i = 0; i < vec_num; i++) { @@ -10631,7 +10654,7 @@ vectorizable_load (vec_info *vinfo, if (slp && !slp_perm) continue; - if (slp_perm) + if (slp_perm && !costing_p) { unsigned n_perms; /* For SLP we know we've seen all possible uses of dr_chain so @@ -10643,7 +10666,7 @@ vectorizable_load (vec_info *vinfo, nullptr, true); gcc_assert (ok); } - else + else if (!costing_p) { if (grouped_load) { @@ -10659,9 +10682,14 @@ vectorizable_load (vec_info *vinfo, } dr_chain.release (); } - if (!slp) + if (!slp && !costing_p) *vec_stmt = STMT_VINFO_VEC_STMTS (stmt_info)[0]; + if (costing_p) + vect_model_load_cost (vinfo, stmt_info, ncopies, vf, memory_access_type, + alignment_support_scheme, misalignment, &gs_info, + slp_node, cost_vec); + return true; }