From patchwork Tue Aug 22 08:44:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kewen.Lin" X-Patchwork-Id: 136471 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b82d:0:b0:3f2:4152:657d with SMTP id z13csp3489687vqi; Tue, 22 Aug 2023 01:45:28 -0700 (PDT) X-Google-Smtp-Source: AGHT+IG1W6w3KFe28BJgG05oY2JlBnRu93Dbg+cNgIJFojplXNYMyG4BNtcf+EmFG2vEcaFLd5Df X-Received: by 2002:a17:907:775a:b0:994:536c:ab45 with SMTP id kx26-20020a170907775a00b00994536cab45mr6708292ejc.50.1692693928057; Tue, 22 Aug 2023 01:45:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1692693928; cv=none; d=google.com; s=arc-20160816; b=RorZK3AJCZiz3x8zNz+x+wgk27N15qcAyjl21hIWsKdD01ijyvaMAgYuL2bhhO+VGD tPn8JB1INoNX3gpAN80ijM94LWpq2D/P92h0C4LWzLYNRLQthF0Uk535CC2vAFWhhIYl 12maM+N3NopIGe8vB6SqHLbBVD/iIAiG3gvcS2RrFKkyfhxPB1Xsy8+dQ8orzFtOaCNe PXLH0Ri9006jtDbu8WHSXDzBOONGQs2pWAcWfhDY1nqipjcDGdgms+0fnNZ+lsZgp/i0 go930cgyfrCj1ersBdMmgXK6UgtjdPlfG8WDBB9je/N1eJxLVCSOU+ONx7YIHNVNrrd9 WgwA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:reply-to:from:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence :content-transfer-encoding:cc:to:subject:content-language:user-agent :mime-version:date:message-id:dmarc-filter:delivered-to :dkim-signature:dkim-filter; bh=Zu9Xjodo7sOtmUhmAw7JqBZKlyL/LjkF74WQna+njtI=; fh=Gv6PAYpoZePgnyLjqy3TiguihZUSZfKa6GhguGNpSZ8=; b=z9Qrt1K/9TK0GBq/oMfywwkNOiTof+7p+7lV1RZDjynvvEr+IG0bXnqwlblNvBn0On kzCvyM5mXPFfstCXi39HpB4e5a7sz+C7JnPLTb5EOhYnv/ua0WOE8P+goeXRJ23BSasR z7dHePmAfpn/6YRFbR9okfxzmgOfuqzN+kXCGaF8UQEOsk4GmojCeg4t7VavYiz3mbz6 JY1iFFBam5eWaNmyOpV5Mp4QLlGdHrky2avGCTxIUz3kUYmlN0m4TEutCUiJtQDgjuvL MSjHfwfii952NiyMQ6K5XWFYSMd2N1XwQpZgiDGuzK7rflzIshaSXVmBmt6n+bI2si3B uChQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=OqSAxOkd; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gnu.org Received: from server2.sourceware.org (ip-8-43-85-97.sourceware.org. [8.43.85.97]) by mx.google.com with ESMTPS id qt11-20020a170906eceb00b00991f9e2a813si6873942ejb.299.2023.08.22.01.45.27 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Aug 2023 01:45:28 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) client-ip=8.43.85.97; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=OqSAxOkd; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gnu.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id BA64D3857012 for ; Tue, 22 Aug 2023 08:45:26 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org BA64D3857012 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1692693926; bh=Zu9Xjodo7sOtmUhmAw7JqBZKlyL/LjkF74WQna+njtI=; h=Date:Subject:To:Cc:List-Id:List-Unsubscribe:List-Archive: List-Post:List-Help:List-Subscribe:From:Reply-To:From; b=OqSAxOkdanPJ6J1faNtGMNPjiOaNlUFiQ+J979dKN+M9woViceLDxFYVSvXBjFq9C wjBcVcYuzkpg0lhSbU/FxiOR3kCbpoKgR7Ks/HJnNauA4+xzcsWzWe0F6Qd4kdcdaS EeU7Wdtvq3dpw4TVEwCReJb8oMC4G+/KB2n2ytAM= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by sourceware.org (Postfix) with ESMTPS id 5FB03385DC01 for ; Tue, 22 Aug 2023 08:44:13 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 5FB03385DC01 Received: from pps.filterd (m0353725.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37M8I0vS002483; Tue, 22 Aug 2023 08:44:11 GMT Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3smscs0n81-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 22 Aug 2023 08:44:10 +0000 Received: from m0353725.ppops.net (m0353725.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 37M8IeD2004669; Tue, 22 Aug 2023 08:44:10 GMT Received: from ppma13.dal12v.mail.ibm.com (dd.9e.1632.ip4.static.sl-reverse.com [50.22.158.221]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3smscs0n7p-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 22 Aug 2023 08:44:10 +0000 Received: from pps.filterd (ppma13.dal12v.mail.ibm.com [127.0.0.1]) by ppma13.dal12v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 37M85Sij007537; Tue, 22 Aug 2023 08:44:09 GMT Received: from smtprelay02.fra02v.mail.ibm.com ([9.218.2.226]) by ppma13.dal12v.mail.ibm.com (PPS) with ESMTPS id 3ska9k1ruq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 22 Aug 2023 08:44:09 +0000 Received: from smtpav06.fra02v.mail.ibm.com (smtpav06.fra02v.mail.ibm.com [10.20.54.105]) by smtprelay02.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 37M8i7uE590484 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 22 Aug 2023 08:44:07 GMT Received: from smtpav06.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A27EA2004D; Tue, 22 Aug 2023 08:44:07 +0000 (GMT) Received: from smtpav06.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 72BE020049; Tue, 22 Aug 2023 08:44:05 +0000 (GMT) Received: from [9.197.233.216] (unknown [9.197.233.216]) by smtpav06.fra02v.mail.ibm.com (Postfix) with ESMTP; Tue, 22 Aug 2023 08:44:05 +0000 (GMT) Message-ID: <31596d79-c607-8180-3399-5013c1b53aef@linux.ibm.com> Date: Tue, 22 Aug 2023 16:44:03 +0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Thunderbird/91.6.1 Content-Language: en-US Subject: [PATCH] vect: Replace DR_GROUP_STORE_COUNT with DR_GROUP_LAST_ELEMENT To: GCC Patches Cc: Richard Biener , Richard Sandiford , Segher Boessenkool , Peter Bergner X-TM-AS-GCONF: 00 X-Proofpoint-GUID: puqpa_B2eAM14FMcrfGu4aS0JSBlK93K X-Proofpoint-ORIG-GUID: G14RGNjPGyN_eCEecqyhPi_o0WOm-rVS X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-22_07,2023-08-18_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 malwarescore=0 mlxlogscore=999 priorityscore=1501 lowpriorityscore=0 mlxscore=0 suspectscore=0 bulkscore=0 adultscore=0 spamscore=0 clxscore=1015 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2306200000 definitions=main-2308220065 X-Spam-Status: No, score=-12.2 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_MSPIKE_H5, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: "Kewen.Lin via Gcc-patches" From: "Kewen.Lin" Reply-To: "Kewen.Lin" Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org Sender: "Gcc-patches" X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1774918228410327655 X-GMAIL-MSGID: 1774918228410327655 Hi, Now we use DR_GROUP_STORE_COUNT to record how many stores in a group have been transformed and only do the actual transform when encountering the last one. I'm making patches to move costing next to the transform code, it's awkward to use this DR_GROUP_STORE_COUNT for both costing and transforming. This patch is to introduce last_element to record the last element to be transformed in the group rather than to sum up the store number we have seen, then we can only check the given stmt is the last or not. It can make it work simply for both costing and transforming. Bootstrapped and regtested on x86_64-redhat-linux, aarch64-linux-gnu and powerpc64{,le}-linux-gnu. Is it ok for trunk? BR, Kewen ----- gcc/ChangeLog: * tree-vect-data-refs.cc (vect_set_group_last_element): New function. (vect_analyze_group_access): Call new function vect_set_group_last_element. * tree-vect-stmts.cc (vectorizable_store): Replace DR_GROUP_STORE_COUNT uses with DR_GROUP_LAST_ELEMENT. (vect_transform_stmt): Likewise. * tree-vect-slp.cc (vect_split_slp_store_group): Likewise. (vect_build_slp_instance): Likewise. * tree-vectorizer.h (DR_GROUP_LAST_ELEMENT): New macro. (DR_GROUP_STORE_COUNT): Remove. (class _stmt_vec_info::store_count): Remove. (class _stmt_vec_info::last_element): New class member. (vect_set_group_last_element): New function declaration. --- gcc/tree-vect-data-refs.cc | 30 ++++++++++++++++++++++++++++++ gcc/tree-vect-slp.cc | 13 +++++++++---- gcc/tree-vect-stmts.cc | 9 +++------ gcc/tree-vectorizer.h | 12 +++++++----- 4 files changed, 49 insertions(+), 15 deletions(-) -- 2.31.1 diff --git a/gcc/tree-vect-data-refs.cc b/gcc/tree-vect-data-refs.cc index 3e9a284666c..c4a495431d5 100644 --- a/gcc/tree-vect-data-refs.cc +++ b/gcc/tree-vect-data-refs.cc @@ -2832,6 +2832,33 @@ vect_analyze_group_access_1 (vec_info *vinfo, dr_vec_info *dr_info) return true; } +/* Given vectorization information VINFO, set the last element in the + group led by FIRST_STMT_INFO. For now, it's only used for loop + vectorization and stores, since for loop-vect the grouped stores + are only transformed till encounting its last one. */ + +void +vect_set_group_last_element (vec_info *vinfo, stmt_vec_info first_stmt_info) +{ + if (first_stmt_info + && is_a (vinfo) + && DR_IS_WRITE (STMT_VINFO_DATA_REF (first_stmt_info))) + { + stmt_vec_info stmt_info = DR_GROUP_NEXT_ELEMENT (first_stmt_info); + stmt_vec_info last_stmt_info = first_stmt_info; + while (stmt_info) + { + gimple *stmt = stmt_info->stmt; + gimple *last_stmt = last_stmt_info->stmt; + gcc_assert (gimple_bb (stmt) == gimple_bb (last_stmt)); + if (gimple_uid (stmt) > gimple_uid (last_stmt)) + last_stmt_info = stmt_info; + stmt_info = DR_GROUP_NEXT_ELEMENT (stmt_info); + } + DR_GROUP_LAST_ELEMENT (first_stmt_info) = last_stmt_info; + } +} + /* Analyze groups of accesses: check that DR_INFO belongs to a group of accesses of legal size, step, etc. Detect gaps, single element interleaving, and other special cases. Set grouped access info. @@ -2853,6 +2880,9 @@ vect_analyze_group_access (vec_info *vinfo, dr_vec_info *dr_info) } return false; } + + stmt_vec_info first_stmt_info = DR_GROUP_FIRST_ELEMENT (dr_info->stmt); + vect_set_group_last_element (vinfo, first_stmt_info); return true; } diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc index 89c3216afac..e9b64efe125 100644 --- a/gcc/tree-vect-slp.cc +++ b/gcc/tree-vect-slp.cc @@ -2827,7 +2827,8 @@ vect_find_first_scalar_stmt_in_slp (slp_tree node) Return the first stmt in the second group. */ static stmt_vec_info -vect_split_slp_store_group (stmt_vec_info first_vinfo, unsigned group1_size) +vect_split_slp_store_group (vec_info *vinfo, stmt_vec_info first_vinfo, + unsigned group1_size) { gcc_assert (DR_GROUP_FIRST_ELEMENT (first_vinfo) == first_vinfo); gcc_assert (group1_size > 0); @@ -2860,6 +2861,9 @@ vect_split_slp_store_group (stmt_vec_info first_vinfo, unsigned group1_size) /* DR_GROUP_GAP of the first group now has to skip over the second group too. */ DR_GROUP_GAP (first_vinfo) += group2_size; + vect_set_group_last_element (vinfo, first_vinfo); + vect_set_group_last_element (vinfo, group2); + if (dump_enabled_p ()) dump_printf_loc (MSG_NOTE, vect_location, "Split group into %d and %d\n", group1_size, group2_size); @@ -3321,7 +3325,7 @@ vect_build_slp_instance (vec_info *vinfo, if (dump_enabled_p ()) dump_printf_loc (MSG_NOTE, vect_location, "Splitting SLP group at stmt %u\n", i); - stmt_vec_info rest = vect_split_slp_store_group (stmt_info, + stmt_vec_info rest = vect_split_slp_store_group (vinfo, stmt_info, group1_size); bool res = vect_analyze_slp_instance (vinfo, bst_map, stmt_info, kind, max_tree_size, @@ -3334,7 +3338,8 @@ vect_build_slp_instance (vec_info *vinfo, || i - group1_size > 1)) { stmt_vec_info rest2 = rest; - rest = vect_split_slp_store_group (rest, i - group1_size); + rest = vect_split_slp_store_group (vinfo, rest, + i - group1_size); if (i - group1_size > 1) res |= vect_analyze_slp_instance (vinfo, bst_map, rest2, kind, max_tree_size, @@ -3361,7 +3366,7 @@ vect_build_slp_instance (vec_info *vinfo, dump_printf_loc (MSG_NOTE, vect_location, "Splitting SLP group at stmt %u\n", i); - stmt_vec_info rest = vect_split_slp_store_group (stmt_info, + stmt_vec_info rest = vect_split_slp_store_group (vinfo, stmt_info, group1_size); /* Loop vectorization cannot handle gaps in stores, make sure the split group appears as strided. */ diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc index 33f62b77710..1580a396301 100644 --- a/gcc/tree-vect-stmts.cc +++ b/gcc/tree-vect-stmts.cc @@ -8429,9 +8429,6 @@ vectorizable_store (vec_info *vinfo, else if (STMT_VINFO_SIMD_LANE_ACCESS_P (stmt_info) >= 3) return vectorizable_scan_store (vinfo, stmt_info, gsi, vec_stmt, ncopies); - if (STMT_VINFO_GROUPED_ACCESS (stmt_info)) - DR_GROUP_STORE_COUNT (DR_GROUP_FIRST_ELEMENT (stmt_info))++; - if (grouped_store) { /* FORNOW */ @@ -8439,8 +8436,8 @@ vectorizable_store (vec_info *vinfo, /* We vectorize all the stmts of the interleaving group when we reach the last stmt in the group. */ - if (DR_GROUP_STORE_COUNT (first_stmt_info) - < DR_GROUP_SIZE (first_stmt_info) + if (STMT_VINFO_GROUPED_ACCESS (stmt_info) + && stmt_info != DR_GROUP_LAST_ELEMENT (first_stmt_info) && !slp) { *vec_stmt = NULL; @@ -12497,7 +12494,7 @@ vect_transform_stmt (vec_info *vinfo, one are skipped, and there vec_stmt_info shouldn't be freed meanwhile. */ stmt_vec_info group_info = DR_GROUP_FIRST_ELEMENT (stmt_info); - if (DR_GROUP_STORE_COUNT (group_info) == DR_GROUP_SIZE (group_info)) + if (stmt_info == DR_GROUP_LAST_ELEMENT (group_info)) is_store = true; } else diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h index 53a3d78d545..6817e113b56 100644 --- a/gcc/tree-vectorizer.h +++ b/gcc/tree-vectorizer.h @@ -1285,11 +1285,12 @@ public: stmt_vec_info first_element; /* Pointer to the next element in the group. */ stmt_vec_info next_element; + /* Pointer to the last element in the group, for now it's only used + for loop-vect stores since they only get transformed till the + last one is being transformed. */ + stmt_vec_info last_element; /* The size of the group. */ unsigned int size; - /* For stores, number of stores from this group seen. We vectorize the last - one. */ - unsigned int store_count; /* For loads only, the gap from the previous load. For consecutive loads, GAP is 1. */ unsigned int gap; @@ -1500,10 +1501,10 @@ struct gather_scatter_info { (gcc_checking_assert ((S)->dr_aux.dr), (S)->first_element) #define DR_GROUP_NEXT_ELEMENT(S) \ (gcc_checking_assert ((S)->dr_aux.dr), (S)->next_element) +#define DR_GROUP_LAST_ELEMENT(S) \ + (gcc_checking_assert ((S)->dr_aux.dr), (S)->last_element) #define DR_GROUP_SIZE(S) \ (gcc_checking_assert ((S)->dr_aux.dr), (S)->size) -#define DR_GROUP_STORE_COUNT(S) \ - (gcc_checking_assert ((S)->dr_aux.dr), (S)->store_count) #define DR_GROUP_GAP(S) \ (gcc_checking_assert ((S)->dr_aux.dr), (S)->gap) @@ -2317,6 +2318,7 @@ extern tree vect_get_new_ssa_name (tree, enum vect_var_kind, extern tree vect_create_addr_base_for_vector_ref (vec_info *, stmt_vec_info, gimple_seq *, tree); +extern void vect_set_group_last_element (vec_info *, stmt_vec_info); /* In tree-vect-loop.cc. */ extern tree neutral_op_for_reduction (tree, code_helper, tree);