From patchwork Tue Aug 22 13:18:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Biener X-Patchwork-Id: 136539 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b82d:0:b0:3f2:4152:657d with SMTP id z13csp3619046vqi; Tue, 22 Aug 2023 06:19:02 -0700 (PDT) X-Google-Smtp-Source: AGHT+IF1XHWzPUF+wn/WanaYlvJQEV6nkU0vI2lKmklZ5zAq45NgkJatso2foDpgUM5q2NFRP99P X-Received: by 2002:a17:906:73da:b0:98d:5ae2:f1c with SMTP id n26-20020a17090673da00b0098d5ae20f1cmr7456373ejl.34.1692710341818; Tue, 22 Aug 2023 06:19:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1692710341; cv=none; d=google.com; s=arc-20160816; b=ECwi1JP3SvCbywpvXSAbC+Domfu/egTo0sKXMCw+QKoEjIrlIPY/OBOXTcjkpnxdcn z7ljANA8W4F+SxgurDFhNUp8ZZ3NRE2LTfhELhblBS0u8q3MTH5wKhqAFhrjEqYc1L4u Fk4OF6dVMv972da1fS3rN/NPTWHp9OMJNv1ZwqALjjVxZ6Th5gf7hySQ1T2OMVifKE/r fWHZKsD1ADtPnaDblkM60SNxH3FXqmiE0Q4afoClwg0yHjUBEO1NCWNDDD5YSi9fPIrk 29jIMmHKP/384FwkNXc4g2WHWK43KrSLsllp7u/6pXyXdHAq0F0w6qRWQD5oQNMbtkBS kBRw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:sender:errors-to:reply-to:from:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence :mime-version:user-agent:subject:cc:to:date:dmarc-filter :delivered-to:dkim-signature:dkim-filter; bh=tpBNY8o6ZjDtIqbFIeE7lsDD/RM3gqEZO+yM0NA3uGA=; fh=5EAlhau9oY3Ed/97atZIJrQg+mP4xpVQo+a0hK7qw9w=; b=s6c8EiZS2DnnW9PfmV5mCZB+I0mjVzS0dd+8jSMHn0vxZv00Qgf8Zp9TvoAwuqRi9S fR+/5lM/mrsMaDf8Qs/URPC0LUqP7uZ33mor63g53d0kLwenW44Qd7lNF95+Yfg+NDHV iv56nDzKnsO2QuTlb3npgk37jZVLhPJxXu1iCUjOf/ieFbaMPLXwbkpa/MKYqYYNzdJC mDePteMmMIequPUI4+QhTcDWT7rCoOZWGt4NtEJOHpLFyz4+kg8Gw0AleP2TfTA2ULJo p4Rk9uKMHlPB7R3Wcs7d7E3HALEaX1L32+SKnIQlVYsGykTyXlDQxDbiXZFDCTXSkD2W 4mww== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=gMYerlsM; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gnu.org Received: from server2.sourceware.org (ip-8-43-85-97.sourceware.org. [8.43.85.97]) by mx.google.com with ESMTPS id k16-20020a170906159000b00992feae3848si7691815ejd.38.2023.08.22.06.19.01 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Aug 2023 06:19:01 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) client-ip=8.43.85.97; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=gMYerlsM; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gnu.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 977183858D1E for ; Tue, 22 Aug 2023 13:19:00 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 977183858D1E DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1692710340; bh=tpBNY8o6ZjDtIqbFIeE7lsDD/RM3gqEZO+yM0NA3uGA=; h=Date:To:cc:Subject:List-Id:List-Unsubscribe:List-Archive: List-Post:List-Help:List-Subscribe:From:Reply-To:From; b=gMYerlsMlVHdK94NmSPkzI/DKEuYwJlfK7OUMlagsiMhNhkjJc5YcqvO5ldldFChB OZ0wzQPXrpi5En/HiJJzl1fp86sommRwnrWZKj7jXK3d1DJvXJPJxu7+1QUQUE5chd JDMDrS+NaxB2x7xeHKOqX8cmhTuZJBzaJf0A4DME= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by sourceware.org (Postfix) with ESMTPS id 246163858D1E for ; Tue, 22 Aug 2023 13:18:18 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 246163858D1E Received: from relay2.suse.de (relay2.suse.de [149.44.160.134]) by smtp-out1.suse.de (Postfix) with ESMTP id 30ECB22C36; Tue, 22 Aug 2023 13:18:17 +0000 (UTC) Received: from wotan.suse.de (wotan.suse.de [10.160.0.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by relay2.suse.de (Postfix) with ESMTPS id 2765E2C145; Tue, 22 Aug 2023 13:18:17 +0000 (UTC) Date: Tue, 22 Aug 2023 13:18:17 +0000 (UTC) To: gcc-patches@gcc.gnu.org cc: linkw@linux.ibm.com Subject: [PATCH] Simplify intereaved store vectorization processing User-Agent: Alpine 2.22 (LSU 394 2020-01-19) MIME-Version: 1.0 X-Spam-Status: No, score=-10.6 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, MISSING_MID, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Richard Biener via Gcc-patches From: Richard Biener Reply-To: Richard Biener Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org Sender: "Gcc-patches" Message-Id: <20230822131900.977183858D1E@sourceware.org> X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1774935439564997902 X-GMAIL-MSGID: 1774935439564997902 When doing interleaving we perform code generation when visiting the last store of a chain. We keep track of this via DR_GROUP_STORE_COUNT, the following localizes this to the caller of vectorizable_store, also avoing redundant non-processing of the other stores. Bootstrapped and tested on x86_64-unknown-linux-gnu, pushed. * tree-vect-stmts.cc (vectorizable_store): Do not bump DR_GROUP_STORE_COUNT here. Remove early out. (vect_transform_stmt): Only call vectorizable_store on the last element of an interleaving chain. --- gcc/tree-vect-stmts.cc | 40 ++++++++++++++-------------------------- 1 file changed, 14 insertions(+), 26 deletions(-) diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc index 33f62b77710..43502dc169f 100644 --- a/gcc/tree-vect-stmts.cc +++ b/gcc/tree-vect-stmts.cc @@ -8429,24 +8429,11 @@ vectorizable_store (vec_info *vinfo, else if (STMT_VINFO_SIMD_LANE_ACCESS_P (stmt_info) >= 3) return vectorizable_scan_store (vinfo, stmt_info, gsi, vec_stmt, ncopies); - if (STMT_VINFO_GROUPED_ACCESS (stmt_info)) - DR_GROUP_STORE_COUNT (DR_GROUP_FIRST_ELEMENT (stmt_info))++; - if (grouped_store) { /* FORNOW */ gcc_assert (!loop || !nested_in_vect_loop_p (loop, stmt_info)); - /* We vectorize all the stmts of the interleaving group when we - reach the last stmt in the group. */ - if (DR_GROUP_STORE_COUNT (first_stmt_info) - < DR_GROUP_SIZE (first_stmt_info) - && !slp) - { - *vec_stmt = NULL; - return true; - } - if (slp) { grouped_store = false; @@ -12487,21 +12474,22 @@ vect_transform_stmt (vec_info *vinfo, break; case store_vec_info_type: - done = vectorizable_store (vinfo, stmt_info, - gsi, &vec_stmt, slp_node, NULL); - gcc_assert (done); - if (STMT_VINFO_GROUPED_ACCESS (stmt_info) && !slp_node) + if (STMT_VINFO_GROUPED_ACCESS (stmt_info) + && !slp_node + && (++DR_GROUP_STORE_COUNT (DR_GROUP_FIRST_ELEMENT (stmt_info)) + < DR_GROUP_SIZE (DR_GROUP_FIRST_ELEMENT (stmt_info)))) + /* In case of interleaving, the whole chain is vectorized when the + last store in the chain is reached. Store stmts before the last + one are skipped, and there vec_stmt_info shouldn't be freed + meanwhile. */ + ; + else { - /* In case of interleaving, the whole chain is vectorized when the - last store in the chain is reached. Store stmts before the last - one are skipped, and there vec_stmt_info shouldn't be freed - meanwhile. */ - stmt_vec_info group_info = DR_GROUP_FIRST_ELEMENT (stmt_info); - if (DR_GROUP_STORE_COUNT (group_info) == DR_GROUP_SIZE (group_info)) - is_store = true; + done = vectorizable_store (vinfo, stmt_info, + gsi, &vec_stmt, slp_node, NULL); + gcc_assert (done); + is_store = true; } - else - is_store = true; break; case condition_vec_info_type: