From patchwork Thu Aug 24 12:41:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Biener X-Patchwork-Id: 136816 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a7d1:0:b0:3f2:4152:657d with SMTP id p17csp1090609vqm; Thu, 24 Aug 2023 05:42:58 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEu/or/wQWfVKfxrqWeG8xyFnYZzt0NOTFPt2igRX3im2jDuclE7q8ol+pHT7khSZdDKZr0 X-Received: by 2002:a17:907:b09:b0:9a2:474:4aa0 with SMTP id h9-20020a1709070b0900b009a204744aa0mr1508150ejl.48.1692880978398; Thu, 24 Aug 2023 05:42:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1692880978; cv=none; d=google.com; s=arc-20160816; b=Cx1Oh8bpHecWLC0IbYfU1fZHrXYVSPDX4NRbb+f17BS0JrDk8NhFpgu07d2jsOc4+Z vouiwcXJtD8OI6tTOw4pexqOKgxcKll0/uLS/cl1fmt9VSaxb5MzkEPES8HHxzpMsILv uL3X6dzHUsYJaYcct0M5zaNMF999tMTar4YX/oNsCf8Sc7q9k8fsEnfstEYBQPj5b7zb Do4YUyzJigvF8vLtOEgwrhOcfsUrD/dbrx/MQCd+rZlHtrh1g0ESD7a+27cxKcKVNBpu X+x3EZ+6cx07DJSwTibgND+Rwc2p8FyBbeZp9UWGQt+KYeGCkWXljIUQ9qunPD9UEFX+ mFTg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:sender:errors-to:reply-to:from:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence :mime-version:user-agent:subject:to:date:dmarc-filter:delivered-to :dkim-signature:dkim-filter; bh=gkynkec21D2bjcmSFCVOjzPf6AnW4bC7hA4JphKFVKw=; fh=etb9MYHN7HLF/sff76ICVdPeKiI8ZsjoOL2bcdG0aog=; b=c1PRwEc+3LcgRbha6HGxmjfaD0LA+KR9vLSP7OBZv9Tu3/glDfrfvmoJcSGRjW15Xv esZR8szlkZE4dH6XBt2iPMRwexmhar3r31Wb2qjpyZv7a6KXAAzpjseee0TINsJ5ipgj 4gEqdbKaTqfPiHVD8nwhsIp+g6U2xdOz+TL65afzmSJlfvNL8faQ5MxMIU85hV85+LyU 0DFf0f/YKgr0TZQL2kad4R9zKz/RhBytEE20dB9IalOmqR4BP5l3KVpuRan/KmqvhsaA JTVkpWbq3Kasu+LGcnf/HTHS319vZ9sfO6LISGKmHK4KWArJZHhl+QTNCXWRLN+xNIOC TbNA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=M8tt6GFv; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gnu.org Received: from server2.sourceware.org (ip-8-43-85-97.sourceware.org. [8.43.85.97]) by mx.google.com with ESMTPS id n9-20020a1709062bc900b0099bce4aa78dsi10517942ejg.865.2023.08.24.05.42.58 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Aug 2023 05:42:58 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) client-ip=8.43.85.97; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=M8tt6GFv; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gnu.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 86E373853D1F for ; Thu, 24 Aug 2023 12:42:31 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 86E373853D1F DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1692880951; bh=gkynkec21D2bjcmSFCVOjzPf6AnW4bC7hA4JphKFVKw=; h=Date:To:Subject:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=M8tt6GFvX80f67g6suSAqKowT1F9iwrxdb8OIUllSKJdXWP2scYBOLJfKuviOAQ5W hPTkYwPQZG1BKXBIImcgVP723yJ990wZnV/6lApZUiwsZUoVfGj6SRAAKC5KyMJ/Ux sk3/X6gkl6OnZM0vcG7VZEdIYT+Stm043pZU2Oqw= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtp-out1.suse.de (smtp-out1.suse.de [IPv6:2001:67c:2178:6::1c]) by sourceware.org (Postfix) with ESMTPS id DB2CE3856961 for ; Thu, 24 Aug 2023 12:41:46 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org DB2CE3856961 Received: from relay2.suse.de (relay2.suse.de [149.44.160.134]) by smtp-out1.suse.de (Postfix) with ESMTP id 0341022BB2 for ; Thu, 24 Aug 2023 12:41:46 +0000 (UTC) Received: from wotan.suse.de (wotan.suse.de [10.160.0.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by relay2.suse.de (Postfix) with ESMTPS id F2F552C143 for ; Thu, 24 Aug 2023 12:41:45 +0000 (UTC) Date: Thu, 24 Aug 2023 12:41:45 +0000 (UTC) To: gcc-patches@gcc.gnu.org Subject: [PATCH] Fix confusion about load_p in vect_build_slp_tree_1 User-Agent: Alpine 2.22 (LSU 394 2020-01-19) MIME-Version: 1.0 X-Spam-Status: No, score=-10.4 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_NUMSUBJECT, MISSING_MID, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Richard Biener via Gcc-patches From: Richard Biener Reply-To: Richard Biener Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org Sender: "Gcc-patches" Message-Id: <20230824124231.86E373853D1F@sourceware.org> X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1775114364511102739 X-GMAIL-MSGID: 1775114364511102739 load_p is set and used as to whether the stmt is a memory operation, not whether it is only a load. The following renames it to ldst_p to avoid this confusion. It also replaces checking for a VUSE with checking STMT_VINFO_DATA_REF since VUSE checking doesn't work for pattern matched stores where no virtual operands are present. Where we want to distinguish between loads and stores we then check DR_IS_READ/WRITE. I've made a classification mistake with .MASK_STORE support and this hits other complications when dealing with single-lane SLP. Bootstrapped and tested on x86_64-unknown-linux-gnu, pushed. * tree-vect-slp.cc (vect_build_slp_tree_1): Rename load_p to ldst_p, fix mistakes and rely on STMT_VINFO_DATA_REF. --- gcc/tree-vect-slp.cc | 42 ++++++++++++++++++++++++------------------ 1 file changed, 24 insertions(+), 18 deletions(-) diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc index 0b1c2233017..0cf6e02285e 100644 --- a/gcc/tree-vect-slp.cc +++ b/gcc/tree-vect-slp.cc @@ -984,7 +984,7 @@ vect_build_slp_tree_1 (vec_info *vinfo, unsigned char *swap, bool need_same_oprnds = false; tree vectype = NULL_TREE, first_op1 = NULL_TREE; stmt_vec_info first_load = NULL, prev_first_load = NULL; - bool first_stmt_load_p = false, load_p = false; + bool first_stmt_ldst_p = false, ldst_p = false; bool first_stmt_phi_p = false, phi_p = false; bool maybe_soft_fail = false; tree soft_fail_nunits_vectype = NULL_TREE; @@ -1074,9 +1074,12 @@ vect_build_slp_tree_1 (vec_info *vinfo, unsigned char *swap, if (cfn == CFN_MASK_LOAD || cfn == CFN_GATHER_LOAD || cfn == CFN_MASK_GATHER_LOAD) - load_p = true; + ldst_p = true; else if (cfn == CFN_MASK_STORE) - rhs_code = CFN_MASK_STORE; + { + ldst_p = true; + rhs_code = CFN_MASK_STORE; + } else if ((internal_fn_p (cfn) && !vectorizable_internal_fn_p (as_internal_fn (cfn))) || gimple_call_tail_p (call_stmt) @@ -1102,7 +1105,7 @@ vect_build_slp_tree_1 (vec_info *vinfo, unsigned char *swap, else { rhs_code = gimple_assign_rhs_code (stmt); - load_p = gimple_vuse (stmt); + ldst_p = STMT_VINFO_DATA_REF (stmt_info) != nullptr; } /* Check the operation. */ @@ -1110,7 +1113,7 @@ vect_build_slp_tree_1 (vec_info *vinfo, unsigned char *swap, { *node_vectype = vectype; first_stmt_code = rhs_code; - first_stmt_load_p = load_p; + first_stmt_ldst_p = ldst_p; first_stmt_phi_p = phi_p; /* Shift arguments should be equal in all the packed stmts for a @@ -1144,7 +1147,7 @@ vect_build_slp_tree_1 (vec_info *vinfo, unsigned char *swap, need_same_oprnds = true; first_op1 = gimple_assign_rhs2 (stmt); } - else if (!load_p + else if (!ldst_p && rhs_code == BIT_FIELD_REF) { tree vec = TREE_OPERAND (gimple_assign_rhs1 (stmt), 0); @@ -1207,7 +1210,7 @@ vect_build_slp_tree_1 (vec_info *vinfo, unsigned char *swap, || rhs_code == INDIRECT_REF || rhs_code == COMPONENT_REF || rhs_code == MEM_REF))) - || first_stmt_load_p != load_p + || first_stmt_ldst_p != ldst_p || first_stmt_phi_p != phi_p) { if (dump_enabled_p ()) @@ -1222,7 +1225,7 @@ vect_build_slp_tree_1 (vec_info *vinfo, unsigned char *swap, continue; } - if (!load_p + if (!ldst_p && first_stmt_code == BIT_FIELD_REF && (TREE_OPERAND (gimple_assign_rhs1 (first_stmt_info->stmt), 0) != TREE_OPERAND (gimple_assign_rhs1 (stmt_info->stmt), 0))) @@ -1291,12 +1294,13 @@ vect_build_slp_tree_1 (vec_info *vinfo, unsigned char *swap, /* Grouped store or load. */ if (STMT_VINFO_GROUPED_ACCESS (stmt_info)) { - if (!load_p) + gcc_assert (ldst_p); + if (DR_IS_WRITE (STMT_VINFO_DATA_REF (stmt_info))) { /* Store. */ gcc_assert (rhs_code == CFN_MASK_STORE - || REFERENCE_CLASS_P (lhs)); - ; + || REFERENCE_CLASS_P (lhs) + || DECL_P (lhs)); } else { @@ -1321,10 +1325,11 @@ vect_build_slp_tree_1 (vec_info *vinfo, unsigned char *swap, else prev_first_load = first_load; } - } /* Grouped access. */ - else + } + /* Non-grouped store or load. */ + else if (ldst_p) { - if (load_p + if (DR_IS_READ (STMT_VINFO_DATA_REF (stmt_info)) && rhs_code != CFN_GATHER_LOAD && rhs_code != CFN_MASK_GATHER_LOAD /* Not grouped loads are handled as externals for BB @@ -1345,10 +1350,11 @@ vect_build_slp_tree_1 (vec_info *vinfo, unsigned char *swap, matches[0] = false; return false; } - - /* Not memory operation. */ - if (!load_p - && !phi_p + } + /* Not memory operation. */ + else + { + if (!phi_p && rhs_code.is_tree_code () && TREE_CODE_CLASS (tree_code (rhs_code)) != tcc_binary && TREE_CODE_CLASS (tree_code (rhs_code)) != tcc_unary