From patchwork Fri Oct 20 09:23:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Biener X-Patchwork-Id: 155915 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2010:b0:403:3b70:6f57 with SMTP id fe16csp928425vqb; Fri, 20 Oct 2023 02:24:10 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHoUSVT8DvPQO20eMCUjM4Caruf2boDRfJXXM2Vy1JLTZQoqF/KFR4gJ75sVdFY2Loesous X-Received: by 2002:a67:e047:0:b0:457:c415:a495 with SMTP id n7-20020a67e047000000b00457c415a495mr1290140vsl.10.1697793850229; Fri, 20 Oct 2023 02:24:10 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1697793850; cv=pass; d=google.com; s=arc-20160816; b=ysabbinKXYrEBRb6xF3mUqxCub3XW+dHLSbUCDeq4uWiBHQ+B06iP3q2A3468tcV/I 9nWr+Zt5SxIxxjgdkVcOSlI3MCoYXM90R7UrkwzmpOce8r4x9BwuP3CgSQ8ucHcq48fh d2YxBjgAZmRDyOEO6+ytKojs5kRi3kHAgutTnYV1JqhSAGd5UBjC4OfsFZiY43JJT3ZJ uuEIgkq66F4gpalUJn8ep1Jf9zwsns6PmppOTP/SWnoXzsAUE56qBhjEKxc2SYNdvop9 JY6oRqdL0ZyIrcJV+uN73PPHo01kbn7OESFzJoCRx8K4twh5ihfr66yonLV6OM0y1gGQ Uqfw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:message-id:mime-version:subject :to:from:date:dkim-signature:dkim-signature:arc-filter:dmarc-filter :delivered-to; bh=JtxE+gboQ84d9SH6kGd98tSxbYy+tk0erzye7FmrJzg=; fh=hPrbWPhweUx4V0GV9uXJqbyAzg2ABmTz7kczrAQqMmM=; b=ls/vA9lLw+WsK6gCUSSCfIjTuNY/UKbrI/h/4r1wCphReaf0s+0N1d9ZpWLb47XM+b 9j9R6pMdFB9qXDmjWsJtewTE4oml7m1R7zVFxyoBr+8ASz0EvJHVnuXP1L2VdnGqImGB 1aFiWDcGcl9DYpjyCmH58IUWsMBIx/nd1Gwoe7gDpKESR9Glxn19Fm6niysIoq5GSCqv eDDodJn/S0NmpELRSeja54AMta0UaLwPJrjeLlD6Fm0CpmaSH16NqERgIwyrbrVUnLKm qLr6yPpiaLKy4TLGWypq2aZ21+4evIzTIQLYP1CMDN3MTyvBK5FWNqIkv0yDKPFX11oA BXWg== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=T5iFfI9Z; dkim=neutral (no key) header.i=@suse.de; arc=pass (i=1); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: from server2.sourceware.org (server2.sourceware.org. [2620:52:3:1:0:246e:9693:128c]) by mx.google.com with ESMTPS id bq32-20020a05620a46a000b00773e066060csi1004904qkb.296.2023.10.20.02.24.10 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Oct 2023 02:24:10 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) client-ip=2620:52:3:1:0:246e:9693:128c; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=T5iFfI9Z; dkim=neutral (no key) header.i=@suse.de; arc=pass (i=1); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 0517A3858024 for ; Fri, 20 Oct 2023 09:24:10 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtp-out1.suse.de (smtp-out1.suse.de [IPv6:2001:67c:2178:6::1c]) by sourceware.org (Postfix) with ESMTPS id 0C9F63858D1E for ; Fri, 20 Oct 2023 09:23:29 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 0C9F63858D1E Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=suse.de ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 0C9F63858D1E Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2001:67c:2178:6::1c ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1697793810; cv=none; b=f7ppxgmx+58zZM8e+sK67VQ9FdLvS9Jg86TnsXq/PWdsAuUr08rOqnptFbw/zsmweWLSaOxvMk5SFB415AjrCLbZU/htkF+tFYD9/y4OGFQqOVyw2H1AB+jOFUHYi6mzp8K3ak6zaEinIONR1NuarLoQvu98UBwqGq25bgZjiPk= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1697793810; c=relaxed/simple; bh=vBEnBF5J32v/O4vz1nBXz1xdugGnSg8LxQQk+b5TTKo=; h=DKIM-Signature:DKIM-Signature:Date:From:To:Subject:MIME-Version: Message-Id; b=vO3ClRdnLNcpjhgzVdOi2Lp4kyaROp6FTQXiozXXboNmahA8fnYp17vpSAXayrtWjYziYNfodApZnmHVOyPfMfbE8XmouEh6MnTsF6B4MeDiO7tbLZ06I497/Oy6f4P/yJJwLbbxHncP1SCzNS8RiVjcNg/47x14BvSz19Ah/JQ= ARC-Authentication-Results: i=1; server2.sourceware.org Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id D8C87219CD for ; Fri, 20 Oct 2023 09:23:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1697793807; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version:content-type:content-type; bh=JtxE+gboQ84d9SH6kGd98tSxbYy+tk0erzye7FmrJzg=; b=T5iFfI9Z7cX4OlDUDZ/ojtv6hqHYKXUzfXhcNuugotQKhhW6yWooyx8v8gkgfAMXPWa+NH s7Nwltr4bfaFsOo8KlieP022LPQj0nbDbYyu9VCYgiw7y3iTVzz2F7mLYjk0zA1yzRdLPs j7a8bW+vYvKeC2mUHcP+pH8LF4+PcTk= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1697793807; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version:content-type:content-type; bh=JtxE+gboQ84d9SH6kGd98tSxbYy+tk0erzye7FmrJzg=; b=A7n3owHxPGbtHb0JjCCHgZBw3NL0JSoLzInH+59278ysIx9E8J/PMDoj6yFC8avpD40IC5 uKZyusu4BvJnuXCQ== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id C732A13584 for ; Fri, 20 Oct 2023 09:23:27 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id P45vLw9HMmUaOgAAMHmgww (envelope-from ) for ; Fri, 20 Oct 2023 09:23:27 +0000 Date: Fri, 20 Oct 2023 11:23:27 +0200 (CEST) From: Richard Biener To: gcc-patches@gcc.gnu.org Subject: [PATCH] Rewrite more refs for epilogue vectorization MIME-Version: 1.0 Message-Id: <20231020092327.C732A13584@imap2.suse-dmz.suse.de> Authentication-Results: smtp-out1.suse.de; none X-Spam-Level: X-Spam-Score: -7.10 X-Spamd-Result: default: False [-7.10 / 50.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-3.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[gcc-patches@gcc.gnu.org]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROMTLD(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; BAYES_HAM(-3.00)[100.00%] X-Spam-Status: No, score=-11.2 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780265884137036533 X-GMAIL-MSGID: 1780265884137036533 The following makes sure to rewrite all gather/scatter detected by dataref analysis plus stmts classified as VMAT_GATHER_SCATTER. Maybe we need to rewrite all refs, the following covers the cases I've run into now. Bootstrapped and tested on x86_64-unknown-linux-gnu, pushed. * tree-vect-loop.cc (update_epilogue_loop_vinfo): Rewrite both STMT_VINFO_GATHER_SCATTER_P and VMAT_GATHER_SCATTER stmt refs. --- gcc/tree-vect-loop.cc | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc index 8877ebde246..4a8b0a18800 100644 --- a/gcc/tree-vect-loop.cc +++ b/gcc/tree-vect-loop.cc @@ -11361,8 +11361,12 @@ update_epilogue_loop_vinfo (class loop *epilogue, tree advance) /* Data references for gather loads and scatter stores do not use the updated offset we set using ADVANCE. Instead we have to make sure the reference in the data references point to the corresponding copy of - the original in the epilogue. */ - if (STMT_VINFO_GATHER_SCATTER_P (vect_stmt_to_vectorize (stmt_vinfo))) + the original in the epilogue. Make sure to update both + gather/scatters recognized by dataref analysis and also other + refs that get_load_store_type classified as VMAT_GATHER_SCATTER. */ + auto vstmt_vinfo = vect_stmt_to_vectorize (stmt_vinfo); + if (STMT_VINFO_MEMORY_ACCESS_TYPE (vstmt_vinfo) == VMAT_GATHER_SCATTER + || STMT_VINFO_GATHER_SCATTER_P (vstmt_vinfo)) { DR_REF (dr) = simplify_replace_tree (DR_REF (dr), NULL_TREE, NULL_TREE, @@ -11371,9 +11375,6 @@ update_epilogue_loop_vinfo (class loop *epilogue, tree advance) = simplify_replace_tree (DR_BASE_ADDRESS (dr), NULL_TREE, NULL_TREE, &find_in_mapping, &mapping); } - else - gcc_assert (STMT_VINFO_MEMORY_ACCESS_TYPE (vect_stmt_to_vectorize (stmt_vinfo)) - != VMAT_GATHER_SCATTER); DR_STMT (dr) = STMT_VINFO_STMT (stmt_vinfo); stmt_vinfo->dr_aux.stmt = stmt_vinfo; /* The vector size of the epilogue is smaller than that of the main loop