From patchwork Mon Jan 22 09:02:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Biener X-Patchwork-Id: 189973 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:2bc4:b0:101:a8e8:374 with SMTP id hx4csp2447353dyb; Mon, 22 Jan 2024 01:04:09 -0800 (PST) X-Google-Smtp-Source: AGHT+IGpZz0QUyNTIZ6gvG2aGaG9V3QYE+jRBK6ds2smWbvEyTsnljcGQ4kMle7HTJ7sIl7LBIwF X-Received: by 2002:ac8:5a50:0:b0:429:c6e3:6bd9 with SMTP id o16-20020ac85a50000000b00429c6e36bd9mr7687177qta.15.1705914249300; Mon, 22 Jan 2024 01:04:09 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1705914249; cv=pass; d=google.com; s=arc-20160816; b=NnrjYkBXom4kbw2Er0L3gChNxqHv+ONNjqHmy5wxFSixGbc3s0JG/IDm77a5wTIKBS bqsvJkC7pGz4FGsLGw+q0TqusFNsJQIEbJPui1OXqkOSJe5xrHR7yBAmaGaTsxnuuvUj gPBGLq5MrGz3UL0nKtoTtgKlHqoC8By84SbwWj9Mwt/t9oCk53TggG2D2CuxttIppZHe FGUQETuDwTsT8J3RtExbow19Aq+Kh7nB+diSVcwl5AZuJonVnPq2/hukJIzJjtTI10Fd L1a+xBYm3yx6aud3SIbVnvfecQza5ALXURmVXiPQL5Q0Ejgj2ENXc6dt44bIoeX2iKmE kqVw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:errors-to:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence:mime-version :subject:to:from:date:dkim-signature:dkim-signature:dkim-signature :dkim-signature:arc-filter:dmarc-filter:delivered-to; bh=DqewgRezKtGRDZhbFwF/dW1i6YdZCSqeRW/DXZ7FDv4=; fh=hPrbWPhweUx4V0GV9uXJqbyAzg2ABmTz7kczrAQqMmM=; b=Uvs/GPtdj73BiW8CmKmfWrMj8v3BI0nLOKTZdngmr9L6LQbxldTXq1iMVPTKsUfxAQ wDZ3sAERR+ghWVnIzU4WB+cLEVHzI7CGg1pmXDzihlzpQMeXegrFFDmPpJLTu7tGpOwI i3XiE9wdpmxuuKnI0Yav78Mx5rMzj361aIGzhufb2WW39y7d7GBZ2YvMkWmtolbAU2yt 83841HDqWqE7OMezU8Y3t2PiY0P9EllvNFHoK97By6N1m9/8qdTjuqGMNPC9jgOapz2C N9FOocvPPtzjl6XXng4C+M3X+F3er66wkZbZFgeipWjqQLg3SYInR6/z+ELPZf0va1S+ CGcw== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=Y88awFRm; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=Y88awFRm; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519; arc=pass (i=1); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: from server2.sourceware.org (server2.sourceware.org. [8.43.85.97]) by mx.google.com with ESMTPS id i10-20020ac85e4a000000b004298facb58asi5064429qtx.768.2024.01.22.01.04.09 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 Jan 2024 01:04:09 -0800 (PST) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) client-ip=8.43.85.97; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=Y88awFRm; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=Y88awFRm; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519; arc=pass (i=1); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 08E533858419 for ; Mon, 22 Jan 2024 09:04:09 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtp-out2.suse.de (smtp-out2.suse.de [IPv6:2a07:de40:b251:101:10:150:64:2]) by sourceware.org (Postfix) with ESMTPS id CCB7A3858C2F for ; Mon, 22 Jan 2024 09:03:09 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org CCB7A3858C2F Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=suse.de ARC-Filter: OpenARC Filter v1.0.0 sourceware.org CCB7A3858C2F Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2a07:de40:b251:101:10:150:64:2 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1705914195; cv=none; b=s6cmpd1B+D3+eLl8S4nenrF9Un4r9IFK6BRFHVj9hPCwpgWKBStmFO/qB81vi7DbAtGHDyRKhm4S8p27xrhLLBeayiNYMT0DTu7ZWZDWnSDgRnQ3tenP+40Xk4TU3eBXGRSSL9/5GHxtccebaGqbiKS6FmmNBT3dmsyEqNE06ng= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1705914195; c=relaxed/simple; bh=RF4LqTMmRY9f/7eZML7QsF+TeuaYVj+Zb3MCAQnD2Ms=; h=DKIM-Signature:DKIM-Signature:DKIM-Signature:DKIM-Signature:Date: From:To:Subject:MIME-Version; b=xXOcSj9b2KcV3c1LMTzsxUi6fQe8xecAvMqck1q6nf8vdpSFLnFUFRYD9mPNYtifrAM24rTkXywQeSPGtF1cO4ico+PzLp73pRhuZNkrv4942sN86wOstPpwFmSxe33xBIE4uK18+FoOjDPhbiVimLXw7F27+cS6YqKx7ovhnDg= ARC-Authentication-Results: i=1; server2.sourceware.org Received: from [10.168.4.150] (unknown [10.168.4.150]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id BFC061FB9E for ; Mon, 22 Jan 2024 09:03:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1705914188; h=from:from:reply-to:date:date:to:to:cc:mime-version:mime-version: content-type:content-type; bh=DqewgRezKtGRDZhbFwF/dW1i6YdZCSqeRW/DXZ7FDv4=; b=Y88awFRmq4fblBbErbkb9U/QYdPEpmc1mg5XGHihJRI2/B748do2vKtIJhn8JLVwIcFyT1 hkB7hBRZ4v6TkEdmaVPFP81jcmE7B1ML6wgKHs3QcaUUIJyRWIFtbxeL5fu8zg41ebQfLD Vni16s5SNvyl1B2ftFaFNbwjCO7xW+A= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1705914188; h=from:from:reply-to:date:date:to:to:cc:mime-version:mime-version: content-type:content-type; bh=DqewgRezKtGRDZhbFwF/dW1i6YdZCSqeRW/DXZ7FDv4=; b=xXOOTyqUYSDEYFAPJRTYsSzEQMvobtLD7vllyxftuZ0Bgr5MeP22lwjq+I+sLZL2xWl2ij PVhn8htXNn2VXmDQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1705914188; h=from:from:reply-to:date:date:to:to:cc:mime-version:mime-version: content-type:content-type; bh=DqewgRezKtGRDZhbFwF/dW1i6YdZCSqeRW/DXZ7FDv4=; b=Y88awFRmq4fblBbErbkb9U/QYdPEpmc1mg5XGHihJRI2/B748do2vKtIJhn8JLVwIcFyT1 hkB7hBRZ4v6TkEdmaVPFP81jcmE7B1ML6wgKHs3QcaUUIJyRWIFtbxeL5fu8zg41ebQfLD Vni16s5SNvyl1B2ftFaFNbwjCO7xW+A= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1705914188; h=from:from:reply-to:date:date:to:to:cc:mime-version:mime-version: content-type:content-type; bh=DqewgRezKtGRDZhbFwF/dW1i6YdZCSqeRW/DXZ7FDv4=; b=xXOOTyqUYSDEYFAPJRTYsSzEQMvobtLD7vllyxftuZ0Bgr5MeP22lwjq+I+sLZL2xWl2ij PVhn8htXNn2VXmDQ== Date: Mon, 22 Jan 2024 10:02:04 +0100 (CET) From: Richard Biener To: gcc-patches@gcc.gnu.org Subject: [PATCH] Refactor exit PHI handling in vectorizer epilogue peeling MIME-Version: 1.0 Authentication-Results: smtp-out2.suse.de; none X-Spam-Score: 5.22 X-Spamd-Result: default: False [5.22 / 50.00]; TO_DN_NONE(0.00)[]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; BAYES_HAM(-3.00)[100.00%]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_SPAM_SHORT(2.99)[0.997]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_ONE(0.00)[1]; MISSING_MID(2.50)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; NEURAL_SPAM_LONG(2.83)[0.807]; FUZZY_BLOCKED(0.00)[rspamd.com] X-Spam-Status: No, score=-10.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, MISSING_MID, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org Message-Id: <20240122090409.08E533858419@sourceware.org> X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788780740295217106 X-GMAIL-MSGID: 1788780740295217106 This refactors the handling of PHIs inbetween the main and the epilogue loop. Instead of trying to handle the multiple exit and original single exit case together the following separates these cases resulting in much easier to understand code. Bootstrap and regtest running on x86_64-unknown-linux-gnu. Richard. * tree-vect-loop-manip.cc (slpeel_tree_duplicate_loop_to_edge_cfg): Separate single and multi-exit case when creating PHIs between the main and epilogue. --- gcc/tree-vect-loop-manip.cc | 135 ++++++++++++++++++++---------------- 1 file changed, 74 insertions(+), 61 deletions(-) diff --git a/gcc/tree-vect-loop-manip.cc b/gcc/tree-vect-loop-manip.cc index eacbc022549..873a28d7c56 100644 --- a/gcc/tree-vect-loop-manip.cc +++ b/gcc/tree-vect-loop-manip.cc @@ -1682,77 +1682,60 @@ slpeel_tree_duplicate_loop_to_edge_cfg (class loop *loop, edge loop_exit, } /* Create the merge PHI nodes in new_preheader and populate the - arguments for the main exit. */ - for (auto gsi_from = gsi_start_phis (loop->header), - gsi_to = gsi_start_phis (new_loop->header); - !gsi_end_p (gsi_from) && !gsi_end_p (gsi_to); - gsi_next (&gsi_from), gsi_next (&gsi_to)) + arguments for the exits. */ + if (multiple_exits_p) { - gimple *from_phi = gsi_stmt (gsi_from); - gimple *to_phi = gsi_stmt (gsi_to); - tree new_arg = PHI_ARG_DEF_FROM_EDGE (from_phi, - loop_latch_edge (loop)); - - /* Check if we've already created a new phi node during edge - redirection. If we have, only propagate the value - downwards in case there is no merge block. */ - tree *res; - if ((res = new_phi_args.get (new_arg))) + for (auto gsi_from = gsi_start_phis (loop->header), + gsi_to = gsi_start_phis (new_loop->header); + !gsi_end_p (gsi_from) && !gsi_end_p (gsi_to); + gsi_next (&gsi_from), gsi_next (&gsi_to)) { - if (multiple_exits_p) - new_arg = *res; + gimple *from_phi = gsi_stmt (gsi_from); + gimple *to_phi = gsi_stmt (gsi_to); + + /* When the vector loop is peeled then we need to use the + value at start of the loop, otherwise the main loop exit + should use the final iter value. */ + tree new_arg; + if (peeled_iters) + new_arg = gimple_phi_result (from_phi); else - { - adjust_phi_and_debug_stmts (to_phi, loop_entry, *res); - continue; - } - } - /* If we have multiple exits and the vector loop is peeled then we - need to use the value at start of loop. If we're looking at - virtual operands we have to keep the original link. Virtual - operands don't all become the same because we'll corrupt the - vUSE chains among others. */ - if (peeled_iters) - { - tree tmp_arg = gimple_phi_result (from_phi); - /* Similar to the single exit case, If we have an existing - LCSSA variable thread through the original value otherwise - skip it and directly use the final value. */ - if ((res = new_phi_args.get (tmp_arg))) + new_arg = PHI_ARG_DEF_FROM_EDGE (from_phi, + loop_latch_edge (loop)); + + /* Check if we've already created a new phi node during edge + redirection and re-use it if so. Otherwise create a + LC PHI node to feed the merge PHI. */ + tree *res; + if (virtual_operand_p (new_arg)) + /* Use the existing virtual LC SSA from exit block. */ + new_arg = gimple_phi_result + (get_virtual_phi (main_loop_exit_block)); + else if ((res = new_phi_args.get (new_arg))) new_arg = *res; - else if (!virtual_operand_p (new_arg)) - new_arg = tmp_arg; - } - - tree new_res = copy_ssa_name (gimple_phi_result (from_phi)); - gphi *lcssa_phi = create_phi_node (new_res, new_preheader); - - /* Otherwise, main loop exit should use the final iter value. */ - if (multiple_exits_p) - { - /* Create a LC PHI if it doesn't already exist. */ - if (!virtual_operand_p (new_arg) && !res) + else { + /* Create the LC PHI node for the exit. */ tree new_def = copy_ssa_name (new_arg); gphi *lc_phi - = create_phi_node (new_def, main_loop_exit_block); + = create_phi_node (new_def, main_loop_exit_block); SET_PHI_ARG_DEF (lc_phi, 0, new_arg); new_arg = new_def; } - SET_PHI_ARG_DEF_ON_EDGE (lcssa_phi, - single_succ_edge (main_loop_exit_block), - new_arg); - } - else - SET_PHI_ARG_DEF_ON_EDGE (lcssa_phi, loop_exit, new_arg); - adjust_phi_and_debug_stmts (to_phi, loop_entry, new_res); - } + /* Create the PHI node in the merge block merging the + main and early exit values. */ + tree new_res = copy_ssa_name (gimple_phi_result (from_phi)); + gphi *lcssa_phi = create_phi_node (new_res, new_preheader); + edge main_e = single_succ_edge (main_loop_exit_block); + SET_PHI_ARG_DEF_ON_EDGE (lcssa_phi, main_e, new_arg); - /* Now fill in the values for the merge PHI in new_preheader - for the alternative exits. */ - if (multiple_exits_p) - { + /* And adjust the epilog entry value. */ + adjust_phi_and_debug_stmts (to_phi, loop_entry, new_res); + } + + /* After creating the merge PHIs handle the early exits those + should use the values at the start of the loop. */ for (auto gsi_from = gsi_start_phis (loop->header), gsi_to = gsi_start_phis (new_preheader); !gsi_end_p (gsi_from) && !gsi_end_p (gsi_to); @@ -1790,8 +1773,38 @@ slpeel_tree_duplicate_loop_to_edge_cfg (class loop *loop, edge loop_exit, SET_PHI_ARG_DEF (lc_phi, i, alt_arg); alt_arg = alt_def; } - edge main_e = single_succ_edge (alt_loop_exit_block); - SET_PHI_ARG_DEF_ON_EDGE (to_phi, main_e, alt_arg); + edge alt_e = single_succ_edge (alt_loop_exit_block); + SET_PHI_ARG_DEF_ON_EDGE (to_phi, alt_e, alt_arg); + } + } + /* For the single exit case only create the missing LC PHI nodes + for the continuation of the loop IVs that are not also already + reductions and thus had LC PHI nodes on the exit already. */ + else + { + for (auto gsi_from = gsi_start_phis (loop->header), + gsi_to = gsi_start_phis (new_loop->header); + !gsi_end_p (gsi_from) && !gsi_end_p (gsi_to); + gsi_next (&gsi_from), gsi_next (&gsi_to)) + { + gimple *from_phi = gsi_stmt (gsi_from); + gimple *to_phi = gsi_stmt (gsi_to); + tree new_arg = PHI_ARG_DEF_FROM_EDGE (from_phi, + loop_latch_edge (loop)); + + /* Check if we've already created a new phi node during edge + redirection. If we have, only propagate the value + downwards. */ + if (tree *res = new_phi_args.get (new_arg)) + { + adjust_phi_and_debug_stmts (to_phi, loop_entry, *res); + continue; + } + + tree new_res = copy_ssa_name (gimple_phi_result (from_phi)); + gphi *lcssa_phi = create_phi_node (new_res, new_preheader); + SET_PHI_ARG_DEF_ON_EDGE (lcssa_phi, loop_exit, new_arg); + adjust_phi_and_debug_stmts (to_phi, loop_entry, new_res); } } }