From patchwork Tue Oct 17 11:34:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lehua Ding X-Patchwork-Id: 154096 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp4069520vqb; Tue, 17 Oct 2023 04:38:39 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFbXtMoGmVD2067mANjZhEE/SSsz4jOvBaj1uIHkdF/HpI23gcQh1qXjy889z/j4oth7WSg X-Received: by 2002:a05:620a:1726:b0:775:93aa:cb91 with SMTP id az38-20020a05620a172600b0077593aacb91mr2111986qkb.35.1697542718867; Tue, 17 Oct 2023 04:38:38 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1697542718; cv=pass; d=google.com; s=arc-20160816; b=b8S7c/HExRQ6MgGTACy3ilUG090FNZSlLn7O6mYDVJEYd4r9erCKgFWEZCRjstdEec 2W/EJ9Ge0U2HCUnUfqDR0eTKUok+7PdklnBqC1NYRJUU/vph6j4rOaYNLos1GM4U2yzU 0HpFKMf7sQ/xnBrMvGQB7+pwPlMtMpqHvS+3kMeLqdozyVYAMyDxjlVQYMK7OwIYCtod NkFA5aNrAe7TOwrxITuLbpulAxKZeuftEjDxQUTrtWlb94tEEyWDb3DQAE8YMaTr+/1l 0hXuFBE4gTNd6ySpIgTz9oSMJQfKjVqnPLBOD0zJNSmEY6vpkHtw3YIpqO6f0tN6qb5u jY5g== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:feedback-id :content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:arc-filter:dmarc-filter :delivered-to; bh=RjdMmKnVX8qlhGzMNwI3PYzAmkVfETWv7/TqeY3RR2o=; fh=x/Q0OlwHuvCZ3FpkiZPiUSvevOYVxUAi4aNnf76mUPQ=; b=nm5lOKpC/0HvJGzYA/1rv4uzdvWr6dPAJiGX0S+d8ivRPyO+z6NefGeYyL3RtPEJD9 9Zrjw2ddk1zAPrn2tA0WKD2Vt+DFJavLhKtBPftMGJ1HAWWzuqFpc8N4HP/yfiOhiLrZ R5BKD0qj1VOSuzDJHdJc0FXZStLwF3qJwGeRm2JTDS1Tfwtec/jmqtwQKSy7IKzpaniY t2U0v+i0xIQupNRcXHpsXhqh15Y4W6qmvkrgS7mi+5PHQcWuSSXnpPfNPuR5oQnhIYjR zxq4Ddhvyd9UvEDx5CS2XpBsOVFHQPOSMBIN6THO5CXnx/Zsv5IMeQruwtSw4GdZLhl1 WxTw== ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (server2.sourceware.org. [2620:52:3:1:0:246e:9693:128c]) by mx.google.com with ESMTPS id b2-20020a05620a04e200b00773fb21ee2bsi889809qkh.105.2023.10.17.04.38.38 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Oct 2023 04:38:38 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) client-ip=2620:52:3:1:0:246e:9693:128c; Authentication-Results: mx.google.com; arc=pass (i=1); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 3867B385CC9E for ; Tue, 17 Oct 2023 11:38:10 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtpbguseast2.qq.com (smtpbguseast2.qq.com [54.204.34.130]) by sourceware.org (Postfix) with ESMTPS id CC16B3857031 for ; Tue, 17 Oct 2023 11:35:37 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org CC16B3857031 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivai.ai Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivai.ai ARC-Filter: OpenARC Filter v1.0.0 sourceware.org CC16B3857031 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=54.204.34.130 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1697542542; cv=none; b=IL7g0czmjbxkRmwp7RhekHFKk/Ddzk5NZCsQ52C8WYmlABYMQkh1Xdtmf8v/os81D9gQbBgN2izmXBSP2iOAUkDH58apgEUAiJYFWiHLF3nmrxtYA2mY9DxvascjkSOqfKNLZxG7wJuEMOa4mSatWYKpkb7AA1mR6JLUBjZ2EFA= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1697542542; c=relaxed/simple; bh=MKk+V5f4X6i3vdAwHexwSPHiM56r5k8E1MyJMGhUhc8=; h=From:To:Subject:Date:Message-Id:MIME-Version; b=aNLEgGkLEItBexUKX2W0YKANvOKTF/7ezLk5djVnqAYIOeU6AY4ayebiN8f+2ANFvTA9ozw9LDRRi2VJbaHkzOrZzUQ16qU19y4cvRGsseGqGqZR49P4BkLvVAiA966HIst863/1xnKiwKWST7ALOYIxDi81zwd2E+kgVL6Llos= ARC-Authentication-Results: i=1; server2.sourceware.org X-QQ-mid: bizesmtp62t1697542531tfpa8vug Received: from rios-cad121.hadoop.rioslab.org ( [58.60.1.9]) by bizesmtp.qq.com (ESMTP) with id ; Tue, 17 Oct 2023 19:35:29 +0800 (CST) X-QQ-SSF: 01400000000000C0F000000A0000000 X-QQ-FEAT: hvKw+PyJ8U5C554ylf3r2DJowySYe5wgT2zI0xi1vEcPizUUmZdx6DEnMT3OD XExar9kHnPosMqUanmxbd43qpF6uJy9XaBib9oUrm0vR3/0m43doMo0jM4Nq5nfi3ng2ff/ ldkAziCoBoURxeAFeXQmV+JZgv5EyMwtBKXNuWIbC7OIN68bFDsLfyfKXccI8maXuRAuMbc ugndpewHXuIJABA6pVMq5bMXGDlxlFU01iCZ8+ngKnxihYXVA1ZwIBbw1POnXapGUYMKREP xBj+vxwC4yFVmt40E2BiiiQhfQs0PYUKXpBbbBgg04A2PYkt74wrOlrK3smTBwl4NlQmTBE nDm/0SHDKt4cmVgEata7Sjf0PZyTEcsUi7ENH4WAR4kCzuVWI+UlfEGTal3tYQhQRKOAjou Y6D3+6WQQko= X-QQ-GoodBg: 2 X-BIZMAIL-ID: 4943574971249258968 From: Lehua Ding To: gcc-patches@gcc.gnu.org Cc: juzhe.zhong@rivai.ai, kito.cheng@gmail.com, rdapp.gcc@gmail.com, palmer@rivosinc.com, jeffreyalaw@gmail.com, lehua.ding@rivai.ai Subject: [PATCH V2 07/14] RISC-V: P7: Move earliest fuse and lcm code to pre_vsetvl class Date: Tue, 17 Oct 2023 19:34:53 +0800 Message-Id: <20231017113500.1160997-8-lehua.ding@rivai.ai> X-Mailer: git-send-email 2.36.3 In-Reply-To: <20231017113500.1160997-1-lehua.ding@rivai.ai> References: <20231017113500.1160997-1-lehua.ding@rivai.ai> MIME-Version: 1.0 X-QQ-SENDSIZE: 520 Feedback-ID: bizesmtp:rivai.ai:qybglogicsvrgz:qybglogicsvrgz6a-0 X-Spam-Status: No, score=-11.5 required=5.0 tests=BAYES_00, GIT_PATCH_0, KAM_DMARC_STATUS, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780002553733665068 X-GMAIL-MSGID: 1780002553733665068 This patch adjust move the code phase 2 and 3 from pass_vsetvl to pre_vsetvl class. gcc/ChangeLog: * config/riscv/riscv-vsetvl.cc (pre_vsetvl::earliest_fuse_vsetvl_info): New. (pre_vsetvl::pre_global_vsetvl_info): New. (pass_vsetvl::prune_expressions): Removed. (pass_vsetvl::compute_local_properties): Removed. (pass_vsetvl::earliest_fusion): Removed. (pass_vsetvl::vsetvl_fusion): Removed. (pass_vsetvl::pre_vsetvl): Removed. (pass_vsetvl::compute_probabilities): Removed. --- gcc/config/riscv/riscv-vsetvl.cc | 829 +++++++++++++++---------------- 1 file changed, 398 insertions(+), 431 deletions(-) -- 2.36.3 diff --git a/gcc/config/riscv/riscv-vsetvl.cc b/gcc/config/riscv/riscv-vsetvl.cc index b1269e8cf4f..a112895a283 100644 --- a/gcc/config/riscv/riscv-vsetvl.cc +++ b/gcc/config/riscv/riscv-vsetvl.cc @@ -3260,6 +3260,404 @@ pre_vsetvl::fuse_local_vsetvl_info () } } +bool +pre_vsetvl::earliest_fuse_vsetvl_info () +{ + compute_avl_def_data (); + compute_vsetvl_def_data (); + compute_vsetvl_lcm_data (); + + unsigned num_exprs = exprs.length (); + struct edge_list *edges = create_edge_list (); + unsigned num_edges = NUM_EDGES (edges); + sbitmap *antin + = sbitmap_vector_alloc (last_basic_block_for_fn (cfun), num_exprs); + sbitmap *antout + = sbitmap_vector_alloc (last_basic_block_for_fn (cfun), num_exprs); + + sbitmap *earliest = sbitmap_vector_alloc (num_edges, num_exprs); + + compute_available (avloc, kill, avout, avin); + compute_antinout_edge (antloc, transp, antin, antout); + compute_earliest (edges, num_exprs, antin, antout, avout, kill, earliest); + + if (dump_file && (dump_flags & TDF_DETAILS)) + { + fprintf (dump_file, "\n Compute LCM earliest insert data:\n\n"); + fprintf (dump_file, " Expression List (%u):\n", num_exprs); + for (unsigned i = 0; i < num_exprs; i++) + { + const auto &info = *exprs[i]; + fprintf (dump_file, " Expr[%u]: ", i); + info.dump (dump_file, " "); + } + fprintf (dump_file, "\n bitmap data:\n"); + for (const bb_info *bb : crtl->ssa->bbs ()) + { + unsigned int i = bb->index (); + fprintf (dump_file, " BB %u:\n", i); + fprintf (dump_file, " avloc: "); + dump_bitmap_file (dump_file, avloc[i]); + fprintf (dump_file, " kill: "); + dump_bitmap_file (dump_file, kill[i]); + fprintf (dump_file, " antloc: "); + dump_bitmap_file (dump_file, antloc[i]); + fprintf (dump_file, " transp: "); + dump_bitmap_file (dump_file, transp[i]); + + fprintf (dump_file, " avin: "); + dump_bitmap_file (dump_file, avin[i]); + fprintf (dump_file, " avout: "); + dump_bitmap_file (dump_file, avout[i]); + fprintf (dump_file, " antin: "); + dump_bitmap_file (dump_file, antin[i]); + fprintf (dump_file, " antout: "); + dump_bitmap_file (dump_file, antout[i]); + } + fprintf (dump_file, "\n"); + fprintf (dump_file, " earliest:\n"); + for (unsigned ed = 0; ed < num_edges; ed++) + { + edge eg = INDEX_EDGE (edges, ed); + + if (bitmap_empty_p (earliest[ed])) + continue; + fprintf (dump_file, " Edge(bb %u -> bb %u): ", eg->src->index, + eg->dest->index); + dump_bitmap_file (dump_file, earliest[ed]); + } + fprintf (dump_file, "\n"); + } + + if (dump_file && (dump_flags & TDF_DETAILS)) + { + fprintf (dump_file, " Fused global info result:\n"); + } + + bool changed = false; + for (unsigned ed = 0; ed < num_edges; ed++) + { + sbitmap e = earliest[ed]; + if (bitmap_empty_p (e)) + continue; + + unsigned int expr_index; + sbitmap_iterator sbi; + EXECUTE_IF_SET_IN_BITMAP (e, 0, expr_index, sbi) + { + vsetvl_info &curr_info = *exprs[expr_index]; + if (!curr_info.valid_p ()) + continue; + + edge eg = INDEX_EDGE (edges, ed); + if (eg->probability == profile_probability::never ()) + continue; + if (eg->src == ENTRY_BLOCK_PTR_FOR_FN (cfun) + || eg->dest == EXIT_BLOCK_PTR_FOR_FN (cfun)) + continue; + + vsetvl_block_info &src_block_info = get_block_info (eg->src); + vsetvl_block_info &dest_block_info = get_block_info (eg->dest); + + if (src_block_info.probability + == profile_probability::uninitialized ()) + continue; + + if (src_block_info.empty_p ()) + { + vsetvl_info new_curr_info = curr_info; + new_curr_info.set_bb (crtl->ssa->bb (eg->dest)); + bool has_compatible_p = false; + unsigned int def_expr_index; + sbitmap_iterator sbi2; + EXECUTE_IF_SET_IN_BITMAP ( + vsetvl_def_in[new_curr_info.get_bb ()->index ()], 0, + def_expr_index, sbi2) + { + vsetvl_info &prev_info = *vsetvl_def_exprs[def_expr_index]; + if (!prev_info.valid_p ()) + continue; + if (dem.compatible_with (prev_info, new_curr_info)) + { + has_compatible_p = true; + break; + } + } + if (!has_compatible_p) + { + if (dump_file && (dump_flags & TDF_DETAILS)) + { + fprintf (dump_file, + " Forbidden lift up vsetvl info into bb %u " + "since there is no vsetvl info that reaching in " + "is compatible with it:", + eg->src->index); + curr_info.dump (dump_file, " "); + } + continue; + } + + if (dump_file && (dump_flags & TDF_DETAILS)) + { + fprintf (dump_file, + " Set empty bb %u to info:", eg->src->index); + curr_info.dump (dump_file, " "); + } + src_block_info.set_info (curr_info); + src_block_info.probability = dest_block_info.probability; + changed = true; + } + else if (src_block_info.has_info ()) + { + vsetvl_info &prev_info = src_block_info.get_footer_info (); + gcc_assert (prev_info.valid_p ()); + + if (dem.compatible_with (prev_info, curr_info)) + { + if (dump_file && (dump_flags & TDF_DETAILS)) + { + fprintf (dump_file, " Fuse curr info since prev info " + "compatible with it:\n"); + fprintf (dump_file, " prev_info: "); + prev_info.dump (dump_file, " "); + fprintf (dump_file, " curr_info: "); + curr_info.dump (dump_file, " "); + } + dem.merge_with (prev_info, curr_info); + if (dump_file && (dump_flags & TDF_DETAILS)) + { + fprintf (dump_file, " prev_info after fused: "); + prev_info.dump (dump_file, " "); + fprintf (dump_file, "\n"); + } + changed = true; + if (src_block_info.has_info ()) + src_block_info.probability += dest_block_info.probability; + } + else if (src_block_info.has_info () + && !dem.compatible_with (prev_info, curr_info)) + { + /* Cancel lift up if probabilities are equal. */ + if (!earliest_fusion_worthwhile_p (eg->src)) + { + if (dump_file && (dump_flags & TDF_DETAILS)) + { + fprintf (dump_file, + " Change empty bb %u to from:", + eg->src->index); + prev_info.dump (dump_file, " "); + fprintf (dump_file, + " to (higher probability):"); + curr_info.dump (dump_file, " "); + } + src_block_info.set_empty_info (); + src_block_info.probability + = profile_probability::uninitialized (); + changed = true; + } + /* Choose the one with higher probability. */ + else if (dest_block_info.probability + > src_block_info.probability) + { + if (dump_file && (dump_flags & TDF_DETAILS)) + { + fprintf (dump_file, + " Change empty bb %u to from:", + eg->src->index); + prev_info.dump (dump_file, " "); + fprintf (dump_file, + " to (higher probability):"); + curr_info.dump (dump_file, " "); + } + src_block_info.set_info (curr_info); + src_block_info.probability = dest_block_info.probability; + changed = true; + } + } + } + else + { + vsetvl_info &prev_info = src_block_info.get_footer_info (); + if (!prev_info.valid_p () + || dem.available_with (prev_info, curr_info)) + continue; + + if (dem.compatible_with (prev_info, curr_info)) + { + if (dump_file && (dump_flags & TDF_DETAILS)) + { + fprintf (dump_file, " Fuse curr info since prev info " + "compatible with it:\n"); + fprintf (dump_file, " prev_info: "); + prev_info.dump (dump_file, " "); + fprintf (dump_file, " curr_info: "); + curr_info.dump (dump_file, " "); + } + dem.merge_with (prev_info, curr_info); + if (dump_file && (dump_flags & TDF_DETAILS)) + { + fprintf (dump_file, " prev_info after fused: "); + prev_info.dump (dump_file, " "); + fprintf (dump_file, "\n"); + } + changed = true; + } + } + } + } + + if (dump_file && (dump_flags & TDF_DETAILS)) + { + fprintf (dump_file, "\n"); + } + + sbitmap_vector_free (antin); + sbitmap_vector_free (antout); + sbitmap_vector_free (earliest); + free_edge_list (edges); + + return changed; +} + +void +pre_vsetvl::pre_global_vsetvl_info () +{ + compute_avl_def_data (); + compute_vsetvl_def_data (); + compute_vsetvl_lcm_data (); + + unsigned num_exprs = exprs.length (); + edges = pre_edge_lcm_avs (num_exprs, transp, avloc, antloc, kill, avin, avout, + &insert, &del); + unsigned num_edges = NUM_EDGES (edges); + + if (dump_file && (dump_flags & TDF_DETAILS)) + { + fprintf (dump_file, "\n Compute LCM insert and delete data:\n\n"); + fprintf (dump_file, " Expression List (%u):\n", num_exprs); + for (unsigned i = 0; i < num_exprs; i++) + { + const auto &info = *exprs[i]; + fprintf (dump_file, " Expr[%u]: ", i); + info.dump (dump_file, " "); + } + fprintf (dump_file, "\n bitmap data:\n"); + for (const bb_info *bb : crtl->ssa->bbs ()) + { + unsigned i = bb->index (); + fprintf (dump_file, " BB %u:\n", i); + fprintf (dump_file, " avloc: "); + dump_bitmap_file (dump_file, avloc[i]); + fprintf (dump_file, " kill: "); + dump_bitmap_file (dump_file, kill[i]); + fprintf (dump_file, " antloc: "); + dump_bitmap_file (dump_file, antloc[i]); + fprintf (dump_file, " transp: "); + dump_bitmap_file (dump_file, transp[i]); + + fprintf (dump_file, " avin: "); + dump_bitmap_file (dump_file, avin[i]); + fprintf (dump_file, " avout: "); + dump_bitmap_file (dump_file, avout[i]); + fprintf (dump_file, " del: "); + dump_bitmap_file (dump_file, del[i]); + } + fprintf (dump_file, "\n"); + fprintf (dump_file, " insert:\n"); + for (unsigned ed = 0; ed < num_edges; ed++) + { + edge eg = INDEX_EDGE (edges, ed); + + if (bitmap_empty_p (insert[ed])) + continue; + fprintf (dump_file, " Edge(bb %u -> bb %u): ", eg->src->index, + eg->dest->index); + dump_bitmap_file (dump_file, insert[ed]); + } + } + + /* Remove vsetvl infos as LCM suggest */ + for (const bb_info *bb : crtl->ssa->bbs ()) + { + sbitmap d = del[bb->index ()]; + if (bitmap_count_bits (d) == 0) + continue; + gcc_assert (bitmap_count_bits (d) == 1); + unsigned expr_index = bitmap_first_set_bit (d); + vsetvl_info &info = *exprs[expr_index]; + gcc_assert (info.valid_p ()); + gcc_assert (info.get_bb () == bb); + const vsetvl_block_info &block_info = get_block_info (info.get_bb ()); + gcc_assert (block_info.get_header_info () == info); + info.set_ignore (); + } + + for (const bb_info *bb : crtl->ssa->bbs ()) + { + vsetvl_block_info &block_info = get_block_info (bb); + if (block_info.empty_p ()) + continue; + vsetvl_info &curr_info = block_info.get_header_info (); + if (curr_info.ignore_p ()) + { + if (block_info.infos.is_empty ()) + continue; + curr_info = block_info.infos[0]; + } + if (curr_info.valid_p () && !curr_info.use_by_non_rvv_insn_p () + && preds_has_same_avl_p (curr_info)) + curr_info.set_change_vtype_only (); + + vsetvl_info prev_info = vsetvl_info (); + prev_info.set_empty (); + for (auto &curr_info : block_info.infos) + { + if (prev_info.valid_p () && curr_info.valid_p () + && dem.available_avl_with (prev_info, curr_info)) + curr_info.set_change_vtype_only (); + prev_info = curr_info; + } + } + + /* Cancel unnecessary insert. */ + for (int ed = 0; ed < NUM_EDGES (edges); ed++) + { + edge eg = INDEX_EDGE (edges, ed); + sbitmap i = insert[ed]; + if (bitmap_count_bits (i) < 1) + continue; + + if (bitmap_count_bits (i) > 1) + /* For code with infinite loop (e.g. pr61634.c) The data flow is + completely wrong. */ + continue; + + gcc_assert (bitmap_count_bits (i) == 1); + unsigned expr_index = bitmap_first_set_bit (i); + const vsetvl_info &info = *exprs[expr_index]; + gcc_assert (info.valid_p ()); + if (eg->src->succs->length () == 1) + { + vsetvl_block_info &block_info = get_block_info (eg->src); + if (block_info.empty_p ()) + continue; + + vsetvl_info &prev_info = block_info.get_footer_info (); + if (!block_info.has_info () && !prev_info.ignore_p () + && prev_info.valid_p ()) + { + vsetvl_info curr_info = info; + curr_info.set_bb (prev_info.get_bb ()); + if (dem.compatible_with (prev_info, curr_info)) + { + dem.merge_with (prev_info, curr_info); + bitmap_clear_bit (i, expr_index); + } + } + } + } +} + const pass_data pass_data_vsetvl = { RTL_PASS, /* type */ "vsetvl", /* name */ @@ -3392,366 +3790,6 @@ make_pass_vsetvl (gcc::context *ctxt) return new pass_vsetvl (ctxt); } -/* Assemble the candidates expressions for LCM. */ -void -pass_vsetvl::prune_expressions (void) -{ - for (const bb_info *bb : crtl->ssa->bbs ()) - { - if (m_vector_manager->vector_block_infos[bb->index ()] - .local_dem.valid_or_dirty_p ()) - m_vector_manager->create_expr ( - m_vector_manager->vector_block_infos[bb->index ()].local_dem); - if (m_vector_manager->vector_block_infos[bb->index ()] - .reaching_out.valid_or_dirty_p ()) - m_vector_manager->create_expr ( - m_vector_manager->vector_block_infos[bb->index ()].reaching_out); - } - - if (dump_file) - { - fprintf (dump_file, "\nThe total VSETVL expression num = %d\n", - m_vector_manager->vector_exprs.length ()); - fprintf (dump_file, "Expression List:\n"); - for (size_t i = 0; i < m_vector_manager->vector_exprs.length (); i++) - { - fprintf (dump_file, "Expr[%ld]:\n", i); - m_vector_manager->vector_exprs[i]->dump (dump_file); - fprintf (dump_file, "\n"); - } - } -} - -/* Compute the local properties of each recorded expression. - - Local properties are those that are defined by the block, irrespective of - other blocks. - - An expression is transparent in a block if its operands are not modified - in the block. - - An expression is computed (locally available) in a block if it is computed - at least once and expression would contain the same value if the - computation was moved to the end of the block. - - An expression is locally anticipatable in a block if it is computed at - least once and expression would contain the same value if the computation - was moved to the beginning of the block. */ -void -pass_vsetvl::compute_local_properties (void) -{ - /* - If T is locally available at the end of a block, then T' must be - available at the end of the same block. Since some optimization has - occurred earlier, T' might not be locally available, however, it must - have been previously computed on all paths. As a formula, T at AVLOC(B) - implies that T' at AVOUT(B). - An "available occurrence" is one that is the last occurrence in the - basic block and the operands are not modified by following statements in - the basic block [including this insn]. - - - If T is locally anticipated at the beginning of a block, then either - T', is locally anticipated or it is already available from previous - blocks. As a formula, this means that T at ANTLOC(B) implies that T' at - ANTLOC(B) at AVIN(B). - An "anticipatable occurrence" is one that is the first occurrence in the - basic block, the operands are not modified in the basic block prior - to the occurrence and the output is not used between the start of - the block and the occurrence. */ - - basic_block cfg_bb; - for (const bb_info *bb : crtl->ssa->bbs ()) - { - unsigned int curr_bb_idx = bb->index (); - if (curr_bb_idx == ENTRY_BLOCK || curr_bb_idx == EXIT_BLOCK) - continue; - const auto local_dem - = m_vector_manager->vector_block_infos[curr_bb_idx].local_dem; - const auto reaching_out - = m_vector_manager->vector_block_infos[curr_bb_idx].reaching_out; - - /* Compute transparent. */ - for (size_t i = 0; i < m_vector_manager->vector_exprs.length (); i++) - { - const auto *expr = m_vector_manager->vector_exprs[i]; - if (local_dem.valid_or_dirty_p () || local_dem.unknown_p ()) - bitmap_clear_bit (m_vector_manager->vector_transp[curr_bb_idx], i); - else if (expr->has_avl_reg ()) - { - rtx reg = expr->get_avl_or_vl_reg (); - for (const insn_info *insn : bb->real_nondebug_insns ()) - { - if (find_access (insn->defs (), REGNO (reg))) - { - bitmap_clear_bit ( - m_vector_manager->vector_transp[curr_bb_idx], i); - break; - } - else if (vlmax_avl_p (expr->get_avl ()) - && find_access (insn->uses (), REGNO (reg))) - { - bitmap_clear_bit ( - m_vector_manager->vector_transp[curr_bb_idx], i); - break; - } - } - } - } - - /* Compute anticipatable occurrences. */ - if (local_dem.valid_or_dirty_p ()) - if (anticipatable_occurrence_p (bb, local_dem)) - bitmap_set_bit (m_vector_manager->vector_antic[curr_bb_idx], - m_vector_manager->get_expr_id (local_dem)); - - /* Compute available occurrences. */ - if (reaching_out.valid_or_dirty_p ()) - { - auto_vec available_list - = m_vector_manager->get_all_available_exprs (reaching_out); - for (size_t i = 0; i < available_list.length (); i++) - { - const vector_insn_info *expr - = m_vector_manager->vector_exprs[available_list[i]]; - if (available_occurrence_p (bb, *expr)) - bitmap_set_bit (m_vector_manager->vector_comp[curr_bb_idx], - available_list[i]); - } - } - - if (loop_basic_block_p (bb->cfg_bb ()) && local_dem.valid_or_dirty_p () - && reaching_out.valid_or_dirty_p () - && !local_dem.compatible_p (reaching_out)) - bitmap_clear_bit (m_vector_manager->vector_antic[curr_bb_idx], - m_vector_manager->get_expr_id (local_dem)); - } - - /* Compute kill for each basic block using: - - ~(TRANSP | COMP) - */ - - FOR_EACH_BB_FN (cfg_bb, cfun) - { - bitmap_ior (m_vector_manager->vector_kill[cfg_bb->index], - m_vector_manager->vector_transp[cfg_bb->index], - m_vector_manager->vector_comp[cfg_bb->index]); - bitmap_not (m_vector_manager->vector_kill[cfg_bb->index], - m_vector_manager->vector_kill[cfg_bb->index]); - } - - FOR_EACH_BB_FN (cfg_bb, cfun) - { - edge e; - edge_iterator ei; - - /* If the current block is the destination of an abnormal edge, we - kill all trapping (for PRE) and memory (for hoist) expressions - because we won't be able to properly place the instruction on - the edge. So make them neither anticipatable nor transparent. - This is fairly conservative. - - ??? For hoisting it may be necessary to check for set-and-jump - instructions here, not just for abnormal edges. The general problem - is that when an expression cannot not be placed right at the end of - a basic block we should account for any side-effects of a subsequent - jump instructions that could clobber the expression. It would - be best to implement this check along the lines of - should_hoist_expr_to_dom where the target block is already known - and, hence, there's no need to conservatively prune expressions on - "intermediate" set-and-jump instructions. */ - FOR_EACH_EDGE (e, ei, cfg_bb->preds) - if (e->flags & EDGE_COMPLEX) - { - bitmap_clear (m_vector_manager->vector_antic[cfg_bb->index]); - bitmap_clear (m_vector_manager->vector_transp[cfg_bb->index]); - } - } -} - -/* Fuse demand info for earliest edge. */ -bool -pass_vsetvl::earliest_fusion (void) -{ - bool changed_p = false; - for (int ed = 0; ed < NUM_EDGES (m_vector_manager->vector_edge_list); ed++) - { - for (size_t i = 0; i < m_vector_manager->vector_exprs.length (); i++) - { - auto &expr = *m_vector_manager->vector_exprs[i]; - if (expr.empty_p ()) - continue; - edge eg = INDEX_EDGE (m_vector_manager->vector_edge_list, ed); - /* If it is the edge that we never reach, skip its possible PRE - fusion conservatively. */ - if (eg->probability == profile_probability::never ()) - break; - if (eg->src == ENTRY_BLOCK_PTR_FOR_FN (cfun) - || eg->dest == EXIT_BLOCK_PTR_FOR_FN (cfun)) - break; - if (bitmap_bit_p (m_vector_manager->vector_earliest[ed], i)) - { - auto &src_block_info = get_block_info (eg->src); - auto &dest_block_info = get_block_info (eg->dest); - if (src_block_info.reaching_out.unknown_p ()) - break; - - gcc_assert (!(eg->flags & EDGE_ABNORMAL)); - vector_insn_info new_info = vector_insn_info (); - profile_probability prob = src_block_info.probability; - /* We don't fuse user vsetvl into EMPTY or - DIRTY (EMPTY but polluted) block for these - following reasons: - - - The user vsetvl instruction is configured as - no side effects that the previous passes - (GSCE, Loop-invariant, ..., etc) - should be able to do a good job on optimization - of user explicit vsetvls so we don't need to - PRE optimization (The user vsetvls should be - on the optimal local already before this pass) - again for user vsetvls in VSETVL PASS here - (Phase 3 && Phase 4). - - - Allowing user vsetvls be optimized in PRE - optimization here (Phase 3 && Phase 4) will - complicate the codes so much so we prefer user - vsetvls be optimized in post-optimization - (Phase 5 && Phase 6). */ - if (vsetvl_insn_p (expr.get_insn ()->rtl ())) - { - if (src_block_info.reaching_out.empty_p ()) - continue; - else if (src_block_info.reaching_out.dirty_p () - && !src_block_info.reaching_out.compatible_p (expr)) - { - new_info.set_empty (); - /* Update probability as uninitialized status so that - we won't try to fuse any demand info into such EMPTY - block any more. */ - prob = profile_probability::uninitialized (); - update_block_info (eg->src->index, prob, new_info); - continue; - } - } - - if (src_block_info.reaching_out.empty_p ()) - { - if (src_block_info.probability - == profile_probability::uninitialized ()) - continue; - new_info = expr.global_merge (expr, eg->src->index); - new_info.set_dirty (); - prob = dest_block_info.probability; - update_block_info (eg->src->index, prob, new_info); - changed_p = true; - } - else if (src_block_info.reaching_out.dirty_p ()) - { - /* DIRTY -> DIRTY or VALID -> DIRTY. */ - if (demands_can_be_fused_p (src_block_info.reaching_out, - expr)) - { - new_info = src_block_info.reaching_out.global_merge ( - expr, eg->src->index); - new_info.set_dirty (); - prob += dest_block_info.probability; - } - else if (!src_block_info.reaching_out.compatible_p (expr) - && !m_vector_manager->earliest_fusion_worthwhile_p ( - eg->src)) - { - new_info.set_empty (); - prob = profile_probability::uninitialized (); - } - else if (!src_block_info.reaching_out.compatible_p (expr) - && dest_block_info.probability - > src_block_info.probability) - { - new_info = expr; - new_info.set_dirty (); - prob = dest_block_info.probability; - } - else - continue; - update_block_info (eg->src->index, prob, new_info); - changed_p = true; - } - else - { - rtx vl = NULL_RTX; - if (vsetvl_insn_p ( - src_block_info.reaching_out.get_insn ()->rtl ()) - && vsetvl_dominated_by_p (eg->src, expr, - src_block_info.reaching_out, - true)) - ; - else if (!demands_can_be_fused_p (src_block_info.reaching_out, - expr)) - continue; - else if (!earliest_pred_can_be_fused_p ( - crtl->ssa->bb (eg->src), - src_block_info.reaching_out, expr, &vl)) - continue; - - vector_insn_info new_info - = src_block_info.reaching_out.global_merge (expr, - eg->src->index); - - prob = std::max (dest_block_info.probability, - src_block_info.probability); - change_vsetvl_insn (new_info.get_insn (), new_info, vl); - update_block_info (eg->src->index, prob, new_info); - changed_p = true; - } - } - } - } - return changed_p; -} - -/* Fuse VSETVL demand info according LCM computed location. */ -void -pass_vsetvl::vsetvl_fusion (void) -{ - /* Fuse VSETVL demand info until VSETVL CFG fixed. */ - bool changed_p = true; - int fusion_no = 0; - while (changed_p) - { - changed_p = false; - fusion_no++; - prune_expressions (); - m_vector_manager->create_bitmap_vectors (); - compute_local_properties (); - /* Compute global availability. */ - compute_available (m_vector_manager->vector_comp, - m_vector_manager->vector_kill, - m_vector_manager->vector_avout, - m_vector_manager->vector_avin); - /* Compute global anticipatability. */ - compute_antinout_edge (m_vector_manager->vector_antic, - m_vector_manager->vector_transp, - m_vector_manager->vector_antin, - m_vector_manager->vector_antout); - /* Compute earliestness. */ - compute_earliest (m_vector_manager->vector_edge_list, - m_vector_manager->vector_exprs.length (), - m_vector_manager->vector_antin, - m_vector_manager->vector_antout, - m_vector_manager->vector_avout, - m_vector_manager->vector_kill, - m_vector_manager->vector_earliest); - changed_p |= earliest_fusion (); - if (dump_file && (dump_flags & TDF_DETAILS)) - { - fprintf (dump_file, "\nEARLIEST fusion %d\n", fusion_no); - m_vector_manager->dump (dump_file); - } - m_vector_manager->free_bitmap_vectors (); - if (!m_vector_manager->vector_exprs.is_empty ()) - m_vector_manager->vector_exprs.release (); - } -} /* Return true if VSETVL in the block can be refined as vsetvl zero,zero. */ bool @@ -4007,33 +4045,6 @@ pass_vsetvl::commit_vsetvls (void) return need_commit; } -void -pass_vsetvl::pre_vsetvl (void) -{ - /* Compute entity list. */ - prune_expressions (); - - m_vector_manager->create_bitmap_vectors (); - compute_local_properties (); - m_vector_manager->vector_edge_list = pre_edge_lcm_avs ( - m_vector_manager->vector_exprs.length (), m_vector_manager->vector_transp, - m_vector_manager->vector_comp, m_vector_manager->vector_antic, - m_vector_manager->vector_kill, m_vector_manager->vector_avin, - m_vector_manager->vector_avout, &m_vector_manager->vector_insert, - &m_vector_manager->vector_del); - - /* We should dump the information before CFG is changed. Otherwise it will - produce ICE (internal compiler error). */ - if (dump_file && (dump_flags & TDF_DETAILS)) - m_vector_manager->dump (dump_file); - - refine_vsetvls (); - cleanup_vsetvls (); - bool need_commit = commit_vsetvls (); - if (need_commit) - commit_edge_insertions (); -} - /* Some instruction can not be accessed in RTL_SSA when we don't re-init the new RTL_SSA framework but it is definetely at the END of the block. @@ -4581,50 +4592,6 @@ pass_vsetvl::done (void) m_vector_manager = nullptr; } -/* Compute probability for each block. */ -void -pass_vsetvl::compute_probabilities (void) -{ - /* Don't compute it in -O0 since we don't need it. */ - if (!optimize) - return; - edge e; - edge_iterator ei; - - for (const bb_info *bb : crtl->ssa->bbs ()) - { - basic_block cfg_bb = bb->cfg_bb (); - auto &curr_prob = get_block_info (cfg_bb).probability; - - /* GCC assume entry block (bb 0) are always so - executed so set its probability as "always". */ - if (ENTRY_BLOCK_PTR_FOR_FN (cfun) == cfg_bb) - curr_prob = profile_probability::always (); - /* Exit block (bb 1) is the block we don't need to process. */ - if (EXIT_BLOCK_PTR_FOR_FN (cfun) == cfg_bb) - continue; - - gcc_assert (curr_prob.initialized_p ()); - FOR_EACH_EDGE (e, ei, cfg_bb->succs) - { - auto &new_prob = get_block_info (e->dest).probability; - /* Normally, the edge probability should be initialized. - However, some special testing code which is written in - GIMPLE IR style force the edge probility uninitialized, - we conservatively set it as never so that it will not - affect PRE (Phase 3 && Phse 4). */ - if (!e->probability.initialized_p ()) - new_prob = profile_probability::never (); - else if (!new_prob.initialized_p ()) - new_prob = curr_prob * e->probability; - else if (new_prob == profile_probability::always ()) - continue; - else - new_prob += curr_prob * e->probability; - } - } -} - /* Lazy vsetvl insertion for optimize > 0. */ void pass_vsetvl::lazy_vsetvl (void)