From patchwork Wed Jun 28 13:45:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tamar Christina X-Patchwork-Id: 113896 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp8939313vqr; Wed, 28 Jun 2023 06:48:13 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7fgm+hhRWuesL8RAJutrPmPtN7OZtK8nEvoNXet43+HwEhwC3FVKm3Yppv3qcSXi9vvEK5 X-Received: by 2002:a17:907:36c5:b0:978:acec:36b1 with SMTP id bj5-20020a17090736c500b00978acec36b1mr28968741ejc.17.1687960092764; Wed, 28 Jun 2023 06:48:12 -0700 (PDT) Received: from sourceware.org (ip-8-43-85-97.sourceware.org. [8.43.85.97]) by mx.google.com with ESMTPS id b11-20020a170906d10b00b00982818cbb54si5653248ejz.593.2023.06.28.06.48.12 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Jun 2023 06:48:12 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) client-ip=8.43.85.97; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=kkg6tqRj; arc=fail (signature failed); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gnu.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id A20863864C6A for ; Wed, 28 Jun 2023 13:46:34 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org A20863864C6A DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1687959994; bh=JRtp83SrSr9NKESgMZiVBn/UYyIox+rUfgMfey6VGoE=; h=Date:To:Cc:Subject:In-Reply-To:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=kkg6tqRjOvptVb/aPgZKJO4qbJoqxc3xLZhYiFbx2ulV2+8QcsM0CtbBiYJz+FBpY mIX8OoGqvf0m+74bsbHVOTPq9vp4uvKcONUfJXb4QdlTB6vhaiZB/WSDLN7LoaW9aI AopZVuTwOsFB1bnXecy4aeBVCr07edaErIbL0YBU= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from EUR04-DB3-obe.outbound.protection.outlook.com (mail-db3eur04on2082.outbound.protection.outlook.com [40.107.6.82]) by sourceware.org (Postfix) with ESMTPS id 40BC43857438 for ; Wed, 28 Jun 2023 13:45:46 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 40BC43857438 Received: from AS9PR06CA0662.eurprd06.prod.outlook.com (2603:10a6:20b:49c::7) by AS8PR08MB10314.eurprd08.prod.outlook.com (2603:10a6:20b:5c0::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6521.26; Wed, 28 Jun 2023 13:45:42 +0000 Received: from AM7EUR03FT045.eop-EUR03.prod.protection.outlook.com (2603:10a6:20b:49c:cafe::6d) by AS9PR06CA0662.outlook.office365.com (2603:10a6:20b:49c::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6521.34 via Frontend Transport; Wed, 28 Jun 2023 13:45:42 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; pr=C Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by AM7EUR03FT045.mail.protection.outlook.com (100.127.140.150) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6544.20 via Frontend Transport; Wed, 28 Jun 2023 13:45:42 +0000 Received: ("Tessian outbound e2424c13b707:v142"); Wed, 28 Jun 2023 13:45:42 +0000 X-CheckRecipientChecked: true X-CR-MTA-CID: 174086eba2bb8df6 X-CR-MTA-TID: 64aa7808 Received: from e8eea0f808ea.1 by 64aa7808-outbound-1.mta.getcheckrecipient.com id 07AF4E92-5AE2-4AF0-B2AE-A57396C88700.1; Wed, 28 Jun 2023 13:45:35 +0000 Received: from EUR05-DB8-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id e8eea0f808ea.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Wed, 28 Jun 2023 13:45:35 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=aRG4fphg11Q+QefwikMADaqrkhaIBHFWJVBborvsoOIDU5dr6k8B7f7t7A0RW8P3kVEI4higTTYCzyFeJf7PysbatimzUIcKH2jwh9Bnhgb+8SeKkkTW1kMBH4YvR+g6us7GeEzFPw4MNJrrDTfM+TR2V/aitEJOJ/1ECSq3jBcrheg5l2Sq9BAG/mczUJVZ4l2F5JtIBIXseVXTmm3lX0t0q2iTJ/I9U50LeGfZYkChbGRtKqfigUNXpfjG0KQrYj1wfi/S5mXo31uS1BX3tixcG8IdHAk4LgjqNTgXjYwzcEWhDP7P6aSLRbsoXkUVPA+JrX9AbMglpH/iwv/N0g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=JRtp83SrSr9NKESgMZiVBn/UYyIox+rUfgMfey6VGoE=; b=fJ0xTiIGrH2u3aBDl9QvIaXW3fHzb9Q8Lk/TI1eg3PF2RJSwIUSc9kli5j0Trq9V3UmwPePDLkuwGGRr3/eokLDw7VebYaabAUaJtUKmTvrwly+cCJVPP36FDgLzAZ8neKnpNXkk8RU/nCvd/opmX4OXb0RKRZwV38O+tcubOaSz3LgrNLECdZhS9UklQFM96MSXhEgZEOblz4IDuu+qlRMse2G0+/re7JLnOEdqH4w7lvhVwRkQoX7QY18q9DCbWSFzKhgh9TyblEl2eCSNTZshfz7W8PRxmG7KSvZJaEyf+DcpPto881H2D1BuaiTAaXBC1XtKJogVq4C7FYDRHg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none Authentication-Results-Original: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; Received: from VI1PR08MB5325.eurprd08.prod.outlook.com (2603:10a6:803:13e::17) by DB5PR08MB9970.eurprd08.prod.outlook.com (2603:10a6:10:489::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6521.23; Wed, 28 Jun 2023 13:45:32 +0000 Received: from VI1PR08MB5325.eurprd08.prod.outlook.com ([fe80::2301:1cde:cfe7:eaf0]) by VI1PR08MB5325.eurprd08.prod.outlook.com ([fe80::2301:1cde:cfe7:eaf0%6]) with mapi id 15.20.6521.026; Wed, 28 Jun 2023 13:45:32 +0000 Date: Wed, 28 Jun 2023 14:45:30 +0100 To: gcc-patches@gcc.gnu.org Cc: nd@arm.com, rguenther@suse.de, jlaw@ventanamicro.com Subject: [PATCH 8/19]middle-end: updated niters analysis to handle multiple exits. Message-ID: Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: LO4P123CA0396.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:18f::23) To VI1PR08MB5325.eurprd08.prod.outlook.com (2603:10a6:803:13e::17) MIME-Version: 1.0 X-MS-TrafficTypeDiagnostic: VI1PR08MB5325:EE_|DB5PR08MB9970:EE_|AM7EUR03FT045:EE_|AS8PR08MB10314:EE_ X-MS-Office365-Filtering-Correlation-Id: 823d9974-ac3e-4319-d608-08db77ddf77b x-checkrecipientrouted: true NoDisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: vBAG/tLsDEyF0rID/xH7vC046qnfmWVoG/SML8JthIpe8XrowVcIJUw8PZcbiq2x5MQXCADoXMGfVFb6CIjcYuQnB4m3MXkxTBgiTBJKjJOU0xk27cVw1R1Df3k0ZK3Xtm2oPuvhik1VAw9SHc1vjF2DdTL0Sb6W51qBK5PaKYIbeuimA8K8lBVJm1dCx4fFn2jRuRPQKfgf1HK5+IwLpnwPnuz2Or4DpoVaeWUbog6ETIX/klIAEmUygmCxQFD9XkbO1AubasT7bgBtVI37sIIOkq7b6KdLlBBdlzYLiP6FDPKtBd5wqg4ZeiqRmpuFXQsA517epyr6v58PCSOTXBtx3KgZGeMU4/tQC/S4zLNIfsgPl1dbAxuJ8d0NZil8zr2qnT7i7tQNlepACV4DGmilFtGLGfBacePgOkNtizvaol8CniNsHCkV3U3cMZZ0b1y9pXS31oNqT6Zpq4oKZSdENqqVvNrEjvuDEpoEt199Yl4FFbqpppkTO/Ooz0qE2EyBetC5u5p7jFfZCm0N/P7w8NCQ5vGpCBSfzskTHuUd33xa3lVMEJbpn0O4/Ie1FXHEBRj7aqDOKHPrLSoD/tlI81SagpkVw7icJ1P5yioTVJKJhBcbMYRPYf8NHJqx X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:VI1PR08MB5325.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(4636009)(136003)(346002)(396003)(376002)(366004)(39860400002)(451199021)(66946007)(66476007)(6916009)(44832011)(316002)(66556008)(478600001)(4326008)(36756003)(8936002)(235185007)(15650500001)(8676002)(5660300002)(86362001)(2906002)(33964004)(41300700001)(44144004)(6486002)(186003)(4743002)(6506007)(6512007)(38100700002)(26005)(83380400001)(2616005)(2700100001); DIR:OUT; SFP:1101; X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB5PR08MB9970 Original-Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AM7EUR03FT045.eop-EUR03.prod.protection.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: 8e2248cc-ae45-4967-2ecb-08db77ddf143 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 3gmEqIoOY407l08xstA5ds4u4Tb+3NzgryAmq3/hd00JSIA8l31oEhPyY8bEAcFID2O5zMEG/riUbTTWWnjWmFTGVEPIt36whIX+7GcadZF6ThuO1u6I9I/h9o5sCTOXArjP/L4qD21EVYmuNuEVhuhTWJ2GuJ4+sLLozXxtDfUzK9BK071vTw3eAF2FvU7fyR8ezgTy38HJHpiFYpPOoMxEeFzLg1iDAZc3Puf+B8srMtec33KLFej6ets/+/39NciucEuW1q0OE6xJJZatabNm2DZGjN/BejIFIz0CZUsAvcy+lVzLRAWMN8wqhctzPGbQ4h2PjrzCwxN6wx+aM7lityD/Z7iAEZ9hLKISz3+N6rPLVTBkOeMOW5rwvotsXSIrKPMiz2bnJWFhyNx5D4WhvbB6ck3Ql5OR8lU+/V9B1HjqU+Husa5PYAFwAKzY+iaZTSL6b04be12+yycXbOWx0FaS/uZHuG4AETdVzs9XaCjUOOok1UcNKlAesYoT1uXDSlaAdQEa1D5vJvA/rdZS+yx89B1V4GDvGh9Odwo70pNHfduLV7qPVQTWNMet4B37Vv23kh+Arp3fjqw6teTbDXeBz2GrYomt618FH1G2YU7Z/Ssge5wK9beiegzmSFqEKqhi6EdffR7dGFgmyeFhAZD4ZfC00JQ9F8Sq7J6FVnPA9HRQ0bfBWojC08+RAAjb3Bb6SP9aRLd9WAqDrNBb5gfNRAJTUrYKYmpPd0RRROBiNDQDnGS31CdlLD4UYVLQ8+olRhRwiqB7bPhtqg== X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFS:(13230028)(4636009)(396003)(39860400002)(136003)(376002)(346002)(451199021)(46966006)(40470700004)(36840700001)(44832011)(235185007)(5660300002)(15650500001)(4326008)(6916009)(70206006)(478600001)(36756003)(316002)(70586007)(8936002)(8676002)(2906002)(40460700003)(4743002)(33964004)(36860700001)(41300700001)(6486002)(44144004)(82310400005)(186003)(40480700001)(336012)(86362001)(26005)(6506007)(6512007)(47076005)(356005)(107886003)(81166007)(2616005)(82740400003)(83380400001)(2700100001); DIR:OUT; SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Jun 2023 13:45:42.5841 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 823d9974-ac3e-4319-d608-08db77ddf77b X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: AM7EUR03FT045.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR08MB10314 X-Spam-Status: No, score=-12.1 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, FORGED_SPF_HELO, GIT_PATCH_0, KAM_DMARC_NONE, KAM_LOTSOFHASH, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_NONE, TXREP, T_SCC_BODY_TEXT_LINE, UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Tamar Christina via Gcc-patches From: Tamar Christina Reply-To: Tamar Christina Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org Sender: "Gcc-patches" X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1769954442292202279?= X-GMAIL-MSGID: =?utf-8?q?1769954442292202279?= Hi All, For early break vectorization we have to update niters analysis to record and analyze all exits of the loop, and so all conds. The niters of the loop is still determined by the main/natural exit of the loop as this is the O(n) bounds. For now we don't do much with the secondary conds, but their assumptions can be used to generate versioning checks later. Bootstrapped Regtested on aarch64-none-linux-gnu and no issues. Ok for master? Thanks, Tamar gcc/ChangeLog: * tree-vect-loop.cc (vect_get_loop_niters): Analyze all exits and return all gconds. (vect_analyze_loop_form): Update code checking for conds. (vect_create_loop_vinfo): Handle having multiple conds. (vect_analyze_loop): Release extra loop conds structures. * tree-vectorizer.h (LOOP_VINFO_LOOP_CONDS, LOOP_VINFO_LOOP_IV_COND): New. (struct vect_loop_form_info): Add conds, loop_iv_cond. --- inline copy of patch -- diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc index 55e69a7ca0b24e0872477141db6f74dbf90b7981..9065811b3b9c2a550baf44768603172b9e26b94b 100644 --- diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc index 55e69a7ca0b24e0872477141db6f74dbf90b7981..9065811b3b9c2a550baf44768603172b9e26b94b 100644 --- a/gcc/tree-vect-loop.cc +++ b/gcc/tree-vect-loop.cc @@ -849,80 +849,106 @@ vect_fixup_scalar_cycles_with_patterns (loop_vec_info loop_vinfo) in NUMBER_OF_ITERATIONSM1. Place the condition under which the niter information holds in ASSUMPTIONS. - Return the loop exit condition. */ + Return the loop exit conditions. */ -static gcond * +static vec vect_get_loop_niters (class loop *loop, tree *assumptions, tree *number_of_iterations, tree *number_of_iterationsm1) { - edge exit = single_exit (loop); + auto_vec exits = get_loop_exit_edges (loop); + vec conds; + conds.create (exits.length ()); class tree_niter_desc niter_desc; tree niter_assumptions, niter, may_be_zero; - gcond *cond = get_loop_exit_condition (loop); *assumptions = boolean_true_node; *number_of_iterationsm1 = chrec_dont_know; *number_of_iterations = chrec_dont_know; + DUMP_VECT_SCOPE ("get_loop_niters"); - if (!exit) - return cond; + if (exits.is_empty ()) + return conds; - may_be_zero = NULL_TREE; - if (!number_of_iterations_exit_assumptions (loop, exit, &niter_desc, NULL) - || chrec_contains_undetermined (niter_desc.niter)) - return cond; + if (dump_enabled_p ()) + dump_printf_loc (MSG_NOTE, vect_location, "Loop has %d exits.\n", + exits.length ()); - niter_assumptions = niter_desc.assumptions; - may_be_zero = niter_desc.may_be_zero; - niter = niter_desc.niter; + edge exit; + unsigned int i; + FOR_EACH_VEC_ELT (exits, i, exit) + { + gcond *cond = get_edge_condition (exit); + if (cond) + conds.safe_push (cond); - if (may_be_zero && integer_zerop (may_be_zero)) - may_be_zero = NULL_TREE; + if (dump_enabled_p ()) + dump_printf_loc (MSG_NOTE, vect_location, "Analyzing exit %d...\n", i); - if (may_be_zero) - { - if (COMPARISON_CLASS_P (may_be_zero)) + may_be_zero = NULL_TREE; + if (!number_of_iterations_exit_assumptions (loop, exit, &niter_desc, NULL) + || chrec_contains_undetermined (niter_desc.niter)) + continue; + + niter_assumptions = niter_desc.assumptions; + may_be_zero = niter_desc.may_be_zero; + niter = niter_desc.niter; + + if (may_be_zero && integer_zerop (may_be_zero)) + may_be_zero = NULL_TREE; + + if (may_be_zero) { - /* Try to combine may_be_zero with assumptions, this can simplify - computation of niter expression. */ - if (niter_assumptions && !integer_nonzerop (niter_assumptions)) - niter_assumptions = fold_build2 (TRUTH_AND_EXPR, boolean_type_node, - niter_assumptions, - fold_build1 (TRUTH_NOT_EXPR, - boolean_type_node, - may_be_zero)); + if (COMPARISON_CLASS_P (may_be_zero)) + { + /* Try to combine may_be_zero with assumptions, this can simplify + computation of niter expression. */ + if (niter_assumptions && !integer_nonzerop (niter_assumptions)) + niter_assumptions = fold_build2 (TRUTH_AND_EXPR, boolean_type_node, + niter_assumptions, + fold_build1 (TRUTH_NOT_EXPR, + boolean_type_node, + may_be_zero)); + else + niter = fold_build3 (COND_EXPR, TREE_TYPE (niter), may_be_zero, + build_int_cst (TREE_TYPE (niter), 0), + rewrite_to_non_trapping_overflow (niter)); + + may_be_zero = NULL_TREE; + } + else if (integer_nonzerop (may_be_zero) && exit == loop->vec_loop_iv) + { + *number_of_iterationsm1 = build_int_cst (TREE_TYPE (niter), 0); + *number_of_iterations = build_int_cst (TREE_TYPE (niter), 1); + continue; + } else - niter = fold_build3 (COND_EXPR, TREE_TYPE (niter), may_be_zero, - build_int_cst (TREE_TYPE (niter), 0), - rewrite_to_non_trapping_overflow (niter)); + continue; + } - may_be_zero = NULL_TREE; - } - else if (integer_nonzerop (may_be_zero)) + /* Loop assumptions are based off the normal exit. */ + if (exit == loop->vec_loop_iv) { - *number_of_iterationsm1 = build_int_cst (TREE_TYPE (niter), 0); - *number_of_iterations = build_int_cst (TREE_TYPE (niter), 1); - return cond; + *assumptions = niter_assumptions; + *number_of_iterationsm1 = niter; + + /* We want the number of loop header executions which is the number + of latch executions plus one. + ??? For UINT_MAX latch executions this number overflows to zero + for loops like do { n++; } while (n != 0); */ + if (niter && !chrec_contains_undetermined (niter)) + niter = fold_build2 (PLUS_EXPR, TREE_TYPE (niter), + unshare_expr (niter), + build_int_cst (TREE_TYPE (niter), 1)); + *number_of_iterations = niter; } - else - return cond; } - *assumptions = niter_assumptions; - *number_of_iterationsm1 = niter; - - /* We want the number of loop header executions which is the number - of latch executions plus one. - ??? For UINT_MAX latch executions this number overflows to zero - for loops like do { n++; } while (n != 0); */ - if (niter && !chrec_contains_undetermined (niter)) - niter = fold_build2 (PLUS_EXPR, TREE_TYPE (niter), unshare_expr (niter), - build_int_cst (TREE_TYPE (niter), 1)); - *number_of_iterations = niter; + if (dump_enabled_p ()) + dump_printf_loc (MSG_NOTE, vect_location, "All loop exits successfully analyzed.\n"); - return cond; + return conds; } /* Function bb_in_loop_p @@ -1768,15 +1794,26 @@ vect_analyze_loop_form (class loop *loop, vect_loop_form_info *info) "not vectorized:" " abnormal loop exit edge.\n"); - info->loop_cond + info->conds = vect_get_loop_niters (loop, &info->assumptions, &info->number_of_iterations, &info->number_of_iterationsm1); - if (!info->loop_cond) + + if (info->conds.is_empty ()) return opt_result::failure_at (vect_location, "not vectorized: complicated exit condition.\n"); + /* Determine what the primary and alternate exit conds are. */ + info->alt_loop_conds.create (info->conds.length () - 1); + for (gcond *cond : info->conds) + { + if (loop->vec_loop_iv->src != gimple_bb (cond)) + info->alt_loop_conds.quick_push (cond); + else + info->loop_cond = cond; + } + if (integer_zerop (info->assumptions) || !info->number_of_iterations || chrec_contains_undetermined (info->number_of_iterations)) @@ -1821,8 +1858,14 @@ vect_create_loop_vinfo (class loop *loop, vec_info_shared *shared, if (!integer_onep (info->assumptions) && !main_loop_info) LOOP_VINFO_NITERS_ASSUMPTIONS (loop_vinfo) = info->assumptions; - stmt_vec_info loop_cond_info = loop_vinfo->lookup_stmt (info->loop_cond); - STMT_VINFO_TYPE (loop_cond_info) = loop_exit_ctrl_vec_info_type; + for (gcond *cond : info->alt_loop_conds) + { + stmt_vec_info loop_cond_info = loop_vinfo->lookup_stmt (cond); + STMT_VINFO_TYPE (loop_cond_info) = loop_exit_ctrl_vec_info_type; + } + LOOP_VINFO_LOOP_CONDS (loop_vinfo).safe_splice (info->alt_loop_conds); + LOOP_VINFO_LOOP_IV_COND (loop_vinfo) = info->loop_cond; + if (info->inner_loop_cond) { stmt_vec_info inner_loop_cond_info @@ -3520,6 +3563,9 @@ vect_analyze_loop (class loop *loop, vec_info_shared *shared) "***** Choosing vector mode %s\n", GET_MODE_NAME (first_loop_vinfo->vector_mode)); + loop_form_info.conds.release (); + loop_form_info.alt_loop_conds.release (); + /* Only vectorize epilogues if PARAM_VECT_EPILOGUES_NOMASK is enabled, SIMDUID is not set, it is the innermost loop and we have either already found the loop's SIMDLEN or there was no SIMDLEN to @@ -3631,6 +3677,9 @@ vect_analyze_loop (class loop *loop, vec_info_shared *shared) (first_loop_vinfo->epilogue_vinfos[0]->vector_mode)); } + loop_form_info.conds.release (); + loop_form_info.alt_loop_conds.release (); + return first_loop_vinfo; } diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h index bd5eceb5da7a45ef036cd14609ebe091799320bf..1cc003c12e2447eca878f56cb019236f56e96f85 100644 --- a/gcc/tree-vectorizer.h +++ b/gcc/tree-vectorizer.h @@ -876,6 +876,12 @@ public: we need to peel off iterations at the end to form an epilogue loop. */ bool peeling_for_niter; + /* List of loop additional IV conditionals found in the loop. */ + auto_vec conds; + + /* Main loop IV cond. */ + gcond* loop_iv_cond; + /* True if there are no loop carried data dependencies in the loop. If loop->safelen <= 1, then this is always true, either the loop didn't have any loop carried data dependencies, or the loop is being @@ -966,6 +972,8 @@ public: #define LOOP_VINFO_REDUCTION_CHAINS(L) (L)->reduction_chains #define LOOP_VINFO_PEELING_FOR_GAPS(L) (L)->peeling_for_gaps #define LOOP_VINFO_PEELING_FOR_NITER(L) (L)->peeling_for_niter +#define LOOP_VINFO_LOOP_CONDS(L) (L)->conds +#define LOOP_VINFO_LOOP_IV_COND(L) (L)->loop_iv_cond #define LOOP_VINFO_NO_DATA_DEPENDENCIES(L) (L)->no_data_dependencies #define LOOP_VINFO_SCALAR_LOOP(L) (L)->scalar_loop #define LOOP_VINFO_SCALAR_LOOP_SCALING(L) (L)->scalar_loop_scaling @@ -2353,7 +2361,9 @@ struct vect_loop_form_info tree number_of_iterations; tree number_of_iterationsm1; tree assumptions; + vec conds; gcond *loop_cond; + vec alt_loop_conds; gcond *inner_loop_cond; }; extern opt_result vect_analyze_loop_form (class loop *, vect_loop_form_info *); --- a/gcc/tree-vect-loop.cc +++ b/gcc/tree-vect-loop.cc @@ -849,80 +849,106 @@ vect_fixup_scalar_cycles_with_patterns (loop_vec_info loop_vinfo) in NUMBER_OF_ITERATIONSM1. Place the condition under which the niter information holds in ASSUMPTIONS. - Return the loop exit condition. */ + Return the loop exit conditions. */ -static gcond * +static vec vect_get_loop_niters (class loop *loop, tree *assumptions, tree *number_of_iterations, tree *number_of_iterationsm1) { - edge exit = single_exit (loop); + auto_vec exits = get_loop_exit_edges (loop); + vec conds; + conds.create (exits.length ()); class tree_niter_desc niter_desc; tree niter_assumptions, niter, may_be_zero; - gcond *cond = get_loop_exit_condition (loop); *assumptions = boolean_true_node; *number_of_iterationsm1 = chrec_dont_know; *number_of_iterations = chrec_dont_know; + DUMP_VECT_SCOPE ("get_loop_niters"); - if (!exit) - return cond; + if (exits.is_empty ()) + return conds; - may_be_zero = NULL_TREE; - if (!number_of_iterations_exit_assumptions (loop, exit, &niter_desc, NULL) - || chrec_contains_undetermined (niter_desc.niter)) - return cond; + if (dump_enabled_p ()) + dump_printf_loc (MSG_NOTE, vect_location, "Loop has %d exits.\n", + exits.length ()); - niter_assumptions = niter_desc.assumptions; - may_be_zero = niter_desc.may_be_zero; - niter = niter_desc.niter; + edge exit; + unsigned int i; + FOR_EACH_VEC_ELT (exits, i, exit) + { + gcond *cond = get_edge_condition (exit); + if (cond) + conds.safe_push (cond); - if (may_be_zero && integer_zerop (may_be_zero)) - may_be_zero = NULL_TREE; + if (dump_enabled_p ()) + dump_printf_loc (MSG_NOTE, vect_location, "Analyzing exit %d...\n", i); - if (may_be_zero) - { - if (COMPARISON_CLASS_P (may_be_zero)) + may_be_zero = NULL_TREE; + if (!number_of_iterations_exit_assumptions (loop, exit, &niter_desc, NULL) + || chrec_contains_undetermined (niter_desc.niter)) + continue; + + niter_assumptions = niter_desc.assumptions; + may_be_zero = niter_desc.may_be_zero; + niter = niter_desc.niter; + + if (may_be_zero && integer_zerop (may_be_zero)) + may_be_zero = NULL_TREE; + + if (may_be_zero) { - /* Try to combine may_be_zero with assumptions, this can simplify - computation of niter expression. */ - if (niter_assumptions && !integer_nonzerop (niter_assumptions)) - niter_assumptions = fold_build2 (TRUTH_AND_EXPR, boolean_type_node, - niter_assumptions, - fold_build1 (TRUTH_NOT_EXPR, - boolean_type_node, - may_be_zero)); + if (COMPARISON_CLASS_P (may_be_zero)) + { + /* Try to combine may_be_zero with assumptions, this can simplify + computation of niter expression. */ + if (niter_assumptions && !integer_nonzerop (niter_assumptions)) + niter_assumptions = fold_build2 (TRUTH_AND_EXPR, boolean_type_node, + niter_assumptions, + fold_build1 (TRUTH_NOT_EXPR, + boolean_type_node, + may_be_zero)); + else + niter = fold_build3 (COND_EXPR, TREE_TYPE (niter), may_be_zero, + build_int_cst (TREE_TYPE (niter), 0), + rewrite_to_non_trapping_overflow (niter)); + + may_be_zero = NULL_TREE; + } + else if (integer_nonzerop (may_be_zero) && exit == loop->vec_loop_iv) + { + *number_of_iterationsm1 = build_int_cst (TREE_TYPE (niter), 0); + *number_of_iterations = build_int_cst (TREE_TYPE (niter), 1); + continue; + } else - niter = fold_build3 (COND_EXPR, TREE_TYPE (niter), may_be_zero, - build_int_cst (TREE_TYPE (niter), 0), - rewrite_to_non_trapping_overflow (niter)); + continue; + } - may_be_zero = NULL_TREE; - } - else if (integer_nonzerop (may_be_zero)) + /* Loop assumptions are based off the normal exit. */ + if (exit == loop->vec_loop_iv) { - *number_of_iterationsm1 = build_int_cst (TREE_TYPE (niter), 0); - *number_of_iterations = build_int_cst (TREE_TYPE (niter), 1); - return cond; + *assumptions = niter_assumptions; + *number_of_iterationsm1 = niter; + + /* We want the number of loop header executions which is the number + of latch executions plus one. + ??? For UINT_MAX latch executions this number overflows to zero + for loops like do { n++; } while (n != 0); */ + if (niter && !chrec_contains_undetermined (niter)) + niter = fold_build2 (PLUS_EXPR, TREE_TYPE (niter), + unshare_expr (niter), + build_int_cst (TREE_TYPE (niter), 1)); + *number_of_iterations = niter; } - else - return cond; } - *assumptions = niter_assumptions; - *number_of_iterationsm1 = niter; - - /* We want the number of loop header executions which is the number - of latch executions plus one. - ??? For UINT_MAX latch executions this number overflows to zero - for loops like do { n++; } while (n != 0); */ - if (niter && !chrec_contains_undetermined (niter)) - niter = fold_build2 (PLUS_EXPR, TREE_TYPE (niter), unshare_expr (niter), - build_int_cst (TREE_TYPE (niter), 1)); - *number_of_iterations = niter; + if (dump_enabled_p ()) + dump_printf_loc (MSG_NOTE, vect_location, "All loop exits successfully analyzed.\n"); - return cond; + return conds; } /* Function bb_in_loop_p @@ -1768,15 +1794,26 @@ vect_analyze_loop_form (class loop *loop, vect_loop_form_info *info) "not vectorized:" " abnormal loop exit edge.\n"); - info->loop_cond + info->conds = vect_get_loop_niters (loop, &info->assumptions, &info->number_of_iterations, &info->number_of_iterationsm1); - if (!info->loop_cond) + + if (info->conds.is_empty ()) return opt_result::failure_at (vect_location, "not vectorized: complicated exit condition.\n"); + /* Determine what the primary and alternate exit conds are. */ + info->alt_loop_conds.create (info->conds.length () - 1); + for (gcond *cond : info->conds) + { + if (loop->vec_loop_iv->src != gimple_bb (cond)) + info->alt_loop_conds.quick_push (cond); + else + info->loop_cond = cond; + } + if (integer_zerop (info->assumptions) || !info->number_of_iterations || chrec_contains_undetermined (info->number_of_iterations)) @@ -1821,8 +1858,14 @@ vect_create_loop_vinfo (class loop *loop, vec_info_shared *shared, if (!integer_onep (info->assumptions) && !main_loop_info) LOOP_VINFO_NITERS_ASSUMPTIONS (loop_vinfo) = info->assumptions; - stmt_vec_info loop_cond_info = loop_vinfo->lookup_stmt (info->loop_cond); - STMT_VINFO_TYPE (loop_cond_info) = loop_exit_ctrl_vec_info_type; + for (gcond *cond : info->alt_loop_conds) + { + stmt_vec_info loop_cond_info = loop_vinfo->lookup_stmt (cond); + STMT_VINFO_TYPE (loop_cond_info) = loop_exit_ctrl_vec_info_type; + } + LOOP_VINFO_LOOP_CONDS (loop_vinfo).safe_splice (info->alt_loop_conds); + LOOP_VINFO_LOOP_IV_COND (loop_vinfo) = info->loop_cond; + if (info->inner_loop_cond) { stmt_vec_info inner_loop_cond_info @@ -3520,6 +3563,9 @@ vect_analyze_loop (class loop *loop, vec_info_shared *shared) "***** Choosing vector mode %s\n", GET_MODE_NAME (first_loop_vinfo->vector_mode)); + loop_form_info.conds.release (); + loop_form_info.alt_loop_conds.release (); + /* Only vectorize epilogues if PARAM_VECT_EPILOGUES_NOMASK is enabled, SIMDUID is not set, it is the innermost loop and we have either already found the loop's SIMDLEN or there was no SIMDLEN to @@ -3631,6 +3677,9 @@ vect_analyze_loop (class loop *loop, vec_info_shared *shared) (first_loop_vinfo->epilogue_vinfos[0]->vector_mode)); } + loop_form_info.conds.release (); + loop_form_info.alt_loop_conds.release (); + return first_loop_vinfo; } diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h index bd5eceb5da7a45ef036cd14609ebe091799320bf..1cc003c12e2447eca878f56cb019236f56e96f85 100644 --- a/gcc/tree-vectorizer.h +++ b/gcc/tree-vectorizer.h @@ -876,6 +876,12 @@ public: we need to peel off iterations at the end to form an epilogue loop. */ bool peeling_for_niter; + /* List of loop additional IV conditionals found in the loop. */ + auto_vec conds; + + /* Main loop IV cond. */ + gcond* loop_iv_cond; + /* True if there are no loop carried data dependencies in the loop. If loop->safelen <= 1, then this is always true, either the loop didn't have any loop carried data dependencies, or the loop is being @@ -966,6 +972,8 @@ public: #define LOOP_VINFO_REDUCTION_CHAINS(L) (L)->reduction_chains #define LOOP_VINFO_PEELING_FOR_GAPS(L) (L)->peeling_for_gaps #define LOOP_VINFO_PEELING_FOR_NITER(L) (L)->peeling_for_niter +#define LOOP_VINFO_LOOP_CONDS(L) (L)->conds +#define LOOP_VINFO_LOOP_IV_COND(L) (L)->loop_iv_cond #define LOOP_VINFO_NO_DATA_DEPENDENCIES(L) (L)->no_data_dependencies #define LOOP_VINFO_SCALAR_LOOP(L) (L)->scalar_loop #define LOOP_VINFO_SCALAR_LOOP_SCALING(L) (L)->scalar_loop_scaling @@ -2353,7 +2361,9 @@ struct vect_loop_form_info tree number_of_iterations; tree number_of_iterationsm1; tree assumptions; + vec conds; gcond *loop_cond; + vec alt_loop_conds; gcond *inner_loop_cond; }; extern opt_result vect_analyze_loop_form (class loop *, vect_loop_form_info *);