From patchwork Tue Oct 24 11:19:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: liuhongt X-Patchwork-Id: 157389 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:ce89:0:b0:403:3b70:6f57 with SMTP id p9csp1867914vqx; Tue, 24 Oct 2023 04:22:16 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHqf7/3UoZDhhtqpdz7hHKamdCsKpxqrrDbfnc2hlIdEQJ/bph200rIa1MxpQhOUzfLNtR0 X-Received: by 2002:ad4:5cc2:0:b0:66d:169a:d41c with SMTP id iu2-20020ad45cc2000000b0066d169ad41cmr12596185qvb.19.1698146536210; Tue, 24 Oct 2023 04:22:16 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1698146536; cv=pass; d=google.com; s=arc-20160816; b=JKfOQEf0/ahN5HlJq+73swJcCXEP1zf9ozWIwOIbs6fkBRmA2S1k0C0u1hIcjtjkBU jCRMXeegehxPlsO30tLOsu+MLN8Fez35RPD9y7LO9KobQ1+Wd8ZC0mQJWEMpsygXdieh 9wUaFvomFlD70ILIUiJjybO6/om6IYT5f/r1TDInFBMwVou91uq016gCkUjIwGD8EMnD SDLl4EzI5gGi9jTuMT7K9v9wT3vFfXWXv7KD80nmb9SQt552EPfpp2W9O3aB9/MkU19U dBNse6KcVnA2q6MwNedplGcnvdIjRO9QfYDBOreIQdCCU28uBiKGDzd7JRkm92NrOKrE 2XVA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature:arc-filter:dmarc-filter:delivered-to; bh=rCk0UHbI4PFZyExUjh+OzswMoFW1AI1Kx7ocKaGHm9A=; fh=ChXOctppJn0KECDRINafwUY5xHRufGHaa0Ju9pddrcQ=; b=HqnxszvJ3nLrOgh+2xMweq35Vow3kh0GYCRGDRsVRSupHoCpAOTzMgohuOth1ib7xR y4WWZ8m4PJbIIx9CR9tOkCw58aEXcJkgodsEEo+kWEylShYFxOL6DZxnzGI8Yf8aXuU8 Di/BIk9Ei26lBXkAB0g2Ko95Mc5ZxIwKisUD4JeJpsRp1pRkDA3ZcYvG7Qqlak1KfMJ2 H/pRCSXf7eqkHwt3BUOFGT/c9J0EGT38uWGEL2gCh9fqKTLHF34XbEE+wEN66uQ7hcEE 7JI6r4eGP6i7uB3vXKed2PywpqgYMzVwUhf57T6r55QnvNplHqw7Fw/EMCiUVvSQMFlq cPNA== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=FPk+VdQI; arc=pass (i=1); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from server2.sourceware.org (server2.sourceware.org. [2620:52:3:1:0:246e:9693:128c]) by mx.google.com with ESMTPS id e12-20020ac85dcc000000b0041cbd1b361fsi6776062qtx.763.2023.10.24.04.22.16 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Oct 2023 04:22:16 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) client-ip=2620:52:3:1:0:246e:9693:128c; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=FPk+VdQI; arc=pass (i=1); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id F30003856243 for ; Tue, 24 Oct 2023 11:22:15 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.115]) by sourceware.org (Postfix) with ESMTPS id D7AD33858C36 for ; Tue, 24 Oct 2023 11:21:45 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org D7AD33858C36 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=intel.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org D7AD33858C36 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=192.55.52.115 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1698146512; cv=none; b=tFgmwoE/rzh9nCTuT3OftBmwN4GFJs99M1qe9fm7rSN3LW/7TJP+V6FDG0J/vkBFJ2rZmTbxrGDXkW1jpM2+AChOI9uW+Zygbj5SLfzL+KJ1TYuvo6a56Xw+tV5My1jzQyzOMB2IDLrH/XwUa0E8qcVLyOj0kGpyAlZt/Q94Ofc= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1698146512; c=relaxed/simple; bh=wMpbf08Ak0VxntwsAcTgdVWLwlVqBi6EllQ6R1ly6m0=; h=DKIM-Signature:From:To:Subject:Date:Message-Id:MIME-Version; b=hBEKEkmgpwSIiWKKg79XSZxxlpwo7lGlxnNkl2LuJkga4w1Oon0OJIY4mBs1i1D5PKEvDfUEz/Xb2B+yxyfJEgdhMi5kgtpyyUztrbCOeT8jo6uInstwv8fMlKO9M5p4kMsIDFyy1EupvImzO64O7ladoS8xKaLvQf9BOZ+oUAE= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1698146505; x=1729682505; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=wMpbf08Ak0VxntwsAcTgdVWLwlVqBi6EllQ6R1ly6m0=; b=FPk+VdQIcECGXBLzed0l/2Cs0pjhatKFzUBhdabz5RsAfZ40qu6WztG3 oGvLxS86LJX5CDokVFhNZyQXyIY5Y2wgyfze73SDkN5kXhUkN0Ugjrt6z 8gzFRZULUDl2YdJcD0ZKeG9UyHMDFHJKwNKChpSmJtyRLCDUoruX94rsG WfYtz3JN6LceNvnuUpe1RqepSpDZPVCyJ/FsQIcamBHDdSM7TFj0I6SNI SGaMYpiWS5KG5S74i46uzNudIvoNUXrJwi3coVtiZYUOtb3JPSqCyHLXn n3l1C6bX37wAZfNvy5jfeF3V0kTs17WeWD0X9JQ6CQcCTlHdFU2EqYiTj A==; X-IronPort-AV: E=McAfee;i="6600,9927,10872"; a="386846410" X-IronPort-AV: E=Sophos;i="6.03,247,1694761200"; d="scan'208";a="386846410" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Oct 2023 04:21:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10872"; a="758445719" X-IronPort-AV: E=Sophos;i="6.03,247,1694761200"; d="scan'208";a="758445719" Received: from shvmail03.sh.intel.com ([10.239.245.20]) by orsmga002.jf.intel.com with ESMTP; 24 Oct 2023 04:21:42 -0700 Received: from shliclel4217.sh.intel.com (shliclel4217.sh.intel.com [10.239.240.127]) by shvmail03.sh.intel.com (Postfix) with ESMTP id B179E1005671; Tue, 24 Oct 2023 19:21:41 +0800 (CST) From: liuhongt To: gcc-patches@gcc.gnu.org Cc: crazylht@gmail.com, hjl.tools@gmail.com Subject: [PATCH GCC13 backport] Avoid compile time hog on vect_peel_nonlinear_iv_init for nonlinear induction vec_step_op_mul when iteration count is too big. Date: Tue, 24 Oct 2023 19:19:41 +0800 Message-Id: <20231024111941.1407250-1-hongtao.liu@intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-12.3 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_LOW, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780163503634650155 X-GMAIL-MSGID: 1780635702097338694 This is the backport patch for releases/gcc-13 branch, the original patch for main trunk is at [1]. The only difference between this backport patch and [1] is GCC13 doesn't support auto_mpz, So this patch manually use mpz_init/mpz_clear. [1] https://gcc.gnu.org/pipermail/gcc-patches/2023-October/633661.html Bootstrapped and regtested on x86_64-pc-linux-gnu{-m32,} Ok for backport to releases/gcc-13? There's loop in vect_peel_nonlinear_iv_init to get init_expr * pow (step_expr, skip_niters). When skipn_iters is too big, compile time hogs. To avoid that, optimize init_expr * pow (step_expr, skip_niters) to init_expr << (exact_log2 (step_expr) * skip_niters) when step_expr is pow of 2, otherwise give up vectorization when skip_niters >= TYPE_PRECISION (TREE_TYPE (init_expr)). Also give up vectorization when niters_skip is negative which will be used for fully masked loop. gcc/ChangeLog: PR tree-optimization/111820 PR tree-optimization/111833 * tree-vect-loop-manip.cc (vect_can_peel_nonlinear_iv_p): Give up vectorization for nonlinear iv vect_step_op_mul when step_expr is not exact_log2 and niters is greater than TYPE_PRECISION (TREE_TYPE (step_expr)). Also don't vectorize for nagative niters_skip which will be used by fully masked loop. (vect_can_advance_ivs_p): Pass whole phi_info to vect_can_peel_nonlinear_iv_p. * tree-vect-loop.cc (vect_peel_nonlinear_iv_init): Optimize init_expr * pow (step_expr, skipn) to init_expr << (log2 (step_expr) * skipn) when step_expr is exact_log2. gcc/testsuite/ChangeLog: * gcc.target/i386/pr111820-1.c: New test. * gcc.target/i386/pr111820-2.c: New test. * gcc.target/i386/pr111820-3.c: New test. * gcc.target/i386/pr103144-mul-1.c: Adjust testcase. * gcc.target/i386/pr103144-mul-2.c: Adjust testcase. --- .../gcc.target/i386/pr103144-mul-1.c | 8 +++--- .../gcc.target/i386/pr103144-mul-2.c | 8 +++--- gcc/testsuite/gcc.target/i386/pr111820-1.c | 16 +++++++++++ gcc/testsuite/gcc.target/i386/pr111820-2.c | 16 +++++++++++ gcc/testsuite/gcc.target/i386/pr111820-3.c | 16 +++++++++++ gcc/tree-vect-loop-manip.cc | 28 +++++++++++++++++-- gcc/tree-vect-loop.cc | 21 +++++++++++--- 7 files changed, 98 insertions(+), 15 deletions(-) create mode 100644 gcc/testsuite/gcc.target/i386/pr111820-1.c create mode 100644 gcc/testsuite/gcc.target/i386/pr111820-2.c create mode 100644 gcc/testsuite/gcc.target/i386/pr111820-3.c diff --git a/gcc/testsuite/gcc.target/i386/pr103144-mul-1.c b/gcc/testsuite/gcc.target/i386/pr103144-mul-1.c index 640c34fd959..913d7737dcd 100644 --- a/gcc/testsuite/gcc.target/i386/pr103144-mul-1.c +++ b/gcc/testsuite/gcc.target/i386/pr103144-mul-1.c @@ -11,7 +11,7 @@ foo_mul (int* a, int b) for (int i = 0; i != N; i++) { a[i] = b; - b *= 3; + b *= 4; } } @@ -23,7 +23,7 @@ foo_mul_const (int* a) for (int i = 0; i != N; i++) { a[i] = b; - b *= 3; + b *= 4; } } @@ -34,7 +34,7 @@ foo_mul_peel (int* a, int b) for (int i = 0; i != 39; i++) { a[i] = b; - b *= 3; + b *= 4; } } @@ -46,6 +46,6 @@ foo_mul_peel_const (int* a) for (int i = 0; i != 39; i++) { a[i] = b; - b *= 3; + b *= 4; } } diff --git a/gcc/testsuite/gcc.target/i386/pr103144-mul-2.c b/gcc/testsuite/gcc.target/i386/pr103144-mul-2.c index 39fdea3a69d..b2ff186e335 100644 --- a/gcc/testsuite/gcc.target/i386/pr103144-mul-2.c +++ b/gcc/testsuite/gcc.target/i386/pr103144-mul-2.c @@ -16,12 +16,12 @@ avx2_test (void) __builtin_memset (epi32_exp, 0, N * sizeof (int)); int b = 8; - v8si init = __extension__(v8si) { b, b * 3, b * 9, b * 27, b * 81, b * 243, b * 729, b * 2187 }; + v8si init = __extension__(v8si) { b, b * 4, b * 16, b * 64, b * 256, b * 1024, b * 4096, b * 16384 }; for (int i = 0; i != N / 8; i++) { memcpy (epi32_exp + i * 8, &init, 32); - init *= 6561; + init *= 65536; } foo_mul (epi32_dst, b); @@ -32,11 +32,11 @@ avx2_test (void) if (__builtin_memcmp (epi32_dst, epi32_exp, 39 * 4) != 0) __builtin_abort (); - init = __extension__(v8si) { 1, 3, 9, 27, 81, 243, 729, 2187 }; + init = __extension__(v8si) { 1, 4, 16, 64, 256, 1024, 4096, 16384 }; for (int i = 0; i != N / 8; i++) { memcpy (epi32_exp + i * 8, &init, 32); - init *= 6561; + init *= 65536; } foo_mul_const (epi32_dst); diff --git a/gcc/testsuite/gcc.target/i386/pr111820-1.c b/gcc/testsuite/gcc.target/i386/pr111820-1.c new file mode 100644 index 00000000000..50e960c39d4 --- /dev/null +++ b/gcc/testsuite/gcc.target/i386/pr111820-1.c @@ -0,0 +1,16 @@ +/* { dg-do compile } */ +/* { dg-options "-O3 -mavx2 -fno-tree-vrp -Wno-aggressive-loop-optimizations -fdump-tree-vect-details" } */ +/* { dg-final { scan-tree-dump "Avoid compile time hog on vect_peel_nonlinear_iv_init for nonlinear induction vec_step_op_mul when iteration count is too big" "vect" } } */ + +int r; +int r_0; + +void f1 (void) +{ + int n = 0; + while (-- n) + { + r_0 += r; + r *= 3; + } +} diff --git a/gcc/testsuite/gcc.target/i386/pr111820-2.c b/gcc/testsuite/gcc.target/i386/pr111820-2.c new file mode 100644 index 00000000000..dbeceb228c3 --- /dev/null +++ b/gcc/testsuite/gcc.target/i386/pr111820-2.c @@ -0,0 +1,16 @@ +/* { dg-do compile } */ +/* { dg-options "-O3 -mavx2 -fno-tree-vrp -fdump-tree-vect-details -Wno-aggressive-loop-optimizations" } */ +/* { dg-final { scan-tree-dump "LOOP VECTORIZED" "vect" } } */ + +int r; +int r_0; + +void f (void) +{ + int n = 0; + while (-- n) + { + r_0 += r ; + r *= 2; + } +} diff --git a/gcc/testsuite/gcc.target/i386/pr111820-3.c b/gcc/testsuite/gcc.target/i386/pr111820-3.c new file mode 100644 index 00000000000..b778f517663 --- /dev/null +++ b/gcc/testsuite/gcc.target/i386/pr111820-3.c @@ -0,0 +1,16 @@ +/* { dg-do compile } */ +/* { dg-options "-O3 -mavx2 -fno-tree-vrp -fdump-tree-vect-details -Wno-aggressive-loop-optimizations" } */ +/* { dg-final { scan-tree-dump "LOOP VECTORIZED" "vect" } } */ + +int r; +int r_0; + +void f (void) +{ + int n = 14; + while (-- n) + { + r_0 += r ; + r *= 3; + } +} diff --git a/gcc/tree-vect-loop-manip.cc b/gcc/tree-vect-loop-manip.cc index f60fa50e8f4..767b19b15a1 100644 --- a/gcc/tree-vect-loop-manip.cc +++ b/gcc/tree-vect-loop-manip.cc @@ -1391,8 +1391,10 @@ iv_phi_p (stmt_vec_info stmt_info) /* Return true if vectorizer can peel for nonlinear iv. */ static bool vect_can_peel_nonlinear_iv_p (loop_vec_info loop_vinfo, - enum vect_induction_op_type induction_type) + stmt_vec_info stmt_info) { + enum vect_induction_op_type induction_type + = STMT_VINFO_LOOP_PHI_EVOLUTION_TYPE (stmt_info); tree niters_skip; /* Init_expr will be update by vect_update_ivs_after_vectorizer, if niters or vf is unkown: @@ -1413,11 +1415,31 @@ vect_can_peel_nonlinear_iv_p (loop_vec_info loop_vinfo, return false; } + /* Avoid compile time hog on vect_peel_nonlinear_iv_init. */ + if (induction_type == vect_step_op_mul) + { + tree step_expr = STMT_VINFO_LOOP_PHI_EVOLUTION_PART (stmt_info); + tree type = TREE_TYPE (step_expr); + + if (wi::exact_log2 (wi::to_wide (step_expr)) == -1 + && LOOP_VINFO_INT_NITERS(loop_vinfo) >= TYPE_PRECISION (type)) + { + if (dump_enabled_p ()) + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, + "Avoid compile time hog on" + " vect_peel_nonlinear_iv_init" + " for nonlinear induction vec_step_op_mul" + " when iteration count is too big.\n"); + return false; + } + } + /* Also doens't support peel for neg when niter is variable. ??? generate something like niter_expr & 1 ? init_expr : -init_expr? */ niters_skip = LOOP_VINFO_MASK_SKIP_NITERS (loop_vinfo); if ((niters_skip != NULL_TREE - && TREE_CODE (niters_skip) != INTEGER_CST) + && (TREE_CODE (niters_skip) != INTEGER_CST + || (HOST_WIDE_INT) TREE_INT_CST_LOW (niters_skip) < 0)) || (!vect_use_loop_mask_for_alignment_p (loop_vinfo) && LOOP_VINFO_PEELING_FOR_ALIGNMENT (loop_vinfo) < 0)) { @@ -1478,7 +1500,7 @@ vect_can_advance_ivs_p (loop_vec_info loop_vinfo) induction_type = STMT_VINFO_LOOP_PHI_EVOLUTION_TYPE (phi_info); if (induction_type != vect_step_op_add) { - if (!vect_can_peel_nonlinear_iv_p (loop_vinfo, induction_type)) + if (!vect_can_peel_nonlinear_iv_p (loop_vinfo, phi_info)) return false; continue; diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc index 92d2e0ef9be..2f098b96b1d 100644 --- a/gcc/tree-vect-loop.cc +++ b/gcc/tree-vect-loop.cc @@ -8728,13 +8728,26 @@ vect_peel_nonlinear_iv_init (gimple_seq* stmts, tree init_expr, { tree utype = unsigned_type_for (type); init_expr = gimple_convert (stmts, utype, init_expr); - unsigned skipn = TREE_INT_CST_LOW (skip_niters); + wide_int skipn = wi::to_wide (skip_niters); wide_int begin = wi::to_wide (step_expr); - for (unsigned i = 0; i != skipn - 1; i++) - begin = wi::mul (begin, wi::to_wide (step_expr)); + mpz_t base, exp, mod, res; + mpz_init (base); + mpz_init (mod); + mpz_init (exp); + mpz_init (res); + wi::to_mpz (begin, base, TYPE_SIGN (type)); + wi::to_mpz (skipn, exp, UNSIGNED); + mpz_ui_pow_ui (mod, 2, TYPE_PRECISION (type)); + mpz_powm (res, base, exp, mod); + begin = wi::from_mpz (type, res, TYPE_SIGN (type)); tree mult_expr = wide_int_to_tree (utype, begin); - init_expr = gimple_build (stmts, MULT_EXPR, utype, init_expr, mult_expr); + init_expr = gimple_build (stmts, MULT_EXPR, utype, + init_expr, mult_expr); init_expr = gimple_convert (stmts, type, init_expr); + mpz_clear (base); + mpz_clear (mod); + mpz_clear (exp); + mpz_clear (res); } break;