From patchwork Mon Aug 21 10:59:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "juzhe.zhong@rivai.ai" X-Patchwork-Id: 136372 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b82d:0:b0:3f2:4152:657d with SMTP id z13csp2918890vqi; Mon, 21 Aug 2023 04:00:46 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHUmrCiEs3EFR1dqV0dsICb5+KU3t3TzVZzTbydd1aZCuGyA4V9e+2Yqs7F6yvTP/JMEdGb X-Received: by 2002:aa7:d397:0:b0:522:cb97:f196 with SMTP id x23-20020aa7d397000000b00522cb97f196mr5129784edq.36.1692615646349; Mon, 21 Aug 2023 04:00:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1692615646; cv=none; d=google.com; s=arc-20160816; b=QAeoKLQfmc6vq9VqSG/U0PerNiIAySh8jZX1R8fnl/egf9YhI563fjKSQjra1OAYu+ CZ2FcgFlU6zPhIH3ei+2VQCLR84UtrtOEoET2IYeBAW72JbM8NME7qtUqixJWFg1mStR LFjo2ySJnTKjQLMbsg8p0VPhP/R4sKlikKiWxK7ZKpxAMpBUEYW0/NTafa4hopCO93VC SQXT5jFk15JZNOhnOyqdISll5HoIwA6i3Afm0BX9EXkwly0MVOwslM3lnBQ1yK4Ocb68 SUSsjOUjc+16Ng3VFskmAp6+6FCOF3ArBWGaQQrH6cc+EWupNoJG/KvaQPcYtsM3fDM5 hdjw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:feedback-id :content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:dmarc-filter:delivered-to; bh=IcbusmlFzS10UlhYhvEXkg84w2SK0j2e6yqkXfyDye4=; fh=u9WsmLzfZIlIt0P/pXEHVk8CLkM78H9RyaM6HQ5JXtk=; b=c5YV4aTHCj4FdswhTy7iAwp9WKEIcRKgkC88c9ZdUxrw6bBZ2j7KvW9Sz0cQbDWwNU 9AzWs55spzee/bXbmzThWENLHNXT8B7wGAivrG3E7034ftZ3r4hVAsu0hXgDEctzAciz 9lS9/wgG4Na9cZzGnmItFeXmItlBOdfTaKarKBB0d7cf904nNbst6d+k2/zws+5U2eN3 tnlaopx5hj3X9M+7JsowV4K81N5b+Kp6nvxg0E3e7R+19s4aHkAak9Yzs2iweNvr80G+ E3dZTujYYDVq8OFa5g8D3wr7NrgbCFFLWzHmmSpVfq8QZf9BaGlDoDQporkXq2yu7DhZ LXMg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (server2.sourceware.org. [2620:52:3:1:0:246e:9693:128c]) by mx.google.com with ESMTPS id i18-20020aa7c712000000b0052570917661si5449235edq.615.2023.08.21.04.00.45 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Aug 2023 04:00:46 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) client-ip=2620:52:3:1:0:246e:9693:128c; Authentication-Results: mx.google.com; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id E77163856962 for ; Mon, 21 Aug 2023 11:00:35 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtpbgsg1.qq.com (smtpbgsg1.qq.com [54.254.200.92]) by sourceware.org (Postfix) with ESMTPS id 7DA9C3858025 for ; Mon, 21 Aug 2023 11:00:07 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 7DA9C3858025 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivai.ai Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivai.ai X-QQ-mid: bizesmtp90t1692615598tmndvkb4 Received: from rios-cad122.hadoop.rioslab.org ( [58.60.1.8]) by bizesmtp.qq.com (ESMTP) with id ; Mon, 21 Aug 2023 18:59:57 +0800 (CST) X-QQ-SSF: 01400000000000G0V000000A0000000 X-QQ-FEAT: cvpDInk2tjUq8Z7sqaQ8oWNZUvTT+2GWsEeTvETw6rFcVQjW0qpIdytAF3Z7o oo+7PRsQAAxUTclUvwDCxcEe4qIHa4+tj/naKl/VC1IlTLUmuixxrSFoTc3vybOY0oSV8iT uo2Y+MyLEBBwjl7PGnR5GC27Vnde4JWJul1lrfPSwkkrC9QtSGHaQDqdPPB1rCt03TpqryS nWbVGnJNUIkikqJyAgZOTbTIW1rWhBHZAZzOwY8jcqqbTL7z8V4SmIgrZq275wrsJzuZerX 0w4e1Xg6JI2fAGab8d+QqEXamBhvE/j6v6m3kW+Z3sfAiymcnEUb7aTtyvYotsjJUmAmozc gD0GtMolWE+4KAFg5p1+Lv2agwehkKVEIQ+ga6ilvQyBce07dYJgkUu74EQa7GsEI1DaLjB 0z/1lddFiSE/Un6ajRh8Sw== X-QQ-GoodBg: 2 X-BIZMAIL-ID: 13669396428601737728 From: Juzhe-Zhong To: gcc-patches@gcc.gnu.org Cc: richard.sandiford@arm.com, rguenther@suse.de, stefansf@linux.ibm.com, linkw@linux.ibm.com, Juzhe-Zhong Subject: [PATCH V5] VECT: Support loop len control on EXTRACT_LAST vectorization Date: Mon, 21 Aug 2023 18:59:55 +0800 Message-Id: <20230821105955.3471098-1-juzhe.zhong@rivai.ai> X-Mailer: git-send-email 2.36.3 MIME-Version: 1.0 X-QQ-SENDSIZE: 520 Feedback-ID: bizesmtp:rivai.ai:qybglogicsvrgz:qybglogicsvrgz7a-one-0 X-Spam-Status: No, score=-10.3 required=5.0 tests=BAYES_00, GIT_PATCH_0, KAM_DMARC_STATUS, KAM_SHORT, RCVD_IN_BARRACUDACENTRAL, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H4, RCVD_IN_MSPIKE_WL, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org Sender: "Gcc-patches" X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1774836144078812288 X-GMAIL-MSGID: 1774836144078812288 Co-Authored-By: Kewen.Lin Hi, @Richi and @Richard, base on previous disscussion, I simpily fix issuses for powerpc and s390 with your suggestions: - machine_mode len_load_mode = get_len_load_store_mode - (loop_vinfo->vector_mode, true).require (); - machine_mode len_store_mode = get_len_load_store_mode - (loop_vinfo->vector_mode, false).require (); + machine_mode len_load_mode, len_store_mode; + if (!get_len_load_store_mode (loop_vinfo->vector_mode, true) + .exists (&len_load_mode)) + return false; + if (!get_len_load_store_mode (loop_vinfo->vector_mode, false) + .exists (&len_store_mode)) + return false; Hi, @Kewen and @Stefan Could you test this patch again ? Thanks. Co-Authored-By: Kewen.Lin gcc/ChangeLog: * tree-vect-loop.cc (vect_verify_loop_lens): Add exists check. (vectorizable_live_operation): Add live vectorization for length loop control. gcc/testsuite/ChangeLog: * gcc.target/riscv/rvv/autovec/partial/live-1.c: New test. * gcc.target/riscv/rvv/autovec/partial/live_run-1.c: New test. --- .../riscv/rvv/autovec/partial/live-1.c | 34 +++++++ .../riscv/rvv/autovec/partial/live_run-1.c | 35 ++++++++ gcc/tree-vect-loop.cc | 89 ++++++++++++++----- 3 files changed, 138 insertions(+), 20 deletions(-) create mode 100644 gcc/testsuite/gcc.target/riscv/rvv/autovec/partial/live-1.c create mode 100644 gcc/testsuite/gcc.target/riscv/rvv/autovec/partial/live_run-1.c diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/partial/live-1.c b/gcc/testsuite/gcc.target/riscv/rvv/autovec/partial/live-1.c new file mode 100644 index 00000000000..75fa2eba8cc --- /dev/null +++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/partial/live-1.c @@ -0,0 +1,34 @@ +/* { dg-do compile } */ +/* { dg-additional-options "-march=rv32gcv -mabi=ilp32d --param riscv-autovec-preference=scalable -fdump-tree-optimized-details" } */ + +#include + +#define EXTRACT_LAST(TYPE) \ + TYPE __attribute__ ((noinline, noclone)) \ + test_##TYPE (TYPE *x, int n, TYPE value) \ + { \ + TYPE last; \ + for (int j = 0; j < n; ++j) \ + { \ + last = x[j]; \ + x[j] = last * value; \ + } \ + return last; \ + } + +#define TEST_ALL(T) \ + T (int8_t) \ + T (int16_t) \ + T (int32_t) \ + T (int64_t) \ + T (uint8_t) \ + T (uint16_t) \ + T (uint32_t) \ + T (uint64_t) \ + T (_Float16) \ + T (float) \ + T (double) + +TEST_ALL (EXTRACT_LAST) + +/* { dg-final { scan-tree-dump-times "\.VEC_EXTRACT" 10 "optimized" } } */ diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/partial/live_run-1.c b/gcc/testsuite/gcc.target/riscv/rvv/autovec/partial/live_run-1.c new file mode 100644 index 00000000000..42913a112c6 --- /dev/null +++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/partial/live_run-1.c @@ -0,0 +1,35 @@ +/* { dg-do run { target { riscv_vector } } } */ +/* { dg-additional-options "--param riscv-autovec-preference=scalable" } */ + +#include "live-1.c" + +#define N 107 +#define OP 70 + +#define TEST_LOOP(TYPE) \ + { \ + TYPE a[N]; \ + for (int i = 0; i < N; ++i) \ + { \ + a[i] = i * 2 + (i % 3); \ + asm volatile ("" ::: "memory"); \ + } \ + TYPE expected = a[N - 1]; \ + TYPE res = test_##TYPE (a, N, OP); \ + if (res != expected) \ + __builtin_abort (); \ + for (int i = 0; i < N; ++i) \ + { \ + TYPE old = i * 2 + (i % 3); \ + if (a[i] != (TYPE) (old * (TYPE) OP)) \ + __builtin_abort (); \ + asm volatile ("" ::: "memory"); \ + } \ + } + +int __attribute__ ((optimize (1))) +main (void) +{ + TEST_ALL (TEST_LOOP); + return 0; +} diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc index 1fcd8d07ea1..1cd6c291377 100644 --- a/gcc/tree-vect-loop.cc +++ b/gcc/tree-vect-loop.cc @@ -32,6 +32,8 @@ along with GCC; see the file COPYING3. If not see #include "tree-pass.h" #include "ssa.h" #include "optabs-tree.h" +#include "memmodel.h" +#include "optabs.h" #include "diagnostic-core.h" #include "fold-const.h" #include "stor-layout.h" @@ -1465,10 +1467,13 @@ vect_verify_loop_lens (loop_vec_info loop_vinfo) if (LOOP_VINFO_LENS (loop_vinfo).is_empty ()) return false; - machine_mode len_load_mode = get_len_load_store_mode - (loop_vinfo->vector_mode, true).require (); - machine_mode len_store_mode = get_len_load_store_mode - (loop_vinfo->vector_mode, false).require (); + machine_mode len_load_mode, len_store_mode; + if (!get_len_load_store_mode (loop_vinfo->vector_mode, true) + .exists (&len_load_mode)) + return false; + if (!get_len_load_store_mode (loop_vinfo->vector_mode, false) + .exists (&len_store_mode)) + return false; signed char partial_load_bias = internal_len_load_store_bias (IFN_LEN_LOAD, len_load_mode); @@ -10301,17 +10306,7 @@ vectorizable_live_operation (vec_info *vinfo, stmt_vec_info stmt_info, /* No transformation required. */ if (loop_vinfo && LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo)) { - if (!direct_internal_fn_supported_p (IFN_EXTRACT_LAST, vectype, - OPTIMIZE_FOR_SPEED)) - { - if (dump_enabled_p ()) - dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, - "can't operate on partial vectors " - "because the target doesn't support extract " - "last reduction.\n"); - LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo) = false; - } - else if (slp_node) + if (slp_node) { if (dump_enabled_p ()) dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, @@ -10331,9 +10326,26 @@ vectorizable_live_operation (vec_info *vinfo, stmt_vec_info stmt_info, else { gcc_assert (ncopies == 1 && !slp_node); - vect_record_loop_mask (loop_vinfo, - &LOOP_VINFO_MASKS (loop_vinfo), - 1, vectype, NULL); + if (direct_internal_fn_supported_p (IFN_EXTRACT_LAST, vectype, + OPTIMIZE_FOR_SPEED)) + vect_record_loop_mask (loop_vinfo, + &LOOP_VINFO_MASKS (loop_vinfo), + 1, vectype, NULL); + else if (can_vec_extract_var_idx_p ( + TYPE_MODE (vectype), TYPE_MODE (TREE_TYPE (vectype)))) + vect_record_loop_len (loop_vinfo, + &LOOP_VINFO_LENS (loop_vinfo), + 1, vectype, 1); + else + { + if (dump_enabled_p ()) + dump_printf_loc ( + MSG_MISSED_OPTIMIZATION, vect_location, + "can't operate on partial vectors " + "because the target doesn't support extract " + "last reduction.\n"); + LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo) = false; + } } } /* ??? Enable for loop costing as well. */ @@ -10359,7 +10371,9 @@ vectorizable_live_operation (vec_info *vinfo, stmt_vec_info stmt_info, gimple *vec_stmt; if (slp_node) { - gcc_assert (!loop_vinfo || !LOOP_VINFO_FULLY_MASKED_P (loop_vinfo)); + gcc_assert (!loop_vinfo + || (!LOOP_VINFO_FULLY_MASKED_P (loop_vinfo) + && !LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo))); /* Get the correct slp vectorized stmt. */ vec_lhs = SLP_TREE_VEC_DEFS (slp_node)[vec_entry]; @@ -10403,7 +10417,42 @@ vectorizable_live_operation (vec_info *vinfo, stmt_vec_info stmt_info, gimple_seq stmts = NULL; tree new_tree; - if (LOOP_VINFO_FULLY_MASKED_P (loop_vinfo)) + if (LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo)) + { + /* Emit: + + SCALAR_RES = VEC_EXTRACT + + where VEC_LHS is the vectorized live-out result and MASK is + the loop mask for the final iteration. */ + gcc_assert (ncopies == 1 && !slp_node); + gimple_seq tem = NULL; + gimple_stmt_iterator gsi = gsi_last (tem); + tree len + = vect_get_loop_len (loop_vinfo, &gsi, + &LOOP_VINFO_LENS (loop_vinfo), + 1, vectype, 0, 0); + + /* BIAS - 1. */ + signed char biasval = LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo); + tree bias_minus_one + = int_const_binop (MINUS_EXPR, + build_int_cst (TREE_TYPE (len), biasval), + build_one_cst (TREE_TYPE (len))); + + /* LAST_INDEX = LEN + (BIAS - 1). */ + tree last_index = gimple_build (&stmts, PLUS_EXPR, TREE_TYPE (len), + len, bias_minus_one); + + /* SCALAR_RES = VEC_EXTRACT . */ + tree scalar_res + = gimple_build (&stmts, CFN_VEC_EXTRACT, TREE_TYPE (vectype), + vec_lhs_phi, last_index); + + /* Convert the extracted vector element to the scalar type. */ + new_tree = gimple_convert (&stmts, lhs_type, scalar_res); + } + else if (LOOP_VINFO_FULLY_MASKED_P (loop_vinfo)) { /* Emit: