From patchwork Fri Apr 7 01:25:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "juzhe.zhong@rivai.ai" X-Patchwork-Id: 80597 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1414300vqo; Thu, 6 Apr 2023 18:25:49 -0700 (PDT) X-Google-Smtp-Source: AKy350bDzizs5F9yvH9EXt1zAyc8eMUHwlzN4wUHYJlK/Vk7Af9bi0eISc3YRpLOgQBJapXJ4g5+ X-Received: by 2002:a17:906:368e:b0:93e:739f:b0b3 with SMTP id a14-20020a170906368e00b0093e739fb0b3mr695105ejc.50.1680830749144; Thu, 06 Apr 2023 18:25:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680830749; cv=none; d=google.com; s=arc-20160816; b=Wid865so5vgfYWIyf3y93SMieeUbQm6x2twdbX+WLvlFVorA3EYzOqmAc5V0IpV+Vz hQxVrddqjnBtAJGWR4VKn1nZ1lOYZGH1vdtlTIY/f9Mm7OwrQ+0XKxWfJfa8rxanhDsh lwMcASfHUXznVKHmLNWI9/CzUyq7EVO2f9dr3osFlRZEKtu6PDy19q8qezVPUw75qGeU 4H7M4WQIa4wpeW/YmhghHtRSwFBvRJtwpeAtzuF+2zHIhRMfUEdOX/DplnZ7g98IvNJr IQkkDPYyUvleFKOpT8T3EEP2BFraJGnOkyBex/UswxlZ4Q5eFZfUZN8EGtcIDvRvNfiw /fRw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:feedback-id :content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:dmarc-filter:delivered-to; bh=tvGhrWVvMElBg0kmEY6/RIOb2PMnXo/kWVpH8jvxIEM=; b=ObRWIYsR9aZrX+YyEpiLd91OZUtCI1qaGMmGTx3CYei/MAnY1nNVGEiofpLMWExisq COZ9lQHvG+aTEUxvIfge07eNOsGAxYwqYK442Ll9WBL3GE1y1tCc4WYp00J5+Ezcp7ox lD+/yqJj/CwH23IqgUplBtvdNr+JPcU7+OmfZwEfcq5yBQ11ERjggGzhRjvH4SymHSJz ZnWVg8DMKLowgfJ8NMryaLK9CmyFz3M5IHCmD2qoSIKJiaeFRmW2XF+NMnt3IGKRZcOy 662gGWEX9bbKZ7VIkfzRbkmH2cTervT2ldfpmQBnYmGDzvZcGSenFFknxAuSTUPh8RiU hi6Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from sourceware.org (server2.sourceware.org. [2620:52:3:1:0:246e:9693:128c]) by mx.google.com with ESMTPS id c20-20020a170906171400b0093295ea18c2si2101526eje.392.2023.04.06.18.25.48 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Apr 2023 18:25:49 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) client-ip=2620:52:3:1:0:246e:9693:128c; Authentication-Results: mx.google.com; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 3A3CC385770A for ; Fri, 7 Apr 2023 01:25:44 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtpbgeu2.qq.com (smtpbgeu2.qq.com [18.194.254.142]) by sourceware.org (Postfix) with ESMTPS id 4CCCB3858D28 for ; Fri, 7 Apr 2023 01:25:11 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 4CCCB3858D28 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivai.ai Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivai.ai X-QQ-mid: bizesmtp79t1680830706tbd8e49v Received: from rios-cad5.localdomain ( [58.60.1.11]) by bizesmtp.qq.com (ESMTP) with id ; Fri, 07 Apr 2023 09:25:05 +0800 (CST) X-QQ-SSF: 01400000000000F0O000000A0000000 X-QQ-FEAT: QityeSR92A0s1OLT+mW+ugOcnH4SDD+lIpIOBQnt/2TBBkRHxL53Lv/ky7QTc ZGxiFWGuy95bkCIr9SO/JwchfVCjNaInAChHG2EeNgr3AG7DovfMpOs1FSmUCYEy2IpGmIa C0Rg7STImU0lIj3eQnFrhPZ/K/+agOh7gQGoo1lqM8+KQgs/XN2WNlwpl1RNcNiVSwi9hcB U3mh3QDC9HVK24Ul9uJdUJN8/UO+5956I+ti+06GCHe0FST6N1/7e2ULYCpdizSbv5q3R+R tD2BeqYp3paM1oXyFNIvGDHrXwGL+Mu+NPIZS7ROba/2/7GGjmss32PV7KgOHWOYo2AUwL5 qhG936CzZK6bbjNb0ztWrzcNt66t9XZBswpGqwX+wgo5SpjHbCUi8zCHTs8Ulc3froRyuaK X-QQ-GoodBg: 2 X-BIZMAIL-ID: 662578759089011493 From: juzhe.zhong@rivai.ai To: gcc-patches@gcc.gnu.org Cc: kito.cheng@gmail.com, palmer@dabbelt.com, jeffreyalaw@gmail.com, Juzhe-Zhong Subject: [PATCH] RISC-V: Enable basic RVV auto-vectorization support Date: Fri, 7 Apr 2023 09:25:03 +0800 Message-Id: <20230407012503.65215-1-juzhe.zhong@rivai.ai> X-Mailer: git-send-email 2.36.3 MIME-Version: 1.0 X-QQ-SENDSIZE: 520 Feedback-ID: bizesmtp:rivai.ai:qybglogicsvr:qybglogicsvr7 X-Spam-Status: No, score=-12.4 required=5.0 tests=BAYES_00, GIT_PATCH_0, KAM_ASCII_DIVIDERS, KAM_DMARC_STATUS, KAM_SHORT, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_PASS, TXREP, T_SPF_HELO_TEMPERROR autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org Sender: "Gcc-patches" X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762478783487930013?= X-GMAIL-MSGID: =?utf-8?q?1762478783487930013?= From: Juzhe-Zhong Enable basic auto-vectorization support of WHILE_LEN/LEN_LOAD/LEN_STORE. gcc/ChangeLog: * config/riscv/riscv-protos.h (preferred_simd_mode): New function. (expand_while_len): Ditto. * config/riscv/riscv-v.cc (autovec_use_vlmax_p): Ditto. (preferred_simd_mode): Ditto. (expand_while_len): Ditto. * config/riscv/riscv.cc (riscv_convert_vector_bits): Add basic auto-vectorization support. (riscv_preferred_simd_mode): New function. (TARGET_VECTORIZE_PREFERRED_SIMD_MODE): New targethook for RVV auto-vectorization support. * config/riscv/vector.md: Add basic autovec. * config/riscv/autovec.md: New file. --- gcc/config/riscv/autovec.md | 63 ++++++++++++++++++++++++++ gcc/config/riscv/riscv-protos.h | 2 + gcc/config/riscv/riscv-v.cc | 78 +++++++++++++++++++++++++++++++++ gcc/config/riscv/riscv.cc | 24 +++++++++- gcc/config/riscv/vector.md | 4 +- 5 files changed, 169 insertions(+), 2 deletions(-) create mode 100644 gcc/config/riscv/autovec.md diff --git a/gcc/config/riscv/autovec.md b/gcc/config/riscv/autovec.md new file mode 100644 index 00000000000..34561383041 --- /dev/null +++ b/gcc/config/riscv/autovec.md @@ -0,0 +1,63 @@ +;; Machine description for auto-vectorization using RVV for GNU compiler. +;; Copyright (C) 2023 Free Software Foundation, Inc. +;; Contributed by Juzhe Zhong (juzhe.zhong@rivai.ai), RiVAI Technologies Ltd. + +;; This file is part of GCC. + +;; GCC is free software; you can redistribute it and/or modify +;; it under the terms of the GNU General Public License as published by +;; the Free Software Foundation; either version 3, or (at your option) +;; any later version. + +;; GCC is distributed in the hope that it will be useful, +;; but WITHOUT ANY WARRANTY; without even the implied warranty of +;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;; GNU General Public License for more details. + +;; You should have received a copy of the GNU General Public License +;; along with GCC; see the file COPYING3. If not see +;; . + +;; ========================================================================= +;; == While_len +;; ========================================================================= + +(define_expand "while_len" + [(match_operand:P 0 "register_operand") + (match_operand:P 1 "vector_length_operand") + (match_operand:P 2 "")] + "TARGET_VECTOR" +{ + riscv_vector::expand_while_len (operands); + DONE; +}) + +;; ========================================================================= +;; == Loads/Stores +;; ========================================================================= + +;; len_load/len_store is a sub-optimal pattern for RVV auto-vectorization support. +;; We will replace them when len_maskload/len_maskstore is supported in loop vectorizer. +(define_expand "len_load_" + [(match_operand:V 0 "register_operand") + (match_operand:V 1 "memory_operand") + (match_operand 2 "vector_length_operand") + (match_operand 3 "const_0_operand")] + "TARGET_VECTOR" +{ + riscv_vector::emit_nonvlmax_op (code_for_pred_mov (mode), operands[0], + operands[1], operands[2], mode); + DONE; +}) + +(define_expand "len_store_" + [(match_operand:V 0 "memory_operand") + (match_operand:V 1 "register_operand") + (match_operand 2 "vector_length_operand") + (match_operand 3 "const_0_operand")] + "TARGET_VECTOR" +{ + riscv_vector::emit_nonvlmax_op (code_for_pred_mov (mode), operands[0], + operands[1], operands[2], mode); + DONE; +}) diff --git a/gcc/config/riscv/riscv-protos.h b/gcc/config/riscv/riscv-protos.h index 4611447ddde..6cd91987199 100644 --- a/gcc/config/riscv/riscv-protos.h +++ b/gcc/config/riscv/riscv-protos.h @@ -206,6 +206,8 @@ enum vlen_enum bool slide1_sew64_helper (int, machine_mode, machine_mode, machine_mode, rtx *); rtx gen_avl_for_scalar_move (rtx); +machine_mode preferred_simd_mode (scalar_mode); +void expand_while_len (rtx *); } /* We classify builtin types into two classes: diff --git a/gcc/config/riscv/riscv-v.cc b/gcc/config/riscv/riscv-v.cc index ed3c5e0756f..84d33fcdd14 100644 --- a/gcc/config/riscv/riscv-v.cc +++ b/gcc/config/riscv/riscv-v.cc @@ -43,6 +43,7 @@ #include "optabs.h" #include "tm-constrs.h" #include "rtx-vector-builder.h" +#include "targhooks.h" using namespace riscv_vector; @@ -729,4 +730,81 @@ gen_avl_for_scalar_move (rtx avl) } } +/* SCALABLE means that the vector-length is agnostic (run-time invariant and + compile-time unknown). FIXED meands that the vector-length is specific + (compile-time known). Both RVV_SCALABLE and RVV_FIXED_VLMAX are doing + auto-vectorization using VLMAX vsetvl configuration. */ +static bool +autovec_use_vlmax_p (void) +{ + return riscv_autovec_preference == RVV_SCALABLE + || riscv_autovec_preference == RVV_FIXED_VLMAX; +} + +/* Return the vectorization machine mode for RVV according to LMUL. */ +machine_mode +preferred_simd_mode (scalar_mode mode) +{ + /* We only enable auto-vectorization when TARGET_MIN_VLEN >= 128 + which is -march=rv64gcv. Since GCC loop vectorizer report ICE + when we enable -march=rv64gc_zve32* and -march=rv32gc_zve64*. + in the 'can_duplicate_and_interleave_p' of tree-vect-slp.cc. Since we have + VNx1SImode in -march=*zve32* and VNx1DImode in -march=*zve64*, they are + enabled in targetm. vector_mode_supported_p and SLP vectorizer will try to + use them. Currently, we can support auto-vectorization in + -march=rv32_zve32x_zvl128b. Wheras, -march=rv32_zve32x_zvl32b or + -march=rv32_zve32x_zvl64b are disabled. + */ + if (autovec_use_vlmax_p ()) + { + /* If TARGET_MIN_VLEN * riscv_autovec_lmul < 128, we don't allow + auto-vectorization since Loop Vectorizer may use VNx1SImode or + VNx1DImode to vectorize which will create ICE in the + 'can_duplicate_and_interleave_p' of tree-vect-slp.cc. */ + if (TARGET_MIN_VLEN * riscv_autovec_lmul < 128) + return word_mode; + /* We use LMUL = 1 as base bytesize which is BYTES_PER_RISCV_VECTOR and + riscv_autovec_lmul as multiply factor to calculate the the NUNITS to + get the auto-vectorization mode. */ + poly_uint64 nunits; + poly_uint64 vector_size + = BYTES_PER_RISCV_VECTOR * ((int) riscv_autovec_lmul); + poly_uint64 scalar_size = GET_MODE_SIZE (mode); + if (!multiple_p (vector_size, scalar_size, &nunits)) + return word_mode; + machine_mode rvv_mode; + if (get_vector_mode (mode, nunits).exists (&rvv_mode)) + return rvv_mode; + } + /* TODO: We will support minimum length VLS auto-vectorization in the future. + */ + return word_mode; +} + +/* Expand WHILE_LEN pattern. If we can find a mode for a corresponding + NUNITS, we emit vsetvl instructions directly. Otherwise, we emit + UMIN (operand1, NUNITS). */ +void +expand_while_len (rtx *ops) +{ + poly_int64 nunits; + gcc_assert (poly_int_rtx_p (ops[2], &nunits)); + /* We arbitrary picked QImode as inner scalar mode to get vector mode. + since vsetvl only demand ratio. We let VSETVL PASS to optimize it. */ + scalar_int_mode mode = QImode; + machine_mode rvv_mode; + if (get_vector_mode (mode, nunits).exists (&rvv_mode)) + { + rtx vsetvl_rtx + = gen_no_side_effects_vsetvl_rtx (rvv_mode, ops[0], ops[1]); + emit_insn (vsetvl_rtx); + } + else + { + rtx tmp = gen_reg_rtx (Pmode); + emit_move_insn (tmp, gen_int_mode (nunits, Pmode)); + expand_binop (Pmode, umin_optab, tmp, ops[1], ops[0], true, OPTAB_LIB); + } +} + } // namespace riscv_vector diff --git a/gcc/config/riscv/riscv.cc b/gcc/config/riscv/riscv.cc index b460c8a0b8b..9e507caabb4 100644 --- a/gcc/config/riscv/riscv.cc +++ b/gcc/config/riscv/riscv.cc @@ -6217,7 +6217,15 @@ riscv_convert_vector_bits (void) to set RVV mode size. The RVV machine modes size are run-time constant if TARGET_VECTOR is enabled. The RVV machine modes size remains default compile-time constant if TARGET_VECTOR is disabled. */ - return TARGET_VECTOR ? poly_uint16 (1, 1) : 1; + if (TARGET_VECTOR) + { + if (riscv_autovec_preference == RVV_FIXED_VLMAX) + return (int) TARGET_MIN_VLEN / (riscv_bytes_per_vector_chunk * 8); + else + return poly_uint16 (1, 1); + } + else + return 1; } /* Implement TARGET_OPTION_OVERRIDE. */ @@ -7076,6 +7084,17 @@ riscv_shamt_matches_mask_p (int shamt, HOST_WIDE_INT mask) return shamt == ctz_hwi (mask); } +/* Implement TARGET_VECTORIZE_PREFERRED_SIMD_MODE. */ + +static machine_mode +riscv_preferred_simd_mode (scalar_mode mode) +{ + if (TARGET_VECTOR) + return riscv_vector::preferred_simd_mode (mode); + + return word_mode; +} + /* Initialize the GCC target structure. */ #undef TARGET_ASM_ALIGNED_HI_OP #define TARGET_ASM_ALIGNED_HI_OP "\t.half\t" @@ -7327,6 +7346,9 @@ riscv_shamt_matches_mask_p (int shamt, HOST_WIDE_INT mask) #undef TARGET_DWARF_POLY_INDETERMINATE_VALUE #define TARGET_DWARF_POLY_INDETERMINATE_VALUE riscv_dwarf_poly_indeterminate_value +#undef TARGET_VECTORIZE_PREFERRED_SIMD_MODE +#define TARGET_VECTORIZE_PREFERRED_SIMD_MODE riscv_preferred_simd_mode + struct gcc_target targetm = TARGET_INITIALIZER; #include "gt-riscv.h" diff --git a/gcc/config/riscv/vector.md b/gcc/config/riscv/vector.md index 07624197a5a..9151a4c9891 100644 --- a/gcc/config/riscv/vector.md +++ b/gcc/config/riscv/vector.md @@ -23,7 +23,7 @@ ;; This file include : ;; ;; - Intrinsics (https://github.com/riscv/rvv-intrinsic-doc) -;; - Auto-vectorization (TBD) +;; - Auto-vectorization (autovec.md) ;; - Combine optimization (TBD) (include "vector-iterators.md") @@ -7688,3 +7688,5 @@ "vleff.v\t%0,%3%p1" [(set_attr "type" "vldff") (set_attr "mode" "")]) + +(include "autovec.md")