From patchwork Mon Sep 18 15:36:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lehua Ding X-Patchwork-Id: 141453 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp2750655vqi; Mon, 18 Sep 2023 08:37:12 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHl6ClH21cczhsVVoesnp7lNobGAk4CNN0E7YgPNKXwe3QR2DPzPFVXYO24YhYGAyHjv6pB X-Received: by 2002:a17:907:d13:b0:9ad:8641:e91b with SMTP id gn19-20020a1709070d1300b009ad8641e91bmr32929ejc.11.1695051431994; Mon, 18 Sep 2023 08:37:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695051431; cv=none; d=google.com; s=arc-20160816; b=A7FJNrhDR7FTieXjPceddujo1uZkztk/N2GAxk8sJqt80X3I+o0PJC2pGOVHFFxWK2 W8cossHDj6w8Pk/6C3s7wL4IhFBuDgtcbsQA3FHG85jxEPJgzdkklKV2ZJYOuIVXSi6N dD0EgruBwac26Cm1qxgyMm5LqeOuBwsRb3qQthRBHn6msYxJs8dR9DORTOn/arOVh64h sZW0eW9P21+UqRiILd3fggnh+6MZ2QVXP8xvMi1YyEgk+fw+zbEpxzjwZR/d8vAeZuOG rg6vprMoZVxI9AKqstqSzcfjwsvO5DJkOe3u1sGcjk7DxUaLGf7U5W2U2gvmow4wH/uH WniQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:feedback-id :content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:dmarc-filter:delivered-to; bh=ZmrR2se9hAftPnGpt9sDPNj2b2AJQSQBUAriDc7P+r4=; fh=x/Q0OlwHuvCZ3FpkiZPiUSvevOYVxUAi4aNnf76mUPQ=; b=VsDBMv4C4uI1pUF3YiNgqVbL1Zas5GrccZ4Ub1MXe6ysqmwRXCTGbyvUKMSfhwlAZh 33TNcdE3Cqd3InEtjJWz+Q872Hi43Vwx1aa2Ryzk0Y2D0X3S887OqbGecGH8LRqAE5cb wu6S0e1wc5a0+rhLaWILvxjZ6NG0ciQJz6lVB79ojuN35fc+QVZKJJlJIo1RADe6DsxF qkyT+v0il0bT2QrSfr2zFhaLNh04huz6OcCsHxbJM4f8dtsO7qmp0b/QMm/QLmNu/dpl FGgTd1oJulxwbQcxD16BBzBXYlMsfzUytEovI6V06MQ1IDR2KGx6aCDF0tJewQ9gM6Vl VY2w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (server2.sourceware.org. [2620:52:3:1:0:246e:9693:128c]) by mx.google.com with ESMTPS id v13-20020a17090606cd00b00992f8116abdsi8460808ejb.480.2023.09.18.08.37.11 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Sep 2023 08:37:11 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) client-ip=2620:52:3:1:0:246e:9693:128c; Authentication-Results: mx.google.com; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 17EED385558A for ; Mon, 18 Sep 2023 15:37:05 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtpbg153.qq.com (smtpbg153.qq.com [13.245.218.24]) by sourceware.org (Postfix) with ESMTPS id 806913858D32 for ; Mon, 18 Sep 2023 15:36:32 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 806913858D32 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivai.ai Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivai.ai X-QQ-mid: bizesmtp71t1695051383taifm5bk Received: from rios-cad121.hadoop.rioslab.org ( [58.60.1.9]) by bizesmtp.qq.com (ESMTP) with id ; Mon, 18 Sep 2023 23:36:22 +0800 (CST) X-QQ-SSF: 01400000000000C0F000000A0000000 X-QQ-FEAT: LE7C6P2vL8TGR8U81VP1m47WqIyIKJ6UG1bbKS0kTaL6km+NWCUyt38XjApy0 SEYKd4dmJy1Z/9SXqCz+BSe0SKGxvW5wYOunWXnUYQnQdhS66abV6nlchH9qSwHgC1QKzQE HhMyeuPh8pwwxsCEQAmovleItuqKVHADURusshUnXrBxH219Oyrw7hYeQrGyQUefRrkXQZC otKEb8ezGUXX1oowAW5c2cRJtnoO+TjehdCO4s/ZuSvcMhGtp1AgTmRZCVPUYf1ojdisFhI /7flOJ2K1ESEzt7JOCNac3pAkcCalRtoIHIPX9e2624wmyN+TihDt9iZPVTWgP9zg8/M+04 ZAWQfsskNvMdypQ/dg5vdk9h6ZyW2mePel2KXBPpIv0GJl5Z2Ha/pnha9Ir8dX2hwDuVxMT krNg2zb+U9Y= X-QQ-GoodBg: 2 X-BIZMAIL-ID: 15403236746069738457 From: Lehua Ding To: gcc-patches@gcc.gnu.org Cc: juzhe.zhong@rivai.ai, kito.cheng@gmail.com, rdapp.gcc@gmail.com, palmer@rivosinc.com, jeffreyalaw@gmail.com, lehua.ding@rivai.ai Subject: [PATCH] RISC-V: Support combine cond extend and reduce sum to cond widen reduce sum Date: Mon, 18 Sep 2023 23:36:22 +0800 Message-Id: <20230918153622.1584614-1-lehua.ding@rivai.ai> X-Mailer: git-send-email 2.36.3 MIME-Version: 1.0 X-QQ-SENDSIZE: 520 Feedback-ID: bizesmtp:rivai.ai:qybglogicsvrgz:qybglogicsvrgz6a-0 X-Spam-Status: No, score=-11.1 required=5.0 tests=BAYES_00, GIT_PATCH_0, KAM_ASCII_DIVIDERS, KAM_DMARC_STATUS, KAM_SHORT, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org Sender: "Gcc-patches" X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777390250429762756 X-GMAIL-MSGID: 1777390250429762756 This patch support combining cond extend and reduce_sum to cond widen reduce_sum like combine the following three insns: (set (reg:RVVM2HI 149) (const_vector:RVVM2HI repeat [ (const_int 0) ])) (set (reg:RVVM2HI 138) (if_then_else:RVVM2HI (reg:RVVMF8BI 135) (reg:RVVM2HI 148) (reg:RVVM2HI 149))) (set (reg:HI 150) (unspec:HI [ (reg:RVVM2HI 138) ] UNSPEC_REDUC_SUM)) into one insn: (set (reg:SI 147) (unspec:SI [ (if_then_else:RVVM2SI (reg:RVVMF16BI 135) (sign_extend:RVVM2SI (reg:RVVM1HI 136)) (const_vector:RVVM2SI repeat [ (const_int 0) ])) ] UNSPEC_REDUC_SUM)) Consider the following C code: int16_t foo (int8_t *restrict a, int8_t *restrict pred) { int16_t sum = 0; for (int i = 0; i < 16; i += 1) if (pred[i]) sum += a[i]; return sum; } assembly before this patch: foo: vsetivli zero,16,e16,m2,ta,ma li a5,0 vmv.v.i v2,0 vsetvli zero,zero,e8,m1,ta,ma vl1re8.v v0,0(a1) vmsne.vi v0,v0,0 vsetvli zero,zero,e16,m2,ta,mu vle8.v v4,0(a0),v0.t vmv.s.x v1,a5 vsext.vf2 v2,v4,v0.t vredsum.vs v2,v2,v1 vmv.x.s a0,v2 slliw a0,a0,16 sraiw a0,a0,16 ret assembly after this patch: foo: li a5,0 vsetivli zero,16,e16,m1,ta,ma vmv.s.x v3,a5 vsetivli zero,16,e8,m1,ta,ma vl1re8.v v0,0(a1) vmsne.vi v0,v0,0 vle8.v v2,0(a0),v0.t vwredsum.vs v1,v2,v3,v0.t vsetivli zero,0,e16,m1,ta,ma vmv.x.s a0,v1 slliw a0,a0,16 sraiw a0,a0,16 ret gcc/ChangeLog: * config/riscv/autovec-opt.md (*cond_widen_reduc_plus_scal_): New combine patterns. * config/riscv/autovec.md (vcond_mask_): Split vcond_mask pattern into three patterns. (vec_duplicate_const_0): Ditto. (*vcond_mask_): Ditto. * config/riscv/predicates.md (vector_register_or_const_0_operand): New. * config/riscv/riscv-protos.h (enum insn_type): Add REDUCE_OP_M. gcc/testsuite/ChangeLog: * gcc.target/riscv/rvv/autovec/cond/cond_widen_reduc-1.c: New test. * gcc.target/riscv/rvv/autovec/cond/cond_widen_reduc_run-1.c: New test. --- gcc/config/riscv/autovec-opt.md | 48 +++++++++++++++ gcc/config/riscv/autovec.md | 59 ++++++++++++++++++- gcc/config/riscv/predicates.md | 5 ++ gcc/config/riscv/riscv-protos.h | 1 + .../rvv/autovec/cond/cond_widen_reduc-1.c | 30 ++++++++++ .../rvv/autovec/cond/cond_widen_reduc_run-1.c | 28 +++++++++ 6 files changed, 170 insertions(+), 1 deletion(-) create mode 100644 gcc/testsuite/gcc.target/riscv/rvv/autovec/cond/cond_widen_reduc-1.c create mode 100644 gcc/testsuite/gcc.target/riscv/rvv/autovec/cond/cond_widen_reduc_run-1.c -- 2.36.3 diff --git a/gcc/config/riscv/autovec-opt.md b/gcc/config/riscv/autovec-opt.md index b47bae16193..eefa4f28a0a 100644 --- a/gcc/config/riscv/autovec-opt.md +++ b/gcc/config/riscv/autovec-opt.md @@ -1284,6 +1284,54 @@ } [(set_attr "type" "vector")]) +;; Combine mask extend + vredsum to mask vwredsum[u] +(define_insn_and_split "*cond_widen_reduc_plus_scal_" + [(set (match_operand: 0 "register_operand") + (unspec: [ + (if_then_else: + (match_operand: 1 "register_operand") + (any_extend: + (match_operand:VI_QHS_NO_M8 2 "register_operand")) + (match_operand: 3 "vector_const_0_operand")) + ] UNSPEC_REDUC_SUM))] + "TARGET_VECTOR && can_create_pseudo_p ()" + "#" + "&& 1" + [(const_int 0)] +{ + rtx ops[] = {operands[0], operands[2], operands[1], + gen_int_mode (GET_MODE_NUNITS (mode), Pmode)}; + riscv_vector::expand_reduction (, + riscv_vector::REDUCE_OP_M, + ops, CONST0_RTX (mode)); + DONE; +} +[(set_attr "type" "vector")]) + +;; Combine mask extend + vfredsum to mask vfwredusum +(define_insn_and_split "*cond_widen_reduc_plus_scal_" + [(set (match_operand: 0 "register_operand") + (unspec: [ + (if_then_else: + (match_operand: 1 "register_operand") + (float_extend: + (match_operand:VF_HS_NO_M8 2 "register_operand")) + (match_operand: 3 "vector_const_0_operand")) + ] UNSPEC_REDUC_SUM_UNORDERED))] + "TARGET_VECTOR && can_create_pseudo_p ()" + "#" + "&& 1" + [(const_int 0)] +{ + rtx ops[] = {operands[0], operands[2], operands[1], + gen_int_mode (GET_MODE_NUNITS (mode), Pmode)}; + riscv_vector::expand_reduction (UNSPEC_WREDUC_SUM_UNORDERED, + riscv_vector::REDUCE_OP_M_FRM_DYN, + ops, CONST0_RTX (mode)); + DONE; +} +[(set_attr "type" "vector")]) + ;; ============================================================================= ;; Misc combine patterns ;; ============================================================================= diff --git a/gcc/config/riscv/autovec.md b/gcc/config/riscv/autovec.md index 493d5745485..20a71ad8ced 100644 --- a/gcc/config/riscv/autovec.md +++ b/gcc/config/riscv/autovec.md @@ -545,7 +545,64 @@ ;; - vfmerge.vf ;; ------------------------------------------------------------------------- -(define_insn_and_split "vcond_mask_" +;; The purpose of splitting the original pattern into three patterns here is +;; to combine the following three insns: +;; (set (reg:RVVM2HI 149) +;; (const_vector:RVVM2HI repeat [ +;; (const_int 0) +;; ])) +;; (set (reg:RVVM2HI 138) +;; (if_then_else:RVVM2HI +;; (reg:RVVMF8BI 135) +;; (reg:RVVM2HI 148) +;; (reg:RVVM2HI 149))) +;; (set (reg:HI 150) +;; (unspec:HI [ +;; (reg:RVVM2HI 138) +;; ] UNSPEC_REDUC_SUM)) +;; +;; into one insn: +;; +;; (set (reg:SI 147) +;; (unspec:SI [ +;; (if_then_else:RVVM2SI +;; (reg:RVVMF16BI 135) +;; (sign_extend:RVVM2SI (reg:RVVM1HI 136)) +;; (const_vector:RVVM2SI repeat [ +;; (const_int 0) +;; ])) +;; ] UNSPEC_REDUC_SUM)) + +(define_expand "vcond_mask_" + [(set (match_operand:V_VLS 0 "register_operand") + (if_then_else:V_VLS + (match_operand: 3 "register_operand") + (match_operand:V_VLS 1 "nonmemory_operand") + (match_operand:V_VLS 2 "vector_register_or_const_0_operand")))] + "TARGET_VECTOR" + { + if (satisfies_constraint_Wc0 (operands[2])) + { + rtx reg = gen_reg_rtx (mode); + emit_insn (gen_vec_duplicate_const_0 (reg, operands[2])); + operands[2] = reg; + } + }) + +(define_insn_and_split "vec_duplicate_const_0" + [(set (match_operand:V_VLS 0 "register_operand") + (match_operand:V_VLS 1 "vector_const_0_operand"))] + "TARGET_VECTOR && can_create_pseudo_p ()" + "#" + "&& 1" + [(const_int 0)] +{ + emit_move_insn (operands[0], operands[1]); + DONE; +} + [(set_attr "type" "vector")]) + +(define_insn_and_split "*vcond_mask_" [(set (match_operand:V_VLS 0 "register_operand") (if_then_else:V_VLS (match_operand: 3 "register_operand") diff --git a/gcc/config/riscv/predicates.md b/gcc/config/riscv/predicates.md index 4bc7ff2c9d8..6abf9d97958 100644 --- a/gcc/config/riscv/predicates.md +++ b/gcc/config/riscv/predicates.md @@ -463,6 +463,11 @@ (ior (match_operand 0 "register_operand") (match_code "const_vector"))) +(define_predicate "vector_register_or_const_0_operand" + (ior (match_operand 0 "register_operand") + (and (match_code "const_vector") + (match_test "satisfies_constraint_Wc0 (op)")))) + (define_predicate "vector_gs_scale_operand_16" (and (match_code "const_int") (match_test "INTVAL (op) == 1 || INTVAL (op) == 2"))) diff --git a/gcc/config/riscv/riscv-protos.h b/gcc/config/riscv/riscv-protos.h index 5a2d218d67b..fd6107ccb5c 100644 --- a/gcc/config/riscv/riscv-protos.h +++ b/gcc/config/riscv/riscv-protos.h @@ -337,6 +337,7 @@ enum insn_type : unsigned int /* For vreduce, no mask policy operand. */ REDUCE_OP = __NORMAL_OP_TA | BINARY_OP_P | VTYPE_MODE_FROM_OP1_P, + REDUCE_OP_M = __MASK_OP_TA | BINARY_OP_P | VTYPE_MODE_FROM_OP1_P, REDUCE_OP_FRM_DYN = REDUCE_OP | FRM_DYN_P | VTYPE_MODE_FROM_OP1_P, REDUCE_OP_M_FRM_DYN = __MASK_OP_TA | BINARY_OP_P | FRM_DYN_P | VTYPE_MODE_FROM_OP1_P, diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/cond/cond_widen_reduc-1.c b/gcc/testsuite/gcc.target/riscv/rvv/autovec/cond/cond_widen_reduc-1.c new file mode 100644 index 00000000000..22a71048684 --- /dev/null +++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/cond/cond_widen_reduc-1.c @@ -0,0 +1,30 @@ +/* { dg-do compile } */ +/* { dg-additional-options "-march=rv64gcv_zvfh_zvl128b -mabi=lp64d --param riscv-autovec-preference=fixed-vlmax --param riscv-autovec-lmul=m2 -fno-vect-cost-model -ffast-math" } */ +#include + +#define TEST_TYPE(TYPE1, TYPE2, N) \ + __attribute__ ((noipa)) \ + TYPE1 reduc_##TYPE1##_##TYPE2 (TYPE2 *restrict a, TYPE2 *restrict pred) \ + { \ + TYPE1 sum = 0; \ + for (int i = 0; i < N; i += 1) \ + if (pred[i]) \ + sum += a[i]; \ + return sum; \ + } + +#define TEST_ALL(TEST) \ + TEST (int16_t, int8_t, 16) \ + TEST (int32_t, int16_t, 8) \ + TEST (int64_t, int32_t, 4) \ + TEST (uint16_t, uint8_t, 16) \ + TEST (uint32_t, uint16_t, 8) \ + TEST (uint64_t, uint32_t, 4) \ + TEST (float, _Float16, 8) \ + TEST (double, float, 4) + +TEST_ALL (TEST_TYPE) + +/* { dg-final { scan-assembler-times {\tvfwredusum\.vs\tv[0-9]+,v[0-9]+,v[0-9]+,v0\.t} 2 } } */ +/* { dg-final { scan-assembler-times {\tvwredsum\.vs\tv[0-9]+,v[0-9]+,v[0-9]+,v0\.t} 3 } } */ +/* { dg-final { scan-assembler-times {\tvwredsumu\.vs\tv[0-9]+,v[0-9]+,v[0-9]+,v0\.t} 3 } } */ diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/cond/cond_widen_reduc_run-1.c b/gcc/testsuite/gcc.target/riscv/rvv/autovec/cond/cond_widen_reduc_run-1.c new file mode 100644 index 00000000000..fdb7e5249ee --- /dev/null +++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/cond/cond_widen_reduc_run-1.c @@ -0,0 +1,28 @@ +/* { dg-do run { target { riscv_vector } } } */ +/* { dg-additional-options "--param=riscv-autovec-preference=scalable -fno-vect-cost-model" } */ + +#include "cond_widen_reduc-1.c" + +#define RUN(TYPE1, TYPE2, N) \ + { \ + TYPE2 a[N]; \ + TYPE2 pred[N]; \ + TYPE1 r = 0; \ + for (int i = 0; i < N; i++) \ + { \ + a[i] = (i * 0.1) * (i & 1 ? 1 : -1); \ + pred[i] = i % 3; \ + if (pred[i]) \ + r += a[i]; \ + asm volatile ("" ::: "memory"); \ + } \ + if (r != reduc_##TYPE1##_##TYPE2 (a, pred)) \ + __builtin_abort (); \ + } + +int __attribute__ ((optimize (1))) +main () +{ + TEST_ALL (RUN) + return 0; +}