From patchwork Tue Oct 18 09:17:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Jiang, Haochen" X-Patchwork-Id: 4046 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp1855222wrs; Tue, 18 Oct 2022 02:18:54 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6yc5pNoPxOofMKsbsEQw7fDivGhheSIls1kBLf6q3QLPrxj7fygRikZhEQsoAHho0ET3VM X-Received: by 2002:a05:6402:3546:b0:45d:a52f:2d38 with SMTP id f6-20020a056402354600b0045da52f2d38mr1653222edd.403.1666084734203; Tue, 18 Oct 2022 02:18:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666084734; cv=none; d=google.com; s=arc-20160816; b=P4OnPdkebqxmbdNhj3VhNRlXOjrHe1Jk5kWCF2CQCQBkVdcn4+FJWOXDqyo0K10f/t vIs3qa9+9UQhbDHeipzmy9/vsyR/2n4bpYxHjGcJpv1CLPkSxyaIvc+Ddk+PyiOIphLb 99JZZac+GQfs2c+ECHLSsfnIBSQKAI62NVL/La3bKpIjBOHaCNiNwGnWAQw1sbauu2FL cV9z2aTL+Cfv7FrQ1PILpeIZuEquMUR3dJsTdtGeWzNF5eICe17napOOSuR08iGJNAoO MEXveq7PEyOCOsOiB2kcnsQv3c+/a1RxemaMOvsdlDOGGPcWYWIb8WgaGbZFCc7NIV6q kywQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:reply-to:from:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence :references:in-reply-to:message-id:date:subject:to:dmarc-filter :delivered-to:dkim-signature:dkim-filter; bh=3qT0XCCBgangvAkjBBZZGva2mNZDekXvmcmmAFcu9w8=; b=H0eRLByJNXEH8SQ4L8082Tkt1c1cUsV/7pmGqPHc4mq0aPS4Kqd+NrSnea1EBUVyfe OrZ7PKjJB1aWODRCvlMZOmInzJX8X2qyHhC5HRGIDj4/Gl7BDMoazXxXZCiOssLPCJwA j4CFR2xsXOQgycMrqDsdIDYpT2bgTSSKQWfLaMzbMpbLsH42evXE98GetaqhzFAnm+Us Ub7aI6q7s/YJ8qeF7Zd5ReLX8w0dFSWI1avlKMOTJkd9vZgymGSp0wqAY4DOtAk2s3tl rrrnmCdZXZVbSNbCT7PqFAHaZsMiCETCA8WZ8CI/RFqAzcFdIAHdxIbyecHxrrWs0BWV q/MA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b="G/5L9VVi"; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gnu.org Received: from sourceware.org (ip-8-43-85-97.sourceware.org. [8.43.85.97]) by mx.google.com with ESMTPS id gn28-20020a1709070d1c00b00732fa13e848si12575643ejc.597.2022.10.18.02.18.54 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Oct 2022 02:18:54 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) client-ip=8.43.85.97; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b="G/5L9VVi"; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gnu.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id A30D33858291 for ; Tue, 18 Oct 2022 09:18:45 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org A30D33858291 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1666084725; bh=3qT0XCCBgangvAkjBBZZGva2mNZDekXvmcmmAFcu9w8=; h=To:Subject:Date:In-Reply-To:References:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=G/5L9VViQqICJEMmdntUcA9MlK4v9L2TVtFXYenAb8DoZNe4W2HqSOqqCS1JzJHgg HkIudE0oJGGSEvk2wSvQcWzGevKSwuL3/TIqmE4mg1uFUf9gVSMjKKG/uIQLijm8v7 uS0TyZkjtcjCowjQTNCfcdW7UKkIbCrUlC0NsflA= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by sourceware.org (Postfix) with ESMTPS id 1CB993858D3C for ; Tue, 18 Oct 2022 09:18:01 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 1CB993858D3C X-IronPort-AV: E=McAfee;i="6500,9779,10503"; a="332590451" X-IronPort-AV: E=Sophos;i="5.95,193,1661842800"; d="scan'208";a="332590451" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Oct 2022 02:17:59 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10503"; a="691714917" X-IronPort-AV: E=Sophos;i="5.95,193,1661842800"; d="scan'208";a="691714917" Received: from shvmail03.sh.intel.com ([10.239.245.20]) by fmsmga008.fm.intel.com with ESMTP; 18 Oct 2022 02:17:29 -0700 Received: from shliclel320.sh.intel.com (shliclel320.sh.intel.com [10.239.240.127]) by shvmail03.sh.intel.com (Postfix) with ESMTP id CED261009810; Tue, 18 Oct 2022 17:17:27 +0800 (CST) To: gcc-patches@gcc.gnu.org Subject: [PATCH] i386: Auto vectorize sdot_prod, udot_prod with VNNIINT8 instruction. Date: Tue, 18 Oct 2022 17:17:27 +0800 Message-Id: <20221018091727.82856-1-haochen.jiang@intel.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20221014075445.7938-1-haochen.jiang@intel.com> References: <20221014075445.7938-1-haochen.jiang@intel.com> X-Spam-Status: No, score=-11.8 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_SHORT, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Haochen Jiang via Gcc-patches From: "Jiang, Haochen" Reply-To: Haochen Jiang Cc: hongtao.liu@intel.com Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org Sender: "Gcc-patches" X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747016466411721601?= X-GMAIL-MSGID: =?utf-8?q?1747016466411721601?= Hi all, We would like to add one more patch to enhance the codegen with avxvnniint8. Also renamed two awkward named mode_attr to make them more aligned with others. Regtested on x86_64-pc-linux-gnu. Ok for trunk? BRs, Haochen gcc/ChangeLog: * config/i386/sse.md (ssedvecmode): Rename from VI1SI. (ssedvecmodelower): Rename from vi1si. (sdot_prod): New define_expand. (udot_prod): Ditto. gcc/testsuite/ChangeLog: * gcc.target/i386/vnniint8-auto-vectorize-1.c: New test. * gcc.target/i386/vnniint8-auto-vectorize-2.c: Ditto. --- gcc/config/i386/sse.md | 61 ++++++++++++--- .../i386/vnniint8-auto-vectorize-1.c | 28 +++++++ .../i386/vnniint8-auto-vectorize-2.c | 75 +++++++++++++++++++ 3 files changed, 153 insertions(+), 11 deletions(-) create mode 100644 gcc/testsuite/gcc.target/i386/vnniint8-auto-vectorize-1.c create mode 100644 gcc/testsuite/gcc.target/i386/vnniint8-auto-vectorize-2.c diff --git a/gcc/config/i386/sse.md b/gcc/config/i386/sse.md index 29cf6fa090b..fc17b5193dc 100644 --- a/gcc/config/i386/sse.md +++ b/gcc/config/i386/sse.md @@ -1043,6 +1043,13 @@ (V16HI "v16hi") (V8HI "v8hi") (V32QI "v32qi") (V16QI "v16qi")]) +;; Mapping of vector modes to an V*SImode of the same size +(define_mode_attr ssedvecmode + [(V64QI "V16SI") (V32QI "V8SI") (V16QI "V4SI")]) + +(define_mode_attr ssedvecmodelower + [(V64QI "v16si") (V32QI "v8si") (V16QI "v4si")]) + ;; Mapping of vector modes to a vector mode of double size (define_mode_attr ssedoublevecmode [(V64QI "V128QI") (V32HI "V64HI") (V16SI "V32SI") (V8DI "V16DI") @@ -28523,29 +28530,23 @@ [(set_attr ("prefix") ("evex")) (set_attr "mode" "")]) -(define_mode_attr VI1SI - [(V64QI "V16SI") (V32QI "V8SI") (V16QI "V4SI")]) - -(define_mode_attr vi1si - [(V64QI "v16si") (V32QI "v8si") (V16QI "v4si")]) - (define_expand "usdot_prod" - [(match_operand: 0 "register_operand") + [(match_operand: 0 "register_operand") (match_operand:VI1_AVX512VNNI 1 "register_operand") (match_operand:VI1_AVX512VNNI 2 "register_operand") - (match_operand: 3 "register_operand")] + (match_operand: 3 "register_operand")] "( == 64 ||((TARGET_AVX512VNNI && TARGET_AVX512VL) || TARGET_AVXVNNI))" { - operands[1] = lowpart_subreg (mode, + operands[1] = lowpart_subreg (mode, force_reg (mode, operands[1]), mode); - operands[2] = lowpart_subreg (mode, + operands[2] = lowpart_subreg (mode, force_reg (mode, operands[2]), mode); emit_insn (gen_rtx_SET (operands[0], operands[3])); - emit_insn (gen_vpdpbusd_ (operands[0], operands[3], + emit_insn (gen_vpdpbusd_ (operands[0], operands[3], operands[1], operands[2])); DONE; }) @@ -29358,6 +29359,44 @@ (UNSPEC_VPDPBSUD "bsud") (UNSPEC_VPDPBSUDS "bsuds") (UNSPEC_VPDPBUUD "buud") (UNSPEC_VPDPBUUDS "buuds")]) +(define_expand "sdot_prod" + [(match_operand: 0 "register_operand") + (match_operand:VI1 1 "register_operand") + (match_operand:VI1 2 "register_operand") + (match_operand: 3 "register_operand")] + "TARGET_AVXVNNIINT8" +{ + operands[1] = lowpart_subreg (mode, + force_reg (mode, operands[1]), + mode); + operands[2] = lowpart_subreg (mode, + force_reg (mode, operands[2]), + mode); + emit_insn (gen_rtx_SET (operands[0], operands[3])); + emit_insn (gen_vpdpbssd_ (operands[0], operands[3], + operands[1], operands[2])); + DONE; +}) + +(define_expand "udot_prod" + [(match_operand: 0 "register_operand") + (match_operand:VI1 1 "register_operand") + (match_operand:VI1 2 "register_operand") + (match_operand: 3 "register_operand")] + "TARGET_AVXVNNIINT8" +{ + operands[1] = lowpart_subreg (mode, + force_reg (mode, operands[1]), + mode); + operands[2] = lowpart_subreg (mode, + force_reg (mode, operands[2]), + mode); + emit_insn (gen_rtx_SET (operands[0], operands[3])); + emit_insn (gen_vpdpbuud_ (operands[0], operands[3], + operands[1], operands[2])); + DONE; +}) + (define_insn "vpdp_" [(set (match_operand:VI4_AVX 0 "register_operand" "=x") (unspec:VI4_AVX diff --git a/gcc/testsuite/gcc.target/i386/vnniint8-auto-vectorize-1.c b/gcc/testsuite/gcc.target/i386/vnniint8-auto-vectorize-1.c new file mode 100644 index 00000000000..9cadab6a845 --- /dev/null +++ b/gcc/testsuite/gcc.target/i386/vnniint8-auto-vectorize-1.c @@ -0,0 +1,28 @@ +/* { dg-do compile } */ +/* { dg-options "-mavxvnniint8 -O2" } */ +/* { dg-final { scan-assembler "vpdpbssd\t" } } */ +/* { dg-final { scan-assembler "vpdpbuud\t" } } */ + +int __attribute__((noinline, noclone, optimize("tree-vectorize"))) +sdot_prod_qi (char * restrict a, char * restrict b, + int c, int n) +{ + int i; + for (i = 0; i < n; i++) + { + c += ((int) a[i] * (int) b[i]); + } + return c; +} + +int __attribute__((noinline, noclone, optimize("tree-vectorize"))) +udot_prod_qi (unsigned char * restrict a, unsigned char *restrict b, + int c, int n) +{ + int i; + for (i = 0; i < n; i++) + { + c += ((int) a[i] * (int) b[i]); + } + return c; +} diff --git a/gcc/testsuite/gcc.target/i386/vnniint8-auto-vectorize-2.c b/gcc/testsuite/gcc.target/i386/vnniint8-auto-vectorize-2.c new file mode 100644 index 00000000000..99853e6c3b7 --- /dev/null +++ b/gcc/testsuite/gcc.target/i386/vnniint8-auto-vectorize-2.c @@ -0,0 +1,75 @@ +/* { dg-do run } */ +/* { dg-options "-O2 -mavxvnniint8" } */ +/* { dg-require-effective-target avxvnniint8 } */ + +#define AVXVNNIINT8 +#ifndef CHECK +#define CHECK "avx-check.h" +#endif + +#ifndef TEST +#define TEST avx_test +#endif + +#include CHECK +#include "vnniint8-auto-vectorize-1.c" + +#define N 256 +char a_i8[N], b_i8[N]; +unsigned char c_u8[N], d_u8[N]; +int i8_exp, i8_ref; + +int __attribute__((noipa, optimize("no-tree-vectorize"))) +sdot_prod_qi_scalar (char * restrict a, char * restrict b, + int c, int n) +{ + int i; + for (i = 0; i < n; i++) + { + c += ((int) a[i] * (int) b[i]); + } + return c; +} + +int __attribute__((noipa, optimize("no-tree-vectorize"))) +udot_prod_qi_scalar (unsigned char * restrict a, unsigned char *restrict b, + int c, int n) +{ + int i; + for (i = 0; i < n; i++) + { + c += ((int) a[i] * (int) b[i]); + } + return c; +} + +void init () +{ + int i; + + i8_exp = i8_ref = 127; + + for (i = 0; i < N; i++) + { + a_i8[i] = (-i + 4) % 128; + b_i8[i] = (i + 1) % 128; + c_u8[i] = (i + 3) % 256; + d_u8[i] = (i + 5) % 256; + } +} + +void +TEST (void) +{ + init (); + i8_exp = sdot_prod_qi (a_i8, b_i8, i8_exp, N); + i8_ref = sdot_prod_qi_scalar (a_i8, b_i8, i8_ref, N); + if (i8_exp != i8_ref) + abort (); + + init (); + i8_exp = udot_prod_qi (c_u8, d_u8, i8_exp, N); + i8_ref = udot_prod_qi_scalar (c_u8, d_u8, i8_ref, N); + if (i8_exp != i8_ref) + abort (); +}