From patchwork Mon Jul 3 11:09:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "juzhe.zhong@rivai.ai" X-Patchwork-Id: 115313 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f45:0:b0:3ea:f831:8777 with SMTP id v5csp438272vqx; Mon, 3 Jul 2023 04:09:54 -0700 (PDT) X-Google-Smtp-Source: APBJJlFDpcezd2yYWw0zYdpTzR8JMPJsZ+x7QSjP/RYNcUsslweUzf9qUNiA+jDCYKym6rdbvOCp X-Received: by 2002:a17:907:100c:b0:992:e76:9c18 with SMTP id ox12-20020a170907100c00b009920e769c18mr6726878ejb.37.1688382593874; Mon, 03 Jul 2023 04:09:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1688382593; cv=none; d=google.com; s=arc-20160816; b=LVdjzCsjPUktkIufugmcJBakklWMItlG2hsLRv91m/dzL7BKLutdHDnCqNL6SaTjlf 3bEI9i6iyHY8Zp20btq3wHERl4qev6VgrLycLDIAyVR/2TUiHmv+ll8yz8fVzwLeDtGh ghiRAK0Sf7PdHiAb8PEZCD6fVznig50DBIDT6x4wh21wBqgLrRHJQKrPhx3V4uyQyuxg cmR3KSt59L2iZjmyx+PvQ5vXd65ReaCHvySlKNfmo9aSnZpMZwduEi8sy7pvzaJioCXa jSlVMA3Wa5aZyuto9oaJl3bNoT4IW+AiJ+39S7Ohtq5rIt+XAwGZsVLPH/5jiQPklq2j 0FgA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:feedback-id :content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:dmarc-filter:delivered-to; bh=rRJvbqKb8L2gZV0BEQCqYk3ZgxpuFXSmBJ0lrtKdtaA=; fh=h6UG6PIw2FD/e8JL+9n/VDDoO75bXx4B3jGj5okR2Rw=; b=Sb3WzwEpFLnLQCHoapGl6dCGr8gHmhAqfebmEc56c2uqGl6Q2iyubiGVx6h49ZjHEO m2nvLu7/rLZd11Fr1vNqMLd8BA2oTAU1upAKyds+wibBwH/FScdq8nqAO1vR7t2q7E9V P39ru41juXmK67/igaBxOq7IUXstbs+sTcvJbu98ZqUPkQ9y9DPm0be1i3mwtn1g4stK 2eYXeX2NWogRqSF6uMRyjL2ZnJGmZ49j/Uy0craXql8L9aXBfVBofPUhtrT8G1loatZN nie6KDrpR2N5tw74g9XlqRELPiZxTCBpM02kJLHAkZdITy327aAD41xssphKsFZylvh9 SMtg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (server2.sourceware.org. [2620:52:3:1:0:246e:9693:128c]) by mx.google.com with ESMTPS id ss26-20020a170907039a00b0098e422d675csi9838464ejb.311.2023.07.03.04.09.53 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Jul 2023 04:09:53 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) client-ip=2620:52:3:1:0:246e:9693:128c; Authentication-Results: mx.google.com; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 1A10F3857C55 for ; Mon, 3 Jul 2023 11:09:49 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtpbguseast2.qq.com (smtpbguseast2.qq.com [54.204.34.130]) by sourceware.org (Postfix) with ESMTPS id 67DB93858D32 for ; Mon, 3 Jul 2023 11:09:22 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 67DB93858D32 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivai.ai Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivai.ai X-QQ-mid: bizesmtp66t1688382557tz76r969 Received: from server1.localdomain ( [58.60.1.22]) by bizesmtp.qq.com (ESMTP) with id ; Mon, 03 Jul 2023 19:09:16 +0800 (CST) X-QQ-SSF: 01400000000000G0T000000A0000000 X-QQ-FEAT: q+EIYT+FhZpvwfIkuT3c2hDNRk272MqQCMdeHS+CygHeBklFZTWhZr6IfJ6YQ pKA7BC3VUApw6mo9oyfHwv4inP6r8d3BicvmaVIe9/ckIt7dlXdvAqUbuC566nzoLmd2bFJ X7RyCxsXvmDgQMpwbzw1nZbdrTZqo//Bukscv2JVw7TlpwnqJhiAxxSfRaVu+okj7iq6qyD Gf70dssFo0jfDTTpOqqGtrUDKlcuanZXE0HBeNXkXkw1AzB5/Y02jUTRaffRJCLtjqtb+Aq akSreKXE83ARUviW6lo1FbGA5ZuKVMocY/BvIzuCZrARUrxlVXZ+CI/EmKX1uVqZgeA3hRf NFuc7iPSuCVdtxSHP+OpepJrmUA8vgO/QoBnvxivWiFMUUqWlCuj8D5FI+o1m8EGlJdCOUy X-QQ-GoodBg: 2 X-BIZMAIL-ID: 8140089069030067765 From: juzhe.zhong@rivai.ai To: gcc-patches@gcc.gnu.org Cc: richard.sandiford@arm.com, rguenther@suse.de, Ju-Zhe Zhong Subject: [PATCH V7] Machine Description: Add LEN_MASK_{GATHER_LOAD, SCATTER_STORE} pattern Date: Mon, 3 Jul 2023 19:09:12 +0800 Message-Id: <20230703110912.15311-1-juzhe.zhong@rivai.ai> X-Mailer: git-send-email 2.36.1 MIME-Version: 1.0 X-QQ-SENDSIZE: 520 Feedback-ID: bizesmtp:rivai.ai:qybglogicsvrgz:qybglogicsvrgz7a-one-0 X-Spam-Status: No, score=-10.3 required=5.0 tests=BAYES_00, GIT_PATCH_0, KAM_DMARC_STATUS, KAM_SHORT, RCVD_IN_BARRACUDACENTRAL, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H4, RCVD_IN_MSPIKE_WL, SPF_HELO_PASS, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org Sender: "Gcc-patches" X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1770363876260769443?= X-GMAIL-MSGID: =?utf-8?q?1770397466824047813?= From: Ju-Zhe Zhong Hi, Richi and Richard. Base one the review comments from Richard: https://gcc.gnu.org/pipermail/gcc-patches/2023-July/623405.html I change len_mask_gather_load/len_mask_scatter_store order into: {len,bias,mask} We adjust adding len and mask using using add_len_and_mask_args which is same as partial_load/parial_store. Now, the codes become more reasonable and easier maintain. This patch is adding LEN_MASK_{GATHER_LOAD,SCATTER_STORE} to allow targets handle flow control by mask and loop control by length on gather/scatter memory operations. Consider this following case: #include void f (uint8_t *restrict a, uint8_t *restrict b, int n, int base, int step, int *restrict cond) { for (int i = 0; i < n; ++i) { if (cond[i]) a[i * step + base] = b[i * step + base]; } } We hope RVV can vectorize such case into following IR: loop_len = SELECT_VL control_mask = comparison v = LEN_MASK_GATHER_LOAD (.., loop_len, bias, control_mask) LEN_SCATTER_STORE (... v, ..., loop_len, bias, control_mask) This patch doesn't apply such patterns into vectorizer, just add patterns and update the documents. Will send patch which apply such patterns into vectorizer soon after this patch is approved. Ok for trunk? gcc/ChangeLog: * doc/md.texi: Add len_mask_gather_load/len_mask_scatter_store. * internal-fn.cc (expand_scatter_store_optab_fn): Ditto. (expand_gather_load_optab_fn): Ditto. (internal_load_fn_p): Ditto. (internal_store_fn_p): Ditto. (internal_gather_scatter_fn_p): Ditto. (internal_fn_len_index): Ditto. (internal_fn_mask_index): Ditto. (internal_fn_stored_value_index): Ditto. * internal-fn.def (LEN_MASK_GATHER_LOAD): Ditto. (LEN_MASK_SCATTER_STORE): Ditto. * optabs.def (OPTAB_CD): Ditto. --- gcc/doc/md.texi | 17 +++++++++++++++++ gcc/internal-fn.cc | 32 +++++++++++++++++--------------- gcc/internal-fn.def | 8 ++++++-- gcc/optabs.def | 2 ++ 4 files changed, 42 insertions(+), 17 deletions(-) diff --git a/gcc/doc/md.texi b/gcc/doc/md.texi index 5e5482265cd..f14dd32b2dc 100644 --- a/gcc/doc/md.texi +++ b/gcc/doc/md.texi @@ -5040,6 +5040,15 @@ operand 5. Bit @var{i} of the mask is set if element @var{i} of the result should be loaded from memory and clear if element @var{i} of the result should be set to zero. +@cindex @code{len_mask_gather_load@var{m}@var{n}} instruction pattern +@item @samp{len_mask_gather_load@var{m}@var{n}} +Like @samp{gather_load@var{m}@var{n}}, but takes an extra length operand (operand 5), +a bias operand (operand 6) as well as a mask operand (operand 7). Similar to len_maskload, +the instruction loads at most (operand 5 + operand 6) elements from memory. +Bit @var{i} of the mask is set if element @var{i} of the result should +be loaded from memory and clear if element @var{i} of the result should be undefined. +Mask elements @var{i} with @var{i} > (operand 5 + operand 6) are ignored. + @cindex @code{scatter_store@var{m}@var{n}} instruction pattern @item @samp{scatter_store@var{m}@var{n}} Store a vector of mode @var{m} into several distinct memory locations. @@ -5069,6 +5078,14 @@ Like @samp{scatter_store@var{m}@var{n}}, but takes an extra mask operand as operand 5. Bit @var{i} of the mask is set if element @var{i} of the result should be stored to memory. +@cindex @code{len_mask_scatter_store@var{m}@var{n}} instruction pattern +@item @samp{len_mask_scatter_store@var{m}@var{n}} +Like @samp{scatter_store@var{m}@var{n}}, but takes an extra length operand (operand 5), +a bias operand (operand 6) as well as a mask operand (operand 7). The instruction stores +at most (operand 5 + operand 6) elements of (operand 4) to memory. +Bit @var{i} of the mask is set if element @var{i} of (operand 4) should be stored. +Mask elements @var{i} with @var{i} > (operand 5 + operand 6) are ignored. + @cindex @code{vec_set@var{m}} instruction pattern @item @samp{vec_set@var{m}} Set given field in the vector value. Operand 0 is the vector to modify, diff --git a/gcc/internal-fn.cc b/gcc/internal-fn.cc index c1fcb38b17b..303df102d81 100644 --- a/gcc/internal-fn.cc +++ b/gcc/internal-fn.cc @@ -3507,7 +3507,6 @@ expand_scatter_store_optab_fn (internal_fn, gcall *stmt, direct_optab optab) { internal_fn ifn = gimple_call_internal_fn (stmt); int rhs_index = internal_fn_stored_value_index (ifn); - int mask_index = internal_fn_mask_index (ifn); tree base = gimple_call_arg (stmt, 0); tree offset = gimple_call_arg (stmt, 1); tree scale = gimple_call_arg (stmt, 2); @@ -3518,19 +3517,14 @@ expand_scatter_store_optab_fn (internal_fn, gcall *stmt, direct_optab optab) HOST_WIDE_INT scale_int = tree_to_shwi (scale); rtx rhs_rtx = expand_normal (rhs); - class expand_operand ops[6]; + class expand_operand ops[8]; int i = 0; create_address_operand (&ops[i++], base_rtx); create_input_operand (&ops[i++], offset_rtx, TYPE_MODE (TREE_TYPE (offset))); create_integer_operand (&ops[i++], TYPE_UNSIGNED (TREE_TYPE (offset))); create_integer_operand (&ops[i++], scale_int); create_input_operand (&ops[i++], rhs_rtx, TYPE_MODE (TREE_TYPE (rhs))); - if (mask_index >= 0) - { - tree mask = gimple_call_arg (stmt, mask_index); - rtx mask_rtx = expand_normal (mask); - create_input_operand (&ops[i++], mask_rtx, TYPE_MODE (TREE_TYPE (mask))); - } + i = add_len_and_mask_args (ops, i, stmt); insn_code icode = convert_optab_handler (optab, TYPE_MODE (TREE_TYPE (rhs)), TYPE_MODE (TREE_TYPE (offset))); @@ -3553,18 +3547,13 @@ expand_gather_load_optab_fn (internal_fn, gcall *stmt, direct_optab optab) HOST_WIDE_INT scale_int = tree_to_shwi (scale); int i = 0; - class expand_operand ops[6]; + class expand_operand ops[8]; create_output_operand (&ops[i++], lhs_rtx, TYPE_MODE (TREE_TYPE (lhs))); create_address_operand (&ops[i++], base_rtx); create_input_operand (&ops[i++], offset_rtx, TYPE_MODE (TREE_TYPE (offset))); create_integer_operand (&ops[i++], TYPE_UNSIGNED (TREE_TYPE (offset))); create_integer_operand (&ops[i++], scale_int); - if (optab == mask_gather_load_optab) - { - tree mask = gimple_call_arg (stmt, 4); - rtx mask_rtx = expand_normal (mask); - create_input_operand (&ops[i++], mask_rtx, TYPE_MODE (TREE_TYPE (mask))); - } + i = add_len_and_mask_args (ops, i, stmt); insn_code icode = convert_optab_handler (optab, TYPE_MODE (TREE_TYPE (lhs)), TYPE_MODE (TREE_TYPE (offset))); expand_insn (icode, i, ops); @@ -4415,6 +4404,7 @@ internal_load_fn_p (internal_fn fn) case IFN_MASK_LOAD_LANES: case IFN_GATHER_LOAD: case IFN_MASK_GATHER_LOAD: + case IFN_LEN_MASK_GATHER_LOAD: case IFN_LEN_LOAD: case IFN_LEN_MASK_LOAD: return true; @@ -4436,6 +4426,7 @@ internal_store_fn_p (internal_fn fn) case IFN_MASK_STORE_LANES: case IFN_SCATTER_STORE: case IFN_MASK_SCATTER_STORE: + case IFN_LEN_MASK_SCATTER_STORE: case IFN_LEN_STORE: case IFN_LEN_MASK_STORE: return true; @@ -4454,8 +4445,10 @@ internal_gather_scatter_fn_p (internal_fn fn) { case IFN_GATHER_LOAD: case IFN_MASK_GATHER_LOAD: + case IFN_LEN_MASK_GATHER_LOAD: case IFN_SCATTER_STORE: case IFN_MASK_SCATTER_STORE: + case IFN_LEN_MASK_SCATTER_STORE: return true; default: @@ -4477,6 +4470,10 @@ internal_fn_len_index (internal_fn fn) case IFN_LEN_MASK_STORE: return 2; + case IFN_LEN_MASK_GATHER_LOAD: + case IFN_LEN_MASK_SCATTER_STORE: + return 4; + default: return -1; } @@ -4502,6 +4499,10 @@ internal_fn_mask_index (internal_fn fn) case IFN_LEN_MASK_STORE: return 4; + case IFN_LEN_MASK_GATHER_LOAD: + case IFN_LEN_MASK_SCATTER_STORE: + return 6; + default: return (conditional_internal_fn_code (fn) != ERROR_MARK || get_unconditional_internal_fn (fn) != IFN_LAST ? 0 : -1); @@ -4520,6 +4521,7 @@ internal_fn_stored_value_index (internal_fn fn) case IFN_MASK_STORE_LANES: case IFN_SCATTER_STORE: case IFN_MASK_SCATTER_STORE: + case IFN_LEN_MASK_SCATTER_STORE: return 3; case IFN_LEN_STORE: diff --git a/gcc/internal-fn.def b/gcc/internal-fn.def index d9fcca8430f..9b73e540d55 100644 --- a/gcc/internal-fn.def +++ b/gcc/internal-fn.def @@ -48,14 +48,14 @@ along with GCC; see the file COPYING3. If not see - mask_load: currently just maskload - load_lanes: currently just vec_load_lanes - mask_load_lanes: currently just vec_mask_load_lanes - - gather_load: used for {mask_,}gather_load + - gather_load: used for {mask_,len_mask,}gather_load - len_load: currently just len_load - len_maskload: currently just len_maskload - mask_store: currently just maskstore - store_lanes: currently just vec_store_lanes - mask_store_lanes: currently just vec_mask_store_lanes - - scatter_store: used for {mask_,}scatter_store + - scatter_store: used for {mask_,len_mask,}scatter_store - len_store: currently just len_store - len_maskstore: currently just len_maskstore @@ -157,6 +157,8 @@ DEF_INTERNAL_OPTAB_FN (MASK_LOAD_LANES, ECF_PURE, DEF_INTERNAL_OPTAB_FN (GATHER_LOAD, ECF_PURE, gather_load, gather_load) DEF_INTERNAL_OPTAB_FN (MASK_GATHER_LOAD, ECF_PURE, mask_gather_load, gather_load) +DEF_INTERNAL_OPTAB_FN (LEN_MASK_GATHER_LOAD, ECF_PURE, + len_mask_gather_load, gather_load) DEF_INTERNAL_OPTAB_FN (LEN_LOAD, ECF_PURE, len_load, len_load) DEF_INTERNAL_OPTAB_FN (LEN_MASK_LOAD, ECF_PURE, len_maskload, len_maskload) @@ -164,6 +166,8 @@ DEF_INTERNAL_OPTAB_FN (LEN_MASK_LOAD, ECF_PURE, len_maskload, len_maskload) DEF_INTERNAL_OPTAB_FN (SCATTER_STORE, 0, scatter_store, scatter_store) DEF_INTERNAL_OPTAB_FN (MASK_SCATTER_STORE, 0, mask_scatter_store, scatter_store) +DEF_INTERNAL_OPTAB_FN (LEN_MASK_SCATTER_STORE, 0, + len_mask_scatter_store, scatter_store) DEF_INTERNAL_OPTAB_FN (MASK_STORE, 0, maskstore, mask_store) DEF_INTERNAL_OPTAB_FN (STORE_LANES, ECF_CONST, vec_store_lanes, store_lanes) diff --git a/gcc/optabs.def b/gcc/optabs.def index a901b68c538..73c9a0c760f 100644 --- a/gcc/optabs.def +++ b/gcc/optabs.def @@ -95,8 +95,10 @@ OPTAB_CD(len_maskload_optab, "len_maskload$a$b") OPTAB_CD(len_maskstore_optab, "len_maskstore$a$b") OPTAB_CD(gather_load_optab, "gather_load$a$b") OPTAB_CD(mask_gather_load_optab, "mask_gather_load$a$b") +OPTAB_CD(len_mask_gather_load_optab, "len_mask_gather_load$a$b") OPTAB_CD(scatter_store_optab, "scatter_store$a$b") OPTAB_CD(mask_scatter_store_optab, "mask_scatter_store$a$b") +OPTAB_CD(len_mask_scatter_store_optab, "len_mask_scatter_store$a$b") OPTAB_CD(vec_extract_optab, "vec_extract$a$b") OPTAB_CD(vec_init_optab, "vec_init$a$b")