From patchwork Tue Nov 14 03:46:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "juzhe.zhong@rivai.ai" X-Patchwork-Id: 164715 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b909:0:b0:403:3b70:6f57 with SMTP id t9csp1630265vqg; Mon, 13 Nov 2023 19:46:47 -0800 (PST) X-Google-Smtp-Source: AGHT+IHw2n3WXRy6K+ISe4M5K7QhkanC7W0hxJd1uFoOsZ2nku/ZZBfDLEAi+izqoi0JLokdszwX X-Received: by 2002:a05:622a:54:b0:41e:1d94:885f with SMTP id y20-20020a05622a005400b0041e1d94885fmr1352246qtw.41.1699933607001; Mon, 13 Nov 2023 19:46:47 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1699933606; cv=pass; d=google.com; s=arc-20160816; b=NltE0l08Gtfiq2GwfEJM1DUXSZkLPd2aiLekj9WM2mLEqhAh9f0X9JbZoVMU2I6yfe 9lCNp6Gk0rjPxsC6ufTuwI6ajz/Xbdn07QB2SuuuRg0sEqyBDWmr7N2R1yyTLvtGeZii Qtf/rd/vU7A/i1RJ/TPmw9DlVHGKk8E749XuqyPggQ8MumCnrmw1VxN1w5cebM1r1EaO +YIS+clbed+gNbB4uNzaBAiOC50eaPoRN5PsSQzUchXLXD7JdyBkIvA/yx1BubNu3o0F a/umtfJLOfPTOZHI8Wb/wa47WnQ0fC+/94dvXfV4O3KDoywL4HDrkSeueP30KFnv7uzX gwPw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:feedback-id :content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:arc-filter:dmarc-filter:delivered-to; bh=kwPuQaV4Jig2nIfmD4uaCyuQ3OeTExVZ7Kv1W9qIgFE=; fh=Hxj0cw/2jAkyitv6gv3v4BQlApRvZwleU4172JThTNQ=; b=ddcfA+i896aTzO/k0K/U83DEymfl8Wm3ymfJWB882LHF3bWk+90KfYm2nhPqELEWGg dTmwzHVlhR0rqcJpKQ7+wU0qM2eP84SZAH68lqE1yElC1C3U0d5mGatmKIAtzDyH/0pC 10i9i4WFiJZOJK44bEuqGrv7qmqfbUT9ioqtpckysmB4OUZYvyrza8DPZmIK0Hp49dHl 67ZbIIMRErU0PX3HJnMEObh4VxQps3ezoOryFWNG2j/bfBxrXtX2L2HKHUnB8desgM+M nJ8nFNYAseFeSJeRdAA6SqJlKpfTZJwvxm/y4ih4nHtac9VXHOn4QWBVDs6GPDMs1DFl 0xOw== ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (server2.sourceware.org. [2620:52:3:1:0:246e:9693:128c]) by mx.google.com with ESMTPS id q7-20020a05622a04c700b004123092feb6si6235996qtx.340.2023.11.13.19.46.46 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Nov 2023 19:46:46 -0800 (PST) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) client-ip=2620:52:3:1:0:246e:9693:128c; Authentication-Results: mx.google.com; arc=pass (i=1); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id C37BE3858C01 for ; Tue, 14 Nov 2023 03:46:46 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtpbg151.qq.com (smtpbg151.qq.com [18.169.211.239]) by sourceware.org (Postfix) with ESMTPS id A32A43858D35 for ; Tue, 14 Nov 2023 03:46:21 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org A32A43858D35 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivai.ai Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivai.ai ARC-Filter: OpenARC Filter v1.0.0 sourceware.org A32A43858D35 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=18.169.211.239 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1699933584; cv=none; b=jfzADFoSG7zlsf63DTDkgOPoJYmiICR4Pnl9Vicc7ijA4WXHLK/g+7oZsNefQO3mFyAfDbOBwwx5Ro2aNItwyM+El6eexx/jNU9N3s8zziZpMkuPLyC/5U6779W9dzbxs9XQfYr4imN0/X0KL42lLUJMicq9+Wi5GeuCQLNaANQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1699933584; c=relaxed/simple; bh=NrEPnqk+Zki8QmFYnUBB9dC0rM7Ck2FTIQ1+cDEa5Ac=; h=From:To:Subject:Date:Message-Id:MIME-Version; b=v6zwWEf2daqPah9I+h2ivg092uWus/7NiW3i7V+wEr4aaAkOgtXembeWASn4My4rHIxAnf33IoR+rBWO0tHLnrJvofikn86UGqgfWHOMAwNvdDOwWB4uFV1U2ahTtSR48m7Fb6Ds2Ej2aWpPUFzrCFYjeZ5gc4khhIW4uP6Gg6o= ARC-Authentication-Results: i=1; server2.sourceware.org X-QQ-mid: bizesmtp80t1699933576t4v8ez15 Received: from rios-cad122.hadoop.rioslab.org ( [58.60.1.26]) by bizesmtp.qq.com (ESMTP) with id ; Tue, 14 Nov 2023 11:46:15 +0800 (CST) X-QQ-SSF: 01400000000000G0V000000A0000000 X-QQ-FEAT: C46Rb8GPIEfq0P+RA140roXaNKw8mAWweC5sin/0DbAAvPo1SrMzUld2XgvfI mx90wtzJQeyJH9/K37rHl+mrRtUtNgE1QiZrLf9E80onbOhRERlYX5cgHXwjaMQdI5KOKsK Zh1H5Ql3TUaSQldAkqNo1a+llCT+XOw7e0T04ljzT3GB8h+5hz3sYuVTWTz4Jan7w4XUflj nd7nWu1GslGQXbVGJsqfSMWvpR6S76Pg2Z8UvU2HXhWwiupekDx2s1ATx9BEVjFNpQH1kq8 mU0BGafR1D8+1AJ+HgVgP5x1uM0x7OnnezdZ3PycL3kpOIybfQ5/ihxVpkrHFapPMUDXhrC rWAvLf5viGNNQ7azRnPUafr9gg0FwZF39lRVGR1IkWl7l+3i0r6xOKpaq8/OFqVSVc6LB8o X-QQ-GoodBg: 2 X-BIZMAIL-ID: 9531856747018420145 From: Juzhe-Zhong To: gcc-patches@gcc.gnu.org Cc: richard.sandiford@arm.com, rguenther@suse.de, Juzhe-Zhong Subject: [PATCH] VECT: Add MASK_LEN_STRIDED_LOAD/MASK_LEN_STRIDED_STORE into loop vectorizer Date: Tue, 14 Nov 2023 11:46:14 +0800 Message-Id: <20231114034614.1697097-1-juzhe.zhong@rivai.ai> X-Mailer: git-send-email 2.36.3 MIME-Version: 1.0 X-QQ-SENDSIZE: 520 Feedback-ID: bizesmtp:rivai.ai:qybglogicsvrgz:qybglogicsvrgz7a-one-0 X-Spam-Status: No, score=-10.3 required=5.0 tests=BAYES_00, GIT_PATCH_0, KAM_DMARC_STATUS, RCVD_IN_BARRACUDACENTRAL, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1782509582216895274 X-GMAIL-MSGID: 1782509582216895274 This patch support generating MASK_LEN_STRIDED_LOAD/MASK_LEN_STRIDED_STORE IR for invariant stride memory access. It's a special optimization for targets like RVV. RVV has both indexed load/store and stride load/store. In RVV, we always have gather/scatter and strided optab at the same time. E.g. void foo (int *__restrict a, int * __restrict b, int n, int *__restrict indice) { for (int i = 0; i < n; i++) a[indice[i]] = b[indice[i]] + a[i]; } Such vector codes, RVV is using indexed load/store for gather/scatter. E.g. void foo (int *__restrict a, int * __restrict b, int n, int m) { for (int i = 0; i < n; i++) a[i] = b[i * m] + a[i]; } Such vector codes, RVV is using stride load/store instructions. We only need to support direct mask_len_stride_xxx optab for invariant stride. gcc/ChangeLog: * tree-vect-stmts.cc (vect_get_strided_load_store_ops): Add MASK_LEN_STRIDED_LOAD/MASK_LEN_STRIDED_STORE. (vectorizable_store): Ditto. (vectorizable_load): Ditto. --- gcc/tree-vect-stmts.cc | 47 ++++++++++++++++++++++++++++++++++-------- 1 file changed, 38 insertions(+), 9 deletions(-) diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc index ee89f47c468..9c65b688510 100644 --- a/gcc/tree-vect-stmts.cc +++ b/gcc/tree-vect-stmts.cc @@ -2863,6 +2863,17 @@ vect_get_strided_load_store_ops (stmt_vec_info stmt_info, *dataref_bump = cse_and_gimplify_to_preheader (loop_vinfo, bump); } + /* Target supports strided load/store use DR_STEP as stride for VEC_OFFSET + directly instead of build VEC_OFFSET with VEC_SERIES. */ + internal_fn ifn + = DR_IS_READ (dr) ? IFN_MASK_LEN_STRIDED_LOAD : IFN_MASK_LEN_STRIDED_STORE; + if (direct_internal_fn_supported_p (ifn, vectype, OPTIMIZE_FOR_SPEED)) + { + *vec_offset = cse_and_gimplify_to_preheader (loop_vinfo, + unshare_expr (DR_STEP (dr))); + return; + } + /* The offset given in GS_INFO can have pointer type, so use the element type of the vector instead. */ tree offset_type = TREE_TYPE (gs_info->offset_vectype); @@ -9012,10 +9023,20 @@ vectorizable_store (vec_info *vinfo, gcall *call; if (final_len && final_mask) - call = gimple_build_call_internal - (IFN_MASK_LEN_SCATTER_STORE, 7, dataref_ptr, - vec_offset, scale, vec_oprnd, final_mask, - final_len, bias); + { + if (VECTOR_TYPE_P (TREE_TYPE (vec_offset))) + call = gimple_build_call_internal ( + IFN_MASK_LEN_SCATTER_STORE, 7, dataref_ptr, + vec_offset, scale, vec_oprnd, final_mask, final_len, + bias); + else + /* non-vector type offset means that target prefers to + use MASK_LEN_STRIDED_STORE instead of + MASK_LEN_GATHER_STORE with direct stride argument. */ + call = gimple_build_call_internal ( + IFN_MASK_LEN_STRIDED_STORE, 6, dataref_ptr, + vec_offset, vec_oprnd, final_mask, final_len, bias); + } else if (final_mask) call = gimple_build_call_internal (IFN_MASK_SCATTER_STORE, 5, dataref_ptr, @@ -10956,11 +10977,19 @@ vectorizable_load (vec_info *vinfo, gcall *call; if (final_len && final_mask) - call - = gimple_build_call_internal (IFN_MASK_LEN_GATHER_LOAD, 7, - dataref_ptr, vec_offset, - scale, zero, final_mask, - final_len, bias); + { + if (VECTOR_TYPE_P (TREE_TYPE (vec_offset))) + call = gimple_build_call_internal ( + IFN_MASK_LEN_GATHER_LOAD, 7, dataref_ptr, vec_offset, + scale, zero, final_mask, final_len, bias); + else + /* non-vector type offset means that target prefers to + use MASK_LEN_STRIDED_LOAD instead of + MASK_LEN_GATHER_LOAD with direct stride argument. */ + call = gimple_build_call_internal ( + IFN_MASK_LEN_STRIDED_LOAD, 6, dataref_ptr, vec_offset, + zero, final_mask, final_len, bias); + } else if (final_mask) call = gimple_build_call_internal (IFN_MASK_GATHER_LOAD, 5, dataref_ptr, vec_offset,