From patchwork Mon May 22 15:36:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Matthias Kretz X-Patchwork-Id: 97514 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1538973vqo; Mon, 22 May 2023 08:37:58 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ66iTUzUODWFIB1jiqPlfEdmPY+YcWW6HXKZC/cvJxEaaT2dymysFqszIK9igb/4X0TIVMG X-Received: by 2002:a17:906:dc94:b0:96a:47d7:8b52 with SMTP id cs20-20020a170906dc9400b0096a47d78b52mr10083279ejc.77.1684769877961; Mon, 22 May 2023 08:37:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684769877; cv=none; d=google.com; s=arc-20160816; b=jMHSkCit97BDNSXT04E5LrHNZhWY13kT7983H7Tfbzyo846ou+WBAbioQXDqjwRmek a23Mty7du1GNF90/MwXmNkD1cCYI1VUIdSCvM85jSWJT1aKqg1t5sLSSugmY+/x9gw6Q umEnR2ntxfHrLxJT22KbnQKyHdI73fPB8Z6uU0LjUQHRkgNlPxqvnnSXtLbyJXqFmkfV 3NkYfpzBQjivHh4IiZDfvT3C/jM1sC6mYs9K7kEv4Gulfzd4CEc5QBeNa7igIvq8maoY KfK+BTgOQPmITkXrqJT1PrbnO3SNf8nMq0OyOuNv3hw3H3jdestuoPjK+jsDv8xhH/Jo l+eQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:reply-to:from:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence :content-transfer-encoding:mime-version:organization:message-id:date :subject:to:dmarc-filter:delivered-to:dkim-signature:dkim-filter; bh=bYjnhe7tcLuQZxFujrWLgFNQX9wW1i5JuIBlp4nVOUA=; b=AtpAE2afU0ZA6TFPSwiPOAuoTPisLLBMHi6jmtgCzR9pvda5a6NcbPpZn2Oiu84keh ltu6Kc8WOcT8x+umgKSsllxkmMB0z9JcewLS4++v2yK+E0v+o5m4DiVmmUTTiWt9at9h Ajna4ZjYVie/+cw43D9mlQ8vqh3+cDmI1adWXHfemdHzKmkkY/TNI4ZyjAhASxJwlI/0 WDp912rhBS6KnDrfnv5l8vlr5R3FRmvi4wpDF99Gg9yrqYJ4+Z2tLBlRRws4qBGvg4mD dnS0xQMvR8NtmUbm1ZS621vIOXJBygirg8vMec/r7eK4jocFYN+9uKRJzVBdKc+gXAyh tsuw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=ynO3jtbE; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gnu.org Received: from sourceware.org (server2.sourceware.org. [2620:52:3:1:0:246e:9693:128c]) by mx.google.com with ESMTPS id g12-20020aa7d1cc000000b0050bd2211c6asi586282edp.374.2023.05.22.08.37.57 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 May 2023 08:37:57 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) client-ip=2620:52:3:1:0:246e:9693:128c; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=ynO3jtbE; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gnu.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 531E43858438 for ; Mon, 22 May 2023 15:37:56 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 531E43858438 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1684769876; bh=bYjnhe7tcLuQZxFujrWLgFNQX9wW1i5JuIBlp4nVOUA=; h=To:Subject:Date:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=ynO3jtbEebbsc73/35UFtrWy7NGQoNQs6VPBm8y3mShEKXtKIQyV1ZByTtwa9wjMn oOh4JIfxLh9cKz9fRBfAqwZ7V7/SSJFr6U+L8jn3spn9gvomXqVwYYCmf9dTHRRNq+ S/Je3aDF4YQiPWdEBXTrfSqfYYDrbigPEmsfyhEI= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from lxmtout2.gsi.de (lxmtout2.gsi.de [140.181.3.112]) by sourceware.org (Postfix) with ESMTPS id 6FCB53858D35; Mon, 22 May 2023 15:36:21 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 6FCB53858D35 Received: from localhost (localhost [127.0.0.1]) by lxmtout2.gsi.de (Postfix) with ESMTP id 9F920202AD46; Mon, 22 May 2023 17:36:20 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at lxmtout2.gsi.de Received: from lxmtout2.gsi.de ([127.0.0.1]) by localhost (lxmtout2.gsi.de [127.0.0.1]) (amavisd-new, port 10024) with LMTP id CcsUz_PBv8Xo; Mon, 22 May 2023 17:36:20 +0200 (CEST) Received: from srvEX6.campus.gsi.de (unknown [10.10.4.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lxmtout2.gsi.de (Postfix) with ESMTPS id 74DF9203E7FB; Mon, 22 May 2023 17:36:20 +0200 (CEST) Received: from minbar.localnet (140.181.3.12) by srvEX6.campus.gsi.de (10.10.4.96) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1118.26; Mon, 22 May 2023 17:36:20 +0200 To: , Subject: [PATCH] libstdc++: Add missing constexpr to simd Date: Mon, 22 May 2023 17:36:19 +0200 Message-ID: <116291317.nniJfEyVGO@minbar> Organization: GSI Helmholtz Centre for Heavy Ion Research MIME-Version: 1.0 X-Originating-IP: [140.181.3.12] X-ClientProxiedBy: srvEX8.Campus.gsi.de (10.10.4.160) To srvEX6.campus.gsi.de (10.10.4.96) X-Spam-Status: No, score=-10.2 required=5.0 tests=BAYES_00, BODY_8BITS, GIT_PATCH_0, KAM_DMARC_STATUS, SPF_HELO_PASS, SPF_PASS, TXREP, T_FILL_THIS_FORM_SHORT, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Matthias Kretz via Gcc-patches From: Matthias Kretz Reply-To: Matthias Kretz Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org Sender: "Gcc-patches" X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766609259125326160?= X-GMAIL-MSGID: =?utf-8?q?1766609259125326160?= OK for trunk and backporting? regtested on x86_64-linux and aarch64-linux The constexpr API is only available with -std=gnu++XX (and proposed for C++26). The proposal is to have the complete simd API usable in constant expressions. This patch resolves several issues with using simd in constant expressions. Issues why constant_evaluated branches are necessary: * subscripting vector builtins is not allowed in constant expressions * if the implementation needs/uses memcpy * if the implementation would otherwise call SIMD intrinsics/builtins Signed-off-by: Matthias Kretz libstdc++-v3/ChangeLog: PR libstdc++/109261 * include/experimental/bits/simd.h (_SimdWrapper::_M_set): Avoid vector builtin subscripting in constant expressions. (resizing_simd_cast): Avoid memcpy if constant_evaluated. (const_where_expression, where_expression, where) (__extract_part, simd_mask, _SimdIntOperators, simd): Add either _GLIBCXX_SIMD_CONSTEXPR (on public APIs), or constexpr (on internal APIs). * include/experimental/bits/simd_builtin.h (__vector_permute) (__vector_shuffle, __extract_part, _GnuTraits::_SimdCastType1) (_GnuTraits::_SimdCastType2, _SimdImplBuiltin) (_MaskImplBuiltin::_S_store): Add constexpr. (_CommonImplBuiltin::_S_store_bool_array) (_SimdImplBuiltin::_S_load, _SimdImplBuiltin::_S_store) (_SimdImplBuiltin::_S_reduce, _MaskImplBuiltin::_S_load): Add constant_evaluated case. * include/experimental/bits/simd_fixed_size.h (_S_masked_load): Reword comment. (__tuple_element_meta, __make_meta, _SimdTuple::_M_apply_r) (_SimdTuple::_M_subscript_read, _SimdTuple::_M_subscript_write) (__make_simd_tuple, __optimize_simd_tuple, __extract_part) (__autocvt_to_simd, _Fixed::__traits::_SimdBase) (_Fixed::__traits::_SimdCastType, _SimdImplFixedSize): Add constexpr. (_SimdTuple::operator[], _M_set): Add constexpr and add constant_evaluated case. (_MaskImplFixedSize::_S_load): Add constant_evaluated case. * include/experimental/bits/simd_scalar.h: Add constexpr. * include/experimental/bits/simd_x86.h (_CommonImplX86): Add constexpr and add constant_evaluated case. (_SimdImplX86::_S_equal_to, _S_not_equal_to, _S_less) (_S_less_equal): Value-initialize to satisfy constexpr evaluation. (_MaskImplX86::_S_load): Add constant_evaluated case. (_MaskImplX86::_S_store): Add constexpr and constant_evaluated case. Value-initialize local variables. (_MaskImplX86::_S_logical_and, _S_logical_or, _S_bit_not) (_S_bit_and, _S_bit_or, _S_bit_xor): Add constant_evaluated case. * testsuite/experimental/simd/pr109261_constexpr_simd.cc: New test. --- libstdc++-v3/include/experimental/bits/simd.h | 153 ++++++++------- .../include/experimental/bits/simd_builtin.h | 100 ++++++---- .../experimental/bits/simd_fixed_size.h | 177 +++++++++--------- .../include/experimental/bits/simd_scalar.h | 78 ++++---- .../include/experimental/bits/simd_x86.h | 68 +++++-- .../simd/pr109261_constexpr_simd.cc | 109 +++++++++++ 6 files changed, 437 insertions(+), 248 deletions(-) create mode 100644 libstdc++-v3/testsuite/experimental/simd/ pr109261_constexpr_simd.cc -- ────────────────────────────────────────────────────────────────────────── Dr. Matthias Kretz https://mattkretz.github.io GSI Helmholtz Centre for Heavy Ion Research https://gsi.de stdₓ::simd ────────────────────────────────────────────────────────────────────────── diff --git a/libstdc++-v3/include/experimental/bits/simd.h b/libstdc++-v3/include/experimental/bits/simd.h index 224153ffbaf..b0571ca26c4 100644 --- a/libstdc++-v3/include/experimental/bits/simd.h +++ b/libstdc++-v3/include/experimental/bits/simd.h @@ -2675,7 +2675,14 @@ _SimdWrapper(_V __x) _GLIBCXX_SIMD_INTRINSIC constexpr void _M_set(size_t __i, _Tp __x) - { _M_data[__i] = __x; } + { + if (__builtin_is_constant_evaluated()) + _M_data = __generate_from_n_evaluations<_Width, _BuiltinType>([&](auto __j) { + return __j == __i ? __x : _M_data[__j()]; + }); + else + _M_data[__i] = __x; + } _GLIBCXX_SIMD_INTRINSIC constexpr bool @@ -3186,6 +3193,10 @@ resizing_simd_cast(const simd<_Up, _Ap>& __x) { if constexpr (is_same_v) return __x; + else if (__builtin_is_constant_evaluated()) + return _Tp([&](auto __i) constexpr { + return __i < simd_size_v<_Up, _Ap> ? __x[__i] : _Up(); + }); else if constexpr (simd_size_v<_Up, _Ap> == 1) { _Tp __r{}; @@ -3321,10 +3332,11 @@ __get_lvalue(const const_where_expression& __x) const_where_expression& operator=(const const_where_expression&) = delete; - _GLIBCXX_SIMD_INTRINSIC const_where_expression(const _M& __kk, const _Tp& dd) - : _M_k(__kk), _M_value(const_cast<_Tp&>(dd)) {} + _GLIBCXX_SIMD_INTRINSIC constexpr + const_where_expression(const _M& __kk, const _Tp& dd) + : _M_k(__kk), _M_value(const_cast<_Tp&>(dd)) {} - _GLIBCXX_SIMD_INTRINSIC _V + _GLIBCXX_SIMD_INTRINSIC _GLIBCXX_SIMD_CONSTEXPR _V operator-() const&& { return {__private_init, @@ -3333,7 +3345,7 @@ __get_lvalue(const const_where_expression& __x) } template - [[nodiscard]] _GLIBCXX_SIMD_INTRINSIC _V + [[nodiscard]] _GLIBCXX_SIMD_INTRINSIC _GLIBCXX_SIMD_CONSTEXPR _V copy_from(const _LoadStorePtr<_Up, value_type>* __mem, _IsSimdFlagType<_Flags>) const&& { return {__private_init, @@ -3342,7 +3354,7 @@ __get_lvalue(const const_where_expression& __x) } template - _GLIBCXX_SIMD_INTRINSIC void + _GLIBCXX_SIMD_INTRINSIC _GLIBCXX_SIMD_CONSTEXPR void copy_to(_LoadStorePtr<_Up, value_type>* __mem, _IsSimdFlagType<_Flags>) const&& { _Impl::_S_masked_store(__data(_M_value), @@ -3381,19 +3393,21 @@ __get_lvalue(const const_where_expression& __x) const_where_expression(const const_where_expression&) = delete; const_where_expression& operator=(const const_where_expression&) = delete; - _GLIBCXX_SIMD_INTRINSIC const_where_expression(const bool __kk, const _Tp& dd) - : _M_k(__kk), _M_value(const_cast<_Tp&>(dd)) {} + _GLIBCXX_SIMD_INTRINSIC constexpr + const_where_expression(const bool __kk, const _Tp& dd) + : _M_k(__kk), _M_value(const_cast<_Tp&>(dd)) {} - _GLIBCXX_SIMD_INTRINSIC _V operator-() const&& + _GLIBCXX_SIMD_INTRINSIC _GLIBCXX_SIMD_CONSTEXPR _V + operator-() const&& { return _M_k ? -_M_value : _M_value; } template - [[nodiscard]] _GLIBCXX_SIMD_INTRINSIC _V + [[nodiscard]] _GLIBCXX_SIMD_INTRINSIC _GLIBCXX_SIMD_CONSTEXPR _V copy_from(const _LoadStorePtr<_Up, value_type>* __mem, _IsSimdFlagType<_Flags>) const&& { return _M_k ? static_cast<_V>(__mem[0]) : _M_value; } template - _GLIBCXX_SIMD_INTRINSIC void + _GLIBCXX_SIMD_INTRINSIC _GLIBCXX_SIMD_CONSTEXPR void copy_to(_LoadStorePtr<_Up, value_type>* __mem, _IsSimdFlagType<_Flags>) const&& { if (_M_k) @@ -3419,18 +3433,21 @@ static_assert( is_same::value, ""); static_assert(_M::size() == _Tp::size(), ""); - _GLIBCXX_SIMD_INTRINSIC friend _Tp& __get_lvalue(where_expression& __x) + _GLIBCXX_SIMD_INTRINSIC friend constexpr _Tp& + __get_lvalue(where_expression& __x) { return __x._M_value; } public: where_expression(const where_expression&) = delete; where_expression& operator=(const where_expression&) = delete; - _GLIBCXX_SIMD_INTRINSIC where_expression(const _M& __kk, _Tp& dd) - : const_where_expression<_M, _Tp>(__kk, dd) {} + _GLIBCXX_SIMD_INTRINSIC _GLIBCXX_SIMD_CONSTEXPR + where_expression(const _M& __kk, _Tp& dd) + : const_where_expression<_M, _Tp>(__kk, dd) {} template - _GLIBCXX_SIMD_INTRINSIC void operator=(_Up&& __x) && + _GLIBCXX_SIMD_INTRINSIC _GLIBCXX_SIMD_CONSTEXPR void + operator=(_Up&& __x) && { _Impl::_S_masked_assign(__data(_M_k), __data(_M_value), __to_value_type_or_member_type<_Tp>( @@ -3439,7 +3456,8 @@ static_assert( #define _GLIBCXX_SIMD_OP_(__op, __name) \ template \ - _GLIBCXX_SIMD_INTRINSIC void operator __op##=(_Up&& __x)&& \ + _GLIBCXX_SIMD_INTRINSIC _GLIBCXX_SIMD_CONSTEXPR void \ + operator __op##=(_Up&& __x)&& \ { \ _Impl::template _S_masked_cassign( \ __data(_M_k), __data(_M_value), \ @@ -3461,28 +3479,28 @@ static_assert( _GLIBCXX_SIMD_OP_(>>, _S_shift_right); #undef _GLIBCXX_SIMD_OP_ - _GLIBCXX_SIMD_INTRINSIC void + _GLIBCXX_SIMD_INTRINSIC _GLIBCXX_SIMD_CONSTEXPR void operator++() && { __data(_M_value) = _Impl::template _S_masked_unary<__increment>(__data(_M_k), __data(_M_value)); } - _GLIBCXX_SIMD_INTRINSIC void + _GLIBCXX_SIMD_INTRINSIC _GLIBCXX_SIMD_CONSTEXPR void operator++(int) && { __data(_M_value) = _Impl::template _S_masked_unary<__increment>(__data(_M_k), __data(_M_value)); } - _GLIBCXX_SIMD_INTRINSIC void + _GLIBCXX_SIMD_INTRINSIC _GLIBCXX_SIMD_CONSTEXPR void operator--() && { __data(_M_value) = _Impl::template _S_masked_unary<__decrement>(__data(_M_k), __data(_M_value)); } - _GLIBCXX_SIMD_INTRINSIC void + _GLIBCXX_SIMD_INTRINSIC _GLIBCXX_SIMD_CONSTEXPR void operator--(int) && { __data(_M_value) @@ -3491,7 +3509,7 @@ static_assert( // intentionally hides const_where_expression::copy_from template - _GLIBCXX_SIMD_INTRINSIC void + _GLIBCXX_SIMD_INTRINSIC _GLIBCXX_SIMD_CONSTEXPR void copy_from(const _LoadStorePtr<_Up, value_type>* __mem, _IsSimdFlagType<_Flags>) && { __data(_M_value) = _Impl::_S_masked_load(__data(_M_value), __data(_M_k), @@ -3513,13 +3531,13 @@ class where_expression where_expression(const where_expression&) = delete; where_expression& operator=(const where_expression&) = delete; - _GLIBCXX_SIMD_INTRINSIC + _GLIBCXX_SIMD_INTRINSIC _GLIBCXX_SIMD_CONSTEXPR where_expression(const _M& __kk, _Tp& dd) : const_where_expression<_M, _Tp>(__kk, dd) {} #define _GLIBCXX_SIMD_OP_(__op) \ template \ - _GLIBCXX_SIMD_INTRINSIC void \ + _GLIBCXX_SIMD_INTRINSIC _GLIBCXX_SIMD_CONSTEXPR void \ operator __op(_Up&& __x)&& \ { if (_M_k) _M_value __op static_cast<_Up&&>(__x); } @@ -3536,68 +3554,71 @@ where_expression(const _M& __kk, _Tp& dd) _GLIBCXX_SIMD_OP_(>>=) #undef _GLIBCXX_SIMD_OP_ - _GLIBCXX_SIMD_INTRINSIC void + _GLIBCXX_SIMD_INTRINSIC _GLIBCXX_SIMD_CONSTEXPR void operator++() && { if (_M_k) ++_M_value; } - _GLIBCXX_SIMD_INTRINSIC void + _GLIBCXX_SIMD_INTRINSIC _GLIBCXX_SIMD_CONSTEXPR void operator++(int) && { if (_M_k) ++_M_value; } - _GLIBCXX_SIMD_INTRINSIC void + _GLIBCXX_SIMD_INTRINSIC _GLIBCXX_SIMD_CONSTEXPR void operator--() && { if (_M_k) --_M_value; } - _GLIBCXX_SIMD_INTRINSIC void + _GLIBCXX_SIMD_INTRINSIC _GLIBCXX_SIMD_CONSTEXPR void operator--(int) && { if (_M_k) --_M_value; } // intentionally hides const_where_expression::copy_from template - _GLIBCXX_SIMD_INTRINSIC void + _GLIBCXX_SIMD_INTRINSIC _GLIBCXX_SIMD_CONSTEXPR void copy_from(const _LoadStorePtr<_Up, value_type>* __mem, _IsSimdFlagType<_Flags>) && { if (_M_k) _M_value = __mem[0]; } }; // where {{{1 template - _GLIBCXX_SIMD_INTRINSIC where_expression, simd<_Tp, _Ap>> + _GLIBCXX_SIMD_INTRINSIC _GLIBCXX_SIMD_CONSTEXPR + where_expression, simd<_Tp, _Ap>> where(const typename simd<_Tp, _Ap>::mask_type& __k, simd<_Tp, _Ap>& __value) { return {__k, __value}; } template - _GLIBCXX_SIMD_INTRINSIC + _GLIBCXX_SIMD_INTRINSIC _GLIBCXX_SIMD_CONSTEXPR const_where_expression, simd<_Tp, _Ap>> where(const typename simd<_Tp, _Ap>::mask_type& __k, const simd<_Tp, _Ap>& __value) { return {__k, __value}; } template - _GLIBCXX_SIMD_INTRINSIC + _GLIBCXX_SIMD_INTRINSIC _GLIBCXX_SIMD_CONSTEXPR where_expression, simd_mask<_Tp, _Ap>> where(const remove_const_t>& __k, simd_mask<_Tp, _Ap>& __value) { return {__k, __value}; } template - _GLIBCXX_SIMD_INTRINSIC + _GLIBCXX_SIMD_INTRINSIC _GLIBCXX_SIMD_CONSTEXPR const_where_expression, simd_mask<_Tp, _Ap>> where(const remove_const_t>& __k, const simd_mask<_Tp, _Ap>& __value) { return {__k, __value}; } template - _GLIBCXX_SIMD_INTRINSIC where_expression + _GLIBCXX_SIMD_INTRINSIC _GLIBCXX_SIMD_CONSTEXPR where_expression where(_ExactBool __k, _Tp& __value) { return {__k, __value}; } template - _GLIBCXX_SIMD_INTRINSIC const_where_expression + _GLIBCXX_SIMD_INTRINSIC _GLIBCXX_SIMD_CONSTEXPR const_where_expression where(_ExactBool __k, const _Tp& __value) { return {__k, __value}; } template - void where(bool __k, simd<_Tp, _Ap>& __value) = delete; + _GLIBCXX_SIMD_CONSTEXPR void + where(bool __k, simd<_Tp, _Ap>& __value) = delete; template - void where(bool __k, const simd<_Tp, _Ap>& __value) = delete; + _GLIBCXX_SIMD_CONSTEXPR void + where(bool __k, const simd<_Tp, _Ap>& __value) = delete; // proposed mask iterations {{{1 namespace __proposed { @@ -3820,12 +3841,12 @@ clamp(const simd<_Tp, _Ap>& __v, const simd<_Tp, _Ap>& __lo, const simd<_Tp, _Ap // __extract_part {{{ template - _GLIBCXX_SIMD_INTRINSIC _GLIBCXX_CONST + _GLIBCXX_SIMD_INTRINSIC _GLIBCXX_CONST constexpr _SimdWrapper<_Tp, _Np / _Total * _Combine> __extract_part(const _SimdWrapper<_Tp, _Np> __x); template - _GLIBCXX_SIMD_INTRINSIC auto + _GLIBCXX_SIMD_INTRINSIC constexpr auto __extract_part(const _SimdTuple<_Tp, _A0, _As...>& __x); // }}} @@ -4551,7 +4572,7 @@ class simd_mask // }}} // access to internal representation (optional feature) {{{ - _GLIBCXX_SIMD_ALWAYS_INLINE explicit + _GLIBCXX_SIMD_ALWAYS_INLINE _GLIBCXX_SIMD_CONSTEXPR explicit simd_mask(typename _Traits::_MaskCastType __init) : _M_data{__init} {} // conversions to internal type is done in _MaskBase @@ -4562,11 +4583,11 @@ class simd_mask // Conversion of simd_mask to and from bitset makes it much easier to // interface with other facilities. I suggest adding `static // simd_mask::from_bitset` and `simd_mask::to_bitset`. - _GLIBCXX_SIMD_ALWAYS_INLINE static simd_mask + _GLIBCXX_SIMD_ALWAYS_INLINE _GLIBCXX_SIMD_CONSTEXPR static simd_mask __from_bitset(bitset bs) { return {__bitset_init, bs}; } - _GLIBCXX_SIMD_ALWAYS_INLINE bitset + _GLIBCXX_SIMD_ALWAYS_INLINE _GLIBCXX_SIMD_CONSTEXPR bitset __to_bitset() const { return _Impl::_S_to_bits(_M_data)._M_to_bitset(); } @@ -4591,7 +4612,7 @@ simd_mask(const simd_mask<_Up, _A2>& __x) template >, is_same<_Up, _Up>>::value>> - _GLIBCXX_SIMD_ALWAYS_INLINE + _GLIBCXX_SIMD_ALWAYS_INLINE _GLIBCXX_SIMD_CONSTEXPR simd_mask(const simd_mask<_Up, simd_abi::fixed_size>& __x) : _M_data(_Impl::_S_from_bitmask(__data(__x), _S_type_tag)) {} #endif @@ -4599,12 +4620,12 @@ simd_mask(const simd_mask<_Up, simd_abi::fixed_size>& __x) // }}} // load constructor {{{ template - _GLIBCXX_SIMD_ALWAYS_INLINE + _GLIBCXX_SIMD_ALWAYS_INLINE _GLIBCXX_SIMD_CONSTEXPR simd_mask(const value_type* __mem, _IsSimdFlagType<_Flags>) : _M_data(_Impl::template _S_load<_Ip>(_Flags::template _S_apply(__mem))) {} template - _GLIBCXX_SIMD_ALWAYS_INLINE + _GLIBCXX_SIMD_ALWAYS_INLINE _GLIBCXX_SIMD_CONSTEXPR simd_mask(const value_type* __mem, simd_mask __k, _IsSimdFlagType<_Flags>) : _M_data{} { @@ -4615,20 +4636,20 @@ simd_mask(const simd_mask<_Up, simd_abi::fixed_size>& __x) // }}} // loads [simd_mask.load] {{{ template - _GLIBCXX_SIMD_ALWAYS_INLINE void + _GLIBCXX_SIMD_ALWAYS_INLINE _GLIBCXX_SIMD_CONSTEXPR void copy_from(const value_type* __mem, _IsSimdFlagType<_Flags>) { _M_data = _Impl::template _S_load<_Ip>(_Flags::template _S_apply(__mem)); } // }}} // stores [simd_mask.store] {{{ template - _GLIBCXX_SIMD_ALWAYS_INLINE void + _GLIBCXX_SIMD_ALWAYS_INLINE _GLIBCXX_SIMD_CONSTEXPR void copy_to(value_type* __mem, _IsSimdFlagType<_Flags>) const { _Impl::_S_store(_M_data, _Flags::template _S_apply(__mem)); } // }}} // scalar access {{{ - _GLIBCXX_SIMD_ALWAYS_INLINE reference + _GLIBCXX_SIMD_ALWAYS_INLINE _GLIBCXX_SIMD_CONSTEXPR reference operator[](size_t __i) { if (__i >= size()) @@ -4636,7 +4657,7 @@ simd_mask(const simd_mask<_Up, simd_abi::fixed_size>& __x) return {_M_data, int(__i)}; } - _GLIBCXX_SIMD_ALWAYS_INLINE value_type + _GLIBCXX_SIMD_ALWAYS_INLINE _GLIBCXX_SIMD_CONSTEXPR value_type operator[](size_t __i) const { if (__i >= size()) @@ -4649,7 +4670,7 @@ simd_mask(const simd_mask<_Up, simd_abi::fixed_size>& __x) // }}} // negation {{{ - _GLIBCXX_SIMD_ALWAYS_INLINE simd_mask + _GLIBCXX_SIMD_ALWAYS_INLINE _GLIBCXX_SIMD_CONSTEXPR simd_mask operator!() const { return {__private_init, _Impl::_S_bit_not(_M_data)}; } @@ -4659,7 +4680,7 @@ simd_mask(const simd_mask<_Up, simd_abi::fixed_size>& __x) // simd_mask && simd_mask needs disambiguation template , simd_mask>>> - _GLIBCXX_SIMD_ALWAYS_INLINE friend simd_mask + _GLIBCXX_SIMD_ALWAYS_INLINE _GLIBCXX_SIMD_CONSTEXPR friend simd_mask operator&&(const simd_mask& __x, const simd_mask<_Up, _A2>& __y) { return {__private_init, @@ -4668,7 +4689,7 @@ simd_mask(const simd_mask<_Up, simd_abi::fixed_size>& __x) template , simd_mask>>> - _GLIBCXX_SIMD_ALWAYS_INLINE friend simd_mask + _GLIBCXX_SIMD_ALWAYS_INLINE _GLIBCXX_SIMD_CONSTEXPR friend simd_mask operator||(const simd_mask& __x, const simd_mask<_Up, _A2>& __y) { return {__private_init, @@ -4676,41 +4697,41 @@ simd_mask(const simd_mask<_Up, simd_abi::fixed_size>& __x) } #endif // _GLIBCXX_SIMD_ENABLE_IMPLICIT_MASK_CAST - _GLIBCXX_SIMD_ALWAYS_INLINE friend simd_mask + _GLIBCXX_SIMD_ALWAYS_INLINE _GLIBCXX_SIMD_CONSTEXPR friend simd_mask operator&&(const simd_mask& __x, const simd_mask& __y) { return {__private_init, _Impl::_S_logical_and(__x._M_data, __y._M_data)}; } - _GLIBCXX_SIMD_ALWAYS_INLINE friend simd_mask + _GLIBCXX_SIMD_ALWAYS_INLINE _GLIBCXX_SIMD_CONSTEXPR friend simd_mask operator||(const simd_mask& __x, const simd_mask& __y) { return {__private_init, _Impl::_S_logical_or(__x._M_data, __y._M_data)}; } - _GLIBCXX_SIMD_ALWAYS_INLINE friend simd_mask + _GLIBCXX_SIMD_ALWAYS_INLINE _GLIBCXX_SIMD_CONSTEXPR friend simd_mask operator&(const simd_mask& __x, const simd_mask& __y) { return {__private_init, _Impl::_S_bit_and(__x._M_data, __y._M_data)}; } - _GLIBCXX_SIMD_ALWAYS_INLINE friend simd_mask + _GLIBCXX_SIMD_ALWAYS_INLINE _GLIBCXX_SIMD_CONSTEXPR friend simd_mask operator|(const simd_mask& __x, const simd_mask& __y) { return {__private_init, _Impl::_S_bit_or(__x._M_data, __y._M_data)}; } - _GLIBCXX_SIMD_ALWAYS_INLINE friend simd_mask + _GLIBCXX_SIMD_ALWAYS_INLINE _GLIBCXX_SIMD_CONSTEXPR friend simd_mask operator^(const simd_mask& __x, const simd_mask& __y) { return {__private_init, _Impl::_S_bit_xor(__x._M_data, __y._M_data)}; } - _GLIBCXX_SIMD_ALWAYS_INLINE friend simd_mask& + _GLIBCXX_SIMD_ALWAYS_INLINE _GLIBCXX_SIMD_CONSTEXPR friend simd_mask& operator&=(simd_mask& __x, const simd_mask& __y) { __x._M_data = _Impl::_S_bit_and(__x._M_data, __y._M_data); return __x; } - _GLIBCXX_SIMD_ALWAYS_INLINE friend simd_mask& + _GLIBCXX_SIMD_ALWAYS_INLINE _GLIBCXX_SIMD_CONSTEXPR friend simd_mask& operator|=(simd_mask& __x, const simd_mask& __y) { __x._M_data = _Impl::_S_bit_or(__x._M_data, __y._M_data); return __x; } - _GLIBCXX_SIMD_ALWAYS_INLINE friend simd_mask& + _GLIBCXX_SIMD_ALWAYS_INLINE _GLIBCXX_SIMD_CONSTEXPR friend simd_mask& operator^=(simd_mask& __x, const simd_mask& __y) { __x._M_data = _Impl::_S_bit_xor(__x._M_data, __y._M_data); @@ -4747,7 +4768,8 @@ simd_mask(const simd_mask<_Up, simd_abi::fixed_size>& __x) // }}} // bitset_init ctor {{{ - _GLIBCXX_SIMD_INTRINSIC simd_mask(_BitsetInit, bitset __init) + _GLIBCXX_SIMD_INTRINSIC constexpr + simd_mask(_BitsetInit, bitset __init) : _M_data(_Impl::_S_from_bitmask(_SanitizedBitMask(__init), _S_type_tag)) {} @@ -5015,7 +5037,8 @@ class _SimdIntOperators<_V, _Tp, _Abi, true> { using _Impl = typename _SimdTraits<_Tp, _Abi>::_SimdImpl; - _GLIBCXX_SIMD_INTRINSIC const _V& __derived() const + _GLIBCXX_SIMD_INTRINSIC constexpr const _V& + __derived() const { return *static_cast(this); } template @@ -5235,7 +5258,7 @@ simd(const simd<_Up, _A2>& __x) // load constructor template - _GLIBCXX_SIMD_ALWAYS_INLINE + _GLIBCXX_SIMD_ALWAYS_INLINE _GLIBCXX_SIMD_CONSTEXPR simd(const _Up* __mem, _IsSimdFlagType<_Flags>) : _M_data( _Impl::_S_load(_Flags::template _S_apply(__mem), _S_type_tag)) @@ -5243,7 +5266,7 @@ simd(const simd<_Up, _A2>& __x) // loads [simd.load] template - _GLIBCXX_SIMD_ALWAYS_INLINE void + _GLIBCXX_SIMD_ALWAYS_INLINE _GLIBCXX_SIMD_CONSTEXPR void copy_from(const _Vectorizable<_Up>* __mem, _IsSimdFlagType<_Flags>) { _M_data = static_cast( @@ -5252,7 +5275,7 @@ simd(const simd<_Up, _A2>& __x) // stores [simd.store] template - _GLIBCXX_SIMD_ALWAYS_INLINE void + _GLIBCXX_SIMD_ALWAYS_INLINE _GLIBCXX_SIMD_CONSTEXPR void copy_to(_Vectorizable<_Up>* __mem, _IsSimdFlagType<_Flags>) const { _Impl::_S_store(_M_data, _Flags::template _S_apply(__mem), @@ -5424,7 +5447,7 @@ _M_is_constprop() const } private: - _GLIBCXX_SIMD_INTRINSIC _GLIBCXX_SIMD_CONSTEXPR static mask_type + _GLIBCXX_SIMD_INTRINSIC static constexpr mask_type _S_make_mask(typename mask_type::_MemberType __k) { return {__private_init, __k}; } diff --git a/libstdc++-v3/include/experimental/bits/simd_builtin.h b/libstdc++-v3/include/experimental/bits/simd_builtin.h index 3d52bc6c96a..8337fa2d9a6 100644 --- a/libstdc++-v3/include/experimental/bits/simd_builtin.h +++ b/libstdc++-v3/include/experimental/bits/simd_builtin.h @@ -52,7 +52,7 @@ // Index == -1 requests zeroing of the output element template , typename = __detail::__odr_helper> - _Tp + constexpr _Tp __vector_permute(_Tp __x) { static_assert(sizeof...(_Indices) == _TVT::_S_full_size); @@ -65,7 +65,7 @@ __vector_permute(_Tp __x) // Index == -1 requests zeroing of the output element template , typename = __detail::__odr_helper> - _Tp + constexpr _Tp __vector_shuffle(_Tp __x, _Tp __y) { return _Tp{(_Indices == -1 ? 0 @@ -205,7 +205,7 @@ __shift_elements_right(_Tp __v) // }}} // __extract_part(_SimdWrapper<_Tp, _Np>) {{{ template - _GLIBCXX_SIMD_INTRINSIC _GLIBCXX_CONST + _GLIBCXX_SIMD_INTRINSIC _GLIBCXX_CONST constexpr _SimdWrapper<_Tp, _Np / _Total * _Combine> __extract_part(const _SimdWrapper<_Tp, _Np> __x) { @@ -905,10 +905,10 @@ class _SimdCastType1 _SimdMember _M_data; public: - _GLIBCXX_SIMD_ALWAYS_INLINE + _GLIBCXX_SIMD_ALWAYS_INLINE constexpr _SimdCastType1(_Ap __a) : _M_data(__vector_bitcast<_Tp>(__a)) {} - _GLIBCXX_SIMD_ALWAYS_INLINE + _GLIBCXX_SIMD_ALWAYS_INLINE constexpr operator _SimdMember() const { return _M_data; } }; @@ -919,13 +919,13 @@ class _SimdCastType2 _SimdMember _M_data; public: - _GLIBCXX_SIMD_ALWAYS_INLINE + _GLIBCXX_SIMD_ALWAYS_INLINE constexpr _SimdCastType2(_Ap __a) : _M_data(__vector_bitcast<_Tp>(__a)) {} - _GLIBCXX_SIMD_ALWAYS_INLINE + _GLIBCXX_SIMD_ALWAYS_INLINE constexpr _SimdCastType2(_Bp __b) : _M_data(__b) {} - _GLIBCXX_SIMD_ALWAYS_INLINE + _GLIBCXX_SIMD_ALWAYS_INLINE constexpr operator _SimdMember() const { return _M_data; } }; @@ -1345,6 +1345,11 @@ _S_store_bool_array(_BitMask<_Np, _Sanitized> __x, bool* __mem) { if constexpr (_Np == 1) __mem[0] = __x[0]; + else if (__builtin_is_constant_evaluated()) + { + for (size_t __i = 0; __i < _Np; ++__i) + __mem[__i] = __x[__i]; + } else if constexpr (_Np == 2) { short __bool2 = (__x._M_to_bits() * 0x81) & 0x0101; @@ -1424,12 +1429,12 @@ struct _SimdImplBuiltin // _M_make_simd(_SimdWrapper/__intrinsic_type_t) {{{2 template - _GLIBCXX_SIMD_INTRINSIC static simd<_Tp, _Abi> + _GLIBCXX_SIMD_INTRINSIC static constexpr simd<_Tp, _Abi> _M_make_simd(_SimdWrapper<_Tp, _Np> __x) { return {__private_init, __x}; } template - _GLIBCXX_SIMD_INTRINSIC static simd<_Tp, _Abi> + _GLIBCXX_SIMD_INTRINSIC static constexpr simd<_Tp, _Abi> _M_make_simd(__intrinsic_type_t<_Tp, _Np> __x) { return {__private_init, __vector_bitcast<_Tp>(__x)}; } @@ -1455,7 +1460,7 @@ _S_broadcast(_Tp __x) noexcept // _S_load {{{2 template - _GLIBCXX_SIMD_INTRINSIC static _SimdMember<_Tp> + _GLIBCXX_SIMD_INTRINSIC static constexpr _SimdMember<_Tp> _S_load(const _Up* __mem, _TypeTag<_Tp>) noexcept { constexpr size_t _Np = _S_size<_Tp>; @@ -1464,7 +1469,12 @@ _S_broadcast(_Tp __x) noexcept : (is_floating_point_v<_Up> && __have_avx) || __have_avx2 ? 32 : 16; constexpr size_t __bytes_to_load = sizeof(_Up) * _Np; - if constexpr (sizeof(_Up) > 8) + if (__builtin_is_constant_evaluated()) + return __generate_vector<_Tp, _S_full_size<_Tp>>( + [&](auto __i) constexpr { + return static_cast<_Tp>(__i < _Np ? __mem[__i] : 0); + }); + else if constexpr (sizeof(_Up) > 8) return __generate_vector<_Tp, _SimdMember<_Tp>::_S_full_size>( [&](auto __i) constexpr _GLIBCXX_SIMD_ALWAYS_INLINE_LAMBDA { return static_cast<_Tp>(__i < _Np ? __mem[__i] : 0); @@ -1511,7 +1521,7 @@ _S_broadcast(_Tp __x) noexcept // _S_masked_load {{{2 template - static inline _SimdWrapper<_Tp, _Np> + static constexpr inline _SimdWrapper<_Tp, _Np> _S_masked_load(_SimdWrapper<_Tp, _Np> __merge, _MaskMember<_Tp> __k, const _Up* __mem) noexcept { @@ -1524,14 +1534,19 @@ _S_masked_load(_SimdWrapper<_Tp, _Np> __merge, _MaskMember<_Tp> __k, // _S_store {{{2 template - _GLIBCXX_SIMD_INTRINSIC static void + _GLIBCXX_SIMD_INTRINSIC static constexpr void _S_store(_SimdMember<_Tp> __v, _Up* __mem, _TypeTag<_Tp>) noexcept { // TODO: converting int -> "smaller int" can be optimized with AVX512 constexpr size_t _Np = _S_size<_Tp>; constexpr size_t __max_store_size = _SuperImpl::template _S_max_store_size<_Up>; - if constexpr (sizeof(_Up) > 8) + if (__builtin_is_constant_evaluated()) + { + for (size_t __i = 0; __i < _Np; ++__i) + __mem[__i] = __v[__i]; + } + else if constexpr (sizeof(_Up) > 8) __execute_n_times<_Np>([&](auto __i) constexpr _GLIBCXX_SIMD_ALWAYS_INLINE_LAMBDA { __mem[__i] = __v[__i]; }); @@ -1562,7 +1577,7 @@ _S_masked_load(_SimdWrapper<_Tp, _Np> __merge, _MaskMember<_Tp> __k, // _S_masked_store_nocvt {{{2 template - _GLIBCXX_SIMD_INTRINSIC static void + _GLIBCXX_SIMD_INTRINSIC static constexpr void _S_masked_store_nocvt(_SimdWrapper<_Tp, _Np> __v, _Tp* __mem, _MaskMember<_Tp> __k) { _BitOps::_S_bit_iteration( @@ -1575,7 +1590,7 @@ _S_masked_store_nocvt(_SimdWrapper<_Tp, _Np> __v, _Tp* __mem, _MaskMember<_Tp> _ // _S_masked_store {{{2 template , typename _Tp = typename _TVT::value_type, typename _Up> - static inline void + static constexpr inline void _S_masked_store(const _TW __v, _Up* __mem, const _MaskMember<_Tp> __k) noexcept { constexpr size_t _TV_size = _S_size<_Tp>; @@ -1803,7 +1818,7 @@ _S_minmax(_SimdWrapper<_Tp, _Np> __a, _SimdWrapper<_Tp, _Np> __b) // reductions {{{2 template - _GLIBCXX_SIMD_INTRINSIC static _Tp + _GLIBCXX_SIMD_INTRINSIC static constexpr _Tp _S_reduce_partial(index_sequence<_Is...>, index_sequence<_Zeros...>, simd<_Tp, _Abi> __x, _BinaryOperation&& __binary_op) { @@ -1833,6 +1848,13 @@ _S_reduce(simd<_Tp, _Abi> __x, _BinaryOperation&& __binary_op) else if constexpr (_Np == 2) return __binary_op(simd<_Tp, simd_abi::scalar>(__x[0]), simd<_Tp, simd_abi::scalar>(__x[1]))[0]; + else if (__builtin_is_constant_evaluated()) + { + simd<_Tp, simd_abi::scalar> __acc = __x[0]; + for (size_t __i = 1; __i < _Np; ++__i) + __acc = __binary_op(__acc, simd<_Tp, simd_abi::scalar>(__x[__i])); + return __acc[0]; + } else if constexpr (_Abi::template _S_is_partial<_Tp>) //{{{ { [[maybe_unused]] constexpr auto __full_size @@ -2445,24 +2467,24 @@ _S_fpclassify(_SimdWrapper<_Tp, _Np> __x) // _S_increment & _S_decrement{{{2 template - _GLIBCXX_SIMD_INTRINSIC static void + _GLIBCXX_SIMD_INTRINSIC static constexpr void _S_increment(_SimdWrapper<_Tp, _Np>& __x) { __x = __x._M_data + 1; } template - _GLIBCXX_SIMD_INTRINSIC static void + _GLIBCXX_SIMD_INTRINSIC static constexpr void _S_decrement(_SimdWrapper<_Tp, _Np>& __x) { __x = __x._M_data - 1; } // smart_reference access {{{2 template - _GLIBCXX_SIMD_INTRINSIC constexpr static void + _GLIBCXX_SIMD_INTRINSIC static constexpr void _S_set(_SimdWrapper<_Tp, _Np>& __v, int __i, _Up&& __x) noexcept { __v._M_set(__i, static_cast<_Up&&>(__x)); } // _S_masked_assign{{{2 template - _GLIBCXX_SIMD_INTRINSIC static void + _GLIBCXX_SIMD_INTRINSIC static constexpr void _S_masked_assign(_SimdWrapper<_K, _Np> __k, _SimdWrapper<_Tp, _Np>& __lhs, __type_identity_t<_SimdWrapper<_Tp, _Np>> __rhs) { @@ -2475,7 +2497,7 @@ _S_masked_assign(_SimdWrapper<_K, _Np> __k, _SimdWrapper<_Tp, _Np>& __lhs, } template - _GLIBCXX_SIMD_INTRINSIC static void + _GLIBCXX_SIMD_INTRINSIC static constexpr void _S_masked_assign(_SimdWrapper<_K, _Np> __k, _SimdWrapper<_Tp, _Np>& __lhs, __type_identity_t<_Tp> __rhs) { @@ -2503,7 +2525,7 @@ _S_masked_assign(_SimdWrapper<_K, _Np> __k, _SimdWrapper<_Tp, _Np>& __lhs, // _S_masked_cassign {{{2 template - _GLIBCXX_SIMD_INTRINSIC static void + _GLIBCXX_SIMD_INTRINSIC static constexpr void _S_masked_cassign(const _SimdWrapper<_K, _Np> __k, _SimdWrapper<_Tp, _Np>& __lhs, const __type_identity_t<_SimdWrapper<_Tp, _Np>> __rhs, @@ -2519,7 +2541,7 @@ _S_masked_cassign(const _SimdWrapper<_K, _Np> __k, } template - _GLIBCXX_SIMD_INTRINSIC static void + _GLIBCXX_SIMD_INTRINSIC static constexpr void _S_masked_cassign(const _SimdWrapper<_K, _Np> __k, _SimdWrapper<_Tp, _Np>& __lhs, const __type_identity_t<_Tp> __rhs, _Op __op) @@ -2528,7 +2550,7 @@ _S_masked_cassign(const _SimdWrapper<_K, _Np> __k, // _S_masked_unary {{{2 template