libstdc++: Allow emergency EH alloc pool size to be tuned [PR68606]
Checks
Commit Message
This needs a little more documentation (see the TODO in the manual),
rather than just the comments in the source. This isn't final, but I
think it's the direction I want to take.
-- >8 --
Implement a long-standing request to support tuning the size of the
emergency buffer for allocating exceptions after malloc fails, or to
disable that buffer entirely.
It's now possible to disable the dynamic allocation of the buffer and
use a fixed-size static buffer, via --enable-libstdcxx-static-eh-pool.
This is a built-time choice that is baked into libstdc++ and so affects
all code linked against that build of libstdc++.
The size of the pool can be set by --with-libstdcxx-eh-pool-obj-count=N
which is measured in units of sizeof(void*) not bytes. A given exception
type such as std::system_error depends on the target, so giving a size
in bytes wouldn't be portable across 16/32/64-bit targets.
When libstdc++ is configured to use a dynamic buffer, the size of that
buffer can now be tuned at runtime by setting the GLIBCXX_TUNABLES
environment variable (c.f. PR libstdc++/88264). The number of exceptions
to reserve space for is controlled by the "glibcxx.eh_pool.obj_count"
and "glibcxx.eh_pool.obj_size" tunables. The pool will be sized to be
able to allocate obj_count exceptions of size obj_size*sizeof(void*) and
obj_count "dependent" exceptions rethrown by std::rethrow_exception.
With the ability to tune the buffer size, we can reduce the default pool
size. Most users never need to throw 1kB exceptions in parallel from
hundreds of threads after malloc is OOM. The users who do need that can
use the tunables to select larger sizes. The old default can be chosen
with:
For 64-bit:
GLIBCXX_TUNABLES="glibcxx.eh_pool.obj_count=64:glibcxx.eh_pool.obj_size=112"
For 32-bit:
GLIBCXX_TUNABLES="glibcxx.eh_pool.obj_count=32:glibcxx.eh_pool.obj_size=112"
libstdc++-v3/ChangeLog:
PR libstdc++/68606
* Makefile.in: Regenerate.
* acinclude.m4 (GLIBCXX_EMERGENCY_EH_ALLOC): New macro.
* configure: Regenerate.
* configure.ac: Use GLIBCXX_EMERGENCY_EH_ALLOC.
* crossconfig.m4: Check for secure_getenv.
* doc/Makefile.in: Regenerate.
* doc/xml/manual/configure.xml: Document new configure options.
* doc/xml/manual/using_exceptions.xml: Document emergency
buffer.
* doc/html/*: Regenerate.
* include/Makefile.in: Regenerate.
* libsupc++/Makefile.am: Use EH_POOL_FLAGS.
* libsupc++/Makefile.in: Regenerate.
* libsupc++/eh_alloc.cc (EMERGENCY_OBJ_SIZE): Define in units
of sizeof(void*) not including the ABI's exception header.
(EMERGENCY_OBJ_COUNT): Define as target-independent calculation
based on word size.
(MAX_OBJ_COUNT): Define macro for upper limit on pool size.
(pool) [_GLIBCXX_EH_POOL_STATIC]: Use fixed-size buffer.
(pool::buffer_size_in_bytes): New static member function.
(pool::pool): Parse GLIBCXX_TUNABLES environment variable to set
pool size at runtime.
(pool::in_pool): Use std::less<void*> for total order.
(__freeres) [_GLIBCXX_EH_POOL_STATIC]: Do nothing.
(__cxa_allocate_exception, __cxa_free_exception)
(__cxa_allocate_dependent_exception)
(__cxa_free_dependent_exception): Add [[unlikely]] attributes.
* po/Makefile.in: Regenerate.
* python/Makefile.in: Regenerate.
* src/Makefile.in: Regenerate.
* src/c++11/Makefile.in: Regenerate.
* src/c++17/Makefile.in: Regenerate.
* src/c++20/Makefile.in: Regenerate.
* src/c++98/Makefile.in: Regenerate.
* src/filesystem/Makefile.in: Regenerate.
* src/libbacktrace/Makefile.in: Regenerate.
* testsuite/Makefile.in: Regenerate.
---
libstdc++-v3/Makefile.in | 1 +
libstdc++-v3/acinclude.m4 | 45 ++++
libstdc++-v3/configure | 67 +++++-
libstdc++-v3/configure.ac | 3 +
libstdc++-v3/crossconfig.m4 | 1 +
libstdc++-v3/doc/Makefile.in | 1 +
libstdc++-v3/doc/html/index.html | 2 +-
libstdc++-v3/doc/html/manual/configure.html | 10 +-
libstdc++-v3/doc/html/manual/index.html | 2 +-
libstdc++-v3/doc/html/manual/intro.html | 2 +-
libstdc++-v3/doc/html/manual/using.html | 2 +-
.../doc/html/manual/using_exceptions.html | 53 +++-
libstdc++-v3/doc/xml/manual/configure.xml | 23 ++
.../doc/xml/manual/using_exceptions.xml | 48 ++++
libstdc++-v3/include/Makefile.in | 1 +
libstdc++-v3/libsupc++/Makefile.am | 2 +-
libstdc++-v3/libsupc++/Makefile.in | 3 +-
libstdc++-v3/libsupc++/eh_alloc.cc | 226 +++++++++++++-----
libstdc++-v3/po/Makefile.in | 1 +
libstdc++-v3/python/Makefile.in | 1 +
libstdc++-v3/src/Makefile.in | 1 +
libstdc++-v3/src/c++11/Makefile.in | 1 +
libstdc++-v3/src/c++17/Makefile.in | 1 +
libstdc++-v3/src/c++20/Makefile.in | 1 +
libstdc++-v3/src/c++98/Makefile.in | 1 +
libstdc++-v3/src/filesystem/Makefile.in | 1 +
libstdc++-v3/src/libbacktrace/Makefile.in | 1 +
libstdc++-v3/testsuite/Makefile.in | 1 +
28 files changed, 414 insertions(+), 88 deletions(-)
Comments
On Fri, Oct 7, 2022 at 5:55 PM Jonathan Wakely via Gcc-patches
<gcc-patches@gcc.gnu.org> wrote:
>
> This needs a little more documentation (see the TODO in the manual),
> rather than just the comments in the source. This isn't final, but I
> think it's the direction I want to take.
>
> -- >8 --
>
> Implement a long-standing request to support tuning the size of the
> emergency buffer for allocating exceptions after malloc fails, or to
> disable that buffer entirely.
>
> It's now possible to disable the dynamic allocation of the buffer and
> use a fixed-size static buffer, via --enable-libstdcxx-static-eh-pool.
> This is a built-time choice that is baked into libstdc++ and so affects
> all code linked against that build of libstdc++.
>
> The size of the pool can be set by --with-libstdcxx-eh-pool-obj-count=N
> which is measured in units of sizeof(void*) not bytes. A given exception
> type such as std::system_error depends on the target, so giving a size
> in bytes wouldn't be portable across 16/32/64-bit targets.
>
> When libstdc++ is configured to use a dynamic buffer, the size of that
> buffer can now be tuned at runtime by setting the GLIBCXX_TUNABLES
> environment variable (c.f. PR libstdc++/88264). The number of exceptions
> to reserve space for is controlled by the "glibcxx.eh_pool.obj_count"
> and "glibcxx.eh_pool.obj_size" tunables. The pool will be sized to be
> able to allocate obj_count exceptions of size obj_size*sizeof(void*) and
> obj_count "dependent" exceptions rethrown by std::rethrow_exception.
>
> With the ability to tune the buffer size, we can reduce the default pool
> size. Most users never need to throw 1kB exceptions in parallel from
> hundreds of threads after malloc is OOM.
But does it hurt? Back in time when I reworked the allocator to be less
wasteful the whole point was to allow more exceptions to be in-flight
during OOM shutdown of a process with many threads.
So if we reduce the default buffer size that should be documented
in changes.html, maybe with a hint how to restore the old buffer size
(configury flags required or runtime ENV setting)?
Otherwise looks OK to me.
Thanks,
Richard.
> The users who do need that can
> use the tunables to select larger sizes. The old default can be chosen
> with:
>
> For 64-bit:
> GLIBCXX_TUNABLES="glibcxx.eh_pool.obj_count=64:glibcxx.eh_pool.obj_size=112"
> For 32-bit:
> GLIBCXX_TUNABLES="glibcxx.eh_pool.obj_count=32:glibcxx.eh_pool.obj_size=112"
>
> libstdc++-v3/ChangeLog:
>
> PR libstdc++/68606
> * Makefile.in: Regenerate.
> * acinclude.m4 (GLIBCXX_EMERGENCY_EH_ALLOC): New macro.
> * configure: Regenerate.
> * configure.ac: Use GLIBCXX_EMERGENCY_EH_ALLOC.
> * crossconfig.m4: Check for secure_getenv.
> * doc/Makefile.in: Regenerate.
> * doc/xml/manual/configure.xml: Document new configure options.
> * doc/xml/manual/using_exceptions.xml: Document emergency
> buffer.
> * doc/html/*: Regenerate.
> * include/Makefile.in: Regenerate.
> * libsupc++/Makefile.am: Use EH_POOL_FLAGS.
> * libsupc++/Makefile.in: Regenerate.
> * libsupc++/eh_alloc.cc (EMERGENCY_OBJ_SIZE): Define in units
> of sizeof(void*) not including the ABI's exception header.
> (EMERGENCY_OBJ_COUNT): Define as target-independent calculation
> based on word size.
> (MAX_OBJ_COUNT): Define macro for upper limit on pool size.
> (pool) [_GLIBCXX_EH_POOL_STATIC]: Use fixed-size buffer.
> (pool::buffer_size_in_bytes): New static member function.
> (pool::pool): Parse GLIBCXX_TUNABLES environment variable to set
> pool size at runtime.
> (pool::in_pool): Use std::less<void*> for total order.
> (__freeres) [_GLIBCXX_EH_POOL_STATIC]: Do nothing.
> (__cxa_allocate_exception, __cxa_free_exception)
> (__cxa_allocate_dependent_exception)
> (__cxa_free_dependent_exception): Add [[unlikely]] attributes.
> * po/Makefile.in: Regenerate.
> * python/Makefile.in: Regenerate.
> * src/Makefile.in: Regenerate.
> * src/c++11/Makefile.in: Regenerate.
> * src/c++17/Makefile.in: Regenerate.
> * src/c++20/Makefile.in: Regenerate.
> * src/c++98/Makefile.in: Regenerate.
> * src/filesystem/Makefile.in: Regenerate.
> * src/libbacktrace/Makefile.in: Regenerate.
> * testsuite/Makefile.in: Regenerate.
> ---
> libstdc++-v3/Makefile.in | 1 +
> libstdc++-v3/acinclude.m4 | 45 ++++
> libstdc++-v3/configure | 67 +++++-
> libstdc++-v3/configure.ac | 3 +
> libstdc++-v3/crossconfig.m4 | 1 +
> libstdc++-v3/doc/Makefile.in | 1 +
> libstdc++-v3/doc/html/index.html | 2 +-
> libstdc++-v3/doc/html/manual/configure.html | 10 +-
> libstdc++-v3/doc/html/manual/index.html | 2 +-
> libstdc++-v3/doc/html/manual/intro.html | 2 +-
> libstdc++-v3/doc/html/manual/using.html | 2 +-
> .../doc/html/manual/using_exceptions.html | 53 +++-
> libstdc++-v3/doc/xml/manual/configure.xml | 23 ++
> .../doc/xml/manual/using_exceptions.xml | 48 ++++
> libstdc++-v3/include/Makefile.in | 1 +
> libstdc++-v3/libsupc++/Makefile.am | 2 +-
> libstdc++-v3/libsupc++/Makefile.in | 3 +-
> libstdc++-v3/libsupc++/eh_alloc.cc | 226 +++++++++++++-----
> libstdc++-v3/po/Makefile.in | 1 +
> libstdc++-v3/python/Makefile.in | 1 +
> libstdc++-v3/src/Makefile.in | 1 +
> libstdc++-v3/src/c++11/Makefile.in | 1 +
> libstdc++-v3/src/c++17/Makefile.in | 1 +
> libstdc++-v3/src/c++20/Makefile.in | 1 +
> libstdc++-v3/src/c++98/Makefile.in | 1 +
> libstdc++-v3/src/filesystem/Makefile.in | 1 +
> libstdc++-v3/src/libbacktrace/Makefile.in | 1 +
> libstdc++-v3/testsuite/Makefile.in | 1 +
> 28 files changed, 414 insertions(+), 88 deletions(-)
>
> diff --git a/libstdc++-v3/Makefile.in b/libstdc++-v3/Makefile.in
> index 58a0acdcc1b..a7c2b60678b 100644
> --- a/libstdc++-v3/Makefile.in
> +++ b/libstdc++-v3/Makefile.in
> @@ -240,6 +240,7 @@ ECHO_C = @ECHO_C@
> ECHO_N = @ECHO_N@
> ECHO_T = @ECHO_T@
> EGREP = @EGREP@
> +EH_POOL_FLAGS = @EH_POOL_FLAGS@
> ERROR_CONSTANTS_SRCDIR = @ERROR_CONSTANTS_SRCDIR@
> EXEEXT = @EXEEXT@
> EXTRA_CFLAGS = @EXTRA_CFLAGS@
> diff --git a/libstdc++-v3/acinclude.m4 b/libstdc++-v3/acinclude.m4
> index 719eab15c77..6523ba9a816 100644
> --- a/libstdc++-v3/acinclude.m4
> +++ b/libstdc++-v3/acinclude.m4
> @@ -5092,6 +5092,51 @@ BACKTRACE_CPPFLAGS="$BACKTRACE_CPPFLAGS -DBACKTRACE_ELF_SIZE=$elfsize"
> GLIBCXX_CONDITIONAL(ENABLE_BACKTRACE, [test "$enable_libstdcxx_backtrace" = yes])
> ])
>
> +dnl
> +dnl Allow the emergency EH pool to be configured.
> +dnl
> +dnl --enable-libstdcxx-static-eh-pool will cause a fixed-size static buffer
> +dnl to be used for allocating exceptions after malloc fails. The default is
> +dnl to allocate a buffer using malloc
> +dnl
> +dnl --with-libstdcxx-eh-pool-obj-count=N will set the default size for the
> +dnl buffer. For a static buffer that size is fixed permanently. For a dynamic
> +dnl buffer it's the default, but it can be overridden from the environment.
> +dnl
> +dnl To set the default to approximately the same values as GCC 12,
> +dnl use --with-libstdcxx-eh-pool-obj-count=94 for 32-bit targets,
> +dnl and --with-libstdcxx-eh-pool-obj-count=252 for 64-bit targets.
> +dnl
> +dnl Defines:
> +dnl _GLIBCXX_EH_POOL_STATIC if a fixed-size static buffer should be used
> +dnl instead of allocating a buffer on startup.
> +dnl _GLIBCXX_EH_POOL_NOBJS to override the default EMERGENCY_OBJ_COUNT value.
> +dnl
> +AC_DEFUN([GLIBCXX_EMERGENCY_EH_ALLOC], [
> + eh_pool_static=
> + eh_pool_nobjs=
> + AC_ARG_ENABLE([libstdcxx-static-eh-pool],
> + AC_HELP_STRING([--enable-libstdcxx-static-eh-pool],
> + [use a fixed-size static buffer for allocating exceptions if malloc fails]),
> + [if test "${enableval}" = yes; then
> + eh_pool_static="-D_GLIBCXX_EH_POOL_STATIC"
> + AC_MSG_NOTICE([EH pool using static buffer])
> + fi],)
> +
> + AC_ARG_WITH([libstdcxx-eh-pool-obj-count],
> + AC_HELP_STRING([--with-libstdcxx-eh-pool-obj-count],
> + [the number of exceptions that can be allocated from the pool if malloc fails]),
> + [if test "${withval}" -ge 0 2>/dev/null; then
> + eh_pool_obj_count="-D_GLIBCXX_EH_POOL_NOBJS=${withval}"
> + AC_MSG_NOTICE([EH pool object count: ${withval}])
> + else
> + AC_MSG_ERROR([EH pool obj count must be a non-negative integer: $withval])
> + fi],)
> +
> + EH_POOL_FLAGS="$eh_pool_static $eh_pool_obj_count"
> + AC_SUBST(EH_POOL_FLAGS)
> +])
> +
> # Macros from the top-level gcc directory.
> m4_include([../config/gc++filt.m4])
> m4_include([../config/tls.m4])
> diff --git a/libstdc++-v3/configure.ac b/libstdc++-v3/configure.ac
> index c05fcdda7e9..42c453099f2 100644
> --- a/libstdc++-v3/configure.ac
> +++ b/libstdc++-v3/configure.ac
> @@ -538,6 +538,9 @@ GLIBCXX_CHECK_SIZE_T_MANGLING
> # Check which release added std::exception_ptr for the target
> GLIBCXX_CHECK_EXCEPTION_PTR_SYMVER
>
> +# For libsupc++/eh_alloc.cc defaults.
> +GLIBCXX_EMERGENCY_EH_ALLOC
> +
> # Define documentation rules conditionally.
>
> # See if makeinfo has been installed and is modern enough
> diff --git a/libstdc++-v3/crossconfig.m4 b/libstdc++-v3/crossconfig.m4
> index 130f47fb1d4..b3269cb88e0 100644
> --- a/libstdc++-v3/crossconfig.m4
> +++ b/libstdc++-v3/crossconfig.m4
> @@ -187,6 +187,7 @@ case "${host}" in
> AC_CHECK_FUNCS(timespec_get)
> AC_CHECK_FUNCS(sockatmark)
> AC_CHECK_FUNCS(uselocale)
> + AC_CHECK_FUNCS(secure_getenv)
> AM_ICONV
> ;;
> *-mingw32*)
> diff --git a/libstdc++-v3/doc/xml/manual/configure.xml b/libstdc++-v3/doc/xml/manual/configure.xml
> index 8c26acc95a7..c6c5981968d 100644
> --- a/libstdc++-v3/doc/xml/manual/configure.xml
> +++ b/libstdc++-v3/doc/xml/manual/configure.xml
> @@ -366,6 +366,11 @@
> </para>
> </listitem></varlistentry>
>
> + <varlistentry><term><code>--disable-libstdcxx-hosted</code></term>
> + <listitem>
> + <para>This is an alias for <code>--disable-hosted-libstdcxx</code>.</para>
> + </listitem></varlistentry>
> +
> <varlistentry><term><code>--disable-libstdcxx-verbose</code></term>
> <listitem>
> <para>
> @@ -446,6 +451,24 @@
> </para>
> </listitem></varlistentry>
>
> + <varlistentry><term><code>--enable-libstdcxx-static-eh-pool</code></term>
> + <listitem>
> + <para>Use a fixed-size static buffer for the emergency exception handling
> + pool (see <xref linkend="intro.using.exception.alloc"/>). The default
> + is to allocate the pool on program startup using <code>malloc</code>.
> + With this option, a static buffer will be provided by libstdc++ instead.
> + This does not change the library ABI.
> + </para>
> + </listitem></varlistentry>
> +
> + <varlistentry><term><code>--with-libstdcxx-eh-pool-obj-count=NUM</code></term>
> + <listitem>
> + <para>Set the size of the emergency exception handling pool. NUM is the
> + number of simultaneous allocated exceptions to support.
> + This does not change the library ABI.
> + </para>
> + </listitem></varlistentry>
> +
> </variablelist>
>
> </section>
> diff --git a/libstdc++-v3/doc/xml/manual/using_exceptions.xml b/libstdc++-v3/doc/xml/manual/using_exceptions.xml
> index 32bff85549f..d17a915f237 100644
> --- a/libstdc++-v3/doc/xml/manual/using_exceptions.xml
> +++ b/libstdc++-v3/doc/xml/manual/using_exceptions.xml
> @@ -188,6 +188,54 @@ exception neutrality and exception safety.
>
> </section>
>
> +<section xml:id="intro.using.exception.alloc" xreflabel="Memory allocation for exceptions"><info><title>Memory allocation</title></info>
> +
> + <para>
> + When the program throws an exception the runtime will obtain storage for
> + a <code>__cxa_exception</code> header and the thrown object itself.
> + Libstdc++ will try to use <code>malloc</code> to obtain storage,
> + but provides an emergency buffer to be used if malloc fails,
> + as described by the <link xmlns:xlink="http://www.w3.org/1999/xlink"
> + xlink:href="https://itanium-cxx-abi.github.io/cxx-abi/abi-eh.html#imp-emergency">Itanium
> + exception handling ABI</link>.
> + </para>
> +
> + <para>
> + Contrary to the ABI, the libstdc++ emergency buffer is not always 64kB,
> + and does not always allocate 1kB chunks. The buffer is used as a pool for
> + variable-sized allocations, so that it doesn't waste space for smaller
> + exception objects, such as <code>std::bad_alloc</code>.
> + The total size of the buffer is scaled appropriately for the target,
> + specifically it depends on <code>sizeof(void*)</code>, so that a 64-bit
> + system uses a larger pool than a 32-bit system. This is done because for
> + 32-bit systems the exception objects (and the exception header) require
> + less space, and core counts and thread counts are typically lower as well.
> + </para>
> +
> + <para>
> + By default, libstdc++ will use <code>malloc</code> to allocate the buffer
> + on program startup.
> + <xref linkend="manual.intro.setup.configure"/> libstdc++ with the
> + <code>--enable-libstdcxx-static-eh-pool</code> option will make it
> + use a static buffer instead of using <code>malloc</code>.
> + The default buffer size is chosen automatically, but can be overridden
> + by configuring with <code>--with-libstdcxx-eh-pool-obj-count=NUM</code>,
> + where NUM is the number of simultaneous allocations that should be
> + supported. The size of the pool will be sufficient for NUM exceptions
> + of <code>6 * sizeof(void*)</code> bytes, plus another NUM exceptions
> + captured in <classname>std::exception_ptr</classname> and rethrown using
> + <code>std::rethrow_exception</code>. That default size applies whether
> + the buffer is reserved as static storage or is allocated dynamically.
> + For a dynamic buffer, the default size can also be changed at runtime,
> + per-process, via the <code>GLIBCXX_TUNABLES</code> environment variable.
> + </para>
> +
> + <para>
> + TODO document GLIBCXX_TUNABLES
> + </para>
> +
> +</section>
> +
> <section xml:id="intro.using.exception.no" xreflabel="-fno-exceptions"><info><title>Doing without</title></info>
>
> <para>
> diff --git a/libstdc++-v3/libsupc++/Makefile.am b/libstdc++-v3/libsupc++/Makefile.am
> index 65b5c1a87fd..cde9fbdb1eb 100644
> --- a/libstdc++-v3/libsupc++/Makefile.am
> +++ b/libstdc++-v3/libsupc++/Makefile.am
> @@ -139,7 +139,7 @@ atomicity.cc: ${atomicity_file}
> # as the occasion call for it.
> AM_CXXFLAGS = \
> $(glibcxx_lt_pic_flag) $(glibcxx_compiler_shared_flag) \
> - $(XTEMPLATE_FLAGS) $(FREESTANDING_FLAGS) \
> + $(XTEMPLATE_FLAGS) $(FREESTANDING_FLAGS) $(EH_POOL_FLAGS) \
> $(WARN_CXXFLAGS) $(OPTIMIZE_CXXFLAGS) $(CONFIG_CXXFLAGS)
>
> AM_MAKEFLAGS = \
> diff --git a/libstdc++-v3/libsupc++/eh_alloc.cc b/libstdc++-v3/libsupc++/eh_alloc.cc
> index 68f319869f9..32378d586ad 100644
> --- a/libstdc++-v3/libsupc++/eh_alloc.cc
> +++ b/libstdc++-v3/libsupc++/eh_alloc.cc
> @@ -25,16 +25,45 @@
> // This is derived from the C++ ABI for IA-64. Where we diverge
> // for cross-architecture compatibility are noted with "@@@".
>
> -#include <bits/c++config.h>
> -#include <cstdlib>
> -#if _GLIBCXX_HOSTED
> -#include <cstring>
> -#endif
> -#include <climits>
> -#include <exception>
> +#include <exception> // std::exception
> +#include <new> // std::terminate
> +#include <cstdlib> // std::malloc, std::free, std::strtoul
> +#include <climits> // INT_MAX
> +#include <bits/stl_function.h> // std::less
> #include "unwind-cxx.h"
> -#include <ext/concurrence.h>
> -#include <new>
> +#if _GLIBCXX_HOSTED
> +# include <string_view> // std::string_view
> +# include <cstring> // std::strchr, std::memset
> +# include <ext/concurrence.h> // __gnu_cxx::__mutex, __gnu_cxx::__scoped_lock
> +#endif
> +
> +// We use an emergency buffer used for exceptions when malloc fails.
> +// If _GLIBCXX_EH_POOL_STATIC is defined (e.g. by configure) then we use
> +// a fixed-size static buffer. Otherwise, allocate on startup using malloc.
> +//
> +// The size of the buffer is N * (S * P + R + D), where:
> +// N == The number of objects to reserve space for.
> +// Defaults to EMERGENCY_OBJ_COUNT, defined below.
> +// S == Estimated size of exception objects to account for.
> +// This size is in units of sizeof(void*) not bytes.
> +// Defaults to EMERGENCY_OBJ_SIZE, defined below.
> +// P == sizeof(void*).
> +// R == sizeof(__cxa_refcounted_exception).
> +// D == sizeof(__cxa_dependent_exception).
> +//
> +// This provides space for N thrown exceptions of S words each, and an
> +// additional N dependent exceptions from std::rethrow_exception.
> +//
> +// The calculation allows values of N and S to be target-independent,
> +// as the size will be scaled by the size of basic types on the target,
> +// and space for the C++ exception header (__cxa_refcounted_exception)
> +// is added automatically.
> +//
> +// For a dynamically allocated buffer, N and S can be set from the environment.
> +// Setting N=0 will disable the emergency buffer.
> +// The GLIBCXX_TUNABLES environment variable will be checked for the following:
> +// - Tunable glibcxx.eh_pool.obj_count overrides EMERGENCY_OBJ_COUNT.
> +// - Tunable glibcxx.eh_pool.obj_size overrides EMERGENCY_OBJ_SIZE.
>
> #if _GLIBCXX_HOSTED
> using std::free;
> @@ -50,46 +79,61 @@ extern "C" void *memset (void *, int, std::size_t);
>
> using namespace __cxxabiv1;
>
> -// ??? How to control these parameters.
> +// Assume that 6 * sizeof(void*) is a reasonable exception object size.
> +// Throwing very many large objects will exhaust the pool more quickly.
> +// N.B. sizeof(std::bad_alloc) == sizeof(void*)
> +// and sizeof(std::runtime_error) == 2 * sizeof(void*)
> +// and sizeof(std::system_error) == 4 * sizeof(void*).
> +#define EMERGENCY_OBJ_SIZE 6
>
> -// Guess from the size of basic types how large a buffer is reasonable.
> -// Note that the basic c++ exception header has 13 pointers and 2 ints,
> -// so on a system with PSImode pointers we're talking about 56 bytes
> -// just for overhead.
> -
> -#if INT_MAX == 32767
> -# define EMERGENCY_OBJ_SIZE 128
> -# define EMERGENCY_OBJ_COUNT 16
> -#elif !defined (_GLIBCXX_LLP64) && LONG_MAX == 2147483647
> -# define EMERGENCY_OBJ_SIZE 512
> -# define EMERGENCY_OBJ_COUNT 32
> +#ifdef __GTHREADS
> +// Assume that the number of concurrent exception objects scales with the
> +// processor word size, i.e., 16-bit systems are not likely to have hundreds
> +// or threads all simultaneously throwing on OOM conditions.
> +# define EMERGENCY_OBJ_COUNT (8 * __SIZEOF_POINTER__)
> +# define MAX_OBJ_COUNT (16 << __SIZEOF_POINTER__)
> #else
> -# define EMERGENCY_OBJ_SIZE 1024
> -# define EMERGENCY_OBJ_COUNT 64
> +# define EMERGENCY_OBJ_COUNT 4
> +# define MAX_OBJ_COUNT 64
> #endif
>
> -#ifndef __GTHREADS
> -# undef EMERGENCY_OBJ_COUNT
> -# define EMERGENCY_OBJ_COUNT 4
> +// This can be set by configure.
> +#ifdef _GLIBCXX_EH_POOL_NOBJS
> +# if _GLIBCXX_EH_POOL_NOBJS > MAX_OBJ_COUNT
> +# warning "_GLIBCXX_EH_POOL_NOBJS value is too large; ignoring it"
> +# else
> +# undef EMERGENCY_OBJ_COUNT
> +# define EMERGENCY_OBJ_COUNT _GLIBCXX_EH_POOL_NOBJS
> +# endif
> #endif
>
> namespace __gnu_cxx
> {
> - void __freeres();
> + void __freeres() noexcept;
> }
>
> namespace
> {
> + static constexpr std::size_t
> + buffer_size_in_bytes(std::size_t obj_count, std::size_t obj_size) noexcept
> + {
> + // N * (S * P + R + D)
> + constexpr std::size_t P = sizeof(void*);
> + constexpr std::size_t R = sizeof(__cxa_refcounted_exception);
> + constexpr std::size_t D = sizeof(__cxa_dependent_exception);
> + return obj_count * (obj_size * P + R + D);
> + }
> +
> // A fixed-size heap, variable size object allocator
> class pool
> {
> public:
> - pool();
> + pool() noexcept;
>
> - _GLIBCXX_NODISCARD void *allocate (std::size_t);
> - void free (void *);
> + _GLIBCXX_NODISCARD void *allocate (std::size_t) noexcept;
> + void free (void *) noexcept;
>
> - bool in_pool (void *);
> + bool in_pool (void *) const noexcept;
>
> private:
> struct free_entry {
> @@ -101,33 +145,87 @@ namespace
> char data[] __attribute__((aligned));
> };
>
> +#ifdef __GTHREADS
> // A single mutex controlling emergency allocations.
> __gnu_cxx::__mutex emergency_mutex;
> + using __scoped_lock = __gnu_cxx::__scoped_lock;
> +#else
> + int emergency_mutex = 0;
> + struct __scoped_lock { explicit __scoped_lock(int) { } };
> +#endif
>
> // The free-list
> - free_entry *first_free_entry;
> + free_entry *first_free_entry = nullptr;
> // The arena itself - we need to keep track of these only
> // to implement in_pool.
> - char *arena;
> - std::size_t arena_size;
> +#ifdef _GLIBCXX_EH_POOL_STATIC
> + static constexpr std::size_t arena_size
> + = buffer_size_in_bytes(EMERGENCY_OBJ_COUNT, EMERGENCY_OBJ_SIZE);
> + alignas(void*) char arena[std::max(arena_size, sizeof(free_entry))];
> +#else
> + char *arena = nullptr;
> + std::size_t arena_size = 0;
> +#endif
>
> - friend void __gnu_cxx::__freeres();
> + friend void __gnu_cxx::__freeres() noexcept;
> };
>
> - pool::pool()
> + pool::pool() noexcept
> {
> - // Allocate the arena - we could add a GLIBCXX_EH_ARENA_SIZE environment
> - // to make this tunable.
> - arena_size = (EMERGENCY_OBJ_SIZE * EMERGENCY_OBJ_COUNT
> - + EMERGENCY_OBJ_COUNT * sizeof (__cxa_dependent_exception));
> +#ifndef _GLIBCXX_EH_POOL_STATIC
> + int obj_size = EMERGENCY_OBJ_SIZE;
> + int obj_count = EMERGENCY_OBJ_COUNT;
> +
> +#if _GLIBCXX_HOSTED
> +#if _GLIBCXX_HAVE_SECURE_GETENV
> + const char* str = ::secure_getenv("GLIBCXX_TUNABLES");
> +#else
> + const char* str = std::getenv("GLIBCXX_TUNABLES");
> +#endif
> + const std::string_view ns_name = "glibcxx.eh_pool";
> + std::pair<std::string_view, int> tunables[]{
> + {"obj_size", 0}, {"obj_count", obj_count}
> + };
> + while (str)
> + {
> + if (*str == ':')
> + ++str;
> +
> + if (!ns_name.compare(0, ns_name.size(), str, ns_name.size())
> + && str[ns_name.size()] == '.')
> + {
> + str += ns_name.size() + 1;
> + for (auto& t : tunables)
> + if (!t.first.compare(0, t.first.size(), str, t.first.size())
> + && str[t.first.size()] == '=')
> + {
> + str += t.first.size() + 1;
> + char* end;
> + unsigned long val = strtoul(str, &end, 0);
> + if ((*end == ':' || *end == '\0') && val <= INT_MAX)
> + t.second = val;
> + str = end;
> + break;
> + }
> + }
> + str = strchr(str, ':');
> + }
> + obj_count = std::min(tunables[1].second, MAX_OBJ_COUNT); // Can be zero.
> + if (tunables[0].second != 0)
> + obj_size = tunables[0].second;
> +#endif // HOSTED
> +
> + arena_size = buffer_size_in_bytes(obj_count, obj_size);
> + if (arena_size == 0)
> + return;
> arena = (char *)malloc (arena_size);
> if (!arena)
> {
> // If the allocation failed go without an emergency pool.
> arena_size = 0;
> - first_free_entry = NULL;
> return;
> }
> +#endif // STATIC
>
> // Populate the free-list with a single entry covering the whole arena
> first_free_entry = reinterpret_cast <free_entry *> (arena);
> @@ -136,7 +234,7 @@ namespace
> first_free_entry->next = NULL;
> }
>
> - void *pool::allocate (std::size_t size)
> + void *pool::allocate (std::size_t size) noexcept
> {
> __gnu_cxx::__scoped_lock sentry(emergency_mutex);
> // We need an additional size_t member plus the padding to
> @@ -188,9 +286,9 @@ namespace
> return &x->data;
> }
>
> - void pool::free (void *data)
> + void pool::free (void *data) noexcept
> {
> - __gnu_cxx::__scoped_lock sentry(emergency_mutex);
> + __scoped_lock sentry(emergency_mutex);
> allocated_entry *e = reinterpret_cast <allocated_entry *>
> (reinterpret_cast <char *> (data) - offsetof (allocated_entry, data));
> std::size_t sz = e->size;
> @@ -252,11 +350,10 @@ namespace
> }
> }
>
> - bool pool::in_pool (void *ptr)
> + inline bool pool::in_pool (void *ptr) const noexcept
> {
> - char *p = reinterpret_cast <char *> (ptr);
> - return (p > arena
> - && p < arena + arena_size);
> + std::less<const void*> less;
> + return less(ptr, arena + arena_size) && less(arena, ptr);
> }
>
> pool emergency_pool;
> @@ -264,29 +361,32 @@ namespace
>
> namespace __gnu_cxx
> {
> + __attribute__((cold))
> void
> - __freeres()
> + __freeres() noexcept
> {
> +#ifndef _GLIBCXX_EH_POOL_STATIC
> if (emergency_pool.arena)
> {
> ::free(emergency_pool.arena);
> emergency_pool.arena = 0;
> }
> +#endif
> }
> }
>
> extern "C" void *
> -__cxxabiv1::__cxa_allocate_exception(std::size_t thrown_size) _GLIBCXX_NOTHROW
> +__cxxabiv1::__cxa_allocate_exception(std::size_t thrown_size) noexcept
> {
> void *ret;
>
> thrown_size += sizeof (__cxa_refcounted_exception);
> ret = malloc (thrown_size);
>
> - if (!ret)
> + if (!ret) [[__unlikely__]]
> ret = emergency_pool.allocate (thrown_size);
>
> - if (!ret)
> + if (!ret) [[__unlikely__]]
> std::terminate ();
>
> memset (ret, 0, sizeof (__cxa_refcounted_exception));
> @@ -296,10 +396,10 @@ __cxxabiv1::__cxa_allocate_exception(std::size_t thrown_size) _GLIBCXX_NOTHROW
>
>
> extern "C" void
> -__cxxabiv1::__cxa_free_exception(void *vptr) _GLIBCXX_NOTHROW
> +__cxxabiv1::__cxa_free_exception(void *vptr) noexcept
> {
> char *ptr = (char *) vptr - sizeof (__cxa_refcounted_exception);
> - if (emergency_pool.in_pool (ptr))
> + if (emergency_pool.in_pool (ptr)) [[__unlikely__]]
> emergency_pool.free (ptr);
> else
> free (ptr);
> @@ -307,31 +407,29 @@ __cxxabiv1::__cxa_free_exception(void *vptr) _GLIBCXX_NOTHROW
>
>
> extern "C" __cxa_dependent_exception*
> -__cxxabiv1::__cxa_allocate_dependent_exception() _GLIBCXX_NOTHROW
> +__cxxabiv1::__cxa_allocate_dependent_exception() noexcept
> {
> - __cxa_dependent_exception *ret;
> + void *ret;
>
> - ret = static_cast<__cxa_dependent_exception*>
> - (malloc (sizeof (__cxa_dependent_exception)));
> + ret = malloc (sizeof (__cxa_dependent_exception));
>
> - if (!ret)
> - ret = static_cast <__cxa_dependent_exception*>
> - (emergency_pool.allocate (sizeof (__cxa_dependent_exception)));
> + if (!ret) [[__unlikely__]]
> + ret = emergency_pool.allocate (sizeof (__cxa_dependent_exception));
>
> if (!ret)
> std::terminate ();
>
> memset (ret, 0, sizeof (__cxa_dependent_exception));
>
> - return ret;
> + return static_cast<__cxa_dependent_exception*>(ret);
> }
>
>
> extern "C" void
> __cxxabiv1::__cxa_free_dependent_exception
> - (__cxa_dependent_exception *vptr) _GLIBCXX_NOTHROW
> + (__cxa_dependent_exception *vptr) noexcept
> {
> - if (emergency_pool.in_pool (vptr))
> + if (emergency_pool.in_pool (vptr)) [[__unlikely__]]
> emergency_pool.free (vptr);
> else
> free (vptr);
>
On Mon, 10 Oct 2022 at 07:18, Richard Biener <richard.guenther@gmail.com> wrote:
>
> On Fri, Oct 7, 2022 at 5:55 PM Jonathan Wakely via Gcc-patches
> <gcc-patches@gcc.gnu.org> wrote:
> >
> > This needs a little more documentation (see the TODO in the manual),
> > rather than just the comments in the source. This isn't final, but I
> > think it's the direction I want to take.
> >
> > -- >8 --
> >
> > Implement a long-standing request to support tuning the size of the
> > emergency buffer for allocating exceptions after malloc fails, or to
> > disable that buffer entirely.
> >
> > It's now possible to disable the dynamic allocation of the buffer and
> > use a fixed-size static buffer, via --enable-libstdcxx-static-eh-pool.
> > This is a built-time choice that is baked into libstdc++ and so affects
> > all code linked against that build of libstdc++.
> >
> > The size of the pool can be set by --with-libstdcxx-eh-pool-obj-count=N
> > which is measured in units of sizeof(void*) not bytes. A given exception
> > type such as std::system_error depends on the target, so giving a size
> > in bytes wouldn't be portable across 16/32/64-bit targets.
> >
> > When libstdc++ is configured to use a dynamic buffer, the size of that
> > buffer can now be tuned at runtime by setting the GLIBCXX_TUNABLES
> > environment variable (c.f. PR libstdc++/88264). The number of exceptions
> > to reserve space for is controlled by the "glibcxx.eh_pool.obj_count"
> > and "glibcxx.eh_pool.obj_size" tunables. The pool will be sized to be
> > able to allocate obj_count exceptions of size obj_size*sizeof(void*) and
> > obj_count "dependent" exceptions rethrown by std::rethrow_exception.
> >
> > With the ability to tune the buffer size, we can reduce the default pool
> > size. Most users never need to throw 1kB exceptions in parallel from
> > hundreds of threads after malloc is OOM.
>
> But does it hurt? Back in time when I reworked the allocator to be less
> wasteful the whole point was to allow more exceptions to be in-flight
> during OOM shutdown of a process with many threads.
It certainly hurts for small systems, but maybe we can keep the large
allocation for 64-bit targets (currently 73kB) and only reduce it for
32-bit (19kB) and 16-bit (3kB IIRC) targets.
And obviously if the new code to check an env var is backported, the
defaults shouldn't be changed in the release branch. Just enable the
new feature, but leave defaults the same.
N.B. the C++ ABI actually requires that the emergency pool should
block if too many threads try to access it at once. That would mean
the program slows down drastically, but doesn't empty the pool and
terminate if there are many threads all trying to throw on OOM. I'm
not convinced blocking is the right default, but making it an option
seems reasonable (I created PR107180 to track that).
> So if we reduce the default buffer size that should be documented
> in changes.html, maybe with a hint how to restore the old buffer size
> (configury flags required or runtime ENV setting)?
The git commit message gives the env setting to do that, and
acinclude.m4 gives the configure option to do that. I can certainly
add the info to changes.html where it's more likely to be noticed.
>
> Otherwise looks OK to me.
Great, thanks for taking a look.
On Mon, 10 Oct 2022 at 12:17, Jonathan Wakely <jwakely@redhat.com> wrote:
>
> On Mon, 10 Oct 2022 at 07:18, Richard Biener <richard.guenther@gmail.com> wrote:
> >
> > On Fri, Oct 7, 2022 at 5:55 PM Jonathan Wakely via Gcc-patches
> > <gcc-patches@gcc.gnu.org> wrote:
> > >
> > > This needs a little more documentation (see the TODO in the manual),
> > > rather than just the comments in the source. This isn't final, but I
> > > think it's the direction I want to take.
> > >
> > > -- >8 --
> > >
> > > Implement a long-standing request to support tuning the size of the
> > > emergency buffer for allocating exceptions after malloc fails, or to
> > > disable that buffer entirely.
> > >
> > > It's now possible to disable the dynamic allocation of the buffer and
> > > use a fixed-size static buffer, via --enable-libstdcxx-static-eh-pool.
> > > This is a built-time choice that is baked into libstdc++ and so affects
> > > all code linked against that build of libstdc++.
> > >
> > > The size of the pool can be set by --with-libstdcxx-eh-pool-obj-count=N
> > > which is measured in units of sizeof(void*) not bytes. A given exception
> > > type such as std::system_error depends on the target, so giving a size
> > > in bytes wouldn't be portable across 16/32/64-bit targets.
> > >
> > > When libstdc++ is configured to use a dynamic buffer, the size of that
> > > buffer can now be tuned at runtime by setting the GLIBCXX_TUNABLES
> > > environment variable (c.f. PR libstdc++/88264). The number of exceptions
> > > to reserve space for is controlled by the "glibcxx.eh_pool.obj_count"
> > > and "glibcxx.eh_pool.obj_size" tunables. The pool will be sized to be
> > > able to allocate obj_count exceptions of size obj_size*sizeof(void*) and
> > > obj_count "dependent" exceptions rethrown by std::rethrow_exception.
> > >
> > > With the ability to tune the buffer size, we can reduce the default pool
> > > size. Most users never need to throw 1kB exceptions in parallel from
> > > hundreds of threads after malloc is OOM.
> >
> > But does it hurt? Back in time when I reworked the allocator to be less
> > wasteful the whole point was to allow more exceptions to be in-flight
> > during OOM shutdown of a process with many threads.
>
> It certainly hurts for small systems, but maybe we can keep the large
> allocation for 64-bit targets (currently 73kB) and only reduce it for
> 32-bit (19kB) and 16-bit (3kB IIRC) targets.
Maybe this incremental diff would be an improvement:
@@ -90,7 +90,7 @@ using namespace __cxxabiv1;
// Assume that the number of concurrent exception objects scales with the
// processor word size, i.e., 16-bit systems are not likely to have hundreds
// of threads all simultaneously throwing on OOM conditions.
-# define EMERGENCY_OBJ_COUNT (8 * __SIZEOF_POINTER__)
+# define EMERGENCY_OBJ_COUNT (4 * __SIZEOF_POINTER__ * __SIZEOF_POINTER__)
# define MAX_OBJ_COUNT (16 << __SIZEOF_POINTER__)
#else
# define EMERGENCY_OBJ_COUNT 4
This makes it quadratic in the word size, so on 64-bit targets we'd
have space for 256 "reasonable size" exceptions (and twice as many
single word exceptions like std::bad_alloc), but only 64 on 32-bit
targets, and only 16 on 16-bit ones.
This slightly increases the initial allocation on x86_64 from 72,704
bytes to 73,728 bytes, but reduces 32-bit from 18,944 bytes to 12,800.
If more is needed, it can be chosen via configure or the environment.
On Mon, Oct 10, 2022 at 5:10 PM Jonathan Wakely <jwakely@redhat.com> wrote:
>
> On Mon, 10 Oct 2022 at 12:17, Jonathan Wakely <jwakely@redhat.com> wrote:
> >
> > On Mon, 10 Oct 2022 at 07:18, Richard Biener <richard.guenther@gmail.com> wrote:
> > >
> > > On Fri, Oct 7, 2022 at 5:55 PM Jonathan Wakely via Gcc-patches
> > > <gcc-patches@gcc.gnu.org> wrote:
> > > >
> > > > This needs a little more documentation (see the TODO in the manual),
> > > > rather than just the comments in the source. This isn't final, but I
> > > > think it's the direction I want to take.
> > > >
> > > > -- >8 --
> > > >
> > > > Implement a long-standing request to support tuning the size of the
> > > > emergency buffer for allocating exceptions after malloc fails, or to
> > > > disable that buffer entirely.
> > > >
> > > > It's now possible to disable the dynamic allocation of the buffer and
> > > > use a fixed-size static buffer, via --enable-libstdcxx-static-eh-pool.
> > > > This is a built-time choice that is baked into libstdc++ and so affects
> > > > all code linked against that build of libstdc++.
> > > >
> > > > The size of the pool can be set by --with-libstdcxx-eh-pool-obj-count=N
> > > > which is measured in units of sizeof(void*) not bytes. A given exception
> > > > type such as std::system_error depends on the target, so giving a size
> > > > in bytes wouldn't be portable across 16/32/64-bit targets.
> > > >
> > > > When libstdc++ is configured to use a dynamic buffer, the size of that
> > > > buffer can now be tuned at runtime by setting the GLIBCXX_TUNABLES
> > > > environment variable (c.f. PR libstdc++/88264). The number of exceptions
> > > > to reserve space for is controlled by the "glibcxx.eh_pool.obj_count"
> > > > and "glibcxx.eh_pool.obj_size" tunables. The pool will be sized to be
> > > > able to allocate obj_count exceptions of size obj_size*sizeof(void*) and
> > > > obj_count "dependent" exceptions rethrown by std::rethrow_exception.
> > > >
> > > > With the ability to tune the buffer size, we can reduce the default pool
> > > > size. Most users never need to throw 1kB exceptions in parallel from
> > > > hundreds of threads after malloc is OOM.
> > >
> > > But does it hurt? Back in time when I reworked the allocator to be less
> > > wasteful the whole point was to allow more exceptions to be in-flight
> > > during OOM shutdown of a process with many threads.
> >
> > It certainly hurts for small systems, but maybe we can keep the large
> > allocation for 64-bit targets (currently 73kB) and only reduce it for
> > 32-bit (19kB) and 16-bit (3kB IIRC) targets.
>
> Maybe this incremental diff would be an improvement:
>
> @@ -90,7 +90,7 @@ using namespace __cxxabiv1;
> // Assume that the number of concurrent exception objects scales with the
> // processor word size, i.e., 16-bit systems are not likely to have hundreds
> // of threads all simultaneously throwing on OOM conditions.
> -# define EMERGENCY_OBJ_COUNT (8 * __SIZEOF_POINTER__)
> +# define EMERGENCY_OBJ_COUNT (4 * __SIZEOF_POINTER__ * __SIZEOF_POINTER__)
> # define MAX_OBJ_COUNT (16 << __SIZEOF_POINTER__)
> #else
> # define EMERGENCY_OBJ_COUNT 4
>
> This makes it quadratic in the word size, so on 64-bit targets we'd
> have space for 256 "reasonable size" exceptions (and twice as many
> single word exceptions like std::bad_alloc), but only 64 on 32-bit
> targets, and only 16 on 16-bit ones.
So can we then commonize some of the #defines by using sizeof(void *)
(taking pointer size as word size?)
>
> This slightly increases the initial allocation on x86_64 from 72,704
> bytes to 73,728 bytes, but reduces 32-bit from 18,944 bytes to 12,800.
> If more is needed, it can be chosen via configure or the environment.
>
On Tue, 11 Oct 2022 at 07:41, Richard Biener <richard.guenther@gmail.com> wrote:
>
> On Mon, Oct 10, 2022 at 5:10 PM Jonathan Wakely <jwakely@redhat.com> wrote:
> >
> > On Mon, 10 Oct 2022 at 12:17, Jonathan Wakely <jwakely@redhat.com> wrote:
> > >
> > > On Mon, 10 Oct 2022 at 07:18, Richard Biener <richard.guenther@gmail.com> wrote:
> > > >
> > > > On Fri, Oct 7, 2022 at 5:55 PM Jonathan Wakely via Gcc-patches
> > > > <gcc-patches@gcc.gnu.org> wrote:
> > > > >
> > > > > This needs a little more documentation (see the TODO in the manual),
> > > > > rather than just the comments in the source. This isn't final, but I
> > > > > think it's the direction I want to take.
> > > > >
> > > > > -- >8 --
> > > > >
> > > > > Implement a long-standing request to support tuning the size of the
> > > > > emergency buffer for allocating exceptions after malloc fails, or to
> > > > > disable that buffer entirely.
> > > > >
> > > > > It's now possible to disable the dynamic allocation of the buffer and
> > > > > use a fixed-size static buffer, via --enable-libstdcxx-static-eh-pool.
> > > > > This is a built-time choice that is baked into libstdc++ and so affects
> > > > > all code linked against that build of libstdc++.
> > > > >
> > > > > The size of the pool can be set by --with-libstdcxx-eh-pool-obj-count=N
> > > > > which is measured in units of sizeof(void*) not bytes. A given exception
> > > > > type such as std::system_error depends on the target, so giving a size
> > > > > in bytes wouldn't be portable across 16/32/64-bit targets.
> > > > >
> > > > > When libstdc++ is configured to use a dynamic buffer, the size of that
> > > > > buffer can now be tuned at runtime by setting the GLIBCXX_TUNABLES
> > > > > environment variable (c.f. PR libstdc++/88264). The number of exceptions
> > > > > to reserve space for is controlled by the "glibcxx.eh_pool.obj_count"
> > > > > and "glibcxx.eh_pool.obj_size" tunables. The pool will be sized to be
> > > > > able to allocate obj_count exceptions of size obj_size*sizeof(void*) and
> > > > > obj_count "dependent" exceptions rethrown by std::rethrow_exception.
> > > > >
> > > > > With the ability to tune the buffer size, we can reduce the default pool
> > > > > size. Most users never need to throw 1kB exceptions in parallel from
> > > > > hundreds of threads after malloc is OOM.
> > > >
> > > > But does it hurt? Back in time when I reworked the allocator to be less
> > > > wasteful the whole point was to allow more exceptions to be in-flight
> > > > during OOM shutdown of a process with many threads.
> > >
> > > It certainly hurts for small systems, but maybe we can keep the large
> > > allocation for 64-bit targets (currently 73kB) and only reduce it for
> > > 32-bit (19kB) and 16-bit (3kB IIRC) targets.
> >
> > Maybe this incremental diff would be an improvement:
> >
> > @@ -90,7 +90,7 @@ using namespace __cxxabiv1;
> > // Assume that the number of concurrent exception objects scales with the
> > // processor word size, i.e., 16-bit systems are not likely to have hundreds
> > // of threads all simultaneously throwing on OOM conditions.
> > -# define EMERGENCY_OBJ_COUNT (8 * __SIZEOF_POINTER__)
> > +# define EMERGENCY_OBJ_COUNT (4 * __SIZEOF_POINTER__ * __SIZEOF_POINTER__)
> > # define MAX_OBJ_COUNT (16 << __SIZEOF_POINTER__)
> > #else
> > # define EMERGENCY_OBJ_COUNT 4
> >
> > This makes it quadratic in the word size, so on 64-bit targets we'd
> > have space for 256 "reasonable size" exceptions (and twice as many
> > single word exceptions like std::bad_alloc), but only 64 on 32-bit
> > targets, and only 16 on 16-bit ones.
>
> So can we then commonize some of the #defines by using sizeof(void *)
> (taking pointer size as word size?)
What did you have in mind? Do you mean use sizeof(void*) instead of
the SIZEOF macro?
MAX_OBJ_COUNT uses the SIZEOF macro so it can be used in a
preprocessor condition:
#ifdef _GLIBCXX_EH_POOL_NOBJS
# if _GLIBCXX_EH_POOL_NOBJS > MAX_OBJ_COUNT
On Tue, Oct 11, 2022 at 1:06 PM Jonathan Wakely <jwakely@redhat.com> wrote:
>
> On Tue, 11 Oct 2022 at 07:41, Richard Biener <richard.guenther@gmail.com> wrote:
> >
> > On Mon, Oct 10, 2022 at 5:10 PM Jonathan Wakely <jwakely@redhat.com> wrote:
> > >
> > > On Mon, 10 Oct 2022 at 12:17, Jonathan Wakely <jwakely@redhat.com> wrote:
> > > >
> > > > On Mon, 10 Oct 2022 at 07:18, Richard Biener <richard.guenther@gmail.com> wrote:
> > > > >
> > > > > On Fri, Oct 7, 2022 at 5:55 PM Jonathan Wakely via Gcc-patches
> > > > > <gcc-patches@gcc.gnu.org> wrote:
> > > > > >
> > > > > > This needs a little more documentation (see the TODO in the manual),
> > > > > > rather than just the comments in the source. This isn't final, but I
> > > > > > think it's the direction I want to take.
> > > > > >
> > > > > > -- >8 --
> > > > > >
> > > > > > Implement a long-standing request to support tuning the size of the
> > > > > > emergency buffer for allocating exceptions after malloc fails, or to
> > > > > > disable that buffer entirely.
> > > > > >
> > > > > > It's now possible to disable the dynamic allocation of the buffer and
> > > > > > use a fixed-size static buffer, via --enable-libstdcxx-static-eh-pool.
> > > > > > This is a built-time choice that is baked into libstdc++ and so affects
> > > > > > all code linked against that build of libstdc++.
> > > > > >
> > > > > > The size of the pool can be set by --with-libstdcxx-eh-pool-obj-count=N
> > > > > > which is measured in units of sizeof(void*) not bytes. A given exception
> > > > > > type such as std::system_error depends on the target, so giving a size
> > > > > > in bytes wouldn't be portable across 16/32/64-bit targets.
> > > > > >
> > > > > > When libstdc++ is configured to use a dynamic buffer, the size of that
> > > > > > buffer can now be tuned at runtime by setting the GLIBCXX_TUNABLES
> > > > > > environment variable (c.f. PR libstdc++/88264). The number of exceptions
> > > > > > to reserve space for is controlled by the "glibcxx.eh_pool.obj_count"
> > > > > > and "glibcxx.eh_pool.obj_size" tunables. The pool will be sized to be
> > > > > > able to allocate obj_count exceptions of size obj_size*sizeof(void*) and
> > > > > > obj_count "dependent" exceptions rethrown by std::rethrow_exception.
> > > > > >
> > > > > > With the ability to tune the buffer size, we can reduce the default pool
> > > > > > size. Most users never need to throw 1kB exceptions in parallel from
> > > > > > hundreds of threads after malloc is OOM.
> > > > >
> > > > > But does it hurt? Back in time when I reworked the allocator to be less
> > > > > wasteful the whole point was to allow more exceptions to be in-flight
> > > > > during OOM shutdown of a process with many threads.
> > > >
> > > > It certainly hurts for small systems, but maybe we can keep the large
> > > > allocation for 64-bit targets (currently 73kB) and only reduce it for
> > > > 32-bit (19kB) and 16-bit (3kB IIRC) targets.
> > >
> > > Maybe this incremental diff would be an improvement:
> > >
> > > @@ -90,7 +90,7 @@ using namespace __cxxabiv1;
> > > // Assume that the number of concurrent exception objects scales with the
> > > // processor word size, i.e., 16-bit systems are not likely to have hundreds
> > > // of threads all simultaneously throwing on OOM conditions.
> > > -# define EMERGENCY_OBJ_COUNT (8 * __SIZEOF_POINTER__)
> > > +# define EMERGENCY_OBJ_COUNT (4 * __SIZEOF_POINTER__ * __SIZEOF_POINTER__)
> > > # define MAX_OBJ_COUNT (16 << __SIZEOF_POINTER__)
> > > #else
> > > # define EMERGENCY_OBJ_COUNT 4
> > >
> > > This makes it quadratic in the word size, so on 64-bit targets we'd
> > > have space for 256 "reasonable size" exceptions (and twice as many
> > > single word exceptions like std::bad_alloc), but only 64 on 32-bit
> > > targets, and only 16 on 16-bit ones.
> >
> > So can we then commonize some of the #defines by using sizeof(void *)
> > (taking pointer size as word size?)
>
> What did you have in mind? Do you mean use sizeof(void*) instead of
> the SIZEOF macro?
I was just confused and didn't see you commonized EMERGENCY_OBJ_SIZE
already, so ignore my comment.
>
> MAX_OBJ_COUNT uses the SIZEOF macro so it can be used in a
> preprocessor condition:
>
> #ifdef _GLIBCXX_EH_POOL_NOBJS
> # if _GLIBCXX_EH_POOL_NOBJS > MAX_OBJ_COUNT
>
@@ -240,6 +240,7 @@ ECHO_C = @ECHO_C@
ECHO_N = @ECHO_N@
ECHO_T = @ECHO_T@
EGREP = @EGREP@
+EH_POOL_FLAGS = @EH_POOL_FLAGS@
ERROR_CONSTANTS_SRCDIR = @ERROR_CONSTANTS_SRCDIR@
EXEEXT = @EXEEXT@
EXTRA_CFLAGS = @EXTRA_CFLAGS@
@@ -5092,6 +5092,51 @@ BACKTRACE_CPPFLAGS="$BACKTRACE_CPPFLAGS -DBACKTRACE_ELF_SIZE=$elfsize"
GLIBCXX_CONDITIONAL(ENABLE_BACKTRACE, [test "$enable_libstdcxx_backtrace" = yes])
])
+dnl
+dnl Allow the emergency EH pool to be configured.
+dnl
+dnl --enable-libstdcxx-static-eh-pool will cause a fixed-size static buffer
+dnl to be used for allocating exceptions after malloc fails. The default is
+dnl to allocate a buffer using malloc
+dnl
+dnl --with-libstdcxx-eh-pool-obj-count=N will set the default size for the
+dnl buffer. For a static buffer that size is fixed permanently. For a dynamic
+dnl buffer it's the default, but it can be overridden from the environment.
+dnl
+dnl To set the default to approximately the same values as GCC 12,
+dnl use --with-libstdcxx-eh-pool-obj-count=94 for 32-bit targets,
+dnl and --with-libstdcxx-eh-pool-obj-count=252 for 64-bit targets.
+dnl
+dnl Defines:
+dnl _GLIBCXX_EH_POOL_STATIC if a fixed-size static buffer should be used
+dnl instead of allocating a buffer on startup.
+dnl _GLIBCXX_EH_POOL_NOBJS to override the default EMERGENCY_OBJ_COUNT value.
+dnl
+AC_DEFUN([GLIBCXX_EMERGENCY_EH_ALLOC], [
+ eh_pool_static=
+ eh_pool_nobjs=
+ AC_ARG_ENABLE([libstdcxx-static-eh-pool],
+ AC_HELP_STRING([--enable-libstdcxx-static-eh-pool],
+ [use a fixed-size static buffer for allocating exceptions if malloc fails]),
+ [if test "${enableval}" = yes; then
+ eh_pool_static="-D_GLIBCXX_EH_POOL_STATIC"
+ AC_MSG_NOTICE([EH pool using static buffer])
+ fi],)
+
+ AC_ARG_WITH([libstdcxx-eh-pool-obj-count],
+ AC_HELP_STRING([--with-libstdcxx-eh-pool-obj-count],
+ [the number of exceptions that can be allocated from the pool if malloc fails]),
+ [if test "${withval}" -ge 0 2>/dev/null; then
+ eh_pool_obj_count="-D_GLIBCXX_EH_POOL_NOBJS=${withval}"
+ AC_MSG_NOTICE([EH pool object count: ${withval}])
+ else
+ AC_MSG_ERROR([EH pool obj count must be a non-negative integer: $withval])
+ fi],)
+
+ EH_POOL_FLAGS="$eh_pool_static $eh_pool_obj_count"
+ AC_SUBST(EH_POOL_FLAGS)
+])
+
# Macros from the top-level gcc directory.
m4_include([../config/gc++filt.m4])
m4_include([../config/tls.m4])
@@ -538,6 +538,9 @@ GLIBCXX_CHECK_SIZE_T_MANGLING
# Check which release added std::exception_ptr for the target
GLIBCXX_CHECK_EXCEPTION_PTR_SYMVER
+# For libsupc++/eh_alloc.cc defaults.
+GLIBCXX_EMERGENCY_EH_ALLOC
+
# Define documentation rules conditionally.
# See if makeinfo has been installed and is modern enough
@@ -187,6 +187,7 @@ case "${host}" in
AC_CHECK_FUNCS(timespec_get)
AC_CHECK_FUNCS(sockatmark)
AC_CHECK_FUNCS(uselocale)
+ AC_CHECK_FUNCS(secure_getenv)
AM_ICONV
;;
*-mingw32*)
@@ -366,6 +366,11 @@
</para>
</listitem></varlistentry>
+ <varlistentry><term><code>--disable-libstdcxx-hosted</code></term>
+ <listitem>
+ <para>This is an alias for <code>--disable-hosted-libstdcxx</code>.</para>
+ </listitem></varlistentry>
+
<varlistentry><term><code>--disable-libstdcxx-verbose</code></term>
<listitem>
<para>
@@ -446,6 +451,24 @@
</para>
</listitem></varlistentry>
+ <varlistentry><term><code>--enable-libstdcxx-static-eh-pool</code></term>
+ <listitem>
+ <para>Use a fixed-size static buffer for the emergency exception handling
+ pool (see <xref linkend="intro.using.exception.alloc"/>). The default
+ is to allocate the pool on program startup using <code>malloc</code>.
+ With this option, a static buffer will be provided by libstdc++ instead.
+ This does not change the library ABI.
+ </para>
+ </listitem></varlistentry>
+
+ <varlistentry><term><code>--with-libstdcxx-eh-pool-obj-count=NUM</code></term>
+ <listitem>
+ <para>Set the size of the emergency exception handling pool. NUM is the
+ number of simultaneous allocated exceptions to support.
+ This does not change the library ABI.
+ </para>
+ </listitem></varlistentry>
+
</variablelist>
</section>
@@ -188,6 +188,54 @@ exception neutrality and exception safety.
</section>
+<section xml:id="intro.using.exception.alloc" xreflabel="Memory allocation for exceptions"><info><title>Memory allocation</title></info>
+
+ <para>
+ When the program throws an exception the runtime will obtain storage for
+ a <code>__cxa_exception</code> header and the thrown object itself.
+ Libstdc++ will try to use <code>malloc</code> to obtain storage,
+ but provides an emergency buffer to be used if malloc fails,
+ as described by the <link xmlns:xlink="http://www.w3.org/1999/xlink"
+ xlink:href="https://itanium-cxx-abi.github.io/cxx-abi/abi-eh.html#imp-emergency">Itanium
+ exception handling ABI</link>.
+ </para>
+
+ <para>
+ Contrary to the ABI, the libstdc++ emergency buffer is not always 64kB,
+ and does not always allocate 1kB chunks. The buffer is used as a pool for
+ variable-sized allocations, so that it doesn't waste space for smaller
+ exception objects, such as <code>std::bad_alloc</code>.
+ The total size of the buffer is scaled appropriately for the target,
+ specifically it depends on <code>sizeof(void*)</code>, so that a 64-bit
+ system uses a larger pool than a 32-bit system. This is done because for
+ 32-bit systems the exception objects (and the exception header) require
+ less space, and core counts and thread counts are typically lower as well.
+ </para>
+
+ <para>
+ By default, libstdc++ will use <code>malloc</code> to allocate the buffer
+ on program startup.
+ <xref linkend="manual.intro.setup.configure"/> libstdc++ with the
+ <code>--enable-libstdcxx-static-eh-pool</code> option will make it
+ use a static buffer instead of using <code>malloc</code>.
+ The default buffer size is chosen automatically, but can be overridden
+ by configuring with <code>--with-libstdcxx-eh-pool-obj-count=NUM</code>,
+ where NUM is the number of simultaneous allocations that should be
+ supported. The size of the pool will be sufficient for NUM exceptions
+ of <code>6 * sizeof(void*)</code> bytes, plus another NUM exceptions
+ captured in <classname>std::exception_ptr</classname> and rethrown using
+ <code>std::rethrow_exception</code>. That default size applies whether
+ the buffer is reserved as static storage or is allocated dynamically.
+ For a dynamic buffer, the default size can also be changed at runtime,
+ per-process, via the <code>GLIBCXX_TUNABLES</code> environment variable.
+ </para>
+
+ <para>
+ TODO document GLIBCXX_TUNABLES
+ </para>
+
+</section>
+
<section xml:id="intro.using.exception.no" xreflabel="-fno-exceptions"><info><title>Doing without</title></info>
<para>
@@ -139,7 +139,7 @@ atomicity.cc: ${atomicity_file}
# as the occasion call for it.
AM_CXXFLAGS = \
$(glibcxx_lt_pic_flag) $(glibcxx_compiler_shared_flag) \
- $(XTEMPLATE_FLAGS) $(FREESTANDING_FLAGS) \
+ $(XTEMPLATE_FLAGS) $(FREESTANDING_FLAGS) $(EH_POOL_FLAGS) \
$(WARN_CXXFLAGS) $(OPTIMIZE_CXXFLAGS) $(CONFIG_CXXFLAGS)
AM_MAKEFLAGS = \
@@ -25,16 +25,45 @@
// This is derived from the C++ ABI for IA-64. Where we diverge
// for cross-architecture compatibility are noted with "@@@".
-#include <bits/c++config.h>
-#include <cstdlib>
-#if _GLIBCXX_HOSTED
-#include <cstring>
-#endif
-#include <climits>
-#include <exception>
+#include <exception> // std::exception
+#include <new> // std::terminate
+#include <cstdlib> // std::malloc, std::free, std::strtoul
+#include <climits> // INT_MAX
+#include <bits/stl_function.h> // std::less
#include "unwind-cxx.h"
-#include <ext/concurrence.h>
-#include <new>
+#if _GLIBCXX_HOSTED
+# include <string_view> // std::string_view
+# include <cstring> // std::strchr, std::memset
+# include <ext/concurrence.h> // __gnu_cxx::__mutex, __gnu_cxx::__scoped_lock
+#endif
+
+// We use an emergency buffer used for exceptions when malloc fails.
+// If _GLIBCXX_EH_POOL_STATIC is defined (e.g. by configure) then we use
+// a fixed-size static buffer. Otherwise, allocate on startup using malloc.
+//
+// The size of the buffer is N * (S * P + R + D), where:
+// N == The number of objects to reserve space for.
+// Defaults to EMERGENCY_OBJ_COUNT, defined below.
+// S == Estimated size of exception objects to account for.
+// This size is in units of sizeof(void*) not bytes.
+// Defaults to EMERGENCY_OBJ_SIZE, defined below.
+// P == sizeof(void*).
+// R == sizeof(__cxa_refcounted_exception).
+// D == sizeof(__cxa_dependent_exception).
+//
+// This provides space for N thrown exceptions of S words each, and an
+// additional N dependent exceptions from std::rethrow_exception.
+//
+// The calculation allows values of N and S to be target-independent,
+// as the size will be scaled by the size of basic types on the target,
+// and space for the C++ exception header (__cxa_refcounted_exception)
+// is added automatically.
+//
+// For a dynamically allocated buffer, N and S can be set from the environment.
+// Setting N=0 will disable the emergency buffer.
+// The GLIBCXX_TUNABLES environment variable will be checked for the following:
+// - Tunable glibcxx.eh_pool.obj_count overrides EMERGENCY_OBJ_COUNT.
+// - Tunable glibcxx.eh_pool.obj_size overrides EMERGENCY_OBJ_SIZE.
#if _GLIBCXX_HOSTED
using std::free;
@@ -50,46 +79,61 @@ extern "C" void *memset (void *, int, std::size_t);
using namespace __cxxabiv1;
-// ??? How to control these parameters.
+// Assume that 6 * sizeof(void*) is a reasonable exception object size.
+// Throwing very many large objects will exhaust the pool more quickly.
+// N.B. sizeof(std::bad_alloc) == sizeof(void*)
+// and sizeof(std::runtime_error) == 2 * sizeof(void*)
+// and sizeof(std::system_error) == 4 * sizeof(void*).
+#define EMERGENCY_OBJ_SIZE 6
-// Guess from the size of basic types how large a buffer is reasonable.
-// Note that the basic c++ exception header has 13 pointers and 2 ints,
-// so on a system with PSImode pointers we're talking about 56 bytes
-// just for overhead.
-
-#if INT_MAX == 32767
-# define EMERGENCY_OBJ_SIZE 128
-# define EMERGENCY_OBJ_COUNT 16
-#elif !defined (_GLIBCXX_LLP64) && LONG_MAX == 2147483647
-# define EMERGENCY_OBJ_SIZE 512
-# define EMERGENCY_OBJ_COUNT 32
+#ifdef __GTHREADS
+// Assume that the number of concurrent exception objects scales with the
+// processor word size, i.e., 16-bit systems are not likely to have hundreds
+// or threads all simultaneously throwing on OOM conditions.
+# define EMERGENCY_OBJ_COUNT (8 * __SIZEOF_POINTER__)
+# define MAX_OBJ_COUNT (16 << __SIZEOF_POINTER__)
#else
-# define EMERGENCY_OBJ_SIZE 1024
-# define EMERGENCY_OBJ_COUNT 64
+# define EMERGENCY_OBJ_COUNT 4
+# define MAX_OBJ_COUNT 64
#endif
-#ifndef __GTHREADS
-# undef EMERGENCY_OBJ_COUNT
-# define EMERGENCY_OBJ_COUNT 4
+// This can be set by configure.
+#ifdef _GLIBCXX_EH_POOL_NOBJS
+# if _GLIBCXX_EH_POOL_NOBJS > MAX_OBJ_COUNT
+# warning "_GLIBCXX_EH_POOL_NOBJS value is too large; ignoring it"
+# else
+# undef EMERGENCY_OBJ_COUNT
+# define EMERGENCY_OBJ_COUNT _GLIBCXX_EH_POOL_NOBJS
+# endif
#endif
namespace __gnu_cxx
{
- void __freeres();
+ void __freeres() noexcept;
}
namespace
{
+ static constexpr std::size_t
+ buffer_size_in_bytes(std::size_t obj_count, std::size_t obj_size) noexcept
+ {
+ // N * (S * P + R + D)
+ constexpr std::size_t P = sizeof(void*);
+ constexpr std::size_t R = sizeof(__cxa_refcounted_exception);
+ constexpr std::size_t D = sizeof(__cxa_dependent_exception);
+ return obj_count * (obj_size * P + R + D);
+ }
+
// A fixed-size heap, variable size object allocator
class pool
{
public:
- pool();
+ pool() noexcept;
- _GLIBCXX_NODISCARD void *allocate (std::size_t);
- void free (void *);
+ _GLIBCXX_NODISCARD void *allocate (std::size_t) noexcept;
+ void free (void *) noexcept;
- bool in_pool (void *);
+ bool in_pool (void *) const noexcept;
private:
struct free_entry {
@@ -101,33 +145,87 @@ namespace
char data[] __attribute__((aligned));
};
+#ifdef __GTHREADS
// A single mutex controlling emergency allocations.
__gnu_cxx::__mutex emergency_mutex;
+ using __scoped_lock = __gnu_cxx::__scoped_lock;
+#else
+ int emergency_mutex = 0;
+ struct __scoped_lock { explicit __scoped_lock(int) { } };
+#endif
// The free-list
- free_entry *first_free_entry;
+ free_entry *first_free_entry = nullptr;
// The arena itself - we need to keep track of these only
// to implement in_pool.
- char *arena;
- std::size_t arena_size;
+#ifdef _GLIBCXX_EH_POOL_STATIC
+ static constexpr std::size_t arena_size
+ = buffer_size_in_bytes(EMERGENCY_OBJ_COUNT, EMERGENCY_OBJ_SIZE);
+ alignas(void*) char arena[std::max(arena_size, sizeof(free_entry))];
+#else
+ char *arena = nullptr;
+ std::size_t arena_size = 0;
+#endif
- friend void __gnu_cxx::__freeres();
+ friend void __gnu_cxx::__freeres() noexcept;
};
- pool::pool()
+ pool::pool() noexcept
{
- // Allocate the arena - we could add a GLIBCXX_EH_ARENA_SIZE environment
- // to make this tunable.
- arena_size = (EMERGENCY_OBJ_SIZE * EMERGENCY_OBJ_COUNT
- + EMERGENCY_OBJ_COUNT * sizeof (__cxa_dependent_exception));
+#ifndef _GLIBCXX_EH_POOL_STATIC
+ int obj_size = EMERGENCY_OBJ_SIZE;
+ int obj_count = EMERGENCY_OBJ_COUNT;
+
+#if _GLIBCXX_HOSTED
+#if _GLIBCXX_HAVE_SECURE_GETENV
+ const char* str = ::secure_getenv("GLIBCXX_TUNABLES");
+#else
+ const char* str = std::getenv("GLIBCXX_TUNABLES");
+#endif
+ const std::string_view ns_name = "glibcxx.eh_pool";
+ std::pair<std::string_view, int> tunables[]{
+ {"obj_size", 0}, {"obj_count", obj_count}
+ };
+ while (str)
+ {
+ if (*str == ':')
+ ++str;
+
+ if (!ns_name.compare(0, ns_name.size(), str, ns_name.size())
+ && str[ns_name.size()] == '.')
+ {
+ str += ns_name.size() + 1;
+ for (auto& t : tunables)
+ if (!t.first.compare(0, t.first.size(), str, t.first.size())
+ && str[t.first.size()] == '=')
+ {
+ str += t.first.size() + 1;
+ char* end;
+ unsigned long val = strtoul(str, &end, 0);
+ if ((*end == ':' || *end == '\0') && val <= INT_MAX)
+ t.second = val;
+ str = end;
+ break;
+ }
+ }
+ str = strchr(str, ':');
+ }
+ obj_count = std::min(tunables[1].second, MAX_OBJ_COUNT); // Can be zero.
+ if (tunables[0].second != 0)
+ obj_size = tunables[0].second;
+#endif // HOSTED
+
+ arena_size = buffer_size_in_bytes(obj_count, obj_size);
+ if (arena_size == 0)
+ return;
arena = (char *)malloc (arena_size);
if (!arena)
{
// If the allocation failed go without an emergency pool.
arena_size = 0;
- first_free_entry = NULL;
return;
}
+#endif // STATIC
// Populate the free-list with a single entry covering the whole arena
first_free_entry = reinterpret_cast <free_entry *> (arena);
@@ -136,7 +234,7 @@ namespace
first_free_entry->next = NULL;
}
- void *pool::allocate (std::size_t size)
+ void *pool::allocate (std::size_t size) noexcept
{
__gnu_cxx::__scoped_lock sentry(emergency_mutex);
// We need an additional size_t member plus the padding to
@@ -188,9 +286,9 @@ namespace
return &x->data;
}
- void pool::free (void *data)
+ void pool::free (void *data) noexcept
{
- __gnu_cxx::__scoped_lock sentry(emergency_mutex);
+ __scoped_lock sentry(emergency_mutex);
allocated_entry *e = reinterpret_cast <allocated_entry *>
(reinterpret_cast <char *> (data) - offsetof (allocated_entry, data));
std::size_t sz = e->size;
@@ -252,11 +350,10 @@ namespace
}
}
- bool pool::in_pool (void *ptr)
+ inline bool pool::in_pool (void *ptr) const noexcept
{
- char *p = reinterpret_cast <char *> (ptr);
- return (p > arena
- && p < arena + arena_size);
+ std::less<const void*> less;
+ return less(ptr, arena + arena_size) && less(arena, ptr);
}
pool emergency_pool;
@@ -264,29 +361,32 @@ namespace
namespace __gnu_cxx
{
+ __attribute__((cold))
void
- __freeres()
+ __freeres() noexcept
{
+#ifndef _GLIBCXX_EH_POOL_STATIC
if (emergency_pool.arena)
{
::free(emergency_pool.arena);
emergency_pool.arena = 0;
}
+#endif
}
}
extern "C" void *
-__cxxabiv1::__cxa_allocate_exception(std::size_t thrown_size) _GLIBCXX_NOTHROW
+__cxxabiv1::__cxa_allocate_exception(std::size_t thrown_size) noexcept
{
void *ret;
thrown_size += sizeof (__cxa_refcounted_exception);
ret = malloc (thrown_size);
- if (!ret)
+ if (!ret) [[__unlikely__]]
ret = emergency_pool.allocate (thrown_size);
- if (!ret)
+ if (!ret) [[__unlikely__]]
std::terminate ();
memset (ret, 0, sizeof (__cxa_refcounted_exception));
@@ -296,10 +396,10 @@ __cxxabiv1::__cxa_allocate_exception(std::size_t thrown_size) _GLIBCXX_NOTHROW
extern "C" void
-__cxxabiv1::__cxa_free_exception(void *vptr) _GLIBCXX_NOTHROW
+__cxxabiv1::__cxa_free_exception(void *vptr) noexcept
{
char *ptr = (char *) vptr - sizeof (__cxa_refcounted_exception);
- if (emergency_pool.in_pool (ptr))
+ if (emergency_pool.in_pool (ptr)) [[__unlikely__]]
emergency_pool.free (ptr);
else
free (ptr);
@@ -307,31 +407,29 @@ __cxxabiv1::__cxa_free_exception(void *vptr) _GLIBCXX_NOTHROW
extern "C" __cxa_dependent_exception*
-__cxxabiv1::__cxa_allocate_dependent_exception() _GLIBCXX_NOTHROW
+__cxxabiv1::__cxa_allocate_dependent_exception() noexcept
{
- __cxa_dependent_exception *ret;
+ void *ret;
- ret = static_cast<__cxa_dependent_exception*>
- (malloc (sizeof (__cxa_dependent_exception)));
+ ret = malloc (sizeof (__cxa_dependent_exception));
- if (!ret)
- ret = static_cast <__cxa_dependent_exception*>
- (emergency_pool.allocate (sizeof (__cxa_dependent_exception)));
+ if (!ret) [[__unlikely__]]
+ ret = emergency_pool.allocate (sizeof (__cxa_dependent_exception));
if (!ret)
std::terminate ();
memset (ret, 0, sizeof (__cxa_dependent_exception));
- return ret;
+ return static_cast<__cxa_dependent_exception*>(ret);
}
extern "C" void
__cxxabiv1::__cxa_free_dependent_exception
- (__cxa_dependent_exception *vptr) _GLIBCXX_NOTHROW
+ (__cxa_dependent_exception *vptr) noexcept
{
- if (emergency_pool.in_pool (vptr))
+ if (emergency_pool.in_pool (vptr)) [[__unlikely__]]
emergency_pool.free (vptr);
else
free (vptr);