[committed] libstdc++: Fix deadlock in debug iterator increment [PR108288]
Checks
Commit Message
Tested x86_64-linux. Pushed to trunk.
I think we should backport this too, after some soak time on trunk.
-- >8 --
With -fno-elide-constructors the debug iterator post-increment and
post-decrement operators are susceptible to deadlock. They take a mutex
lock and then return a temporary, which also attempts to take a lock to
attach itself to the sequence. If the return value and *this happen to
collide and use the same mutex from the pool, then you get a deadlock
trying to lock a mutex that is already held by the current thread.
The solution is to construct the return value before taking the lock.
The copy constructor and pre-inc/pre-dec operators already manage locks
correctly, without deadlock, so just implement post-inc/post-dec in the
conventional way, taking a copy then modifying *this, then returning the
copy.
libstdc++-v3/ChangeLog:
PR libstdc++/108288
* include/debug/safe_iterator.h (_Safe_iterator::operator++(int))
(_Safe_iterator::operator--(int)): Do not hold lock around
construction of return value.
---
libstdc++-v3/include/debug/safe_iterator.h | 18 ++++++------------
1 file changed, 6 insertions(+), 12 deletions(-)
Comments
Thanks for fixing this.
Here is the extension of the fix to all post-increment/decrement
operators we have on _GLIBCXX_DEBUG iterator.
I prefer to restore somehow previous implementation to continue to have
_GLIBCXX_DEBUG post operators implemented in terms of normal post operators.
I also plan to remove the debug check in the _Safe_iterator constructor
from base iterator to avoid the redundant check we have now. But I need
to make sure first that we are never calling it with an unchecked base
iterator. And it might not be the right moment to do such a change.
libstdc++: Fix deadlock in debug local_iterator increment [PR108288]
Complete fix on all _Safe_iterator post-increment and
post-decrement implementations
and on _Safe_local_iterator.
libstdc++-v3/ChangeLog:
* include/debug/safe_iterator.h
(_Safe_iterator<>::operator++(int)): Extend deadlock fix to
other iterator category.
(_Safe_iterator<>::operator--(int)): Likewise.
* include/debug/safe_local_iterator.h
(_Safe_local_iterator<>::operator++(int)): Fix deadlock.
* testsuite/util/debug/unordered_checks.h
(invalid_local_iterator_pre_increment): New.
(invalid_local_iterator_post_increment): New.
*
testsuite/23_containers/unordered_map/debug/invalid_local_iterator_post_increment_neg.cc:
New test.
*
testsuite/23_containers/unordered_map/debug/invalid_local_iterator_pre_increment_neg.cc:
New test.
Tested under Linux x86_64.
Ok to commit ?
François
On 06/01/23 12:54, Jonathan Wakely via Libstdc++ wrote:
> Tested x86_64-linux. Pushed to trunk.
>
> I think we should backport this too, after some soak time on trunk.
>
> -- >8 --
>
> With -fno-elide-constructors the debug iterator post-increment and
> post-decrement operators are susceptible to deadlock. They take a mutex
> lock and then return a temporary, which also attempts to take a lock to
> attach itself to the sequence. If the return value and *this happen to
Note that the chosen mutex depends on the sequence so there is no need
for conditional sentense here, it will necessarily be the same mutex.
> collide and use the same mutex from the pool, then you get a deadlock
> trying to lock a mutex that is already held by the current thread.
Small update for an obvious compilation issue and to review new test
case that could have lead to an infinite loop if the increment issue was
not detected.
I also forgot to ask if there is more chance for the instantiation to be
elided when it is implemented like in the _Safe_local_iterator:
return { __cur, this->_M_sequence };
than in the _Safe_iterator:
return _Safe_iterator(__cur, this->_M_sequence);
In the case where the user code do not use it ?
Fully tested now, ok to commit ?
François
On 11/01/23 07:03, François Dumont wrote:
> Thanks for fixing this.
>
> Here is the extension of the fix to all post-increment/decrement
> operators we have on _GLIBCXX_DEBUG iterator.
>
> I prefer to restore somehow previous implementation to continue to
> have _GLIBCXX_DEBUG post operators implemented in terms of normal post
> operators.
>
> I also plan to remove the debug check in the _Safe_iterator
> constructor from base iterator to avoid the redundant check we have
> now. But I need to make sure first that we are never calling it with
> an unchecked base iterator. And it might not be the right moment to do
> such a change.
>
> libstdc++: Fix deadlock in debug local_iterator increment [PR108288]
>
> Complete fix on all _Safe_iterator post-increment and
> post-decrement implementations
> and on _Safe_local_iterator.
>
> libstdc++-v3/ChangeLog:
>
> * include/debug/safe_iterator.h
> (_Safe_iterator<>::operator++(int)): Extend deadlock fix to
> other iterator category.
> (_Safe_iterator<>::operator--(int)): Likewise.
> * include/debug/safe_local_iterator.h
> (_Safe_local_iterator<>::operator++(int)): Fix deadlock.
> * testsuite/util/debug/unordered_checks.h
> (invalid_local_iterator_pre_increment): New.
> (invalid_local_iterator_post_increment): New.
> *
> testsuite/23_containers/unordered_map/debug/invalid_local_iterator_post_increment_neg.cc:
> New test.
> *
> testsuite/23_containers/unordered_map/debug/invalid_local_iterator_pre_increment_neg.cc:
> New test.
>
> Tested under Linux x86_64.
>
> Ok to commit ?
>
> François
>
> On 06/01/23 12:54, Jonathan Wakely via Libstdc++ wrote:
>> Tested x86_64-linux. Pushed to trunk.
>>
>> I think we should backport this too, after some soak time on trunk.
>>
>> -- >8 --
>>
>> With -fno-elide-constructors the debug iterator post-increment and
>> post-decrement operators are susceptible to deadlock. They take a mutex
>> lock and then return a temporary, which also attempts to take a lock to
>> attach itself to the sequence. If the return value and *this happen to
> Note that the chosen mutex depends on the sequence so there is no need
> for conditional sentense here, it will necessarily be the same mutex.
>> collide and use the same mutex from the pool, then you get a deadlock
>> trying to lock a mutex that is already held by the current thread.
On Thu, 12 Jan 2023 at 05:52, François Dumont wrote:
>
> Small update for an obvious compilation issue and to review new test
> case that could have lead to an infinite loop if the increment issue was
> not detected.
>
> I also forgot to ask if there is more chance for the instantiation to be
> elided when it is implemented like in the _Safe_local_iterator:
> return { __cur, this->_M_sequence };
No, that doesn't make any difference.
>
> than in the _Safe_iterator:
> return _Safe_iterator(__cur, this->_M_sequence);
>
> In the case where the user code do not use it ?
>
> Fully tested now, ok to commit ?
>
> François
>
> On 11/01/23 07:03, François Dumont wrote:
> > Thanks for fixing this.
> >
> > Here is the extension of the fix to all post-increment/decrement
> > operators we have on _GLIBCXX_DEBUG iterator.
Thanks, I completely forgot we have other partial specializations, I
just fixed the one that showed a deadlock in the user's example!
> > I prefer to restore somehow previous implementation to continue to
> > have _GLIBCXX_DEBUG post operators implemented in terms of normal post
> > operators.
Why?
Implementing post-increment as:
auto tmp = *this;
++*this;
return tmp;
is the idiomatic way to write it, and it works fine in this case. I
don't think it performs any more work than your version, does it?
Why not use the idiomatic form?
Is it just so that post-inc of a debug iterator uses post-inc of the
underlying iterator? Why does that matter?
On 12/01/23 13:00, Jonathan Wakely wrote:
> On Thu, 12 Jan 2023 at 05:52, François Dumont wrote:
>> Small update for an obvious compilation issue and to review new test
>> case that could have lead to an infinite loop if the increment issue was
>> not detected.
>>
>> I also forgot to ask if there is more chance for the instantiation to be
>> elided when it is implemented like in the _Safe_local_iterator:
>> return { __cur, this->_M_sequence };
> No, that doesn't make any difference.
>
>> than in the _Safe_iterator:
>> return _Safe_iterator(__cur, this->_M_sequence);
>>
>> In the case where the user code do not use it ?
>>
>> Fully tested now, ok to commit ?
>>
>> François
>>
>> On 11/01/23 07:03, François Dumont wrote:
>>> Thanks for fixing this.
>>>
>>> Here is the extension of the fix to all post-increment/decrement
>>> operators we have on _GLIBCXX_DEBUG iterator.
> Thanks, I completely forgot we have other partial specializations, I
> just fixed the one that showed a deadlock in the user's example!
>
>>> I prefer to restore somehow previous implementation to continue to
>>> have _GLIBCXX_DEBUG post operators implemented in terms of normal post
>>> operators.
> Why?
>
> Implementing post-increment as:
>
> auto tmp = *this;
> ++*this;
> return tmp;
>
> is the idiomatic way to write it, and it works fine in this case. I
> don't think it performs any more work than your version, does it?
> Why not use the idiomatic form?
>
> Is it just so that post-inc of a debug iterator uses post-inc of the
> underlying iterator? Why does that matter?
>
A little yes, but that's a minor reason that is just making me happy.
Main reason is that this form could produce a __msg_init_copy_singular
before the __msg_bad_inc.
And moreover I plan to propose a patch later to skip any check in the
call to _Safe_iterator(__cur, _M_sequence) as we already know that __cur
is ok here like anywhere else in the lib.
There will still be one in the constructor normally elided unless
--no-elide-constructors but there is not much I can do about it.
On Thu, 12 Jan 2023 at 18:25, François Dumont <frs.dumont@gmail.com> wrote:
>
> On 12/01/23 13:00, Jonathan Wakely wrote:
> > On Thu, 12 Jan 2023 at 05:52, François Dumont wrote:
> >> Small update for an obvious compilation issue and to review new test
> >> case that could have lead to an infinite loop if the increment issue was
> >> not detected.
> >>
> >> I also forgot to ask if there is more chance for the instantiation to be
> >> elided when it is implemented like in the _Safe_local_iterator:
> >> return { __cur, this->_M_sequence };
> > No, that doesn't make any difference.
> >
> >> than in the _Safe_iterator:
> >> return _Safe_iterator(__cur, this->_M_sequence);
> >>
> >> In the case where the user code do not use it ?
> >>
> >> Fully tested now, ok to commit ?
> >>
> >> François
> >>
> >> On 11/01/23 07:03, François Dumont wrote:
> >>> Thanks for fixing this.
> >>>
> >>> Here is the extension of the fix to all post-increment/decrement
> >>> operators we have on _GLIBCXX_DEBUG iterator.
> > Thanks, I completely forgot we have other partial specializations, I
> > just fixed the one that showed a deadlock in the user's example!
> >
> >>> I prefer to restore somehow previous implementation to continue to
> >>> have _GLIBCXX_DEBUG post operators implemented in terms of normal post
> >>> operators.
> > Why?
> >
> > Implementing post-increment as:
> >
> > auto tmp = *this;
> > ++*this;
> > return tmp;
> >
> > is the idiomatic way to write it, and it works fine in this case. I
> > don't think it performs any more work than your version, does it?
> > Why not use the idiomatic form?
> >
> > Is it just so that post-inc of a debug iterator uses post-inc of the
> > underlying iterator? Why does that matter?
> >
> A little yes, but that's a minor reason that is just making me happy.
>
> Main reason is that this form could produce a __msg_init_copy_singular
> before the __msg_bad_inc.
Ah yes, I see. That's a shame. I find the idiomatic form much simpler
to read, and it will generate better code (because it just reuses
existing functions, instead of adding new ones).
We could do this though, right?
_GLIBCXX_DEBUG_VERIFY(this->_M_incrementable(),
_M_message(__msg_bad_inc)
._M_iterator(*this, "this"));
_Safe_iterator __tmp = *this;
++*this;
return __tmp;
That does the VERIFY check twice though.
> And moreover I plan to propose a patch later to skip any check in the
> call to _Safe_iterator(__cur, _M_sequence) as we already know that __cur
> is ok here like anywhere else in the lib.
>
> There will still be one in the constructor normally elided unless
> --no-elide-constructors but there is not much I can do about it.
Don't worry about it. Nobody should ever use -fno-elide-constructors
in any real cases (except maybe debugging some very strange corner
cases, and in that case the extra safe iterator checks are not going
to be their biggest problem).
The patch is OK for trunk then.
Committed with the idiomatic approach.
I'll work on this additional check later.
On 12/01/23 22:35, Jonathan Wakely wrote:
> On Thu, 12 Jan 2023 at 18:25, François Dumont <frs.dumont@gmail.com> wrote:
>> On 12/01/23 13:00, Jonathan Wakely wrote:
>>> On Thu, 12 Jan 2023 at 05:52, François Dumont wrote:
>>>> Small update for an obvious compilation issue and to review new test
>>>> case that could have lead to an infinite loop if the increment issue was
>>>> not detected.
>>>>
>>>> I also forgot to ask if there is more chance for the instantiation to be
>>>> elided when it is implemented like in the _Safe_local_iterator:
>>>> return { __cur, this->_M_sequence };
>>> No, that doesn't make any difference.
>>>
>>>> than in the _Safe_iterator:
>>>> return _Safe_iterator(__cur, this->_M_sequence);
>>>>
>>>> In the case where the user code do not use it ?
>>>>
>>>> Fully tested now, ok to commit ?
>>>>
>>>> François
>>>>
>>>> On 11/01/23 07:03, François Dumont wrote:
>>>>> Thanks for fixing this.
>>>>>
>>>>> Here is the extension of the fix to all post-increment/decrement
>>>>> operators we have on _GLIBCXX_DEBUG iterator.
>>> Thanks, I completely forgot we have other partial specializations, I
>>> just fixed the one that showed a deadlock in the user's example!
>>>
>>>>> I prefer to restore somehow previous implementation to continue to
>>>>> have _GLIBCXX_DEBUG post operators implemented in terms of normal post
>>>>> operators.
>>> Why?
>>>
>>> Implementing post-increment as:
>>>
>>> auto tmp = *this;
>>> ++*this;
>>> return tmp;
>>>
>>> is the idiomatic way to write it, and it works fine in this case. I
>>> don't think it performs any more work than your version, does it?
>>> Why not use the idiomatic form?
>>>
>>> Is it just so that post-inc of a debug iterator uses post-inc of the
>>> underlying iterator? Why does that matter?
>>>
>> A little yes, but that's a minor reason that is just making me happy.
>>
>> Main reason is that this form could produce a __msg_init_copy_singular
>> before the __msg_bad_inc.
> Ah yes, I see. That's a shame. I find the idiomatic form much simpler
> to read, and it will generate better code (because it just reuses
> existing functions, instead of adding new ones).
>
> We could do this though, right?
>
> _GLIBCXX_DEBUG_VERIFY(this->_M_incrementable(),
> _M_message(__msg_bad_inc)
> ._M_iterator(*this, "this"));
> _Safe_iterator __tmp = *this;
> ++*this;
> return __tmp;
>
> That does the VERIFY check twice though.
>
>> And moreover I plan to propose a patch later to skip any check in the
>> call to _Safe_iterator(__cur, _M_sequence) as we already know that __cur
>> is ok here like anywhere else in the lib.
>>
>> There will still be one in the constructor normally elided unless
>> --no-elide-constructors but there is not much I can do about it.
> Don't worry about it. Nobody should ever use -fno-elide-constructors
> in any real cases (except maybe debugging some very strange corner
> cases, and in that case the extra safe iterator checks are not going
> to be their biggest problem).
>
> The patch is OK for trunk then.
>
@@ -761,12 +761,9 @@ namespace __gnu_debug
_Safe_iterator
operator++(int) _GLIBCXX_NOEXCEPT
{
- _GLIBCXX_DEBUG_VERIFY(this->_M_incrementable(),
- _M_message(__msg_bad_inc)
- ._M_iterator(*this, "this"));
- __gnu_cxx::__scoped_lock __l(this->_M_get_mutex());
- return _Safe_iterator(this->base()++, this->_M_sequence,
- _Attach_single());
+ _Safe_iterator __ret = *this;
+ ++*this;
+ return __ret;
}
// ------ Bidirectional iterator requirements ------
@@ -788,12 +785,9 @@ namespace __gnu_debug
_Safe_iterator
operator--(int) _GLIBCXX_NOEXCEPT
{
- _GLIBCXX_DEBUG_VERIFY(this->_M_decrementable(),
- _M_message(__msg_bad_dec)
- ._M_iterator(*this, "this"));
- __gnu_cxx::__scoped_lock __l(this->_M_get_mutex());
- return _Safe_iterator(this->base()--, this->_M_sequence,
- _Attach_single());
+ _Safe_iterator __ret = *this;
+ --*this;
+ return __ret;
}
// ------ Random access iterator requirements ------