gh-148259: make list.remove atomic for non-trivial __eq__ comparison

[KowalskiThomas]

What is this PR?

This PR addresses the bug reported in #148259 where calling rich comparison may result in releasing the GIL which may result in list.remove removing the wrong item from the list.

The PR also updates the tests added for #126033.

• Issue: list.remove is not atomic for non trivial __eq__ comparisons #148259

[colesbury]

Please add a NEWS entry (https://devguide.python.org/getting-started/pull-request-lifecycle/#how-to-add-a-news-entry) and fix the failing tests.

[picnixz]

The raises parameter is now misleading. Please correct it. In addition, please add tests to list itself. And please also update the docs if they need to.

Overall, I'm not convinced by this change. Mutating the list while doing the comparison is unsafe. We just don't want to crash. But making it raise a RuntimeError does not necessarily makes it better. Otherwise, we need to do it for all other mutable sequences in C to have more or less the same behavior.

cc @rhettinger

[bedevere-app]

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

[KowalskiThomas]

The raises parameter is now misleading. Please correct it. In addition, please add tests to list itself. And please also update the docs if they need to.

Yes, I had to stop in the middle of my work earlier -- I didn't plan to leave it in that state, sorry.

Otherwise, we need to do it for all other mutable sequences in C to have more or less the same behavior.

I think some sequences already raise RuntimeError when incompatible things are done to them concurrently, right?

[KowalskiThomas]

I just had the time to give this another look. Updated the tests again and now everything passes I believe.

EDIT: just realised you also asked for tests on list; I'll add them later today!

[picnixz]

I think some sequences already raise RuntimeError when incompatible things are done to them concurrently, right?

Yes but if it is not always consistent. I see this as a garbage-in/garbage-out case. Do not mutate your list while doing other things concurrently. We prefer not raising and leaving it as a GIGO rather than adding complexity and changing existing behaviors.

We do fix cases where there hard crashes (segfaults) but ortherwise we try to retain existing behaviors. Anyway, I think we need first to discuss this, perhaps even standardizing this into a PEP, that is, make sure that concurrent mutations consistently raise whatever collection you are using (not just lists or dicts) and how.

[picnixz]

Yes, I had to stop in the middle of my work earlier -- I didn't plan to leave it in that state, sorry.

In this case, please make it a draft.

[KowalskiThomas]

Anyway, I think we need first to discuss this, perhaps even standardizing this into a PEP, that is, make sure that concurrent mutations consistently raise whatever collection you are using (not just lists or dicts) and how.

This makes sense to me. I admit I didn't check what other collections do outside deque (which does raise RuntimeError in case it is mutated concurrently); someone on the original issue did for set and dict (which from what I can tell properly handle those cases).
I'll look more into this and maybe check what the process for a PEP is.

I see this as a garbage-in/garbage-out case. Do not mutate your list while doing other things concurrently. We prefer not raising and leaving it as a GIGO rather than adding complexity and changing existing behaviors.

While I generally understand the GIGO case/rationale, I beg to differ on that one.

The way this can "currently" (with the GIL) happen is a bit convoluted and unlikely to happen in a real life scenario (custom equality operator releasing the GIL at the same time as another thread resumes and mutates the list), so I wouldn't worry too much about it.

However, with the GIL disabled, I can see it happening more commonly (it really just takes two threads working on the same list at the same time, I believe?).
On top of that, many operations on list are atomic in practice (and PEP-703 ensures they stay so without the GIL), so as a normal user who doesn't know anything about how list (or CPython, for that matter) works internally, I would be very confused if sometimes my list.remove didn't remove the right item when all the other list operations always work as expected.

It would seem (at least to me?) much more helpful to get an exception loudly saying that I'm doing something wrong than to have my code randomly fail to do what I want it to every once in a while (which I could potentially only realise because some downstream result of that list.remove isn't behaving as expected, making reproducing and debugging the issue very complicated).

What is your take on that? Happy to be told so if I'm missing part of the picture here, of course.

[picnixz]

My take is:

• equality is not meant to mutate the list, whether it is with or without the GIL.
• legitimate use cases that make it unsage without the GIL must be fixed
• convoluted ones are GIGO. There is no point in making it slower for legitimate use cases just to protect unlikely cases.