[nascheme]

Fixes the free-threaded deadlock described in gh-158195 by removing type-lock pinning. The type lock and class dict lock can now be released normally while a thread waits to stop the world, rather than remaining pinned across that wait.

Current design

In the free-threaded build, updates to published type slots, flags, MROs and bases need stop-the-world synchronization. Waiting to stop the world can release TYPE_LOCK, so state examined before the wait may no longer be current when the world has stopped.

This PR handles that without a generation counter or optimistic retry loop:

• Slot updates: resolve the affected slots and write them within the same stop-the-world interval. Descriptor lookup and subclass traversal therefore use the current type dictionaries and hierarchy, not updates prepared before the wait. There is no longer a queue of prepared slot updates.
• __bases__ assignment: perform hierarchy-dependent validation (inheritance cycles and layout compatibility) and store the new tp_bases and tp_base within the same stop-the-world interval. Argument checks and the audit event happen beforehand; errors from hierarchy validation are raised after restarting the world.
• MRO publication and rollback: custom mro() methods still run outside stop-the-world and may execute arbitrary Python code. Before publishing an MRO, check with the world stopped that the old MRO has not been replaced. Rollback similarly restores an MRO or bases value only if the value being undone is still current. Unrelated type mutations do not cause mro() to be rerun.

The entire __bases__ operation is not one stop-the-world transaction: in particular, custom MRO computation remains reentrant. The identity checks preserve the existing handling of replacement by a nested operation and also cover replacement by another thread while the type lock is released.

Supporting changes

Slot resolution with the world stopped must not run user code, block, or deallocate objects. The lookup and traversal paths are adjusted accordingly:

• Slot-name lookup uses exact-string comparisons and skips non-string dictionary keys, rather than potentially invoking a key's __eq__ method. String-subclass keys supplied in a class namespace are copied to exact strings, as they already are for class attribute assignment.
• Dictionary lookup and subclass traversal use borrowed references while the world is stopped, avoiding reference drops that could invoke deallocation and reacquire locks.
• Type version tags are invalidated after publication while the type lock is held, so cache entries created during the wait cannot remain valid with stale state.
• __bases__ assignment stores a private copy of the assigned tuple. Its identity distinguishes successive assignments even when they use the same input tuple object.
• Deferred refcounting is enabled only for an MRO tuple that is actually stored, so a rejected candidate can be freed normally.

Tests

Added coverage includes stop-the-world calls from custom mro() methods under type-lock contention, concurrent bases assignments, and the custom mro() example that mutated an unrelated type on every call. Additional tests cover dictionary keys whose equality methods must not run during slot lookup, normalization of string-subclass namespace keys, and collection of subclasses after slot updates.

Backporting and compatibility

The deadlock affects 3.14 and 3.15, so the fix should be backported to both. This design adds no per-interpreter generation counter or corresponding interpreter-structure ABI change.

Stop-the-world synchronization remains free-threaded-only. Some supporting changes also apply to the default build, notably namespace-key normalization and copying the tuple assigned to __bases__. Consequently, C.__bases__ need not be the same tuple object that was assigned, although its contents are unchanged.

I backported this change to the 3.14 branch and tested it using snake pit. Some package tests fail but they fail without the patch applied as well. Tests were run on x86-64 Linux, with both ordinary GIL and free-threaded builds and matched package versions and dependencies. The 33-package catalog was attempted, including pytest, Hypothesis, Django, SQLAlchemy, Pydantic Core, SymPy, Cython, CFFI, NumPy, pandas, matplotlib, packaging, requests, AnyIO, greenlet, uvloop, Pillow, lxml, cryptography, and Sphinx.

If we want to be extra careful, I suggest we use ifdefs so that the default build does not get behavioral changes in the backport (the namespace key normalization and copied tuple seem most risky).

Previous approach

Earlier revisions prepared updates before stopping the world, then checked a per-interpreter generation counter and retried if type state had changed. That could retry indefinitely when a custom mro() mutated an unrelated type on every call, as Dino's example demonstrated. The current implementation removes generation-based retries in favor of the synchronization described above. Earlier review comments about the counter, retry loops, and slot-update queue refer to that superseded design.

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @nascheme for commit c7ac4e7 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F159972%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[DinoV]

It looks like this causes an infinite loop:

class Other: pass

class Meta(type):
    def mro(cls):
        Other.__repr__ = lambda self: "x"
        return type.mro(cls)


class B: pass

class C(B, metaclass=Meta): pass

C.__bases__ = (B, )