[NOGIL] Add Free Threading Support - #2347
Draft
Ojasva Jain (ojasvajain) wants to merge 15 commits into
Draft
Conversation
|
🎉 All Contributor License Agreements have been signed. Ready to merge. |
airlock-confluentinc
Bot
force-pushed
the
dev_thread_free_support
branch
2 times, most recently
from
September 7, 2026 09:26
00aa99d to
a916f2f
Compare
* Add CI verification jobs for free-threaded Python 3.14t
- Add a Semaphore block running source package verification and
integration tests on CPython 3.14t (classic and consumer group
protocols). cimpl does not declare free-threading support yet, so
importing it re-enables the GIL: these jobs validate the 3.14t
toolchain and packaging until that declaration ships.
- Skip the CI-only orjson install on free-threaded interpreters: no
free-threaded orjson wheels exist and the source build would fail;
the stdlib JSON fallback path stays covered.
- Add a module-scoped autouse fixture (defined on free-threaded builds
only) that warns when the GIL is re-enabled around a test module.
Hard asserts are staged behind TODO FTS markers, to be enabled in the
same PR that declares Py_MOD_GIL_NOT_USED.
Interpreter detection follows the free-threading HOWTO:
https://docs.python.org/3/howto/free-threading-python.html
* Handle deps without free-threaded wheels in test setup
On free-threaded (no-GIL) builds, the rules and json-fast extras'
compiled deps (tink, google-re2, grpcio; orjson) ship no free-threaded
wheels and fail to build from source, so:
- Add requirements-tests-install-nogil.txt, a variant of
requirements-tests-install.txt without those extras, and install it
from source-package-verification.sh when the interpreter is
free-threaded (detected via Py_GIL_DISABLED).
- Exclude the schema_registry test modules that import tink/celpy/orjson
at the top of the file from collection on free-threaded builds only;
on regular builds a missing dep stays a loud collection error rather
than a silent skip. Plain serdes coverage recovery is marked as a
TODO NOGIL follow-up.
* Style fixes
…usions (#2309) * [NOGIL] Split schema_registry tests to narrow free-threaded test exclusions Each of test_avro_serdes.py, test_config.py, test_json_serdes.py, and test_proto_serdes.py is split into a plain file (no rules/encryption dependency) and a _rules file (CEL/encryption/JSONata-dependent tests). This lets the plain tests run on free-threaded (3.14t) builds, where tink/celpy/orjson have no free-threaded wheels, while only the _rules files stay excluded via conftest.py's collect_ignore. * Fix isort/black formatting in split schema_registry tests
…stry tests from free-threaded collection test_azure_aead.py, test_azure_client.py, test_azure_driver.py, and test_encrypt_executor.py import azure/tink unconditionally at module level but were missing from conftest.py's free-threaded collect_ignore list, causing collection errors on 3.14t CI. These tests were added to master after the original NOGIL schema_registry exclusion list (#2309) landed, so they weren't accounted for; rebasing onto master surfaced the gap.
…lusions test_json.py's orjson import became pytest.importorskip upstream, so it no longer needs excluding. test_dlq_serdes.py is new post-rebase and hard- imports celpy, so it needs excluding.
…ation test collection
…elf (#2313) * Fix Producer.close() races with concurrent calls and with itself Producer.close() previously raced with concurrent produce()/poll()/ flush()/produce_batch()/transaction calls and with itself when called from multiple threads, both leading to use-after-free/double-free on the underlying rd_kafka_t handle. Adds an active_calls/closing guard (Handle_enter_rk_use/Handle_exit_rk_use) so every method that touches self->rk registers itself before use, and close() drains in-flight calls before tearing down; a CAS on `closing` ensures only one concurrent close() call performs the actual teardown, with losing callers waiting for it to finish rather than racing it. Adds tests/parallel/test_producer_close_race.py covering each affected method racing close(), close() racing itself, and close()'s blocking behavior. Uses pytest-forked (POSIX only) so a regression segfault fails only that test. Integration tests against a real broker are still pending. * Replace pytest-forked with subprocess-based test isolation * Fix subprocess_isolated import and flaky close() timing assertion * Rename tests/parallel to tests/concurrency and add integration tests for Producer close()/transaction races * Clarified comment * Make Producer.close() non-blocking for concurrent callers, add reentrancy tests Concurrent close() calls now return False immediately with a warning instead of waiting for the CAS winner, since waiting could deadlock a caller that already holds an active_calls slot (e.g. a callback invoked from its own poll()/flush()). poll()/flush() now also exit early once closing is set instead of blocking the drain-wait. Fixes an ordering bug in produce_batch() where the topic handle was destroyed after releasing the active_calls slot. Adds integration tests for reentrant callbacks and close()'s internal flush delivering all messages, and documents close()-from-callback as unsupported. * Fix close() losers to wait for winner instead of returning False early * Fix CI flakiness in test_close_races_close_losers_wait_for_slow_winner * Move signal/slow-winner close() race tests to integration suite * Allow reentrant Producer calls from close()'s own delivery callback * Update docstring of test case * Harden race checks
* Add a serializing Consumer reentrancy gate for free-threading support Introduces gate_owner/gate_depth in Consumer.c so concurrent, cross-caller access to a single Consumer/AIOConsumer instance waits for the current caller to finish rather than being undefined behavior, since librdkafka's consumer is not thread-safe; legitimate re-entrant calls (e.g. a rebalance/commit callback calling back into the Consumer that triggered it) are still admitted immediately. AIOConsumer identity is tracked via a ContextVar since the owning logical caller can move across ThreadPoolExecutor worker threads. Includes unit and integration test coverage for both the sync Consumer and AIOConsumer. * Style fixes * Fix flaky tests * Add TODO for handling concurrent calls inside callback * Addressed comments * Trigger CLA check * Address comments * Return consistent error when consumer is closed * Fix one test case * Fix styling
#2335) * Exclude Avro tests from running on free threaded python builds * Exclude only fastavro * temoporarily install fastavro for docs
) * Add a serializing Consumer reentrancy gate for free-threading support Introduces gate_owner/gate_depth in Consumer.c so concurrent, cross-caller access to a single Consumer/AIOConsumer instance waits for the current caller to finish rather than being undefined behavior, since librdkafka's consumer is not thread-safe; legitimate re-entrant calls (e.g. a rebalance/commit callback calling back into the Consumer that triggered it) are still admitted immediately. AIOConsumer identity is tracked via a ContextVar since the owning logical caller can move across ThreadPoolExecutor worker threads. Includes unit and integration test coverage for both the sync Consumer and AIOConsumer. * Trigger CLA check * Add Integration test coverage for AIO Producer * Add Tx related test cases + Add abort_tx call in Producer close * Fix flaky AIO Producer test case * Fix for serializing concurrent calls from a Async callback * Minor improvements to tests * Fix styling * Fix flaky test case * Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant * Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant * Revert "Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant" This reverts commit 42b40cc. * Revert "Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant" This reverts commit f75dba3. * Add TODO
…y to test_proto_serdes_rules.py
…threaded collect_ignore entries
…aces (#2317) (#2346) * Add a serializing Consumer reentrancy gate for free-threading support Introduces gate_owner/gate_depth in Consumer.c so concurrent, cross-caller access to a single Consumer/AIOConsumer instance waits for the current caller to finish rather than being undefined behavior, since librdkafka's consumer is not thread-safe; legitimate re-entrant calls (e.g. a rebalance/commit callback calling back into the Consumer that triggered it) are still admitted immediately. AIOConsumer identity is tracked via a ContextVar since the owning logical caller can move across ThreadPoolExecutor worker threads. Includes unit and integration test coverage for both the sync Consumer and AIOConsumer. * Trigger CLA check * Add Integration test coverage for AIO Producer * Add Tx related test cases + Add abort_tx call in Producer close * Fix styling * Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant * Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant * Trigger CLA check * Trigger CLA check * Add a serializing Consumer reentrancy gate for free-threading support Introduces gate_owner/gate_depth in Consumer.c so concurrent, cross-caller access to a single Consumer/AIOConsumer instance waits for the current caller to finish rather than being undefined behavior, since librdkafka's consumer is not thread-safe; legitimate re-entrant calls (e.g. a rebalance/commit callback calling back into the Consumer that triggered it) are still admitted immediately. AIOConsumer identity is tracked via a ContextVar since the owning logical caller can move across ThreadPoolExecutor worker threads. Includes unit and integration test coverage for both the sync Consumer and AIOConsumer. * Style fixes * Addressed comments * Trigger CLA check * Address comments * Add Tx related test cases + Add abort_tx call in Producer close * Fix for serializing concurrent calls from a Async callback * Fix one test case * Fix styling * [NOGIL] Guard AdminClient against concurrent close()-vs-method-call races * Add handle check in common APIs and refactor overall handle logic * Remove conflict marker * Remove another conflict marker * Rename handle functions * Add more tests * Fix flaky test * Address comments * Fix corrupted _common.py * Fix other corrupted files * Fix flaky test case
* Add a serializing Consumer reentrancy gate for free-threading support Introduces gate_owner/gate_depth in Consumer.c so concurrent, cross-caller access to a single Consumer/AIOConsumer instance waits for the current caller to finish rather than being undefined behavior, since librdkafka's consumer is not thread-safe; legitimate re-entrant calls (e.g. a rebalance/commit callback calling back into the Consumer that triggered it) are still admitted immediately. AIOConsumer identity is tracked via a ContextVar since the owning logical caller can move across ThreadPoolExecutor worker threads. Includes unit and integration test coverage for both the sync Consumer and AIOConsumer. * Trigger CLA check * Add Integration test coverage for AIO Producer * Add Tx related test cases + Add abort_tx call in Producer close * Fix styling * Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant * Serialize AIOConsumer calls that outlive their callback invocation instead of admitting them as re-entrant * Trigger CLA check * Trigger CLA check * Add a serializing Consumer reentrancy gate for free-threading support Introduces gate_owner/gate_depth in Consumer.c so concurrent, cross-caller access to a single Consumer/AIOConsumer instance waits for the current caller to finish rather than being undefined behavior, since librdkafka's consumer is not thread-safe; legitimate re-entrant calls (e.g. a rebalance/commit callback calling back into the Consumer that triggered it) are still admitted immediately. AIOConsumer identity is tracked via a ContextVar since the owning logical caller can move across ThreadPoolExecutor worker threads. Includes unit and integration test coverage for both the sync Consumer and AIOConsumer. * Style fixes * Addressed comments * Trigger CLA check * Address comments * Add Tx related test cases + Add abort_tx call in Producer close * Fix for serializing concurrent calls from a Async callback * Fix one test case * Fix styling * [NOGIL] Guard AdminClient against concurrent close()-vs-method-call races * Add handle check in common APIs and refactor overall handle logic * Remove conflict marker * Remove another conflict marker * Rename handle functions * Add more tests * Fix flaky test * Address comments * Fix corrupted _common.py * Fix other corrupted files * Fix flaky test case * [NOGIL] Fix borrowed-reference/stale-count races in Admin/Producer/Consumer parse loop * Fix list mutation in Admin API and a flaky test case * Fix data race on Handle.rk in close()/__exit__()/__len__() by making liveness checks atomic * Fix flaky test * Address comments
airlock-confluentinc
Bot
force-pushed
the
dev_thread_free_support
branch
from
September 8, 2026 06:57
a916f2f to
24155f9
Compare
|
Ojasva Jain (ojasvajain)
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Free threading support has been under development on
dev_thread_free_supportbranch. This PR merges this branch to master.PRs merged to feature branch: https://github.com/confluentinc/confluent-kafka-python/pulls?q=is%3Apr+label%3Anogil+is%3Amerged+
Open PRs: https://github.com/confluentinc/confluent-kafka-python/pulls?q=is%3Apr+label%3Anogil+is%3Aopen
Pending: Documentation changes, CHANGELOG update and automating performance test cases.
Checklist
References
JIRA:
Test & Review
Open questions / Follow-ups