Repository navigation
TEMP: [CI] Run only mac-beta, only the cppjit tests, 20 times each - #23740
Draft
guitargeek wants to merge 10 commits into
Draft
guitargeek wants to merge 10 commits into
guitargeek wants to merge 10 commits into
Conversation
Test Results726 tests 726 ✅ 2h 30m 51s ⏱️ Results for commit 41e8bb8. ♻️ This comment has been updated with latest results. |
…atatypes Measurement commit for the sporadic crash of test_datatypes on the mac-beta ARM64 runner: disable the Windows and Linux jobs, drop the other macOS platforms from the matrix, restrict ctest on mac-beta to the cppjit tests, and register 300 extra copies of test_datatypes (-rep1 to -rep300) that share the dictionary fixture. ctest runs them 8 at a time in the common test directory, like the real suite runs the cppjit tests, instead of the sequential and otherwise idle repetitions --repeat gives. The crash rate measured over 201 PR jobs and 13 nightlies is about 3%, so 300 clean runs without the fix have a probability below one in a thousand, while 300 runs with it should show no failure. About 6 seconds per copy, so a few minutes of test time. Not to be merged.
guitargeek
force-pushed
the
cppjit-mac-beta
branch
from
October 17, 2026 20:56
0949f99 to
5cf8835
Compare
pyunittests-bindings-cppjit-test-datatypes crashes in roughly one of six runs on the mac-beta ARM64 runner, and nowhere else: 2 of 13 nightlies and 4 of 25 PR jobs since the test was introduced on 2026-10-03. Every instance is a segmentation violation or bus error in Python's cyclic garbage collector (gc_collect_main -> list_traverse -> visit_decref) during test26_function_pointers. That is the signature of a Python object that was freed, or whose memory was corrupted, while a container still referred to it, and the damage can come from any earlier test since the collection that trips over it only runs when the allocation count crosses the GC threshold. The crash does not reproduce on Linux, not even with the debug allocator and a full collection forced after every Python line, and the local build uses the free-threaded interpreter whose GC differs from the mac's default build. So instead of guessing, make the next CI failure informative: - Run the test with PYTHONMALLOC=debug, so a dangling reference fails loudly (freed blocks are filled with 0xDD, guard bytes are checked, a double free or overrun reports the block) instead of corrupting whatever reused the memory. - Load a small pytest plugin, root_ci_diagnostics.py, that performs a full gc.collect() after every test, which moves the crash into the teardown of the test that leaves the damage behind, and re-arms Python's faulthandler before and after each test. ROOT's signal handlers replace the one pytest installs at session start and print a C stack without symbols on the macOS runners; with faulthandler back in place the report carries the Python traceback and, on Python 3.14+, a symbolized C stack. The plugin is loaded only for test files registered with the new CRASH_DIAGNOSTICS option of CPPJIT_ADD_PYTEST, currently test_datatypes alone, and lives next to the CMake file rather than in the vendored conftest.py so that syncing cppjit from upstream does not drop it. It can be removed once the crash is understood. Verified locally: the test passes with the plugin active; an injected segfault yields the faulthandler report instead of ROOT's; on a default-build Python 3.14.7, an injected over-decref of a list element crashes in the teardown of the injecting test with a "Garbage-collecting" marker.
Sharpen the crash diagnostics for the measurement runs on mac-beta: the plugin now also forces a full gc.collect() after every line of test24 to test29 of test_datatypes, so a corrupted container is found at the statement that leaves it behind rather than at the next threshold-triggered collection, and reports how many collections it forced. The number of test_datatypes copies goes from 300 to 600: the crash rate in this setup is about 0.5%, so this yields a few reports per run (or, if the debug allocator and the forced collections make the damage deterministic, a few hundred). Not to be merged.
The 600 copies under PYTHONMALLOC=debug and forced collections all passed, where the production configuration fails about 0.5% of the time, so the Python debug allocator changes the memory layout enough to hide the bug. That points away from a plain dangling reference to a Python object and towards the C++ heap, or a stale pointer whose effect depends on where a block lands. Replace the 600 copies by two groups of 300 that keep Python's production allocator and no forced collections (new ROOT_CI_DIAGNOSTICS_NO_GC switch in the plugin, which then only re-arms the faulthandler): - -malloc: MallocScribble, MallocPreScribble and MallocGuardEdges, macOS's debugging for the system allocator, which Python's small objects do not use. A use of freed or uninitialized C++ memory then reads 0x55/0xAA patterns and crashes at the culprit instead of corrupting a neighbour. - -only26: pytest -k test26_function_pointers, to learn whether the test that always notices the damage also causes it. CPPJIT_ADD_PYTEST gains ENV and PYTEST_ARGS for these per-copy overrides; the test driver applies environment entries in order, so the overrides win. Not to be merged.
…cator Both 300-copy groups of the previous round passed (macOS malloc debugging, and test26 alone), as did the 600 copies under PYTHONMALLOC=debug, while every configuration with the production allocator and the whole test file has crashed, always with the collector tripping in test26. Python 3.14's incremental collector reaches an old container at a reproducible point of the allocation sequence, so a list damaged at any earlier point can be found consistently in test26; the test may just be where the damage is noticed, not where it is done. The previous round could not tell, because the forced collections were bundled with the debug allocator, which hides the bug. Separate them: - -gc (700 copies): production allocator, full gc.collect() after every test and after every line of test24 to test29. A crash in the teardown of a test, or at a line, names where the damage is left behind. - -plain (300 copies): production allocator, faulthandler only, for the base rate and the Python line at which the collector trips. Not to be merged.
In the 700 copies with a forced collection after every test, 4 crashed, all identically: every test passed, then a bus error at the session summary in pytest's _get_reports_to_display, on a TestReport taken from the terminal reporter's stats lists. The 300 copies without forced collections all passed. The collections had traversed those lists after every test without tripping, so the report looked valid until late and was garbage by the end: the signature of a report freed early, through a stale Py_DECREF on a pointer to a dead object whose memory the report had reused, and then overwritten once its block was reused again. Under the debug allocator such a pointer lands in a block's padding instead of a reference count, which is why that configuration was silent. So watch for exactly that: after every test, and once more at session finish, the plugin walks the stats lists, records each report's reference count once its own test has finished, and exits the process with a message if a report's count ever drops, or if a report can no longer be read. Each check announces itself on stderr first, so a hard crash inside the scan still shows which test preceded it. The copies become one group of 700 (-rc) with the production allocator and forced collections, the configuration that produced the hits. Verified with a default-build Python 3.14: a clean run stays silent, and an injected Py_DecRef on a report is reported after the test that did it. Not to be merged.
The report watch fired in 33 of 700 copies, always at the same place: in the check after test21 (24 copies, a report in stats['passed'] whose type pointer now pointed at a list) or as a bus error inside the check after test20 (9 copies). The check after test19 was clean in all of them. So the report dies during test20_object_comparisons_with_cpp__eq__, or at the latest during test21_object_validity. Narrow that to a statement: the line tracer now works on test19 to test22 and runs the report check after every line. The check reads the stats lists' item pointers, reference counts and type pointers as raw memory through ctypes, because a freed report has a zero reference count and any Python-level touch of it, even iterating the list, deallocates the block a second time. Baselines are taken at the setup of the next test and at teardown, where the runner holds no temporaries to the reports, so the message can say whose report died. Only default (GIL) builds have the object layout the raw reads assume; elsewhere the check is skipped. Verified with a default-build Python 3.14: a clean run is silent, and an injected Py_DecRef on a report is reported at the following line with the report named. Not to be merged.
…report The previous round checked the reports after every line of test19 to test22, with a full collection before each check. The collections changed the allocation pattern enough that the damage became rare (2 in 700 copies instead of 33 in 700); both remaining copies crashed in the collection right after `d = gbl.CppjitTestPod()` in test21, and ctest still reported them as passed because RootTestDriver.cmake runs a test a second time after a raw segmentation fault. This round goes back to the configuration of the run that found the damage (collections only after each test and in test24 to test29) and adds a finalizer to pytest's TestReport and CollectReport classes. It runs inside the Py_DECREF that drops the last reference, so when a report that the terminal reporter's stats lists still hold is deallocated, the output carries the Python traceback (the statement of the test) and, with Python 3.14, the C stack of the decrement, which names the cpyrt code that performed it. The process then exits with code 98. The second attempt after a segmentation fault in RootTestDriver.cmake is disabled so that such crashes fail the test instead of hiding behind a passing retry.
… test22 The report finalizer did not fire in the copy that crashed: the collection at the teardown of test21 died in list_traverse without any report having been deallocated. Together with the earlier run, where a report's type slot pointed at a list instance while the stats list still held the report, this says the damage is a stray write into live memory, not a stray decrement. So the tracer now snapshots the raw state of the terminal reporter's stats lists (list identity, item array, item pointers, type slots and reference counts of the reports) when the body of test19 to test22 starts, and compares it before every line and at the return, without collecting. Pytest does not touch its reports inside a test body, so any difference is the damage, and the message names the statement that did it. On a difference the plugin dumps the memory around the affected addresses and looks up which live object owns each address (the object itself, a list's item array, or the inside of an object), so that a pointer written into the report can be identified. It also prints the address of every new local of these tests, with the C++ address for cppjit proxies, to relate the damage to the objects the test creates.
When a C++ function takes a proxied object as `T**` (or `void**`), the converter passes the address of the proxy's own pointer slot, so the function can delete the object and store a new one in the proxy, as `CppjitTestData::set_pod_ptrptr_out` does in test_datatypes. The memory regulator still tracks the proxy under the old address, and when the proxy dies, `UnregisterPyObject` looks it up under the new one. That erases nothing, or erases the entry of an unrelated proxy that is tracked under the new address, and leaves the old entry pointing at the dead proxy. Once the allocator hands the old address to a new object of the same class, `RegisterPyObject` finds the dangling entry and clears `kIsRegulated` in the `fFlags` of the dead proxy, i.e. it clears bit 0x800 somewhere in whatever object reused that memory. This was the sporadic crash of pyunittests-bindings-cppjit-test-datatypes on the mac-beta runner: the constructor call in test21 got the address of the pod that test16 had replaced, and the write landed in the item array of pytest's list of passed reports, which the garbage collector later traversed. Only macOS's allocator reused the address at that point. `UnregisterPyObject` now erases an entry only if it belongs to the proxy, and searches the map for the proxy otherwise, which also covers proxies whose held pointer was nulled. The search runs only when a registered proxy is not found under the address it holds now.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Measurement commit for the sporadic crash of test_datatypes on the mac-beta ARM64 runner: disable the Windows and Linux jobs, drop the other macOS platforms from the matrix, and restrict ctest on mac-beta to the cppjit tests with --repeat until-fail:20. At the observed rate of roughly one crash in six runs, 20 repetitions of test_datatypes fail with near certainty without the fix and pass with it.
Not to be merged.
🤖 Done with the help of AI