Conversation
DeviceMemoryResource(device) with no options raises the release threshold of the driver's pool with cuMemPoolGetAttribute and cuMemPoolSetAttribute. The driver refuses both as potentially unsafe calls while the calling thread is inside a global or thread-local capture, and it invalidates the capture. Device.memory_resource constructs the resource lazily, so a first allocation could invalidate a capture in progress. Make the two calls in relaxed capture mode and restore the thread's previous mode afterwards. A failure to restore the mode is attached to the propagating error as a note. Ending an invalidated capture made the builder destroy a graph the driver had already destroyed. cuStreamEndCapture returns a NULL graph for an invalidated (or unjoined) capture and releases the capture graph itself, but the builder kept the owning handle it took from cuStreamGetCaptureInfo and its deleter called cuGraphDestroy again: a use-after-free that segfaulted at close() or garbage collection. Add invalidate_root_graph_state to retire the hierarchy when the driver discards the root graph, and route end_building(), close() and __dealloc__ through one GB_end_capture helper that settles graph ownership from the end-capture result. end_building() now ends an invalidated capture and raises the driver error; the builder then holds no graph (new CAPTURE_INVALIDATED state), and complete(), debug_dot_print(), graph_definition, embed() and Graph.update() say so. close() closes the builder before raising. end_building() on a forked builder is rejected with RuntimeError instead of invalidating the capture. Fixes NVIDIA#2834. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Andy-Jost
added
bug
Contributor
|
The driver also discards the body graph of a conditional node when the body capture ends invalidated, and the parent graph keeps referring to it, which cuda.core cannot repair. Say so in end_building(), the release note, and the GB_end_capture comment. Scope the DeviceMemoryResource note to the default-pool constructor, since cuMemPoolCreate is still refused under capture. Drop the end_building() call on a forked builder from the skip path of test_graph_conditional_on_forked_builder. Issue NVIDIA#2834 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Andy-Jost
force-pushed
the
ajost/issue-2776
branch
from
September 17, 2026 16:02
a56479c to
0569229
Compare
join() waits on each forked builder's stream and then closes it. When a wait raised partway through, the forks the loop had not reached stayed open with capturing streams, and destroying them later during garbage collection crashed the interpreter. Close every fork the loop did not reach before the error propagates. The capture cannot complete without the work captured on those forks, so end_building() then raises the driver's unjoined-capture error and the builder closes cleanly, which NVIDIA#2834 made possible. Issue NVIDIA#2776 Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Andy-Jost
force-pushed
the
ajost/issue-2776
branch
from
September 17, 2026 17:26
0569229 to
0ff8225
Compare
Andy-Jost
added a commit
to Andy-Jost/cuda-python
that referenced
this pull request
Sep 17, 2026
PR NVIDIA#2881 is folded into this PR: its join() cleanup relies on the invalidated-capture handling here, and the NVIDIA#2776 crash is the same double destroy of a graph the driver already discarded. Merge the two release-note entries and correct the test docstring, which attributed the crash to the fork rather than to the builder's teardown. Issue NVIDIA#2834, NVIDIA#2776 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Contributor
Author
Andy-Jost
added a commit
that referenced
this pull request
Sep 22, 2026
…2838) * cuda.core: fix pool setup and builder teardown under stream capture DeviceMemoryResource(device) with no options raises the release threshold of the driver's pool with cuMemPoolGetAttribute and cuMemPoolSetAttribute. The driver refuses both as potentially unsafe calls while the calling thread is inside a global or thread-local capture, and it invalidates the capture. Device.memory_resource constructs the resource lazily, so a first allocation could invalidate a capture in progress. Make the two calls in relaxed capture mode and restore the thread's previous mode afterwards. A failure to restore the mode is attached to the propagating error as a note. Ending an invalidated capture made the builder destroy a graph the driver had already destroyed. cuStreamEndCapture returns a NULL graph for an invalidated (or unjoined) capture and releases the capture graph itself, but the builder kept the owning handle it took from cuStreamGetCaptureInfo and its deleter called cuGraphDestroy again: a use-after-free that segfaulted at close() or garbage collection. Add invalidate_root_graph_state to retire the hierarchy when the driver discards the root graph, and route end_building(), close() and __dealloc__ through one GB_end_capture helper that settles graph ownership from the end-capture result. end_building() now ends an invalidated capture and raises the driver error; the builder then holds no graph (new CAPTURE_INVALIDATED state), and complete(), debug_dot_print(), graph_definition, embed() and Graph.update() say so. close() closes the builder before raising. end_building() on a forked builder is rejected with RuntimeError instead of invalidating the capture. Fixes #2834. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * cuda.core: scope the invalidated-capture recovery to top-level builders The driver also discards the body graph of a conditional node when the body capture ends invalidated, and the parent graph keeps referring to it, which cuda.core cannot repair. Say so in end_building(), the release note, and the GB_end_capture comment. Scope the DeviceMemoryResource note to the default-pool constructor, since cuMemPoolCreate is still refused under capture. Drop the end_building() call on a forked builder from the skip path of test_graph_conditional_on_forked_builder. Issue #2834 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * cuda.core: close unjoined forks when GraphBuilder.join fails join() waits on each forked builder's stream and then closes it. When a wait raised partway through, the forks the loop had not reached stayed open with capturing streams, and destroying them later during garbage collection crashed the interpreter. Close every fork the loop did not reach before the error propagates. The capture cannot complete without the work captured on those forks, so end_building() then raises the driver's unjoined-capture error and the builder closes cleanly, which #2834 made possible. Issue #2776 Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> * cuda.core: fold the join cleanup into the capture-teardown release note PR #2881 is folded into this PR: its join() cleanup relies on the invalidated-capture handling here, and the #2776 crash is the same double destroy of a graph the driver already discarded. Merge the two release-note entries and correct the test docstring, which attributed the crash to the fork rather than to the builder's teardown. Issue #2834, #2776 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * cuda.core: point the conditional-body limitation at issue #2918 An invalidated capture of a conditional body leaves the parent graph invalid, and ending the parent capture afterwards crashes inside the driver (#2918, a CUDA driver bug reproduced with the driver API alone). Say so in the end_building docstring and the release note instead of explaining the mechanism. Issue #2834 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * cuda.core: keep GraphBuilder.join's cleanup going when a close fails join() closes the builders it did not join before its error propagates. That sweep ran in a finally block and called close() on each builder. A builder from another capture makes the root's wait fail, and the driver invalidates that builder's capture as well, so its close() raised too. The raise stopped the sweep, stranded the later builders with capturing streams, and replaced the merge error with the close error. Run the sweep in an except block, close each builder through GB_close, which returns the driver status instead of raising, and attach a failed close to the propagating error as a note (error handling policy). The original error is re-raised unchanged. Review feedback on #2838. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(cuda.core): cover a failing close inside GraphBuilder.join's cleanup The existing test provokes the join failure with a forked builder, whose close() cannot raise, so the sweep's own failure path went untested. Join two separate primaries into a capture: the driver refuses the cross-capture wait and invalidates the other capture, so closing that builder fails. The test checks that every builder is closed, that the merge error propagates, and that the failed close is attached as a note (or reported as a CUDAWarning on Python 3.10). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * cuda.core: factor retire_graph out of the graph invalidation sweeps invalidate_child_graph_state and invalidate_root_graph_state repeated the per-box retirement: detach node handles, drop the registry entry and attachments, move the box to the graveyard. Both now call retire_graph and differ only in which boxes they select. No behavior change. Review feedback on #2838. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #2776.
GraphBuilder.join()waits on each forked builder's stream and then closes it. When a wait raised partway through, the forks the loop had not reached stayed open with capturing streams, and destroying them later during garbage collection crashed the interpreter.Changes
_graph_builder.pyx:join()closes every fork the loop did not reach before the error propagates. On success every fork is already closed, so the cleanup is a no-op. The capture cannot complete without the work captured on those forks, soend_building()afterwards raises the driver's unjoined-capture error and the builder closes cleanly, which cuda.core: DeviceMemoryResource setup is refused under non-relaxed stream capture, and tearing down the invalidated GraphBuilder segfaults #2834 made possible._graph_builder.pyi: docstring.tests/graph/test_graph_builder.py: splits a builder four ways, launches work on one fork, closes another fork's stream so the wait on it raises insidejoin(), and checks that all forks are closed, that collecting them does not crash, and that ending the capture reports the unjoined fork.Related Work
main.🤖 Generated with Claude Code