cuda.core: require a cuda-bindings floor and call the driver through its resolved pointers (#2783) - #2920
Draft
Andy-Jost wants to merge 8 commits into
Draft
cuda.core: require a cuda-bindings floor and call the driver through its resolved pointers (#2783)#2920Andy-Jost wants to merge 8 commits into
Andy-Jost wants to merge 8 commits into
Conversation
cuda.core accepted any cuda-bindings of the right major at build time and at run time, and built against whatever cuda.h was on the include path. Builds succeeded for configurations we never test, and an older cuda-bindings at run time surfaced as an ImportError for a missing C function, a silently disabled feature, or a null-pointer crash (NVIDIA#2783). Each supported CUDA major now has a floor, the newest cuda-bindings release its CI source root can build: 12.9.8 for CUDA 12 and 13.4.1 for CUDA 13, in the import-free module cuda/core/_bindings_floor.py. Build time: the pip build requirement becomes `cuda-bindings>=<floor>,==<major>.*`, and because conda-forge, pixi and --no-build-isolation installs bypass it, the backend itself checks the imported cuda-bindings against the floor and requires the cuda.h it compiles against to have the same major.minor as that cuda-bindings (the header cuda-bindings was generated from), even when CUDA_CORE_BUILD_MAJOR is set. It records the header version and the floor in the generated, gitignored cuda/core/_build_info.py, shipped like _version.py. Run time: cuda/core/__init__.py reads that record from the selected build (the cu12/cu13 subpackage of the merged wheel, or the top level of a plain build) and requires the installed cuda-bindings to be of the build's major and at least the floor, or at least the header's minor when that is newer. The error names the version found, the version required, and the pip command that fixes it. Pre-release and dev builds of an accepted version pass. Packaging: the cu12/cu13 extras pin the floor; a test keeps them and the ci/versions.yml toolkit pins in step with the module. CI: the BINDINGS_SOURCE=published rows, which paired a new wheel with cuda-bindings 13.0 (no longer supported), become BINDINGS_SOURCE=floor: they install the floor bindings, read from the wheel under test by ci/tools/cuda_core_bindings_floor.py, and keep their older CTK libraries. The prior-major rows whose CTK minor differs from prev_build do the same. Docs: support policy section, install guide, 1.3.0 release note. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The two `#if CUDA_VERSION >= 130x0` fences in _cpp/rt/driver_api.* compiled cuDevSmResourceSplit and cuMemcpyWithAttributesAsync out of a source build against an older 13.x header, while the run-time gates, which looked at the bindings and the driver, still reported the features as available; the calls then failed with CUDA_ERROR_NOT_SUPPORTED (NVIDIA#2783, "guards disable working features"). Both features also went through void*-typed C++ shims and has_*() presence probes whose only purpose was to avoid an eager cimport of a cydriver function that cuda-bindings 13.0 did not export. With the cuda-bindings floor in place (13.4.1 for CUDA 13), the cimport is safe, so the shims, the presence probes and the two fenced pointers go away: _device_resources.pyx and _buffer.pyx call cydriver.cuDevSmResourceSplit and cydriver.cuMemcpyWithAttributesAsync directly under the existing `IF CUDA_CORE_BUILD_MAJOR >= 13`, gated on the driver version alone. build_hooks.py now defines CUDA_CORE_BUILD_MAJOR (until now a Cython-only compile-time constant) and CUDA_CORE_MIN_CUDA_VERSION (the floor's major.minor) for the C++ compiler. The new _cpp/rt/versions.hpp, the first include of the tree via types.hpp, re-checks cuda.h against both with #error, so a build that bypasses the backend still cannot compile against an unsupported header. tests/test_rt_layout.py enforces that versions.hpp is the only file under _cpp/ that names CUDA_VERSION. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…VIDIA#2783) The C++ under _cpp/rt/ used to call the driver through cuda-bindings' Cython wrappers, extracted from cydriver.__pyx_capi__ at import. A wrapper for a function the driver lacks raises a Python exception the C++ never sees, and a wrapper the installed cuda-bindings lacks made the pointer optional and probed for null at every use. driver_api.hpp now lists every driver function the C++ calls, with the CUDA version cuda-bindings requests it at. The table is filled on first use from cuda.bindings._internal.driver._inspect_function_pointers(), which holds the driver's own entry points, so `import cuda.core` never touches the driver. Calls go through DRIVER_CALL(name, args...), which fills the table if needed and, when a pointer is still null after the fill (a missing feature gate or a failed fill), reports once and returns an error status from a trampoline instead of dereferencing null. The fill rejects a driver older than the CUDA major series. pw_ deleter wrappers use the same path, so the C++ never null-checks a pointer; functions the driver may lack are gated on the driver version in Cython. NVRTC, NVVM and nvJitLink each have their own one-entry table, filled when a program or linker handle is created. Consequences in this commit: the null-check fallbacks to CUDA_ERROR_NOT_SUPPORTED are gone; green-context stream creation is gated in _stream.pyx on driver 12.5; deviceptr_import_ipc resolves the table before taking ipc_import_mutex and keeps raw calls under the lock (marked `// raw:`); _rt.pyx no longer imports cydriver/cynvrtc/cynvvm/ cynvjitlink or reads __pyx_capi__. test_rt_layout.py checks the table against cuda-bindings' loader and forbids raw p_ calls elsewhere. DESIGN.md and AGENTS.md describe the mechanism and settle the GIL contract for entry points.
…as a floor (NVIDIA#2783) Every cuda-bindings cuda.core accepts (12.9.8+, 13.4.1+) has the full API surface of its CUDA major, so a run-time check of the cuda-bindings version says nothing a build-major fence does not. Double checks (driver and bindings) become driver-only checks; checks of the bindings minor become `IF CUDA_CORE_BUILD_MAJOR` fences in Cython or a comparison with the new `cuda.core._utils.version.BUILD_CUDA_MAJOR` in Python; checks that every accepted cuda-bindings satisfies are deleted, along with the fallbacks they guarded. - Kernel argument info, green contexts, workqueues, graph node updates, conditional graph nodes: driver-only gates. - cuGraphNodeGetParams (13.2): a driver gate on the CUDA 13 build, and the call goes through cydriver instead of the Python driver layer. - Managed memory NUMA locations, virtual memory MANAGED type: build fence (plus the 13.0 driver where a driver entry point is involved). - Copy attribute enums (CUDA 12.8): unconditional. - NVVM, nvJitLink, NVRTC PCH bindings: always present; only the library load is probed. - Checkpoint: the CUDA 13 build only (CUcheckpointGpuPair is a CUDA 13 type; the CUDA 12 bindings never had it), stated plainly. - cuda.core.system: NVML through cuda.bindings unconditionally; the driver/runtime fallbacks are gone and CUDA_BINDINGS_NVML_IS_COMPATIBLE is a deprecated constant True. system.typing exports DeviceArch and FieldId unconditionally. - Error-enum explanations come from cuda-bindings' enum docstrings; the frozen 13.1.1 tables and their loader are removed. - cuda.bindings.utils.warn_if_cuda_major_version_mismatch (13.3+) is called on the CUDA 13 build only, instead of try/except ImportError. cy_binding_version() has no callers left and is removed; binding_version() stays. Tests follow the same rule set.
NVIDIA#2783) test_checkpoint.py probed cuda.core.checkpoint._REQUIRED_BINDING_ATTRS, which the gate collapse removed, so the whole session failed at collection. The helpers build CUcheckpointGpuPair, a CUDA 13 type; skip them on the CUDA 12 build instead.
…untime-floor # Conflicts: # cuda_core/tests/system/test_system_events.py
…ts (NVIDIA#2783) stubgen-pyx regenerated version.pyi (BUILD_CUDA_MAJOR), system/_system.pyi (the constant is now True) and system/_device.pyi (NVLink 6.0 mapping is unconditional). ci/tools/cuda_core_bindings_floor.py reads the CUDA_BINDINGS_FLOOR literal with ast instead of executing the module; its tests and test_build_hooks.py load modules with importlib instead of exec, and the stubbed configuration check takes any arguments.
Contributor
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
github-actions
Bot
added
CI/CD
Contributor
Author
|
/ok to test af88337 |
Contributor
|
…A#1824 namespace repair (NVIDIA#2783) The build-configuration check imported cuda.bindings directly. In an isolated PEP 517 build with an in-tree backend (the wheel-from-sdist CI job), the project's own cuda/ directory is the whole `cuda` namespace, so the cuda-bindings pip installed into the build environment is not importable (NVIDIA#1824) and the check failed with "requires cuda-bindings to build". Import it the way _import_get_cuda_path_or_home() imports cuda.pathfinder: add the site-packages cuda/ directory to the namespace path when the plain import fails. The pxd-path block uses the same helper. Package metadata is not a substitute: pip's in-process hook runner forwards find_distributions without the requested name, so it returns this project's own metadata for `cuda-bindings`. test_managed_ops.py asserted the old wording of the NUMA-host message ("cuda-bindings 13.0+"), which now names the CUDA 13 build; the four regexes follow.
Contributor
Author
|
/ok to test b4fedec |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the plan in #2783:
cuda.corerequires a per-majorcuda-bindingsfloor, and with that floor in place it stops working around oldercuda-bindingsreleases.What changes
1. A
cuda-bindingsfloor per CUDA major, at build and run time.cuda/core/_bindings_floor.pyrecords 12.9.8 for CUDA 12 and 13.4.1 for CUDA 13.build_hooks.pyrefuses to build against an oldercuda-bindingsand writes the header version it compiled against into a generated_build_info.py.import cuda.corechecks the installedcuda-bindingsagainst the build's major and minimum before importing any extension, and fails with a message that names the version found, the version required, and thepipcommand that fixes it. Thecu12/cu13extras pin the floors.2. The header rule. A source build requires a
cuda.hof the same major.minor as itscuda-bindings(the headercuda-bindingswas generated from). Any other configuration fails the build with a clear message. This is what makes the pointer table below sound:cuda-bindingskeys its function table by the symbolcuda.hmaps each name to (cuStreamDestroy->cuStreamDestroy_v2), so both sides must see the same header.3. C++ fences on the CUDA major only.
build_hooks.pypasses-DCUDA_CORE_BUILD_MAJORand-DCUDA_CORE_MIN_CUDA_VERSION; the new_cpp/rt/versions.hppre-checks the header against both. Every#if CUDA_VERSION >= 130x0fence is gone;tests/test_rt_layout.pyenforces thatversions.hppis the only file under_cpp/that namesCUDA_VERSION.4. Driver calls through
cuda-bindings' resolved pointers. The C++ under_cpp/rt/used to call the driver throughcuda-bindings' Cython wrappers, extracted fromcydriver.__pyx_capi__. A wrapper for a function the driver lacks raises a Python exception the C++ never sees.driver_api.hppnow lists the 61 driver functions the C++ calls in an X-macro table, each with the CUDA versioncuda-bindingsrequests it at. The table is filled on first use fromcuda.bindings._internal.driver._inspect_function_pointers(), which holds the driver's own entry points, soimport cuda.corenever loads the driver. Calls go throughDRIVER_CALL(name, args...); a pointer still null after the fill (a missing feature gate, or a failed fill) is reported once through the warning path and returns an error status from a trampoline of the right signature, never a null dereference and never an exception. The fill rejects a driver older than the CUDA major series. Thepw_deleter wrappers use the same path, so the C++ never null-checks a pointer; functions the installed driver may lack are gated on the driver version in Cython. NVRTC, NVVM and nvJitLink each have their own one-entry table.5. Feature gates depend on the driver alone. Checks that also inspected the
cuda-bindingsversion are gone: double checks (driver + bindings) became driver-only checks, checks of the bindings minor became build-major fences, and checks every acceptedcuda-bindingssatisfies were deleted along with the fallbacks they guarded.cuGraphNodeGetParamsnow goes throughcydriverinstead of the Python driver layer. The frozen error-explanation tables, the NVML fallbacks incuda.core.system, and the NVVM, nvJitLink and NVRTC PCH probes are removed.User-visible changes
import cuda.corewith acuda-bindingsolder than the floor fails with an actionableImportError. Previously it imported and failed later (missing C function, silently disabled feature, or crash).cuda-bindingsversion now name a driver version.cuda.core.system.CUDA_BINDINGS_NVML_IS_COMPATIBLEis alwaysTrueand deprecated.cuda.core.checkpointstates that it requires the CUDA 13 build. It did in effect before: the CUDA 12cuda-bindingslackCUcheckpointGpuPair.cuda.core.system.typingexportsDeviceArchandFieldIdunconditionally.CUDA_ERROR_NOT_SUPPORTEDfallback in C++).Supported CUDA drivers and CUDA Toolkit libraries are unchanged. Release notes:
docs/source/release/1.3.0-notes.rst.CI
ci/tools/env-varsreplaces thepublishedbindings source withfloor: the prior-major test rows that do not match the backport branch's minor installcuda-bindings==<floor>(read from the wheel byci/tools/cuda_core_bindings_floor.py), so the floor itself is tested.tests/test_bindings_floor.pychecks thatpyproject.tomlandci/versions.ymlagree with_bindings_floor.py.Testing
cuda_coretest suite on a local 2-GPU system (H20, driver 615.71) againstcuda-bindings13.4.1: 4307 passed, 121 skipped, 5 failed. The five failures are test-infrastructure issues unrelated to this change:test_device_cpu_affinityenumerates NVML devices where the process can only see a subset of the GPUs (fixed on main by tests: enumerate CUDA devices for GPU-only system checks #2916, merged here), and threetest_object_code_load_rdc*cases whose fixture builds object files inside the checkout, where a file mirror on the test host removes them mid-build.import cuda.coredoes not maplibcuda;Device.set_current()does; all 61 table entries resolve to non-null driver entry points.test_bindings_floor.py,test_build_hooks.py(build configuration checks, requirement string, define macros),test_rt_layout.py(table agrees withcuda-bindings' loader; no rawp_calls outside the table machinery),ci/tools/tests/test_cuda_core_bindings_floor.py.Notes for review
The API check reports one finding, and it is intentional.
cuda.core.system.CUDA_BINDINGS_NVML_IS_COMPATIBLEis declaredbool = Truein the regenerated stub where it used to be a barebool(its value was computed at import). The constant is alwaysTrueunder the floor and is deprecated; nothing else in the public API changed._cpp/rt/DESIGN.mdandAGENTS.mddescribe the pointer table and settle the GIL contract for the C++ entry points: they work with or without the GIL, never take a C++ lock while acquiring it, and only the reporting wrappers and the one-time table fill acquire it.deviceptr_import_ipcresolves the table before takingipc_import_mutexand keeps raw pointer calls under the lock, marked// raw:and linted.The commits are independent steps and read well one at a time: floor, major-only fences, pointer table, gate collapse, test fix.
After feat(cuda.core): support nvJitLink incremental linking #2867 merges, its
TODO(#2783)in_linker.pyxbecomes a direct cimport. cuda.core: fix pool setup and builder teardown under stream capture #2838 touches_rt.pyx/graph.cppand needs a rebase after this lands.The conda-forge feedstock should pin
cuda-bindingsper the floors (out of this repo).🤖 Generated with Claude Code