From 045bf5c879f47aa674c2276458c82d3425213cdb Mon Sep 17 00:00:00 2001 From: Ramakrishna Prabhu Date: Thu, 17 Sep 2026 10:52:49 -0500 Subject: [PATCH 1/7] build(wheel): split libcuopt into per-component wheels Adds libcuopt-client, libcuopt-mathopt and libcuopt-routing, each staging only its own install components through scikit-build-core's install.components, which the per-component install rules in #1926 made possible. The shared CMake body moved to python/cmake/cuopt_wheel_build.cmake so the RPATH list and third-party lookups cannot drift between them. libcuopt keeps the ld script and the gRPC server binary and now depends on the three rather than bundling the libraries a second time. Its load_library delegates to the component packages, client first since mathopt and routing both carry a DT_NEEDED on it. cuopt_cli's console script moves to libcuopt-mathopt, where the binary installs. Dependencies were split from measured DT_NEEDED: client needs only rmm and rapids-logger and no CUDA math libraries, routing adds cublas through a narrower cuda_wheels_routing group, and cudss, nccl and nvjitlink stay with mathopt alone. A routing-only install no longer pulls roughly 1.2 GB of libraries it never calls. Each wheel still configures the whole C++ tree, so this trades CI build time for install size; sccache is what keeps that affordable. Building once and packaging three times would need build_wheel.sh restructured, which is worth doing separately if the cost shows up. Co-Authored-By: Claude Opus 5 --- ci/build_wheel_libcuopt_client.sh | 131 ++++++++++++ ci/build_wheel_libcuopt_mathopt.sh | 131 ++++++++++++ ci/build_wheel_libcuopt_routing.sh | 131 ++++++++++++ dependencies.yaml | 111 ++++++++++ python/cmake/cuopt_wheel_build.cmake | 83 ++++++++ python/libcuopt/libcuopt/load.py | 49 ++--- python/libcuopt/pyproject.toml | 6 +- python/libcuopt_client/CMakeLists.txt | 25 +++ python/libcuopt_client/LICENSE | 201 ++++++++++++++++++ python/libcuopt_client/README.md | 5 + .../libcuopt_client/libcuopt_client/VERSION | 1 + .../libcuopt_client/__init__.py | 7 + .../libcuopt_client/_version.py | 22 ++ .../libcuopt_client/libcuopt_client/load.py | 113 ++++++++++ python/libcuopt_client/pyproject.toml | 78 +++++++ python/libcuopt_mathopt/CMakeLists.txt | 25 +++ python/libcuopt_mathopt/LICENSE | 201 ++++++++++++++++++ python/libcuopt_mathopt/README.md | 5 + .../libcuopt_mathopt/libcuopt_mathopt/VERSION | 1 + .../libcuopt_mathopt/__init__.py | 7 + .../libcuopt_mathopt/_cli_wrapper.py | 17 ++ .../libcuopt_mathopt/_version.py | 22 ++ .../libcuopt_mathopt/libcuopt_mathopt/load.py | 118 ++++++++++ python/libcuopt_mathopt/pyproject.toml | 86 ++++++++ python/libcuopt_routing/CMakeLists.txt | 25 +++ python/libcuopt_routing/LICENSE | 201 ++++++++++++++++++ python/libcuopt_routing/README.md | 5 + .../libcuopt_routing/libcuopt_routing/VERSION | 1 + .../libcuopt_routing/__init__.py | 7 + .../libcuopt_routing/_version.py | 22 ++ .../libcuopt_routing/libcuopt_routing/load.py | 118 ++++++++++ python/libcuopt_routing/pyproject.toml | 80 +++++++ 32 files changed, 2005 insertions(+), 30 deletions(-) create mode 100755 ci/build_wheel_libcuopt_client.sh create mode 100755 ci/build_wheel_libcuopt_mathopt.sh create mode 100755 ci/build_wheel_libcuopt_routing.sh create mode 100644 python/cmake/cuopt_wheel_build.cmake create mode 100644 python/libcuopt_client/CMakeLists.txt create mode 100644 python/libcuopt_client/LICENSE create mode 100644 python/libcuopt_client/README.md create mode 100644 python/libcuopt_client/libcuopt_client/VERSION create mode 100644 python/libcuopt_client/libcuopt_client/__init__.py create mode 100644 python/libcuopt_client/libcuopt_client/_version.py create mode 100644 python/libcuopt_client/libcuopt_client/load.py create mode 100644 python/libcuopt_client/pyproject.toml create mode 100644 python/libcuopt_mathopt/CMakeLists.txt create mode 100644 python/libcuopt_mathopt/LICENSE create mode 100644 python/libcuopt_mathopt/README.md create mode 100644 python/libcuopt_mathopt/libcuopt_mathopt/VERSION create mode 100644 python/libcuopt_mathopt/libcuopt_mathopt/__init__.py create mode 100644 python/libcuopt_mathopt/libcuopt_mathopt/_cli_wrapper.py create mode 100644 python/libcuopt_mathopt/libcuopt_mathopt/_version.py create mode 100644 python/libcuopt_mathopt/libcuopt_mathopt/load.py create mode 100644 python/libcuopt_mathopt/pyproject.toml create mode 100644 python/libcuopt_routing/CMakeLists.txt create mode 100644 python/libcuopt_routing/LICENSE create mode 100644 python/libcuopt_routing/README.md create mode 100644 python/libcuopt_routing/libcuopt_routing/VERSION create mode 100644 python/libcuopt_routing/libcuopt_routing/__init__.py create mode 100644 python/libcuopt_routing/libcuopt_routing/_version.py create mode 100644 python/libcuopt_routing/libcuopt_routing/load.py create mode 100644 python/libcuopt_routing/pyproject.toml diff --git a/ci/build_wheel_libcuopt_client.sh b/ci/build_wheel_libcuopt_client.sh new file mode 100755 index 0000000000..5246002c37 --- /dev/null +++ b/ci/build_wheel_libcuopt_client.sh @@ -0,0 +1,131 @@ +#!/bin/bash +# SPDX-FileCopyrightText: Copyright (c) 2023-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +set -euo pipefail + +# Configures the full C++ tree like every other cuOpt wheel, but stages only this +# component's install rules -- see install.components in the package's +# pyproject.toml. sccache is what keeps the repeated configure affordable. + +source rapids-init-pip + +package_name="libcuopt_client" +package_dir="python/libcuopt_client" + +# Install rockylinux repo +if command -v dnf &> /dev/null; then + bash ci/utils/update_rockylinux_repo.sh +fi + +# Install Boost and TBB +bash ci/utils/install_boost_tbb.sh + +# Install libuuid and LLVM's OpenMP runtime +if command -v dnf &> /dev/null; then + # LLVM Toolset is distributed as a module on Rocky/RHEL 8. + dnf module install -y llvm-toolset + dnf install -y libuuid-devel libomp-devel +elif command -v apt-get &> /dev/null; then + apt-get update + apt-get install -y uuid-dev libomp-dev +fi + +# Install Protobuf + gRPC (protoc + grpc_cpp_plugin) +bash ci/utils/install_protobuf_grpc.sh + +# Compile with GCC, but use LLVM libomp as the OpenMP runtime bundled in the wheel. Resolve the +# versioned ELF library rather than an unversioned linker script or compiler-toolset indirection. +LIBOMP_LIBRARY="$( + ldconfig -p | + awk '$1 ~ /^libomp\.so(\.[0-9]+)*$/ && !library { library = $NF } + END { print library }' +)" +if [[ "${LIBOMP_LIBRARY}" != /* || ! -f "${LIBOMP_LIBRARY}" ]]; then + echo "Could not resolve the LLVM OpenMP runtime: '${LIBOMP_LIBRARY}'" >&2 + exit 1 +fi + +echo "Using LLVM OpenMP runtime: ${LIBOMP_LIBRARY}" + +export SKBUILD_CMAKE_ARGS="-DOpenMP_gomp_LIBRARY:FILEPATH=${LIBOMP_LIBRARY}" + +# OpenSSL 3 hints for libcuopt's own find_package(OpenSSL). +# +# install_protobuf_grpc.sh links gRPC against OpenSSL 3 (see that script for +# rationale). libcuopt then re-resolves OpenSSL via find_package because +# gRPC's imported targets propagate it transitively. On Rocky/RHEL 8 the +# EPEL openssl3-devel package installs in non-default paths, so we have to +# point CMake at them; on Rocky/RHEL 9+ and Ubuntu 22.04+ the default +# OpenSSL is already 3.x and no hints are needed. +if [ -f /etc/os-release ]; then + . /etc/os-release + if [[ "$ID" == "rocky" || "$ID" == "centos" || "$ID" == "rhel" || "$ID" == "fedora" ]] && \ + [[ "${VERSION_ID%%.*}" == "8" ]]; then + SKBUILD_CMAKE_ARGS="${SKBUILD_CMAKE_ARGS};-DOPENSSL_INCLUDE_DIR=/usr/include/openssl3;-DOPENSSL_SSL_LIBRARY=/usr/lib64/openssl3/libssl.so;-DOPENSSL_CRYPTO_LIBRARY=/usr/lib64/openssl3/libcrypto.so" + fi +fi + +# For pull requests we are enabling assert mode. +if [ "$RAPIDS_BUILD_TYPE" = "pull-request" ]; then + echo "Building in assert mode" + export SKBUILD_CMAKE_ARGS="${SKBUILD_CMAKE_ARGS};-DDEFINE_ASSERT=True" +else + echo "Building in release mode" +fi + +# Install cudss +bash ci/utils/install_cudss.sh + +rapids-logger "Generating build requirements" + +rapids-dependency-file-generator \ + --output requirements \ + --file-key "py_build_${package_name}" \ + --file-key "py_rapids_build_${package_name}" \ + --matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION};cuda_suffixed=true" \ +| tee /tmp/requirements-build.txt + +rapids-logger "Installing build requirements" +rapids-pip-retry install \ + -v \ + --prefer-binary \ + -r /tmp/requirements-build.txt + +# build with '--no-build-isolation', for better sccache hit rate +# 0 really means "add --no-build-isolation" (ref: https://github.com/pypa/pip/issues/5735) +export PIP_NO_BUILD_ISOLATION=0 + + +EXCLUDE_ARGS=( + --exclude "libraft.so" + --exclude "libcublas.so.*" + --exclude "libcublasLt.so.*" + --exclude "libcuda.so.1" + --exclude "libcudss.so.*" + --exclude "libcurand.so.*" + --exclude "libcusolver.so.*" + --exclude "libcusparse.so.*" + --exclude "libnccl.so.*" + --exclude "libnvJitLink.so*" + --exclude "librapids_logger.so" + --exclude "librmm.so" + # OpenSSL 3 is intentionally NOT bundled. Resolving libssl.so.3 / libcrypto.so.3 + # at runtime via the host (or container image) keeps libcrypto and the FIPS + # provider (system or mounted) byte-version-matched, which is required for + # the FIPS provider's HMAC integrity check and avoids loading two libcrypto.so.3 + # in the same process. Hosts must provide libssl.so.3 / libcrypto.so.3 (Ubuntu + # 22.04+, RHEL/Rocky 9+, manylinux_2_28+ with openssl3, Debian 12+). + --exclude "libssl.so.3" + --exclude "libcrypto.so.3" +) + +ci/build_wheel.sh ${package_name} ${package_dir} + +mkdir -p final_dist +python -m auditwheel repair "${EXCLUDE_ARGS[@]}" -w "${RAPIDS_WHEEL_BLD_OUTPUT_DIR}" ${package_dir}/dist/* + +ci/validate_wheel.sh ${package_dir} "${RAPIDS_WHEEL_BLD_OUTPUT_DIR}" + +RAPIDS_PACKAGE_NAME="$(rapids-artifact-name wheel_cpp libcuopt_client cuopt --cuda "$RAPIDS_CUDA_VERSION")" +export RAPIDS_PACKAGE_NAME diff --git a/ci/build_wheel_libcuopt_mathopt.sh b/ci/build_wheel_libcuopt_mathopt.sh new file mode 100755 index 0000000000..e70d0ca468 --- /dev/null +++ b/ci/build_wheel_libcuopt_mathopt.sh @@ -0,0 +1,131 @@ +#!/bin/bash +# SPDX-FileCopyrightText: Copyright (c) 2023-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +set -euo pipefail + +# Configures the full C++ tree like every other cuOpt wheel, but stages only this +# component's install rules -- see install.components in the package's +# pyproject.toml. sccache is what keeps the repeated configure affordable. + +source rapids-init-pip + +package_name="libcuopt_mathopt" +package_dir="python/libcuopt_mathopt" + +# Install rockylinux repo +if command -v dnf &> /dev/null; then + bash ci/utils/update_rockylinux_repo.sh +fi + +# Install Boost and TBB +bash ci/utils/install_boost_tbb.sh + +# Install libuuid and LLVM's OpenMP runtime +if command -v dnf &> /dev/null; then + # LLVM Toolset is distributed as a module on Rocky/RHEL 8. + dnf module install -y llvm-toolset + dnf install -y libuuid-devel libomp-devel +elif command -v apt-get &> /dev/null; then + apt-get update + apt-get install -y uuid-dev libomp-dev +fi + +# Install Protobuf + gRPC (protoc + grpc_cpp_plugin) +bash ci/utils/install_protobuf_grpc.sh + +# Compile with GCC, but use LLVM libomp as the OpenMP runtime bundled in the wheel. Resolve the +# versioned ELF library rather than an unversioned linker script or compiler-toolset indirection. +LIBOMP_LIBRARY="$( + ldconfig -p | + awk '$1 ~ /^libomp\.so(\.[0-9]+)*$/ && !library { library = $NF } + END { print library }' +)" +if [[ "${LIBOMP_LIBRARY}" != /* || ! -f "${LIBOMP_LIBRARY}" ]]; then + echo "Could not resolve the LLVM OpenMP runtime: '${LIBOMP_LIBRARY}'" >&2 + exit 1 +fi + +echo "Using LLVM OpenMP runtime: ${LIBOMP_LIBRARY}" + +export SKBUILD_CMAKE_ARGS="-DOpenMP_gomp_LIBRARY:FILEPATH=${LIBOMP_LIBRARY}" + +# OpenSSL 3 hints for libcuopt's own find_package(OpenSSL). +# +# install_protobuf_grpc.sh links gRPC against OpenSSL 3 (see that script for +# rationale). libcuopt then re-resolves OpenSSL via find_package because +# gRPC's imported targets propagate it transitively. On Rocky/RHEL 8 the +# EPEL openssl3-devel package installs in non-default paths, so we have to +# point CMake at them; on Rocky/RHEL 9+ and Ubuntu 22.04+ the default +# OpenSSL is already 3.x and no hints are needed. +if [ -f /etc/os-release ]; then + . /etc/os-release + if [[ "$ID" == "rocky" || "$ID" == "centos" || "$ID" == "rhel" || "$ID" == "fedora" ]] && \ + [[ "${VERSION_ID%%.*}" == "8" ]]; then + SKBUILD_CMAKE_ARGS="${SKBUILD_CMAKE_ARGS};-DOPENSSL_INCLUDE_DIR=/usr/include/openssl3;-DOPENSSL_SSL_LIBRARY=/usr/lib64/openssl3/libssl.so;-DOPENSSL_CRYPTO_LIBRARY=/usr/lib64/openssl3/libcrypto.so" + fi +fi + +# For pull requests we are enabling assert mode. +if [ "$RAPIDS_BUILD_TYPE" = "pull-request" ]; then + echo "Building in assert mode" + export SKBUILD_CMAKE_ARGS="${SKBUILD_CMAKE_ARGS};-DDEFINE_ASSERT=True" +else + echo "Building in release mode" +fi + +# Install cudss +bash ci/utils/install_cudss.sh + +rapids-logger "Generating build requirements" + +rapids-dependency-file-generator \ + --output requirements \ + --file-key "py_build_${package_name}" \ + --file-key "py_rapids_build_${package_name}" \ + --matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION};cuda_suffixed=true" \ +| tee /tmp/requirements-build.txt + +rapids-logger "Installing build requirements" +rapids-pip-retry install \ + -v \ + --prefer-binary \ + -r /tmp/requirements-build.txt + +# build with '--no-build-isolation', for better sccache hit rate +# 0 really means "add --no-build-isolation" (ref: https://github.com/pypa/pip/issues/5735) +export PIP_NO_BUILD_ISOLATION=0 + + +EXCLUDE_ARGS=( + --exclude "libraft.so" + --exclude "libcublas.so.*" + --exclude "libcublasLt.so.*" + --exclude "libcuda.so.1" + --exclude "libcudss.so.*" + --exclude "libcurand.so.*" + --exclude "libcusolver.so.*" + --exclude "libcusparse.so.*" + --exclude "libnccl.so.*" + --exclude "libnvJitLink.so*" + --exclude "librapids_logger.so" + --exclude "librmm.so" + # OpenSSL 3 is intentionally NOT bundled. Resolving libssl.so.3 / libcrypto.so.3 + # at runtime via the host (or container image) keeps libcrypto and the FIPS + # provider (system or mounted) byte-version-matched, which is required for + # the FIPS provider's HMAC integrity check and avoids loading two libcrypto.so.3 + # in the same process. Hosts must provide libssl.so.3 / libcrypto.so.3 (Ubuntu + # 22.04+, RHEL/Rocky 9+, manylinux_2_28+ with openssl3, Debian 12+). + --exclude "libssl.so.3" + --exclude "libcrypto.so.3" +) + +ci/build_wheel.sh ${package_name} ${package_dir} + +mkdir -p final_dist +python -m auditwheel repair "${EXCLUDE_ARGS[@]}" -w "${RAPIDS_WHEEL_BLD_OUTPUT_DIR}" ${package_dir}/dist/* + +ci/validate_wheel.sh ${package_dir} "${RAPIDS_WHEEL_BLD_OUTPUT_DIR}" + +RAPIDS_PACKAGE_NAME="$(rapids-artifact-name wheel_cpp libcuopt_mathopt cuopt --cuda "$RAPIDS_CUDA_VERSION")" +export RAPIDS_PACKAGE_NAME diff --git a/ci/build_wheel_libcuopt_routing.sh b/ci/build_wheel_libcuopt_routing.sh new file mode 100755 index 0000000000..87acfab67d --- /dev/null +++ b/ci/build_wheel_libcuopt_routing.sh @@ -0,0 +1,131 @@ +#!/bin/bash +# SPDX-FileCopyrightText: Copyright (c) 2023-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +set -euo pipefail + +# Configures the full C++ tree like every other cuOpt wheel, but stages only this +# component's install rules -- see install.components in the package's +# pyproject.toml. sccache is what keeps the repeated configure affordable. + +source rapids-init-pip + +package_name="libcuopt_routing" +package_dir="python/libcuopt_routing" + +# Install rockylinux repo +if command -v dnf &> /dev/null; then + bash ci/utils/update_rockylinux_repo.sh +fi + +# Install Boost and TBB +bash ci/utils/install_boost_tbb.sh + +# Install libuuid and LLVM's OpenMP runtime +if command -v dnf &> /dev/null; then + # LLVM Toolset is distributed as a module on Rocky/RHEL 8. + dnf module install -y llvm-toolset + dnf install -y libuuid-devel libomp-devel +elif command -v apt-get &> /dev/null; then + apt-get update + apt-get install -y uuid-dev libomp-dev +fi + +# Install Protobuf + gRPC (protoc + grpc_cpp_plugin) +bash ci/utils/install_protobuf_grpc.sh + +# Compile with GCC, but use LLVM libomp as the OpenMP runtime bundled in the wheel. Resolve the +# versioned ELF library rather than an unversioned linker script or compiler-toolset indirection. +LIBOMP_LIBRARY="$( + ldconfig -p | + awk '$1 ~ /^libomp\.so(\.[0-9]+)*$/ && !library { library = $NF } + END { print library }' +)" +if [[ "${LIBOMP_LIBRARY}" != /* || ! -f "${LIBOMP_LIBRARY}" ]]; then + echo "Could not resolve the LLVM OpenMP runtime: '${LIBOMP_LIBRARY}'" >&2 + exit 1 +fi + +echo "Using LLVM OpenMP runtime: ${LIBOMP_LIBRARY}" + +export SKBUILD_CMAKE_ARGS="-DOpenMP_gomp_LIBRARY:FILEPATH=${LIBOMP_LIBRARY}" + +# OpenSSL 3 hints for libcuopt's own find_package(OpenSSL). +# +# install_protobuf_grpc.sh links gRPC against OpenSSL 3 (see that script for +# rationale). libcuopt then re-resolves OpenSSL via find_package because +# gRPC's imported targets propagate it transitively. On Rocky/RHEL 8 the +# EPEL openssl3-devel package installs in non-default paths, so we have to +# point CMake at them; on Rocky/RHEL 9+ and Ubuntu 22.04+ the default +# OpenSSL is already 3.x and no hints are needed. +if [ -f /etc/os-release ]; then + . /etc/os-release + if [[ "$ID" == "rocky" || "$ID" == "centos" || "$ID" == "rhel" || "$ID" == "fedora" ]] && \ + [[ "${VERSION_ID%%.*}" == "8" ]]; then + SKBUILD_CMAKE_ARGS="${SKBUILD_CMAKE_ARGS};-DOPENSSL_INCLUDE_DIR=/usr/include/openssl3;-DOPENSSL_SSL_LIBRARY=/usr/lib64/openssl3/libssl.so;-DOPENSSL_CRYPTO_LIBRARY=/usr/lib64/openssl3/libcrypto.so" + fi +fi + +# For pull requests we are enabling assert mode. +if [ "$RAPIDS_BUILD_TYPE" = "pull-request" ]; then + echo "Building in assert mode" + export SKBUILD_CMAKE_ARGS="${SKBUILD_CMAKE_ARGS};-DDEFINE_ASSERT=True" +else + echo "Building in release mode" +fi + +# Install cudss +bash ci/utils/install_cudss.sh + +rapids-logger "Generating build requirements" + +rapids-dependency-file-generator \ + --output requirements \ + --file-key "py_build_${package_name}" \ + --file-key "py_rapids_build_${package_name}" \ + --matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION};cuda_suffixed=true" \ +| tee /tmp/requirements-build.txt + +rapids-logger "Installing build requirements" +rapids-pip-retry install \ + -v \ + --prefer-binary \ + -r /tmp/requirements-build.txt + +# build with '--no-build-isolation', for better sccache hit rate +# 0 really means "add --no-build-isolation" (ref: https://github.com/pypa/pip/issues/5735) +export PIP_NO_BUILD_ISOLATION=0 + + +EXCLUDE_ARGS=( + --exclude "libraft.so" + --exclude "libcublas.so.*" + --exclude "libcublasLt.so.*" + --exclude "libcuda.so.1" + --exclude "libcudss.so.*" + --exclude "libcurand.so.*" + --exclude "libcusolver.so.*" + --exclude "libcusparse.so.*" + --exclude "libnccl.so.*" + --exclude "libnvJitLink.so*" + --exclude "librapids_logger.so" + --exclude "librmm.so" + # OpenSSL 3 is intentionally NOT bundled. Resolving libssl.so.3 / libcrypto.so.3 + # at runtime via the host (or container image) keeps libcrypto and the FIPS + # provider (system or mounted) byte-version-matched, which is required for + # the FIPS provider's HMAC integrity check and avoids loading two libcrypto.so.3 + # in the same process. Hosts must provide libssl.so.3 / libcrypto.so.3 (Ubuntu + # 22.04+, RHEL/Rocky 9+, manylinux_2_28+ with openssl3, Debian 12+). + --exclude "libssl.so.3" + --exclude "libcrypto.so.3" +) + +ci/build_wheel.sh ${package_name} ${package_dir} + +mkdir -p final_dist +python -m auditwheel repair "${EXCLUDE_ARGS[@]}" -w "${RAPIDS_WHEEL_BLD_OUTPUT_DIR}" ${package_dir}/dist/* + +ci/validate_wheel.sh ${package_dir} "${RAPIDS_WHEEL_BLD_OUTPUT_DIR}" + +RAPIDS_PACKAGE_NAME="$(rapids-artifact-name wheel_cpp libcuopt_routing cuopt --cuda "$RAPIDS_CUDA_VERSION")" +export RAPIDS_PACKAGE_NAME diff --git a/dependencies.yaml b/dependencies.yaml index 3fd82474bc..61eb39910f 100644 --- a/dependencies.yaml +++ b/dependencies.yaml @@ -152,6 +152,92 @@ files: - depends_on_libraft_headers - depends_on_librmm - depends_on_rapids_logger + # One set of keys per component wheel. The point of the split is in py_run_*: + # only mathopt carries cudss / nccl / nvjitlink, so a routing-only install no + # longer pulls libraries it never calls. + py_build_libcuopt_client: + output: pyproject + pyproject_dir: python/libcuopt_client + extras: + table: build-system + includes: + - rapids_build_backend + py_rapids_build_libcuopt_client: + output: pyproject + pyproject_dir: python/libcuopt_client + extras: + table: tool.rapids-build-backend + key: requires + includes: + - build_common + - depends_on_librmm + - depends_on_rapids_logger + py_run_libcuopt_client: + output: pyproject + pyproject_dir: python/libcuopt_client + extras: + table: project + includes: + - depends_on_librmm + - depends_on_rapids_logger + py_build_libcuopt_mathopt: + output: pyproject + pyproject_dir: python/libcuopt_mathopt + extras: + table: build-system + includes: + - rapids_build_backend + py_rapids_build_libcuopt_mathopt: + output: pyproject + pyproject_dir: python/libcuopt_mathopt + extras: + table: tool.rapids-build-backend + key: requires + includes: + - build_common + - depends_on_libraft_headers + - depends_on_librmm + - depends_on_rapids_logger + py_run_libcuopt_mathopt: + output: pyproject + pyproject_dir: python/libcuopt_mathopt + extras: + table: project + includes: + - cuda_wheels + - depends_on_libcuopt_client + - depends_on_libraft_headers + - depends_on_librmm + - depends_on_rapids_logger + py_build_libcuopt_routing: + output: pyproject + pyproject_dir: python/libcuopt_routing + extras: + table: build-system + includes: + - rapids_build_backend + py_rapids_build_libcuopt_routing: + output: pyproject + pyproject_dir: python/libcuopt_routing + extras: + table: tool.rapids-build-backend + key: requires + includes: + - build_common + - depends_on_libraft_headers + - depends_on_librmm + - depends_on_rapids_logger + py_run_libcuopt_routing: + output: pyproject + pyproject_dir: python/libcuopt_routing + extras: + table: project + includes: + - cuda_wheels_routing + - depends_on_libcuopt_client + - depends_on_libraft_headers + - depends_on_librmm + - depends_on_rapids_logger py_build_cuopt: output: pyproject pyproject_dir: python/cuopt @@ -934,6 +1020,31 @@ dependencies: - libgomp >=9 + # cublas only. libcuopt_routing's DT_NEEDED is cublas, rmm and rapids_logger -- + # it never touches cudss, nccl, cusparse or nvjitlink, so pulling the full + # cuda_wheels set here would re-create exactly the bloat this split removes. + cuda_wheels_routing: + specific: + - output_types: [requirements, pyproject] + matrices: + - matrix: + use_cuda_wheels: "false" + packages: + - matrix: + cuda: "12.*" + use_cuda_wheels: "true" + packages: + - cuda-toolkit[cublas,cudart]==12.* + - matrix: + cuda: "13.*" + use_cuda_wheels: "true" + packages: + - &ctk_routing_cu13 cuda-toolkit[cublas,cudart]==13.* + # if no matching matrix selectors passed, list the CUDA 13 requirement, which is + # what populates pyproject.toml in source control + - matrix: + packages: + - *ctk_routing_cu13 cuda_wheels: specific: # cuOpt needs 'nvJitLink>={whatever-cuopt-was-built-against}' at runtime, and mixing diff --git a/python/cmake/cuopt_wheel_build.cmake b/python/cmake/cuopt_wheel_build.cmake new file mode 100644 index 0000000000..1ee1d07ffa --- /dev/null +++ b/python/cmake/cuopt_wheel_build.cmake @@ -0,0 +1,83 @@ +# cmake-format: off +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# cmake-format: on + +# Shared build for the libcuopt wheels. Every wheel configures the same C++ tree and +# differs only in which install components scikit-build-core stages into it, set by +# install.components in each pyproject.toml. Keeping this in one file means the RPATH +# list and the third-party lookups cannot drift between the per-component wheels. + +include(FetchContent) +FetchContent_Declare( + argparse + GIT_REPOSITORY https://github.com/p-ranav/argparse.git + GIT_TAG v3.2 +) +FetchContent_MakeAvailable(argparse) + +# gRPC must be available as an installed CMake package (gRPCConfig.cmake). +# On RockyLinux 8 wheel builds we install it in CI via ci/utils/install_protobuf_grpc.sh. +find_package(gRPC CONFIG REQUIRED) + +find_package(Boost 1.65 REQUIRED) +if(Boost_FOUND) + message(STATUS "Found Boost ${Boost_VERSION} in ${Boost_INCLUDE_DIRS}") +else() + message(FATAL_ERROR "Boost not found. Please install boost-devel.") +endif() + +include_directories(${Boost_INCLUDE_DIRS}) + +set(BUILD_TESTING OFF CACHE BOOL "Disable test build for papilo") +set(PAPILO_NO_BINARIES ON) +option(LUSOL "Disable LUSOL" OFF) + +set(BUILD_TESTS OFF) +set(BUILD_BENCHMARKS OFF) +set(CUOPT_BUILD_TESTUTIL OFF) + +add_subdirectory(../../cpp cuopt-cpp) + +# cuopt is an INTERFACE target (libcuopt.so is a linker script) and compiles nothing, +# so it has no use for argparse. cuopt_cli and cuopt_grpc_server, which do, already link +# argparse::argparse in cpp/CMakeLists.txt. +target_link_libraries(cuopt_cli PRIVATE + argparse +) + +set(rpaths + "$ORIGIN/../lib64" + "$ORIGIN/../../rapids_logger/lib64" + "$ORIGIN/../../librmm/lib64" + "$ORIGIN/../../nvidia/cudss/lib" + "$ORIGIN/../../nvidia/cublas/lib" + "$ORIGIN/../../nvidia/curand/lib" + "$ORIGIN/../../nvidia/cusolver/lib" + "$ORIGIN/../../nvidia/cusparse/lib" + "$ORIGIN/../../nvidia/nccl/lib" + "$ORIGIN/../../nvidia/nvjitlink/lib" +) + +# Add CUDA version-specific paths based on CUDA compiler version +message(STATUS "libcuopt: CMAKE_CUDA_COMPILER_VERSION = ${CMAKE_CUDA_COMPILER_VERSION}") +if(CMAKE_CUDA_COMPILER_VERSION VERSION_GREATER_EQUAL 13.0 AND CMAKE_CUDA_COMPILER_VERSION VERSION_LESS 14.0) + message(STATUS "libcuopt: Adding cu13 RPATH") + list(APPEND rpaths "$ORIGIN/../../nvidia/cu13/lib") +elseif(CMAKE_CUDA_COMPILER_VERSION VERSION_GREATER_EQUAL 12.0 AND CMAKE_CUDA_COMPILER_VERSION VERSION_LESS 13.0) + message(STATUS "libcuopt: Adding cu12 RPATH") + list(APPEND rpaths "$ORIGIN/../../nvidia/cu12/lib") +else() + message(WARNING "libcuopt: Unsupported CUDA version ${CMAKE_CUDA_COMPILER_VERSION}") +endif() +message(STATUS "libcuopt: Final RPATH = ${rpaths}") + +# cuopt is an INTERFACE target (libcuopt.so is a linker script), so it has no RPATH. +# cuopt_client is in the list because it PUBLIC-links rapids_logger, which +# build_wheel_libcuopt.sh excludes from vendoring, so the only way to find it is +# $ORIGIN/../../rapids_logger/lib64 from this list. +foreach(_target cuopt_routing cuopt_mathopt cuopt_client cuopt_cli cuopt_grpc_server) + if(TARGET ${_target}) + set_property(TARGET ${_target} APPEND PROPERTY INSTALL_RPATH ${rpaths}) + endif() +endforeach() diff --git a/python/libcuopt/libcuopt/load.py b/python/libcuopt/libcuopt/load.py index 86f5798f70..e7ea9a65b7 100644 --- a/python/libcuopt/libcuopt/load.py +++ b/python/libcuopt/libcuopt/load.py @@ -34,39 +34,30 @@ def _load_wheel_installation(soname: str): def load_library(): - """Dynamically load libcuopt.so and its dependencies""" + """Load every cuOpt component library. + + The libraries themselves live in the per-component packages now, so this + delegates rather than dlopen()ing them directly. The client goes first: + mathopt and routing both carry a DT_NEEDED on it. + """ + loaded = [] + + import libcuopt_client + + loaded.extend(libcuopt_client.load_library() or []) + + # routing is optional -- a SKIP_ROUTING_BUILD install will not have it. try: - # librmm and libraft must be loaded before libcuopt - # because libcuopt references them. - import libraft - import librmm - import rapids_logger - - rapids_logger.load_library() - librmm.load_library() - libraft.load_library() + import libcuopt_routing + + loaded.extend(libcuopt_routing.load_library() or []) except ModuleNotFoundError: pass - prefer_system_installation = ( - os.getenv("RAPIDS_LIBCUOPT_PREFER_SYSTEM_LIBRARY", "false").lower() - != "false" - ) - - # cuOpt ships as component libraries. libcuopt.so is a linker script, - # not an ELF object, so it cannot be dlopen()ed -- load the components - # instead. Each pulls its own dependencies in through DT_NEEDED, so this - # order only needs to be valid, not exhaustive. routing and grpc are - # optional (SKIP_ROUTING_BUILD, SKIP_GRPC_BUILD) and may be absent. - components = [ - ("libcuopt_routing.so", False), - ("libcuopt_mathopt.so", True), - ] - loaded = [] - for soname, required in components: - lib = _load_component(soname, prefer_system_installation, required) - if lib is not None: - loaded.append(lib) + import libcuopt_mathopt + + loaded.extend(libcuopt_mathopt.load_library() or []) + return loaded diff --git a/python/libcuopt/pyproject.toml b/python/libcuopt/pyproject.toml index d4b415ae6a..fd02478394 100644 --- a/python/libcuopt/pyproject.toml +++ b/python/libcuopt/pyproject.toml @@ -45,8 +45,9 @@ Source = "https://github.com/nvidia/cuopt" [project.entry-points."cmake.prefix"] libcuopt = "libcuopt" +# cuopt_cli moves to libcuopt-mathopt, which is where the binary installs. The +# server console script stays here until cuopt-grpc-server has its own wheel. [project.scripts] -cuopt_cli = "libcuopt._cli_wrapper:main" cuopt_grpc_server = "libcuopt._grpc_server_wrapper:main" [tool.pydistcheck] @@ -65,6 +66,9 @@ sdist.reproducible = true wheel.packages = ["libcuopt"] wheel.install-dir = "libcuopt" wheel.py-api = "py3" +# Only the ld script and the gRPC server binary: the three component libraries now +# arrive through this package's dependencies rather than being bundled again here. +install.components = ["dev", "grpc-server"] [tool.scikit-build.metadata.version] provider = "scikit_build_core.metadata.regex" diff --git a/python/libcuopt_client/CMakeLists.txt b/python/libcuopt_client/CMakeLists.txt new file mode 100644 index 0000000000..e5fde2401b --- /dev/null +++ b/python/libcuopt_client/CMakeLists.txt @@ -0,0 +1,25 @@ +# cmake-format: off +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# cmake-format: on + +cmake_minimum_required(VERSION 4.0 FATAL_ERROR) + +include(../../cmake/rapids_config.cmake) +include(rapids-cuda) +rapids_cuda_init_architectures(libcuopt-client-python) + +project( + libcuopt-client-python + VERSION "${RAPIDS_VERSION}" + LANGUAGES CXX CUDA +) + +# Already installed: nothing to build, the wheel just stages the component. +find_package(cuopt "${RAPIDS_VERSION}") +if(cuopt_FOUND) + return() +endif() +unset(cuopt_FOUND) + +include(../cmake/cuopt_wheel_build.cmake) diff --git a/python/libcuopt_client/LICENSE b/python/libcuopt_client/LICENSE new file mode 100644 index 0000000000..6ed9218d99 --- /dev/null +++ b/python/libcuopt_client/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "{}" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2025 NVIDIA Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/python/libcuopt_client/README.md b/python/libcuopt_client/README.md new file mode 100644 index 0000000000..7147977a60 --- /dev/null +++ b/python/libcuopt_client/README.md @@ -0,0 +1,5 @@ +# libcuopt-client + +Host-side cuOpt client: problem representation, parsers and the gRPC client. Contains no CUDA kernels. + +This package ships one component of cuOpt's C++ library. Install [`libcuopt`](https://pypi.org/project/libcuopt/) instead to get all of them. diff --git a/python/libcuopt_client/libcuopt_client/VERSION b/python/libcuopt_client/libcuopt_client/VERSION new file mode 100644 index 0000000000..6549ba6527 --- /dev/null +++ b/python/libcuopt_client/libcuopt_client/VERSION @@ -0,0 +1 @@ +26.10.00 diff --git a/python/libcuopt_client/libcuopt_client/__init__.py b/python/libcuopt_client/libcuopt_client/__init__.py new file mode 100644 index 0000000000..aad0391082 --- /dev/null +++ b/python/libcuopt_client/libcuopt_client/__init__.py @@ -0,0 +1,7 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from libcuopt_client._version import __git_commit__, __version__ +from libcuopt_client.load import load_library + +__all__ = ["__git_commit__", "__version__", "load_library"] diff --git a/python/libcuopt_client/libcuopt_client/_version.py b/python/libcuopt_client/libcuopt_client/_version.py new file mode 100644 index 0000000000..ca5b7efde7 --- /dev/null +++ b/python/libcuopt_client/libcuopt_client/_version.py @@ -0,0 +1,22 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +import importlib.resources + +__version__ = ( + importlib.resources.files(__package__) + .joinpath("VERSION") + .read_text() + .strip() +) +try: + __git_commit__ = ( + importlib.resources.files(__package__) + .joinpath("GIT_COMMIT") + .read_text() + .strip() + ) +except FileNotFoundError: + __git_commit__ = "" + +__all__ = ["__git_commit__", "__version__"] diff --git a/python/libcuopt_client/libcuopt_client/load.py b/python/libcuopt_client/libcuopt_client/load.py new file mode 100644 index 0000000000..3a663576da --- /dev/null +++ b/python/libcuopt_client/libcuopt_client/load.py @@ -0,0 +1,113 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Loader for the cuOpt client component. This is the leaf: it has no DT_NEEDED on +# another cuOpt library, which is what lets it be installed without the GPU stack. + +import ctypes +import os + +# Loading with RTLD_LOCAL adds the library itself to the loader's +# loaded library cache without loading any symbols into the global +# namespace. This allows libraries that express a dependency on +# this library to be loaded later and successfully satisfy this dependency +# without polluting the global symbol table with symbols from +# libcudf that could conflict with symbols from other DSOs. +PREFERRED_LOAD_FLAG = ctypes.RTLD_LOCAL + + +def _load_system_installation(soname: str): + """Try to dlopen() the library indicated by ``soname`` + Raises ``OSError`` if library cannot be loaded. + """ + return ctypes.CDLL(soname, PREFERRED_LOAD_FLAG) + + +def _load_wheel_installation(soname: str): + """Try to dlopen() the library indicated by ``soname`` + + Returns ``None`` if the library cannot be loaded. + """ + if os.path.isfile( + lib := os.path.join(os.path.dirname(__file__), "lib64", soname) + ): + return ctypes.CDLL(lib, PREFERRED_LOAD_FLAG) + return None + + +def load_library(): + """Dynamically load libcuopt_client.so and its dependencies.""" + try: + # librmm and rapids_logger must be loaded before libcuopt_client.so, + # which references them. + import librmm + import rapids_logger + + rapids_logger.load_library() + librmm.load_library() + except ModuleNotFoundError: + pass + + prefer_system_installation = ( + os.getenv("RAPIDS_LIBCUOPT_PREFER_SYSTEM_LIBRARY", "false").lower() + != "false" + ) + + lib = _load_component( + "libcuopt_client.so", prefer_system_installation, True + ) + return [lib] if lib is not None else [] + + +def _load_component( + soname: str, prefer_system_installation: bool, required: bool +): + """Load one cuOpt component. + + Returns the handle, or ``None`` if it could not be loaded. Failing to + load an optional component is silent; failing a required one warns. + """ + libcuopt_lib = None + try: + if prefer_system_installation: + # Prefer a system library if one is present to + # avoid clobbering symbols that other packages might expect, but if no + # other library is present use the one in the wheel. + try: + libcuopt_lib = _load_system_installation(soname) + except OSError: + # The fallback needs to sit inside the outer handler too. An + # optional component that is present but fails to load would + # otherwise raise out of here and stop the remaining components + # from being loaded at all. + libcuopt_lib = _load_wheel_installation(soname) + else: + # Prefer the libraries bundled in this package. If they aren't found + # (which might be the case in builds where the library + # was prebuilt before packaging the wheel), look for a + # system installation. + libcuopt_lib = _load_wheel_installation(soname) + if libcuopt_lib is None: + libcuopt_lib = _load_system_installation(soname) + except OSError as e: + # If none of the searches above succeed, just silently return None + # and rely on other mechanisms (like RPATHs on other DSOs) to + # help the loader find the library. + if not required: + return None + + import warnings + + warnings.warn( + f"Failed to load libcuopt library: {soname}. " + f"Error: {str(e)}. " + "Falling back to relying on system loader. " + "cuOpt functionality may be unavailable. " + f"This might lead to a generic error such as " + f"'{soname} missing' if the library cannot be found.", + RuntimeWarning, + ) + # The caller almost never needs to do anything with this library, but no + # harm in offering the option since this object at least provides a handle + # to inspect where libcuopt was loaded from. + return libcuopt_lib diff --git a/python/libcuopt_client/pyproject.toml b/python/libcuopt_client/pyproject.toml new file mode 100644 index 0000000000..3766809e38 --- /dev/null +++ b/python/libcuopt_client/pyproject.toml @@ -0,0 +1,78 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +[build-system] +build-backend = "rapids_build_backend.build" +requires = [ + "rapids-build-backend>=0.4.0,<0.5.0", + "scikit-build-core[pyproject]>=0.11.0", +] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`. + +[project] +name = "libcuopt-client" +dynamic = ["version"] +description = "cuOpt host-side client libraries (C++)" +readme = { file = "README.md", content-type = "text/markdown" } +authors = [ + { name = "NVIDIA Corporation" }, +] +license = "Apache-2.0" +requires-python = ">=3.11" +classifiers = [ + "Intended Audience :: Developers", + "Topic :: Database", + "Topic :: Scientific/Engineering", + "Programming Language :: C++", + "Environment :: GPU :: NVIDIA CUDA", + "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", + "Programming Language :: Python :: 3.13", + "Programming Language :: Python :: 3.14", +] +dependencies = [ + "librmm==26.10.*,>=0.0.0a0", + "rapids-logger==0.3.*", +] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`. + +[project.urls] +Homepage = "https://docs.nvidia.com/cuopt/introduction.html" +Source = "https://github.com/nvidia/cuopt" + +[project.entry-points."cmake.prefix"] +libcuopt_client = "libcuopt_client" + +[tool.pydistcheck] +select = [ + # note: max_allowed_size_compressed is set as a command-line arg + "distro-too-large-compressed", +] + +[tool.scikit-build] +build-dir = "build/{wheel_tag}" +cmake.build-type = "Release" +cmake.version = "CMakeLists.txt" +minimum-version = "build-system.requires" +ninja.make-fallback = false +sdist.reproducible = true +wheel.packages = ["libcuopt_client"] +wheel.install-dir = "libcuopt_client" +wheel.py-api = "py3" +# The whole point of this package: stage only this component's install rules, +# so the wheel carries one library instead of all three. +install.components = ["client", "client-dev"] + +[tool.scikit-build.metadata.version] +provider = "scikit_build_core.metadata.regex" +input = "libcuopt_client/VERSION" +regex = "(?P.*)" + +[tool.rapids-build-backend] +build-backend = "scikit_build_core.build" +dependencies-file = "../../dependencies.yaml" +matrix-entry = "cuda_suffixed=true;use_cuda_wheels=true" +requires = [ + "cmake>=4.0", + "librmm==26.10.*,>=0.0.0a0", + "ninja", + "rapids-logger==0.3.*", +] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`. diff --git a/python/libcuopt_mathopt/CMakeLists.txt b/python/libcuopt_mathopt/CMakeLists.txt new file mode 100644 index 0000000000..0ebf0ad09d --- /dev/null +++ b/python/libcuopt_mathopt/CMakeLists.txt @@ -0,0 +1,25 @@ +# cmake-format: off +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# cmake-format: on + +cmake_minimum_required(VERSION 4.0 FATAL_ERROR) + +include(../../cmake/rapids_config.cmake) +include(rapids-cuda) +rapids_cuda_init_architectures(libcuopt-mathopt-python) + +project( + libcuopt-mathopt-python + VERSION "${RAPIDS_VERSION}" + LANGUAGES CXX CUDA +) + +# Already installed: nothing to build, the wheel just stages the component. +find_package(cuopt "${RAPIDS_VERSION}") +if(cuopt_FOUND) + return() +endif() +unset(cuopt_FOUND) + +include(../cmake/cuopt_wheel_build.cmake) diff --git a/python/libcuopt_mathopt/LICENSE b/python/libcuopt_mathopt/LICENSE new file mode 100644 index 0000000000..6ed9218d99 --- /dev/null +++ b/python/libcuopt_mathopt/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "{}" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2025 NVIDIA Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/python/libcuopt_mathopt/README.md b/python/libcuopt_mathopt/README.md new file mode 100644 index 0000000000..11e17c2017 --- /dev/null +++ b/python/libcuopt_mathopt/README.md @@ -0,0 +1,5 @@ +# libcuopt-mathopt + +cuOpt LP / MILP / QP / SOCP solver library and the `cuopt_cli` command-line solver. + +This package ships one component of cuOpt's C++ library. Install [`libcuopt`](https://pypi.org/project/libcuopt/) instead to get all of them. diff --git a/python/libcuopt_mathopt/libcuopt_mathopt/VERSION b/python/libcuopt_mathopt/libcuopt_mathopt/VERSION new file mode 100644 index 0000000000..6549ba6527 --- /dev/null +++ b/python/libcuopt_mathopt/libcuopt_mathopt/VERSION @@ -0,0 +1 @@ +26.10.00 diff --git a/python/libcuopt_mathopt/libcuopt_mathopt/__init__.py b/python/libcuopt_mathopt/libcuopt_mathopt/__init__.py new file mode 100644 index 0000000000..e5128393ef --- /dev/null +++ b/python/libcuopt_mathopt/libcuopt_mathopt/__init__.py @@ -0,0 +1,7 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from libcuopt_mathopt._version import __git_commit__, __version__ +from libcuopt_mathopt.load import load_library + +__all__ = ["__git_commit__", "__version__", "load_library"] diff --git a/python/libcuopt_mathopt/libcuopt_mathopt/_cli_wrapper.py b/python/libcuopt_mathopt/libcuopt_mathopt/_cli_wrapper.py new file mode 100644 index 0000000000..78e95280b7 --- /dev/null +++ b/python/libcuopt_mathopt/libcuopt_mathopt/_cli_wrapper.py @@ -0,0 +1,17 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +import os +import sys + + +def main(): + """ + This connects to cli binary which situated under libcuopt/bin folder + + execv replaces this process rather than spawning a child, so signals sent + to the console script's pid reach the solver directly instead of stopping + at a Python parent that forwards nothing. + """ + cli_path = os.path.join(os.path.dirname(__file__), "bin", "cuopt_cli") + os.execv(cli_path, [cli_path] + sys.argv[1:]) diff --git a/python/libcuopt_mathopt/libcuopt_mathopt/_version.py b/python/libcuopt_mathopt/libcuopt_mathopt/_version.py new file mode 100644 index 0000000000..ca5b7efde7 --- /dev/null +++ b/python/libcuopt_mathopt/libcuopt_mathopt/_version.py @@ -0,0 +1,22 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +import importlib.resources + +__version__ = ( + importlib.resources.files(__package__) + .joinpath("VERSION") + .read_text() + .strip() +) +try: + __git_commit__ = ( + importlib.resources.files(__package__) + .joinpath("GIT_COMMIT") + .read_text() + .strip() + ) +except FileNotFoundError: + __git_commit__ = "" + +__all__ = ["__git_commit__", "__version__"] diff --git a/python/libcuopt_mathopt/libcuopt_mathopt/load.py b/python/libcuopt_mathopt/libcuopt_mathopt/load.py new file mode 100644 index 0000000000..e35fdd6315 --- /dev/null +++ b/python/libcuopt_mathopt/libcuopt_mathopt/load.py @@ -0,0 +1,118 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Loader for the cuOpt LP / MILP / QP / SOCP component. + +import ctypes +import os + +# Loading with RTLD_LOCAL adds the library itself to the loader's +# loaded library cache without loading any symbols into the global +# namespace. This allows libraries that express a dependency on +# this library to be loaded later and successfully satisfy this dependency +# without polluting the global symbol table with symbols from +# libcudf that could conflict with symbols from other DSOs. +PREFERRED_LOAD_FLAG = ctypes.RTLD_LOCAL + + +def _load_system_installation(soname: str): + """Try to dlopen() the library indicated by ``soname`` + Raises ``OSError`` if library cannot be loaded. + """ + return ctypes.CDLL(soname, PREFERRED_LOAD_FLAG) + + +def _load_wheel_installation(soname: str): + """Try to dlopen() the library indicated by ``soname`` + + Returns ``None`` if the library cannot be loaded. + """ + if os.path.isfile( + lib := os.path.join(os.path.dirname(__file__), "lib64", soname) + ): + return ctypes.CDLL(lib, PREFERRED_LOAD_FLAG) + return None + + +def load_library(): + """Dynamically load libcuopt_mathopt.so and its dependencies.""" + try: + # librmm and rapids_logger must be loaded before libcuopt_mathopt.so, + # which references them. + import librmm + import rapids_logger + + rapids_logger.load_library() + librmm.load_library() + except ModuleNotFoundError: + pass + + # This component has a DT_NEEDED on the client, so the client package must + # be loaded first now that the two live in separate wheels. + import libcuopt_client + + libcuopt_client.load_library() + + prefer_system_installation = ( + os.getenv("RAPIDS_LIBCUOPT_PREFER_SYSTEM_LIBRARY", "false").lower() + != "false" + ) + + lib = _load_component( + "libcuopt_mathopt.so", prefer_system_installation, True + ) + return [lib] if lib is not None else [] + + +def _load_component( + soname: str, prefer_system_installation: bool, required: bool +): + """Load one cuOpt component. + + Returns the handle, or ``None`` if it could not be loaded. Failing to + load an optional component is silent; failing a required one warns. + """ + libcuopt_lib = None + try: + if prefer_system_installation: + # Prefer a system library if one is present to + # avoid clobbering symbols that other packages might expect, but if no + # other library is present use the one in the wheel. + try: + libcuopt_lib = _load_system_installation(soname) + except OSError: + # The fallback needs to sit inside the outer handler too. An + # optional component that is present but fails to load would + # otherwise raise out of here and stop the remaining components + # from being loaded at all. + libcuopt_lib = _load_wheel_installation(soname) + else: + # Prefer the libraries bundled in this package. If they aren't found + # (which might be the case in builds where the library + # was prebuilt before packaging the wheel), look for a + # system installation. + libcuopt_lib = _load_wheel_installation(soname) + if libcuopt_lib is None: + libcuopt_lib = _load_system_installation(soname) + except OSError as e: + # If none of the searches above succeed, just silently return None + # and rely on other mechanisms (like RPATHs on other DSOs) to + # help the loader find the library. + if not required: + return None + + import warnings + + warnings.warn( + f"Failed to load libcuopt library: {soname}. " + f"Error: {str(e)}. " + "Falling back to relying on system loader. " + "cuOpt functionality may be unavailable. " + f"This might lead to a generic error such as " + f"'{soname} missing' if the library cannot be found.", + RuntimeWarning, + ) + # The caller almost never needs to do anything with this library, but no + # harm in offering the option since this object at least provides a handle + # to inspect where libcuopt was loaded from. + return libcuopt_lib diff --git a/python/libcuopt_mathopt/pyproject.toml b/python/libcuopt_mathopt/pyproject.toml new file mode 100644 index 0000000000..481b2864df --- /dev/null +++ b/python/libcuopt_mathopt/pyproject.toml @@ -0,0 +1,86 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +[build-system] +build-backend = "rapids_build_backend.build" +requires = [ + "rapids-build-backend>=0.4.0,<0.5.0", + "scikit-build-core[pyproject]>=0.11.0", +] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`. + +[project] +name = "libcuopt-mathopt" +dynamic = ["version"] +description = "cuOpt LP / MILP / QP / SOCP solver libraries (C++)" +readme = { file = "README.md", content-type = "text/markdown" } +authors = [ + { name = "NVIDIA Corporation" }, +] +license = "Apache-2.0" +requires-python = ">=3.11" +classifiers = [ + "Intended Audience :: Developers", + "Topic :: Database", + "Topic :: Scientific/Engineering", + "Programming Language :: C++", + "Environment :: GPU :: NVIDIA CUDA", + "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", + "Programming Language :: Python :: 3.13", + "Programming Language :: Python :: 3.14", +] +dependencies = [ + "cuda-toolkit[cublas,cudart,curand,cusolver,cusparse,nvtx]==13.*", + "libcuopt-client==26.10.*,>=0.0.0a0", + "librmm==26.10.*,>=0.0.0a0", + "nvidia-cudss-cu13>=0.7,<0.8", + "nvidia-nccl-cu13>=2.19", + "nvidia-nvjitlink>=13.3,<14", + "rapids-logger==0.3.*", +] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`. + +[project.urls] +Homepage = "https://docs.nvidia.com/cuopt/introduction.html" +Source = "https://github.com/nvidia/cuopt" + +[project.entry-points."cmake.prefix"] +libcuopt_mathopt = "libcuopt_mathopt" + +[project.scripts] +cuopt_cli = "libcuopt_mathopt._cli_wrapper:main" + +[tool.pydistcheck] +select = [ + # note: max_allowed_size_compressed is set as a command-line arg + "distro-too-large-compressed", +] + +[tool.scikit-build] +build-dir = "build/{wheel_tag}" +cmake.build-type = "Release" +cmake.version = "CMakeLists.txt" +minimum-version = "build-system.requires" +ninja.make-fallback = false +sdist.reproducible = true +wheel.packages = ["libcuopt_mathopt"] +wheel.install-dir = "libcuopt_mathopt" +wheel.py-api = "py3" +# The whole point of this package: stage only this component's install rules, +# so the wheel carries one library instead of all three. +install.components = ["mathopt", "mathopt-dev"] + +[tool.scikit-build.metadata.version] +provider = "scikit_build_core.metadata.regex" +input = "libcuopt_mathopt/VERSION" +regex = "(?P.*)" + +[tool.rapids-build-backend] +build-backend = "scikit_build_core.build" +dependencies-file = "../../dependencies.yaml" +matrix-entry = "cuda_suffixed=true;use_cuda_wheels=true" +requires = [ + "cmake>=4.0", + "librmm==26.10.*,>=0.0.0a0", + "ninja", + "rapids-logger==0.3.*", +] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`. diff --git a/python/libcuopt_routing/CMakeLists.txt b/python/libcuopt_routing/CMakeLists.txt new file mode 100644 index 0000000000..03afc376da --- /dev/null +++ b/python/libcuopt_routing/CMakeLists.txt @@ -0,0 +1,25 @@ +# cmake-format: off +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# cmake-format: on + +cmake_minimum_required(VERSION 4.0 FATAL_ERROR) + +include(../../cmake/rapids_config.cmake) +include(rapids-cuda) +rapids_cuda_init_architectures(libcuopt-routing-python) + +project( + libcuopt-routing-python + VERSION "${RAPIDS_VERSION}" + LANGUAGES CXX CUDA +) + +# Already installed: nothing to build, the wheel just stages the component. +find_package(cuopt "${RAPIDS_VERSION}") +if(cuopt_FOUND) + return() +endif() +unset(cuopt_FOUND) + +include(../cmake/cuopt_wheel_build.cmake) diff --git a/python/libcuopt_routing/LICENSE b/python/libcuopt_routing/LICENSE new file mode 100644 index 0000000000..6ed9218d99 --- /dev/null +++ b/python/libcuopt_routing/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "{}" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2025 NVIDIA Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/python/libcuopt_routing/README.md b/python/libcuopt_routing/README.md new file mode 100644 index 0000000000..6de72cc553 --- /dev/null +++ b/python/libcuopt_routing/README.md @@ -0,0 +1,5 @@ +# libcuopt-routing + +cuOpt VRP / TSP / PDP routing engine library. + +This package ships one component of cuOpt's C++ library. Install [`libcuopt`](https://pypi.org/project/libcuopt/) instead to get all of them. diff --git a/python/libcuopt_routing/libcuopt_routing/VERSION b/python/libcuopt_routing/libcuopt_routing/VERSION new file mode 100644 index 0000000000..6549ba6527 --- /dev/null +++ b/python/libcuopt_routing/libcuopt_routing/VERSION @@ -0,0 +1 @@ +26.10.00 diff --git a/python/libcuopt_routing/libcuopt_routing/__init__.py b/python/libcuopt_routing/libcuopt_routing/__init__.py new file mode 100644 index 0000000000..ea7cf929a8 --- /dev/null +++ b/python/libcuopt_routing/libcuopt_routing/__init__.py @@ -0,0 +1,7 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from libcuopt_routing._version import __git_commit__, __version__ +from libcuopt_routing.load import load_library + +__all__ = ["__git_commit__", "__version__", "load_library"] diff --git a/python/libcuopt_routing/libcuopt_routing/_version.py b/python/libcuopt_routing/libcuopt_routing/_version.py new file mode 100644 index 0000000000..ca5b7efde7 --- /dev/null +++ b/python/libcuopt_routing/libcuopt_routing/_version.py @@ -0,0 +1,22 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +import importlib.resources + +__version__ = ( + importlib.resources.files(__package__) + .joinpath("VERSION") + .read_text() + .strip() +) +try: + __git_commit__ = ( + importlib.resources.files(__package__) + .joinpath("GIT_COMMIT") + .read_text() + .strip() + ) +except FileNotFoundError: + __git_commit__ = "" + +__all__ = ["__git_commit__", "__version__"] diff --git a/python/libcuopt_routing/libcuopt_routing/load.py b/python/libcuopt_routing/libcuopt_routing/load.py new file mode 100644 index 0000000000..b4edda442f --- /dev/null +++ b/python/libcuopt_routing/libcuopt_routing/load.py @@ -0,0 +1,118 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Loader for the cuOpt VRP / TSP / PDP routing component. + +import ctypes +import os + +# Loading with RTLD_LOCAL adds the library itself to the loader's +# loaded library cache without loading any symbols into the global +# namespace. This allows libraries that express a dependency on +# this library to be loaded later and successfully satisfy this dependency +# without polluting the global symbol table with symbols from +# libcudf that could conflict with symbols from other DSOs. +PREFERRED_LOAD_FLAG = ctypes.RTLD_LOCAL + + +def _load_system_installation(soname: str): + """Try to dlopen() the library indicated by ``soname`` + Raises ``OSError`` if library cannot be loaded. + """ + return ctypes.CDLL(soname, PREFERRED_LOAD_FLAG) + + +def _load_wheel_installation(soname: str): + """Try to dlopen() the library indicated by ``soname`` + + Returns ``None`` if the library cannot be loaded. + """ + if os.path.isfile( + lib := os.path.join(os.path.dirname(__file__), "lib64", soname) + ): + return ctypes.CDLL(lib, PREFERRED_LOAD_FLAG) + return None + + +def load_library(): + """Dynamically load libcuopt_routing.so and its dependencies.""" + try: + # librmm and rapids_logger must be loaded before libcuopt_routing.so, + # which references them. + import librmm + import rapids_logger + + rapids_logger.load_library() + librmm.load_library() + except ModuleNotFoundError: + pass + + # This component has a DT_NEEDED on the client, so the client package must + # be loaded first now that the two live in separate wheels. + import libcuopt_client + + libcuopt_client.load_library() + + prefer_system_installation = ( + os.getenv("RAPIDS_LIBCUOPT_PREFER_SYSTEM_LIBRARY", "false").lower() + != "false" + ) + + lib = _load_component( + "libcuopt_routing.so", prefer_system_installation, True + ) + return [lib] if lib is not None else [] + + +def _load_component( + soname: str, prefer_system_installation: bool, required: bool +): + """Load one cuOpt component. + + Returns the handle, or ``None`` if it could not be loaded. Failing to + load an optional component is silent; failing a required one warns. + """ + libcuopt_lib = None + try: + if prefer_system_installation: + # Prefer a system library if one is present to + # avoid clobbering symbols that other packages might expect, but if no + # other library is present use the one in the wheel. + try: + libcuopt_lib = _load_system_installation(soname) + except OSError: + # The fallback needs to sit inside the outer handler too. An + # optional component that is present but fails to load would + # otherwise raise out of here and stop the remaining components + # from being loaded at all. + libcuopt_lib = _load_wheel_installation(soname) + else: + # Prefer the libraries bundled in this package. If they aren't found + # (which might be the case in builds where the library + # was prebuilt before packaging the wheel), look for a + # system installation. + libcuopt_lib = _load_wheel_installation(soname) + if libcuopt_lib is None: + libcuopt_lib = _load_system_installation(soname) + except OSError as e: + # If none of the searches above succeed, just silently return None + # and rely on other mechanisms (like RPATHs on other DSOs) to + # help the loader find the library. + if not required: + return None + + import warnings + + warnings.warn( + f"Failed to load libcuopt library: {soname}. " + f"Error: {str(e)}. " + "Falling back to relying on system loader. " + "cuOpt functionality may be unavailable. " + f"This might lead to a generic error such as " + f"'{soname} missing' if the library cannot be found.", + RuntimeWarning, + ) + # The caller almost never needs to do anything with this library, but no + # harm in offering the option since this object at least provides a handle + # to inspect where libcuopt was loaded from. + return libcuopt_lib diff --git a/python/libcuopt_routing/pyproject.toml b/python/libcuopt_routing/pyproject.toml new file mode 100644 index 0000000000..5e1ac15d61 --- /dev/null +++ b/python/libcuopt_routing/pyproject.toml @@ -0,0 +1,80 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +[build-system] +build-backend = "rapids_build_backend.build" +requires = [ + "rapids-build-backend>=0.4.0,<0.5.0", + "scikit-build-core[pyproject]>=0.11.0", +] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`. + +[project] +name = "libcuopt-routing" +dynamic = ["version"] +description = "cuOpt VRP / TSP / PDP routing libraries (C++)" +readme = { file = "README.md", content-type = "text/markdown" } +authors = [ + { name = "NVIDIA Corporation" }, +] +license = "Apache-2.0" +requires-python = ">=3.11" +classifiers = [ + "Intended Audience :: Developers", + "Topic :: Database", + "Topic :: Scientific/Engineering", + "Programming Language :: C++", + "Environment :: GPU :: NVIDIA CUDA", + "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", + "Programming Language :: Python :: 3.13", + "Programming Language :: Python :: 3.14", +] +dependencies = [ + "cuda-toolkit[cublas,cudart]==13.*", + "libcuopt-client==26.10.*,>=0.0.0a0", + "librmm==26.10.*,>=0.0.0a0", + "rapids-logger==0.3.*", +] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`. + +[project.urls] +Homepage = "https://docs.nvidia.com/cuopt/introduction.html" +Source = "https://github.com/nvidia/cuopt" + +[project.entry-points."cmake.prefix"] +libcuopt_routing = "libcuopt_routing" + +[tool.pydistcheck] +select = [ + # note: max_allowed_size_compressed is set as a command-line arg + "distro-too-large-compressed", +] + +[tool.scikit-build] +build-dir = "build/{wheel_tag}" +cmake.build-type = "Release" +cmake.version = "CMakeLists.txt" +minimum-version = "build-system.requires" +ninja.make-fallback = false +sdist.reproducible = true +wheel.packages = ["libcuopt_routing"] +wheel.install-dir = "libcuopt_routing" +wheel.py-api = "py3" +# The whole point of this package: stage only this component's install rules, +# so the wheel carries one library instead of all three. +install.components = ["routing", "routing-dev"] + +[tool.scikit-build.metadata.version] +provider = "scikit_build_core.metadata.regex" +input = "libcuopt_routing/VERSION" +regex = "(?P.*)" + +[tool.rapids-build-backend] +build-backend = "scikit_build_core.build" +dependencies-file = "../../dependencies.yaml" +matrix-entry = "cuda_suffixed=true;use_cuda_wheels=true" +requires = [ + "cmake>=4.0", + "librmm==26.10.*,>=0.0.0a0", + "ninja", + "rapids-logger==0.3.*", +] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`. From fd8b429cc5e9f63b52483dd554019a9f3f9e3866 Mon Sep 17 00:00:00 2001 From: Ramakrishna Prabhu Date: Fri, 18 Sep 2026 10:13:01 -0500 Subject: [PATCH 2/7] build(wheel): regenerate the component build scripts from current main The three scripts were copied from ci/build_wheel_libcuopt.sh before it moved off LLVM libomp. They still installed llvm-toolset and libomp-devel and set OpenMP_gomp_LIBRARY to the system libomp, missing the modern GNU libgomp fetch and CUOPT_BUILD_CUSTOM_CUDSS_MTLAYER=ON that #1219 needs so cuOpt and cuDSS's threading layer share one OpenMP runtime, as well as use_cuda_wheels in the requirements matrix. Regenerated from the current file, so they now differ from it only in package_name, package_dir and the artifact name. Noted in each that the four have to be regenerated together. Co-Authored-By: Claude Opus 5 --- ci/build_wheel_libcuopt_client.sh | 41 ++++++++++++++---------------- ci/build_wheel_libcuopt_mathopt.sh | 41 ++++++++++++++---------------- ci/build_wheel_libcuopt_routing.sh | 41 ++++++++++++++---------------- 3 files changed, 57 insertions(+), 66 deletions(-) diff --git a/ci/build_wheel_libcuopt_client.sh b/ci/build_wheel_libcuopt_client.sh index 5246002c37..d83f3218aa 100755 --- a/ci/build_wheel_libcuopt_client.sh +++ b/ci/build_wheel_libcuopt_client.sh @@ -4,9 +4,9 @@ set -euo pipefail -# Configures the full C++ tree like every other cuOpt wheel, but stages only this -# component's install rules -- see install.components in the package's -# pyproject.toml. sccache is what keeps the repeated configure affordable. +# Same full C++ configure as every other cuOpt wheel; only the install components +# staged into it differ, set by install.components in this package's pyproject.toml. +# Kept in step with ci/build_wheel_libcuopt.sh -- regenerate all four together. source rapids-init-pip @@ -21,34 +21,31 @@ fi # Install Boost and TBB bash ci/utils/install_boost_tbb.sh -# Install libuuid and LLVM's OpenMP runtime +# Install libuuid (needed by cuopt_grpc_server) if command -v dnf &> /dev/null; then - # LLVM Toolset is distributed as a module on Rocky/RHEL 8. - dnf module install -y llvm-toolset - dnf install -y libuuid-devel libomp-devel + dnf install -y libuuid-devel elif command -v apt-get &> /dev/null; then apt-get update - apt-get install -y uuid-dev libomp-dev + apt-get install -y uuid-dev fi # Install Protobuf + gRPC (protoc + grpc_cpp_plugin) bash ci/utils/install_protobuf_grpc.sh -# Compile with GCC, but use LLVM libomp as the OpenMP runtime bundled in the wheel. Resolve the -# versioned ELF library rather than an unversioned linker script or compiler-toolset indirection. -LIBOMP_LIBRARY="$( - ldconfig -p | - awk '$1 ~ /^libomp\.so(\.[0-9]+)*$/ && !library { library = $NF } - END { print library }' -)" -if [[ "${LIBOMP_LIBRARY}" != /* || ! -f "${LIBOMP_LIBRARY}" ]]; then - echo "Could not resolve the LLVM OpenMP runtime: '${LIBOMP_LIBRARY}'" >&2 - exit 1 -fi +# Compile against a modern GNU libgomp from conda-forge instead of bundled LLVM libomp, to +# unify cuOpt's OpenMP runtime with the one cuDSS's threading layer needs (#1219). +MODERN_LIBGOMP_DIR="$(pwd)/modern_libgomp" +python -m pip install --quiet zstandard +python ci/utils/install_modern_libgomp.py "${MODERN_LIBGOMP_DIR}" -echo "Using LLVM OpenMP runtime: ${LIBOMP_LIBRARY}" +# Also build our own cuDSS threading layer against this same libgomp (see cpp/CMakeLists.txt, +# cpp/src/barrier/cudss_mtlayer_cuopt.cpp), instead of cuDSS's prebuilt one, so both actually +# share one instance, not just the same flavor. Conda keeps cuDSS's default (#1219 discussion). +export SKBUILD_CMAKE_ARGS="-DOpenMP_gomp_LIBRARY:FILEPATH=${MODERN_LIBGOMP_DIR}/libgomp.so.1.0.0;-DCUOPT_BUILD_CUSTOM_CUDSS_MTLAYER=ON" -export SKBUILD_CMAKE_ARGS="-DOpenMP_gomp_LIBRARY:FILEPATH=${LIBOMP_LIBRARY}" +# auditwheel repair does its own dependency resolution separately from the compiler; without +# this it can't see our fetched copy and silently vendors the old Rocky 8 system one instead. +export LD_LIBRARY_PATH="${MODERN_LIBGOMP_DIR}${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}" # OpenSSL 3 hints for libcuopt's own find_package(OpenSSL). # @@ -83,7 +80,7 @@ rapids-dependency-file-generator \ --output requirements \ --file-key "py_build_${package_name}" \ --file-key "py_rapids_build_${package_name}" \ - --matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION};cuda_suffixed=true" \ + --matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION};cuda_suffixed=true;use_cuda_wheels=true" \ | tee /tmp/requirements-build.txt rapids-logger "Installing build requirements" diff --git a/ci/build_wheel_libcuopt_mathopt.sh b/ci/build_wheel_libcuopt_mathopt.sh index e70d0ca468..e72e831602 100755 --- a/ci/build_wheel_libcuopt_mathopt.sh +++ b/ci/build_wheel_libcuopt_mathopt.sh @@ -4,9 +4,9 @@ set -euo pipefail -# Configures the full C++ tree like every other cuOpt wheel, but stages only this -# component's install rules -- see install.components in the package's -# pyproject.toml. sccache is what keeps the repeated configure affordable. +# Same full C++ configure as every other cuOpt wheel; only the install components +# staged into it differ, set by install.components in this package's pyproject.toml. +# Kept in step with ci/build_wheel_libcuopt.sh -- regenerate all four together. source rapids-init-pip @@ -21,34 +21,31 @@ fi # Install Boost and TBB bash ci/utils/install_boost_tbb.sh -# Install libuuid and LLVM's OpenMP runtime +# Install libuuid (needed by cuopt_grpc_server) if command -v dnf &> /dev/null; then - # LLVM Toolset is distributed as a module on Rocky/RHEL 8. - dnf module install -y llvm-toolset - dnf install -y libuuid-devel libomp-devel + dnf install -y libuuid-devel elif command -v apt-get &> /dev/null; then apt-get update - apt-get install -y uuid-dev libomp-dev + apt-get install -y uuid-dev fi # Install Protobuf + gRPC (protoc + grpc_cpp_plugin) bash ci/utils/install_protobuf_grpc.sh -# Compile with GCC, but use LLVM libomp as the OpenMP runtime bundled in the wheel. Resolve the -# versioned ELF library rather than an unversioned linker script or compiler-toolset indirection. -LIBOMP_LIBRARY="$( - ldconfig -p | - awk '$1 ~ /^libomp\.so(\.[0-9]+)*$/ && !library { library = $NF } - END { print library }' -)" -if [[ "${LIBOMP_LIBRARY}" != /* || ! -f "${LIBOMP_LIBRARY}" ]]; then - echo "Could not resolve the LLVM OpenMP runtime: '${LIBOMP_LIBRARY}'" >&2 - exit 1 -fi +# Compile against a modern GNU libgomp from conda-forge instead of bundled LLVM libomp, to +# unify cuOpt's OpenMP runtime with the one cuDSS's threading layer needs (#1219). +MODERN_LIBGOMP_DIR="$(pwd)/modern_libgomp" +python -m pip install --quiet zstandard +python ci/utils/install_modern_libgomp.py "${MODERN_LIBGOMP_DIR}" -echo "Using LLVM OpenMP runtime: ${LIBOMP_LIBRARY}" +# Also build our own cuDSS threading layer against this same libgomp (see cpp/CMakeLists.txt, +# cpp/src/barrier/cudss_mtlayer_cuopt.cpp), instead of cuDSS's prebuilt one, so both actually +# share one instance, not just the same flavor. Conda keeps cuDSS's default (#1219 discussion). +export SKBUILD_CMAKE_ARGS="-DOpenMP_gomp_LIBRARY:FILEPATH=${MODERN_LIBGOMP_DIR}/libgomp.so.1.0.0;-DCUOPT_BUILD_CUSTOM_CUDSS_MTLAYER=ON" -export SKBUILD_CMAKE_ARGS="-DOpenMP_gomp_LIBRARY:FILEPATH=${LIBOMP_LIBRARY}" +# auditwheel repair does its own dependency resolution separately from the compiler; without +# this it can't see our fetched copy and silently vendors the old Rocky 8 system one instead. +export LD_LIBRARY_PATH="${MODERN_LIBGOMP_DIR}${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}" # OpenSSL 3 hints for libcuopt's own find_package(OpenSSL). # @@ -83,7 +80,7 @@ rapids-dependency-file-generator \ --output requirements \ --file-key "py_build_${package_name}" \ --file-key "py_rapids_build_${package_name}" \ - --matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION};cuda_suffixed=true" \ + --matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION};cuda_suffixed=true;use_cuda_wheels=true" \ | tee /tmp/requirements-build.txt rapids-logger "Installing build requirements" diff --git a/ci/build_wheel_libcuopt_routing.sh b/ci/build_wheel_libcuopt_routing.sh index 87acfab67d..d8ba2e5cee 100755 --- a/ci/build_wheel_libcuopt_routing.sh +++ b/ci/build_wheel_libcuopt_routing.sh @@ -4,9 +4,9 @@ set -euo pipefail -# Configures the full C++ tree like every other cuOpt wheel, but stages only this -# component's install rules -- see install.components in the package's -# pyproject.toml. sccache is what keeps the repeated configure affordable. +# Same full C++ configure as every other cuOpt wheel; only the install components +# staged into it differ, set by install.components in this package's pyproject.toml. +# Kept in step with ci/build_wheel_libcuopt.sh -- regenerate all four together. source rapids-init-pip @@ -21,34 +21,31 @@ fi # Install Boost and TBB bash ci/utils/install_boost_tbb.sh -# Install libuuid and LLVM's OpenMP runtime +# Install libuuid (needed by cuopt_grpc_server) if command -v dnf &> /dev/null; then - # LLVM Toolset is distributed as a module on Rocky/RHEL 8. - dnf module install -y llvm-toolset - dnf install -y libuuid-devel libomp-devel + dnf install -y libuuid-devel elif command -v apt-get &> /dev/null; then apt-get update - apt-get install -y uuid-dev libomp-dev + apt-get install -y uuid-dev fi # Install Protobuf + gRPC (protoc + grpc_cpp_plugin) bash ci/utils/install_protobuf_grpc.sh -# Compile with GCC, but use LLVM libomp as the OpenMP runtime bundled in the wheel. Resolve the -# versioned ELF library rather than an unversioned linker script or compiler-toolset indirection. -LIBOMP_LIBRARY="$( - ldconfig -p | - awk '$1 ~ /^libomp\.so(\.[0-9]+)*$/ && !library { library = $NF } - END { print library }' -)" -if [[ "${LIBOMP_LIBRARY}" != /* || ! -f "${LIBOMP_LIBRARY}" ]]; then - echo "Could not resolve the LLVM OpenMP runtime: '${LIBOMP_LIBRARY}'" >&2 - exit 1 -fi +# Compile against a modern GNU libgomp from conda-forge instead of bundled LLVM libomp, to +# unify cuOpt's OpenMP runtime with the one cuDSS's threading layer needs (#1219). +MODERN_LIBGOMP_DIR="$(pwd)/modern_libgomp" +python -m pip install --quiet zstandard +python ci/utils/install_modern_libgomp.py "${MODERN_LIBGOMP_DIR}" -echo "Using LLVM OpenMP runtime: ${LIBOMP_LIBRARY}" +# Also build our own cuDSS threading layer against this same libgomp (see cpp/CMakeLists.txt, +# cpp/src/barrier/cudss_mtlayer_cuopt.cpp), instead of cuDSS's prebuilt one, so both actually +# share one instance, not just the same flavor. Conda keeps cuDSS's default (#1219 discussion). +export SKBUILD_CMAKE_ARGS="-DOpenMP_gomp_LIBRARY:FILEPATH=${MODERN_LIBGOMP_DIR}/libgomp.so.1.0.0;-DCUOPT_BUILD_CUSTOM_CUDSS_MTLAYER=ON" -export SKBUILD_CMAKE_ARGS="-DOpenMP_gomp_LIBRARY:FILEPATH=${LIBOMP_LIBRARY}" +# auditwheel repair does its own dependency resolution separately from the compiler; without +# this it can't see our fetched copy and silently vendors the old Rocky 8 system one instead. +export LD_LIBRARY_PATH="${MODERN_LIBGOMP_DIR}${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}" # OpenSSL 3 hints for libcuopt's own find_package(OpenSSL). # @@ -83,7 +80,7 @@ rapids-dependency-file-generator \ --output requirements \ --file-key "py_build_${package_name}" \ --file-key "py_rapids_build_${package_name}" \ - --matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION};cuda_suffixed=true" \ + --matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION};cuda_suffixed=true;use_cuda_wheels=true" \ | tee /tmp/requirements-build.txt rapids-logger "Installing build requirements" From aaf90d1839667697faad194872b92f05fa5246b8 Mon Sep 17 00:00:00 2001 From: Ramakrishna Prabhu Date: Fri, 18 Sep 2026 10:43:13 -0500 Subject: [PATCH 3/7] build(wheel): fix the dependency file keys and cross-wheel library resolution Three CI failures, all from the split. rapids-dependency-file-generator raised "No matching matrix found in 'depends_on_librmm' for: {}" because the six new pyproject file keys had no matrix, so they resolved against an empty combination that depends_on_librmm's matrices do not cover. They now carry the same cuda_suffixed / use_cuda_wheels matrix as py_run_libcuopt. verify-dependencies rejected cuda_wheels_routing's no-selector entry for listing cuda-toolkit outside a set marked use_cuda_wheels: "true". That entry is what populates pyproject.toml in source control, so it stays, but it now names use_cuda_wheels explicitly, which is also what the file keys resolve against. wheel-build-libcuopt failed in auditwheel with "required library libcuopt_mathopt.so could not be located": the libcuopt wheel now stages only dev and grpc-server, while cuopt_grpc_server links mathopt, which lives in a sibling wheel. The component libraries are excluded from vendoring in all four scripts -- bundling them would ship several copies of each engine -- and the wheels gained $ORIGIN/../../libcuopt_/lib64 so the loader resolves them across packages. Co-Authored-By: Claude Opus 5 --- ci/build_wheel_libcuopt.sh | 5 +++++ ci/build_wheel_libcuopt_client.sh | 5 +++++ ci/build_wheel_libcuopt_mathopt.sh | 5 +++++ ci/build_wheel_libcuopt_routing.sh | 5 +++++ dependencies.yaml | 24 ++++++++++++++++++++++-- python/cmake/cuopt_wheel_build.cmake | 5 +++++ 6 files changed, 47 insertions(+), 2 deletions(-) diff --git a/ci/build_wheel_libcuopt.sh b/ci/build_wheel_libcuopt.sh index 807cfb8f21..02ba7e52ea 100755 --- a/ci/build_wheel_libcuopt.sh +++ b/ci/build_wheel_libcuopt.sh @@ -103,6 +103,11 @@ EXCLUDE_ARGS=( --exclude "libnvJitLink.so*" --exclude "librapids_logger.so" --exclude "librmm.so" + # Provided by the sibling libcuopt-* wheels, so they must not be vendored here: doing so + # would ship several copies of each engine and defeat the split. RPATH resolves them. + --exclude "libcuopt_client.so" + --exclude "libcuopt_mathopt.so" + --exclude "libcuopt_routing.so" # OpenSSL 3 is intentionally NOT bundled. Resolving libssl.so.3 / libcrypto.so.3 # at runtime via the host (or container image) keeps libcrypto and the FIPS # provider (system or mounted) byte-version-matched, which is required for diff --git a/ci/build_wheel_libcuopt_client.sh b/ci/build_wheel_libcuopt_client.sh index d83f3218aa..9e36b553a2 100755 --- a/ci/build_wheel_libcuopt_client.sh +++ b/ci/build_wheel_libcuopt_client.sh @@ -107,6 +107,11 @@ EXCLUDE_ARGS=( --exclude "libnvJitLink.so*" --exclude "librapids_logger.so" --exclude "librmm.so" + # Provided by the sibling libcuopt-* wheels, so they must not be vendored here: doing so + # would ship several copies of each engine and defeat the split. RPATH resolves them. + --exclude "libcuopt_client.so" + --exclude "libcuopt_mathopt.so" + --exclude "libcuopt_routing.so" # OpenSSL 3 is intentionally NOT bundled. Resolving libssl.so.3 / libcrypto.so.3 # at runtime via the host (or container image) keeps libcrypto and the FIPS # provider (system or mounted) byte-version-matched, which is required for diff --git a/ci/build_wheel_libcuopt_mathopt.sh b/ci/build_wheel_libcuopt_mathopt.sh index e72e831602..8a86a4b50a 100755 --- a/ci/build_wheel_libcuopt_mathopt.sh +++ b/ci/build_wheel_libcuopt_mathopt.sh @@ -107,6 +107,11 @@ EXCLUDE_ARGS=( --exclude "libnvJitLink.so*" --exclude "librapids_logger.so" --exclude "librmm.so" + # Provided by the sibling libcuopt-* wheels, so they must not be vendored here: doing so + # would ship several copies of each engine and defeat the split. RPATH resolves them. + --exclude "libcuopt_client.so" + --exclude "libcuopt_mathopt.so" + --exclude "libcuopt_routing.so" # OpenSSL 3 is intentionally NOT bundled. Resolving libssl.so.3 / libcrypto.so.3 # at runtime via the host (or container image) keeps libcrypto and the FIPS # provider (system or mounted) byte-version-matched, which is required for diff --git a/ci/build_wheel_libcuopt_routing.sh b/ci/build_wheel_libcuopt_routing.sh index d8ba2e5cee..a5ec02ce09 100755 --- a/ci/build_wheel_libcuopt_routing.sh +++ b/ci/build_wheel_libcuopt_routing.sh @@ -107,6 +107,11 @@ EXCLUDE_ARGS=( --exclude "libnvJitLink.so*" --exclude "librapids_logger.so" --exclude "librmm.so" + # Provided by the sibling libcuopt-* wheels, so they must not be vendored here: doing so + # would ship several copies of each engine and defeat the split. RPATH resolves them. + --exclude "libcuopt_client.so" + --exclude "libcuopt_mathopt.so" + --exclude "libcuopt_routing.so" # OpenSSL 3 is intentionally NOT bundled. Resolving libssl.so.3 / libcrypto.so.3 # at runtime via the host (or container image) keeps libcrypto and the FIPS # provider (system or mounted) byte-version-matched, which is required for diff --git a/dependencies.yaml b/dependencies.yaml index 61eb39910f..fcf8babdc1 100644 --- a/dependencies.yaml +++ b/dependencies.yaml @@ -165,6 +165,9 @@ files: py_rapids_build_libcuopt_client: output: pyproject pyproject_dir: python/libcuopt_client + matrix: + cuda_suffixed: ["false"] + use_cuda_wheels: ["true"] extras: table: tool.rapids-build-backend key: requires @@ -175,6 +178,9 @@ files: py_run_libcuopt_client: output: pyproject pyproject_dir: python/libcuopt_client + matrix: + cuda_suffixed: ["false"] + use_cuda_wheels: ["true"] extras: table: project includes: @@ -190,6 +196,9 @@ files: py_rapids_build_libcuopt_mathopt: output: pyproject pyproject_dir: python/libcuopt_mathopt + matrix: + cuda_suffixed: ["false"] + use_cuda_wheels: ["true"] extras: table: tool.rapids-build-backend key: requires @@ -201,6 +210,9 @@ files: py_run_libcuopt_mathopt: output: pyproject pyproject_dir: python/libcuopt_mathopt + matrix: + cuda_suffixed: ["false"] + use_cuda_wheels: ["true"] extras: table: project includes: @@ -219,6 +231,9 @@ files: py_rapids_build_libcuopt_routing: output: pyproject pyproject_dir: python/libcuopt_routing + matrix: + cuda_suffixed: ["false"] + use_cuda_wheels: ["true"] extras: table: tool.rapids-build-backend key: requires @@ -230,6 +245,9 @@ files: py_run_libcuopt_routing: output: pyproject pyproject_dir: python/libcuopt_routing + matrix: + cuda_suffixed: ["false"] + use_cuda_wheels: ["true"] extras: table: project includes: @@ -1040,9 +1058,11 @@ dependencies: use_cuda_wheels: "true" packages: - &ctk_routing_cu13 cuda-toolkit[cublas,cudart]==13.* - # if no matching matrix selectors passed, list the CUDA 13 requirement, which is - # what populates pyproject.toml in source control + # No cuda selector: this is what the pyproject file keys resolve against, and it + # is what populates pyproject.toml in source control. use_cuda_wheels is named + # explicitly so the entry still reads as a CUDA-wheel set. - matrix: + use_cuda_wheels: "true" packages: - *ctk_routing_cu13 cuda_wheels: diff --git a/python/cmake/cuopt_wheel_build.cmake b/python/cmake/cuopt_wheel_build.cmake index 1ee1d07ffa..6f86bd7694 100644 --- a/python/cmake/cuopt_wheel_build.cmake +++ b/python/cmake/cuopt_wheel_build.cmake @@ -48,6 +48,11 @@ target_link_libraries(cuopt_cli PRIVATE set(rpaths "$ORIGIN/../lib64" + # The component libraries live in sibling wheels now, so a binary or library in one + # package resolves the others through their install directories rather than its own. + "$ORIGIN/../../libcuopt_client/lib64" + "$ORIGIN/../../libcuopt_mathopt/lib64" + "$ORIGIN/../../libcuopt_routing/lib64" "$ORIGIN/../../rapids_logger/lib64" "$ORIGIN/../../librmm/lib64" "$ORIGIN/../../nvidia/cudss/lib" From 6526e7abb89e623e47970cbcc3ff30307b58484a Mon Sep 17 00:00:00 2001 From: Ramakrishna Prabhu Date: Fri, 18 Sep 2026 17:32:06 -0500 Subject: [PATCH 4/7] ci(wheel): build and publish the component wheels The build scripts existed but nothing invoked them, so CI never built the component wheels and the split had no effect on what gets published. pr.yaml gains a build job per component on the same matrix filter as libcuopt, and build.yaml gains the matching build and publish jobs. wheel-build-cuopt now waits on all three, since the cuopt wheel resolves through them rather than through libcuopt alone, and pr-builder gates on them so a failure blocks the PR instead of passing silently. Co-Authored-By: Claude Opus 5 --- .github/workflows/build.yaml | 123 ++++++++++++++++++++++++++++++++++- .github/workflows/pr.yaml | 64 +++++++++++++++++- 2 files changed, 185 insertions(+), 2 deletions(-) diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index e52701dfc5..4fb826b8f8 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -241,8 +241,129 @@ jobs: package-name: libcuopt package-type: cpp publish-wheel-search-key: cuopt_wheel_cpp_libcuopt + wheel-build-libcuopt-client: + permissions: + actions: read + contents: read + id-token: write + packages: read + pull-requests: read + needs: [build-details] + secrets: inherit # zizmor: ignore[secrets-inherit] + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@release/26.10 + with: + build_type: ${{ inputs.build_type || 'branch' }} + build-datetime: ${{ needs.build-details.outputs.build-datetime }} + branch: ${{ inputs.branch }} + sha: ${{ inputs.sha }} + date: ${{ inputs.date }} + script: ci/build_wheel_libcuopt_client.sh + package-name: libcuopt_client + package-type: cpp + matrix_filter: group_by([.ARCH, (.CUDA_VER|split(".")|map(tonumber)|.[0])]) | map(max_by(.PY_VER|split(".")|map(tonumber))) + wheel-publish-libcuopt-client: + needs: wheel-build-libcuopt-client + permissions: + actions: read + contents: read + id-token: write + packages: read + pull-requests: read + uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@release/26.10 + secrets: + CONDA_RAPIDSAI_WHEELS_NIGHTLY_TOKEN: ${{ secrets.CONDA_RAPIDSAI_WHEELS_NIGHTLY_TOKEN }} + RAPIDSAI_PYPI_TOKEN: ${{ secrets.RAPIDSAI_PYPI_TOKEN }} + with: + build_type: ${{ inputs.build_type || 'branch' }} + branch: ${{ inputs.branch }} + sha: ${{ inputs.sha }} + date: ${{ inputs.date }} + package-name: libcuopt_client + package-type: cpp + publish-wheel-search-key: cuopt_wheel_cpp_libcuopt_client + wheel-build-libcuopt-mathopt: + permissions: + actions: read + contents: read + id-token: write + packages: read + pull-requests: read + needs: [build-details] + secrets: inherit # zizmor: ignore[secrets-inherit] + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@release/26.10 + with: + build_type: ${{ inputs.build_type || 'branch' }} + build-datetime: ${{ needs.build-details.outputs.build-datetime }} + branch: ${{ inputs.branch }} + sha: ${{ inputs.sha }} + date: ${{ inputs.date }} + script: ci/build_wheel_libcuopt_mathopt.sh + package-name: libcuopt_mathopt + package-type: cpp + matrix_filter: group_by([.ARCH, (.CUDA_VER|split(".")|map(tonumber)|.[0])]) | map(max_by(.PY_VER|split(".")|map(tonumber))) + wheel-publish-libcuopt-mathopt: + needs: wheel-build-libcuopt-mathopt + permissions: + actions: read + contents: read + id-token: write + packages: read + pull-requests: read + uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@release/26.10 + secrets: + CONDA_RAPIDSAI_WHEELS_NIGHTLY_TOKEN: ${{ secrets.CONDA_RAPIDSAI_WHEELS_NIGHTLY_TOKEN }} + RAPIDSAI_PYPI_TOKEN: ${{ secrets.RAPIDSAI_PYPI_TOKEN }} + with: + build_type: ${{ inputs.build_type || 'branch' }} + branch: ${{ inputs.branch }} + sha: ${{ inputs.sha }} + date: ${{ inputs.date }} + package-name: libcuopt_mathopt + package-type: cpp + publish-wheel-search-key: cuopt_wheel_cpp_libcuopt_mathopt + wheel-build-libcuopt-routing: + permissions: + actions: read + contents: read + id-token: write + packages: read + pull-requests: read + needs: [build-details] + secrets: inherit # zizmor: ignore[secrets-inherit] + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@release/26.10 + with: + build_type: ${{ inputs.build_type || 'branch' }} + build-datetime: ${{ needs.build-details.outputs.build-datetime }} + branch: ${{ inputs.branch }} + sha: ${{ inputs.sha }} + date: ${{ inputs.date }} + script: ci/build_wheel_libcuopt_routing.sh + package-name: libcuopt_routing + package-type: cpp + matrix_filter: group_by([.ARCH, (.CUDA_VER|split(".")|map(tonumber)|.[0])]) | map(max_by(.PY_VER|split(".")|map(tonumber))) + wheel-publish-libcuopt-routing: + needs: wheel-build-libcuopt-routing + permissions: + actions: read + contents: read + id-token: write + packages: read + pull-requests: read + uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@release/26.10 + secrets: + CONDA_RAPIDSAI_WHEELS_NIGHTLY_TOKEN: ${{ secrets.CONDA_RAPIDSAI_WHEELS_NIGHTLY_TOKEN }} + RAPIDSAI_PYPI_TOKEN: ${{ secrets.RAPIDSAI_PYPI_TOKEN }} + with: + build_type: ${{ inputs.build_type || 'branch' }} + branch: ${{ inputs.branch }} + sha: ${{ inputs.sha }} + date: ${{ inputs.date }} + package-name: libcuopt_routing + package-type: cpp + publish-wheel-search-key: cuopt_wheel_cpp_libcuopt_routing wheel-build-cuopt: - needs: [build-details, wheel-build-libcuopt] + needs: [build-details, wheel-build-libcuopt, wheel-build-libcuopt-client, + wheel-build-libcuopt-mathopt, wheel-build-libcuopt-routing] permissions: actions: read contents: read diff --git a/.github/workflows/pr.yaml b/.github/workflows/pr.yaml index 4be18d74cd..8963224ffd 100644 --- a/.github/workflows/pr.yaml +++ b/.github/workflows/pr.yaml @@ -32,6 +32,9 @@ jobs: - docs-build - multi-gpu-cpp-tests - wheel-build-libcuopt + - wheel-build-libcuopt-client + - wheel-build-libcuopt-mathopt + - wheel-build-libcuopt-routing - wheel-build-cuopt - wheel-tests-cuopt - wheel-build-cuopt-server @@ -608,8 +611,67 @@ jobs: build_type: pull-request build-datetime: ${{ needs.build-details.outputs.build-datetime }} script: ci/build_wheel_libcuopt.sh + wheel-build-libcuopt-client: + needs: [build-details, compute-matrix-filters, changed-files] + if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_python_wheels + permissions: + actions: read + contents: read + id-token: write + packages: read + pull-requests: read + secrets: inherit # zizmor: ignore[secrets-inherit] + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@release/26.10 + with: + # Same filter as libcuopt: every arch and CUDA version, latest Python only. + matrix_filter: ${{ needs.compute-matrix-filters.outputs.libcuopt_filter }} + package-type: cpp + package-name: libcuopt_client + build_type: pull-request + build-datetime: ${{ needs.build-details.outputs.build-datetime }} + script: ci/build_wheel_libcuopt_client.sh + wheel-build-libcuopt-mathopt: + needs: [build-details, compute-matrix-filters, changed-files] + if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_python_wheels + permissions: + actions: read + contents: read + id-token: write + packages: read + pull-requests: read + secrets: inherit # zizmor: ignore[secrets-inherit] + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@release/26.10 + with: + # Same filter as libcuopt: every arch and CUDA version, latest Python only. + matrix_filter: ${{ needs.compute-matrix-filters.outputs.libcuopt_filter }} + package-type: cpp + package-name: libcuopt_mathopt + build_type: pull-request + build-datetime: ${{ needs.build-details.outputs.build-datetime }} + script: ci/build_wheel_libcuopt_mathopt.sh + wheel-build-libcuopt-routing: + needs: [build-details, compute-matrix-filters, changed-files] + if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_python_wheels + permissions: + actions: read + contents: read + id-token: write + packages: read + pull-requests: read + secrets: inherit # zizmor: ignore[secrets-inherit] + uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@release/26.10 + with: + # Same filter as libcuopt: every arch and CUDA version, latest Python only. + matrix_filter: ${{ needs.compute-matrix-filters.outputs.libcuopt_filter }} + package-type: cpp + package-name: libcuopt_routing + build_type: pull-request + build-datetime: ${{ needs.build-details.outputs.build-datetime }} + script: ci/build_wheel_libcuopt_routing.sh wheel-build-cuopt: - needs: [build-details, wheel-build-libcuopt, compute-matrix-filters, changed-files] + needs: [build-details, wheel-build-libcuopt, wheel-build-libcuopt-client, + wheel-build-libcuopt-mathopt, wheel-build-libcuopt-routing, + compute-matrix-filters, changed-files] if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_python_wheels permissions: actions: read From e2ca2e7b1269f8accef1e2b415aa10bdc798b2ec Mon Sep 17 00:00:00 2001 From: Ramakrishna Prabhu Date: Fri, 18 Sep 2026 21:48:43 -0500 Subject: [PATCH 5/7] ci(wheel): teach validate_wheel.sh the component package dirs All four component wheel builds failed at the last step with "unrecognized package_dir: 'python/libcuopt_client'". The wheels built and auditwheel repaired them; validate_wheel.sh rejects any package_dir it does not know, and the three new ones were never added. mathopt and routing take libcuopt's existing limits, which are a safe ceiling since each is strictly smaller than the combined wheel was. The client gets 50Mi: it has no CUDA kernels, so anything near the other limits would mean something was vendored into it that should not have been. Co-Authored-By: Claude Opus 5 --- ci/validate_wheel.sh | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/ci/validate_wheel.sh b/ci/validate_wheel.sh index da044db65a..c4b65225dd 100755 --- a/ci/validate_wheel.sh +++ b/ci/validate_wheel.sh @@ -29,6 +29,25 @@ if [[ "${package_dir}" == "python/libcuopt" ]]; then --max-allowed-size-compressed '550Mi' ) fi +elif [[ "${package_dir}" == "python/libcuopt_mathopt" ]] || \ + [[ "${package_dir}" == "python/libcuopt_routing" ]]; then + # Each is strictly smaller than the combined libcuopt wheel was, so its limits are a + # safe ceiling. Worth tightening once CI reports the real compressed sizes. + if [[ "${RAPIDS_CUDA_MAJOR}" == "12" ]]; then + PYDISTCHECK_ARGS+=( + --max-allowed-size-compressed '725Mi' + ) + else + PYDISTCHECK_ARGS+=( + --max-allowed-size-compressed '550Mi' + ) + fi +elif [[ "${package_dir}" == "python/libcuopt_client" ]]; then + # No CUDA kernels: libcuopt_client.so is single-digit MB, so anything near the other + # limits would mean something got vendored into it that should not have been. + PYDISTCHECK_ARGS+=( + --max-allowed-size-compressed '50Mi' + ) elif [[ "${package_dir}" != "python/cuopt" ]] && \ [[ "${package_dir}" != "python/cuopt/cuopt/linear_programming" ]] && \ [[ "${package_dir}" != "python/cuopt_server" ]] && \ From e95b09c5650334f8813cd9512d6e83d95cc63d24 Mon Sep 17 00:00:00 2001 From: Ramakrishna Prabhu Date: Fri, 18 Sep 2026 21:50:41 -0500 Subject: [PATCH 6/7] review: document and annotate the new public loaders Adds return annotations and documents the contract for the three load_library functions and the cuopt_cli entry point: the empty-list result, that a missing library warns rather than raising, and that main never returns because execv replaces the process image. Co-Authored-By: Claude Opus 5 --- python/libcuopt_client/libcuopt_client/load.py | 10 ++++++++-- .../libcuopt_mathopt/libcuopt_mathopt/_cli_wrapper.py | 11 ++++++++--- python/libcuopt_mathopt/libcuopt_mathopt/load.py | 11 +++++++++-- python/libcuopt_routing/libcuopt_routing/load.py | 11 +++++++++-- 4 files changed, 34 insertions(+), 9 deletions(-) diff --git a/python/libcuopt_client/libcuopt_client/load.py b/python/libcuopt_client/libcuopt_client/load.py index 3a663576da..5fc4cefe29 100644 --- a/python/libcuopt_client/libcuopt_client/load.py +++ b/python/libcuopt_client/libcuopt_client/load.py @@ -35,8 +35,14 @@ def _load_wheel_installation(soname: str): return None -def load_library(): - """Dynamically load libcuopt_client.so and its dependencies.""" +def load_library() -> list: + """Dynamically load libcuopt_client.so and its dependencies. + + Returns a single-element list holding the ``ctypes.CDLL`` handle, or an + empty list when the library could not be loaded; callers rarely need the + handle itself. A missing library warns rather than raising, so that the + system loader still gets a chance to resolve it. + """ try: # librmm and rapids_logger must be loaded before libcuopt_client.so, # which references them. diff --git a/python/libcuopt_mathopt/libcuopt_mathopt/_cli_wrapper.py b/python/libcuopt_mathopt/libcuopt_mathopt/_cli_wrapper.py index 78e95280b7..4a7bdc584f 100644 --- a/python/libcuopt_mathopt/libcuopt_mathopt/_cli_wrapper.py +++ b/python/libcuopt_mathopt/libcuopt_mathopt/_cli_wrapper.py @@ -3,11 +3,16 @@ import os import sys +from typing import NoReturn -def main(): - """ - This connects to cli binary which situated under libcuopt/bin folder +def main() -> NoReturn: + """Exec the cuopt_cli binary, forwarding this process's arguments to it. + + Never returns: execv replaces the process image. Raises OSError if the + binary is missing or not executable. + + This connects to cli binary which situated under libcuopt_mathopt/bin folder execv replaces this process rather than spawning a child, so signals sent to the console script's pid reach the solver directly instead of stopping diff --git a/python/libcuopt_mathopt/libcuopt_mathopt/load.py b/python/libcuopt_mathopt/libcuopt_mathopt/load.py index e35fdd6315..e3ba2e5320 100644 --- a/python/libcuopt_mathopt/libcuopt_mathopt/load.py +++ b/python/libcuopt_mathopt/libcuopt_mathopt/load.py @@ -34,8 +34,15 @@ def _load_wheel_installation(soname: str): return None -def load_library(): - """Dynamically load libcuopt_mathopt.so and its dependencies.""" +def load_library() -> list: + """Dynamically load libcuopt_mathopt.so and its dependencies. It loads libcuopt_client first, which + this library has a DT_NEEDED on. + + Returns a single-element list holding the ``ctypes.CDLL`` handle, or an + empty list when the library could not be loaded; callers rarely need the + handle itself. A missing library warns rather than raising, so that the + system loader still gets a chance to resolve it. + """ try: # librmm and rapids_logger must be loaded before libcuopt_mathopt.so, # which references them. diff --git a/python/libcuopt_routing/libcuopt_routing/load.py b/python/libcuopt_routing/libcuopt_routing/load.py index b4edda442f..1cc2f42360 100644 --- a/python/libcuopt_routing/libcuopt_routing/load.py +++ b/python/libcuopt_routing/libcuopt_routing/load.py @@ -34,8 +34,15 @@ def _load_wheel_installation(soname: str): return None -def load_library(): - """Dynamically load libcuopt_routing.so and its dependencies.""" +def load_library() -> list: + """Dynamically load libcuopt_routing.so and its dependencies. It loads libcuopt_client first, which + this library has a DT_NEEDED on. + + Returns a single-element list holding the ``ctypes.CDLL`` handle, or an + empty list when the library could not be loaded; callers rarely need the + handle itself. A missing library warns rather than raising, so that the + system loader still gets a chance to resolve it. + """ try: # librmm and rapids_logger must be loaded before libcuopt_routing.so, # which references them. From a664b1e01a36c407cbe733677c8ed1fa0aa93dd3 Mon Sep 17 00:00:00 2001 From: Ramakrishna Prabhu Date: Sat, 19 Sep 2026 20:44:07 -0500 Subject: [PATCH 7/7] build(wheel): make libcuopt depend on the component wheels libcuopt stages only the ld script and cuopt_grpc_server, but its dependencies were still the CUDA stack it no longer ships, and it named none of the component wheels that now carry the libraries. pip install libcuopt would have pulled cudss and nccl while getting no engine at all. The earlier edit to python/libcuopt/pyproject.toml did not survive, because dependencies.yaml is the source of truth and py_run_libcuopt was never updated; the generator rewrote the file back. Co-Authored-By: Claude Opus 5 --- dependencies.yaml | 10 ++++++---- python/libcuopt/pyproject.toml | 9 +++------ 2 files changed, 9 insertions(+), 10 deletions(-) diff --git a/dependencies.yaml b/dependencies.yaml index fcf8babdc1..b62823d101 100644 --- a/dependencies.yaml +++ b/dependencies.yaml @@ -139,6 +139,9 @@ files: - depends_on_libraft_headers - depends_on_librmm - depends_on_rapids_logger + # libcuopt now carries only the ld script and cuopt_grpc_server; the libraries live in + # the component wheels, so it depends on those rather than on the CUDA stack directly. + # Their own dependency sets bring rmm, rapids-logger and the CUDA wheels they each need. py_run_libcuopt: output: pyproject pyproject_dir: python/libcuopt @@ -148,10 +151,9 @@ files: extras: table: project includes: - - cuda_wheels - - depends_on_libraft_headers - - depends_on_librmm - - depends_on_rapids_logger + - depends_on_libcuopt_client + - depends_on_libcuopt_mathopt + - depends_on_libcuopt_routing # One set of keys per component wheel. The point of the split is in py_run_*: # only mathopt carries cudss / nccl / nvjitlink, so a routing-only install no # longer pulls libraries it never calls. diff --git a/python/libcuopt/pyproject.toml b/python/libcuopt/pyproject.toml index fd02478394..68829096a5 100644 --- a/python/libcuopt/pyproject.toml +++ b/python/libcuopt/pyproject.toml @@ -30,12 +30,9 @@ classifiers = [ "Programming Language :: Python :: 3.14", ] dependencies = [ - "cuda-toolkit[cublas,cudart,curand,cusolver,cusparse,nvtx]==13.*", - "librmm==26.10.*,>=0.0.0a0", - "nvidia-cudss-cu13>=0.7,<0.8", - "nvidia-nccl-cu13>=2.19", - "nvidia-nvjitlink>=13.3,<14", - "rapids-logger==0.3.*", + "libcuopt-client==26.10.*,>=0.0.0a0", + "libcuopt-mathopt==26.10.*,>=0.0.0a0", + "libcuopt-routing==26.10.*,>=0.0.0a0", ] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`. [project.urls]