ci(release): publish cytnx-cuda GPU wheels to PyPI (nightly + stable)#1023
Merged
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces a new gpu-only test preset in CMakePresets.json to run only GPU-labeled tests, and updates tests/gpu/CMakeLists.txt to label the gpu_test_main tests with gpu. The feedback suggests removing the redundant "configurePreset" property from the new preset, as it is already inherited from cpu-and-cuda.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
IvanaGyro
force-pushed
the
claude/gpu-ci-workflow-plan-24zmuj
branch
2 times, most recently
from
July 9, 2026 13:45
7b3a88b to
328fc17
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1023 +/- ##
===========================================
+ Coverage 60.49% 72.11% +11.61%
===========================================
Files 229 225 -4
Lines 31873 28207 -3666
Branches 71 71
===========================================
+ Hits 19282 20341 +1059
+ Misses 12570 7845 -4725
Partials 21 21
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 28 files with indirect coverage changes Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
IvanaGyro
force-pushed
the
claude/gpu-ci-workflow-plan-24zmuj
branch
3 times, most recently
from
July 9, 2026 15:28
5ded4ba to
083b795
Compare
Member
Author
|
@yingjerkao This PR successfully builds with CUDA. The large action runner label is needed for running tests on the GPU machine. |
IvanaGyro
commented
Jul 15, 2026
tests/gpu/gpu_test_main currently has no ctest label, so `ctest --preset cpu-and-cuda` runs every test in the build (the CPU suite under tests/test_main plus the GPU suite) with no way to select just the GPU tests. A GPU CI job needs to run only the GPU suite, since the CPU suite is already covered for free by ci-cmake_tests.yml. Add a "gpu" ctest label to tests/gpu/CMakeLists.txt's gtest_discover_tests() call, and a "gpu-only" CTest preset in CMakePresets.json that filters on it, so both CI and local runs can do `ctest --preset gpu-only` (or `ctest -L gpu`) to run just gpu_test_main's tests. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds ci-gpu_tests.yml, mirroring ci-cmake_tests.yml's build/test flow but for a CUDA build: - Triggers on pull_request (targeting master) and workflow_dispatch, not push to master, since GitHub-hosted GPU runners are billed by uptime and pull_request review is where GPU-affecting changes need the gate. - Targets a GitHub-hosted GPU runner via the vars.GPU_RUNNER_LABEL repository variable rather than the self-hosted runner main.yml already declares (labels self-hosted/linux/x64/cuda/gpu/v100) — that runner's one recorded workflow_dispatch run (id 28780795955) sat queued for 24 hours with runner_id 0 and was auto-cancelled, i.e. never actually picked up by a runner. - Checks out without submodules, merges the PR's target branch first (pull_request only), then initializes submodules once — so a merge that moves a gitlink only costs one submodule checkout instead of one before the merge plus a refresh after, which is what ci-cmake_tests.yml's checkout-with-submodules-then-merge order costs. - Installs the CUDA toolchain, cuTENSOR, and cuQuantum from PyPI (nvidia-cuda-nvcc, nvidia-cuda-runtime, nvidia-cublas, nvidia-cusolver, nvidia-cusparse, nvidia-curand, cutensor, cuquantum) instead of a system package manager, and locates each package's include/lib roots by searching the installed wheel contents rather than assuming a fixed path, since the on-disk layout isn't pinned by these packages' metadata. - Builds Release (not Debug/ASan — this job isn't collecting coverage, and ASan is known to abort under CUDA) by overriding pyproject.toml's pinned cmake.args (--preset=openblas-cpu) to select openblas-cuda instead; scikit-build-core's cmake.args setting replaces rather than merges with the pyproject.toml value (scikit_build_core.settings.skbuild_model.CMakeSettings.args), so this is safe. RUN_TESTS=ON is layered on top via the (additive) cmake.define setting, same as ci-cmake_tests.yml already does for CMAKE_BUILD_TYPE. - Runs only the "gpu" ctest label added in the preceding commit, so it doesn't redundantly re-run the CPU suite ci-cmake_tests.yml already covers, then runs pytest with no coverage collection or Codecov upload. The runner label is still unset (a repo maintainer needs to configure the GPU_RUNNER_LABEL variable once the GitHub-hosted GPU runner group exists), and this hasn't had a real run yet to confirm the PyPI packages' on-disk layout matches what the "Locate CUDA toolkit and cuTENSOR/cuQuantum roots" step expects, or to produce the per-run cost cited in #538. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…sion
CUDA 13.0 removed offline compilation support for Maxwell, Pascal, and
Volta (sm_70 and below) — nvcc from a CUDA 13 toolkit rejects sm_70
outright ("Unsupported gpu architecture 'compute_70'"), which fails
enable_language(CUDA)'s own internal compiler ABI-detection compile
before any project code is reached, since the unconditional default set
here included 70-real regardless of the detected toolkit version.
Move the caller-override detection (-D/cache/preset/CUDAARCHS) ahead of
enable_language(CUDA) as before, but no longer set CMAKE_CUDA_ARCHITECTURES
before that call — enable_language(CUDA)'s own internal ABI-detection
compile runs with whatever value is set at that point, and CMake's
toolchain-aware default handles it correctly regardless of CUDA version.
After enable_language(CUDA) and find_package(CUDAToolkit) run, set
CMAKE_CUDA_ARCHITECTURES for the actual project targets based on
CUDAToolkit_VERSION: the 70-real floor stays for CUDA <13, and CUDA >=13
starts at 75-real (Turing) instead.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
find_package(CUDAToolkit REQUIRED) failed with "Unable to find cudart library" / "Could NOT find CUDAToolkit (missing: CUDA_CUDART)" even though it correctly detected the nvcc version (13.3.73). Inspecting the actual nvidia-cuda-runtime wheel contents shows why: it ships lib/libcudart.so.13 (and libcudart_static.a), but no unversioned lib/libcudart.so. The same pattern holds across nvidia-cublas (libcublas.so.13, libcublasLt.so.13, libnvblas.so.13) and cutensor-cu13 (libcutensor.so.2, libcutensorMg.so.2, libcutensorMp.so.2) — none of these PyPI wheels ship the unversioned soname that find_library(NAMES foo), used internally by FindCUDAToolkit.cmake, FindCUTENSOR.cmake, and FindCUQUANTUM.cmake, requires to match. Add a step that walks site-packages after the CUDA/cuTENSOR/cuQuantum pip install and symlinks each versioned libFOO.so.N to an unversioned libFOO.so alongside it, so those find_library calls succeed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
FindCUTENSOR.cmake picks a PATH_SUFFIXES value of lib/<CUDA major> (e.g. lib/12 for CUDA 13, since the module's version check only special-cases 10/11 and treats everything >=12 the same), matching the layout of NVIDIA's standalone cuTENSOR tarball releases. The cutensor-cu13 PyPI wheel doesn't use that layout: inspecting its contents shows lib/libcutensor.so.2 and lib/libcutensorMg.so.2 directly under lib/, with no CUDA-version subdirectory, so find_library never found them there and CMake's configure step failed with CUTENSOR_LIB and CUTENSORMg_LIB both NOTFOUND. Add "lib" as an additional PATH_SUFFIXES entry after the versioned one, so find_library falls back to the flat layout when the versioned subdirectory doesn't contain the library. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Device linking failed with "nvlink fatal: elfLink linker library load error (target: sm_75)" during cmake_device_link.o, after ~240/347 objects (including CUDA kernels) had already compiled successfully. nvcc's -dlto device-link mode invokes nvlink, which dlopen()s libnvJitLink.so at runtime to perform the actual link. That's a dynamic loader lookup (LD_LIBRARY_PATH), entirely separate from CMake's find_library — the .so symlinks added in c33c440 only affect CMake's own configure-time variable resolution, not what a spawned nvlink process can dlopen. The workflow never exported LD_LIBRARY_PATH anywhere, so nvlink had no way to find nvidia-nvjitlink's libnvJitLink.so.13 (confirmed by downloading that wheel: it installs to nvidia/cu13/lib, the same shared directory as every other CUDA component here). Export LD_LIBRARY_PATH covering the CUDAToolkit/cuTENSOR/cuQuantum lib directories once they're known, alongside the existing CMake hint variables. This also covers the built cytnx extension needing the same libraries on the loader path at import/test time. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The LD_LIBRARY_PATH export in 2880c22 didn't fix nvlink's "elfLink linker library load error (target: sm_75)" — the same failure recurred on the next run with LD_LIBRARY_PATH correctly set (confirmed via the "Report resolved CUDA locations" step's output). nvlink resolves libnvJitLink.so via a path hardcoded relative to its own bin/ directory, following the traditional standalone CUDA Toolkit layout (.../bin/nvlink -> .../lib64/libnvJitLink.so). A dlopen() call using a relative path like that is resolved directly against that path, bypassing LD_LIBRARY_PATH entirely — unlike a bare-soname dlopen(), which does search LD_LIBRARY_PATH (and is what the extension itself relies on at import time, so that fix stays). The nvidia-cuda-nvcc/nvidia-cuda-runtime wheels install to nvidia/cu13/lib/, never lib64/ (confirmed: no lib64 anywhere in the nvidia-cuda-nvcc wheel contents). Symlink lib64 -> lib inside CUDAToolkit_ROOT so nvlink's hardcoded relative lookup resolves. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both the LD_LIBRARY_PATH export (2880c22) and the lib64 -> lib symlink (80c5d2f) failed to fix "nvlink fatal: elfLink linker library load error (target: sm_75)" during cmake_device_link.o — the identical failure recurred on both follow-up runs with each fix in place. The actual source of -dlto (device link-time optimization) in that nvlink invocation is CMakeLists.txt:311, which sets CMAKE_INTERPROCEDURAL_OPTIMIZATION TRUE unconditionally on non-Apple platforms; for CUDA targets CMake maps that property to -dlto on the device-link step. Two independent, plausible fixes for what nvlink loads at that step both failed to change the outcome, which points at DLTO's device linking needing something the pip-wheel CUDA toolchain doesn't provide here, rather than a simple missing-file/path problem — unlike the monolithic CUDA installs the repo's normal build path is presumably validated against. Override CMAKE_INTERPROCEDURAL_OPTIMIZATION=OFF for this job only, rather than changing the CMakeLists.txt default: this only affects the CI build's binary performance, not correctness, and the repo-wide default is left alone since it's presumably fine for monolithic installs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The GPU CI workflow's --config-settings=cmake.define.CMAKE_INTERPROCEDURAL_OPTIMIZATION=OFF (6944511) had no effect: nvlink's device-link step still received -dlto on the next run, identical to before that commit. CMakeLists.txt:311 (now inside the added if-block) called set(CMAKE_INTERPROCEDURAL_OPTIMIZATION TRUE) unconditionally on non-Apple platforms. A plain set() with no CACHE keyword creates a normal variable that shadows a cache entry of the same name for the rest of the script — so any caller-supplied -D value, no matter what it is, was silently discarded the moment this line ran. This is a general bug independent of the GPU CI workflow: it means -DCMAKE_INTERPROCEDURAL_OPTIMIZATION=OFF never worked for any caller on a non-Apple platform. Guard the default behind NOT DEFINED, the same pattern this file already uses for CMAKE_CUDA_ARCHITECTURES, so a caller-provided value survives. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ble DLTO Both prior attempts at this same "nvlink fatal: elfLink linker library load error" (2880c22's LD_LIBRARY_PATH export, 80c5d2f's top-level lib64 -> lib symlink) targeted the wrong library and the wrong path: nvlink's device-link step needs libnvvm.so specifically (NVVM, the LLVM-based device codegen backend used during DLTO), resolved via -nvvmpath (derived from nvcc's own install prefix) + "/lib64/libnvvm.so" — not a bare-soname dlopen() search, and not relative to CUDAToolkit_ROOT/lib64 either. Inspecting the actual nvidia-nvvm wheel contents confirms the mismatch: it installs a nvvm/ directory (bin/cicc, libdevice/) that nvcc's -nvvmpath points at, but the real library lives at lib/libnvvm.so.N alongside every other CUDA component here — there is no nvvm/lib64/ at all in the wheel, which is exactly the path nvlink tries to open. This mirrors a known issue with the Debian/Ubuntu nvidia-cuda-toolkit apt package, which has the same nvvmpath/lib64/libnvvm.so vs. actual-location mismatch for the same underlying reason (multiarch library layout vs. nvcc's hardcoded expectation). Locate the real libnvvm.so.N and symlink it into place at nvvm/lib64/libnvvm.so. With the actual mechanism fixed, revert 6944511's CMAKE_INTERPROCEDURAL_OPTIMIZATION=OFF override — DLTO no longer needs to be disabled (b18e0a4's CMakeLists.txt fix, which makes that override actually work when needed, stays either way). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Device_class's constructor routed cudaGetDeviceCount()'s return value through checkCudaErrors, which calls exit(EXIT_FAILURE) on any non-success code. cudaGetDeviceCount() legitimately returns cudaErrorNoDevice or cudaErrorInsufficientDriver on a machine with no CUDA-capable device or no NVIDIA driver installed -- the CUDA Runtime API is designed to let a caller probe for GPU absence this way -- so a CUDA-enabled cytnx build crashed the whole process (via a raw C exit(), not a catchable C++/Python exception) the moment it was imported on any GPU-less machine, before any user code could run. Capture the return value once and treat only those two "no hardware" codes as Ngpus == 0; any other, genuinely unexpected error still goes through checkCudaErrors as before. Downstream code already handles Ngpus == 0 gracefully (e.g. Storage.cpp falls back to CPU when a saved tensor's device id is out of range), so this only changes the constructor's own error handling, not how the rest of the codebase reacts to a GPU-less environment. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
#1055 added a repo-wide convention for PR-triggered workflows: key the concurrency group by PR number (falling back to the run id, not the ref, for non-PR events, so a push/dispatch never shares a group with another and risks having a still-pending run silently replaced) and scope cancel-in-progress to pull_request events only. This workflow predates that PR and still used its own group naming; align it now that #1055 has merged. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
IvanaGyro
force-pushed
the
claude/gpu-ci-workflow-plan-24zmuj
branch
from
July 15, 2026 18:44
f71d51d to
fffe488
Compare
Renamed the workflow file and its display name ("GPU Tests" ->
"CUDA Tests") to match ci-cmake_tests.yml's naming convention and
disambiguate it from the upcoming release_pypi_cuda.yml release
pipeline: this workflow is a correctness/test gate, not a release
workflow.
Also switched the GPU_RUNNER_LABEL fallback from the floating
ubuntu-latest alias to the pinned ubuntu-24.04, matching the runner
convention release_pypi.yml already uses.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This PR's CUDA CI coverage moves to the new release_pypi_cuda.yml release pipeline instead of a separate ad-hoc test-only workflow. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Rewrites pyproject.toml in place for a cytnx-cuda release build: renames the PyPI project to cytnx-cuda, adds the NVIDIA/cuTENSOR/ cuQuantum runtime packages cytnx links directly as ordinary [project.dependencies] pinned to the exact versions this wheel is built against (PyTorch approach: dynamically linked at runtime from pip packages, never vendored -- see cytnx/_cuda_preload.py, added next, for how they get resolved without LD_LIBRARY_PATH), switches the CMake preset to openblas-cuda, and overrides [tool.cibuildwheel.linux].repair-wheel-command to exclude those same libraries' sonames from auditwheel's vendoring so the wheel stays small. Only nvidia-cuda-runtime/nvidia-cublas/nvidia-cusparse/nvidia-curand/ nvidia-cusolver/cutensor-cu13/cutensornet-cu13/custatevec-cu13 are listed directly; each pulls in its own transitive NVIDIA dependencies via pip's resolver (verified against each package's requires_dist on PyPI: nvidia-cusolver -> nvidia-cublas/nvidia-nvjitlink/ nvidia-cusparse, nvidia-cublas -> nvidia-cuda-nvrtc, cutensornet-cu13 -> cutensor-cu13), so they don't need separate entries. cuquantum-cu13 itself is deliberately not a dependency: it pulls in cudensitymat/cupauliprop/custabilizer, none of which cytnx links. Verified by running the script against a scratch copy of pyproject.toml and checking the result both parses as valid TOML and has the expected name/dependencies/cmake.args/repair-wheel-command. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
IvanaGyro
commented
Jul 15, 2026
The NVIDIA/cuTENSOR/cuQuantum runtime dependencies in tools/prepare_cuda_release.py were pinned with exact-version (==) specifiers. Switch them to PEP 440 compatible-release (~=) specifiers instead, so pip can resolve a patch release within the same minor version rather than requiring the exact build/link-time version. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
tools/prepare_cuda_release.py excludes libcudart/libcublas/libcutensor/ etc. from auditwheel's vendoring, relying on the pip runtime packages (nvidia-cuda-runtime, nvidia-cublas, cutensor-cu13, ...) instead. Those packages install into site-packages namespace directories (nvidia/cu13/lib, cutensor/lib, cuquantum/lib) that are outside the dynamic linker's default search path, so importing the compiled extension without LD_LIBRARY_PATH set would fail to resolve those sonames. cytnx/_cuda_preload.py loads each discovered library with ctypes.CDLL(..., RTLD_GLOBAL) before the extension import, so its NEEDED entries resolve against libraries already mapped into the process by soname. Dependency ordering among the libraries is handled by retrying failed loads in a fixed-point loop rather than a hardcoded order. cytnx/__init__.py calls preload() only when vinfo.tmp carries a "cuda" marker, which CMakeLists.txt now appends when USE_CUDA is ON, so this stays a no-op on CPU-only installs -- the marker is read directly from the file before `from . import cytnx`, since importing the extension is what would otherwise reveal whether this is a CUDA build. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
cibuildwheel's manylinux container has no system CUDA install, so
compiling cytnx-cuda's extension needs nvcc plus the CUDA/cuTENSOR/
cuQuantum dev headers and import libraries available inside the
container.
tools/cibuildwheel_before_all_cuda.sh installs the toolchain
(nvidia-cuda-nvcc plus the same nvidia-*/cutensor-cu13/cuquantum
packages already declared as wheel runtime dependencies) into an
isolated --target prefix, chosen so it doesn't leak into the wheel's
own dependency resolution. All nvidia-* packages share the "nvidia"
namespace package, so installing them together merges into one
nvidia/cu13/{bin,include,lib} tree usable as a CUDAToolkit_ROOT-style
layout.
tools/prepare_cuda_release.py now chains that script onto
[tool.cibuildwheel.linux].before-all (after the existing arpack/
openblas/boost install, which cytnx-cuda still needs for its non-GPU
paths) and points CMAKE_CUDA_COMPILER/CUTENSOR_ROOT/CUQUANTUM_ROOT/
PATH/CMAKE_PREFIX_PATH at the installed toolchain. The toolchain
package versions are derived from CUDA_RUNTIME_DEPENDENCIES rather
than duplicated, so both stay pinned to the same versions from one
list.
Rebuilds the environment inline table from scratch rather than
assigning new keys onto the parsed one in place, since tomlkit was
found (via a scratch pyproject.toml round-trip test) to omit the
separating comma when a key is added to an already-parsed
InlineTable, silently producing invalid TOML.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
PACKAGE was hardcoded to "cytnx", but the cytnx-nightly-wheels anaconda.org channel is about to host a second package (cytnx-cuda) that needs its own independent pruning. Adds a --package flag (default: cytnx, matching release_pypi.yml's existing invocation unchanged) and threads it through fetch_versions/delete_version instead of the module-level constant. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…line Mirrors release_pypi.yml's DetermineChannel/BuildWheel/publish structure for the GPU package: master push -> nightly upload to the cytnx-nightly-wheels anaconda.org channel (shared with the CPU nightlies; no filename collision since the package name is cytnx-cuda), v* tag push -> stable release to PyPI via trusted publishing, plus workflow_dispatch. Not run on pull_request -- an nvcc build is too slow/expensive to gate every PR on, and this repo has no GitHub-hosted GPU runner budget for it (#538). Single Linux x86_64 manylinux_2_28 leg on a plain (GPU-less) ubuntu-24.04 runner. tools/prepare_cuda_release.py and tools/cibuildwheel_before_all_cuda.sh (both already on this branch) do the actual pyproject.toml rewrite and CUDA toolchain provisioning; this workflow just wires the trigger/version-stamping/publish scaffolding around them, same as release_pypi.yml does for the CPU package. CIBW_TEST_COMMAND asserts cytnx.Device.Ngpus == 0 in addition to the existing ccache-stats check, since this builder has no GPU -- this pipeline can only prove the extension builds, links, and imports cleanly, not that GPU code paths are correct on real hardware. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
IvanaGyro
changed the title
The module's logic (CUDA-build marker detection, namespace-package discovery, the fixed-point RTLD_GLOBAL retry loop) was previously untestable in CI: no GPU-less environment has the real NVIDIA/ cuTENSOR/cuQuantum packages installed, and cytnx/__init__.py's own inline marker check duplicated logic that lived nowhere importable in isolation. Moves the vinfo.tmp marker check into _cuda_preload.is_cuda_build(), independently testable with a tmp_path fixture instead of only being exercised implicitly by every pytest run's `import cytnx`. preload() now accepts optional lib_paths/loader parameters (defaulting to the real discovery/ctypes.CDLL behavior), so its dependency-order fixed-point retry loop and ImportError path can be exercised with fake libraries instead of requiring real CUDA hardware. cytnx/__init__.py now unconditionally imports _cuda_preload (a pure function-definition module with no import-time side effects, so this is safe on CPU-only installs) and calls preload() only when is_cuda_build() returns true, replacing the previous private helper that duplicated the same file-reading logic inline. Verified via a real editable debug-openblas-cpu build (this sandbox's existing build dir had a stale CMake generator cache from before ninja was on PATH, requiring CMAKE_GENERATOR="Unix Makefiles" to reconfigure against it rather than deleting the dir) that `import cytnx` still succeeds cleanly, that the new pytests/cuda_preload_test.py suite passes in full, and that ~44 pre-existing tests spanning several files continue to pass. Running the full suite under this build's AddressSanitizer instrumentation (which needs LD_PRELOAD=libasan.so for pytest, since python3 itself isn't ASan-linked, undocumented in this repo's tooling until now) surfaced that every pytest asserting a raised C++ exception crashes -- a pre-existing, environment-level ASan/exception-propagation issue affecting many unrelated test files, not something this change introduces or touches. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
IvanaGyro
commented
Jul 16, 2026
IvanaGyro
commented
Jul 16, 2026
IvanaGyro
commented
Jul 16, 2026
release_pypi_cuda.yml previously ran only on push/tag/dispatch, so the nvcc build path got no per-PR validation. Add a pull_request trigger and mirror release_pypi.yml's publish-gating so PRs build the CUDA wheel as a build-health check but never upload: - DetermineChannel gains a `publish` output; pull_request resolves to nightly with publish=false, while push/tag/dispatch set publish=true. Both ReleasePyPI and PublishNightlyAnaconda now additionally gate on publish == 'true', so a PR run builds and stops. - The concurrency group keys on `pull_request.number || run_id` with cancel-in-progress only for pull_request events, matching release_pypi.yml (a new PR push cancels its own superseded run; a push/tag/dispatch run is never cancelled). - BuildWheel gains the "Merge with latest target branch (pull_request only)" step release_pypi.yml uses, so a re-run tests against the current target branch rather than the stale merge commit. CIBW_TEST_COMMAND still asserts cytnx.Device.Ngpus == 0, which holds on the GPU-less PR builder just as on the release builder. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…eload
The excluded NVIDIA/cuTENSOR/cuQuantum libraries were resolved at
import time by cytnx/_cuda_preload.py, which ctypes.CDLL'd each one
with RTLD_GLOBAL before the extension import, gated on a "cuda" marker
written to vinfo.tmp. Replace that whole mechanism with an
$ORIGIN-relative RUNPATH baked into the installed extension, set via
CMAKE_INSTALL_RPATH from tools/prepare_cuda_release.py (the Linux
pycytnx target already folds ${CMAKE_INSTALL_RPATH} into its
INSTALL_RPATH).
The extension's RUNPATH points at the three sibling pip-package lib
dirs (nvidia/cu13/lib, cutensor/lib, cuquantum/lib) relative to
site-packages/cytnx/, covering its own direct NEEDED entries; the
NVIDIA wheels each carry their own $ORIGIN-relative RUNPATHs across
those same dirs, so the transitive graph resolves itself. Verified
against the real wheels: e.g. libcutensornet.so.2 ships RUNPATH
$ORIGIN:$ORIGIN/../../cutensor/lib:$ORIGIN/../../nvidia/cu13/lib, and
libcutensor.so.2 / libcustatevec.so.1 reach nvidia/cu13/lib for
libcudart.
Verified auditwheel 6.7.0 preserves this build-set RUNPATH through
`repair` when the CUDA libraries are --excluded, and appends its own
vendored-libs dir to it (rather than replacing), so the excluded CUDA
libs and the vendored openblas/arpack both resolve from the one repaired
RUNPATH. This removes the need for any runtime Python preload, so
cytnx/_cuda_preload.py, its pytest, the vinfo.tmp "cuda" marker in
CMakeLists.txt, and the cytnx/__init__.py gating are all deleted.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
find_package(CUDAToolkit) failed to configure the cytnx-cuda build with "Could NOT find CUDAToolkit (missing: CUDA_CUDART)" even though nvcc 13.3.73 was detected: the pip CUDA wheels ship only versioned sonames (libcudart.so.13, libcutensor.so.2, ...), but CMake resolves the import libraries with find_library(), which matches only the unversioned libX.so name. After installing the toolchain, create the unversioned libX.so -> libX.so.N dev symlinks in each toolchain lib dir (nvidia/cu13/lib plus the cutensor/ and cuquantum/ roots CUTENSOR_ROOT/CUQUANTUM_ROOT point at), so configuration can locate cudart and the cuTENSOR/cuQuantum import libraries. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
With the dev symlinks in place, find_package(CUDAToolkit) now succeeds, but CMake's generate step then fails: the CUDA::toolkit imported target lists nvidia/cu13/include/cccl in its INTERFACE_INCLUDE_DIRECTORIES and errors when that directory is absent. CUDA 13's toolkit ships the CUDA C++ Core Libraries (thrust/cub/libcudacxx) under include/cccl, but the pip layout splits them into their own nvidia-cuda-cccl wheel, which the toolchain wasn't installing. Add nvidia-cuda-cccl to CUDA_BUILD_TOOLCHAIN (build-time only, like nvidia-cuda-nvcc -- not a wheel runtime dependency): it installs the headers under nvidia/cu13/include/cccl, satisfying the path FindCUDAToolkit references. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…kend) With the toolchain in place the cytnx-cuda build compiles every .cu translation unit, then fails at the device link step: nvlink fatal : elfLink linker library load error (target: sm_75) CMakeLists.txt defaults CMAKE_INTERPROCEDURAL_OPTIMIZATION ON for non-Apple builds, which on CUDA emits -dlto and makes nvlink -dlink run the offline device-LTO backend. The pip nvidia-cuda-nvcc wheel bundles only nvcc/cudafe++/nvlink/ptxas/fatbinary/bin2c -- not the device-LTO backend library nvlink dlopens for that step -- so it aborts. Pass -DCMAKE_INTERPROCEDURAL_OPTIMIZATION=OFF from tools/prepare_cuda_release.py (CMakeLists.txt honors an explicit override). IPO is a per-target, not per-language, property, so this also drops host -flto for the CUDA wheel -- an accepted tradeoff for a working build until the device-LTO backend is available through the pip toolchain. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Restore device (and host) LTO for the CUDA wheel by fixing the actual
root cause of the earlier "nvlink fatal: elfLink linker library load
error" instead of disabling IPO.
nvlink dlopens libnvvm for the -dlto device link step (its own strings
expose a -nvvmpath option and the exact error). libnvvm is not missing
-- nvidia-cuda-nvcc pulls nvidia-nvvm transitively -- but that wheel
relocates it to nvidia/cu13/lib/libnvvm.so.4 and leaves nvvm/ with only
bin/ and libdevice/, whereas nvlink looks for it at the canonical
<top>/nvvm/lib64/. tools/cibuildwheel_before_all_cuda.sh now recreates
nvvm/lib64/libnvvm.so{,.4} pointing back at ../../lib/libnvvm.so.4, so
nvlink finds it there.
With that in place, revert the -DCMAKE_INTERPROCEDURAL_OPTIMIZATION=OFF
workaround from tools/prepare_cuda_release.py, so the CUDA wheel keeps
the same LTO the CPU build uses.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
IvanaGyro
marked this pull request as ready for review
July 16, 2026 19:33
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8decb7ba4e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
release_pypi_cuda.yml's PublishNightlyAnaconda job runs prune_nightly_wheels.py before uploading. For the first-ever cytnx-cuda nightly, the cytnx-cuda package does not exist on the cytnx-nightly-wheels channel yet, so fetch_versions() gets a 404 from anaconda.org's /package/<org>/<pkg> endpoint. _urlopen_with_retry deliberately re-raises HTTPError, so that 404 propagated and failed the publish job before the upload could create the package. Handle a 404 in fetch_versions as an empty version list (nothing on the channel yet, nothing to prune); any other HTTP error still propagates. This also covers any future new package pruned by this script, not just cytnx-cuda. Add regression tests: 404 -> [], non-404 -> raises, and a normal version-list parse. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
yingjerkao
approved these changes
Jul 22, 2026
This was referenced Jul 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
#538 asks for GPU testing/CI, and #969 investigated a PyPI/conda CUDA packaging strategy modeled on PyTorch/JAX. cytnx has no CUDA-enabled release today. Along the way,
Device_class's constructor was found to crash the whole process on any GPU-less machine running a CUDA-enabled build (checkCudaErrorstreatscudaGetDeviceCount's legitimate "no GPU here" result as a fatal error) — which would break every import of acytnx-cudapackage outside a real GPU host, so that fix landed here first.Fix
src/Device.cpp: treatcudaErrorNoDevice/cudaErrorInsufficientDriverfromcudaGetDeviceCountasNgpus == 0instead of routing throughcheckCudaErrors's fatal exit.tools/prepare_cuda_release.py: rewritespyproject.tomlin place for acytnx-cudarelease build — renames the project, adds the NVIDIA/cuTENSOR/cuQuantum runtime packages as~=-pinned[project.dependencies](PyTorch-style dynamic linking at runtime, never vendored into the wheel), switches the CMake preset toopenblas-cuda, and excludes those same libraries fromauditwheel repair's vendoring.tools/cibuildwheel_before_all_cuda.sh(chained ontobefore-allby the script above): provisions nvcc and the CUDA/cuTENSOR/cuQuantum dev headers inside the manylinux container from the same pip packages the wheel depends on at runtime, since the container has no system CUDA install.cytnx/_cuda_preload.py: preloads those shared libraries withctypes.CDLL(..., RTLD_GLOBAL)before the compiled extension import, so itsNEEDEDsonames resolve withoutLD_LIBRARY_PATH. Gated on acudamarkerCMakeLists.txtnow writes tovinfo.tmpforUSE_CUDAbuilds, so it's a no-op on the plain CPU package.tools/prune_nightly_wheels.py: parameterized by--packageso the sharedcytnx-nightly-wheelsanaconda.org channel can prunecytnxandcytnx-cudanightlies independently..github/workflows/release_pypi_cuda.yml: new pipeline mirroringrelease_pypi.yml's channel-resolution/publish structure —masterpush → nightly upload,v*tag → stable PyPI release via trusted publishing, plusworkflow_dispatch. Not run onpull_request(an nvcc build is too slow/expensive to gate every PR on, and there's no GitHub-hosted GPU runner budget for it). Single Linux x86_64manylinux_2_28leg on a plain (GPU-less)ubuntu-24.04runner;CIBW_TEST_COMMANDassertscytnx.Device.Ngpus == 0..github/workflows/ci-gpu_tests.ymlremoved entirely — its function is superseded byrelease_pypi_cuda.yml's own build/link/import validation; real GPU-hardware test execution is out of this PR's scope.Known gaps — deliberately left open
release_pypi_cuda.ymlcan only prove the extension builds, links, and imports cleanly on a GPU-less runner — not that GPU code paths are numerically correct on real hardware.cytnx-cudaproject needs a maintainer to configure OIDC trusted publishing for this workflow file, the same one-time setuprelease_pypi.yml'scytnxproject already has.template <CytnxType T>concept-constrained members #1063 (an nvcc internal-compiler-error on master's own Storage refactor, found during this PR's rebase but unrelated to it) is tracked separately and not addressed here.Testing
src/Device.cpp's fix: full local CUDA compile+link via theopenblas-cudapreset (libcytnx.abuilt cleanly with zero errors), both before and after rebasing onto master.tools/prepare_cuda_release.py: round-tripped against scratch copies ofpyproject.tomlin both pre- and post-build(wheels): source arpack/openblas/boost from conda-forge (Linux + macOS) #1057 state, then reparsed withtomllibto confirm every rewritten field (name,dependencies,cmake.args,repair-wheel-command,before-all,environment) is correct; this caught a realtomlkitbug where mutating an already-parsed inline table in place silently drops the separating commas, producing invalid TOML — fixed by rebuilding the table from scratch instead.cytnx/_cuda_preload.py: exercised_discover_lib_paths()/preload()directly on this (CPU-only, no GPU) sandbox, confirming the no-op path behaves correctly when the NVIDIA/cuTENSOR/cuQuantum packages aren't installed.include/lib/binlayout referenced bytools/prepare_cuda_release.pyandtools/cibuildwheel_before_all_cuda.shwas verified against the real PyPI JSON API and by downloading and inspecting the actual wheel contents, not assumed..github/workflows/release_pypi_cuda.yml/tools/cibuildwheel_before_all_cuda.sh: reviewed by hand againstrelease_pypi.yml's proven structure; the actual manylinux+CUDA container build has not run anywhere in this environment (no Docker registry access, no GPU) — relying on this PR's own CI for first real validation.Relates to #538 (adds the GPU wheel-release side of that ask; real GPU-hardware ctest execution is still open elsewhere) and implements the design from #969. See #1063 for the unrelated, separately-tracked nvcc issue found along the way.