Skip to content

ci(gpu): native self-hosted GPU build + test workflow#1044

Open
yingjerkao wants to merge 3 commits into
masterfrom
ci/gpu-selfhosted-tests
Open

ci(gpu): native self-hosted GPU build + test workflow#1044
yingjerkao wants to merge 3 commits into
masterfrom
ci/gpu-selfhosted-tests

Conversation

@yingjerkao

Copy link
Copy Markdown
Collaborator

Problem

No GPU CI exists (origin issue #538). ci-cmake_tests.yml is CPU-only; the old gpu-smoke-test (main.yml) targeted a non-existent v100 runner and shelled out to Docker (not installed), so it never ran a single job. And after #1004 gpu_test_main didn't even compile: its host-compiled .cpp TUs include Type.hpp<cuda/std/complex>, but cytnx adds the CUDA toolkit + CCCL includes PRIVATE, so they never reached the test target (fatal error: cuda/std/complex: No such file or directory).

Fix

  • tests/gpu/CMakeLists.txt — give gpu_test_main the CUDA toolkit + CCCL include dirs (the compile fix); tag its cases with the gpu ctest label; enumerate DISCOVERY_MODE PRE_TEST so a CUDA build links even on a driver-less machine.
  • CytnxBKNDCMakeLists.cmake — expose the detected CCCL dir as CYTNX_CCCL_INCLUDE_DIR (single source of truth, reused by the test target rather than re-detected).
  • CMakePresets.json — add a gpu-only test preset (filters on the gpu label).
  • .github/workflows/ci-gpu_tests.yml — native build on the self-hosted GPU runner using the box's CUDA / cuTENSOR / cuQuantum (roots come from repo vars.CUTENSOR_ROOT / vars.CUQUANTUM_ROOT, already configured); builds gpu_test_main and runs ctest -L gpu. Manual dispatch + same-repo PRs only — fork PRs never execute on self-hosted hardware.
  • Remove the dead gpu-smoke-test workflow (main.yml).

The gpu label + PRE_TEST groundwork is adapted from the draft #1023 (which targeted a GitHub-hosted runner and, being unrun, still carried the CCCL compile blocker). This supersedes #1023's direction — suggest closing it once this lands.

Testing

On the runner box (2× RTX 4070 Ti SUPER, sm_89, CUDA 13.0, cuTENSOR 2.0, cuQuantum):

  • cmake configure + cmake --build --target gpu_test_main succeed — CUDA lib + test binary, no cuda/std/complex error.
  • ctest -L gpu discovers 788 GPU-labeled tests; AccessorTest passes 15/15 on-GPU.
  • This PR's own pull_request run exercises the workflow on the self-hosted runner end-to-end.

Known runner-env caveat (not a blocker)

The runner's cuquantum conda env is CUDA-11-built, so linking warns libcublas.so.11 … may conflict with libcublas.so.13. Core tensor tests are unaffected; updating that env to a CUDA-13 cuQuantum build clears the warning.

Toward #538.

🤖 Generated with Claude Code

Problem: there is still no GPU CI (the origin issue #538). ci-cmake_tests.yml
is CPU-only, and the existing gpu-smoke-test (main.yml) targeted a non-existent
`v100` self-hosted runner and shelled out to Docker (not installed on the
runner), so it never ran a single job. Separately, gpu_test_main could not even
compile after #1004: its host-compiled .cpp TUs include Type.hpp, which pulls
<cuda/std/complex>, but cytnx adds the CUDA toolkit + CCCL include dirs PRIVATE,
so they never propagated to the test target.

Fix:
- tests/gpu/CMakeLists.txt: give gpu_test_main the CUDA toolkit + CCCL include
  dirs (fixes "fatal error: cuda/std/complex: No such file or directory"), tag
  its cases with the `gpu` ctest label, and enumerate them DISCOVERY_MODE
  PRE_TEST so a CUDA-enabled build still links on a driver-less machine.
- CytnxBKNDCMakeLists.cmake: expose the detected CCCL dir as
  CYTNX_CCCL_INCLUDE_DIR so the test target reuses cytnx's single-source
  detection instead of duplicating it.
- CMakePresets.json: add a `gpu-only` test preset (filters on the `gpu` label).
- .github/workflows/ci-gpu_tests.yml: native build on the self-hosted GPU
  runner using the box's CUDA + cuTENSOR + cuQuantum (roots via repo vars),
  building gpu_test_main and running `ctest -L gpu`. Manual dispatch plus
  same-repo PRs only; fork PRs never execute on self-hosted hardware.
- Remove the dead gpu-smoke-test workflow (main.yml).

Testing: on the runner box (2x RTX 4070 Ti SUPER, sm_89, CUDA 13.0, cuTENSOR
2.0, cuQuantum) configure + `cmake --build --target gpu_test_main` succeed,
`ctest -L gpu` discovers 788 tests, and AccessorTest passes 15/15 on-GPU.

Toward #538.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds a "gpu-only" test preset and updates the CMake configuration to ensure host-compiled consumers of Cytnx public headers (like gpu_test_main) can correctly locate CUDA and CCCL headers. It also tags discovered GPU tests with the gpu label and sets the discovery mode to PRE_TEST. The review feedback recommends wrapping the CUDAToolkit_INCLUDE_DIRS include directory command in an if condition to prevent a CMake syntax error if the variable is empty.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread tests/gpu/CMakeLists.txt
# and CCCL include dirs PRIVATEly, so they do not propagate here through
# linking; add them explicitly, otherwise the host compile fails with
# "fatal error: cuda/std/complex: No such file or directory".
target_include_directories(gpu_test_main PRIVATE ${CUDAToolkit_INCLUDE_DIRS})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If CUDAToolkit_INCLUDE_DIRS is empty or undefined (which can happen in some environments where CUDA is installed in standard system paths), the command expands to target_include_directories(gpu_test_main PRIVATE), which is a CMake syntax error and will cause the configuration step to fail. Wrapping it in an if check ensures that the directory is only added if the variable is non-empty, preventing potential configuration failures.

if(CUDAToolkit_INCLUDE_DIRS)
  target_include_directories(gpu_test_main PRIVATE ${CUDAToolkit_INCLUDE_DIRS})
endif()

@codecov

codecov Bot commented Jul 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 57.71%. Comparing base (ba9aff4) to head (b2329d3).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1044   +/-   ##
=======================================
  Coverage   57.71%   57.71%           
=======================================
  Files         229      229           
  Lines       33468    33468           
  Branches       71       71           
=======================================
  Hits        19315    19315           
  Misses      14132    14132           
  Partials       21       21           
Flag Coverage Δ
cpp 57.64% <ø> (ø)
python 63.35% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
C++ backend 55.47% <ø> (ø)
Python bindings 71.99% <ø> (ø)
Python package 63.35% <ø> (ø)

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ba9aff4...b2329d3. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

yingjerkao and others added 2 commits July 11, 2026 22:47
BlockUniTensorTest.gpu_Trace read the rank-0 scalar from UT_diag.Trace(0, 1)
via at({0}), which #1026 (rank-0 UniTensor semantics) now rejects with
"rank-0 scalar expects no locator indices". The CPU Trace test was updated to
at({}) in #1026; the GPU test was missed because GPU tests don't run in CI.
Mirror the CPU fix. Surfaced by this PR's new GPU CI on its first run.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comment on lines +71 to +77
run: >-
cmake -S . -B "$BUILD_DIR"
-DCMAKE_BUILD_TYPE=Release
-DUSE_CUDA=ON -DUSE_CUTENSOR=ON -DUSE_CUQUANTUM=ON
-DRUN_TESTS=ON -DBUILD_PYTHON=OFF
-DCMAKE_CUDA_ARCHITECTURES="$GPU_CUDA_ARCH"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider to use preset debug-openblas-cuda or add a new test preset like #1023

Comment on lines +39 to +40
CUTENSOR_ROOT: ${{ vars.CUTENSOR_ROOT }}
CUQUANTUM_ROOT: ${{ vars.CUQUANTUM_ROOT }}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we can control the dependency versions like #1023 will be better. Because that will be the versions we will release.

@yingjerkao
yingjerkao requested review from IvanaGyro and ianmccul July 24, 2026 09:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants