Skip to content

fix(cmake): quote CUDA architecture list so wheels ship every target - #490

Merged
KemengHuang merged 2 commits into
spiriMirror:mainfrom
Ligo04:fix/wheel-cuda-arch-list-quoting
Sep 3, 2026
Merged

fix(cmake): quote CUDA architecture list so wheels ship every target#490
KemengHuang merged 2 commits into
spiriMirror:mainfrom
Ligo04:fix/wheel-cuda-arch-list-quoting

Conversation

@Ligo04

@Ligo04 Ligo04 commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Summary

Release wheels declared five CUDA architectures but shipped one. pyuipc
0.0.27 (cp312 and cp313 manylinux both verified) contains sm_75 SASS only and
zero PTX, although pyproject.toml and python/src/uipc/compatibility.json
both asked for 75/80/86/89-real plus 89-virtual.

The cause is an unquoted variable:

set_target_properties(cuda PROPERTIES
    CUDA_SEPARABLE_COMPILATION ON
    CUDA_ARCHITECTURES ${UIPC_CUDA_ARCHITECTURES}   # <- unquoted
    CUDA_STANDARD 20)

An unquoted multi-element list expands into separate arguments, which breaks
the PROPERTIES key/value pairing. The property keeps only the first entry and
the rest are absorbed as bogus property names — measured directly: with
75-real;80-real;86-real;89-real;89-virtual, CUDA_ARCHITECTURES ends up as
75-real while 80-real becomes a property whose value is 86-real. The
89-virtual PTX entry disappears the same way. The argument count happens to
stay even here, so CMake raises no error and the build and packaging both
succeed.

Consequence on consumer Blackwell (sm_120): world.init(scene) throws

RuntimeError: [/project/src/backends/cuda/cuda_tool/launch.h:26] CUDA error
cudaOccupancyMaxPotentialBlockSize(...) (500): named symbol not found

Error 500 rather than 209 cudaErrorNoKernelImageForDevice because under
CUDA 12.x lazy module loading a fatbin with no image for the current device
fails to load and its symbols are never registered. The developer default
native is a single element, so the defect cannot reproduce in a local build.

This PR:

  1. Quotes the variable at all three set_target_properties sites
    (src/backends/cuda/CMakeLists.txt, src/backends/cuda/components.cmake,
    apps/tests/backends/cuda/CMakeLists.txt).
  2. Adds a configure-time fast fail that reads CUDA_ARCHITECTURES back off the
    cuda target and aborts if it does not match UIPC_CUDA_ARCHITECTURES, so
    a regression is a configure error instead of a silently narrowed wheel.
  3. Adds 120-real to the release matrix for 50-series cards, mirrored in
    compatibility.json so scripts/check_release_policy.py stays green.
    89-virtual remains the lowest virtual entry on purpose: PTX only JITs
    upward, so it is what still covers sm_90 and sm_100.
  4. Normalizes UIPC_CUDA_ARCHITECTURES in place. The option's documentation
    offers a comma form (75,89), but only CMAKE_CUDA_ARCHITECTURES was
    normalized while the backend targets set their property straight from the
    option, so the comma survived configure and broke the first .cu compile
    with nvcc fatal : '89' is not in 'keyword=value' format. This also gives
    the new assertion a meaningful value to compare.

XMake needs no counterpart. Its architecture surface is a single-valued
add_cugencodes("sm_89") under github_actions and add_cugencodes("native")
otherwise, so it has neither the quoting defect nor a release matrix to mirror,
and the published wheel is built by scikit-build-core plus CMake.

Verification

  • compile_commands.json for the CUDA backend: all 199 CUDA translation units
    now carry the full list including code=[compute_89]. Before the fix only
    sm_75 was emitted.
  • Minimal end-to-end compile with the new six-entry list produces an archive
    whose cuobjdump -all --list-elf reports sm_75 sm_80 sm_86 sm_89 sm_120
    and whose cuobjdump -all -ptx reports .target sm_89, confirming CUDA 12.8
    accepts 120-real.
  • The new assertion was exercised both ways: it fires on the unquoted form and
    passes on the quoted one.
  • Real-project configure is clean and the assertion stays quiet for the full
    six-entry list, for native, and for the comma form. A minimal end-to-end
    build confirms 75,89 now compiles where it previously hit the nvcc fatal
    above.
  • scripts/check_release_policy.py passes; scripts/format_changed.py --check
    passes (no C++ touched).
  • Reference build with native on an RTX 5090 (sm_120) runs the
    hello_affine_body example and the Python 0_check_libuipc sample to
    completion.

Note for release planning: one more architecture increases wheel size and CI
compile time. sm_90 and sm_100 still have no native SASS and rely on the
89-virtual JIT path; extending the matrix further is a separate decision.

Audit commands, the cuobjdump --list-ptx caveat (it reports nothing even for
libraries that do embed PTX — use -ptx), and the quoting rule are recorded in
agent_docs/08-pitfalls-and-debugging.md.

Breaking Changes

None. The published architecture set only grows; pyproject.toml and
compatibility.json stay in sync, and no public API or behavior changes.

Checklist

Fast fail

  • Internal invariants checked with UIPC_ASSERT where feasible — n/a for
    CMake; the equivalent guard is the configure-time FATAL_ERROR added in
    src/backends/cuda/CMakeLists.txt
  • User-facing inputs validated with UIPC_ASSERT_THROW — n/a, no runtime
    input surface changed

C++ style

  • Conforms to .clang-format — n/a, no C++ changed
    (scripts/format_changed.py --check clean)
  • No raw pointer parameters / no const std::string& / no multiple
    inheritance — n/a
  • Naming conventions — n/a

GPU / CUDA (if touched)

  • GPU buffers passed as view types — n/a, no kernel or buffer code changed
  • Index guards / no NaN-Inf hazards — n/a, build configuration only

Constitution / material (if touched)

  • Not touched

Build / bindings (if touched)

  • New CMake options added to the root option block — none added; the
    existing UIPC_CUDA_ARCHITECTURES is only propagated correctly now
  • New public C++ API mirrored in the Python bindings — none added

Tests

  • C++ tests via uipc_add_test — n/a, the defect lives in the build
    configuration and cannot be observed from a test binary that is itself
    compiled with the truncated list
  • Python tests under python/tests/ — n/a, no binding surface changed
  • Bug fixes include a regression test — the configure-time assertion is the
    regression guard, verified to fire on the unquoted form; no existing test
    was modified

…atrix

`set_target_properties(... PROPERTIES ... CUDA_ARCHITECTURES
${UIPC_CUDA_ARCHITECTURES} ...)` did not quote the variable. An unquoted
multi-element list expands into separate arguments, which breaks the
PROPERTIES key/value pairing: the property kept only the first entry and
the remaining ones were silently absorbed as bogus property names
(`80-real` became a property whose value was `86-real`).

Every published wheel up to 0.0.27 therefore shipped `sm_75` SASS only and
lost the `89-virtual` PTX fallback, even though pyproject.toml and
compatibility.json declared 75/80/86/89-real plus 89-virtual. On consumer
Blackwell (`sm_120`) that surfaces as CUDA error 500 `named symbol not
found` at `world.init(scene)`: under CUDA 12.x lazy module loading a
missing device image appears as an unregistered symbol rather than error
209 `cudaErrorNoKernelImageForDevice`. The developer default `native` is a
single element, so the defect never reproduced locally.

Quote the variable at all three sites and add `120-real` to the release
matrix so 50-series cards get native SASS instead of relying on a JIT from
`89-virtual`. `89-virtual` stays as the lowest virtual entry because PTX
only JITs upward, and it is what still covers sm_90 and sm_100.

Verified through `arch=compute_*,code=*` in compile_commands.json: all 199
CUDA translation units now receive the full list including
`code=[compute_89]`, where before only `sm_75` was emitted.
`scripts/check_release_policy.py` stays green with compatibility.json
mirrored.

XMake needs no counterpart: its architecture surface is a single-valued
`add_cugencodes("sm_89")`/`("native")`, so it has neither the quoting
defect nor a release matrix, and it does not build the published wheel.
Two follow-ups to the quoting fix, both about the architecture list reaching
nvcc intact.

Add a configure-time fast fail: read `CUDA_ARCHITECTURES` back off the `cuda`
target and abort when it differs from `UIPC_CUDA_ARCHITECTURES`. The previous
defect passed configure, build and packaging, and only surfaced when the binary
met a GPU that was not in the (silently truncated) list, so a plain
`FATAL_ERROR` here is worth more than any downstream test: a test binary is
itself compiled with the same truncated list.

Normalize `UIPC_CUDA_ARCHITECTURES` in place as well. Its documentation offers
a comma form (`75,89`), but only `CMAKE_CUDA_ARCHITECTURES` was normalized,
while the backend targets set their property straight from the option. The
comma therefore survived configure and broke the first `.cu` compile with
`nvcc fatal : '89' is not in 'keyword=value' format`. Normalizing the cache
entry makes the documented form work and gives the new assertion a meaningful
value to compare.

Verified on the real project: configure succeeds and the assertion stays quiet
for the full `75-real;80-real;86-real;89-real;120-real;89-virtual` list, for
`native`, and for the comma form; all 199 CUDA translation units carry six
`arch=compute_*,code=*` entries. The assertion was also shown to fire on the
unquoted form, and a minimal end-to-end build confirms the comma form now
compiles where it previously failed.
@KemengHuang
KemengHuang merged commit 5b1ff1c into spiriMirror:main Sep 3, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants