OpenACC kernel names truncated for long source paths, causing duplicate device functions (NVVM_ERROR_COMPILATION)

[joshkamm]

Summary

With the cuBLAS/cuSolver header issue (#98) out of the picture, the GPU build still fails on the Perlmutter login node — integrals.cpp dies during OpenACC device code generation:

nvvmCompileProgram: failed with error 9: NVVM_ERROR_COMPILATION
nvnvvmd: error: /tmp/nvacc....gpu (39808, 24): parse invalid redefinition of function
  '_124_..._src_integrals_integrals_cpp___nv__Z18compute_4c_ol_onediiPiPfiiPdS1_RSt6vectorIS2_IdSaIdEESaIS4_EEiiiiiiiiiiS1_S1_PS1_S8_S8_S8_S1_'

The redefined symbol demangles to compute_4c_ol_oned(...) (integrals.cpp:7156) — a long-standing function (June 2024, 45c984f), not recent code.

Key facts

• Deterministic and version-independent on the pixi side: reproduced identically with pixi 0.60.0 / 0.66.0 / 0.70.2 and pixi-build-cmake backends 0.3.6 / 0.3.8 / 0.3.14 (cold caches each time). This is not a packaging/toolchain-plumbing issue.
• CI is green on NVHPC 25.1 (self-hosted GPU runner), while the failure occurs with NVHPC 25.5 on the Perlmutter login node — suggesting an nvc++ 25.5 regression or a target-architecture interaction.
• The login node has no visible GPU, so -acc=gpu compiles for a default compute-capability set rather than a detected device — a possible trigger for duplicate device-function emission across targets.
• Athena also runs NVHPC 25.5, but Srijita's git-main build succeeded there after a clean rebuild (Build reliability: slatergpu git-main builds fail (path-dep workaround); confirmed facet = find_package(MPI REQUIRED) #95), which argues against a pure "25.5 always fails" story and for the no-GPU-visible/default-target hypothesis.
• This failure was previously masked in most environments: with backends <=0.3.11 the build inherited the shell env, and depending on the machine it either failed earlier (GPU build fails: nvc++ can't find cublas_v2.h / cusolverDn.h (NVHPC env not forwarded to pixi build sandbox) #98's cublas facet under sanitized env) or here. It is plausibly Bishnu's uncaptured Perlmutter failure from Build reliability: slatergpu git-main builds fail (path-dep workaround); confirmed facet = find_package(MPI REQUIRED) #95.

Environment

• Perlmutter login node (no GPU visible); NVHPC 25.5 (nvc++ 25.5-0), CUDA math_libs 12.9
• SlaterGPU git-main, USE_ACC=ON, pixi source build (pixi-build-cmake)

Suggested diagnostics

1. Compile integrals.cpp alone with an explicit target, e.g. -gpu=cc80, on the login node — if it passes, the default-target-set hypothesis is confirmed and a pinned -gpu= flag (or building on a GPU node) is the answer.
2. Try the nvhpc/24.11 module available on Perlmutter with the same TU.
3. Compare against a compute node / athena (GPU visible, 25.5) to separate compiler version from device visibility.

Related

• Build reliability: slatergpu git-main builds fail (path-dep workaround); confirmed facet = find_package(MPI REQUIRED) #95 — umbrella "git-main builds fail"; this is likely its Perlmutter facet (Bishnu's error was never captured)
• GPU build fails: nvc++ can't find cublas_v2.h / cusolverDn.h (NVHPC env not forwarded to pixi build sandbox) #98 — the cuBLAS/cuSolver facet; fixing it exposes this failure as the next blocker on affected machines

[joshkamm]

Root cause found — path-length symbol truncation, NOT compiler-version/machine specific

The original framing (NVHPC 25.5, Perlmutter login node, device visibility) was wrong. Retitled accordingly.

Mechanism

NVHPC's OpenACC codegen names each generated device kernel:

_<N><mangled full source path>___nv_<mangled C++ signature>_F1L<line>_<counter>_gpu

When that composite exceeds ~231 characters it is truncated from the right — destroying the _F1L<line>_<counter>_gpu uniquifier. Multiple kernels then share one name and the NVVM stage rejects its own IR: parse invalid redefinition of function.

Evidence (from kept intermediates, -v -gpu=keep)

• Failing IR: one 231-char symbol (truncated mid-signature, no F1L suffix) defined 5× — the five OpenACC regions of compute_4c_ol_oned around integrals.cpp:7198–7283.
• Passing IR (same source, same flags, relative -c integrals.cpp → short _13integrals_cpp prefix): identical kernels keep their suffixes (_F1L7198_*, _F1L7283_*) — no collision.
• Decisive A/B, same flags and file, absolute paths only differing in length: 124-char path → fails; 79-char path (standard checkout location) → compiles clean.

Why it was misattributed

The failure only appeared inside pixi builds because CMake/ninja pass the absolute source path, and the investigation checkout lived under a deep worktree directory (.../SlaterGPU/.claude/worktrees/slater-gpu-issue-98-a65411/...). Manual repro attempts used relative paths, so they always passed — which mimicked a "build-system-only" failure. Compiler version and GPU visibility were red herrings (25.1 vs 25.5 untested for the truncation cap, but irrelevant to the group's builds).

Impact — nobody's normal workflow is affected

Standard checkout paths (athena home dirs, Perlmutter project dirs, CI's /home/runner/work/...) stay well under the threshold. At risk: checkouts whose source-file paths approach ~110+ characters combined with functions whose mangled signatures are long (e.g. nested std::vector parameters). This is probably not Bishnu's #95 facet unless his checkout was unusually deep.

Suggested actions

1. Report upstream to NVIDIA: truncation should preserve/hash the uniquifier instead of dropping it.
2. Local guidance: keep GPU-build checkouts out of very deep directories (this bit a .claude/worktrees/... path).
3. No SlaterGPU code change warranted.