Infrastructure to test DaCe's codegen (in)deterministic behavior

[kotsaloscv]

Adds infrastructure to detect non-determinism in gt4py's DaCe codegen. Two new parametrized nox sessions run gt4py's own DaCe test selection twice with isolated build caches, then byte-compare the generated sources between the two runs. The session is successful when the codegen is identical, fails on any content diff. Supports both gt4py.next and gt4py.cartesian caches, covering the dace_cpu, dace_gpu (CUDA), and HIP backends. Driven entirely from gt4py's own nox sessions — no external repo needed.

Layout

• noxfile.py — two new sessions, test_cartesian_dace_determinism and test_next_dace_determinism. Each calls a shared _run_dace_determinism_check helper that runs gt4py's regular DaCe pytest selection twice with the appropriate cache-isolation env vars per layout, then invokes the comparator. Tagged dace + determinism, so nox -t determinism runs every variant.

• scripts/dace_deterministic_codegen.py — pure-stdlib comparison library + thin argparse CLI with explicit --layout {next,cartesian} dispatch. Each layout has its own snapshot walker:

  • next — snapshots flat <root>/<program>_<sha256>/src/{cpu,cuda}/... (HIP under src/cuda/hip/ covered transparently).
  • cartesian — snapshots the deeply nested <root>/py<pyver>_<cachever>/<backend>/<test.module.path>/<Class>_<backend>_<id>/ structure, comparing m_*.py plus bindings.{cpp,cu} and computation.hpp inside each m_*_pyext_BUILD/.

• ci/cscs-ci-dace-determinism.yml — self-contained CSCS-CI pipeline entry-point (see CI section).

Per-layout env vars

gt4py.next and gt4py.cartesian have completely separate caching subsystems with different knobs. The helper sets them per layout automatically:

• next — GT4PY_BUILD_CACHE_DIR=<run_dir>; cache lands at <run_dir>/.gt4py_cache/.
• cartesian — GT_CACHE_ROOT=<run_dir> plus GT_CACHE_PYTEST_DIR=<run_dir>/gt_cache, and --keep-gtcache passed to pytest. cartesian's conftest unconditionally rmtrees its cache at pytest_sessionfinish unless that CLI flag is present (independent of the env vars), so both knobs are required.

DaCe build folder mode

The check requires DACE_compiler_build_folder_mode=development. gt4py configures DaCe to production by default, which strips the generated source files after compilation — leaving nothing to compare. The nox sessions set this env var automatically for both layouts; the comparator raises NoSourceFilesObservedError with an env-var hint if it ever encounters caches built without it.

CI

Standalone CSCS-CI pipeline, independent from ci/cscs-ci.yml. Pipeline registered separately as dace-determinism and triggered by whitelisted users posting cscs-ci run dace-determinism in a PR comment.

allow_failure: true while the determinism work stabilizes — to be dropped once main stays green. Per-cell artifacts (_dace_deterministic_codegen/ containing diffs/ and report.txt after the session's cleanup hook removes the raw caches) retained for 1 month.

Local use

./noxfile.py -s 'test_next_dace_determinism-3.10(cpu, nomesh)'    # one next variant
./noxfile.py -s 'test_cartesian_dace_determinism-3.10(cpu)'       # one cartesian variant
nox -t determinism                                                # all variants
python scripts/dace_deterministic_codegen.py --run1 PATH --run2 PATH --layout {next,cartesian}

[kotsaloscv]

cscs-ci run dace-determinism

[kotsaloscv]

cscs-ci run dace-determinism

[kotsaloscv]

cscs-ci run dace-determinism

[kotsaloscv]

cscs-ci run dace-determinism

[kotsaloscv]

cscs-ci run dace-determinism

[kotsaloscv]

cscs-ci run dace-determinism

[kotsaloscv]

cscs-ci run dace-determinism

[kotsaloscv]

cscs-ci run dace-determinism

[kotsaloscv]

cscs-ci run dace-determinism

[kotsaloscv]

cscs-ci run dace-determinism