[CI] Cross-platform — Part 2: ARM/Spark workflow#5698
Conversation
There was a problem hiding this comment.
Update (df786d2): Reviewed incremental diff since 5cbba72 (1 file changed, 1 commit).
New commits reviewed
| Commit | PR | Description |
|---|---|---|
| feba370 | #5826 | Defer pxr loading in isaaclab.sim, .assets, .scene, .sim/utils |
| 207f319 | #5844 | Log Kit version info on AppLauncher startup |
| c54e512 | — | Scope test tags to arm_ci only; misc review fixes |
| ea8ecd6 | #5891 | Split tasks into core/contrib |
| acb09f2 | — | Merge remote-tracking branch origin/develop |
| 5cbba72 | — | Move cartpole smoke test into test/core/ after tasks split |
| df786d2 | — | Raise cartpole perception smoke-test timeout for cold shader cache |
Latest change (df786d2)
Raises test_train_cartpole_perception timeout from default (presumably matching state test) to 1800 s to accommodate first-run shader compilation on cold GPU cache. Adds a comment explaining the reasoning. Minimal, sensible change — shader compilation on first camera-enabled run can easily exceed 600 s on some hardware.
Full analysis (unchanged from prior update)
1. Deferred pxr imports (#5826) ✅ — All 9 modules now defer from pxr import ... into function bodies. Type hints stay under TYPE_CHECKING. Fixes Kit startup crash when env-cfg parse pulls pxr before AppLauncher.
2. Kit version logging (#5844) ✅ — Simple _log_kit_version_info() added to AppLauncher. Prints Kit version, kernel version, and git hash to sys.__stderr__.
3. Tasks core/contrib restructuring (#5891) ✅ — Massive but mechanical: direct.* / manager_based.* → flat core.* / contrib.*. Gym IDs unchanged. Docs updated. Changelog fragment present.
4. CI/test adjustments ✅ — Test markers narrowed, cartpole smoke test moved, perception timeout raised for cold cache.
Previous concern status
| Issue | Status |
|---|---|
P1 (conftest.py OSError handling) |
✅ Resolved |
| P2 (TAB delimiter) | ✅ Resolved |
No new issues found. LGTM from bot. 🤖
Update (e6184ab): Branch synced with develop (merge commit, 114 upstream commits incorporated — no PR-specific code changes). Previous LGTM still stands. ✅
Foundation for cross-platform CI. Registers four pytest markers (windows, windows_ci, arm, arm_ci), teaches AppLauncher to recognize them in argv so they do not leak into Isaac Sim's argparse, and moves the AssetConverterBase USD scratch directory from a hardcoded /tmp/IsaacLab to tempfile.gettempdir() for cross-platform compatibility. Tags source/isaaclab/test/deps/test_torch.py and test_scipy.py with the new markers so they are selectable by future cross-platform jobs. Workflow files (arm-ci.yaml, windows-ci.yaml) ship in follow-up PRs.
00966f4 to
ca8fa07
Compare
4b8ecb8 to
3e904c3
Compare
Mirrors build.yaml's spirit but stays minimal for the aarch64 path: Tier 1 (gates none — continue-on-error): general-arm, install-arm, kit-launch-arm Tier 2 (meaningful, marker-filtered): kitless-arm, determinism-arm Every job sets continue-on-error: true while the aarch64 runner setup stabilizes. Every pytest invocation passes --timeout=N --timeout-method=signal so a single hung test cannot consume the whole job slot. Inline scripts use set -e to fail on the first nonzero return. Tags three test_rendering_*_kitless.py files plus test_differential_ik.py and test_operational_space.py with the arm_ci marker so the Tier 2 jobs can select them via pytest -m arm_ci.
3e904c3 to
98a5b68
Compare
b7862d0 to
48819bf
Compare
- source/isaaclab/setup.py: gate pytetwild==0.2.3 with `platform_machine != 'aarch64'`. No aarch64 wheel on PyPI; source build fails because the transitive `geogram` dep hardcodes `-m64` in its CMakeLists. The single call site at sim/schemas/schemas.py already lazy-imports it with a clear "install manually" message, so aarch64 users keep everything except automatic volume-deformable tetrahedralization. - docker/Dockerfile.arm-ci: new lightweight Dockerfile that layers cmake / build-essential / git + EULA env vars + a bash entrypoint onto the multi-arch Isaac Sim base image. Replaces the previous inline `docker run + apt-get install + docker commit` chain in arm-ci.yaml. - docker/Dockerfile.base: reverted the libgmp / libmpfr / libeigen3 / libcgal / libboost additions from the arm64-conditional apt block — those were only needed by pytetwild's fTetWild build, which we no longer install on aarch64. - .github/workflows/arm-ci.yaml: build via docker/Dockerfile.arm-ci instead of the inline apt-install-and-commit pattern. Test steps no longer need to re-specify --entrypoint or EULA env vars on every docker run.
48819bf to
b3d7f7c
Compare
Forces run_docker_tests=false in build.yaml's changes job so all heavy test jobs skip via their existing if-gate. Saves CI runner time + cost during ARM CI iteration on PR isaac-sim#5698. Must be reverted before final review.
The develop merge brought in isaac-sim#5891 (split tasks tests into core/contrib), which relocated every test file under test/core/ or test/contrib/ and left only shared helpers + conftest at the test-suite root. The new test_cartpole_training_smoke.py predated that split and was the lone test_*.py still at top-level. Cartpole is a core task, so move it under test/core/ to match the new layout, and bump its _REPO_ROOT path depth from parents[3] to parents[4] (matching the sibling core/ tests).
test_train_cartpole_perception timed out at the default 600s on a cold-cache Spark runner: the first camera-enabled run compiles shaders (~600s) before training even starts, which alone consumes the budget. It passed earlier only because that runner had a warm shader cache. Give the camera case a 1800s timeout (the state case keeps 600s).
Should be fixed. |
AntoineRichard
left a comment
There was a problem hiding this comment.
I'm not sure about the git action things
There was a problem hiding this comment.
Why not run this similarly to the other training tests and just add a windows/arm test list like we have the short / long list?
There was a problem hiding this comment.
The goal is to delegate config to per test/file level, while keep workflow more stable if I understand the question correctly here
…park-ci-cartpole # Conflicts: # pyproject.toml # tools/conftest.py
The develop merge removed the versioned Isaac-Cartpole-Direct-v0 and Isaac-Cartpole-Camera-Direct-v0 gym registrations in favor of the unversioned Isaac-Cartpole-Direct and Isaac-Cartpole-Camera-Direct ids. The smoke test still requested the -v0 names, so every run failed at gym registration with DeprecatedEnv before training started, breaking both the arm-ci and isaaclab_tasks jobs. Point the test at the current ids and correct the docstring: the camera variant now also registers an rsl_rl_cfg_entry_point, so the old rationale for choosing rl_games no longer held.
…base) (#5695) ## Summary Base PR for cross-platform CI on Windows and NVIDIA DGX Spark (ARM/aarch64 Linux). No user-facing API change. - Registers `windows_ci` and `arm_ci` pytest markers in pyproject.toml. - AppLauncher argv strip recognizes the new markers so `pytest -m windows_ci` (etc.) does not leak into Isaac Sim's argparse. - AssetConverterBase USD scratch dir: hardcoded `/tmp/IsaacLab` → `tempfile.gettempdir()` for cross-platform compatibility. This base PR ships only the marker registration and the shared scaffolding. Tagging specific suites with these markers, and the CI workflows that consume them, ship in the follow-up parts below — that keeps the marker/scaffolding change reviewable on its own and lets the consumer PRs rebase cleanly once it lands. ## Prior art and credits This work is a derivative of and builds directly on two prior PRs by other contributors: - #3900 by Kelly Guo (kellyguo11), against `main`, 2025-10-31 — "test windows CI". Designed and introduced the cross-platform marker scheme (`windows`, `windows_ci`, `arm`, `arm_ci`), tagged ~50 existing tests across the codebase, added the AppLauncher argv-strip pattern so the new pytest `-m` filters do not leak into Isaac Sim's argument parser, and pioneered the Windows CI job wiring. The marker design and AppLauncher behavior in this PR are taken directly from #3900. - #3845 by Alexander Poddubny (nv-apoddubny), against `main`, 2025-10-27 — "Adds basic CI verification on Windows". Built the original Windows self-hosted runner infrastructure: `isaaclab.bat` entry script, the initial `.github/workflows/build.yml` Windows job, and the self-hosted-runner labeling conventions this PR's downstream Part 2/Part 3 workflows still rely on. Both PRs target `main` and have not been merged. This PR rebases that work onto `develop` (where the active development happens — `main` is behind by hundreds of commits), narrows the base PR to just the marker scheme plus shared scaffolding (so the marker/runner plumbing can be validated end-to-end before tests are tagged and re-expanded), and adds an `arm_ci` marker variant + ARM/Spark runner support that #3900/#3845 did not include. Credit for the architecture goes to the authors above; this PR is the rebase + minimization step. ## Series PRs prefixed with `[CI] Cross-platform —` all contribute to bringing Isaac Lab CI up on Windows and NVIDIA DGX Spark runners. This PR is the base everything else builds on. Current siblings: - Part 1 (this PR, base) — Markers and shared scaffolding - Part 2 — #5698 — ARM/Spark workflow (`.github/workflows/arm-ci.yaml`) - Part 3 — #5700 — Windows workflow (`.github/workflows/windows-ci.yaml`) Siblings depend on this PR's marker scheme; the same scaffolding lines are carried in their branches and dedupe on rebase after this lands. Additional follow-up parts can be added later without renumbering existing ones. ## Test plan - [x] `pytest --markers` lists `windows_ci` and `arm_ci` (registered, no `PytestUnknownMarkWarning`). - [x] AppLauncher argv strip drops `-m windows_ci` / `-m arm_ci` before Isaac Sim's argparse runs. - [x] `./isaaclab.sh -f` (pre-commit) passes. - [x] Merged latest `develop`: the markers list is the union with the newly added `device_split` marker, and `--capture=no` argv stripping from develop is preserved.
| try: | ||
| import isaacsim as _isaacsim_for_paths | ||
| except ImportError as e: | ||
| raise RuntimeError( |
There was a problem hiding this comment.
is this path only triggered when we are launching with isaac sim? just want to make sure we don't throw this error if we try to run a kitless path
There was a problem hiding this comment.
I believe that's determined in simulation context related utilities. In kitless path, applauncher should be skipped.
| @@ -0,0 +1,80 @@ | |||
| # Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md). | |||
There was a problem hiding this comment.
is this one similar to some of the cartpole tests we have under source/isaaclab/test/install_ci/cli/test_install_rl_uv_env.py? if we could consolidate some of the tests, that'd help keep files more organized.
There was a problem hiding this comment.
There's a limitation that install tests needs to be in the same folder. the current approach is to keep the test in the install_ci folder, but keep it in separate file such that other CI can use it.
|
|
||
| print(f"ovrtx version: {ovrtx.__version__}") | ||
| except ImportError as exc: | ||
| if platform.machine() == "aarch64": |
There was a problem hiding this comment.
ovrtx and ovphysx seem to provide aarch64 wheels - https://pypi.nvidia.com/ovrtx/ https://pypi.nvidia.com/ovphysx/
Share the kitless Cartpole state, rendering, reward, and camera training probes across installation and ARM CI. Install the published OV runtimes on ARM and remove architecture skips so missing dependencies fail visibly.
Use the config job's image tag for ARM builds and isolate local dependency caches by target platform. Clean up merge artifacts and align the artifact action with current develop.
Use RSL-RL for both shared training probes so every validated install\nscope has the required framework. Install Git LFS in the base image so\nRSL-RL can record repository state from mounted checkouts.
…park-ci-cartpole # Conflicts: # .github/actions/run-tests/action.yml
Refresh the ARM CI branch with current upstream changes and preserve both the per-file pytest filter and CI marker inputs in the shared Docker test action.
The arm-ci job ran the ovphysx-backed rendering test params that every x86 kitless configuration deselects via TEST_K_EXPR. These params require ovphysx >= 0.5.1, which has no aarch64 wheel, and running the 13 extra ovphysx-ovrtx combos in test_rendering_cartpole_kitless.py pushed the process past the ovrtx per-process SyncScopeId limit (~15 renderer creations), crashing HydraEngine initialization on the 16th env and failing the job with a missing test report. Mirror the public pip-index fallback of rendering-correctness-kitless by passing the same k-expression to the arm_ci marker run.
| echo "🔵 Local deps tag: ${LOCAL_DEPS_TAG}" | ||
| echo "LOCAL_DEPS_TAG=${LOCAL_DEPS_TAG}" >> "$GITHUB_ENV" | ||
|
|
||
| if docker image inspect "${LOCAL_DEPS_TAG}" >/dev/null 2>&1; then |
There was a problem hiding this comment.
does this guarantee that it will always use the latest isaac lab code? the agent picked up a concerning behavior that I'm not too sure if it's true or not
Commit A: build dependencies + source A → cached image
Commit B: source changes, dependencies do not
Current behavior:
cached image containing source A → renamed as Commit B image
There was a problem hiding this comment.
image is deduped w.r.t dependency hash. source code is always mounted.
| needs: [changes, config] | ||
| if: needs.changes.outputs.run_docker_tests == 'true' | ||
| timeout-minutes: 60 | ||
| continue-on-error: true |
There was a problem hiding this comment.
would this prevent us from getting an accurate signal on whether the tests passed or failed?
There was a problem hiding this comment.
It just means that if the job failed, merge is not blocked. If we are confident, this can be made required test.
Reconcile tools/conftest.py with isaac-sim#5698 (ARM/Spark): adopt the generalized ctx.ci_marker / run_individual_tests(ci_marker) rename while keeping the multi-GPU inject_shard_select plugin hook.
…rmittent hangs (#6581) # Description ## 1. Summary Removes the `arm_ci` marker from `test_rendering_shadow_hand_kitless.py`. Since the test landed on arm-ci (2026-07-10, #5698) it has been implicated in 15 of 81 concluded arm-ci runs (~19%) and 15 of that job's 18 failures — including 3 failures on `develop` itself — typically stalling mid-file after several passing render cases until the per-file timeout kills the job. x86 coverage is unchanged: the file still runs in `rendering-correctness-kitless` with its full backend matrix. ## 2. Failure data (all concluded runs since 2026-07-10) | Job | Concluded | Failures | Implicating this test | |---|---|---|---| | `arm-ci` (blocking on PRs) | 81 | 18 | 15 (3 on `develop`) | | `rendering-correctness-kitless` x86 (continue-on-error on PRs) | 85 | 32 | 10 (3 timeouts, 7 file failures) | The instability is not arm-exclusive — x86 hits the same file at roughly half the rate — but the x86 job is advisory on PRs while `arm-ci` is blocking, making this test the dominant source of spurious red arm-ci checks. `test_rendering_dexsuite_kuka_homo_kitless.py` shows the same signature on arm-ci at a much lower rate (2 of 18 failures) and stays enabled for now. ## 3. Test plan - [x] Pre-commit clean (`./isaaclab.sh -f`) - [x] arm-ci green on this PR with the file no longer collected ## 4. Out of scope Root-causing the renderer stall (all observed hangs sit in the ovrtx segment of the parametrized cases, between per-case env create/teardown cycles); the two remaining `arm_ci`-marked rendering files (cartpole, kuka-homo) stay enabled.
Summary
arm-cijob to the existingDocker + Testsworkflow for NVIDIA DGX Spark self-hosted runners. It builds and tests on the same ARM runner because ECR is not wired for arm64.CI_MARKER, plus a shared kitless Cartpole state/camera smoke used by ARM and installation CI.1. ARM/Spark CI
1.1 The
arm-cijob shares the existingchangesandconfiggating. It buildsDockerfile.baseforlinux/arm64, runs thearm_ci-selected tests, and then runs the shared Cartpole smoke.1.2 OVRTX and OVPhysX are installed explicitly for the rendering probes. Missing ARM runtime dependencies fail instead of being hidden by architecture-specific skips.
1.3 The job uses the same workflow and shared actions as the amd64 jobs. ARM enables 14-day stale dependency-cache eviction because its runner has persistent local Docker storage and no ECR handoff.
1.4 The
arm_cimarker run passestest-k-expr: not ovphysx, the same per-file-kexpression the x86rendering-correctness-kitlessjob applies on public pip-index runs. The ovphysx-backed rendering params require OVPhysX >= 0.5.1, which has no aarch64 wheel.2. Shared CI utilities
2.1
_lib/compute-deps-hashcomputes one dependency-cache identity from install inputs, the resolved base-image digest, and target platform. Bothdocker-buildandecr-build-push-pullconsume it.2.2
_lib/setup-docker-configcentralizes the credential-helper workaround and optional NGC login. It is idempotent when the ECR action delegates todocker-build.2.3
run-testsforwards both upstream's per-filetest-k-exprand this PR'sci-markerinputs into the test container.3. Shared Cartpole smoke
3.1
source/isaaclab/test/install_ci/misc/cartpole_training_smoke.pyis the common prepared-environment probe for state Cartpole training, camera observation/reward validity, and camera Cartpole training.3.2 CLI and wheel installation tests execute the same probe inside their newly installed environments; ARM executes it inside its prepared image.
3.3 The base image installs
git-lfsso RSL-RL can snapshot repository state from a mounted LFS checkout.4. Cross-platform support
4.1
AppLauncherderivesEXP_PATHfrom the installedisaacsimpackage when the variable is missing, while leaving existing configured environments unchanged.4.2
tools/conftest.pysupportsCI_MARKERas a file-selection and pytest marker filter parallel toISAACSIM_CI_SHORT, exact node-ID selection, and per-file-kfiltering.Known problems
arm_ciat module scope, so ARM collects all 32 cases per module. The ovphysx-backed cases are now deselected viatest-k-expr: not ovphysx(section 1.4), leaving 15 Newton cases running per module. Without the filter, the extra ovphysx-ovrtx cases intest_rendering_cartpole_kitless.pypushed a single process past the OVRTX per-process SyncScopeId limit (~15 renderer creations), crashing HydraEngine initialization on the 16th env; this also reproduces on x86, so the failure is not ARM-specific. Narrowing ARM to the two intended representative tests is still pending.developpins OVPhysX0.5.2+head.f62c22207c, while this job installs the publicly available ARM0.4.13wheel. With thenot ovphysxfilter the ARM rendering selection is Newton-only, matching the x86 public pip-index fallback; revisit if an aarch64 OVPhysX >= 0.5.1 wheel or wheelhouse becomes available.5. Series
Part 2 of
[CI] Cross-platform —. Depends on #5695 (Part 1). The Windows workflow is #5700 (Part 3).Test plan
develop(5f2db9902baa724427b7aa654835aa559fce64e8)../isaaclab.sh -fbefore commit and before push.test-k-expr: not ovphysx), mirroring the x86 kitless public pip-index fallback.Docker + Testsworkflow is green.daily-compatibilitybefore merge.