[CI] Drop shadow-hand kitless rendering test from arm-ci to stop intermittent hangs#6581
Merged
kellyguo11 merged 1 commit intoJul 22, 2026
Conversation
The test intermittently stalls the arm64 runner mid-file: several parametrized render cases pass, then the renderer hangs and the file is killed at the per-file timeout, failing the arm-ci job. The same signature hit unrelated branches within the same hour while develop passed, so the instability is platform-side, not change-related. Keep the test running in the x86 rendering-correctness-kitless job, which retains full backend coverage including ovphysx parameters that arm-ci already excluded.
mataylor-nvidia
approved these changes
Jul 20, 2026
hujc7
marked this pull request as ready for review
July 21, 2026 16:36
hujc7
requested review from
AntoineRichard,
StafaH,
kellyguo11 and
ooctipus
as code owners
July 21, 2026 16:36
Contributor
kellyguo11
approved these changes
Jul 22, 2026
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
1. Summary
Removes the
arm_cimarker fromtest_rendering_shadow_hand_kitless.py. Since the test landed on arm-ci (2026-07-10, #5698) it has been implicated in 15 of 81 concluded arm-ci runs (~19%) and 15 of that job's 18 failures — including 3 failures ondevelopitself — typically stalling mid-file after several passing render cases until the per-file timeout kills the job. x86 coverage is unchanged: the file still runs inrendering-correctness-kitlesswith its full backend matrix.2. Failure data (all concluded runs since 2026-07-10)
arm-ci(blocking on PRs)develop)rendering-correctness-kitlessx86 (continue-on-error on PRs)The instability is not arm-exclusive — x86 hits the same file at roughly half the rate — but the x86 job is advisory on PRs while
arm-ciis blocking, making this test the dominant source of spurious red arm-ci checks.test_rendering_dexsuite_kuka_homo_kitless.pyshows the same signature on arm-ci at a much lower rate (2 of 18 failures) and stays enabled for now.3. Test plan
./isaaclab.sh -f)4. Out of scope
Root-causing the renderer stall (all observed hangs sit in the ovrtx segment of the parametrized cases, between per-case env create/teardown cycles); the two remaining
arm_ci-marked rendering files (cartpole, kuka-homo) stay enabled.