Skip to content

[CI] Drop shadow-hand kitless rendering test from arm-ci to stop intermittent hangs#6581

Merged
kellyguo11 merged 1 commit into
isaac-sim:developfrom
hujc7:jichuanh/disable-shadow-hand-kitless-arm
Jul 22, 2026
Merged

[CI] Drop shadow-hand kitless rendering test from arm-ci to stop intermittent hangs#6581
kellyguo11 merged 1 commit into
isaac-sim:developfrom
hujc7:jichuanh/disable-shadow-hand-kitless-arm

Conversation

@hujc7

@hujc7 hujc7 commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

Description

1. Summary

Removes the arm_ci marker from test_rendering_shadow_hand_kitless.py. Since the test landed on arm-ci (2026-07-10, #5698) it has been implicated in 15 of 81 concluded arm-ci runs (~19%) and 15 of that job's 18 failures — including 3 failures on develop itself — typically stalling mid-file after several passing render cases until the per-file timeout kills the job. x86 coverage is unchanged: the file still runs in rendering-correctness-kitless with its full backend matrix.

2. Failure data (all concluded runs since 2026-07-10)

Job Concluded Failures Implicating this test
arm-ci (blocking on PRs) 81 18 15 (3 on develop)
rendering-correctness-kitless x86 (continue-on-error on PRs) 85 32 10 (3 timeouts, 7 file failures)

The instability is not arm-exclusive — x86 hits the same file at roughly half the rate — but the x86 job is advisory on PRs while arm-ci is blocking, making this test the dominant source of spurious red arm-ci checks. test_rendering_dexsuite_kuka_homo_kitless.py shows the same signature on arm-ci at a much lower rate (2 of 18 failures) and stays enabled for now.

3. Test plan

  • Pre-commit clean (./isaaclab.sh -f)
  • arm-ci green on this PR with the file no longer collected

4. Out of scope

Root-causing the renderer stall (all observed hangs sit in the ovrtx segment of the parametrized cases, between per-case env create/teardown cycles); the two remaining arm_ci-marked rendering files (cartpole, kuka-homo) stay enabled.

The test intermittently stalls the arm64 runner mid-file: several
parametrized render cases pass, then the renderer hangs and the file
is killed at the per-file timeout, failing the arm-ci job. The same
signature hit unrelated branches within the same hour while develop
passed, so the instability is platform-side, not change-related.

Keep the test running in the x86 rendering-correctness-kitless job,
which retains full backend coverage including ovphysx parameters that
arm-ci already excluded.
@github-actions github-actions Bot added the isaac-lab Related to Isaac Lab team label Jul 17, 2026
@hujc7
hujc7 marked this pull request as ready for review July 21, 2026 16:36
@hujc7
hujc7 requested a review from a team July 21, 2026 16:36
@greptile-apps

greptile-apps Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR removes the pytest.mark.arm_ci marker from test_rendering_shadow_hand_kitless.py to stop it from blocking arm64 CI runs, where it was responsible for ~83% of all arm-ci failures since it was added. x86 coverage is unaffected — the test continues to run in the rendering-correctness-kitless job.

  • Test marker removed: pytestmark drops pytest.mark.arm_ci; an inline comment documents the renderer stall root cause.
  • Changelog fragment added: disable-shadow-hand-kitless-arm.skip follows the existing convention for test-only/CI-only entries (.skip extension matches several other fragments in changelog.d/).

Confidence Score: 5/5

Safe to merge — a one-line marker removal with no impact on test logic or x86 coverage.

The change removes a single pytest marker and adds a comment plus a changelog fragment. No test logic, no application code, and no x86 coverage is affected. The .skip changelog extension matches the existing convention for test-only entries.

No files require special attention.

Important Files Changed

Filename Overview
source/isaaclab_tasks/test/core/test_rendering_shadow_hand_kitless.py Removes pytest.mark.arm_ci from pytestmark; adds inline comment explaining the arm64 renderer stall issue. No logic changes.
source/isaaclab_tasks/changelog.d/disable-shadow-hand-kitless-arm.skip New changelog fragment with .skip extension, consistent with other test-only/CI-only entries in the same directory.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[test_rendering_shadow_hand_kitless.py] -->|Before PR| B["pytestmark = [isaacsim_ci, arm_ci]"]
    A -->|After PR| C["pytestmark = [isaacsim_ci]"]
    B --> D[Collected by arm-ci job]
    B --> E[Collected by rendering-correctness-kitless x86]
    C --> F[NOT collected by arm-ci job]
    C --> G[Collected by rendering-correctness-kitless x86]
    D -->|~19% of runs| H[Intermittent hang / timeout]
    G --> I[Full backend matrix coverage retained]
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[test_rendering_shadow_hand_kitless.py] -->|Before PR| B["pytestmark = [isaacsim_ci, arm_ci]"]
    A -->|After PR| C["pytestmark = [isaacsim_ci]"]
    B --> D[Collected by arm-ci job]
    B --> E[Collected by rendering-correctness-kitless x86]
    C --> F[NOT collected by arm-ci job]
    C --> G[Collected by rendering-correctness-kitless x86]
    D -->|~19% of runs| H[Intermittent hang / timeout]
    G --> I[Full backend matrix coverage retained]
Loading

Reviews (1): Last reviewed commit: "Remove arm_ci marker from shadow-hand ki..." | Re-trigger Greptile

@kellyguo11
kellyguo11 merged commit d8e1523 into isaac-sim:develop Jul 22, 2026
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants