Skip to content

test: stabilize benchmark timing contract - #869

Merged
TATP-233 merged 2 commits into
feat/issue-705-manager-mjwarpfrom
fix/issue-855-benchmark-timing-contract
Aug 2, 2026
Merged

test: stabilize benchmark timing contract#869
TATP-233 merged 2 commits into
feat/issue-705-manager-mjwarpfrom
fix/issue-855-benchmark-timing-contract

Conversation

@TATP-233

@TATP-233 TATP-233 commented Aug 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • replace a flaky ordering assertion between two independently sampled wall-clock timings in the G1 motion-tracking Numba benchmark test
  • assert the actual record contract: total time is the full update-state measurement, the NumPy/Numba total fields match their update-state fields, and both reward/termination samples remain positive
  • refresh and promote Phase 7 evidence because the final gate binds the complete tracked source tree

Part of #705
Addresses #855

Real CI Root Cause

GitHub Actions run 30738994557 passed Ruff, mypy, pyright, benchmark smoke, and shards A/B/C/D/F/H. Shard E failed at:

tests/benchmark/test_g1_motion_tracking_numba_benchmark.py::test_g1_motion_tracking_numba_benchmark_builds_records_and_matches_numpy

The test compared numpy_total_ms >= numpy_reward_termination_ms, but those values are produced by separate time_call invocations. On the shared runner, the one-sample reward/termination timing was 1.647 ms while the independently measured full update-state timing was 1.107 ms. That ordering is not a stable wall-clock contract and does not indicate incorrect benchmark output.

The implementation defines numpy_total_ms = numpy_update_state_mean and numba_total_ms = numba_update_state_mean. The revised test checks those exact definitions and keeps parity, positive timings, record identity, thread, and speedup assertions intact.

Validation

  • CI-equivalent shard E: 119 passed, 1 skipped, 5 deselected
  • focused regression: 1 passed
  • LD_LIBRARY_PATH=/tmp/unilab-nvml-570.133.07/usr/lib/x86_64-linux-gnu make test-all: 2350 passed, 26 skipped, 449 deselected, 1 xfailed; coverage 68%; Ruff, mypy, pyright, and benchmark smoke passed
  • benchmark smoke: module mode 36/37 and script mode 37/38; only optional MLX was skipped
  • Phase 0-7 standalone gates, baseline, thresholds, DR inventory, claim inventory, backend isolation, support, task rollout, legacy retirement, workflow trigger, and CI shard audits passed
  • refreshed Phase 7 source: 51fc5014222306dee30a48ddbe2c19b7a1fafe8f
  • refreshed Phase 7 artifact SHA256: 6f3a8576f60efa77a3191d50cd10c78aa4adbfb3d31ebe8cbfbc91e561bcd5de
  • Phase 7: 12/12 mandatory A/B/C/D commands passed with zero deselections, skips, xfails, or xpasses
  • canonical final validator: ok=true, errors=[], promotion required and satisfied
  • git diff --check passed; worktree clean

Impact

  • benchmark implementation and measured values: unchanged
  • training/backend behavior: unchanged
  • test stability: removes an invalid cross-sample timing ordering while preserving the actual benchmark record contract
  • support boundary: unchanged; the only Recommended combination remains ppo_torch + g1_walk_flat + mjwarp + device_resident

Integration

This child PR intentionally targets feat/issue-705-manager-mjwarp. Per #705, final PR #847 remains open for real main-targeted CI and user review; this PR does not approve or merge #847.

@TATP-233
TATP-233 merged commit 4053ad3 into feat/issue-705-manager-mjwarp Aug 2, 2026
@TATP-233
TATP-233 deleted the fix/issue-855-benchmark-timing-contract branch August 2, 2026 16:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant