Skip to content

Add RoboLab (Isaac Lab DROID benchmark) behind the env-server#469

Open
vertix wants to merge 3 commits into
Positronic-Robotics:mainfrom
vertix:robolab
Open

Add RoboLab (Isaac Lab DROID benchmark) behind the env-server#469
vertix wants to merge 3 commits into
Positronic-Robotics:mainfrom
vertix:robolab

Conversation

@vertix

@vertix vertix commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Summary

Integrates NVIDIA RoboLab (Isaac Lab DROID benchmark, pinned to NVLabs/RoboLab@7d45d74) as a remote eval environment behind positronic's env-server wire.

  • Extracts shared env-server client machinery — WireCommandAdapter, the subprocess launcher, remote_franka_embodiment — and migrates libero and the env-server test fixture onto it.
  • Adds positronic/simulator/robolab/ (env, adapter, launcher, validate, e2e, make_fixture) plus positronic/cfg/eval/sim/robolab.py with the pinned 120-task table and named configs benchmark / visual / relational / procedural / banana_in_bowl / rubiks_cube_and_banana.

Design decisions

  • All-to-all transparent env: one jointpos-substrate registration; all command conversion happens server-side — joint_pos is a passthrough (bit-identical to the RoboLab leaderboard stack), joint_velq_measured + dq, holdq_measured, and Cartesian commands go through a standalone Isaac Lab DifferentialIKController (DroidIKActionCfg parameters; commands and observations share the eef_frame = base_link ∘ EEF_OFFSET_ROT (0.5, -0.5, 0.5, -0.5) wxyz).
  • No seed anywhere: RoboLab exposes no seed hook, so trial contexts carry no eval.seed (built directly, not via build_trials).
  • done/success follow RoboLab's frozen-env accounting (all_terminated + get_env_results), not raw term/trunc — their env discards ≤2-step "physics artifact" terminations, so raw term would score false successes.
  • Delivery = host-uv: the launcher clones and pins the repo, git lfs pulls assets, and patches upstream's constraint-dependencies with cffi != 2.1.0. Upstream ships no lockfile, so fresh resolves can break on day-fresh releases, and --exclude-newer is impossible (pypi.nvidia.com publishes no upload timestamps). A full freeze (lockfile or baked image) is a parked TODO in launcher.py.

No CI story

Both verification gates need an RTX-class GPU and run on-box; nothing beyond the normal unit suite runs in GitHub CI.

Test plan

On-box verification (Nebius L40S, driver 570):

  • Smoke: uv sync 79 s (18 GB venv), swept all 120 task envs, abs-IK demo passed, peak VRAM ~10.4 GB.
  • validate.py: ALL CHECKS PASSED — obs contract, grip extremes, bit-identical jointpos passthrough (0.0002 rad convergence), exact q + dq anchoring, eef↔base_link offset round-trip, 13/13 Cartesian IK cases ≤0.12 mm / 0.04° (tolerances 5 mm / 2°; including their known-divergent translate +x case), cartesian_delta 0.06 mm.
  • e2e.py: E2E REPLAY PASSED — exact-state reset drift 0.0115 rad, full 530-action socket replay clean. Replay task success is reported informationally only (0/1 here): the shipped demo recording is stale at the pinned commit — RoboLab's own run_recorded.py also fails it (0/1) on the same box — so demo replay is not a success oracle. The success oracle is the follow-up PR's policy differential (openpi DROID policy vs their leaderboard client).
  • Local: uv run --locked pytest --no-cov → 606 passed, 7 skipped; ruff clean.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant