Skip to content

Integrate Droid lightwheel kitchen example with robot placement#948

Open
qianl-nv wants to merge 19 commits into
zxiao/feature/robo_placement_prototypefrom
qianl/dev/robot_placement_integration_3
Open

Integrate Droid lightwheel kitchen example with robot placement#948
qianl-nv wants to merge 19 commits into
zxiao/feature/robo_placement_prototypefrom
qianl/dev/robot_placement_integration_3

Conversation

@qianl-nv

@qianl-nv qianl-nv commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

Summary

Add Droid kitchen env yaml with robot placement

Detailed description

  • Add droid_pick_and_place_lightwheel_kitchen.yaml with Droid-on-floor/counter relations in the Lightwheel kitchen scene.
  • Use stand footprint for Droid placement bounds and write robot + stand roots on reset via layout_pose_to_scene_writes, adapted to the unified PlacementAsset/EmbodimentBase pose-reset path and main #945 stand-height API.
  • Extend placement-event and relation-solver tests for Droid stand reset, bbox sizing.

Command to run (start pi server first):
/isaac-sim/python.sh isaaclab_arena/evaluation/policy_runner.py --viz kit --policy_type isaaclab_arena_openpi.policy.pi0_remote_policy.Pi0RemotePolicy --num_episodes 2 --enable_cameras --record_viewport_video --output_base_dir output/kitchen_task --env_graph_spec_yaml isaaclab_arena_environments/droid_pick_and_place_lightwheel_kitchen.yaml

Intermediate results:

  • On base:
image
  • Switch to stand bbox without stand height fix:
image
  • Stand bbox + stand height fix:
image image (114)

TODO: Enable mesh-based placement for background and embodiment

zhx06 and others added 12 commits July 23, 2026 20:11
Signed-off-by: zhx06 <zihaox@nvidia.com>
Signed-off-by: zhx06 <zihaox@nvidia.com>
Signed-off-by: zhx06 <zihaox@nvidia.com>
Signed-off-by: zhx06 <zihaox@nvidia.com>
Signed-off-by: zhx06 <zihaox@nvidia.com>
Signed-off-by: zhx06 <zihaox@nvidia.com>
Signed-off-by: zhx06 <zihaox@nvidia.com>
Signed-off-by: zhx06 <zihaox@nvidia.com>
Signed-off-by: zhx06 <zihaox@nvidia.com>
Relation placement reset wrote only the robot root at the solver's
unlifted z, leaving the stand at spawn and causing float when
stand_height_m differs from the default. Map layout poses to all scene
roots via layout_pose_to_scene_writes and apply stand-height lift for
Droid on reset.

Signed-off-by: Qian Lin <qianl@nvidia.com>
@qianl-nv
qianl-nv force-pushed the qianl/dev/robot_placement_integration_3 branch from 79fb19e to 9d391a8 Compare July 24, 2026 06:58

@qianl-nv qianl-nv left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

self review 1

Comment thread isaaclab_arena/embodiments/droid/droid.py Outdated
Comment thread isaaclab_arena/embodiments/droid/droid.py
Comment thread isaaclab_arena/embodiments/droid/droid.py Outdated
Comment thread isaaclab_arena/relations/placement_events.py Outdated
Comment thread isaaclab_arena/tests/test_droid_reset_placement.py Outdated
Comment thread isaaclab_arena/tests/test_embodiment_placement_bbox.py Outdated
Comment thread isaaclab_arena/tests/test_placement_events.py
qianl-nv added 2 commits July 24, 2026 15:42
* Clean up code comments
* Fix bug in droid get_bounding_box offset handling
Merge placement-bbox and reset-alignment coverage into test_droid_stand_height.py so the suite uses one sim startup and drops redundant cases. Remove the overlapping companion-scene write test; the Droid reset write test covers multi-prim layout_pose_to_scene_writes.

Signed-off-by: Qian Lin <qianl@nvidia.com>
@qianl-nv
qianl-nv marked this pull request as ready for review July 24, 2026 08:01
@greptile-apps

greptile-apps Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR integrates the Droid robot into the Lightwheel kitchen environment by adding a new YAML spec and extending the placement/reset pipeline to write both the robot root and its separate stand prim on each reset. The key mechanical change is layout_pose_to_scene_writes on DroidEmbodimentBase, which lifts the solver layout pose by the stand-height offset and composes the stand's local root offset via compose_poses, plus a new _write_scene_root_pose_to_sim helper that dispatches between write_root_pose_to_sim (articulations/rigid bodies) and set_world_poses (static prims like the stand).

  • droid.py: Adds get_bounding_box (stand footprint as placement proxy), _stand_pose_from_robot_pose, and layout_pose_to_scene_writes; refactors stand init_state to use the shared _STAND_ROOT_OFFSET_IN_ROBOT_FRAME constant instead of the robot pose directly.
  • placement_events.py: Extracts _write_scene_root_pose_to_sim to handle both physics and non-physics scene assets polymorphically; the write_layout_to_sim loop now uses it.
  • Test files: Expanded with stand reset alignment, bbox sizing, and relation-solver stand-write tests; the kitchen reset alignment check contains a broken API call (get_world_poses()[0].torch) that will raise AttributeError at runtime, so the alignment invariant is never actually verified.

Confidence Score: 3/5

The production placement code is logically correct, but the new kitchen reset alignment test will always crash before making any assertion, so the robot-z vs stand-z invariant it is meant to guard is never verified.

The core stand-height offset and dual-prim reset logic is well-reasoned and consistent with the existing codebase. The broken .torch access in _check_kitchen_reset_alignment means the test cannot pass on any run, leaving the most important integration-level check non-functional. The stand mock issue in test_placement_events.py further means the set_world_poses dispatch path for the stand is untested, so a regression there would be silent.

test_droid_stand_height.py (_check_kitchen_reset_alignment) and test_placement_events.py (test_solve_and_place_objects_writes_droid_robot_and_stand) need attention before the test suite can provide meaningful coverage of the new reset path.

Important Files Changed

Filename Overview
isaaclab_arena/embodiments/droid/droid.py Adds get_bounding_box (stand footprint proxy), _stand_pose_from_robot_pose, and layout_pose_to_scene_writes; refactors stand init_state to use the shared _STAND_ROOT_OFFSET_IN_ROBOT_FRAME constant. Logic is sound but the stand scene-entity name is hardcoded.
isaaclab_arena/relations/placement_events.py Extracts _write_scene_root_pose_to_sim to handle both articulation (write_root_pose_to_sim) and static-mesh (set_world_poses) assets. Dispatch logic is correct; the unreachable sentinel uses assert False which is silently dropped under -O.
isaaclab_arena/tests/test_droid_stand_height.py Expanded test suite for stand scaling, pose wiring, bbox, and kitchen reset alignment; _check_kitchen_reset_alignment accesses get_world_poses()[0].torch which is not a valid attribute on torch.Tensor and will raise AttributeError at runtime.
isaaclab_arena/tests/test_placement_events.py Adds Droid robot+stand placement test and missing-asset assertion test; the Droid test uses MagicMock which auto-creates write_root_pose_to_sim, hiding that the stand should use set_world_poses in production.
isaaclab_arena_environments/droid_pick_and_place_lightwheel_kitchen.yaml New environment spec for Droid in the Lightwheel kitchen scene; relations (on floor, next_to counter, rotate, object placement) are well-formed and consistent with the arena env graph format.

Reviews (1): Last reviewed commit: "Consolidate Droid stand and placement te..." | Re-trigger Greptile

Comment thread isaaclab_arena/tests/test_droid_stand_height.py Outdated
Comment thread isaaclab_arena/relations/placement_events.py Outdated
Comment thread isaaclab_arena/tests/test_placement_events.py Outdated
Comment thread isaaclab_arena/embodiments/droid/droid.py
Comment thread isaaclab_arena/embodiments/droid/droid.py
Comment thread isaaclab_arena/tests/test_droid_stand_height.py
@arena-review-bot

Copy link
Copy Markdown
Contributor

🤖 Isaac Lab-Arena Review Bot

Summary

This PR adds a Droid + Lightwheel-kitchen environment yaml and, to support it, makes the Droid embodiment place both the robot root and its separate stand on reset (via the layout_pose_to_scene_writes compound-asset hook) and use the stand footprint as its placement bbox. The scene-write plumbing is extended so an AssetBase extra (the stand) can be written with set_world_poses. The composition boundaries look right — using the existing layout_pose_to_scene_writes extension point is the intended pattern — but the stand-aware write dispatch was only added to one of the two reset paths.

Findings

🟡 Warning: embodiments/droid/droid.py:144layout_pose_to_scene_writes now emits a ("stand", …) write for every reset path, but only the relation-solver path (placement_events._write_scene_root_pose_to_sim) handles the stand's AssetBase via set_world_poses. The fixed-pose reset path (terms/events._write_scene_pose, built when a Droid is given an explicit initial_pose) still calls write_root_pose_to_sim unconditionally and will raise AttributeError on reset. The kitchen example is unaffected (it places via relations), but a fixed-pose Droid regresses. The two write routines are near-duplicates that have diverged — worth a single shared dispatch helper.

🔵 Improvement: tests/test_droid_stand_height.py:139 — the kitchen reset check only compares robot/stand z. Since the stand and robot are written through different Isaac APIs (set_world_poses vs write_root_pose_to_sim) and the spec applies a 1.57 rad yaw, a quaternion-order mismatch would leave the stand mis-rotated while z still matches. Consider also asserting the stand's world orientation equals the robot's.

Test Coverage

Good coverage overall: the consolidated test_droid_stand_height.py exercises stand-height scaling, robot/stand pose wiring, placement bbox, and an end-to-end kitchen reset, all under one run_simulation_app_function session (correct inner/outer pattern). test_placement_events.py adds a mock-env test that the Droid robot+stand are both written, plus a missing-asset assertion. Two gaps: (1) no test covers the fixed-pose (reset_placement_asset_pose) Droid path that the finding above breaks; (2) reset alignment only checks z, not orientation.

Verdict

Minor fixes needed — mainly routing the fixed-pose reset path through the same stand-aware write dispatch.

qianl-nv added 3 commits July 24, 2026 16:18
Keep the original _test_droid_stand_height body and move placement bbox and kitchen reset checks into separate pytest entry points.

Signed-off-by: Qian Lin <qianl@nvidia.com>
get_world_poses()[0] is already a torch.Tensor; accessing .torch
raised AttributeError and prevented the kitchen reset check from running.

Signed-off-by: Qian Lin <qianl@nvidia.com>
Extract write_scene_root_poses_to_sim so relation placement and fixed-pose
embodiment resets both route AssetBase extras (e.g. Droid stand) through
set_world_poses instead of write_root_pose_to_sim.

Signed-off-by: Qian Lin <qianl@nvidia.com>
qianl-nv added 2 commits July 24, 2026 16:31
Configure stand mocks without write_root_pose_to_sim so unit tests exercise
the AssetBase dispatch path used in production for the Droid stand.

Signed-off-by: Qian Lin <qianl@nvidia.com>
Compare stand world quaternion to the robot root after relation placement
reset so quaternion-order bugs between write_root_pose_to_sim and
set_world_poses are caught, not just z alignment.

Signed-off-by: Qian Lin <qianl@nvidia.com>
@zhx06
zhx06 force-pushed the zxiao/feature/robo_placement_prototype branch from ee9f659 to dedf07a Compare July 24, 2026 16:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants