feat(teleop): stream robot camera into Quest VR headset

[ruthwikdasyam]

Problem

Adding streaming to the VR, for teleop - sim camera, realsense camera addition, or go2 camera view will be displayed in the headset

Closes DIM-850

Solution

• QuestTeleopModule has an optional color_image: In[Image]. When the blueprint wires it, frames are JPEG-encoded with OpenCV and cached; otherwise the subscribe is skipped and /video stays empty.
• Wired into teleop_quest_xarm7 so dimos --simulation run teleop-quest-xarm7 shows MuJoCo's color_image out of the box; for real hardware, compose at launch: dimos run teleop-quest-xarm7 real-sense-camera.

How to Test

# login through quest, you can see the camera view
dimos --simulation run teleop-quest-xarm7

# for go2
dimos run teleop-quest-go2

Contributor License Agreement

• I have read and approved the CLA.

VR_go2.mp4

VR_arm_sim.mp4

[greptile-apps]

Greptile Summary

This PR adds robot camera streaming into the Meta Quest VR headset for teleop sessions. JPEG frames are encoded server-side via TurboJPEG, pushed as binary WebSocket messages, and rendered as a world-locked WebGL quad in the XR scene.

• Adds VideoArmTeleopModule (xArm7 + camera panel) and Go2TeleopModule (Go2 velocity teleop + camera), each wired with new teleop-quest-xarm7-video and teleop-quest-go2 blueprints.
• Extracts inline HTML/CSS/JS into separate static files; adds videoDirty flag so texImage2D is called only when a new JPEG has decoded, and fixes gamepad axis selection with an xr-standard mapping check.

Confidence Score: 5/5

Safe to merge; changes are additive (new blueprints and modules), the existing arm teleop path is untouched, and the client/server threading model is sound.

All changed code paths are new (new module classes, new blueprints, new static files). The base QuestTeleopModule changes are minimal and correct (client-set tracking with a lock and a finally cleanup). The two observations flagged are style/performance notes with no incorrect runtime behavior on the current code paths.

dimos/teleop/quest/quest_extensions.py — the handle_color_image async/sync interaction is worth a second look, but does not block the existing teleop path.

Important Files Changed

Filename	Overview
dimos/msgs/sensor_msgs/Image.py	Refactored JPEG encoding into a new to_jpeg_bytes() helper; lcm_jpeg_encode now delegates to it. Note: a new TurboJPEG() handle is opened on every call.
dimos/teleop/quest/quest_teleop_module.py	Adds _connected_clients, _clients_lock, and _ws_loop to the base module so subclasses can push frames; the finally block correctly cleans up disconnected clients.
dimos/teleop/quest/quest_extensions.py	Adds _push_jpeg helper and two new modules: VideoArmTeleopModule and Go2TeleopModule. The handle_color_image is declared async def but calls the synchronous, CPU-bound _push_jpeg; if the framework awaits handlers on the event loop, JPEG encoding would block it.
dimos/teleop/quest/blueprints.py	Adds teleop_quest_xarm7_video and teleop_quest_go2 blueprints with correct transport wiring for camera and velocity outputs.
dimos/teleop/quest/web/static/teleop.js	Extracted from inline script; adds MJPEG-over-WebSocket reception, blob-URL management with deferred revocation, WebGL quad rendering with a dirty flag, and corrected xr-standard gamepad axis mapping.
dimos/teleop/quest/web/static/index.html	Moves CSS to teleop.css and JS to teleop.js; adds hidden <img id="videoFeed"> element for JPEG frame decoding.
dimos/teleop/quest/web/static/teleop.css	New file; CSS extracted verbatim from the former inline <style> block in index.html.
dimos/robot/all_blueprints.py	Registers two new blueprints (teleop-quest-go2, teleop-quest-xarm7-video) and two new modules (go2-teleop-module, video-arm-teleop-module) in the global registry.

Sequence Diagram

sequenceDiagram
    participant Cam as Camera/Simulator
    participant Trans as LCM/pSHM Transport
    participant Mod as VideoArmTeleopModule
    participant Push as _push_jpeg()
    participant Ev as asyncio event loop
    participant WS as WebSocket /ws
    participant JS as teleop.js
    participant GL as WebGL Texture

    Cam->>Trans: publish Image frame
    Trans->>Mod: handle_color_image(msg)
    Mod->>Push: _push_jpeg(self, msg, quality)
    Push->>Push: msg.to_jpeg_bytes() JPEG encode
    Push->>Ev: run_coroutine_threadsafe(_ws_send_jpeg)
    Ev->>WS: ws.send_bytes(jpeg)
    WS->>JS: ws.onmessage Blob received
    JS->>JS: URL.createObjectURL(blob)
    JS->>JS: "videoEl.src = newUrl, onload sets videoDirty=true"
    JS->>GL: uploadVideoTexture() on next XR frame
    GL->>GL: gl.texImage2D with videoEl
    GL->>GL: renderVideoPanel() per-eye quad draw

Loading

_{Reviews (4): Last reviewed commit: "Refactor: html into css+js for fontend/U..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 43.47826% with 52 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
dimos/teleop/quest/quest_extensions.py	43.47%	39 Missing ⚠️
dimos/teleop/quest/quest_teleop_module.py	11.11%	8 Missing ⚠️
dimos/msgs/sensor_msgs/Image.py	16.66%	5 Missing ⚠️

📢 Thoughts on this report? Let us know!

[leshy]

yoooo can you add Yaw control for robot body?

[greptile-apps]

Want your agent to iterate on Greptile's feedback? Try greploops.