Camera/audio/cmd web bridges + Mac+LM-Studio agentic compose

[tfius]

Summary

Adds three web bridges and a local-LLM hackathon quickstart, exposing the
robot's I/O over plain HTTP/WS so external processes (e.g. an MLX VLM running
elsewhere on the same Mac) can drive perception + control without speaking
any internal dimos APIs.

New modules (Mac- and sim-friendly, no CUDA)

• camera-mjpeg-module — port 7780
  • GET /video_feed/color_image MJPEG (multipart/x-mixed-replace)
  • GET /snapshot/color_image single JPEG
  • CORS open
• audio-ws-module — port 7781
  • ws://…/audio_out binary int16 PCM frames (with a JSON {event:"format",…} hello)
  • GET /audio_info reports rate/channels
  • POST /play accepts WAV bytes → robot speaker (megaphone path)
• cmd-bridge-module — port 7782
  • POST /cmd_vel, POST /path (raw Twist or semantic forward/left/degrees),
    POST /stop, GET /pose
  • Open-loop; designed for a VLM that re-plans each iteration

Go2 audio I/O (real robot only — sim has no audio)

• UnitreeWebRTCConnection.audio_stream() → AudioMessage(data, sample_rate, channels),
  hooks the existing WebRTC audio transceiver
• UnitreeWebRTCConnection.play_wav_bytes(wav) — uploads via audiohub.upload_megaphone,
  enters megaphone, sleeps for the WAV's duration, then exits cleanly
• New skills on GO2Connection: play_wav(path), play_wav_b64(base64)
• New streams: audio: Out[AudioMessage], audio_in: In[bytes]

Launcher scripts

• go2-start.sh — hackathon quickstart: wifi/ping/NTP checks, env-based LLM
  presets (LMSTUDIO=1, MLXVLM=1), auto-injects mcp-server + mcp-client
  when an LLM preset is set against a non-agentic blueprint
• sim-with-llm.sh — thin wrapper that sets SIMULATION=1 and forwards
• Both probe /v1/models before launch and surface a useful error if the
  backend isn't up

Docs

• journal/2026-05-27-camera-audio-bridges.md — implementation notes,
  review fixes, Mac+local-LLM agentic landscape
• journal/2026-05-27-mlxvlm-robot-integration-prompt.md — drop-in brief
  for an external Claude/agent that consumes these endpoints

Includes (from earlier feat/gemini-go2-2245 work by @grmkris, not by me)

• feat(go2): all-Gemini VL for the Mac-only agentic blueprint
• feat(take_picture), feat(follow_person), feat(go2): tilt_body,
  save/load map, room_scan, etc.

Test plan

• ./sim-with-llm.sh lmstudio boots; agent reaches the endpoint
• ./sim-with-llm.sh mlxvlm reaches mlxvlm at :8080
• Open http://127.0.0.1:7780/video_feed/color_image — MJPEG renders
• curl -X POST -d '{"steps":[{"forward":0.5,"duration":2}]}' http://127.0.0.1:7782/path
  drives the sim Go2 forward
• dimos mcp list-tools shows observe, play_wav, play_wav_b64
• On real Go2: WebSocket audio_out streams PCM; POST /play plays
  through the dog's speaker and exits megaphone cleanly

🤖 Generated with Claude Code

[greptile-apps]

Greptile Summary

This PR adds three HTTP/WebSocket bridges (MJPEG camera, PCM audio, cmd_vel) to expose robot I/O over plain HTTP for external VLM/LLM processes, adds Go2 audio I/O via WebRTC, several new skills (Gemini TTS, macOS say, take_picture, map_uploader), a Gemini-all blueprint, and two hackathon launcher scripts.

• Web bridges (mjpeg_module, audio_ws_module, cmd_bridge_module): each runs uvicorn in a daemon thread, uses proper cross-thread handoffs, and binds to loopback only.
• Audio I/O (UnitreeWebRTCConnection): audio_stream() correctly uses a Subject + finally_action teardown; play_wav_bytes() fires a coroutine into the existing event loop — but exit_megaphone() is missing from the finally block, which can leave the robot unable to move.
• Launchers (go2-start.sh, sim-with-llm.sh): pre-flight wifi/ping/NTP/LLM checks with LLM auto-injection; a hardcoded developer filesystem path appears in one error message.

Confidence Score: 3/5

Safe to merge with one fix: the megaphone-exit bug in play_wav_bytes must be addressed before deploying on a real Go2, as it can leave the robot unresponsive to all motion commands until manually reset.

The audio WebRTC path in connection.py enters megaphone mode and then calls exit_megaphone() outside any finally block. A coroutine cancellation during active playback will leave the robot permanently in megaphone mode, blocking every subsequent motion command. The rest of the PR — the three web bridges, Gemini TTS skill, take-picture skill, and launcher scripts — is well-structured and largely correct.

dimos/robot/unitree/connection.py (play_wav_bytes / _upload_play_exit) requires the exit_megaphone finally-block fix before real-robot use. go2-start.sh has a minor hardcoded path to address.

Important Files Changed

Filename	Overview
dimos/robot/unitree/connection.py	Adds AudioMessage dataclass, audio_stream() observable, and play_wav_bytes(). The exit_megaphone() call is not wrapped in a finally block, so cancellation leaves the robot stuck in megaphone mode.
dimos/web/audio_ws_module.py	New WebSocket bridge for robot audio (mic out + WAV playback). Thread/async handoff via a bounded queue is correctly implemented; _clients is only accessed from the same asyncio loop so no race. Clean.
dimos/web/cmd_bridge_module.py	New HTTP bridge for robot motion commands. Lock-and-cancel design is correct; duration field lacks an upper bound that could stall the thread-pool worker indefinitely.
dimos/web/mjpeg_module.py	New MJPEG + snapshot HTTP module. Frame encoding is correct (RGB to BGR before imencode), snapshot is lock-protected, CORS is open. No issues.
dimos/robot/unitree/go2/connection.py	Adds audio/audio_in streams and play_wav/play_wav_b64 skills to GO2Connection. Uses hasattr guards for sim compatibility. Clean.
go2-start.sh	New hackathon launcher with wifi/ping/NTP/LLM-endpoint checks. Contains a hardcoded developer local path (/Users/tex/repos/…) in the mlxvlm error message.
dimos/agents/skills/gemini_speak_skill.py	New Gemini TTS-backed speak skill. Idempotent consumer wiring, fire-and-forget queue, blocking wait with text matching, and resource cleanup are all handled correctly.
dimos/agents/skills/take_picture_skill.py	New skill for one-shot and explore-and-capture photo upload. Background threads tracked and joined on stop; tilt always re-leveled in finally. Clean.
sim-with-llm.sh	Thin wrapper around go2-start.sh that routes the backend argument (mlxvlm/lmstudio/ollama) and sets SIMULATION=1. No issues.
dimos/stream/audio/tts/node_gemini.py	New Gemini TTS pipeline node. Worker thread and subscription lifecycle are correctly managed; dispose() drains the queue and joins the thread before completing subjects.

Sequence Diagram

sequenceDiagram
    participant ExtClient as External VLM/Client
    participant MjpegMod as CameraMjpegModule (port 7780)
    participant AudioMod as AudioWsModule (port 7781)
    participant CmdMod as CmdBridgeModule (port 7782)
    participant GO2Conn as GO2Connection
    participant UnitreeConn as UnitreeWebRTCConnection

    GO2Conn->>MjpegMod: color_image stream
    GO2Conn->>AudioMod: audio (AudioMessage) stream
    UnitreeConn->>GO2Conn: video_stream / audio_stream

    ExtClient->>MjpegMod: GET /video_feed/color_image (MJPEG)
    MjpegMod-->>ExtClient: multipart/x-mixed-replace JPEG frames

    ExtClient->>AudioMod: WS /audio_out
    AudioMod-->>ExtClient: binary int16 PCM frames

    ExtClient->>AudioMod: POST /play (WAV bytes)
    AudioMod->>GO2Conn: audio_in (bytes)
    GO2Conn->>UnitreeConn: play_wav_bytes(wav)
    UnitreeConn->>UnitreeConn: upload_megaphone, enter, sleep, exit

    ExtClient->>CmdMod: POST /cmd_vel or /path
    CmdMod->>GO2Conn: cmd_vel (Twist)

    ExtClient->>CmdMod: GET /pose
    CmdMod-->>ExtClient: x, y, z, theta

Loading

_{Reviews (1): Last reviewed commit: "fix(go2-start): auto-inject mcp-server/m..." | Re-trigger Greptile}

[greptile-apps]

If asyncio.sleep() is cancelled (e.g. the event loop shuts down mid-play) or any awaitable after enter_megaphone() raises, exit_megaphone() is never called and the robot is permanently stuck in megaphone mode until manually reset, rendering all movement commands ineffective.

Suggested change

	await hub.upload_megaphone(tmp_path)
	await hub.enter_megaphone()
	# Hold megaphone for the clip's duration, plus a small flush margin,
	# then release so other commands work normally.
	await asyncio.sleep(duration + 0.5)
	await hub.exit_megaphone()
	await hub.upload_megaphone(tmp_path)
	await hub.enter_megaphone()
	try:
	# Hold megaphone for the clip's duration, plus a small flush margin,
	# then release so other commands work normally.
	await asyncio.sleep(duration + 0.5)
	finally:
	await hub.exit_megaphone()

[greptile-apps]

This error message contains a hardcoded path from the developer's local machine (/Users/tex/repos/ai/mlx/mlxvlm). Every other user will see an incorrect and confusing instruction.

Suggested change

	c_red " start mlxvlm: cd /Users/tex/repos/ai/mlx/mlxvlm && scripts/start-all.sh"
	c_red " start mlxvlm server and ensure it is listening on port 8080"

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[greptile-apps]

duration has no upper bound. A caller can send duration: 999999, blocking the uvicorn thread-pool worker (and the _drive_lock) for hours, making /stop unable to interrupt ongoing execution and preventing subsequent commands from acquiring the lock.

Suggested change

	duration: float = 0.5 # seconds to hold the command before stopping
	duration: float = Field(default=0.5, ge=0.0, le=30.0) # seconds to hold the command before stopping