- Field note / Process
- 2026.06.05
- >_
+
+
+
+ 
+
+
+Useful AI agents now run for long periods of time, taking on critical tasks to accomplish goals. They call tools, learn new skills, write code, and keep working while no one is watching. How do you give autonomy to such a highly capable agent?
+
+OpenShell is the secure runtime for autonomous agents. It solves this problem by moving the security layer outside the agent. OpenShell runs your agent in a sandbox and sits in the path of everything the agent does. A policy layer, configured separately from the agent, governs what processes the agent can start, which hosts and ports it can reach, which models it can call, and which other services it needs to invoke along its network path.
+
+Every OpenShell deployment has different needs, and different agents want different things, so we made OpenShell extensible. You are not locked into a fixed set of behaviors. You can swap the isolation backend, add your own checks on policy updates, and plug in middleware services that run on traffic as it leaves the sandbox.
+
+The combination of autonomous agents and edge devices like robots brings the digital concerns of traditional software deployments into the physical world. Imagine a home assistant robot that is only supposed to clean. It suffers a prompt injection and starts unlocking the doors instead. At the edge, a compromised agent does not just leak data or crash a process. It can let a stranger into your home, or send what it sees and hears somewhere it should never go.
+
+OpenShell makes an autonomous agent safe to run at the edge by solving two problems:
+
+- **Restrict what the device can do.** Deterministic policy decides what the agent can actually make the device do, and it is enforced on the device.
+- **Keep the data private.** Sensitive data stays local, and only approved services ever see it.
+
+We used the HuggingFace Reachy Mini to see what that looks like in practice. We run a small chat application connected to an OpenAI endpoint within the OpenShell runtime, all on the Reachy Mini's onboard Raspberry Pi. Running the whole stack on a Raspberry Pi is part of the point: OpenShell holds up on the small, resource-constrained hardware that real edge devices ship with, not just a workstation. We use OpenShell to restrict what actions the model can take. Our next step is to route sensitive data to approved models, but more on that in a follow-up post.
+
+If you just want to try this for yourself, check out our tutorial [here](https://github.com/NVIDIA/OpenShell-Research/blob/kirit93/reachy-implementation/projects/reachy-mini-openshell/ONBOARD_SETUP.md).
+
+---
+
+## Give Each Job Its Own Boundary
+
+Start with the first problem: restricting what the device can do. The simplest
+way to build this is to run the agent on the device and give it the robot's SDK.
+That works for a quick demo, but it puts two things that should be separate into
+a single process.
+
+**The agent gets more than it needs.** A robot SDK exposes motors, raw targets,
+camera settings, recorded motions, app management, and system state. An agent
+that only needs a few fixed head moves should not inherit all of that.
+
+**The decision happens too far from the effect.** A remote model can pick which
+tool to call, but it should not be the final say on whether a local motor
+actually moves. That call belongs on the device, right before the request reaches
+the controller.
+
+Both come from the same place: one process holds all the access and makes all the
+decisions. So we split the work into separate parts, each with one job.
+
+1. **The model reasons.** It listens to what the user says and picks from the
+ tools the app offers. The model can be remote or local.
+2. **The sandboxed agent turns that choice into a concrete request.** It owns the
+ conversation state and the tool logic, but it does not own the hardware.
+3. **OpenShell decides whether the request may leave the sandbox.** It checks the
+ tool call request against the OpenShell policy.
+4. **A trusted adapter exposes one narrow capability.** It owns the native device
+ objects and turns an approved request into a single bounded operation.
+5. **The device controller does the work.** Motors, sensors, and cameras are
+ reached only after every boundary above has allowed it.
+
+
+ 
+ Reasoning, policy enforcement, hardware ownership, and device execution are separated into distinct boundaries. The local OpenShell policy decision is the control point, evaluated on the device before a request can leave the sandbox.
+
+
+The sandbox starts with deny-by-default policy instead of inheriting everything
+the host can do. When OpenShell blocks an action, it returns a natural-language
+message explaining why, along with examples of what is allowed. The agent can
+tell the person what happened and look for a safe way to do the same task,
+instead of quietly working around the limit. An operator can change the policy
+without rebuilding the app, and the power to grant new authority stays outside
+the agent.
+
+---
+
+## How We Built It
+
+These were the main choices behind the build.
+
+### Run OpenShell entirely on device
+
+We run all of OpenShell on Reachy's onboard computer. The gateway is the control
+plane: it creates the sandbox, holds the policy, and manages the sandbox
+lifecycle. The agent runs inside the sandbox. When the agent tries to move the
+robot, OpenShell checks the request on the device and decides whether it reaches
+the daemon. The same check controls what leaves the sandbox: every outbound
+connection can be allowed, routed somewhere else, or denied.
+
+### Expose capabilities, not complete APIs
+
+We give the agent three tools:
+
+- `move_head(directions)` accepts `left`, `right`, `up`, `down`, and `front`.
+- `stop_motion()` stops active movements.
+- `camera(question)` requests one image for the current conversation.
+
+The model picks one of these tools. It does not build the HTTP request itself.
+Python handlers in the sandbox turn `move_head` into `POST /api/move/goto`,
+`stop_motion` into `POST /api/move/stop`, and `camera` into
+`POST /camera/capture`. OpenShell then checks that concrete request before it can
+leave the sandbox.
+
+The agent never sees the full Reachy API. Fixed tool inputs become fixed REST
+requests, so the capability you meant to grant is visible right at the network
+boundary. That is much easier to reason about than handing the agent a
+general-purpose SDK and trying to list every unsafe combination after the fact.
+
+Keeping the tools fixed also covers a gap in what policy can see. OpenShell
+matches the calling binary, destination, method, path, and query, but it does not
+yet check the values inside a JSON body. If one endpoint could do many things,
+allowing its path would let the sandbox send any body that endpoint accepts.
+Fixed tools avoid that: `move_head` only ever produces its five poses, a limit
+the app enforces rather than policy, and `POST /camera/capture` takes no
+arguments the model can set at all.
+
+### Keep hardware handles in trusted native code
+
+Reachy's native app already owns the microphone, speaker, and camera. Moving
+those into the sandbox would split hardware ownership in two and drag the full
+Reachy SDK, media stack, and vision dependencies into the agent image.
+
+Instead, a small trusted Reachy App keeps the media objects. It forwards PCM
+audio over a local WebSocket and exposes one camera route with no arguments:
+`POST /camera/capture`. The sandbox can hold a conversation and ask for one
+frame. It never gets the camera handle, the device choice, the resolution
+controls, or a file path.
+
+### Use a remote or local model
+
+The model in our demo is remote, an OpenAI Realtime endpoint. Everything else,
+the conversation, the policy, the adapter, and the controller, runs on Reachy.
+Edge does not have to mean fully offline, and enforcing policy on the device does
+not need a local model. If you later want an on-device model, only the endpoint
+changes. The sandbox, the adapter, and the boundary stay the same.
+
+---
+
+## What This Looks Like on Reachy
+
+
+ 
+ Every request to the hardware passes through OpenShell first. The OpenShell policy allows the camera capture path (green) and denies the movement path (red), so the camera works and the head does not move, enforced by policy rather than by the application choosing to refuse.
+
+
+There is no laptop, browser, or Gradio page in the runtime path. The full
+request flow is:
+
+1. The trusted Reachy App captures microphone audio and streams PCM frames to
+ the audio service in the sandbox.
+2. The agent sends that audio to the OpenAI Realtime API over its
+ policy-approved WebSocket. The model gets the conversation along with the
+ fixed `move_head`, `stop_motion`, and `camera` tools.
+3. The model returns audio or picks a tool. For a tool call, Python code in the
+ sandbox turns the model's arguments into one of the fixed REST requests.
+4. OpenShell checks the destination, port, method, and path. An allowed request
+ reaches the trusted adapter or the Reachy daemon. A denied request comes back
+ as a policy error and never reaches the hardware.
+5. The agent uses the tool result or the policy error to keep talking, and the
+ response audio goes back through the native Reachy App to the speaker.
+
+The demo policy splits two physical capabilities:
+
+```text
+POST host.openshell.internal:8042/camera/capture allowed
+POST host.openshell.internal:8000/api/move/goto denied
+```
+
+When someone says, "Take a picture of me," the agent calls `camera`, OpenShell
+allows the capture path, the trusted adapter returns one bounded JPEG, and
+Reachy describes it aloud.
+
+When they say, "Turn right and take a picture," the agent tries
+`POST /api/move/goto`. OpenShell denies it before it reaches the Reachy daemon.
+The robot stays still, and it says so out loud.
+
+The key point is that the app is not just choosing not to move. The agent makes
+a real attempt to move, and a separate policy boundary stops the physical
+effect.
+
+---
+
+## Keeping Data Private
+
+The demo above covers action restriction, the first of the two problems. The
+second is privacy, and at the edge it matters more than usual, because these
+devices can see and hear the room they are in. A camera should be able to
+describe a scene without sending raw video off the device, and a microphone
+should be able to drive a conversation while the recording stays local. The
+device should capture only what it needs, and anything sensitive should leave
+only when it has to.
+
+The next piece for Reachy is to add privacy routing. We use OpenShell's
+middleware service, which hooks into OpenShell's sandbox proxy. The middleware service is built to detect
+what type of data is leaving the sandbox. We keep things simple here: if it's image data, we route it to a model approved to
+handle image data, which may be sensitive. If it's not image data, it goes to a frontier model in the cloud. This is just an
+example, but the benefit of middleware is its flexibility. You can build in whatever routing logic you want, and use any tool you
+want for PII redaction or replacement.
+
+Keep an eye on this repo for more details on privacy routing!
+
+
+ 
+
+
+
+---
+
+## Conclusion
+
+None of this is specific to Reachy. The same approach fits inspection robots,
+smart cameras, lab instruments, field vehicles, building gateways, and kiosks. In
+each one, the agent can read sensors and describe what it sees, while anything
+with a real consequence has to be approved by policy first: moving an actuator,
+changing a setpoint, or sending raw data off the device.
+
+The main result from putting this on Reachy is that the whole OpenShell stack,
+the gateway and the sandbox, runs on the device's own computer. The isolation and
+the policy do not depend on the network or a cloud service, so the device keeps
+enforcing its boundary even if the network drops or the model is unreachable.
+Running the full stack on device is what gives complete isolation and control at
+the edge.
+
+This is a repeatable pattern for edge and robotics. Agents are going to run
+everywhere: in homes, factories, vehicles, and hospitals, with more and more of
+them running side by side and working together. You cannot govern that many
+agents, in that many places, by writing a guardrail into each app. It takes one
+deterministic boundary you can put on any device and trust the same way every
+time. That is the pattern we are building with OpenShell.
+
+Key resources:
+
+1. [Onboard Reachy Mini + OpenShell setup](https://github.com/NVIDIA/OpenShell-Research/blob/kirit93/reachy-implementation/projects/reachy-mini-openshell/ONBOARD_SETUP.md)
+2. [Reachy Mini OpenShell project source](https://github.com/NVIDIA/OpenShell-Research/tree/kirit93/reachy-implementation/projects/reachy-mini-openshell)
+3. [Camera-enabled, motion-disabled policy](https://github.com/NVIDIA/OpenShell-Research/blob/kirit93/reachy-implementation/projects/reachy-mini-openshell/openshell/policy-camera-enabled-motion-disabled.yaml)
diff --git a/docs/development/index.md b/docs/development/index.md
index 32e171ec..ae97f624 100644
--- a/docs/development/index.md
+++ b/docs/development/index.md
@@ -28,7 +28,9 @@ defined in `docs/dev-notes/authors.json`. Use a dated filename such as
The renderer uses `categories[0]` as the card topic and `card_tags` as its tags,
falling back to `tags`. An optional `card_variant` must have matching card and
-artwork CSS modifiers in `docs/stylesheets/dev-notes.css`.
+artwork CSS modifiers in `docs/stylesheets/dev-notes.css`. Set `hero_image` to
+an image path relative to the post when its card should use the post's hero
+instead of generated artwork. Hero images must live under `docs/`.
Do not edit content inside these generated marker pairs:
diff --git a/docs/projects/index.md b/docs/projects/index.md
new file mode 100644
index 00000000..facfeb5f
--- /dev/null
+++ b/docs/projects/index.md
@@ -0,0 +1,28 @@
+# Projects
+
+Projects are runnable research demos with their own source code, dependencies,
+setup guide, validation commands, and troubleshooting notes.
+
+## Current Projects
+
+### [Reachy Mini + OpenShell](reachy-mini-openshell/index.md)
+
+A physical-robot and simulator conversation demo with OpenAI Realtime, a
+robot-native audio and camera bridge, fixed REST actions, an optional Gradio
+diagnostic UI, and OpenShell policy enforcement for physical robot tools.
+
+**Want the edge architecture and engineering story?** Read
+[Designing OpenShell for the Edge](../dev-notes/posts/2026-07-20-policy-controlling-reachy-mini-with-openshell.md).
+
+**Building it on a robot?** Follow the
+[project-local onboard setup tutorial](https://github.com/NVIDIA/OpenShell-Research/blob/kirit93/reachy-implementation/projects/reachy-mini-openshell/ONBOARD_SETUP.md).
+
+## Project Page Standard
+
+Each project page should help a new contributor answer five questions quickly:
+
+- What does it run?
+- What do I need installed or configured?
+- Which command starts the local demo?
+- How do I know it is working?
+- Where do I look when it fails?
diff --git a/docs/projects/reachy-mini-openshell-demo-tutorial.md b/docs/projects/reachy-mini-openshell-demo-tutorial.md
new file mode 100644
index 00000000..562df5f0
--- /dev/null
+++ b/docs/projects/reachy-mini-openshell-demo-tutorial.md
@@ -0,0 +1,15 @@
+# Reachy Mini + OpenShell: Dev Note and Setup Tutorial
+
+The original combined guide has been split into two focused documents:
+
+- Read **[Designing OpenShell for the Edge: Why Policy Has to Live Next to the
+ Action](../dev-notes/posts/2026-07-20-policy-controlling-reachy-mini-with-openshell.md)**
+ for the reusable edge architecture, design decisions, challenges, and results.
+- Follow the **[project-local onboard setup
+ tutorial](https://github.com/NVIDIA/OpenShell-Research/blob/kirit93/reachy-implementation/projects/reachy-mini-openshell/ONBOARD_SETUP.md)**
+ for build, installation, validation, policy switching, and troubleshooting
+ commands.
+
+Keeping the tutorial next to the Dockerfile, policies, native controller, and
+application source makes it easier to update the procedure whenever the
+implementation changes.
diff --git a/docs/projects/reachy-mini-openshell-sandbox.md b/docs/projects/reachy-mini-openshell-sandbox.md
new file mode 100644
index 00000000..fb4765c1
--- /dev/null
+++ b/docs/projects/reachy-mini-openshell-sandbox.md
@@ -0,0 +1,9 @@
+# Reachy Mini + OpenShell Setup Tutorial
+
+The canonical tutorial now lives beside the runnable project:
+
+[Open the onboard setup tutorial](https://github.com/NVIDIA/OpenShell-Research/blob/kirit93/reachy-implementation/projects/reachy-mini-openshell/ONBOARD_SETUP.md){ .md-button .md-button--primary }
+
+For the engineering story and reusable edge architecture, read
+[Designing OpenShell for the Edge: Why Policy Has to Live Next to the
+Action](../dev-notes/posts/2026-07-20-policy-controlling-reachy-mini-with-openshell.md).
diff --git a/docs/projects/reachy-mini-openshell/index.md b/docs/projects/reachy-mini-openshell/index.md
new file mode 100644
index 00000000..2751ebdd
--- /dev/null
+++ b/docs/projects/reachy-mini-openshell/index.md
@@ -0,0 +1,57 @@
+# Reachy Mini + OpenShell
+
+This project is a reference implementation for running OpenShell on an edge
+device. A conversation agent runs in an onboard sandbox, trusted native code
+owns the hardware, and local policy mediates the REST calls that can produce
+physical effects on Reachy Mini.
+
+## Start here
+
+Read **[Designing OpenShell for the Edge: Why Policy Has to Live Next to the
+Action](../../dev-notes/posts/2026-07-20-policy-controlling-reachy-mini-with-openshell.md)**
+for the reusable edge architecture, implementation decisions, challenges, and
+demo result.
+
+Follow the **[project-local onboard setup
+tutorial](https://github.com/NVIDIA/OpenShell-Research/blob/kirit93/reachy-implementation/projects/reachy-mini-openshell/ONBOARD_SETUP.md)**
+for the copy-and-run build, installation, policy verification, voice demo, and
+troubleshooting steps.
+
+The tutorial progressively adds:
+
+1. A fixed REST tool transport for `move_head` and `stop_motion`.
+2. An OpenShell sandbox around the AI-facing application.
+3. Direct calls to the Reachy daemon on port `8000`.
+4. REST policy rules that enable or disable `/api/move/goto`.
+5. Browser and command-line tests that make policy decisions visible.
+
+## What runs where
+
+| Component | Location | Responsibility |
+| --- | --- | --- |
+| Reachy daemon | Reachy Mini | Controls the physical motors and exposes the REST API. |
+| OpenShell gateway | Reachy Mini | Creates the onboard sandbox and enforces REST method/path policy. |
+| Conversation application | OpenShell sandbox on Reachy | Runs the Realtime conversation session and fixed REST tools. |
+| Native controller | Reachy App on Reachy | Owns microphone, speaker, and camera; starts/stops the sandbox agent. |
+
+## Project resources
+
+- [Implementation Dev Note](../../dev-notes/posts/2026-07-20-policy-controlling-reachy-mini-with-openshell.md)
+- [Onboard setup and troubleshooting tutorial](https://github.com/NVIDIA/OpenShell-Research/blob/kirit93/reachy-implementation/projects/reachy-mini-openshell/ONBOARD_SETUP.md)
+- [Application source and README](https://github.com/NVIDIA/OpenShell-Research/tree/kirit93/reachy-implementation/projects/reachy-mini-openshell)
+- [Motion-disabled policy](https://github.com/NVIDIA/OpenShell-Research/blob/kirit93/reachy-implementation/projects/reachy-mini-openshell/openshell/policy-motion-disabled.yaml)
+- [Camera-enabled, motion-disabled policy](https://github.com/NVIDIA/OpenShell-Research/blob/kirit93/reachy-implementation/projects/reachy-mini-openshell/openshell/policy-camera-enabled-motion-disabled.yaml)
+- [Head-motion-enabled policy](https://github.com/NVIDIA/OpenShell-Research/blob/kirit93/reachy-implementation/projects/reachy-mini-openshell/openshell/policy-head-motion-enabled.yaml)
+- [Sandbox Dockerfile](https://github.com/NVIDIA/OpenShell-Research/blob/kirit93/reachy-implementation/projects/reachy-mini-openshell/Dockerfile.openshell)
+
+## Two ways to use the project
+
+### Run the application locally
+
+Use the application README for simulator-first development or direct operation
+without an OpenShell sandbox.
+
+### Build the OpenShell policy POC
+
+Use the onboard setup tutorial when you want to demonstrate direct REST
+restrictions, visible policy denials, and the physical Reachy Mini architecture.
diff --git a/docs/stylesheets/dev-notes.css b/docs/stylesheets/dev-notes.css
index 15018b54..08fee185 100644
--- a/docs/stylesheets/dev-notes.css
+++ b/docs/stylesheets/dev-notes.css
@@ -479,6 +479,19 @@ body:has(.openshell-home-page) .md-path {
border-bottom: 1px solid var(--openshell-rule);
}
+.dev-note-card--featured.dev-note-card--has-image .dev-note-card__link {
+ grid-template-columns: 1fr;
+}
+
+.dev-note-card--featured.dev-note-card--has-image .dev-note-card__visual {
+ min-height: 0;
+ aspect-ratio: 1200 / 630;
+}
+
+.dev-note-card--featured.dev-note-card--has-image .dev-note-card__copy {
+ border-left: 1px solid var(--openshell-rule);
+}
+
.dev-note-card__visual {
position: relative;
display: block;
@@ -525,6 +538,26 @@ body:has(.openshell-home-page) .md-path {
transition: transform 400ms ease;
}
+.dev-note-card__visual--image {
+ background: var(--openshell-paper-deep);
+}
+
+.dev-note-card__visual--image::before,
+.dev-note-card__visual--image::after {
+ display: none;
+}
+
+.dev-note-card__visual-image {
+ position: absolute;
+ inset: 0;
+ display: block;
+ width: 100%;
+ height: 100%;
+ object-fit: cover;
+ object-position: center;
+ transition: transform 300ms ease;
+}
+
.dev-note-card__visual--runtime,
.dev-note-card__visual--launch {
background:
@@ -703,6 +736,10 @@ body:has(.openshell-home-page) .md-path {
transform: translate(-50%, -50%) rotate(-27deg);
}
+.dev-note-card__link:hover .dev-note-card__visual-image {
+ transform: scale(1.015);
+}
+
.dev-note-card__link:hover .dev-note-card__read::after {
transform: translate(0.15rem, -0.15rem);
}
@@ -1089,3 +1126,40 @@ body[data-md-color-scheme="slate"] .openshell-home-brand__dark {
scroll-behavior: auto;
}
}
+
+/* Dev note diagrams (SVG figures) */
+.md-typeset .dev-note-figure {
+ width: 100%;
+ max-width: 100%;
+ margin: 1.9rem auto;
+ text-align: center;
+}
+
+.md-typeset .dev-note-figure img {
+ display: block;
+ width: 100%;
+ max-width: 620px;
+ height: auto;
+ margin-inline: auto;
+ border-radius: 14px;
+ border: 1px solid var(--openshell-home-hero-border, rgba(0, 0, 0, 0.08));
+ box-shadow: var(--openshell-card-shadow, 0 0.55rem 1.6rem rgba(15, 23, 42, 0.11));
+ background: #ffffff;
+}
+
+.md-typeset .dev-note-figure--wide img {
+ max-width: 960px;
+}
+
+.md-typeset .dev-note-figure--hero img {
+ max-width: 100%;
+ aspect-ratio: 1200 / 630;
+}
+
+.md-typeset .dev-note-figure figcaption {
+ margin: 0.7rem auto 0;
+ max-width: 42rem;
+ font-size: 0.8rem;
+ line-height: 1.5;
+ color: var(--md-default-fg-color--light);
+}
diff --git a/projects/reachy-mini-openshell/.dockerignore b/projects/reachy-mini-openshell/.dockerignore
new file mode 100644
index 00000000..658bb085
--- /dev/null
+++ b/projects/reachy-mini-openshell/.dockerignore
@@ -0,0 +1,12 @@
+.env
+.venv
+.git
+.pytest_cache
+.ruff_cache
+cache
+captures
+captures-sandbox
+__pycache__
+*.pyc
+.reachy-mcp-token
+.run
diff --git a/projects/reachy-mini-openshell/.env.example b/projects/reachy-mini-openshell/.env.example
index e3aa4715..4afbb521 100644
--- a/projects/reachy-mini-openshell/.env.example
+++ b/projects/reachy-mini-openshell/.env.example
@@ -15,6 +15,13 @@ OPENAI_REALTIME_BASE_URL=https://api.openai.com/v1
OPENAI_REALTIME_MODEL=gpt-realtime-2
OPENAI_REALTIME_VOICE=cedar
+# Camera images are analyzed through a separate allowlist-enforced Responses API route.
+# VISION_API_KEY is optional and falls back to the exported OPENAI_API_KEY.
+# VISION_API_KEY=
+VISION_BASE_URL=https://api.openai.com/v1
+VISION_DEFAULT_MODEL=gpt-5.4-mini
+VISION_ALLOWED_MODELS=gpt-5.4-mini
+
# Optional Riva/local STT cascade backend. Used only when BACKEND_PROVIDER=local_stt.
CHAT_API_KEY=${NVIDIA_INFERENCE_API_KEY}
CHAT_BASE_URL=https://inference-api.nvidia.com/v1
@@ -53,4 +60,30 @@ MIC_TRANSCRIPTION_MAX_AUDIO_MS=12000
# Used only with --local-vision.
LOCAL_VISION_MODEL=HuggingFaceTB/SmolVLM2-2.2B-Instruct
+# Scene-scan MP4 output. Relative paths are resolved from the app's working directory.
+REACHY_CAPTURE_DIR=./captures
+
+# Tool execution mode. REST is the default and exposes only fixed head
+# directions plus stop_motion. Use local only for the legacy SDK development path.
+REACHY_TOOL_TRANSPORT=rest
+REACHY_REST_BASE_URL=http://127.0.0.1:8000
+# Onboard only: uncomment when the trusted native Reachy App snapshot adapter is running.
+# REACHY_CAMERA_BASE_URL=http://host.openshell.internal:8042
+REACHY_REST_TIMEOUT_SECONDS=5
+REACHY_MOTION_DURATION_SECONDS=1
+REACHY_MOTION_POLL_INTERVAL_SECONDS=0.1
+REACHY_MOTION_COMPLETION_TIMEOUT_SECONDS=10
+
+# Headless sandbox audio service used by the native Reachy App controller.
+REACHY_AUDIO_HOST=127.0.0.1
+REACHY_AUDIO_PORT=8765
+REACHY_MODEL_LOGS=1
+REACHY_AGENT_STATE_DIR=/sandbox/run
+REACHY_AGENT_LOG_DIR=/sandbox/logs
+# Cold imports take about 40 seconds on the Reachy Mini onboard Raspberry Pi.
+REACHY_AGENT_START_TIMEOUT_SECONDS=120
+
+# Used only by explicit local mode camera tools.
+REQUIRE_ROUTED_VISION=0
+
HF_HOME=./cache
diff --git a/projects/reachy-mini-openshell/.gitignore b/projects/reachy-mini-openshell/.gitignore
index 2451acc1..cdfea660 100644
--- a/projects/reachy-mini-openshell/.gitignore
+++ b/projects/reachy-mini-openshell/.gitignore
@@ -42,11 +42,13 @@ coverage.xml
*.pem
*.crt
*.csr
+.reachy-mcp-token
# Temporary files
tmp/
*.log
cache/
+captures/
# macOS
.DS_Store
diff --git a/projects/reachy-mini-openshell/Dockerfile.openshell b/projects/reachy-mini-openshell/Dockerfile.openshell
new file mode 100644
index 00000000..15644ec8
--- /dev/null
+++ b/projects/reachy-mini-openshell/Dockerfile.openshell
@@ -0,0 +1,53 @@
+FROM python:3.12-slim-bookworm AS wheel-builder
+
+WORKDIR /build
+COPY requirements-rest.txt pyproject.toml README.md ./
+COPY src ./src
+
+RUN python -m pip install --no-cache-dir --upgrade pip \
+ && python -m pip wheel --no-cache-dir --wheel-dir /wheels -r requirements-rest.txt \
+ && python -m pip wheel --no-cache-dir --no-deps --wheel-dir /wheels .
+
+
+FROM python:3.12-slim-bookworm AS runtime
+
+ENV PATH="/opt/venv/bin:${PATH}" \
+ PYTHONDONTWRITEBYTECODE=1 \
+ PYTHONUNBUFFERED=1
+
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends iproute2 nftables \
+ && rm -rf /var/lib/apt/lists/*
+
+RUN groupadd --gid 1000660000 sandbox \
+ && useradd \
+ --no-log-init \
+ --uid 1000660000 \
+ --gid 1000660000 \
+ --create-home \
+ sandbox
+
+RUN python -m venv /opt/venv
+COPY requirements-rest.txt /tmp/requirements-rest.txt
+RUN --mount=type=bind,from=wheel-builder,source=/wheels,target=/wheels,ro \
+ /opt/venv/bin/pip install --no-cache-dir --no-compile --no-index --find-links=/wheels \
+ -r /tmp/requirements-rest.txt \
+ && /opt/venv/bin/pip check \
+ && /opt/venv/bin/pip install --no-cache-dir --no-compile --no-index --find-links=/wheels \
+ --no-deps reachy_mini_conversation_app \
+ && /opt/venv/bin/python -c \
+ "import reachy_mini_conversation_app.main; import reachy_mini_conversation_app.conversation_stream; import reachy_mini_conversation_app.rest_tool_transport; import reachy_mini_conversation_app.sandbox_audio; import reachy_mini_conversation_app.sandbox_control" \
+ && test -x /opt/venv/bin/reachy-agent-control \
+ && test -x /opt/venv/bin/reachy-mini-sandbox-audio \
+ && rm -rf /opt/venv/lib/python3.12/site-packages/pip /opt/venv/lib/python3.12/site-packages/pip-*.dist-info \
+ && rm -f /tmp/requirements-rest.txt
+
+RUN mkdir -p /sandbox/captures /sandbox/run /sandbox/logs /home/sandbox \
+ && chown -R sandbox:sandbox /sandbox /home/sandbox
+
+WORKDIR /sandbox
+USER sandbox
+
+# Keep the pre-created sandbox alive. The trusted native Reachy App starts and
+# stops the detached conversation process with reachy-agent-control.
+CMD ["/bin/sleep", "infinity"]
diff --git a/projects/reachy-mini-openshell/ONBOARD_SETUP.md b/projects/reachy-mini-openshell/ONBOARD_SETUP.md
new file mode 100644
index 00000000..48aadddf
--- /dev/null
+++ b/projects/reachy-mini-openshell/ONBOARD_SETUP.md
@@ -0,0 +1,762 @@
+# Onboard Reachy Mini + OpenShell Setup
+
+This is the canonical setup guide for running the Reachy Mini conversation app
+on the robot itself. The conversation agent runs inside an OpenShell sandbox;
+a trusted native Reachy App owns microphone, speaker, and one-frame camera
+access; and OpenShell independently allows or denies the REST requests that can
+cause physical actions.
+
+For the implementation story, architecture decisions, and lessons from building
+the demo, read the [Dev Note](https://github.com/NVIDIA/OpenShell-Research/blob/kirit93/reachy-implementation/docs/dev-notes/posts/2026-07-20-policy-controlling-reachy-mini-with-openshell.md).
+
+The commands alternate between a **development machine** and **Reachy**. Build
+the wheel and ARM64 image on the development machine; install and run the final
+artifacts on Reachy.
+
+## What this demonstrates
+
+The model can translate a request such as:
+
+```text
+Reachy, look up and then right.
+```
+
+into the fixed application tool call:
+
+```json
+{"directions": ["up", "right"]}
+```
+
+The application maps those names to fixed head poses and sends one ordered
+`POST /api/move/goto` request at a time. OpenShell decides whether that HTTP
+method and path may leave the sandbox.
+
+For a request such as `Reachy, what do you see?`, the model calls the fixed
+`camera(question)` tool. That tool sends an argument-free
+`POST /camera/capture` request. OpenShell independently decides whether that
+single capture endpoint may leave the sandbox.
+
+## Architecture
+
+```mermaid
+flowchart LR
+ U["Reachy microphone and speaker"] <--> N["Trusted native Reachy App"]
+ N <-->|"PCM audio"| A["Conversation app
OpenShell sandbox"] + A -->|"Realtime WebSocket"| O["OpenAI Realtime"] + A -->|"Motion REST"| P["OpenShell network policy"] + P -->|"host.openshell.internal:8000"| D["Reachy Mini daemon"] + D --> H["Head motors"] + A -->|"POST /camera/capture"| P + P -->|"host.openshell.internal:8042"| N + N --> C["Reachy camera"] +``` + +The sandbox does not start the Reachy SDK, movement manager, camera worker, or +vision router. The trusted native Reachy App owns the SDK media object and only +bridges audio plus one bounded JPEG capture operation. + +## Security boundary + +OpenShell REST rules can match: + +- Calling binary +- Destination host and port +- HTTP method +- URL path +- Query parameters + +OpenShell can therefore allow or deny: + +```text +POST /api/move/goto +POST /camera/capture +``` + +It does not currently enforce arbitrary JSON values inside that REST request. +Once `/api/move/goto` is allowed, OpenShell cannot prove that the body contains +only a head pose or distinguish `up` from `down`. + +The application reduces normal model behavior to fixed values: + +| Direction | Pitch | Yaw | +| --- | ---: | ---: | +| `up` | -30 degrees | 0 degrees | +| `down` | 30 degrees | 0 degrees | +| `left` | 0 degrees | 40 degrees | +| `right` | 0 degrees | -40 degrees | +| `front` | 0 degrees | 0 degrees | + +It also fixes duration to one second, uses `minjerk`, and omits antennas and +body yaw. These body constraints are application validation, not OpenShell +policy enforcement. + +Camera capture has a tighter adapter boundary: the request has no body fields +or query parameters. The native adapter chooses the already-open Reachy camera, +captures one JPEG, never writes it to disk, limits the response to 2 MiB, and +rate-limits calls. OpenShell still enforces the calling binary, host, port, +method, and exact path; the adapter enforces the capture semantics. + +## Requirements and resource budget + +The proven environment was a Reachy Mini Wireless running ARM64 Debian, Reachy +Mini 1.8.3, 3.7 GiB usable RAM, 2 GiB swap, and a 14 GiB root filesystem. + +| Resource | Requirement for this setup | +| --- | --- | +| Development machine | Git, Docker Buildx, Python 3.10–3.12, and `uv` | +| Reachy architecture | `aarch64`; the container is built for `linux/arm64` | +| Reachy RAM | A 4 GB unit is known to work. The sandbox has a 2 GiB ceiling and the host retains the remaining memory. A 2 GB device is unvalidated. | +| Reachy CPU | The sandbox is limited to 2 CPUs. Build the image off-device. | +| Reachy disk | Start with at least 4 GiB free; 5 GiB is preferred. Retain 1–2 GiB free after cleanup. | +| Network | Reachy must reach the configured Realtime API and must be reachable over SSH during installation. | + +The expanded sandbox image is approximately 339 MB. Installation needs more +temporary space because the compressed archive, expanded image, Docker layer +extraction, Docker/OpenShell packages, and `/venvs/apps_venv` may coexist. The +native controller wheel is only about 13 KB, but creating the shared Reachy Apps +environment and installing its Reachy SDK dependencies can require roughly +1–1.5 GiB if that environment does not already exist. + +The model runs remotely, so no local model weights or GPU are required. The +`--memory 2Gi` value used later is a limit, not a claim that the agent +continuously consumes 2 GiB. + +## Prepare the development checkout (development machine) + +```bash +git clone git@github.com:NVIDIA/OpenShell-Research.git +cd OpenShell-Research +git switch kirit93/reachy-implementation +cd projects/reachy-mini-openshell +``` + +Run the relevant checks before building deployable artifacts: + +```bash +uv run ruff check src tests +uv run pytest -q +PYTHONPATH=native-controller/src uv run pytest -q native-controller/tests +``` + +## Verify and prepare Reachy (Reachy) + +Connect and inspect the robot before installing anything: + +```bash +ssh pollen@reachy-mini.local + +uname -m +free -h +df -h / +curl --silent --show-error http://127.0.0.1:8000/api/daemon/status +``` + +Require `aarch64`, a running physical daemon, and at least 4 GiB free. Then +verify Docker and OpenShell: + +```bash +docker --version +sudo systemctl status docker --no-pager +openshell --version +openshell sandbox list +``` + +If either command is missing, install Docker Engine using Docker's current +Debian ARM64 instructions and install OpenShell using the current OpenShell +instructions. Do not copy credentials into the image, policy, or repository. + +Create the provider once on Reachy, then verify it: + +```bash +openshell provider create \ + --name reachy-openai \ + --type openai \ + --from-existing + +openshell provider get reachy-openai +``` + +Finally, verify that containers can reach the host-side Reachy daemon: + +```bash +docker run --rm --add-host host.openshell.internal:host-gateway \ + curlimages/curl:latest \ + http://host.openshell.internal:8000/api/daemon/status +``` + +Do not continue until the daemon reports `state: running`. + +## Run locally before sandboxing + +From the project directory: + +```bash +cp .env.example .env +export OPENAI_API_KEY=sk-... +./scripts/start-local.sh +``` + +The default `.env.example` selects: + +```dotenv +REACHY_TOOL_TRANSPORT=rest +REACHY_REST_BASE_URL=http://127.0.0.1:8000 +``` + +Test these prompts in text mode first: + +```text +Reachy, look up. +Reachy, look front. +Reachy, look left and then right. +Stop moving. +``` + +## REST transport behavior + +The REST transport always advertises these physical tools: + +- `move_head(directions)` +- `stop_motion()` + +When `REACHY_CAMERA_BASE_URL` is configured, it additionally advertises: + +- `camera(question)` + +`move_head` accepts one to eight values from `left`, `right`, `up`, `down`, and +`front`. Extra keys and raw pose values are rejected before a network request is +made. + +Each successful `goto` returns a move UUID. The client polls +`GET /api/move/running` and waits for that UUID to finish before sending the next +direction. A timed-out POST is reported as `unknown_delivery` and is never +automatically retried. + +`stop_motion` lists active move UUIDs and calls `POST /api/move/stop` once for +each one. + +`camera` posts no model-supplied capture settings. It accepts only a short +question, validates the JPEG response, sends the image into the existing +Realtime conversation, and asks the assistant to answer aloud. An OpenShell +`403` becomes `status: policy_denied` and is not retried. + +## OpenShell policies + +Three relevant policies are checked in: + +```text +openshell/policy-motion-disabled.yaml +openshell/policy-camera-enabled-motion-disabled.yaml +openshell/policy-head-motion-enabled.yaml +``` + +All three allow: + +```text +GET /api/daemon/status +GET /api/move/running +POST /api/move/stop +``` + +Only `policy-head-motion-enabled.yaml` allows: + +```text +POST /api/move/goto +``` + +Only `policy-camera-enabled-motion-disabled.yaml` allows: + +```text +POST /camera/capture +``` + +That camera policy still blocks `POST /api/move/goto`. The base +`policy-motion-disabled.yaml` blocks both camera capture and motion start. + +None of the three policies allow `/api/move/set_target`, `/api/motors/**`, +`/api/apps/**`, raw movement WebSockets, wake/sleep, or recorded motions. + +The permitted binary is `/opt/venv/bin/python`. A denial seen with `curl` could +therefore be a binary denial rather than a path denial; use the application or +the same Python executable for final policy tests. + +## Build the deployable artifacts (development machine) + +Build the native Reachy App wheel: + +```bash +uv build --project native-controller + +sha256sum \ + native-controller/dist/reachy_mini_openshell_controller-0.2.0-py3-none-any.whl +``` + +Build the sandbox specifically for Reachy's ARM64 computer: + +```bash +docker buildx build \ + --platform linux/arm64 \ + --load \ + --tag reachy-mini-openshell:rest-arm64 \ + --file Dockerfile.openshell \ + . + +docker image inspect reachy-mini-openshell:rest-arm64 \ + --format 'architecture={{.Architecture}} size={{.Size}} cmd={{json .Config.Cmd}}' +``` + +Require `architecture=arm64` and +`cmd=["/bin/sleep","infinity"]`. The persistent command keeps the pre-created +sandbox alive; the native Reachy App starts and stops the agent process inside +it. + +### Why this image is small + +The image is custom-built rather than copying the complete Reachy environment: + +- Both stages start from `python:3.12-slim-bookworm`. +- A disposable builder stage constructs wheels; build files do not enter the + runtime stage. +- `requirements-rest.txt` contains the sandbox's REST, Realtime, and audio + requirements. +- The application wheel is installed with `--no-deps`, preventing its normal + Reachy SDK, MuJoCo, OpenCV, simulator, dance, camera-worker, and local vision + dependencies from entering the sandbox. +- Pip caches, install-time bytecode, runtime `pip`, and APT metadata are + removed. +- Debian packages use `--no-install-recommends`. + +The image deliberately retains `iproute2` and `nftables`. OpenShell needs the +trusted `ip` helper to create the isolated sandbox network, so removing it to +save a few megabytes breaks provisioning. The tested image was 339,495,096 +bytes according to Docker. + +Export and compress the image for transfer: + +```bash +docker save reachy-mini-openshell:rest-arm64 \ + | gzip > reachy-mini-openshell-rest-arm64.tar.gz + +sha256sum reachy-mini-openshell-rest-arm64.tar.gz +``` + +## Transfer and load the artifacts + +Copy the image, controller wheel, and main demo policy from the development +machine: + +```bash +scp reachy-mini-openshell-rest-arm64.tar.gz \ + pollen@reachy-mini.local:/home/pollen/ + +scp native-controller/dist/reachy_mini_openshell_controller-0.2.0-py3-none-any.whl \ + pollen@reachy-mini.local:/home/pollen/ + +scp openshell/policy-camera-enabled-motion-disabled.yaml \ + openshell/policy-motion-disabled.yaml \ + openshell/policy-head-motion-enabled.yaml \ + pollen@reachy-mini.local:/home/pollen/ +``` + +On Reachy, compare the received checksums with the development machine, then +load and inspect the image: + +```bash +df -h / +docker system df +docker load --input ~/reachy-mini-openshell-rest-arm64.tar.gz + +docker image inspect reachy-mini-openshell:rest-arm64 \ + --format 'architecture={{.Architecture}} size={{.Size}} cmd={{json .Config.Cmd}}' +``` + +`docker load` may print nothing for several minutes on microSD storage. In a +second SSH session, check `ps`, `df -h /`, and `journalctl -u docker` before +assuming it has stalled. After a verified load, the transferred `.tar.gz` can +be deleted to recover space. + +Put policies in a stable operator-owned directory: + +```bash +mkdir -p ~/reachy-openshell +cp ~/policy-*.yaml ~/reachy-openshell/ +``` + +## Create the sandbox once (Reachy) + +Create an idle sandbox with camera enabled and motion disabled: + +```bash +openshell sandbox create \ + --name reachy-agent \ + --from reachy-mini-openshell:rest-arm64 \ + --policy ~/reachy-openshell/policy-camera-enabled-motion-disabled.yaml \ + --provider reachy-openai \ + --cpu 2 \ + --memory 2Gi \ + --env REACHY_MINI_SKIP_DOTENV=1 \ + --env BACKEND_PROVIDER=openai_realtime \ + --env REACHY_TOOL_TRANSPORT=rest \ + --env REACHY_REST_BASE_URL=http://host.openshell.internal:8000 \ + --env REACHY_CAMERA_BASE_URL=http://host.openshell.internal:8042 \ + --env REACHY_REST_TIMEOUT_SECONDS=5 \ + --env REACHY_MOTION_DURATION_SECONDS=1 \ + --env REACHY_MOTION_POLL_INTERVAL_SECONDS=0.1 \ + --env REACHY_MOTION_COMPLETION_TIMEOUT_SECONDS=10 \ + --env REACHY_AUDIO_HOST=127.0.0.1 \ + --env REACHY_AUDIO_PORT=8765 \ + --env REACHY_AGENT_START_TIMEOUT_SECONDS=120 \ + --env REACHY_MODEL_LOGS=1 \ + --env OPENAI_REALTIME_BASE_URL=https://api.openai.com/v1 \ + --env OPENAI_REALTIME_MODEL=gpt-realtime-2 \ + --env OPENAI_REALTIME_VOICE=cedar +``` + +If the CLI enters an interactive sandbox prompt after creation, type `exit`. +That exits only the shell; the image's `sleep infinity` command keeps the +sandbox alive. + +Check that it is ready: + +```bash +openshell sandbox get reachy-agent +``` + +Require `Phase: Ready`. Read the printed policy and confirm that it allows +`POST host.openshell.internal:8042/camera/capture` but has no allow rule for +`POST host.openshell.internal:8000/api/move/goto`. + +The 120-second agent startup window is intentional. A cold import of the audio +stack takes about 40 seconds on the Reachy Mini onboard Raspberry Pi. + +## Robot-native media and lifecycle + +The normal onboard path uses a small trusted Reachy App from +`projects/reachy-mini-openshell/native-controller`. It owns only the robot +microphone, speaker, camera snapshot adapter, and fixed OpenShell lifecycle +commands. The model, tools, and every requested action remain inside +`reachy-agent`. + +Before installing the native app, test the inner lifecycle directly. A cold +start took approximately 41 seconds on the tested robot: + +```bash +time openshell sandbox exec \ + --name reachy-agent \ + --no-tty \ + -- \ + /usr/bin/env REACHY_AGENT_START_TIMEOUT_SECONDS=120 \ + /opt/venv/bin/reachy-agent-control start + +openshell sandbox exec --name reachy-agent --no-tty -- \ + /opt/venv/bin/reachy-agent-control status + +openshell sandbox exec --name reachy-agent --no-tty -- \ + tail -n 100 /sandbox/logs/reachy-agent.log +``` + +Require `running` and `Application startup complete` before exposing the audio +service. + +### Expose the sandbox audio listener + +The agent listens on `127.0.0.1:8765` **inside the sandbox**. Sandbox loopback +is intentionally private, so the native Reachy App on the host cannot connect +to that address directly. Exposing the service creates a local OpenShell gateway +route from `reachy-agent--audio.openshell.localhost:17670` to the sandbox +listener: + +```bash +openshell service expose reachy-agent 8765 audio +openshell service get reachy-agent audio + +curl --silent --show-error \ + http://reachy-agent--audio.openshell.localhost:17670/health +``` + +Expected output includes: + +```json +{ + "status": "ok", + "active_audio_client": false, + "format": "pcm_s16le", + "sample_rate": 16000, + "channels": 1 +} +``` + +`active_audio_client: false` is correct until the native app connects Reachy's +microphone and speaker. The route is local to the onboard OpenShell gateway; it +does not publish the audio service to the internet or directly to the robot's +LAN. + +The controller's WebSocket URI is: + +```text +ws://reachy-agent--audio.openshell.localhost:17670/audio +``` + +The service accepts one client, mono signed 16-bit PCM at 16 kHz, and exposes +`GET /health` plus `WS /audio`. The Reachy App reads and plays audio through the +SDK media manager. No laptop, browser, Gradio page, or SSH tunnel is required +after installation. + +The same native app serves only this camera operation on the robot host: + +```text +POST http://127.0.0.1:8042/camera/capture +``` + +The sandbox addresses it as +`http://host.openshell.internal:8042/camera/capture`, so the call crosses the +OpenShell REST policy. It is not exposed as an OpenShell browser service. + +When the Reachy App is stopped it runs: + +```bash +openshell sandbox exec --name reachy-agent --no-tty -- \ + /opt/venv/bin/reachy-agent-control stop +``` + +The sandbox, provider, service endpoint, and policy remain provisioned for the +next Start. + +Stop the manually started agent before testing the native lifecycle: + +```bash +openshell sandbox exec --name reachy-agent --no-tty -- \ + /opt/venv/bin/reachy-agent-control stop +``` + +### Install the native Reachy App (Reachy) + +Reachy's application installer does not accept an arbitrary local wheel. Install +the controller into the daemon's shared application environment instead. + +Check whether that environment already exists: + +```bash +ls -l /venvs/apps_venv/bin/python +``` + +If it does not exist, create it with the same Python generation as the Reachy +daemon and install the matching Reachy SDK: + +```bash +/opt/uv/uv venv \ + --python /venvs/mini_daemon/bin/python \ + /venvs/apps_venv + +/opt/uv/uv pip install \ + --no-cache \ + --python /venvs/apps_venv/bin/python \ + 'reachy-mini==1.8.3' +``` + +Use the daemon's actual Reachy Mini version instead of `1.8.3` if it differs: + +```bash +/venvs/mini_daemon/bin/python -c \ + 'import importlib.metadata as m; print(m.version("reachy-mini"))' +``` + +Install the controller wheel that was transferred earlier: + +```bash +/opt/uv/uv pip install --no-cache \ + --python /venvs/apps_venv/bin/python \ + /home/pollen/reachy_mini_openshell_controller-0.2.0-py3-none-any.whl +``` + +The `/api/apps/install` endpoint intentionally does not accept `source_kind: +local`; that endpoint installs catalog/Hugging Face apps. After the manual pip +install, `reachy_mini_openshell_controller` appears in the installed app list +and can be started from the Reachy Apps UI. The daemon launches the controller +in its shared apps environment and supplies the local `ReachyMini` media object. + +Verify the entry point and installed version: + +```bash +/venvs/apps_venv/bin/python -c \ + 'import importlib.metadata as m; print(m.version("reachy-mini-openshell-controller")); print([ep.name for ep in m.entry_points(group="reachy_mini_apps") if "openshell" in ep.name])' +``` + +### Start and verify the native app (Reachy) + +Start it from the Reachy Apps UI, or call the same daemon endpoint: + +```bash +curl --silent --show-error -X POST \ + http://127.0.0.1:8000/api/apps/start-app/reachy_mini_openshell_controller +``` + +Verify all three layers: + +```bash +curl --silent --show-error \ + http://127.0.0.1:8000/api/apps/current-app-status + +openshell sandbox exec --name reachy-agent --no-tty -- \ + /opt/venv/bin/reachy-agent-control status + +openshell service get reachy-agent audio + +curl --silent --show-error \ + http://reachy-agent--audio.openshell.localhost:17670/health +``` + +Require the Reachy App state `running`, inner agent state `running`, an `audio` +service targeting `127.0.0.1:8765`, and a healthy response. Once the native app +connects, `active_audio_client` should become `true`. + +Inspect the inner log when troubleshooting: + +```bash +openshell sandbox exec --name reachy-agent --no-tty -- \ + tail -n 100 /sandbox/logs/reachy-agent.log +``` + +The project README retains an optional Gradio diagnostic path for testing the +model and tool flow independently of robot media. It is not part of normal +onboard operation. + +## Verify the denied action + +With `policy-camera-enabled-motion-disabled.yaml` active, say: + +```text +Reachy, look up. +``` + +Expected behavior: + +1. The model selects `move_head`. +2. The app attempts `POST /api/move/goto`. +3. OpenShell returns HTTP `403`. +4. The tool result has `status: policy_denied`. +5. Reachy does not move. +6. The assistant explains that policy blocked the action and does not retry. + +## Verify camera allow and deny + +With `policy-camera-enabled-motion-disabled.yaml` active, say: + +```text +Reachy, take a picture and tell me what you see. +``` + +Expected behavior: the model selects `camera`, OpenShell permits only +`POST /camera/capture`, one JPEG is delivered to the Realtime session, and +Reachy answers aloud. Asking `Reachy, look up` remains denied. + +To prove the camera boundary, hot-reload the policy that blocks both camera and +motion: + +```bash +openshell policy set reachy-agent \ + --policy ~/reachy-openshell/policy-motion-disabled.yaml \ + --wait +``` + +Ask the picture question again. Reachy must not capture a frame and should +explain that policy blocked the action. Restore the main demo policy afterward: + +```bash +openshell policy set reachy-agent \ + --policy ~/reachy-openshell/policy-camera-enabled-motion-disabled.yaml \ + --wait +``` + +Inspect logs: + +```bash +openshell logs reachy-agent --tail +``` + +## Enable head motion without restarting + +Hot-reload the enabled policy: + +```bash +openshell policy set reachy-agent \ + --policy ~/reachy-openshell/policy-head-motion-enabled.yaml \ + --wait +``` + +Repeat the same request. Reachy should now move through the fixed application +pose. + +Return to the restrictive policy: + +```bash +openshell policy set reachy-agent \ + --policy ~/reachy-openshell/policy-camera-enabled-motion-disabled.yaml \ + --wait +``` + +## Negative policy tests + +Use the permitted Python binary inside the sandbox to test a dangerous path: + +```bash +openshell sandbox exec -n reachy-agent -- \ + /opt/venv/bin/python -c \ + 'import httpx; print(httpx.post("http://host.openshell.internal:8000/api/move/set_target", json={}).status_code)' +``` + +Expected result: `403`. + +Repeat for a motor path or app-management path. Those requests must remain +denied under both policies. + +## Normal operation + +After provisioning, normal use does not require SSH commands: + +1. Start `reachy_mini_openshell_controller` from the Reachy Apps UI. +2. Wait for the app to report `running`; the first cold start may take about 40 + seconds. +3. Speak directly to Reachy. +4. Stop the app from the Reachy Apps UI when finished. + +Stopping the app closes the media bridge and stops the inner conversation +process. It does not delete the sandbox, provider, policy, or audio-service +definition. + +## Troubleshooting quick reference + +| Symptom | First checks | +| --- | --- | +| `docker load` is silent | In another SSH session run `ps -eo pid,etime,stat,%cpu,%mem,cmd \| grep '[d]ocker load'`, `df -h /`, and `sudo journalctl -u docker -n 30 --no-pager`. Slow microSD extraction is normal. | +| Sandbox enters `ContainerRestarting` | Inspect the image command. It must be `sleep infinity`; do not append `/bin/true` to `sandbox create`. | +| `reachy-agent failed to become healthy` | Retry with the 120-second timeout and inspect `/sandbox/logs/reachy-agent.log`. A cold start took about 41 seconds. | +| Audio URL says `Service endpoint is not available` | Confirm the inner agent is `running`, then recreate or expose `audio` and check `/health`. The target listener must exist before the route is usable. | +| Audio bridge reports `Name or service not known` | Verify controller version `0.2.0`. It preserves the virtual routing hostname while connecting the socket to `127.0.0.1:17670`. | +| Reachy says it cannot take a picture | Test `POST http://127.0.0.1:8042/camera/capture`, then test the same path from the sandbox with `/opt/venv/bin/python`. If both work, inspect the model/tool logs and session instructions. | +| Local wheel install is rejected by `/api/apps/install` | This endpoint does not accept `source_kind: local`; install the wheel into `/venvs/apps_venv` with `/opt/uv/uv pip install`. | +| Root filesystem is almost full | Run `docker system df` and `sudo du -sh /var/lib/docker /venvs/apps_venv`. Remove transferred archives, obsolete images, failed layers, and old wheel versions, but not the active image or `/venvs/mini_daemon`. | + +## Development checks + +```bash +uv run ruff check src tests +uv run pytest -q +PYTHONPATH=native-controller/src uv run pytest -q native-controller/tests +``` + +The unit suite covers fixed schemas, pose mapping, argument rejection, ordered +movement, stop behavior, OpenShell `403` conversion, no retry after an uncertain +motion POST, native JPEG limits, and exact camera-policy rules. + +## Completion criteria + +- The sandbox starts no local robot SDK or camera workers. +- Only fixed head directions, stop, and the optional one-frame camera tool are + model-visible robot actions. +- Motion-disabled policy blocks `goto` while preserving stop. +- Camera-enabled/motion-disabled policy permits only one fixed capture endpoint + and still blocks `goto`. +- Motion-enabled policy allows `goto` but no raw target or motor endpoints. +- Policy can be hot-reloaded without recreating the sandbox. +- Documentation states that JSON body values remain application-enforced. diff --git a/projects/reachy-mini-openshell/README.md b/projects/reachy-mini-openshell/README.md index f9384131..a7dec086 100644 --- a/projects/reachy-mini-openshell/README.md +++ b/projects/reachy-mini-openshell/README.md @@ -1,8 +1,13 @@ # Reachy OpenShell -Reachy Mini conversation demo for OpenShell: Gradio UI, simulator support, -microphone or text input, Reachy movement tools, and selectable model backends. -The default and preferred starting point is OpenAI Realtime. +Reachy Mini conversation demo for OpenShell: native robot microphone, speaker, +and single-frame camera capture; optional Gradio input; OpenAI Realtime; and a +deliberately small REST-controlled action surface. + +> **Building the OpenShell policy demo with a physical Reachy?** Follow the +> [onboard setup tutorial](ONBOARD_SETUP.md). For the architecture, implementation +> decisions, and lessons learned, read the +> [Dev Note](../../docs/dev-notes/posts/2026-07-20-policy-controlling-reachy-mini-with-openshell.md). Commands: @@ -10,6 +15,9 @@ Commands: - app: `reachy-mini-conversation-app` - module: `python -m reachy_mini_conversation_app` - check: `reachy-mini-backend-check` +- sandbox audio: `reachy-mini-sandbox-audio` +- sandbox lifecycle: `reachy-agent-control start|stop|status` +- native Reachy App: `native-controller/` ## Quick Start @@ -36,13 +44,18 @@ The launcher creates `.venv`, runs `uv sync`, validates `.env`, starts `reachy-mini-daemon --sim`, then prints the Gradio URL:`: connect to a matching daemon robot name
+- `--tool-transport rest`: fixed direct REST action tools; this is the default
- `--debug`: debug logging
-- `--local-vision`: local vision model; requires `local_vision`
-- `--head-tracker yolo`: YOLO head tracking; requires `yolo_vision`
-- `--head-tracker mediapipe`: MediaPipe head tracking; requires
- `mediapipe_vision`
+- `--tool-transport local`: legacy in-process SDK development mode
+- `--no-camera`, `--robot-name`, `--local-vision`, and `--head-tracker`: legacy
+ local-mode options; the REST snapshot tool is controlled only by
+ `REACHY_CAMERA_BASE_URL`
## Customize Reachy
@@ -295,17 +399,22 @@ src/reachy_mini_conversation_app/profiles/_reachy_mini_conversation_app_locked_p
- `tools.txt`: allowed profile tools
- `*.py`: profile-local tool implementations
-Current profile tools:
+REST-mode model tools:
```text
-dance
-stop_dance
-play_emotion
-stop_emotion
-sweep_look
+move_head
+stop_motion
+do_nothing
+task_status
+task_cancel
```
-## Optional Vision Extras
+The REST transport supplies the two physical tools. Only `do_nothing` and the
+task-management helpers remain local. The files in the locked profile still
+support explicit legacy local-mode development, but those extra physical tools
+are not merged into a REST-mode model session.
+
+## Legacy Local-Mode Vision Extras
The default install includes the MuJoCo simulator backend. There are no
project-level `backend` or `sim` extras.
@@ -348,7 +457,7 @@ uv run reachy-mini-app-assistant check .
| Riva ASR readiness fails | Check `http://:9000/v1/health/ready`, GPU/container logs, and that the app can reach the host from macOS. |
| vLLM STT says audio support is missing | Redeploy the service with vLLM audio support, then rerun `stt-probe`. |
| `uv sync` builds `pygobject` or `pycairo` on macOS | Run `uv cache clean reachy-mini pygobject pycairo`, then `uv sync`. |
-| Daemon uses `--no-media` | Start the app with `--no-camera`; the launcher already does this. |
+| REST motion returns `policy_denied` | Inspect the active OpenShell policy and allow `POST /api/move/goto` only when motion should be enabled. |
The checked-in uv resolution targets macOS/Darwin. For Linux deployment,
update `[tool.uv].environments` and regenerate `uv.lock`.
diff --git a/projects/reachy-mini-openshell/native-controller/README.md b/projects/reachy-mini-openshell/native-controller/README.md
new file mode 100644
index 00000000..3a327172
--- /dev/null
+++ b/projects/reachy-mini-openshell/native-controller/README.md
@@ -0,0 +1,99 @@
+# Reachy Mini OpenShell Controller
+
+This is the small trusted Reachy App that owns the robot microphone, speaker,
+and camera; starts and stops the conversation process inside the existing
+`reachy-agent` OpenShell sandbox; bridges PCM audio over a loopback WebSocket;
+and exposes one bounded snapshot endpoint to the sandbox.
+
+It intentionally contains no model client, tool implementation, or robot motion
+call. Model-driven requests remain inside the OpenShell sandbox and reach the
+robot only through policy-controlled REST requests.
+
+The trusted camera surface is exactly `POST /camera/capture` on port `8042`.
+It accepts no request arguments, does not write a file, returns one JPEG no
+larger than 2 MiB, permits one capture per second, and rejects concurrent
+captures. The Reachy App process—not the sandbox—owns the camera SDK object.
+
+## Runtime contract
+
+Before this app starts, the Reachy host must already have:
+
+- a working `openshell` CLI and local gateway;
+- a `Ready` sandbox named `reachy-agent`;
+- `/opt/venv/bin/reachy-agent-control` in that sandbox image; and
+- the motion-disabled policy attached to the sandbox.
+
+Start performs only these fixed actions:
+
+1. `openshell sandbox get reachy-agent`
+2. `openshell sandbox exec ... reachy-agent-control start`
+3. `openshell service get reachy-agent audio`
+4. `openshell service expose reachy-agent 8765 audio` when missing, after the listener is live
+5. connect robot audio to the loopback WebSocket service
+6. serve the fixed camera capture route for the Reachy App lifetime
+
+The lifecycle command verifies readiness by inspecting the sandbox's Linux TCP
+listener table. It does not make an HTTP request to `127.0.0.1`, because
+OpenShell intentionally blocks sandbox egress to loopback even though the
+gateway can forward an explicitly exposed service to a loopback listener.
+The sandbox configuration gives the first cold start 120 seconds; measured
+startup on the Reachy Mini onboard Raspberry Pi is approximately 40 seconds.
+
+Stop closes robot media and the camera adapter, then invokes
+`reachy-agent-control stop`. It never
+creates or deletes a sandbox and never accepts a command from the model.
+
+## Configuration
+
+Defaults are suitable for the documented onboard deployment. Supported
+overrides are:
+
+| Variable | Default |
+| --- | --- |
+| `REACHY_OPENSHELL_BIN` | discovered from `PATH` and standard `pollen` locations |
+| `REACHY_OPENSHELL_SANDBOX` | `reachy-agent` |
+| `REACHY_OPENSHELL_AUDIO_SERVICE` | `audio` |
+| `REACHY_OPENSHELL_AUDIO_PORT` | `8765` |
+| `REACHY_OPENSHELL_GATEWAY_PORT` | `17670` |
+| `REACHY_OPENSHELL_COMMAND_TIMEOUT_SECONDS` | `150` |
+
+The app process must run as a user that can read the OpenShell gateway
+configuration. On the standard Reachy image this is expected to be `pollen`,
+but verify it during device installation.
+
+## Development checks
+
+From the parent project:
+
+```bash
+PYTHONPATH=native-controller/src uv run pytest -q native-controller/tests
+uv run ruff check native-controller/src native-controller/tests
+```
+
+Build the installable package without installing the sandbox dependencies:
+
+```bash
+uv build --project native-controller
+```
+
+For a local Wireless robot install, copy the resulting universal wheel to
+Reachy and install it in the daemon's shared app environment. This is Pollen's
+documented manual-deployment path:
+
+```bash
+/opt/uv/uv pip install --no-cache \
+ --python /venvs/apps_venv/bin/python \
+ /home/pollen/reachy_mini_openshell_controller-0.2.0-py3-none-any.whl
+```
+
+The daemon API intentionally rejects `source_kind: local`; its install endpoint
+is for catalog/Hugging Face apps. The wheel's `reachy_mini_apps` entry point
+makes it discoverable after the manual install. It appears as
+`reachy_mini_openshell_controller` and can then be started/stopped from the
+Reachy dashboard. The same lifecycle can be exercised directly with:
+
+```bash
+curl -X POST \
+ http://127.0.0.1:8000/api/apps/start-app/reachy_mini_openshell_controller
+curl -X POST http://127.0.0.1:8000/api/apps/stop-current-app
+```
diff --git a/projects/reachy-mini-openshell/native-controller/pyproject.toml b/projects/reachy-mini-openshell/native-controller/pyproject.toml
new file mode 100644
index 00000000..a5bd1990
--- /dev/null
+++ b/projects/reachy-mini-openshell/native-controller/pyproject.toml
@@ -0,0 +1,34 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "reachy-mini-openshell-controller"
+version = "0.2.0"
+description = "Trusted Reachy Mini media and OpenShell lifecycle controller."
+readme = "README.md"
+requires-python = ">=3.10,<3.13"
+dependencies = [
+ "fastapi>=0.115,<1",
+ "numpy>=1.26,<3",
+ "pillow>=10,<13",
+ "reachy-mini>=1.8,<1.9",
+ "uvicorn>=0.34,<1",
+ "websockets>=15,<16",
+]
+
+[project.entry-points.reachy_mini_apps]
+reachy_mini_openshell_controller = "reachy_mini_openshell_controller.app:ReachyOpenShellApp"
+
+[tool.setuptools.package-dir]
+"" = "src"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.ruff]
+line-length = 119
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "I", "C4", "D"]
+ignore = ["D203", "D213"]
diff --git a/projects/reachy-mini-openshell/native-controller/src/reachy_mini_openshell_controller/__init__.py b/projects/reachy-mini-openshell/native-controller/src/reachy_mini_openshell_controller/__init__.py
new file mode 100644
index 00000000..1e4d1294
--- /dev/null
+++ b/projects/reachy-mini-openshell/native-controller/src/reachy_mini_openshell_controller/__init__.py
@@ -0,0 +1,5 @@
+"""Trusted native controller for the Reachy Mini OpenShell agent."""
+
+from reachy_mini_openshell_controller.settings import ControllerSettings
+
+__all__ = ["ControllerSettings"]
diff --git a/projects/reachy-mini-openshell/native-controller/src/reachy_mini_openshell_controller/app.py b/projects/reachy-mini-openshell/native-controller/src/reachy_mini_openshell_controller/app.py
new file mode 100644
index 00000000..da453f6b
--- /dev/null
+++ b/projects/reachy-mini-openshell/native-controller/src/reachy_mini_openshell_controller/app.py
@@ -0,0 +1,57 @@
+"""Reachy Mini Apps lifecycle integration for the OpenShell agent."""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+from threading import Event
+from typing import Any
+
+from reachy_mini import ReachyMini, ReachyMiniApp
+
+from reachy_mini_openshell_controller.bridge import NativeAudioBridge
+from reachy_mini_openshell_controller.camera_adapter import TrustedCameraAdapter
+from reachy_mini_openshell_controller.openshell import OpenShellController
+from reachy_mini_openshell_controller.settings import ControllerSettings
+
+logger = logging.getLogger(__name__)
+
+
+class ReachyOpenShellApp(ReachyMiniApp): # type: ignore[misc]
+ """Start the sandbox agent and bridge audio for the lifetime of the Reachy App."""
+
+ custom_app_url = "http://0.0.0.0:8042"
+ dont_start_webserver = False
+
+ def __init__(self, *args: Any, **kwargs: Any) -> None:
+ """Initialize the native app and its fixed trusted camera route."""
+ super().__init__(*args, **kwargs)
+ self._media: Any | None = None
+ if self.settings_app is None:
+ raise RuntimeError("Reachy camera adapter web server was not initialized")
+ self._camera_adapter = TrustedCameraAdapter(lambda: self._media)
+ self._camera_adapter.register(self.settings_app)
+
+ def run(self, reachy_mini: ReachyMini, stop_event: Event) -> None:
+ """Start the sandbox agent and bridge audio until stopped."""
+ logging.basicConfig(
+ level=logging.INFO,
+ format="%(asctime)s %(levelname)s %(name)s | %(message)s",
+ )
+ settings = ControllerSettings.from_environment()
+ controller = OpenShellController(settings)
+ self._media = reachy_mini.media
+ try:
+ controller.start_agent()
+ asyncio.run(NativeAudioBridge(reachy_mini.media, settings).run(stop_event))
+ finally:
+ self._media = None
+ controller.stop_agent()
+
+
+if __name__ == "__main__":
+ app = ReachyOpenShellApp()
+ try:
+ app.wrapped_run()
+ except KeyboardInterrupt:
+ app.stop()
diff --git a/projects/reachy-mini-openshell/native-controller/src/reachy_mini_openshell_controller/audio.py b/projects/reachy-mini-openshell/native-controller/src/reachy_mini_openshell_controller/audio.py
new file mode 100644
index 00000000..5ab4b281
--- /dev/null
+++ b/projects/reachy-mini-openshell/native-controller/src/reachy_mini_openshell_controller/audio.py
@@ -0,0 +1,62 @@
+"""Small NumPy-only codec helpers for robot PCM audio."""
+
+from __future__ import annotations
+
+import numpy as np
+from numpy.typing import NDArray
+
+WIRE_SAMPLE_RATE = 16_000
+
+
+def first_mono_channel(audio: NDArray[np.generic]) -> NDArray[np.generic]:
+ """Select one channel from common frame-first and channel-first layouts."""
+ if audio.ndim == 1:
+ return audio
+ if audio.ndim != 2:
+ raise ValueError(f"Expected 1D or 2D audio, got {audio.shape}")
+ rows, columns = audio.shape
+ if rows == 1:
+ return audio[0]
+ if columns == 1:
+ return audio[:, 0]
+ if rows in {2, 6, 8} and columns > rows:
+ return audio[0]
+ return audio[:, 0]
+
+
+def resample_linear(audio: NDArray[np.float32], source_rate: int, target_rate: int) -> NDArray[np.float32]:
+ """Resample a mono float32 frame without pulling SciPy into the native app."""
+ if source_rate <= 0 or target_rate <= 0:
+ raise ValueError("sample rates must be positive")
+ if source_rate == target_rate or audio.size == 0:
+ return audio.astype(np.float32, copy=False)
+ target_length = max(1, round(audio.size * target_rate / source_rate))
+ source_positions = np.linspace(0.0, 1.0, num=audio.size, endpoint=False)
+ target_positions = np.linspace(0.0, 1.0, num=target_length, endpoint=False)
+ return np.interp(target_positions, source_positions, audio).astype(np.float32)
+
+
+def encode_robot_audio(audio: NDArray[np.generic], source_rate: int) -> bytes:
+ """Convert Reachy float/int audio to mono 16 kHz signed PCM."""
+ mono = first_mono_channel(np.asarray(audio))
+ if np.issubdtype(mono.dtype, np.floating):
+ normalized = np.nan_to_num(mono, nan=0.0, posinf=1.0, neginf=-1.0).astype(np.float32)
+ max_abs = float(np.max(np.abs(normalized))) if normalized.size else 0.0
+ if max_abs > 1.0:
+ normalized = normalized / max_abs
+ elif np.issubdtype(mono.dtype, np.integer):
+ normalized = mono.astype(np.float32) / 32768.0
+ else:
+ raise TypeError(f"Unsupported robot audio dtype {mono.dtype}")
+ resampled = resample_linear(normalized, source_rate, WIRE_SAMPLE_RATE)
+ pcm = np.clip(resampled, -1.0, 1.0) * 32767.0
+ return pcm.astype(" NDArray[np.float32]:
+ """Convert mono 16 kHz signed PCM into Reachy float32 playback audio."""
+ if len(payload) % 2:
+ raise ValueError("PCM payload must contain complete int16 samples")
+ pcm = np.frombuffer(payload, dtype=" None:
+ """Initialize the bridge around Reachy's media manager."""
+ self.media = media
+ self.settings = settings
+ self._mute_microphone_until = 0.0
+
+ async def run(self, stop_event: Event) -> None:
+ """Run audio until the Reachy App lifecycle requests a stop."""
+ self.media.start_recording()
+ self.media.start_playing()
+ await asyncio.sleep(1.0)
+ input_rate = int(self.media.get_input_audio_samplerate())
+ output_rate = int(self.media.get_output_audio_samplerate())
+ reconnect_delay = self.settings.reconnect_initial_seconds
+ try:
+ while not stop_event.is_set():
+ try:
+ async with connect(
+ self.settings.audio_websocket_url,
+ host=self.settings.gateway_connect_host,
+ port=self.settings.gateway_port,
+ proxy=None,
+ max_size=4 * 1024 * 1024,
+ ping_interval=20,
+ ping_timeout=20,
+ ) as websocket:
+ await websocket.send(
+ json.dumps(
+ {
+ "type": "hello",
+ "format": "pcm_s16le",
+ "sample_rate": WIRE_SAMPLE_RATE,
+ "channels": 1,
+ }
+ )
+ )
+ ready = await asyncio.wait_for(websocket.recv(), timeout=10.0)
+ if not isinstance(ready, str) or json.loads(ready).get("type") != "ready":
+ raise RuntimeError("sandbox audio service did not return ready")
+ logger.info("Connected Reachy audio to %s", self.settings.audio_websocket_url)
+ reconnect_delay = self.settings.reconnect_initial_seconds
+ await self._run_session(websocket, stop_event, input_rate, output_rate)
+ except asyncio.CancelledError:
+ raise
+ except Exception as exc:
+ if stop_event.is_set():
+ break
+ logger.warning("Audio bridge disconnected (%s); reconnecting in %.1fs", exc, reconnect_delay)
+ await self._wait_or_stop(stop_event, reconnect_delay)
+ reconnect_delay = min(reconnect_delay * 2, self.settings.reconnect_max_seconds)
+ finally:
+ self._safe_media_call("stop_recording")
+ self._safe_media_call("stop_playing")
+
+ async def _run_session(self, websocket: Any, stop_event: Event, input_rate: int, output_rate: int) -> None:
+ record_task = asyncio.create_task(
+ self._record_loop(websocket, stop_event, input_rate),
+ name="reachy-microphone",
+ )
+ play_task = asyncio.create_task(
+ self._play_loop(websocket, stop_event, output_rate),
+ name="reachy-speaker",
+ )
+ stop_task = asyncio.create_task(self._wait_or_stop(stop_event, None), name="reachy-stop-event")
+ done, pending = await asyncio.wait(
+ {record_task, play_task, stop_task},
+ return_when=asyncio.FIRST_COMPLETED,
+ )
+ for task in pending:
+ task.cancel()
+ await asyncio.gather(*pending, return_exceptions=True)
+ for task in done:
+ if task is not stop_task:
+ task.result()
+
+ async def _record_loop(self, websocket: Any, stop_event: Event, input_rate: int) -> None:
+ while not stop_event.is_set():
+ frame = self.media.get_audio_sample()
+ if frame is None:
+ await asyncio.sleep(0.005)
+ continue
+ if time.monotonic() < self._mute_microphone_until:
+ await asyncio.sleep(0)
+ continue
+ payload = encode_robot_audio(frame, input_rate)
+ if payload:
+ await websocket.send(payload)
+ await asyncio.sleep(0)
+
+ async def _play_loop(self, websocket: Any, stop_event: Event, output_rate: int) -> None:
+ while not stop_event.is_set():
+ message = await websocket.recv()
+ if isinstance(message, bytes):
+ audio = decode_agent_audio(message, output_rate)
+ if audio.size:
+ duration = audio.size / output_rate
+ self._mute_microphone_until = max(
+ self._mute_microphone_until,
+ time.monotonic() + duration + 0.15,
+ )
+ self.media.push_audio_sample(audio)
+ continue
+ try:
+ event = json.loads(message)
+ except (TypeError, json.JSONDecodeError):
+ logger.debug("Ignoring malformed sandbox control message")
+ continue
+ if event.get("type") == "messages":
+ for item in event.get("messages", []):
+ if isinstance(item, dict):
+ logger.info("agent role=%s content=%s", item.get("role"), item.get("content"))
+ elif event.get("type") == "error":
+ logger.error("Sandbox audio error: %s", event.get("message"))
+
+ @staticmethod
+ async def _wait_or_stop(stop_event: Event, timeout: float | None) -> None:
+ started = time.monotonic()
+ while not stop_event.is_set():
+ if timeout is not None and time.monotonic() - started >= timeout:
+ return
+ await asyncio.sleep(0.1)
+
+ def _safe_media_call(self, method_name: str) -> None:
+ try:
+ getattr(self.media, method_name)()
+ except Exception as exc:
+ logger.debug("Ignoring %s failure during shutdown: %s", method_name, exc)
diff --git a/projects/reachy-mini-openshell/native-controller/src/reachy_mini_openshell_controller/camera_adapter.py b/projects/reachy-mini-openshell/native-controller/src/reachy_mini_openshell_controller/camera_adapter.py
new file mode 100644
index 00000000..dddbc50d
--- /dev/null
+++ b/projects/reachy-mini-openshell/native-controller/src/reachy_mini_openshell_controller/camera_adapter.py
@@ -0,0 +1,136 @@
+"""Narrow trusted HTTP adapter for one-frame Reachy camera capture."""
+
+from __future__ import annotations
+
+import math
+import threading
+import time
+from collections.abc import Callable
+from io import BytesIO
+from typing import Any
+
+import numpy as np
+from fastapi import FastAPI, HTTPException, Response
+from PIL import Image
+
+JPEG_MEDIA_TYPE = "image/jpeg"
+DEFAULT_MAX_JPEG_BYTES = 2 * 1024 * 1024
+DEFAULT_MIN_CAPTURE_INTERVAL_SECONDS = 1.0
+DEFAULT_FRAME_WAIT_SECONDS = 2.0
+DEFAULT_JPEG_QUALITY = 85
+
+
+def _encode_bgr_frame_as_jpeg(frame: Any) -> bytes:
+ """Encode one Reachy SDK BGR uint8 frame as a bounded-quality JPEG."""
+ if (
+ not isinstance(frame, np.ndarray)
+ or frame.dtype != np.uint8
+ or frame.ndim != 3
+ or frame.shape[2] != 3
+ or frame.size == 0
+ ):
+ raise ValueError("Reachy returned an invalid BGR camera frame")
+
+ rgb_frame = np.ascontiguousarray(frame[:, :, ::-1])
+ output = BytesIO()
+ Image.fromarray(rgb_frame).save(
+ output,
+ format="JPEG",
+ quality=DEFAULT_JPEG_QUALITY,
+ optimize=True,
+ )
+ return output.getvalue()
+
+
+class TrustedCameraAdapter:
+ """Expose only one bounded snapshot operation from the trusted native app."""
+
+ def __init__(
+ self,
+ media_provider: Callable[[], Any | None],
+ *,
+ max_jpeg_bytes: int = DEFAULT_MAX_JPEG_BYTES,
+ min_capture_interval_seconds: float = DEFAULT_MIN_CAPTURE_INTERVAL_SECONDS,
+ frame_wait_seconds: float = DEFAULT_FRAME_WAIT_SECONDS,
+ clock: Callable[[], float] = time.monotonic,
+ sleeper: Callable[[float], None] = time.sleep,
+ frame_encoder: Callable[[Any], bytes] = _encode_bgr_frame_as_jpeg,
+ ) -> None:
+ """Configure bounded capture without accepting model-controlled device options."""
+ if max_jpeg_bytes <= 0:
+ raise ValueError("max_jpeg_bytes must be positive")
+ if min_capture_interval_seconds < 0:
+ raise ValueError("min_capture_interval_seconds must be non-negative")
+ if frame_wait_seconds <= 0:
+ raise ValueError("frame_wait_seconds must be positive")
+
+ self._media_provider = media_provider
+ self._max_jpeg_bytes = max_jpeg_bytes
+ self._min_capture_interval_seconds = min_capture_interval_seconds
+ self._frame_wait_seconds = frame_wait_seconds
+ self._clock = clock
+ self._sleeper = sleeper
+ self._frame_encoder = frame_encoder
+ self._capture_lock = threading.Lock()
+ self._last_capture_at: float | None = None
+
+ def register(self, application: FastAPI) -> None:
+ """Register the fixed capture route on the Reachy App settings server."""
+
+ @application.post("/camera/capture", response_class=Response)
+ def capture() -> Response:
+ return self.capture()
+
+ def capture(self) -> Response:
+ """Capture one JPEG or return a bounded, non-sensitive error."""
+ if not self._capture_lock.acquire(blocking=False):
+ raise HTTPException(status_code=429, detail="A camera capture is already in progress")
+
+ try:
+ now = self._clock()
+ if self._last_capture_at is not None:
+ retry_after = self._min_capture_interval_seconds - (now - self._last_capture_at)
+ if retry_after > 0:
+ raise HTTPException(
+ status_code=429,
+ detail="Camera capture rate limit exceeded",
+ headers={"Retry-After": str(max(1, math.ceil(retry_after)))},
+ )
+
+ media = self._media_provider()
+ if media is None:
+ raise HTTPException(status_code=503, detail="Reachy camera is not ready")
+
+ deadline = now + self._frame_wait_seconds
+ frame: Any = None
+ while self._clock() < deadline:
+ frame = media.get_frame()
+ if frame is not None:
+ break
+ self._sleeper(0.05)
+
+ if frame is None:
+ raise HTTPException(status_code=503, detail="No camera frame is available")
+ try:
+ jpeg = self._frame_encoder(frame)
+ except (TypeError, ValueError, OSError) as exc:
+ raise HTTPException(status_code=502, detail="Reachy returned an invalid camera frame") from exc
+ if not isinstance(jpeg, bytes) or not jpeg:
+ raise HTTPException(status_code=502, detail="Reachy returned an invalid JPEG frame")
+ if len(jpeg) > self._max_jpeg_bytes:
+ raise HTTPException(status_code=413, detail="Camera frame exceeds the configured size limit")
+ if not jpeg.startswith(b"\xff\xd8"):
+ raise HTTPException(status_code=502, detail="Reachy returned an invalid JPEG frame")
+
+ self._last_capture_at = self._clock()
+ return Response(
+ content=jpeg,
+ media_type=JPEG_MEDIA_TYPE,
+ headers={
+ "Cache-Control": "no-store",
+ "Content-Disposition": 'inline; filename="reachy-capture.jpg"',
+ "X-Content-Type-Options": "nosniff",
+ },
+ )
+ finally:
+ self._capture_lock.release()
diff --git a/projects/reachy-mini-openshell/native-controller/src/reachy_mini_openshell_controller/openshell.py b/projects/reachy-mini-openshell/native-controller/src/reachy_mini_openshell_controller/openshell.py
new file mode 100644
index 00000000..319ef19b
--- /dev/null
+++ b/projects/reachy-mini-openshell/native-controller/src/reachy_mini_openshell_controller/openshell.py
@@ -0,0 +1,154 @@
+"""Fixed OpenShell lifecycle commands used by the trusted Reachy App."""
+
+from __future__ import annotations
+
+import logging
+import os
+import re
+import shutil
+import subprocess
+from dataclasses import dataclass
+from pathlib import Path
+
+from reachy_mini_openshell_controller.settings import ControllerSettings
+
+logger = logging.getLogger(__name__)
+
+ANSI_CONTROL_SEQUENCE = re.compile(r"\x1b\[[0-?]*[ -/]*[@-~]")
+
+
+class OpenShellLifecycleError(RuntimeError):
+ """Raised when the fixed sandbox lifecycle cannot be completed."""
+
+
+@dataclass(frozen=True)
+class CommandResult:
+ """Captured output from one fixed OpenShell invocation."""
+
+ returncode: int
+ stdout: str
+ stderr: str
+
+
+class OpenShellController:
+ """Control one pre-created sandbox without accepting arbitrary commands."""
+
+ def __init__(self, settings: ControllerSettings) -> None:
+ """Resolve the OpenShell CLI used for lifecycle commands."""
+ self.settings = settings
+ self.executable = self._resolve_executable(settings.openshell_executable)
+
+ @staticmethod
+ def _resolve_executable(configured: str | None) -> str:
+ if configured:
+ path = Path(configured).expanduser()
+ if path.is_file() and path.stat().st_mode & 0o111:
+ return str(path)
+ raise OpenShellLifecycleError(f"REACHY_OPENSHELL_BIN is not executable: {path}")
+
+ discovered = shutil.which("openshell")
+ if discovered:
+ return discovered
+ for candidate in (
+ Path("/home/pollen/.local/bin/openshell"),
+ Path("/home/pollen/.cargo/bin/openshell"),
+ Path("/usr/local/bin/openshell"),
+ Path("/usr/bin/openshell"),
+ ):
+ if candidate.is_file() and candidate.stat().st_mode & 0o111:
+ return str(candidate)
+ raise OpenShellLifecycleError("openshell executable not found; set REACHY_OPENSHELL_BIN")
+
+ def _run(
+ self,
+ arguments: list[str],
+ *,
+ check: bool = True,
+ timeout_seconds: float | None = None,
+ ) -> CommandResult:
+ environment = os.environ.copy()
+ environment["NO_COLOR"] = "1"
+ environment["CLICOLOR"] = "0"
+ environment["CLICOLOR_FORCE"] = "0"
+ environment["FORCE_COLOR"] = "0"
+ environment["TERM"] = "dumb"
+ completed = subprocess.run( # noqa: S603 - executable and arguments are fixed by this class
+ [self.executable, *arguments],
+ capture_output=True,
+ text=True,
+ timeout=timeout_seconds or self.settings.command_timeout_seconds,
+ check=False,
+ env=environment,
+ )
+ result = CommandResult(completed.returncode, completed.stdout, completed.stderr)
+ if check and result.returncode != 0:
+ detail = (result.stderr or result.stdout).strip()
+ raise OpenShellLifecycleError(f"openshell {' '.join(arguments[:3])} failed: {detail}")
+ return result
+
+ def verify_ready(self) -> None:
+ """Require the pre-created sandbox to be Ready."""
+ result = self._run(["sandbox", "get", self.settings.sandbox_name])
+ combined = ANSI_CONTROL_SEQUENCE.sub("", f"{result.stdout}\n{result.stderr}").lower()
+ ready_line = re.search(r"(?im)^\s*(?:phase\s*:?\s*)?ready\s*$", combined)
+ if ready_line is None:
+ raise OpenShellLifecycleError(
+ f"sandbox {self.settings.sandbox_name!r} is not Ready; output was: {result.stdout.strip()}"
+ )
+
+ def ensure_audio_service(self) -> None:
+ """Create the fixed audio service endpoint when missing."""
+ existing = self._run(
+ ["service", "get", self.settings.sandbox_name, self.settings.service_name],
+ check=False,
+ )
+ if existing.returncode == 0:
+ return
+ self._run(
+ [
+ "service",
+ "expose",
+ self.settings.sandbox_name,
+ str(self.settings.service_port),
+ self.settings.service_name,
+ ]
+ )
+
+ def start_agent(self) -> None:
+ """Start the agent inside the existing sandbox."""
+ self.verify_ready()
+ self._run(
+ [
+ "sandbox",
+ "exec",
+ "--name",
+ self.settings.sandbox_name,
+ "--no-tty",
+ "--",
+ "/opt/venv/bin/reachy-agent-control",
+ "start",
+ ]
+ )
+ # Register the service only after its loopback listener is live. The
+ # gateway can otherwise retain an unavailable target for the service.
+ self.ensure_audio_service()
+ logger.info("Started agent inside sandbox %s", self.settings.sandbox_name)
+
+ def stop_agent(self) -> None:
+ """Stop the sandbox agent without deleting its sandbox."""
+ result = self._run(
+ [
+ "sandbox",
+ "exec",
+ "--name",
+ self.settings.sandbox_name,
+ "--no-tty",
+ "--",
+ "/opt/venv/bin/reachy-agent-control",
+ "stop",
+ ],
+ check=False,
+ timeout_seconds=min(self.settings.command_timeout_seconds, 15.0),
+ )
+ if result.returncode != 0:
+ logger.warning("Unable to stop sandbox agent cleanly: %s", (result.stderr or result.stdout).strip())
diff --git a/projects/reachy-mini-openshell/native-controller/src/reachy_mini_openshell_controller/settings.py b/projects/reachy-mini-openshell/native-controller/src/reachy_mini_openshell_controller/settings.py
new file mode 100644
index 00000000..536ec916
--- /dev/null
+++ b/projects/reachy-mini-openshell/native-controller/src/reachy_mini_openshell_controller/settings.py
@@ -0,0 +1,62 @@
+"""Configuration for the trusted native controller."""
+
+from __future__ import annotations
+
+import os
+import re
+from dataclasses import dataclass
+
+
+@dataclass(frozen=True)
+class ControllerSettings:
+ """Fixed sandbox and audio bridge settings."""
+
+ sandbox_name: str = "reachy-agent"
+ service_name: str = "audio"
+ service_port: int = 8765
+ gateway_port: int = 17670
+ openshell_executable: str | None = None
+ command_timeout_seconds: float = 150.0
+ reconnect_initial_seconds: float = 0.5
+ reconnect_max_seconds: float = 5.0
+
+ def __post_init__(self) -> None:
+ """Reject settings that could be interpreted as CLI options or invalid URLs."""
+ name_pattern = re.compile(r"^[a-z0-9][a-z0-9-]{0,62}$")
+ for label, value in (("sandbox_name", self.sandbox_name), ("service_name", self.service_name)):
+ if name_pattern.fullmatch(value) is None:
+ raise ValueError(f"{label} must contain only lowercase letters, numbers, and hyphens")
+ for label, value in (("service_port", self.service_port), ("gateway_port", self.gateway_port)):
+ if not 1 <= value <= 65_535:
+ raise ValueError(f"{label} must be between 1 and 65535")
+ if self.command_timeout_seconds <= 0:
+ raise ValueError("command_timeout_seconds must be positive")
+ if self.reconnect_initial_seconds <= 0 or self.reconnect_max_seconds < self.reconnect_initial_seconds:
+ raise ValueError("audio reconnect delays are invalid")
+
+ @property
+ def audio_websocket_url(self) -> str:
+ """Return the loopback OpenShell service URL."""
+ return (
+ f"ws://{self.sandbox_name}--{self.service_name}.openshell.localhost:"
+ f"{self.gateway_port}/audio"
+ )
+
+ @property
+ def gateway_connect_host(self) -> str:
+ """Return the fixed TCP destination for the onboard OpenShell gateway."""
+ return "127.0.0.1"
+
+ @classmethod
+ def from_environment(cls) -> "ControllerSettings":
+ """Load supported controller overrides from the environment."""
+ return cls(
+ sandbox_name=os.getenv("REACHY_OPENSHELL_SANDBOX", "reachy-agent"),
+ service_name=os.getenv("REACHY_OPENSHELL_AUDIO_SERVICE", "audio"),
+ service_port=int(os.getenv("REACHY_OPENSHELL_AUDIO_PORT", "8765")),
+ gateway_port=int(os.getenv("REACHY_OPENSHELL_GATEWAY_PORT", "17670")),
+ openshell_executable=os.getenv("REACHY_OPENSHELL_BIN") or None,
+ command_timeout_seconds=float(os.getenv("REACHY_OPENSHELL_COMMAND_TIMEOUT_SECONDS", "150")),
+ reconnect_initial_seconds=float(os.getenv("REACHY_AUDIO_RECONNECT_INITIAL_SECONDS", "0.5")),
+ reconnect_max_seconds=float(os.getenv("REACHY_AUDIO_RECONNECT_MAX_SECONDS", "5")),
+ )
diff --git a/projects/reachy-mini-openshell/native-controller/tests/test_audio.py b/projects/reachy-mini-openshell/native-controller/tests/test_audio.py
new file mode 100644
index 00000000..80a9597a
--- /dev/null
+++ b/projects/reachy-mini-openshell/native-controller/tests/test_audio.py
@@ -0,0 +1,63 @@
+"""Tests for the trusted native audio bridge."""
+
+# ruff: noqa: D103
+
+from threading import Event
+from typing import Any
+
+import numpy as np
+import pytest
+
+from reachy_mini_openshell_controller.audio import decode_agent_audio, encode_robot_audio
+from reachy_mini_openshell_controller.bridge import NativeAudioBridge
+from reachy_mini_openshell_controller.settings import ControllerSettings
+
+
+def test_audio_service_uses_gateway_hostname_with_loopback_tcp_destination() -> None:
+ settings = ControllerSettings()
+
+ assert settings.audio_websocket_url == "ws://reachy-agent--audio.openshell.localhost:17670/audio"
+ assert settings.gateway_connect_host == "127.0.0.1"
+
+
+def test_encode_robot_audio_selects_mono_and_resamples() -> None:
+ stereo = np.column_stack(
+ [
+ np.linspace(-1.0, 1.0, 800, dtype=np.float32),
+ np.ones(800, dtype=np.float32),
+ ]
+ )
+
+ payload = encode_robot_audio(stereo, 8_000)
+
+ decoded = np.frombuffer(payload, dtype=" 0
+
+
+def test_decode_agent_audio_rejects_incomplete_sample() -> None:
+ with pytest.raises(ValueError, match="complete int16"):
+ decode_agent_audio(b"\x01", 16_000)
+
+
+@pytest.mark.asyncio
+async def test_play_loop_pushes_audio_and_mutes_microphone() -> None:
+ stop_event = Event()
+ pushed: list[np.ndarray[Any, Any]] = []
+
+ class Media:
+ def push_audio_sample(self, audio: np.ndarray[Any, Any]) -> None:
+ pushed.append(audio)
+
+ class WebSocket:
+ async def recv(self) -> bytes:
+ stop_event.set()
+ return np.array([0, 16_384, -16_384], dtype=" 0
diff --git a/projects/reachy-mini-openshell/native-controller/tests/test_camera_adapter.py b/projects/reachy-mini-openshell/native-controller/tests/test_camera_adapter.py
new file mode 100644
index 00000000..6ddba8c3
--- /dev/null
+++ b/projects/reachy-mini-openshell/native-controller/tests/test_camera_adapter.py
@@ -0,0 +1,90 @@
+"""Tests for the trusted native camera adapter."""
+
+# ruff: noqa: D101, D102, D103, D107
+
+from typing import Any
+
+import numpy as np
+from fastapi import FastAPI
+from fastapi.testclient import TestClient
+
+from reachy_mini_openshell_controller.camera_adapter import TrustedCameraAdapter
+
+JPEG = b"\xff\xd8test-jpeg\xff\xd9"
+FRAME = np.zeros((8, 12, 3), dtype=np.uint8)
+
+
+class Media:
+ def __init__(self, frames: list[Any]) -> None:
+ self.frames = frames
+ self.calls = 0
+
+ def get_frame(self) -> Any:
+ self.calls += 1
+ return self.frames.pop(0) if self.frames else None
+
+
+def app_for(adapter: TrustedCameraAdapter) -> FastAPI:
+ application = FastAPI()
+ adapter.register(application)
+ return application
+
+
+def test_capture_returns_one_uncached_bounded_jpeg() -> None:
+ media = Media([FRAME])
+ adapter = TrustedCameraAdapter(lambda: media, min_capture_interval_seconds=0)
+
+ with TestClient(app_for(adapter)) as client:
+ response = client.post("/camera/capture")
+
+ assert response.status_code == 200
+ assert response.content.startswith(b"\xff\xd8")
+ assert response.content.endswith(b"\xff\xd9")
+ assert response.headers["content-type"] == "image/jpeg"
+ assert response.headers["cache-control"] == "no-store"
+ assert media.calls == 1
+
+
+def test_capture_exposes_no_get_route_or_model_controlled_options() -> None:
+ adapter = TrustedCameraAdapter(lambda: Media([FRAME, FRAME]), min_capture_interval_seconds=0)
+
+ with TestClient(app_for(adapter)) as client:
+ get_response = client.get("/camera/capture")
+ body_response = client.post("/camera/capture", json={"filename": "/tmp/anything"})
+
+ assert get_response.status_code == 405
+ assert body_response.status_code == 200
+ assert body_response.content.startswith(b"\xff\xd8")
+
+
+def test_capture_rejects_unavailable_oversized_and_invalid_frames() -> None:
+ cases = [
+ (lambda: None, {}, 503),
+ (lambda: Media([FRAME]), {"max_jpeg_bytes": 4}, 413),
+ (lambda: Media([object()]), {}, 502),
+ ]
+
+ for provider, options, expected_status in cases:
+ adapter = TrustedCameraAdapter(
+ provider,
+ min_capture_interval_seconds=0,
+ frame_wait_seconds=0.01,
+ **options,
+ )
+ with TestClient(app_for(adapter)) as client:
+ response = client.post("/camera/capture")
+ assert response.status_code == expected_status
+
+
+def test_capture_rate_limits_successive_snapshots() -> None:
+ times = iter([10.0, 10.0, 10.0, 10.2])
+ media = Media([FRAME, FRAME])
+ adapter = TrustedCameraAdapter(media_provider=lambda: media, clock=lambda: next(times))
+
+ with TestClient(app_for(adapter)) as client:
+ first = client.post("/camera/capture")
+ second = client.post("/camera/capture")
+
+ assert first.status_code == 200
+ assert second.status_code == 429
+ assert second.headers["retry-after"] == "1"
diff --git a/projects/reachy-mini-openshell/native-controller/tests/test_openshell.py b/projects/reachy-mini-openshell/native-controller/tests/test_openshell.py
new file mode 100644
index 00000000..9aaec242
--- /dev/null
+++ b/projects/reachy-mini-openshell/native-controller/tests/test_openshell.py
@@ -0,0 +1,89 @@
+"""Tests for fixed native OpenShell lifecycle commands."""
+
+# ruff: noqa: D103
+
+import os
+import subprocess
+from pathlib import Path
+
+from reachy_mini_openshell_controller.openshell import OpenShellController
+from reachy_mini_openshell_controller.settings import ControllerSettings
+
+
+def executable(tmp_path: Path) -> Path:
+ path = tmp_path / "openshell"
+ path.write_text("#!/bin/sh\nexit 0\n", encoding="utf-8")
+ path.chmod(0o755)
+ return path
+
+
+def test_start_agent_uses_only_fixed_lifecycle_commands(tmp_path: Path, monkeypatch) -> None:
+ openshell = executable(tmp_path)
+ calls: list[list[str]] = []
+
+ def run(command: list[str], **kwargs):
+ calls.append(command)
+ if command[1:4] == ["sandbox", "get", "reachy-agent"]:
+ return subprocess.CompletedProcess(command, 0, "Phase: Ready\n", "")
+ if command[1:5] == ["service", "get", "reachy-agent", "audio"]:
+ return subprocess.CompletedProcess(command, 1, "", "service endpoint not found")
+ return subprocess.CompletedProcess(command, 0, "", "")
+
+ monkeypatch.setattr(subprocess, "run", run)
+ controller = OpenShellController(ControllerSettings(openshell_executable=os.fspath(openshell)))
+
+ controller.start_agent()
+
+ assert [call[1:] for call in calls] == [
+ ["sandbox", "get", "reachy-agent"],
+ [
+ "sandbox",
+ "exec",
+ "--name",
+ "reachy-agent",
+ "--no-tty",
+ "--",
+ "/opt/venv/bin/reachy-agent-control",
+ "start",
+ ],
+ ["service", "get", "reachy-agent", "audio"],
+ ["service", "expose", "reachy-agent", "8765", "audio"],
+ ]
+
+
+def test_verify_ready_accepts_ansi_formatted_output(tmp_path: Path, monkeypatch) -> None:
+ openshell = executable(tmp_path)
+
+ def run(command: list[str], **kwargs):
+ return subprocess.CompletedProcess(command, 0, "\x1b[2m Phase:\x1b[0m \x1b[32mReady\x1b[0m\n", "")
+
+ monkeypatch.setattr(subprocess, "run", run)
+ controller = OpenShellController(ControllerSettings(openshell_executable=os.fspath(openshell)))
+
+ controller.verify_ready()
+
+
+def test_stop_agent_does_not_delete_sandbox(tmp_path: Path, monkeypatch) -> None:
+ openshell = executable(tmp_path)
+ calls: list[list[str]] = []
+
+ def run(command: list[str], **kwargs):
+ calls.append(command)
+ return subprocess.CompletedProcess(command, 0, "", "")
+
+ monkeypatch.setattr(subprocess, "run", run)
+ controller = OpenShellController(ControllerSettings(openshell_executable=os.fspath(openshell)))
+
+ controller.stop_agent()
+
+ assert calls[0][1:] == [
+ "sandbox",
+ "exec",
+ "--name",
+ "reachy-agent",
+ "--no-tty",
+ "--",
+ "/opt/venv/bin/reachy-agent-control",
+ "stop",
+ ]
+ assert "delete" not in calls[0]
diff --git a/projects/reachy-mini-openshell/openshell/Dockerfile.camera-prompt-hotfix b/projects/reachy-mini-openshell/openshell/Dockerfile.camera-prompt-hotfix
new file mode 100644
index 00000000..61f147c7
--- /dev/null
+++ b/projects/reachy-mini-openshell/openshell/Dockerfile.camera-prompt-hotfix
@@ -0,0 +1,11 @@
+FROM reachy-mini-openshell:rest-arm64
+
+USER root
+COPY reachy_mini_conversation_app-0.3.0-py3-none-any.whl /tmp/reachy-app.whl
+RUN /usr/local/bin/python -c \
+ 'import zipfile; zipfile.ZipFile("/tmp/reachy-app.whl").extractall("/opt/venv/lib/python3.12/site-packages")' \
+ && rm /tmp/reachy-app.whl
+
+USER sandbox
+WORKDIR /sandbox
+CMD ["/bin/sleep", "infinity"]
diff --git a/projects/reachy-mini-openshell/openshell/policy-camera-enabled-motion-disabled.yaml b/projects/reachy-mini-openshell/openshell/policy-camera-enabled-motion-disabled.yaml
new file mode 100644
index 00000000..b53d159e
--- /dev/null
+++ b/projects/reachy-mini-openshell/openshell/policy-camera-enabled-motion-disabled.yaml
@@ -0,0 +1,92 @@
+version: 1
+
+# Camera capture is allowed while every motion-start endpoint remains blocked.
+filesystem_policy:
+ include_workdir: true
+ read_only:
+ - /bin
+ - /usr
+ - /lib
+ - /lib64
+ - /proc
+ - /sys
+ - /etc
+ - /opt
+ - /var/log
+ - /dev/urandom
+ read_write:
+ - /sandbox
+ - /tmp
+ - /dev/null
+ - /home/sandbox
+
+landlock:
+ compatibility: best_effort
+
+process:
+ run_as_user: sandbox
+ run_as_group: sandbox
+
+network_policies:
+ reachy_rest:
+ name: reachy-rest-motion-disabled
+ endpoints:
+ - host: host.openshell.internal
+ port: 8000
+ protocol: rest
+ enforcement: enforce
+ allowed_ips:
+ - 10.0.0.0/8
+ - 172.16.0.0/12
+ - 192.168.0.0/16
+ rules:
+ - allow:
+ method: GET
+ path: /api/daemon/status
+ - allow:
+ method: GET
+ path: /api/move/running
+ - allow:
+ method: POST
+ path: /api/move/stop
+ binaries:
+ - path: /opt/venv/bin/python
+
+ reachy_camera:
+ name: reachy-camera-single-frame
+ endpoints:
+ - host: host.openshell.internal
+ port: 8042
+ protocol: rest
+ enforcement: enforce
+ allowed_ips:
+ - 10.0.0.0/8
+ - 172.16.0.0/12
+ - 192.168.0.0/16
+ rules:
+ - allow:
+ method: POST
+ path: /camera/capture
+ binaries:
+ - path: /opt/venv/bin/python
+
+ openai_realtime:
+ name: openai-realtime
+ endpoints:
+ - host: api.openai.com
+ port: 443
+ protocol: websocket
+ enforcement: enforce
+ rules:
+ - allow:
+ method: GET
+ path: /v1/realtime
+ query:
+ model: gpt-realtime-2
+ - allow:
+ method: WEBSOCKET_TEXT
+ path: /v1/realtime
+ query:
+ model: gpt-realtime-2
+ binaries:
+ - path: /opt/venv/bin/python
diff --git a/projects/reachy-mini-openshell/openshell/policy-head-motion-enabled.yaml b/projects/reachy-mini-openshell/openshell/policy-head-motion-enabled.yaml
new file mode 100644
index 00000000..0fb7dd47
--- /dev/null
+++ b/projects/reachy-mini-openshell/openshell/policy-head-motion-enabled.yaml
@@ -0,0 +1,79 @@
+version: 1
+
+# Static sections are fixed when the sandbox is created.
+filesystem_policy:
+ include_workdir: true
+ read_only:
+ - /bin
+ - /usr
+ - /lib
+ - /lib64
+ - /proc
+ - /sys
+ - /etc
+ - /opt
+ - /var/log
+ - /dev/urandom
+ read_write:
+ - /sandbox
+ - /tmp
+ - /dev/null
+ - /home/sandbox
+
+landlock:
+ compatibility: best_effort
+
+process:
+ run_as_user: sandbox
+ run_as_group: sandbox
+
+# This policy permits the whole /api/move/goto endpoint. OpenShell does not
+# currently verify that its JSON body contains only a fixed head direction.
+network_policies:
+ reachy_rest:
+ name: reachy-rest-head-motion-enabled
+ endpoints:
+ - host: host.openshell.internal
+ port: 8000
+ protocol: rest
+ enforcement: enforce
+ allowed_ips:
+ - 10.0.0.0/8
+ - 172.16.0.0/12
+ - 192.168.0.0/16
+ rules:
+ - allow:
+ method: GET
+ path: /api/daemon/status
+ - allow:
+ method: GET
+ path: /api/move/running
+ - allow:
+ method: POST
+ path: /api/move/goto
+ - allow:
+ method: POST
+ path: /api/move/stop
+ binaries:
+ - path: /opt/venv/bin/python
+
+ openai_realtime:
+ name: openai-realtime
+ endpoints:
+ - host: api.openai.com
+ port: 443
+ protocol: websocket
+ enforcement: enforce
+ rules:
+ - allow:
+ method: GET
+ path: /v1/realtime
+ query:
+ model: gpt-realtime-2
+ - allow:
+ method: WEBSOCKET_TEXT
+ path: /v1/realtime
+ query:
+ model: gpt-realtime-2
+ binaries:
+ - path: /opt/venv/bin/python
diff --git a/projects/reachy-mini-openshell/openshell/policy-motion-disabled.yaml b/projects/reachy-mini-openshell/openshell/policy-motion-disabled.yaml
new file mode 100644
index 00000000..558491b7
--- /dev/null
+++ b/projects/reachy-mini-openshell/openshell/policy-motion-disabled.yaml
@@ -0,0 +1,77 @@
+version: 1
+
+# Static sections are fixed when the sandbox is created.
+filesystem_policy:
+ include_workdir: true
+ read_only:
+ - /bin
+ - /usr
+ - /lib
+ - /lib64
+ - /proc
+ - /sys
+ - /etc
+ - /opt
+ - /var/log
+ - /dev/urandom
+ read_write:
+ - /sandbox
+ - /tmp
+ - /dev/null
+ - /home/sandbox
+
+landlock:
+ compatibility: best_effort
+
+process:
+ run_as_user: sandbox
+ run_as_group: sandbox
+
+# REST rules match the HTTP method and path, not JSON request-body values.
+# The app may inspect Reachy's state and stop movement, but cannot start motion.
+# Port 8042 is intentionally absent, so camera capture is also policy-denied.
+network_policies:
+ reachy_rest:
+ name: reachy-rest-motion-disabled
+ endpoints:
+ - host: host.openshell.internal
+ port: 8000
+ protocol: rest
+ enforcement: enforce
+ allowed_ips:
+ - 10.0.0.0/8
+ - 172.16.0.0/12
+ - 192.168.0.0/16
+ rules:
+ - allow:
+ method: GET
+ path: /api/daemon/status
+ - allow:
+ method: GET
+ path: /api/move/running
+ - allow:
+ method: POST
+ path: /api/move/stop
+ binaries:
+ - path: /opt/venv/bin/python
+
+ openai_realtime:
+ name: openai-realtime
+ endpoints:
+ - host: api.openai.com
+ port: 443
+ protocol: websocket
+ enforcement: enforce
+ rules:
+ - allow:
+ method: GET
+ path: /v1/realtime
+ query:
+ model: gpt-realtime-2
+ - allow:
+ method: WEBSOCKET_TEXT
+ path: /v1/realtime
+ query:
+ model: gpt-realtime-2
+ binaries:
+ - path: /opt/venv/bin/python
diff --git a/projects/reachy-mini-openshell/pyproject.toml b/projects/reachy-mini-openshell/pyproject.toml
index ce47a1b0..351cd449 100644
--- a/projects/reachy-mini-openshell/pyproject.toml
+++ b/projects/reachy-mini-openshell/pyproject.toml
@@ -8,7 +8,21 @@ version = "0.3.0"
description = "Reachy Mini conversation demo for OpenShell."
readme = "README.md"
requires-python = ">=3.10,<3.13"
-dependencies = [ "aiortc>=1.13.0", "fastrtc>=0.0.34", "gradio==5.50.1.dev1", "huggingface-hub==1.3.0", "opencv-python>=4.12.0.88", "python-dotenv", "openai>=2.1", "reachy_mini_dances_library", "reachy_mini_toolbox", "reachy-mini[mujoco]==1.8.0", "eclipse-zenoh~=1.7.0", "gradio_client>=1.13.3",]
+dependencies = [
+ "aiortc>=1.13.0",
+ "fastrtc>=0.0.34",
+ "gradio==5.50.1.dev1",
+ "huggingface-hub==1.3.0",
+ "opencv-python>=4.12.0.88",
+ "python-dotenv",
+ "openai>=2.1",
+ "httpx>=0.27",
+ "reachy_mini_dances_library",
+ "reachy_mini_toolbox",
+ "reachy-mini[mujoco]==1.8.0",
+ "eclipse-zenoh~=1.7.0",
+ "gradio_client>=1.13.3",
+]
[[project.authors]]
name = "Pollen Robotics"
email = "contact@pollen-robotics.com"
@@ -38,6 +52,8 @@ all_vision = [ "torch>=2.1", "transformers==5.0.0rc2", "num2words", "ultralytics
[project.scripts]
reachy-mini-conversation-app = "reachy_mini_conversation_app.main:main"
reachy-mini-backend-check = "reachy_mini_conversation_app.backend_check:main"
+reachy-mini-sandbox-audio = "reachy_mini_conversation_app.sandbox_audio:main"
+reachy-agent-control = "reachy_mini_conversation_app.sandbox_control:main"
[tool.setuptools]
include-package-data = true
@@ -46,9 +62,6 @@ include-package-data = true
line-length = 119
exclude = [ ".venv", "dist", "build", "**/__pycache__", "*.egg-info", ".pytest_cache",]
-[project.entry-points.reachy_mini_apps]
-reachy_mini_conversation_app = "reachy_mini_conversation_app.main:ReachyMiniConversationApp"
-
[tool.setuptools.package-dir]
"" = "src"
@@ -65,6 +78,9 @@ indent-style = "space"
skip-magic-trailing-comma = false
line-ending = "auto"
+[tool.pytest.ini_options]
+pythonpath = ["src", "native-controller/src"]
+
[tool.setuptools.packages.find]
where = [ "src",]
diff --git a/projects/reachy-mini-openshell/requirements-rest.txt b/projects/reachy-mini-openshell/requirements-rest.txt
new file mode 100644
index 00000000..0903eb13
--- /dev/null
+++ b/projects/reachy-mini-openshell/requirements-rest.txt
@@ -0,0 +1,13 @@
+# Runtime dependencies for the ARM64 OpenShell REST container.
+# Native Reachy SDK, camera, vision, dance, simulator, and Zenoh packages are
+# intentionally omitted because robot actions go through the daemon REST API.
+fastrtc>=0.0.34
+fastapi>=0.115,<1
+gradio==5.50.1.dev1
+huggingface-hub==1.3.0
+httpx>=0.27
+numpy<3
+openai[realtime]>=2.1
+python-dotenv
+uvicorn>=0.34
+websockets>=15,<16
diff --git a/projects/reachy-mini-openshell/scripts/start-local.sh b/projects/reachy-mini-openshell/scripts/start-local.sh
index f5250a11..2065d463 100755
--- a/projects/reachy-mini-openshell/scripts/start-local.sh
+++ b/projects/reachy-mini-openshell/scripts/start-local.sh
@@ -59,6 +59,28 @@ except Exception:
PY
}
+daemon_camera_available() {
+ "${PROJECT_DIR}/.venv/bin/python" - "$DAEMON_HOST" "$DAEMON_PORT" <<'PY' >/dev/null 2>&1
+import json
+import sys
+import urllib.request
+
+host, port = sys.argv[1], sys.argv[2]
+try:
+ with urllib.request.urlopen(f"http://{host}:{port}/api/daemon/status", timeout=1.5) as response:
+ payload = json.loads(response.read().decode("utf-8"))
+ available = (
+ response.status == 200
+ and payload.get("state") == "running"
+ and not payload.get("no_media", False)
+ and not payload.get("media_released", False)
+ )
+ raise SystemExit(0 if available else 1)
+except Exception:
+ raise SystemExit(1)
+PY
+}
+
pick_app_port() {
if [[ -n "${APP_PORT}" ]]; then
"${PROJECT_DIR}/.venv/bin/python" - "$APP_HOST" "$APP_PORT" <<'PY'
@@ -159,4 +181,11 @@ export GRADIO_SERVER_PORT="${SELECTED_APP_PORT}"
log "Starting Reachy conversation app"
log "Open: http://${APP_HOST}:${SELECTED_APP_PORT}/"
-"${PROJECT_DIR}/.venv/bin/python" -m reachy_mini_conversation_app --gradio --no-camera "$@"
+APP_ARGS=(--gradio)
+if daemon_camera_available; then
+ log "Reachy media is available; enabling the camera tool"
+else
+ log "Reachy media is unavailable; disabling camera support"
+ APP_ARGS+=(--no-camera)
+fi
+"${PROJECT_DIR}/.venv/bin/python" -m reachy_mini_conversation_app "${APP_ARGS[@]}" "$@"
diff --git a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/camera_worker.py b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/camera_worker.py
index 046bd8db..e20dce9a 100644
--- a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/camera_worker.py
+++ b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/camera_worker.py
@@ -79,6 +79,15 @@ def set_head_tracking_enabled(self, enabled: bool) -> None:
self.is_head_tracking_enabled = enabled
logger.info(f"Head tracking {'enabled' if enabled else 'disabled'}")
+ def clear_face_tracking_offsets(self) -> None:
+ """Immediately clear the additive face-tracking pose.
+
+ Scene scans use this after temporarily disabling tracking so the sweep
+ follows its deterministic primary motion without a stale face offset.
+ """
+ with self.face_tracking_lock:
+ self.face_tracking_offsets = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
+
def start(self) -> None:
"""Start the camera worker loop in a thread."""
self._stop_event.clear()
diff --git a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/chat_completions.py b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/chat_completions.py
index 882f6d0b..dc0dde55 100644
--- a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/chat_completions.py
+++ b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/chat_completions.py
@@ -8,12 +8,15 @@
from typing import Any, Final, cast
from reachy_mini_conversation_app.prompts import get_session_instructions
+from reachy_mini_conversation_app.tool_transport import ToolTransport
from reachy_mini_conversation_app.tools.core_tools import (
ToolDependencies,
get_tool_specs,
dispatch_tool_call_with_manager,
+ get_tool_specs_for_dependencies,
)
-from reachy_mini_conversation_app.tools.background_tool_manager import BackgroundToolManager
+from reachy_mini_conversation_app.media_result_processor import MediaResultProcessor
+from reachy_mini_conversation_app.tools.background_tool_manager import ToolCallRoutine, BackgroundToolManager
logger = logging.getLogger(__name__)
@@ -22,16 +25,25 @@
_RATE_LIMIT_DEFAULT_DELAY: Final[float] = 5.0
_RATE_LIMIT_MAX_DELAY: Final[float] = 30.0
_TOOL_ROUND_LIMIT: Final[int] = 5
+_MAX_TOOL_IMAGES: Final[int] = 12
_WAIT_RE: Final[re.Pattern[str]] = re.compile(
r"(?:please\s+)?wait\s+(\d+(?:\.\d+)?)\s+seconds?",
re.IGNORECASE,
)
-def chat_completion_tool_specs() -> list[dict[str, Any]]:
+def chat_completion_tool_specs(
+ deps: ToolDependencies | None = None,
+ tool_specs: list[dict[str, Any]] | None = None,
+) -> list[dict[str, Any]]:
"""Convert Realtime-style tool specs to Chat Completions tool specs."""
chat_tools: list[dict[str, Any]] = []
- for tool in get_tool_specs():
+ realtime_tools = (
+ tool_specs
+ if tool_specs is not None
+ else (get_tool_specs_for_dependencies(deps) if deps is not None else get_tool_specs())
+ )
+ for tool in realtime_tools:
if tool.get("type") != "function":
continue
chat_tools.append(
@@ -92,6 +104,66 @@ def _serialize_tool_call(tool_call: Any) -> dict[str, Any]:
}
+def _separate_tool_images(tool_name: str, tool_result: dict[str, Any]) -> tuple[dict[str, Any], list[str]]:
+ """Remove raw image bytes from tool JSON and return validated vision inputs."""
+ model_tool_result = dict(tool_result)
+ images: list[str] = []
+
+ if tool_name == "camera" and "b64_im" in model_tool_result:
+ raw_image = model_tool_result.pop("b64_im")
+ if isinstance(raw_image, str) and raw_image:
+ images = [raw_image]
+ model_tool_result["status"] = "image_captured"
+ else:
+ return {"error": "Camera returned an invalid image"}, []
+ elif tool_name == "scan_scene" and "b64_images" in model_tool_result:
+ raw_images = model_tool_result.pop("b64_images")
+ if (
+ isinstance(raw_images, list)
+ and 0 < len(raw_images) <= _MAX_TOOL_IMAGES
+ and all(isinstance(image, str) and image for image in raw_images)
+ ):
+ images = raw_images
+ else:
+ return {"error": "Scene scan returned invalid analysis images"}, []
+
+ return model_tool_result, images
+
+
+def _vision_followup_message(
+ tool_name: str,
+ tool_result: dict[str, Any],
+ images: list[str],
+) -> dict[str, Any]:
+ """Build a Chat Completions user message containing tool-captured images."""
+ if tool_name == "scan_scene":
+ question = tool_result.get("question", "Describe everything visible during the scan.")
+ timestamps = tool_result.get("frame_timestamps_seconds", [])
+ prompt = (
+ "These are chronological frames sampled across one Reachy scene sweep. "
+ f"Frame timestamps in seconds: {timestamps}. Combine evidence across all frames, "
+ "deduplicate repeated people and objects, and describe only visibly supported details. "
+ f"User question: {question}"
+ )
+ else:
+ question = tool_result.get("question", "Describe this image.")
+ prompt = f"Answer the camera question using this image: {question}"
+
+ return {
+ "role": "user",
+ "content": [
+ {"type": "text", "text": prompt},
+ *[
+ {
+ "type": "image_url",
+ "image_url": {"url": f"data:image/jpeg;base64,{image}"},
+ }
+ for image in images
+ ],
+ ],
+ }
+
+
def _rate_limit_delay(exc: Exception) -> float | None:
"""Return a retry delay when a Chat Completions error is a provider rate limit."""
status_code = getattr(exc, "status_code", None)
@@ -136,11 +208,15 @@ def __init__(
tool_manager: BackgroundToolManager,
model_name: str,
base_url: str | None,
+ tool_transport: ToolTransport | None = None,
+ media_result_processor: MediaResultProcessor | None = None,
) -> None:
"""Initialize the runner."""
self.client = client
self.deps = deps
self.tool_manager = tool_manager
+ self.tool_transport = tool_transport
+ self.media_result_processor = media_result_processor
self.model_name = model_name
self.base_url = base_url
@@ -172,7 +248,19 @@ async def send_text_message(self, text: str) -> list[dict[str, Any]]:
{"role": "system", "content": get_session_instructions()},
{"role": "user", "content": text},
]
- chat_tools = chat_completion_tool_specs()
+ try:
+ transport_specs = await self.tool_transport.list_tools() if self.tool_transport is not None else None
+ except Exception as e:
+ logger.exception("Tool discovery failed")
+ chatbot_messages.append(
+ {
+ "role": "assistant",
+ "content": f"[error] Tool discovery failed: {type(e).__name__}: {e}",
+ }
+ )
+ return chatbot_messages
+
+ chat_tools = chat_completion_tool_specs(self.deps, transport_specs)
operation = "Chat Completions request"
request_kwargs: dict[str, Any] = {
@@ -227,12 +315,20 @@ async def send_text_message(self, text: str) -> list[dict[str, Any]]:
}
)
+ vision_followups: list[dict[str, Any]] = []
for tool_call in tool_calls:
tool_call_id = _tool_call_value(tool_call, "id") or str(uuid.uuid4())
tool_name = _tool_call_function_value(tool_call, "name")
args_json = _tool_call_function_value(tool_call, "arguments") or "{}"
if not isinstance(tool_name, str):
tool_result = {"error": "tool call did not include a valid function name"}
+ elif self.tool_transport is not None:
+ tool_result = await ToolCallRoutine(
+ tool_name=tool_name,
+ args_json_str=args_json,
+ deps=self.deps,
+ transport=self.tool_transport,
+ )(self.tool_manager)
else:
tool_result = await dispatch_tool_call_with_manager(
tool_name,
@@ -241,7 +337,13 @@ async def send_text_message(self, text: str) -> list[dict[str, Any]]:
self.tool_manager,
)
- tool_result_json = json.dumps(tool_result)
+ if self.media_result_processor is not None:
+ processed_media = await self.media_result_processor.process(tool_name or "", tool_result)
+ model_tool_result = processed_media.model_payload
+ vision_images: list[str] = []
+ else:
+ model_tool_result, vision_images = _separate_tool_images(tool_name or "", tool_result)
+ tool_result_json = json.dumps(model_tool_result)
chat_messages.append(
{
"role": "tool",
@@ -259,6 +361,12 @@ async def send_text_message(self, text: str) -> list[dict[str, Any]]:
},
}
)
+ if vision_images:
+ vision_followups.append(
+ _vision_followup_message(tool_name or "", model_tool_result, vision_images)
+ )
+
+ chat_messages.extend(vision_followups)
operation = "Chat Completions tool follow-up"
request_kwargs = {
diff --git a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/config.py b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/config.py
index d6ab93f2..d1dd903c 100644
--- a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/config.py
+++ b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/config.py
@@ -22,6 +22,10 @@
BACKEND_LOCAL_STT,
}
+TOOL_TRANSPORT_LOCAL = "local"
+TOOL_TRANSPORT_REST = "rest"
+TOOL_TRANSPORTS = {TOOL_TRANSPORT_LOCAL, TOOL_TRANSPORT_REST}
+
HF_REALTIME_CONNECTION_DEPLOYED = "deployed"
HF_REALTIME_CONNECTION_LOCAL = "local"
HF_REALTIME_SESSION_PROXY_URL = "https://pollen-robotics-reachy-mini-realtime-url.hf.space/session"
@@ -31,11 +35,11 @@
_ENV_REF_RE = re.compile(r"\$\{(?P[A-Za-z_][A-Za-z0-9_]*)\}")
_PLACEHOLDER_VALUES = {"", "set-me"}
_ORIGINAL_PROCESS_ENV = dict(os.environ)
+_DEFAULT_VISION_ALLOWED_MODELS = ("gpt-5.4-mini",)
-def _env_flag(name: str, default: bool = False) -> bool:
- """Parse a boolean environment flag."""
- raw = os.getenv(name)
+def _parse_bool_value(name: str, raw: str | None, default: bool = False) -> bool:
+ """Parse a boolean configuration value."""
if raw is None:
return default
@@ -49,6 +53,11 @@ def _env_flag(name: str, default: bool = False) -> bool:
return default
+def _env_flag(name: str, default: bool = False) -> bool:
+ """Parse a boolean environment flag."""
+ return _parse_bool_value(name, os.getenv(name), default)
+
+
_profile_path = DEFAULT_PROFILES_DIRECTORY / LOCKED_PROFILE
_instructions_file = _profile_path / "instructions.txt"
if not _profile_path.is_dir():
@@ -217,6 +226,28 @@ def _dotenv_float(name: str, default: float) -> float:
return default
+def _runtime_value(name: str, default: str | None = None) -> str | None:
+ """Return a process-injected value first, then dotenv, then the default."""
+ return _process_env_value(name) or _dotenv_value(name, default)
+
+
+def _runtime_url(name: str, default: str | None = None) -> str | None:
+ """Return a normalized URL using process environment precedence."""
+ return _clean_url_value(name, _runtime_value(name, default))
+
+
+def _runtime_float(name: str, default: float) -> float:
+ """Return a float using process environment precedence."""
+ value = _runtime_value(name)
+ if value is None:
+ return default
+ try:
+ return float(value)
+ except ValueError:
+ logger.warning("Invalid float value for %s=%r, using default=%s", name, value, default)
+ return default
+
+
def _mapping_value(values: Mapping[str, str | None], name: str, default: str | None = None) -> str | None:
"""Return a stripped value from a dotenv mapping."""
value = values.get(name)
@@ -243,11 +274,53 @@ def _mapping_float(values: Mapping[str, str | None], name: str, default: float)
return default
+def _mapping_bool(values: Mapping[str, str | None], name: str, default: bool) -> bool:
+ """Return a boolean from an arbitrary dotenv mapping."""
+ return _parse_bool_value(name, _mapping_value(values, name), default)
+
+
+def _csv_values(value: str | None, default: tuple[str, ...]) -> tuple[str, ...]:
+ """Parse a comma-separated list while preserving order and removing duplicates."""
+ if value is None:
+ return default
+
+ parsed = tuple(dict.fromkeys(item.strip() for item in value.split(",") if item.strip()))
+ return parsed or default
+
+
+def _dotenv_csv(name: str, default: tuple[str, ...]) -> tuple[str, ...]:
+ """Return a comma-separated tuple from the loaded dotenv file."""
+ return _csv_values(_dotenv_value(name), default)
+
+
+def _runtime_csv(name: str, default: tuple[str, ...]) -> tuple[str, ...]:
+ """Return a comma-separated tuple using process environment precedence."""
+ return _csv_values(_runtime_value(name), default)
+
+
+def _mapping_csv(
+ values: Mapping[str, str | None],
+ name: str,
+ default: tuple[str, ...],
+) -> tuple[str, ...]:
+ """Return a comma-separated tuple from an arbitrary dotenv mapping."""
+ return _csv_values(values.get(name), default)
+
+
def _normalize_backend_provider(value: str | None) -> str:
"""Normalize the configured conversation backend provider."""
return (value or "").strip().lower()
+def _normalize_tool_transport(value: str | None) -> str:
+ """Normalize the selected conversation tool transport."""
+ candidate = (value or TOOL_TRANSPORT_REST).strip().lower()
+ if candidate in TOOL_TRANSPORTS:
+ return candidate
+ logger.warning("Invalid REACHY_TOOL_TRANSPORT=%r; using rest.", value)
+ return TOOL_TRANSPORT_REST
+
+
def _normalize_hf_connection_mode(value: str | None) -> str:
"""Normalize the Hugging Face realtime connection mode."""
candidate = (value or HF_REALTIME_CONNECTION_DEPLOYED).strip().lower()
@@ -293,46 +366,66 @@ def parse_hf_realtime_url(realtime_url: str) -> HFRealtimeURLParts:
class Config:
"""Configuration class for the conversation app."""
- BACKEND_PROVIDER = _normalize_backend_provider(_dotenv_value("BACKEND_PROVIDER"))
+ BACKEND_PROVIDER = _normalize_backend_provider(_runtime_value("BACKEND_PROVIDER"))
- REALTIME_TRANSCRIPTION_LANGUAGE = _dotenv_value("REALTIME_TRANSCRIPTION_LANGUAGE", "en")
+ REALTIME_TRANSCRIPTION_LANGUAGE = _runtime_value("REALTIME_TRANSCRIPTION_LANGUAGE", "en")
- OPENAI_REALTIME_API_KEY = (
- _configured_value(_dotenv_value("OPENAI_REALTIME_API_KEY"))
- or _configured_value(_dotenv_value("OPENAI_API_KEY"))
- or _process_env_value("OPENAI_API_KEY")
+ OPENAI_REALTIME_API_KEY = _configured_value(_runtime_value("OPENAI_REALTIME_API_KEY")) or _configured_value(
+ _runtime_value("OPENAI_API_KEY")
+ )
+ OPENAI_REALTIME_BASE_URL = _runtime_url("OPENAI_REALTIME_BASE_URL", "https://api.openai.com/v1")
+ OPENAI_REALTIME_MODEL = _runtime_value("OPENAI_REALTIME_MODEL", "gpt-realtime-2")
+ OPENAI_REALTIME_VOICE = _runtime_value("OPENAI_REALTIME_VOICE", "cedar")
+
+ VISION_API_KEY = _configured_value(_runtime_value("VISION_API_KEY"))
+ VISION_BASE_URL = _runtime_url("VISION_BASE_URL", "https://api.openai.com/v1")
+ VISION_DEFAULT_MODEL = _runtime_value("VISION_DEFAULT_MODEL", "gpt-5.4-mini")
+ VISION_ALLOWED_MODELS = _runtime_csv("VISION_ALLOWED_MODELS", _DEFAULT_VISION_ALLOWED_MODELS)
+
+ REACHY_TOOL_TRANSPORT = _normalize_tool_transport(_runtime_value("REACHY_TOOL_TRANSPORT", TOOL_TRANSPORT_REST))
+ REACHY_REST_BASE_URL = _runtime_url("REACHY_REST_BASE_URL", "http://127.0.0.1:8000")
+ REACHY_CAMERA_BASE_URL = _runtime_url("REACHY_CAMERA_BASE_URL")
+ REACHY_REST_TIMEOUT_SECONDS = _runtime_float("REACHY_REST_TIMEOUT_SECONDS", 5.0)
+ REACHY_MOTION_DURATION_SECONDS = _runtime_float("REACHY_MOTION_DURATION_SECONDS", 1.0)
+ REACHY_MOTION_POLL_INTERVAL_SECONDS = _runtime_float("REACHY_MOTION_POLL_INTERVAL_SECONDS", 0.1)
+ REACHY_MOTION_COMPLETION_TIMEOUT_SECONDS = _runtime_float(
+ "REACHY_MOTION_COMPLETION_TIMEOUT_SECONDS",
+ 10.0,
+ )
+ REQUIRE_ROUTED_VISION = _parse_bool_value(
+ "REQUIRE_ROUTED_VISION",
+ _runtime_value("REQUIRE_ROUTED_VISION"),
+ False,
)
- OPENAI_REALTIME_BASE_URL = _dotenv_url("OPENAI_REALTIME_BASE_URL", "https://api.openai.com/v1")
- OPENAI_REALTIME_MODEL = _dotenv_value("OPENAI_REALTIME_MODEL", "gpt-realtime-2")
- OPENAI_REALTIME_VOICE = _dotenv_value("OPENAI_REALTIME_VOICE", "cedar")
-
- HF_REALTIME_CONNECTION_MODE = _normalize_hf_connection_mode(_dotenv_value("HF_REALTIME_CONNECTION_MODE"))
- HF_REALTIME_SESSION_URL = _dotenv_url("HF_REALTIME_SESSION_URL", HF_REALTIME_SESSION_PROXY_URL)
- HF_REALTIME_WS_URL = _dotenv_url("HF_REALTIME_WS_URL")
- HF_REALTIME_MODEL = _dotenv_value("HF_REALTIME_MODEL", "")
- HF_REALTIME_VOICE = _dotenv_value("HF_REALTIME_VOICE", "Aiden")
-
- CHAT_API_KEY = _dotenv_value("CHAT_API_KEY")
- CHAT_BASE_URL = _dotenv_url("CHAT_BASE_URL")
- CHAT_MODEL_NAME = _dotenv_value("CHAT_MODEL_NAME")
-
- STT_API_KEY = _dotenv_value("STT_API_KEY", "not-needed")
- STT_BASE_URL = _dotenv_url("STT_BASE_URL")
- STT_MODEL_NAME = _dotenv_value("STT_MODEL_NAME", "whisper-1")
- TTS_API_KEY = _dotenv_value("TTS_API_KEY", "not-needed")
- TTS_BASE_URL = _dotenv_url("TTS_BASE_URL")
- TTS_MODEL_NAME = _dotenv_value("TTS_MODEL_NAME", "gpt-4o-mini-tts")
- TTS_VOICE = _dotenv_value("TTS_VOICE", OPENAI_REALTIME_VOICE)
- MIC_TRANSCRIPTION_RMS_THRESHOLD = _dotenv_float("MIC_TRANSCRIPTION_RMS_THRESHOLD", 500.0)
- MIC_TRANSCRIPTION_MIN_AUDIO_MS = _dotenv_float("MIC_TRANSCRIPTION_MIN_AUDIO_MS", 250.0)
- MIC_TRANSCRIPTION_SILENCE_MS = _dotenv_float("MIC_TRANSCRIPTION_SILENCE_MS", 800.0)
- MIC_TRANSCRIPTION_MAX_AUDIO_MS = _dotenv_float("MIC_TRANSCRIPTION_MAX_AUDIO_MS", 12_000.0)
- HF_HOME = _dotenv_value("HF_HOME", "./cache")
- LOCAL_VISION_MODEL = _dotenv_value("LOCAL_VISION_MODEL", "HuggingFaceTB/SmolVLM2-2.2B-Instruct")
- HF_TOKEN = _dotenv_value("HF_TOKEN")
+
+ HF_REALTIME_CONNECTION_MODE = _normalize_hf_connection_mode(_runtime_value("HF_REALTIME_CONNECTION_MODE"))
+ HF_REALTIME_SESSION_URL = _runtime_url("HF_REALTIME_SESSION_URL", HF_REALTIME_SESSION_PROXY_URL)
+ HF_REALTIME_WS_URL = _runtime_url("HF_REALTIME_WS_URL")
+ HF_REALTIME_MODEL = _runtime_value("HF_REALTIME_MODEL", "")
+ HF_REALTIME_VOICE = _runtime_value("HF_REALTIME_VOICE", "Aiden")
+
+ CHAT_API_KEY = _runtime_value("CHAT_API_KEY")
+ CHAT_BASE_URL = _runtime_url("CHAT_BASE_URL")
+ CHAT_MODEL_NAME = _runtime_value("CHAT_MODEL_NAME")
+
+ STT_API_KEY = _runtime_value("STT_API_KEY", "not-needed")
+ STT_BASE_URL = _runtime_url("STT_BASE_URL")
+ STT_MODEL_NAME = _runtime_value("STT_MODEL_NAME", "whisper-1")
+ TTS_API_KEY = _runtime_value("TTS_API_KEY", "not-needed")
+ TTS_BASE_URL = _runtime_url("TTS_BASE_URL")
+ TTS_MODEL_NAME = _runtime_value("TTS_MODEL_NAME", "gpt-4o-mini-tts")
+ TTS_VOICE = _runtime_value("TTS_VOICE", OPENAI_REALTIME_VOICE)
+ MIC_TRANSCRIPTION_RMS_THRESHOLD = _runtime_float("MIC_TRANSCRIPTION_RMS_THRESHOLD", 500.0)
+ MIC_TRANSCRIPTION_MIN_AUDIO_MS = _runtime_float("MIC_TRANSCRIPTION_MIN_AUDIO_MS", 250.0)
+ MIC_TRANSCRIPTION_SILENCE_MS = _runtime_float("MIC_TRANSCRIPTION_SILENCE_MS", 800.0)
+ MIC_TRANSCRIPTION_MAX_AUDIO_MS = _runtime_float("MIC_TRANSCRIPTION_MAX_AUDIO_MS", 12_000.0)
+ HF_HOME = _runtime_value("HF_HOME", "./cache")
+ LOCAL_VISION_MODEL = _runtime_value("LOCAL_VISION_MODEL", "HuggingFaceTB/SmolVLM2-2.2B-Instruct")
+ HF_TOKEN = _runtime_value("HF_TOKEN")
logger.debug(
- "Backend: %s, realtime_model=%s, chat_model=%s, STT=%s, TTS=%s, HF mode=%s, HF_HOME=%s, Vision Model=%s",
+ "Backend: %s, realtime_model=%s, chat_model=%s, STT=%s, TTS=%s, HF mode=%s, "
+ "HF_HOME=%s, local_vision_model=%s, routed_vision_default=%s, routed_vision_allowed=%s",
BACKEND_PROVIDER,
OPENAI_REALTIME_MODEL,
CHAT_MODEL_NAME,
@@ -341,6 +434,8 @@ class Config:
HF_REALTIME_CONNECTION_MODE,
HF_HOME,
LOCAL_VISION_MODEL,
+ VISION_DEFAULT_MODEL,
+ VISION_ALLOWED_MODELS,
)
logger.debug(f"Locked profile: {LOCKED_PROFILE}")
logger.debug("Dotenv path: %s", _dotenv_path or "")
@@ -362,6 +457,20 @@ def apply_config_values(values: Mapping[str, str | None], *, inherit_current: bo
)
openai_realtime_model_default = config.OPENAI_REALTIME_MODEL if inherit_current else "gpt-realtime-2"
openai_realtime_voice_default = config.OPENAI_REALTIME_VOICE if inherit_current else "cedar"
+ vision_api_key_default = config.VISION_API_KEY if inherit_current else None
+ vision_base_url_default = config.VISION_BASE_URL if inherit_current else "https://api.openai.com/v1"
+ vision_default_model_default = config.VISION_DEFAULT_MODEL if inherit_current else "gpt-5.4-mini"
+ vision_allowed_models_default = config.VISION_ALLOWED_MODELS if inherit_current else _DEFAULT_VISION_ALLOWED_MODELS
+ tool_transport_default = config.REACHY_TOOL_TRANSPORT if inherit_current else TOOL_TRANSPORT_REST
+ rest_base_url_default = config.REACHY_REST_BASE_URL if inherit_current else "http://127.0.0.1:8000"
+ camera_base_url_default = config.REACHY_CAMERA_BASE_URL if inherit_current else None
+ rest_timeout_default = config.REACHY_REST_TIMEOUT_SECONDS if inherit_current else 5.0
+ motion_duration_default = config.REACHY_MOTION_DURATION_SECONDS if inherit_current else 1.0
+ motion_poll_interval_default = config.REACHY_MOTION_POLL_INTERVAL_SECONDS if inherit_current else 0.1
+ motion_completion_timeout_default = (
+ config.REACHY_MOTION_COMPLETION_TIMEOUT_SECONDS if inherit_current else 10.0
+ )
+ require_routed_vision_default = config.REQUIRE_ROUTED_VISION if inherit_current else False
hf_realtime_connection_mode_default = (
config.HF_REALTIME_CONNECTION_MODE if inherit_current else HF_REALTIME_CONNECTION_DEPLOYED
)
@@ -423,6 +532,57 @@ def apply_config_values(values: Mapping[str, str | None], *, inherit_current: bo
openai_realtime_voice_default,
)
+ config.VISION_API_KEY = _configured_value(_mapping_value(values, "VISION_API_KEY", vision_api_key_default))
+ config.VISION_BASE_URL = _mapping_url(values, "VISION_BASE_URL", vision_base_url_default)
+ config.VISION_DEFAULT_MODEL = (
+ _mapping_value(values, "VISION_DEFAULT_MODEL", vision_default_model_default) or vision_default_model_default
+ )
+ config.VISION_ALLOWED_MODELS = _mapping_csv(
+ values,
+ "VISION_ALLOWED_MODELS",
+ vision_allowed_models_default,
+ )
+
+ config.REACHY_TOOL_TRANSPORT = _normalize_tool_transport(
+ _process_env_value("REACHY_TOOL_TRANSPORT")
+ or _mapping_value(values, "REACHY_TOOL_TRANSPORT", tool_transport_default)
+ )
+ config.REACHY_REST_BASE_URL = _clean_url_value(
+ "REACHY_REST_BASE_URL",
+ _process_env_value("REACHY_REST_BASE_URL")
+ or _mapping_value(values, "REACHY_REST_BASE_URL", rest_base_url_default),
+ )
+ config.REACHY_CAMERA_BASE_URL = _clean_url_value(
+ "REACHY_CAMERA_BASE_URL",
+ _process_env_value("REACHY_CAMERA_BASE_URL")
+ or _mapping_value(values, "REACHY_CAMERA_BASE_URL", camera_base_url_default),
+ )
+ config.REACHY_REST_TIMEOUT_SECONDS = _mapping_float(
+ values,
+ "REACHY_REST_TIMEOUT_SECONDS",
+ rest_timeout_default,
+ )
+ config.REACHY_MOTION_DURATION_SECONDS = _mapping_float(
+ values,
+ "REACHY_MOTION_DURATION_SECONDS",
+ motion_duration_default,
+ )
+ config.REACHY_MOTION_POLL_INTERVAL_SECONDS = _mapping_float(
+ values,
+ "REACHY_MOTION_POLL_INTERVAL_SECONDS",
+ motion_poll_interval_default,
+ )
+ config.REACHY_MOTION_COMPLETION_TIMEOUT_SECONDS = _mapping_float(
+ values,
+ "REACHY_MOTION_COMPLETION_TIMEOUT_SECONDS",
+ motion_completion_timeout_default,
+ )
+ config.REQUIRE_ROUTED_VISION = _parse_bool_value(
+ "REQUIRE_ROUTED_VISION",
+ _process_env_value("REQUIRE_ROUTED_VISION") or _mapping_value(values, "REQUIRE_ROUTED_VISION"),
+ require_routed_vision_default,
+ )
+
config.HF_REALTIME_CONNECTION_MODE = _normalize_hf_connection_mode(
_mapping_value(values, "HF_REALTIME_CONNECTION_MODE", hf_realtime_connection_mode_default)
)
@@ -497,3 +657,10 @@ def loaded_dotenv_path() -> str | None:
def openai_realtime_api_key() -> str | None:
"""Return the OpenAI Realtime key, falling back to the standard OpenAI key."""
return _configured_value(config.OPENAI_REALTIME_API_KEY) or _process_env_value("OPENAI_API_KEY")
+
+
+def vision_api_key() -> str | None:
+ """Return the routed-vision key, falling back to the app's standard OpenAI key."""
+ return (
+ _configured_value(config.VISION_API_KEY) or _process_env_value("OPENAI_API_KEY") or openai_realtime_api_key()
+ )
diff --git a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/console.py b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/console.py
index e1f6a81c..dd81b69b 100644
--- a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/console.py
+++ b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/console.py
@@ -74,7 +74,9 @@ def launch(self) -> None:
missing_reason,
)
else:
- logger.error("%s Add the missing value to .env and restart the conversation app.", missing_reason)
+ logger.error(
+ "%s Add it to .env or the process environment, then restart the conversation app.", missing_reason
+ )
return
# Start media after key is set/available
diff --git a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/conversation_stream.py b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/conversation_stream.py
index 2ea04359..574ecffa 100644
--- a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/conversation_stream.py
+++ b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/conversation_stream.py
@@ -4,8 +4,10 @@
import random
import asyncio
import logging
-from typing import Any, Final, Tuple, Optional, cast
+from typing import Any, Final, Tuple, Callable, Optional, cast
+from pathlib import Path
from datetime import datetime
+from collections import deque
import numpy as np
import gradio as gr
@@ -23,6 +25,7 @@
)
from reachy_mini_conversation_app.prompts import get_session_voice, get_session_instructions
from reachy_mini_conversation_app.audio.pcm import prepare_mono_int16_audio
+from reachy_mini_conversation_app.tool_transport import ToolTransport
from reachy_mini_conversation_app.backend_runtime import (
selected_backend,
backend_config_error,
@@ -35,7 +38,7 @@
)
from reachy_mini_conversation_app.tools.core_tools import (
ToolDependencies,
- get_tool_specs,
+ get_tool_specs_for_dependencies,
)
from reachy_mini_conversation_app.local_stt_backend import LocalSTTBackend
from reachy_mini_conversation_app.realtime_backends import (
@@ -45,6 +48,13 @@
build_realtime_connect_kwargs,
build_realtime_session_config,
)
+from reachy_mini_conversation_app.media_result_processor import (
+ MediaSecurityError,
+ ProcessedToolResult,
+ MediaResultProcessor,
+ contains_raw_media,
+ assert_no_raw_media,
+)
from reachy_mini_conversation_app.tools.background_tool_manager import (
ToolCallRoutine,
ToolNotification,
@@ -62,6 +72,65 @@
IMAGE_INPUT_COST_PER_1M = 5.0
_RESPONSE_DONE_TIMEOUT: Final[float] = 30.0
+_TYPED_TOOL_TIMEOUT: Final[float] = 180.0
+_MODEL_IO_MAX_STRING: Final[int] = 8_000
+_MAX_TOOL_IMAGES: Final[int] = 12
+_MODEL_IO_SECRET_FIELDS: Final[frozenset[str]] = frozenset(
+ {
+ "api_key",
+ "authorization",
+ "hf_token",
+ "openai_api_key",
+ "openai_realtime_api_key",
+ "token",
+ }
+)
+
+
+def _encoded_payload_summary(value: str, media_type: str) -> str:
+ """Describe a Base64 payload without writing the payload to logs."""
+ encoded = value.split(",", 1)[1] if "," in value else value
+ padding = len(encoded) - len(encoded.rstrip("="))
+ estimated_bytes = max(0, (len(encoded) * 3 // 4) - padding)
+ return f"<{media_type}: base64_chars={len(encoded)}, estimated_bytes={estimated_bytes}>"
+
+
+def _sanitize_model_io(value: Any, field_name: str = "") -> Any:
+ """Convert model I/O to JSON-safe values while removing secrets and binary blobs."""
+ normalized_field = field_name.lower()
+ if normalized_field in _MODEL_IO_SECRET_FIELDS:
+ return ""
+
+ model_dump = getattr(value, "model_dump", None)
+ if callable(model_dump):
+ try:
+ value = model_dump(mode="json")
+ except TypeError:
+ value = model_dump()
+
+ if isinstance(value, dict):
+ return {str(key): _sanitize_model_io(item, str(key)) for key, item in value.items()}
+ if isinstance(value, (list, tuple)):
+ return [_sanitize_model_io(item, field_name) for item in value]
+ if isinstance(value, str):
+ if normalized_field == "image_url" and value.startswith("data:image/"):
+ media_type = value.split(";", 1)[0].removeprefix("data:")
+ return _encoded_payload_summary(value, media_type)
+ if normalized_field in {"audio", "b64_im", "b64_images", "delta"} and len(value) > 128:
+ return _encoded_payload_summary(value, normalized_field)
+ if len(value) > _MODEL_IO_MAX_STRING:
+ omitted = len(value) - _MODEL_IO_MAX_STRING
+ return f"{value[:_MODEL_IO_MAX_STRING]}... <{omitted} characters omitted>"
+ return value
+ if value is None or isinstance(value, (bool, int, float)):
+ return value
+
+ return repr(value)
+
+
+def _model_io_json(value: Any) -> str:
+ """Return a compact, redacted JSON representation for debug logs."""
+ return json.dumps(_sanitize_model_io(value), ensure_ascii=False, separators=(",", ":"), default=repr)
def _compute_response_cost(usage: Any) -> float:
@@ -82,7 +151,16 @@ def _compute_response_cost(usage: Any) -> float:
class ConversationStreamHandler(AsyncStreamHandler):
"""Conversation audio/text stream handler for the selected backend."""
- def __init__(self, deps: ToolDependencies, gradio_mode: bool = False, instance_path: Optional[str] = None):
+ def __init__(
+ self,
+ deps: ToolDependencies,
+ gradio_mode: bool = False,
+ instance_path: Optional[str] = None,
+ model_logs: bool = False,
+ tool_transport: ToolTransport | None = None,
+ tool_transport_factory: Callable[[], ToolTransport] | None = None,
+ media_result_processor: MediaResultProcessor | None = None,
+ ):
"""Initialize the handler."""
backend = selected_backend()
stream_sample_rate = backend.stream_sample_rate
@@ -93,6 +171,13 @@ def __init__(self, deps: ToolDependencies, gradio_mode: bool = False, instance_p
)
self.deps = deps
+ if tool_transport is not None and tool_transport_factory is not None:
+ raise ValueError("Provide tool_transport or tool_transport_factory, not both")
+ self._tool_transport_factory = tool_transport_factory
+ self.tool_transport = tool_transport or (tool_transport_factory() if tool_transport_factory else None)
+ self._tool_specs_cache: list[dict[str, Any]] | None = None
+ self._tool_transport_closed = False
+ self.media_result_processor = media_result_processor
self.output_sample_rate = stream_sample_rate
self.input_sample_rate = stream_sample_rate
@@ -101,12 +186,19 @@ def __init__(self, deps: ToolDependencies, gradio_mode: bool = False, instance_p
self.local_stt_backend: LocalSTTBackend | None = None
self._realtime_connect_query: dict[str, str] = {}
self.output_queue: "asyncio.Queue[Tuple[int, NDArray[np.int16]] | AdditionalOutputs]" = asyncio.Queue()
+ self._typed_output_queue: asyncio.Queue[AdditionalOutputs] | None = None
+ self._typed_request_lock: asyncio.Lock = asyncio.Lock()
+ self._typed_tool_calls_awaiting_followup: set[str] = set()
+ self._typed_followup_call_order: deque[str] = deque()
+ self._tool_call_response_ids: set[str] = set()
+ self._chat_response_ids: set[str] = set()
self.last_activity_time = asyncio.get_event_loop().time()
self.start_time = asyncio.get_event_loop().time()
self.is_idle_tool_call = False
self.gradio_mode = gradio_mode
self.instance_path = instance_path
+ self.model_logs = model_logs
# Debouncing for partial transcripts
self.partial_transcript_task: asyncio.Task[None] | None = None
@@ -148,7 +240,24 @@ def __init__(self, deps: ToolDependencies, gradio_mode: bool = False, instance_p
def copy(self) -> "ConversationStreamHandler":
"""Create a copy of the handler."""
- return ConversationStreamHandler(self.deps, self.gradio_mode, self.instance_path)
+ return ConversationStreamHandler(
+ self.deps,
+ self.gradio_mode,
+ self.instance_path,
+ self.model_logs,
+ tool_transport=self.tool_transport if self._tool_transport_factory is None else None,
+ tool_transport_factory=self._tool_transport_factory,
+ media_result_processor=self.media_result_processor,
+ )
+
+ async def _available_tool_specs(self) -> list[dict[str, Any]]:
+ """Return and cache the schemas advertised by the selected transport."""
+ if self._tool_specs_cache is None:
+ if self.tool_transport is None:
+ self._tool_specs_cache = get_tool_specs_for_dependencies(self.deps)
+ else:
+ self._tool_specs_cache = await self.tool_transport.list_tools()
+ return self._tool_specs_cache
def _record_startup_error(self, message: str) -> None:
"""Store a startup failure so the UI can show a useful error."""
@@ -176,7 +285,7 @@ def _record_backend_config_error(self, error: str) -> None:
"Then restart the conversation app."
)
else:
- message = f"{error} Add the missing value to .env and restart the conversation app."
+ message = f"{error} Add it to .env or the process environment, then restart the conversation app."
self._record_startup_error(message)
logger.error(message)
@@ -192,6 +301,8 @@ def _get_local_stt_backend(self) -> LocalSTTBackend:
deps=self.deps,
tool_manager=self.tool_manager,
client_factory=AsyncOpenAI,
+ tool_transport=self.tool_transport,
+ media_result_processor=self.media_result_processor,
)
return self.local_stt_backend
@@ -358,7 +469,51 @@ async def _report_microphone_error_once(self, message: str) -> None:
if self._microphone_error_reported:
return
self._microphone_error_reported = True
- await self.output_queue.put(AdditionalOutputs({"role": "assistant", "content": message}))
+ await self._publish_chat_output({"role": "assistant", "content": message})
+
+ async def _publish_chat_output(self, message: dict[str, Any]) -> None:
+ """Publish a chat update to the live stream and any active typed request.
+
+ FastRTC continuously drains ``output_queue``. Typed Gradio callbacks
+ therefore need their own copy so the stream cannot consume the model's
+ final transcript before ``send_text_message`` sees it.
+ """
+ output = AdditionalOutputs(message)
+ await self.output_queue.put(output)
+ typed_output_queue = self._typed_output_queue
+ if typed_output_queue is not None:
+ await typed_output_queue.put(output)
+
+ async def _create_conversation_item(self, item: dict[str, Any]) -> None:
+ """Log and send a Realtime conversation.item.create request."""
+ self._log_model_request("conversation.item.create", item)
+ await self.connection.conversation.item.create(item=item)
+
+ def _log_model_request(self, request_type: str, payload: Any) -> None:
+ """Log a sanitized model request at the selected detail level."""
+ log = logger.info if self.model_logs else logger.debug
+ log("MODEL request type=%s payload=%s", request_type, _model_io_json(payload))
+
+ @staticmethod
+ def _response_message_text(response: Any) -> str:
+ """Extract final assistant text or audio transcript from response.done."""
+ output_items = getattr(response, "output", None)
+ if not isinstance(output_items, list):
+ return ""
+
+ text_parts: list[str] = []
+ for item in output_items:
+ if getattr(item, "type", None) != "message" or getattr(item, "role", None) != "assistant":
+ continue
+ content_parts = getattr(item, "content", None)
+ if not isinstance(content_parts, list):
+ continue
+ for content_part in content_parts:
+ text = getattr(content_part, "transcript", None) or getattr(content_part, "text", None)
+ if isinstance(text, str) and text.strip():
+ text_parts.append(text.strip())
+
+ return "\n".join(text_parts)
@staticmethod
def _chat_message_from_output(output: AdditionalOutputs) -> dict[str, Any] | None:
@@ -374,7 +529,50 @@ def _chat_message_from_output(output: AdditionalOutputs) -> dict[str, Any] | Non
return None
return candidate
- async def send_text_message(self, message: str, timeout: float = 30.0) -> list[dict[str, Any]]:
+ @staticmethod
+ def _is_final_assistant_message(message: dict[str, Any]) -> bool:
+ """Return whether a chat update is a user-facing assistant answer.
+
+ Tool lifecycle cards and camera previews also use the assistant role in
+ Gradio, but they are intermediate updates. Treating either as the final
+ answer makes typed requests return before the post-tool model response.
+ """
+ if message.get("role") != "assistant" or message.get("metadata"):
+ return False
+
+ content = message.get("content")
+ if not isinstance(content, str):
+ return False
+
+ return not content.startswith("🛠️ Used tool")
+
+ def _typed_turn_is_complete(self, saw_assistant_message: bool) -> bool:
+ """Return whether typed chat has a final answer and no pending tool follow-up."""
+ return (
+ saw_assistant_message
+ and self._response_done_event.is_set()
+ and not self._typed_tool_calls_awaiting_followup
+ )
+
+ def _mark_typed_followup_response(self, response_id: Any) -> None:
+ """Match a non-tool response to the next tool awaiting a spoken follow-up."""
+ if not self._typed_tool_calls_awaiting_followup:
+ return
+ if isinstance(response_id, str) and response_id in self._tool_call_response_ids:
+ return
+ if not self._typed_followup_call_order:
+ return
+
+ call_id = self._typed_followup_call_order.popleft()
+ self._typed_tool_calls_awaiting_followup.discard(call_id)
+ logger.debug("Typed tool follow-up completed for call_id=%s response_id=%s", call_id, response_id)
+
+ async def send_text_message(
+ self,
+ message: str,
+ timeout: float = 30.0,
+ tool_timeout: float = _TYPED_TOOL_TIMEOUT,
+ ) -> list[dict[str, Any]]:
"""Send a typed user message and collect chat updates."""
text = message.strip()
if not text:
@@ -401,42 +599,89 @@ async def send_text_message(self, message: str, timeout: float = 30.0) -> list[d
},
]
- await self.connection.conversation.item.create(
- item={
+ async with self._typed_request_lock:
+ typed_output_queue: asyncio.Queue[AdditionalOutputs] = asyncio.Queue()
+ self._typed_output_queue = typed_output_queue
+ self._typed_tool_calls_awaiting_followup.clear()
+ self._typed_followup_call_order.clear()
+ self._tool_call_response_ids.clear()
+ try:
+ return await self._send_realtime_text_message(text, timeout, tool_timeout, typed_output_queue)
+ finally:
+ self._typed_output_queue = None
+ self._typed_tool_calls_awaiting_followup.clear()
+ self._typed_followup_call_order.clear()
+ self._tool_call_response_ids.clear()
+
+ async def _send_realtime_text_message(
+ self,
+ text: str,
+ timeout: float,
+ tool_timeout: float,
+ typed_output_queue: asyncio.Queue[AdditionalOutputs],
+ ) -> list[dict[str, Any]]:
+ """Send one typed Realtime turn and collect its dedicated chat updates."""
+ await self._create_conversation_item(
+ {
"type": "message",
"role": "user",
"content": [{"type": "input_text", "text": text}],
- },
+ }
)
await self._safe_response_create()
messages: list[dict[str, Any]] = [{"role": "user", "content": text}]
saw_assistant_message = False
- deadline = asyncio.get_event_loop().time() + timeout
+ turn_completed = False
+ loop = asyncio.get_event_loop()
+ response_deadline = loop.time() + timeout
+ active_tool_deadline: float | None = None
+
+ while True:
+ now = loop.time()
+ has_pending_tool = bool(self._typed_tool_calls_awaiting_followup)
+ if has_pending_tool and active_tool_deadline is None:
+ active_tool_deadline = now + tool_timeout
+ logger.debug("Extended typed request timeout for active tool by %.1f seconds", tool_timeout)
+ elif not has_pending_tool and active_tool_deadline is not None:
+ active_tool_deadline = None
+ response_deadline = now + timeout
+
+ if has_pending_tool:
+ assert active_tool_deadline is not None
+ deadline = active_tool_deadline
+ else:
+ deadline = response_deadline
+ if now >= deadline:
+ break
- while asyncio.get_event_loop().time() < deadline:
try:
- output = await asyncio.wait_for(self.output_queue.get(), timeout=0.5)
+ output = await asyncio.wait_for(typed_output_queue.get(), timeout=min(0.5, deadline - now))
except asyncio.TimeoutError:
- if saw_assistant_message and self._response_done_event.is_set():
+ if self._typed_turn_is_complete(saw_assistant_message):
+ turn_completed = True
break
continue
- if not isinstance(output, AdditionalOutputs):
- continue
-
+ response_deadline = loop.time() + timeout
chat_message = self._chat_message_from_output(output)
if chat_message is None:
continue
messages.append(chat_message)
- if chat_message.get("role") == "assistant":
+ if self._is_final_assistant_message(chat_message):
saw_assistant_message = True
- if saw_assistant_message and self._response_done_event.is_set():
+ if self._typed_turn_is_complete(saw_assistant_message):
+ turn_completed = True
break
- if not saw_assistant_message:
+ if not turn_completed:
+ logger.warning(
+ "Typed Realtime request timed out (pending_tools=%s, saw_assistant=%s)",
+ sorted(self._typed_tool_calls_awaiting_followup),
+ saw_assistant_message,
+ )
messages.append({"role": "assistant", "content": "[error] Timed out waiting for a response."})
return messages
@@ -468,6 +713,14 @@ async def _emit_debounced_partial(self, transcript: str, sequence: int) -> None:
async def start_up(self) -> None:
"""Start the handler with minimal retries on unexpected websocket closure."""
+ try:
+ await self._available_tool_specs()
+ except Exception as e:
+ message = f"Tool discovery failed: {type(e).__name__}: {e}"
+ self._record_startup_error(message)
+ logger.exception(message)
+ return
+
if not self._text_model_uses_realtime():
logger.info(
"Skipping Realtime startup because BACKEND_PROVIDER=%r uses local STT/chat", config.BACKEND_PROVIDER
@@ -602,7 +855,7 @@ async def _response_sender_loop(self) -> None:
try:
await asyncio.wait_for(self._response_done_event.wait(), timeout=_RESPONSE_DONE_TIMEOUT)
except asyncio.TimeoutError:
- logger.debug("Timed out waiting for previous response to finish; forcing ahead")
+ logger.warning("Timed out waiting for previous response to finish; forcing ahead")
self._response_done_event.set()
if not self.connection:
@@ -611,16 +864,19 @@ async def _response_sender_loop(self) -> None:
self._last_response_rejected = False
try:
self._response_done_event.clear()
+ logger.info("Sending queued Realtime response")
+ self._log_model_request("response.create", kwargs)
await self.connection.response.create(**kwargs)
+ logger.info("Realtime response request sent; waiting for model output")
except Exception as e:
- logger.debug("_response_sender_loop: send failed: %s", e)
+ logger.exception("Failed to send queued Realtime response: %s", e)
self._response_done_event.set()
break
try:
await asyncio.wait_for(self._response_done_event.wait(), timeout=_RESPONSE_DONE_TIMEOUT)
except asyncio.TimeoutError:
- logger.debug("Timed out waiting for response.done; assuming response completed")
+ logger.warning("Timed out waiting for response.done; assuming response completed")
self._response_done_event.set()
break
@@ -628,7 +884,7 @@ async def _response_sender_loop(self) -> None:
if self._last_response_rejected:
attempts += 1
if attempts >= max_retries:
- logger.debug("response.create rejected %d times; giving up", attempts)
+ logger.warning("response.create rejected %d times; giving up", attempts)
break
logger.debug("response.create was rejected; retrying (%d/%d)", attempts, max_retries)
continue
@@ -637,17 +893,20 @@ async def _response_sender_loop(self) -> None:
async def _handle_tool_result(self, bg_tool: ToolNotification) -> None:
"""Process the result of a tool call."""
- if bg_tool.error is not None:
+ if bg_tool.result is not None:
+ tool_result = bg_tool.result
+ if bg_tool.error is not None:
+ logger.error("Tool '%s' (id=%s) failed with error: %s", bg_tool.tool_name, bg_tool.id, bg_tool.error)
+ else:
+ logger.info(
+ "Tool '%s' (id=%s) executed successfully.",
+ bg_tool.tool_name,
+ bg_tool.id,
+ )
+ logger.debug("TOOL response name=%s result=%s", bg_tool.tool_name, _model_io_json(tool_result))
+ elif bg_tool.error is not None:
logger.error("Tool '%s' (id=%s) failed with error: %s", bg_tool.tool_name, bg_tool.id, bg_tool.error)
tool_result = {"error": bg_tool.error}
- elif bg_tool.result is not None:
- tool_result = bg_tool.result
- logger.info(
- "Tool '%s' (id=%s) executed successfully.",
- bg_tool.tool_name,
- bg_tool.id,
- )
- logger.debug("Tool '%s' full result: %s", bg_tool.tool_name, tool_result)
else:
logger.warning("Tool '%s' (id=%s) returned no result and no error", bg_tool.tool_name, bg_tool.id)
tool_result = {"error": "No result returned from tool execution"}
@@ -662,78 +921,204 @@ async def _handle_tool_result(self, bg_tool: ToolNotification) -> None:
return
try:
+ vision_images: list[str] = []
+ processed_media: ProcessedToolResult | None = None
+ if self.media_result_processor is not None:
+ try:
+ processed_media = await self.media_result_processor.process(bg_tool.tool_name, tool_result)
+ model_tool_result = processed_media.model_payload
+ except MediaSecurityError:
+ logger.error("Raw media was rejected before Realtime serialization")
+ model_tool_result = {
+ "status": "media_security_error",
+ "tool": bg_tool.tool_name,
+ "error": "Raw media was rejected before reaching the conversation model",
+ }
+ elif config.REQUIRE_ROUTED_VISION and contains_raw_media(tool_result):
+ logger.error("Raw media reached a strict Realtime handler without a media processor")
+ model_tool_result = {
+ "status": "media_security_error",
+ "tool": bg_tool.tool_name,
+ "error": "Routed vision is required; raw media was discarded",
+ }
+ else:
+ model_tool_result = dict(tool_result)
+ if bg_tool.tool_name == "camera" and "b64_im" in model_tool_result:
+ raw_camera_image = model_tool_result.pop("b64_im")
+ if isinstance(raw_camera_image, str) and raw_camera_image:
+ vision_images = [raw_camera_image]
+ model_tool_result["status"] = "image_captured"
+ else:
+ logger.warning("Unexpected camera image type: %s", type(raw_camera_image))
+ model_tool_result = {"error": "Camera returned an invalid image"}
+ elif bg_tool.tool_name == "scan_scene" and "b64_images" in model_tool_result:
+ raw_scan_images = model_tool_result.pop("b64_images")
+ if (
+ isinstance(raw_scan_images, list)
+ and 0 < len(raw_scan_images) <= _MAX_TOOL_IMAGES
+ and all(isinstance(image, str) and image for image in raw_scan_images)
+ ):
+ vision_images = raw_scan_images
+ else:
+ logger.warning("Unexpected scene-scan image payload")
+ model_tool_result = {"error": "Scene scan returned invalid analysis images"}
+
+ assert_no_raw_media(model_tool_result)
+ serialized_tool_result = json.dumps(model_tool_result)
+
# Send the tool result back
if isinstance(bg_tool.id, str):
- await self.connection.conversation.item.create(
- item={
+ await self._create_conversation_item(
+ {
"type": "function_call_output",
"call_id": bg_tool.id,
- "output": json.dumps(tool_result),
- },
+ "output": serialized_tool_result,
+ }
)
- await self.output_queue.put(
- AdditionalOutputs(
- {
- "role": "assistant",
- "content": json.dumps(tool_result),
- # Gradio UI metadata.status accept only "pending" and "done". Do not accept bg.tool.status values.
- "metadata": {
- "title": f"🛠️ Used tool {bg_tool.tool_name}",
- "status": "done",
- },
+ policy_denied = model_tool_result.get("status") == "policy_denied"
+ await self._publish_chat_output(
+ {
+ "role": "assistant",
+ "content": serialized_tool_result,
+ # Gradio UI metadata.status accept only "pending" and "done". Do not accept bg.tool.status values.
+ "metadata": {
+ "title": (
+ f"🚫 OpenShell blocked tool {bg_tool.tool_name}"
+ if policy_denied
+ else f"🛠️ Used tool {bg_tool.tool_name}"
+ ),
+ "status": "done",
},
- ),
+ },
)
- if bg_tool.tool_name == "camera" and "b64_im" in tool_result:
- # use raw base64, don't json.dumps (which adds quotes)
- b64_im = tool_result["b64_im"]
- if not isinstance(b64_im, str):
- logger.warning("Unexpected type for b64_im: %s", type(b64_im))
- b64_im = str(b64_im)
- await self.connection.conversation.item.create(
- item={
+ if vision_images:
+ image_content: list[dict[str, Any]] = []
+ if bg_tool.tool_name == "scan_scene":
+ question = model_tool_result.get("question", "Describe everything visible during the scan.")
+ timestamps = model_tool_result.get("frame_timestamps_seconds", [])
+ image_content.append(
+ {
+ "type": "input_text",
+ "text": (
+ "These are chronological frames sampled across one Reachy scene sweep. "
+ f"Frame timestamps in seconds: {timestamps}. "
+ "Combine evidence across all frames, deduplicate repeated people and objects, "
+ "and do not claim details that are not visibly supported. "
+ f"User question: {question}"
+ ),
+ }
+ )
+ image_content.extend(
+ {
+ "type": "input_image",
+ "image_url": f"data:image/jpeg;base64,{image}",
+ }
+ for image in vision_images
+ )
+ await self._create_conversation_item(
+ {
"type": "message",
"role": "user",
- "content": [
- {
- "type": "input_image",
- "image_url": f"data:image/jpeg;base64,{b64_im}",
- },
- ],
+ "content": image_content,
+ }
+ )
+ logger.info("Added %d image(s) from tool '%s' to conversation", len(vision_images), bg_tool.tool_name)
+
+ # Show the local camera preview even when a dedicated vision model
+ # consumed the raw image and only returned a text description.
+ preview_image = processed_media.preview_image if processed_media is not None else None
+ if preview_image is None and (
+ bg_tool.tool_name == "camera"
+ and self.deps.camera_worker is not None
+ and (vision_images or model_tool_result.get("status") == "image_analyzed")
+ ):
+ np_img = self.deps.camera_worker.get_latest_frame()
+ if np_img is not None:
+ import cv2
+
+ # Camera frames are BGR from OpenCV; convert so Gradio displays correct colors.
+ preview_image = cv2.cvtColor(np_img, cv2.COLOR_BGR2RGB)
+
+ if preview_image is not None:
+ img = gr.Image(value=preview_image)
+
+ await self._publish_chat_output(
+ {
+ "role": "assistant",
+ "content": img,
},
)
- logger.info("Added camera image to conversation")
- if self.deps.camera_worker is not None:
- np_img = self.deps.camera_worker.get_latest_frame()
- if np_img is not None:
- import cv2
-
- # Camera frames are BGR from OpenCV; convert so Gradio displays correct colors.
- rgb_frame = cv2.cvtColor(np_img, cv2.COLOR_BGR2RGB)
- else:
- rgb_frame = None
- img = gr.Image(value=rgb_frame)
-
- await self.output_queue.put(
- AdditionalOutputs(
- {
- "role": "assistant",
- "content": img,
- },
- ),
+ if bg_tool.tool_name == "scan_scene":
+ video_path: str | Path | None = (
+ processed_media.video_path if processed_media is not None else model_tool_result.get("video_path")
+ )
+ if isinstance(video_path, (str, Path)) and Path(video_path).is_file():
+ await self._publish_chat_output(
+ {
+ "role": "assistant",
+ "content": gr.Video(value=str(video_path)),
+ },
)
# If this tool call was triggered by an idle signal, don't make the robot speak.
# For other tool calls, let the robot reply out loud.
if not bg_tool.is_idle_tool_call:
+ follow_up_instructions = "Use the tool result just returned and answer concisely in speech."
+ if policy_denied:
+ follow_up_instructions = (
+ f"Tell the user that the requested {bg_tool.tool_name} action was blocked by the "
+ "OpenShell policy. Do not claim the robot lacks the physical capability, do not imply "
+ "that the action succeeded, and do not retry the tool."
+ )
+ elif bg_tool.tool_name == "camera" and vision_images:
+ follow_up_instructions = (
+ "Answer the user's camera question using the input image just added. "
+ "Describe only what is visibly supported by the image, and answer concisely in speech."
+ )
+ elif bg_tool.tool_name == "camera" and model_tool_result.get("status") == "image_analyzed":
+ follow_up_instructions = (
+ "Relay the image_description returned by the approved vision model. "
+ "Answer the user's camera question concisely in speech and do not claim that you "
+ "personally received the raw image."
+ )
+ elif bg_tool.tool_name == "scan_scene" and vision_images:
+ follow_up_instructions = (
+ "Answer the user's scene-scan question using every chronological image just added. "
+ "Give one concise combined account, deduplicate things visible in multiple frames, and "
+ "describe only visibly supported details. Mention that the recording was saved, but do "
+ "not read the full local filesystem path aloud."
+ )
+ elif bg_tool.tool_name == "scan_scene" and model_tool_result.get("status") == "scene_analyzed":
+ if model_tool_result.get("scan_status") == "scene_scan_incomplete":
+ follow_up_instructions = (
+ "Explain that Reachy's physical scene sweep was interrupted and may be incomplete. "
+ "Use scan_warning and returned_to_front from the tool result to say whether Reachy "
+ "recovered to its front pose. Then relay the approved vision model's description of "
+ "the frames that were successfully recorded. Do not claim a complete room scan."
+ )
+ elif model_tool_result.get("recording_status") == "preview_unavailable":
+ follow_up_instructions = (
+ "Relay the image_description returned by the approved vision model as one concise "
+ "combined account. Briefly mention that the recording preview is unavailable, without "
+ "exposing internal paths."
+ )
+ else:
+ follow_up_instructions = (
+ "Relay the image_description returned by the approved vision model as one concise "
+ "combined account. Mention that the recording was saved, but do not expose internal paths."
+ )
await self._safe_response_create(
response={
- "instructions": "Use the tool result just returned and answer concisely in speech.",
+ "instructions": follow_up_instructions,
+ "tool_choice": "none",
},
)
+ if isinstance(bg_tool.id, str) and bg_tool.id in self._typed_tool_calls_awaiting_followup:
+ self._typed_followup_call_order.append(bg_tool.id)
+ logger.info("Queued spoken follow-up for tool '%s'", bg_tool.tool_name)
# Re-synchronize the head wobble after a tool call that may have taken some time
if self.deps.head_wobbler is not None:
@@ -758,16 +1143,27 @@ async def _run_realtime_session(self) -> None:
output_sample_rate=self.output_sample_rate,
instructions=get_session_instructions(),
voice=session_voice,
- tools=get_tool_specs(),
+ tools=await self._available_tool_specs(),
transcription_language=config.REALTIME_TRANSCRIPTION_LANGUAGE,
)
+ logger.info("Realtime model connection: %s", realtime_context())
+ logger.debug("MODEL request session.update=%s", _model_io_json({"session": session_config}))
await conn.session.update(session=cast(Any, session_config))
logger.info(
- "Realtime session initialized with backend=%r locked_profile=%r voice=%r",
+ "Realtime session initialized with backend=%r model=%r locked_profile=%r voice=%r tools=%s",
backend.provider,
+ backend.realtime_model,
LOCKED_PROFILE,
session_voice,
+ [tool.get("name", "") for tool in session_config["tools"]],
)
+ if self.model_logs:
+ logger.info(
+ "MODEL selected provider=%s model=%s voice=%s",
+ backend.provider,
+ backend.realtime_model,
+ session_voice,
+ )
except Exception as e:
message = (
f"Realtime session.update failed ({realtime_context()}): "
@@ -801,11 +1197,13 @@ async def _run_realtime_session(self) -> None:
self._clear_queue()
if self.deps.head_wobbler is not None:
self.deps.head_wobbler.reset()
- self.deps.movement_manager.set_listening(True)
+ if self.deps.movement_manager is not None:
+ self.deps.movement_manager.set_listening(True)
logger.debug("User speech started")
if event.type == "input_audio_buffer.speech_stopped":
- self.deps.movement_manager.set_listening(False)
+ if self.deps.movement_manager is not None:
+ self.deps.movement_manager.set_listening(False)
logger.debug("User speech stopped - server will auto-commit with VAD")
if event.type in (
@@ -823,14 +1221,41 @@ async def _run_realtime_session(self) -> None:
if event.type == "response.done":
# Doesn't mean the audio is done playing
self._response_done_event.set()
- logger.debug("Response done")
-
response = getattr(event, "response", None)
+ response_status = getattr(response, "status", "unknown") if response else "unknown"
+ logger.info("Realtime response completed (status=%s)", response_status)
+ logger.debug("MODEL response response.done=%s", _model_io_json(response))
+
+ # The normal transcript/text events are preferred, but
+ # response.done also contains the complete assistant
+ # message. Use it as a fallback for compatible backends
+ # that omit the dedicated done event.
+ response_id = getattr(response, "id", None) if response else None
+ if isinstance(response_id, str) and response_id not in self._chat_response_ids:
+ response_text = self._response_message_text(response)
+ if response_text:
+ self._mark_typed_followup_response(response_id)
+ logger.info(
+ "Recovered assistant text from response.done (%d characters)", len(response_text)
+ )
+ await self._publish_chat_output({"role": "assistant", "content": response_text})
+ self._chat_response_ids.add(response_id)
+
usage = getattr(response, "usage", None) if response else None
if usage:
cost = _compute_response_cost(usage)
self.cumulative_cost += cost
- logger.debug("Cost: $%.4f | Cumulative: $%.4f", cost, self.cumulative_cost)
+ if self.model_logs:
+ logger.info(
+ "MODEL usage model=%s response_id=%s tokens=%s cost_usd=%.6f cumulative_cost_usd=%.6f",
+ selected_backend().realtime_model,
+ response_id,
+ _model_io_json(usage),
+ cost,
+ self.cumulative_cost,
+ )
+ else:
+ logger.debug("Cost: $%.4f | Cumulative: $%.4f", cost, self.cumulative_cost)
else:
logger.warning("No usage data available for cost tracking")
@@ -881,7 +1306,27 @@ async def _run_realtime_session(self) -> None:
if not isinstance(transcript, str):
transcript = ""
logger.debug(f"Assistant transcript: {transcript}")
- await self.output_queue.put(AdditionalOutputs({"role": "assistant", "content": transcript}))
+ logger.info("Received assistant transcript (%d characters)", len(transcript))
+ logger.debug("MODEL response assistant.transcript=%s", _model_io_json(transcript))
+ await self._publish_chat_output({"role": "assistant", "content": transcript})
+ response_id = getattr(event, "response_id", None)
+ self._mark_typed_followup_response(response_id)
+ if isinstance(response_id, str):
+ self._chat_response_ids.add(response_id)
+
+ # Some Realtime responses use text content instead of an
+ # audio transcript. Surface those in the same chat path.
+ if event.type in ("response.text.done", "response.output_text.done"):
+ text = getattr(event, "text", "")
+ if not isinstance(text, str):
+ text = ""
+ logger.info("Received assistant text (%d characters)", len(text))
+ logger.debug("MODEL response assistant.text=%s", _model_io_json(text))
+ await self._publish_chat_output({"role": "assistant", "content": text})
+ response_id = getattr(event, "response_id", None)
+ self._mark_typed_followup_response(response_id)
+ if isinstance(response_id, str):
+ self._chat_response_ids.add(response_id)
# Handle audio delta
if event.type in ("response.audio.delta", "response.output_audio.delta"):
@@ -913,6 +1358,17 @@ async def _run_realtime_session(self) -> None:
self.is_idle_tool_call,
args_json_str,
)
+ logger.debug(
+ "MODEL response function_call=%s",
+ _model_io_json(
+ {
+ "name": tool_name,
+ "call_id": call_id,
+ "arguments": args_json_str,
+ "is_idle": self.is_idle_tool_call,
+ }
+ ),
+ )
if not isinstance(tool_name, str) or not isinstance(args_json_str, str):
logger.error(
@@ -925,23 +1381,28 @@ async def _run_realtime_session(self) -> None:
)
continue
+ response_id = getattr(event, "response_id", None)
+ if self._typed_output_queue is not None and not self.is_idle_tool_call:
+ self._typed_tool_calls_awaiting_followup.add(call_id)
+ if isinstance(response_id, str):
+ self._tool_call_response_ids.add(response_id)
+
bg_tool = await self.tool_manager.start_tool(
call_id=call_id,
tool_call_routine=ToolCallRoutine(
tool_name=tool_name,
args_json_str=args_json_str,
deps=self.deps,
+ transport=self.tool_transport,
),
is_idle_tool_call=self.is_idle_tool_call,
)
- await self.output_queue.put(
- AdditionalOutputs(
- {
- "role": "assistant",
- "content": f"🛠️ Used tool {tool_name} with args {args_json_str}. The tool is now running. Tool ID: {bg_tool.tool_id}",
- },
- ),
+ await self._publish_chat_output(
+ {
+ "role": "assistant",
+ "content": f"🛠️ Used tool {tool_name} with args {args_json_str}. The tool is now running. Tool ID: {bg_tool.tool_id}",
+ },
)
if self.is_idle_tool_call:
@@ -968,9 +1429,7 @@ async def _run_realtime_session(self) -> None:
# Only show user-facing errors, not internal state errors
if code not in ("input_audio_buffer_commit_empty",):
- await self.output_queue.put(
- AdditionalOutputs({"role": "assistant", "content": f"[error] {msg}"})
- )
+ await self._publish_chat_output({"role": "assistant", "content": f"[error] {msg}"})
finally:
# Stop the response sender worker.
if response_sender_task is not None:
@@ -1007,7 +1466,7 @@ async def receive(self, frame: Tuple[int, NDArray[Any]]) -> None:
config_error = backend_config_error()
if config_error:
await self._report_microphone_error_once(
- f"[error] {config_error} Add the missing value to .env and restart the conversation app."
+ f"[error] {config_error} Add it to .env or the process environment, then restart the conversation app."
)
return
await self._receive_transcribed_text_frame(frame)
@@ -1036,7 +1495,12 @@ async def emit(self) -> Tuple[int, NDArray[np.int16]] | AdditionalOutputs | None
# Handle idle
idle_duration = asyncio.get_event_loop().time() - self.last_activity_time
- if self._text_model_uses_realtime() and idle_duration > 15.0 and self.deps.movement_manager.is_idle():
+ if (
+ self._text_model_uses_realtime()
+ and idle_duration > 15.0
+ and self.deps.movement_manager is not None
+ and self.deps.movement_manager.is_idle()
+ ):
try:
await self.send_idle_signal(idle_duration)
except Exception as e:
@@ -1085,6 +1549,10 @@ async def shutdown(self) -> None:
finally:
self.connection = None
+ if self.tool_transport is not None and not self._tool_transport_closed:
+ self._tool_transport_closed = True
+ await self.tool_transport.close()
+
# Clear any remaining items in the output queue
while not self.output_queue.empty():
try:
@@ -1107,12 +1575,12 @@ async def send_idle_signal(self, idle_duration: float) -> None:
if not self.connection:
logger.debug("No connection, cannot send idle signal")
return
- await self.connection.conversation.item.create(
- item={
+ await self._create_conversation_item(
+ {
"type": "message",
"role": "user",
"content": [{"type": "input_text", "text": timestamp_msg}],
- },
+ }
)
await self._safe_response_create(
response={
diff --git a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/local_stt_backend.py b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/local_stt_backend.py
index 2e1dade1..9b1636f7 100644
--- a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/local_stt_backend.py
+++ b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/local_stt_backend.py
@@ -7,9 +7,11 @@
from numpy.typing import NDArray
from reachy_mini_conversation_app.config import config
+from reachy_mini_conversation_app.tool_transport import ToolTransport
from reachy_mini_conversation_app.chat_completions import ChatCompletionRunner
from reachy_mini_conversation_app.speech_endpoints import SpeechEndpointClient
from reachy_mini_conversation_app.tools.core_tools import ToolDependencies
+from reachy_mini_conversation_app.media_result_processor import MediaResultProcessor
from reachy_mini_conversation_app.tools.background_tool_manager import BackgroundToolManager
@@ -22,10 +24,14 @@ def __init__(
deps: ToolDependencies,
tool_manager: BackgroundToolManager,
client_factory: Callable[..., Any],
+ tool_transport: ToolTransport | None = None,
+ media_result_processor: MediaResultProcessor | None = None,
) -> None:
"""Initialize the local-STT backend adapter."""
self.deps = deps
self.tool_manager = tool_manager
+ self.tool_transport = tool_transport
+ self.media_result_processor = media_result_processor
self.client_factory = client_factory
self._speech_endpoint_client: SpeechEndpointClient | None = None
self._chat_client: Any = None
@@ -96,6 +102,8 @@ async def send_text_message(self, text: str) -> list[dict[str, Any]]:
client=self._get_chat_client(),
deps=self.deps,
tool_manager=self.tool_manager,
+ media_result_processor=self.media_result_processor,
+ tool_transport=self.tool_transport,
model_name=config.CHAT_MODEL_NAME or "",
base_url=config.CHAT_BASE_URL,
).send_text_message(text)
diff --git a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/main.py b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/main.py
index fcccefe1..ce5b3bc3 100644
--- a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/main.py
+++ b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/main.py
@@ -1,19 +1,16 @@
"""Entrypoint for the Reachy Mini conversation app."""
+from __future__ import annotations
import os
import sys
-import time
-import asyncio
import argparse
import threading
-from typing import Any, Dict, List, Optional
+from typing import Any, Dict, List, Callable, Optional
from pathlib import Path
-from reachy_mini import ReachyMini, ReachyMiniApp
from reachy_mini_conversation_app.utils import (
parse_args,
setup_logger,
- handle_vision_stuff,
log_connection_troubleshooting,
)
@@ -25,7 +22,7 @@ def update_chatbot(chatbot: List[Dict[str, Any]], response: Dict[str, Any]) -> L
def _shutdown_step(logger: Any, name: str, callback: Any) -> None:
- """Run one shutdown callback without turning cleanup interrupts into tracebacks."""
+ """Preserve the original cleanup helper for callers and regression tests."""
try:
callback()
except KeyboardInterrupt:
@@ -34,6 +31,50 @@ def _shutdown_step(logger: Any, name: str, callback: Any) -> None:
logger.debug("Error while stopping %s: %s", name, exc)
+def _build_tool_transport_factory(
+ mode: str,
+ dependencies: Any,
+ *,
+ rest_base_url: str | None = None,
+ camera_base_url: str | None = None,
+ rest_timeout_seconds: float = 5.0,
+ motion_duration_seconds: float = 1.0,
+ motion_poll_interval_seconds: float = 0.1,
+ motion_completion_timeout_seconds: float = 10.0,
+) -> Callable[[], Any]:
+ """Build per-conversation transports without opening a network connection yet."""
+ from reachy_mini_conversation_app.tool_transport import (
+ LocalToolTransport,
+ RoutedToolTransport,
+ ConversationUtilityTransport,
+ )
+ from reachy_mini_conversation_app.rest_tool_transport import RestToolTransport, RestTransportSettings
+
+ if mode == "local":
+ return lambda: LocalToolTransport(dependencies)
+ if mode != "rest":
+ raise ValueError(f"Unsupported tool transport: {mode!r}")
+ if not rest_base_url or rest_base_url.strip().lower() in {"", "", "set-me"}:
+ raise ValueError("REACHY_REST_BASE_URL must be set when REACHY_TOOL_TRANSPORT=rest")
+
+ settings = RestTransportSettings(
+ base_url=rest_base_url,
+ camera_base_url=camera_base_url,
+ request_timeout_seconds=rest_timeout_seconds,
+ motion_duration_seconds=motion_duration_seconds,
+ poll_interval_seconds=motion_poll_interval_seconds,
+ completion_timeout_seconds=motion_completion_timeout_seconds,
+ )
+
+ def create_rest_transport() -> Any:
+ return RoutedToolTransport(
+ remote=RestToolTransport(settings),
+ local=ConversationUtilityTransport(),
+ )
+
+ return create_rest_transport
+
+
def main() -> None:
"""Entrypoint for the Reachy Mini conversation app."""
args, _ = parse_args()
@@ -42,7 +83,7 @@ def main() -> None:
def run(
args: argparse.Namespace,
- robot: ReachyMini | None = None,
+ robot: Any | None = None,
app_stop_event: Optional[threading.Event] = None,
settings_app: Optional[Any] = None,
instance_path: Optional[str] = None,
@@ -59,80 +100,104 @@ def run(
) from exc
# Putting these dependencies here makes the dashboard faster to load when the conversation app is installed
- from reachy_mini_conversation_app.moves import MovementManager
- from reachy_mini_conversation_app.config import load_dotenv_file
- from reachy_mini_conversation_app.console import LocalStream
+ from reachy_mini_conversation_app.config import (
+ TOOL_TRANSPORT_REST,
+ config,
+ load_dotenv_file,
+ )
from reachy_mini_conversation_app.tools.core_tools import ToolDependencies
- from reachy_mini_conversation_app.audio.head_wobbler import HeadWobbler
from reachy_mini_conversation_app.conversation_stream import ConversationStreamHandler
logger = setup_logger(args.debug)
logger.info("Starting Reachy Mini Conversation App")
- if args.no_camera and args.head_tracker is not None:
- logger.warning("Head tracking disabled: --no-camera flag is set. Remove --no-camera to enable head tracking.")
-
if instance_path:
try:
load_dotenv_file(Path(instance_path) / ".env")
except Exception as exc:
logger.debug("Instance .env loading skipped: %s", exc)
- if robot is None:
- try:
- robot_kwargs = {}
- if args.robot_name is not None:
- robot_kwargs["robot_name"] = args.robot_name
-
- logger.info("Initializing ReachyMini (SDK will auto-detect appropriate backend)")
- robot = ReachyMini(**robot_kwargs)
+ tool_transport_mode = getattr(args, "tool_transport", None) or config.REACHY_TOOL_TRANSPORT
+ runtime: Any | None = None
+
+ if tool_transport_mode == TOOL_TRANSPORT_REST:
+ logger.info("Using direct Reachy REST tool transport; local robot SDK workers are disabled")
+ if args.head_tracker is not None or args.local_vision:
+ logger.warning("--head-tracker and --local-vision are ignored in REST mode")
+ if not args.gradio:
+ logger.info("REST mode has no in-process robot audio device; automatically enabling Gradio")
+ args.gradio = True
+ dependencies = ToolDependencies(
+ capture_directory=Path(os.getenv("REACHY_CAPTURE_DIR", "captures")).expanduser(),
+ )
+ robot = None
+ else:
+ from reachy_mini_conversation_app.robot_runtime import ReachyRuntime
+ if args.no_camera and args.head_tracker is not None:
+ logger.warning(
+ "Head tracking disabled: --no-camera flag is set. Remove --no-camera to enable head tracking."
+ )
+ try:
+ runtime = ReachyRuntime.connect(
+ robot_name=args.robot_name,
+ robot=robot,
+ no_camera=args.no_camera,
+ head_tracker=args.head_tracker,
+ local_vision=args.local_vision,
+ capture_directory=Path(os.getenv("REACHY_CAPTURE_DIR", "captures")),
+ log=logger,
+ )
except TimeoutError as e:
logger.error(f"Connection timeout: Failed to connect to Reachy Mini daemon. Details: {e}")
log_connection_troubleshooting(logger, args.robot_name)
sys.exit(1)
-
except ConnectionError as e:
logger.error(f"Connection failed: Unable to establish connection to Reachy Mini. Details: {e}")
log_connection_troubleshooting(logger, args.robot_name)
sys.exit(1)
-
except Exception as e:
logger.error(f"Unexpected error during robot initialization: {type(e).__name__}: {e}")
logger.error("Please check your configuration and try again.")
sys.exit(1)
- # Auto-enable Gradio in simulation mode (both MuJoCo for daemon and mockup-sim for desktop app)
- status = robot.client.get_status()
- if isinstance(status, dict):
- simulation_enabled = status.get("simulation_enabled", False)
- mockup_sim_enabled = status.get("mockup_sim_enabled", False)
- else:
- simulation_enabled = getattr(status, "simulation_enabled", False)
- mockup_sim_enabled = getattr(status, "mockup_sim_enabled", False)
-
- is_simulation = simulation_enabled or mockup_sim_enabled
+ robot = runtime.robot
+ dependencies = runtime.dependencies
- if is_simulation and not args.gradio:
- logger.info("Simulation mode detected. Automatically enabling gradio flag.")
- args.gradio = True
+ # Auto-enable Gradio in simulation mode (both MuJoCo for daemon and mockup-sim for desktop app)
+ if runtime.is_simulation and not args.gradio:
+ logger.info("Simulation mode detected. Automatically enabling gradio flag.")
+ args.gradio = True
- camera_worker, _, vision_manager = handle_vision_stuff(args, robot)
+ try:
+ tool_transport_factory = _build_tool_transport_factory(
+ tool_transport_mode,
+ dependencies,
+ rest_base_url=config.REACHY_REST_BASE_URL,
+ camera_base_url=config.REACHY_CAMERA_BASE_URL,
+ rest_timeout_seconds=config.REACHY_REST_TIMEOUT_SECONDS,
+ motion_duration_seconds=config.REACHY_MOTION_DURATION_SECONDS,
+ motion_poll_interval_seconds=config.REACHY_MOTION_POLL_INTERVAL_SECONDS,
+ motion_completion_timeout_seconds=config.REACHY_MOTION_COMPLETION_TIMEOUT_SECONDS,
+ )
+ except ValueError as e:
+ logger.error("Invalid tool transport configuration: %s", e)
+ sys.exit(1)
- movement_manager = MovementManager(
- current_robot=robot,
- camera_worker=camera_worker,
- )
+ media_result_processor = None
+ if tool_transport_mode != TOOL_TRANSPORT_REST:
+ from reachy_mini_conversation_app.media_result_processor import MediaResultProcessor
- head_wobbler = HeadWobbler(set_speech_offsets=movement_manager.set_speech_offsets)
+ try:
+ media_result_processor = MediaResultProcessor(
+ vision_router=dependencies.vision_router,
+ capture_directory=dependencies.capture_directory or Path("captures"),
+ require_routed_vision=config.REQUIRE_ROUTED_VISION,
+ )
+ except ValueError as e:
+ logger.error("Invalid routed vision configuration: %s", e)
+ sys.exit(1)
- deps = ToolDependencies(
- reachy_mini=robot,
- movement_manager=movement_manager,
- camera_worker=camera_worker,
- vision_manager=vision_manager,
- head_wobbler=head_wobbler,
- )
current_file_path = os.path.dirname(os.path.abspath(__file__))
logger.debug(f"Current file absolute path: {current_file_path}")
chatbot = gr.Chatbot(
@@ -145,9 +210,16 @@ def run(
)
logger.debug(f"Chatbot avatar images: {chatbot.avatar_images}")
- handler = ConversationStreamHandler(deps, gradio_mode=args.gradio, instance_path=instance_path)
+ handler = ConversationStreamHandler(
+ dependencies,
+ gradio_mode=args.gradio,
+ instance_path=instance_path,
+ model_logs=args.model_logs,
+ tool_transport_factory=tool_transport_factory,
+ media_result_processor=media_result_processor,
+ )
- stream_manager: gr.Blocks | LocalStream | None = None
+ stream_manager: Any | None = None
if args.gradio:
stream = Stream(
@@ -217,7 +289,11 @@ async def send_text_message(
app = gr.mount_gradio_app(app, stream.ui, path="/")
else:
+ from reachy_mini_conversation_app.console import LocalStream
+
# In headless mode, wire settings_app + instance_path to console LocalStream
+ if robot is None:
+ raise RuntimeError("Headless mode requires the local Reachy tool transport")
stream_manager = LocalStream(
handler,
robot,
@@ -225,14 +301,6 @@ async def send_text_message(
instance_path=instance_path,
)
- # Each async service → its own thread/loop
- movement_manager.start()
- head_wobbler.start()
- if camera_worker:
- camera_worker.start()
- if vision_manager:
- vision_manager.start()
-
def poll_stop_event() -> None:
"""Poll the stop event to allow graceful shutdown."""
if app_stop_event is not None:
@@ -248,49 +316,16 @@ def poll_stop_event() -> None:
threading.Thread(target=poll_stop_event, daemon=True).start()
try:
+ # Each robot service owns its own thread/loop behind the shared runtime.
+ if runtime is not None:
+ runtime.start()
stream_manager.launch()
except KeyboardInterrupt:
logger.info("Keyboard interruption in main thread... closing server.")
finally:
- _shutdown_step(logger, "movement manager", movement_manager.stop)
- _shutdown_step(logger, "head wobbler", head_wobbler.stop)
- if camera_worker:
- _shutdown_step(logger, "camera worker", camera_worker.stop)
- if vision_manager:
- _shutdown_step(logger, "vision manager", vision_manager.stop)
-
- _shutdown_step(logger, "media", robot.media.close)
- _shutdown_step(logger, "robot client", robot.client.disconnect)
- time.sleep(1)
- logger.info("Shutdown complete.")
-
-
-class ReachyMiniConversationApp(ReachyMiniApp): # type: ignore[misc]
- """Reachy Mini Apps entry point for the conversation app."""
-
- custom_app_url = "http://0.0.0.0:7860/"
- dont_start_webserver = False
-
- def run(self, reachy_mini: ReachyMini, stop_event: threading.Event) -> None:
- """Run the Reachy Mini conversation app."""
- loop = asyncio.new_event_loop()
- asyncio.set_event_loop(loop)
-
- args, _ = parse_args()
-
- instance_path = self._get_instance_path().parent
- run(
- args,
- robot=reachy_mini,
- app_stop_event=stop_event,
- settings_app=self.settings_app,
- instance_path=str(instance_path),
- )
+ if runtime is not None:
+ runtime.stop()
if __name__ == "__main__":
- app = ReachyMiniConversationApp()
- try:
- app.wrapped_run()
- except KeyboardInterrupt:
- app.stop()
+ main()
diff --git a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/media_result_processor.py b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/media_result_processor.py
new file mode 100644
index 00000000..ae270689
--- /dev/null
+++ b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/media_result_processor.py
@@ -0,0 +1,278 @@
+"""Process raw local-tool media before any result reaches a conversation model."""
+
+from __future__ import annotations
+import base64
+import logging
+from typing import TYPE_CHECKING, Any
+from pathlib import Path
+from dataclasses import dataclass
+
+
+if TYPE_CHECKING:
+ from reachy_mini_conversation_app.vision_router import VisionRouter, VisionAnalysis
+
+
+logger = logging.getLogger(__name__)
+
+MAX_SCENE_IMAGES = 9
+_RAW_MEDIA_KEYS = frozenset({"b64_im", "b64_images"})
+
+
+class MediaSecurityError(RuntimeError):
+ """Raised when raw media would cross the model-visible boundary."""
+
+
+@dataclass
+class ProcessedToolResult:
+ """Separate model-visible text from UI-only media artifacts."""
+
+ model_payload: dict[str, Any]
+ preview_image: Any | None = None
+ video_path: Path | None = None
+
+
+class MediaResultProcessor:
+ """Route camera media to approved vision and sanitize the tool result."""
+
+ def __init__(
+ self,
+ *,
+ vision_router: VisionRouter | None,
+ capture_directory: Path,
+ require_routed_vision: bool,
+ ) -> None:
+ """Configure media routing and fail startup when strict routing is unavailable."""
+ if require_routed_vision and vision_router is None:
+ raise ValueError("REQUIRE_ROUTED_VISION=1 requires a configured VisionRouter")
+
+ self.vision_router = vision_router
+ self.capture_directory = capture_directory.expanduser().resolve()
+ self.require_routed_vision = require_routed_vision
+
+ async def process(self, tool_name: str, tool_result: dict[str, Any]) -> ProcessedToolResult:
+ """Process supported media results and verify that model output has no raw bytes."""
+ if tool_name == "camera" and "b64_im" in tool_result:
+ processed = await self._process_camera(tool_result)
+ elif tool_name == "scan_scene" and "b64_images" in tool_result:
+ processed = await self._process_scene_scan(tool_result)
+ elif contains_raw_media(tool_result):
+ processed = self._security_failure(tool_name, "Unexpected raw media field in tool result")
+ else:
+ processed = ProcessedToolResult(model_payload=dict(tool_result))
+
+ assert_no_raw_media(processed.model_payload)
+ return processed
+
+ async def _process_camera(self, tool_result: dict[str, Any]) -> ProcessedToolResult:
+ result = dict(tool_result)
+ raw_image = result.pop("b64_im", None)
+ question = result.get("question")
+ if not isinstance(raw_image, str) or not raw_image:
+ return self._security_failure("camera", "Camera returned an invalid image")
+ if not isinstance(question, str) or not question.strip():
+ return self._security_failure("camera", "Camera returned an invalid question")
+
+ try:
+ preview_image = _decode_preview(raw_image)
+ except ValueError:
+ return self._security_failure("camera", "Camera returned an invalid JPEG")
+
+ analysis = await self._analyze_images(
+ tool_name="camera",
+ images=[raw_image],
+ question=question.strip(),
+ timestamps=None,
+ )
+ if analysis is None:
+ return self._vision_failure("camera")
+
+ model_payload = _analysis_payload(analysis, question=question.strip(), status="image_analyzed")
+ return ProcessedToolResult(model_payload=model_payload, preview_image=preview_image)
+
+ async def _process_scene_scan(self, tool_result: dict[str, Any]) -> ProcessedToolResult:
+ result = dict(tool_result)
+ raw_images = result.pop("b64_images", None)
+ question = result.get("question")
+ timestamps = result.get("frame_timestamps_seconds")
+
+ if (
+ not isinstance(raw_images, list)
+ or not 1 <= len(raw_images) <= MAX_SCENE_IMAGES
+ or not all(isinstance(image, str) and image for image in raw_images)
+ ):
+ return self._security_failure(
+ "scan_scene",
+ f"Scene scan must contain between 1 and {MAX_SCENE_IMAGES} images",
+ )
+ if not isinstance(question, str) or not question.strip():
+ return self._security_failure("scan_scene", "Scene scan returned an invalid question")
+ if not _valid_timestamps(timestamps, len(raw_images)):
+ return self._security_failure("scan_scene", "Scene scan returned invalid frame timestamps")
+ try:
+ for image in raw_images:
+ base64.b64decode(image, validate=True)
+ except (ValueError, TypeError):
+ return self._security_failure("scan_scene", "Scene scan returned invalid image data")
+
+ analysis = await self._analyze_images(
+ tool_name="scan_scene",
+ images=raw_images,
+ question=question.strip(),
+ timestamps=timestamps,
+ )
+ if analysis is None:
+ return self._vision_failure("scan_scene")
+
+ excluded = {
+ "b64_images",
+ "video_url",
+ "video_path",
+ "image_description",
+ "selected_model",
+ "response_id",
+ "usage",
+ }
+ recording_metadata = {key: value for key, value in result.items() if key not in excluded}
+ model_payload = {
+ **recording_metadata,
+ **_analysis_payload(analysis, question=question.strip(), status="scene_analyzed"),
+ }
+
+ try:
+ video_path = await self._scene_video_path(result)
+ except Exception as exc:
+ logger.error("Scene video retrieval failed: %s", type(exc).__name__)
+ model_payload["recording_status"] = "preview_unavailable"
+ model_payload["recording_error"] = (
+ "The scene was analyzed successfully, but the recording preview could not be retrieved"
+ )
+ return ProcessedToolResult(model_payload=model_payload)
+
+ model_payload["recording_status"] = "available"
+ return ProcessedToolResult(model_payload=model_payload, video_path=video_path)
+
+ async def _analyze_images(
+ self,
+ *,
+ tool_name: str,
+ images: list[str],
+ question: str,
+ timestamps: list[float] | None,
+ ) -> VisionAnalysis | None:
+ if self.vision_router is None:
+ logger.error("Routed vision unavailable for tool=%s", tool_name)
+ return None
+ try:
+ return await self.vision_router.analyze_images(
+ images_base64=images,
+ question=question,
+ frame_timestamps=timestamps,
+ )
+ except Exception as exc:
+ logger.error("Approved vision request failed for tool=%s error=%s", tool_name, type(exc).__name__)
+ return None
+
+ async def _scene_video_path(self, result: dict[str, Any]) -> Path:
+ local_path = result.get("video_path")
+ if isinstance(local_path, str):
+ return self._validated_local_video(local_path)
+ raise MediaSecurityError("Missing local scene recording path")
+
+ def _validated_local_video(self, raw_path: str) -> Path:
+ path = Path(raw_path).expanduser().resolve(strict=True)
+ if path.suffix.lower() != ".mp4" or path.is_symlink():
+ raise MediaSecurityError("Invalid local scene recording")
+ if path.parent != self.capture_directory:
+ raise MediaSecurityError("Scene recording is outside the configured capture directory")
+ return path
+
+ @staticmethod
+ def _security_failure(tool_name: str, message: str) -> ProcessedToolResult:
+ logger.error("Media security check failed for tool=%s: %s", tool_name, message)
+ return ProcessedToolResult(
+ model_payload={
+ "status": "media_security_error",
+ "tool": tool_name,
+ "error": message,
+ }
+ )
+
+ @staticmethod
+ def _vision_failure(tool_name: str) -> ProcessedToolResult:
+ return ProcessedToolResult(
+ model_payload={
+ "status": "vision_error",
+ "tool": tool_name,
+ "error": "Approved vision analysis failed; raw media was discarded",
+ }
+ )
+
+
+def contains_raw_media(value: Any) -> bool:
+ """Return whether a value contains raw image fields or data URLs."""
+ if isinstance(value, dict):
+ for key, item in value.items():
+ if str(key).lower() in _RAW_MEDIA_KEYS:
+ return True
+ if str(key).lower() == "image_url" and isinstance(item, str) and item.startswith("data:image/"):
+ return True
+ if contains_raw_media(item):
+ return True
+ elif isinstance(value, (list, tuple)):
+ return any(contains_raw_media(item) for item in value)
+ return False
+
+
+def assert_no_raw_media(value: Any) -> None:
+ """Reject any model payload that still contains raw images."""
+ if contains_raw_media(value):
+ raise MediaSecurityError("Raw media reached the model-visible tool result")
+
+
+def _analysis_payload(analysis: VisionAnalysis, *, question: str, status: str) -> dict[str, Any]:
+ payload: dict[str, Any] = {
+ "status": status,
+ "question": question,
+ "image_description": analysis.description,
+ "selected_model": analysis.selected_model,
+ }
+ if analysis.response_id:
+ payload["response_id"] = analysis.response_id
+ if analysis.usage is not None:
+ payload["usage"] = _jsonable(analysis.usage)
+ return payload
+
+
+def _jsonable(value: Any) -> Any:
+ model_dump = getattr(value, "model_dump", None)
+ if callable(model_dump):
+ try:
+ return model_dump(mode="json")
+ except TypeError:
+ return model_dump()
+ return value
+
+
+def _decode_preview(image_base64: str) -> Any:
+ import cv2
+ import numpy as np
+
+ try:
+ encoded = base64.b64decode(image_base64, validate=True)
+ except (ValueError, TypeError) as exc:
+ raise ValueError("Invalid Base64 image") from exc
+ try:
+ frame = cv2.imdecode(np.frombuffer(encoded, dtype=np.uint8), cv2.IMREAD_COLOR)
+ except cv2.error as exc:
+ raise ValueError("Invalid JPEG image") from exc
+ if frame is None:
+ raise ValueError("Invalid JPEG image")
+ return cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
+
+
+def _valid_timestamps(value: Any, image_count: int) -> bool:
+ return (
+ isinstance(value, list)
+ and len(value) == image_count
+ and all(isinstance(timestamp, (int, float)) and not isinstance(timestamp, bool) for timestamp in value)
+ )
diff --git a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/moves.py b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/moves.py
index 496305f6..cbb8a3bd 100644
--- a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/moves.py
+++ b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/moves.py
@@ -247,10 +247,16 @@ def __init__(
self,
current_robot: ReachyMini,
camera_worker: "Any" = None,
+ target_frequency_hz: float = CONTROL_LOOP_FREQUENCY_HZ,
+ enable_idle_breathing: bool = True,
):
"""Initialize movement manager."""
+ if target_frequency_hz <= 0:
+ raise ValueError("target_frequency_hz must be positive")
+
self.current_robot = current_robot
self.camera_worker = camera_worker
+ self.enable_idle_breathing = enable_idle_breathing
# Single timing source for durations
self._now = time.monotonic
@@ -260,6 +266,18 @@ def __init__(
self.state.last_activity_time = self._now()
neutral_pose = create_head_pose(0, 0, 0, 0, 0, 0, degrees=True)
initial_pose: FullBodyPose = (neutral_pose, (0.0, 0.0), 0.0)
+ initial_pose_is_observed = False
+ try:
+ current_head_pose = current_robot.get_current_head_pose()
+ current_head_joints, current_antennas = current_robot.get_current_joint_positions()
+ initial_pose = (
+ current_head_pose.copy(),
+ (float(current_antennas[0]), float(current_antennas[1])),
+ float(current_head_joints[0]),
+ )
+ initial_pose_is_observed = True
+ except (AttributeError, AssertionError, IndexError, TypeError, ValueError):
+ logger.debug("Could not seed movement output from current robot state; using neutral state")
self.state.last_primary_pose = initial_pose
# Move queue (primary moves)
@@ -267,13 +285,14 @@ def __init__(
# Configuration
self.idle_inactivity_delay = 0.3 # seconds
- self.target_frequency = CONTROL_LOOP_FREQUENCY_HZ
+ self.target_frequency = target_frequency_hz
self.target_period = 1.0 / self.target_frequency
self._stop_event = threading.Event()
self._thread: threading.Thread | None = None
self._is_listening = False
self._last_commanded_pose: FullBodyPose = clone_full_body_pose(initial_pose)
+ self._has_sent_target = initial_pose_is_observed
self._listening_antennas: Tuple[float, float] = self._last_commanded_pose[1]
self._antenna_unfreeze_blend = 1.0
self._antenna_blend_duration = 0.4 # seconds to blend back after listening
@@ -281,9 +300,10 @@ def __init__(
self._breathing_active = False # true when breathing move is running or queued
self._listening_debounce_s = 0.15
self._last_listening_toggle_time = self._now()
- self._last_set_target_err = 0.0
- self._set_target_err_interval = 1.0 # seconds between error logs
- self._set_target_err_suppressed = 0
+ self._delivery_condition = threading.Condition()
+ self._delivery_sequence = 0
+ self._delivery_error: str | None = None
+ self._delivery_failed = False
# Cross-thread signalling
self._command_queue: "Queue[Tuple[str, Any]]" = Queue()
@@ -331,6 +351,38 @@ def clear_move_queue(self) -> None:
"""
self._command_queue.put(("clear_queue", None))
+ def delivery_checkpoint(self) -> int:
+ """Return a sequence number callers can use to await a later send."""
+ with self._delivery_condition:
+ return self._delivery_sequence
+
+ def wait_for_delivery(self, checkpoint: int, timeout: float = 1.0) -> tuple[bool, str | None]:
+ """Wait for a target send after ``checkpoint`` or a terminal send error."""
+ with self._delivery_condition:
+ self._delivery_condition.wait_for(
+ lambda: self._delivery_sequence > checkpoint or self._delivery_failed,
+ timeout=max(0.0, timeout),
+ )
+ if self._delivery_sequence > checkpoint:
+ return True, None
+ return False, self._delivery_error or "Timed out waiting to send a target to Reachy"
+
+ def connection_healthy(self) -> bool:
+ """Return whether the SDK and this movement loop can still send commands."""
+ with self._delivery_condition:
+ if self._delivery_failed:
+ return False
+
+ client = getattr(self.current_robot, "client", None)
+ sdk_alive = getattr(client, "_is_alive", None)
+ return True if sdk_alive is None else bool(sdk_alive)
+
+ @property
+ def delivery_error(self) -> str | None:
+ """Return the terminal target-delivery error, if one occurred."""
+ with self._delivery_condition:
+ return self._delivery_error
+
def set_speech_offsets(self, offsets: Tuple[float, float, float, float, float, float]) -> None:
"""Update speech-induced secondary offsets (x, y, z, roll, pitch, yaw).
@@ -411,6 +463,9 @@ def _apply_pending_offsets(self) -> None:
def _handle_command(self, command: str, payload: Any, current_time: float) -> None:
"""Handle a single cross-thread command."""
if command == "queue_move":
+ if not self.connection_healthy():
+ logger.warning("Ignored queued move because the Reachy control connection is unavailable")
+ return
if isinstance(payload, Move):
self.move_queue.append(payload)
self.state.update_activity()
@@ -494,6 +549,10 @@ def _manage_move_queue(self, current_time: float) -> None:
def _manage_breathing(self, current_time: float) -> None:
"""Manage automatic breathing when idle."""
+ if not self.enable_idle_breathing:
+ self._breathing_active = False
+ return
+
if (
self.state.current_move is None
and not self.move_queue
@@ -638,7 +697,11 @@ def _calculate_blended_antennas(self, target_antennas: Tuple[float, float]) -> T
def _issue_control_command(
self, head: NDArray[np.float64], antennas: Tuple[float, float], body_yaw: float
) -> None:
- """Send the fused pose to the robot with throttled error logging."""
+ """Send one fused pose, stopping permanently after uncertain delivery."""
+ with self._delivery_condition:
+ if self._delivery_failed:
+ return
+
try:
self.current_robot.set_target(
head=head,
@@ -646,19 +709,41 @@ def _issue_control_command(
body_yaw=body_yaw,
)
except Exception as e:
- now = self._now()
- if now - self._last_set_target_err >= self._set_target_err_interval:
- msg = f"Failed to set robot target: {e}"
- if self._set_target_err_suppressed:
- msg += f" (suppressed {self._set_target_err_suppressed} repeats)"
- self._set_target_err_suppressed = 0
- logger.error(msg)
- self._last_set_target_err = now
- else:
- self._set_target_err_suppressed += 1
+ error = f"Failed to set robot target: {type(e).__name__}: {e}"
+ with self._delivery_condition:
+ self._delivery_failed = True
+ self._delivery_error = error
+ self._delivery_condition.notify_all()
+
+ self.move_queue.clear()
+ self.state.current_move = None
+ self.state.move_start_time = None
+ self._breathing_active = False
+ logger.error("%s; motion output paused until the runtime reconnects", error)
else:
with self._status_lock:
self._last_commanded_pose = clone_full_body_pose((head, antennas, body_yaw))
+ self._has_sent_target = True
+ with self._delivery_condition:
+ self._delivery_sequence += 1
+ self._delivery_condition.notify_all()
+
+ def _target_changed(
+ self,
+ head: NDArray[np.float64],
+ antennas: Tuple[float, float],
+ body_yaw: float,
+ ) -> bool:
+ """Return whether a target differs enough to justify a network send."""
+ with self._status_lock:
+ if not self._has_sent_target:
+ return True
+ previous = clone_full_body_pose(self._last_commanded_pose)
+ return not (
+ np.allclose(head, previous[0], rtol=0.0, atol=1e-7)
+ and np.allclose(antennas, previous[1], rtol=0.0, atol=1e-7)
+ and abs(body_yaw - previous[2]) <= 1e-7
+ )
def _update_frequency_stats(
self,
@@ -725,7 +810,7 @@ def _update_face_tracking(self, current_time: float) -> None:
self.state.face_tracking_offsets = (0.0, 0.0, 0.0, 0.0, 0.0, 0.0)
def start(self) -> None:
- """Start the worker thread that drives the 100 Hz control loop."""
+ """Start the worker thread that drives the configured control loop."""
if self._thread is not None and self._thread.is_alive():
logger.warning("Move worker already running; start() ignored")
return
@@ -755,6 +840,10 @@ def stop(self) -> None:
self._thread = None
logger.debug("Move worker stopped")
+ if not self.connection_healthy():
+ logger.info("Skipping neutral reset because the Reachy control connection is unavailable")
+ return
+
# Reset to neutral position using goto_target (same approach as wake_up)
try:
neutral_head_pose = create_head_pose(0, 0, 0, 0, 0, 0, degrees=True)
@@ -795,6 +884,8 @@ def get_status(self) -> Dict[str, Any]:
"queue_size": len(self.move_queue),
"is_listening": self._is_listening,
"breathing_active": self._breathing_active,
+ "connection_healthy": self.connection_healthy(),
+ "delivery_error": self.delivery_error,
"last_commanded_pose": {
"head": head_matrix,
"antennas": antennas,
@@ -814,7 +905,7 @@ def working_loop(self) -> None:
Single set_target() call with pose fusion.
"""
- logger.debug("Starting enhanced movement control loop (100Hz)")
+ logger.debug("Starting enhanced movement control loop (%.1fHz)", self.target_frequency)
loop_count = 0
prev_loop_start = self._now()
@@ -844,8 +935,11 @@ def working_loop(self) -> None:
# 5) Apply listening antenna freeze or blend-back
antennas_cmd = self._calculate_blended_antennas(antennas)
- # 6) Single set_target call - the only control point
- self._issue_control_command(head, antennas_cmd, body_yaw)
+ # 6) Send only changed targets. Avoiding idle writes is especially
+ # important over Wi-Fi, where a 100 Hz no-op stream can starve the
+ # daemon's WebSocket and eventually trigger a keepalive timeout.
+ if self._target_changed(head, antennas_cmd, body_yaw):
+ self._issue_control_command(head, antennas_cmd, body_yaw)
# 7) Adaptive sleep to align to next tick, then publish shared state
sleep_time, freq_stats = self._schedule_next_tick(loop_start, freq_stats)
diff --git a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/native_app.py b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/native_app.py
new file mode 100644
index 00000000..ae2ac48d
--- /dev/null
+++ b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/native_app.py
@@ -0,0 +1,38 @@
+"""Native Reachy Mini Apps integration for local robot installations."""
+
+import asyncio
+import threading
+
+from reachy_mini import ReachyMini, ReachyMiniApp
+from reachy_mini_conversation_app.main import run
+from reachy_mini_conversation_app.utils import parse_args
+
+
+class ReachyMiniConversationApp(ReachyMiniApp): # type: ignore[misc]
+ """Reachy Mini Apps entry point for the conversation app."""
+
+ custom_app_url = "http://0.0.0.0:7860/"
+ dont_start_webserver = False
+
+ def run(self, reachy_mini: ReachyMini, stop_event: threading.Event) -> None:
+ """Run the Reachy Mini conversation app."""
+ loop = asyncio.new_event_loop()
+ asyncio.set_event_loop(loop)
+
+ args, _ = parse_args()
+ instance_path = self._get_instance_path().parent
+ run(
+ args,
+ robot=reachy_mini,
+ app_stop_event=stop_event,
+ settings_app=self.settings_app,
+ instance_path=str(instance_path),
+ )
+
+
+if __name__ == "__main__":
+ app = ReachyMiniConversationApp()
+ try:
+ app.wrapped_run()
+ except KeyboardInterrupt:
+ app.stop()
diff --git a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/profiles/_reachy_mini_conversation_app_locked_profile/instructions.txt b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/profiles/_reachy_mini_conversation_app_locked_profile/instructions.txt
index 358102c5..5c50c9a8 100644
--- a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/profiles/_reachy_mini_conversation_app_locked_profile/instructions.txt
+++ b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/profiles/_reachy_mini_conversation_app_locked_profile/instructions.txt
@@ -1,3 +1,13 @@
You are a helpful assistant controlling a Reachy Mini robot for OpenShell research.
Keep responses concise, friendly, and concrete.
-Use the sweep_look tool when looking around would help.
+Only use move_head when the human user's latest message explicitly asks Reachy to look left, right, up, down, or front.
+When the user requests multiple head directions, call move_head once with every direction in the directions array and preserve their order.
+Use stop_motion immediately when the user asks Reachy to stop moving.
+When the human user explicitly asks Reachy to take a picture or photo, see them, or describe what Reachy sees, call camera with a concise question that preserves their request.
+Do not claim that taking a picture is unavailable when the camera tool is present. Call camera and use its result.
+Before a physical tool returns its result, say only that you will try the action; do not claim that motion has started or succeeded.
+Claim motion succeeded only when the tool result has status completed.
+If a tool result has status policy_denied, clearly say that OpenShell policy blocked the requested action, do not claim that the robot lacks the capability, and do not retry it.
+If a tool result has status unknown_delivery, explain that the result is uncertain and do not retry it.
+Do not offer scene scanning, dance, emotion, or head-tracking capabilities. Do not take pictures unless the human explicitly requests one.
+For automatic idle updates, use do_nothing instead of moving or taking a picture.
diff --git a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/profiles/_reachy_mini_conversation_app_locked_profile/scan_scene.py b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/profiles/_reachy_mini_conversation_app_locked_profile/scan_scene.py
new file mode 100644
index 00000000..48d81a87
--- /dev/null
+++ b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/profiles/_reachy_mini_conversation_app_locked_profile/scan_scene.py
@@ -0,0 +1,218 @@
+"""Record a synchronized Reachy sweep and return representative vision frames."""
+
+from __future__ import annotations
+import time
+import base64
+import asyncio
+import logging
+from typing import Any, Dict
+from pathlib import Path
+from datetime import datetime
+from dataclasses import dataclass
+
+import cv2
+import numpy as np
+from numpy.typing import NDArray
+
+from reachy_mini_conversation_app.tools.core_tools import Tool, ToolDependencies
+from reachy_mini_conversation_app.profiles._reachy_mini_conversation_app_locked_profile.sweep_look import (
+ SWEEP_TOTAL_DURATION_SECONDS,
+ SweepLook,
+)
+
+
+logger = logging.getLogger(__name__)
+
+CAPTURE_FPS = 15.0
+MAX_ANALYSIS_FRAMES = 9
+FRAME_WAIT_TIMEOUT_SECONDS = 3.0
+SWEEP_RECORDING_SETTLE_SECONDS = 0.25
+JPEG_QUALITY = 85
+
+
+@dataclass
+class _FrameCandidate:
+ """Sharpest frame observed in one chronological section of the sweep."""
+
+ sharpness: float
+ elapsed_seconds: float
+ frame: NDArray[np.uint8]
+
+
+def _frame_sharpness(frame: NDArray[np.uint8]) -> float:
+ """Return a simple focus score used to avoid motion-blurred samples."""
+ gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
+ return float(cv2.Laplacian(gray, cv2.CV_64F).var())
+
+
+def _open_video_writer(path: Path, frame: NDArray[np.uint8]) -> Any:
+ """Create an MP4 writer matching the camera frame size."""
+ height, width = frame.shape[:2]
+ fourcc = cv2.VideoWriter.fourcc(*"mp4v")
+ return cv2.VideoWriter(str(path), fourcc, CAPTURE_FPS, (width, height))
+
+
+def _encode_analysis_frames(
+ candidates: list[_FrameCandidate | None],
+) -> tuple[list[str], list[float]]:
+ """JPEG/base64 encode selected frames in chronological order."""
+ images: list[str] = []
+ timestamps: list[float] = []
+ for candidate in candidates:
+ if candidate is None:
+ continue
+ success, buffer = cv2.imencode(
+ ".jpg",
+ candidate.frame,
+ [int(cv2.IMWRITE_JPEG_QUALITY), JPEG_QUALITY],
+ )
+ if not success:
+ logger.warning("Skipping a scene-scan frame that failed JPEG encoding")
+ continue
+ images.append(base64.b64encode(buffer.tobytes()).decode("utf-8"))
+ timestamps.append(round(candidate.elapsed_seconds, 2))
+ return images, timestamps
+
+
+class ScanScene(Tool):
+ """Sweep, record a video, and provide chronological frames for visual analysis."""
+
+ name = "scan_scene"
+ description = (
+ "Sweep Reachy from left to right while recording a video, then analyze representative "
+ "frames to answer a question about everything visible during the sweep. Use this instead "
+ "of separate sweep_look and camera calls when the user asks to scan, record, or survey a scene."
+ )
+ parameters_schema: Dict[str, Any] = {
+ "type": "object",
+ "properties": {
+ "question": {
+ "type": "string",
+ "description": (
+ "What to determine from the complete scene scan, for example: "
+ "'List the people, objects, text, and notable surroundings you saw.'"
+ ),
+ },
+ },
+ "required": ["question"],
+ }
+
+ async def _wait_for_frame(self, camera_worker: Any) -> NDArray[np.uint8] | None:
+ """Wait briefly for the camera worker to publish its first frame."""
+ deadline = time.monotonic() + FRAME_WAIT_TIMEOUT_SECONDS
+ while time.monotonic() < deadline:
+ frame = camera_worker.get_latest_frame()
+ if frame is not None:
+ return frame
+ await asyncio.sleep(0.05)
+ return None
+
+ async def __call__(self, deps: ToolDependencies, **kwargs: Any) -> Dict[str, Any]:
+ """Record the complete sweep and return sampled frames to the conversation model."""
+ question = (kwargs.get("question") or "").strip()
+ if not question:
+ return {"error": "question must be a non-empty string"}
+ if deps.camera_worker is None:
+ return {"error": "Camera worker not available"}
+
+ first_frame = await self._wait_for_frame(deps.camera_worker)
+ if first_frame is None:
+ return {"error": "No frame available from camera worker"}
+
+ capture_directory = (deps.capture_directory or Path("captures")).expanduser().resolve()
+ capture_directory.mkdir(parents=True, exist_ok=True)
+ timestamp = datetime.now().strftime("%Y%m%d-%H%M%S-%f")
+ video_path = capture_directory / f"reachy-scene-scan-{timestamp}.mp4"
+ writer = _open_video_writer(video_path, first_frame)
+ if not writer.isOpened():
+ writer.release()
+ return {"error": f"Could not open video writer for {video_path}"}
+
+ was_tracking_enabled = bool(getattr(deps.camera_worker, "is_head_tracking_enabled", False))
+ scan_completed = False
+ frames_recorded = 0
+ candidates: list[_FrameCandidate | None] = [None] * MAX_ANALYSIS_FRAMES
+ recording_duration = SWEEP_TOTAL_DURATION_SECONDS + SWEEP_RECORDING_SETTLE_SECONDS
+ started_at = time.monotonic()
+
+ logger.info(
+ "Tool call: scan_scene question=%s video=%s duration=%.2fs",
+ question[:120],
+ video_path,
+ recording_duration,
+ )
+
+ try:
+ deps.camera_worker.set_head_tracking_enabled(False)
+ clear_offsets = getattr(deps.camera_worker, "clear_face_tracking_offsets", None)
+ if callable(clear_offsets):
+ clear_offsets()
+
+ await SweepLook()(deps)
+
+ frame_period = 1.0 / CAPTURE_FPS
+ next_frame_at = started_at
+ while True:
+ now = time.monotonic()
+ elapsed = now - started_at
+ if elapsed > recording_duration:
+ break
+
+ frame = deps.camera_worker.get_latest_frame()
+ if frame is not None:
+ writer.write(frame)
+ frames_recorded += 1
+
+ bin_index = min(
+ MAX_ANALYSIS_FRAMES - 1,
+ int((elapsed / recording_duration) * MAX_ANALYSIS_FRAMES),
+ )
+ sharpness = _frame_sharpness(frame)
+ current = candidates[bin_index]
+ if current is None or sharpness > current.sharpness:
+ candidates[bin_index] = _FrameCandidate(sharpness, elapsed, frame.copy())
+
+ next_frame_at += frame_period
+ await asyncio.sleep(max(0.0, next_frame_at - time.monotonic()))
+
+ scan_completed = True
+ finally:
+ writer.release()
+ if was_tracking_enabled:
+ deps.camera_worker.set_head_tracking_enabled(True)
+ if not scan_completed:
+ deps.require_movement_manager().clear_move_queue()
+ video_path.unlink(missing_ok=True)
+
+ b64_images, frame_timestamps = await asyncio.to_thread(_encode_analysis_frames, candidates)
+ if not b64_images:
+ video_path.unlink(missing_ok=True)
+ return {"error": "The sweep recorded no usable analysis frames"}
+
+ elapsed_total = round(time.monotonic() - started_at, 2)
+ movement_manager = deps.require_movement_manager()
+ connection_checker = getattr(movement_manager, "connection_healthy", None)
+ connection_healthy = True if not callable(connection_checker) else bool(connection_checker())
+ scan_status = "scene_scan_complete" if connection_healthy else "scene_scan_incomplete"
+ logger.info(
+ "Scene scan captured video=%s frames_recorded=%d analysis_frames=%d elapsed=%.2fs status=%s",
+ video_path,
+ frames_recorded,
+ len(b64_images),
+ elapsed_total,
+ scan_status,
+ )
+ result = {
+ "status": scan_status,
+ "scan_status": scan_status,
+ "question": question,
+ "video_path": str(video_path),
+ "duration_seconds": elapsed_total,
+ "frames_recorded": frames_recorded,
+ "frames_selected": len(b64_images),
+ "frame_timestamps_seconds": frame_timestamps,
+ "b64_images": b64_images,
+ }
+ if not connection_healthy:
+ result["scan_warning"] = "Reachy lost its control connection before the sweep returned to front"
+ return result
diff --git a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/profiles/_reachy_mini_conversation_app_locked_profile/sweep_look.py b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/profiles/_reachy_mini_conversation_app_locked_profile/sweep_look.py
index ec7bfb56..1e4e882d 100644
--- a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/profiles/_reachy_mini_conversation_app_locked_profile/sweep_look.py
+++ b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/profiles/_reachy_mini_conversation_app_locked_profile/sweep_look.py
@@ -10,6 +10,11 @@
logger = logging.getLogger(__name__)
+SWEEP_MAX_ANGLE_RADIANS = 0.9 * np.pi
+SWEEP_TRANSITION_DURATION_SECONDS = 3.0
+SWEEP_HOLD_DURATION_SECONDS = 1.0
+SWEEP_TOTAL_DURATION_SECONDS = SWEEP_TRANSITION_DURATION_SECONDS * 4 + SWEEP_HOLD_DURATION_SECONDS * 2
+
class SweepLook(Tool):
"""Sweep head from left to right and back to center, pausing at each position."""
@@ -28,18 +33,20 @@ async def __call__(self, deps: ToolDependencies, **kwargs: Any) -> Dict[str, Any
"""Execute sweep look: left -> hold -> right -> hold -> center."""
logger.info("Tool call: sweep_look")
- deps.movement_manager.clear_move_queue()
+ movement_manager = deps.require_movement_manager()
+ reachy_mini = deps.require_reachy_mini()
+ movement_manager.clear_move_queue()
- current_head_pose = deps.reachy_mini.get_current_head_pose()
- head_joints, antenna_joints = deps.reachy_mini.get_current_joint_positions()
+ current_head_pose = reachy_mini.get_current_head_pose()
+ head_joints, antenna_joints = reachy_mini.get_current_joint_positions()
current_body_yaw = head_joints[0]
current_antenna1 = antenna_joints[0]
current_antenna2 = antenna_joints[1]
- max_angle = 0.9 * np.pi
- transition_duration = 3.0
- hold_duration = 1.0
+ max_angle = SWEEP_MAX_ANGLE_RADIANS
+ transition_duration = SWEEP_TRANSITION_DURATION_SECONDS
+ hold_duration = SWEEP_HOLD_DURATION_SECONDS
left_head_pose = create_head_pose(0, 0, 0, 0, 0, max_angle, degrees=False)
move_to_left = GotoQueueMove(
@@ -47,7 +54,7 @@ async def __call__(self, deps: ToolDependencies, **kwargs: Any) -> Dict[str, Any
start_head_pose=current_head_pose,
target_antennas=(current_antenna1, current_antenna2),
start_antennas=(current_antenna1, current_antenna2),
- target_body_yaw=current_body_yaw + max_angle,
+ target_body_yaw=max_angle,
start_body_yaw=current_body_yaw,
duration=transition_duration,
)
@@ -57,8 +64,8 @@ async def __call__(self, deps: ToolDependencies, **kwargs: Any) -> Dict[str, Any
start_head_pose=left_head_pose,
target_antennas=(current_antenna1, current_antenna2),
start_antennas=(current_antenna1, current_antenna2),
- target_body_yaw=current_body_yaw + max_angle,
- start_body_yaw=current_body_yaw + max_angle,
+ target_body_yaw=max_angle,
+ start_body_yaw=max_angle,
duration=hold_duration,
)
@@ -68,8 +75,8 @@ async def __call__(self, deps: ToolDependencies, **kwargs: Any) -> Dict[str, Any
start_head_pose=left_head_pose,
target_antennas=(current_antenna1, current_antenna2),
start_antennas=(current_antenna1, current_antenna2),
- target_body_yaw=current_body_yaw,
- start_body_yaw=current_body_yaw + max_angle,
+ target_body_yaw=0,
+ start_body_yaw=max_angle,
duration=transition_duration,
)
@@ -79,8 +86,8 @@ async def __call__(self, deps: ToolDependencies, **kwargs: Any) -> Dict[str, Any
start_head_pose=center_head_pose,
target_antennas=(current_antenna1, current_antenna2),
start_antennas=(current_antenna1, current_antenna2),
- target_body_yaw=current_body_yaw - max_angle,
- start_body_yaw=current_body_yaw,
+ target_body_yaw=-max_angle,
+ start_body_yaw=0,
duration=transition_duration,
)
@@ -89,8 +96,8 @@ async def __call__(self, deps: ToolDependencies, **kwargs: Any) -> Dict[str, Any
start_head_pose=right_head_pose,
target_antennas=(current_antenna1, current_antenna2),
start_antennas=(current_antenna1, current_antenna2),
- target_body_yaw=current_body_yaw - max_angle,
- start_body_yaw=current_body_yaw - max_angle,
+ target_body_yaw=-max_angle,
+ start_body_yaw=-max_angle,
duration=hold_duration,
)
@@ -99,19 +106,19 @@ async def __call__(self, deps: ToolDependencies, **kwargs: Any) -> Dict[str, Any
start_head_pose=right_head_pose,
target_antennas=(current_antenna1, current_antenna2),
start_antennas=(current_antenna1, current_antenna2),
- target_body_yaw=current_body_yaw,
- start_body_yaw=current_body_yaw - max_angle,
+ target_body_yaw=0,
+ start_body_yaw=-max_angle,
duration=transition_duration,
)
- deps.movement_manager.queue_move(move_to_left)
- deps.movement_manager.queue_move(hold_left)
- deps.movement_manager.queue_move(return_to_center_from_left)
- deps.movement_manager.queue_move(move_to_right)
- deps.movement_manager.queue_move(hold_right)
- deps.movement_manager.queue_move(return_to_center_final)
+ movement_manager.queue_move(move_to_left)
+ movement_manager.queue_move(hold_left)
+ movement_manager.queue_move(return_to_center_from_left)
+ movement_manager.queue_move(move_to_right)
+ movement_manager.queue_move(hold_right)
+ movement_manager.queue_move(return_to_center_final)
- total_duration = transition_duration * 4 + hold_duration * 2
- deps.movement_manager.set_moving_state(total_duration)
+ total_duration = SWEEP_TOTAL_DURATION_SECONDS
+ movement_manager.set_moving_state(total_duration)
return {"status": f"sweeping look left-right-center, total {total_duration:.1f}s"}
diff --git a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/profiles/_reachy_mini_conversation_app_locked_profile/tools.txt b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/profiles/_reachy_mini_conversation_app_locked_profile/tools.txt
index d5f05cb2..bfb3b4a4 100644
--- a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/profiles/_reachy_mini_conversation_app_locked_profile/tools.txt
+++ b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/profiles/_reachy_mini_conversation_app_locked_profile/tools.txt
@@ -3,3 +3,7 @@ stop_dance
play_emotion
stop_emotion
sweep_look
+camera
+scan_scene
+move_head
+do_nothing
diff --git a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/rest_tool_transport.py b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/rest_tool_transport.py
new file mode 100644
index 00000000..d412c8cd
--- /dev/null
+++ b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/rest_tool_transport.py
@@ -0,0 +1,465 @@
+"""Direct, policy-aware Reachy REST tool transport."""
+
+from __future__ import annotations
+import math
+import base64
+import asyncio
+from copy import deepcopy
+from uuid import UUID
+from typing import Any, Final, cast
+from contextlib import suppress
+from dataclasses import dataclass
+from urllib.parse import urlsplit
+
+import httpx
+
+
+POLICY_DENIED_ERROR: Final = "Blocked by OpenShell policy"
+MAX_CAMERA_QUESTION_CHARACTERS: Final = 500
+MAX_CAMERA_JPEG_BYTES: Final = 2 * 1024 * 1024
+
+_MOVE_HEAD_SPEC: Final[dict[str, Any]] = {
+ "type": "function",
+ "name": "move_head",
+ "description": (
+ "Move Reachy's head through one or more fixed directions in order. "
+ "Valid directions are left, right, up, down, and front."
+ ),
+ "parameters": {
+ "type": "object",
+ "properties": {
+ "directions": {
+ "type": "array",
+ "description": "Ordered directions to perform.",
+ "items": {
+ "type": "string",
+ "enum": ["left", "right", "up", "down", "front"],
+ },
+ "minItems": 1,
+ "maxItems": 8,
+ }
+ },
+ "required": ["directions"],
+ "additionalProperties": False,
+ },
+}
+
+_STOP_MOTION_SPEC: Final[dict[str, Any]] = {
+ "type": "function",
+ "name": "stop_motion",
+ "description": "Stop every currently running Reachy movement.",
+ "parameters": {
+ "type": "object",
+ "properties": {},
+ "additionalProperties": False,
+ },
+}
+
+_CAMERA_SPEC: Final[dict[str, Any]] = {
+ "type": "function",
+ "name": "camera",
+ "description": (
+ "Capture one still image from Reachy's camera and answer a question about what is visibly present. "
+ "The capture is a separate OpenShell policy-controlled action."
+ ),
+ "parameters": {
+ "type": "object",
+ "properties": {
+ "question": {
+ "type": "string",
+ "description": "A concise question to answer using the captured image.",
+ "minLength": 1,
+ "maxLength": MAX_CAMERA_QUESTION_CHARACTERS,
+ }
+ },
+ "required": ["question"],
+ "additionalProperties": False,
+ },
+}
+
+_MOTION_TOOL_SPECS: Final[list[dict[str, Any]]] = [_MOVE_HEAD_SPEC, _STOP_MOTION_SPEC]
+
+# REST XYZRPYPose values are expressed in radians. These absolute poses match
+# the original local MoveHead tool's fixed direction mapping.
+_DIRECTION_POSES: Final[dict[str, tuple[float, float]]] = {
+ "left": (0.0, math.radians(40.0)),
+ "right": (0.0, math.radians(-40.0)),
+ "up": (math.radians(-30.0), 0.0),
+ "down": (math.radians(30.0), 0.0),
+ "front": (0.0, 0.0),
+}
+
+
+@dataclass(frozen=True)
+class RestTransportSettings:
+ """Validated settings for direct Reachy REST calls."""
+
+ base_url: str = "http://127.0.0.1:8000"
+ camera_base_url: str | None = None
+ request_timeout_seconds: float = 5.0
+ motion_duration_seconds: float = 1.0
+ poll_interval_seconds: float = 0.1
+ completion_timeout_seconds: float = 10.0
+
+ def __post_init__(self) -> None:
+ """Reject invalid network and timing settings before tool discovery."""
+ self._validate_base_url("REACHY_REST_BASE_URL", self.base_url)
+ if self.camera_base_url is not None:
+ self._validate_base_url("REACHY_CAMERA_BASE_URL", self.camera_base_url)
+
+ for name, value in (
+ ("REACHY_REST_TIMEOUT_SECONDS", self.request_timeout_seconds),
+ ("REACHY_MOTION_DURATION_SECONDS", self.motion_duration_seconds),
+ ("REACHY_MOTION_POLL_INTERVAL_SECONDS", self.poll_interval_seconds),
+ ("REACHY_MOTION_COMPLETION_TIMEOUT_SECONDS", self.completion_timeout_seconds),
+ ):
+ if not math.isfinite(value) or value <= 0:
+ raise ValueError(f"{name} must be a positive finite number")
+
+ @staticmethod
+ def _validate_base_url(name: str, value: str) -> None:
+ parsed = urlsplit(value)
+ if parsed.scheme not in {"http", "https"} or not parsed.netloc:
+ raise ValueError(f"{name} must be an absolute HTTP(S) URL")
+ if parsed.path not in {"", "/"}:
+ raise ValueError(f"{name} must not contain a path")
+ if parsed.query or parsed.fragment:
+ raise ValueError(f"{name} must not contain a query string or fragment")
+
+
+class RestToolTransport:
+ """Expose a deliberately small Reachy tool set through its daemon REST API."""
+
+ def __init__(
+ self,
+ settings: RestTransportSettings,
+ *,
+ client: httpx.AsyncClient | None = None,
+ camera_client: httpx.AsyncClient | None = None,
+ ) -> None:
+ """Create the transport without probing or moving the robot."""
+ self.settings = settings
+ self._base_url = settings.base_url.rstrip("/")
+ self._camera_base_url = settings.camera_base_url.rstrip("/") if settings.camera_base_url else None
+ if camera_client is not None and self._camera_base_url is None:
+ raise ValueError("camera_client requires REACHY_CAMERA_BASE_URL")
+ self._owns_client = client is None
+ self._client = client or httpx.AsyncClient(
+ base_url=f"{settings.base_url.rstrip('/')}/",
+ timeout=httpx.Timeout(settings.request_timeout_seconds),
+ follow_redirects=False,
+ )
+ self._owns_camera_client = camera_client is None and self._camera_base_url is not None
+ self._camera_client = camera_client
+ if self._camera_client is None and self._camera_base_url is not None:
+ self._camera_client = httpx.AsyncClient(
+ base_url=f"{self._camera_base_url}/",
+ timeout=httpx.Timeout(settings.request_timeout_seconds),
+ follow_redirects=False,
+ )
+ self._active_move_ids: set[str] = set()
+ self._closed = False
+
+ async def list_tools(self) -> list[dict[str, Any]]:
+ """Return fixed motion schemas plus camera only when its adapter is configured."""
+ schemas = [*_MOTION_TOOL_SPECS]
+ if self._camera_client is not None:
+ schemas.append(_CAMERA_SPEC)
+ return deepcopy(schemas)
+
+ async def call_tool(self, name: str, arguments: dict[str, Any]) -> dict[str, Any]:
+ """Validate model arguments and invoke the corresponding REST operation."""
+ if self._closed:
+ return {"status": "robot_unavailable", "tool": name, "error": "Reachy REST transport is closed"}
+ if name == "move_head":
+ return await self._move_head(arguments)
+ if name == "stop_motion":
+ if arguments:
+ return self._invalid_arguments(name, "stop_motion does not accept arguments")
+ return await self._stop_motion()
+ if name == "camera" and self._camera_client is not None:
+ return await self._capture_image(arguments)
+ return {"status": "unknown_tool", "tool": name, "error": f"Unknown REST tool: {name}"}
+
+ async def close(self) -> None:
+ """Close the owned HTTP client once."""
+ if self._closed:
+ return
+ self._closed = True
+ if self._owns_client:
+ await self._client.aclose()
+ if self._owns_camera_client and self._camera_client is not None and self._camera_client is not self._client:
+ await self._camera_client.aclose()
+
+ async def _capture_image(self, arguments: dict[str, Any]) -> dict[str, Any]:
+ if set(arguments) != {"question"}:
+ return self._invalid_arguments("camera", "camera accepts only the question field")
+ question = arguments.get("question")
+ if not isinstance(question, str) or not question.strip():
+ return self._invalid_arguments("camera", "question must be a non-empty string")
+ question = question.strip()
+ if len(question) > MAX_CAMERA_QUESTION_CHARACTERS:
+ return self._invalid_arguments(
+ "camera",
+ f"question must contain at most {MAX_CAMERA_QUESTION_CHARACTERS} characters",
+ )
+
+ assert self._camera_client is not None
+ assert self._camera_base_url is not None
+ path = "/camera/capture"
+ try:
+ response = await self._camera_client.post(f"{self._camera_base_url}{path}")
+ except httpx.TimeoutException:
+ return {
+ "status": "unknown_delivery",
+ "tool": "camera",
+ "error": "Camera request timed out; a picture may have been captured and the request was not retried",
+ }
+ except httpx.RequestError as exc:
+ return {
+ "status": "camera_unavailable",
+ "tool": "camera",
+ "error": f"Reachy camera adapter is unavailable: {type(exc).__name__}",
+ }
+
+ error = self._response_error(response, "camera", path)
+ if error is not None:
+ return error
+
+ jpeg = response.content
+ content_type = response.headers.get("content-type", "").split(";", 1)[0].strip().lower()
+ if content_type != "image/jpeg" or not jpeg.startswith(b"\xff\xd8"):
+ return {
+ "status": "camera_rejected",
+ "tool": "camera",
+ "error": "Reachy camera adapter returned an invalid JPEG",
+ }
+ if not jpeg or len(jpeg) > MAX_CAMERA_JPEG_BYTES:
+ return {
+ "status": "camera_rejected",
+ "tool": "camera",
+ "error": "Reachy camera adapter returned an image outside the allowed size limit",
+ }
+
+ return {
+ "status": "captured",
+ "tool": "camera",
+ "question": question,
+ "b64_im": base64.b64encode(jpeg).decode("ascii"),
+ }
+
+ async def _move_head(self, arguments: dict[str, Any]) -> dict[str, Any]:
+ if set(arguments) != {"directions"}:
+ return self._invalid_arguments("move_head", "move_head accepts only the directions field")
+
+ raw_directions = arguments.get("directions")
+ if not isinstance(raw_directions, list) or not 1 <= len(raw_directions) <= 8:
+ return self._invalid_arguments("move_head", "directions must contain between 1 and 8 values")
+ if not all(isinstance(direction, str) and direction in _DIRECTION_POSES for direction in raw_directions):
+ return self._invalid_arguments(
+ "move_head",
+ f"directions must contain only {list(_DIRECTION_POSES)}",
+ )
+
+ directions = cast(list[str], list(raw_directions))
+ completed: list[str] = []
+ try:
+ for direction in directions:
+ result = await self._goto_and_wait(direction)
+ if result.get("status") != "completed":
+ result["directions"] = directions
+ result["completed_directions"] = completed
+ return result
+ completed.append(direction)
+ except asyncio.CancelledError:
+ with suppress(Exception):
+ await asyncio.shield(self._stop_motion())
+ raise
+
+ return {
+ "status": "completed",
+ "tool": "move_head",
+ "directions": directions,
+ "total_duration_seconds": self.settings.motion_duration_seconds * len(directions),
+ }
+
+ async def _goto_and_wait(self, direction: str) -> dict[str, Any]:
+ pitch, yaw = _DIRECTION_POSES[direction]
+ payload = {
+ "head_pose": {
+ "x": 0.0,
+ "y": 0.0,
+ "z": 0.0,
+ "roll": 0.0,
+ "pitch": pitch,
+ "yaw": yaw,
+ },
+ "duration": self.settings.motion_duration_seconds,
+ "interpolation": "minjerk",
+ }
+
+ try:
+ response = await self._client.post(self._url("/api/move/goto"), json=payload)
+ except httpx.TimeoutException:
+ return {
+ "status": "unknown_delivery",
+ "tool": "move_head",
+ "direction": direction,
+ "error": "Reachy motion request timed out; it was not retried",
+ }
+ except httpx.RequestError as exc:
+ return self._request_error("move_head", exc)
+
+ error = self._response_error(response, "move_head", "/api/move/goto")
+ if error is not None:
+ return error
+
+ move_id = self._move_id_from_response(response)
+ if move_id is None:
+ return {
+ "status": "robot_rejected",
+ "tool": "move_head",
+ "direction": direction,
+ "error": "Reachy returned an invalid move identifier",
+ }
+
+ self._active_move_ids.add(move_id)
+ deadline = asyncio.get_running_loop().time() + self.settings.completion_timeout_seconds
+ while True:
+ running = await self._running_moves("move_head")
+ if isinstance(running, dict):
+ running["direction"] = direction
+ return running
+ if move_id not in running:
+ self._active_move_ids.discard(move_id)
+ return {"status": "completed", "tool": "move_head", "direction": direction, "move_id": move_id}
+ if asyncio.get_running_loop().time() >= deadline:
+ await self._stop_move_best_effort(move_id)
+ return {
+ "status": "motion_timeout",
+ "tool": "move_head",
+ "direction": direction,
+ "move_id": move_id,
+ "error": "Reachy motion did not complete before the safety timeout",
+ }
+ await asyncio.sleep(self.settings.poll_interval_seconds)
+
+ async def _stop_motion(self) -> dict[str, Any]:
+ running = await self._running_moves("stop_motion")
+ if isinstance(running, dict):
+ return running
+
+ move_ids = sorted(running | self._active_move_ids)
+ stopped: list[str] = []
+ for move_id in move_ids:
+ try:
+ response = await self._client.post(self._url("/api/move/stop"), json={"uuid": move_id})
+ except httpx.TimeoutException:
+ return {
+ "status": "unknown_delivery",
+ "tool": "stop_motion",
+ "stopped_move_ids": stopped,
+ "error": "Reachy stop request timed out; it was not retried",
+ }
+ except httpx.RequestError as exc:
+ result = self._request_error("stop_motion", exc)
+ result["stopped_move_ids"] = stopped
+ return result
+
+ error = self._response_error(response, "stop_motion", "/api/move/stop")
+ if error is not None:
+ error["stopped_move_ids"] = stopped
+ return error
+ stopped.append(move_id)
+ self._active_move_ids.discard(move_id)
+
+ return {"status": "stopped", "tool": "stop_motion", "stopped_move_ids": stopped}
+
+ async def _running_moves(self, tool: str) -> set[str] | dict[str, Any]:
+ try:
+ response = await self._client.get(self._url("/api/move/running"))
+ except httpx.RequestError as exc:
+ return self._request_error(tool, exc)
+
+ error = self._response_error(response, tool, "/api/move/running")
+ if error is not None:
+ return error
+ try:
+ payload = response.json()
+ except ValueError:
+ payload = None
+ if not isinstance(payload, list):
+ return {
+ "status": "robot_rejected",
+ "tool": tool,
+ "error": "Reachy returned an invalid running-moves response",
+ }
+
+ move_ids: set[str] = set()
+ for item in payload:
+ move_id = self._validated_move_id(item.get("uuid") if isinstance(item, dict) else None)
+ if move_id is None:
+ return {
+ "status": "robot_rejected",
+ "tool": tool,
+ "error": "Reachy returned an invalid running move identifier",
+ }
+ move_ids.add(move_id)
+ return move_ids
+
+ async def _stop_move_best_effort(self, move_id: str) -> None:
+ try:
+ response = await self._client.post(self._url("/api/move/stop"), json={"uuid": move_id})
+ if response.is_success:
+ self._active_move_ids.discard(move_id)
+ except httpx.RequestError:
+ return
+
+ @classmethod
+ def _move_id_from_response(cls, response: httpx.Response) -> str | None:
+ try:
+ payload = response.json()
+ except ValueError:
+ return None
+ return cls._validated_move_id(payload.get("uuid") if isinstance(payload, dict) else None)
+
+ @staticmethod
+ def _validated_move_id(value: Any) -> str | None:
+ if not isinstance(value, str):
+ return None
+ try:
+ return str(UUID(value))
+ except ValueError:
+ return None
+
+ def _url(self, path: str) -> str:
+ return f"{self._base_url}{path}"
+
+ @staticmethod
+ def _invalid_arguments(tool: str, message: str) -> dict[str, Any]:
+ return {"status": "invalid_arguments", "tool": tool, "error": message}
+
+ @staticmethod
+ def _request_error(tool: str, exc: httpx.RequestError) -> dict[str, Any]:
+ return {
+ "status": "robot_unavailable",
+ "tool": tool,
+ "error": f"Reachy REST API is unavailable: {type(exc).__name__}",
+ }
+
+ @staticmethod
+ def _response_error(response: httpx.Response, tool: str, path: str) -> dict[str, Any] | None:
+ if response.status_code == httpx.codes.FORBIDDEN:
+ return {
+ "status": "policy_denied",
+ "tool": tool,
+ "error": f"{POLICY_DENIED_ERROR}: {response.request.method} {path}",
+ }
+ if response.is_error:
+ return {
+ "status": "robot_rejected",
+ "tool": tool,
+ "http_status": response.status_code,
+ "error": f"Reachy rejected {response.request.method} {path}",
+ }
+ return None
diff --git a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/robot_runtime.py b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/robot_runtime.py
new file mode 100644
index 00000000..f6235b94
--- /dev/null
+++ b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/robot_runtime.py
@@ -0,0 +1,223 @@
+"""Reusable lifecycle for a Reachy Mini robot and its local workers."""
+
+from __future__ import annotations
+import time
+import logging
+from types import SimpleNamespace
+from typing import Any, Callable
+from pathlib import Path
+from dataclasses import field, dataclass
+
+from reachy_mini import ReachyMini
+from reachy_mini_conversation_app.moves import MovementManager
+from reachy_mini_conversation_app.utils import handle_vision_stuff
+from reachy_mini_conversation_app.vision_router import build_vision_router
+from reachy_mini_conversation_app.tools.core_tools import ToolDependencies
+from reachy_mini_conversation_app.audio.head_wobbler import HeadWobbler
+
+
+logger = logging.getLogger(__name__)
+
+
+def shutdown_step(log: Any, name: str, callback: Callable[[], Any]) -> None:
+ """Run one shutdown callback without interrupting later cleanup steps."""
+ try:
+ callback()
+ except KeyboardInterrupt:
+ log.warning("Shutdown interrupted while stopping %s; continuing cleanup.", name)
+ except Exception as exc:
+ log.debug("Error while stopping %s: %s", name, exc)
+
+
+def _status_flag(status: Any, name: str) -> bool:
+ """Read a boolean daemon-status field from a mapping or SDK object."""
+ if isinstance(status, dict):
+ return bool(status.get(name, False))
+ return bool(getattr(status, name, False))
+
+
+@dataclass
+class ReachyRuntime:
+ """Own the connected robot, its workers, and their coordinated lifecycle."""
+
+ robot: ReachyMini
+ camera_worker: Any | None
+ movement_manager: MovementManager
+ dependencies: ToolDependencies
+ head_wobbler: HeadWobbler
+ head_tracker: Any | None = None
+ vision_manager: Any | None = None
+ vision_router: Any | None = None
+ simulation_enabled: bool = False
+ mockup_sim_enabled: bool = False
+ log: Any = logger
+ shutdown_delay_seconds: float = 1.0
+ _started: bool = field(default=False, init=False, repr=False)
+ _stopped: bool = field(default=False, init=False, repr=False)
+
+ @property
+ def is_simulation(self) -> bool:
+ """Return whether either supported simulation backend is active."""
+ return self.simulation_enabled or self.mockup_sim_enabled
+
+ @classmethod
+ def connect(
+ cls,
+ *,
+ robot_name: str | None = None,
+ robot_host: str | None = None,
+ robot_port: int | None = None,
+ connection_mode: str | None = None,
+ media_backend: str | None = None,
+ robot: ReachyMini | None = None,
+ no_camera: bool = False,
+ head_tracker: str | None = None,
+ local_vision: bool = False,
+ enable_vision_router: bool = True,
+ movement_frequency_hz: float = 100.0,
+ enable_idle_breathing: bool = True,
+ capture_directory: Path | None = None,
+ log: Any = logger,
+ shutdown_delay_seconds: float = 1.0,
+ ) -> "ReachyRuntime":
+ """Connect to Reachy and construct every worker that depends on it.
+
+ Passing an existing ``robot`` preserves the Reachy Mini Apps integration,
+ while standalone local-mode callers can let the runtime create its own
+ SDK connection.
+ """
+ current_robot = robot
+ if current_robot is None:
+ robot_kwargs: dict[str, Any] = {}
+ if robot_name is not None:
+ robot_kwargs["robot_name"] = robot_name
+ if robot_host is not None:
+ robot_kwargs["host"] = robot_host
+ if robot_port is not None:
+ robot_kwargs["port"] = robot_port
+ if connection_mode is not None:
+ robot_kwargs["connection_mode"] = connection_mode
+ if media_backend is not None:
+ robot_kwargs["media_backend"] = media_backend
+
+ log.info("Initializing ReachyMini (SDK will auto-detect appropriate backend)")
+ current_robot = ReachyMini(**robot_kwargs)
+
+ status = current_robot.client.get_status()
+ simulation_enabled = _status_flag(status, "simulation_enabled")
+ mockup_sim_enabled = _status_flag(status, "mockup_sim_enabled")
+
+ vision_args = SimpleNamespace(
+ no_camera=no_camera,
+ head_tracker=head_tracker,
+ local_vision=local_vision,
+ )
+ camera_worker, initialized_head_tracker, vision_manager = handle_vision_stuff(
+ vision_args,
+ current_robot,
+ )
+
+ vision_router = None
+ if enable_vision_router and camera_worker is not None and vision_manager is None:
+ vision_router = build_vision_router()
+
+ movement_manager = MovementManager(
+ current_robot=current_robot,
+ camera_worker=camera_worker,
+ target_frequency_hz=movement_frequency_hz,
+ enable_idle_breathing=enable_idle_breathing,
+ )
+ head_wobbler = HeadWobbler(set_speech_offsets=movement_manager.set_speech_offsets)
+ dependencies = ToolDependencies(
+ reachy_mini=current_robot,
+ movement_manager=movement_manager,
+ camera_worker=camera_worker,
+ vision_manager=vision_manager,
+ vision_router=vision_router,
+ head_wobbler=head_wobbler,
+ capture_directory=(capture_directory or Path("captures")).expanduser(),
+ )
+
+ return cls(
+ robot=current_robot,
+ camera_worker=camera_worker,
+ movement_manager=movement_manager,
+ dependencies=dependencies,
+ head_wobbler=head_wobbler,
+ head_tracker=initialized_head_tracker,
+ vision_manager=vision_manager,
+ vision_router=vision_router,
+ simulation_enabled=simulation_enabled,
+ mockup_sim_enabled=mockup_sim_enabled,
+ log=log,
+ shutdown_delay_seconds=shutdown_delay_seconds,
+ )
+
+ def start(self) -> None:
+ """Start each robot worker once, preserving the app's existing order."""
+ if self._started:
+ self.log.debug("Reachy runtime already started; start() ignored")
+ return
+ if self._stopped:
+ raise RuntimeError("A stopped Reachy runtime cannot be restarted; create a new runtime")
+
+ started_components: list[tuple[str, Any]] = []
+ components = [
+ ("movement manager", self.movement_manager),
+ ("head wobbler", self.head_wobbler),
+ ("camera worker", self.camera_worker),
+ ("vision manager", self.vision_manager),
+ ]
+ try:
+ for name, component in components:
+ if component is None:
+ continue
+ component.start()
+ started_components.append((name, component))
+ except BaseException:
+ for name, component in reversed(started_components):
+ shutdown_step(self.log, name, component.stop)
+ raise
+
+ self._started = True
+ self._stopped = False
+ self.log.debug("Reachy runtime started")
+
+ @property
+ def is_connected(self) -> bool:
+ """Return the live SDK/movement connection state used by hardware tools."""
+ if self._stopped:
+ return False
+ checker = getattr(self.movement_manager, "connection_healthy", None)
+ if callable(checker):
+ return bool(checker())
+ client_alive = getattr(self.robot.client, "_is_alive", None)
+ return True if client_alive is None else bool(client_alive)
+
+ @property
+ def connection_error(self) -> str | None:
+ """Return the movement transport's terminal error when available."""
+ return getattr(self.movement_manager, "delivery_error", None)
+
+ def stop(self) -> None:
+ """Stop all workers and disconnect the SDK without skipping cleanup."""
+ if self._stopped:
+ self.log.debug("Reachy runtime already stopped; stop() ignored")
+ return
+
+ shutdown_step(self.log, "movement manager", self.movement_manager.stop)
+ shutdown_step(self.log, "head wobbler", self.head_wobbler.stop)
+ if self.camera_worker is not None:
+ shutdown_step(self.log, "camera worker", self.camera_worker.stop)
+ if self.vision_manager is not None:
+ shutdown_step(self.log, "vision manager", self.vision_manager.stop)
+
+ shutdown_step(self.log, "media", self.robot.media.close)
+ shutdown_step(self.log, "robot client", self.robot.client.disconnect)
+
+ if self.shutdown_delay_seconds > 0:
+ time.sleep(self.shutdown_delay_seconds)
+
+ self._started = False
+ self._stopped = True
+ self.log.info("Shutdown complete.")
diff --git a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/sandbox_audio.py b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/sandbox_audio.py
new file mode 100644
index 00000000..6e3600f6
--- /dev/null
+++ b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/sandbox_audio.py
@@ -0,0 +1,289 @@
+"""Headless audio service for the OpenShell-hosted conversation agent."""
+
+from __future__ import annotations
+import os
+import json
+import asyncio
+import logging
+from typing import Any, Protocol
+from pathlib import Path
+from collections.abc import Callable
+
+import numpy as np
+import uvicorn
+from fastapi import FastAPI, WebSocket, WebSocketDisconnect
+from fastrtc import AdditionalOutputs
+from numpy.typing import NDArray
+
+from reachy_mini_conversation_app.audio.pcm import prepare_mono_int16_audio
+
+
+logger = logging.getLogger(__name__)
+
+WIRE_FORMAT = "pcm_s16le"
+WIRE_SAMPLE_RATE = 16_000
+WIRE_CHANNELS = 1
+DEFAULT_HOST = "127.0.0.1"
+DEFAULT_PORT = 8765
+MAX_AUDIO_FRAME_BYTES = WIRE_SAMPLE_RATE * 2 * 2
+
+
+class ConversationHandler(Protocol):
+ """Subset of ConversationStreamHandler used by the audio service."""
+
+ async def start_up(self) -> None:
+ """Run the model session until shutdown."""
+ ...
+
+ async def receive(self, frame: tuple[int, NDArray[Any]]) -> None:
+ """Accept one microphone frame."""
+ ...
+
+ async def emit(self) -> Any:
+ """Return the next audio or message output."""
+ ...
+
+ async def shutdown(self) -> None:
+ """Close model and tool resources."""
+ ...
+
+
+HandlerFactory = Callable[[], ConversationHandler]
+
+
+def _build_handler() -> ConversationHandler:
+ """Build the normal conversation handler with policy-routed REST tools."""
+ from reachy_mini_conversation_app.main import _build_tool_transport_factory
+ from reachy_mini_conversation_app.config import TOOL_TRANSPORT_REST, config
+ from reachy_mini_conversation_app.tools.core_tools import ToolDependencies
+ from reachy_mini_conversation_app.conversation_stream import ConversationStreamHandler
+
+ tool_transport_mode = config.REACHY_TOOL_TRANSPORT
+ if tool_transport_mode != TOOL_TRANSPORT_REST:
+ raise RuntimeError(
+ "The sandbox audio service requires REACHY_TOOL_TRANSPORT=rest so all robot actions cross OpenShell policy"
+ )
+
+ dependencies = ToolDependencies(
+ capture_directory=Path(os.getenv("REACHY_CAPTURE_DIR", "/sandbox/captures")).expanduser(),
+ )
+ tool_transport_factory = _build_tool_transport_factory(
+ tool_transport_mode,
+ dependencies,
+ rest_base_url=config.REACHY_REST_BASE_URL,
+ camera_base_url=config.REACHY_CAMERA_BASE_URL,
+ rest_timeout_seconds=config.REACHY_REST_TIMEOUT_SECONDS,
+ motion_duration_seconds=config.REACHY_MOTION_DURATION_SECONDS,
+ motion_poll_interval_seconds=config.REACHY_MOTION_POLL_INTERVAL_SECONDS,
+ motion_completion_timeout_seconds=config.REACHY_MOTION_COMPLETION_TIMEOUT_SECONDS,
+ )
+ return ConversationStreamHandler(
+ dependencies,
+ gradio_mode=False,
+ model_logs=os.getenv("REACHY_MODEL_LOGS", "1").strip().lower() not in {"0", "false", "no", "off"},
+ tool_transport_factory=tool_transport_factory,
+ )
+
+
+def _hello_payload() -> dict[str, Any]:
+ return {
+ "type": "hello",
+ "format": WIRE_FORMAT,
+ "sample_rate": WIRE_SAMPLE_RATE,
+ "channels": WIRE_CHANNELS,
+ }
+
+
+def _validate_hello(payload: Any) -> str | None:
+ if not isinstance(payload, dict) or payload.get("type") != "hello":
+ return "first message must be a hello object"
+ if payload.get("format") != WIRE_FORMAT:
+ return f"format must be {WIRE_FORMAT}"
+ if payload.get("sample_rate") != WIRE_SAMPLE_RATE:
+ return f"sample_rate must be {WIRE_SAMPLE_RATE}"
+ if payload.get("channels") != WIRE_CHANNELS:
+ return f"channels must be {WIRE_CHANNELS}"
+ return None
+
+
+def _safe_additional_outputs(output: AdditionalOutputs) -> list[dict[str, str]]:
+ """Return only text metadata needed by the trusted native bridge."""
+ messages: list[dict[str, str]] = []
+ for item in output.args:
+ if not isinstance(item, dict):
+ continue
+ role = item.get("role")
+ content = item.get("content")
+ if isinstance(role, str) and isinstance(content, str):
+ messages.append({"role": role, "content": content})
+ return messages
+
+
+async def _receive_audio(websocket: WebSocket, handler: ConversationHandler) -> None:
+ while True:
+ message = await websocket.receive()
+ message_type = message.get("type")
+ if message_type == "websocket.disconnect":
+ return
+
+ payload = message.get("bytes")
+ if payload is not None:
+ if len(payload) == 0:
+ continue
+ if len(payload) > MAX_AUDIO_FRAME_BYTES:
+ await websocket.close(code=1009, reason="audio frame too large")
+ return
+ if len(payload) % 2:
+ await websocket.close(code=1003, reason="PCM frame must contain complete int16 samples")
+ return
+ audio = np.frombuffer(payload, dtype=" None:
+ while True:
+ output = await handler.emit()
+ if output is None:
+ continue
+ if isinstance(output, AdditionalOutputs):
+ messages = _safe_additional_outputs(output)
+ if messages:
+ await websocket.send_json({"type": "messages", "messages": messages})
+ continue
+
+ if not isinstance(output, tuple) or len(output) != 2:
+ logger.warning("Ignoring unsupported handler output type %s", type(output).__name__)
+ continue
+
+ sample_rate, audio = output
+ if not isinstance(sample_rate, int) or not isinstance(audio, np.ndarray):
+ logger.warning("Ignoring malformed audio output")
+ continue
+ wire_audio = prepare_mono_int16_audio((sample_rate, audio), WIRE_SAMPLE_RATE)
+ if wire_audio.size:
+ await websocket.send_bytes(wire_audio.astype(" None:
+ try:
+ await handler.shutdown()
+ finally:
+ if not startup_task.done():
+ try:
+ await asyncio.wait_for(startup_task, timeout=3.0)
+ except TimeoutError:
+ startup_task.cancel()
+ if startup_task.cancelled():
+ return
+ if startup_task.done():
+ try:
+ startup_task.result()
+ except asyncio.CancelledError:
+ pass
+ except Exception:
+ logger.exception("Conversation startup task failed during shutdown")
+
+
+def create_audio_app(handler_factory: HandlerFactory = _build_handler) -> FastAPI:
+ """Create the single-client, loopback-only audio service."""
+ application = FastAPI(title="Reachy OpenShell Audio", docs_url=None, redoc_url=None)
+ client_lock = asyncio.Lock()
+
+ @application.get("/health")
+ async def health() -> dict[str, Any]:
+ return {
+ "status": "ok",
+ "active_audio_client": client_lock.locked(),
+ **_hello_payload(),
+ }
+
+ @application.websocket("/audio")
+ async def audio(websocket: WebSocket) -> None:
+ if client_lock.locked():
+ await websocket.close(code=1013, reason="another Reachy audio client is already connected")
+ return
+
+ async with client_lock:
+ await websocket.accept()
+ try:
+ hello = await asyncio.wait_for(websocket.receive_json(), timeout=5.0)
+ except (TimeoutError, WebSocketDisconnect, json.JSONDecodeError):
+ await websocket.close(code=1002, reason="valid hello message required")
+ return
+
+ hello_error = _validate_hello(hello)
+ if hello_error is not None:
+ await websocket.close(code=1002, reason=hello_error)
+ return
+ await websocket.send_json({**_hello_payload(), "type": "ready"})
+
+ try:
+ handler = handler_factory()
+ except Exception as exc:
+ logger.exception("Unable to build conversation handler")
+ await websocket.send_json({"type": "error", "message": f"agent startup failed: {type(exc).__name__}"})
+ await websocket.close(code=1011)
+ return
+
+ startup_task = asyncio.create_task(handler.start_up(), name="conversation-handler")
+ receive_task = asyncio.create_task(_receive_audio(websocket, handler), name="robot-audio-input")
+ send_task = asyncio.create_task(_send_outputs(websocket, handler), name="robot-audio-output")
+ try:
+ done, pending = await asyncio.wait(
+ {startup_task, receive_task, send_task},
+ return_when=asyncio.FIRST_COMPLETED,
+ )
+ for task in done:
+ if task is startup_task and task.exception() is not None:
+ raise task.exception() # type: ignore[misc]
+ for task in pending:
+ if task is not startup_task:
+ task.cancel()
+ except WebSocketDisconnect:
+ pass
+ except Exception:
+ logger.exception("Audio session failed")
+ finally:
+ receive_task.cancel()
+ send_task.cancel()
+ await asyncio.gather(receive_task, send_task, return_exceptions=True)
+ await _close_handler(handler, startup_task)
+ try:
+ await websocket.close(code=1000)
+ except RuntimeError:
+ # The peer may already have sent a disconnect frame.
+ pass
+
+ return application
+
+
+app = create_audio_app()
+
+
+def main() -> None:
+ """Run the loopback audio service inside the OpenShell sandbox."""
+ host = os.getenv("REACHY_AUDIO_HOST", DEFAULT_HOST)
+ port = int(os.getenv("REACHY_AUDIO_PORT", str(DEFAULT_PORT)))
+ if host not in {"127.0.0.1", "localhost", "::1"}:
+ raise SystemExit("REACHY_AUDIO_HOST must remain loopback-only")
+ uvicorn.run(app, host=host, port=port, log_level=os.getenv("REACHY_AUDIO_LOG_LEVEL", "info"))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/sandbox_control.py b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/sandbox_control.py
new file mode 100644
index 00000000..e11d59f0
--- /dev/null
+++ b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/sandbox_control.py
@@ -0,0 +1,202 @@
+"""Idempotent lifecycle control for the agent process inside a sandbox."""
+
+from __future__ import annotations
+import os
+import sys
+import time
+import signal
+import argparse
+import subprocess
+from pathlib import Path
+from dataclasses import dataclass
+
+
+@dataclass(frozen=True)
+class ControlSettings:
+ """Filesystem, process, and health settings for sandbox lifecycle control."""
+
+ state_directory: Path = Path("/sandbox/run")
+ log_directory: Path = Path("/sandbox/logs")
+ listen_port: int = 8765
+ startup_timeout_seconds: float = 120.0
+ shutdown_timeout_seconds: float = 10.0
+
+ @property
+ def pid_path(self) -> Path:
+ """Return the PID file path."""
+ return self.state_directory / "reachy-agent.pid"
+
+ @property
+ def log_path(self) -> Path:
+ """Return the detached agent log path."""
+ return self.log_directory / "reachy-agent.log"
+
+
+def _read_pid(path: Path) -> int | None:
+ try:
+ pid = int(path.read_text(encoding="utf-8").strip())
+ except (FileNotFoundError, ValueError, OSError):
+ return None
+ return pid if pid > 1 else None
+
+
+def _process_exists(pid: int) -> bool:
+ try:
+ os.kill(pid, 0)
+ except ProcessLookupError:
+ return False
+ except PermissionError:
+ return True
+ return True
+
+
+def _port_is_listening(port: int) -> bool:
+ """Check Linux TCP tables without making a policy-controlled network call.
+
+ OpenShell deliberately blocks sandbox egress to loopback. The gateway can
+ still forward an exposed service to a loopback listener, so lifecycle
+ readiness must be established from kernel state instead of an HTTP probe.
+ """
+ expected_port = f"{port:04X}"
+ for table_path in (Path("/proc/net/tcp"), Path("/proc/net/tcp6")):
+ try:
+ rows = table_path.read_text(encoding="ascii").splitlines()[1:]
+ except OSError:
+ continue
+ for row in rows:
+ fields = row.split()
+ if len(fields) < 4:
+ continue
+ local_address = fields[1]
+ state = fields[3]
+ _, separator, encoded_port = local_address.rpartition(":")
+ if separator and encoded_port.upper() == expected_port and state == "0A":
+ return True
+ return False
+
+
+def _wait_until(predicate: object, timeout: float, interval: float = 0.1) -> bool:
+ deadline = time.monotonic() + timeout
+ while time.monotonic() < deadline:
+ if callable(predicate) and predicate():
+ return True
+ time.sleep(interval)
+ return bool(callable(predicate) and predicate())
+
+
+class SandboxAgentControl:
+ """Start, stop, and inspect the detached sandbox audio agent."""
+
+ def __init__(self, settings: ControlSettings | None = None) -> None:
+ """Initialize lifecycle control with environment-aware defaults."""
+ self.settings = settings or ControlSettings(
+ state_directory=Path(os.getenv("REACHY_AGENT_STATE_DIR", "/sandbox/run")),
+ log_directory=Path(os.getenv("REACHY_AGENT_LOG_DIR", "/sandbox/logs")),
+ listen_port=int(os.getenv("REACHY_AUDIO_PORT", "8765")),
+ startup_timeout_seconds=float(os.getenv("REACHY_AGENT_START_TIMEOUT_SECONDS", "120")),
+ shutdown_timeout_seconds=float(os.getenv("REACHY_AGENT_STOP_TIMEOUT_SECONDS", "10")),
+ )
+
+ def status(self) -> str:
+ """Return stopped, unhealthy, or running."""
+ pid = _read_pid(self.settings.pid_path)
+ if pid is None or not _process_exists(pid):
+ return "stopped"
+ return "running" if _port_is_listening(self.settings.listen_port) else "unhealthy"
+
+ def start(self) -> int:
+ """Start the agent if needed and wait for health."""
+ self.settings.state_directory.mkdir(parents=True, exist_ok=True)
+ self.settings.log_directory.mkdir(parents=True, exist_ok=True)
+
+ current_status = self.status()
+ if current_status == "running":
+ print("reachy-agent is already running")
+ return 0
+ if current_status == "unhealthy":
+ print("reachy-agent process exists but is unhealthy; stop it before restarting", file=sys.stderr)
+ return 1
+
+ self.settings.pid_path.unlink(missing_ok=True)
+ command = [sys.executable, "-m", "reachy_mini_conversation_app.sandbox_audio"]
+ with self.settings.log_path.open("ab", buffering=0) as log_file:
+ process = subprocess.Popen( # noqa: S603 - fixed internal command
+ command,
+ stdin=subprocess.DEVNULL,
+ stdout=log_file,
+ stderr=subprocess.STDOUT,
+ cwd="/sandbox",
+ start_new_session=True,
+ close_fds=True,
+ )
+
+ temporary_pid = self.settings.pid_path.with_suffix(".pid.tmp")
+ temporary_pid.write_text(f"{process.pid}\n", encoding="utf-8")
+ temporary_pid.replace(self.settings.pid_path)
+
+ ready = _wait_until(
+ lambda: process.poll() is None and _port_is_listening(self.settings.listen_port),
+ self.settings.startup_timeout_seconds,
+ )
+ if ready:
+ print(f"reachy-agent started pid={process.pid}")
+ return 0
+
+ self._signal_process(process.pid, signal.SIGTERM)
+ _wait_until(lambda: not _process_exists(process.pid), 2.0)
+ if _process_exists(process.pid):
+ self._signal_process(process.pid, signal.SIGKILL)
+ self.settings.pid_path.unlink(missing_ok=True)
+ print(f"reachy-agent failed to become healthy; inspect {self.settings.log_path}", file=sys.stderr)
+ return 1
+
+ def stop(self) -> int:
+ """Stop the agent process group and clear its PID file."""
+ pid = _read_pid(self.settings.pid_path)
+ if pid is None or not _process_exists(pid):
+ self.settings.pid_path.unlink(missing_ok=True)
+ print("reachy-agent is already stopped")
+ return 0
+
+ self._signal_process(pid, signal.SIGTERM)
+ stopped = _wait_until(lambda: not _process_exists(pid), self.settings.shutdown_timeout_seconds)
+ if not stopped:
+ self._signal_process(pid, signal.SIGKILL)
+ stopped = _wait_until(lambda: not _process_exists(pid), 2.0)
+ if stopped:
+ self.settings.pid_path.unlink(missing_ok=True)
+ print("reachy-agent stopped")
+ return 0
+ print(f"reachy-agent pid={pid} did not stop", file=sys.stderr)
+ return 1
+
+ @staticmethod
+ def _signal_process(pid: int, requested_signal: signal.Signals) -> None:
+ try:
+ os.killpg(pid, requested_signal)
+ except ProcessLookupError:
+ return
+
+
+def build_parser() -> argparse.ArgumentParser:
+ """Build the lifecycle CLI parser."""
+ parser = argparse.ArgumentParser(description="Control the Reachy agent inside an OpenShell sandbox")
+ parser.add_argument("command", choices=["start", "stop", "status"])
+ return parser
+
+
+def main() -> None:
+ """Run the sandbox lifecycle command."""
+ args = build_parser().parse_args()
+ control = SandboxAgentControl()
+ if args.command == "start":
+ raise SystemExit(control.start())
+ if args.command == "stop":
+ raise SystemExit(control.stop())
+ status = control.status()
+ print(status)
+ raise SystemExit(0 if status == "running" else 1)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tool_transport.py b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tool_transport.py
new file mode 100644
index 00000000..1e068c50
--- /dev/null
+++ b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tool_transport.py
@@ -0,0 +1,165 @@
+"""Tool transport abstraction and direct in-process implementation."""
+
+from __future__ import annotations
+import json
+from copy import deepcopy
+from typing import Any, Protocol, runtime_checkable
+
+from reachy_mini_conversation_app.tools.core_tools import (
+ ToolDependencies,
+ dispatch_tool_call,
+ get_tool_specs_for_dependencies,
+)
+from reachy_mini_conversation_app.tools.tool_constants import SystemTool
+
+
+CONVERSATION_LOCAL_TOOL_NAMES = frozenset(
+ {
+ "do_nothing",
+ *(tool.value for tool in SystemTool),
+ }
+)
+
+_CONVERSATION_UTILITY_SPECS: tuple[dict[str, Any], ...] = (
+ {
+ "type": "function",
+ "name": "do_nothing",
+ "description": "Stay still and silent.",
+ "parameters": {
+ "type": "object",
+ "properties": {
+ "reason": {
+ "type": "string",
+ "description": "Optional reason for staying still.",
+ }
+ },
+ "required": [],
+ },
+ },
+ {
+ "type": "function",
+ "name": "task_status",
+ "description": "Check the status of background robot actions.",
+ "parameters": {
+ "type": "object",
+ "properties": {
+ "tool_id": {
+ "type": "string",
+ "description": "Specific tool ID to check; omit to list running actions.",
+ }
+ },
+ "required": [],
+ },
+ },
+ {
+ "type": "function",
+ "name": "task_cancel",
+ "description": "Cancel a running background robot action.",
+ "parameters": {
+ "type": "object",
+ "properties": {
+ "tool_id": {
+ "type": "string",
+ "description": "The tool ID to cancel.",
+ }
+ },
+ "required": ["tool_id"],
+ },
+ },
+)
+
+
+@runtime_checkable
+class ToolTransport(Protocol):
+ """Common interface for discovering and invoking conversation tools."""
+
+ async def list_tools(self) -> list[dict[str, Any]]:
+ """Return the tools available through this transport."""
+ ...
+
+ async def call_tool(self, name: str, arguments: dict[str, Any]) -> dict[str, Any]:
+ """Invoke a tool and return its model-visible result."""
+ ...
+
+ async def close(self) -> None:
+ """Release resources owned by the transport."""
+ ...
+
+
+class LocalToolTransport:
+ """Invoke the existing Python tool registry in the conversation process."""
+
+ def __init__(self, dependencies: ToolDependencies) -> None:
+ """Bind the registry to the application's local hardware dependencies."""
+ self._dependencies = dependencies
+
+ async def list_tools(self) -> list[dict[str, Any]]:
+ """Return dependency-compatible local tool schemas."""
+ return deepcopy(get_tool_specs_for_dependencies(self._dependencies))
+
+ async def call_tool(self, name: str, arguments: dict[str, Any]) -> dict[str, Any]:
+ """Dispatch a tool through the existing local Python implementation."""
+ try:
+ arguments_json = json.dumps(arguments)
+ except (TypeError, ValueError) as exc:
+ return {"error": f"Tool arguments are not JSON serializable: {exc}"}
+ return await dispatch_tool_call(name, arguments_json, self._dependencies)
+
+ async def close(self) -> None:
+ """Close the transport; local dispatch owns no transport resources."""
+
+
+class ConversationUtilityTransport:
+ """Provide hardware-free conversation utilities for the REST runtime."""
+
+ async def list_tools(self) -> list[dict[str, Any]]:
+ """Return the small fixed utility set without loading the robot tool registry."""
+ return deepcopy(list(_CONVERSATION_UTILITY_SPECS))
+
+ async def call_tool(self, name: str, arguments: dict[str, Any]) -> dict[str, Any]:
+ """Execute utilities that do not need the background manager."""
+ if name == "do_nothing":
+ reason = arguments.get("reason", "just chilling")
+ return {"status": "doing nothing", "reason": reason}
+ if name in {tool.value for tool in SystemTool}:
+ return {"error": f"{name} requires the conversation background manager"}
+ return {"error": f"unknown local conversation tool: {name}"}
+
+ async def close(self) -> None:
+ """Close the transport; utilities own no external resources."""
+
+
+class RoutedToolTransport:
+ """Route an explicit local allowlist locally and every other tool remotely."""
+
+ def __init__(
+ self,
+ *,
+ remote: ToolTransport,
+ local: ToolTransport,
+ local_tool_names: frozenset[str] = CONVERSATION_LOCAL_TOOL_NAMES,
+ ) -> None:
+ """Configure the remote transport and local-only tool names."""
+ self._remote = remote
+ self._local = local
+ self._local_tool_names = local_tool_names
+
+ async def list_tools(self) -> list[dict[str, Any]]:
+ """Merge remote schemas with only the explicitly permitted local schemas."""
+ remote_tools = await self._remote.list_tools()
+ local_tools = await self._local.list_tools()
+ routed_remote_tools = [tool for tool in remote_tools if tool.get("name") not in self._local_tool_names]
+ routed_local_tools = [tool for tool in local_tools if tool.get("name") in self._local_tool_names]
+ return [*routed_remote_tools, *routed_local_tools]
+
+ async def call_tool(self, name: str, arguments: dict[str, Any]) -> dict[str, Any]:
+ """Invoke local-only tools locally and route all other names remotely."""
+ transport = self._local if name in self._local_tool_names else self._remote
+ return await transport.call_tool(name, arguments)
+
+ async def close(self) -> None:
+ """Close both underlying transports even if one close operation fails."""
+ try:
+ await self._remote.close()
+ finally:
+ await self._local.close()
diff --git a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tools/background_tool_manager.py b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tools/background_tool_manager.py
index 2120aea1..60e5e781 100644
--- a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tools/background_tool_manager.py
+++ b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tools/background_tool_manager.py
@@ -6,6 +6,7 @@
"""
from __future__ import annotations
+import json
import time
import asyncio
import logging
@@ -13,11 +14,8 @@
from pydantic import Field, BaseModel, PrivateAttr
-from reachy_mini_conversation_app.tools.core_tools import (
- ToolDependencies,
- dispatch_tool_call,
- dispatch_tool_call_with_manager,
-)
+from reachy_mini_conversation_app.tool_transport import ToolTransport
+from reachy_mini_conversation_app.tools.core_tools import ToolDependencies
from reachy_mini_conversation_app.tools.tool_constants import ToolState, SystemTool
@@ -26,6 +24,99 @@
_SYSTEM_TOOL_NAMES: set[str] = {t.value for t in SystemTool}
+async def dispatch_tool_call(tool_name: str, args_json: str, deps: ToolDependencies) -> dict[str, Any]:
+ """Load the native registry only when a local tool actually executes."""
+ from reachy_mini_conversation_app.tools.core_tools import dispatch_tool_call as dispatch_local_tool
+
+ return await dispatch_local_tool(tool_name=tool_name, args_json=args_json, deps=deps)
+
+
+def _tool_arguments(args_json_str: str) -> dict[str, Any]:
+ """Parse model tool arguments into a safe object."""
+ try:
+ arguments = json.loads(args_json_str or "{}")
+ except (TypeError, ValueError):
+ return {}
+ return arguments if isinstance(arguments, dict) else {}
+
+
+async def _call_system_tool(
+ tool_name: str,
+ arguments: dict[str, Any],
+ tool_manager: "BackgroundToolManager",
+) -> dict[str, Any]:
+ """Handle manager tools without loading the local robot tool registry."""
+ tool_id = arguments.get("tool_id")
+ if tool_name == SystemTool.TASK_STATUS.value:
+ if isinstance(tool_id, str) and tool_id:
+ tool = tool_manager.get_tool(tool_id)
+ if tool is None:
+ return {"error": f"Tool {tool_id} not found."}
+ result: dict[str, Any] = {
+ "tool_id": tool.tool_id,
+ "name": tool.tool_name,
+ "status": tool.status.value,
+ "started_at": tool.started_at,
+ }
+ if tool.completed_at is not None:
+ result["completed_at"] = tool.completed_at
+ if tool.progress is not None:
+ result["progress_percent"] = f"{tool.progress.progress:.0%}"
+ if tool.progress.message:
+ result["progress_message"] = tool.progress.message
+ if tool.result:
+ result["result"] = tool.result
+ if tool.error:
+ result["error"] = tool.error
+ return result
+
+ running = [tool for tool in tool_manager.get_running_tools() if tool.tool_name not in _SYSTEM_TOOL_NAMES]
+ if not running:
+ return {"status": "idle", "message": "No tools running in the background."}
+ tools_info = []
+ for tool in running:
+ tool_info: dict[str, Any] = {
+ "tool_id": tool.tool_id,
+ "name": tool.tool_name,
+ "status": tool.status.value,
+ "elapsed_seconds": round(time.monotonic() - tool.started_at, 1),
+ }
+ if tool.progress is not None:
+ tool_info["progress_percent"] = f"{tool.progress.progress:.0%}"
+ if tool.progress.message:
+ tool_info["progress_message"] = tool.progress.message
+ tools_info.append(tool_info)
+ return {
+ "status": "running",
+ "count": len(tools_info),
+ "message": f"{len(tools_info)} tool(s) running in the background.",
+ "tools": tools_info,
+ }
+
+ if tool_name == SystemTool.TASK_CANCEL.value:
+ if not isinstance(tool_id, str) or not tool_id:
+ return {"error": "Tool ID is required."}
+ tool = tool_manager.get_tool(tool_id)
+ if tool is None:
+ return {"error": f"Tool {tool_id} not found."}
+ if tool.status != ToolState.RUNNING:
+ return {
+ "status": tool.status.value,
+ "message": f"Tool '{tool.tool_name}' is not running (status: {tool.status.value}).",
+ "tool_id": tool_id,
+ }
+ if await tool_manager.cancel_tool(tool_id):
+ return {
+ "status": "cancelled",
+ "message": f"Tool '{tool.tool_name}' has been cancelled.",
+ "tool_id": tool_id,
+ "tool_name": tool.tool_name,
+ }
+ return {"error": f"Could not cancel tool {tool_id}. It may have already completed."}
+
+ return {"error": f"unknown system tool: {tool_name}"}
+
+
class ToolProgress(BaseModel):
"""Progress of a background tool."""
@@ -50,13 +141,16 @@ class ToolCallRoutine(BaseModel):
"""the dependencies for the tool call"""
deps: "ToolDependencies"
+ """the selected local or remote tool transport"""
+ transport: ToolTransport | None = None
+
async def __call__(self, tool_manager: BackgroundToolManager) -> Any:
"""Execute the stored callable with its arguments."""
+ arguments = _tool_arguments(self.args_json_str)
if self.tool_name in _SYSTEM_TOOL_NAMES:
- # For safety purposes, we only allow system tools to be called with the tool manager
- return await dispatch_tool_call_with_manager(
- tool_name=self.tool_name, args_json=self.args_json_str, deps=self.deps, tool_manager=tool_manager
- )
+ return await _call_system_tool(self.tool_name, arguments, tool_manager)
+ if self.transport is not None:
+ return await self.transport.call_tool(self.tool_name, arguments)
return await dispatch_tool_call(tool_name=self.tool_name, args_json=self.args_json_str, deps=self.deps)
@@ -213,6 +307,7 @@ async def _run_tool(
result = {"error": "Tool cancelled"}
bg_tool.completed_at = time.monotonic()
+ bg_tool.result = result
error = result.get("error")
if error is not None:
@@ -225,7 +320,6 @@ async def _run_tool(
bg_tool.error = result["error"]
else:
- bg_tool.result = result
bg_tool.status = ToolState.COMPLETED
logger.debug(f"Background tool completed: {bg_tool.tool_name} (id={bg_tool.id})")
diff --git a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tools/camera.py b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tools/camera.py
index 8fd9999d..7fdb330e 100644
--- a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tools/camera.py
+++ b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tools/camera.py
@@ -25,6 +25,7 @@ class Camera(Tool):
},
},
"required": ["question"],
+ "additionalProperties": False,
}
async def __call__(self, deps: ToolDependencies, **kwargs: Any) -> Dict[str, Any]:
@@ -46,7 +47,9 @@ async def __call__(self, deps: ToolDependencies, **kwargs: Any) -> Dict[str, Any
logger.error("Camera worker not available")
return {"error": "Camera worker not available"}
- # Use vision manager for processing if available
+ # Use the explicitly enabled local vision manager before the legacy
+ # active-conversation path. Main does not install both local vision and
+ # routed cloud vision at the same time.
if deps.vision_manager is not None:
vision_result = await asyncio.to_thread(
deps.vision_manager.processor.process_image,
@@ -67,4 +70,12 @@ async def __call__(self, deps: ToolDependencies, **kwargs: Any) -> Dict[str, Any
raise RuntimeError("Failed to encode frame as JPEG")
b64_encoded = base64.b64encode(buffer.tobytes()).decode("utf-8")
- return {"b64_im": b64_encoded}
+
+ if deps.vision_router is not None:
+ analysis = await deps.vision_router.analyze_images(
+ images_base64=[b64_encoded],
+ question=image_query,
+ )
+ return analysis.as_tool_result()
+
+ return {"b64_im": b64_encoded, "question": image_query}
diff --git a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tools/core_tools.py b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tools/core_tools.py
index fea84ffb..847949ee 100644
--- a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tools/core_tools.py
+++ b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tools/core_tools.py
@@ -6,9 +6,9 @@
import logging
import importlib
from typing import TYPE_CHECKING, Any, Dict, List
+from pathlib import Path
from dataclasses import dataclass
-from reachy_mini import ReachyMini
from reachy_mini_conversation_app.config import LOCKED_PROFILE, DEFAULT_PROFILES_DIRECTORY
from reachy_mini_conversation_app.tools.tool_constants import SystemTool
@@ -50,12 +50,26 @@ def get_concrete_subclasses(base: type[Tool]) -> List[type[Tool]]:
class ToolDependencies:
"""External dependencies injected into tools."""
- reachy_mini: ReachyMini
- movement_manager: Any
+ reachy_mini: Any | None = None
+ movement_manager: Any | None = None
camera_worker: Any | None = None
vision_manager: Any | None = None
+ vision_router: Any | None = None
head_wobbler: Any | None = None
motion_duration_s: float = 1.0
+ capture_directory: Path | None = None
+
+ def require_reachy_mini(self) -> Any:
+ """Return the local robot or fail clearly in a hardware-free process."""
+ if self.reachy_mini is None:
+ raise RuntimeError("This tool requires a local ReachyMini connection")
+ return self.reachy_mini
+
+ def require_movement_manager(self) -> Any:
+ """Return the local movement manager or fail clearly in remote mode."""
+ if self.movement_manager is None:
+ raise RuntimeError("This tool requires a local movement manager")
+ return self.movement_manager
class Tool(abc.ABC):
@@ -167,15 +181,21 @@ def _initialize_tools() -> None:
_TOOLS_INITIALIZED = True
-_initialize_tools()
-
-
def get_tool_specs(exclusion_list: list[str] | None = None) -> list[Dict[str, Any]]:
"""Get tool specs, optionally excluding some tools."""
+ _initialize_tools()
exclusion_list = exclusion_list or []
return [spec for spec in ALL_TOOL_SPECS if spec.get("name") not in exclusion_list]
+def get_tool_specs_for_dependencies(deps: ToolDependencies) -> list[Dict[str, Any]]:
+ """Return only tools whose runtime dependencies are available."""
+ exclusions: list[str] = []
+ if deps.camera_worker is None:
+ exclusions.extend(("camera", "head_tracking", "scan_scene"))
+ return get_tool_specs(exclusions)
+
+
# Dispatcher
def _safe_load_obj(args_json: str) -> Dict[str, Any]:
try:
@@ -203,6 +223,7 @@ async def _dispatch_tool_call(tool_name: str, args: Dict[str, Any], deps: ToolDe
async def dispatch_tool_call(tool_name: str, args_json: str, deps: ToolDependencies) -> Dict[str, Any]:
"""Dispatch a tool call by name with JSON args and dependencies."""
+ _initialize_tools()
return await _dispatch_tool_call(tool_name, _safe_load_obj(args_json), deps)
@@ -210,6 +231,7 @@ async def dispatch_tool_call_with_manager(
tool_name: str, args_json: str, deps: ToolDependencies, tool_manager: "BackgroundToolManager"
) -> Dict[str, Any]:
"""Dispatch a tool call, injecting a BackgroundToolManager into the args."""
+ _initialize_tools()
args = _safe_load_obj(args_json)
args["tool_manager"] = tool_manager
return await _dispatch_tool_call(tool_name, args, deps)
diff --git a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tools/dance.py b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tools/dance.py
index 833cd552..c67229ee 100644
--- a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tools/dance.py
+++ b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tools/dance.py
@@ -78,7 +78,7 @@ async def __call__(self, deps: ToolDependencies, **kwargs: Any) -> Dict[str, Any
return {"error": f"Unknown dance move '{move_name}'. Available: {list(AVAILABLE_MOVES.keys())}"}
# Add dance moves to queue
- movement_manager = deps.movement_manager
+ movement_manager = deps.require_movement_manager()
for _ in range(repeat):
dance_move = DanceQueueMove(move_name)
movement_manager.queue_move(dance_move)
diff --git a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tools/move_head.py b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tools/move_head.py
index 83bc1455..e129ca95 100644
--- a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tools/move_head.py
+++ b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tools/move_head.py
@@ -12,19 +12,28 @@
class MoveHead(Tool):
- """Move head in a given direction."""
+ """Move the head through one or more ordered directions."""
name = "move_head"
- description = "Move your head in a given direction: left, right, up, down or front."
- parameters_schema = {
+ description = (
+ "Move your head through an ordered list of directions. Include every direction the user requests, "
+ "in the same order. Valid directions are left, right, up, down, and front."
+ )
+ parameters_schema: Dict[str, Any] = {
"type": "object",
"properties": {
- "direction": {
- "type": "string",
- "enum": ["left", "right", "up", "down", "front"],
+ "directions": {
+ "type": "array",
+ "description": ("Ordered directions to perform. For 'look up and then right', use ['up', 'right']."),
+ "items": {
+ "type": "string",
+ "enum": ["left", "right", "up", "down", "front"],
+ },
+ "minItems": 1,
+ "maxItems": 8,
},
},
- "required": ["direction"],
+ "required": ["directions"],
}
# mapping: direction -> args for create_head_pose
@@ -37,42 +46,63 @@ class MoveHead(Tool):
}
async def __call__(self, deps: ToolDependencies, **kwargs: Any) -> Dict[str, Any]:
- """Move head in a given direction."""
- direction_raw = kwargs.get("direction")
- if not isinstance(direction_raw, str):
- return {"error": "direction must be a string"}
- direction: Direction = direction_raw # type: ignore[assignment]
- logger.info("Tool call: move_head direction=%s", direction)
+ """Queue each requested head direction in order."""
+ directions_raw = kwargs.get("directions")
- deltas = self.DELTAS.get(direction, self.DELTAS["front"])
- target = create_head_pose(*deltas, degrees=True)
+ # Preserve compatibility with older callers that send {"direction": "up"}.
+ if directions_raw is None and "direction" in kwargs:
+ directions_raw = [kwargs.get("direction")]
+
+ if not isinstance(directions_raw, list) or not directions_raw:
+ return {"error": "directions must be a non-empty list"}
+ if len(directions_raw) > 8:
+ return {"error": "directions supports at most 8 movements"}
+
+ directions: list[Direction] = []
+ for direction_raw in directions_raw:
+ if not isinstance(direction_raw, str) or direction_raw not in self.DELTAS:
+ return {"error": (f"invalid direction {direction_raw!r}; expected one of {list(self.DELTAS.keys())}")}
+ directions.append(direction_raw) # type: ignore[arg-type]
+
+ logger.info("Tool call: move_head directions=%s", directions)
# Use new movement manager
try:
- movement_manager = deps.movement_manager
+ movement_manager = deps.require_movement_manager()
+ reachy_mini = deps.require_reachy_mini()
# Get current state for interpolation
- current_head_pose = deps.reachy_mini.get_current_head_pose()
- head_joints, current_antennas = deps.reachy_mini.get_current_joint_positions()
-
- # Create goto move
- goto_move = GotoQueueMove(
- target_head_pose=target,
- start_head_pose=current_head_pose,
- target_antennas=(0, 0), # Reset antennas to default
- start_antennas=(
- current_antennas[0],
- current_antennas[1],
- ), # Skip body_yaw
- target_body_yaw=0, # Reset body yaw
- start_body_yaw=head_joints[0], # body_yaw is first in head joint positions
- duration=deps.motion_duration_s,
- )
-
- movement_manager.queue_move(goto_move)
- movement_manager.set_moving_state(deps.motion_duration_s)
-
- return {"status": f"looking {direction}"}
+ current_head_pose = reachy_mini.get_current_head_pose()
+ head_joints, current_antennas = reachy_mini.get_current_joint_positions()
+
+ start_head_pose = current_head_pose
+ start_antennas = (current_antennas[0], current_antennas[1])
+ start_body_yaw = head_joints[0]
+
+ for direction in directions:
+ target_head_pose = create_head_pose(*self.DELTAS[direction], degrees=True)
+ movement_manager.queue_move(
+ GotoQueueMove(
+ target_head_pose=target_head_pose,
+ start_head_pose=start_head_pose,
+ target_antennas=(0, 0),
+ start_antennas=start_antennas,
+ target_body_yaw=0,
+ start_body_yaw=start_body_yaw,
+ duration=deps.motion_duration_s,
+ )
+ )
+ start_head_pose = target_head_pose
+ start_antennas = (0, 0)
+ start_body_yaw = 0
+
+ movement_manager.set_moving_state(deps.motion_duration_s * len(directions))
+
+ return {
+ "status": "queued",
+ "directions": directions,
+ "total_duration_seconds": deps.motion_duration_s * len(directions),
+ }
except Exception as e:
logger.error("move_head failed")
diff --git a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tools/play_emotion.py b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tools/play_emotion.py
index 8694f2ea..e425b68f 100644
--- a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tools/play_emotion.py
+++ b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tools/play_emotion.py
@@ -99,7 +99,7 @@ async def __call__(self, deps: ToolDependencies, **kwargs: Any) -> Dict[str, Any
return {"error": f"Unknown emotion '{emotion_name}'. Available: {emotion_names}"}
# Add emotion to queue
- movement_manager = deps.movement_manager
+ movement_manager = deps.require_movement_manager()
emotion_move = EmotionQueueMove(emotion_name, recorded_moves)
movement_manager.queue_move(emotion_move)
diff --git a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tools/stop_dance.py b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tools/stop_dance.py
index ab14e84a..cc3980a3 100644
--- a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tools/stop_dance.py
+++ b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tools/stop_dance.py
@@ -26,6 +26,6 @@ class StopDance(Tool):
async def __call__(self, deps: ToolDependencies, **kwargs: Any) -> Dict[str, Any]:
"""Stop the current dance move."""
logger.info("Tool call: stop_dance")
- movement_manager = deps.movement_manager
+ movement_manager = deps.require_movement_manager()
movement_manager.clear_move_queue()
return {"status": "stopped dance and cleared queue"}
diff --git a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tools/stop_emotion.py b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tools/stop_emotion.py
index b5d2323f..0e323722 100644
--- a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tools/stop_emotion.py
+++ b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/tools/stop_emotion.py
@@ -26,6 +26,6 @@ class StopEmotion(Tool):
async def __call__(self, deps: ToolDependencies, **kwargs: Any) -> Dict[str, Any]:
"""Stop the current emotion."""
logger.info("Tool call: stop_emotion")
- movement_manager = deps.movement_manager
+ movement_manager = deps.require_movement_manager()
movement_manager.clear_move_queue()
return {"status": "stopped emotion and cleared queue"}
diff --git a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/utils.py b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/utils.py
index 0cdea7b0..e0b9247c 100644
--- a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/utils.py
+++ b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/utils.py
@@ -1,10 +1,13 @@
+from __future__ import annotations
import logging
import argparse
import warnings
-from typing import Any, Tuple, Optional
+from typing import TYPE_CHECKING, Any, Tuple, Optional
-from reachy_mini import ReachyMini
-from reachy_mini_conversation_app.camera_worker import CameraWorker
+
+if TYPE_CHECKING:
+ from reachy_mini import ReachyMini
+ from reachy_mini_conversation_app.camera_worker import CameraWorker
def parse_args() -> Tuple[argparse.Namespace, list]:
@@ -25,6 +28,18 @@ def parse_args() -> Tuple[argparse.Namespace, list]:
)
parser.add_argument("--gradio", default=False, action="store_true", help="Open gradio interface")
parser.add_argument("--debug", default=False, action="store_true", help="Enable debug logging")
+ parser.add_argument(
+ "--model-logs",
+ default=False,
+ action="store_true",
+ help="Log only focused model selection, sanitized requests, and token usage at INFO level",
+ )
+ parser.add_argument(
+ "--tool-transport",
+ choices=["local", "rest"],
+ default=None,
+ help="Execute Reachy tools locally or through the daemon REST API (default: REACHY_TOOL_TRANSPORT)",
+ )
parser.add_argument(
"--robot-name",
type=str,
@@ -45,6 +60,8 @@ def handle_vision_stuff(args: argparse.Namespace, current_robot: ReachyMini) ->
vision_manager = None
if not args.no_camera:
+ from reachy_mini_conversation_app.camera_worker import CameraWorker
+
# Initialize head tracker if specified
if args.head_tracker is not None:
if args.head_tracker == "yolo":
diff --git a/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/vision_router.py b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/vision_router.py
new file mode 100644
index 00000000..8da4bad9
--- /dev/null
+++ b/projects/reachy-mini-openshell/src/reachy_mini_conversation_app/vision_router.py
@@ -0,0 +1,176 @@
+"""Single-model routing for camera and scene-scan image analysis."""
+
+from __future__ import annotations
+import logging
+from typing import Any, Iterable
+from dataclasses import dataclass
+
+from openai import AsyncOpenAI
+
+from reachy_mini_conversation_app.config import config, vision_api_key
+
+
+logger = logging.getLogger(__name__)
+
+MAX_VISION_IMAGES = 9
+
+
+@dataclass(frozen=True)
+class VisionAnalysis:
+ """Text and routing metadata returned by an approved vision request."""
+
+ description: str
+ selected_model: str
+ response_id: str | None
+ usage: Any | None
+
+ def as_tool_result(self) -> dict[str, Any]:
+ """Return a JSON-safe tool result without including the source image."""
+ result: dict[str, Any] = {
+ "status": "image_analyzed",
+ "image_description": self.description,
+ "selected_model": self.selected_model,
+ }
+ if self.response_id:
+ result["response_id"] = self.response_id
+ if self.usage is not None:
+ result["usage"] = _jsonable(self.usage)
+ return result
+
+
+def _jsonable(value: Any) -> Any:
+ """Convert OpenAI SDK response values to plain JSON-compatible objects."""
+ model_dump = getattr(value, "model_dump", None)
+ if callable(model_dump):
+ try:
+ return model_dump(mode="json")
+ except TypeError:
+ return model_dump()
+ return value
+
+
+class VisionRouter:
+ """Route one or more images to the single approved Responses model."""
+
+ def __init__(
+ self,
+ *,
+ client: Any,
+ default_model: str,
+ allowed_models: Iterable[str],
+ ) -> None:
+ """Initialize and validate the routing policy."""
+ self.client = client
+ self.default_model = default_model.strip()
+ self.allowed_models = tuple(dict.fromkeys(model.strip() for model in allowed_models if model.strip()))
+
+ if len(self.allowed_models) != 1:
+ raise ValueError("VISION_ALLOWED_MODELS must contain exactly one model")
+ if not self.default_model:
+ raise ValueError("VISION_DEFAULT_MODEL must be configured")
+ if self.default_model != self.allowed_models[0]:
+ raise ValueError(
+ f"VISION_DEFAULT_MODEL={self.default_model!r} must equal the only VISION_ALLOWED_MODELS entry"
+ )
+
+ async def analyze_images(
+ self,
+ *,
+ images_base64: list[str],
+ question: str,
+ frame_timestamps: list[float] | None = None,
+ ) -> VisionAnalysis:
+ """Analyze one or more chronological images with the configured default model."""
+ if not 1 <= len(images_base64) <= MAX_VISION_IMAGES:
+ raise ValueError(f"Vision analysis requires between 1 and {MAX_VISION_IMAGES} images")
+ if not all(isinstance(image, str) and image for image in images_base64):
+ raise ValueError("Vision images must be non-empty Base64 strings")
+ if frame_timestamps is not None and len(frame_timestamps) != len(images_base64):
+ raise ValueError("Frame timestamp count must match image count")
+
+ return await self._analyze(
+ images_base64=images_base64,
+ question=question,
+ frame_timestamps=frame_timestamps,
+ )
+
+ async def _analyze(
+ self,
+ *,
+ images_base64: list[str],
+ question: str,
+ frame_timestamps: list[float] | None,
+ ) -> VisionAnalysis:
+ """Send one Responses request containing text followed by ordered images."""
+ prompt = question
+ if len(images_base64) > 1:
+ prompt = (
+ "These are chronological frames sampled across one Reachy scene sweep. "
+ f"Frame timestamps in seconds: {frame_timestamps or []}. "
+ "Combine evidence across every frame, deduplicate people and objects visible more than once, "
+ "and describe only details supported by the images. "
+ f"User question: {question}"
+ )
+ logger.info(
+ "VISION request selected_model=%s image_count=%d question=%s",
+ self.default_model,
+ len(images_base64),
+ question[:160],
+ )
+
+ response = await self.client.responses.create(
+ model=self.default_model,
+ input=[
+ {
+ "role": "user",
+ "content": [
+ {"type": "input_text", "text": prompt},
+ *[
+ {
+ "type": "input_image",
+ "image_url": f"data:image/jpeg;base64,{image_base64}",
+ }
+ for image_base64 in images_base64
+ ],
+ ],
+ }
+ ],
+ )
+
+ description = (getattr(response, "output_text", None) or "").strip()
+ if not description:
+ raise RuntimeError(f"Vision model {self.default_model!r} returned no text")
+
+ response_id = getattr(response, "id", None)
+ usage = getattr(response, "usage", None)
+ logger.info(
+ "VISION response selected_model=%s response_id=%s usage=%s",
+ self.default_model,
+ response_id,
+ _jsonable(usage),
+ )
+ return VisionAnalysis(
+ description=description,
+ selected_model=self.default_model,
+ response_id=response_id if isinstance(response_id, str) else None,
+ usage=usage,
+ )
+
+
+def build_vision_router() -> VisionRouter | None:
+ """Build the configured router, or preserve the legacy path when no key exists."""
+ api_key = vision_api_key()
+ if not api_key:
+ logger.warning(
+ "Routed camera vision is disabled because neither VISION_API_KEY nor OPENAI_API_KEY is configured."
+ )
+ return None
+
+ if not config.VISION_BASE_URL:
+ raise ValueError("VISION_BASE_URL must be configured when routed camera vision is enabled")
+
+ return VisionRouter(
+ client=AsyncOpenAI(api_key=api_key, base_url=config.VISION_BASE_URL),
+ default_model=config.VISION_DEFAULT_MODEL or "",
+ allowed_models=config.VISION_ALLOWED_MODELS,
+ )
diff --git a/projects/reachy-mini-openshell/tests/test_camera_policy.py b/projects/reachy-mini-openshell/tests/test_camera_policy.py
new file mode 100644
index 00000000..6cf85ddd
--- /dev/null
+++ b/projects/reachy-mini-openshell/tests/test_camera_policy.py
@@ -0,0 +1,51 @@
+"""Regression tests for the two explicit camera policy modes."""
+
+# ruff: noqa: D103
+
+from typing import Any
+from pathlib import Path
+
+import yaml
+
+
+POLICY_DIRECTORY = Path(__file__).parents[1] / "openshell"
+
+
+def _load_policy(filename: str) -> dict[str, Any]:
+ with (POLICY_DIRECTORY / filename).open(encoding="utf-8") as stream:
+ return yaml.safe_load(stream)
+
+
+def _allowed_requests(policy: dict[str, Any]) -> set[tuple[str, int, str, str]]:
+ requests: set[tuple[str, int, str, str]] = set()
+ for network_policy in policy["network_policies"].values():
+ for endpoint in network_policy["endpoints"]:
+ for rule in endpoint.get("rules", []):
+ allowed = rule["allow"]
+ requests.add(
+ (
+ endpoint["host"],
+ endpoint["port"],
+ allowed["method"],
+ allowed["path"],
+ )
+ )
+ return requests
+
+
+def test_motion_disabled_policy_blocks_camera_and_motion_start() -> None:
+ requests = _allowed_requests(_load_policy("policy-motion-disabled.yaml"))
+
+ assert not any(port == 8042 for _, port, _, _ in requests)
+ assert not any(path == "/api/move/goto" for _, _, _, path in requests)
+ assert ("host.openshell.internal", 8000, "POST", "/api/move/stop") in requests
+
+
+def test_camera_enabled_policy_allows_only_fixed_capture_and_still_blocks_motion_start() -> None:
+ requests = _allowed_requests(_load_policy("policy-camera-enabled-motion-disabled.yaml"))
+ camera_requests = {request for request in requests if request[1] == 8042}
+
+ assert camera_requests == {
+ ("host.openshell.internal", 8042, "POST", "/camera/capture")
+ }
+ assert not any(path == "/api/move/goto" for _, _, _, path in requests)
diff --git a/projects/reachy-mini-openshell/tests/test_chat_completions_media.py b/projects/reachy-mini-openshell/tests/test_chat_completions_media.py
new file mode 100644
index 00000000..bf8489d0
--- /dev/null
+++ b/projects/reachy-mini-openshell/tests/test_chat_completions_media.py
@@ -0,0 +1,194 @@
+"""Tests for camera media passed through the Chat Completions tool loop."""
+
+import json
+from typing import Any, cast
+from unittest.mock import MagicMock
+
+import pytest
+
+import reachy_mini_conversation_app.chat_completions as chat_mod
+from reachy_mini_conversation_app.tools.core_tools import ToolDependencies
+from reachy_mini_conversation_app.media_result_processor import ProcessedToolResult
+
+
+@pytest.mark.asyncio
+async def test_scene_scan_images_are_sent_as_vision_content(monkeypatch: Any) -> None:
+ """Local-STT Chat Completions should analyze frames without embedding them in tool JSON."""
+ create_calls: list[dict[str, Any]] = []
+
+ class FakeFunction:
+ name = "scan_scene"
+ arguments = '{"question":"What did you see?"}'
+
+ class FakeToolCall:
+ id = "call_scan"
+ type = "function"
+ function = FakeFunction()
+
+ def model_dump(self) -> dict[str, Any]:
+ return {
+ "id": self.id,
+ "type": self.type,
+ "function": {
+ "name": self.function.name,
+ "arguments": self.function.arguments,
+ },
+ }
+
+ class FakeToolMessage:
+ content = ""
+ tool_calls = [FakeToolCall()]
+
+ class FakeFinalMessage:
+ content = "I saw a desk and a person."
+ tool_calls: list[Any] = []
+
+ class FakeChoice:
+ def __init__(self, message: Any) -> None:
+ self.message = message
+
+ class FakeCompletion:
+ def __init__(self, message: Any) -> None:
+ self.choices = [FakeChoice(message)]
+
+ class FakeCompletions:
+ async def create(self, **kwargs: Any) -> FakeCompletion:
+ create_calls.append(kwargs)
+ if len(create_calls) == 1:
+ return FakeCompletion(FakeToolMessage())
+ return FakeCompletion(FakeFinalMessage())
+
+ class FakeChat:
+ completions = FakeCompletions()
+
+ class FakeClient:
+ chat = FakeChat()
+
+ async def fake_dispatch(*_args: Any, **_kwargs: Any) -> dict[str, Any]:
+ return {
+ "status": "scene_scan_complete",
+ "question": "What did you see?",
+ "frame_timestamps_seconds": [0.5, 7.0],
+ "b64_images": ["first-jpeg", "second-jpeg"],
+ }
+
+ monkeypatch.setattr(chat_mod, "dispatch_tool_call_with_manager", fake_dispatch)
+ runner = chat_mod.ChatCompletionRunner(
+ client=FakeClient(),
+ deps=ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock()),
+ tool_manager=MagicMock(),
+ model_name="vision-chat-model",
+ base_url="https://example.test/v1",
+ )
+
+ chatbot_messages = await runner.send_text_message("Scan the room and tell me what you saw.")
+
+ followup_messages = create_calls[1]["messages"]
+ tool_message = next(message for message in followup_messages if message["role"] == "tool")
+ vision_message = next(
+ message for message in followup_messages if message["role"] == "user" and isinstance(message["content"], list)
+ )
+
+ assert "b64_images" not in json.loads(tool_message["content"])
+ assert vision_message["content"][0]["type"] == "text"
+ assert "chronological" in vision_message["content"][0]["text"]
+ assert vision_message["content"][1:] == [
+ {
+ "type": "image_url",
+ "image_url": {"url": "data:image/jpeg;base64,first-jpeg"},
+ },
+ {
+ "type": "image_url",
+ "image_url": {"url": "data:image/jpeg;base64,second-jpeg"},
+ },
+ ]
+ assert "first-jpeg" not in chatbot_messages[1]["content"]
+ assert chatbot_messages[-1]["content"] == "I saw a desk and a person."
+
+
+@pytest.mark.asyncio
+async def test_media_processor_keeps_raw_camera_image_out_of_chat_completions(monkeypatch: Any) -> None:
+ """Local STT should use routed vision text instead of sending the image to chat."""
+ create_calls: list[dict[str, Any]] = []
+
+ class FakeFunction:
+ name = "camera"
+ arguments = '{"question":"What am I doing?"}'
+
+ class FakeToolCall:
+ id = "call_camera"
+ type = "function"
+ function = FakeFunction()
+
+ def model_dump(self) -> dict[str, Any]:
+ return {
+ "id": self.id,
+ "type": self.type,
+ "function": {
+ "name": self.function.name,
+ "arguments": self.function.arguments,
+ },
+ }
+
+ class FakeMessage:
+ def __init__(self, content: str, tool_calls: list[Any]) -> None:
+ self.content = content
+ self.tool_calls = tool_calls
+
+ class FakeChoice:
+ def __init__(self, message: FakeMessage) -> None:
+ self.message = message
+
+ class FakeCompletion:
+ def __init__(self, message: FakeMessage) -> None:
+ self.choices = [FakeChoice(message)]
+
+ class FakeCompletions:
+ async def create(self, **kwargs: Any) -> FakeCompletion:
+ create_calls.append(kwargs)
+ if len(create_calls) == 1:
+ return FakeCompletion(FakeMessage("", [FakeToolCall()]))
+ return FakeCompletion(FakeMessage("You are waving.", []))
+
+ class FakeChat:
+ completions = FakeCompletions()
+
+ class FakeClient:
+ chat = FakeChat()
+
+ class FakeProcessor:
+ async def process(self, _tool_name: str, result: dict[str, Any]) -> ProcessedToolResult:
+ assert result["b64_im"] == "private-image"
+ return ProcessedToolResult(
+ model_payload={
+ "status": "image_analyzed",
+ "question": result["question"],
+ "image_description": "The person is waving.",
+ "selected_model": "approved-vision-model",
+ }
+ )
+
+ async def fake_dispatch(*_args: Any, **_kwargs: Any) -> dict[str, Any]:
+ return {
+ "question": "What am I doing?",
+ "b64_im": "private-image",
+ }
+
+ monkeypatch.setattr(chat_mod, "dispatch_tool_call_with_manager", fake_dispatch)
+ runner = chat_mod.ChatCompletionRunner(
+ client=FakeClient(),
+ deps=ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock()),
+ tool_manager=MagicMock(),
+ model_name="chat-model",
+ base_url="https://example.test/v1",
+ media_result_processor=cast(Any, FakeProcessor()),
+ )
+
+ chatbot_messages = await runner.send_text_message("Use the camera.")
+
+ followup_messages = create_calls[1]["messages"]
+ serialized_messages = json.dumps(followup_messages)
+ assert "private-image" not in serialized_messages
+ assert "data:image" not in serialized_messages
+ assert "The person is waving." in serialized_messages
+ assert chatbot_messages[-1]["content"] == "You are waving."
diff --git a/projects/reachy-mini-openshell/tests/test_config.py b/projects/reachy-mini-openshell/tests/test_config.py
index f01d89f8..be194693 100644
--- a/projects/reachy-mini-openshell/tests/test_config.py
+++ b/projects/reachy-mini-openshell/tests/test_config.py
@@ -1,4 +1,7 @@
import os
+import sys
+import json
+import subprocess
from typing import Any
from pathlib import Path
@@ -14,6 +17,18 @@
"OPENAI_REALTIME_BASE_URL",
"OPENAI_REALTIME_MODEL",
"OPENAI_REALTIME_VOICE",
+ "VISION_API_KEY",
+ "VISION_BASE_URL",
+ "VISION_DEFAULT_MODEL",
+ "VISION_ALLOWED_MODELS",
+ "REACHY_TOOL_TRANSPORT",
+ "REACHY_REST_BASE_URL",
+ "REACHY_CAMERA_BASE_URL",
+ "REACHY_REST_TIMEOUT_SECONDS",
+ "REACHY_MOTION_DURATION_SECONDS",
+ "REACHY_MOTION_POLL_INTERVAL_SECONDS",
+ "REACHY_MOTION_COMPLETION_TIMEOUT_SECONDS",
+ "REQUIRE_ROUTED_VISION",
"HF_REALTIME_CONNECTION_MODE",
"HF_REALTIME_SESSION_URL",
"HF_REALTIME_WS_URL",
@@ -75,6 +90,13 @@ def test_documented_env_template_validates_with_exported_system_keys(monkeypatch
try:
assert not list(PROJECT_ROOT.glob(".env.*.example"))
assert raw_values["BACKEND_PROVIDER"] == config_mod.BACKEND_OPENAI_REALTIME
+ assert raw_values["VISION_DEFAULT_MODEL"] == "gpt-5.4-mini"
+ assert raw_values["VISION_ALLOWED_MODELS"] == "gpt-5.4-mini"
+ assert raw_values["REACHY_TOOL_TRANSPORT"] == "rest"
+ assert raw_values["REACHY_REST_BASE_URL"] == "http://127.0.0.1:8000"
+ assert "REACHY_CAMERA_BASE_URL" not in raw_values
+ assert raw_values["REACHY_MOTION_DURATION_SECONDS"] == "1"
+ assert raw_values["REQUIRE_ROUTED_VISION"] == "0"
assert "OPENAI_REALTIME_API_KEY" not in raw_values
assert "OPENAI_API_KEY" not in raw_values
@@ -96,6 +118,122 @@ def test_documented_env_template_validates_with_exported_system_keys(monkeypatch
_restore_config_snapshot(snapshot)
+def test_sandbox_process_environment_loads_when_dotenv_is_disabled() -> None:
+ """OpenShell --env values should configure the app without a bundled dotenv file."""
+ env: dict[str, str] = dict(os.environ)
+ env.update(
+ {
+ "REACHY_MINI_SKIP_DOTENV": "1",
+ "BACKEND_PROVIDER": "openai_realtime",
+ "OPENAI_API_KEY": "sandbox-provider-key",
+ "OPENAI_REALTIME_BASE_URL": "https://api.openai.com/v1",
+ "OPENAI_REALTIME_MODEL": "gpt-realtime-2",
+ "OPENAI_REALTIME_VOICE": "cedar",
+ "REACHY_TOOL_TRANSPORT": "rest",
+ "REACHY_REST_BASE_URL": "http://host.openshell.internal:8000",
+ "REACHY_CAMERA_BASE_URL": "http://host.openshell.internal:8042",
+ "REACHY_REST_TIMEOUT_SECONDS": "4",
+ "REACHY_MOTION_DURATION_SECONDS": "0.75",
+ "REACHY_MOTION_POLL_INTERVAL_SECONDS": "0.2",
+ "REACHY_MOTION_COMPLETION_TIMEOUT_SECONDS": "8",
+ }
+ )
+ env.pop("REACHY_MINI_DOTENV_PATH", None)
+ code = """
+import json
+from reachy_mini_conversation_app.config import config
+print(json.dumps({
+ "backend": config.BACKEND_PROVIDER,
+ "realtime_base_url": config.OPENAI_REALTIME_BASE_URL,
+ "realtime_model": config.OPENAI_REALTIME_MODEL,
+ "tool_transport": config.REACHY_TOOL_TRANSPORT,
+ "rest_base_url": config.REACHY_REST_BASE_URL,
+ "camera_base_url": config.REACHY_CAMERA_BASE_URL,
+ "rest_timeout": config.REACHY_REST_TIMEOUT_SECONDS,
+ "motion_duration": config.REACHY_MOTION_DURATION_SECONDS,
+ "motion_poll": config.REACHY_MOTION_POLL_INTERVAL_SECONDS,
+ "motion_completion_timeout": config.REACHY_MOTION_COMPLETION_TIMEOUT_SECONDS,
+}))
+"""
+
+ completed = subprocess.run(
+ [sys.executable, "-c", code],
+ cwd=PROJECT_ROOT,
+ env=env,
+ check=True,
+ capture_output=True,
+ text=True,
+ )
+ loaded = json.loads(completed.stdout)
+
+ assert loaded == {
+ "backend": "openai_realtime",
+ "realtime_base_url": "https://api.openai.com/v1",
+ "realtime_model": "gpt-realtime-2",
+ "tool_transport": "rest",
+ "rest_base_url": "http://host.openshell.internal:8000",
+ "camera_base_url": "http://host.openshell.internal:8042",
+ "rest_timeout": 4.0,
+ "motion_duration": 0.75,
+ "motion_poll": 0.2,
+ "motion_completion_timeout": 8.0,
+ }
+
+
+def test_apply_config_values_parses_vision_route_and_key_fallback(monkeypatch: Any) -> None:
+ """Vision configuration should parse its single model and reuse the standard OpenAI key."""
+ snapshot = _config_snapshot()
+ monkeypatch.setitem(config_mod._ORIGINAL_PROCESS_ENV, "OPENAI_API_KEY", "global-openai-key")
+
+ try:
+ config_mod.apply_config_values(
+ {
+ "VISION_BASE_URL": "https://vision.example.test/v1",
+ "VISION_DEFAULT_MODEL": "gpt-5.4-mini",
+ "VISION_ALLOWED_MODELS": "gpt-5.4-mini, gpt-5.4-mini",
+ },
+ inherit_current=False,
+ )
+
+ assert config_mod.config.VISION_BASE_URL == "https://vision.example.test/v1"
+ assert config_mod.config.VISION_DEFAULT_MODEL == "gpt-5.4-mini"
+ assert config_mod.config.VISION_ALLOWED_MODELS == ("gpt-5.4-mini",)
+ assert config_mod.vision_api_key() == "global-openai-key"
+ finally:
+ _restore_config_snapshot(snapshot)
+
+
+def test_apply_config_values_parses_rest_conversation_transport(monkeypatch: Any) -> None:
+ """REST mode should load its endpoint and bounded motion timing settings."""
+ snapshot = _config_snapshot()
+ monkeypatch.setattr(config_mod, "_dotenv_loaded_keys", set())
+ monkeypatch.setattr(config_mod, "_dotenv_values", {})
+
+ try:
+ config_mod.apply_config_values(
+ {
+ "REACHY_TOOL_TRANSPORT": "rest",
+ "REACHY_REST_BASE_URL": "http://127.0.0.1:8000",
+ "REACHY_CAMERA_BASE_URL": "http://127.0.0.1:8042",
+ "REACHY_REST_TIMEOUT_SECONDS": "3",
+ "REACHY_MOTION_DURATION_SECONDS": "0.5",
+ "REACHY_MOTION_POLL_INTERVAL_SECONDS": "0.05",
+ "REACHY_MOTION_COMPLETION_TIMEOUT_SECONDS": "6",
+ },
+ inherit_current=False,
+ )
+
+ assert config_mod.config.REACHY_TOOL_TRANSPORT == "rest"
+ assert config_mod.config.REACHY_REST_BASE_URL == "http://127.0.0.1:8000"
+ assert config_mod.config.REACHY_CAMERA_BASE_URL == "http://127.0.0.1:8042"
+ assert config_mod.config.REACHY_REST_TIMEOUT_SECONDS == 3.0
+ assert config_mod.config.REACHY_MOTION_DURATION_SECONDS == 0.5
+ assert config_mod.config.REACHY_MOTION_POLL_INTERVAL_SECONDS == 0.05
+ assert config_mod.config.REACHY_MOTION_COMPLETION_TIMEOUT_SECONDS == 6.0
+ finally:
+ _restore_config_snapshot(snapshot)
+
+
def test_load_dotenv_file_expands_system_env_and_applies_values(tmp_path: Any, monkeypatch: Any) -> None:
"""Explicit instance .env loading supports values sourced from system env names."""
tracked_attrs = {
diff --git a/projects/reachy-mini-openshell/tests/test_conversation_stream.py b/projects/reachy-mini-openshell/tests/test_conversation_stream.py
index f0b89be6..760435ee 100644
--- a/projects/reachy-mini-openshell/tests/test_conversation_stream.py
+++ b/projects/reachy-mini-openshell/tests/test_conversation_stream.py
@@ -1,8 +1,10 @@
+import json
import base64
import random
import asyncio
import logging
from typing import Any, cast
+from pathlib import Path
from datetime import datetime, timezone
from unittest.mock import MagicMock
@@ -20,8 +22,14 @@
)
from reachy_mini_conversation_app.audio.pcm import wav_bytes, prepare_mono_int16_audio
from reachy_mini_conversation_app.tools.core_tools import ToolDependencies
-from reachy_mini_conversation_app.conversation_stream import ConversationStreamHandler, _compute_response_cost
-from reachy_mini_conversation_app.tools.background_tool_manager import ToolCallRoutine
+from reachy_mini_conversation_app.conversation_stream import (
+ ConversationStreamHandler,
+ _model_io_json,
+ _compute_response_cost,
+)
+from reachy_mini_conversation_app.tools.tool_constants import ToolState
+from reachy_mini_conversation_app.media_result_processor import ProcessedToolResult
+from reachy_mini_conversation_app.tools.background_tool_manager import ToolCallRoutine, ToolNotification
def _build_handler(loop: asyncio.AbstractEventLoop) -> ConversationStreamHandler:
@@ -82,6 +90,44 @@ def test_format_timestamp_uses_wall_clock() -> None:
assert year == datetime.now(timezone.utc).year
+def test_model_io_logging_redacts_secrets_and_binary_payloads() -> None:
+ """Detailed model logs keep useful text while omitting credentials and media bytes."""
+ raw_base64 = "A" * 1_000
+
+ logged = _model_io_json(
+ {
+ "api_key": "sk-secret-value",
+ "text": "Describe this image in detail.",
+ "image_url": f"data:image/jpeg;base64,{raw_base64}",
+ "audio": raw_base64,
+ "b64_images": [raw_base64, raw_base64],
+ }
+ )
+
+ assert "sk-secret-value" not in logged
+ assert raw_base64 not in logged
+ assert "" in logged
+ assert "image/jpeg" in logged
+ assert "estimated_bytes" in logged
+ assert "Describe this image in detail." in logged
+
+
+@pytest.mark.asyncio
+async def test_focused_model_request_logging_uses_info(caplog: Any) -> None:
+ """Focused model logs should work without enabling global DEBUG output."""
+ caplog.set_level(logging.INFO)
+ deps = ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock())
+ handler = ConversationStreamHandler(deps, model_logs=True)
+
+ handler._log_model_request(
+ "response.create",
+ {"response": {"instructions": "Describe the image.", "tool_choice": "none"}},
+ )
+
+ assert "MODEL request type=response.create" in caplog.text
+ assert "Describe the image." in caplog.text
+
+
def test_parse_hf_realtime_url_removes_realtime_path_and_preserves_query() -> None:
"""HF realtime URLs are split into OpenAI-compatible HTTP and websocket bases."""
parsed = realtime_mod.parse_hf_realtime_url("wss://example.test/v1/realtime?session_id=session_123&model=ignored")
@@ -169,7 +215,7 @@ async def test_hf_realtime_session_uses_configured_model_and_connect_query(monke
"""HF realtime sessions pass the selected model separately from session query params."""
_set_hf_realtime_test_config(monkeypatch)
monkeypatch.setattr(stream_mod, "get_session_instructions", lambda: "test instructions")
- monkeypatch.setattr(stream_mod, "get_tool_specs", lambda: [])
+ monkeypatch.setattr(stream_mod, "get_tool_specs_for_dependencies", lambda _deps: [])
connect_calls: list[dict[str, Any]] = []
session_updates: list[dict[str, Any]] = []
@@ -221,12 +267,74 @@ class FakeClient:
assert handler.connection is None
+@pytest.mark.asyncio
+async def test_transport_discovery_precedes_realtime_configuration(monkeypatch: Any) -> None:
+ """Remote schemas must be discovered before the model session is configured."""
+ _set_openai_test_config(monkeypatch)
+ events: list[str] = []
+ session_updates: list[dict[str, Any]] = []
+
+ class FakeTransport:
+ async def list_tools(self) -> list[dict[str, Any]]:
+ events.append("tools/list")
+ return [
+ {
+ "type": "function",
+ "name": "move_head",
+ "description": "Move Reachy's head",
+ "parameters": {"type": "object"},
+ }
+ ]
+
+ async def call_tool(self, name: str, arguments: dict[str, Any]) -> dict[str, Any]:
+ return {"name": name, "arguments": arguments}
+
+ async def close(self) -> None:
+ events.append("transport.close")
+
+ class FakeSession:
+ async def update(self, **kwargs: Any) -> None:
+ events.append("session.update")
+ session_updates.append(kwargs)
+
+ class FakeConn:
+ session = FakeSession()
+
+ async def __aenter__(self) -> "FakeConn":
+ return self
+
+ async def __aexit__(self, exc_type: Any, exc: Any, tb: Any) -> bool:
+ return False
+
+ def __aiter__(self) -> "FakeConn":
+ return self
+
+ async def __anext__(self) -> None:
+ raise StopAsyncIteration
+
+ class FakeRealtime:
+ def connect(self, **_kwargs: Any) -> FakeConn:
+ return FakeConn()
+
+ class FakeClient:
+ realtime = FakeRealtime()
+
+ handler = ConversationStreamHandler(ToolDependencies(), tool_transport=FakeTransport())
+ handler.client = cast(Any, FakeClient())
+
+ await handler._run_realtime_session()
+ await handler.shutdown()
+
+ assert events == ["tools/list", "session.update", "transport.close"]
+ assert session_updates[0]["session"]["tools"][0]["name"] == "move_head"
+
+
@pytest.mark.asyncio
async def test_hf_start_up_refreshes_client_between_realtime_retries(monkeypatch: Any) -> None:
"""HF session retries should allocate a fresh client/connect URL after abrupt closes."""
_set_hf_realtime_test_config(monkeypatch)
monkeypatch.setattr(stream_mod, "get_session_instructions", lambda: "test instructions")
- monkeypatch.setattr(stream_mod, "get_tool_specs", lambda: [])
+ monkeypatch.setattr(stream_mod, "get_tool_specs_for_dependencies", lambda _deps: [])
fake_closed_error = type("FakeConnectionClosedError", (Exception,), {})
monkeypatch.setattr(stream_mod, "ConnectionClosedError", fake_closed_error)
@@ -650,7 +758,7 @@ async def test_realtime_tool_call_waits_for_tool_result_before_response_create(m
"""Realtime tool calls should not request a follow-up before function_call_output is available."""
_set_openai_test_config(monkeypatch)
monkeypatch.setattr(stream_mod, "get_session_instructions", lambda: "test instructions")
- monkeypatch.setattr(stream_mod, "get_tool_specs", lambda: [])
+ monkeypatch.setattr(stream_mod, "get_tool_specs_for_dependencies", lambda _deps: [])
event_queue: asyncio.Queue[Any] = asyncio.Queue()
response_create_calls: list[dict[str, Any]] = []
@@ -748,6 +856,577 @@ class FakeBackgroundTool:
assert response_create_calls == []
+@pytest.mark.asyncio
+async def test_camera_result_sends_image_separately_from_function_output() -> None:
+ """Camera bytes belong in input_image, not the text function result."""
+ item_create_calls: list[dict[str, Any]] = []
+
+ class FakeConversationItem:
+ async def create(self, **kwargs: Any) -> None:
+ item_create_calls.append(kwargs)
+
+ class FakeConversation:
+ item = FakeConversationItem()
+
+ class FakeConnection:
+ conversation = FakeConversation()
+
+ deps = ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock())
+ handler = stream_mod.ConversationStreamHandler(deps)
+ handler.connection = FakeConnection()
+
+ await handler._handle_tool_result(
+ ToolNotification(
+ id="call_camera",
+ tool_name="camera",
+ is_idle_tool_call=False,
+ status=ToolState.COMPLETED,
+ result={
+ "b64_im": "jpeg-base64-data",
+ "question": "What is the person doing?",
+ },
+ )
+ )
+
+ function_output = item_create_calls[0]["item"]
+ image_message = item_create_calls[1]["item"]
+ parsed_output = json.loads(function_output["output"])
+ queued_response = handler._pending_responses.get_nowait()
+
+ assert function_output["type"] == "function_call_output"
+ assert parsed_output == {
+ "question": "What is the person doing?",
+ "status": "image_captured",
+ }
+ assert "jpeg-base64-data" not in function_output["output"]
+ assert image_message["content"] == [
+ {
+ "type": "input_image",
+ "image_url": "data:image/jpeg;base64,jpeg-base64-data",
+ }
+ ]
+ assert queued_response["response"]["tool_choice"] == "none"
+ assert "input image" in queued_response["response"]["instructions"]
+
+
+@pytest.mark.asyncio
+async def test_camera_processor_routes_raw_image_before_realtime() -> None:
+ """Configured routing should send Realtime only the approved text result."""
+ item_create_calls: list[dict[str, Any]] = []
+ processor_calls: list[tuple[str, dict[str, Any]]] = []
+
+ class FakeConversationItem:
+ async def create(self, **kwargs: Any) -> None:
+ item_create_calls.append(kwargs)
+
+ class FakeConversation:
+ item = FakeConversationItem()
+
+ class FakeConnection:
+ conversation = FakeConversation()
+
+ class FakeProcessor:
+ async def process(self, tool_name: str, result: dict[str, Any]) -> ProcessedToolResult:
+ processor_calls.append((tool_name, result))
+ return ProcessedToolResult(
+ model_payload={
+ "status": "image_analyzed",
+ "question": result["question"],
+ "image_description": "The person is waving.",
+ "selected_model": "approved-vision-model",
+ }
+ )
+
+ deps = ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock())
+ handler = stream_mod.ConversationStreamHandler(
+ deps,
+ media_result_processor=cast(Any, FakeProcessor()),
+ )
+ handler.connection = FakeConnection()
+
+ await handler._handle_tool_result(
+ ToolNotification(
+ id="call_camera_routed_before_realtime",
+ tool_name="camera",
+ is_idle_tool_call=False,
+ status=ToolState.COMPLETED,
+ result={
+ "b64_im": "raw-camera-bytes",
+ "question": "What am I doing?",
+ },
+ )
+ )
+
+ assert processor_calls == [
+ (
+ "camera",
+ {
+ "b64_im": "raw-camera-bytes",
+ "question": "What am I doing?",
+ },
+ )
+ ]
+ assert len(item_create_calls) == 1
+ function_output = item_create_calls[0]["item"]
+ assert function_output["type"] == "function_call_output"
+ assert "raw-camera-bytes" not in function_output["output"]
+ assert json.loads(function_output["output"])["image_description"] == "The person is waving."
+ queued_response = handler._pending_responses.get_nowait()
+ assert "approved vision model" in queued_response["response"]["instructions"]
+
+
+@pytest.mark.asyncio
+async def test_interrupted_routed_scan_is_explained_as_partial() -> None:
+ """The follow-up must not describe interrupted scan frames as a complete room scan."""
+ item_create_calls: list[dict[str, Any]] = []
+
+ class FakeConversationItem:
+ async def create(self, **kwargs: Any) -> None:
+ item_create_calls.append(kwargs)
+
+ class FakeConversation:
+ item = FakeConversationItem()
+
+ class FakeConnection:
+ conversation = FakeConversation()
+
+ class FakeProcessor:
+ async def process(self, tool_name: str, result: dict[str, Any]) -> ProcessedToolResult:
+ assert tool_name == "scan_scene"
+ assert result["scan_status"] == "scene_scan_incomplete"
+ return ProcessedToolResult(
+ model_payload={
+ "status": "scene_analyzed",
+ "scan_status": "scene_scan_incomplete",
+ "scan_warning": "Reachy lost its control connection during the sweep",
+ "returned_to_front": True,
+ "image_description": "The recorded frames show a desk and one chair.",
+ "selected_model": "approved-vision-model",
+ }
+ )
+
+ handler = stream_mod.ConversationStreamHandler(
+ ToolDependencies(),
+ media_result_processor=cast(Any, FakeProcessor()),
+ )
+ handler.connection = FakeConnection()
+
+ await handler._handle_tool_result(
+ ToolNotification(
+ id="call_interrupted_scan",
+ tool_name="scan_scene",
+ is_idle_tool_call=False,
+ status=ToolState.COMPLETED,
+ result={
+ "status": "scene_scan_incomplete",
+ "scan_status": "scene_scan_incomplete",
+ "scan_warning": "Reachy lost its control connection during the sweep",
+ "returned_to_front": True,
+ "question": "What did you see?",
+ "frame_timestamps_seconds": [0.5],
+ "b64_images": ["raw-frame"],
+ },
+ )
+ )
+
+ function_output = json.loads(item_create_calls[0]["item"]["output"])
+ queued_response = handler._pending_responses.get_nowait()
+ assert function_output["scan_status"] == "scene_scan_incomplete"
+ assert function_output["returned_to_front"] is True
+ assert "physical scene sweep was interrupted" in queued_response["response"]["instructions"]
+ assert "Do not claim a complete room scan" in queued_response["response"]["instructions"]
+
+
+@pytest.mark.asyncio
+async def test_policy_denial_preserves_structured_transport_result() -> None:
+ """Realtime should receive the policy status as well as its human-readable error."""
+ item_create_calls: list[dict[str, Any]] = []
+
+ class FakeConversationItem:
+ async def create(self, **kwargs: Any) -> None:
+ item_create_calls.append(kwargs)
+
+ class FakeConversation:
+ item = FakeConversationItem()
+
+ class FakeConnection:
+ conversation = FakeConversation()
+
+ handler = stream_mod.ConversationStreamHandler(ToolDependencies())
+ handler.connection = FakeConnection()
+
+ await handler._handle_tool_result(
+ ToolNotification(
+ id="call_dance",
+ tool_name="dance",
+ is_idle_tool_call=False,
+ status=ToolState.FAILED,
+ result={
+ "status": "policy_denied",
+ "tool": "dance",
+ "error": "Blocked by OpenShell policy",
+ },
+ error="Blocked by OpenShell policy",
+ )
+ )
+
+ function_output = item_create_calls[0]["item"]
+ assert json.loads(function_output["output"]) == {
+ "status": "policy_denied",
+ "tool": "dance",
+ "error": "Blocked by OpenShell policy",
+ }
+ tool_card = handler.output_queue.get_nowait().args[0]
+ assert tool_card["metadata"]["title"] == "🚫 OpenShell blocked tool dance"
+ queued_response = handler._pending_responses.get_nowait()
+ assert "blocked by the OpenShell policy" in queued_response["response"]["instructions"]
+ assert "Do not claim the robot lacks the physical capability" in queued_response["response"]["instructions"]
+
+
+@pytest.mark.asyncio
+async def test_routed_camera_result_returns_description_without_realtime_image() -> None:
+ """A dedicated vision route should keep raw image bytes out of the Realtime session."""
+ item_create_calls: list[dict[str, Any]] = []
+
+ class FakeConversationItem:
+ async def create(self, **kwargs: Any) -> None:
+ item_create_calls.append(kwargs)
+
+ class FakeConversation:
+ item = FakeConversationItem()
+
+ class FakeConnection:
+ conversation = FakeConversation()
+
+ deps = ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock())
+ handler = stream_mod.ConversationStreamHandler(deps)
+ handler.connection = FakeConnection()
+
+ await handler._handle_tool_result(
+ ToolNotification(
+ id="call_camera_routed",
+ tool_name="camera",
+ is_idle_tool_call=False,
+ status=ToolState.COMPLETED,
+ result={
+ "status": "image_analyzed",
+ "image_description": "The person is waving.",
+ "selected_model": "gpt-5.4-mini",
+ "response_id": "resp_vision",
+ },
+ )
+ )
+
+ assert len(item_create_calls) == 1
+ function_output = item_create_calls[0]["item"]
+ parsed_output = json.loads(function_output["output"])
+ queued_response = handler._pending_responses.get_nowait()
+
+ assert function_output["type"] == "function_call_output"
+ assert parsed_output["image_description"] == "The person is waving."
+ assert parsed_output["selected_model"] == "gpt-5.4-mini"
+ assert queued_response["response"]["tool_choice"] == "none"
+ assert "approved vision model" in queued_response["response"]["instructions"]
+
+
+@pytest.mark.asyncio
+async def test_scene_scan_result_sends_chronological_images_and_video_preview(
+ monkeypatch: Any,
+ tmp_path: Path,
+) -> None:
+ """Scene-scan media stays out of tool JSON and reaches the vision model as images."""
+ item_create_calls: list[dict[str, Any]] = []
+
+ class FakeConversationItem:
+ async def create(self, **kwargs: Any) -> None:
+ item_create_calls.append(kwargs)
+
+ class FakeConversation:
+ item = FakeConversationItem()
+
+ class FakeConnection:
+ conversation = FakeConversation()
+
+ video_path = tmp_path / "scene-scan.mp4"
+ video_path.write_bytes(b"fake-mp4")
+ monkeypatch.setattr(stream_mod.gr, "Video", lambda **kwargs: {"video": kwargs["value"]})
+
+ deps = ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock())
+ handler = stream_mod.ConversationStreamHandler(deps)
+ handler.connection = FakeConnection()
+
+ await handler._handle_tool_result(
+ ToolNotification(
+ id="call_scan",
+ tool_name="scan_scene",
+ is_idle_tool_call=False,
+ status=ToolState.COMPLETED,
+ result={
+ "status": "scene_scan_complete",
+ "question": "What did you see?",
+ "video_path": str(video_path),
+ "frame_timestamps_seconds": [0.5, 7.0],
+ "b64_images": ["first-jpeg", "second-jpeg"],
+ },
+ )
+ )
+
+ function_output = item_create_calls[0]["item"]
+ image_message = item_create_calls[1]["item"]
+ parsed_output = json.loads(function_output["output"])
+ queued_response = handler._pending_responses.get_nowait()
+
+ assert "b64_images" not in parsed_output
+ assert parsed_output["video_path"] == str(video_path)
+ assert image_message["content"][0]["type"] == "input_text"
+ assert "chronological" in image_message["content"][0]["text"]
+ assert image_message["content"][1:] == [
+ {
+ "type": "input_image",
+ "image_url": "data:image/jpeg;base64,first-jpeg",
+ },
+ {
+ "type": "input_image",
+ "image_url": "data:image/jpeg;base64,second-jpeg",
+ },
+ ]
+ assert queued_response["response"]["tool_choice"] == "none"
+ assert "every chronological image" in queued_response["response"]["instructions"]
+
+ tool_card = await handler.output_queue.get()
+ video_preview = await handler.output_queue.get()
+ assert isinstance(tool_card, stream_mod.AdditionalOutputs)
+ assert isinstance(video_preview, stream_mod.AdditionalOutputs)
+ assert tool_card.args[0]["metadata"]["title"] == "🛠️ Used tool scan_scene"
+ assert video_preview.args[0]["content"] == {"video": str(video_path)}
+
+
+@pytest.mark.asyncio
+async def test_typed_realtime_message_waits_for_answer_after_tool_updates(monkeypatch: Any) -> None:
+ """A tool card is an intermediate update, not the final typed-chat answer."""
+ monkeypatch.setattr(stream_mod, "backend_config_error", lambda: None)
+
+ class FakeConversationItem:
+ async def create(self, **_kwargs: Any) -> None:
+ return None
+
+ class FakeConversation:
+ item = FakeConversationItem()
+
+ class FakeConnection:
+ conversation = FakeConversation()
+
+ deps = ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock())
+ handler = stream_mod.ConversationStreamHandler(deps)
+ handler.connection = FakeConnection()
+ monkeypatch.setattr(handler, "_text_model_uses_realtime", lambda: True)
+
+ async def ensure_text_session() -> bool:
+ return True
+
+ monkeypatch.setattr(handler, "_ensure_text_session", ensure_text_session)
+ handler._response_done_event.clear()
+
+ send_task = asyncio.create_task(handler.send_text_message("Take a picture", timeout=2.0))
+ await asyncio.sleep(0)
+ await handler._publish_chat_output(
+ {
+ "role": "assistant",
+ "content": "🛠️ Used tool camera with args {}. The tool is now running.",
+ }
+ )
+ handler._response_done_event.set()
+
+ await asyncio.sleep(0.05)
+ assert not send_task.done()
+
+ handler._response_done_event.clear()
+ await handler._publish_chat_output(
+ {
+ "role": "assistant",
+ "content": "You are sitting in front of the camera.",
+ }
+ )
+ handler._response_done_event.set()
+
+ messages = await asyncio.wait_for(send_task, timeout=1.0)
+
+ assert messages[-1] == {
+ "role": "assistant",
+ "content": "You are sitting in front of the camera.",
+ }
+
+
+@pytest.mark.asyncio
+async def test_typed_realtime_output_is_not_stolen_by_stream_consumer(monkeypatch: Any) -> None:
+ """FastRTC and typed chat should each receive a copy of assistant text."""
+ monkeypatch.setattr(stream_mod, "backend_config_error", lambda: None)
+
+ class FakeConversationItem:
+ async def create(self, **_kwargs: Any) -> None:
+ return None
+
+ class FakeConversation:
+ item = FakeConversationItem()
+
+ class FakeConnection:
+ conversation = FakeConversation()
+
+ deps = ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock())
+ handler = stream_mod.ConversationStreamHandler(deps)
+ handler.connection = FakeConnection()
+ monkeypatch.setattr(handler, "_text_model_uses_realtime", lambda: True)
+
+ async def ensure_text_session() -> bool:
+ return True
+
+ monkeypatch.setattr(handler, "_ensure_text_session", ensure_text_session)
+ handler._response_done_event.clear()
+
+ send_task = asyncio.create_task(handler.send_text_message("Describe the picture", timeout=2.0))
+ await asyncio.sleep(0)
+ await handler._publish_chat_output(
+ {
+ "role": "assistant",
+ "content": "I can see a person standing by a desk.",
+ }
+ )
+
+ stream_output = await asyncio.wait_for(handler.output_queue.get(), timeout=1.0)
+ handler._response_done_event.set()
+ messages = await asyncio.wait_for(send_task, timeout=1.0)
+
+ assert isinstance(stream_output, stream_mod.AdditionalOutputs)
+ assert stream_output.args[0]["content"] == "I can see a person standing by a desk."
+ assert messages[-1]["content"] == "I can see a person standing by a desk."
+
+
+@pytest.mark.asyncio
+async def test_typed_realtime_waits_past_tool_commentary_for_followup(monkeypatch: Any) -> None:
+ """Pre-tool commentary must not finish a typed turn before the tool's final answer."""
+ monkeypatch.setattr(stream_mod, "backend_config_error", lambda: None)
+
+ class FakeConversationItem:
+ async def create(self, **_kwargs: Any) -> None:
+ return None
+
+ class FakeConversation:
+ item = FakeConversationItem()
+
+ class FakeConnection:
+ conversation = FakeConversation()
+
+ deps = ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock())
+ handler = stream_mod.ConversationStreamHandler(deps)
+ handler.connection = FakeConnection()
+ monkeypatch.setattr(handler, "_text_model_uses_realtime", lambda: True)
+
+ async def ensure_text_session() -> bool:
+ return True
+
+ monkeypatch.setattr(handler, "_ensure_text_session", ensure_text_session)
+ handler._response_done_event.clear()
+
+ send_task = asyncio.create_task(handler.send_text_message("What am I doing?", timeout=2.0))
+ await asyncio.sleep(0)
+ await handler._publish_chat_output(
+ {
+ "role": "assistant",
+ "content": "Let me check the camera.",
+ }
+ )
+ handler._typed_tool_calls_awaiting_followup.add("call_camera")
+ handler._typed_followup_call_order.append("call_camera")
+ handler._tool_call_response_ids.add("resp_tool_call")
+ handler._response_done_event.set()
+
+ await asyncio.sleep(0.05)
+ assert not send_task.done()
+
+ handler._response_done_event.clear()
+ handler._mark_typed_followup_response("resp_camera_answer")
+ await handler._publish_chat_output(
+ {
+ "role": "assistant",
+ "content": "You are sitting at a desk with one hand near your mouth.",
+ }
+ )
+ handler._response_done_event.set()
+
+ messages = await asyncio.wait_for(send_task, timeout=1.0)
+
+ assert messages[-1]["content"] == "You are sitting at a desk with one hand near your mouth."
+
+
+@pytest.mark.asyncio
+async def test_typed_realtime_uses_longer_timeout_while_tool_is_pending(monkeypatch: Any) -> None:
+ """A long scan should stay attached after the normal response timeout expires."""
+ monkeypatch.setattr(stream_mod, "backend_config_error", lambda: None)
+
+ class FakeConversationItem:
+ async def create(self, **_kwargs: Any) -> None:
+ return None
+
+ class FakeConversation:
+ item = FakeConversationItem()
+
+ class FakeConnection:
+ conversation = FakeConversation()
+
+ handler = stream_mod.ConversationStreamHandler(ToolDependencies())
+ handler.connection = FakeConnection()
+ monkeypatch.setattr(handler, "_text_model_uses_realtime", lambda: True)
+
+ async def ensure_text_session() -> bool:
+ return True
+
+ monkeypatch.setattr(handler, "_ensure_text_session", ensure_text_session)
+ handler._response_done_event.clear()
+
+ send_task = asyncio.create_task(handler.send_text_message("Scan the room", timeout=0.05, tool_timeout=0.5))
+ await asyncio.sleep(0)
+ await handler._publish_chat_output({"role": "assistant", "content": "Okay, I’ll scan the room."})
+ handler._typed_tool_calls_awaiting_followup.add("call_scan")
+ handler._typed_followup_call_order.append("call_scan")
+ handler._response_done_event.set()
+
+ await asyncio.sleep(0.1)
+ assert not send_task.done()
+
+ handler._response_done_event.clear()
+ handler._mark_typed_followup_response("resp_scan_answer")
+ await handler._publish_chat_output(
+ {
+ "role": "assistant",
+ "content": "I saw a desk, a chair, and a clear walking path.",
+ }
+ )
+ handler._response_done_event.set()
+
+ messages = await asyncio.wait_for(send_task, timeout=1.0)
+
+ assert messages[-1]["content"] == "I saw a desk, a chair, and a clear walking path."
+ assert not any(message.get("content", "").startswith("[error] Timed out") for message in messages)
+
+
+def test_response_message_text_extracts_audio_transcript_fallback() -> None:
+ """response.done can recover text when a backend omits transcript.done."""
+
+ class Content:
+ transcript = "I can see someone wearing a blue shirt."
+ text = None
+
+ class Message:
+ type = "message"
+ role = "assistant"
+ content = [Content()]
+
+ class Response:
+ output = [Message()]
+
+ assert ConversationStreamHandler._response_message_text(Response()) == "I can see someone wearing a blue shirt."
+
+
@pytest.mark.asyncio
async def test_receive_transcribes_microphone_audio_for_non_realtime_model(monkeypatch: Any) -> None:
"""Local-STT mic mode transcribes speech and sends the transcript through Chat Completions."""
@@ -1108,8 +1787,8 @@ async def test_chat_completion_tool_follow_up_keeps_tools_param(monkeypatch: Any
monkeypatch.setattr(stream_mod.config, "CHAT_MODEL_NAME", "azure/anthropic/claude-opus-4-8")
monkeypatch.setattr(
chat_mod,
- "get_tool_specs",
- lambda: [
+ "get_tool_specs_for_dependencies",
+ lambda _deps: [
{
"type": "function",
"name": "dance",
@@ -1194,8 +1873,8 @@ async def test_chat_completion_supports_multiple_tool_rounds(monkeypatch: Any) -
_set_local_stt_test_config(monkeypatch)
monkeypatch.setattr(
chat_mod,
- "get_tool_specs",
- lambda: [
+ "get_tool_specs_for_dependencies",
+ lambda _deps: [
{
"type": "function",
"name": "play_emotion",
@@ -1302,8 +1981,8 @@ async def test_chat_completion_accepts_dict_shaped_messages_and_tool_calls(monke
_set_local_stt_test_config(monkeypatch)
monkeypatch.setattr(
chat_mod,
- "get_tool_specs",
- lambda: [
+ "get_tool_specs_for_dependencies",
+ lambda _deps: [
{
"type": "function",
"name": "play_emotion",
@@ -1379,8 +2058,8 @@ async def test_chat_completion_tool_follow_up_retries_rate_limit(monkeypatch: An
monkeypatch.setattr(stream_mod.config, "CHAT_MODEL_NAME", "azure/anthropic/claude-opus-4-8")
monkeypatch.setattr(
chat_mod,
- "get_tool_specs",
- lambda: [
+ "get_tool_specs_for_dependencies",
+ lambda _deps: [
{
"type": "function",
"name": "sweep_look",
@@ -1550,7 +2229,7 @@ async def test_response_sender_retries_on_active_response_rejection(monkeypatch:
monkeypatch.setattr(stream_mod, "ConnectionClosedError", FakeCCE)
monkeypatch.setattr(stream_mod, "get_session_instructions", lambda: "test")
monkeypatch.setattr(stream_mod, "get_session_voice", lambda *_args: "alloy")
- monkeypatch.setattr(stream_mod, "get_tool_specs", lambda: [])
+ monkeypatch.setattr(stream_mod, "get_tool_specs_for_dependencies", lambda _deps: [])
N_TOOL_RESULTS = 400
REJECT_CALL_NUMBERS = {1, 3, 5, 10, 25, 50, 75, 100, 150, 200, 300, 399}
diff --git a/projects/reachy-mini-openshell/tests/test_main.py b/projects/reachy-mini-openshell/tests/test_main.py
index ba613e35..7a7a90ed 100644
--- a/projects/reachy-mini-openshell/tests/test_main.py
+++ b/projects/reachy-mini-openshell/tests/test_main.py
@@ -1,6 +1,13 @@
+import sys
from typing import Any
+import pytest
+
import reachy_mini_conversation_app.main as main_mod
+from reachy_mini_conversation_app.utils import parse_args
+from reachy_mini_conversation_app.tool_transport import RoutedToolTransport, ConversationUtilityTransport
+from reachy_mini_conversation_app.tools.core_tools import ToolDependencies
+from reachy_mini_conversation_app.rest_tool_transport import RestToolTransport
class _FakeLogger:
@@ -42,3 +49,85 @@ def stop() -> None:
assert args[0] == "media"
assert isinstance(args[1], RuntimeError)
assert str(args[1]) == "already stopped"
+
+
+def test_parse_args_accepts_rest_tool_transport(monkeypatch: Any) -> None:
+ """The standalone app should expose REST mode as a CLI choice."""
+ monkeypatch.setattr(sys, "argv", ["reachy-mini-conversation-app", "--tool-transport", "rest"])
+
+ args, unknown = parse_args()
+
+ assert args.tool_transport == "rest"
+ assert unknown == []
+
+
+def test_rest_transport_factory_requires_endpoint() -> None:
+ """REST mode should fail before robot or model startup when its endpoint is absent."""
+ dependencies = ToolDependencies()
+
+ with pytest.raises(ValueError, match="REACHY_REST_BASE_URL"):
+ main_mod._build_tool_transport_factory("rest", dependencies, rest_base_url=None)
+
+
+def test_rest_transport_factory_builds_routed_transport() -> None:
+ """Each conversation connection should receive its own REST/local routing stack."""
+ dependencies = ToolDependencies()
+ factory = main_mod._build_tool_transport_factory(
+ "rest",
+ dependencies,
+ rest_base_url="http://127.0.0.1:8000",
+ )
+
+ first = factory()
+ second = factory()
+
+ assert isinstance(first, RoutedToolTransport)
+ assert isinstance(first._remote, RestToolTransport)
+ assert isinstance(first._local, ConversationUtilityTransport)
+ assert first is not second
+
+
+@pytest.mark.asyncio
+async def test_rest_transport_factory_exposes_only_v1_tools() -> None:
+ """REST mode should hide every legacy camera, dance, emotion, and tracking tool."""
+ factory = main_mod._build_tool_transport_factory(
+ "rest",
+ ToolDependencies(),
+ rest_base_url="http://127.0.0.1:8000",
+ )
+ transport = factory()
+
+ tools = await transport.list_tools()
+ await transport.close()
+
+ assert {tool["name"] for tool in tools} == {
+ "move_head",
+ "stop_motion",
+ "do_nothing",
+ "task_status",
+ "task_cancel",
+ }
+
+
+@pytest.mark.asyncio
+async def test_rest_transport_factory_adds_camera_only_with_adapter_endpoint() -> None:
+ """REST mode should advertise camera only when the trusted native adapter is configured."""
+ factory = main_mod._build_tool_transport_factory(
+ "rest",
+ ToolDependencies(),
+ rest_base_url="http://127.0.0.1:8000",
+ camera_base_url="http://host.openshell.internal:8042",
+ )
+ transport = factory()
+
+ tools = await transport.list_tools()
+ await transport.close()
+
+ assert {tool["name"] for tool in tools} == {
+ "move_head",
+ "stop_motion",
+ "camera",
+ "do_nothing",
+ "task_status",
+ "task_cancel",
+ }
diff --git a/projects/reachy-mini-openshell/tests/test_media_result_processor.py b/projects/reachy-mini-openshell/tests/test_media_result_processor.py
new file mode 100644
index 00000000..9ae8fc29
--- /dev/null
+++ b/projects/reachy-mini-openshell/tests/test_media_result_processor.py
@@ -0,0 +1,231 @@
+"""Tests for the optional local raw-media boundary."""
+
+import base64
+from typing import Any
+from pathlib import Path
+
+import cv2
+import numpy as np
+import pytest
+
+from reachy_mini_conversation_app.vision_router import VisionAnalysis
+from reachy_mini_conversation_app.media_result_processor import (
+ MediaResultProcessor,
+ contains_raw_media,
+ assert_no_raw_media,
+)
+
+
+def _jpeg_base64(value: int = 0) -> str:
+ frame = np.full((4, 4, 3), value, dtype=np.uint8)
+ encoded, buffer = cv2.imencode(".jpg", frame)
+ assert encoded
+ return base64.b64encode(buffer.tobytes()).decode("ascii")
+
+
+class _FakeVisionRouter:
+ def __init__(self, *, fail: bool = False) -> None:
+ self.fail = fail
+ self.calls: list[dict[str, Any]] = []
+
+ async def analyze_images(self, **kwargs: Any) -> VisionAnalysis:
+ self.calls.append(kwargs)
+ if self.fail:
+ raise RuntimeError("vision unavailable")
+ return VisionAnalysis(
+ description="A person is sitting at a desk.",
+ selected_model="approved-vision-model",
+ response_id="resp_vision",
+ usage={"total_tokens": 42},
+ )
+
+
+def _processor(tmp_path: Path, router: Any) -> MediaResultProcessor:
+ return MediaResultProcessor(
+ vision_router=router,
+ capture_directory=tmp_path,
+ require_routed_vision=True,
+ )
+
+
+@pytest.mark.asyncio
+async def test_camera_is_analyzed_before_model_payload_is_created(tmp_path: Path) -> None:
+ """A local camera result should expose text to the model and pixels only to the UI."""
+ router = _FakeVisionRouter()
+ processor = _processor(tmp_path, router)
+ image = _jpeg_base64(25)
+
+ processed = await processor.process(
+ "camera",
+ {
+ "status": "image_captured",
+ "question": "What am I doing?",
+ "b64_im": image,
+ },
+ )
+
+ assert router.calls == [
+ {
+ "images_base64": [image],
+ "question": "What am I doing?",
+ "frame_timestamps": None,
+ }
+ ]
+ assert processed.model_payload == {
+ "status": "image_analyzed",
+ "question": "What am I doing?",
+ "image_description": "A person is sitting at a desk.",
+ "selected_model": "approved-vision-model",
+ "response_id": "resp_vision",
+ "usage": {"total_tokens": 42},
+ }
+ assert processed.preview_image is not None
+ assert processed.preview_image.shape == (4, 4, 3)
+ assert not contains_raw_media(processed.model_payload)
+
+
+@pytest.mark.asyncio
+async def test_scene_scan_routes_ordered_frames_and_keeps_local_video(tmp_path: Path) -> None:
+ """A local scan should retain its validated MP4 as a UI-only artifact."""
+ router = _FakeVisionRouter()
+ processor = _processor(tmp_path, router)
+ images = [_jpeg_base64(value) for value in range(3)]
+ timestamps = [0.0, 1.0, 2.0]
+ video_path = tmp_path / "scan.mp4"
+ video_path.write_bytes(b"fake-mp4")
+
+ processed = await processor.process(
+ "scan_scene",
+ {
+ "status": "scene_scan_complete",
+ "question": "What did you see?",
+ "video_path": str(video_path),
+ "frame_timestamps_seconds": timestamps,
+ "frames_selected": 3,
+ "b64_images": images,
+ },
+ )
+
+ assert router.calls == [
+ {
+ "images_base64": images,
+ "question": "What did you see?",
+ "frame_timestamps": timestamps,
+ }
+ ]
+ assert processed.model_payload["status"] == "scene_analyzed"
+ assert processed.model_payload["recording_status"] == "available"
+ assert processed.model_payload["image_description"] == "A person is sitting at a desk."
+ assert "b64_images" not in processed.model_payload
+ assert "video_path" not in processed.model_payload
+ assert processed.video_path == video_path
+
+
+@pytest.mark.asyncio
+async def test_scene_scan_preserves_interruption_metadata(tmp_path: Path) -> None:
+ """Vision analysis must not erase the physical scan's incomplete status."""
+ router = _FakeVisionRouter()
+ processor = _processor(tmp_path, router)
+ video_path = tmp_path / "partial.mp4"
+ video_path.write_bytes(b"partial")
+
+ processed = await processor.process(
+ "scan_scene",
+ {
+ "status": "scene_scan_incomplete",
+ "scan_status": "scene_scan_incomplete",
+ "scan_warning": "Reachy lost its control connection during the sweep",
+ "returned_to_front": True,
+ "front_verified": True,
+ "question": "What did you see?",
+ "video_path": str(video_path),
+ "frame_timestamps_seconds": [0.0],
+ "b64_images": [_jpeg_base64()],
+ },
+ )
+
+ assert processed.model_payload["status"] == "scene_analyzed"
+ assert processed.model_payload["scan_status"] == "scene_scan_incomplete"
+ assert processed.model_payload["returned_to_front"] is True
+ assert processed.model_payload["front_verified"] is True
+ assert "lost its control connection" in processed.model_payload["scan_warning"]
+
+
+@pytest.mark.asyncio
+async def test_scene_scan_preserves_analysis_when_local_preview_is_invalid(tmp_path: Path) -> None:
+ """An invalid local recording must not discard successful vision output."""
+ router = _FakeVisionRouter()
+ processor = _processor(tmp_path, router)
+
+ processed = await processor.process(
+ "scan_scene",
+ {
+ "status": "scene_scan_complete",
+ "question": "What did you see?",
+ "video_path": str(tmp_path / "missing.mp4"),
+ "frame_timestamps_seconds": [0.0],
+ "b64_images": [_jpeg_base64()],
+ },
+ )
+
+ assert processed.model_payload["status"] == "scene_analyzed"
+ assert processed.model_payload["recording_status"] == "preview_unavailable"
+ assert processed.video_path is None
+ assert not contains_raw_media(processed.model_payload)
+
+
+@pytest.mark.asyncio
+async def test_vision_failure_discards_camera_bytes(tmp_path: Path) -> None:
+ """A failed local vision route must discard raw camera bytes."""
+ processor = _processor(tmp_path, _FakeVisionRouter(fail=True))
+
+ processed = await processor.process(
+ "camera",
+ {"question": "Describe this.", "b64_im": _jpeg_base64()},
+ )
+
+ assert processed.model_payload == {
+ "status": "vision_error",
+ "tool": "camera",
+ "error": "Approved vision analysis failed; raw media was discarded",
+ }
+ assert processed.preview_image is None
+ assert not contains_raw_media(processed.model_payload)
+
+
+@pytest.mark.asyncio
+async def test_scene_scan_rejects_more_than_nine_frames_before_upload(tmp_path: Path) -> None:
+ """Oversized local frame sets should fail before model upload."""
+ router = _FakeVisionRouter()
+ processor = _processor(tmp_path, router)
+
+ processed = await processor.process(
+ "scan_scene",
+ {
+ "question": "What did you see?",
+ "frame_timestamps_seconds": list(range(10)),
+ "b64_images": [_jpeg_base64()] * 10,
+ },
+ )
+
+ assert processed.model_payload["status"] == "media_security_error"
+ assert router.calls == []
+
+
+def test_strict_routing_requires_a_vision_router(tmp_path: Path) -> None:
+ """Explicit local strict mode should require a configured vision router."""
+ with pytest.raises(ValueError, match="requires a configured VisionRouter"):
+ MediaResultProcessor(
+ vision_router=None,
+ capture_directory=tmp_path,
+ require_routed_vision=True,
+ )
+
+
+def test_recursive_raw_media_guard_rejects_data_urls() -> None:
+ """The serialization guard should detect nested raw image data URLs."""
+ payload = {"nested": [{"image_url": "data:image/jpeg;base64,secret"}]}
+
+ assert contains_raw_media(payload)
+ with pytest.raises(RuntimeError, match="Raw media reached"):
+ assert_no_raw_media(payload)
diff --git a/projects/reachy-mini-openshell/tests/test_moves_delivery.py b/projects/reachy-mini-openshell/tests/test_moves_delivery.py
new file mode 100644
index 00000000..446ad191
--- /dev/null
+++ b/projects/reachy-mini-openshell/tests/test_moves_delivery.py
@@ -0,0 +1,94 @@
+from __future__ import annotations
+from typing import Any
+
+import numpy as np
+
+from reachy_mini.utils import create_head_pose
+from reachy_mini_conversation_app.moves import MovementManager
+
+
+class _Client:
+ def __init__(self) -> None:
+ self._is_alive = True
+
+
+class _Robot:
+ def __init__(self, *, failure: Exception | None = None) -> None:
+ self.client = _Client()
+ self.failure = failure
+ self.targets: list[dict[str, Any]] = []
+
+ def set_target(self, **kwargs: Any) -> None:
+ self.targets.append(kwargs)
+ if self.failure is not None:
+ raise self.failure
+
+
+class _ObservedRobot(_Robot):
+ def __init__(self) -> None:
+ super().__init__()
+ self.head_pose = create_head_pose(0, 0, 0, 0, 0, 10, degrees=True)
+
+ def get_current_head_pose(self) -> np.ndarray[Any, Any]:
+ return self.head_pose.copy()
+
+ def get_current_joint_positions(self) -> tuple[list[float], list[float]]:
+ return [0.1] + [0.0] * 6, [0.2, -0.2]
+
+
+def _target() -> tuple[np.ndarray[Any, Any], tuple[float, float], float]:
+ return create_head_pose(0, 0, 0, 0, 0, 20, degrees=True), (0.0, 0.0), 0.0
+
+
+def test_successful_target_send_advances_delivery_checkpoint() -> None:
+ """A successful socket write should release delivery waiters."""
+ robot = _Robot()
+ manager = MovementManager(robot) # type: ignore[arg-type]
+ checkpoint = manager.delivery_checkpoint()
+
+ manager._issue_control_command(*_target())
+
+ assert manager.wait_for_delivery(checkpoint, timeout=0) == (True, None)
+ assert manager.connection_healthy() is True
+ assert len(robot.targets) == 1
+
+
+def test_first_delivery_failure_pauses_output_without_retry_flood() -> None:
+ """An uncertain send should be terminal for this runtime and logged only once."""
+ robot = _Robot(failure=ConnectionError("socket closed"))
+ manager = MovementManager(robot) # type: ignore[arg-type]
+ checkpoint = manager.delivery_checkpoint()
+
+ manager._issue_control_command(*_target())
+ manager._issue_control_command(*_target())
+
+ delivered, error = manager.wait_for_delivery(checkpoint, timeout=0)
+ assert delivered is False
+ assert error is not None and "socket closed" in error
+ assert manager.connection_healthy() is False
+ assert len(robot.targets) == 1
+
+
+def test_unchanged_target_is_not_sent_continuously() -> None:
+ """Idle control ticks should not flood the wireless daemon with no-op targets."""
+ robot = _Robot()
+ manager = MovementManager(robot, target_frequency_hz=50, enable_idle_breathing=False) # type: ignore[arg-type]
+ target = _target()
+
+ assert manager._target_changed(*target) is True
+ manager._issue_control_command(*target)
+ assert manager._target_changed(*target) is False
+
+ changed = (target[0].copy(), target[1], target[2])
+ changed[0][0, 3] += 0.001
+ assert manager._target_changed(*changed) is True
+
+
+def test_observed_starting_pose_does_not_command_neutral_on_startup() -> None:
+ """Starting the standalone runtime should preserve the robot's current physical pose."""
+ robot = _ObservedRobot()
+ manager = MovementManager(robot, enable_idle_breathing=False) # type: ignore[arg-type]
+ observed = (robot.head_pose, (0.2, -0.2), 0.1)
+
+ assert manager._target_changed(*observed) is False
+ assert robot.targets == []
diff --git a/projects/reachy-mini-openshell/tests/test_prompts.py b/projects/reachy-mini-openshell/tests/test_prompts.py
new file mode 100644
index 00000000..3ade4c10
--- /dev/null
+++ b/projects/reachy-mini-openshell/tests/test_prompts.py
@@ -0,0 +1,13 @@
+"""Tests for locked Realtime session instructions."""
+
+from reachy_mini_conversation_app.prompts import get_session_instructions
+
+
+def test_camera_instructions_require_explicit_request_and_tool_use() -> None:
+ """The prompt should use camera on request without enabling automatic capture."""
+ instructions = get_session_instructions()
+
+ assert "explicitly asks Reachy to take a picture or photo" in instructions
+ assert "call camera" in instructions
+ assert "Do not take pictures unless the human explicitly requests one" in instructions
+ assert "Do not offer camera" not in instructions
diff --git a/projects/reachy-mini-openshell/tests/test_rest_tool_transport.py b/projects/reachy-mini-openshell/tests/test_rest_tool_transport.py
new file mode 100644
index 00000000..802b4255
--- /dev/null
+++ b/projects/reachy-mini-openshell/tests/test_rest_tool_transport.py
@@ -0,0 +1,284 @@
+"""Tests for the fixed Reachy daemon REST transport."""
+
+from __future__ import annotations
+import json
+import math
+import base64
+from typing import Any
+
+import httpx
+import pytest
+
+from reachy_mini_conversation_app.rest_tool_transport import RestToolTransport, RestTransportSettings
+
+
+MOVE_1 = "11111111-1111-4111-8111-111111111111"
+MOVE_2 = "22222222-2222-4222-8222-222222222222"
+
+
+def _settings(**overrides: Any) -> RestTransportSettings:
+ values = {
+ "base_url": "http://reachy.test:8000",
+ "request_timeout_seconds": 1.0,
+ "motion_duration_seconds": 1.0,
+ "poll_interval_seconds": 0.001,
+ "completion_timeout_seconds": 1.0,
+ }
+ values.update(overrides)
+ return RestTransportSettings(**values)
+
+
+def _json_body(request: httpx.Request) -> dict[str, Any]:
+ return json.loads(request.content.decode("utf-8"))
+
+
+@pytest.mark.asyncio
+async def test_rest_transport_advertises_only_fixed_motion_tools() -> None:
+ """REST discovery should expose only immutable motion and stop schemas."""
+ client = httpx.AsyncClient(transport=httpx.MockTransport(lambda request: httpx.Response(500, request=request)))
+ transport = RestToolTransport(_settings(), client=client)
+
+ first = await transport.list_tools()
+ first[0]["parameters"]["properties"]["directions"]["items"]["enum"].append("raw_pose")
+ second = await transport.list_tools()
+
+ assert [tool["name"] for tool in second] == ["move_head", "stop_motion"]
+ assert second[0]["parameters"]["additionalProperties"] is False
+ assert second[0]["parameters"]["properties"]["directions"]["items"]["enum"] == [
+ "left",
+ "right",
+ "up",
+ "down",
+ "front",
+ ]
+ assert second[1]["parameters"] == {"type": "object", "properties": {}, "additionalProperties": False}
+ await client.aclose()
+
+
+@pytest.mark.asyncio
+async def test_configured_camera_tool_captures_one_bounded_jpeg() -> None:
+ """Camera discovery and capture should use only the fixed adapter endpoint."""
+ jpeg = b"\xff\xd8camera-frame\xff\xd9"
+ camera_requests: list[httpx.Request] = []
+
+ def camera_handler(request: httpx.Request) -> httpx.Response:
+ camera_requests.append(request)
+ return httpx.Response(200, content=jpeg, headers={"content-type": "image/jpeg"}, request=request)
+
+ async with (
+ httpx.AsyncClient(transport=httpx.MockTransport(lambda request: httpx.Response(500, request=request))) as client,
+ httpx.AsyncClient(transport=httpx.MockTransport(camera_handler)) as camera_client,
+ ):
+ transport = RestToolTransport(
+ _settings(camera_base_url="http://camera.test:8042"),
+ client=client,
+ camera_client=camera_client,
+ )
+ tools = await transport.list_tools()
+ result = await transport.call_tool("camera", {"question": "What is in front of me?"})
+
+ assert [tool["name"] for tool in tools] == ["move_head", "stop_motion", "camera"]
+ assert [(request.method, request.url.path, request.content) for request in camera_requests] == [
+ ("POST", "/camera/capture", b""),
+ ]
+ assert result == {
+ "status": "captured",
+ "tool": "camera",
+ "question": "What is in front of me?",
+ "b64_im": base64.b64encode(jpeg).decode("ascii"),
+ }
+
+
+@pytest.mark.asyncio
+async def test_camera_policy_denial_and_invalid_arguments_are_model_visible() -> None:
+ """Camera policy denial should be explicit and invalid arguments must not reach the adapter."""
+ request_count = 0
+
+ def camera_handler(request: httpx.Request) -> httpx.Response:
+ nonlocal request_count
+ request_count += 1
+ return httpx.Response(403, json={"error": "denied"}, request=request)
+
+ async with (
+ httpx.AsyncClient(transport=httpx.MockTransport(lambda request: httpx.Response(500, request=request))) as client,
+ httpx.AsyncClient(transport=httpx.MockTransport(camera_handler)) as camera_client,
+ ):
+ transport = RestToolTransport(
+ _settings(camera_base_url="http://camera.test:8042"),
+ client=client,
+ camera_client=camera_client,
+ )
+ denied = await transport.call_tool("camera", {"question": "What do you see?"})
+ empty = await transport.call_tool("camera", {"question": " "})
+ extra = await transport.call_tool("camera", {"question": "What?", "filename": "/tmp/x"})
+
+ assert denied == {
+ "status": "policy_denied",
+ "tool": "camera",
+ "error": "Blocked by OpenShell policy: POST /camera/capture",
+ }
+ assert empty["status"] == "invalid_arguments"
+ assert extra["status"] == "invalid_arguments"
+ assert request_count == 1
+
+
+@pytest.mark.asyncio
+async def test_move_head_maps_directions_to_fixed_rest_poses_in_order() -> None:
+ """Directions should become fixed head-only poses sent sequentially."""
+ requests: list[httpx.Request] = []
+ goto_ids = iter((MOVE_1, MOVE_2))
+ current_move: list[str | None] = [None]
+ poll_counts: dict[str, int] = {}
+
+ def handler(request: httpx.Request) -> httpx.Response:
+ requests.append(request)
+ if request.method == "POST" and request.url.path == "/api/move/goto":
+ move_id = next(goto_ids)
+ current_move[0] = move_id
+ return httpx.Response(200, json={"uuid": move_id}, request=request)
+ if request.method == "GET" and request.url.path == "/api/move/running":
+ move_id = current_move[0]
+ assert move_id is not None
+ poll_counts[move_id] = poll_counts.get(move_id, 0) + 1
+ if poll_counts[move_id] == 1:
+ return httpx.Response(200, json=[{"uuid": move_id}], request=request)
+ current_move[0] = None
+ return httpx.Response(200, json=[], request=request)
+ return httpx.Response(500, request=request)
+
+ async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
+ transport = RestToolTransport(_settings(), client=client)
+ result = await transport.call_tool("move_head", {"directions": ["up", "right"]})
+
+ goto_requests = [request for request in requests if request.url.path == "/api/move/goto"]
+ assert result == {
+ "status": "completed",
+ "tool": "move_head",
+ "directions": ["up", "right"],
+ "total_duration_seconds": 2.0,
+ }
+ assert len(goto_requests) == 2
+ assert [(request.method, request.url.path) for request in requests] == [
+ ("POST", "/api/move/goto"),
+ ("GET", "/api/move/running"),
+ ("GET", "/api/move/running"),
+ ("POST", "/api/move/goto"),
+ ("GET", "/api/move/running"),
+ ("GET", "/api/move/running"),
+ ]
+ up = _json_body(goto_requests[0])
+ right = _json_body(goto_requests[1])
+ assert up == {
+ "head_pose": {
+ "x": 0.0,
+ "y": 0.0,
+ "z": 0.0,
+ "roll": 0.0,
+ "pitch": pytest.approx(math.radians(-30.0)),
+ "yaw": 0.0,
+ },
+ "duration": 1.0,
+ "interpolation": "minjerk",
+ }
+ assert right["head_pose"]["pitch"] == 0.0
+ assert right["head_pose"]["yaw"] == pytest.approx(math.radians(-40.0))
+ assert "antennas" not in up
+ assert "body_yaw" not in up
+
+
+@pytest.mark.asyncio
+async def test_move_head_rejects_raw_or_invalid_arguments_without_network_access() -> None:
+ """Raw pose fields and invalid direction lists must fail before networking."""
+ request_count = 0
+
+ def handler(request: httpx.Request) -> httpx.Response:
+ nonlocal request_count
+ request_count += 1
+ return httpx.Response(500, request=request)
+
+ async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
+ transport = RestToolTransport(_settings(), client=client)
+ raw_pose = await transport.call_tool("move_head", {"directions": ["up"], "pitch": -2.0})
+ invalid_direction = await transport.call_tool("move_head", {"directions": ["backward"]})
+ too_many = await transport.call_tool("move_head", {"directions": ["up"] * 9})
+
+ assert raw_pose["status"] == "invalid_arguments"
+ assert invalid_direction["status"] == "invalid_arguments"
+ assert too_many["status"] == "invalid_arguments"
+ assert request_count == 0
+
+
+@pytest.mark.asyncio
+async def test_open_shell_forbidden_response_becomes_policy_denial() -> None:
+ """An OpenShell HTTP 403 should remain visible to the conversation model."""
+ def handler(request: httpx.Request) -> httpx.Response:
+ return httpx.Response(403, json={"error": "denied"}, request=request)
+
+ async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
+ transport = RestToolTransport(_settings(), client=client)
+ result = await transport.call_tool("move_head", {"directions": ["up"]})
+
+ assert result == {
+ "status": "policy_denied",
+ "tool": "move_head",
+ "error": "Blocked by OpenShell policy: POST /api/move/goto",
+ "directions": ["up"],
+ "completed_directions": [],
+ }
+
+
+@pytest.mark.asyncio
+async def test_motion_post_timeout_is_uncertain_and_is_not_retried() -> None:
+ """A timed-out motion POST must not be repeated after uncertain delivery."""
+ request_count = 0
+
+ def handler(request: httpx.Request) -> httpx.Response:
+ nonlocal request_count
+ request_count += 1
+ raise httpx.ReadTimeout("timeout after send", request=request)
+
+ async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
+ transport = RestToolTransport(_settings(), client=client)
+ result = await transport.call_tool("move_head", {"directions": ["left"]})
+
+ assert result["status"] == "unknown_delivery"
+ assert result["completed_directions"] == []
+ assert request_count == 1
+
+
+@pytest.mark.asyncio
+async def test_stop_motion_lists_and_stops_every_running_move() -> None:
+ """The stop tool should stop all daemon-reported move identifiers."""
+ stopped: list[str] = []
+
+ def handler(request: httpx.Request) -> httpx.Response:
+ if request.method == "GET" and request.url.path == "/api/move/running":
+ return httpx.Response(200, json=[{"uuid": MOVE_2}, {"uuid": MOVE_1}], request=request)
+ if request.method == "POST" and request.url.path == "/api/move/stop":
+ stopped.append(_json_body(request)["uuid"])
+ return httpx.Response(200, json={"status": "stopped"}, request=request)
+ return httpx.Response(500, request=request)
+
+ async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
+ transport = RestToolTransport(_settings(), client=client)
+ result = await transport.call_tool("stop_motion", {})
+
+ assert result == {"status": "stopped", "tool": "stop_motion", "stopped_move_ids": [MOVE_1, MOVE_2]}
+ assert stopped == [MOVE_1, MOVE_2]
+
+
+@pytest.mark.parametrize(
+ ("overrides", "message"),
+ [
+ ({"base_url": "reachy.test:8000"}, "absolute HTTP"),
+ ({"camera_base_url": "http://camera.test:8042/capture"}, "REACHY_CAMERA_BASE_URL"),
+ ({"request_timeout_seconds": 0.0}, "REACHY_REST_TIMEOUT_SECONDS"),
+ ({"motion_duration_seconds": float("nan")}, "REACHY_MOTION_DURATION_SECONDS"),
+ ({"poll_interval_seconds": -1.0}, "REACHY_MOTION_POLL_INTERVAL_SECONDS"),
+ ({"completion_timeout_seconds": 0.0}, "REACHY_MOTION_COMPLETION_TIMEOUT_SECONDS"),
+ ],
+)
+def test_rest_settings_reject_invalid_values(overrides: dict[str, Any], message: str) -> None:
+ """Invalid URLs and timing values should fail before transport creation."""
+ with pytest.raises(ValueError, match=message):
+ _settings(**overrides)
diff --git a/projects/reachy-mini-openshell/tests/test_robot_runtime.py b/projects/reachy-mini-openshell/tests/test_robot_runtime.py
new file mode 100644
index 00000000..75c292f2
--- /dev/null
+++ b/projects/reachy-mini-openshell/tests/test_robot_runtime.py
@@ -0,0 +1,329 @@
+from __future__ import annotations
+from typing import Any
+from pathlib import Path
+
+import pytest
+
+import reachy_mini_conversation_app.robot_runtime as runtime_mod
+
+
+class _Component:
+ def __init__(self, name: str, events: list[str], *, fail_start: bool = False) -> None:
+ self.name = name
+ self.events = events
+ self.fail_start = fail_start
+
+ def start(self) -> None:
+ self.events.append(f"{self.name}.start")
+ if self.fail_start:
+ raise RuntimeError(f"{self.name} failed")
+
+ def stop(self) -> None:
+ self.events.append(f"{self.name}.stop")
+
+
+class _MovementManager(_Component):
+ def set_speech_offsets(self, offsets: Any) -> None:
+ del offsets
+
+ def connection_healthy(self) -> bool:
+ return True
+
+
+class _Client:
+ def __init__(self, events: list[str], status: Any) -> None:
+ self.events = events
+ self.status = status
+
+ def get_status(self) -> Any:
+ return self.status
+
+ def disconnect(self) -> None:
+ self.events.append("client.disconnect")
+
+
+class _Media:
+ def __init__(self, events: list[str]) -> None:
+ self.events = events
+
+ def close(self) -> None:
+ self.events.append("media.close")
+
+
+class _Robot:
+ def __init__(self, events: list[str], status: Any) -> None:
+ self.client = _Client(events, status)
+ self.media = _Media(events)
+
+
+def _patch_runtime_factories(
+ monkeypatch: pytest.MonkeyPatch,
+ events: list[str],
+ *,
+ camera_worker: Any | None,
+ head_tracker: Any | None = None,
+ vision_manager: Any | None = None,
+ vision_router: Any | None = None,
+) -> tuple[_MovementManager, _Component]:
+ movement_manager = _MovementManager("movement", events)
+ head_wobbler = _Component("wobbler", events)
+
+ monkeypatch.setattr(
+ runtime_mod,
+ "handle_vision_stuff",
+ lambda args, robot: (camera_worker, head_tracker, vision_manager),
+ )
+ monkeypatch.setattr(runtime_mod, "build_vision_router", lambda: vision_router)
+ monkeypatch.setattr(runtime_mod, "MovementManager", lambda **kwargs: movement_manager)
+ monkeypatch.setattr(runtime_mod, "HeadWobbler", lambda **kwargs: head_wobbler)
+ return movement_manager, head_wobbler
+
+
+def test_connect_builds_workers_dependencies_and_status(monkeypatch: pytest.MonkeyPatch) -> None:
+ """Runtime construction should wire every robot-dependent service once."""
+ events: list[str] = []
+ camera_worker = _Component("camera", events)
+ head_tracker = object()
+ vision_router = object()
+ movement_manager, head_wobbler = _patch_runtime_factories(
+ monkeypatch,
+ events,
+ camera_worker=camera_worker,
+ head_tracker=head_tracker,
+ vision_router=vision_router,
+ )
+ robot = _Robot(
+ events,
+ {
+ "simulation_enabled": True,
+ "mockup_sim_enabled": False,
+ },
+ )
+
+ runtime = runtime_mod.ReachyRuntime.connect(
+ robot=robot, # type: ignore[arg-type]
+ no_camera=False,
+ head_tracker="mediapipe",
+ capture_directory=Path("~/reachy-captures"),
+ shutdown_delay_seconds=0,
+ )
+
+ assert runtime.robot is robot
+ assert runtime.camera_worker is camera_worker
+ assert runtime.head_tracker is head_tracker
+ assert runtime.movement_manager is movement_manager
+ assert runtime.head_wobbler is head_wobbler
+ assert runtime.vision_router is vision_router
+ assert runtime.is_simulation is True
+ assert runtime.dependencies.reachy_mini is robot
+ assert runtime.dependencies.movement_manager is movement_manager
+ assert runtime.dependencies.camera_worker is camera_worker
+ assert runtime.dependencies.vision_router is vision_router
+ assert runtime.dependencies.capture_directory == Path("~/reachy-captures").expanduser()
+
+
+def test_connect_constructs_robot_with_name(monkeypatch: pytest.MonkeyPatch) -> None:
+ """Standalone callers should be able to select the Reachy topic prefix."""
+ events: list[str] = []
+ robot = _Robot(events, {"simulation_enabled": False, "mockup_sim_enabled": False})
+ received_kwargs: dict[str, Any] = {}
+
+ def build_robot(**kwargs: Any) -> _Robot:
+ received_kwargs.update(kwargs)
+ return robot
+
+ monkeypatch.setattr(runtime_mod, "ReachyMini", build_robot)
+ _patch_runtime_factories(monkeypatch, events, camera_worker=None)
+
+ runtime = runtime_mod.ReachyRuntime.connect(
+ robot_name="test_reachy",
+ shutdown_delay_seconds=0,
+ )
+
+ assert runtime.robot is robot
+ assert received_kwargs == {"robot_name": "test_reachy"}
+ assert runtime.is_simulation is False
+
+
+def test_connect_passes_explicit_network_and_movement_settings(monkeypatch: pytest.MonkeyPatch) -> None:
+ """A standalone runtime should honor the configured robot address and lower send rate."""
+ events: list[str] = []
+ robot = _Robot(events, {})
+ received_robot_kwargs: dict[str, Any] = {}
+ received_movement_kwargs: dict[str, Any] = {}
+
+ def build_robot(**kwargs: Any) -> _Robot:
+ received_robot_kwargs.update(kwargs)
+ return robot
+
+ monkeypatch.setattr(runtime_mod, "ReachyMini", build_robot)
+ monkeypatch.setattr(runtime_mod, "handle_vision_stuff", lambda args, robot: (None, None, None))
+ monkeypatch.setattr(runtime_mod, "build_vision_router", lambda: None)
+ monkeypatch.setattr(
+ runtime_mod,
+ "MovementManager",
+ lambda **kwargs: received_movement_kwargs.update(kwargs) or _MovementManager("movement", events),
+ )
+ monkeypatch.setattr(runtime_mod, "HeadWobbler", lambda **kwargs: _Component("wobbler", events))
+
+ runtime_mod.ReachyRuntime.connect(
+ robot_host="192.168.0.107",
+ robot_port=8000,
+ connection_mode="network",
+ movement_frequency_hz=50,
+ enable_idle_breathing=False,
+ shutdown_delay_seconds=0,
+ )
+
+ assert received_robot_kwargs == {
+ "host": "192.168.0.107",
+ "port": 8000,
+ "connection_mode": "network",
+ }
+ assert received_movement_kwargs["target_frequency_hz"] == 50
+ assert received_movement_kwargs["enable_idle_breathing"] is False
+
+
+def test_connect_skips_cloud_router_for_local_vision(monkeypatch: pytest.MonkeyPatch) -> None:
+ """Explicit local vision should not also construct the routed cloud client."""
+ events: list[str] = []
+ camera_worker = _Component("camera", events)
+ vision_manager = _Component("vision", events)
+ router_builds = 0
+
+ monkeypatch.setattr(
+ runtime_mod,
+ "handle_vision_stuff",
+ lambda args, robot: (camera_worker, None, vision_manager),
+ )
+
+ def build_router() -> object:
+ nonlocal router_builds
+ router_builds += 1
+ return object()
+
+ monkeypatch.setattr(runtime_mod, "build_vision_router", build_router)
+ monkeypatch.setattr(runtime_mod, "MovementManager", lambda **kwargs: _MovementManager("movement", events))
+ monkeypatch.setattr(runtime_mod, "HeadWobbler", lambda **kwargs: _Component("wobbler", events))
+
+ runtime = runtime_mod.ReachyRuntime.connect(
+ robot=_Robot(events, {}), # type: ignore[arg-type]
+ local_vision=True,
+ shutdown_delay_seconds=0,
+ )
+
+ assert runtime.vision_manager is vision_manager
+ assert runtime.vision_router is None
+ assert router_builds == 0
+
+
+def test_connect_can_disable_cloud_router_for_camera_only_runtime(monkeypatch: pytest.MonkeyPatch) -> None:
+ """The local robot runtime should capture images without constructing an OpenAI client."""
+ events: list[str] = []
+ camera_worker = _Component("camera", events)
+ router_builds = 0
+
+ monkeypatch.setattr(
+ runtime_mod,
+ "handle_vision_stuff",
+ lambda args, robot: (camera_worker, None, None),
+ )
+
+ def build_router() -> object:
+ nonlocal router_builds
+ router_builds += 1
+ return object()
+
+ monkeypatch.setattr(runtime_mod, "build_vision_router", build_router)
+ monkeypatch.setattr(runtime_mod, "MovementManager", lambda **kwargs: _MovementManager("movement", events))
+ monkeypatch.setattr(runtime_mod, "HeadWobbler", lambda **kwargs: _Component("wobbler", events))
+
+ runtime = runtime_mod.ReachyRuntime.connect(
+ robot=_Robot(events, {}), # type: ignore[arg-type]
+ enable_vision_router=False,
+ shutdown_delay_seconds=0,
+ )
+
+ assert runtime.camera_worker is camera_worker
+ assert runtime.vision_router is None
+ assert runtime.dependencies.vision_router is None
+ assert router_builds == 0
+
+
+def test_start_and_stop_manage_the_complete_lifecycle_once() -> None:
+ """Lifecycle methods should be ordered and idempotent."""
+ events: list[str] = []
+ robot = _Robot(events, {})
+ movement_manager = _MovementManager("movement", events)
+ head_wobbler = _Component("wobbler", events)
+ camera_worker = _Component("camera", events)
+ vision_manager = _Component("vision", events)
+
+ runtime = runtime_mod.ReachyRuntime(
+ robot=robot, # type: ignore[arg-type]
+ camera_worker=camera_worker,
+ movement_manager=movement_manager, # type: ignore[arg-type]
+ dependencies=object(), # type: ignore[arg-type]
+ head_wobbler=head_wobbler, # type: ignore[arg-type]
+ vision_manager=vision_manager,
+ shutdown_delay_seconds=0,
+ )
+
+ runtime.start()
+ runtime.start()
+ runtime.stop()
+ runtime.stop()
+
+ assert events == [
+ "movement.start",
+ "wobbler.start",
+ "camera.start",
+ "vision.start",
+ "movement.stop",
+ "wobbler.stop",
+ "camera.stop",
+ "vision.stop",
+ "media.close",
+ "client.disconnect",
+ ]
+
+
+def test_start_cleans_up_already_started_components_on_failure() -> None:
+ """A partial start should stop components that were already running."""
+ events: list[str] = []
+ runtime = runtime_mod.ReachyRuntime(
+ robot=_Robot(events, {}), # type: ignore[arg-type]
+ camera_worker=_Component("camera", events),
+ movement_manager=_MovementManager("movement", events), # type: ignore[arg-type]
+ dependencies=object(), # type: ignore[arg-type]
+ head_wobbler=_Component("wobbler", events, fail_start=True), # type: ignore[arg-type]
+ shutdown_delay_seconds=0,
+ )
+
+ with pytest.raises(RuntimeError, match="wobbler failed"):
+ runtime.start()
+
+ assert events == [
+ "movement.start",
+ "wobbler.start",
+ "movement.stop",
+ ]
+
+
+def test_stopped_runtime_cannot_restart_disconnected_robot() -> None:
+ """Restart should fail after the SDK connection has been closed."""
+ events: list[str] = []
+ runtime = runtime_mod.ReachyRuntime(
+ robot=_Robot(events, {}), # type: ignore[arg-type]
+ camera_worker=None,
+ movement_manager=_MovementManager("movement", events), # type: ignore[arg-type]
+ dependencies=object(), # type: ignore[arg-type]
+ head_wobbler=_Component("wobbler", events), # type: ignore[arg-type]
+ shutdown_delay_seconds=0,
+ )
+
+ runtime.start()
+ runtime.stop()
+
+ with pytest.raises(RuntimeError, match="cannot be restarted"):
+ runtime.start()
diff --git a/projects/reachy-mini-openshell/tests/test_sandbox_audio.py b/projects/reachy-mini-openshell/tests/test_sandbox_audio.py
new file mode 100644
index 00000000..1f650ca8
--- /dev/null
+++ b/projects/reachy-mini-openshell/tests/test_sandbox_audio.py
@@ -0,0 +1,99 @@
+"""Tests for the sandbox audio WebSocket service."""
+
+# ruff: noqa: D101, D102, D103, D107
+
+import time
+import asyncio
+from typing import Any
+
+import numpy as np
+from fastrtc import AdditionalOutputs
+from fastapi.testclient import TestClient
+
+from reachy_mini_conversation_app.sandbox_audio import WIRE_SAMPLE_RATE, create_audio_app
+
+
+class FakeHandler:
+ def __init__(self, outputs: list[Any] | None = None) -> None:
+ self.outputs: asyncio.Queue[Any] = asyncio.Queue()
+ for output in outputs or []:
+ self.outputs.put_nowait(output)
+ self.received: list[tuple[int, np.ndarray[Any, Any]]] = []
+ self.stopped = asyncio.Event()
+ self.shutdown_called = False
+
+ async def start_up(self) -> None:
+ await self.stopped.wait()
+
+ async def receive(self, frame: tuple[int, np.ndarray[Any, Any]]) -> None:
+ self.received.append(frame)
+
+ async def emit(self) -> Any:
+ return await self.outputs.get()
+
+ async def shutdown(self) -> None:
+ self.shutdown_called = True
+ self.stopped.set()
+
+
+def hello() -> dict[str, Any]:
+ return {
+ "type": "hello",
+ "format": "pcm_s16le",
+ "sample_rate": WIRE_SAMPLE_RATE,
+ "channels": 1,
+ }
+
+
+def test_health_describes_audio_contract() -> None:
+ with TestClient(create_audio_app(lambda: FakeHandler())) as client:
+ response = client.get("/health")
+
+ assert response.status_code == 200
+ assert response.json() == {
+ "status": "ok",
+ "active_audio_client": False,
+ **hello(),
+ }
+
+
+def test_audio_websocket_bridges_pcm_and_text_outputs() -> None:
+ expected_audio = np.array([100, -100, 200, -200], dtype=np.int16)
+ handler = FakeHandler(
+ [
+ (WIRE_SAMPLE_RATE, expected_audio.reshape(1, -1)),
+ AdditionalOutputs({"role": "assistant", "content": "OpenShell blocked that movement."}),
+ ]
+ )
+ with TestClient(create_audio_app(lambda: handler)) as client:
+ with client.websocket_connect("/audio") as websocket:
+ websocket.send_json(hello())
+ assert websocket.receive_json()["type"] == "ready"
+
+ microphone_audio = np.array([1, -2, 3], dtype=" None:
+ with TestClient(create_audio_app(lambda: FakeHandler())) as client:
+ with client.websocket_connect("/audio") as websocket:
+ websocket.send_json({**hello(), "format": "float32"})
+ message = websocket.receive()
+
+ assert message["type"] == "websocket.close"
+ assert message["code"] == 1002
diff --git a/projects/reachy-mini-openshell/tests/test_sandbox_control.py b/projects/reachy-mini-openshell/tests/test_sandbox_control.py
new file mode 100644
index 00000000..2f54f1f4
--- /dev/null
+++ b/projects/reachy-mini-openshell/tests/test_sandbox_control.py
@@ -0,0 +1,79 @@
+"""Tests for in-sandbox lifecycle control."""
+
+# ruff: noqa: D103
+
+import signal
+from pathlib import Path
+
+from reachy_mini_conversation_app import sandbox_control
+from reachy_mini_conversation_app.sandbox_control import ControlSettings, SandboxAgentControl
+
+
+def settings(tmp_path: Path) -> ControlSettings:
+ return ControlSettings(
+ state_directory=tmp_path / "run",
+ log_directory=tmp_path / "logs",
+ startup_timeout_seconds=0.01,
+ shutdown_timeout_seconds=0.01,
+ )
+
+
+def test_status_reports_stopped_without_pid(tmp_path: Path) -> None:
+ assert SandboxAgentControl(settings(tmp_path)).status() == "stopped"
+
+
+def test_start_is_idempotent_when_agent_is_healthy(tmp_path: Path, monkeypatch) -> None:
+ configured = settings(tmp_path)
+ configured.state_directory.mkdir(parents=True)
+ configured.pid_path.write_text("123\n", encoding="utf-8")
+ monkeypatch.setattr(sandbox_control, "_process_exists", lambda pid: pid == 123)
+ monkeypatch.setattr(sandbox_control, "_port_is_listening", lambda port: port == 8765)
+
+ assert SandboxAgentControl(configured).start() == 0
+
+
+def test_port_is_listening_reads_linux_tcp_table(tmp_path: Path, monkeypatch) -> None:
+ tcp_table = tmp_path / "tcp"
+ tcp_table.write_text(
+ " sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt\n"
+ " 0: 0100007F:223D 00000000:0000 0A 00000000:00000000 00:00000000 00000000\n",
+ encoding="ascii",
+ )
+
+ real_path = sandbox_control.Path
+
+ def fake_path(value: str):
+ if value == "/proc/net/tcp":
+ return tcp_table
+ if value == "/proc/net/tcp6":
+ return tmp_path / "missing"
+ return real_path(value)
+
+ monkeypatch.setattr(sandbox_control, "Path", fake_path)
+
+ assert sandbox_control._port_is_listening(8765) is True
+ assert sandbox_control._port_is_listening(8766) is False
+
+
+def test_stop_terminates_process_group_and_removes_pid(tmp_path: Path, monkeypatch) -> None:
+ configured = settings(tmp_path)
+ configured.state_directory.mkdir(parents=True)
+ configured.pid_path.write_text("321\n", encoding="utf-8")
+ alive = True
+ signals: list[tuple[int, signal.Signals]] = []
+
+ def process_exists(pid: int) -> bool:
+ assert pid == 321
+ return alive
+
+ def killpg(pid: int, requested_signal: signal.Signals) -> None:
+ nonlocal alive
+ signals.append((pid, requested_signal))
+ alive = False
+
+ monkeypatch.setattr(sandbox_control, "_process_exists", process_exists)
+ monkeypatch.setattr(sandbox_control.os, "killpg", killpg)
+
+ assert SandboxAgentControl(configured).stop() == 0
+ assert signals == [(321, signal.SIGTERM)]
+ assert not configured.pid_path.exists()
diff --git a/projects/reachy-mini-openshell/tests/test_tool_transports.py b/projects/reachy-mini-openshell/tests/test_tool_transports.py
new file mode 100644
index 00000000..0618c017
--- /dev/null
+++ b/projects/reachy-mini-openshell/tests/test_tool_transports.py
@@ -0,0 +1,129 @@
+"""Tests for local and routed tool transports."""
+
+from __future__ import annotations
+from typing import Any, cast
+
+import pytest
+
+from reachy_mini_conversation_app.tool_transport import (
+ ToolTransport,
+ LocalToolTransport,
+ RoutedToolTransport,
+ ConversationUtilityTransport,
+)
+from reachy_mini_conversation_app.tools.core_tools import ToolDependencies
+from reachy_mini_conversation_app.tools.background_tool_manager import ToolCallRoutine, BackgroundToolManager
+
+
+class _RecordingTransport:
+ def __init__(self, tools: list[dict[str, Any]]) -> None:
+ self.tools = tools
+ self.calls: list[tuple[str, dict[str, Any]]] = []
+ self.close_calls = 0
+
+ async def list_tools(self) -> list[dict[str, Any]]:
+ return self.tools
+
+ async def call_tool(self, name: str, arguments: dict[str, Any]) -> dict[str, Any]:
+ self.calls.append((name, arguments))
+ return {"transport": "called", "name": name}
+
+ async def close(self) -> None:
+ self.close_calls += 1
+
+
+@pytest.mark.asyncio
+async def test_local_transport_lists_and_dispatches_existing_tools() -> None:
+ """Local mode should preserve dependency filtering and Python dispatch."""
+ dependencies = ToolDependencies(
+ reachy_mini=cast(Any, None),
+ movement_manager=object(),
+ camera_worker=None,
+ )
+ transport = LocalToolTransport(dependencies)
+
+ tools = await transport.list_tools()
+ result = await transport.call_tool("do_nothing", {"reason": "testing"})
+
+ assert isinstance(transport, ToolTransport)
+ assert "camera" not in {tool["name"] for tool in tools}
+ assert result == {"status": "doing nothing", "reason": "testing"}
+ await transport.close()
+
+
+@pytest.mark.asyncio
+async def test_conversation_utility_transport_does_not_load_robot_registry() -> None:
+ """REST utilities should remain usable without the native robot dependency graph."""
+ transport = ConversationUtilityTransport()
+
+ tools = await transport.list_tools()
+ result = await transport.call_tool("do_nothing", {"reason": "testing"})
+
+ assert {tool["name"] for tool in tools} == {"do_nothing", "task_status", "task_cancel"}
+ assert result == {"status": "doing nothing", "reason": "testing"}
+
+
+@pytest.mark.asyncio
+async def test_routed_transport_keeps_only_conversation_system_tools_local() -> None:
+ """Remote mode should route hardware remotely while retaining narrow local utilities."""
+ remote = _RecordingTransport(
+ [
+ {"type": "function", "name": "move_head"},
+ {"type": "function", "name": "stop_motion"},
+ {"type": "function", "name": "do_nothing", "source": "remote"},
+ ]
+ )
+ local = _RecordingTransport(
+ [
+ {"type": "function", "name": "camera"},
+ {"type": "function", "name": "do_nothing", "source": "local"},
+ {"type": "function", "name": "task_status"},
+ {"type": "function", "name": "task_cancel"},
+ ]
+ )
+ transport = RoutedToolTransport(remote=remote, local=local)
+
+ tools = await transport.list_tools()
+ remote_result = await transport.call_tool("move_head", {"directions": ["left"]})
+ local_result = await transport.call_tool("do_nothing", {"reason": "testing"})
+ await transport.close()
+
+ assert [tool["name"] for tool in tools] == [
+ "move_head",
+ "stop_motion",
+ "do_nothing",
+ "task_status",
+ "task_cancel",
+ ]
+ assert tools[2]["source"] == "local"
+ assert remote_result["name"] == "move_head"
+ assert local_result["name"] == "do_nothing"
+ assert remote.calls == [("move_head", {"directions": ["left"]})]
+ assert local.calls == [("do_nothing", {"reason": "testing"})]
+ assert remote.close_calls == 1
+ assert local.close_calls == 1
+
+
+@pytest.mark.asyncio
+async def test_tool_call_routine_uses_transport_but_keeps_manager_tools_local() -> None:
+ """Deferred hardware calls use the transport while task controls retain manager access."""
+ transport = _RecordingTransport([])
+ manager = BackgroundToolManager()
+ dependencies = ToolDependencies()
+
+ hardware_result = await ToolCallRoutine(
+ tool_name="move_head",
+ args_json_str='{"directions":["left"]}',
+ deps=dependencies,
+ transport=transport,
+ )(manager)
+ status_result = await ToolCallRoutine(
+ tool_name="task_status",
+ args_json_str="{}",
+ deps=dependencies,
+ transport=transport,
+ )(manager)
+
+ assert hardware_result == {"transport": "called", "name": "move_head"}
+ assert transport.calls == [("move_head", {"directions": ["left"]})]
+ assert status_result == {"status": "idle", "message": "No tools running in the background."}
diff --git a/projects/reachy-mini-openshell/tests/test_vision_router.py b/projects/reachy-mini-openshell/tests/test_vision_router.py
new file mode 100644
index 00000000..067467d4
--- /dev/null
+++ b/projects/reachy-mini-openshell/tests/test_vision_router.py
@@ -0,0 +1,133 @@
+"""Tests for single-model camera and scene-scan routing."""
+
+from typing import Any
+
+import pytest
+
+from reachy_mini_conversation_app.vision_router import VisionRouter
+
+
+class _FakeUsage:
+ def model_dump(self, mode: str = "python") -> dict[str, int]:
+ assert mode in {"python", "json"}
+ return {"input_tokens": 42, "output_tokens": 12, "total_tokens": 54}
+
+
+class _FakeResponse:
+ id = "resp_vision_123"
+ output_text = "The person is sitting at a desk."
+ usage = _FakeUsage()
+
+
+class _FakeResponses:
+ def __init__(self) -> None:
+ self.calls: list[dict[str, Any]] = []
+
+ async def create(self, **kwargs: Any) -> _FakeResponse:
+ self.calls.append(kwargs)
+ return _FakeResponse()
+
+
+class _FakeClient:
+ def __init__(self) -> None:
+ self.responses = _FakeResponses()
+
+
+@pytest.mark.asyncio
+async def test_router_sends_one_camera_frame_to_the_only_approved_model() -> None:
+ """One camera frame should upload once and return no source image bytes."""
+ client = _FakeClient()
+ router = VisionRouter(
+ client=client,
+ default_model="gpt-5.4-mini",
+ allowed_models=("gpt-5.4-mini",),
+ )
+
+ analysis = await router.analyze_images(
+ images_base64=["jpeg-base64-data"],
+ question="Use gpt-5.5 to tell me what the person is doing.",
+ )
+ result = analysis.as_tool_result()
+
+ assert len(client.responses.calls) == 1
+ assert client.responses.calls[0]["model"] == "gpt-5.4-mini"
+ assert client.responses.calls[0]["input"][0]["content"] == [
+ {"type": "input_text", "text": "Use gpt-5.5 to tell me what the person is doing."},
+ {
+ "type": "input_image",
+ "image_url": "data:image/jpeg;base64,jpeg-base64-data",
+ },
+ ]
+ assert result["selected_model"] == "gpt-5.4-mini"
+ assert result["image_description"] == "The person is sitting at a desk."
+ assert result["usage"]["total_tokens"] == 54
+ assert "b64_im" not in result
+ assert "requested_model" not in result
+ assert "jpeg-base64-data" not in str(result)
+
+
+@pytest.mark.asyncio
+async def test_router_sends_nine_ordered_frames_in_one_response_request() -> None:
+ """A scene scan should become one request with frames preserved in chronological order."""
+ client = _FakeClient()
+ router = VisionRouter(
+ client=client,
+ default_model="gpt-5.4-mini",
+ allowed_models=("gpt-5.4-mini",),
+ )
+ images = [f"jpeg-{index}" for index in range(9)]
+ timestamps = [float(index) for index in range(9)]
+
+ analysis = await router.analyze_images(
+ images_base64=images,
+ question="What did Reachy see?",
+ frame_timestamps=timestamps,
+ )
+
+ assert len(client.responses.calls) == 1
+ request = client.responses.calls[0]
+ assert request["model"] == "gpt-5.4-mini"
+ content = request["input"][0]["content"]
+ assert "deduplicate" in content[0]["text"]
+ assert str(timestamps) in content[0]["text"]
+ assert [item["image_url"] for item in content[1:]] == [f"data:image/jpeg;base64,{image}" for image in images]
+ assert analysis.selected_model == "gpt-5.4-mini"
+
+
+@pytest.mark.asyncio
+async def test_router_rejects_more_than_nine_images_before_upload() -> None:
+ """An oversized frame set must fail before the client receives any images."""
+ client = _FakeClient()
+ router = VisionRouter(
+ client=client,
+ default_model="gpt-5.4-mini",
+ allowed_models=("gpt-5.4-mini",),
+ )
+
+ with pytest.raises(ValueError, match="between 1 and 9"):
+ await router.analyze_images(
+ images_base64=["must-not-upload"] * 10,
+ question="Describe this image.",
+ )
+
+ assert client.responses.calls == []
+
+
+def test_router_requires_exactly_one_allowed_model() -> None:
+ """Multiple allowed models would reintroduce a user-selectable routing path."""
+ with pytest.raises(ValueError, match="exactly one model"):
+ VisionRouter(
+ client=_FakeClient(),
+ default_model="gpt-5.4-mini",
+ allowed_models=("gpt-5.4-mini", "gpt-5.5"),
+ )
+
+
+def test_router_requires_default_model_to_be_allowed() -> None:
+ """An invalid routing policy should fail during startup."""
+ with pytest.raises(ValueError, match="must equal the only"):
+ VisionRouter(
+ client=_FakeClient(),
+ default_model="gpt-5.5",
+ allowed_models=("gpt-5.4-mini",),
+ )
diff --git a/projects/reachy-mini-openshell/tests/tools/test_background_tool_manager.py b/projects/reachy-mini-openshell/tests/tools/test_background_tool_manager.py
index 2d0d41d6..d7ee29b5 100644
--- a/projects/reachy-mini-openshell/tests/tools/test_background_tool_manager.py
+++ b/projects/reachy-mini-openshell/tests/tools/test_background_tool_manager.py
@@ -572,3 +572,4 @@ async def test_notifications_queued_on_failure(self, manager: BackgroundToolMana
n = manager._notification_queue.get_nowait()
assert n.status == ToolState.FAILED
assert "RuntimeError: oops" in (n.error or "")
+ assert n.result == {"error": "RuntimeError: oops"}
diff --git a/projects/reachy-mini-openshell/tests/tools/test_camera.py b/projects/reachy-mini-openshell/tests/tools/test_camera.py
new file mode 100644
index 00000000..91877f61
--- /dev/null
+++ b/projects/reachy-mini-openshell/tests/tools/test_camera.py
@@ -0,0 +1,89 @@
+"""Tests for the camera tool's routed vision behavior."""
+
+from typing import Any
+from unittest.mock import MagicMock
+
+import numpy as np
+import pytest
+
+from reachy_mini_conversation_app.tools.camera import Camera
+from reachy_mini_conversation_app.vision_router import VisionAnalysis
+from reachy_mini_conversation_app.tools.core_tools import ToolDependencies
+
+
+class _FakeCameraWorker:
+ def get_latest_frame(self) -> np.ndarray[Any, Any]:
+ return np.zeros((8, 8, 3), dtype=np.uint8)
+
+
+class _FakeVisionRouter:
+ def __init__(self) -> None:
+ self.calls: list[dict[str, Any]] = []
+
+ async def analyze_images(self, **kwargs: Any) -> VisionAnalysis:
+ self.calls.append(kwargs)
+ return VisionAnalysis(
+ description="The person is waving.",
+ selected_model="gpt-5.4-mini",
+ response_id="resp_camera",
+ usage={"total_tokens": 20},
+ )
+
+
+@pytest.mark.asyncio
+async def test_camera_sends_one_image_without_a_model_selection() -> None:
+ """The tool should send one frame while leaving model selection to the router."""
+ router = _FakeVisionRouter()
+ deps = ToolDependencies(
+ reachy_mini=MagicMock(),
+ movement_manager=MagicMock(),
+ camera_worker=_FakeCameraWorker(),
+ vision_router=router,
+ )
+
+ result = await Camera()(deps, question="What am I doing?")
+
+ assert len(router.calls[0]["images_base64"]) == 1
+ assert router.calls[0]["question"] == "What am I doing?"
+ assert "requested_model" not in router.calls[0]
+ assert result["status"] == "image_analyzed"
+ assert result["selected_model"] == "gpt-5.4-mini"
+ assert "b64_im" not in result
+
+
+@pytest.mark.asyncio
+async def test_camera_schema_and_runtime_ignore_user_model_selection() -> None:
+ """Neither the public schema nor direct kwargs should provide a model override."""
+ router = _FakeVisionRouter()
+ deps = ToolDependencies(
+ reachy_mini=MagicMock(),
+ movement_manager=MagicMock(),
+ camera_worker=_FakeCameraWorker(),
+ vision_router=router,
+ )
+
+ result = await Camera()(deps, question="Use gpt-5.5.", requested_model="gpt-5.5")
+
+ properties = Camera.parameters_schema["properties"]
+ assert isinstance(properties, dict)
+ assert "requested_model" not in properties
+ assert Camera.parameters_schema["additionalProperties"] is False
+ assert "requested_model" not in router.calls[0]
+ assert result["selected_model"] == "gpt-5.4-mini"
+ assert "b64_im" not in result
+
+
+@pytest.mark.asyncio
+async def test_camera_without_a_router_returns_raw_media_for_the_internal_processor() -> None:
+ """The unprocessed path should return an internal image for MediaResultProcessor."""
+ deps = ToolDependencies(
+ reachy_mini=MagicMock(),
+ movement_manager=MagicMock(),
+ camera_worker=_FakeCameraWorker(),
+ )
+
+ result = await Camera()(deps, question="What am I doing?")
+
+ assert result["question"] == "What am I doing?"
+ assert isinstance(result["b64_im"], str)
+ assert result["b64_im"]
diff --git a/projects/reachy-mini-openshell/tests/tools/test_core_tools.py b/projects/reachy-mini-openshell/tests/tools/test_core_tools.py
new file mode 100644
index 00000000..58376ed0
--- /dev/null
+++ b/projects/reachy-mini-openshell/tests/tools/test_core_tools.py
@@ -0,0 +1,43 @@
+"""Tests for profile tool registration and dependency-aware exposure."""
+
+from typing import Any, cast
+
+from reachy_mini_conversation_app.tools.core_tools import (
+ ToolDependencies,
+ get_tool_specs,
+ get_tool_specs_for_dependencies,
+)
+
+
+def _tool_names(specs: list[dict[str, object]]) -> set[object]:
+ return {spec.get("name") for spec in specs}
+
+
+def test_requested_profile_tools_are_registered() -> None:
+ """The locked profile exposes the newly enabled tools."""
+ names = _tool_names(cast(list[dict[str, object]], get_tool_specs()))
+
+ assert {"camera", "scan_scene", "move_head", "do_nothing"} <= names
+
+
+def test_camera_tool_requires_a_camera_worker() -> None:
+ """Models should not see the camera tool when the app disabled camera support."""
+ without_camera = ToolDependencies(
+ reachy_mini=cast(Any, None),
+ movement_manager=object(),
+ camera_worker=None,
+ )
+ with_camera = ToolDependencies(
+ reachy_mini=cast(Any, None),
+ movement_manager=object(),
+ camera_worker=object(),
+ )
+
+ without_names = _tool_names(cast(list[dict[str, object]], get_tool_specs_for_dependencies(without_camera)))
+ with_names = _tool_names(cast(list[dict[str, object]], get_tool_specs_for_dependencies(with_camera)))
+
+ assert "camera" not in without_names
+ assert "scan_scene" not in without_names
+ assert {"move_head", "do_nothing"} <= without_names
+ assert "camera" in with_names
+ assert "scan_scene" in with_names
diff --git a/projects/reachy-mini-openshell/tests/tools/test_move_head.py b/projects/reachy-mini-openshell/tests/tools/test_move_head.py
new file mode 100644
index 00000000..bac9f0b7
--- /dev/null
+++ b/projects/reachy-mini-openshell/tests/tools/test_move_head.py
@@ -0,0 +1,79 @@
+"""Tests for ordered Reachy Mini head movements."""
+
+from unittest.mock import MagicMock
+
+import numpy as np
+import pytest
+
+import reachy_mini_conversation_app.tools.core_tools as core_tools
+from reachy_mini_conversation_app.tools.move_head import MoveHead
+
+
+def _dependencies() -> tuple[core_tools.ToolDependencies, MagicMock]:
+ reachy = MagicMock()
+ reachy.get_current_head_pose.return_value = np.eye(4)
+ reachy.get_current_joint_positions.return_value = ([0.25], [0.1, -0.1])
+ movement_manager = MagicMock()
+ deps = core_tools.ToolDependencies(
+ reachy_mini=reachy,
+ movement_manager=movement_manager,
+ motion_duration_s=0.5,
+ )
+ return deps, movement_manager
+
+
+def test_move_head_schema_requests_an_ordered_direction_list() -> None:
+ """The model should receive a schema capable of representing a sequence."""
+ schema = MoveHead.parameters_schema
+
+ assert schema["required"] == ["directions"]
+ assert schema["properties"]["directions"]["type"] == "array"
+ assert schema["properties"]["directions"]["items"]["enum"] == [
+ "left",
+ "right",
+ "up",
+ "down",
+ "front",
+ ]
+
+
+@pytest.mark.asyncio
+async def test_move_head_queues_multiple_directions_in_order() -> None:
+ """A compound request should become one deterministic queued sequence."""
+ deps, movement_manager = _dependencies()
+
+ result = await MoveHead()(deps, directions=["up", "right"])
+
+ queued_moves = [call.args[0] for call in movement_manager.queue_move.call_args_list]
+ assert len(queued_moves) == 2
+ assert np.array_equal(queued_moves[0].start_head_pose, np.eye(4))
+ assert np.array_equal(queued_moves[1].start_head_pose, queued_moves[0].target_head_pose)
+ assert not np.array_equal(queued_moves[0].target_head_pose, queued_moves[1].target_head_pose)
+ movement_manager.set_moving_state.assert_called_once_with(1.0)
+ assert result == {
+ "status": "queued",
+ "directions": ["up", "right"],
+ "total_duration_seconds": 1.0,
+ }
+
+
+@pytest.mark.asyncio
+async def test_move_head_accepts_legacy_single_direction() -> None:
+ """Existing direct callers using direction= remain compatible."""
+ deps, movement_manager = _dependencies()
+
+ result = await MoveHead()(deps, direction="left")
+
+ assert movement_manager.queue_move.call_count == 1
+ assert result["directions"] == ["left"]
+
+
+@pytest.mark.asyncio
+async def test_move_head_rejects_invalid_sequence() -> None:
+ """Invalid directions should not enqueue partial motion."""
+ deps, movement_manager = _dependencies()
+
+ result = await MoveHead()(deps, directions=["up", "backwards"])
+
+ assert "error" in result
+ movement_manager.queue_move.assert_not_called()
diff --git a/projects/reachy-mini-openshell/tests/tools/test_scan_scene.py b/projects/reachy-mini-openshell/tests/tools/test_scan_scene.py
new file mode 100644
index 00000000..2a770481
--- /dev/null
+++ b/projects/reachy-mini-openshell/tests/tools/test_scan_scene.py
@@ -0,0 +1,206 @@
+"""Tests for synchronized Reachy scene scanning and recording."""
+
+import base64
+from typing import Any
+from pathlib import Path
+from unittest.mock import MagicMock
+
+import numpy as np
+import pytest
+from numpy.typing import NDArray
+
+import reachy_mini_conversation_app.tools.core_tools as core_tools
+import reachy_mini_conversation_app.profiles._reachy_mini_conversation_app_locked_profile.scan_scene as scan_mod
+import reachy_mini_conversation_app.profiles._reachy_mini_conversation_app_locked_profile.sweep_look as sweep_mod
+
+
+ToolDependencies = core_tools.ToolDependencies
+
+
+class _FakeCameraWorker:
+ def __init__(self, frame: NDArray[np.uint8] | None) -> None:
+ self.frame = frame
+ self.is_head_tracking_enabled = True
+ self.tracking_changes: list[bool] = []
+ self.offsets_cleared = False
+
+ def get_latest_frame(self) -> NDArray[np.uint8] | None:
+ return None if self.frame is None else self.frame.copy()
+
+ def set_head_tracking_enabled(self, enabled: bool) -> None:
+ self.tracking_changes.append(enabled)
+ self.is_head_tracking_enabled = enabled
+
+ def clear_face_tracking_offsets(self) -> None:
+ self.offsets_cleared = True
+
+
+class _FakeVideoWriter:
+ def __init__(self, path: Path) -> None:
+ self.path = path
+ path.touch()
+ self.frames: list[NDArray[np.uint8]] = []
+ self.released = False
+
+ def isOpened(self) -> bool:
+ return True
+
+ def write(self, frame: NDArray[np.uint8]) -> None:
+ self.frames.append(frame.copy())
+
+ def release(self) -> None:
+ self.released = True
+
+
+def test_scan_scene_schema_requires_an_analysis_question() -> None:
+ """The model should state what it wants determined from the complete scan."""
+ schema = scan_mod.ScanScene.parameters_schema
+
+ assert schema["required"] == ["question"]
+ assert schema["properties"]["question"]["type"] == "string"
+
+
+@pytest.mark.asyncio
+async def test_sweep_look_uses_absolute_left_right_and_front_body_targets() -> None:
+ """A scan that begins off-center must still end at absolute front."""
+ movement_manager = MagicMock()
+ robot = MagicMock()
+ robot.get_current_head_pose.return_value = np.eye(4)
+ robot.get_current_joint_positions.return_value = ([0.6, *([0.0] * 6)], [0.1, -0.1])
+ deps = ToolDependencies(reachy_mini=robot, movement_manager=movement_manager)
+
+ await sweep_mod.SweepLook()(deps)
+
+ queued_moves = [call.args[0] for call in movement_manager.queue_move.call_args_list]
+ max_angle = sweep_mod.SWEEP_MAX_ANGLE_RADIANS
+ assert [move.target_body_yaw for move in queued_moves] == [
+ max_angle,
+ max_angle,
+ 0,
+ -max_angle,
+ -max_angle,
+ 0,
+ ]
+ assert queued_moves[0].start_body_yaw == 0.6
+ assert queued_moves[-1].target_body_yaw == 0
+
+
+@pytest.mark.asyncio
+async def test_scan_scene_records_video_and_returns_chronological_frames(
+ monkeypatch: Any,
+ tmp_path: Path,
+) -> None:
+ """One tool call should coordinate motion, recording, and model-ready frames."""
+ frame = np.zeros((48, 64, 3), dtype=np.uint8)
+ frame[:, 16:32] = 255
+ camera_worker = _FakeCameraWorker(frame)
+ movement_manager = MagicMock()
+ deps = ToolDependencies(
+ reachy_mini=MagicMock(),
+ movement_manager=movement_manager,
+ camera_worker=camera_worker,
+ capture_directory=tmp_path,
+ )
+
+ writer_holder: list[_FakeVideoWriter] = []
+
+ def fake_open_writer(path: Path, _frame: NDArray[np.uint8]) -> _FakeVideoWriter:
+ writer = _FakeVideoWriter(path)
+ writer_holder.append(writer)
+ return writer
+
+ sweep_calls: list[ToolDependencies] = []
+
+ async def fake_sweep(_self: Any, sweep_deps: ToolDependencies, **_kwargs: Any) -> dict[str, str]:
+ sweep_calls.append(sweep_deps)
+ return {"status": "queued"}
+
+ monkeypatch.setattr(scan_mod, "_open_video_writer", fake_open_writer)
+ monkeypatch.setattr(scan_mod.SweepLook, "__call__", fake_sweep)
+ monkeypatch.setattr(scan_mod, "SWEEP_TOTAL_DURATION_SECONDS", 0.12)
+ monkeypatch.setattr(scan_mod, "SWEEP_RECORDING_SETTLE_SECONDS", 0.0)
+ monkeypatch.setattr(scan_mod, "CAPTURE_FPS", 60.0)
+ monkeypatch.setattr(scan_mod, "MAX_ANALYSIS_FRAMES", 3)
+
+ result = await scan_mod.ScanScene()(deps, question="What people and objects did you see?")
+
+ assert result["status"] == "scene_scan_complete"
+ assert result["question"] == "What people and objects did you see?"
+ assert result["frames_recorded"] >= 3
+ assert result["frames_selected"] == 3
+ assert len(result["frame_timestamps_seconds"]) == 3
+ assert len(result["b64_images"]) == 3
+ assert all(base64.b64decode(image) for image in result["b64_images"])
+ assert Path(result["video_path"]).is_file()
+ assert sweep_calls == [deps]
+ assert writer_holder[0].released is True
+ assert len(writer_holder[0].frames) == result["frames_recorded"]
+ assert camera_worker.offsets_cleared is True
+ assert camera_worker.tracking_changes == [False, True]
+
+
+@pytest.mark.asyncio
+async def test_scan_scene_fails_before_moving_when_no_camera_frame(
+ monkeypatch: Any,
+ tmp_path: Path,
+) -> None:
+ """A missing camera feed must not start physical movement."""
+ camera_worker = _FakeCameraWorker(None)
+ movement_manager = MagicMock()
+ deps = ToolDependencies(
+ reachy_mini=MagicMock(),
+ movement_manager=movement_manager,
+ camera_worker=camera_worker,
+ capture_directory=tmp_path,
+ )
+ monkeypatch.setattr(scan_mod, "FRAME_WAIT_TIMEOUT_SECONDS", 0.01)
+
+ result = await scan_mod.ScanScene()(deps, question="What do you see?")
+
+ assert result == {"error": "No frame available from camera worker"}
+ movement_manager.queue_move.assert_not_called()
+ assert not list(tmp_path.iterdir())
+
+
+@pytest.mark.asyncio
+async def test_scan_scene_stops_motion_and_removes_partial_video_on_recording_failure(
+ monkeypatch: Any,
+ tmp_path: Path,
+) -> None:
+ """Recording failures should restore tracking and stop the in-progress sweep."""
+ frame = np.zeros((48, 64, 3), dtype=np.uint8)
+ camera_worker = _FakeCameraWorker(frame)
+ movement_manager = MagicMock()
+ deps = ToolDependencies(
+ reachy_mini=MagicMock(),
+ movement_manager=movement_manager,
+ camera_worker=camera_worker,
+ capture_directory=tmp_path,
+ )
+
+ class FailingWriter(_FakeVideoWriter):
+ def write(self, frame: NDArray[np.uint8]) -> None:
+ raise RuntimeError("disk full")
+
+ writer_holder: list[FailingWriter] = []
+
+ def fake_open_writer(path: Path, _frame: NDArray[np.uint8]) -> FailingWriter:
+ writer = FailingWriter(path)
+ writer_holder.append(writer)
+ return writer
+
+ async def fake_sweep(_self: Any, _deps: ToolDependencies, **_kwargs: Any) -> dict[str, str]:
+ return {"status": "queued"}
+
+ monkeypatch.setattr(scan_mod, "_open_video_writer", fake_open_writer)
+ monkeypatch.setattr(scan_mod.SweepLook, "__call__", fake_sweep)
+ monkeypatch.setattr(scan_mod, "SWEEP_TOTAL_DURATION_SECONDS", 0.1)
+ monkeypatch.setattr(scan_mod, "SWEEP_RECORDING_SETTLE_SECONDS", 0.0)
+
+ with pytest.raises(RuntimeError, match="disk full"):
+ await scan_mod.ScanScene()(deps, question="What do you see?")
+
+ movement_manager.clear_move_queue.assert_called_once_with()
+ assert writer_holder[0].released is True
+ assert not writer_holder[0].path.exists()
+ assert camera_worker.tracking_changes == [False, True]
diff --git a/projects/reachy-mini-openshell/uv.lock b/projects/reachy-mini-openshell/uv.lock
index 15530f52..c64f162c 100644
--- a/projects/reachy-mini-openshell/uv.lock
+++ b/projects/reachy-mini-openshell/uv.lock
@@ -2135,6 +2135,7 @@ dependencies = [
{ name = "fastrtc", marker = "sys_platform == 'darwin'" },
{ name = "gradio", marker = "sys_platform == 'darwin'" },
{ name = "gradio-client", marker = "sys_platform == 'darwin'" },
+ { name = "httpx", marker = "sys_platform == 'darwin'" },
{ name = "huggingface-hub", marker = "sys_platform == 'darwin'" },
{ name = "openai", marker = "sys_platform == 'darwin'" },
{ name = "opencv-python", marker = "sys_platform == 'darwin'" },
@@ -2188,6 +2189,7 @@ requires-dist = [
{ name = "fastrtc", specifier = ">=0.0.34" },
{ name = "gradio", specifier = "==5.50.1.dev1" },
{ name = "gradio-client", specifier = ">=1.13.3" },
+ { name = "httpx", specifier = ">=0.27" },
{ name = "huggingface-hub", specifier = "==1.3.0" },
{ name = "mediapipe", marker = "extra == 'all-vision'", specifier = "==0.10.14" },
{ name = "mediapipe", marker = "extra == 'mediapipe-vision'", specifier = "==0.10.14" },
diff --git a/scripts/render-dev-notes.py b/scripts/render-dev-notes.py
index 4f9cdb66..3f4232dd 100644
--- a/scripts/render-dev-notes.py
+++ b/scripts/render-dev-notes.py
@@ -199,12 +199,39 @@ def card_visual_class(post: dict[str, Any]) -> str:
return slug or "research"
-def render_card_visual(post: dict[str, Any]) -> str:
+def card_hero_image_url(post: dict[str, Any]) -> str | None:
+ """Resolve an optional post-relative hero image for the Dev Notes index."""
+ raw_path = str(post["metadata"].get("hero_image", "")).strip()
+ if not raw_path:
+ return None
+
+ docs_root = (ROOT / "docs").resolve()
+ hero_path = (post["path"].parent / raw_path).resolve()
+ try:
+ docs_relative = hero_path.relative_to(docs_root)
+ except ValueError as exc:
+ raise ValueError(f"{post['path']} hero_image must stay inside {docs_root}") from exc
+ if not hero_path.is_file():
+ raise ValueError(f"{post['path']} hero_image does not exist: {hero_path}")
+
+ return (Path("..") / docs_relative).as_posix()
+
+
+def render_card_visual(post: dict[str, Any], *, eager: bool = False) -> str:
metadata = post["metadata"]
+ variant = html.escape(card_visual_class(post), quote=True)
+ hero_image = card_hero_image_url(post)
+ if hero_image:
+ loading = "eager" if eager else "lazy"
+ priority = ' fetchpriority="high"' if eager else ""
+ return f"""
+ 
-
- Process
+
+ Edge AI
- Making Dev Notes Repeatable
-A repeatable author and post workflow keeps Dev Notes easy to extend as the research log grows.
+Bringing Privacy and Security to the Edge with OpenShell
+Edge agents handle sensitive data and make decisions with physical consequences. Reachy Mini shows why privacy and safety controls must be deterministic and local.
- dev-notes
- authors
- workflow
+ edge-ai
+ reachy-mini
+ policy
@@ -61,43 +59,5 @@ hide:
-
-
diff --git a/docs/dev-notes/posts/2026-06-04-bootstrapping-openshell-research.md b/docs/dev-notes/posts/2026-06-04-bootstrapping-openshell-research.md
deleted file mode 100644
index fd1121ee..00000000
--- a/docs/dev-notes/posts/2026-06-04-bootstrapping-openshell-research.md
+++ /dev/null
@@ -1,60 +0,0 @@
----
-title: Bootstrapping OpenShell Research
-date: 2026-06-04
-updated: 2026-06-05
-description: Initial setup for OpenShell Research documentation and Dev Notes.
-card_variant: launch
-categories:
- - Runtime
-tags:
- - openshell
- - research-engineering
- - docs
- - ci
-authors:
- - johnnygreco
- - zredlined
-card_tags:
- - openshell
- - docs
- - ci
----
-
-# Bootstrapping OpenShell Research
-
-
-
-
-
-
- Recent notes
- The working archive -
- 
-
-
-
-
-OpenShell Research starts as a shared engineering home for applications that
-turn current research into working systems on the OpenShell runtime.
-
-
-
-The first version of this repository establishes the documentation structure,
-the Dev Notes section, and CI verification for the generated Zensical site. The
-next useful step is to add the first runnable application skeleton and document
-the runtime assumptions it depends on.
diff --git a/docs/dev-notes/posts/2026-06-05-making-dev-notes-repeatable.md b/docs/dev-notes/posts/2026-06-05-making-dev-notes-repeatable.md
deleted file mode 100644
index 7dfb71de..00000000
--- a/docs/dev-notes/posts/2026-06-05-making-dev-notes-repeatable.md
+++ /dev/null
@@ -1,49 +0,0 @@
----
-title: Making Dev Notes Repeatable
-date: 2026-06-05
-updated: 2026-06-05
-description: A repeatable author and post workflow keeps Dev Notes easy to extend as the research log grows.
-categories:
- - Process
-tags:
- - dev-notes
- - authors
- - workflow
-authors:
- - kirit93
-card_tags:
- - dev-notes
- - authors
- - workflow
----
-
-# Making Dev Notes Repeatable
-
-
-
-- Field note - - Runtime -
- -
- 
-
-
-
-
-The Dev Notes section should stay easy to grow: add a post, name its authors,
-and let the documentation build keep the landing page and navigation aligned.
-
-
-
-This pass makes author metadata reusable across posts and keeps generated
-elements inside clearly marked regions. The goal is to make each new update feel
-like writing a note, not maintaining a miniature website by hand.
diff --git a/docs/dev-notes/posts/2026-07-20-policy-controlling-reachy-mini-with-openshell.md b/docs/dev-notes/posts/2026-07-20-policy-controlling-reachy-mini-with-openshell.md
new file mode 100644
index 00000000..64f88af8
--- /dev/null
+++ b/docs/dev-notes/posts/2026-07-20-policy-controlling-reachy-mini-with-openshell.md
@@ -0,0 +1,272 @@
+---
+title: "Bringing Privacy and Security to the Edge with OpenShell"
+date: 2026-07-20
+updated: 2026-07-20
+description: "Edge agents handle sensitive data and make decisions with physical consequences. Reachy Mini shows why privacy and safety controls must be deterministic and local."
+hero_image: "../../assets/reachy-mini-openshell/hero.svg"
+categories:
+ - Edge AI
+tags:
+ - reachy-mini
+ - openshell
+ - edge-ai
+ - sandbox
+ - robotics
+ - policy
+authors:
+ - kirit93
+card_tags:
+ - edge-ai
+ - reachy-mini
+ - policy
+---
+
+# Bringing Privacy and Security to the Edge with OpenShell
+
+
+
+- Field note - - Process -
- -
-
+
+
+
++ Field note + + Edge AI +
+ +
+
+
+
+OpenShell sandbox"] + A -->|"Realtime WebSocket"| O["OpenAI Realtime"] + A -->|"Motion REST"| P["OpenShell network policy"] + P -->|"host.openshell.internal:8000"| D["Reachy Mini daemon"] + D --> H["Head motors"] + A -->|"POST /camera/capture"| P + P -->|"host.openshell.internal:8042"| N + N --> C["Reachy camera"] +``` + +The sandbox does not start the Reachy SDK, movement manager, camera worker, or +vision router. The trusted native Reachy App owns the SDK media object and only +bridges audio plus one bounded JPEG capture operation. + +## Security boundary + +OpenShell REST rules can match: + +- Calling binary +- Destination host and port +- HTTP method +- URL path +- Query parameters + +OpenShell can therefore allow or deny: + +```text +POST /api/move/goto +POST /camera/capture +``` + +It does not currently enforce arbitrary JSON values inside that REST request. +Once `/api/move/goto` is allowed, OpenShell cannot prove that the body contains +only a head pose or distinguish `up` from `down`. + +The application reduces normal model behavior to fixed values: + +| Direction | Pitch | Yaw | +| --- | ---: | ---: | +| `up` | -30 degrees | 0 degrees | +| `down` | 30 degrees | 0 degrees | +| `left` | 0 degrees | 40 degrees | +| `right` | 0 degrees | -40 degrees | +| `front` | 0 degrees | 0 degrees | + +It also fixes duration to one second, uses `minjerk`, and omits antennas and +body yaw. These body constraints are application validation, not OpenShell +policy enforcement. + +Camera capture has a tighter adapter boundary: the request has no body fields +or query parameters. The native adapter chooses the already-open Reachy camera, +captures one JPEG, never writes it to disk, limits the response to 2 MiB, and +rate-limits calls. OpenShell still enforces the calling binary, host, port, +method, and exact path; the adapter enforces the capture semantics. + +## Requirements and resource budget + +The proven environment was a Reachy Mini Wireless running ARM64 Debian, Reachy +Mini 1.8.3, 3.7 GiB usable RAM, 2 GiB swap, and a 14 GiB root filesystem. + +| Resource | Requirement for this setup | +| --- | --- | +| Development machine | Git, Docker Buildx, Python 3.10–3.12, and `uv` | +| Reachy architecture | `aarch64`; the container is built for `linux/arm64` | +| Reachy RAM | A 4 GB unit is known to work. The sandbox has a 2 GiB ceiling and the host retains the remaining memory. A 2 GB device is unvalidated. | +| Reachy CPU | The sandbox is limited to 2 CPUs. Build the image off-device. | +| Reachy disk | Start with at least 4 GiB free; 5 GiB is preferred. Retain 1–2 GiB free after cleanup. | +| Network | Reachy must reach the configured Realtime API and must be reachable over SSH during installation. | + +The expanded sandbox image is approximately 339 MB. Installation needs more +temporary space because the compressed archive, expanded image, Docker layer +extraction, Docker/OpenShell packages, and `/venvs/apps_venv` may coexist. The +native controller wheel is only about 13 KB, but creating the shared Reachy Apps +environment and installing its Reachy SDK dependencies can require roughly +1–1.5 GiB if that environment does not already exist. + +The model runs remotely, so no local model weights or GPU are required. The +`--memory 2Gi` value used later is a limit, not a claim that the agent +continuously consumes 2 GiB. + +## Prepare the development checkout (development machine) + +```bash +git clone git@github.com:NVIDIA/OpenShell-Research.git +cd OpenShell-Research +git switch kirit93/reachy-implementation +cd projects/reachy-mini-openshell +``` + +Run the relevant checks before building deployable artifacts: + +```bash +uv run ruff check src tests +uv run pytest -q +PYTHONPATH=native-controller/src uv run pytest -q native-controller/tests +``` + +## Verify and prepare Reachy (Reachy) + +Connect and inspect the robot before installing anything: + +```bash +ssh pollen@reachy-mini.local + +uname -m +free -h +df -h / +curl --silent --show-error http://127.0.0.1:8000/api/daemon/status +``` + +Require `aarch64`, a running physical daemon, and at least 4 GiB free. Then +verify Docker and OpenShell: + +```bash +docker --version +sudo systemctl status docker --no-pager +openshell --version +openshell sandbox list +``` + +If either command is missing, install Docker Engine using Docker's current +Debian ARM64 instructions and install OpenShell using the current OpenShell +instructions. Do not copy credentials into the image, policy, or repository. + +Create the provider once on Reachy, then verify it: + +```bash +openshell provider create \ + --name reachy-openai \ + --type openai \ + --from-existing + +openshell provider get reachy-openai +``` + +Finally, verify that containers can reach the host-side Reachy daemon: + +```bash +docker run --rm --add-host host.openshell.internal:host-gateway \ + curlimages/curl:latest \ + http://host.openshell.internal:8000/api/daemon/status +``` + +Do not continue until the daemon reports `state: running`. + +## Run locally before sandboxing + +From the project directory: + +```bash +cp .env.example .env +export OPENAI_API_KEY=sk-... +./scripts/start-local.sh +``` + +The default `.env.example` selects: + +```dotenv +REACHY_TOOL_TRANSPORT=rest +REACHY_REST_BASE_URL=http://127.0.0.1:8000 +``` + +Test these prompts in text mode first: + +```text +Reachy, look up. +Reachy, look front. +Reachy, look left and then right. +Stop moving. +``` + +## REST transport behavior + +The REST transport always advertises these physical tools: + +- `move_head(directions)` +- `stop_motion()` + +When `REACHY_CAMERA_BASE_URL` is configured, it additionally advertises: + +- `camera(question)` + +`move_head` accepts one to eight values from `left`, `right`, `up`, `down`, and +`front`. Extra keys and raw pose values are rejected before a network request is +made. + +Each successful `goto` returns a move UUID. The client polls +`GET /api/move/running` and waits for that UUID to finish before sending the next +direction. A timed-out POST is reported as `unknown_delivery` and is never +automatically retried. + +`stop_motion` lists active move UUIDs and calls `POST /api/move/stop` once for +each one. + +`camera` posts no model-supplied capture settings. It accepts only a short +question, validates the JPEG response, sends the image into the existing +Realtime conversation, and asks the assistant to answer aloud. An OpenShell +`403` becomes `status: policy_denied` and is not retried. + +## OpenShell policies + +Three relevant policies are checked in: + +```text +openshell/policy-motion-disabled.yaml +openshell/policy-camera-enabled-motion-disabled.yaml +openshell/policy-head-motion-enabled.yaml +``` + +All three allow: + +```text +GET /api/daemon/status +GET /api/move/running +POST /api/move/stop +``` + +Only `policy-head-motion-enabled.yaml` allows: + +```text +POST /api/move/goto +``` + +Only `policy-camera-enabled-motion-disabled.yaml` allows: + +```text +POST /camera/capture +``` + +That camera policy still blocks `POST /api/move/goto`. The base +`policy-motion-disabled.yaml` blocks both camera capture and motion start. + +None of the three policies allow `/api/move/set_target`, `/api/motors/**`, +`/api/apps/**`, raw movement WebSockets, wake/sleep, or recorded motions. + +The permitted binary is `/opt/venv/bin/python`. A denial seen with `curl` could +therefore be a binary denial rather than a path denial; use the application or +the same Python executable for final policy tests. + +## Build the deployable artifacts (development machine) + +Build the native Reachy App wheel: + +```bash +uv build --project native-controller + +sha256sum \ + native-controller/dist/reachy_mini_openshell_controller-0.2.0-py3-none-any.whl +``` + +Build the sandbox specifically for Reachy's ARM64 computer: + +```bash +docker buildx build \ + --platform linux/arm64 \ + --load \ + --tag reachy-mini-openshell:rest-arm64 \ + --file Dockerfile.openshell \ + . + +docker image inspect reachy-mini-openshell:rest-arm64 \ + --format 'architecture={{.Architecture}} size={{.Size}} cmd={{json .Config.Cmd}}' +``` + +Require `architecture=arm64` and +`cmd=["/bin/sleep","infinity"]`. The persistent command keeps the pre-created +sandbox alive; the native Reachy App starts and stops the agent process inside +it. + +### Why this image is small + +The image is custom-built rather than copying the complete Reachy environment: + +- Both stages start from `python:3.12-slim-bookworm`. +- A disposable builder stage constructs wheels; build files do not enter the + runtime stage. +- `requirements-rest.txt` contains the sandbox's REST, Realtime, and audio + requirements. +- The application wheel is installed with `--no-deps`, preventing its normal + Reachy SDK, MuJoCo, OpenCV, simulator, dance, camera-worker, and local vision + dependencies from entering the sandbox. +- Pip caches, install-time bytecode, runtime `pip`, and APT metadata are + removed. +- Debian packages use `--no-install-recommends`. + +The image deliberately retains `iproute2` and `nftables`. OpenShell needs the +trusted `ip` helper to create the isolated sandbox network, so removing it to +save a few megabytes breaks provisioning. The tested image was 339,495,096 +bytes according to Docker. + +Export and compress the image for transfer: + +```bash +docker save reachy-mini-openshell:rest-arm64 \ + | gzip > reachy-mini-openshell-rest-arm64.tar.gz + +sha256sum reachy-mini-openshell-rest-arm64.tar.gz +``` + +## Transfer and load the artifacts + +Copy the image, controller wheel, and main demo policy from the development +machine: + +```bash +scp reachy-mini-openshell-rest-arm64.tar.gz \ + pollen@reachy-mini.local:/home/pollen/ + +scp native-controller/dist/reachy_mini_openshell_controller-0.2.0-py3-none-any.whl \ + pollen@reachy-mini.local:/home/pollen/ + +scp openshell/policy-camera-enabled-motion-disabled.yaml \ + openshell/policy-motion-disabled.yaml \ + openshell/policy-head-motion-enabled.yaml \ + pollen@reachy-mini.local:/home/pollen/ +``` + +On Reachy, compare the received checksums with the development machine, then +load and inspect the image: + +```bash +df -h / +docker system df +docker load --input ~/reachy-mini-openshell-rest-arm64.tar.gz + +docker image inspect reachy-mini-openshell:rest-arm64 \ + --format 'architecture={{.Architecture}} size={{.Size}} cmd={{json .Config.Cmd}}' +``` + +`docker load` may print nothing for several minutes on microSD storage. In a +second SSH session, check `ps`, `df -h /`, and `journalctl -u docker` before +assuming it has stalled. After a verified load, the transferred `.tar.gz` can +be deleted to recover space. + +Put policies in a stable operator-owned directory: + +```bash +mkdir -p ~/reachy-openshell +cp ~/policy-*.yaml ~/reachy-openshell/ +``` + +## Create the sandbox once (Reachy) + +Create an idle sandbox with camera enabled and motion disabled: + +```bash +openshell sandbox create \ + --name reachy-agent \ + --from reachy-mini-openshell:rest-arm64 \ + --policy ~/reachy-openshell/policy-camera-enabled-motion-disabled.yaml \ + --provider reachy-openai \ + --cpu 2 \ + --memory 2Gi \ + --env REACHY_MINI_SKIP_DOTENV=1 \ + --env BACKEND_PROVIDER=openai_realtime \ + --env REACHY_TOOL_TRANSPORT=rest \ + --env REACHY_REST_BASE_URL=http://host.openshell.internal:8000 \ + --env REACHY_CAMERA_BASE_URL=http://host.openshell.internal:8042 \ + --env REACHY_REST_TIMEOUT_SECONDS=5 \ + --env REACHY_MOTION_DURATION_SECONDS=1 \ + --env REACHY_MOTION_POLL_INTERVAL_SECONDS=0.1 \ + --env REACHY_MOTION_COMPLETION_TIMEOUT_SECONDS=10 \ + --env REACHY_AUDIO_HOST=127.0.0.1 \ + --env REACHY_AUDIO_PORT=8765 \ + --env REACHY_AGENT_START_TIMEOUT_SECONDS=120 \ + --env REACHY_MODEL_LOGS=1 \ + --env OPENAI_REALTIME_BASE_URL=https://api.openai.com/v1 \ + --env OPENAI_REALTIME_MODEL=gpt-realtime-2 \ + --env OPENAI_REALTIME_VOICE=cedar +``` + +If the CLI enters an interactive sandbox prompt after creation, type `exit`. +That exits only the shell; the image's `sleep infinity` command keeps the +sandbox alive. + +Check that it is ready: + +```bash +openshell sandbox get reachy-agent +``` + +Require `Phase: Ready`. Read the printed policy and confirm that it allows +`POST host.openshell.internal:8042/camera/capture` but has no allow rule for +`POST host.openshell.internal:8000/api/move/goto`. + +The 120-second agent startup window is intentional. A cold import of the audio +stack takes about 40 seconds on the Reachy Mini onboard Raspberry Pi. + +## Robot-native media and lifecycle + +The normal onboard path uses a small trusted Reachy App from +`projects/reachy-mini-openshell/native-controller`. It owns only the robot +microphone, speaker, camera snapshot adapter, and fixed OpenShell lifecycle +commands. The model, tools, and every requested action remain inside +`reachy-agent`. + +Before installing the native app, test the inner lifecycle directly. A cold +start took approximately 41 seconds on the tested robot: + +```bash +time openshell sandbox exec \ + --name reachy-agent \ + --no-tty \ + -- \ + /usr/bin/env REACHY_AGENT_START_TIMEOUT_SECONDS=120 \ + /opt/venv/bin/reachy-agent-control start + +openshell sandbox exec --name reachy-agent --no-tty -- \ + /opt/venv/bin/reachy-agent-control status + +openshell sandbox exec --name reachy-agent --no-tty -- \ + tail -n 100 /sandbox/logs/reachy-agent.log +``` + +Require `running` and `Application startup complete` before exposing the audio +service. + +### Expose the sandbox audio listener + +The agent listens on `127.0.0.1:8765` **inside the sandbox**. Sandbox loopback +is intentionally private, so the native Reachy App on the host cannot connect +to that address directly. Exposing the service creates a local OpenShell gateway +route from `reachy-agent--audio.openshell.localhost:17670` to the sandbox +listener: + +```bash +openshell service expose reachy-agent 8765 audio +openshell service get reachy-agent audio + +curl --silent --show-error \ + http://reachy-agent--audio.openshell.localhost:17670/health +``` + +Expected output includes: + +```json +{ + "status": "ok", + "active_audio_client": false, + "format": "pcm_s16le", + "sample_rate": 16000, + "channels": 1 +} +``` + +`active_audio_client: false` is correct until the native app connects Reachy's +microphone and speaker. The route is local to the onboard OpenShell gateway; it +does not publish the audio service to the internet or directly to the robot's +LAN. + +The controller's WebSocket URI is: + +```text +ws://reachy-agent--audio.openshell.localhost:17670/audio +``` + +The service accepts one client, mono signed 16-bit PCM at 16 kHz, and exposes +`GET /health` plus `WS /audio`. The Reachy App reads and plays audio through the +SDK media manager. No laptop, browser, Gradio page, or SSH tunnel is required +after installation. + +The same native app serves only this camera operation on the robot host: + +```text +POST http://127.0.0.1:8042/camera/capture +``` + +The sandbox addresses it as +`http://host.openshell.internal:8042/camera/capture`, so the call crosses the +OpenShell REST policy. It is not exposed as an OpenShell browser service. + +When the Reachy App is stopped it runs: + +```bash +openshell sandbox exec --name reachy-agent --no-tty -- \ + /opt/venv/bin/reachy-agent-control stop +``` + +The sandbox, provider, service endpoint, and policy remain provisioned for the +next Start. + +Stop the manually started agent before testing the native lifecycle: + +```bash +openshell sandbox exec --name reachy-agent --no-tty -- \ + /opt/venv/bin/reachy-agent-control stop +``` + +### Install the native Reachy App (Reachy) + +Reachy's application installer does not accept an arbitrary local wheel. Install +the controller into the daemon's shared application environment instead. + +Check whether that environment already exists: + +```bash +ls -l /venvs/apps_venv/bin/python +``` + +If it does not exist, create it with the same Python generation as the Reachy +daemon and install the matching Reachy SDK: + +```bash +/opt/uv/uv venv \ + --python /venvs/mini_daemon/bin/python \ + /venvs/apps_venv + +/opt/uv/uv pip install \ + --no-cache \ + --python /venvs/apps_venv/bin/python \ + 'reachy-mini==1.8.3' +``` + +Use the daemon's actual Reachy Mini version instead of `1.8.3` if it differs: + +```bash +/venvs/mini_daemon/bin/python -c \ + 'import importlib.metadata as m; print(m.version("reachy-mini"))' +``` + +Install the controller wheel that was transferred earlier: + +```bash +/opt/uv/uv pip install --no-cache \ + --python /venvs/apps_venv/bin/python \ + /home/pollen/reachy_mini_openshell_controller-0.2.0-py3-none-any.whl +``` + +The `/api/apps/install` endpoint intentionally does not accept `source_kind: +local`; that endpoint installs catalog/Hugging Face apps. After the manual pip +install, `reachy_mini_openshell_controller` appears in the installed app list +and can be started from the Reachy Apps UI. The daemon launches the controller +in its shared apps environment and supplies the local `ReachyMini` media object. + +Verify the entry point and installed version: + +```bash +/venvs/apps_venv/bin/python -c \ + 'import importlib.metadata as m; print(m.version("reachy-mini-openshell-controller")); print([ep.name for ep in m.entry_points(group="reachy_mini_apps") if "openshell" in ep.name])' +``` + +### Start and verify the native app (Reachy) + +Start it from the Reachy Apps UI, or call the same daemon endpoint: + +```bash +curl --silent --show-error -X POST \ + http://127.0.0.1:8000/api/apps/start-app/reachy_mini_openshell_controller +``` + +Verify all three layers: + +```bash +curl --silent --show-error \ + http://127.0.0.1:8000/api/apps/current-app-status + +openshell sandbox exec --name reachy-agent --no-tty -- \ + /opt/venv/bin/reachy-agent-control status + +openshell service get reachy-agent audio + +curl --silent --show-error \ + http://reachy-agent--audio.openshell.localhost:17670/health +``` + +Require the Reachy App state `running`, inner agent state `running`, an `audio` +service targeting `127.0.0.1:8765`, and a healthy response. Once the native app +connects, `active_audio_client` should become `true`. + +Inspect the inner log when troubleshooting: + +```bash +openshell sandbox exec --name reachy-agent --no-tty -- \ + tail -n 100 /sandbox/logs/reachy-agent.log +``` + +The project README retains an optional Gradio diagnostic path for testing the +model and tool flow independently of robot media. It is not part of normal +onboard operation. + +## Verify the denied action + +With `policy-camera-enabled-motion-disabled.yaml` active, say: + +```text +Reachy, look up. +``` + +Expected behavior: + +1. The model selects `move_head`. +2. The app attempts `POST /api/move/goto`. +3. OpenShell returns HTTP `403`. +4. The tool result has `status: policy_denied`. +5. Reachy does not move. +6. The assistant explains that policy blocked the action and does not retry. + +## Verify camera allow and deny + +With `policy-camera-enabled-motion-disabled.yaml` active, say: + +```text +Reachy, take a picture and tell me what you see. +``` + +Expected behavior: the model selects `camera`, OpenShell permits only +`POST /camera/capture`, one JPEG is delivered to the Realtime session, and +Reachy answers aloud. Asking `Reachy, look up` remains denied. + +To prove the camera boundary, hot-reload the policy that blocks both camera and +motion: + +```bash +openshell policy set reachy-agent \ + --policy ~/reachy-openshell/policy-motion-disabled.yaml \ + --wait +``` + +Ask the picture question again. Reachy must not capture a frame and should +explain that policy blocked the action. Restore the main demo policy afterward: + +```bash +openshell policy set reachy-agent \ + --policy ~/reachy-openshell/policy-camera-enabled-motion-disabled.yaml \ + --wait +``` + +Inspect logs: + +```bash +openshell logs reachy-agent --tail +``` + +## Enable head motion without restarting + +Hot-reload the enabled policy: + +```bash +openshell policy set reachy-agent \ + --policy ~/reachy-openshell/policy-head-motion-enabled.yaml \ + --wait +``` + +Repeat the same request. Reachy should now move through the fixed application +pose. + +Return to the restrictive policy: + +```bash +openshell policy set reachy-agent \ + --policy ~/reachy-openshell/policy-camera-enabled-motion-disabled.yaml \ + --wait +``` + +## Negative policy tests + +Use the permitted Python binary inside the sandbox to test a dangerous path: + +```bash +openshell sandbox exec -n reachy-agent -- \ + /opt/venv/bin/python -c \ + 'import httpx; print(httpx.post("http://host.openshell.internal:8000/api/move/set_target", json={}).status_code)' +``` + +Expected result: `403`. + +Repeat for a motor path or app-management path. Those requests must remain +denied under both policies. + +## Normal operation + +After provisioning, normal use does not require SSH commands: + +1. Start `reachy_mini_openshell_controller` from the Reachy Apps UI. +2. Wait for the app to report `running`; the first cold start may take about 40 + seconds. +3. Speak directly to Reachy. +4. Stop the app from the Reachy Apps UI when finished. + +Stopping the app closes the media bridge and stops the inner conversation +process. It does not delete the sandbox, provider, policy, or audio-service +definition. + +## Troubleshooting quick reference + +| Symptom | First checks | +| --- | --- | +| `docker load` is silent | In another SSH session run `ps -eo pid,etime,stat,%cpu,%mem,cmd \| grep '[d]ocker load'`, `df -h /`, and `sudo journalctl -u docker -n 30 --no-pager`. Slow microSD extraction is normal. | +| Sandbox enters `ContainerRestarting` | Inspect the image command. It must be `sleep infinity`; do not append `/bin/true` to `sandbox create`. | +| `reachy-agent failed to become healthy` | Retry with the 120-second timeout and inspect `/sandbox/logs/reachy-agent.log`. A cold start took about 41 seconds. | +| Audio URL says `Service endpoint is not available` | Confirm the inner agent is `running`, then recreate or expose `audio` and check `/health`. The target listener must exist before the route is usable. | +| Audio bridge reports `Name or service not known` | Verify controller version `0.2.0`. It preserves the virtual routing hostname while connecting the socket to `127.0.0.1:17670`. | +| Reachy says it cannot take a picture | Test `POST http://127.0.0.1:8042/camera/capture`, then test the same path from the sandbox with `/opt/venv/bin/python`. If both work, inspect the model/tool logs and session instructions. | +| Local wheel install is rejected by `/api/apps/install` | This endpoint does not accept `source_kind: local`; install the wheel into `/venvs/apps_venv` with `/opt/uv/uv pip install`. | +| Root filesystem is almost full | Run `docker system df` and `sudo du -sh /var/lib/docker /venvs/apps_venv`. Remove transferred archives, obsolete images, failed layers, and old wheel versions, but not the active image or `/venvs/mini_daemon`. | + +## Development checks + +```bash +uv run ruff check src tests +uv run pytest -q +PYTHONPATH=native-controller/src uv run pytest -q native-controller/tests +``` + +The unit suite covers fixed schemas, pose mapping, argument rejection, ordered +movement, stop behavior, OpenShell `403` conversion, no retry after an uncertain +motion POST, native JPEG limits, and exact camera-policy rules. + +## Completion criteria + +- The sandbox starts no local robot SDK or camera workers. +- Only fixed head directions, stop, and the optional one-frame camera tool are + model-visible robot actions. +- Motion-disabled policy blocks `goto` while preserving stop. +- Camera-enabled/motion-disabled policy permits only one fixed capture endpoint + and still blocks `goto`. +- Motion-enabled policy allows `goto` but no raw target or motor endpoints. +- Policy can be hot-reloaded without recreating the sandbox. +- Documentation states that JSON body values remain application-enforced. diff --git a/projects/reachy-mini-openshell/README.md b/projects/reachy-mini-openshell/README.md index f9384131..a7dec086 100644 --- a/projects/reachy-mini-openshell/README.md +++ b/projects/reachy-mini-openshell/README.md @@ -1,8 +1,13 @@ # Reachy OpenShell -Reachy Mini conversation demo for OpenShell: Gradio UI, simulator support, -microphone or text input, Reachy movement tools, and selectable model backends. -The default and preferred starting point is OpenAI Realtime. +Reachy Mini conversation demo for OpenShell: native robot microphone, speaker, +and single-frame camera capture; optional Gradio input; OpenAI Realtime; and a +deliberately small REST-controlled action surface. + +> **Building the OpenShell policy demo with a physical Reachy?** Follow the +> [onboard setup tutorial](ONBOARD_SETUP.md). For the architecture, implementation +> decisions, and lessons learned, read the +> [Dev Note](../../docs/dev-notes/posts/2026-07-20-policy-controlling-reachy-mini-with-openshell.md). Commands: @@ -10,6 +15,9 @@ Commands: - app: `reachy-mini-conversation-app` - module: `python -m reachy_mini_conversation_app` - check: `reachy-mini-backend-check` +- sandbox audio: `reachy-mini-sandbox-audio` +- sandbox lifecycle: `reachy-agent-control start|stop|status` +- native Reachy App: `native-controller/` ## Quick Start @@ -36,13 +44,18 @@ The launcher creates `.venv`, runs `uv sync`, validates `.env`, starts `reachy-mini-daemon --sim`, then prints the Gradio URL:
+ })
+
"""
+
categories = require_list(metadata, "categories", post["path"])
label = categories[0] if categories else "Research"
date_stamp = post["published"].strftime("%Y.%m.%d")
- return f"""
+ return f"""
Field note / {html.escape(label)}
{date_stamp}
>_
@@ -257,9 +284,10 @@ def render_card_copy(post: dict[str, Any]) -> str:
def render_featured_card(post: dict[str, Any]) -> str:
relative_url = post["path"].relative_to(DEV_NOTES_DIR).with_suffix("").as_posix() + "/"
variant = card_visual_class(post)
- return f"""
+ image_class = " dev-note-card--has-image" if card_hero_image_url(post) else ""
+ return f"""
-{render_card_visual(post)}
+{render_card_visual(post, eager=True)}
{render_card_copy(post)}
"""
@@ -268,7 +296,8 @@ def render_featured_card(post: dict[str, Any]) -> str:
def render_recent_card(post: dict[str, Any]) -> str:
relative_url = post["path"].relative_to(DEV_NOTES_DIR).with_suffix("").as_posix() + "/"
variant = card_visual_class(post)
- return f"""
+ image_class = " dev-note-card--has-image" if card_hero_image_url(post) else ""
+ return f"""
{render_card_visual(post)}
{render_card_copy(post)}
diff --git a/tests/test_render_dev_notes.py b/tests/test_render_dev_notes.py
index 9e1b0807..d9c39aa0 100644
--- a/tests/test_render_dev_notes.py
+++ b/tests/test_render_dev_notes.py
@@ -29,6 +29,7 @@ def make_post(
tags: list[str] | None = None,
card_tags: list[str] | None = None,
variant: str = "",
+ hero_image: str = "",
) -> dict[str, object]:
path = renderer.POSTS_DIR / filename
metadata: dict[str, object] = {
@@ -42,6 +43,8 @@ def make_post(
metadata["card_tags"] = card_tags
if variant:
metadata["card_variant"] = variant
+ if hero_image:
+ metadata["hero_image"] = hero_image
return {
"path": path,
"metadata": metadata,
@@ -125,6 +128,32 @@ def test_card_tags_override_tags_and_variant_reaches_outer_card(self) -> None:
self.assertIn(">evaluation<", card)
self.assertNotIn(">fallback<", card)
+ def test_hero_image_replaces_generated_visual(self) -> None:
+ post = make_post(
+ "2026-06-05-note.md",
+ hero_image="../../assets/reachy-mini-openshell/hero.svg",
+ )
+
+ featured = renderer.render_featured_card(post)
+ self.assertIn("dev-note-card--has-image", featured)
+ self.assertIn("dev-note-card__visual--image", featured)
+ self.assertIn('src="../assets/reachy-mini-openshell/hero.svg"', featured)
+ self.assertIn('loading="eager" fetchpriority="high"', featured)
+ self.assertNotIn("dev-note-card__visual-label", featured)
+
+ recent = renderer.render_recent_card(post)
+ self.assertIn('loading="lazy"', recent)
+ self.assertNotIn("fetchpriority", recent)
+
+ def test_hero_image_must_exist_inside_docs(self) -> None:
+ outside = make_post("2026-06-05-note.md", hero_image="../../../../outside.svg")
+ with self.assertRaisesRegex(ValueError, "must stay inside"):
+ renderer.render_featured_card(outside)
+
+ missing = make_post("2026-06-05-note.md", hero_image="../../assets/missing.svg")
+ with self.assertRaisesRegex(ValueError, "does not exist"):
+ renderer.render_featured_card(missing)
+
def test_metadata_is_html_escaped(self) -> None:
post = make_post(
"2026-06-05-note.md",
diff --git a/zensical.toml b/zensical.toml
index ab56020d..5a16db11 100644
--- a/zensical.toml
+++ b/zensical.toml
@@ -15,8 +15,7 @@ nav = [
"dev-notes/index.md",
{"Posts" = [
# dev-notes:nav:start
- {"Making Dev Notes Repeatable" = "dev-notes/posts/2026-06-05-making-dev-notes-repeatable.md"},
- {"Bootstrapping OpenShell Research" = "dev-notes/posts/2026-06-04-bootstrapping-openshell-research.md"}
+ {"Bringing Privacy and Security to the Edge with OpenShell" = "dev-notes/posts/2026-07-20-policy-controlling-reachy-mini-with-openshell.md"}
# dev-notes:nav:end
]}
]},