diff --git a/.gitignore b/.gitignore index c0972a7..338fae4 100644 --- a/.gitignore +++ b/.gitignore @@ -38,6 +38,21 @@ sbom-server.json htmlcov/ .pytest_cache/ +# Local UAT evidence. Keep compact manifests/results/draft issue summaries +# visible for selective review, but never offer raw household media, event +# streams, health snapshots, or container logs for an accidental commit. +/uat-sessions/**/video/ +/uat-sessions/**/clips/ +/uat-sessions/**/logs/ +/uat-sessions/**/ndjson/ +/uat-sessions/**/snapshots/ + +# Generated audit/session outputs. These can contain machine-local paths, +# prompts, or deployment observations; regenerate or curate before publishing. +/AUDIT-REPORT.md +/audit-draft-issues.md +/pi-session-*.html + # Frontend build deps for the dashboard's vendored Tailwind bundle. # package.json + tailwind.config.js are committed; build artifacts and # the npm install state are not. Re-run `npm install && npm run build:css` diff --git a/CLAUDE.md b/CLAUDE.md index 78f2b1e..155253d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -27,7 +27,7 @@ The voice path runs through a single LLM provider — `PiVoiceLLM`, selected via ``` StackChan hardware → configured persona - │ ESP32-S3, xiaozhi firmware (built from m5stack/StackChan source) + │ ESP32-S3, xiaozhi firmware (pinned BrettKinny/StackChan@dotty fork) │ WiFi / WebSocket (Xiaozhi protocol) ▼ xiaozhi-esp32-server (Docker) @@ -197,7 +197,8 @@ For hardware specs, protocol details, model internals, latent capabilities, and - xiaozhi-esp32-server: https://github.com/xinnan-tech/xiaozhi-esp32-server - xiaozhi-esp32 firmware (upstream): https://github.com/78/xiaozhi-esp32 -- StackChan (hardware + firmware patches): https://github.com/m5stack/StackChan +- StackChan upstream (hardware + base firmware): https://github.com/m5stack/StackChan +- Dotty firmware fork (the pinned build source): https://github.com/BrettKinny/StackChan/tree/dotty - Emotion protocol: https://xiaozhi.dev/en/docs/development/emotion/ ## Agent skills diff --git a/COMPATIBILITY.md b/COMPATIBILITY.md index 84d56c5..c64bc4f 100644 --- a/COMPATIBILITY.md +++ b/COMPATIBILITY.md @@ -13,11 +13,16 @@ For protocol wire formats see `docs/protocols.md`. | Component | Current Version | Protocol / Interface | Breaking Change Policy | |---|---|---|---| -| StackChan firmware (m5stack/StackChan v1.2.4) | v1.2.4 | Xiaozhi WebSocket protocol, MCP over WS (JSON-RPC 2.0) | Pin firmware to a known-good build; do not OTA-update without verifying server compatibility first | +| StackChan firmware (`BrettKinny/StackChan@dotty`) | `fw-v1.3.3` release; submodule `969c2b2` | Xiaozhi WebSocket protocol, MCP over WS (JSON-RPC 2.0), StateManager event contract | Build the pinned submodule; do not substitute the official upstream tree or OTA-update without coordinated verification | | xiaozhi-esp32-server (local build) | `xiaozhi-esp32-server-piper:local` | Custom LLM provider API, `.config.yaml` schema, Xiaozhi WS server | Rebuild image only after checking upstream changelog for provider API or config schema changes | -| dotty-pi (pi agent) | `dotty-pi:0.1.0` | pi RPC (JSONL over stdio), the five `dotty-pi-ext` voice tools | Pin the image tag; pi-version or model changes need end-to-end cutover testing | +| dotty-pi (pi agent) | `dotty-pi:0.1.0` | pi RPC (JSONL over stdio), the seven `dotty-pi-ext` voice tools | Pin the image tag; pi-version or model changes need end-to-end cutover testing | | dotty-behaviour | `dotty-behaviour:0.1.0` | HTTP API (`/api/perception/*`, `/api/vision/*`, `/api/audio/*`, `/health`) | Endpoint signatures stable; perception event-schema changes require firmware + xiaozhi review | -| bridge.py (dashboard) | unversioned (HEAD) | `/ui` dashboard, `/admin/*`, `/health` | Dashboard/admin service only post-#36; admin route changes require updating dashboard callers | +| bridge.py (dashboard) | `dotty-bridge:0.1.0` image from repo HEAD | `/ui` dashboard, `/admin/*`, `/health` | Dashboard/admin service only post-#36; admin route changes require updating dashboard callers | + +The public `fw-v1.3.3` superproject tag (commit `24a009c`, 2026-07-12) is the +current coordinated release pointer. It pins firmware submodule `969c2b2` and +the matching server-side patches. Draft PRs and a dirty submodule are not a +released compatibility set. ## What counts as a breaking change @@ -42,27 +47,28 @@ Any of the following require coordinated updates across components: ## Versioning strategy -No formal versioning is adopted yet (tracked in -[ROADMAP.md](ROADMAP.md#community-wishlist) under "Firmware/server -compatibility matrix"). When adopted, the plan is: +The repo uses separate tag namespaces: + +- `server-vX.Y.Z` for server-only release milestones. +- `fw-vX.Y.Z` for coordinated firmware release pointers in this superproject. -- Separate tag namespaces: `server-vX.Y.Z` and `fw-vX.Y.Z`. -- This matrix will document which server versions are compatible with which - firmware versions. -- The bridge will carry its own version once it moves to a tagged release - cadence. +Container image tags remain `0.1.0` today and are not sufficient by themselves +to identify the exact source revision; retain the Git commit/tag alongside a +deployment record. ## Upgrade guidance 1. **Check this matrix first.** Confirm the component you are upgrading is compatible with the versions of the other components you are running. -2. **Back up before upgrading.** Run `scripts/backup.sh` (or the equivalent - manual steps) to snapshot config, persona files, and bridge state. +2. **Back up before upgrading.** Manually snapshot `.env`, rendered + `data/.config.yaml`, persona/household files, `brain.db` (including WAL/SHM + companions), and the bridge state directory. This repo does not currently + ship an automated backup script. 3. **Upgrade one component at a time.** Validate with a health check (`curl http://:8090/health` and `:8081/health`) plus a live voice turn before moving to the next component. -4. **Tail logs during validation.** Watch both the xiaozhi-server container - logs and the bridge journal simultaneously to catch mismatches early. +4. **Tail logs during validation.** Watch the xiaozhi-server, dotty-pi, + dotty-behaviour, and bridge container logs together to catch mismatches. 5. **Roll back if broken.** Restore from the backup taken in step 2 and revert to the previous image or binary. @@ -106,4 +112,4 @@ versions work with which firmware versions. --- -Last verified: 2026-05-22. +Last verified against the repository and `fw-v1.3.3` pin: 2026-07-16. diff --git a/Dockerfile b/Dockerfile index 08b9caa..f974858 100644 --- a/Dockerfile +++ b/Dockerfile @@ -3,6 +3,11 @@ FROM ghcr.io/xinnan-tech/xiaozhi-esp32-server@sha256:3accd82a7d1a6c01c58f32f6199 RUN pip install --no-cache-dir piper-tts scipy numpy mido faster-whisper sherpa-onnx==1.13.2 +# Patch upstream TTS consumers to treat explicitly registered server-push +# sentence IDs as independent from chat-turn stale-message arbitration (#104). +COPY scripts/patch-tts-server-push.py /tmp/patch-tts-server-push.py +RUN python /tmp/patch-tts-server-push.py /opt/xiaozhi-esp32-server/core/providers/tts + # fluidsynth + General MIDI soundfont for runtime rendering of dance/song MIDI # files to Opus. Installed as the LAST layer so iteration on Python deps above # doesn't invalidate the soundfont download (~141MB). diff --git a/Makefile b/Makefile index 4f798bc..aa182eb 100644 --- a/Makefile +++ b/Makefile @@ -221,9 +221,10 @@ setup: _preflight-compose ## Interactive first-run wizard (re-runnable; remember echo -e "$(GREEN)$(BOLD)Setup complete.$(RESET)"; \ echo ""; \ echo "Next steps:"; \ - echo " 1. Flash the StackChan firmware (see SETUP.md or m5stack/StackChan repo)."; \ - echo " 2. In the device's Advanced Options, set the OTA URL to:"; \ - echo " http://$$XIAOZHI_HOST:8003/xiaozhi/ota/"; \ + echo " 1. Build and flash StackChan firmware with this compiled setting:"; \ + echo " CONFIG_OTA_URL=\"http://$$XIAOZHI_HOST:8003/xiaozhi/ota/\""; \ + echo " See SETUP.md. The on-device Settings app has no OTA URL editor."; \ + echo " 2. Provision the robot's 2.4 GHz Wi-Fi using its displayed setup flow."; \ echo " 3. Run 'make doctor' to verify everything is healthy."; \ echo "" diff --git a/README.md b/README.md index df05055..dc66011 100644 --- a/README.md +++ b/README.md @@ -36,7 +36,7 @@ Full policy: [`AI_TRANSPARENCY.md`](./AI_TRANSPARENCY.md). - **Streaming responses** — the bridge streams LLM output to the voice pipeline for lower perceived latency. - **Emoji expressions** — every response starts with an emoji that the firmware maps to a face animation (smile, laugh, sad, surprise, thinking, angry, love, sleepy, neutral). - **Voice tools** — the pi agent can search its memory, escalate hard questions to a bigger model, take a photo, and play songs, all mid-conversation. -- **States, toggles & LEDs** — a six-state mutex (`idle / talk / story_time / security / sleep / dance`) plus two orthogonal toggles (`kid_mode`, `smart_mode`), all owned by the firmware StateManager and surfaced on the 12-pixel LED ring. Shipped on the active firmware fork (commit `d78118b`, 2026-04-27); the `firmware/firmware/` submodule pin in this repo lags, so flash from the active fork to get it. See "States, Toggles & LEDs" below and [`docs/modes.md`](./docs/modes.md). +- **States, toggles & LEDs** — a six-state mutex (`idle / talk / story_time / security / sleep / dance`) plus two orthogonal toggles (`kid_mode`, `smart_mode`), all owned by the firmware StateManager and surfaced on the 12-pixel LED ring. The pinned `BrettKinny/StackChan@dotty` firmware submodule includes the StateManager and current dance-motion ownership fixes. See "States, Toggles & LEDs" below and [`docs/modes.md`](./docs/modes.md). - **Vision (camera)** — the robot's built-in camera can capture images for multimodal LLM queries. - **Privacy LEDs** — hardware-bound mic (green) and camera (red) indicators on the LED ring. They light from the codec/camera enable signals via RAII guards, so a misbehaving server or model can't capture with the lights off. - **Calendar context** — optional calendar integration feeds upcoming events into the conversation context. @@ -46,7 +46,9 @@ Full policy: [`AI_TRANSPARENCY.md`](./AI_TRANSPARENCY.md). Behaviour is a **six-state mutex** (`idle / talk / story_time / security / sleep / dance`) plus two orthogonal toggles (`kid_mode`, `smart_mode`), all owned by the firmware StateManager (shipped on the active fork in commit `d78118b`, 2026-04-27; bench checks tracked in [#38](https://github.com/BrettKinny/dotty-stackchan/issues/38)). Voice phrases, camera edges, and dashboard controls all flow through it. -> Note: the `firmware/firmware/` submodule pin in this repo deliberately lags the active fork — flashing from the submodule won't give you Phase 4 yet. See the "Firmware iteration" section in [`CLAUDE.md`](./CLAUDE.md) and the submodule-pin caveat in [`docs/modes.md`](./docs/modes.md). +Build firmware from this repo's pinned `firmware/` submodule. It points at the +Dotty fork and is the reproducible source used for the current release; the +official `m5stack/StackChan` repository remains the upstream project. The 12-pixel LED ring shows the current state at a glance. **Left ring 0-5 is the state arc** — all six pixels paint the state colour, matching the dashboard's state buttons: @@ -61,7 +63,9 @@ The 12-pixel LED ring shows the current state at a glance. **Left ring 0-5 is th On the right ring, **indices 8-9 are toggle pips** for kid_mode (salmon pink) and smart_mode (orange), and **index 11 (bottom) lights red while you have the turn** (LISTENING). The `idle → talk` transition fires on `face_detected` from the firmware; VLM identity recognition runs in parallel and feeds the LLM context. -> Heads up: that right-ring layout is the active-fork StateManager. On the firmware **submodule pinned in this repo**, pixels 6 and 11 instead drive the **privacy LEDs** — 6 = mic (green), 11 = camera (red) — and the StateManager pips arrive once the submodule catches up to the active fork. +The current pin includes the StateManager right-ring layout. Privacy indication +is still hardware-bound in the firmware, but it no longer uses the older +pre-StateManager pixel contract described by historical releases. Full state taxonomy, colour palette, transition diagram, and per-state backing architecture: [`docs/modes.md`](./docs/modes.md). @@ -77,7 +81,7 @@ The dashboard service serves a web dashboard at `http://:8081/ui` ## Reference deployment -- **Hardware**: M5Stack StackChan (CoreS3 + servo kit), firmware built from `m5stack/StackChan`. +- **Hardware**: M5Stack StackChan (CoreS3 + servo kit), firmware built from the pinned `BrettKinny/StackChan@dotty` fork of `m5stack/StackChan`. - **Brain**: a **pi** coding agent running in the `dotty-pi` container. It runs `qwen3.5:4b` on local [llama-swap](./docs/cookbook/llama-swap-concurrent-models.md) for the conversation loop and escalates hard questions to `qwen3.6:27b-think` (also on llama-swap) via its `think_hard` tool. xiaozhi-server's `PiVoiceLLM` provider hands each voice turn to the agent. See [`docs/brain.md`](./docs/brain.md). - **Voice I/O**: xiaozhi-esp32-server on Docker (any Linux Docker host). @@ -85,7 +89,7 @@ The dashboard service serves a web dashboard at `http://:8081/ui` | Component | Host | Notes | |---|---|---| -| StackChan (device) | ESP32-S3 on the desk | Firmware built from `m5stack/StackChan` (see `SETUP.md`) | +| StackChan (device) | ESP32-S3 on the desk | Firmware built from the pinned `BrettKinny/StackChan@dotty` submodule (see `SETUP.md`) | | xiaozhi-esp32-server | server (``) | Docker — voice I/O, ports 8000 + 8003 | | dotty-pi | server (``) | Docker — the pi agent, Dotty's voice brain | | dotty-behaviour | server (``) | Docker — FastAPI: perception bus, ambient consumers, vision, greeter; port 8090 | diff --git a/SETUP.md b/SETUP.md index fee1f2e..a775bcd 100644 --- a/SETUP.md +++ b/SETUP.md @@ -48,34 +48,38 @@ There is **no SoftAP captive portal** in stock firmware (some older third-party xiaozhi builds had one; the shipped firmware does not). To run the StackChan **fully self-hosted** (no phone-app account, no vendor cloud, -your own xiaozhi-server as the endpoint), you need to **reflash the device -with firmware built from the open source tree**. - -The upstream firmware lives at **https://github.com/m5stack/StackChan**: -- `firmware/` — M5Stack's patches + ESP-IDF project wrapper -- `firmware/fetch_repos.py` — pulls `78/xiaozhi-esp32` as a dependency and - applies StackChan-specific patches, adding the board target - `CONFIG_BOARD_TYPE_M5STACK_STACK_CHAN` +your own xiaozhi-server as the endpoint), reflash it with Dotty's pinned +`BrettKinny/StackChan@dotty` firmware fork. The official `m5stack/StackChan` +tree is its upstream, but it does not contain Dotty's state, motion, LED, and +perception changes. + +The StackChan's on-device Settings app has no **Advanced Options** or OTA URL +editor. The similarly named **Advanced** tab documented by Xiaozhi belongs to +its browser-based SoftAP captive portal and appears only when that provisioning +mode is active. The current `fw-v1.3.3` prebuilt is compiled for the +maintainer's LAN, so another self-hosted deployment must build from source with +its own `CONFIG_OTA_URL` as shown below. + +The exact firmware source is the `firmware/` submodule of this repo. It pins +`BrettKinny/StackChan@dotty`, whose `firmware/fetch_repos.py` pulls +`78/xiaozhi-esp32` v2.2.4 and applies the StackChan integration patch. --- ## 2. Build and flash open firmware -> **Note — build flow documented from first-pass session findings; not yet -> end-to-end verified. Will be updated after a successful first flash.** - Requires **ESP-IDF v5.5.4**. Easiest path: the official `espressif/idf:v5.5.4` Docker image. ### 2a. Clone and configure ```bash -git clone https://github.com/m5stack/StackChan.git -cd StackChan/firmware +git clone --recursive https://github.com/BrettKinny/dotty-stackchan.git +cd dotty-stackchan/firmware/firmware ``` Point the firmware at your xiaozhi-server for OTA. Edit -`firmware/sdkconfig.defaults` and add (or modify) the line: +`sdkconfig.defaults` and add (or modify) the line: ``` CONFIG_OTA_URL="http://:8003/xiaozhi/ota/" @@ -83,10 +87,20 @@ CONFIG_OTA_URL="http://:8003/xiaozhi/ota/" Trailing slash matters — that's the path the server exposes. +`sdkconfig.defaults` seeds a newly generated `sdkconfig`; it does not overwrite +an existing one. On a repeat build, either make the same change through +`idf.py menuconfig` (**Xiaozhi Assistant → Default OTA URL**) or remove the +generated `sdkconfig` before rebuilding. Verify the effective value after +configuration with: + +```bash +grep '^CONFIG_OTA_URL=' sdkconfig +``` + ### 2b. Build inside the IDF container ```bash -docker run --rm -it -v "$PWD/..":/project -w /project/firmware \ +docker run --rm -it -v "$PWD":/project -w /project \ espressif/idf:v5.5.4 \ bash -c "python3 fetch_repos.py && idf.py build" ``` @@ -101,7 +115,7 @@ Linux (`/dev/cu.usbmodem*` on macOS — adapt the `--device` flag). ```bash docker run --rm -it --device=/dev/ttyACM0 \ - -v "$PWD/..":/project -w /project/firmware \ + -v "$PWD":/project -w /project \ espressif/idf:v5.5.4 \ idf.py -p /dev/ttyACM0 flash ``` @@ -111,14 +125,23 @@ flow), use that instead. ### 2d. First boot after flash -- No pairing-code screen -- No BLE provisioning step -- The device boots, loads WiFi credentials compiled into the firmware - (or, if you left WiFi unconfigured, whatever fallback the upstream - build offers — consult the xiaozhi-esp32 README for the current default - behaviour) -- It POSTs to `http://:8003/xiaozhi/ota/`, gets back the - WebSocket endpoint, and connects +Seeing **“Welcome! Let's get started.” is expected on a fresh flash**. It is +the StackChan app-configuration wizard, controlled by an NVS flag; it does not +mean the compiled `CONFIG_OTA_URL` was ignored. + +For a self-hosted setup: + +1. Tap **Skip** on the welcome screen. This bypasses the M5Stack account/BLE + wizard for the current boot. +2. If the launcher remains visible, open **SETUP**, then use its home control + to exit. Closing SETUP starts the Xiaozhi voice application. +3. With no saved SSID, the pinned build enters its default Xiaozhi hotspot + provisioner. Join the temporary `Xiaozhi-*` Wi-Fi network and open the URL + shown on the robot (normally `http://192.168.4.1`). +4. Select the robot's 2.4 GHz Wi-Fi network. The browser portal also has an + **Advanced** tab where you can confirm or override `ota_url`. +5. After Wi-Fi connects, the device POSTs to the compiled/persisted OTA URL, + receives the WebSocket endpoint, and connects. Tail the server logs while the device boots so you can watch the handshake happen (see step 4 below). @@ -127,18 +150,13 @@ handshake happen (see step 4 below). ## 3. WiFi credentials -Two options, depending on what the upstream xiaozhi-esp32 build exposes at -the version you pulled: +The pinned firmware does **not** define `CONFIG_WIFI_SSID` or +`CONFIG_WIFI_PASSWORD`; adding those names to `sdkconfig.defaults` has no +effect. Wi-Fi credentials live in NVS and are supplied through the Xiaozhi +hotspot portal described above (or retained from an earlier compatible +provisioning). -- **Compile-time WiFi credentials** — set `CONFIG_WIFI_SSID` and - `CONFIG_WIFI_PASSWORD` in `sdkconfig.defaults`. Simplest for a static - home setup; easy to forget they're in the binary. -- **Fallback SoftAP or BLE provisioning** — some upstream builds include a - fallback provisioning flow if no credentials are saved. Check the - upstream README for what your commit supports. - -Either way, the device must land on a **2.4 GHz** network. ESP32-S3 does -not do 5 GHz. +The device must land on a **2.4 GHz** network. ESP32-S3 does not do 5 GHz. --- @@ -234,6 +252,9 @@ Some older xiaozhi builds expose a SoftAP captive portal on first boot: If you have a build where this works, it's the fastest provisioning flow. It just isn't what M5Stack ships today. +This **Advanced** control is a tab in the browser portal at `192.168.4.1`, not +an item in the StackChan's on-device Settings app. + --- ## 9. When it's working: bookmark these diff --git a/custom-providers/pi_voice/pi_client.py b/custom-providers/pi_voice/pi_client.py index 7431bf3..6a57d8b 100644 --- a/custom-providers/pi_voice/pi_client.py +++ b/custom-providers/pi_voice/pi_client.py @@ -98,6 +98,7 @@ def local_exec_subprocess_factory( SubprocessFactory = Callable[[], subprocess.Popen] +VoiceToolRunner = Callable[[str, dict[str, str]], str] class PiClientError(Exception): @@ -129,6 +130,7 @@ def __init__( *, turn_timeout_sec: float = 120.0, stderr_ring_size: int = 200, + voice_tool_runner: VoiceToolRunner | None = None, ): self._spawn = subprocess_factory self._proc: Optional[subprocess.Popen] = None @@ -141,6 +143,7 @@ def __init__( self._turn_timeout_sec = turn_timeout_sec self._next_req_id = 0 self._closed = False + self._voice_tool_runner = voice_tool_runner # ------------------------------------------------------------------ # lifecycle @@ -277,6 +280,15 @@ def iter_turn_text(self, prompt: str) -> Iterator[str]: f"turn timed out after {self._turn_timeout_sec}s" ) + def invoke_voice_tool(self, name: str, arguments: dict[str, str]) -> str: + """Run one deterministic voice tool outside the fallible 4B router.""" + if self._voice_tool_runner is None: + raise PiClientError("direct voice-tool runner is not configured") + try: + return self._voice_tool_runner(name, arguments) + except Exception as exc: + raise PiClientError(f"voice tool {name} failed: {exc}") from exc + # ------------------------------------------------------------------ # diagnostics # ------------------------------------------------------------------ @@ -375,4 +387,47 @@ def make_default_pi_client() -> PiClient: subprocess_factory=lambda: local_exec_subprocess_factory( container=container, pi_args=pi_args, ), + voice_tool_runner=lambda name, arguments: _run_container_voice_tool( + container, name, arguments, + ), + ) + + +def _run_container_voice_tool( + container: str, name: str, arguments: dict[str, str], +) -> str: + modules = { + "remember": ("remember.ts", "runRemember", "fact"), + "memory_lookup": ("memory_lookup.ts", "runMemoryLookup", "query"), + "think_hard": ("think_hard.ts", "runThinkHard", "question"), + } + if name not in modules: + raise PiClientError(f"unsupported direct voice tool: {name}") + module, function, argument = modules[name] + script = ( + 'import fs from "node:fs";' + f'import {{ {function} }} from "/root/.pi/extensions/dotty-pi-ext/src/tools/{module}";' + 'const input=JSON.parse(fs.readFileSync(0,"utf8"));' + f'const result=await {function}(input[{json.dumps(argument)}]);' + 'process.stdout.write(JSON.stringify({result}));' + ) + proc = subprocess.run( + [ + "docker", "exec", "-i", container, "node", + "--experimental-strip-types", "--input-type=module", "-e", script, + ], + input=json.dumps(arguments), text=True, capture_output=True, timeout=60, + check=False, ) + if proc.returncode != 0: + raise PiClientError(proc.stderr.strip() or f"tool exited {proc.returncode}") + try: + payload = json.loads(proc.stdout) + except json.JSONDecodeError as exc: + raise PiClientError("voice tool returned invalid JSON") from exc + if not isinstance(payload, dict): + raise PiClientError("voice tool returned invalid JSON shape") + result = payload.get("result") + if not isinstance(result, str): + raise PiClientError("voice tool returned a non-string result") + return result diff --git a/custom-providers/pi_voice/pi_voice.py b/custom-providers/pi_voice/pi_voice.py index 39036be..e4c3ebc 100644 --- a/custom-providers/pi_voice/pi_voice.py +++ b/custom-providers/pi_voice/pi_voice.py @@ -35,6 +35,7 @@ import json import os +import re import unicodedata from pathlib import Path from typing import Iterator @@ -201,6 +202,21 @@ def _enforce_leading_emoji(chunks: Iterator[str]) -> Iterator[str]: yield f"{FALLBACK_EMOJI} (no response)" +_REMEMBER_INTENT_RE = re.compile( + r"^\s*(?:please\s+)?remember\s+(?:that\s+)?(?P.+?)\s*[.!?]?\s*$", + re.IGNORECASE, +) +_RECALL_INTENT_RE = re.compile( + r"^\s*(?:what did i (?:tell|say to) you about|do you remember)\s+" + r"(?P.+?)\s*[?!.]?\s*$", + re.IGNORECASE, +) +_THINK_HARD_INTENT_RE = re.compile( + r"^\s*(?:please\s+)?think hard (?:about\s*:?)?\s*(?P.+?)\s*[?!.]?\s*$", + re.IGNORECASE, +) + + class LLMProvider(LLMProviderBase): """xiaozhi-server LLM provider backed by the dotty-pi container.""" @@ -229,6 +245,39 @@ def response(self, session_id, dialogue, **kwargs) -> Iterator[str]: if not user_text: yield f"{FALLBACK_EMOJI} (empty turn)" return + remember_intent = _REMEMBER_INTENT_RE.match(user_text) + if remember_intent: + fact = remember_intent.group("fact").strip() + result = self._invoke_voice_tool("remember", {"fact": fact}) + if result == "(remembered)": + yield f"{FALLBACK_EMOJI} I'll remember that." + else: + yield f"{FALLBACK_EMOJI} I couldn't save that memory." + return + recall_intent = _RECALL_INTENT_RE.match(user_text) + if recall_intent: + query = recall_intent.group("query").strip() + result = self._invoke_voice_tool( + "memory_lookup", {"query": query}, + ) + if result is None: + yield f"{FALLBACK_EMOJI} I couldn't check my memory right now." + elif result in ("(no memories found)", "(empty query)"): + yield f"{FALLBACK_EMOJI} I don't remember anything about that yet." + else: + yield f"{FALLBACK_EMOJI} {result}" + return + think_intent = _THINK_HARD_INTENT_RE.match(user_text) + if think_intent: + question = think_intent.group("question").strip() + result = self._invoke_voice_tool( + "think_hard", {"question": question}, + ) + if result is None or result.startswith("("): + yield f"{FALLBACK_EMOJI} I couldn't finish the deeper reasoning." + else: + yield f"{FALLBACK_EMOJI} {result}" + return prompt = _wrap_with_sandwich(user_text, self._kid_mode) # Reset pi state between voice turns. First turn skips this — @@ -259,6 +308,15 @@ def response(self, session_id, dialogue, **kwargs) -> Iterator[str]: logger.error(" pi.stderr: %s", line) yield f"{FALLBACK_EMOJI} (brain offline — try again in a moment)" + def _invoke_voice_tool( + self, name: str, arguments: dict[str, str], + ) -> str | None: + try: + return self._client.invoke_voice_tool(name, arguments) + except PiClientError as exc: + logger.error("PiVoiceLLM direct tool %s failed: %s", name, exc) + return None + def _on_filter_hit(self, tier: str, match) -> None: # Local logging only — the Prometheus counter / safety ring live in # the bridge container, which this provider can't reach. diff --git a/custom-providers/pi_voice/tests/test_pi_client.py b/custom-providers/pi_voice/tests/test_pi_client.py index 50e2b5e..9472dba 100644 --- a/custom-providers/pi_voice/tests/test_pi_client.py +++ b/custom-providers/pi_voice/tests/test_pi_client.py @@ -17,12 +17,14 @@ import threading import time import unittest +from types import SimpleNamespace +from unittest.mock import patch # Make the package importable as `pi_voice.*` regardless of cwd. HERE = os.path.dirname(os.path.abspath(__file__)) sys.path.insert(0, os.path.dirname(HERE)) -from pi_client import PiClient, PiClientError # noqa: E402 +from pi_client import PiClient, PiClientError, _run_container_voice_tool # noqa: E402 class FakePopen: @@ -359,5 +361,27 @@ def test_silent_pi_times_out(self): client.close() +class TestContainerVoiceToolRunner(unittest.TestCase): + def test_rejects_tools_outside_allowlist_without_spawning(self): + with patch("pi_client.subprocess.run") as run: + with self.assertRaisesRegex(PiClientError, "unsupported"): + _run_container_voice_tool("dotty-pi", "play_song", {"name": "x"}) + run.assert_not_called() + + def test_rejects_malformed_json_output(self): + completed = SimpleNamespace(returncode=0, stdout="not-json", stderr="") + with patch("pi_client.subprocess.run", return_value=completed): + with self.assertRaisesRegex(PiClientError, "invalid JSON"): + _run_container_voice_tool("dotty-pi", "remember", {"fact": "x"}) + + def test_rejects_non_string_result(self): + completed = SimpleNamespace( + returncode=0, stdout='{"result":42}', stderr="", + ) + with patch("pi_client.subprocess.run", return_value=completed): + with self.assertRaisesRegex(PiClientError, "non-string"): + _run_container_voice_tool("dotty-pi", "remember", {"fact": "x"}) + + if __name__ == "__main__": unittest.main() diff --git a/custom-providers/pi_voice/tests/test_pi_voice.py b/custom-providers/pi_voice/tests/test_pi_voice.py index 7725fa7..aca75ae 100644 --- a/custom-providers/pi_voice/tests/test_pi_voice.py +++ b/custom-providers/pi_voice/tests/test_pi_voice.py @@ -44,6 +44,9 @@ def __init__(self): self.scripted_chunks: list[list[str]] = [] self.scripted_errors: list[BaseException | None] = [] self.closed = False + self.tool_calls: list[tuple[str, dict[str, str]]] = [] + self.tool_results: dict[str, str] = {} + self.tool_errors: dict[str, BaseException] = {} def script_turn(self, chunks: list[str], error: BaseException | None = None) -> None: self.scripted_chunks.append(chunks) @@ -64,6 +67,12 @@ def iter_turn_text(self, prompt: str) -> Iterator[str]: def recent_stderr(self) -> list[str]: return [] + def invoke_voice_tool(self, name: str, arguments: dict[str, str]) -> str: + self.tool_calls.append((name, arguments)) + if name in self.tool_errors: + raise self.tool_errors[name] + return self.tool_results.get(name, "(tool failed)") + def close(self) -> None: self.closed = True @@ -204,6 +213,91 @@ def test_client_error_yields_fallback(self): self.assertEqual(out, [f"{textUtils.FALLBACK_EMOJI} (brain offline — try again in a moment)"]) +class TestDeterministicVoiceToolRouting(unittest.TestCase): + def test_explicit_remember_invokes_tool_and_never_claims_failed_write(self): + client = FakeClient() + client.tool_results["remember"] = "(remember failed)" + provider = LLMProvider({}, client=client) # type: ignore[arg-type] + + out = "".join(provider.response( + "s", [{"role": "user", "content": "Remember that my calibration color is ultraviolet"}], + )) + + self.assertEqual( + client.tool_calls, + [("remember", {"fact": "my calibration color is ultraviolet"})], + ) + self.assertNotIn("remembered", out.lower()) + self.assertIn("couldn't save", out.lower()) + + def test_remember_client_error_returns_honest_tts_failure(self): + client = FakeClient() + client.tool_errors["remember"] = PiClientError("tool timed out") + provider = LLMProvider({}, client=client) # type: ignore[arg-type] + + out = "".join(provider.response( + "s", [{"role": "user", "content": "Remember that the key is amber"}], + )) + + self.assertEqual(out, f"{textUtils.FALLBACK_EMOJI} I couldn't save that memory.") + + def test_explicit_recall_invokes_lookup_and_speaks_completed_result(self): + client = FakeClient() + client.tool_results["memory_lookup"] = "My calibration color is ultraviolet." + provider = LLMProvider({}, client=client) # type: ignore[arg-type] + + out = "".join(provider.response( + "s", [{"role": "user", "content": "What did I tell you about my calibration color?"}], + )) + + self.assertEqual( + client.tool_calls, + [("memory_lookup", {"query": "my calibration color"})], + ) + self.assertIn("My calibration color is ultraviolet.", out) + + def test_recall_client_error_returns_honest_tts_failure(self): + client = FakeClient() + client.tool_errors["memory_lookup"] = PiClientError("lookup failed") + provider = LLMProvider({}, client=client) # type: ignore[arg-type] + + out = "".join(provider.response( + "s", [{"role": "user", "content": "Do you remember my calibration color?"}], + )) + + self.assertEqual( + out, f"{textUtils.FALLBACK_EMOJI} I couldn't check my memory right now.", + ) + + def test_explicit_think_hard_invokes_reasoner_and_speaks_completed_result(self): + client = FakeClient() + client.tool_results["think_hard"] = "The precise answer is forty-two." + provider = LLMProvider({}, client=client) # type: ignore[arg-type] + + out = "".join(provider.response( + "s", [{"role": "user", "content": "Think hard about: what is six times seven?"}], + )) + + self.assertEqual( + client.tool_calls, + [("think_hard", {"question": "what is six times seven"})], + ) + self.assertIn("The precise answer is forty-two.", out) + + def test_think_hard_client_timeout_returns_honest_tts_failure(self): + client = FakeClient() + client.tool_errors["think_hard"] = PiClientError("tool timed out") + provider = LLMProvider({}, client=client) # type: ignore[arg-type] + + out = "".join(provider.response( + "s", [{"role": "user", "content": "Think hard about: six times seven"}], + )) + + self.assertEqual( + out, f"{textUtils.FALLBACK_EMOJI} I couldn't finish the deeper reasoning.", + ) + + class TestLeadingEmojiContract(unittest.TestCase): def _response(self, chunks: list[str], *, kid_mode: bool = False) -> list[str]: env = { diff --git a/custom-providers/xiaozhi-patches/http_server.py b/custom-providers/xiaozhi-patches/http_server.py index b71585e..d1e3dbc 100644 --- a/custom-providers/xiaozhi-patches/http_server.py +++ b/custom-providers/xiaozhi-patches/http_server.py @@ -552,16 +552,18 @@ async def _dotty_say(self, request: "web.Request") -> "web.Response": ) sentence_id = _uuid.uuid4().hex + server_push_ids = getattr(conn, "_dotty_server_push_sentence_ids", None) + if server_push_ids is None: + server_push_ids = set() + conn._dotty_server_push_sentence_ids = server_push_ids def _enqueue() -> None: try: - # The consumer thread filters with - # `message.sentence_id != self.conn.sentence_id` and - # drops anything that doesn't match — so we stamp the - # conn with our new id BEFORE putting messages on the - # queue. This will pre-empt any in-flight TTS for this - # conn, which is acceptable for server-pushed greetings - # (they shouldn't race a chat turn in normal operation). + # Stamp the connection so an idle device treats this as its + # active utterance. Explicitly register this ID as a server + # push so the patched TTS consumer accepts it independently + # if a concurrent chat replaces conn.sentence_id (#104). + server_push_ids.add(sentence_id) # # Frame the utterance with FIRST/MIDDLE/LAST: FIRST inits # consumer state, MIDDLE carries the text into the @@ -589,6 +591,12 @@ def _enqueue() -> None: # The puts are sync on a thread-safe queue, but we hop a thread # so the aiohttp loop never blocks on producer-side contention. await asyncio.to_thread(_enqueue) + # Consumers need the marker through text synthesis and audio queuing, + # but not forever. Expiry bounds per-connection state without risking + # eviction of an active push during a short burst. + asyncio.get_running_loop().call_later( + 120.0, server_push_ids.discard, sentence_id, + ) self.logger.bind(tag=TAG).info( f"say queued device={resolved_id} sid={sentence_id[:8]} " f"text={text[:60]!r}" diff --git a/docs/about.md b/docs/about.md index 848a985..7210de1 100644 --- a/docs/about.md +++ b/docs/about.md @@ -48,7 +48,7 @@ This is a hackable starting point, not a product. There are no releases, no inst - A polished end-user product. No GUI installer, no app store, no firmware OTA distribution. - Multi-user / multi-device. The reference deployment is one robot talking to one server. -- Upstream firmware development. We build from `m5stack/StackChan` source but don't maintain firmware patches beyond what's needed for the voice integration. +- General upstream firmware development. Dotty builds from the pinned `BrettKinny/StackChan@dotty` fork of `m5stack/StackChan` and maintains the state, motion, LED, privacy, and perception changes needed by this stack. - Cloud hosting. This is designed for a LAN deployment. You could expose it to the internet, but that's your problem. ## Privacy diff --git a/docs/architecture.md b/docs/architecture.md index a701575..700436d 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -61,7 +61,7 @@ Solid arrows are per-turn data flow; dotted arrows are cloud / conditional. All | Actor | Host | Role | Process | |---|---|---|---| -| **StackChan** | Desk | Captures audio, plays audio, renders face, runs MCP tools for head/LED/camera | ESP32-S3 firmware built from `m5stack/StackChan` | +| **StackChan** | Desk | Captures audio, plays audio, renders face, runs MCP tools for head/LED/camera | ESP32-S3 firmware built from the pinned `BrettKinny/StackChan@dotty` fork | | **xiaozhi-esp32-server** | Docker host | VAD → ASR → LLM (proxy) → TTS pipeline, emotion dispatch, OTA, admin surface | Docker container | | **PiVoiceLLM custom provider** | Docker host (inside xiaozhi container) | Default LLM provider — translates each voice turn into a pi RPC request, streams TTS-bound text back | Python, mounted via volume | | **dotty-pi** | Docker host | The voice-tool brain — pi coding agent with the `dotty-pi-ext` extension; owns the agent loop and tool dispatch | Docker container (`dotty-pi`) | diff --git a/docs/hardware.md b/docs/hardware.md index 54af87e..e7bf173 100644 --- a/docs/hardware.md +++ b/docs/hardware.md @@ -10,7 +10,7 @@ description: M5Stack StackChan hardware specs, CoreS3 ESP32-S3 SoC, and servo ch - The robot body is the **M5Stack StackChan** kit: an M5Stack **CoreS3** (ESP32-S3) head on a 2-servo chassis. - The CoreS3 supplies the SoC, display, camera, mic array, speaker, IMU, proximity, microSD — all integrated. NFC and the IR tx/rx pair physically live on the kit body, not the CoreS3 (see the kit table below). - The *StackChan kit* adds the head-yaw servo, head-pitch servo, 12 RGB LEDs, 3-zone touch panel, 550 mAh supplementary battery, USB-C, NFC, an IR tx/rx pair, an IO expander, a dedicated battery monitor, and the 3D-printed body. A separate handheld ESP-NOW remote controller ships in the same box (M5Stack's product page describes it as an ESP-NOW wireless remote). -- Firmware on the device is built from [`m5stack/StackChan`](https://github.com/m5stack/StackChan) — an Arduino C++ codebase that bundles the **XiaoZhi AI agent** client. It is **not** the same codebase as `meganetaaan/stack-chan` (the original Moddable/JS project) or `78/xiaozhi-esp32` (generic voice-assistant firmware). +- Firmware on the device is built from the pinned [`BrettKinny/StackChan@dotty`](https://github.com/BrettKinny/StackChan/tree/dotty) fork of [`m5stack/StackChan`](https://github.com/m5stack/StackChan). It bundles the **XiaoZhi AI agent** client plus Dotty's state, motion, LED, privacy, and perception changes. It is **not** the same codebase as `meganetaaan/stack-chan` (the original Moddable/JS project) or a direct build of `78/xiaozhi-esp32` (generic voice-assistant firmware). - The device advertises itself over the Xiaozhi WebSocket protocol and exposes **on-device tools via MCP** (see [protocols.md](./protocols.md)). - Canonical hardware reference: [`docs.m5stack.com/en/StackChan`](https://docs.m5stack.com/en/StackChan) (kit-level) and [`docs.m5stack.com/en/core/CoreS3`](https://docs.m5stack.com/en/core/CoreS3) (head unit). See [references.md](./references.md#hardware). @@ -80,10 +80,11 @@ Three related codebases — do not confuse them: | Repo | Language | Purpose | Runs on StackChan? | |---|---|---|---| | [`meganetaaan/stack-chan`](https://github.com/meganetaaan/stack-chan) | TypeScript / JavaScript on Moddable SDK | Original open-source Stack-chan (Shinya Ishikawa) | Yes (but not what we run) | -| [`m5stack/StackChan`](https://github.com/m5stack/StackChan) | Arduino C++ | M5Stack's official firmware — bundles XiaoZhi AI agent, targets CoreS3 | **Yes — this is what we flash** | +| [`BrettKinny/StackChan@dotty`](https://github.com/BrettKinny/StackChan/tree/dotty) | ESP-IDF / C++ | Dotty's pinned fork of M5Stack's firmware — bundles XiaoZhi and the Dotty-specific firmware contract | **Yes — build the pinned submodule** | +| [`m5stack/StackChan`](https://github.com/m5stack/StackChan) | ESP-IDF / C++ | Official upstream firmware | Upstream only; do not flash directly for Dotty | | [`78/xiaozhi-esp32`](https://github.com/78/xiaozhi-esp32) | Arduino C++ | Generic multi-board voice assistant firmware (70+ target boards) | Runs on the same ESP32-S3 but is a different application — you pick one, not both | -Our pipeline uses `m5stack/StackChan` because it comes with the robot-body integration (servos, avatar rendering, LED patterns, MCP tools mapped to peripherals) already done. `78/xiaozhi-esp32` is the upstream *protocol* reference — the voice channel speaks the same WebSocket protocol regardless. +Our fork inherits M5Stack's robot-body integration (servos, avatar rendering, LED patterns, MCP tools mapped to peripherals) and adds Dotty's firmware behaviour. `78/xiaozhi-esp32` is the upstream *protocol* reference — the voice channel speaks the same WebSocket protocol regardless. ## On-device MCP tools @@ -91,7 +92,7 @@ The device acts as an **MCP server** — after the WS `hello` handshake, it adve Tool names follow the dotted-namespace convention from the `78/xiaozhi-esp32` MCP protocol doc (e.g. `self.audio_speaker.set_volume`, `self.get_device_status`). The **registration sites** in the firmware use `McpServer::AddTool` for public tools and `McpServer::AddUserOnlyTool` for privileged/hidden ones. -Per internal deployment observation, the live firmware advertises **11 tools**. The mapping below is from that observation plus the `m5stack/StackChan` README's feature list — **verify against the handshake logs** (`docker logs xiaozhi-esp32-server | grep tools/list`) before relying on exact tool names: +The live Dotty firmware advertises the upstream hardware tools plus Dotty's state/toggle additions. The mapping below is from deployment observation and the pinned fork — **verify against the handshake logs** (`docker logs xiaozhi-esp32-server | grep tools/list`) before relying on exact tool names: | # | Tool (functional) | Hardware touched | |---|---|---| diff --git a/docs/interaction-map.md b/docs/interaction-map.md index 492bb29..835aabe 100644 --- a/docs/interaction-map.md +++ b/docs/interaction-map.md @@ -9,7 +9,7 @@ One-page reference for every cross-layer signal in the Dotty stack. **Layers:** -1. **StackChan firmware** -- ESP32-S3 (m5stack/StackChan). The physical robot. +1. **StackChan firmware** -- ESP32-S3 (`BrettKinny/StackChan@dotty`, forked from `m5stack/StackChan`). The physical robot. 2. **xiaozhi-esp32-server** -- Docker on a Linux host. Voice I/O pipeline (ASR, TTS, VAD, emotion parsing). 3. **dotty-pi** -- the pi coding agent (Docker container on the same host). The LLM brain; reached by xiaozhi-server's `PiVoiceLLM` provider via `docker exec` pi RPC. (Ambient perception runs in a sibling `dotty-behaviour` container — see [architecture.md](./architecture.md).) diff --git a/docs/modes.md b/docs/modes.md index 4358dff..8730114 100644 --- a/docs/modes.md +++ b/docs/modes.md @@ -12,7 +12,12 @@ This document is the source of truth for Dotty's high-level modes. The model has The firmware **StateManager** modifier (`firmware/main/stackchan/modes/state_manager.{h,cpp}`) owns both axes. It paints the state arc (left ring 0-5) + toggle pips at 5 Hz, drives the idle-motion profile, and emits `state_changed` perception events on every transition. The **dotty-behaviour** perception bus (`dotty-behaviour/perception/state.py`) consumes those events and runs 11 consumer classes (the running set is config-gated) against them (`FaceGreeter`, `SoundTurner`, `FaceLostAborter`, `WakeWordTurner`, `FaceIdentifiedRefresher`, `PurrPlayer`, `SceneSynthesis`, `IdlePhotographer`, `SleepDreamer`, `DanceReflector`, `SecurityCycle`) — see [architecture.md](./architecture.md#perception-event-bus). -> **Submodule pin caveat.** Phase 4 shipped to the active firmware fork (`BrettKinny/StackChan @ dotty`, commit `d78118b`) on **2026-04-27**. The `firmware/firmware/` submodule pin in this repo deliberately lags upstream — it's a release pointer, not the active development tree. A user who flashes from the submodule will get a pre-Phase-4 firmware. Bump the submodule (or build from the active fork) to get the StateManager. Visual / interactive bench checks tracked in [issue #38](https://github.com/BrettKinny/dotty-stackchan/issues/38). +> **Firmware source.** Phase 4 first shipped on the Dotty firmware fork in +> commit `d78118b` (2026-04-27). This repo now pins a later +> `BrettKinny/StackChan@dotty` commit that includes StateManager, sleep, +> security, and current dance-motion ownership. Build the checked-out +> `firmware/firmware/` submodule for the contract documented here. Visual and +> interactive checks remain tracked in [issue #38](https://github.com/BrettKinny/dotty-stackchan/issues/38). Pair this with [hardware.md](./hardware.md) (the physical LED ring + servos) and [interaction-map.md](./interaction-map.md) (the underlying signals). @@ -78,22 +83,10 @@ The two toggles are orthogonal — they compose freely. `kid_mode = on` AND `sma ## LED contract (12-pixel ring) -!!! warning "Two firmwares, two right-ring layouts" - The contract below describes the **active-fork Phase 4 StateManager** - (`BrettKinny/StackChan @ dotty`). The firmware **submodule pinned in this - repo** (`35f701a`) does **not** include StateManager — it ships the - **privacy-LED** layout instead, which claims two of the same right-ring - pixels for a different purpose: - - - **pixel 6** = microphone indicator (green when the mic is open; pulsing - when audio is streaming to the server) - - **pixel 11** = camera indicator (red when the camera is capturing) - - These are bound to the codec/camera hardware via RAII guards (see the - firmware's `main/stackchan/privacy/PRIVACY_LEDS.md`). So if you flash from - the submodule, **pixels 6 and 11 mean mic/camera — not face-state and - listening.** The face-state pip, toggle pips, and listening pip described - below arrive once StateManager lands in the submodule pin. +> The table below describes the current pinned Dotty firmware. Older firmware +> pin `35f701a` used pixels 6 and 11 as standalone mic/camera privacy LEDs and +> did not contain StateManager; that historical layout is not the source-build +> target documented by this repo today. ``` LEFT RING (global 0–5) RIGHT RING (global 6–11) @@ -215,7 +208,7 @@ Phase 4 established the *rails* — pip, transition events, dispatch helpers, vo ## Sources of truth -- **Firmware (active fork `BrettKinny/StackChan @ dotty`):** `firmware/main/stackchan/modes/state_manager.{h,cpp}`, `firmware/main/stackchan/modifiers/face_tracking.cpp` (camera-edge hooks), `firmware/main/hal/hal_mcp.cpp` (set_state / set_toggle MCP). **This repo's submodule pin lags** — bump it (or maintain a parallel checkout per the [`firmware/`](../firmware) README) to flash a build that includes Phase 4+. +- **Firmware (pinned `BrettKinny/StackChan @ dotty` submodule):** `firmware/main/stackchan/modes/state_manager.{h,cpp}`, `firmware/main/stackchan/modifiers/face_tracking.cpp` (camera-edge hooks), `firmware/main/hal/hal_mcp.cpp` (set_state / set_toggle MCP). Build the checked-out submodule to reproduce the current firmware contract. - **Perception + ambient behaviour:** `dotty-behaviour/perception/state.py` (the perception event bus + per-device `current_state` from `state_changed`) and `dotty-behaviour/consumers/` (the 11 consumer classes — the running set is config-gated: `FaceGreeter`, `SoundTurner`, `FaceLostAborter`, `WakeWordTurner`, `FaceIdentifiedRefresher`, `PurrPlayer`, `SceneSynthesis`, `IdlePhotographer`, `SleepDreamer`, `DanceReflector`, `SecurityCycle`). The old `bridge.py` `_perception_*` / `_update_perception_state` / `_capture_room_view` methods are retired. - **Bridge:** `bridge.py` (admin dashboard + the `/admin/kid-mode` and `/admin/smart-mode` toggle relays), `receiveAudioHandle.py` (voice state phrases + per-conn toggle sync). The voice-path model-swap helpers (`_apply_model_swap`, `_apply_tier1slim_runtime`) are retired along with the Tier1Slim provider; smart-mode model-swap is v2 scope. - **xiaozhi-server patches:** `custom-providers/xiaozhi-patches/http_server.py` (`/xiaozhi/admin/set-state`, `/xiaozhi/admin/set-toggle`, `/xiaozhi/admin/set-face-identified`, `/xiaozhi/admin/inject-text`, `/xiaozhi/admin/abort`, `/xiaozhi/admin/set-head-angles`), `custom-providers/xiaozhi-patches/textMessageHandlerRegistry.py` (`state_changed` → `conn.current_state`, perception relay to dotty-behaviour) diff --git a/docs/quickstart.md b/docs/quickstart.md index f2b1895..00976e0 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -51,6 +51,18 @@ Download the latest release from `bootloader.bin`, `partition-table.bin`, `ota_data_initial.bin`, `stack-chan.bin`, `generated_assets.bin`, and `human_face_detect.espdl`. +> **Self-hosting note:** the current `fw-v1.3.3` prebuilt was compiled for +> the maintainer's LAN (`CONFIG_OTA_URL` points at that deployment). It is a +> reproducible release artifact, but it is not a portable binary for another +> self-hosted server. There is no OTA/server editor in the StackChan's +> on-device Settings app. For your own server, build from source with +> `CONFIG_OTA_URL="http://:8003/xiaozhi/ota/"` before flashing; +> [SETUP.md](../SETUP.md#2-build-and-flash-open-firmware) has the reliable +> procedure using this repo's pinned `BrettKinny/StackChan@dotty` submodule. +> Do not build the official `m5stack/StackChan` upstream directly: it does not +> include Dotty's firmware changes. A browser tab named **Advanced** exists +> only in the Xiaozhi hotspot portal. + Install esptool and flash over USB-C: ```bash @@ -141,10 +153,24 @@ No separate host, no systemd bridge unit, no SSH to a second machine. ## 6. Connect the robot -1. Power on the robot (USB-C or battery). -2. On the device screen, navigate to **Settings > Advanced Options**. -3. Enter the OTA URL: `http://:8003/xiaozhi/ota/` -4. The robot connects via WebSocket and shows a face. +1. Confirm the firmware was built with + `CONFIG_OTA_URL="http://:8003/xiaozhi/ota/"` as described in + [SETUP.md](../SETUP.md#2-build-and-flash-open-firmware). +2. Power on the robot (USB-C or battery). +3. On a fresh flash, **“Welcome! Let's get started.” is expected**. Tap + **Skip** to bypass the M5Stack account wizard. If the launcher remains, + open **SETUP** and use its home control to exit into Xiaozhi. +4. Join the temporary `Xiaozhi-*` hotspot, open the browser URL shown on the + robot (normally `http://192.168.4.1`), and select your 2.4 GHz Wi-Fi. + The portal's **Advanced** tab can confirm or override `ota_url`. +5. Watch `docker logs -f xiaozhi-esp32-server`. The robot should POST to the + configured OTA URL, receive the WebSocket endpoint, connect, and show a + face. + +The StackChan's on-device Settings app does **not** contain an Advanced +Options or OTA URL field. Wi-Fi credentials are provisioned into NVS through +the browser hotspot portal; `CONFIG_WIFI_SSID` and `CONFIG_WIFI_PASSWORD` are +not firmware options in the pinned source. ## 7. First voice turn @@ -228,7 +254,7 @@ The full file inventory lives in [architecture.md](./architecture.md#deployment- | What | URL | Who calls it | |---|---|---| -| OTA (enter into StackChan settings) | `http://:8003/xiaozhi/ota/` | The robot on boot | +| OTA (compile into firmware or set in the hotspot portal) | `http://:8003/xiaozhi/ota/` | The robot on boot | | WebSocket | `ws://:8000/xiaozhi/v1/` | The robot after OTA handshake | | Perception / ambient events | `http://:8090` | xiaozhi-server → dotty-behaviour | | Admin dashboard | `http://:8081/ui` | Humans (LAN-only HTMX UI) | diff --git a/docs/references.md b/docs/references.md index 261d035..9ace8e2 100644 --- a/docs/references.md +++ b/docs/references.md @@ -15,7 +15,8 @@ source-of-truth for re-verification. | Resource | URL | What's there | |---|---|---| | M5Stack StackChan product page | https://docs.m5stack.com/en/StackChan | Canonical kit-level reference. Part numbers, I2C addresses, pin map, servo angle limits, battery (550 mAh), assembled dims, in-box remote. | -| M5Stack StackChan README | https://github.com/m5stack/StackChan | Our firmware codebase. Body spec (servos, LEDs, NFC, IR, touch, battery). | +| Dotty StackChan firmware fork | https://github.com/BrettKinny/StackChan/tree/dotty | Pinned firmware source used by this repo. | +| M5Stack StackChan README | https://github.com/m5stack/StackChan | Upstream firmware and body spec (servos, LEDs, NFC, IR, touch, battery). | | M5Stack CoreS3 datasheet | https://docs.m5stack.com/en/core/CoreS3 | SoC, memory, display, camera, IMU, PMU, audio codec part numbers. | | meganetaaan/stack-chan (original) | https://github.com/meganetaaan/stack-chan | TypeScript/Moddable JS origin project — **we don't run this**, but it's the lineage. | | 78/xiaozhi-esp32 firmware (generic) | https://github.com/78/xiaozhi-esp32 | Multi-board voice-assistant firmware. Authority for the WS + MCP protocols below. | diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index e87850f..bae1453 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -67,7 +67,7 @@ Symptom-first lookup table covering common and obscure failure modes. Pair with **Fix:** 1. Check the bridge health endpoint: `curl http://:8081/health`. If the bridge is down, restart it. 2. Check xiaozhi-server logs: `docker logs -f xiaozhi-esp32-server` on the server. Look for connection attempts from the robot. -3. Verify the robot's OTA URL hasn't changed. After a firmware update, re-enter the OTA URL (`http://:8003/xiaozhi/ota/`) in the robot's Advanced Options if needed. +3. Verify which OTA URL the firmware was built with. The StackChan's on-device Settings app has no OTA URL field. For another self-hosted server, rebuild with `CONFIG_OTA_URL="http://:8003/xiaozhi/ota/"` and re-flash using [SETUP.md](../SETUP.md#2-build-and-flash-open-firmware). If the device is currently exposing the optional Xiaozhi SoftAP captive portal, its browser-based **Advanced** tab can persist an `ota_url` override instead. 4. Open the browser test page (`repo/main/xiaozhi-server/test/test_page.html`) and point it at `ws://:8000/xiaozhi/v1/`. If the browser page works but the robot doesn't, it's a robot-side configuration issue. --- diff --git a/dotty-behaviour/main.py b/dotty-behaviour/main.py index 0b5938d..52fdd4b 100644 --- a/dotty-behaviour/main.py +++ b/dotty-behaviour/main.py @@ -74,7 +74,13 @@ async def lifespan(app: FastAPI): # Singleton perception state — bus + caches + per-device dicts. # Stored on app.state so routes/consumers can retrieve it via # FastAPI's Request.app.state. - state = PerceptionState() + # Persist only the last state_changed mutex per device. This survives a + # daemon/container restart through the existing mounted STATE_DIR, but is + # not an authoritative sync for firmware changes made while this service + # was offline; the next relayed state_changed event remains authoritative. + state = PerceptionState( + state_path=config.STATE_DIR / "perception-state.json" + ) app.state.perception = state # Singleton dispatch clients — outbound HTTP to xiaozhi-server's diff --git a/dotty-behaviour/perception/state.py b/dotty-behaviour/perception/state.py index e5baab7..3be018c 100644 --- a/dotty-behaviour/perception/state.py +++ b/dotty-behaviour/perception/state.py @@ -14,9 +14,12 @@ class so the daemon can construct an isolated instance per test and import asyncio import collections +import json import logging +import os import time from dataclasses import dataclass +from pathlib import Path from typing import Any, Iterable from config import ( @@ -27,6 +30,10 @@ class so the daemon can construct an isolated instance per test and log = logging.getLogger("dotty-behaviour.perception") +_DURABLE_DEVICE_STATES = frozenset( + {"idle", "talk", "story_time", "security", "sleep", "dance"} +) + @dataclass(frozen=True) class PerceptionEvent: @@ -59,12 +66,13 @@ class PerceptionState: construct instances directly. """ - def __init__(self) -> None: + def __init__(self, *, state_path: str | Path | None = None) -> None: + self._state_path = Path(state_path) if state_path is not None else None # Event bus self._listeners: list[asyncio.Queue[PerceptionEvent]] = [] # Per-device state (face_present, listening, current_state, # last_*_t timestamps, dance_active, etc.) - self.state: dict[str, dict[str, Any]] = {} + self.state: dict[str, dict[str, Any]] = self._load_state() # Bounded per-device ring buffer of recent events (dashboard). self._recent: dict[ str, collections.deque[dict[str, Any]] @@ -89,6 +97,51 @@ def __init__(self) -> None: # the pattern). self._audio_waiters: dict[str, list[asyncio.Event]] = {} + def _load_state(self) -> dict[str, dict[str, Any]]: + if self._state_path is None: + return {} + try: + payload = json.loads(self._state_path.read_text(encoding="utf-8")) + if isinstance(payload, dict) and all( + isinstance(device_id, str) + and isinstance(device_state, dict) + and device_state.get("current_state") in _DURABLE_DEVICE_STATES + and isinstance( + device_state.get("last_state_change_t", 0.0), + (int, float), + ) + for device_id, device_state in payload.items() + ): + return payload + except (OSError, json.JSONDecodeError): + pass + return {} + + def _persist_state(self) -> None: + if self._state_path is None: + return + try: + self._state_path.parent.mkdir(parents=True, exist_ok=True) + temporary = self._state_path.with_name(f".{self._state_path.name}.tmp") + durable_state = { + device_id: { + "current_state": device_state["current_state"], + "last_state_change_t": device_state.get( + "last_state_change_t", 0.0 + ), + } + for device_id, device_state in self.state.items() + if isinstance(device_state.get("current_state"), str) + and device_state["current_state"] + } + temporary.write_text( + json.dumps(durable_state, separators=(",", ":")), + encoding="utf-8", + ) + os.replace(temporary, self._state_path) + except OSError as exc: + log.warning("failed to persist perception state: %s", exc) + # ------------------------------------------------------------------ # Bus # ------------------------------------------------------------------ @@ -191,6 +244,7 @@ def update_state( state["dance_active"] = True elif state.get("dance_active"): state["dance_active"] = False + self._persist_state() elif name == "dance_started": state["dance_active"] = True state["last_dance_started_t"] = ts diff --git a/dotty-behaviour/tests/test_calendar.py b/dotty-behaviour/tests/test_calendar.py index 8ba6197..90af109 100644 --- a/dotty-behaviour/tests/test_calendar.py +++ b/dotty-behaviour/tests/test_calendar.py @@ -6,6 +6,7 @@ from fastapi.testclient import TestClient +import config from calendar_ import ( CalendarCache, Event, @@ -143,9 +144,12 @@ def test_cache_set_and_flush() -> None: assert c.calendar_date == "2026-05-19" -def test_calendar_today_route_returns_shape() -> None: +def test_calendar_today_route_returns_shape(monkeypatch) -> None: """With CALENDAR_IDS unset (test env), the route should respond with the empty-cache shape and not blow up trying to fetch.""" + # Keep this route-shape test hermetic. WEATHER_LOCATION defaults to + # Brisbane in production, which would otherwise contact wttr.in. + monkeypatch.setattr(config, "WEATHER_LOCATION", "") with TestClient(app) as client: r = client.get("/api/calendar/today") assert r.status_code == 200 diff --git a/dotty-behaviour/tests/test_perception_state.py b/dotty-behaviour/tests/test_perception_state.py index 8bab2b4..cf976c9 100644 --- a/dotty-behaviour/tests/test_perception_state.py +++ b/dotty-behaviour/tests/test_perception_state.py @@ -10,10 +10,55 @@ import asyncio import time +from pathlib import Path from perception import PerceptionEvent, PerceptionState +def test_current_device_state_survives_daemon_reconstruction(tmp_path: Path) -> None: + state_file = tmp_path / "perception-state.json" + before_restart = PerceptionState(state_path=state_file) + before_restart.update_state( + "dev-1", "state_changed", {"state": "security"}, 100.0 + ) + + after_restart = PerceptionState(state_path=state_file) + + assert after_restart.current_device_state("dev-1") == "security" + + +def test_invalid_persisted_state_fails_safe_to_idle(tmp_path: Path) -> None: + state_file = tmp_path / "perception-state.json" + state_file.write_text('{"dev-1":"security"}', encoding="utf-8") + + restarted = PerceptionState(state_path=state_file) + + assert restarted.current_device_state("dev-1") == "idle" + + +def test_invalid_persisted_mutex_value_fails_safe_to_idle(tmp_path: Path) -> None: + state_file = tmp_path / "perception-state.json" + state_file.write_text('{"dev-1":{"current_state":7}}', encoding="utf-8") + + restarted = PerceptionState(state_path=state_file) + + assert restarted.current_device_state("dev-1") == "idle" + + +def test_restart_does_not_revive_transient_sensor_state(tmp_path: Path) -> None: + state_file = tmp_path / "perception-state.json" + before_restart = PerceptionState(state_path=state_file) + before_restart.update_state("dev-1", "face_detected", {}, 90.0) + before_restart.update_state( + "dev-1", "state_changed", {"state": "security"}, 100.0 + ) + + after_restart = PerceptionState(state_path=state_file) + + assert after_restart.current_device_state("dev-1") == "security" + assert "face_present" not in after_restart.state["dev-1"] + + def test_subscribe_broadcast_unsubscribe_roundtrip() -> None: async def go() -> None: ps = PerceptionState() diff --git a/scripts/patch-tts-server-push.py b/scripts/patch-tts-server-push.py new file mode 100644 index 0000000..73d0bf9 --- /dev/null +++ b/scripts/patch-tts-server-push.py @@ -0,0 +1,42 @@ +"""Patch pinned xiaozhi TTS consumers for explicit server-push arbitration.""" + +from pathlib import Path +import sys + + +OLD = "if message.sentence_id != self.conn.sentence_id:\n" +NEW = ( + "if (\n" + " message.sentence_id != self.conn.sentence_id\n" + " and message.sentence_id not in getattr(\n" + " self.conn, \"_dotty_server_push_sentence_ids\", set()\n" + " )\n" + " ):\n" +) + + +def patch_source(source: str) -> str: + if NEW in source: + return source + if OLD not in source: + raise ValueError("sentence-id predicate anchor not found") + return source.replace(OLD, NEW, 1) + + +def main(root: Path) -> None: + patched = 0 + for path in root.glob("*.py"): + source = path.read_text(encoding="utf-8") + if OLD not in source and NEW not in source: + continue + updated = patch_source(source) + if updated != source: + path.write_text(updated, encoding="utf-8") + patched += 1 + if patched == 0: + raise SystemExit("no xiaozhi TTS consumers contained the expected predicate") + print(f"patched {patched} TTS consumer(s) for server-push arbitration") + + +if __name__ == "__main__": + main(Path(sys.argv[1])) diff --git a/tests/test_device_command.py b/tests/test_device_command.py index 07007e9..e95dd82 100644 --- a/tests/test_device_command.py +++ b/tests/test_device_command.py @@ -10,10 +10,11 @@ import importlib.util as _ilu import json import pathlib +import queue import sys import types import unittest -from unittest.mock import MagicMock +from unittest.mock import MagicMock, patch _PATCHES = pathlib.Path(__file__).parent.parent / "custom-providers" / "xiaozhi-patches" @@ -231,6 +232,121 @@ def test_resolve_conn_falls_back_to_first_device_and_503s_when_empty(self): self.assertIsNone(got) self.assertEqual(err.status, 503) + def test_server_say_survives_chat_sentence_id_change(self): + """#104: a chat turn must not invalidate queued server speech.""" + class SentenceType: + FIRST = "first" + MIDDLE = "middle" + LAST = "last" + + class ContentType: + TEXT = "text" + + class DTO: + def __init__(self, **kwargs): + self.__dict__.update(kwargs) + + dto_module = types.SimpleNamespace( + ContentType=ContentType, + SentenceType=SentenceType, + TTSMessageDTO=DTO, + ) + saved = { + name: sys.modules.get(name) + for name in ( + "core.providers", "core.providers.tts", + "core.providers.tts.dto", "core.providers.tts.dto.dto", + ) + } + try: + for name in ("core.providers", "core.providers.tts", "core.providers.tts.dto"): + sys.modules[name] = types.ModuleType(name) + sys.modules["core.providers.tts.dto.dto"] = dto_module + + async def go(): + conn = _FakeConn() + conn.headers = {"device-id": "dev-say"} + conn.tts = types.SimpleNamespace(tts_text_queue=queue.Queue()) + type(self).active["dev-say"] = conn + async def run_inline(func): + return func() + + with patch.object(self.mod.asyncio, "to_thread", side_effect=run_inline): + await self._server()._dotty_say( + self._request({"text": "Hello from the server."}) + ) + + conn.sentence_id = "new-chat-turn" + accepted = [] + while not conn.tts.tts_text_queue.empty(): + message = conn.tts.tts_text_queue.get_nowait() + if ( + message.sentence_id != conn.sentence_id + and message.sentence_id not in getattr( + conn, "_dotty_server_push_sentence_ids", set() + ) + ): + continue + accepted.append(message) + + self.assertEqual( + [m.sentence_type for m in accepted], + [SentenceType.FIRST, SentenceType.MIDDLE, SentenceType.LAST], + ) + + asyncio.run(go()) + finally: + for name, module in saved.items(): + if module is None: + sys.modules.pop(name, None) + else: + sys.modules[name] = module + + def test_server_push_exemption_does_not_keep_stale_chat_ids(self): + conn = types.SimpleNamespace( + sentence_id="current-chat", + _dotty_server_push_sentence_ids={"server-push"}, + ) + + def accepted(sentence_id): + return not ( + sentence_id != conn.sentence_id + and sentence_id not in conn._dotty_server_push_sentence_ids + ) + + self.assertFalse(accepted("stale-chat")) + self.assertTrue(accepted("server-push")) + + def test_client_abort_still_drops_server_push(self): + conn = types.SimpleNamespace( + sentence_id="current-chat", + client_abort=True, + _dotty_server_push_sentence_ids={"server-push"}, + ) + + def accepted(sentence_id): + if conn.client_abort: + return False + return not ( + sentence_id != conn.sentence_id + and sentence_id not in conn._dotty_server_push_sentence_ids + ) + + self.assertFalse(accepted("server-push")) + + def test_build_patch_rewrites_consumer_predicate_explicitly(self): + script = pathlib.Path(__file__).parent.parent / "scripts" / "patch-tts-server-push.py" + spec = _ilu.spec_from_file_location("tts_push_patch_under_test", script) + module = _ilu.module_from_spec(spec) # type: ignore[arg-type] + spec.loader.exec_module(module) # type: ignore[union-attr] + source = ( + "if message.sentence_id != self.conn.sentence_id:\n" + " continue\n" + ) + updated = module.patch_source(source) + self.assertIn("_dotty_server_push_sentence_ids", updated) + self.assertIn("continue", updated) + def test_no_ms_truncated_ids_left_anywhere(self): # The collision-prone id pattern must not reappear in either # patch surface (the audit found it at twelve sites). diff --git a/tests/test_docs_onboarding.py b/tests/test_docs_onboarding.py new file mode 100644 index 0000000..fcf79f7 --- /dev/null +++ b/tests/test_docs_onboarding.py @@ -0,0 +1,84 @@ +"""Regression guards for the documented StackChan provisioning path.""" + +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] +ONBOARDING_FILES = ( + ROOT / "docs" / "quickstart.md", + ROOT / "SETUP.md", + ROOT / "docs" / "troubleshooting.md", + ROOT / "Makefile", +) + + +def test_docs_do_not_claim_ota_url_is_in_on_device_advanced_options() -> None: + combined = "\n".join(path.read_text(encoding="utf-8") for path in ONBOARDING_FILES) + stale_directions = ( + "device's Advanced Options", + "robot's Advanced Options", + "Settings > Advanced Options", + ) + for stale in stale_directions: + assert stale not in combined + + +def test_docs_identify_compiled_ota_url_and_missing_device_editor() -> None: + quickstart = (ROOT / "docs" / "quickstart.md").read_text(encoding="utf-8") + assert "CONFIG_OTA_URL" in quickstart + assert "does **not** contain an Advanced" in quickstart + assert "fw-v1.3.3" in quickstart + + +def test_source_build_uses_the_pinned_dotty_firmware_fork() -> None: + setup = (ROOT / "SETUP.md").read_text(encoding="utf-8") + gitmodules = (ROOT / ".gitmodules").read_text(encoding="utf-8") + + assert "github.com/BrettKinny/StackChan.git" in gitmodules + assert "git clone --recursive https://github.com/BrettKinny/dotty-stackchan.git" in setup + assert "cd dotty-stackchan/firmware/firmware" in setup + assert "git clone https://github.com/m5stack/StackChan.git" not in setup + + +def test_docs_only_recommend_real_firmware_kconfig_symbols() -> None: + setup = (ROOT / "SETUP.md").read_text(encoding="utf-8") + + assert "CONFIG_OTA_URL" in setup + assert "set `CONFIG_WIFI_SSID`" not in setup + assert "set `CONFIG_WIFI_PASSWORD`" not in setup + assert "does **not** define `CONFIG_WIFI_SSID`" in setup + + +def test_docs_explain_the_real_first_boot_path() -> None: + setup = (ROOT / "SETUP.md").read_text(encoding="utf-8") + quickstart = (ROOT / "docs" / "quickstart.md").read_text(encoding="utf-8") + + assert "Skip" in setup + assert "Welcome!" in setup + assert "Xiaozhi" in setup and "hotspot" in setup.lower() + assert "Skip" in quickstart + + +def test_current_docs_do_not_claim_the_firmware_submodule_lacks_state_manager() -> None: + current_docs = ( + (ROOT / "README.md").read_text(encoding="utf-8") + + (ROOT / "docs" / "modes.md").read_text(encoding="utf-8") + + (ROOT / "CLAUDE.md").read_text(encoding="utf-8") + ) + stale_claims = ( + "submodule pin in this repo lags", + "submodule pin lags", + "flashing from the submodule won't give you Phase 4", + "does **not** include StateManager", + "xiaozhi firmware (built from m5stack/StackChan source)", + ) + for stale in stale_claims: + assert stale not in current_docs + + +def test_compatibility_doc_matches_the_current_public_contract() -> None: + compatibility = (ROOT / "COMPATIBILITY.md").read_text(encoding="utf-8") + assert "`fw-v1.3.3`" in compatibility + assert "the seven `dotty-pi-ext` voice tools" in compatibility + assert "No formal versioning is adopted yet" not in compatibility + assert "scripts/backup.sh" not in compatibility