Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,21 @@ sbom-server.json
htmlcov/
.pytest_cache/

# Local UAT evidence. Keep compact manifests/results/draft issue summaries
# visible for selective review, but never offer raw household media, event
# streams, health snapshots, or container logs for an accidental commit.
/uat-sessions/**/video/
/uat-sessions/**/clips/
/uat-sessions/**/logs/
/uat-sessions/**/ndjson/
/uat-sessions/**/snapshots/

# Generated audit/session outputs. These can contain machine-local paths,
# prompts, or deployment observations; regenerate or curate before publishing.
/AUDIT-REPORT.md
/audit-draft-issues.md
/pi-session-*.html

# Frontend build deps for the dashboard's vendored Tailwind bundle.
# package.json + tailwind.config.js are committed; build artifacts and
# the npm install state are not. Re-run `npm install && npm run build:css`
Expand Down
5 changes: 3 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ The voice path runs through a single LLM provider — `PiVoiceLLM`, selected via

```
StackChan hardware → configured persona
│ ESP32-S3, xiaozhi firmware (built from m5stack/StackChan source)
│ ESP32-S3, xiaozhi firmware (pinned BrettKinny/StackChan@dotty fork)
│ WiFi / WebSocket (Xiaozhi protocol)
xiaozhi-esp32-server (Docker)
Expand Down Expand Up @@ -197,7 +197,8 @@ For hardware specs, protocol details, model internals, latent capabilities, and

- xiaozhi-esp32-server: https://github.com/xinnan-tech/xiaozhi-esp32-server
- xiaozhi-esp32 firmware (upstream): https://github.com/78/xiaozhi-esp32
- StackChan (hardware + firmware patches): https://github.com/m5stack/StackChan
- StackChan upstream (hardware + base firmware): https://github.com/m5stack/StackChan
- Dotty firmware fork (the pinned build source): https://github.com/BrettKinny/StackChan/tree/dotty
- Emotion protocol: https://xiaozhi.dev/en/docs/development/emotion/

## Agent skills
Expand Down
38 changes: 22 additions & 16 deletions COMPATIBILITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,16 @@ For protocol wire formats see `docs/protocols.md`.

| Component | Current Version | Protocol / Interface | Breaking Change Policy |
|---|---|---|---|
| StackChan firmware (m5stack/StackChan v1.2.4) | v1.2.4 | Xiaozhi WebSocket protocol, MCP over WS (JSON-RPC 2.0) | Pin firmware to a known-good build; do not OTA-update without verifying server compatibility first |
| StackChan firmware (`BrettKinny/StackChan@dotty`) | `fw-v1.3.3` release; submodule `969c2b2` | Xiaozhi WebSocket protocol, MCP over WS (JSON-RPC 2.0), StateManager event contract | Build the pinned submodule; do not substitute the official upstream tree or OTA-update without coordinated verification |
| xiaozhi-esp32-server (local build) | `xiaozhi-esp32-server-piper:local` | Custom LLM provider API, `.config.yaml` schema, Xiaozhi WS server | Rebuild image only after checking upstream changelog for provider API or config schema changes |
| dotty-pi (pi agent) | `dotty-pi:0.1.0` | pi RPC (JSONL over stdio), the five `dotty-pi-ext` voice tools | Pin the image tag; pi-version or model changes need end-to-end cutover testing |
| dotty-pi (pi agent) | `dotty-pi:0.1.0` | pi RPC (JSONL over stdio), the seven `dotty-pi-ext` voice tools | Pin the image tag; pi-version or model changes need end-to-end cutover testing |
| dotty-behaviour | `dotty-behaviour:0.1.0` | HTTP API (`/api/perception/*`, `/api/vision/*`, `/api/audio/*`, `/health`) | Endpoint signatures stable; perception event-schema changes require firmware + xiaozhi review |
| bridge.py (dashboard) | unversioned (HEAD) | `/ui` dashboard, `/admin/*`, `/health` | Dashboard/admin service only post-#36; admin route changes require updating dashboard callers |
| bridge.py (dashboard) | `dotty-bridge:0.1.0` image from repo HEAD | `/ui` dashboard, `/admin/*`, `/health` | Dashboard/admin service only post-#36; admin route changes require updating dashboard callers |

The public `fw-v1.3.3` superproject tag (commit `24a009c`, 2026-07-12) is the
current coordinated release pointer. It pins firmware submodule `969c2b2` and
the matching server-side patches. Draft PRs and a dirty submodule are not a
released compatibility set.

## What counts as a breaking change

Expand All @@ -42,27 +47,28 @@ Any of the following require coordinated updates across components:

## Versioning strategy

No formal versioning is adopted yet (tracked in
[ROADMAP.md](ROADMAP.md#community-wishlist) under "Firmware/server
compatibility matrix"). When adopted, the plan is:
The repo uses separate tag namespaces:

- `server-vX.Y.Z` for server-only release milestones.
- `fw-vX.Y.Z` for coordinated firmware release pointers in this superproject.

- Separate tag namespaces: `server-vX.Y.Z` and `fw-vX.Y.Z`.
- This matrix will document which server versions are compatible with which
firmware versions.
- The bridge will carry its own version once it moves to a tagged release
cadence.
Container image tags remain `0.1.0` today and are not sufficient by themselves
to identify the exact source revision; retain the Git commit/tag alongside a
deployment record.

## Upgrade guidance

1. **Check this matrix first.** Confirm the component you are upgrading is
compatible with the versions of the other components you are running.
2. **Back up before upgrading.** Run `scripts/backup.sh` (or the equivalent
manual steps) to snapshot config, persona files, and bridge state.
2. **Back up before upgrading.** Manually snapshot `.env`, rendered
`data/.config.yaml`, persona/household files, `brain.db` (including WAL/SHM
companions), and the bridge state directory. This repo does not currently
ship an automated backup script.
3. **Upgrade one component at a time.** Validate with a health check
(`curl http://<XIAOZHI_HOST>:8090/health` and `:8081/health`) plus a live
voice turn before moving to the next component.
4. **Tail logs during validation.** Watch both the xiaozhi-server container
logs and the bridge journal simultaneously to catch mismatches early.
4. **Tail logs during validation.** Watch the xiaozhi-server, dotty-pi,
dotty-behaviour, and bridge container logs together to catch mismatches.
5. **Roll back if broken.** Restore from the backup taken in step 2 and
revert to the previous image or binary.

Expand Down Expand Up @@ -106,4 +112,4 @@ versions work with which firmware versions.

---

Last verified: 2026-05-22.
Last verified against the repository and `fw-v1.3.3` pin: 2026-07-16.
5 changes: 5 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,11 @@ FROM ghcr.io/xinnan-tech/xiaozhi-esp32-server@sha256:3accd82a7d1a6c01c58f32f6199

RUN pip install --no-cache-dir piper-tts scipy numpy mido faster-whisper sherpa-onnx==1.13.2

# Patch upstream TTS consumers to treat explicitly registered server-push
# sentence IDs as independent from chat-turn stale-message arbitration (#104).
COPY scripts/patch-tts-server-push.py /tmp/patch-tts-server-push.py
RUN python /tmp/patch-tts-server-push.py /opt/xiaozhi-esp32-server/core/providers/tts

# fluidsynth + General MIDI soundfont for runtime rendering of dance/song MIDI
# files to Opus. Installed as the LAST layer so iteration on Python deps above
# doesn't invalidate the soundfont download (~141MB).
Expand Down
7 changes: 4 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -221,9 +221,10 @@ setup: _preflight-compose ## Interactive first-run wizard (re-runnable; remember
echo -e "$(GREEN)$(BOLD)Setup complete.$(RESET)"; \
echo ""; \
echo "Next steps:"; \
echo " 1. Flash the StackChan firmware (see SETUP.md or m5stack/StackChan repo)."; \
echo " 2. In the device's Advanced Options, set the OTA URL to:"; \
echo " http://$$XIAOZHI_HOST:8003/xiaozhi/ota/"; \
echo " 1. Build and flash StackChan firmware with this compiled setting:"; \
echo " CONFIG_OTA_URL=\"http://$$XIAOZHI_HOST:8003/xiaozhi/ota/\""; \
echo " See SETUP.md. The on-device Settings app has no OTA URL editor."; \
echo " 2. Provision the robot's 2.4 GHz Wi-Fi using its displayed setup flow."; \
echo " 3. Run 'make doctor' to verify everything is healthy."; \
echo ""

Expand Down
14 changes: 9 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ Full policy: [`AI_TRANSPARENCY.md`](./AI_TRANSPARENCY.md).
- **Streaming responses** — the bridge streams LLM output to the voice pipeline for lower perceived latency.
- **Emoji expressions** — every response starts with an emoji that the firmware maps to a face animation (smile, laugh, sad, surprise, thinking, angry, love, sleepy, neutral).
- **Voice tools** — the pi agent can search its memory, escalate hard questions to a bigger model, take a photo, and play songs, all mid-conversation.
- **States, toggles & LEDs** — a six-state mutex (`idle / talk / story_time / security / sleep / dance`) plus two orthogonal toggles (`kid_mode`, `smart_mode`), all owned by the firmware StateManager and surfaced on the 12-pixel LED ring. Shipped on the active firmware fork (commit `d78118b`, 2026-04-27); the `firmware/firmware/` submodule pin in this repo lags, so flash from the active fork to get it. See "States, Toggles & LEDs" below and [`docs/modes.md`](./docs/modes.md).
- **States, toggles & LEDs** — a six-state mutex (`idle / talk / story_time / security / sleep / dance`) plus two orthogonal toggles (`kid_mode`, `smart_mode`), all owned by the firmware StateManager and surfaced on the 12-pixel LED ring. The pinned `BrettKinny/StackChan@dotty` firmware submodule includes the StateManager and current dance-motion ownership fixes. See "States, Toggles & LEDs" below and [`docs/modes.md`](./docs/modes.md).
- **Vision (camera)** — the robot's built-in camera can capture images for multimodal LLM queries.
- **Privacy LEDs** — hardware-bound mic (green) and camera (red) indicators on the LED ring. They light from the codec/camera enable signals via RAII guards, so a misbehaving server or model can't capture with the lights off.
- **Calendar context** — optional calendar integration feeds upcoming events into the conversation context.
Expand All @@ -46,7 +46,9 @@ Full policy: [`AI_TRANSPARENCY.md`](./AI_TRANSPARENCY.md).

Behaviour is a **six-state mutex** (`idle / talk / story_time / security / sleep / dance`) plus two orthogonal toggles (`kid_mode`, `smart_mode`), all owned by the firmware StateManager (shipped on the active fork in commit `d78118b`, 2026-04-27; bench checks tracked in [#38](https://github.com/BrettKinny/dotty-stackchan/issues/38)). Voice phrases, camera edges, and dashboard controls all flow through it.

> Note: the `firmware/firmware/` submodule pin in this repo deliberately lags the active fork — flashing from the submodule won't give you Phase 4 yet. See the "Firmware iteration" section in [`CLAUDE.md`](./CLAUDE.md) and the submodule-pin caveat in [`docs/modes.md`](./docs/modes.md).
Build firmware from this repo's pinned `firmware/` submodule. It points at the
Dotty fork and is the reproducible source used for the current release; the
official `m5stack/StackChan` repository remains the upstream project.

The 12-pixel LED ring shows the current state at a glance. **Left ring 0-5 is the state arc** — all six pixels paint the state colour, matching the dashboard's state buttons:

Expand All @@ -61,7 +63,9 @@ The 12-pixel LED ring shows the current state at a glance. **Left ring 0-5 is th

On the right ring, **indices 8-9 are toggle pips** for kid_mode (salmon pink) and smart_mode (orange), and **index 11 (bottom) lights red while you have the turn** (LISTENING). The `idle → talk` transition fires on `face_detected` from the firmware; VLM identity recognition runs in parallel and feeds the LLM context.

> Heads up: that right-ring layout is the active-fork StateManager. On the firmware **submodule pinned in this repo**, pixels 6 and 11 instead drive the **privacy LEDs** — 6 = mic (green), 11 = camera (red) — and the StateManager pips arrive once the submodule catches up to the active fork.
The current pin includes the StateManager right-ring layout. Privacy indication
is still hardware-bound in the firmware, but it no longer uses the older
pre-StateManager pixel contract described by historical releases.

Full state taxonomy, colour palette, transition diagram, and per-state backing architecture: [`docs/modes.md`](./docs/modes.md).

Expand All @@ -77,15 +81,15 @@ The dashboard service serves a web dashboard at `http://<XIAOZHI_HOST>:8081/ui`

## Reference deployment

- **Hardware**: M5Stack StackChan (CoreS3 + servo kit), firmware built from `m5stack/StackChan`.
- **Hardware**: M5Stack StackChan (CoreS3 + servo kit), firmware built from the pinned `BrettKinny/StackChan@dotty` fork of `m5stack/StackChan`.
- **Brain**: a **pi** coding agent running in the `dotty-pi` container. It runs `qwen3.5:4b` on local [llama-swap](./docs/cookbook/llama-swap-concurrent-models.md) for the conversation loop and escalates hard questions to `qwen3.6:27b-think` (also on llama-swap) via its `think_hard` tool. xiaozhi-server's `PiVoiceLLM` provider hands each voice turn to the agent. See [`docs/brain.md`](./docs/brain.md).
- **Voice I/O**: xiaozhi-esp32-server on Docker (any Linux Docker host).

## What runs where

| Component | Host | Notes |
|---|---|---|
| StackChan (device) | ESP32-S3 on the desk | Firmware built from `m5stack/StackChan` (see `SETUP.md`) |
| StackChan (device) | ESP32-S3 on the desk | Firmware built from the pinned `BrettKinny/StackChan@dotty` submodule (see `SETUP.md`) |
| xiaozhi-esp32-server | server (`<XIAOZHI_HOST>`) | Docker — voice I/O, ports 8000 + 8003 |
| dotty-pi | server (`<XIAOZHI_HOST>`) | Docker — the pi agent, Dotty's voice brain |
| dotty-behaviour | server (`<XIAOZHI_HOST>`) | Docker — FastAPI: perception bus, ambient consumers, vision, greeter; port 8090 |
Expand Down
91 changes: 56 additions & 35 deletions SETUP.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,45 +48,59 @@ There is **no SoftAP captive portal** in stock firmware (some older
third-party xiaozhi builds had one; the shipped firmware does not).

To run the StackChan **fully self-hosted** (no phone-app account, no vendor cloud,
your own xiaozhi-server as the endpoint), you need to **reflash the device
with firmware built from the open source tree**.

The upstream firmware lives at **https://github.com/m5stack/StackChan**:
- `firmware/` — M5Stack's patches + ESP-IDF project wrapper
- `firmware/fetch_repos.py` — pulls `78/xiaozhi-esp32` as a dependency and
applies StackChan-specific patches, adding the board target
`CONFIG_BOARD_TYPE_M5STACK_STACK_CHAN`
your own xiaozhi-server as the endpoint), reflash it with Dotty's pinned
`BrettKinny/StackChan@dotty` firmware fork. The official `m5stack/StackChan`
tree is its upstream, but it does not contain Dotty's state, motion, LED, and
perception changes.

The StackChan's on-device Settings app has no **Advanced Options** or OTA URL
editor. The similarly named **Advanced** tab documented by Xiaozhi belongs to
its browser-based SoftAP captive portal and appears only when that provisioning
mode is active. The current `fw-v1.3.3` prebuilt is compiled for the
maintainer's LAN, so another self-hosted deployment must build from source with
its own `CONFIG_OTA_URL` as shown below.

The exact firmware source is the `firmware/` submodule of this repo. It pins
`BrettKinny/StackChan@dotty`, whose `firmware/fetch_repos.py` pulls
`78/xiaozhi-esp32` v2.2.4 and applies the StackChan integration patch.

---

## 2. Build and flash open firmware

> **Note — build flow documented from first-pass session findings; not yet
> end-to-end verified. Will be updated after a successful first flash.**

Requires **ESP-IDF v5.5.4**. Easiest path: the official
`espressif/idf:v5.5.4` Docker image.

### 2a. Clone and configure

```bash
git clone https://github.com/m5stack/StackChan.git
cd StackChan/firmware
git clone --recursive https://github.com/BrettKinny/dotty-stackchan.git
cd dotty-stackchan/firmware/firmware
```

Point the firmware at your xiaozhi-server for OTA. Edit
`firmware/sdkconfig.defaults` and add (or modify) the line:
`sdkconfig.defaults` and add (or modify) the line:

```
CONFIG_OTA_URL="http://<XIAOZHI_HOST>:8003/xiaozhi/ota/"
```

Trailing slash matters — that's the path the server exposes.

`sdkconfig.defaults` seeds a newly generated `sdkconfig`; it does not overwrite
an existing one. On a repeat build, either make the same change through
`idf.py menuconfig` (**Xiaozhi Assistant → Default OTA URL**) or remove the
generated `sdkconfig` before rebuilding. Verify the effective value after
configuration with:

```bash
grep '^CONFIG_OTA_URL=' sdkconfig
```

### 2b. Build inside the IDF container

```bash
docker run --rm -it -v "$PWD/..":/project -w /project/firmware \
docker run --rm -it -v "$PWD":/project -w /project \
espressif/idf:v5.5.4 \
bash -c "python3 fetch_repos.py && idf.py build"
```
Expand All @@ -101,7 +115,7 @@ Linux (`/dev/cu.usbmodem*` on macOS — adapt the `--device` flag).

```bash
docker run --rm -it --device=/dev/ttyACM0 \
-v "$PWD/..":/project -w /project/firmware \
-v "$PWD":/project -w /project \
espressif/idf:v5.5.4 \
idf.py -p /dev/ttyACM0 flash
```
Expand All @@ -111,14 +125,23 @@ flow), use that instead.

### 2d. First boot after flash

- No pairing-code screen
- No BLE provisioning step
- The device boots, loads WiFi credentials compiled into the firmware
(or, if you left WiFi unconfigured, whatever fallback the upstream
build offers — consult the xiaozhi-esp32 README for the current default
behaviour)
- It POSTs to `http://<XIAOZHI_HOST>:8003/xiaozhi/ota/`, gets back the
WebSocket endpoint, and connects
Seeing **“Welcome! Let's get started.” is expected on a fresh flash**. It is
the StackChan app-configuration wizard, controlled by an NVS flag; it does not
mean the compiled `CONFIG_OTA_URL` was ignored.

For a self-hosted setup:

1. Tap **Skip** on the welcome screen. This bypasses the M5Stack account/BLE
wizard for the current boot.
2. If the launcher remains visible, open **SETUP**, then use its home control
to exit. Closing SETUP starts the Xiaozhi voice application.
3. With no saved SSID, the pinned build enters its default Xiaozhi hotspot
provisioner. Join the temporary `Xiaozhi-*` Wi-Fi network and open the URL
shown on the robot (normally `http://192.168.4.1`).
4. Select the robot's 2.4 GHz Wi-Fi network. The browser portal also has an
**Advanced** tab where you can confirm or override `ota_url`.
5. After Wi-Fi connects, the device POSTs to the compiled/persisted OTA URL,
receives the WebSocket endpoint, and connects.

Tail the server logs while the device boots so you can watch the
handshake happen (see step 4 below).
Expand All @@ -127,18 +150,13 @@ handshake happen (see step 4 below).

## 3. WiFi credentials

Two options, depending on what the upstream xiaozhi-esp32 build exposes at
the version you pulled:
The pinned firmware does **not** define `CONFIG_WIFI_SSID` or
`CONFIG_WIFI_PASSWORD`; adding those names to `sdkconfig.defaults` has no
effect. Wi-Fi credentials live in NVS and are supplied through the Xiaozhi
hotspot portal described above (or retained from an earlier compatible
provisioning).

- **Compile-time WiFi credentials** — set `CONFIG_WIFI_SSID` and
`CONFIG_WIFI_PASSWORD` in `sdkconfig.defaults`. Simplest for a static
home setup; easy to forget they're in the binary.
- **Fallback SoftAP or BLE provisioning** — some upstream builds include a
fallback provisioning flow if no credentials are saved. Check the
upstream README for what your commit supports.

Either way, the device must land on a **2.4 GHz** network. ESP32-S3 does
not do 5 GHz.
The device must land on a **2.4 GHz** network. ESP32-S3 does not do 5 GHz.

---

Expand Down Expand Up @@ -234,6 +252,9 @@ Some older xiaozhi builds expose a SoftAP captive portal on first boot:
If you have a build where this works, it's the fastest provisioning flow.
It just isn't what M5Stack ships today.

This **Advanced** control is a tab in the browser portal at `192.168.4.1`, not
an item in the StackChan's on-device Settings app.

---

## 9. When it's working: bookmark these
Expand Down
Loading
Loading