feat(cli): larql speak-serve — keep the speech model resident

[sam-shepp]

Summary

run --speak pays the whole load bill for one utterance: safetensors load, the Q4_K
quantisation pass, tokenizer. Measured on an M-series 32 GB machine, a 3.92 s utterance
cost 33.3 s wall for ~5 s of actual work — model load 12.2 s, quantisation 4.1 s,
generation 4.8 s, external codec 11.7 s. Roughly 26 s of every invocation was startup,
paid again each time.

speak-serve pays it once and then serves utterance after utterance against the hot model.
That is what the existing API was already shaped for — MossSpeech is "the loaded model
borrowed as one unit" and MossSession is per-utterance
(larql-inference/src/speech/moss_session.rs) — so this is a loop around that seam rather
than new inference machinery.

Result: 33.3 s → 5.3 s steady state per utterance, with generation now ~95 % of what
remains. With the per-frame stream: path, first sound at 2.0–2.1 s on a 7.76 s
utterance, 0 underruns.

Where this sits on the roadmap

This is the process-level step toward docs/tts-funnel.md §5's "keep the speech
machine hot", and it is deliberately not the realtime runtime that section describes:
no PCM ring buffer, no audio callback, and the codec stays external (§6 defers
codec-in-LARQL). It is also the "server-side home for a persistent engine" that §7 lists
under session/turn state.

It gives the post-step-5 perf gate (first-turn TTFA below 500 ms) a measurement surface:
every reply carries first_frame=Nms, and stderr carries the prefill/steady split.

What's in the diff

File	Change
commands/primary/speak_serve.rs	New. The subcommand.
commands/primary/mod.rs	pub mod speak_serve;
main.rs	SpeakServe variant + dispatch arm (6 lines)
commands/primary/run_cmd_speak.rs	read_token_rows / write_token_rows widened fn → pub(super)

Nothing existing changes behaviour. The two widened fns are shared rather than
reimplemented, so run --speak stays the reference for the token-row format and the two
paths cannot drift.

Protocol

One request is a block of key: value lines terminated by a blank line (or EOF); text:
is the only required key. Every other key falls back to the startup flag, so the common
case is one line.

text: Good evening Sam.
voice: /path/to/jarvis.tokens
tokens: /path/to/out.tokens.txt
stream: /path/to/frames.fifo
seed: 5
max-frames: 200
reply: /path/to/reply.fifo

Reply is a single line — ok frames=N audio=Ns gen=Ns first_frame=Nms or err <message> —
on reply: when given, else stdout.

Design points worth calling out, each of which was earned rather than guessed:

• Unknown keys are rejected, not ignored. A typo'd voise: would otherwise silently
  synthesise the default voice and look like a model bug.
• voice: - is an explicit "unconditioned", distinct from an absent key (which inherits
  the server default). So switching voices needs no restart.
• stream: flushes per frame, and closing the FIFO is the end-of-utterance signal —
  no in-band terminator that could be mistaken for a frame.
• Steady-state rate is reported separately from end-to-end. Prefill is a fixed cost a
  short utterance amortises badly, and the end-to-end number flatters or damns a run
  depending only on its length. (Related: an early Metal reading here looked like a 15 %
  win and was warm-up contamination — the same trap tts-funnel.md records.)
• Requests serialise. One model, one utterance at a time.

Placement — a CLI subcommand now, larql-server later

Flagging the structural question up front, since it's the obvious one: for always-on
serving this belongs in larql-server, that being the inference server. No argument — treat
this CLI subcommand as the near-term shape rather than the destination, and move it when the
server-side session/turn-state work (docs/tts-funnel.md §7) gives it a proper home.

Why it's still worth landing here first:

• The load-bearing claim is the amortisation (33.3 s → 5.3 s), and that's independent of
  transport. Proving it in the smallest surface keeps this to 10 lines outside one new file,
  and makes the eventual move a transport swap rather than a rewrite.
• Today's consumer is a local shell script on the same machine, where a request is
  printf 'text: hi\n\n' > fifo — no client library, no port, no JSON.
• The stream: path needs per-frame flush latency with nothing between generation and
  the player's decoder. Better that constraint informs the server route's design than gets
  discovered after it.
• It keeps tokio and a network stack off a CLI path that has neither.

The reusable half is serve_one: resident model in, one utterance out, no I/O policy of its
own beyond the paths it's handed. An HTTP or gRPC route should be able to call it more or less
unchanged — the FIFO/stdin plumbing is the throwaway half. Move or rename freely.

The one design question: --voice and roadmap P1

speak-serve --voice <TOKENS> takes a path to a token-rows file, matching run --speak
today. ROADMAP P1 ("Voice bank") plans larql speak --voice aru-12 "…" resolving a named
voice from a bank, and says outright that maintaining aru-12.tokens as a manually prepared
magic file is "beneath the abstraction level of the rest of the system" — i.e. exactly what
this flag currently takes.

I've kept the path form so this is consistent with the command it amortises, rather than
inventing half a voice bank. The intended seam: when voice clone / voice list land,
--voice and the per-request voice: key resolve a name through the bank and this
subcommand doesn't otherwise change. Flagging it so it's a decision rather than a
fait accompli.

Portability

The FIFO paths are Unix-only and refuse with a clear message elsewhere; the stdin request
path and tokens: output work on every platform, so the amortisation is available
everywhere. --listen needs mkfifo(3), and reply: needs a non-blocking write-open to
be safe against a dead client:

A blocking write-open on a FIFO with no reader never returns. Interrupting a client after
generation had finished left the daemon blocked forever — 0 % CPU, no error, every
later request stuck behind a pipe nobody drained. It took the whole speech path down until
a restart. The reply FIFO is now opened O_NONBLOCK and retried to a 10 s deadline
(short deliberately: a client opens its read end before generation even finishes, so if
nobody is there seconds later, nobody is coming). A dead client now costs one dropped
reply and a log line.

Tests

19 unit tests over the parsing and reporting surface — Request::absorb (every key, unknown-key
rejection, the voice: - sentinel, unparseable numbers, trimming), drain (block splitting,
trailing block at EOF, and that neither a malformed block nor a failing request takes the
server down), stage_split (medians, and that frame 0's absent backbone time doesn't drag
them down), truncate (multibyte safety).

Verification

• cargo test -p larql-cli --bins speak_serve — 19 passed.
• cargo fmt -p larql-cli -- --check — clean.
• cargo clippy -p larql-cli --bins --tests --no-deps -- -D warnings — clean.
• Non-Unix branch typechecked by inverting the cfgs locally (no Windows host available);
  clean apart from the artificial cfg warnings that inversion introduces.
• Parity, not assumed: same text + seed + voice through the resident path produces
  bit-identical tokens to the one-shot run --speak path.

Measurements

M-series, 32 GB, MOSS-TTS-Realtime, --q4.

	One-shot run --speak	speak-serve
Per utterance (3.92 s audio)	33.3 s	5.3 s
Model load	12.2 s, every time	12.2 s, once
Q4_K quantisation	4.1 s, every time	4.1 s, once
First sound (7.76 s utterance, stream:)	—	2.0–2.1 s, 0 underruns

Steady state on a 7.76 s utterance: prefill 1.45 s fixed, then 41 ms/frame = 1.95×
realtime. Inside a frame, depth is ~70 % (28–34 ms vs backbone 12 ms) — consistent
with §5 naming a fused moss_depth_frame as the steady-state target.

Two null/negative results, recorded so they don't get rediscovered:

• --metal changes nothing measurable in the speech path today. CPU 1.45–1.54 s prefill
  / 39–44 ms per frame vs Metal 1.44–1.66 s / 40–44 ms; backbone p50 identical at 12 ms. So
  §5's "Metal FFN ⇒ TTFA ≈ 550 ms" remains a projection for an unbuilt kernel.
• Residency is only as durable as spare RAM. Running a 4B LLM in front of the daemon
  pages MOSS out: the next utterance measured first frame 13.3 s / 147 ms per frame against
  1.5 s / 52 ms, putting first sound at 33 s. Re-speaking with no LLM in front returned
  straight to 1.5 s / 52 ms. Anything wanting both models resident needs to re-warm.

Not included

Barge-in / cancellation (§5's 100 ms budget) needs a cancel path rather than more speed.
Voice-prefix KV caching — identical for every utterance in a voice and currently recomputed
each time — is the next amortisation win and is not attempted here.

---

Unrelated: pre-existing red CI on main, same root cause as #256

Not from this PR, and already reported — adding one data point rather than a new issue. The
--routed-from --metal block in run_cmd.rs references larql_compute_metal with no gating,
and that single omission is reachable from two directions:

• Build breaks at larql-compute-metal on ubuntu #256's — Linux with default features, where the crate is linked but its items are
  #[cfg(target_os = "macos")]-gated, so route_witness isn't found.
• The other side — macOS with --no-default-features, where the crate isn't linked at all:

error[E0433]: cannot find module or crate `larql_compute_metal` in this scope
    --> crates/larql-cli/src/commands/primary/run_cmd.rs:1034:23
error[E0432]: unresolved import `larql_compute_metal`
    --> crates/larql-cli/src/commands/primary/run_cmd.rs:1042:13

larql-compute-metal is optional = true (larql-cli/Cargo.toml:17), pulled in only by
feature gpu, and default = ["gpu"] — so macOS with default features is the one
configuration where this block compiles, which is presumably why it slipped through.
Introduced by 7da7df35. Worth noting that a #[cfg(feature = "gpu")] alone won't close
both: #256's Linux case needs target_os = "macos" too, and per that thread the
let backend = … binding then falls out of scope at run_cmd.rs:1054.

Mentioning it here only because .github/workflows/larql-cli.yml runs the
--no-default-features clippy step on all three OSes, so this PR's checks may show a red job
that predates the branch.