H-6763: Speak finalized Brunch responses with OpenAI

[kostandinang]

🌟 What is the purpose of this PR?

Speak finalized Brunch responses through OpenAI's dedicated Speech API. The exact canonical text already visible in Petrinaut—including a validated pending brunch_ask question—is selected once, sent unchanged for synthesis, and played while the microphone remains closed.

OpenAI Realtime continues to perform transcription only; it never generates assistant responses.

🔗 Related links

• Linear: H-6763 (internal)
• Depends on #9356
• Stack: #9346 → #9355 → #9356 → H-6763: Speak finalized Brunch responses with OpenAI #9357
• #9297 is historical evidence only and is unchanged.

🚫 Blocked by

• #9356 and its downstack prerequisites

🔍 What does this change?

• Selects only finalized canonical assistant text and schema-validated brunch_ask.input.question values.
  • Reasoning, partial text, tool output, malformed asks, other tools, and non-assistant roles are excluded.
  • Questions come from structured AI SDK parts, never DOM scraping.
  • The real Brunch ask integration test proves an awaiting question is emitted without duplicate visible assistant text.
• Assigns every speakable segment stable message/part identity and an exact-text fingerprint.
  • The browser sends { segmentId, text }.
  • The server recomputes the fingerprint and rejects any mismatched or malformed request.
• Adds the app-owned /api/voice/speech edge.
  • It validates same-origin, content type, body size, text length, and segment identity.
  • It sends the exact selected text to OpenAI gpt-4o-mini-tts with server-fixed marin voice and MP3 output.
  • It accepts only audio/mpeg, disables caching, sanitizes upstream errors, and propagates timeouts/client cancellation.
  • It sends no content-changing instruction, “say exactly” prompt, or Realtime response.create event.
• Adds cancelable, stale-safe browser playback.
  • New generations cancel old fetch/audio work.
  • Audio elements, response streams, blobs, and object URLs are cleaned up on success, failure, cancellation, and disposal.
• Extends the half-duplex controller so the microphone stays closed throughout synthesis and playback and reopens only after speech drains and Brunch is ready.
• Keeps canonical text visible if speech fails; there is no generated or alternate speech fallback.
• Adds accessible synthesis/playback status and the required disclosure: spoken responses use an AI-generated OpenAI voice.
• Updates ADR-0009, the website README, and the Petrinaut AI-assistant guide. No provider abstraction, package dependency, lockfile change, or ElevenLabs path is added.

Stack record: direct base 7c042206e919b80ef27ce46649bfe5202005b27c; PR head b98ec4772a9702e1cba096e5604e7dc77f46706d.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

• does not modify any publishable blocks or libraries, or modifications do not need publishing

📜 Does this require a change to the docs?

The changes in this PR:

• require changes to docs which are made as part of this PR

No existing documentation screenshot covers this voice surface.

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

• do not affect the execution graph

⚠️ Known issues

• Public production rollout is intentionally blocked on FE-1439, FE-1420, platform authentication, distributed quotas, FE-1505 telemetry, and FE-1438/FE-1440 completion/projection contracts.
• The pruned @apps/brunch-agent CI jobs cannot see the non-workspace condition-5 runner on the FE-1506: Show captures and completion while a panel elicitation runs #9346 base. Parallel PR #9348 already contains the exact tested prune fix (ade09e9c36), so this sibling stack does not duplicate it. Full-checkout Brunch tests, lint, typecheck, and build pass locally.
• The inherited Brunch docs-index check remains 6/7 because FE-1506: Show captures and completion while a panel elicitation runs #9346 references ADR-0008 from the parallel prerequisite stack.
• Transport emits 4 inherited no-await-in-loop warnings and Brunch emits 12 inherited warnings.

🐾 Next steps

Voice PR4 will add delivery caching, authoritative reload reconciliation, stale-ask recovery, authentication, distributed quotas, telemetry, retention cleanup, a kill switch, and real completion/projection evidence. It must not start until the listed platform contracts are available.

🛡 What tests cover this?

• Canonical-selection tests prove exact finalized assistant/ask selection and exclusion of reasoning, tool output, partial, malformed, and duplicate content.
• Speech-handler tests prove request bounds, exact-text forwarding, fixed server policy, media-type enforcement, sanitized errors, timeout, disconnect, and stream cancellation.
• Playback-controller tests prove cancellation, stale-generation rejection, and resource cleanup.
• Turn-controller/component tests prove queued speech ordering, microphone suppression, failure fallback, disclosure, and accessible status.
• The Brunch ask integration test proves the structured question reaches Petrinaut without duplicate assistant text.

Validation run

yarn workspace @hashintel/petrinaut test:unit --run
yarn workspace @hashintel/petrinaut lint:eslint
yarn workspace @hashintel/petrinaut lint:tsc
yarn workspace @hashintel/petrinaut build
yarn workspace @apps/petrinaut-website test:unit
yarn workspace @apps/petrinaut-website lint:eslint
yarn workspace @apps/petrinaut-website lint:tsc
yarn workspace @apps/petrinaut-website build
yarn workspace @hashintel/brunch-agent-transport-aisdk test:unit
yarn workspace @hashintel/brunch-agent-transport-aisdk lint:eslint
yarn workspace @hashintel/brunch-agent-transport-aisdk lint:tsc
yarn workspace @hashintel/brunch-agent-transport-aisdk build
yarn workspace @apps/brunch-agent test:unit
yarn workspace @apps/brunch-agent lint:eslint
yarn workspace @apps/brunch-agent lint:tsc
yarn workspace @apps/brunch-agent build
yarn workspace @local/petrinaut-arch-docs lint:arch-docs
yarn oxfmt --check apps/petrinaut-website apps/brunch-agent libs/@hashintel/petrinaut libs/@hashintel/brunch-agent

Results: 214 Petrinaut tests, 72 website tests, 12 transport tests, and 35 Brunch tests passed locally. Lint, typecheck, build, formatting, and architecture-doc checks passed.

The direct-base no-ElevenLabs audit and git diff --check passed. No package manifest or lockfile changed; provider-owned paths are explicitly OpenAI-named.

Credential-dependent checks not run: real OpenAI synthesis, remote Brunch, microphone/browser permission, exact spoken output in desktop Chrome/Edge/Safari, and deployed cancellation/failure behavior.

❓ How to test this?

1. Run yarn workspace @apps/petrinaut-website test:unit plus its lint, typecheck, and build commands above.
2. In a protected preview with OpenAI credentials and a real Brunch endpoint, complete one text turn and one voice turn.
3. Confirm finalized assistant text and a pending structured question are each spoken once and match the visible text exactly.
4. Cancel/end voice and induce a speech failure; confirm playback stops, the microphone remains safely closed until Brunch is ready, and visible text remains usable.

📹 Demo

Not included: credentialed browser/provider evidence remains an explicit acceptance requirement.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
petrinaut	Ready	Preview	Aug 27, 2026 2:18pm

3 Skipped Deployments

Project	Deployment	Actions	Updated (UTC)
hash	Ignored	Preview	Aug 27, 2026 2:18pm
hashdotdesign-tokens	Ignored	Preview	Aug 27, 2026 2:18pm
petrinaut-docs	Skipped		Aug 27, 2026 2:18pm

[kostandinang]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• H-6763: Add the voice interview experience #9380
• H-6763: Improve voice preview reliability diagnostics #9378
• H-6763: Speak finalized Brunch responses with OpenAI #9357 👈 (View in Graphite)
• H-6763: Add OpenAI Realtime voice input to Brunch #9356
• H-6763: Add a generic Petrinaut composer submission API #9355
• FE-1507: Move the repertoire into core behind a guarded prompts subpath #9349 : 1 other dependent PR (#9381 )
• FE-1431: Build the plugin authoring surface (cycle two) #9348
• FE-1505: Time each interviewer turn by purpose #9347
• FE-1506: Show captures and completion while a panel elicitation runs #9346
• FE-1504: Run the process-model elicitor from the Petrinaut panel on one owned document #9345
• FE-1431: Build the plugin authoring surface #9337
• FE-1406: Audit the harness-teaching lineage and propose the fixed-key plugin contract (ADR-0007) #9327
• FE-1497: Build the controller read path and the SDCPN plugin package #9325
• FE-1404: Run condition 3 as the walking skeleton #9322
• FE-1403: Assemble and test the CPS interview guidance #9321
• FE-1402: Define and rehearse the elicitation completion contract #9320
• FE-1407: Catalogue elicitor failures that published measures miss #9319
• FE-1480: Project and realize a CPS model as a Petrinaut SDCPN #9313
• FE-1495: Prevent Brunch TypeScript contracts from drifting #9312
• FE-1494: Keep Brunch strategy and control records coherent #9311
• FE-1437: Move brunch-agent into hashintel/hash with its history #9274
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[cursor]

PR Summary

Medium Risk
Introduces a new server route that forwards user-visible text to OpenAI with API-key handling and complex async voice state; impact is limited to the gated voice preview, with validation and sanitized errors, but failures or cancellation races could still affect UX.

Overview
Adds text-to-speech for the Brunch voice preview: finalized assistant text and pending validated brunch_ask questions are selected as canonical segments (stable IDs + FNV fingerprint), spoken unchanged through a new /api/voice/speech proxy to OpenAI gpt-4o-mini-tts, while Realtime stays transcription-only.

The voice turn flow now atomically ingests chat status and canonical segments via updateChat, queues speech in order (history is marked seen, not replayed), runs synthesizing/playing phases with the mic closed, and cancels stale fetch/audio on end or failure—leaving visible text and a recoverable error if synthesis fails. Client playback verifies segment fingerprints before POSTing; the server enforces same-origin, bounds, and matching segmentId/text.

UI adds accessible status for synthesis/playback and disclosure that speech is AI-generated. Brunch integration tests assert an awaiting ask emits no duplicate assistant text (initialText empty). Docs (README, ADR-0009, AI assistant guide) and dev/Vercel routing for the speech route are updated.

^{Reviewed by Cursor Bugbot for commit 9ef8f75. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 7990e51. Configure here.}