Skip to content

[codex] Improve voice input quality pipeline#43

Open
neta-zjj wants to merge 33 commits into
mainfrom
fix_voice_input_audio_quality
Open

[codex] Improve voice input quality pipeline#43
neta-zjj wants to merge 33 commits into
mainfrom
fix_voice_input_audio_quality

Conversation

@neta-zjj

@neta-zjj neta-zjj commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add session-level ASR observability for the websocket voice path: stable request/session ids, structured summary logs, OpenTelemetry session spans, stop reasons, provider/client timing, audio/result counts, and experiment metadata without logging raw audio or transcript text
  • add configurable Volc ASR endpoint experiments with stable hash bucketing, plus a local real-Volc experiment script that streams 16k mono PCM in configurable frames instead of uploading one blob
  • add optional OpenAI-compatible final-result rewrite for ASR cleanup; it is disabled by default and only runs when ASR_REWRITE_ENABLED=true and an API key is configured
  • extend SDK voice events with transcript metadata, alternatives, stop reasons, and client-side telemetry; keep legacy callback usage compatible
  • keep AudioWorklet-first capture, browser speech cleanup constraints, improved resampling, and client VAD endpoint stop
  • add web composer controls for local voice terms, continuous dictation, candidate replacement, undo, and press-and-hold hotkey release stop
  • feed local terms, current composer context, model name, prompt names, and mentions into ASR hotwords/context for the current session
  • fix SDK build reproducibility by adding the unrun peer needed by tsdown, and fix the API typecheck path mapping for @cohub/core/content/sanitize

Validation

  • pnpm --filter @cohub/gateway typecheck
  • pnpm --filter @cohub/gateway test
  • pnpm --filter @cohub/gateway lint
  • pnpm --filter @cohub/gateway build
  • pnpm --filter @neta-art/cohub typecheck
  • pnpm --filter @neta-art/cohub exec tsx --test tests/voice-input.test.ts
  • pnpm --filter @neta-art/cohub build (passes; existing zod d.ts bundling warnings remain)
  • PUBLIC_COHUB_ENV=dev PUBLIC_API_ORIGIN=http://localhost:8787 PUBLIC_GATEWAY_ORIGIN=http://localhost:8788 pnpm --filter web typecheck
  • PUBLIC_COHUB_ENV=dev PUBLIC_API_ORIGIN=http://localhost:8787 PUBLIC_GATEWAY_ORIGIN=http://localhost:8788 pnpm --filter web lint
  • PUBLIC_COHUB_ENV=dev PUBLIC_API_ORIGIN=http://localhost:8787 PUBLIC_GATEWAY_ORIGIN=http://localhost:8788 pnpm --filter web build (passes; existing large chunk warnings remain)
  • pnpm --filter @cohub/api typecheck
  • PUBLIC_COHUB_ENV=dev PUBLIC_API_ORIGIN=http://localhost:8787 PUBLIC_GATEWAY_ORIGIN=http://localhost:8788 pnpm -r typecheck
  • git diff --check

Notes

  • No online/dev Cohub environment was contacted for functional testing.
  • No real Volc ASR experiment was run because VOLC_ASR_API_KEY and VOLC_ASR_EXPERIMENT_AUDIO are not configured locally; the experiment script is implemented and exits unless both are provided.
  • LLM rewrite remains off by default and will not make external calls without explicit rewrite config and key.

@neta-zjj
neta-zjj force-pushed the fix_voice_input_audio_quality branch from 22124d2 to 3833627 Compare June 9, 2026 10:40
@neta-zjj neta-zjj changed the title [codex] Improve voice input audio capture [codex] Improve voice input quality pipeline Jun 9, 2026
@neta-zjj
neta-zjj force-pushed the fix_voice_input_audio_quality branch from afc4def to 0061798 Compare June 10, 2026 05:21
@neta-zjj
neta-zjj marked this pull request as ready for review June 10, 2026 05:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant