Skip to content

Voice: personal MCP tools + standalone assistant mode - #449

Open
jamesrochabrun wants to merge 1 commit into
mainfrom
voice-agent-access
Open

Voice: personal MCP tools + standalone assistant mode#449
jamesrochabrun wants to merge 1 commit into
mainfrom
voice-agent-access

Conversation

@jamesrochabrun

Copy link
Copy Markdown
Owner

Summary

Two features that turn AgentHub Voice into more than a session controller:

MCP tools from the user's personal configs (one source of truth)

  • New VoiceMCPToolProvider reads MCP servers from both Claude (~/.claude.json, global + per-project) and Codex (~/.codex/config.toml), merged by name (Claude wins), and exposes each enabled server's tools to voice conversations as namespaced {server}__{tool} function tools.
  • Calls proxy through a dedicated MCPAppDiscoveryService instance (AgentHubProvider.voiceMCPDiscoveryService, 45s request timeout) so the MCP Apps panel keeps its fast 6s instance. Both shut down on app quit.
  • Per-server opt-in checklist in Settings → Voice → MCP Tools (default off) — only enabled servers are ever spawned or contacted. Unsupported (authenticated remote) servers are shown but not connectable.
  • Tool handlers carry their own 60s deadline and truncate output to 6k chars, so a hung MCP server returns error JSON instead of freezing the mic-muted conversation. The tool cache warms when the HUD opens.
  • MCP configs can carry secrets in env blocks — they are never logged and never leave the machine; only tool schemas and results enter the conversation.

Assistant mode ("Assistant · no session")

  • New target-chip option runs conversations as a standalone assistant: no session snapshot, and a registry that hard-excludes every session-mutating tool (send_prompt, launch_session, create_worktree_tasks, approve_pending_tool) and screen capture — so "check my email" can only be answered directly, never forwarded into a session.
  • Read-only session tools (list/status/read/watch/worktrees) and MCP tools remain.
  • RealtimeSessionConfigurationBuilder keys the persona on send_prompt presence: assistant registries get an answer-directly persona; an MCP server roster line is injected so the assistant can name its capabilities when asked.
  • Flipping the chip mid-conversation stops the engine so a live call can't keep its old tool scope. Choice persists (voice.assistantMode).

Testing

  • AgentHubVoice package: 66 tests / 16 suites pass (swift test), incl. new assistant-persona, read-only-discipline, and MCP-roster instruction tests.
  • AgentHubCore targeted suites pass via AgentHubCore-Tests scheme: VoiceMCPToolProviderTests (8 new tests: dedup/namespacing, handler proxying, timeout, truncation, opt-in gating), VoiceToolCatalogTests (assistant-mode exclusion + MCP append), VoiceControlCoordinatorTests, MCPAppDiscoveryServiceTests.
  • ./scripts/test.sh packages (CI gate): all pass.

🤖 Generated with Claude Code

…stant mode

- VoiceMCPToolProvider bridges the user's personal MCP servers (~/.claude.json
  + ~/.codex/config.toml, merged by name, Claude wins) into voice conversations
  as namespaced {server}__{tool} function tools, proxied through a dedicated
  MCPAppDiscoveryService instance with a longer request timeout. Handlers have
  their own deadline and truncate output so a slow server can't freeze the
  mic-muted conversation.
- Per-server opt-in checklist in Voice settings (default off); only enabled
  servers are ever spawned. Tool cache warms when the HUD is shown.
- Assistant mode via the HUD target chip ("Assistant · no session"): no
  session snapshot and a registry that hard-excludes send_prompt,
  launch_session, create_worktree_tasks, approve_pending_tool, and screen
  capture — read-only session tools + MCP tools only. The instructions builder
  keys the persona on send_prompt presence and announces the MCP server roster
  so the assistant can say what it can do.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant