Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -252,7 +252,9 @@ The voice stack is designed to be droppable into another app. Layering, strictly

- **`AgentHubVoice` package** (`app/modules/AgentHubVoice`) — app-agnostic voice infrastructure: `RealtimeVoiceEngine`, audio capture/playback, dictation/STT, `VoiceTool`/`VoiceToolRegistry`, realtime session config builder, OpenAI key handling, `VoiceScreenCaptureService` (+ `VoiceScreenCaptureTools` factory for `list_displays`/`capture_screen`), and the HUD contract (`VoiceHUDHost`, `VoiceHUDConfiguration`, `VoiceHUDPresenting`). Depends only on SwiftOpenAI. Never import AgentHub types here.
- **`AgentHubVoicePanel` target** (same package) — reusable SwiftUI HUD + onboarding (`VoiceOnboardingView`, `VoiceOption`) + `AppKitVoiceHUDPresenter`. Depends only on `AgentHubVoice`; branding and UserDefaults keys arrive via `VoiceHUDConfiguration` — never hardcode an app's name or preference keys in this target.
- **`AgentHubCore`** — the AgentHub-specific glue: `VoiceAgentToolExecutor`, `VoiceToolCatalog` (session tools; appends the capture tools from the factory), target resolver, completion watcher, transcript reader, `VoiceControlCoordinator`, settings UI, and `AgentHubVoiceHUDHost` (the `VoiceHUDHost` adapter) plus `VoiceHUDConfiguration.agentHub`. Core imports both voice products; the voice package must never import core.
- **`AgentHubCore`** — the AgentHub-specific glue: `VoiceAgentToolExecutor`, `VoiceToolCatalog` (session tools; appends the capture tools from the factory and the MCP tools from `VoiceMCPToolProvider`), target resolver, completion watcher, transcript reader, `VoiceControlCoordinator`, settings UI, and `AgentHubVoiceHUDHost` (the `VoiceHUDHost` adapter) plus `VoiceHUDConfiguration.agentHub`. Core imports both voice products; the voice package must never import core.
- **Voice MCP tools** — `VoiceMCPToolProvider` bridges the user's personal MCP servers (Claude `~/.claude.json` + Codex `~/.codex/config.toml`, merged by name, Claude wins) into voice conversations as namespaced `{server}__{tool}` function tools, proxied through a dedicated `MCPAppDiscoveryService` instance with a longer request timeout (`AgentHubProvider.voiceMCPDiscoveryService`). Servers are opt-in per name via `AgentHubDefaults.voiceMCPEnabledServers` (Voice settings checklist, default off) — only enabled servers are ever spawned/contacted; MCP configs may carry secrets in `env` and must never be logged. Tool handlers have their own deadline and truncate output so a slow server can't freeze the mic-muted conversation.
- **Voice assistant mode** — the HUD target chip's "Assistant · no session" option (`AgentHubDefaults.voiceAssistantMode`, mirrored by `VoiceHUDSettingsKeys.assistantMode`) runs conversations as a standalone assistant: no session snapshot, and the registry (`VoiceToolCatalog.makeTools(assistantMode:)`) hard-excludes every session-mutating tool (`send_prompt`, `launch_session`, `create_worktree_tasks`, `approve_pending_tool`) and screen capture, keeping read-only session tools + MCP tools. `RealtimeSessionConfigurationBuilder` keys the persona on `send_prompt` presence — registries without it get the answer-directly assistant persona. Toggling the chip mid-conversation stops the engine so a live session can't keep its old tool scope.

To reuse voice in another app: depend on the `AgentHubVoice` package, implement `VoiceHUDHost`, supply a `VoiceHUDConfiguration`, and present `AppKitVoiceHUDPresenter` — plus your own tool catalog for app-specific tools.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -218,6 +218,16 @@ public enum AgentHubDefaults {
/// visualizer. Type: Bool (default: false)
public static let voiceShowTranscript = "\(keyPrefix)voice.showTranscript"

/// MCP servers (by name, from the user's Claude/Codex configs) whose tools
/// are exposed to voice conversations. Empty means no MCP tools.
/// Type: [String] (default: empty)
public static let voiceMCPEnabledServers = "\(keyPrefix)voice.mcpEnabledServers"

/// Whether voice conversations run as a standalone assistant: no session
/// target, no session snapshot, and no tools that push prompts or content
/// into a session. Type: Bool (default: false)
public static let voiceAssistantMode = "\(keyPrefix)voice.assistantMode"

// MARK: - Feature Flags

/// Whether smart mode (AI-powered orchestration planning) is enabled
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -270,9 +270,19 @@ public final class AgentHubProvider {
targetResolver: voiceSessionTargetResolver
)

/// Dedicated MCP gateway for voice tool calls. Separate instance from the
/// MCP Apps discovery service so long-running tool calls get a longer
/// request timeout without affecting app-resource discovery.
public private(set) lazy var voiceMCPDiscoveryService: any MCPAppDiscoveryServiceProtocol =
MCPAppDiscoveryService(requestTimeoutSeconds: 45)

public private(set) lazy var voiceMCPToolProvider: any VoiceMCPToolProviding =
VoiceMCPToolProvider(discovery: voiceMCPDiscoveryService)

public private(set) lazy var voiceToolCatalog: any VoiceToolCataloging =
VoiceToolCatalog(
executor: voiceToolExecutor,
mcpToolProvider: voiceMCPToolProvider,
onBackgroundWaitCountChanged: { [weak self] count in
self?.realtimeVoiceEngine.setAwaitingBackgroundWork(count > 0)
}
Expand Down Expand Up @@ -532,10 +542,15 @@ public final class AgentHubProvider {
}

public func shutdownMCPAppDiscoveryService() {
let service = mcpAppDiscoveryService
let services: [any MCPAppDiscoveryServiceProtocol] = [
mcpAppDiscoveryService,
voiceMCPDiscoveryService,
]
let semaphore = DispatchSemaphore(value: 0)
Task.detached(priority: .utility) {
await service.shutdown()
for service in services {
await service.shutdown()
}
semaphore.signal()
}
_ = semaphore.wait(timeout: .now() + 2.0)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,8 @@ extension VoiceHUDConfiguration {
hudFrame: AgentHubDefaults.voiceHUDFrame,
screenCaptureEnabled: AgentHubDefaults.voiceScreenCaptureEnabled,
onboardingCompleted: AgentHubDefaults.voiceOnboardingCompleted,
showTranscript: AgentHubDefaults.voiceShowTranscript
showTranscript: AgentHubDefaults.voiceShowTranscript,
assistantMode: AgentHubDefaults.voiceAssistantMode
),
accentColor: .brandSecondary
)
Expand Down Expand Up @@ -62,14 +63,20 @@ public final class AgentHubVoiceHUDHost: VoiceHUDHost {
.map(Self.hudTarget)
}

private var isAssistantMode: Bool {
defaults.bool(forKey: AgentHubDefaults.voiceAssistantMode)
}

public func makeToolRegistry() -> VoiceToolRegistry {
VoiceToolRegistry(
tools: provider?.voiceToolCatalog.makeTools() ?? []
tools: provider?.voiceToolCatalog.makeTools(
assistantMode: isAssistantMode
) ?? []
)
}

public func makeSessionContext() -> String? {
guard let provider else { return nil }
guard let provider, !isAssistantMode else { return nil }
return VoiceSessionContextBuilder.make(
summary: provider.voiceToolExecutor.listSessions()
)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ public final class VoiceControlCoordinator {
private let presenter: any VoiceHUDPresenting
private let defaults: UserDefaults
private let hotKey: GlobalHotKey
private let onHUDShown: (@MainActor () -> Void)?
private var isStarted = false

public var isHUDVisible: Bool {
Expand All @@ -25,12 +26,14 @@ public final class VoiceControlCoordinator {
registrar: any GlobalHotKeyRegistrarProtocol,
presenter: any VoiceHUDPresenting,
defaults: UserDefaults = .standard,
hotKey: GlobalHotKey = .voiceHUDDefault
hotKey: GlobalHotKey = .voiceHUDDefault,
onHUDShown: (@MainActor () -> Void)? = nil
) {
self.registrar = registrar
self.presenter = presenter
self.defaults = defaults
self.hotKey = hotKey
self.onHUDShown = onHUDShown
}

public convenience init(
Expand All @@ -40,7 +43,12 @@ public final class VoiceControlCoordinator {
self.init(
registrar: CarbonGlobalHotKeyRegistrar(),
presenter: provider.makeVoiceHUDPresenter(defaults: defaults),
defaults: defaults
defaults: defaults,
onHUDShown: { [weak provider] in
// Warm the MCP tool cache so the conversation the user is about to
// start sees the enabled servers' tools.
provider?.voiceMCPToolProvider.scheduleRefresh()
}
)
}

Expand Down Expand Up @@ -87,6 +95,9 @@ public final class VoiceControlCoordinator {
}

public func toggleHUD() {
if !presenter.isVisible {
onHUDShown?()
}
presenter.toggle()
}
}
Loading
Loading