Local models & API providers via in-app agentic harness - #93
Merged
Conversation
Contracts-first scaffold for raw LLM API providers (local models + hosted OpenAI-compatible endpoints) driven by an in-app agentic harness: - New Packages/EaselAgentHarness: AgentModelClient/AgentModelEvent, AgentMessage/AgentTranscript, AgentTool/ToolResult/ToolExecutionContext, JSONValue, ModelCapabilities, EndpointProfile (+presets), PathConfinementPolicy, AgentLoop signature, AgentHarnessError - Stub targets: AgentTools, AgentProviderOpenAI (SwiftOpenAI 4.4.9), AgentProviderOllama - ClaudeCodeCore: ChatProvider.api case with defensive decode, APIChatRuntime stub, CredentialStore + AgentTranscriptStore protocols, ChatViewModel runtime wiring, GlobalSettingsView provider switch
… matrix - ToolCallAccumulator: index-keyed fragment reassembly, synthesized ids, empty-args defaulting; invalid JSON deferred to model-visible errors - AgentLoop actor: trim -> stream (auto non-streaming fallback when the backend can't stream tool calls) -> accumulate -> execute (read-only concurrent, mutating serial) -> append -> repeat; no-data watchdog, cumulative usage, maxTurns cap - Invariant: every emitted transcriptUpdated is API-valid on its own, so the host can persist blindly and cancel consumption at any moment - TranscriptTrimmer: elide old tool-result bodies first, then drop whole turn groups; system + last-user protected; throws when irreducible
api_transcripts table keyed by session id; SimplifiedClaudeCodeSQLiteStorage conforms to AgentTranscriptStore with upsert save, corrupt-row-tolerant load, and deletion/re-key hooks in the session lifecycle
Bash (structured-concurrency ProcessRunner: SIGTERM->SIGKILL watchdog, concurrent pipe drain, middle truncation), Read/Write/Edit with shared FileReadRegistry enforcing read-before-modify and stale-read detection, Glob/Grep/LS. Path confinement + codebase-references write denial enforced via PathConfinementPolicy; expected failures are instructive model-visible error results
…model catalog - GlobalPreferencesStorage/GeneralPreferences: apiEndpointProfiles, selectedAPIProfileId, apiModel, apiMaxTurns with preset seeding and selection repair (Ollama-first) - KeychainCredentialStore (generic password, service com.easel.api-provider) - APIModelCatalog dispatching per profile kind with short cache - GlobalSettingsView: profile picker + editor sheet (base URL, SecureField API key, capability toggles, context window, Test Connection), model picker with refresh + text fallback, max-turns stepper
- apiAgentInstructionsPrefix: compact harness-toolset instructions sized for small local-model context windows (+ WithFrontendSkill variant) - ChatViewModel(apiInstructionsPrefix:) + combinedAPIInstructions(), passed from ChatService - APIImageContentConverter: 'Analyze this image:' marker lines -> downscaled (<=1568px) base64 JPEG data-URL content blocks for vision models
- OpenAIRequestMapper: AgentMessage/AgentToolSchema -> ChatCompletionParameters (image data-URL content arrays, tool-call echo, error-prefixed tool results, JSONValue->JSONSchema round-trip) - OpenAIStreamMapper: chunk -> AgentModelEvent incl. reasoning_content, indexed tool-call fragments, stream usage; buffered-completion synthesis for backends with broken streamed tool calls - Base-URL parsing into factory components (root/proxyPath/version) and APIError -> AgentHarnessError mapping
Wire types per Ollama's documented chat API (tool_call arguments as JSON objects, tool_name result pairing, done_reason/eval counts, error payloads). Complete tool calls are serialized with synthesized ids; thinking maps to reasoning deltas; num_ctx follows the profile context window. Deliberately avoids the /v1 OpenAI shim and its streaming+tools bugs. URLProtocol-stubbed test suite covers streaming, tool calls, malformed lines, and encoding
- send(): mints .api-tagged sessions, loads/persists AgentTranscript per session (system prompt refreshed each send), converts image markers to vision blocks per profile capability, runs AgentLoop with the built-in toolset (per-session read registry) and Claude-style tool names - APIMessageMapper: loop events -> streaming assistant/thinking bubbles, tool-use cards with display+raw parameters, tool results keyed by call id - Per-turn SessionUsageRecord deltas; activeGeneration guard copied from CodexChatRuntime; ChatViewModel configure/factory wiring - UnavailableModelClient placeholder keeps ClaudeCodeCore free of the MLX dependency (factory injected by the app at integration)
- MLXModelClient (AgentModelClient) on mlx-swift-lm 3.31: one completion per call, streamed .chunk/.toolCall/.info generations mapped to harness events - MLXModelRuntime actor owns the loaded ModelContainer (single resident model, GPU cache limit, local-directory loading via huggingFaceTokenizerLoader) - MLXModelManager actor: HubApi snapshot downloads with progress into ~/Library/Application Support/Easel/Models, install detection, disk usage, delete; MLXModelManagerView for settings embedding - Availability gating (Apple Silicon + RAM tiers) and curated Qwen-coder starter models
- ChatViewModel(apiModelClientFactory:) lets the app inject backends ClaudeCodeCore doesn't link - ChatService owns one MLXModelManager + MLXModelRuntime and routes .mlxLocal profiles to MLXModelClient (HTTP kinds unchanged) - EaselChatSettingsView embeds the MLX download manager below provider settings - Note: building with Xcode now needs -skipPackagePluginValidation (mlx-swift ships a build plugin)
Some models emit tool calls as plain text (bare or fenced JSON like
{"name": "LS", "arguments": {...}}) that the backend's template parser
fails to convert to structured tool_calls — observed with qwen2.5-coder:7b
on Ollama 0.11.x. When a turn yields no structured calls, the loop now
recognizes whole-payload JSON (entire message or fenced block body, name
matching a known tool) and executes it, stripping the consumed payload from
the transcript echo. Prose merely mentioning JSON never converts.
- PseudoToolCallParser now extracts tool-call JSON embedded in prose (not
just whole-message or fenced), using a quote-aware balanced-brace scanner.
Inline objects inside a sentence must carry an arguments/parameters key to
convert (so a stray mention isn't executed); objects alone on a line or
fenced convert even with no args.
- APIMessageMapper reconciles the streamed bubble with the loop's cleaned
assistant text on completion: the consumed JSON payload is replaced with
the residual prose, or the bubble is removed when the whole message was a
tool call. No more raw {"name": ...} blobs in the chat.
The old prompt told the model resources/design-system/DESIGN.md was 'the design source of truth' to read 'before writing any UI', which steered weak local models (llama3.1) to write page content INTO DESIGN.md and never touch index.html — so nothing rendered. Rewrite to lead with the load-bearing fact: index.html is the entry point that renders in the preview; edit it (and the CSS/JS it links) to change what the user sees. Mark design-system/ explicitly read-only, demand real markup over placeholder text, and tell the model not to narrate JSON. Verified against llama3.1: now targets index.html with real HTML across runs.
It was rendering below the version footer (and off-screen) because it was appended outside GlobalSettingsView's Form. Thread an injectable apiExtraContent closure through ClaudeCodeGlobalSettingsSceneView into the Form so the MLX manager renders as a grouped Section under the .api provider config, above the version row. Injection keeps ClaudeCodeCore free of the MLX dependency; the section only shows for the Local / API provider.
The catalog short-circuited .mlxLocal profiles to an empty list, so a downloaded on-device model never appeared in the dropdown. Now MLX profiles dispatch to MLXModelClient.listModels() (installed models), bypassing the cache so a fresh download shows immediately. ChatService exposes an MLX-aware APIModelCatalog, threaded through the settings scene view. A NotificationCenter signal (agentInstalledModelsDidChange) posted on download/delete refreshes the picker live. Picker shows the friendly display name while storing the hub repo id.
The composer footer had model badges for Codex and Claude but nothing for the API provider. Add APIModelBadge showing the selected endpoint name and model (MLX hub ids shortened to their leaf), rendered whenever the API provider is active — so the active model is always visible next to the session token count.
Loads a downloaded MLX model and runs a tools-enabled prompt, asserting a structured tool call is emitted. Skipped by default; run with EASEL_MLX_INTEGRATION=1. (MLX's Metal library only resolves inside an app bundle, so this runs in-app rather than via headless swift test.)
Weak local models (MLX Qwen, some Ollama models) answer 'build a page' by pasting the HTML into the chat as a code block rather than calling Write, so nothing reaches disk and the canvas stays blank. Now, when a turn makes NO tool calls but its final message contains a complete page: - CodeFileExtractor (harness, pure/tested) recovers the file(s): a complete HTML document -> index.html; fenced blocks with a named file -> that file - APIChatRuntime writes them through the path policy (workspace-confined, codebase-references denied) and renders a Write tool card per file, so it's visible, not silent; the 600ms watcher reloads the preview Guarded to fire only when a complete page is present (not explanatory snippets) and only when the turn used no tools (no double-writing).
Weak models label pasted files with a header line above the fence
("script.js" then a plain ```javascript block) rather than putting the
filename in the fence info string. The extractor only read the info string,
so it wrote index.html but dropped styles.css/script.js — the linked assets
404'd and the page rendered unstyled. Now the extractor also reads a short
label line (<=3 tokens: "script.js", "**styles.css**", "styles.css:",
"File: styles.css") directly above a fence, so all authored files are
written together. Full sentences that merely mention a filename are still
ignored.
Models use a 'Step N: Create `script.js`' header, then a description sentence, then the fence — so the filename is not on the line directly above the code block. The extractor now scans all lines since the previous fence (nearest first) and recognizes a backtick-wrapped filename in a header, the strongest prose-safe signal. This recovers styles.css/script.js that were previously dropped, so the linked assets exist and the page renders styled with working JS.
The selected model was a single global value (apiModel), independent of the endpoint. Switching endpoints never updated it, so the model and endpoint drifted out of sync (e.g. Groq selected but an MLX model stored) and the selector showed the wrong model after relaunch. Now the chosen model is stored on each EndpointProfile.defaultModel: the settings picker, the model badge, and the runtime all read the selected profile's model, and switching endpoints restores that endpoint's own model. The legacy apiModel field is kept as a mirror/fallback for pre-migration state.
…config The endpoint UI was a flat provider list with edit/add/delete buttons that made no sense for fixed presets (you can't edit Groq's URL — you need a key). Redesigned around what each provider actually needs: - Provider picker grouped into 'On your Mac' (Ollama, LM Studio, llama.cpp, MLX) and 'Hosted API' (OpenRouter, Groq, DeepSeek, xAI), plus Custom - Contextual inline config: an API-key SecureField for hosted providers, an editable Server URL for local servers, a note for on-device MLX — no more opening a sheet just to enter a key - Inline Test Connection with status - Edit/Remove only for user-added custom endpoints; presets can't be renamed or deleted. 'Add Custom Endpoint' for advanced OpenAI-compatible URLs - Dedicated Model and Advanced sections; MLX download manager shows only when MLX is selected - EndpointProfile.category/isPreset drive the grouping API keys and server URLs persist as you type (Keychain / profile).
EaselAgentMLX pulls in mlx-swift-lm, whose MLXHuggingFace target ships swift-syntax macros. Non-interactive CI can't approve them, so the build failed at dependency-graph/plugin resolution (surfacing as SwiftLint plugin failures). Pair the existing -skipPackagePluginValidation with -skipMacroValidation in the build/test and release xcodebuild invocations, matching the flags that build cleanly locally.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a third chat-provider class, Local / API, alongside the Codex and Claude CLI providers: raw LLM endpoints driven by a new in-app agentic harness. Unlike the CLIs, raw APIs bring no tools — so the harness supplies them: a streaming tool-call loop with workspace-confined Bash/Read/Write/Edit/Glob/Grep/LS tools, rendered through the existing chat UI (Claude-style tool names reuse the existing tool cards), with file edits hot-reloading the preview.
Backends
/api/chat(not its/v1shim, which has known streaming+tool-call bugs)Architecture
Packages/EaselAgentHarness(new, Swift 6, UI-free, OSS-extractable): frozen contracts (AgentModelClient,AgentTool,AgentTranscript,EndpointProfile), theAgentLoopactor,ToolCallAccumulator,TranscriptTrimmer, the 7 built-in tools withPathConfinementPolicy, and the OpenAI/Ollama adapters. Invariant: every emitted transcript is API-valid, so cancellation never persists dangling tool calls.Packages/EaselAgentMLX(new):MLXModelClient, single-residentMLXModelRuntime,MLXModelManager,MLXModelManagerView.ChatProvider.apicase (defensive decode),APIChatRuntime+APIMessageMappermirroringCodexChatRuntime'sactiveGenerationdiscipline, KeychainCredentialStore, endpoint-profile settings UI, SQLiteapi_transcriptstable (migration V7) for resumable model context, image→vision-block conversion, per-turn usage records.ChatServiceinjects the MLX-aware client factory so ClaudeCodeCore never links MLX.Weak-model recovery layer
Small local models don't always call tools cleanly; the harness degrades gracefully:
tool_calls, and strips the raw JSON from the chat bubble.index.html+ labeledstyles.css/script.js), the files are written to disk (workspace-confined) with a visible Write card, so the preview updates. Guarded to fire only on a complete page, never on explanatory snippets or turns that already used tools.Settings & model selection
EndpointProfile.defaultModel) so each provider remembers its own model across relaunches and endpoint switches.Safety
Parity with the CLI providers: file tools hard-confined to the project directory in code (writes to
resources/codebase-references/denied); Bash cwd-scoped with timeout/output caps; Easel constraints prompt-enforced. Send Handoff is unaffected (CLI-only by design).Testing
APIChatRuntimeend-to-end with scripted clients + real tool execution, transcript SQLite, code-file extractor, per-profile persistence.CodexChatRuntime*,ClaudeChatRuntimeOptionsBuilder,CodexMessageMapper, model catalogs, storage, prefs) plusEaselChatTests/ChatServiceTests. Residual risk is confined to two additive UI surfaces (settings sections, composer badge) — recommend a quick manual smoke of Codex/Claude before merge.-skipPackagePluginValidation -skipMacroValidation(mlx-swift ships a build plugin + macros).🤖 Generated with Claude Code