One-click local LLM setup for AI translations (managed Ollama sidecar)

[robflowk]

Goal

One-click setup for fully local AI translation: the app downloads, configures, and runs a local LLM if the machine supports it — no cloud account, no API key, nothing leaves the machine. Relevant for teams with sensitive strings and for users who just don't want to manage keys.

Builds on the provider-agnostic LLM layer from #8: the suggestion pipeline only talks to an OpenAI-compatible endpoint at a configurable base URL, so a managed local runtime plugs in as a preconfigured provider pointed at localhost — zero changes to the pipeline, glossary, or draft flow.

Proposed approach

Runtime: Ollama as a managed sidecar

• Detect an existing Ollama install first and reuse it (many devs already run one) — manage our own only if none is found
• Otherwise download the Ollama binary and run it as a Tauri sidecar; the app owns its lifecycle (start on demand, stop on quit)
• Ollama handles the hard parts: model downloads, quantization variants, GPU offload, and it already speaks the OpenAI-compatible API

Hardware check gates the offer

• Probe RAM, VRAM / Apple Silicon unified memory before offering the feature at all
• Gate not just "can it run" but translation quality: small models are weak at multilingual output — an 8B-class model is roughly the floor for decent quality, which implies ~8–16 GB RAM as a practical minimum
• Curate a small model list per hardware tier rather than exposing the full Ollama catalog

UX

• Surfaces as a "Local (managed)" option in the provider settings from Concept: AI translations with a learned per-project glossary and leading language #8 — selecting it triggers the guided setup (hardware check → model download with progress → ready)
• Once set up, it's an ordinary OpenAI-compatible provider entry; the per-user local override from Concept: AI translations with a learned per-project glossary and leading language #8 means one teammate can run local while others use cloud providers

Open questions

• Detect-and-reuse vs. always-managed: how to handle version skew with a user's existing Ollama?
• Model curation: which models per hardware tier, and who updates the list (baked into releases vs. remote manifest)?
• Disk footprint transparency: models are multi-GB — show size before download, offer cleanup/uninstall
• Windows/Linux GPU detection (CUDA/ROCm) vs. macOS unified memory — how much per-platform probing is worth it in v1?
• Fallback UX when hardware is below the floor: hide the option entirely, or show it disabled with an explanation?

Dependencies

• Blocked by Concept: AI translations with a learned per-project glossary and leading language #8 (provider-agnostic LLM layer, specifically the OpenAI-compatible tier and provider settings UI)