Screen Solver

[jpka]

Destination

An implementation-ready spec for a Windows desktop app that captures a chosen browser window on an interval, calls a vision LLM only when the screen has meaningfully changed, and streams the solution to a coding exercise into its own read-only window.

Done means: nothing left to decide before someone starts coding.

Notes

Domain. Greenfield Windows 11 desktop app. Node 24.13 / npm 11.6 available; no Python, .NET, or Rust toolchain installed.

Tracker. GitHub Issues (this map + its child issues). See docs/agents/issue-tracker.md. Original planning history lives at .scratch/solver/.

Posture. Plan, don't do. Every ticket resolves a decision; the map is done when the way is clear. Do not build the app from inside this map.

Skills to consult. /grilling and /domain-modeling by default; /research for research tickets; /prototype for prototype tickets; /codebase-design when designing the provider seam.

Settled constraints. Pinned down while naming the destination — these fix the scope and are not up for re-litigation inside a ticket:

Constraint	Answer
Destination	Implementation-ready spec (plan, don't build)
First target	Coding exercises / katas (LeetCode, Exercism, Advent of Code, HackerRank)
Trigger	Configured interval + change detection — the LLM fires only on a meaningful diff, never on an unchanged screen
Capture target	A named window the user picks from a list, captured by handle so it follows moves and resizes
Scope boundary	Strictly read-only — the app displays answers and never acts on the page
LLM provider	Anthropic API only for v1 (decided, Vision provider comparison), behind a seam designed to admit a second provider later

Decisions so far

• Window capture on Windows 11 — Target Windows.Graphics.Capture; Electron's desktopCapturer is already WGC-backed and sidesteps PrintWindow's black-frame problem on GPU-composited Chrome/Edge. Two open risks flagged for the stack ticket: unconfirmed border-suppression on an unpackaged app, and unguaranteed stale-vs-black behavior on minimized/other-desktop windows. (The border risk is since closed by Packaging and distribution.)
• Vision provider comparison — Decided directly by the user, skipping the planned cross-provider research: Anthropic API only for v1. The seam (Provider seam design) is still designed to admit a second provider later.
• App shell and stack — Electron + TypeScript. Capture loop, change detection, and the streaming Anthropic call all live in the main process; the output window is a thin renderer driven by IPC events, never touching the network or capture API directly.
• Provider seam design — A deep module: createProvider(config) → Provider with solve(image, {signal}) → AsyncIterable. System prompt is configured once at construction, not per-call. Streaming normalizes to delta/done{usage}/error{kind}; transient errors (rate limit, overloaded, network) retry internally, auth/refusal surface immediately. Budget enforcement, change detection, and image capture stay outside the seam.
• What counts as "changed" — dHash on the instructions-pane region as the cheap per-tick signal, gated by a debounce (sample twice, ~500ms-1s apart, escalate only if the change persists against the last-solved baseline) rather than a tuned magnitude threshold — a real prototype against live kata captures found that an incidental hover flyout produced more apparent change than the real navigation event, but the flyout's effect fully cancelled once checked past it while real changes didn't. Scroll-vs-navigate ambiguity is left as an accepted risk bounded by Cost control's cooldown, not solved here. Editor-pane changes are excluded from the trigger gate.
• The output window — "Focus pane": one dominant streaming answer pane, history/settings behind titlebar icons, new problem interrupts-and-replaces (previous pushed to history, tagged interrupted). Three variants prototyped; code wraps rather than horizontal-scrolls, window is always-on-top by default and remembers size/position across restarts.
• Configuration and API key storage — API key stored via Electron's safeStorage (DPAPI-backed, no extra native dep), entered once through an in-app masked input, decrypted into a main-process-only variable and never sent over IPC or logged. Settings (interval, target window, provider/model, change-detection params) live in one config.json in userData, alongside the encrypted key blob; target window is identified by process name + title (not the unstable OS handle), re-resolved on startup with a fallback to reopening the picker. All settings apply live, no restart needed.
• Cost control — Four nested limits — near-duplicate suppression, a sticky runaway breaker (8 solves/5min), a 20s cooldown, a 40/hour ceiling, and a $5/day spend cap — sized against a computed cost model: a solve costs $0.02–$0.09 ($0.21 worst case), and an unguarded misfiring detector on a 5s interval costs $18–$153/hour. Defaults ship conservative (claude-sonnet-5, effort: medium, max_tokens: 8000, capture downscaled to 1568px, no crop) because effort moves cost more than image size does — thinking bills at the output rate. Limits escalate silent → auto-recovering → sticky; hitting one stops and warns, never silently degrades the model. Keeping the last 3 solved baselines for near-duplicate suppression also partially closes What counts as "changed"'s scroll-vs-navigate risk. Prompt caching was left conditional on Solve-call prompt design, which has since measured the prompt over the minimum cacheable prefix and turned it on at ttl: 1h.
• Packaging and distribution — electron-builder NSIS one-click per-user installer (no admin, no UAC, unsigned — a locally built installer never carries mark-of-the-web, so SmartScreen never fires); config committed. Auto-start is an opt-in setting, default off, via setLoginItemSettings, and an auto-started launch begins paused — login must never start spending. Updates are a manual local rebuild; electron-updater is deferred but not precluded, since NSIS is the only target it supports. The border-suppression spike is skipped, on a finding that closes it for good: graphicsCaptureWithoutBorder turns out to be a general-use capability, not a restricted one (correcting Window capture on Windows 11) — but that's moot, because WebRTC's WGC session never sets IsBorderRequired and Electron exposes no way to, so no packaging format can suppress the border from the chosen capture path. The yellow border ships as permanent, expected behavior — and doubles as a free OS-drawn "watching" indicator that vanishes on pause, provided the app holds one long-lived capture session rather than one per tick, or it strobes. Also mandates a single-instance lock, which is a budget control: two instances is two BudgetGuards and two bills.
• Solve-call prompt design — Answer-first (# title → fenced code → two or three short paragraphs), matching The output window's code-streams-first contract; the heading doubles as the history drawer's per-entry label. Prompt committed, run against the real kata captures. The biggest correctness lever wasn't on the ticket's list: the answer must reproduce the editor pane's exact signature, and the sample tests pin the return shape more precisely than the prose does — which is why Cost control was right to refuse cropping. Partial screens are judged by determinacy, not by whether anything is cut off (a scrolled frame with title, examples, signature and tests is solved silently; only a missing load-bearing fact earns a > **Missing:** line, placed before the code because the code streams first). Prompt caching turns on — the prompt measures 1196–1496 t against claude-sonnet-5's 1024 minimum, and above that threshold a longer prompt is a cheaper one. The measured visible answer is 255–305 t, under 4% of max_tokens: 8000, so that ceiling is in practice a thinking budget and the right answer to Cost control #9's truncation question is to make truncation detectable (stop_reason) rather than assert it won't happen. A literal # No exercise on screen bail token closes the map's "detection as a separate pass" fog: no separate detector ships in v1 — the in-prompt bail is the v1 detector, and counting how often it fires produces exactly the suppression rate Cost control #9's 4% break-even arithmetic was missing, turning a quality guess into a measurement. No API key was reachable, so every figure carries its measured-vs-estimated label and its direction of bias.

Not yet specified

• Answer history persistence — whether past solutions survive a restart, and where they live. Cost control requires a 30-day spend/usage history in userData regardless, and Packaging and distribution froze the %APPDATA%\Screen Solver\ path and made it survive uninstall — so the storage question is now largely forced, and only the retention and shape of the history are really open. Solve-call prompt design settled where an entry's title comes from (the answer's first line) and recommended that "no exercise on screen" responses get no entry at all.

Out of scope

• Auto-typing answers into the browser — read-only was chosen deliberately; input injection is a different category of tool.
• Clipboard copy of the answer — strictly-read-only was chosen over the read-only-plus-clipboard variant.
• Quizzes, puzzles, crosswords, general Q&A — katas are the first and only target; generalizing the detection problem is a separate effort.
• macOS and Linux — Windows only.
• A local on-device model as the v1 provider — the seam should not preclude one later, but v1 does not ship it.
• Comparing or supporting non-Anthropic vision providers for v1 — decided directly (Vision provider comparison): Anthropic only, no comparison research. The seam stays generic enough to add one later, but nothing beyond Anthropic ships now.
• Silently degrading to a cheaper model when the budget is hit — considered and rejected in Cost control. The app's whole value is a correct answer; changing answer quality without the user asking, and invisibly, is worse than pausing. Switching models stays a deliberate one-click setting.
• Distribution to anyone but the author — Microsoft Store submission, a purchased code-signing certificate, and a hosted auto-update feed were all weighed and ruled out in Packaging and distribution. Each exists to move a binary to a machine other than the one that built it, which is past this map's destination.

Tickets

• Window capture on Windows 11 #2 — Window capture on Windows 11
• App shell and stack #3 — App shell and stack
• Vision provider comparison #4 — Vision provider comparison
• Provider seam design #5 — Provider seam design
• What counts as "changed" #6 — What counts as "changed"
• The output window #7 — The output window
• Configuration and API key storage #8 — Configuration and API key storage
• Cost control #9 — Cost control
• Packaging and distribution #10 — Packaging and distribution
• Solve-call prompt design #11 — Solve-call prompt design
• Failure and edge-case behavior #13 — Failure and edge-case behavior

---

Generated by Claude Code