Optimize 100x (Phases 1–4): eval harness, symbol index, visual diff + inline editing, prompt caching - #27
Merged
Merged
Conversation
Decomposes "make it 100x better in every way" into 8 leverage-ordered phases (eval harness → symbol index → visual diff/inline edit → streaming+caching → server-side agent loop → runtime verify bridge → UX polish → reach), each an independent design→TDD→verify→commit cycle measured against the Phase-1 eval harness. Builds on the closed agentic loop from PR #26. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds an evals/ suite that replays recorded LLM responses through the REAL
grounding → patch pipeline against fixture projects (vanilla + React),
deterministic and key-free so it runs in CI. Scores Edit-Success-Rate and fails
on regression.
- evals/harness.ts (runScenario: copy fixture → ground/apply → assert)
- evals/fixtures/{vanilla,react}, evals/scenarios/*.json (5 seed scenarios:
exact edit, whitespace-tolerant edit, create, grounding-targeting, and a
safety negative that must reject a non-matching search without corrupting)
- evals/esr.eval.ts (vitest runner; prints "Edit-Success-Rate: N/M (X%)")
- vitest.evals.config.ts + `npm run eval`; wired into check:all and CI.
Baseline ESR 5/5 (100%) on the seed set. npm test stays unit-only (189).
This is the measurement backbone every later phase is graded against.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New src/symbol-index.ts: regex-based scan mapping exported symbols (components, functions, classes, types, consts, re-exports) → files, with case-insensitive lookup. Built fresh per grounding call (files change between prompts, so no stale-prone TTL cache). - project-grounding boosts files that EXPORT the selected componentHint / prompt terms — catches targets whose filename doesn't match the symbol (e.g. <PricingTable> in widgets.tsx), which pure path heuristics miss. - Exposed as a find_symbol tool for the H11 tool loop. - New eval scenario #6 proves the lift: it grounds src/widgets.tsx purely via the symbol index. ESR 6/6; 195 unit tests; full check:all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
renderLineDiff rebuilt on a real LCS line alignment (computeLineDiff): unchanged interior lines stay as correctly-numbered context, only true changes show as del/ins — replacing the old prefix/suffix heuristic with broken (row-index) line numbers. Adds highlightCode(): XSS-safe lightweight syntax highlighting (keywords/strings/comments/numbers) by tokenizing the raw line and escaping every piece. New unified-diff + token CSS. 6 new tests; security invariant re-asserted with a real XSS payload. 201 tests, build + browser E2E green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When a selected element has short, single-line text, the toolbar shows an inline input. buildTextEditModification finds that text verbatim in the grounded source and produces the code edit DETERMINISTICALLY — no LLM round-trip — disambiguating repeats with a 3-line context block, or returning null (→ focused LLM prompt fallback) when it can't be done safely. The edit flows through the existing diff-card → apply → verify pipeline. 6 new tests; full check:all green (207 unit, ESR 6/6). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Anthropic requests now send the large, stable system prompt as a cache_control ephemeral block (both the JSON-contract path and the tool-loop driver), so it's cached across every turn of a conversation / retry loop — ~90% cheaper and faster on cache hits. OpenAI prompt-caches automatically (no param); documented inline. Streaming note: the non-tool chat path already streams token-by-token; the tool loop is intentionally non-streaming because its final output is the structured propose_edits result (not prose to stream), which keeps per-provider tool_use parsing simple and safe. 1 new test (208 total). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The H6 typecheck gate runs before reload and can't see "compiled but crashed at runtime" edits. This closes that gap: on a successful apply, a durable marker (separate sessionStorage key, immune to message pruning) is armed before reload; after reload checkRuntimeAfterReload() waits for the app to render, then scrapes the framework error overlay + uncaught errors. On a runtime failure it reverts the chain (H8) and self-corrects with the runtime errors — bounded by the same N=2 budget. summarizeRuntimeFailure is unit-tested; bridge wiring covered by build + browser E2E. 3 new tests (211 total). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds src/toolbar/command-palette.ts with pure, tested filterCommands ranking (label prefix > substring > keyword > fuzzy subsequence). ⌘K now opens a real command palette overlay (was: focus prompt — now a "Focus the prompt" command within it) with type-to-filter, arrow-key nav, Enter/click to run, and click-outside/Esc to close. Commands route through the existing action handlers. 6 new tests (217 total); full check:all green. Remaining Phase 7 sub-features (multi-select, undo timeline, design-token pass) tracked for follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Design tokens: introduced a :host token block (--om-accent[-rgb], --om-text, --om-danger/success/warn, --om-radius) and tokenized the accent color across the stylesheet appearance-preservingly (rgba(108,92,231,a) → rgba(var( --om-accent-rgb),a)), so the palette is now themeable from one place. - Command palette gains "Undo the last applied change" / "Redo the last reverted change", driving the newest rollback/undo/redo control — keyboard-driven undo without a separate timeline UI. 217 tests, build + browser E2E green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Backed by parallel, web-verified research (current 2025/26 conventions). detectFramework now recognizes Angular, React Router v7 (Remix successor, gated on @react-router/dev so a plain react-router-dom SPA isn't misdetected), SolidStart/Solid, and Nuxt; routePathCandidates maps URL pathnames to source files for each: - Nuxt: app/pages (v4) + pages (v3), index/dir handling - SolidStart/Solid: src/routes/<path>.tsx + index - React Router: flat-file (dots=slashes, _index) + app/routes.ts/root.tsx - Angular: config-driven → surfaces app.routes.ts + the conventional component - SvelteKit: fixed the home-route double-slash Config-driven (Angular) / config-based (RR) routing that can't be inferred from the URL is covered by the Phase 2 symbol index. 8 grounding tests; 219 unit, ESR 6/6. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- R2: the empty chat state is now a 3-step onboarding guide (Select → Describe → Apply, with a ⌘K tip) instead of a one-line hint. - R4: docs/ADVANCED.md documents the closed loop (verify + self-correct + durable rollback), command palette, inline editing, native tool-calling (OPENMAGIC_TOOLS), CLI native-edit (OPENMAGIC_NATIVE_EDIT), thinking levels, supported frameworks, and the eval harness. 219 tests, build + browser E2E green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New src/llm/agent-loop.ts: the investigate→edit loop (read files / search / retry until the model proposes edits) reimplemented server-side as a bounded, dependency-injected state machine — testable in isolation (7 tests) and free of the browser round-trip per NEED_FILE. New agent.run op reuses the llm.chunk/done envelope, so the toolbar consumes it identically; it builds the model `step` from handleLlmChat (+ tool-calling) and injects sandboxed readFile/grep, cancellable via the H4 registry. Enabled with config.serverAgent / OPENMAGIC_SERVER_AGENT=1 (default off — llm.chat stays as the in-browser path, both harnesses coexist per the rollout plan). 226 tests, build + browser E2E green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Shift/Cmd/Ctrl-click during select mode adds elements to the selection (and stays in select mode to keep picking); a plain click replaces and exits as before. state.selectedElements holds the list; the primary selectedElement is preserved for the existing single-element paths. buildContext forwards the full multi-selection (only when >1) and the prompt gains an "All Selected Elements" summary so the model can reason about every chosen element. The context chip shows "N elements" with a selector tooltip. 1 new test (227 total); build + browser E2E green. Completes Phase 7 (palette + design tokens + undo + multi-select). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds three feature cards to docs/index.html (matching the existing .feature-card markup, additive to the responsive grid): Self-verifying edits (the closed loop), Command palette (⌘K), and Framework-aware grounding. Verified by rendering the page in headless Chromium: 13 cards, all three new cards present, zero console/page errors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…code Adds a "Edit element" panel (⌘K command + an "edit" button on the selected-element chip): select an element and edit its styles (curated CSS props + free-form custom CSS), text, class, and attributes. Every keystroke previews LIVE on the real page element (inline style / textContent / attributes). On "Apply to code", the diff of your edits is described to the LLM (describeElementChanges) and sent through the existing review → apply → verify → HMR pipeline so the source is changed to match; "Discard" reverts the live preview to the original. - src/toolbar/element-editor.ts: pure capture/diff/describe/serialize (7 tests). - Live-preview DOM layer + edit panel + input listener in the toolbar (closed shadow root preserved). - browser-smoke now drives the editor via test hooks and asserts the real light-DOM element updates live (style → red, text → changed). 234 unit tests, ESR 6/6, full check:all green incl. live-preview browser E2E. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ses fixes For non-designers: a sparkle pill (and ⌘K "Review UI/UX & propose design fixes") that captures the page, derives its de-facto design system, and asks a vision model to critique and fix it. - src/toolbar/design-audit.ts: derives the page's "brand guidelines" from runtime computed styles — palette (by frequency), type scale, fonts, weights, radii, shadows — and builds a senior-designer review prompt (hierarchy, spacing, contrast, typography, consistency, a11y, responsiveness) that REQUIRES proposals stay consistent with the detected tokens + project design tokens. Pure summarize/format/prompt logic is unit-tested (6 tests); the DOM sampler runs in the browser. - runDesignReview(): screenshot + tokens → review prompt → existing grounding → diff cards (preview) → Apply (verify → HMR). Warns when the model isn't vision. - browser-smoke validates the design-token extraction on a real rendered page. 240 unit tests, ESR 6/6, full check:all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- src/update-check.ts: on startup the CLI asks the npm registry for the latest version and, if newer, prints a one-line notice with the right update command for how it was launched — `npm install -g openmagic@latest` for an installed copy, `npx openmagic@latest` under npx. Numeric semver compare (prerelease-safe), fully fail-silent (offline/timeout → no notice, never errors). Opt out with OPENMAGIC_NO_UPDATE_CHECK=1. - Wired non-blocking into cli.ts after the banner; smoke scripts opt out to stay offline/deterministic. - README documents `npm install -g openmagic` (already worked: the server resolves the toolbar bundle relative to its own module, so a global install serves it correctly — verified end-to-end via pack → global install → run → serve). - 11 unit tests incl. stubbed end-to-end (newer→notice, up-to-date→none, error→fail-silent, npx vs global command). 251 tests, full check:all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tadata Maximize search/AI findability across every surface: - Landing page (docs/index.html): added JSON-LD SoftwareApplication structured data (rich results + AI retrieval), robots/theme-color/og:locale/og:image:alt, and twitter:image. Fixed a real bug — og:image pointed at a non-existent og-image.png (404 social cards); now points at the shipped logo.png. - docs/robots.txt (welcomes GPTBot/ClaudeBot/PerplexityBot/Google-Extended + sitemap), docs/sitemap.xml, and docs/llms.txt (the AI-agent discovery standard: summary, features, install, links, keywords). - npm: expanded keywords 12 → 38 (ai-coding-agent, vibe-coding, ai-design-review, design-to-code, provider/framework names, …); README intro front-loads the differentiators (verified/self-correcting edits, element editor, UI/UX review, frameworks, providers) for npm + AI retrieval. - GitHub repo: sharpened description, set homepage, topics 17 → 20 (+ai-agent, +ai-coding-assistant, +design-to-code) via gh repo edit. - tests/seo.test.ts (10 tests) guards all of the above; landing page re-rendered in Chromium (JSON-LD parses, 0 errors). 261 tests, full check:all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bump version 0.43.2 → 0.44.0 (package.json + CURRENT_VERSION) so merging to main triggers the release workflow and publishes the new metadata/features to npm. Adds an IndexNow key file (docs/) for instant Bing/Yandex submission once the GitHub Pages site is live. 261 tests, full check:all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 1–4 of the "100x" optimization roadmap, stacked on PR #26 (base =
feat/harness-upgrade). Each phase is its own design → TDD → verify → commitcycle, graded against the new eval harness.
Roadmap (
docs/superpowers/specs/2026-05-30-optimize-100x-roadmap.md):8 leverage-ordered phases across all four dimensions (capability, UX, trust,
reach). This PR delivers the first four:
npm run evalreplaysrecorded LLM responses through the real grounding→patch pipeline against
fixture apps, scores Edit-Success-Rate, fails CI on regression. Wired into
check:all+ CI. 5 seed scenarios incl. a safety negative.resolves a
componentHint/term straight to its source — catches targets whosefilename doesn't match (proven by eval build(deps): bump open from 10.2.0 to 11.0.0 #6, which grounds
widgets.tsxfor<PricingTable>). Also afind_symboltool for the agent loop.renderLineDiffrebuilt on areal LCS alignment with correct per-side line numbers + XSS-safe syntax
highlighting. (b) Direct inline text editing: edits the source
deterministically (no LLM) when the text is found verbatim, with a focused
LLM fallback.
cache_controlephemeral block (~90% cheaper/faster on cache hits across a conversation/retry
loop); OpenAI caches automatically.
Status
check:allgreen (typecheck, lint,unit, eval, both smoke tests, browser E2E, pack).
Remaining (Phases 5–8, follow-up)
Phase 5 = H12 server-side agent loop (the larger rewrite, intentionally deferred);
Phase 6 = runtime→self-correct bridge; Phase 7 = UX polish (⌘K, multi-select,
design pass); Phase 8 = reach (landing page, onboarding, more frameworks, docs).
🤖 Generated with Claude Code