feat(ui): doc-backed timeline editor + MCP server (v0.3.0)#36
Merged
Conversation
…s, playhead) The browser UI was built entirely on the legacy session/output-file model — it never read the CompositionDoc the Phase 0 engine maintains (`grep api/timeline src/ui/web/src` was zero hits), so it looked like an op log, not an editor. That is the real source of "sketchy": architectural, not cosmetic. Add the human window onto the doc, reusing endpoints that already exist: - useTimeline() polls GET /api/timeline (mirrors useSession). - lib/composition.ts mirrors the doc geometry helpers (clipDuration / clipEndSec / compositionDuration) so clips position by their document extent. - DocTimeline renders a seconds ruler, one lane per track, clips as blocks (startSec → clipDuration), transitions as boundary markers, clip selection, and an accessible range-input playhead. Wired as the primary surface above the (now demoted) legacy outputs/render-queue strip. - .app switches to a flex column so stacked strips size to content and `main` flexes to fill. Read-only for now (no doc mutation): editing via the existing verb layer and the composited-frame preview are the next increments. Verified in a real browser — a 2-clip + transition composition renders with correct lanes/ruler/marker, clip select highlights, and the playhead scrubs.
…surface) The browser chat panel called Anthropic directly (POST /api/chat → streamText) — the sole thing needing ANTHROPIC_API_KEY and the sole network call in `clip ui`. The agent-editing story is better served by Claude Code (skill → verbs, no key) and, next, an MCP server for Claude Desktop — both drive the same verb layer, and the timeline view live-updates via polling regardless of which agent edits. The in-app chat returns later for the hosted/web product. Remove: ChatPanel, the /api/chat routes + chat.json persistence + buildSystemPrompt, buildTimelineTools/buildChatTools, the Chat header button, the chat CSS, and the @ai-sdk/* + ai dependencies. Keep makeVerbContext + summarizeComposition (used by the verbs route, reusable by the MCP server). Result: `clip ui` makes zero network calls and needs no key; the JS bundle drops 437 kB → 235 kB (gzip 120 → 71). Verified in-browser: no Chat button, the timeline and ops intact, no console errors. 539 tests pass.
The UI read as a light, utilitarian tool, not an editor. Re-theme it as a cohesive dark NLE and restructure toward the familiar editor shape. - Token-driven dark design system (:root vars for bg/border/text/accent/radii/ shadow) applied across every surface — header, op-list, inspector, modal, forms, import zone, outputs strip, snapshots, and the doc timeline — so nothing is left as a stray light panel. Polished buttons, inputs, focus rings, thin dark scrollbars. - Dock the CompositionDoc timeline at the BOTTOM (work area above, timeline below — the NLE layout), with a bounded height. - Timeline made the centerpiece: dark lane bed, clips as gradient blocks with a per-kind accent top-border (media/text/color), a glowing selection ring, a crisp red playhead with a handle, refined ruler, amber transition markers. Verified live: cohesive dark editor look, timeline at the bottom, clip selection + playhead scrub work, no console errors. (Bundle unchanged; CSS-only + a small App.tsx reorder.)
…nsition/remove) Make the timeline editable, not just viewable. Selecting a clip opens a ClipInspector that emits CompositionVerbs through POST /api/timeline/verbs — the same op-aware, undoable path the agent and CLI use, so human edits and agent edits share one reducer. - ClipInspector: numeric trim (media source window), move (timeline start), split-at-playhead, add-transition-into-next-clip, remove. set_transform is intentionally omitted — the compiler rejects a non-identity transform, so the UI must not author one (the guardrail: never produce a doc that can't export). - Playhead lifted to App (controlled) so split-at-playhead knows the time; the range scrubber drives it. Op-selection and clip-selection are mutually exclusive — each owns the inspector pane. - lib/verbs.ts posts verbs and surfaces the server's 4xx message (e.g. 422 bad clip ref); after a successful edit the timeline + session refresh immediately. - findClip / nextClipOnTrack helpers; inspector styled to the dark theme. Verified live in the browser: select a clip → trim out 3s→2s → the document mutates (rev 4→5) and the timeline + inspector reflect it. Closes e2e #3–6.
Add the monitor — the most "NLE" piece — and one-click export, both through
the real export compiler so the preview can't diverge from the render.
Server:
- GET /api/timeline/frame?at=<sec> — the composited frame, rendered via
buildFrameAtPlan (the real compiler path); image/jpeg, 422 + reason when the
doc can't render a frame there. Transient frame files are cleaned up.
- GET /api/timeline/exportable — dry-run compile (pure) → { exportable, blockers }.
- POST /api/timeline/export — compile the whole doc to an mp4 and log it as a
session op so it lands in Outputs, playable.
UI:
- Viewer pane (top of the work area): shows the frame at the playhead, debounced
so scrubbing doesn't render per pixel, with a Renderable / blocker banner.
- Export button in the timeline header → POST export → the result appears in
Outputs and selects for playback.
Verified live: the frame endpoint returns a real composited JPEG (img loads at
1920px); CPU stays ~0% idle (no re-render churn); the Export button produces a
timeline_export op whose mp4 decodes. Closes e2e #9 (export) and #10 (frame).
The header "Undo" acts on the session log; the timeline needed its own undo/redo on the CompositionDoc op-log. Add ↶/↷ buttons in the timeline header, gated by /api/timeline/history (canUndo/canRedo, refetched on every doc change), backed by new POST /api/timeline/undo and /redo routes over undoLastDocOp/redoDocOp. Verified live: undo reverts the last edit (trim 3→2, rev 6→7), redo reapplies it (2→3, rev 7→8); the buttons enable/disable from the op-log cursor. Closes e2e #8.
A second front door onto the SAME op-aware, undoable CompositionDoc the CLI, clip ui, and Claude Code skill edit — so Claude Desktop / Cursor / any MCP client can run the editor on the user's own Claude subscription, no API key. - src/mcp/server.ts: an McpServer over @modelcontextprotocol/sdk (MIT) exposing timeline_show / timeline_edit (verbs) / timeline_undo / timeline_redo / timeline_export. Reuses makeVerbContext, so MCP media paths are confined to the workspace (AGENTS.md non-negotiable #3); Zod validates verbs at the boundary. - `clip mcp` runs it over stdio (lazy-imported so the SDK never loads for other CLI commands; branched before any stdout so the protocol channel stays clean). - README: an "As an MCP server" install block with the Claude Desktop config. Verified: an in-process MCP client (3 tests — list tools, edit+undo/redo, reject bad verb) AND a real stdio smoke (`node dist/cli.js mcp` spawned, tools listed, edit applied, doc read back, clean protocol I/O). 542 tests pass.
…e editor The chat sidebar (and the AI SDK, chat.json, ANTHROPIC_API_KEY) were removed, but the shipped docs still described them — false claims headed for v0.3.0. Correct README, SKILL.md, and llms.txt: - Remove every chat-panel / Vercel AI SDK / chat.json / API-key reference. The editor embeds no model; agent editing runs on the user's own Claude (Claude Code skill or the MCP server). No API key anywhere. - Flip the "MCP: not yet" FAQ to "yes" and document `clip mcp` + the Claude Desktop config; the architecture diagram and dep list now show MCP, not the AI SDK. - Reframe the browser UI as the dark timeline editor (monitor, editable timeline, export, undo/redo). Surfaces 3 → 4 (skill, CLI, UI, MCP). Test count → 542.
The MCP server (clip mcp) is now a first-class surface, so feat(mcp) needs a sanctioned scope. Add it to both the AGENTS.md one-liner and the CONTRIBUTING.md scope reference, placed next to cli.
The viewer's frame-error reset effect and the exportable/history fetch effects re-ran on rev / playhead change but never referenced those values in their bodies, so biome's correctness/useExhaustiveDependencies flagged them as errors — which fails the release pipeline's lint gate. - Viewer: frameError is now derived (erroredSrc === src) instead of a sticky boolean reset by an effect, so it clears itself the instant the playhead or rev yields a new src. Drops a whole effect. - exportable + history fetches now carry ?rev=<rev>, which both references the dependency (satisfying the rule) and cache-busts the request. No behavior change beyond the self-clearing error; lint is clean (0 errors).
yultyyev
force-pushed
the
feat/ui-timeline
branch
from
June 18, 2026 22:31
9e49673 to
b1cbbad
Compare
yultyyev
changed the title
yultyyev
marked this pull request as ready for review
June 18, 2026 22:43
The 'dry-run compileTimeline and turn a CompileError into {exportable,
blockers}' dance was copy-pasted in the CLI timeline show and the clip ui
exportable route. Lift it into one exported checkExportable in compile.ts so
a third caller (the MCP server) reuses it and every surface reports
renderability identically. Pure refactor — no behaviour change.
Split set_transform out of the verb union into its own schema and expose EditableVerbSchema — the eight verbs whose results the compiler can render. CompositionVerbSchema is unchanged (the full set incl. set_transform, for the CLI/op layer). This lets the UI and MCP accept only renderable edits without re-listing the verbs. No behaviour change on its own.
Three pre-release-audit follow-ups for the MCP server:
- timeline_show/edit now report {exportable, blockers} (via the shared
checkExportable), so an agent learns a doc cannot render up front instead
of only at timeline_export. summarizeComposition gains a typed return.
- timeline_edit validates against EditableVerbSchema, so set_transform — which
the compiler refuses — is no longer accepted (mirrors the ClipInspector UI).
- the server version is read from package.json (walking up so it resolves from
both src under vitest and the bundled dist/cli.js) instead of a hardcoded
string that would go stale at the next release. A test asserts the handshake
version matches package.json.
Verified the bundled path with a real node dist/cli.js mcp stdio handshake.
Intermediate segment/text/fold filenames were deterministic (tl-seg-<index>-<clip>), so a preview frame render and a concurrent export wrote and ffmpeg -y-overwrote the SAME files — and each unlinked the other's in run-plan's finally — yielding a transient garbled frame or 500. The timeline doc (source of truth) was never touched, so it self-healed on retry, but it is a real race on the unauthenticated localhost dev server. Add an optional tag to CompileContext, mixed into the intermediate names. Export stays deterministic (no tag); the clip ui sets a per-request tag on each frame render (next commit), so previews get their own paths and cannot collide. Default (untagged) paths are unchanged — existing plans/tests are untouched.
- The frame route passes a per-request tag to buildFrameAtPlan, so a preview's intermediate segments never collide with a concurrent export's. - Frame and export routes translate a non-CompileError (a transient FFmpeg failure) into a clean 503 instead of a raw 500. - Viewer fetches the frame as a blob behind an AbortController, so a superseded scrub is cancelled instead of left rendering, and a 503 reads as a message rather than a broken img. The previous frame stays on screen until the next decodes — no blank flash.
fix(timeline): isolate preview renders from concurrent exports
feat(mcp): harden the MCP server — renderability, verb narrowing, real version
GET /api/media/thumb?mediaId=&t=&h= extracts one small still from a SOURCE media file (not the composited timeline) via a fast input-seek frame grab (buildThumbnailArgs). Each thumb is keyed by (mediaId, t, h) and cached under the workspace, generated to a temp path then atomically renamed so concurrent filmstrip requests never serve a half-written file. Unknown media → 404; a non-thumbnailable source (e.g. audio-only) → 422, so a clip just falls back to its gradient block.
Media clips now show a row of source-frame thumbnails across their trimmed window — a filmstrip — instead of a flat gradient block. The clip samples one /api/media/thumb still per ~72px slot (capped at 16), at the middle of each slot; the gradient shows underneath until the frames load, a scrim keeps the label/duration legible, and text/color clips keep their solid block. Thumbs are lazy-loaded so long timelines don't fetch off-screen frames.
The playhead was a non-interactive line (pointer-events:none) and the only scrub control was a disconnected 220px slider in the header — there was nothing to grab on the timeline. Now: - The playhead is a draggable handle: pointer-down + drag scrubs to the cursor anywhere along the timeline (setPointerCapture + a clientX→sec map that accounts for the gutter and horizontal scroll). It renders as the conventional thin line + a downward-triangle marker. - A transparent native range over the ruler is the scrub surface: click or drag to position, arrow keys (one frame each) move the playhead — the accessible, keyboard-friendly control that replaces the removed header slider. Also removes a stale duplicate .doc-tl-playhead::before that survived an earlier refactor and was offsetting the line with a stray border wedge.
feat(ui): timeline polish — filmstrips + draggable playhead
|
🎉 This PR is included in version 0.3.0 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
v0.3.0 — doc-backed timeline editor + MCP server
The release bundle. Rebuilds the browser UI on the CompositionDoc the Phase 0 engine maintains, adds an MCP server so Claude Desktop (or any MCP client) can drive the editor key-free, and removes the only API-key-requiring surface. Merging this PR to
maincuts v0.3.0 (minor, from 0.2.1) via semantic-release → one CHANGELOG → signed npm publish with provenance.Bundles the follow-ups that were folded into this branch: #37 (MCP hardening), #38 (concurrent-render fix), #39 (filmstrips + draggable playhead) — so it all ships as one release.
What's in it
useTimeline()+DocTimelinerender the document (ruler, per-track lanes, clips by extent, transitions). Edit via the shared verb layer (trim / move / split / transition / remove); a composited-frame viewer renders the playhead through the real export compiler; export from the UI; undo/redo on the doc op-log./api/media/thumb); the playhead is a draggable handle (line + downward-triangle marker) and the ruler is a click/drag/keyboard scrub rail.clip mcp) —timeline_show/timeline_edit/timeline_undo/timeline_redo/timeline_exportover stdio for Claude Desktop. Media paths stay workspace-confined (AGENTS.md non-negotiable feat(tools): add ingest — probe media and return MediaRef #3).show/editreport renderability;timeline_editaccepts only the renderable verb set (noset_transform); the handshake version derives frompackage.json.clip ui· MCP).One engine, four surfaces
The verb layer is shared by the CLI, the
clip ui, and the MCP server — all driving the same op-aware, undoable CompositionDoc. Agent/UI/MCP ingest paths are workspace-confined; the trusted CLI is not.Verification (on the merged tip)
pnpm type-check✓ · 548 tests ✓ ·pnpm lintexit 0 ✓ ·pnpm build✓ · CI green · commit history brand-clean.Live e2e against the running stack with real FFmpeg:
node dist/cli.js mcp) — version frompackage.json· 5 tools ·timeline_showreports exportable and sees the UI-built timeline (cross-surface) ·set_transformrejected · edit + undo.After merge
semantic-release writes the CHANGELOG, tags v0.3.0, and publishes to npm with SLSA provenance — same as v0.2.0/v0.2.1.
🤖 Generated with Claude Code