feat: local-first Videoboom — on-device MLX pipeline + optional BYOK cloud, per-stage auto-config

.github/workflows/ci.yml

@@ -22,32 +22,52 @@ jobs:
         os: [ubuntu-latest, windows-latest, macos-latest]
     runs-on: ${{ matrix.os }}
     steps:
-      - uses: actions/checkout@v4
-      - uses: actions/setup-node@v4
+      - uses: actions/checkout@v7
+      - uses: actions/setup-node@v7
         with:
-          node-version: 20
+          node-version: 24
           cache: npm
       - run: npm ci
       - name: Typecheck
         run: npm run typecheck
       - name: Unit tests
         run: npm run test:unit
+      # `npm test` is typecheck + test:unit + check:no-cloud, but CI only ran the first two, so the guard
+      # that keeps cloud calls out of a local-resolved render was never enforced on a PR.
+      - name: No-cloud guard
+        run: npm run check:no-cloud
       - name: Build (main + renderer)
         run: npm run build
 
   # Electron integration: the offline engine self-test (real ffmpeg) + the renderer smoke test.
-  # Headless on Linux via xvfb; exit code is the pass/fail signal (app.exit(0|1)).
+  # Exit code is the pass/fail signal (app.exit(0|1)).
+  #
+  # Runs on all three OSes, not just Linux. Installers are shipped for Windows and Linux, but until now
+  # nothing had ever launched the app there — the matrix job below only typechecks, unit-tests and builds,
+  # which cannot catch a main-process crash, a broken preload bridge or an ffmpeg binary that does not
+  # resolve. On-device generation is Apple-Silicon-only by design (localCapabilities), so what this proves
+  # off macOS is the shell: the app boots, the bridge is up, the renderer mounts, and the bundled
+  # ffmpeg/ffprobe work — which is exactly what a Windows or Linux user on a cloud key depends on.
   electron:
-    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, windows-latest, macos-latest]
+    runs-on: ${{ matrix.os }}
     steps:
-      - uses: actions/checkout@v4
-      - uses: actions/setup-node@v4
+      - uses: actions/checkout@v7
+      - uses: actions/setup-node@v7
         with:
-          node-version: 20
+          node-version: 24
           cache: npm
       - run: npm ci
       - run: npm run build
+      # Only Linux needs a virtual display; macOS and Windows runners have a window server.
       - name: Engine self-test (offline ffmpeg path)
-        run: xvfb-run -a env VB_ENGINE_TEST=1 npx electron --no-sandbox .
+        run: ${{ matrix.os == 'ubuntu-latest' && 'xvfb-run -a ' || '' }}npx electron --no-sandbox .
+        env:
+          VB_ENGINE_TEST: '1'
       - name: Renderer smoke test
-        run: xvfb-run -a env VB_SMOKE=1 npx electron --no-sandbox .
+        run: ${{ matrix.os == 'ubuntu-latest' && 'xvfb-run -a ' || '' }}npx electron --no-sandbox .
+        env:
+          VB_SMOKE: '1'

.github/workflows/pages.yml

@@ -23,10 +23,10 @@ jobs:
       url: ${{ steps.deployment.outputs.page_url }}
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v4
-      - uses: actions/configure-pages@v5
-      - uses: actions/upload-pages-artifact@v3
+      - uses: actions/checkout@v7
+      - uses: actions/configure-pages@v6
+      - uses: actions/upload-pages-artifact@v5
         with:
           path: site
       - id: deployment
-        uses: actions/deploy-pages@v4
+        uses: actions/deploy-pages@v5

.github/workflows/release.yml

@@ -16,10 +16,10 @@ jobs:
         os: [macos-latest, windows-latest, ubuntu-latest]
     runs-on: ${{ matrix.os }}
     steps:
-      - uses: actions/checkout@v4
-      - uses: actions/setup-node@v4
+      - uses: actions/checkout@v7
+      - uses: actions/setup-node@v7
         with:
-          node-version: 20
+          node-version: 24
           cache: npm
       - run: npm ci
 
@@ -38,7 +38,7 @@ jobs:
 
       # CI artifacts for every run (also for workflow_dispatch, which has no tag/release).
       - name: Upload build artifacts
-        uses: actions/upload-artifact@v4
+        uses: actions/upload-artifact@v7
         with:
           name: installers-${{ matrix.os }}
           path: |
@@ -51,7 +51,7 @@ jobs:
       # Attach to the GitHub Release for the tag (each OS job appends its own files).
       - name: Publish to release
         if: startsWith(github.ref, 'refs/tags/')
-        uses: softprops/action-gh-release@v2
+        uses: softprops/action-gh-release@v3
         with:
           fail_on_unmatched_files: false
           files: |

.gitignore

@@ -5,6 +5,8 @@ node_modules/
 # builds (electron main bundle + renderer + packaged installers)
 dist/
 build/
+# ...except the signing entitlements, which package.json references and codesign needs.
+!build/entitlements.mac.plist
 renderer-dist/
 release/
 *.tsbuildinfo
@@ -15,6 +17,16 @@ __pycache__/
 sidecar/build/
 sidecar/dist/
 
+# local-model sidecar: venv, downloaded/converted weights, and the path marker (code is tracked)
+local/.venv/
+local/models/
+local/.model-path
+local/.model-path-5b
+local/.model-path-ltx
+local/.lightning-dir
+# generated by scripts/vendor-runtime.sh (uv+wheels live under build/ which is already ignored)
+local/requirements.macos.lock
+
 # bundled sidecar (built per-OS by scripts/build-sidecar.sh)
 resources/sidecar
 
@@ -33,3 +45,5 @@ resources/sidecar
 
 # encrypted key store (lives in userData at runtime; never commit if it lands in-tree)
 keys.json
+# Draw Things headless spike: binaries + model zoo (30GB+), never in the repo
+local/dt/

AGENTS.md

@@ -3,23 +3,25 @@
 Rules any agent (human or AI) MUST follow in this repo. Violations have caused real bugs.
 
 ## What this is
-Videoboom is an **open-source, bring-your-own-key (BYOK) desktop app** that turns a song into a music
-video on the user's own machine. An **Electron** shell (`src/main`, `src/preload`, `renderer/`) runs an
-**in-process TypeScript render engine** (`src/engine/`); ffmpeg ships bundled (`ffmpeg-static`). The user
-pastes their own provider keys; the app calls those providers directly and the user pays them at cost.
-**No accounts, no server, no wallet, no Python, nothing leaves the machine** except the generation API
-calls. Ships for Windows / macOS / Linux.
+Videoboom is an **open-source, local-only desktop app** that turns a song into a music video entirely on
+the user's own machine. An **Electron** shell (`src/main`, `src/preload`, `renderer/`) runs an **in-process
+TypeScript render engine** (`src/engine/`); ffmpeg ships bundled (`ffmpeg-static`). Every generation stage
+runs **on-device** (Apple Silicon / MLX) through a resident Python sidecar (`local/server.py`). **No
+accounts, no server, no wallet, no API keys** — the only network use is downloading the model weights once.
+Nothing about the song or the video leaves the machine. Targets macOS (Apple Silicon).
 
-> History: this was once a local-first Mac app (Phase 0), then an AWS serverless SaaS (coins / Cognito /
-> DynamoDB / Step Functions). Both are gone — that code was deleted (recoverable from git history).
-> Anything mentioning coins, wallets, Cognito, DynamoDB, S3, Lambda, or CDK is **historical**.
+> History: this was once an AWS serverless SaaS (coins / Cognito / DynamoDB), then a bring-your-own-key
+> cloud build (OpenRouter / Replicate). Both are gone — that code was deleted (recoverable from git
+> history). Anything mentioning coins, wallets, Cognito/DynamoDB/S3/Lambda/CDK, **or cloud provider keys
+> (OpenRouter, Replicate, `VB_*_MODEL` slugs, `safeStorage` keychain)** is **historical**.
 
 ## Docs
 - **Keep docs in sync with the code.** Any change to the architecture, the render flow, the engine/IPC
-  contract, key storage, providers, or env/config MUST update the matching doc in the SAME change —
+  contract, the on-device model stack, or env/config MUST update the matching doc in the SAME change —
   stale docs that claim the wrong behavior are treated as bugs.
 - Current docs that MUST stay accurate (keep minimal + truthful): `README.md`, `AGENTS.md`,
-  `docs/ARCHITECTURE.md`, `docs/FOLDER-STRUCTURE.md`, `docs/PROVIDERS.md`, `docs/ROADMAP.md`.
+  `docs/ARCHITECTURE.md`, `docs/FOLDER-STRUCTURE.md`, `docs/MODELS.md`, `docs/LOCAL-MODELS.md`,
+  `docs/ROADMAP.md`.
 - Don't hoard docs — few accurate ones beat many stale ones. `docs/research/*` is point-in-time
   reference; truly dead docs are deleted, not left to rot.
 
@@ -36,19 +38,21 @@ calls. Ships for Windows / macOS / Linux.
 - **English only** — all UI text, code, comments, docs, and commit messages. (Assistant chat replies may
   match the user's language; anything written into the repo is English.)
 
-## Keys & privacy (BYOK)
-- API keys are the **user's**. Stored **encrypted via the OS keychain** (Electron `safeStorage`) in the
-  app's userData; decrypted only in-memory and injected into the render engine per operation as config
-  (`VB_OPENROUTER_API_KEY`, `REPLICATE_API_TOKEN`, …, read via `src/engine/config.ts`). **Never log
-  secrets**; never write them to the project store or to git. The gitignored `keys.json` must never be
-  committed.
-- Nothing is uploaded to a Videoboom server — there is none. The only network calls are to the provider
-  APIs the user configured.
+## Privacy (local-only)
+- There are **no API keys and no secrets** — every stage runs on-device. Do NOT reintroduce cloud clients,
+  key storage (`safeStorage`/`keys.json`), or `VB_*_API_KEY` / provider-token env. Boot removes any stale
+  `keys.json` left by the old cloud build.
+- Nothing is uploaded anywhere. The **only** network use is downloading model weights (Hugging Face) via
+  `local/setup.sh` / `local/download.py`. Generation itself is fully offline.
 
-## Models are user-facing (the opposite of the old SaaS)
-- This is BYOK: the user **chooses** the LLM / image / video models in **Settings**. Every model is a
-  `VB_*_MODEL` env var with a sensible default. Naming models in the UI/docs is fine here. Do NOT
-  re-introduce the old "hide the provider" stripping — that was a SaaS concern.
+## On-device models
+- Stages run locally on Apple Silicon (MLX) through the sidecar: STT (mlx-whisper), story/shot-list LLM
+  (mlx-lm), keyframes (mflux FLUX + Kontext), image-to-video (mlx-video Wan 2.2), portrait caption + safety
+  (mlx-vlm). The TypeScript wrappers in `src/engine/stages.ts` delegate to the `src/engine/local*.ts`
+  modules; keep that indirection (no cloud branch). Video model + fast/hd quality are chosen in Settings
+  (`src/main/settings.ts`, injected as `VB_*` env). See `docs/MODELS.md`.
+- A machine that can't run on-device (not Apple Silicon, or under the RAM floor) is surfaced via
+  `localCapabilities()`; the required-hardware spec lives in ONE place (`HARDWARE_SPEC`).
 
 ## Engine contract
 - The engine runs **in-process** (`src/engine/`), driven by `runEngine(command, args, env, onEvent)`
@@ -58,16 +62,21 @@ calls. Ships for Windows / macOS / Linux.
   contract). Same command set as before: `create-project`, `render` (`--preview`), `resume`,
   `regenerate-scene`, `character-create`, `character-portrait`, `get-project`.
 - The engine is async throughout — it runs in the main process, so it must **never block the event loop**:
-  ffmpeg runs as async child processes, model calls are `fetch`. No `spawnSync` on hot paths.
+  ffmpeg runs as async child processes, model calls are async HTTP to the localhost sidecar. No `spawnSync`
+  on hot paths.
 - State + media are **plain files** under `VB_DATA_DIR` (default the app's userData `data/`).
   `src/engine/storage.ts` is **local filesystem only** — do not reintroduce any cloud coupling.
 - ffmpeg/ffprobe come from `ffmpeg-static` / `ffprobe-static`; packaged builds `asarUnpack` them and the
   engine rewrites `app.asar` → `app.asar.unpacked` in the binary path.
 
 ## Image / video generation
-- **Video model = Kling** (`kwaivgi/kling-v3.0-std` for Fast, `-pro` for HD) via OpenRouter, first+last
-  frame morph (duration {5,10}). Kling animates realistic adults, children, AND toon. Keep
-  `genVideo()` model-agnostic so the user can swap in another i2v model from Settings.
+- **Video model = Wan 2.2** on-device (mlx-video). The Fast/Quality choice IS the model choice: **Fast** =
+  FastWan-5B (DMD 3-step draft, `.model-path-5b` → FastWan2.2-TI2V-5B-MLX, marker-forced in
+  `local/wan_i2v.py`); **Quality** = Wan I2V-A14B bf16-relay. **Both finish at 1080p** — the shot renders
+  at 480p on-device, then the finish pass interpolates (RIFE) + upscales (Real-ESRGAN) to 1080p (native
+  1080p diffusion OOMs on-device). Each scene renders as one continuous shot of chained native sub-clips
+  (single start frame), then trims to the frame grid — never a stretched slow-mo clip. Keep the `5b`/`14b`
+  selection in `localVideo.ts` intact (`localVideoModel` in settings drives it).
 - **Identity**: keyframes are built from the cast's reference portraits with a strong "reproduce every
   facial feature exactly, no blending/de-aging" prompt; the cap (`VB_MAX_SUBJECTS`) must cover the whole
   cast (a dropped reference = an invented subject). The clip animates the keyframe, so keyframe identity
@@ -77,13 +86,8 @@ calls. Ships for Windows / macOS / Linux.
   explicit character/reference image.
 - Prefer **medium/wide shots**; avoid tight close-ups until lip-sync is solved.
 
-## Cost
-- Track REAL provider cost in **cents** (`src/engine/cost.ts`, `costTotal()`); OpenRouter returns it in
-  `usage.cost` (send `usage:{include:true}`). The user sees the **at-cost** number — there is no margin
-  and no wallet. Record cost on success AND failure (partial cost on failed ops).
-
 ## Code quality
-- **No duplication.** Shared logic lives in ONE helper (config/env reads, storage paths, cost accounting,
+- **No duplication.** Shared logic lives in ONE helper (config/env reads, storage paths,
   the progress event emitter). Never copy a block across the engine modules — extract it.
 - **Reusable UI.** Build on the shared `renderer/components/ui.tsx` primitives (Button, Field, Card, …);
   don't re-implement inputs/buttons/modals per screen.