diff --git a/.agents/skills/testing-livepeer-fal-deploy/SKILL.md b/.agents/skills/testing-livepeer-fal-deploy/SKILL.md new file mode 100644 index 000000000..e9aec715f --- /dev/null +++ b/.agents/skills/testing-livepeer-fal-deploy/SKILL.md @@ -0,0 +1,327 @@ +--- +name: testing-livepeer-fal-deploy +description: End-to-end test harness for Scope's Livepeer cloud path against a deployed fal.ai app — the only supported cloud path going forward (the old cloud-relay / direct mode using `fal_app.py` + `CloudConnectionManager` is being deprecated). Primary path is a Playwright browser test that drives the full UI flow (camera → local scope WebRTC → livepeer trickle → fal runner → back), producing every session-lifecycle Kafka event. Secondary path is `test-cloud-connect.sh` — a bash/curl smoke test for the `/api/v1/cloud/connect` path only. Has two modes: "deploy then test" (default — runs `deploy-staging.sh` first) and "test existing deploy" (skips deploy, points at whatever is already live, e.g. `scope-livepeer--prod`). TRIGGER any time a user says "test cloud", "test the fal deploy", "test cloud streaming", "run the e2e test", "run playwright", "verify cloud connect", "verify kafka events", "diagnose fal", "debug fal deploy", "did my stream work", "deploy-staging.sh", "test against prod", "test prod cloud", "test the prod deploy", "don't deploy", "skip deploy", "no deploy", "target the existing deploy", "target prod", "scope-livepeer--prod", OR pastes any of these errors — "All orchestrators failed (N tried)", "ACCESS_DENIED", "did not receive ready message from websocket", "discover_orchestrators requires discovery_url", "cold start" — OR has just changed `src/scope/cloud/livepeer_fal_app.py` / `src/scope/cloud/livepeer_app.py` / `src/scope/server/livepeer.py` / `src/scope/server/livepeer_client.py`. Use `testing-livepeer` instead for a fully-local livepeer stack (prebuilt go-livepeer binary, no fal involvement). +--- + +# Testing Livepeer fal Deploy + +## When to use + +Use when testing the **deployed** livepeer path end-to-end — local Scope +client → daydream orchestrator → deployed fal app. This exercises: + +- The wrapper in `src/scope/cloud/livepeer_fal_app.py` that fal runs +- The runner in `src/scope/cloud/livepeer_app.py` that spawns inside the + fal container +- The orchestrator → fal handshake (headers, auth, cold start) +- Kafka event publishing across wrapper + runner (full lifecycle) + +**Two paths, pick the right one:** + +- **Playwright (primary)** — real browser drives the Perform-mode UI + with a synthetic camera, streams through, verifies the output video + comes back from the cloud. This is the only path that exercises the + full livepeer trickle round-trip and produces every lifecycle Kafka + event (`pipeline_loaded`, `session_created`, `stream_started`, + `stream_heartbeat`, `session_closed`). Takes 2–5 minutes. +- **`test-cloud-connect.sh` (secondary, HTTP-only)** — bash script that + POSTs `/api/v1/cloud/connect` and polls `/api/v1/cloud/status`. Only + verifies the `websocket_connected` / `websocket_disconnected` pair at + the wrapper layer. Useful as a fast smoke test ("did the container + come up?") or in `git bisect run` against cloud-connect regressions. + Does not produce pipeline/session/stream events. + +Do **not** use this skill for local-only livepeer testing — that's +`testing-livepeer` (prebuilt go-livepeer + local runner, no fal). + +## One-time setup + +1. **`.env.local`**: copy `.env.example` to `.env.local` (gitignored) + and fill in real values: + - `SCOPE_CLOUD_APP_ID` — your fal app URL. For the default `main` + env, the URL does **not** include a `--main` suffix (e.g. + `daydream/scope-livepeer-emran/ws`). Non-default envs do include + the suffix (e.g. `--preview/ws`). + - `SCOPE_CLOUD_API_KEY` — daydream cloud API key (sk_...). Without + this the scope client can't hit `signer.daydream.live` and fails + with `discover_orchestrators requires discovery_url or signer_url`. + - `SCOPE_USER_ID` — daydream user id. The runner's + `validate_user_access` rejects with `ACCESS_DENIED` when missing. + Find it in `~/.daydream-scope/logs/scope-logs-*.log` after a + successful UI connect, or in devtools Network on + `/api/v1/cloud/connect`. + - (Optional) `LIVEPEER_DEBUG=1` — surfaces per-orchestrator + rejection reasons in scope.log; essential for diagnosing + `All orchestrators failed (N tried)`. +2. **Frontend rebuild with baked-in auth** (once per local workspace): + ```bash + source .env.local + cd frontend && VITE_DAYDREAM_API_KEY="$SCOPE_CLOUD_API_KEY" npm run build + cd .. + ``` + This bakes the API key into the dist bundle so the app appears + signed-in (otherwise Playwright hits the login screen). +3. **Playwright setup** (once per machine): + ```bash + cd e2e + npm install + npx playwright install chromium + ``` + Then install Chromium's system deps (sudo required — one-time): + ```bash + sudo apt-get install -y libnss3 libnspr4 libasound2t64 + # or the Playwright-managed superset: + sudo npx playwright install-deps chromium + ``` + Without these the browser fails to launch with + `error while loading shared libraries: libnspr4.so`. + +## Running the Playwright test (primary) + +There are two modes. Pick by what the user said: + +- **Deploy-then-test (default)** — user said "test cloud" / "test the + fal deploy" / changed cloud code and wants to verify it. Run all + steps below including Step 3 (deploy). +- **Test-existing-deploy (no deploy)** — user said "test against + prod", "don't deploy", "no deploy", "target the existing deploy", + "scope-livepeer--prod", or otherwise made clear they want to test + whatever is *already live*. **Skip Step 3 entirely.** See + ["Variant: target an existing deploy"](#variant-target-an-existing-deploy-no-deploy) + below before running. + +When the user says "test cloud" (or any trigger in the description) +without indicating they want to skip deploy, **always deploy their +current working tree before running Playwright**. Otherwise the test +runs against whatever stale code was last deployed and can +false-positive on their change. + +### Step 0 — Ask the user where to deploy + +Before anything else, confirm the deploy target. Use AskUserQuestion +(or plain text prompts) and persist answers for the session: + +1. **Fal app name** — required. If `SCOPE_FAL_APP_NAME` is set in + `.env.local`, show that value and ask the user to confirm or + override. Otherwise ask outright (e.g. `scope-livepeer-`). +2. **Fal env** — defaults to `main`. If `SCOPE_FAL_ENV` is set in + `.env.local`, show and offer to override. Non-default envs (e.g. + `preview`) change the URL suffix in `SCOPE_CLOUD_APP_ID` — see + below. + +Once confirmed, export both for the current shell, and derive / +overwrite `SCOPE_CLOUD_APP_ID`: + +| Env | `SCOPE_CLOUD_APP_ID` | +|---|---| +| `main` | `daydream//ws` (no suffix) | +| anything else | `daydream/--/ws` (with suffix) | + +This is a fal convention — the default `main` env is exposed without +a suffix; all other envs include `--` in the URL. Getting this +wrong produces `did not receive ready message from websocket`. + +### Step 1 — Sanity-check `.env.local` + +- `SCOPE_CLOUD_API_KEY` must be set (otherwise: + `discover_orchestrators requires discovery_url or signer_url`) +- `SCOPE_USER_ID` must be set (otherwise the runner's + `validate_user_access` rejects with `ACCESS_DENIED`) + +If either is missing, stop and ask the user before deploying. + +### Step 2 — Kill any scope already on :8000 + +If another scope process is bound to the port, stop it (or ask the +user) before continuing. The run-app.sh the script starts must be the +one under test. + +### Step 3 — Deploy + +```bash +SCOPE_FAL_APP_NAME= SCOPE_FAL_ENV= ./deploy-staging.sh +``` + +Abort with a clear error if this fails — don't run Playwright against +stale deployed code. Common failure: the `{git-short-sha}-cloud` +Docker base image isn't built yet (CI for the current commit is still +running). If that's the case, either wait for CI or have the user +confirm they want to deploy against an older base image. + +### Step 4 — Start scope and run Playwright + +```bash +# Terminal 1 — scope (port 8000) +SCOPE_CLOUD_APP_ID= ./run-app.sh + +# Terminal 2 — test +cd e2e && npx playwright test +``` + +Expected on success (≤5 min cold, ~20 s warm): + +``` +Enabling cloud mode... ✅ +Waiting for cloud connection... ✅ +Selecting passthrough model... ✅ +Switching input source to Camera... ✅ +Starting stream... ✅ +Verifying output stream processing... ✅ Output frames flowing +Stopping stream... ✅ +1 passed +``` + +**What the test does in livepeer terms:** + +1. Navigates to `localhost:8000`, switches the UI to Perform mode. +2. Opens settings, flips Remote Inference on, waits for Connection ID + (proves the fal WebSocket handshake completed and + `websocket_connected` fired in Kafka). +3. Selects the `passthrough` pipeline — triggers `pipeline/load`, which + runs on the fal runner and emits `pipeline_load_start` + + `pipeline_loaded`. +4. Switches the input source to Camera — Playwright's launch args + `--use-fake-device-for-media-stream` and + `--use-fake-ui-for-media-stream` (configured in + `e2e/playwright.config.ts`) give `getUserMedia()` a synthetic feed. + This is essential: without a real MediaStream, the browser↔local + scope WebRTC ICE never completes, `CloudTrack._start()` is never + called, and the runner never gets `start_stream`. +5. Clicks the play overlay (`[data-testid="start-stream-button"]`). + Frames flow via livepeer trickle through the orchestrator to the + fal runner; the runner emits `session_created` and `stream_started`. +6. Waits 15 s so at least one `stream_heartbeat` fires on the runner. +7. Asserts the **output** `