From c12b9c5728fb3b5922130e420b446071df5da710 Mon Sep 17 00:00:00 2001 From: emranemran Date: Sun, 19 Apr 2026 21:19:54 -0700 Subject: [PATCH 1/8] feat: add end-to-end cloud-connect test harness and skill MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ship the scripts and skill we developed while debugging the livepeer fal deploy path, so other contributors can run the same test loop from their own Claude Code session. - test-cloud-connect.sh orchestrates push → CI build-cloud wait → deploy-staging → local scope start → /cloud/connect → status poll with bisect-friendly exit codes (0/1/2/3/4). Supports --skip-push, --skip-build-wait, --skip-deploy, --full-session, --keep-scope. - run-app.sh launches daydream-scope in livepeer cloud mode, sourcing secrets from .env.local (gitignored). - .env.example documents the required SCOPE_CLOUD_APP_ID / SCOPE_CLOUD_API_KEY / SCOPE_USER_ID env vars plus optional LIVEPEER_DEBUG. - .agents/skills/testing-livepeer-fal-deploy/SKILL.md teaches future agents when to use this loop, common failure signatures (ACCESS_DENIED, All orchestrators failed, did not receive ready message), and the known gap around /api/v1/session/start not being livepeer-compatible. Users need to supply their own deploy-staging.sh (a thin wrapper around `fal deploy src/scope/cloud/livepeer_fal_app.py --app --auth public --env main`); the test script errors out with a clear message if it's missing. Co-Authored-By: Claude Opus 4.7 (1M context) Signed-off-by: emranemran --- .../testing-livepeer-fal-deploy/SKILL.md | 144 +++++++ .env.example | 20 + run-app.sh | 27 ++ test-cloud-connect.sh | 390 ++++++++++++++++++ 4 files changed, 581 insertions(+) create mode 100644 .agents/skills/testing-livepeer-fal-deploy/SKILL.md create mode 100644 .env.example create mode 100755 run-app.sh create mode 100755 test-cloud-connect.sh diff --git a/.agents/skills/testing-livepeer-fal-deploy/SKILL.md b/.agents/skills/testing-livepeer-fal-deploy/SKILL.md new file mode 100644 index 000000000..bcc05d6dc --- /dev/null +++ b/.agents/skills/testing-livepeer-fal-deploy/SKILL.md @@ -0,0 +1,144 @@ +--- +name: testing-livepeer-fal-deploy +description: End-to-end test harness for Scope running in Livepeer cloud mode against a deployed fal.ai app. Orchestrates git push, GitHub Actions CI build-cloud wait, deploy-staging, local scope startup, and /cloud/connect verification. Use when the user wants to test a change to `src/scope/cloud/livepeer_fal_app.py` or diagnose cloud-connect failures ("All orchestrators failed", "ACCESS_DENIED", etc.) through the real fal + orchestrator path. For a fully-local livepeer stack (no fal), use `testing-livepeer` instead. +--- + +# Testing Livepeer fal Deploy + +## When to use + +Use when testing the **deployed** livepeer cloud path end-to-end — i.e. local +Scope client → daydream orchestrator → deployed fal app. This exercises: + +- The wrapper in `src/scope/cloud/livepeer_fal_app.py` that fal actually runs. +- The orchestrator → fal handshake (headers, auth, cold start). +- Kafka event publishing from the fal wrapper. + +Do **not** use for local-only livepeer testing — that's what `testing-livepeer` +is for (uses a prebuilt go-livepeer binary + local runner, no fal involvement). + +## One-time setup + +1. Copy `.env.example` to `.env.local` and fill in real values. + `.env.local` is gitignored — never commit it. +2. Required values: + - `SCOPE_CLOUD_APP_ID` — your fal app. The `main` env is exposed **without** + a `--main` suffix in the URL; non-default envs (e.g. `--preview`) include + the suffix. + - `SCOPE_CLOUD_API_KEY` — daydream cloud API key (sk_...). Without it the + scope client cannot call `signer.daydream.live` and orchestrator + discovery fails with `discover_orchestrators requires discovery_url or + signer_url`. + - `SCOPE_USER_ID` — daydream user id. Without it the runner's + `validate_user_access` rejects with `ACCESS_DENIED`. Find it in + `~/.daydream-scope/logs/scope-logs-*.log` after a successful UI connect, + or in a browser devtools Network tab on `/api/v1/cloud/connect`. +3. Optional: `LIVEPEER_DEBUG=1` to surface per-orchestrator rejection reasons + in the scope log (crucial for diagnosing `All orchestrators failed (N tried)`). + +## Running the test + +### One-shot + +```bash +./test-cloud-connect.sh [flags] +``` + +Default flow: git push current branch → wait for GitHub Actions +`docker-build.yml build-cloud` to succeed → run `./deploy-staging.sh` → +start scope via `./run-app.sh` → POST `/api/v1/cloud/connect` → poll +`/api/v1/cloud/status` until connected, errored, or timed out. + +### Flags + +- `--skip-push` — don't `git push` (useful when re-testing without code + changes, or testing `main`). +- `--skip-build-wait` — don't wait for CI (assumes the `-cloud` image is + already built for HEAD). +- `--skip-deploy` — don't run `deploy-staging.sh` (fast iteration when only + the scope client changed). +- `--full-session` — after connect, load a pipeline, start a session, verify + frames flow, stop, and cloud-disconnect. **Known limitation:** in livepeer + mode the `/api/v1/session/start` endpoint is not livepeer-compatible + (see `TODO` at `src/scope/server/mcp_router.py:252`), so this flag hits a + "Pipeline X not loaded" error. Use a manual UI test if you need the + `pipeline_loaded` / `session_created` / `stream_started` / `stream_heartbeat` + Kafka events. +- `--keep-scope` — leave scope running after the test (don't kill). +- `--port N` — change the local scope port (default 8000). + +### Env overrides + +`PORT`, `TIMEOUT_CONNECT` (default 180s, bump to 300+ for cold starts), +`TIMEOUT_HEALTH`, `TIMEOUT_CI`, `TIMEOUT_PIPELINE`, `TIMEOUT_FRAMES`, +`PIPELINE_ID`, `TEST_VIDEO`. + +## Exit codes + +Bisect-friendly — `git bisect run ./test-cloud-connect.sh` works. + +| Code | Meaning | +|---|---| +| 0 | Connected (and if `--full-session`, frames flowed) | +| 1 | Cloud reported an error — see `error` field in status response | +| 2 | Timed out waiting for connect / pipeline / frames | +| 3 | Infra failure — push / CI / deploy / scope startup | +| 4 | Session-level failure (pipeline load, session start, no frames) | + +## Logs + +- `/tmp/test-cloud-connect/driver.log` — script's own progress log +- `/tmp/test-cloud-connect/scope.log` — stdout/stderr of the local scope + process. If cloud connect fails, grep here for `livepeer_gateway` — + that's where rejection reasons land when `LIVEPEER_DEBUG=1`. +- `~/.daydream-scope/logs/scope-logs-*.log` — scope's rolling app logs + (separate from the test-captured log; useful for historical runs). +- fal deployment dashboard — the fal container's stdout/stderr, including + `[KAFKA-DEBUG]` lines, runner subprocess logs, and Kafka publisher state. + Not accessible via CLI; open the fal.ai dashboard for the app. + +## Common failure signatures + +- **`All orchestrators failed (N tried)`** — generic; enable `LIVEPEER_DEBUG=1` + and re-run to get the specific per-orchestrator reason. Typical underlying + causes: + - `did not receive ready message from websocket` → fal URL wrong (e.g. + extra `--main` suffix) or container still cold-starting. + - `serverless handshake failed (ACCESS_DENIED)` → runner's + `validate_user_access` rejected because `SCOPE_USER_ID` was missing + or the daydream API couldn't find the user. +- **`discover_orchestrators requires discovery_url or signer_url`** → + `SCOPE_CLOUD_API_KEY` not set, so the signer fallback isn't configured. +- **`FrameProcessor failed to start: Pipeline not loaded`** — you used + `--full-session` in livepeer mode. Known gap; use UI for pipeline/session + events. + +## Typical workflows + +**Verify a new commit works:** +```bash +./test-cloud-connect.sh # full cycle +``` + +**Fast iteration (no redeploy needed):** +```bash +./test-cloud-connect.sh --skip-push --skip-build-wait --skip-deploy +``` + +**Bisect a regression:** +```bash +git bisect start HEAD known-good-sha +git bisect run ./test-cloud-connect.sh --skip-push +``` +(Each iteration triggers a full CI+deploy, so bisects are slow — budget +~10+ min per step.) + +## What this skill does NOT cover + +- Full-session event coverage (`pipeline_loaded` / `session_created` / + `stream_started` / `stream_heartbeat`) — those fire from the cloud runner + (or via WebRTC on the local side) and require a real streaming session + that `/api/v1/session/start` doesn't support in livepeer mode yet. Today, + trigger them by running scope with the UI and pushing frames through. +- ClickHouse verification — this skill produces Kafka events; use the + user's ClickHouse MCP or dashboard to verify they land downstream. diff --git a/.env.example b/.env.example new file mode 100644 index 000000000..eebbcbaaa --- /dev/null +++ b/.env.example @@ -0,0 +1,20 @@ +# Copy this file to `.env.local` (gitignored) and fill in real values. +# Used by run-app.sh and test-cloud-connect.sh. + +# Required — fal app ID for your livepeer deployment. +# Format: daydream//ws (no --main suffix for the default env; +# for non-default envs the URL includes the env, e.g. --preview/ws) +export SCOPE_CLOUD_APP_ID=daydream//ws + +# Required — daydream cloud API key (used to auth with signer.daydream.live). +export SCOPE_CLOUD_API_KEY=sk_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + +# Required for the full automated test — your daydream user id. +# Find it via the Scope UI cloud-connect request body, or in +# ~/.daydream-scope/logs/scope-logs-*.log after a UI-driven connect. +export SCOPE_USER_ID=user_xxxxxxxxxxxxxxxxxxxxxxxxx + +# Optional — enable DEBUG logs from livepeer_gateway so per-orchestrator +# rejection reasons appear in scope.log (e.g. "ACCESS_DENIED", +# "did not receive ready message from websocket"). +# export LIVEPEER_DEBUG=1 diff --git a/run-app.sh b/run-app.sh new file mode 100755 index 000000000..b9a68d4da --- /dev/null +++ b/run-app.sh @@ -0,0 +1,27 @@ +#!/bin/bash +# Run daydream-scope in livepeer cloud mode. +# +# Requires `.env.local` (gitignored) exporting at minimum: +# SCOPE_CLOUD_APP_ID e.g. daydream/scope-livepeer-/ws +# SCOPE_CLOUD_API_KEY daydream cloud API key (sk_...) +# Optional in `.env.local`: +# SCOPE_USER_ID daydream user id (used by test-cloud-connect.sh) +# LIVEPEER_DEBUG=1 surface per-orchestrator rejection reasons +# +# See .env.example for a template. + +set -euo pipefail +HERE="$(cd "$(dirname "$0")" && pwd)" + +if [ -f "$HERE/.env.local" ]; then + # shellcheck disable=SC1091 + source "$HERE/.env.local" +fi + +: "${SCOPE_CLOUD_APP_ID:?Set SCOPE_CLOUD_APP_ID in .env.local (see .env.example)}" + +SCOPE_CLOUD_MODE=livepeer \ +SCOPE_CLOUD_APP_ID="$SCOPE_CLOUD_APP_ID" \ +${SCOPE_CLOUD_API_KEY:+SCOPE_CLOUD_API_KEY="$SCOPE_CLOUD_API_KEY"} \ +${LIVEPEER_DEBUG:+LIVEPEER_DEBUG="$LIVEPEER_DEBUG"} \ +uv run daydream-scope "$@" diff --git a/test-cloud-connect.sh b/test-cloud-connect.sh new file mode 100755 index 000000000..6366de1b3 --- /dev/null +++ b/test-cloud-connect.sh @@ -0,0 +1,390 @@ +#!/bin/bash +# End-to-end cloud-connect test for the livepeer fal deploy. +# +# Flow: +# 1. (optional) push current branch to origin +# 2. (optional) wait for CI `build-cloud` to succeed for HEAD +# 3. (optional) run deploy-staging.sh to deploy the fal wrapper +# 4. start daydream-scope locally via ./run-app.sh +# 5. POST /api/v1/cloud/connect +# 6. poll /api/v1/cloud/status until connected, errored, or timed out +# 7. (--full-session) load pipeline, start session, wait for frames, +# stop session, cloud disconnect +# +# Exit codes (bisect-friendly): +# 0 success (connected, and if --full-session then frames flowed) +# 1 cloud reported error +# 2 timed out waiting for connect / pipeline / frames +# 3 infra failure (push / CI / deploy / scope startup) +# 4 session-level failure (pipeline load, session start, no frames) + +set -euo pipefail + +PORT="${PORT:-8000}" +TIMEOUT_CONNECT="${TIMEOUT_CONNECT:-180}" +TIMEOUT_HEALTH="${TIMEOUT_HEALTH:-60}" +TIMEOUT_CI="${TIMEOUT_CI:-1800}" +TIMEOUT_PIPELINE="${TIMEOUT_PIPELINE:-300}" +TIMEOUT_FRAMES="${TIMEOUT_FRAMES:-60}" +PIPELINE_ID="${PIPELINE_ID:-passthrough}" +TEST_VIDEO="${TEST_VIDEO:-/tmp/test_input.mp4}" +SKIP_PUSH=0 +SKIP_BUILD_WAIT=0 +SKIP_DEPLOY=0 +KEEP_SCOPE=0 +FULL_SESSION=0 + +usage() { + cat < "$DRIVER_LOG" +: > "$SCOPE_LOG" + +log() { echo "[$(date +%H:%M:%S)] $*" | tee -a "$DRIVER_LOG"; } +fail() { log "FAIL: $*"; exit "${2:-3}"; } + +SCOPE_PID="" +cleanup() { + local ec=$? + if [[ $KEEP_SCOPE -eq 0 && -n "$SCOPE_PID" ]]; then + log "Stopping scope (pid=$SCOPE_PID)" + kill "$SCOPE_PID" 2>/dev/null || true + wait "$SCOPE_PID" 2>/dev/null || true + elif [[ $KEEP_SCOPE -eq 1 && -n "$SCOPE_PID" ]]; then + log "Leaving scope running (pid=$SCOPE_PID, logs $SCOPE_LOG)" + fi + log "Exit code: $ec" + exit $ec +} +trap cleanup EXIT INT TERM + +# JSON field extractor via python3 (jq not available everywhere) +json_get() { + # $1 = field path (e.g. ".connected" or ".error") + # stdin = json + python3 -c " +import json, sys +try: + d = json.load(sys.stdin) +except Exception as e: + print(f'', file=sys.stderr) + sys.exit(0) +path = '$1'.lstrip('.').split('.') +v = d +for p in path: + if isinstance(v, dict): + v = v.get(p) + else: + v = None + break +if v is None: + print('') +elif isinstance(v, bool): + print('true' if v else 'false') +else: + print(v) +" +} + +# --- 1. Push ------------------------------------------------------- +if [[ $SKIP_PUSH -eq 0 ]]; then + if ! git diff-index --quiet HEAD --; then + fail "Uncommitted changes present. Commit first or pass --skip-push." 3 + fi + BRANCH=$(git rev-parse --abbrev-ref HEAD) + log "Pushing $BRANCH to origin..." + git push origin "$BRANCH" 2>&1 | tee -a "$DRIVER_LOG" +fi + +SHA=$(git rev-parse HEAD) +SHORT_SHA=$(git rev-parse --short HEAD) +log "Testing commit: $SHORT_SHA" + +# --- 2. Wait for CI build-cloud ------------------------------------ +if [[ $SKIP_BUILD_WAIT -eq 0 ]]; then + log "Locating CI build-cloud run for $SHORT_SHA..." + START=$(date +%s) + RUN_ID="" + while [[ -z "$RUN_ID" ]]; do + if [[ $(($(date +%s) - START)) -gt 180 ]]; then + fail "No CI run found for $SHORT_SHA after 3 min" 3 + fi + RUN_ID=$(gh run list --workflow=docker-build.yml --commit "$SHA" \ + --json databaseId --jq '.[0].databaseId' 2>/dev/null || true) + [[ -z "$RUN_ID" ]] && sleep 5 + done + log "Watching CI run $RUN_ID (timeout ${TIMEOUT_CI}s)..." + if ! timeout "$TIMEOUT_CI" gh run watch "$RUN_ID" --exit-status --interval 15 \ + 2>&1 | tee -a "$DRIVER_LOG"; then + fail "CI run $RUN_ID did not succeed" 3 + fi + log "CI succeeded" +fi + +# --- 3. Deploy ----------------------------------------------------- +if [[ $SKIP_DEPLOY -eq 0 ]]; then + if [[ ! -x ./deploy-staging.sh ]]; then + fail "./deploy-staging.sh not found or not executable. Create one that runs \`fal deploy src/scope/cloud/livepeer_fal_app.py --app --auth public --env main\`, or pass --skip-deploy." 3 + fi + log "Running ./deploy-staging.sh..." + if ! ./deploy-staging.sh 2>&1 | tee -a "$DRIVER_LOG"; then + fail "deploy-staging.sh failed" 3 + fi + log "Deploy completed" +fi + +# --- 4. Start scope ------------------------------------------------ +log "Freeing port $PORT..." +lsof -ti:"$PORT" 2>/dev/null | xargs -r kill -9 2>/dev/null || true +sleep 1 + +log "Starting scope (logs: $SCOPE_LOG)..." +./run-app.sh --port "$PORT" > "$SCOPE_LOG" 2>&1 & +SCOPE_PID=$! +log "Scope pid=$SCOPE_PID" + +log "Waiting for /health..." +START=$(date +%s) +while ! curl -sf "$SCOPE_URL/health" > /dev/null 2>&1; do + if [[ $(($(date +%s) - START)) -gt $TIMEOUT_HEALTH ]]; then + log "Scope health timeout. Last 50 log lines:" + tail -50 "$SCOPE_LOG" | tee -a "$DRIVER_LOG" + fail "Scope did not become healthy" 3 + fi + if ! kill -0 "$SCOPE_PID" 2>/dev/null; then + log "Scope process died. Last 50 log lines:" + tail -50 "$SCOPE_LOG" | tee -a "$DRIVER_LOG" + fail "Scope process exited" 3 + fi + sleep 1 +done +log "Scope healthy" + +# --- 5. Connect ---------------------------------------------------- +# Source .env.local so SCOPE_USER_ID is available for the connect body. +if [ -f "$(dirname "$0")/.env.local" ]; then + # shellcheck disable=SC1091 + source "$(dirname "$0")/.env.local" +fi +CONNECT_BODY='{}' +if [[ -n "${SCOPE_USER_ID:-}" ]]; then + CONNECT_BODY=$(python3 -c "import json,os; print(json.dumps({'user_id': os.environ['SCOPE_USER_ID']}))") +fi +log "POST /api/v1/cloud/connect (user_id=${SCOPE_USER_ID:-})" +CONNECT_RESP=$(curl -sf -X POST "$SCOPE_URL/api/v1/cloud/connect" \ + -H 'Content-Type: application/json' -d "$CONNECT_BODY") +log "Connect response: $CONNECT_RESP" + +# --- 6. Poll status ------------------------------------------------ +log "Polling /api/v1/cloud/status (timeout ${TIMEOUT_CONNECT}s)..." +START=$(date +%s) +LAST_STAGE="" +while true; do + ELAPSED=$(($(date +%s) - START)) + if [[ $ELAPSED -gt $TIMEOUT_CONNECT ]]; then + log "TIMEOUT after ${ELAPSED}s" + curl -s "$SCOPE_URL/api/v1/cloud/status" | tee -a "$DRIVER_LOG" + echo + log "Last 30 scope log lines:" + tail -30 "$SCOPE_LOG" | tee -a "$DRIVER_LOG" + exit 2 + fi + STATUS=$(curl -s "$SCOPE_URL/api/v1/cloud/status") + CONNECTED=$(echo "$STATUS" | json_get ".connected") + ERROR=$(echo "$STATUS" | json_get ".error") + STAGE=$(echo "$STATUS" | json_get ".connect_stage") + + if [[ "$CONNECTED" == "true" ]]; then + log "CONNECTED (${ELAPSED}s)" + echo "$STATUS" | tee -a "$DRIVER_LOG" + echo + break + fi + if [[ -n "$ERROR" && "$ERROR" != "None" ]]; then + log "CLOUD ERROR (${ELAPSED}s): $ERROR" + echo "$STATUS" | tee -a "$DRIVER_LOG" + echo + log "Last 30 scope log lines:" + tail -30 "$SCOPE_LOG" | tee -a "$DRIVER_LOG" + exit 1 + fi + if [[ "$STAGE" != "$LAST_STAGE" ]]; then + log " stage: $STAGE (${ELAPSED}s)" + LAST_STAGE="$STAGE" + fi + sleep 3 +done + +if [[ $FULL_SESSION -eq 0 ]]; then + exit 0 +fi + +# --- 7. Full session: pipeline + session + frames + cleanup -------- + +# 7a. Ensure test video exists +if [[ ! -f "$TEST_VIDEO" ]]; then + log "Creating $TEST_VIDEO (512x512 red frames @30fps, 10s)..." + uv run --with opencv-python --with numpy python -c " +import cv2, numpy as np +w = cv2.VideoWriter('$TEST_VIDEO', cv2.VideoWriter_fourcc(*'mp4v'), 30, (512, 512)) +frame = np.zeros((512, 512, 3), dtype=np.uint8) +frame[:] = (0, 0, 255) +for _ in range(300): + w.write(frame) +w.release() +" 2>&1 | tee -a "$DRIVER_LOG" + [[ -f "$TEST_VIDEO" ]] || fail "Failed to create $TEST_VIDEO" 4 +fi +log "Test video: $TEST_VIDEO" + +# 7b. Load pipeline +log "POST /api/v1/pipeline/load (pipeline_id=$PIPELINE_ID)" +LOAD_BODY=$(python3 -c "import json; print(json.dumps({'pipeline_ids': ['$PIPELINE_ID']}))") +LOAD_RESP=$(curl -sf -X POST "$SCOPE_URL/api/v1/pipeline/load" \ + -H 'Content-Type: application/json' -d "$LOAD_BODY") \ + || fail "pipeline/load request failed" 4 +log "Load response: $LOAD_RESP" + +# 7c. Poll pipeline status — require both status=loaded AND pipeline_id +# matches what we loaded (cloud-mode status can show a stale "loaded" +# from a previous session for a brief window after POST). +log "Polling /api/v1/pipeline/status (timeout ${TIMEOUT_PIPELINE}s)..." +# Give the async load a moment to propagate before first check. +sleep 5 +START=$(date +%s) +LAST_KEY="" +while true; do + ELAPSED=$(($(date +%s) - START)) + if [[ $ELAPSED -gt $TIMEOUT_PIPELINE ]]; then + log "Pipeline load TIMEOUT after ${ELAPSED}s. Last status:" + curl -s "$SCOPE_URL/api/v1/pipeline/status" | tee -a "$DRIVER_LOG" + echo + exit 2 + fi + PSTATUS=$(curl -s "$SCOPE_URL/api/v1/pipeline/status") + PS=$(echo "$PSTATUS" | json_get ".status") + PID=$(echo "$PSTATUS" | json_get ".pipeline_id") + STAGE=$(echo "$PSTATUS" | json_get ".loading_stage") + if [[ "$PS" == "loaded" && "$PID" == "$PIPELINE_ID" ]]; then + log "Pipeline loaded (${ELAPSED}s, id=$PID)" + break + fi + if [[ "$PS" == "error" ]]; then + log "Pipeline load ERROR after ${ELAPSED}s" + echo "$PSTATUS" | tee -a "$DRIVER_LOG" + echo + exit 4 + fi + KEY="${PS}|${PID}|${STAGE}" + if [[ "$KEY" != "$LAST_KEY" ]]; then + log " pipeline status=$PS pipeline_id=$PID stage=$STAGE (${ELAPSED}s)" + LAST_KEY="$KEY" + fi + sleep 3 +done + +# 7d. Start session with video-file input +log "POST /api/v1/session/start (pipeline=$PIPELINE_ID, source=$TEST_VIDEO)" +SESSION_BODY=$(python3 -c " +import json, os +body = { + 'pipeline_id': '$PIPELINE_ID', + 'input_mode': 'video', + 'input_source': { + 'enabled': True, + 'source_type': 'video_file', + 'source_name': os.environ.get('TEST_VIDEO', '$TEST_VIDEO'), + }, +} +print(json.dumps(body)) +") +SESSION_RESP=$(curl -s -o /tmp/session_start.json -w '%{http_code}' \ + -X POST "$SCOPE_URL/api/v1/session/start" \ + -H 'Content-Type: application/json' -d "$SESSION_BODY") || true +if [[ "$SESSION_RESP" != "200" ]]; then + log "session/start failed (http $SESSION_RESP)" + cat /tmp/session_start.json | tee -a "$DRIVER_LOG" + echo + exit 4 +fi +log "Session started" + +# 7e. Wait for frames +log "Waiting for frames to flow (timeout ${TIMEOUT_FRAMES}s)..." +START=$(date +%s) +FRAMES_IN=0 +FRAMES_OUT=0 +while true; do + ELAPSED=$(($(date +%s) - START)) + if [[ $ELAPSED -gt $TIMEOUT_FRAMES ]]; then + log "Frame-wait TIMEOUT (frames_in=$FRAMES_IN frames_out=$FRAMES_OUT)" + curl -s "$SCOPE_URL/api/v1/session/metrics" | tee -a "$DRIVER_LOG" + echo + exit 2 + fi + METRICS=$(curl -s "$SCOPE_URL/api/v1/session/metrics") + FRAMES_IN=$(echo "$METRICS" | json_get ".frames_in") + FRAMES_OUT=$(echo "$METRICS" | json_get ".frames_out") + FRAMES_IN=${FRAMES_IN:-0} + FRAMES_OUT=${FRAMES_OUT:-0} + if [[ "$FRAMES_OUT" != "0" && "$FRAMES_OUT" != "" ]]; then + log "Frames flowing: in=$FRAMES_IN out=$FRAMES_OUT (${ELAPSED}s)" + break + fi + sleep 2 +done + +# 7f. Let it run a bit so stream_heartbeat events fire +log "Streaming for 10s to let heartbeat events fire..." +sleep 10 +METRICS=$(curl -s "$SCOPE_URL/api/v1/session/metrics") +log "Final metrics: $METRICS" + +# 7g. Stop session +log "POST /api/v1/session/stop" +curl -sf -X POST "$SCOPE_URL/api/v1/session/stop" > /dev/null \ + || log "session/stop returned non-2xx (continuing)" + +# 7h. Cloud disconnect (explicit, to cleanly fire websocket_disconnected) +log "POST /api/v1/cloud/disconnect" +curl -sf -X POST "$SCOPE_URL/api/v1/cloud/disconnect" > /dev/null \ + || log "cloud/disconnect returned non-2xx (continuing)" + +log "Full-session test OK" +exit 0 From 5f45236b967e31785f278bd3421f7eaed1f72b27 Mon Sep 17 00:00:00 2001 From: emranemran Date: Tue, 19 May 2026 09:38:30 -0700 Subject: [PATCH 2/8] fix --- e2e/tests/cloud-streaming.spec.ts | 260 ++++++++++++++++++++++++++++++ 1 file changed, 260 insertions(+) create mode 100644 e2e/tests/cloud-streaming.spec.ts diff --git a/e2e/tests/cloud-streaming.spec.ts b/e2e/tests/cloud-streaming.spec.ts new file mode 100644 index 000000000..950b5359b --- /dev/null +++ b/e2e/tests/cloud-streaming.spec.ts @@ -0,0 +1,260 @@ +import { test, expect, Page } from "@playwright/test"; + +/** + * E2E tests for Scope cloud streaming via fal.ai. + * + * The app is started with: + * VITE_DAYDREAM_API_KEY=... → baked into the frontend, makes the app + * behave as signed-in so the cloud toggle + * is enabled + * SCOPE_CLOUD_APP_ID=daydream//ws → points scope at a fal deploy + * + * Flow: + * 1. App loads (already logged in via baked-in API key) + * 2. Switch to Perform mode (default is Workflow/graph mode after the + * graph-mode redesign) + * 3. Toggle Remote Inference on from the settings dialog + * 4. Wait for cloud connection (Connection ID rendered) + * 5. Select the passthrough pipeline + * 6. Click the play overlay to start the stream + * 7. Verify the output