URBANCLIPATLAS

A Visual Analytics Framework for Event and Scene Retrieval in Urban Videos

URBANCLIPATLAS turns long urban traffic recordings into a searchable index. A Dagster pipeline splits source videos into overlapping clips, describes each clip with a vision-language model, extracts agent trajectories, embeds the resulting event descriptions and indexes them in Qdrant. A FastAPI backend serves retrieval over that index, and a SvelteKit frontend provides the visual analytics interface.

Requirements

• Python 3.11–3.12 (.python-version pins 3.11; pyproject.toml allows >=3.11,<3.13)
• uv
• Node.js with pnpm 10.9
• Docker and Docker Compose
• ffmpeg and ffprobe on PATH
• An NVIDIA GPU with CUDA 13. Torch is pinned to the pytorch-cu130 index; for a CPU-only install, edit [tool.uv.sources] in pyproject.toml to point torch, torchvision and torchaudio at pytorch-cpu.

Setup

The sam3 submodule is an editable path dependency, so uv sync fails without it. Cloning with submodules is mandatory:

git clone --recurse-submodules https://github.com/visual-ds/urbanclipatlas.git
cd urbanclipatlas

If the repository is already cloned:

git submodule update --init --recursive

Then configure the environment and install:

cp .env.example .env   # fill in the keys, see Environment below
make py-install
make ui-install

Running

Command	What it starts	Port
make res	MinIO and Qdrant via Docker Compose	MinIO 9020 / 9021, Qdrant 6333 / 6334
make down	Stops the Compose services	—
make py-run	FastAPI backend	8000
make ui-dev	SvelteKit dev server	5173
make ws	WebSocket video server	8200
make dag	Dagster UI	3001
make dag-kill	Kills Dagster processes	—
make restart	Restarts the WebSocket server	—

Port 5173 is the only origin in the backend CORS allow-list, so run the frontend there. The frontend reaches the backend through the Vite /api proxy defined in web/vite.config.ts; no frontend environment variables are needed.

make ws downloads YOLO and sentence-transformers weights on first run and expects DEFAULT_VIDEO to name a file under infra/videos.

Environment

Copy .env.example to .env. The keys that matter:

Variable	Used by	Required
GOOGLE_CLOUD_API_KEY	Dagster VLM resource (Vertex AI, gemini-3.1-pro-preview)	For clip description
AI_STUDIO_API_KEY	Embeddings and answer generation (AI Studio)	For indexing and /ask
DAGSTER_HOME	Dagster instance directory, absolute path	For the pipeline
QDRANT_HOST, QDRANT_PORT, QDRANT_URL	Vector store connection	Defaults to localhost
QDRANT_CLIP_COLLECTION, QDRANT_EVENT_COLLECTION	Collection names	Have defaults
GEMINI_API_KEY	Query embedding for GET /api/v2/videos/query	Required by that endpoint, which returns 500 without it
OPENAI_KEY	Query enrichment and clip analysis	Optional, those steps are skipped without it
COHERE_API_KEY, JINA_API_KEY	Reranking	Optional, reranking is skipped without them
DEFAULT_VIDEO, MASK_DIR, COLLISION_EVENT_PHRASE, COLLISION_EVENT_TIMESTAMP	core/ws_server.py video streaming, masks and the collision-event jump	Optional

The notebooks under notebooks/segmentation/ read VIDEOS_DIR and DETECTION_DIR to locate inputs, falling back to the in-repo relative paths when unset.

GOOGLE_CLOUD_API_KEY and AI_STUDIO_API_KEY target different Google endpoints and are not interchangeable.

Repository layout

• core/ — FastAPI backend. main.py serves GET /api/v2/videos, GET /api/v2/videos/query, GET /api/v2/videos/reply and POST /api/v3/events/ask. ws_server.py streams YOLO-annotated video frames over a WebSocket at /ws/experiments.
• web/ — SvelteKit 1.x frontend (Svelte 4, Tailwind CSS v4, shadcn-svelte).
• workflows/ — Dagster assets, resources and utilities for the processing pipeline.
• infra/ — Docker Compose definition plus the data directories the pipeline reads and writes (videos, clips, descriptions, trajectories). Only the Compose file is tracked.
• notebooks/ — exploratory notebooks that prototyped the pipeline stages.
• third_party/sam3 — SAM 3 submodule.

Pipeline

video_clips
  └─ clip_descriptions
       └─ double_check_descriptions
            ├─ agent_trajectories
            └─ clip_embeddings
                 └─ clip_index

Asset	Group	Does
video_clips	video	Splits source videos into overlapping clips with ffmpeg (15 s clips, 3 s overlap)
clip_descriptions	description	Describes each clip with the Gemini VLM using a traffic-analysis prompt
double_check_descriptions	description	Second VLM pass that refines and corrects the descriptions
agent_trajectories	detection	Grounding DINO plus ByteTrack tracking for each described agent
clip_embeddings	embedding	Gemini embeddings for clip and event descriptions
clip_index	embedding	Upserts the embeddings into the Qdrant collections

Every stage is gated by a manifest file; without them the assets find no work to do.

• infra/videos/<scene>/videos.json gates clipping:

  { "videos": ["sensor_2/right.mp4"] }

• clips.json, written under infra/clips/, gates description:

  { "enabled": true }

• embeddings.json, written under infra/descriptions/, gates embedding and indexing.

Generated clips follow infra/clips/<scene>/<sensor>/<side>/<duration>_<overlap>/, for example infra/clips/chase_1/sensor_2/right/15_3/.

Start the Dagster UI with make dag and materialize the assets from there.

Status

This repository ships the source code. It does not include a sample dataset or a prepopulated index, so retrieval returns nothing until you run the Dagster pipeline over your own videos and populate Qdrant. The POST /api/v3/events/ask endpoint is served by the backend but is not yet wired to the frontend.

Known limitation: the pipeline indexes video_id as the source file stem, so scoping a search to a specific video matches on that stem alone. Two scenes whose files share a name (for example chase_1/sensor_2/right.mp4 and chase_3/sensor_2/right.mp4) are not distinguishable in a filtered query.

The Compose file uses the default MinIO credentials (minioadmin / minioadmin). Change them before exposing the service anywhere.