-
A Real-Time Video Search and Analysis system powered by VAST DataEngine, using Nvidia NIM's & Cosmos-Reason Models.
-
This Blueprint is were released to bridge the gap between NVIDIA’s VSS reference architectures, and showcases the e2e utilization of the full VAST AI OS production-grade capabilities, including Agentic & Serverless Event-based Compute Framework and VastDB Vector-Store.
The system has three main parts:
- K8s Application - Web UI and REST API (Kubernetes)
- Ingest Pipeline - Serverless video processing (VAST DataEngine)
- Enrichment Pipeline - Scheduled prompt-suggester (search chips + key events)
— see also the interactive diagram in the app (Show Blueprint Diagram) or blueprint.html
| Component | Guide |
|---|---|
| VSS - Retrieval Web UI; K8s Application (Backend, Frontend, Streaming, Batch Sync) | vss-k8s-application |
| VSS - DataEngine; Ingest Pipeline (Segmenter, Detector, Reasoner, Embedder, Writer) | dataengine-vss-ingest-pipeline |
| VSS - DataEngine; Enrichment Pipeline (prompt-suggester, optional) | dataengine-vss-ingest-pipeline |
| GPU models (Cosmos-Reason2, Cosmos-Embed1, YOLO11 on Docker) | vss-blueprint-models |
-
GPU models (Reason2, Embed1, YOLO on a GPU host): vss-blueprint-models
-
Build and push images (
REGISTRYis required;TAGdefaults tov1):# K8s app (backend, frontend, streaming, batch-sync) REGISTRY=your.registry/vss source-code/scripts/build-retrieval-images.sh # DataEngine functions (ingest + prompt-suggester) REGISTRY=your.registry/vss source-code/scripts/build-vastde-functions.sh
Details: K8s Step 2 · DataEngine build · scripts
-
Deploy K8s Application:
cd deployments/vss-k8s-application vim backend-secret.yaml # Configure credentials ./QUICK_DEPLOY.sh <namespace> <cluster_name>
-
Deploy Ingest Pipeline (choose one):
- Using GUI: Configure
vss-gui-secret-file-template.yaml, then use DataEngine UI - Using CLI: Configure
vss-cli-secret-file-template.yaml, then run vastde commands
See Ingest Pipeline Guide for full instructions (optional enrichment is the last step of the same GUI or CLI path).
- Using GUI: Configure
-
Test: Upload a video and search at
http://video-lab.<cluster_name>.vastdata.com
| Feature | Description | Documentation |
|---|---|---|
| Video Analysis Prompts | Configurable AI scenarios (surveillance, traffic, live_driving, etc.) | video-reasoner |
| Ingest metadata config | Single definition for upload metadata UI + S3 mapping (ingest_metadata.py) |
shared |
| Custom AI Prompts | Per-video custom prompts (max length in ingest_metadata.py) |
video-reasoner |
| Metadata Filters | Filter by camera_id, location, capture_type | ingest |
| Advanced Search & AI Settings | Max clip cards, synthesis clip count, caption/video weight, similarity | video-backend |
| Explore mode | Browse indexed uploads by day and location — no query; summarize any video on demand | video-frontend |
| Data Dashboard | VastDB stats, ingest health, S3 pipeline inventory, live key events | video-frontend |
| Search suggestions & key events | Grounded ≤8-word rephrases of segment reasoning_content → vss-prompts-events |
prompt-suggester |
| Object detection counts | YOLO peak concurrent per class (object_counts); UI chips + dashboard heatmap |
video-detector |
| Agent APIs | Tool wrappers + grounded Q&A for external agents | video-backend |
| Time Filtering | Filter by upload time (presets or custom range) | video-backend |
| Video Streaming | Capture YouTube videos to S3 | video-streaming |
| Batch Sync | Copy MP4 files between S3 buckets | video-batch-sync |
| Authentication | VAST username + password | video-backend |
| Component | Description |
|---|---|
| shared | Cross-service modules (ingest_metadata.py — upload metadata UI + S3) |
| scripts | Build scripts for retrieval images and DataEngine functions |
| video-backend | REST API, authentication, search |
| video-frontend | Angular web UI (Search, Explore, Dashboard) |
| prompt-suggester | Grounded search chips + key events → VastDB |
| video-streaming | YouTube capture service |
| video-batch-sync | S3 batch copy service |
| video-segmenter | Splits videos into segments |
| video-detector | YOLO11 object detection + peak counts + bbox sidecars |
| video-reasoner | Plain searchable reasoning_content (Cosmos-Reason2) |
| video-embedder | Text (reasoning_content) + visual embeddings |
| vastdb-writer | Stores vectors and segment rows in VastDB |
Upload Video → vss-chunks bucket
↓
video-segmenter (~5s clips, trim to capture_interval)
↓
video-detector (YOLO11 → object_classes + peak object_counts + bbox sidecars)
↓
video-reasoner (Cosmos-Reason2 → searchable plain reasoning_content ≤1024)
↓
video-embedder (vectors from reasoning_content + vectors_visual from MP4)
↓
vastdb-writer (segment rows in VastDB)
↓
Search Ready
↓
prompt-suggester (optional enrichment)
↓
vss-prompts-events (grounded search chips + key events)
Search flow: query embed (Cosmos-Embed1) → hybrid search (reasoning_content + visual) → clip cards with upload-time badge and match timeline → Cosmos-Reason2 synthesis → player with bbox overlay and label N object chips
Explore flow: browse by upload date and location (fully indexed chunks only) → clip cards with segment timeline and upload-time badge → full-chunk player with bbox toggle and Summarize Video
Dashboard flow: VastDB KPIs + ingest quality + object instance heatmap (sum of peak counts) + S3 vs index alignment → key events (grounded rephrases) with in-place segment preview
Enrichment flow: sample recent reasoning_content → one grounded ≤8-word phrase per sample (no inventing) → search chips + key events
Agent flow: GET/POST /api/v1/tools/* for VastDB/search/explore/synthesize → POST /api/v1/agent/ask or /search-and-answer for grounded answers
Interactive diagram: open Show Blueprint Diagram in the app, or source-code/retrieval/video-frontend/src/assets/blueprint.html
- K8s Deployment: See K8s Application Guide
- Ingest / Enrichment Pipeline: See DataEngine Pipeline Guide
- Community: VAST Community Forums