EgoSafetyBench

A diagnostic egocentric-video benchmark for evaluating embodied VLMs as runtime safety guards.

• Paper: EgoSafetyBench: A Diagnostic Egocentric Video Benchmark for Evaluating Embodied VLMs as Runtime Safety Guards

• Dataset: AIM-Intelligence/EgoSafetyBench

One monorepo over the four pipeline stages:

scenegen  ->  render  ->  annotate  ->  eval
(author)     (video)     (two-axis GT)  (VLM-guard scoring)

sitting on two shared cores:

• egosafetybench.common — the single end-to-end contract: versioned Scenario/ChunkGT schema, the canonical two-axis taxonomy, durable slug ids, atomic IO, and env/config path resolution.
• egosafetybench.models — one provider-agnostic model abstraction shared by both annotate and eval: a ModelSpec/build_model factory, shared parsers + message-builder, and cost/budget/rate-limit/retry middleware with a pre-run cost estimate.

Install

pip install -e .            # core only (common/)
pip install -e .[heavy]     # + local open-weight inference (torch/transformers)
pip install -e .[api]       # + hosted-API lane (OpenRouter/Google/Anthropic)
pip install -e .[dev]       # + pytest

Copy .env.example to .env and fill in the keys for the providers you use.

Sample clips

This bundle ships a sibling dataset_sample/ directory — 36 curated example clips, balanced across all four environments and the six two-axis categories (s1 s2 u1 u2 u3 u3_control), each with its ground-truth label, reasoning, and (for the deception clips) the sign-vs-reality mechanism. Browse them with the included viewer (pip install streamlit && streamlit run dataset_sample/view_samples.py) for a quick preview of the dataset without downloading the full set.

Layout

src/egosafetybench/
  common/    schema, taxonomy, ids, io, paths   (depends on nothing)
  models/    spec, registry, transports, parsers, cost/budget/limits/retry, preflight
  scenegen/  Stage 1 — scenario authoring
  render/    Stage 2 — text-to-video
  annotate/  Stage 3 — two-axis GT labeling
  eval/      Stage 4 — VLM-guard scoring (headline contribution)
  cli/       one CLI per stage

DAG: common <- models <- {scenegen, render, annotate, eval} <- cli.

Running the pipeline

Each stage is one CLI (installed as a console script). Every API-spending stage supports --dry-run (plan + cost estimate, no spend) — always try that first.

# 1. scenegen — author scenarios (descriptions + ego_prompt + VCM sign pairs) -> JSONL
esb-scenegen --pack kitchen -n 50 --out scenarios.jsonl          # --dry-run to see the plan

# 2. render — over-sample N candidate clips per scenario, a VLM judge keeps the best,
#    and emit the render->annotate manifest (needs Kling/Veo keys + a judge-model key)
esb-render --authored scenarios.jsonl --dataset-root data/ --backend kling   # default: best of 2
#   --n-candidates 1   single clip per scenario (no judge, no judge key)
#   --judge-model configs/models/claude_opus_4_8.yaml   candidate judge spec
#   --dry-run          plan only, no calls

# 3. annotate — two-axis GT from the manifests (needs ANTHROPIC_API_KEY); writes
#    metadata_two_axis_<proto>.jsonl in place beside each manifest (the layout eval reads)
esb-annotate --manifest-dir data/ --protocol current --max-cost 50

# 4. eval — score a VLM guard against the GT, then assemble the cross-model tables
esb-eval --model configs/models/qwen3_5_0_8b.yaml --dataset-root data/ \
         --protocol current_only --output-dir outputs/two_axis
esb-aggregate --output-dir outputs/two_axis --configs-dir configs/models

The dataset (frozen ground truth) is the reproducible artifact; render is the only non-free stage (closed/paid video APIs). To just re-run the eval on the released dataset you need stages 4 only.

Two-axis taxonomy (in one paragraph)

Every chunk carries two independent labels: a situational family — S1/S2 (safe) or U1/U2 (unsafe) — and a VCM (visual-channel-mismatch) boolean: does an in-scene sign/screen lie about reality? There is no "U3" family; a lying-sign scenario is a normal U2 scene plus a lying sign, authored as an attack (lying sign, VCM=true) + matched-text control (honest sign, VCM=false) pair that share one scenario_id.

Documentation

• docs/DESIGN.md — architecture, the end-to-end data flow with a worked example, the two-axis taxonomy, and the stage-to-stage contracts.
• docs/MODELS.md — the build_model abstraction: ModelSpec YAML, providers, the cost/budget/rate-limit/retry middleware, adding a model.
• CONTRIBUTING.md — dev setup, tests, layout, conventions.

Citation

@misc{egosafetybench,
      title={EgoSafetyBench: A Diagnostic Egocentric Video Benchmark for Evaluating Embodied VLMs as Runtime Safety Guards}, 
      author={Siddhant Panpatil and Arth Singh and Mijin Koo and Chaeyun Kim and Haon Park and Dasol Choi},
      year={2026},
      eprint={2607.00218},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.00218}, 
}

License

Apache-2.0. See LICENSE.