See, Symbolize, Act: Grounding VLMs with Spatial Representations for Better Gameplay

We test whether giving vision-language models explicit object coordinates improves their gameplay across Atari, VizDoom, and AI2-THOR. We find that symbolic grounding helps — but only when the model can accurately extract those symbols itself. Claude-4-Sonnet benefits consistently. GPT-4o and Gemini degrade.

Preprint: https://arxiv.org/abs/2603.11601

---

Repository Structure

• advanced_zero_shot_pipeline.py — Frame + Self-Extracted Symbols pipeline
• direct_frame_runner.py — Frame-only baseline pipeline
• ocatari_symbolic_runner.py — Frame + Ground-Truth Symbols pipeline
• symbol_only_pipeline.py — Symbols-only pipeline (no visual frame)
• ocatari_ground_truth.py — Ground truth extraction from Atari RAM via OCAtari
• vizdoom_symbolic_runner.py — All four pipelines for VizDoom
• vizdoom_ground_truth.py — VizDoom ground truth extraction
• ai2thor_symbolic_runner.py — All four pipelines for AI2-THOR
• ai2thor_ground_truth.py — AI2-THOR ground truth extraction
• aws_model.py — Unified model client (AWS Bedrock, OpenRouter, OpenAI)
• models_for_symbol_only.py — Text-only model wrappers for the symbols-only pipeline
• run_benchmark.py — Runs the full evaluation benchmark
• run_ablation_study.py — Runs the noise and dropout ablation study
• noise_applier.py — Injects coordinate noise and object dropout into symbol data
• coordinate_accuracy_evaluator.py — Measures object detection quality against OCAtari ground truth
• gameplay_metrics.py — Game performance metrics
• automatic_benchmark/ — Benchmarking framework with rule-based, semantic, and LLM-judge scoring
• ablation_config.yaml — Ablation study configuration

---

Setup

pip install -r requirements.txt

Models are accessed via AWS Bedrock or OpenRouter. Set the relevant environment variable before running:

export OPENROUTER_API_KEY=...

How to Run

python direct_frame_runner.py \
  --env_name "ALE/Pong-v5" \
  --provider openrouter \
  --model_id anthropic/claude-sonnet-4 \
  --api_key YOUR_OPENROUTER_KEY \
  --game_type pong \
  --num_frames 600 \
  --seed 42 \
  --output_dir ./experiments/

2. Gameplay Evaluation (Vision+Symbol)

python advance_game_runner.py \
  --env_name "ALE/Pong-v5" \
  --provider openrouter \
  --model_id anthropic/claude-sonnet-4 \
  --openrouter_key_file OPENROUTER_API_KEY.txt \
  --detection_model anthropic/claude-sonnet-4 \
  --game_type pong \
  --num_frames 600 \
  --seed 42 \
  --output_dir ./experiments/