See2Think

Do Multimodal Models Really Use Intermediate Visual States?

Siyu Yan^1,3,†   Zhuoran Yan^2,†   Haiying Xu^3,4,†   Panhao Zhou²   Jingyu Chen²   Chenhao Ji³   Shuo Cao^3,5
Yongheng Zhang²   Haoze Liu³   Siyu Zhang^3,6   Xiwen Gu⁷   Yihao Liu³   Alex Jinpeng Wang^2,§

The Question · Benchmark · VAoT · Results · Quick Start · Citation

---

The Question

A model can produce a useful-looking intermediate image—but does its later reasoning actually depend on that visual state?

Multimodal models can draw auxiliary lines, crop regions, highlight objects, and request rendered intermediate images. Final-answer accuracy alone cannot reveal whether the model chose a relevant visual action, whether the renderer executed it faithfully, or whether the returned visual state affected subsequent reasoning.

See2Think turns this hidden process into something measurable.

_{Figure 1. From final-answer evaluation to controlled diagnosis of visual-state use.}

01 · Benchmark 1,200 open-ended, visually dependent problems.

02 · Three visual worlds 2D structures, 3D scenes, and real-world reasoning.

03 · Process diagnosis Measure action, rendering, and feedback separately.

04 · Intervention Corrupt feedback to reveal behavioral dependence.

See2ThinkBench

See2ThinkBench contains 1,200 samples across 12 task categories. Every problem is open-ended and visually dependent, spanning three complementary reasoning worlds:

• 2D structured reasoning — diagrams, charts, geometry, and symbolic visual structure.
• 3D scene reasoning — spatial relations, embodied scenes, and manipulation.
• Real-world visual reasoning — natural images and grounded multimodal questions.

_{Figure 2. Representative See2ThinkBench examples across 12 task categories and three visual worlds.}

Visual Action-of-Thought

Visual Action-of-Thought (VAoT) records the complete reasoning trajectory: textual thought, visual action, rendered state, follow-up reasoning, and final answer.

Four matched inference settings

Setting	Visual action	Rendered feedback	Diagnostic role
CoT	—	—	Text-only reasoning baseline
VAoT-NoRender	✓	—	Is proposing an action alone useful?
VAoT-Full	✓	✓	Does genuine visual feedback help?
VAoT-WrongRender	✓	Corrupted	Does later reasoning depend on returned visual evidence?

Three process-level measurements

Measurement	Question
Action Relevance	Does the selected visual operation target task-relevant evidence?
Render Faithfulness	Does the renderer faithfully execute the requested operation?
Feedback Uptake	Does the model actually use the rendered state in later reasoning?

Results

See2Think reports final-answer performance and process behavior together. This separates looking correct from using intermediate visual states correctly.

Final-answer accuracy across models and settings.

Process-level diagnosis of action relevance, render faithfulness, and feedback uptake.

The main analyses include:

• matched comparisons among CoT, NoRender, Full, and WrongRender;
• paired render-benefit and corrupted-feedback sensitivity;
• process judging for relevance, faithfulness, and uptake;
• human audits of process judges and WrongRender quality.

Quick Start

1. Install

git clone https://github.com/CSU-JPG/See2Think.git
cd See2Think

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Windows PowerShell

python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt

2. Configure endpoints

cp config.example.sh config.sh
source config.sh

Fill in the model endpoints and local paths in config.sh. This file is ignored by git and should never contain committed credentials.

Configuration variables

export SEE2THINK_LLM_BACKEND="openai"      # openai | vllm
export OPENAI_API_KEY="..."
export OPENAI_BASE_URL="..."

export GEMINI_API_KEY="..."
export GEMINI_BASE_URL="..."

export SEE2THINK_DATA_BASE="/path/to/See2Think"
export SEE2THINK_OUTPUT_BASE="/path/to/outputs"
export SEE2THINK_LOG_DIR="/path/to/logs"

3. Prepare tasks

The public repository does not include the paper's full task manifests, benchmark images, or generated model outputs. Provide your own manifest with --tasks; see examples/tasks.example.json for the minimal format.

4. Run an inference setting

python -u solve/run_tasks.py \
  --tasks /path/to/tasks.json \
  --mode banana \
  --model gpt-5.5 \
  --setting vaot_full \
  --workers 4 \
  --prompt_dir prompt

Supported settings: text_cot, vaot_no_render, vaot_full, and vaot_wrong_render.

Run answer and process evaluation

Build answer-judge inputs:

python eval/build_answer_input.py \
  --tasks /path/to/tasks.json \
  --data-base /path/to/See2Think \
  --manifest /path/to/final_results/_manifest.csv \
  --output-jsonl eval/results/answer_inputs/input.jsonl \
  --model gpt-5.5 \
  --setting vaot_full

Run process judging for VAoT-Full:

python -u eval/process_judge.py \
  --tasks /path/to/tasks.json \
  --results-root /path/to/vaot_full_outputs \
  --model gpt-5.5 \
  --setting vaot_full \
  --judge-model gpt-5.4 \
  --run-name gpt55_vaot_full_process_judge \
  --workers 1

Repository Map

Path	Purpose
solve/	Core inference pipeline and VAoT execution
convert/	Parsing and answer-evaluation helpers
eval/	Answer judging and process-level evaluation
prompt/	Prompts for inference, rendering, and intervention
examples/	Minimal public task-manifest examples

Large benchmark data, generated trajectories, rendered images, logs, audit packets, and paper output bundles are intentionally excluded from git.

Citation

If See2Think helps your research, please cite:

@article{yan2026see2think,
  title   = {See2Think: Do Multimodal Models Really Use Intermediate Visual States?},
  author  = {Yan, Siyu and Yan, Zhuoran and Xu, Haiying and Zhou, Panhao and Chen, Jingyu and Ji, Chenhao and Cao, Shuo and Zhang, Yongheng and Liu, Haoze and Zhang, Siyu and Gu, Xiwen and Liu, Yihao and Wang, Alex Jinpeng},
  journal = {arXiv preprint},
  year    = {2026}
}

Acknowledgments

See2Think builds on public benchmark resources across diagrammatic reasoning, 3D scene reasoning, embodied manipulation, and real-world visual reasoning. Please also cite the original benchmark sources when using released See2Think task manifests.

---

See the answer. Inspect the process. Test the dependence.