Shuang Liang1,3†, Zeqing Wang2†, Yuxian Li1†, Xihui Liu1, Han Wang1,3*
1The University of Hong Kong, 2Sun Yat-sen University, 3CASIC, HKU
†Equal contribution. *Corresponding author.
- [2026-May-14] 📄 Paper released on arXiv.
- [2026-May-09] 🤖 CAFE-SAM3 agent code released under
agent/. - [2026-May-09] 🧪 CAFEval2026 evaluation toolkit released under
tools/(cgF1 + AFPR/UFPR/IL-FPR/ACSR/UCSR/CSR/SoftSwap). - [2026-May-09] 🤗 CAFEval2026 benchmark (2,146 paired samples) released on HuggingFace.
- [2026-May-09] 🌐 Project page with interactive leaderboard live.
- CAFE-SAM3 agent release
- CAFEval2026 evaluation toolkit release
- CAFEval2026 benchmark release
- Project page with interactive leaderboard
CAFE asks a sharper question than is the mask accurate?: does the model actually ground the queried concept, or is it riding on visually salient but semantically misleading cues?
- ✅ Attribute-level counterfactual evaluation: target region and ground-truth mask are fixed; only attributes (appearance, context, material) change.
- ✅ Paired positive vs. misleading-negative prompts expose the gap between mask quality and concept discrimination.
- ✅ Three failure regimes: Superficial Mimicry, Context Conflict, Ontological Conflict — 2,146 paired samples in total.
- ✅ CAFE-SAM3 agent: an MLLM-driven loop that corrects SAM3 failures via concept verification, zoom-in inspection, and re-prompting.
| Split | # samples | What changes | Stress test |
|---|---|---|---|
| Superficial Mimicry (SM) | 1,111 | Surface texture / appearance | Pattern vs. object identity |
| Context Conflict (CC) | 593 | Surrounding environment | Scene prior vs. true category |
| Ontological Conflict (OC) | 442 | Material / substance | Shape shortcut vs. concept |
The mask stays put; the attribute moves. A faithful model should flip its prediction between the positive and the misleading-negative prompt, not ride the leftover visual cue.
cgF₁ is the headline metric, combining concept discrimination (IL_MCC) with mask quality (pmF₁).
| # | Model | Type | SM | CC | OC | Overall |
|---|---|---|---|---|---|---|
| 1 | CAFE-SAM3 (GPT-5.5) | Agentic | 69.7 | 66.1 | 44.7 | 63.3 |
| 2 | SAM 3 | End-to-end | 53.0 | 61.4 | -10.5 | 38.5 |
| 3 | OWLv2 + SAM 1 | Multi-model | 43.2 | 41.0 | -8.0 | 27.9 |
| 4 | YOLO-World | End-to-end | 39.4 | 20.8 | -5.9 | 21.1 |
| 5 | OpenSeeD | End-to-end | 28.9 | 29.8 | -4.0 | 15.1 |
| 6 | Grounded SAM 2 | Multi-model | 13.0 | 5.9 | 3.6 | 9.9 |
💡 Strong mask predictors are not automatically strong concept grounders. SAM 3 collapses on OC (negative cgF1); the CAFE-SAM3 agent recovers most of the loss through reasoning. See the interactive leaderboard for
IL_MCC,pmF₁,FPR,AFPR,ACSR.
git clone https://github.com/T-S-Liang/CAFE.git
cd CAFE
conda create -n cafe python=3.10 -y
conda activate cafe
pip install numpy scipy tqdm pycocotools requeststools/cgf1_eval_wrapper.py imports SAM 3's official CGF1Evaluator, and the agent uses SAM 3 as the segmentation backbone. Clone SAM 3 alongside CAFE and point --sam3-dir (or the SAM3_DIR env var) at its repository root:
git clone https://github.com/facebookresearch/sam3.git
export SAM3_DIR=$PWD/sam3huggingface-cli download teemosliang/CAFE --repo-type dataset --local-dir CAFEval2026Expected layout used by tools/run_eval.sh defaults:
CAFEval2026/
├── CAFEval2026_annotations.json # full set (SM + CC + OC)
├── CAFEval2026_SM.json # Superficial Mimicry subset
├── CAFEval2026_CC.json # Context Conflict subset
├── CAFEval2026_OC.json # Ontological Conflict subset
└── CAFEval2026_imgs/ # image directory
CAFEval2026 evaluates any COCO-format segmentation prediction dump ([{image_id, category_id, segmentation: RLE, score, ...}, ...]). One command produces both the SAM 3 cgF₁ family (per-subset) and the CAFE custom metrics (AFPR, UFPR, IL-FPR, ACSR, UCSR, CSR, SoftSwap).
bash tools/run_eval.sh \
--pred path/to/coco_predictions_segm.json \
--out-dir results/<your_run_tag> \
--gt-ann CAFEval2026/CAFEval2026_annotations.json \
--gt-sm CAFEval2026/CAFEval2026_SM.json \
--gt-cc CAFEval2026/CAFEval2026_CC.json \
--gt-oc CAFEval2026/CAFEval2026_OC.json \
--img-dir CAFEval2026/CAFEval2026_imgs \
--sam3-dir $SAM3_DIROutputs written to --out-dir:
| File | Contents |
|---|---|
<TAG>__cgf1_t<T>_{Overall,SM,CC,OC}.log |
SAM 3 cgF1 stdout per subset |
<TAG>__cafe_t<T>_tau<TAU>.json |
CAFE custom metrics: per-subset summary + per-pair audit |
<TAG>__SUMMARY_t<T>_tau<TAU>.md |
Consolidated markdown report (cgF1 + CAFE) |
Tunable knobs:
| Flag | Default | Meaning |
|---|---|---|
--score-thr |
0.5 (SAM 3 paper) |
Presence-confidence threshold; preds below this are dropped |
--iou-thr |
0.3 (CAFE paper) |
Target-alignment IoU threshold (CAFE custom metrics only; cgF1 sweeps IoU 0.50:0.95) |
--sam3-dir |
$SAM3_DIR |
SAM 3 repository root, must contain sam3/eval/cgf1_eval.py |
--python |
python3 |
Python interpreter |
Drop your model's predictions into COCO segm format and rerun the same script — the toolkit is model-agnostic:
[
{"image_id": 12345, "category_id": 1, "score": 0.87,
"segmentation": {"size": [H, W], "counts": "<rle>"}},
...
]If you already have cgF₁ numbers and want just AFPR / ACSR / SoftSwap / etc.:
python tools/eval_cafe_metrics.py \
--ann CAFEval2026/CAFEval2026_annotations.json \
--pred path/to/coco_predictions_segm.json \
--score-threshold 0.5 \
--iou-threshold 0.3 \
--out results/<tag>/cafe_metrics.jsonThe CAFE-SAM3 agent wraps SAM 3 with an MLLM-driven loop that performs concept verification, zoom-in inspection, and re-prompting on hard counterfactual cases. Lives in agent/.
export OPENAI_API_KEY=...
export CAFE_ANN=CAFEval2026/CAFEval2026_annotations.json
export CAFE_IMG_DIR=CAFEval2026/CAFEval2026_imgs
python agent/eval_agent.py \
--provider ttapi_openai \
--model gpt-5.5 \
--output-dir agent_eval_output \
--concurrency 4 \
--max-turns 10Other modes:
python agent/eval_agent.py --model gpt-5.5 --force # re-run all (ignore checkpoint)The runner is resumable: completed cases are written to agent_eval_output/checkpoint.json and skipped on restart. Predictions are produced in the same COCO segm format consumed by tools/run_eval.sh, so you can score the agent the same way as any baseline:
bash tools/run_eval.sh \
--pred agent_eval_output/coco_predictions_segm.json \
--out-dir results/cafe_sam3_gpt55A click-through of the agent's reasoning trace (including cases where SAM 3 alone fails) is on the project page:
If you find CAFE useful in your research, please cite:
@article{liang2026pixels,
title={From Pixels to Concepts: Do Segmentation Models Understand What They Segment?},
author={Liang, Shuang and Wang, Zeqing and Li, Yuxian and Liu, Xihui and Wang, Han},
journal={arXiv preprint arXiv:2605.09591},
year={2026}
}This project is released under the MIT License.
CAFE is built on top of, and grateful to, the following open-source efforts:
- SAM 3 and SAM 2: Promptable segmentation backbones, evaluated as baselines and used inside the CAFE-SAM3 agent.
- OWLv2, YOLO-World, OpenSeeD, Grounded SAM 2: Open-vocabulary baselines included in the leaderboard.
- LVIS and COCO: Source images and category vocabulary used for counterfactual construction.
For questions, suggestions, or collaboration inquiries:
- Shuang Liang: sliang57@connect.hku.hk
- Project page: https://t-s-liang.github.io/CAFE