Predicting Radiologist Expertise from 3D Gaze Patterns During CT Interpretation

This is the official implementation of the paper "Predicting Radiologist Expertise from 3D Gaze Patterns During CT Interpretation", early accepted at MICCAI'26.

Abstract

Accurate interpretation of volumetric CT requires efficient navigation of 3D image volumes and attention to diagnostically relevant regions. While eye-tracking has been widely studied in 2D medical imaging, its use for expertise assessment in CT settings remains limited. We propose a gaze-informed transformer framework for expertise classification in thoracic CT. Using a DINOv2 ViT-B/14 backbone, radiologist fixation patterns are integrated into volumetric feature learning through (1) a learnable log-space bias in self-attention and (2) gaze-weighted pooling of patch embeddings. We trained and evaluated our approach on 182 CT reading sessions from five radiologists with varying levels of experience. On a held-out test set, the model achieves a ROC-AUC of 0.91 and F1 score of 0.86, outperforming adapted methods. These findings suggest that incorporating visual search behavior into transformers may support objective, process-based expertise assessment in radiology.

Model

The proposed model is implemented on a DINOv2 ViT-B/14 backbone that injects radiologist fixation patterns into volumetric feature learning through a learnable log-space bias in self-attention and gaze-weighted pooling of patch embeddings:

Pretrained Checkpoints

You can download the pretrained model weights here: Download Checkpoint.

Getting Started

Data Preprocessing

Generate the gaze heatmaps consumed during training from the raw fixation data:

python utils/gen_heatmaps.py

This produces the per-session gaze weights aligned to the CT volumes used by the classifier.

Train DINOv2 Gaze-Weighted Classifier

Examples:

• Default settings (no gaze attention, gaze-weighted pooling, backbone unfrozen):

  python models/train.py
  

• Use gaze-bias + gaze-weighted pooling (full gaze model):

  python models/train.py --attn-mode gaze_bias --pooling gaze_weighted
  

• Freeze backbone, use CLS pooling, train for 50 epochs:

  python models/train.py --freeze-backbone --pooling cls --epochs 50
  

• Fewer folds, fewer slices per step (faster, for debugging):

  python models/train.py --folds 3 --slices-per-step 4 --epochs 10
  

• All options at a glance:

  python models/train.py --help
  

Software and Dependencies

Package	Version
Python	3.9.13
PyTorch	2.7.0+cu126
CUDA (build)	12.6
PyTorch Lightning	2.6.0
timm	1.0.24
NumPy	2.0.2
scikit-learn	1.6.1
nibabel	5.3.2
Pillow	11.2.1

Evaluation on Held-Out Test Set

We evaluate our proposed model against adapted models. Performance is reported using ROC-AUC, F1-score, sensitivity, and specificity.

• SGP: Spatial Gaze Map
• GTI: Gaze Trajectory Image
• FO-CT+Gaze: Fixation-ordered CT slices with gaze overlay

Results

Model	Input	Prediction	ROC-AUC	F1	Sens.	Spec.
TF-CNN (Sharma et al., 2021)	SGP, GTI	Skill	0.7793	0.8372	0.9231	0.6071
TF-CNN (Sharma et al., 2021)	FO-CT+Gaze, SGP, GTI	Skill	0.7454	0.7568	0.7179	0.7500
IF-CNN (Sharma et al., 2021)	SGP, GTI	Skill	0.7308	0.6866	0.5897	0.8214
IF-CNN (Sharma et al., 2021)	FO-CT+Gaze, SGP, GTI	Skill	0.7363	0.7532	0.7436	0.6786
LF-CNN (Sharma et al., 2021)	SGP, GTI	Skill	0.7463	0.7838	0.7436	0.7857
LF-CNN (Sharma et al., 2021)	FO-CT+Gaze, SGP, GTI	Skill	0.7637	0.8205	0.8205	0.7500
HF-CNN (Sharma et al., 2021)	SGP, GTI	Skill	0.7518	0.7838	0.7436	0.7857
HF-CNN (Sharma et al., 2021)	FO-CT+Gaze, SGP, GTI	Skill	0.7363	0.7733	0.7436	0.7500
Lou et al. (2023)	FO-CT session-level	Saliency map	0.6782	0.7123	0.6667	0.7241
Lou et al. (2023)	FO-CT chunk-level	Saliency map	0.5570	0.6000	0.5385	0.6552
CT-Searcher (Pham et al., 2025)	CT volume	Scanpath	0.8750	0.8421	0.8205	0.8214
Ours	CT volume	Skill	0.9089	0.8611	0.7949	0.9310

Ablation Study

Attention variants: Gaze-Bias - soft additive bias injected into self-attention layers; Fix-Mask - hard binary mask on attention, meaning patches with zero gaze weight are blocked from being attended to; None - standard DINOv2 self-attention, no gaze signal enters the transformer. Pooling strategies: Gaze-Weighted - weighted average of all patch tokens using normalized gaze weights, i.e., patches the radiologist looked at more contribute more to the session embedding; CLS - uses only CLS token.

Attention	Pooling	Mean Val AUC	AUC	F1	Sens.	Spec.
Gaze-Bias	Gaze-Weighted	0.796 ± 0.091	0.909	0.861	0.795	0.931
Gaze-Bias	CLS	0.828 ± 0.101	0.897	0.824	0.751	0.892
None	Gaze-Weighted	0.632 ± 0.090	0.856	0.759	0.623	0.969
Fix-Mask	Gaze-Weighted	0.871 ± 0.101	0.842	0.783	0.688	0.892
Fix-Mask	CLS	0.823 ± 0.105	0.799	0.850	0.872	0.759