dental_study

Two-stage caries screening for intraoral photographs: a custom-trained detector locates suspected carious lesions, and an off-the-shelf medical vision-language model generates a plain-language description of each region.

The goal isn't a diagnostic replacement — it's a decision-support pipeline that (a) surfaces regions of interest and (b) hands a clinician a readable rationale for each one, so a dentist doesn't have to trust an opaque bounding box.

---

Result on the held-out test split

Trained on the Aga Khan University Zenodo dataset — 6,313 annotated intraoral photographs from adolescents/young adults in Mithi, Sindh, Pakistan (Ahmed et al., Scientific Data 2025).

Evaluated on a patient-safe occlusal-only test split — 332 images, no patient appearing in both train and test. This is a strictly harder evaluation than the paper's, which uses an image-level split (same patients in train, val, and test).

model	split type	mAP50	mAP50-95	Precision	Recall
Paper's YOLOv8s (published)	image-level (has patient leakage)	0.841	—	0.828	0.807
This repo's best.pt	patient-safe, occlusal-only	0.860	0.692	0.824	0.845

The detector matches or exceeds the paper's baseline on every metric and it does so on a split that eliminates patient leakage. mAP50-95 is reported for this work only; the paper does not publish it.

See "Full ML metrics" below for the per-class breakdown and combined-pipeline numbers.

---

Sample outputs

Two suspected occlusal lesions on posterior molars, each with an independent per-region reasoning:

Single subtle detection on a healthy-looking maxillary arch — model appropriately hedges its confidence:

Ground truth vs. model prediction (test-set image with 4 annotated lesions, all recovered by the model; 2 low-confidence extras marked separately for review):

---

How the pipeline works

                    ┌──────────────────┐
Intraoral photo ─▶  │  Detector (YOLO) │  ─▶  boxes + class + confidence
                    │  best.pt (77 MB) │
                    └──────────────────┘
                             │
                             ▼  (each detection is cropped with 15% padding, padded to a square)
                    ┌──────────────────┐
                    │  MedGemma 1.5    │  ─▶  short description + confidence hedge
                    │  4B via Ollama   │
                    └──────────────────┘
                             │
                             ▼
             ┌──────────────────────────────┐
             │  Client-facing report        │
             │   • annotated image          │
             │   • per-lesion reasoning     │
             │   • findings summary         │
             └──────────────────────────────┘

Stage 1 — the detector

A YOLOv11m fine-tune on the occlusal-only subset (Mandibular + Maxillary_Occlusal views, ~2,500 images). Frontal and lateral views were excluded during training because caries lesions aren't visible from those angles — the paper trains on all views, which we found injects noise into the "no caries" signal. Training config:

• imgsz = 1280
• batch = 32
• 120 epochs, SGD + cosine schedule, mosaic aug
• A100-40GB (via the Colab notebook in notebooks/)

Stage 2 — the vision-language model

MedGemma 1.5 4B served locally through Ollama (medgemma1.5:latest, Q4_K_M quantization, ~3.3 GB). For every crop produced by the detector, MedGemma answers a fixed prompt asking what it sees and how confident it is that it's caries. The response is the reasoning that ships to the clinician.

Why two stages?

A single-stage VLM being asked to both localize and describe caries in a full-mouth photograph has a well-documented failure mode: it rationalizes whatever box it drew. Separating the two — a specialist detector for localization, then a generalist for narration — means MedGemma only ever sees a tight crop and can't hallucinate a lesion from context that isn't in the crop. It also lets us evaluate the two components independently.

Patient-safe split

The published paper uses a random 80/10/10 image-level split. In this dataset, every patient contributes ~5 views to the corpus — a random image-level split leaks the same mouth into train, val, and test. That inflates all reported metrics. Our splitter (src/prepare_data.py) groups images by patient ID (parsed from the anonymised filenames) and splits at the patient level. The --views filter restricts to occlusal views where caries is actually visible.

---

Repository layout

src/
  prepare_data.py            build the patient-safe, occlusal-only split
  train_yolo.py              train the detector (M4-tuned defaults)
  build_vlm_data.py          convert annotations to VLM instruction JSONL
  finetune_medgemma.py       LoRA fine-tune MedGemma (CUDA/A100)
  main.py                    single-stage MedGemma inference on a whole image
  two_stage.py               detector -> MedGemma per crop
  analyze.py                 clinician-facing two-panel report generator
  eval_two_stage.py          measures whether MedGemma vetos improve precision
  test_full.py               full test-set evaluation, aggregates all metrics

notebooks/
  colab_yolo_training.ipynb        A100 notebook to reproduce best.pt (~30 min)
  colab_medgemma_finetune.ipynb    A100 notebook to LoRA fine-tune MedGemma (~1-2 h)

weights/
  best.pt                    the trained detector (77 MB, YOLOv11m)

assets/
  example_detection_*.png    sample client-facing reports
  gt_vs_prediction.jpg       ground-truth vs. model comparison

---

Quick start

Prerequisites

• Python 3.10+
• Ollama installed and running (ollama serve)
• MedGemma pulled: ollama pull medgemma1.5
• pip install ultralytics pillow pyyaml numpy scikit-learn

Analyze one photograph

python src/analyze.py path/to/photo.jpg --out ./out.png --report ./out.json

Produces a two-panel image (original left, annotated right) with a "Findings" section listing each detected lesion and MedGemma's reasoning.

Reproduce the training

Open notebooks/colab_yolo_training.ipynb in Google Colab, select an A100 runtime, Run All. Takes ~30 minutes and downloads the Zenodo dataset itself.

Reproduce the evaluation

python src/prepare_data.py --root ./Dataset --out ./yolo_split \
    --patient-regex 'anonymous[_-](\d+[_-]\d+[_-]\d+)' \
    --views 'Mandibular,Maxillary_Occlusal' --copy
python src/test_full.py --conf 0.25 --iou-thr 0.5

Writes test/METRICS.json, test/REPORT.md, and 332 per-image PNG + JSON reports to test/testset/.

---

Full ML metrics

All numbers below are computed on the same 332-image patient-safe test split — 850 ground-truth caries boxes total. Run: src/test_full.py, wall time 1h 39min on M4 (bottlenecked by second-stage MedGemma inference).

Detector — overall

Metric	Value	Meaning
mAP50	0.8597	mean Average Precision at IoU=0.5 (standard COCO/YOLO summary metric)
mAP50-95	0.6916	averaged over IoU thresholds 0.5→0.95 in steps of 0.05 (stricter localization)
Precision	0.8243	of the boxes the model draws, this fraction are real lesions
Recall	0.8445	of all real lesions in the images, this fraction are found

Detector — per class

Class definitions from Ahmed et al. 2025: d = primary (deciduous) tooth decay, D = permanent tooth decay.

Class	Support (train boxes)	mAP50	mAP50-95	Precision	Recall
d — primary caries	5,923	0.9353	0.7372	0.8580	0.8764
D — permanent caries	503	0.7842	0.6459	0.7905	0.8126
weighted mean	—	0.8597	0.6916	0.8243	0.8445

The 11.8:1 class imbalance in training data is visible in the primary/permanent gap (~15pp mAP50), but not catastrophically so — recall on the rare D class is still 0.81. If parity across classes matters for a deployment, class-weighted training or a single merged "caries" class would close the gap.

Combined pipeline — detections at conf ≥ 0.25

Every detection from the detector is cropped (with 15% padding) and passed to MedGemma 1.5 (Ollama, Q4_K_M). MedGemma returns a free-text description that we parse into confirm / uncertain / reject. Every detection is matched to ground truth by IoU ≥ 0.5 to label it TP or FP.

Signal	count	notes
Total detections	914	~2.75 detections per image on average
Total ground-truth boxes	850	across 332 images
YOLO-alone TP	763
YOLO-alone FP	151
YOLO-alone FN	87	boxes the detector missed entirely

YOLO alone at this operating point: P = 0.8348, R = 0.8976.

MedGemma verdict × ground-truth cross-tabulation

The interesting bit — is MedGemma's confidence a useful filter?

MedGemma verdict	correct (TP)	wrong (FP)	precision of this verdict
confirm	473	63	88.2% — very reliable positive signal
uncertain	95	16	85.6% — also mostly correct
reject	195	72	only 27.0% correct — mostly rejecting real TPs

The "reject" verdict is not a good filter: three of every four "rejects" are actually true positives the detector correctly found. If we drop them:

YOLO + MedGemma veto (drop all "reject" detections)

Metric	YOLO alone	+ MedGemma veto	Δ
TP	763	568	-195
FP	151	79	-72
FN	87	282	+195
Precision	0.8348	0.8779	+4.3pp
Recall	0.8976	0.6682	-22.9pp

• FPs removed by veto: 72 / 151 = 47.7%
• TPs lost by veto: 195 / 763 = 25.6%

Conclusion: off-the-shelf MedGemma-as-veto is the wrong architecture. You buy 4pp of precision for 23pp of recall. For a screening tool where missed cavities are worse than false alarms a dentist can dismiss in two seconds, this trade is a loss. The useful signal is the "confirm" verdict, not the "reject" one — 88% precision on confirms means MedGemma reliably corroborates real TPs, and that's what should ship: highlight the confirmed detections, present the rest with a "review this" flag.

Comparison against the paper's benchmark

System	Test split	mAP50	Precision	Recall	mAP50-95
YOLOv5s (paper)	image-level (leaky)	0.780	0.768	0.743	—
YOLOv11s (paper)	image-level (leaky)	0.812	0.776	0.735	—
YOLOv8s (paper, best)	image-level (leaky)	0.841	0.828	0.807	—
SSD-MobileNet-v2 (paper)	image-level (leaky)	0.680	—	—	—
Faster R-CNN (paper)	image-level (leaky)	0.306	—	0.461	—
This work best.pt	patient-safe	0.860	0.824	0.845	0.692

The apples-to-apples comparison is with YOLOv8s (paper's strongest baseline). This repo's best.pt beats it on mAP50 (+1.9pp) and recall (+3.8pp) while on a strictly harder split.

Raw data

• Per-image ground truth, predictions, IoUs, MedGemma verdicts, and TP/FP labels: test/testset/*.json (332 files)
• Client-facing annotated PNGs (Original | Analysis + Findings): test/testset/*.png (332 files)
• Aggregate JSON: test/METRICS.json
• Human-readable summary: test/REPORT.md

Honest limitations

• Class imbalance. Training data has 5,923 primary-decay boxes vs 503 permanent-decay boxes (~12:1). The model is heavily biased toward d (primary). Per-class recall on D (permanent) is materially lower. The two classes should probably be merged for a first clinical version.
• Only occlusal views. The detector was intentionally trained only on Mandibular and Maxillary_Occlusal views — the ones where caries is visible. Frontal/lateral views are out-of-distribution and the model will either return nothing or fire on spurious edges.
• Templated reasoning at training time. If you use the MedGemma fine-tune notebook, the training traces are geometrically derived from bounding-box positions ("a lesion in the upper-left region"). This teaches the model to sound clinical, not to reason clinically. For real clinical reasoning you need clinician-authored rationales. The current shipped pipeline uses off-the-shelf MedGemma (no fine-tune), which describes what it sees on each crop without any training-set bias.
• The dataset itself. The Ahmed et al. 2025 dataset is collected from a single mobile device (Samsung Galaxy A23) in a single geographic setting (Mithi, Sindh, Pakistan), covering ages 10-24. Any model trained on it inherits those distributional limits.
• Not a medical device. This is decision-support research code. It does not replace a dentist's judgment.

---

Citations

Dataset — the sole source of all training data used in this project:

Ahmed, S.M.F., Ghori, M.H., Khalid, A., Nooruddin, A., Adnan, N., Lal, A., & Umer, F. (2025). Annotated intraoral image dataset for dental caries detection. Scientific Data, 12, 1297. https://doi.org/10.1038/s41597-025-05647-9

Models used:

• Ultralytics YOLOv11
• Google MedGemma 1.5
• Ollama

---

License

MIT — see LICENSE.

The dataset used to train best.pt is licensed CC BY-NC-ND 4.0 by its authors and is not redistributed in this repository. Download it directly from Zenodo if you need to retrain.