Validate zip members before extracting KITTI

README.md

@@ -288,6 +288,49 @@ confidently wrong one.
 comparable with the KITTI benchmark. This removes a limitation the README used
 to carry.
 
+## And what kind of mistake was the wrong box?
+
+The section above explains the misses. Until now nothing explained the false
+positives: every wrong box counted the same, so a detector that fires twice on
+one pedestrian and a detector that invents pedestrians in empty road produced
+the same number. AP cannot separate them either.
+
+Counted over the whole curve, so these are every box the detector emits at any
+confidence, not the ones a vehicle would act on:
+
+| class | false positives | duplicate | misclassified | mislocalised | **hallucinated** |
+|---|---|---|---|---|---|
+| Car | 33752 | 266 (1%) | 81 (0%) | 5305 (16%) | **28100 (83%)** |
+| Pedestrian | 10808 | 40 (0%) | 674 (6%) | 847 (8%) | **9247 (86%)** |
+
+**Four fifths of the wrong boxes are on nothing at all**, and that is the
+category a safety argument cares about most: it is the only one that makes a
+vehicle brake for empty road, and the only one whose cause is invisible in the
+ground truth. Duplicates are almost absent, so non-maximum suppression is not
+the problem. Misclassification is a rounding error for Car and 6% for
+Pedestrian, where the confusions are with the Cyclist and Car boxes a road
+scene puts people next to.
+
+The categories are defined in `src/ape/outcomes.py` and every one of them is
+read off the same match the metric used, at the same threshold, in the same
+order. Nothing here matches a second time.
+
+## Beyond AP: what no threshold choice can buy
+
+AP integrates over every operating point, which is a question no vehicle asks.
+These are the ones it does ask.
+
+| class | false-negative rate | recall at 90% precision | recall at 50% precision |
+|---|---|---|---|
+| Car | 18.2% | 64.7% | 80.9% |
+| Pedestrian | 31.3% | **0.2%** | 60.7% |
+
+**The pedestrian row is the finding.** An AP of 0.506 reads as a mediocre but
+usable detector. It is not usable at high precision at all: demand 90%
+precision and it returns two pedestrians in a thousand. There is no threshold
+that buys both, and the aggregate hides that completely, which is the argument
+for reporting more than one number per slice.
+
 ## Where would you set the threshold?
 
 Average precision integrates over every confidence threshold at once. That is
@@ -442,6 +485,98 @@ uv sync --group dev
 uv run pytest
 ```
 
+## Metamorphic robustness: the same scene, degraded a stated amount
+
+A second dataset changes the scene, the camera, the labelling policy and the class balance at once, so a drop in AP has four candidate causes. A perturbation changes exactly one thing by a stated amount and **leaves the ground truth identical**, so the curve is attributable. That is what makes these metamorphic relations rather than augmentations.
+
+500 KITTI frames, `yolov8s`, IoU 0.5, worst drop relative to the unperturbed baseline (`scripts/sweep_robustness.py`, full curves in `outputs/robustness.md`):
+
+| perturbation | at | Car | Pedestrian |
+|---|---|---|---|
+| blur | 4 px radius | -16.3% | **-17.4%** |
+| contrast removed | 0.8 | -13.0% | **-17.6%** |
+| JPEG | quality 10 | -8.4% | -11.7% |
+| fog veil | 0.6 opacity | -8.0% | -7.2% |
+| brightness | ±0.6 | **-0.2%** | -2.2% |
+
+**Three findings.**
+
+**Exposure is free and defocus is not.** Brightness at ±60% costs Car essentially nothing, which is a real result rather than a broken perturbation: the tests assert the image actually changed. A pipeline worrying about tunnel mouths and low sun is worrying about the wrong thing; one worrying about a dirty or misfocused lens is not.
+
+**Pedestrians degrade faster than cars under every perturbation except fog.** The class that matters most for a braking decision is the more fragile one, and the gap widens with strength: at blur radius 2 the Car cost is 3.3% and the Pedestrian cost is 8.5%. A single aggregate mAP hides that completely.
+
+**Nothing here falls off a cliff.** Every curve is gradual, so there is no threshold below which the detector stops working, and a degradation curve is the honest way to report that. A single number at one operating point would suggest a robustness the smooth decline does not contradict but also does not demonstrate.
+
+**Read with three caveats, all of them stated in the code.** This is 500 frames, so the baselines here (Car 0.758, Pedestrian 0.443) are not the headline figures above, which come from all 7481. The fog is a **uniform veil, not depth-aware**, so it understates exactly the distance dependence that matters most for ADAS; `vkitti` is where depth-aware weather belongs. And Cyclist is mapping-limited to the point of meaninglessness here, so its column is omitted.
+
+**Crop is deliberately not included.** It is a reasonable perturbation and it moves the boxes, so the ground truth would have to be transformed with it, which makes a bug in the box transform indistinguishable from a real drop. The whole point of this module is that nothing about the labels changes.
+
+## Calibration, and why the sign matters more than the size
+
+mAP asks how often the detector is right. **Calibration asks whether it knows how
+often it is right**, and nothing else here measured that. A detector at 0.68 mAP
+that reports 0.95 on every box it will get wrong is a worse engineering problem
+than one reporting 0.4 on those boxes, because the second can be gated by a
+threshold and the first cannot.
+
+`src/ape/calibration.py` bins detections by confidence and reports what each band
+actually delivered: a reliability diagram as data, plus expected calibration
+error, maximum calibration error, and **overconfidence error**.
+
+That last one is the point. **ECE is symmetric.** A detector claiming 0.4 while
+being right 0.9 of the time scores exactly as badly as one claiming 0.9 while
+being right 0.4 of the time, and those are not equally dangerous. The first is
+timid and merely wastes performance; the second is **confidently wrong**, which
+is the failure ISO 21448 exists for. A test constructs that exact pair and
+asserts ECE cannot tell them apart while overconfidence error can.
+
+**Slices are cut on the detection, not the ground truth**, which is the decision
+here worth arguing with. Every other slice in this repository cuts on
+ground-truth attributes: range, occlusion, truncation. Those exist only for
+objects that are really there, so slicing calibration that way would silently
+drop every false positive, and false positives are exactly where overconfidence
+does its damage. Box height stands in for range. It is a weaker proxy than
+KITTI's labelled distance and it is the only one a box corresponding to nothing
+can have.
+
+## Is this frame the kind of thing we validated on?
+
+Every other measurement here asks how well the detector did on some data.
+`src/ape/ood.py` asks the prior question: **is this data the data we validated
+against.** A frame that is not is a triggering condition whether or not the
+detector happened to get it right, which is the ISO 21448 case where nothing has
+failed and the world is simply outside the design envelope.
+
+It fits an operating envelope over six cheap image statistics and scores new
+frames by Mahalanobis distance. **Mahalanobis rather than a z-score per feature
+because the features covary**: a foggy frame is brighter *and* lower contrast
+*and* has fewer edges together, and scoring each independently treats one
+moderate joint excursion as three unremarkable ones. A test puts two probes the
+same distance out on every individual feature, one along the correlation and one
+across it, and asserts the second scores an order of magnitude higher.
+
+**What it is not:** a learned OOD method. There is no network and nothing is
+trained, consistent with the rest of this repository. It will notice fog, night,
+blur, a blown exposure and compression artefacts. **It will not notice a
+semantically novel object rendered at ordinary brightness and contrast**, and
+that limit is the interesting half of the honesty, because it is exactly the
+failure a statistics-only detector cannot see.
+
+**An OOD score nobody has validated is a number, not evidence.** `agreement()`
+measures whether high-scoring frames actually did worse, reporting an AUC that
+sits at 0.5 for a score carrying no information. A score that cannot rank the
+degraded frames first has not earned the right to gate anything. The output is
+called `triggering_candidates` rather than triggering conditions on purpose: a
+triggering condition is a scenario a person describes and reasons about, and
+promoting a statistic straight into a safety artefact is the shortcut that name
+refuses to take.
+
+## Roadmap
+
+- ~~**Calibration and OOD scoring**~~ **Done, 1 September.** `src/ape/calibration.py` and `src/ape/ood.py`. See the section above.
+
+Not doing: **nuScenes, BDD100K or Waymo before the metamorphic curves exist** (large, licence-gated, and they answer a question the harness has not yet shown it can express). Not training a better detector either, which would make the numbers nicer and the point weaker.
+
 ## Licence
 
 Code under MIT. KITTI is CC BY-NC-SA 3.0 and is not included.

outputs/report.html

@@ -130,4 +130,4 @@ <h2>By position</h2>
 </tbody></table></div>
 <div class="note">The small figures under each number are a 95% confidence interval from resampling FRAMES, not objects: people standing in one group are not independent observations, and resampling objects would understate the range. Two cells whose intervals do not overlap differ by more than the sample explains. Two whose intervals DO overlap are not thereby shown to be the same, which is a weaker statement than it looks. A tilde still marks a cell computed from fewer than ten objects.</div>
 <h2>What this does not claim</h2><div class="note">Cyclist is reported but excluded from the headline: KITTI annotates a rider and bicycle as one box and a COCO detector emits two, so that number measures box convention as much as detection. The IoU threshold is 0.5 for every class, where KITTI's own benchmark uses 0.7 for Car. Average precision follows COCO 101-point interpolation, not KITTI 40-point, so these figures are not directly comparable to the KITTI leaderboard. SOTIF vocabulary is borrowed; its process is not performed and no compliance is claimed.</div>
-<p class="meta">Generated 2026-08-02 13:15 UTC from 7481 frames, 000000 to 007480, detections kept above score 0.05. The mAP implementation is checked against pycocotools to within 0.001 by the test suite.</p></main></body></html>
+<p class="meta">Generated 2026-08-31 10:28 UTC from 7481 frames, 000000 to 007480, detections kept above score 0.05. The mAP implementation is checked against pycocotools to within 0.001 by the test suite.</p></main></body></html>

outputs/results.json

@@ -85,6 +85,73 @@
       "mislocation_share": 0.1904127829560586
     }
   },
+  "false_positive_kinds": {
+    "Car": {
+      "total": 33752,
+      "counts": {
+        "duplicate": 266,
+        "misclassified": 81,
+        "mislocalised": 5305,
+        "hallucinated": 28100
+      },
+      "shares": {
+        "duplicate": 0.007881014458402466,
+        "misclassified": 0.002399857786205262,
+        "mislocalised": 0.15717587105949277,
+        "hallucinated": 0.8325432566958995
+      }
+    },
+    "Pedestrian": {
+      "total": 10808,
+      "counts": {
+        "duplicate": 40,
+        "misclassified": 674,
+        "mislocalised": 847,
+        "hallucinated": 9247
+      },
+      "shares": {
+        "duplicate": 0.003700962250185048,
+        "misclassified": 0.06236121391561806,
+        "mislocalised": 0.0783678756476684,
+        "hallucinated": 0.8555699481865285
+      }
+    },
+    "Cyclist": {
+      "total": 8100,
+      "counts": {
+        "duplicate": 2,
+        "misclassified": 50,
+        "mislocalised": 525,
+        "hallucinated": 7523
+      },
+      "shares": {
+        "duplicate": 0.0002469135802469136,
+        "misclassified": 0.006172839506172839,
+        "mislocalised": 0.06481481481481481,
+        "hallucinated": 0.9287654320987654
+      }
+    }
+  },
+  "beyond_ap": {
+    "Car": {
+      "false_negative_rate": 0.18199846913923878,
+      "max_recall": 0.8180015308607612,
+      "recall_at_precision_90": 0.6471713868206805,
+      "recall_at_precision_50": 0.8085380279730012
+    },
+    "Pedestrian": {
+      "false_negative_rate": 0.31268107867171835,
+      "max_recall": 0.6873189213282817,
+      "recall_at_precision_90": 0.0020057945174949856,
+      "recall_at_precision_50": 0.6068642745709828
+    },
+    "Cyclist": {
+      "false_negative_rate": 0.9231714812538414,
+      "max_recall": 0.07682851874615858,
+      "recall_at_precision_90": 0.0,
+      "recall_at_precision_50": 0.0
+    }
+  },
   "operating_points": {
     "Car": [
       {

scripts/evaluate.py

@@ -22,6 +22,8 @@
 from ape.classes import EVALUATED, HEADLINE  # noqa: E402
 from ape.evaluate import IOU, evaluate, operating_table  # noqa: E402
 from ape.kitti import frame_ids, load_labels  # noqa: E402
+from ape.metrics import false_negative_rate, recall_at_precision  # noqa: E402
+from ape.outcomes import FAILURES  # noqa: E402
 from ape.report import render  # noqa: E402
 
 
@@ -102,6 +104,24 @@ def main() -> int:
               f"{diagnosis.mislocated} mislocated, {diagnosis.unseen} unseen "
               f"-> {diagnosis.mislocation_share:.0%} of misses are a box problem")
 
+    print("\nwhat kind of mistake was it? (AP counts every wrong box the same)")
+    for label in HEADLINE:
+        breakdown = result.outcomes[label]
+        parts = "  ".join(f"{outcome.value} {breakdown.counts[outcome]}"
+                          f" ({breakdown.share(outcome):.0%})"
+                          for outcome in FAILURES)
+        print(f"  {label:<11} {breakdown.false_positives} false positives")
+        print(f"              {parts}")
+
+    print("\nbeyond AP: what no threshold choice can buy")
+    for label in HEADLINE:
+        curve = result.overall[label]
+        print(f"  {label:<11} false-negative rate "
+              f"{false_negative_rate(curve):.1%} at the recall ceiling")
+        print(f"              recall at 90% precision "
+              f"{recall_at_precision(curve, 0.90):.1%}, "
+              f"at 50% precision {recall_at_precision(curve, 0.50):.1%}")
+
     print("\nchoosing an operating point (AP integrates over all of them; "
           "a vehicle runs at one)")
     for label in HEADLINE:
@@ -131,6 +151,17 @@ def main() -> int:
                           "mislocated": d.mislocated, "unseen": d.unseen,
                           "mislocation_share": d.mislocation_share}
                       for k, d in result.diagnosis.items()},
+        "false_positive_kinds": {
+            k: {"total": b.false_positives,
+                "counts": {o.value: b.counts[o] for o in FAILURES},
+                "shares": {o.value: b.share(o) for o in FAILURES}}
+            for k, b in result.outcomes.items()},
+        "beyond_ap": {
+            k: {"false_negative_rate": false_negative_rate(c),
+                "max_recall": c.best_recall,
+                "recall_at_precision_90": recall_at_precision(c, 0.90),
+                "recall_at_precision_50": recall_at_precision(c, 0.50)}
+            for k, c in result.overall.items()},
         "operating_points": {
             label: [{"target": target,
                      "threshold": p.threshold if p else None,

scripts/fetch_kitti.py

@@ -96,6 +96,20 @@ def download(archive: str, into: Path, expected: int) -> Path:
 def extract(archive: Path, into: Path, must_contain: str) -> None:
     print(f"  extracting {archive.name}")
     with zipfile.ZipFile(archive) as zf:
+        # Every member is checked to land inside `into` before anything is
+        # written. ZipFile has no equivalent of tarfile's filter="data", which
+        # is what fetch_vkitti.py uses, so the check is written out here. An
+        # entry named "../../etc/thing" or an absolute path would otherwise
+        # extract outside the data directory; these archives come from a fixed
+        # official URL, so this is a guard against the URL or the host
+        # changing, not against KITTI.
+        root = into.resolve()
+        for member in zf.infolist():
+            destination = (root / member.filename).resolve()
+            if destination != root and root not in destination.parents:
+                raise SystemExit(
+                    f"{archive.name} contains {member.filename!r}, which would "
+                    f"extract outside {into}. Refusing to unpack it.")
         zf.extractall(into)
     if not (into / must_contain).is_dir():
         raise SystemExit(

scripts/render_demo.py

@@ -53,7 +53,7 @@
 from ape.classes import neutral_labels  # noqa: E402
 from ape.evaluate import IOU  # noqa: E402
 from ape.kitti import frame_ids, load_labels  # noqa: E402
-from ape.match import partition  # noqa: E402
+from ape.match import judge_frame  # noqa: E402
 from ape.slices import dimension  # noqa: E402
 
 GREEN, RED, BLUE, GREY, INK, PAPER = ((60, 200, 120), (235, 60, 80),
@@ -86,27 +86,19 @@ def match(truth, detections, label):
     detector did not see this" and "the detector saw it and boxed it badly",
     which are different failures with different fixes and looked identical in
     the first version of this scene.
+
+    THIS USED TO BE ITS OWN COPY OF THE MATCHER. It reimplemented the greedy
+    descending-score loop from `ape.match`, closely but not identically: it had
+    no notion of neutral ground truth, so a box on a Person_sitting counted as
+    claiming nothing here while the metric declined to score it at all. The
+    picture and the reported number could therefore disagree about the same
+    frame, with nothing in the repository able to notice. It now asks the same
+    function the metric does, and `tests/test_outcomes.py` asserts that
+    function and `assign_frame` cannot diverge.
     """
-    counts, _ = partition(truth, label, neutral_labels(label), None)
-    claimed: set[int] = set()
-    for detection in sorted(detections, key=lambda d: d.score, reverse=True):
-        if detection.label != label:
-            continue
-        best, index = 0.0, -1
-        for i, candidate in enumerate(counts):
-            if i in claimed:
-                continue
-            overlap = detection.box.iou(candidate.box)
-            if overlap > best:
-                best, index = overlap, i
-        if index >= 0 and best >= IOU:
-            claimed.add(index)
-
-    found = [g for i, g in enumerate(counts) if i in claimed]
-    gone = [(g, max((d.box.iou(g.box) for d in detections
-                     if d.label == label), default=0.0))
-            for i, g in enumerate(counts) if i not in claimed]
-    return found, gone
+    outcome = judge_frame(detections, truth, label, neutral_labels(label), IOU)
+    return outcome.found, [(item, outcome.best_overlap_on(item))
+                           for item in outcome.missed]
 
 
 def main() -> int: