Validate zip members before extracting KITTI - #1
Merged
Merged
Conversation
fetch_kitti called zf.extractall() with no member validation while its sibling fetch_vkitti already used tarfile's filter="data". ZipFile has no equivalent, so the check is written out: an entry resolving outside the target directory now refuses to unpack. The archives come from a fixed official URL, so this guards against that URL or host changing rather than against KITTI.
… slice Three roadmap items, and the first two turn out to be the same refactor. ONE MATCHER. scripts/render_demo.py carried its own copy of the greedy descending-score assignment loop. Close to the real one and not identical: it had no notion of neutral ground truth, so a box on a Person_sitting counted as claiming nothing in the picture while the metric declined to score it at all. The demo and the reported number could disagree about the same frame and nothing in the repository could notice. ape.match.judge_frame now does the matching and keeps the per-object detail; assign_frame is a view of it that throws the detail away, and the demo asks the same function the metric does. tests/test_outcomes.py asserts the two cannot diverge across seven frame shapes, including the ones where matching order decides the answer. The pycocotools comparison still agrees, which is the real evidence the refactor changed nothing: 238 tests pass, up from 218. A TAXONOMY FOR THE FALSE POSITIVES. ape.localisation already splits the MISSES into "never seen" and "seen and boxed badly" and explains why that matters. The false positives had no such split, so a detector that fires twice on one pedestrian and a detector that invents pedestrians in empty road produced the same number. ape.outcomes sorts every wrong box into duplicate, misclassified, mislocalised or hallucinated, reading each verdict off the match the metric already performed rather than matching a second time. Measured on KITTI with yolov8s: Car 33752 FP 1% duplicate 0% misclassified 16% mislocalised 83% hallucinated Pedestrian 10808 FP 0% duplicate 6% misclassified 8% mislocalised 86% hallucinated Four fifths of the wrong boxes are on nothing at all, which is the category a safety argument cares about most: the only one that makes a vehicle brake for empty road, and the only one with no cause visible in the ground truth. Duplicates are almost absent, so NMS is not the problem. MORE THAN AP PER SLICE. false_negative_rate and recall_at_precision, both read off the precision envelope for the same reason AP is. Reported by the CLI and carried in results.json. Car FNR 18.2% recall at 90% precision 64.7% at 50% 80.9% Pedestrian FNR 31.3% recall at 90% precision 0.2% at 50% 60.7% The pedestrian row is the finding, and it is the argument for the whole item. An AP of 0.506 reads as a mediocre but usable detector. It is not usable at high precision at all: demand 90% precision and it returns two pedestrians in a thousand. No threshold buys both, and the aggregate hides that completely.
Three items and one decline. Metamorphic robustness first because it gives most of the domain-shift story from data already fetched, and because it proves the harness can express a degradation result before any licence-gated download is worth doing.
A second dataset changes the scene, the camera, the labelling policy and the
class balance at once, so a drop in AP has four candidate causes and the result
is a number rather than a finding. A perturbation changes exactly one thing by
a stated amount and leaves the ground truth identical, which is what makes
these metamorphic relations rather than augmentations.
Five perturbations, each one a camera actually suffers: brightness, contrast,
blur, JPEG round-trip and a fog veil. `Detector.detect_image` is extracted so a
perturbed frame is scored through the same pipeline as the baseline; a sweep
that ran a different pipeline would be measuring the pipeline.
Measured over 500 KITTI frames with yolov8s, worst drop against the
unperturbed baseline:
blur 4 px Car -16.3% Pedestrian -17.4%
contrast 0.8 Car -13.0% Pedestrian -17.6%
jpeg quality 10 Car -8.4% Pedestrian -11.7%
fog 0.6 Car -8.0% Pedestrian -7.2%
brightness +/-0.6 Car -0.2% Pedestrian -2.2%
Three findings. Exposure is free and defocus is not, so a pipeline worrying
about tunnel mouths is worrying about the wrong thing. Pedestrians degrade
faster than cars under every perturbation except fog, and the gap widens with
strength: at blur radius 2 it is 3.3% against 8.5%. And nothing falls off a
cliff, which is a result a single number at one operating point could not have
shown either way.
Crop is deliberately excluded. It moves the boxes, so the ground truth would
have to be transformed with it, and a bug in that transform would be
indistinguishable from a real drop.
The fog is a uniform veil rather than depth-aware, so it understates the
distance dependence that matters most for ADAS. Said in the module, in the
report and in the README rather than left for a reader to assume otherwise.
32 tests on the perturbations themselves: strength zero is exactly the
identity, size and mode are preserved, each one changes the image, each is
deterministic, and each does the specific thing it claims rather than merely
something.
The overall figures and every slice cell already carried a 95% interval; the Easy, Moderate and Hard rows did not, and they sat in the same table as the overall row that did. Reporting them as bare numbers beside a row showing a range invites exactly the comparison the intervals exist to prevent: Easy against Hard looks like a finding until you can see how much of the gap the sample explains. Same bootstrap, resampling frames rather than objects, and the tiers are now assembled per frame so the interval and the point estimate come from the same assignment rather than from two passes.
Two questions nothing here could answer. mAP asks how often the detector is right; calibration asks whether it KNOWS how often it is right. And every measurement here asks how well it did on some data; the OOD score asks whether that data is the data we validated against. src/ape/calibration.py bins detections by confidence and reports what each band delivered: a reliability diagram as data, ECE, MCE, and overconfidence error. THE SIGN IS THE POINT. ECE is symmetric, so a detector claiming 0.4 while right 0.9 of the time scores exactly as badly as one claiming 0.9 while right 0.4 of the time. Those are not equally dangerous: the first is timid and wastes performance, the second is confidently wrong, which is the failure ISO 21448 exists for. A test constructs that pair and asserts ECE cannot separate them while overconfidence error can. Slices are cut on the DETECTION rather than the ground truth. Every other slice here cuts on ground-truth attributes, which exist only for objects that are really there, so cutting calibration that way would silently drop every false positive. False positives are exactly where overconfidence does its damage. Box height stands in for range, which is weaker than KITTI's labelled distance and is the only proxy a box corresponding to nothing can have. src/ape/ood.py fits an operating envelope over six image statistics and scores frames by Mahalanobis distance. Mahalanobis and not a per-feature z-score because the features covary: a foggy frame is brighter AND lower contrast AND has fewer edges together, and scoring each independently treats one joint excursion as three unremarkable ones. A test puts two probes equally far out on every individual feature, one along the correlation and one across it, and asserts the second scores an order of magnitude higher. Stated as a limit rather than left to be discovered: this is a feature-space novelty detector, nothing is trained, and it cannot see a semantically novel object rendered at ordinary brightness and contrast. agreement() exists because an OOD score nobody has validated is a number rather than evidence. It reports an AUC that sits at 0.5 for a score carrying no information, so a score that cannot rank the degraded frames first can be seen not to have earned the right to gate anything. The output is named triggering_candidates, not triggering conditions: a triggering condition is a scenario a person reasons about, and promoting a statistic straight into a safety artefact is the shortcut that name refuses to take. Verified: 315 passed, both new modules at 100% statement and branch coverage, ruff clean, mypy strict clean. Break-it pass: 8 of 8 seeded defects killed, including overconfidence collapsing back into ECE, ignored detections counted as failures, and Mahalanobis degrading to a per-feature distance.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fetch_kitti called zf.extractall() with no member validation while its sibling
fetch_vkitti already used tarfile's filter="data". ZipFile has no equivalent,
so the check is written out: an entry resolving outside the target directory
now refuses to unpack. The archives come from a fixed official URL, so this
guards against that URL or host changing rather than against KITTI.