Summary
The Assessment Report's damaged-building population estimate can report a 95% confidence interval whose lower bound is negative, which is impossible for a count.
Observed
Spokane wildfire layer, DINOv2 ViT-B/14, 647 predicted damaged of 112,687 footprints, validated with the default 200-building sample (199 sure-labeled, 2 damaged):
Extrapolating to all 97,541 buildings with area > 50 m², we estimate 980 damaged buildings (1.0%) with a 95% CI of [-373, 2,334].
| Field |
Value |
| N (buildings with area > 50 m²) |
97,541 |
| Sample size (n) |
199 |
| Damaged in sample (x) |
2 |
| Sample damage rate (p̂) |
1.0% |
| Std. error of p̂ |
0.007081 |
| Estimated damaged (Ŷ) |
980 |
| 95% CI |
[-373, 2,334] |
Cause
The interval appears to be a normal (Wald) approximation, p̂ ± 1.96·SE, applied to x = 2. Wald is not valid at small counts — the usual guidance is np̂ ≥ 5 and n(1-p̂) ≥ 5, and here np̂ = 2. With p̂ = 0.01 and SE = 0.00708, the lower bound goes negative.
Suggested fix
- Use an interval valid at small counts — Wilson score or Clopper–Pearson — instead of Wald.
- Clamp reported bounds to
[0, N] regardless of method.
- Warn in the report when
x is small (say < 10) that the estimate is low-confidence.
Two related observations on the same report
- The point estimate contradicts the model's own count. The report extrapolates from the sample to 980, while the model predicted 647. Both numbers appear in the product with equal authority and no explanation of why they differ or which to cite.
- Precision/recall are reported as bare 100%. "Estimated recall 100.0% and precision 100.0%" comes from those same 2 positives, with no indication of how little that constrains the true value.
Related
The interval's width here is a direct consequence of the unstratified validation sample (#50) — a uniform sample of a 0.57%-prevalence class yielded only 2 positives. Fixing #50 improves the input, but this issue is about the estimator being invalid regardless of how the sample was drawn.
Summary
The Assessment Report's damaged-building population estimate can report a 95% confidence interval whose lower bound is negative, which is impossible for a count.
Observed
Spokane wildfire layer, DINOv2 ViT-B/14, 647 predicted damaged of 112,687 footprints, validated with the default 200-building sample (199 sure-labeled, 2 damaged):
Cause
The interval appears to be a normal (Wald) approximation,
p̂ ± 1.96·SE, applied tox = 2. Wald is not valid at small counts — the usual guidance isnp̂ ≥ 5andn(1-p̂) ≥ 5, and herenp̂ = 2. Withp̂ = 0.01andSE = 0.00708, the lower bound goes negative.Suggested fix
[0, N]regardless of method.xis small (say< 10) that the estimate is low-confidence.Two related observations on the same report
Related
The interval's width here is a direct consequence of the unstratified validation sample (#50) — a uniform sample of a 0.57%-prevalence class yielded only 2 positives. Fixing #50 improves the input, but this issue is about the estimator being invalid regardless of how the sample was drawn.