Skip to content

Redesign calibration loss as capped weighted MAPE#96

Merged
MaxGhenis merged 1 commit into
mainfrom
codex/loss-redesign-20260617
Jun 17, 2026
Merged

Redesign calibration loss as capped weighted MAPE#96
MaxGhenis merged 1 commit into
mainfrom
codex/loss-redesign-20260617

Conversation

@MaxGhenis

Copy link
Copy Markdown
Contributor

Summary

Redesigns Populace calibration loss from squared bounded relative error to capped weighted MAPE:

loss = weighted_mean(min(abs((estimate - target) / scale), cap))
scale = max(abs(target), abs(initial_estimate), 1)
cap = 10  # 1000% per target

This fixes the a912 failure mode where zero-valued/tiny fiscal rows dominated the objective and prevented meaningful movement on major targets like CTC.

Changes

  • Adds fixed target-loss scales in populace.calibrate, defaulting to max(abs(target), abs(initial_estimate), 1) after matrix compilation.
  • Caps each target's scaled miss at 1000%.
  • Lowers the default Adam learning rate to 0.02, which is better behaved for absolute-loss gradients.
  • Removes the obsolete compiled-target -1 guard from matrix compilation; the new loss has no (target + 1) denominator.
  • Changes the US fiscal refresh builder from raw dollar target weighting / CTC-only boost to semantic weights:
    • national critical fiscal roles get higher priority,
    • state rows are downweighted,
    • state rows do not inherit national role boosts.
  • Updates PUF aggregate diagnostics and docs to report the new loss formula.

Sanity check

On the published a912 diagnostics, the old objective's top 10 rows accounted for ~99.9999989% of total loss. Under the new capped MAPE scoring on the same final estimates, the top 10 rows account for ~0.6%, so loss is no longer owned by a handful of zero/tiny rows.

Verification

  • uv run python -m pytest -q passed
  • uv run ruff check ... passed on changed Python files
  • git diff --check passed
  • /cycle review: first pass found state rows were still getting national role boosts; fixed with regression coverage. Second pass found no actionable issues.

@MaxGhenis MaxGhenis merged commit 8aa3dbb into main Jun 17, 2026
4 checks passed
@MaxGhenis MaxGhenis deleted the codex/loss-redesign-20260617 branch June 17, 2026 15:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant