EPIC: certified populace-us default correctness (gates not fully wired / reference rotted)

[MaxGhenis]

Tracking epic — certified populace-us default correctness

The certified US default (populace_us_2024.h5, bundle 4.16.2) has correctness issues its gate suite did not catch — some because a gate's reference rotted, some because a defined gate was never wired into the build. All findings below are reproduced (sha-verified certified artifact; build's own check_parity.py recovered from git history; column classification against policyengine-us variable definitions). This epic centralizes the fixes.

Root meta-issue

The acceptance suite (exported_nonzero, formula_owned_export, parity, support, aggregate_vs_admin, per_family_fit) exists in populace.build.gates, but the build pipeline only ran exported_nonzero + parity, against an unpinned eCPS reference, and the runner + reference were deleted from HEAD (fda3838 "Remove retired build and benchmark surfaces"). So "parity 0" was real but non-reproducible and silently rotted, and formula_owned_export never ran at all.

Issues (suggested order)

• Formula-owned export must supersede eCPS parity matching (drop formula-owned vars even when eCPS stores them) #25 — Formula-owned supersedes parity (gate-precedence decision; eCPS storing formula-derived IRA contributions etc. is an eCPS bug populace must not inherit). Sets the formula-owned drop list to 22 and makes parity exempt formula-owned layers.
• Certified populace-us persists 18 formula-owned variables eCPS doesn't (formula_owned_export gate never wired into the build) #24 — Wire formula_owned_export into the build + drop the masked columns (18→22 per Formula-owned export must supersede eCPS parity matching (drop formula-owned vars even when eCPS stores them) #25). Stops the certified data from freezing engine-computed values (ssi, taxable_unemployment_compensation, social_security, the income family, …) that then can't respond to reform.
• #19 — Pin the parity reference + record the result in the manifest + restore/keep the runner + CI drift-check (fix PR populace-build: reference-pinned, recorded, reconstructable parity (part of #19) #21 already in review for the pin/record/runner half).
• Drop raw-source and build-intermediate columns from the published populace-us artifact #27 — Drop raw-source + build-intermediate columns from the published artifact (keep geo identifiers + weights).
• Close missing eCPS-populated input layers in populace-us (6 PUF credits + SE-health upstream + reported aggregates) #26 — Impute the missing pure-input layers (6 PUF credits + fix SE-health upstream + decide on the reported aggregates).
• Rebuild + re-certify with the full gate suite running against a pinned reference, all gate results recorded in the release manifest (and content-hashed into the build TRO).

Decision input

• #20 — Downstream blocking assessment (CRFB taxation-of-benefits, PolicyBench) — informs how urgently the rebuild ships vs. the current certified default.

PR #21 (pin-parity-reference) already lands the reference-pinned, recorded, reconstructable parity runner (part of PolicyEngine/populace-benchmarks#1).

[MaxGhenis]

Coverage audit addendum: I checked the current open issues/PRs against the export-mask/eCPS needs. The US core is covered by #24/#25/#19/#26/#27, and the poverty/SPM target/diagnostic gaps are now covered by #29/#30/#32/#33. I filed #34 as a focused subtask for the in_nyc county-input edge case so the formula-owned cleanup does not accidentally recompute NYC to zero. I also filed #31 for the UK-side formula-owned export audit/gate; that is outside this US epic but came from the same class of bug.

[MaxGhenis]

Superseded after #37 and the benchmark split. The live Populace repo now has targeted open issues for actual production work (#27, #29-#35, #38), while incumbent comparison/reference-drift discussion moved to PolicyEngine/populace-benchmarks#1 and #2.