Fix PUF real-half QRF imputation

[daphnehanse11]

Summary

• keep the existing rich-preserving QRF draw for the zero-weight PUF clone half
• add a separate PUF-weighted, income-conditioned QRF draw for positive-weight CPS records
• use broader observed income/capital-gain/pension/social-security predictors for the real-half draw, while dropping any predictor that is itself being imputed
• add unit coverage for separate real-vs-clone draws plus a raw-file charitable smoke test

Validation

• uv run ruff check policyengine_us_data/calibration/puf_impute.py tests/unit/calibration/test_calibration_puf_impute.py validation/puf_qrf_real_half_smoke.py
• uv run pytest tests/unit/calibration/test_calibration_puf_impute.py
• uv run pytest tests/unit/test_puf_impute.py tests/unit/test_extended_cps.py tests/unit/calibration/test_retirement_imputation.py
• uv run python -m compileall -q policyengine_us_data/calibration/puf_impute.py tests/unit/calibration/test_calibration_puf_impute.py validation/puf_qrf_real_half_smoke.py
• uv run python validation/puf_qrf_real_half_smoke.py

Smoke result: raw PUF weighted combined charitable is $236.4B; old unweighted demographic-only QRF gives $15,523.1B (65.7x); fixed weighted income-conditioned QRF gives $357.0B (1.5x).

Notes

• The smoke script is diagnostic, not release validation. It works from raw puf_2015.csv, demographics_2015.csv, and cps_2024.h5 because the local dataset-backed Microsimulation path currently fails on stale CPS schema/shape issues.
• Full acceptance still needs a rebuilt enhanced CPS artifact and Daphne's 2026 dashboard/JCT comparison rerun.

[anth-volk]

Rebased onto current main (f7458313) to resolve the merge conflicts, with one substantive integration fix: main's Forbes/top-tail training exclusion landed after this branch's base, and the new weighted real-half training frame (X_train_real) bypassed it. X_train_real now flows through the same non_forbes_mask filtering as the clone-half frames, with the positive-weight filter moved below the Forbes block so the person-level mask stays index-aligned.

Test integration:

• TestForbesTrainingExclusion fakes now answer household_weight and accept the weight_col/max_train_samples kwargs; assertions cover all four sequential-QRF training frames (clone full/override + real full/override), so they now also prove Forbes donors stay out of the weighted real-half draw.
• The new test_run_qrf_imputation_splits_weighted_real_half_from_clone_half fake gained a load_dataset() stub for main's Forbes metadata detection.
• test_retirement_imputation.py's 2-tuple _run_qrf_imputation mocks are covered by the existing backwards-compat shim.
• Added the towncrier fragment (changelog.d/1170.fixed.md).

Validation: ruff format/ruff check clean on the three touched files; tests/unit/calibration/test_calibration_puf_impute.py 33 passed; tests/unit/test_puf_impute.py + tests/unit/test_extended_cps.py + tests/unit/calibration/test_retirement_imputation.py 124 passed; scripts/run_quality_guards.py all passed.

🤖 Generated with Claude Code