Synthetic-n recoverability across real Q' sources (teacher/student twin experiment) - #29
Merged
Conversation
Ground-truth-anchored control test for the equifinality campaign's bias-absorption hypothesis: prescribe a known Manning's n field, fix geometry to the 4-checkpoint consensus median, route a known Q' through the real MC solver to generate noise-free synthetic gauge observations, then check whether recovered n diverges by real Q'-source while geometry does not.
Two gaps flagged in review: (1) the disagg-head-is-independent-of-geometry claim was asserted, not tested — S3's cross-arm consistency check could be confounded by every arm sharing the same frozen disagg head, so this needs an explicit caveat rather than treatment as closed. (2) this campaign's 4 arm names (aorc2f_distributed/lumped, daily/hourly-lstm) are a different naming scheme from the pre-registered equifinality campaign's R1/R2/R3 — must not conflate the two in any findings doc or paper table.
Reuses eval-loss mode's load_comid_field/gather_by_comid/RoutingParamOverride machinery so teacher mode can generate synthetic-twin ground truth for the routing parameters, not just leakance. --plant-file/--zeta-output/use_leakance are now independent of the new --donor-params-nc path.
Runs dump_parameters against the 4 already-converged real-Q'-source checkpoints from this campaign and computes the per-COMID median q_spatial/p_spatial as the fixed geometry truth for the synthetic-n recoverability experiment.
A prior run killed mid-write could leave a truncated .nc that "exists" but is missing variables; silently reusing it would poison the fixed geometry truth every synthetic-n student depends on. Check q_spatial/p_spatial are actually readable before skipping, else delete and regenerate. Also drops an unused import flagged in code review.
…ea range The b=0.15 placeholder exponent, centered on the median in log-space, undershot both bounds (realized range [0.017,0.099] vs the intended [0.015,0.15]) because log10_uparea is right-skewed — the design spec explicitly flagged this calibration as needing verification against the real attribute distribution, which hadn't happened yet. Anchor instead on the 1st/99th percentile of log10_uparea so the field actually spans the full intended range, with the percentile (not true min/max) anchor avoiding a few extreme-tail reaches compressing everyone else's scale.
…olerance The donor round-trip goes through denormalize -> physical_to_normalized (an f32 ln/exp round-trip for log-space params like p_spatial), which src/training/forward.rs's own unit test documents as accurate to < 1e-3 m3/s absolute, not bit-exact. A byte-identical zarr-chunk comparison failed on exactly this expected ~7.6e-6 abs noise (~130x inside that floor) — compare decoded f64 discharge values against the same tolerance the donor mechanism is actually specified to instead.
…ic obs) Each is an exact copy of its real-campaign counterpart (aorc2f_distributed_frozen_chunk1, aorc2f_lumped_frozen_chunk1, lstm_daily_frozen_chunk1, lstm_hourly_native) with only data_sources.observations repointed to the synthetic-n teacher's gauge-observation store (output/synthetic_n/synthetic_obs_lm, not yet produced by the in-flight teacher run). Q' source, disagg-head settings, and all other arm-specific config stay identical to the real campaign so results are comparable against the shared synthetic-truth answer key.
…figs Each header was the untouched source config's header, still describing itself as the real-campaign arm (wrong filename, "Companion arm:" cross-references pointing at the real campaign instead of the synthetic-n sibling). Prepend a short derived-config header matching the synthetic_n_teacher.yaml precedent; keep the original header below for provenance.
The 365-day default peaks at ~65 GB RSS over the 64,892-reach eval network; the 2026-07-23 synthetic-n teacher run was OOM-killed at the end of chunk 1 on the 93 GB desktop (54 GB anon RSS + ambient apps). --chunk-days 180 halves the peak at the cost of doubling disagg boundary-artifact density (0.55% -> 1.1% of days over 29 years). State continuity across chunks is exact either way; the donor parity test still passes (max abs diff 6e-5 m3/s). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…bility Also commits the implementation plan doc that was left untracked. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…d (T15) traps Both hit during the 2026-07-23..29 synthetic-n campaign execution. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Experiment paused after arm 1/4 (machine reallocated to unit-catchment runs). Records teacher execution, the OOM and transient-icechunk incidents, and the distributed arm's preview recovery (n corr 0.736, slope sign recovered but attenuated) with the notebook + 7 PNGs. S1-S5 verdicts remain pending on the 3 unfinished arms. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner
Author
|
Interim artifacts pushed (experiment paused after arm 1/4 — machine reallocated to unit-catchment runs):
S1–S5 verdicts still require the 3 remaining arms ( 🤖 Generated with Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Implements the synthetic-n recoverability experiment (spec:
docs/superpowers/specs/2026-07-22-synthetic-n-recoverability-design.md, plan:docs/superpowers/plans/2026-07-22-synthetic-n-recoverability.md): a ground-truth-anchored synthetic-twin control testing whether learned Manning's n absorbs Q'-source bias while channel geometry (q_spatial/p_spatial) does not, forced by this campaign's 4 real Q' stores.Code changes
probe_zeta_gradient --mode teachergains an optional n/q/p donor override (--donor-params-nc): reuses eval-loss mode'sload_comid_field/gather_by_comid/RoutingParamOverridemachinery so teacher mode can generate synthetic-twin ground truth for the routing parameters, not just leakance.--plant-file/--zeta-output/use_leakanceare now independent of it.--chunk-daysflag on teacher mode (default 365 = old behavior). The 365-day chunk peaks at ~65 GB RSS over the 64,892-reach eval network; the first teacher launch was OOM-killed on the 93 GB desktop. 180-day chunks halve the peak (verified: ~45 GB in-chunk, ~4 GB at boundaries) at a negligible boundary-artifact cost (0.55% → 1.1% of days).tests/teacher_donor_override_parity.rs: parity gate — injecting a checkpoint's owndump_parametersoutput as the donor must reproduce the un-overridden synthetic obs (passes at 6e-5 m³/s max diff vs the 1e-3 tolerance).synthetic_n_consensus_geometry.py(per-COMID median q/p across the 4 campaign checkpoints),synthetic_n_truth_fields.py(Leopold-Maddock + Gaussian-null truth-n donors),synthetic_n_recoverability_analysis.py(pre-registered S1–S5 verdicts).synthetic_n_teacher.yaml(full 1981–2010 window, standard benchmark Q' store) + 4 student configs (exact copies of the campaign arms, observations repointed at the synthetic obs store).No changes to
src/routing/,src/geometry.rs,src/sparse.rs, or anyBackwardimpl —compare_ddr_sandboxblast radius untouched.Experiment status — PAUSED after arm 1/4 (machine reallocated to new unit-catchment runs, 2026-07-29)
distributed: training complete,recovered_distributed.ncdumped (its eval died at chunk 364/366 on a transient icechunk "object not found" — store probes clean; training unaffected)lumped/daily_lstm/hourly_lstm: stopped at user request (lumped was at epoch 1); resume withoutput/synthetic_n/run_students_sequential.sh lumped daily_lstm hourly_lstmPreview (arm 1 only, NOT a verdict — S1–S5 need all 4 arms): distributed recovers the truth-n field partially: median |n error| 0.035, corr(truth, recovered) = 0.74, Leopold-Maddock slope recovered with correct sign but attenuated (−0.019 vs true −0.042). Geometry: median |q error| 0.027, |p error| 5.9.
🤖 Generated with Claude Code