Synthetic-n recoverability across real Q' sources (teacher/student twin experiment)

.claude/skills/ddrs-debugging-playbook/SKILL.md

@@ -73,6 +73,8 @@ Scan this table first. Each row points to a Part 2 entry with the full story and
 | Resumed run drifts from uninterrupted run | Expected: weights stored as f16 (CompactRecorder) | [T11](#t11-checkpoint-resume-issues) |
 | `ddrs run --strict` exits with code 4 | Source fingerprint drift vs `.ddrs/sources.lock` | [T12](#t12-source-lock-drift) |
 | Recoverability / identifiability experiment fails | Hotstart transient noise floor issue (Phase B not yet met) | [T13](#t13-leakance-identifiability-status) |
+| `probe_zeta_gradient --mode teacher` killed with no error in its own log | 365-day chunk peaks ~65 GB RSS on the 64,892-reach eval network; kernel OOM kill | [T14](#t14-teacher-mode-oom) |
+| Workflow fails `icechunk read failed ... object not found` deep into eval; store probes clean | Transient icechunk read failure, not a data hole | [T15](#t15-transient-icechunk-read-failure) |
 
 ---
 
@@ -460,6 +462,39 @@ cat .ddrs/sources.lock   # shows last-locked fingerprints
 
 ---
 
+### T14: Teacher mode OOM
+
+**Story (2026-07-23).** The synthetic-n teacher run (`probe_zeta_gradient --mode teacher`, full 29-year window, 64,892-reach eval network, CPU backend) died 11 minutes after launch with NOTHING in its own log past the setup lines — no panic, no error. The kernel OOM killer had taken it at 54 GB anon RSS (213 GB total-vm) during its FIRST 365-day chunk; desktop apps (Slack/Hyprland/browser) held ~24 GB of the 93 GB machine and the teacher had the highest oom_score.
+
+**Discriminating test.** Log ends abruptly after `teacher: N plants, ...` with no `chunk k/n` lines and the process is gone:
+```bash
+journalctl -k --since <launch date> | grep -iE 'oom|killed process'
+# → "Out of memory: Killed process <pid> (probe_zeta_grad)"
+```
+
+**Memory profile (measured 2026-07-28):** RSS climbs steadily WITHIN a chunk (transient per-timestep allocations) and collapses back to ~4 GB at every chunk boundary. Peak scales with chunk length: ~65 GB at 365 days, ~45 GB at 180 days.
+
+**Fix:** `--chunk-days 180` (added 2026-07-28; default 365 = old behavior). Cost: disagg boundary-artifact density doubles (0.55% → 1.1% of days over 29 years) — still negligible. State continuity across chunks is exact either way. The 180-day teacher completed the full 59-chunk window in ~14 h wall.
+
+**Caveat:** `run_state_cache` still hardcodes 365; if you ever pair a state cache with a teacher run, their chunk lengths must match (state boundaries align).
+
+---
+
+### T15: Transient icechunk read failure
+
+**Story (2026-07-28).** The synthetic-n `distributed` student completed all 5 training epochs, then its eval died at chunk 364/366 with `icechunk read failed at .../daily_dhbv_aorc2f_merit_unit_catchments.ic: object not found` (empty context, no panic). Looked like the store ends before 2010-09-30 — it doesn't.
+
+**Why it can't be a data hole:** these icechunk Q' stores are divide-major — `Qr` chunk shape `(200 divides, ALL 14,976 days)`. Any time-slice read touches every divide-block chunk object, so eval chunks 1–363 had already read the exact objects chunk 364 "couldn't find". A direct Python probe of the same date range read clean immediately after. Verdict: transient.
+
+**Triage:**
+1. Probe the store at the failing range (from `~/projects/ddr` venv): `icechunk.Repository.open(icechunk.local_filesystem_storage(<path>))` → read the failing day-slice. Clean read ⇒ transient.
+2. Check `ddrs import <store> --dry-run` — reports the declared time axis and sample-read health.
+3. Only if BOTH fail is it a real store problem.
+
+**Recovery:** training and eval are separate phases — a post-training eval failure leaves the epoch-5 checkpoint valid. Dump parameters from it (`dump_parameters --checkpoint <ckpt>/head`) and re-run eval-only later (legacy `eval` binary) if the diagnostics matter. Drivers chaining multiple runs should treat "workflow exited nonzero but final checkpoint exists" as continue-with-warning, not abort (see `output/synthetic_n/run_students_sequential.sh`).
+
+---
+
 ## Part 3 — Pre-flight checklist before any training run
 
 Use this before starting a new experiment to prevent the most common traps:

.gitignore

@@ -48,4 +48,5 @@ __pycache__/
 
 .cargo/
 .ddrs/
+.ddrs-synthetic-n-*/
 ddrs.yaml

config/experiments/synthetic_n_student_daily_lstm.yaml

@@ -0,0 +1,104 @@
+# synthetic_n_student_daily_lstm.yaml — synthetic-n recoverability student:
+# GENERATED from lstm_daily_frozen_chunk1.yaml, observations repointed to the
+# synthetic-n teacher's ground-truth store (output/synthetic_n/synthetic_obs_lm)
+# — everything else (Q' source, disagg head, architecture) stays identical to
+# the real campaign arm below.
+# Spec: docs/superpowers/specs/2026-07-22-synthetic-n-recoverability-design.md
+# Companion synthetic-n students: synthetic_n_student_{distributed,lumped,hourly_lstm}.yaml
+#
+# --- original lstm_daily_frozen_chunk1.yaml header, unmodified below ---
+# lstm_daily_frozen_chunk1.yaml — full CONUS routing train against the daily
+# CudaLSTM (NH) Q' forecast, updated to the capacity-boosted frozen disagg
+# head (hidden 16, 2 KanLayers, grid 20, chunk_days=1, warm-started from
+# capacity_chunk1.mpk, FROZEN) — supersedes the older
+# equif_daily_lstm_disagg.yaml arm, which pre-dates the 2026-07-10/12 KAN
+# disagg fixes and used a plain hidden_size:16 head with no pretraining.
+#
+# Both the KAN head and the router run on GPU (sparse_solver: cuda).
+# Companion arm: lstm_hourly_native.yaml (hourly-native Q', no disagg head).
+
+mode: training
+workflow: train-and-test
+geodataset: merit
+device: 0
+seed: 42
+np_seed: 42
+
+data_sources:
+  attributes: /home/tbindas/projects/ddr/data/merit_global_attributes_v2.nc
+  conus_adjacency: /home/tbindas/projects/ddr/data/merit_conus_adjacency.zarr
+  gages_adjacency: /home/tbindas/projects/ddr/data/merit_gages_conus_adjacency.zarr
+  streamflow: /mnt/ssd1/data/icechunk/daily_lstm_merit_unit_catchments.ic
+  observations: /home/tbindas/projects/ddrs/output/synthetic_n/synthetic_obs_lm
+  gages: /home/tbindas/projects/ddr/references/gage_info/gages_3000.csv
+  aorc_precip: /mnt/ssd1/data/aorc/merit_unit_catchments.zarr
+
+experiment:
+  batch_size: 64
+  start_time: 1981/10/01
+  end_time: 1995/09/30
+  epochs: 5
+  rho: 90
+  shuffle: true
+  warmup: 5
+  learning_rate:
+    1: 0.001
+    3: 0.0005
+  grad_clip_max_norm: 1.0
+
+kan_head:
+  hidden_size: 21
+  num_hidden_layers: 2
+  grid: 50
+  k: 2
+  input_var_names:
+    - SoilGrids1km_clay
+    - aridity
+    - meanelevation
+    - meanP
+    - NDVI
+    - meanslope
+    - log10_uparea
+    - SoilGrids1km_sand
+    - ETPOT_Hargr
+    - Porosity
+  learnable_parameters:
+    - n
+    - q_spatial
+    - p_spatial
+  # Capacity-boosted disagg head — architecture MUST match capacity_chunk1.mpk
+  # (pretrain_disagg_capacity --chunk-days 1: hidden 16, 2 layers, grid 20, k 3).
+  disaggregation:
+    hidden_size: 16
+    num_hidden_layers: 2
+    grid: 20
+    k: 3
+    boundary_blend: 0.0
+    chunk_days: 1
+    pretrained_checkpoint: /home/tbindas/projects/ddrs/output/disagg_pretrain/capacity_chunk1.mpk
+    freeze: true
+
+params:
+  parameter_ranges:
+    n: [0.015, 0.25]
+    q_spatial: [0.0, 1.0]
+    p_spatial: [1.0, 200.0]
+  attribute_minimums:
+    discharge: 1.0e-4
+    slope: 1.0e-3
+    velocity: 0.01
+    depth: 0.01
+    bottom_width: 0.01
+  defaults:
+    p_spatial: 21.0
+  log_space_parameters:
+    - p_spatial
+  sparse_solver: cuda
+  use_cuda_graphs: false
+  use_leakance: false
+
+testing:
+  start_time: 1995/10/01
+  end_time: 2010/09/30
+  batch_size: 15
+  rho: null

config/experiments/synthetic_n_student_distributed.yaml

@@ -0,0 +1,105 @@
+# synthetic_n_student_distributed.yaml — synthetic-n recoverability student:
+# GENERATED from aorc2f_distributed_frozen_chunk1.yaml, observations repointed
+# to the synthetic-n teacher's ground-truth store (output/synthetic_n/
+# synthetic_obs_lm) — everything else (Q' source, disagg head, architecture)
+# stays identical to the real campaign arm below.
+# Spec: docs/superpowers/specs/2026-07-22-synthetic-n-recoverability-design.md
+# Companion synthetic-n students: synthetic_n_student_{lumped,daily_lstm,hourly_lstm}.yaml
+#
+# --- original aorc2f_distributed_frozen_chunk1.yaml header, unmodified below ---
+# aorc2f_distributed_frozen_chunk1.yaml — full CONUS routing train against the
+# distributed+UH-routing dHBV AORC2F v2 Q' forecast (upstream checkpoint
+# CONUS2717_AORC2F_v2 ep69 — a DDR/PyTorch model that produced this store's
+# Q' data; ddrs only reads the resulting streamflow, it does not load that
+# checkpoint). Same capacity-boosted frozen disagg head as
+# kan_disagg_conus_frozen_chunk1.yaml (hidden 16, 2 KanLayers, grid 20,
+# chunk_days=1, warm-started from capacity_chunk1.mpk, FROZEN).
+#
+# Both the KAN head and the router run on GPU (sparse_solver: cuda).
+# Companion arm: aorc2f_lumped_frozen_chunk1.yaml (same settings, lumped Q').
+
+mode: training
+workflow: train-and-test
+geodataset: merit
+device: 0
+seed: 42
+np_seed: 42
+
+data_sources:
+  attributes: /home/tbindas/projects/ddr/data/merit_global_attributes_v2.nc
+  conus_adjacency: /home/tbindas/projects/ddr/data/merit_conus_adjacency.zarr
+  gages_adjacency: /home/tbindas/projects/ddr/data/merit_gages_conus_adjacency.zarr
+  streamflow: /mnt/ssd1/data/icechunk/daily_dhbv_aorc2f_merit_unit_catchments.ic
+  observations: /home/tbindas/projects/ddrs/output/synthetic_n/synthetic_obs_lm
+  gages: /home/tbindas/projects/ddr/references/gage_info/gages_3000.csv
+  aorc_precip: /mnt/ssd1/data/aorc/merit_unit_catchments.zarr
+
+experiment:
+  batch_size: 64
+  start_time: 1981/10/01
+  end_time: 1995/09/30
+  epochs: 5
+  rho: 90
+  shuffle: true
+  warmup: 5
+  learning_rate:
+    1: 0.001
+    3: 0.0005
+  grad_clip_max_norm: 1.0
+
+kan_head:
+  hidden_size: 21
+  num_hidden_layers: 2
+  grid: 50
+  k: 2
+  input_var_names:
+    - SoilGrids1km_clay
+    - aridity
+    - meanelevation
+    - meanP
+    - NDVI
+    - meanslope
+    - log10_uparea
+    - SoilGrids1km_sand
+    - ETPOT_Hargr
+    - Porosity
+  learnable_parameters:
+    - n
+    - q_spatial
+    - p_spatial
+  # Capacity-boosted disagg head — architecture MUST match capacity_chunk1.mpk
+  # (pretrain_disagg_capacity --chunk-days 1: hidden 16, 2 layers, grid 20, k 3).
+  disaggregation:
+    hidden_size: 16
+    num_hidden_layers: 2
+    grid: 20
+    k: 3
+    boundary_blend: 0.0
+    chunk_days: 1
+    pretrained_checkpoint: /home/tbindas/projects/ddrs/output/disagg_pretrain/capacity_chunk1.mpk
+    freeze: true
+
+params:
+  parameter_ranges:
+    n: [0.015, 0.25]
+    q_spatial: [0.0, 1.0]
+    p_spatial: [1.0, 200.0]
+  attribute_minimums:
+    discharge: 1.0e-4
+    slope: 1.0e-3
+    velocity: 0.01
+    depth: 0.01
+    bottom_width: 0.01
+  defaults:
+    p_spatial: 21.0
+  log_space_parameters:
+    - p_spatial
+  sparse_solver: cuda
+  use_cuda_graphs: false
+  use_leakance: false
+
+testing:
+  start_time: 1995/10/01
+  end_time: 2010/09/30
+  batch_size: 15
+  rho: null

config/experiments/synthetic_n_student_hourly_lstm.yaml

@@ -0,0 +1,91 @@
+# synthetic_n_student_hourly_lstm.yaml — synthetic-n recoverability student:
+# GENERATED from lstm_hourly_native.yaml, observations repointed to the
+# synthetic-n teacher's ground-truth store (output/synthetic_n/synthetic_obs_lm)
+# — everything else (Q' source, no-disagg-head architecture) stays identical
+# to the real campaign arm below.
+# Spec: docs/superpowers/specs/2026-07-22-synthetic-n-recoverability-design.md
+# Companion synthetic-n students: synthetic_n_student_{distributed,lumped,daily_lstm}.yaml
+#
+# --- original lstm_hourly_native.yaml header, unmodified below ---
+# lstm_hourly_native.yaml — full CONUS routing train against the hourly-native
+# MTS-LSTM (NH) Q' forecast. No disaggregation block — the store is already
+# hourly-native, and kan_head.disaggregation + an hourly-native source is a
+# hard config-load error (src/data/dataset.rs::validate_disagg_vs_resolution).
+#
+# Both the KAN head and the router run on GPU (sparse_solver: cuda).
+# Companion arm: lstm_daily_frozen_chunk1.yaml (daily Q', frozen disagg head).
+
+mode: training
+workflow: train-and-test
+geodataset: merit
+device: 0
+seed: 42
+np_seed: 42
+
+data_sources:
+  attributes: /home/tbindas/projects/ddr/data/merit_global_attributes_v2.nc
+  conus_adjacency: /home/tbindas/projects/ddr/data/merit_conus_adjacency.zarr
+  gages_adjacency: /home/tbindas/projects/ddr/data/merit_gages_conus_adjacency.zarr
+  streamflow: /mnt/ssd1/data/icechunk/hourly_lstm_merit_unit_catchments.ic
+  observations: /home/tbindas/projects/ddrs/output/synthetic_n/synthetic_obs_lm
+  gages: /home/tbindas/projects/ddr/references/gage_info/gages_3000.csv
+  aorc_precip: /mnt/ssd1/data/aorc/merit_unit_catchments.zarr
+
+experiment:
+  batch_size: 64
+  start_time: 1981/10/01
+  end_time: 1995/09/30
+  epochs: 5
+  rho: 90
+  shuffle: true
+  warmup: 5
+  learning_rate:
+    1: 0.001
+    3: 0.0005
+  grad_clip_max_norm: 1.0
+
+kan_head:
+  hidden_size: 21
+  num_hidden_layers: 2
+  grid: 50
+  k: 2
+  input_var_names:
+    - SoilGrids1km_clay
+    - aridity
+    - meanelevation
+    - meanP
+    - NDVI
+    - meanslope
+    - log10_uparea
+    - SoilGrids1km_sand
+    - ETPOT_Hargr
+    - Porosity
+  learnable_parameters:
+    - n
+    - q_spatial
+    - p_spatial
+
+params:
+  parameter_ranges:
+    n: [0.015, 0.25]
+    q_spatial: [0.0, 1.0]
+    p_spatial: [1.0, 200.0]
+  attribute_minimums:
+    discharge: 1.0e-4
+    slope: 1.0e-3
+    velocity: 0.01
+    depth: 0.01
+    bottom_width: 0.01
+  defaults:
+    p_spatial: 21.0
+  log_space_parameters:
+    - p_spatial
+  sparse_solver: cuda
+  use_cuda_graphs: false
+  use_leakance: false
+
+testing:
+  start_time: 1995/10/01
+  end_time: 2010/09/30
+  batch_size: 15
+  rho: null