feat(registry): run status + supersession provenance model (Phase 1) - #79
Merged
Conversation
Give each provenance concern one home instead of overloading `label`:
run_hash is what it is, label is what I call it (a pure alias), tags are
what it's about, and a new `status` column is whether I should use it.
"Current" collapses entirely into status='active'. See
REGISTRY_PROVENANCE.md for the full model.
Schema (additive, backward-compatible):
- Add status / superseded_by / status_reason / status_updated_at columns to
job_configs via probe-and-ALTER migration. NULL status reads as 'active'
everywhere via coalesce, so existing databases are unchanged.
- Add non-materialized cell_data_current view (active runs only), created
after _init_spatial so its SELECT c.* captures the geom column when present.
API:
- mark_run(run_hash, status, superseded_by=, reason=) — closed enum
{active, superseded, bad}, validated, with link/self-supersede checks.
- get_run_status() -> RunStatus; supersede() sugar; run_history() walks the
superseded_by chain (replaces resolve_latest).
- label_run: retire on_collision="timestamp" in favor of "supersede" (old run
releases its label and is marked superseded, pointing at the new run) plus a
reason arg. force=True unchanged.
- Surface status on ConfigInfo (effective_status/is_current) and in
describe_run; wire SweepManagerBuilder.with_label(on_collision=, reason=).
Docs/tests: update tutorials + llms-full to the supersession model, regenerate
the quartodoc reference index, and replace the timestamp/resolve_latest tests
with status/supersession/migration coverage.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…cs freeze Add tests/test_provenance_integration.py: runs real Josh sims through SweepManager, ingests real cell_data, and verifies the Phase 1 provenance model against actual data — supersession provenance, label handover, run_history lineage, and that the cell_data_current view filters a superseded (and a mark_run 'bad') run's real rows without deleting them. Passes against the DEV jar. Regenerate the manual-workflow and complete-example tutorial _freeze caches (Quarto freeze:true) so the deployed site reflects the supersession-model edits, and drop the orphaned figures left by the changed chunk numbering. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Regenerating the manual-workflow and complete-example _freeze against the freshly-fetched DEV jar produced degenerate figures: for the variant_sweep model (computed max_growth.step + external soil_quality), tree height stops accumulating across steps (flat lines ~ maxGrowth/4 instead of a running sum). A plain-config model (tutorial_sweep.josh) still accumulates correctly under the same jar, so this is a Josh DEV-jar behavior change in how prior.<state> composes with a computed .step variable, not a joshpy change — and out of scope for the registry-provenance work. Keep the .qmd prose/code edits (timestamp -> supersede) but leave the committed freeze untouched; the tutorials should be re-rendered against a known-good / pinned jar once the accumulation regression is resolved upstream. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ate freeze The DEV jar's new `external` read semantics resolve a bare `external X` to "the current raw timestep as index" (and warn once), instead of a fixed record. The example models read *static* spatial maps every step, so a bare read now walks past the single record on later steps and collapses tree growth (flat curves in the regenerated tutorial figures). Pin these reads to the intended record with the explicit, non-warning `at index 0` form: - examples/external_sweep.josh, examples/variant_sweep.josh.j2: soil_quality.step = external soil_quality at index 0 - examples/timestep_intervention.josh: initial_density / fire_severity .init reads Also adapt timestep_intervention to the jar's stricter entity-count validation (`create ... of` now rejects fractional counts): round the scaled density. Update the matching inline snippets in the external-data-sweep and timestep-interventions tutorials, and regenerate the _freeze for all four tutorials touched this PR (external-data-sweep, timestep-interventions, complete-example, manual-workflow) against the real jar. Verified the figures now show correct accumulation (e.g. complete-example maxGrowth 5/10/15 -> ~12/25/37 over 10 steps) and the fire-intervention drop at step 5. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements Phase 1 of
REGISTRY_PROVENANCE.md: a first-class run provenance model. Today "current" is an emergent side-effect of a label-renaming hack (on_collision="timestamp"), which is why tags can't tell current from archived and supersession keeps no record of why or what replaced what. This gives each concern one home:"Current" collapses entirely into
status = 'active'.Schema (additive, backward-compatible)
job_configs:status,superseded_by,status_reason,status_updated_at, applied through the existing probe-and-ALTERmigration path.NULLstatus is read as'active'everywhere viacoalesce— zero behavior change for existing databases (covered by a migration test that opens a pre-status DB).cell_data_currentview (active runs only). Created after_init_spatial()so itsSELECT c.*captures thegeomcolumn when spatial is enabled.API
mark_run(run_hash, status, *, superseded_by=None, reason=None)— closed enum{active, superseded, bad}, enum-validated, with target-exists / no-self-supersede checks. Returning a run toactiveclears its supersession link.get_run_status(label_or_hash) -> RunStatus(witheffective_status/is_current).supersede(new_hash, replaces=, reason=)sugar;run_history(label)walks thesuperseded_bychain (current → oldest), replacing the removedresolve_latest().label_run: retireon_collision="timestamp"in favor of"supersede"— the old run releases its label (label=NULL) and is markedstatus='superseded'pointing at the new run, so it self-cleans out oflist_labels(). Added areasonarg;force=Trueunchanged.statussurfaced onConfigInfoand indescribe_run;SweepManagerBuilder.with_label(on_collision=, reason=)wired through.Testing — including real-JAR integration
tests/test_provenance_integration.py(-m integration, real DEV jar, no mocking): runs actual Josh sims throughSweepManager, ingests realcell_data, and asserts against it — supersession provenance, label handover,run_historylineage, and thatcell_data_currentfilters a superseded (and amark_run("bad")) run's real rows while leaving rawcell_dataintact. Both pass.timestamp/resolve_latesttests with status/supersession/migration coverage. Full suite 1206 passed, 24 skipped;-m integration(jshdz + new provenance) pass, MinIO skipped (no server).Docs — regenerated against the real jar
complete-example.qmd,manual-workflow.qmd) andllms-full.txtrewritten to the supersession model; quartodoc reference index (objects.json) regenerated._freezeregenerated for all four affected tutorials by executing the real sims, and verified figure-by-figure (e.g. complete-examplemaxGrowth5/10/15 → ~12/25/37 over 10 steps; the fire-intervention drop at step 5).Bundled: adapt example models to new josh-jar semantics
Regenerating the freeze surfaced two intended josh behavior changes (confirmed against josh's own
llms-full.txt, not regressions) that broke the example models. Fixed here so the docs render correctly:externalreads now index by raw timestep. A bareexternal Xresolves to "current raw timestep as index" (and warns once). The example models read static spatial maps every step, so a bare read walked past the single record on later steps and collapsed growth (flat figures). Pinned toexternal X at index 0(explicit, non-warning) inexternal_sweep.josh,variant_sweep.josh.j2, andtimestep_intervention.josh.create ... ofnow rejects fractional counts; wrapped the scaled density inround()intimestep_intervention.josh.Follow-ups (not in this PR)
REGISTRY_PROVENANCE.md§10.status='bad'should also trigger a rerun (acollision_policyconcern) vs. purely exclude from reads. This PR does exclude-only.🤖 Generated with Claude Code