Skip to content

feat(registry): run status + supersession provenance model (Phase 1) - #79

Merged
GondekNP merged 4 commits into
mainfrom
feat/registry-provenance-status
Aug 3, 2026
Merged

feat(registry): run status + supersession provenance model (Phase 1)#79
GondekNP merged 4 commits into
mainfrom
feat/registry-provenance-status

Conversation

@GondekNP

@GondekNP GondekNP commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Summary

Implements Phase 1 of REGISTRY_PROVENANCE.md: a first-class run provenance model. Today "current" is an emergent side-effect of a label-renaming hack (on_collision="timestamp"), which is why tags can't tell current from archived and supersession keeps no record of why or what replaced what. This gives each concern one home:

run_hash is what it is · label is what I call it (a pure alias) · tags are what it's about · status is whether I should use it.

"Current" collapses entirely into status = 'active'.

Schema (additive, backward-compatible)

  • New columns on job_configs: status, superseded_by, status_reason, status_updated_at, applied through the existing probe-and-ALTER migration path. NULL status is read as 'active' everywhere via coalescezero behavior change for existing databases (covered by a migration test that opens a pre-status DB).
  • New non-materialized cell_data_current view (active runs only). Created after _init_spatial() so its SELECT c.* captures the geom column when spatial is enabled.

API

  • mark_run(run_hash, status, *, superseded_by=None, reason=None) — closed enum {active, superseded, bad}, enum-validated, with target-exists / no-self-supersede checks. Returning a run to active clears its supersession link.
  • get_run_status(label_or_hash) -> RunStatus (with effective_status / is_current).
  • supersede(new_hash, replaces=, reason=) sugar; run_history(label) walks the superseded_by chain (current → oldest), replacing the removed resolve_latest().
  • label_run: retire on_collision="timestamp" in favor of "supersede" — the old run releases its label (label=NULL) and is marked status='superseded' pointing at the new run, so it self-cleans out of list_labels(). Added a reason arg; force=True unchanged.
  • status surfaced on ConfigInfo and in describe_run; SweepManagerBuilder.with_label(on_collision=, reason=) wired through.

Testing — including real-JAR integration

  • New tests/test_provenance_integration.py (-m integration, real DEV jar, no mocking): runs actual Josh sims through SweepManager, ingests real cell_data, and asserts against it — supersession provenance, label handover, run_history lineage, and that cell_data_current filters a superseded (and a mark_run("bad")) run's real rows while leaving raw cell_data intact. Both pass.
  • Unit tests: replaced the timestamp/resolve_latest tests with status/supersession/migration coverage. Full suite 1206 passed, 24 skipped; -m integration (jshdz + new provenance) pass, MinIO skipped (no server).

Docs — regenerated against the real jar

  • Tutorials (complete-example.qmd, manual-workflow.qmd) and llms-full.txt rewritten to the supersession model; quartodoc reference index (objects.json) regenerated.
  • _freeze regenerated for all four affected tutorials by executing the real sims, and verified figure-by-figure (e.g. complete-example maxGrowth 5/10/15 → ~12/25/37 over 10 steps; the fire-intervention drop at step 5).

Bundled: adapt example models to new josh-jar semantics

Regenerating the freeze surfaced two intended josh behavior changes (confirmed against josh's own llms-full.txt, not regressions) that broke the example models. Fixed here so the docs render correctly:

  1. external reads now index by raw timestep. A bare external X resolves to "current raw timestep as index" (and warns once). The example models read static spatial maps every step, so a bare read walked past the single record on later steps and collapsed growth (flat figures). Pinned to external X at index 0 (explicit, non-warning) in external_sweep.josh, variant_sweep.josh.j2, and timestep_intervention.josh.
  2. Stricter entity-count validation. create ... of now rejects fractional counts; wrapped the scaled density in round() in timestep_intervention.josh.

Follow-ups (not in this PR)

  • Phase 2 (tag currency) and Phase 3 (remote bucket aggregation), per REGISTRY_PROVENANCE.md §10.
  • Open design question (§11): whether status='bad' should also trigger a rerun (a collision_policy concern) vs. purely exclude from reads. This PR does exclude-only.

🤖 Generated with Claude Code

GondekNP and others added 4 commits August 3, 2026 18:20
Give each provenance concern one home instead of overloading `label`:
run_hash is what it is, label is what I call it (a pure alias), tags are
what it's about, and a new `status` column is whether I should use it.
"Current" collapses entirely into status='active'. See
REGISTRY_PROVENANCE.md for the full model.

Schema (additive, backward-compatible):
- Add status / superseded_by / status_reason / status_updated_at columns to
  job_configs via probe-and-ALTER migration. NULL status reads as 'active'
  everywhere via coalesce, so existing databases are unchanged.
- Add non-materialized cell_data_current view (active runs only), created
  after _init_spatial so its SELECT c.* captures the geom column when present.

API:
- mark_run(run_hash, status, superseded_by=, reason=) — closed enum
  {active, superseded, bad}, validated, with link/self-supersede checks.
- get_run_status() -> RunStatus; supersede() sugar; run_history() walks the
  superseded_by chain (replaces resolve_latest).
- label_run: retire on_collision="timestamp" in favor of "supersede" (old run
  releases its label and is marked superseded, pointing at the new run) plus a
  reason arg. force=True unchanged.
- Surface status on ConfigInfo (effective_status/is_current) and in
  describe_run; wire SweepManagerBuilder.with_label(on_collision=, reason=).

Docs/tests: update tutorials + llms-full to the supersession model, regenerate
the quartodoc reference index, and replace the timestamp/resolve_latest tests
with status/supersession/migration coverage.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…cs freeze

Add tests/test_provenance_integration.py: runs real Josh sims through
SweepManager, ingests real cell_data, and verifies the Phase 1 provenance
model against actual data — supersession provenance, label handover,
run_history lineage, and that the cell_data_current view filters a superseded
(and a mark_run 'bad') run's real rows without deleting them. Passes against
the DEV jar.

Regenerate the manual-workflow and complete-example tutorial _freeze caches
(Quarto freeze:true) so the deployed site reflects the supersession-model
edits, and drop the orphaned figures left by the changed chunk numbering.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Regenerating the manual-workflow and complete-example _freeze against the
freshly-fetched DEV jar produced degenerate figures: for the variant_sweep
model (computed max_growth.step + external soil_quality), tree height stops
accumulating across steps (flat lines ~ maxGrowth/4 instead of a running sum).
A plain-config model (tutorial_sweep.josh) still accumulates correctly under
the same jar, so this is a Josh DEV-jar behavior change in how prior.<state>
composes with a computed .step variable, not a joshpy change — and out of scope
for the registry-provenance work.

Keep the .qmd prose/code edits (timestamp -> supersede) but leave the committed
freeze untouched; the tutorials should be re-rendered against a known-good /
pinned jar once the accumulation regression is resolved upstream.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ate freeze

The DEV jar's new `external` read semantics resolve a bare `external X` to
"the current raw timestep as index" (and warn once), instead of a fixed record.
The example models read *static* spatial maps every step, so a bare read now
walks past the single record on later steps and collapses tree growth (flat
curves in the regenerated tutorial figures). Pin these reads to the intended
record with the explicit, non-warning `at index 0` form:

- examples/external_sweep.josh, examples/variant_sweep.josh.j2:
  soil_quality.step = external soil_quality at index 0
- examples/timestep_intervention.josh: initial_density / fire_severity .init reads

Also adapt timestep_intervention to the jar's stricter entity-count validation
(`create ... of` now rejects fractional counts): round the scaled density.

Update the matching inline snippets in the external-data-sweep and
timestep-interventions tutorials, and regenerate the _freeze for all four
tutorials touched this PR (external-data-sweep, timestep-interventions,
complete-example, manual-workflow) against the real jar. Verified the figures
now show correct accumulation (e.g. complete-example maxGrowth 5/10/15 ->
~12/25/37 over 10 steps) and the fire-intervention drop at step 5.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@GondekNP
GondekNP merged commit 10ce674 into main Aug 3, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant