PRD: Optimizer-trained METR-LA merit probe and further-investment gate

[RMKruse]

Problem Statement

The GraphNAMLSS Concept Demonstrator now gives a Methodologically Sympathetic Colleague a runnable, reader-facing path through the Localized Congestion Spillover Demo Scenario, the local additive predictor, parameter-specific graph contributions, Graph Gate Diagnostics, numerical checks, and the Concept Demo Evidence Boundary. That achieves the communication goal, but it does not answer the research-investment question.

The current deterministic Tracer-Bullet Fit is intentionally a teaching simplification. It cannot satisfy the Merit Probe Training Standard and must not be used to pass or fail the Further-Investment Gate. The project therefore needs a disciplined transition from a validated Concept Demonstrator to an optimizer-trained METR-LA Merit Probe. Without that transition, the repository risks either treating synthetic or simplified results as empirical evidence, or expanding into PEMS-BAY replication, the Primary Experiment Matrix, Bayesian Graph Streams, and application tracks before the central modeling idea has earned further investment.

The immediate user problem is to obtain trustworthy, reproducible, real-data evidence for one narrow decision: whether optimizer-trained GraphNAMLSS shows enough distributional signal over deterministic NAMLSS on METR-LA to justify deeper research work, while preserving the interpretability that motivates the model family.

Solution

Complete one bounded transition from Concept Demonstrator to Merit Probe.

First, run a structured colleague review of the existing Guided Demo Path and freeze the Concept Demonstrator after one targeted feedback round. In parallel, add a multi-seed Concept Demo Robustness Probe that checks whether the synthetic teaching claims remain directionally stable rather than depending on one favorable seed. These activities validate communication and robustness only; they do not produce benchmark evidence.

Next, implement an optimizer-trained GraphNAMLSS path that jointly updates the local additive predictor, shared graph representation, parameter-specific graph heads, and Graph Contribution Gates under a Gaussian location-scale objective. Training must use only the training partition, validation evidence must control model selection or stopping, and the test partition must remain untouched until the candidate is selected. The run must persist a reloadable checkpoint and enough provenance to reproduce the selected candidate.

Then run the METR-LA Merit Probe under the existing evidence contract. The primary comparison is deterministic NAMLSS versus optimizer-trained Tracer-Bullet GraphNAMLSS. Distributional GNN Baseline and Point GNN Reference provide secondary context. The probe must report every seed, an aggregate summary, NLL and CRPS as primary distributional signals, and coverage, PIT, MAE, and RMSE as guardrails. Local Shape Contributions, Graph Contributions, and Graph Gate Diagnostics must remain inspectable.

Finally, evaluate the predeclared Further-Investment Gate and record a human go/no-go decision. A promising result authorizes consideration of PEMS-BAY as the Replication Gate. An unclear or negative result triggers a bounded diagnosis of data, baseline, fitting, graph stream, scale predictor, and horizon choices before any architecture expansion.

User Stories

1. As a GraphNAMLSS researcher, I want the Concept Demonstrator reviewed by a Methodologically Sympathetic Colleague, so that its clarity is tested by someone other than its author.
2. As a Methodologically Sympathetic Colleague, I want a short review guide, so that I can assess the model idea, decomposition, and claim boundary consistently.
3. As a project maintainer, I want feedback captured as concrete observations and actions, so that the Concept Demonstrator receives one focused improvement round rather than indefinite polishing.
4. As a project maintainer, I want the Concept Demonstrator frozen after accepted feedback, so that research work can move to the Merit Probe.
5. As a GraphNAMLSS researcher, I want the Concept Demo Evidence Boundary preserved during review, so that communication feedback is not mistaken for empirical validation.
6. As a model developer, I want the Localized Congestion Spillover Demo Scenario checked across multiple seeds, so that its teaching signal is not seed-fragile.
7. As a model developer, I want the routine-versus-spillover location direction checked across seeds, so that the expected slowdown story remains coherent.
8. As a model developer, I want the routine-versus-spillover scale direction checked across seeds, so that Heteroskedastic Spillover Uncertainty remains visible.
9. As a project maintainer, I want a bounded robustness summary artifact, so that the synthetic checks are reproducible without being presented as benchmark evidence.
10. As a future contributor, I want the robustness probe callable through the project workflow, so that it can be rerun after changes to the Concept Demonstrator.
11. As a GraphNAMLSS researcher, I want an optimizer-trained Tracer-Bullet GraphNAMLSS, so that the METR-LA candidate matches the conceptual architecture closely enough for an investment decision.
12. As a model developer, I want local additive components, the shared graph representation, parameter-specific graph heads, and Graph Contribution Gates updated by the declared objective, so that the candidate is not a fixed post-fit approximation.
13. As a model developer, I want Gaussian location-scale NLL to drive training, so that location and scale are learned under one distributional objective.
14. As a model developer, I want positive scale enforced throughout training and inference, so that every saved prediction defines a valid Gaussian distribution.
15. As a model developer, I want optimization to consume only training observations, so that validation and test evidence remain uncontaminated.
16. As a model developer, I want validation evidence to control stopping or model selection, so that the selected candidate is not chosen using the test partition.
17. As a reviewer, I want the test partition evaluated only after candidate selection, so that the reported Merit Probe result has a defensible evaluation story.
18. As a project maintainer, I want a reloadable selected checkpoint, so that reported predictions and artifacts can be reproduced without retraining.
19. As a project maintainer, I want training provenance saved with the checkpoint, so that seed, environment, objective, optimizer, epoch selection, and resolved settings are auditable.
20. As a model developer, I want restored-checkpoint inference to reproduce saved predictions within tolerance, so that checkpoint artifacts are trustworthy.
21. As a Methodologically Sympathetic Colleague, I want Local Shape Contributions from the optimizer-trained candidate, so that local NAMLSS-style interpretability remains visible.
22. As a Methodologically Sympathetic Colleague, I want parameter-specific Graph Contributions and Graph Gate Diagnostics from the selected checkpoint, so that graph reliance remains inspectable for mu and sigma.
23. As a reviewer, I want graph gates described as diagnostics rather than causal attributions, so that parameter magnitude is not overinterpreted.
24. As a GraphNAMLSS researcher, I want deterministic NAMLSS and optimizer-trained GraphNAMLSS evaluated under one METR-LA protocol, so that the primary comparison changes the model rather than the data recipe.
25. As a GraphNAMLSS researcher, I want the same temporal split, feature policy, scaling policy, missingness policy, horizon, and metric definitions across primary candidates, so that the comparison is interpretable.
26. As a GraphNAMLSS researcher, I want multiple declared seeds reported individually, so that the decision does not depend on selecting the best run.
27. As a GraphNAMLSS researcher, I want aggregate location and dispersion summaries across seeds, so that result stability is visible without hiding per-seed outcomes.
28. As a statistical collaborator, I want NLL and CRPS treated as primary distributional signals, so that the gate reflects the probabilistic modeling claim.
29. As a statistical collaborator, I want coverage and PIT reported as calibration guardrails, so that improvements in one score do not hide distributional failure.
30. As a statistical collaborator, I want MAE and RMSE reported as point-accuracy guardrails, so that distributional improvements do not conceal serious location degradation.
31. As a reviewer, I want the Distributional GNN Baseline shown as secondary distributional context, so that interpretability-preserving modeling can be compared with a less additive graph model.
32. As a reviewer, I want the Point GNN Reference shown only as a point-accuracy guardrail, so that it is not treated as a distributional peer.
33. As a project maintainer, I want one Merit Probe Evidence Bundle, so that metrics, per-seed runs, checkpoints, predictions, interpretability artifacts, and provenance can be reviewed together.
34. As a GraphNAMLSS researcher, I want the Further-Investment Gate evaluated from the predeclared evidence contract, so that the threshold is not invented after seeing results.
35. As a GraphNAMLSS researcher, I want the evaluator to distinguish pass, fail, and inconclusive evidence, so that ambiguity is not forced into a misleading binary result.
36. As a project maintainer, I want a human review step before recording the final investment decision, so that methodological caveats and artifact quality are considered alongside metric summaries.
37. As a future contributor, I want a recorded decision and rationale, so that later work knows whether to replicate, diagnose, or pause.
38. As a GraphNAMLSS researcher, I want an inconclusive or negative result to name bounded diagnostic follow-ups, so that the response is investigation rather than uncontrolled model expansion.
39. As a GraphNAMLSS researcher, I want a promising result to unlock PEMS-BAY only as the next Replication Gate, so that replication follows evidence rather than precedes it.
40. As a project maintainer, I want the Merit Probe to remain separate from the Concept Demo Bundle, so that future readers can distinguish teaching artifacts from research evidence.

Implementation Decisions

• This PRD is the narrow follow-up to the broader Concept Demonstrator and Merit Probe plan. It begins after the Concept Demonstrator is runnable and reader-facing.
• The Colleague Review Contract is a HITL workflow. It records whether a reader can explain the local predictor, graph contribution, mu/sigma separation, and evidence boundary in their own words.
• The Concept Demonstrator receives at most one targeted feedback round under this PRD before being frozen for the Merit Probe phase.
• The Concept Demo Robustness Probe is a separate deep module with a simple interface: run the existing Concept Demonstrator for a declared seed set and emit a bounded summary of scenario and contribution-direction checks.
• Robustness Probe results remain synthetic sanity checks. They must never be included as evidence for the Further-Investment Gate.
• The Optimizer-Trained GraphNAMLSS Trainer is a deep module that owns objective evaluation, parameter updates, validation-based selection or stopping, selected-checkpoint persistence, and training provenance behind one stable experiment-facing interface.
• The optimizer-trained candidate jointly updates local additive components, the shared graph representation, parameter-specific graph heads, and Graph Contribution Gates.
• The first optimizer-trained candidate remains the Tracer-Bullet GraphNAMLSS cell: deterministic weights, shared graph representation, additive graph contribution, neighbor-focused graph stream, Gaussian location-scale outcome, and no Bayesian Graph Stream.
• Gaussian NLL is the training objective. NLL and CRPS remain evaluation metrics; CRPS does not need to be the optimization objective for this PRD.
• Positive scale is enforced through the existing positive link contract during both training and restored-checkpoint inference.
• Training, validation, and test roles are strict: training updates parameters, validation selects or stops, and test is evaluated only after the selected checkpoint is fixed.
• Scaling and any learned preprocessing statistics are fit on the training partition only and reused unchanged for validation and test.
• A selected checkpoint is incomplete without resolved configuration, training seed, optimizer settings, selected epoch, selection criterion, environment provenance, training history, and a reload verification result.
• The checkpoint-loading interface must reproduce saved candidate predictions within a declared numerical tolerance without rerunning optimization.
• Prediction, Local Shape Contribution, Graph Contribution, and Graph Gate artifacts are generated from the selected checkpoint rather than from transient training state.
• The METR-LA Merit Probe Runner is a deep orchestration module that applies one canonical data and evaluation protocol to every declared seed and model role, then records independently reviewable run artifacts.
• The METR-LA Merit Probe Primary Comparison is deterministic NAMLSS versus optimizer-trained Tracer-Bullet GraphNAMLSS.
• Distributional GNN Baseline and Point GNN Reference remain secondary context. Point GNN is not ranked as a distributional peer.
• The Merit Probe uses multiple predeclared seeds. Every seed is retained in the evidence bundle; the system must not publish only the best candidate.
• Aggregate evidence reports the per-seed values plus a compact mean and dispersion summary. Formal publication-grade significance testing is not required by this PRD.
• The Merit Probe Evidence Evaluator is a deep module that consumes saved run artifacts and the existing evidence contract, classifies the result as pass, fail, or inconclusive, and explains which signals and guardrails drove that classification.
• A pass requires a clear distributional signal, preferably NLL or CRPS improvement, no serious guardrail degradation, and preserved interpretability artifacts. Exact numeric thresholds must come from the declared evidence contract or an explicitly recorded pre-run decision, not post-hoc tuning.
• A fail or inconclusive result does not automatically schedule architecture expansion. It produces a bounded diagnostic agenda covering data, deterministic baseline, optimizer behavior, graph stream, scale predictor, and forecast horizon.
• The final research-investment decision is HITL. The automated evaluator prepares evidence; it does not autonomously authorize PEMS-BAY, the Primary Experiment Matrix, or Bayesian work.
• Existing experiment configuration, traffic data, scaling, missingness, metric, artifact, and report contracts should be extended rather than replaced.
• The project glossary and ADR decisions remain authoritative for names, scope boundaries, and evidence claims.

Testing Decisions

• Good tests verify observable scientific and workflow contracts rather than private optimizer implementation details.
• The Colleague Review Contract is validated through a human checklist and recorded feedback, not unit tests.
• The Concept Demo Robustness Probe is tested against a small declared seed set and must emit deterministic, bounded summaries with explicit synthetic-evidence wording.
• Robustness tests verify directional scenario behavior for location and scale without requiring identical floating-point values for every seed.
• Trainer unit tests verify that optimizer steps update the intended parameter groups, the Gaussian training objective is finite, and the positive scale contract always holds.
• Trainer behavior tests verify that a learnable synthetic fixture produces a lower training objective than initialization without asserting a specific internal optimization trajectory.
• Split-discipline tests verify that training consumes only the training partition, validation controls selection, and the test partition is not read before the selected checkpoint is fixed.
• Checkpoint tests verify save, reload, selected-epoch metadata, optimizer and seed provenance, and prediction reproduction within tolerance.
• Interpretability artifact tests verify that Local Shape Contributions, parameter-specific Graph Contributions, and Graph Gate Diagnostics are produced from the restored selected checkpoint.
• Merit Probe Runner integration tests use small synthetic or local fixture artifacts rather than requiring METR-LA downloads or benchmark-scale compute in CI.
• Runner tests verify that every declared seed produces a complete run, that failures are reported rather than silently dropped, and that the same protocol is applied to both primary candidates.
• Evidence aggregation tests verify per-seed retention, aggregate mean and dispersion calculations, model-role labeling, and Point GNN guardrail treatment.
• Further-Investment Gate tests use fixture evidence bundles representing pass, fail, and inconclusive cases and verify the explanation and bounded next-step outcome.
• Claim-boundary tests verify that Concept Demo Robustness results are excluded from the Merit Probe decision and that synthetic results are not described as benchmark evidence.
• Report tests verify that the evidence bundle exposes primary signals, guardrails, interpretability artifacts, provenance, caveats, and the final HITL decision status.
• Prior art includes the repository's existing config validation, traffic fixture, training CLI, metrics, saved artifact, report ordering, scope guard, Concept Demo Bundle, and executed notebook contract tests.

Out of Scope

• Treating colleague feedback or Concept Demo Robustness Probe results as empirical evidence for GraphNAMLSS.
• Replacing the existing Concept Demonstrator with the optimizer-trained Merit Probe workflow.
• Publication-grade hyperparameter optimization or exhaustive architecture search.
• The full Primary Experiment Matrix.
• Per-parameter graph encoders or graph-aware shape-function variants beyond the first Tracer-Bullet GraphNAMLSS cell.
• Bayesian Graph Streams or variational graph weights.
• Structural Uncertainty Experiments.
• PEMS-BAY execution before a promising METR-LA decision; only the decision to unlock the Replication Gate is in scope.
• Spatial Health Graph or other application tracks.
• Completing GAMLSS or BAMLSS External Code Adapters.
• Claiming causal meaning from Graph Contribution Gates or Graph Contribution maps.
• Claiming general GraphNAMLSS superiority from one dataset or from the Further-Investment Gate.
• Automatically making the final research-investment decision without human review.

Further Notes

This PRD follows #27 and operationalizes the remaining Merit Probe half of that plan after the Concept Demonstrator work was completed. It relies on the METR-LA Merit Probe Evidence Contract captured in #29 and the optimizer-trained Merit Probe Training Standard captured in #30 and ADR 0004.

The intended product is not merely a trained model or a benchmark table. It is a trustworthy decision package: a selected, reloadable optimizer-trained candidate; a protocol-matched primary comparison; per-seed and aggregate evidence; preserved interpretability artifacts; explicit caveats; and a recorded human decision about whether GraphNAMLSS deserves replication or diagnosis.

The project should stop after this gate long enough to make the decision. A larger experiment matrix is not a substitute for a clear first signal.