Evaluate the Further-Investment Gate

[RMKruse]

Parent

#35

What to build

Type: AFK.

Build a saved-artifact evaluator for the existing Merit Probe Evidence Contract. It must consume the completed primary and secondary evidence bundles, classify the result as pass, fail, or inconclusive, explain which primary signals and guardrails drove that classification, and produce a bounded diagnostic agenda without making the final human investment decision.

Acceptance criteria

• The evaluator consumes saved run artifacts and the declared evidence contract without retraining models.
• NLL and CRPS are treated as primary distributional signals; coverage, PIT, MAE, and RMSE are treated as guardrails.
• The evaluator verifies that required selected-checkpoint interpretability artifacts are present before a pass is possible.
• Concept Demo and Concept Demo Robustness results are explicitly excluded from the Merit Probe decision.
• Pass, fail, and inconclusive fixture bundles produce the expected classification and a human-readable explanation.
• Fail or inconclusive results produce bounded diagnostic areas covering data, baseline, optimizer behavior, graph stream, scale predictor, and forecast horizon.
• A pass may recommend human consideration of PEMS-BAY as the Replication Gate but cannot automatically authorize it.

Blocked by

• Produce a Multi-Seed METR-LA Evidence Bundle #42
• Add Secondary Distributional and Point Context #43