feat(e2e): historian eval lane runner script and CI wiring (U7) - #73
feat(e2e): historian eval lane runner script and CI wiring (U7)#73ahrav wants to merge 1 commit into
Conversation
|
You have reached your Codex usage limits for security reviews. Please try again later. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 064feb2291
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| const { scenarios, releaseVersion } = loadCorpus(args); | ||
| const mode = liveModeFromEnv(); | ||
| const sha = repoCommitSha(); | ||
| const artifactsRoot = join(dirname(resolve(args.reportPath)), "historian-eval-runs"); |
There was a problem hiding this comment.
In the scheduled/dispatch workflow, resolving the report path makes every artifactDir absolute, and runScenario serializes the resulting absolute contextDbSnapshotPath into each run record. After GitHub uploads and an operator downloads the artifact, that runner-local path no longer exists, while scoreRunRecord opens it directly; consequently the archived records cannot provide the documented byte-identical re-scoring. Store a portable path relative to the run record/artifact root and resolve it when replaying.
Useful? React with 👍 / 👎.
| } else { | ||
| throw new Error(`unknown argument: ${arg}`); |
There was a problem hiding this comment.
When an operator follows either documented freeze command using --mutations --evidence <path>, argument parsing reaches this branch and exits with unknown argument: --evidence; runMutations also never writes the evidence artifact. This breaks the README's stated operator workflow for generating reviewable mutation evidence, so either implement the option and serialize the returned battery artifact or remove that option from the documented freeze procedure.
Useful? React with 👍 / 👎.
| function runLint(scenarios: readonly HistorianEvalScenario[]): number { | ||
| const diagnostics = scenarios.flatMap((scenario) => lintScenario(scenario)); | ||
| const families = new Set(scenarios.flatMap((scenario) => scenario.families)); |
There was a problem hiding this comment.
If two development JSON files accidentally declare the same scenario ID, each document can pass lintScenario and the corpus is reported clean because no corpus-level uniqueness check runs here. The default live lane then maps both scenarios to the same historian-eval-runs/<id> directory, and the second iteration deletes and overwrites the first run record while the aggregate report still counts both scores. Reject duplicate IDs before mutation or live execution, as the frozen-release promotion path already does.
Useful? React with 👍 / 👎.
| - name: Install opencode | ||
| run: | | ||
| curl -fsSL https://opencode.ai/install | bash | ||
| echo "$HOME/.opencode/bin" >> "$GITHUB_PATH" |
There was a problem hiding this comment.
Whenever the OpenCode installer begins serving a new release, otherwise identical weekly evaluations start running against a different harness/runtime, but the serialized system tuple records only the repository SHA, model IDs, parser implementation, and chunk budget. Reports from before and after that upgrade therefore appear to have the same system identity even though request routing or session behavior may have changed, undermining longitudinal comparisons. Install a pinned OpenCode release or add its resolved version to the recorded system tuple.
Useful? React with 👍 / 👎.
Code Review SummaryStatus: No Issues Found | Recommendation: Merge Files Reviewed (4 files)
Previous Review Summary (commit 064feb2)Current summary above is authoritative. Previous snapshots are kept for context only. Previous review (commit 064feb2)Status: No Issues Found | Recommendation: Merge Files Reviewed (4 files)
Reviewed by gemini-3.7-flash · Input: 148.4K · Output: 13.6K · Cached: 1M |
run-historian-eval.ts drives three modes: --lint (freeze lint plus corpus-level family coverage), --mutations (the invalid-state battery with per-class output), and --live (scenario runs against the dev split or a frozen release, aggregated into one schema-versioned report with the KTD8 exit mapping: 0 green, 1 red, 2 run-fatal on any false- authoritative FAIL). Per-PR CI runs only the deterministic parts with no credentials (historian-eval-deterministic job); live scenario runs are scheduled or operator-dispatched via historian-eval.yml, where the corpus selector flows through env into a closed case statement rather than template interpolation so a crafted dispatch input cannot execute with the API key in scope. The lane README documents the layout, the raw-output scorer seam for task x4l.13, the operator gates (live prototype run, 3-run stability audits, freeze), and every deviation from the plan discovered at implementation time.
064feb2 to
511d673
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 511d673b9c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| contents: read | ||
|
|
||
| jobs: | ||
| live-lane: |
There was a problem hiding this comment.
A user with permission to dispatch workflows can select a non-default repository branch, and actions/checkout will check out that selected ref before this job exposes HISTORIAN_EVAL_ANTHROPIC_API_KEY. Such a branch can modify the runner or package lifecycle scripts to exfiltrate the secret without merging; gate manual dispatches on github.ref == refs/heads/<default branch> as the repository's reference-host workflow does.
Useful? React with 👍 / 👎.
| const [providerID, ...modelParts] = probeModel.split("/"); | ||
| if (!providerID || modelParts.length === 0) { | ||
| throw new Error("HISTORIAN_EVAL_PROBE_MODEL must be provider/model"); |
There was a problem hiding this comment.
When HISTORIAN_EVAL_PROBE_MODEL is set to a value such as anthropic/, split("/") produces one empty model component, so this check accepts it and the runner performs the expensive historian work before every probe fails through an invalid model route. The historian model is not shape-validated at all. Validate that both configured routes have a nonempty provider and nonempty model portion before iterating over the corpus.
Useful? React with 👍 / 👎.
1 participant
Stack 5/5 for the historian structural eval lane (beads
magic-context-x4l.11). Base: #72 (corpus + governance).run-historian-eval.tsdrives three modes:--lint: freeze lint plus corpus-level hard-negative family coverage.--mutations: the invalid-state battery with per-class output.--live: scenario runs against the dev split or a frozen release (env:ANTHROPIC_API_KEY,HISTORIAN_EVAL_MODEL,HISTORIAN_EVAL_PROBE_MODEL), aggregated into one schema-versioned report with the KTD8 exit mapping: 0 green, 1 red (any FAIL or ERROR), 2 run-fatal (any false-authoritative FAIL).CI split (R14): per-PR runs only the deterministic parts with no credentials (
historian-eval-deterministicjob in ci.yml: unit selection + lint + battery). Live runs are scheduled or operator-dispatched viahistorian-eval.yml; the corpus selector flows throughenvinto a closedcasestatement — never template interpolation — so a crafted dispatch input cannot execute with the API key in scope. Defaults to the dev split until the operator freezesreleases/v1.README documents the layout, the raw-output scorer seam for task x4l.13 (metamorphic lane, now unblocked), the operator gates (U2 live prototype run, U5 3-run stability audits, freeze), and every deviation from the plan discovered at implementation time (verification bridge, claim-memory bootstrap for
magic-context-e1c, matcher-keyed scripting, live-mode probe-payload capture limits, force-band trigger numbers).Stack: 1/5 plumbing → 2/5 contract → 3/5 engine → 4/5 corpus+governance → 5/5 (this)