ci: run AgentMeasure conformance on the two urusilla vectors - #15
ci: run AgentMeasure conformance on the two urusilla vectors#15roy-tong wants to merge 4 commits into
Conversation
Turns the two fixtures already maintained for conformance passes (001 retry/cache boundary, 002 multi-operation grain) into a push/PR check via the AgentMeasure conformance Action. Reports PASS/FAIL/ UNPROVABLE per invariant; no new semantics.
jaden3824
left a comment
There was a problem hiding this comment.
Thank you for turning the two existing fixtures into a bounded CI proposal and for pinning AgentMeasure to a full commit SHA.
Before this workflow runs in the repository, please match the repository's existing workflow hardening:
- Add top-level
permissions: contents: read. - Replace
actions/checkout@v4with the repository-pinnedactions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.
I can verify the two fixture paths exist and that the AgentMeasure action is pinned, but there is currently no visible agentmeasure-conformance check result on this PR, so the fixture verdict remains unverified until the workflow actually runs. This review is about CI safety and observability only; it does not treat the PR as adoption, independent reproduction, or performance evidence.
|
Both hardening changes are in: One note since this workflow is newly introduced from a fork: GitHub may require maintainer approval before workflow runs execute on fork PRs. Once approved, the two fixture verdicts should become visible here — vector 001 is the retry/cache boundary that caught #8/#9, vector 002 is the multi-operation grain case (AM-U-007), so both should report PASS/UNPROVABLE per invariant rather than silently passing. |
|
Thanks — the permissions scope and checkout pin are correct. However, commit |
|
One additional bounded audit before workflow approval: I ran the exact full AgentMeasure commit in a sanitized offline environment against both unchanged fixtures. Both returned exit 0 and PASS for the five currently supported invariants: execution-grain, retry-reconciliation, cost-preservation, operation-grain, and evidence-boundary. Please make these final small changes:
Scope note: this action checks the generic AgentMeasure invariants only. |
|
All three items from the audit are in the branch head as of 9d85ecb:
Ready for re-review whenever convenient; no rush on the workflow approval. |
This just turns the two fixtures you've already been running manually into a push/PR check — no new semantics.
What it does: on every push and PR, the AgentMeasure conformance Action runs against
agentmeasure_urusilla_fixture_001.events.jsonland_002, reporting PASS / FAIL / UNPROVABLE per invariant (execution grain, retry reconciliation, cost preservation, operation grain, evidence boundary). Vector 001 is the retry/cache boundary that caught #8/#9; vector 002 is the multi-operation shape that caught the grain-mixing defect (AM-U-007) — so this also pins both of those fixes permanently against your inputs.If the checks aren't useful here, close freely — no hard feelings either way. And thank you again; a repeat external contributor who found three real defects is the best thing that has happened to this project.