Skip to content

ci: run AgentMeasure conformance on the two urusilla vectors - #15

Open
roy-tong wants to merge 4 commits into
jaden3824:mainfrom
roy-tong:add-agentmeasure-conformance
Open

ci: run AgentMeasure conformance on the two urusilla vectors#15
roy-tong wants to merge 4 commits into
jaden3824:mainfrom
roy-tong:add-agentmeasure-conformance

Conversation

@roy-tong

@roy-tong roy-tong commented Sep 1, 2026

Copy link
Copy Markdown

This just turns the two fixtures you've already been running manually into a push/PR check — no new semantics.

What it does: on every push and PR, the AgentMeasure conformance Action runs against agentmeasure_urusilla_fixture_001.events.jsonl and _002, reporting PASS / FAIL / UNPROVABLE per invariant (execution grain, retry reconciliation, cost preservation, operation grain, evidence boundary). Vector 001 is the retry/cache boundary that caught #8/#9; vector 002 is the multi-operation shape that caught the grain-mixing defect (AM-U-007) — so this also pins both of those fixes permanently against your inputs.

If the checks aren't useful here, close freely — no hard feelings either way. And thank you again; a repeat external contributor who found three real defects is the best thing that has happened to this project.

Turns the two fixtures already maintained for conformance passes
(001 retry/cache boundary, 002 multi-operation grain) into a push/PR
check via the AgentMeasure conformance Action. Reports PASS/FAIL/
UNPROVABLE per invariant; no new semantics.
@roy-tong
roy-tong requested a review from jaden3824 as a code owner September 1, 2026 02:58

@jaden3824 jaden3824 left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for turning the two existing fixtures into a bounded CI proposal and for pinning AgentMeasure to a full commit SHA.

Before this workflow runs in the repository, please match the repository's existing workflow hardening:

  1. Add top-level permissions: contents: read.
  2. Replace actions/checkout@v4 with the repository-pinned actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.

I can verify the two fixture paths exist and that the AgentMeasure action is pinned, but there is currently no visible agentmeasure-conformance check result on this PR, so the fixture verdict remains unverified until the workflow actually runs. This review is about CI safety and observability only; it does not treat the PR as adoption, independent reproduction, or performance evidence.

@roy-tong

roy-tong commented Sep 2, 2026

Copy link
Copy Markdown
Author

Both hardening changes are in: permissions: contents: read at the workflow level, and actions/checkout pinned to a commit SHA (3a1661c).

One note since this workflow is newly introduced from a fork: GitHub may require maintainer approval before workflow runs execute on fork PRs. Once approved, the two fixture verdicts should become visible here — vector 001 is the retry/cache boundary that caught #8/#9, vector 002 is the multi-operation grain case (AM-U-007), so both should report PASS/UNPROVABLE per invariant rather than silently passing.

@jaden3824

Copy link
Copy Markdown
Owner

Thanks — the permissions scope and checkout pin are correct. However, commit 3a1661c shortened the AgentMeasure reference from the full SHA to @2cf476d. Please restore @2cf476d6f7d0fc45401db5a822e1f12de009ac74. I’ll approve the workflow after that.

@jaden3824

Copy link
Copy Markdown
Owner

One additional bounded audit before workflow approval: I ran the exact full AgentMeasure commit in a sanitized offline environment against both unchanged fixtures. Both returned exit 0 and PASS for the five currently supported invariants: execution-grain, retry-reconciliation, cost-preservation, operation-grain, and evidence-boundary.

Please make these final small changes:

  1. Restore the full pin: roy-tong/AgentMeasure@2cf476d6f7d0fc45401db5a822e1f12de009ac74.
  2. Add require: execution-grain,retry-reconciliation,cost-preservation,operation-grain,evidence-boundary to both AgentMeasure steps, so a future UNPROVABLE result cannot remain green.
  3. As low-cost hardening, add persist-credentials: false to the checkout step.

Scope note: this action checks the generic AgentMeasure invariants only. token-subset and cache-distinction are currently NOT-SUPPORTED, and the action does not evaluate the x_urusilla sidecar or the fixture expected.json contracts. This PR therefore complements rather than replaces the existing fixture-specific validation. After the full pin and required-invariant settings are present, I can approve the workflow run.

@roy-tong

roy-tong commented Sep 7, 2026

Copy link
Copy Markdown
Author

All three items from the audit are in the branch head as of 9d85ecb:

  • full pin restored: roy-tong/AgentMeasure@2cf476d6f7d0fc45401db5a822e1f12de009ac74
  • require: execution-grain,retry-reconciliation,cost-preservation,operation-grain,evidence-boundary on both AgentMeasure steps, so a future UNPROVABLE cannot stay green
  • persist-credentials: false on the pinned checkout (workflow-level permissions: contents: read unchanged)

Ready for re-review whenever convenient; no rush on the workflow approval.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants