One subdir benchmark for the get_artifact MCP tool via rac resolve <id> corpus/ --json. Conformance cases (GAB- ids): exact-id hit, legacy-alias hit, case-insensitivity, duplicate id → duplicate error shape with every path, unknown id → not-found error shape, and the ADR-007 stable payload fields. Metric: conformance pass-rate; gate = 1.0 with zero tolerance.
Recorded limitation: provenance enrichment is not present in the current CLI resolve payload, so provenance-field conformance is deferred to the MCP-stdio harness workstream rather than silently dropped.
Roadmap: rac/roadmaps/benchmarks/tool-benchmarks.md in rac-core [roadmap:tool-benchmarks].
One subdir benchmark for the
get_artifactMCP tool viarac resolve <id> corpus/ --json. Conformance cases (GAB-ids): exact-id hit, legacy-alias hit, case-insensitivity, duplicate id → duplicate error shape with every path, unknown id → not-found error shape, and the ADR-007 stable payload fields. Metric: conformance pass-rate; gate = 1.0 with zero tolerance.Recorded limitation: provenance enrichment is not present in the current CLI resolve payload, so provenance-field conformance is deferred to the MCP-stdio harness workstream rather than silently dropped.
Roadmap:
rac/roadmaps/benchmarks/tool-benchmarks.mdin rac-core [roadmap:tool-benchmarks].