Context
The current validation gates (validate_policy, test_policy) catch structural and syntax errors in generated Rego policies but not semantic incorrectness — a policy that enforces the wrong condition can pass all current gates.
Models below Sonnet-class capability (e.g., Haiku) may skip MCP tool calls and generate from training data instead of catalog content, producing plausible-looking but incorrect output.
Without explicit approval steps, is it easier to mistake this tool for an autonomous workflow and not the human-in-loop authoring workflow is supposed to be.
Proposal
Add a test-first workflow to the pack-assessment skill. Test cases are generated from MCP-sourced requirements and presented to the user for review before any policy is written. The user reviews pass/fail scenarios in domain terms rather than Rego syntax.
Acceptance Criteria
Context
The current validation gates (
validate_policy,test_policy) catch structural and syntax errors in generated Rego policies but not semantic incorrectness — a policy that enforces the wrong condition can pass all current gates.Models below Sonnet-class capability (e.g., Haiku) may skip MCP tool calls and generate from training data instead of catalog content, producing plausible-looking but incorrect output.
Without explicit approval steps, is it easier to mistake this tool for an autonomous workflow and not the human-in-loop authoring workflow is supposed to be.
Proposal
Add a test-first workflow to the pack-assessment skill. Test cases are generated from MCP-sourced requirements and presented to the user for review before any policy is written. The user reviews pass/fail scenarios in domain terms rather than Rego syntax.
Acceptance Criteria