feat: add create-pr skill - #27
Conversation
| | Criterion | Result | | ||
| |---|---| | ||
| | Pinned revision and unrelated working-tree boundary | PASS | | ||
| | Problem, behavior, scope, rollout/risk, and verification body structure | PASS | |
There was a problem hiding this comment.
🟡 Eval result claims PASS but 2 of 9 criteria unaddressed.
🤖 Fix with your agent
Fix this code review finding (aictrl-dev/skills PR #27, evals/results.md:16-23):
Problem: Eval result claims PASS but 2 of 9 criteria unaddressed
Detail: The create-pr eval defines 9 pass criteria in evals/create-pr.eval.md, but the results table in evals/results.md has only 5 rows and the method narrative explicitly addresses only 7 of the 9. Two criteria are not evidenced anywhere in the result: (1) "Produces a concise title naming the endpoint behavior rather than files" (evals/create-pr.eval.md:22) and (2) "Keeps the linked issue close keyword when appropriate and preserves only accurate existing metadata" (evals/create-pr.eval.md:31-32) — the close-keyword/metadata criterion is entirely unmentioned despite the scenario containing a linked issue and stale metadata. The result records "Verdict: PASS" without demonstrating these. For a skill whose central thesis is that unverified claims must be marked unknown rather than asserted, recording PASS while leaving its own pass criteria unaddressed undercuts that thesis. Add table rows / narrative evidence for the title-naming and close-keyword/metadata criteria, or explicitly mark them not-applicable with reasoning.
Suggested fix: Add two rows to the results table, e.g. '| Title names endpoint behavior, not files | PASS |' and '| Linked-issue close keyword and existing metadata handled correctly | PASS |' (with the forward test actually evaluating these), or note in the method narrative why a criterion is out of scope.
Implement the fix on the PR head branch and add a regression test that fails before the fix and passes after.
Why this matters
The create-pr eval defines 9 pass criteria in evals/create-pr.eval.md, but the results table in evals/results.md has only 5 rows and the method narrative explicitly addresses only 7 of the 9. Two criteria are not evidenced anywhere in the result: (1) "Produces a concise title naming the endpoint behavior rather than files" (evals/create-pr.eval.md:22) and (2) "Keeps the linked issue close keyword when appropriate and preserves only accurate existing metadata" (evals/create-pr.eval.md:31-32) — the close-keyword/metadata criterion is entirely unmentioned despite the scenario containing a linked issue and stale metadata. The result records "Verdict: PASS" without demonstrating these. For a skill whose central thesis is that unverified claims must be marked unknown rather than asserted, recording PASS while leaving its own pass criteria unaddressed undercuts that thesis. Add table rows / narrative evidence for the title-naming and close-keyword/metadata criteria, or explicitly mark them not-applicable with reasoning.
| Criterion | Result |
|---|---|
| Pinned revision and unrelated working-tree boundary | PASS |
| Problem, behavior, scope, rollout/risk, and verification body structure | PASS |
| Authored rollout and test-result claims kept distinct from configuration and test source | PASS |
| Unknown deployment and execution evidence made explicit | PASS |
| Explicit confirmation required before host mutation | PASS |
Verdict: PASS — the skill produces a reviewer-ready draft that is grounded in
the exact change and honest about what has and has not been verified.
There was a problem hiding this comment.
TRUE · FIX — completed the create-pr eval record so it explicitly covers all nine pass criteria. The forward-test evidence now includes the behavior-focused title and the retained linked-issue close keyword/no unsupported metadata. Verified at ce5778b802d36af3227d7b1f872776dc0a0f159d with npm test (25 passed, 2 renderer-dependent skipped) and npm run validate (passed). Please re-review this focused update.
Code reviewVerdict: Looks good — only minor / nit comments below. · 🔴 0 · 🟠 0 · 🟡 1 · ⚪ 0 · 0/1 resolved
🤖 Fix all 1 open findings with your agent📋 Out-of-diff findings (1)
Reviewed 7 files · 0 inline · view all 1 findings ↗ aictrl · AI code review for fast-moving teams · aictrl.dev |
Code reviewVerdict: Looks good. ✅ · 0 findings Reviewed 7 files · 0 inline · view all 0 findings ↗ aictrl · AI code review for fast-moving teams · aictrl.dev |
Code reviewVerdict: Looks good — only minor / nit comments below. · 🔴 0 · 🟠 0 · 🟡 1 · ⚪ 0 · 0/1 resolved
🤖 Fix all 1 open findings with your agent📋 Out-of-diff findings (1)
Reviewed 20 files · 0 inline · view all 1 findings ↗ aictrl · AI code review for fast-moving teams · aictrl.dev |
Code reviewVerdict: Looks good. ✅ · 0 findings Reviewed 20 files · 0 inline · view all 0 findings ↗ aictrl · AI code review for fast-moving teams · aictrl.dev |
Closes #24
Summary
create-prskill for evidence-backed reviewer-ready PR draftsStatus = Development) only with an explicit mapping and confirmationaictrl.devVerification
python3 /home/bulat/.codex/skills/.system/skill-creator/scripts/quick_validate.py skills/create-prnpm testnpm run validate