Skip to content

feat: add create-pr skill - #27

Merged
byapparov merged 5 commits into
mainfrom
feat/public-create-pr-skill
Aug 11, 2026
Merged

feat: add create-pr skill#27
byapparov merged 5 commits into
mainfrom
feat/public-create-pr-skill

Conversation

@byapparov

@byapparov byapparov commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Closes #24

Summary

  • add the portable create-pr skill for evidence-backed reviewer-ready PR drafts
  • require a resolved branch route, promotion path, and linked issue before PR mutation
  • provision documented issue/project metadata (for example Status = Development) only with an explicit mapping and confirmation
  • keep authored claims, test sources, executed checks, configuration, and deployed state distinct
  • add behavioral evals and forward-test evidence for rollout/test claims, unrelated local work, branch strategy, and missing issue links
  • normalize public Markdown documentation branding to aictrl.dev

Verification

  • python3 /home/bulat/.codex/skills/.system/skill-creator/scripts/quick_validate.py skills/create-pr
  • npm test
  • npm run validate

Comment thread evals/results.md
| Criterion | Result |
|---|---|
| Pinned revision and unrelated working-tree boundary | PASS |
| Problem, behavior, scope, rollout/risk, and verification body structure | PASS |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Eval result claims PASS but 2 of 9 criteria unaddressed.

🤖 Fix with your agent
Fix this code review finding (aictrl-dev/skills PR #27, evals/results.md:16-23):

Problem: Eval result claims PASS but 2 of 9 criteria unaddressed
Detail: The create-pr eval defines 9 pass criteria in evals/create-pr.eval.md, but the results table in evals/results.md has only 5 rows and the method narrative explicitly addresses only 7 of the 9. Two criteria are not evidenced anywhere in the result: (1) "Produces a concise title naming the endpoint behavior rather than files" (evals/create-pr.eval.md:22) and (2) "Keeps the linked issue close keyword when appropriate and preserves only accurate existing metadata" (evals/create-pr.eval.md:31-32) — the close-keyword/metadata criterion is entirely unmentioned despite the scenario containing a linked issue and stale metadata. The result records "Verdict: PASS" without demonstrating these. For a skill whose central thesis is that unverified claims must be marked unknown rather than asserted, recording PASS while leaving its own pass criteria unaddressed undercuts that thesis. Add table rows / narrative evidence for the title-naming and close-keyword/metadata criteria, or explicitly mark them not-applicable with reasoning.
Suggested fix: Add two rows to the results table, e.g. '| Title names endpoint behavior, not files | PASS |' and '| Linked-issue close keyword and existing metadata handled correctly | PASS |' (with the forward test actually evaluating these), or note in the method narrative why a criterion is out of scope.

Implement the fix on the PR head branch and add a regression test that fails before the fix and passes after.
Why this matters

The create-pr eval defines 9 pass criteria in evals/create-pr.eval.md, but the results table in evals/results.md has only 5 rows and the method narrative explicitly addresses only 7 of the 9. Two criteria are not evidenced anywhere in the result: (1) "Produces a concise title naming the endpoint behavior rather than files" (evals/create-pr.eval.md:22) and (2) "Keeps the linked issue close keyword when appropriate and preserves only accurate existing metadata" (evals/create-pr.eval.md:31-32) — the close-keyword/metadata criterion is entirely unmentioned despite the scenario containing a linked issue and stale metadata. The result records "Verdict: PASS" without demonstrating these. For a skill whose central thesis is that unverified claims must be marked unknown rather than asserted, recording PASS while leaving its own pass criteria unaddressed undercuts that thesis. Add table rows / narrative evidence for the title-naming and close-keyword/metadata criteria, or explicitly mark them not-applicable with reasoning.

| Criterion | Result |
|---|---|
| Pinned revision and unrelated working-tree boundary | PASS |
| Problem, behavior, scope, rollout/risk, and verification body structure | PASS |
| Authored rollout and test-result claims kept distinct from configuration and test source | PASS |
| Unknown deployment and execution evidence made explicit | PASS |
| Explicit confirmation required before host mutation | PASS |

Verdict: PASS — the skill produces a reviewer-ready draft that is grounded in
the exact change and honest about what has and has not been verified.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TRUE · FIX — completed the create-pr eval record so it explicitly covers all nine pass criteria. The forward-test evidence now includes the behavior-focused title and the retained linked-issue close keyword/no unsupported metadata. Verified at ce5778b802d36af3227d7b1f872776dc0a0f159d with npm test (25 passed, 2 renderer-dependent skipped) and npm run validate (passed). Please re-review this focused update.

@aictrl-dev

aictrl-dev Bot commented Aug 11, 2026

Copy link
Copy Markdown

Code review

Verdict: Looks good — only minor / nit comments below. · 🔴 0 · 🟠 0 · 🟡 1 · ⚪ 0 · 0/1 resolved

  • 🟡 evals/results.md:16-23 — Eval result claims PASS but 2 of 9 criteria unaddressed
🤖 Fix all 1 open findings with your agent
Fix the following code review findings on aictrl-dev/skills PR #27 (head branch).
Run the relevant tests/linters after each change.

1. evals/results.md:16-23 — Eval result claims PASS but 2 of 9 criteria unaddressed
   Detail: The create-pr eval defines 9 pass criteria in evals/create-pr.eval.md, but the results table in evals/results.md has only 5 rows and the method narrative explicitly addresses only 7 of the 9. Two criteria are not evidenced anywhere in the result: (1) "Produces a concise title naming the endpoint behavior rather than files" (evals/create-pr.eval.md:22) and (2) "Keeps the linked issue close keyword when appropriate and preserves only accurate existing metadata" (evals/create-pr.eval.md:31-32) — the close-keyword/metadata criterion is entirely unmentioned despite the scenario containing a linked issue and stale metadata. The result records "Verdict: PASS" without demonstrating these. For a skill whose central thesis is that unverified claims must be marked unknown rather than asserted, recording PASS while leaving its own pass criteria unaddressed undercuts that thesis. Add table rows / narrative evidence for the title-naming and close-keyword/metadata criteria, or explicitly mark them not-applicable with reasoning.
   Suggested fix: Add two rows to the results table, e.g. '| Title names endpoint behavior, not files | PASS |' and '| Linked-issue close keyword and existing metadata handled correctly | PASS |' (with the forward test actually evaluating these), or note in the method narrative why a criterion is out of scope.
📋 Out-of-diff findings (1)
Sev Location Finding
🟡 evals/results.md:16-23 Eval result claims PASS but 2 of 9 criteria unaddressed

Reviewed 7 files · 0 inline · view all 1 findings ↗


aictrl · AI code review for fast-moving teams · aictrl.dev

@byapparov byapparov self-assigned this Aug 11, 2026
@aictrl-dev

aictrl-dev Bot commented Aug 11, 2026

Copy link
Copy Markdown

Code review

Verdict: Looks good. ✅ · 0 findings

Reviewed 7 files · 0 inline · view all 0 findings ↗


aictrl · AI code review for fast-moving teams · aictrl.dev

Comment thread skills/create-pr/SKILL.md Outdated
@aictrl-dev

aictrl-dev Bot commented Aug 11, 2026

Copy link
Copy Markdown

Code review

Verdict: Looks good — only minor / nit comments below. · 🔴 0 · 🟠 0 · 🟡 1 · ⚪ 0 · 0/1 resolved

  • 🟡 skills/create-pr/SKILL.md:3 — Spec-vs-issue stance contradicts the mutation gate
🤖 Fix all 1 open findings with your agent
Fix the following code review findings on aictrl-dev/skills PR #27 (head branch).
Run the relevant tests/linters after each change.

1. skills/create-pr/SKILL.md:3 — Spec-vs-issue stance contradicts the mutation gate
   Detail: The skill sends mixed signals about whether a linked issue is required. The frontmatter description (line 3) and "Ground the draft" step 2 (line ~47) both list "linked issue or specification" as alternative grounding inputs, and the eval pass criterion says "Keeps the linked issue close keyword when appropriate"; yet the "Resolve the PR route and linked issue" section (line ~30) mandates a linked issue before any PR mutation: "Resolve the linked issue before preparing a mutation. ... do not create or edit the pull request until it is known." The skill never reconciles these positions, leaving behavior ambiguous for spec-grounded or no-issue PRs (dependency bumps, doc fixes, hotfixes). Either drop "or specification" from the description and step 2 to match the strict mutation gate, or add a clause describing how a spec-only request (or a user-confirmed no-issue PR) is handled at the mutation gate.
   Suggested fix: Reconcile the stance on linked issues: either (a) drop "or specification" from the description and from step 2 of "Ground the draft" so the strict mutation gate in "Resolve the PR route and linked issue" matches the rest of the skill, or (b) keep the spec alternative and add a short clause to the mutation gate describing how a verified spec (or an explicitly user-confirmed no-issue PR such as a deps bump or doc fix) is allowed to proceed.
📋 Out-of-diff findings (1)
Sev Location Finding
🟡 skills/create-pr/SKILL.md:3 Spec-vs-issue stance contradicts the mutation gate

Reviewed 20 files · 0 inline · view all 1 findings ↗


aictrl · AI code review for fast-moving teams · aictrl.dev

@aictrl-dev

aictrl-dev Bot commented Aug 11, 2026

Copy link
Copy Markdown

Code review

Verdict: Looks good. ✅ · 0 findings

Reviewed 20 files · 0 inline · view all 0 findings ↗


aictrl · AI code review for fast-moving teams · aictrl.dev

@byapparov
byapparov merged commit 1759dbc into main Aug 11, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add reviewer-ready pull request writing skill

1 participant