Skip to content

fix(pr-swarm): react to first failing CI check instead of full matrix - #85

Merged
Mathews-Tom merged 4 commits into
mainfrom
fix/pr-swarm-react-to-first-ci-failure
Aug 1, 2026
Merged

fix(pr-swarm): react to first failing CI check instead of full matrix#85
Mathews-Tom merged 4 commits into
mainfrom
fix/pr-swarm-react-to-first-ci-failure

Conversation

@Mathews-Tom

Copy link
Copy Markdown
Owner

Summary

pr-swarm lanes previously treated CI/feedback state too passively:

  • On launch, a lane only checked state/mergeable/mergeStateStatus — it never looked at whether the PR already had failing checks or open review feedback, and nothing told it to act on that immediately.
  • The watch loop was a flat "push, wait ~10 minutes, re-check everything" cycle keyed off the run-level aggregate, so a lane sat idle even after an individual check had already failed.
  • Neither the skill nor its lane prompt said anything about gh run rerun requiring repo admin rights — a lane could plausibly read "can't force a fresh CI run" as a reason to stall, when the normal fix-and-push cycle never depends on that command at all.

This PR makes lanes react to what's already visible instead of waiting on a clean baseline or a full CI matrix, and documents why an admin-gated rerun is a non-issue for the normal flow.

What changed, by commit

fix(pr-swarm): react to first failing CI check instead of full matrixreferences/lane-prompt-template.md, references/verification-gates.md

  • Lane ORIENTATION now runs gh pr checks {PR_NUMBER} alongside gh pr view, and instructs the lane to treat a PR that already has failing checks or unresolved feedback at launch as normal starting state — fix it as the first action, don't wait for a clean baseline.
  • WATCH LOOP rewritten from waiting the full ~10-minute quiet window before checking anything, to polling per-job status (gh pr checks) and reacting to the first individually-failing check or new feedback, without waiting for the rest of that run's still-pending checks. The passive quiet-window wait now only applies once nothing is currently actionable.
  • New verification-gates.md section "React to the first failing check, not the full matrix": explains gh pr checks is per-job, not run-level, and that a lane rarely needs to force a fresh CI run at all since pushing a fix triggers one automatically — gh run rerun (admin-gated) is reserved for the narrow flaky-retest case, not the default response to red CI.

docs(pr-swarm): note first-failure reaction in skill and bump versionSKILL.md

  • Phase 6 (Monitor & Verify) now states that a lane inheriting red CI or open feedback at launch is normal input, not a swarm-level stop condition — Phase 2's independence check is what actually gates the swarm, not the target PR's current health.
  • Adds a matching row to the Error Handling table and a bullet to Red Flags.
  • Updates the verification-gates.md reference-table description to mention the new first-failure-reaction content.
  • Bumps metadata.version 1.0.0 → 1.1.0 for the behavioral change.

test(pr-swarm): cover pre-existing CI failure at swarm launchevals/cases.yaml

  • New case pr_with_existing_ci_failures: a swarm request where one PR already has failing checks and open feedback at launch. Asserts the skill still activates and the response treats the existing state as immediate work, not a blocker.

chore(pr-swarm): regenerate manifest for version bumpmanifest.yaml

  • scripts/generate_manifest.py output reflecting the pr-swarm version bump. No other manifest content changed.

Verification

  • uv run python scripts/generate_manifest.py — regenerated cleanly, diff limited to the version bump line.
  • uv run python scripts/validate_evals.py — all packages pass, including the new eval case (73 skills validated).
  • uv run python scripts/evaluate_package.py --path skills/pr-swarm — 96/100 (PASS), unchanged from before this change.
  • python -c "import yaml; yaml.safe_load(open('skills/pr-swarm/evals/cases.yaml'))" — new eval case parses correctly (6 cases total).

Skill Evaluator Results

Package: pr-swarm (skill)
  D1 Frontmatter Quality:            20/20
  D2 Trigger Coverage:               18/18
  D3 Structural Completeness:        20/20
  D4 Content Depth:                  18/22
  D5 Consistency:                    12/12
  D6 Compliance:                       8/8
  Overall:                          96/100 (96%)
  Status: PASS

Checklist

  • SKILL.md has valid YAML frontmatter with name and description
  • Skill name is kebab-case, under 64 characters
  • Description is 200-1024 characters with trigger phrases and "Use when" clause
  • No angle brackets or pushy language in description
  • No secrets, credentials, or internal URLs in any file
  • Tested locally (scripts/validate_evals.py, scripts/evaluate_package.py)
  • All file references in SKILL.md resolve to existing files
  • Skill evaluator score is 70% or above (96%)
  • No CRITICAL or HIGH findings from skill evaluator

Lanes previously waited a flat ~10-minute quiet window before re-checking anything, and could read an already-red PR or an admin-gated `gh run rerun` as a reason to stall. Neither is necessary: pushing a fix always triggers a fresh CI run on its own.

- Lane orientation now runs `gh pr checks` alongside `gh pr view` and treats a PR that already has failing checks or open feedback at launch as normal starting state to fix immediately, not a reason to wait.
- The watch loop polls per-job status instead of only the run-level aggregate, and reacts to the first individually-failing check or new feedback rather than waiting for every check in a run to finish.
- verification-gates.md documents the per-job polling rationale and clarifies `gh run rerun` (which requires admin rights) is not part of the normal fix-and-push cycle.
Ties the reference-doc behavior change into the top-level skill: Phase 6 now states that an inherited red PR at launch is normal input, not a swarm-level stop condition, and points at the independence check as the actual gate. Adds matching Error Handling and Red Flags entries, and updates the verification-gates.md reference-table summary.

Bumps metadata version 1.0.0 -> 1.1.0 for the behavioral change.
Adds an eval case asserting the skill still activates when a targeted PR already has failing checks and open feedback, and that the response treats it as immediate work rather than a blocker.
Output of scripts/generate_manifest.py after the pr-swarm SKILL.md version bump.
@Mathews-Tom
Mathews-Tom merged commit d9ef25b into main Aug 1, 2026
3 checks passed
@Mathews-Tom
Mathews-Tom deleted the fix/pr-swarm-react-to-first-ci-failure branch August 1, 2026 13:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant