Skip to content

fix(review): fail closed on incomplete coverage - #35

Merged
pierrick-fonquerne merged 2 commits into
mainfrom
fix/ai-review-incomplete-verdict
Aug 12, 2026
Merged

fix(review): fail closed on incomplete coverage#35
pierrick-fonquerne merged 2 commits into
mainfrom
fix/ai-review-incomplete-verdict

Conversation

@pierrick-fonquerne

@pierrick-fonquerne pierrick-fonquerne commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

What changed

  • add a deterministic INCOMPLETE team verdict whenever coverage gaps remain or the 15-finding verification cap leaves findings unchecked
  • keep NEEDS_WORK authoritative when a confirmed critical finding is already known
  • render fail-closed results when no textual patch or no synthesis is available
  • paginate global bot-comment discovery across every issue-comment page
  • group inline findings by location and upsert them with stable mode/head/location markers
  • delete stale and duplicate inline bot comments within the same mode and head scope
  • update the reviewer documentation with the new verdict and rerun semantics

Why

The coverage work in #33 and #34 made every unreviewed input visible, but the global verdict could still say SHIP after partial coverage. Re-running a review also created duplicate inline comments, a global marker beyond GitHub's first comment page could be missed, and inline findings that disappeared on a later run remained visible.

Impact

A partial review can no longer be presented as shippable. Re-runs on the same head commit converge the global and inline bot comments to the current result: existing comments are updated, new locations are created, and results no longer confirmed by that mode are removed.

Validation

  • cargo fmt --manifest-path tools/ai-review/Cargo.toml -- --check
  • cargo clippy --manifest-path tools/ai-review/Cargo.toml --all-targets -- -D warnings
  • cargo test --manifest-path tools/ai-review/Cargo.toml
  • Rust 1.88: tests and Clippy/check strict
  • 57 Rust tests and 5 workflow policy tests pass
  • HTTP integration fixtures cover page-2 updates and empty-result stale deletion
  • real multi-batch/idempotence smoke: lightshuttle#316, run 31614721376

Merge policy

Keep this pull request in draft until CI, the real rerun/idempotence smoke test, and review are complete. Do not merge without explicit approval.

@pierrick-fonquerne

Copy link
Copy Markdown
Contributor Author

Final smoke validation passed on lightshuttle run 31614721376 against exact SHA 4a55945c74571ba6414838297dd2b28af13dddfb.

  • a 38-finding synthesis capped at 15 rendered INCOMPLETE instead of SHIP;
  • the seed job created scoped stale inline comment 3768039640, and the team upsert deleted it through the real GitHub API;
  • the review mode created three scoped inline comments on the current head, then a same-SHA rerun with zero inline findings removed all three;
  • global bot comment IDs remained stable across reruns;
  • team, review, and describe jobs completed successfully.

The temporary PR will be closed without merge and cleaned up.

@pierrick-fonquerne
pierrick-fonquerne marked this pull request as ready for review August 12, 2026 16:13
@pierrick-fonquerne
pierrick-fonquerne merged commit 99b0d33 into main Aug 12, 2026
4 checks passed
@pierrick-fonquerne
pierrick-fonquerne deleted the fix/ai-review-incomplete-verdict branch August 12, 2026 16:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant