Add a reusable, traceable review framework - #14
Conversation
|
Warning Review limit reached
Next review available in: 18 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Free Run ID: 📒 Files selected for processing (6)
📝 WalkthroughWalkthroughAdds an ChangesReview framework
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant CLI
participant ReviewPipeline
participant Harbor
participant RunDirectory
CLI->>ReviewPipeline: load task and resolve base/head refs
ReviewPipeline->>Harbor: run review in detached worktree
Harbor->>RunDirectory: write report, trajectory, and result artifacts
ReviewPipeline->>RunDirectory: validate and collect artifacts
RunDirectory-->>CLI: return verdict and output paths
Note 🎁 Summarized by CodeRabbit FreeYour organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/login. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 97e290a1d1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| raise RuntimeError(f"missing {label}: {path}") | ||
| _validate_trajectory(trajectory_path) | ||
| report = ReviewReport.load(report_path, task) | ||
| resolved_wall = wall_s if wall_s is not None else _result_wall_s(result_path) |
There was a problem hiding this comment.
For every successful review, the initial collection records run_owned().wall_s, but a later collect_review_run(run_dir) recomputes the value from the trial's started_at and finished_at. Those measure different intervals because the process duration includes Harbor startup and shutdown, so merely recovering a run can overwrite run.json with a different wall time and make repeated analysis inconsistent. Use one timing source consistently, or reuse the already persisted wall_s during recovery.
Useful? React with 👍 / 👎.
| if result.returncode != 0: | ||
| raise RuntimeError(f"review agent failed with exit {result.returncode}; artifacts: {run_dir}") |
There was a problem hiding this comment.
When Harbor exits nonzero, this branch raises after saving only stdout and stderr, leaving run.json without the exit code, failure state, or elapsed wall time and making collect_review_run unusable. Agent/setup failures are precisely the runs for which retained operational evidence is needed, so persist the failure metadata before raising rather than producing less trace information than a successful run.
Useful? React with 👍 / 👎.
3 participants
Summary
Validation
uv run ruff check .uv run ty checkuv run pytest -q(379 passed)Usage
uv run evolve review . --base origin/main --head HEADThe generated job can be inspected with Harbor View, and the full task snapshot is retained in
run.json.Summary by CodeRabbit
New Features
evolve reviewcommand.Documentation
Tests