ci: integrate pre-commit hook package and optimize AI code reviewer pipeline

[JayKay24]

Summary

This Pull Request integrates the pre-commit Python package for local Git hooks management and adds robustness, performance, and cost-optimization updates to the automated AI Code Reviewer pipeline.

Key Changes

• 3rdparty/requirements.txt & 3rdparty/user_reqs.lock: Added pre-commit>=3.0.0 to manage git hooks and compiled updated locks.
• .pre-commit-config.yaml: Configured the local hook to run Pants format (./pants fmt) and Pants lint (./pants lint) on all staged files before committing.
• scripts/BUILD: Created target file to include the scripts directory under Pants build system target management.
• scripts/ai_pr_reviewer.py:
  • Refactored ignore parsing to load .gitignore wildcard rules dynamically using the pathspec library.
  • Made the Gemini model name dynamic by reading from the GEMINI_MODEL environment variable.
  • Added early-exit check to stop retrieving PR files once size limit is exceeded.
  • Wrapped API calls (Gemini and GitHub Review) in safety try-except blocks.
  • Handled 403 Forbidden exceptions gracefully to exit with status code 0 on external fork PRs where GITHUB_TOKEN has read-only access.
• .github/workflows/ai-review.yml:
  • Added a concurrency group to auto-cancel outdated runs if a new commit is pushed.
  • Added path filters to trigger the action only on changes to Python, scripts, or build files.
  • Injected GEMINI_MODEL from GitHub Action variables.

Verification

• Verified that ./pants fmt :: and ./pants lint :: pass successfully on the codebase.
• Ran pre-commit run --all-files locally to confirm both hooks execute and pass:

  Pants Format.............................................................Passed
  Pants Lint...............................................................Passed
  

[github-actions]

🤖 AI PR Review Summary

This Pull Request introduces significant enhancements to the project's CI/CD pipeline and code quality enforcement. Key improvements include:

1. Pre-commit Hooks: Integration of pants fmt and pants lint via pre-commit ensures that code is automatically formatted and linted before commits, maintaining a high standard of code quality and consistency.
2. GitHub Actions Optimizations: The ai-review.yml workflow now uses paths filtering to trigger only on relevant file changes and concurrency to prevent redundant runs for the same PR, improving efficiency and reducing resource consumption.
3. Robust AI Reviewer Script: The ai_pr_reviewer.py script has been refined with better error handling, graceful exits for external forks or API failures, and clear handling of large diffs to stay within token limits.
4. Pants Build Integration: The addition of scripts/BUILD properly registers the Python sources with Pants, supporting the new pre-commit setup.

Overall, this PR demonstrates a strong commitment to maintainable code, efficient development workflows, and robust error handling in a CI/CD context.

💡 Key Feedback & Recommendations

1. Python Code Quality: Consistent Blank Lines (PEP8/Ruff)

The changes introduce some inconsistent blank lines, especially around function definitions and multi-line statements. While a formatter (like pants fmt which likely uses Ruff/Black) should handle this, it's good to be aware. For example, some multi-line statements end with an extra blank line, or there are multiple blank lines within a function.

Recommendation: Ensure consistent application of blank lines for readability as per PEP8/Ruff standards (e.g., two blank lines between top-level definitions, one blank line between method definitions and the first line of code). The addition of pants fmt should largely resolve this automatically once run.

Before (Example):

def get_ignore_spec() -> pathspec.PathSpec:
    """Loads .gitignore patterns and appends custom file exclusion wildcards."""
    ignore_patterns = []
    # ...
            print(f"Warning: Failed to parse .gitignore: {e}", file=sys.stderr)
            
    # Custom wildcards for lockfiles and binary assets to skip
    ignore_patterns.extend([
        "*.lock",
        "*.png",
        "*.jpg",
        "*.jpeg",
        "*.zip",
        "*.pdf"
    ])
    
    return pathspec.PathSpec.from_lines("gitwildmatch", ignore_patterns)

After (Suggests more compact extend and removes redundant blank lines):

def get_ignore_spec() -> pathspec.PathSpec:
    """Loads .gitignore patterns and appends custom file exclusion wildcards."""
    ignore_patterns = []
    # ...
            print(f"Warning: Failed to parse .gitignore: {e}", file=sys.stderr)

    # Custom wildcards for lockfiles and binary assets to skip
    ignore_patterns.extend(["*.lock", "*.png", "*.jpg", "*.jpeg", "*.zip", "*.pdf"])

    return pathspec.PathSpec.from_lines("gitwildmatch", ignore_patterns)

Note: The PR itself already applies the ignore_patterns.extend change. The point here is about the surrounding blank lines that might need further alignment with formatter settings.

2. Error Handling & Graceful Exits (Data Engineering Best Practice for CI)

The change from sys.exit(1) to sys.exit(0) for API-related errors (generate_review, post_review) is a deliberate and well-justified design choice. In a data engineering context, having CI/CD steps that are "nice-to-have" but not critical for core functionality often benefit from graceful failure (warning and continuing) to avoid blocking primary development workflows.

Recommendation: While this is a good choice for a non-critical step, ensure that the warnings (print(f"Warning: ...", file=sys.stderr)) are sufficiently clear for debugging purposes if such errors occur frequently. No code change is needed, but this is a good pattern to acknowledge.

Before:

    try:
        review_body = generate_review(gemini_api_key, gemini_model, diff)
    except Exception as e:
        print(f"Error generating review via Gemini API: {e}", file=sys.stderr)
        sys.exit(1) # This would fail the CI job

After (Current PR):

    try:
        review_body = generate_review(gemini_api_key, gemini_model, diff)
    except Exception as e:
        print(
            f"Warning: Failed to generate review via Gemini API: {e}", file=sys.stderr
        )
        print("Exiting gracefully with code 0 to avoid failing the CI build.")
        sys.exit(0) # This will let the CI job pass

This is a good design choice for a non-critical review step.

3. Consistency in print Statements

There are a few instances where print statements include an explicit file=sys.stderr for warnings/errors, and others where they do not. While standard print often defaults to sys.stdout, for warnings and errors, explicitly directing to sys.stderr is a good practice for clarity and for systems that differentiate between stdout and stderr.

Recommendation: For all warning and error messages, explicitly use file=sys.stderr. This makes logs clearer, especially in CI environments where stdout and stderr might be streamed or captured separately.

Before (Example from get_ignore_spec):

        except Exception as e:
            print(f"Warning: Failed to parse .gitignore: {e}", file=sys.stderr)

This is already good.

Before (Example from build_diff_content):

            print("Max character limit reached during diff generation. Stopping file retrieval.")

After (Example from build_diff_content):

            print(
                "Max character limit reached during diff generation. Stopping file retrieval.",
                file=sys.stderr # Add this for consistency
            )

This is a minor point, but contributes to overall robustness.

✅ Verdict

Approve

This PR significantly improves the project's CI/CD quality and the robustness of the AI review process. The changes are well-implemented, address important edge cases, and align with best practices for automated code quality and CI efficiency. The minor points mentioned above are mostly stylistic or consistency refinements that can be addressed in follow-up work or by the integrated formatter.

[github-actions]

🤖 AI PR Review Summary

This Pull Request introduces significant enhancements to the project's development workflow and CI/CD pipeline. Key additions include:

1. Pre-commit Hooks: Integration of pre-commit to automatically run pants fmt and pants lint locally, enforcing code style and quality before commits.
2. AI Code Review Workflow: A new GitHub Actions workflow (ai-review.yml) for automated AI-powered code reviews using the Gemini API, triggered on PR events and filtered by file paths.
3. CI Optimization: Introduction of concurrency control in the GitHub Actions workflow to prevent redundant CI runs.
4. Documentation Updates: Comprehensive updates to README.md and agents.md to reflect the new tools and repository structure.

The core Python script scripts/ai_pr_reviewer.py is well-structured, handles various edge cases gracefully (e.g., missing API keys, large diffs, read-only GitHub tokens for external forks), and implements robust error handling. The use of sys.stderr for warnings/errors and exiting with code 0 for non-critical failures in CI is a pragmatic and excellent design choice.

💡 Key Feedback & Recommendations

1. Python Code Quality: Formatting Consistency

The Python script ai_pr_reviewer.py has numerous new empty lines inserted that deviate from standard PEP8 practices and automatic formatters like Ruff or Black (which are mentioned in the README.md). While not a functional bug, it creates minor visual inconsistencies. Ensuring the script is formatted with pants fmt (which uses Ruff/Black) would resolve these.

Recommendation: Run pants fmt scripts/ai_pr_reviewer.py to automatically align the script with the project's formatting standards. This typically involves removing extraneous empty lines and consistent spacing.

Before (examples from diff):

MAX_DIFF_CHARACTERS = 150000


def get_ignore_spec() -> pathspec.PathSpec:
    """Loads .gitignore patterns and appends custom file exclusion wildcards."""
    ignore_patterns = []
    try:
        with open(".gitignore", "r") as f:
            lines = [line.strip() for line in f if line.strip() and not line.startswith("#")]
            ignore_patterns.extend(lines)
    except Exception as e:
        print(f"Warning: Failed to parse .gitignore: {e}", file=sys.stderr)
        
    # Custom wildcards for lockfiles and binary assets to skip
    ignore_patterns.extend([
        "*.lock",
        "*.png",
        "*.jpg",
        "*.jpeg",
        "*.zip",
        "*.pdf"
    ])
    
    return pathspec.PathSpec.from_lines("gitwildmatch", ignore_patterns)

After (as Ruff/Black would format):

MAX_DIFF_CHARACTERS = 150000

def get_ignore_spec() -> pathspec.PathSpec:
    """Loads .gitignore patterns and appends custom file exclusion wildcards."""
    ignore_patterns = []
    try:
        with open(".gitignore", "r") as f:
            lines = [line.strip() for line in f if line.strip() and not line.startswith("#")]
            ignore_patterns.extend(lines)
    except Exception as e:
        print(f"Warning: Failed to parse .gitignore: {e}", file=sys.stderr)

    # Custom wildcards for lockfiles and binary assets to skip
    ignore_patterns.extend(["*.lock", "*.png", "*.jpg", "*.jpeg", "*.zip", "*.pdf"])

    return pathspec.PathSpec.from_lines("gitwildmatch", ignore_patterns)

(This example demonstrates common spacing adjustments; the ignore_patterns.extend multi-line list was already correctly condensed in the PR.)

2. PySpark & Data Engineering Best Practices

This PR focuses on infrastructure and tooling rather than PySpark code. No PySpark or data processing code was modified in this diff, so there are no specific best practices to review in this area.

3. Bugs & Edge Cases

The script scripts/ai_pr_reviewer.py demonstrates excellent handling of bugs and edge cases:

• Robust Environment Variable Handling: Graceful exits for missing API keys (exit 0 for external forks) and other essential variables (exit 1 for configuration errors).
• Diff Size Limits: Implements MAX_DIFF_CHARACTERS and single-file size limit to prevent token overruns with large diffs.
• File Exclusion: Correctly uses .gitignore and custom patterns to skip irrelevant files.
• API Resilience: The main function wraps calls to generate_review and post_review in try...except blocks, exiting gracefully with code 0 on failure. This ensures the CI pipeline doesn't fail due to transient external API issues or read-only GitHub token permissions for external forks.
• Standard Error Stream: Consistent use of file=sys.stderr for all warning and error messages, which is a good practice for command-line scripts.

This level of robustness is highly commendable for an automation script running in a CI environment.

✅ Verdict

Approve

This is an excellent PR. The changes significantly improve the developer experience and CI automation. The ai_pr_reviewer.py script is particularly well-designed and robust. The only minor point is formatting consistency, which can be easily resolved by running pants fmt on the new script.