Skip to content

feat: migrate LLM rule evaluation to evidence-based validation model#30

Merged
Tanishq1030 merged 1 commit into
mainfrom
feature/evidence-based-evaluation
Jul 9, 2026
Merged

feat: migrate LLM rule evaluation to evidence-based validation model#30
Tanishq1030 merged 1 commit into
mainfrom
feature/evidence-based-evaluation

Conversation

@Tanishq1030

Copy link
Copy Markdown
Member

Summary

This PR separates pattern-matching evidence collection from final policy evaluation within the engine, addressing the false-positive limitations of static regex rules like ALN-001.

Instead of immediately generating a violation on an LLM API call pattern match, the match is treated as a Finding Candidate and evaluated in a post-scan validation pipeline. By default, all rules fall back to direct promotion, but specific rules (such as ALN-001/MIT-003-A) run through specialized context-aware validators to inspect the file for validation safeguards.

Closes #20

Acceptance Criteria Verification

  • [GD-1] LLM API invocations are treated as candidate evidence rather than immediate violations.
    • Status: Verified. Matches are appended to a temporary list of candidates instead of generating violations instantly.
  • [GD-2] Introduce a rule evaluation stage before generating findings.
    • Status: Verified. Added evaluate_candidates to act as an evaluation filter at the end of the file scan.
  • [GD-3] ALN-001 findings are only emitted after evaluation determines validation is absent.
    • Status: Verified. The pipeline discards the candidate finding if it detects any validation markers (such as Pydantic, Instructor, Guardrails, or explicit # anchor: validate comments).
  • [GD-4] Existing reporting format remains unchanged.
    • Status: Verified. The shape of findings returned to CLI and output templates is completely preserved.
  • [GD-5] Documentation updated to describe the new evidence-based evaluation pipeline.
    • Status: Verified. Created DOCS/research/evidence_evaluation.md explaining the new pipeline design.

Proposed Changes

  • Rule Evaluation Pipeline: Refactored scan_file in anchor/core/engine.py to route findings through evaluate_candidates.
  • ALN-001 Context Evaluator: Added specialized logic to check for validation indicators (e.g. imports of pydantic, instructor, guardrails, marshmallow, jsonschema, .validate(), .parse_obj(), json.loads(), BaseModel, or a # anchor: validate instruction comment). Candidates with validation markers are safely discarded.
  • Unit Tests: Added tests/unit/test_evaluation.py to cover both invalid and validated LLM invocation scenarios.
  • Documentation: Created DOCS/research/evidence_evaluation.md explaining the new pipeline design.

Verification Results

  • All 53/53 tests pass successfully (including new evaluation validation tests).

Copilot AI review requested due to automatic review settings July 9, 2026 14:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@Tanishq1030
Tanishq1030 merged commit b4f4433 into main Jul 9, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement Evidence-Based Evaluation for ALN-001

2 participants