Skip to content

feat: distill learned rules from human feedback#35

Merged
SimonOyaneder merged 6 commits into
mainfrom
feat/learned-rules-from-feedback
Feb 26, 2026
Merged

feat: distill learned rules from human feedback#35
SimonOyaneder merged 6 commits into
mainfrom
feat/learned-rules-from-feedback

Conversation

@SimonOyaneder

Copy link
Copy Markdown
Owner

Summary

  • Closes the feedback loop: accumulated human feedback on Canon findings is distilled into concise team preference rules via LLM
  • Rules are stored per-repo and injected into the system prompt on subsequent reviews, so Canon learns from team corrections
  • Distillation is event-driven (triggered when feedback count crosses a threshold, default 5) and fire-and-forget (non-blocking)

Changes

  • Migration 0003: New repo_learned_rules table (one row per repo, upserted)
  • src/db/learned-rules.ts: DB access functions (get/store rules, fetch feedback for distillation, count helpers)
  • src/intelligence/distiller.ts: Sends feedback to LLM, returns max 10 bullet-point rules
  • src/config/index.ts: New distillThreshold config (env DISTILL_THRESHOLD, default 5)
  • src/review/prompt-builder.ts: New learnedRules field in PromptContext, injected after CANON.md section
  • src/github/pr-handler.ts: Fetches learned rules from DB and passes to prompt context
  • src/github/feedback-handler.ts: Triggers distillation after feedback storage when threshold is reached

Test plan

  • npm run lint — type-check passes
  • npm test — all 330 tests pass (20 new tests added)
  • Integration test with real DB (testcontainers)
  • Manual verification: simulate feedback → verify distillation → verify rules in prompt

🤖 Generated with Claude Code

SimonOyaneder and others added 4 commits February 26, 2026 18:57
Add migration 0003 with repo_learned_rules table (one row per repo,
upserted on re-distillation). Add learned-rules.ts with functions for
getting/storing rules and fetching feedback for distillation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Distills accumulated human feedback into concise team preference rules
via LLM. Uses generateText() for dual-API support.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add distillThreshold config (env DISTILL_THRESHOLD, default 5)
- Inject learned rules into system prompt after CANON.md section
- Fetch learned rules in pr-handler and pass to PromptContext
- Trigger fire-and-forget distillation in feedback-handler when
  accumulated feedback reaches threshold

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Extend feedback-handler tests with distillation trigger tests
- Extend prompt-builder tests for learned rules section
- Extend pr-handler tests for learnedRules passing
- Add distillThreshold to all test makeConfig() functions

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@canon-review canon-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found 2 pattern deviations: 🟡 2 medium

Comment thread src/db/learned-rules.ts Outdated
category: string;
severity: string;
finding: string;
existing_pattern: string;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Naming Convention · medium · confidence: 95%

DB row interface uses snake_case property names, while existing TypeScript interfaces in the repo use camelCase and map snake_case at query-read time.

Existing pattern: Use camelCase in TS interfaces/types even when SQL columns are snake_case (e.g., map row.file_path -> path, row.content_hash -> contentHash).
Reference: src/intelligence/embeddings.ts

Suggested change
existing_pattern: string;
export interface FeedbackRow {
category: string;
severity: string;
finding: string;
existingPattern: string;
replyBody: string;
replyAuthor: string;
}
// In query:
SELECT
f.category,
f.severity,
f.finding,
f.existing_pattern AS "existingPattern",
rf.reply_body AS "replyBody",
rf.reply_author AS "replyAuthor"

Comment thread src/intelligence/distiller.ts Outdated
`Category: ${row.category}`,
`Severity: ${row.severity}`,
`Canon's finding: ${row.finding}`,
`Existing pattern referenced: ${row.existing_pattern}`,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Pattern Violation · medium · confidence: 93%

Uses snake_case fields from FeedbackRow directly in app-layer code, deviating from the repo’s camelCase object-property pattern.

Existing pattern: Application-layer object properties are camelCase; SQL snake_case is translated at DB boundary.
Reference: src/intelligence/embeddings.ts

Suggested change
`Existing pattern referenced: ${row.existing_pattern}`,
`Existing pattern referenced: ${row.existingPattern}`,
`Developer reply (${row.replyAuthor}): ${row.replyBody}`,

SimonOyaneder and others added 2 commits February 26, 2026 19:00
Map snake_case SQL columns to camelCase at the query boundary using
column aliases, following existing repo conventions (e.g. embeddings.ts).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Document the feedback distillation pipeline, DISTILL_THRESHOLD config,
new files (learned-rules.ts, distiller.ts), and updated test counts.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@SimonOyaneder

Copy link
Copy Markdown
Owner Author

/canon review

@canon-review canon-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found 1 pattern deviations: ⚪ 1 low

@@ -1,5 +1,11 @@
import type { Context } from "probot";
import { storeFeedback } from "../db/review-store";
import { getFeedbackCount, getStoredFeedbackCount, getFeedbackForDistillation, storeLearnedRules } from "../db/learned-rules";

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style Inconsistency · low · confidence: 95%

Import list exceeds the repository’s established multiline import formatting pattern used for long named imports.

Existing pattern: Long named imports are split across multiple lines with one symbol per line for readability.
Reference: src/github/init-handler.ts

Suggested change
import { getFeedbackCount, getStoredFeedbackCount, getFeedbackForDistillation, storeLearnedRules } from "../db/learned-rules";
import {
getFeedbackCount,
getStoredFeedbackCount,
getFeedbackForDistillation,
storeLearnedRules,
} from "../db/learned-rules";

@SimonOyaneder SimonOyaneder merged commit 7148fb6 into main Feb 26, 2026
2 checks passed
@SimonOyaneder SimonOyaneder deleted the feat/learned-rules-from-feedback branch February 26, 2026 22:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant