Skip to content

Add skill health reporting#257

Merged
paudley merged 5 commits into
mainfrom
paudley/183-skill-provenance-health
Jun 23, 2026
Merged

Add skill health reporting#257
paudley merged 5 commits into
mainfrom
paudley/183-skill-provenance-health

Conversation

@paudley

@paudley paudley commented Jun 23, 2026

Copy link
Copy Markdown
Owner

Summary

  • add code-intel skill health aggregation over remediation outcome evidence
  • add CLI and MCP JSON/TOON skill health surfaces
  • record MCP skill_lookup and skill_recommend usage as unknown-outcome observations when code-intel storage is configured
  • document measurement-only skill health reporting and no automatic skill promotion

Validation

  • go test ./internal/codeintel ./internal/codeintelcli ./internal/mcp
  • ./bin/lint --staged
  • make build
  • make check (passes with existing non-blocking testing.go_coverage_goal warning)

Closes #183

@paudley paudley requested a review from ErinAudley as a code owner June 23, 2026 10:20
Copilot AI review requested due to automatic review settings June 23, 2026 10:20
@coderabbitai

coderabbitai Bot commented Jun 23, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@paudley, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 48 minutes and 16 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate.

For paid Pro and Pro+ PR reviews, CodeRabbit uses rolling per-developer review limits. Reviews become available again as older review attempts age out of the rolling limit window.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: c9b5c541-f321-4261-a368-78f5c12fd6b8

📥 Commits

Reviewing files that changed from the base of the PR and between 11a31e5 and d953dab.

📒 Files selected for processing (3)
  • README.md
  • docs/CAPABILITY_SURFACE_DECISION.md
  • docs/MCP_SERVER.md
📝 Walkthrough

Walkthrough

Adds end-to-end skill health tracking: a new skill_health.go module in the codeintel package records skill observations as remediation outcomes and computes 7d/30d windowed health metrics (unused, frequently failing, improving, stale, unknown). A skill-health CLI command and a code_intel_skill_health MCP tool expose JSON and TOON output. Existing skill_lookup and skill_recommend MCP tools now persist unknown-outcome observations when a code-intel root is configured. Documentation is updated accordingly.

Changes

Skill Health Feature

Layer / File(s) Summary
Data model and constants
go/internal/codeintel/skill_health.go
Defines exported types: SkillObservation, SkillProvenance, SkillHealthQuery, SkillHealthReport, SkillHealthSummary, SkillHealthRecord, SkillHealthWindow, SkillHealthWindowID. Adds configuration constants for status/trend labels, window day counts (7/30), and status ordering weights.
Database read and RecordSkillObservation
go/internal/codeintel/skill_health.go
Implements skillOutcomeRows to query remediation_outcomes filtered by skill ID, ordered by skill and recency. Implements Store.RecordSkillObservation to normalize observations, derive stable remediation/outcome IDs and search text, and persist via RecordRemediationOutcome.
SkillHealth query and record assembly pipeline
go/internal/codeintel/skill_health.go
Implements Store.SkillHealth query orchestration, query/observation normalization, per-skill record building from outcomes and known-skills catalog, outcome accumulation with window/counter logic, record finalization with trend/status derivation, record sorting by status/activity/ID, limit truncation, and summary computation. Includes 23 helper functions for window inclusion, staleness, time parsing, outcome counting, trend/status selection, and sorting.
TOON serialization
go/internal/codeintel/skill_health.go
Implements FormatSkillHealthTOON to serialize SkillHealthReport into delimited multi-line text format with header and per-skill rows. Implements skillHealthTOONCell to safely quote values containing delimiters.
codeintel package unit tests
go/internal/codeintel/skill_health_test.go
Tests SkillHealth trend and status computation with multiple skills and fixed reference time. Tests RecordSkillObservation produces expected "unknown" outcome rows. Tests summary counts are computed before pagination limit. Provides test helpers for outcome recording and status assertion.
CLI skill-health command and tests
go/internal/codeintelcli/main.go, go/internal/codeintelcli/main_internal_test.go
Adds defaultSkillHealthStaleDays constant, command dispatch entry, and printSkillHealth handler with flag parsing (--skill-id, --format, --limit, --stale-days), store query execution, and JSON/TOON rendering. Tests verify both output formats contain stable identifiers and skill-health is invoked in the command query suite.
MCP skill-health tool definition and protocol
go/internal/mcp/protocol.go
Defines codeIntelSkillHealthInput request struct and codeIntelSkillHealthToolDefinition with input schema and metadata. Increases tool-definition capacity constants. Sets TracePersisted: true on skill_lookup and skill_recommend. Appends new tool to code-intel tool definitions list.
MCP server handler and observation recording
go/internal/mcp/server.go, go/internal/mcp/skill_health.go
Implements Server.codeIntelSkillHealth handler parsing input, opening code-intel store, querying skill health with server-provided provenance, and returning JSON with optional TOON content. Implements observation recording helpers: recordSkillLookupObservation, recordSkillRecommendationObservations, recordSkillObservation, skillHealthProvenance, and logging support. Wires observation recording into lookupSkill and recommendSkills flows. Registers new tool in toolHandlers().
MCP server tests
go/internal/mcp/server_test.go
Updates tool-count assertion to 38 and adds code_intel_skill_health to expected tool names. Adds TestServerRecordsSkillToolUsageAndReportsSkillHealth verifying end-to-end observation recording in skill_lookup/skill_recommend, persistence to remediation_outcomes, and complete health report output with expected observations and structure.
Documentation and configuration
README.md, docs/CODE_INTEL.md, docs/MCP_SERVER.md, .gitattributes
Documents new code_intel_skill_health MCP tool, skill measurement behavioral model (unknown-outcome observation recording, outcome tracking, 7d/30d windowed reporting), CLI example with --format toon, and measurement-only promotion policy. Updates Python whitespace enforcement rules in .gitattributes.

Sequence Diagram(s)

sequenceDiagram
  rect rgba(100, 149, 237, 0.5)
    Note over Client,CodeIntelStore: Observation Recording Phase
    Client->>MCPServer: skill_lookup(skill_id: "safe-git-workflow")
    MCPServer->>MCPServer: lookupSkill(skill)
    MCPServer->>CodeIntelStore: RecordSkillObservation(tool=skill_lookup, outcome=unknown)
    MCPServer-->>Client: skill response

    Client->>MCPServer: skill_recommend(diagnostic)
    MCPServer->>MCPServer: recommendSkills(recommendations)
    MCPServer->>CodeIntelStore: RecordSkillObservation per recommendation (tool=skill_recommend)
    MCPServer-->>Client: {recommendations: [...]}
  end

  rect rgba(144, 238, 144, 0.5)
    Note over Client,CodeIntelStore: Health Reporting Phase
    Client->>MCPServer: code_intel_skill_health(skill_id, format)
    MCPServer->>MCPServer: codeIntelSkillHealth(skillHealthProvenance)
    MCPServer->>CodeIntelStore: SkillHealth(SkillHealthQuery with 7d/30d windows)
    CodeIntelStore->>CodeIntelStore: skillOutcomeRows → buildSkillHealthRecords → finalizeSkillHealthRecord
    CodeIntelStore-->>MCPServer: SkillHealthReport
    MCPServer-->>Client: JSON or TOON report
  end
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Suggested reviewers

  • ErinAudley

Poem

🐇 Hippity-hop, the skills get tracked,
Seven days, thirty days — nothing is slack!
Unknown, improving, or frequently failing,
The code-intel store keeps a tally most telling.
No skill gets promoted without a clear sign —
Measurement-only, by careful design! 🌟

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The PR title 'Add skill health reporting' accurately and concisely summarizes the main change: implementing skill health aggregation and reporting functionality.
Description check ✅ Passed The PR description covers the summary (what and why), includes validation steps, documents capability surfaces (CLI and MCP JSON/TOON), and references the linked issue. All required template sections are addressed with relevant content.
Linked Issues check ✅ Passed The PR fully implements the acceptance criteria from issue #183: records skill provenance/usage/outcomes, provides CLI/MCP JSON/TOON health reports with 7/30-day windows, enforces measurement-only approach without automatic promotion, and includes tests for recording and trend computation.
Out of Scope Changes check ✅ Passed Changes are focused on skill health reporting implementation. The .gitattributes Python pattern addition and minor CAPABILITY_SURFACE_DECISION.md formatting are trivial incidental updates unrelated to core objectives but not substantive scope violations.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch paudley/183-skill-provenance-health

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the code_intel_skill_health tool and CLI command to measure and report skill usage, trends, and health from remediation-outcome evidence. The changes span documentation, CLI commands, MCP server integration, and core logic in Go. The review feedback highlights three key performance optimization opportunities: opening the database connection outside the loop when recording multiple skill recommendations, parsing the constant nowUTC timestamp once rather than repeatedly inside loops, and simplifying the maxUTC helper by using direct lexicographical string comparison instead of parsing RFC3339 strings into time.Time objects.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread go/internal/mcp/skill_health.go
Comment thread go/internal/codeintel/skill_health.go Outdated
Comment thread go/internal/codeintel/skill_health.go

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@go/internal/codeintel/skill_health.go`:
- Around line 208-218: The summarizeSkillHealth function is being called after
the records have been truncated to query.Limit, causing the summary to only
reflect the limited subset of records instead of all skills. Move the
summarizeSkillHealth call to execute before the records are sliced to
query.Limit, so the summary is calculated on the complete records set rather
than the paginated set. This ensures the summary accurately represents all
skills.
- Around line 461-475: The UnknownSkill assignment in the
finalizeSkillHealthRecord function relies on an implicit invariant that the
Generated field correctly indicates catalog membership, which is fragile. Since
catalog membership is already explicitly known through the knownCatalog
parameter passed at record-creation time, refactor the UnknownSkill logic to use
knownCatalog directly rather than inferring it from the Generated field.
Simplify the assignment on line 472 to remove the dependency on the Generated
field, making the code more maintainable and less susceptible to future
refactoring errors by directly checking only knownCatalog and record.Total.

In `@go/internal/mcp/server.go`:
- Around line 842-845: The recordSkillLookupObservation method calls are
currently propagating errors that cause the entire skill lookup and skill
recommend tools to fail. Since observation recording is telemetry-only and not
critical to tool functionality, log the error instead of returning it. Find both
occurrences of the recordSkillLookupObservation error handling (around lines
842-845 and 887-890) and replace the error propagation with a log statement that
records the error but allows execution to continue.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 0164c406-f43a-4105-8919-0dbbbaa35283

📥 Commits

Reviewing files that changed from the base of the PR and between 12debae and af53138.

📒 Files selected for processing (11)
  • README.md
  • docs/CODE_INTEL.md
  • docs/MCP_SERVER.md
  • go/internal/codeintel/skill_health.go
  • go/internal/codeintel/skill_health_test.go
  • go/internal/codeintelcli/main.go
  • go/internal/codeintelcli/main_internal_test.go
  • go/internal/mcp/protocol.go
  • go/internal/mcp/server.go
  • go/internal/mcp/server_test.go
  • go/internal/mcp/skill_health.go
📜 Review details
⏰ Context from checks skipped due to timeout. (8)
  • GitHub Check: Validate GitHub workflows
  • GitHub Check: Coding Ethos SARIF Gate / Coding Ethos SARIF Gate
  • GitHub Check: Test (Python 3.11)
  • GitHub Check: Test (Python 3.13)
  • GitHub Check: Go coverage
  • GitHub Check: Unified lint
  • GitHub Check: Go fuzz smoke
  • GitHub Check: CodeQL (go)
🔇 Additional comments (11)
go/internal/codeintel/skill_health_test.go (1)

14-202: LGTM!

go/internal/codeintelcli/main.go (1)

25-25: LGTM!

Also applies to: 100-100, 206-260

go/internal/codeintelcli/main_internal_test.go (1)

957-1011: LGTM!

Also applies to: 1105-1107

go/internal/codeintel/skill_health.go (1)

680-729: LGTM!

go/internal/mcp/protocol.go (1)

366-371: LGTM!

Also applies to: 481-482, 879-879, 927-927, 1270-1295, 1590-1590

go/internal/mcp/skill_health.go (1)

17-54: LGTM!

Also applies to: 56-65, 67-115, 117-136, 138-173

go/internal/mcp/server.go (1)

228-228: LGTM!

go/internal/mcp/server_test.go (1)

54-55: LGTM!

Also applies to: 85-85, 2029-2105

README.md (1)

368-370: LGTM!

Also applies to: 433-439, 472-472

docs/CODE_INTEL.md (1)

192-200: LGTM!

Also applies to: 280-280, 596-598

docs/MCP_SERVER.md (1)

71-73: LGTM!

Also applies to: 105-111

Comment thread go/internal/codeintel/skill_health.go
Comment thread go/internal/codeintel/skill_health.go
Comment thread go/internal/mcp/server.go
Copilot AI review requested due to automatic review settings June 23, 2026 10:54
coderabbitai[bot]
coderabbitai Bot previously approved these changes Jun 23, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@paudley

paudley commented Jun 23, 2026

Copy link
Copy Markdown
Owner Author

Stage 2 update

  • Addressed Gemini review feedback in af53138.
  • Addressed CodeRabbit follow-up feedback in a29d660: summary counts now use the full unpaginated record set, known skill catalog membership is explicit, and telemetry write failures are logged without failing skill tools.
  • Resolved all reviewed threads after verification.
  • Validation: go test ./internal/codeintel ./internal/codeintelcli ./internal/mcp from the Go module root; ./bin/lint --staged; make check. The managed gates pass with the existing non-blocking testing.go_coverage_goal warning.

PR is ready for merge once required review and pending checks complete.

@paudley paudley enabled auto-merge (squash) June 23, 2026 10:56

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
README.md (1)

374-376: 📐 Maintainability & Code Quality | 🟡 Minor

Fix inconsistent window descriptions in MCP_SERVER.md (lines 82-84).

The skill health documentation has inconsistencies across files. README.md and CODE_INTEL.md consistently specify "7-day and 30-day" windows, but MCP_SERVER.md (line 82-84) vaguely states "recent usage windows". Align MCP_SERVER.md to match the specific window terminology used elsewhere. Additionally, clarify the evidence source: README.md (line 374-376) should mention that the report draws from "remediation-outcome evidence" or "code-intel outcome evidence" to match CODE_INTEL.md (line 605-607) and MCP_SERVER.md (line 124-127).

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@README.md` around lines 374 - 376, The skill health documentation contains
inconsistencies across README.md, CODE_INTEL.md, and MCP_SERVER.md. In
MCP_SERVER.md (lines 82-84), replace the vague term "recent usage windows" with
the specific terminology "7-day and 30-day" to match what is documented in
README.md and CODE_INTEL.md. Additionally, update the code_intel_skill_health
description in README.md (lines 374-376) to explicitly mention that the report
draws from "remediation-outcome evidence" or "code-intel outcome evidence" to
align with the evidence source terminology referenced in CODE_INTEL.md (lines
605-607) and MCP_SERVER.md (lines 124-127).
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/CAPABILITY_SURFACE_DECISION.md`:
- Around line 74-84: The minimum grounding checklist under "Minimum grounding by
category" is missing an explicit requirement for code-intel storage contracts
related to skill health surfaces. Add a new bullet point to the existing list of
grounding requirements (which currently covers MCP tools, CLI commands, policy
IDs, provider behavior, and output contracts) that specifically requires new
code-intel measurement features to document their storage table, schema, and
windowing contract in the code-intel capability documentation. This ensures that
skill health observations persisted in code-intel tables like
remediation_outcomes are properly grounded and discoverable.

---

Outside diff comments:
In `@README.md`:
- Around line 374-376: The skill health documentation contains inconsistencies
across README.md, CODE_INTEL.md, and MCP_SERVER.md. In MCP_SERVER.md (lines
82-84), replace the vague term "recent usage windows" with the specific
terminology "7-day and 30-day" to match what is documented in README.md and
CODE_INTEL.md. Additionally, update the code_intel_skill_health description in
README.md (lines 374-376) to explicitly mention that the report draws from
"remediation-outcome evidence" or "code-intel outcome evidence" to align with
the evidence source terminology referenced in CODE_INTEL.md (lines 605-607) and
MCP_SERVER.md (lines 124-127).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 373be931-6e43-40d5-abc9-0834bea1727c

📥 Commits

Reviewing files that changed from the base of the PR and between af53138 and 11a31e5.

📒 Files selected for processing (13)
  • .gitattributes
  • README.md
  • docs/CAPABILITY_SURFACE_DECISION.md
  • docs/CODE_INTEL.md
  • docs/MCP_SERVER.md
  • go/internal/codeintel/skill_health.go
  • go/internal/codeintel/skill_health_test.go
  • go/internal/codeintelcli/main.go
  • go/internal/codeintelcli/main_internal_test.go
  • go/internal/mcp/protocol.go
  • go/internal/mcp/server.go
  • go/internal/mcp/server_test.go
  • go/internal/mcp/skill_health.go
📜 Review details
⏰ Context from checks skipped due to timeout. (8)
  • GitHub Check: Unified lint
  • GitHub Check: Test (Python 3.11)
  • GitHub Check: Coding Ethos SARIF Gate / Coding Ethos SARIF Gate
  • GitHub Check: Validate GitHub workflows
  • GitHub Check: Go coverage
  • GitHub Check: Test (Python 3.13)
  • GitHub Check: Go fuzz smoke
  • GitHub Check: CodeQL (go)
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2026-06-23T12:26:47.373Z
Learnt from: paudley
Repo: paudley/coding-ethos PR: 258
File: go/internal/geminiprompts/render.go:125-132
Timestamp: 2026-06-23T12:26:47.373Z
Learning: In the paudley/coding-ethos repo, the documented guideline “Gemini prompt pack should be generated in go/internal/geminiprompts/” refers to the generator implementation location (Go source), not the location of the generated prompt-pack artifact. Always treat PromptPackPath as the enforced output artifact path `.coding-ethos/gemini/prompt-pack.json` (e.g., render_test.go asserts PromptPackPath, and compiler_policies.go / hookrunner lookup / README document the expected path). Do not flag a code review issue just because PromptPackPath points to `.coding-ethos/gemini/prompt-pack.json`; only flag violations if the artifact output path logic breaks that enforced contract.

Applied to files:

  • go/internal/codeintel/skill_health_test.go
  • go/internal/mcp/protocol.go
  • go/internal/mcp/server.go
  • go/internal/codeintelcli/main_internal_test.go
  • go/internal/codeintelcli/main.go
  • go/internal/mcp/skill_health.go
  • go/internal/mcp/server_test.go
  • go/internal/codeintel/skill_health.go
🪛 LanguageTool
docs/CAPABILITY_SURFACE_DECISION.md

[grammar] ~23-~23: Ensure spelling is correct
Context: ...s, MCP tools, or runnable commands when those own the facts. 3. **Does an agent need...

(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)


[style] ~79-~79: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ... must appear in docs/CODE_INTEL.md. - New policy IDs must live with their ETHOS p...

(ENGLISH_WORD_REPEAT_BEGINNING_RULE)

🔇 Additional comments (15)
go/internal/codeintelcli/main.go (1)

45-47: LGTM!

Also applies to: 71-71, 104-104

go/internal/codeintelcli/main_internal_test.go (1)

17-18: LGTM!

Also applies to: 45-146

go/internal/codeintel/skill_health.go (1)

122-123: LGTM!

Also applies to: 212-213, 385-386, 475-475

go/internal/codeintel/skill_health_test.go (1)

103-104: LGTM!

Also applies to: 114-140

.gitattributes (1)

8-8: LGTM!

docs/CAPABILITY_SURFACE_DECISION.md (1)

14-51: 📐 Maintainability & Code Quality | ⚡ Quick win

Surface decision tree accurately grounds skill health across MCP, CLI, and code-intel storage.

The decision tree (question 5 at line 37-40) correctly identifies that durable evidence, audit, and trend analysis require SARIF/code-intel storage—precisely how skill health observations are persisted and reported. Question 3 (line 25-29) correctly routes interactive structured agent access to MCP, grounding the code_intel_skill_health tool. Question 4 (line 31-35) justifies the CLI skill-health command as a one-shot local operation. The guide documents why multiple surfaces own parts of the skill health feature—a pattern the PR follows.

docs/CODE_INTEL.md (2)

192-200: 📐 Maintainability & Code Quality | ⚡ Quick win

Skill health measurement model clearly documented.

The new "Skill use" section (lines 192-200) accurately documents the measurement-only behavioral model: unknown-outcome observations from skill_lookup/recommend when code-intel is configured, explicit remediation outcomes for success/failure/abandonment, and 7d/30d windowed health reports. The clarification that learned/evolved skills are excluded from repo artifact promotion without explicit separate policy change aligns precisely with PR objectives and prevents scope creep.


280-283: 📐 Maintainability & Code Quality | ⚡ Quick win

CLI command and MCP tool documentation is consistent with README and matches implementation.

The skill-health --format toon CLI command (line 283) and code_intel_skill_health MCP tool (line 605-607) are consistently documented across CODE_INTEL.md, README.md, and MCP_SERVER.md. Both mention 7-day/30-day windows and unknown skill ID handling. The integration into the existing command/tool lists follows established patterns.

Also applies to: 605-607

docs/MCP_SERVER.md (1)

82-84: 📐 Maintainability & Code Quality | ⚡ Quick win

Tool documentation and dataflow clearly explain skill health measurement model.

The code_intel_skill_health tool entry (lines 82-84) and dataflow explanation (lines 122-128) consistently describe the measurement-only behavior across all three documentation files. The dataflow correctly identifies that unknown-outcome observations flow from skill_lookup/skill_recommend through code-intel storage, that code_intel_skill_health reads both observations and remediation outcomes, and that the result is measurement-only with no artifact promotion. The placement in the tool list (between code_intel_change_risk and code_intel_health) is logically ordered.

Also applies to: 122-128

go/internal/mcp/protocol.go (1)

312-330: LGTM!

Also applies to: 500-512, 942-1022, 1697-1741

go/internal/mcp/server.go (2)

196-239: LGTM!

Also applies to: 242-330


938-942: LGTM!

Also applies to: 981-988

go/internal/mcp/skill_health.go (1)

12-16: LGTM!

Also applies to: 141-148

go/internal/mcp/server_test.go (2)

56-57: LGTM!

Also applies to: 75-77, 113-116, 119-190


338-397: LGTM!

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Inline review comments failed to post. This is likely due to GitHub's internal server error or limits when posting large numbers of comments. If you are seeing this consistently it is likely a permissions issue. Please check "Moderation" -> "Code review limits" under your organization settings.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
README.md (1)

374-376: 📐 Maintainability & Code Quality | 🟡 Minor

Fix inconsistent window descriptions in MCP_SERVER.md (lines 82-84).

The skill health documentation has inconsistencies across files. README.md and CODE_INTEL.md consistently specify "7-day and 30-day" windows, but MCP_SERVER.md (line 82-84) vaguely states "recent usage windows". Align MCP_SERVER.md to match the specific window terminology used elsewhere. Additionally, clarify the evidence source: README.md (line 374-376) should mention that the report draws from "remediation-outcome evidence" or "code-intel outcome evidence" to match CODE_INTEL.md (line 605-607) and MCP_SERVER.md (line 124-127).

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@README.md` around lines 374 - 376, The skill health documentation contains
inconsistencies across README.md, CODE_INTEL.md, and MCP_SERVER.md. In
MCP_SERVER.md (lines 82-84), replace the vague term "recent usage windows" with
the specific terminology "7-day and 30-day" to match what is documented in
README.md and CODE_INTEL.md. Additionally, update the code_intel_skill_health
description in README.md (lines 374-376) to explicitly mention that the report
draws from "remediation-outcome evidence" or "code-intel outcome evidence" to
align with the evidence source terminology referenced in CODE_INTEL.md (lines
605-607) and MCP_SERVER.md (lines 124-127).
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/CAPABILITY_SURFACE_DECISION.md`:
- Around line 74-84: The minimum grounding checklist under "Minimum grounding by
category" is missing an explicit requirement for code-intel storage contracts
related to skill health surfaces. Add a new bullet point to the existing list of
grounding requirements (which currently covers MCP tools, CLI commands, policy
IDs, provider behavior, and output contracts) that specifically requires new
code-intel measurement features to document their storage table, schema, and
windowing contract in the code-intel capability documentation. This ensures that
skill health observations persisted in code-intel tables like
remediation_outcomes are properly grounded and discoverable.

---

Outside diff comments:
In `@README.md`:
- Around line 374-376: The skill health documentation contains inconsistencies
across README.md, CODE_INTEL.md, and MCP_SERVER.md. In MCP_SERVER.md (lines
82-84), replace the vague term "recent usage windows" with the specific
terminology "7-day and 30-day" to match what is documented in README.md and
CODE_INTEL.md. Additionally, update the code_intel_skill_health description in
README.md (lines 374-376) to explicitly mention that the report draws from
"remediation-outcome evidence" or "code-intel outcome evidence" to align with
the evidence source terminology referenced in CODE_INTEL.md (lines 605-607) and
MCP_SERVER.md (lines 124-127).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 373be931-6e43-40d5-abc9-0834bea1727c

📥 Commits

Reviewing files that changed from the base of the PR and between af53138 and 11a31e5.

📒 Files selected for processing (13)
  • .gitattributes
  • README.md
  • docs/CAPABILITY_SURFACE_DECISION.md
  • docs/CODE_INTEL.md
  • docs/MCP_SERVER.md
  • go/internal/codeintel/skill_health.go
  • go/internal/codeintel/skill_health_test.go
  • go/internal/codeintelcli/main.go
  • go/internal/codeintelcli/main_internal_test.go
  • go/internal/mcp/protocol.go
  • go/internal/mcp/server.go
  • go/internal/mcp/server_test.go
  • go/internal/mcp/skill_health.go
📜 Review details
🔇 Additional comments (15)
go/internal/codeintelcli/main.go (1)

45-47: LGTM!

Also applies to: 71-71, 104-104

go/internal/codeintelcli/main_internal_test.go (1)

17-18: LGTM!

Also applies to: 45-146

go/internal/codeintel/skill_health.go (1)

122-123: LGTM!

Also applies to: 212-213, 385-386, 475-475

go/internal/codeintel/skill_health_test.go (1)

103-104: LGTM!

Also applies to: 114-140

.gitattributes (1)

8-8: LGTM!

docs/CAPABILITY_SURFACE_DECISION.md (1)

14-51: 📐 Maintainability & Code Quality | ⚡ Quick win

Surface decision tree accurately grounds skill health across MCP, CLI, and code-intel storage.

The decision tree (question 5 at line 37-40) correctly identifies that durable evidence, audit, and trend analysis require SARIF/code-intel storage—precisely how skill health observations are persisted and reported. Question 3 (line 25-29) correctly routes interactive structured agent access to MCP, grounding the code_intel_skill_health tool. Question 4 (line 31-35) justifies the CLI skill-health command as a one-shot local operation. The guide documents why multiple surfaces own parts of the skill health feature—a pattern the PR follows.

docs/CODE_INTEL.md (2)

192-200: 📐 Maintainability & Code Quality | ⚡ Quick win

Skill health measurement model clearly documented.

The new "Skill use" section (lines 192-200) accurately documents the measurement-only behavioral model: unknown-outcome observations from skill_lookup/recommend when code-intel is configured, explicit remediation outcomes for success/failure/abandonment, and 7d/30d windowed health reports. The clarification that learned/evolved skills are excluded from repo artifact promotion without explicit separate policy change aligns precisely with PR objectives and prevents scope creep.


280-283: 📐 Maintainability & Code Quality | ⚡ Quick win

CLI command and MCP tool documentation is consistent with README and matches implementation.

The skill-health --format toon CLI command (line 283) and code_intel_skill_health MCP tool (line 605-607) are consistently documented across CODE_INTEL.md, README.md, and MCP_SERVER.md. Both mention 7-day/30-day windows and unknown skill ID handling. The integration into the existing command/tool lists follows established patterns.

Also applies to: 605-607

docs/MCP_SERVER.md (1)

82-84: 📐 Maintainability & Code Quality | ⚡ Quick win

Tool documentation and dataflow clearly explain skill health measurement model.

The code_intel_skill_health tool entry (lines 82-84) and dataflow explanation (lines 122-128) consistently describe the measurement-only behavior across all three documentation files. The dataflow correctly identifies that unknown-outcome observations flow from skill_lookup/skill_recommend through code-intel storage, that code_intel_skill_health reads both observations and remediation outcomes, and that the result is measurement-only with no artifact promotion. The placement in the tool list (between code_intel_change_risk and code_intel_health) is logically ordered.

Also applies to: 122-128

go/internal/mcp/protocol.go (1)

312-330: LGTM!

Also applies to: 500-512, 942-1022, 1697-1741

go/internal/mcp/server.go (2)

196-239: LGTM!

Also applies to: 242-330


938-942: LGTM!

Also applies to: 981-988

go/internal/mcp/skill_health.go (1)

12-16: LGTM!

Also applies to: 141-148

go/internal/mcp/server_test.go (2)

56-57: LGTM!

Also applies to: 75-77, 113-116, 119-190


338-397: LGTM!

🛑 Comments failed to post (1)
docs/CAPABILITY_SURFACE_DECISION.md (1)

74-84: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Add explicit grounding requirement for skill health surfaces.

The minimum grounding checklist (lines 74-84) lists MCP tools, CLI commands, policy IDs, provider behavior, and output contracts, but does not explicitly mention code-intel storage contracts. Since skill health observations are persisted in code-intel tables (remediation_outcomes), the checklist should clarify that new measurement/outcome features must document their code-intel storage contract alongside MCP/CLI surfaces. Optionally add a new bullet: "New code-intel measurement features must document their storage table, schema, and windowing contract in the code-intel capability docs."

🧰 Tools
🪛 LanguageTool

[style] ~79-~79: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ... must appear in docs/CODE_INTEL.md. - New policy IDs must live with their ETHOS p...

(ENGLISH_WORD_REPEAT_BEGINNING_RULE)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/CAPABILITY_SURFACE_DECISION.md` around lines 74 - 84, The minimum
grounding checklist under "Minimum grounding by category" is missing an explicit
requirement for code-intel storage contracts related to skill health surfaces.
Add a new bullet point to the existing list of grounding requirements (which
currently covers MCP tools, CLI commands, policy IDs, provider behavior, and
output contracts) that specifically requires new code-intel measurement features
to document their storage table, schema, and windowing contract in the
code-intel capability documentation. This ensures that skill health observations
persisted in code-intel tables like remediation_outcomes are properly grounded
and discoverable.

Copilot AI review requested due to automatic review settings June 23, 2026 17:07

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@paudley paudley merged commit 8833a19 into main Jun 23, 2026
22 of 23 checks passed
@paudley paudley deleted the paudley/183-skill-provenance-health branch June 23, 2026 17:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[feature] Track skill provenance, usage, and health

2 participants