Skip to content

Add MCP annotation tools - #309

Merged
johnnygreco merged 1 commit into
feature/trace-annotationsfrom
issue-304-mcp-annotation-tools
Jun 17, 2026
Merged

Add MCP annotation tools#309
johnnygreco merged 1 commit into
feature/trace-annotationsfrom
issue-304-mcp-annotation-tools

Conversation

@johnnygreco

Copy link
Copy Markdown
Owner

Summary

Adds MCP annotation tools for agents: create_annotation, update_annotation, list_annotations, get_annotation, and delete_annotation.

This also makes message-level annotations first-class by adding a message annotation target type backed by activity_events.event_uid and validated against event_kind = 'message'. Transcript message controls now create message annotations, while non-message transcript blocks remain generic event annotations. MCP uses the current tool surface only; no legacy framing or compatibility shims were added.

Closes #304

Validation

  • generate: make generate-check -> passed after commit 06562ce
  • fmt: make fmt-check -> passed
  • test: make test -> passed
  • lint: make lint -> passed
  • build: make build -> passed
  • frontend: npm run test:frontend -> passed
  • targeted MCP protocol: go test ./internal/mcp -run 'TestTool(CreateAnnotationSupportsMessageTarget|ListAnnotationsSessionTargetListsAllSessionAnnotations|UpdateAndDeleteAnnotationVerifyMessageScope|GetAnnotationScopedMissReturnsForbidden|DefinitionsMatchImplementedArguments|DefinitionsAreOpenAIFunctionCompatible)|TestDataBackedToolsReturnDatabaseUnavailableToolErrors' -> passed
  • e2e: npm run test:e2e -> passed, 52 tests
  • a11y: npm run test:a11y -> passed, 8 tests
  • whitespace: git diff --check -> passed

Sample MCP Calls

Create message annotation:

{"name":"create_annotation","arguments":{"target_type":"message","message_id":"event:msg-1","session_id":null,"event_id":null,"open_ref":null,"author_id":"agent-1","author_name":"Reviewer Agent","category":"quality","outcome":null,"quality_score":4,"confidence":80,"needs_followup":false,"labels":["dataset:eval"],"note":"Message-level finding.","metadata_json":"{\"rubric\":\"qa\"}","source_name":null,"source_names":null,"runtime":null,"runtimes":null,"project_key":null,"project_keys":null}}

List session annotations:

{"name":"list_annotations","arguments":{"target_type":null,"session_id":"session:session-1","message_id":null,"event_id":null,"open_ref":null,"include_deleted":false,"limit":200,"offset":0,"source_name":null,"source_names":null,"runtime":null,"runtimes":null,"project_key":null,"project_keys":null}}

Risks

MCP now opens ClickHouse through the writable store path so write tools can insert annotation revisions. That is intentional for this issue and covered by backend-unavailable redaction tests.

Message annotations introduce a new target_type value (message) without a schema migration because trace_annotations.target_type is already a string column. Validation rejects message targets unless the referenced activity event is currently a message event.

Reviewer Notes

Focus on target resolution and scope behavior in internal/mcp/annotation_tools.go, the new message model/API handling, and the fact that session annotation listing intentionally returns all annotation targets for that session unless a target type is specified. Backward compatibility was explicitly not required and no compatibility work was added.

@johnnygreco
johnnygreco force-pushed the issue-304-mcp-annotation-tools branch from 06562ce to 4c4f7f9 Compare June 17, 2026 05:11
@johnnygreco

Copy link
Copy Markdown
Owner Author

Review Record

Frozen target: feature/trace-annotations f9bfcf7a4d8e646d639e096e34f4e17f09315067..issue-304-mcp-annotation-tools 4c4f7f9732a02745a8e2510908a984ad7a1f7c3b
Diff command: git diff f9bfcf7a4d8e646d639e096e34f4e17f09315067..4c4f7f9732a02745a8e2510908a984ad7a1f7c3b

Panel:

  • Codex: Robustness/correctness, security/privacy, MCP API contracts and scoped target resolution -> findings, then clean on amended head
  • Codex: Data/model/API/UI integration correctness, data contracts, target-type consistency -> findings, then clean on amended head
  • Codex: Tests/coverage, maintainability/complexity discipline -> findings, then clean on amended head
  • Codex: Agent experience, user experience/accessibility, documentation/adoption -> clean
    Skipped Review-Standard lenses: none

Accepted findings fixed:

  • list_annotations could expose out-of-scope event/message annotation rows from a session-scoped list -> added per-row scoped target validation before formatting list results, skipping forbidden/not-found rows and preserving internal-error failures.
  • Message annotation results emitted event-shaped open_ref values, so returned refs did not round-trip through list_annotations as message targets -> added message-shaped refs with message_id, taught annotation/open target resolution to honor them, and added create-to-list round-trip coverage.
  • E2E fixture type excluded target_type: "message" -> added a local AnnotationTarget = 'session' | 'message' | 'event' alias and used it in FixtureAnnotation.

Rejected findings:

  • none

Validation after fixes:

  • go test ./internal/mcp -run 'TestTool(CreateAnnotationSupportsMessageTarget|ListAnnotationsSessionTargetListsAllSessionAnnotations|ListAnnotationsFiltersOutOfScopeTargets|ListAnnotationsMessageOpenRefRoundTripsFromCreate|UpdateAndDeleteAnnotationVerifyMessageScope|GetAnnotationScopedMissReturnsForbidden)' -> passed
  • make generate-check -> passed
  • make fmt-check -> passed
  • make test -> passed
  • make lint -> passed
  • make build -> passed
  • npm run test:frontend -> passed
  • npm run test:e2e -> passed, 52 tests
  • npm run test:a11y -> passed, 8 tests
  • git diff --check -> passed
  • GitHub CI on head 4c4f7f9732a02745a8e2510908a984ad7a1f7c3b -> all checks passed: format, generated, govulncheck, npm-audit, frontend, dependency-review, test, lint, build, playwright-dashboard, playwright-accessibility

Final status: all reviewer findings fixed and rechecked clean on current head; no unresolved blockers.

@johnnygreco
johnnygreco merged commit 65c3ff9 into feature/trace-annotations Jun 17, 2026
11 checks passed
@johnnygreco
johnnygreco deleted the issue-304-mcp-annotation-tools branch June 17, 2026 05:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant