Skip to content

Measure generic model-facing tool ergonomics - #128

Merged
yyjeqhc merged 3 commits into
mainfrom
feat/model-ergonomics-telemetry
Aug 23, 2026
Merged

Measure generic model-facing tool ergonomics#128
yyjeqhc merged 3 commits into
mainfrom
feat/model-ergonomics-telemetry

Conversation

@yyjeqhc

@yyjeqhc yyjeqhc commented Aug 23, 2026

Copy link
Copy Markdown
Owner

Summary

Add low-cardinality, privacy-bounded telemetry for model-visible runtime tool ergonomics by reusing the existing durable/queryable Action Audit sink rather than introducing a second telemetry subsystem.

  • record one summary.model_ergonomics object per outer model-visible runtime invocation
  • measure tool_name, registry-owned category, success, invocation duration, exact final model-facing ToolResult UTF-8 byte length, and bounded structured failure/recovery metadata
  • keep pre-ToolResult failures observable with serialized_result_bytes = null
  • preserve batch-as-one-outer-invocation semantics and existing edit-specific telemetry without duplicate generic counting
  • deliberately defer generic result_truncated because there is no authoritative final-result truncation fact
  • keep telemetry failure isolated from tool success/failure semantics

Storage / query model

The existing SQLite Action Audit action_events row remains the durable source of truth. Generic telemetry is stored only under summary.model_ergonomics and is available through the existing audit query path. No new table, migration, dashboard, stats service, or background recorder is added.

The record intentionally excludes args, commands/argv/scripts/stdin/stdout/stderr, ToolResult bodies, error prose, paths/cwd/project paths, queries/file contents, clipboard/Computer contents, Session message bodies, prompts/answers, credentials, native identities, and new high-cardinality correlation IDs.

Exact result-byte contract

serialized_result_bytes is the UTF-8 byte length of the exact final model-facing ToolResult JSON projection.

  • API/GPT Actions count the final ToolResult returned by the API path.
  • MCP counts the final structuredContent ToolResult after MCP-only image/resource framing; JSON-RPC envelopes and MCP content blocks are excluded.
  • If a recognized model-visible tool fails before a ToolResult exists, the field is null rather than measuring a transport error envelope.

Independent review corrected MCP exact-byte accounting to include the final error: null field and closed pre-kernel MCP validation and hard-dispatch-timeout telemetry gaps.

Coverage

Instrumentation identity comes from the live model-visible ToolDefinition registry; the implementation does not hardcode a tool count. Hidden/internal helpers and the Canonical Connector surface do not start generic model-usage telemetry. After integrating current main (including #126 and #127), the registry coverage regression still passes.

Validation

Implementation and independent review covered:

  • generic telemetry helper regressions, including UTF-8 byte accounting and structured failure classification
  • API success/failure/pre-result/batch/failure-isolation regressions
  • MCP exact structuredContent byte accounting
  • MCP pre-kernel and pre-result invalid-argument telemetry
  • edit/generic de-duplication
  • model-visible registry coverage
  • Action Audit queryability/privacy checks
  • cargo check -p webcodex
  • cargo fmt -- --check
  • git diff --check

Latest-main integration closure additionally passed:

  • cargo test every_registered_model_visible_tool_has_generic_telemetry_identity -p webcodex
  • cargo test mcp_tools_list_returns_same_names_as_runtime -p webcodex
  • cargo check -p webcodex
  • git diff --check origin/main...HEAD

The PR diff remains limited to the 10 telemetry implementation/test/documentation files. Deployment/restart/release operations are not part of this PR.

@yyjeqhc yyjeqhc added the run-ci Run the heavy pull-request CI jobs for an owner-authored PR label Aug 23, 2026
@yyjeqhc
yyjeqhc merged commit e2c0f9a into main Aug 23, 2026
8 of 14 checks passed
@yyjeqhc
yyjeqhc deleted the feat/model-ergonomics-telemetry branch August 23, 2026 06:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-ci Run the heavy pull-request CI jobs for an owner-authored PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant