Measure generic model-facing tool ergonomics - #128
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add low-cardinality, privacy-bounded telemetry for model-visible runtime tool ergonomics by reusing the existing durable/queryable Action Audit sink rather than introducing a second telemetry subsystem.
summary.model_ergonomicsobject per outer model-visible runtime invocationtool_name, registry-owned category, success, invocation duration, exact final model-facing ToolResult UTF-8 byte length, and bounded structured failure/recovery metadataserialized_result_bytes = nullresult_truncatedbecause there is no authoritative final-result truncation factStorage / query model
The existing SQLite Action Audit
action_eventsrow remains the durable source of truth. Generic telemetry is stored only undersummary.model_ergonomicsand is available through the existing audit query path. No new table, migration, dashboard, stats service, or background recorder is added.The record intentionally excludes args, commands/argv/scripts/stdin/stdout/stderr, ToolResult bodies, error prose, paths/cwd/project paths, queries/file contents, clipboard/Computer contents, Session message bodies, prompts/answers, credentials, native identities, and new high-cardinality correlation IDs.
Exact result-byte contract
serialized_result_bytesis the UTF-8 byte length of the exact final model-facing ToolResult JSON projection.structuredContentToolResult after MCP-only image/resource framing; JSON-RPC envelopes and MCP content blocks are excluded.Independent review corrected MCP exact-byte accounting to include the final
error: nullfield and closed pre-kernel MCP validation and hard-dispatch-timeout telemetry gaps.Coverage
Instrumentation identity comes from the live model-visible
ToolDefinitionregistry; the implementation does not hardcode a tool count. Hidden/internal helpers and the Canonical Connector surface do not start generic model-usage telemetry. After integrating currentmain(including #126 and #127), the registry coverage regression still passes.Validation
Implementation and independent review covered:
cargo check -p webcodexcargo fmt -- --checkgit diff --checkLatest-main integration closure additionally passed:
cargo test every_registered_model_visible_tool_has_generic_telemetry_identity -p webcodexcargo test mcp_tools_list_returns_same_names_as_runtime -p webcodexcargo check -p webcodexgit diff --check origin/main...HEADThe PR diff remains limited to the 10 telemetry implementation/test/documentation files. Deployment/restart/release operations are not part of this PR.