Skip to content

feat: telemetry — tracing spans with token/cost fields (C5)#65

Merged
yuanhao merged 1 commit into
mainfrom
feat/telemetry
Jul 10, 2026
Merged

feat: telemetry — tracing spans with token/cost fields (C5)#65
yuanhao merged 1 commit into
mainfrom
feat/telemetry

Conversation

@yuanhao

@yuanhao yuanhao commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

What

Phase C5 — the final Phase C item. The loop now emits structured tracing spans, not just log lines:

agent_loop                (model)
└─ llm_stream             (turn, model, tokens_in, tokens_out, tokens_cached, cost_usd)
└─ tool                   (tool, tool_call_id, is_error)

cargo run --example telemetry shows it live:

INFO agent_loop{model=mock}:llm_stream{turn=1 model=mock tokens_in=0 ...}: close time.busy=347µs
INFO agent_loop{model=mock}: close time.busy=1.32ms

Design

  • OTel stays app-side (deviation from the roadmap's literal "feature flag", as discussed): the library emits plain tracing spans and gains no OTel dependency — apps install the tracing-opentelemetry layer and the same spans flow to Datadog/Tempo/Honeycomb/Jaeger. Zero overhead with no subscriber.
  • cost_usd recorded from CostConfig when pricing is configured — cost attribution lands in dashboards for free.
  • Futures instrumented with .instrument(span); no entered guards across .await.
  • tracing-subscriber added as dev-dependency only (test + example).

Tests (1 new, 361 total green)

Capturing Layer + WithSubscriber-attached agent_loop run asserting exact span counts (1× agent_loop, 2× llm_stream, 1× tool).

Verification

clippy -Dwarnings --all-features clean · 361 passed / 0 failed · docs -Dwarnings clean · example smoke-run verified · new mdBook page concepts/telemetry.md with the OTLP bridge snippet.

🤖 Generated with Claude Code

Phase C5 of the roadmap (final Phase C item): structured observability.

- Spans: `agent_loop` (model) → `llm_stream` per turn (turn, model, and
  recorded-on-completion tokens_in/out/cached + cost_usd when the
  ModelConfig has pricing) → `tool` per execution (tool, tool_call_id,
  is_error). Durations come free with spans.
- Futures instrumented with .instrument(span) — no entered guards held
  across .await.
- OTel deliberately stays app-side (tracing-opentelemetry layer): the
  library gains no OTel dependency, spans compile to no-ops without a
  subscriber. Documented with an OTLP bridge snippet.
- New examples/telemetry.rs (fmt subscriber with span timings; runs on
  MockProvider, no API key needed) — smoke-run verified.
- tracing-subscriber added as a dev-dependency only.

Tests: a capturing Layer + WithSubscriber-attached agent_loop run asserting
the span set (1× agent_loop, 2× llm_stream for two turns, 1× tool).

Docs: concepts/telemetry.md (+ SUMMARY), lib.rs/README bullets, CLAUDE.md,
CHANGELOG.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@yuanhao yuanhao merged commit ccec3bd into main Jul 10, 2026
4 checks passed
@yuanhao yuanhao deleted the feat/telemetry branch July 10, 2026 15:33
yuanhao added a commit that referenced this pull request Jul 10, 2026
Five-agent review of the combined v0.10.0...main diff (PRs #61-#65)
surfaced convergent findings; this is the full fix batch.

Critical fixes:
- prompt_structured error handling overhauled: provider failures now
  surface as StructuredPromptError::Provider carrying the real error
  (previously laundered into Parse { raw: "" }); the raw-text scan is
  scoped to messages produced by THIS call (a failed run can no longer
  return an earlier turn's JSON as Ok(T)); the payload is taken from the
  LAST text block (tool-forcing appends after preamble); Parse now carries
  #[source] serde_json::Error.
- Schema drop-leak closed structurally: the output_schema field on Agent is
  gone — the schema threads per-call through prompt_messages_internal into
  the run's AgentLoopConfig, so a timed-out/dropped future can't leave the
  agent schema-forced.
- Bedrock replays Content::Thinking (with signature) as reasoningContent —
  previously captured then dropped, guaranteeing a ValidationException on
  multi-turn thinking + tool use.
- Anthropic structured outputs disable thinking for that request (forced
  tool_choice + extended thinking is an API-level 400) with a warning.
- Session::append_new verifies the history extends the current path and
  returns HistoryDiverged instead of silently dropping turns or corrupting
  the tree when compaction (on by default) rewrites agent messages; returns
  the appended count.
- Session::from_jsonl single-pass validation: DuplicateId + UnknownParent
  (parents must precede children, which also makes cycles impossible —
  path_ids provably terminates).

API shape (free now — these types are unreleased):
- ToolMiddleware::before_tool takes a #[non_exhaustive] ToolCallRequest
  context struct instead of three frozen positional params.
- StreamConfig and OutputSchema are #[non_exhaustive]; StreamConfig::new
  added (this release alone added output_schema to it).
- ToolDecision documents the deliberate StopReason-style exhaustive policy.

Hardening:
- A panicking ToolMiddleware is contained (catch_unwind) as a Deny instead
  of killing the loop task and stripping the agent of its tools.
- Middleware denials emit tracing::warn! (operator visibility).
- unwrap_structured_tool_call removes only the synthetic tool call and
  preserves non-ToolUse stop reasons (Length truncation no longer laundered
  into success).
- llm_stream span gains an error field; seek_checkpoint is latest-wins on
  duplicate labels; Vertex round-trips Gemini thought signatures on
  function calls (parity with google.rs).

Docs: tools.md example on the new signature + panic note; structured-outputs
error semantics + schema-dialect caveats + thinking caveat; telemetry OTel
snippet updated to the current builder API (old one was removed in 0.17) and
overhead claims made honest; session-trees compaction caveat + validation;
"enforced natively where supported"; CHANGELOG Fixed section; CLAUDE.md.

Tests (+15, 376 total): schema propagation via capturing provider (replaces
the tautological reset test); Provider-error and stale-history-never-parsed;
middleware never sees the synthetic tool; middleware panic containment;
Batched-strategy deny; session divergence/dup-id/dangling-parent/latest-wins;
bedrock thinking-replay body; anthropic structured-disables-thinking body;
google thinking-drop pin; vertex signature round-trip; telemetry field values
including exact cost math (on_record collector).

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
@yuanhao yuanhao mentioned this pull request Jul 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant