feat(tui): usage & tool diagnostics — token accounting (per-component, per-model + cache hit rate, per-tool sinks, compaction cost) and tool-call error patterns

[7jrxt42BxFZo4iAnN4CX]

Problem

Session cost and context are a black box. In-session surfaces exist (/tokens, /cost, /context report, the TUI Context panel) but they are session-scoped and live only: after the session ends there is no way to answer "where did my tokens/cache/money go over the last week", "which model is burning cache", or "which tool calls fail constantly". When a session eats 2M tokens you cannot tell whether it was replayed read output, cache-thrashing tool-schema churn, thinking spend, or a model re-running the same failing tool call. When the model "keeps making the same mistake" there is no report that names the tool + error signature so you can fix the tool description or the global AGENTS.md.

Affected: everyone who debugs token spend, tunes prompt-cache layout, or wants data-driven fixes to tool descriptions / AGENTS.md.

Proposed solution

A global CLI codewhale stats that aggregates persisted session data (already on disk — see code refs) into token/cost/cache/tool reports, mirroring opencode stats:

codewhale stats [--days N] [--models N] [--tools N] [--project <id>]

• --days N — window (default: all time)
• --models N — top-N model usage breakdown (default: all)
• --tools N — top-N tool usage (default: all)
• --project <id> — filter by project/worktree

Report sections (mirroring opencode, plus improvements):

A. Request composition (per-session, best-effort)

1. Per-component breakdown of the latest request: system prompt, tool schemas (wire payload), message history, tool results, reasoning/ledger — mirroring VTCode's token_budget_breakdown and Claude Code's /context. Uses the existing conservative estimator (estimate_input_tokens_conservative); no tokenizer dependency required for the first pass.

B. Lifetime accounting (global)
2. Per-model usage + cost + cache hit rate: input / output / cache read / cache write + $ per model + hit rate % (cache_read/(cache_read+input+cache_write)), plus reasoning/thinking tokens as their own line (Codewhale already tracks thinking replay tokens; opencode hides reasoning inside output — we can do better).
3. Prompt-cache statistics (per session, aggregate): requests, share of input served from cache, misses with likely cause, expected rebuilds (compaction / tool-result clearing), warm/cold — mirroring Claude Code's Prompt cache (main) line. /cache today is per-turn DeepSeek-specific.
4. Per-tool / per-command token sinks: read, grep, bash, git diff, webfetch, … — calls, tokens added to history, and how much was truncated/spooled (recoverable) vs replayed verbatim (the real leak). Sizes/truncation decisions already exist (truncate.rs, fetch_url.rs); missing is the ledger + aggregation.
5. Compaction cost as a metered line: count, trigger (manual/auto/threshold), summarizer input/output tokens + cost, history tokens dropped. Codewhale already attributes compaction to a turn owner (runtime_cost_owner); this is reporting, not new plumbing — compaction looks "free" today but costs tokens+cache (Claude Code documents it).

C. Reliability analytics (global)
6. Tool-call error analytics: per tool (and per model/provider): calls, failures, failure rate, error taxonomy (schema/validation, execution, timeout, policy/permission denial, empty/unusable result, parse failure), retry recovery rate (does the model correct itself or repeat the identical mistake), and recurring failure signatures across sessions. Actionable: fix a confusing tool description; put repeated bash mistakes into global AGENTS.md. Data exists (ToolResult error status, lifecycle outbox events); missing is the aggregation + patterns view.

D. Sub-agent / fleet statistics (from the durable fleet ledger)
7. Per-subagent and per-fleet analytics: aggregate worker usage by role (built-in roles + custom names from [fleet.roles]) and per run/worker:

• calls, tokens, cost, and status counts (queued / running / completed / partial / failed / restarted / escalated / cancelled / stale);
• failure sources (verifier / transport / task) and retry/recovery rate — same "where do agents keep failing" signal as slice 6, but per delegated worker;
• grouping survives custom roles: the ledger records the role name at run time, so aggregation keys on that name even when [fleet.roles] changes later.
• Data already exists: .codewhale/fleet.jsonl ledger, codewhale fleet status|inspect|logs|artifacts, Runtime API /v1/fleet/runs/{run_id}/workers, and SessionCostSnapshot.subagent_cost_usd/cny for money already attributed to sub-agents.

Use case

• codewhale stats --days 30 --tools 10 → bash shows 60% of replayed tokens → switch to spooling/truncation or smaller commands.
• codewhale stats --days 30 --models 5 → hit rate dropped 96% → 70% last week → something (tool-catalog churn, model/effort switches, compaction) is invalidating the cache; slice 3 names the cause.
• "The model keeps failing at X" → codewhale stats --days 30 --tools 10 shows bash failing 30% with the same usage error → fix lands in AGENTS.md, next sessions stop repeating it.
• Extended thinking on → per-model reasoning split shows thinking is 40% of spend → tune reasoning_effort.
• codewhale stats --days 30 --roles 5 (or codewhale fleet stats) → role explore shows 70% of sub-agent cost with 30% verifier-failure receipts → fix the explore role instructions / verifier scorers.

Alternatives considered

• opencode stats — per-model tokens/cost/cache read-write with --days/--tools/--models/--project; but no cache hit rate (must hand-compute), no per-component request breakdown, no tool errors. Real output below — this is the target shape.
• Claude Code /usage + /context — closest: per-model usage, prompt-cache line with misses/expected rebuilds, category breakdown with optimization suggestions. Missing: per-tool sinks, compaction cost ledger, error analytics; and it's in-session only.
• VTCode token_budget_breakdown — per-request component metrics + spooling; closest on slice 1. Requires HF tokenizers for per-component tracking (follow-up option).
• rtk-style shell hooks — measure pre-agent trimming savings; cannot see actual in-context cost of built-in tools, no error patterns, requires an external wrapper. Real output below.
• Chosen: persist + aggregate inside the runtime — covers all built-in tools, measures real context cost, adds reliability signals no competitor has. In-session TUI /tokens enrichments remain an open question (see below).

Impact

Every session; high value for token-spend debugging, cache tuning, and turning "model keeps failing" from anecdote into a data-driven fix. All source data already exists and persists — this is aggregation + presentation, not new capture.

Additional context

Precedent: opencode (real output, opencode stats --days 30 --models 8)

This is the actual current output of opencode stats — per-model Cache Read / Cache Write are already rendered. But there is no cache hit rate anywhere — you must compute it by hand per model (e.g. deepseek-v4-flash: 3473.7 / (3473.7+110.1) ≈ 96.9%):

┌────────────────────────────────────────────────────────┐
│                       OVERVIEW                         │
├────────────────────────────────────────────────────────┤
│Sessions                                            976 │
│Messages                                         68,219 │
│Days                                                 30 │
└────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────┐
│                    COST & TOKENS                       │
├────────────────────────────────────────────────────────┤
│Total Cost                                       $14.49 │
│Avg Cost/Day                                      $0.48 │
│Avg Tokens/Session                                 9.1M │
│Median Tokens/Session                              1.4M │
│Input                                            445.4M │
│Output                                            23.8M │
│Cache Read                                      8402.2M │
│Cache Write                                           0 │
└────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────┐
│                      MODEL USAGE                       │
├────────────────────────────────────────────────────────┤
│ commandcode/deepseek/deepseek-v4-flash                 │
│  Messages                                       25,371 │
│  Input Tokens                                   110.1M │
│  Output Tokens                                   14.7M │
│  Cache Read                                    3473.7M │   → hit ≈ 96.9%
│  Cache Write                                         0 │
│  Cost                                          $0.0000 │
├────────────────────────────────────────────────────────┤
│ opencode/x-preview-f-free                              │
│  Messages                                       17,109 │
│  Input Tokens                                   162.6M │
│  Output Tokens                                   10.1M │
│  Cache Read                                    1941.9M │   → hit ≈ 92.3%
│  Cache Write                                         0 │
│  Cost                                          $0.0000 │
├────────────────────────────────────────────────────────┤
│ opencode-go/deepseek-v4-flash                          │
│  Messages                                       10,460 │
│  Input Tokens                                    85.9M │
│  Output Tokens                                    7.9M │
│  Cache Read                                    1946.7M │   → hit ≈ 95.8%
│  Cache Write                                         0 │
│  Cost                                         $14.4830 │
├────────────────────────────────────────────────────────┤

(Hand-computed hit rates annotated inline with →; opencode prints no such column.)

Precedent: rtk hook (real output)

Users already track token leakage manually with shell-wrapper hooks. This real report shows where tokens actually go across 89k commands (hook-side savings, i.e. what the wrapper trimmed before the agent saw it):

RTK Token Savings (Global Scope)
════════════════════════════════════════════════════════════
Total commands:    89010
Input tokens:      2006.2M
Output tokens:     576.9M
Tokens saved:      1429.3M (71.2%)
Efficiency meter: █████████████████░░░░░░░ 71.2%

By Command
─────────────────────────────────────────────────────────────────────────
  #  Command                   Count    Saved    Avg%    Time  Impact
 1.  rtk read                   2343  1030.7M   23.2%    45ms  ██████████
 2.  rtk grep                  12298   158.4M   23.9%   197ms  ██░░░░░░░░
 3.  rtk git diff output/p...      3   105.2M  100.0%    1.2s  █░░░░░░░░░
 4.  rtk git diff -- outpu...      1    36.5M  100.0%    1.2s  ░░░░░░░░░░
 5.  rtk diff                     27    35.2M   64.3%   153ms  ░░░░░░░░░░
 6.  rtk find                    535    24.3M   12.7%   789ms  ░░░░░░░░░░
 7.  rtk:toml ps aux             286     8.9M   97.5%    51ms  ░░░░░░░░░░
 8.  rtk:toml jq ...              5     4.4M  100.0%   288ms  ░░░░░░░░░░
9.  rtk:toml jq ...              11     3.2M   99.9%    75ms  ░░░░░░░░░░
10.  rtk psql postgresql:/...      1     2.6M  100.0%   652ms  ░░░░░░░░░░
─────────────────────────────────────────────────────────────────────────

Key insight: read is the single biggest sink (1030.7M = 72% of all savings), then grep (158M), then git diff (105M) — the same tools dominate both hook-side savings and (we should verify) in-context cost. CW can measure the actual in-context footprint of its own built-in tools without any external hook.

Code refs (data already exists)

• Persisted session stats: crates/tui/src/session_manager.rs — SavedSession (total_tokens, cost: SessionCostSnapshot incl. session/subagent USD+CNY, high-water), sessions dir + journal (#5262).
• In-session surfaces: /tokens//cost (crates/tui/src/commands/groups/debug/tokens.rs), /cache (.../debug/cache.rs), TUI Context panel (crates/tui/src/tui/app.rs:1826 context_panel, work_surface/views.rs).
• Tool-result sizes/truncation: crates/tui/src/tools/truncate.rs (spillover + bounded preview), crates/tui/src/tools/web/extract.rs (format=markdown|text|raw).
• Compaction ownership: crates/tui/src/core/engine.rs (runtime_cost_owner), crates/tui/src/runtime_threads.rs.
• Thinking replay tokens: crates/tui/src/client.rs ("multi-turn thinking-mode conversation should report replay tokens").

Related work

• Shipped: feat(tui): show live session token totals #5624 (live session token totals), feat(tui): report post-compaction input tokens #5623 (post-compaction input tokens)
• Open, related: Session metrics: the tok/s number doesn't look right — audit the denominator and the token source #5977 (tok/s audit), Context pressure warning is transient and the agent does not proactively react to it #5620 (agent doesn't react to context pressure), Event-granularity audit: surfaces that stall at turn boundaries despite live per-step sources #5581 (event-granularity audit — feeds error analytics), Fleet/agents: first-class sub-agent and workflow management in the TUI (agents bottom view: live list, per-agent status/elapsed/tokens, focus, message, stop, history over time) #5479 (fleet/agents TUI view)
• Outbox/events (can feed error analytics): lifecycle outbox [lifecycle_outbox] in docs/CONFIGURATION.md
• Fleet ledger / status: codewhale fleet status|inspect|logs|artifacts, durable ledger .codewhale/fleet.jsonl, Runtime API /v1/fleet/runs, /v1/fleet/runs/{id}/workers (docs/FLEET.md "Status Surfaces"); SessionCostSnapshot.subagent_cost_usd/cny in session_manager.rs; /fleet workers = session sub-agents view (/subagents compat).

Open questions (decide during implementation)

• TUI surface: /tokens (and subcommands like /tokens models, /tokens tools, /tokens errors) as in-TUI commands are an open question — the TUI Context panel already covers live session view, and the confirmed deliverable here is the global codewhale stats CLI. Whether the same reports should also be reachable from inside the TUI can be decided separately.
• Duplicate / re-send waste ("token survival") — how many tokens re-enter history verbatim (same file read twice, git diff + read of same files)? Measurable via content hashing (VTCode drops duplicate reads during compaction). Worth the complexity? Measure first, optionally dedupe later.
• Tokenizer-estimate accuracy (unique) — Codewhale uses ~4 chars/token and marks windows (unverified). Accumulate estimate-vs-provider-actual bias per request. Nobody ships this; great for budget debugging. How much is derivable from TurnUsage receipts?

Acceptance criteria

• codewhale stats --days N prints: sessions, messages, total cost, tokens (input/output/cache read/write), per-session average/median tokens
• codewhale stats --models N prints per-model rows: input/output, cache read/write, cost, cache hit %, reasoning split
• codewhale stats --tools N prints per-tool rows: calls, tokens added to history, truncated/spooled share, failure rate + top error
• codewhale stats --project <id> filters by project
• compaction ledger (count, trigger, summarizer tokens/cost, history dropped) visible in stats or a dedicated subcommand
• --days/--models/--tools/--project behave like opencode's flags
• sub-agent/fleet stats: per-role (incl. custom) and per-worker calls, tokens, cost, status counts, failure sources — in stats --roles N or codewhale fleet stats

[github-actions]

Thanks @7jrxt42BxFZo4iAnN4CX for the report.

This issue is staying open for maintainer triage. CodeWhale gets better because people bring us real edge cases from real machines, providers, regions, and workflows.

If you can add a reproduction, logs, version output, screenshots, or the provider/model involved, that makes it much easier for us to verify and harvest the fix. Maintainers may comment /lgtmi to mark recurring issue reporters as approved so this intake note is skipped next time.