Find repeated failures and retries in your Codex sessions, with local evidence.
Healthcheck reads existing Codex Desktop rollout logs and produces a terminal summary and a local HTML report. It checks duplicate records, retry chains, and consecutive tool failures. Missing evidence is UNPROVABLE, never silently zero.
# Requires Python 3.9+, Git, and pipx. Installation uses the network.
pipx install "git+https://github.com/roy-tong/AgentMeasure#subdirectory=healthcheck"
agentmeasure demo # synthetic example; no personal logs needed
agentmeasure check # your local Codex sessions, last 7 daysAnalysis runs locally with no runtime network calls. Engineering preview: Codex CLI is not yet independently verified; Claude Code is not supported yet. The Git install above is available now; PyPI publishing is being prepared.
Quick start and supported formats · Try it and share feedback safely · What our contributions changed · 中文
Found a useful result? Keep a snapshot and compare your next run. Feedback is optional: you can use the tool without opening an issue or uploading logs.
The open yardstick for agent usage and AI outcomes. Test whether your agent metrics mean what their labels claim.
Conformance checks for AI-agent telemetry — a retry is one logical operation, not two requests; a reasoning-token subset must not be added into totals; a cache hit is not a new measurement. Every check reports PASS / FAIL / UNPROVABLE, and UNPROVABLE is a first-class result: when the evidence to decide is absent, it is disclosed, never zeroed.
# .github/workflows/conformance.yml — turn measurement assumptions into CI checks
- uses: roy-tong/AgentMeasure@2cf476d6f7d0fc45401db5a822e1f12de009ac74
with:
fixture: fixtures/telemetry.jsonl # your FMT-002 event fixtureRun conformance locally · See real failures · Read the spec · 中文
AgentMeasure separates execution facts from logical operations, evidence from inference, and economic settlement from value — an open measurement layer for the Agent Capability Economy. One number for your dashboards (logical operations), the evidence for what each number counts, and explicit disclosure for what it cannot prove.
Today: measure agent-facing capability usage — attempts, operations, retry inflation, success rates with numerators you can audit. Next: make capabilities comparable and meterable — including the outcome units now being billed (what counts as one resolution, one completed task). Long term: provide the measurement foundation for Capability as a Service (CaaS).
Reach → Choice → Use → Utility → Value
AgentMeasure is not a payment protocol, marketplace, or universal ranking system. It standardizes the facts and measurement semantics those systems can build on.
AI services have started charging by the outcome: Zendesk at $1.50–2.00 per automated resolution since Aug 2024, Intercom Fin at $0.99 per resolution with a money-back guarantee, Sierra's outcome-based enterprise contracts. But what counts as one outcome has no standard — retries, reopens, and silent "assumed resolutions" all change the number, and the bill. When money rides on a measured unit, someone has to define the unit. That is what we build. → Read the note
- Free, forever. Spec, engine, SDK, conformance, local dashboard — the free list is part of governance, not revocable later. Adoption is the product.
- Local-first, opt-in. Raw data never leaves your machine by default; only aggregated, anonymized evidence returns, and only if you opt in.
- The referee doesn't open a store. No paid rankings, no recommendation slots, no custody of funds. Neutrality is the product.
Website · Send a trace → get a measurement check · Free 7-day audit — apply · 30 Projects / 30 Days campaign · Whitepaper · Core Specification
Start with the story: When the Software Consumer Becomes an Agent (EN) · 当软件的消费者变成 Agent (ZH)
./examples/demo-e2e.shMock MCP server → canonical observations → local metrics, all on your machine, no
cloud. The demo is reproducible: it runs in an isolated workspace (never touches
~/.agentmeasure) — same fixture + same policy = same result (42 calls → 84
canonical observations, caller claims claude:14 · codex:14 · unknown:14).
Then read why we audit the ecosystem's usage claims: Benchmark Run #001 — six real claims profiled with the Evidence Profile (multi-axis, no composite scores), and Pipeline Validation #001 — our own fixture, kept out of the ranking as a reference baseline.
python3 lab/am lab selftest # planted uplift recovered + honest null
python3 lab/am lab init # workspace + example experiment
python3 lab/am lab preregister am-lab/experiments/example-manifest.json
python3 lab/am lab run am-lab/experiments/example-manifest.prereg.jsonThe open experiment engine (lab/): task set × harness matrix × factor
variants → Reach → Choice → Success → Consumption funnel → effect sizes with
confidence intervals, guardrails, honest nulls, and an offline HTML report that opens
with a bilingual decision-maker one-pager. Preregistration is enforced (hypothesis /
primary metric / guardrails / analysis plan hashed before the run — with a scale /
power / budget preview), seeds replay deterministically, and a budget circuit breaker
stops safely with the data it has. Selection uplift that loses consumption is
rejected at the decision exit (unverified_growth — do not ship), and candidates
that make no more money at higher cost are flagged as dominated.
The shipped demo runs on a synthetic harness (planted ground truth at realistic, literature-scale amplitudes, disclosed in every report) — it validates the engine, not real-agent claims. Real harness adapters (Claude Code / Codex) are runner plugins against the same interface; that is the highest-value contribution right now. Docs: lab/README.md · formats: lab/schemas/ (experiment manifest / funnel events / report).
1 user intent
2 provider attempts
1 final success
That is 1 operation, 2 attempts, operation success 100%, attempt success 50%, attempts per operation 2.
It is not "2 operations with 50% success".
Attempts are execution facts. Operations are logical intents. Mixing the two is how agent reliability reports get distorted.
Start smaller than a trial: send a trace, get a measurement check — 20–100 anonymized trace/log rows (or a public export), mapped locally, with a short report of what safely counts as attempts vs operations, where retries may inflate usage, and what the telemetry cannot prove. No SDK, no integration; raw data stays local unless you explicitly authorize sharing a sanitized sample. A sample check is not a full audit and claims no causal effect.
If the check pays for itself, AgentMeasure is looking for 3 MCP/API providers for short external measurement runs.
The goal is not to prove the model works. It is to find where it breaks on real provider traffic.
A trial can stay fully local, requires no agent-side install, and can start with one capability.
Best feedback: a concrete trace, counterexample, or field that cannot be measured safely.
→ The formal offer, schedule, and FAQ live in product/AUDIT.md · apply in Issue #2 — Looking for 3 MCP/API providers for external measurement runs.
The software consumer is changing from humans to agents, and the economic unit is shifting from software seats toward callable capabilities.
Skill / MCP / CLI / SDK
↓
Describe / expose / distribute a capability
Capability
↓
Data / Compute / Action / Permission / Transaction
↓
Creates scarce economic value
Interfaces may become cheap to create; capabilities remain scarce to deliver.
The first generation of capability distribution is already here — open Skills, open MCP adapters, open CLIs. The scarce layer underneath them is what the next economy is built on: proprietary data, compute, execution, permissions, and real-world fulfillment.
Human Software Economy
User → UI → SaaS → Seat / Month
↓
Agent Capability Economy
Agent → Capability → Execution → Outcome
↓
Usage / Value / Transaction
If a capability is to become an economic unit that agents can discover, compare, and eventually purchase automatically, it must first be identifiable, measurable, and comparable under a shared semantics. That is what AgentMeasure provides.
Traditional usage metrics cannot support this economy — the old chain breaks at every link, and the last link is new:
Install ≠ Available
Available ≠ Presented
Presented ≠ Selected
Selected ≠ Used
Used ≠ Useful
Useful ≠ Incremental Value
Measured Usage ≠ Billable Usage
The last inequality is why the Operation/Attempt model matters commercially: 3 attempts of one operation are not 3 billable operations — unless the metering policy says so.
The extended thesis — economic units, scarcity, measurement before monetization, metering semantics, and the evidence that commerce is arriving before measurement — lives in docs/CAPABILITY-ECONOMY.md.
AgentMeasure defines metric families, not a universal KPI. A search capability, a booking API, and a compute job have different value structures.
| Layer | Question | Representative metrics |
|---|---|---|
| Reach | Did the capability enter the agent's choice range? | Eligible Opportunities · Presentations · Presentation Rate · Distribution Coverage |
| Choice | When the agent had the chance, did it choose it? | Observed Selection Rate · Observed Head-to-Head Choice Share |
| Use | After selection, was it actually used? | Operations · Attempts · Completion Rate · Success Rate |
| Utility | Did it produce a usable result or confirmed effect? | Result Consumption · Effect Confirmation |
| Value | Did it improve the task outcome? | Incremental Task Success（Draft 0.5） |
The five layers are the measurement view. The same facts map onto an economic view:
| CaaS Domain | AgentMeasure |
|---|---|
| Demand | Reach + Choice |
| Delivery | Use + Utility |
| Outcome | Value |
| Economics | Metering / Attribution（future extension） |
Claim discipline throughout: observed choice is not preference. A selection can be made by the model, a router, a workflow, the user, a policy, or the platform; Observed Selection Rate reports what was observed, and Observed Head-to-Head Choice Share is an observed head-to-head choice share under comparable candidate conditions — comparable means the same candidate set, category, choice mode, and decision axes (Decision Authority / Selection Constraint) are declared.
Agent Runtime Capability Provider
Claude / Codex
│
│ MCP / API
▼
┌──────────────────────┐
│ Customer Capability │
│ │
│ AgentMeasure SDK │
│ Business Handler │
└──────────┬───────────┘
│
observations
│
▼
Collector
│
▼
AgentMeasure Cloud
Four things a real developer needs to know:
Your software does not need to be open source.
MCP is not required — it is the first reference surface.
Third-party agents do not need AgentMeasure installed for provider-side usage measurement.
AgentMeasure is not on the critical request path.
Product architecture (Provider SDK → local buffer → hosted ingestion → dashboard): product/ARCHITECTURE.md.
- Decision Opportunity / Candidate Set / Presentation / Selection — the four objects of choice; the Observed Selection Rate denominator is Presented, not Available
- Software Entity → Capability → Interaction Surface — what exists, what it can do, and the observable interface; observation happens on surfaces, attribution resolves to entities through the machine-readable registry
- Operation / Attempt — one logical use vs. one execution; retries are multiple attempts of one operation, not a validity class (they are kept as reliability signals, not counted as distinct logical uses)
- Qualified Usage — production usage after excluding test, benchmark, synthetic, replay, duplicate and other invalid traffic according to policy
- Result Consumption — defined, reference partial: the result was used by the task
- Effect Confirmation — domain model defined, metric planned for Draft 0.5: the intended world-state change was confirmed
- Measurement Label — a nutrition label for every public number (coverage / sampling / policy / method)
The full model lives in the Core Specification, Metrics, Entity, and Quality. The README is not a spec summary.
AgentMeasure is progressively standardizing the measurement chain from discovery and choice through execution, utility and value: core usage semantics are defined today; utility/value and commercial metering remain active drafts. Metering and commercial attribution are future extensions; payment rails can be provided by existing payment infrastructure.
AgentMeasure standardizes economic facts, not money movement.
Extended thesis: docs/CAPABILITY-ECONOMY.md · Economic semantics: extensions/COMMERCIAL.md（Experimental）
Portable semantics need a public record of observation blind spots. Per-harness profiles map each runtime's native objects to AgentMeasure semantics:
| Harness | Profile | Highlights |
|---|---|---|
| Codex | profiles/codex.md | hook 观察，无 trace/精确时间戳；App Server 事件流为优先生效观察面 |
| Claude Code | profiles/claude-code.md | 内置成败判定；第一个 Consumption 可实证平台 |
| DeepSeek Harness | profiles/deepseek-harness.md | append-only session log；subagent lineage/depth 是 Delegation 的首个真实数据源 |
| Pydantic AI | profiles/pydantic-ai.md | Logfire spans → attempt 语义 |
| OpenTelemetry GenAI | profiles/opentelemetry-genai.md | Route B mapping |
As harnesses compose software at runtime, one behavior gets described by different objects and units in different runtimes. Experiment D turns that into evidence; Proposal: Delegation defines the agent-to-agent boundary the object model was missing.
| AgentMeasure is | AgentMeasure is not |
|---|---|
| Measurement standard | Payment protocol |
| Usage analytics foundation | Marketplace |
| Metering semantics | Wallet |
| Comparable quality signals | Universal reputation score |
| Attribution framework | Single global source of truth |
| Audience | Why |
|---|---|
| Capability Provider | measure and eventually meter agent usage of your capabilities |
| Agent Runtime | expose decision / usage signals consistently |
| Registry / Marketplace | compare capabilities using standardized signals |
| Data / Measurement Provider | produce comparable agent-usage analytics |
| Commerce / Payment Infrastructure | consume standardized billable events in future profiles |
| Researchers / Standard Contributors | evolve the methodology |
git clone https://github.com/roy-tong/AgentMeasure && cd AgentMeasure
python3 conformance/runners/run_metrics.py # metric vectors (M2.2 / M2.5 / M4.1)
python3 verify_vectors.py # verification / correlation / operation vectors
python3 registry/validate_entities.py # validate the machine-readable registryThe first product path is Remote MCP / API Capability Measurement: an
AgentMeasure Provider SDK (@agentmeasure/mcp) that emits observations from
the provider side (no agent-side install), feeding a local collector. Local analytics
run without any cloud:
npm install https://github.com/roy-tong/AgentMeasure/releases/download/v0.1.1/agentmeasure-mcp-0.1.1.tgz
# (npm registry publish pending scope/token — tarball is the current install path)
# wrap your MCP server's tool handlers: server.tool = (name, schema, mw.wrapTool(name, handler))
node examples/mcp-integration.js # synthetic traffic → local JSONL
python3 product/local-analytics.py ~/.agentmeasure/events/agentmeasure-events.jsonlStatus: SDK v0.1.1 — External-Ready (canonical output, non-blocking spool with loss accounting, per-request caller, MCP v1/v2, 21 tests, deterministic fixture) + local analytics implemented; hosted ingestion and dashboard next. First real external Provider = Product Gate A (ROADMAP.md, MVP.md).
Scope and acceptance: product/MVP.md · SDK contract: product/PROVIDER-SDK.md · Deployment: product/DEPLOYMENT.md
AgentMeasure/
├── standard/ # the normative standard (CORE / METRICS / QUALITY / DATA / ...)
├── extensions/ # experimental, non-normative profiles (COMMERCIAL.md)
├── product/ # product architecture (SDK / hosted analytics, in development)
├── whitepaper/ # methodology papers (EN/CN)
├── conformance/ # language-neutral test vectors + runners
├── reference/ # reference implementation (collector + adapters)
│ ├── collector/ # normalization, correlation, aggregation, evidence
│ └── adapters/ # codex / claude / dsh / mcp observation adapters
├── schemas/ # machine-readable schemas (entity registry)
├── registry/ # machine-readable registries (entities / project identity)
├── experiments/ # empirical experiment designs
├── reports/ # public reports (Discrepancy Report)
├── proposals/ # standard change proposals (AUP)
└── archive/ # retired early documents
The standard is the artifact; the code is a reference implementation. Using the standard does not mean uploading data to any central server.
Draft 0.4.4（Canonicalization & Reference Convergence） — 唯一 Canonical Observation（schemas/observation.schema.json，6 类 payload）；Choice/Execution 从同一 Envelope 派生；M3.1 只计已解析 operation（无回退）；Attempt 级 qualification 派生； metrics.yaml 单一事实源；四维正交（Evidence/Caller/Use Profile/Billing）。
| Capability | Standard | Reference | Real Runtime |
|---|---|---|---|
| Observed Selection Rate | Defined | Implemented | Limited |
| Observed Head-to-Head Choice Share | Defined | Implemented | Experimental |
| Operations / Attempts | Defined | Implemented | Yes |
| Operation Resolution Coverage | Defined | Implemented | No |
| Result Consumption | Defined | Implemented | Claude partial |
| Incrementality | Defined (formula) | Planned | No |
| Qualified Usage (Strict) | Defined | Implemented | Yes |
The roadmap runs on two tracks — the standard (0.4 objects & quality → 0.5 utility & economic semantics → 1.0) and the product (Remote Capability Analytics → Provider SDK + hosted analytics → metering). See ROADMAP.md.
- Join the community: Discussions — categories: Metric Semantics · Runtime Observation · Experiments · Capability Economy · Implementers; ground rules in docs/DISCUSSIONS.md
- Where to start: the open debate on Strict Qualified Usage as the default, or onboard the first external Provider
- Propose standard changes:
proposals/(AUP: Draft → Discussion → Accepted → Experimental → Stable) - Report measurement discrepancies:
reports/(Discrepancy Report template) - Fix the reference implementation: PRs must pass all
conformance/vectors
AgentMeasure does not define who owns the truth. It defines what evidence, under what rules, can support what conclusions.