roy-tong · roy-tong · Sep 6, 2026 · Sep 6, 2026
diff --git a/.github/ISSUE_TEMPLATE/6-healthcheck-first-run.yml b/.github/ISSUE_TEMPLATE/6-healthcheck-first-run.yml
@@ -0,0 +1,64 @@
+name: Healthcheck first run or installation feedback
+description: Report a useful result, confusing result, or installation blocker. No raw logs needed.
+title: "[Healthcheck] "
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for trying the engineering preview. Feedback is optional.
+        Do not upload raw logs, full JSON reports, HTML reports, or snapshots.
+        They may contain local paths and identifiers. Use `agentmeasure share report.json`
+        to preview an aggregate summary; inspect the exported summary yourself before posting.
+  - type: dropdown
+    id: outcome
+    attributes:
+      label: What happened?
+      options:
+        - Installation blocked
+        - Installed, but own-data check blocked
+        - Own-data report generated, but result unclear
+        - Own-data report generated and understood
+        - Synthetic demo only
+        - Meaningful repeat run or comparison
+    validations:
+      required: true
+  - type: input
+    id: environment
+    attributes:
+      label: OS, Python, AgentMeasure version, and Codex Desktop or CLI
+      placeholder: "macOS; Python 3.11; AgentMeasure 0.3.0; Codex Desktop"
+    validations:
+      required: true
+  - type: textarea
+    id: result
+    attributes:
+      label: What did you learn, or where did you get stuck?
+      description: Explain one result in your own words and your next step. For blockers, redact any personal paths from errors.
+    validations:
+      required: true
+  - type: input
+    id: time
+    attributes:
+      label: Optional time to install and time to first own-data report
+      placeholder: "Installation 2 min; own-data report 20 sec"
+  - type: textarea
+    id: summary
+    attributes:
+      label: Optional reviewed aggregate summary
+      description: Only the output of the share command, reviewed by you. You may leave this blank.
+  - type: dropdown
+    id: first_touch
+    attributes:
+      label: Optional — where did you first hear about AgentMeasure?
+      options:
+        - GitHub issue, PR, or Discussion
+        - Personal email
+        - X
+        - Search, documentation, or resource list
+        - A colleague or community
+        - Other or unknown
+  - type: input
+    id: conversion_touch
+    attributes:
+      label: Optional — what made you actually try it?
+      description: A case study, email, integration, or another reason; no private email contents needed.
diff --git a/README.md b/README.md
@@ -1,5 +1,32 @@
 # AgentMeasure
 
+**Find repeated failures and retries in your Codex sessions, with local evidence.**
+
+Healthcheck reads existing **Codex Desktop rollout logs** and produces a terminal
+summary and a local HTML report. It checks duplicate records, retry chains, and
+consecutive tool failures. Missing evidence is **UNPROVABLE**, never silently zero.
+
+```bash
+# Requires Python 3.9+, Git, and pipx. Installation uses the network.
+pipx install "git+https://github.com/roy-tong/AgentMeasure#subdirectory=healthcheck"
+agentmeasure demo   # synthetic example; no personal logs needed
+agentmeasure check  # your local Codex sessions, last 7 days
+```
+
+Analysis runs locally with no runtime network calls. **Engineering preview**:
+Codex CLI is not yet independently verified; Claude Code is not supported yet.
+The Git install above is available now; PyPI publishing is being prepared.
+
+[**Quick start and supported formats**](healthcheck/README.md) ·
+[**Try it and share feedback safely**](campaigns/healthcheck-first-run.md) ·
+[**What our contributions changed**](campaigns/measurement-casebook.md) ·
+[中文](README.zh-CN.md)
+
+Found a useful result? Keep a snapshot and compare your next run. Feedback is
+optional: you can use the tool without opening an issue or uploading logs.
+
+## Measurement infrastructure
+
 **The open yardstick for agent usage and AI outcomes.**
 **Test whether your agent metrics mean what their labels claim.**
 

diff --git a/README.zh-CN.md b/README.zh-CN.md
@@ -1,5 +1,28 @@
 # AgentMeasure
 
+**用本地证据，找出 Codex 会话中的重复记录、重试和连续工具失败。**
+
+Healthcheck 直接读取已有的 **Codex Desktop rollout 日志**，生成终端摘要和
+本地 HTML 报告。每项发现有证据；无法判断就显示 UNPROVABLE，不伪装成零。
+
+```bash
+# 需已有 Python 3.9+、Git、pipx；安装需要网络
+pipx install "git+https://github.com/roy-tong/AgentMeasure#subdirectory=healthcheck"
+agentmeasure demo   # 合成示例，无需个人日志
+agentmeasure check  # 本机 Codex 最近 7 天日志
+```
+
+分析过程本地运行，不发网络请求。当前为工程预览版：Codex CLI 尚未独立验证，
+Claude Code 暂不支持。上面的 Git 安装现在可用，PyPI 发布仍在准备中。
+
+[**快速开始与支持范围**](healthcheck/README.md) ·
+[**试跑和安全反馈**](campaigns/healthcheck-first-run.md) ·
+[**公开贡献案例**](campaigns/measurement-casebook.md) · [English](README.md)
+
+有用的结果可以保存快照，下次运行时比较。无需上传日志或开 issue 才能使用。
+
+## 计量基础设施
+
 **Agent 经济缺一把公尺——AI 用了什么、干得怎么样，行业还没有统一的算法。**
 **度量 Agent 的真实使用——别把重试当成用户。**
 

diff --git a/campaigns/30-projects-30-days.md b/campaigns/30-projects-30-days.md
@@ -25,7 +25,7 @@ This sprint asks each project one question:
 
 | # | Project | Surface checked | Invariant | Status | Upstream artifact |
 | --- | --- | --- | --- | --- | --- |
-| 1 | [OpenLIT](https://github.com/openlit/openlit) | token accounting | token subset (reasoning ⊂ output) | **FAIL → fixed** | [PR #1476 merged](https://github.com/openlit/openlit/pull/1476) |
+| 1 | [OpenLIT](https://github.com/openlit/openlit) | token accounting | token subset (reasoning ⊂ output) | **subset attribute + invariant merged** | [PR #1476 merged](https://github.com/openlit/openlit/pull/1476) |
 | 2 | [Urusilla](https://github.com/jaden3824/urusilla) | operation aggregation (checked AgentMeasure itself) | execution / reconciliation / operation grain | **3 FAILs → fixed** | vectors [001](../conformance/vectors/external/urusilla-001/) · [002](../conformance/vectors/external/urusilla-002/) · issues [#8](https://github.com/roy-tong/AgentMeasure/issues/8) [#9](https://github.com/roy-tong/AgentMeasure/issues/9) [#12](https://github.com/roy-tong/AgentMeasure/issues/12) · CI integration [PR jaden3824/urusilla#15](https://github.com/jaden3824/urusilla/pull/15) |
 | 3 | [pydantic-ai](https://github.com/pydantic/pydantic-ai) | OTel usage attributes | token subset emission | **intended / consumer-contract gap** | [issue #7975](https://github.com/pydantic/pydantic-ai/issues/7975) — emission confirmed intended; the consumer subset contract is unstated; routing to semconv |
 | 4 | [LiteLLM](https://github.com/BerriAI/litellm) | cache-hit usage logging | cache accounting | **confirmed by third party** | [issue #39057](https://github.com/BerriAI/litellm/issues/39057) — independent dev confirmed the ambiguity changes budget enforcement, not just reports |
@@ -40,6 +40,15 @@ This sprint asks each project one question:
 | 14 | [Phoenix](https://github.com/Arize-ai/phoenix) | cost attribution | total-only span classification | reviewing | [issue #15840](https://github.com/Arize-ai/phoenix/issues/15840) |
 | 12 | [Langfuse](https://github.com/langfuse/langfuse) | OTel vs REST ingestion | cache normalization | **current paths normalized; residual scope unverified** | [issue #16884 correction](https://github.com/langfuse/langfuse/issues/16884#issuecomment-5502836090) — the original 2.3× claim used stale line references and was withdrawn |
 
+| 15 | [Weave](https://github.com/wandb/weave) | usage aggregation | self+descendants double counting | reviewing | [issue #7831](https://github.com/wandb/weave/issues/7831) |
+| 16 | [OpenInference](https://github.com/Arize-ai/openinference) | openai instrumentation | retry invisible in single span | reviewing | [issue #3664](https://github.com/Arize-ai/openinference/issues/3664) |
+| 17 | [TruLens](https://github.com/truera/trulens) | leaderboard aggregation | avg masquerading as total; mixed currency sum | reviewing | [issue #2759](https://github.com/truera/trulens/issues/2759) |
+| 18 | [Helicone](https://github.com/Helicone/helicone) | cost calculation | accepted_prediction_tokens double count | reviewing | [issue #5805](https://github.com/Helicone/helicone/issues/5805) |
+| 19 | [Braintrust](https://github.com/braintrustdata/braintrust-sdk-python) | span idempotency | span_parents skip on merge | reviewing | (drafted, template blocked) |
+| 20 | [LiteLLM](https://github.com/BerriAI/litellm) | mid-stream fallback | partial usage loss asymmetry | reviewing | [issue #39462](https://github.com/BerriAI/litellm/issues/39462) |
+| 21 | [AgentOps](https://github.com/AgentOps-AI/agentops) | semconv naming | old prompt_tokens vs input_tokens | reviewing | [issue #1447](https://github.com/AgentOps-AI/agentops/issues/1447) |
+| 22 | [OpenLLMetry](https://github.com/traceloop/openllmetry) | anthropic reasoning | reasoning subset not emitted | reviewing | [issue #4458](https://github.com/traceloop/openllmetry/issues/4458) |
+
 *(Rows update as checks complete. Status meanings: checking = audit in
 progress; reviewing = upstream issue open; discussing = public thread active;
 PASS/FAIL/UNPROVABLE = invariant verdict recorded; fixed = upstream accepted
@@ -56,14 +65,6 @@ a change.)*
 | cost preservation | does grouping remove or duplicate real attempt cost? |
 | evidence boundary | are returned / available / influential conflated? |
 | eval repeatability | are n runs n measurements, or retries of one verdict? |
-| 15 | [Weave](https://github.com/wandb/weave) | usage aggregation | self+descendants double counting | reviewing | [issue #7831](https://github.com/wandb/weave/issues/7831) |
-| 16 | [OpenInference](https://github.com/Arize-ai/openinference) | openai instrumentation | retry invisible in single span | reviewing | [issue #3664](https://github.com/Arize-ai/openinference/issues/3664) |
-| 17 | [TruLens](https://github.com/truera/trulens) | leaderboard aggregation | avg masquerading as total; mixed currency sum | reviewing | [issue #2759](https://github.com/truera/trulens/issues/2759) |
-| 18 | [Helicone](https://github.com/Helicone/helicone) | cost calculation | accepted_prediction_tokens double count | reviewing | [issue #5805](https://github.com/Helicone/helicone/issues/5805) |
-| 19 | [Braintrust](https://github.com/braintrustdata/braintrust-sdk-python) | span idempotency | span_parents skip on merge | reviewing | (drafted, template blocked) |
-| 20 | [LiteLLM](https://github.com/BerriAI/litellm) | mid-stream fallback | partial usage loss asymmetry | reviewing | [issue #39462](https://github.com/BerriAI/litellm/issues/39462) |
-| 21 | [AgentOps](https://github.com/AgentOps-AI/agentops) | semconv naming | old prompt_tokens vs input_tokens | reviewing | [issue #1447](https://github.com/AgentOps-AI/agentops/issues/1447) |
-| 22 | [OpenLLMetry](https://github.com/traceloop/openllmetry) | anthropic reasoning | reasoning subset not emitted | reviewing | [issue #4458](https://github.com/traceloop/openllmetry/issues/4458) |
 
 ## Discipline
 
@@ -78,5 +79,9 @@ a change.)*
 
 ## Follow along
 
+[Try Healthcheck on local Codex logs](healthcheck-first-run.md) ·
+[Read the evidence casebook](measurement-casebook.md). Open issues and proposed
+PRs are work in progress, not accepted integrations or product adoption.
+
 Repo → [conformance pack](../conformance/pack/README.md) (run the same checks
 on your own fixture) · X → [@elliwoodtong](https://x.com/elliwoodtong)
diff --git a/campaigns/healthcheck-first-run.md b/campaigns/healthcheck-first-run.md
@@ -0,0 +1,64 @@
+# Try Healthcheck on your own Codex logs
+
+We are looking for the first five non-author testers to help verify installation
+and whether the report is understandable. This is an engineering preview, not a
+claim that five testers have already completed it. No account, star, issue, or
+data upload is required to use it.
+
+## Run
+
+Requires Python 3.9+, Git and pipx. Installation downloads the package; analysis
+is local and has zero runtime dependencies. Supported samples currently come
+from Codex Desktop; standalone Codex CLI remains unverified and Claude Code is
+not supported.
+
+```bash
+pipx install "git+https://github.com/roy-tong/AgentMeasure#subdirectory=healthcheck"
+agentmeasure demo
+agentmeasure check --json report.json
+```
+
+Without pipx, install into a Python virtual environment using pip and the same
+Git URL. See [the full guide](../healthcheck/README.md).
+
+Note separately how long installation and the first own-data report take.
+An empty or unsupported directory is useful feedback, but is not a successful
+own-data run. The demo is synthetic and never counts as an own-data run.
+
+Try answering: **What did the report establish, and what would you do next?**
+Finding no issue can be useful too, provided coverage supports that conclusion.
+These are log checks, not a verdict on overall agent quality or task success.
+
+## Optional feedback, with a preview
+
+The HTML report, full `report.json`, and snapshots are personal artifacts. Do
+not attach them to a public issue: they may contain local paths and identifiers.
+
+```bash
+agentmeasure share report.json                  # preview; writes nothing
+agentmeasure share report.json --out summary.md # export aggregate summary
+```
+
+Open `summary.md` yourself before sharing. It contains aggregate counts, without
+prompts, commands, paths, project names, or session IDs. You can also omit the
+summary and describe only the installation problem or confusing result.
+
+[**Open a first-run feedback issue**](https://github.com/roy-tong/AgentMeasure/issues/new?template=6-healthcheck-first-run.yml)
+
+Please include OS, Python and tool version, whether the data is your own or the
+demo, and what was useful or blocked you. How you first found us and what made
+you try it are optional, separate questions. Feedback is not an endorsement and
+opening an issue does not automatically count as successful use.
+
+## Come back after a change
+
+```bash
+agentmeasure check --save-snapshot before.json
+# Make an actual change and run your agent again.
+agentmeasure check --save-snapshot after.json
+agentmeasure compare before.json after.json
+```
+
+Describe the change and whether the comparison informed a decision. Different
+windows or workloads limit comparisons; metric deltas alone do not prove the
+change caused an improvement. Keep the snapshots local.
diff --git a/campaigns/measurement-casebook.md b/campaigns/measurement-casebook.md
@@ -0,0 +1,52 @@
+# Measurement contributions with inspectable evidence
+
+These are engineering contributions and counterexamples, not product adoption
+claims or endorsements. Each case links to its public source and states what
+the evidence does not establish.
+
+## Exposing reasoning tokens without adding them twice
+
+[OpenLIT PR #1476](https://github.com/openlit/openlit/pull/1476) was merged on
+2026-08-26. The Python/OpenAI change exposes reasoning output tokens for chat
+and Responses paths while preserving input/output as the token-usage metric's
+categories. The subset invariant is explicit: with output 1,000 and reasoning
+700, output usage remains 1,000, not 1,700.
+
+That is a contribution to telemetry and regression protection. It does not
+show that OpenLIT adopted AgentMeasure Healthcheck, and does not establish that
+every previous OpenLIT path double-counted output.
+
+## An external fixture found defects in our own checker
+
+Urusilla's project-authored synthetic fixtures uncovered our validator's
+`oneOf` sibling-constraint gap and declared-operation reconciliation gap:
+[issue #8](https://github.com/roy-tong/AgentMeasure/issues/8) and
+[issue #9](https://github.com/roy-tong/AgentMeasure/issues/9).
+The [first vector](../conformance/vectors/external/urusilla-001/) became a
+regression fixture. A [second vector](../conformance/vectors/external/urusilla-002/)
+exercised a different operation-grain boundary, tracked in
+[issue #12](https://github.com/roy-tong/AgentMeasure/issues/12).
+
+The lesson is useful even if you never install AgentMeasure: preserve attempt
+costs while checking declared operation summaries against their underlying
+attempts. A successful synthetic fixture is not production usage or a live
+provider-cost observation.
+
+[Urusilla PR #15](https://github.com/jaden3824/urusilla/pull/15) proposes running
+the generic conformance checks in CI. As checked on 2026-09-06, it is open;
+review fixes have been pushed, but upstream workflow approval and merging are
+still pending. It complements Urusilla's fixture-specific validators rather
+than replacing them.
+
+## Try the local product or contribute a bounded example
+
+For Codex Desktop logs, [try Healthcheck](healthcheck-first-run.md). For generic
+FMT-002 event fixtures, use the [conformance pack](../conformance/pack/README.md).
+These are different input paths; Healthcheck does not require converting your
+logs into FMT-002.
+
+Useful contributions include a minimal synthetic parsing counterexample, a
+repeat-run snapshot consumer, or a documentation correction with a versioned
+source. Start from a real question and a small reproducer. Do not share private
+logs or infer that a missing field means zero. See the
+[campaign tracker](30-projects-30-days.md) for existing work before duplicating it.