diff --git a/.github/ISSUE_TEMPLATE/6-healthcheck-first-run.yml b/.github/ISSUE_TEMPLATE/6-healthcheck-first-run.yml new file mode 100644 index 0000000..9f4338b --- /dev/null +++ b/.github/ISSUE_TEMPLATE/6-healthcheck-first-run.yml @@ -0,0 +1,64 @@ +name: Healthcheck first run or installation feedback +description: Report a useful result, confusing result, or installation blocker. No raw logs needed. +title: "[Healthcheck] " +body: + - type: markdown + attributes: + value: | + Thanks for trying the engineering preview. Feedback is optional. + Do not upload raw logs, full JSON reports, HTML reports, or snapshots. + They may contain local paths and identifiers. Use `agentmeasure share report.json` + to preview an aggregate summary; inspect the exported summary yourself before posting. + - type: dropdown + id: outcome + attributes: + label: What happened? + options: + - Installation blocked + - Installed, but own-data check blocked + - Own-data report generated, but result unclear + - Own-data report generated and understood + - Synthetic demo only + - Meaningful repeat run or comparison + validations: + required: true + - type: input + id: environment + attributes: + label: OS, Python, AgentMeasure version, and Codex Desktop or CLI + placeholder: "macOS; Python 3.11; AgentMeasure 0.3.0; Codex Desktop" + validations: + required: true + - type: textarea + id: result + attributes: + label: What did you learn, or where did you get stuck? + description: Explain one result in your own words and your next step. For blockers, redact any personal paths from errors. + validations: + required: true + - type: input + id: time + attributes: + label: Optional time to install and time to first own-data report + placeholder: "Installation 2 min; own-data report 20 sec" + - type: textarea + id: summary + attributes: + label: Optional reviewed aggregate summary + description: Only the output of the share command, reviewed by you. You may leave this blank. + - type: dropdown + id: first_touch + attributes: + label: Optional — where did you first hear about AgentMeasure? + options: + - GitHub issue, PR, or Discussion + - Personal email + - X + - Search, documentation, or resource list + - A colleague or community + - Other or unknown + - type: input + id: conversion_touch + attributes: + label: Optional — what made you actually try it? + description: A case study, email, integration, or another reason; no private email contents needed. diff --git a/README.md b/README.md index 0618cc0..2a04dad 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,32 @@ # AgentMeasure +**Find repeated failures and retries in your Codex sessions, with local evidence.** + +Healthcheck reads existing **Codex Desktop rollout logs** and produces a terminal +summary and a local HTML report. It checks duplicate records, retry chains, and +consecutive tool failures. Missing evidence is **UNPROVABLE**, never silently zero. + +```bash +# Requires Python 3.9+, Git, and pipx. Installation uses the network. +pipx install "git+https://github.com/roy-tong/AgentMeasure#subdirectory=healthcheck" +agentmeasure demo # synthetic example; no personal logs needed +agentmeasure check # your local Codex sessions, last 7 days +``` + +Analysis runs locally with no runtime network calls. **Engineering preview**: +Codex CLI is not yet independently verified; Claude Code is not supported yet. +The Git install above is available now; PyPI publishing is being prepared. + +[**Quick start and supported formats**](healthcheck/README.md) · +[**Try it and share feedback safely**](campaigns/healthcheck-first-run.md) · +[**What our contributions changed**](campaigns/measurement-casebook.md) · +[中文](README.zh-CN.md) + +Found a useful result? Keep a snapshot and compare your next run. Feedback is +optional: you can use the tool without opening an issue or uploading logs. + +## Measurement infrastructure + **The open yardstick for agent usage and AI outcomes.** **Test whether your agent metrics mean what their labels claim.** diff --git a/README.zh-CN.md b/README.zh-CN.md index 017302b..f0ffa97 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -1,5 +1,28 @@ # AgentMeasure +**用本地证据,找出 Codex 会话中的重复记录、重试和连续工具失败。** + +Healthcheck 直接读取已有的 **Codex Desktop rollout 日志**,生成终端摘要和 +本地 HTML 报告。每项发现有证据;无法判断就显示 UNPROVABLE,不伪装成零。 + +```bash +# 需已有 Python 3.9+、Git、pipx;安装需要网络 +pipx install "git+https://github.com/roy-tong/AgentMeasure#subdirectory=healthcheck" +agentmeasure demo # 合成示例,无需个人日志 +agentmeasure check # 本机 Codex 最近 7 天日志 +``` + +分析过程本地运行,不发网络请求。当前为工程预览版:Codex CLI 尚未独立验证, +Claude Code 暂不支持。上面的 Git 安装现在可用,PyPI 发布仍在准备中。 + +[**快速开始与支持范围**](healthcheck/README.md) · +[**试跑和安全反馈**](campaigns/healthcheck-first-run.md) · +[**公开贡献案例**](campaigns/measurement-casebook.md) · [English](README.md) + +有用的结果可以保存快照,下次运行时比较。无需上传日志或开 issue 才能使用。 + +## 计量基础设施 + **Agent 经济缺一把公尺——AI 用了什么、干得怎么样,行业还没有统一的算法。** **度量 Agent 的真实使用——别把重试当成用户。** diff --git a/campaigns/30-projects-30-days.md b/campaigns/30-projects-30-days.md index bf0dc43..1692797 100644 --- a/campaigns/30-projects-30-days.md +++ b/campaigns/30-projects-30-days.md @@ -25,7 +25,7 @@ This sprint asks each project one question: | # | Project | Surface checked | Invariant | Status | Upstream artifact | | --- | --- | --- | --- | --- | --- | -| 1 | [OpenLIT](https://github.com/openlit/openlit) | token accounting | token subset (reasoning ⊂ output) | **FAIL → fixed** | [PR #1476 merged](https://github.com/openlit/openlit/pull/1476) | +| 1 | [OpenLIT](https://github.com/openlit/openlit) | token accounting | token subset (reasoning ⊂ output) | **subset attribute + invariant merged** | [PR #1476 merged](https://github.com/openlit/openlit/pull/1476) | | 2 | [Urusilla](https://github.com/jaden3824/urusilla) | operation aggregation (checked AgentMeasure itself) | execution / reconciliation / operation grain | **3 FAILs → fixed** | vectors [001](../conformance/vectors/external/urusilla-001/) · [002](../conformance/vectors/external/urusilla-002/) · issues [#8](https://github.com/roy-tong/AgentMeasure/issues/8) [#9](https://github.com/roy-tong/AgentMeasure/issues/9) [#12](https://github.com/roy-tong/AgentMeasure/issues/12) · CI integration [PR jaden3824/urusilla#15](https://github.com/jaden3824/urusilla/pull/15) | | 3 | [pydantic-ai](https://github.com/pydantic/pydantic-ai) | OTel usage attributes | token subset emission | **intended / consumer-contract gap** | [issue #7975](https://github.com/pydantic/pydantic-ai/issues/7975) — emission confirmed intended; the consumer subset contract is unstated; routing to semconv | | 4 | [LiteLLM](https://github.com/BerriAI/litellm) | cache-hit usage logging | cache accounting | **confirmed by third party** | [issue #39057](https://github.com/BerriAI/litellm/issues/39057) — independent dev confirmed the ambiguity changes budget enforcement, not just reports | @@ -40,6 +40,15 @@ This sprint asks each project one question: | 14 | [Phoenix](https://github.com/Arize-ai/phoenix) | cost attribution | total-only span classification | reviewing | [issue #15840](https://github.com/Arize-ai/phoenix/issues/15840) | | 12 | [Langfuse](https://github.com/langfuse/langfuse) | OTel vs REST ingestion | cache normalization | **current paths normalized; residual scope unverified** | [issue #16884 correction](https://github.com/langfuse/langfuse/issues/16884#issuecomment-5502836090) — the original 2.3× claim used stale line references and was withdrawn | +| 15 | [Weave](https://github.com/wandb/weave) | usage aggregation | self+descendants double counting | reviewing | [issue #7831](https://github.com/wandb/weave/issues/7831) | +| 16 | [OpenInference](https://github.com/Arize-ai/openinference) | openai instrumentation | retry invisible in single span | reviewing | [issue #3664](https://github.com/Arize-ai/openinference/issues/3664) | +| 17 | [TruLens](https://github.com/truera/trulens) | leaderboard aggregation | avg masquerading as total; mixed currency sum | reviewing | [issue #2759](https://github.com/truera/trulens/issues/2759) | +| 18 | [Helicone](https://github.com/Helicone/helicone) | cost calculation | accepted_prediction_tokens double count | reviewing | [issue #5805](https://github.com/Helicone/helicone/issues/5805) | +| 19 | [Braintrust](https://github.com/braintrustdata/braintrust-sdk-python) | span idempotency | span_parents skip on merge | reviewing | (drafted, template blocked) | +| 20 | [LiteLLM](https://github.com/BerriAI/litellm) | mid-stream fallback | partial usage loss asymmetry | reviewing | [issue #39462](https://github.com/BerriAI/litellm/issues/39462) | +| 21 | [AgentOps](https://github.com/AgentOps-AI/agentops) | semconv naming | old prompt_tokens vs input_tokens | reviewing | [issue #1447](https://github.com/AgentOps-AI/agentops/issues/1447) | +| 22 | [OpenLLMetry](https://github.com/traceloop/openllmetry) | anthropic reasoning | reasoning subset not emitted | reviewing | [issue #4458](https://github.com/traceloop/openllmetry/issues/4458) | + *(Rows update as checks complete. Status meanings: checking = audit in progress; reviewing = upstream issue open; discussing = public thread active; PASS/FAIL/UNPROVABLE = invariant verdict recorded; fixed = upstream accepted @@ -56,14 +65,6 @@ a change.)* | cost preservation | does grouping remove or duplicate real attempt cost? | | evidence boundary | are returned / available / influential conflated? | | eval repeatability | are n runs n measurements, or retries of one verdict? | -| 15 | [Weave](https://github.com/wandb/weave) | usage aggregation | self+descendants double counting | reviewing | [issue #7831](https://github.com/wandb/weave/issues/7831) | -| 16 | [OpenInference](https://github.com/Arize-ai/openinference) | openai instrumentation | retry invisible in single span | reviewing | [issue #3664](https://github.com/Arize-ai/openinference/issues/3664) | -| 17 | [TruLens](https://github.com/truera/trulens) | leaderboard aggregation | avg masquerading as total; mixed currency sum | reviewing | [issue #2759](https://github.com/truera/trulens/issues/2759) | -| 18 | [Helicone](https://github.com/Helicone/helicone) | cost calculation | accepted_prediction_tokens double count | reviewing | [issue #5805](https://github.com/Helicone/helicone/issues/5805) | -| 19 | [Braintrust](https://github.com/braintrustdata/braintrust-sdk-python) | span idempotency | span_parents skip on merge | reviewing | (drafted, template blocked) | -| 20 | [LiteLLM](https://github.com/BerriAI/litellm) | mid-stream fallback | partial usage loss asymmetry | reviewing | [issue #39462](https://github.com/BerriAI/litellm/issues/39462) | -| 21 | [AgentOps](https://github.com/AgentOps-AI/agentops) | semconv naming | old prompt_tokens vs input_tokens | reviewing | [issue #1447](https://github.com/AgentOps-AI/agentops/issues/1447) | -| 22 | [OpenLLMetry](https://github.com/traceloop/openllmetry) | anthropic reasoning | reasoning subset not emitted | reviewing | [issue #4458](https://github.com/traceloop/openllmetry/issues/4458) | ## Discipline @@ -78,5 +79,9 @@ a change.)* ## Follow along +[Try Healthcheck on local Codex logs](healthcheck-first-run.md) · +[Read the evidence casebook](measurement-casebook.md). Open issues and proposed +PRs are work in progress, not accepted integrations or product adoption. + Repo → [conformance pack](../conformance/pack/README.md) (run the same checks on your own fixture) · X → [@elliwoodtong](https://x.com/elliwoodtong) diff --git a/campaigns/healthcheck-first-run.md b/campaigns/healthcheck-first-run.md new file mode 100644 index 0000000..d8ea6df --- /dev/null +++ b/campaigns/healthcheck-first-run.md @@ -0,0 +1,64 @@ +# Try Healthcheck on your own Codex logs + +We are looking for the first five non-author testers to help verify installation +and whether the report is understandable. This is an engineering preview, not a +claim that five testers have already completed it. No account, star, issue, or +data upload is required to use it. + +## Run + +Requires Python 3.9+, Git and pipx. Installation downloads the package; analysis +is local and has zero runtime dependencies. Supported samples currently come +from Codex Desktop; standalone Codex CLI remains unverified and Claude Code is +not supported. + +```bash +pipx install "git+https://github.com/roy-tong/AgentMeasure#subdirectory=healthcheck" +agentmeasure demo +agentmeasure check --json report.json +``` + +Without pipx, install into a Python virtual environment using pip and the same +Git URL. See [the full guide](../healthcheck/README.md). + +Note separately how long installation and the first own-data report take. +An empty or unsupported directory is useful feedback, but is not a successful +own-data run. The demo is synthetic and never counts as an own-data run. + +Try answering: **What did the report establish, and what would you do next?** +Finding no issue can be useful too, provided coverage supports that conclusion. +These are log checks, not a verdict on overall agent quality or task success. + +## Optional feedback, with a preview + +The HTML report, full `report.json`, and snapshots are personal artifacts. Do +not attach them to a public issue: they may contain local paths and identifiers. + +```bash +agentmeasure share report.json # preview; writes nothing +agentmeasure share report.json --out summary.md # export aggregate summary +``` + +Open `summary.md` yourself before sharing. It contains aggregate counts, without +prompts, commands, paths, project names, or session IDs. You can also omit the +summary and describe only the installation problem or confusing result. + +[**Open a first-run feedback issue**](https://github.com/roy-tong/AgentMeasure/issues/new?template=6-healthcheck-first-run.yml) + +Please include OS, Python and tool version, whether the data is your own or the +demo, and what was useful or blocked you. How you first found us and what made +you try it are optional, separate questions. Feedback is not an endorsement and +opening an issue does not automatically count as successful use. + +## Come back after a change + +```bash +agentmeasure check --save-snapshot before.json +# Make an actual change and run your agent again. +agentmeasure check --save-snapshot after.json +agentmeasure compare before.json after.json +``` + +Describe the change and whether the comparison informed a decision. Different +windows or workloads limit comparisons; metric deltas alone do not prove the +change caused an improvement. Keep the snapshots local. diff --git a/campaigns/measurement-casebook.md b/campaigns/measurement-casebook.md new file mode 100644 index 0000000..b451f41 --- /dev/null +++ b/campaigns/measurement-casebook.md @@ -0,0 +1,52 @@ +# Measurement contributions with inspectable evidence + +These are engineering contributions and counterexamples, not product adoption +claims or endorsements. Each case links to its public source and states what +the evidence does not establish. + +## Exposing reasoning tokens without adding them twice + +[OpenLIT PR #1476](https://github.com/openlit/openlit/pull/1476) was merged on +2026-08-26. The Python/OpenAI change exposes reasoning output tokens for chat +and Responses paths while preserving input/output as the token-usage metric's +categories. The subset invariant is explicit: with output 1,000 and reasoning +700, output usage remains 1,000, not 1,700. + +That is a contribution to telemetry and regression protection. It does not +show that OpenLIT adopted AgentMeasure Healthcheck, and does not establish that +every previous OpenLIT path double-counted output. + +## An external fixture found defects in our own checker + +Urusilla's project-authored synthetic fixtures uncovered our validator's +`oneOf` sibling-constraint gap and declared-operation reconciliation gap: +[issue #8](https://github.com/roy-tong/AgentMeasure/issues/8) and +[issue #9](https://github.com/roy-tong/AgentMeasure/issues/9). +The [first vector](../conformance/vectors/external/urusilla-001/) became a +regression fixture. A [second vector](../conformance/vectors/external/urusilla-002/) +exercised a different operation-grain boundary, tracked in +[issue #12](https://github.com/roy-tong/AgentMeasure/issues/12). + +The lesson is useful even if you never install AgentMeasure: preserve attempt +costs while checking declared operation summaries against their underlying +attempts. A successful synthetic fixture is not production usage or a live +provider-cost observation. + +[Urusilla PR #15](https://github.com/jaden3824/urusilla/pull/15) proposes running +the generic conformance checks in CI. As checked on 2026-09-06, it is open; +review fixes have been pushed, but upstream workflow approval and merging are +still pending. It complements Urusilla's fixture-specific validators rather +than replacing them. + +## Try the local product or contribute a bounded example + +For Codex Desktop logs, [try Healthcheck](healthcheck-first-run.md). For generic +FMT-002 event fixtures, use the [conformance pack](../conformance/pack/README.md). +These are different input paths; Healthcheck does not require converting your +logs into FMT-002. + +Useful contributions include a minimal synthetic parsing counterexample, a +repeat-run snapshot consumer, or a documentation correction with a versioned +source. Start from a real question and a small reproducer. Do not share private +logs or infer that a missing field means zero. See the +[campaign tracker](30-projects-30-days.md) for existing work before duplicating it.