feat: TODO batch — OpenRouter cost reconciliation, safer cleanup manifest, pinned harness tests, docs

[tardigrde]

Summary

Works through the remaining TODO.md items (everything except the subagent-evals generalization, which is deferred).

feat(harness): reconcile claude-code cost against OpenRouter pricing

• The claude CLI prices runs at Anthropic list prices regardless of endpoint; cost_usd was wrong whenever claude-code was routed through OpenRouter.
• The OpenRouter generation API would be exact but needs per-request generation ids the CLI doesn't expose, so cost is recomputed from the run's token counts × OpenRouter's published per-model rates (public models endpoint, fetched once per process, failure-safe).
• timing.json gains cost_usd_source (cli / openrouter-pricing / cli-unreconciled) and cost_usd_cli (the CLI's original estimate), plus cache_creation_tokens so cache-write charges are included.

fix(runner): only record eval-created open PRs in cleanup manifest

• The PR snapshot uses gh pr list --state all, so the manifest could record PRs merged/closed during the run — and even PRs someone else opened mid-run, which cleanup would then close with --delete-branch.
• Now a new PR number is recorded only when the PR is still OPEN and its head branch was pushed by this run. Old snapshots without state/headRefName keep the previous behavior.

test(harness)

• Full argv pinned for claude-code, opencode, and codex build_command (was codex-only, flag-presence-only).
• _build_cleanup_entry covered for merged/closed/external/pre-existing/number-less PRs.

docs / examples

• pyproject authors filled with the real maintainer.
• README: python -m agent_skill_eval usage, cost reporting + reconciliation semantics, multi-agent comparison walkthrough (how to read per-agent with/without-skill deltas), stale codex invocation row fixed.
• examples/commit-push-pr/evals/files/sample_change.py expanded from two print lines into a realistic user-directory fixture matching what the eval prompts claim about it (login fix + search feature).

local cleanup (not in the diff)

• Stale merged local branches deleted (confirmed merged via PRs feat: improve commit-push-pr skill to 100% pass rate and add cleanup command #2–0.2.0: pass@k, per-agent deltas, model pinning, validate/list/compare, write-release-notes example #8), both leftover .claude/worktrees/ copies of the old skill_eval module tree removed, stale untracked HANDOFF.md deleted.

Test plan

• make test — 260 passed (includes free e2e tier)
• make lint — clean
• make test-live — 7 passed (all real harnesses, zero skipped grades); details in comment

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces cost reconciliation for Claude models routed through OpenRouter, captures cache creation tokens, and refines PR cleanup logic to exclude merged, closed, or external PRs. It also expands documentation, adds realistic test fixtures, and introduces comprehensive unit tests. Feedback on the changes highlights a concurrency issue in the newly added OpenRouter pricing fetcher, suggesting the use of a thread lock, a custom User-Agent to prevent CDN blocking, and defensive type-checking on the API response.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

The fetch_openrouter_pricing function is called concurrently by multiple threads when evaluations are run in parallel. Without synchronization, multiple threads can concurrently trigger redundant HTTP requests to the OpenRouter API when _pricing_cache is initially None.

Additionally, public APIs protected by Cloudflare or other CDNs often block the default Python urllib User-Agent with a 403 Forbidden error. It is safer to specify a custom User-Agent header. Finally, we should defensively verify that the parsed JSON data is a dictionary before calling .get() to avoid potential AttributeError exceptions if the API returns an unexpected format.

from __future__ import annotations

import json
import re
import threading
import urllib.request
from typing import Optional

from agent_skill_eval.models import TimingData

OPENROUTER_MODELS_URL = "https://openrouter.ai/api/v1/models"
_FETCH_TIMEOUT_SECONDS = 10

# model id -> pricing dict, populated once per process. The models endpoint
# is public (no API key) but we still never want it on a hot path.
_pricing_cache: Optional[dict[str, dict]] = None
_pricing_lock = threading.Lock()


def fetch_openrouter_pricing() -> dict[str, dict]:
    """Return {model_id: pricing} from the OpenRouter models endpoint.

    Cached for the process lifetime. Raises on network/parse errors; callers
    that must not fail (the harness) wrap this in try/except.
    """
    global _pricing_cache
    if _pricing_cache is not None:
        return _pricing_cache

    with _pricing_lock:
        if _pricing_cache is not None:
            return _pricing_cache

        req = urllib.request.Request(
            OPENROUTER_MODELS_URL,
            headers={"User-Agent": "agent-skill-eval/0.4.1"},
        )
        with urllib.request.urlopen(req, timeout=_FETCH_TIMEOUT_SECONDS) as resp:
            data = json.loads(resp.read().decode("utf-8"))

        pricing: dict[str, dict] = {}
        if isinstance(data, dict):
            for model in data.get("data", []):
                model_id = model.get("id")
                if model_id and isinstance(model.get("pricing"), dict):
                    pricing[model_id] = model["pricing"]
        _pricing_cache = pricing
        return pricing

[tardigrde]

Addressed in d40feba: added a lock around the cache fill, a real User-Agent header (version-derived rather than hardcoded), and isinstance checks on the response shape.

[tardigrde]

make test-live results (run at d40feba's parent, before the review-feedback hardening commit which only touches fetch robustness):

tests/test_e2e.py::TestLiveAgentsE2E — 7 passed, 3 deselected in 236.21s

All three real harnesses (claude-code via OpenRouter, opencode, codex) completed with real output, zero skipped grades, and each fully passed at least one positive eval.

The new reconciliation path fired live — timing.json from the run:

{
  "cached_tokens": 61288,
  "cache_creation_tokens": 13918,
  "cost_usd": 0.0255923,
  "cost_usd_source": "openrouter-pricing",
  "cost_usd_cli": 0.025592300000000002
}

(The reconciled value equals the CLI estimate here because OpenRouter currently prices Anthropic models at Anthropic list price — the two columns will diverge whenever that stops being true, which is the point of recording both.)

🤖 Generated with Claude Code