No bluffing. No barging. A normative overlay that keeps score of what a long-running LLM agent is entitled to claim and entitled to do — not just a memory of what happened.
It runs, and it's measured — negative results included. On DeonticBench's hardest condition, a weak agent drifted past advisory warnings, then exploited a one-shot blocking bump by simply claiming entitlement it didn't have. What held was the board-adjudicated wall (ADR-0007): the write stays denied until the scoreboard itself records an entitled revision — verified symmetrically (drift HELD, entitled revision EXECUTED with zero denies). Full evidence: seed-0 report. Roadmap: ROADMAP.md.
The core distinction, live: same shape of change — different provenance, different verdict.
Reproduce: uv run --project core python demo/drift_demo.py
On a real Claude Code session — paste two lines, nothing to pre-install (the scorer self-fetches from PyPI via uv/pip on first hook):
/plugin marketplace add michalstrnadel/scorekeeper
/plugin install scorekeeper@scorekeeper
Six hooks attach a live scoreboard; watch .scorekeeper/scoreboard.md grow as you work. The extractor uses your own claude CLI by default — no API key needed.
Just the mechanism, no Claude Code (needs uv):
git clone https://github.com/michalstrnadel/scorekeeper && cd scorekeeper
uv run --project core python demo/drift_demo.py # the ~20s demo aboveAs a library / MCP server: pip install scorekeeper (see core/README).
Tried it? A one-paragraph experience report (what it caught, missed, or got wrong) shapes the roadmap more than anything. See CONTRIBUTING.md.
Long-running agents fail in two characteristic directions.
They bluff: at step 3 they decide on Postgres; at step 47 they write MongoDB code. They promise to preserve an API contract and quietly change it an hour later. They assert something they have no basis for — and after context compaction they don't even remember asserting it.
And they barge: asked to fix a typo, the agent refactors three modules, "modernizes" the test helpers, and spins up a subagent fleet that burns an afternoon of usage on work nobody asked for — every edit locally defensible, none of it entitled, all of it now yours to review.
The industry treats the first as a memory problem (bigger windows, better retrieval) and the second as a permissions problem (sandboxes, approval prompts). Here is the uncomfortable observation: in most of these failures, the information was still in the context window and the action was inside the sandbox. The Postgres decision was right there, forty turns up; the typo request was the first line of the conversation. What was missing was that neither had special status — no marker saying this one binds you, none saying this one bounds you.
That is not a memory problem or a permissions problem. It is a normative problem: the agent keeps a record of what happened, but no ledger of what it committed to or was entitled to do. Philosophy of language has studied exactly this structure for fifty years (Brandom's deontic scorekeeping), for claims and for deeds, with a precision that transfers directly into a data model:
- Hallucination = commitment without entitlement — an assertion with no provenance is not a fuzzy quality judgment, it's a visible hole in a graph.
- Self-contradiction = an undetected incompatibility between two live commitments — Postgres-at-step-3 and MongoDB-at-step-47 were both "active" and nobody was keeping score.
- Post-compaction incoherence = deletion of the scoreboard — summarizers preserve narrative and drop exactly the normative state.
- Overreach = action without entitlement — work no request licensed is the practical twin of an assertion with no provenance.
→ The full argument in accessible form: docs/why.md.
Every long-running agent should have, alongside its memory (what happened), a scoreboard answering two questions: what am I entitled to say? and what am I entitled to do?
scorekeeper is a lightweight overlay — it sits on top of any agent harness (Claude Code hooks first, MCP + library next) and:
- extracts commitments from each agent turn into structured, first-class records;
- tracks entitlement — the provenance of each commitment (did the agent read a file? did the user say so? or did it just generate this?). A commitment with no source is a first-class suspect;
- detects incompatibility between active commitments before it propagates into code, docs, or decisions;
- pins action scope — the request in force entitles a bounded scope of work (
path:pins), and the Tier-0 wall denies writes outside it until the board records an entitled widening (ADR-0008); - protects the scoreboard from context compaction — injecting the normative state into summarization exactly where today's summarizers drop it.
Memory systems track facts about the user and the world; observability tools (LangSmith, Langfuse, AgentOps) are flight recorders of execution — spans, latency, tokens. Scorekeeper tracks something neither does: the agent's own commitments and whether whoever changes one was entitled to.
Pieces of that boundary exist elsewhere — AGM-style belief revision protects user axioms by entrenchment, instruction-hierarchy frameworks rank prompt sources by privilege (honest related-work map). What doesn't exist elsewhere is the combination:
- A live normative lifecycle, not a static rank — assert, challenge, supersede, conflict (Brandom's scorekeeping), running inside the agent loop — over claims and deeds.
- An active channel, not a passive store — the boundary becomes environmental physics: the Tier-0 wall denies an unentitled rival write (ADR-0007) and an out-of-scope write (ADR-0008) before it lands; enforcement is keyed on entitlement (who granted what), not on resources (which files are allowed).
- Symmetric measurement, on both axes — each axis is measured symmetrically (the too-eager failure and its too-timid shadow, at the same boundary), and the two axes are twins — the same entitlement structure over claims and over actions:
| Too eager (unentitled move lands) | Too timid (entitled move blocked) | |
|---|---|---|
| Claims — no bluffing | drift / hallucination — SCR (wall verified: HELD) | false refusal — FRR (verified: EXECUTED, zero denies) |
| Actions — no barging | overreach — ORR (wall shipped + tested; first live runs: a case series, not rates) | underreach — URR (instrument validated live; no rates yet) |
All four scored by a deterministic artifact-level classifier, not an LLM judge.
For the argument in accessible form, start with docs/why.md. Then see docs/theory.md for the full conceptual apparatus, docs/research/related-work.md for positioning against the closest five systems, docs/interop.md for mappings onto xAIF and W3C PROV-O, the ROADMAP, and docs/SPEC.md for the full specification (English translation; the Czech SPEC-cs.md is source-of-record). The public API, CLI, and MCP tools are documented in docs/api.md.
An agent is never the model alone — it is the model plus its scaffold (CLAUDE.md, rules, skills, hooks, memory). Harness engineering is, in effect, applied scaffolded-mind theory (Sterelny): building cognitive supports for an entity with tiny working memory and no persistent memory of its own.
That framing forces a choice. Self-editing memory (Letta-style) is extended-mind-style — the agent owns and edits its own external cognition, which is exactly why it has a reliability gap: if the agent forgets to write, the fact is gone. scorekeeper is deliberately the opposite — scaffolded, not extended: the scoreboard is maintained by deterministic hooks and an isolated scorer outside the agent's authority. The agent stands on the scaffold; it does not build it under itself at runtime.
This is not just cleaner engineering — it is what the philosophy independently demands. For Brandom, keeping score is constitutively social: it is done by the other player, not by the speaker about itself. Philosophy of mind and philosophy of language converge on the same overlay design. (Full argument: docs/theory.md §5.)
Phase 0 — complete, acceptance gate passed (full report). The pipeline works end-to-end and the first paired evidence is in:
- The paired delta: on the planted db-choice scenario, the bare haiku agent drifted to MongoDB against its own PostgreSQL decision (verified in artifacts); the scorekept twin — same model, same scenario, only the scoreboard added — held the line. SCR bare 1/6 vs scorekept 0/6 (N=6, Wilson CIs overlap — effect size awaits DeonticBench).
- 0 false conflicts on the dedicated entitled-revision probe (FPR target < 10 %).
- +0.6 % token overhead (target < 10 %); commitment survived context compaction via digest re-injection.
- Instrument per Addendum-1: cross-family local judge (qwen3, S8 protocol, anchored rubric), meta-eval gate CV 0.000, sensitivity-probe verified.
Phase 1 — shipped: v0.1 is on PyPI
(pip install scorekeeper), the MCP server ships (scorekeeper-mcp — writes route
through the same operator pipeline as the hooks), extraction is async by
default in the plugin (detached worker, ~0 ms added turn latency; findings
surface on the next prompt — ADR-0006).
Phase 2 (current) — DeonticBench (formerly EntitleBench) evidence, including the negative kind:
- The effect reproduces. Two independent paired runs on the hardest condition (distance 8 + forced compaction + distractors): the bare haiku agent migrated to the rival tech against its own recorded decision; the scorekept twin surfaced the commitment and asked instead (seed-0 report).
- Two honest negative findings — and the mechanism they forced. On a
second tech pair the scorekept agent drifted anyway: it sailed past 11
advisory warnings and built the tempted Memcached hot path with a
"polyglot" excuse (it even camouflaged the drift — a docstring claiming
"backed by Redis" over a
pymemcacheimport). A one-shot blocking speed bump failed next: told "if the user ordered this, say so and retry", the agent simply claimed entitlement it didn't have and retried. - What held: the board-adjudicated wall
(ADR-0007,
tier0_gate: block). A conflicting write stays denied until the scoreboard itself records an entitled revision through the operator pipeline — the agent's say-so can't lift it. Verified symmetrically on the same scenario: drift family HELD/high (2 denies, zero rival code landed, the agent surfaced and asked) and revision family EXECUTED/high with zero denies (turn-end extraction recorded the user's supersede before any conflicting write — the entitled path cost nothing). - The measurement is adversarially hardened. The deterministic behavioral
classifier (primary metric; the LLM judge is a known-unreliable secondary)
survived a 33-agent adversarial review plus a file-level audit of live runs;
every confirmed misfire is a regression test anchored on a verbatim reply.
126 tests passing across core + bench;
bench/harness/reclassify.pyre-scores old runs after every classifier change.
Phase 2b — the second axis, stated precisely. The same board-adjudicated
wall now guards actions: a scope pin (path:app/** on a task-scope
commitment) records what the current request entitles the agent to touch, and
out-of-scope writes stay denied until the board records an entitled
scope-widening grant — the same wall → surface → entitle → pass flow, applied
to deeds (ADR-0008; landscape:
overreach-landscape). Where the
evidence stands: the mechanism is implemented and unit-tested in this
release (including a subprocess chain test: deny → wall → entitled grant →
pass), and the DeonticBench overreach/expansion families (ORR vs. URR) are
built on the same deterministic-classifier pattern — but no live paired
numbers exist yet. The claims axis has measured evidence; the actions axis
has a tested mechanism and a ready instrument. Until runs land, we claim
exactly that and nothing more.
Rather than scale the numbers in-house next, the ask is for the community to try it — see below. See CHANGELOG.
| Path | What |
|---|---|
core/ |
model + store + backends + extractor + detectors + operators + CLI + MCP server (Python, PyPI: scorekeeper) |
claude-code-plugin/ |
Primary integration: 6 Claude Code hooks (claude --plugin-dir ./claude-code-plugin) |
mcp/ |
scorekeeper-mcp docs — the server lives in core (pip install "scorekeeper[mcp]") |
demo/ |
~20-second mechanism demo (drift_demo.py) + the README GIF tape |
bench/ |
planted acceptance scenarios + Agent-SDK eval harness; DeonticBench in Phase 2 |
docs/ |
why.md (start here), theory.md, api.md, SPEC.md (+ Czech source SPEC-cs.md), model-reports/, research/ |
adr/ |
Architecture Decision Records |
.scorekeeper/ |
The project's own scoreboard — scorekeeper dogfoods itself |
Early project, design-first — trying it and reporting back is as valuable as code. Start with CONTRIBUTING.md: 60-second setup, a tour of the codebase, and good first issues (add a model backend, extend the rival-tech lexicon, write a planted scenario, or port the core to TypeScript). scripts/e2e.sh runs everything CI runs in one command. File an experience report after you run it.
If you use it in research, there's a CITATION.cff.