Skip to content

fix(opencode): capture every assistant turn and stop losing concurrent extraction records#252

Merged
edheltzel merged 17 commits into
mainfrom
fm/recall-opencode-phase4-o1
Jul 25, 2026
Merged

fix(opencode): capture every assistant turn and stop losing concurrent extraction records#252
edheltzel merged 17 commits into
mainfrom
fm/recall-opencode-phase4-o1

Conversation

@edheltzel

Copy link
Copy Markdown
Owner

Intent

Deliver OpenCode integration Phase 4 (Testing and Polish) for Recall: genuinely exercise its four acceptance items against a real OpenCode runtime and record the results, including the session.idle frequency as a measured number.

Key context a reviewer reading only the diff would not know:

  1. Phase 4 was ALREADY marked complete by merged PR test(opencode): complete Phase 4 testing and polish #248, but that was not a real proof. Its e2e (scripts/e2e-opencode.ts) imports the adapter directly and SYNTHESIZES the session.idle payload and its own $ shell. So it proved Recall ACCEPTS that event shape - never that OpenCode emits it, and never that OpenCode loads the plugin at all. Closing that blind spot is the point of this branch, which is why a second e2e script exists rather than extending the first. The two are deliberately kept separate: pipeline vs live-server contract.

  2. The measurement: session.idle fires 1.00 times per assistant turn (3 turns -> 3 events), measured from OpenCode's own /event SSE stream inside ONE long-lived server process so it cannot be a process-teardown artifact, against OpenCode 1.18.5. The design doc previously hedged toward adding DEBOUNCING; the measured evidence points the opposite way, so the doc hedge was replaced.

  3. That measurement exposed three real, pre-existing defects, each reproduced first and each with a regression test I verified FAILS on the pre-fix code:

  • opencode/RecallExtract.ts kept a permanent dedup tracker, so with per-turn idle only turn 1 was ever captured; a real 3-turn session left a 154-byte drop missing 2 of 3 markers. Tracker now keys on a content digest instead of 'session seen'.
  • OpenCode invokes EVERY export of a top-level plugins/*.ts as a plugin factory, so the plugin's exported test helpers were called with the plugin context and OpenCode logged 'failed to load plugin ... Object is not a function' on every launch. Helpers moved to opencode/lib/session-export.ts; I verified empirically that OpenCode does NOT glob subdirectories before choosing that layout, and the installer/uninstaller carry the nested file.
  • hooks/lib/sqlite-writers.ts lost records when two hosts extracted concurrently. The skipDuplicates probe makes those transactions read-then-write, and SQLite fails that upgrade with SQLITE_BUSY INSTANTLY, never consulting busy_timeout - measured 1ms hard failure vs the full timeout honoured once the write lock is taken up front. Hence insertMany.immediate(...) at exactly four call sites. I checked the other hook transactions (consolidate-core, RecallPreCompact): they begin with a write, so busy_timeout already applies and they were deliberately left alone. I also probed and DISPROVED an earlier theory that pragma ordering caused it.
  1. Deliberate scope decisions: uninstall does NOT auto-restore collision backups and I did not add that - the documented rollback path is ./install.sh restore, so I tested that instead of inventing product behaviour. Restore gates on a default-No confirm that returns the default when stdin is not a TTY, making it a no-op in CI; rather than weakening that safety default I asserted it directly and exercised the restoring path by overriding _confirm in the sourced shell. Compaction context injection (RecallPreCompact) is deliberately left runtime-unverified and README/docs say so rather than implying otherwise, which is why the README row moves to Beta and not Stable.

  2. 10 lifecycle tests fail in any git worktree — the environment AGENTS.md mandates for workers has a permanently red baseline #243 is closed by this branch: all three of its acceptance items are met (fresh-worktree suite green, both config helpers exit non-zero leaving a malformed config byte-identical, and CI now runs the suite inside a worktree). chore: fresh /ce-worktree worktrees miss node_modules → ~10 MCP-config tests fail spuriously #165 is deliberately NOT closed - the worktree suite is green only after bun install inside it, which is chore: fresh /ce-worktree worktrees miss node_modules → ~10 MCP-config tests fail spuriously #165's gap, so the CI job asserts the empty-node_modules precondition rather than hiding it.

  3. The OpenCode version pin moved 1.18.4 -> 1.18.5 and now lives once in scripts/lib/opencode-runtime.ts (DRY), which is why the older e2e no longer defines its own. Two pre-existing tests that re-implemented tracker logic inline - and therefore passed with the data-loss bug present - were replaced with tests that drive the real plugin factory.

Verification state: bun test 1298 pass / 0 fail, bun run lint clean, bash -n clean on all lifecycle scripts, both OpenCode e2e scripts pass against a live OpenCode 1.18.5, and the runtime gate was confirmed non-vacuous on the bunx path CI uses by deliberately re-introducing the defect.

What Changed

  • OpenCode plugin (opencode/RecallExtract.ts)session.idle fires once per assistant turn, not once per session, so the permanent "session seen" dedup tracker froze each drop on turn 1 and discarded the rest of the conversation. The tracker now keys on a SHA-256 digest of the rendered markdown (legacy array-format trackers load with an empty digest and converge on the next idle), and both the drop and the tracker are written through a temp file plus renameSync so the cron-driven RecallBatchExtract in a separate process can't read a torn file. OpenCode also calls every export of a top-level plugins/*.ts as a plugin factory, which made it log failed to load plugin ... Object is not a function on every launch — the shared helpers moved to opencode/lib/session-export.ts, with lib/install-lib.sh and uninstall.sh now installing and removing that nested file from one shared RECALL_OPENCODE_PLUGIN_HELPERS list. The shipped agent definition's /dump rule now requires explicit messages + source: "opencode" (title-only resolved to Claude Code's transcript on a dual-host machine), and its tool table is replaced by the prefix rule plus a pointer to docs/mcp-tools.md.
  • Shared SQLite write path (hooks/lib/sqlite-writers.ts, all hosts) — the four batch writers now commit through insertMany.immediate(...) instead of the default DEFERRED. The skipDuplicates probe makes those transactions read-then-write, and SQLite fails that lock upgrade with SQLITE_BUSY immediately without ever consulting busy_timeout, so a peer host extracting into the same WAL database lost records; taking the write lock up front is what makes the timeout apply. consolidate-core and RecallPreCompact open with a write and were left DEFERRED.
  • Runtime verification and coverage — added scripts/e2e-opencode-runtime.ts plus a shared scripts/lib/opencode-runtime.ts (the OpenCode version pin now lives there only) that drives a real opencode serve, subscribes to its /event SSE stream, and asserts the plugin loads with zero errors and the measured once-per-turn session.idle frequency; the existing scripts/e2e-opencode.ts stays as the pipeline-level harness. CI gains a job running that runtime gate and a worktree job that runs lint + the suite inside a fresh git worktree (asserting the empty-node_modules precondition rather than hiding it). New tests/hooks/sqlite-writers-concurrency.test.ts and tests/install/restore.test.ts; two OpenCode tests that re-implemented tracker logic inline now drive the real plugin factory. README and docs/OPENCODE_INTEGRATION.md move OpenCode from Alpha to Beta, record the measured event frequency in place of the old debouncing hedge, and state that compaction injection remains runtime-unverified.

Risk Assessment

✅ Low: All nine prior findings are resolved with surgical, verifiable changes and no warnings remain; the shared SQLite write-path surface is unchanged but is now guarded by a cross-process regression test that provably fails on the pre-fix DEFERRED code, leaving only one info-level hardening nit.

Testing

Ran both OpenCode e2e scripts against a live OpenCode 1.18.5 server (runtime contract and pipeline, both EXIT=0), capturing the branch's headline measurement - session.idle fires exactly 1.00 times per assistant turn across three turns of one long-lived process - plus zero plugin load errors and a 560-byte drop containing all three turn markers, which is the direct product-level contrast to the 154-byte 1-of-3 drop the fixes address. Ran the full suite twice: 1298 pass / 0 fail under --timeout 30000 (matching the handoff's claim exactly) and again under CI's bare bun test, both inside a confirmed detached-HEAD git worktree, which satisfies #243's fresh-worktree acceptance item. Verified the newly added pi stub does not buy speed with coverage: it is behaviorally transparent because remove_pi discards the call's exit code and output, and a mutation that disables Pi memory-section removal still turns the stubbed test red, with uninstall.sh restored and bash -n clean afterward. The stub's AFTER timings (0.781s / 0.681s under load) match the handoff, though its ~10s BEFORE figure did not reproduce on this host - real pi remove costs 0.50s solo and the pre-stub file passes under the default 5s timeout - so only that magnitude claim in the handoff prose is unconfirmed, not the fix. Captured reviewer-visible screenshots of the rendered README showing the intro line and the compatibility table now agreeing on Beta with matching caveats. Left #251's cold-network install flake untouched as directed; the worktree is clean and dist/ was removed.

Evidence: Live OpenCode 1.18.5 runtime contract e2e - measured session.idle frequency

opencode.version=1.18.5 opencode.installed_plugin=.../xdg-config/opencode/plugins/RecallExtract.ts opencode.installed_helper=.../xdg-config/opencode/plugins/lib/session-export.ts opencode.plugin_loaded_by_host=true opencode.plugin_load_errors=0 opencode.session_idle_total=3 opencode.session_idle_turns=3 opencode.session_idle_per_turn=1.00 opencode.session_idle_deltas=[{"turn":1,"idleDelta":1},{"turn":2,"idleDelta":1},{"turn":3,"idleDelta":1}] opencode.drop_contains_all_turns=true (3 turns, 560 bytes) opencode.tracker_records_content_digest=true opencode.export_contains_all_turns=true isolation.production_db_opened=false opencode.runtime_contract=verified against 1.18.5 EXIT=0

$ bun run build && bun run scripts/e2e-opencode-runtime.ts
$ tsup src/index.ts src/mcp-server.ts --format esm --dts --clean --external bun:sqlite && node -e "const fs=require('fs');['dist/index.js','dist/mcp-server.js'].forEach(f=>{const c=fs.readFileSync(f,'utf8');if(!c.startsWith('#!/usr/bin/env node'))throw new Error(f+' missing expected shebang');fs.writeFileSync(f,c.replace('#!/usr/bin/env node','#!/usr/bin/env bun'))})"
CLI Building entry: src/index.ts, src/mcp-server.ts
CLI Using tsconfig: tsconfig.json
CLI tsup v8.5.1
CLI Target: es2022
CLI Cleaning output folder
ESM Build start
ESM dist/mcp-server.js     30.68 KB
ESM dist/chunk-YHD4BVUY.js 6.50 KB
ESM dist/dump-IMJCGEP7.js  225.00 B
ESM dist/chunk-76L3MJB6.js 14.08 KB
ESM dist/chunk-6QFGMCES.js 94.06 KB
ESM dist/index.js          236.11 KB
ESM ⚡️ Build success in 984ms
DTS Build start
DTS ⚡️ Build success in 3620ms
DTS dist/index.d.ts      20.00 B
DTS dist/mcp-server.d.ts 1.42 KB
isolation.recall_home=/var/folders/88/3h9cyc4979d2l6p7xkn79yqr0000gn/T/recall-opencode-runtime-x739jt/recall-home
isolation.opencode_config=/var/folders/88/3h9cyc4979d2l6p7xkn79yqr0000gn/T/recall-opencode-runtime-x739jt/xdg-config/opencode
isolation.test_db=/var/folders/88/3h9cyc4979d2l6p7xkn79yqr0000gn/T/recall-opencode-runtime-x739jt/recall.db
isolation.production_db=/Users/ed/.agents/Recall/recall.db
isolation.production_before={"exists":true,"size":180125696,"mtimeMs":1785000903977.856,"ino":571168746}
isolation.production_db_opened=false
opencode.version=1.18.5
opencode.installed_plugin=/var/folders/88/3h9cyc4979d2l6p7xkn79yqr0000gn/T/recall-opencode-runtime-x739jt/xdg-config/opencode/plugins/RecallExtract.ts
opencode.installed_helper=/var/folders/88/3h9cyc4979d2l6p7xkn79yqr0000gn/T/recall-opencode-runtime-x739jt/xdg-config/opencode/plugins/lib/session-export.ts
stub.port=61992
opencode.serve_port=61993
opencode.session=ses_065a669ddffeqg4jV1sOuHheNo
opencode.plugin_loaded_by_host=true
opencode.plugin_load_errors=0
opencode.session_idle_total=3
opencode.session_idle_turns=3
opencode.session_idle_per_turn=1.00
opencode.session_idle_deltas=[{"turn":1,"idleDelta":1},{"turn":2,"idleDelta":1},{"turn":3,"idleDelta":1}]
opencode.drop_contains_all_turns=true (3 turns, 560 bytes)
opencode.tracker_records_content_digest=true
opencode.export_contains_all_turns=true
isolation.production_db_opened=false
opencode.runtime_contract=verified against 1.18.5
EXIT=0
- Evidence: README intro line 21 rendered - now reads Beta for OpenCode (local file: /var/folders/88/3h9cyc4979d2l6p7xkn79yqr0000gn/T/no-mistakes-evidence/01KYCZPQNZX62ATN5XVEWZYTQA/readme-intro-line21.png) - Evidence: README compatibility table rendered - OpenCode row agrees with the intro (local file: /var/folders/88/3h9cyc4979d2l6p7xkn79yqr0000gn/T/no-mistakes-evidence/01KYCZPQNZX62ATN5XVEWZYTQA/readme-compat-table.png)
Evidence: Non-vacuity proof: pi-stubbed test still catches a broken Pi removal path

# MUTANT: uninstall.sh remove_memory_section "$agents_md" pi -> disabled 579: : # MUTANT error: expect(received).not.toContain(expected) Expected to not contain: "## MEMORY" Received: "# Existing Pi instructions\n\n## MEMORY\n\nYou have persistent memory via Recall..." at tests/install/uninstall.test.ts:456:25 (fail) uninstall.sh > Pi AGENTS.md: complete legacy Recall section is removed [1399.13ms] 1 pass 1 fail Ran 2 tests across 1 file. [2.19s]

# Non-vacuity proof: the pi-stubbed Pi-path tests still catch a broken removal
# MUTANT: uninstall.sh remove_memory_section "$agents_md" pi  ->  disabled

579:  : # MUTANT

                              ^
error: expect(received).not.toContain(expected)

Expected to not contain: "## MEMORY"
Received: "# Existing Pi instructions\n\n## MEMORY\n\nYou have persistent memory via Recall. **Read the full guide:** ~/.pi/agent/Recall_GUIDE.md\n\nCore rules:\n1. Before asking user to repeat anything → search first with `recall-memory_memory_search`\n2. When decisions are made → record with `recall-memory_memory_add`\n3. End of session when user says `/dump` → call `recall-memory_memory_dump({ title: \"Descriptive Title\" })`\n4. Dumped sessions are immediately searchable from new sessions via `recall-memory_memory_search`\n\nTool syntax:\n- `recall-memory_memory_search({ query: \"search terms\" })`\n- `recall-memory_memory_add({ type: \"decision\", content: \"what\", detail: \"why\" })`\n- `recall-memory_memory_dump({ title: \"Session title\", skip_fabric: true })`\n- `recall-memory_context_for_agent({ task_description: \"what the agent will do\" })`\n\n## After-memory section\n\nPreserve this.\n"

      at <anonymous> (/Users/ed/.no-mistakes/worktrees/eead88ef695d/01KYCZPQNZX62ATN5XVEWZYTQA/tests/install/uninstall.test.ts:456:25)
(fail) uninstall.sh > Pi AGENTS.md: complete legacy Recall section is removed [1399.13ms]

 1 pass
 11 filtered out
 1 fail
 6 expect() calls
Ran 2 tests across 1 file. [2.19s]
Evidence: Pipeline e2e at HEAD against live OpenCode 1.18.5

opencode.uninstall_preserves_user_plugin_lib=verified opencode.version=1.18.5 opencode.search_markers=OPENCODE_PHASE4_SEARCHABLE_MARKER,OPENCODE_PHASE4_SEARCHABLE_MARKER_RETRY,OPENCODEPHASE4CONCURRENTA,OPENCODEPHASE4CONCURRENTB opencode.export_failure_retry=verified opencode.concurrent_wal_writers=verified opencode.installer_jsonc_rollback=verified isolation.production_db_opened=false EXIT=0

$ bun run build && bun run scripts/e2e-opencode.ts
$ tsup src/index.ts src/mcp-server.ts --format esm --dts --clean --external bun:sqlite && node -e "const fs=require('fs');['dist/index.js','dist/mcp-server.js'].forEach(f=>{const c=fs.readFileSync(f,'utf8');if(!c.startsWith('#!/usr/bin/env node'))throw new Error(f+' missing expected shebang');fs.writeFileSync(f,c.replace('#!/usr/bin/env node','#!/usr/bin/env bun'))})"
CLI Building entry: src/index.ts, src/mcp-server.ts
CLI Using tsconfig: tsconfig.json
CLI tsup v8.5.1
CLI Target: es2022
CLI Cleaning output folder
ESM Build start
ESM dist/dump-IMJCGEP7.js  225.00 B
ESM dist/mcp-server.js     30.68 KB
ESM dist/chunk-YHD4BVUY.js 6.50 KB
ESM dist/chunk-76L3MJB6.js 14.08 KB
ESM dist/chunk-6QFGMCES.js 94.06 KB
ESM dist/index.js          236.11 KB
ESM ⚡️ Build success in 84ms
DTS Build start
DTS ⚡️ Build success in 1790ms
DTS dist/index.d.ts      20.00 B
DTS dist/mcp-server.d.ts 1.42 KB
isolation.recall_home=/var/folders/88/3h9cyc4979d2l6p7xkn79yqr0000gn/T/recall-opencode-e2e-MeSDuM/recall-home
isolation.opencode_config=/var/folders/88/3h9cyc4979d2l6p7xkn79yqr0000gn/T/recall-opencode-e2e-MeSDuM/xdg-config/opencode
isolation.test_db=/var/folders/88/3h9cyc4979d2l6p7xkn79yqr0000gn/T/recall-opencode-e2e-MeSDuM/recall.db
isolation.production_db=/Users/ed/.agents/Recall/recall.db
isolation.production_before={"exists":true,"size":180125696,"mtimeMs":1785000903977.856,"ino":571168746}
isolation.production_db_opened=false
[recall] OpenCode session export failed for ses_recall_phase4_retry; will retry on a later idle event: synthetic export failure
opencode.uninstall_preserves_user_plugin_lib=verified
opencode.version=1.18.5
opencode.search_markers=OPENCODE_PHASE4_SEARCHABLE_MARKER,OPENCODE_PHASE4_SEARCHABLE_MARKER_RETRY,OPENCODEPHASE4CONCURRENTA,OPENCODEPHASE4CONCURRENTB
opencode.export_failure_retry=verified
opencode.concurrent_wal_writers=verified
opencode.installer_jsonc_rollback=verified
isolation.production_db_opened=false
EXIT=0
Evidence: pi stub timing A/B on a host that has Pi installed

host pi: /Users/ed/.bun/bin/pi BEFORE (pre-stub file @6f59fdd), --timeout 60000: 13 pass / 0 fail [8.44s] BEFORE (pre-stub file @6f59fdd), bun DEFAULT 5s: 13 pass / 0 fail [8.45s] <- no individual test exceeded 5s here AFTER (HEAD, pi stubbed), bun DEFAULT 5s: 13 pass / 0 fail [7.30s] direct cost of the call remove_pi makes (isolated PI_CODING_AGENT_DIR): real 0.50 / real 0.50 / real 0.49 junit per-test, under full-suite load (matches handoff's "stable 0.7-1.0s each"): 0.781s Pi AGENTS.md: complete legacy Recall section is removed 0.681s Pi AGENTS.md: customized legacy-looking section is preserved

host pi: /Users/ed/.bun/bin/pi   (this machine HAS Pi installed - the environment that exhibits the bug)

===== BEFORE: pre-stub test file (commit 6f59fdd), run with --timeout 60000 =====
Ran 13 tests across 1 file. [8.44s]

===== AFTER: HEAD (pi stubbed on PATH), run with bun's DEFAULT 5s per-test timeout =====
Ran 13 tests across 1 file. [7.30s]
Evidence: Full suite under CI's exact command (bare bun test, 5s default)

$ bun test # .github/workflows/ci.yml:38 1298 pass 0 fail 47918 expect() calls Ran 1298 tests across 107 files. [44.10s] EXIT=0 (also: bun test --timeout 30000 -> 1298 pass / 0 fail [47.02s], run inside git worktree .../01KYCZPQNZX62ATN5XVEWZYTQA d5a53d5 (detached HEAD))

bun test v1.3.14 (0d9b296a)

tests/opencode-integration.test.ts:
[recall] OpenCode tracker unreadable; starting fresh: JSON Parse error: Unexpected identifier "not"
[recall] OpenCode session export failed for ses-retry; will retry on a later idle event: no export configured for ses-retry

tests/mcp-server-hybrid-dedup.test.ts:
[recall] vec KNN failed — falling back to brute-force scan: Dimension mismatch for query vector for the "embedding" column. Expected 1024 dimensions but received 4.

tests/hooks/extract-model-provider.test.ts:
[FabricExtract] Trying first-host extraction...
[FabricExtract] Trying second-host extraction...
RECALL_CONSOLIDATE_PROGRESS {"written":2,"demoted":5}

tests/commands/import-conversations.test.ts:
Conversation Import
===================


tests/commands/embed-rebackfill.test.ts:
Re-backfilling 2 embeddings → qwen3-embedding:0.6b (1024d)...

  [1/2] Re-embedding decisions#1... ✓ (1024d)
  [2/2] Re-embedding decisions#2... ✓ (1024d)

Done: 2 re-embedded, 0 pruned (orphan/too-short).
Marker set to qwen3-embedding:0.6b (1024d).
Vector index rebuilt: 2 entries.
Re-backfilling 2 embeddings → qwen3-embedding:0.6b (1024d)...

  [1/2] Re-embedding decisions#1... ✓ (1024d)
  [2/2] Re-embedding decisions#2... No embeddings to re-backfill. Marker set to qwen3-embedding:0.6b (1024d).

tests/commands/lifecycle.test.ts:
recall install: lifecycle script not found at /var/folders/88/3h9cyc4979d2l6p7xkn79yqr0000gn/T/recall-install-hJ7cFN/install.sh.
This command delegates to install.sh, which ships in the Recall package (and the git checkout).
Run it from a Recall install, or set RECALL_REPO_DIR to the directory that holds install.sh.
recall uninstall: lifecycle script not found at /var/folders/88/3h9cyc4979d2l6p7xkn79yqr0000gn/T/recall-lc-g8Hdmx/uninstall.sh.
This command delegates to uninstall.sh, which ships in the Recall package (and the git checkout).
Run it from a Recall install, or set RECALL_REPO_DIR to the directory that holds uninstall.sh.

 1298 pass
 0 fail
 47918 expect() calls
Ran 1298 tests across 107 files. [44.10s]
EXIT=0
Evidence: Round 2 validation summary (all checks, with reasoning)
# Round 2 validation — OpenCode Phase 4 (commit d5a53d5)

Round 1 validated the branch body. Round 2 re-validates at HEAD after the two
requested fixes, and adds the check nobody had done: whether the new `pi` stub
made the uninstall tests vacuous.

## 1. User intent: measured session.idle against a live OpenCode 1.18.5

`bun run test:e2e:opencode:runtime` -> `e2e-opencode-runtime.log`

    opencode.version=1.18.5
    opencode.plugin_loaded_by_host=true
    opencode.plugin_load_errors=0
    opencode.session_idle_total=3
    opencode.session_idle_turns=3
    opencode.session_idle_per_turn=1.00
    opencode.session_idle_deltas=[{turn:1,idleDelta:1},{turn:2,idleDelta:1},{turn:3,idleDelta:1}]
    opencode.drop_contains_all_turns=true (3 turns, 560 bytes)
    opencode.tracker_records_content_digest=true
    opencode.runtime_contract=verified against 1.18.5

560 bytes carrying all 3 markers is the fix for the pre-fix 154-byte drop that
held 1 of 3. Zero plugin load errors is the fix for the per-launch
"Object is not a function". Both measured in ONE long-lived server process.

`bun run test:e2e:opencode` (pipeline) -> `e2e-opencode-pipeline.log`, EXIT=0.

## 2. The `pi` stub is NOT vacuous

`remove_pi` (uninstall.sh:518-525) calls `pi remove ... >/dev/null 2>&1 || true`
— exit code discarded, output discarded, no test asserts on it. Stub is
behaviorally transparent.

Mutant proof — disabled `remove_memory_section "$agents_md" pi`:

    (fail) uninstall.sh > Pi AGENTS.md: complete legacy Recall section is removed [774.07ms]
    expect(received).not.toContain("## MEMORY")

The stubbed test still catches a broken Pi removal path. uninstall.sh restored,
`bash -n` clean.

## 3. Timing claim: AFTER matches, BEFORE magnitude does not

junit, under real full-suite load:

    Pi AGENTS.md: complete legacy Recall section is removed   0.781s
    Pi AGENTS.md: customized legacy-looking section is preserved  0.681s

Matches the handoff's "stable 0.7-1.0s each". But the handoff's ~10s BEFORE
figure did not reproduce here: real `pi remove` against the test's isolated
PI_CODING_AGENT_DIR costs 0.50s (3 runs), and the pre-stub file passes under
bun's DEFAULT 5s timeout today (8.45s total / 13 tests, none individually >5s).
See `pi-stub-timing-before-after.txt`. The fix is harmless and correct; only the
magnitude in the handoff prose is unconfirmed on this host.

## 4. Full suite, inside a git worktree

    git worktree list -> .../01KYCZPQNZX62ATN5XVEWZYTQA d5a53d5 (detached HEAD)
    bun test --timeout 30000 -> 1298 pass / 0 fail / 40335 expect() / 107 files [47.02s]

Matches the handoff's 1298/0 exactly, and satisfies #243 acceptance item 1
(fresh-worktree suite green). Slowest single test in the run: 4.167s.

## 5. README self-contradiction fixed (screenshots)

`readme-intro-line21.png`  — "Beta on Pi and Beta for OpenCode (capture is
verified against a live server; compaction injection is not)"
`readme-compat-table.png` — "OpenCode | Beta — recall-extract plugin;
session.idle capture verified against a live OpenCode 1.18.5 server |
Capture runtime-verified; compaction injection not yet"

Both regions of the rendered README now agree. No "Alpha for" remains anywhere.

## 6. Also green under CI's exact command

`.github/workflows/ci.yml` runs bare `bun test` (no `--timeout`), so the suite
was re-run that way too:

    bun test -> 1298 pass / 0 fail / 47918 expect() / 107 files [44.10s], EXIT=0

No test failed under bun's 5s default in this run. All 13 `uninstall.test.ts`
tests land between 0.431s and 0.862s (junit, under full-suite load).

Artifacts: `nonvacuity-pi-stub-mutant.log`, `full-suite-ci-command.log`,
`full-suite.log`, `junit-full-suite.xml`, `e2e-opencode-runtime.log`,
`e2e-opencode-pipeline.log`, `pi-stub-timing-before-after.txt`,
`readme-intro-line21.png`, `readme-compat-table.png`, `readme-rendered.html`.
Evidence: Rendered README used for the screenshots
<!doctype html><html><head><meta charset="utf-8">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/github-markdown-css@5/github-markdown-light.min.css">
<script src="https://cdn.jsdelivr.net/npm/marked@12/marked.min.js"></script>
<style>body{margin:0;background:#fff}.markdown-body{box-sizing:border-box;max-width:900px;margin:0 auto;padding:32px}
mark.hl{background:#fff3c4;outline:2px solid #d4a017;border-radius:3px}</style>
</head><body><article class="markdown-body" id="c"></article>
<script id="src" type="text/plain"><p align="center">
  <img src="assets/banner.png" alt="Recall — Persistent Memory for Coding Agents" width="100%">
</p>

## TL;DR

Recall is a retrieval-first memory layer: everything lands in one searchable database, the best of it is ranked and injected at session start, and decisions carry confidence, importance, and a lifecycle across any coding agent/harness.


> **A SQLite-backed persistent memory layer for coding agents.** Stop-hook extraction captures sessions where a host lifecycle adapter exists, MCP tools expose them mid-session, hybrid search (FTS5 + embeddings) retrieves them, and a tiered L0/L1 recall block injects identity + top-ranked records on supported hosts. Works across Claude Code, OpenCode, Pi, and Codex from one local database.

Got questions about the project? I'd suggest using [DeepWiki](https://deepwiki.com/edheltzel/Recall) from Devin/Cognition to ask questions about the project.


# Recall — Persistent Memory for Any Agent Harness

All coding agents forget when a session ends. Recall doesn't — it extracts, indexes, and recalls what matters across every session, across every agent you use.

Built on the [Model Context Protocol](https://modelcontextprotocol.io). One SQLite file. No phone-home. No vendor lock-in.

> Stable on [Claude Code](https://claude.com/claude-code). Beta on [Pi](https://pi.dev/) and Beta for [OpenCode](https://opencode.ai/) (capture is verified against a live server; compaction injection is not). [Codex CLI](https://github.com/openai/codex) uses a native plugin for MCP and skills; lifecycle auto-capture is not yet supported. [Gemini CLI](https://github.com/google-gemini/gemini-cli) remains on the roadmap. See [Roadmap](#roadmap).

---

[Jump to the Docs](#documentation)

## The Problem

AI agents have no memory between sessions. Context is lost. You repeat yourself. Decisions made last week are forgotten today. Every new session re-learns the basics.

## How Recall Fixes It

Install once, then forget about it. Recall runs silently in the background:

`` `
┌──────────┐    ┌────────────────┐    ┌──────────────┐    ┌───────────────┐    ┌──────────────┐
│ You Work │───▶│ Stop hook fires│───▶│ Auto-Extract │───▶│ SQLite + FTS5 │───▶│ Next Session │
└─────▲────┘    │ (end of turn)  │    └──────────────┘    └───────────────┘    └──────┬───────┘
      │         └────────────────┘                                                    │
      └───────────────────────────── Memory Available ───────────────────────────────┘
`` `

- **Auto-extraction** — sessions are parsed into structured summaries incrementally as you work (Stop hook fires at the end of every turn, not only when you exit)
- **Full-text + semantic search** — find anything from any past session
- **Tiered session-start context** — L0 identity (who you are) + L1 importance-ranked top records load automatically
- **Zero friction** — no workflow changes, no manual steps
- **MCP integration** — your agent searches memory automatically through standard MCP tools

## Why Recall

Four things that set Recall apart from cloud-hosted memory layers and from agent-specific scratch files:

- **Local-first, zero infrastructure.** One SQLite file at `~/.agents/Recall/recall.db` (override via `RECALL_DB_PATH`). WAL mode, `0600` perms. No vector database, no graph database, no agent server, no API keys for retrieval. Nothing leaves your machine — no telemetry, no phone-home. Optional Ollama for embeddings (also local).
- **Multi-agent native.** One memory layer across the agents you actually use. Stable on Claude Code today; Pi, OpenCode, and Codex connect through MCP. Memories captured by one agent are searchable from any other agent on the same machine.
- **Structured taxonomy, not a flat blob.** Decisions (with supersede/revert lifecycle and confidence scoring), learnings, breadcrumbs, and curated **Library of Alexandria** entries — each has a purpose and a query path. Importance scoring (1–10) surfaces what matters first.
- **Hybrid search that works offline.** FTS5 keyword search ships with SQLite — no embedding infrastructure required to find anything. Optional Ollama embeddings layer on top for semantic queries. Both are merged via Reciprocal Rank Fusion. Lose Ollama, lose nothing — the keyword path keeps working.

## Quick Start

Recall requires [Bun](https://bun.sh) (it uses `bun:sqlite` and Bun-native hooks).

`` `bash
# Primary — install from npm with Bun, then configure
bun install -g recall-memory
recall install

# Secondary — one-shot via npx (Bun must be on PATH)
npx --package=recall-memory recall install
`` `

`recall install` runs the canonical setup (MCP server, hooks, agent skills, guides) for all detected agents. Prefer `bun install -g`: with `npm install -g`, the `#!/usr/bin/env bun` shebang depends on Bun being on PATH (nvm/fnm shells can hide it).

<details>
<summary>Install from source instead</summary>

`` `bash
git clone https://github.com/edheltzel/Recall.git
cd Recall
./install.sh
`` `

</details>

Verify it works:

`` `bash
recall stats        # Database overview
recall doctor       # Health check
`` `

Restart your agent (Claude Code, Pi, or OpenCode) to load the MCP server and hooks.

Codex uses its native plugin marketplace instead of the lifecycle installer; see [Codex Integration](docs/CODEX_INTEGRATION.md).

Claude Code can additionally install Recall as a native plugin, which takes over the nine `recall-*` skills and the `recall-memory` MCP server while the installer keeps owning the lifecycle hooks. Existing installs need one reconciliation step — see [Claude Integration](docs/CLAUDE_INTEGRATION.md).

### First run: set your identity

Recall's tiered RecallStart injects a small identity file at the top
of every session (the L0 tier — your role, projects, tools, and working
preferences). Without it, L0 is empty and every new session has to
re-learn the basics.

`` `bash
recall onboard
`` `

A 7-question interview that writes `~/.claude/MEMORY/identity.md`. Run
it once. Re-run whenever your role, active projects, or working
preferences change. Use `|` (not `,`) to separate values so a phrase
like `no force-push, ever` survives as a single entry.

### Updating

From inside Claude Code, `/recall-update` prints the current vs. latest
release and the exact command to run. From a shell:

`` `bash
./update.sh --check   # version check only
./update.sh           # full update: pull, build, migrate, re-register hooks
`` `

Installed from npm? Use `recall update` (same flags) — or `bun install -g recall-memory@latest && recall install` to bump the binary.

### Uninstalling

`` `bash
./uninstall.sh --dry-run   # preview, touch nothing
./uninstall.sh             # surgical remove; preserves ~/.agents/Recall/ (DB + backups)
./uninstall.sh --purge     # also destroy ~/.agents/Recall/ and any legacy DB (confirmed)
`` `

Installed from npm? Use `recall uninstall` (same flags, e.g. `--dry-run` / `--purge`).

> [Full installation guide](docs/installation.md) — prerequisites, platform support, session extraction setup, uninstalling

## How Recall Works

Recall sits between your agent and a single SQL

... [14723 bytes truncated] ...

 `/recall-scout [focus]` produces a memory-first scout report (repo map, key paths, tests, risks, next steps) for orienting in an unfamiliar repo, with a strict no-secrets boundary and chat-only-by-default output
- **Benchmark harness** — `recall benchmark run B` measures wake-up context efficiency against locked baselines so regressions are visible
- **Onboarding** — `recall onboard` runs a 7-question interview that writes your L0 identity file

## Measured wake-up efficiency

Suite B measures the byte cost of session-start memory injection. Latest tracked run ([2026-04-18, scope `atlas-recall`](benchmarks/results/2026-04-18T20-13-59-suite-B.md)):

| Variant                                      |     Chars | Tokens (est, 4 ch/tok) |
| -------------------------------------------- | --------: | ---------------------: |
| **v2 tiered RecallStart** (L0 + L1 top 12) | **5,306** |             **~1,327** |
| v1 flat-blob RecallStart (simulated)       |     8,020 |                 ~2,005 |
| CLAUDE.md static baseline                    |     8,760 |                 ~2,190 |

v2 is **51% smaller than v1** on this corpus. CLAUDE.md is hand-written static context; Recall is auto-extracted dynamic memory — the two are complementary, not competitors. Numbers scale with your own DB and L0 identity; reproduce with `recall benchmark run B`. Methodology and caveats live in [`benchmarks/README.md`](benchmarks/README.md).

## CLI at a Glance

`` `bash
recall "kubernetes auth"          # Search your memory
recall onboard                    # Seed your L0 identity tier (one-time)
recall dump "Session Title"       # Save this session
recall add decision "Use X" ...   # Record a decision
recall decision list              # List decisions with status and confidence
recall pin decisions 42           # Pin a record to high importance
recall benchmark run B            # Measure wake-up context efficiency
recall prune                      # Preview stale records for removal
recall stats                      # See what's stored
recall doctor                     # Health check
`` `

<details>
<summary>See it in action</summary>

| Search                                | Stats                               |
| ------------------------------------- | ----------------------------------- |
| ![recall search](assets/demo-search.gif) | ![recall stats](assets/demo-stats.gif) |

| Health Check                          | Recent Memory                         |
| ------------------------------------- | ------------------------------------- |
| ![recall doctor](assets/demo-doctor.gif) | ![recall recent](assets/demo-recent.gif) |

</details>

> [Full CLI reference](docs/cli-reference.md)

## For AI Agents

If you're an AI agent reading this repository:

| What you need                                                  | Where to find it                     |
| -------------------------------------------------------------- | ------------------------------------ |
| **Using Recall from Claude Code** (MCP tools, CLI, core rules) | [`FOR_CLAUDE.md`](FOR_CLAUDE.md)     |
| **Installing the Claude Code plugin**                          | [`docs/CLAUDE_INTEGRATION.md`](docs/CLAUDE_INTEGRATION.md) |
| **Using Recall from OpenCode**                                 | [`FOR_OPENCODE.md`](FOR_OPENCODE.md) |
| **Using Recall from Pi**                                       | [`FOR_PI.md`](FOR_PI.md)             |
| **Using Recall from Codex**                                    | [`docs/CODEX_INTEGRATION.md`](docs/CODEX_INTEGRATION.md) |
| **Developing Recall** (build, test, conventions)               | [`CLAUDE.md`](CLAUDE.md)             |

## Roadmap

Recall is built around two integration surfaces: **MCP** (memory search and add, available from inside the agent) and **lifecycle hooks** (auto-extraction, session-start context injection, pre-compact flushes). Different agents support different surfaces — the table below tracks where each one stands.

| Agent                                                         | MCP |                      Lifecycle hooks                       | Status                                |
| ------------------------------------------------------------- | :-: | :--------------------------------------------------------: | ------------------------------------- |
| [**Claude Code**](https://claude.com/claude-code)             | ✅  |            ✅ Stop · SessionStart · PreCompact             | **Stable** — reference implementation; native plugin ships skills + MCP |
| [**Pi**](https://pi.dev/)                                     | ✅  | ⚠ Beta — native package with memory-injection + shutdown-capture extensions | Package + separate MCP adapter/config |
| [**OpenCode**](https://opencode.ai/)                          | ✅  | ⚠ Beta — `recall-extract` plugin; `session.idle` capture verified against a live OpenCode 1.18.5 server | Capture runtime-verified; compaction injection not yet |
| [**Codex CLI**](https://github.com/openai/codex)              | ✅  |               — native plugin, explicit dump only          | MCP + skills available                |
| [**Gemini CLI**](https://github.com/google-gemini/gemini-cli) |  —  |                             —                              | Coming soon                           |

**Candidate** — [Cursor](https://cursor.com): both `.cursor/hooks.json` and MCP are first-class; the integration model maps cleanly onto Recall's existing hook architecture. Tracked but not started.

Have an agent you'd like to see supported? [Open an issue](https://github.com/edheltzel/Recall/issues) — Recall is designed to be agent-agnostic, and any host that speaks MCP is a candidate.

## Documentation

| Guide                                      | Description                                                               |
| ------------------------------------------ | ------------------------------------------------------------------------- |
| [Installation](docs/installation.md)       | Prerequisites, install, verify, session extraction                        |
| [Managing Recall](docs/lifecycle.md)       | Which command when: install, update, uninstall, custom DB, recovery       |
| [CLI Reference](docs/cli-reference.md)     | All commands and options                                                  |
| [MCP Tools](docs/mcp-tools.md)             | Tools available to AI agents                                              |
| [Architecture](docs/architecture.md)       | Database, search, extraction pipeline                                     |
| Codebase Map (local)                       | Interactive visual map at `.agents/atlas/artifacts/2026-06-10-recall-codebase-map.html` — generated from the codegraph index, not committed (`.agents/` is gitignored) |
| [Agent Skills](docs/agent-skills.md)       | `recall-*` skills for Claude Code, Pi, and omp                            |
| [Codex Integration](docs/CODEX_INTEGRATION.md) | Native plugin install, MCP coverage, and lifecycle limits             |
| [Claude Integration](docs/CLAUDE_INTEGRATION.md) | Native plugin install, plugin/installer ownership split, migration    |
| [Pi Integration](docs/PI_INTEGRATION.md)   | Native package, separate MCP setup, lifecycle coverage, and host limits  |
| [Upgrading](docs/upgrading.md)             | Update, backup, migration system                                          |
| [Troubleshooting](docs/troubleshooting.md) | Common issues and fixes                                                   |
| [Changelog](CHANGELOG.md)                  | Release notes and breaking changes                                        |
## Acknowledgments

Graciously borrowing and features inspired by:

- [MemPalace](https://github.com/MemPalace/mempalace) — tiered session-start context, PreCompact hook, importance scoring
- [Personal AI Infrastructure (PAI)](https://github.com/danielmiessler/Personal_AI_Infrastructure) — TELOS framework integration

## License

MIT
</script>
<script>
document.getElementById('c').innerHTML = marked.parse(document.getElementById('src').textContent);
document.title='README render';
</script></body></html>
- Outcome: 🔧 2 issues found → auto-fixed ✅ across 2 runs (1h12m17s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 1 info
  • ⚠️ opencode/RecallExtract.ts:95 - The markdown drop is written with a bare writeFileSync (truncate-then-write), and the fix changes how often that happens: from once per session to once per assistant turn. RecallBatchExtract runs in a separate process on a 30-minute cron and reads the same path, so the torn-read window now recurs for the whole life of a session instead of existing once. Write to ${sessionId}.md.tmp and renameSync onto the final path — the temp name does not end in .md, so findMarkdownSessions (hooks/RecallBatchExtract.ts:227) skips it.
  • ⚠️ tests/hooks/sqlite-writers-concurrency.test.ts:110 - writeDecisionsBatch is fully synchronous, so Promise.all([Promise.resolve().then(fn), Promise.resolve().then(fn)]) runs the two calls in sequential microtasks with zero overlap — the first opens, commits, and closes its connection before the second begins. The test therefore passes identically on the pre-fix DEFERRED code and adds no protection for the immediate change, despite the name and the "two hosts writing the same tables" comment. Either drive the second writer from a subprocess (as peerHoldingWriteLock already does) or rename it to what it actually asserts: that two sequential writes keep their own project attribution.
  • ℹ️ hooks/lib/sqlite-writers.ts:434 - The rationale comment above insertMany.immediate(items) in writeExtractionErrors says these batches "read (the skipDuplicates probe) before they insert" — but this function takes no WriteOptions and has no probe; it is a single INSERT ... ON CONFLICT DO UPDATE, so the transaction already begins with a write and busy_timeout applied before the change. immediate is harmless here, but the stated justification does not describe the code, which will mislead the next person auditing which transactions need it. The same 5-line block is also pasted verbatim at all four sites while the canonical statement of the rule now lives in hooks/AGENTS.md:28; collapse it to one comment near openDb plus one-line pointers, with the upsert site described accurately.
  • ℹ️ hooks/AGENTS.md:28 - The new invariant ("A write transaction that reads first must be IMMEDIATE") reads as covering every read-then-write path in lib/sqlite-writers.ts, and lists only "a transaction whose first statement is a write" as the exemption. writeLoaEntryFromExtraction (sqlite-writers.ts:361) probes with a SELECT and then inserts, but is in neither category: it uses no explicit transaction, so the SELECT autocommits and releases its read lock before the INSERT takes the write lock with busy_timeout honoured. Naming that third case in the rule prevents a future reader from either "fixing" a fifth call site that needs nothing, or assuming the invariant guarantees atomicity it does not (two concurrent replays of the same extraction can still both pass the LoA probe and insert).
  • ℹ️ tests/hooks/sqlite-writers-concurrency.test.ts:83 - expect(waited).toBeGreaterThan(200) against a 600 ms peer hold fails if the test process is descheduled for ~400 ms between reading LOCKED and starting the write — exactly what a loaded macOS runner executing the full suite can produce, and exactly what an isolated run cannot. That matters because the handoff (.agents/atlas/handoffs/2026-07-25-opencode-phase4-runtime-verification.md:104) clears the unreproduced full-suite failure from this branch on the strength of 15 targeted runs of this file; targeted runs are the wrong instrument for a wall-clock floor. Have the peer print a timestamp when it commits and assert the writer's completion is after that, instead of against a fixed floor.
  • ℹ️ scripts/e2e-opencode-runtime.ts:57 - STUB_PORT/SERVER_PORT are hardcoded to 47611/47612. If either is occupied, Bun.serve throws EADDRINUSE before any diagnostic output, or waitForServer burns 60 s and then reports "never became healthy" with a log that will not explain why. Teardown also SIGKILLs the bunx wrapper, which is not a reliable way to reap the OpenCode server it launched, so an interrupted local run can leave the port held and make every rerun fail opaquely. Bind the stub on port 0 and read back the assigned port, and pass a free port to opencode serve (or spawn detached and kill the process group).
  • ℹ️ docs/OPENCODE_INTEGRATION.md:162 - "RecallBatchExtract already re-extracts a drop that grows, so the fuller transcript reaches memory without new machinery" is true only past a threshold: findCandidates requires GROWTH_THRESHOLD = 0.5 (hooks/RecallBatchExtract.ts:64), so once a drop is extracted at size S, later turns must add >50% before it is re-extracted. The drop file now holds every turn (the fix works), but the last portion of a long session may never reach the database. Worth one clause of precision on a claim this branch newly makes, since the same threshold governs Claude Code JSONL and is presumably intentional.
  • ℹ️ CHANGELOG.md:18 - The existing Unreleased entry still says the e2e harness "provisions OpenCode 1.18.4", while the pin moved to 1.18.5 in this branch (scripts/lib/opencode-runtime.ts:19). It is the only remaining 1.18.4 reference in the repo, and it ships in the next release notes alongside the new entry that says 1.18.5.
  • ℹ️ uninstall.sh:470 - for helper in session-export.ts is hardcoded here and again at lib/install-lib.sh:2304, so adding a second helper requires editing two files or the file is orphaned in every existing install — the drift the repo's DRY rule and the RECALL_SKILL_NAMES precedent exist to prevent, and which opencode/AGENTS.md:29 currently documents as a manual obligation. A single RECALL_OPENCODE_PLUGIN_HELPERS array in lib/install-lib.sh consumed by both loops removes the obligation (the adjacent plugin-name list has the same shape and could follow).

🔧 Fix: atomic drops, real concurrency test, ephemeral e2e ports
1 info still open:

  • ℹ️ opencode/RecallExtract.ts:75 - writeAtomic derives the temp name from the target alone (${path}.tmp), so it is a fixed, process-independent path. For the drop file that is unreachable in practice — the path is keyed by session ID, inFlight serializes same-session exports, and a session lives in one server process. For the tracker it is a genuinely shared path: .extracted.json.tmp is the same for every OpenCode instance, two instances across two projects is a normal condition, and both rewrite it on every turn, so their writes can interleave into that temp file before either rename publishes the mixed bytes. Not a regression — the pre-fix code interleaved on the final file instead, and the failure self-heals through the existing "tracker unreadable; starting fresh" path at the cost of one redundant re-export. Interpolating process.pid into the temp name removes the window entirely and keeps the suffix outside findMarkdownSessions's *.md filter.
🔧 **Test** - 2 issues found → auto-fixed ✅
  • ⚠️ tests/install/npm-pack.test.ts - Pre-existing flake, not from this branch: tests/install/npm-pack.test.ts and tests/install/update.test.ts trip bun's 5s default per-test timeout when the npm registry / GitHub release lookup are cold. Across three full-suite runs I saw 4 fail → 2 fail → 0 fail. Both pass in isolation with headroom (bun test --timeout 60000 ./tests/install/npm-pack.test.ts → 9 pass; update.test.ts → 17 pass once warm), and standalone the underlying commands take 0.4-0.7s. Neither file, nor update.sh, nor the packaging path is touched by this branch, so this is out of scope here — but the suite has a load-dependent red baseline that an explicit per-test timeout would remove.
  • ℹ️ README.md:21 - README now contradicts itself on OpenCode's maturity. The compatibility table at line 324 moved to "⚠ Beta", but the intro paragraph at line 21 still reads "Alpha for OpenCode (MCP works; lifecycle extensions are early)". Before this diff both said Alpha, so the inconsistency is introduced here — a one-line copy fix on the front page.
  • bun test — full suite, three runs (final: 1298 pass / 0 fail)
  • bun run test:e2e:opencode:runtime — live OpenCode 1.18.5 server, run twice; produced session_idle_per_turn=1.00, deltas [1,1,1], plugin_load_errors=0
  • RECALL_E2E_KEEP=1 bun run test:e2e:opencode:runtime — kept the temp root to capture the real drop markdown and .extracted.json digest tracker
  • bun run test:e2e:opencode — pipeline e2e, including uninstall_preserves_user_plugin_lib=verified and installer_jsonc_rollback=verified
  • Manual full-chain bridge: bun run &lt;evidence&gt;/bridge-live-opencode-to-recall-search.ts — live opencode serve session → installed plugin → drop → RecallBatchExtract --allrecall search LIVEBRIDGEMARKER_BRAVO --table loaFound 1 result(s); production DB asserted byte-identical (size/ino unchanged)
  • Non-vacuity: reverted insertMany.immediate(items)insertMany(items) at all 4 sites, then bun test ./tests/hooks/sqlite-writers-concurrency.test.ts → 0 pass / 3 fail with SQLITE_BUSY at 64-218ms
  • Non-vacuity: added a top-level export { exportSession, renderSessionExport } to opencode/RecallExtract.ts, then bun run test:e2e:opencode:runtime → exit 1, quoting OpenCode's failed to load plugin … &#34;Object is not a function&#34;
  • Non-vacuity: restored tracker.has(sessionId) || to the idle early-return, then bun test ./tests/opencode-integration.test.ts → 3 fail, and bun run test:e2e:opencode:runtimethe drop lost RUNTIMETURNBRAVO, RUNTIMETURNCHARLIE … drop bytes=250
  • bun test ./tests/opencode-integration.test.ts ./tests/pi-integration.test.ts --reporter=junit inside a linked git worktree — extracted the 10 tests #243 named, incl. both helpers' V-1/V-4 non-zero-exit + byte-identical assertions, 0 failures
  • bun test ./tests/install/restore.test.ts — documented ./install.sh restore rollback path, incl. a non-interactive restore declines and changes nothing
  • bun test --timeout 60000 ./tests/install/npm-pack.test.ts → 9 pass — discriminates the cold-network flake from a real failure
  • grep -c &#39;insertMany.immediate(&#39; hooks/lib/sqlite-writers.ts → exactly 4 sites, no bare insertMany( left; inspected hooks/lib/consolidate-core.ts:192 and hooks/RecallPreCompact.ts:370 to confirm both open with a write (so leaving them DEFERRED is correct)
  • npm pack --dry-run --json → confirms opencode/lib/session-export.ts ships in the published package
  • git status --porcelain after every revert experiment → clean; git worktree list → no stray worktrees

🔧 Fix: stub host pi in uninstall tests; align README OpenCode maturity
✅ Re-checked - no issues remain.

  • bun run test:e2e:opencode:runtime - live OpenCode 1.18.5 server, EXIT=0, session_idle_per_turn=1.00, plugin_load_errors=0, drop_contains_all_turns=true (560 bytes)
  • bun run test:e2e:opencode - pipeline e2e at HEAD, EXIT=0 (export-failure retry, concurrent WAL writers, JSONC rollback all verified)
  • bun test --timeout 30000 - 1298 pass / 0 fail / 40335 expect() across 107 files, run inside a confirmed detached-HEAD git worktree (#243 acceptance item 1)
  • bun test - CI's exact command (bare, bun's 5s default), 1298 pass / 0 fail, no failures
  • Mutation proof of the pi stub: disabled remove_memory_section &#34;$agents_md&#34; pi in uninstall.sh, ran bun test ./tests/install/uninstall.test.ts -t &#34;Pi AGENTS.md&#34; -> (fail) with expect(received).not.toContain(&#34;## MEMORY&#34;); uninstall.sh restored, bash -n uninstall.sh clean
  • Read remove_pi (uninstall.sh:513-525) to confirm pi remove exit code and output are both discarded, and grepped tests for assertions on its log lines (none)
  • Timing A/B: git checkout 6f59fdd -- tests/install/uninstall.test.ts then bun test --timeout 60000 and bare bun test, vs HEAD; plus 3x /usr/bin/time -p env PI_CODING_AGENT_DIR=... pi remove ... --no-approve (0.50s each)
  • Per-test timings extracted from --reporter=junit: all 13 uninstall.test.ts tests between 0.431s and 0.862s under full-suite load; slowest test in whole suite 4.167s
  • Rendered README.md to HTML and screenshotted both maturity statements via chrome-devtools-axi (intro blockquote at line 21 and the Roadmap compatibility table row)
  • rg -n &#39;Alpha for|lifecycle extensions are early&#39; across the tree - no matches remain
  • git status --short clean and dist/ build output removed after both e2e runs
🔧 **Document** - 1 issue found → auto-fixed (2) ✅
  • ⚠️ docs/OPENCODE_INTEGRATION.md:360 - This change corrected the doc's architecture diagram from "7 tools" to "9 tools", which makes the same file's "Tool Name Mapping" table (7 rows) self-contradictory, and FOR_OPENCODE.md enumerates the same 7. docs/mcp-tools.md is the owner and documents 9: memory_dump and decision_update are missing from both OpenCode-facing lists. This is pre-existing drift, not caused by this change, and the fix is a consolidation rather than an edit: the mapping table adds nothing over the one prefix rule (memory_search -> recall-memory_memory_search) plus a pointer to the owner, while FOR_OPENCODE.md genuinely needs two new tool sections. Adding rows to a duplicate list would be synchronization; deleting the table is restructuring beyond this change's scope. No tracking issue exists (searched open and closed via gh issue list). Proposed follow-up: reduce the mapping table to the prefix rule plus a link to docs/mcp-tools.md, and add the two missing tools to FOR_OPENCODE.md.

🔧 Fix: point OpenCode tool docs at mcp-tools owner
1 warning still open:

  • ⚠️ opencode/recall-memory.md:27 - The installed OpenCode agent definition now contradicts the guide this pass corrected. Its rule 5 tells the agent "When the user says /dump, capture the session: recall-memory_memory_dump with a descriptive title" — the title-only call I just documented in FOR_OPENCODE.md as resolving to the wrong host's transcript, because coreDump (src/commands/dump.ts:122) falls through to discoverCurrentSession() and claudeSessionSource is tried first with no host gate. That is not theoretical: FOR_OPENCODE.md:165 advertises the shared-database dual-host setup, which is exactly the configuration where the Claude adapter wins. Separately, its Available Tools table (lines 41-50) lists 8 of 9 tools, omitting decision_update; the frontmatter recall-memory_*: true wildcard means no capability is actually lost, so that half is completeness rather than breakage. I did not fix either: this is a shipped agent prompt (package.json#files includes opencode/) whose rule text shapes agent behavior rather than describing it, it is outside the two files the round-1 directive authorized, and adding a table row would be the same list synchronization that directive rejected. Proposed follow-up: correct rule 5 to require explicit messages + source: &#34;opencode&#34;, and decide whether the tool table should stay an enumeration at all or become a pointer to docs/mcp-tools.md.

🔧 Fix: fix OpenCode agent dump rule, point tools at owner
✅ Re-checked - no issues remain.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

edheltzel added 17 commits July 25, 2026 11:03
…ss it exposes

Phase 4 was marked complete, but its e2e synthesizes the session.idle payload
instead of observing OpenCode emit it, and the frequency the design hedged on
was never measured.

Measured against a live OpenCode 1.18.5 server: session.idle fires once per
assistant turn (3 turns -> 3 events). Because the adapter's tracker suppresses
a session permanently after the first idle, every turn after the first is
silently discarded.
session.idle fires once per assistant turn, not once per session (measured
against OpenCode 1.18.5: three turns produced three events, one per turn). The
adapter gated on a permanent tracker, so the first idle wrote a drop containing
only turn 1 and every later idle returned early. Reproduced end to end with a
real three-turn session and the real plugin loaded by a real OpenCode server:
the drop froze at 154 bytes and two of three markers never reached it.

The tracker now records a content digest per session instead of 'this session
is done', so a grown session overwrites its earlier, shorter drop while an
unchanged one still costs no write. RecallBatchExtract already re-extracts a
drop that grows, so the fuller transcript reaches memory without new
machinery. Legacy array trackers load with an empty digest and converge after
one re-export.

Replaces two tracker tests that re-implemented the logic inline — and so passed
with the bug present — with tests that drive the real plugin factory. All four
new behavioural cases fail on the pre-fix adapter.
OpenCode invokes EVERY export of a top-level plugins/*.ts as a plugin factory.
opencode/RecallExtract.ts exported three test helpers beside its factory, so on
each launch OpenCode called exportSession() with its plugin context as the
shell, the tagged-template call hit a non-function, and the server logged
'failed to load plugin ... Object is not a function'.

Observed against a real OpenCode 1.18.5 server; reproduced with the plugin
installed by the real installer, and confirmed by exporting only the factory,
which loads clean. The existing e2e could not catch it: it imports the adapter
directly and supplies its own $, fabricating the very shape OpenCode breaks on.

Helpers move to opencode/lib/session-export.ts. OpenCode globs only top-level
plugins/*.ts, so a nested module is importable without being mistaken for a
plugin — verified empirically before choosing this layout. Install and uninstall
carry the nested helper, and uninstall only removes plugins/lib when Recall
emptied it.

A new test asserts each plugin entry exports exactly one function, so a stray
helper export cannot silently reintroduce this.
…erver

The existing e2e imports the adapter and synthesizes the session.idle payload,
so it proves Recall's handler accepts that shape — not that OpenCode emits it,
and not that OpenCode ever loads the plugin. This harness closes both gaps:

- Discovery: installs via the real recall_install_opencode_platform, then
  observes OpenCode load the plugin from that path and asserts zero plugin load
  errors.
- Frequency: counts session.idle from OpenCode's own /event stream across three
  turns of ONE session in ONE long-lived server process, so the number cannot be
  an artifact of process teardown. Measured 1.00 per assistant turn, and the
  script fails loudly if that ever changes.
- Completeness: asserts the drop carries every turn, which is the end-to-end
  regression for the discarded-turns fix.

The model is a local OpenAI-compatible stub — the subject under test is
OpenCode's session lifecycle, so a hosted model would only add flakiness. A
PATH shim supplies `opencode` because the plugin shells out to it, exactly as
in a real install.

The pinned version now lives once in scripts/lib/opencode-runtime.ts; the
existing e2e resolves the CLI through it instead of repeating the pin.
Concurrent Claude and OpenCode extraction against the shared WAL database lost
records to 'database is locked' even though openDb sets busy_timeout=5000 — the
isolated OpenCode e2e failed its own concurrency assertion this way.

Cause: the batch writers probe for duplicates (skipDuplicates) before
inserting, and SQLite fails a read-to-write upgrade inside a DEFERRED
transaction with SQLITE_BUSY *immediately*, never consulting busy_timeout.
Measured on this machine while a peer held the write lock: the deferred
read-then-write path failed in 1ms, while taking the write lock up front waited
the full timeout. So the timeout the comment relies on was never reachable on
the retry-safe path extraction actually uses.

Taking the transactions IMMEDIATE makes the peer wait honoured. Scoped to the
four batch writers: the other hook transactions (consolidate-core,
RecallPreCompact) already begin with a write, so busy_timeout applies to them
and they are left alone.

The new tests hold the write lock from a separate process — the writers are
synchronous, so an in-process timer could never release it — and assert the
write waits and lands. They fail with 'database is locked' on the deferred
version.
./install.sh restore is documented in docs/installation.md but recall_do_restore
had no coverage, and neither did the collision backup that is the other half of
rollback.

Covers restoring the latest snapshot, restoring a specific timestamp, the
pre-restore snapshot that makes a restore itself reversible, and non-zero exits
for an unknown timestamp and for no backups at all.

Also pins the automation contract: restore gates on a default-No confirm, and
_confirm returns the default when stdin is not a TTY, so restore is a
deliberate no-op in CI or scripts. That is asserted directly — it declines,
mutates nothing, and writes no pre-restore snapshot — rather than left as a
surprise. The restoring path is exercised by overriding _confirm in the sourced
shell, so the safety default is tested rather than weakened.

The collision tests prove a hand-edited user file at a Recall-managed path
survives install as recoverable bytes, and that an identical file is replaced
without a needless backup.
AGENTS.md mandates workers run in an isolated worktree, so a worktree is the
default environment for automated work. Ten lifecycle/config tests once failed
there while passing in a normal checkout (#243), which teaches agents to read a
red suite as normal — the exact signal loss a suite exists to prevent.

This leg creates a real worktree, asserts it starts without node_modules (#165's
condition, so the workaround stays visible rather than accidental), installs
there, and runs lint plus the full suite. Verified locally in a fresh worktree:
1298 pass, 0 fail.

Closes the last open acceptance item on #243; the other two were already met —
recall_configure_opencode_mcp and recall_configure_pi_mcp both exit 1 and leave
a malformed config byte-identical.
The design doc asserted OpenCode 1.18.4 'emits session.idle through the event
hook' and cited the pipeline e2e as pinning it. That e2e imports the adapter and
supplies the payload itself, so it proved Recall ACCEPTS that shape — not that
OpenCode sends it, and not that OpenCode loads the plugin at all. Phase 4
Evidence now says which of the two scripts proves what, and names compaction
context injection as still unverified at runtime.

Records the measured session.idle frequency (once per assistant turn, OpenCode
1.18.5) in place of the hedge about adding debouncing 'if future evidence shows
repeated idle events' — the evidence exists now, and it points the other way.
Also documents the single-export plugin contract and the PATH requirement.

README moves OpenCode from Alpha/In progress to Beta with the scope stated:
capture is runtime-verified, compaction injection is not. Fixes stale details in
the same doc (7 tools -> 9, the tracker's digest semantics, missing files).
opencode/ became a durable boundary with rules a future session cannot infer
from the code, so it gets a child AGENTS.md: plugin entry points export exactly
one factory, helpers nest under lib/ and must be installed and uninstalled with
the plugins, and session.idle is per-turn so dedup is by content digest rather
than 'session seen'.

hooks/ gains the transaction rule the concurrency fix established: a write
transaction that reads first must be IMMEDIATE, because SQLite fails the upgrade
with SQLITE_BUSY instantly and never consults busy_timeout.

tests/ records why OpenCode has two e2e scripts and which claims belong in each,
so a host-behaviour claim does not get added to the script that structurally
cannot support it.

Root index gains opencode/, and scripts/ is named in the owned-at-root list it
was missing from.
…s/lib

The runtime e2e was not wired into CI, so nothing on a PR actually enforced the
measured once-per-turn session.idle frequency that CHANGELOG and the design doc
both claim is asserted. It runs on the macOS leg now, making that claim true.

Verified the gate is not vacuous on the path CI takes: with OPENCODE_BIN unset
(so the run goes through bunx, adding a process layer between the server and the
captured log) a deliberately reintroduced second plugin export still produced
'OpenCode reported 1 plugin load error(s)' and failed the run.

The pipeline e2e now also proves uninstall removes Recall's plugin and its
nested helper while leaving a user-owned file in plugins/lib untouched — the
directory Recall creates is only removed when Recall emptied it.
@edheltzel
edheltzel merged commit 38f8a4c into main Jul 25, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant