feat(memory): project-local default for the files-backend store (0.10.0)#23
Merged
Conversation
…-to-plan) Converged spec and build plan for making eidetic's files-backend store default to the git repo root (public->repo, private->$HOME, override wins). Two-store read model; 5 tasks across 4 waves. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WR1vQvraNdkhuYNpy68fzh
Add _home_store_dir, _override_dir, _git_toplevel, _resolve_write_dir, and _candidate_read_dirs helpers to eidetic/memory/backend.py with CWD-based caching for git lookups. Includes 17 unit tests covering in-repo, subdir, outside-repo, git-absent, override, dedup, and caching scenarios.
…mem...
TASK t1 (from docs/plans/2026-06-23-eidetic-cli-s-default-memory-store-becomes-project.md):
Add visibility-aware store-path RESOLVER HELPERS to eidetic/memory/backend.py. THIS TASK IS ONLY THE PURE HELPERS + THEIR UNIT TESTS. Do NOT change upsert/search/all yet (that is task t2). Do NOT touch mongo/neo4j logic.
Work TEST-FIRST: write tests/test_store_resolver.py first, then implement until green.
Add these helpers near the existing _bridge_env in eidetic/memory/backend.py (study that file's style/imports; Path and os are already imported):
_home_store_dir() -> str
return str(Path.home() / ".eidetic" / "memory") # today's default
_override_dir() -> str | None
return os.environ.get("EIDETIC_DATA_DIR") or None # explicit override
_git_toplevel() -> str | None
Return `git rev-parse --show-toplevel` (stripped) when the CURRENT WORKING DIR
is inside a git repo; return None when outside a repo OR git is unavailable OR
any error. Use subprocess.run([...], capture_output=True, text=True) with NO shell.
Catch non-zero return code AND FileNotFoundError -> return None. Never raise, never
print, never construct a CliError.
CACHE per process keyed by os.getcwd(): use a module-level dict {cwd: result} so a
batch ingest of N records (fixed cwd) spawns AT MOST ONE git subprocess, while a
test that os.chdir()s to a different dir gets a fresh result. Do NOT use a bare
lru_cache with no args (it would never refresh across chdir).
_resolve_write_dir(visibility: str) -> str
Precedence EXACTLY:
1. if _override_dir(): return it
2. if visibility == "public" and _git_toplevel(): return str(Path(top)/".eidetic"/"memory")
3. return _home_store_dir()
(So a PRIVATE record, or ANY record outside a repo, resolves to home.)
_candidate_read_dirs() -> list[str]
For multi-store reads. Precedence:
- if _override_dir(): return [override] # single dir, byte-identical to today
- else: dirs = [_home_store_dir()]; if _git_toplevel(): repo = str(Path(top)/".eidetic"/"memory");
if repo not in dirs: dirs.append(repo); return dirs
Must contain NO duplicate when the repo toplevel store == home store.
CWD-based only. Do NOT call or reuse find_culture_yaml (that walks the installed module path; wrong target here).
tests/test_store_resolver.py MUST cover, using tmp_path + monkeypatch (monkeypatch HOME and EIDETIC_DATA_DIR; use subprocess to `git init` a temp repo and os.chdir into it / a subdir):
- in-repo: _git_toplevel == repo root; _resolve_write_dir("public") == <repo>/.eidetic/memory
- subdir-of-repo: still resolves to the repo root
- _resolve_write_dir("private") == home even inside a repo; outside a repo public also == home
- git-absent: monkeypatch subprocess.run to raise FileNotFoundError -> _git_toplevel() is None (no raise)
- EIDETIC_DATA_DIR set: _resolve_write_dir(any) and _candidate_read_dirs() return exactly that dir (single)
- _candidate_read_dirs() has no duplicate when toplevel store == home
- caching: calling _git_toplevel() twice in the same cwd invokes the git subprocess at most once (assert via a mock/counter)
Run `uv run pytest tests/test_store_resolver.py -q` and make it pass. Keep style consistent with the file (black/isort/flake8 clean). Commit your work.
Implement the task above in this repository.
Rules:
- Make the SMALLEST change that correctly satisfies the task.
- Follow the repository's existing patterns, style, and conventions — read the
neighbouring files first so your change reads like the surrounding code.
- Keep edits lint-clean: respect the project's maximum line length and end every
text file with exactly one trailing newline.
- You may read, create, modify files, and run commands as needed.
- Don't widen the scope: do exactly what was asked, nothing more.
When you are done, call finish with a short summary of exactly what you changed
and why.
colleague-authored (drive colleague/b4d5e6b27b4b-task-t1-from-docs-plans-2026-06-23), TDD-gated: baseline 304 passed before, 321 passed after (+17 resolver unit tests). Pure additive helpers in backend.py; existing code untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WR1vQvraNdkhuYNpy68fzh
…mem...
TASK t2 (from docs/plans/2026-06-23-eidetic-cli-s-default-memory-store-becomes-project.md):
Wire upsert/search/all in eidetic/memory/backend.py onto the t1 resolver helpers
(_resolve_write_dir, _candidate_read_dirs, _git_toplevel, _override_dir, _home_store_dir),
which already exist and are tested. This is the CRUX — the public/private no-leak invariant
rides on it. Work TEST-FIRST where practical; keep all existing tests green.
ALL new directory logic MUST be gated behind `self._name == "files"`. The mongo and neo4j
branches of _bridge_env and their single-store reads MUST be byte-unchanged.
1) _bridge_env: replace ONLY the files branch's hardcoded
os.environ["DR_DATA_DIR"] = EIDETIC_DATA_DIR or str(Path.home()/".eidetic"/"memory")
Keep _bridge_env able to set DR_DATA_DIR to an EXPLICIT directory. Simplest: add a
parameter `data_dir: str | None = None`; when name=="files", set DR_DATA_DIR to
`data_dir` if given else `_resolve_write_dir("private")` (the safe home default for any
caller that doesn't pass a dir). mongo/neo4j branches unchanged.
2) upsert(record): for the files backend, set DR_DATA_DIR via the resolver using the
RECORD's own visibility -> _bridge_env("files", data_dir=_resolve_write_dir(record.scope.visibility)).
So a public record lands in the repo store, a private record in the home store. For
mongo/neo4j keep calling _bridge_env(self._name) exactly as today.
3) search(...): for the files backend, gather candidates from EVERY dir in
_candidate_read_dirs():
seen = {} # id -> Record, dedup by id (first dir wins is fine)
for d in _candidate_read_dirs():
_bridge_env("files", data_dir=d)
for env in drstore.list(scope=DRScope(name=scope.name, visibility=scope.visibility),
backend="files", **self._kwargs):
r = record_from_envelope(env)
seen.setdefault(r.id, r)
candidates = list(seen.values())
Then apply the EXISTING pipeline UNCHANGED: the defensive
`[r for r in candidates if can_serve(scope, r.scope)]`, then facet filters, then
rank(mode, query, candidates, self._embed, top_k, alpha=alpha, case_sensitive=...).
For mongo/neo4j keep today's single drstore.list call exactly.
INVARIANT TO PRESERVE: reading the home dir for a PUBLIC query must NOT leak private
records — can_serve already fails closed (private served only on exact scope match), so
do not weaken or bypass it. A private-scope query returns its own private records (home)
PLUS public records (repo); a public-scope query returns ONLY public records from any dir.
4) all(): for the files backend, enumerate across EVERY dir in _candidate_read_dirs(),
union by id:
seen = {}
for d in _candidate_read_dirs():
_bridge_env("files", data_dir=d)
backend = drstore.get_backend("files", **self._kwargs)
for env in backend.all():
r = record_from_envelope(env); seen.setdefault(r.id, r)
return list(seen.values())
For mongo/neo4j keep today's single-store all().
(sweep then re-upserts each changed record via upsert(), which step 2 routes back to the
dir matching that record's OWN visibility — no extra work needed here, but DO confirm it.)
5) migrate_store: unchanged behavior is acceptable for this task (it still resolves the home
default via _bridge_env("files")); do NOT regress it.
Acceptance (all must hold; add/keep tests):
- files upsert writes public -> _resolve_write_dir("public"), private -> _resolve_write_dir("private")
(verify which <scope>__<visibility>.jsonl appears under which dir)
- files search unions candidates across _candidate_read_dirs() with no duplicate records;
private-scope query returns own-private (home) + public (repo); public-scope query never
returns private records even though home is read (no-leak holds)
- files all() spans both dirs; sweep re-upsert lands each record back in its visibility's dir
- new logic gated behind self._name=="files"; mongo/neo4j paths untouched; old hardcoded
Path.home()/.eidetic/memory line is the only path logic replaced
- with EIDETIC_DATA_DIR set, _candidate_read_dirs() is a single dir so search/all/upsert all
use exactly that dir -> behavior byte-identical to before (run the full existing suite green)
Run `uv run pytest -q` (whole suite) and make it pass. Keep black/isort/flake8 clean. Commit.
Implement the task above in this repository.
Rules:
- Make the SMALLEST change that correctly satisfies the task.
- Follow the repository's existing patterns, style, and conventions — read the
neighbouring files first so your change reads like the surrounding code.
- Keep edits lint-clean: respect the project's maximum line length and end every
text file with exactly one trailing newline.
- You may read, create, modify files, and run commands as needed.
- Don't widen the scope: do exactly what was asked, nothing more.
When you are done, call finish with a short summary of exactly what you changed
and why.
…al ~/.eidetic store The autouse fixture cleared EIDETIC_DATA_DIR + the git cache but not HOME, so private and outside-repo upserts wrote real records into the developer's live ~/.eidetic/memory. Redirect HOME to an isolated tmp dir. (integrator fix on colleague's drive branch.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WR1vQvraNdkhuYNpy68fzh
colleague-authored crux (drive colleague/066538dfdbb7-task-t2-from-docs-plans-2026-06-23) + integrator HOME-isolation fix. TDD-gated: 321 passed before, 338 after (+17 multi-store tests). No-leak invariant doubly-enforced (drstore can_serve + eidetic defensive can_serve); mongo/neo4j untouched; EIDETIC_DATA_DIR override short-circuits to single-dir (regression lock). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WR1vQvraNdkhuYNpy68fzh
…eview follow-up) Addresses the independent no-leak review's top suggestion: clarify that union-by-id prefers the home copy and can only under-serve a duplicate id, never leak (the survivor is still can_serve-gated). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WR1vQvraNdkhuYNpy68fzh
…mem...
TASK t3 (from docs/plans/2026-06-23-eidetic-cli-s-default-memory-store-becomes-project.md):
Add END-TO-END + regression tests for the two-store routing in a NEW file
tests/test_store_routing_e2e.py. Do NOT modify production code (t1/t2 already did it).
Tests must pass: `uv run pytest tests/test_store_routing_e2e.py -q`.
Use the same pattern as tests/test_added_by_e2e.py: import and call
eidetic.cli._commands.remember.cmd_remember and recall.cmd_recall with a constructed
argparse.Namespace, OR call eidetic.cli.main([...]). Capture stdout with capsys when you
need the recall JSON (pass json=True / --json).
CRITICAL test-isolation rules (the routing depends on cwd + HOME + git, so set them up real):
- monkeypatch.setenv("HOME", str(tmp_home)) so Path.home() points at an isolated dir
- monkeypatch.delenv("EIDETIC_DATA_DIR", raising=False) for the REPO-ROUTING cases
(an override would short-circuit the very behavior under test)
- create a real git repo: subprocess.run(["git","init",str(repo)], capture_output=True, check=True)
- os.chdir into the repo (restore in finally), and CLEAR the resolver cache each time:
import eidetic.memory.backend as be; be._GIT_CACHE.clear()
- in recall, use mode="exact" or "keyword" (lexical) so tests never depend on the live
embed endpoint (model-gear). Follow how existing recall tests pass mode.
Cases (each its own test):
1. public-in-repo: remember a record with --visibility public while cwd is inside the temp
git repo. Assert the file <repo>/.eidetic/memory/<scope>__public.jsonl EXISTS and the
record is in it; assert NOTHING was written under tmp_home/.eidetic/memory. Then recall
the same query (public scope) from the repo and assert the record is returned.
2. private-to-home: remember --visibility private inside the repo. Assert it lands in
tmp_home/.eidetic/memory/<scope>__private.jsonl and NOT under <repo>/.eidetic/memory.
3. private-recall-merges: with a private record (home) AND a public record (repo) both
present for the same scope name, a recall with a PRIVATE query scope returns BOTH
(own private + the public pool). A recall with a PUBLIC query scope returns ONLY the
public record (no private leak).
4. override-byte-identical: set EIDETIC_DATA_DIR=tmp/override. remember (public AND private)
+ recall all touch ONLY that dir; nothing under repo or home. (Locks the override regression.)
5. outside-repo: cwd in a NON-git tmp dir. remember (public or private) lands under
tmp_home/.eidetic/memory. _git_toplevel is None.
6. clean-break: pre-populate tmp_home/.eidetic/memory with a record (write a record there
first, via override or by remembering outside a repo). Then remember a public record
while inside the repo. Assert the pre-existing home file is byte-UNCHANGED (read bytes
before/after) — the repo write never touches home.
7. sweep-both-dirs: put one record in the repo public store and one in the home private
store such that lifecycle transitions would apply (e.g. force via supersedes or an old
created date — reuse helpers/patterns from the existing sweep tests). Run sweep and
assert records in BOTH dirs are transitioned, and each transitioned record is written
back to the dir it came from (public->repo, private->home).
8. mongo/neo4j unaffected: a quick assertion that get_backend("mongo")/("neo4j") path
resolution is unchanged (no repo/home dir logic applied) — or assert _bridge_env for
mongo/neo4j only sets DR_MONGO_URI/DR_NEO4J_URI as before. Keep it light; skip if it
needs a live server (use the existing skip pattern).
Keep black/isort/flake8 clean. Commit.
Implement the task above in this repository.
Rules:
- Make the SMALLEST change that correctly satisfies the task.
- Follow the repository's existing patterns, style, and conventions — read the
neighbouring files first so your change reads like the surrounding code.
- Keep edits lint-clean: respect the project's maximum line length and end every
text file with exactly one trailing newline.
- You may read, create, modify files, and run commands as needed.
- Don't widen the scope: do exactly what was asked, nothing more.
When you are done, call finish with a short summary of exactly what you changed
and why.
colleague-authored (drive colleague/ae1d41412fda-task-t3-from-docs-plans-2026-06-23), hermetic (HOME-isolated). 8 e2e tests across the CLI: public-in-repo, private-to-home, private-recall merges (public query never leaks private), override byte-identical, outside-repo, clean-break (byte-compare), sweep across both dirs, mongo/neo4j unaffected. TDD-gated: 338 -> 346. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WR1vQvraNdkhuYNpy68fzh
Update header comments in remember.sh and recall.sh, SKILL.md descriptions for both remember and recall, migrate --data-dir help text, catalog.py store-location prose, and CLAUDE.md to describe the new default resolution: PUBLIC records inside a git repo go to <repo-root>/.eidetic/memory (committed), PRIVATE/outside-repo records go to $HOME/.eidetic/memory (never committed), EIDETIC_DATA_DIR override wins, recall merges both. No executable behavior changed — docs/comments only.
colleague-authored (drive colleague/dcde193859ad-task-t4-from-docs-plans-2026-06-23), docs/comments only (no executable change): remember.sh/recall.sh headers, /remember + /recall SKILL.md descriptions, CLAUDE.md store-location subsection, migrate.py + catalog.py help. markdownlint + teken rubric green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WR1vQvraNdkhuYNpy68fzh
- version 0.9.3 -> 0.10.0 + CHANGELOG entry for the project-local store default - bandit: # nosec B607 on the intentional git-on-PATH subprocess in _git_toplevel - markdownlint: backtick <repo-root> paths in CHANGELOG (MD033 inline-HTML) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WR1vQvraNdkhuYNpy68fzh
Contributor
Author
|
/agentic_review |
Code Review by QodoNew Review StartedThis review has been superseded by a new analysis |
PR Summary by Qodofeat(memory): route files-backend store by repo+visibility (default 0.10.0) Description
Diagram
High-Level Assessment
Files changed (19)
|
Resolves SonarCloud S1192 (literal ".eidetic" duplicated 4x). Adds _STORE_SUBPATH + _store_dir(base) and routes _home_store_dir / _resolve_write_dir / _candidate_read_dirs / _bridge_env through it. No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WR1vQvraNdkhuYNpy68fzh
Code Review by Qodo
1.
|
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
…oplevel Addresses two Qodo review findings on PR #23 (inline comment 3463350770 + review 4557593486). #1 (correctness, defensive): StoreBackend.search() deduped multi-store candidates by id (seen.setdefault, home-before-repo) BEFORE eidetic's own can_serve filter ran. eidetic re-applies can_serve as defense-in-depth so the no-leak invariant holds "regardless of the store's behavior" — but the ordering left that defensive layer non-self-sufficient: a non-serveable private copy could win the id slot and then be dropped, hiding a serveable public copy of the same id. Move can_serve INTO the read loop, before setdefault, so only serveable copies enter the dedup map. (Not currently exploitable through the real data_refinery.store.list(), which pre-filters by can_serve at the source — but this makes eidetic's own layer correct on its own terms.) #2 (reliability): _git_toplevel() promised "Never raises" but os.getcwd() ran outside the try and only FileNotFoundError was caught. Wrap getcwd (return None without caching on failure) and broaden the subprocess except to OSError so the helper truly fails closed before every files-backend op. Tests: both regressions reproduce at eidetic's layer (fail pre-fix, pass post-). The dedup test simulates an unfiltered store via monkeypatch — a test through the real list() cannot reproduce it (DR pre-filters), which is why the first colleague-written test passed on old code. 348 passed, 2 skipped. #3 (recall mutates committed store) deferred to a follow-up issue — it changes documented recall-reinforcement semantics, out of scope for this routing PR. Co-Authored-By: colleague (Qwen3.6-27B via ask-colleague) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WR1vQvraNdkhuYNpy68fzh
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WR1vQvraNdkhuYNpy68fzh
Contributor
Author
Qodo review triage — addressed in 0.10.1 (
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Make eidetic's files-backend memory store default to the git repo it's run in, instead of always using a single global
$HOME/.eidetic/memory. Resolution is now per-operation, by visibility and cwd:<repo-root>/.eidetic/memory(committed, team-shared)$HOME/.eidetic/memory$HOME/.eidetic/memory(never committed)$HOME/.eidetic/memoryPrecedence: explicit
--data-dir/EIDETIC_DATA_DIR> (public-in-repo → repo store) >$HOME. SettingEIDETIC_DATA_DIRcollapses to a single dir — byte-identical to prior behavior (the regression lock).recallandsweepread and merge across both stores, so a private-scope query returns your own private records plus the public pool.Why
Project work should recall that project's memory by default rather than a cross-project global soup, and team-relevant (public) memory should travel with the repo via git — without breaking the existing Claude↔colleague shared-memory story. Private records stay in the shared
$HOMEstore (so the colleague backend in a throwaway worktree still shares them); public records travel viagit worktree addcheckout of the committed store. Sharing survives on both paths.The no-leak invariant
The security-critical property — a public-scope recall must never return a private record, even though
searchnow reads the home dir that holds private records — is doubly enforced:data_refinery.store.list()filters bycan_serve, and eidetic re-applies its own defensivecan_serveto the unioned candidates.can_servefails closed (a private record serves only on exact same-scope match). Verified by unit tests, an end-to-end test (priv-mergeis absent from a public query's results), an independent review, and a live CLI run.Scope / boundaries
Files-backend default directory only. Unchanged: record schema, ranking modes, scoring/freshness, the
can_serveno-leak semantics, the mongo/neo4j network backends, and theEIDETIC_DATA_DIR/--data-diroverride path. The existing$HOMEstore is left intact (clean break — no migration).How it was built
Worked the idea → spec (
/think) → plan (/spec-to-plan) → 5 tasks across 4 dependency waves, then fanned out to the colleague backend (ask-colleague, Qwen3.6-27B) as the worker, with this agent integrating + TDD-gating each merge (tests green before and after) and reviewing the security-critical crux.ask-colleaguereview of the no-leak invariant (concurred) + a final whole-diff review and live-test.HOME, so private-record tests were polluting the real~/.eideticstore. Fixed (HOME→tmp) and cleaned the pollution.Tests & gates
346 passed, 2 skipped. New:
test_store_resolver.py(17),test_backend_multi_store.py(17),test_store_routing_e2e.py(8). All hermetic (HOME-isolated). black / isort / flake8 / bandit / markdownlint /teken cli doctor --strictall green. Version0.9.3→0.10.0.Follow-ups (tracked, out of scope here)
/remember&/recallwrapper docs are first-party here but fan out byte-verbatim to ~57 org repos — that re-sync is a separate rollout-cli job.migrate storewithout--data-dirupgrades only the$HOMEstore; the repo store needs an explicit--data-dir(documented in--help).*__public.jsonlproduces a git diff per public remember; data-refinery's atomic-per-file format's merge-friendliness across teammates is worth validating.🤖 Generated with Claude Code
https://claude.ai/code/session_01WR1vQvraNdkhuYNpy68fzh