fix: make agentcore memory actually work end-to-end (Hive Mind)

README.md

@@ -2,7 +2,7 @@
 
 **better-memory gives your AI coding assistant a memory that grows with your project.** It remembers what worked and what didn't, picks up the preferences and conventions you care about, and when you have to point it back to something it should have used, it records that too — so the same lesson surfaces on its own next time. Every lesson is captured the moment a decision is made and distilled into short, relevance-ranked guidance the assistant pulls up on its own. The result: an assistant that *compounds* — getting sharper on your codebase the more you work together.
 
-It's a memory layer for Claude Code that runs entirely on your machine. Your observations, distilled lessons, and knowledge base live in a local SQLite database — nothing is sent to a cloud service. Even the work of turning raw notes into durable lessons runs inside your own Claude Code session — no separate cloud LLM, no second subscription, no third party seeing your code.
+It's a memory layer for Claude Code that (with the default sqlite backend) runs entirely on your machine. Your observations, distilled lessons, and knowledge base live in a local SQLite database — nothing is sent to a cloud service. Even the work of turning raw notes into durable lessons runs inside your own Claude Code session — no separate cloud LLM, no second subscription, no third party seeing your code. (An optional [AWS-backed backend](website/agentcore-setup.md) exists for teams that want cloud-managed memory.)
 
 ## How it works
 
@@ -34,9 +34,9 @@ better-memory has two storage backends. Pick one:
 | Backend | When to pick | Setup |
 |---|---|---|
 | **`sqlite`** (default) | Single-machine usage; full offline operation; no cloud cost. | None — works out of the box. |
-| **`agentcore`** | Multi-machine syncing; managed extraction by AWS; team-shared memory bucket. | Requires AWS account with Bedrock AgentCore Memory enabled in `eu-west-2`. See [AgentCore setup](website/agentcore-setup.md). |
+| **`agentcore`** | Multi-machine syncing; managed extraction by AWS; team-shared memory bucket. | Requires an AWS account with Bedrock AgentCore Memory available in your chosen region (`init --region` defaults to `eu-west-2`). See [AgentCore setup](website/agentcore-setup.md). |
 
-Switch backends via `BETTER_MEMORY_STORAGE_BACKEND=agentcore` (default: `sqlite`). The MCP server reads the env var at startup and dispatches accordingly. Switching is one-way today — there is no bulk migration tool (deferred; clean start in agentcore mode is the supported path).
+Switching to agentcore is done by `better-memory agentcore init`, which provisions the AWS memories and activates the backend by writing `{"storage_backend": "agentcore"}` to `$BETTER_MEMORY_HOME/settings.json`. The MCP server, hooks, and CLI all resolve the backend the same way: `BETTER_MEMORY_STORAGE_BACKEND` env var if set (always wins), else `settings.json`, else `sqlite`. To revert, remove the `storage_backend` key from `settings.json` or set the env var to `sqlite`. Switching is one-way today — there is no bulk migration tool (deferred; clean start in agentcore mode is the supported path).
 
 `agentcore` mode needs the optional dependency group: `pip install 'better-memory[agentcore]'` (or `uv pip install '.[agentcore]'`). Sqlite-only installs skip boto3 entirely.
 
@@ -87,7 +87,7 @@ Then add to `~/.claude.json` (user-scope MCP — create the file if it doesn't e
 
 And add hooks to `~/.claude/settings.json`:
 
-Four hooks ship. `session_bootstrap` (SessionStart) opens (or reuses) a background episode for the session and injects the project's curated context — project-scoped and general-scope semantic memories plus distilled reflections (`do` / `dont` / `neutral` buckets) — as `additionalContext` so Claude has prior memory available without needing to call any retrieval tool first. Only the top `BETTER_MEMORY_BOOTSTRAP_TOP_N` project-scoped items (default 5) render in full; the rest collapse into a one-line index plus a retrieve affordance (`BETTER_MEMORY_BOOTSTRAP_TOP_N=0` restores the old full-dump behavior). The hook does its work in-process against `memory.db`; if it fails for any reason, a fallback directive is injected instructing Claude to call `mcp__better-memory__memory_session_bootstrap` manually. `contextual_inject` (UserPromptSubmit + PreToolUse) additionally surfaces memories relevant to the current prompt/tool-input mid-session — see [Configuration](website/configuration.md) and [Architecture](website/architecture.md#injection-strategies) for how it scores, floors, and dedups candidates.
+Four hooks ship. `session_bootstrap` (SessionStart) opens (or reuses) a background episode for the session and injects the project's curated context — project-scoped and general-scope semantic memories plus distilled reflections (`do` / `dont` / `neutral` buckets) — as `additionalContext` so Claude has prior memory available without needing to call any retrieval tool first. Only the top `BETTER_MEMORY_BOOTSTRAP_TOP_N` project-scoped items (default 5) render in full; the rest collapse into a one-line index plus a retrieve affordance (`BETTER_MEMORY_BOOTSTRAP_TOP_N=0` restores the old full-dump behavior). The hook does its work in-process against `memory.db` (in agentcore mode it routes through the storage backend instead and never opens the local database); if it fails for any reason, a fallback directive is injected instructing Claude to call `mcp__better-memory__memory_session_bootstrap` manually. `contextual_inject` (UserPromptSubmit + PreToolUse) additionally surfaces memories relevant to the current prompt/tool-input mid-session — see [Configuration](website/configuration.md) and [Architecture](website/architecture.md#injection-strategies) for how it scores, floors, and dedups candidates.
 
 ```json
 {
@@ -167,6 +167,7 @@ One env var roots the runtime filesystem layout:
 | `EMBED_MODEL` | `nomic-embed-text` | Embedding model (must produce 768-dim vectors) |
 | `AUDIT_LOG_RETRIEVED` | `true` | Whether `memory.retrieve` writes per-result audit rows |
 | `BETTER_MEMORY_EMBEDDINGS_BACKEND` | `ollama` | `ollama` (default) — local Ollama at `OLLAMA_HOST`; `sqlite` — pure-SQL trigram-FTS5 fusion, no model downloads and no in-memory state. |
+| `BETTER_MEMORY_STORAGE_BACKEND` | unset | `sqlite` or `agentcore`. Explicit override of the storage backend; when unset, `$BETTER_MEMORY_HOME/settings.json` (written by `better-memory agentcore init`) decides, falling back to `sqlite`. The env var always wins over settings.json. |
 | `BETTER_MEMORY_AUTO_PRUNE` | (unset = `false`) | When set to `1`, the auto-retention runner (which fires on `memory.retrieve`, throttled to once per 24h) ALSO hard-deletes archived observations older than 365 days. **Irreversible.** Default is archive-only (status flip, reversible). Opt in only if you actively want disk space reclaimed. |
 | `BETTER_MEMORY_PROJECT` | unset | Force the project name for all calls in this process. Highest-priority project-resolution signal — overrides both the `.better-memory` file and the git-derived name. Designed for subprocess scoping (e.g. ralph's executor sets it per-iteration so subagent observations land in the PBI's target_repo regardless of the worktree's cwd). Empty/whitespace-only values are treated as unset. |
 | `BETTER_MEMORY_CONTEXT_INJECT_MODE` | `both` | Contextual memory-injection hook trigger: `userprompt`, `pretool`, `both` (default), or `off`. |
@@ -211,7 +212,7 @@ despite the shared name.
 
 ## MCP tools
 
-The server registers 22 tools, grouped below. Full schemas are in [`website/mcp-tools.md`](website/mcp-tools.md) and live in `better_memory/mcp/tools.py`.
+The server registers 22 tools, grouped below. Full schemas are in [`website/mcp-tools.md`](website/mcp-tools.md) and live in `better_memory/mcp/tools.py`. (In agentcore mode the two synthesis tools, the four episode tools, and `memory.run_retention` are hidden from the advertised list — 15 tools; the memory-data tools dispatch to AWS instead of SQLite.)
 
 **Episodic memory** — observations the AI writes during a session.
 

better_memory/cli/agentcore.py

@@ -3,6 +3,7 @@
 from __future__ import annotations
 
 import argparse
+import json
 import sys
 import time
 from pathlib import Path
@@ -28,9 +29,9 @@
 
 _POLL_INTERVAL_S = 5
 # Bumped to 240s vs the 180s the Plan 2 smoke uses: smoke runs solo against
-# a clean account, but `init` runs after the user has just `export`ed env
-# vars and may be hitting a fresh / cold region — small extra headroom is
-# cheap and prevents the user thinking init hung.
+# a clean account, but `init` is often the user's very first call into a
+# fresh / cold region — small extra headroom is cheap and prevents the user
+# thinking init hung.
 _POLL_TIMEOUT_S = 240
 
 
@@ -50,6 +51,15 @@ def add_subparsers(parent: argparse.ArgumentParser) -> None:
         action="store_true",
         help="Overwrite existing agentcore.json",
     )
+    p_init.add_argument(
+        "--no-activate",
+        action="store_true",
+        help=(
+            "Provision the AWS memories and write agentcore.json without "
+            "activating the agentcore backend in settings.json "
+            "(provision-only scripting)"
+        ),
+    )
 
     p_status = subparsers.add_parser(
         "status",
@@ -88,6 +98,59 @@ def _resolve_home(arg_home: str | None) -> Path:
     return Path(os.environ.get("BETTER_MEMORY_HOME", "~/.better-memory")).expanduser()
 
 
+def _write_settings_activation(home: Path) -> Path:
+    """Persist ``{"storage_backend": "agentcore"}`` into ``<home>/settings.json``.
+
+    Merges into an existing JSON object so unrelated keys survive; a missing
+    or corrupt file is replaced wholesale (init IS the remediation path for a
+    broken settings.json, so it must not crash on one). Written atomically —
+    tmp file + replace, the same pattern as ``save_agentcore_config``.
+    """
+    settings_path = home / "settings.json"
+    data: dict[str, Any] = {}
+    try:
+        raw = json.loads(settings_path.read_text(encoding="utf-8"))
+        if isinstance(raw, dict):
+            data = raw
+    except (OSError, json.JSONDecodeError):
+        data = {}
+    data["storage_backend"] = "agentcore"
+    home.mkdir(parents=True, exist_ok=True)
+    tmp = settings_path.with_suffix(".json.tmp")
+    tmp.write_text(json.dumps(data, indent=2, sort_keys=True), encoding="utf-8")
+    tmp.replace(settings_path)
+    return settings_path
+
+
+def _effective_backend(home: Path) -> tuple[str, str]:
+    """Resolve the backend ``home`` would use, plus where the answer came from.
+
+    Mirrors :func:`better_memory.config.resolve_storage_backend` but honours
+    the CLI's ``--home`` override (the config resolver only reads
+    ``$BETTER_MEMORY_HOME``) and reports the source: ``env`` / ``settings`` /
+    ``default``. Where the runtime resolver would RAISE on an invalid env
+    value (killing the server at boot), status keeps going and flags the
+    value instead — a diagnostic tool must surface the misconfiguration, not
+    reproduce the crash. May raise ``ValueError`` on a corrupt settings.json.
+    """
+    import os
+
+    from better_memory.config import (
+        _VALID_STORAGE_BACKENDS,
+        _read_settings_storage_backend,
+    )
+
+    raw = os.environ.get("BETTER_MEMORY_STORAGE_BACKEND")
+    if raw is not None:
+        if raw not in _VALID_STORAGE_BACKENDS:
+            return raw, "env — INVALID value; the MCP server will refuse to start"
+        return raw, "env"
+    from_file = _read_settings_storage_backend(home)
+    if from_file is not None:
+        return from_file, "settings"
+    return "sqlite", "default"
+
+
 def _build_control_client(region: str) -> Any:
     """Build the bedrock-agentcore-control boto3 client. Patched out in tests."""
     import boto3
@@ -304,15 +367,53 @@ def _handle_init(args: argparse.Namespace) -> int:
     )
     save_agentcore_config(cfg, home)
 
+    settings_path = home / "settings.json"
+    activate = not args.no_activate
+    if activate:
+        _write_settings_activation(home)
+
     print()
     print(f"agentcore.json written to {config_path}")
     print(f"  episodic memory_id: {episodic.memory_id}")
     print(f"  semantic memory_id: {semantic.memory_id}")
     print()
+    if activate:
+        print(
+            f"agentcore is now the default backend for {home} (persisted in "
+            f"{settings_path}; the BETTER_MEMORY_STORAGE_BACKEND env var "
+            f"still overrides — unset it or set it to agentcore). To revert: "
+            f"remove 'storage_backend' from settings.json or set the env var "
+            f"to sqlite."
+        )
+    else:
+        print(
+            f"Activation skipped (--no-activate): agentcore.json is written "
+            f"but the backend for {home} is unchanged. To activate later, "
+            f'add {{"storage_backend": "agentcore"}} to {settings_path} or '
+            f"set BETTER_MEMORY_STORAGE_BACKEND=agentcore."
+        )
+    print()
     print("Next steps:")
-    print("  1. Export BETTER_MEMORY_STORAGE_BACKEND=agentcore")
-    print("  2. Restart your MCP server (or Claude Code session)")
-    print("  3. Run `better-memory agentcore smoke` to verify the round-trip")
+    if activate:
+        print(
+            "  1. Restart Claude Code (or your MCP server) so it picks up the new backend"
+        )
+        print(
+            "  2. Run `better-memory agentcore status` to confirm the effective backend"
+        )
+        print("     and that both memories are ACTIVE")
+        print("  3. Run `better-memory agentcore smoke` to verify the AWS round-trip")
+        print(
+            "     (smoke validates AWS credentials and wire access, not MCP registration)"
+        )
+    else:
+        print("  1. Activate when ready (see above) — no backend change was made yet")
+        print("  2. Run `better-memory agentcore status` to confirm both memories are")
+        print("     ACTIVE (the effective backend is unchanged until you activate)")
+        print("  3. Run `better-memory agentcore smoke` to verify the AWS round-trip")
+        print(
+            "     (smoke validates AWS credentials and wire access, not MCP registration)"
+        )
     return 0
 
 
@@ -327,6 +428,17 @@ def _handle_status(args: argparse.Namespace) -> int:
         )
         return 1
 
+    try:
+        backend, source = _effective_backend(home)
+        print(f"effective backend: {backend} (source: {source})")
+    except ValueError as exc:
+        # A corrupt settings.json must not take `status` down — it is the
+        # diagnostic tool the user reaches for. Warn and keep reporting.
+        print(
+            f"WARN: could not resolve the effective backend: {exc}",
+            file=sys.stderr,
+        )
+
     region = args.region or cfg.region
     control = _build_control_client(region)
 
@@ -423,13 +535,20 @@ def _handle_smoke(args: argparse.Namespace) -> int:
         print(f"   ok ({len(events)} events)")
 
         print(">> 4. BatchCreateMemoryRecords — semantic write")
-        record_id = f"smoke-rec-{int(time.time())}"
+        # Live-verified request shape (aws_record_dialect.md §1): per-record
+        # required keys are requestIdentifier + namespaces + content +
+        # timestamp. memoryRecordId is NOT a valid input key (botocore
+        # ParamValidationError) — the SERVER mints the durable mem-<uuid4>
+        # id, returned in successfulRecords[0].memoryRecordId;
+        # requestIdentifier is correlation-only.
+        request_identifier = f"smoke-rec-{int(time.time())}"
         create_resp = data.batch_create_memory_records(
             memoryId=cfg.semantic.memory_id,
             records=[{
-                "memoryRecordId": record_id,
+                "requestIdentifier": request_identifier,
                 "namespaces": [f"projects/{actor_id}/semantic/"],
                 "content": {"text": "smoke test semantic record"},
+                "timestamp": datetime.now(UTC),
                 "metadata": {
                     "useful_count": {"numberValue": 0},
                     "status": {"stringValue": "active"},
@@ -447,16 +566,31 @@ def _handle_smoke(args: argparse.Namespace) -> int:
         real_id = successful[0]["memoryRecordId"]
         print(f"   ok (id={real_id})")
 
-        print(">> 5. ListMemoryRecords — readback")
-        list_resp = data.list_memory_records(
-            memoryId=cfg.semantic.memory_id,
-            namespace=f"projects/{actor_id}/semantic/",
-            maxResults=10,
-        )
-        summaries = list_resp.get("memoryRecordSummaries", [])
-        if not summaries:
-            raise RuntimeError("list_memory_records returned no summaries")
-        print(f"   ok ({len(summaries)} records)")
+        print(">> 5. GetMemoryRecord — readback")
+        # get_memory_record is read-your-write (~1s); list_memory_records
+        # shares an index with ~60s lag and would force the smoke to poll
+        # for a minute (aws_record_dialect.md §3). Retry a few times to
+        # cover the sub-second window all the same.
+        record = None
+        for attempt in range(1, 6):
+            try:
+                record = data.get_memory_record(
+                    memoryId=cfg.semantic.memory_id,
+                    memoryRecordId=real_id,
+                )["memoryRecord"]
+                break
+            except Exception as get_exc:  # noqa: BLE001 — retried, re-raised below
+                code = getattr(get_exc, "response", {}).get(
+                    "Error", {}
+                ).get("Code", "")
+                if code != "ResourceNotFoundException" or attempt == 5:
+                    raise
+                time.sleep(2)
+        if record is None or record.get("memoryRecordId") != real_id:
+            raise RuntimeError(
+                f"get_memory_record readback mismatch: {record!r}"
+            )
+        print("   ok (readback id matches)")
 
         print(">> 6. BatchDeleteMemoryRecords — cleanup")
         del_resp = data.batch_delete_memory_records(