feat(agentcore): idempotent sqlite -> agentcore migration (agentcore migrate)#80
Merged
Conversation
… migrate`)
Add `better-memory agentcore migrate`, a first-class, idempotent migration of
local SQLite reflections + semantic memories into AWS Bedrock AgentCore, so a
user on the default sqlite backend can switch to agentcore without losing
curated knowledge or its usage metadata.
Architecture validated live against real AWS (throwaway memories, torn down):
- AWS silently drops the custom metadata map on client-authored BASE records in
the episodic reflections namespace (schema-gated: episodic declares its schema
under reflectionConfiguration, which governs only extraction), but the JSON
content body round-trips and content-body updates persist. So ALL migrated
reflection state (polarity, status, counters, source_row_id, updated_at) lives
in the content body; `_parse_reflection_record` reads body-first with metadata
fallback (AWS-extracted records unchanged).
- userPreference (semantic) DOES honour a top-level declared memoryRecordSchema,
so semantic state lives in declared metadata (content stays raw text); the
migration idempotency key source_row_id is declared in that schema.
Feature:
- Writer + local ledger (better_memory/storage/agentcore_migrate.py): pure
record builders, deterministic requestIdentifier (bm-{kind}-{sqlite_id}),
canonical content_hash, batching, and an agentcore_migration ledger in the
source db (create/update/skip/retire decisions) for resumable idempotency.
- CLI: --dry-run, --include, --restart, --provision, --db, --project, --region,
--home, --batch-size, --verify; no-rollback partial-failure handling; single
-run advisory lock; exit 0 converged / 2 partial / 1 fatal. Does not touch
settings.json (activation stays a separate, deliberate step).
- Read parity: rating on migrated reflections rewrites the content body (metadata
writes are a no-op on client records); semantic_list returns SemanticMemory
-shaped objects so relevant.py injection works in agentcore mode.
- Client-side reconcile-by-source_row_id rebuilds ledger state after loss.
Tests: hermetic unit (builders/ledger/reader/rating parity) + T2 fake-endpoint
integration modelling the live-proven metadata-drop / body-persist behavior.
Full suite: 1482 passed, 22 skipped, 0 failures.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…pyright build_reflection_record returns dict|None (None for retired/superseded rows); CI pyright checks test files, so active-row call sites that subscript the result need narrowing. Add a _rr() helper / inline asserts. No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
better-memory agentcore migrate— a first-class, idempotent migration of local SQLite reflections + semantic memories (with full usage metadata) into AWS Bedrock AgentCore, so a user on the default sqlite backend can switch to agentcore without losing curated knowledge or itsuseful_count/times_misled/times_overlooked/polarity/status/confidence.Why it was non-trivial — validated live against real AWS
Three live probes (throwaway memories, torn down) overturned the naive design:
reflectionConfiguration, which governs only AWS-extracted records). The JSON content body, however, round-trips and content-body updates persist.memoryRecordSchemathat DOES govern client writes — declared keys are retained, updatable, and listable; undeclared keys drop.Resulting hybrid architecture
source_row_id,evidence_count,updated_at) lives in the content body._parse_reflection_recordreads body-first with metadata fallback, so AWS-extracted records are unchanged and migrated ones resolve from the body.source_row_idis declared in the userPreference schema; content stays raw text.Feature
better_memory/storage/agentcore_migrate.py: pure record builders, deterministicrequestIdentifier(bm-{kind}-{sqlite_id}), canonical content-hash, batching, and anagentcore_migrationledger in the source db (create/update/skip/retire) for resumable idempotency + client-side reconcile-by-source_row_id.--dry-run --include --restart --provision --db --project --region --home --batch-size --verify. No-rollback partial-failure; single-run advisory lock; exit 0/2/1. Does not touchsettings.json(activation stays a separate, deliberate step).semantic_listreturnsSemanticMemory-shaped objects sorelevant.pyinjection works in agentcore mode.Testing
retrieve()on all three buckets + semantic; idempotency held across initial + rerun + reconcile-after-ledger-loss (final record count == seeded count, zero duplicates).Built by agent teams (design scouts → build with per-task spec + code-quality review + fix loop → live validation). Design doc:
docs/superpowers/specs/2026-07-13-sqlite-agentcore-migration-design.md.🤖 Generated with Claude Code