Run the periodic memory sync loops inside the module - #100
Conversation
The host starts `sync::composio::start_periodic_sync` and `start_workspace_periodic_sync` against the second, in-process engine it also boots. openhuman#5560 deletes that engine, so the loops move in here beside the queue worker pool. Three things had to be closed first, each of which failed quietly rather than loudly. The cadence read as manual-only: `EngineRuntimeConfig` answered the constant `Some(0)`, which `effective_interval_secs` maps to `None`, so both loops skipped every source on every tick with nothing logged. `ModuleConfig` now carries `memory_sync_interval_secs` and the accessor answers it. An absent field defaults to `None` — "no explicit choice", the 24h fallback — not to `Some(0)`, because an over-sync is bounded and visible while a no-sync is invisible by construction, and an older host's payload means whatever the default says. The Composio branch was never selected: `composio()` answered an empty mode, so `composio_config` fell to its backend branch and failed on a session bearer. `composio_mode` and `composio_entity_id` now cross as routing — the direct-mode key still comes from `ComposioHost` per call, and there is no field for a bearer. `session_token` therefore returns a named refusal instead of `Ok(None)`, which used to surface as "not configured" and send a reader after a sign-in that cannot help, and the loop is gated to direct mode rather than started to fail every tick forever. The module's client was not in the global slot: every pipeline run opens with `global::client_if_ready()`, and the module builds its store through `store::factories`, which never touches it. `global::bind` publishes the already-built client into the slot and the per-workspace cache; `init` would have built a second `MemoryClient` over the same SQLite file. A different client for one workspace is refused rather than absorbed. One degradation is documented, not fixed: the module's scheduler-gate stub always answers `Normal`, so neither loop honours the "Memory Tree off" and "signed out" pauses, and a re-enable does not wake them early. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: Warning Your free Security trial is over. An organization admin can activate billing to continue. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
How this change flows3 changed behaviours across 12 relationships. 4 surrounding behaviours are shown (60 graph nodes walked). 42 further behaviours left out to keep the diagram readable. flowchart LR
n0["client_if_ready<br/>changed"]:::changed
n1["client_returns_a_handle_after_explicit_init<br/>changed"]:::changed
n2["ModuleConfig<br/>changed"]:::changed
n3["install"]:::impacted
n4["test_config"]:::impacted
n5["...ed_and_only_success_counts_toward_the_cap"]:::impacted
n6["...e_cap_is_reached_through_successful_opens"]:::impacted
n1 -->|calls| n0
n1 -->|tests| n0
n3 -->|uses| n2
n4 -->|uses| n2
n5 -->|calls| n3
n5 -->|tests| n3
n5 -->|calls| n4
n5 -->|tests| n4
n6 -->|calls| n3
n6 -->|tests| n3
n6 -->|calls| n4
n6 -->|tests| n4
classDef changed fill:#0d4429,stroke:#238636,color:#e6edf3
classDef impacted fill:#161b22,stroke:#6e7681,color:#c9d1d9
classDef flagged fill:#5a1e02,stroke:#d93f0b,color:#ffffff
classDef blocking fill:#67060c,stroke:#f85149,color:#ffffff
Green: changed behaviour. Grey: surrounding behaviour. Arrows name the call, use, implementation, or test relationship. Orange: has findings. Red: has a finding that blocks the merge. |
…crate The host is down from 285 direct references to the memory engine to 89, and every one that remains is here: something it asks for that the contract cannot express. This is measured rather than estimated — deleting both memory crates from the host's `[dependencies]` and compiling the product profile yields exactly those 89 errors across 31 distinct symbols. Closing all of them in one release is the point; a gap left open is another release cycle. ## Two new families `MemorySync` carries the seven sync surfaces the host still reaches into the engine for: `run_connection_sync` (the host's five call sites are manual "sync now" paths — the periodic loops moved into the module in #100, but a user pressing a button is still the host's to trigger), `source_sync_state`, `sync_audit_log`, `estimate_sync_cost_usd`, `sync_statuses`, `raw_archive_coverage` and `rebuild_from_raw_archive`. `MemorySessions` carries the coding-session pipeline: `coding_session_status` and `ingest_coding_sessions`. `MemoryMaintenance::diagnose` replaces the host's reach for the engine's `async_run_doctor`, answering a structured `Diagnosis` rather than the maintenance report `doctor` already returns. `estimate_sync_cost_usd` is a member rather than something the host could compute, and deliberately: the same constants back `SyncAuditEntry`'s own cost field, so a host-side copy becomes a second price that drifts from the one audit rows were written with. ## Types that were never engine-internal About twenty types reached the host only through `pub use tinymemory_core::…` shims — source descriptors, the composio provider vocabulary, tree scoring and summarising, retrieval types, diff ops, a facet class. They are data, so they move to `tinymemory-bus` and the engine re-exports them from there. Moving rather than copying is the whole point: a copy is the drifting-duplicate failure where a field added on one side is a decode failure on the other with nothing to catch it. `SyncState` moves with them, but `load` and `save` do not — they do I/O, and the contract crate stays free of storage engines, HTTP clients and async runtimes. They become the `PersistedSyncState` extension trait in the engine, which is where the `SyncStateStore` seam already lives. `apply_kind_defaults` moves to `tinymemory-sources` so a host can fill a new entry's caps without linking the engine. Its defaults are the ones the retroactive Composio caps migration applies, so a change here is a change to what already-registered sources reconcile against — the tests say so. ## Additive throughout Every new trait member has a default body returning `Unsupported`, so no existing driver stops compiling, and every new field is `#[serde(default)]`. The host and module are separately compiled and separately released; an older peer must decode a newer payload rather than fail to load. All new bus methods are registered in the four places that have to agree, in declaration order, because the drift assertion compares sequences rather than sets.
Depends on nothing; unblocks the last phase of openhuman#5560.
Why
openhuman#5560's final phase deletes the second, in-process memory engine the host boots. Everything that engine was doing then has to run here. The queue worker pool and the
ComposioHost/ConfigLoaderseams already moved. The periodic composio and workspace-source sync loops had not, and three things stopped them working in module mode — each verified at source, each failing quietly rather than loudly.The three
Cadence read as manual-only.
EngineRuntimeConfig::memory_sync_interval_secs()returnedSome(0), whicheffective_interval_secsdefines as manual-only, so both loops skipped every source with no error at all. The value now travels inModuleConfigand the engine answers it.The Composio credential branch was never selected.
composio_configtakes its direct branch only when the mode isdirect, otherwise it needs a session token; the module answeredComposioMode::default()andOk(None), so backend mode failed and direct mode was unreachable. Note this could not be fixed by theComposioHostseam alone —api_keyis consulted inside the branch that was not taken. Mode and entity id now travel; the key still resolves through the seam per call, so no credential is stored module-side.The module's client was not in the global slot.
run_composio_connection_with_capsbegins atglobal::client_if_ready(), which wasNonehere. Callingglobal::initwould have built a secondMemoryClientover the same SQLite file — two ingestion workers, duplicate extraction and embedding. Newglobal::bindregisters the already-built client into both the slot and the per-workspace cache so all three resolution paths converge on one client.Three decisions worth review
memory_sync_interval_secsdefaults toNone, notSome(0). The two candidates fail asymmetrically.Some(0)is manual-only: every source skipped, every tick, no error and nothing in the log — indistinguishable from a sync that ran and found nothing, which is the exact failure this field exists to remove.Nonemeans "the user chose nothing", which is true of a host that sent nothing, and lands on the same 24h fallback the host applies. GettingNonewrong costs a "Manual only" user one background sync per day until their host learns the field: bounded, visible, and still gated by each source's ownenabledtoggle. GettingSome(0)wrong is invisible by construction.Backend-mode Composio cannot run here, and now fails by name.
session_token()returnsErr(NO_BACKEND_SESSION)instead ofOk(None). The old path reached "backend bearer token is not configured", which reads as "signed out" and points a reader at a sign-in that cannot help; the cause is structural — there is no field for a bearer, and a load-time snapshot could not follow a token the host refreshes mid-session. The Composio loop is also gated off unless the mode resolves to direct, because starting it otherwise would list the user's connections every 20 minutes and fail every due one forever, appending a failed audit row each time. The workspace loop starts in every mode; it never touches Composio.bindrefuses a different client for the same workspace. The cache's usual "racing caller wins" rule is free forinit, whose caller only wanted a client. Abindcaller is already using the one it passed, so handing back someone else's would neither retire the caller's client nor stop its ingestion worker — two would exist and the API would have said fine. Same workspace and sameArcis idempotent; a different workspace rebinds, asinitalready does for an active-user switch.What the host must do
Three optional
ModuleConfigkeys, all#[serde(default)]so an older host still loads a newer module:memory_sync_interval_secs,composio_mode,composio_entity_id.And one thing it must stop doing. The
cdylibcarries its own copy oftinymemory-core, so each loop'sOnceLockis a different static from the host's. A host that keeps itsstart_periodic_sync/start_workspace_periodic_synccalls while loading this module gets two pairs of loops over one store, and the in-process claim guard cannot see across that boundary. The host's call site has to go in the same change that deletes the engine it was calling against.Known degradation
The module's scheduler-gate stub always answers
Normal, so these loops do not honourperiodic_pause_reason's "signed out" and "user disabled" pauses. Documented where a reader will find it rather than left to be discovered.Validation
cargo check --workspace --all-targetsclean in both workspaces;cargo test --workspace1927 passing.