Skip to content

Run the periodic memory sync loops inside the module - #100

Merged
YellowSnnowmann merged 2 commits into
tinyhumansai:mainfrom
YellowSnnowmann:feat/5560-module-periodic-sync-loops
Aug 25, 2026
Merged

Run the periodic memory sync loops inside the module#100
YellowSnnowmann merged 2 commits into
tinyhumansai:mainfrom
YellowSnnowmann:feat/5560-module-periodic-sync-loops

Conversation

@YellowSnnowmann

Copy link
Copy Markdown
Contributor

Depends on nothing; unblocks the last phase of openhuman#5560.

Why

openhuman#5560's final phase deletes the second, in-process memory engine the host boots. Everything that engine was doing then has to run here. The queue worker pool and the ComposioHost / ConfigLoader seams already moved. The periodic composio and workspace-source sync loops had not, and three things stopped them working in module mode — each verified at source, each failing quietly rather than loudly.

The three

Cadence read as manual-only. EngineRuntimeConfig::memory_sync_interval_secs() returned Some(0), which effective_interval_secs defines as manual-only, so both loops skipped every source with no error at all. The value now travels in ModuleConfig and the engine answers it.

The Composio credential branch was never selected. composio_config takes its direct branch only when the mode is direct, otherwise it needs a session token; the module answered ComposioMode::default() and Ok(None), so backend mode failed and direct mode was unreachable. Note this could not be fixed by the ComposioHost seam alone — api_key is consulted inside the branch that was not taken. Mode and entity id now travel; the key still resolves through the seam per call, so no credential is stored module-side.

The module's client was not in the global slot. run_composio_connection_with_caps begins at global::client_if_ready(), which was None here. Calling global::init would have built a second MemoryClient over the same SQLite file — two ingestion workers, duplicate extraction and embedding. New global::bind registers the already-built client into both the slot and the per-workspace cache so all three resolution paths converge on one client.

Three decisions worth review

memory_sync_interval_secs defaults to None, not Some(0). The two candidates fail asymmetrically. Some(0) is manual-only: every source skipped, every tick, no error and nothing in the log — indistinguishable from a sync that ran and found nothing, which is the exact failure this field exists to remove. None means "the user chose nothing", which is true of a host that sent nothing, and lands on the same 24h fallback the host applies. Getting None wrong costs a "Manual only" user one background sync per day until their host learns the field: bounded, visible, and still gated by each source's own enabled toggle. Getting Some(0) wrong is invisible by construction.

Backend-mode Composio cannot run here, and now fails by name. session_token() returns Err(NO_BACKEND_SESSION) instead of Ok(None). The old path reached "backend bearer token is not configured", which reads as "signed out" and points a reader at a sign-in that cannot help; the cause is structural — there is no field for a bearer, and a load-time snapshot could not follow a token the host refreshes mid-session. The Composio loop is also gated off unless the mode resolves to direct, because starting it otherwise would list the user's connections every 20 minutes and fail every due one forever, appending a failed audit row each time. The workspace loop starts in every mode; it never touches Composio.

bind refuses a different client for the same workspace. The cache's usual "racing caller wins" rule is free for init, whose caller only wanted a client. A bind caller is already using the one it passed, so handing back someone else's would neither retire the caller's client nor stop its ingestion worker — two would exist and the API would have said fine. Same workspace and same Arc is idempotent; a different workspace rebinds, as init already does for an active-user switch.

What the host must do

Three optional ModuleConfig keys, all #[serde(default)] so an older host still loads a newer module: memory_sync_interval_secs, composio_mode, composio_entity_id.

And one thing it must stop doing. The cdylib carries its own copy of tinymemory-core, so each loop's OnceLock is a different static from the host's. A host that keeps its start_periodic_sync / start_workspace_periodic_sync calls while loading this module gets two pairs of loops over one store, and the in-process claim guard cannot see across that boundary. The host's call site has to go in the same change that deletes the engine it was calling against.

Known degradation

The module's scheduler-gate stub always answers Normal, so these loops do not honour periodic_pause_reason's "signed out" and "user disabled" pauses. Documented where a reader will find it rather than left to be discovered.

Validation

cargo check --workspace --all-targets clean in both workspaces; cargo test --workspace 1927 passing.

The host starts `sync::composio::start_periodic_sync` and
`start_workspace_periodic_sync` against the second, in-process engine it
also boots. openhuman#5560 deletes that engine, so the loops move in here
beside the queue worker pool. Three things had to be closed first, each of
which failed quietly rather than loudly.

The cadence read as manual-only: `EngineRuntimeConfig` answered the constant
`Some(0)`, which `effective_interval_secs` maps to `None`, so both loops
skipped every source on every tick with nothing logged. `ModuleConfig` now
carries `memory_sync_interval_secs` and the accessor answers it. An absent
field defaults to `None` — "no explicit choice", the 24h fallback — not to
`Some(0)`, because an over-sync is bounded and visible while a no-sync is
invisible by construction, and an older host's payload means whatever the
default says.

The Composio branch was never selected: `composio()` answered an empty mode,
so `composio_config` fell to its backend branch and failed on a session
bearer. `composio_mode` and `composio_entity_id` now cross as routing — the
direct-mode key still comes from `ComposioHost` per call, and there is no
field for a bearer. `session_token` therefore returns a named refusal instead
of `Ok(None)`, which used to surface as "not configured" and send a reader
after a sign-in that cannot help, and the loop is gated to direct mode rather
than started to fail every tick forever.

The module's client was not in the global slot: every pipeline run opens with
`global::client_if_ready()`, and the module builds its store through
`store::factories`, which never touches it. `global::bind` publishes the
already-built client into the slot and the per-workspace cache; `init` would
have built a second `MemoryClient` over the same SQLite file. A different
client for one workspace is refused rather than absorbed.

One degradation is documented, not fixed: the module's scheduler-gate stub
always answers `Normal`, so neither loop honours the "Memory Tree off" and
"signed out" pauses, and a re-enable does not wake them early.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Aug 25, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 233d8337-436c-4c8d-8731-b043da5a13ad

Warning

Your free Security trial is over. An organization admin can activate billing to continue.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@tinysweeper

tinysweeper Bot commented Aug 25, 2026

Copy link
Copy Markdown

How this change flows

3 changed behaviours across 12 relationships. 4 surrounding behaviours are shown (60 graph nodes walked). 42 further behaviours left out to keep the diagram readable.

flowchart LR
  n0["client_if_ready<br/>changed"]:::changed
  n1["client_returns_a_handle_after_explicit_init<br/>changed"]:::changed
  n2["ModuleConfig<br/>changed"]:::changed
  n3["install"]:::impacted
  n4["test_config"]:::impacted
  n5["...ed_and_only_success_counts_toward_the_cap"]:::impacted
  n6["...e_cap_is_reached_through_successful_opens"]:::impacted
  n1 -->|calls| n0
  n1 -->|tests| n0
  n3 -->|uses| n2
  n4 -->|uses| n2
  n5 -->|calls| n3
  n5 -->|tests| n3
  n5 -->|calls| n4
  n5 -->|tests| n4
  n6 -->|calls| n3
  n6 -->|tests| n3
  n6 -->|calls| n4
  n6 -->|tests| n4
  classDef changed fill:#0d4429,stroke:#238636,color:#e6edf3
  classDef impacted fill:#161b22,stroke:#6e7681,color:#c9d1d9
  classDef flagged fill:#5a1e02,stroke:#d93f0b,color:#ffffff
  classDef blocking fill:#67060c,stroke:#f85149,color:#ffffff
Loading

Green: changed behaviour. Grey: surrounding behaviour. Arrows name the call, use, implementation, or test relationship. Orange: has findings. Red: has a finding that blocks the merge.

tinysweeper 0.1.0

@tinysweeper tinysweeper Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tinysweeper found nothing blocking. Approving.

$0.0000 · 0 in / 0 out

@tinysweeper tinysweeper Bot added the priority: p3 Whenever. Cosmetic, a nicety, or a cleanup with no user visible effect. label Aug 25, 2026
@YellowSnnowmann
YellowSnnowmann merged commit 61be7f5 into tinyhumansai:main Aug 25, 2026
27 checks passed
YellowSnnowmann added a commit that referenced this pull request Aug 25, 2026
…crate

The host is down from 285 direct references to the memory engine to 89, and
every one that remains is here: something it asks for that the contract cannot
express. This is measured rather than estimated — deleting both memory crates
from the host's `[dependencies]` and compiling the product profile yields
exactly those 89 errors across 31 distinct symbols. Closing all of them in one
release is the point; a gap left open is another release cycle.

## Two new families

`MemorySync` carries the seven sync surfaces the host still reaches into the
engine for: `run_connection_sync` (the host's five call sites are manual "sync
now" paths — the periodic loops moved into the module in #100, but a user
pressing a button is still the host's to trigger), `source_sync_state`,
`sync_audit_log`, `estimate_sync_cost_usd`, `sync_statuses`,
`raw_archive_coverage` and `rebuild_from_raw_archive`.

`MemorySessions` carries the coding-session pipeline: `coding_session_status`
and `ingest_coding_sessions`.

`MemoryMaintenance::diagnose` replaces the host's reach for the engine's
`async_run_doctor`, answering a structured `Diagnosis` rather than the
maintenance report `doctor` already returns.

`estimate_sync_cost_usd` is a member rather than something the host could
compute, and deliberately: the same constants back `SyncAuditEntry`'s own cost
field, so a host-side copy becomes a second price that drifts from the one audit
rows were written with.

## Types that were never engine-internal

About twenty types reached the host only through `pub use tinymemory_core::…`
shims — source descriptors, the composio provider vocabulary, tree scoring and
summarising, retrieval types, diff ops, a facet class. They are data, so they
move to `tinymemory-bus` and the engine re-exports them from there. Moving
rather than copying is the whole point: a copy is the drifting-duplicate failure
where a field added on one side is a decode failure on the other with nothing to
catch it.

`SyncState` moves with them, but `load` and `save` do not — they do I/O, and the
contract crate stays free of storage engines, HTTP clients and async runtimes.
They become the `PersistedSyncState` extension trait in the engine, which is
where the `SyncStateStore` seam already lives.

`apply_kind_defaults` moves to `tinymemory-sources` so a host can fill a new
entry's caps without linking the engine. Its defaults are the ones the
retroactive Composio caps migration applies, so a change here is a change to
what already-registered sources reconcile against — the tests say so.

## Additive throughout

Every new trait member has a default body returning `Unsupported`, so no
existing driver stops compiling, and every new field is `#[serde(default)]`.
The host and module are separately compiled and separately released; an older
peer must decode a newer payload rather than fail to load.

All new bus methods are registered in the four places that have to agree, in
declaration order, because the drift assertion compares sequences rather than
sets.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority: p3 Whenever. Cosmetic, a nicety, or a cleanup with no user visible effect.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant