From 727b6b800faf9420b053e2d6f299bc5509da69af Mon Sep 17 00:00:00 2001 From: Shanu Date: Tue, 25 Aug 2026 16:24:48 +0530 Subject: [PATCH 1/2] Close the last contract gaps openhuman#5560 needs to drop the engine crate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The host is down from 285 direct references to the memory engine to 89, and every one that remains is here: something it asks for that the contract cannot express. This is measured rather than estimated — deleting both memory crates from the host's `[dependencies]` and compiling the product profile yields exactly those 89 errors across 31 distinct symbols. Closing all of them in one release is the point; a gap left open is another release cycle. ## Two new families `MemorySync` carries the seven sync surfaces the host still reaches into the engine for: `run_connection_sync` (the host's five call sites are manual "sync now" paths — the periodic loops moved into the module in #100, but a user pressing a button is still the host's to trigger), `source_sync_state`, `sync_audit_log`, `estimate_sync_cost_usd`, `sync_statuses`, `raw_archive_coverage` and `rebuild_from_raw_archive`. `MemorySessions` carries the coding-session pipeline: `coding_session_status` and `ingest_coding_sessions`. `MemoryMaintenance::diagnose` replaces the host's reach for the engine's `async_run_doctor`, answering a structured `Diagnosis` rather than the maintenance report `doctor` already returns. `estimate_sync_cost_usd` is a member rather than something the host could compute, and deliberately: the same constants back `SyncAuditEntry`'s own cost field, so a host-side copy becomes a second price that drifts from the one audit rows were written with. ## Types that were never engine-internal About twenty types reached the host only through `pub use tinymemory_core::…` shims — source descriptors, the composio provider vocabulary, tree scoring and summarising, retrieval types, diff ops, a facet class. They are data, so they move to `tinymemory-bus` and the engine re-exports them from there. Moving rather than copying is the whole point: a copy is the drifting-duplicate failure where a field added on one side is a decode failure on the other with nothing to catch it. `SyncState` moves with them, but `load` and `save` do not — they do I/O, and the contract crate stays free of storage engines, HTTP clients and async runtimes. They become the `PersistedSyncState` extension trait in the engine, which is where the `SyncStateStore` seam already lives. `apply_kind_defaults` moves to `tinymemory-sources` so a host can fill a new entry's caps without linking the engine. Its defaults are the ones the retroactive Composio caps migration applies, so a change here is a change to what already-registered sources reconcile against — the tests say so. ## Additive throughout Every new trait member has a default body returning `Unsupported`, so no existing driver stops compiling, and every new field is `#[serde(default)]`. The host and module are separately compiled and separately released; an older peer must decode a newer payload rather than fail to load. All new bus methods are registered in the four places that have to agree, in declaration order, because the drift assertion compares sequences rather than sets. --- README.md | 12 +- clippy.toml | 2 + crates/tinymemory-api/src/lib.rs | 21 +- crates/tinymemory-api/src/null.rs | 98 +++- crates/tinymemory-api/src/null_tests.rs | 56 ++- .../src/provider/audit_tests.rs | 9 +- crates/tinymemory-api/src/provider/content.rs | 47 ++ crates/tinymemory-api/src/provider/driver.rs | 25 +- crates/tinymemory-api/src/provider/mod.rs | 47 +- crates/tinymemory-api/src/provider/records.rs | 54 ++ .../tinymemory-api/src/provider/sessions.rs | 102 ++++ crates/tinymemory-api/src/provider/sync.rs | 229 +++++++++ crates/tinymemory-bus/README.md | 6 +- crates/tinymemory-bus/src/capabilities.rs | 27 +- .../tinymemory-bus/src/capabilities_tests.rs | 22 +- crates/tinymemory-bus/src/composio/mod.rs | 71 +++ crates/tinymemory-bus/src/composio/profile.rs | 299 +++++++++++ .../src/composio/profile_tests.rs | 239 +++++++++ crates/tinymemory-bus/src/composio/runs.rs | 112 +++++ .../tinymemory-bus/src/composio/runs_tests.rs | 113 +++++ crates/tinymemory-bus/src/composio/scopes.rs | 246 +++++++++ .../src/composio/scopes_tests.rs | 178 +++++++ crates/tinymemory-bus/src/composio/state.rs | 275 ++++++++++ .../src/composio/state_tests.rs | 194 +++++++ crates/tinymemory-bus/src/composio/tasks.rs | 206 ++++++++ .../src/composio/tasks_tests.rs | 139 +++++ crates/tinymemory-bus/src/learning.rs | 144 ++++++ crates/tinymemory-bus/src/learning_tests.rs | 111 ++++ crates/tinymemory-bus/src/lib.rs | 13 +- crates/tinymemory-bus/src/names.rs | 46 +- crates/tinymemory-bus/src/namespace.rs | 2 +- .../tinymemory-bus/src/provider/diagnosis.rs | 189 +++++++ .../src/provider/diagnosis_tests.rs | 107 ++++ crates/tinymemory-bus/src/provider/mod.rs | 3 + .../tinymemory-bus/src/provider/sessions.rs | 157 ++++++ .../src/provider/sessions_tests.rs | 63 +++ crates/tinymemory-bus/src/provider/sync.rs | 312 ++++++++++++ .../tinymemory-bus/src/provider/sync_tests.rs | 147 ++++++ crates/tinymemory-core/src/engine/sync.rs | 4 + .../tinymemory-core/src/learning_candidate.rs | 137 ++--- .../tinymemory-core/src/sources/registry.rs | 43 +- .../src/sync/composio/providers/mod.rs | 52 +- .../src/sync/composio/providers/profile.rs | 217 ++------ .../sync/composio/providers/profile_tests.rs | 6 +- .../src/sync/composio/providers/sync_state.rs | 262 ++++------ .../src/sync/composio/providers/tool_scope.rs | 124 +---- .../src/sync/composio/providers/types.rs | 306 ++--------- .../sync/composio/providers/user_scopes.rs | 55 +- .../composio/providers/user_scopes_tests.rs | 3 + .../sync/pipelines/composio/orchestrator.rs | 2 +- .../composio/providers/provider_tests.rs | 2 +- .../pipelines/composio/providers/slack.rs | 2 +- .../composio_gmail_non_tinycortex_e2e.rs | 7 +- crates/tinymemory-module/src/lib.rs | 27 + crates/tinymemory-module/src/service/mod.rs | 195 ++++++++ crates/tinymemory-module/src/service/test.rs | 41 ++ crates/tinymemory-module/tests/module_e2e.rs | 15 +- crates/tinymemory-sources/src/lib.rs | 4 +- crates/tinymemory-sources/src/registry.rs | 37 ++ .../tinymemory-sources/src/registry_tests.rs | 73 +++ .../src/conformance_test.rs | 2 +- .../tinymemory-tinycortex/src/engine/mod.rs | 473 +++++++++++++++++- .../tinymemory-tinycortex/src/engine/test.rs | 94 +++- .../tests/full_provider_conformance.rs | 2 +- crates/tinymemory/examples/tinycortex.rs | 2 +- 65 files changed, 5282 insertions(+), 1028 deletions(-) create mode 100644 crates/tinymemory-api/src/provider/sessions.rs create mode 100644 crates/tinymemory-api/src/provider/sync.rs create mode 100644 crates/tinymemory-bus/src/composio/mod.rs create mode 100644 crates/tinymemory-bus/src/composio/profile.rs create mode 100644 crates/tinymemory-bus/src/composio/profile_tests.rs create mode 100644 crates/tinymemory-bus/src/composio/runs.rs create mode 100644 crates/tinymemory-bus/src/composio/runs_tests.rs create mode 100644 crates/tinymemory-bus/src/composio/scopes.rs create mode 100644 crates/tinymemory-bus/src/composio/scopes_tests.rs create mode 100644 crates/tinymemory-bus/src/composio/state.rs create mode 100644 crates/tinymemory-bus/src/composio/state_tests.rs create mode 100644 crates/tinymemory-bus/src/composio/tasks.rs create mode 100644 crates/tinymemory-bus/src/composio/tasks_tests.rs create mode 100644 crates/tinymemory-bus/src/learning.rs create mode 100644 crates/tinymemory-bus/src/learning_tests.rs create mode 100644 crates/tinymemory-bus/src/provider/diagnosis.rs create mode 100644 crates/tinymemory-bus/src/provider/diagnosis_tests.rs create mode 100644 crates/tinymemory-bus/src/provider/sessions.rs create mode 100644 crates/tinymemory-bus/src/provider/sessions_tests.rs create mode 100644 crates/tinymemory-bus/src/provider/sync.rs create mode 100644 crates/tinymemory-bus/src/provider/sync_tests.rs diff --git a/README.md b/README.md index 31b91f80..16ddea77 100644 --- a/README.md +++ b/README.md @@ -97,7 +97,7 @@ is deliberately outside `full`: "give me the whole workspace" is not the same request as "give me the test doubles". This table says which crate each feature brings in. For what each *engine* -feature actually serves — driver class, and how many of the eighteen capability +feature actually serves — driver class, and how many of the twenty capability families answer — see the engine table under [Using from your project](#using-from-your-project). @@ -173,7 +173,7 @@ let provider = Arc::new(provider(Arc::new(InMemoryMemoryStore::new()))); ``` That is a complete embedded setup for the mandatory three families. The full -eighteen-family engine (`TinycortexProvider`) additionally needs the host +twenty-family engine (`TinycortexProvider`) additionally needs the host seams (`EmbeddingHost` et al.) installed — see `crates/tinymemory-tinycortex/tests/full_provider_conformance.rs` for the minimal working wiring. @@ -200,10 +200,10 @@ for assistant-memory workloads; wrong for high-volume keyed storage. ## The contract `MemoryProvider` is an object-safe trait with **three mandatory** capability -families and **fifteen optional** ones. The mandatory three are supertraits, so a -driver missing any of them cannot be constructed; the optional fifteen are reached -through `as_ingest()` / `as_tree()` / … accessors that default to `None`, so a -minimal driver implements what it supports and inherits correct absence for +families and **seventeen optional** ones. The mandatory three are supertraits, so +a driver missing any of them cannot be constructed; the optional seventeen are +reached through `as_ingest()` / `as_tree()` / … accessors that default to `None`, +so a minimal driver implements what it supports and inherits correct absence for everything else. A driver's advertised set and its reachable accessors must agree. diff --git a/clippy.toml b/clippy.toml index 8485eecb..5fa90648 100644 --- a/clippy.toml +++ b/clippy.toml @@ -13,4 +13,6 @@ doc-valid-idents = [ # Product names in `chunks::SourceKind`'s prose, not Rust items. "WhatsApp", "FastMail", + # Toolkit names in `composio`'s prose, likewise. + "ClickUp", ] diff --git a/crates/tinymemory-api/src/lib.rs b/crates/tinymemory-api/src/lib.rs index df78204c..9eb1e1da 100644 --- a/crates/tinymemory-api/src/lib.rs +++ b/crates/tinymemory-api/src/lib.rs @@ -55,13 +55,26 @@ //! ## Module map //! //! - [`types`]: pure data contracts (entries, hits, taint, namespaces). +//! - [`evidence`]: [`evidence::EvidenceRef`], the pointer a learned fact keeps +//! back to what it was learned from. Also re-exported as +//! [`host::EvidenceRef`], which is where the memory store's callers name it. +//! - [`learning`]: the learning-candidate taxonomy +//! ([`learning::FacetClass`], [`learning::CueFamily`], +//! [`learning::LearningCandidate`]) — what a producer asserts about the user +//! and how strongly, with the buffer that queues it left in the engine crate. +//! - [`composio`]: the connector-sync vocabulary — [`composio::SyncOutcome`], +//! [`composio::NormalizedTask`], [`composio::SyncState`], +//! [`composio::ToolScope`] and friends. **Not** [`host::composio`], which is +//! the *client* seam: connections, execute responses and the capability +//! matrix a host serves to the memory layer. This one is what a provider run +//! produces and remembers; that one is how it reaches Composio at all. //! - [`recall`]: the borrowed [`recall::RecallOpts`] and owned, serde-derived //! [`recall::OwnedRecallOpts`] recall filters (both re-exported from //! [`types`]). -//! - [`capabilities`]: the eighteen [`capabilities::Capability`] families and +//! - [`capabilities`]: the twenty [`capabilities::Capability`] families and //! the [`capabilities::Capabilities`] set negotiated at bind time. //! - [`provider`]: the driver contract — [`provider::MemoryProvider`] plus the -//! eighteen capability family traits and the value types they need. +//! twenty capability family traits and the value types they need. //! - [`null`]: [`null::NullMemoryProvider`], the reference driver a //! compiled-out or unconfigured memory subsystem binds to. //! - [`health`]: [`health::MemoryHealth`], the liveness state a driver reports. @@ -108,8 +121,8 @@ pub mod sync_events; // point: a second definition would need a conversion at the module seam that // nothing type-checks. pub use tinymemory_bus::{ - capabilities, chunks, error, goals, graph, health, namespace, recall, tool_memory, tree, types, - version, wire, + capabilities, chunks, composio, error, evidence, goals, graph, health, learning, namespace, + recall, tool_memory, tree, types, version, wire, }; /// The mandatory-family composition: wrap any [`traits::Memory`] backend as a /// complete [`provider::MemoryProvider`]. diff --git a/crates/tinymemory-api/src/null.rs b/crates/tinymemory-api/src/null.rs index 47d2bc4c..9e737c4f 100644 --- a/crates/tinymemory-api/src/null.rs +++ b/crates/tinymemory-api/src/null.rs @@ -10,7 +10,7 @@ //! `stub.rs` files with one generic answer. //! //! It is also the fixture the capability-degradation tests bind: with it in the -//! slot, the fifteen optional families are unadvertised, so their RPC methods are +//! slot, the optional families are unadvertised, so their RPC methods are //! unregistered and their agent tools are absent — and the core still boots. //! //! And it is the existence proof for the mandatory set: if @@ -32,9 +32,9 @@ //! driver that failed to bind — **that** case falls back to the embedded //! default, never to this. Do not wire it as a general-purpose failure mode. //! -//! ## Why it implements all eighteen families but advertises three +//! ## Why it implements the optional families but advertises three //! -//! The fifteen optional families are implemented and every method returns +//! The optional families are implemented and every method returns //! [`crate::error::MemoryError::Unsupported`] naming its family, but the //! `as_*` accessors return `None` and //! [`crate::provider::MemoryProvider::capabilities`] lists only the mandatory @@ -60,13 +60,15 @@ use crate::provider::types::{ IngestOutcome, MaintenanceReport, ResetOutcome, SnapshotRef, SourceItem, SourceScope, }; use crate::provider::{ - AddressBookSeedOutcome, ChunkDetail, ChunkEmbedding, ChunkQuery, CoverWindowQuery, EntityMatch, - FacetType, FastRetrieveQuery, MemoryChunks, MemoryCore, MemoryDiff, MemoryDocuments, + AddressBookSeedOutcome, ChunkDetail, ChunkEmbedding, ChunkQuery, CodingSessionIngestReport, + CodingSessionIngestRequest, CodingSessionSource, CoverWindowQuery, EntityMatch, FacetType, + FastRetrieveQuery, MemoryChunks, MemoryCodingSessions, MemoryCore, MemoryDiff, MemoryDocuments, MemoryEntities, MemoryGoals, MemoryGraph, MemoryIngest, MemoryMaintenance, MemoryPeople, MemoryPortability, MemoryProfile, MemoryProvider, MemoryRecall, MemoryRetrieval, - MemorySourceSink, MemoryToolMemory, MemoryTree, PersonHandle, PersonInteraction, PersonRecord, - PersonScore, ProfileFacet, RankedPerson, ResolvedPerson, RetrievalHit, RetrievalResponse, - SourceRetrievalQuery, UserState, + MemorySourceSink, MemorySourceSync, MemoryToolMemory, MemoryTree, PersonHandle, + PersonInteraction, PersonRecord, PersonScore, ProfileFacet, RankedPerson, RawArchiveCoverage, + RawRebuildOutcome, ResolvedPerson, RetrievalHit, RetrievalResponse, SourceRetrievalQuery, + SourceSyncState, SourceSyncStatus, SyncAuditEntry, SyncRunOutcome, UserState, }; use crate::recall::OwnedRecallOpts; use crate::tool_memory::ToolMemoryRule; @@ -106,7 +108,7 @@ impl MemoryProvider for NullMemoryProvider { NULL_DRIVER_ID } - /// Exactly the mandatory three. The fifteen optional families are implemented + /// Exactly the mandatory three. The optional families are implemented /// below but deliberately not advertised, so they stay unreachable through /// the trait object. fn capabilities(&self) -> Capabilities { @@ -705,6 +707,84 @@ impl MemoryProfile for NullMemoryProvider { } } +#[async_trait] +impl MemorySourceSync for NullMemoryProvider { + async fn run_connection_sync( + &self, + _toolkit: &str, + _connection_id: &str, + ) -> Result { + unsupported(Capability::SourceSync) + } + + async fn source_sync_state( + &self, + _toolkit: &str, + _connection_id: &str, + ) -> Result, MemoryError> { + // Not `Ok(None)`, which the trait defines as "this connection has never + // synced". This driver cannot sync at all, and answering "never synced" + // would put a connection with a plausible empty state in front of a + // caller that would then offer to sync it. + unsupported(Capability::SourceSync) + } + + async fn sync_audit_log( + &self, + _limit: Option, + ) -> Result, MemoryError> { + unsupported(Capability::SourceSync) + } + + async fn estimate_sync_cost_usd( + &self, + _input_tokens: u64, + _output_tokens: u64, + ) -> Result { + // The trait lets a driver whose sync is free answer `0.0`. This one has + // no sync to price, and quoting a free one would be a price rather than + // an absence — the same distinction the state read above draws. + unsupported(Capability::SourceSync) + } + + async fn sync_statuses(&self) -> Result, MemoryError> { + unsupported(Capability::SourceSync) + } + + async fn raw_archive_coverage( + &self, + _tree_scope: &str, + _archive_source_id: &str, + ) -> Result { + unsupported(Capability::SourceSync) + } + + async fn rebuild_from_raw_archive( + &self, + _tree_scope: &str, + _archive_source_id: &str, + ) -> Result { + unsupported(Capability::SourceSync) + } +} + +#[async_trait] +impl MemoryCodingSessions for NullMemoryProvider { + async fn coding_session_status(&self) -> Result, MemoryError> { + // Not an empty list. The trait defines one row per agent the driver + // knows about, so an empty answer is "I looked and found no agents + // installed" — which this driver did not do. + unsupported(Capability::CodingSessions) + } + + async fn ingest_coding_sessions( + &self, + _request: CodingSessionIngestRequest, + ) -> Result { + unsupported(Capability::CodingSessions) + } +} + #[cfg(test)] #[path = "null_tests.rs"] mod tests; diff --git a/crates/tinymemory-api/src/null_tests.rs b/crates/tinymemory-api/src/null_tests.rs index 36d2621a..d473a0ac 100644 --- a/crates/tinymemory-api/src/null_tests.rs +++ b/crates/tinymemory-api/src/null_tests.rs @@ -168,6 +168,8 @@ fn every_unadvertised_family_is_unreachable_through_the_trait_object() { assert!(provider.as_tool_memory().is_none()); assert!(provider.as_sources().is_none()); assert!(provider.as_maintenance().is_none()); + assert!(provider.as_source_sync().is_none()); + assert!(provider.as_coding_sessions().is_none()); } #[test] @@ -231,8 +233,8 @@ fn every_optional_method_fails_with_its_advertised_family_name() { use crate::goals::GoalsDoc; use crate::provider::types::{IngestItem, SourceItem}; use crate::provider::{ - ChunkQuery, CoverWindowQuery, FacetType, FastRetrieveQuery, PersonHandle, - PersonInteraction, SourceRetrievalQuery, UserState, + ChunkQuery, CodingSessionIngestRequest, CoverWindowQuery, FacetType, FastRetrieveQuery, + PersonHandle, PersonInteraction, SourceRetrievalQuery, UserState, }; use crate::tool_memory::{ToolMemoryPriority, ToolMemoryRule, ToolMemorySource}; use crate::tree::IngestRequest; @@ -507,6 +509,56 @@ fn every_optional_method_fails_with_its_advertised_family_name() { ); assert_unsupported(block_on(driver.drop_facets_below(0.5)), Capability::Profile); assert!(!block_on(driver.workflow_identity_matches("*", "value"))); + + assert_unsupported( + block_on(driver.run_connection_sync("gmail", "conn-1")), + Capability::SourceSync, + ); + assert_unsupported( + block_on(driver.source_sync_state("gmail", "conn-1")), + Capability::SourceSync, + ); + assert_unsupported( + block_on(driver.sync_audit_log(None)), + Capability::SourceSync, + ); + assert_unsupported( + block_on(driver.estimate_sync_cost_usd(1_000, 100)), + Capability::SourceSync, + ); + assert_unsupported(block_on(driver.sync_statuses()), Capability::SourceSync); + assert_unsupported( + block_on(driver.raw_archive_coverage("gmail:conn-1", "archive")), + Capability::SourceSync, + ); + assert_unsupported( + block_on(driver.rebuild_from_raw_archive("gmail:conn-1", "archive")), + Capability::SourceSync, + ); + + assert_unsupported( + block_on(driver.coding_session_status()), + Capability::CodingSessions, + ); + assert_unsupported( + block_on(driver.ingest_coding_sessions(CodingSessionIngestRequest::default())), + Capability::CodingSessions, + ); +} + +#[test] +fn the_two_members_added_to_existing_families_refuse_rather_than_report_nothing() { + // Both inherit their trait's default body, and both defaults are a refusal + // on purpose. A `flush_source_tree` answering `Ok(0)` would tell a user + // their source was flushed and had nothing to write; a `diagnose` + // answering an empty report would have to claim `healthy` one way or the + // other, and both claims are untrue of a driver that never looked. + let driver = NullMemoryProvider::new(); + assert_unsupported( + block_on(driver.flush_source_tree("gmail:conn-1")), + Capability::Tree, + ); + assert_unsupported(block_on(driver.diagnose()), Capability::Maintenance); } #[test] diff --git a/crates/tinymemory-api/src/provider/audit_tests.rs b/crates/tinymemory-api/src/provider/audit_tests.rs index 2d7b12eb..4be86df4 100644 --- a/crates/tinymemory-api/src/provider/audit_tests.rs +++ b/crates/tinymemory-api/src/provider/audit_tests.rs @@ -135,13 +135,18 @@ fn honest_driver_passes_the_audit() { #[test] fn over_claiming_driver_is_reported_as_advertised_but_absent() { // Advertises everything, exposes no optional accessor. Every one of the - // thirteen optional families would fail on first call — the exact + // optional families would fail on first call — the exact // registered-but-failing outcome the capability filter exists to prevent. let liar = Fixture::new(Capabilities::all(), false); let audit = audit_provider(&liar).expect_err("over-claiming driver must fail the audit"); assert_eq!(audit.present_but_unadvertised, Vec::new()); - assert_eq!(audit.advertised_but_absent.len(), 15); + // Everything except the mandatory three, derived rather than spelled out: + // a family added to the contract must land here without editing a literal. + assert_eq!( + audit.advertised_but_absent.len(), + Capability::ALL.len() - Capability::MANDATORY.len() + ); assert!(audit.advertised_but_absent.contains(&Capability::Tree)); // The mandatory three are supertraits, so they can never be missing. assert!(!audit.advertised_but_absent.contains(&Capability::Core)); diff --git a/crates/tinymemory-api/src/provider/content.rs b/crates/tinymemory-api/src/provider/content.rs index ff7b9b19..83baf139 100644 --- a/crates/tinymemory-api/src/provider/content.rs +++ b/crates/tinymemory-api/src/provider/content.rs @@ -363,4 +363,51 @@ pub trait MemoryTree: Send + Sync { ) -> Result, MemoryError> { Err(MemoryError::unsupported(Capability::Tree)) } + + /// Seal and cascade one source's tree now, and report how many summaries + /// were written. + /// + /// The "flush this source" control, for a user who does not want to wait + /// for the scheduled window. Everything else in this family is addressed + /// by *namespace*; this one is addressed by **source scope** — the + /// `{platform}:{connection}` string a sync writes under — because that is + /// the identity a caller has when it is looking at one connected source. + /// + /// # Why not `seal` plus `cascade` on the same namespace + /// + /// Because a source scope is not a namespace, and the mapping between them + /// is the driver's. A source's content may sit under a tree the driver + /// created for it, named however the driver names trees; a caller that + /// tried to derive the namespace would be reimplementing that naming, and + /// would get it wrong for exactly the sources whose trees were created + /// before whatever convention it copied. + /// + /// It is also one operation rather than two on purpose. Sealing without + /// cascading leaves a tier of leaves with no summary above them, which + /// reads as an empty tree to every structural query — and a caller that + /// made the second call separately would have a window where that is the + /// state. + /// + /// # Why a count and not a tree + /// + /// The engine's own flush hands back a live tree object, and the caller's + /// question is "did anything happen". A handle to a driver's internal + /// object is precisely what this contract exists not to pass, and once the + /// labelling decision that flush needs is made driver-side — which is + /// where it comes from anyway — there is nothing else the object was + /// carrying that a caller can use. + /// + /// # Errors + /// + /// [`MemoryError::Unsupported`] from a driver with a tree family but no + /// source-scoped flush. Backend failures otherwise. + /// + /// A scope with nothing buffered is `Ok(0)`, not an error: idempotent for + /// the same reason [`Self::seal`] is, so a caller may offer the control + /// unconditionally. An **unknown** scope is also `Ok(0)` — the driver + /// creates the tree if it has to, so there is no scope it can refuse, and + /// a caller cannot use this to probe which scopes exist. + async fn flush_source_tree(&self, _source_scope: &str) -> Result { + Err(MemoryError::unsupported(Capability::Tree)) + } } diff --git a/crates/tinymemory-api/src/provider/driver.rs b/crates/tinymemory-api/src/provider/driver.rs index e23b82c8..b3768ce3 100644 --- a/crates/tinymemory-api/src/provider/driver.rs +++ b/crates/tinymemory-api/src/provider/driver.rs @@ -66,6 +66,8 @@ use crate::provider::records::{ MemoryGoals, MemoryMaintenance, MemorySourceSink, MemoryToolMemory, }; use crate::provider::retrieval::MemoryRetrieval; +use crate::provider::sessions::MemoryCodingSessions; +use crate::provider::sync::MemorySourceSync; /// A bound memory driver. /// @@ -74,8 +76,8 @@ use crate::provider::retrieval::MemoryRetrieval; /// supertraits, so a driver missing any of them cannot be constructed as a /// provider at all. /// -/// The thirteen optional families are reached through the `as_*` accessors below. -/// Each defaults to `None`, so a minimal driver implements only what it +/// The seventeen optional families are reached through the `as_*` accessors +/// below. Each defaults to `None`, so a minimal driver implements only what it /// supports and inherits correct absence for everything else. #[async_trait] pub trait MemoryProvider: MemoryCore + MemoryRecall + MemoryPortability + 'static { @@ -197,6 +199,21 @@ pub trait MemoryProvider: MemoryCore + MemoryRecall + MemoryPortability + 'stati None } + /// Syncs this driver runs itself, when advertised. + /// + /// Distinct from [`Self::as_sources`], which is the write seam a caller + /// pushes fetched items through. A driver may serve either alone: pushing + /// a batch needs storage, walking a connection needs pipelines and a + /// credential seam. + fn as_source_sync(&self) -> Option<&dyn MemorySourceSync> { + None + } + + /// Local coding-agent transcript ingestion, when advertised. + fn as_coding_sessions(&self) -> Option<&dyn MemoryCodingSessions> { + None + } + /// Whether `capability` is actually **reachable** on this driver. /// /// This is the implementation-side truth, as opposed to @@ -204,7 +221,7 @@ pub trait MemoryProvider: MemoryCore + MemoryRecall + MemoryPortability + 'stati /// agree; [`crate::provider::audit_provider`] is where they are compared. /// /// The mandatory three are always `true` because they are supertraits. The - /// remaining ten delegate to their accessor. + /// remaining seventeen delegate to their accessor. /// /// The `match` is deliberately exhaustive: [`Capability`] is not /// `#[non_exhaustive]`, so adding a family without adding an accessor and @@ -227,6 +244,8 @@ pub trait MemoryProvider: MemoryCore + MemoryRecall + MemoryPortability + 'stati Capability::Retrieval => self.as_retrieval().is_some(), Capability::Profile => self.as_profile().is_some(), Capability::Episodic => self.as_episodic().is_some(), + Capability::SourceSync => self.as_source_sync().is_some(), + Capability::CodingSessions => self.as_coding_sessions().is_some(), } } } diff --git a/crates/tinymemory-api/src/provider/mod.rs b/crates/tinymemory-api/src/provider/mod.rs index de1a99da..01c15914 100644 --- a/crates/tinymemory-api/src/provider/mod.rs +++ b/crates/tinymemory-api/src/provider/mod.rs @@ -1,4 +1,4 @@ -//! The memory driver contract: [`MemoryProvider`] plus the eighteen capability +//! The memory driver contract: [`MemoryProvider`] plus the twenty capability //! family traits a driver may implement. //! //! ## Shape @@ -22,12 +22,15 @@ //! ├─ as_chunks() -> Option<&dyn MemoryChunks> //! ├─ as_retrieval() -> Option<&dyn MemoryRetrieval> //! ├─ as_profile() -> Option<&dyn MemoryProfile> -//! └─ as_episodic() -> Option<&dyn MemoryEpisodic> +//! ├─ as_episodic() -> Option<&dyn MemoryEpisodic> +//! ├─ as_source_sync() -> Option<&dyn MemorySourceSync> +//! └─ as_coding_sessions() +//! -> Option<&dyn MemoryCodingSessions> //! ``` //! //! The mandatory three are supertraits, so "mandatory" is enforced by the type -//! system rather than by a runtime check. The optional fifteen are accessors that -//! default to `None`, so absence is the default and presence is opt-in. +//! system rather than by a runtime check. The optional seventeen are accessors +//! that default to `None`, so absence is the default and presence is opt-in. //! //! ## Rules that bind every family //! @@ -42,18 +45,23 @@ //! third-party driver depends on this crate alone. //! 4. **The driver never assigns provenance.** [`crate::types::MemoryTaint`] is //! an argument on every write path and a preserved field on every import. -//! 5. **The host owns the loop.** Sealing, cascading, maintenance, and source -//! sync are all "run one step when asked"; no driver installs a background -//! task or hooks the agent turn. +//! 5. **The host owns the loop** — with one recorded exception. Sealing, +//! cascading and maintenance are all "run one step when asked", and no +//! driver hooks the agent turn. *Source sync* is the exception, and it +//! moved deliberately: a host that stops compiling an engine has no +//! periodic loop left to run, so the loops went into the module beside the +//! queue pool. What the caller kept is the manual trigger — see +//! [`MemorySourceSync`], which exists because a user's "sync now" is not a +//! schedule and no member of [`MemorySourceSink`] can express it. //! 6. **Object safety throughout.** No generics, no `Self` returns, no //! associated constants — every family is usable as `&dyn`. //! //! ## Reference implementation //! -//! [`crate::null::NullMemoryProvider`] implements all eighteen families: +//! [`crate::null::NullMemoryProvider`] implements all twenty families: //! `/dev/null` semantics for the mandatory three, and -//! [`crate::error::MemoryError::Unsupported`] for the other fifteen, which it does -//! not advertise. It is what a compiled-out or unconfigured memory subsystem +//! [`crate::error::MemoryError::Unsupported`] for the other seventeen, which it +//! does not advertise. It is what a compiled-out or unconfigured memory subsystem //! binds to, and it doubles as the proof that the mandatory set is //! implementable without a storage engine. @@ -68,16 +76,25 @@ pub mod people; pub mod profile; pub mod records; pub mod retrieval; +pub mod sessions; +pub mod sync; // The value types every family exchanges, defined in `tinymemory-bus` and // re-exported at their historical path. See this crate's `lib.rs` for why the // vocabulary sits a layer below the traits. -pub use tinymemory_bus::provider::types; +// +// `diagnosis` is re-exported rather than wrapped in a module of its own here +// because it carries no trait: it is the return shape of one maintenance +// member, so there is nothing for a file on this side to hold. +pub use tinymemory_bus::provider::{diagnosis, types}; pub use audit::{audit_provider, CapabilityAudit}; pub use chunks::{ ChunkDetail, ChunkEmbedding, ChunkListRow, ChunkQuery, MemoryChunks, SourceTotal, }; pub use content::{MemoryDocuments, MemoryIngest, MemoryTree}; +pub use diagnosis::{ + DegradedCapabilities, Diagnosis, DiagnosisCounters, DiagnosisFailure, DiagnosisStage, +}; pub use driver::MemoryProvider; pub use episodic::{ConversationSegment, EpisodicEvent, EpisodicTurn, EventKind, MemoryEpisodic}; pub use knowledge::{MemoryDiff, MemoryEntities, MemoryGraph, INBOUND_SCAN_LIMIT}; @@ -92,6 +109,14 @@ pub use retrieval::{ CoverWindowQuery, EntityMatch, FastRetrieveQuery, MemoryRetrieval, RetrievalHit, RetrievalNodeKind, RetrievalResponse, SourceRetrievalQuery, }; +pub use sessions::{ + CodingSessionIngestReport, CodingSessionIngestRequest, CodingSessionSource, + MemoryCodingSessions, +}; +pub use sync::{ + MemorySourceSync, RawArchiveCoverage, RawRebuildOutcome, SourceSyncState, SourceSyncStatus, + SyncAuditEntry, SyncFreshness, SyncRunOutcome, +}; pub use types::{ ChangeKind, ChunkEntityOccurrence, DiffReport, EntityHit, EntityOccurrence, EntityRef, ExportPage, ExportRecord, FlushOutcome, ForgetOutcome, ForgetSelector, ImportOutcome, diff --git a/crates/tinymemory-api/src/provider/records.rs b/crates/tinymemory-api/src/provider/records.rs index 80ce4509..fec952f3 100644 --- a/crates/tinymemory-api/src/provider/records.rs +++ b/crates/tinymemory-api/src/provider/records.rs @@ -19,6 +19,7 @@ use async_trait::async_trait; use crate::capabilities::Capability; use crate::error::MemoryError; use crate::goals::GoalsDoc; +use crate::provider::diagnosis::Diagnosis; use crate::provider::types::{ FlushOutcome, ForgetOutcome, ForgetSelector, IngestOutcome, MaintenanceReport, PurgeOutcome, QueueFailure, QueueStats, ResetOutcome, SourceItem, StoreStats, @@ -416,4 +417,57 @@ pub trait MemoryMaintenance: Send + Sync { async fn purge_all(&self) -> Result { Err(MemoryError::unsupported(Capability::Maintenance)) } + + /// The typed, per-stage diagnosis of the driver's ingest pipeline. + /// + /// Read-only in exactly the sense [`Self::doctor`] is, and driven by the + /// same pass. What differs is who reads the answer. + /// + /// # Why this is not [`Self::doctor`] widened + /// + /// [`MaintenanceReport`] is deliberately one shape across `reembed`, + /// `compact`, `consolidate` and `doctor`, so a scheduler running all four + /// on a timer does not special-case one. That is the right shape for a + /// scheduler and the wrong one for an operator: it flattens a classified + /// cause into a line of prose, and a caller that wants to *act* on the + /// cause — localise the remediation, decide whether a retry could help, + /// tell "nothing ingested" apart from "ingested, not yet embedded" — has + /// to parse that prose back into the structure it was flattened from. + /// + /// Adding those fields to [`MaintenanceReport`] was the alternative. Four + /// of its five producers would leave every one of them empty, so the type + /// would stop describing what any single call returns; and changing + /// `doctor`'s return type instead is a breaking change to a member drivers + /// already implement. + /// + /// So the two coexist and a driver derives both from one pass: + /// [`Self::doctor`] is the lossy projection a scheduler reads, + /// [`Self::diagnose`] the full one a human or an agent reads. + /// + /// # Why a caller cannot compute this itself + /// + /// Two of the four parts of a [`Diagnosis`] exist only inside the driver's + /// process. [`crate::provider::diagnosis::DegradedCapabilities`] is set by + /// the embed and extract stages as they run, and + /// [`crate::provider::diagnosis::DiagnosisCounters`] is a read of the + /// driver's own storage. A caller that hosts no engine has neither, and + /// what it would produce is not a stale diagnosis but a confident + /// all-clear over counters of zero. + /// + /// # Errors + /// + /// [`MemoryError::Unsupported`] from a driver that cannot diagnose itself + /// — deliberately, and unlike the reads above, which default to an empty + /// answer. An empty [`Diagnosis`] is not "nothing to report": its + /// `healthy` flag would have to say something, and both answers are lies. + /// `false` with no stages sends a user hunting a fault that was never + /// found; `true` reports a clean bill of health from a driver that never + /// looked. + /// + /// Backend failures otherwise. A *finding* is not an error, for the reason + /// [`Self::doctor`] gives: a pipeline with problems still returns `Ok` + /// with the problems in it. + async fn diagnose(&self) -> Result { + Err(MemoryError::unsupported(Capability::Maintenance)) + } } diff --git a/crates/tinymemory-api/src/provider/sessions.rs b/crates/tinymemory-api/src/provider/sessions.rs new file mode 100644 index 00000000..591b777d --- /dev/null +++ b/crates/tinymemory-api/src/provider/sessions.rs @@ -0,0 +1,102 @@ +//! [`MemoryCodingSessions`] — distilling the user's coding-agent transcripts. +//! +//! A driver advertising +//! [`Capability::CodingSessions`](crate::capabilities::Capability::CodingSessions) +//! knows where a coding agent leaves its session transcripts, can say how much +//! is there without ingesting any of it, and can run the pass that turns those +//! transcripts into observations about the user. +//! +//! # Why not the source-sync family +//! +//! Both fetch and report, and that is the whole of the resemblance. +//! [`MemorySourceSync`](super::MemorySourceSync) walks a *remote* connection +//! the user authorised, is billed per provider action, and resumes from a +//! provider cursor. This walks *local* files the user's own tools wrote, is +//! billed per inference window, and resumes from a per-file state store. The +//! two fail independently, which is the test that decides a family: a driver +//! running server-side has no `~/.claude` to read, and a driver fronting a +//! local vault may have no authorised connection to walk. Advertising them +//! together puts a dead control in front of whichever half is absent. +//! +//! # No path crosses this contract +//! +//! Not one member takes a directory. Which agents are supported, where each +//! keeps its sessions, and how the environment overrides those locations are +//! resolved driver-side. A caller passing roots would be choosing which files +//! the driver opens — the shape a source gate exists to prevent — and would +//! freeze the supported-agent list into the contract, where adding an agent +//! becomes a version bump instead of a driver release. +//! +//! # Both members are bounded, and say when the bound bit +//! +//! A status scan caps the files it opens and the bytes it reads; an ingest +//! caps the sessions it processes. Neither is a promise to finish: a large +//! history drains across repeated calls, and +//! [`CodingSessionSource::scan_truncated`] and +//! [`CodingSessionIngestReport::budget_hit`] are how a caller knows to ask +//! again rather than to report a total it has not seen. + +use async_trait::async_trait; + +use crate::error::MemoryError; + +// The value types this family exchanges — defined in `tinymemory-bus` because +// they cross the module boundary, re-exported here so the two trees stay the +// same shape and the types stay the same types. +pub use tinymemory_bus::provider::sessions::{ + CodingSessionIngestReport, CodingSessionIngestRequest, CodingSessionSource, +}; + +/// Reading and distilling local coding-agent session transcripts. +#[async_trait] +pub trait MemoryCodingSessions: Send + Sync { + /// What each supported agent's session store holds right now. + /// + /// One row per agent the driver knows about, present or not — an absent + /// agent is a row with [`CodingSessionSource::available`] `false`, not a + /// missing row, because a caller rendering a picker needs to show what it + /// could offer as well as what it can. + /// + /// Bounded by the driver's own scan caps rather than by an argument: the + /// caps exist to keep a status call from reading a multi-gigabyte history, + /// and a caller able to raise them could turn a status probe into one. + /// + /// # Errors + /// + /// Backend failures only. A file that cannot be read is counted in + /// [`CodingSessionSource::invalid_files`] rather than raised — one + /// half-written transcript from a session that is still running must not + /// fail a scan of four hundred. + async fn coding_session_status(&self) -> Result, MemoryError>; + + /// Distil coding sessions into observations, and report what the pass did. + /// + /// # This costs inference, and the caller cannot bound the time + /// + /// Each session is one or more sequential model calls, so the wall-clock + /// cost scales with [`CodingSessionIngestRequest::max_sessions`] and with + /// how long the individual transcripts are. The driver clamps the request + /// to its own ceiling; a caller that needs a deadline enforces it on its + /// own side, because a driver that abandoned a run mid-session would leave + /// a state store that disagrees with what was written. + /// + /// # Errors + /// + /// [`MemoryError::Unsupported`] from a driver with no summarisation + /// provider resolvable — deliberately not an empty report, which would + /// tell a user their history was imported and found nothing in it. + /// + /// [`MemoryError::BudgetExceeded`] when an inference budget stops the run + /// before it processed anything. A run that stopped on its *session* + /// budget after doing work is an `Ok` with + /// [`CodingSessionIngestReport::budget_hit`] set, because that is progress + /// the caller should keep and continue from. + /// + /// Otherwise backend failures. Individual failed sessions are counted in + /// [`CodingSessionIngestReport::sessions_failed`], for the same reason the + /// status scan counts unreadable files. + async fn ingest_coding_sessions( + &self, + request: CodingSessionIngestRequest, + ) -> Result; +} diff --git a/crates/tinymemory-api/src/provider/sync.rs b/crates/tinymemory-api/src/provider/sync.rs new file mode 100644 index 00000000..5f626552 --- /dev/null +++ b/crates/tinymemory-api/src/provider/sync.rs @@ -0,0 +1,229 @@ +//! [`MemorySourceSync`] — the family for syncs the *driver* runs. +//! +//! [`crate::provider::MemorySourceSink`] is the seam a caller writes fetched +//! items through. This is the other half of the same subject and deliberately +//! not the same family: here the driver owns the pipelines, walks the +//! connection, holds the cursor and the budget, and can price what it spent. +//! +//! ## What changed, and what did not +//! +//! The contract's fifth rule — "the host owns the loop" — was written when +//! every sync was driven from outside the driver. It still holds for +//! *sealing, cascading and maintenance*, and it no longer describes source +//! sync: the periodic Composio and workspace loops run inside the module, next +//! to the queue pool, because a host that stops compiling the engine has no +//! loop left to run. What stayed with the caller is the part a loop cannot +//! provide — the **manual** trigger. A user pressing "sync now" is not a +//! schedule, and no member of the sink family can express it. +//! +//! That is why this is a family and not four more methods on +//! [`crate::provider::MemorySourceSink`]. A driver that accepts a batch is not +//! thereby a driver that can walk an OAuth connection: a remote HTTP backend +//! and [`crate::null::NullMemoryProvider`] both do the first and neither can do +//! the second. Advertising them together would put a "sync now" control in +//! front of a driver that fails on first press — the registered-but-failing +//! outcome [`crate::capabilities`] exists to avoid — and, because a new method +//! on an already-advertised family is a **major** contract bump while a new +//! family is a minor one, it would also break every existing driver. +//! +//! ## Credentials still do not cross this contract +//! +//! No signature here names a token, a key, or a session. The driver resolves +//! whatever it needs through its own host seam, at call time — which is the +//! only way that works, since a connection can be authorised in a browser +//! minutes after the driver was bound. +//! +//! ## Neither does configuration +//! +//! [`Self::run_connection_sync`] takes no budget arguments. The per-source +//! caps — item limits, depth windows, token and cost ceilings — live in the +//! registry the driver already reads, so passing them would be a caller +//! restating something the driver knows, with two sources of truth for a limit +//! that costs money when it is wrong. + +use async_trait::async_trait; + +use crate::error::MemoryError; + +// The value types this family exchanges. They are defined in `tinymemory-bus` +// — they cross the module boundary, and a host that only makes calls must be +// able to name them without compiling this trait — and re-exported here so the +// two trees stay the same shape and the types stay the same types. +pub use tinymemory_bus::provider::sync::{ + RawArchiveCoverage, RawRebuildOutcome, SourceSyncState, SourceSyncStatus, SyncAuditEntry, + SyncFreshness, SyncRunOutcome, +}; + +/// Running a source sync on demand, and reporting what past runs cost. +#[async_trait] +pub trait MemorySourceSync: Send + Sync { + /// Sync one connection now, and report what the run moved and spent. + /// + /// `toolkit` is the provider slug (`gmail`, `slack`, `github`, …) and + /// `connection_id` the authorised connection under it. Both are wire + /// strings rather than enums, for the reason the sink family's + /// `source_kind` is one: the set belongs to whoever integrates providers + /// and grows without a contract change. + /// + /// # This is the manual path, and it is not idempotent + /// + /// It is what a user's "sync now" reaches. Calling it twice runs the + /// pipeline twice — the cursor makes the second run cheap rather than + /// free, and both runs append an audit row. A driver that is already + /// syncing this connection should serialise rather than run a second walk + /// concurrently; two walks sharing one cursor lose items. + /// + /// # Errors + /// + /// [`MemoryError::Invalid`] for a toolkit the driver has no pipeline for + /// or a connection it cannot resolve — deliberately not an outcome of + /// zero, which a caller would render as "nothing new" over a source that + /// can never sync. + /// + /// [`MemoryError::BudgetExceeded`] when a per-source token or cost ceiling + /// stops the run. + /// + /// Otherwise backend and provider failures. A run that failed **after** + /// spending is still a failure: what it burned belongs in the error the + /// driver returns and in the audit row it writes, not in an `Ok` that + /// reports partial progress as success. + async fn run_connection_sync( + &self, + toolkit: &str, + connection_id: &str, + ) -> Result; + + /// The persisted cursor, dedup and budget state for one connection. + /// + /// `Ok(None)` for a connection that has never synced — a valid state, not + /// a missing record, and the reason this is not + /// [`MemoryError::NotFound`]: a status surface listing every connection + /// would otherwise turn "never synced" into an error row. + /// + /// # This is the status read, not the whole row + /// + /// [`SourceSyncState`] carries counts where the persisted row carries + /// sets, and says why. A caller that needs the dedup set itself — a + /// disconnect walking it to decide which per-item documents to forget — + /// reads the row through [`crate::provider::MemoryGraph::kv_get`] instead. + /// What this member adds over that read is the driver's own day-rollover + /// rule applied to the budget, and the absence of a row reported as + /// `Ok(None)` rather than as a namespace-and-key convention the caller has + /// to know. + /// + /// # Errors + /// + /// Backend failures only. + async fn source_sync_state( + &self, + toolkit: &str, + connection_id: &str, + ) -> Result, MemoryError>; + + /// Past sync runs, newest first. + /// + /// `limit` caps the rows and the driver clamps it to its own ceiling — a + /// caller cannot raise it by asking for more, the same rule + /// [`crate::provider::ChunkQuery::limit`] carries. `None` means "the + /// driver's own cap", **not** unbounded: the log is append-only for the + /// life of a workspace, so an unbounded read is a response that grows + /// without limit and eventually cannot cross a frame at all. + /// + /// A caller totalling a period therefore reads the newest rows and stops + /// when it passes the period's start. That is the one reduction against + /// reading the log file directly, and it is the shape a total wants + /// anyway: newest-first ordering means the rows a period needs are the + /// first ones returned. + /// + /// # Errors + /// + /// Backend failures only. A driver that has never synced returns an empty + /// log, which is true of it. + async fn sync_audit_log( + &self, + limit: Option, + ) -> Result, MemoryError>; + + /// Price a token count at the same rate the driver stamped onto its audit + /// rows. + /// + /// # Why this is a call and not a constant a caller could hold + /// + /// It looks like arithmetic, and copying it is the mistake this member + /// exists to prevent. The same constants produce + /// [`SyncAuditEntry::estimated_cost_usd`] on every row this driver writes. + /// A caller holding its own copy has a second price the moment either side + /// is retuned, and it would then present a projected cost and a historical + /// total computed at two different rates, on the same screen, with nothing + /// to say which. + /// + /// So the price stays where the rows are written, and a caller that wants + /// to quote one asks. + /// + /// # Errors + /// + /// Backend failures only; a driver that prices nothing answers `0.0` + /// rather than refusing, which is true of a driver whose sync costs the + /// user nothing. + async fn estimate_sync_cost_usd( + &self, + input_tokens: u64, + output_tokens: u64, + ) -> Result; + + /// Per-provider sync progress, derived from stored content. + /// + /// Not from the sync machinery's own counters, and the difference is what + /// makes it survive a restart: a run killed mid-wave leaves its chunks + /// behind, so a count taken from the content is real where a counter that + /// was never decremented is not. + /// + /// # Errors + /// + /// Backend failures only; a store with no synced content returns an empty + /// list. + async fn sync_statuses(&self) -> Result, MemoryError>; + + /// How much of one raw archive the tree derived from it covers. + /// + /// `tree_scope` names the summary tree and `archive_source_id` the raw + /// archive beneath it; a sync writes both, and a run that died between + /// them leaves an archive the tree does not cover. This is the read behind + /// a "reconcile" control, and [`Self::rebuild_from_raw_archive`] is its + /// repair. + /// + /// # Errors + /// + /// Backend failures only. An archive the driver has never written is a + /// coverage of zero over a total of zero, not [`MemoryError::NotFound`]: + /// the caller is asking whether anything is missing, and "there is nothing + /// there" answers that. + async fn raw_archive_coverage( + &self, + tree_scope: &str, + archive_source_id: &str, + ) -> Result; + + /// Re-derive a summary tree from its raw archive. + /// + /// The repair [`Self::raw_archive_coverage`] diagnoses. It re-reads the + /// archive and re-summarises what the tree is missing, so it costs + /// inference and can be slow; a caller runs it in the background and + /// reports progress from the outcome rather than blocking a user on it. + /// + /// Safe to repeat: a file the tree already covers is not summarised twice, + /// so a rebuild interrupted halfway resumes rather than starting over. + /// + /// # Errors + /// + /// [`MemoryError::BudgetExceeded`] when an inference budget stops the + /// rebuild mid-run — what it managed is in the error, not in an `Ok` that + /// would read as a completed repair. + /// + /// Otherwise backend failures. + async fn rebuild_from_raw_archive( + &self, + tree_scope: &str, + archive_source_id: &str, + ) -> Result; +} diff --git a/crates/tinymemory-bus/README.md b/crates/tinymemory-bus/README.md index 5598697f..c28f5fa8 100644 --- a/crates/tinymemory-bus/README.md +++ b/crates/tinymemory-bus/README.md @@ -4,7 +4,7 @@ Every type that crosses the TinyMemory `TinyBus` boundary, and the names of the members that carry them. TinyMemory ships as a loadable module so a host does not compile the engine: -`crates/tinymemory-module` exports one object with 89 members on it, built as a +`crates/tinymemory-module` exports one object with 120 members on it, built as a `cdylib`. A host can load that binary but cannot `use` anything out of it, so the payload vocabulary has to be published as an ordinary library. This is it. @@ -13,6 +13,8 @@ the payload vocabulary has to be published as an ordinary library. This is it. | `names` | bus name, object path, one constant per member | | `types`, `chunks`, `recall`, `tree`, `goals`, `tool_memory`, `health`, `capabilities`, `evidence` | the value vocabulary | | `provider` | the value types each capability family exchanges | +| `learning` | the learning-candidate taxonomy — what a producer asserts about the user, and how strongly | +| `composio` | the connector-sync vocabulary: run reports, task envelopes, per-connection sync state, scope preferences | | `error`, `wire` | `MemoryError` and the name table it round-trips through | | `version` | `CONTRACT_VERSION` and the bind rule | @@ -47,7 +49,7 @@ A host depends on `tinymemory-bus` and gets vocabulary alone. ## What is deliberately absent -**No traits.** `MemoryProvider` and the eighteen capability-family traits +**No traits.** `MemoryProvider` and the twenty capability-family traits describe what an engine must implement, not what a frame carries. They stay in `tinymemory-api`. The split is readable off the path: a name here is data, a name there is an obligation. diff --git a/crates/tinymemory-bus/src/capabilities.rs b/crates/tinymemory-bus/src/capabilities.rs index e7641ac4..2a4600d7 100644 --- a/crates/tinymemory-bus/src/capabilities.rs +++ b/crates/tinymemory-bus/src/capabilities.rs @@ -51,7 +51,7 @@ use crate::error::MemoryError; /// One capability family a memory driver may advertise. /// -/// The variants are exactly the sixteen families of the memory contract. Each +/// The variants are exactly the twenty families of the memory contract. Each /// maps to a trait family in the contract, a group of RPC methods, and a group /// of agent tools; a driver that does not advertise a family simply has that /// surface absent. @@ -96,6 +96,23 @@ pub enum Capability { Profile, /// The turn-by-turn conversation record and its segment lifecycle. Episodic, + /// Running a source sync on demand, and reporting what past runs cost. + /// + /// The counterpart of [`Self::Sources`], not a widening of it. That family + /// is the *sink* — the driver accepts items a caller fetched — and it stays + /// true of every driver that can store a batch. This one says the driver + /// owns the pipelines: it walks the connection itself, holds the cursor and + /// the budget, and can price what it spent. A driver may serve either + /// without the other. + SourceSync, + /// Distilling the user's local coding-agent transcripts into observations. + /// + /// Separate from [`Self::SourceSync`] because the two fail independently: a + /// driver running server-side has no local transcript store to read, and a + /// driver fronting a local vault has no authorised remote connection to + /// walk. Advertising them together would put a dead control in front of + /// whichever half is absent. + CodingSessions, } impl Capability { @@ -104,7 +121,7 @@ impl Capability { /// Declaration order is also bit order in [`Capabilities`] and iteration /// order in its serialized form, so this slice is the single ordering /// authority for the whole module. - pub const ALL: [Capability; 18] = [ + pub const ALL: [Capability; 20] = [ Capability::Core, Capability::Recall, Capability::Ingest, @@ -126,6 +143,8 @@ impl Capability { Capability::Retrieval, Capability::Profile, Capability::Episodic, + Capability::SourceSync, + Capability::CodingSessions, ]; /// The families a driver must advertise to be bindable at all. @@ -169,6 +188,8 @@ impl Capability { Self::Retrieval => "retrieval", Self::Profile => "profile", Self::Episodic => "episodic", + Self::SourceSync => "source_sync", + Self::CodingSessions => "coding_sessions", } } @@ -216,6 +237,8 @@ impl Capability { Self::Retrieval => 15, Self::Profile => 16, Self::Episodic => 17, + Self::SourceSync => 18, + Self::CodingSessions => 19, } } diff --git a/crates/tinymemory-bus/src/capabilities_tests.rs b/crates/tinymemory-bus/src/capabilities_tests.rs index 02544181..6f0ce9e6 100644 --- a/crates/tinymemory-bus/src/capabilities_tests.rs +++ b/crates/tinymemory-bus/src/capabilities_tests.rs @@ -2,7 +2,7 @@ //! //! Three properties are load-bearing and each has its own test: //! -//! 1. the enum has exactly the sixteen contract families and no more; +//! 1. the enum has exactly the twenty contract families and no more; //! 2. the serialized form is stable snake_case **strings**, never discriminant //! integers — a driver deployed against an older build must keep advertising //! the same set after a variant is inserted mid-enum; @@ -19,9 +19,9 @@ use super::*; use serde_json::json; #[test] -fn capability_has_exactly_the_eighteen_contract_families() { - assert_eq!(Capability::ALL.len(), 18); - assert_eq!(Capability::all().len(), 18); +fn capability_has_exactly_the_twenty_contract_families() { + assert_eq!(Capability::ALL.len(), 20); + assert_eq!(Capability::all().len(), 20); let names: Vec<&str> = Capability::ALL.iter().map(|c| c.as_str()).collect(); assert_eq!( @@ -45,6 +45,8 @@ fn capability_has_exactly_the_eighteen_contract_families() { "retrieval", "profile", "episodic", + "source_sync", + "coding_sessions", ] ); } @@ -152,12 +154,14 @@ fn capabilities_empty_contains_nothing() { } #[test] -fn capabilities_bit_width_has_room_well_beyond_the_current_sixteen_families() { - // A `u16` bitset (the original representation) has exactly 16 bit - // positions, leaving room for only 3 more families before a family's - // `1 << index` bit-shift overflows. Pin the wider `u64` representation so - // a future family addition doesn't have to rediscover that ceiling. +fn capabilities_bit_width_has_room_well_beyond_the_current_family_count() { + // A `u16` bitset (the original representation) had exactly 16 bit + // positions — already fewer than the contract now has, so a family's + // `1 << index` bit-shift would overflow today. Pin the wider `u64` + // representation, and pin that the current count still fits inside it, so + // the next addition does not have to rediscover the ceiling. assert!(std::mem::size_of::() * 8 >= 64); + assert!(Capability::ALL.len() < std::mem::size_of::() * 8); } #[test] diff --git a/crates/tinymemory-bus/src/composio/mod.rs b/crates/tinymemory-bus/src/composio/mod.rs new file mode 100644 index 00000000..760c8824 --- /dev/null +++ b/crates/tinymemory-bus/src/composio/mod.rs @@ -0,0 +1,71 @@ +//! The Composio sync vocabulary: what a provider run produces, what it +//! remembers between runs, and what a user is willing to let it do. +//! +//! Composio is the connector layer the memory stack syncs through — Gmail, +//! Slack, Notion, GitHub, Linear, ClickUp and the catalog-only toolkits behind +//! them. Each toolkit's *provider* fetches a profile, pulls items, normalises +//! tasks and reports what it did. This module owns the shapes those runs +//! exchange; the providers themselves, the HTTP client, the registry and the +//! persistence live in the engine crate, where their dependencies belong. +//! +//! # Why the vocabulary is here and the providers are not +//! +//! The split is the same one [`crate::goals`] makes, and for the same reason. +//! A host reads these shapes: it renders a [`runs::SyncOutcome`] in the sync +//! status panel, files a [`tasks::NormalizedTask`] onto the agent's todo board, +//! gates a tool call on a [`scopes::UserScopePref`], and reports a +//! [`state::SyncState`]'s remaining daily budget. It does not run a provider — +//! the module does that, behind the bus. So the *values* have to be nameable +//! from the contract the host already links, while the code that produces them +//! must not be: a provider needs `reqwest`, an async runtime and the chunk +//! store, and none of those may enter this crate. +//! +//! The alternative — a parallel set of host-side structs — is the failure this +//! whole crate exists to prevent. A `SyncOutcome` decoded from the module would +//! not be *the* `SyncOutcome`, and every field added on one side would be a +//! silent decode gap on the other with nothing to catch it. One definition, +//! here, at the bottom. +//! +//! # What stayed in the engine crate, and why +//! +//! Read this before concluding something is missing: +//! +//! - **`ProviderContext`** — holds an `Arc` and dispatches Composio +//! actions through the host seam. Behaviour, not a payload. +//! - **`SyncStateStore`** and [`state::SyncState`]'s `load` / `save` — a +//! key/value I/O seam and its two async methods. This crate publishes no +//! traits (see [`crate`]); the engine crate carries the trait and offers the +//! two methods as an extension trait over the type defined here. +//! - **`user_scopes::{load, save}`** — the same, one namespace over: the +//! preference *shape* is here, reading and writing it is not. +//! - **`profile::{persist_provider_profile, load_connected_identities, +//! delete_connected_identity_facets, is_self_identity}`** — every one of +//! them reaches the profile facet store. +//! - **`profile_md`** — rewrites managed blocks in the host's `PROFILE.md`. +//! Filesystem mutation against a host-owned file; it is host policy that +//! happens to be written in the memory stack, not a wire type. +//! - **the curated catalogs and the provider registry** — several thousand +//! `&'static str` action slugs and a process-global `HashMap` of trait +//! objects. The [`scopes::CuratedTool`] *shape* is here so a catalog can be +//! typed; the catalogs are the engine's. + +pub mod profile; +pub mod runs; +pub mod scopes; +pub mod state; +pub mod tasks; + +pub use profile::{ + canonicalize, normalize_connection_identifier, render_connected_identities_section, + ConnectedIdentity, IdentityKind, ProviderUserProfile, +}; +pub use runs::{ComposioUsage, ComposioUsageHandle, SyncOutcome, SyncReason}; +pub use scopes::{ + agent_ready_toolkits, classify_unknown, find_curated, toolkit_from_slug, CuratedTool, + ToolScope, UserScopePref, +}; +pub use state::{ + extract_item_id, DailyBudget, SyncState, DEFAULT_DAILY_REQUEST_LIMIT, KV_NAMESPACE, + STATE_NAMESPACE, +}; +pub use tasks::{GithubFetchMode, NormalizedTask, TaskContainer, TaskFetchFilter, TaskKind}; diff --git a/crates/tinymemory-bus/src/composio/profile.rs b/crates/tinymemory-bus/src/composio/profile.rs new file mode 100644 index 00000000..a0651e10 --- /dev/null +++ b/crates/tinymemory-bus/src/composio/profile.rs @@ -0,0 +1,299 @@ +//! Who the user is on a connected account: the identifier kinds, how each one +//! is canonicalised, and what a loaded set of identities looks like. +//! +//! A provider hands back one [`ProviderUserProfile`] per connection. The engine +//! crate expands that into one facet row per identifier, so the self-identity +//! matcher can answer "is this message *from the user*?" with a +//! `(toolkit, kind, canonical value)` lookup rather than a fuzzy comparison. +//! [`IdentityKind`] is that matching axis and [`canonicalize`] is the routine +//! both sides of the comparison run. +//! +//! # Why canonicalisation is contract vocabulary +//! +//! Because equality of canonical forms is the matcher's *only* test. The value +//! is canonicalised once when a profile is persisted and again when a candidate +//! identifier is checked against it — no `COLLATE NOCASE`, no per-call +//! lowercasing. If the writer and the reader ran two different implementations +//! of that routine, the matcher would fail open: a user's own Slack messages +//! would stop being recognised as theirs, silently, with nothing to catch it. +//! Those two calls are on opposite sides of the module boundary, which is what +//! puts [`canonicalize`] here and not in the engine. +//! +//! The same argument covers [`normalize_connection_identifier`]: it produces +//! the key segment a facet row is *stored under*, so writer and reader must +//! spell it identically or a disconnect leaves rows behind and the removed +//! account keeps being treated as the user. +//! +//! # What is not here +//! +//! Everything that touches the profile facet store: persisting a profile, +//! loading the identities back, deleting a connection's rows, and the +//! `is_self_identity` lookups. Those are the engine crate's, along with the +//! `PROFILE.md` markdown bridge, which rewrites a file in the host's workspace +//! and is host policy rather than a wire type. + +use serde::{Deserialize, Serialize}; + +/// Normalized user profile shape returned by every provider. +/// +/// The shared fields (`display_name`, `email`, `username`, `avatar_url`, +/// `profile_url`) cover what a desktop UI needs to render a connected-account +/// card. Anything provider-specific — Gmail's `messagesTotal`, Notion's +/// workspace ids — goes into [`extras`](Self::extras), so callers do not widen +/// the shape every time a new toolkit lands, and so an identifier a provider +/// only exposes there (a Slack screen name, for instance) is still available to +/// the row expansion. +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct ProviderUserProfile { + /// Composio toolkit slug the profile was fetched from, e.g. `"gmail"`. + pub toolkit: String, + /// The connection the profile belongs to; `None` on toolkit-wide fetches. + pub connection_id: Option, + /// Human display label, when the provider exposes one. + pub display_name: Option, + /// Primary email address on the connected account. + pub email: Option, + /// Platform username or screen name, without any leading `@`. + pub username: Option, + /// URL of the account's avatar image. + pub avatar_url: Option, + /// URL of the account's public profile page. + pub profile_url: Option, + /// Provider-specific extras (raw JSON object). + #[serde(default)] + pub extras: serde_json::Value, +} + +/// Shape of an identifier persisted against a connection. +/// +/// Mirrors the matching dimensions of the memory tree's entity index, so the +/// self-check is a direct `(toolkit, kind, value)` lookup. The string form is +/// the last segment of the stored facet key, which makes every variant name a +/// durable value rather than a label. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum IdentityKind { + /// Platform-canonical immutable id — a Slack `U123ABC`, a Notion UUID. + UserId, + /// Email address. + Email, + /// An `@`-style screen name, canonicalised without the leading `@`. + Handle, + /// E.164 phone number. + Phone, + /// Human display label. A weak signal — never auto-promotes to "is self". + DisplayName, + /// Not for matching; kept for UI and prompt rendering. + AvatarUrl, + /// Not for matching; kept for UI and prompt rendering. + ProfileUrl, +} + +impl IdentityKind { + /// The stored key segment for this kind. + /// + /// Durable: it is the last segment of a persisted facet key, so renaming a + /// variant's string orphans every row filed under the old one. + pub fn as_str(self) -> &'static str { + match self { + Self::UserId => "user_id", + Self::Email => "email", + Self::Handle => "handle", + Self::Phone => "phone", + Self::DisplayName => "display_name", + Self::AvatarUrl => "avatar_url", + Self::ProfileUrl => "profile_url", + } + } + + /// Parse a stored key segment back into a kind. + /// + /// Returns `None` for anything unrecognised — including the legacy + /// `username` segment written before the identifier rewrite. Callers skip + /// those rows rather than failing the load, so one stale row cannot make a + /// user's whole identity set unreadable. + pub fn parse(s: &str) -> Option { + Some(match s { + "user_id" => Self::UserId, + "email" => Self::Email, + "handle" => Self::Handle, + "phone" => Self::Phone, + "display_name" => Self::DisplayName, + "avatar_url" => Self::AvatarUrl, + "profile_url" => Self::ProfileUrl, + _ => return None, + }) + } + + /// Confidence the matcher records on a row of this kind. + /// + /// Hard kinds auto-promote a chunk to "is self"; weak kinds require + /// corroboration. A display name is deliberately low — two people share a + /// name far more often than they share a user id. + pub fn confidence(self) -> f64 { + match self { + Self::UserId | Self::Phone => 1.00, + Self::Email => 0.95, + Self::Handle => 0.70, + Self::DisplayName => 0.40, + Self::AvatarUrl | Self::ProfileUrl => 0.50, + } + } + + /// Whether this kind is a real identity signal worth running through the + /// matcher, as opposed to a UI-only field. + pub fn is_matchable(self) -> bool { + matches!( + self, + Self::UserId | Self::Email | Self::Handle | Self::Phone | Self::DisplayName + ) + } +} + +/// Canonicalize a raw identifier for storage and lookup. +/// +/// The same routine runs on the entity side at match time, so equality of +/// canonical forms is the matcher's only test. Returns `None` for an empty or +/// whitespace-only value — storing one would match every chunk that happened to +/// carry a blank sender. +pub fn canonicalize(kind: IdentityKind, raw: &str) -> Option { + let trimmed = raw.trim(); + if trimmed.is_empty() { + return None; + } + Some(match kind { + IdentityKind::Email => trimmed.to_lowercase(), + IdentityKind::Handle => trimmed.trim_start_matches('@').to_lowercase(), + IdentityKind::Phone => trimmed + .chars() + .filter(|c| c.is_ascii_digit() || *c == '+') + .collect(), + IdentityKind::DisplayName => trimmed.split_whitespace().collect::>().join(" "), + IdentityKind::UserId | IdentityKind::AvatarUrl | IdentityKind::ProfileUrl => { + trimmed.to_string() + } + }) +} + +/// Every identifier known for one `(source, connection)` pair, collapsed into +/// one row. +/// +/// This is the read shape: the store holds one facet per identifier, and a +/// loader groups them back into this so a caller does not have to reassemble an +/// account from seven rows. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct ConnectedIdentity { + /// Toolkit slug the identity came from, normalised. + pub source: String, + /// Connection identifier, normalised — see + /// [`normalize_connection_identifier`]. + pub identifier: String, + /// Human display label, when one was stored. + pub display_name: Option, + /// Canonicalised email address. + pub email: Option, + /// Canonicalised screen name, without the leading `@`. + pub handle: Option, + /// Canonicalised phone number. + pub phone: Option, + /// Platform-canonical immutable id. + pub user_id: Option, + /// Avatar image URL. + pub avatar_url: Option, + /// Public profile URL. + pub profile_url: Option, +} + +/// Render a compact prompt section for a set of identities. +/// +/// Skips `user_id` (not human-readable) and prefixes a handle with `@`. Returns +/// an empty string — rather than a bare heading — when there is nothing worth +/// showing, so a caller can concatenate the result unconditionally. +/// +/// Every value is flattened onto one line and has `|` replaced before it is +/// joined with `|` separators: an identifier is user-controlled text arriving +/// from a third-party provider, and a display name containing a newline would +/// otherwise let it forge additional prompt lines. +pub fn render_connected_identities_section(identities: &[ConnectedIdentity]) -> String { + if identities.is_empty() { + return String::new(); + } + let mut out = String::from("## Connected Identities\n\n"); + for id in identities { + let mut fields = Vec::::new(); + if let Some(v) = id.display_name.as_deref() { + let v = sanitize_prompt_value(v); + if !v.is_empty() { + fields.push(v); + } + } + if let Some(v) = id.email.as_deref() { + let v = sanitize_prompt_value(v); + if !v.is_empty() { + fields.push(v); + } + } + if let Some(v) = id.handle.as_deref() { + let v = sanitize_prompt_value(v); + if !v.is_empty() { + fields.push(format!("@{v}")); + } + } + if let Some(v) = id.profile_url.as_deref() { + let v = sanitize_prompt_value(v); + if !v.is_empty() { + fields.push(v); + } + } + if fields.is_empty() { + continue; + } + let identifier = sanitize_prompt_value(&id.identifier); + out.push_str(&format!( + "- {} ({}): {}\n", + title_case(&id.source), + identifier, + fields.join(" | ") + )); + } + if out.trim() == "## Connected Identities" { + return String::new(); + } + out +} + +/// Normalize a raw toolkit slug or connection id into the form facet keys are +/// stored under. +/// +/// Lowercases, replaces every character outside `[a-z0-9_-]` with `_`, and +/// trims leading and trailing underscores. Writer and reader must both call +/// this: a caller passing a raw connection id to a delete would otherwise match +/// no rows, and the disconnected account would keep being treated as the user. +pub fn normalize_connection_identifier(raw: &str) -> String { + let mut out = String::with_capacity(raw.len()); + for ch in raw.chars() { + let lower = ch.to_ascii_lowercase(); + if lower.is_ascii_alphanumeric() || lower == '-' || lower == '_' { + out.push(lower); + } else { + out.push('_'); + } + } + out.trim_matches('_').to_string() +} + +fn title_case(raw: &str) -> String { + let mut chars = raw.chars(); + match chars.next() { + Some(first) => first.to_ascii_uppercase().to_string() + chars.as_str(), + None => String::new(), + } +} + +fn sanitize_prompt_value(raw: &str) -> String { + let replaced = raw.replace(['\n', '\r', '\t'], " ").replace('|', "/"); + replaced.split_whitespace().collect::>().join(" ") +} + +#[cfg(test)] +#[path = "profile_tests.rs"] +mod tests; diff --git a/crates/tinymemory-bus/src/composio/profile_tests.rs b/crates/tinymemory-bus/src/composio/profile_tests.rs new file mode 100644 index 00000000..b89310ea --- /dev/null +++ b/crates/tinymemory-bus/src/composio/profile_tests.rs @@ -0,0 +1,239 @@ +//! Tests for the identity vocabulary — the pure-data half. +//! +//! The facet-store half (persisting a profile, loading identities back, the +//! self-identity lookups, the disconnect delete) is tested in the engine crate +//! next to the store it drives. What is pinned here is what both sides of the +//! module boundary have to compute identically: the stored key segments, the +//! canonical form each kind reduces to, and the identifier normalisation a +//! delete has to reproduce exactly to match the rows a write created. + +#![allow(clippy::expect_used, clippy::unwrap_used, clippy::panic)] + +use super::{ + canonicalize, normalize_connection_identifier, render_connected_identities_section, + ConnectedIdentity, IdentityKind, ProviderUserProfile, +}; + +const ALL_KINDS: [IdentityKind; 7] = [ + IdentityKind::UserId, + IdentityKind::Email, + IdentityKind::Handle, + IdentityKind::Phone, + IdentityKind::DisplayName, + IdentityKind::AvatarUrl, + IdentityKind::ProfileUrl, +]; + +#[test] +fn every_identity_kind_round_trips_through_its_stored_segment() { + for kind in ALL_KINDS { + assert_eq!( + IdentityKind::parse(kind.as_str()), + Some(kind), + "the stored segment for {kind:?} does not parse back" + ); + } +} + +#[test] +fn the_stored_segments_are_the_durable_strings() { + assert_eq!(IdentityKind::UserId.as_str(), "user_id"); + assert_eq!(IdentityKind::Email.as_str(), "email"); + assert_eq!(IdentityKind::Handle.as_str(), "handle"); + assert_eq!(IdentityKind::Phone.as_str(), "phone"); + assert_eq!(IdentityKind::DisplayName.as_str(), "display_name"); + assert_eq!(IdentityKind::AvatarUrl.as_str(), "avatar_url"); + assert_eq!(IdentityKind::ProfileUrl.as_str(), "profile_url"); +} + +#[test] +fn an_unknown_segment_parses_to_none_rather_than_a_wrong_kind() { + // `username` is the legacy segment written before the rewrite; a loader + // skips those rows instead of failing the whole identity set. + assert_eq!(IdentityKind::parse("username"), None); + assert_eq!(IdentityKind::parse(""), None); + assert_eq!(IdentityKind::parse("USER_ID"), None); +} + +#[test] +fn only_the_matchable_kinds_are_matchable() { + assert!(IdentityKind::UserId.is_matchable()); + assert!(IdentityKind::Email.is_matchable()); + assert!(IdentityKind::Handle.is_matchable()); + assert!(IdentityKind::Phone.is_matchable()); + assert!(IdentityKind::DisplayName.is_matchable()); + // UI-only fields never enter the matcher. + assert!(!IdentityKind::AvatarUrl.is_matchable()); + assert!(!IdentityKind::ProfileUrl.is_matchable()); +} + +#[test] +fn a_display_name_never_outranks_a_hard_identifier() { + // The ordering is what stops two people sharing a name from being treated + // as one another; the absolute numbers matter less than the ranking. + assert!(IdentityKind::UserId.confidence() > IdentityKind::Handle.confidence()); + assert!(IdentityKind::Email.confidence() > IdentityKind::Handle.confidence()); + assert!(IdentityKind::Handle.confidence() > IdentityKind::DisplayName.confidence()); +} + +#[test] +fn every_confidence_is_a_probability() { + for kind in ALL_KINDS { + let confidence = kind.confidence(); + assert!( + (0.0..=1.0).contains(&confidence), + "{kind:?} reports a confidence outside 0..=1" + ); + } +} + +#[test] +fn an_email_canonicalises_case_insensitively() { + assert_eq!( + canonicalize(IdentityKind::Email, " Alice@Example.COM ").as_deref(), + Some("alice@example.com") + ); +} + +#[test] +fn a_handle_loses_its_at_sign_and_its_casing() { + assert_eq!( + canonicalize(IdentityKind::Handle, "@AliceW").as_deref(), + Some("alicew") + ); +} + +#[test] +fn a_phone_keeps_only_digits_and_the_country_plus() { + assert_eq!( + canonicalize(IdentityKind::Phone, "+1 (555) 010-9999").as_deref(), + Some("+15550109999") + ); +} + +#[test] +fn a_display_name_collapses_its_whitespace_but_keeps_its_casing() { + assert_eq!( + canonicalize(IdentityKind::DisplayName, " Alice W. ").as_deref(), + Some("Alice W.") + ); +} + +#[test] +fn an_opaque_identifier_is_only_trimmed() { + // A platform id is case-significant; lowercasing `U123ABC` would stop it + // matching the sender field it is compared against. + assert_eq!( + canonicalize(IdentityKind::UserId, " U123ABC ").as_deref(), + Some("U123ABC") + ); + assert_eq!( + canonicalize(IdentityKind::ProfileUrl, " https://x/Alice ").as_deref(), + Some("https://x/Alice") + ); +} + +#[test] +fn a_blank_identifier_canonicalises_to_nothing() { + // Storing an empty canonical form would match every chunk carrying a blank + // sender, which is the matcher failing open. + for kind in ALL_KINDS { + assert_eq!(canonicalize(kind, " "), None, "{kind:?} accepted a blank"); + assert_eq!(canonicalize(kind, ""), None, "{kind:?} accepted an empty"); + } +} + +#[test] +fn normalising_an_identifier_is_idempotent() { + // A delete re-normalises what a write already normalised; if the routine + // were not idempotent the second pass would miss the stored rows. + let once = normalize_connection_identifier("Conn ID/42!"); + assert_eq!(normalize_connection_identifier(&once), once); + assert_eq!(once, "conn_id_42"); +} + +#[test] +fn normalising_lowercases_and_replaces_and_trims() { + assert_eq!(normalize_connection_identifier("GMAIL"), "gmail"); + assert_eq!(normalize_connection_identifier("a.b:c"), "a_b_c"); + assert_eq!(normalize_connection_identifier("__lead__"), "lead"); + assert_eq!(normalize_connection_identifier("keep-me_1"), "keep-me_1"); +} + +fn identity(source: &str, identifier: &str) -> ConnectedIdentity { + ConnectedIdentity { + source: source.into(), + identifier: identifier.into(), + ..ConnectedIdentity::default() + } +} + +#[test] +fn rendering_no_identities_yields_nothing_at_all() { + assert_eq!(render_connected_identities_section(&[]), ""); +} + +#[test] +fn rendering_identities_with_no_showable_fields_yields_nothing() { + // A bare heading with no rows under it is worse than no section: it spends + // prompt budget telling the model nothing. + let identities = [identity("gmail", "conn-1")]; + assert_eq!(render_connected_identities_section(&identities), ""); +} + +#[test] +fn rendering_prefixes_a_handle_and_skips_the_opaque_user_id() { + let identities = [ConnectedIdentity { + display_name: Some("Alice W".into()), + email: Some("alice@example.com".into()), + handle: Some("alicew".into()), + user_id: Some("U123ABC".into()), + ..identity("slack", "conn-1") + }]; + let rendered = render_connected_identities_section(&identities); + + assert!(rendered.starts_with("## Connected Identities\n\n")); + assert!(rendered.contains("- Slack (conn-1): Alice W | alice@example.com | @alicew")); + assert!( + !rendered.contains("U123ABC"), + "the opaque user id is not human-readable and must not be rendered" + ); +} + +#[test] +fn rendering_neutralises_a_value_that_would_forge_prompt_lines() { + // Every field here is third-party text the user does not control. A newline + // or a pipe in a display name must not be able to invent a row. + let identities = [ConnectedIdentity { + display_name: Some("Bob\n- Admin (root): owner".into()), + email: Some("b|o@example.com".into()), + ..identity("gmail", "conn-2") + }]; + let rendered = render_connected_identities_section(&identities); + + assert_eq!( + rendered.lines().filter(|l| l.starts_with("- ")).count(), + 1, + "a value with a newline forged an extra row" + ); + assert!(rendered.contains("Bob - Admin (root): owner")); + assert!(rendered.contains("b/o@example.com")); +} + +#[test] +fn a_provider_profile_round_trips_with_its_open_extras() { + let profile = ProviderUserProfile { + toolkit: "slack".into(), + connection_id: Some("conn-1".into()), + display_name: Some("Alice".into()), + extras: serde_json::json!({ "handle": "alicew" }), + ..ProviderUserProfile::default() + }; + let json = serde_json::to_string(&profile).expect("serialize"); + let back: ProviderUserProfile = serde_json::from_str(&json).expect("deserialize"); + + assert_eq!(back.toolkit, "slack"); + assert_eq!(back.connection_id.as_deref(), Some("conn-1")); + assert_eq!(back.display_name.as_deref(), Some("Alice")); + assert_eq!(back.extras["handle"], "alicew"); +} diff --git a/crates/tinymemory-bus/src/composio/runs.rs b/crates/tinymemory-bus/src/composio/runs.rs new file mode 100644 index 00000000..91c38213 --- /dev/null +++ b/crates/tinymemory-bus/src/composio/runs.rs @@ -0,0 +1,112 @@ +//! What one provider sync run was for, what it cost, and what it produced. +//! +//! Three shapes, read in three different places: [`SyncReason`] is an *input* a +//! provider branches on (backfill everything, or pull since the cursor), +//! [`ComposioUsage`] is a running tally the execute chokepoint accumulates, and +//! [`SyncOutcome`] is the *report* a finished run hands back for the status +//! panel and the sync audit log. +//! +//! All three are serde shapes with no behaviour beyond arithmetic. The run +//! itself — the HTTP calls, the ingestion, the audit-log write — is the engine +//! crate's. + +use std::sync::{Arc, Mutex}; + +use serde::{Deserialize, Serialize}; + +/// Reason a sync was triggered. Providers use this to decide whether to do a +/// full backfill or an incremental pull. +/// +/// The serde form is `snake_case` and is mirrored into audit rows, so the +/// variant names are a compatibility surface. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum SyncReason { + /// First sync immediately after an OAuth handoff completes. + ConnectionCreated, + /// Periodic background sync from the scheduler. + Periodic, + /// Explicit user-driven sync from RPC or the UI. + Manual, +} + +impl SyncReason { + /// Stable lowercase tag, matching the serde representation. + /// + /// Callers that stamp the reason into a log line or an audit row want the + /// string without a serde round-trip; this is that string, and the pin test + /// holds the two forms equal. + pub fn as_str(&self) -> &'static str { + match self { + SyncReason::ConnectionCreated => "connection_created", + SyncReason::Periodic => "periodic", + SyncReason::Manual => "manual", + } + } +} + +/// Result of a provider sync run. Read by the sync status panel and written to +/// the sync audit log. +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct SyncOutcome { + /// Composio toolkit slug the run covered. + pub toolkit: String, + /// The connection that was synced; `None` for toolkit-wide runs. + pub connection_id: Option, + /// Why the run happened — normally a [`SyncReason::as_str`] tag, kept as a + /// `String` because a caller may report a reason the enum does not model. + pub reason: String, + /// How many items the run ingested. + pub items_ingested: usize, + /// Wall-clock start, epoch milliseconds. + pub started_at_ms: u64, + /// Wall-clock finish, epoch milliseconds. + pub finished_at_ms: u64, + /// One-line human summary for the status panel. + pub summary: String, + /// Provider-specific extras (raw JSON object). + #[serde(default)] + pub details: serde_json::Value, +} + +impl SyncOutcome { + /// How long the run took, in milliseconds. + /// + /// Saturating rather than panicking: the two timestamps come from separate + /// clock reads and a backwards system-clock adjustment between them would + /// otherwise take a status panel down over a cosmetic number. + pub fn elapsed_ms(&self) -> u64 { + self.finished_at_ms.saturating_sub(self.started_at_ms) + } +} + +/// Per-sync accumulator for Composio billable-action usage. +/// +/// Lives behind a shared handle on the provider context so the single `execute` +/// chokepoint can tally every action a provider fires during one run, whichever +/// provider it is and however many pages it paginates. The finished tally is +/// reported alongside the [`SyncOutcome`] for the sync audit log. +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct ComposioUsage { + /// Count of `execute` calls that returned a response this run. + /// + /// A provider-reported failure still counts — it reached Composio and was + /// billed. Transport errors do not. + pub actions_called: u32, + /// Sum of each response's backend-reported `cost_usd`. + pub cost_usd: f64, +} + +/// Shared, interior-mutable handle to a [`ComposioUsage`] tally. +/// +/// Cloning a provider context shares the same underlying counter, so the count +/// is stable no matter how the context is passed around within one sync. +/// +/// A `std` `Mutex` rather than an async one on purpose: the lock is taken for a +/// single increment and never held across an `await`, and this crate carries no +/// async runtime to borrow one from. +pub type ComposioUsageHandle = Arc>; + +#[cfg(test)] +#[path = "runs_tests.rs"] +mod tests; diff --git a/crates/tinymemory-bus/src/composio/runs_tests.rs b/crates/tinymemory-bus/src/composio/runs_tests.rs new file mode 100644 index 00000000..748a0dd2 --- /dev/null +++ b/crates/tinymemory-bus/src/composio/runs_tests.rs @@ -0,0 +1,113 @@ +//! Tests for the sync-run report vocabulary — the pure-data half. +//! +//! What is pinned here is what two separately compiled processes have to agree +//! on: the `snake_case` reason tags that end up in audit rows, and the +//! arithmetic on a report that a status panel renders without re-deriving. + +#![allow(clippy::expect_used, clippy::unwrap_used, clippy::panic)] + +use super::{ComposioUsage, ComposioUsageHandle, SyncOutcome, SyncReason}; + +#[test] +fn every_sync_reason_tag_matches_its_serde_form() { + for reason in [ + SyncReason::ConnectionCreated, + SyncReason::Periodic, + SyncReason::Manual, + ] { + let json = serde_json::to_string(&reason).expect("serialize"); + assert_eq!( + json, + format!("\"{}\"", reason.as_str()), + "as_str and the serde form disagree for {reason:?}" + ); + let back: SyncReason = serde_json::from_str(&json).expect("deserialize"); + assert_eq!(back, reason); + } +} + +#[test] +fn sync_reason_tags_are_the_stable_strings() { + assert_eq!(SyncReason::ConnectionCreated.as_str(), "connection_created"); + assert_eq!(SyncReason::Periodic.as_str(), "periodic"); + assert_eq!(SyncReason::Manual.as_str(), "manual"); +} + +#[test] +fn elapsed_is_the_difference_between_the_two_stamps() { + let outcome = SyncOutcome { + started_at_ms: 1_000, + finished_at_ms: 1_750, + ..SyncOutcome::default() + }; + assert_eq!(outcome.elapsed_ms(), 750); +} + +#[test] +fn elapsed_saturates_when_the_clock_went_backwards() { + // Two separate clock reads; an NTP step between them must not panic a + // status panel over a cosmetic number. + let outcome = SyncOutcome { + started_at_ms: 2_000, + finished_at_ms: 1_000, + ..SyncOutcome::default() + }; + assert_eq!(outcome.elapsed_ms(), 0); +} + +#[test] +fn an_outcome_round_trips_with_its_open_details_object() { + let outcome = SyncOutcome { + toolkit: "gmail".into(), + connection_id: Some("conn-1".into()), + reason: SyncReason::Periodic.as_str().to_string(), + items_ingested: 12, + started_at_ms: 5, + finished_at_ms: 9, + summary: "12 messages".into(), + details: serde_json::json!({ "pages": 3 }), + }; + let json = serde_json::to_string(&outcome).expect("serialize"); + let back: SyncOutcome = serde_json::from_str(&json).expect("deserialize"); + + assert_eq!(back.toolkit, "gmail"); + assert_eq!(back.connection_id.as_deref(), Some("conn-1")); + assert_eq!(back.reason, "periodic"); + assert_eq!(back.items_ingested, 12); + assert_eq!(back.elapsed_ms(), 4); + assert_eq!(back.summary, "12 messages"); + assert_eq!(back.details["pages"], 3); +} + +#[test] +fn an_outcome_decodes_when_details_is_absent() { + // `details` is `#[serde(default)]`, so an older peer that never wrote the + // field still decodes rather than failing the whole frame. + let back: SyncOutcome = serde_json::from_str( + r#"{"toolkit":"slack","connection_id":null,"reason":"manual", + "items_ingested":0,"started_at_ms":0,"finished_at_ms":0,"summary":""}"#, + ) + .expect("deserialize without details"); + assert!(back.details.is_null()); +} + +#[test] +fn cloning_a_usage_handle_shares_one_tally() { + let handle = ComposioUsageHandle::default(); + let clone = handle.clone(); + { + let mut usage = clone.lock().expect("usage lock"); + usage.actions_called += 2; + usage.cost_usd += 0.5; + } + let usage = handle.lock().expect("usage lock"); + assert_eq!(usage.actions_called, 2); + assert_eq!(usage.cost_usd, 0.5); +} + +#[test] +fn a_usage_tally_starts_at_zero() { + let usage = ComposioUsage::default(); + assert_eq!(usage.actions_called, 0); + assert_eq!(usage.cost_usd, 0.0); +} diff --git a/crates/tinymemory-bus/src/composio/scopes.rs b/crates/tinymemory-bus/src/composio/scopes.rs new file mode 100644 index 00000000..09714e9d --- /dev/null +++ b/crates/tinymemory-bus/src/composio/scopes.rs @@ -0,0 +1,246 @@ +//! How invasive an action is, and how much of that the user has agreed to. +//! +//! Composio publishes sixty-odd actions per toolkit and most of them are noise +//! for an agent's planning loop, so each provider hand-curates a slice of +//! [`CuratedTool`] entries that pares the surface down and tags every action +//! with a [`ToolScope`]. The user's [`UserScopePref`] then gates execution per +//! toolkit: reads and writes on by default, destructive and permission-changing +//! actions off until explicitly opted into. +//! +//! # Why the classification is here and the catalogs are not +//! +//! Two different consumers ask the same question from opposite sides of the +//! module boundary. The host asks it when it renders the integrations panel and +//! when it filters the agent's visible tool list; the sync pipelines ask it +//! inside the module before firing an action. Both have to reach the same +//! verdict, which makes [`ToolScope`], [`UserScopePref::allows`] and the +//! heuristic fallback [`classify_unknown`] shared vocabulary rather than either +//! side's private policy. +//! +//! The catalogs themselves — thousands of `&'static str` action slugs across +//! thirty toolkits — stay in the engine crate. They are provider data, they +//! change whenever a provider does, and nothing about them has to cross a +//! frame: what crosses is the verdict. +//! +//! Reading and writing a preference is likewise the engine crate's; this module +//! defines what a preference *is*, not where it is stored. + +use serde::{Deserialize, Serialize}; + +/// Classification of how invasive an action is. +/// +/// Used both to filter the agent's visible tool list and to enforce per-user +/// scope preferences at execution time. The serde form is lowercase and is +/// persisted inside a [`UserScopePref`] key/value row. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(rename_all = "lowercase")] +pub enum ToolScope { + /// Pure reads — `GET` / `FETCH` / `LIST` / `SEARCH` / `GET_PROFILE`. + Read, + /// Side-effectful actions that create or mutate user data — + /// `SEND` / `CREATE` / `UPDATE` / `REPLY` / `APPEND`. + Write, + /// Destructive or permission-changing actions — `DELETE` / `TRASH` / + /// `REMOVE` / `MODIFY_LABELS` / `SHARE`. + Admin, +} + +impl ToolScope { + /// Stable lowercase tag, matching the serde representation. + pub fn as_str(self) -> &'static str { + match self { + ToolScope::Read => "read", + ToolScope::Write => "write", + ToolScope::Admin => "admin", + } + } +} + +/// One curated entry in a provider's tool catalog. +/// +/// `slug` is the Composio action slug as the toolkit listing returns it, e.g. +/// `"GMAIL_SEND_EMAIL"`. `scope` controls whether the action is gated by the +/// user's read / write / admin preference. +/// +/// Deliberately `&'static str` and `Copy`: catalogs are `const` slices built at +/// compile time, and giving this owned `String` fields would turn thirty static +/// tables into thirty heap allocations at startup for no gain. +#[derive(Debug, Clone, Copy)] +pub struct CuratedTool { + /// The Composio action slug, e.g. `"GMAIL_SEND_EMAIL"`. + pub slug: &'static str, + /// How invasive the action is, for preference gating. + pub scope: ToolScope, +} + +/// Per-toolkit scope preference. +/// +/// Defaults are `read = true`, `write = true`, `admin = false` — the agent can +/// use a connected integration productively out of the box, but destructive and +/// permission-changing actions require an explicit opt-in. +/// +/// The two `default_true` helpers matter for decoding: a row written before a +/// field existed must not read back as "denied", which is what a bare +/// `#[serde(default)]` would give a `bool`. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +pub struct UserScopePref { + /// Whether the agent may call [`ToolScope::Read`] actions. + #[serde(default = "default_true")] + pub read: bool, + /// Whether the agent may call [`ToolScope::Write`] actions. + #[serde(default = "default_true")] + pub write: bool, + /// Whether the agent may call [`ToolScope::Admin`] actions. + #[serde(default)] + pub admin: bool, +} + +fn default_true() -> bool { + true +} + +impl Default for UserScopePref { + fn default() -> Self { + Self { + read: true, + write: true, + admin: false, + } + } +} + +impl UserScopePref { + /// Whether the given scope is enabled in this preference. + pub fn allows(&self, scope: ToolScope) -> bool { + match scope { + ToolScope::Read => self.read, + ToolScope::Write => self.write, + ToolScope::Admin => self.admin, + } + } +} + +/// Heuristic fallback for gating a tool that is not in any provider's curated +/// list. +/// +/// Prefer the curated classification when one exists; only reach for this when +/// a toolkit has no catalog or the catalog does not mention the slug. Admin +/// verbs are checked first so `MODIFY_LABELS` does not slip into the write +/// bucket on the `UPDATE` substring rule — the ordering is the whole point of +/// the function and not an implementation detail. +pub fn classify_unknown(slug: &str) -> ToolScope { + let upper = slug.to_ascii_uppercase(); + const ADMIN: &[&str] = &[ + "DELETE", + "TRASH", + "REMOVE", + "MODIFY_LABELS", + "SHARE", + "REVOKE", + "DESTROY", + ]; + const WRITE: &[&str] = &[ + "SEND", "CREATE", "UPDATE", "REPLY", "APPEND", "INSERT", "ADD", "POST", "PATCH", "WRITE", + "DRAFT", + ]; + if ADMIN.iter().any(|kw| upper.contains(kw)) { + return ToolScope::Admin; + } + if WRITE.iter().any(|kw| upper.contains(kw)) { + return ToolScope::Write; + } + ToolScope::Read +} + +/// Look up a slug inside a curated catalog, case-insensitively. +pub fn find_curated<'a>(catalog: &'a [CuratedTool], slug: &str) -> Option<&'a CuratedTool> { + catalog.iter().find(|t| t.slug.eq_ignore_ascii_case(slug)) +} + +/// Extract the toolkit slug from a Composio action slug. +/// +/// Most action slugs follow `__…` — `GMAIL_SEND_EMAIL` yields +/// `gmail`. A few toolkit identifiers contain underscores themselves, so those +/// need known-prefix handling or a connected-toolkit check silently drops every +/// action for them (`ZOHO_MAIL_*` would resolve to the non-existent `zoho`). +/// +/// Returns `None` only for an empty or whitespace-only slug. +pub fn toolkit_from_slug(slug: &str) -> Option { + let trimmed = slug.trim(); + if trimmed.is_empty() { + return None; + } + const MULTI_SEGMENT_TOOLKIT_PREFIXES: &[(&str, &str)] = &[ + ("MICROSOFT_TEAMS_", "microsoft_teams"), + ("ONE_DRIVE_", "one_drive"), + ("ZOHO_MAIL_", "zoho_mail"), + ]; + let upper = trimmed.to_ascii_uppercase(); + for (prefix, toolkit) in MULTI_SEGMENT_TOOLKIT_PREFIXES { + if upper.starts_with(prefix) { + return Some((*toolkit).to_string()); + } + } + let prefix = trimmed.split('_').next()?; + if prefix.is_empty() { + None + } else { + Some(prefix.to_ascii_lowercase()) + } +} + +/// Every toolkit slug that has a curated, agent-ready catalog. +/// +/// This is the source of truth behind the "preview / agent integration coming +/// soon" badge: a connected toolkit whose slug is *not* in this list can be +/// authorized but has no curated tool surface, so the agent cannot use it +/// productively and the UI should say so rather than offering it. +/// +/// Returned sorted, so the RPC response is stable across builds. +/// +/// The list is here rather than with the catalogs it names because the *host* +/// renders the badge. Keeping it beside the catalogs would mean the host asking +/// the module a question — "is this toolkit worth showing?" — that has no +/// user-visible state behind it and would answer identically forever. +pub fn agent_ready_toolkits() -> Vec<&'static str> { + let mut slugs: Vec<&'static str> = vec![ + // Native providers. + "gmail", + "notion", + "github", + // Catalog-only toolkits. + "slack", + "discord", + "googlecalendar", + "googledrive", + "googledocs", + "googlesheets", + "outlook", + "microsoft_teams", + "linear", + "jira", + "trello", + "asana", + "dropbox", + "twitter", + "spotify", + "telegram", + "whatsapp", + "shopify", + "stripe", + "hubspot", + "salesforce", + "airtable", + "figma", + "youtube", + "one_drive", + "excel", + "todoist", + ]; + slugs.sort_unstable(); + slugs +} + +#[cfg(test)] +#[path = "scopes_tests.rs"] +mod tests; diff --git a/crates/tinymemory-bus/src/composio/scopes_tests.rs b/crates/tinymemory-bus/src/composio/scopes_tests.rs new file mode 100644 index 00000000..c1b038d8 --- /dev/null +++ b/crates/tinymemory-bus/src/composio/scopes_tests.rs @@ -0,0 +1,178 @@ +//! Tests for the action-scope classification and the per-toolkit preference. +//! +//! The storage half — reading and writing a [`super::UserScopePref`] through +//! the key/value seam — is tested in the engine crate next to the code that +//! performs it. What is pinned here is everything two separately compiled +//! processes have to agree on: the verb precedence in +//! [`super::classify_unknown`], the multi-segment toolkit prefixes, the +//! persisted preference shape, and the default that decides what a brand-new +//! connection is allowed to do. + +#![allow(clippy::expect_used, clippy::unwrap_used, clippy::panic)] + +use super::{ + agent_ready_toolkits, classify_unknown, find_curated, toolkit_from_slug, CuratedTool, + ToolScope, UserScopePref, +}; + +#[test] +fn destructive_verbs_classify_as_admin() { + assert_eq!(classify_unknown("GMAIL_DELETE_EMAIL"), ToolScope::Admin); + assert_eq!(classify_unknown("GMAIL_TRASH_EMAIL"), ToolScope::Admin); + assert_eq!(classify_unknown("GMAIL_MODIFY_LABELS"), ToolScope::Admin); + assert_eq!(classify_unknown("DRIVE_SHARE_FILE"), ToolScope::Admin); +} + +#[test] +fn mutating_verbs_classify_as_write() { + assert_eq!(classify_unknown("GMAIL_SEND_EMAIL"), ToolScope::Write); + assert_eq!(classify_unknown("NOTION_CREATE_PAGE"), ToolScope::Write); + assert_eq!(classify_unknown("NOTION_UPDATE_PAGE"), ToolScope::Write); +} + +#[test] +fn anything_else_classifies_as_read() { + assert_eq!(classify_unknown("GMAIL_FETCH_EMAILS"), ToolScope::Read); + assert_eq!(classify_unknown("NOTION_SEARCH"), ToolScope::Read); + assert_eq!(classify_unknown("GMAIL_GET_PROFILE"), ToolScope::Read); +} + +#[test] +fn admin_verbs_are_checked_before_write_verbs() { + // `DELETE_DRAFT` contains `DRAFT`, a write verb. If the two lists were + // checked in the other order this would gate as a write and a destructive + // action would run under a write-only preference. + assert_eq!(classify_unknown("GMAIL_DELETE_DRAFT"), ToolScope::Admin); +} + +#[test] +fn classification_ignores_slug_casing() { + assert_eq!(classify_unknown("gmail_delete_email"), ToolScope::Admin); + assert_eq!(classify_unknown("gmail_send_email"), ToolScope::Write); +} + +#[test] +fn a_toolkit_slug_is_the_lowercased_first_segment() { + assert_eq!( + toolkit_from_slug("GMAIL_SEND_EMAIL").as_deref(), + Some("gmail") + ); + assert_eq!( + toolkit_from_slug("NOTION_FETCH_DATA").as_deref(), + Some("notion") + ); + assert_eq!( + toolkit_from_slug("noUnderscore").as_deref(), + Some("nounderscore") + ); +} + +#[test] +fn an_empty_slug_names_no_toolkit() { + assert_eq!(toolkit_from_slug(""), None); + assert_eq!(toolkit_from_slug(" "), None); +} + +#[test] +fn multi_segment_toolkits_keep_their_whole_prefix() { + // Without these three, `ZOHO_MAIL_*` resolves to `zoho`, matches no + // connected toolkit, and every action for it is silently dropped. + assert_eq!( + toolkit_from_slug("ZOHO_MAIL_SEND_EMAIL").as_deref(), + Some("zoho_mail") + ); + assert_eq!( + toolkit_from_slug("ONE_DRIVE_GET_FILE").as_deref(), + Some("one_drive") + ); + assert_eq!( + toolkit_from_slug("MICROSOFT_TEAMS_SEND_MESSAGE").as_deref(), + Some("microsoft_teams") + ); +} + +#[test] +fn a_curated_lookup_ignores_casing_and_reports_a_miss() { + let catalog = &[CuratedTool { + slug: "GMAIL_SEND_EMAIL", + scope: ToolScope::Write, + }]; + assert!(find_curated(catalog, "gmail_send_email").is_some()); + assert!(find_curated(catalog, "GMAIL_SEND_EMAIL").is_some()); + assert!(find_curated(catalog, "GMAIL_DELETE_EMAIL").is_none()); +} + +#[test] +fn every_tool_scope_tag_matches_its_serde_form() { + for scope in [ToolScope::Read, ToolScope::Write, ToolScope::Admin] { + let json = serde_json::to_string(&scope).expect("serialize"); + assert_eq!(json, format!("\"{}\"", scope.as_str())); + } +} + +#[test] +fn a_new_connection_may_read_and_write_but_not_administer() { + let pref = UserScopePref::default(); + assert!(pref.read); + assert!(pref.write); + assert!(!pref.admin); +} + +#[test] +fn allows_answers_per_scope() { + let pref = UserScopePref { + read: true, + write: false, + admin: false, + }; + assert!(pref.allows(ToolScope::Read)); + assert!(!pref.allows(ToolScope::Write)); + assert!(!pref.allows(ToolScope::Admin)); +} + +#[test] +fn a_preference_round_trips() { + let pref = UserScopePref { + read: true, + write: true, + admin: true, + }; + let value = serde_json::to_value(pref).expect("serialize"); + let back: UserScopePref = serde_json::from_value(value).expect("deserialize"); + assert_eq!(pref, back); +} + +#[test] +fn a_row_missing_read_and_write_decodes_as_permitted_not_denied() { + // A stored row written before a field existed must not read back as a + // denial: `#[serde(default)]` on a `bool` would silently revoke access the + // user never revoked. + let stored = serde_json::json!({ "admin": true }); + let pref: UserScopePref = serde_json::from_value(stored).expect("deserialize"); + assert!(pref.read); + assert!(pref.write); + assert!(pref.admin); +} + +#[test] +fn the_agent_ready_list_is_sorted_and_free_of_duplicates() { + // The RPC response has to be stable across builds, and the badge logic is a + // membership test — a duplicate would be invisible there and confusing in + // the panel. + let slugs = agent_ready_toolkits(); + let mut sorted = slugs.clone(); + sorted.sort_unstable(); + assert_eq!(slugs, sorted, "the list must come back sorted"); + + let mut deduped = sorted.clone(); + deduped.dedup(); + assert_eq!(deduped.len(), slugs.len(), "the list has a duplicate slug"); +} + +#[test] +fn the_agent_ready_list_names_the_native_providers() { + let slugs = agent_ready_toolkits(); + for native in ["gmail", "notion", "github", "linear"] { + assert!(slugs.contains(&native), "{native} is missing from the list"); + } +} diff --git a/crates/tinymemory-bus/src/composio/state.rs b/crates/tinymemory-bus/src/composio/state.rs new file mode 100644 index 00000000..08057632 --- /dev/null +++ b/crates/tinymemory-bus/src/composio/state.rs @@ -0,0 +1,275 @@ +//! What a connection remembers between sync runs: a cursor, a dedup set, and +//! a daily request budget. +//! +//! One [`SyncState`] per `(toolkit, connection)` pair, persisted as JSON under +//! [`STATE_NAMESPACE`] in whatever key/value store the driver provides. It is +//! the reason a second Gmail sync does not re-ingest the first sync's messages +//! and the reason a runaway pipeline stops at five hundred requests instead of +//! exhausting a user's quota. +//! +//! # Why this is contract vocabulary rather than engine state +//! +//! Both sides read it. The module advances the cursor and records requests; the +//! host shows "synced 4 minutes ago, 312 of 500 requests used today" and, on +//! disconnect, walks the dedup set to decide what to forget. A structural twin +//! on the host side would decode today and diverge the first time a field was +//! added — and this shape is *persisted*, so a divergence is not a wire bug +//! that reconnects away, it is a stranded cursor and a re-ingested inbox. +//! +//! # What is not here +//! +//! `SyncStateStore`, and the `load` / `save` that use it. This crate publishes +//! no traits and holds no I/O (see [`crate`]); the engine crate carries the +//! trait and offers the two methods as an extension trait over the type defined +//! here. Everything below is arithmetic on a struct. +//! +//! # Durability +//! +//! [`STATE_NAMESPACE`] and the serde field names are a compatibility surface: +//! changing the namespace strands every cursor, and renaming a field silently +//! resets it to its default on the next load. The engine keeps a +//! structurally-identical copy for its own internal pipelines, persisting under +//! the same namespace with the same shape, until those pipelines retire — the +//! pin tests below are what hold the two together. + +use std::collections::{HashMap, HashSet}; + +use chrono::Utc; +use serde::{Deserialize, Serialize}; + +/// The key/value namespace every persisted sync cursor lives under. +/// +/// An alias for [`STATE_NAMESPACE`], kept because callers reach for one name or +/// the other depending on whether they are writing state or cleaning it up. +/// Durable: changing it strands every cursor. +pub const KV_NAMESPACE: &str = STATE_NAMESPACE; + +/// Requests one connection may spend in a day before its budget is exhausted. +/// +/// A backstop against a paginating pipeline that never terminates, not a +/// billing limit — the provider-reported cost is tallied separately. +pub const DEFAULT_DAILY_REQUEST_LIMIT: u32 = 500; + +/// The key/value namespace every persisted sync cursor lives under. +/// +/// Durable: changing it strands every cursor. +pub const STATE_NAMESPACE: &str = "composio-sync-state"; + +/// A per-connection, per-day request allowance. +/// +/// `date` is the UTC day the counter belongs to. Every accessor compares it +/// against today and treats a stale date as a fresh allowance, so a state +/// loaded from yesterday reports a full budget without anyone having to reset +/// it — which is what makes a missed midnight rollover a non-event. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct DailyBudget { + /// The UTC day this counter belongs to, as `YYYY-MM-DD`. + pub date: String, + /// Requests spent on [`date`](Self::date). + pub requests_used: u32, + /// Allowance for one day; defaults to [`DEFAULT_DAILY_REQUEST_LIMIT`]. + pub limit: u32, +} + +impl Default for DailyBudget { + fn default() -> Self { + Self { + date: today(), + requests_used: 0, + limit: DEFAULT_DAILY_REQUEST_LIMIT, + } + } +} + +impl DailyBudget { + /// Requests still available today. + /// + /// A counter from an earlier day reports the full limit rather than its + /// stale remainder: the rollover happens on read, so nothing has to run at + /// midnight for a budget to refresh. + pub fn remaining(&self) -> u32 { + if self.date != today() { + self.limit + } else { + self.limit.saturating_sub(self.requests_used) + } + } + + /// Whether today's allowance is spent. + pub fn is_exhausted(&self) -> bool { + self.remaining() == 0 + } + + /// Charge `count` requests against today's allowance, rolling the counter + /// over first if it belongs to an earlier day. + pub fn record_requests(&mut self, count: u32) { + self.roll_over_if_stale(); + self.requests_used = self.requests_used.saturating_add(count); + } + + /// Reset the counter when it belongs to an earlier day. + /// + /// Every accessor already rolls over lazily, so this exists for the one + /// caller that wants the *stored* value normalised rather than the answer: + /// a state just loaded from yesterday, so that what is written back is + /// today's row and not a stale one that later reads have to keep + /// compensating for. + pub fn roll_over_if_stale(&mut self) { + let today = today(); + if self.date != today { + self.date = today; + self.requests_used = 0; + } + } + + /// Charge a single request. Shorthand for [`record_requests`](Self::record_requests). + pub fn record_request(&mut self) { + self.record_requests(1); + } +} + +/// Everything one `(toolkit, connection)` pair carries between sync runs. +/// +/// The two `#[serde(skip)]` fields are per-run counters rather than state: they +/// exist so a finished run can report what it spent, and persisting them would +/// make the tally cumulative, which is not what any caller reads it as. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct SyncState { + /// Composio toolkit slug, e.g. `"gmail"`. + pub toolkit: String, + /// The connection this state belongs to. + pub connection_id: String, + /// Provider-native pagination cursor, when the provider issues one. + #[serde(default)] + pub cursor: Option, + /// Upstream item ids already ingested, so a re-run does not duplicate them. + #[serde(default)] + pub synced_ids: HashSet, + /// Item id to version string, for providers whose items can be edited after + /// they were first seen. + #[serde(default)] + pub item_versions: HashMap, + /// Today's request allowance. + #[serde(default)] + pub daily_budget: DailyBudget, + /// Newest item id seen, for providers that page newest-first. + #[serde(default)] + pub last_seen_id: Option, + /// When the last run finished, epoch milliseconds. + #[serde(default)] + pub last_sync_at_ms: Option, + /// Requests spent by the *current* run. Not persisted. + #[serde(skip)] + pub run_requests: u32, + /// Provider-reported cost accumulated by the *current* run. Not persisted. + #[serde(skip)] + pub run_provider_cost_usd: f64, +} + +impl SyncState { + /// A fresh state for a connection that has never synced. + pub fn new(toolkit: impl Into, connection_id: impl Into) -> Self { + Self { + toolkit: toolkit.into(), + connection_id: connection_id.into(), + cursor: None, + synced_ids: HashSet::new(), + item_versions: HashMap::new(), + daily_budget: DailyBudget::default(), + last_seen_id: None, + last_sync_at_ms: None, + run_requests: 0, + run_provider_cost_usd: 0.0, + } + } + + /// The key/value key a state is stored under, within [`STATE_NAMESPACE`]. + /// + /// Durable, like the namespace: a different separator strands every cursor. + pub fn key(toolkit: &str, connection_id: &str) -> String { + format!("{toolkit}:{connection_id}") + } + + /// Whether this item has already been ingested. + pub fn is_synced(&self, id: &str) -> bool { + self.synced_ids.contains(id) + } + + /// Record an item as ingested. + pub fn mark_synced(&mut self, id: impl Into) { + self.synced_ids.insert(id.into()); + } + + /// Move the pagination cursor forward. + pub fn advance_cursor(&mut self, cursor: impl Into) { + self.cursor = Some(cursor.into()); + } + + /// Record the newest item id this run saw. + pub fn set_last_seen_id(&mut self, id: impl Into) { + self.last_seen_id = Some(id.into()); + } + + /// Stamp when the run finished, epoch milliseconds. + pub fn set_last_sync_at_ms(&mut self, timestamp_ms: u64) { + self.last_sync_at_ms = Some(timestamp_ms); + } + + /// Whether today's request allowance is spent. + pub fn budget_exhausted(&self) -> bool { + self.daily_budget.is_exhausted() + } + + /// Requests still available today. + pub fn budget_remaining(&self) -> u32 { + self.daily_budget.remaining() + } + + /// Charge `count` requests against both the daily allowance and this run's + /// counter. + pub fn record_requests(&mut self, count: u32) { + self.daily_budget.record_requests(count); + self.run_requests = self.run_requests.saturating_add(count); + } + + /// Record one completed Composio action: its request attempts and the + /// provider-reported cost. + /// + /// `attempts` is floored at one — an action that reached the provider spent + /// at least one request however the caller counted its retries. A cost that + /// is negative, infinite or `NaN` is discarded rather than propagated into + /// a total someone reads as money. + pub fn record_action(&mut self, attempts: u32, cost_usd: f64) { + self.record_requests(attempts.max(1)); + if cost_usd.is_finite() && cost_usd > 0.0 { + self.run_provider_cost_usd += cost_usd; + } + } +} + +/// First non-empty string found at any of `paths` — dot-separated — in `item`. +/// +/// Providers disagree about where an item's stable id lives (`id`, `messageId`, +/// `data.id`, …), so a pipeline hands this the candidates in priority order and +/// takes the first that is actually populated. Whitespace-only values count as +/// absent: an id of `" "` dedupes nothing and would poison the synced set. +pub fn extract_item_id(item: &serde_json::Value, paths: &[&str]) -> Option { + paths.iter().find_map(|path| { + let value = path + .split('.') + .try_fold(item, |current, segment| current.get(segment))?; + value + .as_str() + .map(str::trim) + .filter(|value| !value.is_empty()) + .map(str::to_owned) + }) +} + +fn today() -> String { + Utc::now().format("%Y-%m-%d").to_string() +} + +#[cfg(test)] +#[path = "state_tests.rs"] +mod tests; diff --git a/crates/tinymemory-bus/src/composio/state_tests.rs b/crates/tinymemory-bus/src/composio/state_tests.rs new file mode 100644 index 00000000..efecb95d --- /dev/null +++ b/crates/tinymemory-bus/src/composio/state_tests.rs @@ -0,0 +1,194 @@ +//! Tests for the persisted sync-state shape. +//! +//! The load/save round-trip through a key/value store is tested in the engine +//! crate, next to the `SyncStateStore` trait that performs it. What is pinned +//! here is what a migration would have to care about: the namespace, the +//! serialised field set, and the day-rollover arithmetic that decides whether a +//! connection is allowed to make another request. + +#![allow(clippy::expect_used, clippy::unwrap_used, clippy::panic)] + +use super::{ + extract_item_id, DailyBudget, SyncState, DEFAULT_DAILY_REQUEST_LIMIT, KV_NAMESPACE, + STATE_NAMESPACE, +}; + +/// The namespace is durable: every persisted Composio sync cursor lives under +/// this string, so a change strands all of them. The engine's own copy of this +/// type must agree — failing here means a coordinated migration, never a local +/// edit. +#[test] +fn the_state_namespace_is_pinned() { + assert_eq!( + STATE_NAMESPACE, "composio-sync-state", + "the Composio sync-state namespace changed; every persisted cursor is \ + stored under the old value and needs migrating" + ); + assert_eq!(KV_NAMESPACE, STATE_NAMESPACE); +} + +/// The serialised shape is persisted and is also what the engine's copy writes. +/// Pinned so the two cannot drift silently. +#[test] +fn the_state_wire_shape_is_pinned() { + let mut state = SyncState::new("gmail", "conn-1"); + state.advance_cursor("c2"); + state.mark_synced("m1"); + state.item_versions.insert("m1".into(), "v1".into()); + state.set_last_seen_id("m1"); + state.set_last_sync_at_ms(1_000); + // Written directly rather than through `record_requests`, which would roll + // the stale date forward to today and make the expectation clock-dependent. + state.daily_budget.date = "2026-01-02".into(); + state.daily_budget.requests_used = 3; + + let value = serde_json::to_value(&state).expect("serialize"); + assert_eq!( + value, + serde_json::json!({ + "toolkit": "gmail", + "connection_id": "conn-1", + "cursor": "c2", + "synced_ids": ["m1"], + "item_versions": {"m1": "v1"}, + "daily_budget": {"date": "2026-01-02", "requests_used": 3, "limit": 500}, + "last_seen_id": "m1", + "last_sync_at_ms": 1000 + }) + ); +} + +#[test] +fn per_run_counters_never_reach_the_wire() { + // A persisted tally would make every subsequent run report the sum of all + // the runs before it, which is not what the audit log reads it as. + let mut state = SyncState::new("gmail", "conn-1"); + state.record_action(2, 0.25); + assert_eq!(state.run_requests, 2); + assert_eq!(state.run_provider_cost_usd, 0.25); + + let value = serde_json::to_value(&state).expect("serialize"); + let object = value.as_object().expect("state serialises as an object"); + assert!(!object.contains_key("run_requests")); + assert!(!object.contains_key("run_provider_cost_usd")); +} + +#[test] +fn the_key_is_toolkit_then_connection() { + assert_eq!(SyncState::key("gmail", "conn-1"), "gmail:conn-1"); +} + +#[test] +fn a_fresh_state_has_synced_nothing_and_spent_nothing() { + let state = SyncState::new("slack", "conn-2"); + assert!(state.cursor.is_none()); + assert!(!state.is_synced("anything")); + assert!(!state.budget_exhausted()); + assert_eq!(state.budget_remaining(), DEFAULT_DAILY_REQUEST_LIMIT); + assert_eq!(state.run_requests, 0); +} + +#[test] +fn a_state_missing_every_optional_field_still_decodes() { + // Only `toolkit` and `connection_id` are required; a row written by an + // older peer that never knew about `item_versions` must load rather than + // fail the whole connection. + let stored = serde_json::json!({ "toolkit": "gmail", "connection_id": "c" }); + let state: SyncState = serde_json::from_value(stored).expect("deserialize"); + assert!(state.cursor.is_none()); + assert!(state.synced_ids.is_empty()); + assert!(state.item_versions.is_empty()); + assert_eq!(state.daily_budget.limit, DEFAULT_DAILY_REQUEST_LIMIT); +} + +#[test] +fn a_stale_budget_reports_full_and_resets_on_the_next_charge() { + let mut budget = DailyBudget { + date: "2000-01-01".into(), + requests_used: 499, + limit: 500, + }; + assert_eq!(budget.remaining(), 500); + budget.record_requests(1); + assert_eq!(budget.requests_used, 1); + assert_eq!(budget.remaining(), 499); +} + +#[test] +fn an_exhausted_budget_reports_zero_remaining() { + let mut budget = DailyBudget { + limit: 2, + ..DailyBudget::default() + }; + budget.record_request(); + assert!(!budget.is_exhausted()); + budget.record_request(); + assert!(budget.is_exhausted()); + assert_eq!(budget.remaining(), 0); +} + +#[test] +fn charging_past_the_limit_saturates_rather_than_wrapping() { + let mut budget = DailyBudget { + limit: 1, + ..DailyBudget::default() + }; + budget.record_requests(u32::MAX); + budget.record_requests(10); + assert_eq!(budget.requests_used, u32::MAX); + assert_eq!(budget.remaining(), 0); +} + +#[test] +fn an_action_always_costs_at_least_one_request() { + let mut state = SyncState::new("gmail", "conn-1"); + state.record_action(0, 0.0); + assert_eq!(state.run_requests, 1); + assert_eq!(state.daily_budget.requests_used, 1); +} + +#[test] +fn a_nonsense_action_cost_is_discarded_rather_than_totalled() { + let mut state = SyncState::new("gmail", "conn-1"); + state.record_action(1, f64::NAN); + state.record_action(1, f64::INFINITY); + state.record_action(1, -5.0); + assert_eq!(state.run_provider_cost_usd, 0.0); + state.record_action(1, 0.5); + assert_eq!(state.run_provider_cost_usd, 0.5); +} + +#[test] +fn an_item_id_is_taken_from_the_first_populated_path() { + let item = serde_json::json!({ "data": { "id": "inner" }, "messageId": "outer" }); + assert_eq!( + extract_item_id(&item, &["missing", "data.id", "messageId"]).as_deref(), + Some("inner") + ); +} + +#[test] +fn a_blank_item_id_counts_as_absent() { + // An id of `" "` dedupes nothing and would poison the synced set, so it + // must not win over a later path that is actually populated. + let item = serde_json::json!({ "id": " ", "messageId": "m-1" }); + assert_eq!( + extract_item_id(&item, &["id", "messageId"]).as_deref(), + Some("m-1") + ); +} + +#[test] +fn a_non_string_item_id_is_skipped() { + let item = serde_json::json!({ "id": 7, "messageId": "m-1" }); + assert_eq!( + extract_item_id(&item, &["id", "messageId"]).as_deref(), + Some("m-1") + ); +} + +#[test] +fn no_matching_path_yields_none() { + let item = serde_json::json!({ "id": "x" }); + assert_eq!(extract_item_id(&item, &["nope", "also.nope"]), None); +} diff --git a/crates/tinymemory-bus/src/composio/tasks.rs b/crates/tinymemory-bus/src/composio/tasks.rs new file mode 100644 index 00000000..74ec2c67 --- /dev/null +++ b/crates/tinymemory-bus/src/composio/tasks.rs @@ -0,0 +1,206 @@ +//! The provider-agnostic work-item envelope: what a task fetch asks for, and +//! what it hands back. +//! +//! This is the second of the two things a Composio provider does. `sync` +//! persists upstream items into the memory store as passive context; +//! `fetch_tasks` *returns* [`NormalizedTask`]s so the host can enrich them and +//! route them onto the agent's todo board. Every native task provider (GitHub, +//! Notion, Linear, ClickUp) maps its upstream payload into this one envelope, +//! which is why the envelope — and not the payloads — is what crosses the bus. +//! +//! # A note on the wire casing +//! +//! [`NormalizedTask`], [`TaskContainer`] and [`TaskFetchFilter`] serialise +//! `camelCase`; the enums serialise `snake_case`. That is not an oversight: the +//! structs are read by the task-source UI, which is TypeScript, while the enum +//! tags are also written into a card's `source_metadata` and compared as +//! strings on the Rust side. Both forms are persisted — treat every field name +//! and every variant name as a compatibility surface. + +use serde::{Deserialize, Serialize}; + +/// What kind of work an ingested task implies. +/// +/// GitHub's issues-and-pull-requests search returns both shapes and the job +/// differs fundamentally — *resolve* an issue versus *review* a pull request — +/// so providers tag each task and the enrichment phrases the objective and the +/// agent prompt accordingly. Providers that do not distinguish (Notion, Linear, +/// ClickUp) leave this [`Generic`](Self::Generic). +#[derive(Debug, Clone, Copy, Default, Serialize, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "snake_case")] +pub enum TaskKind { + /// No issue/pull-request distinction — the default for non-code providers. + #[default] + Generic, + /// A tracker issue: the job is to resolve or implement it. + Issue, + /// A pull request: the job is to review it — read the diff, give feedback. + PullRequest, +} + +impl TaskKind { + /// Stable lowercase tag, mirrored into the card's `source_metadata`. + pub fn as_str(&self) -> &'static str { + match self { + TaskKind::Generic => "generic", + TaskKind::Issue => "issue", + TaskKind::PullRequest => "pull_request", + } + } +} + +/// How the GitHub task-source fetch reaches GitHub. +/// +/// Shipped desktop users connect GitHub through Composio OAuth — no `gh` on +/// `PATH`, no `GITHUB_TOKEN` — while local development and self-hosted setups +/// often have the reverse. [`Auto`](Self::Auto) does the right thing for both; +/// the other two force a path when the user wants one. +#[derive(Debug, Clone, Copy, Default, Serialize, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "snake_case")] +pub enum GithubFetchMode { + /// Try the connected Composio account first; fall back to local `gh` or + /// REST only when Composio is unavailable. + /// + /// The safe default: no regression for shipped users, still a true fallback + /// for local and development setups. + #[default] + Auto, + /// Force the connected Composio account — the classic shipped-app path. + Composio, + /// Force the local `gh` CLI or REST with a `GH_TOKEN` / `GITHUB_TOKEN` + /// environment token. + Local, +} + +/// A provider-agnostic, structured work item returned by a task fetch. +/// +/// `source_id` is left empty by providers and stamped by the host's task-source +/// pipeline with the originating source id — a provider has no knowledge of +/// which configured source asked for the fetch. +#[derive(Debug, Clone, Default, Serialize, Deserialize, PartialEq)] +#[serde(rename_all = "camelCase")] +pub struct NormalizedTask { + /// The upstream provider's stable id for the item — issue, task or page id. + pub external_id: String, + /// The task source that produced this task. Empty until the pipeline + /// stamps it. + #[serde(default)] + pub source_id: String, + /// Toolkit slug, e.g. `"github"`. + pub provider: String, + /// Whether this task is an issue, a pull request, or undifferentiated. + /// + /// Drives intent-aware objective and prompt phrasing during enrichment. + #[serde(default)] + pub kind: TaskKind, + /// Human-readable title, as the provider spells it. + pub title: String, + /// Body text or description, when the provider returns one. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub body: Option, + /// Canonical web URL for the item. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub url: Option, + /// Provider-native status string, e.g. `"open"` or `"todo"`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub status: Option, + /// Whoever the item is assigned to upstream. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub assignee: Option, + /// Due date as an ISO-8601 string, when the provider exposes one. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub due: Option, + /// Provider-native labels or tags. + #[serde(default)] + pub labels: Vec, + /// Provider-native priority string. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub priority: Option, + /// Last-updated ISO-8601 timestamp — used for cursor advancement and + /// edit-aware dedup (`{external_id}@{updated_at}`). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub updated_at: Option, + /// The raw upstream payload, retained for enrichment and debugging. + #[serde(default)] + pub raw: serde_json::Value, +} + +/// A selectable upstream task container — a board, database or list. +/// +/// Populates a picker so the user chooses from a list instead of pasting a raw +/// id. Today this is a Notion database; later a Linear team or a ClickUp list. +/// Surfaced to the task-source UI as `{ id, title }`. +#[derive(Debug, Clone, Default, Serialize, Deserialize, PartialEq)] +#[serde(rename_all = "camelCase")] +pub struct TaskContainer { + /// Provider-native id, e.g. a Notion database id, used as the filter id. + pub id: String, + /// Human-readable label for the picker. + pub title: String, +} + +/// Provider-agnostic filter passed into a task fetch. +/// +/// The host builds this from a user-configured, per-provider filter spec. Each +/// provider reads only the fields that apply to it — GitHub reads `repo` and +/// `labels`, Notion reads `database_id`, Linear and ClickUp read `team_id` — +/// and ignores the rest. [`extra`](Self::extra) is a free-form escape hatch +/// surfaced in the UI for advanced provider-native query fragments. +#[derive(Debug, Clone, Default, Serialize, Deserialize, PartialEq)] +#[serde(rename_all = "camelCase")] +pub struct TaskFetchFilter { + /// Scope to items assigned to — or involving — the authenticated user. + #[serde(default)] + pub assignee_is_me: bool, + /// GitHub fetch-path selector. Defaults to [`GithubFetchMode::Auto`]. + #[serde(default)] + pub github_fetch_mode: GithubFetchMode, + /// GitHub `owner/name` repository scope. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub repo: Option, + /// GitHub label filter. + #[serde(default)] + pub labels: Vec, + /// Issue or task state filter, e.g. `"open"` or `"todo"`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub state: Option, + /// Notion database — board — id. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub database_id: Option, + /// Notion status property filter. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub status: Option, + /// Linear or ClickUp team — workspace — id. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub team_id: Option, + /// ClickUp list id. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub list_id: Option, + /// Free-form provider-native filter fragment, for advanced users. + #[serde(default)] + pub extra: serde_json::Value, + /// Hard cap on how many tasks a single fetch returns. `0` means "unset"; + /// see [`effective_max`](Self::effective_max). + #[serde(default)] + pub max: u32, +} + +impl TaskFetchFilter { + /// Effective per-fetch item cap. + /// + /// `max` is `#[serde(default)]`, so an unset filter arrives as `0`. Reading + /// that literally would mean "fetch nothing", which is never what a caller + /// who omitted the field wanted, so an unset cap becomes a safe bound of 25 + /// instead. + pub fn effective_max(&self) -> usize { + if self.max == 0 { + 25 + } else { + self.max as usize + } + } +} + +#[cfg(test)] +#[path = "tasks_tests.rs"] +mod tests; diff --git a/crates/tinymemory-bus/src/composio/tasks_tests.rs b/crates/tinymemory-bus/src/composio/tasks_tests.rs new file mode 100644 index 00000000..f669e5cd --- /dev/null +++ b/crates/tinymemory-bus/src/composio/tasks_tests.rs @@ -0,0 +1,139 @@ +//! Tests for the task-fetch envelope — the pure-data half. +//! +//! The provider mappings that populate a [`super::NormalizedTask`] live in the +//! engine crate with the providers that own them. What is pinned here is the +//! envelope itself: the wire casing the task-source UI reads, the enum tags +//! that end up in a card's `source_metadata`, and the unset-cap rule that would +//! otherwise read as "fetch nothing". + +#![allow(clippy::expect_used, clippy::unwrap_used, clippy::panic)] + +use super::{GithubFetchMode, NormalizedTask, TaskContainer, TaskFetchFilter, TaskKind}; + +#[test] +fn every_task_kind_tag_matches_its_serde_form() { + for kind in [TaskKind::Generic, TaskKind::Issue, TaskKind::PullRequest] { + let json = serde_json::to_string(&kind).expect("serialize"); + assert_eq!( + json, + format!("\"{}\"", kind.as_str()), + "as_str and the serde form disagree for {kind:?}" + ); + } +} + +#[test] +fn task_kind_tags_are_the_stable_strings() { + assert_eq!(TaskKind::Generic.as_str(), "generic"); + assert_eq!(TaskKind::Issue.as_str(), "issue"); + assert_eq!(TaskKind::PullRequest.as_str(), "pull_request"); +} + +#[test] +fn an_undifferentiated_task_defaults_to_generic() { + assert_eq!(TaskKind::default(), TaskKind::Generic); + assert_eq!(NormalizedTask::default().kind, TaskKind::Generic); +} + +#[test] +fn the_github_fetch_mode_defaults_to_auto() { + // `Auto` is the safe default: a shipped user with no `gh` on `PATH` still + // reaches GitHub through the connected Composio account. + assert_eq!(GithubFetchMode::default(), GithubFetchMode::Auto); + assert_eq!( + TaskFetchFilter::default().github_fetch_mode, + GithubFetchMode::Auto + ); +} + +#[test] +fn every_github_fetch_mode_round_trips_through_its_snake_case_tag() { + let cases = [ + (GithubFetchMode::Auto, "\"auto\""), + (GithubFetchMode::Composio, "\"composio\""), + (GithubFetchMode::Local, "\"local\""), + ]; + for (mode, wire) in cases { + let json = serde_json::to_string(&mode).expect("serialize"); + assert_eq!(json, wire, "wire form changed for {mode:?}"); + let back: GithubFetchMode = serde_json::from_str(&json).expect("deserialize"); + assert_eq!(back, mode); + } +} + +#[test] +fn a_normalized_task_serialises_camel_case_for_the_ui() { + let task = NormalizedTask { + external_id: "42".into(), + source_id: "src-1".into(), + provider: "github".into(), + kind: TaskKind::PullRequest, + title: "Fix the thing".into(), + updated_at: Some("2026-08-25T10:00:00Z".into()), + ..NormalizedTask::default() + }; + let json = serde_json::to_value(&task).expect("serialize to value"); + let object = json.as_object().expect("task serialises as an object"); + + assert!(object.contains_key("externalId")); + assert!(object.contains_key("sourceId")); + assert!(object.contains_key("updatedAt")); + assert_eq!(object["kind"], "pull_request"); + // Absent optionals are skipped rather than emitted as null, so the UI can + // distinguish "the provider had nothing" from "the field is new". + assert!(!object.contains_key("body")); + assert!(!object.contains_key("url")); +} + +#[test] +fn a_normalized_task_round_trips() { + let task = NormalizedTask { + external_id: "7".into(), + provider: "linear".into(), + title: "Ship it".into(), + labels: vec!["p1".into()], + raw: serde_json::json!({ "id": 7 }), + ..NormalizedTask::default() + }; + let json = serde_json::to_string(&task).expect("serialize"); + let back: NormalizedTask = serde_json::from_str(&json).expect("deserialize"); + assert_eq!(back, task); +} + +#[test] +fn a_normalized_task_decodes_from_the_minimum_an_older_peer_wrote() { + // Every field beyond the three required ones carries `#[serde(default)]`, + // so a peer built before any of them existed still decodes. + let back: NormalizedTask = + serde_json::from_str(r#"{"externalId":"1","provider":"notion","title":"t"}"#) + .expect("deserialize minimal"); + assert_eq!(back.external_id, "1"); + assert_eq!(back.source_id, ""); + assert_eq!(back.kind, TaskKind::Generic); + assert!(back.labels.is_empty()); +} + +#[test] +fn an_unset_cap_becomes_a_safe_bound_rather_than_zero() { + assert_eq!(TaskFetchFilter::default().effective_max(), 25); +} + +#[test] +fn an_explicit_cap_is_honoured() { + let filter = TaskFetchFilter { + max: 3, + ..TaskFetchFilter::default() + }; + assert_eq!(filter.effective_max(), 3); +} + +#[test] +fn a_task_container_serialises_the_picker_shape() { + let container = TaskContainer { + id: "db-1".into(), + title: "Roadmap".into(), + }; + let json = serde_json::to_value(&container).expect("serialize to value"); + assert_eq!(json["id"], "db-1"); + assert_eq!(json["title"], "Roadmap"); +} diff --git a/crates/tinymemory-bus/src/learning.rs b/crates/tinymemory-bus/src/learning.rs new file mode 100644 index 00000000..c0a0b88a --- /dev/null +++ b/crates/tinymemory-bus/src/learning.rs @@ -0,0 +1,144 @@ +//! The learning-candidate taxonomy: what a producer asserts about the user, +//! and how strongly. +//! +//! A *candidate* is one observation — "this user prefers `pnpm`", "this user's +//! timezone is `UTC+5:30`" — emitted by a producer and later aggregated by a +//! stability detector into a durable profile facet. The detector weights each +//! candidate by its [`CueFamily`] and decays it by age; the [`FacetClass`] +//! decides the half-life and the per-class budget it is scored against. +//! +//! These three types moved here from the engine crate for the same reason +//! [`crate::evidence::EvidenceRef`] did, one module over: **the producer and +//! the consumer are on opposite sides of the module boundary**. The Composio +//! provider-profile sync emits an identity candidate on every run and runs +//! inside `tinymemory-module`; the stability detector that consumes it runs in +//! the host. Two structurally identical enums either side of that seam would +//! round-trip through serde and diverge silently on the first added variant — +//! and `FacetClass` is exactly the kind of enum that grows. +//! +//! ## What is deliberately *not* here +//! +//! The **queue** is not. The engine crate keeps the bounded ring buffer and its +//! process-global singleton, because a global is not a payload: this crate is +//! compiled into the host binary *and* into the module `cdylib`, so a `static` +//! declared here would be two statics, and a producer pushing into one while a +//! consumer drains the other is worse than no queue at all. See +//! `tinymemory_core::learning_candidate` for the buffer, and the note there on +//! why crossing the module boundary needs a bus member rather than a shared +//! `static`. +//! +//! Also not here: the stability formula itself (`TAU_*` / `HALF_LIFE_*` / +//! `BUDGET_*`, the aggregation and the promotion rules). That is host policy — +//! it decides what the product is willing to believe about a user — and it has +//! never lived in the memory stack. + +use serde::{Deserialize, Serialize}; + +use crate::evidence::EvidenceRef; + +/// Six-class taxonomy of what the learned-facet cache can hold. +/// +/// Keys are stored with a class prefix, e.g. `style/verbosity` or +/// `tooling/package_manager`. The class determines the half-life and the class +/// budget the stability detector scores a candidate against, so it is part of +/// the *storage* key, not only a label: renaming a variant strands every facet +/// filed under the old name. +/// +/// The serde form is `snake_case` and is persisted; treat each variant name as +/// a compatibility surface. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum FacetClass { + /// Communication style preferences — verbosity, formality, code format. + Style, + /// Stable biographical facts — timezone, name, language, role. + Identity, + /// Developer toolchain preferences — package manager, editor, OS, language. + Tooling, + /// Hard user vetoes — things the user has explicitly rejected or forbidden. + Veto, + /// Active user goals or ongoing projects. + Goal, + /// Preferred communication channel or platform. + Channel, +} + +/// How a candidate signal was produced — determines the weight multiplier +/// applied in the stability formula. +/// +/// Higher-weight families contribute more strongly per evidence item. The +/// weights are the canonical values the detector was tuned against: +/// `Explicit=1.0`, `Structural=0.9`, `Behavioral=0.7`, `Recurrence=0.6`. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum CueFamily { + /// Direct declaration of intent by the user (highest weight — 1.0). + /// + /// Examples: "I prefer pnpm", "my timezone is PST", "always use terse replies". + Explicit, + /// Inferred from structured file or provider metadata (weight 0.9). + /// + /// Examples: `package.json#packageManager`, Gmail display name, Slack workspace. + Structural, + /// Inferred by heuristics or an LLM from observed behaviour (weight 0.7). + /// + /// Examples: rolling edit-window ratio, correction-repeat signal, reflection hook output. + Behavioral, + /// Materialized from recurrence statistics in the memory tree (weight 0.6). + /// + /// Examples: tree-topic hotness, `source_weight` per channel. + Recurrence, +} + +impl CueFamily { + /// Weight multiplier for this cue family in the stability formula. + /// + /// Canonical values: `Explicit=1.0`, `Structural=0.9`, `Behavioral=0.7`, + /// `Recurrence=0.6`. They live on the enum rather than in the detector + /// because a producer on the far side of the module boundary has to be + /// able to reason about how much its signal is worth without linking the + /// detector. + pub fn weight(self) -> f64 { + match self { + CueFamily::Explicit => 1.0, + CueFamily::Structural => 0.9, + CueFamily::Behavioral => 0.7, + CueFamily::Recurrence => 0.6, + } + } +} + +/// A single unit of learning evidence emitted by a producer and queued for the +/// stability detector. +/// +/// Each candidate asserts a specific `(class, key, value)` triple alongside the +/// evidence that backs it. The detector aggregates competing candidates for the +/// same `(class, key)` pair and resolves them into a single cache entry, so two +/// producers disagreeing about the user's timezone is a normal input, not an +/// error. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct LearningCandidate { + /// Which facet class this evidence touches. + pub class: FacetClass, + /// Canonical slug key within the class, e.g. `"verbosity"`, `"package_manager"`. + /// + /// Convention: `snake_case`, lowercase, no class prefix (the class carries that). + pub key: String, + /// Canonical value string, e.g. `"terse"`, `"pnpm"`, `"UTC+5:30"`. + pub value: String, + /// How this candidate was produced. + pub cue_family: CueFamily, + /// Pointer to the backing evidence in the memory substrate. + pub evidence: EvidenceRef, + /// Source-provided confidence hint, `0.0..=1.0`. + /// + /// This is an initial hint; the stability detector reweights it using the + /// cue-family weight and recency decay. + pub initial_confidence: f64, + /// When this candidate was observed, as seconds since the Unix epoch. + pub observed_at: f64, +} + +#[cfg(test)] +#[path = "learning_tests.rs"] +mod tests; diff --git a/crates/tinymemory-bus/src/learning_tests.rs b/crates/tinymemory-bus/src/learning_tests.rs new file mode 100644 index 00000000..ebfce282 --- /dev/null +++ b/crates/tinymemory-bus/src/learning_tests.rs @@ -0,0 +1,111 @@ +//! Tests for the learning-candidate taxonomy — the pure-data half. +//! +//! The buffer tests (FIFO order, bounded eviction, the process-global +//! singleton) stay in the engine crate next to the buffer that owns them: +//! `tinymemory_core::learning_candidate`. Nothing here touches a queue. +//! +//! What is pinned below is the part a *second* process can observe: the serde +//! discriminants, which are persisted alongside profile facets and which a +//! producer in the module and a consumer in the host have to agree on. + +#![allow(clippy::expect_used, clippy::unwrap_used, clippy::panic)] + +use super::{CueFamily, FacetClass, LearningCandidate}; +use crate::evidence::EvidenceRef; + +fn candidate(class: FacetClass, cue_family: CueFamily) -> LearningCandidate { + LearningCandidate { + class, + key: "verbosity".into(), + value: "terse".into(), + cue_family, + evidence: EvidenceRef::Episodic { episodic_id: 1 }, + initial_confidence: 0.8, + observed_at: 1_700_000_000.0, + } +} + +#[test] +fn every_facet_class_serialises_to_its_stable_snake_case_name() { + let cases = [ + (FacetClass::Style, "\"style\""), + (FacetClass::Identity, "\"identity\""), + (FacetClass::Tooling, "\"tooling\""), + (FacetClass::Veto, "\"veto\""), + (FacetClass::Goal, "\"goal\""), + (FacetClass::Channel, "\"channel\""), + ]; + for (class, wire) in cases { + let json = serde_json::to_string(&class).expect("serialize"); + assert_eq!(json, wire, "wire form changed for {class:?}"); + let back: FacetClass = serde_json::from_str(&json).expect("deserialize"); + assert_eq!(back, class); + } +} + +#[test] +fn every_cue_family_serialises_to_its_stable_snake_case_name() { + let cases = [ + (CueFamily::Explicit, "\"explicit\""), + (CueFamily::Structural, "\"structural\""), + (CueFamily::Behavioral, "\"behavioral\""), + (CueFamily::Recurrence, "\"recurrence\""), + ]; + for (family, wire) in cases { + let json = serde_json::to_string(&family).expect("serialize"); + assert_eq!(json, wire, "wire form changed for {family:?}"); + let back: CueFamily = serde_json::from_str(&json).expect("deserialize"); + assert_eq!(back, family); + } +} + +#[test] +fn cue_family_weights_are_the_canonical_values() { + assert_eq!(CueFamily::Explicit.weight(), 1.0); + assert_eq!(CueFamily::Structural.weight(), 0.9); + assert_eq!(CueFamily::Behavioral.weight(), 0.7); + assert_eq!(CueFamily::Recurrence.weight(), 0.6); +} + +#[test] +fn weights_are_ordered_explicit_down_to_recurrence() { + // The formula only makes sense if a stated preference outranks an inferred + // one. Asserting the ordering catches a retune that inverts two families + // without anyone noticing the ranking flipped. + assert!(CueFamily::Explicit.weight() > CueFamily::Structural.weight()); + assert!(CueFamily::Structural.weight() > CueFamily::Behavioral.weight()); + assert!(CueFamily::Behavioral.weight() > CueFamily::Recurrence.weight()); +} + +#[test] +fn a_candidate_round_trips_with_its_evidence_pointer() { + let original = candidate(FacetClass::Tooling, CueFamily::Structural); + let json = serde_json::to_string(&original).expect("serialize"); + let back: LearningCandidate = serde_json::from_str(&json).expect("deserialize"); + + assert_eq!(back.class, original.class); + assert_eq!(back.key, original.key); + assert_eq!(back.value, original.value); + assert_eq!(back.cue_family, original.cue_family); + assert_eq!(back.evidence, original.evidence); + assert_eq!(back.initial_confidence, original.initial_confidence); + assert_eq!(back.observed_at, original.observed_at); +} + +#[test] +fn candidate_field_names_are_the_persisted_ones() { + let json = serde_json::to_value(candidate(FacetClass::Goal, CueFamily::Explicit)) + .expect("serialize to value"); + let object = json.as_object().expect("candidate serialises as an object"); + for field in [ + "class", + "key", + "value", + "cue_family", + "evidence", + "initial_confidence", + "observed_at", + ] { + assert!(object.contains_key(field), "missing field {field}"); + } +} diff --git a/crates/tinymemory-bus/src/lib.rs b/crates/tinymemory-bus/src/lib.rs index 9b988117..4e743acf 100644 --- a/crates/tinymemory-bus/src/lib.rs +++ b/crates/tinymemory-bus/src/lib.rs @@ -2,7 +2,7 @@ //! the members that carry them. //! //! TinyMemory ships as a loadable `TinyBus` module: `crates/tinymemory-module` -//! exports one object with 109 members on it, built as a `cdylib`. A host that +//! exports one object with 120 members on it, built as a `cdylib`. A host that //! loads it — OpenHuman — can call into it but cannot `use` anything out of it, //! so the payload vocabulary has to be published as an ordinary library. This //! is that library. @@ -12,6 +12,13 @@ //! - [`names`] — the bus name, the object path, and one constant per member. //! - [`types`], [`chunks`], [`recall`], [`tree`], [`goals`], [`tool_memory`], //! [`health`], [`capabilities`], [`evidence`] — the value vocabulary. +//! - [`learning`] — the learning-candidate taxonomy ([`learning::FacetClass`], +//! [`learning::CueFamily`], [`learning::LearningCandidate`]), whose producer +//! and consumer sit on opposite sides of the module boundary. +//! - [`composio`] — the connector-sync vocabulary: what a provider run +//! produces ([`composio::SyncOutcome`], [`composio::NormalizedTask`]), what +//! it remembers between runs ([`composio::SyncState`]) and what the user has +//! allowed it to do ([`composio::UserScopePref`]). //! - [`graph`] — the bounded graph-view model ([`graph::GraphView`], //! [`graph::GraphViewQuery`]), the graph counterpart of [`tree`]. //! - [`namespace`] — the `
:` namespace convention @@ -24,7 +31,7 @@ //! //! ## What is deliberately not here //! -//! **No traits.** `MemoryProvider` and the eighteen capability-family traits +//! **No traits.** `MemoryProvider` and the twenty capability-family traits //! are driver obligations: they describe what an engine must implement, not //! what a frame carries. They stay in `tinymemory-api`, which depends on this //! crate. @@ -73,11 +80,13 @@ pub mod capabilities; pub mod chunks; +pub mod composio; pub mod error; pub mod evidence; pub mod goals; pub mod graph; pub mod health; +pub mod learning; pub mod names; pub mod namespace; pub mod provider; diff --git a/crates/tinymemory-bus/src/names.rs b/crates/tinymemory-bus/src/names.rs index f3eb2e48..79fd7f6c 100644 --- a/crates/tinymemory-bus/src/names.rs +++ b/crates/tinymemory-bus/src/names.rs @@ -275,6 +275,39 @@ pub mod methods { pub const UPSERT_SEGMENT_EMBEDDING: &str = "UpsertSegmentEmbedding"; /// `InsertEvent` — record one extracted event against its segment. pub const INSERT_EVENT: &str = "InsertEvent"; + + // The summary tree's flush door, addressed by source scope rather than by + // a tree handle. + /// `FlushSourceTree` — seal and cascade one source's tree now. + pub const FLUSH_SOURCE_TREE: &str = "FlushSourceTree"; + + // The typed pipeline diagnosis, beside the maintenance family's uniform + // report. + /// `Diagnose` — the typed, per-stage pipeline diagnosis. + pub const DIAGNOSE: &str = "Diagnose"; + + // Syncs the driver runs itself: the manual trigger, the persisted state, + // and what past runs cost. + /// `RunConnectionSync` — run one connection's sync now. + pub const RUN_CONNECTION_SYNC: &str = "RunConnectionSync"; + /// `SourceSyncState` — the persisted cursor and budget for one connection. + pub const SOURCE_SYNC_STATE: &str = "SourceSyncState"; + /// `SyncAuditLog` — past sync runs, newest first. + pub const SYNC_AUDIT_LOG: &str = "SyncAuditLog"; + /// `EstimateSyncCostUsd` — price a token count at the driver's own rate. + pub const ESTIMATE_SYNC_COST_USD: &str = "EstimateSyncCostUsd"; + /// `SyncStatuses` — per-provider progress, derived from stored content. + pub const SYNC_STATUSES: &str = "SyncStatuses"; + /// `RawArchiveCoverage` — how much of a raw archive its tree covers. + pub const RAW_ARCHIVE_COVERAGE: &str = "RawArchiveCoverage"; + /// `RebuildFromRawArchive` — re-derive a tree from its raw archive. + pub const REBUILD_FROM_RAW_ARCHIVE: &str = "RebuildFromRawArchive"; + + // The local coding-agent transcripts the driver distils. + /// `CodingSessionStatus` — what each agent's session store holds. + pub const CODING_SESSION_STATUS: &str = "CodingSessionStatus"; + /// `IngestCodingSessions` — distil coding sessions into observations. + pub const INGEST_CODING_SESSIONS: &str = "IngestCodingSessions"; } /// Every member name, in the order the module declares them. @@ -282,7 +315,7 @@ pub mod methods { /// The order matters: `tinybus`'s `Interface::members()` returns declaration /// order, and the module compares the two sequences directly rather than as /// sets, so a reordering is caught alongside an addition or a removal. -pub const METHODS: [&str; 109] = [ +pub const METHODS: [&str; 120] = [ methods::DRIVER_ID, methods::CAPABILITIES, methods::HEALTH, @@ -392,6 +425,17 @@ pub const METHODS: [&str; 109] = [ methods::SOURCE_TOTALS, methods::FORGET_MATCHING, methods::PURGE_ALL, + methods::FLUSH_SOURCE_TREE, + methods::DIAGNOSE, + methods::RUN_CONNECTION_SYNC, + methods::SOURCE_SYNC_STATE, + methods::SYNC_AUDIT_LOG, + methods::ESTIMATE_SYNC_COST_USD, + methods::SYNC_STATUSES, + methods::RAW_ARCHIVE_COVERAGE, + methods::REBUILD_FROM_RAW_ARCHIVE, + methods::CODING_SESSION_STATUS, + methods::INGEST_CODING_SESSIONS, ]; #[cfg(test)] diff --git a/crates/tinymemory-bus/src/namespace.rs b/crates/tinymemory-bus/src/namespace.rs index 04c942e4..6ff3eda9 100644 --- a/crates/tinymemory-bus/src/namespace.rs +++ b/crates/tinymemory-bus/src/namespace.rs @@ -3,7 +3,7 @@ //! Namespaces are the only partitioning primitive this contract has, and every //! family takes them as a bare `&str`. That is deliberate — a driver's //! container vocabulary is its own, and a typed namespace threaded through -//! eighteen trait families would force every engine to agree on a shape none of +//! twenty trait families would force every engine to agree on a shape none of //! them share. What was missing was not a type in the signatures but a *shared //! convention* for what goes in the string, so that "conversational memory", //! "document memory", and "learnings" mean the same thing to every caller and diff --git a/crates/tinymemory-bus/src/provider/diagnosis.rs b/crates/tinymemory-bus/src/provider/diagnosis.rs new file mode 100644 index 00000000..79a8490b --- /dev/null +++ b/crates/tinymemory-bus/src/provider/diagnosis.rs @@ -0,0 +1,189 @@ +//! [`Diagnosis`] — the typed, per-stage answer to "why is memory empty?". +//! +//! Returned by the `Diagnose` member of the maintenance family. It sits beside +//! [`crate::provider::types::MaintenanceReport`] rather than replacing it, and +//! the two are not redundant: +//! +//! - [`crate::provider::types::MaintenanceReport`] is what a **scheduler** +//! reads. Its shape is uniform across reembed, compact, consolidate and +//! doctor precisely so a caller driving all four on a timer does not +//! special-case one, and its findings are prose because that is all a log +//! line needs. +//! - [`Diagnosis`] is what an **operator, an agent, or a status panel** reads. +//! Every field it adds is one a caller acts on rather than prints: the +//! remediation key a frontend localises, the class that decides whether to +//! offer a retry, the degradation flags that say results are reduced rather +//! than absent, and the counters that distinguish "nothing ingested" from +//! "ingested and not yet embedded". +//! +//! Widening `MaintenanceReport` to carry all of that was the alternative, and +//! it is the worse one: four of its five producers would leave every new field +//! empty, so the type would stop describing what any single call returns. +//! +//! # The failure vocabulary is the driver's, not this contract's +//! +//! [`DiagnosisFailure::code`] and [`DiagnosisFailure::class`] are strings. +//! Every engine classifies its own pipeline failures, and an enum here would +//! either freeze one engine's taxonomy into the contract or force a second +//! engine to squeeze its causes into someone else's variants — reporting a +//! cause as the nearest wrong one, which is worse than reporting it verbatim. +//! Same reasoning [`crate::provider::types::QueueFailure`] gives for keeping +//! the driver's own words. +//! +//! [`DiagnosisFailure::remediation_key`] is what makes that safe. The caller +//! resolves it to localised text and stays presentational, so an unrecognised +//! code degrades to "we have no localised advice for this" rather than to a +//! mis-rendered one. +//! +//! # Nothing here is memory content +//! +//! Stage notes, details and remediation keys are all operator-facing and are +//! logged. A driver must not put a namespace key, an entry body, a recall +//! query or a credential in any of them. + +use serde::{Deserialize, Serialize}; + +/// One classified reason a stage is not healthy. +#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)] +pub struct DiagnosisFailure { + /// The driver's stable identifier for this cause, in `snake_case`. + /// + /// Compared for equality, never parsed. A caller that does not recognise a + /// code still has [`Self::remediation_key`] and [`Self::detail`] to show. + pub code: String, + /// Whether retrying could help, in the driver's vocabulary — conventionally + /// `transient` or `unrecoverable`. + /// + /// Optional because a driver may classify a cause without deciding its + /// retry policy, and a caller that must guess is better served by an absent + /// answer than by a defaulted wrong one: defaulting to `transient` invites + /// a retry loop against a cause that can never clear. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub class: Option, + /// The i18n key a caller resolves to localised remediation text. + /// + /// Carried so the caller stays presentational — the driver decides what the + /// user should be told to do, the caller decides in which language. + pub remediation_key: String, + /// A non-localised detail for logs and diagnosis. Never a secret. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub detail: Option, +} + +/// The health of one named stage of the driver's ingest pipeline. +#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)] +pub struct DiagnosisStage { + /// The driver's stable id for the stage (`routing`, `embeddings`, `queue`, + /// …). + /// + /// The set is the driver's: a second engine has different stages, and a + /// caller renders whatever it is given in order rather than looking for + /// stages it knows by name. + pub stage: String, + /// Whether this stage is healthy. + pub ok: bool, + /// Why it is not, when it is not. Always `None` when [`Self::ok`]. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub failure: Option, + /// A short operator-facing note, healthy or not. + pub note: String, +} + +/// Which capabilities are running in a reduced mode. +/// +/// "The pipeline ran, but the output is worse than it looks." Surfaced as its +/// own shape because a degraded result is otherwise indistinguishable from a +/// good one: a recall that fell back to recency because no embedder resolved +/// returns rows, and a caller with no way to know that presents them as +/// semantic hits. +#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)] +pub struct DegradedCapabilities { + /// Semantic recall is falling back to recency — no usable embedder. + #[serde(default)] + pub semantic_recall: bool, + /// Extraction is producing no structure, so the entity index is empty. + #[serde(default)] + pub structure: bool, + /// The driver's own storage path is unusable. + /// + /// The most severe of the three: the others reduce quality, this one stops + /// the pipeline before it starts. + #[serde(default)] + pub storage: bool, + /// The cause of the most significant degradation, when the driver knows it. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub cause: Option, +} + +/// The counters a diagnosis is read against. +/// +/// Present so "nothing comes back from recall" can be told apart from "nothing +/// was ever ingested" without a second call that could be answered either side +/// of a write. +#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)] +pub struct DiagnosisCounters { + /// Chunks the driver holds. + pub total_chunks: u64, + /// Jobs waiting. + pub jobs_ready: u64, + /// Jobs a worker currently holds. + pub jobs_running: u64, + /// Jobs in a terminal failure. + pub jobs_failed: u64, + /// Fraction of chunks with at least one extracted entity, in `[0.0, 1.0]`. + /// + /// `None` when the driver could not measure it — deliberately distinct from + /// `Some(0.0)`, which is a real measurement of no structure. Collapsing the + /// two reports a broken read as a broken pipeline. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub extraction_coverage: Option, +} + +/// A one-shot, read-only diagnosis of the driver's ingest pipeline. +/// +/// Read-only in the same sense as +/// [`crate::provider::types::MaintenanceReport`]'s doctor: it inspects +/// configuration, persisted state and counters, and changes nothing. It is +/// specified not to make a live provider call — a network probe would make the +/// diagnosis slow, flaky and order-dependent, and the degradation flags already +/// record what the last real run did. +/// +/// # Why this must be asked of the driver rather than computed by the caller +/// +/// Two of its four parts exist only in the driver's process. +/// [`Self::degraded`] is set by the embed and extract stages as they run, and +/// [`Self::counters`] is a read of the driver's own database. A caller that +/// hosts no engine has neither — it would report an all-clear degradation over +/// counters of zero, which is not a stale diagnosis but a confidently wrong +/// one. +#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)] +pub struct Diagnosis { + /// Whether nothing is blocking. Equivalent to + /// [`Self::first_blocking_cause`] being `None`, carried so a caller can + /// answer the yes/no question without reasoning about an `Option`. + pub healthy: bool, + /// Per-stage health, in the driver's own pipeline order. + /// + /// Order is meaningful: the stages run in it, so the first unhealthy one is + /// the one to fix first. A caller renders the list as given rather than + /// sorting it. + #[serde(default)] + pub stages: Vec, + /// The single cause to act on first. + /// + /// One cause rather than every failing stage, because a stage that cannot + /// run makes the ones after it fail too, and a wall of consequences buries + /// the one thing a user can do about it. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub first_blocking_cause: Option, + /// What is running in a reduced mode even where nothing is blocking. + #[serde(default)] + pub degraded: DegradedCapabilities, + /// The counters the rest of the report is read against. + #[serde(default)] + pub counters: DiagnosisCounters, +} + +#[cfg(test)] +#[path = "diagnosis_tests.rs"] +mod tests; diff --git a/crates/tinymemory-bus/src/provider/diagnosis_tests.rs b/crates/tinymemory-bus/src/provider/diagnosis_tests.rs new file mode 100644 index 00000000..b64b1b69 --- /dev/null +++ b/crates/tinymemory-bus/src/provider/diagnosis_tests.rs @@ -0,0 +1,107 @@ +//! Tests for the pipeline diagnosis. +//! +//! The invariant worth pinning is that an older module's report still decodes: +//! every compound field defaults, so a diagnosis missing `degraded` or +//! `counters` reads as "nothing reported" rather than failing the call — which +//! is the difference between a status panel that degrades and one that goes +//! blank. + +// A failed assertion in a test is a panic either way; `unwrap`/`expect` here say +// what the invariant was. Same allowance the crate's other test modules take. +#![allow(clippy::expect_used, clippy::unwrap_used, clippy::panic)] + +use super::*; + +#[test] +fn a_minimal_diagnosis_decodes_to_nothing_reported() { + let diagnosis: Diagnosis = + serde_json::from_value(serde_json::json!({ "healthy": true })).expect("decode a minimal"); + assert!(diagnosis.healthy); + assert!(diagnosis.stages.is_empty()); + assert_eq!(diagnosis.first_blocking_cause, None); + assert_eq!(diagnosis.degraded, DegradedCapabilities::default()); + assert_eq!(diagnosis.counters.total_chunks, 0); + assert_eq!(diagnosis.counters.extraction_coverage, None); +} + +#[test] +fn an_unmeasured_coverage_is_not_a_measured_zero() { + // `None` is "the read failed"; `Some(0.0)` is "nothing has structure". A + // caller escalates on the second and retries the first. + let unmeasured = DiagnosisCounters::default(); + let measured = DiagnosisCounters { + extraction_coverage: Some(0.0), + ..DiagnosisCounters::default() + }; + assert_ne!(unmeasured, measured); + assert!(serde_json::to_value(&unmeasured) + .expect("serialize counters") + .get("extraction_coverage") + .is_none()); +} + +#[test] +fn a_failure_carries_the_drivers_own_code_unparsed() { + // A code this build has never heard of must survive the round trip: the + // whole reason these are strings is that a newer driver classifies causes + // this one cannot name. + let failure = DiagnosisFailure { + code: "a_cause_from_a_newer_driver".to_string(), + class: None, + remediation_key: "memory.doctor.unknown".to_string(), + detail: Some("the driver's own words".to_string()), + }; + let round_tripped: DiagnosisFailure = + serde_json::from_value(serde_json::to_value(&failure).expect("serialize failure")) + .expect("decode failure"); + assert_eq!(round_tripped, failure); + assert_eq!(round_tripped.class, None); +} + +#[test] +fn healthy_and_a_blocking_cause_are_kept_consistent_by_the_producer() { + // The contract's rule, asserted on the shape a driver is expected to build: + // `healthy` is `first_blocking_cause.is_none()`. The type cannot enforce + // it, so the test states it where a reader will find it. + let stages = vec![ + DiagnosisStage { + stage: "routing".to_string(), + ok: true, + failure: None, + note: "routed".to_string(), + }, + DiagnosisStage { + stage: "embeddings".to_string(), + ok: false, + failure: Some(DiagnosisFailure { + code: "embeddings_unconfigured".to_string(), + class: Some("unrecoverable".to_string()), + remediation_key: "memory.embeddings.unconfigured".to_string(), + detail: None, + }), + note: "no embedder resolved".to_string(), + }, + ]; + let first = stages + .iter() + .find(|stage| !stage.ok) + .and_then(|stage| stage.failure.clone()); + let diagnosis = Diagnosis { + healthy: first.is_none(), + stages, + first_blocking_cause: first, + degraded: DegradedCapabilities { + semantic_recall: true, + ..DegradedCapabilities::default() + }, + counters: DiagnosisCounters::default(), + }; + assert!(!diagnosis.healthy); + assert_eq!( + diagnosis + .first_blocking_cause + .as_ref() + .map(|failure| failure.code.as_str()), + Some("embeddings_unconfigured") + ); +} diff --git a/crates/tinymemory-bus/src/provider/mod.rs b/crates/tinymemory-bus/src/provider/mod.rs index aa1250bc..12e07e1b 100644 --- a/crates/tinymemory-bus/src/provider/mod.rs +++ b/crates/tinymemory-bus/src/provider/mod.rs @@ -11,8 +11,11 @@ //! argument. pub mod chunks; +pub mod diagnosis; pub mod episodic; pub mod people; pub mod profile; pub mod retrieval; +pub mod sessions; +pub mod sync; pub mod types; diff --git a/crates/tinymemory-bus/src/provider/sessions.rs b/crates/tinymemory-bus/src/provider/sessions.rs new file mode 100644 index 00000000..c36c064c --- /dev/null +++ b/crates/tinymemory-bus/src/provider/sessions.rs @@ -0,0 +1,157 @@ +//! The coding-sessions family: transcripts of the user's agent sessions, read +//! and distilled by the driver. +//! +//! A driver advertising +//! [`Capability::CodingSessions`](crate::capabilities::Capability::CodingSessions) +//! knows where a coding agent leaves its session transcripts, can say how much +//! is there without reading any of it into memory, and can run the distillation +//! pass that turns those transcripts into observations about the user. +//! +//! # Why this is not the source-sync family +//! +//! Both are "go and fetch, then tell me what you got", and that is where the +//! resemblance stops. A source sync walks a *remote* connection the user +//! authorised, is billed per provider action, and resumes from a cursor. This +//! walks *local* files the user's own tools wrote, is billed per inference +//! window, and resumes from a per-file state store. A driver that can do one +//! and not the other is the ordinary case rather than the exotic one — a +//! server-side driver has no `~/.claude` to read, and a driver fronting a +//! local vault has no Composio connection — so they negotiate separately. +//! +//! # Where the transcripts are is the driver's business +//! +//! No member here takes a path. Which agents are supported, where each keeps +//! its sessions, and how the environment overrides those locations are all +//! resolved driver-side. A caller passing roots would be choosing which files +//! the driver reads, which is exactly the shape a source gate exists to +//! prevent — and it would freeze the supported-agent list into the contract, +//! where adding one becomes a version bump. + +use serde::{Deserialize, Serialize}; + +/// What one coding agent's session store holds, without ingesting any of it. +/// +/// The answer behind a "there are 412 sessions to import" prompt. Reading it is +/// bounded work: the driver caps how many files it opens and how many bytes it +/// reads, and says so in [`Self::scan_truncated`] rather than taking however +/// long a large history needs. +#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)] +pub struct CodingSessionSource { + /// Which agent this row is for, as the driver names it (`claude_code`, + /// `codex`, …). + /// + /// A wire string rather than an enum for the same reason a source kind is + /// one on the sink family: the supported set is the driver's, and it grows + /// as agents are added without a contract change. + pub kind: String, + /// Whether the driver found this agent's session store at all. + /// + /// `false` with zero counts is "this agent is not installed"; `true` with + /// zero counts is "installed, nothing recorded". A caller prompts for the + /// second and stays quiet about the first. + pub available: bool, + /// Session files the scan saw. + pub session_files: usize, + /// Evidence units those files parse into — the unit the ingest budget is + /// spent in, so this is what a caller sizes an import against. + pub evidence_units: usize, + /// Files the scan could not read or parse. + /// + /// Not an error: a half-written transcript from a session that is still + /// running is normal, and the count is what tells a caller its total is a + /// floor. + pub invalid_files: usize, + /// Whether the scan stopped at one of its own caps rather than at the end. + /// + /// Every count above is then a floor. The caller shows "412+" rather than + /// "412", and the difference matters on the one screen where the number is + /// a promise about how long an import will take. + pub scan_truncated: bool, +} + +/// A request to distil coding sessions into observations. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +pub struct CodingSessionIngestRequest { + /// Re-read sessions the driver has already processed. + /// + /// `false` — the default — processes only what is new since the last run. + /// `true` is the "import my history" pass, and costs an inference window + /// per session all over again. + #[serde(default)] + pub backfill: bool, + /// How many sessions this run may process. + /// + /// The driver clamps it to its own floor and ceiling: a caller cannot raise + /// the limit by asking for more, the same rule + /// [`crate::provider::chunks::ChunkQuery::limit`] carries. Bounded because + /// each session is one or more sequential LLM calls, so an unbounded run is + /// an unbounded bill and an unbounded wall-clock wait. + #[serde(default = "default_max_sessions")] + pub max_sessions: usize, +} + +/// The `max_sessions` an older caller's payload means. +/// +/// A caller that omits the field is asking for the driver's ordinary batch, not +/// for none and not for all of history — so the default is a real batch size +/// rather than `0` (which would silently do nothing) or `usize::MAX` (which +/// would silently do everything). The driver clamps it either way. +fn default_max_sessions() -> usize { + 100 +} + +impl Default for CodingSessionIngestRequest { + /// Incremental, at the default batch size — the shape a scheduler asks for. + fn default() -> Self { + Self { + backfill: false, + max_sessions: default_max_sessions(), + } + } +} + +/// What one coding-session ingest run read, distilled and skipped. +#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)] +pub struct CodingSessionIngestReport { + /// Which pass ran, in the driver's own words (`incremental`, `backfill`). + /// + /// Echoed rather than assumed: a driver that has never run before may + /// upgrade an incremental request to a full pass, and a caller reporting + /// "up to date" over that would be describing the wrong run. + pub mode: String, + /// Session files the run looked at. + pub files_seen: usize, + /// Sessions it distilled. + pub sessions_processed: usize, + /// Sessions it skipped because their state said they were already done. + pub sessions_skipped: usize, + /// Sessions it attempted and failed. + /// + /// Counted rather than raised: one unreadable transcript must not abandon + /// the other four hundred, and a caller decides from the ratio whether + /// anything is actually wrong. + pub sessions_failed: usize, + /// Evidence units the run consumed. + pub evidence_units: usize, + /// Observations it wrote. + pub observations: usize, + /// Whether the run stopped on its budget rather than on running out of + /// sessions. + /// + /// `true` means calling again makes more progress, which is how a caller + /// drains a large history across several passes instead of one call that + /// cannot finish. + pub budget_hit: bool, + /// Where the driver wrote the distilled pack, when it writes one and when + /// the location is meaningful to the caller. + /// + /// `None` from a driver that keeps the result in its own storage. A caller + /// must not require this to be `Some` — it is a convenience for a local + /// driver, not a promise that the output is a file. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub pack_path: Option, +} + +#[cfg(test)] +#[path = "sessions_tests.rs"] +mod tests; diff --git a/crates/tinymemory-bus/src/provider/sessions_tests.rs b/crates/tinymemory-bus/src/provider/sessions_tests.rs new file mode 100644 index 00000000..c0cb1539 --- /dev/null +++ b/crates/tinymemory-bus/src/provider/sessions_tests.rs @@ -0,0 +1,63 @@ +//! Tests for the coding-session value types. +//! +//! The load-bearing part is [`CodingSessionIngestRequest`]'s default: the field +//! is `#[serde(default)]`, so what an older caller's payload *means* is decided +//! here rather than at whichever call site forgot to set it. + +// A failed assertion in a test is a panic either way; `unwrap`/`expect` here say +// what the invariant was. Same allowance the crate's other test modules take. +#![allow(clippy::expect_used, clippy::unwrap_used, clippy::panic)] + +use super::*; + +#[test] +fn an_omitted_max_sessions_means_a_batch_not_none_and_not_everything() { + // `0` would silently ingest nothing and report success; `usize::MAX` would + // silently start an unbounded, billable run. Both are worse than a batch. + let request: CodingSessionIngestRequest = + serde_json::from_value(serde_json::json!({})).expect("decode an empty request"); + assert!(!request.backfill); + assert_eq!(request.max_sessions, 100); + assert_eq!(request, CodingSessionIngestRequest::default()); +} + +#[test] +fn backfill_defaults_to_the_cheap_pass() { + // An absent flag must not mean "re-read all of history": incremental is the + // pass a scheduler can run unattended. + let request: CodingSessionIngestRequest = + serde_json::from_value(serde_json::json!({ "max_sessions": 5 })) + .expect("decode a partial request"); + assert!(!request.backfill); + assert_eq!(request.max_sessions, 5); +} + +#[test] +fn an_absent_agent_is_distinguishable_from_an_empty_one() { + // The pair `(available, session_files)` carries two different prompts, and + // a caller that read only the count would nag about an agent that is not + // installed. + let absent = CodingSessionSource { + kind: "codex".to_string(), + available: false, + ..CodingSessionSource::default() + }; + let empty = CodingSessionSource { + kind: "codex".to_string(), + available: true, + ..CodingSessionSource::default() + }; + assert_ne!(absent, empty); + assert_eq!(absent.session_files, empty.session_files); +} + +#[test] +fn an_ingest_report_without_a_pack_path_omits_it() { + let report = CodingSessionIngestReport { + mode: "incremental".to_string(), + ..CodingSessionIngestReport::default() + }; + let encoded = serde_json::to_value(&report).expect("serialize report"); + assert!(encoded.get("pack_path").is_none()); + assert_eq!(encoded["budget_hit"], serde_json::json!(false)); +} diff --git a/crates/tinymemory-bus/src/provider/sync.rs b/crates/tinymemory-bus/src/provider/sync.rs new file mode 100644 index 00000000..83cc4f3a --- /dev/null +++ b/crates/tinymemory-bus/src/provider/sync.rs @@ -0,0 +1,312 @@ +//! The source-sync family: what the driver reports about a sync it ran itself. +//! +//! A driver advertising +//! [`Capability::SourceSync`](crate::capabilities::Capability::SourceSync) does +//! not merely *accept* items a caller fetched — that is +//! [`Capability::Sources`](crate::capabilities::Capability::Sources) and the +//! sink it names. This family is the other direction: the driver holds the +//! pipelines, walks the connection itself, and answers for what the walk cost. +//! +//! # Why the two are different families +//! +//! [`Capability::Sources`](crate::capabilities::Capability::Sources) documents +//! itself as "accepting synced source items; the host still owns credentials +//! and scheduling", and that premise is still true of the sink. It stopped +//! being true of the *loop*: the periodic Composio and workspace loops now run +//! inside the module beside the queue pool, so the schedule and the credential +//! resolution are the driver's. What the caller kept is the **manual** trigger +//! — a user pressing "sync now" — which is a call it must be able to make and +//! which no member of the sink family can express. +//! +//! Folding these onto the sink instead would advertise them for every driver +//! that can accept a batch. A remote HTTP driver and the null driver both +//! accept batches and neither owns a Composio pipeline, so their callers would +//! get a registered "sync now" button that fails on first press — the +//! registered-but-failing outcome [`crate::capabilities`] exists to avoid. +//! +//! # Money is reported, never recomputed +//! +//! [`SyncAuditEntry::effective_cost_usd`] is arithmetic over fields the row +//! already carries, so it is safe here. The *price* — what a token costs — is +//! deliberately **not** here: it is asked of the driver, because the same +//! constants are what stamped `estimated_cost_usd` onto every row this module +//! hands back. A second copy of those constants in a caller becomes a second +//! price the moment either side is retuned, and the audit rows would then be +//! summed at a rate they were never written with. +//! +//! # No path leaves the driver +//! +//! [`RawArchiveCoverage`] counts pending files and does not name them. The +//! engine's own coverage scan carries absolute paths into the driver's content +//! vault; those describe the driver's storage layout, which no caller may +//! depend on and which is the one thing a bus payload should never teach it. + +use chrono::{DateTime, Utc}; +use serde::{Deserialize, Serialize}; + +/// What one sync run moved, and what it spent doing it. +/// +/// The same five numbers a failed run reports in its error message, so a caller +/// that logs both paths logs the same vocabulary either way. +/// +/// # Not `composio::runs::SyncOutcome` +/// +/// That one is the *report* a caller assembles about a run — which toolkit, +/// which connection, why it ran, when it started and finished, and a one-line +/// summary for a status panel. This is what the run itself produced: how much +/// landed, whether more is waiting, and what the provider charged. A caller +/// building the first from the second is the normal direction; nothing builds +/// the second from the first, which is why they are two shapes rather than one +/// with half its fields unset on every call. +#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)] +pub struct SyncRunOutcome { + /// Items the run stored. + pub records_ingested: u32, + /// Whether the source has more waiting than this run took. + /// + /// A cap was hit — the per-source item limit, the depth window, or the + /// daily request budget — and another run would fetch more. Distinct from + /// `records_ingested == 0`, which can equally mean "nothing new". + pub more_pending: bool, + /// Provider actions the run called. + /// + /// The unit the daily budget is counted in, so a caller showing "requests + /// used today" adds these rather than counting runs. + #[serde(default)] + pub actions_called: u32, + /// What the provider charged for those actions, in USD. + /// + /// The provider's own charge, not the inference cost — that lands on the + /// audit row as [`SyncAuditEntry::estimated_cost_usd`]. Kept apart because + /// they are billed by different parties. + #[serde(default)] + pub provider_cost_usd: f64, + /// A short operator-facing note, when the driver has one. + /// + /// Never memory content: this is rendered in sync status and written to the + /// log. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub note: Option, +} + +/// The persisted cursor, dedup and budget state for one connection. +/// +/// Read-only on this contract. A caller inspects it to render status; it is +/// written by the runs themselves, and a caller that could set a cursor could +/// silently re-fetch or skip a window with no record of having done so. +/// +/// # Why counts and not the sets +/// +/// The persisted state holds the full set of synced item ids and their content +/// versions. Those are unbounded — a mature Gmail connection carries tens of +/// thousands — and a status row needs the size, not the members. Sending the +/// sets would put an ever-growing payload behind a call whose only consumer +/// renders one number from it, and would leak per-message identifiers to a +/// surface that has no use for them. +/// +/// The one caller that genuinely walks the set — a disconnect deciding which +/// per-item documents to forget — reads the persisted row directly through the +/// graph family's `KvGet`, on the sync-state namespace, and decodes it into the +/// shape `composio::state` defines. That path exists, it is not this one, and +/// keeping them apart is what lets a status poll stay small while a disconnect +/// still gets everything. +/// +/// # What this adds over reading that row +/// +/// Two things a raw read cannot give a caller. The daily budget rolls over on +/// the driver's own day boundary, and the persisted row is only rewritten when +/// a sync runs — so `requests_used` read raw is yesterday's number until the +/// next run, while [`Self::daily_requests_used`] has the rollover applied. And +/// the absence of a row means "never synced", which a caller can only learn by +/// knowing the namespace and key convention the driver writes under; asking +/// here spells neither. +#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)] +pub struct SourceSyncState { + /// The toolkit this state belongs to (`gmail`, `slack`, …), lowercased. + pub toolkit: String, + /// The connection this state belongs to. + pub connection_id: String, + /// The provider cursor the next run resumes from, in the provider's own + /// encoding. + /// + /// Opaque: round-trip it, show it, never parse it. Slack's is a JSON map of + /// per-channel cursors and Gmail's is a page token, and a caller that + /// learned to read one would break on the other. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub cursor: Option, + /// How many item ids the dedup set holds. + pub synced_item_count: u64, + /// The newest item id the connection has seen, when it tracks one. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub last_seen_id: Option, + /// When the last run finished. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub last_sync_at_ms: Option, + /// Provider requests spent today against [`Self::daily_request_limit`]. + /// + /// Rolls over on the driver's own day boundary. A caller reading a used + /// count above the limit is reading a limit that was lowered after the + /// spend, not a budget overrun. + pub daily_requests_used: u32, + /// The connection's daily provider-request budget. + pub daily_request_limit: u32, +} + +/// One sync run, as the driver's audit log recorded it. +/// +/// The field names are the driver's on-disk format. They are reproduced here +/// rather than renamed so a caller reading a row over the bus and a caller +/// reading the log file directly see the same keys, and so the driver's +/// conversion is a field-for-field map with nothing to get wrong. +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +pub struct SyncAuditEntry { + /// When the run finished. + pub timestamp: DateTime, + /// The source the run was for. + pub source_id: String, + /// The source's kind (`composio`, `folder`, `github`, …). + pub source_kind: String, + /// The tree scope the run wrote under. + pub scope: String, + /// Items the run fetched from the provider. + pub items_fetched: u32, + /// Summary batches the run sealed. + pub batches: u32, + /// Inference input tokens the run spent. + pub input_tokens: u64, + /// Inference output tokens the run spent. + pub output_tokens: u64, + /// What those tokens were *estimated* to cost, priced by the driver. + /// + /// Stamped at write time from the driver's own price table. Two rows + /// written either side of a retune carry two different rates, which is + /// correct: each says what it was priced at. + pub estimated_cost_usd: f64, + /// Composio actions the run called. + #[serde(default)] + pub composio_actions_called: u32, + /// What Composio charged for those actions. + #[serde(default)] + pub composio_cost_usd: f64, + /// What the inference provider actually billed, when it reported a figure. + /// + /// `None` means no charge was reported and the estimate stands — not that + /// the run was free. + #[serde(default)] + pub actual_charged_usd: Option, + /// Wall-clock duration of the run. + pub duration_ms: u64, + /// Whether the run completed. + pub success: bool, + /// Why it did not, when it did not. Never memory content. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub error: Option, +} + +impl SyncAuditEntry { + /// What the run cost, as the audit views it. + /// + /// The real charge when the provider reported one, the estimate otherwise, + /// plus Composio's own action cost. This is arithmetic over fields the row + /// already carries and introduces no price of its own — which is why it can + /// live here while the price behind + /// [`ESTIMATE_SYNC_COST_USD`](crate::names::methods::ESTIMATE_SYNC_COST_USD) + /// has to be asked of the driver. + #[must_use] + pub fn effective_cost_usd(&self) -> f64 { + self.actual_charged_usd.unwrap_or(self.estimated_cost_usd) + self.composio_cost_usd + } +} + +/// How recently a provider last landed content. +/// +/// Three buckets rather than an age, because the caller renders a badge and the +/// thresholds are the driver's to choose. A caller that computed its own +/// buckets from a timestamp would disagree with the driver's status surface by +/// exactly the drift between the two threshold tables. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum SyncFreshness { + /// Content landed within the driver's "just now" window. + Active, + /// Content landed recently, but the provider is no longer streaming. + Recent, + /// Nothing has landed lately, or ever. + Idle, +} + +/// One provider's share of the store, and how far its last wave got. +/// +/// Derived from stored content rather than from the sync machinery, which is +/// what makes it survivable across a restart: a run that died mid-wave still +/// leaves its chunks, so the pending count is real rather than a counter that +/// was never decremented. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +pub struct SourceSyncStatus { + /// The provider these counts are for, as the driver names it. + pub provider: String, + /// Chunks the provider has contributed in total. + pub chunks_synced: u64, + /// Of those, how many still await derived work (embedding or extraction). + pub chunks_pending: u64, + /// Chunks in the most recent wave. + /// + /// A "wave" is the driver's grouping of one burst of arrivals; with + /// [`Self::batch_processed`] it is what a progress bar needs. Zero when + /// nothing is pending — there is no wave in flight to show. + pub batch_total: u64, + /// Of that wave, how many are finished. + pub batch_processed: u64, + /// When the provider last landed content. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub last_chunk_at_ms: Option, + /// The badge [`Self::last_chunk_at_ms`] resolves to, bucketed by the + /// driver. + pub freshness: SyncFreshness, +} + +/// How much of a raw archive has made it into the tree derived from it. +/// +/// The crosscheck behind a "reconcile" control: a sync writes raw files and +/// then derives a summary tree from them, and a run that died between the two +/// leaves an archive the tree does not cover. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)] +pub struct RawArchiveCoverage { + /// Raw files the archive holds. + pub total: u64, + /// Of those, how many the tree covers. + pub covered: u64, + /// How many are still uncovered. + /// + /// A count, deliberately, and the one reduction this family makes against + /// what the engine computes: the engine's scan carries each pending file's + /// absolute path inside the driver's content vault. A path is the driver's + /// storage layout, which the contract never hands out — see the module + /// docs — and no caller needs it: the pending set is not addressable + /// through any member here, because the repair — + /// [`REBUILD_FROM_RAW_ARCHIVE`](crate::names::methods::REBUILD_FROM_RAW_ARCHIVE) + /// — takes the same scope rather than a file list. + pub pending: u64, +} + +/// What rebuilding a tree from its raw archive read, sealed and spent. +#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)] +pub struct RawRebuildOutcome { + /// Raw files the rebuild read. + pub files_read: u64, + /// Summary batches it sealed. + pub batches: u64, + /// Inference input tokens it spent. + pub input_tokens: u64, + /// Inference output tokens it spent. + pub output_tokens: u64, + /// What those tokens were estimated to cost, priced by the driver. + pub estimated_cost_usd: f64, + /// What the inference provider actually billed, when it reported a figure. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub actual_charged_usd: Option, +} + +#[cfg(test)] +#[path = "sync_tests.rs"] +mod tests; diff --git a/crates/tinymemory-bus/src/provider/sync_tests.rs b/crates/tinymemory-bus/src/provider/sync_tests.rs new file mode 100644 index 00000000..2d50f8b6 --- /dev/null +++ b/crates/tinymemory-bus/src/provider/sync_tests.rs @@ -0,0 +1,147 @@ +//! Tests for the source-sync value types. +//! +//! Two things a later slice can silently break: the cost rule +//! ([`SyncAuditEntry::effective_cost_usd`] must prefer the real charge and must +//! always add Composio's), and the serde defaults an older peer's payload +//! relies on — every one of those fields is a `#[serde(default)]` precisely so +//! a module and a host built a release apart still decode each other. + +// A failed assertion in a test is a panic either way; `unwrap`/`expect` here say +// what the invariant was. Same allowance the crate's other test modules take. +#![allow(clippy::expect_used, clippy::unwrap_used, clippy::panic)] + +use super::*; + +fn entry() -> SyncAuditEntry { + SyncAuditEntry { + timestamp: DateTime::::from_timestamp(1_700_000_000, 0).expect("valid timestamp"), + source_id: "composio:gmail:conn-1".to_string(), + source_kind: "composio".to_string(), + scope: "gmail:conn-1".to_string(), + items_fetched: 12, + batches: 2, + input_tokens: 1_000_000, + output_tokens: 1_000_000, + estimated_cost_usd: 0.35, + composio_actions_called: 4, + composio_cost_usd: 0.02, + actual_charged_usd: None, + duration_ms: 4_200, + success: true, + error: None, + } +} + +#[test] +fn effective_cost_falls_back_to_the_estimate_and_always_adds_composio() { + let entry = entry(); + // No reported charge: the estimate stands, plus the provider's own cost. + assert!((entry.effective_cost_usd() - 0.37).abs() < 1e-9); +} + +#[test] +fn effective_cost_prefers_the_real_charge_over_the_estimate() { + let mut entry = entry(); + entry.actual_charged_usd = Some(0.10); + // The estimate is superseded, not averaged with, and Composio's cost is + // still additive — it is billed by a different party. + assert!((entry.effective_cost_usd() - 0.12).abs() < 1e-9); +} + +#[test] +fn a_reported_charge_of_zero_is_not_an_absent_one() { + // `Some(0.0)` is "the provider billed nothing"; `None` is "the provider said + // nothing". Collapsing them would price a free run at the estimate. + let mut entry = entry(); + entry.actual_charged_usd = Some(0.0); + assert!((entry.effective_cost_usd() - 0.02).abs() < 1e-9); +} + +#[test] +fn an_audit_row_from_an_older_writer_still_decodes() { + // The four `#[serde(default)]` fields were added after the log format + // existed, and the file is append-only across releases: a row written + // before they existed must still read. + let raw = serde_json::json!({ + "timestamp": "2023-11-14T22:13:20Z", + "source_id": "folder:notes", + "source_kind": "folder", + "scope": "folder:notes", + "items_fetched": 3, + "batches": 1, + "input_tokens": 10, + "output_tokens": 5, + "estimated_cost_usd": 0.5, + "duration_ms": 10, + "success": true, + }); + let entry: SyncAuditEntry = serde_json::from_value(raw).expect("decode a pre-default row"); + assert_eq!(entry.composio_actions_called, 0); + assert!((entry.composio_cost_usd - 0.0).abs() < 1e-9); + assert_eq!(entry.actual_charged_usd, None); + assert!((entry.effective_cost_usd() - 0.5).abs() < 1e-9); +} + +#[test] +fn a_sync_run_outcome_from_an_older_module_decodes_to_no_usage() { + // `actions_called`, `provider_cost_usd` and `note` all default: a module + // that predates them reports a run without them, and the caller must read + // that as "no usage recorded", not fail the call. + let outcome: SyncRunOutcome = + serde_json::from_value(serde_json::json!({ "records_ingested": 7, "more_pending": true })) + .expect("decode a minimal outcome"); + assert_eq!(outcome.records_ingested, 7); + assert!(outcome.more_pending); + assert_eq!(outcome.actions_called, 0); + assert_eq!(outcome.note, None); +} + +#[test] +fn freshness_wire_strings_are_snake_case() { + // Rendered as a badge by name, so these strings are the contract. + for (freshness, expected) in [ + (SyncFreshness::Active, "active"), + (SyncFreshness::Recent, "recent"), + (SyncFreshness::Idle, "idle"), + ] { + assert_eq!( + serde_json::to_value(freshness).expect("serialize freshness"), + serde_json::Value::String(expected.to_string()) + ); + } +} + +#[test] +fn coverage_counts_pending_and_names_nothing() { + // The reduction is deliberate and is asserted rather than left to the docs: + // a path inside the driver's content vault must not appear on the wire. + let coverage = RawArchiveCoverage { + total: 10, + covered: 7, + pending: 3, + }; + let encoded = serde_json::to_value(coverage).expect("serialize coverage"); + assert_eq!(encoded["pending"], serde_json::json!(3)); + assert!( + encoded.as_object().is_some_and(|map| map.len() == 3), + "coverage carries exactly total/covered/pending: {encoded}" + ); +} + +#[test] +fn sync_state_omits_the_absent_optionals_rather_than_nulling_them() { + // The status row is rendered straight from this shape, and a `null` cursor + // and an omitted one are the same fact; emitting both spellings over the + // life of one connection makes a caller handle two. + let state = SourceSyncState { + toolkit: "slack".to_string(), + connection_id: "conn-1".to_string(), + daily_request_limit: 500, + ..SourceSyncState::default() + }; + let encoded = serde_json::to_value(&state).expect("serialize sync state"); + assert!(encoded.get("cursor").is_none()); + assert!(encoded.get("last_seen_id").is_none()); + assert!(encoded.get("last_sync_at_ms").is_none()); + assert_eq!(encoded["daily_requests_used"], serde_json::json!(0)); +} diff --git a/crates/tinymemory-core/src/engine/sync.rs b/crates/tinymemory-core/src/engine/sync.rs index 4880b117..a93783f6 100644 --- a/crates/tinymemory-core/src/engine/sync.rs +++ b/crates/tinymemory-core/src/engine/sync.rs @@ -465,6 +465,10 @@ pub async fn load_composio_sync_state( toolkit: &str, connection_id: &str, ) -> anyhow::Result { + // `load` is an extension-trait method since the state shape moved to the + // contract crate (#5560); the trait has to be in scope to call it. + use crate::sync::composio::providers::sync_state::PersistedSyncState; + let memory = crate::global::client_if_ready() .ok_or_else(|| anyhow::anyhow!("memory client is not ready"))?; let host = crate::sync::pipelines::host::PipelineHost::without_tree_ingest(memory); diff --git a/crates/tinymemory-core/src/learning_candidate.rs b/crates/tinymemory-core/src/learning_candidate.rs index 1ce93e35..b161e4e4 100644 --- a/crates/tinymemory-core/src/learning_candidate.rs +++ b/crates/tinymemory-core/src/learning_candidate.rs @@ -1,125 +1,58 @@ //! Learning candidate buffer — Phase 1 of issue #566. //! -//! Defines the taxonomy types ([`FacetClass`], [`CueFamily`], [`EvidenceRef`]), -//! the unit-of-work [`LearningCandidate`], and a thread-safe ring-buffer -//! [`Buffer`] that collects candidates emitted by producers (Phase 2) before -//! they are consumed by the stability detector (Phase 3). +//! The taxonomy ([`FacetClass`], [`CueFamily`], [`EvidenceRef`]) and the +//! unit-of-work [`LearningCandidate`] are defined in the contract crate; this +//! module re-exports them and owns the thread-safe ring-buffer [`Buffer`] that +//! collects candidates emitted by producers (Phase 2) before the stability +//! detector consumes them (Phase 3). //! //! The buffer is bounded: when full it evicts the oldest entry (FIFO overflow). //! A global singleton is exposed via [`global()`]; individual tests may //! construct their own [`Buffer`] with `Buffer::new(capacity)`. +//! +//! # Why the types moved out and the buffer did not (#5560) +//! +//! The types moved to [`tinymemory_api::learning`] because a *host* names them: +//! the stability detector, the facet cache and the reflection hooks all live in +//! OpenHuman, and reaching them through this crate is one of the compile-time +//! links #5560 removes. They are inert serde data, so the contract crate is the +//! right floor for them — same argument, and the same destination, as +//! [`EvidenceRef`], which went there first. +//! +//! The buffer stayed because a **`static` is not a payload**. This crate is +//! compiled into the module `cdylib`; the contract crate is compiled into both +//! that and the host binary. Moving [`global()`] down would not give the two +//! sides one queue, it would give them two, and the producer would push into +//! the copy the consumer never drains. +//! +//! **That split is already live, and moving the types does not close it.** The +//! one producer in this workspace is +//! `crate::sync::composio::providers::profile`, which pushes an identity +//! candidate on every provider-profile sync — and that code runs inside the +//! module. The host's detector drains the host's buffer. Delivering a candidate +//! across that boundary needs a bus member (or an event), which is contract +//! work rather than a re-export, and is called out in the upstream gap notes +//! rather than papered over here. use std::collections::VecDeque; use std::sync::OnceLock; use parking_lot::Mutex; -use serde::{Deserialize, Serialize}; - -// ── Taxonomy ──────────────────────────────────────────────────────────────── - -/// Six-class taxonomy of what the cache can hold. -/// -/// Keys are stored with a class prefix, e.g. `style/verbosity` or -/// `tooling/package_manager`. The class determines the half-life and -/// class budget used by the stability detector (Phase 3). -#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] -#[serde(rename_all = "snake_case")] -pub enum FacetClass { - /// Communication style preferences — verbosity, formality, code format. - Style, - /// Stable biographical facts — timezone, name, language, role. - Identity, - /// Developer toolchain preferences — package manager, editor, OS, language. - Tooling, - /// Hard user vetoes — things the user has explicitly rejected or forbidden. - Veto, - /// Active user goals or ongoing projects. - Goal, - /// Preferred communication channel or platform. - Channel, -} -/// How a candidate signal was produced — determines the weight multiplier -/// applied in the stability formula. +/// The learning-candidate taxonomy, defined in the contract crate. /// -/// Higher-weight families contribute more strongly per evidence item. -/// The weights here are the canonical values from the Phase 1 plan: -/// `Explicit=1.0`, `Structural=0.9`, `Behavioral=0.7`, `Recurrence=0.6`. -#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] -#[serde(rename_all = "snake_case")] -pub enum CueFamily { - /// Direct declaration of intent by the user (highest weight — 1.0). - /// - /// Examples: "I prefer pnpm", "my timezone is PST", "always use terse replies". - Explicit, - /// Inferred from structured file or provider metadata (weight 0.9). - /// - /// Examples: `package.json#packageManager`, Gmail display name, Slack workspace. - Structural, - /// Inferred by heuristics or LLM from observed behaviour (weight 0.7). - /// - /// Examples: rolling edit-window ratio, correction-repeat signal, reflection hook output. - Behavioral, - /// Materialized from recurrence statistics in the memory tree (weight 0.6). - /// - /// Examples: tree-topic hotness, source_weight per channel. - Recurrence, -} - -impl CueFamily { - /// Weight multiplier for this cue family in the stability formula. - /// - /// Phase 1 canonical values (matches the plan): - /// `Explicit=1.0`, `Structural=0.9`, `Behavioral=0.7`, `Recurrence=0.6`. - pub fn weight(self) -> f64 { - match self { - CueFamily::Explicit => 1.0, - CueFamily::Structural => 0.9, - CueFamily::Behavioral => 0.7, - CueFamily::Recurrence => 0.6, - } - } -} +/// Re-exported at this path because ~30 call sites in this crate and in +/// OpenHuman already spell it `learning_candidate::FacetClass`, and the move +/// delivers the decoupling without spending that churn. +pub use tinymemory_api::learning::{CueFamily, FacetClass, LearningCandidate}; -// ── Evidence reference ─────────────────────────────────────────────────────── +// ── Evidence reference ────────────────────────────────────────── /// Where a candidate's evidence points. Defined in the contract crate — the /// memory store persists it, so both sides must name one type. See /// [`tinymemory_api::host::EvidenceRef`]. pub use tinymemory_api::host::EvidenceRef; -// ── Learning candidate ─────────────────────────────────────────────────────── - -/// A single unit of learning evidence emitted by a producer and queued in the -/// [`Buffer`]. -/// -/// Each candidate asserts a specific `(class, key, value)` triple alongside -/// the evidence that backs it. The stability detector (Phase 3) aggregates -/// competing candidates for the same `(class, key)` pair and resolves them -/// into a single cache entry. -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct LearningCandidate { - /// Which facet class this evidence touches. - pub class: FacetClass, - /// Canonical slug key within the class, e.g. `"verbosity"`, `"package_manager"`. - /// - /// Convention: `snake_case`, lowercase, no class prefix (the class carries that). - pub key: String, - /// Canonical value string, e.g. `"terse"`, `"pnpm"`, `"UTC+5:30"`. - pub value: String, - /// How this candidate was produced. - pub cue_family: CueFamily, - /// Pointer to the backing evidence in the memory substrate. - pub evidence: EvidenceRef, - /// Source-provided confidence hint, `0.0..=1.0`. - /// - /// This is an initial hint; the stability detector will reweight it using - /// the cue-family weight and recency decay. - pub initial_confidence: f64, - /// When this candidate was observed, as seconds since the Unix epoch. - pub observed_at: f64, -} - // ── Buffer ─────────────────────────────────────────────────────────────────── /// Thread-safe, bounded ring-buffer of [`LearningCandidate`] items. diff --git a/crates/tinymemory-core/src/sources/registry.rs b/crates/tinymemory-core/src/sources/registry.rs index 93691737..4d0151ed 100644 --- a/crates/tinymemory-core/src/sources/registry.rs +++ b/crates/tinymemory-core/src/sources/registry.rs @@ -2,6 +2,14 @@ //! //! The registry itself moved to `tinymemory-sources` (#18 §B4); this layer adds //! the host's config path and the lock that serialises writes to it. +//! +//! [`apply_kind_defaults`] followed the registry down in #5560. It is pure +//! policy over a [`MemorySourceEntry`] — it fills caps that are still `None` +//! and nothing else — and OpenHuman calls it when a user adds a source, so it +//! had to be reachable without a compile-time link to this crate. It now sits +//! beside `memory_sync_defaults_for_toolkit`, the Composio half of the same +//! decision, which is where it should have been all along: creation-time and +//! migration-time defaults only stay in step while the policy has one address. use std::sync::OnceLock; @@ -9,7 +17,7 @@ use crate::config_loader as config_rpc; use crate::sources::types::{MemorySourceEntry, SourceKind}; pub use tinymemory_sources::{ - memory_sync_defaults_for_toolkit, ComposioUpsertTarget, MemorySourcePatch, + apply_kind_defaults, memory_sync_defaults_for_toolkit, ComposioUpsertTarget, MemorySourcePatch, }; static MEMORY_SOURCES_WRITE_LOCK: OnceLock> = OnceLock::new(); @@ -136,39 +144,6 @@ pub async fn apply_all_in() -> Result, String> { .map_err(|error| error.to_string()) } -/// Apply conservative per-kind cap defaults to a new source entry. -/// -/// Only fills fields that are still `None` — never overwrites a -/// caller-supplied value. This mirrors the retroactive migration logic in -/// `reconcile::apply_composio_source_caps_migration` so the same defaults -/// are applied consistently at creation time and during migration. -pub fn apply_kind_defaults(entry: &mut MemorySourceEntry) { - match entry.kind { - SourceKind::GithubRepo => { - if entry.max_prs.is_none() { - entry.max_prs = Some(10); - } - if entry.max_issues.is_none() { - entry.max_issues = Some(10); - } - if entry.max_commits.is_none() { - entry.max_commits = Some(50); - } - } - SourceKind::RssFeed => { - if entry.max_items.is_none() { - entry.max_items = Some(20); - } - } - SourceKind::TwitterQuery if entry.since_days.is_none() => { - entry.since_days = Some(7); - } - // Folder / WebPage / Composio: no defaults to apply here. - // Composio defaults are set at upsert time in registry::upsert_composio_source. - _ => {} - } -} - /// Decode the source registry a host config carries. /// /// The registry crosses the host seam as JSON: [`MemorySourceEntry`] is defined diff --git a/crates/tinymemory-core/src/sync/composio/providers/mod.rs b/crates/tinymemory-core/src/sync/composio/providers/mod.rs index dd8e96ab..a46b9523 100644 --- a/crates/tinymemory-core/src/sync/composio/providers/mod.rs +++ b/crates/tinymemory-core/src/sync/composio/providers/mod.rs @@ -229,51 +229,15 @@ pub fn catalog_for_toolkit(toolkit: &str) -> Option<&'static [CuratedTool]> { /// All toolkit slugs that have a curated agent-ready catalog. /// -/// Source of truth for the UI "preview / agent integration coming -/// soon" badge: any connected toolkit whose slug is NOT in this list -/// can be authorized but lacks a curated tool surface, so the agent -/// can't use it productively. +/// Source of truth for the UI "preview / agent integration coming soon" badge: +/// any connected toolkit whose slug is NOT in this list can be authorized but +/// lacks a curated tool surface, so the agent can't use it productively. /// -/// Returned in sorted order to keep the RPC response stable across -/// builds. -pub fn agent_ready_toolkits() -> Vec<&'static str> { - let mut slugs: Vec<&'static str> = vec![ - // Native providers - "gmail", - "notion", - "github", - // Catalog-only toolkits - "slack", - "discord", - "googlecalendar", - "googledrive", - "googledocs", - "googlesheets", - "outlook", - "microsoft_teams", - "linear", - "jira", - "trello", - "asana", - "dropbox", - "twitter", - "spotify", - "telegram", - "whatsapp", - "shopify", - "stripe", - "hubspot", - "salesforce", - "airtable", - "figma", - "youtube", - "one_drive", - "excel", - "todoist", - ]; - slugs.sort_unstable(); - slugs -} +/// Defined in the contract crate (#5560) because the *host* renders that badge +/// and reaching this crate to spell the list is one of the compile-time links +/// the issue removes. Re-exported here so every historical +/// `providers::agent_ready_toolkits()` call keeps resolving. +pub use tinymemory_api::composio::scopes::agent_ready_toolkits; pub use descriptions::toolkit_description; pub(crate) use helpers::{first_array_str, merge_extra}; diff --git a/crates/tinymemory-core/src/sync/composio/providers/profile.rs b/crates/tinymemory-core/src/sync/composio/providers/profile.rs index ebbc6347..d2546717 100644 --- a/crates/tinymemory-core/src/sync/composio/providers/profile.rs +++ b/crates/tinymemory-core/src/sync/composio/providers/profile.rs @@ -16,8 +16,27 @@ //! `fetch_user_profile` call — from `on_connection_created`, periodic syncs, //! and the `composio_get_user_profile` / `composio_refresh_all_identities` //! RPC ops. +//! +//! # Where the vocabulary lives (#5560) +//! +//! [`IdentityKind`], [`canonicalize`], [`ConnectedIdentity`], +//! [`render_connected_identities_section`] and +//! [`normalize_connection_identifier`] are defined in +//! [`tinymemory_api::composio::profile`] and re-exported here. +//! +//! Canonicalisation had to go down because equality of canonical forms is the +//! matcher's *only* test, and the two calls it compares are on opposite sides +//! of the module boundary: the value is canonicalised here when a profile is +//! persisted, and again in OpenHuman when a candidate identifier is checked +//! against it. Two implementations would fail open — a user's own messages +//! would quietly stop being recognised as theirs. The same argument covers the +//! identifier normalisation, which produces the key segment a row is *stored* +//! under: a delete that spelled it differently would leave rows behind and keep +//! treating a disconnected account as the user. +//! +//! Everything that touches the facet store stayed here, because the contract +//! crate holds no storage. -use super::ProviderUserProfile; use crate::learning_candidate::{ self as learning_candidate, CueFamily, EvidenceRef, FacetClass, LearningCandidate, }; @@ -25,100 +44,17 @@ use crate::store::profile::FacetType; use serde_json::Value; use std::collections::BTreeMap; -// ──────────────────────────────────────────────────────────────────────── -// IdentityKind — the matching axis -// ──────────────────────────────────────────────────────────────────────── - -/// Shape of an identifier persisted against a connection. Mirrors the -/// matching dimensions of the memory tree's -/// `crate::tree::score::extract::EntityKind` so the -/// self-check is a direct `(toolkit, kind, value)` lookup. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub enum IdentityKind { - /// Platform-canonical immutable id — Slack `U123ABC`, Notion UUID. - UserId, - Email, - /// `@`-style screen name, canonicalised without the leading `@`. - Handle, - /// E.164 phone number. - Phone, - /// Human display label. Weak signal — never auto-promotes to is_self. - DisplayName, - /// Not for matching; kept for UI / prompt rendering. - AvatarUrl, - /// Not for matching; kept for UI / prompt rendering. - ProfileUrl, -} - -impl IdentityKind { - pub fn as_str(self) -> &'static str { - match self { - Self::UserId => "user_id", - Self::Email => "email", - Self::Handle => "handle", - Self::Phone => "phone", - Self::DisplayName => "display_name", - Self::AvatarUrl => "avatar_url", - Self::ProfileUrl => "profile_url", - } - } - - pub fn parse(s: &str) -> Option { - Some(match s { - "user_id" => Self::UserId, - "email" => Self::Email, - "handle" => Self::Handle, - "phone" => Self::Phone, - "display_name" => Self::DisplayName, - "avatar_url" => Self::AvatarUrl, - "profile_url" => Self::ProfileUrl, - _ => return None, - }) - } - - /// Confidence the matcher records on the row. Hard kinds auto-promote - /// a chunk to `is_self`; weak kinds require corroboration. - pub fn confidence(self) -> f64 { - match self { - Self::UserId | Self::Phone => 1.00, - Self::Email => 0.95, - Self::Handle => 0.70, - Self::DisplayName => 0.40, - Self::AvatarUrl | Self::ProfileUrl => 0.50, - } - } +use tinymemory_api::composio::profile::normalize_connection_identifier as normalize_token; - /// True if this kind is a real identity signal worth running through - /// the matcher (vs. UI-only fields). - pub fn is_matchable(self) -> bool { - matches!( - self, - Self::UserId | Self::Email | Self::Handle | Self::Phone | Self::DisplayName - ) - } -} - -/// Canonicalize a raw value for storage and lookup. The same routine runs -/// on the entity side at match time, so equality of canonical forms is the -/// matcher's only test — no `COLLATE NOCASE`, no per-call lowercasing. -pub fn canonicalize(kind: IdentityKind, raw: &str) -> Option { - let trimmed = raw.trim(); - if trimmed.is_empty() { - return None; - } - Some(match kind { - IdentityKind::Email => trimmed.to_lowercase(), - IdentityKind::Handle => trimmed.trim_start_matches('@').to_lowercase(), - IdentityKind::Phone => trimmed - .chars() - .filter(|c| c.is_ascii_digit() || *c == '+') - .collect(), - IdentityKind::DisplayName => trimmed.split_whitespace().collect::>().join(" "), - IdentityKind::UserId | IdentityKind::AvatarUrl | IdentityKind::ProfileUrl => { - trimmed.to_string() - } - }) -} +/// The identity vocabulary, defined in the contract crate. +/// +/// Re-exported at this path so every historical +/// `providers::profile::IdentityKind` reference keeps resolving. See the module +/// docs for why the shapes went down and the store access stayed. +pub use tinymemory_api::composio::profile::{ + canonicalize, normalize_connection_identifier, render_connected_identities_section, + ConnectedIdentity, IdentityKind, ProviderUserProfile, +}; // ──────────────────────────────────────────────────────────────────────── // Persist @@ -261,19 +197,6 @@ fn json_str<'a>(v: &'a Value, key: &str) -> Option<&'a str> { // Read paths // ──────────────────────────────────────────────────────────────────────── -#[derive(Debug, Clone, Default, PartialEq, Eq)] -pub struct ConnectedIdentity { - pub source: String, - pub identifier: String, - pub display_name: Option, - pub email: Option, - pub handle: Option, - pub phone: Option, - pub user_id: Option, - pub avatar_url: Option, - pub profile_url: Option, -} - /// Load all provider-sourced identities, grouped by `(source, conn_id)`. /// Rows whose last segment is not a known [`IdentityKind`] are silently /// skipped — that includes legacy `username` rows from before the rewrite. @@ -361,56 +284,6 @@ pub fn is_self_identity_any_toolkit(kind: IdentityKind, raw_value: &str) -> bool .skill_identity_matches(&key_pattern, &canonical) } -/// Render a compact section for prompt injection. Skips `user_id` (not -/// human-readable), prefixes `handle` with `@`. -pub fn render_connected_identities_section(identities: &[ConnectedIdentity]) -> String { - if identities.is_empty() { - return String::new(); - } - let mut out = String::from("## Connected Identities\n\n"); - for id in identities { - let mut fields = Vec::::new(); - if let Some(v) = id.display_name.as_deref() { - let v = sanitize_prompt_value(v); - if !v.is_empty() { - fields.push(v); - } - } - if let Some(v) = id.email.as_deref() { - let v = sanitize_prompt_value(v); - if !v.is_empty() { - fields.push(v); - } - } - if let Some(v) = id.handle.as_deref() { - let v = sanitize_prompt_value(v); - if !v.is_empty() { - fields.push(format!("@{v}")); - } - } - if let Some(v) = id.profile_url.as_deref() { - let v = sanitize_prompt_value(v); - if !v.is_empty() { - fields.push(v); - } - } - if fields.is_empty() { - continue; - } - let identifier = sanitize_prompt_value(&id.identifier); - out.push_str(&format!( - "- {} ({}): {}\n", - title_case(&id.source), - identifier, - fields.join(" | ") - )); - } - if out.trim() == "## Connected Identities" { - return String::new(); - } - out -} - /// Delete every row for a `(source, conn_id)` pair — used on disconnect. pub fn delete_connected_identity_facets(source: &str, identifier: &str) -> usize { // `persist_provider_profile` writes keys with `normalize_token`-applied @@ -464,36 +337,6 @@ fn parse_skill_identity_key(key: &str) -> Option<(String, String, String)> { Some((source.to_string(), identifier.to_string(), kind.to_string())) } -fn normalize_token(raw: &str) -> String { - let mut out = String::with_capacity(raw.len()); - for ch in raw.chars() { - let lower = ch.to_ascii_lowercase(); - if lower.is_ascii_alphanumeric() || lower == '-' || lower == '_' { - out.push(lower); - } else { - out.push('_'); - } - } - out.trim_matches('_').to_string() -} - -pub fn normalize_connection_identifier(raw: &str) -> String { - normalize_token(raw) -} - -fn title_case(raw: &str) -> String { - let mut chars = raw.chars(); - match chars.next() { - Some(first) => first.to_ascii_uppercase().to_string() + chars.as_str(), - None => String::new(), - } -} - -fn sanitize_prompt_value(raw: &str) -> String { - let replaced = raw.replace(['\n', '\r', '\t'], " ").replace('|', "/"); - replaced.split_whitespace().collect::>().join(" ") -} - fn now_secs() -> f64 { use std::time::{SystemTime, UNIX_EPOCH}; SystemTime::now() diff --git a/crates/tinymemory-core/src/sync/composio/providers/profile_tests.rs b/crates/tinymemory-core/src/sync/composio/providers/profile_tests.rs index 438f808f..28eafaeb 100644 --- a/crates/tinymemory-core/src/sync/composio/providers/profile_tests.rs +++ b/crates/tinymemory-core/src/sync/composio/providers/profile_tests.rs @@ -343,8 +343,10 @@ fn helper_parsing_and_normalization_cover_malformed_inputs() { "team_name__me" ); assert_eq!(normalize_connection_identifier("___"), ""); - assert_eq!(title_case(""), ""); - assert_eq!(title_case("slack"), "Slack"); + // `title_case` went down with the renderer that is its only caller + // (#5560); its behaviour is asserted through + // `render_connected_identities_section` in the contract crate's tests, + // which is the only way it is observable. } #[test] diff --git a/crates/tinymemory-core/src/sync/composio/providers/sync_state.rs b/crates/tinymemory-core/src/sync/composio/providers/sync_state.rs index 6f161ac0..d39e4364 100644 --- a/crates/tinymemory-core/src/sync/composio/providers/sync_state.rs +++ b/crates/tinymemory-core/src/sync/composio/providers/sync_state.rs @@ -1,33 +1,64 @@ //! Cursor, dedup and daily-budget state for Composio sync (#18 §B2). //! -//! Owned here, engine-neutral, persisted through the [`SyncStateStore`] KV -//! seam — any provider whose KV family can get/set a JSON value can carry -//! sync state. This was a re-export of the engine's copy; §B2 asks for the -//! state to be engine-neutral, and the type is nothing but serde shapes over -//! std/chrono, so owning it costs one copy. +//! Engine-neutral, persisted through the [`SyncStateStore`] KV seam — any +//! provider whose KV family can get and set a JSON value can carry sync state. //! -//! The engine keeps its own copy for its internal pipelines until §B1's -//! orchestrator move retires them. The two persist under the same KV -//! namespace with the same serde shape; `the_state_namespace_is_pinned` and -//! `state_line_format_is_pinned` below hold this copy to that contract. - -use std::collections::{HashMap, HashSet}; +//! The engine keeps its own copy of the shape for its internal pipelines until +//! §B1's orchestrator move retires them. The two persist under the same KV +//! namespace with the same serde form; the pin tests either side hold this copy +//! to that contract. +//! +//! # Where the shape lives (#5560) +//! +//! [`SyncState`], [`DailyBudget`], the namespaces and [`extract_item_id`] are +//! defined in [`tinymemory_api::composio::state`] and re-exported here. Both +//! sides read them: the module advances the cursor and spends the budget, while +//! OpenHuman renders "312 of 500 requests used today" and, on disconnect, walks +//! the dedup set to decide what to forget. A host-side twin would decode today +//! and diverge on the first added field — and because this shape is +//! *persisted*, divergence is a stranded cursor and a re-ingested inbox rather +//! than a wire error someone notices. +//! +//! What stayed here is the I/O: the [`SyncStateStore`] seam and the two methods +//! that use it, offered as the [`PersistedSyncState`] extension trait because +//! an inherent `impl` has to live in the crate that defines the type. Call +//! sites are unchanged — `SyncState::load(store, …)` and `state.save(store)` +//! still resolve — but the trait has to be in scope, so the four call sites in +//! this crate import it alongside the type. use async_trait::async_trait; -use chrono::Utc; -use serde::{Deserialize, Serialize}; -/// The KV namespace every persisted sync cursor lives under. +/// The persisted sync-state shape, defined in the contract crate. /// -/// Durable: changing it strands every cursor. See the pin test. -pub const KV_NAMESPACE: &str = STATE_NAMESPACE; - -pub const DEFAULT_DAILY_REQUEST_LIMIT: u32 = 500; -pub const STATE_NAMESPACE: &str = "composio-sync-state"; - +/// Re-exported at this path so every historical +/// `providers::sync_state::SyncState` reference keeps resolving. +pub use tinymemory_api::composio::state::{ + extract_item_id, DailyBudget, SyncState, DEFAULT_DAILY_REQUEST_LIMIT, KV_NAMESPACE, + STATE_NAMESPACE, +}; + +/// The key/value seam a [`SyncState`] is persisted through. +/// +/// Deliberately narrower than a memory client: get and set one JSON value by +/// `(namespace, key)`. That is the whole requirement, and stating it as two +/// methods is what lets a non-TinyCortex driver carry Composio sync state +/// without implementing anything else. #[async_trait] pub trait SyncStateStore: Send + Sync { + /// Read the value at `(namespace, key)`, or `None` when nothing is stored. + /// + /// # Errors + /// + /// Returns an error when the underlying store cannot be reached. "Nothing + /// stored" is `Ok(None)`, not an error — a first sync is the normal case. async fn get(&self, namespace: &str, key: &str) -> anyhow::Result>; + + /// Write `value` at `(namespace, key)`, replacing anything already there. + /// + /// # Errors + /// + /// Returns an error when the underlying store rejects or cannot persist the + /// write. async fn set( &self, namespace: &str, @@ -36,133 +67,45 @@ pub trait SyncStateStore: Send + Sync { ) -> anyhow::Result<()>; } -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct DailyBudget { - pub date: String, - pub requests_used: u32, - pub limit: u32, -} - -impl Default for DailyBudget { - fn default() -> Self { - Self { - date: today(), - requests_used: 0, - limit: DEFAULT_DAILY_REQUEST_LIMIT, - } - } -} - -impl DailyBudget { - pub fn remaining(&self) -> u32 { - if self.date != today() { - self.limit - } else { - self.limit.saturating_sub(self.requests_used) - } - } - - pub fn is_exhausted(&self) -> bool { - self.remaining() == 0 - } - - pub fn record_requests(&mut self, count: u32) { - let today = today(); - if self.date != today { - self.date = today; - self.requests_used = 0; - } - self.requests_used = self.requests_used.saturating_add(count); - } - - pub fn record_request(&mut self) { - self.record_requests(1); - } -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct SyncState { - pub toolkit: String, - pub connection_id: String, - #[serde(default)] - pub cursor: Option, - #[serde(default)] - pub synced_ids: HashSet, - #[serde(default)] - pub item_versions: HashMap, - #[serde(default)] - pub daily_budget: DailyBudget, - #[serde(default)] - pub last_seen_id: Option, - #[serde(default)] - pub last_sync_at_ms: Option, - #[serde(skip)] - pub run_requests: u32, - #[serde(skip)] - pub run_provider_cost_usd: f64, +/// Loading and saving a [`SyncState`] through a [`SyncStateStore`]. +/// +/// An extension trait rather than an inherent `impl` because the type is +/// defined in the contract crate, which holds no I/O and publishes no traits. +/// The method names and signatures are the ones the inherent versions had, so +/// existing call sites only need this trait in scope. +#[async_trait] +pub trait PersistedSyncState: Sized { + /// Load the state for one `(toolkit, connection)` pair. + /// + /// A connection with nothing stored yields a fresh state rather than an + /// error — that is a first sync, not a failure. A loaded state has its + /// daily budget rolled forward before it is returned, so what a caller + /// spends and later writes back is today's row rather than yesterday's. + /// + /// # Errors + /// + /// Returns an error when the store cannot be reached, or when the stored + /// value is not a decodable state. Both are genuine faults: silently + /// starting from a fresh state would re-ingest everything the connection + /// had already synced. + async fn load( + store: &dyn SyncStateStore, + toolkit: &str, + connection_id: &str, + ) -> anyhow::Result; + + /// Persist this state under its `(toolkit, connection)` key. + /// + /// # Errors + /// + /// Returns an error when the state cannot be serialised or the store + /// rejects the write. + async fn save(&self, store: &dyn SyncStateStore) -> anyhow::Result<()>; } -impl SyncState { - pub fn new(toolkit: impl Into, connection_id: impl Into) -> Self { - Self { - toolkit: toolkit.into(), - connection_id: connection_id.into(), - cursor: None, - synced_ids: HashSet::new(), - item_versions: HashMap::new(), - daily_budget: DailyBudget::default(), - last_seen_id: None, - last_sync_at_ms: None, - run_requests: 0, - run_provider_cost_usd: 0.0, - } - } - - pub fn key(toolkit: &str, connection_id: &str) -> String { - format!("{toolkit}:{connection_id}") - } - - pub fn is_synced(&self, id: &str) -> bool { - self.synced_ids.contains(id) - } - - pub fn mark_synced(&mut self, id: impl Into) { - self.synced_ids.insert(id.into()); - } - - pub fn advance_cursor(&mut self, cursor: impl Into) { - self.cursor = Some(cursor.into()); - } - - pub fn set_last_seen_id(&mut self, id: impl Into) { - self.last_seen_id = Some(id.into()); - } - - pub fn set_last_sync_at_ms(&mut self, timestamp_ms: u64) { - self.last_sync_at_ms = Some(timestamp_ms); - } - - pub fn budget_exhausted(&self) -> bool { - self.daily_budget.is_exhausted() - } - - pub fn budget_remaining(&self) -> u32 { - self.daily_budget.remaining() - } - - pub fn record_requests(&mut self, count: u32) { - self.daily_budget.record_requests(count); - self.run_requests = self.run_requests.saturating_add(count); - } - - pub fn record_action(&mut self, attempts: u32, cost_usd: f64) { - self.record_requests(attempts.max(1)); - if cost_usd.is_finite() && cost_usd > 0.0 { - self.run_provider_cost_usd += cost_usd; - } - } - - pub async fn load( +#[async_trait] +impl PersistedSyncState for SyncState { + async fn load( store: &dyn SyncStateStore, toolkit: &str, connection_id: &str, @@ -171,17 +114,14 @@ impl SyncState { match store.get(STATE_NAMESPACE, &key).await? { Some(value) => { let mut state: Self = serde_json::from_value(value)?; - if state.daily_budget.date != today() { - state.daily_budget.date = today(); - state.daily_budget.requests_used = 0; - } + state.daily_budget.roll_over_if_stale(); Ok(state) } None => Ok(Self::new(toolkit, connection_id)), } } - pub async fn save(&self, store: &dyn SyncStateStore) -> anyhow::Result<()> { + async fn save(&self, store: &dyn SyncStateStore) -> anyhow::Result<()> { let value = serde_json::to_value(self)?; store .set( @@ -193,28 +133,6 @@ impl SyncState { } } -/// First non-empty string at any of `paths` (dot-separated) in `item`. -/// -/// Removed in the §B1a move as dead within this workspace; restored because -/// OpenHuman's raw-coverage integration tests import and exercise it through -/// the pin — "dead here" was measured with too small a grep. -pub fn extract_item_id(item: &serde_json::Value, paths: &[&str]) -> Option { - paths.iter().find_map(|path| { - let value = path - .split('.') - .try_fold(item, |current, segment| current.get(segment))?; - value - .as_str() - .map(str::trim) - .filter(|value| !value.is_empty()) - .map(str::to_owned) - }) -} - -fn today() -> String { - Utc::now().format("%Y-%m-%d").to_string() -} - #[cfg(test)] #[path = "sync_state_tests.rs"] mod tests; diff --git a/crates/tinymemory-core/src/sync/composio/providers/tool_scope.rs b/crates/tinymemory-core/src/sync/composio/providers/tool_scope.rs index ae3d4bef..8044c622 100644 --- a/crates/tinymemory-core/src/sync/composio/providers/tool_scope.rs +++ b/crates/tinymemory-core/src/sync/composio/providers/tool_scope.rs @@ -7,111 +7,25 @@ //! [`CuratedTool`] slice via [`super::ComposioProvider::curated_tools`] //! that pares the surface down to a useful subset and tags every action //! with a [`ToolScope`] so per-user scope preferences can gate execution. - -use serde::{Deserialize, Serialize}; - -/// Classification of how invasive an action is. -/// -/// Used both to filter the agent's visible tool list and to enforce -/// per-user scope preferences at execution time. -#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] -#[serde(rename_all = "lowercase")] -pub enum ToolScope { - /// Pure reads — `GET` / `FETCH` / `LIST` / `SEARCH` / `GET_PROFILE`. - Read, - /// Side-effectful actions that create or mutate user data — - /// `SEND` / `CREATE` / `UPDATE` / `REPLY` / `APPEND`. - Write, - /// Destructive or permission-changing actions — `DELETE` / `TRASH` / - /// `REMOVE` / `MODIFY_LABELS` / `SHARE`. - Admin, -} - -impl ToolScope { - pub fn as_str(self) -> &'static str { - match self { - ToolScope::Read => "read", - ToolScope::Write => "write", - ToolScope::Admin => "admin", - } - } -} - -/// One curated entry in a provider's tool catalog. -/// -/// `slug` is the Composio action slug as returned by `composio_list_tools` -/// (e.g. `"GMAIL_SEND_EMAIL"`). `scope` controls whether the action is -/// gated by the user's read / write / admin preference. -#[derive(Debug, Clone, Copy)] -pub struct CuratedTool { - pub slug: &'static str, - pub scope: ToolScope, -} - -/// Heuristic fallback when we need to gate a tool that isn't in any -/// provider's curated list. Prefer the curated classification when -/// available; only call this when [`super::ComposioProvider::curated_tools`] -/// returned `None` or didn't include the slug. -pub fn classify_unknown(slug: &str) -> ToolScope { - let upper = slug.to_ascii_uppercase(); - // Admin verbs are checked first so e.g. `MODIFY_LABELS` doesn't slip - // into the Write bucket on the `UPDATE`-substring rule. - const ADMIN: &[&str] = &[ - "DELETE", - "TRASH", - "REMOVE", - "MODIFY_LABELS", - "SHARE", - "REVOKE", - "DESTROY", - ]; - const WRITE: &[&str] = &[ - "SEND", "CREATE", "UPDATE", "REPLY", "APPEND", "INSERT", "ADD", "POST", "PATCH", "WRITE", - "DRAFT", - ]; - if ADMIN.iter().any(|kw| upper.contains(kw)) { - return ToolScope::Admin; - } - if WRITE.iter().any(|kw| upper.contains(kw)) { - return ToolScope::Write; - } - ToolScope::Read -} - -/// Look up a slug inside a curated catalog. -pub fn find_curated<'a>(catalog: &'a [CuratedTool], slug: &str) -> Option<&'a CuratedTool> { - catalog.iter().find(|t| t.slug.eq_ignore_ascii_case(slug)) -} - -/// Extract the toolkit slug from a Composio action slug. -/// -/// Most Composio action slugs follow `__…` -/// (e.g. `GMAIL_SEND_EMAIL` → `gmail`). A few toolkit identifiers contain -/// underscores themselves; those need known-prefix handling so connected -/// toolkit checks do not drop actions such as `ZOHO_MAIL_*`. -pub fn toolkit_from_slug(slug: &str) -> Option { - let trimmed = slug.trim(); - if trimmed.is_empty() { - return None; - } - const MULTI_SEGMENT_TOOLKIT_PREFIXES: &[(&str, &str)] = &[ - ("MICROSOFT_TEAMS_", "microsoft_teams"), - ("ONE_DRIVE_", "one_drive"), - ("ZOHO_MAIL_", "zoho_mail"), - ]; - let upper = trimmed.to_ascii_uppercase(); - for (prefix, toolkit) in MULTI_SEGMENT_TOOLKIT_PREFIXES { - if upper.starts_with(prefix) { - return Some((*toolkit).to_string()); - } - } - let prefix = trimmed.split('_').next()?; - if prefix.is_empty() { - None - } else { - Some(prefix.to_ascii_lowercase()) - } -} +//! +//! # Where the definitions live (#5560) +//! +//! All of it moved to [`tinymemory_api::composio::scopes`] and is re-exported +//! here at its historical path. The reason is that the *same verdict* has to be +//! reached on both sides of the module boundary: OpenHuman filters the agent's +//! visible tool list with [`classify_unknown`] and [`find_curated`], and the +//! sync pipelines gate execution with them inside the module. Two copies of the +//! verb-precedence rule would be two different answers to "may this action +//! run", which is not a shape mismatch but a permissions bug. +//! +//! The curated catalogs themselves stay in this crate — see +//! [`super::catalogs`] and the per-toolkit modules. They are provider data +//! rather than contract vocabulary, they change whenever a provider does, and +//! nothing about them has to cross a frame. + +pub use tinymemory_api::composio::scopes::{ + classify_unknown, find_curated, toolkit_from_slug, CuratedTool, ToolScope, +}; #[cfg(test)] #[path = "tool_scope_tests.rs"] diff --git a/crates/tinymemory-core/src/sync/composio/providers/types.rs b/crates/tinymemory-core/src/sync/composio/providers/types.rs index 0b9cf82f..6d251b1e 100644 --- a/crates/tinymemory-core/src/sync/composio/providers/types.rs +++ b/crates/tinymemory-core/src/sync/composio/providers/types.rs @@ -1,7 +1,20 @@ //! Shared types for Composio provider implementations. - -use serde::{Deserialize, Serialize}; -use std::sync::{Arc, Mutex}; +//! +//! # What is here, and what moved down (#5560) +//! +//! The *values* a provider exchanges — the run report, the task envelope, the +//! normalized profile — are defined in the contract crate +//! ([`tinymemory_api::composio`]) and re-exported below at their historical +//! paths. OpenHuman names every one of them in its own signatures, so they had +//! to be reachable without a compile-time link to this crate; they are inert +//! serde data, so moving them cost nothing. +//! +//! What stayed is [`ProviderContext`], and it stayed because it is not a value: +//! it holds an `Arc`, resolves a Composio client through the host seam +//! on every call, and awaits an HTTP round-trip. None of that may enter the +//! contract crate. + +use std::sync::Arc; // Test-only: the tests below build a `TestHostConfig` and call // `MemoryHostConfig` methods on it directly. Production code in this module @@ -13,281 +26,34 @@ use crate::composio_host::{self, ComposioExecuteResponse}; use crate::config_loader as config_rpc; use crate::Config; -/// Reason a sync was triggered. Providers can use this to decide -/// whether to do a full backfill or an incremental pull. -#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] -#[serde(rename_all = "snake_case")] -pub enum SyncReason { - /// First sync immediately after an OAuth handoff completes. - ConnectionCreated, - /// Periodic background sync from the scheduler. - Periodic, - /// Explicit user-driven sync from RPC / UI. - Manual, -} - -impl SyncReason { - pub fn as_str(&self) -> &'static str { - match self { - SyncReason::ConnectionCreated => "connection_created", - SyncReason::Periodic => "periodic", - SyncReason::Manual => "manual", - } - } -} - -/// What kind of work an ingested task implies. GitHub's issues-and-PRs -/// search returns both shapes, and the job differs fundamentally — -/// *resolve* an issue vs *review* a pull request — so providers tag each -/// task and the `task_sources` enrichment phrases the objective / agent -/// prompt accordingly (the triage LLM then knows what to do). Providers -/// that don't distinguish (notion, linear, clickup) leave this `Generic`. -#[derive(Debug, Clone, Copy, Default, Serialize, Deserialize, PartialEq, Eq)] -#[serde(rename_all = "snake_case")] -pub enum TaskKind { - /// No issue/PR distinction — the default for non-code providers. - #[default] - Generic, - /// A tracker issue: the job is to resolve / implement it. - Issue, - /// A pull request: the job is to review it (read the diff, give feedback). - PullRequest, -} - -impl TaskKind { - /// Stable lowercase tag, mirrored into the card's `source_metadata`. - pub fn as_str(&self) -> &'static str { - match self { - TaskKind::Generic => "generic", - TaskKind::Issue => "issue", - TaskKind::PullRequest => "pull_request", - } - } -} - -/// Normalized user profile shape returned by every provider. -/// -/// The shared fields (`display_name`, `email`, `username`, `avatar_url`, -/// `profile_url`) -/// cover what the desktop UI actually needs to render a connected -/// account card. Anything provider-specific (Gmail's `messagesTotal`, -/// Notion's workspace ids, …) goes into [`extras`](Self::extras) so -/// callers don't have to widen the shape every time a new toolkit -/// lands. -#[derive(Debug, Clone, Default, Serialize, Deserialize)] -pub struct ProviderUserProfile { - pub toolkit: String, - pub connection_id: Option, - pub display_name: Option, - pub email: Option, - pub username: Option, - pub avatar_url: Option, - pub profile_url: Option, - /// Provider-specific extras (raw JSON object). - #[serde(default)] - pub extras: serde_json::Value, -} - -/// Result of a provider sync run. Mostly used for logging + UI status. -#[derive(Debug, Clone, Default, Serialize, Deserialize)] -pub struct SyncOutcome { - pub toolkit: String, - pub connection_id: Option, - pub reason: String, - pub items_ingested: usize, - pub started_at_ms: u64, - pub finished_at_ms: u64, - pub summary: String, - /// Provider-specific extras (raw JSON object). - #[serde(default)] - pub details: serde_json::Value, -} - -impl SyncOutcome { - pub fn elapsed_ms(&self) -> u64 { - self.finished_at_ms.saturating_sub(self.started_at_ms) - } -} - -/// A provider-agnostic, structured work item produced by -/// [`super::ComposioProvider::fetch_tasks`]. -/// -/// Unlike the `sync()` path — which persists upstream items into the -/// memory store as passive context — `fetch_tasks` *returns* normalized -/// tasks so the `task_sources` domain can enrich them and route them -/// onto the agent's todo board. Every native task provider (github, -/// notion, linear, clickup) maps its upstream payload shape into this -/// common envelope. +/// The Composio sync vocabulary, defined in the contract crate. /// -/// `source_id` is left empty by providers and stamped by the -/// `task_sources` pipeline with the originating `TaskSource.id` — a -/// provider has no knowledge of which configured source asked for the -/// fetch. -#[derive(Debug, Clone, Default, Serialize, Deserialize, PartialEq)] -#[serde(rename_all = "camelCase")] -pub struct NormalizedTask { - /// Upstream provider's stable id for the item (issue/task/page id). - pub external_id: String, - /// The `TaskSource.id` that produced this task. Empty until the - /// pipeline stamps it. - #[serde(default)] - pub source_id: String, - /// Toolkit slug, e.g. `"github"`. - pub provider: String, - /// Whether this task is an issue, a pull request, or undifferentiated. - /// Drives intent-aware objective / prompt phrasing in enrichment. - #[serde(default)] - pub kind: TaskKind, - pub title: String, - #[serde(default, skip_serializing_if = "Option::is_none")] - pub body: Option, - #[serde(default, skip_serializing_if = "Option::is_none")] - pub url: Option, - #[serde(default, skip_serializing_if = "Option::is_none")] - pub status: Option, - #[serde(default, skip_serializing_if = "Option::is_none")] - pub assignee: Option, - /// Due date as an ISO-8601 string, when the provider exposes one. - #[serde(default, skip_serializing_if = "Option::is_none")] - pub due: Option, - #[serde(default)] - pub labels: Vec, - #[serde(default, skip_serializing_if = "Option::is_none")] - pub priority: Option, - /// Last-updated ISO-8601 timestamp — used for cursor advancement and - /// edit-aware dedup (`{external_id}@{updated_at}`). - #[serde(default, skip_serializing_if = "Option::is_none")] - pub updated_at: Option, - /// The raw upstream payload, retained for enrichment / debugging. - #[serde(default)] - pub raw: serde_json::Value, -} - -/// A selectable upstream task container (board / database / list) used to -/// populate a picker so the user chooses from a list instead of pasting a -/// raw id. Today this is a Notion database, later a Linear team or ClickUp -/// list. Surfaced to the task-source UI as `{ id, title }`. -#[derive(Debug, Clone, Default, Serialize, Deserialize, PartialEq)] -#[serde(rename_all = "camelCase")] -pub struct TaskContainer { - /// Provider-native id (e.g. a Notion database id) used as the filter id. - pub id: String, - /// Human-readable label for the picker. - pub title: String, -} - -/// Provider-agnostic filter passed into -/// [`super::ComposioProvider::fetch_tasks`]. -/// -/// The `task_sources` domain builds this from a user-configured, -/// per-provider `FilterSpec`. Each provider reads only the fields that -/// apply to it (github reads `repo`/`labels`; notion reads -/// `database_id`; linear/clickup read `team_id`; …) and ignores the -/// rest. `extra` is a free-form escape hatch surfaced in the UI for -/// advanced provider-native query fragments. -/// How the GitHub task-source fetch reaches GitHub. Shipped desktop users -/// connect GitHub via Composio OAuth (no `gh` on PATH, no `GITHUB_TOKEN`), -/// while local dev / self-host setups often have the reverse. `Auto` does the -/// right thing for both; `Composio` / `Local` force a path when the user wants. -#[derive(Debug, Clone, Copy, Default, Serialize, Deserialize, PartialEq, Eq)] -#[serde(rename_all = "snake_case")] -pub enum GithubFetchMode { - /// Try the connected Composio account first; fall back to local `gh`/REST - /// only when Composio is unavailable. The safe default — no regression for - /// shipped users, still a true fallback for local/dev. - #[default] - Auto, - /// Force the connected Composio account (classic shipped-app behaviour). - Composio, - /// Force local `gh` CLI / REST with a `GH_TOKEN`/`GITHUB_TOKEN` env token. - Local, -} - -#[derive(Debug, Clone, Default, Serialize, Deserialize, PartialEq)] -#[serde(rename_all = "camelCase")] -pub struct TaskFetchFilter { - /// Scope to items assigned to (or involving) the authenticated user. - #[serde(default)] - pub assignee_is_me: bool, - /// GitHub fetch path selector (Composio vs local `gh`/REST). Default `Auto`. - #[serde(default)] - pub github_fetch_mode: GithubFetchMode, - /// GitHub `owner/name` repository scope. - #[serde(default, skip_serializing_if = "Option::is_none")] - pub repo: Option, - /// GitHub label filter. - #[serde(default)] - pub labels: Vec, - /// Issue/task state filter (e.g. `"open"`, `"todo"`). - #[serde(default, skip_serializing_if = "Option::is_none")] - pub state: Option, - /// Notion database (board) id. - #[serde(default, skip_serializing_if = "Option::is_none")] - pub database_id: Option, - /// Notion status property filter. - #[serde(default, skip_serializing_if = "Option::is_none")] - pub status: Option, - /// Linear / ClickUp team (workspace) id. - #[serde(default, skip_serializing_if = "Option::is_none")] - pub team_id: Option, - /// ClickUp list id. - #[serde(default, skip_serializing_if = "Option::is_none")] - pub list_id: Option, - /// Free-form provider-native filter fragment (advanced). - #[serde(default)] - pub extra: serde_json::Value, - /// Hard cap on how many tasks a single fetch returns. - #[serde(default)] - pub max: u32, -} - -impl TaskFetchFilter { - /// Effective per-fetch item cap, defaulting to a safe bound when the - /// caller leaves `max` unset (0). - pub fn effective_max(&self) -> usize { - if self.max == 0 { - 25 - } else { - self.max as usize - } - } -} +/// Re-exported at this path because roughly a hundred call sites here and in +/// OpenHuman already spell these `providers::SyncOutcome`, +/// `providers::NormalizedTask` and so on, and the move delivers the decoupling +/// without spending that churn. +pub use tinymemory_api::composio::{ + ComposioUsage, ComposioUsageHandle, GithubFetchMode, NormalizedTask, ProviderUserProfile, + SyncOutcome, SyncReason, TaskContainer, TaskFetchFilter, TaskKind, +}; /// Per-call context handed to provider methods. /// -/// `connection_id` is `None` when a method runs in a "no specific -/// connection" mode (e.g. an across-the-board periodic sync that -/// already iterated). For per-connection paths it is always populated. +/// `connection_id` is `None` when a method runs in a "no specific connection" +/// mode (e.g. an across-the-board periodic sync that already iterated). For +/// per-connection paths it is always populated. /// /// **Mode-aware dispatch (#1710)**: pre-fix, `ProviderContext` cached a /// pre-baked `ComposioClient` built once at construction time. Toggling -/// `composio.mode = "direct"` mid-session left provider syncs still -/// routing through the backend tinyhumans tenant. The current shape -/// keeps an [`Arc`] and resolves the underlying client per call -/// through [`ProviderContext::execute`], mirroring the agent-tool -/// migration in the host's `integrations::composio::tools::ComposioExecuteTool`. -/// Per-sync accumulator for Composio billable-action usage. +/// `composio.mode = "direct"` mid-session left provider syncs still routing +/// through the backend tinyhumans tenant. The current shape keeps an +/// [`Arc`] and resolves the underlying client per call through +/// [`ProviderContext::execute`], mirroring the agent-tool migration in the +/// host's `integrations::composio::tools::ComposioExecuteTool`. /// -/// Lives behind a shared handle on [`ProviderContext`] so the single -/// `execute` chokepoint can tally every action a provider fires during one -/// sync run, regardless of which provider (gmail / slack / github / notion / -/// linear / clickup) or how many pages it paginates. -/// [`crate::sync::composio::run_connection_sync`] returns -/// the final tally alongside the [`SyncOutcome`] for the sync audit log -/// (#3111). -#[derive(Debug, Clone, Default, Serialize, Deserialize)] -pub struct ComposioUsage { - /// Count of `execute` calls that returned a response this run. - pub actions_called: u32, - /// Sum of each response's backend-reported `cost_usd`. - pub cost_usd: f64, -} - -/// Shared, interior-mutable handle to a [`ComposioUsage`] tally. Cloning a -/// [`ProviderContext`] shares the same underlying counter, so the count is -/// stable no matter how the context is passed around within a sync. -pub type ComposioUsageHandle = Arc>; - +/// This is the one item in this module that is *not* contract vocabulary: a +/// context is a live handle onto the host seam, not something a frame can +/// carry. See the module docs. #[derive(Clone)] pub struct ProviderContext { pub config: Arc, diff --git a/crates/tinymemory-core/src/sync/composio/providers/user_scopes.rs b/crates/tinymemory-core/src/sync/composio/providers/user_scopes.rs index af7beecc..78b70872 100644 --- a/crates/tinymemory-core/src/sync/composio/providers/user_scopes.rs +++ b/crates/tinymemory-core/src/sync/composio/providers/user_scopes.rs @@ -11,53 +11,30 @@ //! Storage uses the same KV surface as [`super::sync_state`] //! (`MemoryClient::kv_get` / `kv_set`) under a dedicated namespace so //! prefs survive process restarts without any extra file management. - -use serde::{Deserialize, Serialize}; +//! +//! # Where the shape lives (#5560) +//! +//! [`UserScopePref`] itself is defined in the contract crate and re-exported +//! here: OpenHuman reads a preference to decide which Composio actions to show +//! the user and which to offer the agent, so the *shape* — and the defaults +//! that decide what a brand-new connection may do — has to be nameable without +//! a compile-time link to this crate. +//! +//! The three functions below stayed, because reading and writing a preference +//! is I/O against a memory client and the contract crate holds none. use crate::store::MemoryClientRef; -use super::tool_scope::ToolScope; +/// The preference shape, defined in the contract crate. +/// +/// Re-exported at this path so every historical +/// `providers::user_scopes::UserScopePref` reference keeps resolving. +pub use tinymemory_api::composio::scopes::UserScopePref; /// KV namespace for scope prefs. Separate from `composio-sync-state` so /// the two never collide. const KV_NAMESPACE: &str = "composio-user-scopes"; -/// Per-toolkit scope preference. -#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] -pub struct UserScopePref { - #[serde(default = "default_true")] - pub read: bool, - #[serde(default = "default_true")] - pub write: bool, - #[serde(default)] - pub admin: bool, -} - -fn default_true() -> bool { - true -} - -impl Default for UserScopePref { - fn default() -> Self { - Self { - read: true, - write: true, - admin: false, - } - } -} - -impl UserScopePref { - /// Returns `true` if the given scope is enabled in this preference. - pub fn allows(&self, scope: ToolScope) -> bool { - match scope { - ToolScope::Read => self.read, - ToolScope::Write => self.write, - ToolScope::Admin => self.admin, - } - } -} - fn kv_key(toolkit: &str) -> String { toolkit.trim().to_ascii_lowercase() } diff --git a/crates/tinymemory-core/src/sync/composio/providers/user_scopes_tests.rs b/crates/tinymemory-core/src/sync/composio/providers/user_scopes_tests.rs index b6c9e0ee..eea0890d 100644 --- a/crates/tinymemory-core/src/sync/composio/providers/user_scopes_tests.rs +++ b/crates/tinymemory-core/src/sync/composio/providers/user_scopes_tests.rs @@ -1,4 +1,7 @@ use super::*; +// `allows` is defined on the contract's type, so `ToolScope` is no longer +// imported by the module under test and has to be named here (#5560). +use super::super::tool_scope::ToolScope; use crate::store::MemoryClient; use std::sync::Arc; use tempfile::TempDir; diff --git a/crates/tinymemory-core/src/sync/pipelines/composio/orchestrator.rs b/crates/tinymemory-core/src/sync/pipelines/composio/orchestrator.rs index fd201371..1f19dbfa 100644 --- a/crates/tinymemory-core/src/sync/pipelines/composio/orchestrator.rs +++ b/crates/tinymemory-core/src/sync/pipelines/composio/orchestrator.rs @@ -5,7 +5,7 @@ use serde_json::Value; use super::client::ActionExecutor; use super::page_size::{apply_page_size, is_payload_too_large, shrink_page_size}; -use crate::sync::composio::providers::sync_state::SyncState; +use crate::sync::composio::providers::sync_state::{PersistedSyncState, SyncState}; use crate::sync::pipelines::traits::PipelineConfig; use crate::sync::pipelines::traits::{ SkillDocument, SyncContext, SyncEvent, SyncOutcome, SyncRunError, SyncStage, diff --git a/crates/tinymemory-core/src/sync/pipelines/composio/providers/provider_tests.rs b/crates/tinymemory-core/src/sync/pipelines/composio/providers/provider_tests.rs index a3416a73..073817bf 100644 --- a/crates/tinymemory-core/src/sync/pipelines/composio/providers/provider_tests.rs +++ b/crates/tinymemory-core/src/sync/pipelines/composio/providers/provider_tests.rs @@ -11,7 +11,7 @@ use super::{ GoogleDriveSyncPipeline, GoogleSheetsSyncPipeline, LinearSyncPipeline, NotionSyncPipeline, OutlookSyncPipeline, SlackSearchBackfillPipeline, SlackSyncPipeline, TodoistSyncPipeline, }; -use crate::sync::composio::providers::sync_state::{SyncState, SyncStateStore}; +use crate::sync::composio::providers::sync_state::{PersistedSyncState, SyncState, SyncStateStore}; use crate::sync::pipelines::composio::{ ActionExecutor, ComposioClient, ExecuteResponse, IncrementalSource, SyncItem, SyncScope, }; diff --git a/crates/tinymemory-core/src/sync/pipelines/composio/providers/slack.rs b/crates/tinymemory-core/src/sync/pipelines/composio/providers/slack.rs index 3953d6fb..5a7ea599 100644 --- a/crates/tinymemory-core/src/sync/pipelines/composio/providers/slack.rs +++ b/crates/tinymemory-core/src/sync/pipelines/composio/providers/slack.rs @@ -8,7 +8,7 @@ use super::common::{checked_execute, document, first_array, pick_str}; use super::slack_parse::{ decode_cursors, next_cursor, parse_ts, replace_mentions, search_matches, search_total_pages, }; -use crate::sync::composio::providers::sync_state::SyncState; +use crate::sync::composio::providers::sync_state::{PersistedSyncState, SyncState}; use crate::sync::pipelines::composio::{ run_incremental_sync, ActionExecutor, ComposioClient, IncrementalSource, PageFetch, SyncItem, SyncScope, diff --git a/crates/tinymemory-core/tests/composio_gmail_non_tinycortex_e2e.rs b/crates/tinymemory-core/tests/composio_gmail_non_tinycortex_e2e.rs index 4488a62a..a6448443 100644 --- a/crates/tinymemory-core/tests/composio_gmail_non_tinycortex_e2e.rs +++ b/crates/tinymemory-core/tests/composio_gmail_non_tinycortex_e2e.rs @@ -101,7 +101,12 @@ impl tinymemory_api::host::EmbeddingHost for NoopEmbeddingHost { Ok(Box::new(tinymemory_api::host::NoopEmbedding)) } } -use tinymemory_core::sync::composio::providers::sync_state::{SyncState, KV_NAMESPACE}; +// `load`/`save` are the extension trait, not inherent methods: `SyncState` +// itself moved to the contract crate, which stays free of I/O, so persistence +// lives here in the engine and arrives through `PersistedSyncState`. +use tinymemory_core::sync::composio::providers::sync_state::{ + PersistedSyncState, SyncState, KV_NAMESPACE, +}; use tinymemory_core::sync::pipelines::composio::ComposioClient; use tinymemory_core::sync::pipelines::composio::GmailSyncPipeline; use tinymemory_core::sync::pipelines::dispatcher::SyncDispatcher; diff --git a/crates/tinymemory-module/src/lib.rs b/crates/tinymemory-module/src/lib.rs index 0411ee77..fb3ea52c 100644 --- a/crates/tinymemory-module/src/lib.rs +++ b/crates/tinymemory-module/src/lib.rs @@ -616,6 +616,16 @@ mod exports { // `BusComposioHost::probe`, which blocks its caller for one bus round // trip and is bounded at twice per tick — see the note on `probe` for // why that bridge blocks at all. + // + // Nor do the long-running on-demand members. `RunConnectionSync` and + // `RebuildFromRawArchive` await network and inference, so they yield + // their worker between every step; every synchronous read behind + // `SyncAuditLog`, `SyncStatuses` and `RawArchiveCoverage` hops to + // `spawn_blocking`, which draws on the blocking pool rather than on + // these eight. `IngestCodingSessions` is the one that occupies a thread + // outright for its whole run — the persona pipeline is not `Send`, so + // the driver drives it from a blocking worker — and that is again the + // blocking pool, not a runtime worker. worker_threads = 8, provides = ["ai.tinyhumans.tinymemory.Memory"], methods = [ @@ -735,6 +745,23 @@ mod exports { // Tree, structural: the forest walk and its leaf edge. "SummaryForest", "RecentLeaves", + // Tree, by source scope: the flush a user triggers on one source. + "FlushSourceTree", + // Maintenance, typed: the diagnosis an operator or an agent reads, + // beside the uniform report a scheduler reads. + "Diagnose", + // Source sync this process runs itself. The periodic loops already + // live here; these are the on-demand half plus what past runs cost. + "RunConnectionSync", + "SourceSyncState", + "SyncAuditLog", + "EstimateSyncCostUsd", + "SyncStatuses", + "RawArchiveCoverage", + "RebuildFromRawArchive", + // Local coding-agent transcripts. + "CodingSessionStatus", + "IngestCodingSessions", ], signals = [], // The host's embedder is deliberately NOT declared as `requires`. That diff --git a/crates/tinymemory-module/src/service/mod.rs b/crates/tinymemory-module/src/service/mod.rs index c5c12a51..5ae63de4 100644 --- a/crates/tinymemory-module/src/service/mod.rs +++ b/crates/tinymemory-module/src/service/mod.rs @@ -60,6 +60,20 @@ //! //! ForgetMatching(selector) -> ForgetOutcome //! PurgeAll() -> PurgeOutcome +//! +//! FlushSourceTree(source_scope) -> u64 +//! Diagnose() -> Diagnosis +//! +//! RunConnectionSync(toolkit, connection_id) -> SyncRunOutcome +//! SourceSyncState(toolkit, connection_id) -> Option +//! SyncAuditLog(limit) -> [SyncAuditEntry] +//! EstimateSyncCostUsd(input_tokens, output_tokens) -> f64 +//! SyncStatuses() -> [SourceSyncStatus] +//! RawArchiveCoverage(tree_scope, archive_source_id) -> RawArchiveCoverage +//! RebuildFromRawArchive(tree_scope, archive_source_id) -> RawRebuildOutcome +//! +//! CodingSessionStatus() -> [CodingSessionSource] +//! IngestCodingSessions(request) -> CodingSessionIngestReport //! ``` //! //! # Source scope crosses as an argument, never as ambient state @@ -144,6 +158,7 @@ use tinymemory_api::provider::types::{ use tinymemory_api::provider::chunks::{ ChunkDetail, ChunkEmbedding, ChunkListRow, ChunkQuery, SourceTotal, }; +use tinymemory_api::provider::diagnosis::Diagnosis; use tinymemory_api::provider::episodic::{ConversationSegment, EpisodicEvent, EpisodicTurn}; use tinymemory_api::provider::people::{ AddressBookSeedOutcome, PersonHandle, PersonInteraction, PersonRecord, PersonScore, @@ -154,6 +169,13 @@ use tinymemory_api::provider::retrieval::{ CoverWindowQuery, EntityMatch, FastRetrieveQuery, RetrievalHit, RetrievalResponse, SourceRetrievalQuery, }; +use tinymemory_api::provider::sessions::{ + CodingSessionIngestReport, CodingSessionIngestRequest, CodingSessionSource, +}; +use tinymemory_api::provider::sync::{ + RawArchiveCoverage, RawRebuildOutcome, SourceSyncState, SourceSyncStatus, SyncAuditEntry, + SyncRunOutcome, +}; use tinymemory_api::provider::MemoryProvider; use tinymemory_api::recall::OwnedRecallOpts; use tinymemory_api::tool_memory::ToolMemoryRule; @@ -1660,6 +1682,179 @@ impl MemoryService { .await .map_err(|error| into_bus_error(&error)) } + + // ── Tree, by source scope ─────────────────────────────────────────────── + + /// Seal and cascade one source's tree now. + /// + /// Addressed by source scope rather than by namespace, unlike every other + /// member of this family: the caller is looking at one connected source and + /// that is the identity it holds. The scope is **not** logged — it carries + /// a platform and a connection id, and the second is user data. + async fn flush_source_tree(&self, source_scope: String) -> BusResult { + require_family!(self, as_tree, Capability::Tree) + .flush_source_tree(&source_scope) + .await + .map_err(|error| into_bus_error(&error)) + } + + // ── Maintenance, typed ────────────────────────────────────────────────── + + /// The typed, per-stage pipeline diagnosis. + /// + /// Beside `Doctor` rather than replacing it. `Doctor` returns the uniform + /// `MaintenanceReport` a scheduler reads across all four upkeep calls; this + /// returns the classified causes, degradation flags and counters an + /// operator or an agent acts on. Both come from one pass driver-side. + /// + /// Not size-checked: the report is bounded by the driver's stage list. + async fn diagnose(&self) -> BusResult { + require_family!(self, as_maintenance, Capability::Maintenance) + .diagnose() + .await + .map_err(|error| into_bus_error(&error)) + } + + // ── Source sync the driver runs itself ────────────────────────────────── + + /// Sync one connection now. + /// + /// The manual "sync now" a user presses. The periodic loops already run in + /// this process; this is the on-demand half, which a schedule cannot + /// express. + /// + /// Neither argument is logged. A toolkit is harmless, a connection id is + /// not, and logging one without the other says nothing useful. + async fn run_connection_sync( + &self, + toolkit: String, + connection_id: String, + ) -> BusResult { + require_family!(self, as_source_sync, Capability::SourceSync) + .run_connection_sync(&toolkit, &connection_id) + .await + .map_err(|error| into_bus_error(&error)) + } + + /// The persisted cursor, dedup and budget state for one connection. + /// + /// `None` is "never synced", which is a state and not an error — a status + /// list covering every connection would otherwise be all errors on a fresh + /// install. + async fn source_sync_state( + &self, + toolkit: String, + connection_id: String, + ) -> BusResult> { + require_family!(self, as_source_sync, Capability::SourceSync) + .source_sync_state(&toolkit, &connection_id) + .await + .map_err(|error| into_bus_error(&error)) + } + + /// Past sync runs, newest first. + /// + /// Size-checked, and the only member of this family that needs to be: the + /// audit log is append-only for the life of a workspace, so it is the one + /// response here that grows without a bound the caller controls. `limit` + /// bounds the *count*; the bytes are bounded here, and a refusal names + /// `BudgetExceeded` so the caller knows to ask for fewer rows rather than + /// reading a silently short log as a complete one. + async fn sync_audit_log(&self, limit: Option) -> BusResult> { + let entries = require_family!(self, as_source_sync, Capability::SourceSync) + .sync_audit_log(limit) + .await + .map_err(|error| into_bus_error(&error))?; + ensure_response_fits(&entries, "SyncAuditLog")?; + Ok(entries) + } + + /// Price a token count at the driver's own rate. + /// + /// A bus round trip for two multiplications, and deliberately so: the same + /// constants stamped `estimated_cost_usd` onto every audit row above, and a + /// caller holding its own copy would show a projection and a historical + /// total computed at two different prices. + async fn estimate_sync_cost_usd( + &self, + input_tokens: u64, + output_tokens: u64, + ) -> BusResult { + require_family!(self, as_source_sync, Capability::SourceSync) + .estimate_sync_cost_usd(input_tokens, output_tokens) + .await + .map_err(|error| into_bus_error(&error)) + } + + /// Per-provider sync progress, derived from stored content. + /// + /// Not size-checked: one row per provider, and a store with enough distinct + /// providers to fill a frame has a different problem — the same reasoning + /// `Namespaces` is left unchecked under. + async fn sync_statuses(&self) -> BusResult> { + require_family!(self, as_source_sync, Capability::SourceSync) + .sync_statuses() + .await + .map_err(|error| into_bus_error(&error)) + } + + /// How much of one raw archive its summary tree covers. + async fn raw_archive_coverage( + &self, + tree_scope: String, + archive_source_id: String, + ) -> BusResult { + require_family!(self, as_source_sync, Capability::SourceSync) + .raw_archive_coverage(&tree_scope, &archive_source_id) + .await + .map_err(|error| into_bus_error(&error)) + } + + /// Re-derive a summary tree from its raw archive. + /// + /// Costs inference and can run long. It is a call rather than a background + /// job on purpose: the module holds no notion of a caller's request, so a + /// fire-and-forget rebuild would have nowhere to report to and no way to be + /// cancelled. A caller that does not want to wait runs it off its own task. + async fn rebuild_from_raw_archive( + &self, + tree_scope: String, + archive_source_id: String, + ) -> BusResult { + require_family!(self, as_source_sync, Capability::SourceSync) + .rebuild_from_raw_archive(&tree_scope, &archive_source_id) + .await + .map_err(|error| into_bus_error(&error)) + } + + // ── Local coding-agent transcripts ────────────────────────────────────── + + /// What each supported coding agent's session store holds. + /// + /// Not size-checked: one row per agent the driver supports. + async fn coding_session_status(&self) -> BusResult> { + require_family!(self, as_coding_sessions, Capability::CodingSessions) + .coding_session_status() + .await + .map_err(|error| into_bus_error(&error)) + } + + /// Distil coding sessions into observations. + /// + /// The longest-running member on this object: one or more sequential model + /// calls per session, bounded by the request's session count and by the + /// driver's own clamp on it. A caller enforcing a deadline does so on its + /// own side — abandoning a run here would leave the driver's per-file state + /// disagreeing with what it wrote. + async fn ingest_coding_sessions( + &self, + request: CodingSessionIngestRequest, + ) -> BusResult { + require_family!(self, as_coding_sessions, Capability::CodingSessions) + .ingest_coding_sessions(request) + .await + .map_err(|error| into_bus_error(&error)) + } } /// The response-size ceiling for a method that returns a list of entries. diff --git a/crates/tinymemory-module/src/service/test.rs b/crates/tinymemory-module/src/service/test.rs index 14e73365..9e1640b0 100644 --- a/crates/tinymemory-module/src/service/test.rs +++ b/crates/tinymemory-module/src/service/test.rs @@ -716,3 +716,44 @@ fn the_served_members_are_exactly_the_published_contract() { "the two lists hold the same members in different orders" ); } + +#[tokio::test] +async fn the_two_new_families_are_gated_on_their_own_capability() { + // `test_provider` wraps a bare `Memory` backend through the mandatory + // composition, so it advertises Core/Recall/Portability and nothing else. + // The gate has to be per family: a method reached on a driver that does not + // serve its family must refuse by name, not fall through to whatever the + // trait's default body happens to return. + let service = super::MemoryService::new(test_provider()); + + let refusal = |error: BusError| match error { + BusError::MethodFailed { name, .. } => name, + other => panic!("expected a named MethodFailed, got {other:?}"), + }; + + let error = service + .run_connection_sync("gmail".to_string(), "conn-1".to_string()) + .await + .expect_err("a driver without the source-sync family must refuse"); + assert_eq!(refusal(error), wire::UNSUPPORTED); + + let error = service + .coding_session_status() + .await + .expect_err("a driver without the coding-sessions family must refuse"); + assert_eq!(refusal(error), wire::UNSUPPORTED); + + // The two members added to *existing* families refuse through their own + // family's gate — Tree and Maintenance — rather than through a new one. + let error = service + .flush_source_tree("gmail:conn-1".to_string()) + .await + .expect_err("a driver without the tree family must refuse"); + assert_eq!(refusal(error), wire::UNSUPPORTED); + + let error = service + .diagnose() + .await + .expect_err("a driver without the maintenance family must refuse"); + assert_eq!(refusal(error), wire::UNSUPPORTED); +} diff --git a/crates/tinymemory-module/tests/module_e2e.rs b/crates/tinymemory-module/tests/module_e2e.rs index 02608fcb..99837fb6 100644 --- a/crates/tinymemory-module/tests/module_e2e.rs +++ b/crates/tinymemory-module/tests/module_e2e.rs @@ -691,6 +691,17 @@ const EXPECTED_METHODS: &[&str] = &[ "RecallNamespaceRecent", "SummaryForest", "RecentLeaves", + "FlushSourceTree", + "Diagnose", + "RunConnectionSync", + "SourceSyncState", + "SyncAuditLog", + "EstimateSyncCostUsd", + "SyncStatuses", + "RawArchiveCoverage", + "RebuildFromRawArchive", + "CodingSessionStatus", + "IngestCodingSessions", ]; #[tokio::test] @@ -795,10 +806,10 @@ async fn what_is_written_lands_in_the_workspace_it_was_given() { #[ignore = "drives a real dlopen'ed module; must be the only such test in the process — see the module docs"] async fn every_declared_method_is_actually_routed() { // Issue #18 §E5 asks the E2E to cover every family the module advertises. - // It advertises `Capabilities::all()` — eighteen families — and the tests + // It advertises `Capabilities::all()` — twenty families — and the tests // above exercise three of them. // - // Rather than eighteen bespoke round trips, this asserts the property that + // Rather than twenty bespoke round trips, this asserts the property that // makes the advertisement honest at this layer: every method the manifest // declares is actually *reachable*. `the_manifest_declares_every_method_the // _module_serves` compares two lists and would pass for a method that is diff --git a/crates/tinymemory-sources/src/lib.rs b/crates/tinymemory-sources/src/lib.rs index 56c56fc4..f4ff7779 100644 --- a/crates/tinymemory-sources/src/lib.rs +++ b/crates/tinymemory-sources/src/lib.rs @@ -57,7 +57,9 @@ pub type SourceResult = Result; /// only its previous address. pub const FOLDER_FILE_SIZE_CAP_BYTES: u64 = 10 * 1024 * 1024; -pub use registry::{memory_sync_defaults_for_toolkit, ComposioUpsertTarget, SourceRegistry}; +pub use registry::{ + apply_kind_defaults, memory_sync_defaults_for_toolkit, ComposioUpsertTarget, SourceRegistry, +}; pub use types::{ ContentType, MemorySourceEntry, MemorySourcePatch, SourceContent, SourceItem, SourceKind, }; diff --git a/crates/tinymemory-sources/src/registry.rs b/crates/tinymemory-sources/src/registry.rs index f22b33e5..88a6a7f4 100644 --- a/crates/tinymemory-sources/src/registry.rs +++ b/crates/tinymemory-sources/src/registry.rs @@ -64,6 +64,43 @@ pub fn memory_sync_defaults_for_toolkit(toolkit: &str) -> (Option, Option { + if entry.max_prs.is_none() { + entry.max_prs = Some(10); + } + if entry.max_issues.is_none() { + entry.max_issues = Some(10); + } + if entry.max_commits.is_none() { + entry.max_commits = Some(50); + } + } + SourceKind::RssFeed => { + if entry.max_items.is_none() { + entry.max_items = Some(20); + } + } + SourceKind::TwitterQuery if entry.since_days.is_none() => { + entry.since_days = Some(7); + } + _ => {} + } +} + /// A registry of [`MemorySourceEntry`] values backed by a TOML config file. /// /// Construct one with [`SourceRegistry::new`], pointing at the `config.toml` diff --git a/crates/tinymemory-sources/src/registry_tests.rs b/crates/tinymemory-sources/src/registry_tests.rs index 9932b5bb..5ba235dd 100644 --- a/crates/tinymemory-sources/src/registry_tests.rs +++ b/crates/tinymemory-sources/src/registry_tests.rs @@ -436,3 +436,76 @@ fn every_mutation_path_leaves_the_config_owner_only() { assert!(reg.remove("src_2").unwrap()); assert_eq!(mode_of(&path) & 0o077, 0, "remove() widened the config"); } + +// ── apply_kind_defaults ───────────────────────────────────────────────────── +// +// Moved here from the engine crate in #5560 so a host can fill a new entry's +// caps without linking the engine. The defaults are the ones the retroactive +// Composio caps migration also applies, so any change here is a change to what +// already-registered sources are reconciled against. + +fn entry_of_kind(kind: SourceKind) -> MemorySourceEntry { + let mut entry = folder_entry("defaults"); + entry.kind = kind; + entry +} + +#[test] +fn github_defaults_fill_only_the_caps_left_unset() { + let mut entry = entry_of_kind(SourceKind::GithubRepo); + entry.max_issues = Some(3); + apply_kind_defaults(&mut entry); + assert_eq!(entry.max_prs, Some(10)); + assert_eq!(entry.max_issues, Some(3), "a user-set cap must survive"); + assert_eq!(entry.max_commits, Some(50)); +} + +#[test] +fn an_rss_feed_gets_an_item_cap() { + let mut entry = entry_of_kind(SourceKind::RssFeed); + apply_kind_defaults(&mut entry); + assert_eq!(entry.max_items, Some(20)); +} + +#[test] +fn a_twitter_query_gets_a_lookback_window() { + let mut entry = entry_of_kind(SourceKind::TwitterQuery); + apply_kind_defaults(&mut entry); + assert_eq!(entry.since_days, Some(7)); + + entry.since_days = Some(2); + apply_kind_defaults(&mut entry); + assert_eq!(entry.since_days, Some(2), "a user-set window must survive"); +} + +#[test] +fn kinds_with_no_defaults_are_left_alone() { + // Composio caps come from the toolkit slug at upsert time, which this + // function does not have; folders and web pages have no caps at all. + for kind in [ + SourceKind::Composio, + SourceKind::Conversation, + SourceKind::Folder, + SourceKind::WebPage, + ] { + let mut entry = entry_of_kind(kind.clone()); + apply_kind_defaults(&mut entry); + assert!(entry.max_items.is_none(), "{kind:?} gained an item cap"); + assert!(entry.since_days.is_none(), "{kind:?} gained a lookback"); + assert!( + entry.max_prs.is_none(), + "{kind:?} gained a pull-request cap" + ); + } +} + +#[test] +fn applying_the_defaults_twice_changes_nothing() { + let mut once = entry_of_kind(SourceKind::GithubRepo); + apply_kind_defaults(&mut once); + let mut twice = once.clone(); + apply_kind_defaults(&mut twice); + assert_eq!(twice.max_prs, once.max_prs); + assert_eq!(twice.max_issues, once.max_issues); + assert_eq!(twice.max_commits, once.max_commits); +} diff --git a/crates/tinymemory-tinycortex/src/conformance_test.rs b/crates/tinymemory-tinycortex/src/conformance_test.rs index 51483a56..067ddf6f 100644 --- a/crates/tinymemory-tinycortex/src/conformance_test.rs +++ b/crates/tinymemory-tinycortex/src/conformance_test.rs @@ -11,7 +11,7 @@ //! `tinycortex::memory::Memory` backend. It needs nothing but the backend, so //! the suite runs against it here with the engine's own `InMemoryMemoryStore`. //! -//! [`crate::engine::TinycortexProvider`] serves all eighteen families, and +//! [`crate::engine::TinycortexProvider`] serves all twenty families, and //! needs a `MemoryClient` — which needs the host's process-global seams //! (`set_embedding_host` and friends) installed before it will open. A test //! that installs a process global is order-dependent, which `AGENTS.md` rules diff --git a/crates/tinymemory-tinycortex/src/engine/mod.rs b/crates/tinymemory-tinycortex/src/engine/mod.rs index db4e404b..dd3c92db 100644 --- a/crates/tinymemory-tinycortex/src/engine/mod.rs +++ b/crates/tinymemory-tinycortex/src/engine/mod.rs @@ -43,13 +43,17 @@ use tinymemory_api::provider::types::{ use tinymemory_api::provider::types::{ChangeKind, DiffReport, SnapshotRef, SourceChange}; use tinymemory_api::provider::{ AddressBookSeedOutcome, ChunkDetail, ChunkEmbedding, ChunkListRow, ChunkQuery, - ConversationSegment, CoverWindowQuery, EntityMatch, EpisodicEvent, EpisodicTurn, EventKind, - FacetType, FastRetrieveQuery, MemoryChunks, MemoryCore, MemoryDiff, MemoryDocuments, - MemoryEntities, MemoryEpisodic, MemoryGoals, MemoryGraph, MemoryIngest, MemoryMaintenance, - MemoryPeople, MemoryPortability, MemoryProfile, MemoryProvider, MemoryRecall, MemoryRetrieval, - MemorySourceSink, MemoryToolMemory, MemoryTree, PersonHandle, PersonInteraction, PersonRecord, - PersonScore, ProfileFacet, RankedPerson, ResolvedPerson, RetrievalHit, RetrievalResponse, - SourceRetrievalQuery, SourceTotal, UserState, + CodingSessionIngestReport, CodingSessionIngestRequest, CodingSessionSource, + ConversationSegment, CoverWindowQuery, DegradedCapabilities, Diagnosis, DiagnosisCounters, + DiagnosisFailure, DiagnosisStage, EntityMatch, EpisodicEvent, EpisodicTurn, EventKind, + FacetType, FastRetrieveQuery, MemoryChunks, MemoryCodingSessions, MemoryCore, MemoryDiff, + MemoryDocuments, MemoryEntities, MemoryEpisodic, MemoryGoals, MemoryGraph, MemoryIngest, + MemoryMaintenance, MemoryPeople, MemoryPortability, MemoryProfile, MemoryProvider, + MemoryRecall, MemoryRetrieval, MemorySourceSink, MemorySourceSync, MemoryToolMemory, + MemoryTree, PersonHandle, PersonInteraction, PersonRecord, PersonScore, ProfileFacet, + RankedPerson, RawArchiveCoverage, RawRebuildOutcome, ResolvedPerson, RetrievalHit, + RetrievalResponse, SourceRetrievalQuery, SourceSyncState, SourceSyncStatus, SourceTotal, + SyncAuditEntry, SyncFreshness, SyncRunOutcome, UserState, }; use tinymemory_api::recall::OwnedRecallOpts; use tinymemory_api::tool_memory::ToolMemoryRule; @@ -1216,6 +1220,42 @@ impl MemoryTree for TinycortexProvider { }) .collect()) } + + async fn flush_source_tree(&self, source_scope: &str) -> Result { + let scope = source_scope.to_string(); + // The lookup is synchronous SQLite and it also (re)writes the source's + // `_source.md` mirror, so it goes to a blocking thread like every other + // synchronous read in this file. It creates the tree when the scope has + // none, which is what makes the member idempotent rather than a probe + // for which scopes exist. + let tree = blocking(self.config.clone(), "open the source tree", move |config| { + tinymemory_core::tree_source::get_or_create_source_tree(config, &scope) + }) + .await?; + + // The labelling policy comes from the tree's own kind and scope, which + // is the reason the contract passes a scope rather than a namespace: + // the caller has no way to make this choice and should not be making + // it. `from_tree` reads the kind off the row just fetched, so a topic + // or global tree reached through this member still gets its own + // policy rather than the source default. + let strategy = + tinymemory_core::tree::tree::TreeFactory::from_tree(&tree).label_strategy(&self.config); + + // Seal *and* cascade, in one call: `force_flush_tree` cascades from + // level zero, so the leaves it seals are rolled up in the same pass. + // Stopping after the seal would leave a tier of leaves under no + // summary, which every structural query reads as an empty tree. + let sealed = tinymemory_core::tree::tree::flush::force_flush_tree( + &self.config, + &tree.id, + None, + &strategy, + ) + .await + .map_err(|error| Self::other("flush the source tree", error))?; + Ok(u64::try_from(sealed.len()).unwrap_or(u64::MAX)) + } } /// Validate entity-kind wire strings and re-emit them in the index's spelling. @@ -2153,6 +2193,68 @@ impl MemoryMaintenance for TinycortexProvider { .collect(), }) } + + /// The same pass [`MemoryMaintenance::doctor`] runs, reported in full. + /// + /// Both members call `async_run_doctor` and neither runs it twice for the + /// other: `doctor` throws away the classification, the degradation flags + /// and the counters to fit the family's uniform report, and this one keeps + /// them. That is the whole difference, and it is why the pair is not a + /// duplicate — the engine work is one call, and the two projections have + /// different readers. + /// + /// It cannot fail. `async_run_doctor` is best-effort by construction — + /// counter reads that error degrade to zero, and a panicking blocking task + /// still yields a shaped report with a transient cause — so there is no + /// error path to map. `Ok` is the honest return, not a swallowed failure. + async fn diagnose(&self) -> Result { + let report = tinymemory_core::tree::health::async_run_doctor(&self.config).await; + Ok(Diagnosis { + healthy: report.healthy, + stages: report + .stages + .into_iter() + .map(|stage| DiagnosisStage { + stage: stage.stage, + ok: stage.ok, + failure: stage.failure.as_ref().map(diagnosis_failure), + note: stage.note, + }) + .collect(), + first_blocking_cause: report.first_blocking_cause.as_ref().map(diagnosis_failure), + degraded: DegradedCapabilities { + semantic_recall: report.degraded.semantic_recall, + structure: report.degraded.structure, + storage: report.degraded.storage, + cause: report.degraded.cause.as_ref().map(diagnosis_failure), + }, + counters: DiagnosisCounters { + total_chunks: report.counters.total_chunks, + jobs_ready: report.counters.jobs_ready, + jobs_running: report.counters.jobs_running, + jobs_failed: report.counters.jobs_failed, + extraction_coverage: report.counters.extraction_coverage, + }, + }) + } +} + +/// Carry one engine pipeline failure across as the contract's own shape. +/// +/// The codes and classes cross as **strings**, and the strings are the +/// engine's own `as_str` rather than anything invented here: the frontend +/// resolves `remediation_key` to localised text and compares `code` for +/// equality, so a re-spelling on this side would silently stop matching the +/// keys that already exist. `class` is always `Some` because this engine always +/// derives one from the code; the contract keeps it optional for an engine that +/// classifies a cause without deciding a retry policy for it. +fn diagnosis_failure(failure: &tinymemory_core::tree::health::PipelineFailure) -> DiagnosisFailure { + DiagnosisFailure { + code: failure.code.as_str().to_string(), + class: Some(failure.class.as_str().to_string()), + remediation_key: failure.remediation_key.clone(), + detail: failure.detail.clone(), + } } #[async_trait] @@ -2224,6 +2326,363 @@ impl MemoryProvider for TinycortexProvider { fn as_episodic(&self) -> Option<&dyn MemoryEpisodic> { Some(self) } + fn as_source_sync(&self) -> Option<&dyn MemorySourceSync> { + Some(self) + } + fn as_coding_sessions(&self) -> Option<&dyn MemoryCodingSessions> { + Some(self) + } +} + +// ── Source sync ────────────────────────────────────────────────────────────── +// +// Everything below delegates to `tinymemory_core`'s sync layer, which is where +// the pipelines, the cursor store and the audit log already live. Nothing here +// re-implements a fetch; this file's whole job is to say the same things in the +// contract's vocabulary. +// +// The conversions destructure rather than round-trip through `Self::cross`, for +// the reason the People section below gives at length: a serde round-trip +// agrees only while the field *names* agree on both sides, and it fails at +// runtime rather than at compile time when they stop. + +/// Toolkits with no native pipeline are refused before anything is dispatched. +/// +/// The pipeline builder refuses them too, but as a `PipelineFailure` carrying a +/// message — and by the time it does, this adapter can no longer tell "you +/// asked for a provider that does not exist" apart from "the provider failed". +/// The contract promises [`MemoryError::Invalid`] for the first, and on a call +/// that costs money the difference decides whether a caller retries. +fn ensure_syncable_toolkit(toolkit: &str) -> Result<(), MemoryError> { + if tinymemory_core::sync::pipelines::host::is_composio_toolkit_syncable(toolkit) { + return Ok(()); + } + Err(MemoryError::Invalid(format!( + "memory sync has no pipeline for toolkit '{toolkit}'" + ))) +} + +/// Carry one audit row across as the contract's own shape. +/// +/// Field-for-field, and the field names are identical on both sides because +/// the contract copied the driver's on-disk format deliberately — see +/// `tinymemory_bus::provider::sync::SyncAuditEntry`. The `estimated_cost_usd` +/// this carries was priced when the row was written, which is why +/// `estimate_sync_cost_usd` has to answer from the same constants rather than +/// letting a caller re-derive it. +fn audit_entry(entry: tinymemory_core::sync::audit::SyncAuditEntry) -> SyncAuditEntry { + let tinymemory_core::sync::audit::SyncAuditEntry { + timestamp, + source_id, + source_kind, + scope, + items_fetched, + batches, + input_tokens, + output_tokens, + estimated_cost_usd, + composio_actions_called, + composio_cost_usd, + actual_charged_usd, + duration_ms, + success, + error, + } = entry; + SyncAuditEntry { + timestamp, + source_id, + source_kind, + scope, + items_fetched, + batches, + input_tokens, + output_tokens, + estimated_cost_usd, + composio_actions_called, + composio_cost_usd, + actual_charged_usd, + duration_ms, + success, + error, + } +} + +/// How many audit rows one read may return when the caller names no limit. +/// +/// The log is append-only for the life of a workspace, so "all of it" is not a +/// bound. A thousand rows is far more than any surface renders and still fits a +/// frame with room to spare; a caller wanting a longer history asks for it and +/// is refused by the module's size check rather than by silence. +const DEFAULT_AUDIT_ROWS: usize = 1_000; + +/// The ceiling a caller's own `limit` is clamped to. +const MAX_AUDIT_ROWS: usize = 10_000; + +#[async_trait] +impl MemorySourceSync for TinycortexProvider { + async fn run_connection_sync( + &self, + toolkit: &str, + connection_id: &str, + ) -> Result { + ensure_syncable_toolkit(toolkit)?; + // The registry lookup, the per-source budgets and the pipeline dispatch + // all happen inside this call — which is why the contract carries no + // budget arguments: they are already recorded against the source. + let outcome = tinymemory_core::tinycortex::run_composio_connection( + toolkit, + connection_id, + &self.config, + ) + .await + .map_err(|failure| { + // The usage travels in the message rather than being dropped: a run + // that failed after calling four provider actions and spending real + // money has to say so somewhere, and `IngestOutcome`-style partial + // success is not available on an error path. + MemoryError::Other(anyhow::anyhow!( + "sync {toolkit} connection: {} (actions_called={}, provider_cost_usd={})", + failure.message, + failure.actions_called, + failure.provider_cost_usd + )) + })?; + Ok(SyncRunOutcome { + records_ingested: outcome.records_ingested, + more_pending: outcome.more_pending, + actions_called: outcome.actions_called, + provider_cost_usd: outcome.provider_cost_usd, + note: outcome.note, + }) + } + + async fn source_sync_state( + &self, + toolkit: &str, + connection_id: &str, + ) -> Result, MemoryError> { + use tinymemory_core::sync::composio::providers::sync_state::{SyncState, STATE_NAMESPACE}; + + // Read the row rather than calling `SyncState::load`, which materialises + // a fresh default when nothing is persisted. That default is right for a + // *run* — it is the state a first sync starts from — and wrong here: the + // contract distinguishes "never synced" from "synced and holding no + // cursor", and `load` cannot. + // + // The namespace and the key come from the engine's own constant and its + // own `key`, so this read cannot address a different row than the writes + // do; a literal here would be a second spelling of a durable key. + let key = SyncState::key(toolkit, connection_id); + let stored = self + .client + .kv_get(Some(STATE_NAMESPACE), &key) + .await + .map_err(|error| Self::other("read composio sync state", error))?; + let Some(stored) = stored else { + return Ok(None); + }; + let state: SyncState = serde_json::from_value(stored) + .map_err(|error| Self::other("decode composio sync state", error))?; + + // `remaining()` applies the engine's own day-rollover rule, so a budget + // last written yesterday reads as fully available today. Deriving the + // used count from it rather than from `requests_used` keeps that rule in + // one place — a second date comparison here would show yesterday's spend + // as today's the moment the two disagreed about what a day is. + let limit = state.daily_budget.limit; + let used = limit.saturating_sub(state.daily_budget.remaining()); + Ok(Some(SourceSyncState { + toolkit: state.toolkit, + connection_id: state.connection_id, + cursor: state.cursor, + synced_item_count: u64::try_from(state.synced_ids.len()).unwrap_or(u64::MAX), + last_seen_id: state.last_seen_id, + last_sync_at_ms: state.last_sync_at_ms, + daily_requests_used: used, + daily_request_limit: limit, + })) + } + + async fn sync_audit_log( + &self, + limit: Option, + ) -> Result, MemoryError> { + // Clamped rather than trusted: `limit` reaches this from an RPC + // argument, and the log grows without bound, so an unclamped read is a + // response size a caller chooses. + let take = limit.unwrap_or(DEFAULT_AUDIT_ROWS).min(MAX_AUDIT_ROWS); + let entries = blocking( + self.config.clone(), + "read the sync audit log", + move |config| { + // Already newest-first: the reader reverses the append-only file. + Ok(tinymemory_core::tinycortex::read_audit_log(config)) + }, + ) + .await?; + Ok(entries.into_iter().take(take).map(audit_entry).collect()) + } + + async fn estimate_sync_cost_usd( + &self, + input_tokens: u64, + output_tokens: u64, + ) -> Result { + // The one place these constants are read from outside the audit writer. + // Pure arithmetic, so no blocking hop and no failure path. + Ok(tinymemory_core::tinycortex::estimate_cost_usd( + input_tokens, + output_tokens, + )) + } + + async fn sync_statuses(&self) -> Result, MemoryError> { + let statuses = blocking(self.config.clone(), "list sync statuses", move |config| { + // `engine_config` is the engine's `MemoryConfig` built from the host + // config this driver already holds. It is built *here*, inside the + // driver, which is the whole point: the caller used to build it and + // pass it in, which meant the caller had to name an engine type. + let engine = tinymemory_core::tinycortex::engine_config(config); + tinycortex::memory::sync::list_sync_statuses(&engine) + }) + .await?; + Ok(statuses + .into_iter() + .map(|status| SourceSyncStatus { + provider: status.provider, + chunks_synced: status.chunks_synced, + chunks_pending: status.chunks_pending, + batch_total: status.batch_total, + batch_processed: status.batch_processed, + last_chunk_at_ms: status.last_chunk_at_ms, + freshness: match status.freshness { + tinycortex::memory::sync::FreshnessLabel::Active => SyncFreshness::Active, + tinycortex::memory::sync::FreshnessLabel::Recent => SyncFreshness::Recent, + tinycortex::memory::sync::FreshnessLabel::Idle => SyncFreshness::Idle, + }, + }) + .collect()) + } + + async fn raw_archive_coverage( + &self, + tree_scope: &str, + archive_source_id: &str, + ) -> Result { + let tree_scope = tree_scope.to_string(); + let archive_source_id = archive_source_id.to_string(); + let coverage = blocking( + self.config.clone(), + "scan raw archive coverage", + move |config| { + tinymemory_core::tinycortex::raw_coverage(config, &tree_scope, &archive_source_id) + }, + ) + .await?; + // The engine's scan carries each pending file's absolute path inside + // this driver's content vault. Only the count crosses — a path describes + // storage layout no caller may depend on, and the repair takes the same + // scope rather than a file list, so nothing downstream can use one. + Ok(RawArchiveCoverage { + total: u64::try_from(coverage.total).unwrap_or(u64::MAX), + covered: u64::try_from(coverage.covered).unwrap_or(u64::MAX), + pending: u64::try_from(coverage.pending.len()).unwrap_or(u64::MAX), + }) + } + + async fn rebuild_from_raw_archive( + &self, + tree_scope: &str, + archive_source_id: &str, + ) -> Result { + // Async rather than `blocking`: the rebuild summarises through the host + // summariser, so it awaits inference between batches. + let outcome = tinymemory_core::tinycortex::rebuild_tree_from_raw( + &self.config, + tree_scope, + archive_source_id, + ) + .await + .map_err(|error| Self::other("rebuild the tree from its raw archive", error))?; + Ok(RawRebuildOutcome { + files_read: u64::try_from(outcome.files_read).unwrap_or(u64::MAX), + batches: u64::try_from(outcome.batches).unwrap_or(u64::MAX), + input_tokens: outcome.input_tokens, + output_tokens: outcome.output_tokens, + estimated_cost_usd: outcome.estimated_cost_usd, + actual_charged_usd: outcome.actual_charged_usd, + }) + } +} + +// ── Coding sessions ────────────────────────────────────────────────────────── + +#[async_trait] +impl MemoryCodingSessions for TinycortexProvider { + async fn coding_session_status(&self) -> Result, MemoryError> { + // A bounded `walkdir` over two session roots plus a parse of each file: + // synchronous filesystem work, so it hops off the executor like every + // other synchronous read here. It takes no config — the roots come from + // the environment, driver-side, which is why no path appears in the + // contract. + let statuses = + tokio::task::spawn_blocking(tinymemory_core::tinycortex::coding_session_status) + .await + .map_err(|error| Self::other("scan coding sessions", error))?; + Ok(statuses + .into_iter() + .map(|status| CodingSessionSource { + kind: status.kind, + available: status.available, + session_files: status.session_files, + evidence_units: status.evidence_units, + invalid_files: status.invalid_files, + scan_truncated: status.scan_truncated, + }) + .collect()) + } + + async fn ingest_coding_sessions( + &self, + request: CodingSessionIngestRequest, + ) -> Result { + let config = self.config.clone(); + // The persona pipeline holds borrowed path state across its awaits and + // is therefore **not** `Send`, while this trait's future must be. So the + // future is built and driven inside a blocking worker, on the ambient + // runtime, and only the finished report crosses back. Dropping the + // `spawn_blocking` and awaiting directly does not compile; replacing it + // with a second runtime would give the pipeline a different reactor from + // the one its inference calls are registered on. + let handle = tokio::runtime::Handle::current(); + let engine_request = tinymemory_core::tinycortex::CodingSessionIngestRequest { + backfill: request.backfill, + // Clamped driver-side, as the contract says: `max_sessions` reaches + // here from an RPC argument, and each session is one or more + // sequential model calls. + max_sessions: request.max_sessions, + }; + let response = tokio::task::spawn_blocking(move || { + handle.block_on(tinymemory_core::tinycortex::ingest_coding_sessions( + &config, + engine_request, + )) + }) + .await + .map_err(|error| Self::other("join coding-session ingestion", error))? + .map_err(|error| Self::other("ingest coding sessions", error))?; + Ok(CodingSessionIngestReport { + mode: response.mode, + files_seen: response.files_seen, + sessions_processed: response.sessions_processed, + sessions_skipped: response.sessions_skipped, + sessions_failed: response.sessions_failed, + evidence_units: response.evidence_units, + observations: response.observations, + budget_hit: response.budget_hit, + pack_path: response.pack_path, + }) + } } // ── People ─────────────────────────────────────────────────────────────────── diff --git a/crates/tinymemory-tinycortex/src/engine/test.rs b/crates/tinymemory-tinycortex/src/engine/test.rs index 78e1b0b7..1caeb188 100644 --- a/crates/tinymemory-tinycortex/src/engine/test.rs +++ b/crates/tinymemory-tinycortex/src/engine/test.rs @@ -23,8 +23,9 @@ use tinymemory_api::provider::types::IngestItem; use tinymemory_api::types::MemoryTaint; use super::{ - advertised_capabilities, facet_type_to_engine, handle_to_contract, handle_to_engine, - parse_person_id, scope_to_engine, validate_ingest_item, EngineRuntimeConfig, + advertised_capabilities, audit_entry, diagnosis_failure, ensure_syncable_toolkit, + facet_type_to_engine, handle_to_contract, handle_to_engine, parse_person_id, scope_to_engine, + validate_ingest_item, EngineRuntimeConfig, }; fn ingest_item(content: &str, mime: Option<&str>, taint: MemoryTaint) -> IngestItem { @@ -333,3 +334,92 @@ fn people_profile_and_scope_boundary_conversions_are_total_and_fail_closed() { 2 ); } + +#[test] +fn an_unsyncable_toolkit_is_refused_before_anything_is_dispatched() { + // The pipeline builder refuses these too, but as a message inside a + // `PipelineFailure` — by which point this adapter can no longer tell + // "there is no such provider" from "the provider failed". On a call that + // spends money, the caller acts differently on each. + let error = ensure_syncable_toolkit("definitely-not-a-provider") + .expect_err("a toolkit with no pipeline must be refused"); + assert!( + matches!(error, MemoryError::Invalid(_)), + "expected Invalid, got {error:?}" + ); + + // Every toolkit the engine-free pipelines actually build, and the + // case/whitespace forms a caller may send: the gate normalises, so a + // padded slug must not be refused here and then accepted downstream. + for toolkit in ["gmail", "Slack", " github ", "notion", "linear", "clickup"] { + assert!( + ensure_syncable_toolkit(toolkit).is_ok(), + "`{toolkit}` has a native pipeline and must not be refused" + ); + } +} + +#[test] +fn a_pipeline_failure_crosses_with_the_engines_own_wire_strings() { + // The frontend resolves `remediation_key` to localised text and compares + // `code` for equality, so a re-spelling on this side stops matching keys + // that already exist. Pinned against the engine's own `as_str`. + use tinymemory_core::tree::health::{FailureCode, PipelineFailure}; + + let failure = PipelineFailure::new(FailureCode::EmbeddingsUnconfigured); + let crossed = diagnosis_failure(&failure); + assert_eq!(crossed.code, FailureCode::EmbeddingsUnconfigured.as_str()); + assert_eq!( + crossed.class.as_deref(), + Some(FailureCode::EmbeddingsUnconfigured.class().as_str()) + ); + assert_eq!( + crossed.remediation_key, + FailureCode::EmbeddingsUnconfigured.remediation_key() + ); + assert_eq!(crossed.detail, None); +} + +#[test] +fn an_audit_row_crosses_field_for_field_and_keeps_its_price() { + // The row was priced when it was written. Carrying `estimated_cost_usd` + // verbatim — rather than re-deriving it from the token counts on this side + // — is what keeps a historical total summed at the rate it was recorded at. + let entry = tinymemory_core::sync::audit::SyncAuditEntry { + timestamp: chrono::DateTime::::from_timestamp(1_700_000_000, 0) + .expect("valid timestamp"), + source_id: "composio:gmail:conn-1".to_string(), + source_kind: "composio".to_string(), + scope: "gmail:conn-1".to_string(), + items_fetched: 12, + batches: 2, + input_tokens: 1_000, + output_tokens: 100, + estimated_cost_usd: 0.42, + composio_actions_called: 4, + composio_cost_usd: 0.02, + actual_charged_usd: None, + duration_ms: 4_200, + success: true, + error: None, + }; + let crossed = audit_entry(entry); + assert_eq!(crossed.source_id, "composio:gmail:conn-1"); + assert_eq!(crossed.items_fetched, 12); + assert!((crossed.estimated_cost_usd - 0.42).abs() < 1e-9); + // The contract's own arithmetic, over the same fields the engine's copy + // uses: estimate when nothing was charged, plus Composio's action cost. + assert!((crossed.effective_cost_usd() - 0.44).abs() < 1e-9); + assert!(crossed.success); + assert_eq!(crossed.error, None); +} + +#[test] +fn the_two_new_families_are_advertised_by_the_full_engine() { + // The families exist because the driver serves them; advertising is what + // makes a host register their RPC surface, and `audit_provider` fails the + // bind if the accessor and the advertisement disagree. + let caps = advertised_capabilities(); + assert!(caps.contains(Capability::SourceSync)); + assert!(caps.contains(Capability::CodingSessions)); +} diff --git a/crates/tinymemory-tinycortex/tests/full_provider_conformance.rs b/crates/tinymemory-tinycortex/tests/full_provider_conformance.rs index 2b276062..95b487cb 100644 --- a/crates/tinymemory-tinycortex/tests/full_provider_conformance.rs +++ b/crates/tinymemory-tinycortex/tests/full_provider_conformance.rs @@ -1,4 +1,4 @@ -//! The conformance suite over the FULL eighteen-family driver (#18 §E1/§E3). +//! The conformance suite over the FULL twenty-family driver (#18 §E1/§E3). //! //! `conformance_test.rs` (in-lib) covers `crate::provider` — the mandatory //! three families over any engine backend. This target covers diff --git a/crates/tinymemory/examples/tinycortex.rs b/crates/tinymemory/examples/tinycortex.rs index 68f1e5d2..3dace678 100644 --- a/crates/tinymemory/examples/tinycortex.rs +++ b/crates/tinymemory/examples/tinycortex.rs @@ -10,7 +10,7 @@ //! one proves the first real engine binds the same way and actually retains. //! The backend is the engine's own in-memory store — a complete embedded //! setup for the mandatory three families: no workspace, no host seams. (The -//! full eighteen-family `TinycortexProvider` additionally needs the host +//! full twenty-family `TinycortexProvider` additionally needs the host //! seams installed; `crates/tinymemory-tinycortex/tests/full_provider_conformance.rs` //! is the minimal working wiring for that.) From d8037263321b1583d49ed30dde03c451734746e1 Mon Sep 17 00:00:00 2001 From: Shanu Date: Tue, 25 Aug 2026 16:29:24 +0530 Subject: [PATCH 2/2] Name the trait in a module-level doc link, not Self MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `Self` has no meaning in a `//!` comment — it is module documentation, not an impl — so rustdoc could not resolve the target and the docs lane fails the whole crate under `-D warnings`. --- crates/tinymemory-api/src/provider/sync.rs | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/crates/tinymemory-api/src/provider/sync.rs b/crates/tinymemory-api/src/provider/sync.rs index 5f626552..144353dc 100644 --- a/crates/tinymemory-api/src/provider/sync.rs +++ b/crates/tinymemory-api/src/provider/sync.rs @@ -35,9 +35,9 @@ //! //! ## Neither does configuration //! -//! [`Self::run_connection_sync`] takes no budget arguments. The per-source -//! caps — item limits, depth windows, token and cost ceilings — live in the -//! registry the driver already reads, so passing them would be a caller +//! [`MemorySourceSync::run_connection_sync`] takes no budget arguments. The +//! per-source caps — item limits, depth windows, token and cost ceilings — live +//! in the registry the driver already reads, so passing them would be a caller //! restating something the driver knows, with two sources of truth for a limit //! that costs money when it is wrong.